2024-03-06T07:11:07.091 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:11:07.091 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583073 --verbose --owner scheduled_leonidus@teuthology --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583073/orig.config.yaml 2024-03-06T07:11:07.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:11:07.226 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583073 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583073' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCq8nxrAaiDTym51uoTjdO1MXuKLphpyesa1QR7FnsXDWN8xTvy9X6wuKhLYZBNaX5iL+eUChDM82aLwP418bSE= smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZFHy6EPcycZSqUJx74UYkgnDWoG3lbn8x5b+vVB4Pyvl/bDHZ3SJz1AzMdVn6O6ysZuxFyBmf4D8M1AM4pJ/k= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:11:07.226 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:11:07.227 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:11:07.227 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:11:07.228 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:11:07.257 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:11:07.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:11:07.257 INFO:teuthology.packaging:ref: None 2024-03-06T07:11:07.257 INFO:teuthology.packaging:tag: None 2024-03-06T07:11:07.257 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:11:07.257 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:11:07.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:11:07.414 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:11:07.417 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:11:07.424 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:11:07.424 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:11:07.484 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:11:07.501 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:11:07.517 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:11:07.542 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583073', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:45.489917', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCq8nxrAaiDTym51uoTjdO1MXuKLphpyesa1QR7FnsXDWN8xTvy9X6wuKhLYZBNaX5iL+eUChDM82aLwP418bSE='} 2024-03-06T07:11:07.562 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583073', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:45.491663', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZFHy6EPcycZSqUJx74UYkgnDWoG3lbn8x5b+vVB4Pyvl/bDHZ3SJz1AzMdVn6O6ysZuxFyBmf4D8M1AM4pJ/k='} 2024-03-06T07:11:07.562 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:11:07.569 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:11:07.569 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:11:07.569 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:11:07.668 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb66c0c4f70>, signals=[15]) 2024-03-06T07:11:07.668 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:11:07.675 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:11:07.675 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:11:07.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:11:07.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:11:07.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:11:07.899 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:11:07.905 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-03-06T07:11:07.922 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-03-06T07:11:07.923 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:11:07.979 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-03-06T07:11:07.980 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:11:07.980 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:11:07.980 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:11:07.980 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:11:07.980 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:11:07.980 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:11:07.981 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-03-06T07:11:08.004 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-03-06T07:11:08.021 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-03-06T07:11:08.022 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-03-06T07:11:08.078 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:11:08.079 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:11:08.080 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-03-06T07:11:08.104 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:11:08.113 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:11:08.141 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:11:08.142 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:11:08.144 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:11:08.161 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:11:08.175 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:11:08.175 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:11:08.201 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:11:08.233 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:11:08.289 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:11:08.304 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-06T07:11:08.304 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-06T07:11:08.304 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-06T07:11:08.305 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:11:08.305 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:11:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:11:08.322 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:11:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:11:08.326 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:11:08.653 DEBUG:teuthology.task.kernel:sha1 for client.0 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:11:08.653 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:11:08.654 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-06T07:11:08.655 DEBUG:teuthology.task.kernel:sha1 for client.1 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:11:08.655 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:11:08.655 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-03-06T07:11:08.672 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:11:08.673 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:11:08.673 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:11:08.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum upgrade -y linux-firmware 2024-03-06T07:11:08.675 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:11:08.675 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:11:08.675 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:11:08.675 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum upgrade -y linux-firmware 2024-03-06T07:11:11.401 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.7 MB 00:01 2024-03-06T07:11:11.409 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 5.8 MB/s | 6.7 MB 00:01 2024-03-06T07:11:13.300 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-06T07:11:13.301 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-06T07:11:14.283 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-06T07:11:14.416 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-06T07:11:17.650 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 19 MB 00:02 2024-03-06T07:11:18.009 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-03-06T07:11:22.259 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-06T07:11:22.502 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.1 MB 00:00 2024-03-06T07:11:23.760 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2024-03-06T07:11:24.111 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-06T07:11:27.431 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 21 MB 00:03 2024-03-06T07:11:32.595 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-06T07:11:34.943 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-06T07:11:34.950 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 1 Package 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 375 M 2024-03-06T07:11:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-06T07:11:37.927 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 21 MB 00:13 2024-03-06T07:11:43.119 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-06T07:11:43.196 INFO:teuthology.orchestra.run.smithi032.stdout:linux-firmware-20240219-143.el9.noarch.rpm 45 MB/s | 375 MB 00:08 2024-03-06T07:11:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:11:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:Total 45 MB/s | 375 MB 00:08 2024-03-06T07:11:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-06T07:11:43.298 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-06T07:11:43.298 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-06T07:11:45.544 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-03-06T07:11:45.548 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-03-06T07:11:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-03-06T07:11:45.548 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 1 Package 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:11:45.549 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 375 M 2024-03-06T07:11:45.550 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-03-06T07:11:45.791 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-06T07:11:45.792 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-06T07:11:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-06T07:11:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:11:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:11:54.113 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:11:54.113 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:11:54.523 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:11:54.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:11:54.523 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-03-06T07:11:54.523 INFO:teuthology.orchestra.run.smithi032.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:11:54.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:11:54.523 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-06T07:11:54.824 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:11:54.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:11:54.987 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:11:54.987 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:11:54.990 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi032.front.sepia.ceph.com to complete... 2024-03-06T07:12:18.625 INFO:teuthology.orchestra.run.smithi039.stdout:linux-firmware-20240219-143.el9.noarch.rpm 11 MB/s | 375 MB 00:33 2024-03-06T07:12:18.626 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:12:18.626 INFO:teuthology.orchestra.run.smithi039.stdout:Total 11 MB/s | 375 MB 00:33 2024-03-06T07:12:18.626 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-03-06T07:12:18.713 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-03-06T07:12:18.713 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-03-06T07:12:21.202 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-03-06T07:12:21.202 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-03-06T07:12:27.753 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-03-06T07:12:27.979 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:28.077 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:29.497 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:29.497 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:12:29.498 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-03-06T07:12:29.498 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:12:29.498 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:12:29.498 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-03-06T07:12:29.793 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:12:29.793 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:12:30.021 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:12:30.031 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:12:30.035 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi039.front.sepia.ceph.com to complete... 2024-03-06T07:13:15.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06 07:13:15 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:13:15.507 INFO:teuthology.orchestra.run.smithi032.stderr:FINISHED --2024-03-06 07:13:15-- 2024-03-06T07:13:15.507 INFO:teuthology.orchestra.run.smithi032.stderr:Total wall clock time: 1m 20s 2024-03-06T07:13:15.507 INFO:teuthology.orchestra.run.smithi032.stderr:Downloaded: 1 files, 320M in 1m 20s (4.00 MB/s) 2024-03-06T07:13:15.510 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:13:15.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:13:17.542 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying... ######################################## 2024-03-06T07:13:17.604 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing... ######################################## 2024-03-06T07:13:17.687 INFO:teuthology.orchestra.run.smithi032.stdout:Updating / installing... 2024-03-06T07:13:25.602 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:13:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:14:03.041 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:14:03.041 DEBUG:teuthology.orchestra.run.smithi032:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:14:03.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:14:03.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:14:03.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:14:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:14:03.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:14:03.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:14:03.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:14:03.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:14:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:14:03.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:14:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:14:03.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:14:03.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:14:03.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:14:03.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:14:03.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:14:03.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:14:03.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:14:03.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:14:03.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:14:03.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:14:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:14:03.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:14:03.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:14:03.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:14:03.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:14:03.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:14:03.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:14:03.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:14:03.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:14:03.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:14:03.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:14:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:14:03.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:14:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:14:03.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:14:03.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:14:03.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:14:03.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:14:03.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:14:03.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:14:03.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:14:03.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:14:03.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:14:03.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:14:03.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:14:03.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:14:03.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:14:03.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:14:03.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:14:03.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:14:03.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:14:03.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:14:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:14:03.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:14:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:14:03.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:14:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:14:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:14:03.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:14:03.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:14:03.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:14:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:14:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:14:03.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:14:03.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:14:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:14:03.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:14:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:14:03.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:14:03.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:14:03.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:14:03.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:14:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:14:03.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:14:03.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:14:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:14:03.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:14:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:14:03.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:14:03.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:14:03.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:14:03.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:14:03.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:14:03.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:14:03.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:14:03.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:14:03.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:14:03.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:14:03.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:14:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:14:03.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:14:03.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:14:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:14:03.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:14:03.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:14:03.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:14:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:14:03.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:14:03.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:14:03.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:14:03.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:14:03.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:14:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:14:03.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:14:03.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:14:03.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:14:03.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:14:03.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:14:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:14:03.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:14:03.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:14:03.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:14:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:14:03.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:14:03.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:14:03.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:14:03.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:14:03.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:14:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:14:03.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:14:03.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:14:03.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:14:03.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:14:03.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:14:03.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:14:03.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:14:03.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:14:03.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:14:03.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:14:03.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:14:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:14:03.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:14:03.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:14:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:14:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:14:03.253 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:14:03.253 DEBUG:teuthology.orchestra.run.smithi032:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:14:03.276 INFO:teuthology.orchestra.run.smithi032.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:14:03.276 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:14:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: fi 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-03-06T07:14:03.277 INFO:teuthology.orchestra.run.smithi032.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:14:03.278 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:14:03.278 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:14:03.278 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-03-06T07:14:03.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-03-06T07:14:03.368 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-03-06T07:14:03.368 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-03-06T07:14:03.368 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-03-06T07:14:03.368 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-03-06T07:14:03.368 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-03-06T07:14:03.369 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:14:03.370 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:14:03.370 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:14:03.370 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:14:03.370 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-03-06T07:14:03.370 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:14:03.370 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:14:03.371 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:14:03.371 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:14:03.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:14:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-03-06T07:14:05.405 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:14:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-03-06T07:14:05.449 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-06T07:14:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.iMqUAzS8OB 2024-03-06T07:14:05.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iMqUAzS8OB 2024-03-06T07:14:05.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.iMqUAzS8OB 2024-03-06T07:14:05.677 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.iMqUAzS8OB is 6KB 2024-03-06T07:14:05.727 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.iMqUAzS8OB 2024-03-06T07:14:05.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:14:05.815 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:14:05.816 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:14:05.816 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:14:05.816 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:14:05.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:14:05.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-03-06T07:14:14.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06 07:14:14 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:14:14.810 INFO:teuthology.orchestra.run.smithi039.stderr:FINISHED --2024-03-06 07:14:14-- 2024-03-06T07:14:14.810 INFO:teuthology.orchestra.run.smithi039.stderr:Total wall clock time: 1m 45s 2024-03-06T07:14:14.810 INFO:teuthology.orchestra.run.smithi039.stderr:Downloaded: 1 files, 320M in 1m 44s (3.08 MB/s) 2024-03-06T07:14:14.814 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:14:14.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:14:16.849 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying... ######################################## 2024-03-06T07:14:16.917 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing... ######################################## 2024-03-06T07:14:17.001 INFO:teuthology.orchestra.run.smithi039.stdout:Updating / installing... 2024-03-06T07:14:24.879 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:14:35.981 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-06T07:14:35.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-06T07:14:35.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:03.161 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:15:03.161 DEBUG:teuthology.orchestra.run.smithi039:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:03.242 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:03.242 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:15:03.243 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:15:03.244 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:15:03.245 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:15:03.246 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:15:03.247 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:15:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:15:03.249 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:15:03.250 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:15:03.251 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:15:03.252 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:15:03.253 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:15:03.254 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:15:03.255 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:15:03.256 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:15:03.257 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:15:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:15:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:15:03.260 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:15:03.261 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:15:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:15:03.263 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:15:03.264 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:15:03.265 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:15:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:15:03.267 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:15:03.268 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:15:03.269 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:15:03.270 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:15:03.271 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:15:03.272 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:15:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:15:03.274 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:15:03.275 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:15:03.276 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:15:03.277 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:15:03.278 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:15:03.279 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:15:03.280 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:15:03.281 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:15:03.282 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:15:03.283 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:15:03.284 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:15:03.285 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:15:03.286 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:15:03.287 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:15:03.288 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:15:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:15:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:15:03.291 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:15:03.292 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:15:03.293 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:15:03.294 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:15:03.295 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:15:03.296 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:15:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:15:03.298 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:15:03.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:15:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:15:03.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:15:03.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:15:03.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:15:03.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:15:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:15:03.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:15:03.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:15:03.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:15:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:15:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:15:03.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:15:03.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:15:03.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:15:03.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:15:03.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:15:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:15:03.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:15:03.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:15:03.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:15:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:15:03.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:15:03.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:15:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:15:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:15:03.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:15:03.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:15:03.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:15:03.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:15:03.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:15:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:15:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:15:03.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:15:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:15:03.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:15:03.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:15:03.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:15:03.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:15:03.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:15:03.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:15:03.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:15:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:15:03.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:15:03.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:15:03.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:15:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:15:03.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:15:03.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:15:03.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:15:03.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:15:03.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:15:03.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:15:03.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:15:03.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:15:03.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:15:03.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:15:03.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:15:03.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:15:03.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:15:03.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:15:03.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:15:03.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:15:03.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:15:03.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:15:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:15:03.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:15:03.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:15:03.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:15:03.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:15:03.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:15:03.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:15:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:15:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:15:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:15:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:15:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:15:03.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:15:03.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:15:03.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:15:03.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:15:03.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:15:03.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:15:03.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:15:03.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:15:03.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:15:03.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:15:03.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:15:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:15:03.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:15:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:15:03.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:15:03.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:15:03.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:15:03.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:15:03.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:15:03.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:15:03.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:15:03.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:15:03.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:03.395 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:03.395 DEBUG:teuthology.orchestra.run.smithi039:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:03.420 INFO:teuthology.orchestra.run.smithi039.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:15:03.420 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:03.420 INFO:teuthology.orchestra.run.smithi039.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:03.420 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-03-06T07:15:03.420 INFO:teuthology.orchestra.run.smithi039.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout: fi 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:03.421 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-03-06T07:15:03.422 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-03-06T07:15:03.502 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-03-06T07:15:03.502 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-03-06T07:15:03.502 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-03-06T07:15:03.502 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 68.el9 2024-03-06T07:15:03.502 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8264254 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:15:03.503 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:15:03.504 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:15:03.504 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:15:03.504 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-03-06T07:15:03.504 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:15:03.504 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:15:03.505 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:15:03.505 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:03.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:15:04.109 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-03-06T07:15:05.464 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:15:05.510 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-03-06T07:15:05.511 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-03-06T07:15:05.526 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.3Bc3GvNBBj 2024-03-06T07:15:05.527 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3Bc3GvNBBj 2024-03-06T07:15:05.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.3Bc3GvNBBj 2024-03-06T07:15:05.763 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.3Bc3GvNBBj is 6KB 2024-03-06T07:15:05.812 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.3Bc3GvNBBj 2024-03-06T07:15:05.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:15:05.903 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:15:05.903 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:15:05.903 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:15:05.904 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:15:05.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:06.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-03-06T07:15:36.013 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:15:36.074 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-06T07:15:36.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:15:36.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:45.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-06T07:15:45.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:45.519 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-06T07:15:46.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-06T07:15:46.313 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:15:46.313 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-06T07:15:46.361 INFO:teuthology.orchestra.run.smithi032.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:46.362 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:46.362 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:15:46.362 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:15:47.366 DEBUG:teuthology.parallel:result is None 2024-03-06T07:16:10.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-03-06T07:16:19.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:16:19.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:16:22.752 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-03-06T07:16:34.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:16:34.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:16:37.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-03-06T07:16:52.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:16:52.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:52.825 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:18:10.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:18:10.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:11.198 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-03-06T07:18:11.875 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:18:11.875 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:18:11.875 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-03-06T07:18:11.932 INFO:teuthology.orchestra.run.smithi039.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:18:11.932 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:18:11.932 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:18:11.933 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:18:12.934 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:12.934 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:18:12.942 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:18:12.942 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:18:12.951 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:18:13.007 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:18:13.014 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:18:13.045 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:18:13.045 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:18:13.053 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:18:13.127 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:18:13.134 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:18:13.134 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:18:13.141 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:18:13.292 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:13.334 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:13.355 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:13.409 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:13.414 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:18:13.421 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:18:13.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:18:13.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:18:13.532 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:18:13.546 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:18:13.546 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:18:13.554 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:18:13.611 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:13.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:13.814 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:13.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:14.003 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:18:14.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:18:14.097 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:14.187 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:14.296 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:14.386 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:14.488 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:18:14.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:18:14.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-03-06T07:18:14.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-03-06T07:18:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:18:14.754 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:18:15.145 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:18:15.157 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:18:15.157 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:18:15.196 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:18:15.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-03-06T07:18:15.355 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-03-06T07:18:15.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-03-06T07:18:15.497 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-03-06T07:18:15.501 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:18:15.501 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-03-06T07:18:15.558 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-03-06T07:18:15.559 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-03-06T07:18:15.604 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-03-06T07:18:15.604 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi039.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:18:15.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:18:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:15.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:18:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:15.786 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:18:15.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:18:15.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:18:15.991 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:18:16.001 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:18:16.019 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:18:16.020 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:24:25.278 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2024-03-06T07:24:25.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-06T07:24:25.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:24:25.436 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-06T07:24:25.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-06T07:24:25.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:24:25.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:24:25.741 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-03-06T07:24:25.896 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-06T07:24:25.896 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:24:25.907 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:24:25.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:24:25.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:24:25.915 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:24:25.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:24:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:24:26.023 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:24:26.079 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:24:26.080 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:24:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-03-06T07:24:26.187 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-03-06T07:24:26.222 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-03-06T07:24:26.233 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-03-06T07:24:26.280 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:24:26.291 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:24:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:24:26.348 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:24:26.476 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:24:26.481 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:24:26.486 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:24:26.486 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-06T07:24:26.486 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.486 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.487 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.487 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.490 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:24:26.491 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-03-06T07:24:26.491 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.491 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.491 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.491 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:26.492 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:24:26.501 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:24:26.501 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:24:26.501 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:24:26.501 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:24:26.508 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:24:26.508 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:24:26.508 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:24:26.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:24:26.511 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:24:26.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:24:26.660 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:24:26.660 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:24:26.739 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:24:26.739 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:24:27.010 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:24:27.010 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:24:27.010 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:24:27.095 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:24:27.095 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:24:27.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:24:27.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:24:27.117 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:24:27.195 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:24:27.195 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:24:27.260 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:24:27.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:24:27.395 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:24:27.472 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-03-06T07:24:27.493 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-03-06T07:24:27.565 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-03-06T07:24:27.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-03-06T07:24:27.857 INFO:teuthology.orchestra.run.smithi039.stdout:74 files removed 2024-03-06T07:24:27.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:24:27.934 INFO:teuthology.orchestra.run.smithi032.stdout:74 files removed 2024-03-06T07:24:27.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:24:29.019 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 137 kB/s | 83 kB 00:00 2024-03-06T07:24:29.305 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 103 kB/s | 83 kB 00:00 2024-03-06T07:24:29.837 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-06T07:24:29.981 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-06T07:24:30.720 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-06T07:24:30.882 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-06T07:24:31.340 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:24:31.527 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:24:33.033 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-06T07:24:33.144 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-06T07:24:34.615 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.0 MB 00:01 2024-03-06T07:24:36.587 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-03-06T07:24:41.929 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-06T07:24:42.432 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-06T07:24:48.123 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-03-06T07:24:49.250 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-06T07:24:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-06T07:25:23.075 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 660 kB/s | 20 MB 00:31 2024-03-06T07:25:29.050 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-06T07:25:31.861 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:25:31.863 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:25:31.883 INFO:teuthology.orchestra.run.smithi032.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi032.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:25:31.886 INFO:teuthology.orchestra.run.smithi032.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:25:31.887 INFO:teuthology.orchestra.run.smithi032.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:25:31.890 INFO:teuthology.orchestra.run.smithi032.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:25:31.893 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:25:31.895 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:25:31.994 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-06T07:25:32.023 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-06T07:25:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-06T07:25:32.023 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-06T07:25:32.023 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:25:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:25:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-03-06T07:25:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:25:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-06T07:25:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:25:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:25:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:25:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:25:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:25:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:25:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:25:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout:Install 150 Packages 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 26 Packages 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:25:32.038 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 224 M 2024-03-06T07:25:32.038 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-06T07:25:32.827 INFO:teuthology.orchestra.run.smithi032.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-03-06T07:25:33.285 INFO:teuthology.orchestra.run.smithi032.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.9 MB/s | 888 kB 00:00 2024-03-06T07:25:33.444 INFO:teuthology.orchestra.run.smithi032.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 924 kB/s | 145 kB 00:00 2024-03-06T07:25:34.102 INFO:teuthology.orchestra.run.smithi032.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 3.5 MB/s | 2.3 MB 00:00 2024-03-06T07:25:34.236 INFO:teuthology.orchestra.run.smithi032.stdout:(5/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 3.2 MB/s | 5.3 MB 00:01 2024-03-06T07:25:34.511 INFO:teuthology.orchestra.run.smithi032.stdout:(6/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 10 MB/s | 20 MB 00:01 2024-03-06T07:25:34.603 INFO:teuthology.orchestra.run.smithi032.stdout:(7/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-03-06T07:25:35.103 INFO:teuthology.orchestra.run.smithi032.stdout:(8/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 5.4 MB/s | 4.7 MB 00:00 2024-03-06T07:25:35.195 INFO:teuthology.orchestra.run.smithi032.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 277 kB/s | 25 kB 00:00 2024-03-06T07:25:35.904 INFO:teuthology.orchestra.run.smithi032.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 12 MB/s | 17 MB 00:01 2024-03-06T07:25:35.970 INFO:teuthology.orchestra.run.smithi032.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 3.3 MB/s | 220 kB 00:00 2024-03-06T07:25:36.013 INFO:teuthology.orchestra.run.smithi032.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 656 kB/s | 27 kB 00:00 2024-03-06T07:25:36.171 INFO:teuthology.orchestra.run.smithi032.stdout:(13/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 6.2 MB/s | 9.8 MB 00:01 2024-03-06T07:25:36.188 INFO:teuthology.orchestra.run.smithi032.stdout:(14/176): libcephfs-devel-19.0.0-1739.g1bc250c7 179 kB/s | 31 kB 00:00 2024-03-06T07:25:36.263 INFO:teuthology.orchestra.run.smithi032.stdout:(15/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 8.0 MB/s | 747 kB 00:00 2024-03-06T07:25:36.297 INFO:teuthology.orchestra.run.smithi032.stdout:(16/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.5 MB/s | 163 kB 00:00 2024-03-06T07:25:36.339 INFO:teuthology.orchestra.run.smithi032.stdout:(17/176): librados-devel-19.0.0-1739.g1bc250c7. 1.7 MB/s | 127 kB 00:00 2024-03-06T07:25:36.414 INFO:teuthology.orchestra.run.smithi032.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 4.2 MB/s | 501 kB 00:00 2024-03-06T07:25:36.498 INFO:teuthology.orchestra.run.smithi032.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 546 kB/s | 45 kB 00:00 2024-03-06T07:25:36.590 INFO:teuthology.orchestra.run.smithi032.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.4 MB/s | 131 kB 00:00 2024-03-06T07:25:36.674 INFO:teuthology.orchestra.run.smithi032.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.9 MB/s | 162 kB 00:00 2024-03-06T07:25:36.757 INFO:teuthology.orchestra.run.smithi032.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 3.8 MB/s | 321 kB 00:00 2024-03-06T07:25:37.024 INFO:teuthology.orchestra.run.smithi032.stdout:(23/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 7.5 MB/s | 5.1 MB 00:00 2024-03-06T07:25:37.066 INFO:teuthology.orchestra.run.smithi032.stdout:(24/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 968 kB/s | 298 kB 00:00 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi032.stdout:(25/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.3 MB/s | 99 kB 00:00 2024-03-06T07:25:37.125 INFO:teuthology.orchestra.run.smithi032.stdout:(26/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-06T07:25:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 2.9 MB/s | 171 kB 00:00 2024-03-06T07:25:37.251 INFO:teuthology.orchestra.run.smithi032.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 478 kB/s | 31 kB 00:00 2024-03-06T07:25:37.318 INFO:teuthology.orchestra.run.smithi032.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 2.1 MB/s | 138 kB 00:00 2024-03-06T07:25:37.418 INFO:teuthology.orchestra.run.smithi032.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 9.7 MB/s | 3.1 MB 00:00 2024-03-06T07:25:37.510 INFO:teuthology.orchestra.run.smithi032.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 9.3 MB/s | 1.8 MB 00:00 2024-03-06T07:25:37.594 INFO:teuthology.orchestra.run.smithi032.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 2.9 MB/s | 248 kB 00:00 2024-03-06T07:25:37.669 INFO:teuthology.orchestra.run.smithi032.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 664 kB/s | 49 kB 00:00 2024-03-06T07:25:37.744 INFO:teuthology.orchestra.run.smithi032.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 207 kB/s | 15 kB 00:00 2024-03-06T07:25:37.828 INFO:teuthology.orchestra.run.smithi032.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 3.1 MB/s | 264 kB 00:00 2024-03-06T07:25:38.054 INFO:teuthology.orchestra.run.smithi032.stdout:(36/176): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-06T07:25:38.095 INFO:teuthology.orchestra.run.smithi032.stdout:(37/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 2.7 MB/s | 748 kB 00:00 2024-03-06T07:25:38.121 INFO:teuthology.orchestra.run.smithi032.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 277 kB/s | 18 kB 00:00 2024-03-06T07:25:38.346 INFO:teuthology.orchestra.run.smithi032.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-06T07:25:38.388 INFO:teuthology.orchestra.run.smithi032.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-06T07:25:38.413 INFO:teuthology.orchestra.run.smithi032.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.7 MB/s | 106 kB 00:00 2024-03-06T07:25:38.430 INFO:teuthology.orchestra.run.smithi032.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 507 kB/s | 20 kB 00:00 2024-03-06T07:25:38.464 INFO:teuthology.orchestra.run.smithi032.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 307 kB 00:00 2024-03-06T07:25:38.489 INFO:teuthology.orchestra.run.smithi032.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 580 kB/s | 32 kB 00:00 2024-03-06T07:25:38.723 INFO:teuthology.orchestra.run.smithi032.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-03-06T07:25:38.739 INFO:teuthology.orchestra.run.smithi032.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 73 kB/s | 17 kB 00:00 2024-03-06T07:25:38.776 INFO:teuthology.orchestra.run.smithi032.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 313 kB/s | 10 kB 00:00 2024-03-06T07:25:38.790 INFO:teuthology.orchestra.run.smithi032.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-06T07:25:38.815 INFO:teuthology.orchestra.run.smithi032.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-06T07:25:38.857 INFO:teuthology.orchestra.run.smithi032.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-06T07:25:38.930 INFO:teuthology.orchestra.run.smithi032.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 578 kB 00:00 2024-03-06T07:25:38.958 INFO:teuthology.orchestra.run.smithi032.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 258 kB/s | 24 kB 00:00 2024-03-06T07:25:38.983 INFO:teuthology.orchestra.run.smithi032.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 898 kB/s | 46 kB 00:00 2024-03-06T07:25:39.017 INFO:teuthology.orchestra.run.smithi032.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-06T07:25:39.051 INFO:teuthology.orchestra.run.smithi032.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-06T07:25:39.076 INFO:teuthology.orchestra.run.smithi032.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 607 kB/s | 34 kB 00:00 2024-03-06T07:25:39.226 INFO:teuthology.orchestra.run.smithi032.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-06T07:25:39.310 INFO:teuthology.orchestra.run.smithi032.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-03-06T07:25:39.368 INFO:teuthology.orchestra.run.smithi032.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-06T07:25:39.410 INFO:teuthology.orchestra.run.smithi032.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 9.3 MB/s | 811 kB 00:00 2024-03-06T07:25:39.435 INFO:teuthology.orchestra.run.smithi032.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-03-06T07:25:39.578 INFO:teuthology.orchestra.run.smithi032.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-06T07:25:39.619 INFO:teuthology.orchestra.run.smithi032.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-06T07:25:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-06T07:25:39.812 INFO:teuthology.orchestra.run.smithi032.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-06T07:25:39.836 INFO:teuthology.orchestra.run.smithi032.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 227 kB 00:00 2024-03-06T07:25:39.887 INFO:teuthology.orchestra.run.smithi032.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 500 kB/s | 24 kB 00:00 2024-03-06T07:25:39.912 INFO:teuthology.orchestra.run.smithi032.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 245 kB/s | 18 kB 00:00 2024-03-06T07:25:39.962 INFO:teuthology.orchestra.run.smithi032.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-06T07:25:39.987 INFO:teuthology.orchestra.run.smithi032.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 428 kB/s | 32 kB 00:00 2024-03-06T07:25:40.030 INFO:teuthology.orchestra.run.smithi032.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.3 MB/s | 217 kB 00:00 2024-03-06T07:25:40.147 INFO:teuthology.orchestra.run.smithi032.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-06T07:25:40.172 INFO:teuthology.orchestra.run.smithi032.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-06T07:25:40.205 INFO:teuthology.orchestra.run.smithi032.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 2.1 MB/s | 69 kB 00:00 2024-03-06T07:25:40.230 INFO:teuthology.orchestra.run.smithi032.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 722 kB/s | 41 kB 00:00 2024-03-06T07:25:40.272 INFO:teuthology.orchestra.run.smithi032.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 5.8 MB/s | 238 kB 00:00 2024-03-06T07:25:40.289 INFO:teuthology.orchestra.run.smithi032.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi032.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 5.1 MB/s | 211 kB 00:00 2024-03-06T07:25:40.348 INFO:teuthology.orchestra.run.smithi032.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 829 kB/s | 47 kB 00:00 2024-03-06T07:25:40.390 INFO:teuthology.orchestra.run.smithi032.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi032.stdout:(81/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-06T07:25:40.740 INFO:teuthology.orchestra.run.smithi032.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 115 kB/s | 39 kB 00:00 2024-03-06T07:25:40.757 INFO:teuthology.orchestra.run.smithi032.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-06T07:25:40.791 INFO:teuthology.orchestra.run.smithi032.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.3 MB/s | 215 kB 00:00 2024-03-06T07:25:40.808 INFO:teuthology.orchestra.run.smithi032.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 712 kB/s | 34 kB 00:00 2024-03-06T07:25:40.825 INFO:teuthology.orchestra.run.smithi032.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:25:40.858 INFO:teuthology.orchestra.run.smithi032.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-06T07:25:40.884 INFO:teuthology.orchestra.run.smithi032.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-06T07:25:40.909 INFO:teuthology.orchestra.run.smithi032.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-06T07:25:40.960 INFO:teuthology.orchestra.run.smithi032.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 851 kB/s | 63 kB 00:00 2024-03-06T07:25:40.985 INFO:teuthology.orchestra.run.smithi032.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 597 kB/s | 44 kB 00:00 2024-03-06T07:25:41.010 INFO:teuthology.orchestra.run.smithi032.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-06T07:25:41.044 INFO:teuthology.orchestra.run.smithi032.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-03-06T07:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.7 MB/s | 802 kB 00:00 2024-03-06T07:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 194 kB 00:00 2024-03-06T07:25:41.161 INFO:teuthology.orchestra.run.smithi032.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-06T07:25:41.195 INFO:teuthology.orchestra.run.smithi032.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 241 kB 00:00 2024-03-06T07:25:41.245 INFO:teuthology.orchestra.run.smithi032.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-06T07:25:41.262 INFO:teuthology.orchestra.run.smithi032.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-06T07:25:41.287 INFO:teuthology.orchestra.run.smithi032.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 3.0 MB/s | 124 kB 00:00 2024-03-06T07:25:41.313 INFO:teuthology.orchestra.run.smithi032.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.3 MB/s | 113 kB 00:00 2024-03-06T07:25:41.347 INFO:teuthology.orchestra.run.smithi032.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.2 MB/s | 187 kB 00:00 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi032.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 246 kB/s | 22 kB 00:00 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi032.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.5 MB/s | 245 kB 00:00 2024-03-06T07:25:41.589 INFO:teuthology.orchestra.run.smithi032.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-03-06T07:25:41.714 INFO:teuthology.orchestra.run.smithi032.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-03-06T07:25:41.765 INFO:teuthology.orchestra.run.smithi032.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 396 kB/s | 19 kB 00:00 2024-03-06T07:25:41.798 INFO:teuthology.orchestra.run.smithi032.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.5 MB/s | 309 kB 00:00 2024-03-06T07:25:41.857 INFO:teuthology.orchestra.run.smithi032.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 453 kB/s | 26 kB 00:00 2024-03-06T07:25:41.907 INFO:teuthology.orchestra.run.smithi032.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 990 kB/s | 49 kB 00:00 2024-03-06T07:25:41.957 INFO:teuthology.orchestra.run.smithi032.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-06T07:25:42.041 INFO:teuthology.orchestra.run.smithi032.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-06T07:25:42.233 INFO:teuthology.orchestra.run.smithi032.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-06T07:25:42.367 INFO:teuthology.orchestra.run.smithi032.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-03-06T07:25:42.467 INFO:teuthology.orchestra.run.smithi032.stdout:(115/176): libarrow-9.0.0-10.el9.x86_64.rpm 6.3 MB/s | 4.4 MB 00:00 2024-03-06T07:25:42.484 INFO:teuthology.orchestra.run.smithi032.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 372 kB/s | 43 kB 00:00 2024-03-06T07:25:42.626 INFO:teuthology.orchestra.run.smithi032.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 203 kB/s | 32 kB 00:00 2024-03-06T07:25:42.643 INFO:teuthology.orchestra.run.smithi032.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 95 kB/s | 15 kB 00:00 2024-03-06T07:25:42.677 INFO:teuthology.orchestra.run.smithi032.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-06T07:25:43.202 INFO:teuthology.orchestra.run.smithi032.stdout:(120/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 5.9 MB/s | 47 MB 00:08 2024-03-06T07:25:43.236 INFO:teuthology.orchestra.run.smithi032.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 606 kB/s | 358 kB 00:00 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 513 kB/s | 303 kB 00:00 2024-03-06T07:25:43.303 INFO:teuthology.orchestra.run.smithi032.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 356 kB/s | 35 kB 00:00 2024-03-06T07:25:43.328 INFO:teuthology.orchestra.run.smithi032.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-06T07:25:43.370 INFO:teuthology.orchestra.run.smithi032.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 261 kB/s | 11 kB 00:00 2024-03-06T07:25:43.403 INFO:teuthology.orchestra.run.smithi032.stdout:(126/176): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-06T07:25:43.429 INFO:teuthology.orchestra.run.smithi032.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 306 kB/s | 18 kB 00:00 2024-03-06T07:25:43.454 INFO:teuthology.orchestra.run.smithi032.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 470 kB/s | 23 kB 00:00 2024-03-06T07:25:43.479 INFO:teuthology.orchestra.run.smithi032.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 394 kB/s | 19 kB 00:00 2024-03-06T07:25:43.504 INFO:teuthology.orchestra.run.smithi032.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 399 kB/s | 20 kB 00:00 2024-03-06T07:25:43.555 INFO:teuthology.orchestra.run.smithi032.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 944 kB/s | 46 kB 00:00 2024-03-06T07:25:43.605 INFO:teuthology.orchestra.run.smithi032.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-03-06T07:25:43.664 INFO:teuthology.orchestra.run.smithi032.stdout:(133/176): python3-grpcio-1.46.7-10.el9.x86_64. 5.2 MB/s | 2.0 MB 00:00 2024-03-06T07:25:43.689 INFO:teuthology.orchestra.run.smithi032.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 698 kB/s | 58 kB 00:00 2024-03-06T07:25:43.739 INFO:teuthology.orchestra.run.smithi032.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-03-06T07:25:43.764 INFO:teuthology.orchestra.run.smithi032.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 221 kB/s | 16 kB 00:00 2024-03-06T07:25:43.806 INFO:teuthology.orchestra.run.smithi032.stdout:(137/176): python3-kubernetes-26.1.0-2.el9.noar 3.1 MB/s | 1.0 MB 00:00 2024-03-06T07:25:43.840 INFO:teuthology.orchestra.run.smithi032.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 379 kB/s | 28 kB 00:00 2024-03-06T07:25:43.865 INFO:teuthology.orchestra.run.smithi032.stdout:(139/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 720 kB/s | 90 kB 00:00 2024-03-06T07:25:43.890 INFO:teuthology.orchestra.run.smithi032.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 371 kB/s | 31 kB 00:00 2024-03-06T07:25:43.916 INFO:teuthology.orchestra.run.smithi032.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-06T07:25:43.949 INFO:teuthology.orchestra.run.smithi032.stdout:(142/176): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-06T07:25:43.974 INFO:teuthology.orchestra.run.smithi032.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 435 kB/s | 36 kB 00:00 2024-03-06T07:25:44.000 INFO:teuthology.orchestra.run.smithi032.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 614 kB/s | 51 kB 00:00 2024-03-06T07:25:44.025 INFO:teuthology.orchestra.run.smithi032.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-03-06T07:25:44.059 INFO:teuthology.orchestra.run.smithi032.stdout:(146/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-06T07:25:44.092 INFO:teuthology.orchestra.run.smithi032.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 302 kB/s | 20 kB 00:00 2024-03-06T07:25:44.126 INFO:teuthology.orchestra.run.smithi032.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-06T07:25:44.185 INFO:teuthology.orchestra.run.smithi032.stdout:(149/176): python3-werkzeug-2.0.3-3.el9.1.noarc 2.3 MB/s | 427 kB 00:00 2024-03-06T07:25:44.494 INFO:teuthology.orchestra.run.smithi032.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-03-06T07:25:44.527 INFO:teuthology.orchestra.run.smithi032.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:25:44.744 INFO:teuthology.orchestra.run.smithi032.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-06T07:25:44.844 INFO:teuthology.orchestra.run.smithi032.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 18 MB/s | 1.8 MB 00:00 2024-03-06T07:25:44.878 INFO:teuthology.orchestra.run.smithi032.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:25:44.920 INFO:teuthology.orchestra.run.smithi032.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-06T07:25:44.962 INFO:teuthology.orchestra.run.smithi032.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 69 kB 00:00 2024-03-06T07:25:44.995 INFO:teuthology.orchestra.run.smithi032.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-03-06T07:25:45.037 INFO:teuthology.orchestra.run.smithi032.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-03-06T07:25:45.079 INFO:teuthology.orchestra.run.smithi032.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-03-06T07:25:45.142 INFO:teuthology.orchestra.run.smithi032.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:25:45.226 INFO:teuthology.orchestra.run.smithi032.stdout:(161/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 3.0 MB/s | 3.1 MB 00:01 2024-03-06T07:25:45.260 INFO:teuthology.orchestra.run.smithi032.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 3.6 MB/s | 433 kB 00:00 2024-03-06T07:25:45.285 INFO:teuthology.orchestra.run.smithi032.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 404 kB/s | 22 kB 00:00 2024-03-06T07:25:45.311 INFO:teuthology.orchestra.run.smithi032.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-06T07:25:45.336 INFO:teuthology.orchestra.run.smithi032.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 154 kB 00:00 2024-03-06T07:25:45.361 INFO:teuthology.orchestra.run.smithi032.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.8 MB/s | 135 kB 00:00 2024-03-06T07:25:45.386 INFO:teuthology.orchestra.run.smithi032.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.3 MB/s | 63 kB 00:00 2024-03-06T07:25:45.412 INFO:teuthology.orchestra.run.smithi032.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 564 kB/s | 28 kB 00:00 2024-03-06T07:25:45.462 INFO:teuthology.orchestra.run.smithi032.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 16 MB/s | 1.2 MB 00:00 2024-03-06T07:25:45.537 INFO:teuthology.orchestra.run.smithi032.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 19 MB/s | 2.2 MB 00:00 2024-03-06T07:25:45.621 INFO:teuthology.orchestra.run.smithi032.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-06T07:25:45.655 INFO:teuthology.orchestra.run.smithi032.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 438 kB 00:00 2024-03-06T07:25:45.672 INFO:teuthology.orchestra.run.smithi032.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 656 kB/s | 30 kB 00:00 2024-03-06T07:25:45.705 INFO:teuthology.orchestra.run.smithi032.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-06T07:25:45.755 INFO:teuthology.orchestra.run.smithi032.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-06T07:25:45.839 INFO:teuthology.orchestra.run.smithi032.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.0 MB/s | 3.4 MB 00:01 2024-03-06T07:25:45.848 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:25:45.848 INFO:teuthology.orchestra.run.smithi032.stdout:Total 16 MB/s | 224 MB 00:13 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-06T07:25:46.381 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-06T07:25:46.381 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-06T07:25:48.796 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-06T07:25:48.798 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-06T07:25:56.423 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-06T07:25:56.674 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:56.737 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:56.872 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:56.944 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:57.033 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:57.141 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:57.202 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:57.233 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:57.332 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:57.762 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:57.797 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:58.381 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:58.440 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:58.470 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:58.655 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:58.746 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:58.808 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:58.840 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:58.904 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:59.312 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:59.357 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:59.399 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:59.721 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:59.898 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:26:00.208 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:26:00.259 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:26:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:26:00.771 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:26:00.901 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:26:01.029 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:26:01.152 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:26:01.230 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:26:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:26:01.342 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:26:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:26:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:26:01.588 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:26:01.666 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:26:01.714 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:26:01.863 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:26:02.015 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:26:02.082 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:26:02.120 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:26:02.251 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:26:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:26:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:26:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:26:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:26:03.384 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:26:03.487 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:26:03.616 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:26:03.665 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:26:03.697 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:26:03.762 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:26:03.823 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:26:03.870 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:26:04.063 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:26:04.111 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:26:04.491 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:26:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:26:04.848 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:26:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:26:05.143 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:26:05.223 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:26:05.338 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:26:05.376 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:26:05.417 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:26:05.488 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:26:05.540 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:26:05.604 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:26:05.673 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:26:05.882 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:26:05.946 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:26:05.985 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:26:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:26:06.211 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:26:06.258 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:26:06.305 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:26:06.340 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:26:06.373 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:26:06.540 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:26:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:26:06.868 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:26:06.933 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:26:06.998 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:26:07.076 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:26:07.209 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 267 kB/s | 21 MB 01:18 2024-03-06T07:26:07.331 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:26:07.396 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:26:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:26:07.626 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:26:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:26:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:26:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:26:07.897 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:26:07.958 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:26:08.353 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:26:08.437 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:26:08.488 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:26:08.631 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:26:08.678 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:26:08.713 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:26:08.785 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:26:08.825 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:26:08.987 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:26:09.105 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:26:09.162 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:26:09.198 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:26:09.435 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:26:09.506 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:26:10.517 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:26:10.571 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:26:10.605 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:26:10.662 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:26:10.822 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:26:11.063 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:26:11.093 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:26:11.189 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:26:12.031 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:26:12.178 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:26:13.208 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-06T07:26:13.772 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:26:13.874 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:26:14.295 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:26:14.327 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:26:14.449 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:26:14.483 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:26:15.070 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:26:15.097 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:26:15.189 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:26:15.215 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:26:16.016 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:26:16.018 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:26:16.039 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:26:16.041 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:26:16.042 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:26:16.043 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:26:16.046 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:26:16.049 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:26:16.051 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:26:16.149 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-03-06T07:26:16.174 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2024-03-06T07:26:16.174 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-03-06T07:26:16.174 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2024-03-06T07:26:16.174 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-03-06T07:26:16.174 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:26:16.175 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi039.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-03-06T07:26:16.179 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:26:16.182 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:26:16.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:26:16.187 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout:Install 150 Packages 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 26 Packages 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:26:16.189 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 224 M 2024-03-06T07:26:16.189 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-03-06T07:26:16.943 INFO:teuthology.orchestra.run.smithi039.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-06T07:26:17.301 INFO:teuthology.orchestra.run.smithi039.stdout:(2/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 7.6 MB/s | 5.3 MB 00:00 2024-03-06T07:26:17.360 INFO:teuthology.orchestra.run.smithi039.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 2.5 MB/s | 145 kB 00:00 2024-03-06T07:26:17.452 INFO:teuthology.orchestra.run.smithi039.stdout:(4/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.7 MB/s | 888 kB 00:00 2024-03-06T07:26:17.610 INFO:teuthology.orchestra.run.smithi039.stdout:(5/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 9.3 MB/s | 2.3 MB 00:00 2024-03-06T07:26:17.854 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:26:17.887 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:26:18.019 INFO:teuthology.orchestra.run.smithi039.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-06T07:26:18.078 INFO:teuthology.orchestra.run.smithi039.stdout:(7/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-03-06T07:26:18.547 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:26:18.599 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:26:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:26:19.320 INFO:teuthology.orchestra.run.smithi039.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 13 MB/s | 17 MB 00:01 2024-03-06T07:26:19.379 INFO:teuthology.orchestra.run.smithi039.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 438 kB/s | 25 kB 00:00 2024-03-06T07:26:19.629 INFO:teuthology.orchestra.run.smithi039.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 6.7 MB/s | 20 MB 00:03 2024-03-06T07:26:19.704 INFO:teuthology.orchestra.run.smithi039.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.9 MB/s | 220 kB 00:00 2024-03-06T07:26:19.796 INFO:teuthology.orchestra.run.smithi039.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 298 kB/s | 27 kB 00:00 2024-03-06T07:26:19.947 INFO:teuthology.orchestra.run.smithi039.stdout:(13/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 5.2 MB/s | 9.8 MB 00:01 2024-03-06T07:26:19.971 INFO:teuthology.orchestra.run.smithi039.stdout:(14/176): libcephfs-devel-19.0.0-1739.g1bc250c7 179 kB/s | 31 kB 00:00 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi039.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 2.4 MB/s | 163 kB 00:00 2024-03-06T07:26:20.081 INFO:teuthology.orchestra.run.smithi039.stdout:(16/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 5.5 MB/s | 747 kB 00:00 2024-03-06T07:26:20.148 INFO:teuthology.orchestra.run.smithi039.stdout:(17/176): librados-devel-19.0.0-1739.g1bc250c7. 1.1 MB/s | 127 kB 00:00 2024-03-06T07:26:20.190 INFO:teuthology.orchestra.run.smithi039.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 4.5 MB/s | 501 kB 00:00 2024-03-06T07:26:20.273 INFO:teuthology.orchestra.run.smithi039.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 547 kB/s | 45 kB 00:00 2024-03-06T07:26:20.374 INFO:teuthology.orchestra.run.smithi039.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.3 MB/s | 131 kB 00:00 2024-03-06T07:26:20.491 INFO:teuthology.orchestra.run.smithi039.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.4 MB/s | 162 kB 00:00 2024-03-06T07:26:20.608 INFO:teuthology.orchestra.run.smithi039.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 2.7 MB/s | 321 kB 00:00 2024-03-06T07:26:20.741 INFO:teuthology.orchestra.run.smithi039.stdout:(23/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 8.6 MB/s | 5.1 MB 00:00 2024-03-06T07:26:20.783 INFO:teuthology.orchestra.run.smithi039.stdout:(24/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.7 MB/s | 298 kB 00:00 2024-03-06T07:26:20.850 INFO:teuthology.orchestra.run.smithi039.stdout:(25/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 922 kB/s | 99 kB 00:00 2024-03-06T07:26:20.876 INFO:teuthology.orchestra.run.smithi039.stdout:(26/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 934 kB/s | 85 kB 00:00 2024-03-06T07:26:20.943 INFO:teuthology.orchestra.run.smithi039.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 2.5 MB/s | 171 kB 00:00 2024-03-06T07:26:21.035 INFO:teuthology.orchestra.run.smithi039.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 346 kB/s | 31 kB 00:00 2024-03-06T07:26:21.143 INFO:teuthology.orchestra.run.smithi039.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.3 MB/s | 138 kB 00:00 2024-03-06T07:26:21.260 INFO:teuthology.orchestra.run.smithi039.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 7.6 MB/s | 3.1 MB 00:00 2024-03-06T07:26:21.427 INFO:teuthology.orchestra.run.smithi039.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 6.3 MB/s | 1.8 MB 00:00 2024-03-06T07:26:21.574 INFO:teuthology.orchestra.run.smithi039.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.7 MB/s | 248 kB 00:00 2024-03-06T07:26:21.686 INFO:teuthology.orchestra.run.smithi039.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 444 kB/s | 49 kB 00:00 2024-03-06T07:26:21.795 INFO:teuthology.orchestra.run.smithi039.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 143 kB/s | 15 kB 00:00 2024-03-06T07:26:21.940 INFO:teuthology.orchestra.run.smithi039.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.8 MB/s | 264 kB 00:00 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi039.stdout:(36/176): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-03-06T07:26:22.221 INFO:teuthology.orchestra.run.smithi039.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 247 kB/s | 18 kB 00:00 2024-03-06T07:26:22.755 INFO:teuthology.orchestra.run.smithi039.stdout:(38/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 14 MB/s | 47 MB 00:03 2024-03-06T07:26:22.872 INFO:teuthology.orchestra.run.smithi039.stdout:(39/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 804 kB/s | 748 kB 00:00 2024-03-06T07:26:23.055 INFO:teuthology.orchestra.run.smithi039.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 920 kB 00:00 2024-03-06T07:26:23.097 INFO:teuthology.orchestra.run.smithi039.stdout:(41/176): automake-1.16.2-8.el9.noarch.rpm 756 kB/s | 662 kB 00:00 2024-03-06T07:26:23.122 INFO:teuthology.orchestra.run.smithi039.stdout:(42/176): boost-program-options-1.75.0-8.el9.x8 576 kB/s | 106 kB 00:00 2024-03-06T07:26:23.147 INFO:teuthology.orchestra.run.smithi039.stdout:(43/176): elfutils-libelf-devel-0.190-2.el9.x86 230 kB/s | 20 kB 00:00 2024-03-06T07:26:23.181 INFO:teuthology.orchestra.run.smithi039.stdout:(44/176): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 307 kB 00:00 2024-03-06T07:26:23.215 INFO:teuthology.orchestra.run.smithi039.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 501 kB/s | 32 kB 00:00 2024-03-06T07:26:23.315 INFO:teuthology.orchestra.run.smithi039.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 132 kB/s | 17 kB 00:00 2024-03-06T07:26:23.415 INFO:teuthology.orchestra.run.smithi039.stdout:(47/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-03-06T07:26:23.432 INFO:teuthology.orchestra.run.smithi039.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 48 kB/s | 10 kB 00:00 2024-03-06T07:26:23.466 INFO:teuthology.orchestra.run.smithi039.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-06T07:26:23.492 INFO:teuthology.orchestra.run.smithi039.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 787 kB/s | 44 kB 00:00 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi039.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 7.2 MB/s | 661 kB 00:00 2024-03-06T07:26:23.575 INFO:teuthology.orchestra.run.smithi039.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-03-06T07:26:23.592 INFO:teuthology.orchestra.run.smithi039.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 401 kB/s | 24 kB 00:00 2024-03-06T07:26:23.609 INFO:teuthology.orchestra.run.smithi039.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 599 kB/s | 46 kB 00:00 2024-03-06T07:26:23.642 INFO:teuthology.orchestra.run.smithi039.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-06T07:26:23.676 INFO:teuthology.orchestra.run.smithi039.stdout:(56/176): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-06T07:26:23.701 INFO:teuthology.orchestra.run.smithi039.stdout:(57/176): openblas-0.3.21-2.el9.x86_64.rpm 388 kB/s | 34 kB 00:00 2024-03-06T07:26:23.910 INFO:teuthology.orchestra.run.smithi039.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-06T07:26:23.994 INFO:teuthology.orchestra.run.smithi039.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 5.5 MB/s | 1.5 MB 00:00 2024-03-06T07:26:24.069 INFO:teuthology.orchestra.run.smithi039.stdout:(60/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.4 MB/s | 3.0 MB 00:00 2024-03-06T07:26:24.127 INFO:teuthology.orchestra.run.smithi039.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 3.9 MB/s | 811 kB 00:00 2024-03-06T07:26:24.152 INFO:teuthology.orchestra.run.smithi039.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 913 kB/s | 134 kB 00:00 2024-03-06T07:26:24.262 INFO:teuthology.orchestra.run.smithi039.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 7.7 MB/s | 1.0 MB 00:00 2024-03-06T07:26:24.387 INFO:teuthology.orchestra.run.smithi039.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-06T07:26:24.412 INFO:teuthology.orchestra.run.smithi039.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 319 kB/s | 78 kB 00:00 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi039.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-06T07:26:24.604 INFO:teuthology.orchestra.run.smithi039.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 227 kB 00:00 2024-03-06T07:26:24.629 INFO:teuthology.orchestra.run.smithi039.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 121 kB/s | 24 kB 00:00 2024-03-06T07:26:24.663 INFO:teuthology.orchestra.run.smithi039.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 567 kB/s | 18 kB 00:00 2024-03-06T07:26:24.688 INFO:teuthology.orchestra.run.smithi039.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-03-06T07:26:24.715 INFO:teuthology.orchestra.run.smithi039.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 385 kB/s | 32 kB 00:00 2024-03-06T07:26:24.864 INFO:teuthology.orchestra.run.smithi039.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-06T07:26:24.889 INFO:teuthology.orchestra.run.smithi039.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-06T07:26:24.914 INFO:teuthology.orchestra.run.smithi039.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 952 kB/s | 174 kB 00:00 2024-03-06T07:26:24.956 INFO:teuthology.orchestra.run.smithi039.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-03-06T07:26:24.981 INFO:teuthology.orchestra.run.smithi039.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 631 kB/s | 41 kB 00:00 2024-03-06T07:26:25.015 INFO:teuthology.orchestra.run.smithi039.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 238 kB 00:00 2024-03-06T07:26:25.057 INFO:teuthology.orchestra.run.smithi039.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 132 kB 00:00 2024-03-06T07:26:25.090 INFO:teuthology.orchestra.run.smithi039.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 1.9 MB/s | 211 kB 00:00 2024-03-06T07:26:25.116 INFO:teuthology.orchestra.run.smithi039.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 490 kB/s | 47 kB 00:00 2024-03-06T07:26:25.174 INFO:teuthology.orchestra.run.smithi039.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 521 kB/s | 43 kB 00:00 2024-03-06T07:26:25.491 INFO:teuthology.orchestra.run.smithi039.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-06T07:26:25.541 INFO:teuthology.orchestra.run.smithi039.stdout:(83/176): python3-toml-0.10.2-6.el9.noarch.rpm 95 kB/s | 39 kB 00:00 2024-03-06T07:26:25.567 INFO:teuthology.orchestra.run.smithi039.stdout:(84/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 109 kB/s | 41 kB 00:00 2024-03-06T07:26:25.617 INFO:teuthology.orchestra.run.smithi039.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.3 MB/s | 215 kB 00:00 2024-03-06T07:26:25.634 INFO:teuthology.orchestra.run.smithi039.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 518 kB/s | 34 kB 00:00 2024-03-06T07:26:25.651 INFO:teuthology.orchestra.run.smithi039.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 530 kB/s | 43 kB 00:00 2024-03-06T07:26:25.695 INFO:teuthology.orchestra.run.smithi039.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 300 kB 00:00 2024-03-06T07:26:25.710 INFO:teuthology.orchestra.run.smithi039.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-06T07:26:25.735 INFO:teuthology.orchestra.run.smithi039.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi039.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-06T07:26:25.777 INFO:teuthology.orchestra.run.smithi039.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-06T07:26:25.802 INFO:teuthology.orchestra.run.smithi039.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-06T07:26:25.844 INFO:teuthology.orchestra.run.smithi039.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2024-03-06T07:26:25.912 INFO:teuthology.orchestra.run.smithi039.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 7.4 MB/s | 802 kB 00:00 2024-03-06T07:26:25.945 INFO:teuthology.orchestra.run.smithi039.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 194 kB 00:00 2024-03-06T07:26:25.979 INFO:teuthology.orchestra.run.smithi039.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi039.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 241 kB 00:00 2024-03-06T07:26:26.071 INFO:teuthology.orchestra.run.smithi039.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 9.4 MB/s | 1.1 MB 00:00 2024-03-06T07:26:26.096 INFO:teuthology.orchestra.run.smithi039.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 996 kB/s | 103 kB 00:00 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi039.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 2.5 MB/s | 124 kB 00:00 2024-03-06T07:26:26.147 INFO:teuthology.orchestra.run.smithi039.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.5 MB/s | 113 kB 00:00 2024-03-06T07:26:26.180 INFO:teuthology.orchestra.run.smithi039.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.3 MB/s | 187 kB 00:00 2024-03-06T07:26:26.273 INFO:teuthology.orchestra.run.smithi039.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 182 kB/s | 22 kB 00:00 2024-03-06T07:26:26.390 INFO:teuthology.orchestra.run.smithi039.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.0 MB/s | 245 kB 00:00 2024-03-06T07:26:26.523 INFO:teuthology.orchestra.run.smithi039.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-06T07:26:26.632 INFO:teuthology.orchestra.run.smithi039.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-06T07:26:26.682 INFO:teuthology.orchestra.run.smithi039.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-03-06T07:26:26.724 INFO:teuthology.orchestra.run.smithi039.stdout:(109/176): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-03-06T07:26:26.766 INFO:teuthology.orchestra.run.smithi039.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-06T07:26:26.850 INFO:teuthology.orchestra.run.smithi039.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 814 kB/s | 67 kB 00:00 2024-03-06T07:26:26.883 INFO:teuthology.orchestra.run.smithi039.stdout:(112/176): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-03-06T07:26:27.009 INFO:teuthology.orchestra.run.smithi039.stdout:(113/176): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-06T07:26:27.042 INFO:teuthology.orchestra.run.smithi039.stdout:(114/176): luarocks-3.9.2-1.el9.noarch.rpm 789 kB/s | 151 kB 00:00 2024-03-06T07:26:27.126 INFO:teuthology.orchestra.run.smithi039.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-06T07:26:27.151 INFO:teuthology.orchestra.run.smithi039.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 402 kB/s | 43 kB 00:00 2024-03-06T07:26:27.218 INFO:teuthology.orchestra.run.smithi039.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 353 kB/s | 32 kB 00:00 2024-03-06T07:26:27.252 INFO:teuthology.orchestra.run.smithi039.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-06T07:26:27.352 INFO:teuthology.orchestra.run.smithi039.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-06T07:26:27.411 INFO:teuthology.orchestra.run.smithi039.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-03-06T07:26:27.461 INFO:teuthology.orchestra.run.smithi039.stdout:(121/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.4 MB/s | 838 kB 00:00 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run.smithi039.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-06T07:26:27.528 INFO:teuthology.orchestra.run.smithi039.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-03-06T07:26:27.670 INFO:teuthology.orchestra.run.smithi039.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-06T07:26:27.696 INFO:teuthology.orchestra.run.smithi039.stdout:(125/176): python3-google-auth-2.28.0-1.el9.noa 867 kB/s | 203 kB 00:00 2024-03-06T07:26:27.780 INFO:teuthology.orchestra.run.smithi039.stdout:(126/176): python3-grpcio-1.46.7-10.el9.x86_64. 7.2 MB/s | 2.0 MB 00:00 2024-03-06T07:26:27.805 INFO:teuthology.orchestra.run.smithi039.stdout:(127/176): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-06T07:26:27.830 INFO:teuthology.orchestra.run.smithi039.stdout:(128/176): python3-jaraco-classes-3.2.1-5.el9.n 133 kB/s | 18 kB 00:00 2024-03-06T07:26:27.897 INFO:teuthology.orchestra.run.smithi039.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 213 kB/s | 19 kB 00:00 2024-03-06T07:26:27.922 INFO:teuthology.orchestra.run.smithi039.stdout:(130/176): python3-jaraco-collections-3.0.0-8.e 163 kB/s | 23 kB 00:00 2024-03-06T07:26:27.947 INFO:teuthology.orchestra.run.smithi039.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-06T07:26:28.040 INFO:teuthology.orchestra.run.smithi039.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 397 kB/s | 46 kB 00:00 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi039.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 567 kB/s | 79 kB 00:00 2024-03-06T07:26:28.163 INFO:teuthology.orchestra.run.smithi039.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 3.9 MB/s | 1.0 MB 00:00 2024-03-06T07:26:28.188 INFO:teuthology.orchestra.run.smithi039.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 394 kB/s | 58 kB 00:00 2024-03-06T07:26:28.263 INFO:teuthology.orchestra.run.smithi039.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 165 kB/s | 16 kB 00:00 2024-03-06T07:26:28.289 INFO:teuthology.orchestra.run.smithi039.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 899 kB/s | 90 kB 00:00 2024-03-06T07:26:28.414 INFO:teuthology.orchestra.run.smithi039.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 188 kB/s | 28 kB 00:00 2024-03-06T07:26:28.440 INFO:teuthology.orchestra.run.smithi039.stdout:(139/176): python3-pecan-1.4.2-3.el9.noarch.rpm 775 kB/s | 272 kB 00:00 2024-03-06T07:26:28.465 INFO:teuthology.orchestra.run.smithi039.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 176 kB/s | 31 kB 00:00 2024-03-06T07:26:28.532 INFO:teuthology.orchestra.run.smithi039.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi039.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-03-06T07:26:28.583 INFO:teuthology.orchestra.run.smithi039.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2024-03-06T07:26:28.625 INFO:teuthology.orchestra.run.smithi039.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 557 kB/s | 51 kB 00:00 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi039.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-06T07:26:28.742 INFO:teuthology.orchestra.run.smithi039.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 566 kB/s | 90 kB 00:00 2024-03-06T07:26:28.767 INFO:teuthology.orchestra.run.smithi039.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 201 kB/s | 20 kB 00:00 2024-03-06T07:26:28.801 INFO:teuthology.orchestra.run.smithi039.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-03-06T07:26:28.909 INFO:teuthology.orchestra.run.smithi039.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-06T07:26:28.985 INFO:teuthology.orchestra.run.smithi039.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-06T07:26:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:26:29.127 INFO:teuthology.orchestra.run.smithi039.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi039.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-06T07:26:29.244 INFO:teuthology.orchestra.run.smithi039.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:26:29.286 INFO:teuthology.orchestra.run.smithi039.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-06T07:26:29.320 INFO:teuthology.orchestra.run.smithi039.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.1 MB/s | 69 kB 00:00 2024-03-06T07:26:29.353 INFO:teuthology.orchestra.run.smithi039.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-03-06T07:26:29.404 INFO:teuthology.orchestra.run.smithi039.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 191 kB 00:00 2024-03-06T07:26:29.479 INFO:teuthology.orchestra.run.smithi039.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.4 MB/s | 254 kB 00:00 2024-03-06T07:26:29.555 INFO:teuthology.orchestra.run.smithi039.stdout:(160/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 4.9 MB/s | 3.1 MB 00:00 2024-03-06T07:26:29.605 INFO:teuthology.orchestra.run.smithi039.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 6.3 MB/s | 802 kB 00:00 2024-03-06T07:26:29.647 INFO:teuthology.orchestra.run.smithi039.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 5.0 MB/s | 433 kB 00:00 2024-03-06T07:26:29.664 INFO:teuthology.orchestra.run.smithi039.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 407 kB/s | 22 kB 00:00 2024-03-06T07:26:29.697 INFO:teuthology.orchestra.run.smithi039.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-06T07:26:29.731 INFO:teuthology.orchestra.run.smithi039.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 154 kB 00:00 2024-03-06T07:26:29.756 INFO:teuthology.orchestra.run.smithi039.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 135 kB 00:00 2024-03-06T07:26:29.815 INFO:teuthology.orchestra.run.smithi039.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 767 kB/s | 63 kB 00:00 2024-03-06T07:26:29.841 INFO:teuthology.orchestra.run.smithi039.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 335 kB/s | 28 kB 00:00 2024-03-06T07:26:29.891 INFO:teuthology.orchestra.run.smithi039.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 16 MB/s | 1.2 MB 00:00 2024-03-06T07:26:29.991 INFO:teuthology.orchestra.run.smithi039.stdout:(170/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.9 MB/s | 3.4 MB 00:01 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi039.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-06T07:26:30.134 INFO:teuthology.orchestra.run.smithi039.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 9.7 MB/s | 2.2 MB 00:00 2024-03-06T07:26:30.184 INFO:teuthology.orchestra.run.smithi039.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 2.4 MB/s | 438 kB 00:00 2024-03-06T07:26:30.209 INFO:teuthology.orchestra.run.smithi039.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 229 kB/s | 30 kB 00:00 2024-03-06T07:26:30.251 INFO:teuthology.orchestra.run.smithi039.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 4.8 MB/s | 321 kB 00:00 2024-03-06T07:26:30.293 INFO:teuthology.orchestra.run.smithi039.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-06T07:26:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:26:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:Total 16 MB/s | 224 MB 00:14 2024-03-06T07:26:30.447 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-03-06T07:26:30.843 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-03-06T07:26:30.844 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-03-06T07:26:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:26:32.034 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-03-06T07:26:32.034 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-03-06T07:26:32.034 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-03-06T07:26:32.034 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-03-06T07:26:32.034 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-03-06T07:26:32.035 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-03-06T07:26:32.035 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-03-06T07:26:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:32.238 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:32.284 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:32.284 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:32.284 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:26:32.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:33.226 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-03-06T07:26:33.227 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-03-06T07:26:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:34.061 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:34.061 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:34.061 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:26:34.061 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:34.061 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:34.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:34.314 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:26:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:34.924 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:35.019 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:35.019 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:35.019 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:26:35.019 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:35.019 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:35.019 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:35.151 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:26:35.699 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:35.760 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:26:35.919 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:26:35.955 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:36.290 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:36.323 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:37.490 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:37.523 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:37.853 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:26:37.892 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:38.051 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:38.091 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:38.192 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:26:39.206 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:39.343 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:39.343 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:39.343 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:26:39.343 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:39.343 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:39.344 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:39.395 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:39.495 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:39.495 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:39.495 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:26:39.495 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:39.571 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:39.678 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:39.678 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:39.678 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:26:39.678 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:39.678 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:39.678 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:40.096 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:40.200 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:40.200 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:40.200 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:26:40.200 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:40.200 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:40.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:41.118 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-03-06T07:26:41.209 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:26:41.253 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:26:41.319 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:26:41.429 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:26:41.619 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:26:41.747 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:26:41.864 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:26:41.929 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:26:41.967 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:26:42.067 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:26:42.497 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:26:42.539 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:42.704 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:42.897 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:42.944 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:43.222 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:26:43.338 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:26:43.363 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:43.547 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:26:43.622 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:26:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:26:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:26:43.782 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:26:44.158 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:26:44.237 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:26:44.707 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:26:44.874 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:26:44.995 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:26:45.199 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:26:45.253 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:26:45.259 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:26:45.297 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:26:45.342 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:26:45.380 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:26:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:26:45.665 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:26:45.795 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:26:45.798 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:26:45.855 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:26:45.923 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:26:45.928 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:26:45.966 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:26:46.051 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:26:46.066 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:26:46.138 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:26:46.169 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:26:46.185 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:26:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:26:46.228 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:26:46.261 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:26:46.275 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:26:46.363 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:26:46.367 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:26:46.434 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:26:46.490 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:26:46.508 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:26:46.564 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:26:46.570 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:26:46.601 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:26:46.620 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:26:46.727 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:26:46.833 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:26:46.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:26:47.031 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:26:47.133 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:26:47.485 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:26:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:47.541 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:48.218 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:26:48.318 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:26:48.452 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:26:48.506 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:26:48.592 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:26:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:26:48.684 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:26:48.869 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:26:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:26:49.021 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:49.106 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:49.285 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:49.325 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:49.358 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:26:49.392 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:49.433 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:49.434 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:49.567 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:49.597 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:26:49.618 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:49.671 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:49.672 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:49.712 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:26:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:49.871 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:49.928 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:49.966 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:50.006 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:26:50.019 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:26:50.098 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:50.142 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:26:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:50.243 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:26:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:50.283 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:26:50.322 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:50.324 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:26:50.355 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:50.397 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:50.409 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:50.449 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:26:50.472 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:50.510 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:50.510 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:50.515 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:26:50.581 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:26:50.785 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:26:50.846 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:26:50.848 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:50.848 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:50.876 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:26:50.926 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:26:50.978 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:51.030 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:51.067 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:51.070 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:26:51.115 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:26:51.125 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:51.161 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:26:51.162 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:51.198 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:26:51.231 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:26:51.293 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:51.406 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:26:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:51.559 INFO:teuthology.orchestra.run.smithi032.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:51.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:51.588 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:26:51.738 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:26:51.800 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:26:52.191 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:26:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:26:52.433 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:26:52.509 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:26:52.560 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:26:52.601 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:26:52.651 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:26:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:26:52.746 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:26:52.801 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:26:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:26:52.984 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:26:53.030 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:26:53.174 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:26:53.300 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:26:53.368 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:26:53.411 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:26:53.563 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:26:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:26:53.774 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:26:53.808 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:26:53.866 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:26:54.010 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:26:54.075 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:26:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:26:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:26:55.174 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:26:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:26:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:26:55.391 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:26:55.664 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:26:55.696 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:26:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:56.383 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:56.384 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:56.385 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:56.403 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:56.408 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:56.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:56.411 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:56.412 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:56.645 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:26:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:26:58.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:26:58.518 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:26:58.935 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:26:58.963 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:26:59.053 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:59.054 INFO:teuthology.orchestra.run.smithi032.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.055 INFO:teuthology.orchestra.run.smithi032.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.059 INFO:teuthology.orchestra.run.smithi032.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:59.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:59.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:59.066 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:26:59.067 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-06T07:26:59.093 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:26:59.128 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:26:59.718 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:59.729 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:26:59.766 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:26:59.867 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:26:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:27:02.450 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:27:02.482 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:27:03.139 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:27:03.177 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:27:03.310 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:27:03.310 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:27:03.311 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:27:03.311 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:03.369 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-03-06T07:27:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:27:17.323 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:27:17.323 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:17.323 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:27:17.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:19.003 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:27:19.085 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:27:19.085 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:19.085 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:27:19.085 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:27:19.085 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:27:19.085 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:19.336 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:27:19.407 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:27:19.407 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:19.407 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:27:19.407 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:27:19.407 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:27:19.407 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:19.951 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:27:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:27:20.035 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:20.035 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:27:20.035 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:27:20.035 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:27:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:27:20.799 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:27:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:27:20.890 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:27:20.996 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:27:21.043 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:27:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:27:21.427 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:27:22.612 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:27:22.644 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:27:22.793 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:27:22.991 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:27:23.021 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:27:23.095 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:27:23.096 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:23.096 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:27:23.096 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:27:23.096 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:27:23.096 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:23.173 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:27:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:27:23.318 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:27:24.338 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:27:24.365 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:27:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:27:24.438 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:24.438 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:27:24.438 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:27:24.438 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:27:24.438 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:24.499 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:27:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:27:24.574 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:24.574 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:27:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:24.659 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:27:24.740 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:27:24.740 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:24.741 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:27:24.741 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:27:24.741 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:27:24.741 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:27:25.233 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:27:25.233 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:27:25.234 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:27:25.234 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:27:25.234 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:27:25.234 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:30.315 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:27:30.393 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:27:30.430 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:27:30.469 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:27:30.769 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:27:30.949 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:27:31.010 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:27:31.091 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:27:31.206 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:27:31.249 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:27:31.295 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:27:31.323 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:27:31.378 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:27:31.546 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:27:31.608 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:27:31.643 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:27:32.504 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:27:32.582 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:27:32.641 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:27:34.055 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:27:34.128 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:27:34.306 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:27:34.372 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:27:34.405 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:27:34.439 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:27:34.483 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:27:34.484 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:27:34.614 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:27:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:27:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:27:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:27:34.865 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:27:34.926 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:27:34.975 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:27:35.013 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:27:35.099 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:27:35.219 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:27:35.287 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:27:35.335 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:27:35.369 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:27:35.427 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:27:35.461 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:27:35.494 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:27:35.992 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:27:36.043 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:27:36.073 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:27:36.130 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:27:36.166 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:27:36.298 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:27:36.573 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:27:36.573 INFO:teuthology.orchestra.run.smithi039.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:27:36.573 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:27:41.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:27:41.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:27:41.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:27:41.713 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:27:41.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:27:41.725 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:27:44.357 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:27:44.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:44.357 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:27:44.358 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.359 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.360 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.361 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:27:44.362 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:27:44.363 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:27:44.364 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:27:44.365 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:27:44.366 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:27:44.367 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:27:44.368 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:27:44.369 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:27:44.370 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:27:44.371 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:27:44.372 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:27:44.372 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:44.372 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-03-06T07:27:45.143 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:45.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:27:45.286 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:27:45.362 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:27:45.362 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:27:45.363 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:27:45.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:27:45.505 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:27:45.563 INFO:teuthology.orchestra.run.smithi039.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:27:45.563 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:27:45.563 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:27:45.564 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:27:45.565 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:45.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:27:45.674 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:45.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:27:45.754 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:27:45.754 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:45.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:27:45.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:27:45.959 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:45.959 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:27:46.040 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:27:46.152 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:27:46.152 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:46.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:27:46.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:27:46.381 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:46.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:27:46.460 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:27:46.572 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:27:46.572 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:46.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:27:46.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:27:46.787 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:46.787 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:27:46.856 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:27:46.967 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:27:47.052 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:27:47.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-03-06T07:27:47.060 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-03-06T07:27:47.142 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:27:47.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:27:47.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:27:47.234 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:27:47.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:27:47.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:27:47.336 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:27:47.336 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:27:47.336 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:27:47.336 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:47.336 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:27:47.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:47.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:47.731 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:47.732 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:27:47.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:47.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:48.054 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:27:48.054 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:27:48.055 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583073', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583073', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCq8nxrAaiDTym51uoTjdO1MXuKLphpyesa1QR7FnsXDWN8xTvy9X6wuKhLYZBNaX5iL+eUChDM82aLwP418bSE=', 'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZFHy6EPcycZSqUJx74UYkgnDWoG3lbn8x5b+vVB4Pyvl/bDHZ3SJz1AzMdVn6O6ysZuxFyBmf4D8M1AM4pJ/k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:27:48.055 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:27:48.063 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:27:48.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:27:48.121 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:27:48.216 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:48.217 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:27:48.275 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:27:48.275 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-03-06T07:27:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-06 07:26:52.026698469 +0000 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-06 07:23:51.997663293 +0000 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-06 07:23:51.997663293 +0000 2024-03-06T07:27:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-06 07:23:51.997663293 +0000 2024-03-06T07:27:48.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:27:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-06T07:27:48.423 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-06T07:27:48.423 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000254272 s, 2.0 MB/s 2024-03-06T07:27:48.427 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:27:48.488 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-03-06T07:27:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:27:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:48.542 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-06T07:27:48.542 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:48.542 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-06 07:26:52.026698469 +0000 2024-03-06T07:27:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-06 07:23:52.554651665 +0000 2024-03-06T07:27:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-06 07:23:52.554651665 +0000 2024-03-06T07:27:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-06 07:23:52.554651665 +0000 2024-03-06T07:27:48.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:27:48.638 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-06T07:27:48.638 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-06T07:27:48.638 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000268025 s, 1.9 MB/s 2024-03-06T07:27:48.642 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:27:48.702 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-06 07:26:52.027698449 +0000 2024-03-06T07:27:48.755 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-06 07:23:53.145639326 +0000 2024-03-06T07:27:48.756 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-06 07:23:53.145639326 +0000 2024-03-06T07:27:48.756 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-06 07:23:53.145639326 +0000 2024-03-06T07:27:48.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:27:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-06T07:27:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-06T07:27:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000224677 s, 2.3 MB/s 2024-03-06T07:27:48.856 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:27:48.916 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-06T07:27:48.968 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-06 07:26:52.027698449 +0000 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-06 07:23:53.720627322 +0000 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-06 07:23:53.720627322 +0000 2024-03-06T07:27:48.969 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-06 07:23:53.720627322 +0000 2024-03-06T07:27:48.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:27:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-06T07:27:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-06T07:27:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.00026595 s, 1.9 MB/s 2024-03-06T07:27:49.062 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:27:49.122 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:49.122 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:49.122 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:27:49.169 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:27:49.169 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-03-06T07:27:49.254 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:27:49.254 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:49.254 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-06T07:27:49.254 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:49.255 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:49.255 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-06 07:27:37.034538885 +0000 2024-03-06T07:27:49.255 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-06 07:23:51.767368175 +0000 2024-03-06T07:27:49.255 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-06 07:23:51.767368175 +0000 2024-03-06T07:27:49.255 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-06 07:23:51.767368175 +0000 2024-03-06T07:27:49.255 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:27:49.363 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-06T07:27:49.364 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-06T07:27:49.364 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000524796 s, 976 kB/s 2024-03-06T07:27:49.367 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:27:49.410 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-03-06T07:27:49.492 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-06 07:27:37.034538885 +0000 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-06 07:23:52.363356361 +0000 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-06 07:23:52.363356361 +0000 2024-03-06T07:27:49.493 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-06 07:23:52.363356361 +0000 2024-03-06T07:27:49.493 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000370956 s, 1.4 MB/s 2024-03-06T07:27:49.606 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:27:49.650 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-03-06T07:27:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:27:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:49.730 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-06T07:27:49.731 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:49.731 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:49.731 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-06 07:27:37.035538865 +0000 2024-03-06T07:27:49.731 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-06 07:23:52.972344289 +0000 2024-03-06T07:27:49.731 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-06 07:23:52.972344289 +0000 2024-03-06T07:27:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-06 07:23:52.972344289 +0000 2024-03-06T07:27:49.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:27:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-06T07:27:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-06T07:27:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000365445 s, 1.4 MB/s 2024-03-06T07:27:49.842 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:27:49.882 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-06 07:27:37.036538845 +0000 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-06 07:23:53.571332416 +0000 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-06 07:23:53.571332416 +0000 2024-03-06T07:27:49.963 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-06 07:23:53.571332416 +0000 2024-03-06T07:27:49.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:27:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-06T07:27:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-06T07:27:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000382399 s, 1.3 MB/s 2024-03-06T07:27:50.076 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:27:50.131 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:50.131 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:27:50.131 INFO:tasks.ceph:Generating config... 2024-03-06T07:27:50.132 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:27:50.132 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:27:50.132 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:27:50.133 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:27:50.133 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:27:50.133 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:27:50.133 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:27:50.133 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:27:50.134 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:27:50.134 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:27:50.134 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:27:50.134 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:27:50.134 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:27:50.135 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:27:50.135 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:27:50.135 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:27:50.135 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:27:50.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:50.292 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:27:50.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:27:50.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:27:50.545 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.32'), ('mon.b', '172.21.15.39'), ('mon.c', '[v2:172.21.15.39:3301,v1:172.21.15.39:6790]')] 2024-03-06T07:27:50.545 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:27:50.546 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:50.546 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:27:50.605 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.32 --add b 172.21.15.39 --addv c '[v2:172.21.15.39:3301,v1:172.21.15.39:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:fsid e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-03-06T07:27:50.766470+0000 2024-03-06T07:27:50.767 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-03-06T07:27:50.766470+0000 2024-03-06T07:27:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 17 (quincy) 2024-03-06T07:27:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 1 2024-03-06T07:27:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-03-06T07:27:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-03-06T07:27:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:2: [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] mon.c 2024-03-06T07:27:50.768 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:27:50.771 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:27:50.820 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e91bbd07-5556-4949-b381-99a73b242ae4... 2024-03-06T07:27:50.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:50.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:27:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.998 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:27:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:27:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:27:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:27:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:27:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:27:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:27:51.001 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.012 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.013 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:51.014 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:27:51.015 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:27:51.016 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:27:51.017 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:27:51.018 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.019 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:51.020 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.021 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:27:51.022 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:27:51.023 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:27:51.024 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:27:51.058 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:27:51.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:27:51.214 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:27:51.214 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:51.214 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:27:51.266 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:51.267 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:27:51.319 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:27:51.319 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:51.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:27:51.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:27:51.468 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:27:51.468 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:51.568 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:27:51.568 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:51.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:27:51.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:27:51.731 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:27:51.731 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:51.831 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:27:51.831 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:27:51.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:27:52.030 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:27:52.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:27:52.264 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:27:52.276 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:27:52.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:27:52.479 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:27:52.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:27:52.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:27:52.768 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:27:52.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:27:52.872 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:27:53.067 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:27:53.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:27:53.172 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:27:53.365 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:27:53.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:27:53.465 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:27:53.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:27:53.605 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:27:53.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:27:53.809 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:27:53.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:27:54.055 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:27:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:27:54.247 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:27:54.247 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:27:54.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:27:54.352 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:54.352 INFO:tasks.ceph:role: osd.0 2024-03-06T07:27:54.352 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:27:54.352 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:54.450 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:54.484 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-06T07:27:54.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-06T07:27:54.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:27:54.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:27:54.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:27:54.768 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:54.768 INFO:tasks.ceph:role: osd.1 2024-03-06T07:27:54.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:27:54.768 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:27:54.858 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:54.858 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:54.858 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:54.858 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:54.859 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:54.859 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:54.859 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:54.859 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:54.859 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:54.859 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:54.893 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-06T07:27:54.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-06T07:27:54.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:27:55.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:27:55.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:27:55.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:55.196 INFO:tasks.ceph:role: osd.2 2024-03-06T07:27:55.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:27:55.197 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:27:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:55.328 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-06T07:27:55.335 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-06T07:27:55.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:27:55.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:27:55.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:27:55.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:55.634 INFO:tasks.ceph:role: osd.3 2024-03-06T07:27:55.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:27:55.634 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:27:55.728 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:55.729 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:55.762 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-06T07:27:55.768 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-06T07:27:55.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:27:55.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:27:55.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:56.110+0000 7f00fd015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:27:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:56.111+0000 7f00fd015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:27:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:56.111+0000 7f00fd015640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:27:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:56.111+0000 7f00fd015640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:27:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:56.111+0000 7f00fd015640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:27:56.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:56.114+0000 7f00fd015640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:27:59.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:27:59.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:59.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:59.286+0000 7f5abf60f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:27:59.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:59.286+0000 7f5abf60f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:27:59.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:59.286+0000 7f5abf60f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:27:59.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:59.286+0000 7f5abf60f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:27:59.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:59.286+0000 7f5abf60f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:27:59.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:27:59.289+0000 7f5abf60f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:28:02.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:28:02.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:02.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:02.724+0000 7f4ad7ec4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:28:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:02.724+0000 7f4ad7ec4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:28:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:02.725+0000 7f4ad7ec4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:28:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:02.725+0000 7f4ad7ec4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:28:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:02.725+0000 7f4ad7ec4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:28:02.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:02.727+0000 7f4ad7ec4640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:28:05.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:28:06.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:06.154+0000 7f32c87dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:06.154+0000 7f32c87dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:06.155+0000 7f32c87dd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:06.155+0000 7f32c87dd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:06.155+0000 7f32c87dd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:06.156+0000 7f32c87dd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:28:09.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:28:09.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:28:09.537 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:28:09.538 INFO:tasks.ceph:role: osd.4 2024-03-06T07:28:09.538 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:28:09.538 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:28:10.977 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-03-06T07:28:10.984 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-03-06T07:28:10.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:28:11.099 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:28:11.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:28:11.286 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:28:11.286 INFO:tasks.ceph:role: osd.5 2024-03-06T07:28:11.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:28:11.287 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:28:11.394 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:28:11.394 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:28:11.394 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:28:11.395 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:28:12.714 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-03-06T07:28:12.724 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-03-06T07:28:12.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:28:12.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:28:12.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:28:12.986 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:28:12.986 INFO:tasks.ceph:role: osd.6 2024-03-06T07:28:12.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:28:12.986 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:28:13.103 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:28:13.103 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:28:13.103 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:28:13.103 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:28:13.104 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:28:14.423 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-03-06T07:28:14.430 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-03-06T07:28:14.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:28:14.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:28:14.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:28:14.726 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:28:14.727 INFO:tasks.ceph:role: osd.7 2024-03-06T07:28:14.727 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:28:14.727 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:28:16.170 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-03-06T07:28:16.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-03-06T07:28:16.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:28:16.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:28:16.385 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:16.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:16.528+0000 7f8021b49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:28:16.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:16.529+0000 7f8021b49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:28:16.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:16.529+0000 7f8021b49640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:28:16.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:16.529+0000 7f8021b49640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:28:16.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:16.529+0000 7f8021b49640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:28:16.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:16.532+0000 7f8021b49640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:28:18.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:18.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:19.747 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:28:19.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:19.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:19.984+0000 7f4fc961e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:19.985+0000 7f4fc961e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:19.985+0000 7f4fc961e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:19.985+0000 7f4fc961e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:19.985+0000 7f4fc961e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:28:19.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:19.988+0000 7f4fc961e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:28:22.961 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:28:23.056 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:23.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:23.199+0000 7f59e28aa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:28:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:23.200+0000 7f59e28aa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:28:23.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:23.200+0000 7f59e28aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:28:23.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:23.200+0000 7f59e28aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:28:23.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:23.200+0000 7f59e28aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:28:23.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:23.202+0000 7f59e28aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:28:26.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:28:26.272 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:26.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:26.416+0000 7fa531e22640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:28:26.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:26.417+0000 7fa531e22640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:28:26.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:26.417+0000 7fa531e22640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:28:26.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:26.417+0000 7fa531e22640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:28:26.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:26.417+0000 7fa531e22640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:28:26.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:26.419+0000 7fa531e22640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:28:29.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:28:29.705 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:28:29.705 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:29.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:28:29.812 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:29.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:28:29.915 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:29.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:28:30.015 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:30.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:28:30.113 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:30.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:28:30.212 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:30.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:28:30.309 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:30.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:28:30.408 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:30.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:28:30.512 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:30.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:28:30.610 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:30.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:28:30.699 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:30.699 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:28:30.794 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:30.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:28:30.944 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:30.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:28:31.043 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:31.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:28:31.143 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:31.143 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:28:31.202 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:31.202 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:28:31.258 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:31.258 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:28:31.318 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:31.319 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:28:31.371 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:28:31.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:28:31.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew== 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg== 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ== 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ== 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g== 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw== 2024-03-06T07:28:31.472 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg== 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ== 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout:[client.2] 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw== 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout:[client.1] 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg== 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout:[client.3] 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q== 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.y] 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew== 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA== 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:28:31.475 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.6] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.7] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[client.2] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[client.1] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg== 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout:[client.3] 2024-03-06T07:28:31.476 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q== 2024-03-06T07:28:31.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:28:31.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:28:31.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:31.640 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:31.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:31.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:31.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:31.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.416 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:32.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:28:32.575 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:28:32.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:32.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:32.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:32.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:28:33.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:28:33.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:33.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:33.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:33.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:34.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:34.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:34.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:34.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:28:34.337 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:28:34.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:28:34.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:28:34.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:28:34.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:28:34.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:28:34.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:28:35.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:28:35.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:28:35.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:28:35.416 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:35.423 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:28:35.479 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:28:35.480 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:28:35.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:28:35.487 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:28:35.487 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:28:35.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:28:35.492 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:28:35.492 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:28:35.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:28:35.498 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:28:35.499 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:28:35.499 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:28:35.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:28:35.504 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:28:35.504 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:28:35.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:28:35.510 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:28:35.510 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:28:35.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.690+0000 7f25eb600640 1 Processor -- start 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.690+0000 7f25eb600640 1 -- start start 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.691+0000 7f25eb600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.691+0000 7f25eb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.691+0000 7f25eb600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40cf780 con 0x7f25e40d4930 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.691+0000 7f25eb600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e40b3770 con 0x7f25e40d07d0 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.691+0000 7f25eb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e40cfa60 con 0x7f25e415e6c0 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33336/0 (socket says 172.21.15.32:33336) 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 learned_addr learned my addr 172.21.15.32:0/3663848571 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:28:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.692+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:28:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.893+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.893+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:28:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:35.894+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.295+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.295+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.295+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.295+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.295+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.296+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.296+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.296+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.296+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:28:36.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:36.296+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.097+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.097+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e8c00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:28:37.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:37.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:37.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:37.098+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:28:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 msgr2=0x7f25e40b3c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25e415e6c0 0x7f25e40b3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 msgr2=0x7f25e40b4f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.32:3300/0 conn(0x7f25e40d07d0 0x7f25e40b4f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 -- 172.21.15.32:0/3663848571 >> v1:172.21.15.32:6789/0 conn(0x7f25e40d4930 legacy=0x7f25e415dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 -- 172.21.15.32:0/3663848571 --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40cfd40 con 0x7f25c40036c0 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.692+0000 7f25e2000640 1 -- 172.21.15.32:0/3663848571 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e40b3770 con 0x7f25c40073e0 2024-03-06T07:28:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.693+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.694+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:28:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.694+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.694+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.694+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:28:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.895+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.896+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:28:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.896+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.896+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:38.896+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:39.297+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:39.298+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:39.298+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:39.298+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:39.298+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:28:40.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.100+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.100+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:28:40.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.100+0000 7f25e3e00640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:28:40.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.100+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:28:40.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.100+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:28:40.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.970+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528857323 0 0) 0x7f25e40cfd40 con 0x7f25c40036c0 2024-03-06T07:28:40.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.971+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc003520 con 0x7f25c40036c0 2024-03-06T07:28:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 <== mon.1 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2788013941 0 0) 0x7f25cc003520 con 0x7f25c40036c0 2024-03-06T07:28:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 msgr2=0x7f25c40098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 --2- 172.21.15.32:0/3663848571 >> v2:172.21.15.39:3300/0 conn(0x7f25c40073e0 0x7f25c40098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c400afa0 con 0x7f25c40036c0 2024-03-06T07:28:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 <== mon.1 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f25d800f040 con 0x7f25c40036c0 2024-03-06T07:28:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 >> v1:172.21.15.39:6789/0 conn(0x7f25c40036c0 legacy=0x7f25c4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 reap_dead start 2024-03-06T07:28:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc003a00 0x7f25cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc0053c0 0x7f25cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e40b3770 con 0x7f25cc0053c0 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc00c280 con 0x7f25cc003a00 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.977+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc00c480 con 0x7f25cc004800 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.978+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc0053c0 0x7f25cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.978+0000 7f25e3e00640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.978+0000 7f25e8c00640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc003a00 0x7f25cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.979+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 msgr2=0x7f25cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.979+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.979+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc003a00 msgr2=0x7f25cc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.979+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc003a00 0x7f25cc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.979+0000 7f25e9600640 1 -- 172.21.15.32:0/3663848571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc003520 con 0x7f25cc0053c0 2024-03-06T07:28:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.981+0000 7f25e9600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc0053c0 0x7f25cc00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f25e415eaa0 tx=0x7f25cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31bbfb5211f25204 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.984+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc03a070 con 0x7f25cc0053c0 2024-03-06T07:28:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.984+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc035410 con 0x7f25cc0053c0 2024-03-06T07:28:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.984+0000 7f25e3400640 1 -- 172.21.15.32:0/3663848571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc003dd0 con 0x7f25cc0053c0 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.985+0000 7f25eb600640 1 -- 172.21.15.32:0/3663848571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc0053c0 msgr2=0x7f25cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.985+0000 7f25eb600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc0053c0 0x7f25cc00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f25e415eaa0 tx=0x7f25cc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.985+0000 7f25eb600640 1 -- 172.21.15.32:0/3663848571 shutdown_connections 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.986+0000 7f25eb600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc0053c0 0x7f25cc00b9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.986+0000 7f25eb600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25cc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.986+0000 7f25eb600640 1 --2- 172.21.15.32:0/3663848571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc003a00 0x7f25cc003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.986+0000 7f25eb600640 1 -- 172.21.15.32:0/3663848571 >> 172.21.15.32:0/3663848571 conn(0x7f25e40d3e10 msgr2=0x7f25e40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.986+0000 7f25eb600640 1 -- 172.21.15.32:0/3663848571 shutdown_connections 2024-03-06T07:28:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.986+0000 7f25eb600640 1 -- 172.21.15.32:0/3663848571 wait complete. 2024-03-06T07:28:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.992+0000 7f25eb600640 1 Processor -- start 2024-03-06T07:28:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.992+0000 7f25eb600640 1 -- start start 2024-03-06T07:28:40.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.995+0000 7f25eb600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 0x7f25e40c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.995+0000 7f25eb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25e415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.995+0000 7f25eb600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0053c0 0x7f25e40c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.995+0000 7f25eb600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e415dfb0 con 0x7f25cc003a00 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.995+0000 7f25eb600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e40d0dc0 con 0x7f25cc0053c0 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.995+0000 7f25eb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e415e130 con 0x7f25cc004800 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 0x7f25e40c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 0x7f25e40c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41852/0 (socket says 172.21.15.32:41852) 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 -- 172.21.15.32:0/924781927 learned_addr learned my addr 172.21.15.32:0/924781927 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e3e00640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25e415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e9600640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0053c0 0x7f25e40c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 -- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 msgr2=0x7f25e415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25e415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 -- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0053c0 msgr2=0x7f25e40c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0053c0 0x7f25e40c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.996+0000 7f25e8c00640 1 -- 172.21.15.32:0/924781927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e415ca00 con 0x7f25cc003a00 2024-03-06T07:28:41.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.997+0000 7f25e8c00640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 0x7f25e40c93f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f25d8001eb0 tx=0x7f25d8015e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.997+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8013a40 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.997+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d8013be0 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.997+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8004020 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.997+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e40c8350 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.998+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e40c8910 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.998+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f25d80041c0 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.998+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f25d80179d0 con 0x7f25cc003a00 2024-03-06T07:28:41.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:40.999+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e404f330 con 0x7f25cc003a00 2024-03-06T07:28:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.004+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f25d8017bd0 con 0x7f25cc003a00 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.079+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f25e40b9b70 con 0x7f25cc003a00 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.083+0000 7f25e0c00640 1 -- 172.21.15.32:0/924781927 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f25d8017db0 con 0x7f25cc003a00 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.084+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 msgr2=0x7f25e40c93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.084+0000 7f25eb600640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 0x7f25e40c93f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f25d8001eb0 tx=0x7f25d8015e10 comp rx=0 tx=0).stop 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 shutdown_connections 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0053c0 0x7f25e40c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25cc004800 0x7f25e415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 --2- 172.21.15.32:0/924781927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25cc003a00 0x7f25e40c93f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 >> 172.21.15.32:0/924781927 conn(0x7f25e40d3e10 msgr2=0x7f25e415d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 shutdown_connections 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:41.085+0000 7f25eb600640 1 -- 172.21.15.32:0/924781927 wait complete. 2024-03-06T07:28:41.106 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:28:41.106 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:28:41.106 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:41.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:28:41.189 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:41.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:28:41.325 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:41.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:28:41.421 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:28:41.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:28:41.514 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:41.515 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:28:41.617 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:41.618 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:28:41.627 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:41.625+0000 7f8284649040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:28:41.697 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:41.695+0000 7f8284649040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:28:41.714 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:41.714 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:28:41.786 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:28:41.787 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:28:41.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 381a2d5a-6911-40e0-93b2-252fbd01807f 0 2024-03-06T07:28:42.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.154+0000 7f6c7a800640 1 Processor -- start 2024-03-06T07:28:42.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.155+0000 7f6c7a800640 1 -- start start 2024-03-06T07:28:42.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.156+0000 7f6c7a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6c74161840 0x7f6c740d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.156+0000 7f6c7a800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c740d4a40 0x7f6c740d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.156+0000 7f6c7a800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740c7a00 con 0x7f6c740c7f00 2024-03-06T07:28:42.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.156+0000 7f6c7a800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c74161c20 con 0x7f6c740d4a40 2024-03-06T07:28:42.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.156+0000 7f6c7a800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740d2530 con 0x7f6c74161840 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.157+0000 7f6c73400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6c74161840 0x7f6c740d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.157+0000 7f6c73e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6c740c7f00 0x7f6c740c8390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.39:56054/0 (socket says 172.21.15.39:56054) 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.157+0000 7f6c73e00640 1 -- 172.21.15.39:0/3907355588 learned_addr learned my addr 172.21.15.39:0/3907355588 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.157+0000 7f6c72a00640 1 --2- 172.21.15.39:0/3907355588 >> v2:172.21.15.32:3300/0 conn(0x7f6c740d4a40 0x7f6c740d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.157+0000 7f6c73400640 1 -- 172.21.15.39:0/3907355588 >> v2:172.21.15.32:3300/0 conn(0x7f6c740d4a40 msgr2=0x7f6c740d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.157+0000 7f6c73400640 1 --2- 172.21.15.39:0/3907355588 >> v2:172.21.15.32:3300/0 conn(0x7f6c740d4a40 0x7f6c740d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.158+0000 7f6c73400640 1 -- 172.21.15.39:0/3907355588 >> v1:172.21.15.32:6789/0 conn(0x7f6c740c7f00 legacy=0x7f6c740c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.158+0000 7f6c72000640 1 -- 172.21.15.39:0/3907355588 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260080082 0 0) 0x7f6c740c7a00 con 0x7f6c740c7f00 2024-03-06T07:28:42.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.158+0000 7f6c73400640 1 -- 172.21.15.39:0/3907355588 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740d2810 con 0x7f6c74161840 2024-03-06T07:28:42.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.159+0000 7f6c73400640 1 --2- 172.21.15.39:0/3907355588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c74161840 0x7f6c740d07f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6c7408da60 tx=0x7f6c6402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8cf776e3666b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.159+0000 7f6c72000640 1 -- 172.21.15.39:0/3907355588 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c640315a0 con 0x7f6c74161840 2024-03-06T07:28:42.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.159+0000 7f6c72000640 1 -- 172.21.15.39:0/3907355588 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64037070 con 0x7f6c74161840 2024-03-06T07:28:42.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.159+0000 7f6c72000640 1 -- 172.21.15.39:0/3907355588 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64031960 con 0x7f6c74161840 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 -- 172.21.15.39:0/3907355588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c74161840 msgr2=0x7f6c740d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 --2- 172.21.15.39:0/3907355588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c74161840 0x7f6c740d07f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6c7408da60 tx=0x7f6c6402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 -- 172.21.15.39:0/3907355588 shutdown_connections 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 --2- 172.21.15.39:0/3907355588 >> v2:172.21.15.32:3300/0 conn(0x7f6c740d4a40 0x7f6c740d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 --2- 172.21.15.39:0/3907355588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c74161840 0x7f6c740d07f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 -- 172.21.15.39:0/3907355588 >> 172.21.15.39:0/3907355588 conn(0x7f6c740d3eb0 msgr2=0x7f6c740d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 -- 172.21.15.39:0/3907355588 shutdown_connections 2024-03-06T07:28:42.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.160+0000 7f6c7a800640 1 -- 172.21.15.39:0/3907355588 wait complete. 2024-03-06T07:28:42.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.162+0000 7f6c7a800640 1 Processor -- start 2024-03-06T07:28:42.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.162+0000 7f6c7a800640 1 -- start start 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c7a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 0x7f6c740a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c7a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c740d4a40 0x7f6c740a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c7a800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c74161840 0x7f6c741f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c7a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740d1750 con 0x7f6c740d4a40 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c7a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740d2dd0 con 0x7f6c740c7f00 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c7a800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740cfad0 con 0x7f6c74161840 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c73400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 0x7f6c740a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c73e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c74161840 0x7f6c741f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.163+0000 7f6c73400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 0x7f6c740a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48500/0 (socket says 172.21.15.39:48500) 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 -- 172.21.15.39:0/1481001279 learned_addr learned my addr 172.21.15.39:0/1481001279 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c72a00640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c740d4a40 0x7f6c740a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 -- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c74161840 msgr2=0x7f6c741f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c74161840 0x7f6c741f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 -- 172.21.15.39:0/1481001279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c740d4a40 msgr2=0x7f6c740a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c740d4a40 0x7f6c740a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 -- 172.21.15.39:0/1481001279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740a0df0 con 0x7f6c740c7f00 2024-03-06T07:28:42.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.164+0000 7f6c73400640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 0x7f6c740a0370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009890 tx=0x7f6c640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64032630 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c640327d0 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64032bc0 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c740a10a0 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c740a1660 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6c64006020 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.165+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6c6402f030 con 0x7f6c740c7f00 2024-03-06T07:28:42.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.166+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c74068010 con 0x7f6c740c7f00 2024-03-06T07:28:42.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.169+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6c64040070 con 0x7f6c740c7f00 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.250+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "381a2d5a-6911-40e0-93b2-252fbd01807f", "id": 0} v 0) v1 -- 0x7f6c740ca7b0 con 0x7f6c740c7f00 2024-03-06T07:28:42.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.263+0000 7f6c6b400640 1 -- 172.21.15.39:0/1481001279 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "381a2d5a-6911-40e0-93b2-252fbd01807f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6c64032d60 con 0x7f6c740c7f00 2024-03-06T07:28:42.265 INFO:teuthology.orchestra.run.smithi039.stdout:0 2024-03-06T07:28:42.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 msgr2=0x7f6c740a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 0x7f6c740a0370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009890 tx=0x7f6c640046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 shutdown_connections 2024-03-06T07:28:42.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c74161840 0x7f6c741f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c740d4a40 0x7f6c740a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 --2- 172.21.15.39:0/1481001279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c740c7f00 0x7f6c740a0370 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.264+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 >> 172.21.15.39:0/1481001279 conn(0x7f6c740d3eb0 msgr2=0x7f6c740b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.265+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 shutdown_connections 2024-03-06T07:28:42.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.265+0000 7f6c7a800640 1 -- 172.21.15.39:0/1481001279 wait complete. 2024-03-06T07:28:42.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new ec67ac0c-0584-4a74-8116-1480561db4a3 1 2024-03-06T07:28:42.303 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:42.302+0000 7f8645e73040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:28:42.350 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.348+0000 7f8284649040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:28:42.376 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:42.375+0000 7f8645e73040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:28:42.418 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.415+0000 7f8284649040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:28:42.470 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.468+0000 7f8284649040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:28:42.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.486+0000 7f2304a00640 1 Processor -- start 2024-03-06T07:28:42.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.486+0000 7f2304a00640 1 -- start start 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.488+0000 7f2304a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2300161580 0x7f23000b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.488+0000 7f2304a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23000d4800 0x7f23000b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.488+0000 7f2304a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23000b3150 con 0x7f2300161960 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.488+0000 7f2304a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000d26f0 con 0x7f23000d4800 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.488+0000 7f2304a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000b2b90 con 0x7f2300161580 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2300161580 0x7f23000b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2300161580 0x7f23000b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48506/0 (socket says 172.21.15.39:48506) 2024-03-06T07:28:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 -- 172.21.15.39:0/2676829687 learned_addr learned my addr 172.21.15.39:0/2676829687 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:42.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ffe00640 1 --2- 172.21.15.39:0/2676829687 >> v2:172.21.15.32:3300/0 conn(0x7f23000d4800 0x7f23000b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 -- 172.21.15.39:0/2676829687 >> v2:172.21.15.32:3300/0 conn(0x7f23000d4800 msgr2=0x7f23000b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 --2- 172.21.15.39:0/2676829687 >> v2:172.21.15.32:3300/0 conn(0x7f23000d4800 0x7f23000b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 -- 172.21.15.39:0/2676829687 >> v1:172.21.15.32:6789/0 conn(0x7f2300161960 legacy=0x7f23000b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.489+0000 7f22ff400640 1 -- 172.21.15.39:0/2676829687 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000b2e70 con 0x7f2300161580 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.490+0000 7f22ff400640 1 --2- 172.21.15.39:0/2676829687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000b2240 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f230006b530 tx=0x7f22f002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67da38682ff338d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.490+0000 7f22fe000640 1 -- 172.21.15.39:0/2676829687 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f002e070 con 0x7f2300161580 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f22fe000640 1 -- 172.21.15.39:0/2676829687 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0037070 con 0x7f2300161580 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f22fe000640 1 -- 172.21.15.39:0/2676829687 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0031a00 con 0x7f2300161580 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 -- 172.21.15.39:0/2676829687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 msgr2=0x7f23000b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 --2- 172.21.15.39:0/2676829687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000b2240 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f230006b530 tx=0x7f22f002b880 comp rx=0 tx=0).stop 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 -- 172.21.15.39:0/2676829687 shutdown_connections 2024-03-06T07:28:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 --2- 172.21.15.39:0/2676829687 >> v2:172.21.15.32:3300/0 conn(0x7f23000d4800 0x7f23000b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 --2- 172.21.15.39:0/2676829687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000b2240 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 -- 172.21.15.39:0/2676829687 >> 172.21.15.39:0/2676829687 conn(0x7f23000d3c70 msgr2=0x7f23000d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 -- 172.21.15.39:0/2676829687 shutdown_connections 2024-03-06T07:28:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.491+0000 7f2304a00640 1 -- 172.21.15.39:0/2676829687 wait complete. 2024-03-06T07:28:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.493+0000 7f2304a00640 1 Processor -- start 2024-03-06T07:28:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.493+0000 7f2304a00640 1 -- start start 2024-03-06T07:28:42.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f2304a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23000d4800 0x7f23001f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f2304a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f2304a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 0x7f23000cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f2304a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000b1470 con 0x7f23000d4800 2024-03-06T07:28:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f2304a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000b2780 con 0x7f2300161580 2024-03-06T07:28:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f2304a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23000b15f0 con 0x7f2300161960 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f22ffe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 0x7f23000cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f22fea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f22ffe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 0x7f23000cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.39:49994/0 (socket says 172.21.15.39:49994) 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.494+0000 7f22ffe00640 1 -- 172.21.15.39:0/3903325746 learned_addr learned my addr 172.21.15.39:0/3903325746 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22ff400640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23000d4800 0x7f23001f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22fea00640 1 -- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 msgr2=0x7f23000cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22fea00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 0x7f23000cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22fea00640 1 -- 172.21.15.39:0/3903325746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23000d4800 msgr2=0x7f23001f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22fea00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23000d4800 0x7f23001f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22fea00640 1 -- 172.21.15.39:0/3903325746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000c7940 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22ffe00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 0x7f23000cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.495+0000 7f22fea00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000cc990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00b3e0 tx=0x7f22ec00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec015030 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec004020 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec004410 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23000c7c20 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23000cd410 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f22ec018070 con 0x7f2300161580 2024-03-06T07:28:42.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.496+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f22ec003ad0 con 0x7f2300161580 2024-03-06T07:28:42.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.497+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23000d1630 con 0x7f2300161580 2024-03-06T07:28:42.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.499+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f22ec0035f0 con 0x7f2300161580 2024-03-06T07:28:42.537 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.535+0000 7f8284649040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:28:42.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.581+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ec67ac0c-0584-4a74-8116-1480561db4a3", "id": 1} v 0) v1 -- 0x7f230004f330 con 0x7f2300161580 2024-03-06T07:28:42.589 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.587+0000 7f8284649040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:28:42.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.591+0000 7f22f7400640 1 -- 172.21.15.39:0/3903325746 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ec67ac0c-0584-4a74-8116-1480561db4a3", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f22ec003da0 con 0x7f2300161580 2024-03-06T07:28:42.593 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-03-06T07:28:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 msgr2=0x7f23000cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000cc990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00b3e0 tx=0x7f22ec00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 shutdown_connections 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2300161960 0x7f23000cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2300161580 0x7f23000cc990 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 --2- 172.21.15.39:0/3903325746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23000d4800 0x7f23001f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 >> 172.21.15.39:0/3903325746 conn(0x7f23000d3c70 msgr2=0x7f23000d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 shutdown_connections 2024-03-06T07:28:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.592+0000 7f2304a00640 1 -- 172.21.15.39:0/3903325746 wait complete. 2024-03-06T07:28:42.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new c45c108c-3f06-4132-9202-b9553ebeb4eb 2 2024-03-06T07:28:42.726 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.724+0000 7f8284649040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:28:42.810 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.808+0000 7f8284649040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:28:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.849+0000 7fe26a000640 1 Processor -- start 2024-03-06T07:28:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.849+0000 7fe26a000640 1 -- start start 2024-03-06T07:28:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe26a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe26415e130 0x7fe264071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe26a000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26406b050 con 0x7fe2640b51b0 2024-03-06T07:28:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe26a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26406b330 con 0x7fe2640cfff0 2024-03-06T07:28:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe26a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26406b610 con 0x7fe26415e130 2024-03-06T07:28:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe263e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe26415e130 0x7fe264071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe269600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe2640cfff0 0x7fe26415e950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58522/0 (socket says 172.21.15.39:58522) 2024-03-06T07:28:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.851+0000 7fe269600640 1 -- 172.21.15.39:0/3197983054 learned_addr learned my addr 172.21.15.39:0/3197983054 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-03-06T07:28:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.852+0000 7fe263e00640 1 -- 172.21.15.39:0/3197983054 >> v1:172.21.15.39:6789/0 conn(0x7fe2640cfff0 legacy=0x7fe26415e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.852+0000 7fe263e00640 1 -- 172.21.15.39:0/3197983054 >> v1:172.21.15.32:6789/0 conn(0x7fe2640b51b0 legacy=0x7fe2640b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.852+0000 7fe263e00640 1 -- 172.21.15.39:0/3197983054 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26406b8f0 con 0x7fe26415e130 2024-03-06T07:28:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.853+0000 7fe263e00640 1 --2- 172.21.15.39:0/3197983054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe26415e130 0x7fe264071a20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe250009800 tx=0x7fe25002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486cd2039eedd066 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.853+0000 7fe263400640 1 -- 172.21.15.39:0/3197983054 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250034070 con 0x7fe26415e130 2024-03-06T07:28:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.853+0000 7fe263400640 1 -- 172.21.15.39:0/3197983054 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250030800 con 0x7fe26415e130 2024-03-06T07:28:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.853+0000 7fe263400640 1 -- 172.21.15.39:0/3197983054 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250030bd0 con 0x7fe26415e130 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 -- 172.21.15.39:0/3197983054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe26415e130 msgr2=0x7fe264071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 --2- 172.21.15.39:0/3197983054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe26415e130 0x7fe264071a20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe250009800 tx=0x7fe25002b940 comp rx=0 tx=0).stop 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 -- 172.21.15.39:0/3197983054 shutdown_connections 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 --2- 172.21.15.39:0/3197983054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe26415e130 0x7fe264071a20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 -- 172.21.15.39:0/3197983054 >> 172.21.15.39:0/3197983054 conn(0x7fe2640d0980 msgr2=0x7fe2640d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 -- 172.21.15.39:0/3197983054 shutdown_connections 2024-03-06T07:28:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.854+0000 7fe26a000640 1 -- 172.21.15.39:0/3197983054 wait complete. 2024-03-06T07:28:42.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.856+0000 7fe26a000640 1 Processor -- start 2024-03-06T07:28:42.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.856+0000 7fe26a000640 1 -- start start 2024-03-06T07:28:42.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe26a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2640b51b0 0x7fe26415f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe26a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 0x7fe264152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe26a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe26415e130 0x7fe2641f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe26a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640d1780 con 0x7fe2640b51b0 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe26a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640d1600 con 0x7fe26415e130 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe26a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640d0480 con 0x7fe2640cfff0 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe263e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 0x7fe264152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe269600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe26415e130 0x7fe2641f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe263e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 0x7fe264152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.39:50012/0 (socket says 172.21.15.39:50012) 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.857+0000 7fe263e00640 1 -- 172.21.15.39:0/673132695 learned_addr learned my addr 172.21.15.39:0/673132695 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe268c00640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2640b51b0 0x7fe26415f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe263e00640 1 -- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe26415e130 msgr2=0x7fe2641f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe263e00640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe26415e130 0x7fe2641f7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe263e00640 1 -- 172.21.15.39:0/673132695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2640b51b0 msgr2=0x7fe26415f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe263e00640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2640b51b0 0x7fe26415f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe263e00640 1 -- 172.21.15.39:0/673132695 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2641e7a60 con 0x7fe2640cfff0 2024-03-06T07:28:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe269600640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe26415e130 0x7fe2641f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.858+0000 7fe263e00640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 0x7fe264152300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe25002bf50 tx=0x7fe2500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250034040 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2500443f0 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500447e0 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe264161eb0 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe264162470 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe25004b060 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.859+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe2500306c0 con 0x7fe2640cfff0 2024-03-06T07:28:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.860+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26404f330 con 0x7fe2640cfff0 2024-03-06T07:28:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.863+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe25003f070 con 0x7fe2640cfff0 2024-03-06T07:28:42.866 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.864+0000 7f8284649040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:28:42.930 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.928+0000 7f8284649040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:28:42.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.943+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c45c108c-3f06-4132-9202-b9553ebeb4eb", "id": 2} v 0) v1 -- 0x7fe2640b3c00 con 0x7fe2640cfff0 2024-03-06T07:28:42.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.951+0000 7fe260c00640 1 -- 172.21.15.39:0/673132695 <== mon.2 v2:172.21.15.39:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c45c108c-3f06-4132-9202-b9553ebeb4eb", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe250044980 con 0x7fe2640cfff0 2024-03-06T07:28:42.953 INFO:teuthology.orchestra.run.smithi039.stdout:2 2024-03-06T07:28:42.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.952+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 msgr2=0x7fe264152300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.952+0000 7fe26a000640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 0x7fe264152300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe25002bf50 tx=0x7fe2500046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.952+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 shutdown_connections 2024-03-06T07:28:42.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.952+0000 7fe26a000640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe26415e130 0x7fe2641f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.952+0000 7fe26a000640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2640cfff0 0x7fe264152300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.952+0000 7fe26a000640 1 --2- 172.21.15.39:0/673132695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2640b51b0 0x7fe26415f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.953+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 >> 172.21.15.39:0/673132695 conn(0x7fe2640d0980 msgr2=0x7fe2640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.953+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 shutdown_connections 2024-03-06T07:28:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:42.953+0000 7fe26a000640 1 -- 172.21.15.39:0/673132695 wait complete. 2024-03-06T07:28:42.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 27319d10-b65b-4934-ab15-58281f33ecc1 3 2024-03-06T07:28:42.989 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:42.986+0000 7f8284649040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:28:43.041 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.040+0000 7f8645e73040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:28:43.110 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.109+0000 7f8645e73040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:28:43.165 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.164+0000 7f8645e73040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:28:43.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.167+0000 7f4694a00640 1 Processor -- start 2024-03-06T07:28:43.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.168+0000 7f4694a00640 1 -- start start 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f4694a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4690161840 0x7f46900d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f4694a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46900c7a00 con 0x7f46900d4a40 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f4694a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4690161c20 con 0x7f46900cfad0 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f4694a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46900d2530 con 0x7f4690161840 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4690161840 0x7f46900d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4690161840 0x7f46900d07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48532/0 (socket says 172.21.15.39:48532) 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468de00640 1 -- 172.21.15.39:0/1504404595 learned_addr learned my addr 172.21.15.39:0/1504404595 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:43.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468e800640 1 --1- 172.21.15.39:0/1504404595 >> v1:172.21.15.39:6789/0 conn(0x7f46900cfad0 0x7f46900c8610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58526/0 (socket says 172.21.15.39:58526) 2024-03-06T07:28:43.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468de00640 1 -- 172.21.15.39:0/1504404595 >> v1:172.21.15.39:6789/0 conn(0x7f46900cfad0 legacy=0x7f46900c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468de00640 1 -- 172.21.15.39:0/1504404595 >> v1:172.21.15.32:6789/0 conn(0x7f46900d4a40 legacy=0x7f46900c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.170+0000 7f468de00640 1 -- 172.21.15.39:0/1504404595 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900d2810 con 0x7f4690161840 2024-03-06T07:28:43.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.169+0000 7f468ca00640 1 -- 172.21.15.39:0/1504404595 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705180099 0 0) 0x7f4690161c20 con 0x7f46900cfad0 2024-03-06T07:28:43.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f468de00640 1 --2- 172.21.15.39:0/1504404595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900d07f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4680009870 tx=0x7f468002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f42a77c94f968956 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f468ca00640 1 -- 172.21.15.39:0/1504404595 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680034070 con 0x7f4690161840 2024-03-06T07:28:43.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f468ca00640 1 -- 172.21.15.39:0/1504404595 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680030860 con 0x7f4690161840 2024-03-06T07:28:43.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f468ca00640 1 -- 172.21.15.39:0/1504404595 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680030c30 con 0x7f4690161840 2024-03-06T07:28:43.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f4694a00640 1 -- 172.21.15.39:0/1504404595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 msgr2=0x7f46900d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f4694a00640 1 --2- 172.21.15.39:0/1504404595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900d07f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4680009870 tx=0x7f468002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.171+0000 7f4694a00640 1 -- 172.21.15.39:0/1504404595 shutdown_connections 2024-03-06T07:28:43.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.172+0000 7f4694a00640 1 --2- 172.21.15.39:0/1504404595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900d07f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.172+0000 7f4694a00640 1 -- 172.21.15.39:0/1504404595 >> 172.21.15.39:0/1504404595 conn(0x7f46900d3eb0 msgr2=0x7f46900d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.172+0000 7f4694a00640 1 -- 172.21.15.39:0/1504404595 shutdown_connections 2024-03-06T07:28:43.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.172+0000 7f4694a00640 1 -- 172.21.15.39:0/1504404595 wait complete. 2024-03-06T07:28:43.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.173+0000 7f4694a00640 1 Processor -- start 2024-03-06T07:28:43.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.173+0000 7f4694a00640 1 -- start start 2024-03-06T07:28:43.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f4694a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46900cfad0 0x7f469014f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f4694a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 0x7f46900b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f4694a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f4694a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46900d1750 con 0x7f46900cfad0 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f4694a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46900d2dd0 con 0x7f4690161840 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f4694a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46900d0d30 con 0x7f46900d4a40 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f468e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f468d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 0x7f46900b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f468e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48542/0 (socket says 172.21.15.39:48542) 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f468d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 0x7f46900b8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.39:50024/0 (socket says 172.21.15.39:50024) 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.174+0000 7f468e800640 1 -- 172.21.15.39:0/3113523822 learned_addr learned my addr 172.21.15.39:0/3113523822 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468de00640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46900cfad0 0x7f469014f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468e800640 1 -- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 msgr2=0x7f46900b8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468e800640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 0x7f46900b8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468e800640 1 -- 172.21.15.39:0/3113523822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46900cfad0 msgr2=0x7f469014f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468e800640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46900cfad0 0x7f469014f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468e800640 1 -- 172.21.15.39:0/3113523822 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900b7aa0 con 0x7f4690161840 2024-03-06T07:28:43.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468d400640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 0x7f46900b8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.175+0000 7f468e800640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900cd470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f468800c7a0 tx=0x7f468800cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688018070 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468800a640 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46900b7d80 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468800aa70 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46900cb260 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f468801b070 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.176+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f46880078a0 con 0x7f4690161840 2024-03-06T07:28:43.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.177+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4690074620 con 0x7f4690161840 2024-03-06T07:28:43.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.180+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4690074620 con 0x7f4690161840 2024-03-06T07:28:43.234 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.233+0000 7f8645e73040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:28:43.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.259+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "27319d10-b65b-4934-ab15-58281f33ecc1", "id": 3} v 0) v1 -- 0x7f46900d1570 con 0x7f4690161840 2024-03-06T07:28:43.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.272+0000 7f4686000640 1 -- 172.21.15.39:0/3113523822 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27319d10-b65b-4934-ab15-58281f33ecc1", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f46900d1570 con 0x7f4690161840 2024-03-06T07:28:43.274 INFO:teuthology.orchestra.run.smithi039.stdout:3 2024-03-06T07:28:43.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.273+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 msgr2=0x7f46900cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.273+0000 7f4694a00640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900cd470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f468800c7a0 tx=0x7f468800cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:43.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 shutdown_connections 2024-03-06T07:28:43.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4690161840 0x7f46900cd470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46900d4a40 0x7f46900b8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 --2- 172.21.15.39:0/3113523822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46900cfad0 0x7f469014f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 >> 172.21.15.39:0/3113523822 conn(0x7f46900d3eb0 msgr2=0x7f469015f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 shutdown_connections 2024-03-06T07:28:43.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.274+0000 7f4694a00640 1 -- 172.21.15.39:0/3113523822 wait complete. 2024-03-06T07:28:43.289 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.287+0000 7f8645e73040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:28:43.291 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new c234adf2-055d-4dd0-adac-7e68bc54ebd7 4 2024-03-06T07:28:43.429 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.428+0000 7f8645e73040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:28:43.462 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:43.459+0000 7f8284649040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:28:43.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.468+0000 7f2da0600640 1 Processor -- start 2024-03-06T07:28:43.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.468+0000 7f2da0600640 1 -- start start 2024-03-06T07:28:43.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2da0600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2d9815bb90 0x7f2d9806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2da0600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2d9815efd0 0x7f2d9806b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2da0600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9815f4a0 con 0x7f2d9815bf70 2024-03-06T07:28:43.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2da0600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d981521d0 con 0x7f2d9815efd0 2024-03-06T07:28:43.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2da0600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9815f780 con 0x7f2d9815bb90 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2d9dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2d9815bb90 0x7f2d9806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2d9dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2d9815bb90 0x7f2d9806b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48552/0 (socket says 172.21.15.39:48552) 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.470+0000 7f2d9dc00640 1 -- 172.21.15.39:0/2490471707 learned_addr learned my addr 172.21.15.39:0/2490471707 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.471+0000 7f2d9e600640 1 --2- 172.21.15.39:0/2490471707 >> v2:172.21.15.32:3300/0 conn(0x7f2d9815efd0 0x7f2d9806b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.471+0000 7f2d9dc00640 1 -- 172.21.15.39:0/2490471707 >> v2:172.21.15.32:3300/0 conn(0x7f2d9815efd0 msgr2=0x7f2d9806b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.471+0000 7f2d9dc00640 1 --2- 172.21.15.39:0/2490471707 >> v2:172.21.15.32:3300/0 conn(0x7f2d9815efd0 0x7f2d9806b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.471+0000 7f2d9dc00640 1 -- 172.21.15.39:0/2490471707 >> v1:172.21.15.32:6789/0 conn(0x7f2d9815bf70 legacy=0x7f2d9815e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.471+0000 7f2d9dc00640 1 -- 172.21.15.39:0/2490471707 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98072050 con 0x7f2d9815bb90 2024-03-06T07:28:43.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.472+0000 7f2d9dc00640 1 --2- 172.21.15.39:0/2490471707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bb90 0x7f2d9806b1f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d98071ef0 tx=0x7f2d8c02e060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=549859cf6a0081da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.472+0000 7f2d97e00640 1 -- 172.21.15.39:0/2490471707 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c03b070 con 0x7f2d9815bb90 2024-03-06T07:28:43.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.472+0000 7f2d97e00640 1 -- 172.21.15.39:0/2490471707 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c02ee10 con 0x7f2d9815bb90 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.472+0000 7f2d97e00640 1 -- 172.21.15.39:0/2490471707 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c036660 con 0x7f2d9815bb90 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 -- 172.21.15.39:0/2490471707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bb90 msgr2=0x7f2d9806b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 --2- 172.21.15.39:0/2490471707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bb90 0x7f2d9806b1f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d98071ef0 tx=0x7f2d8c02e060 comp rx=0 tx=0).stop 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 -- 172.21.15.39:0/2490471707 shutdown_connections 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 --2- 172.21.15.39:0/2490471707 >> v2:172.21.15.32:3300/0 conn(0x7f2d9815efd0 0x7f2d9806b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 --2- 172.21.15.39:0/2490471707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bb90 0x7f2d9806b1f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 -- 172.21.15.39:0/2490471707 >> 172.21.15.39:0/2490471707 conn(0x7f2d980b1ad0 msgr2=0x7f2d980b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 -- 172.21.15.39:0/2490471707 shutdown_connections 2024-03-06T07:28:43.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.473+0000 7f2da0600640 1 -- 172.21.15.39:0/2490471707 wait complete. 2024-03-06T07:28:43.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.475+0000 7f2da0600640 1 Processor -- start 2024-03-06T07:28:43.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.475+0000 7f2da0600640 1 -- start start 2024-03-06T07:28:43.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2da0600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 0x7f2d981f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2da0600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bf70 0x7f2d9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2da0600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d9815efd0 0x7f2d9806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2da0600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98072610 con 0x7f2d9815efd0 2024-03-06T07:28:43.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2da0600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9815b660 con 0x7f2d9815bf70 2024-03-06T07:28:43.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2da0600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98072b30 con 0x7f2d9815bb90 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 0x7f2d981f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bf70 0x7f2d9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 0x7f2d981f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.39:50034/0 (socket says 172.21.15.39:50034) 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 -- 172.21.15.39:0/3260063843 learned_addr learned my addr 172.21.15.39:0/3260063843 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9e600640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d9815efd0 0x7f2d9806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 -- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bf70 msgr2=0x7f2d9806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bf70 0x7f2d9806d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 -- 172.21.15.39:0/3260063843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d9815efd0 msgr2=0x7f2d9806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d9815efd0 0x7f2d9806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.476+0000 7f2d9dc00640 1 -- 172.21.15.39:0/3260063843 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d981f54b0 con 0x7f2d9815bb90 2024-03-06T07:28:43.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.477+0000 7f2d9dc00640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 0x7f2d981f4f70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c02e0a0 tx=0x7f2d8c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.477+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c03b040 con 0x7f2d9815bb90 2024-03-06T07:28:43.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.477+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c046070 con 0x7f2d9815bb90 2024-03-06T07:28:43.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.477+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c041690 con 0x7f2d9815bb90 2024-03-06T07:28:43.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.477+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d981527e0 con 0x7f2d9815bb90 2024-03-06T07:28:43.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.477+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d98152da0 con 0x7f2d9815bb90 2024-03-06T07:28:43.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.478+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2d8c041830 con 0x7f2d9815bb90 2024-03-06T07:28:43.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.478+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f2d8c04b460 con 0x7f2d9815bb90 2024-03-06T07:28:43.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.478+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d98068010 con 0x7f2d9815bb90 2024-03-06T07:28:43.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.481+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2d981f54b0 con 0x7f2d9815bb90 2024-03-06T07:28:43.517 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.516+0000 7f8645e73040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.560+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c234adf2-055d-4dd0-adac-7e68bc54ebd7", "id": 4} v 0) v1 -- 0x7f2d9809d820 con 0x7f2d9815bb90 2024-03-06T07:28:43.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.570+0000 7f2d95600640 1 -- 172.21.15.39:0/3260063843 <== mon.2 v2:172.21.15.39:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c234adf2-055d-4dd0-adac-7e68bc54ebd7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2d8c041b20 con 0x7f2d9815bb90 2024-03-06T07:28:43.571 INFO:teuthology.orchestra.run.smithi039.stdout:4 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 msgr2=0x7f2d981f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 0x7f2d981f4f70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c02e0a0 tx=0x7f2d8c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 shutdown_connections 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d9815efd0 0x7f2d9806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d9815bf70 0x7f2d9806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 --2- 172.21.15.39:0/3260063843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2d9815bb90 0x7f2d981f4f70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 >> 172.21.15.39:0/3260063843 conn(0x7f2d980b1ad0 msgr2=0x7f2d9806b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 shutdown_connections 2024-03-06T07:28:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.571+0000 7f2da0600640 1 -- 172.21.15.39:0/3260063843 wait complete. 2024-03-06T07:28:43.574 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.573+0000 7f8645e73040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:28:43.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 3be1b0dc-2152-420f-b869-5cb82c5136a4 5 2024-03-06T07:28:43.632 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:43.629+0000 7f8284649040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:28:43.640 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.638+0000 7f8645e73040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:28:43.699 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:43.698+0000 7f8645e73040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:28:43.769 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:43.767+0000 7f8284649040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.773+0000 7f8f7de00640 1 Processor -- start 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.774+0000 7f8f7de00640 1 -- start start 2024-03-06T07:28:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.775+0000 7f8f7de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8f780d4800 0x7f8f780b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.775+0000 7f8f7de00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780b3150 con 0x7f8f78161960 2024-03-06T07:28:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.775+0000 7f8f7de00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780b2b90 con 0x7f8f78161580 2024-03-06T07:28:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.775+0000 7f8f7de00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f780d26f0 con 0x7f8f780d4800 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f7ca00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8f78161580 0x7f8f780b2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58532/0 (socket says 172.21.15.39:58532) 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f7ca00640 1 -- 172.21.15.39:0/2109590209 learned_addr learned my addr 172.21.15.39:0/2109590209 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f7d400640 1 --2- 172.21.15.39:0/2109590209 >> v2:172.21.15.39:3300/0 conn(0x7f8f780d4800 0x7f8f780b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f77400640 1 -- 172.21.15.39:0/2109590209 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356081006 0 0) 0x7f8f780b2b90 con 0x7f8f78161580 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f7d400640 1 -- 172.21.15.39:0/2109590209 >> v1:172.21.15.39:6789/0 conn(0x7f8f78161580 legacy=0x7f8f780b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f7d400640 1 -- 172.21.15.39:0/2109590209 >> v1:172.21.15.32:6789/0 conn(0x7f8f78161960 legacy=0x7f8f780b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.776+0000 7f8f7d400640 1 -- 172.21.15.39:0/2109590209 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780b2e70 con 0x7f8f780d4800 2024-03-06T07:28:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.777+0000 7f8f7d400640 1 --2- 172.21.15.39:0/2109590209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f780b4bb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f780d2870 tx=0x7f8f7002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f5b88f6478e1959 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.777+0000 7f8f77400640 1 -- 172.21.15.39:0/2109590209 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70039070 con 0x7f8f780d4800 2024-03-06T07:28:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.777+0000 7f8f77400640 1 -- 172.21.15.39:0/2109590209 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7002cc60 con 0x7f8f780d4800 2024-03-06T07:28:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.777+0000 7f8f77400640 1 -- 172.21.15.39:0/2109590209 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70034660 con 0x7f8f780d4800 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 -- 172.21.15.39:0/2109590209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 msgr2=0x7f8f780b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 --2- 172.21.15.39:0/2109590209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f780b4bb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f780d2870 tx=0x7f8f7002beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 -- 172.21.15.39:0/2109590209 shutdown_connections 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 --2- 172.21.15.39:0/2109590209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f780b4bb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 -- 172.21.15.39:0/2109590209 >> 172.21.15.39:0/2109590209 conn(0x7f8f780d3c70 msgr2=0x7f8f780d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 -- 172.21.15.39:0/2109590209 shutdown_connections 2024-03-06T07:28:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.778+0000 7f8f7de00640 1 -- 172.21.15.39:0/2109590209 wait complete. 2024-03-06T07:28:43.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.780+0000 7f8f7de00640 1 Processor -- start 2024-03-06T07:28:43.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.780+0000 7f8f7de00640 1 -- start start 2024-03-06T07:28:43.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f7814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7de00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8f78161580 0x7f8f781517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f78161960 0x7f8f781568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780b5270 con 0x7f8f78161960 2024-03-06T07:28:43.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7de00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780b50f0 con 0x7f8f780d4800 2024-03-06T07:28:43.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7de00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78168e40 con 0x7f8f78161580 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f7814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f7814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48574/0 (socket says 172.21.15.39:48574) 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7ca00640 1 -- 172.21.15.39:0/604935912 learned_addr learned my addr 172.21.15.39:0/604935912 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f77e00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8f78161580 0x7f8f781517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.781+0000 7f8f7d400640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f78161960 0x7f8f781568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f77e00640 1 -- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 msgr2=0x7f8f7814f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f77e00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f7814f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f77e00640 1 -- 172.21.15.39:0/604935912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f78161960 msgr2=0x7f8f781568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f77e00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f78161960 0x7f8f781568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f77e00640 1 -- 172.21.15.39:0/604935912 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f78150730 con 0x7f8f78161580 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f7ca00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f7814f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f77e00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8f78161580 0x7f8f781517d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400c390 tx=0x7f8f6400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64017070 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f640052d0 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64003830 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f78150a10 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.782+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f78150fd0 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.783+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8f6401a070 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.783+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f8f64011590 con 0x7f8f78161580 2024-03-06T07:28:43.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.784+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7804f330 con 0x7f8f78161580 2024-03-06T07:28:43.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.788+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8f64021070 con 0x7f8f78161580 2024-03-06T07:28:43.825 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:43.823+0000 7f8284649040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:28:43.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.868+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3be1b0dc-2152-420f-b869-5cb82c5136a4", "id": 5} v 0) v1 -- 0x7f8f780d06a0 con 0x7f8f78161580 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.879+0000 7f8f74c00640 1 -- 172.21.15.39:0/604935912 <== mon.2 v2:172.21.15.39:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3be1b0dc-2152-420f-b869-5cb82c5136a4", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f64011bd0 con 0x7f8f78161580 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi039.stdout:5 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8f78161580 msgr2=0x7f8f781517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8f78161580 0x7f8f781517d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400c390 tx=0x7f8f6400c850 comp rx=0 tx=0).stop 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 shutdown_connections 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f78161960 0x7f8f781568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8f78161580 0x7f8f781517d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 --2- 172.21.15.39:0/604935912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f780d4800 0x7f8f7814f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 >> 172.21.15.39:0/604935912 conn(0x7f8f780d3c70 msgr2=0x7f8f780b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 shutdown_connections 2024-03-06T07:28:43.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:43.880+0000 7f8f7de00640 1 -- 172.21.15.39:0/604935912 wait complete. 2024-03-06T07:28:43.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 39783030-63d8-4a9d-8909-7bac4d53da31 6 2024-03-06T07:28:43.959 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:43.957+0000 7f8284649040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:28:44.076 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:44.074+0000 7f8284649040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:28:44.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.088+0000 7f6996400640 1 Processor -- start 2024-03-06T07:28:44.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.089+0000 7f6996400640 1 -- start start 2024-03-06T07:28:44.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6996400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f69900d2180 0x7f69900d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6996400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f699015e1e0 0x7f69900b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6996400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69900b1480 con 0x7f69900b3040 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6996400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69900b1230 con 0x7f699015e1e0 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6996400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69900b1db0 con 0x7f69900d2180 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6995000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f69900d2180 0x7f69900d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6995a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f69900b3040 0x7f69900d2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58548/0 (socket says 172.21.15.39:58548) 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f6995a00640 1 -- 172.21.15.39:0/2120578155 learned_addr learned my addr 172.21.15.39:0/2120578155 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.090+0000 7f698fe00640 1 --2- 172.21.15.39:0/2120578155 >> v2:172.21.15.32:3300/0 conn(0x7f699015e1e0 0x7f69900b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.091+0000 7f6995000640 1 -- 172.21.15.39:0/2120578155 >> v2:172.21.15.32:3300/0 conn(0x7f699015e1e0 msgr2=0x7f69900b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.091+0000 7f6995000640 1 --2- 172.21.15.39:0/2120578155 >> v2:172.21.15.32:3300/0 conn(0x7f699015e1e0 0x7f69900b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.091+0000 7f698f400640 1 -- 172.21.15.39:0/2120578155 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747337704 0 0) 0x7f69900b1480 con 0x7f69900b3040 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.091+0000 7f6995000640 1 -- 172.21.15.39:0/2120578155 >> v1:172.21.15.39:6789/0 conn(0x7f69900b3040 legacy=0x7f69900d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.091+0000 7f6995000640 1 -- 172.21.15.39:0/2120578155 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69900b1760 con 0x7f69900d2180 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.092+0000 7f6995000640 1 --2- 172.21.15.39:0/2120578155 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f69900d2180 0x7f69900d1980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69840097a0 tx=0x7f698402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=304d81e39fdfab1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.092+0000 7f698f400640 1 -- 172.21.15.39:0/2120578155 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984035020 con 0x7f69900d2180 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.092+0000 7f698f400640 1 -- 172.21.15.39:0/2120578155 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6984031830 con 0x7f69900d2180 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.092+0000 7f698f400640 1 -- 172.21.15.39:0/2120578155 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984031c00 con 0x7f69900d2180 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 -- 172.21.15.39:0/2120578155 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f69900d2180 msgr2=0x7f69900d1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 --2- 172.21.15.39:0/2120578155 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f69900d2180 0x7f69900d1980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69840097a0 tx=0x7f698402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 -- 172.21.15.39:0/2120578155 shutdown_connections 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 --2- 172.21.15.39:0/2120578155 >> v2:172.21.15.32:3300/0 conn(0x7f699015e1e0 0x7f69900b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 --2- 172.21.15.39:0/2120578155 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f69900d2180 0x7f69900d1980 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 -- 172.21.15.39:0/2120578155 >> 172.21.15.39:0/2120578155 conn(0x7f69900d3c30 msgr2=0x7f69900d4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 -- 172.21.15.39:0/2120578155 shutdown_connections 2024-03-06T07:28:44.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.093+0000 7f6996400640 1 -- 172.21.15.39:0/2120578155 wait complete. 2024-03-06T07:28:44.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.095+0000 7f6996400640 1 Processor -- start 2024-03-06T07:28:44.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.095+0000 7f6996400640 1 -- start start 2024-03-06T07:28:44.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6996400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69900b3040 0x7f69900ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6996400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 0x7f6990150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6996400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f699015e1e0 0x7f699014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6996400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900cfae0 con 0x7f69900b3040 2024-03-06T07:28:44.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6996400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900b6fc0 con 0x7f69900d2180 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6996400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69900b2240 con 0x7f699015e1e0 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f698fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 0x7f6990150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f699015e1e0 0x7f699014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f698fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 0x7f6990150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48588/0 (socket says 172.21.15.39:48588) 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f698fe00640 1 -- 172.21.15.39:0/3301355537 learned_addr learned my addr 172.21.15.39:0/3301355537 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995000640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69900b3040 0x7f69900ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995a00640 1 -- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 msgr2=0x7f6990150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995a00640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 0x7f6990150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995a00640 1 -- 172.21.15.39:0/3301355537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69900b3040 msgr2=0x7f69900ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995a00640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69900b3040 0x7f69900ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995a00640 1 -- 172.21.15.39:0/3301355537 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69900ce370 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f698fe00640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 0x7f6990150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.096+0000 7f6995000640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69900b3040 0x7f69900ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f6995a00640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f699015e1e0 0x7f699014d780 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f698800c6d0 tx=0x7f698800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988018070 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6988004d60 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69900cd2d0 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698800a880 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69900cd890 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.097+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f698801b070 con 0x7f699015e1e0 2024-03-06T07:28:44.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.098+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f6988007990 con 0x7f699015e1e0 2024-03-06T07:28:44.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.098+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699004f330 con 0x7f699015e1e0 2024-03-06T07:28:44.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.100+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f698800ac30 con 0x7f699015e1e0 2024-03-06T07:28:44.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.180+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "osd new", "uuid": "39783030-63d8-4a9d-8909-7bac4d53da31", "id": 6} v 0) v1 -- 0x7f6990074180 con 0x7f699015e1e0 2024-03-06T07:28:44.188 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:44.186+0000 7f8645e73040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:28:44.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.187+0000 7f698cc00640 1 -- 172.21.15.39:0/3301355537 <== mon.2 v2:172.21.15.39:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "39783030-63d8-4a9d-8909-7bac4d53da31", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f69880070d0 con 0x7f699015e1e0 2024-03-06T07:28:44.189 INFO:teuthology.orchestra.run.smithi039.stdout:6 2024-03-06T07:28:44.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f699015e1e0 msgr2=0x7f699014d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f699015e1e0 0x7f699014d780 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f698800c6d0 tx=0x7f698800cb90 comp rx=0 tx=0).stop 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 shutdown_connections 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f699015e1e0 0x7f699014d780 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69900d2180 0x7f6990150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 --2- 172.21.15.39:0/3301355537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69900b3040 0x7f69900ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 >> 172.21.15.39:0/3301355537 conn(0x7f69900d3c30 msgr2=0x7f69900ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 shutdown_connections 2024-03-06T07:28:44.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.189+0000 7f6996400640 1 -- 172.21.15.39:0/3301355537 wait complete. 2024-03-06T07:28:44.208 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new f10c34e2-acc4-460e-97f6-7da3d7ae631c 7 2024-03-06T07:28:44.262 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:44.259+0000 7f8284649040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:28:44.314 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:44.312+0000 7f8284649040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:28:44.361 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:44.359+0000 7f8645e73040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:28:44.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.404+0000 7f09d9600640 1 Processor -- start 2024-03-06T07:28:44.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.404+0000 7f09d9600640 1 -- start start 2024-03-06T07:28:44.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d9600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 0x7f09d40d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d9600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09d40d4a40 0x7f09d40d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d9600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f09d40c7f00 0x7f09d40c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d9600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d40c7a00 con 0x7f09d40d4a40 2024-03-06T07:28:44.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d9600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d4161c20 con 0x7f09d40c7f00 2024-03-06T07:28:44.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d9600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40d2530 con 0x7f09d4161840 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d2a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 0x7f09d40d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d2a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 0x7f09d40d07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.39:50072/0 (socket says 172.21.15.39:50072) 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d2a00640 1 -- 172.21.15.39:0/2802173513 learned_addr learned my addr 172.21.15.39:0/2802173513 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d3400640 1 --2- 172.21.15.39:0/2802173513 >> v2:172.21.15.39:3300/0 conn(0x7f09d40c7f00 0x7f09d40c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d2000640 1 --2- 172.21.15.39:0/2802173513 >> v2:172.21.15.32:3300/0 conn(0x7f09d40d4a40 0x7f09d40d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d2a00640 1 -- 172.21.15.39:0/2802173513 >> v2:172.21.15.39:3300/0 conn(0x7f09d40c7f00 msgr2=0x7f09d40c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.406+0000 7f09d2a00640 1 --2- 172.21.15.39:0/2802173513 >> v2:172.21.15.39:3300/0 conn(0x7f09d40c7f00 0x7f09d40c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.407+0000 7f09d2a00640 1 -- 172.21.15.39:0/2802173513 >> v2:172.21.15.32:3300/0 conn(0x7f09d40d4a40 msgr2=0x7f09d40d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.407+0000 7f09d2a00640 1 --2- 172.21.15.39:0/2802173513 >> v2:172.21.15.32:3300/0 conn(0x7f09d40d4a40 0x7f09d40d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.407+0000 7f09d2a00640 1 -- 172.21.15.39:0/2802173513 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40d2810 con 0x7f09d4161840 2024-03-06T07:28:44.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.408+0000 7f09d2a00640 1 --2- 172.21.15.39:0/2802173513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 0x7f09d40d07f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009870 tx=0x7f09c402b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aad78a2e2e0637ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.408+0000 7f09d1600640 1 -- 172.21.15.39:0/2802173513 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c40317d0 con 0x7f09d4161840 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.408+0000 7f09d1600640 1 -- 172.21.15.39:0/2802173513 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c4037070 con 0x7f09d4161840 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.408+0000 7f09d1600640 1 -- 172.21.15.39:0/2802173513 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4031b90 con 0x7f09d4161840 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.408+0000 7f09d9600640 1 -- 172.21.15.39:0/2802173513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 msgr2=0x7f09d40d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.408+0000 7f09d9600640 1 --2- 172.21.15.39:0/2802173513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 0x7f09d40d07f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009870 tx=0x7f09c402b870 comp rx=0 tx=0).stop 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 -- 172.21.15.39:0/2802173513 shutdown_connections 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 --2- 172.21.15.39:0/2802173513 >> v2:172.21.15.39:3300/0 conn(0x7f09d40c7f00 0x7f09d40c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 --2- 172.21.15.39:0/2802173513 >> v2:172.21.15.32:3300/0 conn(0x7f09d40d4a40 0x7f09d40d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 --2- 172.21.15.39:0/2802173513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d4161840 0x7f09d40d07f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 -- 172.21.15.39:0/2802173513 >> 172.21.15.39:0/2802173513 conn(0x7f09d40d3eb0 msgr2=0x7f09d40d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 -- 172.21.15.39:0/2802173513 shutdown_connections 2024-03-06T07:28:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.409+0000 7f09d9600640 1 -- 172.21.15.39:0/2802173513 wait complete. 2024-03-06T07:28:44.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.410+0000 7f09d9600640 1 Processor -- start 2024-03-06T07:28:44.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.410+0000 7f09d9600640 1 -- start start 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.411+0000 7f09d9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09d40c7f00 0x7f09d40b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.411+0000 7f09d9600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d40d4a40 0x7f09d40cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.411+0000 7f09d9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 0x7f09d40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.411+0000 7f09d9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40d1750 con 0x7f09d40c7f00 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.411+0000 7f09d9600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40d2dd0 con 0x7f09d4161840 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.411+0000 7f09d9600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40c88d0 con 0x7f09d40d4a40 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 0x7f09d40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d40d4a40 0x7f09d40cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 0x7f09d40b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48614/0 (socket says 172.21.15.39:48614) 2024-03-06T07:28:44.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 -- 172.21.15.39:0/440295952 learned_addr learned my addr 172.21.15.39:0/440295952 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d2a00640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09d40c7f00 0x7f09d40b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 -- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d40d4a40 msgr2=0x7f09d40cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d40d4a40 0x7f09d40cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 -- 172.21.15.39:0/440295952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09d40c7f00 msgr2=0x7f09d40b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09d40c7f00 0x7f09d40b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 -- 172.21.15.39:0/440295952 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40cbf90 con 0x7f09d4161840 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.412+0000 7f09d3400640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 0x7f09d40b3f30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00efc0 tx=0x7f09cc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc018020 con 0x7f09d4161840 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc014480 con 0x7f09d4161840 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d40cbcb0 con 0x7f09d4161840 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc014870 con 0x7f09d4161840 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d40b3a10 con 0x7f09d4161840 2024-03-06T07:28:44.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f09cc005d10 con 0x7f09d4161840 2024-03-06T07:28:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.413+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f09cc013a40 con 0x7f09d4161840 2024-03-06T07:28:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.414+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d4068010 con 0x7f09d4161840 2024-03-06T07:28:44.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.418+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f09cc022030 con 0x7f09d4161840 2024-03-06T07:28:44.497 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:44.495+0000 7f8284649040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:28:44.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.498+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f10c34e2-acc4-460e-97f6-7da3d7ae631c", "id": 7} v 0) v1 -- 0x7f09d40b3be0 con 0x7f09d4161840 2024-03-06T07:28:44.504 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:44.502+0000 7f8645e73040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:28:44.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.507+0000 7f09caa00640 1 -- 172.21.15.39:0/440295952 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f10c34e2-acc4-460e-97f6-7da3d7ae631c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f09cc010040 con 0x7f09d4161840 2024-03-06T07:28:44.509 INFO:teuthology.orchestra.run.smithi039.stdout:7 2024-03-06T07:28:44.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 msgr2=0x7f09d40b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 0x7f09d40b3f30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00efc0 tx=0x7f09cc00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:44.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 shutdown_connections 2024-03-06T07:28:44.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d4161840 0x7f09d40b3f30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09d40d4a40 0x7f09d40cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 --2- 172.21.15.39:0/440295952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09d40c7f00 0x7f09d40b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.508+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 >> 172.21.15.39:0/440295952 conn(0x7f09d40d3eb0 msgr2=0x7f09d406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.509+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 shutdown_connections 2024-03-06T07:28:44.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-06T07:28:44.509+0000 7f09d9600640 1 -- 172.21.15.39:0/440295952 wait complete. 2024-03-06T07:28:44.527 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:28:44.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:28:44.534 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:28:44.535 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:28:44.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:28:44.541 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:28:44.542 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:28:44.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:28:44.547 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:28:44.547 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:28:44.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:28:44.552 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:28:44.553 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:28:44.553 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:28:44.558 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:28:44.558 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:28:44.559 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:28:44.564 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:28:44.564 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:28:44.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:28:44.565 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:44.564+0000 7f8645e73040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:28:44.569 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:28:44.569 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:28:44.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:28:44.575 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:28:44.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:44.750 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:44.748+0000 7f8645e73040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:28:44.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.790+0000 7f8613600640 1 Processor -- start 2024-03-06T07:28:44.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.790+0000 7f8613600640 1 -- start start 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8613600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f860c161670 0x7f860c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8613600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f860c06e240 0x7f860c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8613600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8613600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0730d0 con 0x7f860c06e240 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8613600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0ce9a0 con 0x7f860c161670 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8613600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0cec80 con 0x7f860c06cfe0 2024-03-06T07:28:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8603e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f860c06e240 0x7f860c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8603e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f860c06e240 0x7f860c0711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41898/0 (socket says 172.21.15.32:41898) 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8603e00640 1 -- 172.21.15.32:0/1998853431 learned_addr learned my addr 172.21.15.32:0/1998853431 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8611600640 1 --2- 172.21.15.32:0/1998853431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.791+0000 7f8610c00640 1 --2- 172.21.15.32:0/1998853431 >> v2:172.21.15.39:3300/0 conn(0x7f860c161670 0x7f860c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.792+0000 7f8603e00640 1 -- 172.21.15.32:0/1998853431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 msgr2=0x7f860c06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.792+0000 7f8603e00640 1 --2- 172.21.15.32:0/1998853431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c06d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.792+0000 7f8603e00640 1 -- 172.21.15.32:0/1998853431 >> v2:172.21.15.39:3300/0 conn(0x7f860c161670 msgr2=0x7f860c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.792+0000 7f8603e00640 1 --2- 172.21.15.32:0/1998853431 >> v2:172.21.15.39:3300/0 conn(0x7f860c161670 0x7f860c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.792+0000 7f8603e00640 1 -- 172.21.15.32:0/1998853431 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0cef60 con 0x7f860c06e240 2024-03-06T07:28:44.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.793+0000 7f8603e00640 1 --2- 172.21.15.32:0/1998853431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0711b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f860c073610 tx=0x7f85f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb96e8bc72db1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.793+0000 7f860be00640 1 -- 172.21.15.32:0/1998853431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4035020 con 0x7f860c06e240 2024-03-06T07:28:44.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.793+0000 7f860be00640 1 -- 172.21.15.32:0/1998853431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f4031870 con 0x7f860c06e240 2024-03-06T07:28:44.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.793+0000 7f860be00640 1 -- 172.21.15.32:0/1998853431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4031c40 con 0x7f860c06e240 2024-03-06T07:28:44.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 -- 172.21.15.32:0/1998853431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 msgr2=0x7f860c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 --2- 172.21.15.32:0/1998853431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0711b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f860c073610 tx=0x7f85f402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 -- 172.21.15.32:0/1998853431 shutdown_connections 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 --2- 172.21.15.32:0/1998853431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c06d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 --2- 172.21.15.32:0/1998853431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0711b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 --2- 172.21.15.32:0/1998853431 >> v2:172.21.15.39:3300/0 conn(0x7f860c161670 0x7f860c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 -- 172.21.15.32:0/1998853431 >> 172.21.15.32:0/1998853431 conn(0x7f860c06afb0 msgr2=0x7f860c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 -- 172.21.15.32:0/1998853431 shutdown_connections 2024-03-06T07:28:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.794+0000 7f8613600640 1 -- 172.21.15.32:0/1998853431 wait complete. 2024-03-06T07:28:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.796+0000 7f8613600640 1 Processor -- start 2024-03-06T07:28:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.796+0000 7f8613600640 1 -- start start 2024-03-06T07:28:44.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.796+0000 7f8613600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c14b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8613600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8613600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f860c161670 0x7f860c0cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8613600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0ce600 con 0x7f860c06e240 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8613600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0cf520 con 0x7f860c161670 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8613600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0ce780 con 0x7f860c06cfe0 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41912/0 (socket says 172.21.15.32:41912) 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 -- 172.21.15.32:0/2639477708 learned_addr learned my addr 172.21.15.32:0/2639477708 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8610c00640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c14b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8611600640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f860c161670 0x7f860c0cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 -- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 msgr2=0x7f860c14b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c14b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 -- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f860c161670 msgr2=0x7f860c0cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f860c161670 0x7f860c0cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.797+0000 7f8603e00640 1 -- 172.21.15.32:0/2639477708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c09b7f0 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8603e00640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0ca740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f85f402b950 tx=0x7f85f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4040070 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f4031e10 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4035040 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c09bad0 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c14bcf0 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f85f404b060 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.798+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f85f403ab90 con 0x7f860c06e240 2024-03-06T07:28:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.799+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c069510 con 0x7f860c06e240 2024-03-06T07:28:44.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.802+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f85f4063dd0 con 0x7f860c06e240 2024-03-06T07:28:44.869 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:44.867+0000 7f8645e73040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:28:44.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.868+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f860c04f3a0 con 0x7f860c06e240 2024-03-06T07:28:44.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.869+0000 7f8609600640 1 -- 172.21.15.32:0/2639477708 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f85f4052070 con 0x7f860c06e240 2024-03-06T07:28:44.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:28:44.871 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:28:44.501404+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 msgr2=0x7f860c0ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0ca740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f85f402b950 tx=0x7f85f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 shutdown_connections 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f860c161670 0x7f860c0cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c06e240 0x7f860c0ca740 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 --2- 172.21.15.32:0/2639477708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f860c06cfe0 0x7f860c14b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 >> 172.21.15.32:0/2639477708 conn(0x7f860c06afb0 msgr2=0x7f860c070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 shutdown_connections 2024-03-06T07:28:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:44.870+0000 7f8613600640 1 -- 172.21.15.32:0/2639477708 wait complete. 2024-03-06T07:28:44.888 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:28:44.888 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:28:45.056 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:45.054+0000 7f8645e73040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:28:45.108 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:45.106+0000 7f8645e73040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:28:45.242 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-03-06T07:28:45.240+0000 7ff5d7a10640 -1 Falling back to public interface 2024-03-06T07:28:45.246 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-03-06T07:28:45.244+0000 7f59e380f640 -1 Falling back to public interface 2024-03-06T07:28:45.259 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-06T07:28:45.257+0000 7ff890136640 -1 Falling back to public interface 2024-03-06T07:28:45.260 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-06T07:28:45.258+0000 7f2a6a668640 -1 Falling back to public interface 2024-03-06T07:28:45.272 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-06T07:28:45.270+0000 7f2baf9df640 -1 Falling back to public interface 2024-03-06T07:28:45.272 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-06T07:28:45.270+0000 7f073c302640 -1 Falling back to public interface 2024-03-06T07:28:45.274 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-03-06T07:28:45.272+0000 7f2fb9613640 -1 Falling back to public interface 2024-03-06T07:28:45.281 INFO:tasks.ceph.osd.7.smithi039.stderr:2024-03-06T07:28:45.280+0000 7ff0d16c8640 -1 Falling back to public interface 2024-03-06T07:28:45.297 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:45.295+0000 7f8645e73040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:28:45.627 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:45.625+0000 7f8284649040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:28:45.694 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:45.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.848+0000 7f87a3a00640 1 Processor -- start 2024-03-06T07:28:45.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.848+0000 7f87a3a00640 1 -- start start 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.849+0000 7f87a3a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c1626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.849+0000 7f87a3a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f879c0b2f40 0x7f879c0b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.849+0000 7f87a3a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879c162be0 con 0x7f879c0b1bd0 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.849+0000 7f87a3a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879c162210 con 0x7f879c0b2f40 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.849+0000 7f87a3a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879c0b39e0 con 0x7f879c1478f0 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f879c0b1bd0 0x7f879c0ad850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:39474/0 (socket says 172.21.15.32:39474) 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1a00640 1 -- 172.21.15.32:0/2065119684 learned_addr learned my addr 172.21.15.32:0/2065119684 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1000640 1 --2- 172.21.15.32:0/2065119684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c1626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f879be00640 1 --2- 172.21.15.32:0/2065119684 >> v2:172.21.15.32:3300/0 conn(0x7f879c0b2f40 0x7f879c0b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f879b400640 1 -- 172.21.15.32:0/2065119684 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266496228 0 0) 0x7f879c162be0 con 0x7f879c0b1bd0 2024-03-06T07:28:45.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f879b400640 1 -- 172.21.15.32:0/2065119684 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784003520 con 0x7f879c0b1bd0 2024-03-06T07:28:45.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1000640 1 -- 172.21.15.32:0/2065119684 >> v2:172.21.15.32:3300/0 conn(0x7f879c0b2f40 msgr2=0x7f879c0b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1000640 1 --2- 172.21.15.32:0/2065119684 >> v2:172.21.15.32:3300/0 conn(0x7f879c0b2f40 0x7f879c0b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1000640 1 -- 172.21.15.32:0/2065119684 >> v1:172.21.15.39:6789/0 conn(0x7f879c0b1bd0 legacy=0x7f879c0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.850+0000 7f87a1000640 1 -- 172.21.15.32:0/2065119684 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c0b3cc0 con 0x7f879c1478f0 2024-03-06T07:28:45.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.852+0000 7f87a1000640 1 --2- 172.21.15.32:0/2065119684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c1626a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f87900097a0 tx=0x7f879002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74f27fb9e576d9f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.852+0000 7f879b400640 1 -- 172.21.15.32:0/2065119684 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790035020 con 0x7f879c1478f0 2024-03-06T07:28:45.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.852+0000 7f879b400640 1 -- 172.21.15.32:0/2065119684 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8790031830 con 0x7f879c1478f0 2024-03-06T07:28:45.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.852+0000 7f879b400640 1 -- 172.21.15.32:0/2065119684 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790031c00 con 0x7f879c1478f0 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 -- 172.21.15.32:0/2065119684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 msgr2=0x7f879c1626a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 --2- 172.21.15.32:0/2065119684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c1626a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f87900097a0 tx=0x7f879002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 -- 172.21.15.32:0/2065119684 shutdown_connections 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 --2- 172.21.15.32:0/2065119684 >> v2:172.21.15.32:3300/0 conn(0x7f879c0b2f40 0x7f879c0b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 --2- 172.21.15.32:0/2065119684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c1626a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 -- 172.21.15.32:0/2065119684 >> 172.21.15.32:0/2065119684 conn(0x7f879c06af70 msgr2=0x7f879c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 -- 172.21.15.32:0/2065119684 shutdown_connections 2024-03-06T07:28:45.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.853+0000 7f87a3a00640 1 -- 172.21.15.32:0/2065119684 wait complete. 2024-03-06T07:28:45.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.854+0000 7f87a3a00640 1 Processor -- start 2024-03-06T07:28:45.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 -- start start 2024-03-06T07:28:45.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 0x7f879c0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f879c0b2f40 0x7f879c131540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c0b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c147cd0 con 0x7f879c0b1bd0 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c0aefc0 con 0x7f879c0b2f40 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.855+0000 7f87a3a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879c147e50 con 0x7f879c1478f0 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 0x7f879c0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 0x7f879c0cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41942/0 (socket says 172.21.15.32:41942) 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 -- 172.21.15.32:0/1334984712 learned_addr learned my addr 172.21.15.32:0/1334984712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f879be00640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f879c0b2f40 0x7f879c131540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1a00640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c0b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 -- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 msgr2=0x7f879c0b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c0b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 -- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f879c0b2f40 msgr2=0x7f879c131540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f879c0b2f40 0x7f879c131540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.856+0000 7f87a1000640 1 -- 172.21.15.32:0/1334984712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c134510 con 0x7f879c0b1bd0 2024-03-06T07:28:45.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f87a1000640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 0x7f879c0cee80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f87900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790040070 con 0x7f879c0b1bd0 2024-03-06T07:28:45.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879003a850 con 0x7f879c0b1bd0 2024-03-06T07:28:45.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879c133990 con 0x7f879c0b1bd0 2024-03-06T07:28:45.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790035040 con 0x7f879c0b1bd0 2024-03-06T07:28:45.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879c133f50 con 0x7f879c0b1bd0 2024-03-06T07:28:45.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8790038020 con 0x7f879c0b1bd0 2024-03-06T07:28:45.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.857+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f87900317b0 con 0x7f879c0b1bd0 2024-03-06T07:28:45.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.858+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879c04f330 con 0x7f879c0b1bd0 2024-03-06T07:28:45.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.863+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f879004dd40 con 0x7f879c0b1bd0 2024-03-06T07:28:45.930 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.928+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f879c1f71d0 con 0x7f879c0b1bd0 2024-03-06T07:28:45.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.929+0000 7f8798c00640 1 -- 172.21.15.32:0/1334984712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f879004d430 con 0x7f879c0b1bd0 2024-03-06T07:28:45.932 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-06T07:28:45.932 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:28:44.501404+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:45.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.930+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 msgr2=0x7f879c0cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.930+0000 7f87a3a00640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 0x7f879c0cee80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f87900046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 shutdown_connections 2024-03-06T07:28:45.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f879c1478f0 0x7f879c0b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f879c0b2f40 0x7f879c131540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 --2- 172.21.15.32:0/1334984712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879c0b1bd0 0x7f879c0cee80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.935 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 >> 172.21.15.32:0/1334984712 conn(0x7f879c06af70 msgr2=0x7f879c1607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.935 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 shutdown_connections 2024-03-06T07:28:45.935 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:45.931+0000 7f87a3a00640 1 -- 172.21.15.32:0/1334984712 wait complete. 2024-03-06T07:28:45.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:28:45.962 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:45.960+0000 7f8284649040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:28:46.025 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:46.023+0000 7f8284649040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:28:46.163 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:46.160+0000 7f8284649040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:28:46.413 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:46.411+0000 7f8645e73040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:28:46.763 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:46.762+0000 7f8645e73040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:28:46.826 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:46.824+0000 7f8645e73040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:28:46.964 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-03-06T07:28:46.963+0000 7f8645e73040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:28:47.194 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:47.192+0000 7f8224e00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:28:47.199 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-06T07:28:47.197+0000 7f8224e00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:28:48.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:48.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:48.806 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-03-06T07:28:48.805+0000 7ff5d7a10640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:28:48.807 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-06T07:28:48.805+0000 7f2a6a668640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:28:48.820 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-06T07:28:48.818+0000 7ff890136640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:28:48.821 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-03-06T07:28:48.819+0000 7f2fb9613640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:28:48.842 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-06T07:28:48.841+0000 7f073c302640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:28:48.848 INFO:tasks.ceph.osd.7.smithi039.stderr:2024-03-06T07:28:48.847+0000 7ff0d16c8640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:28:48.849 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-06T07:28:48.847+0000 7f2baf9df640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:28:49.053 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-03-06T07:28:49.052+0000 7f59e380f640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:28:51.987 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-06T07:28:51.986+0000 7f2babc00640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:28:51.995 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-06T07:28:51.994+0000 7f2ba0800640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:51.999 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-06T07:28:51.998+0000 7ff88c400640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:28:52.003 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-06T07:28:52.002+0000 7f2a64c00640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:28:52.004 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-06T07:28:52.003+0000 7f0736800640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:28:52.008 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-06T07:28:52.007+0000 7ff881000640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.010 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-06T07:28:52.009+0000 7f072d200640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.021 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-06T07:28:52.020+0000 7f2a5b600640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.752 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:52.829 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-03-06T07:28:52.828+0000 7f59dd600640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:28:52.836 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-03-06T07:28:52.835+0000 7f59d4000640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.860 INFO:tasks.ceph.osd.7.smithi039.stderr:2024-03-06T07:28:52.858+0000 7ff0cbc00640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:28:52.867 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-03-06T07:28:52.865+0000 7f2fb3400640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:28:52.867 INFO:tasks.ceph.osd.7.smithi039.stderr:2024-03-06T07:28:52.866+0000 7ff0c2600640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.874 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-03-06T07:28:52.873+0000 7ff5d1a00640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:28:52.877 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-03-06T07:28:52.875+0000 7f2fa9e00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.886 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-03-06T07:28:52.885+0000 7ff5c8400640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:28:52.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.949+0000 7fe27c000640 1 Processor -- start 2024-03-06T07:28:52.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.950+0000 7fe27c000640 1 -- start start 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe27c000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2741457d0 0x7fe2740cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe27c000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe2740b22e0 0x7fe2740b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe27c000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe274145bb0 con 0x7fe27415c380 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe27c000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2740b33b0 con 0x7fe2741457d0 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe27c000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2740b3690 con 0x7fe2740b22e0 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe279600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2741457d0 0x7fe2740cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe279600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2741457d0 0x7fe2740cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45410/0 (socket says 172.21.15.32:45410) 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe279600640 1 -- 172.21.15.32:0/1574533499 learned_addr learned my addr 172.21.15.32:0/1574533499 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.951+0000 7fe27a000640 1 --2- 172.21.15.32:0/1574533499 >> v2:172.21.15.39:3300/0 conn(0x7fe2740b22e0 0x7fe2740b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.952+0000 7fe279600640 1 -- 172.21.15.32:0/1574533499 >> v2:172.21.15.39:3300/0 conn(0x7fe2740b22e0 msgr2=0x7fe2740b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.952+0000 7fe279600640 1 --2- 172.21.15.32:0/1574533499 >> v2:172.21.15.39:3300/0 conn(0x7fe2740b22e0 0x7fe2740b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.952+0000 7fe279600640 1 -- 172.21.15.32:0/1574533499 >> v1:172.21.15.39:6789/0 conn(0x7fe27415c380 legacy=0x7fe2740b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.952+0000 7fe279600640 1 -- 172.21.15.32:0/1574533499 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2740b3970 con 0x7fe2741457d0 2024-03-06T07:28:52.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.952+0000 7fe27a000640 1 --2- 172.21.15.32:0/1574533499 >> v2:172.21.15.39:3300/0 conn(0x7fe2740b22e0 0x7fe2740b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.953+0000 7fe279600640 1 --2- 172.21.15.32:0/1574533499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2741457d0 0x7fe2740cf620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe2740b27b0 tx=0x7fe26802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7a95e5aa2477d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.953+0000 7fe273e00640 1 -- 172.21.15.32:0/1574533499 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268035020 con 0x7fe2741457d0 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.953+0000 7fe273e00640 1 -- 172.21.15.32:0/1574533499 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268031870 con 0x7fe2741457d0 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.953+0000 7fe273e00640 1 -- 172.21.15.32:0/1574533499 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268031c40 con 0x7fe2741457d0 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 -- 172.21.15.32:0/1574533499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2741457d0 msgr2=0x7fe2740cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 --2- 172.21.15.32:0/1574533499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2741457d0 0x7fe2740cf620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe2740b27b0 tx=0x7fe26802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 -- 172.21.15.32:0/1574533499 shutdown_connections 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 --2- 172.21.15.32:0/1574533499 >> v2:172.21.15.39:3300/0 conn(0x7fe2740b22e0 0x7fe2740b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 --2- 172.21.15.32:0/1574533499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2741457d0 0x7fe2740cf620 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 -- 172.21.15.32:0/1574533499 >> 172.21.15.32:0/1574533499 conn(0x7fe27406af70 msgr2=0x7fe27406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 -- 172.21.15.32:0/1574533499 shutdown_connections 2024-03-06T07:28:52.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.954+0000 7fe27c000640 1 -- 172.21.15.32:0/1574533499 wait complete. 2024-03-06T07:28:52.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.955+0000 7fe27c000640 1 Processor -- start 2024-03-06T07:28:52.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 -- start start 2024-03-06T07:28:52.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 0x7fe274148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2741457d0 0x7fe274149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe27415c380 0x7fe274145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740731f0 con 0x7fe2740b22e0 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740b3f30 con 0x7fe27415c380 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.956+0000 7fe27c000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe274073370 con 0x7fe2741457d0 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 0x7fe274148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 0x7fe274148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45418/0 (socket says 172.21.15.32:45418) 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 -- 172.21.15.32:0/1137158675 learned_addr learned my addr 172.21.15.32:0/1137158675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe27a000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe27415c380 0x7fe274145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe278c00640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2741457d0 0x7fe274149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 -- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2741457d0 msgr2=0x7fe274149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2741457d0 0x7fe274149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 -- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe27415c380 msgr2=0x7fe274145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe27415c380 0x7fe274145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe279600640 1 -- 172.21.15.32:0/1137158675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274138250 con 0x7fe2740b22e0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.957+0000 7fe278c00640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2741457d0 0x7fe274149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.958+0000 7fe279600640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 0x7fe274148ef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe268000c00 tx=0x7fe2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.958+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26803ad60 con 0x7fe2740b22e0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.958+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2680443f0 con 0x7fe2740b22e0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.958+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe274157830 con 0x7fe2740b22e0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.958+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2680447e0 con 0x7fe2740b22e0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.958+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe274135e40 con 0x7fe2740b22e0 2024-03-06T07:28:52.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.959+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe274138520 con 0x7fe2740b22e0 2024-03-06T07:28:52.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.960+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fe26804b060 con 0x7fe2740b22e0 2024-03-06T07:28:52.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.961+0000 7fe271600640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe25406b470 0x7fe25406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.961+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe2680aba20 con 0x7fe2740b22e0 2024-03-06T07:28:52.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.961+0000 7fe278c00640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe25406b470 0x7fe25406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.962+0000 7fe278c00640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe25406b470 0x7fe25406d930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2741f9750 tx=0x7fe260008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:52.966+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe26803f070 con 0x7fe2740b22e0 2024-03-06T07:28:53.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.169+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe27406efb0 con 0x7fe2740b22e0 2024-03-06T07:28:53.172 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.171+0000 7fe271600640 1 -- 172.21.15.32:0/1137158675 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10801 (secure 0 0 0) 0x7fe26802f320 con 0x7fe2740b22e0 2024-03-06T07:28:53.172 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-06T07:28:53.173 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":13,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:28:52.988900+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:53.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe25406b470 msgr2=0x7fe25406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe25406b470 0x7fe25406d930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2741f9750 tx=0x7fe260008040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 msgr2=0x7fe274148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 0x7fe274148ef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe268000c00 tx=0x7fe2680046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 shutdown_connections 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe25406b470 0x7fe25406d930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe27415c380 0x7fe274145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2741457d0 0x7fe274149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 --2- 172.21.15.32:0/1137158675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2740b22e0 0x7fe274148ef0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.173+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 >> 172.21.15.32:0/1137158675 conn(0x7fe27406af70 msgr2=0x7fe2740729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.174+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 shutdown_connections 2024-03-06T07:28:53.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-06T07:28:53.174+0000 7fe27c000640 1 -- 172.21.15.32:0/1137158675 wait complete. 2024-03-06T07:28:53.188 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:28:53.188 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:28:53.188 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:53.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:53.193 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:53.193 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:53.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:53.197 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:53.197 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:53.198 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:53.205 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:53.205 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:53.206 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:53.213 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:53.213 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:28:53.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:53.345 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:28:53.346 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:28:53.368 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:28:53.375 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:28:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.395+0000 7f9b60000640 1 Processor -- start 2024-03-06T07:28:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.395+0000 7f9b60000640 1 -- start start 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.396+0000 7f9b60000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b5815e000 0x7f9b5806cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.396+0000 7f9b60000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58162810 0x7f9b580af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.396+0000 7f9b60000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5806d060 con 0x7f9b58161c70 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.396+0000 7f9b60000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5806d8a0 con 0x7f9b5815e000 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.396+0000 7f9b60000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5806d2e0 con 0x7f9b58162810 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5d600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b5815e000 0x7f9b5806cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5cc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9b58161c70 0x7f9b58162100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60126/0 (socket says 172.21.15.32:60126) 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5cc00640 1 -- 172.21.15.32:0/3012383803 learned_addr learned my addr 172.21.15.32:0/3012383803 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5e000640 1 --2- 172.21.15.32:0/3012383803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58162810 0x7f9b580af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5d600640 1 -- 172.21.15.32:0/3012383803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58162810 msgr2=0x7f9b580af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5d600640 1 --2- 172.21.15.32:0/3012383803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58162810 0x7f9b580af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5d600640 1 -- 172.21.15.32:0/3012383803 >> v1:172.21.15.32:6789/0 conn(0x7f9b58161c70 legacy=0x7f9b58162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b5d600640 1 -- 172.21.15.32:0/3012383803 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5806d5c0 con 0x7f9b5815e000 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.397+0000 7f9b57e00640 1 -- 172.21.15.32:0/3012383803 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554017677 0 0) 0x7f9b44001110 con 0x7f9b58161c70 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.398+0000 7f9b5d600640 1 --2- 172.21.15.32:0/3012383803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b5815e000 0x7f9b5806cb20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0097a0 tx=0x7f9b4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e189057c4f8288e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.398+0000 7f9b57e00640 1 -- 172.21.15.32:0/3012383803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c0315a0 con 0x7f9b5815e000 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.398+0000 7f9b57e00640 1 -- 172.21.15.32:0/3012383803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4c037070 con 0x7f9b5815e000 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.398+0000 7f9b57e00640 1 -- 172.21.15.32:0/3012383803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c031960 con 0x7f9b5815e000 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 -- 172.21.15.32:0/3012383803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b5815e000 msgr2=0x7f9b5806cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 --2- 172.21.15.32:0/3012383803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b5815e000 0x7f9b5806cb20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0097a0 tx=0x7f9b4c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 -- 172.21.15.32:0/3012383803 shutdown_connections 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 --2- 172.21.15.32:0/3012383803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58162810 0x7f9b580af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 --2- 172.21.15.32:0/3012383803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b5815e000 0x7f9b5806cb20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 -- 172.21.15.32:0/3012383803 >> 172.21.15.32:0/3012383803 conn(0x7f9b5809aa10 msgr2=0x7f9b5809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 -- 172.21.15.32:0/3012383803 shutdown_connections 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.399+0000 7f9b60000640 1 -- 172.21.15.32:0/3012383803 wait complete. 2024-03-06T07:28:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.400+0000 7f9b60000640 1 Processor -- start 2024-03-06T07:28:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 -- start start 2024-03-06T07:28:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b5815e000 0x7f9b58160e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58161c70 0x7f9b58148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 0x7f9b58148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5815e3e0 con 0x7f9b58162810 2024-03-06T07:28:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b580afa30 con 0x7f9b5815e000 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.401+0000 7f9b60000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b581483a0 con 0x7f9b58161c70 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 0x7f9b58148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 0x7f9b58148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45476/0 (socket says 172.21.15.32:45476) 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 -- 172.21.15.32:0/3665399548 learned_addr learned my addr 172.21.15.32:0/3665399548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5d600640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b5815e000 0x7f9b58160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5cc00640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58161c70 0x7f9b58148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 -- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58161c70 msgr2=0x7f9b58148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58161c70 0x7f9b58148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 -- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b5815e000 msgr2=0x7f9b58160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b5815e000 0x7f9b58160e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 -- 172.21.15.32:0/3665399548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b58147300 con 0x7f9b58162810 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5d600640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b5815e000 0x7f9b58160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.402+0000 7f9b5e000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 0x7f9b58148ba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b50007e60 tx=0x7f9b5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.403+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50014420 con 0x7f9b58162810 2024-03-06T07:28:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.403+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b50019070 con 0x7f9b58162810 2024-03-06T07:28:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.403+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50014820 con 0x7f9b58162810 2024-03-06T07:28:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.403+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b581475e0 con 0x7f9b58162810 2024-03-06T07:28:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.403+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b58147ba0 con 0x7f9b58162810 2024-03-06T07:28:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.404+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5804f3a0 con 0x7f9b58162810 2024-03-06T07:28:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.405+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f9b500149c0 con 0x7f9b58162810 2024-03-06T07:28:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.405+0000 7f9b55600640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b3806b440 0x7f9b3806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.405+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f9b5008a440 con 0x7f9b58162810 2024-03-06T07:28:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.406+0000 7f9b5d600640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b3806b440 0x7f9b3806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.406+0000 7f9b5d600640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b3806b440 0x7f9b3806d900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c036000 tx=0x7f9b4c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.410+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b500597d0 con 0x7f9b58162810 2024-03-06T07:28:53.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.600+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b5809b750 con 0x7f9b58162810 2024-03-06T07:28:53.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.602+0000 7f9b55600640 1 -- 172.21.15.32:0/3665399548 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10801 (secure 0 0 0) 0x7f9b500590d0 con 0x7f9b58162810 2024-03-06T07:28:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:28:53.603 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:28:52.988900+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b3806b440 msgr2=0x7f9b3806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b3806b440 0x7f9b3806d900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c036000 tx=0x7f9b4c03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 msgr2=0x7f9b58148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 0x7f9b58148ba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b50007e60 tx=0x7f9b5000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 shutdown_connections 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b3806b440 0x7f9b3806d900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b58162810 0x7f9b58148ba0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b58161c70 0x7f9b58148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 --2- 172.21.15.32:0/3665399548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b5815e000 0x7f9b58160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 >> 172.21.15.32:0/3665399548 conn(0x7f9b5809aa10 msgr2=0x7f9b580afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 shutdown_connections 2024-03-06T07:28:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.604+0000 7f9b60000640 1 -- 172.21.15.32:0/3665399548 wait complete. 2024-03-06T07:28:53.623 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-06T07:28:53.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:53.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.824+0000 7fcfcc800640 1 Processor -- start 2024-03-06T07:28:53.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.825+0000 7fcfcc800640 1 -- start start 2024-03-06T07:28:53.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfcc800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcfc40af1a0 0x7fcfc415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfcc800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcfc40cdc20 0x7fcfc40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfcc800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc415cc40 con 0x7fcfc40af580 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfcc800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc415cf20 con 0x7fcfc40af1a0 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfcc800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc415d200 con 0x7fcfc40cdc20 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfc9400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcfc40af580 0x7fcfc40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60130/0 (socket says 172.21.15.32:60130) 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfc9400640 1 -- 172.21.15.32:0/206267037 learned_addr learned my addr 172.21.15.32:0/206267037 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:28:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.826+0000 7fcfc9e00640 1 --2- 172.21.15.32:0/206267037 >> v2:172.21.15.32:3300/0 conn(0x7fcfc40af1a0 0x7fcfc415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfca800640 1 --2- 172.21.15.32:0/206267037 >> v2:172.21.15.39:3300/0 conn(0x7fcfc40cdc20 0x7fcfc40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfc8a00640 1 -- 172.21.15.32:0/206267037 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230663851 0 0) 0x7fcfc415cc40 con 0x7fcfc40af580 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfc8a00640 1 -- 172.21.15.32:0/206267037 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb0003520 con 0x7fcfc40af580 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfc9e00640 1 -- 172.21.15.32:0/206267037 >> v2:172.21.15.39:3300/0 conn(0x7fcfc40cdc20 msgr2=0x7fcfc40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfc9e00640 1 --2- 172.21.15.32:0/206267037 >> v2:172.21.15.39:3300/0 conn(0x7fcfc40cdc20 0x7fcfc40b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfc9e00640 1 -- 172.21.15.32:0/206267037 >> v1:172.21.15.32:6789/0 conn(0x7fcfc40af580 legacy=0x7fcfc40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.827+0000 7fcfc9e00640 1 -- 172.21.15.32:0/206267037 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc415d4e0 con 0x7fcfc40af1a0 2024-03-06T07:28:53.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.828+0000 7fcfc9e00640 1 --2- 172.21.15.32:0/206267037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc415c600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcfc406c0c0 tx=0x7fcfbc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d73bfc35e983187 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.830+0000 7fcfc8a00640 1 -- 172.21.15.32:0/206267037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc039070 con 0x7fcfc40af1a0 2024-03-06T07:28:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.830+0000 7fcfc8a00640 1 -- 172.21.15.32:0/206267037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc02cbe0 con 0x7fcfc40af1a0 2024-03-06T07:28:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.830+0000 7fcfc8a00640 1 -- 172.21.15.32:0/206267037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc034440 con 0x7fcfc40af1a0 2024-03-06T07:28:53.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 -- 172.21.15.32:0/206267037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 msgr2=0x7fcfc415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 --2- 172.21.15.32:0/206267037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc415c600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcfc406c0c0 tx=0x7fcfbc02be30 comp rx=0 tx=0).stop 2024-03-06T07:28:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 -- 172.21.15.32:0/206267037 shutdown_connections 2024-03-06T07:28:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 --2- 172.21.15.32:0/206267037 >> v2:172.21.15.39:3300/0 conn(0x7fcfc40cdc20 0x7fcfc40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 --2- 172.21.15.32:0/206267037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc415c600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 -- 172.21.15.32:0/206267037 >> 172.21.15.32:0/206267037 conn(0x7fcfc406afb0 msgr2=0x7fcfc406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 -- 172.21.15.32:0/206267037 shutdown_connections 2024-03-06T07:28:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.831+0000 7fcfcc800640 1 -- 172.21.15.32:0/206267037 wait complete. 2024-03-06T07:28:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.833+0000 7fcfcc800640 1 Processor -- start 2024-03-06T07:28:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.833+0000 7fcfcc800640 1 -- start start 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfcc800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc41f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfcc800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcfc40af580 0x7fcfc409da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfcc800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfc40cdc20 0x7fcfc40ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfcc800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40b36a0 con 0x7fcfc40af1a0 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfcc800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40b3520 con 0x7fcfc40cdc20 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfcc800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40b3820 con 0x7fcfc40af580 2024-03-06T07:28:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfc9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc41f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfc9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc41f7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45488/0 (socket says 172.21.15.32:45488) 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.834+0000 7fcfc9e00640 1 -- 172.21.15.32:0/3882322958 learned_addr learned my addr 172.21.15.32:0/3882322958 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfc9e00640 1 -- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcfc40af580 msgr2=0x7fcfc409da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfc9e00640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcfc40af580 0x7fcfc409da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfc9e00640 1 -- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfc40cdc20 msgr2=0x7fcfc40ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfca800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfc40cdc20 0x7fcfc40ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfc9e00640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfc40cdc20 0x7fcfc40ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfc9e00640 1 -- 172.21.15.32:0/3882322958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc409b090 con 0x7fcfc40af1a0 2024-03-06T07:28:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfc9e00640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc41f7cf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009e80 tx=0x7fcfbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc039050 con 0x7fcfc40af1a0 2024-03-06T07:28:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc045070 con 0x7fcfc40af1a0 2024-03-06T07:28:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc03e850 con 0x7fcfc40af1a0 2024-03-06T07:28:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.835+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc409adb0 con 0x7fcfc40af1a0 2024-03-06T07:28:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.836+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc41f74f0 con 0x7fcfc40af1a0 2024-03-06T07:28:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.837+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc4069510 con 0x7fcfc40af1a0 2024-03-06T07:28:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.837+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fcfbc03e9f0 con 0x7fcfc40af1a0 2024-03-06T07:28:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.838+0000 7fcfba000640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcfa806b440 0x7fcfa806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.838+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fcfbc0acb80 con 0x7fcfc40af1a0 2024-03-06T07:28:53.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.839+0000 7fcfc9400640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcfa806b440 0x7fcfa806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.839+0000 7fcfc9400640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcfa806b440 0x7fcfa806d900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4000f60 tx=0x7fcfb40078f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:53.846+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfbc07c0c0 con 0x7fcfc40af1a0 2024-03-06T07:28:54.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.090+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfc404f3a0 con 0x7fcfc40af1a0 2024-03-06T07:28:54.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.091+0000 7fcfba000640 1 -- 172.21.15.32:0/3882322958 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fcfbc07b9c0 con 0x7fcfc40af1a0 2024-03-06T07:28:54.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:28:54.092 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:28:54.093 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.093+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcfa806b440 msgr2=0x7fcfa806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.093+0000 7fcfcc800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcfa806b440 0x7fcfa806d900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4000f60 tx=0x7fcfb40078f0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.093+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 msgr2=0x7fcfc41f7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.093+0000 7fcfcc800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc41f7cf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009e80 tx=0x7fcfbc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 shutdown_connections 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcfa806b440 0x7fcfa806d900 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcfc40cdc20 0x7fcfc40ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcfc40af580 0x7fcfc409da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 --2- 172.21.15.32:0/3882322958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfc40af1a0 0x7fcfc41f7cf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 >> 172.21.15.32:0/3882322958 conn(0x7fcfc406afb0 msgr2=0x7fcfc40cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 shutdown_connections 2024-03-06T07:28:54.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:54.094+0000 7fcfcc800640 1 -- 172.21.15.32:0/3882322958 wait complete. 2024-03-06T07:28:56.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.350+0000 7f3713400640 1 Processor -- start 2024-03-06T07:28:56.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.350+0000 7f3713400640 1 -- start start 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3713400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0af580 0x7f370c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3713400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f370c0cdc20 0x7f370c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3713400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370c15cc40 con 0x7f370c0af1a0 2024-03-06T07:28:56.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3713400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c15cf20 con 0x7f370c0cdc20 2024-03-06T07:28:56.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3713400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c15d200 con 0x7f370c0af580 2024-03-06T07:28:56.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3710a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f370c0af1a0 0x7f370c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60140/0 (socket says 172.21.15.32:60140) 2024-03-06T07:28:56.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.352+0000 7f3710a00640 1 -- 172.21.15.32:0/2500336148 learned_addr learned my addr 172.21.15.32:0/2500336148 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f3711400640 1 --2- 172.21.15.32:0/2500336148 >> v2:172.21.15.39:3300/0 conn(0x7f370c0cdc20 0x7f370c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f370be00640 1 --2- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0af580 0x7f370c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f370b400640 1 -- 172.21.15.32:0/2500336148 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59607743 0 0) 0x7f370c15cc40 con 0x7f370c0af1a0 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f370b400640 1 -- 172.21.15.32:0/2500336148 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f4003520 con 0x7f370c0af1a0 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f3711400640 1 -- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0af580 msgr2=0x7f370c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f3711400640 1 --2- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0af580 0x7f370c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f3711400640 1 -- 172.21.15.32:0/2500336148 >> v1:172.21.15.32:6789/0 conn(0x7f370c0af1a0 legacy=0x7f370c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.353+0000 7f3711400640 1 -- 172.21.15.32:0/2500336148 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c15d4e0 con 0x7f370c0cdc20 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.355+0000 7f3711400640 1 --2- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0cdc20 0x7f370c0b2dd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3704009d90 tx=0x7f370402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82bea7db5b03e7f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f370b400640 1 -- 172.21.15.32:0/2500336148 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704039070 con 0x7f370c0cdc20 2024-03-06T07:28:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f370b400640 1 -- 172.21.15.32:0/2500336148 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370402cbd0 con 0x7f370c0cdc20 2024-03-06T07:28:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f370b400640 1 -- 172.21.15.32:0/2500336148 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704034440 con 0x7f370c0cdc20 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 -- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0cdc20 msgr2=0x7f370c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 --2- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0cdc20 0x7f370c0b2dd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3704009d90 tx=0x7f370402be20 comp rx=0 tx=0).stop 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 -- 172.21.15.32:0/2500336148 shutdown_connections 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 --2- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0cdc20 0x7f370c0b2dd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 --2- 172.21.15.32:0/2500336148 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0af580 0x7f370c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 -- 172.21.15.32:0/2500336148 >> 172.21.15.32:0/2500336148 conn(0x7f370c06afb0 msgr2=0x7f370c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 -- 172.21.15.32:0/2500336148 shutdown_connections 2024-03-06T07:28:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.356+0000 7f3713400640 1 -- 172.21.15.32:0/2500336148 wait complete. 2024-03-06T07:28:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.358+0000 7f3713400640 1 Processor -- start 2024-03-06T07:28:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.358+0000 7f3713400640 1 -- start start 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f3713400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0af1a0 0x7f370c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f3713400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 0x7f370c14f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f3713400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0cdc20 0x7f370c158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f3713400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0b36a0 con 0x7f370c0af580 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f3713400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0b3520 con 0x7f370c0af1a0 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f3713400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0b3820 con 0x7f370c0cdc20 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f370be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 0x7f370c14f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f370be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 0x7f370c14f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45492/0 (socket says 172.21.15.32:45492) 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.359+0000 7f370be00640 1 -- 172.21.15.32:0/272631157 learned_addr learned my addr 172.21.15.32:0/272631157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3710a00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0af1a0 0x7f370c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f370be00640 1 -- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0cdc20 msgr2=0x7f370c158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3711400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0cdc20 0x7f370c158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f370be00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0cdc20 0x7f370c158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f370be00640 1 -- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0af1a0 msgr2=0x7f370c154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f370be00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0af1a0 0x7f370c154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f370be00640 1 -- 172.21.15.32:0/272631157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c1593c0 con 0x7f370c0af580 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3710a00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0af1a0 0x7f370c154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f370be00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 0x7f370c14f050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36f800e9e0 tx=0x7f36f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8017070 con 0x7f370c0af580 2024-03-06T07:28:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f800c970 con 0x7f370c0af580 2024-03-06T07:28:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f800cd60 con 0x7f370c0af580 2024-03-06T07:28:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.360+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c09abb0 con 0x7f370c0af580 2024-03-06T07:28:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.361+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c09b450 con 0x7f370c0af580 2024-03-06T07:28:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.362+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c04f3a0 con 0x7f370c0af580 2024-03-06T07:28:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.363+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f36f801a070 con 0x7f370c0af580 2024-03-06T07:28:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.363+0000 7f3708c00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f36ec06b440 0x7f36ec06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.364+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f36f8089e20 con 0x7f370c0af580 2024-03-06T07:28:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.364+0000 7f3710a00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f36ec06b440 0x7f36ec06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.365+0000 7f3710a00640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f36ec06b440 0x7f36ec06d900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3700005880 tx=0x7f3700009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.369+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36f8059030 con 0x7f370c0af580 2024-03-06T07:28:56.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.631+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f370c15c090 con 0x7f370c0af580 2024-03-06T07:28:56.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.632+0000 7f3708c00640 1 -- 172.21.15.32:0/272631157 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f36f8058930 con 0x7f370c0af580 2024-03-06T07:28:56.634 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:28:56.634 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:28:56.634 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-03-06T07:28:56.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f36ec06b440 msgr2=0x7f36ec06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f36ec06b440 0x7f36ec06d900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3700005880 tx=0x7f3700009040 comp rx=0 tx=0).stop 2024-03-06T07:28:56.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 msgr2=0x7f370c14f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 0x7f370c14f050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36f800e9e0 tx=0x7f36f800eea0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 shutdown_connections 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f36ec06b440 0x7f36ec06d900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f370c0cdc20 0x7f370c158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f370c0af580 0x7f370c14f050 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 --2- 172.21.15.32:0/272631157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f370c0af1a0 0x7f370c154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 >> 172.21.15.32:0/272631157 conn(0x7f370c06afb0 msgr2=0x7f370c071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 shutdown_connections 2024-03-06T07:28:56.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.635+0000 7f3713400640 1 -- 172.21.15.32:0/272631157 wait complete. 2024-03-06T07:28:56.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:56.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.842+0000 7f7246200640 1 Processor -- start 2024-03-06T07:28:56.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.842+0000 7f7246200640 1 -- start start 2024-03-06T07:28:56.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.843+0000 7f7246200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.843+0000 7f7246200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f72400cdc20 0x7f72400b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.843+0000 7f7246200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724015cc40 con 0x7f72400af1a0 2024-03-06T07:28:56.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.843+0000 7f7246200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f724015cf20 con 0x7f72400cdc20 2024-03-06T07:28:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.843+0000 7f7246200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724015d200 con 0x7f72400af580 2024-03-06T07:28:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.844+0000 7f723fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f72400cdc20 0x7f72400b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.844+0000 7f723fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f72400cdc20 0x7f72400b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:46196/0 (socket says 172.21.15.32:46196) 2024-03-06T07:28:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.844+0000 7f723fe00640 1 -- 172.21.15.32:0/3173072851 learned_addr learned my addr 172.21.15.32:0/3173072851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:56.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.844+0000 7f7237e00640 1 --2- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.844+0000 7f723fe00640 1 -- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 msgr2=0x7f724015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.845+0000 7f723fe00640 1 --2- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.845+0000 7f723fe00640 1 -- 172.21.15.32:0/3173072851 >> v1:172.21.15.39:6789/0 conn(0x7f72400af1a0 legacy=0x7f72400b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.845+0000 7f723fe00640 1 -- 172.21.15.32:0/3173072851 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724015d4e0 con 0x7f72400cdc20 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.846+0000 7f723fe00640 1 --2- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400cdc20 0x7f72400b2dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7238009e50 tx=0x7f723802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e248846884bb79f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.846+0000 7f723ea00640 1 -- 172.21.15.32:0/3173072851 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723802ca70 con 0x7f72400cdc20 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.846+0000 7f723ea00640 1 -- 172.21.15.32:0/3173072851 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723802cc10 con 0x7f72400cdc20 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.846+0000 7f723ea00640 1 -- 172.21.15.32:0/3173072851 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72380343f0 con 0x7f72400cdc20 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 -- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400cdc20 msgr2=0x7f72400b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 --2- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400cdc20 0x7f72400b2dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7238009e50 tx=0x7f723802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 -- 172.21.15.32:0/3173072851 shutdown_connections 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 --2- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400cdc20 0x7f72400b2dd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 --2- 172.21.15.32:0/3173072851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 -- 172.21.15.32:0/3173072851 >> 172.21.15.32:0/3173072851 conn(0x7f724006afb0 msgr2=0x7f724006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 -- 172.21.15.32:0/3173072851 shutdown_connections 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.847+0000 7f7246200640 1 -- 172.21.15.32:0/3173072851 wait complete. 2024-03-06T07:28:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.849+0000 7f7246200640 1 Processor -- start 2024-03-06T07:28:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.849+0000 7f7246200640 1 -- start start 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f7246200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400af1a0 0x7f724014c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f7246200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724014f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f7246200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 0x7f724006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f7246200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400b36a0 con 0x7f72400cdc20 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f7246200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400b3520 con 0x7f72400af1a0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f7246200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72400b3820 con 0x7f72400af580 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f723fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 0x7f724006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f723fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 0x7f724006db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45522/0 (socket says 172.21.15.32:45522) 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.850+0000 7f723fe00640 1 -- 172.21.15.32:0/722679660 learned_addr learned my addr 172.21.15.32:0/722679660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723f400640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400af1a0 0x7f724014c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f7237e00640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724014f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723fe00640 1 -- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 msgr2=0x7f724014f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723fe00640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724014f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723fe00640 1 -- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400af1a0 msgr2=0x7f724014c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723fe00640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400af1a0 0x7f724014c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723fe00640 1 -- 172.21.15.32:0/722679660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724006e4b0 con 0x7f72400cdc20 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f723fe00640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 0x7f724006db60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7238036040 tx=0x7f7238004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.851+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238043020 con 0x7f72400cdc20 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.852+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723803e450 con 0x7f72400cdc20 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.852+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724006e790 con 0x7f72400cdc20 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.852+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238009510 con 0x7f72400cdc20 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.852+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724014f6c0 con 0x7f72400cdc20 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.853+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7240069510 con 0x7f72400cdc20 2024-03-06T07:28:56.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.854+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f7238046020 con 0x7f72400cdc20 2024-03-06T07:28:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.855+0000 7f7237400640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f721c06b440 0x7f721c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.855+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f72380ac570 con 0x7f72400cdc20 2024-03-06T07:28:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.855+0000 7f723f400640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f721c06b440 0x7f721c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.856+0000 7f723f400640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f721c06b440 0x7f721c06d900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7230001640 tx=0x7f7230009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:56.859+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f723804d030 con 0x7f72400cdc20 2024-03-06T07:28:57.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.089+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f724015bd40 con 0x7f72400cdc20 2024-03-06T07:28:57.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.090+0000 7f7237400640 1 -- 172.21.15.32:0/722679660 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12776 (secure 0 0 0) 0x7f723807b780 con 0x7f72400cdc20 2024-03-06T07:28:57.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:28:57.092 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:28:57.016271+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:57.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f721c06b440 msgr2=0x7f721c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f721c06b440 0x7f721c06d900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7230001640 tx=0x7f7230009040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 msgr2=0x7f724006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 0x7f724006db60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7238036040 tx=0x7f7238004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 shutdown_connections 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f721c06b440 0x7f721c06d900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72400cdc20 0x7f724006db60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f72400af580 0x7f724014f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 --2- 172.21.15.32:0/722679660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72400af1a0 0x7f724014c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 >> 172.21.15.32:0/722679660 conn(0x7f724006afb0 msgr2=0x7f72400cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 shutdown_connections 2024-03-06T07:28:57.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.093+0000 7f7246200640 1 -- 172.21.15.32:0/722679660 wait complete. 2024-03-06T07:28:57.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:57.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.301+0000 7fafb8800640 1 Processor -- start 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.301+0000 7fafb8800640 1 -- start start 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb8800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafb015bf20 0x7fafb00acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb8800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fafb0072520 0x7fafb00cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb8800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 0x7fafb00ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb8800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb0072df0 con 0x7fafb015bf20 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb8800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb00cdab0 con 0x7fafb0072520 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb8800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00b4d70 con 0x7fafb0071e20 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb5e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafb015bf20 0x7fafb00acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb5e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafb015bf20 0x7fafb00acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45524/0 (socket says 172.21.15.32:45524) 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.302+0000 7fafb5e00640 1 -- 172.21.15.32:0/818784555 learned_addr learned my addr 172.21.15.32:0/818784555 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5e00640 1 -- 172.21.15.32:0/818784555 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 msgr2=0x7fafb00ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5e00640 1 --2- 172.21.15.32:0/818784555 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 0x7fafb00ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5e00640 1 -- 172.21.15.32:0/818784555 >> v2:172.21.15.39:3300/0 conn(0x7fafb0072520 msgr2=0x7fafb00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5400640 1 --2- 172.21.15.32:0/818784555 >> v2:172.21.15.39:3300/0 conn(0x7fafb0072520 0x7fafb00cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5e00640 1 --2- 172.21.15.32:0/818784555 >> v2:172.21.15.39:3300/0 conn(0x7fafb0072520 0x7fafb00cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5e00640 1 -- 172.21.15.32:0/818784555 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00b5050 con 0x7fafb015bf20 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.303+0000 7fafb5400640 1 --2- 172.21.15.32:0/818784555 >> v2:172.21.15.39:3300/0 conn(0x7fafb0072520 0x7fafb00cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.304+0000 7fafb5e00640 1 --2- 172.21.15.32:0/818784555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb015bf20 0x7fafb00acd40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafb0070b90 tx=0x7fafa402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8938a601aa2197 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.304+0000 7fafb4a00640 1 -- 172.21.15.32:0/818784555 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4035020 con 0x7fafb015bf20 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.304+0000 7fafb4a00640 1 -- 172.21.15.32:0/818784555 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa402d040 con 0x7fafb015bf20 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.304+0000 7fafb4a00640 1 -- 172.21.15.32:0/818784555 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4031a80 con 0x7fafb015bf20 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 -- 172.21.15.32:0/818784555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb015bf20 msgr2=0x7fafb00acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 --2- 172.21.15.32:0/818784555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb015bf20 0x7fafb00acd40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafb0070b90 tx=0x7fafa402b900 comp rx=0 tx=0).stop 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 -- 172.21.15.32:0/818784555 shutdown_connections 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 --2- 172.21.15.32:0/818784555 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 0x7fafb00ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 --2- 172.21.15.32:0/818784555 >> v2:172.21.15.39:3300/0 conn(0x7fafb0072520 0x7fafb00cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 --2- 172.21.15.32:0/818784555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb015bf20 0x7fafb00acd40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 -- 172.21.15.32:0/818784555 >> 172.21.15.32:0/818784555 conn(0x7fafb006b3d0 msgr2=0x7fafb006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 -- 172.21.15.32:0/818784555 shutdown_connections 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.305+0000 7fafb8800640 1 -- 172.21.15.32:0/818784555 wait complete. 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.307+0000 7fafb8800640 1 Processor -- start 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.307+0000 7fafb8800640 1 -- start start 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.308+0000 7fafb8800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 0x7fafb01493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.308+0000 7fafb8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 0x7fafb009cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.308+0000 7fafb8800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb015bf20 0x7fafb009d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.308+0000 7fafb8800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00b4b10 con 0x7fafb0072520 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.308+0000 7fafb8800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00b4990 con 0x7fafb015bf20 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.308+0000 7fafb8800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00ad280 con 0x7fafb0071e20 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 0x7fafb009cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 0x7fafb009cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45536/0 (socket says 172.21.15.32:45536) 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 -- 172.21.15.32:0/3611804754 learned_addr learned my addr 172.21.15.32:0/3611804754 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 -- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 msgr2=0x7fafb01493f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 0x7fafb01493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 -- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb015bf20 msgr2=0x7fafb009d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb6800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb015bf20 0x7fafb009d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb015bf20 0x7fafb009d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 -- 172.21.15.32:0/3611804754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb015e220 con 0x7fafb0072520 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb6800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb015bf20 0x7fafb009d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.309+0000 7fafb5400640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 0x7fafb009cbe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafa000c7f0 tx=0x7fafa000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.310+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0015070 con 0x7fafb0072520 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.310+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa0007a30 con 0x7fafb0072520 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.310+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0007e00 con 0x7fafb0072520 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.310+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb015d180 con 0x7fafb0072520 2024-03-06T07:28:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.310+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb015d740 con 0x7fafb0072520 2024-03-06T07:28:57.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.311+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb004f330 con 0x7fafb0072520 2024-03-06T07:28:57.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.312+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fafa0018070 con 0x7fafb0072520 2024-03-06T07:28:57.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.313+0000 7fafaa000640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9406b440 0x7faf9406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.313+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fafa0089d80 con 0x7fafb0072520 2024-03-06T07:28:57.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.313+0000 7fafb5e00640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9406b440 0x7faf9406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.314+0000 7fafb5e00640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9406b440 0x7faf9406d900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafa4036000 tx=0x7fafa4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.321+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafa0058f90 con 0x7fafb0072520 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.523+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fafb0159850 con 0x7fafb0072520 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.524+0000 7fafaa000640 1 -- 172.21.15.32:0/3611804754 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fafa0058890 con 0x7fafb0072520 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.526+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9406b440 msgr2=0x7faf9406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.526+0000 7fafb8800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9406b440 0x7faf9406d900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafa4036000 tx=0x7fafa4030000 comp rx=0 tx=0).stop 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.526+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 msgr2=0x7fafb009cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.526+0000 7fafb8800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 0x7fafb009cbe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafa000c7f0 tx=0x7fafa000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 shutdown_connections 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9406b440 0x7faf9406d900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb015bf20 0x7fafb009d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb0072520 0x7fafb009cbe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 --2- 172.21.15.32:0/3611804754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafb0071e20 0x7fafb01493f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 >> 172.21.15.32:0/3611804754 conn(0x7fafb006b3d0 msgr2=0x7fafb00b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 shutdown_connections 2024-03-06T07:28:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.527+0000 7fafb8800640 1 -- 172.21.15.32:0/3611804754 wait complete. 2024-03-06T07:28:57.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.746+0000 7fe7f9e00640 1 Processor -- start 2024-03-06T07:28:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.747+0000 7fe7f9e00640 1 -- start start 2024-03-06T07:28:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.748+0000 7fe7f9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 0x7fe7f4162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.748+0000 7fe7f9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f40712c0 0x7fe7f406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.748+0000 7fe7f9e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f40ce090 con 0x7fe7f4070ee0 2024-03-06T07:28:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.748+0000 7fe7f9e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f40ceca0 con 0x7fe7f4072da0 2024-03-06T07:28:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.748+0000 7fe7f9e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40ce370 con 0x7fe7f40712c0 2024-03-06T07:28:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.749+0000 7fe7f2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 0x7fe7f4162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.749+0000 7fe7f3400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f40712c0 0x7fe7f406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.749+0000 7fe7f2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 0x7fe7f4162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:46230/0 (socket says 172.21.15.32:46230) 2024-03-06T07:28:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.749+0000 7fe7f2000640 1 -- 172.21.15.32:0/2552964249 learned_addr learned my addr 172.21.15.32:0/2552964249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.750+0000 7fe7f3400640 1 -- 172.21.15.32:0/2552964249 >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 msgr2=0x7fe7f4162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.750+0000 7fe7f3400640 1 --2- 172.21.15.32:0/2552964249 >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 0x7fe7f4162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.750+0000 7fe7f3400640 1 -- 172.21.15.32:0/2552964249 >> v1:172.21.15.39:6789/0 conn(0x7fe7f4070ee0 legacy=0x7fe7f406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.750+0000 7fe7f3400640 1 -- 172.21.15.32:0/2552964249 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40ce650 con 0x7fe7f40712c0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.750+0000 7fe7f2000640 1 --2- 172.21.15.32:0/2552964249 >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 0x7fe7f4162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.751+0000 7fe7f3400640 1 --2- 172.21.15.32:0/2552964249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f40712c0 0x7fe7f406e7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40cee20 tx=0x7fe7ec02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d3ad6d60ef44a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.751+0000 7fe7f1600640 1 -- 172.21.15.32:0/2552964249 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec039070 con 0x7fe7f40712c0 2024-03-06T07:28:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.752+0000 7fe7f1600640 1 -- 172.21.15.32:0/2552964249 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec02cc40 con 0x7fe7f40712c0 2024-03-06T07:28:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.752+0000 7fe7f1600640 1 -- 172.21.15.32:0/2552964249 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec034440 con 0x7fe7f40712c0 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.752+0000 7fe7f9e00640 1 -- 172.21.15.32:0/2552964249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f40712c0 msgr2=0x7fe7f406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.752+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/2552964249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f40712c0 0x7fe7f406e7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40cee20 tx=0x7fe7ec02be90 comp rx=0 tx=0).stop 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.753+0000 7fe7f9e00640 1 -- 172.21.15.32:0/2552964249 shutdown_connections 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.753+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/2552964249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f40712c0 0x7fe7f406e7d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.753+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/2552964249 >> v2:172.21.15.39:3300/0 conn(0x7fe7f4072da0 0x7fe7f4162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.753+0000 7fe7f9e00640 1 -- 172.21.15.32:0/2552964249 >> 172.21.15.32:0/2552964249 conn(0x7fe7f406afb0 msgr2=0x7fe7f406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.753+0000 7fe7f9e00640 1 -- 172.21.15.32:0/2552964249 shutdown_connections 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.753+0000 7fe7f9e00640 1 -- 172.21.15.32:0/2552964249 wait complete. 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.755+0000 7fe7f9e00640 1 Processor -- start 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.756+0000 7fe7f9e00640 1 -- start start 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 0x7fe7f409ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7f40712c0 0x7fe7f415c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f4072da0 0x7fe7f415c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f9e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40cdda0 con 0x7fe7f4070ee0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f9e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40cdc20 con 0x7fe7f40712c0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f9e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40cf130 con 0x7fe7f4072da0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 0x7fe7f409ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 0x7fe7f409ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45540/0 (socket says 172.21.15.32:45540) 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.757+0000 7fe7f2a00640 1 -- 172.21.15.32:0/777373647 learned_addr learned my addr 172.21.15.32:0/777373647 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2000640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7f40712c0 0x7fe7f415c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f3400640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f4072da0 0x7fe7f415c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2a00640 1 -- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f4072da0 msgr2=0x7fe7f415c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2a00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f4072da0 0x7fe7f415c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2a00640 1 -- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7f40712c0 msgr2=0x7fe7f415c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2a00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7f40712c0 0x7fe7f415c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2a00640 1 -- 172.21.15.32:0/777373647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40ae560 con 0x7fe7f4070ee0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2000640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7f40712c0 0x7fe7f415c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.758+0000 7fe7f2a00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 0x7fe7f409ce80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400c400 tx=0x7fe7e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.759+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40052d0 con 0x7fe7f4070ee0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.759+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e4005490 con 0x7fe7f4070ee0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.759+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4007260 con 0x7fe7f4070ee0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.759+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f40ae840 con 0x7fe7f4070ee0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.759+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f409af40 con 0x7fe7f4070ee0 2024-03-06T07:28:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.760+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f4069510 con 0x7fe7f4070ee0 2024-03-06T07:28:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.761+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fe7e4007400 con 0x7fe7f4070ee0 2024-03-06T07:28:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.762+0000 7fe7eb400640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe7d406b470 0x7fe7d406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.762+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe7e4089240 con 0x7fe7f4070ee0 2024-03-06T07:28:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.762+0000 7fe7f2000640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe7d406b470 0x7fe7d406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.763+0000 7fe7f2000640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe7d406b470 0x7fe7d406d930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7f41624d0 tx=0x7fe7e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.766+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7e40584e0 con 0x7fe7f4070ee0 2024-03-06T07:28:57.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.971+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe7f404f3a0 con 0x7fe7f4070ee0 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.972+0000 7fe7eb400640 1 -- 172.21.15.32:0/777373647 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe7e4057de0 con 0x7fe7f4070ee0 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:28:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:28:57.974 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe7d406b470 msgr2=0x7fe7d406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe7d406b470 0x7fe7d406d930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7f41624d0 tx=0x7fe7e0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 msgr2=0x7fe7f409ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 0x7fe7f409ce80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400c400 tx=0x7fe7e400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 shutdown_connections 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe7d406b470 0x7fe7d406d930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7f4072da0 0x7fe7f415c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7f40712c0 0x7fe7f415c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 --2- 172.21.15.32:0/777373647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f4070ee0 0x7fe7f409ce80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.974+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 >> 172.21.15.32:0/777373647 conn(0x7fe7f406afb0 msgr2=0x7fe7f40af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.975+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 shutdown_connections 2024-03-06T07:28:57.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:57.975+0000 7fe7f9e00640 1 -- 172.21.15.32:0/777373647 wait complete. 2024-03-06T07:28:57.995 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:57.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:58.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.224+0000 7f3cb1a00640 1 Processor -- start 2024-03-06T07:28:58.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.224+0000 7f3cb1a00640 1 -- start start 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.225+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac06e200 0x7f3cac0720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.225+0000 7f3cb1a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cac0725f0 0x7f3cac0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.225+0000 7f3cb1a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0b3ea0 con 0x7f3cac144320 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.225+0000 7f3cb1a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac0b4180 con 0x7f3cac0725f0 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.225+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0b4460 con 0x7f3cac06e200 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cac0725f0 0x7f3cac0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cac0725f0 0x7f3cac0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57660/0 (socket says 172.21.15.32:57660) 2024-03-06T07:28:58.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 -- 172.21.15.32:0/2513858263 learned_addr learned my addr 172.21.15.32:0/2513858263 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3ca3e00640 1 --2- 172.21.15.32:0/2513858263 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac06e200 0x7f3cac0720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 -- 172.21.15.32:0/2513858263 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac06e200 msgr2=0x7f3cac0720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 --2- 172.21.15.32:0/2513858263 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac06e200 0x7f3cac0720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 -- 172.21.15.32:0/2513858263 >> v1:172.21.15.39:6789/0 conn(0x7f3cac144320 legacy=0x7f3cac06daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.226+0000 7f3cb1000640 1 -- 172.21.15.32:0/2513858263 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0b4740 con 0x7f3cac0725f0 2024-03-06T07:28:58.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.227+0000 7f3cb1000640 1 --2- 172.21.15.32:0/2513858263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac0ad880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009e50 tx=0x7f3ca402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadc63160a6c5758 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cab400640 1 -- 172.21.15.32:0/2513858263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca402ca70 con 0x7f3cac0725f0 2024-03-06T07:28:58.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cab400640 1 -- 172.21.15.32:0/2513858263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca402cc10 con 0x7f3cac0725f0 2024-03-06T07:28:58.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cab400640 1 -- 172.21.15.32:0/2513858263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca40343f0 con 0x7f3cac0725f0 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 -- 172.21.15.32:0/2513858263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 msgr2=0x7f3cac0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/2513858263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac0ad880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009e50 tx=0x7f3ca402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 -- 172.21.15.32:0/2513858263 shutdown_connections 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/2513858263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac0ad880 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/2513858263 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac06e200 0x7f3cac0720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 -- 172.21.15.32:0/2513858263 >> 172.21.15.32:0/2513858263 conn(0x7f3cac0b17c0 msgr2=0x7f3cac0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.228+0000 7f3cb1a00640 1 -- 172.21.15.32:0/2513858263 shutdown_connections 2024-03-06T07:28:58.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.229+0000 7f3cb1a00640 1 -- 172.21.15.32:0/2513858263 wait complete. 2024-03-06T07:28:58.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.230+0000 7f3cb1a00640 1 Processor -- start 2024-03-06T07:28:58.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.230+0000 7f3cb1a00640 1 -- start start 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.231+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cac06e200 0x7f3cac15cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.231+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac146e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.231+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac144320 0x7f3cac147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.231+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac072aa0 con 0x7f3cac0725f0 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.231+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0b4f40 con 0x7f3cac06e200 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.231+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac144700 con 0x7f3cac144320 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac146e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57674/0 (socket says 172.21.15.32:57674) 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 -- 172.21.15.32:0/1495919848 learned_addr learned my addr 172.21.15.32:0/1495919848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3cb1000640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac144320 0x7f3cac147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3cabe00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cac06e200 0x7f3cac15cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 -- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac144320 msgr2=0x7f3cac147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac144320 0x7f3cac147340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 -- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cac06e200 msgr2=0x7f3cac15cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cac06e200 0x7f3cac15cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 -- 172.21.15.32:0/1495919848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac15f1f0 con 0x7f3cac0725f0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.232+0000 7f3ca3e00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac146e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400d830 tx=0x7f3c9400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.233+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400b6c0 con 0x7f3cac0725f0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.233+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c94019070 con 0x7f3cac0725f0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.233+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400ba80 con 0x7f3cac0725f0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.233+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac15f4d0 con 0x7f3cac0725f0 2024-03-06T07:28:58.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.233+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac15fa90 con 0x7f3cac0725f0 2024-03-06T07:28:58.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.234+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac04f330 con 0x7f3cac0725f0 2024-03-06T07:28:58.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.235+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f3c9401e460 con 0x7f3cac0725f0 2024-03-06T07:28:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.236+0000 7f3ca8c00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8806b470 0x7f3c8806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.236+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3c9408a440 con 0x7f3cac0725f0 2024-03-06T07:28:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.236+0000 7f3cabe00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8806b470 0x7f3c8806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.237+0000 7f3cabe00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8806b470 0x7f3c8806d930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.241+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c940596e0 con 0x7f3cac0725f0 2024-03-06T07:28:58.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:58.440+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3cac0b28b0 con 0x7f3cac0725f0 2024-03-06T07:28:59.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.265+0000 7f3ca8c00640 1 -- 172.21.15.32:0/1495919848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3c94058fe0 con 0x7f3cac0725f0 2024-03-06T07:28:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8806b470 msgr2=0x7f3c8806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8806b470 0x7f3c8806d930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 msgr2=0x7f3cac146e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac146e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400d830 tx=0x7f3c9400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 shutdown_connections 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8806b470 0x7f3c8806d930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cac144320 0x7f3cac147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cac0725f0 0x7f3cac146e00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 --2- 172.21.15.32:0/1495919848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cac06e200 0x7f3cac15cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 >> 172.21.15.32:0/1495919848 conn(0x7f3cac0b17c0 msgr2=0x7f3cac06d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.268+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 shutdown_connections 2024-03-06T07:28:59.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.269+0000 7f3cb1a00640 1 -- 172.21.15.32:0/1495919848 wait complete. 2024-03-06T07:28:59.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.512+0000 7fec98a00640 1 Processor -- start 2024-03-06T07:28:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.512+0000 7fec98a00640 1 -- start start 2024-03-06T07:28:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.513+0000 7fec98a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec94161320 0x7fec940d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.513+0000 7fec98a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec940d48d0 0x7fec940d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.513+0000 7fec98a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec940ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.513+0000 7fec98a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940d0f60 con 0x7fec94161320 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.513+0000 7fec98a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940d2be0 con 0x7fec940d48d0 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.513+0000 7fec98a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec940b3550 con 0x7fec940d0b50 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec94161320 0x7fec940d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec94161320 0x7fec940d2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57694/0 (socket says 172.21.15.32:57694) 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 -- 172.21.15.32:0/833116641 learned_addr learned my addr 172.21.15.32:0/833116641 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec93200640 1 --2- 172.21.15.32:0/833116641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec940ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 -- 172.21.15.32:0/833116641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 msgr2=0x7fec940ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 --2- 172.21.15.32:0/833116641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec940ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 -- 172.21.15.32:0/833116641 >> v2:172.21.15.39:3300/0 conn(0x7fec940d48d0 msgr2=0x7fec940d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 --2- 172.21.15.32:0/833116641 >> v2:172.21.15.39:3300/0 conn(0x7fec940d48d0 0x7fec940d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.514+0000 7fec92800640 1 -- 172.21.15.32:0/833116641 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940b3830 con 0x7fec94161320 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.515+0000 7fec92800640 1 --2- 172.21.15.32:0/833116641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec940d2320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec940d1820 tx=0x7fec8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce7aa4fc7214c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec91400640 1 -- 172.21.15.32:0/833116641 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8402ca70 con 0x7fec94161320 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec91400640 1 -- 172.21.15.32:0/833116641 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8402cc10 con 0x7fec94161320 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec91400640 1 -- 172.21.15.32:0/833116641 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec840343f0 con 0x7fec94161320 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 -- 172.21.15.32:0/833116641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 msgr2=0x7fec940d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 --2- 172.21.15.32:0/833116641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec940d2320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec940d1820 tx=0x7fec8402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 -- 172.21.15.32:0/833116641 shutdown_connections 2024-03-06T07:28:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 --2- 172.21.15.32:0/833116641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec940ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 --2- 172.21.15.32:0/833116641 >> v2:172.21.15.39:3300/0 conn(0x7fec940d48d0 0x7fec940d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 --2- 172.21.15.32:0/833116641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec940d2320 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.516+0000 7fec98a00640 1 -- 172.21.15.32:0/833116641 >> 172.21.15.32:0/833116641 conn(0x7fec940d3d40 msgr2=0x7fec940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.517+0000 7fec98a00640 1 -- 172.21.15.32:0/833116641 shutdown_connections 2024-03-06T07:28:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.517+0000 7fec98a00640 1 -- 172.21.15.32:0/833116641 wait complete. 2024-03-06T07:28:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.518+0000 7fec98a00640 1 Processor -- start 2024-03-06T07:28:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.518+0000 7fec98a00640 1 -- start start 2024-03-06T07:28:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.519+0000 7fec98a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec9414e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.519+0000 7fec98a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec940d48d0 0x7fec94150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.519+0000 7fec98a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec9414dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.519+0000 7fec98a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec940d2860 con 0x7fec94161320 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.519+0000 7fec98a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec940b3df0 con 0x7fec940d48d0 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.519+0000 7fec98a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec94161700 con 0x7fec940d0b50 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec9414dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec9414dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57708/0 (socket says 172.21.15.32:57708) 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 -- 172.21.15.32:0/843522149 learned_addr learned my addr 172.21.15.32:0/843522149 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec92800640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec9414e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 -- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 msgr2=0x7fec9414e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec9414e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 -- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec940d48d0 msgr2=0x7fec94150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec91e00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec940d48d0 0x7fec94150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec940d48d0 0x7fec94150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 -- 172.21.15.32:0/843522149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9414efd0 con 0x7fec94161320 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.520+0000 7fec93200640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec9414dbc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00d600 tx=0x7fec8c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.521+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c0125f0 con 0x7fec94161320 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.521+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c019070 con 0x7fec94161320 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.521+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c0129f0 con 0x7fec94161320 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.521+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9414f2b0 con 0x7fec94161320 2024-03-06T07:28:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.521+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9414f870 con 0x7fec94161320 2024-03-06T07:28:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.522+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9404f330 con 0x7fec94161320 2024-03-06T07:28:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.523+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec8c012b90 con 0x7fec94161320 2024-03-06T07:28:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.524+0000 7fec8aa00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.524+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fec8c08a400 con 0x7fec94161320 2024-03-06T07:28:59.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.524+0000 7fec92800640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.525+0000 7fec92800640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec7006b4d0 0x7fec7006d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fec84035000 tx=0x7fec84030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.529+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec8c059470 con 0x7fec94161320 2024-03-06T07:28:59.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:28:59.737+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fec94074380 con 0x7fec94161320 2024-03-06T07:29:00.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.284+0000 7fec8aa00640 1 -- 172.21.15.32:0/843522149 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fec8c058d70 con 0x7fec94161320 2024-03-06T07:29:00.286 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:29:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec7006b4d0 msgr2=0x7fec7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec7006b4d0 0x7fec7006d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fec84035000 tx=0x7fec84030000 comp rx=0 tx=0).stop 2024-03-06T07:29:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 msgr2=0x7fec9414dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec9414dbc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00d600 tx=0x7fec8c00dac0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 shutdown_connections 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec94161320 0x7fec9414dbc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec940d48d0 0x7fec94150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 --2- 172.21.15.32:0/843522149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec940d0b50 0x7fec9414e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 >> 172.21.15.32:0/843522149 conn(0x7fec940d3d40 msgr2=0x7fec940b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 shutdown_connections 2024-03-06T07:29:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.288+0000 7fec98a00640 1 -- 172.21.15.32:0/843522149 wait complete. 2024-03-06T07:29:00.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:29:00.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.517+0000 7f758fc00640 1 Processor -- start 2024-03-06T07:29:00.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.517+0000 7f758fc00640 1 -- start start 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758fc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f758815afa0 0x7f75880cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758fc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7588070ce0 0x7f75880ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f758806c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758fc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75880cfa10 con 0x7f758815afa0 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758fc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7588071c60 con 0x7f7588070ce0 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7588071f40 con 0x7f75880cd010 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.518+0000 7f758d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f758815afa0 0x7f75880cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f758815afa0 0x7f75880cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57712/0 (socket says 172.21.15.32:57712) 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 -- 172.21.15.32:0/1923564015 learned_addr learned my addr 172.21.15.32:0/1923564015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f7587e00640 1 --2- 172.21.15.32:0/1923564015 >> v2:172.21.15.39:3300/0 conn(0x7f7588070ce0 0x7f75880ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758dc00640 1 --2- 172.21.15.32:0/1923564015 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f758806c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 -- 172.21.15.32:0/1923564015 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 msgr2=0x7f758806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 --2- 172.21.15.32:0/1923564015 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f758806c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 -- 172.21.15.32:0/1923564015 >> v2:172.21.15.39:3300/0 conn(0x7f7588070ce0 msgr2=0x7f75880ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 --2- 172.21.15.32:0/1923564015 >> v2:172.21.15.39:3300/0 conn(0x7f7588070ce0 0x7f75880ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.519+0000 7f758d200640 1 -- 172.21.15.32:0/1923564015 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588072220 con 0x7f758815afa0 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.520+0000 7f758d200640 1 --2- 172.21.15.32:0/1923564015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f75880cf3a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f75880b3650 tx=0x7f757802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d56d0ab08140fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.520+0000 7f7587400640 1 -- 172.21.15.32:0/1923564015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757802e070 con 0x7f758815afa0 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.520+0000 7f7587400640 1 -- 172.21.15.32:0/1923564015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7578037070 con 0x7f758815afa0 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.520+0000 7f7587400640 1 -- 172.21.15.32:0/1923564015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578031a00 con 0x7f758815afa0 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 -- 172.21.15.32:0/1923564015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 msgr2=0x7f75880cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 --2- 172.21.15.32:0/1923564015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f75880cf3a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f75880b3650 tx=0x7f757802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 -- 172.21.15.32:0/1923564015 shutdown_connections 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 --2- 172.21.15.32:0/1923564015 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f758806c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 --2- 172.21.15.32:0/1923564015 >> v2:172.21.15.39:3300/0 conn(0x7f7588070ce0 0x7f75880ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 --2- 172.21.15.32:0/1923564015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f75880cf3a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 -- 172.21.15.32:0/1923564015 >> 172.21.15.32:0/1923564015 conn(0x7f758806b3d0 msgr2=0x7f758806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 -- 172.21.15.32:0/1923564015 shutdown_connections 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.521+0000 7f758fc00640 1 -- 172.21.15.32:0/1923564015 wait complete. 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.523+0000 7f758fc00640 1 Processor -- start 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.523+0000 7f758fc00640 1 -- start start 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7588070ce0 0x7f75881e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f75881e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f7588142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758806cec0 con 0x7f758815afa0 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7588071170 con 0x7f7588070ce0 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758806d040 con 0x7f75880cd010 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f7588142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f7588142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57718/0 (socket says 172.21.15.32:57718) 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758dc00640 1 -- 172.21.15.32:0/851629369 learned_addr learned my addr 172.21.15.32:0/851629369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.524+0000 7f758d200640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7588070ce0 0x7f75881e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f7587e00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f75881e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758dc00640 1 -- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 msgr2=0x7f75881e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758dc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f75881e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758dc00640 1 -- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7588070ce0 msgr2=0x7f75881e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758dc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7588070ce0 0x7f75881e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758dc00640 1 -- 172.21.15.32:0/851629369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75881565b0 con 0x7f758815afa0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758dc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f7588142310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f758000cd70 tx=0x7f758000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580003b90 con 0x7f758815afa0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580003d30 con 0x7f758815afa0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580004240 con 0x7f758815afa0 2024-03-06T07:29:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.525+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7588156890 con 0x7f758815afa0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.526+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7588140560 con 0x7f758815afa0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.527+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758804f330 con 0x7f758815afa0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.528+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75800043e0 con 0x7f758815afa0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.529+0000 7f7584c00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f756806b4d0 0x7f756806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.529+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f758008abc0 con 0x7f758815afa0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.529+0000 7f758d200640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f756806b4d0 0x7f756806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.529+0000 7f758d200640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f756806b4d0 0x7f756806d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7578036000 tx=0x7f757803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.533+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7580059b30 con 0x7f758815afa0 2024-03-06T07:29:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:00.776+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f758806e500 con 0x7f758815afa0 2024-03-06T07:29:01.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.303+0000 7f7584c00640 1 -- 172.21.15.32:0/851629369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7580059430 con 0x7f758815afa0 2024-03-06T07:29:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:29:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f756806b4d0 msgr2=0x7f756806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f756806b4d0 0x7f756806d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7578036000 tx=0x7f757803d040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 msgr2=0x7f7588142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f7588142310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f758000cd70 tx=0x7f758000a660 comp rx=0 tx=0).stop 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 shutdown_connections 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f756806b4d0 0x7f756806d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f758815afa0 0x7f7588142310 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75880cd010 0x7f75881e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 --2- 172.21.15.32:0/851629369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7588070ce0 0x7f75881e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 >> 172.21.15.32:0/851629369 conn(0x7f758806b3d0 msgr2=0x7f758806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.311+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 shutdown_connections 2024-03-06T07:29:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.312+0000 7f758fc00640 1 -- 172.21.15.32:0/851629369 wait complete. 2024-03-06T07:29:01.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.525+0000 7f4d65a00640 1 Processor -- start 2024-03-06T07:29:01.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.525+0000 7f4d65a00640 1 -- start start 2024-03-06T07:29:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d65a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 0x7f4d6015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d65a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4d600cdc20 0x7f4d600b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d65a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6015cc40 con 0x7f4d600af580 2024-03-06T07:29:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d65a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6015cf20 con 0x7f4d600cdc20 2024-03-06T07:29:01.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d65a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6015d200 con 0x7f4d600af1a0 2024-03-06T07:29:01.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d5e000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4d600af580 0x7f4d600b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46640/0 (socket says 172.21.15.32:46640) 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.527+0000 7f4d5e000640 1 -- 172.21.15.32:0/916576474 learned_addr learned my addr 172.21.15.32:0/916576474 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 0x7f4d6015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5f400640 1 --2- 172.21.15.32:0/916576474 >> v2:172.21.15.39:3300/0 conn(0x7f4d600cdc20 0x7f4d600b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046422327 0 0) 0x7f4d6015cc40 con 0x7f4d600af580 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d44003520 con 0x7f4d600af580 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240958817 0 0) 0x7f4d44003520 con 0x7f4d600af580 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 msgr2=0x7f4d6015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 0x7f4d6015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 >> v2:172.21.15.39:3300/0 conn(0x7f4d600cdc20 msgr2=0x7f4d600b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 --2- 172.21.15.32:0/916576474 >> v2:172.21.15.39:3300/0 conn(0x7f4d600cdc20 0x7f4d600b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6015d4e0 con 0x7f4d600af580 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f4d500019f0 con 0x7f4d600af580 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 >> v1:172.21.15.32:6789/0 conn(0x7f4d600af580 legacy=0x7f4d600b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.528+0000 7f4d5d600640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d44003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5d600640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 0x7f4d44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5d600640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d44004d40 0x7f4d440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6015cf20 con 0x7f4d44003a20 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6015d200 con 0x7f4d44004d40 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4400bac0 con 0x7f4d600af1a0 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d44003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5ea00640 1 -- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 msgr2=0x7f4d44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d600af1a0 0x7f4d44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5ea00640 1 -- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d44004d40 msgr2=0x7f4d440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5f400640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d44004d40 0x7f4d440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d44004d40 0x7f4d440051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.529+0000 7f4d5ea00640 1 -- 172.21.15.32:0/916576474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44003520 con 0x7f4d44003a20 2024-03-06T07:29:01.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.530+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d44003e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009ac0 tx=0x7f4d44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9e457fa1530646 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48039070 con 0x7f4d44003a20 2024-03-06T07:29:01.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4802c710 con 0x7f4d44003a20 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d5d600640 1 -- 172.21.15.32:0/916576474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48009690 con 0x7f4d44003a20 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d65a00640 1 -- 172.21.15.32:0/916576474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 msgr2=0x7f4d44003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d65a00640 1 --2- 172.21.15.32:0/916576474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d44003e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009ac0 tx=0x7f4d44005830 comp rx=0 tx=0).stop 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d5f400640 1 -- 172.21.15.32:0/916576474 reap_dead start 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d65a00640 1 -- 172.21.15.32:0/916576474 shutdown_connections 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d65a00640 1 -- 172.21.15.32:0/916576474 >> 172.21.15.32:0/916576474 conn(0x7f4d6006afb0 msgr2=0x7f4d6006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d65a00640 1 -- 172.21.15.32:0/916576474 shutdown_connections 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.531+0000 7f4d65a00640 1 -- 172.21.15.32:0/916576474 wait complete. 2024-03-06T07:29:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.533+0000 7f4d65a00640 1 Processor -- start 2024-03-06T07:29:01.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.533+0000 7f4d65a00640 1 -- start start 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d65a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d6009d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d65a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d6009b3f0 0x7f4d6006cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d65a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6009b850 0x7f4d6006d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d65a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6015c990 con 0x7f4d44003a20 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d65a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6015c810 con 0x7f4d6009b850 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d65a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600b3310 con 0x7f4d6009b3f0 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d5ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d6009d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d5ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d6009d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57742/0 (socket says 172.21.15.32:57742) 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.534+0000 7f4d5ea00640 1 -- 172.21.15.32:0/327941370 learned_addr learned my addr 172.21.15.32:0/327941370 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5ea00640 1 -- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d6009b3f0 msgr2=0x7f4d6006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5e000640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d6009b3f0 0x7f4d6006cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5f400640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6009b850 0x7f4d6006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d6009b3f0 0x7f4d6006cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5ea00640 1 -- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6009b850 msgr2=0x7f4d6006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6009b850 0x7f4d6006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5ea00640 1 -- 172.21.15.32:0/327941370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6009bdd0 con 0x7f4d44003a20 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5e000640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d6009b3f0 0x7f4d6006cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.535+0000 7f4d5ea00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d6009d740 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4d48000c00 tx=0x7f4d48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.536+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48042020 con 0x7f4d44003a20 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.536+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4803a040 con 0x7f4d44003a20 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.536+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48039070 con 0x7f4d44003a20 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.536+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600c93f0 con 0x7f4d44003a20 2024-03-06T07:29:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.536+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d600c99b0 con 0x7f4d44003a20 2024-03-06T07:29:01.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.537+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60069510 con 0x7f4d44003a20 2024-03-06T07:29:01.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.538+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d48045020 con 0x7f4d44003a20 2024-03-06T07:29:01.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.539+0000 7f4d56a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d3c06b4d0 0x7f4d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.539+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d480ab800 con 0x7f4d44003a20 2024-03-06T07:29:01.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.539+0000 7f4d5e000640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d3c06b4d0 0x7f4d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.540+0000 7f4d5e000640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d3c06b4d0 0x7f4d3c06d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d500012e0 tx=0x7f4d50009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.543+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d4804c070 con 0x7f4d44003a20 2024-03-06T07:29:01.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.743+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d600b38f0 con 0x7f4d44003a20 2024-03-06T07:29:01.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.745+0000 7f4d56a00640 1 -- 172.21.15.32:0/327941370 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16510 (secure 0 0 0) 0x7f4d4807a550 con 0x7f4d44003a20 2024-03-06T07:29:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:01.747 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:00.777798+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:01.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d3c06b4d0 msgr2=0x7f4d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d3c06b4d0 0x7f4d3c06d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d500012e0 tx=0x7f4d50009040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 msgr2=0x7f4d6009d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d6009d740 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4d48000c00 tx=0x7f4d48004730 comp rx=0 tx=0).stop 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 shutdown_connections 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d3c06b4d0 0x7f4d3c06d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6009b850 0x7f4d6006d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d6009b3f0 0x7f4d6006cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 --2- 172.21.15.32:0/327941370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d44003a20 0x7f4d6009d740 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 >> 172.21.15.32:0/327941370 conn(0x7f4d6006afb0 msgr2=0x7f4d600ad970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 shutdown_connections 2024-03-06T07:29:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.748+0000 7f4d65a00640 1 -- 172.21.15.32:0/327941370 wait complete. 2024-03-06T07:29:01.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:01.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.982+0000 7fe30d400640 1 Processor -- start 2024-03-06T07:29:01.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.982+0000 7fe30d400640 1 -- start start 2024-03-06T07:29:01.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.983+0000 7fe30d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe3081552b0 0x7fe30815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.983+0000 7fe30d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe30815e810 0x7fe3080cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3080cce90 0x7fe30806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30d400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30815ed10 con 0x7fe3080cce90 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30d400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30815eff0 con 0x7fe3081552b0 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30815f2d0 con 0x7fe30815e810 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3080cce90 0x7fe30806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3080cce90 0x7fe30806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57752/0 (socket says 172.21.15.32:57752) 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 -- 172.21.15.32:0/1497693530 learned_addr learned my addr 172.21.15.32:0/1497693530 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe307400640 1 --2- 172.21.15.32:0/1497693530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe30815e810 0x7fe3080cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe307e00640 1 --2- 172.21.15.32:0/1497693530 >> v2:172.21.15.39:3300/0 conn(0x7fe3081552b0 0x7fe30815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 -- 172.21.15.32:0/1497693530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe30815e810 msgr2=0x7fe3080cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 --2- 172.21.15.32:0/1497693530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe30815e810 0x7fe3080cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 -- 172.21.15.32:0/1497693530 >> v2:172.21.15.39:3300/0 conn(0x7fe3081552b0 msgr2=0x7fe30815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 --2- 172.21.15.32:0/1497693530 >> v2:172.21.15.39:3300/0 conn(0x7fe3081552b0 0x7fe30815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.984+0000 7fe30ca00640 1 -- 172.21.15.32:0/1497693530 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30815f5b0 con 0x7fe3080cce90 2024-03-06T07:29:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.986+0000 7fe30ca00640 1 --2- 172.21.15.32:0/1497693530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe30806c7c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3080cd540 tx=0x7fe30002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ed054d4e3da81c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.986+0000 7fe306a00640 1 -- 172.21.15.32:0/1497693530 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30002c9e0 con 0x7fe3080cce90 2024-03-06T07:29:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.986+0000 7fe306a00640 1 -- 172.21.15.32:0/1497693530 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30002cb80 con 0x7fe3080cce90 2024-03-06T07:29:01.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.986+0000 7fe306a00640 1 -- 172.21.15.32:0/1497693530 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3000343f0 con 0x7fe3080cce90 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.986+0000 7fe30d400640 1 -- 172.21.15.32:0/1497693530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 msgr2=0x7fe30806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.986+0000 7fe30d400640 1 --2- 172.21.15.32:0/1497693530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe30806c7c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3080cd540 tx=0x7fe30002be40 comp rx=0 tx=0).stop 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 -- 172.21.15.32:0/1497693530 shutdown_connections 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 --2- 172.21.15.32:0/1497693530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe30806c7c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 --2- 172.21.15.32:0/1497693530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe30815e810 0x7fe3080cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 --2- 172.21.15.32:0/1497693530 >> v2:172.21.15.39:3300/0 conn(0x7fe3081552b0 0x7fe30815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 -- 172.21.15.32:0/1497693530 >> 172.21.15.32:0/1497693530 conn(0x7fe30806b3d0 msgr2=0x7fe30806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 -- 172.21.15.32:0/1497693530 shutdown_connections 2024-03-06T07:29:01.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.987+0000 7fe30d400640 1 -- 172.21.15.32:0/1497693530 wait complete. 2024-03-06T07:29:01.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.989+0000 7fe30d400640 1 Processor -- start 2024-03-06T07:29:01.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.989+0000 7fe30d400640 1 -- start start 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe308150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3081552b0 0x7fe30813a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe30815e810 0x7fe30814ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30806dd30 con 0x7fe3080cce90 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080b3760 con 0x7fe30815e810 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30806deb0 con 0x7fe3081552b0 2024-03-06T07:29:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe308150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe308150020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57766/0 (socket says 172.21.15.32:57766) 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 -- 172.21.15.32:0/4286011025 learned_addr learned my addr 172.21.15.32:0/4286011025 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe30ca00640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe30815e810 0x7fe30814ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 -- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3081552b0 msgr2=0x7fe30813a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3081552b0 0x7fe30813a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 -- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe30815e810 msgr2=0x7fe30814ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe30815e810 0x7fe30814ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.990+0000 7fe307e00640 1 -- 172.21.15.32:0/4286011025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe308138f80 con 0x7fe3080cce90 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.991+0000 7fe307e00640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe308150020 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800d800 tx=0x7fe2f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.991+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8018030 con 0x7fe3080cce90 2024-03-06T07:29:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.991+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f80096a0 con 0x7fe3080cce90 2024-03-06T07:29:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.991+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8009870 con 0x7fe3080cce90 2024-03-06T07:29:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.991+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30813b2b0 con 0x7fe3080cce90 2024-03-06T07:29:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.991+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30813b870 con 0x7fe3080cce90 2024-03-06T07:29:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.992+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30804f330 con 0x7fe3080cce90 2024-03-06T07:29:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.993+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2f801b070 con 0x7fe3080cce90 2024-03-06T07:29:01.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.994+0000 7fe2ffe00640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2e406b4d0 0x7fe2e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.994+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2f808a550 con 0x7fe3080cce90 2024-03-06T07:29:01.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.994+0000 7fe307400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2e406b4d0 0x7fe2e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.995+0000 7fe307400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2e406b4d0 0x7fe2e406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe3081517b0 tx=0x7fe2f0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:01.998+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2f8059330 con 0x7fe3080cce90 2024-03-06T07:29:02.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.201+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3080cf4a0 con 0x7fe3080cce90 2024-03-06T07:29:02.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.204+0000 7fe2ffe00640 1 -- 172.21.15.32:0/4286011025 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16510 (secure 0 0 0) 0x7fe2f8058c30 con 0x7fe3080cce90 2024-03-06T07:29:02.206 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:02.206 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:00.777798+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:02.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.206+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2e406b4d0 msgr2=0x7fe2e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.206+0000 7fe30d400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2e406b4d0 0x7fe2e406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe3081517b0 tx=0x7fe2f0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.206+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 msgr2=0x7fe308150020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.206+0000 7fe30d400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe308150020 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800d800 tx=0x7fe2f800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 shutdown_connections 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2e406b4d0 0x7fe2e406d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe30815e810 0x7fe30814ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3081552b0 0x7fe30813a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 --2- 172.21.15.32:0/4286011025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3080cce90 0x7fe308150020 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 >> 172.21.15.32:0/4286011025 conn(0x7fe30806b3d0 msgr2=0x7fe308072060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 shutdown_connections 2024-03-06T07:29:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.207+0000 7fe30d400640 1 -- 172.21.15.32:0/4286011025 wait complete. 2024-03-06T07:29:02.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.445+0000 7f9462c00640 1 Processor -- start 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.445+0000 7f9462c00640 1 -- start start 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.446+0000 7f9462c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f945c0b3c20 0x7f945c0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.446+0000 7f9462c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f945c0b4000 0x7f945c15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9462c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c071930 con 0x7f945c0b1b40 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9462c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0b4490 con 0x7f945c0b3c20 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9462c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c071c10 con 0x7f945c0b4000 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f945c0b3c20 0x7f945c0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f945c0b3c20 0x7f945c0729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57774/0 (socket says 172.21.15.32:57774) 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 -- 172.21.15.32:0/3155409868 learned_addr learned my addr 172.21.15.32:0/3155409868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9460e00640 1 --2- 172.21.15.32:0/3155409868 >> v2:172.21.15.39:3300/0 conn(0x7f945c0b4000 0x7f945c15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 -- 172.21.15.32:0/3155409868 >> v2:172.21.15.39:3300/0 conn(0x7f945c0b4000 msgr2=0x7f945c15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 --2- 172.21.15.32:0/3155409868 >> v2:172.21.15.39:3300/0 conn(0x7f945c0b4000 0x7f945c15b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 -- 172.21.15.32:0/3155409868 >> v1:172.21.15.39:6789/0 conn(0x7f945c0b1b40 legacy=0x7f945c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9461800640 1 -- 172.21.15.32:0/3155409868 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c071ef0 con 0x7f945c0b3c20 2024-03-06T07:29:02.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.447+0000 7f9460e00640 1 --2- 172.21.15.32:0/3155409868 >> v2:172.21.15.39:3300/0 conn(0x7f945c0b4000 0x7f945c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.448+0000 7f9461800640 1 --2- 172.21.15.32:0/3155409868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b3c20 0x7f945c0729d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f945c064970 tx=0x7f945002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94725a2d6d6c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f945be00640 1 -- 172.21.15.32:0/3155409868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450035020 con 0x7f945c0b3c20 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f945be00640 1 -- 172.21.15.32:0/3155409868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450031870 con 0x7f945c0b3c20 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f945be00640 1 -- 172.21.15.32:0/3155409868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450031c40 con 0x7f945c0b3c20 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f9462c00640 1 -- 172.21.15.32:0/3155409868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b3c20 msgr2=0x7f945c0729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f9462c00640 1 --2- 172.21.15.32:0/3155409868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b3c20 0x7f945c0729d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f945c064970 tx=0x7f945002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f9462c00640 1 -- 172.21.15.32:0/3155409868 shutdown_connections 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f9462c00640 1 --2- 172.21.15.32:0/3155409868 >> v2:172.21.15.39:3300/0 conn(0x7f945c0b4000 0x7f945c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f9462c00640 1 --2- 172.21.15.32:0/3155409868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b3c20 0x7f945c0729d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.449+0000 7f9462c00640 1 -- 172.21.15.32:0/3155409868 >> 172.21.15.32:0/3155409868 conn(0x7f945c06b3a0 msgr2=0x7f945c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.450+0000 7f9462c00640 1 -- 172.21.15.32:0/3155409868 shutdown_connections 2024-03-06T07:29:02.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.450+0000 7f9462c00640 1 -- 172.21.15.32:0/3155409868 wait complete. 2024-03-06T07:29:02.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.451+0000 7f9462c00640 1 Processor -- start 2024-03-06T07:29:02.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.451+0000 7f9462c00640 1 -- start start 2024-03-06T07:29:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.452+0000 7f9462c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f945c0b1b40 0x7f945c1482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.452+0000 7f9462c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f945c0b3c20 0x7f945c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.452+0000 7f9462c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 0x7f945c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.452+0000 7f9462c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c15bde0 con 0x7f945c0b4000 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.452+0000 7f9462c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0724b0 con 0x7f945c0b1b40 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.452+0000 7f9462c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c15bf60 con 0x7f945c0b3c20 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 0x7f945c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 0x7f945c0b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57778/0 (socket says 172.21.15.32:57778) 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 -- 172.21.15.32:0/1408467363 learned_addr learned my addr 172.21.15.32:0/1408467363 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9461800640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f945c0b1b40 0x7f945c1482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9460e00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f945c0b3c20 0x7f945c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 -- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f945c0b3c20 msgr2=0x7f945c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f945c0b3c20 0x7f945c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 -- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f945c0b1b40 msgr2=0x7f945c1482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f945c0b1b40 0x7f945c1482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.453+0000 7f9462200640 1 -- 172.21.15.32:0/1408467363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c1e4130 con 0x7f945c0b4000 2024-03-06T07:29:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.454+0000 7f9462200640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 0x7f945c0b0940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f945400c780 tx=0x7f945400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.454+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454017070 con 0x7f945c0b4000 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.454+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9454003da0 con 0x7f945c0b4000 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.454+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540052b0 con 0x7f945c0b4000 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.454+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c1e4410 con 0x7f945c0b4000 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.454+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c1e49d0 con 0x7f945c0b4000 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.455+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c04f330 con 0x7f945c0b4000 2024-03-06T07:29:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.456+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9454005450 con 0x7f945c0b4000 2024-03-06T07:29:02.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.457+0000 7f9459600640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f943c06b4d0 0x7f943c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.457+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9454089780 con 0x7f945c0b4000 2024-03-06T07:29:02.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.457+0000 7f9461800640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f943c06b4d0 0x7f943c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.458+0000 7f9461800640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f943c06b4d0 0x7f943c06d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9450036000 tx=0x7f945003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.462+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94540585f0 con 0x7f945c0b4000 2024-03-06T07:29:02.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:02.705+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f945c074ab0 con 0x7f945c0b4000 2024-03-06T07:29:03.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.323+0000 7f9459600640 1 -- 172.21.15.32:0/1408467363 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9454057ef0 con 0x7f945c0b4000 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f943c06b4d0 msgr2=0x7f943c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f943c06b4d0 0x7f943c06d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9450036000 tx=0x7f945003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 msgr2=0x7f945c0b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 0x7f945c0b0940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f945400c780 tx=0x7f945400cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 shutdown_connections 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f943c06b4d0 0x7f943c06d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f945c0b4000 0x7f945c0b0940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f945c0b3c20 0x7f945c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 --2- 172.21.15.32:0/1408467363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f945c0b1b40 0x7f945c1482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.327+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 >> 172.21.15.32:0/1408467363 conn(0x7f945c06b3a0 msgr2=0x7f945c0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.328+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 shutdown_connections 2024-03-06T07:29:03.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.328+0000 7f9462c00640 1 -- 172.21.15.32:0/1408467363 wait complete. 2024-03-06T07:29:03.352 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:29:03.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:29:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.577+0000 7fa7a6600640 1 Processor -- start 2024-03-06T07:29:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.577+0000 7fa7a6600640 1 -- start start 2024-03-06T07:29:03.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a6600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7a015f810 0x7fa7a00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a6600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 0x7fa7a00b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a6600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a015fe40 con 0x7fa7a0145430 2024-03-06T07:29:03.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a6600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a0160120 con 0x7fa7a015f810 2024-03-06T07:29:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a6600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a0160400 con 0x7fa7a00b2080 2024-03-06T07:29:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a5200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7a0145430 0x7fa7a015cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46644/0 (socket says 172.21.15.32:46644) 2024-03-06T07:29:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa7a5200640 1 -- 172.21.15.32:0/2736584056 learned_addr learned my addr 172.21.15.32:0/2736584056 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.579+0000 7fa79fe00640 1 --2- 172.21.15.32:0/2736584056 >> v2:172.21.15.39:3300/0 conn(0x7fa7a015f810 0x7fa7a00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117902570 0 0) 0x7fa7a015fe40 con 0x7fa7a0145430 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa788003520 con 0x7fa7a0145430 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa7a5c00640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 0x7fa7a00b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986785516 0 0) 0x7fa788003520 con 0x7fa7a0145430 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 msgr2=0x7fa7a00b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 0x7fa7a00b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 >> v2:172.21.15.39:3300/0 conn(0x7fa7a015f810 msgr2=0x7fa7a00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 --2- 172.21.15.32:0/2736584056 >> v2:172.21.15.39:3300/0 conn(0x7fa7a015f810 0x7fa7a00b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a01606e0 con 0x7fa7a0145430 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa794001670 con 0x7fa7a0145430 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 >> v1:172.21.15.32:6789/0 conn(0x7fa7a0145430 legacy=0x7fa7a015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7880039b0 0x7fa788003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa788004800 0x7fa788004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 0x7fa7880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a0160120 con 0x7fa788004800 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a0160400 con 0x7fa7880039b0 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78800b9c0 con 0x7fa7a00b2080 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.580+0000 7fa79fe00640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa788004800 0x7fa788004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa7a5200640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7880039b0 0x7fa788003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa7a5c00640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 0x7fa7880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa79fe00640 1 -- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 msgr2=0x7fa7880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa79fe00640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7a00b2080 0x7fa7880051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa79fe00640 1 -- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7880039b0 msgr2=0x7fa788003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa79fe00640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7880039b0 0x7fa788003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa79fe00640 1 -- 172.21.15.32:0/2736584056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788003520 con 0x7fa788004800 2024-03-06T07:29:03.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.581+0000 7fa7a5200640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7880039b0 0x7fa788003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.582+0000 7fa79fe00640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa788004800 0x7fa788004c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009ac0 tx=0x7fa788005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e983e083fc0633d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.582+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c039070 con 0x7fa788004800 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.582+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c02c710 con 0x7fa788004800 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.582+0000 7fa79f400640 1 -- 172.21.15.32:0/2736584056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c009690 con 0x7fa788004800 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a6600640 1 -- 172.21.15.32:0/2736584056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa788004800 msgr2=0x7fa788004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a6600640 1 --2- 172.21.15.32:0/2736584056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa788004800 0x7fa788004c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009ac0 tx=0x7fa788005830 comp rx=0 tx=0).stop 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a5c00640 1 -- 172.21.15.32:0/2736584056 reap_dead start 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a6600640 1 -- 172.21.15.32:0/2736584056 shutdown_connections 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a6600640 1 -- 172.21.15.32:0/2736584056 >> 172.21.15.32:0/2736584056 conn(0x7fa7a006b3a0 msgr2=0x7fa7a006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a6600640 1 -- 172.21.15.32:0/2736584056 shutdown_connections 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.583+0000 7fa7a6600640 1 -- 172.21.15.32:0/2736584056 wait complete. 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.585+0000 7fa7a6600640 1 Processor -- start 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.585+0000 7fa7a6600640 1 -- start start 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a6600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa788004800 0x7fa7a0141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a6600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 0x7fa7a0151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a6600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a0152170 0x7fa7a01436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a6600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00b2490 con 0x7fa7a01517d0 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a6600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a015fbf0 con 0x7fa7a0152170 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a6600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00cde00 con 0x7fa788004800 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 0x7fa7a0151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa79fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 0x7fa7a0151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57784/0 (socket says 172.21.15.32:57784) 2024-03-06T07:29:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa79fe00640 1 -- 172.21.15.32:0/1853294022 learned_addr learned my addr 172.21.15.32:0/1853294022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a5200640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa788004800 0x7fa7a0141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.586+0000 7fa7a5c00640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a0152170 0x7fa7a01436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79fe00640 1 -- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa788004800 msgr2=0x7fa7a0141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79fe00640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa788004800 0x7fa7a0141e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79fe00640 1 -- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a0152170 msgr2=0x7fa7a01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79fe00640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a0152170 0x7fa7a01436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79fe00640 1 -- 172.21.15.32:0/1853294022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a0144f80 con 0x7fa7a01517d0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa7a5c00640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a0152170 0x7fa7a01436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79fe00640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 0x7fa7a0151c30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa78c037040 tx=0x7fa78c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c039050 con 0x7fa7a01517d0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c045070 con 0x7fa7a01517d0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a01525f0 con 0x7fa7a01517d0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.587+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c003a50 con 0x7fa7a01517d0 2024-03-06T07:29:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.588+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a0142c10 con 0x7fa7a01517d0 2024-03-06T07:29:03.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.589+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a004f330 con 0x7fa7a01517d0 2024-03-06T07:29:03.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.589+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa78c003bf0 con 0x7fa7a01517d0 2024-03-06T07:29:03.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.590+0000 7fa79cc00640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa78006b4d0 0x7fa78006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.591+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa78c0ab900 con 0x7fa7a01517d0 2024-03-06T07:29:03.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.591+0000 7fa7a5200640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa78006b4d0 0x7fa78006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.591+0000 7fa7a5200640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa78006b4d0 0x7fa78006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa794005b70 tx=0x7fa794009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.595+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7a0144f80 con 0x7fa7a01517d0 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:03.834+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa7a006e240 con 0x7fa7a01517d0 2024-03-06T07:29:04.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.336+0000 7fa79cc00640 1 -- 172.21.15.32:0/1853294022 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa78c07a780 con 0x7fa7a01517d0 2024-03-06T07:29:04.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa78006b4d0 msgr2=0x7fa78006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa78006b4d0 0x7fa78006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa794005b70 tx=0x7fa794009040 comp rx=0 tx=0).stop 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 msgr2=0x7fa7a0151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 0x7fa7a0151c30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa78c037040 tx=0x7fa78c004730 comp rx=0 tx=0).stop 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 shutdown_connections 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa78006b4d0 0x7fa78006d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a0152170 0x7fa7a01436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7a01517d0 0x7fa7a0151c30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 --2- 172.21.15.32:0/1853294022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa788004800 0x7fa7a0141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 >> 172.21.15.32:0/1853294022 conn(0x7fa7a006b3a0 msgr2=0x7fa7a006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 shutdown_connections 2024-03-06T07:29:04.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.341+0000 7fa7a6600640 1 -- 172.21.15.32:0/1853294022 wait complete. 2024-03-06T07:29:04.362 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:29:04.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:04.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T07:29:04.371+0000 7f67d8200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.576+0000 7f73a2e00640 1 Processor -- start 2024-03-06T07:29:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.577+0000 7f73a2e00640 1 -- start start 2024-03-06T07:29:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.578+0000 7f73a2e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f739c144c00 0x7f739c145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.578+0000 7f73a2e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f739c1457c0 0x7f739c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.578+0000 7f73a2e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c0d0910 con 0x7f739c06b4f0 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.578+0000 7f73a2e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739c0d0bf0 con 0x7f739c1457c0 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.578+0000 7f73a2e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739c0d0ed0 con 0x7f739c144c00 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f739c1457c0 0x7f739c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f739c1457c0 0x7f739c06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57786/0 (socket says 172.21.15.32:57786) 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a2400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f739c06b4f0 0x7f739c15f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46650/0 (socket says 172.21.15.32:46650) 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 -- 172.21.15.32:0/2838240570 learned_addr learned my addr 172.21.15.32:0/2838240570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1a00640 1 --2- 172.21.15.32:0/2838240570 >> v2:172.21.15.39:3300/0 conn(0x7f739c144c00 0x7f739c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 -- 172.21.15.32:0/2838240570 >> v2:172.21.15.39:3300/0 conn(0x7f739c144c00 msgr2=0x7f739c145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 --2- 172.21.15.32:0/2838240570 >> v2:172.21.15.39:3300/0 conn(0x7f739c144c00 0x7f739c145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 -- 172.21.15.32:0/2838240570 >> v1:172.21.15.32:6789/0 conn(0x7f739c06b4f0 legacy=0x7f739c15f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f73a1000640 1 -- 172.21.15.32:0/2838240570 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c0d11b0 con 0x7f739c1457c0 2024-03-06T07:29:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.579+0000 7f739be00640 1 -- 172.21.15.32:0/2838240570 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051905600 0 0) 0x7f739c0d0910 con 0x7f739c06b4f0 2024-03-06T07:29:04.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.580+0000 7f73a1000640 1 --2- 172.21.15.32:0/2838240570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c1457c0 0x7f739c06afb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f739c06bcf0 tx=0x7f738c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e00f72dcac4afc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f739be00640 1 -- 172.21.15.32:0/2838240570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c034070 con 0x7f739c1457c0 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f739be00640 1 -- 172.21.15.32:0/2838240570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c030860 con 0x7f739c1457c0 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f739be00640 1 -- 172.21.15.32:0/2838240570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c030c30 con 0x7f739c1457c0 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 -- 172.21.15.32:0/2838240570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c1457c0 msgr2=0x7f739c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 --2- 172.21.15.32:0/2838240570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c1457c0 0x7f739c06afb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f739c06bcf0 tx=0x7f738c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 -- 172.21.15.32:0/2838240570 shutdown_connections 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 --2- 172.21.15.32:0/2838240570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c1457c0 0x7f739c06afb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 --2- 172.21.15.32:0/2838240570 >> v2:172.21.15.39:3300/0 conn(0x7f739c144c00 0x7f739c145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 -- 172.21.15.32:0/2838240570 >> 172.21.15.32:0/2838240570 conn(0x7f739c06c0c0 msgr2=0x7f739c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 -- 172.21.15.32:0/2838240570 shutdown_connections 2024-03-06T07:29:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.581+0000 7f73a2e00640 1 -- 172.21.15.32:0/2838240570 wait complete. 2024-03-06T07:29:04.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.583+0000 7f73a2e00640 1 Processor -- start 2024-03-06T07:29:04.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.583+0000 7f73a2e00640 1 -- start start 2024-03-06T07:29:04.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.584+0000 7f73a2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 0x7f739c1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.584+0000 7f73a2e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f739c144c00 0x7f739c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.584+0000 7f73a2e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f739c1457c0 0x7f739c156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.584+0000 7f73a2e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c0d19b0 con 0x7f739c06b4f0 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.584+0000 7f73a2e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c144fe0 con 0x7f739c1457c0 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.584+0000 7f73a2e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739c0d0140 con 0x7f739c144c00 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 0x7f739c1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 0x7f739c1f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57798/0 (socket says 172.21.15.32:57798) 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 -- 172.21.15.32:0/1730467103 learned_addr learned my addr 172.21.15.32:0/1730467103 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 -- 172.21.15.32:0/1730467103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f739c144c00 msgr2=0x7f739c15aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f739c144c00 0x7f739c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 -- 172.21.15.32:0/1730467103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f739c1457c0 msgr2=0x7f739c156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f739c1457c0 0x7f739c156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 -- 172.21.15.32:0/1730467103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c155970 con 0x7f739c06b4f0 2024-03-06T07:29:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.585+0000 7f73a1a00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 0x7f739c1f2c60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f739000e9b0 tx=0x7f739000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.586+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7390017070 con 0x7f739c06b4f0 2024-03-06T07:29:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.586+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739000c9a0 con 0x7f739c06b4f0 2024-03-06T07:29:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.586+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739000cd90 con 0x7f739c06b4f0 2024-03-06T07:29:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.586+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c155c50 con 0x7f739c06b4f0 2024-03-06T07:29:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.586+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c156210 con 0x7f739c06b4f0 2024-03-06T07:29:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.587+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739c04f330 con 0x7f739c06b4f0 2024-03-06T07:29:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.588+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f739001a070 con 0x7f739c06b4f0 2024-03-06T07:29:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.589+0000 7f7399600640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f738006b500 0x7f738006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.589+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f739008a200 con 0x7f739c06b4f0 2024-03-06T07:29:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.590+0000 7f73a1000640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f738006b500 0x7f738006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.590+0000 7f73a1000640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f738006b500 0x7f738006d9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f738c036000 tx=0x7f738c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.594+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7390059070 con 0x7f739c06b4f0 2024-03-06T07:29:04.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.831+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f739c0722f0 con 0x7f739c06b4f0 2024-03-06T07:29:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.832+0000 7f7399600640 1 -- 172.21.15.32:0/1730467103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f7390058970 con 0x7f739c06b4f0 2024-03-06T07:29:04.835 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:04.835 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.835+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f738006b500 msgr2=0x7f738006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.835+0000 7f73a2e00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f738006b500 0x7f738006d9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f738c036000 tx=0x7f738c03b040 comp rx=0 tx=0).stop 2024-03-06T07:29:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.835+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 msgr2=0x7f739c1f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.835+0000 7f73a2e00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 0x7f739c1f2c60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f739000e9b0 tx=0x7f739000ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 shutdown_connections 2024-03-06T07:29:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f738006b500 0x7f738006d9c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f739c1457c0 0x7f739c156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f739c144c00 0x7f739c15aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 --2- 172.21.15.32:0/1730467103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c06b4f0 0x7f739c1f2c60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 >> 172.21.15.32:0/1730467103 conn(0x7f739c06c0c0 msgr2=0x7f739c15ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 shutdown_connections 2024-03-06T07:29:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:04.836+0000 7f73a2e00640 1 -- 172.21.15.32:0/1730467103 wait complete. 2024-03-06T07:29:04.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.077+0000 7f502ba00640 1 Processor -- start 2024-03-06T07:29:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.077+0000 7f502ba00640 1 -- start start 2024-03-06T07:29:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f502ba00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50240af1a0 0x7f502415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f502ba00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f50240af580 0x7f50240b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f502ba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240cdc20 0x7f50240b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f502ba00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502415cc40 con 0x7f50240af1a0 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f502ba00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502415cf20 con 0x7f50240af580 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f502ba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502415d200 con 0x7f50240cdc20 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f5029000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50240af1a0 0x7f502415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f5029000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50240af1a0 0x7f502415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57806/0 (socket says 172.21.15.32:57806) 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.079+0000 7f5029000640 1 -- 172.21.15.32:0/3351166380 learned_addr learned my addr 172.21.15.32:0/3351166380 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029000640 1 -- 172.21.15.32:0/3351166380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240cdc20 msgr2=0x7f50240b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029a00640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240cdc20 0x7f50240b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5023e00640 1 --2- 172.21.15.32:0/3351166380 >> v2:172.21.15.39:3300/0 conn(0x7f50240af580 0x7f50240b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029000640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240cdc20 0x7f50240b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029000640 1 -- 172.21.15.32:0/3351166380 >> v2:172.21.15.39:3300/0 conn(0x7f50240af580 msgr2=0x7f50240b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029000640 1 --2- 172.21.15.32:0/3351166380 >> v2:172.21.15.39:3300/0 conn(0x7f50240af580 0x7f50240b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029000640 1 -- 172.21.15.32:0/3351166380 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502415d4e0 con 0x7f50240af1a0 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.080+0000 7f5029a00640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240cdc20 0x7f50240b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.081+0000 7f5029000640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240af1a0 0x7f502415c600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50240b54c0 tx=0x7f501802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680027e124755cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.081+0000 7f5023400640 1 -- 172.21.15.32:0/3351166380 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501802e070 con 0x7f50240af1a0 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.081+0000 7f5023400640 1 -- 172.21.15.32:0/3351166380 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5018037070 con 0x7f50240af1a0 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.081+0000 7f5023400640 1 -- 172.21.15.32:0/3351166380 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018031a00 con 0x7f50240af1a0 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 -- 172.21.15.32:0/3351166380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240af1a0 msgr2=0x7f502415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240af1a0 0x7f502415c600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50240b54c0 tx=0x7f501802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 -- 172.21.15.32:0/3351166380 shutdown_connections 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240cdc20 0x7f50240b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 --2- 172.21.15.32:0/3351166380 >> v2:172.21.15.39:3300/0 conn(0x7f50240af580 0x7f50240b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 --2- 172.21.15.32:0/3351166380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240af1a0 0x7f502415c600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 -- 172.21.15.32:0/3351166380 >> 172.21.15.32:0/3351166380 conn(0x7f502406afb0 msgr2=0x7f502406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 -- 172.21.15.32:0/3351166380 shutdown_connections 2024-03-06T07:29:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.082+0000 7f502ba00640 1 -- 172.21.15.32:0/3351166380 wait complete. 2024-03-06T07:29:05.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.084+0000 7f502ba00640 1 Processor -- start 2024-03-06T07:29:05.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.084+0000 7f502ba00640 1 -- start start 2024-03-06T07:29:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f502ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50240af1a0 0x7f50240ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f502ba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240af580 0x7f50240adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f502ba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 0x7f502409cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f502ba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50240b3490 con 0x7f50240cdc20 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f502ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50240b3310 con 0x7f50240af1a0 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f502ba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50240b3610 con 0x7f50240af580 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 0x7f502409cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 0x7f502409cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57808/0 (socket says 172.21.15.32:57808) 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 -- 172.21.15.32:0/4294593430 learned_addr learned my addr 172.21.15.32:0/4294593430 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029000640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50240af1a0 0x7f50240ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5023e00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240af580 0x7f50240adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 -- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240af580 msgr2=0x7f50240adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240af580 0x7f50240adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 -- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50240af1a0 msgr2=0x7f50240ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50240af1a0 0x7f50240ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.085+0000 7f5029a00640 1 -- 172.21.15.32:0/4294593430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50241f3e60 con 0x7f50240cdc20 2024-03-06T07:29:05.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.086+0000 7f5029a00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 0x7f502409cf20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f501c00cd50 tx=0x7f501c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.086+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c00f680 con 0x7f50240cdc20 2024-03-06T07:29:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.086+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c00f840 con 0x7f50240cdc20 2024-03-06T07:29:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.086+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c00fc30 con 0x7f50240cdc20 2024-03-06T07:29:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.086+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50241f4140 con 0x7f50240cdc20 2024-03-06T07:29:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.086+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50241f4700 con 0x7f50240cdc20 2024-03-06T07:29:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.087+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5024069510 con 0x7f50240cdc20 2024-03-06T07:29:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.088+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f501c00e3e0 con 0x7f50240cdc20 2024-03-06T07:29:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.089+0000 7f5020c00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.089+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f501c08b000 con 0x7f50240cdc20 2024-03-06T07:29:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.089+0000 7f5029000640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.090+0000 7f5029000640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f500406b4d0 0x7f500406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5018036000 tx=0x7f5018008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.094+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f501c014030 con 0x7f50240cdc20 2024-03-06T07:29:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.294+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50240ad070 con 0x7f50240cdc20 2024-03-06T07:29:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.298+0000 7f5020c00640 1 -- 172.21.15.32:0/4294593430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f501c059e80 con 0x7f50240cdc20 2024-03-06T07:29:05.300 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:05.301 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:05.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.300+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f500406b4d0 msgr2=0x7f500406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.300+0000 7f502ba00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f500406b4d0 0x7f500406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5018036000 tx=0x7f5018008030 comp rx=0 tx=0).stop 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.300+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 msgr2=0x7f502409cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.300+0000 7f502ba00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 0x7f502409cf20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f501c00cd50 tx=0x7f501c00a650 comp rx=0 tx=0).stop 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 shutdown_connections 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50240cdc20 0x7f502409cf20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50240af580 0x7f50240adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 --2- 172.21.15.32:0/4294593430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50240af1a0 0x7f50240ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 >> 172.21.15.32:0/4294593430 conn(0x7f502406afb0 msgr2=0x7f50240cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 shutdown_connections 2024-03-06T07:29:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.301+0000 7f502ba00640 1 -- 172.21.15.32:0/4294593430 wait complete. 2024-03-06T07:29:05.323 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:29:05.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.538+0000 7f8715c00640 1 Processor -- start 2024-03-06T07:29:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.539+0000 7f8715c00640 1 -- start start 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.540+0000 7f8715c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f8710070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.540+0000 7f8715c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f871006e240 0x7f87100711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.540+0000 7f8715c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87100ce9a0 con 0x7f871006cfe0 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.540+0000 7f8715c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87100730d0 con 0x7f871006e240 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.540+0000 7f8715c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87100cec80 con 0x7f8710161670 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f871006e240 0x7f87100711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f871006e240 0x7f87100711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57820/0 (socket says 172.21.15.32:57820) 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 -- 172.21.15.32:0/938743002 learned_addr learned my addr 172.21.15.32:0/938743002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870ea00640 1 --2- 172.21.15.32:0/938743002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f8710070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 -- 172.21.15.32:0/938743002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 msgr2=0x7f8710070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 --2- 172.21.15.32:0/938743002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f8710070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 -- 172.21.15.32:0/938743002 >> v1:172.21.15.39:6789/0 conn(0x7f871006cfe0 legacy=0x7f871006d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.541+0000 7f870e000640 1 -- 172.21.15.32:0/938743002 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87100cef60 con 0x7f871006e240 2024-03-06T07:29:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.542+0000 7f870e000640 1 --2- 172.21.15.32:0/938743002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006e240 0x7f87100711b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009870 tx=0x7f86fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bad9dbdb7e9ec18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f870d600640 1 -- 172.21.15.32:0/938743002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc035020 con 0x7f871006e240 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f870d600640 1 -- 172.21.15.32:0/938743002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc031640 con 0x7f871006e240 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f870d600640 1 -- 172.21.15.32:0/938743002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc02d040 con 0x7f871006e240 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f8715c00640 1 -- 172.21.15.32:0/938743002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006e240 msgr2=0x7f87100711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f8715c00640 1 --2- 172.21.15.32:0/938743002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006e240 0x7f87100711b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009870 tx=0x7f86fc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f8715c00640 1 -- 172.21.15.32:0/938743002 shutdown_connections 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.543+0000 7f8715c00640 1 --2- 172.21.15.32:0/938743002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006e240 0x7f87100711b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.544+0000 7f8715c00640 1 --2- 172.21.15.32:0/938743002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f8710070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.544+0000 7f8715c00640 1 -- 172.21.15.32:0/938743002 >> 172.21.15.32:0/938743002 conn(0x7f871006afb0 msgr2=0x7f871006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.544+0000 7f8715c00640 1 -- 172.21.15.32:0/938743002 shutdown_connections 2024-03-06T07:29:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.544+0000 7f8715c00640 1 -- 172.21.15.32:0/938743002 wait complete. 2024-03-06T07:29:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.546+0000 7f8715c00640 1 Processor -- start 2024-03-06T07:29:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.546+0000 7f8715c00640 1 -- start start 2024-03-06T07:29:05.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f8715c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 0x7f8710148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f8715c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871006e240 0x7f8710148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f8715c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f87100b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f8715c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87100ce600 con 0x7f871006cfe0 2024-03-06T07:29:05.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f8715c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87100cf520 con 0x7f871006e240 2024-03-06T07:29:05.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f8715c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87100ce780 con 0x7f8710161670 2024-03-06T07:29:05.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 0x7f8710148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 0x7f8710148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57830/0 (socket says 172.21.15.32:57830) 2024-03-06T07:29:05.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 -- 172.21.15.32:0/2470600242 learned_addr learned my addr 172.21.15.32:0/2470600242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 -- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 msgr2=0x7f87100b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f87100b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870e000640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871006e240 0x7f8710148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 -- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871006e240 msgr2=0x7f8710148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871006e240 0x7f8710148000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.547+0000 7f870ea00640 1 -- 172.21.15.32:0/2470600242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710148e90 con 0x7f871006cfe0 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.548+0000 7f870ea00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 0x7f8710148950 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f870000d5f0 tx=0x7f870000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.548+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000125f0 con 0x7f871006cfe0 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.548+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700019070 con 0x7f871006cfe0 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.548+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000129d0 con 0x7f871006cfe0 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.548+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87100cba90 con 0x7f871006cfe0 2024-03-06T07:29:05.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.548+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871009c1f0 con 0x7f871006cfe0 2024-03-06T07:29:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.549+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8710069510 con 0x7f871006cfe0 2024-03-06T07:29:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.550+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8700012b70 con 0x7f871006cfe0 2024-03-06T07:29:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.551+0000 7f8706a00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f86f006b500 0x7f86f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.551+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f870008a640 con 0x7f871006cfe0 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.551+0000 7f870e000640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f86f006b500 0x7f86f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.552+0000 7f870e000640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f86f006b500 0x7f86f006d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f86fc036000 tx=0x7f86fc008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.556+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8700059420 con 0x7f871006cfe0 2024-03-06T07:29:05.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.753+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f871004f3a0 con 0x7f86f006b500 2024-03-06T07:29:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.775+0000 7f8706a00640 1 -- 172.21.15.32:0/2470600242 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251949 (secure 0 0 0) 0x7f871004f3a0 con 0x7f86f006b500 2024-03-06T07:29:05.782 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f86f006b500 msgr2=0x7f86f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f86f006b500 0x7f86f006d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f86fc036000 tx=0x7f86fc008030 comp rx=0 tx=0).stop 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 msgr2=0x7f8710148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 0x7f8710148950 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f870000d5f0 tx=0x7f870000dab0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 shutdown_connections 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f86f006b500 0x7f86f006d9c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8710161670 0x7f87100b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871006e240 0x7f8710148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 --2- 172.21.15.32:0/2470600242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f871006cfe0 0x7f8710148950 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 >> 172.21.15.32:0/2470600242 conn(0x7f871006afb0 msgr2=0x7f87100b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 shutdown_connections 2024-03-06T07:29:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:05.782+0000 7f8715c00640 1 -- 172.21.15.32:0/2470600242 wait complete. 2024-03-06T07:29:05.804 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-06T07:29:04.186125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252692,"kb_used_data":2884,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722028,"statfs":{"total":773094113280,"available":772835356672,"internally_reserved":0,"allocated":2953216,"data_stored":1771496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.658107"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306066+0000","last_change":"2024-03-06T07:29:01.317028+0000","last_active":"2024-03-06T07:29:02.306066+0000","last_peered":"2024-03-06T07:29:02.306066+0000","last_clean":"2024-03-06T07:29:02.306066+0000","last_became_active":"2024-03-06T07:29:01.316273+0000","last_became_peered":"2024-03-06T07:29:01.316273+0000","last_unstale":"2024-03-06T07:29:02.306066+0000","last_undegraded":"2024-03-06T07:29:02.306066+0000","last_fullsized":"2024-03-06T07:29:02.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338412+0000","last_change":"2024-03-06T07:29:01.311231+0000","last_active":"2024-03-06T07:29:02.338412+0000","last_peered":"2024-03-06T07:29:02.338412+0000","last_clean":"2024-03-06T07:29:02.338412+0000","last_became_active":"2024-03-06T07:29:01.310868+0000","last_became_peered":"2024-03-06T07:29:01.310868+0000","last_unstale":"2024-03-06T07:29:02.338412+0000","last_undegraded":"2024-03-06T07:29:02.338412+0000","last_fullsized":"2024-03-06T07:29:02.338412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856022+0000","last_change":"2024-03-06T07:29:01.307601+0000","last_active":"2024-03-06T07:29:02.856022+0000","last_peered":"2024-03-06T07:29:02.856022+0000","last_clean":"2024-03-06T07:29:02.856022+0000","last_became_active":"2024-03-06T07:29:01.307310+0000","last_became_peered":"2024-03-06T07:29:01.307310+0000","last_unstale":"2024-03-06T07:29:02.856022+0000","last_undegraded":"2024-03-06T07:29:02.856022+0000","last_fullsized":"2024-03-06T07:29:02.856022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331881+0000","last_change":"2024-03-06T07:29:01.326299+0000","last_active":"2024-03-06T07:29:02.331881+0000","last_peered":"2024-03-06T07:29:02.331881+0000","last_clean":"2024-03-06T07:29:02.331881+0000","last_became_active":"2024-03-06T07:29:01.326026+0000","last_became_peered":"2024-03-06T07:29:01.326026+0000","last_unstale":"2024-03-06T07:29:02.331881+0000","last_undegraded":"2024-03-06T07:29:02.331881+0000","last_fullsized":"2024-03-06T07:29:02.331881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304952+0000","last_change":"2024-03-06T07:29:01.314492+0000","last_active":"2024-03-06T07:29:02.304952+0000","last_peered":"2024-03-06T07:29:02.304952+0000","last_clean":"2024-03-06T07:29:02.304952+0000","last_became_active":"2024-03-06T07:29:01.313926+0000","last_became_peered":"2024-03-06T07:29:01.313926+0000","last_unstale":"2024-03-06T07:29:02.304952+0000","last_undegraded":"2024-03-06T07:29:02.304952+0000","last_fullsized":"2024-03-06T07:29:02.304952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331429+0000","last_change":"2024-03-06T07:29:01.328648+0000","last_active":"2024-03-06T07:29:02.331429+0000","last_peered":"2024-03-06T07:29:02.331429+0000","last_clean":"2024-03-06T07:29:02.331429+0000","last_became_active":"2024-03-06T07:29:01.328347+0000","last_became_peered":"2024-03-06T07:29:01.328347+0000","last_unstale":"2024-03-06T07:29:02.331429+0000","last_undegraded":"2024-03-06T07:29:02.331429+0000","last_fullsized":"2024-03-06T07:29:02.331429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304834+0000","last_change":"2024-03-06T07:29:01.310565+0000","last_active":"2024-03-06T07:29:02.304834+0000","last_peered":"2024-03-06T07:29:02.304834+0000","last_clean":"2024-03-06T07:29:02.304834+0000","last_became_active":"2024-03-06T07:29:01.310220+0000","last_became_peered":"2024-03-06T07:29:01.310220+0000","last_unstale":"2024-03-06T07:29:02.304834+0000","last_undegraded":"2024-03-06T07:29:02.304834+0000","last_fullsized":"2024-03-06T07:29:02.304834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.325938+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.325579+0000","last_became_peered":"2024-03-06T07:29:01.325579+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306146+0000","last_change":"2024-03-06T07:29:01.341575+0000","last_active":"2024-03-06T07:29:02.306146+0000","last_peered":"2024-03-06T07:29:02.306146+0000","last_clean":"2024-03-06T07:29:02.306146+0000","last_became_active":"2024-03-06T07:29:01.341372+0000","last_became_peered":"2024-03-06T07:29:01.341372+0000","last_unstale":"2024-03-06T07:29:02.306146+0000","last_undegraded":"2024-03-06T07:29:02.306146+0000","last_fullsized":"2024-03-06T07:29:02.306146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344143+0000","last_change":"2024-03-06T07:29:01.337188+0000","last_active":"2024-03-06T07:29:02.344143+0000","last_peered":"2024-03-06T07:29:02.344143+0000","last_clean":"2024-03-06T07:29:02.344143+0000","last_became_active":"2024-03-06T07:29:01.336845+0000","last_became_peered":"2024-03-06T07:29:01.336845+0000","last_unstale":"2024-03-06T07:29:02.344143+0000","last_undegraded":"2024-03-06T07:29:02.344143+0000","last_fullsized":"2024-03-06T07:29:02.344143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343789+0000","last_change":"2024-03-06T07:29:01.340996+0000","last_active":"2024-03-06T07:29:02.343789+0000","last_peered":"2024-03-06T07:29:02.343789+0000","last_clean":"2024-03-06T07:29:02.343789+0000","last_became_active":"2024-03-06T07:29:01.336560+0000","last_became_peered":"2024-03-06T07:29:01.336560+0000","last_unstale":"2024-03-06T07:29:02.343789+0000","last_undegraded":"2024-03-06T07:29:02.343789+0000","last_fullsized":"2024-03-06T07:29:02.343789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310075+0000","last_change":"2024-03-06T07:29:01.335731+0000","last_active":"2024-03-06T07:29:02.310075+0000","last_peered":"2024-03-06T07:29:02.310075+0000","last_clean":"2024-03-06T07:29:02.310075+0000","last_became_active":"2024-03-06T07:29:01.331758+0000","last_became_peered":"2024-03-06T07:29:01.331758+0000","last_unstale":"2024-03-06T07:29:02.310075+0000","last_undegraded":"2024-03-06T07:29:02.310075+0000","last_fullsized":"2024-03-06T07:29:02.310075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:19.716910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070024+0000","last_change":"2024-03-06T07:29:01.317550+0000","last_active":"2024-03-06T07:29:03.070024+0000","last_peered":"2024-03-06T07:29:03.070024+0000","last_clean":"2024-03-06T07:29:03.070024+0000","last_became_active":"2024-03-06T07:29:01.317098+0000","last_became_peered":"2024-03-06T07:29:01.317098+0000","last_unstale":"2024-03-06T07:29:03.070024+0000","last_undegraded":"2024-03-06T07:29:03.070024+0000","last_fullsized":"2024-03-06T07:29:03.070024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855737+0000","last_change":"2024-03-06T07:29:01.306850+0000","last_active":"2024-03-06T07:29:02.855737+0000","last_peered":"2024-03-06T07:29:02.855737+0000","last_clean":"2024-03-06T07:29:02.855737+0000","last_became_active":"2024-03-06T07:29:01.306532+0000","last_became_peered":"2024-03-06T07:29:01.306532+0000","last_unstale":"2024-03-06T07:29:02.855737+0000","last_undegraded":"2024-03-06T07:29:02.855737+0000","last_fullsized":"2024-03-06T07:29:02.855737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:02.899318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.337987+0000","last_change":"2024-03-06T07:29:01.313116+0000","last_active":"2024-03-06T07:29:02.337987+0000","last_peered":"2024-03-06T07:29:02.337987+0000","last_clean":"2024-03-06T07:29:02.337987+0000","last_became_active":"2024-03-06T07:29:01.311783+0000","last_became_peered":"2024-03-06T07:29:01.311783+0000","last_unstale":"2024-03-06T07:29:02.337987+0000","last_undegraded":"2024-03-06T07:29:02.337987+0000","last_fullsized":"2024-03-06T07:29:02.337987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331678+0000","last_change":"2024-03-06T07:29:01.323368+0000","last_active":"2024-03-06T07:29:02.331678+0000","last_peered":"2024-03-06T07:29:02.331678+0000","last_clean":"2024-03-06T07:29:02.331678+0000","last_became_active":"2024-03-06T07:29:01.323066+0000","last_became_peered":"2024-03-06T07:29:01.323066+0000","last_unstale":"2024-03-06T07:29:02.331678+0000","last_undegraded":"2024-03-06T07:29:02.331678+0000","last_fullsized":"2024-03-06T07:29:02.331678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332654+0000","last_change":"2024-03-06T07:29:01.332332+0000","last_active":"2024-03-06T07:29:02.332654+0000","last_peered":"2024-03-06T07:29:02.332654+0000","last_clean":"2024-03-06T07:29:02.332654+0000","last_became_active":"2024-03-06T07:29:01.331999+0000","last_became_peered":"2024-03-06T07:29:01.331999+0000","last_unstale":"2024-03-06T07:29:02.332654+0000","last_undegraded":"2024-03-06T07:29:02.332654+0000","last_fullsized":"2024-03-06T07:29:02.332654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069247+0000","last_change":"2024-03-06T07:29:01.319453+0000","last_active":"2024-03-06T07:29:03.069247+0000","last_peered":"2024-03-06T07:29:03.069247+0000","last_clean":"2024-03-06T07:29:03.069247+0000","last_became_active":"2024-03-06T07:29:01.319182+0000","last_became_peered":"2024-03-06T07:29:01.319182+0000","last_unstale":"2024-03-06T07:29:03.069247+0000","last_undegraded":"2024-03-06T07:29:03.069247+0000","last_fullsized":"2024-03-06T07:29:03.069247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:42.291305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305802+0000","last_change":"2024-03-06T07:29:01.304420+0000","last_active":"2024-03-06T07:29:02.305802+0000","last_peered":"2024-03-06T07:29:02.305802+0000","last_clean":"2024-03-06T07:29:02.305802+0000","last_became_active":"2024-03-06T07:29:01.303897+0000","last_became_peered":"2024-03-06T07:29:01.303897+0000","last_unstale":"2024-03-06T07:29:02.305802+0000","last_undegraded":"2024-03-06T07:29:02.305802+0000","last_fullsized":"2024-03-06T07:29:02.305802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343811+0000","last_change":"2024-03-06T07:29:01.340804+0000","last_active":"2024-03-06T07:29:02.343811+0000","last_peered":"2024-03-06T07:29:02.343811+0000","last_clean":"2024-03-06T07:29:02.343811+0000","last_became_active":"2024-03-06T07:29:01.338819+0000","last_became_peered":"2024-03-06T07:29:01.338819+0000","last_unstale":"2024-03-06T07:29:02.343811+0000","last_undegraded":"2024-03-06T07:29:02.343811+0000","last_fullsized":"2024-03-06T07:29:02.343811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856684+0000","last_change":"2024-03-06T07:29:01.329232+0000","last_active":"2024-03-06T07:29:02.856684+0000","last_peered":"2024-03-06T07:29:02.856684+0000","last_clean":"2024-03-06T07:29:02.856684+0000","last_became_active":"2024-03-06T07:29:01.328855+0000","last_became_peered":"2024-03-06T07:29:01.328855+0000","last_unstale":"2024-03-06T07:29:02.856684+0000","last_undegraded":"2024-03-06T07:29:02.856684+0000","last_fullsized":"2024-03-06T07:29:02.856684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331471+0000","last_change":"2024-03-06T07:29:01.334776+0000","last_active":"2024-03-06T07:29:02.331471+0000","last_peered":"2024-03-06T07:29:02.331471+0000","last_clean":"2024-03-06T07:29:02.331471+0000","last_became_active":"2024-03-06T07:29:01.332072+0000","last_became_peered":"2024-03-06T07:29:01.332072+0000","last_unstale":"2024-03-06T07:29:02.331471+0000","last_undegraded":"2024-03-06T07:29:02.331471+0000","last_fullsized":"2024-03-06T07:29:02.331471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.312963+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.312645+0000","last_became_peered":"2024-03-06T07:29:01.312645+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331763+0000","last_change":"2024-03-06T07:29:01.322856+0000","last_active":"2024-03-06T07:29:02.331763+0000","last_peered":"2024-03-06T07:29:02.331763+0000","last_clean":"2024-03-06T07:29:02.331763+0000","last_became_active":"2024-03-06T07:29:01.322485+0000","last_became_peered":"2024-03-06T07:29:01.322485+0000","last_unstale":"2024-03-06T07:29:02.331763+0000","last_undegraded":"2024-03-06T07:29:02.331763+0000","last_fullsized":"2024-03-06T07:29:02.331763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070502+0000","last_change":"2024-03-06T07:29:01.328113+0000","last_active":"2024-03-06T07:29:03.070502+0000","last_peered":"2024-03-06T07:29:03.070502+0000","last_clean":"2024-03-06T07:29:03.070502+0000","last_became_active":"2024-03-06T07:29:01.327700+0000","last_became_peered":"2024-03-06T07:29:01.327700+0000","last_unstale":"2024-03-06T07:29:03.070502+0000","last_undegraded":"2024-03-06T07:29:03.070502+0000","last_fullsized":"2024-03-06T07:29:03.070502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344258+0000","last_change":"2024-03-06T07:29:01.328971+0000","last_active":"2024-03-06T07:29:02.344258+0000","last_peered":"2024-03-06T07:29:02.344258+0000","last_clean":"2024-03-06T07:29:02.344258+0000","last_became_active":"2024-03-06T07:29:01.328471+0000","last_became_peered":"2024-03-06T07:29:01.328471+0000","last_unstale":"2024-03-06T07:29:02.344258+0000","last_undegraded":"2024-03-06T07:29:02.344258+0000","last_fullsized":"2024-03-06T07:29:02.344258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338529+0000","last_change":"2024-03-06T07:29:01.310033+0000","last_active":"2024-03-06T07:29:02.338529+0000","last_peered":"2024-03-06T07:29:02.338529+0000","last_clean":"2024-03-06T07:29:02.338529+0000","last_became_active":"2024-03-06T07:29:01.309715+0000","last_became_peered":"2024-03-06T07:29:01.309715+0000","last_unstale":"2024-03-06T07:29:02.338529+0000","last_undegraded":"2024-03-06T07:29:02.338529+0000","last_fullsized":"2024-03-06T07:29:02.338529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069900+0000","last_change":"2024-03-06T07:29:01.338107+0000","last_active":"2024-03-06T07:29:03.069900+0000","last_peered":"2024-03-06T07:29:03.069900+0000","last_clean":"2024-03-06T07:29:03.069900+0000","last_became_active":"2024-03-06T07:29:01.337793+0000","last_became_peered":"2024-03-06T07:29:01.337793+0000","last_unstale":"2024-03-06T07:29:03.069900+0000","last_undegraded":"2024-03-06T07:29:03.069900+0000","last_fullsized":"2024-03-06T07:29:03.069900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856096+0000","last_change":"2024-03-06T07:29:01.309168+0000","last_active":"2024-03-06T07:29:02.856096+0000","last_peered":"2024-03-06T07:29:02.856096+0000","last_clean":"2024-03-06T07:29:02.856096+0000","last_became_active":"2024-03-06T07:29:01.308853+0000","last_became_peered":"2024-03-06T07:29:01.308853+0000","last_unstale":"2024-03-06T07:29:02.856096+0000","last_undegraded":"2024-03-06T07:29:02.856096+0000","last_fullsized":"2024-03-06T07:29:02.856096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332020+0000","last_change":"2024-03-06T07:29:01.339291+0000","last_active":"2024-03-06T07:29:02.332020+0000","last_peered":"2024-03-06T07:29:02.332020+0000","last_clean":"2024-03-06T07:29:02.332020+0000","last_became_active":"2024-03-06T07:29:01.338961+0000","last_became_peered":"2024-03-06T07:29:01.338961+0000","last_unstale":"2024-03-06T07:29:02.332020+0000","last_undegraded":"2024-03-06T07:29:02.332020+0000","last_fullsized":"2024-03-06T07:29:02.332020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068361+0000","last_change":"2024-03-06T07:29:01.339412+0000","last_active":"2024-03-06T07:29:03.068361+0000","last_peered":"2024-03-06T07:29:03.068361+0000","last_clean":"2024-03-06T07:29:03.068361+0000","last_became_active":"2024-03-06T07:29:01.338930+0000","last_became_peered":"2024-03-06T07:29:01.338930+0000","last_unstale":"2024-03-06T07:29:03.068361+0000","last_undegraded":"2024-03-06T07:29:03.068361+0000","last_fullsized":"2024-03-06T07:29:03.068361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343463+0000","last_change":"2024-03-06T07:29:01.316617+0000","last_active":"2024-03-06T07:29:02.343463+0000","last_peered":"2024-03-06T07:29:02.343463+0000","last_clean":"2024-03-06T07:29:02.343463+0000","last_became_active":"2024-03-06T07:29:01.315094+0000","last_became_peered":"2024-03-06T07:29:01.315094+0000","last_unstale":"2024-03-06T07:29:02.343463+0000","last_undegraded":"2024-03-06T07:29:02.343463+0000","last_fullsized":"2024-03-06T07:29:02.343463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309312+0000","last_change":"2024-03-06T07:29:01.314610+0000","last_active":"2024-03-06T07:29:02.309312+0000","last_peered":"2024-03-06T07:29:02.309312+0000","last_clean":"2024-03-06T07:29:02.309312+0000","last_became_active":"2024-03-06T07:29:01.314249+0000","last_became_peered":"2024-03-06T07:29:01.314249+0000","last_unstale":"2024-03-06T07:29:02.309312+0000","last_undegraded":"2024-03-06T07:29:02.309312+0000","last_fullsized":"2024-03-06T07:29:02.309312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:32.840745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070367+0000","last_change":"2024-03-06T07:29:00.318669+0000","last_active":"2024-03-06T07:29:03.070367+0000","last_peered":"2024-03-06T07:29:03.070367+0000","last_clean":"2024-03-06T07:29:03.070367+0000","last_became_active":"2024-03-06T07:29:00.318368+0000","last_became_peered":"2024-03-06T07:29:00.318368+0000","last_unstale":"2024-03-06T07:29:03.070367+0000","last_undegraded":"2024-03-06T07:29:03.070367+0000","last_fullsized":"2024-03-06T07:29:03.070367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856486+0000","last_change":"2024-03-06T07:29:01.330131+0000","last_active":"2024-03-06T07:29:02.856486+0000","last_peered":"2024-03-06T07:29:02.856486+0000","last_clean":"2024-03-06T07:29:02.856486+0000","last_became_active":"2024-03-06T07:29:01.329564+0000","last_became_peered":"2024-03-06T07:29:01.329564+0000","last_unstale":"2024-03-06T07:29:02.856486+0000","last_undegraded":"2024-03-06T07:29:02.856486+0000","last_fullsized":"2024-03-06T07:29:02.856486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343246+0000","last_change":"2024-03-06T07:29:00.325104+0000","last_active":"2024-03-06T07:29:02.343246+0000","last_peered":"2024-03-06T07:29:02.343246+0000","last_clean":"2024-03-06T07:29:02.343246+0000","last_became_active":"2024-03-06T07:29:00.324737+0000","last_became_peered":"2024-03-06T07:29:00.324737+0000","last_unstale":"2024-03-06T07:29:02.343246+0000","last_undegraded":"2024-03-06T07:29:02.343246+0000","last_fullsized":"2024-03-06T07:29:02.343246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068981+0000","last_change":"2024-03-06T07:29:00.322124+0000","last_active":"2024-03-06T07:29:03.068981+0000","last_peered":"2024-03-06T07:29:03.068981+0000","last_clean":"2024-03-06T07:29:03.068981+0000","last_became_active":"2024-03-06T07:29:00.321658+0000","last_became_peered":"2024-03-06T07:29:00.321658+0000","last_unstale":"2024-03-06T07:29:03.068981+0000","last_undegraded":"2024-03-06T07:29:03.068981+0000","last_fullsized":"2024-03-06T07:29:03.068981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069549+0000","last_change":"2024-03-06T07:29:01.306201+0000","last_active":"2024-03-06T07:29:03.069549+0000","last_peered":"2024-03-06T07:29:03.069549+0000","last_clean":"2024-03-06T07:29:03.069549+0000","last_became_active":"2024-03-06T07:29:01.305652+0000","last_became_peered":"2024-03-06T07:29:01.305652+0000","last_unstale":"2024-03-06T07:29:03.069549+0000","last_undegraded":"2024-03-06T07:29:03.069549+0000","last_fullsized":"2024-03-06T07:29:03.069549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331796+0000","last_change":"2024-03-06T07:29:00.330308+0000","last_active":"2024-03-06T07:29:02.331796+0000","last_peered":"2024-03-06T07:29:02.331796+0000","last_clean":"2024-03-06T07:29:02.331796+0000","last_became_active":"2024-03-06T07:29:00.329727+0000","last_became_peered":"2024-03-06T07:29:00.329727+0000","last_unstale":"2024-03-06T07:29:02.331796+0000","last_undegraded":"2024-03-06T07:29:02.331796+0000","last_fullsized":"2024-03-06T07:29:02.331796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338071+0000","last_change":"2024-03-06T07:29:01.308157+0000","last_active":"2024-03-06T07:29:02.338071+0000","last_peered":"2024-03-06T07:29:02.338071+0000","last_clean":"2024-03-06T07:29:02.338071+0000","last_became_active":"2024-03-06T07:29:01.307819+0000","last_became_peered":"2024-03-06T07:29:01.307819+0000","last_unstale":"2024-03-06T07:29:02.338071+0000","last_undegraded":"2024-03-06T07:29:02.338071+0000","last_fullsized":"2024-03-06T07:29:02.338071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.334131+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.333816+0000","last_became_peered":"2024-03-06T07:29:00.333816+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332394+0000","last_change":"2024-03-06T07:29:01.332596+0000","last_active":"2024-03-06T07:29:02.332394+0000","last_peered":"2024-03-06T07:29:02.332394+0000","last_clean":"2024-03-06T07:29:02.332394+0000","last_became_active":"2024-03-06T07:29:01.332140+0000","last_became_peered":"2024-03-06T07:29:01.332140+0000","last_unstale":"2024-03-06T07:29:02.332394+0000","last_undegraded":"2024-03-06T07:29:02.332394+0000","last_fullsized":"2024-03-06T07:29:02.332394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332369+0000","last_change":"2024-03-06T07:29:00.332773+0000","last_active":"2024-03-06T07:29:02.332369+0000","last_peered":"2024-03-06T07:29:02.332369+0000","last_clean":"2024-03-06T07:29:02.332369+0000","last_became_active":"2024-03-06T07:29:00.332477+0000","last_became_peered":"2024-03-06T07:29:00.332477+0000","last_unstale":"2024-03-06T07:29:02.332369+0000","last_undegraded":"2024-03-06T07:29:02.332369+0000","last_fullsized":"2024-03-06T07:29:02.332369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856820+0000","last_change":"2024-03-06T07:29:01.325722+0000","last_active":"2024-03-06T07:29:02.856820+0000","last_peered":"2024-03-06T07:29:02.856820+0000","last_clean":"2024-03-06T07:29:02.856820+0000","last_became_active":"2024-03-06T07:29:01.325401+0000","last_became_peered":"2024-03-06T07:29:01.325401+0000","last_unstale":"2024-03-06T07:29:02.856820+0000","last_undegraded":"2024-03-06T07:29:02.856820+0000","last_fullsized":"2024-03-06T07:29:02.856820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338454+0000","last_change":"2024-03-06T07:29:00.331332+0000","last_active":"2024-03-06T07:29:02.338454+0000","last_peered":"2024-03-06T07:29:02.338454+0000","last_clean":"2024-03-06T07:29:02.338454+0000","last_became_active":"2024-03-06T07:29:00.331008+0000","last_became_peered":"2024-03-06T07:29:00.331008+0000","last_unstale":"2024-03-06T07:29:02.338454+0000","last_undegraded":"2024-03-06T07:29:02.338454+0000","last_fullsized":"2024-03-06T07:29:02.338454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310152+0000","last_change":"2024-03-06T07:29:01.335452+0000","last_active":"2024-03-06T07:29:02.310152+0000","last_peered":"2024-03-06T07:29:02.310152+0000","last_clean":"2024-03-06T07:29:02.310152+0000","last_became_active":"2024-03-06T07:29:01.331241+0000","last_became_peered":"2024-03-06T07:29:01.331241+0000","last_unstale":"2024-03-06T07:29:02.310152+0000","last_undegraded":"2024-03-06T07:29:02.310152+0000","last_fullsized":"2024-03-06T07:29:02.310152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343888+0000","last_change":"2024-03-06T07:29:00.322219+0000","last_active":"2024-03-06T07:29:02.343888+0000","last_peered":"2024-03-06T07:29:02.343888+0000","last_clean":"2024-03-06T07:29:02.343888+0000","last_became_active":"2024-03-06T07:29:00.321921+0000","last_became_peered":"2024-03-06T07:29:00.321921+0000","last_unstale":"2024-03-06T07:29:02.343888+0000","last_undegraded":"2024-03-06T07:29:02.343888+0000","last_fullsized":"2024-03-06T07:29:02.343888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304565+0000","last_change":"2024-03-06T07:29:01.309672+0000","last_active":"2024-03-06T07:29:02.304565+0000","last_peered":"2024-03-06T07:29:02.304565+0000","last_clean":"2024-03-06T07:29:02.304565+0000","last_became_active":"2024-03-06T07:29:01.309369+0000","last_became_peered":"2024-03-06T07:29:01.309369+0000","last_unstale":"2024-03-06T07:29:02.304565+0000","last_undegraded":"2024-03-06T07:29:02.304565+0000","last_fullsized":"2024-03-06T07:29:02.304565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856650+0000","last_change":"2024-03-06T07:29:00.324810+0000","last_active":"2024-03-06T07:29:02.856650+0000","last_peered":"2024-03-06T07:29:02.856650+0000","last_clean":"2024-03-06T07:29:02.856650+0000","last_became_active":"2024-03-06T07:29:00.324524+0000","last_became_peered":"2024-03-06T07:29:00.324524+0000","last_unstale":"2024-03-06T07:29:02.856650+0000","last_undegraded":"2024-03-06T07:29:02.856650+0000","last_fullsized":"2024-03-06T07:29:02.856650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:20.411708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855986+0000","last_change":"2024-03-06T07:29:01.329235+0000","last_active":"2024-03-06T07:29:02.855986+0000","last_peered":"2024-03-06T07:29:02.855986+0000","last_clean":"2024-03-06T07:29:02.855986+0000","last_became_active":"2024-03-06T07:29:01.328953+0000","last_became_peered":"2024-03-06T07:29:01.328953+0000","last_unstale":"2024-03-06T07:29:02.855986+0000","last_undegraded":"2024-03-06T07:29:02.855986+0000","last_fullsized":"2024-03-06T07:29:02.855986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331601+0000","last_change":"2024-03-06T07:29:00.325328+0000","last_active":"2024-03-06T07:29:02.331601+0000","last_peered":"2024-03-06T07:29:02.331601+0000","last_clean":"2024-03-06T07:29:02.331601+0000","last_became_active":"2024-03-06T07:29:00.323961+0000","last_became_peered":"2024-03-06T07:29:00.323961+0000","last_unstale":"2024-03-06T07:29:02.331601+0000","last_undegraded":"2024-03-06T07:29:02.331601+0000","last_fullsized":"2024-03-06T07:29:02.331601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305341+0000","last_change":"2024-03-06T07:29:01.314222+0000","last_active":"2024-03-06T07:29:02.305341+0000","last_peered":"2024-03-06T07:29:02.305341+0000","last_clean":"2024-03-06T07:29:02.305341+0000","last_became_active":"2024-03-06T07:29:01.313409+0000","last_became_peered":"2024-03-06T07:29:01.313409+0000","last_unstale":"2024-03-06T07:29:02.305341+0000","last_undegraded":"2024-03-06T07:29:02.305341+0000","last_fullsized":"2024-03-06T07:29:02.305341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343329+0000","last_change":"2024-03-06T07:29:00.326529+0000","last_active":"2024-03-06T07:29:02.343329+0000","last_peered":"2024-03-06T07:29:02.343329+0000","last_clean":"2024-03-06T07:29:02.343329+0000","last_became_active":"2024-03-06T07:29:00.326202+0000","last_became_peered":"2024-03-06T07:29:00.326202+0000","last_unstale":"2024-03-06T07:29:02.343329+0000","last_undegraded":"2024-03-06T07:29:02.343329+0000","last_fullsized":"2024-03-06T07:29:02.343329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343344+0000","last_change":"2024-03-06T07:29:01.316333+0000","last_active":"2024-03-06T07:29:02.343344+0000","last_peered":"2024-03-06T07:29:02.343344+0000","last_clean":"2024-03-06T07:29:02.343344+0000","last_became_active":"2024-03-06T07:29:01.315969+0000","last_became_peered":"2024-03-06T07:29:01.315969+0000","last_unstale":"2024-03-06T07:29:02.343344+0000","last_undegraded":"2024-03-06T07:29:02.343344+0000","last_fullsized":"2024-03-06T07:29:02.343344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069217+0000","last_change":"2024-03-06T07:29:00.306241+0000","last_active":"2024-03-06T07:29:03.069217+0000","last_peered":"2024-03-06T07:29:03.069217+0000","last_clean":"2024-03-06T07:29:03.069217+0000","last_became_active":"2024-03-06T07:29:00.305883+0000","last_became_peered":"2024-03-06T07:29:00.305883+0000","last_unstale":"2024-03-06T07:29:03.069217+0000","last_undegraded":"2024-03-06T07:29:03.069217+0000","last_fullsized":"2024-03-06T07:29:03.069217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306244+0000","last_change":"2024-03-06T07:29:01.326203+0000","last_active":"2024-03-06T07:29:02.306244+0000","last_peered":"2024-03-06T07:29:02.306244+0000","last_clean":"2024-03-06T07:29:02.306244+0000","last_became_active":"2024-03-06T07:29:01.325861+0000","last_became_peered":"2024-03-06T07:29:01.325861+0000","last_unstale":"2024-03-06T07:29:02.306244+0000","last_undegraded":"2024-03-06T07:29:02.306244+0000","last_fullsized":"2024-03-06T07:29:02.306244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310043+0000","last_change":"2024-03-06T07:29:00.320015+0000","last_active":"2024-03-06T07:29:02.310043+0000","last_peered":"2024-03-06T07:29:02.310043+0000","last_clean":"2024-03-06T07:29:02.310043+0000","last_became_active":"2024-03-06T07:29:00.319492+0000","last_became_peered":"2024-03-06T07:29:00.319492+0000","last_unstale":"2024-03-06T07:29:02.310043+0000","last_undegraded":"2024-03-06T07:29:02.310043+0000","last_fullsized":"2024-03-06T07:29:02.310043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:49.576279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304906+0000","last_change":"2024-03-06T07:29:01.325857+0000","last_active":"2024-03-06T07:29:02.304906+0000","last_peered":"2024-03-06T07:29:02.304906+0000","last_clean":"2024-03-06T07:29:02.304906+0000","last_became_active":"2024-03-06T07:29:01.325522+0000","last_became_peered":"2024-03-06T07:29:01.325522+0000","last_unstale":"2024-03-06T07:29:02.304906+0000","last_undegraded":"2024-03-06T07:29:02.304906+0000","last_fullsized":"2024-03-06T07:29:02.304906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069075+0000","last_change":"2024-03-06T07:29:00.311734+0000","last_active":"2024-03-06T07:29:03.069075+0000","last_peered":"2024-03-06T07:29:03.069075+0000","last_clean":"2024-03-06T07:29:03.069075+0000","last_became_active":"2024-03-06T07:29:00.309932+0000","last_became_peered":"2024-03-06T07:29:00.309932+0000","last_unstale":"2024-03-06T07:29:03.069075+0000","last_undegraded":"2024-03-06T07:29:03.069075+0000","last_fullsized":"2024-03-06T07:29:03.069075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306517+0000","last_change":"2024-03-06T07:29:01.331125+0000","last_active":"2024-03-06T07:29:02.306517+0000","last_peered":"2024-03-06T07:29:02.306517+0000","last_clean":"2024-03-06T07:29:02.306517+0000","last_became_active":"2024-03-06T07:29:01.330860+0000","last_became_peered":"2024-03-06T07:29:01.330860+0000","last_unstale":"2024-03-06T07:29:02.306517+0000","last_undegraded":"2024-03-06T07:29:02.306517+0000","last_fullsized":"2024-03-06T07:29:02.306517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:44.856694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338728+0000","last_change":"2024-03-06T07:29:00.325056+0000","last_active":"2024-03-06T07:29:02.338728+0000","last_peered":"2024-03-06T07:29:02.338728+0000","last_clean":"2024-03-06T07:29:02.338728+0000","last_became_active":"2024-03-06T07:29:00.324757+0000","last_became_peered":"2024-03-06T07:29:00.324757+0000","last_unstale":"2024-03-06T07:29:02.338728+0000","last_undegraded":"2024-03-06T07:29:02.338728+0000","last_fullsized":"2024-03-06T07:29:02.338728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332058+0000","last_change":"2024-03-06T07:29:01.326654+0000","last_active":"2024-03-06T07:29:02.332058+0000","last_peered":"2024-03-06T07:29:02.332058+0000","last_clean":"2024-03-06T07:29:02.332058+0000","last_became_active":"2024-03-06T07:29:01.325612+0000","last_became_peered":"2024-03-06T07:29:01.325612+0000","last_unstale":"2024-03-06T07:29:02.332058+0000","last_undegraded":"2024-03-06T07:29:02.332058+0000","last_fullsized":"2024-03-06T07:29:02.332058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068894+0000","last_change":"2024-03-06T07:29:00.332332+0000","last_active":"2024-03-06T07:29:03.068894+0000","last_peered":"2024-03-06T07:29:03.068894+0000","last_clean":"2024-03-06T07:29:03.068894+0000","last_became_active":"2024-03-06T07:29:00.331881+0000","last_became_peered":"2024-03-06T07:29:00.331881+0000","last_unstale":"2024-03-06T07:29:03.068894+0000","last_undegraded":"2024-03-06T07:29:03.068894+0000","last_fullsized":"2024-03-06T07:29:03.068894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304989+0000","last_change":"2024-03-06T07:29:01.336946+0000","last_active":"2024-03-06T07:29:02.304989+0000","last_peered":"2024-03-06T07:29:02.304989+0000","last_clean":"2024-03-06T07:29:02.304989+0000","last_became_active":"2024-03-06T07:29:01.336594+0000","last_became_peered":"2024-03-06T07:29:01.336594+0000","last_unstale":"2024-03-06T07:29:02.304989+0000","last_undegraded":"2024-03-06T07:29:02.304989+0000","last_fullsized":"2024-03-06T07:29:02.304989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303958+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:29:02.303958+0000","last_peered":"2024-03-06T07:29:02.303958+0000","last_clean":"2024-03-06T07:29:02.303958+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:29:02.303958+0000","last_undegraded":"2024-03-06T07:29:02.303958+0000","last_fullsized":"2024-03-06T07:29:02.303958+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331455+0000","last_change":"2024-03-06T07:29:00.331164+0000","last_active":"2024-03-06T07:29:02.331455+0000","last_peered":"2024-03-06T07:29:02.331455+0000","last_clean":"2024-03-06T07:29:02.331455+0000","last_became_active":"2024-03-06T07:29:00.330531+0000","last_became_peered":"2024-03-06T07:29:00.330531+0000","last_unstale":"2024-03-06T07:29:02.331455+0000","last_undegraded":"2024-03-06T07:29:02.331455+0000","last_fullsized":"2024-03-06T07:29:02.331455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069681+0000","last_change":"2024-03-06T07:29:01.342482+0000","last_active":"2024-03-06T07:29:03.069681+0000","last_peered":"2024-03-06T07:29:03.069681+0000","last_clean":"2024-03-06T07:29:03.069681+0000","last_became_active":"2024-03-06T07:29:01.342176+0000","last_became_peered":"2024-03-06T07:29:01.342176+0000","last_unstale":"2024-03-06T07:29:03.069681+0000","last_undegraded":"2024-03-06T07:29:03.069681+0000","last_fullsized":"2024-03-06T07:29:03.069681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303790+0000","last_change":"2024-03-06T07:29:00.308988+0000","last_active":"2024-03-06T07:29:02.303790+0000","last_peered":"2024-03-06T07:29:02.303790+0000","last_clean":"2024-03-06T07:29:02.303790+0000","last_became_active":"2024-03-06T07:29:00.304843+0000","last_became_peered":"2024-03-06T07:29:00.304843+0000","last_unstale":"2024-03-06T07:29:02.303790+0000","last_undegraded":"2024-03-06T07:29:02.303790+0000","last_fullsized":"2024-03-06T07:29:02.303790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855676+0000","last_change":"2024-03-06T07:29:01.328580+0000","last_active":"2024-03-06T07:29:02.855676+0000","last_peered":"2024-03-06T07:29:02.855676+0000","last_clean":"2024-03-06T07:29:02.855676+0000","last_became_active":"2024-03-06T07:29:01.328319+0000","last_became_peered":"2024-03-06T07:29:01.328319+0000","last_unstale":"2024-03-06T07:29:02.855676+0000","last_undegraded":"2024-03-06T07:29:02.855676+0000","last_fullsized":"2024-03-06T07:29:02.855676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:33.356914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068423+0000","last_change":"2024-03-06T07:29:00.307455+0000","last_active":"2024-03-06T07:29:03.068423+0000","last_peered":"2024-03-06T07:29:03.068423+0000","last_clean":"2024-03-06T07:29:03.068423+0000","last_became_active":"2024-03-06T07:29:00.307117+0000","last_became_peered":"2024-03-06T07:29:00.307117+0000","last_unstale":"2024-03-06T07:29:03.068423+0000","last_undegraded":"2024-03-06T07:29:03.068423+0000","last_fullsized":"2024-03-06T07:29:03.068423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338099+0000","last_change":"2024-03-06T07:29:01.298549+0000","last_active":"2024-03-06T07:29:02.338099+0000","last_peered":"2024-03-06T07:29:02.338099+0000","last_clean":"2024-03-06T07:29:02.338099+0000","last_became_active":"2024-03-06T07:29:01.298043+0000","last_became_peered":"2024-03-06T07:29:01.298043+0000","last_unstale":"2024-03-06T07:29:02.338099+0000","last_undegraded":"2024-03-06T07:29:02.338099+0000","last_fullsized":"2024-03-06T07:29:02.338099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331760+0000","last_change":"2024-03-06T07:29:00.323986+0000","last_active":"2024-03-06T07:29:02.331760+0000","last_peered":"2024-03-06T07:29:02.331760+0000","last_clean":"2024-03-06T07:29:02.331760+0000","last_became_active":"2024-03-06T07:29:00.323442+0000","last_became_peered":"2024-03-06T07:29:00.323442+0000","last_unstale":"2024-03-06T07:29:02.331760+0000","last_undegraded":"2024-03-06T07:29:02.331760+0000","last_fullsized":"2024-03-06T07:29:02.331760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309938+0000","last_change":"2024-03-06T07:29:01.319824+0000","last_active":"2024-03-06T07:29:02.309938+0000","last_peered":"2024-03-06T07:29:02.309938+0000","last_clean":"2024-03-06T07:29:02.309938+0000","last_became_active":"2024-03-06T07:29:01.317792+0000","last_became_peered":"2024-03-06T07:29:01.317792+0000","last_unstale":"2024-03-06T07:29:02.309938+0000","last_undegraded":"2024-03-06T07:29:02.309938+0000","last_fullsized":"2024-03-06T07:29:02.309938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.562992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338550+0000","last_change":"2024-03-06T07:29:00.312181+0000","last_active":"2024-03-06T07:29:02.338550+0000","last_peered":"2024-03-06T07:29:02.338550+0000","last_clean":"2024-03-06T07:29:02.338550+0000","last_became_active":"2024-03-06T07:29:00.311915+0000","last_became_peered":"2024-03-06T07:29:00.311915+0000","last_unstale":"2024-03-06T07:29:02.338550+0000","last_undegraded":"2024-03-06T07:29:02.338550+0000","last_fullsized":"2024-03-06T07:29:02.338550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332017+0000","last_change":"2024-03-06T07:29:01.332290+0000","last_active":"2024-03-06T07:29:02.332017+0000","last_peered":"2024-03-06T07:29:02.332017+0000","last_clean":"2024-03-06T07:29:02.332017+0000","last_became_active":"2024-03-06T07:29:01.331904+0000","last_became_peered":"2024-03-06T07:29:01.331904+0000","last_unstale":"2024-03-06T07:29:02.332017+0000","last_undegraded":"2024-03-06T07:29:02.332017+0000","last_fullsized":"2024-03-06T07:29:02.332017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306067+0000","last_change":"2024-03-06T07:29:00.315219+0000","last_active":"2024-03-06T07:29:02.306067+0000","last_peered":"2024-03-06T07:29:02.306067+0000","last_clean":"2024-03-06T07:29:02.306067+0000","last_became_active":"2024-03-06T07:29:00.314808+0000","last_became_peered":"2024-03-06T07:29:00.314808+0000","last_unstale":"2024-03-06T07:29:02.306067+0000","last_undegraded":"2024-03-06T07:29:02.306067+0000","last_fullsized":"2024-03-06T07:29:02.306067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338430+0000","last_change":"2024-03-06T07:29:01.303022+0000","last_active":"2024-03-06T07:29:02.338430+0000","last_peered":"2024-03-06T07:29:02.338430+0000","last_clean":"2024-03-06T07:29:02.338430+0000","last_became_active":"2024-03-06T07:29:01.302528+0000","last_became_peered":"2024-03-06T07:29:01.302528+0000","last_unstale":"2024-03-06T07:29:02.338430+0000","last_undegraded":"2024-03-06T07:29:02.338430+0000","last_fullsized":"2024-03-06T07:29:02.338430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.341932+0000","last_change":"2024-03-06T07:29:00.323588+0000","last_active":"2024-03-06T07:29:02.341932+0000","last_peered":"2024-03-06T07:29:02.341932+0000","last_clean":"2024-03-06T07:29:02.341932+0000","last_became_active":"2024-03-06T07:29:00.323304+0000","last_became_peered":"2024-03-06T07:29:00.323304+0000","last_unstale":"2024-03-06T07:29:02.341932+0000","last_undegraded":"2024-03-06T07:29:02.341932+0000","last_fullsized":"2024-03-06T07:29:02.341932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070661+0000","last_change":"2024-03-06T07:29:01.314796+0000","last_active":"2024-03-06T07:29:03.070661+0000","last_peered":"2024-03-06T07:29:03.070661+0000","last_clean":"2024-03-06T07:29:03.070661+0000","last_became_active":"2024-03-06T07:29:01.313844+0000","last_became_peered":"2024-03-06T07:29:01.313844+0000","last_unstale":"2024-03-06T07:29:03.070661+0000","last_undegraded":"2024-03-06T07:29:03.070661+0000","last_fullsized":"2024-03-06T07:29:03.070661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.332089+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.331805+0000","last_became_peered":"2024-03-06T07:29:00.331805+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856581+0000","last_change":"2024-03-06T07:29:01.309249+0000","last_active":"2024-03-06T07:29:02.856581+0000","last_peered":"2024-03-06T07:29:02.856581+0000","last_clean":"2024-03-06T07:29:02.856581+0000","last_became_active":"2024-03-06T07:29:01.308932+0000","last_became_peered":"2024-03-06T07:29:01.308932+0000","last_unstale":"2024-03-06T07:29:02.856581+0000","last_undegraded":"2024-03-06T07:29:02.856581+0000","last_fullsized":"2024-03-06T07:29:02.856581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303785+0000","last_change":"2024-03-06T07:29:00.317215+0000","last_active":"2024-03-06T07:29:02.303785+0000","last_peered":"2024-03-06T07:29:02.303785+0000","last_clean":"2024-03-06T07:29:02.303785+0000","last_became_active":"2024-03-06T07:29:00.314874+0000","last_became_peered":"2024-03-06T07:29:00.314874+0000","last_unstale":"2024-03-06T07:29:02.303785+0000","last_undegraded":"2024-03-06T07:29:02.303785+0000","last_fullsized":"2024-03-06T07:29:02.303785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309601+0000","last_change":"2024-03-06T07:29:01.328026+0000","last_active":"2024-03-06T07:29:02.309601+0000","last_peered":"2024-03-06T07:29:02.309601+0000","last_clean":"2024-03-06T07:29:02.309601+0000","last_became_active":"2024-03-06T07:29:01.327519+0000","last_became_peered":"2024-03-06T07:29:01.327519+0000","last_unstale":"2024-03-06T07:29:02.309601+0000","last_undegraded":"2024-03-06T07:29:02.309601+0000","last_fullsized":"2024-03-06T07:29:02.309601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:29.282488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338149+0000","last_change":"2024-03-06T07:29:00.318811+0000","last_active":"2024-03-06T07:29:02.338149+0000","last_peered":"2024-03-06T07:29:02.338149+0000","last_clean":"2024-03-06T07:29:02.338149+0000","last_became_active":"2024-03-06T07:29:00.318251+0000","last_became_peered":"2024-03-06T07:29:00.318251+0000","last_unstale":"2024-03-06T07:29:02.338149+0000","last_undegraded":"2024-03-06T07:29:02.338149+0000","last_fullsized":"2024-03-06T07:29:02.338149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069507+0000","last_change":"2024-03-06T07:29:01.298030+0000","last_active":"2024-03-06T07:29:03.069507+0000","last_peered":"2024-03-06T07:29:03.069507+0000","last_clean":"2024-03-06T07:29:03.069507+0000","last_became_active":"2024-03-06T07:29:01.297714+0000","last_became_peered":"2024-03-06T07:29:01.297714+0000","last_unstale":"2024-03-06T07:29:03.069507+0000","last_undegraded":"2024-03-06T07:29:03.069507+0000","last_fullsized":"2024-03-06T07:29:03.069507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338187+0000","last_change":"2024-03-06T07:29:00.310808+0000","last_active":"2024-03-06T07:29:02.338187+0000","last_peered":"2024-03-06T07:29:02.338187+0000","last_clean":"2024-03-06T07:29:02.338187+0000","last_became_active":"2024-03-06T07:29:00.310511+0000","last_became_peered":"2024-03-06T07:29:00.310511+0000","last_unstale":"2024-03-06T07:29:02.338187+0000","last_undegraded":"2024-03-06T07:29:02.338187+0000","last_fullsized":"2024-03-06T07:29:02.338187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070152+0000","last_change":"2024-03-06T07:29:01.307662+0000","last_active":"2024-03-06T07:29:03.070152+0000","last_peered":"2024-03-06T07:29:03.070152+0000","last_clean":"2024-03-06T07:29:03.070152+0000","last_became_active":"2024-03-06T07:29:01.307375+0000","last_became_peered":"2024-03-06T07:29:01.307375+0000","last_unstale":"2024-03-06T07:29:03.070152+0000","last_undegraded":"2024-03-06T07:29:03.070152+0000","last_fullsized":"2024-03-06T07:29:03.070152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305649+0000","last_change":"2024-03-06T07:29:00.327844+0000","last_active":"2024-03-06T07:29:02.305649+0000","last_peered":"2024-03-06T07:29:02.305649+0000","last_clean":"2024-03-06T07:29:02.305649+0000","last_became_active":"2024-03-06T07:29:00.327512+0000","last_became_peered":"2024-03-06T07:29:00.327512+0000","last_unstale":"2024-03-06T07:29:02.305649+0000","last_undegraded":"2024-03-06T07:29:02.305649+0000","last_fullsized":"2024-03-06T07:29:02.305649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343323+0000","last_change":"2024-03-06T07:29:01.328118+0000","last_active":"2024-03-06T07:29:02.343323+0000","last_peered":"2024-03-06T07:29:02.343323+0000","last_clean":"2024-03-06T07:29:02.343323+0000","last_became_active":"2024-03-06T07:29:01.327753+0000","last_became_peered":"2024-03-06T07:29:01.327753+0000","last_unstale":"2024-03-06T07:29:02.343323+0000","last_undegraded":"2024-03-06T07:29:02.343323+0000","last_fullsized":"2024-03-06T07:29:02.343323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332010+0000","last_change":"2024-03-06T07:29:01.343767+0000","last_active":"2024-03-06T07:29:02.332010+0000","last_peered":"2024-03-06T07:29:02.332010+0000","last_clean":"2024-03-06T07:29:02.332010+0000","last_became_active":"2024-03-06T07:29:01.343223+0000","last_became_peered":"2024-03-06T07:29:01.343223+0000","last_unstale":"2024-03-06T07:29:02.332010+0000","last_undegraded":"2024-03-06T07:29:02.332010+0000","last_fullsized":"2024-03-06T07:29:02.332010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304611+0000","last_change":"2024-03-06T07:29:00.330067+0000","last_active":"2024-03-06T07:29:02.304611+0000","last_peered":"2024-03-06T07:29:02.304611+0000","last_clean":"2024-03-06T07:29:02.304611+0000","last_became_active":"2024-03-06T07:29:00.329686+0000","last_became_peered":"2024-03-06T07:29:00.329686+0000","last_unstale":"2024-03-06T07:29:02.304611+0000","last_undegraded":"2024-03-06T07:29:02.304611+0000","last_fullsized":"2024-03-06T07:29:02.304611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331745+0000","last_change":"2024-03-06T07:29:01.338965+0000","last_active":"2024-03-06T07:29:02.331745+0000","last_peered":"2024-03-06T07:29:02.331745+0000","last_clean":"2024-03-06T07:29:02.331745+0000","last_became_active":"2024-03-06T07:29:01.338503+0000","last_became_peered":"2024-03-06T07:29:01.338503+0000","last_unstale":"2024-03-06T07:29:02.331745+0000","last_undegraded":"2024-03-06T07:29:02.331745+0000","last_fullsized":"2024-03-06T07:29:02.331745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305289+0000","last_change":"2024-03-06T07:29:00.326670+0000","last_active":"2024-03-06T07:29:02.305289+0000","last_peered":"2024-03-06T07:29:02.305289+0000","last_clean":"2024-03-06T07:29:02.305289+0000","last_became_active":"2024-03-06T07:29:00.326336+0000","last_became_peered":"2024-03-06T07:29:00.326336+0000","last_unstale":"2024-03-06T07:29:02.305289+0000","last_undegraded":"2024-03-06T07:29:02.305289+0000","last_fullsized":"2024-03-06T07:29:02.305289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856601+0000","last_change":"2024-03-06T07:29:01.332554+0000","last_active":"2024-03-06T07:29:02.856601+0000","last_peered":"2024-03-06T07:29:02.856601+0000","last_clean":"2024-03-06T07:29:02.856601+0000","last_became_active":"2024-03-06T07:29:01.332081+0000","last_became_peered":"2024-03-06T07:29:01.332081+0000","last_unstale":"2024-03-06T07:29:02.856601+0000","last_undegraded":"2024-03-06T07:29:02.856601+0000","last_fullsized":"2024-03-06T07:29:02.856601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070670+0000","last_change":"2024-03-06T07:29:00.317549+0000","last_active":"2024-03-06T07:29:03.070670+0000","last_peered":"2024-03-06T07:29:03.070670+0000","last_clean":"2024-03-06T07:29:03.070670+0000","last_became_active":"2024-03-06T07:29:00.317097+0000","last_became_peered":"2024-03-06T07:29:00.317097+0000","last_unstale":"2024-03-06T07:29:03.070670+0000","last_undegraded":"2024-03-06T07:29:03.070670+0000","last_fullsized":"2024-03-06T07:29:03.070670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070604+0000","last_change":"2024-03-06T07:29:01.319129+0000","last_active":"2024-03-06T07:29:03.070604+0000","last_peered":"2024-03-06T07:29:03.070604+0000","last_clean":"2024-03-06T07:29:03.070604+0000","last_became_active":"2024-03-06T07:29:01.318772+0000","last_became_peered":"2024-03-06T07:29:01.318772+0000","last_unstale":"2024-03-06T07:29:03.070604+0000","last_undegraded":"2024-03-06T07:29:03.070604+0000","last_fullsized":"2024-03-06T07:29:03.070604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856882+0000","last_change":"2024-03-06T07:29:00.316149+0000","last_active":"2024-03-06T07:29:02.856882+0000","last_peered":"2024-03-06T07:29:02.856882+0000","last_clean":"2024-03-06T07:29:02.856882+0000","last_became_active":"2024-03-06T07:29:00.315348+0000","last_became_peered":"2024-03-06T07:29:00.315348+0000","last_unstale":"2024-03-06T07:29:02.856882+0000","last_undegraded":"2024-03-06T07:29:02.856882+0000","last_fullsized":"2024-03-06T07:29:02.856882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:26.363805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:05.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.064+0000 7f4c9fc00640 1 Processor -- start 2024-03-06T07:29:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.064+0000 7f4c9fc00640 1 -- start start 2024-03-06T07:29:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.065+0000 7f4c9fc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c9815e080 0x7f4c9806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.065+0000 7f4c9fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c980cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.065+0000 7f4c9fc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9806c920 con 0x7f4c9806de20 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.065+0000 7f4c9fc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9806d530 con 0x7f4c9815e080 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.065+0000 7f4c9fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9806cc00 con 0x7f4c980d2340 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c9815e080 0x7f4c9806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c9815e080 0x7f4c9806c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57834/0 (socket says 172.21.15.32:57834) 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 -- 172.21.15.32:0/1042073979 learned_addr learned my addr 172.21.15.32:0/1042073979 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9dc00640 1 --2- 172.21.15.32:0/1042073979 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c980cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 -- 172.21.15.32:0/1042073979 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 msgr2=0x7f4c980cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 --2- 172.21.15.32:0/1042073979 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c980cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 -- 172.21.15.32:0/1042073979 >> v1:172.21.15.39:6789/0 conn(0x7f4c9806de20 legacy=0x7f4c980d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.066+0000 7f4c9d200640 1 -- 172.21.15.32:0/1042073979 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9806cee0 con 0x7f4c9815e080 2024-03-06T07:29:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.067+0000 7f4c9d200640 1 --2- 172.21.15.32:0/1042073979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9815e080 0x7f4c9806c0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c9806d6b0 tx=0x7f4c8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac1477b4374504d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c97400640 1 -- 172.21.15.32:0/1042073979 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c880317c0 con 0x7f4c9815e080 2024-03-06T07:29:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c97400640 1 -- 172.21.15.32:0/1042073979 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88037070 con 0x7f4c9815e080 2024-03-06T07:29:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c97400640 1 -- 172.21.15.32:0/1042073979 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88031b80 con 0x7f4c9815e080 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c9fc00640 1 -- 172.21.15.32:0/1042073979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9815e080 msgr2=0x7f4c9806c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/1042073979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9815e080 0x7f4c9806c0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c9806d6b0 tx=0x7f4c8802b860 comp rx=0 tx=0).stop 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c9fc00640 1 -- 172.21.15.32:0/1042073979 shutdown_connections 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/1042073979 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c980cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/1042073979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9815e080 0x7f4c9806c0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.068+0000 7f4c9fc00640 1 -- 172.21.15.32:0/1042073979 >> 172.21.15.32:0/1042073979 conn(0x7f4c9809aa10 msgr2=0x7f4c9809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.069+0000 7f4c9fc00640 1 -- 172.21.15.32:0/1042073979 shutdown_connections 2024-03-06T07:29:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.069+0000 7f4c9fc00640 1 -- 172.21.15.32:0/1042073979 wait complete. 2024-03-06T07:29:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.071+0000 7f4c9fc00640 1 Processor -- start 2024-03-06T07:29:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.071+0000 7f4c9fc00640 1 -- start start 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.072+0000 7f4c9fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 0x7f4c980c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.072+0000 7f4c9fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c98160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.072+0000 7f4c9fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9815e080 0x7f4c9809d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.072+0000 7f4c9fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9815e460 con 0x7f4c9806de20 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.072+0000 7f4c9fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9806c600 con 0x7f4c9815e080 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.072+0000 7f4c9fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d0290 con 0x7f4c980d2340 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 0x7f4c980c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 0x7f4c980c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57838/0 (socket says 172.21.15.32:57838) 2024-03-06T07:29:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 -- 172.21.15.32:0/685638827 learned_addr learned my addr 172.21.15.32:0/685638827 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9dc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9815e080 0x7f4c9809d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c97e00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c98160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 -- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 msgr2=0x7f4c98160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c98160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 -- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9815e080 msgr2=0x7f4c9809d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9815e080 0x7f4c9809d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.073+0000 7f4c9d200640 1 -- 172.21.15.32:0/685638827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9809e150 con 0x7f4c9806de20 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.074+0000 7f4c9d200640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 0x7f4c980c8d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4c88000c00 tx=0x7f4c880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.074+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88032860 con 0x7f4c9806de20 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.074+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88032a00 con 0x7f4c9806de20 2024-03-06T07:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.074+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88032df0 con 0x7f4c9806de20 2024-03-06T07:29:06.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.074+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9815e830 con 0x7f4c9806de20 2024-03-06T07:29:06.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.074+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c981611a0 con 0x7f4c9806de20 2024-03-06T07:29:06.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.075+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9804f3a0 con 0x7f4c9806de20 2024-03-06T07:29:06.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.076+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c880453f0 con 0x7f4c9806de20 2024-03-06T07:29:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.077+0000 7f4c94c00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c7406b4d0 0x7f4c7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.077+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c880ac0e0 con 0x7f4c9806de20 2024-03-06T07:29:06.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.078+0000 7f4c97e00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c7406b4d0 0x7f4c7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.078+0000 7f4c97e00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c7406b4d0 0x7f4c7406d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c980c8d40 tx=0x7f4c80009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.082+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c88040070 con 0x7f4c9806de20 2024-03-06T07:29:06.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.285+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c9815e5e0 con 0x7f4c7406b4d0 2024-03-06T07:29:06.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.299+0000 7f4c94c00640 1 -- 172.21.15.32:0/685638827 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251949 (secure 0 0 0) 0x7f4c9815e5e0 con 0x7f4c7406b4d0 2024-03-06T07:29:06.303 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:06.306 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:29:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c7406b4d0 msgr2=0x7f4c7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c7406b4d0 0x7f4c7406d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c980c8d40 tx=0x7f4c80009040 comp rx=0 tx=0).stop 2024-03-06T07:29:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 msgr2=0x7f4c980c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 0x7f4c980c8d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4c88000c00 tx=0x7f4c880046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 shutdown_connections 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c7406b4d0 0x7f4c7406d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9815e080 0x7f4c9809d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c980d2340 0x7f4c98160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 --2- 172.21.15.32:0/685638827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9806de20 0x7f4c980c8d90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 >> 172.21.15.32:0/685638827 conn(0x7f4c9809aa10 msgr2=0x7f4c980b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 shutdown_connections 2024-03-06T07:29:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.306+0000 7f4c9fc00640 1 -- 172.21.15.32:0/685638827 wait complete. 2024-03-06T07:29:06.329 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:29:06.187097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252692,"kb_used_data":2884,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722028,"statfs":{"total":773094113280,"available":772835356672,"internally_reserved":0,"allocated":2953216,"data_stored":1771496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002165"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306066+0000","last_change":"2024-03-06T07:29:01.317028+0000","last_active":"2024-03-06T07:29:02.306066+0000","last_peered":"2024-03-06T07:29:02.306066+0000","last_clean":"2024-03-06T07:29:02.306066+0000","last_became_active":"2024-03-06T07:29:01.316273+0000","last_became_peered":"2024-03-06T07:29:01.316273+0000","last_unstale":"2024-03-06T07:29:02.306066+0000","last_undegraded":"2024-03-06T07:29:02.306066+0000","last_fullsized":"2024-03-06T07:29:02.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338412+0000","last_change":"2024-03-06T07:29:01.311231+0000","last_active":"2024-03-06T07:29:02.338412+0000","last_peered":"2024-03-06T07:29:02.338412+0000","last_clean":"2024-03-06T07:29:02.338412+0000","last_became_active":"2024-03-06T07:29:01.310868+0000","last_became_peered":"2024-03-06T07:29:01.310868+0000","last_unstale":"2024-03-06T07:29:02.338412+0000","last_undegraded":"2024-03-06T07:29:02.338412+0000","last_fullsized":"2024-03-06T07:29:02.338412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856022+0000","last_change":"2024-03-06T07:29:01.307601+0000","last_active":"2024-03-06T07:29:02.856022+0000","last_peered":"2024-03-06T07:29:02.856022+0000","last_clean":"2024-03-06T07:29:02.856022+0000","last_became_active":"2024-03-06T07:29:01.307310+0000","last_became_peered":"2024-03-06T07:29:01.307310+0000","last_unstale":"2024-03-06T07:29:02.856022+0000","last_undegraded":"2024-03-06T07:29:02.856022+0000","last_fullsized":"2024-03-06T07:29:02.856022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331881+0000","last_change":"2024-03-06T07:29:01.326299+0000","last_active":"2024-03-06T07:29:02.331881+0000","last_peered":"2024-03-06T07:29:02.331881+0000","last_clean":"2024-03-06T07:29:02.331881+0000","last_became_active":"2024-03-06T07:29:01.326026+0000","last_became_peered":"2024-03-06T07:29:01.326026+0000","last_unstale":"2024-03-06T07:29:02.331881+0000","last_undegraded":"2024-03-06T07:29:02.331881+0000","last_fullsized":"2024-03-06T07:29:02.331881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304952+0000","last_change":"2024-03-06T07:29:01.314492+0000","last_active":"2024-03-06T07:29:02.304952+0000","last_peered":"2024-03-06T07:29:02.304952+0000","last_clean":"2024-03-06T07:29:02.304952+0000","last_became_active":"2024-03-06T07:29:01.313926+0000","last_became_peered":"2024-03-06T07:29:01.313926+0000","last_unstale":"2024-03-06T07:29:02.304952+0000","last_undegraded":"2024-03-06T07:29:02.304952+0000","last_fullsized":"2024-03-06T07:29:02.304952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331429+0000","last_change":"2024-03-06T07:29:01.328648+0000","last_active":"2024-03-06T07:29:02.331429+0000","last_peered":"2024-03-06T07:29:02.331429+0000","last_clean":"2024-03-06T07:29:02.331429+0000","last_became_active":"2024-03-06T07:29:01.328347+0000","last_became_peered":"2024-03-06T07:29:01.328347+0000","last_unstale":"2024-03-06T07:29:02.331429+0000","last_undegraded":"2024-03-06T07:29:02.331429+0000","last_fullsized":"2024-03-06T07:29:02.331429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304834+0000","last_change":"2024-03-06T07:29:01.310565+0000","last_active":"2024-03-06T07:29:02.304834+0000","last_peered":"2024-03-06T07:29:02.304834+0000","last_clean":"2024-03-06T07:29:02.304834+0000","last_became_active":"2024-03-06T07:29:01.310220+0000","last_became_peered":"2024-03-06T07:29:01.310220+0000","last_unstale":"2024-03-06T07:29:02.304834+0000","last_undegraded":"2024-03-06T07:29:02.304834+0000","last_fullsized":"2024-03-06T07:29:02.304834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.325938+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.325579+0000","last_became_peered":"2024-03-06T07:29:01.325579+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306146+0000","last_change":"2024-03-06T07:29:01.341575+0000","last_active":"2024-03-06T07:29:02.306146+0000","last_peered":"2024-03-06T07:29:02.306146+0000","last_clean":"2024-03-06T07:29:02.306146+0000","last_became_active":"2024-03-06T07:29:01.341372+0000","last_became_peered":"2024-03-06T07:29:01.341372+0000","last_unstale":"2024-03-06T07:29:02.306146+0000","last_undegraded":"2024-03-06T07:29:02.306146+0000","last_fullsized":"2024-03-06T07:29:02.306146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344143+0000","last_change":"2024-03-06T07:29:01.337188+0000","last_active":"2024-03-06T07:29:02.344143+0000","last_peered":"2024-03-06T07:29:02.344143+0000","last_clean":"2024-03-06T07:29:02.344143+0000","last_became_active":"2024-03-06T07:29:01.336845+0000","last_became_peered":"2024-03-06T07:29:01.336845+0000","last_unstale":"2024-03-06T07:29:02.344143+0000","last_undegraded":"2024-03-06T07:29:02.344143+0000","last_fullsized":"2024-03-06T07:29:02.344143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343789+0000","last_change":"2024-03-06T07:29:01.340996+0000","last_active":"2024-03-06T07:29:02.343789+0000","last_peered":"2024-03-06T07:29:02.343789+0000","last_clean":"2024-03-06T07:29:02.343789+0000","last_became_active":"2024-03-06T07:29:01.336560+0000","last_became_peered":"2024-03-06T07:29:01.336560+0000","last_unstale":"2024-03-06T07:29:02.343789+0000","last_undegraded":"2024-03-06T07:29:02.343789+0000","last_fullsized":"2024-03-06T07:29:02.343789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310075+0000","last_change":"2024-03-06T07:29:01.335731+0000","last_active":"2024-03-06T07:29:02.310075+0000","last_peered":"2024-03-06T07:29:02.310075+0000","last_clean":"2024-03-06T07:29:02.310075+0000","last_became_active":"2024-03-06T07:29:01.331758+0000","last_became_peered":"2024-03-06T07:29:01.331758+0000","last_unstale":"2024-03-06T07:29:02.310075+0000","last_undegraded":"2024-03-06T07:29:02.310075+0000","last_fullsized":"2024-03-06T07:29:02.310075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:19.716910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070024+0000","last_change":"2024-03-06T07:29:01.317550+0000","last_active":"2024-03-06T07:29:03.070024+0000","last_peered":"2024-03-06T07:29:03.070024+0000","last_clean":"2024-03-06T07:29:03.070024+0000","last_became_active":"2024-03-06T07:29:01.317098+0000","last_became_peered":"2024-03-06T07:29:01.317098+0000","last_unstale":"2024-03-06T07:29:03.070024+0000","last_undegraded":"2024-03-06T07:29:03.070024+0000","last_fullsized":"2024-03-06T07:29:03.070024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855737+0000","last_change":"2024-03-06T07:29:01.306850+0000","last_active":"2024-03-06T07:29:02.855737+0000","last_peered":"2024-03-06T07:29:02.855737+0000","last_clean":"2024-03-06T07:29:02.855737+0000","last_became_active":"2024-03-06T07:29:01.306532+0000","last_became_peered":"2024-03-06T07:29:01.306532+0000","last_unstale":"2024-03-06T07:29:02.855737+0000","last_undegraded":"2024-03-06T07:29:02.855737+0000","last_fullsized":"2024-03-06T07:29:02.855737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:02.899318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.337987+0000","last_change":"2024-03-06T07:29:01.313116+0000","last_active":"2024-03-06T07:29:02.337987+0000","last_peered":"2024-03-06T07:29:02.337987+0000","last_clean":"2024-03-06T07:29:02.337987+0000","last_became_active":"2024-03-06T07:29:01.311783+0000","last_became_peered":"2024-03-06T07:29:01.311783+0000","last_unstale":"2024-03-06T07:29:02.337987+0000","last_undegraded":"2024-03-06T07:29:02.337987+0000","last_fullsized":"2024-03-06T07:29:02.337987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331678+0000","last_change":"2024-03-06T07:29:01.323368+0000","last_active":"2024-03-06T07:29:02.331678+0000","last_peered":"2024-03-06T07:29:02.331678+0000","last_clean":"2024-03-06T07:29:02.331678+0000","last_became_active":"2024-03-06T07:29:01.323066+0000","last_became_peered":"2024-03-06T07:29:01.323066+0000","last_unstale":"2024-03-06T07:29:02.331678+0000","last_undegraded":"2024-03-06T07:29:02.331678+0000","last_fullsized":"2024-03-06T07:29:02.331678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332654+0000","last_change":"2024-03-06T07:29:01.332332+0000","last_active":"2024-03-06T07:29:02.332654+0000","last_peered":"2024-03-06T07:29:02.332654+0000","last_clean":"2024-03-06T07:29:02.332654+0000","last_became_active":"2024-03-06T07:29:01.331999+0000","last_became_peered":"2024-03-06T07:29:01.331999+0000","last_unstale":"2024-03-06T07:29:02.332654+0000","last_undegraded":"2024-03-06T07:29:02.332654+0000","last_fullsized":"2024-03-06T07:29:02.332654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069247+0000","last_change":"2024-03-06T07:29:01.319453+0000","last_active":"2024-03-06T07:29:03.069247+0000","last_peered":"2024-03-06T07:29:03.069247+0000","last_clean":"2024-03-06T07:29:03.069247+0000","last_became_active":"2024-03-06T07:29:01.319182+0000","last_became_peered":"2024-03-06T07:29:01.319182+0000","last_unstale":"2024-03-06T07:29:03.069247+0000","last_undegraded":"2024-03-06T07:29:03.069247+0000","last_fullsized":"2024-03-06T07:29:03.069247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:42.291305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305802+0000","last_change":"2024-03-06T07:29:01.304420+0000","last_active":"2024-03-06T07:29:02.305802+0000","last_peered":"2024-03-06T07:29:02.305802+0000","last_clean":"2024-03-06T07:29:02.305802+0000","last_became_active":"2024-03-06T07:29:01.303897+0000","last_became_peered":"2024-03-06T07:29:01.303897+0000","last_unstale":"2024-03-06T07:29:02.305802+0000","last_undegraded":"2024-03-06T07:29:02.305802+0000","last_fullsized":"2024-03-06T07:29:02.305802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343811+0000","last_change":"2024-03-06T07:29:01.340804+0000","last_active":"2024-03-06T07:29:02.343811+0000","last_peered":"2024-03-06T07:29:02.343811+0000","last_clean":"2024-03-06T07:29:02.343811+0000","last_became_active":"2024-03-06T07:29:01.338819+0000","last_became_peered":"2024-03-06T07:29:01.338819+0000","last_unstale":"2024-03-06T07:29:02.343811+0000","last_undegraded":"2024-03-06T07:29:02.343811+0000","last_fullsized":"2024-03-06T07:29:02.343811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856684+0000","last_change":"2024-03-06T07:29:01.329232+0000","last_active":"2024-03-06T07:29:02.856684+0000","last_peered":"2024-03-06T07:29:02.856684+0000","last_clean":"2024-03-06T07:29:02.856684+0000","last_became_active":"2024-03-06T07:29:01.328855+0000","last_became_peered":"2024-03-06T07:29:01.328855+0000","last_unstale":"2024-03-06T07:29:02.856684+0000","last_undegraded":"2024-03-06T07:29:02.856684+0000","last_fullsized":"2024-03-06T07:29:02.856684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331471+0000","last_change":"2024-03-06T07:29:01.334776+0000","last_active":"2024-03-06T07:29:02.331471+0000","last_peered":"2024-03-06T07:29:02.331471+0000","last_clean":"2024-03-06T07:29:02.331471+0000","last_became_active":"2024-03-06T07:29:01.332072+0000","last_became_peered":"2024-03-06T07:29:01.332072+0000","last_unstale":"2024-03-06T07:29:02.331471+0000","last_undegraded":"2024-03-06T07:29:02.331471+0000","last_fullsized":"2024-03-06T07:29:02.331471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.312963+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.312645+0000","last_became_peered":"2024-03-06T07:29:01.312645+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331763+0000","last_change":"2024-03-06T07:29:01.322856+0000","last_active":"2024-03-06T07:29:02.331763+0000","last_peered":"2024-03-06T07:29:02.331763+0000","last_clean":"2024-03-06T07:29:02.331763+0000","last_became_active":"2024-03-06T07:29:01.322485+0000","last_became_peered":"2024-03-06T07:29:01.322485+0000","last_unstale":"2024-03-06T07:29:02.331763+0000","last_undegraded":"2024-03-06T07:29:02.331763+0000","last_fullsized":"2024-03-06T07:29:02.331763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070502+0000","last_change":"2024-03-06T07:29:01.328113+0000","last_active":"2024-03-06T07:29:03.070502+0000","last_peered":"2024-03-06T07:29:03.070502+0000","last_clean":"2024-03-06T07:29:03.070502+0000","last_became_active":"2024-03-06T07:29:01.327700+0000","last_became_peered":"2024-03-06T07:29:01.327700+0000","last_unstale":"2024-03-06T07:29:03.070502+0000","last_undegraded":"2024-03-06T07:29:03.070502+0000","last_fullsized":"2024-03-06T07:29:03.070502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344258+0000","last_change":"2024-03-06T07:29:01.328971+0000","last_active":"2024-03-06T07:29:02.344258+0000","last_peered":"2024-03-06T07:29:02.344258+0000","last_clean":"2024-03-06T07:29:02.344258+0000","last_became_active":"2024-03-06T07:29:01.328471+0000","last_became_peered":"2024-03-06T07:29:01.328471+0000","last_unstale":"2024-03-06T07:29:02.344258+0000","last_undegraded":"2024-03-06T07:29:02.344258+0000","last_fullsized":"2024-03-06T07:29:02.344258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338529+0000","last_change":"2024-03-06T07:29:01.310033+0000","last_active":"2024-03-06T07:29:02.338529+0000","last_peered":"2024-03-06T07:29:02.338529+0000","last_clean":"2024-03-06T07:29:02.338529+0000","last_became_active":"2024-03-06T07:29:01.309715+0000","last_became_peered":"2024-03-06T07:29:01.309715+0000","last_unstale":"2024-03-06T07:29:02.338529+0000","last_undegraded":"2024-03-06T07:29:02.338529+0000","last_fullsized":"2024-03-06T07:29:02.338529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069900+0000","last_change":"2024-03-06T07:29:01.338107+0000","last_active":"2024-03-06T07:29:03.069900+0000","last_peered":"2024-03-06T07:29:03.069900+0000","last_clean":"2024-03-06T07:29:03.069900+0000","last_became_active":"2024-03-06T07:29:01.337793+0000","last_became_peered":"2024-03-06T07:29:01.337793+0000","last_unstale":"2024-03-06T07:29:03.069900+0000","last_undegraded":"2024-03-06T07:29:03.069900+0000","last_fullsized":"2024-03-06T07:29:03.069900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856096+0000","last_change":"2024-03-06T07:29:01.309168+0000","last_active":"2024-03-06T07:29:02.856096+0000","last_peered":"2024-03-06T07:29:02.856096+0000","last_clean":"2024-03-06T07:29:02.856096+0000","last_became_active":"2024-03-06T07:29:01.308853+0000","last_became_peered":"2024-03-06T07:29:01.308853+0000","last_unstale":"2024-03-06T07:29:02.856096+0000","last_undegraded":"2024-03-06T07:29:02.856096+0000","last_fullsized":"2024-03-06T07:29:02.856096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332020+0000","last_change":"2024-03-06T07:29:01.339291+0000","last_active":"2024-03-06T07:29:02.332020+0000","last_peered":"2024-03-06T07:29:02.332020+0000","last_clean":"2024-03-06T07:29:02.332020+0000","last_became_active":"2024-03-06T07:29:01.338961+0000","last_became_peered":"2024-03-06T07:29:01.338961+0000","last_unstale":"2024-03-06T07:29:02.332020+0000","last_undegraded":"2024-03-06T07:29:02.332020+0000","last_fullsized":"2024-03-06T07:29:02.332020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068361+0000","last_change":"2024-03-06T07:29:01.339412+0000","last_active":"2024-03-06T07:29:03.068361+0000","last_peered":"2024-03-06T07:29:03.068361+0000","last_clean":"2024-03-06T07:29:03.068361+0000","last_became_active":"2024-03-06T07:29:01.338930+0000","last_became_peered":"2024-03-06T07:29:01.338930+0000","last_unstale":"2024-03-06T07:29:03.068361+0000","last_undegraded":"2024-03-06T07:29:03.068361+0000","last_fullsized":"2024-03-06T07:29:03.068361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343463+0000","last_change":"2024-03-06T07:29:01.316617+0000","last_active":"2024-03-06T07:29:02.343463+0000","last_peered":"2024-03-06T07:29:02.343463+0000","last_clean":"2024-03-06T07:29:02.343463+0000","last_became_active":"2024-03-06T07:29:01.315094+0000","last_became_peered":"2024-03-06T07:29:01.315094+0000","last_unstale":"2024-03-06T07:29:02.343463+0000","last_undegraded":"2024-03-06T07:29:02.343463+0000","last_fullsized":"2024-03-06T07:29:02.343463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309312+0000","last_change":"2024-03-06T07:29:01.314610+0000","last_active":"2024-03-06T07:29:02.309312+0000","last_peered":"2024-03-06T07:29:02.309312+0000","last_clean":"2024-03-06T07:29:02.309312+0000","last_became_active":"2024-03-06T07:29:01.314249+0000","last_became_peered":"2024-03-06T07:29:01.314249+0000","last_unstale":"2024-03-06T07:29:02.309312+0000","last_undegraded":"2024-03-06T07:29:02.309312+0000","last_fullsized":"2024-03-06T07:29:02.309312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:32.840745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070367+0000","last_change":"2024-03-06T07:29:00.318669+0000","last_active":"2024-03-06T07:29:03.070367+0000","last_peered":"2024-03-06T07:29:03.070367+0000","last_clean":"2024-03-06T07:29:03.070367+0000","last_became_active":"2024-03-06T07:29:00.318368+0000","last_became_peered":"2024-03-06T07:29:00.318368+0000","last_unstale":"2024-03-06T07:29:03.070367+0000","last_undegraded":"2024-03-06T07:29:03.070367+0000","last_fullsized":"2024-03-06T07:29:03.070367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856486+0000","last_change":"2024-03-06T07:29:01.330131+0000","last_active":"2024-03-06T07:29:02.856486+0000","last_peered":"2024-03-06T07:29:02.856486+0000","last_clean":"2024-03-06T07:29:02.856486+0000","last_became_active":"2024-03-06T07:29:01.329564+0000","last_became_peered":"2024-03-06T07:29:01.329564+0000","last_unstale":"2024-03-06T07:29:02.856486+0000","last_undegraded":"2024-03-06T07:29:02.856486+0000","last_fullsized":"2024-03-06T07:29:02.856486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343246+0000","last_change":"2024-03-06T07:29:00.325104+0000","last_active":"2024-03-06T07:29:02.343246+0000","last_peered":"2024-03-06T07:29:02.343246+0000","last_clean":"2024-03-06T07:29:02.343246+0000","last_became_active":"2024-03-06T07:29:00.324737+0000","last_became_peered":"2024-03-06T07:29:00.324737+0000","last_unstale":"2024-03-06T07:29:02.343246+0000","last_undegraded":"2024-03-06T07:29:02.343246+0000","last_fullsized":"2024-03-06T07:29:02.343246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068981+0000","last_change":"2024-03-06T07:29:00.322124+0000","last_active":"2024-03-06T07:29:03.068981+0000","last_peered":"2024-03-06T07:29:03.068981+0000","last_clean":"2024-03-06T07:29:03.068981+0000","last_became_active":"2024-03-06T07:29:00.321658+0000","last_became_peered":"2024-03-06T07:29:00.321658+0000","last_unstale":"2024-03-06T07:29:03.068981+0000","last_undegraded":"2024-03-06T07:29:03.068981+0000","last_fullsized":"2024-03-06T07:29:03.068981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069549+0000","last_change":"2024-03-06T07:29:01.306201+0000","last_active":"2024-03-06T07:29:03.069549+0000","last_peered":"2024-03-06T07:29:03.069549+0000","last_clean":"2024-03-06T07:29:03.069549+0000","last_became_active":"2024-03-06T07:29:01.305652+0000","last_became_peered":"2024-03-06T07:29:01.305652+0000","last_unstale":"2024-03-06T07:29:03.069549+0000","last_undegraded":"2024-03-06T07:29:03.069549+0000","last_fullsized":"2024-03-06T07:29:03.069549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331796+0000","last_change":"2024-03-06T07:29:00.330308+0000","last_active":"2024-03-06T07:29:02.331796+0000","last_peered":"2024-03-06T07:29:02.331796+0000","last_clean":"2024-03-06T07:29:02.331796+0000","last_became_active":"2024-03-06T07:29:00.329727+0000","last_became_peered":"2024-03-06T07:29:00.329727+0000","last_unstale":"2024-03-06T07:29:02.331796+0000","last_undegraded":"2024-03-06T07:29:02.331796+0000","last_fullsized":"2024-03-06T07:29:02.331796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338071+0000","last_change":"2024-03-06T07:29:01.308157+0000","last_active":"2024-03-06T07:29:02.338071+0000","last_peered":"2024-03-06T07:29:02.338071+0000","last_clean":"2024-03-06T07:29:02.338071+0000","last_became_active":"2024-03-06T07:29:01.307819+0000","last_became_peered":"2024-03-06T07:29:01.307819+0000","last_unstale":"2024-03-06T07:29:02.338071+0000","last_undegraded":"2024-03-06T07:29:02.338071+0000","last_fullsized":"2024-03-06T07:29:02.338071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.334131+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.333816+0000","last_became_peered":"2024-03-06T07:29:00.333816+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332394+0000","last_change":"2024-03-06T07:29:01.332596+0000","last_active":"2024-03-06T07:29:02.332394+0000","last_peered":"2024-03-06T07:29:02.332394+0000","last_clean":"2024-03-06T07:29:02.332394+0000","last_became_active":"2024-03-06T07:29:01.332140+0000","last_became_peered":"2024-03-06T07:29:01.332140+0000","last_unstale":"2024-03-06T07:29:02.332394+0000","last_undegraded":"2024-03-06T07:29:02.332394+0000","last_fullsized":"2024-03-06T07:29:02.332394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332369+0000","last_change":"2024-03-06T07:29:00.332773+0000","last_active":"2024-03-06T07:29:02.332369+0000","last_peered":"2024-03-06T07:29:02.332369+0000","last_clean":"2024-03-06T07:29:02.332369+0000","last_became_active":"2024-03-06T07:29:00.332477+0000","last_became_peered":"2024-03-06T07:29:00.332477+0000","last_unstale":"2024-03-06T07:29:02.332369+0000","last_undegraded":"2024-03-06T07:29:02.332369+0000","last_fullsized":"2024-03-06T07:29:02.332369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856820+0000","last_change":"2024-03-06T07:29:01.325722+0000","last_active":"2024-03-06T07:29:02.856820+0000","last_peered":"2024-03-06T07:29:02.856820+0000","last_clean":"2024-03-06T07:29:02.856820+0000","last_became_active":"2024-03-06T07:29:01.325401+0000","last_became_peered":"2024-03-06T07:29:01.325401+0000","last_unstale":"2024-03-06T07:29:02.856820+0000","last_undegraded":"2024-03-06T07:29:02.856820+0000","last_fullsized":"2024-03-06T07:29:02.856820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338454+0000","last_change":"2024-03-06T07:29:00.331332+0000","last_active":"2024-03-06T07:29:02.338454+0000","last_peered":"2024-03-06T07:29:02.338454+0000","last_clean":"2024-03-06T07:29:02.338454+0000","last_became_active":"2024-03-06T07:29:00.331008+0000","last_became_peered":"2024-03-06T07:29:00.331008+0000","last_unstale":"2024-03-06T07:29:02.338454+0000","last_undegraded":"2024-03-06T07:29:02.338454+0000","last_fullsized":"2024-03-06T07:29:02.338454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310152+0000","last_change":"2024-03-06T07:29:01.335452+0000","last_active":"2024-03-06T07:29:02.310152+0000","last_peered":"2024-03-06T07:29:02.310152+0000","last_clean":"2024-03-06T07:29:02.310152+0000","last_became_active":"2024-03-06T07:29:01.331241+0000","last_became_peered":"2024-03-06T07:29:01.331241+0000","last_unstale":"2024-03-06T07:29:02.310152+0000","last_undegraded":"2024-03-06T07:29:02.310152+0000","last_fullsized":"2024-03-06T07:29:02.310152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343888+0000","last_change":"2024-03-06T07:29:00.322219+0000","last_active":"2024-03-06T07:29:02.343888+0000","last_peered":"2024-03-06T07:29:02.343888+0000","last_clean":"2024-03-06T07:29:02.343888+0000","last_became_active":"2024-03-06T07:29:00.321921+0000","last_became_peered":"2024-03-06T07:29:00.321921+0000","last_unstale":"2024-03-06T07:29:02.343888+0000","last_undegraded":"2024-03-06T07:29:02.343888+0000","last_fullsized":"2024-03-06T07:29:02.343888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304565+0000","last_change":"2024-03-06T07:29:01.309672+0000","last_active":"2024-03-06T07:29:02.304565+0000","last_peered":"2024-03-06T07:29:02.304565+0000","last_clean":"2024-03-06T07:29:02.304565+0000","last_became_active":"2024-03-06T07:29:01.309369+0000","last_became_peered":"2024-03-06T07:29:01.309369+0000","last_unstale":"2024-03-06T07:29:02.304565+0000","last_undegraded":"2024-03-06T07:29:02.304565+0000","last_fullsized":"2024-03-06T07:29:02.304565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856650+0000","last_change":"2024-03-06T07:29:00.324810+0000","last_active":"2024-03-06T07:29:02.856650+0000","last_peered":"2024-03-06T07:29:02.856650+0000","last_clean":"2024-03-06T07:29:02.856650+0000","last_became_active":"2024-03-06T07:29:00.324524+0000","last_became_peered":"2024-03-06T07:29:00.324524+0000","last_unstale":"2024-03-06T07:29:02.856650+0000","last_undegraded":"2024-03-06T07:29:02.856650+0000","last_fullsized":"2024-03-06T07:29:02.856650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:20.411708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855986+0000","last_change":"2024-03-06T07:29:01.329235+0000","last_active":"2024-03-06T07:29:02.855986+0000","last_peered":"2024-03-06T07:29:02.855986+0000","last_clean":"2024-03-06T07:29:02.855986+0000","last_became_active":"2024-03-06T07:29:01.328953+0000","last_became_peered":"2024-03-06T07:29:01.328953+0000","last_unstale":"2024-03-06T07:29:02.855986+0000","last_undegraded":"2024-03-06T07:29:02.855986+0000","last_fullsized":"2024-03-06T07:29:02.855986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331601+0000","last_change":"2024-03-06T07:29:00.325328+0000","last_active":"2024-03-06T07:29:02.331601+0000","last_peered":"2024-03-06T07:29:02.331601+0000","last_clean":"2024-03-06T07:29:02.331601+0000","last_became_active":"2024-03-06T07:29:00.323961+0000","last_became_peered":"2024-03-06T07:29:00.323961+0000","last_unstale":"2024-03-06T07:29:02.331601+0000","last_undegraded":"2024-03-06T07:29:02.331601+0000","last_fullsized":"2024-03-06T07:29:02.331601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305341+0000","last_change":"2024-03-06T07:29:01.314222+0000","last_active":"2024-03-06T07:29:02.305341+0000","last_peered":"2024-03-06T07:29:02.305341+0000","last_clean":"2024-03-06T07:29:02.305341+0000","last_became_active":"2024-03-06T07:29:01.313409+0000","last_became_peered":"2024-03-06T07:29:01.313409+0000","last_unstale":"2024-03-06T07:29:02.305341+0000","last_undegraded":"2024-03-06T07:29:02.305341+0000","last_fullsized":"2024-03-06T07:29:02.305341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343329+0000","last_change":"2024-03-06T07:29:00.326529+0000","last_active":"2024-03-06T07:29:02.343329+0000","last_peered":"2024-03-06T07:29:02.343329+0000","last_clean":"2024-03-06T07:29:02.343329+0000","last_became_active":"2024-03-06T07:29:00.326202+0000","last_became_peered":"2024-03-06T07:29:00.326202+0000","last_unstale":"2024-03-06T07:29:02.343329+0000","last_undegraded":"2024-03-06T07:29:02.343329+0000","last_fullsized":"2024-03-06T07:29:02.343329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343344+0000","last_change":"2024-03-06T07:29:01.316333+0000","last_active":"2024-03-06T07:29:02.343344+0000","last_peered":"2024-03-06T07:29:02.343344+0000","last_clean":"2024-03-06T07:29:02.343344+0000","last_became_active":"2024-03-06T07:29:01.315969+0000","last_became_peered":"2024-03-06T07:29:01.315969+0000","last_unstale":"2024-03-06T07:29:02.343344+0000","last_undegraded":"2024-03-06T07:29:02.343344+0000","last_fullsized":"2024-03-06T07:29:02.343344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069217+0000","last_change":"2024-03-06T07:29:00.306241+0000","last_active":"2024-03-06T07:29:03.069217+0000","last_peered":"2024-03-06T07:29:03.069217+0000","last_clean":"2024-03-06T07:29:03.069217+0000","last_became_active":"2024-03-06T07:29:00.305883+0000","last_became_peered":"2024-03-06T07:29:00.305883+0000","last_unstale":"2024-03-06T07:29:03.069217+0000","last_undegraded":"2024-03-06T07:29:03.069217+0000","last_fullsized":"2024-03-06T07:29:03.069217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306244+0000","last_change":"2024-03-06T07:29:01.326203+0000","last_active":"2024-03-06T07:29:02.306244+0000","last_peered":"2024-03-06T07:29:02.306244+0000","last_clean":"2024-03-06T07:29:02.306244+0000","last_became_active":"2024-03-06T07:29:01.325861+0000","last_became_peered":"2024-03-06T07:29:01.325861+0000","last_unstale":"2024-03-06T07:29:02.306244+0000","last_undegraded":"2024-03-06T07:29:02.306244+0000","last_fullsized":"2024-03-06T07:29:02.306244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310043+0000","last_change":"2024-03-06T07:29:00.320015+0000","last_active":"2024-03-06T07:29:02.310043+0000","last_peered":"2024-03-06T07:29:02.310043+0000","last_clean":"2024-03-06T07:29:02.310043+0000","last_became_active":"2024-03-06T07:29:00.319492+0000","last_became_peered":"2024-03-06T07:29:00.319492+0000","last_unstale":"2024-03-06T07:29:02.310043+0000","last_undegraded":"2024-03-06T07:29:02.310043+0000","last_fullsized":"2024-03-06T07:29:02.310043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:49.576279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304906+0000","last_change":"2024-03-06T07:29:01.325857+0000","last_active":"2024-03-06T07:29:02.304906+0000","last_peered":"2024-03-06T07:29:02.304906+0000","last_clean":"2024-03-06T07:29:02.304906+0000","last_became_active":"2024-03-06T07:29:01.325522+0000","last_became_peered":"2024-03-06T07:29:01.325522+0000","last_unstale":"2024-03-06T07:29:02.304906+0000","last_undegraded":"2024-03-06T07:29:02.304906+0000","last_fullsized":"2024-03-06T07:29:02.304906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069075+0000","last_change":"2024-03-06T07:29:00.311734+0000","last_active":"2024-03-06T07:29:03.069075+0000","last_peered":"2024-03-06T07:29:03.069075+0000","last_clean":"2024-03-06T07:29:03.069075+0000","last_became_active":"2024-03-06T07:29:00.309932+0000","last_became_peered":"2024-03-06T07:29:00.309932+0000","last_unstale":"2024-03-06T07:29:03.069075+0000","last_undegraded":"2024-03-06T07:29:03.069075+0000","last_fullsized":"2024-03-06T07:29:03.069075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306517+0000","last_change":"2024-03-06T07:29:01.331125+0000","last_active":"2024-03-06T07:29:02.306517+0000","last_peered":"2024-03-06T07:29:02.306517+0000","last_clean":"2024-03-06T07:29:02.306517+0000","last_became_active":"2024-03-06T07:29:01.330860+0000","last_became_peered":"2024-03-06T07:29:01.330860+0000","last_unstale":"2024-03-06T07:29:02.306517+0000","last_undegraded":"2024-03-06T07:29:02.306517+0000","last_fullsized":"2024-03-06T07:29:02.306517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:44.856694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338728+0000","last_change":"2024-03-06T07:29:00.325056+0000","last_active":"2024-03-06T07:29:02.338728+0000","last_peered":"2024-03-06T07:29:02.338728+0000","last_clean":"2024-03-06T07:29:02.338728+0000","last_became_active":"2024-03-06T07:29:00.324757+0000","last_became_peered":"2024-03-06T07:29:00.324757+0000","last_unstale":"2024-03-06T07:29:02.338728+0000","last_undegraded":"2024-03-06T07:29:02.338728+0000","last_fullsized":"2024-03-06T07:29:02.338728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332058+0000","last_change":"2024-03-06T07:29:01.326654+0000","last_active":"2024-03-06T07:29:02.332058+0000","last_peered":"2024-03-06T07:29:02.332058+0000","last_clean":"2024-03-06T07:29:02.332058+0000","last_became_active":"2024-03-06T07:29:01.325612+0000","last_became_peered":"2024-03-06T07:29:01.325612+0000","last_unstale":"2024-03-06T07:29:02.332058+0000","last_undegraded":"2024-03-06T07:29:02.332058+0000","last_fullsized":"2024-03-06T07:29:02.332058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068894+0000","last_change":"2024-03-06T07:29:00.332332+0000","last_active":"2024-03-06T07:29:03.068894+0000","last_peered":"2024-03-06T07:29:03.068894+0000","last_clean":"2024-03-06T07:29:03.068894+0000","last_became_active":"2024-03-06T07:29:00.331881+0000","last_became_peered":"2024-03-06T07:29:00.331881+0000","last_unstale":"2024-03-06T07:29:03.068894+0000","last_undegraded":"2024-03-06T07:29:03.068894+0000","last_fullsized":"2024-03-06T07:29:03.068894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304989+0000","last_change":"2024-03-06T07:29:01.336946+0000","last_active":"2024-03-06T07:29:02.304989+0000","last_peered":"2024-03-06T07:29:02.304989+0000","last_clean":"2024-03-06T07:29:02.304989+0000","last_became_active":"2024-03-06T07:29:01.336594+0000","last_became_peered":"2024-03-06T07:29:01.336594+0000","last_unstale":"2024-03-06T07:29:02.304989+0000","last_undegraded":"2024-03-06T07:29:02.304989+0000","last_fullsized":"2024-03-06T07:29:02.304989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303958+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:29:02.303958+0000","last_peered":"2024-03-06T07:29:02.303958+0000","last_clean":"2024-03-06T07:29:02.303958+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:29:02.303958+0000","last_undegraded":"2024-03-06T07:29:02.303958+0000","last_fullsized":"2024-03-06T07:29:02.303958+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331455+0000","last_change":"2024-03-06T07:29:00.331164+0000","last_active":"2024-03-06T07:29:02.331455+0000","last_peered":"2024-03-06T07:29:02.331455+0000","last_clean":"2024-03-06T07:29:02.331455+0000","last_became_active":"2024-03-06T07:29:00.330531+0000","last_became_peered":"2024-03-06T07:29:00.330531+0000","last_unstale":"2024-03-06T07:29:02.331455+0000","last_undegraded":"2024-03-06T07:29:02.331455+0000","last_fullsized":"2024-03-06T07:29:02.331455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069681+0000","last_change":"2024-03-06T07:29:01.342482+0000","last_active":"2024-03-06T07:29:03.069681+0000","last_peered":"2024-03-06T07:29:03.069681+0000","last_clean":"2024-03-06T07:29:03.069681+0000","last_became_active":"2024-03-06T07:29:01.342176+0000","last_became_peered":"2024-03-06T07:29:01.342176+0000","last_unstale":"2024-03-06T07:29:03.069681+0000","last_undegraded":"2024-03-06T07:29:03.069681+0000","last_fullsized":"2024-03-06T07:29:03.069681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303790+0000","last_change":"2024-03-06T07:29:00.308988+0000","last_active":"2024-03-06T07:29:02.303790+0000","last_peered":"2024-03-06T07:29:02.303790+0000","last_clean":"2024-03-06T07:29:02.303790+0000","last_became_active":"2024-03-06T07:29:00.304843+0000","last_became_peered":"2024-03-06T07:29:00.304843+0000","last_unstale":"2024-03-06T07:29:02.303790+0000","last_undegraded":"2024-03-06T07:29:02.303790+0000","last_fullsized":"2024-03-06T07:29:02.303790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855676+0000","last_change":"2024-03-06T07:29:01.328580+0000","last_active":"2024-03-06T07:29:02.855676+0000","last_peered":"2024-03-06T07:29:02.855676+0000","last_clean":"2024-03-06T07:29:02.855676+0000","last_became_active":"2024-03-06T07:29:01.328319+0000","last_became_peered":"2024-03-06T07:29:01.328319+0000","last_unstale":"2024-03-06T07:29:02.855676+0000","last_undegraded":"2024-03-06T07:29:02.855676+0000","last_fullsized":"2024-03-06T07:29:02.855676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:33.356914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068423+0000","last_change":"2024-03-06T07:29:00.307455+0000","last_active":"2024-03-06T07:29:03.068423+0000","last_peered":"2024-03-06T07:29:03.068423+0000","last_clean":"2024-03-06T07:29:03.068423+0000","last_became_active":"2024-03-06T07:29:00.307117+0000","last_became_peered":"2024-03-06T07:29:00.307117+0000","last_unstale":"2024-03-06T07:29:03.068423+0000","last_undegraded":"2024-03-06T07:29:03.068423+0000","last_fullsized":"2024-03-06T07:29:03.068423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338099+0000","last_change":"2024-03-06T07:29:01.298549+0000","last_active":"2024-03-06T07:29:02.338099+0000","last_peered":"2024-03-06T07:29:02.338099+0000","last_clean":"2024-03-06T07:29:02.338099+0000","last_became_active":"2024-03-06T07:29:01.298043+0000","last_became_peered":"2024-03-06T07:29:01.298043+0000","last_unstale":"2024-03-06T07:29:02.338099+0000","last_undegraded":"2024-03-06T07:29:02.338099+0000","last_fullsized":"2024-03-06T07:29:02.338099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331760+0000","last_change":"2024-03-06T07:29:00.323986+0000","last_active":"2024-03-06T07:29:02.331760+0000","last_peered":"2024-03-06T07:29:02.331760+0000","last_clean":"2024-03-06T07:29:02.331760+0000","last_became_active":"2024-03-06T07:29:00.323442+0000","last_became_peered":"2024-03-06T07:29:00.323442+0000","last_unstale":"2024-03-06T07:29:02.331760+0000","last_undegraded":"2024-03-06T07:29:02.331760+0000","last_fullsized":"2024-03-06T07:29:02.331760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309938+0000","last_change":"2024-03-06T07:29:01.319824+0000","last_active":"2024-03-06T07:29:02.309938+0000","last_peered":"2024-03-06T07:29:02.309938+0000","last_clean":"2024-03-06T07:29:02.309938+0000","last_became_active":"2024-03-06T07:29:01.317792+0000","last_became_peered":"2024-03-06T07:29:01.317792+0000","last_unstale":"2024-03-06T07:29:02.309938+0000","last_undegraded":"2024-03-06T07:29:02.309938+0000","last_fullsized":"2024-03-06T07:29:02.309938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.562992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338550+0000","last_change":"2024-03-06T07:29:00.312181+0000","last_active":"2024-03-06T07:29:02.338550+0000","last_peered":"2024-03-06T07:29:02.338550+0000","last_clean":"2024-03-06T07:29:02.338550+0000","last_became_active":"2024-03-06T07:29:00.311915+0000","last_became_peered":"2024-03-06T07:29:00.311915+0000","last_unstale":"2024-03-06T07:29:02.338550+0000","last_undegraded":"2024-03-06T07:29:02.338550+0000","last_fullsized":"2024-03-06T07:29:02.338550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332017+0000","last_change":"2024-03-06T07:29:01.332290+0000","last_active":"2024-03-06T07:29:02.332017+0000","last_peered":"2024-03-06T07:29:02.332017+0000","last_clean":"2024-03-06T07:29:02.332017+0000","last_became_active":"2024-03-06T07:29:01.331904+0000","last_became_peered":"2024-03-06T07:29:01.331904+0000","last_unstale":"2024-03-06T07:29:02.332017+0000","last_undegraded":"2024-03-06T07:29:02.332017+0000","last_fullsized":"2024-03-06T07:29:02.332017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306067+0000","last_change":"2024-03-06T07:29:00.315219+0000","last_active":"2024-03-06T07:29:02.306067+0000","last_peered":"2024-03-06T07:29:02.306067+0000","last_clean":"2024-03-06T07:29:02.306067+0000","last_became_active":"2024-03-06T07:29:00.314808+0000","last_became_peered":"2024-03-06T07:29:00.314808+0000","last_unstale":"2024-03-06T07:29:02.306067+0000","last_undegraded":"2024-03-06T07:29:02.306067+0000","last_fullsized":"2024-03-06T07:29:02.306067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338430+0000","last_change":"2024-03-06T07:29:01.303022+0000","last_active":"2024-03-06T07:29:02.338430+0000","last_peered":"2024-03-06T07:29:02.338430+0000","last_clean":"2024-03-06T07:29:02.338430+0000","last_became_active":"2024-03-06T07:29:01.302528+0000","last_became_peered":"2024-03-06T07:29:01.302528+0000","last_unstale":"2024-03-06T07:29:02.338430+0000","last_undegraded":"2024-03-06T07:29:02.338430+0000","last_fullsized":"2024-03-06T07:29:02.338430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.341932+0000","last_change":"2024-03-06T07:29:00.323588+0000","last_active":"2024-03-06T07:29:02.341932+0000","last_peered":"2024-03-06T07:29:02.341932+0000","last_clean":"2024-03-06T07:29:02.341932+0000","last_became_active":"2024-03-06T07:29:00.323304+0000","last_became_peered":"2024-03-06T07:29:00.323304+0000","last_unstale":"2024-03-06T07:29:02.341932+0000","last_undegraded":"2024-03-06T07:29:02.341932+0000","last_fullsized":"2024-03-06T07:29:02.341932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070661+0000","last_change":"2024-03-06T07:29:01.314796+0000","last_active":"2024-03-06T07:29:03.070661+0000","last_peered":"2024-03-06T07:29:03.070661+0000","last_clean":"2024-03-06T07:29:03.070661+0000","last_became_active":"2024-03-06T07:29:01.313844+0000","last_became_peered":"2024-03-06T07:29:01.313844+0000","last_unstale":"2024-03-06T07:29:03.070661+0000","last_undegraded":"2024-03-06T07:29:03.070661+0000","last_fullsized":"2024-03-06T07:29:03.070661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.332089+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.331805+0000","last_became_peered":"2024-03-06T07:29:00.331805+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856581+0000","last_change":"2024-03-06T07:29:01.309249+0000","last_active":"2024-03-06T07:29:02.856581+0000","last_peered":"2024-03-06T07:29:02.856581+0000","last_clean":"2024-03-06T07:29:02.856581+0000","last_became_active":"2024-03-06T07:29:01.308932+0000","last_became_peered":"2024-03-06T07:29:01.308932+0000","last_unstale":"2024-03-06T07:29:02.856581+0000","last_undegraded":"2024-03-06T07:29:02.856581+0000","last_fullsized":"2024-03-06T07:29:02.856581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303785+0000","last_change":"2024-03-06T07:29:00.317215+0000","last_active":"2024-03-06T07:29:02.303785+0000","last_peered":"2024-03-06T07:29:02.303785+0000","last_clean":"2024-03-06T07:29:02.303785+0000","last_became_active":"2024-03-06T07:29:00.314874+0000","last_became_peered":"2024-03-06T07:29:00.314874+0000","last_unstale":"2024-03-06T07:29:02.303785+0000","last_undegraded":"2024-03-06T07:29:02.303785+0000","last_fullsized":"2024-03-06T07:29:02.303785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309601+0000","last_change":"2024-03-06T07:29:01.328026+0000","last_active":"2024-03-06T07:29:02.309601+0000","last_peered":"2024-03-06T07:29:02.309601+0000","last_clean":"2024-03-06T07:29:02.309601+0000","last_became_active":"2024-03-06T07:29:01.327519+0000","last_became_peered":"2024-03-06T07:29:01.327519+0000","last_unstale":"2024-03-06T07:29:02.309601+0000","last_undegraded":"2024-03-06T07:29:02.309601+0000","last_fullsized":"2024-03-06T07:29:02.309601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:29.282488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338149+0000","last_change":"2024-03-06T07:29:00.318811+0000","last_active":"2024-03-06T07:29:02.338149+0000","last_peered":"2024-03-06T07:29:02.338149+0000","last_clean":"2024-03-06T07:29:02.338149+0000","last_became_active":"2024-03-06T07:29:00.318251+0000","last_became_peered":"2024-03-06T07:29:00.318251+0000","last_unstale":"2024-03-06T07:29:02.338149+0000","last_undegraded":"2024-03-06T07:29:02.338149+0000","last_fullsized":"2024-03-06T07:29:02.338149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069507+0000","last_change":"2024-03-06T07:29:01.298030+0000","last_active":"2024-03-06T07:29:03.069507+0000","last_peered":"2024-03-06T07:29:03.069507+0000","last_clean":"2024-03-06T07:29:03.069507+0000","last_became_active":"2024-03-06T07:29:01.297714+0000","last_became_peered":"2024-03-06T07:29:01.297714+0000","last_unstale":"2024-03-06T07:29:03.069507+0000","last_undegraded":"2024-03-06T07:29:03.069507+0000","last_fullsized":"2024-03-06T07:29:03.069507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338187+0000","last_change":"2024-03-06T07:29:00.310808+0000","last_active":"2024-03-06T07:29:02.338187+0000","last_peered":"2024-03-06T07:29:02.338187+0000","last_clean":"2024-03-06T07:29:02.338187+0000","last_became_active":"2024-03-06T07:29:00.310511+0000","last_became_peered":"2024-03-06T07:29:00.310511+0000","last_unstale":"2024-03-06T07:29:02.338187+0000","last_undegraded":"2024-03-06T07:29:02.338187+0000","last_fullsized":"2024-03-06T07:29:02.338187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070152+0000","last_change":"2024-03-06T07:29:01.307662+0000","last_active":"2024-03-06T07:29:03.070152+0000","last_peered":"2024-03-06T07:29:03.070152+0000","last_clean":"2024-03-06T07:29:03.070152+0000","last_became_active":"2024-03-06T07:29:01.307375+0000","last_became_peered":"2024-03-06T07:29:01.307375+0000","last_unstale":"2024-03-06T07:29:03.070152+0000","last_undegraded":"2024-03-06T07:29:03.070152+0000","last_fullsized":"2024-03-06T07:29:03.070152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305649+0000","last_change":"2024-03-06T07:29:00.327844+0000","last_active":"2024-03-06T07:29:02.305649+0000","last_peered":"2024-03-06T07:29:02.305649+0000","last_clean":"2024-03-06T07:29:02.305649+0000","last_became_active":"2024-03-06T07:29:00.327512+0000","last_became_peered":"2024-03-06T07:29:00.327512+0000","last_unstale":"2024-03-06T07:29:02.305649+0000","last_undegraded":"2024-03-06T07:29:02.305649+0000","last_fullsized":"2024-03-06T07:29:02.305649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343323+0000","last_change":"2024-03-06T07:29:01.328118+0000","last_active":"2024-03-06T07:29:02.343323+0000","last_peered":"2024-03-06T07:29:02.343323+0000","last_clean":"2024-03-06T07:29:02.343323+0000","last_became_active":"2024-03-06T07:29:01.327753+0000","last_became_peered":"2024-03-06T07:29:01.327753+0000","last_unstale":"2024-03-06T07:29:02.343323+0000","last_undegraded":"2024-03-06T07:29:02.343323+0000","last_fullsized":"2024-03-06T07:29:02.343323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332010+0000","last_change":"2024-03-06T07:29:01.343767+0000","last_active":"2024-03-06T07:29:02.332010+0000","last_peered":"2024-03-06T07:29:02.332010+0000","last_clean":"2024-03-06T07:29:02.332010+0000","last_became_active":"2024-03-06T07:29:01.343223+0000","last_became_peered":"2024-03-06T07:29:01.343223+0000","last_unstale":"2024-03-06T07:29:02.332010+0000","last_undegraded":"2024-03-06T07:29:02.332010+0000","last_fullsized":"2024-03-06T07:29:02.332010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304611+0000","last_change":"2024-03-06T07:29:00.330067+0000","last_active":"2024-03-06T07:29:02.304611+0000","last_peered":"2024-03-06T07:29:02.304611+0000","last_clean":"2024-03-06T07:29:02.304611+0000","last_became_active":"2024-03-06T07:29:00.329686+0000","last_became_peered":"2024-03-06T07:29:00.329686+0000","last_unstale":"2024-03-06T07:29:02.304611+0000","last_undegraded":"2024-03-06T07:29:02.304611+0000","last_fullsized":"2024-03-06T07:29:02.304611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331745+0000","last_change":"2024-03-06T07:29:01.338965+0000","last_active":"2024-03-06T07:29:02.331745+0000","last_peered":"2024-03-06T07:29:02.331745+0000","last_clean":"2024-03-06T07:29:02.331745+0000","last_became_active":"2024-03-06T07:29:01.338503+0000","last_became_peered":"2024-03-06T07:29:01.338503+0000","last_unstale":"2024-03-06T07:29:02.331745+0000","last_undegraded":"2024-03-06T07:29:02.331745+0000","last_fullsized":"2024-03-06T07:29:02.331745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305289+0000","last_change":"2024-03-06T07:29:00.326670+0000","last_active":"2024-03-06T07:29:02.305289+0000","last_peered":"2024-03-06T07:29:02.305289+0000","last_clean":"2024-03-06T07:29:02.305289+0000","last_became_active":"2024-03-06T07:29:00.326336+0000","last_became_peered":"2024-03-06T07:29:00.326336+0000","last_unstale":"2024-03-06T07:29:02.305289+0000","last_undegraded":"2024-03-06T07:29:02.305289+0000","last_fullsized":"2024-03-06T07:29:02.305289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856601+0000","last_change":"2024-03-06T07:29:01.332554+0000","last_active":"2024-03-06T07:29:02.856601+0000","last_peered":"2024-03-06T07:29:02.856601+0000","last_clean":"2024-03-06T07:29:02.856601+0000","last_became_active":"2024-03-06T07:29:01.332081+0000","last_became_peered":"2024-03-06T07:29:01.332081+0000","last_unstale":"2024-03-06T07:29:02.856601+0000","last_undegraded":"2024-03-06T07:29:02.856601+0000","last_fullsized":"2024-03-06T07:29:02.856601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070670+0000","last_change":"2024-03-06T07:29:00.317549+0000","last_active":"2024-03-06T07:29:03.070670+0000","last_peered":"2024-03-06T07:29:03.070670+0000","last_clean":"2024-03-06T07:29:03.070670+0000","last_became_active":"2024-03-06T07:29:00.317097+0000","last_became_peered":"2024-03-06T07:29:00.317097+0000","last_unstale":"2024-03-06T07:29:03.070670+0000","last_undegraded":"2024-03-06T07:29:03.070670+0000","last_fullsized":"2024-03-06T07:29:03.070670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070604+0000","last_change":"2024-03-06T07:29:01.319129+0000","last_active":"2024-03-06T07:29:03.070604+0000","last_peered":"2024-03-06T07:29:03.070604+0000","last_clean":"2024-03-06T07:29:03.070604+0000","last_became_active":"2024-03-06T07:29:01.318772+0000","last_became_peered":"2024-03-06T07:29:01.318772+0000","last_unstale":"2024-03-06T07:29:03.070604+0000","last_undegraded":"2024-03-06T07:29:03.070604+0000","last_fullsized":"2024-03-06T07:29:03.070604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856882+0000","last_change":"2024-03-06T07:29:00.316149+0000","last_active":"2024-03-06T07:29:02.856882+0000","last_peered":"2024-03-06T07:29:02.856882+0000","last_clean":"2024-03-06T07:29:02.856882+0000","last_became_active":"2024-03-06T07:29:00.315348+0000","last_became_peered":"2024-03-06T07:29:00.315348+0000","last_unstale":"2024-03-06T07:29:02.856882+0000","last_undegraded":"2024-03-06T07:29:02.856882+0000","last_fullsized":"2024-03-06T07:29:02.856882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:26.363805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:06.332 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:06.333 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:29:06.333 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:29:06.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:29:06.333 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:29:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.595+0000 7f26fba00640 1 Processor -- start 2024-03-06T07:29:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.595+0000 7f26fba00640 1 -- start start 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26fba00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26f40af1a0 0x7f26f415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26fba00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f26f40af580 0x7f26f40b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26fba00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f415cc40 con 0x7f26f40cdc20 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26fba00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f415cf20 con 0x7f26f40af1a0 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26fba00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f415d200 con 0x7f26f40af580 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26f9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26f40af1a0 0x7f26f415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26f9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26f40af1a0 0x7f26f415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57848/0 (socket says 172.21.15.32:57848) 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26f9000640 1 -- 172.21.15.32:0/1009170540 learned_addr learned my addr 172.21.15.32:0/1009170540 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.597+0000 7f26ebe00640 1 --2- 172.21.15.32:0/1009170540 >> v2:172.21.15.39:3300/0 conn(0x7f26f40af580 0x7f26f40b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.598+0000 7f26f9000640 1 -- 172.21.15.32:0/1009170540 >> v2:172.21.15.39:3300/0 conn(0x7f26f40af580 msgr2=0x7f26f40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.598+0000 7f26f9000640 1 --2- 172.21.15.32:0/1009170540 >> v2:172.21.15.39:3300/0 conn(0x7f26f40af580 0x7f26f40b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.598+0000 7f26f9000640 1 -- 172.21.15.32:0/1009170540 >> v1:172.21.15.32:6789/0 conn(0x7f26f40cdc20 legacy=0x7f26f40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.598+0000 7f26f9000640 1 -- 172.21.15.32:0/1009170540 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f415d4e0 con 0x7f26f40af1a0 2024-03-06T07:29:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.599+0000 7f26f9000640 1 --2- 172.21.15.32:0/1009170540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af1a0 0x7f26f415c600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26f40b54c0 tx=0x7f26e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db6fe603b12501f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.599+0000 7f26f3e00640 1 -- 172.21.15.32:0/1009170540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4035020 con 0x7f26f40af1a0 2024-03-06T07:29:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.599+0000 7f26f3e00640 1 -- 172.21.15.32:0/1009170540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e4031870 con 0x7f26f40af1a0 2024-03-06T07:29:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.599+0000 7f26f3e00640 1 -- 172.21.15.32:0/1009170540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4031c40 con 0x7f26f40af1a0 2024-03-06T07:29:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 -- 172.21.15.32:0/1009170540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af1a0 msgr2=0x7f26f415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 --2- 172.21.15.32:0/1009170540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af1a0 0x7f26f415c600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26f40b54c0 tx=0x7f26e402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 -- 172.21.15.32:0/1009170540 shutdown_connections 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 --2- 172.21.15.32:0/1009170540 >> v2:172.21.15.39:3300/0 conn(0x7f26f40af580 0x7f26f40b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 --2- 172.21.15.32:0/1009170540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af1a0 0x7f26f415c600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 -- 172.21.15.32:0/1009170540 >> 172.21.15.32:0/1009170540 conn(0x7f26f406afb0 msgr2=0x7f26f406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 -- 172.21.15.32:0/1009170540 shutdown_connections 2024-03-06T07:29:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.600+0000 7f26fba00640 1 -- 172.21.15.32:0/1009170540 wait complete. 2024-03-06T07:29:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.602+0000 7f26fba00640 1 Processor -- start 2024-03-06T07:29:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.602+0000 7f26fba00640 1 -- start start 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26fba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26f40af1a0 0x7f26f414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26fba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 0x7f26f40acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26fba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f40cdc20 0x7f26f40ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26fba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40b3660 con 0x7f26f40af580 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26fba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40b34e0 con 0x7f26f40cdc20 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26fba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40b37e0 con 0x7f26f40af1a0 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 0x7f26f40acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 0x7f26f40acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57856/0 (socket says 172.21.15.32:57856) 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.603+0000 7f26ebe00640 1 -- 172.21.15.32:0/3778754548 learned_addr learned my addr 172.21.15.32:0/3778754548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26f9000640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26f40af1a0 0x7f26f414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26f9a00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f40cdc20 0x7f26f40ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26ebe00640 1 -- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26f40af1a0 msgr2=0x7f26f414b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26ebe00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26f40af1a0 0x7f26f414b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26ebe00640 1 -- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f40cdc20 msgr2=0x7f26f40ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26ebe00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f40cdc20 0x7f26f40ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26ebe00640 1 -- 172.21.15.32:0/3778754548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f409d750 con 0x7f26f40af580 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26ebe00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 0x7f26f40acdb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00d7d0 tx=0x7f26dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00b6c0 con 0x7f26f40af580 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc019070 con 0x7f26f40af580 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00ba80 con 0x7f26f40af580 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.604+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41f3d10 con 0x7f26f40af580 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.605+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41f42d0 con 0x7f26f40af580 2024-03-06T07:29:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.606+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f4069510 con 0x7f26f40af580 2024-03-06T07:29:06.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.607+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26dc00bc20 con 0x7f26f40af580 2024-03-06T07:29:06.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.608+0000 7f26f1600640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f26d006b4d0 0x7f26d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.608+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26dc08a5d0 con 0x7f26f40af580 2024-03-06T07:29:06.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.608+0000 7f26f9000640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f26d006b4d0 0x7f26d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.608+0000 7f26f9000640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f26d006b4d0 0x7f26d006d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f26e4036000 tx=0x7f26e403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.612+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26dc01c360 con 0x7f26f40af580 2024-03-06T07:29:06.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.861+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f26f40cd4f0 con 0x7f26f40af580 2024-03-06T07:29:06.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.867+0000 7f26f1600640 1 -- 172.21.15.32:0/3778754548 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171783 (secure 0 0 0) 0x7f26dc0594e0 con 0x7f26f40af580 2024-03-06T07:29:06.872 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f26d006b4d0 msgr2=0x7f26d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f26d006b4d0 0x7f26d006d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f26e4036000 tx=0x7f26e403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 msgr2=0x7f26f40acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 0x7f26f40acdb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00d7d0 tx=0x7f26dc00dc90 comp rx=0 tx=0).stop 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 shutdown_connections 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f26d006b4d0 0x7f26d006d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f40cdc20 0x7f26f40ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26f40af580 0x7f26f40acdb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 --2- 172.21.15.32:0/3778754548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26f40af1a0 0x7f26f414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.871+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 >> 172.21.15.32:0/3778754548 conn(0x7f26f406afb0 msgr2=0x7f26f40b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.872+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 shutdown_connections 2024-03-06T07:29:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:06.872+0000 7f26fba00640 1 -- 172.21.15.32:0/3778754548 wait complete. 2024-03-06T07:29:06.891 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1434773126},{"type":"v1","addr":"172.21.15.32:6833","nonce":1434773126}]},"active_addr":"172.21.15.32:6833/1434773126","active_change":"2024-03-06T07:28:46.164417+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4114,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2051556695}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":566372995}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":371022697}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2542649247}]}]} 2024-03-06T07:29:06.894 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:29:06.894 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:29:06.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:07.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.107+0000 7fb27d400640 1 Processor -- start 2024-03-06T07:29:07.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.107+0000 7fb27d400640 1 -- start start 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.108+0000 7fb27d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb2780cf5c0 0x7fb2780cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.108+0000 7fb27d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 0x7fb278071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.108+0000 7fb27d400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2780b4860 con 0x7fb2780b3c90 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.108+0000 7fb27d400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2780ce160 con 0x7fb2780cf5c0 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.108+0000 7fb27d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780b4b40 con 0x7fb278071050 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb276800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb2780b3c90 0x7fb2780b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46670/0 (socket says 172.21.15.32:46670) 2024-03-06T07:29:07.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb276800640 1 -- 172.21.15.32:0/3762349783 learned_addr learned my addr 172.21.15.32:0/3762349783 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:07.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb277200640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 0x7fb278071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275e00640 1 --2- 172.21.15.32:0/3762349783 >> v2:172.21.15.39:3300/0 conn(0x7fb2780cf5c0 0x7fb2780cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634585027 0 0) 0x7fb2780b4860 con 0x7fb2780b3c90 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c003520 con 0x7fb2780b3c90 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 847406332 0 0) 0x7fb25c003520 con 0x7fb2780b3c90 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 msgr2=0x7fb278071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 0x7fb278071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 >> v2:172.21.15.39:3300/0 conn(0x7fb2780cf5c0 msgr2=0x7fb2780cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 --2- 172.21.15.32:0/3762349783 >> v2:172.21.15.39:3300/0 conn(0x7fb2780cf5c0 0x7fb2780cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2780b4e20 con 0x7fb2780b3c90 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb268001670 con 0x7fb2780b3c90 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 >> v1:172.21.15.32:6789/0 conn(0x7fb2780b3c90 legacy=0x7fb2780b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb25c003a20 0x7fb25c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.109+0000 7fb275400640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c004800 0x7fb25c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb275400640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 0x7fb25c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780ce160 con 0x7fb25c003a20 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780b4b40 con 0x7fb25c004800 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c00ddc0 con 0x7fb278071050 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb275e00640 1 --2- 172.21.15.32:0/3762349783 >> v2:172.21.15.39:3300/0 conn(0x7fb2780cf5c0 0x7fb2780cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb276800640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb25c003a20 0x7fb25c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb277200640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 0x7fb25c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb276800640 1 -- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 msgr2=0x7fb25c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb275e00640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c004800 0x7fb25c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb276800640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb278071050 0x7fb25c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb276800640 1 -- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c004800 msgr2=0x7fb25c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb276800640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c004800 0x7fb25c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.110+0000 7fb276800640 1 -- 172.21.15.32:0/3762349783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c003520 con 0x7fb25c003a20 2024-03-06T07:29:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.111+0000 7fb276800640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb25c003a20 0x7fb25c003e40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb268007fd0 tx=0x7fb25c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427c0bc631e30407 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.111+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268007d50 con 0x7fb25c003a20 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.111+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb268030070 con 0x7fb25c003a20 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.111+0000 7fb275400640 1 -- 172.21.15.32:0/3762349783 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268005d30 con 0x7fb25c003a20 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb27d400640 1 -- 172.21.15.32:0/3762349783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb25c003a20 msgr2=0x7fb25c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb27d400640 1 --2- 172.21.15.32:0/3762349783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb25c003a20 0x7fb25c003e40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb268007fd0 tx=0x7fb25c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb277200640 1 -- 172.21.15.32:0/3762349783 reap_dead start 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb27d400640 1 -- 172.21.15.32:0/3762349783 shutdown_connections 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb27d400640 1 -- 172.21.15.32:0/3762349783 >> 172.21.15.32:0/3762349783 conn(0x7fb27806afb0 msgr2=0x7fb27806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb27d400640 1 -- 172.21.15.32:0/3762349783 shutdown_connections 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.112+0000 7fb27d400640 1 -- 172.21.15.32:0/3762349783 wait complete. 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.114+0000 7fb27d400640 1 Processor -- start 2024-03-06T07:29:07.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.114+0000 7fb27d400640 1 -- start start 2024-03-06T07:29:07.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb27d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c003a20 0x7fb278151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb27d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2780c9150 0x7fb2781535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb27d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 0x7fb2780af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb27d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb278071490 con 0x7fb278153ae0 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb27d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27815b340 con 0x7fb25c003a20 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb27d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb278071610 con 0x7fb2780c9150 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb277200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 0x7fb2780af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb277200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 0x7fb2780af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57884/0 (socket says 172.21.15.32:57884) 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.115+0000 7fb277200640 1 -- 172.21.15.32:0/1513742093 learned_addr learned my addr 172.21.15.32:0/1513742093 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb275e00640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2780c9150 0x7fb2781535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb276800640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c003a20 0x7fb278151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb277200640 1 -- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2780c9150 msgr2=0x7fb2781535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb277200640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2780c9150 0x7fb2781535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb277200640 1 -- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c003a20 msgr2=0x7fb278151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb277200640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c003a20 0x7fb278151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb277200640 1 -- 172.21.15.32:0/1513742093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2780c96c0 con 0x7fb278153ae0 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb277200640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 0x7fb2780af940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb270009890 tx=0x7fb270009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270007c40 con 0x7fb278153ae0 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb270019070 con 0x7fb278153ae0 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270014410 con 0x7fb278153ae0 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.116+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb278152ac0 con 0x7fb278153ae0 2024-03-06T07:29:07.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.117+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb278151d70 con 0x7fb278153ae0 2024-03-06T07:29:07.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.118+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27804f3a0 con 0x7fb278153ae0 2024-03-06T07:29:07.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.119+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2700145b0 con 0x7fb278153ae0 2024-03-06T07:29:07.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.120+0000 7fb26ea00640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25806b4d0 0x7fb25806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.120+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb27008a690 con 0x7fb278153ae0 2024-03-06T07:29:07.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.120+0000 7fb276800640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25806b4d0 0x7fb25806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.121+0000 7fb276800640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25806b4d0 0x7fb25806d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb268034000 tx=0x7fb268007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.125+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb270059510 con 0x7fb278153ae0 2024-03-06T07:29:07.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.315+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2780b3460 con 0x7fb278153ae0 2024-03-06T07:29:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.317+0000 7fb26ea00640 1 -- 172.21.15.32:0/1513742093 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7fb270058e10 con 0x7fb278153ae0 2024-03-06T07:29:07.319 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:07.319 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25806b4d0 msgr2=0x7fb25806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25806b4d0 0x7fb25806d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb268034000 tx=0x7fb268007000 comp rx=0 tx=0).stop 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 msgr2=0x7fb2780af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 0x7fb2780af940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb270009890 tx=0x7fb270009d50 comp rx=0 tx=0).stop 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 shutdown_connections 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25806b4d0 0x7fb25806d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb278153ae0 0x7fb2780af940 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2780c9150 0x7fb2781535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 --2- 172.21.15.32:0/1513742093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb25c003a20 0x7fb278151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 >> 172.21.15.32:0/1513742093 conn(0x7fb27806afb0 msgr2=0x7fb278155ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.319+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 shutdown_connections 2024-03-06T07:29:07.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.320+0000 7fb27d400640 1 -- 172.21.15.32:0/1513742093 wait complete. 2024-03-06T07:29:07.341 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:29:07.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:07.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.550+0000 7f5262800640 1 Processor -- start 2024-03-06T07:29:07.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.550+0000 7f5262800640 1 -- start start 2024-03-06T07:29:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5262800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f525c1552b0 0x7f525c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5262800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f525c0cd060 0x7f525c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5262800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c15ed10 con 0x7f525c15e810 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5262800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c15eff0 con 0x7f525c1552b0 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5262800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c15f2d0 con 0x7f525c0cd060 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5261400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f525c1552b0 0x7f525c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5261400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f525c1552b0 0x7f525c15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57890/0 (socket says 172.21.15.32:57890) 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5261400640 1 -- 172.21.15.32:0/333829324 learned_addr learned my addr 172.21.15.32:0/333829324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5260a00640 1 --1- 172.21.15.32:0/333829324 >> v1:172.21.15.32:6789/0 conn(0x7f525c15e810 0x7f525c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46680/0 (socket says 172.21.15.32:46680) 2024-03-06T07:29:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.552+0000 7f5261e00640 1 --2- 172.21.15.32:0/333829324 >> v2:172.21.15.39:3300/0 conn(0x7f525c0cd060 0x7f525c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.553+0000 7f5261400640 1 -- 172.21.15.32:0/333829324 >> v2:172.21.15.39:3300/0 conn(0x7f525c0cd060 msgr2=0x7f525c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.553+0000 7f5261400640 1 --2- 172.21.15.32:0/333829324 >> v2:172.21.15.39:3300/0 conn(0x7f525c0cd060 0x7f525c0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.553+0000 7f5261400640 1 -- 172.21.15.32:0/333829324 >> v1:172.21.15.32:6789/0 conn(0x7f525c15e810 legacy=0x7f525c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.553+0000 7f5261400640 1 -- 172.21.15.32:0/333829324 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c15f5b0 con 0x7f525c1552b0 2024-03-06T07:29:07.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.554+0000 7f5261400640 1 --2- 172.21.15.32:0/333829324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c15b1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f525c0cf100 tx=0x7f525002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39fc9fa908e46f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.554+0000 7f525be00640 1 -- 172.21.15.32:0/333829324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525002ca80 con 0x7f525c1552b0 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.554+0000 7f525be00640 1 -- 172.21.15.32:0/333829324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525002cc20 con 0x7f525c1552b0 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.554+0000 7f525be00640 1 -- 172.21.15.32:0/333829324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52500343f0 con 0x7f525c1552b0 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 -- 172.21.15.32:0/333829324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 msgr2=0x7f525c15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 --2- 172.21.15.32:0/333829324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c15b1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f525c0cf100 tx=0x7f525002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 -- 172.21.15.32:0/333829324 shutdown_connections 2024-03-06T07:29:07.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 --2- 172.21.15.32:0/333829324 >> v2:172.21.15.39:3300/0 conn(0x7f525c0cd060 0x7f525c0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 --2- 172.21.15.32:0/333829324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c15b1d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 -- 172.21.15.32:0/333829324 >> 172.21.15.32:0/333829324 conn(0x7f525c06b3d0 msgr2=0x7f525c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 -- 172.21.15.32:0/333829324 shutdown_connections 2024-03-06T07:29:07.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.555+0000 7f5262800640 1 -- 172.21.15.32:0/333829324 wait complete. 2024-03-06T07:29:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.557+0000 7f5262800640 1 Processor -- start 2024-03-06T07:29:07.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.557+0000 7f5262800640 1 -- start start 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5262800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0cd060 0x7f525c1413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5262800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c0b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5262800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c15e810 0x7f525c0b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5262800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c06dd30 con 0x7f525c1552b0 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5262800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0b3760 con 0x7f525c15e810 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5262800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c06deb0 con 0x7f525c0cd060 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5260a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c0b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5260a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c0b1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57894/0 (socket says 172.21.15.32:57894) 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5260a00640 1 -- 172.21.15.32:0/3902281305 learned_addr learned my addr 172.21.15.32:0/3902281305 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5261400640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0cd060 0x7f525c1413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5261e00640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c15e810 0x7f525c0b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.558+0000 7f5260a00640 1 -- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0cd060 msgr2=0x7f525c1413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5260a00640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0cd060 0x7f525c1413a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5260a00640 1 -- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c15e810 msgr2=0x7f525c0b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5260a00640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c15e810 0x7f525c0b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5260a00640 1 -- 172.21.15.32:0/3902281305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c1558a0 con 0x7f525c1552b0 2024-03-06T07:29:07.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5260a00640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c0b1990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f524800c430 tx=0x7f524800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480074e0 con 0x7f525c1552b0 2024-03-06T07:29:07.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5248007680 con 0x7f525c1552b0 2024-03-06T07:29:07.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248004a00 con 0x7f525c1552b0 2024-03-06T07:29:07.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c155b80 con 0x7f525c1552b0 2024-03-06T07:29:07.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.559+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c156140 con 0x7f525c1552b0 2024-03-06T07:29:07.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.560+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c04f330 con 0x7f525c1552b0 2024-03-06T07:29:07.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.561+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f524801a020 con 0x7f525c1552b0 2024-03-06T07:29:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.562+0000 7f5259600640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.562+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5248089aa0 con 0x7f525c1552b0 2024-03-06T07:29:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.562+0000 7f5261400640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.563+0000 7f5261400640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5250035000 tx=0x7f5250030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.566+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52480587f0 con 0x7f525c1552b0 2024-03-06T07:29:07.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.768+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f525c0cf4a0 con 0x7f525c1552b0 2024-03-06T07:29:07.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.770+0000 7f5259600640 1 -- 172.21.15.32:0/3902281305 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f52480580f0 con 0x7f525c1552b0 2024-03-06T07:29:07.773 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:07.773 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 msgr2=0x7f523c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5250035000 tx=0x7f5250030000 comp rx=0 tx=0).stop 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 msgr2=0x7f525c0b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c0b1990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f524800c430 tx=0x7f524800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 shutdown_connections 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c15e810 0x7f525c0b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c1552b0 0x7f525c0b1990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 --2- 172.21.15.32:0/3902281305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0cd060 0x7f525c1413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 >> 172.21.15.32:0/3902281305 conn(0x7f525c06b3d0 msgr2=0x7f525c071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 shutdown_connections 2024-03-06T07:29:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:07.773+0000 7f5262800640 1 -- 172.21.15.32:0/3902281305 wait complete. 2024-03-06T07:29:07.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:29:07.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:29:07.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:29:07.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:29:07.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:29:07.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:29:07.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:29:07.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:29:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.098+0000 7f793b000640 1 Processor -- start 2024-03-06T07:29:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.099+0000 7f793b000640 1 -- start start 2024-03-06T07:29:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.101+0000 7f793b000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79340712c0 0x7f7934162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.101+0000 7f5d0a400640 1 Processor -- start 2024-03-06T07:29:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.101+0000 7f793b000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79340ce090 con 0x7f7934070ee0 2024-03-06T07:29:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.101+0000 7f793b000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79340ce370 con 0x7f7934072da0 2024-03-06T07:29:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.101+0000 7f793b000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79340ceca0 con 0x7f79340712c0 2024-03-06T07:29:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.102+0000 7f5d0a400640 1 -- start start 2024-03-06T07:29:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.102+0000 7f7933e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7934070ee0 0x7f793406e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56580/0 (socket says 172.21.15.32:56580) 2024-03-06T07:29:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.102+0000 7f7933e00640 1 -- 172.21.15.32:0/116166543 learned_addr learned my addr 172.21.15.32:0/116166543 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.102+0000 7f7939000640 1 --2- 172.21.15.32:0/116166543 >> v2:172.21.15.32:3300/0 conn(0x7f79340712c0 0x7f7934162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.103+0000 7fdf41c00640 1 Processor -- start 2024-03-06T07:29:08.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.103+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917275371 0 0) 0x7f79340ce370 con 0x7f7934072da0 2024-03-06T07:29:08.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.103+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7918003520 con 0x7f7934072da0 2024-03-06T07:29:08.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.103+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476446827 0 0) 0x7f79340ce090 con 0x7f7934070ee0 2024-03-06T07:29:08.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.103+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79340ce370 con 0x7f7934070ee0 2024-03-06T07:29:08.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.103+0000 7fdf41c00640 1 -- start start 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.104+0000 7f5d0a400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5d04070ee0 0x7f5d04162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.104+0000 7f5d0a400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d04072da0 0x7f5d040cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.104+0000 7f5d0a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d040ce090 con 0x7f5d040712c0 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.104+0000 7f5d0a400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d040ceca0 con 0x7f5d04072da0 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.104+0000 7f5d0a400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d040ce370 con 0x7f5d04070ee0 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7f5d03e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f5d040712c0 0x7f5d0406e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:51056/0 (socket says 172.21.15.32:51056) 2024-03-06T07:29:08.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7f5d03e00640 1 -- 172.21.15.32:0/1182591092 learned_addr learned my addr 172.21.15.32:0/1182591092 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7f7939000640 1 -- 172.21.15.32:0/116166543 >> v1:172.21.15.39:6789/0 conn(0x7f7934072da0 legacy=0x7f793406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7f7939000640 1 -- 172.21.15.32:0/116166543 >> v1:172.21.15.32:6789/0 conn(0x7f7934070ee0 legacy=0x7f793406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7f7939000640 1 -- 172.21.15.32:0/116166543 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79340ce650 con 0x7f79340712c0 2024-03-06T07:29:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7f5d02a00640 1 --2- 172.21.15.32:0/1182591092 >> v2:172.21.15.32:3300/0 conn(0x7f5d04072da0 0x7f5d040cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.105+0000 7fdf41c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fdf41c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdf3c161940 0x7fdf3c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fa79d000640 1 Processor -- start 2024-03-06T07:29:08.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fdf41c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf3c0c8d60 0x7fdf3c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7f5d03400640 1 --2- 172.21.15.32:0/1182591092 >> v2:172.21.15.39:3300/0 conn(0x7f5d04070ee0 0x7f5d04162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fdf3a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdf3c161940 0x7fdf3c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fdf3a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdf3c161940 0x7fdf3c070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54316/0 (socket says 172.21.15.32:54316) 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fdf3a000640 1 -- 172.21.15.32:0/1797701403 learned_addr learned my addr 172.21.15.32:0/1797701403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3c0730d0 con 0x7fdf3c0c8d60 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3c0cec80 con 0x7fdf3c161940 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c0cef60 con 0x7fdf3c06e240 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7fdf3b400640 1 --2- 172.21.15.32:0/1797701403 >> v2:172.21.15.32:3300/0 conn(0x7fdf3c0c8d60 0x7fdf3c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7f5d02a00640 1 -- 172.21.15.32:0/1182591092 >> v2:172.21.15.39:3300/0 conn(0x7f5d04070ee0 msgr2=0x7f5d04162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7f5d02a00640 1 --2- 172.21.15.32:0/1182591092 >> v2:172.21.15.39:3300/0 conn(0x7f5d04070ee0 0x7f5d04162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7f5d02a00640 1 -- 172.21.15.32:0/1182591092 >> v1:172.21.15.39:6789/0 conn(0x7f5d040712c0 legacy=0x7f5d0406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7f5d02a00640 1 -- 172.21.15.32:0/1182591092 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d040ce650 con 0x7f5d04072da0 2024-03-06T07:29:08.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.107+0000 7fdf3aa00640 1 --2- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.106+0000 7fa79d000640 1 -- start start 2024-03-06T07:29:08.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fdf3a000640 1 -- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 msgr2=0x7fdf3c161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fdf3a000640 1 --2- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7f7939000640 1 --2- 172.21.15.32:0/116166543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79340712c0 0x7f7934162c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f79340cee20 tx=0x7f792002b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f65676b773d5006 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920037070 con 0x7f79340712c0 2024-03-06T07:29:08.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7f3cb5000640 1 Processor -- start 2024-03-06T07:29:08.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7920030510 con 0x7f79340712c0 2024-03-06T07:29:08.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7f7932a00640 1 -- 172.21.15.32:0/116166543 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79240015a0 con 0x7f79340712c0 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fdf3a000640 1 -- 172.21.15.32:0/1797701403 >> v2:172.21.15.32:3300/0 conn(0x7fdf3c0c8d60 msgr2=0x7fdf3c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fdf3a000640 1 --2- 172.21.15.32:0/1797701403 >> v2:172.21.15.32:3300/0 conn(0x7fdf3c0c8d60 0x7fdf3c0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fdf3a000640 1 -- 172.21.15.32:0/1797701403 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c0cf240 con 0x7fdf3c161940 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fa79d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa798070ee0 0x7fa798162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fa79d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa798072da0 0x7fa7980cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fa79d000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7980ce090 con 0x7fa7980712c0 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fa79d000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7980ceca0 con 0x7fa798072da0 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.108+0000 7fa79d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7980ce370 con 0x7fa798070ee0 2024-03-06T07:29:08.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa798070ee0 0x7fa798162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa798072da0 0x7fa7980cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa798072da0 0x7fa7980cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50790/0 (socket says 172.21.15.32:50790) 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 -- 172.21.15.32:0/3352471779 learned_addr learned my addr 172.21.15.32:0/3352471779 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 -- 172.21.15.32:0/3352471779 >> v2:172.21.15.39:3300/0 conn(0x7fa798070ee0 msgr2=0x7fa798162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 --2- 172.21.15.32:0/3352471779 >> v2:172.21.15.39:3300/0 conn(0x7fa798070ee0 0x7fa798162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 -- 172.21.15.32:0/3352471779 >> v1:172.21.15.32:6789/0 conn(0x7fa7980712c0 legacy=0x7fa79806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.109+0000 7fa795400640 1 -- 172.21.15.32:0/3352471779 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7980ce650 con 0x7fa798072da0 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f3cb5000640 1 -- start start 2024-03-06T07:29:08.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f5d02a00640 1 --2- 172.21.15.32:0/1182591092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04072da0 0x7f5d040cf130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d04073610 tx=0x7f5cf402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947cf7f1df02e5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7fdf3a000640 1 --2- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c161940 0x7fdf3c070c70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0748c0 tx=0x7fdf3402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a7c577188b11a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f5d02000640 1 -- 172.21.15.32:0/1182591092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4035020 con 0x7f5d04072da0 2024-03-06T07:29:08.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f5d02000640 1 -- 172.21.15.32:0/1182591092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf4031830 con 0x7f5d04072da0 2024-03-06T07:29:08.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f5d02000640 1 -- 172.21.15.32:0/1182591092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4031c00 con 0x7f5d04072da0 2024-03-06T07:29:08.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f793b000640 1 -- 172.21.15.32:0/116166543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79340712c0 msgr2=0x7f7934162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.110+0000 7f793b000640 1 --2- 172.21.15.32:0/116166543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79340712c0 0x7f7934162c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f79340cee20 tx=0x7f792002b6f0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f793b000640 1 -- 172.21.15.32:0/116166543 shutdown_connections 2024-03-06T07:29:08.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f793b000640 1 --2- 172.21.15.32:0/116166543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79340712c0 0x7f7934162c40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f793b000640 1 -- 172.21.15.32:0/116166543 >> 172.21.15.32:0/116166543 conn(0x7f793406afb0 msgr2=0x7f793406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7fdf39600640 1 -- 172.21.15.32:0/1797701403 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34035020 con 0x7fdf3c161940 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7fdf39600640 1 -- 172.21.15.32:0/1797701403 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34031810 con 0x7fdf3c161940 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7fdf39600640 1 -- 172.21.15.32:0/1797701403 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34031c20 con 0x7fdf3c161940 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f5d0a400640 1 -- 172.21.15.32:0/1182591092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04072da0 msgr2=0x7f5d040cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f5d0a400640 1 --2- 172.21.15.32:0/1182591092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04072da0 0x7f5d040cf130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5d04073610 tx=0x7f5cf402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f5d0a400640 1 -- 172.21.15.32:0/1182591092 shutdown_connections 2024-03-06T07:29:08.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f5d0a400640 1 --2- 172.21.15.32:0/1182591092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04072da0 0x7f5d040cf130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f5d0a400640 1 --2- 172.21.15.32:0/1182591092 >> v2:172.21.15.39:3300/0 conn(0x7f5d04070ee0 0x7f5d04162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.111+0000 7f5d0a400640 1 -- 172.21.15.32:0/1182591092 >> 172.21.15.32:0/1182591092 conn(0x7f5d0406afb0 msgr2=0x7f5d0406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7f5d0a400640 1 -- 172.21.15.32:0/1182591092 shutdown_connections 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7f5d0a400640 1 -- 172.21.15.32:0/1182591092 wait complete. 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c161940 msgr2=0x7fdf3c070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7fdf41c00640 1 --2- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c161940 0x7fdf3c070c70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0748c0 tx=0x7fdf3402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7fa795400640 1 --2- 172.21.15.32:0/3352471779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa798072da0 0x7fa7980cf130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa798073610 tx=0x7fa78403c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2bdd396ae32231 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7f3cb5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3cb015e120 0x7f3cb0070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7f3cb5000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00b2dd0 0x7f3cb00711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.112+0000 7f3cb5000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cb00716f0 0x7f3cb0161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cb5000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb00cdc20 con 0x7f3cb00716f0 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cb5000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb006d180 con 0x7f3cb015e120 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cb5000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb00cde70 con 0x7f3cb00b2dd0 2024-03-06T07:29:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f793b000640 1 -- 172.21.15.32:0/116166543 shutdown_connections 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 shutdown_connections 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 --2- 172.21.15.32:0/1797701403 >> v2:172.21.15.32:3300/0 conn(0x7fdf3c0c8d60 0x7fdf3c0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 --2- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c161940 0x7fdf3c070c70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 --2- 172.21.15.32:0/1797701403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 >> 172.21.15.32:0/1797701403 conn(0x7fdf3c06afb0 msgr2=0x7fdf3c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fa794a00640 1 -- 172.21.15.32:0/3352471779 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784046020 con 0x7fa798072da0 2024-03-06T07:29:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fa794a00640 1 -- 172.21.15.32:0/3352471779 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa784042880 con 0x7fa798072da0 2024-03-06T07:29:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cae800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cb00716f0 0x7f3cb0161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cae800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cb00716f0 0x7f3cb0161dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50798/0 (socket says 172.21.15.32:50798) 2024-03-06T07:29:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cae800640 1 -- 172.21.15.32:0/1215296406 learned_addr learned my addr 172.21.15.32:0/1215296406 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cad400640 1 --2- 172.21.15.32:0/1215296406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00b2dd0 0x7f3cb00711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f3cade00640 1 --2- 172.21.15.32:0/1215296406 >> v2:172.21.15.39:3300/0 conn(0x7f3cb015e120 0x7f3cb0070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f5d0a400640 1 Processor -- start 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 shutdown_connections 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7fa79d000640 1 -- 172.21.15.32:0/3352471779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa798072da0 msgr2=0x7fa7980cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7fa79d000640 1 --2- 172.21.15.32:0/3352471779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa798072da0 0x7fa7980cf130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa798073610 tx=0x7fa78403c920 comp rx=0 tx=0).stop 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7fa79d000640 1 -- 172.21.15.32:0/3352471779 shutdown_connections 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7fa79d000640 1 --2- 172.21.15.32:0/3352471779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa798072da0 0x7fa7980cf130 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa798073610 tx=0x7fa78403c920 comp rx=0 tx=0).stop 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7fa79d000640 1 --2- 172.21.15.32:0/3352471779 >> v2:172.21.15.39:3300/0 conn(0x7fa798070ee0 0x7fa798162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7fa79d000640 1 -- 172.21.15.32:0/3352471779 >> 172.21.15.32:0/3352471779 conn(0x7fa79806afb0 msgr2=0x7fa79806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f3cae800640 1 -- 172.21.15.32:0/1215296406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00b2dd0 msgr2=0x7f3cb00711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f3cae800640 1 --2- 172.21.15.32:0/1215296406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00b2dd0 0x7f3cb00711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f3cae800640 1 -- 172.21.15.32:0/1215296406 >> v2:172.21.15.39:3300/0 conn(0x7f3cb015e120 msgr2=0x7f3cb0070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f3cae800640 1 --2- 172.21.15.32:0/1215296406 >> v2:172.21.15.39:3300/0 conn(0x7f3cb015e120 0x7f3cb0070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f3cae800640 1 -- 172.21.15.32:0/1215296406 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00ce150 con 0x7f3cb00716f0 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7fdf41c00640 1 -- 172.21.15.32:0/1797701403 wait complete. 2024-03-06T07:29:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.114+0000 7f5d0a400640 1 -- start start 2024-03-06T07:29:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.113+0000 7f793b000640 1 -- 172.21.15.32:0/116166543 wait complete. 2024-03-06T07:29:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.115+0000 7f793b000640 1 Processor -- start 2024-03-06T07:29:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f793b000640 1 -- start start 2024-03-06T07:29:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.115+0000 7f3cae800640 1 --2- 172.21.15.32:0/1215296406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb00716f0 0x7f3cb0161dd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00692c0 tx=0x7f3ca802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5cc49ca16ff9c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f5d0a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04070ee0 0x7f5d0414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7fdf41c00640 1 Processor -- start 2024-03-06T07:29:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f5d0a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d040712c0 0x7f5d041ec070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f3caca00640 1 -- 172.21.15.32:0/1215296406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca802c950 con 0x7f3cb00716f0 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f3caca00640 1 -- 172.21.15.32:0/1215296406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca802caf0 con 0x7f3cb00716f0 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f3caca00640 1 -- 172.21.15.32:0/1215296406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80343f0 con 0x7f3cb00716f0 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f5d0a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d041ec5b0 0x7f5d040b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f5d0a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d040cdda0 con 0x7f5d04070ee0 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f5d0a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d040cdc20 con 0x7f5d040712c0 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7f5d0a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0406e7d0 con 0x7f5d041ec5b0 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7fa79d000640 1 -- 172.21.15.32:0/3352471779 shutdown_connections 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7fa79d000640 1 -- 172.21.15.32:0/3352471779 wait complete. 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f5d03400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04070ee0 0x7f5d0414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f5d03400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04070ee0 0x7f5d0414d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50812/0 (socket says 172.21.15.32:50812) 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f5d03400640 1 -- 172.21.15.32:0/1206894945 learned_addr learned my addr 172.21.15.32:0/1206894945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f5d03e00640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d041ec5b0 0x7f5d040b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f3cb5000640 1 -- 172.21.15.32:0/1215296406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb00716f0 msgr2=0x7f3cb0161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f3cb5000640 1 --2- 172.21.15.32:0/1215296406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb00716f0 0x7f3cb0161dd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00692c0 tx=0x7f3ca802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f5d02a00640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d040712c0 0x7f5d041ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.116+0000 7fdf41c00640 1 -- start start 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f3cb5000640 1 -- 172.21.15.32:0/1215296406 shutdown_connections 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.117+0000 7f3cb5000640 1 --2- 172.21.15.32:0/1215296406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb00716f0 0x7f3cb0161dd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f3cb5000640 1 --2- 172.21.15.32:0/1215296406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00b2dd0 0x7f3cb00711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f3cb5000640 1 --2- 172.21.15.32:0/1215296406 >> v2:172.21.15.39:3300/0 conn(0x7f3cb015e120 0x7f3cb0070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f3cb5000640 1 -- 172.21.15.32:0/1215296406 >> 172.21.15.32:0/1215296406 conn(0x7f3cb006afb0 msgr2=0x7f3cb006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f793b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7934070ee0 0x7f793409d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f54e1a00640 1 Processor -- start 2024-03-06T07:29:08.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f793b000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79340712c0 0x7f7934152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f793b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7934072da0 0x7f79340ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf41c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f793b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79340cdda0 con 0x7f7934072da0 2024-03-06T07:29:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f5d03400640 1 -- 172.21.15.32:0/1206894945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d041ec5b0 msgr2=0x7f5d040b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f5d03400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d041ec5b0 0x7f5d040b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f5d03400640 1 -- 172.21.15.32:0/1206894945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d040712c0 msgr2=0x7f5d041ec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f793b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79340cdc20 con 0x7f7934070ee0 2024-03-06T07:29:08.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f793b000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f793406e7d0 con 0x7f79340712c0 2024-03-06T07:29:08.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f5d03400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d040712c0 0x7f5d041ec070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7f5d03400640 1 -- 172.21.15.32:0/1206894945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0415ea70 con 0x7f5d04070ee0 2024-03-06T07:29:08.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7933e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7934070ee0 0x7f793409d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf41c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf3c161940 0x7fdf3c158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf41c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c0cb900 0x7fdf3c0c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf41c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c0ce780 con 0x7fdf3c161940 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf41c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c0ce600 con 0x7fdf3c0cb900 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf41c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c06e940 con 0x7fdf3c06e240 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.118+0000 7fdf3aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7fdf3aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c1581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:32868/0 (socket says 172.21.15.32:32868) 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7fdf3aa00640 1 -- 172.21.15.32:0/1148991841 learned_addr learned my addr 172.21.15.32:0/1148991841 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7933e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7934070ee0 0x7f793409d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54348/0 (socket says 172.21.15.32:54348) 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7fdf3a000640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf3c161940 0x7fdf3c158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7933e00640 1 -- 172.21.15.32:0/1578595136 learned_addr learned my addr 172.21.15.32:0/1578595136 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7939000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7934072da0 0x7f79340ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7fdf3b400640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c0cb900 0x7fdf3c0c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7fa79d000640 1 Processor -- start 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7939000640 1 -- 172.21.15.32:0/1578595136 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79340712c0 msgr2=0x7f7934152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fa79d000640 1 -- start start 2024-03-06T07:29:08.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7939000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79340712c0 0x7f7934152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7939000640 1 -- 172.21.15.32:0/1578595136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7934070ee0 msgr2=0x7f793409d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7939000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7934070ee0 0x7f793409d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fdf3aa00640 1 -- 172.21.15.32:0/1148991841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c0cb900 msgr2=0x7fdf3c0c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fdf3aa00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf3c0cb900 0x7fdf3c0c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fdf3aa00640 1 -- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf3c161940 msgr2=0x7fdf3c158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fdf3aa00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf3c161940 0x7fdf3c158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fdf3aa00640 1 -- 172.21.15.32:0/1148991841 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c0b1580 con 0x7fdf3c06e240 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f7939000640 1 -- 172.21.15.32:0/1578595136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793409c180 con 0x7f7934072da0 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7f7939000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7934072da0 0x7f79340ae060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7920032000 tx=0x7f79200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.120+0000 7fdf3aa00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c1581c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c007e60 tx=0x7fdf2c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f3cb5000640 1 -- 172.21.15.32:0/1215296406 shutdown_connections 2024-03-06T07:29:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920040020 con 0x7f7934072da0 2024-03-06T07:29:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7920038040 con 0x7f7934072da0 2024-03-06T07:29:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920037070 con 0x7f7934072da0 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.119+0000 7f54e1a00640 1 -- start start 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f54e1a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f54dc0712c0 0x7f54dc162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f54e1a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0ce090 con 0x7f54dc070ee0 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f54e1a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0ce370 con 0x7f54dc072da0 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f54e1a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ceca0 con 0x7f54dc0712c0 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c014620 con 0x7fdf3c06e240 2024-03-06T07:29:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7fdf41c00640 1 -- 172.21.15.32:0/1148991841 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c0b1860 con 0x7fdf3c06e240 2024-03-06T07:29:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f5d03400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04070ee0 0x7f5d0414d0b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00e9e0 tx=0x7f5cec00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fa79d000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa798070ee0 0x7fa798155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fa79d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7980712c0 0x7fa7980b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fa79d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa798150700 0x7fa7980b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7f54da000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f54dc072da0 0x7f54dc06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:51064/0 (socket says 172.21.15.32:51064) 2024-03-06T07:29:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7f54da000640 1 -- 172.21.15.32:0/1276635280 learned_addr learned my addr 172.21.15.32:0/1276635280 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec017070 con 0x7f5d04070ee0 2024-03-06T07:29:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec00c9d0 con 0x7f5d04070ee0 2024-03-06T07:29:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec00cdc0 con 0x7f5d04070ee0 2024-03-06T07:29:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7fdf41c00640 1 -- 172.21.15.32:0/1148991841 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c0cb100 con 0x7fdf3c06e240 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fa795400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7980712c0 0x7fa7980b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fa795400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7980712c0 0x7fa7980b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50838/0 (socket says 172.21.15.32:50838) 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fa795400640 1 -- 172.21.15.32:0/975462802 learned_addr learned my addr 172.21.15.32:0/975462802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980cdda0 con 0x7fa7980712c0 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7fa795e00640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa798070ee0 0x7fa798155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980cdc20 con 0x7fa798150700 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79806e7d0 con 0x7fa798070ee0 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7fa796800640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa798150700 0x7fa7980b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7f54db400640 1 --2- 172.21.15.32:0/1276635280 >> v2:172.21.15.39:3300/0 conn(0x7f54dc0712c0 0x7f54dc162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0415ed50 con 0x7f5d04070ee0 2024-03-06T07:29:08.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0415f310 con 0x7f5d04070ee0 2024-03-06T07:29:08.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793409c460 con 0x7f7934072da0 2024-03-06T07:29:08.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793409cd00 con 0x7f7934072da0 2024-03-06T07:29:08.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c019070 con 0x7fdf3c06e240 2024-03-06T07:29:08.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.122+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c014a00 con 0x7fdf3c06e240 2024-03-06T07:29:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7fdf41c00640 1 -- 172.21.15.32:0/1148991841 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf14001030 con 0x7fdf3c06e240 2024-03-06T07:29:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7fba19200640 1 Processor -- start 2024-03-06T07:29:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7f54d9600640 1 -- 172.21.15.32:0/1276635280 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945286290 0 0) 0x7f54dc0ce090 con 0x7f54dc070ee0 2024-03-06T07:29:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7f54d9600640 1 -- 172.21.15.32:0/1276635280 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c0003520 con 0x7f54dc070ee0 2024-03-06T07:29:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7f89a9a00640 1 Processor -- start 2024-03-06T07:29:08.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7f89a9a00640 1 -- start start 2024-03-06T07:29:08.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7fa795e00640 1 -- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa798150700 msgr2=0x7fa7980b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.121+0000 7f3cb5000640 1 -- 172.21.15.32:0/1215296406 wait complete. 2024-03-06T07:29:08.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.123+0000 7f3cb5000640 1 Processor -- start 2024-03-06T07:29:08.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f793404f3a0 con 0x7f7934072da0 2024-03-06T07:29:08.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7fa795e00640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa798150700 0x7fa7980b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7fa795e00640 1 -- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7980712c0 msgr2=0x7fa7980b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7fa795e00640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7980712c0 0x7fa7980b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7f3cb5000640 1 -- start start 2024-03-06T07:29:08.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fa795e00640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798155d80 con 0x7fa798070ee0 2024-03-06T07:29:08.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7f54db400640 1 -- 172.21.15.32:0/1276635280 >> v1:172.21.15.39:6789/0 conn(0x7f54dc072da0 legacy=0x7f54dc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7f54db400640 1 -- 172.21.15.32:0/1276635280 >> v1:172.21.15.32:6789/0 conn(0x7f54dc070ee0 legacy=0x7f54dc06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.125+0000 7f54db400640 1 -- 172.21.15.32:0/1276635280 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0ce650 con 0x7f54dc0712c0 2024-03-06T07:29:08.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fa796800640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa798150700 0x7fa7980b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.124+0000 7fba19200640 1 -- start start 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fba19200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fba1415e2e0 0x7fba140d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fba19200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba140d0d70 0x7fba140d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fba19200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140b3940 con 0x7fba140d1b30 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fba19200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba140d4e70 con 0x7fba140d0d70 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fba19200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba140b3c40 con 0x7fba1415e2e0 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf2c014ba0 con 0x7fdf3c06e240 2024-03-06T07:29:08.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7fa795e00640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa798070ee0 0x7fa798155840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa79000d5f0 tx=0x7fa79000dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a415a210 0x7f89a40cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.126+0000 7f89a9a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89a406d440 0x7f89a40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f89a9a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a406d900 con 0x7f89a406e5e0 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f89a9a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a406dbe0 con 0x7f89a406d440 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f89a9a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40cddc0 con 0x7f89a415a210 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f89a2000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89a406e5e0 0x7f89a406cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56602/0 (socket says 172.21.15.32:56602) 2024-03-06T07:29:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f54db400640 1 --2- 172.21.15.32:0/1276635280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 0x7f54dc162c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0cee20 tx=0x7f54d402bd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9806a3f357d3665f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f89a2000640 1 -- 172.21.15.32:0/3340200079 learned_addr learned my addr 172.21.15.32:0/3340200079 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790018020 con 0x7fa798070ee0 2024-03-06T07:29:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa790014620 con 0x7fa798070ee0 2024-03-06T07:29:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790014a10 con 0x7fa798070ee0 2024-03-06T07:29:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fba12800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fba140d1b30 0x7fba140d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:51078/0 (socket says 172.21.15.32:51078) 2024-03-06T07:29:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fba0be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba140d0d70 0x7fba140d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fba0be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba140d0d70 0x7fba140d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50854/0 (socket says 172.21.15.32:50854) 2024-03-06T07:29:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fba0be00640 1 -- 172.21.15.32:0/2658613117 learned_addr learned my addr 172.21.15.32:0/2658613117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7f89a3400640 1 --2- 172.21.15.32:0/3340200079 >> v2:172.21.15.39:3300/0 conn(0x7f89a406d440 0x7f89a40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.127+0000 7fba11e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fba1415e2e0 0x7fba140d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d0404f3a0 con 0x7f5d04070ee0 2024-03-06T07:29:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f89a1600640 1 -- 172.21.15.32:0/3340200079 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058713580 0 0) 0x7f89a406d900 con 0x7f89a406e5e0 2024-03-06T07:29:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f89a1600640 1 -- 172.21.15.32:0/3340200079 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8988003520 con 0x7f89a406e5e0 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fba0be00640 1 -- 172.21.15.32:0/2658613117 >> v2:172.21.15.39:3300/0 conn(0x7fba1415e2e0 msgr2=0x7fba140d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fdf32a00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf1806b500 0x7fdf1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fba0be00640 1 --2- 172.21.15.32:0/2658613117 >> v2:172.21.15.39:3300/0 conn(0x7fba1415e2e0 0x7fba140d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fba0be00640 1 -- 172.21.15.32:0/2658613117 >> v1:172.21.15.39:6789/0 conn(0x7fba140d1b30 legacy=0x7fba140d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fa795400640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7980712c0 0x7fa7980b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa798151be0 con 0x7fa798070ee0 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fba0be00640 1 -- 172.21.15.32:0/2658613117 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140cf580 con 0x7fba140d0d70 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f3cb5000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00716f0 0x7f3cb0149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f89a3400640 1 -- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a415a210 msgr2=0x7f89a40cbfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f89a3400640 1 --2- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a415a210 0x7f89a40cbfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f89a3400640 1 -- 172.21.15.32:0/3340200079 >> v1:172.21.15.32:6789/0 conn(0x7f89a406e5e0 legacy=0x7f89a406cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7f89a3400640 1 -- 172.21.15.32:0/3340200079 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40acdb0 con 0x7f89a406d440 2024-03-06T07:29:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa798160c70 con 0x7fa798070ee0 2024-03-06T07:29:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f3cb5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cb00b2dd0 0x7f3cb01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.128+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf2c08b7a0 con 0x7fdf3c06e240 2024-03-06T07:29:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7fdf32a00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7fdf18073780 0x7fdf18075ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf2c012e50 con 0x7fdf18073780 2024-03-06T07:29:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf2c08bb90 con 0x7fdf3c06e240 2024-03-06T07:29:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f3cb5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb015e120 0x7f3cb00b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f3cb5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb006df00 con 0x7f3cb015e120 2024-03-06T07:29:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cec01a070 con 0x7f5d04070ee0 2024-03-06T07:29:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f3cb5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb006e3f0 con 0x7f3cb00b2dd0 2024-03-06T07:29:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f3cb5000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb015e500 con 0x7f3cb00716f0 2024-03-06T07:29:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.129+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7920043020 con 0x7f7934072da0 2024-03-06T07:29:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7f3cade00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00716f0 0x7f3cb0149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7fdf3b400640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7fdf18073780 0x7fdf18075ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7f3cade00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00716f0 0x7f3cb0149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:32892/0 (socket says 172.21.15.32:32892) 2024-03-06T07:29:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7f3cade00640 1 -- 172.21.15.32:0/2424205511 learned_addr learned my addr 172.21.15.32:0/2424205511 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7fba11e00640 1 --2- 172.21.15.32:0/2658613117 >> v2:172.21.15.39:3300/0 conn(0x7fba1415e2e0 0x7fba140d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7f3cae800640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb015e120 0x7f3cb00b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.130+0000 7f3cad400640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cb00b2dd0 0x7f3cb01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f3cae800640 1 -- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00716f0 msgr2=0x7f3cb0149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f3cae800640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3cb00716f0 0x7f3cb0149690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f3cae800640 1 -- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cb00b2dd0 msgr2=0x7f3cb01573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f3cae800640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3cb00b2dd0 0x7f3cb01573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f3cae800640 1 -- 172.21.15.32:0/2424205511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0149bd0 con 0x7f3cb015e120 2024-03-06T07:29:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f54d9600640 1 -- 172.21.15.32:0/1276635280 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4039070 con 0x7f54dc0712c0 2024-03-06T07:29:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f54d9600640 1 -- 172.21.15.32:0/1276635280 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d402c870 con 0x7f54dc0712c0 2024-03-06T07:29:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f54d9600640 1 -- 172.21.15.32:0/1276635280 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc001670 con 0x7f54dc0712c0 2024-03-06T07:29:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa798151ec0 con 0x7fa798070ee0 2024-03-06T07:29:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f792be00640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f791406b5a0 0x7f791406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79200abba0 con 0x7f7934072da0 2024-03-06T07:29:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7fba0be00640 1 --2- 172.21.15.32:0/2658613117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba140d11d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba1408d8b0 tx=0x7fb9fc02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932661a22d10814f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f792be00640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f79140737c0 0x7f7914075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7920030510 con 0x7f79140737c0 2024-03-06T07:29:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79200abf90 con 0x7f7934072da0 2024-03-06T07:29:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f5cfb400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5ce006b4d0 0x7f5ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cec08a5d0 con 0x7f5d04070ee0 2024-03-06T07:29:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7f3cae800640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb015e120 0x7f3cb00b0610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3cb006c3a0 tx=0x7f3ca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7fba11400640 1 -- 172.21.15.32:0/2658613117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc034070 con 0x7fba140d0d70 2024-03-06T07:29:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f54e1a00640 1 -- 172.21.15.32:0/1276635280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 msgr2=0x7f54dc162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f54e1a00640 1 --2- 172.21.15.32:0/1276635280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 0x7f54dc162c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0cee20 tx=0x7f54d402bd40 comp rx=0 tx=0).stop 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f89a3400640 1 --2- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a406d440 0x7f89a40ce5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f899c009c80 tx=0x7f899c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a8d58070102d45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f7933e00640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f791406b5a0 0x7f791406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7fba11400640 1 -- 172.21.15.32:0/2658613117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc030870 con 0x7fba140d0d70 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7fba11400640 1 -- 172.21.15.32:0/2658613117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc030c40 con 0x7fba140d0d70 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8043020 con 0x7f3cb015e120 2024-03-06T07:29:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.131+0000 7fdf3a000640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf1806b500 0x7fdf1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7fdf3b400640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7fdf18073780 0x7fdf18075ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca803e450 con 0x7f3cb015e120 2024-03-06T07:29:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca803e880 con 0x7f3cb015e120 2024-03-06T07:29:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7fdf3a000640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf1806b500 0x7fdf1806d9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf34036000 tx=0x7fdf34008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f5cfb400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f5ce0073670 0x7f5ce0075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec016ec0 con 0x7f5ce0073670 2024-03-06T07:29:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cec091050 con 0x7f5d04070ee0 2024-03-06T07:29:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f89a1600640 1 -- 172.21.15.32:0/3340200079 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c039070 con 0x7f89a406d440 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f5d02a00640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5ce006b4d0 0x7f5ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f89a1600640 1 -- 172.21.15.32:0/3340200079 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c02cc00 con 0x7f89a406d440 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f7933400640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f79140737c0 0x7f7914075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f89a1600640 1 -- 172.21.15.32:0/3340200079 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c034440 con 0x7f89a406d440 2024-03-06T07:29:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f5d03e00640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f5ce0073670 0x7f5ce0075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f54e1a00640 1 -- 172.21.15.32:0/1276635280 shutdown_connections 2024-03-06T07:29:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f54e1a00640 1 --2- 172.21.15.32:0/1276635280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 0x7f54dc162c40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f54e1a00640 1 -- 172.21.15.32:0/1276635280 >> 172.21.15.32:0/1276635280 conn(0x7f54dc06afb0 msgr2=0x7f54dc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.132+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb0156330 con 0x7f3cb015e120 2024-03-06T07:29:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb01568f0 con 0x7f3cb015e120 2024-03-06T07:29:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7fba19200640 1 -- 172.21.15.32:0/2658613117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 msgr2=0x7fba140d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7fba19200640 1 --2- 172.21.15.32:0/2658613117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba140d11d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba1408d8b0 tx=0x7fb9fc02b9b0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7fba19200640 1 -- 172.21.15.32:0/2658613117 shutdown_connections 2024-03-06T07:29:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== osd.2 v2:172.21.15.32:6800/1600044506 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdf2c012e50 con 0x7fdf18073780 2024-03-06T07:29:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7fba19200640 1 --2- 172.21.15.32:0/2658613117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba140d11d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7fba19200640 1 --2- 172.21.15.32:0/2658613117 >> v2:172.21.15.39:3300/0 conn(0x7fba1415e2e0 0x7fba140d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.133+0000 7f54e1a00640 1 -- 172.21.15.32:0/1276635280 shutdown_connections 2024-03-06T07:29:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7fba19200640 1 -- 172.21.15.32:0/2658613117 >> 172.21.15.32:0/2658613117 conn(0x7fba140d3f60 msgr2=0x7fba140d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7f54e1a00640 1 -- 172.21.15.32:0/1276635280 wait complete. 2024-03-06T07:29:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7f5d03e00640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f5ce0073670 0x7f5ce0075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7f5d02a00640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5ce006b4d0 0x7f5ce006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4036000 tx=0x7f5cf4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7f89a9a00640 1 -- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a406d440 msgr2=0x7f89a40ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7f89a9a00640 1 --2- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a406d440 0x7f89a40ce5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f899c009c80 tx=0x7f899c02be50 comp rx=0 tx=0).stop 2024-03-06T07:29:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.135+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3cb004f3a0 con 0x7f3cb015e120 2024-03-06T07:29:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.135+0000 7fba19200640 1 -- 172.21.15.32:0/2658613117 shutdown_connections 2024-03-06T07:29:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.135+0000 7fba19200640 1 -- 172.21.15.32:0/2658613117 wait complete. 2024-03-06T07:29:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.134+0000 7f7933400640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f79140737c0 0x7f7914075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.135+0000 7f7933e00640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f791406b5a0 0x7f791406da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f792c001560 tx=0x7f792c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5cec016ec0 con 0x7f5ce0073670 2024-03-06T07:29:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== osd.1 v2:172.21.15.32:6817/50998742 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7920030510 con 0x7f79140737c0 2024-03-06T07:29:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f89a9a00640 1 -- 172.21.15.32:0/3340200079 shutdown_connections 2024-03-06T07:29:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f89a9a00640 1 --2- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a406d440 0x7f89a40ce5c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f89a9a00640 1 --2- 172.21.15.32:0/3340200079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a415a210 0x7f89a40cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f89a9a00640 1 -- 172.21.15.32:0/3340200079 >> 172.21.15.32:0/3340200079 conn(0x7f89a406afb0 msgr2=0x7f89a406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f54e1a00640 1 Processor -- start 2024-03-06T07:29:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f89a9a00640 1 -- 172.21.15.32:0/3340200079 shutdown_connections 2024-03-06T07:29:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f54e1a00640 1 -- start start 2024-03-06T07:29:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.137+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa79001b020 con 0x7fa798070ee0 2024-03-06T07:29:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.137+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ca8046020 con 0x7f3cb015e120 2024-03-06T07:29:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7fa78e000640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa77806b500 0x7fa77806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa79008aaa0 con 0x7fa798070ee0 2024-03-06T07:29:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7fa78e000640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7fa778073720 0x7fa778075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7780761f0 con 0x7fa778073720 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa79008ae90 con 0x7fa798070ee0 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7fba19200640 1 Processor -- start 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7fba19200640 1 -- start start 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7f54e1a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54dc070ee0 0x7f54dc0ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7f54e1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 0x7f54dc15f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7f54e1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54dc072da0 0x7f54dc152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7f54e1a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0cdda0 con 0x7f54dc072da0 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7f54e1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0cdc20 con 0x7f54dc0712c0 2024-03-06T07:29:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.138+0000 7f54e1a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc06e7d0 con 0x7f54dc070ee0 2024-03-06T07:29:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54daa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54dc070ee0 0x7f54dc0ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54dc072da0 0x7f54dc152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54dc072da0 0x7f54dc152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50870/0 (socket says 172.21.15.32:50870) 2024-03-06T07:29:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 -- 172.21.15.32:0/2669272048 learned_addr learned my addr 172.21.15.32:0/2669272048 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.136+0000 7f89a9a00640 1 -- 172.21.15.32:0/3340200079 wait complete. 2024-03-06T07:29:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f89a9a00640 1 Processor -- start 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54da000640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 0x7f54dc15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 -- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54dc070ee0 msgr2=0x7f54dc0ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54dc070ee0 0x7f54dc0ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 -- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 msgr2=0x7f54dc15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54dc0712c0 0x7f54dc15f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f54db400640 1 -- 172.21.15.32:0/2669272048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc160c20 con 0x7f54dc072da0 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.140+0000 7f89a9a00640 1 -- start start 2024-03-06T07:29:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f3ca6000640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.140+0000 7fa795400640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa77806b500 0x7fa77806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ca80ac880 con 0x7f3cb015e120 2024-03-06T07:29:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.139+0000 7f3ca6000640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f3c8c073670 0x7f3c8c075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.140+0000 7fa796800640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7fa778073720 0x7fa778075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.140+0000 7f54db400640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54dc072da0 0x7f54dc152a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009e80 tx=0x7f54d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c8c076140 con 0x7f3c8c073670 2024-03-06T07:29:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7f3cade00640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7fa796800640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7fa778073720 0x7fa778075b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7fba19200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba1415ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7fba19200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba140d1b30 0x7fba140cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7fba11e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba1415ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7fa795400640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa77806b500 0x7fa77806d9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7980b15d0 tx=0x7fa784008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7fba11e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba1415ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50872/0 (socket says 172.21.15.32:50872) 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba19200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba1415e2e0 0x7fba1414b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba11e00640 1 -- 172.21.15.32:0/1063525822 learned_addr learned my addr 172.21.15.32:0/1063525822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140cfe20 con 0x7fba140d0d70 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140cfb40 con 0x7fba140d1b30 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba140b50e0 con 0x7fba1415e2e0 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4044020 con 0x7f54dc072da0 2024-03-06T07:29:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba0be00640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba140d1b30 0x7fba140cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7fba12800640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba1415e2e0 0x7fba1414b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ca8077340 con 0x7f3cb015e120 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.141+0000 7f3cad400640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f3c8c073670 0x7f3c8c075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7f3cad400640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f3c8c073670 0x7f3c8c075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.142+0000 7f3cade00640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009c00 tx=0x7f3c98009190 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d403a040 con 0x7f54dc072da0 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4039070 con 0x7f54dc072da0 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== osd.3 v2:172.21.15.32:6806/238384581 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa7780761f0 con 0x7fa778073720 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7fba11e00640 1 -- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba1415e2e0 msgr2=0x7fba1414b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7fba11e00640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba1415e2e0 0x7fba1414b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7fba11e00640 1 -- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba140d1b30 msgr2=0x7fba140cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7fba11e00640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba140d1b30 0x7fba140cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7fba11e00640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1415f260 con 0x7fba140d0d70 2024-03-06T07:29:08.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7fba11e00640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba1415ec40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fba0400d7d0 tx=0x7fba0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc160f00 con 0x7f54dc072da0 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc1614c0 con 0x7f54dc072da0 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a406d440 0x7f89a4154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a406e5e0 0x7f89a41e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a415a210 0x7f89a409cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f89a9a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a406bc20 con 0x7f89a406d440 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f89a9a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a406baa0 con 0x7f89a415a210 2024-03-06T07:29:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.143+0000 7f89a9a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a415a5f0 con 0x7f89a406e5e0 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54dc04f3a0 con 0x7f54dc072da0 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7f89a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a406d440 0x7f89a4154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7f89a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a406d440 0x7f89a4154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50880/0 (socket says 172.21.15.32:50880) 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7f89a2a00640 1 -- 172.21.15.32:0/1335422319 learned_addr learned my addr 172.21.15.32:0/1335422319 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7f89a2000640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a406e5e0 0x7f89a41e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.144+0000 7f89a3400640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a415a210 0x7f89a409cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7f89a2000640 1 -- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a415a210 msgr2=0x7f89a409cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7f89a2000640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89a415a210 0x7f89a409cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7f89a2000640 1 -- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a406d440 msgr2=0x7f89a4154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7f89a2000640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89a406d440 0x7f89a4154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7f89a2000640 1 -- 172.21.15.32:0/1335422319 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4156ec0 con 0x7f89a406e5e0 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04004030 con 0x7fba140d0d70 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1415f540 con 0x7fba140d0d70 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.145+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1415fb00 con 0x7fba140d0d70 2024-03-06T07:29:08.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.146+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== osd.4 v2:172.21.15.39:6800/89256438 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c8c076140 con 0x7f3c8c073670 2024-03-06T07:29:08.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.146+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba040041d0 con 0x7fba140d0d70 2024-03-06T07:29:08.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.146+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba040096a0 con 0x7fba140d0d70 2024-03-06T07:29:08.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.146+0000 7f89a2000640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a406e5e0 0x7f89a41e8140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f899000c430 tx=0x7f899000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.146+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9ec001030 con 0x7fba140d0d70 2024-03-06T07:29:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.146+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990017070 con 0x7f89a406e5e0 2024-03-06T07:29:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.147+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8990005ce0 con 0x7f89a406e5e0 2024-03-06T07:29:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.147+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89900072f0 con 0x7f89a406e5e0 2024-03-06T07:29:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.147+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a41571a0 con 0x7f89a406e5e0 2024-03-06T07:29:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.147+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a409ac60 con 0x7f89a406e5e0 2024-03-06T07:29:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.148+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f897c001030 con 0x7f89a406e5e0 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.148+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54d4047020 con 0x7f54dc072da0 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.152+0000 7f54d2a00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54bc06b520 0x7f54bc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54d40acfd0 con 0x7f54dc072da0 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f54daa00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54bc06b520 0x7f54bc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f54d2a00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f54bc073740 0x7f54bc075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4006b90 con 0x7f54bc073740 2024-03-06T07:29:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba04009840 con 0x7fba140d0d70 2024-03-06T07:29:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.151+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f899001a070 con 0x7f89a406e5e0 2024-03-06T07:29:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f899aa00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898406b4d0 0x7f898406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8990089c60 con 0x7f89a406e5e0 2024-03-06T07:29:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f899aa00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x7f8984073670 0x7f8984075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8990017e70 con 0x7f8984073670 2024-03-06T07:29:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f899008a010 con 0x7f89a406e5e0 2024-03-06T07:29:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.153+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54d40ad440 con 0x7f54dc072da0 2024-03-06T07:29:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.154+0000 7f54daa00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54bc06b520 0x7f54bc06d9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54cc001eb0 tx=0x7f54cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7fba0a000640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9f006b4d0 0x7fb9f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7f89a2a00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898406b4d0 0x7f898406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7f54da000640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f54bc073740 0x7f54bc075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba0408afe0 con 0x7fba140d0d70 2024-03-06T07:29:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7fba0a000640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7fb9f00736f0 0x7fb9f0075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba04008f10 con 0x7fb9f00736f0 2024-03-06T07:29:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.155+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba0408b3d0 con 0x7fba140d0d70 2024-03-06T07:29:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.156+0000 7fba0be00640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9f006b4d0 0x7fb9f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.156+0000 7f89a3400640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x7f8984073670 0x7f8984075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.156+0000 7f89a2a00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898406b4d0 0x7f898406d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89940099a0 tx=0x7f89940090e0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.156+0000 7fba12800640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7fb9f00736f0 0x7fb9f0075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.156+0000 7f54da000640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f54bc073740 0x7f54bc075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.157+0000 7fba0be00640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9f006b4d0 0x7fb9f006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc036000 tx=0x7fb9fc03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.157+0000 7f89a3400640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x7f8984073670 0x7f8984075a90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.158+0000 7fba12800640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7fb9f00736f0 0x7fb9f0075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.159+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== osd.6 v2:172.21.15.39:6808/2944464285 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f54d4006b90 con 0x7f54bc073740 2024-03-06T07:29:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.160+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== osd.7 v2:172.21.15.39:6823/3019464823 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8990017e70 con 0x7f8984073670 2024-03-06T07:29:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.161+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d040b52d0 con 0x7f5ce0073670 2024-03-06T07:29:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.162+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== osd.5 v2:172.21.15.39:6816/969666024 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fba04008f10 con 0x7fb9f00736f0 2024-03-06T07:29:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.162+0000 7f5cfb400640 1 -- 172.21.15.32:0/1206894945 <== osd.0 v2:172.21.15.32:6816/3728617102 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5d040b52d0 con 0x7f5ce0073670 2024-03-06T07:29:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.163+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f5ce0073670 msgr2=0x7f5ce0075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.163+0000 7f5d0a400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f5ce0073670 0x7f5ce0075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5ce006b4d0 msgr2=0x7f5ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d0a400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5ce006b4d0 0x7f5ce006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4036000 tx=0x7f5cf4008030 comp rx=0 tx=0).stop 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04070ee0 msgr2=0x7f5d0414d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d0a400640 1 --2- 172.21.15.32:0/1206894945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d04070ee0 0x7f5d0414d0b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00e9e0 tx=0x7f5cec00eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d03e00640 1 -- 172.21.15.32:0/1206894945 reap_dead start 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 shutdown_connections 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.166+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 >> 172.21.15.32:0/1206894945 conn(0x7f5d0406afb0 msgr2=0x7f5d04166030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.167+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 shutdown_connections 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.167+0000 7f5d0a400640 1 -- 172.21.15.32:0/1206894945 wait complete. 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.168+0000 7fdf41c00640 1 -- 172.21.15.32:0/1148991841 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdf14002dd0 con 0x7fdf18073780 2024-03-06T07:29:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.170+0000 7fdf32a00640 1 -- 172.21.15.32:0/1148991841 <== osd.2 v2:172.21.15.32:6800/1600044506 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdf14002dd0 con 0x7fdf18073780 2024-03-06T07:29:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.170+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f79340b52d0 con 0x7f79140737c0 2024-03-06T07:29:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.171+0000 7f792be00640 1 -- 172.21.15.32:0/1578595136 <== osd.1 v2:172.21.15.32:6817/50998742 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f79340b52d0 con 0x7f79140737c0 2024-03-06T07:29:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.171+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7fdf18073780 msgr2=0x7fdf18075ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.171+0000 7fdf2be00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7fdf18073780 0x7fdf18075ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f79140737c0 msgr2=0x7f7914075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf1806b500 msgr2=0x7fdf1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7f793b000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f79140737c0 0x7f7914075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7fdf2be00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf1806b500 0x7fdf1806d9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf34036000 tx=0x7fdf34008030 comp rx=0 tx=0).stop 2024-03-06T07:29:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 msgr2=0x7fdf3c1581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7fdf2be00640 1 --2- 172.21.15.32:0/1148991841 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf3c06e240 0x7fdf3c1581c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c007e60 tx=0x7fdf2c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.172+0000 7fdf3b400640 1 -- 172.21.15.32:0/1148991841 reap_dead start 2024-03-06T07:29:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa79804f3a0 con 0x7fa778073720 2024-03-06T07:29:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f791406b5a0 msgr2=0x7f791406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7f793b000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f791406b5a0 0x7f791406da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f792c001560 tx=0x7f792c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7934072da0 msgr2=0x7f79340ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7f793b000640 1 --2- 172.21.15.32:0/1578595136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7934072da0 0x7f79340ae060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7920032000 tx=0x7f79200046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 shutdown_connections 2024-03-06T07:29:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.173+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 >> 172.21.15.32:0/1148991841 conn(0x7fdf3c06afb0 msgr2=0x7fdf3c0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.174+0000 7f7939000640 1 -- 172.21.15.32:0/1578595136 reap_dead start 2024-03-06T07:29:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.174+0000 7fa78e000640 1 -- 172.21.15.32:0/975462802 <== osd.3 v2:172.21.15.32:6806/238384581 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa79804f3a0 con 0x7fa778073720 2024-03-06T07:29:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.174+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 shutdown_connections 2024-03-06T07:29:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.174+0000 7fdf2be00640 1 -- 172.21.15.32:0/1148991841 wait complete. 2024-03-06T07:29:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.174+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 shutdown_connections 2024-03-06T07:29:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.174+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 >> 172.21.15.32:0/1578595136 conn(0x7f793406afb0 msgr2=0x7f793406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.175+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 shutdown_connections 2024-03-06T07:29:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.175+0000 7f793b000640 1 -- 172.21.15.32:0/1578595136 wait complete. 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.175+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7fa778073720 msgr2=0x7fa778075b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.175+0000 7fa79d000640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7fa778073720 0x7fa778075b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa77806b500 msgr2=0x7fa77806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa79d000640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa77806b500 0x7fa77806d9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7980b15d0 tx=0x7fa784008030 comp rx=0 tx=0).stop 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa798070ee0 msgr2=0x7fa798155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa79d000640 1 --2- 172.21.15.32:0/975462802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa798070ee0 0x7fa798155840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa79000d5f0 tx=0x7fa79000dab0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa796800640 1 -- 172.21.15.32:0/975462802 reap_dead start 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 shutdown_connections 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.176+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 >> 172.21.15.32:0/975462802 conn(0x7fa79806afb0 msgr2=0x7fa798165540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.177+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 shutdown_connections 2024-03-06T07:29:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.177+0000 7fa79d000640 1 -- 172.21.15.32:0/975462802 wait complete. 2024-03-06T07:29:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.185+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3cb00cd190 con 0x7f3c8c073670 2024-03-06T07:29:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.186+0000 7f3ca6000640 1 -- 172.21.15.32:0/2424205511 <== osd.4 v2:172.21.15.39:6800/89256438 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3cb00cd190 con 0x7f3c8c073670 2024-03-06T07:29:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.187+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54dc0b52d0 con 0x7f54bc073740 2024-03-06T07:29:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.187+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f897c002dd0 con 0x7f8984073670 2024-03-06T07:29:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.189+0000 7f54d2a00640 1 -- 172.21.15.32:0/2669272048 <== osd.6 v2:172.21.15.39:6808/2944464285 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f54dc0b52d0 con 0x7f54bc073740 2024-03-06T07:29:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.189+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f54bc073740 msgr2=0x7f54bc075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.189+0000 7f899aa00640 1 -- 172.21.15.32:0/1335422319 <== osd.7 v2:172.21.15.39:6823/3019464823 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f897c002dd0 con 0x7f8984073670 2024-03-06T07:29:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.189+0000 7f54e1a00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f54bc073740 0x7f54bc075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x7f8984073670 msgr2=0x7f8984075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54bc06b520 msgr2=0x7f54bc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54e1a00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54bc06b520 0x7f54bc06d9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54cc001eb0 tx=0x7f54cc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54dc072da0 msgr2=0x7f54dc152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54e1a00640 1 --2- 172.21.15.32:0/2669272048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54dc072da0 0x7f54dc152a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009e80 tx=0x7f54d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54db400640 1 -- 172.21.15.32:0/2669272048 reap_dead start 2024-03-06T07:29:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f89a9a00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x7f8984073670 0x7f8984075a90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f3c8c073670 msgr2=0x7f3c8c075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 shutdown_connections 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 >> 172.21.15.32:0/2669272048 conn(0x7f54dc06afb0 msgr2=0x7f54dc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f3cb5000640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f3c8c073670 0x7f3c8c075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8c06b4d0 msgr2=0x7f3c8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f3cb5000640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009c00 tx=0x7f3c98009190 comp rx=0 tx=0).stop 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb015e120 msgr2=0x7f3cb00b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.190+0000 7f3cb5000640 1 --2- 172.21.15.32:0/2424205511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb015e120 0x7f3cb00b0610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3cb006c3a0 tx=0x7f3ca8004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:08.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.191+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 shutdown_connections 2024-03-06T07:29:08.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.191+0000 7f54e1a00640 1 -- 172.21.15.32:0/2669272048 wait complete. 2024-03-06T07:29:08.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.191+0000 7f3cae800640 1 -- 172.21.15.32:0/2424205511 reap_dead start 2024-03-06T07:29:08.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.191+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 shutdown_connections 2024-03-06T07:29:08.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.191+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 >> 172.21.15.32:0/2424205511 conn(0x7f3cb006afb0 msgr2=0x7f3cb0073fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.191+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 shutdown_connections 2024-03-06T07:29:08.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f3cb5000640 1 -- 172.21.15.32:0/2424205511 wait complete. 2024-03-06T07:29:08.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898406b4d0 msgr2=0x7f898406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898406b4d0 0x7f898406d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89940099a0 tx=0x7f89940090e0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a406e5e0 msgr2=0x7f89a41e8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 --2- 172.21.15.32:0/1335422319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89a406e5e0 0x7f89a41e8140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f899000c430 tx=0x7f899000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a3400640 1 -- 172.21.15.32:0/1335422319 reap_dead start 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 shutdown_connections 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 >> 172.21.15.32:0/1335422319 conn(0x7f89a406afb0 msgr2=0x7f89a4158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 shutdown_connections 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.192+0000 7f89a9a00640 1 -- 172.21.15.32:0/1335422319 wait complete. 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.194+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb9ec002dd0 con 0x7fb9f00736f0 2024-03-06T07:29:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.195+0000 7fba0a000640 1 -- 172.21.15.32:0/1063525822 <== osd.5 v2:172.21.15.39:6816/969666024 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb9ec002dd0 con 0x7fb9f00736f0 2024-03-06T07:29:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.196+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7fb9f00736f0 msgr2=0x7fb9f0075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.196+0000 7fba19200640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7fb9f00736f0 0x7fb9f0075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.196+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9f006b4d0 msgr2=0x7fb9f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.196+0000 7fba19200640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9f006b4d0 0x7fb9f006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc036000 tx=0x7fb9fc03b040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.214 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:08.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:29:08.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.197+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 msgr2=0x7fba1415ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.197+0000 7fba19200640 1 --2- 172.21.15.32:0/1063525822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba140d0d70 0x7fba1415ec40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fba0400d7d0 tx=0x7fba0400dc90 comp rx=0 tx=0).stop 2024-03-06T07:29:08.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.198+0000 7fba12800640 1 -- 172.21.15.32:0/1063525822 reap_dead start 2024-03-06T07:29:08.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.198+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 shutdown_connections 2024-03-06T07:29:08.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.198+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 >> 172.21.15.32:0/1063525822 conn(0x7fba140d3f60 msgr2=0x7fba140d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.216 INFO:teuthology.orchestra.run.smithi032.stdout:55834574852 2024-03-06T07:29:08.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:29:08.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.202+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 shutdown_connections 2024-03-06T07:29:08.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.202+0000 7fba19200640 1 -- 172.21.15.32:0/1063525822 wait complete. 2024-03-06T07:29:08.217 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:08.218 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2024-03-06T07:29:08.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:29:08.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:29:08.220 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:08.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:29:08.221 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2024-03-06T07:29:08.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:29:08.222 INFO:teuthology.orchestra.run.smithi032.stdout:55834574852 2024-03-06T07:29:08.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:29:08.225 INFO:teuthology.orchestra.run.smithi032.stdout:55834574852 2024-03-06T07:29:08.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:29:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.508+0000 7f8edac00640 1 Processor -- start 2024-03-06T07:29:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.508+0000 7f7795600640 1 Processor -- start 2024-03-06T07:29:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.508+0000 7f7795600640 1 -- start start 2024-03-06T07:29:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.508+0000 7f8edac00640 1 -- start start 2024-03-06T07:29:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.509+0000 7f8edac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40b31f0 0x7f8ed4162950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f8edac00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed40b4e10 con 0x7f8ed4165f40 2024-03-06T07:29:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f8edac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed41630c0 con 0x7f8ed4145520 2024-03-06T07:29:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f8edac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4163960 con 0x7f8ed40b31f0 2024-03-06T07:29:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f7795600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 0x7f7788052c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f7795600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77880e3640 con 0x7f77880531b0 2024-03-06T07:29:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f7795600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7788053930 con 0x7f77880e3260 2024-03-06T07:29:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f7795600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7788053c10 con 0x7f778804ff80 2024-03-06T07:29:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f8eda200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8ed4165f40 0x7f8ed40cdff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56606/0 (socket says 172.21.15.32:56606) 2024-03-06T07:29:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f8eda200640 1 -- 172.21.15.32:0/498852309 learned_addr learned my addr 172.21.15.32:0/498852309 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.510+0000 7f8ed8e00640 1 --2- 172.21.15.32:0/498852309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40b31f0 0x7f8ed4162950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f965f800640 1 Processor -- start 2024-03-06T07:29:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f965f800640 1 -- start start 2024-03-06T07:29:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f778fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f77880e3260 0x7f778804f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:51102/0 (socket says 172.21.15.32:51102) 2024-03-06T07:29:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f7794c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f77880531b0 0x7f7788037bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56610/0 (socket says 172.21.15.32:56610) 2024-03-06T07:29:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.512+0000 7f7794c00640 1 -- 172.21.15.32:0/3261639402 learned_addr learned my addr 172.21.15.32:0/3261639402 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.512+0000 7f778f400640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 0x7f7788052c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.512+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809916961 0 0) 0x7f77880e3640 con 0x7f77880531b0 2024-03-06T07:29:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f8ed8e00640 1 -- 172.21.15.32:0/498852309 >> v1:172.21.15.39:6789/0 conn(0x7f8ed4145520 legacy=0x7f8ed40b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f8ed8e00640 1 -- 172.21.15.32:0/498852309 >> v1:172.21.15.32:6789/0 conn(0x7f8ed4165f40 legacy=0x7f8ed40cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.511+0000 7f8ed8e00640 1 -- 172.21.15.32:0/498852309 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed41633a0 con 0x7f8ed40b31f0 2024-03-06T07:29:08.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.513+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7778003520 con 0x7f77880531b0 2024-03-06T07:29:08.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.513+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228001855 0 0) 0x7f7788053930 con 0x7f77880e3260 2024-03-06T07:29:08.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.513+0000 7f8ed8e00640 1 --2- 172.21.15.32:0/498852309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40b31f0 0x7f8ed4162950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80097d0 tx=0x7f8ec802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66bcc444cb017aed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.513+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77880e3640 con 0x7f77880e3260 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3460674163 0 0) 0x7f7778003520 con 0x7f77880531b0 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 msgr2=0x7f7788052c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 0x7f7788052c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 >> v1:172.21.15.39:6789/0 conn(0x7f77880e3260 legacy=0x7f778804f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788053ef0 con 0x7f77880531b0 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f7780001670 con 0x7f77880531b0 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 >> v1:172.21.15.32:6789/0 conn(0x7f77880531b0 legacy=0x7f7788037bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7778003b70 0x7f7778003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f7778004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 0x7f77780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7788053c10 con 0x7f7778004800 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778014260 con 0x7f7778003b70 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7778014460 con 0x7f778804ff80 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.514+0000 7f8ed3e00640 1 -- 172.21.15.32:0/498852309 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8034070 con 0x7f8ed40b31f0 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.515+0000 7f8ed3e00640 1 -- 172.21.15.32:0/498852309 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec8030610 con 0x7f8ed40b31f0 2024-03-06T07:29:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.515+0000 7f8ed3e00640 1 -- 172.21.15.32:0/498852309 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8030a20 con 0x7f8ed40b31f0 2024-03-06T07:29:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.515+0000 7f778f400640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f7778004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.515+0000 7f7794c00640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 0x7f77780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.515+0000 7f778fe00640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7778003b70 0x7f7778003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f778f400640 1 -- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 msgr2=0x7f77780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f8edac00640 1 -- 172.21.15.32:0/498852309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40b31f0 msgr2=0x7f8ed4162950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f778f400640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f778804ff80 0x7f77780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f778f400640 1 -- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7778003b70 msgr2=0x7f7778003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f778f400640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7778003b70 0x7f7778003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f778f400640 1 -- 172.21.15.32:0/3261639402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778003520 con 0x7f7778004800 2024-03-06T07:29:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f8edac00640 1 --2- 172.21.15.32:0/498852309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40b31f0 0x7f8ed4162950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80097d0 tx=0x7f8ec802b960 comp rx=0 tx=0).stop 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f8edac00640 1 -- 172.21.15.32:0/498852309 shutdown_connections 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f8edac00640 1 --2- 172.21.15.32:0/498852309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40b31f0 0x7f8ed4162950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.516+0000 7f8edac00640 1 -- 172.21.15.32:0/498852309 >> 172.21.15.32:0/498852309 conn(0x7f8ed40b17e0 msgr2=0x7f8ed40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.517+0000 7f8edac00640 1 -- 172.21.15.32:0/498852309 shutdown_connections 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.517+0000 7f8edac00640 1 -- 172.21.15.32:0/498852309 wait complete. 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.518+0000 7f965f800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f9658151200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.518+0000 7f965f800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96580aedd0 0x7f96580af260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.518+0000 7f965ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f9658151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f965ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f9658151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:32970/0 (socket says 172.21.15.32:32970) 2024-03-06T07:29:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.518+0000 7f778f400640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f7778004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7784009ac0 tx=0x7f7778005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e81f71d53447b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f965ce00640 1 -- 172.21.15.32:0/2063036367 learned_addr learned my addr 172.21.15.32:0/2063036367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965806ddf0 con 0x7f965806d6a0 2024-03-06T07:29:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965806e360 con 0x7f96580aedd0 2024-03-06T07:29:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784039070 con 0x7f7778004800 2024-03-06T07:29:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965806e5b0 con 0x7f9658150e20 2024-03-06T07:29:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778402c6b0 con 0x7f7778004800 2024-03-06T07:29:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f778ea00640 1 -- 172.21.15.32:0/3261639402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784009690 con 0x7f7778004800 2024-03-06T07:29:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.520+0000 7f8edac00640 1 Processor -- start 2024-03-06T07:29:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.519+0000 7f965d800640 1 --2- 172.21.15.32:0/2063036367 >> v2:172.21.15.39:3300/0 conn(0x7f96580aedd0 0x7f96580af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.520+0000 7f8edac00640 1 -- start start 2024-03-06T07:29:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f7795600640 1 -- 172.21.15.32:0/3261639402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 msgr2=0x7f7778004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f7795600640 1 --2- 172.21.15.32:0/3261639402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f7778004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7784009ac0 tx=0x7f7778005830 comp rx=0 tx=0).stop 2024-03-06T07:29:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f7794c00640 1 -- 172.21.15.32:0/3261639402 reap_dead start 2024-03-06T07:29:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f965ce00640 1 -- 172.21.15.32:0/2063036367 >> v2:172.21.15.39:3300/0 conn(0x7f96580aedd0 msgr2=0x7f96580af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f965ce00640 1 --2- 172.21.15.32:0/2063036367 >> v2:172.21.15.39:3300/0 conn(0x7f96580aedd0 0x7f96580af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f965ce00640 1 -- 172.21.15.32:0/2063036367 >> v1:172.21.15.32:6789/0 conn(0x7f965806d6a0 legacy=0x7f9658071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.521+0000 7f965ce00640 1 -- 172.21.15.32:0/2063036367 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965806e890 con 0x7f9658150e20 2024-03-06T07:29:08.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8edac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 0x7f8ed41ebef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f7795600640 1 -- 172.21.15.32:0/3261639402 shutdown_connections 2024-03-06T07:29:08.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f7795600640 1 -- 172.21.15.32:0/3261639402 >> 172.21.15.32:0/3261639402 conn(0x7f7788009fb0 msgr2=0x7f778800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8ed9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 0x7f8ed41ebef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8ed9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 0x7f8ed41ebef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50904/0 (socket says 172.21.15.32:50904) 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8ed9800640 1 -- 172.21.15.32:0/1866664471 learned_addr learned my addr 172.21.15.32:0/1866664471 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed4145520 0x7f8ed41f8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed4165f40 0x7f8ed413f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40cdad0 con 0x7f8ed40b31f0 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40ce700 con 0x7f8ed4145520 2024-03-06T07:29:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40ce880 con 0x7f8ed4165f40 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.522+0000 7f7795600640 1 -- 172.21.15.32:0/3261639402 shutdown_connections 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f7795600640 1 -- 172.21.15.32:0/3261639402 wait complete. 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f8ed9800640 1 -- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed4165f40 msgr2=0x7f8ed413f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f8ed9800640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed4165f40 0x7f8ed413f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f8ed9800640 1 -- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed4145520 msgr2=0x7f8ed41f8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f8ed9800640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed4145520 0x7f8ed41f8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f8ed9800640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed413fde0 con 0x7f8ed40b31f0 2024-03-06T07:29:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.523+0000 7f7f15400640 1 Processor -- start 2024-03-06T07:29:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.524+0000 7f8ed9800640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 0x7f8ed41ebef0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000c7c0 tx=0x7f8ec000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.524+0000 7f7f15400640 1 -- start start 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.524+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0016020 con 0x7f8ed40b31f0 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.524+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed41400c0 con 0x7f8ed40b31f0 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.524+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed4140680 con 0x7f8ed40b31f0 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.525+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0006d50 con 0x7f8ed40b31f0 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.525+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0003bb0 con 0x7f8ed40b31f0 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.525+0000 7f7f15400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.525+0000 7f7f15400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f100b3940 con 0x7f7f100d0d70 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.525+0000 7f7f15400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f100cf580 con 0x7f7f100b50e0 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.525+0000 7f7f15400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100d4e70 con 0x7f7f1015e2e0 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.526+0000 7f7f0de00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7f100d0d70 0x7f7f100d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56632/0 (socket says 172.21.15.32:56632) 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.526+0000 7f7f0de00640 1 -- 172.21.15.32:0/1458086386 learned_addr learned my addr 172.21.15.32:0/1458086386 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.526+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb0005290 con 0x7f8ed40b31f0 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.526+0000 7f965ce00640 1 --2- 172.21.15.32:0/2063036367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f9658151200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f965000c2e0 tx=0x7f965002e400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cff6d4a56c47f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f9657400640 1 -- 172.21.15.32:0/2063036367 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965003a070 con 0x7f9658150e20 2024-03-06T07:29:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f9657400640 1 -- 172.21.15.32:0/2063036367 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965000a4e0 con 0x7f9658150e20 2024-03-06T07:29:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f7795600640 1 Processor -- start 2024-03-06T07:29:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f9657400640 1 -- 172.21.15.32:0/2063036367 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965000a8b0 con 0x7f9658150e20 2024-03-06T07:29:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 msgr2=0x7f9658151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f965f800640 1 --2- 172.21.15.32:0/2063036367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f9658151200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f965000c2e0 tx=0x7f965002e400 comp rx=0 tx=0).stop 2024-03-06T07:29:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 shutdown_connections 2024-03-06T07:29:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f965f800640 1 --2- 172.21.15.32:0/2063036367 >> v2:172.21.15.39:3300/0 conn(0x7f96580aedd0 0x7f96580af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f965f800640 1 --2- 172.21.15.32:0/2063036367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f9658151200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 >> 172.21.15.32:0/2063036367 conn(0x7f965806afb0 msgr2=0x7f965806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f7795600640 1 -- start start 2024-03-06T07:29:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.529+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 shutdown_connections 2024-03-06T07:29:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.529+0000 7f965f800640 1 -- 172.21.15.32:0/2063036367 wait complete. 2024-03-06T07:29:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f7f0e800640 1 --2- 172.21.15.32:0/1458086386 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f7f0d400640 1 -- 172.21.15.32:0/1458086386 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675620039 0 0) 0x7f7f100cf580 con 0x7f7f100b50e0 2024-03-06T07:29:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.527+0000 7f7f0d400640 1 -- 172.21.15.32:0/1458086386 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef4003520 con 0x7f7f100b50e0 2024-03-06T07:29:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f7f0e800640 1 -- 172.21.15.32:0/1458086386 >> v1:172.21.15.39:6789/0 conn(0x7f7f100b50e0 legacy=0x7f7f100d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f7f0e800640 1 -- 172.21.15.32:0/1458086386 >> v1:172.21.15.32:6789/0 conn(0x7f7f100d0d70 legacy=0x7f7f100d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.528+0000 7f7f0e800640 1 -- 172.21.15.32:0/1458086386 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f100cf860 con 0x7f7f1015e2e0 2024-03-06T07:29:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f9800200640 1 Processor -- start 2024-03-06T07:29:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f9800200640 1 -- start start 2024-03-06T07:29:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7f0e800640 1 --2- 172.21.15.32:0/1458086386 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100d0830 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f000097d0 tx=0x7f7f0002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dfc82b0a1e023a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7795600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f77880da9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7795600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77880e3a70 0x7f77880e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7795600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 0x7f778804a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7795600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7788053640 con 0x7f7778004800 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7795600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77880544b0 con 0x7f7788049c80 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7795600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7788052c70 con 0x7f77880e3a70 2024-03-06T07:29:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7f0d400640 1 -- 172.21.15.32:0/1458086386 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00035020 con 0x7f7f1015e2e0 2024-03-06T07:29:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7f0d400640 1 -- 172.21.15.32:0/1458086386 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f00031830 con 0x7f7f1015e2e0 2024-03-06T07:29:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7f7f0d400640 1 -- 172.21.15.32:0/1458086386 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00031be0 con 0x7f7f1015e2e0 2024-03-06T07:29:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.531+0000 7f7794c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 0x7f778804a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.531+0000 7f7794c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 0x7f778804a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54432/0 (socket says 172.21.15.32:54432) 2024-03-06T07:29:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.531+0000 7f7794c00640 1 -- 172.21.15.32:0/3063793180 learned_addr learned my addr 172.21.15.32:0/3063793180 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.531+0000 7f4c79a00640 1 Processor -- start 2024-03-06T07:29:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.531+0000 7f4c79a00640 1 -- start start 2024-03-06T07:29:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.531+0000 7f9800200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f97f815e120 0x7f97f8070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.530+0000 7fc2b4600640 1 Processor -- start 2024-03-06T07:29:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f778f400640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77880e3a70 0x7f77880e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f965f800640 1 Processor -- start 2024-03-06T07:29:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f7f15400640 1 -- 172.21.15.32:0/1458086386 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 msgr2=0x7f7f100d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f7f15400640 1 --2- 172.21.15.32:0/1458086386 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100d0830 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f000097d0 tx=0x7f7f0002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7fc2b4600640 1 -- start start 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ec000a850 con 0x7f8ed40b31f0 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f965f800640 1 -- start start 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f9800200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 0x7f97f80711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f9800200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f97f80716f0 0x7f97f8161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f9800200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f80cdc20 con 0x7f97f815e120 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f9800200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f806d180 con 0x7f97f80716f0 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.532+0000 7f9800200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80cde70 con 0x7f97f80b2dd0 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.533+0000 7f97fce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 0x7f97f80711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.533+0000 7f97fce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 0x7f97f80711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:32994/0 (socket says 172.21.15.32:32994) 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.533+0000 7f97fce00640 1 -- 172.21.15.32:0/3943126296 learned_addr learned my addr 172.21.15.32:0/3943126296 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.533+0000 7f778fe00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f77880da9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.533+0000 7f4c79a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c740cd060 0x7f4c74161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7fc2b4600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc2ac0cfd90 0x7fc2ac0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7fc2b4600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 0x7fc2ac0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f97fce00640 1 -- 172.21.15.32:0/3943126296 >> v2:172.21.15.39:3300/0 conn(0x7f97f80716f0 msgr2=0x7f97f8161dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f97fce00640 1 --2- 172.21.15.32:0/3943126296 >> v2:172.21.15.39:3300/0 conn(0x7f97f80716f0 0x7f97f8161dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f97fd800640 1 --2- 172.21.15.32:0/3943126296 >> v2:172.21.15.32:3300/0 conn(0x7f97f815e120 0x7f97f8070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f97fce00640 1 -- 172.21.15.32:0/3943126296 >> v2:172.21.15.32:3300/0 conn(0x7f97f815e120 msgr2=0x7f97f8070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7f15400640 1 -- 172.21.15.32:0/1458086386 shutdown_connections 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7f15400640 1 --2- 172.21.15.32:0/1458086386 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100d0830 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7f15400640 1 -- 172.21.15.32:0/1458086386 >> 172.21.15.32:0/1458086386 conn(0x7f7f100d3f60 msgr2=0x7f7f100d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f8ed1600640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b500 0x7f8eb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7794c00640 1 -- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77880e3a70 msgr2=0x7f77880e3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7794c00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77880e3a70 0x7f77880e3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7794c00640 1 -- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 msgr2=0x7f77880da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ec008ac90 con 0x7f8ed40b31f0 2024-03-06T07:29:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7fc2b4600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ac0cea50 con 0x7fc2ac161b40 2024-03-06T07:29:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7fc2b4600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ac0b5770 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7fc2b4600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0cf900 con 0x7fc2ac0d2760 2024-03-06T07:29:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f97fce00640 1 --2- 172.21.15.32:0/3943126296 >> v2:172.21.15.32:3300/0 conn(0x7f97f815e120 0x7f97f8070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f97fce00640 1 -- 172.21.15.32:0/3943126296 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80ce150 con 0x7f97f80b2dd0 2024-03-06T07:29:08.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f4c73400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c740cd060 0x7f4c74161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f4c79a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c741627f0 con 0x7f4c74162000 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f4c79a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7406c860 con 0x7f4c7415efc0 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f4c79a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7415f3a0 con 0x7f4c740cd060 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f4c73e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4c7415efc0 0x7f4c740cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:51118/0 (socket says 172.21.15.32:51118) 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f4c73e00640 1 -- 172.21.15.32:0/3221417893 learned_addr learned my addr 172.21.15.32:0/3221417893 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.534+0000 7f7f15400640 1 -- 172.21.15.32:0/1458086386 shutdown_connections 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f8ed8e00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b500 0x7f8eb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f965f800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 0x7f9658146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f965f800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96580aedd0 0x7f9658149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f965f800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f965815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f965f800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965806eee0 con 0x7f965806d6a0 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f965f800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965806ed60 con 0x7f96580aedd0 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f965f800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965815dd20 con 0x7f9658150e20 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f7794c00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f77880da9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f7794c00640 1 -- 172.21.15.32:0/3063793180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778804aa30 con 0x7f7788049c80 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f7794c00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 0x7f778804a0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f778800b300 tx=0x7f778000e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923876332 0 0) 0x7f4c7406c860 con 0x7f4c7415efc0 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c58003520 con 0x7f4c7415efc0 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 <== mon.1 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030902851 0 0) 0x7f4c58003520 con 0x7f4c7415efc0 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 >> v2:172.21.15.32:3300/0 conn(0x7f4c740cd060 msgr2=0x7f4c74161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f4c72a00640 1 --2- 172.21.15.32:0/3221417893 >> v2:172.21.15.32:3300/0 conn(0x7f4c740cd060 0x7f4c74161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 >> v1:172.21.15.32:6789/0 conn(0x7f4c74162000 legacy=0x7f4c74070c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 0x7fc2ac0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 0x7fc2ac0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:33000/0 (socket says 172.21.15.32:33000) 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7fc2b2600640 1 -- 172.21.15.32:0/3244096875 learned_addr learned my addr 172.21.15.32:0/3244096875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7fc2b1200640 1 --2- 172.21.15.32:0/3244096875 >> v2:172.21.15.32:3300/0 conn(0x7fc2ac0cfd90 0x7fc2ac0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780012650 con 0x7f7788049c80 2024-03-06T07:29:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7780018070 con 0x7f7788049c80 2024-03-06T07:29:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780012a50 con 0x7f7788049c80 2024-03-06T07:29:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.536+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778804ad10 con 0x7f7788049c80 2024-03-06T07:29:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.537+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77880dbc40 con 0x7f7788049c80 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.537+0000 7f965ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 0x7f9658146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f965ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 0x7f9658146ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50942/0 (socket says 172.21.15.32:50942) 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f965ce00640 1 -- 172.21.15.32:0/2643665313 learned_addr learned my addr 172.21.15.32:0/2643665313 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.535+0000 7f7f15400640 1 -- 172.21.15.32:0/1458086386 wait complete. 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7406cb40 con 0x7f4c7415efc0 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 <== mon.1 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f4c64001670 con 0x7f4c7415efc0 2024-03-06T07:29:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 >> v1:172.21.15.39:6789/0 conn(0x7f4c7415efc0 legacy=0x7f4c740cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f7f15400640 1 Processor -- start 2024-03-06T07:29:08.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c580039b0 0x7f4c58003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c580051c0 0x7f4c5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7415f3a0 con 0x7f4c58004800 2024-03-06T07:29:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5800c040 con 0x7f4c580051c0 2024-03-06T07:29:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5800c240 con 0x7f4c580039b0 2024-03-06T07:29:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f8ed8e00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b500 0x7f8eb406d9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8036000 tx=0x7f8ec8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f7f15400640 1 -- start start 2024-03-06T07:29:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ec0020020 con 0x7f8ed40b31f0 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f4c79000640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c580051c0 0x7f4c5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f4c73e00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c580039b0 0x7f4c58003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f4c73400640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f965d800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f965815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f9657e00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96580aedd0 0x7f9658149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f965ce00640 1 -- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 msgr2=0x7f965815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f965ce00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f965815c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f965ce00640 1 -- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96580aedd0 msgr2=0x7f9658149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f965ce00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96580aedd0 0x7f9658149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f965ce00640 1 -- 172.21.15.32:0/2643665313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965815cb10 con 0x7f965806d6a0 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.537+0000 7fc2b2600640 1 -- 172.21.15.32:0/3244096875 >> v2:172.21.15.32:3300/0 conn(0x7fc2ac0cfd90 msgr2=0x7fc2ac0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.537+0000 7fc2b2600640 1 --2- 172.21.15.32:0/3244096875 >> v2:172.21.15.32:3300/0 conn(0x7fc2ac0cfd90 0x7fc2ac0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.537+0000 7fc2b2600640 1 -- 172.21.15.32:0/3244096875 >> v1:172.21.15.39:6789/0 conn(0x7fc2ac161b40 legacy=0x7fc2ac0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.537+0000 7fc2b2600640 1 -- 172.21.15.32:0/3244096875 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac0cecf0 con 0x7fc2ac0d2760 2024-03-06T07:29:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.538+0000 7f97fce00640 1 --2- 172.21.15.32:0/3943126296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 0x7f97f80711b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97f806ea60 tx=0x7f97e402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e091d40089a74222 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f965ce00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 0x7f9658146ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f965000c400 tx=0x7f9650004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.539+0000 7f4c73400640 1 -- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c580039b0 msgr2=0x7f4c58003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f97f7e00640 1 -- 172.21.15.32:0/3943126296 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4031760 con 0x7f97f80b2dd0 2024-03-06T07:29:08.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f97f7e00640 1 -- 172.21.15.32:0/3943126296 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e4037070 con 0x7f97f80b2dd0 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f97f7e00640 1 -- 172.21.15.32:0/3943126296 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4031b20 con 0x7f97f80b2dd0 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f4c73400640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c580039b0 0x7f4c58003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f4c73400640 1 -- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c580051c0 msgr2=0x7f4c5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f4c73400640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c580051c0 0x7f4c5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f4c73400640 1 -- 172.21.15.32:0/3221417893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7406cb40 con 0x7f4c58004800 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f4c79000640 1 -- 172.21.15.32:0/3221417893 reap_dead start 2024-03-06T07:29:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f4c73e00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c580039b0 0x7f4c58003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965003a040 con 0x7f965806d6a0 2024-03-06T07:29:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9650043070 con 0x7f965806d6a0 2024-03-06T07:29:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.540+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96500074f0 con 0x7f965806d6a0 2024-03-06T07:29:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.541+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965815cda0 con 0x7f965806d6a0 2024-03-06T07:29:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.541+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965815d360 con 0x7f965806d6a0 2024-03-06T07:29:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.541+0000 7f9800200640 1 -- 172.21.15.32:0/3943126296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 msgr2=0x7f97f80711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.541+0000 7f9800200640 1 --2- 172.21.15.32:0/3943126296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 0x7f97f80711b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97f806ea60 tx=0x7f97e402b800 comp rx=0 tx=0).stop 2024-03-06T07:29:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.541+0000 7f4c73400640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c58004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009ac0 tx=0x7f4c58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c7942d8484d403 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.542+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60039070 con 0x7f4c58004800 2024-03-06T07:29:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.542+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6002c6f0 con 0x7f4c58004800 2024-03-06T07:29:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.542+0000 7f4c72a00640 1 -- 172.21.15.32:0/3221417893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60009690 con 0x7f4c58004800 2024-03-06T07:29:08.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.542+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965804f3a0 con 0x7f965806d6a0 2024-03-06T07:29:08.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.542+0000 7fc2b2600640 1 --2- 172.21.15.32:0/3244096875 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 0x7fc2ac0b3740 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009e50 tx=0x7fc2a402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8de00cf0a9b843 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.542+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778804b230 con 0x7f7788049c80 2024-03-06T07:29:08.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f4c79a00640 1 -- 172.21.15.32:0/3221417893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 msgr2=0x7f4c58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f4c79a00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c58004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009ac0 tx=0x7f4c58005830 comp rx=0 tx=0).stop 2024-03-06T07:29:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f15400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f100b50e0 0x7f7f100cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f15400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f100d0d70 0x7f7f100b9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f15400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f15400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100cfe20 con 0x7f7f100d0d70 2024-03-06T07:29:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7fc2abe00640 1 -- 172.21.15.32:0/3244096875 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a402cb20 con 0x7fc2ac0d2760 2024-03-06T07:29:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7fc2abe00640 1 -- 172.21.15.32:0/3244096875 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a402ccc0 con 0x7fc2ac0d2760 2024-03-06T07:29:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f15400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100b3c40 con 0x7f7f100b50e0 2024-03-06T07:29:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f15400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100d2950 con 0x7f7f1015e2e0 2024-03-06T07:29:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7780012bf0 con 0x7f7788049c80 2024-03-06T07:29:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7fc2abe00640 1 -- 172.21.15.32:0/3244096875 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4034630 con 0x7fc2ac0d2760 2024-03-06T07:29:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f9800200640 1 -- 172.21.15.32:0/3943126296 shutdown_connections 2024-03-06T07:29:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f9800200640 1 --2- 172.21.15.32:0/3943126296 >> v2:172.21.15.39:3300/0 conn(0x7f97f80716f0 0x7f97f8161dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f9800200640 1 --2- 172.21.15.32:0/3943126296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f80b2dd0 0x7f97f80711b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f9800200640 1 --2- 172.21.15.32:0/3943126296 >> v2:172.21.15.32:3300/0 conn(0x7f97f815e120 0x7f97f8070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f9800200640 1 -- 172.21.15.32:0/3943126296 >> 172.21.15.32:0/3943126296 conn(0x7f97f806afb0 msgr2=0x7f97f806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f4c79a00640 1 -- 172.21.15.32:0/3221417893 shutdown_connections 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f4c79a00640 1 --2- 172.21.15.32:0/3221417893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c58004c80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f4c79a00640 1 -- 172.21.15.32:0/3221417893 >> 172.21.15.32:0/3221417893 conn(0x7f4c7406b3d0 msgr2=0x7f4c7406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7f4c79a00640 1 -- 172.21.15.32:0/3221417893 shutdown_connections 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7f4c79a00640 1 -- 172.21.15.32:0/3221417893 wait complete. 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7fc2b4600640 1 -- 172.21.15.32:0/3244096875 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 msgr2=0x7fc2ac0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7fc2b4600640 1 --2- 172.21.15.32:0/3244096875 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 0x7fc2ac0b3740 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009e50 tx=0x7fc2a402bfa0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96500359f0 con 0x7f965806d6a0 2024-03-06T07:29:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7fc2b4600640 1 -- 172.21.15.32:0/3244096875 shutdown_connections 2024-03-06T07:29:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f0de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f100d0d70 0x7f7f100b9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f0f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f0f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:33024/0 (socket says 172.21.15.32:33024) 2024-03-06T07:29:08.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.543+0000 7f7f0f200640 1 -- 172.21.15.32:0/1559322812 learned_addr learned my addr 172.21.15.32:0/1559322812 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7f7f0f200640 1 -- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f100b50e0 msgr2=0x7f7f100cb030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:29:08.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7fc2b4600640 1 --2- 172.21.15.32:0/3244096875 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac0d2760 0x7fc2ac0b3740 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.544+0000 7fc2b4600640 1 --2- 172.21.15.32:0/3244096875 >> v2:172.21.15.32:3300/0 conn(0x7fc2ac0cfd90 0x7fc2ac0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7fc2b4600640 1 -- 172.21.15.32:0/3244096875 >> 172.21.15.32:0/3244096875 conn(0x7fc2ac09d220 msgr2=0x7fc2ac0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7f9800200640 1 -- 172.21.15.32:0/3943126296 shutdown_connections 2024-03-06T07:29:08.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7fc2b4600640 1 -- 172.21.15.32:0/3244096875 shutdown_connections 2024-03-06T07:29:08.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7fc2b4600640 1 -- 172.21.15.32:0/3244096875 wait complete. 2024-03-06T07:29:08.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7f9800200640 1 -- 172.21.15.32:0/3943126296 wait complete. 2024-03-06T07:29:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7f777fe00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f777406b4d0 0x7f777406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7780089bd0 con 0x7f7788049c80 2024-03-06T07:29:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.545+0000 7f778fe00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f777406b4d0 0x7f777406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.546+0000 7f9654c00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f963806b520 0x7f963806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.546+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96500ac660 con 0x7f965806d6a0 2024-03-06T07:29:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.546+0000 7f9657e00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f963806b520 0x7f963806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.546+0000 7f4c79a00640 1 Processor -- start 2024-03-06T07:29:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.547+0000 7f7f0f200640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f100b50e0 0x7f7f100cb030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.547+0000 7f7f0f200640 1 -- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f100d0d70 msgr2=0x7f7f100b9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.547+0000 7f7f0f200640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f100d0d70 0x7f7f100b9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.547+0000 7f7f0f200640 1 -- 172.21.15.32:0/1559322812 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10204020 con 0x7f7f1015e2e0 2024-03-06T07:29:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.547+0000 7f7f0de00640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f100d0d70 0x7f7f100b9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7f7f0f200640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100b9840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c780 tx=0x7f7f0800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.547+0000 7f4c79a00640 1 -- start start 2024-03-06T07:29:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7fc2b4600640 1 Processor -- start 2024-03-06T07:29:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7fc2b4600640 1 -- start start 2024-03-06T07:29:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7f4c79a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c74145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7f4c79a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c741f45c0 0x7f4c741464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7f4c73e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c74145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7f4c79a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c740b3900 0x7f4c740b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.548+0000 7f4c73e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c74145950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50962/0 (socket says 172.21.15.32:50962) 2024-03-06T07:29:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08003da0 con 0x7f7f1015e2e0 2024-03-06T07:29:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f4c79a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74162470 con 0x7f4c58004800 2024-03-06T07:29:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f4c79a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7406d190 con 0x7f4c740b3900 2024-03-06T07:29:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f4c79a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7406d620 con 0x7f4c741f45c0 2024-03-06T07:29:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f4c73e00640 1 -- 172.21.15.32:0/2024408761 learned_addr learned my addr 172.21.15.32:0/2024408761 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f4c73400640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c741f45c0 0x7f4c741464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f4c79000640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c740b3900 0x7f4c740b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f9800200640 1 Processor -- start 2024-03-06T07:29:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08005070 con 0x7f7f1015e2e0 2024-03-06T07:29:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f100b8260 con 0x7f7f1015e2e0 2024-03-06T07:29:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f4c73400640 1 -- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c740b3900 msgr2=0x7f4c740b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f4c73400640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c740b3900 0x7f4c740b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f4c73400640 1 -- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 msgr2=0x7f4c74145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f4c73400640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c74145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f4c73400640 1 -- 172.21.15.32:0/2024408761 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c74145e90 con 0x7f4c741f45c0 2024-03-06T07:29:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f9800200640 1 -- start start 2024-03-06T07:29:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f778fe00640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f777406b4d0 0x7f777406d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77900505a0 tx=0x7f779008d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.549+0000 7f9657e00640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f963806b520 0x7f963806d9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9658148760 tx=0x7f9644009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f080054a0 con 0x7f7f1015e2e0 2024-03-06T07:29:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f100b8820 con 0x7f7f1015e2e0 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7fc2b4600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2ac0cfd90 0x7fc2ac0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7fc2b4600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2ac0d2760 0x7fc2ac0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7fc2b4600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 0x7fc2ac0c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7fc2b4600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0cf590 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7fc2b4600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0cf2b0 con 0x7fc2ac0d2760 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.550+0000 7fc2b4600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0b2ff0 con 0x7fc2ac161b40 2024-03-06T07:29:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.551+0000 7f4c73400640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c741f45c0 0x7f4c741464f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c60033000 tx=0x7f4c60004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.551+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 0x7fc2ac0c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.551+0000 7fc2b1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2ac0cfd90 0x7fc2ac0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.551+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 0x7fc2ac0c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:33028/0 (socket says 172.21.15.32:33028) 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.551+0000 7fc2b2600640 1 -- 172.21.15.32:0/4150320313 learned_addr learned my addr 172.21.15.32:0/4150320313 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f9800200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 0x7f97f814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f9800200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97f80b2dd0 0x7f97f81603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f9800200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f815e120 0x7f97f8160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f9800200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f806df00 con 0x7f97f80716f0 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f9800200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f806e3f0 con 0x7f97f80b2dd0 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f9800200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f815e500 con 0x7f97f815e120 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60042020 con 0x7f4c741f45c0 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6003a040 con 0x7f4c741f45c0 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60039070 con 0x7f4c741f45c0 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c741f35c0 con 0x7f4c741f45c0 2024-03-06T07:29:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c741f3b80 con 0x7f4c741f45c0 2024-03-06T07:29:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f97fce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97f80b2dd0 0x7f97f81603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7fc2b1c00640 1 -- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 msgr2=0x7fc2ac0c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7fc2b1c00640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 0x7fc2ac0c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7fc2b1c00640 1 -- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2ac0d2760 msgr2=0x7fc2ac0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f97fd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 0x7f97f814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f97fd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 0x7f97f814d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50986/0 (socket says 172.21.15.32:50986) 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f97fce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97f80b2dd0 0x7f97f81603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54494/0 (socket says 172.21.15.32:54494) 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f97fd800640 1 -- 172.21.15.32:0/167443587 learned_addr learned my addr 172.21.15.32:0/167443587 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7f97fe200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f815e120 0x7f97f8160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7fc2b1c00640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2ac0d2760 0x7fc2ac0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7fc2b1c00640 1 -- 172.21.15.32:0/4150320313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac0c8df0 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.552+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1004f3a0 con 0x7f7f1015e2e0 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7fc2b2600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 0x7fc2ac0c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.553+0000 7fc2b1c00640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2ac0cfd90 0x7fc2ac0cafc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc29c00c340 tx=0x7fc29c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7404f330 con 0x7f4c741f45c0 2024-03-06T07:29:08.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7f97fd800640 1 -- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f815e120 msgr2=0x7f97f8160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0052d0 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7f97fd800640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f815e120 0x7f97f8160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c005490 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c007260 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ac0b8530 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ac0cb500 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7f97fd800640 1 -- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97f80b2dd0 msgr2=0x7f97f81603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7f97fd800640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97f80b2dd0 0x7f97f81603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.554+0000 7f97fd800640 1 -- 172.21.15.32:0/167443587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f809b700 con 0x7f97f80716f0 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.555+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f965007b5c0 con 0x7f965806d6a0 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ac04f3a0 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.555+0000 7f97fd800640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 0x7f97f814d510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c7f0 tx=0x7f97ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec015070 con 0x7f97f80716f0 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec004240 con 0x7f97f80716f0 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec007620 con 0x7f97f80716f0 2024-03-06T07:29:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c60045020 con 0x7f4c741f45c0 2024-03-06T07:29:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.555+0000 7fb334400640 1 Processor -- start 2024-03-06T07:29:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.555+0000 7fb334400640 1 -- start start 2024-03-06T07:29:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.555+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f08005640 con 0x7f7f1015e2e0 2024-03-06T07:29:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f7f06a00640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ef006b520 0x7f7ef006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f0808a030 con 0x7f7f1015e2e0 2024-03-06T07:29:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.556+0000 7f7f0e800640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ef006b520 0x7f7ef006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.557+0000 7f7f0e800640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ef006b520 0x7f7ef006d9e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7f00036000 tx=0x7f7f00008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.557+0000 7f4c6be00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c5406b520 0x7f4c5406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.557+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c600ab650 con 0x7f4c741f45c0 2024-03-06T07:29:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.558+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc29c007400 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.557+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f809b9e0 con 0x7f97f80716f0 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.558+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f809bfa0 con 0x7f97f80716f0 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.558+0000 7fb334400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb32c0b52d0 0x7fb32c0d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.558+0000 7fb334400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb32c0b7300 0x7fb32c0b7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7fb334400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c0b6e50 con 0x7fb32c0c7320 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7fb334400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0cf720 con 0x7fb32c0b52d0 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7fb334400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0cfa00 con 0x7fb32c0b7300 2024-03-06T07:29:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7fc2a9600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc29c0896e0 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f8069510 con 0x7f97f80716f0 2024-03-06T07:29:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7f4c73e00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c5406b520 0x7f4c5406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.559+0000 7fb331a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb32c0b52d0 0x7fb32c0d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb32c0b7300 0x7fb32c0b7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb32c0b7300 0x7fb32c0b7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54498/0 (socket says 172.21.15.32:54498) 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 -- 172.21.15.32:0/2234094680 learned_addr learned my addr 172.21.15.32:0/2234094680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fc2b1200640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 -- 172.21.15.32:0/2234094680 >> v2:172.21.15.32:3300/0 conn(0x7fb32c0b52d0 msgr2=0x7fb32c0d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 --2- 172.21.15.32:0/2234094680 >> v2:172.21.15.32:3300/0 conn(0x7fb32c0b52d0 0x7fb32c0d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 -- 172.21.15.32:0/2234094680 >> v1:172.21.15.32:6789/0 conn(0x7fb32c0c7320 legacy=0x7fb32c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7fb332400640 1 -- 172.21.15.32:0/2234094680 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0cfce0 con 0x7fb32c0b7300 2024-03-06T07:29:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.560+0000 7f4c73e00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c5406b520 0x7f4c5406d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4c640048a0 tx=0x7f4c64000f90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.561+0000 7fc2b1200640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0726b0 tx=0x7fc298008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.562+0000 7fb332400640 1 --2- 172.21.15.32:0/2234094680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b7300 0x7fb32c0b7770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb32c08d250 tx=0x7fb32402b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cc9f87eac8c377d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.562+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97ec0079d0 con 0x7f97f80716f0 2024-03-06T07:29:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.563+0000 7f97f5600640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97d806b520 0x7f97d806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.564+0000 7f97fce00640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97d806b520 0x7f97d806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.564+0000 7f97fce00640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97d806b520 0x7f97d806d9e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97e4036000 tx=0x7f97e403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.565+0000 7fb32be00640 1 -- 172.21.15.32:0/2234094680 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324034070 con 0x7fb32c0b7300 2024-03-06T07:29:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.565+0000 7fb32be00640 1 -- 172.21.15.32:0/2234094680 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb324030870 con 0x7fb32c0b7300 2024-03-06T07:29:08.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.565+0000 7fb32be00640 1 -- 172.21.15.32:0/2234094680 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324030c40 con 0x7fb32c0b7300 2024-03-06T07:29:08.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f778804b230 con 0x7f7788049c80 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7fb334400640 1 -- 172.21.15.32:0/2234094680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b7300 msgr2=0x7fb32c0b7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7fb334400640 1 --2- 172.21.15.32:0/2234094680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b7300 0x7fb32c0b7770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb32c08d250 tx=0x7fb32402b9b0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7fb334400640 1 -- 172.21.15.32:0/2234094680 shutdown_connections 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7fb334400640 1 --2- 172.21.15.32:0/2234094680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b7300 0x7fb32c0b7770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7fb334400640 1 --2- 172.21.15.32:0/2234094680 >> v2:172.21.15.32:3300/0 conn(0x7fb32c0b52d0 0x7fb32c0d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.566+0000 7fb334400640 1 -- 172.21.15.32:0/2234094680 >> 172.21.15.32:0/2234094680 conn(0x7fb32c0d3f60 msgr2=0x7fb32c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.567+0000 7fb334400640 1 -- 172.21.15.32:0/2234094680 shutdown_connections 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.567+0000 7fb334400640 1 -- 172.21.15.32:0/2234094680 wait complete. 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.568+0000 7fb334400640 1 Processor -- start 2024-03-06T07:29:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.569+0000 7fb334400640 1 -- start start 2024-03-06T07:29:08.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.569+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97ec057030 con 0x7f97f80716f0 2024-03-06T07:29:08.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb334400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b52d0 0x7fb32c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb334400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb32c0b7300 0x7fb32c0c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb334400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32c0c7320 0x7fb32c200460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb334400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0d2ed0 con 0x7fb32c0c7320 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb334400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0d2d50 con 0x7fb32c0b52d0 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb334400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b56b0 con 0x7fb32c0b7300 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb332400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32c0c7320 0x7fb32c200460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb331a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b52d0 0x7fb32c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb331000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb32c0b7300 0x7fb32c0c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb331a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b52d0 0x7fb32c15e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54512/0 (socket says 172.21.15.32:54512) 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.570+0000 7fb331a00640 1 -- 172.21.15.32:0/4261464429 learned_addr learned my addr 172.21.15.32:0/4261464429 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.571+0000 7fb332400640 1 -- 172.21.15.32:0/4261464429 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb32c0b7300 msgr2=0x7fb32c0c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.571+0000 7fb332400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb32c0b7300 0x7fb32c0c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.571+0000 7fb332400640 1 -- 172.21.15.32:0/4261464429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b52d0 msgr2=0x7fb32c15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.571+0000 7fb332400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b52d0 0x7fb32c15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.571+0000 7fb332400640 1 -- 172.21.15.32:0/4261464429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0c8d00 con 0x7fb32c0c7320 2024-03-06T07:29:08.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.571+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2ac0b8530 con 0x7fc2ac0cfd90 2024-03-06T07:29:08.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.572+0000 7fb332400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32c0c7320 0x7fb32c200460 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb324033000 tx=0x7fb3240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.573+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c6004f020 con 0x7f4c741f45c0 2024-03-06T07:29:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.579+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324034040 con 0x7fb32c0c7320 2024-03-06T07:29:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.579+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3240433f0 con 0x7fb32c0c7320 2024-03-06T07:29:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.579+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97ec059690 con 0x7f97f80716f0 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.579+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3240437e0 con 0x7fb32c0c7320 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.579+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0c8fa0 con 0x7fb32c0c7320 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.579+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c0c9560 con 0x7fb32c0c7320 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.581+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c04f3a0 con 0x7fb32c0c7320 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.583+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb324037070 con 0x7fb32c0c7320 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.584+0000 7fb329600640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb30c06b520 0x7fb30c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.584+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3240ac330 con 0x7fb32c0c7320 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.585+0000 7fb331a00640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb30c06b520 0x7fb30c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.587+0000 7fb331a00640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb30c06b520 0x7fb30c06d9e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb32000f4c0 tx=0x7fb3200077d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.588+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f08059020 con 0x7f7f1015e2e0 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.593+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb32403e070 con 0x7fb32c0c7320 2024-03-06T07:29:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.986+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f96580adb10 con 0x7f965806d6a0 2024-03-06T07:29:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.987+0000 7f9654c00640 1 -- 172.21.15.32:0/2643665313 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f965007aec0 con 0x7f965806d6a0 2024-03-06T07:29:08.990 INFO:teuthology.orchestra.run.smithi032.stdout:55834574852 2024-03-06T07:29:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.989+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8eb0005850 con 0x7f8ed40b31f0 2024-03-06T07:29:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.990+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f963806b520 msgr2=0x7f963806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.990+0000 7f8ed1600640 1 -- 172.21.15.32:0/1866664471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ec0011380 con 0x7f8ed40b31f0 2024-03-06T07:29:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.990+0000 7f965f800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f963806b520 0x7f963806d9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9658148760 tx=0x7f9644009040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.993 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.990+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 msgr2=0x7f9658146ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.990+0000 7f965f800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 0x7f9658146ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f965000c400 tx=0x7f9650004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:08.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.991+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 shutdown_connections 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.991+0000 7f965f800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f963806b520 0x7f963806d9e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.991+0000 7f965f800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9658150e20 0x7f965815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.991+0000 7f965f800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96580aedd0 0x7f9658149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.991+0000 7f965f800640 1 --2- 172.21.15.32:0/2643665313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965806d6a0 0x7f9658146ef0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.991+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 >> 172.21.15.32:0/2643665313 conn(0x7f965806afb0 msgr2=0x7f96580ce9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.992+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 shutdown_connections 2024-03-06T07:29:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.992+0000 7f965f800640 1 -- 172.21.15.32:0/2643665313 wait complete. 2024-03-06T07:29:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.994+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b500 msgr2=0x7f8eb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.994+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b500 0x7f8eb406d9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8036000 tx=0x7f8ec8030000 comp rx=0 tx=0).stop 2024-03-06T07:29:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.994+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 msgr2=0x7f8ed41ebef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.994+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 0x7f8ed41ebef0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000c7c0 tx=0x7f8ec000cc80 comp rx=0 tx=0).stop 2024-03-06T07:29:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 shutdown_connections 2024-03-06T07:29:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b500 0x7f8eb406d9c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed4165f40 0x7f8ed413f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed4145520 0x7f8ed41f8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 --2- 172.21.15.32:0/1866664471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40b31f0 0x7f8ed41ebef0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 >> 172.21.15.32:0/1866664471 conn(0x7f8ed40b17e0 msgr2=0x7f8ed40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 shutdown_connections 2024-03-06T07:29:08.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:08.995+0000 7f8edac00640 1 -- 172.21.15.32:0/1866664471 wait complete. 2024-03-06T07:29:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.004+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f77880e4410 con 0x7f7788049c80 2024-03-06T07:29:09.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.005+0000 7f777fe00640 1 -- 172.21.15.32:0/3063793180 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77880e4410 con 0x7f7788049c80 2024-03-06T07:29:09.011 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2024-03-06T07:29:09.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.009+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f777406b4d0 msgr2=0x7f777406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.009+0000 7f7795600640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f777406b4d0 0x7f777406d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77900505a0 tx=0x7f779008d040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.009+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 msgr2=0x7f778804a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.009+0000 7f7795600640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 0x7f778804a0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f778800b300 tx=0x7f778000e9e0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.010+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 shutdown_connections 2024-03-06T07:29:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.010+0000 7f7795600640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f777406b4d0 0x7f777406d990 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77900505a0 tx=0x7f779008d040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.010+0000 7f7795600640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7788049c80 0x7f778804a0e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.010+0000 7f7795600640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77880e3a70 0x7f77880e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.010+0000 7f7795600640 1 --2- 172.21.15.32:0/3063793180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7778004800 0x7f77880da9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.010+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 >> 172.21.15.32:0/3063793180 conn(0x7f7788009fb0 msgr2=0x7f77880e5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.011+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 shutdown_connections 2024-03-06T07:29:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.011+0000 7f7795600640 1 -- 172.21.15.32:0/3063793180 wait complete. 2024-03-06T07:29:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.011+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4c740af2f0 con 0x7f4c741f45c0 2024-03-06T07:29:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.012+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc2ac0b4850 con 0x7fc2ac0cfd90 2024-03-06T07:29:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.012+0000 7fc2a9600640 1 -- 172.21.15.32:0/4150320313 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc29c057e70 con 0x7fc2ac0cfd90 2024-03-06T07:29:09.015 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.013+0000 7f4c6be00640 1 -- 172.21.15.32:0/2024408761 <== mon.2 v2:172.21.15.39:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c74145e90 con 0x7f4c741f45c0 2024-03-06T07:29:09.017 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:09.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.016+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c5406b520 msgr2=0x7f4c5406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.016+0000 7f4c79a00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c5406b520 0x7f4c5406d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4c640048a0 tx=0x7f4c64000f90 comp rx=0 tx=0).stop 2024-03-06T07:29:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.016+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c741f45c0 msgr2=0x7f4c741464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.016+0000 7f4c79a00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c741f45c0 0x7f4c741464f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c60033000 tx=0x7f4c60004730 comp rx=0 tx=0).stop 2024-03-06T07:29:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 shutdown_connections 2024-03-06T07:29:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c5406b520 0x7f4c5406d9e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c740b3900 0x7f4c740b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c741f45c0 0x7f4c741464f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 --2- 172.21.15.32:0/2024408761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c58004800 0x7f4c74145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 >> 172.21.15.32:0/2024408761 conn(0x7f4c7406b3d0 msgr2=0x7f4c740ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 shutdown_connections 2024-03-06T07:29:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.017+0000 7f4c79a00640 1 -- 172.21.15.32:0/2024408761 wait complete. 2024-03-06T07:29:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 msgr2=0x7fc28c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0726b0 tx=0x7fc298008040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2ac0cfd90 msgr2=0x7fc2ac0cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2ac0cfd90 0x7fc2ac0cafc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc29c00c340 tx=0x7fc29c00c800 comp rx=0 tx=0).stop 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 shutdown_connections 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0726b0 tx=0x7fc298008040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2ac161b40 0x7fc2ac0c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2ac0d2760 0x7fc2ac0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 --2- 172.21.15.32:0/4150320313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2ac0cfd90 0x7fc2ac0cafc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 >> 172.21.15.32:0/4150320313 conn(0x7fc2ac09d220 msgr2=0x7fc2ac06c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.018+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 shutdown_connections 2024-03-06T07:29:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.019+0000 7fc2b4600640 1 -- 172.21.15.32:0/4150320313 wait complete. 2024-03-06T07:29:09.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-03-06T07:29:09.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-03-06T07:29:09.024 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:09.024 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.023+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f97f8071c00 con 0x7f97f80716f0 2024-03-06T07:29:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.024+0000 7f97f5600640 1 -- 172.21.15.32:0/167443587 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f97ec058f90 con 0x7f97f80716f0 2024-03-06T07:29:09.026 INFO:teuthology.orchestra.run.smithi032.stdout:55834574851 2024-03-06T07:29:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.025+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7f100742f0 con 0x7f7f1015e2e0 2024-03-06T07:29:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.026+0000 7f7f06a00640 1 -- 172.21.15.32:0/1559322812 <== mon.2 v2:172.21.15.39:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f08058920 con 0x7f7f1015e2e0 2024-03-06T07:29:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.027+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97d806b520 msgr2=0x7f97d806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.027+0000 7f9800200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97d806b520 0x7f97d806d9e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97e4036000 tx=0x7f97e403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.027+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 msgr2=0x7f97f814d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.027+0000 7f9800200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 0x7f97f814d510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c7f0 tx=0x7f97ec00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 shutdown_connections 2024-03-06T07:29:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97d806b520 0x7f97d806d9e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97f815e120 0x7f97f8160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97f80b2dd0 0x7f97f81603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 --2- 172.21.15.32:0/167443587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f80716f0 0x7f97f814d510 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c7f0 tx=0x7f97ec00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 >> 172.21.15.32:0/167443587 conn(0x7f97f806afb0 msgr2=0x7f97f8070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.031 INFO:teuthology.orchestra.run.smithi032.stdout:55834574852 2024-03-06T07:29:09.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 shutdown_connections 2024-03-06T07:29:09.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f9800200640 1 -- 172.21.15.32:0/167443587 wait complete. 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ef006b520 msgr2=0x7f7ef006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f7f15400640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ef006b520 0x7f7ef006d9e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7f00036000 tx=0x7f7f00008030 comp rx=0 tx=0).stop 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 msgr2=0x7f7f100b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.028+0000 7f7f15400640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100b9840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c780 tx=0x7f7f0800cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 shutdown_connections 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ef006b520 0x7f7ef006d9e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f1015e2e0 0x7f7f100b9840 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f100d0d70 0x7f7f100b9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 --2- 172.21.15.32:0/1559322812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f100b50e0 0x7f7f100cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 >> 172.21.15.32:0/1559322812 conn(0x7f7f100d3f60 msgr2=0x7f7f100cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 shutdown_connections 2024-03-06T07:29:09.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.029+0000 7f7f15400640 1 -- 172.21.15.32:0/1559322812 wait complete. 2024-03-06T07:29:09.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.032+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb32c0b8f00 con 0x7fb32c0c7320 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.033+0000 7fb329600640 1 -- 172.21.15.32:0/4261464429 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb32407abd0 con 0x7fb32c0c7320 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi032.stdout:55834574851 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.036+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb30c06b520 msgr2=0x7fb30c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.036+0000 7fb334400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb30c06b520 0x7fb30c06d9e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb32000f4c0 tx=0x7fb3200077d0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.036+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32c0c7320 msgr2=0x7fb32c200460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.036+0000 7fb334400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32c0c7320 0x7fb32c200460 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb324033000 tx=0x7fb3240046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 shutdown_connections 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb30c06b520 0x7fb30c06d9e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32c0c7320 0x7fb32c200460 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb32c0b7300 0x7fb32c0c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 --2- 172.21.15.32:0/4261464429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb32c0b52d0 0x7fb32c15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 >> 172.21.15.32:0/4261464429 conn(0x7fb32c0d3f60 msgr2=0x7fb32c0b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 shutdown_connections 2024-03-06T07:29:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:09.037+0000 7fb334400640 1 -- 172.21.15.32:0/4261464429 wait complete. 2024-03-06T07:29:09.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-06T07:29:09.041 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:09.045 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-03-06T07:29:09.046 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:09.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2024-03-06T07:29:09.053 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2024-03-06T07:29:09.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.4 2024-03-06T07:29:09.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2024-03-06T07:29:10.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:29:10.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:29:10.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:29:10.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:29:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.252+0000 7f74ed800640 1 Processor -- start 2024-03-06T07:29:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.253+0000 7f74ed800640 1 -- start start 2024-03-06T07:29:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74ed800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f74e80af580 0x7f74e815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74ed800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 0x7f74e80b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74ed800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e815cc40 con 0x7f74e80af1a0 2024-03-06T07:29:10.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74ed800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e815cf20 con 0x7f74e80af580 2024-03-06T07:29:10.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74ed800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e815d200 con 0x7f74e80cdc20 2024-03-06T07:29:10.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74e6a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f74e80af1a0 0x7f74e80b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56658/0 (socket says 172.21.15.32:56658) 2024-03-06T07:29:10.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74e6a00640 1 -- 172.21.15.32:0/880688480 learned_addr learned my addr 172.21.15.32:0/880688480 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:10.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.254+0000 7f74e6000640 1 --2- 172.21.15.32:0/880688480 >> v2:172.21.15.39:3300/0 conn(0x7f74e80af580 0x7f74e815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e7400640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 0x7f74e80b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015071213 0 0) 0x7f74e815cc40 con 0x7f74e80af1a0 2024-03-06T07:29:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc003520 con 0x7f74e80af1a0 2024-03-06T07:29:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065603754 0 0) 0x7f74cc003520 con 0x7f74e80af1a0 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 msgr2=0x7f74e80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 0x7f74e80b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 >> v2:172.21.15.39:3300/0 conn(0x7f74e80af580 msgr2=0x7f74e815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 --2- 172.21.15.32:0/880688480 >> v2:172.21.15.39:3300/0 conn(0x7f74e80af580 0x7f74e815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e815d4e0 con 0x7f74e80af1a0 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f74d8001670 con 0x7f74e80af1a0 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 >> v1:172.21.15.32:6789/0 conn(0x7f74e80af1a0 legacy=0x7f74e80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc003a20 0x7f74cc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 0x7f74cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc004d40 0x7f74cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e815cf20 con 0x7f74cc004d40 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e815d200 con 0x7f74cc003a20 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.255+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc00ba10 con 0x7f74e80cdc20 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e7400640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc004d40 0x7f74cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e6a00640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc003a20 0x7f74cc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e6000640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 0x7f74cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e7400640 1 -- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 msgr2=0x7f74cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e7400640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e80cdc20 0x7f74cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e7400640 1 -- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc003a20 msgr2=0x7f74cc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e7400640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc003a20 0x7f74cc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.256+0000 7f74e7400640 1 -- 172.21.15.32:0/880688480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc003520 con 0x7f74cc004d40 2024-03-06T07:29:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.258+0000 7f74e7400640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc004d40 0x7f74cc0051c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009fc0 tx=0x7f74cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94aa3f5017d3332 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.258+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0039070 con 0x7f74cc004d40 2024-03-06T07:29:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.258+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e002cc20 con 0x7f74cc004d40 2024-03-06T07:29:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.258+0000 7f74e5600640 1 -- 172.21.15.32:0/880688480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0009c10 con 0x7f74cc004d40 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74ed800640 1 -- 172.21.15.32:0/880688480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc004d40 msgr2=0x7f74cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74ed800640 1 --2- 172.21.15.32:0/880688480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc004d40 0x7f74cc0051c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009fc0 tx=0x7f74cc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74e7400640 1 -- 172.21.15.32:0/880688480 reap_dead start 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74ed800640 1 -- 172.21.15.32:0/880688480 shutdown_connections 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74ed800640 1 -- 172.21.15.32:0/880688480 >> 172.21.15.32:0/880688480 conn(0x7f74e806afb0 msgr2=0x7f74e806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74ed800640 1 -- 172.21.15.32:0/880688480 shutdown_connections 2024-03-06T07:29:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.259+0000 7f74ed800640 1 -- 172.21.15.32:0/880688480 wait complete. 2024-03-06T07:29:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.261+0000 7f74ed800640 1 Processor -- start 2024-03-06T07:29:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.261+0000 7f74ed800640 1 -- start start 2024-03-06T07:29:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74ed800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc004d40 0x7f74e8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74ed800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 0x7f74e81f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74ed800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e8153730 0x7f74e8071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74ed800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e80b31a0 con 0x7f74e81f2660 2024-03-06T07:29:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74ed800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e80b3020 con 0x7f74cc004d40 2024-03-06T07:29:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74ed800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e80b3320 con 0x7f74e8153730 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74e6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 0x7f74e81f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74e6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 0x7f74e81f13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51018/0 (socket says 172.21.15.32:51018) 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.262+0000 7f74e6000640 1 -- 172.21.15.32:0/2518232609 learned_addr learned my addr 172.21.15.32:0/2518232609 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6a00640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc004d40 0x7f74e8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6000640 1 -- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e8153730 msgr2=0x7f74e8071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6000640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e8153730 0x7f74e8071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6000640 1 -- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc004d40 msgr2=0x7f74e8159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6000640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc004d40 0x7f74e8159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6000640 1 -- 172.21.15.32:0/2518232609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8154030 con 0x7f74e81f2660 2024-03-06T07:29:10.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.263+0000 7f74e6000640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 0x7f74e81f13a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f74d40116a0 tx=0x7f74d4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.264+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d400f940 con 0x7f74e81f2660 2024-03-06T07:29:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.264+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d400fae0 con 0x7f74e81f2660 2024-03-06T07:29:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.264+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4015460 con 0x7f74e81f2660 2024-03-06T07:29:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.264+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e8158830 con 0x7f74e81f2660 2024-03-06T07:29:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.264+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e8158df0 con 0x7f74e81f2660 2024-03-06T07:29:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.265+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e8069510 con 0x7f74e81f2660 2024-03-06T07:29:10.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.267+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74d4015600 con 0x7f74e81f2660 2024-03-06T07:29:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.269+0000 7f74dea00640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74c806b500 0x7f74c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.269+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74d408abc0 con 0x7f74e81f2660 2024-03-06T07:29:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.269+0000 7f74e6a00640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74c806b500 0x7f74c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.270+0000 7f74e6a00640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74c806b500 0x7f74c806d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f74d8007cb0 tx=0x7f74d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.277+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74d4019030 con 0x7f74e81f2660 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.287+0000 7f2e97c00640 1 Processor -- start 2024-03-06T07:29:10.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.288+0000 7f2e97c00640 1 -- start start 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.289+0000 7f2e97c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900af580 0x7f2e9015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.289+0000 7f2e97c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9015cc40 con 0x7f2e900af1a0 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.289+0000 7f2e97c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9015cf20 con 0x7f2e900cdc20 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.289+0000 7f2e97c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9015d200 con 0x7f2e900af580 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.289+0000 7f2e95200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2e900af1a0 0x7f2e900b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56666/0 (socket says 172.21.15.32:56666) 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.289+0000 7f2e95200640 1 -- 172.21.15.32:0/1789829832 learned_addr learned my addr 172.21.15.32:0/1789829832 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8fe00640 1 --2- 172.21.15.32:0/1789829832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900af580 0x7f2e9015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785832183 0 0) 0x7f2e9015cc40 con 0x7f2e900af1a0 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e78003520 con 0x7f2e900af1a0 2024-03-06T07:29:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f49b6800640 1 Processor -- start 2024-03-06T07:29:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f49b6800640 1 -- start start 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202205228 0 0) 0x7f2e9015cf20 con 0x7f2e900cdc20 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9015cc40 con 0x7f2e900cdc20 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2580935270 0 0) 0x7f2e78003520 con 0x7f2e900af1a0 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8fe00640 1 -- 172.21.15.32:0/1789829832 >> v1:172.21.15.39:6789/0 conn(0x7f2e900cdc20 legacy=0x7f2e900b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.290+0000 7f2e8fe00640 1 -- 172.21.15.32:0/1789829832 >> v1:172.21.15.32:6789/0 conn(0x7f2e900af1a0 legacy=0x7f2e900b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.291+0000 7f2e8fe00640 1 -- 172.21.15.32:0/1789829832 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9015d4e0 con 0x7f2e900af580 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.291+0000 7fb629400640 1 Processor -- start 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.291+0000 7fb629400640 1 -- start start 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb629400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb62415e2e0 0x7fb6240d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb629400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6240d0d70 0x7fb6240d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb629400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6240b3940 con 0x7fb6240d1b30 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb629400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6240d4e70 con 0x7fb6240d0d70 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb629400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6240b3c40 con 0x7fb62415e2e0 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7f49b6800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49b00d2b40 0x7f49b00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7f49b6800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49b00d1480 0x7f49b00b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7f49b6800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00b3940 con 0x7f49b015e2e0 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7f49b6800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b00d4e70 con 0x7f49b00d2b40 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7f49b6800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b00b3c40 con 0x7f49b00d1480 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb621e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6240d0d70 0x7fb6240d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb621e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6240d0d70 0x7fb6240d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51028/0 (socket says 172.21.15.32:51028) 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb621e00640 1 -- 172.21.15.32:0/1173842613 learned_addr learned my addr 172.21.15.32:0/1173842613 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.292+0000 7fb622800640 1 --2- 172.21.15.32:0/1173842613 >> v2:172.21.15.39:3300/0 conn(0x7fb62415e2e0 0x7fb6240d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49af400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f49b015e2e0 0x7f49b00d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:51164/0 (socket says 172.21.15.32:51164) 2024-03-06T07:29:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49af400640 1 -- 172.21.15.32:0/67359289 learned_addr learned my addr 172.21.15.32:0/67359289 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f2e8fe00640 1 --2- 172.21.15.32:0/1789829832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900af580 0x7f2e9015c600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0097d0 tx=0x7f2e7c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=172a8ad677583eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49aea00640 1 --2- 172.21.15.32:0/67359289 >> v2:172.21.15.32:3300/0 conn(0x7f49b00d2b40 0x7f49b00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7fb621e00640 1 -- 172.21.15.32:0/1173842613 >> v2:172.21.15.39:3300/0 conn(0x7fb62415e2e0 msgr2=0x7fb6240d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7fb621e00640 1 --2- 172.21.15.32:0/1173842613 >> v2:172.21.15.39:3300/0 conn(0x7fb62415e2e0 0x7fb6240d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7fb621e00640 1 -- 172.21.15.32:0/1173842613 >> v1:172.21.15.39:6789/0 conn(0x7fb6240d1b30 legacy=0x7fb6240d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7fb621e00640 1 -- 172.21.15.32:0/1173842613 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6240cf580 con 0x7fb6240d0d70 2024-03-06T07:29:10.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49afe00640 1 --2- 172.21.15.32:0/67359289 >> v2:172.21.15.39:3300/0 conn(0x7f49b00d1480 0x7f49b00b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49aea00640 1 -- 172.21.15.32:0/67359289 >> v2:172.21.15.39:3300/0 conn(0x7f49b00d1480 msgr2=0x7f49b00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c037070 con 0x7f2e900af580 2024-03-06T07:29:10.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49aea00640 1 --2- 172.21.15.32:0/67359289 >> v2:172.21.15.39:3300/0 conn(0x7f49b00d1480 0x7f49b00b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49aea00640 1 -- 172.21.15.32:0/67359289 >> v1:172.21.15.39:6789/0 conn(0x7f49b015e2e0 legacy=0x7f49b00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49aea00640 1 -- 172.21.15.32:0/67359289 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00cf580 con 0x7f49b00d2b40 2024-03-06T07:29:10.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f49afe00640 1 --2- 172.21.15.32:0/67359289 >> v2:172.21.15.39:3300/0 conn(0x7f49b00d1480 0x7f49b00b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c030510 con 0x7f2e900af580 2024-03-06T07:29:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.293+0000 7f2e8f400640 1 -- 172.21.15.32:0/1789829832 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840015f0 con 0x7f2e900af580 2024-03-06T07:29:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 -- 172.21.15.32:0/1789829832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900af580 msgr2=0x7f2e9015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 --2- 172.21.15.32:0/1789829832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900af580 0x7f2e9015c600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0097d0 tx=0x7f2e7c02b770 comp rx=0 tx=0).stop 2024-03-06T07:29:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 -- 172.21.15.32:0/1789829832 shutdown_connections 2024-03-06T07:29:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 --2- 172.21.15.32:0/1789829832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900af580 0x7f2e9015c600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 -- 172.21.15.32:0/1789829832 >> 172.21.15.32:0/1789829832 conn(0x7f2e9006afb0 msgr2=0x7f2e9006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 -- 172.21.15.32:0/1789829832 shutdown_connections 2024-03-06T07:29:10.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.294+0000 7f2e97c00640 1 -- 172.21.15.32:0/1789829832 wait complete. 2024-03-06T07:29:10.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.295+0000 7fb621e00640 1 --2- 172.21.15.32:0/1173842613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6240d0d70 0x7fb6240d11d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb62408d8b0 tx=0x7fb61002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d18614342ea956 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.295+0000 7f49aea00640 1 --2- 172.21.15.32:0/67359289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b00d0f40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f49b00ce540 tx=0x7f499c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d1b21b94c33b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.295+0000 7fb621400640 1 -- 172.21.15.32:0/1173842613 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610035020 con 0x7fb6240d0d70 2024-03-06T07:29:10.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.295+0000 7fb621400640 1 -- 172.21.15.32:0/1173842613 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb610031870 con 0x7fb6240d0d70 2024-03-06T07:29:10.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.295+0000 7fb621400640 1 -- 172.21.15.32:0/1173842613 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610031c40 con 0x7fb6240d0d70 2024-03-06T07:29:10.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7f49ae000640 1 -- 172.21.15.32:0/67359289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c035020 con 0x7f49b00d2b40 2024-03-06T07:29:10.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7f49ae000640 1 -- 172.21.15.32:0/67359289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c031830 con 0x7f49b00d2b40 2024-03-06T07:29:10.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7f49ae000640 1 -- 172.21.15.32:0/67359289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c031c00 con 0x7f49b00d2b40 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7fb629400640 1 -- 172.21.15.32:0/1173842613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6240d0d70 msgr2=0x7fb6240d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7fb629400640 1 --2- 172.21.15.32:0/1173842613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6240d0d70 0x7fb6240d11d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb62408d8b0 tx=0x7fb61002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7fb629400640 1 -- 172.21.15.32:0/1173842613 shutdown_connections 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7fb629400640 1 --2- 172.21.15.32:0/1173842613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6240d0d70 0x7fb6240d11d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7fb629400640 1 --2- 172.21.15.32:0/1173842613 >> v2:172.21.15.39:3300/0 conn(0x7fb62415e2e0 0x7fb6240d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7fb629400640 1 -- 172.21.15.32:0/1173842613 >> 172.21.15.32:0/1173842613 conn(0x7fb6240d3f60 msgr2=0x7fb6240d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7f49b6800640 1 -- 172.21.15.32:0/67359289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 msgr2=0x7f49b00d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.296+0000 7f49b6800640 1 --2- 172.21.15.32:0/67359289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b00d0f40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f49b00ce540 tx=0x7f499c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7fb629400640 1 -- 172.21.15.32:0/1173842613 shutdown_connections 2024-03-06T07:29:10.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7fb629400640 1 -- 172.21.15.32:0/1173842613 wait complete. 2024-03-06T07:29:10.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f2e97c00640 1 Processor -- start 2024-03-06T07:29:10.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f2e97c00640 1 -- start start 2024-03-06T07:29:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f49b6800640 1 -- 172.21.15.32:0/67359289 shutdown_connections 2024-03-06T07:29:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f49b6800640 1 --2- 172.21.15.32:0/67359289 >> v2:172.21.15.39:3300/0 conn(0x7f49b00d1480 0x7f49b00b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f49b6800640 1 --2- 172.21.15.32:0/67359289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b00d0f40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f49b6800640 1 -- 172.21.15.32:0/67359289 >> 172.21.15.32:0/67359289 conn(0x7f49b00d3f60 msgr2=0x7f49b00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f49b6800640 1 -- 172.21.15.32:0/67359289 shutdown_connections 2024-03-06T07:29:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.297+0000 7f49b6800640 1 -- 172.21.15.32:0/67359289 wait complete. 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.298+0000 7f2e97c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e900af1a0 0x7f2e9014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.298+0000 7f2e97c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 0x7f2e9009b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.298+0000 7f2e97c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900cdc20 0x7f2e9014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.298+0000 7f2e97c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90155e70 con 0x7f2e900af580 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.298+0000 7f2e97c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900b36f0 con 0x7f2e900af1a0 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.298+0000 7f2e97c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90155ff0 con 0x7f2e900cdc20 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 0x7f2e9009b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7fb629400640 1 Processor -- start 2024-03-06T07:29:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 0x7f2e9009b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51048/0 (socket says 172.21.15.32:51048) 2024-03-06T07:29:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 -- 172.21.15.32:0/749493826 learned_addr learned my addr 172.21.15.32:0/749493826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e95200640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e900af1a0 0x7f2e9014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e95c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900cdc20 0x7f2e9014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 -- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900cdc20 msgr2=0x7f2e9014d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7fb629400640 1 -- start start 2024-03-06T07:29:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900cdc20 0x7f2e9014d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 -- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e900af1a0 msgr2=0x7f2e9014ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e900af1a0 0x7f2e9014ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f2e8fe00640 1 -- 172.21.15.32:0/749493826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9014d390 con 0x7f2e900af580 2024-03-06T07:29:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f49b6800640 1 Processor -- start 2024-03-06T07:29:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.299+0000 7f49b6800640 1 -- start start 2024-03-06T07:29:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7f2e8fe00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 0x7f2e9009b9e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c035040 tx=0x7f2e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c037050 con 0x7f2e900af580 2024-03-06T07:29:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c043070 con 0x7f2e900af580 2024-03-06T07:29:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7fb629400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 0x7fb62415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7fb629400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6240d1b30 0x7fb62415e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c003850 con 0x7f2e900af580 2024-03-06T07:29:10.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7fb629400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb62415e2e0 0x7fb6240cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7fb629400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6240cfe20 con 0x7fb62415e2e0 2024-03-06T07:29:10.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7fb629400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6240cfb40 con 0x7fb6240d0d70 2024-03-06T07:29:10.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7fb629400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6240b50e0 con 0x7fb6240d1b30 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49b6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b0153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49b6800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49b015e2e0 0x7f49b01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49b6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b01608c0 0x7f49b01f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49b6800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00cfe20 con 0x7f49b00d2b40 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49b6800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00cfb40 con 0x7f49b01608c0 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49b6800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00b4610 con 0x7f49b015e2e0 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b0153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b0153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51062/0 (socket says 172.21.15.32:51062) 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7f49af400640 1 -- 172.21.15.32:0/1576341961 learned_addr learned my addr 172.21.15.32:0/1576341961 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49aea00640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49b015e2e0 0x7f49b01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49afe00640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b01608c0 0x7f49b01f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7fb622800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 0x7fb62415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7fb622800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 0x7fb62415a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54550/0 (socket says 172.21.15.32:54550) 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7fb622800640 1 -- 172.21.15.32:0/3600909 learned_addr learned my addr 172.21.15.32:0/3600909 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7fb623200640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb62415e2e0 0x7fb6240cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.301+0000 7fb621e00640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6240d1b30 0x7fb62415e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb623200640 1 -- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6240d1b30 msgr2=0x7fb62415e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb623200640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6240d1b30 0x7fb62415e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9009ac20 con 0x7f2e900af580 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.300+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9009b1e0 con 0x7f2e900af580 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49af400640 1 -- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49b015e2e0 msgr2=0x7f49b01541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49af400640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49b015e2e0 0x7f49b01541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49af400640 1 -- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b01608c0 msgr2=0x7f49b01f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49af400640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b01608c0 0x7f49b01f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f49af400640 1 -- 172.21.15.32:0/1576341961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0160480 con 0x7f49b00d2b40 2024-03-06T07:29:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb623200640 1 -- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 msgr2=0x7fb62415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb623200640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 0x7fb62415a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb623200640 1 -- 172.21.15.32:0/3600909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62415aaa0 con 0x7fb62415e2e0 2024-03-06T07:29:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90069510 con 0x7f2e900af580 2024-03-06T07:29:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb622800640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 0x7fb62415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:10.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.302+0000 7fb623200640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb62415e2e0 0x7fb6240cea30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c7b0 tx=0x7fb61c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f49af400640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b0153810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49a000c480 tx=0x7f49a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c016020 con 0x7fb62415e2e0 2024-03-06T07:29:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c00aa50 con 0x7fb62415e2e0 2024-03-06T07:29:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c016020 con 0x7fb62415e2e0 2024-03-06T07:29:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0017070 con 0x7f49b00d2b40 2024-03-06T07:29:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a0005ce0 con 0x7f49b00d2b40 2024-03-06T07:29:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62415f100 con 0x7fb62415e2e0 2024-03-06T07:29:10.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a00072f0 con 0x7f49b00d2b40 2024-03-06T07:29:10.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62415fe90 con 0x7fb62415e2e0 2024-03-06T07:29:10.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e7c0039f0 con 0x7f2e900af580 2024-03-06T07:29:10.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b01f3b20 con 0x7f49b00d2b40 2024-03-06T07:29:10.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.303+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b0154b60 con 0x7f49b00d2b40 2024-03-06T07:29:10.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.305+0000 7f2e8cc00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2e7006b5a0 0x7f2e7006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.305+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b0160cd0 con 0x7f49b00d2b40 2024-03-06T07:29:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.305+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e7c0ac090 con 0x7f2e900af580 2024-03-06T07:29:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.305+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb624160090 con 0x7fb62415e2e0 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.305+0000 7f2e95200640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2e7006b5a0 0x7f2e7006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.306+0000 7f2e95200640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2e7006b5a0 0x7f2e7006da60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e840013e0 tx=0x7f2e84009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.309+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e9014d390 con 0x7f2e900af580 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.311+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49a001a070 con 0x7f49b00d2b40 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.312+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb61c019020 con 0x7fb62415e2e0 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.312+0000 7f49a7400640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f499006b500 0x7f499006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.312+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49a0089950 con 0x7f49b00d2b40 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.312+0000 7f49aea00640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f499006b500 0x7f499006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.313+0000 7f49aea00640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f499006b500 0x7f499006d9c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f499c036000 tx=0x7f499c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.313+0000 7fb61aa00640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb60406b8d0 0x7fb60406dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.313+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb61c00a850 con 0x7fb62415e2e0 2024-03-06T07:29:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.313+0000 7fb622800640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb60406b8d0 0x7fb60406dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.314+0000 7fb622800640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb60406b8d0 0x7fb60406dd90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb6240726b0 tx=0x7fb614008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.317+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb624160090 con 0x7fb62415e2e0 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.322+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49b0160cd0 con 0x7f49b00d2b40 2024-03-06T07:29:10.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.512+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb62404f3a0 con 0x7fb62415e2e0 2024-03-06T07:29:10.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.512+0000 7fb61aa00640 1 -- 172.21.15.32:0/3600909 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb61c020070 con 0x7fb62415e2e0 2024-03-06T07:29:10.514 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb60406b8d0 msgr2=0x7fb60406dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb60406b8d0 0x7fb60406dd90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb6240726b0 tx=0x7fb614008040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb62415e2e0 msgr2=0x7fb6240cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb62415e2e0 0x7fb6240cea30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c7b0 tx=0x7fb61c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 shutdown_connections 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb60406b8d0 0x7fb60406dd90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb62415e2e0 0x7fb6240cea30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6240d1b30 0x7fb62415e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 --2- 172.21.15.32:0/3600909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6240d0d70 0x7fb62415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.514+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 >> 172.21.15.32:0/3600909 conn(0x7fb6240d3f60 msgr2=0x7fb62406d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.515+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 shutdown_connections 2024-03-06T07:29:10.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.515+0000 7fb629400640 1 -- 172.21.15.32:0/3600909 wait complete. 2024-03-06T07:29:10.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.519+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f49b004f3a0 con 0x7f49b00d2b40 2024-03-06T07:29:10.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.519+0000 7f49a7400640 1 -- 172.21.15.32:0/1576341961 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f49a005e030 con 0x7f49b00d2b40 2024-03-06T07:29:10.521 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2024-03-06T07:29:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f499006b500 msgr2=0x7f499006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f499006b500 0x7f499006d9c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f499c036000 tx=0x7f499c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 msgr2=0x7f49b0153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b0153810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49a000c480 tx=0x7f49a000c940 comp rx=0 tx=0).stop 2024-03-06T07:29:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 shutdown_connections 2024-03-06T07:29:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f499006b500 0x7f499006d9c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b01608c0 0x7f49b01f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49b015e2e0 0x7f49b01541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 --2- 172.21.15.32:0/1576341961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b00d2b40 0x7f49b0153810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.521+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 >> 172.21.15.32:0/1576341961 conn(0x7f49b00d3f60 msgr2=0x7f49b00b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.522+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 shutdown_connections 2024-03-06T07:29:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.522+0000 7f49b6800640 1 -- 172.21.15.32:0/1576341961 wait complete. 2024-03-06T07:29:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.523+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f74e8158ff0 con 0x7f74e81f2660 2024-03-06T07:29:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.523+0000 7f74dea00640 1 -- 172.21.15.32:0/2518232609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74e8158ff0 con 0x7f74e81f2660 2024-03-06T07:29:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.526+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74c806b500 msgr2=0x7f74c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.526+0000 7f74ed800640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74c806b500 0x7f74c806d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f74d8007cb0 tx=0x7f74d8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.526+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 msgr2=0x7f74e81f13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.526+0000 7f74ed800640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 0x7f74e81f13a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f74d40116a0 tx=0x7f74d4011b60 comp rx=0 tx=0).stop 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 shutdown_connections 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74c806b500 0x7f74c806d9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74e8153730 0x7f74e8071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74e81f2660 0x7f74e81f13a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 --2- 172.21.15.32:0/2518232609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc004d40 0x7f74e8159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 >> 172.21.15.32:0/2518232609 conn(0x7f74e806afb0 msgr2=0x7f74e8074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 shutdown_connections 2024-03-06T07:29:10.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.527+0000 7f74ed800640 1 -- 172.21.15.32:0/2518232609 wait complete. 2024-03-06T07:29:10.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.5 2024-03-06T07:29:10.536 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:10.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.4 2024-03-06T07:29:10.542 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:10.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2024-03-06T07:29:10.547 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:10.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.548+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2e9004f3a0 con 0x7f2e900af580 2024-03-06T07:29:10.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.548+0000 7f2e8cc00640 1 -- 172.21.15.32:0/749493826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e7c07ae60 con 0x7f2e900af580 2024-03-06T07:29:10.550 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2e7006b5a0 msgr2=0x7f2e7006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2e7006b5a0 0x7f2e7006da60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e840013e0 tx=0x7f2e84009040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 msgr2=0x7f2e9009b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 0x7f2e9009b9e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c035040 tx=0x7f2e7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 shutdown_connections 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2e7006b5a0 0x7f2e7006da60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e900cdc20 0x7f2e9014d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e900af580 0x7f2e9009b9e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 --2- 172.21.15.32:0/749493826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e900af1a0 0x7f2e9014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 >> 172.21.15.32:0/749493826 conn(0x7f2e9006afb0 msgr2=0x7f2e900ad640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.550+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 shutdown_connections 2024-03-06T07:29:10.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.551+0000 7f2e97c00640 1 -- 172.21.15.32:0/749493826 wait complete. 2024-03-06T07:29:10.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2024-03-06T07:29:10.569 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:10.569 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:29:10.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:10.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.825+0000 7fb521000640 1 Processor -- start 2024-03-06T07:29:10.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.826+0000 7fb521000640 1 -- start start 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.827+0000 7fb521000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb51c0d4a10 0x7fb51c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.827+0000 7fb521000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51c0b3710 con 0x7fb51c0d0c80 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.827+0000 7fb521000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51c0b39f0 con 0x7fb51c15e0b0 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.827+0000 7fb521000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c0d10f0 con 0x7fb51c0d4a10 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb51c0d4a10 0x7fb51c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb51c0d0c80 0x7fb51c0b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56670/0 (socket says 172.21.15.32:56670) 2024-03-06T07:29:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51be00640 1 -- 172.21.15.32:0/3118239304 learned_addr learned my addr 172.21.15.32:0/3118239304 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51aa00640 1 -- 172.21.15.32:0/3118239304 >> v1:172.21.15.39:6789/0 conn(0x7fb51c15e0b0 legacy=0x7fb51c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51a000640 1 -- 172.21.15.32:0/3118239304 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258012595 0 0) 0x7fb51c0b3710 con 0x7fb51c0d0c80 2024-03-06T07:29:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51aa00640 1 -- 172.21.15.32:0/3118239304 >> v1:172.21.15.32:6789/0 conn(0x7fb51c0d0c80 legacy=0x7fb51c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.828+0000 7fb51aa00640 1 -- 172.21.15.32:0/3118239304 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c0b3cd0 con 0x7fb51c0d4a10 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.829+0000 7fb51aa00640 1 --2- 172.21.15.32:0/3118239304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c0d0740 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb5040097a0 tx=0x7fb50402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc8dba8587810a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb51a000640 1 -- 172.21.15.32:0/3118239304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504034070 con 0x7fb51c0d4a10 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb51a000640 1 -- 172.21.15.32:0/3118239304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504030610 con 0x7fb51c0d4a10 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb51a000640 1 -- 172.21.15.32:0/3118239304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5040309e0 con 0x7fb51c0d4a10 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 -- 172.21.15.32:0/3118239304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 msgr2=0x7fb51c0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 --2- 172.21.15.32:0/3118239304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c0d0740 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb5040097a0 tx=0x7fb50402b960 comp rx=0 tx=0).stop 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 -- 172.21.15.32:0/3118239304 shutdown_connections 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 --2- 172.21.15.32:0/3118239304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c0d0740 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 -- 172.21.15.32:0/3118239304 >> 172.21.15.32:0/3118239304 conn(0x7fb51c0d3d30 msgr2=0x7fb51c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 -- 172.21.15.32:0/3118239304 shutdown_connections 2024-03-06T07:29:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.830+0000 7fb521000640 1 -- 172.21.15.32:0/3118239304 wait complete. 2024-03-06T07:29:10.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.832+0000 7fb521000640 1 Processor -- start 2024-03-06T07:29:10.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.832+0000 7fb521000640 1 -- start start 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb521000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb51c0d0c80 0x7fb51c0b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb521000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb521000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb51c15e0b0 0x7fb51c1f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb521000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0cfa20 con 0x7fb51c0d4a10 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb521000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c15e490 con 0x7fb51c0d0c80 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb521000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0cfc60 con 0x7fb51c15e0b0 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.833+0000 7fb51aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c1f3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51086/0 (socket says 172.21.15.32:51086) 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 -- 172.21.15.32:0/3631177762 learned_addr learned my addr 172.21.15.32:0/3631177762 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51be00640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb51c15e0b0 0x7fb51c1f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51b400640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb51c0d0c80 0x7fb51c0b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 -- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb51c15e0b0 msgr2=0x7fb51c1f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb51c15e0b0 0x7fb51c1f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 -- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb51c0d0c80 msgr2=0x7fb51c0b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb51c0d0c80 0x7fb51c0b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 -- 172.21.15.32:0/3631177762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c1f4d70 con 0x7fb51c0d4a10 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.834+0000 7fb51aa00640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c1f3a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb50402bf70 tx=0x7fb5040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.835+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504034040 con 0x7fb51c0d4a10 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.835+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504039a10 con 0x7fb51c0d4a10 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.835+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504039de0 con 0x7fb51c0d4a10 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.835+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c1f47d0 con 0x7fb51c0d4a10 2024-03-06T07:29:10.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.835+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c1f5310 con 0x7fb51c0d4a10 2024-03-06T07:29:10.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.836+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c04f330 con 0x7fb51c0d4a10 2024-03-06T07:29:10.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.837+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb504037070 con 0x7fb51c0d4a10 2024-03-06T07:29:10.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.837+0000 7fb513400640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb4f806b4d0 0x7fb4f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.838+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5040ac450 con 0x7fb51c0d4a10 2024-03-06T07:29:10.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.838+0000 7fb51b400640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb4f806b4d0 0x7fb4f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.838+0000 7fb51b400640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb4f806b4d0 0x7fb4f806d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0012e0 tx=0x7fb50c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:10.843+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb504037360 con 0x7fb51c0d4a10 2024-03-06T07:29:11.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.031+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb51c0b15a0 con 0x7fb4f806b4d0 2024-03-06T07:29:11.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.039+0000 7fb513400640 1 -- 172.21.15.32:0/3631177762 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251838 (secure 0 0 0) 0x7fb51c0b15a0 con 0x7fb4f806b4d0 2024-03-06T07:29:11.045 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:11.046 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:29:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb4f806b4d0 msgr2=0x7fb4f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb4f806b4d0 0x7fb4f806d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0012e0 tx=0x7fb50c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 msgr2=0x7fb51c1f3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c1f3a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb50402bf70 tx=0x7fb5040046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 shutdown_connections 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb4f806b4d0 0x7fb4f806d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb51c15e0b0 0x7fb51c1f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb51c0d4a10 0x7fb51c1f3a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 --2- 172.21.15.32:0/3631177762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb51c0d0c80 0x7fb51c0b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 >> 172.21.15.32:0/3631177762 conn(0x7fb51c0d3d30 msgr2=0x7fb51c0b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 shutdown_connections 2024-03-06T07:29:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.044+0000 7fb521000640 1 -- 172.21.15.32:0/3631177762 wait complete. 2024-03-06T07:29:11.063 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:29:10.188209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252724,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721996,"statfs":{"total":773094113280,"available":772835323904,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.906764"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306066+0000","last_change":"2024-03-06T07:29:01.317028+0000","last_active":"2024-03-06T07:29:02.306066+0000","last_peered":"2024-03-06T07:29:02.306066+0000","last_clean":"2024-03-06T07:29:02.306066+0000","last_became_active":"2024-03-06T07:29:01.316273+0000","last_became_peered":"2024-03-06T07:29:01.316273+0000","last_unstale":"2024-03-06T07:29:02.306066+0000","last_undegraded":"2024-03-06T07:29:02.306066+0000","last_fullsized":"2024-03-06T07:29:02.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338412+0000","last_change":"2024-03-06T07:29:01.311231+0000","last_active":"2024-03-06T07:29:02.338412+0000","last_peered":"2024-03-06T07:29:02.338412+0000","last_clean":"2024-03-06T07:29:02.338412+0000","last_became_active":"2024-03-06T07:29:01.310868+0000","last_became_peered":"2024-03-06T07:29:01.310868+0000","last_unstale":"2024-03-06T07:29:02.338412+0000","last_undegraded":"2024-03-06T07:29:02.338412+0000","last_fullsized":"2024-03-06T07:29:02.338412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856022+0000","last_change":"2024-03-06T07:29:01.307601+0000","last_active":"2024-03-06T07:29:02.856022+0000","last_peered":"2024-03-06T07:29:02.856022+0000","last_clean":"2024-03-06T07:29:02.856022+0000","last_became_active":"2024-03-06T07:29:01.307310+0000","last_became_peered":"2024-03-06T07:29:01.307310+0000","last_unstale":"2024-03-06T07:29:02.856022+0000","last_undegraded":"2024-03-06T07:29:02.856022+0000","last_fullsized":"2024-03-06T07:29:02.856022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331881+0000","last_change":"2024-03-06T07:29:01.326299+0000","last_active":"2024-03-06T07:29:02.331881+0000","last_peered":"2024-03-06T07:29:02.331881+0000","last_clean":"2024-03-06T07:29:02.331881+0000","last_became_active":"2024-03-06T07:29:01.326026+0000","last_became_peered":"2024-03-06T07:29:01.326026+0000","last_unstale":"2024-03-06T07:29:02.331881+0000","last_undegraded":"2024-03-06T07:29:02.331881+0000","last_fullsized":"2024-03-06T07:29:02.331881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304952+0000","last_change":"2024-03-06T07:29:01.314492+0000","last_active":"2024-03-06T07:29:02.304952+0000","last_peered":"2024-03-06T07:29:02.304952+0000","last_clean":"2024-03-06T07:29:02.304952+0000","last_became_active":"2024-03-06T07:29:01.313926+0000","last_became_peered":"2024-03-06T07:29:01.313926+0000","last_unstale":"2024-03-06T07:29:02.304952+0000","last_undegraded":"2024-03-06T07:29:02.304952+0000","last_fullsized":"2024-03-06T07:29:02.304952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331429+0000","last_change":"2024-03-06T07:29:01.328648+0000","last_active":"2024-03-06T07:29:02.331429+0000","last_peered":"2024-03-06T07:29:02.331429+0000","last_clean":"2024-03-06T07:29:02.331429+0000","last_became_active":"2024-03-06T07:29:01.328347+0000","last_became_peered":"2024-03-06T07:29:01.328347+0000","last_unstale":"2024-03-06T07:29:02.331429+0000","last_undegraded":"2024-03-06T07:29:02.331429+0000","last_fullsized":"2024-03-06T07:29:02.331429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304834+0000","last_change":"2024-03-06T07:29:01.310565+0000","last_active":"2024-03-06T07:29:02.304834+0000","last_peered":"2024-03-06T07:29:02.304834+0000","last_clean":"2024-03-06T07:29:02.304834+0000","last_became_active":"2024-03-06T07:29:01.310220+0000","last_became_peered":"2024-03-06T07:29:01.310220+0000","last_unstale":"2024-03-06T07:29:02.304834+0000","last_undegraded":"2024-03-06T07:29:02.304834+0000","last_fullsized":"2024-03-06T07:29:02.304834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.325938+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.325579+0000","last_became_peered":"2024-03-06T07:29:01.325579+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306146+0000","last_change":"2024-03-06T07:29:01.341575+0000","last_active":"2024-03-06T07:29:02.306146+0000","last_peered":"2024-03-06T07:29:02.306146+0000","last_clean":"2024-03-06T07:29:02.306146+0000","last_became_active":"2024-03-06T07:29:01.341372+0000","last_became_peered":"2024-03-06T07:29:01.341372+0000","last_unstale":"2024-03-06T07:29:02.306146+0000","last_undegraded":"2024-03-06T07:29:02.306146+0000","last_fullsized":"2024-03-06T07:29:02.306146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344143+0000","last_change":"2024-03-06T07:29:01.337188+0000","last_active":"2024-03-06T07:29:02.344143+0000","last_peered":"2024-03-06T07:29:02.344143+0000","last_clean":"2024-03-06T07:29:02.344143+0000","last_became_active":"2024-03-06T07:29:01.336845+0000","last_became_peered":"2024-03-06T07:29:01.336845+0000","last_unstale":"2024-03-06T07:29:02.344143+0000","last_undegraded":"2024-03-06T07:29:02.344143+0000","last_fullsized":"2024-03-06T07:29:02.344143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343789+0000","last_change":"2024-03-06T07:29:01.340996+0000","last_active":"2024-03-06T07:29:02.343789+0000","last_peered":"2024-03-06T07:29:02.343789+0000","last_clean":"2024-03-06T07:29:02.343789+0000","last_became_active":"2024-03-06T07:29:01.336560+0000","last_became_peered":"2024-03-06T07:29:01.336560+0000","last_unstale":"2024-03-06T07:29:02.343789+0000","last_undegraded":"2024-03-06T07:29:02.343789+0000","last_fullsized":"2024-03-06T07:29:02.343789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310075+0000","last_change":"2024-03-06T07:29:01.335731+0000","last_active":"2024-03-06T07:29:02.310075+0000","last_peered":"2024-03-06T07:29:02.310075+0000","last_clean":"2024-03-06T07:29:02.310075+0000","last_became_active":"2024-03-06T07:29:01.331758+0000","last_became_peered":"2024-03-06T07:29:01.331758+0000","last_unstale":"2024-03-06T07:29:02.310075+0000","last_undegraded":"2024-03-06T07:29:02.310075+0000","last_fullsized":"2024-03-06T07:29:02.310075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:19.716910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070024+0000","last_change":"2024-03-06T07:29:01.317550+0000","last_active":"2024-03-06T07:29:03.070024+0000","last_peered":"2024-03-06T07:29:03.070024+0000","last_clean":"2024-03-06T07:29:03.070024+0000","last_became_active":"2024-03-06T07:29:01.317098+0000","last_became_peered":"2024-03-06T07:29:01.317098+0000","last_unstale":"2024-03-06T07:29:03.070024+0000","last_undegraded":"2024-03-06T07:29:03.070024+0000","last_fullsized":"2024-03-06T07:29:03.070024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855737+0000","last_change":"2024-03-06T07:29:01.306850+0000","last_active":"2024-03-06T07:29:02.855737+0000","last_peered":"2024-03-06T07:29:02.855737+0000","last_clean":"2024-03-06T07:29:02.855737+0000","last_became_active":"2024-03-06T07:29:01.306532+0000","last_became_peered":"2024-03-06T07:29:01.306532+0000","last_unstale":"2024-03-06T07:29:02.855737+0000","last_undegraded":"2024-03-06T07:29:02.855737+0000","last_fullsized":"2024-03-06T07:29:02.855737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:02.899318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.337987+0000","last_change":"2024-03-06T07:29:01.313116+0000","last_active":"2024-03-06T07:29:02.337987+0000","last_peered":"2024-03-06T07:29:02.337987+0000","last_clean":"2024-03-06T07:29:02.337987+0000","last_became_active":"2024-03-06T07:29:01.311783+0000","last_became_peered":"2024-03-06T07:29:01.311783+0000","last_unstale":"2024-03-06T07:29:02.337987+0000","last_undegraded":"2024-03-06T07:29:02.337987+0000","last_fullsized":"2024-03-06T07:29:02.337987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331678+0000","last_change":"2024-03-06T07:29:01.323368+0000","last_active":"2024-03-06T07:29:02.331678+0000","last_peered":"2024-03-06T07:29:02.331678+0000","last_clean":"2024-03-06T07:29:02.331678+0000","last_became_active":"2024-03-06T07:29:01.323066+0000","last_became_peered":"2024-03-06T07:29:01.323066+0000","last_unstale":"2024-03-06T07:29:02.331678+0000","last_undegraded":"2024-03-06T07:29:02.331678+0000","last_fullsized":"2024-03-06T07:29:02.331678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332654+0000","last_change":"2024-03-06T07:29:01.332332+0000","last_active":"2024-03-06T07:29:02.332654+0000","last_peered":"2024-03-06T07:29:02.332654+0000","last_clean":"2024-03-06T07:29:02.332654+0000","last_became_active":"2024-03-06T07:29:01.331999+0000","last_became_peered":"2024-03-06T07:29:01.331999+0000","last_unstale":"2024-03-06T07:29:02.332654+0000","last_undegraded":"2024-03-06T07:29:02.332654+0000","last_fullsized":"2024-03-06T07:29:02.332654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069247+0000","last_change":"2024-03-06T07:29:01.319453+0000","last_active":"2024-03-06T07:29:03.069247+0000","last_peered":"2024-03-06T07:29:03.069247+0000","last_clean":"2024-03-06T07:29:03.069247+0000","last_became_active":"2024-03-06T07:29:01.319182+0000","last_became_peered":"2024-03-06T07:29:01.319182+0000","last_unstale":"2024-03-06T07:29:03.069247+0000","last_undegraded":"2024-03-06T07:29:03.069247+0000","last_fullsized":"2024-03-06T07:29:03.069247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:42.291305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305802+0000","last_change":"2024-03-06T07:29:01.304420+0000","last_active":"2024-03-06T07:29:02.305802+0000","last_peered":"2024-03-06T07:29:02.305802+0000","last_clean":"2024-03-06T07:29:02.305802+0000","last_became_active":"2024-03-06T07:29:01.303897+0000","last_became_peered":"2024-03-06T07:29:01.303897+0000","last_unstale":"2024-03-06T07:29:02.305802+0000","last_undegraded":"2024-03-06T07:29:02.305802+0000","last_fullsized":"2024-03-06T07:29:02.305802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343811+0000","last_change":"2024-03-06T07:29:01.340804+0000","last_active":"2024-03-06T07:29:02.343811+0000","last_peered":"2024-03-06T07:29:02.343811+0000","last_clean":"2024-03-06T07:29:02.343811+0000","last_became_active":"2024-03-06T07:29:01.338819+0000","last_became_peered":"2024-03-06T07:29:01.338819+0000","last_unstale":"2024-03-06T07:29:02.343811+0000","last_undegraded":"2024-03-06T07:29:02.343811+0000","last_fullsized":"2024-03-06T07:29:02.343811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856684+0000","last_change":"2024-03-06T07:29:01.329232+0000","last_active":"2024-03-06T07:29:02.856684+0000","last_peered":"2024-03-06T07:29:02.856684+0000","last_clean":"2024-03-06T07:29:02.856684+0000","last_became_active":"2024-03-06T07:29:01.328855+0000","last_became_peered":"2024-03-06T07:29:01.328855+0000","last_unstale":"2024-03-06T07:29:02.856684+0000","last_undegraded":"2024-03-06T07:29:02.856684+0000","last_fullsized":"2024-03-06T07:29:02.856684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331471+0000","last_change":"2024-03-06T07:29:01.334776+0000","last_active":"2024-03-06T07:29:02.331471+0000","last_peered":"2024-03-06T07:29:02.331471+0000","last_clean":"2024-03-06T07:29:02.331471+0000","last_became_active":"2024-03-06T07:29:01.332072+0000","last_became_peered":"2024-03-06T07:29:01.332072+0000","last_unstale":"2024-03-06T07:29:02.331471+0000","last_undegraded":"2024-03-06T07:29:02.331471+0000","last_fullsized":"2024-03-06T07:29:02.331471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.312963+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.312645+0000","last_became_peered":"2024-03-06T07:29:01.312645+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331763+0000","last_change":"2024-03-06T07:29:01.322856+0000","last_active":"2024-03-06T07:29:02.331763+0000","last_peered":"2024-03-06T07:29:02.331763+0000","last_clean":"2024-03-06T07:29:02.331763+0000","last_became_active":"2024-03-06T07:29:01.322485+0000","last_became_peered":"2024-03-06T07:29:01.322485+0000","last_unstale":"2024-03-06T07:29:02.331763+0000","last_undegraded":"2024-03-06T07:29:02.331763+0000","last_fullsized":"2024-03-06T07:29:02.331763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070502+0000","last_change":"2024-03-06T07:29:01.328113+0000","last_active":"2024-03-06T07:29:03.070502+0000","last_peered":"2024-03-06T07:29:03.070502+0000","last_clean":"2024-03-06T07:29:03.070502+0000","last_became_active":"2024-03-06T07:29:01.327700+0000","last_became_peered":"2024-03-06T07:29:01.327700+0000","last_unstale":"2024-03-06T07:29:03.070502+0000","last_undegraded":"2024-03-06T07:29:03.070502+0000","last_fullsized":"2024-03-06T07:29:03.070502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344258+0000","last_change":"2024-03-06T07:29:01.328971+0000","last_active":"2024-03-06T07:29:02.344258+0000","last_peered":"2024-03-06T07:29:02.344258+0000","last_clean":"2024-03-06T07:29:02.344258+0000","last_became_active":"2024-03-06T07:29:01.328471+0000","last_became_peered":"2024-03-06T07:29:01.328471+0000","last_unstale":"2024-03-06T07:29:02.344258+0000","last_undegraded":"2024-03-06T07:29:02.344258+0000","last_fullsized":"2024-03-06T07:29:02.344258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338529+0000","last_change":"2024-03-06T07:29:01.310033+0000","last_active":"2024-03-06T07:29:02.338529+0000","last_peered":"2024-03-06T07:29:02.338529+0000","last_clean":"2024-03-06T07:29:02.338529+0000","last_became_active":"2024-03-06T07:29:01.309715+0000","last_became_peered":"2024-03-06T07:29:01.309715+0000","last_unstale":"2024-03-06T07:29:02.338529+0000","last_undegraded":"2024-03-06T07:29:02.338529+0000","last_fullsized":"2024-03-06T07:29:02.338529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069900+0000","last_change":"2024-03-06T07:29:01.338107+0000","last_active":"2024-03-06T07:29:03.069900+0000","last_peered":"2024-03-06T07:29:03.069900+0000","last_clean":"2024-03-06T07:29:03.069900+0000","last_became_active":"2024-03-06T07:29:01.337793+0000","last_became_peered":"2024-03-06T07:29:01.337793+0000","last_unstale":"2024-03-06T07:29:03.069900+0000","last_undegraded":"2024-03-06T07:29:03.069900+0000","last_fullsized":"2024-03-06T07:29:03.069900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856096+0000","last_change":"2024-03-06T07:29:01.309168+0000","last_active":"2024-03-06T07:29:02.856096+0000","last_peered":"2024-03-06T07:29:02.856096+0000","last_clean":"2024-03-06T07:29:02.856096+0000","last_became_active":"2024-03-06T07:29:01.308853+0000","last_became_peered":"2024-03-06T07:29:01.308853+0000","last_unstale":"2024-03-06T07:29:02.856096+0000","last_undegraded":"2024-03-06T07:29:02.856096+0000","last_fullsized":"2024-03-06T07:29:02.856096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332020+0000","last_change":"2024-03-06T07:29:01.339291+0000","last_active":"2024-03-06T07:29:02.332020+0000","last_peered":"2024-03-06T07:29:02.332020+0000","last_clean":"2024-03-06T07:29:02.332020+0000","last_became_active":"2024-03-06T07:29:01.338961+0000","last_became_peered":"2024-03-06T07:29:01.338961+0000","last_unstale":"2024-03-06T07:29:02.332020+0000","last_undegraded":"2024-03-06T07:29:02.332020+0000","last_fullsized":"2024-03-06T07:29:02.332020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068361+0000","last_change":"2024-03-06T07:29:01.339412+0000","last_active":"2024-03-06T07:29:03.068361+0000","last_peered":"2024-03-06T07:29:03.068361+0000","last_clean":"2024-03-06T07:29:03.068361+0000","last_became_active":"2024-03-06T07:29:01.338930+0000","last_became_peered":"2024-03-06T07:29:01.338930+0000","last_unstale":"2024-03-06T07:29:03.068361+0000","last_undegraded":"2024-03-06T07:29:03.068361+0000","last_fullsized":"2024-03-06T07:29:03.068361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343463+0000","last_change":"2024-03-06T07:29:01.316617+0000","last_active":"2024-03-06T07:29:02.343463+0000","last_peered":"2024-03-06T07:29:02.343463+0000","last_clean":"2024-03-06T07:29:02.343463+0000","last_became_active":"2024-03-06T07:29:01.315094+0000","last_became_peered":"2024-03-06T07:29:01.315094+0000","last_unstale":"2024-03-06T07:29:02.343463+0000","last_undegraded":"2024-03-06T07:29:02.343463+0000","last_fullsized":"2024-03-06T07:29:02.343463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309312+0000","last_change":"2024-03-06T07:29:01.314610+0000","last_active":"2024-03-06T07:29:02.309312+0000","last_peered":"2024-03-06T07:29:02.309312+0000","last_clean":"2024-03-06T07:29:02.309312+0000","last_became_active":"2024-03-06T07:29:01.314249+0000","last_became_peered":"2024-03-06T07:29:01.314249+0000","last_unstale":"2024-03-06T07:29:02.309312+0000","last_undegraded":"2024-03-06T07:29:02.309312+0000","last_fullsized":"2024-03-06T07:29:02.309312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:32.840745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070367+0000","last_change":"2024-03-06T07:29:00.318669+0000","last_active":"2024-03-06T07:29:03.070367+0000","last_peered":"2024-03-06T07:29:03.070367+0000","last_clean":"2024-03-06T07:29:03.070367+0000","last_became_active":"2024-03-06T07:29:00.318368+0000","last_became_peered":"2024-03-06T07:29:00.318368+0000","last_unstale":"2024-03-06T07:29:03.070367+0000","last_undegraded":"2024-03-06T07:29:03.070367+0000","last_fullsized":"2024-03-06T07:29:03.070367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856486+0000","last_change":"2024-03-06T07:29:01.330131+0000","last_active":"2024-03-06T07:29:02.856486+0000","last_peered":"2024-03-06T07:29:02.856486+0000","last_clean":"2024-03-06T07:29:02.856486+0000","last_became_active":"2024-03-06T07:29:01.329564+0000","last_became_peered":"2024-03-06T07:29:01.329564+0000","last_unstale":"2024-03-06T07:29:02.856486+0000","last_undegraded":"2024-03-06T07:29:02.856486+0000","last_fullsized":"2024-03-06T07:29:02.856486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343246+0000","last_change":"2024-03-06T07:29:00.325104+0000","last_active":"2024-03-06T07:29:02.343246+0000","last_peered":"2024-03-06T07:29:02.343246+0000","last_clean":"2024-03-06T07:29:02.343246+0000","last_became_active":"2024-03-06T07:29:00.324737+0000","last_became_peered":"2024-03-06T07:29:00.324737+0000","last_unstale":"2024-03-06T07:29:02.343246+0000","last_undegraded":"2024-03-06T07:29:02.343246+0000","last_fullsized":"2024-03-06T07:29:02.343246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068981+0000","last_change":"2024-03-06T07:29:00.322124+0000","last_active":"2024-03-06T07:29:03.068981+0000","last_peered":"2024-03-06T07:29:03.068981+0000","last_clean":"2024-03-06T07:29:03.068981+0000","last_became_active":"2024-03-06T07:29:00.321658+0000","last_became_peered":"2024-03-06T07:29:00.321658+0000","last_unstale":"2024-03-06T07:29:03.068981+0000","last_undegraded":"2024-03-06T07:29:03.068981+0000","last_fullsized":"2024-03-06T07:29:03.068981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069549+0000","last_change":"2024-03-06T07:29:01.306201+0000","last_active":"2024-03-06T07:29:03.069549+0000","last_peered":"2024-03-06T07:29:03.069549+0000","last_clean":"2024-03-06T07:29:03.069549+0000","last_became_active":"2024-03-06T07:29:01.305652+0000","last_became_peered":"2024-03-06T07:29:01.305652+0000","last_unstale":"2024-03-06T07:29:03.069549+0000","last_undegraded":"2024-03-06T07:29:03.069549+0000","last_fullsized":"2024-03-06T07:29:03.069549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331796+0000","last_change":"2024-03-06T07:29:00.330308+0000","last_active":"2024-03-06T07:29:02.331796+0000","last_peered":"2024-03-06T07:29:02.331796+0000","last_clean":"2024-03-06T07:29:02.331796+0000","last_became_active":"2024-03-06T07:29:00.329727+0000","last_became_peered":"2024-03-06T07:29:00.329727+0000","last_unstale":"2024-03-06T07:29:02.331796+0000","last_undegraded":"2024-03-06T07:29:02.331796+0000","last_fullsized":"2024-03-06T07:29:02.331796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338071+0000","last_change":"2024-03-06T07:29:01.308157+0000","last_active":"2024-03-06T07:29:02.338071+0000","last_peered":"2024-03-06T07:29:02.338071+0000","last_clean":"2024-03-06T07:29:02.338071+0000","last_became_active":"2024-03-06T07:29:01.307819+0000","last_became_peered":"2024-03-06T07:29:01.307819+0000","last_unstale":"2024-03-06T07:29:02.338071+0000","last_undegraded":"2024-03-06T07:29:02.338071+0000","last_fullsized":"2024-03-06T07:29:02.338071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.334131+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.333816+0000","last_became_peered":"2024-03-06T07:29:00.333816+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332394+0000","last_change":"2024-03-06T07:29:01.332596+0000","last_active":"2024-03-06T07:29:02.332394+0000","last_peered":"2024-03-06T07:29:02.332394+0000","last_clean":"2024-03-06T07:29:02.332394+0000","last_became_active":"2024-03-06T07:29:01.332140+0000","last_became_peered":"2024-03-06T07:29:01.332140+0000","last_unstale":"2024-03-06T07:29:02.332394+0000","last_undegraded":"2024-03-06T07:29:02.332394+0000","last_fullsized":"2024-03-06T07:29:02.332394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332369+0000","last_change":"2024-03-06T07:29:00.332773+0000","last_active":"2024-03-06T07:29:02.332369+0000","last_peered":"2024-03-06T07:29:02.332369+0000","last_clean":"2024-03-06T07:29:02.332369+0000","last_became_active":"2024-03-06T07:29:00.332477+0000","last_became_peered":"2024-03-06T07:29:00.332477+0000","last_unstale":"2024-03-06T07:29:02.332369+0000","last_undegraded":"2024-03-06T07:29:02.332369+0000","last_fullsized":"2024-03-06T07:29:02.332369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856820+0000","last_change":"2024-03-06T07:29:01.325722+0000","last_active":"2024-03-06T07:29:02.856820+0000","last_peered":"2024-03-06T07:29:02.856820+0000","last_clean":"2024-03-06T07:29:02.856820+0000","last_became_active":"2024-03-06T07:29:01.325401+0000","last_became_peered":"2024-03-06T07:29:01.325401+0000","last_unstale":"2024-03-06T07:29:02.856820+0000","last_undegraded":"2024-03-06T07:29:02.856820+0000","last_fullsized":"2024-03-06T07:29:02.856820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338454+0000","last_change":"2024-03-06T07:29:00.331332+0000","last_active":"2024-03-06T07:29:02.338454+0000","last_peered":"2024-03-06T07:29:02.338454+0000","last_clean":"2024-03-06T07:29:02.338454+0000","last_became_active":"2024-03-06T07:29:00.331008+0000","last_became_peered":"2024-03-06T07:29:00.331008+0000","last_unstale":"2024-03-06T07:29:02.338454+0000","last_undegraded":"2024-03-06T07:29:02.338454+0000","last_fullsized":"2024-03-06T07:29:02.338454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310152+0000","last_change":"2024-03-06T07:29:01.335452+0000","last_active":"2024-03-06T07:29:02.310152+0000","last_peered":"2024-03-06T07:29:02.310152+0000","last_clean":"2024-03-06T07:29:02.310152+0000","last_became_active":"2024-03-06T07:29:01.331241+0000","last_became_peered":"2024-03-06T07:29:01.331241+0000","last_unstale":"2024-03-06T07:29:02.310152+0000","last_undegraded":"2024-03-06T07:29:02.310152+0000","last_fullsized":"2024-03-06T07:29:02.310152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343888+0000","last_change":"2024-03-06T07:29:00.322219+0000","last_active":"2024-03-06T07:29:02.343888+0000","last_peered":"2024-03-06T07:29:02.343888+0000","last_clean":"2024-03-06T07:29:02.343888+0000","last_became_active":"2024-03-06T07:29:00.321921+0000","last_became_peered":"2024-03-06T07:29:00.321921+0000","last_unstale":"2024-03-06T07:29:02.343888+0000","last_undegraded":"2024-03-06T07:29:02.343888+0000","last_fullsized":"2024-03-06T07:29:02.343888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304565+0000","last_change":"2024-03-06T07:29:01.309672+0000","last_active":"2024-03-06T07:29:02.304565+0000","last_peered":"2024-03-06T07:29:02.304565+0000","last_clean":"2024-03-06T07:29:02.304565+0000","last_became_active":"2024-03-06T07:29:01.309369+0000","last_became_peered":"2024-03-06T07:29:01.309369+0000","last_unstale":"2024-03-06T07:29:02.304565+0000","last_undegraded":"2024-03-06T07:29:02.304565+0000","last_fullsized":"2024-03-06T07:29:02.304565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856650+0000","last_change":"2024-03-06T07:29:00.324810+0000","last_active":"2024-03-06T07:29:02.856650+0000","last_peered":"2024-03-06T07:29:02.856650+0000","last_clean":"2024-03-06T07:29:02.856650+0000","last_became_active":"2024-03-06T07:29:00.324524+0000","last_became_peered":"2024-03-06T07:29:00.324524+0000","last_unstale":"2024-03-06T07:29:02.856650+0000","last_undegraded":"2024-03-06T07:29:02.856650+0000","last_fullsized":"2024-03-06T07:29:02.856650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:20.411708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855986+0000","last_change":"2024-03-06T07:29:01.329235+0000","last_active":"2024-03-06T07:29:02.855986+0000","last_peered":"2024-03-06T07:29:02.855986+0000","last_clean":"2024-03-06T07:29:02.855986+0000","last_became_active":"2024-03-06T07:29:01.328953+0000","last_became_peered":"2024-03-06T07:29:01.328953+0000","last_unstale":"2024-03-06T07:29:02.855986+0000","last_undegraded":"2024-03-06T07:29:02.855986+0000","last_fullsized":"2024-03-06T07:29:02.855986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331601+0000","last_change":"2024-03-06T07:29:00.325328+0000","last_active":"2024-03-06T07:29:02.331601+0000","last_peered":"2024-03-06T07:29:02.331601+0000","last_clean":"2024-03-06T07:29:02.331601+0000","last_became_active":"2024-03-06T07:29:00.323961+0000","last_became_peered":"2024-03-06T07:29:00.323961+0000","last_unstale":"2024-03-06T07:29:02.331601+0000","last_undegraded":"2024-03-06T07:29:02.331601+0000","last_fullsized":"2024-03-06T07:29:02.331601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305341+0000","last_change":"2024-03-06T07:29:01.314222+0000","last_active":"2024-03-06T07:29:02.305341+0000","last_peered":"2024-03-06T07:29:02.305341+0000","last_clean":"2024-03-06T07:29:02.305341+0000","last_became_active":"2024-03-06T07:29:01.313409+0000","last_became_peered":"2024-03-06T07:29:01.313409+0000","last_unstale":"2024-03-06T07:29:02.305341+0000","last_undegraded":"2024-03-06T07:29:02.305341+0000","last_fullsized":"2024-03-06T07:29:02.305341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343329+0000","last_change":"2024-03-06T07:29:00.326529+0000","last_active":"2024-03-06T07:29:02.343329+0000","last_peered":"2024-03-06T07:29:02.343329+0000","last_clean":"2024-03-06T07:29:02.343329+0000","last_became_active":"2024-03-06T07:29:00.326202+0000","last_became_peered":"2024-03-06T07:29:00.326202+0000","last_unstale":"2024-03-06T07:29:02.343329+0000","last_undegraded":"2024-03-06T07:29:02.343329+0000","last_fullsized":"2024-03-06T07:29:02.343329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343344+0000","last_change":"2024-03-06T07:29:01.316333+0000","last_active":"2024-03-06T07:29:02.343344+0000","last_peered":"2024-03-06T07:29:02.343344+0000","last_clean":"2024-03-06T07:29:02.343344+0000","last_became_active":"2024-03-06T07:29:01.315969+0000","last_became_peered":"2024-03-06T07:29:01.315969+0000","last_unstale":"2024-03-06T07:29:02.343344+0000","last_undegraded":"2024-03-06T07:29:02.343344+0000","last_fullsized":"2024-03-06T07:29:02.343344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069217+0000","last_change":"2024-03-06T07:29:00.306241+0000","last_active":"2024-03-06T07:29:03.069217+0000","last_peered":"2024-03-06T07:29:03.069217+0000","last_clean":"2024-03-06T07:29:03.069217+0000","last_became_active":"2024-03-06T07:29:00.305883+0000","last_became_peered":"2024-03-06T07:29:00.305883+0000","last_unstale":"2024-03-06T07:29:03.069217+0000","last_undegraded":"2024-03-06T07:29:03.069217+0000","last_fullsized":"2024-03-06T07:29:03.069217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306244+0000","last_change":"2024-03-06T07:29:01.326203+0000","last_active":"2024-03-06T07:29:02.306244+0000","last_peered":"2024-03-06T07:29:02.306244+0000","last_clean":"2024-03-06T07:29:02.306244+0000","last_became_active":"2024-03-06T07:29:01.325861+0000","last_became_peered":"2024-03-06T07:29:01.325861+0000","last_unstale":"2024-03-06T07:29:02.306244+0000","last_undegraded":"2024-03-06T07:29:02.306244+0000","last_fullsized":"2024-03-06T07:29:02.306244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310043+0000","last_change":"2024-03-06T07:29:00.320015+0000","last_active":"2024-03-06T07:29:02.310043+0000","last_peered":"2024-03-06T07:29:02.310043+0000","last_clean":"2024-03-06T07:29:02.310043+0000","last_became_active":"2024-03-06T07:29:00.319492+0000","last_became_peered":"2024-03-06T07:29:00.319492+0000","last_unstale":"2024-03-06T07:29:02.310043+0000","last_undegraded":"2024-03-06T07:29:02.310043+0000","last_fullsized":"2024-03-06T07:29:02.310043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:49.576279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304906+0000","last_change":"2024-03-06T07:29:01.325857+0000","last_active":"2024-03-06T07:29:02.304906+0000","last_peered":"2024-03-06T07:29:02.304906+0000","last_clean":"2024-03-06T07:29:02.304906+0000","last_became_active":"2024-03-06T07:29:01.325522+0000","last_became_peered":"2024-03-06T07:29:01.325522+0000","last_unstale":"2024-03-06T07:29:02.304906+0000","last_undegraded":"2024-03-06T07:29:02.304906+0000","last_fullsized":"2024-03-06T07:29:02.304906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069075+0000","last_change":"2024-03-06T07:29:00.311734+0000","last_active":"2024-03-06T07:29:03.069075+0000","last_peered":"2024-03-06T07:29:03.069075+0000","last_clean":"2024-03-06T07:29:03.069075+0000","last_became_active":"2024-03-06T07:29:00.309932+0000","last_became_peered":"2024-03-06T07:29:00.309932+0000","last_unstale":"2024-03-06T07:29:03.069075+0000","last_undegraded":"2024-03-06T07:29:03.069075+0000","last_fullsized":"2024-03-06T07:29:03.069075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306517+0000","last_change":"2024-03-06T07:29:01.331125+0000","last_active":"2024-03-06T07:29:02.306517+0000","last_peered":"2024-03-06T07:29:02.306517+0000","last_clean":"2024-03-06T07:29:02.306517+0000","last_became_active":"2024-03-06T07:29:01.330860+0000","last_became_peered":"2024-03-06T07:29:01.330860+0000","last_unstale":"2024-03-06T07:29:02.306517+0000","last_undegraded":"2024-03-06T07:29:02.306517+0000","last_fullsized":"2024-03-06T07:29:02.306517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:44.856694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338728+0000","last_change":"2024-03-06T07:29:00.325056+0000","last_active":"2024-03-06T07:29:02.338728+0000","last_peered":"2024-03-06T07:29:02.338728+0000","last_clean":"2024-03-06T07:29:02.338728+0000","last_became_active":"2024-03-06T07:29:00.324757+0000","last_became_peered":"2024-03-06T07:29:00.324757+0000","last_unstale":"2024-03-06T07:29:02.338728+0000","last_undegraded":"2024-03-06T07:29:02.338728+0000","last_fullsized":"2024-03-06T07:29:02.338728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332058+0000","last_change":"2024-03-06T07:29:01.326654+0000","last_active":"2024-03-06T07:29:02.332058+0000","last_peered":"2024-03-06T07:29:02.332058+0000","last_clean":"2024-03-06T07:29:02.332058+0000","last_became_active":"2024-03-06T07:29:01.325612+0000","last_became_peered":"2024-03-06T07:29:01.325612+0000","last_unstale":"2024-03-06T07:29:02.332058+0000","last_undegraded":"2024-03-06T07:29:02.332058+0000","last_fullsized":"2024-03-06T07:29:02.332058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068894+0000","last_change":"2024-03-06T07:29:00.332332+0000","last_active":"2024-03-06T07:29:03.068894+0000","last_peered":"2024-03-06T07:29:03.068894+0000","last_clean":"2024-03-06T07:29:03.068894+0000","last_became_active":"2024-03-06T07:29:00.331881+0000","last_became_peered":"2024-03-06T07:29:00.331881+0000","last_unstale":"2024-03-06T07:29:03.068894+0000","last_undegraded":"2024-03-06T07:29:03.068894+0000","last_fullsized":"2024-03-06T07:29:03.068894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304989+0000","last_change":"2024-03-06T07:29:01.336946+0000","last_active":"2024-03-06T07:29:02.304989+0000","last_peered":"2024-03-06T07:29:02.304989+0000","last_clean":"2024-03-06T07:29:02.304989+0000","last_became_active":"2024-03-06T07:29:01.336594+0000","last_became_peered":"2024-03-06T07:29:01.336594+0000","last_unstale":"2024-03-06T07:29:02.304989+0000","last_undegraded":"2024-03-06T07:29:02.304989+0000","last_fullsized":"2024-03-06T07:29:02.304989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303958+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:29:02.303958+0000","last_peered":"2024-03-06T07:29:02.303958+0000","last_clean":"2024-03-06T07:29:02.303958+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:29:02.303958+0000","last_undegraded":"2024-03-06T07:29:02.303958+0000","last_fullsized":"2024-03-06T07:29:02.303958+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331455+0000","last_change":"2024-03-06T07:29:00.331164+0000","last_active":"2024-03-06T07:29:02.331455+0000","last_peered":"2024-03-06T07:29:02.331455+0000","last_clean":"2024-03-06T07:29:02.331455+0000","last_became_active":"2024-03-06T07:29:00.330531+0000","last_became_peered":"2024-03-06T07:29:00.330531+0000","last_unstale":"2024-03-06T07:29:02.331455+0000","last_undegraded":"2024-03-06T07:29:02.331455+0000","last_fullsized":"2024-03-06T07:29:02.331455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069681+0000","last_change":"2024-03-06T07:29:01.342482+0000","last_active":"2024-03-06T07:29:03.069681+0000","last_peered":"2024-03-06T07:29:03.069681+0000","last_clean":"2024-03-06T07:29:03.069681+0000","last_became_active":"2024-03-06T07:29:01.342176+0000","last_became_peered":"2024-03-06T07:29:01.342176+0000","last_unstale":"2024-03-06T07:29:03.069681+0000","last_undegraded":"2024-03-06T07:29:03.069681+0000","last_fullsized":"2024-03-06T07:29:03.069681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303790+0000","last_change":"2024-03-06T07:29:00.308988+0000","last_active":"2024-03-06T07:29:02.303790+0000","last_peered":"2024-03-06T07:29:02.303790+0000","last_clean":"2024-03-06T07:29:02.303790+0000","last_became_active":"2024-03-06T07:29:00.304843+0000","last_became_peered":"2024-03-06T07:29:00.304843+0000","last_unstale":"2024-03-06T07:29:02.303790+0000","last_undegraded":"2024-03-06T07:29:02.303790+0000","last_fullsized":"2024-03-06T07:29:02.303790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855676+0000","last_change":"2024-03-06T07:29:01.328580+0000","last_active":"2024-03-06T07:29:02.855676+0000","last_peered":"2024-03-06T07:29:02.855676+0000","last_clean":"2024-03-06T07:29:02.855676+0000","last_became_active":"2024-03-06T07:29:01.328319+0000","last_became_peered":"2024-03-06T07:29:01.328319+0000","last_unstale":"2024-03-06T07:29:02.855676+0000","last_undegraded":"2024-03-06T07:29:02.855676+0000","last_fullsized":"2024-03-06T07:29:02.855676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:33.356914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068423+0000","last_change":"2024-03-06T07:29:00.307455+0000","last_active":"2024-03-06T07:29:03.068423+0000","last_peered":"2024-03-06T07:29:03.068423+0000","last_clean":"2024-03-06T07:29:03.068423+0000","last_became_active":"2024-03-06T07:29:00.307117+0000","last_became_peered":"2024-03-06T07:29:00.307117+0000","last_unstale":"2024-03-06T07:29:03.068423+0000","last_undegraded":"2024-03-06T07:29:03.068423+0000","last_fullsized":"2024-03-06T07:29:03.068423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338099+0000","last_change":"2024-03-06T07:29:01.298549+0000","last_active":"2024-03-06T07:29:02.338099+0000","last_peered":"2024-03-06T07:29:02.338099+0000","last_clean":"2024-03-06T07:29:02.338099+0000","last_became_active":"2024-03-06T07:29:01.298043+0000","last_became_peered":"2024-03-06T07:29:01.298043+0000","last_unstale":"2024-03-06T07:29:02.338099+0000","last_undegraded":"2024-03-06T07:29:02.338099+0000","last_fullsized":"2024-03-06T07:29:02.338099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331760+0000","last_change":"2024-03-06T07:29:00.323986+0000","last_active":"2024-03-06T07:29:02.331760+0000","last_peered":"2024-03-06T07:29:02.331760+0000","last_clean":"2024-03-06T07:29:02.331760+0000","last_became_active":"2024-03-06T07:29:00.323442+0000","last_became_peered":"2024-03-06T07:29:00.323442+0000","last_unstale":"2024-03-06T07:29:02.331760+0000","last_undegraded":"2024-03-06T07:29:02.331760+0000","last_fullsized":"2024-03-06T07:29:02.331760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309938+0000","last_change":"2024-03-06T07:29:01.319824+0000","last_active":"2024-03-06T07:29:02.309938+0000","last_peered":"2024-03-06T07:29:02.309938+0000","last_clean":"2024-03-06T07:29:02.309938+0000","last_became_active":"2024-03-06T07:29:01.317792+0000","last_became_peered":"2024-03-06T07:29:01.317792+0000","last_unstale":"2024-03-06T07:29:02.309938+0000","last_undegraded":"2024-03-06T07:29:02.309938+0000","last_fullsized":"2024-03-06T07:29:02.309938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.562992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338550+0000","last_change":"2024-03-06T07:29:00.312181+0000","last_active":"2024-03-06T07:29:02.338550+0000","last_peered":"2024-03-06T07:29:02.338550+0000","last_clean":"2024-03-06T07:29:02.338550+0000","last_became_active":"2024-03-06T07:29:00.311915+0000","last_became_peered":"2024-03-06T07:29:00.311915+0000","last_unstale":"2024-03-06T07:29:02.338550+0000","last_undegraded":"2024-03-06T07:29:02.338550+0000","last_fullsized":"2024-03-06T07:29:02.338550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332017+0000","last_change":"2024-03-06T07:29:01.332290+0000","last_active":"2024-03-06T07:29:02.332017+0000","last_peered":"2024-03-06T07:29:02.332017+0000","last_clean":"2024-03-06T07:29:02.332017+0000","last_became_active":"2024-03-06T07:29:01.331904+0000","last_became_peered":"2024-03-06T07:29:01.331904+0000","last_unstale":"2024-03-06T07:29:02.332017+0000","last_undegraded":"2024-03-06T07:29:02.332017+0000","last_fullsized":"2024-03-06T07:29:02.332017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306067+0000","last_change":"2024-03-06T07:29:00.315219+0000","last_active":"2024-03-06T07:29:02.306067+0000","last_peered":"2024-03-06T07:29:02.306067+0000","last_clean":"2024-03-06T07:29:02.306067+0000","last_became_active":"2024-03-06T07:29:00.314808+0000","last_became_peered":"2024-03-06T07:29:00.314808+0000","last_unstale":"2024-03-06T07:29:02.306067+0000","last_undegraded":"2024-03-06T07:29:02.306067+0000","last_fullsized":"2024-03-06T07:29:02.306067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338430+0000","last_change":"2024-03-06T07:29:01.303022+0000","last_active":"2024-03-06T07:29:02.338430+0000","last_peered":"2024-03-06T07:29:02.338430+0000","last_clean":"2024-03-06T07:29:02.338430+0000","last_became_active":"2024-03-06T07:29:01.302528+0000","last_became_peered":"2024-03-06T07:29:01.302528+0000","last_unstale":"2024-03-06T07:29:02.338430+0000","last_undegraded":"2024-03-06T07:29:02.338430+0000","last_fullsized":"2024-03-06T07:29:02.338430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.341932+0000","last_change":"2024-03-06T07:29:00.323588+0000","last_active":"2024-03-06T07:29:02.341932+0000","last_peered":"2024-03-06T07:29:02.341932+0000","last_clean":"2024-03-06T07:29:02.341932+0000","last_became_active":"2024-03-06T07:29:00.323304+0000","last_became_peered":"2024-03-06T07:29:00.323304+0000","last_unstale":"2024-03-06T07:29:02.341932+0000","last_undegraded":"2024-03-06T07:29:02.341932+0000","last_fullsized":"2024-03-06T07:29:02.341932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070661+0000","last_change":"2024-03-06T07:29:01.314796+0000","last_active":"2024-03-06T07:29:03.070661+0000","last_peered":"2024-03-06T07:29:03.070661+0000","last_clean":"2024-03-06T07:29:03.070661+0000","last_became_active":"2024-03-06T07:29:01.313844+0000","last_became_peered":"2024-03-06T07:29:01.313844+0000","last_unstale":"2024-03-06T07:29:03.070661+0000","last_undegraded":"2024-03-06T07:29:03.070661+0000","last_fullsized":"2024-03-06T07:29:03.070661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.332089+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.331805+0000","last_became_peered":"2024-03-06T07:29:00.331805+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856581+0000","last_change":"2024-03-06T07:29:01.309249+0000","last_active":"2024-03-06T07:29:02.856581+0000","last_peered":"2024-03-06T07:29:02.856581+0000","last_clean":"2024-03-06T07:29:02.856581+0000","last_became_active":"2024-03-06T07:29:01.308932+0000","last_became_peered":"2024-03-06T07:29:01.308932+0000","last_unstale":"2024-03-06T07:29:02.856581+0000","last_undegraded":"2024-03-06T07:29:02.856581+0000","last_fullsized":"2024-03-06T07:29:02.856581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303785+0000","last_change":"2024-03-06T07:29:00.317215+0000","last_active":"2024-03-06T07:29:02.303785+0000","last_peered":"2024-03-06T07:29:02.303785+0000","last_clean":"2024-03-06T07:29:02.303785+0000","last_became_active":"2024-03-06T07:29:00.314874+0000","last_became_peered":"2024-03-06T07:29:00.314874+0000","last_unstale":"2024-03-06T07:29:02.303785+0000","last_undegraded":"2024-03-06T07:29:02.303785+0000","last_fullsized":"2024-03-06T07:29:02.303785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309601+0000","last_change":"2024-03-06T07:29:01.328026+0000","last_active":"2024-03-06T07:29:02.309601+0000","last_peered":"2024-03-06T07:29:02.309601+0000","last_clean":"2024-03-06T07:29:02.309601+0000","last_became_active":"2024-03-06T07:29:01.327519+0000","last_became_peered":"2024-03-06T07:29:01.327519+0000","last_unstale":"2024-03-06T07:29:02.309601+0000","last_undegraded":"2024-03-06T07:29:02.309601+0000","last_fullsized":"2024-03-06T07:29:02.309601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:29.282488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338149+0000","last_change":"2024-03-06T07:29:00.318811+0000","last_active":"2024-03-06T07:29:02.338149+0000","last_peered":"2024-03-06T07:29:02.338149+0000","last_clean":"2024-03-06T07:29:02.338149+0000","last_became_active":"2024-03-06T07:29:00.318251+0000","last_became_peered":"2024-03-06T07:29:00.318251+0000","last_unstale":"2024-03-06T07:29:02.338149+0000","last_undegraded":"2024-03-06T07:29:02.338149+0000","last_fullsized":"2024-03-06T07:29:02.338149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069507+0000","last_change":"2024-03-06T07:29:01.298030+0000","last_active":"2024-03-06T07:29:03.069507+0000","last_peered":"2024-03-06T07:29:03.069507+0000","last_clean":"2024-03-06T07:29:03.069507+0000","last_became_active":"2024-03-06T07:29:01.297714+0000","last_became_peered":"2024-03-06T07:29:01.297714+0000","last_unstale":"2024-03-06T07:29:03.069507+0000","last_undegraded":"2024-03-06T07:29:03.069507+0000","last_fullsized":"2024-03-06T07:29:03.069507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338187+0000","last_change":"2024-03-06T07:29:00.310808+0000","last_active":"2024-03-06T07:29:02.338187+0000","last_peered":"2024-03-06T07:29:02.338187+0000","last_clean":"2024-03-06T07:29:02.338187+0000","last_became_active":"2024-03-06T07:29:00.310511+0000","last_became_peered":"2024-03-06T07:29:00.310511+0000","last_unstale":"2024-03-06T07:29:02.338187+0000","last_undegraded":"2024-03-06T07:29:02.338187+0000","last_fullsized":"2024-03-06T07:29:02.338187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070152+0000","last_change":"2024-03-06T07:29:01.307662+0000","last_active":"2024-03-06T07:29:03.070152+0000","last_peered":"2024-03-06T07:29:03.070152+0000","last_clean":"2024-03-06T07:29:03.070152+0000","last_became_active":"2024-03-06T07:29:01.307375+0000","last_became_peered":"2024-03-06T07:29:01.307375+0000","last_unstale":"2024-03-06T07:29:03.070152+0000","last_undegraded":"2024-03-06T07:29:03.070152+0000","last_fullsized":"2024-03-06T07:29:03.070152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305649+0000","last_change":"2024-03-06T07:29:00.327844+0000","last_active":"2024-03-06T07:29:02.305649+0000","last_peered":"2024-03-06T07:29:02.305649+0000","last_clean":"2024-03-06T07:29:02.305649+0000","last_became_active":"2024-03-06T07:29:00.327512+0000","last_became_peered":"2024-03-06T07:29:00.327512+0000","last_unstale":"2024-03-06T07:29:02.305649+0000","last_undegraded":"2024-03-06T07:29:02.305649+0000","last_fullsized":"2024-03-06T07:29:02.305649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343323+0000","last_change":"2024-03-06T07:29:01.328118+0000","last_active":"2024-03-06T07:29:02.343323+0000","last_peered":"2024-03-06T07:29:02.343323+0000","last_clean":"2024-03-06T07:29:02.343323+0000","last_became_active":"2024-03-06T07:29:01.327753+0000","last_became_peered":"2024-03-06T07:29:01.327753+0000","last_unstale":"2024-03-06T07:29:02.343323+0000","last_undegraded":"2024-03-06T07:29:02.343323+0000","last_fullsized":"2024-03-06T07:29:02.343323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332010+0000","last_change":"2024-03-06T07:29:01.343767+0000","last_active":"2024-03-06T07:29:02.332010+0000","last_peered":"2024-03-06T07:29:02.332010+0000","last_clean":"2024-03-06T07:29:02.332010+0000","last_became_active":"2024-03-06T07:29:01.343223+0000","last_became_peered":"2024-03-06T07:29:01.343223+0000","last_unstale":"2024-03-06T07:29:02.332010+0000","last_undegraded":"2024-03-06T07:29:02.332010+0000","last_fullsized":"2024-03-06T07:29:02.332010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304611+0000","last_change":"2024-03-06T07:29:00.330067+0000","last_active":"2024-03-06T07:29:02.304611+0000","last_peered":"2024-03-06T07:29:02.304611+0000","last_clean":"2024-03-06T07:29:02.304611+0000","last_became_active":"2024-03-06T07:29:00.329686+0000","last_became_peered":"2024-03-06T07:29:00.329686+0000","last_unstale":"2024-03-06T07:29:02.304611+0000","last_undegraded":"2024-03-06T07:29:02.304611+0000","last_fullsized":"2024-03-06T07:29:02.304611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331745+0000","last_change":"2024-03-06T07:29:01.338965+0000","last_active":"2024-03-06T07:29:02.331745+0000","last_peered":"2024-03-06T07:29:02.331745+0000","last_clean":"2024-03-06T07:29:02.331745+0000","last_became_active":"2024-03-06T07:29:01.338503+0000","last_became_peered":"2024-03-06T07:29:01.338503+0000","last_unstale":"2024-03-06T07:29:02.331745+0000","last_undegraded":"2024-03-06T07:29:02.331745+0000","last_fullsized":"2024-03-06T07:29:02.331745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305289+0000","last_change":"2024-03-06T07:29:00.326670+0000","last_active":"2024-03-06T07:29:02.305289+0000","last_peered":"2024-03-06T07:29:02.305289+0000","last_clean":"2024-03-06T07:29:02.305289+0000","last_became_active":"2024-03-06T07:29:00.326336+0000","last_became_peered":"2024-03-06T07:29:00.326336+0000","last_unstale":"2024-03-06T07:29:02.305289+0000","last_undegraded":"2024-03-06T07:29:02.305289+0000","last_fullsized":"2024-03-06T07:29:02.305289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856601+0000","last_change":"2024-03-06T07:29:01.332554+0000","last_active":"2024-03-06T07:29:02.856601+0000","last_peered":"2024-03-06T07:29:02.856601+0000","last_clean":"2024-03-06T07:29:02.856601+0000","last_became_active":"2024-03-06T07:29:01.332081+0000","last_became_peered":"2024-03-06T07:29:01.332081+0000","last_unstale":"2024-03-06T07:29:02.856601+0000","last_undegraded":"2024-03-06T07:29:02.856601+0000","last_fullsized":"2024-03-06T07:29:02.856601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070670+0000","last_change":"2024-03-06T07:29:00.317549+0000","last_active":"2024-03-06T07:29:03.070670+0000","last_peered":"2024-03-06T07:29:03.070670+0000","last_clean":"2024-03-06T07:29:03.070670+0000","last_became_active":"2024-03-06T07:29:00.317097+0000","last_became_peered":"2024-03-06T07:29:00.317097+0000","last_unstale":"2024-03-06T07:29:03.070670+0000","last_undegraded":"2024-03-06T07:29:03.070670+0000","last_fullsized":"2024-03-06T07:29:03.070670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070604+0000","last_change":"2024-03-06T07:29:01.319129+0000","last_active":"2024-03-06T07:29:03.070604+0000","last_peered":"2024-03-06T07:29:03.070604+0000","last_clean":"2024-03-06T07:29:03.070604+0000","last_became_active":"2024-03-06T07:29:01.318772+0000","last_became_peered":"2024-03-06T07:29:01.318772+0000","last_unstale":"2024-03-06T07:29:03.070604+0000","last_undegraded":"2024-03-06T07:29:03.070604+0000","last_fullsized":"2024-03-06T07:29:03.070604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856882+0000","last_change":"2024-03-06T07:29:00.316149+0000","last_active":"2024-03-06T07:29:02.856882+0000","last_peered":"2024-03-06T07:29:02.856882+0000","last_clean":"2024-03-06T07:29:02.856882+0000","last_became_active":"2024-03-06T07:29:00.315348+0000","last_became_peered":"2024-03-06T07:29:00.315348+0000","last_unstale":"2024-03-06T07:29:02.856882+0000","last_undegraded":"2024-03-06T07:29:02.856882+0000","last_fullsized":"2024-03-06T07:29:02.856882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:26.363805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:11.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:11.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.324+0000 7fafc1000640 1 Processor -- start 2024-03-06T07:29:11.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.324+0000 7fafc1000640 1 -- start start 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafc1000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af1a0 0x7fafbc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafc1000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fafbc0af580 0x7fafbc0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafc1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafbc0cdc20 0x7fafbc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafc1000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc15cc40 con 0x7fafbc0cdc20 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafc1000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc15cf20 con 0x7fafbc0af580 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafc1000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc15d200 con 0x7fafbc0af1a0 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafba800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafbc0cdc20 0x7fafbc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafba800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafbc0cdc20 0x7fafbc0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51100/0 (socket says 172.21.15.32:51100) 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafba800640 1 -- 172.21.15.32:0/1745677113 learned_addr learned my addr 172.21.15.32:0/1745677113 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafb9e00640 1 --2- 172.21.15.32:0/1745677113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af1a0 0x7fafbc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.326+0000 7fafb3e00640 1 --2- 172.21.15.32:0/1745677113 >> v2:172.21.15.39:3300/0 conn(0x7fafbc0af580 0x7fafbc0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.327+0000 7fafba800640 1 -- 172.21.15.32:0/1745677113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af1a0 msgr2=0x7fafbc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.327+0000 7fafba800640 1 --2- 172.21.15.32:0/1745677113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af1a0 0x7fafbc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.327+0000 7fafba800640 1 -- 172.21.15.32:0/1745677113 >> v2:172.21.15.39:3300/0 conn(0x7fafbc0af580 msgr2=0x7fafbc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.327+0000 7fafba800640 1 --2- 172.21.15.32:0/1745677113 >> v2:172.21.15.39:3300/0 conn(0x7fafbc0af580 0x7fafbc0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.327+0000 7fafba800640 1 -- 172.21.15.32:0/1745677113 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc15d4e0 con 0x7fafbc0cdc20 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.328+0000 7fafba800640 1 --2- 172.21.15.32:0/1745677113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0cdc20 0x7fafbc0b2dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0748c0 tx=0x7fafb402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd40ae100dc835b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.328+0000 7fafb9400640 1 -- 172.21.15.32:0/1745677113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb402c9e0 con 0x7fafbc0cdc20 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.328+0000 7fafb9400640 1 -- 172.21.15.32:0/1745677113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb402cb80 con 0x7fafbc0cdc20 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.328+0000 7fafb9400640 1 -- 172.21.15.32:0/1745677113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40343f0 con 0x7fafbc0cdc20 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.328+0000 7fafc1000640 1 -- 172.21.15.32:0/1745677113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0cdc20 msgr2=0x7fafbc0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.328+0000 7fafc1000640 1 --2- 172.21.15.32:0/1745677113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0cdc20 0x7fafbc0b2dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0748c0 tx=0x7fafb402be40 comp rx=0 tx=0).stop 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 -- 172.21.15.32:0/1745677113 shutdown_connections 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 --2- 172.21.15.32:0/1745677113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0cdc20 0x7fafbc0b2dd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 --2- 172.21.15.32:0/1745677113 >> v2:172.21.15.39:3300/0 conn(0x7fafbc0af580 0x7fafbc0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 --2- 172.21.15.32:0/1745677113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af1a0 0x7fafbc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 -- 172.21.15.32:0/1745677113 >> 172.21.15.32:0/1745677113 conn(0x7fafbc06afb0 msgr2=0x7fafbc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 -- 172.21.15.32:0/1745677113 shutdown_connections 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.329+0000 7fafc1000640 1 -- 172.21.15.32:0/1745677113 wait complete. 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.330+0000 7fafc1000640 1 Processor -- start 2024-03-06T07:29:11.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 -- start start 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 0x7fafbc0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af580 0x7fafbc0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafbc0cdc20 0x7fafbc14c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0b3490 con 0x7fafbc0af1a0 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0b3310 con 0x7fafbc0cdc20 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.331+0000 7fafc1000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0b3610 con 0x7fafbc0af580 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 0x7fafbc0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 0x7fafbc0c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51104/0 (socket says 172.21.15.32:51104) 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 -- 172.21.15.32:0/225919325 learned_addr learned my addr 172.21.15.32:0/225919325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb3e00640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af580 0x7fafbc0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 -- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af580 msgr2=0x7fafbc0cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af580 0x7fafbc0cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 -- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafbc0cdc20 msgr2=0x7fafbc14c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafba800640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafbc0cdc20 0x7fafbc14c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafbc0cdc20 0x7fafbc14c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 -- 172.21.15.32:0/225919325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc0cb2b0 con 0x7fafbc0af1a0 2024-03-06T07:29:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.332+0000 7fafb9e00640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 0x7fafbc0c9150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fafac00d800 tx=0x7fafac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.333+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac018020 con 0x7fafbc0af1a0 2024-03-06T07:29:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.333+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac0096a0 con 0x7fafbc0af1a0 2024-03-06T07:29:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.333+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc0afbb0 con 0x7fafbc0af1a0 2024-03-06T07:29:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.333+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac009a70 con 0x7fafbc0af1a0 2024-03-06T07:29:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.333+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc14cad0 con 0x7fafbc0af1a0 2024-03-06T07:29:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.334+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc069510 con 0x7fafbc0af1a0 2024-03-06T07:29:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.335+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafac01b020 con 0x7fafbc0af1a0 2024-03-06T07:29:11.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.336+0000 7fafb2000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9806b500 0x7faf9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.336+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafac08a4f0 con 0x7fafbc0af1a0 2024-03-06T07:29:11.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.336+0000 7fafb3e00640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9806b500 0x7faf9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.336+0000 7fafb3e00640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9806b500 0x7faf9806d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fafbc06df70 tx=0x7fafa4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.340+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafac022030 con 0x7fafbc0af1a0 2024-03-06T07:29:11.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.529+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafbc04f3a0 con 0x7faf9806b500 2024-03-06T07:29:11.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.537+0000 7fafb2000640 1 -- 172.21.15.32:0/225919325 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251838 (secure 0 0 0) 0x7fafbc04f3a0 con 0x7faf9806b500 2024-03-06T07:29:11.542 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:29:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.541+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9806b500 msgr2=0x7faf9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.541+0000 7fafc1000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9806b500 0x7faf9806d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fafbc06df70 tx=0x7fafa4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.541+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 msgr2=0x7fafbc0c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.541+0000 7fafc1000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 0x7fafbc0c9150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fafac00d800 tx=0x7fafac00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 shutdown_connections 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf9806b500 0x7faf9806d9c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafbc0cdc20 0x7fafbc14c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafbc0af580 0x7fafbc0cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 --2- 172.21.15.32:0/225919325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafbc0af1a0 0x7fafbc0c9150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 >> 172.21.15.32:0/225919325 conn(0x7fafbc06afb0 msgr2=0x7fafbc0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 shutdown_connections 2024-03-06T07:29:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.542+0000 7fafc1000640 1 -- 172.21.15.32:0/225919325 wait complete. 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:29:10.188209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252724,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721996,"statfs":{"total":773094113280,"available":772835323904,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.906764"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306066+0000","last_change":"2024-03-06T07:29:01.317028+0000","last_active":"2024-03-06T07:29:02.306066+0000","last_peered":"2024-03-06T07:29:02.306066+0000","last_clean":"2024-03-06T07:29:02.306066+0000","last_became_active":"2024-03-06T07:29:01.316273+0000","last_became_peered":"2024-03-06T07:29:01.316273+0000","last_unstale":"2024-03-06T07:29:02.306066+0000","last_undegraded":"2024-03-06T07:29:02.306066+0000","last_fullsized":"2024-03-06T07:29:02.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338412+0000","last_change":"2024-03-06T07:29:01.311231+0000","last_active":"2024-03-06T07:29:02.338412+0000","last_peered":"2024-03-06T07:29:02.338412+0000","last_clean":"2024-03-06T07:29:02.338412+0000","last_became_active":"2024-03-06T07:29:01.310868+0000","last_became_peered":"2024-03-06T07:29:01.310868+0000","last_unstale":"2024-03-06T07:29:02.338412+0000","last_undegraded":"2024-03-06T07:29:02.338412+0000","last_fullsized":"2024-03-06T07:29:02.338412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856022+0000","last_change":"2024-03-06T07:29:01.307601+0000","last_active":"2024-03-06T07:29:02.856022+0000","last_peered":"2024-03-06T07:29:02.856022+0000","last_clean":"2024-03-06T07:29:02.856022+0000","last_became_active":"2024-03-06T07:29:01.307310+0000","last_became_peered":"2024-03-06T07:29:01.307310+0000","last_unstale":"2024-03-06T07:29:02.856022+0000","last_undegraded":"2024-03-06T07:29:02.856022+0000","last_fullsized":"2024-03-06T07:29:02.856022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331881+0000","last_change":"2024-03-06T07:29:01.326299+0000","last_active":"2024-03-06T07:29:02.331881+0000","last_peered":"2024-03-06T07:29:02.331881+0000","last_clean":"2024-03-06T07:29:02.331881+0000","last_became_active":"2024-03-06T07:29:01.326026+0000","last_became_peered":"2024-03-06T07:29:01.326026+0000","last_unstale":"2024-03-06T07:29:02.331881+0000","last_undegraded":"2024-03-06T07:29:02.331881+0000","last_fullsized":"2024-03-06T07:29:02.331881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304952+0000","last_change":"2024-03-06T07:29:01.314492+0000","last_active":"2024-03-06T07:29:02.304952+0000","last_peered":"2024-03-06T07:29:02.304952+0000","last_clean":"2024-03-06T07:29:02.304952+0000","last_became_active":"2024-03-06T07:29:01.313926+0000","last_became_peered":"2024-03-06T07:29:01.313926+0000","last_unstale":"2024-03-06T07:29:02.304952+0000","last_undegraded":"2024-03-06T07:29:02.304952+0000","last_fullsized":"2024-03-06T07:29:02.304952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331429+0000","last_change":"2024-03-06T07:29:01.328648+0000","last_active":"2024-03-06T07:29:02.331429+0000","last_peered":"2024-03-06T07:29:02.331429+0000","last_clean":"2024-03-06T07:29:02.331429+0000","last_became_active":"2024-03-06T07:29:01.328347+0000","last_became_peered":"2024-03-06T07:29:01.328347+0000","last_unstale":"2024-03-06T07:29:02.331429+0000","last_undegraded":"2024-03-06T07:29:02.331429+0000","last_fullsized":"2024-03-06T07:29:02.331429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304834+0000","last_change":"2024-03-06T07:29:01.310565+0000","last_active":"2024-03-06T07:29:02.304834+0000","last_peered":"2024-03-06T07:29:02.304834+0000","last_clean":"2024-03-06T07:29:02.304834+0000","last_became_active":"2024-03-06T07:29:01.310220+0000","last_became_peered":"2024-03-06T07:29:01.310220+0000","last_unstale":"2024-03-06T07:29:02.304834+0000","last_undegraded":"2024-03-06T07:29:02.304834+0000","last_fullsized":"2024-03-06T07:29:02.304834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.325938+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.325579+0000","last_became_peered":"2024-03-06T07:29:01.325579+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306146+0000","last_change":"2024-03-06T07:29:01.341575+0000","last_active":"2024-03-06T07:29:02.306146+0000","last_peered":"2024-03-06T07:29:02.306146+0000","last_clean":"2024-03-06T07:29:02.306146+0000","last_became_active":"2024-03-06T07:29:01.341372+0000","last_became_peered":"2024-03-06T07:29:01.341372+0000","last_unstale":"2024-03-06T07:29:02.306146+0000","last_undegraded":"2024-03-06T07:29:02.306146+0000","last_fullsized":"2024-03-06T07:29:02.306146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344143+0000","last_change":"2024-03-06T07:29:01.337188+0000","last_active":"2024-03-06T07:29:02.344143+0000","last_peered":"2024-03-06T07:29:02.344143+0000","last_clean":"2024-03-06T07:29:02.344143+0000","last_became_active":"2024-03-06T07:29:01.336845+0000","last_became_peered":"2024-03-06T07:29:01.336845+0000","last_unstale":"2024-03-06T07:29:02.344143+0000","last_undegraded":"2024-03-06T07:29:02.344143+0000","last_fullsized":"2024-03-06T07:29:02.344143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343789+0000","last_change":"2024-03-06T07:29:01.340996+0000","last_active":"2024-03-06T07:29:02.343789+0000","last_peered":"2024-03-06T07:29:02.343789+0000","last_clean":"2024-03-06T07:29:02.343789+0000","last_became_active":"2024-03-06T07:29:01.336560+0000","last_became_peered":"2024-03-06T07:29:01.336560+0000","last_unstale":"2024-03-06T07:29:02.343789+0000","last_undegraded":"2024-03-06T07:29:02.343789+0000","last_fullsized":"2024-03-06T07:29:02.343789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310075+0000","last_change":"2024-03-06T07:29:01.335731+0000","last_active":"2024-03-06T07:29:02.310075+0000","last_peered":"2024-03-06T07:29:02.310075+0000","last_clean":"2024-03-06T07:29:02.310075+0000","last_became_active":"2024-03-06T07:29:01.331758+0000","last_became_peered":"2024-03-06T07:29:01.331758+0000","last_unstale":"2024-03-06T07:29:02.310075+0000","last_undegraded":"2024-03-06T07:29:02.310075+0000","last_fullsized":"2024-03-06T07:29:02.310075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:19.716910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070024+0000","last_change":"2024-03-06T07:29:01.317550+0000","last_active":"2024-03-06T07:29:03.070024+0000","last_peered":"2024-03-06T07:29:03.070024+0000","last_clean":"2024-03-06T07:29:03.070024+0000","last_became_active":"2024-03-06T07:29:01.317098+0000","last_became_peered":"2024-03-06T07:29:01.317098+0000","last_unstale":"2024-03-06T07:29:03.070024+0000","last_undegraded":"2024-03-06T07:29:03.070024+0000","last_fullsized":"2024-03-06T07:29:03.070024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855737+0000","last_change":"2024-03-06T07:29:01.306850+0000","last_active":"2024-03-06T07:29:02.855737+0000","last_peered":"2024-03-06T07:29:02.855737+0000","last_clean":"2024-03-06T07:29:02.855737+0000","last_became_active":"2024-03-06T07:29:01.306532+0000","last_became_peered":"2024-03-06T07:29:01.306532+0000","last_unstale":"2024-03-06T07:29:02.855737+0000","last_undegraded":"2024-03-06T07:29:02.855737+0000","last_fullsized":"2024-03-06T07:29:02.855737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:02.899318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.337987+0000","last_change":"2024-03-06T07:29:01.313116+0000","last_active":"2024-03-06T07:29:02.337987+0000","last_peered":"2024-03-06T07:29:02.337987+0000","last_clean":"2024-03-06T07:29:02.337987+0000","last_became_active":"2024-03-06T07:29:01.311783+0000","last_became_peered":"2024-03-06T07:29:01.311783+0000","last_unstale":"2024-03-06T07:29:02.337987+0000","last_undegraded":"2024-03-06T07:29:02.337987+0000","last_fullsized":"2024-03-06T07:29:02.337987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331678+0000","last_change":"2024-03-06T07:29:01.323368+0000","last_active":"2024-03-06T07:29:02.331678+0000","last_peered":"2024-03-06T07:29:02.331678+0000","last_clean":"2024-03-06T07:29:02.331678+0000","last_became_active":"2024-03-06T07:29:01.323066+0000","last_became_peered":"2024-03-06T07:29:01.323066+0000","last_unstale":"2024-03-06T07:29:02.331678+0000","last_undegraded":"2024-03-06T07:29:02.331678+0000","last_fullsized":"2024-03-06T07:29:02.331678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:05.203547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332654+0000","last_change":"2024-03-06T07:29:01.332332+0000","last_active":"2024-03-06T07:29:02.332654+0000","last_peered":"2024-03-06T07:29:02.332654+0000","last_clean":"2024-03-06T07:29:02.332654+0000","last_became_active":"2024-03-06T07:29:01.331999+0000","last_became_peered":"2024-03-06T07:29:01.331999+0000","last_unstale":"2024-03-06T07:29:02.332654+0000","last_undegraded":"2024-03-06T07:29:02.332654+0000","last_fullsized":"2024-03-06T07:29:02.332654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069247+0000","last_change":"2024-03-06T07:29:01.319453+0000","last_active":"2024-03-06T07:29:03.069247+0000","last_peered":"2024-03-06T07:29:03.069247+0000","last_clean":"2024-03-06T07:29:03.069247+0000","last_became_active":"2024-03-06T07:29:01.319182+0000","last_became_peered":"2024-03-06T07:29:01.319182+0000","last_unstale":"2024-03-06T07:29:03.069247+0000","last_undegraded":"2024-03-06T07:29:03.069247+0000","last_fullsized":"2024-03-06T07:29:03.069247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:42.291305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305802+0000","last_change":"2024-03-06T07:29:01.304420+0000","last_active":"2024-03-06T07:29:02.305802+0000","last_peered":"2024-03-06T07:29:02.305802+0000","last_clean":"2024-03-06T07:29:02.305802+0000","last_became_active":"2024-03-06T07:29:01.303897+0000","last_became_peered":"2024-03-06T07:29:01.303897+0000","last_unstale":"2024-03-06T07:29:02.305802+0000","last_undegraded":"2024-03-06T07:29:02.305802+0000","last_fullsized":"2024-03-06T07:29:02.305802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343811+0000","last_change":"2024-03-06T07:29:01.340804+0000","last_active":"2024-03-06T07:29:02.343811+0000","last_peered":"2024-03-06T07:29:02.343811+0000","last_clean":"2024-03-06T07:29:02.343811+0000","last_became_active":"2024-03-06T07:29:01.338819+0000","last_became_peered":"2024-03-06T07:29:01.338819+0000","last_unstale":"2024-03-06T07:29:02.343811+0000","last_undegraded":"2024-03-06T07:29:02.343811+0000","last_fullsized":"2024-03-06T07:29:02.343811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856684+0000","last_change":"2024-03-06T07:29:01.329232+0000","last_active":"2024-03-06T07:29:02.856684+0000","last_peered":"2024-03-06T07:29:02.856684+0000","last_clean":"2024-03-06T07:29:02.856684+0000","last_became_active":"2024-03-06T07:29:01.328855+0000","last_became_peered":"2024-03-06T07:29:01.328855+0000","last_unstale":"2024-03-06T07:29:02.856684+0000","last_undegraded":"2024-03-06T07:29:02.856684+0000","last_fullsized":"2024-03-06T07:29:02.856684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331471+0000","last_change":"2024-03-06T07:29:01.334776+0000","last_active":"2024-03-06T07:29:02.331471+0000","last_peered":"2024-03-06T07:29:02.331471+0000","last_clean":"2024-03-06T07:29:02.331471+0000","last_became_active":"2024-03-06T07:29:01.332072+0000","last_became_peered":"2024-03-06T07:29:01.332072+0000","last_unstale":"2024-03-06T07:29:02.331471+0000","last_undegraded":"2024-03-06T07:29:02.331471+0000","last_fullsized":"2024-03-06T07:29:02.331471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069032+0000","last_change":"2024-03-06T07:29:01.312963+0000","last_active":"2024-03-06T07:29:03.069032+0000","last_peered":"2024-03-06T07:29:03.069032+0000","last_clean":"2024-03-06T07:29:03.069032+0000","last_became_active":"2024-03-06T07:29:01.312645+0000","last_became_peered":"2024-03-06T07:29:01.312645+0000","last_unstale":"2024-03-06T07:29:03.069032+0000","last_undegraded":"2024-03-06T07:29:03.069032+0000","last_fullsized":"2024-03-06T07:29:03.069032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331763+0000","last_change":"2024-03-06T07:29:01.322856+0000","last_active":"2024-03-06T07:29:02.331763+0000","last_peered":"2024-03-06T07:29:02.331763+0000","last_clean":"2024-03-06T07:29:02.331763+0000","last_became_active":"2024-03-06T07:29:01.322485+0000","last_became_peered":"2024-03-06T07:29:01.322485+0000","last_unstale":"2024-03-06T07:29:02.331763+0000","last_undegraded":"2024-03-06T07:29:02.331763+0000","last_fullsized":"2024-03-06T07:29:02.331763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070502+0000","last_change":"2024-03-06T07:29:01.328113+0000","last_active":"2024-03-06T07:29:03.070502+0000","last_peered":"2024-03-06T07:29:03.070502+0000","last_clean":"2024-03-06T07:29:03.070502+0000","last_became_active":"2024-03-06T07:29:01.327700+0000","last_became_peered":"2024-03-06T07:29:01.327700+0000","last_unstale":"2024-03-06T07:29:03.070502+0000","last_undegraded":"2024-03-06T07:29:03.070502+0000","last_fullsized":"2024-03-06T07:29:03.070502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:15.286815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344258+0000","last_change":"2024-03-06T07:29:01.328971+0000","last_active":"2024-03-06T07:29:02.344258+0000","last_peered":"2024-03-06T07:29:02.344258+0000","last_clean":"2024-03-06T07:29:02.344258+0000","last_became_active":"2024-03-06T07:29:01.328471+0000","last_became_peered":"2024-03-06T07:29:01.328471+0000","last_unstale":"2024-03-06T07:29:02.344258+0000","last_undegraded":"2024-03-06T07:29:02.344258+0000","last_fullsized":"2024-03-06T07:29:02.344258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338529+0000","last_change":"2024-03-06T07:29:01.310033+0000","last_active":"2024-03-06T07:29:02.338529+0000","last_peered":"2024-03-06T07:29:02.338529+0000","last_clean":"2024-03-06T07:29:02.338529+0000","last_became_active":"2024-03-06T07:29:01.309715+0000","last_became_peered":"2024-03-06T07:29:01.309715+0000","last_unstale":"2024-03-06T07:29:02.338529+0000","last_undegraded":"2024-03-06T07:29:02.338529+0000","last_fullsized":"2024-03-06T07:29:02.338529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069900+0000","last_change":"2024-03-06T07:29:01.338107+0000","last_active":"2024-03-06T07:29:03.069900+0000","last_peered":"2024-03-06T07:29:03.069900+0000","last_clean":"2024-03-06T07:29:03.069900+0000","last_became_active":"2024-03-06T07:29:01.337793+0000","last_became_peered":"2024-03-06T07:29:01.337793+0000","last_unstale":"2024-03-06T07:29:03.069900+0000","last_undegraded":"2024-03-06T07:29:03.069900+0000","last_fullsized":"2024-03-06T07:29:03.069900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856096+0000","last_change":"2024-03-06T07:29:01.309168+0000","last_active":"2024-03-06T07:29:02.856096+0000","last_peered":"2024-03-06T07:29:02.856096+0000","last_clean":"2024-03-06T07:29:02.856096+0000","last_became_active":"2024-03-06T07:29:01.308853+0000","last_became_peered":"2024-03-06T07:29:01.308853+0000","last_unstale":"2024-03-06T07:29:02.856096+0000","last_undegraded":"2024-03-06T07:29:02.856096+0000","last_fullsized":"2024-03-06T07:29:02.856096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332020+0000","last_change":"2024-03-06T07:29:01.339291+0000","last_active":"2024-03-06T07:29:02.332020+0000","last_peered":"2024-03-06T07:29:02.332020+0000","last_clean":"2024-03-06T07:29:02.332020+0000","last_became_active":"2024-03-06T07:29:01.338961+0000","last_became_peered":"2024-03-06T07:29:01.338961+0000","last_unstale":"2024-03-06T07:29:02.332020+0000","last_undegraded":"2024-03-06T07:29:02.332020+0000","last_fullsized":"2024-03-06T07:29:02.332020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:35.472420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068361+0000","last_change":"2024-03-06T07:29:01.339412+0000","last_active":"2024-03-06T07:29:03.068361+0000","last_peered":"2024-03-06T07:29:03.068361+0000","last_clean":"2024-03-06T07:29:03.068361+0000","last_became_active":"2024-03-06T07:29:01.338930+0000","last_became_peered":"2024-03-06T07:29:01.338930+0000","last_unstale":"2024-03-06T07:29:03.068361+0000","last_undegraded":"2024-03-06T07:29:03.068361+0000","last_fullsized":"2024-03-06T07:29:03.068361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343463+0000","last_change":"2024-03-06T07:29:01.316617+0000","last_active":"2024-03-06T07:29:02.343463+0000","last_peered":"2024-03-06T07:29:02.343463+0000","last_clean":"2024-03-06T07:29:02.343463+0000","last_became_active":"2024-03-06T07:29:01.315094+0000","last_became_peered":"2024-03-06T07:29:01.315094+0000","last_unstale":"2024-03-06T07:29:02.343463+0000","last_undegraded":"2024-03-06T07:29:02.343463+0000","last_fullsized":"2024-03-06T07:29:02.343463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:58.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309312+0000","last_change":"2024-03-06T07:29:01.314610+0000","last_active":"2024-03-06T07:29:02.309312+0000","last_peered":"2024-03-06T07:29:02.309312+0000","last_clean":"2024-03-06T07:29:02.309312+0000","last_became_active":"2024-03-06T07:29:01.314249+0000","last_became_peered":"2024-03-06T07:29:01.314249+0000","last_unstale":"2024-03-06T07:29:02.309312+0000","last_undegraded":"2024-03-06T07:29:02.309312+0000","last_fullsized":"2024-03-06T07:29:02.309312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:32.840745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070367+0000","last_change":"2024-03-06T07:29:00.318669+0000","last_active":"2024-03-06T07:29:03.070367+0000","last_peered":"2024-03-06T07:29:03.070367+0000","last_clean":"2024-03-06T07:29:03.070367+0000","last_became_active":"2024-03-06T07:29:00.318368+0000","last_became_peered":"2024-03-06T07:29:00.318368+0000","last_unstale":"2024-03-06T07:29:03.070367+0000","last_undegraded":"2024-03-06T07:29:03.070367+0000","last_fullsized":"2024-03-06T07:29:03.070367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856486+0000","last_change":"2024-03-06T07:29:01.330131+0000","last_active":"2024-03-06T07:29:02.856486+0000","last_peered":"2024-03-06T07:29:02.856486+0000","last_clean":"2024-03-06T07:29:02.856486+0000","last_became_active":"2024-03-06T07:29:01.329564+0000","last_became_peered":"2024-03-06T07:29:01.329564+0000","last_unstale":"2024-03-06T07:29:02.856486+0000","last_undegraded":"2024-03-06T07:29:02.856486+0000","last_fullsized":"2024-03-06T07:29:02.856486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343246+0000","last_change":"2024-03-06T07:29:00.325104+0000","last_active":"2024-03-06T07:29:02.343246+0000","last_peered":"2024-03-06T07:29:02.343246+0000","last_clean":"2024-03-06T07:29:02.343246+0000","last_became_active":"2024-03-06T07:29:00.324737+0000","last_became_peered":"2024-03-06T07:29:00.324737+0000","last_unstale":"2024-03-06T07:29:02.343246+0000","last_undegraded":"2024-03-06T07:29:02.343246+0000","last_fullsized":"2024-03-06T07:29:02.343246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068981+0000","last_change":"2024-03-06T07:29:00.322124+0000","last_active":"2024-03-06T07:29:03.068981+0000","last_peered":"2024-03-06T07:29:03.068981+0000","last_clean":"2024-03-06T07:29:03.068981+0000","last_became_active":"2024-03-06T07:29:00.321658+0000","last_became_peered":"2024-03-06T07:29:00.321658+0000","last_unstale":"2024-03-06T07:29:03.068981+0000","last_undegraded":"2024-03-06T07:29:03.068981+0000","last_fullsized":"2024-03-06T07:29:03.068981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069549+0000","last_change":"2024-03-06T07:29:01.306201+0000","last_active":"2024-03-06T07:29:03.069549+0000","last_peered":"2024-03-06T07:29:03.069549+0000","last_clean":"2024-03-06T07:29:03.069549+0000","last_became_active":"2024-03-06T07:29:01.305652+0000","last_became_peered":"2024-03-06T07:29:01.305652+0000","last_unstale":"2024-03-06T07:29:03.069549+0000","last_undegraded":"2024-03-06T07:29:03.069549+0000","last_fullsized":"2024-03-06T07:29:03.069549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331796+0000","last_change":"2024-03-06T07:29:00.330308+0000","last_active":"2024-03-06T07:29:02.331796+0000","last_peered":"2024-03-06T07:29:02.331796+0000","last_clean":"2024-03-06T07:29:02.331796+0000","last_became_active":"2024-03-06T07:29:00.329727+0000","last_became_peered":"2024-03-06T07:29:00.329727+0000","last_unstale":"2024-03-06T07:29:02.331796+0000","last_undegraded":"2024-03-06T07:29:02.331796+0000","last_fullsized":"2024-03-06T07:29:02.331796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338071+0000","last_change":"2024-03-06T07:29:01.308157+0000","last_active":"2024-03-06T07:29:02.338071+0000","last_peered":"2024-03-06T07:29:02.338071+0000","last_clean":"2024-03-06T07:29:02.338071+0000","last_became_active":"2024-03-06T07:29:01.307819+0000","last_became_peered":"2024-03-06T07:29:01.307819+0000","last_unstale":"2024-03-06T07:29:02.338071+0000","last_undegraded":"2024-03-06T07:29:02.338071+0000","last_fullsized":"2024-03-06T07:29:02.338071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:49.242218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.334131+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.333816+0000","last_became_peered":"2024-03-06T07:29:00.333816+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332394+0000","last_change":"2024-03-06T07:29:01.332596+0000","last_active":"2024-03-06T07:29:02.332394+0000","last_peered":"2024-03-06T07:29:02.332394+0000","last_clean":"2024-03-06T07:29:02.332394+0000","last_became_active":"2024-03-06T07:29:01.332140+0000","last_became_peered":"2024-03-06T07:29:01.332140+0000","last_unstale":"2024-03-06T07:29:02.332394+0000","last_undegraded":"2024-03-06T07:29:02.332394+0000","last_fullsized":"2024-03-06T07:29:02.332394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.365360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332369+0000","last_change":"2024-03-06T07:29:00.332773+0000","last_active":"2024-03-06T07:29:02.332369+0000","last_peered":"2024-03-06T07:29:02.332369+0000","last_clean":"2024-03-06T07:29:02.332369+0000","last_became_active":"2024-03-06T07:29:00.332477+0000","last_became_peered":"2024-03-06T07:29:00.332477+0000","last_unstale":"2024-03-06T07:29:02.332369+0000","last_undegraded":"2024-03-06T07:29:02.332369+0000","last_fullsized":"2024-03-06T07:29:02.332369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856820+0000","last_change":"2024-03-06T07:29:01.325722+0000","last_active":"2024-03-06T07:29:02.856820+0000","last_peered":"2024-03-06T07:29:02.856820+0000","last_clean":"2024-03-06T07:29:02.856820+0000","last_became_active":"2024-03-06T07:29:01.325401+0000","last_became_peered":"2024-03-06T07:29:01.325401+0000","last_unstale":"2024-03-06T07:29:02.856820+0000","last_undegraded":"2024-03-06T07:29:02.856820+0000","last_fullsized":"2024-03-06T07:29:02.856820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338454+0000","last_change":"2024-03-06T07:29:00.331332+0000","last_active":"2024-03-06T07:29:02.338454+0000","last_peered":"2024-03-06T07:29:02.338454+0000","last_clean":"2024-03-06T07:29:02.338454+0000","last_became_active":"2024-03-06T07:29:00.331008+0000","last_became_peered":"2024-03-06T07:29:00.331008+0000","last_unstale":"2024-03-06T07:29:02.338454+0000","last_undegraded":"2024-03-06T07:29:02.338454+0000","last_fullsized":"2024-03-06T07:29:02.338454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310152+0000","last_change":"2024-03-06T07:29:01.335452+0000","last_active":"2024-03-06T07:29:02.310152+0000","last_peered":"2024-03-06T07:29:02.310152+0000","last_clean":"2024-03-06T07:29:02.310152+0000","last_became_active":"2024-03-06T07:29:01.331241+0000","last_became_peered":"2024-03-06T07:29:01.331241+0000","last_unstale":"2024-03-06T07:29:02.310152+0000","last_undegraded":"2024-03-06T07:29:02.310152+0000","last_fullsized":"2024-03-06T07:29:02.310152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.332529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343888+0000","last_change":"2024-03-06T07:29:00.322219+0000","last_active":"2024-03-06T07:29:02.343888+0000","last_peered":"2024-03-06T07:29:02.343888+0000","last_clean":"2024-03-06T07:29:02.343888+0000","last_became_active":"2024-03-06T07:29:00.321921+0000","last_became_peered":"2024-03-06T07:29:00.321921+0000","last_unstale":"2024-03-06T07:29:02.343888+0000","last_undegraded":"2024-03-06T07:29:02.343888+0000","last_fullsized":"2024-03-06T07:29:02.343888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304565+0000","last_change":"2024-03-06T07:29:01.309672+0000","last_active":"2024-03-06T07:29:02.304565+0000","last_peered":"2024-03-06T07:29:02.304565+0000","last_clean":"2024-03-06T07:29:02.304565+0000","last_became_active":"2024-03-06T07:29:01.309369+0000","last_became_peered":"2024-03-06T07:29:01.309369+0000","last_unstale":"2024-03-06T07:29:02.304565+0000","last_undegraded":"2024-03-06T07:29:02.304565+0000","last_fullsized":"2024-03-06T07:29:02.304565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.202943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856650+0000","last_change":"2024-03-06T07:29:00.324810+0000","last_active":"2024-03-06T07:29:02.856650+0000","last_peered":"2024-03-06T07:29:02.856650+0000","last_clean":"2024-03-06T07:29:02.856650+0000","last_became_active":"2024-03-06T07:29:00.324524+0000","last_became_peered":"2024-03-06T07:29:00.324524+0000","last_unstale":"2024-03-06T07:29:02.856650+0000","last_undegraded":"2024-03-06T07:29:02.856650+0000","last_fullsized":"2024-03-06T07:29:02.856650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:20.411708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855986+0000","last_change":"2024-03-06T07:29:01.329235+0000","last_active":"2024-03-06T07:29:02.855986+0000","last_peered":"2024-03-06T07:29:02.855986+0000","last_clean":"2024-03-06T07:29:02.855986+0000","last_became_active":"2024-03-06T07:29:01.328953+0000","last_became_peered":"2024-03-06T07:29:01.328953+0000","last_unstale":"2024-03-06T07:29:02.855986+0000","last_undegraded":"2024-03-06T07:29:02.855986+0000","last_fullsized":"2024-03-06T07:29:02.855986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331601+0000","last_change":"2024-03-06T07:29:00.325328+0000","last_active":"2024-03-06T07:29:02.331601+0000","last_peered":"2024-03-06T07:29:02.331601+0000","last_clean":"2024-03-06T07:29:02.331601+0000","last_became_active":"2024-03-06T07:29:00.323961+0000","last_became_peered":"2024-03-06T07:29:00.323961+0000","last_unstale":"2024-03-06T07:29:02.331601+0000","last_undegraded":"2024-03-06T07:29:02.331601+0000","last_fullsized":"2024-03-06T07:29:02.331601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305341+0000","last_change":"2024-03-06T07:29:01.314222+0000","last_active":"2024-03-06T07:29:02.305341+0000","last_peered":"2024-03-06T07:29:02.305341+0000","last_clean":"2024-03-06T07:29:02.305341+0000","last_became_active":"2024-03-06T07:29:01.313409+0000","last_became_peered":"2024-03-06T07:29:01.313409+0000","last_unstale":"2024-03-06T07:29:02.305341+0000","last_undegraded":"2024-03-06T07:29:02.305341+0000","last_fullsized":"2024-03-06T07:29:02.305341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343329+0000","last_change":"2024-03-06T07:29:00.326529+0000","last_active":"2024-03-06T07:29:02.343329+0000","last_peered":"2024-03-06T07:29:02.343329+0000","last_clean":"2024-03-06T07:29:02.343329+0000","last_became_active":"2024-03-06T07:29:00.326202+0000","last_became_peered":"2024-03-06T07:29:00.326202+0000","last_unstale":"2024-03-06T07:29:02.343329+0000","last_undegraded":"2024-03-06T07:29:02.343329+0000","last_fullsized":"2024-03-06T07:29:02.343329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343344+0000","last_change":"2024-03-06T07:29:01.316333+0000","last_active":"2024-03-06T07:29:02.343344+0000","last_peered":"2024-03-06T07:29:02.343344+0000","last_clean":"2024-03-06T07:29:02.343344+0000","last_became_active":"2024-03-06T07:29:01.315969+0000","last_became_peered":"2024-03-06T07:29:01.315969+0000","last_unstale":"2024-03-06T07:29:02.343344+0000","last_undegraded":"2024-03-06T07:29:02.343344+0000","last_fullsized":"2024-03-06T07:29:02.343344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:57.300340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069217+0000","last_change":"2024-03-06T07:29:00.306241+0000","last_active":"2024-03-06T07:29:03.069217+0000","last_peered":"2024-03-06T07:29:03.069217+0000","last_clean":"2024-03-06T07:29:03.069217+0000","last_became_active":"2024-03-06T07:29:00.305883+0000","last_became_peered":"2024-03-06T07:29:00.305883+0000","last_unstale":"2024-03-06T07:29:03.069217+0000","last_undegraded":"2024-03-06T07:29:03.069217+0000","last_fullsized":"2024-03-06T07:29:03.069217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:18.017185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306244+0000","last_change":"2024-03-06T07:29:01.326203+0000","last_active":"2024-03-06T07:29:02.306244+0000","last_peered":"2024-03-06T07:29:02.306244+0000","last_clean":"2024-03-06T07:29:02.306244+0000","last_became_active":"2024-03-06T07:29:01.325861+0000","last_became_peered":"2024-03-06T07:29:01.325861+0000","last_unstale":"2024-03-06T07:29:02.306244+0000","last_undegraded":"2024-03-06T07:29:02.306244+0000","last_fullsized":"2024-03-06T07:29:02.306244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:02.581933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.310043+0000","last_change":"2024-03-06T07:29:00.320015+0000","last_active":"2024-03-06T07:29:02.310043+0000","last_peered":"2024-03-06T07:29:02.310043+0000","last_clean":"2024-03-06T07:29:02.310043+0000","last_became_active":"2024-03-06T07:29:00.319492+0000","last_became_peered":"2024-03-06T07:29:00.319492+0000","last_unstale":"2024-03-06T07:29:02.310043+0000","last_undegraded":"2024-03-06T07:29:02.310043+0000","last_fullsized":"2024-03-06T07:29:02.310043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:49.576279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304906+0000","last_change":"2024-03-06T07:29:01.325857+0000","last_active":"2024-03-06T07:29:02.304906+0000","last_peered":"2024-03-06T07:29:02.304906+0000","last_clean":"2024-03-06T07:29:02.304906+0000","last_became_active":"2024-03-06T07:29:01.325522+0000","last_became_peered":"2024-03-06T07:29:01.325522+0000","last_unstale":"2024-03-06T07:29:02.304906+0000","last_undegraded":"2024-03-06T07:29:02.304906+0000","last_fullsized":"2024-03-06T07:29:02.304906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069075+0000","last_change":"2024-03-06T07:29:00.311734+0000","last_active":"2024-03-06T07:29:03.069075+0000","last_peered":"2024-03-06T07:29:03.069075+0000","last_clean":"2024-03-06T07:29:03.069075+0000","last_became_active":"2024-03-06T07:29:00.309932+0000","last_became_peered":"2024-03-06T07:29:00.309932+0000","last_unstale":"2024-03-06T07:29:03.069075+0000","last_undegraded":"2024-03-06T07:29:03.069075+0000","last_fullsized":"2024-03-06T07:29:03.069075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306517+0000","last_change":"2024-03-06T07:29:01.331125+0000","last_active":"2024-03-06T07:29:02.306517+0000","last_peered":"2024-03-06T07:29:02.306517+0000","last_clean":"2024-03-06T07:29:02.306517+0000","last_became_active":"2024-03-06T07:29:01.330860+0000","last_became_peered":"2024-03-06T07:29:01.330860+0000","last_unstale":"2024-03-06T07:29:02.306517+0000","last_undegraded":"2024-03-06T07:29:02.306517+0000","last_fullsized":"2024-03-06T07:29:02.306517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:44.856694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338728+0000","last_change":"2024-03-06T07:29:00.325056+0000","last_active":"2024-03-06T07:29:02.338728+0000","last_peered":"2024-03-06T07:29:02.338728+0000","last_clean":"2024-03-06T07:29:02.338728+0000","last_became_active":"2024-03-06T07:29:00.324757+0000","last_became_peered":"2024-03-06T07:29:00.324757+0000","last_unstale":"2024-03-06T07:29:02.338728+0000","last_undegraded":"2024-03-06T07:29:02.338728+0000","last_fullsized":"2024-03-06T07:29:02.338728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332058+0000","last_change":"2024-03-06T07:29:01.326654+0000","last_active":"2024-03-06T07:29:02.332058+0000","last_peered":"2024-03-06T07:29:02.332058+0000","last_clean":"2024-03-06T07:29:02.332058+0000","last_became_active":"2024-03-06T07:29:01.325612+0000","last_became_peered":"2024-03-06T07:29:01.325612+0000","last_unstale":"2024-03-06T07:29:02.332058+0000","last_undegraded":"2024-03-06T07:29:02.332058+0000","last_fullsized":"2024-03-06T07:29:02.332058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:44.423326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068894+0000","last_change":"2024-03-06T07:29:00.332332+0000","last_active":"2024-03-06T07:29:03.068894+0000","last_peered":"2024-03-06T07:29:03.068894+0000","last_clean":"2024-03-06T07:29:03.068894+0000","last_became_active":"2024-03-06T07:29:00.331881+0000","last_became_peered":"2024-03-06T07:29:00.331881+0000","last_unstale":"2024-03-06T07:29:03.068894+0000","last_undegraded":"2024-03-06T07:29:03.068894+0000","last_fullsized":"2024-03-06T07:29:03.068894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304989+0000","last_change":"2024-03-06T07:29:01.336946+0000","last_active":"2024-03-06T07:29:02.304989+0000","last_peered":"2024-03-06T07:29:02.304989+0000","last_clean":"2024-03-06T07:29:02.304989+0000","last_became_active":"2024-03-06T07:29:01.336594+0000","last_became_peered":"2024-03-06T07:29:01.336594+0000","last_unstale":"2024-03-06T07:29:02.304989+0000","last_undegraded":"2024-03-06T07:29:02.304989+0000","last_fullsized":"2024-03-06T07:29:02.304989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303958+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:29:02.303958+0000","last_peered":"2024-03-06T07:29:02.303958+0000","last_clean":"2024-03-06T07:29:02.303958+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:29:02.303958+0000","last_undegraded":"2024-03-06T07:29:02.303958+0000","last_fullsized":"2024-03-06T07:29:02.303958+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331455+0000","last_change":"2024-03-06T07:29:00.331164+0000","last_active":"2024-03-06T07:29:02.331455+0000","last_peered":"2024-03-06T07:29:02.331455+0000","last_clean":"2024-03-06T07:29:02.331455+0000","last_became_active":"2024-03-06T07:29:00.330531+0000","last_became_peered":"2024-03-06T07:29:00.330531+0000","last_unstale":"2024-03-06T07:29:02.331455+0000","last_undegraded":"2024-03-06T07:29:02.331455+0000","last_fullsized":"2024-03-06T07:29:02.331455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:11.757764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069681+0000","last_change":"2024-03-06T07:29:01.342482+0000","last_active":"2024-03-06T07:29:03.069681+0000","last_peered":"2024-03-06T07:29:03.069681+0000","last_clean":"2024-03-06T07:29:03.069681+0000","last_became_active":"2024-03-06T07:29:01.342176+0000","last_became_peered":"2024-03-06T07:29:01.342176+0000","last_unstale":"2024-03-06T07:29:03.069681+0000","last_undegraded":"2024-03-06T07:29:03.069681+0000","last_fullsized":"2024-03-06T07:29:03.069681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303790+0000","last_change":"2024-03-06T07:29:00.308988+0000","last_active":"2024-03-06T07:29:02.303790+0000","last_peered":"2024-03-06T07:29:02.303790+0000","last_clean":"2024-03-06T07:29:02.303790+0000","last_became_active":"2024-03-06T07:29:00.304843+0000","last_became_peered":"2024-03-06T07:29:00.304843+0000","last_unstale":"2024-03-06T07:29:02.303790+0000","last_undegraded":"2024-03-06T07:29:02.303790+0000","last_fullsized":"2024-03-06T07:29:02.303790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.855676+0000","last_change":"2024-03-06T07:29:01.328580+0000","last_active":"2024-03-06T07:29:02.855676+0000","last_peered":"2024-03-06T07:29:02.855676+0000","last_clean":"2024-03-06T07:29:02.855676+0000","last_became_active":"2024-03-06T07:29:01.328319+0000","last_became_peered":"2024-03-06T07:29:01.328319+0000","last_unstale":"2024-03-06T07:29:02.855676+0000","last_undegraded":"2024-03-06T07:29:02.855676+0000","last_fullsized":"2024-03-06T07:29:02.855676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:33.356914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.068423+0000","last_change":"2024-03-06T07:29:00.307455+0000","last_active":"2024-03-06T07:29:03.068423+0000","last_peered":"2024-03-06T07:29:03.068423+0000","last_clean":"2024-03-06T07:29:03.068423+0000","last_became_active":"2024-03-06T07:29:00.307117+0000","last_became_peered":"2024-03-06T07:29:00.307117+0000","last_unstale":"2024-03-06T07:29:03.068423+0000","last_undegraded":"2024-03-06T07:29:03.068423+0000","last_fullsized":"2024-03-06T07:29:03.068423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338099+0000","last_change":"2024-03-06T07:29:01.298549+0000","last_active":"2024-03-06T07:29:02.338099+0000","last_peered":"2024-03-06T07:29:02.338099+0000","last_clean":"2024-03-06T07:29:02.338099+0000","last_became_active":"2024-03-06T07:29:01.298043+0000","last_became_peered":"2024-03-06T07:29:01.298043+0000","last_unstale":"2024-03-06T07:29:02.338099+0000","last_undegraded":"2024-03-06T07:29:02.338099+0000","last_fullsized":"2024-03-06T07:29:02.338099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331760+0000","last_change":"2024-03-06T07:29:00.323986+0000","last_active":"2024-03-06T07:29:02.331760+0000","last_peered":"2024-03-06T07:29:02.331760+0000","last_clean":"2024-03-06T07:29:02.331760+0000","last_became_active":"2024-03-06T07:29:00.323442+0000","last_became_peered":"2024-03-06T07:29:00.323442+0000","last_unstale":"2024-03-06T07:29:02.331760+0000","last_undegraded":"2024-03-06T07:29:02.331760+0000","last_fullsized":"2024-03-06T07:29:02.331760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309938+0000","last_change":"2024-03-06T07:29:01.319824+0000","last_active":"2024-03-06T07:29:02.309938+0000","last_peered":"2024-03-06T07:29:02.309938+0000","last_clean":"2024-03-06T07:29:02.309938+0000","last_became_active":"2024-03-06T07:29:01.317792+0000","last_became_peered":"2024-03-06T07:29:01.317792+0000","last_unstale":"2024-03-06T07:29:02.309938+0000","last_undegraded":"2024-03-06T07:29:02.309938+0000","last_fullsized":"2024-03-06T07:29:02.309938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.562992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338550+0000","last_change":"2024-03-06T07:29:00.312181+0000","last_active":"2024-03-06T07:29:02.338550+0000","last_peered":"2024-03-06T07:29:02.338550+0000","last_clean":"2024-03-06T07:29:02.338550+0000","last_became_active":"2024-03-06T07:29:00.311915+0000","last_became_peered":"2024-03-06T07:29:00.311915+0000","last_unstale":"2024-03-06T07:29:02.338550+0000","last_undegraded":"2024-03-06T07:29:02.338550+0000","last_fullsized":"2024-03-06T07:29:02.338550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332017+0000","last_change":"2024-03-06T07:29:01.332290+0000","last_active":"2024-03-06T07:29:02.332017+0000","last_peered":"2024-03-06T07:29:02.332017+0000","last_clean":"2024-03-06T07:29:02.332017+0000","last_became_active":"2024-03-06T07:29:01.331904+0000","last_became_peered":"2024-03-06T07:29:01.331904+0000","last_unstale":"2024-03-06T07:29:02.332017+0000","last_undegraded":"2024-03-06T07:29:02.332017+0000","last_fullsized":"2024-03-06T07:29:02.332017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.306067+0000","last_change":"2024-03-06T07:29:00.315219+0000","last_active":"2024-03-06T07:29:02.306067+0000","last_peered":"2024-03-06T07:29:02.306067+0000","last_clean":"2024-03-06T07:29:02.306067+0000","last_became_active":"2024-03-06T07:29:00.314808+0000","last_became_peered":"2024-03-06T07:29:00.314808+0000","last_unstale":"2024-03-06T07:29:02.306067+0000","last_undegraded":"2024-03-06T07:29:02.306067+0000","last_fullsized":"2024-03-06T07:29:02.306067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:44.319923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338430+0000","last_change":"2024-03-06T07:29:01.303022+0000","last_active":"2024-03-06T07:29:02.338430+0000","last_peered":"2024-03-06T07:29:02.338430+0000","last_clean":"2024-03-06T07:29:02.338430+0000","last_became_active":"2024-03-06T07:29:01.302528+0000","last_became_peered":"2024-03-06T07:29:01.302528+0000","last_unstale":"2024-03-06T07:29:02.338430+0000","last_undegraded":"2024-03-06T07:29:02.338430+0000","last_fullsized":"2024-03-06T07:29:02.338430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.341932+0000","last_change":"2024-03-06T07:29:00.323588+0000","last_active":"2024-03-06T07:29:02.341932+0000","last_peered":"2024-03-06T07:29:02.341932+0000","last_clean":"2024-03-06T07:29:02.341932+0000","last_became_active":"2024-03-06T07:29:00.323304+0000","last_became_peered":"2024-03-06T07:29:00.323304+0000","last_unstale":"2024-03-06T07:29:02.341932+0000","last_undegraded":"2024-03-06T07:29:02.341932+0000","last_fullsized":"2024-03-06T07:29:02.341932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070661+0000","last_change":"2024-03-06T07:29:01.314796+0000","last_active":"2024-03-06T07:29:03.070661+0000","last_peered":"2024-03-06T07:29:03.070661+0000","last_clean":"2024-03-06T07:29:03.070661+0000","last_became_active":"2024-03-06T07:29:01.313844+0000","last_became_peered":"2024-03-06T07:29:01.313844+0000","last_unstale":"2024-03-06T07:29:03.070661+0000","last_undegraded":"2024-03-06T07:29:03.070661+0000","last_fullsized":"2024-03-06T07:29:03.070661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.344249+0000","last_change":"2024-03-06T07:29:00.332089+0000","last_active":"2024-03-06T07:29:02.344249+0000","last_peered":"2024-03-06T07:29:02.344249+0000","last_clean":"2024-03-06T07:29:02.344249+0000","last_became_active":"2024-03-06T07:29:00.331805+0000","last_became_peered":"2024-03-06T07:29:00.331805+0000","last_unstale":"2024-03-06T07:29:02.344249+0000","last_undegraded":"2024-03-06T07:29:02.344249+0000","last_fullsized":"2024-03-06T07:29:02.344249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:28.269881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856581+0000","last_change":"2024-03-06T07:29:01.309249+0000","last_active":"2024-03-06T07:29:02.856581+0000","last_peered":"2024-03-06T07:29:02.856581+0000","last_clean":"2024-03-06T07:29:02.856581+0000","last_became_active":"2024-03-06T07:29:01.308932+0000","last_became_peered":"2024-03-06T07:29:01.308932+0000","last_unstale":"2024-03-06T07:29:02.856581+0000","last_undegraded":"2024-03-06T07:29:02.856581+0000","last_fullsized":"2024-03-06T07:29:02.856581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.185991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.303785+0000","last_change":"2024-03-06T07:29:00.317215+0000","last_active":"2024-03-06T07:29:02.303785+0000","last_peered":"2024-03-06T07:29:02.303785+0000","last_clean":"2024-03-06T07:29:02.303785+0000","last_became_active":"2024-03-06T07:29:00.314874+0000","last_became_peered":"2024-03-06T07:29:00.314874+0000","last_unstale":"2024-03-06T07:29:02.303785+0000","last_undegraded":"2024-03-06T07:29:02.303785+0000","last_fullsized":"2024-03-06T07:29:02.303785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.309601+0000","last_change":"2024-03-06T07:29:01.328026+0000","last_active":"2024-03-06T07:29:02.309601+0000","last_peered":"2024-03-06T07:29:02.309601+0000","last_clean":"2024-03-06T07:29:02.309601+0000","last_became_active":"2024-03-06T07:29:01.327519+0000","last_became_peered":"2024-03-06T07:29:01.327519+0000","last_unstale":"2024-03-06T07:29:02.309601+0000","last_undegraded":"2024-03-06T07:29:02.309601+0000","last_fullsized":"2024-03-06T07:29:02.309601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:29.282488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338149+0000","last_change":"2024-03-06T07:29:00.318811+0000","last_active":"2024-03-06T07:29:02.338149+0000","last_peered":"2024-03-06T07:29:02.338149+0000","last_clean":"2024-03-06T07:29:02.338149+0000","last_became_active":"2024-03-06T07:29:00.318251+0000","last_became_peered":"2024-03-06T07:29:00.318251+0000","last_unstale":"2024-03-06T07:29:02.338149+0000","last_undegraded":"2024-03-06T07:29:02.338149+0000","last_fullsized":"2024-03-06T07:29:02.338149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.069507+0000","last_change":"2024-03-06T07:29:01.298030+0000","last_active":"2024-03-06T07:29:03.069507+0000","last_peered":"2024-03-06T07:29:03.069507+0000","last_clean":"2024-03-06T07:29:03.069507+0000","last_became_active":"2024-03-06T07:29:01.297714+0000","last_became_peered":"2024-03-06T07:29:01.297714+0000","last_unstale":"2024-03-06T07:29:03.069507+0000","last_undegraded":"2024-03-06T07:29:03.069507+0000","last_fullsized":"2024-03-06T07:29:03.069507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:56.704260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.338187+0000","last_change":"2024-03-06T07:29:00.310808+0000","last_active":"2024-03-06T07:29:02.338187+0000","last_peered":"2024-03-06T07:29:02.338187+0000","last_clean":"2024-03-06T07:29:02.338187+0000","last_became_active":"2024-03-06T07:29:00.310511+0000","last_became_peered":"2024-03-06T07:29:00.310511+0000","last_unstale":"2024-03-06T07:29:02.338187+0000","last_undegraded":"2024-03-06T07:29:02.338187+0000","last_fullsized":"2024-03-06T07:29:02.338187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:31.828138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070152+0000","last_change":"2024-03-06T07:29:01.307662+0000","last_active":"2024-03-06T07:29:03.070152+0000","last_peered":"2024-03-06T07:29:03.070152+0000","last_clean":"2024-03-06T07:29:03.070152+0000","last_became_active":"2024-03-06T07:29:01.307375+0000","last_became_peered":"2024-03-06T07:29:01.307375+0000","last_unstale":"2024-03-06T07:29:03.070152+0000","last_undegraded":"2024-03-06T07:29:03.070152+0000","last_fullsized":"2024-03-06T07:29:03.070152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305649+0000","last_change":"2024-03-06T07:29:00.327844+0000","last_active":"2024-03-06T07:29:02.305649+0000","last_peered":"2024-03-06T07:29:02.305649+0000","last_clean":"2024-03-06T07:29:02.305649+0000","last_became_active":"2024-03-06T07:29:00.327512+0000","last_became_peered":"2024-03-06T07:29:00.327512+0000","last_unstale":"2024-03-06T07:29:02.305649+0000","last_undegraded":"2024-03-06T07:29:02.305649+0000","last_fullsized":"2024-03-06T07:29:02.305649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:50.550385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.343323+0000","last_change":"2024-03-06T07:29:01.328118+0000","last_active":"2024-03-06T07:29:02.343323+0000","last_peered":"2024-03-06T07:29:02.343323+0000","last_clean":"2024-03-06T07:29:02.343323+0000","last_became_active":"2024-03-06T07:29:01.327753+0000","last_became_peered":"2024-03-06T07:29:01.327753+0000","last_unstale":"2024-03-06T07:29:02.343323+0000","last_undegraded":"2024-03-06T07:29:02.343323+0000","last_fullsized":"2024-03-06T07:29:02.343323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:19.846343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.332010+0000","last_change":"2024-03-06T07:29:01.343767+0000","last_active":"2024-03-06T07:29:02.332010+0000","last_peered":"2024-03-06T07:29:02.332010+0000","last_clean":"2024-03-06T07:29:02.332010+0000","last_became_active":"2024-03-06T07:29:01.343223+0000","last_became_peered":"2024-03-06T07:29:01.343223+0000","last_unstale":"2024-03-06T07:29:02.332010+0000","last_undegraded":"2024-03-06T07:29:02.332010+0000","last_fullsized":"2024-03-06T07:29:02.332010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:18.207924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.304611+0000","last_change":"2024-03-06T07:29:00.330067+0000","last_active":"2024-03-06T07:29:02.304611+0000","last_peered":"2024-03-06T07:29:02.304611+0000","last_clean":"2024-03-06T07:29:02.304611+0000","last_became_active":"2024-03-06T07:29:00.329686+0000","last_became_peered":"2024-03-06T07:29:00.329686+0000","last_unstale":"2024-03-06T07:29:02.304611+0000","last_undegraded":"2024-03-06T07:29:02.304611+0000","last_fullsized":"2024-03-06T07:29:02.304611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:18.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.331745+0000","last_change":"2024-03-06T07:29:01.338965+0000","last_active":"2024-03-06T07:29:02.331745+0000","last_peered":"2024-03-06T07:29:02.331745+0000","last_clean":"2024-03-06T07:29:02.331745+0000","last_became_active":"2024-03-06T07:29:01.338503+0000","last_became_peered":"2024-03-06T07:29:01.338503+0000","last_unstale":"2024-03-06T07:29:02.331745+0000","last_undegraded":"2024-03-06T07:29:02.331745+0000","last_fullsized":"2024-03-06T07:29:02.331745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.794830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.305289+0000","last_change":"2024-03-06T07:29:00.326670+0000","last_active":"2024-03-06T07:29:02.305289+0000","last_peered":"2024-03-06T07:29:02.305289+0000","last_clean":"2024-03-06T07:29:02.305289+0000","last_became_active":"2024-03-06T07:29:00.326336+0000","last_became_peered":"2024-03-06T07:29:00.326336+0000","last_unstale":"2024-03-06T07:29:02.305289+0000","last_undegraded":"2024-03-06T07:29:02.305289+0000","last_fullsized":"2024-03-06T07:29:02.305289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:41.278698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856601+0000","last_change":"2024-03-06T07:29:01.332554+0000","last_active":"2024-03-06T07:29:02.856601+0000","last_peered":"2024-03-06T07:29:02.856601+0000","last_clean":"2024-03-06T07:29:02.856601+0000","last_became_active":"2024-03-06T07:29:01.332081+0000","last_became_peered":"2024-03-06T07:29:01.332081+0000","last_unstale":"2024-03-06T07:29:02.856601+0000","last_undegraded":"2024-03-06T07:29:02.856601+0000","last_fullsized":"2024-03-06T07:29:02.856601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:38.311914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070670+0000","last_change":"2024-03-06T07:29:00.317549+0000","last_active":"2024-03-06T07:29:03.070670+0000","last_peered":"2024-03-06T07:29:03.070670+0000","last_clean":"2024-03-06T07:29:03.070670+0000","last_became_active":"2024-03-06T07:29:00.317097+0000","last_became_peered":"2024-03-06T07:29:00.317097+0000","last_unstale":"2024-03-06T07:29:03.070670+0000","last_undegraded":"2024-03-06T07:29:03.070670+0000","last_fullsized":"2024-03-06T07:29:03.070670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.920045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:03.070604+0000","last_change":"2024-03-06T07:29:01.319129+0000","last_active":"2024-03-06T07:29:03.070604+0000","last_peered":"2024-03-06T07:29:03.070604+0000","last_clean":"2024-03-06T07:29:03.070604+0000","last_became_active":"2024-03-06T07:29:01.318772+0000","last_became_peered":"2024-03-06T07:29:01.318772+0000","last_unstale":"2024-03-06T07:29:03.070604+0000","last_undegraded":"2024-03-06T07:29:03.070604+0000","last_fullsized":"2024-03-06T07:29:03.070604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:00.266879+0000","last_clean_scrub_stamp":"2024-03-06T07:29:00.266879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:54.101339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:29:02.856882+0000","last_change":"2024-03-06T07:29:00.316149+0000","last_active":"2024-03-06T07:29:02.856882+0000","last_peered":"2024-03-06T07:29:02.856882+0000","last_clean":"2024-03-06T07:29:02.856882+0000","last_became_active":"2024-03-06T07:29:00.315348+0000","last_became_peered":"2024-03-06T07:29:00.315348+0000","last_unstale":"2024-03-06T07:29:02.856882+0000","last_undegraded":"2024-03-06T07:29:02.856882+0000","last_fullsized":"2024-03-06T07:29:02.856882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:59.254272+0000","last_clean_scrub_stamp":"2024-03-06T07:28:59.254272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:26.363805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:11.566 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:29:11.566 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:29:11.566 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:29:11.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:29:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.809+0000 7efd5b200640 1 Processor -- start 2024-03-06T07:29:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.810+0000 7efd5b200640 1 -- start start 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.811+0000 7efd5b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd5c1455e0 0x7efd5c15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.811+0000 7efd5b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efd5c0cc4a0 0x7efd5c0cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.811+0000 7efd5b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c070cd0 con 0x7efd5c06daf0 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.811+0000 7efd5b200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c070a80 con 0x7efd5c1455e0 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.811+0000 7efd5b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c070fb0 con 0x7efd5c0cc4a0 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd5c1455e0 0x7efd5c15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd5c1455e0 0x7efd5c15c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51116/0 (socket says 172.21.15.32:51116) 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 -- 172.21.15.32:0/3873899709 learned_addr learned my addr 172.21.15.32:0/3873899709 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd51400640 1 --2- 172.21.15.32:0/3873899709 >> v2:172.21.15.39:3300/0 conn(0x7efd5c0cc4a0 0x7efd5c0cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 -- 172.21.15.32:0/3873899709 >> v2:172.21.15.39:3300/0 conn(0x7efd5c0cc4a0 msgr2=0x7efd5c0cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 --2- 172.21.15.32:0/3873899709 >> v2:172.21.15.39:3300/0 conn(0x7efd5c0cc4a0 0x7efd5c0cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 -- 172.21.15.32:0/3873899709 >> v1:172.21.15.39:6789/0 conn(0x7efd5c06daf0 legacy=0x7efd5c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.812+0000 7efd59e00640 1 -- 172.21.15.32:0/3873899709 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c071290 con 0x7efd5c1455e0 2024-03-06T07:29:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.814+0000 7efd59e00640 1 --2- 172.21.15.32:0/3873899709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c15c2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd5c06eeb0 tx=0x7efd4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e5cb4bab75628e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.814+0000 7efd59400640 1 -- 172.21.15.32:0/3873899709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c035020 con 0x7efd5c1455e0 2024-03-06T07:29:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd59400640 1 -- 172.21.15.32:0/3873899709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c031870 con 0x7efd5c1455e0 2024-03-06T07:29:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd59400640 1 -- 172.21.15.32:0/3873899709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c031c40 con 0x7efd5c1455e0 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 -- 172.21.15.32:0/3873899709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 msgr2=0x7efd5c15c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 --2- 172.21.15.32:0/3873899709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c15c2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd5c06eeb0 tx=0x7efd4c02b910 comp rx=0 tx=0).stop 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 -- 172.21.15.32:0/3873899709 shutdown_connections 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 --2- 172.21.15.32:0/3873899709 >> v2:172.21.15.39:3300/0 conn(0x7efd5c0cc4a0 0x7efd5c0cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 --2- 172.21.15.32:0/3873899709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c15c2f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 -- 172.21.15.32:0/3873899709 >> 172.21.15.32:0/3873899709 conn(0x7efd5c0b17c0 msgr2=0x7efd5c0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 -- 172.21.15.32:0/3873899709 shutdown_connections 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.815+0000 7efd5b200640 1 -- 172.21.15.32:0/3873899709 wait complete. 2024-03-06T07:29:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.817+0000 7efd5b200640 1 Processor -- start 2024-03-06T07:29:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.817+0000 7efd5b200640 1 -- start start 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd5c06daf0 0x7efd5c0cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd5c0cc4a0 0x7efd5c135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c135790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0cf4c0 con 0x7efd5c1455e0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c1459c0 con 0x7efd5c0cc4a0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0cf640 con 0x7efd5c06daf0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c135790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c135790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51120/0 (socket says 172.21.15.32:51120) 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd5a800640 1 -- 172.21.15.32:0/3084024097 learned_addr learned my addr 172.21.15.32:0/3084024097 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd59e00640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd5c06daf0 0x7efd5c0cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.818+0000 7efd51400640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd5c0cc4a0 0x7efd5c135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5a800640 1 -- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd5c06daf0 msgr2=0x7efd5c0cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5a800640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd5c06daf0 0x7efd5c0cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5a800640 1 -- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd5c0cc4a0 msgr2=0x7efd5c135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5a800640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd5c0cc4a0 0x7efd5c135250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5a800640 1 -- 172.21.15.32:0/3084024097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0cb8c0 con 0x7efd5c1455e0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5a800640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c135790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efd5400c7a0 tx=0x7efd5400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54003b70 con 0x7efd5c1455e0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54003d30 con 0x7efd5c1455e0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5400aa90 con 0x7efd5c1455e0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c1424b0 con 0x7efd5c1455e0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.819+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c142a70 con 0x7efd5c1455e0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.820+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c04f330 con 0x7efd5c1455e0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.821+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd5400ac30 con 0x7efd5c1455e0 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.822+0000 7efd52a00640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd3806b4d0 0x7efd3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.822+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd5408a270 con 0x7efd5c1455e0 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.822+0000 7efd59e00640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd3806b4d0 0x7efd3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.823+0000 7efd59e00640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd3806b4d0 0x7efd3806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd4c036000 tx=0x7efd4c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:11.826+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd5c0cb8c0 con 0x7efd5c1455e0 2024-03-06T07:29:12.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.056+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efd5c0adcd0 con 0x7efd5c1455e0 2024-03-06T07:29:12.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.057+0000 7efd52a00640 1 -- 172.21.15.32:0/3084024097 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7efd54058fc0 con 0x7efd5c1455e0 2024-03-06T07:29:12.059 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:12.059 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:29:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd3806b4d0 msgr2=0x7efd3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd3806b4d0 0x7efd3806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd4c036000 tx=0x7efd4c03c040 comp rx=0 tx=0).stop 2024-03-06T07:29:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 msgr2=0x7efd5c135790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c135790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efd5400c7a0 tx=0x7efd5400cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 shutdown_connections 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd3806b4d0 0x7efd3806d990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c1455e0 0x7efd5c135790 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd5c0cc4a0 0x7efd5c135250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 --2- 172.21.15.32:0/3084024097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd5c06daf0 0x7efd5c0cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.059+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 >> 172.21.15.32:0/3084024097 conn(0x7efd5c0b17c0 msgr2=0x7efd5c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.060+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 shutdown_connections 2024-03-06T07:29:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.060+0000 7efd5b200640 1 -- 172.21.15.32:0/3084024097 wait complete. 2024-03-06T07:29:12.080 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:29:12.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.292+0000 7f13e1400640 1 Processor -- start 2024-03-06T07:29:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.292+0000 7f13e1400640 1 -- start start 2024-03-06T07:29:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.294+0000 7f13e1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f13dc1552b0 0x7f13dc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.294+0000 7f13e1400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc15ed10 con 0x7f13dc0cd060 2024-03-06T07:29:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.294+0000 7f13e1400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc15eff0 con 0x7f13dc15e810 2024-03-06T07:29:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.294+0000 7f13e1400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc15f2d0 con 0x7f13dc1552b0 2024-03-06T07:29:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.294+0000 7f13e0a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f13dc0cd060 0x7f13dc0b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56676/0 (socket says 172.21.15.32:56676) 2024-03-06T07:29:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.294+0000 7f13e0a00640 1 -- 172.21.15.32:0/3811686547 learned_addr learned my addr 172.21.15.32:0/3811686547 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13dbe00640 1 --2- 172.21.15.32:0/3811686547 >> v2:172.21.15.39:3300/0 conn(0x7f13dc1552b0 0x7f13dc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095214706 0 0) 0x7f13dc15ed10 con 0x7f13dc0cd060 2024-03-06T07:29:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0005730 con 0x7f13dc0cd060 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2304169067 0 0) 0x7f13c0005730 con 0x7f13dc0cd060 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 >> v2:172.21.15.39:3300/0 conn(0x7f13dc1552b0 msgr2=0x7f13dc15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 --2- 172.21.15.32:0/3811686547 >> v2:172.21.15.39:3300/0 conn(0x7f13dc1552b0 0x7f13dc15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 >> v1:172.21.15.39:6789/0 conn(0x7f13dc15e810 legacy=0x7f13dc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc15f5b0 con 0x7f13dc0cd060 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f13d4001670 con 0x7f13dc0cd060 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 >> v1:172.21.15.32:6789/0 conn(0x7f13dc0cd060 legacy=0x7f13dc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13c0005c10 0x7f13c0006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13c0006a10 0x7f13c0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0007420 0x7f13c000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc15f2d0 con 0x7f13c0005c10 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c000e370 con 0x7f13c0006a10 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.295+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c000e570 con 0x7f13c0007420 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13dbe00640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13c0005c10 0x7f13c0006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13dbe00640 1 -- 172.21.15.32:0/3811686547 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0007420 msgr2=0x7f13c000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13dbe00640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0007420 0x7f13c000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13dbe00640 1 -- 172.21.15.32:0/3811686547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13c0006a10 msgr2=0x7f13c0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13dbe00640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13c0006a10 0x7f13c0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13dbe00640 1 -- 172.21.15.32:0/3811686547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c0005730 con 0x7f13c0005c10 2024-03-06T07:29:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.296+0000 7f13e0a00640 1 -- 172.21.15.32:0/3811686547 reap_dead start 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.297+0000 7f13dbe00640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13c0005c10 0x7f13c0006030 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009a90 tx=0x7f13c0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590f65c71f89e41f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.297+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc039070 con 0x7f13c0005c10 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.297+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc02c680 con 0x7f13c0005c10 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.297+0000 7f13db400640 1 -- 172.21.15.32:0/3811686547 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc009690 con 0x7f13c0005c10 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 -- 172.21.15.32:0/3811686547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13c0005c10 msgr2=0x7f13c0006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13c0005c10 0x7f13c0006030 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009a90 tx=0x7f13c0007a40 comp rx=0 tx=0).stop 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 -- 172.21.15.32:0/3811686547 shutdown_connections 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 --2- 172.21.15.32:0/3811686547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13c0005c10 0x7f13c0006030 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 -- 172.21.15.32:0/3811686547 >> 172.21.15.32:0/3811686547 conn(0x7f13dc06b3d0 msgr2=0x7f13dc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 -- 172.21.15.32:0/3811686547 shutdown_connections 2024-03-06T07:29:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.298+0000 7f13e1400640 1 -- 172.21.15.32:0/3811686547 wait complete. 2024-03-06T07:29:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.300+0000 7f13e1400640 1 Processor -- start 2024-03-06T07:29:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.300+0000 7f13e1400640 1 -- start start 2024-03-06T07:29:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e1400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0005c10 0x7f13dc1e4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 0x7f13dc14e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc1509d0 0x7f13dc150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e1400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc06dd30 con 0x7f13dc14dbb0 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e1400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0b3760 con 0x7f13dc1509d0 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e1400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc06deb0 con 0x7f13c0005c10 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13d3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 0x7f13dc14e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13d3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 0x7f13dc14e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51136/0 (socket says 172.21.15.32:51136) 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13d3400640 1 -- 172.21.15.32:0/1924908657 learned_addr learned my addr 172.21.15.32:0/1924908657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13e0a00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc1509d0 0x7f13dc150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.301+0000 7f13dbe00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0005c10 0x7f13dc1e4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d3400640 1 -- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0005c10 msgr2=0x7f13dc1e4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d3400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0005c10 0x7f13dc1e4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d3400640 1 -- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc1509d0 msgr2=0x7f13dc150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d3400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc1509d0 0x7f13dc150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d3400640 1 -- 172.21.15.32:0/1924908657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc1e6410 con 0x7f13dc14dbb0 2024-03-06T07:29:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13dbe00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0005c10 0x7f13dc1e4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13e0a00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc1509d0 0x7f13dc150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d3400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 0x7f13dc14e030 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c770 tx=0x7f13c400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4003b70 con 0x7f13dc14dbb0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4003d30 con 0x7f13dc14dbb0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c400aa30 con 0x7f13dc14dbb0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc14e740 con 0x7f13dc14dbb0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.302+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc1501d0 con 0x7f13dc14dbb0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.304+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc04f330 con 0x7f13dc14dbb0 2024-03-06T07:29:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.304+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13c400abd0 con 0x7f13dc14dbb0 2024-03-06T07:29:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.305+0000 7f13d8c00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f13b806b500 0x7f13b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.305+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13c408a0b0 con 0x7f13dc14dbb0 2024-03-06T07:29:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.306+0000 7f13dbe00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f13b806b500 0x7f13b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.306+0000 7f13dbe00640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f13b806b500 0x7f13b806d9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13cc036000 tx=0x7f13cc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.310+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13c4058f20 con 0x7f13dc14dbb0 2024-03-06T07:29:12.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.505+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13dc0cf4a0 con 0x7f13dc14dbb0 2024-03-06T07:29:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.508+0000 7f13d8c00640 1 -- 172.21.15.32:0/1924908657 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f13c4058820 con 0x7f13dc14dbb0 2024-03-06T07:29:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:12.511 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f13b806b500 msgr2=0x7f13b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f13b806b500 0x7f13b806d9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13cc036000 tx=0x7f13cc030000 comp rx=0 tx=0).stop 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 msgr2=0x7f13dc14e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 0x7f13dc14e030 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c770 tx=0x7f13c400cc30 comp rx=0 tx=0).stop 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 shutdown_connections 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f13b806b500 0x7f13b806d9c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13dc1509d0 0x7f13dc150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13dc14dbb0 0x7f13dc14e030 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 --2- 172.21.15.32:0/1924908657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13c0005c10 0x7f13dc1e4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.511+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 >> 172.21.15.32:0/1924908657 conn(0x7f13dc06b3d0 msgr2=0x7f13dc071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.512+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 shutdown_connections 2024-03-06T07:29:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.512+0000 7f13e1400640 1 -- 172.21.15.32:0/1924908657 wait complete. 2024-03-06T07:29:12.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:12.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.745+0000 7f9e4e800640 1 Processor -- start 2024-03-06T07:29:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.745+0000 7f9e4e800640 1 -- start start 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.747+0000 7f9e4e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9e48162bb0 0x7f9e48163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.747+0000 7f9e4e800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e48163950 con 0x7f9e480ccad0 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.747+0000 7f9e4e800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480acd40 con 0x7f9e480adb00 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.747+0000 7f9e4e800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e480ad020 con 0x7f9e48162bb0 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e47400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9e480ccad0 0x7f9e480cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56678/0 (socket says 172.21.15.32:56678) 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e47400640 1 -- 172.21.15.32:0/4134681181 learned_addr learned my addr 172.21.15.32:0/4134681181 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46a00640 1 --2- 172.21.15.32:0/4134681181 >> v2:172.21.15.39:3300/0 conn(0x7f9e48162bb0 0x7f9e48163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223992829 0 0) 0x7f9e48163950 con 0x7f9e480ccad0 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2c003520 con 0x7f9e480ccad0 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289233851 0 0) 0x7f9e2c003520 con 0x7f9e480ccad0 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 >> v2:172.21.15.39:3300/0 conn(0x7f9e48162bb0 msgr2=0x7f9e48163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 --2- 172.21.15.32:0/4134681181 >> v2:172.21.15.39:3300/0 conn(0x7f9e48162bb0 0x7f9e48163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 >> v1:172.21.15.39:6789/0 conn(0x7f9e480adb00 legacy=0x7f9e480adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.748+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e480ad300 con 0x7f9e480ccad0 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f9e38001670 con 0x7f9e480ccad0 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 >> v1:172.21.15.32:6789/0 conn(0x7f9e480ccad0 legacy=0x7f9e480cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e2c003a00 0x7f9e2c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e2c005210 0x7f9e2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e480ad020 con 0x7f9e2c004800 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c00c160 con 0x7f9e2c003a00 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c00c360 con 0x7f9e2c005210 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46a00640 1 --2- 172.21.15.32:0/4134681181 >> v2:172.21.15.39:3300/0 conn(0x7f9e48162bb0 0x7f9e48163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46a00640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e47400640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e2c003a00 0x7f9e2c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e46a00640 1 -- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e2c005210 msgr2=0x7f9e2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.749+0000 7f9e47e00640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e2c005210 0x7f9e2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e46a00640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e2c005210 0x7f9e2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e46a00640 1 -- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e2c003a00 msgr2=0x7f9e2c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e46a00640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e2c003a00 0x7f9e2c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e46a00640 1 -- 172.21.15.32:0/4134681181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c003520 con 0x7f9e2c004800 2024-03-06T07:29:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e47400640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e2c003a00 0x7f9e2c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e47e00640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e2c005210 0x7f9e2c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:12.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.750+0000 7f9e47e00640 1 -- 172.21.15.32:0/4134681181 reap_dead start 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.751+0000 7f9e46a00640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e2c004c80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9e340098a0 tx=0x7f9e2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a0d24c1a1b0745 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.751+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34034070 con 0x7f9e2c004800 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.751+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e34030610 con 0x7f9e2c004800 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.751+0000 7f9e46000640 1 -- 172.21.15.32:0/4134681181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400f040 con 0x7f9e2c004800 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 -- 172.21.15.32:0/4134681181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 msgr2=0x7f9e2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e2c004c80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9e340098a0 tx=0x7f9e2c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 -- 172.21.15.32:0/4134681181 shutdown_connections 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 --2- 172.21.15.32:0/4134681181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e2c004c80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 -- 172.21.15.32:0/4134681181 >> 172.21.15.32:0/4134681181 conn(0x7f9e4806b3d0 msgr2=0x7f9e4806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 -- 172.21.15.32:0/4134681181 shutdown_connections 2024-03-06T07:29:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.752+0000 7f9e4e800640 1 -- 172.21.15.32:0/4134681181 wait complete. 2024-03-06T07:29:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.753+0000 7f9e4e800640 1 Processor -- start 2024-03-06T07:29:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.754+0000 7f9e4e800640 1 -- start start 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e4e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e480b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e4e800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4814cf00 0x7f9e480af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e4e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e4809af50 0x7f9e480b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e4e800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4806c980 con 0x7f9e2c004800 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e4e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4806e7d0 con 0x7f9e4809af50 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e4e800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e480cceb0 con 0x7f9e4814cf00 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e480b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e480b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51156/0 (socket says 172.21.15.32:51156) 2024-03-06T07:29:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 -- 172.21.15.32:0/537700424 learned_addr learned my addr 172.21.15.32:0/537700424 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 -- 172.21.15.32:0/537700424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4814cf00 msgr2=0x7f9e480af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4814cf00 0x7f9e480af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 -- 172.21.15.32:0/537700424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e4809af50 msgr2=0x7f9e480b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e4809af50 0x7f9e480b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 -- 172.21.15.32:0/537700424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4809b3e0 con 0x7f9e2c004800 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.755+0000 7f9e47400640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e480b1060 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800e7d0 tx=0x7f9e3800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.756+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3800c7f0 con 0x7f9e2c004800 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.756+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e38018070 con 0x7f9e2c004800 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.756+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3800cb90 con 0x7f9e2c004800 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.756+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e480affc0 con 0x7f9e2c004800 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.756+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e480b0580 con 0x7f9e2c004800 2024-03-06T07:29:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.757+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4804f330 con 0x7f9e2c004800 2024-03-06T07:29:12.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.758+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e3801d460 con 0x7f9e2c004800 2024-03-06T07:29:12.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.759+0000 7f9e3f400640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2806b500 0x7f9e2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.759+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e3808a110 con 0x7f9e2c004800 2024-03-06T07:29:12.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.759+0000 7f9e46a00640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2806b500 0x7f9e2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.760+0000 7f9e46a00640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2806b500 0x7f9e2806d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e34036000 tx=0x7f9e34030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.764+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e38058ef0 con 0x7f9e2c004800 2024-03-06T07:29:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.962+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e481603a0 con 0x7f9e2c004800 2024-03-06T07:29:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.963+0000 7f9e3f400640 1 -- 172.21.15.32:0/537700424 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e380587f0 con 0x7f9e2c004800 2024-03-06T07:29:12.965 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.965+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2806b500 msgr2=0x7f9e2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2806b500 0x7f9e2806d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e34036000 tx=0x7f9e34030000 comp rx=0 tx=0).stop 2024-03-06T07:29:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 msgr2=0x7f9e480b1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e480b1060 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800e7d0 tx=0x7f9e3800ec90 comp rx=0 tx=0).stop 2024-03-06T07:29:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 shutdown_connections 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2806b500 0x7f9e2806d9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e4809af50 0x7f9e480b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4814cf00 0x7f9e480af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 --2- 172.21.15.32:0/537700424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e2c004800 0x7f9e480b1060 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 >> 172.21.15.32:0/537700424 conn(0x7f9e4806b3d0 msgr2=0x7f9e481615d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 shutdown_connections 2024-03-06T07:29:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:12.966+0000 7f9e4e800640 1 -- 172.21.15.32:0/537700424 wait complete. 2024-03-06T07:29:12.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:13.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.211+0000 7fa4f2400640 1 Processor -- start 2024-03-06T07:29:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.211+0000 7fa4f2400640 1 -- start start 2024-03-06T07:29:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f2400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4ec0729d0 0x7fa4ec0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f2400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f2400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ec071930 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f2400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec0b24e0 con 0x7fa4ec0729d0 2024-03-06T07:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f2400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec15bfb0 con 0x7fa4ec0b1b40 2024-03-06T07:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4ebe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4ec0729d0 0x7fa4ec0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f1000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa4ec0b3c20 0x7fa4ec15b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56682/0 (socket says 172.21.15.32:56682) 2024-03-06T07:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.213+0000 7fa4f1000640 1 -- 172.21.15.32:0/2552415807 learned_addr learned my addr 172.21.15.32:0/2552415807 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.214+0000 7fa4f1a00640 1 --2- 172.21.15.32:0/2552415807 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.214+0000 7fa4ebe00640 1 -- 172.21.15.32:0/2552415807 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 msgr2=0x7fa4ec0b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.214+0000 7fa4ebe00640 1 --2- 172.21.15.32:0/2552415807 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec0b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.214+0000 7fa4ebe00640 1 -- 172.21.15.32:0/2552415807 >> v1:172.21.15.32:6789/0 conn(0x7fa4ec0b3c20 legacy=0x7fa4ec15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.214+0000 7fa4ebe00640 1 -- 172.21.15.32:0/2552415807 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec071c10 con 0x7fa4ec0729d0 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.214+0000 7fa4eb400640 1 -- 172.21.15.32:0/2552415807 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125191717 0 0) 0x7fa4ec071930 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.215+0000 7fa4ebe00640 1 --2- 172.21.15.32:0/2552415807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec0b4000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009d90 tx=0x7fa4dc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84103eb37cca4d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.215+0000 7fa4eb400640 1 -- 172.21.15.32:0/2552415807 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc039070 con 0x7fa4ec0729d0 2024-03-06T07:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.215+0000 7fa4eb400640 1 -- 172.21.15.32:0/2552415807 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc02cc50 con 0x7fa4ec0729d0 2024-03-06T07:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4eb400640 1 -- 172.21.15.32:0/2552415807 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc034440 con 0x7fa4ec0729d0 2024-03-06T07:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 -- 172.21.15.32:0/2552415807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0729d0 msgr2=0x7fa4ec0b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2552415807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec0b4000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009d90 tx=0x7fa4dc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 -- 172.21.15.32:0/2552415807 shutdown_connections 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2552415807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec0b4000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2552415807 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec0b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 -- 172.21.15.32:0/2552415807 >> 172.21.15.32:0/2552415807 conn(0x7fa4ec06b3a0 msgr2=0x7fa4ec06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 -- 172.21.15.32:0/2552415807 shutdown_connections 2024-03-06T07:29:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.216+0000 7fa4f2400640 1 -- 172.21.15.32:0/2552415807 wait complete. 2024-03-06T07:29:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.218+0000 7fa4f2400640 1 Processor -- start 2024-03-06T07:29:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.218+0000 7fa4f2400640 1 -- start start 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec14e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f2400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec1424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 0x7fa4ec1f7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f2400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0724b0 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f2400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0721d0 con 0x7fa4ec0729d0 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f2400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec15c220 con 0x7fa4ec0b1b40 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 0x7fa4ec1f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 0x7fa4ec1f7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51172/0 (socket says 172.21.15.32:51172) 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.219+0000 7fa4f1a00640 1 -- 172.21.15.32:0/2007239000 learned_addr learned my addr 172.21.15.32:0/2007239000 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4ebe00640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec1424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1000640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec14e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1a00640 1 -- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 msgr2=0x7fa4ec1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1a00640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec1424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1a00640 1 -- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4ec0729d0 msgr2=0x7fa4ec14e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1a00640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec14e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1a00640 1 -- 172.21.15.32:0/2007239000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec1f7e80 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4f1a00640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 0x7fa4ec1f7510 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d800 tx=0x7fa4e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400b6c0 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e4019070 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.220+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400bac0 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.221+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec1f8160 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.221+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec151230 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.222+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec04f330 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.223+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4e400bc60 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.224+0000 7fa4e8c00640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4c806b4d0 0x7fa4c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.224+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4e408a710 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.224+0000 7fa4f1000640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4c806b4d0 0x7fa4c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.225+0000 7fa4f1000640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4c806b4d0 0x7fa4c806d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4000f60 tx=0x7fa4d4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.229+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4e40594f0 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.428+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa4ec15d7c0 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.429+0000 7fa4e8c00640 1 -- 172.21.15.32:0/2007239000 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa4e4058df0 con 0x7fa4ec0b3c20 2024-03-06T07:29:13.431 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4c806b4d0 msgr2=0x7fa4c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4c806b4d0 0x7fa4c806d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4000f60 tx=0x7fa4d4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 msgr2=0x7fa4ec1f7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 0x7fa4ec1f7510 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d800 tx=0x7fa4e400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 shutdown_connections 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4c806b4d0 0x7fa4c806d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ec0b3c20 0x7fa4ec1f7510 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4ec0b1b40 0x7fa4ec1424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 --2- 172.21.15.32:0/2007239000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4ec0729d0 0x7fa4ec14e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 >> 172.21.15.32:0/2007239000 conn(0x7fa4ec06b3a0 msgr2=0x7fa4ec15b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 shutdown_connections 2024-03-06T07:29:13.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.431+0000 7fa4f2400640 1 -- 172.21.15.32:0/2007239000 wait complete. 2024-03-06T07:29:13.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:13.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.670+0000 7f3c4b200640 1 Processor -- start 2024-03-06T07:29:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.670+0000 7f3c4b200640 1 -- start start 2024-03-06T07:29:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c4b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3c4c15f810 0x7f3c4c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c4b200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4c15fe40 con 0x7f3c4c145430 2024-03-06T07:29:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c4b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4c160120 con 0x7f3c4c0b2080 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c4b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c0b24d0 con 0x7f3c4c15f810 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c49e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c4c145430 0x7f3c4c15cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56688/0 (socket says 172.21.15.32:56688) 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c49e00640 1 -- 172.21.15.32:0/2910315298 learned_addr learned my addr 172.21.15.32:0/2910315298 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c49400640 1 --2- 172.21.15.32:0/2910315298 >> v2:172.21.15.39:3300/0 conn(0x7f3c4c15f810 0x7f3c4c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.672+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756190159 0 0) 0x7f3c4c15fe40 con 0x7f3c4c145430 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c30003520 con 0x7f3c4c145430 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064215140 0 0) 0x7f3c30003520 con 0x7f3c4c145430 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 >> v2:172.21.15.39:3300/0 conn(0x7f3c4c15f810 msgr2=0x7f3c4c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 --2- 172.21.15.32:0/2910315298 >> v2:172.21.15.39:3300/0 conn(0x7f3c4c15f810 0x7f3c4c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 >> v1:172.21.15.39:6789/0 conn(0x7f3c4c0b2080 legacy=0x7f3c4c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c160400 con 0x7f3c4c145430 2024-03-06T07:29:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3c3c001670 con 0x7f3c4c145430 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 >> v1:172.21.15.32:6789/0 conn(0x7f3c4c145430 legacy=0x7f3c4c15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c300039b0 0x7f3c30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c300051c0 0x7f3c3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0b24d0 con 0x7f3c30004800 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3000c060 con 0x7f3c300051c0 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3000c260 con 0x7f3c300039b0 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.673+0000 7f3c49400640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c49e00640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c300039b0 0x7f3c30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c49400640 1 -- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c300039b0 msgr2=0x7f3c30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c4a800640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c300051c0 0x7f3c3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c49400640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c300039b0 0x7f3c30003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c49400640 1 -- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c300051c0 msgr2=0x7f3c3000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c49400640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c300051c0 0x7f3c3000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c49400640 1 -- 172.21.15.32:0/2910315298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30003520 con 0x7f3c30004800 2024-03-06T07:29:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c4a800640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c300051c0 0x7f3c3000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.674+0000 7f3c4a800640 1 -- 172.21.15.32:0/2910315298 reap_dead start 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.675+0000 7f3c49400640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c30004c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009ac0 tx=0x7f3c30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96714e2477bfa9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.675+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38039070 con 0x7f3c30004800 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.675+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3802c6b0 con 0x7f3c30004800 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.675+0000 7f3c48a00640 1 -- 172.21.15.32:0/2910315298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38009690 con 0x7f3c30004800 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 -- 172.21.15.32:0/2910315298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 msgr2=0x7f3c30004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c30004c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009ac0 tx=0x7f3c30005830 comp rx=0 tx=0).stop 2024-03-06T07:29:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 -- 172.21.15.32:0/2910315298 shutdown_connections 2024-03-06T07:29:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 --2- 172.21.15.32:0/2910315298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c30004c80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 -- 172.21.15.32:0/2910315298 >> 172.21.15.32:0/2910315298 conn(0x7f3c4c06b3a0 msgr2=0x7f3c4c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 -- 172.21.15.32:0/2910315298 shutdown_connections 2024-03-06T07:29:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.676+0000 7f3c4b200640 1 -- 172.21.15.32:0/2910315298 wait complete. 2024-03-06T07:29:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.678+0000 7f3c4b200640 1 Processor -- start 2024-03-06T07:29:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.678+0000 7f3c4b200640 1 -- start start 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c4c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c4c0aeed0 0x7f3c4c0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c4c157160 0x7f3c4c1575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c145990 con 0x7f3c30004800 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c145810 con 0x7f3c4c157160 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0cde00 con 0x7f3c4c0aeed0 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c4c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c4c157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51194/0 (socket says 172.21.15.32:51194) 2024-03-06T07:29:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 -- 172.21.15.32:0/533219202 learned_addr learned my addr 172.21.15.32:0/533219202 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c4a800640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c4c157160 0x7f3c4c1575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49400640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c4c0aeed0 0x7f3c4c0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 -- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c4c0aeed0 msgr2=0x7f3c4c0b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c4c0aeed0 0x7f3c4c0b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 -- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c4c157160 msgr2=0x7f3c4c1575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c4c157160 0x7f3c4c1575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.679+0000 7f3c49e00640 1 -- 172.21.15.32:0/533219202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c0af350 con 0x7f3c30004800 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.680+0000 7f3c49e00640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c4c157e60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00d7e0 tx=0x7f3c3c00dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.680+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c019070 con 0x7f3c30004800 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.680+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c0074e0 con 0x7f3c30004800 2024-03-06T07:29:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.680+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c004a00 con 0x7f3c30004800 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.680+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4c157b20 con 0x7f3c30004800 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.680+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4c13e710 con 0x7f3c30004800 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.681+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c04f330 con 0x7f3c30004800 2024-03-06T07:29:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.682+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c3c004ba0 con 0x7f3c30004800 2024-03-06T07:29:13.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.683+0000 7f3c42000640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c2c06b4d0 0x7f3c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.683+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c3c089c90 con 0x7f3c30004800 2024-03-06T07:29:13.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.683+0000 7f3c49400640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c2c06b4d0 0x7f3c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.684+0000 7f3c49400640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c2c06b4d0 0x7f3c2c06d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3c38036000 tx=0x7f3c38030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.687+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c4c157b20 con 0x7f3c30004800 2024-03-06T07:29:13.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.879+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c4c06e240 con 0x7f3c30004800 2024-03-06T07:29:13.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.879+0000 7f3c42000640 1 -- 172.21.15.32:0/533219202 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c3c058a70 con 0x7f3c30004800 2024-03-06T07:29:13.882 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.881+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c2c06b4d0 msgr2=0x7f3c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.881+0000 7f3c4b200640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c2c06b4d0 0x7f3c2c06d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3c38036000 tx=0x7f3c38030000 comp rx=0 tx=0).stop 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.881+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 msgr2=0x7f3c4c157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c4c157e60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00d7e0 tx=0x7f3c3c00dca0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 shutdown_connections 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3c2c06b4d0 0x7f3c2c06d990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c4c157160 0x7f3c4c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c4c0aeed0 0x7f3c4c0b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 --2- 172.21.15.32:0/533219202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c30004800 0x7f3c4c157e60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 >> 172.21.15.32:0/533219202 conn(0x7f3c4c06b3a0 msgr2=0x7f3c4c0b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 shutdown_connections 2024-03-06T07:29:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:13.882+0000 7f3c4b200640 1 -- 172.21.15.32:0/533219202 wait complete. 2024-03-06T07:29:13.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:14.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.118+0000 7ffaa6000640 1 Processor -- start 2024-03-06T07:29:14.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.118+0000 7ffaa6000640 1 -- start start 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffaa6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0071380 0x7ffaa00ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffaa6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffaa006e240 0x7ffaa00ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffaa6000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa00ce9a0 con 0x7ffaa0161670 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffaa6000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa00730d0 con 0x7ffaa006e240 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffaa6000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00cec80 con 0x7ffaa0071380 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffa9f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffaa0161670 0x7ffaa0070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56698/0 (socket says 172.21.15.32:56698) 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffa9f400640 1 -- 172.21.15.32:0/1144881728 learned_addr learned my addr 172.21.15.32:0/1144881728 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffa9fe00640 1 --2- 172.21.15.32:0/1144881728 >> v2:172.21.15.32:3300/0 conn(0x7ffaa006e240 0x7ffaa00ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.120+0000 7ffa97e00640 1 --2- 172.21.15.32:0/1144881728 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0071380 0x7ffaa00ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.121+0000 7ffa9ea00640 1 -- 172.21.15.32:0/1144881728 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936629356 0 0) 0x7ffaa00ce9a0 con 0x7ffaa0161670 2024-03-06T07:29:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.121+0000 7ffa9ea00640 1 -- 172.21.15.32:0/1144881728 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa84003520 con 0x7ffaa0161670 2024-03-06T07:29:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.121+0000 7ffa9fe00640 1 -- 172.21.15.32:0/1144881728 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0071380 msgr2=0x7ffaa00ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.121+0000 7ffa9fe00640 1 --2- 172.21.15.32:0/1144881728 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0071380 0x7ffaa00ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.121+0000 7ffa9fe00640 1 -- 172.21.15.32:0/1144881728 >> v1:172.21.15.32:6789/0 conn(0x7ffaa0161670 legacy=0x7ffaa0070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.121+0000 7ffa9fe00640 1 -- 172.21.15.32:0/1144881728 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa00cef60 con 0x7ffaa006e240 2024-03-06T07:29:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.122+0000 7ffa9fe00640 1 --2- 172.21.15.32:0/1144881728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa00ad5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009d90 tx=0x7ffa9802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54990c98c5ec47c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.122+0000 7ffa9ea00640 1 -- 172.21.15.32:0/1144881728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98039070 con 0x7ffaa006e240 2024-03-06T07:29:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.122+0000 7ffa9ea00640 1 -- 172.21.15.32:0/1144881728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9802cc50 con 0x7ffaa006e240 2024-03-06T07:29:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.122+0000 7ffa9ea00640 1 -- 172.21.15.32:0/1144881728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98034440 con 0x7ffaa006e240 2024-03-06T07:29:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 -- 172.21.15.32:0/1144881728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 msgr2=0x7ffaa00ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 --2- 172.21.15.32:0/1144881728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa00ad5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009d90 tx=0x7ffa9802bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 -- 172.21.15.32:0/1144881728 shutdown_connections 2024-03-06T07:29:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 --2- 172.21.15.32:0/1144881728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa00ad5d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 --2- 172.21.15.32:0/1144881728 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0071380 0x7ffaa00ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 -- 172.21.15.32:0/1144881728 >> 172.21.15.32:0/1144881728 conn(0x7ffaa006afb0 msgr2=0x7ffaa006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 -- 172.21.15.32:0/1144881728 shutdown_connections 2024-03-06T07:29:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.123+0000 7ffaa6000640 1 -- 172.21.15.32:0/1144881728 wait complete. 2024-03-06T07:29:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.125+0000 7ffaa6000640 1 Processor -- start 2024-03-06T07:29:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.125+0000 7ffaa6000640 1 -- start start 2024-03-06T07:29:14.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffaa6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa01486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffaa6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaa0071380 0x7ffaa009b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffaa6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0161670 0x7ffaa009b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffaa6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00ce600 con 0x7ffaa006e240 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffaa6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00cf520 con 0x7ffaa0071380 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffaa6000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00ce780 con 0x7ffaa0161670 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa01486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa01486f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51216/0 (socket says 172.21.15.32:51216) 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 -- 172.21.15.32:0/868857222 learned_addr learned my addr 172.21.15.32:0/868857222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa97e00640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaa0071380 0x7ffaa009b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 -- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0161670 msgr2=0x7ffaa009b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9fe00640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0161670 0x7ffaa009b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0161670 0x7ffaa009b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 -- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaa0071380 msgr2=0x7ffaa009b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaa0071380 0x7ffaa009b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa9f400640 1 -- 172.21.15.32:0/868857222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa009bf90 con 0x7ffaa006e240 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.126+0000 7ffa97e00640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaa0071380 0x7ffaa009b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.127+0000 7ffa9f400640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa01486f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c390 tx=0x7ffa9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.127+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90017070 con 0x7ffaa006e240 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.127+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa90005ce0 con 0x7ffaa006e240 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.127+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa009c270 con 0x7ffaa006e240 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.127+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa900072f0 con 0x7ffaa006e240 2024-03-06T07:29:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.127+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa009c830 con 0x7ffaa006e240 2024-03-06T07:29:14.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.128+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa0069510 con 0x7ffaa006e240 2024-03-06T07:29:14.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.130+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa9001a070 con 0x7ffaa006e240 2024-03-06T07:29:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.130+0000 7ffa97400640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa7c06b4d0 0x7ffa7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.131+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa900896a0 con 0x7ffaa006e240 2024-03-06T07:29:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.131+0000 7ffa97e00640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa7c06b4d0 0x7ffa7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.132+0000 7ffa97e00640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa7c06b4d0 0x7ffa7c06d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00cc150 tx=0x7ffa88008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.135+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa90058480 con 0x7ffaa006e240 2024-03-06T07:29:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.373+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaa004f3a0 con 0x7ffaa006e240 2024-03-06T07:29:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.374+0000 7ffa97400640 1 -- 172.21.15.32:0/868857222 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7ffa90057d80 con 0x7ffaa006e240 2024-03-06T07:29:14.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:14.377 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa7c06b4d0 msgr2=0x7ffa7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa7c06b4d0 0x7ffa7c06d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00cc150 tx=0x7ffa88008040 comp rx=0 tx=0).stop 2024-03-06T07:29:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 msgr2=0x7ffaa01486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa01486f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c390 tx=0x7ffa9000c850 comp rx=0 tx=0).stop 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 shutdown_connections 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa7c06b4d0 0x7ffa7c06d990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffaa0161670 0x7ffaa009b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaa0071380 0x7ffaa009b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 --2- 172.21.15.32:0/868857222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa006e240 0x7ffaa01486f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 >> 172.21.15.32:0/868857222 conn(0x7ffaa006afb0 msgr2=0x7ffaa00b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 shutdown_connections 2024-03-06T07:29:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.377+0000 7ffaa6000640 1 -- 172.21.15.32:0/868857222 wait complete. 2024-03-06T07:29:14.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:14.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.621+0000 7f318cc00640 1 Processor -- start 2024-03-06T07:29:14.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.622+0000 7f318cc00640 1 -- start start 2024-03-06T07:29:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.623+0000 7f318cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f31881477e0 0x7f3188162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f318cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f318806b6c0 0x7f31880d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f318cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188147bc0 con 0x7f318806ef20 2024-03-06T07:29:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f318cc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3188161f90 con 0x7f318806b6c0 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f318cc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3188162960 con 0x7f31881477e0 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f3187e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f318806b6c0 0x7f31880d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f3187e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f318806b6c0 0x7f31880d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51232/0 (socket says 172.21.15.32:51232) 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f3187e00640 1 -- 172.21.15.32:0/3315715107 learned_addr learned my addr 172.21.15.32:0/3315715107 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.624+0000 7f3187400640 1 --2- 172.21.15.32:0/3315715107 >> v2:172.21.15.39:3300/0 conn(0x7f31881477e0 0x7f3188162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.625+0000 7f3187e00640 1 -- 172.21.15.32:0/3315715107 >> v2:172.21.15.39:3300/0 conn(0x7f31881477e0 msgr2=0x7f3188162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.625+0000 7f3187e00640 1 --2- 172.21.15.32:0/3315715107 >> v2:172.21.15.39:3300/0 conn(0x7f31881477e0 0x7f3188162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.625+0000 7f3187e00640 1 -- 172.21.15.32:0/3315715107 >> v1:172.21.15.39:6789/0 conn(0x7f318806ef20 legacy=0x7f318806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.625+0000 7f3187e00640 1 -- 172.21.15.32:0/3315715107 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880b3d70 con 0x7f318806b6c0 2024-03-06T07:29:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.626+0000 7f3187e00640 1 --2- 172.21.15.32:0/3315715107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f318806b6c0 0x7f31880d0910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f318806c050 tx=0x7f318002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3e098d7ed58c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.626+0000 7f3186000640 1 -- 172.21.15.32:0/3315715107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318002ca80 con 0x7f318806b6c0 2024-03-06T07:29:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.626+0000 7f3186000640 1 -- 172.21.15.32:0/3315715107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318002cc20 con 0x7f318806b6c0 2024-03-06T07:29:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.626+0000 7f3186000640 1 -- 172.21.15.32:0/3315715107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31800343f0 con 0x7f318806b6c0 2024-03-06T07:29:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 -- 172.21.15.32:0/3315715107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f318806b6c0 msgr2=0x7f31880d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 --2- 172.21.15.32:0/3315715107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f318806b6c0 0x7f31880d0910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f318806c050 tx=0x7f318002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 -- 172.21.15.32:0/3315715107 shutdown_connections 2024-03-06T07:29:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 --2- 172.21.15.32:0/3315715107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f318806b6c0 0x7f31880d0910 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 --2- 172.21.15.32:0/3315715107 >> v2:172.21.15.39:3300/0 conn(0x7f31881477e0 0x7f3188162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 -- 172.21.15.32:0/3315715107 >> 172.21.15.32:0/3315715107 conn(0x7f318806c0c0 msgr2=0x7f318806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 -- 172.21.15.32:0/3315715107 shutdown_connections 2024-03-06T07:29:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.627+0000 7f318cc00640 1 -- 172.21.15.32:0/3315715107 wait complete. 2024-03-06T07:29:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.629+0000 7f318cc00640 1 Processor -- start 2024-03-06T07:29:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.629+0000 7f318cc00640 1 -- start start 2024-03-06T07:29:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f318cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f318806b6c0 0x7f318813e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f318cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f318806ef20 0x7f3188143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f318cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 0x7f318815d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f318cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3188161c20 con 0x7f31881477e0 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f318cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31880b4330 con 0x7f318806b6c0 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f318cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318806f8f0 con 0x7f318806ef20 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f3187e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 0x7f318815d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f3187e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 0x7f318815d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51244/0 (socket says 172.21.15.32:51244) 2024-03-06T07:29:14.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.630+0000 7f3187e00640 1 -- 172.21.15.32:0/1152448380 learned_addr learned my addr 172.21.15.32:0/1152448380 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187400640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f318806b6c0 0x7f318813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3186a00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f318806ef20 0x7f3188143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187e00640 1 -- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f318806ef20 msgr2=0x7f3188143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187e00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f318806ef20 0x7f3188143100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187e00640 1 -- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f318806b6c0 msgr2=0x7f318813e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187e00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f318806b6c0 0x7f318813e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187e00640 1 -- 172.21.15.32:0/1152448380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318815ec80 con 0x7f31881477e0 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3186a00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f318806ef20 0x7f3188143100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187400640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f318806b6c0 0x7f318813e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.631+0000 7f3187e00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 0x7f318815d680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3180036040 tx=0x7f3180004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.632+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180043020 con 0x7f31881477e0 2024-03-06T07:29:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.632+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318003e450 con 0x7f31881477e0 2024-03-06T07:29:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.632+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180009510 con 0x7f31881477e0 2024-03-06T07:29:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.632+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318815ef60 con 0x7f31881477e0 2024-03-06T07:29:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.632+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318815f520 con 0x7f31881477e0 2024-03-06T07:29:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.633+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318804f330 con 0x7f31881477e0 2024-03-06T07:29:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.634+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3180046020 con 0x7f31881477e0 2024-03-06T07:29:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.635+0000 7f317f400640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f316806b4d0 0x7f316806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.635+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31800ac9c0 con 0x7f31881477e0 2024-03-06T07:29:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.635+0000 7f3187400640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f316806b4d0 0x7f316806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.636+0000 7f3187400640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f316806b4d0 0x7f316806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3178006fd0 tx=0x7f3178008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.639+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31800aee20 con 0x7f31881477e0 2024-03-06T07:29:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.840+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31880724b0 con 0x7f31881477e0 2024-03-06T07:29:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.843+0000 7f317f400640 1 -- 172.21.15.32:0/1152448380 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f318004d030 con 0x7f31881477e0 2024-03-06T07:29:14.846 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:14.846 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f316806b4d0 msgr2=0x7f316806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f316806b4d0 0x7f316806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3178006fd0 tx=0x7f3178008040 comp rx=0 tx=0).stop 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 msgr2=0x7f318815d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 0x7f318815d680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3180036040 tx=0x7f3180004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 shutdown_connections 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f316806b4d0 0x7f316806d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31881477e0 0x7f318815d680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f318806ef20 0x7f3188143100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 --2- 172.21.15.32:0/1152448380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f318806b6c0 0x7f318813e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 >> 172.21.15.32:0/1152448380 conn(0x7f318806c0c0 msgr2=0x7f318806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 shutdown_connections 2024-03-06T07:29:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:14.846+0000 7f318cc00640 1 -- 172.21.15.32:0/1152448380 wait complete. 2024-03-06T07:29:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:14.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.084+0000 7f9c7cc00640 1 Processor -- start 2024-03-06T07:29:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.084+0000 7f9c7cc00640 1 -- start start 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.085+0000 7f9c7cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c7815e2d0 0x7f9c780b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c7cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78167f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c7cc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c780b5da0 con 0x7f9c7806cab0 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c7cc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c780b6080 con 0x7f9c7815e2d0 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c7cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780b6360 con 0x7f9c78165b60 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c7815e2d0 0x7f9c780b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c7815e2d0 0x7f9c780b5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51260/0 (socket says 172.21.15.32:51260) 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76a00640 1 -- 172.21.15.32:0/2732723902 learned_addr learned my addr 172.21.15.32:0/2732723902 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c77400640 1 --1- 172.21.15.32:0/2732723902 >> v1:172.21.15.32:6789/0 conn(0x7f9c7806cab0 0x7f9c7806ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56714/0 (socket says 172.21.15.32:56714) 2024-03-06T07:29:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c77e00640 1 --2- 172.21.15.32:0/2732723902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78167f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76a00640 1 -- 172.21.15.32:0/2732723902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 msgr2=0x7f9c78167f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76a00640 1 --2- 172.21.15.32:0/2732723902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78167f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76a00640 1 -- 172.21.15.32:0/2732723902 >> v1:172.21.15.32:6789/0 conn(0x7f9c7806cab0 legacy=0x7f9c7806ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.087+0000 7f9c76a00640 1 -- 172.21.15.32:0/2732723902 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780b6640 con 0x7f9c7815e2d0 2024-03-06T07:29:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.086+0000 7f9c76000640 1 -- 172.21.15.32:0/2732723902 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867338738 0 0) 0x7f9c780b5da0 con 0x7f9c7806cab0 2024-03-06T07:29:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.088+0000 7f9c76a00640 1 --2- 172.21.15.32:0/2732723902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7815e2d0 0x7f9c780b5450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009870 tx=0x7f9c6002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4962c27b40127a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.088+0000 7f9c76000640 1 -- 172.21.15.32:0/2732723902 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60034070 con 0x7f9c7815e2d0 2024-03-06T07:29:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.088+0000 7f9c76000640 1 -- 172.21.15.32:0/2732723902 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c60030840 con 0x7f9c7815e2d0 2024-03-06T07:29:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.088+0000 7f9c76000640 1 -- 172.21.15.32:0/2732723902 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60030c10 con 0x7f9c7815e2d0 2024-03-06T07:29:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.088+0000 7f9c7cc00640 1 -- 172.21.15.32:0/2732723902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7815e2d0 msgr2=0x7f9c780b5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.088+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/2732723902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7815e2d0 0x7f9c780b5450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009870 tx=0x7f9c6002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.089+0000 7f9c7cc00640 1 -- 172.21.15.32:0/2732723902 shutdown_connections 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.089+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/2732723902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7815e2d0 0x7f9c780b5450 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.089+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/2732723902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78167f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.089+0000 7f9c7cc00640 1 -- 172.21.15.32:0/2732723902 >> 172.21.15.32:0/2732723902 conn(0x7f9c7809aa10 msgr2=0x7f9c7809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.089+0000 7f9c7cc00640 1 -- 172.21.15.32:0/2732723902 shutdown_connections 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.089+0000 7f9c7cc00640 1 -- 172.21.15.32:0/2732723902 wait complete. 2024-03-06T07:29:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.090+0000 7f9c7cc00640 1 Processor -- start 2024-03-06T07:29:15.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.091+0000 7f9c7cc00640 1 -- start start 2024-03-06T07:29:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.091+0000 7f9c7cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 0x7f9c78159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c7cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c7815e2d0 0x7f9c781643d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c7cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78164910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c7cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7815e8b0 con 0x7f9c7806cab0 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c7cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7815e730 con 0x7f9c7815e2d0 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c7cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780b5990 con 0x7f9c78165b60 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 0x7f9c78159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 0x7f9c78159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51266/0 (socket says 172.21.15.32:51266) 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 -- 172.21.15.32:0/277282792 learned_addr learned my addr 172.21.15.32:0/277282792 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 -- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 msgr2=0x7f9c78164910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78164910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c76a00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c7815e2d0 0x7f9c781643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 -- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c7815e2d0 msgr2=0x7f9c781643d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c7815e2d0 0x7f9c781643d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.092+0000 7f9c77400640 1 -- 172.21.15.32:0/277282792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780af8a0 con 0x7f9c7806cab0 2024-03-06T07:29:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.093+0000 7f9c77400640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 0x7f9c78159280 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800c500 tx=0x7f9c6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.093+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68018070 con 0x7f9c7806cab0 2024-03-06T07:29:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.093+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c68007580 con 0x7f9c7806cab0 2024-03-06T07:29:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.093+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68004a20 con 0x7f9c7806cab0 2024-03-06T07:29:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.093+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c780afb80 con 0x7f9c7806cab0 2024-03-06T07:29:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.093+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c780b0140 con 0x7f9c7806cab0 2024-03-06T07:29:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.094+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7804f330 con 0x7f9c7806cab0 2024-03-06T07:29:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.095+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c68004bc0 con 0x7f9c7806cab0 2024-03-06T07:29:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.096+0000 7f9c6f400640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c5406b4d0 0x7f9c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.096+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c68089510 con 0x7f9c7806cab0 2024-03-06T07:29:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.096+0000 7f9c76a00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c5406b4d0 0x7f9c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.097+0000 7f9c76a00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c5406b4d0 0x7f9c5406d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9c60036000 tx=0x7f9c6003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.100+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c68058380 con 0x7f9c7806cab0 2024-03-06T07:29:15.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.298+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c7806c4b0 con 0x7f9c7806cab0 2024-03-06T07:29:15.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.299+0000 7f9c6f400640 1 -- 172.21.15.32:0/277282792 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c6805e080 con 0x7f9c7806cab0 2024-03-06T07:29:15.301 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c5406b4d0 msgr2=0x7f9c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c5406b4d0 0x7f9c5406d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9c60036000 tx=0x7f9c6003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 msgr2=0x7f9c78159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 0x7f9c78159280 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800c500 tx=0x7f9c6800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 shutdown_connections 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c5406b4d0 0x7f9c5406d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c78165b60 0x7f9c78164910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c7815e2d0 0x7f9c781643d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 --2- 172.21.15.32:0/277282792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c7806cab0 0x7f9c78159280 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.301+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 >> 172.21.15.32:0/277282792 conn(0x7f9c7809aa10 msgr2=0x7f9c7815d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.302+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 shutdown_connections 2024-03-06T07:29:15.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.302+0000 7f9c7cc00640 1 -- 172.21.15.32:0/277282792 wait complete. 2024-03-06T07:29:15.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.552+0000 7f0c0ce00640 1 Processor -- start 2024-03-06T07:29:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.552+0000 7f0c0ce00640 1 -- start start 2024-03-06T07:29:15.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.553+0000 7f0c0ce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c08147590 0x7f0c08071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.553+0000 7f0c0ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c080b2250 0x7f0c080b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.553+0000 7f0c0ce00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080719c0 con 0x7f0c0806d8d0 2024-03-06T07:29:15.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.553+0000 7f0c0ce00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c08147970 con 0x7f0c08147590 2024-03-06T07:29:15.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.553+0000 7f0c0ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080b31b0 con 0x7f0c080b2250 2024-03-06T07:29:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c08147590 0x7f0c08071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c08147590 0x7f0c08071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51268/0 (socket says 172.21.15.32:51268) 2024-03-06T07:29:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 -- 172.21.15.32:0/2773809754 learned_addr learned my addr 172.21.15.32:0/2773809754 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07e00640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c080b2250 0x7f0c080b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 -- 172.21.15.32:0/2773809754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c080b2250 msgr2=0x7f0c080b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c080b2250 0x7f0c080b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 -- 172.21.15.32:0/2773809754 >> v1:172.21.15.39:6789/0 conn(0x7f0c0806d8d0 legacy=0x7f0c080b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.554+0000 7f0c07400640 1 -- 172.21.15.32:0/2773809754 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0806de40 con 0x7f0c08147590 2024-03-06T07:29:15.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.555+0000 7f0c07e00640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c080b2250 0x7f0c080b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:15.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.556+0000 7f0c07400640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c08147590 0x7f0c08071480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009870 tx=0x7f0bf802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68aef3e5c98ec00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c06000640 1 -- 172.21.15.32:0/2773809754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8034070 con 0x7f0c08147590 2024-03-06T07:29:15.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c06000640 1 -- 172.21.15.32:0/2773809754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf8030840 con 0x7f0c08147590 2024-03-06T07:29:15.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c06000640 1 -- 172.21.15.32:0/2773809754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8030c10 con 0x7f0c08147590 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2773809754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c08147590 msgr2=0x7f0c08071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c08147590 0x7f0c08071480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009870 tx=0x7f0bf802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2773809754 shutdown_connections 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c080b2250 0x7f0c080b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2773809754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c08147590 0x7f0c08071480 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2773809754 >> 172.21.15.32:0/2773809754 conn(0x7f0c0806b3a0 msgr2=0x7f0c0806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2773809754 shutdown_connections 2024-03-06T07:29:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.557+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2773809754 wait complete. 2024-03-06T07:29:15.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.559+0000 7f0c0ce00640 1 Processor -- start 2024-03-06T07:29:15.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.559+0000 7f0c0ce00640 1 -- start start 2024-03-06T07:29:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c0ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c0806d8d0 0x7f0c08141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c0ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 0x7f0c081e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c0ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c08147590 0x7f0c081603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c0ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080b2b50 con 0x7f0c080b2250 2024-03-06T07:29:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c0ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080b2e40 con 0x7f0c08147590 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c0ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c080b3b80 con 0x7f0c0806d8d0 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 0x7f0c081e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 0x7f0c081e7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51280/0 (socket says 172.21.15.32:51280) 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.560+0000 7f0c06a00640 1 -- 172.21.15.32:0/2146658720 learned_addr learned my addr 172.21.15.32:0/2146658720 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c07e00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c08147590 0x7f0c081603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c07400640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c0806d8d0 0x7f0c08141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c06a00640 1 -- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c0806d8d0 msgr2=0x7f0c08141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c06a00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c0806d8d0 0x7f0c08141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c06a00640 1 -- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c08147590 msgr2=0x7f0c081603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c06a00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c08147590 0x7f0c081603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c06a00640 1 -- 172.21.15.32:0/2146658720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08160920 con 0x7f0c080b2250 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c07400640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c0806d8d0 0x7f0c08141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0c06a00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 0x7f0c081e7620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c460 tx=0x7f0bf400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.561+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40052d0 con 0x7f0c080b2250 2024-03-06T07:29:15.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.562+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf4012630 con 0x7f0c080b2250 2024-03-06T07:29:15.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.562+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40054b0 con 0x7f0c080b2250 2024-03-06T07:29:15.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.562+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c081e6580 con 0x7f0c080b2250 2024-03-06T07:29:15.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.562+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c081e6b40 con 0x7f0c080b2250 2024-03-06T07:29:15.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.563+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0804f330 con 0x7f0c080b2250 2024-03-06T07:29:15.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.564+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bf4012a20 con 0x7f0c080b2250 2024-03-06T07:29:15.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.565+0000 7f0bff400640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0be806b500 0x7f0be806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.565+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bf4089e90 con 0x7f0c080b2250 2024-03-06T07:29:15.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.565+0000 7f0c07400640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0be806b500 0x7f0be806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.566+0000 7f0c07400640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0be806b500 0x7f0be806d9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8036000 tx=0x7f0bf803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.569+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c08160920 con 0x7f0c080b2250 2024-03-06T07:29:15.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.770+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c08074ab0 con 0x7f0c080b2250 2024-03-06T07:29:15.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.770+0000 7f0bff400640 1 -- 172.21.15.32:0/2146658720 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bf4058c70 con 0x7f0c080b2250 2024-03-06T07:29:15.773 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0be806b500 msgr2=0x7f0be806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0be806b500 0x7f0be806d9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8036000 tx=0x7f0bf803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 msgr2=0x7f0c081e7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 0x7f0c081e7620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c460 tx=0x7f0bf400c920 comp rx=0 tx=0).stop 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 shutdown_connections 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0be806b500 0x7f0be806d9c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c08147590 0x7f0c081603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c080b2250 0x7f0c081e7620 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 --2- 172.21.15.32:0/2146658720 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c0806d8d0 0x7f0c08141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 >> 172.21.15.32:0/2146658720 conn(0x7f0c0806b3a0 msgr2=0x7f0c0806b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 shutdown_connections 2024-03-06T07:29:15.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:15.773+0000 7f0c0ce00640 1 -- 172.21.15.32:0/2146658720 wait complete. 2024-03-06T07:29:15.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:16.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.015+0000 7f6135600640 1 Processor -- start 2024-03-06T07:29:16.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.016+0000 7f6135600640 1 -- start start 2024-03-06T07:29:16.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.017+0000 7f6135600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f61300d4860 0x7f61300d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.017+0000 7f6135600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61300cfa60 0x7f61300d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.017+0000 7f6135600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61300ceca0 con 0x7f613015dee0 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.017+0000 7f6135600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61300cf5d0 con 0x7f61300d4860 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.017+0000 7f6135600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61300cef80 con 0x7f61300cfa60 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f61300d4860 0x7f61300d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f613015dee0 0x7f61300d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56726/0 (socket says 172.21.15.32:56726) 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f61300d4860 0x7f61300d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51290/0 (socket says 172.21.15.32:51290) 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612fe00640 1 -- 172.21.15.32:0/688374399 learned_addr learned my addr 172.21.15.32:0/688374399 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f6134c00640 1 --2- 172.21.15.32:0/688374399 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61300cfa60 0x7f61300d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612f400640 1 -- 172.21.15.32:0/688374399 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61300cfa60 msgr2=0x7f61300d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612f400640 1 --2- 172.21.15.32:0/688374399 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61300cfa60 0x7f61300d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612f400640 1 -- 172.21.15.32:0/688374399 >> v1:172.21.15.32:6789/0 conn(0x7f613015dee0 legacy=0x7f61300d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.018+0000 7f612f400640 1 -- 172.21.15.32:0/688374399 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300cf260 con 0x7f61300d4860 2024-03-06T07:29:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.019+0000 7f612f400640 1 --2- 172.21.15.32:0/688374399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300d4860 0x7f61300d0630 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f611c0097a0 tx=0x7f611c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573323d660207f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.020+0000 7f612ea00640 1 -- 172.21.15.32:0/688374399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c035020 con 0x7f61300d4860 2024-03-06T07:29:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.020+0000 7f612ea00640 1 -- 172.21.15.32:0/688374399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c031810 con 0x7f61300d4860 2024-03-06T07:29:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.020+0000 7f612ea00640 1 -- 172.21.15.32:0/688374399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c031be0 con 0x7f61300d4860 2024-03-06T07:29:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.020+0000 7f6135600640 1 -- 172.21.15.32:0/688374399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300d4860 msgr2=0x7f61300d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.020+0000 7f6135600640 1 --2- 172.21.15.32:0/688374399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300d4860 0x7f61300d0630 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f611c0097a0 tx=0x7f611c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.020+0000 7f6135600640 1 -- 172.21.15.32:0/688374399 shutdown_connections 2024-03-06T07:29:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.021+0000 7f6135600640 1 --2- 172.21.15.32:0/688374399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300d4860 0x7f61300d0630 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.021+0000 7f6135600640 1 --2- 172.21.15.32:0/688374399 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61300cfa60 0x7f61300d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.021+0000 7f6135600640 1 -- 172.21.15.32:0/688374399 >> 172.21.15.32:0/688374399 conn(0x7f61300d3cd0 msgr2=0x7f61300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.021+0000 7f6135600640 1 -- 172.21.15.32:0/688374399 shutdown_connections 2024-03-06T07:29:16.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.021+0000 7f6135600640 1 -- 172.21.15.32:0/688374399 wait complete. 2024-03-06T07:29:16.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.022+0000 7f6135600640 1 Processor -- start 2024-03-06T07:29:16.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.023+0000 7f6135600640 1 -- start start 2024-03-06T07:29:16.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6135600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 0x7f61300b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6135600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61300d4860 0x7f61300cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6135600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f613015dee0 0x7f613014a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6135600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61300b3510 con 0x7f61300cfa60 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6135600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613015e2c0 con 0x7f61300d4860 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6135600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61300b3690 con 0x7f613015dee0 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 0x7f61300b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 0x7f61300b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51298/0 (socket says 172.21.15.32:51298) 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 -- 172.21.15.32:0/602135939 learned_addr learned my addr 172.21.15.32:0/602135939 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612f400640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61300d4860 0x7f61300cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f6134c00640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f613015dee0 0x7f613014a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 -- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f613015dee0 msgr2=0x7f613014a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f613015dee0 0x7f613014a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 -- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61300d4860 msgr2=0x7f61300cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.024+0000 7f612fe00640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61300d4860 0x7f61300cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f612fe00640 1 -- 172.21.15.32:0/602135939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300b8c70 con 0x7f61300cfa60 2024-03-06T07:29:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f612fe00640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 0x7f61300b6b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f612000c3c0 tx=0x7f612000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120017070 con 0x7f61300cfa60 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6120005ce0 con 0x7f61300cfa60 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61200072f0 con 0x7f61300cfa60 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61300b8fb0 con 0x7f61300cfa60 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.025+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61300b97f0 con 0x7f61300cfa60 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.026+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613004f330 con 0x7f61300cfa60 2024-03-06T07:29:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.027+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f612001a070 con 0x7f61300cfa60 2024-03-06T07:29:16.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.028+0000 7f6127e00640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f611006b4d0 0x7f611006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.028+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6120089680 con 0x7f61300cfa60 2024-03-06T07:29:16.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.029+0000 7f612f400640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f611006b4d0 0x7f611006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.029+0000 7f612f400640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f611006b4d0 0x7f611006d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f611c036000 tx=0x7f611c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.033+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61300b8fb0 con 0x7f61300cfa60 2024-03-06T07:29:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.233+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f613008d7d0 con 0x7f61300cfa60 2024-03-06T07:29:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.234+0000 7f6127e00640 1 -- 172.21.15.32:0/602135939 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6120058460 con 0x7f61300cfa60 2024-03-06T07:29:16.237 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f611006b4d0 msgr2=0x7f611006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f611006b4d0 0x7f611006d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f611c036000 tx=0x7f611c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 msgr2=0x7f61300b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 0x7f61300b6b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f612000c3c0 tx=0x7f612000c880 comp rx=0 tx=0).stop 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 shutdown_connections 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f611006b4d0 0x7f611006d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f613015dee0 0x7f613014a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61300d4860 0x7f61300cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.237+0000 7f6135600640 1 --2- 172.21.15.32:0/602135939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61300cfa60 0x7f61300b6b60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.238+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 >> 172.21.15.32:0/602135939 conn(0x7f61300d3cd0 msgr2=0x7f61300b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.238+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 shutdown_connections 2024-03-06T07:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.238+0000 7f6135600640 1 -- 172.21.15.32:0/602135939 wait complete. 2024-03-06T07:29:16.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:16.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.478+0000 7efd02c00640 1 Processor -- start 2024-03-06T07:29:16.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.478+0000 7efd02c00640 1 -- start start 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.479+0000 7efd02c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcfc0b34e0 0x7efcfc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.479+0000 7efd02c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc15cc40 con 0x7efcfc0af1a0 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.479+0000 7efd02c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc15cf20 con 0x7efcfc0af580 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.479+0000 7efd02c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcfc15d200 con 0x7efcfc0b34e0 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efcfbe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efcfc0af1a0 0x7efcfc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56730/0 (socket says 172.21.15.32:56730) 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efcfbe00640 1 -- 172.21.15.32:0/276427079 learned_addr learned my addr 172.21.15.32:0/276427079 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efd00c00640 1 --2- 172.21.15.32:0/276427079 >> v2:172.21.15.32:3300/0 conn(0x7efcfc0b34e0 0x7efcfc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efcfaa00640 1 -- 172.21.15.32:0/276427079 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867342111 0 0) 0x7efcfc15cc40 con 0x7efcfc0af1a0 2024-03-06T07:29:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efd00c00640 1 -- 172.21.15.32:0/276427079 >> v1:172.21.15.39:6789/0 conn(0x7efcfc0af580 legacy=0x7efcfc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efd00c00640 1 -- 172.21.15.32:0/276427079 >> v1:172.21.15.32:6789/0 conn(0x7efcfc0af1a0 legacy=0x7efcfc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.480+0000 7efd00c00640 1 -- 172.21.15.32:0/276427079 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc15d4e0 con 0x7efcfc0b34e0 2024-03-06T07:29:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.481+0000 7efd00c00640 1 --2- 172.21.15.32:0/276427079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0b34e0 0x7efcfc15c600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcf4009d90 tx=0x7efcf402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac1f8075dffe28b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efcfaa00640 1 -- 172.21.15.32:0/276427079 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4039070 con 0x7efcfc0b34e0 2024-03-06T07:29:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efcfaa00640 1 -- 172.21.15.32:0/276427079 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf402cbd0 con 0x7efcfc0b34e0 2024-03-06T07:29:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efcfaa00640 1 -- 172.21.15.32:0/276427079 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4034440 con 0x7efcfc0b34e0 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 -- 172.21.15.32:0/276427079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0b34e0 msgr2=0x7efcfc15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 --2- 172.21.15.32:0/276427079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0b34e0 0x7efcfc15c600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcf4009d90 tx=0x7efcf402be20 comp rx=0 tx=0).stop 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 -- 172.21.15.32:0/276427079 shutdown_connections 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 --2- 172.21.15.32:0/276427079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0b34e0 0x7efcfc15c600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 -- 172.21.15.32:0/276427079 >> 172.21.15.32:0/276427079 conn(0x7efcfc06afb0 msgr2=0x7efcfc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 -- 172.21.15.32:0/276427079 shutdown_connections 2024-03-06T07:29:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.482+0000 7efd02c00640 1 -- 172.21.15.32:0/276427079 wait complete. 2024-03-06T07:29:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.484+0000 7efd02c00640 1 Processor -- start 2024-03-06T07:29:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.484+0000 7efd02c00640 1 -- start start 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efd02c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 0x7efcfc1506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efd02c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcfc0af580 0x7efcfc09cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efd02c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcfc0b34e0 0x7efcfc15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efd02c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc155e70 con 0x7efcfc0af1a0 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efd02c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc0cde30 con 0x7efcfc0af580 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efd02c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc155ff0 con 0x7efcfc0b34e0 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efcfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 0x7efcfc1506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efcfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 0x7efcfc1506e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51308/0 (socket says 172.21.15.32:51308) 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.485+0000 7efcfbe00640 1 -- 172.21.15.32:0/3633495572 learned_addr learned my addr 172.21.15.32:0/3633495572 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfb400640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcfc0af580 0x7efcfc09cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efd00c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcfc0b34e0 0x7efcfc15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfbe00640 1 -- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcfc0b34e0 msgr2=0x7efcfc15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfbe00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcfc0b34e0 0x7efcfc15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfbe00640 1 -- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcfc0af580 msgr2=0x7efcfc09cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfbe00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcfc0af580 0x7efcfc09cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfbe00640 1 -- 172.21.15.32:0/3633495572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc09d4c0 con 0x7efcfc0af1a0 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcfbe00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 0x7efcfc1506e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c4d0 tx=0x7efcec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec018070 con 0x7efcfc0af1a0 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec007500 con 0x7efcfc0af1a0 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.486+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec004a20 con 0x7efcfc0af1a0 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.487+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc06cbb0 con 0x7efcfc0af1a0 2024-03-06T07:29:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.487+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc06d170 con 0x7efcfc0af1a0 2024-03-06T07:29:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.488+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc069510 con 0x7efcfc0af1a0 2024-03-06T07:29:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.489+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcec01b070 con 0x7efcfc0af1a0 2024-03-06T07:29:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.489+0000 7efcf3e00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.490+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcec0895e0 con 0x7efcfc0af1a0 2024-03-06T07:29:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.490+0000 7efcfb400640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.491+0000 7efcfb400640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcdc06b4d0 0x7efcdc06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efcfc15b7d0 tx=0x7efce8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.494+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcec0583c0 con 0x7efcfc0af1a0 2024-03-06T07:29:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.732+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcfc04f3a0 con 0x7efcfc0af1a0 2024-03-06T07:29:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.734+0000 7efcf3e00640 1 -- 172.21.15.32:0/3633495572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7efcec057cc0 con 0x7efcfc0af1a0 2024-03-06T07:29:16.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:16.736 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:16.737 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:16.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.736+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcdc06b4d0 msgr2=0x7efcdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.736+0000 7efd02c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcdc06b4d0 0x7efcdc06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efcfc15b7d0 tx=0x7efce8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:16.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.736+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 msgr2=0x7efcfc1506e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.736+0000 7efd02c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 0x7efcfc1506e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c4d0 tx=0x7efcec00c990 comp rx=0 tx=0).stop 2024-03-06T07:29:16.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 shutdown_connections 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcfc0b34e0 0x7efcfc15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcfc0af580 0x7efcfc09cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 --2- 172.21.15.32:0/3633495572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcfc0af1a0 0x7efcfc1506e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 >> 172.21.15.32:0/3633495572 conn(0x7efcfc06afb0 msgr2=0x7efcfc0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 shutdown_connections 2024-03-06T07:29:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.737+0000 7efd02c00640 1 -- 172.21.15.32:0/3633495572 wait complete. 2024-03-06T07:29:16.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:16.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.966+0000 7ff1dd200640 1 Processor -- start 2024-03-06T07:29:16.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.966+0000 7ff1dd200640 1 -- start start 2024-03-06T07:29:16.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.968+0000 7ff1dd200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 0x7ff1d815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.968+0000 7ff1dd200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d815cc40 con 0x7ff1d80af1a0 2024-03-06T07:29:16.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.968+0000 7ff1dd200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d815cf20 con 0x7ff1d80af580 2024-03-06T07:29:16.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.968+0000 7ff1dd200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d815d200 con 0x7ff1d80b34e0 2024-03-06T07:29:16.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff1d80af1a0 0x7ff1d80b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56742/0 (socket says 172.21.15.32:56742) 2024-03-06T07:29:16.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5e00640 1 -- 172.21.15.32:0/1359444066 learned_addr learned my addr 172.21.15.32:0/1359444066 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d6800640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 0x7ff1d815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290028671 0 0) 0x7ff1d815cc40 con 0x7ff1d80af1a0 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1bc003520 con 0x7ff1d80af1a0 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411945153 0 0) 0x7ff1bc003520 con 0x7ff1d80af1a0 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 msgr2=0x7ff1d815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5400640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 0x7ff1d815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.969+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 >> v1:172.21.15.39:6789/0 conn(0x7ff1d80af580 legacy=0x7ff1d80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d815d4e0 con 0x7ff1d80af1a0 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff1c8001670 con 0x7ff1d80af1a0 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 >> v1:172.21.15.32:6789/0 conn(0x7ff1d80af1a0 legacy=0x7ff1d80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 0x7ff1bc003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc004d40 0x7ff1bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d815d200 con 0x7ff1bc003fd0 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00bbb0 con 0x7ff1bc004d40 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00bdb0 con 0x7ff1d80b34e0 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1cfe00640 1 -- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 msgr2=0x7ff1bc003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d80b34e0 0x7ff1bc003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1cfe00640 1 -- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc004d40 msgr2=0x7ff1bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1d6800640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc004d40 0x7ff1bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc004d40 0x7ff1bc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.970+0000 7ff1cfe00640 1 -- 172.21.15.32:0/1359444066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc003520 con 0x7ff1bc003fd0 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.971+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1bc004800 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0001720 tx=0x7ff1bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a178ef118f4dc261 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0001370 con 0x7ff1bc003fd0 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c00052d0 con 0x7ff1bc003fd0 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1d5400640 1 -- 172.21.15.32:0/1359444066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0003830 con 0x7ff1bc003fd0 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1dd200640 1 -- 172.21.15.32:0/1359444066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 msgr2=0x7ff1bc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1dd200640 1 --2- 172.21.15.32:0/1359444066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1bc004800 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0001720 tx=0x7ff1bc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1d6800640 1 -- 172.21.15.32:0/1359444066 reap_dead start 2024-03-06T07:29:16.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1dd200640 1 -- 172.21.15.32:0/1359444066 shutdown_connections 2024-03-06T07:29:16.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.972+0000 7ff1dd200640 1 -- 172.21.15.32:0/1359444066 >> 172.21.15.32:0/1359444066 conn(0x7ff1d806afb0 msgr2=0x7ff1d806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.973+0000 7ff1dd200640 1 -- 172.21.15.32:0/1359444066 shutdown_connections 2024-03-06T07:29:16.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.973+0000 7ff1dd200640 1 -- 172.21.15.32:0/1359444066 wait complete. 2024-03-06T07:29:16.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.974+0000 7ff1dd200640 1 Processor -- start 2024-03-06T07:29:16.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.975+0000 7ff1dd200640 1 -- start start 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1dd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1d809db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1dd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1d8151d90 0x7ff1d80ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1dd200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d81f3100 0x7ff1d809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1dd200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d815c600 con 0x7ff1bc003fd0 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1dd200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80cde30 con 0x7ff1d8151d90 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1dd200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d815c780 con 0x7ff1d81f3100 2024-03-06T07:29:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1d809db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1d809db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51332/0 (socket says 172.21.15.32:51332) 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 -- 172.21.15.32:0/2992845251 learned_addr learned my addr 172.21.15.32:0/2992845251 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 -- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d81f3100 msgr2=0x7ff1d809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d81f3100 0x7ff1d809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 -- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1d8151d90 msgr2=0x7ff1d80ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1d8151d90 0x7ff1d80ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1d8151d90 0x7ff1d80ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.976+0000 7ff1d5e00640 1 -- 172.21.15.32:0/2992845251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d814ef50 con 0x7ff1bc003fd0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.977+0000 7ff1d5e00640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1d809db10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c680 tx=0x7ff1c800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.977+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8018070 con 0x7ff1bc003fd0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.977+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c80072f0 con 0x7ff1bc003fd0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.977+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d809d030 con 0x7ff1bc003fd0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.977+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80047e0 con 0x7ff1bc003fd0 2024-03-06T07:29:16.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.977+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d809d5f0 con 0x7ff1bc003fd0 2024-03-06T07:29:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.978+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d8069510 con 0x7ff1bc003fd0 2024-03-06T07:29:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.979+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1c801b070 con 0x7ff1bc003fd0 2024-03-06T07:29:16.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.980+0000 7ff1ce000640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff1b406b4d0 0x7ff1b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.980+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1c8089650 con 0x7ff1bc003fd0 2024-03-06T07:29:16.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.980+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff1b406b4d0 0x7ff1b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.981+0000 7ff1cfe00640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff1b406b4d0 0x7ff1b406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0033000 tx=0x7ff1c0009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:16.986+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1c8058430 con 0x7ff1bc003fd0 2024-03-06T07:29:17.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.186+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1d815c2e0 con 0x7ff1bc003fd0 2024-03-06T07:29:17.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.189+0000 7ff1ce000640 1 -- 172.21.15.32:0/2992845251 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7ff1c8057d30 con 0x7ff1bc003fd0 2024-03-06T07:29:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:17.192 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:17.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff1b406b4d0 msgr2=0x7ff1b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff1b406b4d0 0x7ff1b406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0033000 tx=0x7ff1c0009030 comp rx=0 tx=0).stop 2024-03-06T07:29:17.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 msgr2=0x7ff1d809db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1d809db10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c680 tx=0x7ff1c800cb40 comp rx=0 tx=0).stop 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 shutdown_connections 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff1b406b4d0 0x7ff1b406d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff1d81f3100 0x7ff1d809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1d8151d90 0x7ff1d80ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 --2- 172.21.15.32:0/2992845251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1bc003fd0 0x7ff1d809db10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 >> 172.21.15.32:0/2992845251 conn(0x7ff1d806afb0 msgr2=0x7ff1d80ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 shutdown_connections 2024-03-06T07:29:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.192+0000 7ff1dd200640 1 -- 172.21.15.32:0/2992845251 wait complete. 2024-03-06T07:29:17.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.438+0000 7f42a1200640 1 Processor -- start 2024-03-06T07:29:17.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.438+0000 7f42a1200640 1 -- start start 2024-03-06T07:29:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f42a1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f429c145300 0x7f429c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f42a1200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f429c070f10 0x7f429c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f42a1200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429c0714d0 con 0x7f429c0cce90 2024-03-06T07:29:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f42a1200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429c0cd300 con 0x7f429c145300 2024-03-06T07:29:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f42a1200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429c0717b0 con 0x7f429c070f10 2024-03-06T07:29:17.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f429b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f429c145300 0x7f429c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f429be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f429c0cce90 0x7f429c0cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56750/0 (socket says 172.21.15.32:56750) 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.440+0000 7f429be00640 1 -- 172.21.15.32:0/3760759882 learned_addr learned my addr 172.21.15.32:0/3760759882 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.441+0000 7f429aa00640 1 --2- 172.21.15.32:0/3760759882 >> v2:172.21.15.39:3300/0 conn(0x7f429c070f10 0x7f429c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.441+0000 7f429a000640 1 -- 172.21.15.32:0/3760759882 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444531283 0 0) 0x7f429c0714d0 con 0x7f429c0cce90 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.441+0000 7f429b400640 1 -- 172.21.15.32:0/3760759882 >> v2:172.21.15.39:3300/0 conn(0x7f429c070f10 msgr2=0x7f429c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.441+0000 7f429b400640 1 --2- 172.21.15.32:0/3760759882 >> v2:172.21.15.39:3300/0 conn(0x7f429c070f10 0x7f429c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.441+0000 7f429b400640 1 -- 172.21.15.32:0/3760759882 >> v1:172.21.15.32:6789/0 conn(0x7f429c0cce90 legacy=0x7f429c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.441+0000 7f429b400640 1 -- 172.21.15.32:0/3760759882 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c071a90 con 0x7f429c145300 2024-03-06T07:29:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.442+0000 7f429b400640 1 --2- 172.21.15.32:0/3760759882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c0b49a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f429c0b5150 tx=0x7f428c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256809420cd01cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.442+0000 7f429a000640 1 -- 172.21.15.32:0/3760759882 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c0315a0 con 0x7f429c145300 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.442+0000 7f429a000640 1 -- 172.21.15.32:0/3760759882 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c037070 con 0x7f429c145300 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.442+0000 7f429a000640 1 -- 172.21.15.32:0/3760759882 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c031960 con 0x7f429c145300 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 -- 172.21.15.32:0/3760759882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 msgr2=0x7f429c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 --2- 172.21.15.32:0/3760759882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c0b49a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f429c0b5150 tx=0x7f428c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 -- 172.21.15.32:0/3760759882 shutdown_connections 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 --2- 172.21.15.32:0/3760759882 >> v2:172.21.15.39:3300/0 conn(0x7f429c070f10 0x7f429c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 --2- 172.21.15.32:0/3760759882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c0b49a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 -- 172.21.15.32:0/3760759882 >> 172.21.15.32:0/3760759882 conn(0x7f429c06b3d0 msgr2=0x7f429c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 -- 172.21.15.32:0/3760759882 shutdown_connections 2024-03-06T07:29:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.443+0000 7f42a1200640 1 -- 172.21.15.32:0/3760759882 wait complete. 2024-03-06T07:29:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.445+0000 7f42a1200640 1 Processor -- start 2024-03-06T07:29:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.445+0000 7f42a1200640 1 -- start start 2024-03-06T07:29:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f42a1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f429c070f10 0x7f429c1f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f42a1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429c0cce90 0x7f429c13e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f42a1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f42a1200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c1456e0 con 0x7f429c145300 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f42a1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0b4ee0 con 0x7f429c0cce90 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f42a1200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c15d580 con 0x7f429c070f10 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f429be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f429be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c15b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51350/0 (socket says 172.21.15.32:51350) 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.446+0000 7f429be00640 1 -- 172.21.15.32:0/2678083886 learned_addr learned my addr 172.21.15.32:0/2678083886 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429b400640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f429c070f10 0x7f429c1f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429aa00640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429c0cce90 0x7f429c13e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429be00640 1 -- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f429c070f10 msgr2=0x7f429c1f8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429be00640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f429c070f10 0x7f429c1f8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429be00640 1 -- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429c0cce90 msgr2=0x7f429c13e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429be00640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429c0cce90 0x7f429c13e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429be00640 1 -- 172.21.15.32:0/2678083886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c06c940 con 0x7f429c145300 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f429be00640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c15b630 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f429400c810 tx=0x7f429400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294018070 con 0x7f429c145300 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429400a640 con 0x7f429c145300 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429400aa30 con 0x7f429c145300 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.447+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429c06cc20 con 0x7f429c145300 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.448+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429c15bb70 con 0x7f429c145300 2024-03-06T07:29:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.449+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c04f330 con 0x7f429c145300 2024-03-06T07:29:17.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.451+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f429401b070 con 0x7f429c145300 2024-03-06T07:29:17.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.452+0000 7f4293400640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f427c06b4d0 0x7f427c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.452+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4294089890 con 0x7f429c145300 2024-03-06T07:29:17.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.452+0000 7f429b400640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f427c06b4d0 0x7f427c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.453+0000 7f429b400640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f427c06b4d0 0x7f427c06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f428c036000 tx=0x7f428c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.457+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4294058700 con 0x7f429c145300 2024-03-06T07:29:17.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.700+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f429c15be10 con 0x7f429c145300 2024-03-06T07:29:17.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.701+0000 7f4293400640 1 -- 172.21.15.32:0/2678083886 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f4294058000 con 0x7f429c145300 2024-03-06T07:29:17.703 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:17.703 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:17.703 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.703+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f427c06b4d0 msgr2=0x7f427c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.703+0000 7f42a1200640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f427c06b4d0 0x7f427c06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f428c036000 tx=0x7f428c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.703+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 msgr2=0x7f429c15b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.703+0000 7f42a1200640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c15b630 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f429400c810 tx=0x7f429400ccd0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 shutdown_connections 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f427c06b4d0 0x7f427c06d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f429c145300 0x7f429c15b630 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429c0cce90 0x7f429c13e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 --2- 172.21.15.32:0/2678083886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f429c070f10 0x7f429c1f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 >> 172.21.15.32:0/2678083886 conn(0x7f429c06b3d0 msgr2=0x7f429c0ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 shutdown_connections 2024-03-06T07:29:17.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:17.704+0000 7f42a1200640 1 -- 172.21.15.32:0/2678083886 wait complete. 2024-03-06T07:29:18.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:00.778364+0000', 'modified': '2024-03-06T07:29:03.836328+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4258}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4258': {'gid': 4258, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6833/3857082618', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 3857082618}, {'type': 'v1', 'addr': '172.21.15.39:6833', 'nonce': 3857082618}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258]} 2024-03-06T07:29:18.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:18.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:29:18.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:18.348 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.355+0000 7f4c63a00640 1 Processor -- start 2024-03-06T07:29:18.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.355+0000 7f4c63a00640 1 -- start start 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c63a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c5c0b3c90 0x7f4c5c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c63a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c5c0cf5c0 0x7f4c5c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c63a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c06cb10 0x7f4c5c06cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c63a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5c0ce160 con 0x7f4c5c0cf5c0 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c63a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5c0b45b0 con 0x7f4c5c0b3c90 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c63a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0b4860 con 0x7f4c5c06cb10 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c5be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c5c0cf5c0 0x7f4c5c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c5be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c5c0cf5c0 0x7f4c5c0b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37952/0 (socket says 172.21.15.32:37952) 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c5be00640 1 -- 172.21.15.32:0/109629652 learned_addr learned my addr 172.21.15.32:0/109629652 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c61000640 1 --2- 172.21.15.32:0/109629652 >> v2:172.21.15.39:3300/0 conn(0x7f4c5c0b3c90 0x7f4c5c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.357+0000 7f4c61a00640 1 --2- 172.21.15.32:0/109629652 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c06cb10 0x7f4c5c06cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.358+0000 7f4c5be00640 1 -- 172.21.15.32:0/109629652 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c06cb10 msgr2=0x7f4c5c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.358+0000 7f4c5be00640 1 --2- 172.21.15.32:0/109629652 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c06cb10 0x7f4c5c06cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.358+0000 7f4c5be00640 1 -- 172.21.15.32:0/109629652 >> v2:172.21.15.39:3300/0 conn(0x7f4c5c0b3c90 msgr2=0x7f4c5c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.358+0000 7f4c5be00640 1 --2- 172.21.15.32:0/109629652 >> v2:172.21.15.39:3300/0 conn(0x7f4c5c0b3c90 0x7f4c5c0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.358+0000 7f4c5be00640 1 -- 172.21.15.32:0/109629652 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0b4b40 con 0x7f4c5c0cf5c0 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.359+0000 7f4c5be00640 1 --2- 172.21.15.32:0/109629652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c0b4070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c070800 tx=0x7f4c4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf203619fb29e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.359+0000 7f4c5b400640 1 -- 172.21.15.32:0/109629652 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48034070 con 0x7f4c5c0cf5c0 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.359+0000 7f4c5b400640 1 -- 172.21.15.32:0/109629652 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48030840 con 0x7f4c5c0cf5c0 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.359+0000 7f4c5b400640 1 -- 172.21.15.32:0/109629652 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48030c10 con 0x7f4c5c0cf5c0 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 -- 172.21.15.32:0/109629652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c0cf5c0 msgr2=0x7f4c5c0b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 --2- 172.21.15.32:0/109629652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c0b4070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c070800 tx=0x7f4c4802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 -- 172.21.15.32:0/109629652 shutdown_connections 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 --2- 172.21.15.32:0/109629652 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c06cb10 0x7f4c5c06cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 --2- 172.21.15.32:0/109629652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c0b4070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 --2- 172.21.15.32:0/109629652 >> v2:172.21.15.39:3300/0 conn(0x7f4c5c0b3c90 0x7f4c5c0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.360+0000 7f4c63a00640 1 -- 172.21.15.32:0/109629652 >> 172.21.15.32:0/109629652 conn(0x7f4c5c06afb0 msgr2=0x7f4c5c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.361+0000 7f4c63a00640 1 -- 172.21.15.32:0/109629652 shutdown_connections 2024-03-06T07:29:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.361+0000 7f4c63a00640 1 -- 172.21.15.32:0/109629652 wait complete. 2024-03-06T07:29:18.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.362+0000 7f4c63a00640 1 Processor -- start 2024-03-06T07:29:18.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.362+0000 7f4c63a00640 1 -- start start 2024-03-06T07:29:18.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.363+0000 7f4c63a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 0x7f4c5c14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.363+0000 7f4c63a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c0b3c90 0x7f4c5c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.363+0000 7f4c63a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.363+0000 7f4c63a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c15b340 con 0x7f4c5c06cb10 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.363+0000 7f4c63a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0b5100 con 0x7f4c5c0cf5c0 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.363+0000 7f4c63a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c06d4c0 con 0x7f4c5c0b3c90 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 0x7f4c5c14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 0x7f4c5c14d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37964/0 (socket says 172.21.15.32:37964) 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 -- 172.21.15.32:0/2121622447 learned_addr learned my addr 172.21.15.32:0/2121622447 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 -- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c0b3c90 msgr2=0x7f4c5c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c0b3c90 0x7f4c5c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 -- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5c0cf5c0 msgr2=0x7f4c5c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 -- 172.21.15.32:0/2121622447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c156c30 con 0x7f4c5c06cb10 2024-03-06T07:29:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.364+0000 7f4c61000640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 0x7f4c5c14d9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000d800 tx=0x7f4c5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.365+0000 7f4c58c00640 1 -- 172.21.15.32:0/2121622447 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000b6c0 con 0x7f4c5c06cb10 2024-03-06T07:29:18.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.365+0000 7f4c58c00640 1 -- 172.21.15.32:0/2121622447 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50019070 con 0x7f4c5c06cb10 2024-03-06T07:29:18.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.365+0000 7f4c58c00640 1 -- 172.21.15.32:0/2121622447 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000ba80 con 0x7f4c5c06cb10 2024-03-06T07:29:18.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.365+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c156670 con 0x7f4c5c06cb10 2024-03-06T07:29:18.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.365+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c150740 con 0x7f4c5c06cb10 2024-03-06T07:29:18.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.367+0000 7f4c58c00640 1 -- 172.21.15.32:0/2121622447 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c5000bc20 con 0x7f4c5c06cb10 2024-03-06T07:29:18.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.368+0000 7f4c58c00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3c06b4d0 0x7f4c3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.368+0000 7f4c58c00640 1 -- 172.21.15.32:0/2121622447 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c5008a6d0 con 0x7f4c5c06cb10 2024-03-06T07:29:18.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.368+0000 7f4c5be00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3c06b4d0 0x7f4c3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.369+0000 7f4c5be00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3c06b4d0 0x7f4c3c06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c48036000 tx=0x7f4c48030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.370+0000 7f4c6676e740 1 Processor -- start 2024-03-06T07:29:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.370+0000 7f4c6676e740 1 -- start start 2024-03-06T07:29:18.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b42f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b45f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b42fc90 0x55575b45fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c6676e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55575b399600 con 0x55575b45cb90 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55575b342930 con 0x55575b45c7b0 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55575b3956a0 con 0x55575b42fc90 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b45f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b45f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37972/0 (socket says 172.21.15.32:37972) 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 -- 172.21.15.32:0/1784897441 learned_addr learned my addr 172.21.15.32:0/1784897441 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c61000640 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b42f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c61a00640 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b42fc90 0x55575b45fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 -- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b42fc90 msgr2=0x55575b45fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b42fc90 0x55575b45fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 -- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 msgr2=0x55575b42f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b42f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.371+0000 7f4c5be00640 1 -- 172.21.15.32:0/1784897441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55575b0ae2d0 con 0x55575b45cb90 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.372+0000 7f4c5be00640 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b45f540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4c4802bf90 tx=0x7f4c48031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506e57b036c5c1f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.372+0000 7f4c4d600640 1 -- 172.21.15.32:0/1784897441 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48034040 con 0x55575b45cb90 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.372+0000 7f4c4d600640 1 -- 172.21.15.32:0/1784897441 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48004230 con 0x55575b45cb90 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.372+0000 7f4c4d600640 1 -- 172.21.15.32:0/1784897441 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48003610 con 0x55575b45cb90 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 -- 172.21.15.32:0/1784897441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 msgr2=0x55575b45f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b45f540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4c4802bf90 tx=0x7f4c48031000 comp rx=0 tx=0).stop 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 -- 172.21.15.32:0/1784897441 shutdown_connections 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b42fc90 0x55575b45fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b45f540 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1784897441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b42f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 -- 172.21.15.32:0/1784897441 >> 172.21.15.32:0/1784897441 conn(0x55575b4319c0 msgr2=0x55575b47a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 -- 172.21.15.32:0/1784897441 shutdown_connections 2024-03-06T07:29:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.373+0000 7f4c6676e740 1 -- 172.21.15.32:0/1784897441 wait complete. 2024-03-06T07:29:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.374+0000 7f4c6676e740 1 -- start start 2024-03-06T07:29:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.374+0000 7f4c6676e740 1 Processor -- start 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b461580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b45cb90 0x55575b474f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b4742b0 0x55575b473260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55575b433f60 con 0x55575b45c7b0 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55575b355d90 con 0x55575b4742b0 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55575b433350 con 0x55575b45cb90 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.375+0000 7f4c6676e740 10 client.0 _ll_register_callbacks cb 0x55575b391350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b461580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b461580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37984/0 (socket says 172.21.15.32:37984) 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 -- 172.21.15.32:0/1460797333 learned_addr learned my addr 172.21.15.32:0/1460797333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61a00640 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b4742b0 0x55575b473260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 -- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b45cb90 msgr2=0x55575b474f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b45cb90 0x55575b474f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 -- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b4742b0 msgr2=0x55575b473260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b4742b0 0x55575b473260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 -- 172.21.15.32:0/1460797333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55575b1786b0 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c61000640 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b461580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000d7d0 tx=0x7f4c50059600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c33b7195da6c184 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.376+0000 7f4c42000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c42000640 1 -- 172.21.15.32:0/1460797333 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50086020 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c42000640 1 -- 172.21.15.32:0/1460797333 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c50057140 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c48004230 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c42000640 1 -- 172.21.15.32:0/1460797333 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c50075050 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 10 client.4647 fetch_fsmap learned FSMap version 8 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55575b461b80 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c42000640 1 -- 172.21.15.32:0/1460797333 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48004230 con 0x55575b45c7b0 2024-03-06T07:29:18.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c42000640 1 -- 172.21.15.32:0/1460797333 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4c50013070 con 0x55575b45c7b0 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 10 client.4647 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 10 client.4647 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 20 client.4647 populate_metadata read hostname 'smithi032' 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b45b700 0x55575b48e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 4 client.4647 mds_command: new command op to 4258 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.377+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 --> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55575b3a6b40 con 0x55575b45b700 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.378+0000 7f4c5be00640 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b45b700 0x55575b48e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.379+0000 7f4c5be00640 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b45b700 0x55575b48e3f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd7819fe3a089533 server_cookie=3da4c96424b16715 in_seq=0 out_seq=0 2024-03-06T07:29:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.379+0000 7f4c42000640 10 client.4647 ms_handle_connect on v2:172.21.15.39:6832/3857082618 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c42000640 1 -- 172.21.15.32:0/1460797333 <== mds.0 v2:172.21.15.39:6832/3857082618 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55575b3a6b40 con 0x55575b45b700 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c42000640 10 client.4647 handle_command_reply: tid=0 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 client.4647 shutdown 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 msgr2=0x55575b461580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b461580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000d7d0 tx=0x7f4c50059600 comp rx=0 tx=0).stop 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 shutdown_connections 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b45b700 0x55575b48e3f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b4742b0 0x55575b473260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b45cb90 0x55575b474f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 --2- 172.21.15.32:0/1460797333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b461580 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 >> 172.21.15.32:0/1460797333 conn(0x55575b42fc90 msgr2=0x55575b42eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c42000640 0 client.4647 ms_handle_reset on v2:172.21.15.39:6832/3857082618 2024-03-06T07:29:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 shutdown_connections 2024-03-06T07:29:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.381+0000 7f4c6676e740 1 -- 172.21.15.32:0/1460797333 wait complete. 2024-03-06T07:29:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.382+0000 7f4c6676e740 20 client.4647 trim_cache size 0 max 16384 2024-03-06T07:29:18.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.400+0000 7f4c6676e740 1 Processor -- start 2024-03-06T07:29:18.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.400+0000 7f4c6676e740 1 -- start start 2024-03-06T07:29:18.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b47a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45cb90 0x55575b4716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b471c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c6676e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c071980 con 0x55575b45c7b0 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c156100 con 0x55575b45cb90 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c071380 con 0x55575b4742b0 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.401+0000 7f4c61000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b47a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b47a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37996/0 (socket says 172.21.15.32:37996) 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 -- 172.21.15.32:0/4191672474 learned_addr learned my addr 172.21.15.32:0/4191672474 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61a00640 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b471c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c5be00640 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45cb90 0x55575b4716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 -- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 msgr2=0x55575b471c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b471c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 -- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45cb90 msgr2=0x55575b4716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45cb90 0x55575b4716c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 -- 172.21.15.32:0/4191672474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55575b494c20 con 0x55575b45c7b0 2024-03-06T07:29:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c61000640 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b47a520 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c50007170 tx=0x7f4c5006e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82edca4d8743f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.402+0000 7f4c4cc00640 1 -- 172.21.15.32:0/4191672474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50086070 con 0x55575b45c7b0 2024-03-06T07:29:18.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c4cc00640 1 -- 172.21.15.32:0/4191672474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5007c070 con 0x55575b45c7b0 2024-03-06T07:29:18.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c4cc00640 1 -- 172.21.15.32:0/4191672474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5005b040 con 0x55575b45c7b0 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 -- 172.21.15.32:0/4191672474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 msgr2=0x55575b47a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b47a520 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c50007170 tx=0x7f4c5006e000 comp rx=0 tx=0).stop 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 -- 172.21.15.32:0/4191672474 shutdown_connections 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b471c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45cb90 0x55575b4716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 --2- 172.21.15.32:0/4191672474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45c7b0 0x55575b47a520 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 -- 172.21.15.32:0/4191672474 >> 172.21.15.32:0/4191672474 conn(0x55575b42fc90 msgr2=0x55575b473020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 -- 172.21.15.32:0/4191672474 shutdown_connections 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.403+0000 7f4c6676e740 1 -- 172.21.15.32:0/4191672474 wait complete. 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.404+0000 7f4c6676e740 1 -- start start 2024-03-06T07:29:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.404+0000 7f4c6676e740 1 Processor -- start 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b4f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b434400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b4f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55575b45b580 con 0x55575b45cb90 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55575b48b140 con 0x55575b45c7b0 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55575b45bb80 con 0x55575b4742b0 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.405+0000 7f4c6676e740 10 client.0 _ll_register_callbacks cb 0x7f4c50006c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b434400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b434400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38010/0 (socket says 172.21.15.32:38010) 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 -- 172.21.15.32:0/2117431929 learned_addr learned my addr 172.21.15.32:0/2117431929 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c61000640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b4f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c61a00640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b4f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 -- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 msgr2=0x55575b4f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b4f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 -- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 msgr2=0x55575b4f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b4f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 -- 172.21.15.32:0/2117431929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55575b433af0 con 0x55575b45cb90 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c61a00640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b4f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c5be00640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b434400 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4c48032010 tx=0x7f4c4802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6e86edc5008558 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.406+0000 7f4c42000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c42000640 1 -- 172.21.15.32:0/2117431929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4805d060 con 0x55575b45cb90 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c42000640 1 -- 172.21.15.32:0/2117431929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c48057290 con 0x55575b45cb90 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c5007c070 con 0x55575b45cb90 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c42000640 1 -- 172.21.15.32:0/2117431929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c48063050 con 0x55575b45cb90 2024-03-06T07:29:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 10 client.4653 fetch_fsmap learned FSMap version 8 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55575b4879d0 con 0x55575b45cb90 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c42000640 1 -- 172.21.15.32:0/2117431929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5007c070 con 0x55575b45cb90 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c42000640 1 -- 172.21.15.32:0/2117431929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4c48066020 con 0x55575b45cb90 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 10 client.4653 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 20 client.4653 populate_metadata read hostname 'smithi032' 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b487000 0x55575b4873e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 4 client.4653 mds_command: new command op to 4258 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.407+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 --> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55575b476a30 con 0x55575b487000 2024-03-06T07:29:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.408+0000 7f4c61000640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b487000 0x55575b4873e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c61000640 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b487000 0x55575b4873e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5767d96c6848bf1e server_cookie=6e2174e874807deb in_seq=0 out_seq=0 2024-03-06T07:29:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c42000640 10 client.4653 ms_handle_connect on v2:172.21.15.39:6832/3857082618 2024-03-06T07:29:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c42000640 1 -- 172.21.15.32:0/2117431929 <== mds.0 v2:172.21.15.39:6832/3857082618 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55575b476a30 con 0x55575b487000 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c42000640 10 client.4653 handle_command_reply: tid=0 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 client.4653 shutdown 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 msgr2=0x55575b434400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b434400 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4c48032010 tx=0x7f4c4802e000 comp rx=0 tx=0).stop 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 shutdown_connections 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:6832/3857082618,v1:172.21.15.39:6833/3857082618] conn(0x55575b487000 0x55575b4873e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55575b4742b0 0x55575b4f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55575b45cb90 0x55575b434400 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 --2- 172.21.15.32:0/2117431929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55575b45c7b0 0x55575b4f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.409+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 >> 172.21.15.32:0/2117431929 conn(0x55575b42fc90 msgr2=0x55575b4332b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.410+0000 7f4c42000640 0 client.4653 ms_handle_reset on v2:172.21.15.39:6832/3857082618 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.410+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 shutdown_connections 2024-03-06T07:29:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.410+0000 7f4c6676e740 1 -- 172.21.15.32:0/2117431929 wait complete. 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.410+0000 7f4c6676e740 20 client.4653 trim_cache size 0 max 16384 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4258, 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 17.099105882, 2024-03-06T07:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 8, 2024-03-06T07:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 21, 2024-03-06T07:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 21, 2024-03-06T07:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 25.039452409999999 2024-03-06T07:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3c06b4d0 msgr2=0x7f4c3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3c06b4d0 0x7f4c3c06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c48036000 tx=0x7f4c48030000 comp rx=0 tx=0).stop 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 msgr2=0x7f4c5c14d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 0x7f4c5c14d9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000d800 tx=0x7f4c5000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 shutdown_connections 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3c06b4d0 0x7f4c3c06d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5c0cf5c0 0x7f4c5c153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5c0b3c90 0x7f4c5c1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 --2- 172.21.15.32:0/2121622447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c06cb10 0x7f4c5c14d9f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 >> 172.21.15.32:0/2121622447 conn(0x7f4c5c06afb0 msgr2=0x7f4c5c06d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 shutdown_connections 2024-03-06T07:29:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:18.411+0000 7f4c63a00640 1 -- 172.21.15.32:0/2121622447 wait complete. 2024-03-06T07:29:18.430 INFO:teuthology.run_tasks:Running task kclient... 2024-03-06T07:29:18.453 INFO:tasks.kclient:Mounting kernel clients... 2024-03-06T07:29:18.453 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:29:18.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.453 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:29:18.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-03-06T07:29:18.651 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-03-06T07:29:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:18.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.656 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:29:18.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-03-06T07:29:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-03-06T07:29:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:18.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.819 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-06T07:29:18.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.878 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-03-06T07:29:18.981 INFO:teuthology.orchestra.run.smithi039.stderr:Cannot find device "ceph-brx" 2024-03-06T07:29:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:18.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:18.985 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-06T07:29:19.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:19.032 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-03-06T07:29:19.119 INFO:teuthology.orchestra.run.smithi039.stderr:Cannot find device "ceph-brx" 2024-03-06T07:29:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:19.122 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-06T07:29:19.123 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:29:19.123 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:19.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:19.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.352+0000 7f901aa00640 1 Processor -- start 2024-03-06T07:29:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.352+0000 7f901aa00640 1 -- start start 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90140d2960 0x7f90140d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90140d0b00 0x7f90140b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90140d2dc0 con 0x7f901415e070 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901aa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90140d0f70 con 0x7f90140d0b00 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90140d18c0 con 0x7f90140d2960 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90140d0b00 0x7f90140b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90140d0b00 0x7f90140b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38024/0 (socket says 172.21.15.32:38024) 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f901a000640 1 -- 172.21.15.32:0/629125232 learned_addr learned my addr 172.21.15.32:0/629125232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.354+0000 7f9018c00640 1 --2- 172.21.15.32:0/629125232 >> v2:172.21.15.39:3300/0 conn(0x7f90140d2960 0x7f90140d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.355+0000 7f901a000640 1 -- 172.21.15.32:0/629125232 >> v2:172.21.15.39:3300/0 conn(0x7f90140d2960 msgr2=0x7f90140d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.355+0000 7f901a000640 1 --2- 172.21.15.32:0/629125232 >> v2:172.21.15.39:3300/0 conn(0x7f90140d2960 0x7f90140d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.355+0000 7f901a000640 1 -- 172.21.15.32:0/629125232 >> v1:172.21.15.39:6789/0 conn(0x7f901415e070 legacy=0x7f90140cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.355+0000 7f901a000640 1 -- 172.21.15.32:0/629125232 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140d1ba0 con 0x7f90140d0b00 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.356+0000 7f901a000640 1 --2- 172.21.15.32:0/629125232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f90140b3630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90140b6880 tx=0x7f900c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a365b0edc4f161f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.356+0000 7f9013e00640 1 -- 172.21.15.32:0/629125232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c02ca80 con 0x7f90140d0b00 2024-03-06T07:29:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.356+0000 7f9013e00640 1 -- 172.21.15.32:0/629125232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c02cc20 con 0x7f90140d0b00 2024-03-06T07:29:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.356+0000 7f9013e00640 1 -- 172.21.15.32:0/629125232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0343f0 con 0x7f90140d0b00 2024-03-06T07:29:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 -- 172.21.15.32:0/629125232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 msgr2=0x7f90140b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 --2- 172.21.15.32:0/629125232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f90140b3630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90140b6880 tx=0x7f900c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 -- 172.21.15.32:0/629125232 shutdown_connections 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 --2- 172.21.15.32:0/629125232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f90140b3630 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 --2- 172.21.15.32:0/629125232 >> v2:172.21.15.39:3300/0 conn(0x7f90140d2960 0x7f90140d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 -- 172.21.15.32:0/629125232 >> 172.21.15.32:0/629125232 conn(0x7f901408d2a0 msgr2=0x7f90140d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 -- 172.21.15.32:0/629125232 shutdown_connections 2024-03-06T07:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.357+0000 7f901aa00640 1 -- 172.21.15.32:0/629125232 wait complete. 2024-03-06T07:29:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.359+0000 7f901aa00640 1 Processor -- start 2024-03-06T07:29:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.359+0000 7f901aa00640 1 -- start start 2024-03-06T07:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f901aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f901415e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f901aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90140d2960 0x7f9014160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f901aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f901415e070 0x7f90141f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f901aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90140d2440 con 0x7f90140d0b00 2024-03-06T07:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f901aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90140d2160 con 0x7f90140d2960 2024-03-06T07:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f901aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140d45b0 con 0x7f901415e070 2024-03-06T07:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f9019600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f901415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f9019600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f901415e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38030/0 (socket says 172.21.15.32:38030) 2024-03-06T07:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f9019600640 1 -- 172.21.15.32:0/2394818763 learned_addr learned my addr 172.21.15.32:0/2394818763 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.360+0000 7f9018c00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90140d2960 0x7f9014160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f901a000640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f901415e070 0x7f90141f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9019600640 1 -- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f901415e070 msgr2=0x7f90141f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9019600640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f901415e070 0x7f90141f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9019600640 1 -- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90140d2960 msgr2=0x7f9014160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9019600640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90140d2960 0x7f9014160d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9019600640 1 -- 172.21.15.32:0/2394818763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901415eb50 con 0x7f90140d0b00 2024-03-06T07:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9019600640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f901415e450 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f900800b420 tx=0x7f900800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008011830 con 0x7f90140d0b00 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008016070 con 0x7f90140d0b00 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901415ee30 con 0x7f90140d0b00 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.361+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008011c10 con 0x7f90140d0b00 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.362+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901415f3f0 con 0x7f90140d0b00 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.363+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901404f330 con 0x7f90140d0b00 2024-03-06T07:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.364+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f900801b460 con 0x7f90140d0b00 2024-03-06T07:29:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.365+0000 7f9011600640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ff406b4d0 0x7f8ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.365+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9008089d90 con 0x7f90140d0b00 2024-03-06T07:29:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.365+0000 7f9018c00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ff406b4d0 0x7f8ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.366+0000 7f9018c00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ff406b4d0 0x7f8ff406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90140ceb10 tx=0x7f9000008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.370+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9008020020 con 0x7f90140d0b00 2024-03-06T07:29:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.610+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f90140cfc10 con 0x7f90140d0b00 2024-03-06T07:29:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.611+0000 7f9011600640 1 -- 172.21.15.32:0/2394818763 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9008058b70 con 0x7f90140d0b00 2024-03-06T07:29:19.613 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ff406b4d0 msgr2=0x7f8ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ff406b4d0 0x7f8ff406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90140ceb10 tx=0x7f9000008040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 msgr2=0x7f901415e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f901415e450 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f900800b420 tx=0x7f900800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 shutdown_connections 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ff406b4d0 0x7f8ff406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f901415e070 0x7f90141f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90140d2960 0x7f9014160d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 --2- 172.21.15.32:0/2394818763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90140d0b00 0x7f901415e450 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 >> 172.21.15.32:0/2394818763 conn(0x7f901408d2a0 msgr2=0x7f901415d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 shutdown_connections 2024-03-06T07:29:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:19.614+0000 7f901aa00640 1 -- 172.21.15.32:0/2394818763 wait complete. 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:29:19.637 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:19.638 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:19.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:19.638 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42378sec preferred_lft 42378sec 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:29:19.694 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:29:19.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:29:19.695 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:20.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:20.162 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:20.244 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:29:20.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:20.248 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:29:20.303 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:20.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:20.304 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:20.305 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:20.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:29:20.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:29:20.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:29:20.305 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:21.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:21.080 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:29:21.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:21.139 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-03-06T07:29:21.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:21.196 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:21.196 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:21.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:21.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:29:21.197 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:21.345 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:29:21.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:29:21.346 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:21.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:21.661 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:21.661 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:21.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:29:21.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:29:21.661 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:21.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:21.837 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:21.883 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:21.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:21.883 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:21.981 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:29:21.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:21.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:22.119 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:22.119 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:22.747 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:22.748 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:22.748 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:29:22.748 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:29:22.748 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:22.758 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:22.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:22.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:22.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:22.857 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:22.934 INFO:teuthology.orchestra.run.smithi032.stdout:client4543 2024-03-06T07:29:22.934 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:22.934 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:22.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:22.993 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:23.068 INFO:teuthology.orchestra.run.smithi032.stdout:client4543 2024-03-06T07:29:23.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.068 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:23.150 INFO:teuthology.orchestra.run.smithi032.stdout:client4543 2024-03-06T07:29:23.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.150 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e91bbd07-5556-4949-b381-99a73b242ae4.client4543/status') 2024-03-06T07:29:23.265 INFO:teuthology.orchestra.run.smithi032.stdout:instance: client.4543 (3)172.21.15.32:0/69169075 2024-03-06T07:29:23.265 INFO:teuthology.orchestra.run.smithi032.stdout:blocklisted: false 2024-03-06T07:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records in 2024-03-06T07:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records out 2024-03-06T07:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:68 bytes copied, 9.8069e-05 s, 693 kB/s 2024-03-06T07:29:23.270 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-06T07:29:23.270 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:29:23.270 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:29:23.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.487+0000 7eff5aa00640 1 Processor -- start 2024-03-06T07:29:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.488+0000 7eff5aa00640 1 -- start start 2024-03-06T07:29:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.489+0000 7eff5aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 0x7eff540ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.489+0000 7eff5aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7eff5406e240 0x7eff540ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.489+0000 7eff5aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff540ce9a0 con 0x7eff54161670 2024-03-06T07:29:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.489+0000 7eff5aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff540730d0 con 0x7eff5406e240 2024-03-06T07:29:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.489+0000 7eff5aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff540cec80 con 0x7eff54071380 2024-03-06T07:29:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.490+0000 7eff53400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 0x7eff540ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.490+0000 7eff53400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 0x7eff540ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:51240/0 (socket says 172.21.15.32:51240) 2024-03-06T07:29:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.490+0000 7eff53400640 1 -- 172.21.15.32:0/1193182343 learned_addr learned my addr 172.21.15.32:0/1193182343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.490+0000 7eff58a00640 1 --2- 172.21.15.32:0/1193182343 >> v2:172.21.15.39:3300/0 conn(0x7eff5406e240 0x7eff540ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.491+0000 7eff53400640 1 -- 172.21.15.32:0/1193182343 >> v2:172.21.15.39:3300/0 conn(0x7eff5406e240 msgr2=0x7eff540ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.491+0000 7eff53400640 1 --2- 172.21.15.32:0/1193182343 >> v2:172.21.15.39:3300/0 conn(0x7eff5406e240 0x7eff540ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.491+0000 7eff53400640 1 -- 172.21.15.32:0/1193182343 >> v1:172.21.15.39:6789/0 conn(0x7eff54161670 legacy=0x7eff54070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.491+0000 7eff53400640 1 -- 172.21.15.32:0/1193182343 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff540cef60 con 0x7eff54071380 2024-03-06T07:29:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.493+0000 7eff53400640 1 --2- 172.21.15.32:0/1193182343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 0x7eff540ad090 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7eff40009870 tx=0x7eff4002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc8bef4c1bea2035 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.493+0000 7eff52a00640 1 -- 172.21.15.32:0/1193182343 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40034070 con 0x7eff54071380 2024-03-06T07:29:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.493+0000 7eff52a00640 1 -- 172.21.15.32:0/1193182343 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff40030840 con 0x7eff54071380 2024-03-06T07:29:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.493+0000 7eff52a00640 1 -- 172.21.15.32:0/1193182343 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40030c10 con 0x7eff54071380 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 -- 172.21.15.32:0/1193182343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 msgr2=0x7eff540ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 --2- 172.21.15.32:0/1193182343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 0x7eff540ad090 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7eff40009870 tx=0x7eff4002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 -- 172.21.15.32:0/1193182343 shutdown_connections 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 --2- 172.21.15.32:0/1193182343 >> v2:172.21.15.39:3300/0 conn(0x7eff5406e240 0x7eff540ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 --2- 172.21.15.32:0/1193182343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff54071380 0x7eff540ad090 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 -- 172.21.15.32:0/1193182343 >> 172.21.15.32:0/1193182343 conn(0x7eff5406afb0 msgr2=0x7eff5406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 -- 172.21.15.32:0/1193182343 shutdown_connections 2024-03-06T07:29:23.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.494+0000 7eff5aa00640 1 -- 172.21.15.32:0/1193182343 wait complete. 2024-03-06T07:29:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.496+0000 7eff5aa00640 1 Processor -- start 2024-03-06T07:29:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.497+0000 7eff5aa00640 1 -- start start 2024-03-06T07:29:23.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff5aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff5406e240 0x7eff540cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff5aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 0x7eff54156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff5aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff54161670 0x7eff5409aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff5aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff540ce600 con 0x7eff54071380 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff5aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff540cf520 con 0x7eff54161670 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff5aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff540ce780 con 0x7eff5406e240 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff53400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 0x7eff54156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff53400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 0x7eff54156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38036/0 (socket says 172.21.15.32:38036) 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.498+0000 7eff53400640 1 -- 172.21.15.32:0/3841324991 learned_addr learned my addr 172.21.15.32:0/3841324991 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53e00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff5406e240 0x7eff540cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff58a00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff54161670 0x7eff5409aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53400640 1 -- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff5406e240 msgr2=0x7eff540cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53400640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff5406e240 0x7eff540cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53400640 1 -- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff54161670 msgr2=0x7eff5409aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53400640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff54161670 0x7eff5409aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53400640 1 -- 172.21.15.32:0/3841324991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5415b9d0 con 0x7eff54071380 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff58a00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff54161670 0x7eff5409aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.499+0000 7eff53400640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 0x7eff54156a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff40000c00 tx=0x7eff40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.500+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40048070 con 0x7eff54071380 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.500+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff40030640 con 0x7eff54071380 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.500+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40034040 con 0x7eff54071380 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.500+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff540c9ff0 con 0x7eff54071380 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.500+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff540ca5b0 con 0x7eff54071380 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.501+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff54069510 con 0x7eff54071380 2024-03-06T07:29:23.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.502+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff40037070 con 0x7eff54071380 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.503+0000 7eff4be00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff3406b4d0 0x7eff3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.503+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff400abe10 con 0x7eff54071380 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.504+0000 7eff53e00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff3406b4d0 0x7eff3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.504+0000 7eff53e00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff3406b4d0 0x7eff3406d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff44007fd0 tx=0x7eff44009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.508+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff400af030 con 0x7eff54071380 2024-03-06T07:29:23.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.747+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7eff540adfa0 con 0x7eff54071380 2024-03-06T07:29:23.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.748+0000 7eff4be00640 1 -- 172.21.15.32:0/3841324991 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7eff540adfa0 con 0x7eff54071380 2024-03-06T07:29:23.751 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:23.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff3406b4d0 msgr2=0x7eff3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff3406b4d0 0x7eff3406d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff44007fd0 tx=0x7eff44009040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 msgr2=0x7eff54156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 0x7eff54156a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff40000c00 tx=0x7eff40004730 comp rx=0 tx=0).stop 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 shutdown_connections 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff3406b4d0 0x7eff3406d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff54161670 0x7eff5409aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff54071380 0x7eff54156a80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 --2- 172.21.15.32:0/3841324991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff5406e240 0x7eff540cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 >> 172.21.15.32:0/3841324991 conn(0x7eff5406afb0 msgr2=0x7eff540b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 shutdown_connections 2024-03-06T07:29:23.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:23.751+0000 7eff5aa00640 1 -- 172.21.15.32:0/3841324991 wait complete. 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:23.773 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:29:23.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.774 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42373sec preferred_lft 42373sec 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:23.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.828 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:23.921 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:29:23.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.926 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:29:23.979 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:29:23.979 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:23.980 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:29:23.980 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:29:23.980 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:23.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.981 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:23.981 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:23.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:29:23.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:29:23.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:29:23.981 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:24.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.181 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:29:24.233 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:24.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.235 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-03-06T07:29:24.286 INFO:teuthology.orchestra.run.smithi032.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:29:24.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.288 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:24.288 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:24.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:24.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:29:24.288 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:24.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 3e:7d:2f:8d:75:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.535 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::3c7d:2fff:fe8d:75fb/64 scope link proto kernel_ll 2024-03-06T07:29:24.536 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:29:24.640 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:24.640 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:24.640 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:24.640 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 3e:7d:2f:8d:75:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::3c7d:2fff:fe8d:75fb/64 scope link proto kernel_ll 2024-03-06T07:29:24.641 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:24.645 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:29:24.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:29:24.646 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:24.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:24.938 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:29:24.939 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:29:24.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.1 up 2024-03-06T07:29:24.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:29:24.939 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:29:25.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:25.114 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:29:25.164 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:29:25.165 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:25.165 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:29:25.215 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-06T07:29:25.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:25.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:25.362 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:25.362 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 2@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:29:25.400 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-06T07:29:25.400 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:25.406 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:25.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:25.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:29:25.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:25.495 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:25.554 INFO:teuthology.orchestra.run.smithi032.stdout:client4665 2024-03-06T07:29:25.555 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:25.555 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:25.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:25.607 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:client4665 2024-03-06T07:29:25.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:25.664 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:client4665 2024-03-06T07:29:25.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:25.723 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e91bbd07-5556-4949-b381-99a73b242ae4.client4665/status') 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi032.stdout:instance: client.4665 (3)192.168.0.2:0/774475130 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi032.stdout:blocklisted: false 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records in 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records out 2024-03-06T07:29:25.821 INFO:teuthology.orchestra.run.smithi032.stderr:68 bytes copied, 0.000169731 s, 401 kB/s 2024-03-06T07:29:25.825 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-06T07:29:25.825 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:29:25.825 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:29:25.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:26.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.046+0000 7f4686400640 1 Processor -- start 2024-03-06T07:29:26.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.046+0000 7f4686400640 1 -- start start 2024-03-06T07:29:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4686400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46801552b0 0x7f468015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4686400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46800cd060 0x7f46800b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4686400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468015ed10 con 0x7f468015e810 2024-03-06T07:29:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4686400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468015eff0 con 0x7f46801552b0 2024-03-06T07:29:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4686400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468015f2d0 con 0x7f46800cd060 2024-03-06T07:29:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4685000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46801552b0 0x7f468015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4685000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46801552b0 0x7f468015b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38044/0 (socket says 172.21.15.32:38044) 2024-03-06T07:29:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.048+0000 7f4685000640 1 -- 172.21.15.32:0/1949985521 learned_addr learned my addr 172.21.15.32:0/1949985521 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:26.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.049+0000 7f4685a00640 1 --2- 172.21.15.32:0/1949985521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46800cd060 0x7f46800b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.049+0000 7f4685000640 1 -- 172.21.15.32:0/1949985521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46800cd060 msgr2=0x7f46800b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.049+0000 7f4685000640 1 --2- 172.21.15.32:0/1949985521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46800cd060 0x7f46800b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.049+0000 7f4685000640 1 -- 172.21.15.32:0/1949985521 >> v1:172.21.15.39:6789/0 conn(0x7f468015e810 legacy=0x7f46800cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.049+0000 7f4685000640 1 -- 172.21.15.32:0/1949985521 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468015f5b0 con 0x7f46801552b0 2024-03-06T07:29:26.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.050+0000 7f4685000640 1 --2- 172.21.15.32:0/1949985521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46801552b0 0x7f468015b1d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4674009870 tx=0x7f467402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307175698fe1a1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.050+0000 7f467f400640 1 -- 172.21.15.32:0/1949985521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674035020 con 0x7f46801552b0 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.050+0000 7f467f400640 1 -- 172.21.15.32:0/1949985521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4674031640 con 0x7f46801552b0 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.050+0000 7f467f400640 1 -- 172.21.15.32:0/1949985521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467402d040 con 0x7f46801552b0 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 -- 172.21.15.32:0/1949985521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46801552b0 msgr2=0x7f468015b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 --2- 172.21.15.32:0/1949985521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46801552b0 0x7f468015b1d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4674009870 tx=0x7f467402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 -- 172.21.15.32:0/1949985521 shutdown_connections 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 --2- 172.21.15.32:0/1949985521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46800cd060 0x7f46800b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 --2- 172.21.15.32:0/1949985521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46801552b0 0x7f468015b1d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 -- 172.21.15.32:0/1949985521 >> 172.21.15.32:0/1949985521 conn(0x7f468006b3d0 msgr2=0x7f468006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 -- 172.21.15.32:0/1949985521 shutdown_connections 2024-03-06T07:29:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.051+0000 7f4686400640 1 -- 172.21.15.32:0/1949985521 wait complete. 2024-03-06T07:29:26.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.053+0000 7f4686400640 1 Processor -- start 2024-03-06T07:29:26.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.053+0000 7f4686400640 1 -- start start 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4686400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46800cd060 0x7f4680154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4686400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46801552b0 0x7f46801495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4686400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 0x7f46800cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4686400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468006dd30 con 0x7f468015e810 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4686400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46800b3760 con 0x7f46800cd060 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4686400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468006deb0 con 0x7f46801552b0 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4685a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 0x7f46800cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4685a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 0x7f46800cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38048/0 (socket says 172.21.15.32:38048) 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.054+0000 7f4685a00640 1 -- 172.21.15.32:0/2919813452 learned_addr learned my addr 172.21.15.32:0/2919813452 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685000640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46800cd060 0x7f4680154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f467fe00640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46801552b0 0x7f46801495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685a00640 1 -- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46801552b0 msgr2=0x7f46801495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685a00640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46801552b0 0x7f46801495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685a00640 1 -- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46800cd060 msgr2=0x7f4680154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685a00640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46800cd060 0x7f4680154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685a00640 1 -- 172.21.15.32:0/2919813452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46801f7ba0 con 0x7f468015e810 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f4685a00640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 0x7f46800cd9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4678007e60 tx=0x7f467800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.055+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678014620 con 0x7f468015e810 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.056+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4678019070 con 0x7f468015e810 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.056+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678014a20 con 0x7f468015e810 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.056+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46801f7e80 con 0x7f468015e810 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.056+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4680154aa0 con 0x7f468015e810 2024-03-06T07:29:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.057+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468004f330 con 0x7f468015e810 2024-03-06T07:29:26.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.058+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4678014bc0 con 0x7f468015e810 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.059+0000 7f467cc00640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f466006b4d0 0x7f466006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.059+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f467808aae0 con 0x7f468015e810 2024-03-06T07:29:26.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.060+0000 7f4685000640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f466006b4d0 0x7f466006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.060+0000 7f4685000640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f466006b4d0 0x7f466006d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4674036000 tx=0x7f467403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.064+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4678059270 con 0x7f468015e810 2024-03-06T07:29:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.309+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f46800cf4a0 con 0x7f468015e810 2024-03-06T07:29:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.309+0000 7f467cc00640 1 -- 172.21.15.32:0/2919813452 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f467805d310 con 0x7f468015e810 2024-03-06T07:29:26.312 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f466006b4d0 msgr2=0x7f466006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f466006b4d0 0x7f466006d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4674036000 tx=0x7f467403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 msgr2=0x7f46800cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 0x7f46800cd9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4678007e60 tx=0x7f467800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 shutdown_connections 2024-03-06T07:29:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f466006b4d0 0x7f466006d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f468015e810 0x7f46800cd9c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46801552b0 0x7f46801495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 --2- 172.21.15.32:0/2919813452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46800cd060 0x7f4680154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 >> 172.21.15.32:0/2919813452 conn(0x7f468006b3d0 msgr2=0x7f46800b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 shutdown_connections 2024-03-06T07:29:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:26.312+0000 7f4686400640 1 -- 172.21.15.32:0/2919813452 wait complete. 2024-03-06T07:29:26.333 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:29:26.333 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:26.333 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-03-06T07:29:26.333 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:29:26.334 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:29:26.334 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:29:26.334 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:29:26.334 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:26.334 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:29:26.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:26.334 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-03-06T07:29:26.391 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42489sec preferred_lft 42489sec 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-03-06T07:29:26.393 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:26.393 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:26.394 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:26.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:29:26.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-03-06T07:29:26.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-03-06T07:29:26.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:29:26.394 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:26.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:26.860 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:26.935 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-03-06T07:29:26.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:26.938 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-03-06T07:29:26.992 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-03-06T07:29:26.992 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:26.992 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:29:26.992 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:29:26.993 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:26.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:26.994 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:26.994 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:26.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:29:26.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:29:26.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:29:26.994 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:27.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:27.704 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-06T07:29:27.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:27.756 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-03-06T07:29:27.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:27.801 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:27.801 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:27.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:29:27.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:29:27.801 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:27.985 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:29:27.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:29:27.985 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:28.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:28.267 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:28.267 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:28.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-03-06T07:29:28.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:29:28.268 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:28.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:28.431 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:29:28.475 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:29:28.475 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:28.475 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:29:28.563 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-06T07:29:28.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:28.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:28.694 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:28.694 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:29.335 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:29.335 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:29.335 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:29:29.335 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: trying mount with new device syntax: 1@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:29:29.335 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:29:29.338 INFO:teuthology.orchestra.run.smithi039.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-06T07:29:29.338 INFO:teuthology.orchestra.run.smithi039.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:29.338 INFO:teuthology.orchestra.run.smithi039.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:29.338 INFO:teuthology.orchestra.run.smithi039.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:29.338 INFO:teuthology.orchestra.run.smithi039.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:29.342 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:29.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:29.342 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:29:29.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:29.430 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:29.483 INFO:teuthology.orchestra.run.smithi039.stdout:client4561 2024-03-06T07:29:29.483 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:29.484 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:29.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:29.530 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:29.628 INFO:teuthology.orchestra.run.smithi039.stdout:client4561 2024-03-06T07:29:29.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:29.629 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:29.680 INFO:teuthology.orchestra.run.smithi039.stdout:client4561 2024-03-06T07:29:29.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:29.681 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e91bbd07-5556-4949-b381-99a73b242ae4.client4561/status') 2024-03-06T07:29:29.767 INFO:teuthology.orchestra.run.smithi039.stdout:instance: client.4561 (3)192.168.0.1:0/701682536 2024-03-06T07:29:29.767 INFO:teuthology.orchestra.run.smithi039.stdout:blocklisted: false 2024-03-06T07:29:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records in 2024-03-06T07:29:29.768 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records out 2024-03-06T07:29:29.768 INFO:teuthology.orchestra.run.smithi039.stderr:68 bytes copied, 6.9647e-05 s, 976 kB/s 2024-03-06T07:29:29.771 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-06T07:29:29.771 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:29:29.771 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:29:29.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:30.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.001+0000 7f4eeac00640 1 Processor -- start 2024-03-06T07:29:30.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.001+0000 7f4eeac00640 1 -- start start 2024-03-06T07:29:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.003+0000 7f4eeac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ee4145300 0x7f4ee40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.003+0000 7f4eeac00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee40714d0 con 0x7f4ee4070f10 2024-03-06T07:29:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.003+0000 7f4eeac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee40717b0 con 0x7f4ee40cd060 2024-03-06T07:29:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.003+0000 7f4eeac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee4071a90 con 0x7f4ee4145300 2024-03-06T07:29:30.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.003+0000 7f4ee8e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ee4070f10 0x7f4ee40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43652/0 (socket says 172.21.15.32:43652) 2024-03-06T07:29:30.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.003+0000 7f4ee8e00640 1 -- 172.21.15.32:0/1929581256 learned_addr learned my addr 172.21.15.32:0/1929581256 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:30.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee9800640 1 --2- 172.21.15.32:0/1929581256 >> v2:172.21.15.39:3300/0 conn(0x7f4ee4145300 0x7f4ee40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306273813 0 0) 0x7f4ee40714d0 con 0x7f4ee4070f10 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ecc003520 con 0x7f4ee4070f10 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1802602194 0 0) 0x7f4ecc003520 con 0x7f4ee4070f10 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 >> v2:172.21.15.39:3300/0 conn(0x7f4ee4145300 msgr2=0x7f4ee40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 --2- 172.21.15.32:0/1929581256 >> v2:172.21.15.39:3300/0 conn(0x7f4ee4145300 0x7f4ee40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 >> v1:172.21.15.39:6789/0 conn(0x7f4ee40cd060 legacy=0x7f4ee415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4071d70 con 0x7f4ee4070f10 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f4ed0001670 con 0x7f4ee4070f10 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 >> v1:172.21.15.32:6789/0 conn(0x7f4ee4070f10 legacy=0x7f4ee40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.004+0000 7f4ee3e00640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ecc0039b0 0x7f4ecc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee3e00640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc004800 0x7f4ecc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee3e00640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ecc0051c0 0x7f4ecc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee4071a90 con 0x7f4ecc0039b0 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc00c160 con 0x7f4ecc0051c0 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc00c360 con 0x7f4ecc004800 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee9800640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ecc0039b0 0x7f4ecc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4eea200640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ecc0051c0 0x7f4ecc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee8e00640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc004800 0x7f4ecc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee9800640 1 -- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc004800 msgr2=0x7f4ecc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee9800640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc004800 0x7f4ecc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee9800640 1 -- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ecc0051c0 msgr2=0x7f4ecc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee9800640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ecc0051c0 0x7f4ecc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4ee9800640 1 -- 172.21.15.32:0/1929581256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc003520 con 0x7f4ecc0039b0 2024-03-06T07:29:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.005+0000 7f4eea200640 1 -- 172.21.15.32:0/1929581256 reap_dead start 2024-03-06T07:29:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.006+0000 7f4ee9800640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ecc0039b0 0x7f4ecc003dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009ac0 tx=0x7f4ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41212502c9af09c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8039070 con 0x7f4ecc0039b0 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed802c6b0 con 0x7f4ecc0039b0 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4ee3e00640 1 -- 172.21.15.32:0/1929581256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8009690 con 0x7f4ecc0039b0 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4eeac00640 1 -- 172.21.15.32:0/1929581256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ecc0039b0 msgr2=0x7f4ecc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4eeac00640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ecc0039b0 0x7f4ecc003dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009ac0 tx=0x7f4ecc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4eeac00640 1 -- 172.21.15.32:0/1929581256 shutdown_connections 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4eeac00640 1 --2- 172.21.15.32:0/1929581256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ecc0039b0 0x7f4ecc003dd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.007+0000 7f4eeac00640 1 -- 172.21.15.32:0/1929581256 >> 172.21.15.32:0/1929581256 conn(0x7f4ee406b3d0 msgr2=0x7f4ee406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.008+0000 7f4eeac00640 1 -- 172.21.15.32:0/1929581256 shutdown_connections 2024-03-06T07:29:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.008+0000 7f4eeac00640 1 -- 172.21.15.32:0/1929581256 wait complete. 2024-03-06T07:29:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.009+0000 7f4eeac00640 1 Processor -- start 2024-03-06T07:29:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.009+0000 7f4eeac00640 1 -- start start 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.010+0000 7f4eeac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc0039b0 0x7f4ee4137290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.010+0000 7f4eeac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 0x7f4ee41f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.010+0000 7f4eeac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee41f4260 0x7f4ee41e5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.010+0000 7f4eeac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee40ccad0 con 0x7f4ee41e92e0 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.010+0000 7f4eeac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee40cc950 con 0x7f4ee41f4260 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.010+0000 7f4eeac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee40ccc50 con 0x7f4ecc0039b0 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 0x7f4ee41f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 0x7f4ee41f3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35982/0 (socket says 172.21.15.32:35982) 2024-03-06T07:29:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 -- 172.21.15.32:0/2951847115 learned_addr learned my addr 172.21.15.32:0/2951847115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:30.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4eea200640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee41f4260 0x7f4ee41e5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee9800640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc0039b0 0x7f4ee4137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 -- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc0039b0 msgr2=0x7f4ee4137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc0039b0 0x7f4ee4137290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 -- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee41f4260 msgr2=0x7f4ee41e5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee41f4260 0x7f4ee41e5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 -- 172.21.15.32:0/2951847115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee40c4da0 con 0x7f4ee41e92e0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.011+0000 7f4ee8e00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 0x7f4ee41f3d20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000d770 tx=0x7f4ed000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.012+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0019070 con 0x7f4ee41e92e0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.012+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed00074e0 con 0x7f4ee41e92e0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.012+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee41e7820 con 0x7f4ee41e92e0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.012+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0004a20 con 0x7f4ee41e92e0 2024-03-06T07:29:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.012+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee41e55b0 con 0x7f4ee41e92e0 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.013+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee404f330 con 0x7f4ee41e92e0 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.014+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ed001c070 con 0x7f4ee41e92e0 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.015+0000 7f4ee1600640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.015+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ed0089d00 con 0x7f4ee41e92e0 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.015+0000 7f4ee9800640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.016+0000 7f4ee9800640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ec406b4d0 0x7f4ec406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8036000 tx=0x7f4ed8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.020+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ed0058b70 con 0x7f4ee41e92e0 2024-03-06T07:29:30.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.259+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4ee41e5780 con 0x7f4ee41e92e0 2024-03-06T07:29:30.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.260+0000 7f4ee1600640 1 -- 172.21.15.32:0/2951847115 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4ed0058470 con 0x7f4ee41e92e0 2024-03-06T07:29:30.262 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:30.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ec406b4d0 msgr2=0x7f4ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ec406b4d0 0x7f4ec406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8036000 tx=0x7f4ed8030000 comp rx=0 tx=0).stop 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 msgr2=0x7f4ee41f3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 0x7f4ee41f3d20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000d770 tx=0x7f4ed000dc30 comp rx=0 tx=0).stop 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 shutdown_connections 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee41f4260 0x7f4ee41e5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ee41e92e0 0x7f4ee41f3d20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 --2- 172.21.15.32:0/2951847115 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ecc0039b0 0x7f4ee4137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 >> 172.21.15.32:0/2951847115 conn(0x7f4ee406b3d0 msgr2=0x7f4ee40b4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 shutdown_connections 2024-03-06T07:29:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:30.263+0000 7f4eeac00640 1 -- 172.21.15.32:0/2951847115 wait complete. 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:29:30.283 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:29:30.284 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:30.284 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:29:30.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.284 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-03-06T07:29:30.341 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:30.341 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:30.341 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:30.341 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:29:30.342 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42485sec preferred_lft 42485sec 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 3e:59:15:36:1a:10 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::6c6c:6eff:fee8:622b/64 scope link proto kernel_ll 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 3e:59:15:36:1a:10 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:29:30.343 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::3c59:15ff:fe36:1a10/64 scope link proto kernel_ll 2024-03-06T07:29:30.344 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:30.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.345 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:30.433 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-03-06T07:29:30.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.437 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-03-06T07:29:30.490 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-03-06T07:29:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:29:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:29:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.493 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:30.493 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:30.493 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:29:30.493 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:29:30.494 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:29:30.494 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:30.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.674 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-06T07:29:30.720 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:29:30.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.721 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-03-06T07:29:30.763 INFO:teuthology.orchestra.run.smithi039.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:29:30.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.764 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:30.765 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:30.765 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:29:30.765 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:29:30.765 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:30.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:30.949 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:31.041 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 6e:d4:23:c5:4b:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::6cd4:23ff:fec5:4b0d/64 scope link proto kernel_ll 2024-03-06T07:29:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:31.047 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:29:31.139 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:31.139 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:31.139 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 6e:d4:23:c5:4b:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::6cd4:23ff:fec5:4b0d/64 scope link proto kernel_ll 2024-03-06T07:29:31.140 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:31.145 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:29:31.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:31.145 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:31.145 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:31.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:29:31.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:29:31.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:29:31.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:29:31.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:29:31.146 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:31.436 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-03-06T07:29:31.436 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-06T07:29:31.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.1 up 2024-03-06T07:29:31.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:29:31.436 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-06T07:29:31.611 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:31.611 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:29:31.657 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:29:31.657 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:31.657 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:29:31.750 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-06T07:29:31.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:31.750 DEBUG:teuthology.orchestra.run.smithi039:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:31.920 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:31.920 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:31.920 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:29:31.920 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: trying mount with new device syntax: 3@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:29:31.920 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:29:31.923 INFO:teuthology.orchestra.run.smithi039.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-06T07:29:31.923 INFO:teuthology.orchestra.run.smithi039.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:31.923 INFO:teuthology.orchestra.run.smithi039.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:31.923 INFO:teuthology.orchestra.run.smithi039.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:31.923 INFO:teuthology.orchestra.run.smithi039.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:31.927 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:31.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:31.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:29:32.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:32.014 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:32.066 INFO:teuthology.orchestra.run.smithi039.stdout:client4680 2024-03-06T07:29:32.066 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:32.066 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:32.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:32.113 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:32.212 INFO:teuthology.orchestra.run.smithi039.stdout:client4680 2024-03-06T07:29:32.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:32.212 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:32.262 INFO:teuthology.orchestra.run.smithi039.stdout:client4680 2024-03-06T07:29:32.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:32.263 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e91bbd07-5556-4949-b381-99a73b242ae4.client4680/status') 2024-03-06T07:29:32.339 INFO:teuthology.orchestra.run.smithi039.stdout:instance: client.4680 (3)172.21.15.39:0/814723343 2024-03-06T07:29:32.340 INFO:teuthology.orchestra.run.smithi039.stdout:blocklisted: false 2024-03-06T07:29:32.340 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records in 2024-03-06T07:29:32.340 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records out 2024-03-06T07:29:32.340 INFO:teuthology.orchestra.run.smithi039.stderr:69 bytes copied, 6.5509e-05 s, 1.1 MB/s 2024-03-06T07:29:32.343 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:29:32.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:32.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.581+0000 7f5264400640 1 Processor -- start 2024-03-06T07:29:32.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.581+0000 7f5264400640 1 -- start start 2024-03-06T07:29:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.583+0000 7f5264400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af1a0 0x7f525c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.583+0000 7f5264400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f525c0cdc20 0x7f525c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.583+0000 7f5264400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c15cc40 con 0x7f525c0af580 2024-03-06T07:29:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.583+0000 7f5264400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c15cf20 con 0x7f525c0cdc20 2024-03-06T07:29:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.583+0000 7f5264400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c15d200 con 0x7f525c0af1a0 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f525c0cdc20 0x7f525c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f525c0cdc20 0x7f525c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35988/0 (socket says 172.21.15.32:35988) 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 -- 172.21.15.32:0/590659041 learned_addr learned my addr 172.21.15.32:0/590659041 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5261a00640 1 --2- 172.21.15.32:0/590659041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af1a0 0x7f525c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 -- 172.21.15.32:0/590659041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af1a0 msgr2=0x7f525c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 --2- 172.21.15.32:0/590659041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af1a0 0x7f525c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 -- 172.21.15.32:0/590659041 >> v1:172.21.15.39:6789/0 conn(0x7f525c0af580 legacy=0x7f525c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.584+0000 7f5262400640 1 -- 172.21.15.32:0/590659041 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c15d4e0 con 0x7f525c0cdc20 2024-03-06T07:29:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.585+0000 7f5262400640 1 --2- 172.21.15.32:0/590659041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0cdc20 0x7f525c0b2dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5254009e50 tx=0x7f525402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5194529cfb2792a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f525be00640 1 -- 172.21.15.32:0/590659041 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525402c9e0 con 0x7f525c0cdc20 2024-03-06T07:29:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f525be00640 1 -- 172.21.15.32:0/590659041 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525402cb80 con 0x7f525c0cdc20 2024-03-06T07:29:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f525be00640 1 -- 172.21.15.32:0/590659041 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52540343f0 con 0x7f525c0cdc20 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 -- 172.21.15.32:0/590659041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0cdc20 msgr2=0x7f525c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 --2- 172.21.15.32:0/590659041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0cdc20 0x7f525c0b2dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5254009e50 tx=0x7f525402be40 comp rx=0 tx=0).stop 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 -- 172.21.15.32:0/590659041 shutdown_connections 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 --2- 172.21.15.32:0/590659041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0cdc20 0x7f525c0b2dd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 --2- 172.21.15.32:0/590659041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af1a0 0x7f525c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 -- 172.21.15.32:0/590659041 >> 172.21.15.32:0/590659041 conn(0x7f525c06afb0 msgr2=0x7f525c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 -- 172.21.15.32:0/590659041 shutdown_connections 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.586+0000 7f5264400640 1 -- 172.21.15.32:0/590659041 wait complete. 2024-03-06T07:29:32.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.588+0000 7f5264400640 1 Processor -- start 2024-03-06T07:29:32.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.588+0000 7f5264400640 1 -- start start 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5264400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 0x7f525c1549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5264400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af580 0x7f525c155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5264400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c0cdc20 0x7f525c15d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5264400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0b36a0 con 0x7f525c0af1a0 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5264400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0b3520 con 0x7f525c0cdc20 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5264400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0b3820 con 0x7f525c0af580 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.589+0000 7f5261a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 0x7f525c1549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 0x7f525c1549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36002/0 (socket says 172.21.15.32:36002) 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 -- 172.21.15.32:0/1011472738 learned_addr learned my addr 172.21.15.32:0/1011472738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261000640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af580 0x7f525c155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5262400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c0cdc20 0x7f525c15d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 -- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af580 msgr2=0x7f525c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af580 0x7f525c155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 -- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c0cdc20 msgr2=0x7f525c15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c0cdc20 0x7f525c15d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 -- 172.21.15.32:0/1011472738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c156550 con 0x7f525c0af1a0 2024-03-06T07:29:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5262400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c0cdc20 0x7f525c15d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5261a00640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 0x7f525c1549d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f525000d830 tx=0x7f525000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.590+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525000b6c0 con 0x7f525c0af1a0 2024-03-06T07:29:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.591+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250019070 con 0x7f525c0af1a0 2024-03-06T07:29:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.591+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525000ba80 con 0x7f525c0af1a0 2024-03-06T07:29:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.591+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c156830 con 0x7f525c0af1a0 2024-03-06T07:29:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.591+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c09b850 con 0x7f525c0af1a0 2024-03-06T07:29:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.592+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c069510 con 0x7f525c0af1a0 2024-03-06T07:29:32.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.593+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f525001e460 con 0x7f525c0af1a0 2024-03-06T07:29:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.593+0000 7f5259600640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.594+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f525008a9a0 con 0x7f525c0af1a0 2024-03-06T07:29:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.594+0000 7f5261000640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.594+0000 7f5261000640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f525c1f4e10 tx=0x7f5248009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.598+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5250059780 con 0x7f525c0af1a0 2024-03-06T07:29:32.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.801+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f525c04f3a0 con 0x7f525c0af1a0 2024-03-06T07:29:32.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.804+0000 7f5259600640 1 -- 172.21.15.32:0/1011472738 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f5250059080 con 0x7f525c0af1a0 2024-03-06T07:29:32.806 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:32.807 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 msgr2=0x7f523c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f525c1f4e10 tx=0x7f5248009040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 msgr2=0x7f525c1549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 0x7f525c1549d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f525000d830 tx=0x7f525000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 shutdown_connections 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f523c06b4d0 0x7f523c06d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f525c0cdc20 0x7f525c15d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f525c0af580 0x7f525c155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 --2- 172.21.15.32:0/1011472738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f525c0af1a0 0x7f525c1549d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 >> 172.21.15.32:0/1011472738 conn(0x7f525c06afb0 msgr2=0x7f525c0ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 shutdown_connections 2024-03-06T07:29:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:32.807+0000 7f5264400640 1 -- 172.21.15.32:0/1011472738 wait complete. 2024-03-06T07:29:32.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:32.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.057+0000 7f8f01400640 1 Processor -- start 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.058+0000 7f8f01400640 1 -- start start 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.059+0000 7f8f01400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8efc0d1a80 0x7f8efc0d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.059+0000 7f8f01400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8efc0d2810 0x7f8efc0cb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.059+0000 7f8f01400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc0cf670 con 0x7f8efc1615b0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.059+0000 7f8f01400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc161990 con 0x7f8efc0d1a80 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.059+0000 7f8f01400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0d1ff0 con 0x7f8efc0d2810 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8efa800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8efc1615b0 0x7f8efc0d47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43658/0 (socket says 172.21.15.32:43658) 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8efa800640 1 -- 172.21.15.32:0/1506619926 learned_addr learned my addr 172.21.15.32:0/1506619926 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9e00640 1 --2- 172.21.15.32:0/1506619926 >> v2:172.21.15.32:3300/0 conn(0x7f8efc0d1a80 0x7f8efc0d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8efb200640 1 --2- 172.21.15.32:0/1506619926 >> v2:172.21.15.39:3300/0 conn(0x7f8efc0d2810 0x7f8efc0cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9400640 1 -- 172.21.15.32:0/1506619926 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268134584 0 0) 0x7f8efc0cf670 con 0x7f8efc1615b0 2024-03-06T07:29:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9400640 1 -- 172.21.15.32:0/1506619926 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003520 con 0x7f8efc1615b0 2024-03-06T07:29:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9e00640 1 -- 172.21.15.32:0/1506619926 >> v2:172.21.15.39:3300/0 conn(0x7f8efc0d2810 msgr2=0x7f8efc0cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9e00640 1 --2- 172.21.15.32:0/1506619926 >> v2:172.21.15.39:3300/0 conn(0x7f8efc0d2810 0x7f8efc0cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9e00640 1 -- 172.21.15.32:0/1506619926 >> v1:172.21.15.32:6789/0 conn(0x7f8efc1615b0 legacy=0x7f8efc0d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.060+0000 7f8ef9e00640 1 -- 172.21.15.32:0/1506619926 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc0cf910 con 0x7f8efc0d1a80 2024-03-06T07:29:33.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.062+0000 7f8ef9e00640 1 --2- 172.21.15.32:0/1506619926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d1a80 0x7f8efc0d22d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0d2fe0 tx=0x7f8ee802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0dda06c892aa779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.062+0000 7f8ef9400640 1 -- 172.21.15.32:0/1506619926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8035020 con 0x7f8efc0d1a80 2024-03-06T07:29:33.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.062+0000 7f8ef9400640 1 -- 172.21.15.32:0/1506619926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee8031830 con 0x7f8efc0d1a80 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.062+0000 7f8ef9400640 1 -- 172.21.15.32:0/1506619926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8031be0 con 0x7f8efc0d1a80 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 -- 172.21.15.32:0/1506619926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d1a80 msgr2=0x7f8efc0d22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 --2- 172.21.15.32:0/1506619926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d1a80 0x7f8efc0d22d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0d2fe0 tx=0x7f8ee802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 -- 172.21.15.32:0/1506619926 shutdown_connections 2024-03-06T07:29:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 --2- 172.21.15.32:0/1506619926 >> v2:172.21.15.39:3300/0 conn(0x7f8efc0d2810 0x7f8efc0cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 --2- 172.21.15.32:0/1506619926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d1a80 0x7f8efc0d22d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 -- 172.21.15.32:0/1506619926 >> 172.21.15.32:0/1506619926 conn(0x7f8efc09d180 msgr2=0x7f8efc0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 -- 172.21.15.32:0/1506619926 shutdown_connections 2024-03-06T07:29:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.063+0000 7f8f01400640 1 -- 172.21.15.32:0/1506619926 wait complete. 2024-03-06T07:29:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.065+0000 7f8f01400640 1 Processor -- start 2024-03-06T07:29:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.065+0000 7f8f01400640 1 -- start start 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8f01400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0d1a80 0x7f8efc15dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8f01400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 0x7f8efc0b9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8f01400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8efc1615b0 0x7f8efc1e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8f01400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0d2e20 con 0x7f8efc0d2810 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8f01400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0d2ca0 con 0x7f8efc0d1a80 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8f01400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0cb670 con 0x7f8efc1615b0 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8ef9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 0x7f8efc0b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8ef9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 0x7f8efc0b9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36020/0 (socket says 172.21.15.32:36020) 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8ef9e00640 1 -- 172.21.15.32:0/2891383794 learned_addr learned my addr 172.21.15.32:0/2891383794 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.066+0000 7f8efa800640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0d1a80 0x7f8efc15dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8efb200640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8efc1615b0 0x7f8efc1e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef9e00640 1 -- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8efc1615b0 msgr2=0x7f8efc1e83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef9e00640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8efc1615b0 0x7f8efc1e83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef9e00640 1 -- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0d1a80 msgr2=0x7f8efc15dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef9e00640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0d1a80 0x7f8efc15dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef9e00640 1 -- 172.21.15.32:0/2891383794 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc1e69c0 con 0x7f8efc0d2810 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8efb200640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8efc1615b0 0x7f8efc1e83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef9e00640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 0x7f8efc0b9230 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ee802b910 tx=0x7f8ee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.067+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8035040 con 0x7f8efc0d2810 2024-03-06T07:29:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.068+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee803b860 con 0x7f8efc0d2810 2024-03-06T07:29:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.068+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc1e6ca0 con 0x7f8efc0d2810 2024-03-06T07:29:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.068+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee803bc50 con 0x7f8efc0d2810 2024-03-06T07:29:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.068+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc1e7260 con 0x7f8efc0d2810 2024-03-06T07:29:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.069+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc04f330 con 0x7f8efc0d2810 2024-03-06T07:29:33.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.070+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ee8038020 con 0x7f8efc0d2810 2024-03-06T07:29:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.071+0000 7f8ef2a00640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.071+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ee80ac390 con 0x7f8efc0d2810 2024-03-06T07:29:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.072+0000 7f8efa800640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.072+0000 7f8efa800640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8edc06b4d0 0x7f8edc06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8eec005880 tx=0x7f8eec009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.076+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ee807b170 con 0x7f8efc0d2810 2024-03-06T07:29:33.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.278+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8efc06c700 con 0x7f8efc0d2810 2024-03-06T07:29:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.279+0000 7f8ef2a00640 1 -- 172.21.15.32:0/2891383794 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ee807aa70 con 0x7f8efc0d2810 2024-03-06T07:29:33.282 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8edc06b4d0 msgr2=0x7f8edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8edc06b4d0 0x7f8edc06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8eec005880 tx=0x7f8eec009040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 msgr2=0x7f8efc0b9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 0x7f8efc0b9230 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ee802b910 tx=0x7f8ee80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 shutdown_connections 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8efc1615b0 0x7f8efc1e83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8efc0d2810 0x7f8efc0b9230 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 --2- 172.21.15.32:0/2891383794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0d1a80 0x7f8efc15dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 >> 172.21.15.32:0/2891383794 conn(0x7f8efc09d180 msgr2=0x7f8efc06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 shutdown_connections 2024-03-06T07:29:33.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.282+0000 7f8f01400640 1 -- 172.21.15.32:0/2891383794 wait complete. 2024-03-06T07:29:33.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.522+0000 7fb805800640 1 Processor -- start 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.522+0000 7fb805800640 1 -- start start 2024-03-06T07:29:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.524+0000 7fb805800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb8000cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.524+0000 7fb805800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb800071e20 0x7fb8000ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.524+0000 7fb805800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb800072df0 con 0x7fb80015bf20 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.524+0000 7fb805800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8000cdab0 con 0x7fb800071e20 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.524+0000 7fb805800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8000b4d70 con 0x7fb800072520 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.525+0000 7fb7ff400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb800071e20 0x7fb8000ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.525+0000 7fb7f6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb8000cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.525+0000 7fb7f6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb8000cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:34674/0 (socket says 172.21.15.32:34674) 2024-03-06T07:29:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.525+0000 7fb7ff400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb800071e20 0x7fb8000ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:58898/0 (socket says 172.21.15.32:58898) 2024-03-06T07:29:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.525+0000 7fb7f6000640 1 -- 172.21.15.32:0/1533139812 learned_addr learned my addr 172.21.15.32:0/1533139812 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.526+0000 7fb7ff400640 1 -- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 msgr2=0x7fb8000cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.526+0000 7fb7ff400640 1 --2- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb8000cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.526+0000 7fb7ff400640 1 -- 172.21.15.32:0/1533139812 >> v1:172.21.15.39:6789/0 conn(0x7fb80015bf20 legacy=0x7fb8000acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.526+0000 7fb7ff400640 1 -- 172.21.15.32:0/1533139812 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8000b5050 con 0x7fb800071e20 2024-03-06T07:29:33.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.527+0000 7fb7ff400640 1 --2- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb8000ce4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80097a0 tx=0x7fb7f802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4373b34eb4344bfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.528+0000 7fb7fe000640 1 -- 172.21.15.32:0/1533139812 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8035020 con 0x7fb800071e20 2024-03-06T07:29:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.528+0000 7fb7fe000640 1 -- 172.21.15.32:0/1533139812 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8031810 con 0x7fb800071e20 2024-03-06T07:29:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.528+0000 7fb7fe000640 1 -- 172.21.15.32:0/1533139812 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8031be0 con 0x7fb800071e20 2024-03-06T07:29:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 -- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 msgr2=0x7fb8000ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 --2- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb8000ce4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80097a0 tx=0x7fb7f802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 -- 172.21.15.32:0/1533139812 shutdown_connections 2024-03-06T07:29:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 --2- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb8000ce4f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 --2- 172.21.15.32:0/1533139812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb8000cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 -- 172.21.15.32:0/1533139812 >> 172.21.15.32:0/1533139812 conn(0x7fb80006b3d0 msgr2=0x7fb80006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 -- 172.21.15.32:0/1533139812 shutdown_connections 2024-03-06T07:29:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.529+0000 7fb805800640 1 -- 172.21.15.32:0/1533139812 wait complete. 2024-03-06T07:29:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.531+0000 7fb805800640 1 Processor -- start 2024-03-06T07:29:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.531+0000 7fb805800640 1 -- start start 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb805800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb800156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb805800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb800143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb805800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 0x7fb80014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb805800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8000ad450 con 0x7fb80015bf20 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb805800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8000b4990 con 0x7fb800071e20 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb805800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80014b130 con 0x7fb800072520 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 0x7fb80014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 0x7fb80014d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36022/0 (socket says 172.21.15.32:36022) 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 -- 172.21.15.32:0/702346159 learned_addr learned my addr 172.21.15.32:0/702346159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7fea00640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb800156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 -- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 msgr2=0x7fb800143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb800143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 -- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 msgr2=0x7fb800156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb800156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7ff400640 1 -- 172.21.15.32:0/702346159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800143ea0 con 0x7fb80015bf20 2024-03-06T07:29:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.532+0000 7fb7fea00640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb800156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.533+0000 7fb7ff400640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 0x7fb80014d7f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098c0 tx=0x7fb7f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.533+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8035040 con 0x7fb80015bf20 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.533+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8039e10 con 0x7fb80015bf20 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.533+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80456c0 con 0x7fb80015bf20 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.533+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb800144180 con 0x7fb80015bf20 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.533+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80014b380 con 0x7fb80015bf20 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.535+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8000cc4a0 con 0x7fb80015bf20 2024-03-06T07:29:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.536+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7f8038020 con 0x7fb80015bf20 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.537+0000 7fb7f7400640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.537+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7f80ac2a0 con 0x7fb80015bf20 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.537+0000 7fb7fea00640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.538+0000 7fb7fea00640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0001d70 tx=0x7fb7f0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.541+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8000cc4a0 con 0x7fb80015bf20 2024-03-06T07:29:33.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.740+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb80004f330 con 0x7fb80015bf20 2024-03-06T07:29:33.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.741+0000 7fb7f7400640 1 -- 172.21.15.32:0/702346159 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7f807b110 con 0x7fb80015bf20 2024-03-06T07:29:33.744 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.743+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7dc06b4d0 msgr2=0x7fb7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.743+0000 7fb805800640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0001d70 tx=0x7fb7f0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.743+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 msgr2=0x7fb80014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.743+0000 7fb805800640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 0x7fb80014d7f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098c0 tx=0x7fb7f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 shutdown_connections 2024-03-06T07:29:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7dc06b4d0 0x7fb7dc06d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80015bf20 0x7fb80014d7f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb800072520 0x7fb800143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 --2- 172.21.15.32:0/702346159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb800071e20 0x7fb800156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 >> 172.21.15.32:0/702346159 conn(0x7fb80006b3d0 msgr2=0x7fb80015adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 shutdown_connections 2024-03-06T07:29:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.744+0000 7fb805800640 1 -- 172.21.15.32:0/702346159 wait complete. 2024-03-06T07:29:33.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:33.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.974+0000 7fc9c3e00640 1 Processor -- start 2024-03-06T07:29:33.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.975+0000 7fc9c3e00640 1 -- start start 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.976+0000 7fc9c3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc9c40d4860 0x7fc9c40d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.976+0000 7fc9c3e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40ceca0 con 0x7fc9c415dee0 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.976+0000 7fc9c3e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40cef80 con 0x7fc9c40cfa60 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.976+0000 7fc9c3e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40cf5d0 con 0x7fc9c40d4860 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.977+0000 7fc9c2000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc9c40d4860 0x7fc9c40d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.977+0000 7fc9c2a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc9c415dee0 0x7fc9c40d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43670/0 (socket says 172.21.15.32:43670) 2024-03-06T07:29:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.977+0000 7fc9c2a00640 1 -- 172.21.15.32:0/375169934 learned_addr learned my addr 172.21.15.32:0/375169934 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.977+0000 7fc9c2000640 1 -- 172.21.15.32:0/375169934 >> v1:172.21.15.39:6789/0 conn(0x7fc9c40cfa60 legacy=0x7fc9c40b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.977+0000 7fc9c2000640 1 -- 172.21.15.32:0/375169934 >> v1:172.21.15.32:6789/0 conn(0x7fc9c415dee0 legacy=0x7fc9c40d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.977+0000 7fc9c2000640 1 -- 172.21.15.32:0/375169934 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40cf260 con 0x7fc9c40d4860 2024-03-06T07:29:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.978+0000 7fc9c2000640 1 --2- 172.21.15.32:0/375169934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40d4860 0x7fc9c40d0630 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40cf750 tx=0x7fc9ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8bf475a8907fda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c1600640 1 -- 172.21.15.32:0/375169934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac039070 con 0x7fc9c40d4860 2024-03-06T07:29:33.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c1600640 1 -- 172.21.15.32:0/375169934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac02cc60 con 0x7fc9c40d4860 2024-03-06T07:29:33.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c1600640 1 -- 172.21.15.32:0/375169934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac034660 con 0x7fc9c40d4860 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c3e00640 1 -- 172.21.15.32:0/375169934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40d4860 msgr2=0x7fc9c40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/375169934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40d4860 0x7fc9c40d0630 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40cf750 tx=0x7fc9ac02beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c3e00640 1 -- 172.21.15.32:0/375169934 shutdown_connections 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/375169934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40d4860 0x7fc9c40d0630 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.979+0000 7fc9c3e00640 1 -- 172.21.15.32:0/375169934 >> 172.21.15.32:0/375169934 conn(0x7fc9c40d3cd0 msgr2=0x7fc9c40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.980+0000 7fc9c3e00640 1 -- 172.21.15.32:0/375169934 shutdown_connections 2024-03-06T07:29:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.980+0000 7fc9c3e00640 1 -- 172.21.15.32:0/375169934 wait complete. 2024-03-06T07:29:33.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.981+0000 7fc9c3e00640 1 Processor -- start 2024-03-06T07:29:33.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.982+0000 7fc9c3e00640 1 -- start start 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.982+0000 7fc9c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 0x7fc9c40b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.982+0000 7fc9c3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c415dee0 0x7fc9c4159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9c40ca510 0x7fc9c40b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40d0b70 con 0x7fc9c40cfa60 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c415e2c0 con 0x7fc9c415dee0 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c3e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40d0cf0 con 0x7fc9c40ca510 2024-03-06T07:29:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 0x7fc9c40b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 0x7fc9c40b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36036/0 (socket says 172.21.15.32:36036) 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 -- 172.21.15.32:0/3272506356 learned_addr learned my addr 172.21.15.32:0/3272506356 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2000640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c415dee0 0x7fc9c4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c3400640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9c40ca510 0x7fc9c40b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 -- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9c40ca510 msgr2=0x7fc9c40b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9c40ca510 0x7fc9c40b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 -- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c415dee0 msgr2=0x7fc9c4159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c415dee0 0x7fc9c4159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.983+0000 7fc9c2a00640 1 -- 172.21.15.32:0/3272506356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c4159b50 con 0x7fc9c40cfa60 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.984+0000 7fc9c2a00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 0x7fc9c40b4140 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400c3c0 tx=0x7fc9b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.984+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4017070 con 0x7fc9c40cfa60 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.984+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4005ce0 con 0x7fc9c40cfa60 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.984+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40072f0 con 0x7fc9c40cfa60 2024-03-06T07:29:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.984+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c40b4680 con 0x7fc9c40cfa60 2024-03-06T07:29:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.984+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c40c7f70 con 0x7fc9c40cfa60 2024-03-06T07:29:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.985+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c4072bd0 con 0x7fc9c40cfa60 2024-03-06T07:29:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.986+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9b401a070 con 0x7fc9c40cfa60 2024-03-06T07:29:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.987+0000 7fc9baa00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9a006b500 0x7fc9a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.988+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9b4089570 con 0x7fc9c40cfa60 2024-03-06T07:29:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.988+0000 7fc9c2000640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9a006b500 0x7fc9a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.988+0000 7fc9c2000640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9a006b500 0x7fc9a006d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac036000 tx=0x7fc9ac030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:33.992+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9c4072bd0 con 0x7fc9c40cfa60 2024-03-06T07:29:34.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.189+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc9c40b5210 con 0x7fc9c40cfa60 2024-03-06T07:29:34.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.190+0000 7fc9baa00640 1 -- 172.21.15.32:0/3272506356 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc9c40b5210 con 0x7fc9c40cfa60 2024-03-06T07:29:34.193 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9a006b500 msgr2=0x7fc9a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9a006b500 0x7fc9a006d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac036000 tx=0x7fc9ac030000 comp rx=0 tx=0).stop 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 msgr2=0x7fc9c40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 0x7fc9c40b4140 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400c3c0 tx=0x7fc9b400c880 comp rx=0 tx=0).stop 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 shutdown_connections 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9a006b500 0x7fc9a006d9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9c40ca510 0x7fc9c40b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9c415dee0 0x7fc9c4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 --2- 172.21.15.32:0/3272506356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9c40cfa60 0x7fc9c40b4140 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.193+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 >> 172.21.15.32:0/3272506356 conn(0x7fc9c40d3cd0 msgr2=0x7fc9c415a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.194+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 shutdown_connections 2024-03-06T07:29:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.194+0000 7fc9c3e00640 1 -- 172.21.15.32:0/3272506356 wait complete. 2024-03-06T07:29:34.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:34.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.426+0000 7f7b2f600640 1 Processor -- start 2024-03-06T07:29:34.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.426+0000 7f7b2f600640 1 -- start start 2024-03-06T07:29:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b2f600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7b280af580 0x7f7b2815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b2f600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2815cc40 con 0x7f7b280af1a0 2024-03-06T07:29:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b2f600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2815cf20 con 0x7f7b280cdc20 2024-03-06T07:29:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b2f600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2815d200 con 0x7f7b280af580 2024-03-06T07:29:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b2cc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7b280af1a0 0x7f7b280b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43682/0 (socket says 172.21.15.32:43682) 2024-03-06T07:29:34.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b2cc00640 1 -- 172.21.15.32:0/633603617 learned_addr learned my addr 172.21.15.32:0/633603617 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:34.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b27e00640 1 --2- 172.21.15.32:0/633603617 >> v2:172.21.15.39:3300/0 conn(0x7f7b280af580 0x7f7b2815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.428+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838397345 0 0) 0x7f7b2815cc40 con 0x7f7b280af1a0 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b10003520 con 0x7f7b280af1a0 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642518902 0 0) 0x7f7b10003520 con 0x7f7b280af1a0 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 >> v2:172.21.15.39:3300/0 conn(0x7f7b280af580 msgr2=0x7f7b2815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 --2- 172.21.15.32:0/633603617 >> v2:172.21.15.39:3300/0 conn(0x7f7b280af580 0x7f7b2815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 >> v1:172.21.15.39:6789/0 conn(0x7f7b280cdc20 legacy=0x7f7b280b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2815d4e0 con 0x7f7b280af1a0 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f7b1c001670 con 0x7f7b280af1a0 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 >> v1:172.21.15.32:6789/0 conn(0x7f7b280af1a0 legacy=0x7f7b280b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b10003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b10004800 0x7f7b10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b10005230 0x7f7b1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815d200 con 0x7f7b10003a20 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1000c190 con 0x7f7b10005230 2024-03-06T07:29:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1000c390 con 0x7f7b10004800 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.429+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b10003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2d600640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b10005230 0x7f7b1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2cc00640 1 -- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b10004800 msgr2=0x7f7b10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b10004800 0x7f7b10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2cc00640 1 -- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b10005230 msgr2=0x7f7b1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b10005230 0x7f7b1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2cc00640 1 -- 172.21.15.32:0/633603617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10003520 con 0x7f7b10003a20 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2d600640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b10005230 0x7f7b1000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.430+0000 7f7b2d600640 1 -- 172.21.15.32:0/633603617 reap_dead start 2024-03-06T07:29:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.431+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b10003e40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c210 tx=0x7f7b10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9829942142d45dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.431+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c007d30 con 0x7f7b10003a20 2024-03-06T07:29:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.431+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c030070 con 0x7f7b10003a20 2024-03-06T07:29:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.431+0000 7f7b27400640 1 -- 172.21.15.32:0/633603617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c005d70 con 0x7f7b10003a20 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 -- 172.21.15.32:0/633603617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 msgr2=0x7f7b10003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b10003e40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c210 tx=0x7f7b10005830 comp rx=0 tx=0).stop 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 -- 172.21.15.32:0/633603617 shutdown_connections 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 --2- 172.21.15.32:0/633603617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b10003e40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 -- 172.21.15.32:0/633603617 >> 172.21.15.32:0/633603617 conn(0x7f7b2806afb0 msgr2=0x7f7b2806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 -- 172.21.15.32:0/633603617 shutdown_connections 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.432+0000 7f7b2f600640 1 -- 172.21.15.32:0/633603617 wait complete. 2024-03-06T07:29:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.434+0000 7f7b2f600640 1 Processor -- start 2024-03-06T07:29:34.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.434+0000 7f7b2f600640 1 -- start start 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b2814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2f600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b2814eb20 0x7f7b2809ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b2806cb10 0x7f7b281588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2f600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815c990 con 0x7f7b10003a20 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815c810 con 0x7f7b2806cb10 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2f600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b4c20 con 0x7f7b2814eb20 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.435+0000 7f7b2cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b2814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b2814e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36062/0 (socket says 172.21.15.32:36062) 2024-03-06T07:29:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 -- 172.21.15.32:0/1264328695 learned_addr learned my addr 172.21.15.32:0/1264328695 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:34.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2d600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b2806cb10 0x7f7b281588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 -- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b2814eb20 msgr2=0x7f7b2809ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b2814eb20 0x7f7b2809ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 -- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b2806cb10 msgr2=0x7f7b281588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b2806cb10 0x7f7b281588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 -- 172.21.15.32:0/1264328695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2809b2e0 con 0x7f7b10003a20 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2d600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b2806cb10 0x7f7b281588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.436+0000 7f7b2cc00640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b2814e3b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c210 tx=0x7f7b1c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.437+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c004bf0 con 0x7f7b10003a20 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.437+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c004d90 con 0x7f7b10003a20 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.437+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c007270 con 0x7f7b10003a20 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.437+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b28157310 con 0x7f7b10003a20 2024-03-06T07:29:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.437+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b281578d0 con 0x7f7b10003a20 2024-03-06T07:29:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.438+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28069510 con 0x7f7b10003a20 2024-03-06T07:29:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.439+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b1c048020 con 0x7f7b10003a20 2024-03-06T07:29:34.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.440+0000 7f7b24c00640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b0806b500 0x7f7b0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.440+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b1c0ab730 con 0x7f7b10003a20 2024-03-06T07:29:34.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.440+0000 7f7b27e00640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b0806b500 0x7f7b0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.441+0000 7f7b27e00640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b0806b500 0x7f7b0806d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b140093d0 tx=0x7f7b14009300 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.444+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b1c07a5a0 con 0x7f7b10003a20 2024-03-06T07:29:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.645+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b280b4fb0 con 0x7f7b10003a20 2024-03-06T07:29:34.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.647+0000 7f7b24c00640 1 -- 172.21.15.32:0/1264328695 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f7b280b4fb0 con 0x7f7b10003a20 2024-03-06T07:29:34.650 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:34.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b0806b500 msgr2=0x7f7b0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b0806b500 0x7f7b0806d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b140093d0 tx=0x7f7b14009300 comp rx=0 tx=0).stop 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 msgr2=0x7f7b2814e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b2814e3b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c210 tx=0x7f7b1c02e070 comp rx=0 tx=0).stop 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 shutdown_connections 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b0806b500 0x7f7b0806d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b2806cb10 0x7f7b281588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b2814eb20 0x7f7b2809ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 --2- 172.21.15.32:0/1264328695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10003a20 0x7f7b2814e3b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 >> 172.21.15.32:0/1264328695 conn(0x7f7b2806afb0 msgr2=0x7f7b280b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 shutdown_connections 2024-03-06T07:29:34.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.650+0000 7f7b2f600640 1 -- 172.21.15.32:0/1264328695 wait complete. 2024-03-06T07:29:34.672 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:34.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.885+0000 7f2735800640 1 Processor -- start 2024-03-06T07:29:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.885+0000 7f2735800640 1 -- start start 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.886+0000 7f2735800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f27300d4400 0x7f27300d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.886+0000 7f2735800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273015f360 con 0x7f27300d0430 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.886+0000 7f2735800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27300ce800 con 0x7f27300d18c0 2024-03-06T07:29:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.886+0000 7f2735800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27300cea80 con 0x7f27300d4400 2024-03-06T07:29:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.886+0000 7f272fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27300d0430 0x7f27300d0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43690/0 (socket says 172.21.15.32:43690) 2024-03-06T07:29:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.886+0000 7f272fe00640 1 -- 172.21.15.32:0/1812067591 learned_addr learned my addr 172.21.15.32:0/1812067591 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272f400640 1 --2- 172.21.15.32:0/1812067591 >> v2:172.21.15.39:3300/0 conn(0x7f27300d4400 0x7f27300d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989178807 0 0) 0x7f273015f360 con 0x7f27300d0430 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2714003520 con 0x7f27300d0430 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1639823465 0 0) 0x7f2714003520 con 0x7f27300d0430 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 >> v2:172.21.15.39:3300/0 conn(0x7f27300d4400 msgr2=0x7f27300d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 --2- 172.21.15.32:0/1812067591 >> v2:172.21.15.39:3300/0 conn(0x7f27300d4400 0x7f27300d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 >> v1:172.21.15.39:6789/0 conn(0x7f27300d18c0 legacy=0x7f27300d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27300ced60 con 0x7f27300d0430 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f2720001670 con 0x7f27300d0430 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 >> v1:172.21.15.32:6789/0 conn(0x7f27300d0430 legacy=0x7f27300d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2714003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2714004800 0x7f2714004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.887+0000 7f272ea00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27140051c0 0x7f271400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27300cea80 con 0x7f27140039b0 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271400c060 con 0x7f27140051c0 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271400c260 con 0x7f2714004800 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272fe00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2714003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272f400640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2714004800 0x7f2714004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272fe00640 1 -- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2714004800 msgr2=0x7f2714004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272fe00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2714004800 0x7f2714004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272fe00640 1 -- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27140051c0 msgr2=0x7f271400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272fe00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27140051c0 0x7f271400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f272fe00640 1 -- 172.21.15.32:0/1812067591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2714003520 con 0x7f27140039b0 2024-03-06T07:29:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.888+0000 7f2734e00640 1 -- 172.21.15.32:0/1812067591 reap_dead start 2024-03-06T07:29:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.889+0000 7f272fe00640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2714003dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f272000c210 tx=0x7f2714005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54edeea24e0e465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720007d30 con 0x7f27140039b0 2024-03-06T07:29:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720030070 con 0x7f27140039b0 2024-03-06T07:29:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f272ea00640 1 -- 172.21.15.32:0/1812067591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272002cda0 con 0x7f27140039b0 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 -- 172.21.15.32:0/1812067591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 msgr2=0x7f2714003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2714003dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f272000c210 tx=0x7f2714005830 comp rx=0 tx=0).stop 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 -- 172.21.15.32:0/1812067591 shutdown_connections 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 --2- 172.21.15.32:0/1812067591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2714003dd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 -- 172.21.15.32:0/1812067591 >> 172.21.15.32:0/1812067591 conn(0x7f273008d2a0 msgr2=0x7f27300d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 -- 172.21.15.32:0/1812067591 shutdown_connections 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.890+0000 7f2735800640 1 -- 172.21.15.32:0/1812067591 wait complete. 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.892+0000 7f2735800640 1 Processor -- start 2024-03-06T07:29:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.892+0000 7f2735800640 1 -- start start 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.893+0000 7f2735800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2730152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.893+0000 7f2735800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27301f4550 0x7f27300cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.893+0000 7f2735800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f27301f49d0 0x7f27300cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.893+0000 7f2735800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27300cf600 con 0x7f27140039b0 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.893+0000 7f2735800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27300cf320 con 0x7f27301f4550 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.893+0000 7f2735800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27300cfb20 con 0x7f27301f49d0 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2730152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2730152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36076/0 (socket says 172.21.15.32:36076) 2024-03-06T07:29:34.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 -- 172.21.15.32:0/817502939 learned_addr learned my addr 172.21.15.32:0/817502939 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272f400640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27301f4550 0x7f27300cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f2734e00640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f27301f49d0 0x7f27300cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 -- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f27301f49d0 msgr2=0x7f27300cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f27301f49d0 0x7f27300cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 -- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27301f4550 msgr2=0x7f27300cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27301f4550 0x7f27300cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272fe00640 1 -- 172.21.15.32:0/817502939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27300c86e0 con 0x7f27140039b0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f272f400640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27301f4550 0x7f27300cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.894+0000 7f2734e00640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f27301f49d0 0x7f27300cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.895+0000 7f272fe00640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2730152b00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f272000c210 tx=0x7f272002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.895+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720045070 con 0x7f27140039b0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.895+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720000ed0 con 0x7f27140039b0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.895+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720004e50 con 0x7f27140039b0 2024-03-06T07:29:34.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.895+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27300c89c0 con 0x7f27140039b0 2024-03-06T07:29:34.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.895+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27300c8f80 con 0x7f27140039b0 2024-03-06T07:29:34.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.896+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273004f330 con 0x7f27140039b0 2024-03-06T07:29:34.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.897+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2720007070 con 0x7f27140039b0 2024-03-06T07:29:34.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.899+0000 7f2727e00640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f271006b6a0 0x7f271006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.899+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27200ab220 con 0x7f27140039b0 2024-03-06T07:29:34.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.899+0000 7f272f400640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f271006b6a0 0x7f271006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.900+0000 7f272f400640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f271006b6a0 0x7f271006db60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f27300b29e0 tx=0x7f271c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:34.903+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2720079a40 con 0x7f27140039b0 2024-03-06T07:29:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.104+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27300d3ad0 con 0x7f27140039b0 2024-03-06T07:29:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.105+0000 7f2727e00640 1 -- 172.21.15.32:0/817502939 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f272007dae0 con 0x7f27140039b0 2024-03-06T07:29:35.108 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f271006b6a0 msgr2=0x7f271006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f271006b6a0 0x7f271006db60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f27300b29e0 tx=0x7f271c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 msgr2=0x7f2730152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2730152b00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f272000c210 tx=0x7f272002e070 comp rx=0 tx=0).stop 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 shutdown_connections 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f271006b6a0 0x7f271006db60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f27301f49d0 0x7f27300cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27301f4550 0x7f27300cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 --2- 172.21.15.32:0/817502939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27140039b0 0x7f2730152b00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 >> 172.21.15.32:0/817502939 conn(0x7f273008d2a0 msgr2=0x7f273015f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 shutdown_connections 2024-03-06T07:29:35.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.108+0000 7f2735800640 1 -- 172.21.15.32:0/817502939 wait complete. 2024-03-06T07:29:35.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.351+0000 7fe462600640 1 Processor -- start 2024-03-06T07:29:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.352+0000 7fe462600640 1 -- start start 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.353+0000 7fe462600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 0x7fe45c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.353+0000 7fe462600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe45c0b2080 0x7fe45c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.353+0000 7fe462600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c15fe40 con 0x7fe45c145430 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.353+0000 7fe462600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c160120 con 0x7fe45c0b2080 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.353+0000 7fe462600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c160400 con 0x7fe45c15f810 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.354+0000 7fe461200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe45c145430 0x7fe45c15cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43692/0 (socket says 172.21.15.32:43692) 2024-03-06T07:29:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.354+0000 7fe461200640 1 -- 172.21.15.32:0/831667068 learned_addr learned my addr 172.21.15.32:0/831667068 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:35.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.354+0000 7fe461c00640 1 --2- 172.21.15.32:0/831667068 >> v2:172.21.15.39:3300/0 conn(0x7fe45c0b2080 0x7fe45c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.354+0000 7fe45be00640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 0x7fe45c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.354+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142534048 0 0) 0x7fe45c15fe40 con 0x7fe45c145430 2024-03-06T07:29:35.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe440003520 con 0x7fe45c145430 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1563432765 0 0) 0x7fe440003520 con 0x7fe45c145430 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 msgr2=0x7fe45c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 0x7fe45c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 >> v2:172.21.15.39:3300/0 conn(0x7fe45c0b2080 msgr2=0x7fe45c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 --2- 172.21.15.32:0/831667068 >> v2:172.21.15.39:3300/0 conn(0x7fe45c0b2080 0x7fe45c0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c1606e0 con 0x7fe45c145430 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fe44c0019f0 con 0x7fe45c145430 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 >> v1:172.21.15.32:6789/0 conn(0x7fe45c145430 legacy=0x7fe45c15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4400039b0 0x7fe440003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 0x7fe440004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe440004d40 0x7fe4400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c160120 con 0x7fe440004d40 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c160400 con 0x7fe4400039b0 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44000b9c0 con 0x7fe45c15f810 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.355+0000 7fe461c00640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe440004d40 0x7fe4400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.356+0000 7fe461200640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4400039b0 0x7fe440003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.356+0000 7fe461c00640 1 -- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 msgr2=0x7fe440004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.356+0000 7fe461c00640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c15f810 0x7fe440004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.356+0000 7fe461c00640 1 -- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4400039b0 msgr2=0x7fe440003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.356+0000 7fe461c00640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4400039b0 0x7fe440003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.356+0000 7fe461c00640 1 -- 172.21.15.32:0/831667068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe440003520 con 0x7fe440004d40 2024-03-06T07:29:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.357+0000 7fe461c00640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe440004d40 0x7fe4400051c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe454009ac0 tx=0x7fe440005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92553b09e9db9089 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.357+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454039070 con 0x7fe440004d40 2024-03-06T07:29:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.357+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45402c710 con 0x7fe440004d40 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.357+0000 7fe45b400640 1 -- 172.21.15.32:0/831667068 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454009690 con 0x7fe440004d40 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe462600640 1 -- 172.21.15.32:0/831667068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe440004d40 msgr2=0x7fe4400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe462600640 1 --2- 172.21.15.32:0/831667068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe440004d40 0x7fe4400051c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe454009ac0 tx=0x7fe440005830 comp rx=0 tx=0).stop 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe461c00640 1 -- 172.21.15.32:0/831667068 reap_dead start 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe462600640 1 -- 172.21.15.32:0/831667068 shutdown_connections 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe462600640 1 -- 172.21.15.32:0/831667068 >> 172.21.15.32:0/831667068 conn(0x7fe45c06b3a0 msgr2=0x7fe45c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe462600640 1 -- 172.21.15.32:0/831667068 shutdown_connections 2024-03-06T07:29:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.358+0000 7fe462600640 1 -- 172.21.15.32:0/831667068 wait complete. 2024-03-06T07:29:35.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.360+0000 7fe462600640 1 Processor -- start 2024-03-06T07:29:35.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.360+0000 7fe462600640 1 -- start start 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe462600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe440004d40 0x7fe45c15c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe462600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 0x7fe45c0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe462600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0cb420 0x7fe45c0a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe462600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0cdf80 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe462600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0cde00 con 0x7fe440004d40 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe462600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0b1b40 con 0x7fe45c0cb420 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.361+0000 7fe45be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 0x7fe45c0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 0x7fe45c0caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36092/0 (socket says 172.21.15.32:36092) 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 -- 172.21.15.32:0/160397077 learned_addr learned my addr 172.21.15.32:0/160397077 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe461200640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe440004d40 0x7fe45c15c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe461c00640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0cb420 0x7fe45c0a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 -- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0cb420 msgr2=0x7fe45c0a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0cb420 0x7fe45c0a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 -- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe440004d40 msgr2=0x7fe45c15c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe440004d40 0x7fe45c15c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 -- 172.21.15.32:0/160397077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0aa050 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe45be00640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 0x7fe45c0caee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe44400e7f0 tx=0x7fe44400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.362+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444017070 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.363+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44400c940 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.363+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44400eea0 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.363+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c0aa330 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.363+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c0aa8f0 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.364+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45c0cb830 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.365+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe44401a070 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.366+0000 7fe458c00640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.366+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe44408a4a0 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.366+0000 7fe461200640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.367+0000 7fe461200640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe44c001330 tx=0x7fe44c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.370+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe45c0cb830 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.572+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe45c04f330 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.573+0000 7fe458c00640 1 -- 172.21.15.32:0/160397077 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe444059280 con 0x7fe45c0ab0f0 2024-03-06T07:29:35.575 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:35.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 msgr2=0x7fe43806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe44c001330 tx=0x7fe44c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 msgr2=0x7fe45c0caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 0x7fe45c0caee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe44400e7f0 tx=0x7fe44400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 shutdown_connections 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0cb420 0x7fe45c0a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0ab0f0 0x7fe45c0caee0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 --2- 172.21.15.32:0/160397077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe440004d40 0x7fe45c15c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 >> 172.21.15.32:0/160397077 conn(0x7fe45c06b3a0 msgr2=0x7fe45c1431a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 shutdown_connections 2024-03-06T07:29:35.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.576+0000 7fe462600640 1 -- 172.21.15.32:0/160397077 wait complete. 2024-03-06T07:29:35.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.821+0000 7fa4c9800640 1 Processor -- start 2024-03-06T07:29:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.821+0000 7fa4c9800640 1 -- start start 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c9800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa4c415e2d0 0x7fa4c40b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c9800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4c4167f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c9800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c40b5da0 con 0x7fa4c406cab0 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c9800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c40b6080 con 0x7fa4c415e2d0 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c9800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40b6360 con 0x7fa4c4165b60 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa4c406cab0 0x7fa4c406ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43708/0 (socket says 172.21.15.32:43708) 2024-03-06T07:29:35.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c3e00640 1 -- 172.21.15.32:0/1145552516 learned_addr learned my addr 172.21.15.32:0/1145552516 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:35.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120282720 0 0) 0x7fa4c40b5da0 con 0x7fa4c406cab0 2024-03-06T07:29:35.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.823+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a8003520 con 0x7fa4c406cab0 2024-03-06T07:29:35.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c3400640 1 --2- 172.21.15.32:0/1145552516 >> v2:172.21.15.39:3300/0 conn(0x7fa4c415e2d0 0x7fa4c40b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c8e00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4c4167f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 332339941 0 0) 0x7fa4a8003520 con 0x7fa4c406cab0 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 msgr2=0x7fa4c4167f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4c4167f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 >> v2:172.21.15.39:3300/0 conn(0x7fa4c415e2d0 msgr2=0x7fa4c40b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 --2- 172.21.15.32:0/1145552516 >> v2:172.21.15.39:3300/0 conn(0x7fa4c415e2d0 0x7fa4c40b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c40b6640 con 0x7fa4c406cab0 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa4ac001670 con 0x7fa4c406cab0 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 >> v1:172.21.15.32:6789/0 conn(0x7fa4c406cab0 legacy=0x7fa4c406ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4a80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a8004d40 0x7fa4a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40b6080 con 0x7fa4a8004030 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40b6360 con 0x7fa4a8004d40 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a800bb20 con 0x7fa4c4165b60 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.824+0000 7fa4c3400640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3e00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4a80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3400640 1 -- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 msgr2=0x7fa4a80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c8e00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a8004d40 0x7fa4a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3400640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4a80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3400640 1 -- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a8004d40 msgr2=0x7fa4a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3400640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a8004d40 0x7fa4a80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3400640 1 -- 172.21.15.32:0/1145552516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a8003520 con 0x7fa4a8004030 2024-03-06T07:29:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.825+0000 7fa4c3e00640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c4165b60 0x7fa4a80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.826+0000 7fa4c3400640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4a8004800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400ba30 tx=0x7fa4a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d232b540dc77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.826+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4036020 con 0x7fa4a8004030 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.826+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b4032680 con 0x7fa4a8004030 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.826+0000 7fa4c2a00640 1 -- 172.21.15.32:0/1145552516 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b400b600 con 0x7fa4a8004030 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c9800640 1 -- 172.21.15.32:0/1145552516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 msgr2=0x7fa4a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c9800640 1 --2- 172.21.15.32:0/1145552516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4a8004800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400ba30 tx=0x7fa4a8005830 comp rx=0 tx=0).stop 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c8e00640 1 -- 172.21.15.32:0/1145552516 reap_dead start 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c9800640 1 -- 172.21.15.32:0/1145552516 shutdown_connections 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c9800640 1 -- 172.21.15.32:0/1145552516 >> 172.21.15.32:0/1145552516 conn(0x7fa4c409aa10 msgr2=0x7fa4c409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c9800640 1 -- 172.21.15.32:0/1145552516 shutdown_connections 2024-03-06T07:29:35.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.827+0000 7fa4c9800640 1 -- 172.21.15.32:0/1145552516 wait complete. 2024-03-06T07:29:35.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.829+0000 7fa4c9800640 1 Processor -- start 2024-03-06T07:29:35.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.829+0000 7fa4c9800640 1 -- start start 2024-03-06T07:29:35.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4c41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c9800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c4160d60 0x7fa4c4153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c9800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c415b3a0 0x7fa4c415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c9800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c415e8b0 con 0x7fa4a8004030 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c9800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c415e730 con 0x7fa4c4160d60 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c9800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40b5990 con 0x7fa4c415b3a0 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4c41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4c41542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36108/0 (socket says 172.21.15.32:36108) 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c3e00640 1 -- 172.21.15.32:0/2723806456 learned_addr learned my addr 172.21.15.32:0/2723806456 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.830+0000 7fa4c3400640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c4160d60 0x7fa4c4153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c3e00640 1 -- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c415b3a0 msgr2=0x7fa4c415fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c8e00640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c415b3a0 0x7fa4c415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c3e00640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c415b3a0 0x7fa4c415fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c3e00640 1 -- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c4160d60 msgr2=0x7fa4c4153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c3e00640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c4160d60 0x7fa4c4153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c3e00640 1 -- 172.21.15.32:0/2723806456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c41573b0 con 0x7fa4a8004030 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c8e00640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c415b3a0 0x7fa4c415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c3e00640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4c41542e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00d8e0 tx=0x7fa4ac00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac019070 con 0x7fa4a8004030 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac0074e0 con 0x7fa4a8004030 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0049e0 con 0x7fa4a8004030 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c4157690 con 0x7fa4a8004030 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.831+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c415ae80 con 0x7fa4a8004030 2024-03-06T07:29:35.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.832+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c404f330 con 0x7fa4a8004030 2024-03-06T07:29:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.833+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4ac01c070 con 0x7fa4a8004030 2024-03-06T07:29:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.834+0000 7fa4bbe00640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4a006b500 0x7fa4a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.834+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4ac089e10 con 0x7fa4a8004030 2024-03-06T07:29:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.834+0000 7fa4c3400640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4a006b500 0x7fa4a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.835+0000 7fa4c3400640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4a006b500 0x7fa4a006d9c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400ba00 tx=0x7fa4b403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:35.839+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4ac08d030 con 0x7fa4a8004030 2024-03-06T07:29:36.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.046+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4c40b3270 con 0x7fa4a8004030 2024-03-06T07:29:36.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.047+0000 7fa4bbe00640 1 -- 172.21.15.32:0/2723806456 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa4ac058c80 con 0x7fa4a8004030 2024-03-06T07:29:36.049 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.049+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4a006b500 msgr2=0x7fa4a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.049+0000 7fa4c9800640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4a006b500 0x7fa4a006d9c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400ba00 tx=0x7fa4b403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.049+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 msgr2=0x7fa4c41542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.049+0000 7fa4c9800640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4c41542e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00d8e0 tx=0x7fa4ac00dda0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 shutdown_connections 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa4a006b500 0x7fa4a006d9c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4c415b3a0 0x7fa4c415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4c4160d60 0x7fa4c4153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 --2- 172.21.15.32:0/2723806456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a8004030 0x7fa4c41542e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 >> 172.21.15.32:0/2723806456 conn(0x7fa4c409aa10 msgr2=0x7fa4c4070530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 shutdown_connections 2024-03-06T07:29:36.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.050+0000 7fa4c9800640 1 -- 172.21.15.32:0/2723806456 wait complete. 2024-03-06T07:29:36.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:36.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.288+0000 7f1e38e00640 1 Processor -- start 2024-03-06T07:29:36.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.288+0000 7f1e38e00640 1 -- start start 2024-03-06T07:29:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.289+0000 7f1e38e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1e3415f450 0x7f1e34070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.289+0000 7f1e38e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e340cd0c0 con 0x7f1e340700c0 2024-03-06T07:29:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.289+0000 7f1e38e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e340cd3a0 con 0x7f1e340d04b0 2024-03-06T07:29:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.289+0000 7f1e38e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e340d29e0 con 0x7f1e3415f450 2024-03-06T07:29:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.289+0000 7f1e32a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1e340700c0 0x7f1e340cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43720/0 (socket says 172.21.15.32:43720) 2024-03-06T07:29:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.289+0000 7f1e32a00640 1 -- 172.21.15.32:0/1900662409 learned_addr learned my addr 172.21.15.32:0/1900662409 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e33400640 1 --2- 172.21.15.32:0/1900662409 >> v2:172.21.15.39:3300/0 conn(0x7f1e3415f450 0x7f1e34070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504055346 0 0) 0x7f1e340cd0c0 con 0x7f1e340700c0 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e14003520 con 0x7f1e340700c0 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628963938 0 0) 0x7f1e14003520 con 0x7f1e340700c0 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 >> v2:172.21.15.39:3300/0 conn(0x7f1e3415f450 msgr2=0x7f1e34070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 --2- 172.21.15.32:0/1900662409 >> v2:172.21.15.39:3300/0 conn(0x7f1e3415f450 0x7f1e34070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 >> v1:172.21.15.39:6789/0 conn(0x7f1e340d04b0 legacy=0x7f1e340cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e340cd680 con 0x7f1e340700c0 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f1e1c0019f0 con 0x7f1e340700c0 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 >> v1:172.21.15.32:6789/0 conn(0x7f1e340700c0 legacy=0x7f1e340cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e14003a00 0x7f1e14003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e140051c0 0x7f1e1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340d29e0 con 0x7f1e14003a00 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1400c0c0 con 0x7f1e14004800 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.290+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1400c2c0 con 0x7f1e140051c0 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33400640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e14003a00 0x7f1e14003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e32a00640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33400640 1 -- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e140051c0 msgr2=0x7f1e1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33400640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e140051c0 0x7f1e1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33400640 1 -- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e14004800 msgr2=0x7f1e14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33400640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e14004800 0x7f1e14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33400640 1 -- 172.21.15.32:0/1900662409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e14003520 con 0x7f1e14003a00 2024-03-06T07:29:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.291+0000 7f1e33e00640 1 -- 172.21.15.32:0/1900662409 reap_dead start 2024-03-06T07:29:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.292+0000 7f1e33400640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e14003a00 0x7f1e14003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009ac0 tx=0x7f1e14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57cc897c8b7bd425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24039070 con 0x7f1e14003a00 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2402c710 con 0x7f1e14003a00 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e32000640 1 -- 172.21.15.32:0/1900662409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24009690 con 0x7f1e14003a00 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 -- 172.21.15.32:0/1900662409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e14003a00 msgr2=0x7f1e14003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e14003a00 0x7f1e14003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009ac0 tx=0x7f1e14005830 comp rx=0 tx=0).stop 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 -- 172.21.15.32:0/1900662409 shutdown_connections 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1900662409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e14003a00 0x7f1e14003e20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 -- 172.21.15.32:0/1900662409 >> 172.21.15.32:0/1900662409 conn(0x7f1e340d1ae0 msgr2=0x7f1e340d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 -- 172.21.15.32:0/1900662409 shutdown_connections 2024-03-06T07:29:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.293+0000 7f1e38e00640 1 -- 172.21.15.32:0/1900662409 wait complete. 2024-03-06T07:29:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.295+0000 7f1e38e00640 1 Processor -- start 2024-03-06T07:29:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.295+0000 7f1e38e00640 1 -- start start 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e38e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e14003a00 0x7f1e341f7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e38e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e340cf900 0x7f1e34156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e38e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 0x7f1e34157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e38e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3415f830 con 0x7f1e340cfd80 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e38e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340ae8b0 con 0x7f1e340cf900 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e38e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3415f9b0 con 0x7f1e14003a00 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 0x7f1e34157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 0x7f1e34157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36138/0 (socket says 172.21.15.32:36138) 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e33e00640 1 -- 172.21.15.32:0/1437991689 learned_addr learned my addr 172.21.15.32:0/1437991689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e33400640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e14003a00 0x7f1e341f7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.297+0000 7f1e32a00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e340cf900 0x7f1e34156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e33e00640 1 -- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e14003a00 msgr2=0x7f1e341f7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e33e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e14003a00 0x7f1e341f7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e33e00640 1 -- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e340cf900 msgr2=0x7f1e34156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e33e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e340cf900 0x7f1e34156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e33e00640 1 -- 172.21.15.32:0/1437991689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e341f8170 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e33e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 0x7f1e34157380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1e340718e0 tx=0x7f1e2c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c004220 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c0043c0 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c007310 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e340c6800 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.298+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e340c6dc0 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.300+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3404f330 con 0x7f1e340cfd80 2024-03-06T07:29:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.300+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e2c0074b0 con 0x7f1e340cfd80 2024-03-06T07:29:36.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.302+0000 7f1e2b400640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e1006b4d0 0x7f1e1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.302+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e2c0892c0 con 0x7f1e340cfd80 2024-03-06T07:29:36.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.302+0000 7f1e33400640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e1006b4d0 0x7f1e1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.303+0000 7f1e33400640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e1006b4d0 0x7f1e1006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e24036000 tx=0x7f1e24030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.307+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e2c058130 con 0x7f1e340cfd80 2024-03-06T07:29:36.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.506+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e340cbdf0 con 0x7f1e340cfd80 2024-03-06T07:29:36.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.508+0000 7f1e2b400640 1 -- 172.21.15.32:0/1437991689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f1e2c0890c0 con 0x7f1e340cfd80 2024-03-06T07:29:36.511 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:36.511 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.510+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e1006b4d0 msgr2=0x7f1e1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.510+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e1006b4d0 0x7f1e1006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e24036000 tx=0x7f1e24030000 comp rx=0 tx=0).stop 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 msgr2=0x7f1e34157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 0x7f1e34157380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1e340718e0 tx=0x7f1e2c00c820 comp rx=0 tx=0).stop 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 shutdown_connections 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e1006b4d0 0x7f1e1006d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e340cfd80 0x7f1e34157380 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e340cf900 0x7f1e34156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 --2- 172.21.15.32:0/1437991689 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e14003a00 0x7f1e341f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 >> 172.21.15.32:0/1437991689 conn(0x7f1e340d1ae0 msgr2=0x7f1e340b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 shutdown_connections 2024-03-06T07:29:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.511+0000 7f1e38e00640 1 -- 172.21.15.32:0/1437991689 wait complete. 2024-03-06T07:29:36.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:36.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:36.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.757+0000 7fe87be00640 1 Processor -- start 2024-03-06T07:29:36.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.758+0000 7fe87be00640 1 -- start start 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.759+0000 7fe87be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe87c0b3c20 0x7fe87c0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.759+0000 7fe87be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe87c0b4000 0x7fe87c15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.759+0000 7fe87be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c071930 con 0x7fe87c0b1b40 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87c0b4490 con 0x7fe87c0b4000 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87c071c10 con 0x7fe87c0b3c20 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe87c0b4000 0x7fe87c15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe87c0b4000 0x7fe87c15b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36146/0 (socket says 172.21.15.32:36146) 2024-03-06T07:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 -- 172.21.15.32:0/1419274301 learned_addr learned my addr 172.21.15.32:0/1419274301 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:36.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87aa00640 1 --2- 172.21.15.32:0/1419274301 >> v2:172.21.15.39:3300/0 conn(0x7fe87c0b3c20 0x7fe87c0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 -- 172.21.15.32:0/1419274301 >> v2:172.21.15.39:3300/0 conn(0x7fe87c0b3c20 msgr2=0x7fe87c0729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 --2- 172.21.15.32:0/1419274301 >> v2:172.21.15.39:3300/0 conn(0x7fe87c0b3c20 0x7fe87c0729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 -- 172.21.15.32:0/1419274301 >> v1:172.21.15.39:6789/0 conn(0x7fe87c0b1b40 legacy=0x7fe87c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.760+0000 7fe87a000640 1 -- 172.21.15.32:0/1419274301 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c071ef0 con 0x7fe87c0b4000 2024-03-06T07:29:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.761+0000 7fe87a000640 1 --2- 172.21.15.32:0/1419274301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b4000 0x7fe87c15b8a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe87c06ead0 tx=0x7fe86802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d769612a3132433c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.762+0000 7fe879600640 1 -- 172.21.15.32:0/1419274301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868035020 con 0x7fe87c0b4000 2024-03-06T07:29:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.762+0000 7fe879600640 1 -- 172.21.15.32:0/1419274301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe868031870 con 0x7fe87c0b4000 2024-03-06T07:29:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.762+0000 7fe879600640 1 -- 172.21.15.32:0/1419274301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868031c40 con 0x7fe87c0b4000 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.762+0000 7fe87be00640 1 -- 172.21.15.32:0/1419274301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b4000 msgr2=0x7fe87c15b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.762+0000 7fe87be00640 1 --2- 172.21.15.32:0/1419274301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b4000 0x7fe87c15b8a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe87c06ead0 tx=0x7fe86802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.763+0000 7fe87be00640 1 -- 172.21.15.32:0/1419274301 shutdown_connections 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.763+0000 7fe87be00640 1 --2- 172.21.15.32:0/1419274301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b4000 0x7fe87c15b8a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.763+0000 7fe87be00640 1 --2- 172.21.15.32:0/1419274301 >> v2:172.21.15.39:3300/0 conn(0x7fe87c0b3c20 0x7fe87c0729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.763+0000 7fe87be00640 1 -- 172.21.15.32:0/1419274301 >> 172.21.15.32:0/1419274301 conn(0x7fe87c06b3a0 msgr2=0x7fe87c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.763+0000 7fe87be00640 1 -- 172.21.15.32:0/1419274301 shutdown_connections 2024-03-06T07:29:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.763+0000 7fe87be00640 1 -- 172.21.15.32:0/1419274301 wait complete. 2024-03-06T07:29:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.764+0000 7fe87be00640 1 Processor -- start 2024-03-06T07:29:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 -- start start 2024-03-06T07:29:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 0x7fe87c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe87c0b3c20 0x7fe87c155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe87c0b4000 0x7fe87c1486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c15bde0 con 0x7fe87c0b1b40 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c0724b0 con 0x7fe87c0b3c20 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.765+0000 7fe87be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c15bf60 con 0x7fe87c0b4000 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 0x7fe87c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 0x7fe87c15ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36160/0 (socket says 172.21.15.32:36160) 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 -- 172.21.15.32:0/4249923679 learned_addr learned my addr 172.21.15.32:0/4249923679 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87a000640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe87c0b3c20 0x7fe87c155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87b400640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe87c0b4000 0x7fe87c1486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 -- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe87c0b4000 msgr2=0x7fe87c1486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe87c0b4000 0x7fe87c1486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 -- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe87c0b3c20 msgr2=0x7fe87c155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe87c0b3c20 0x7fe87c155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87aa00640 1 -- 172.21.15.32:0/4249923679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c155c60 con 0x7fe87c0b1b40 2024-03-06T07:29:36.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.766+0000 7fe87b400640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe87c0b4000 0x7fe87c1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:36.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.767+0000 7fe87aa00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 0x7fe87c15ab40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d800 tx=0x7fe86c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.767+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00b6c0 con 0x7fe87c0b1b40 2024-03-06T07:29:36.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.767+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c019070 con 0x7fe87c0b1b40 2024-03-06T07:29:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.767+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00ba80 con 0x7fe87c0b1b40 2024-03-06T07:29:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.767+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c155f00 con 0x7fe87c0b1b40 2024-03-06T07:29:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.767+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c1564c0 con 0x7fe87c0b1b40 2024-03-06T07:29:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.768+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c04f330 con 0x7fe87c0b1b40 2024-03-06T07:29:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.769+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe86c00bc20 con 0x7fe87c0b1b40 2024-03-06T07:29:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.770+0000 7fe872a00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe85c06b4d0 0x7fe85c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.770+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe86c08a6c0 con 0x7fe87c0b1b40 2024-03-06T07:29:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.770+0000 7fe87a000640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe85c06b4d0 0x7fe85c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.771+0000 7fe87a000640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe85c06b4d0 0x7fe85c06d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe868036000 tx=0x7fe86803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.775+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe86c059530 con 0x7fe87c0b1b40 2024-03-06T07:29:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.975+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe87c0744c0 con 0x7fe87c0b1b40 2024-03-06T07:29:36.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.976+0000 7fe872a00640 1 -- 172.21.15.32:0/4249923679 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe86c058e30 con 0x7fe87c0b1b40 2024-03-06T07:29:36.978 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe85c06b4d0 msgr2=0x7fe85c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe85c06b4d0 0x7fe85c06d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe868036000 tx=0x7fe86803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 msgr2=0x7fe87c15ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 0x7fe87c15ab40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d800 tx=0x7fe86c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 shutdown_connections 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe85c06b4d0 0x7fe85c06d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe87c0b4000 0x7fe87c1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe87c0b3c20 0x7fe87c155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 --2- 172.21.15.32:0/4249923679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87c0b1b40 0x7fe87c15ab40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 >> 172.21.15.32:0/4249923679 conn(0x7fe87c06b3a0 msgr2=0x7fe87c0b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 shutdown_connections 2024-03-06T07:29:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:36.978+0000 7fe87be00640 1 -- 172.21.15.32:0/4249923679 wait complete. 2024-03-06T07:29:36.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:37.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.224+0000 7fae63c00640 1 Processor -- start 2024-03-06T07:29:37.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.224+0000 7fae63c00640 1 -- start start 2024-03-06T07:29:37.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.225+0000 7fae63c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c3f0 0x7fae5c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.225+0000 7fae63c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fae5c0afcf0 0x7fae5c0b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.225+0000 7fae63c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0af250 con 0x7fae5c15c010 2024-03-06T07:29:37.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.225+0000 7fae63c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c15cf50 con 0x7fae5c0afcf0 2024-03-06T07:29:37.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.225+0000 7fae63c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c15d230 con 0x7fae5c15c3f0 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.226+0000 7fae61200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fae5c15c010 0x7fae5c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:57346/0 (socket says 172.21.15.32:57346) 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.226+0000 7fae61200640 1 -- 172.21.15.32:0/2431662026 learned_addr learned my addr 172.21.15.32:0/2431662026 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.226+0000 7fae53e00640 1 --2- 172.21.15.32:0/2431662026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c3f0 0x7fae5c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.226+0000 7fae61c00640 1 --2- 172.21.15.32:0/2431662026 >> v2:172.21.15.39:3300/0 conn(0x7fae5c0afcf0 0x7fae5c0b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.226+0000 7fae5be00640 1 -- 172.21.15.32:0/2431662026 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445658616 0 0) 0x7fae5c0af250 con 0x7fae5c15c010 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.226+0000 7fae5be00640 1 -- 172.21.15.32:0/2431662026 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae40003520 con 0x7fae5c15c010 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.227+0000 7fae53e00640 1 -- 172.21.15.32:0/2431662026 >> v2:172.21.15.39:3300/0 conn(0x7fae5c0afcf0 msgr2=0x7fae5c0b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.227+0000 7fae53e00640 1 --2- 172.21.15.32:0/2431662026 >> v2:172.21.15.39:3300/0 conn(0x7fae5c0afcf0 0x7fae5c0b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.227+0000 7fae53e00640 1 -- 172.21.15.32:0/2431662026 >> v1:172.21.15.39:6789/0 conn(0x7fae5c15c010 legacy=0x7fae5c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.227+0000 7fae53e00640 1 -- 172.21.15.32:0/2431662026 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0af4f0 con 0x7fae5c15c3f0 2024-03-06T07:29:37.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.228+0000 7fae53e00640 1 --2- 172.21.15.32:0/2431662026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c3f0 0x7fae5c15c870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae440097a0 tx=0x7fae4402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da90e6ccd4010ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.229+0000 7fae5be00640 1 -- 172.21.15.32:0/2431662026 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44035020 con 0x7fae5c15c3f0 2024-03-06T07:29:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.229+0000 7fae5be00640 1 -- 172.21.15.32:0/2431662026 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae44031830 con 0x7fae5c15c3f0 2024-03-06T07:29:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.229+0000 7fae5be00640 1 -- 172.21.15.32:0/2431662026 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44031be0 con 0x7fae5c15c3f0 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.229+0000 7fae63c00640 1 -- 172.21.15.32:0/2431662026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c3f0 msgr2=0x7fae5c15c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.229+0000 7fae63c00640 1 --2- 172.21.15.32:0/2431662026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c3f0 0x7fae5c15c870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae440097a0 tx=0x7fae4402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.230+0000 7fae63c00640 1 -- 172.21.15.32:0/2431662026 shutdown_connections 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.230+0000 7fae63c00640 1 --2- 172.21.15.32:0/2431662026 >> v2:172.21.15.39:3300/0 conn(0x7fae5c0afcf0 0x7fae5c0b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.230+0000 7fae63c00640 1 --2- 172.21.15.32:0/2431662026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c3f0 0x7fae5c15c870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.230+0000 7fae63c00640 1 -- 172.21.15.32:0/2431662026 >> 172.21.15.32:0/2431662026 conn(0x7fae5c0732d0 msgr2=0x7fae5c09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.230+0000 7fae63c00640 1 -- 172.21.15.32:0/2431662026 shutdown_connections 2024-03-06T07:29:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.230+0000 7fae63c00640 1 -- 172.21.15.32:0/2431662026 wait complete. 2024-03-06T07:29:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 Processor -- start 2024-03-06T07:29:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 -- start start 2024-03-06T07:29:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 0x7fae5c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c010 0x7fae5c1452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae5c15c3f0 0x7fae5c1457e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0d1c10 con 0x7fae5c0afcf0 2024-03-06T07:29:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0d1a90 con 0x7fae5c15c3f0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae63c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0b0a30 con 0x7fae5c15c010 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 0x7fae5c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 0x7fae5c1585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36176/0 (socket says 172.21.15.32:36176) 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 -- 172.21.15.32:0/2772567674 learned_addr learned my addr 172.21.15.32:0/2772567674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae5c15c3f0 0x7fae5c1457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 -- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c010 msgr2=0x7fae5c1452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c010 0x7fae5c1452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 -- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae5c15c3f0 msgr2=0x7fae5c1457e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae5c15c3f0 0x7fae5c1457e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.233+0000 7fae61200640 1 -- 172.21.15.32:0/2772567674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0d0320 con 0x7fae5c0afcf0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.234+0000 7fae61200640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 0x7fae5c1585c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00cbf0 tx=0x7fae4c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.234+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c017070 con 0x7fae5c0afcf0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.234+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c00ad50 con 0x7fae5c0afcf0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.234+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c012620 con 0x7fae5c0afcf0 2024-03-06T07:29:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.234+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c0cf280 con 0x7fae5c0afcf0 2024-03-06T07:29:37.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.234+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c0cf840 con 0x7fae5c0afcf0 2024-03-06T07:29:37.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.236+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c04f330 con 0x7fae5c0afcf0 2024-03-06T07:29:37.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.236+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae4c01a070 con 0x7fae5c0afcf0 2024-03-06T07:29:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.237+0000 7fae59600640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae3806b4d0 0x7fae3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.237+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae4c08a050 con 0x7fae5c0afcf0 2024-03-06T07:29:37.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.238+0000 7fae53e00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae3806b4d0 0x7fae3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.238+0000 7fae53e00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae3806b4d0 0x7fae3806d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fae44036000 tx=0x7fae44030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.242+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae4c058f60 con 0x7fae5c0afcf0 2024-03-06T07:29:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.441+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae5c0d2a60 con 0x7fae5c0afcf0 2024-03-06T07:29:37.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.442+0000 7fae59600640 1 -- 172.21.15.32:0/2772567674 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae4c058860 con 0x7fae5c0afcf0 2024-03-06T07:29:37.444 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae3806b4d0 msgr2=0x7fae3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae3806b4d0 0x7fae3806d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fae44036000 tx=0x7fae44030000 comp rx=0 tx=0).stop 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 msgr2=0x7fae5c1585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 0x7fae5c1585c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00cbf0 tx=0x7fae4c00a430 comp rx=0 tx=0).stop 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 shutdown_connections 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae3806b4d0 0x7fae3806d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae5c15c3f0 0x7fae5c1457e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae5c15c010 0x7fae5c1452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 --2- 172.21.15.32:0/2772567674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae5c0afcf0 0x7fae5c1585c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 >> 172.21.15.32:0/2772567674 conn(0x7fae5c0732d0 msgr2=0x7fae5c159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.444+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 shutdown_connections 2024-03-06T07:29:37.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.445+0000 7fae63c00640 1 -- 172.21.15.32:0/2772567674 wait complete. 2024-03-06T07:29:37.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.691+0000 7f508fc00640 1 Processor -- start 2024-03-06T07:29:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.691+0000 7f508fc00640 1 -- start start 2024-03-06T07:29:37.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f508fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 0x7f50880cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f508fc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5088072df0 con 0x7f5088071e20 2024-03-06T07:29:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f508fc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50880cdab0 con 0x7f508815bf20 2024-03-06T07:29:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f508fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50880b4d70 con 0x7f5088072520 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f508dc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5088071e20 0x7f50880ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43728/0 (socket says 172.21.15.32:43728) 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f508dc00640 1 -- 172.21.15.32:0/1830128604 learned_addr learned my addr 172.21.15.32:0/1830128604 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f5087e00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 0x7f50880cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140173474 0 0) 0x7f5088072df0 con 0x7f5088071e20 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.693+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c003520 con 0x7f5088071e20 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4101662871 0 0) 0x7f506c003520 con 0x7f5088071e20 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 msgr2=0x7f50880cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 0x7f50880cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 >> v1:172.21.15.39:6789/0 conn(0x7f508815bf20 legacy=0x7f50880acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50880b5050 con 0x7f5088071e20 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f5080001670 con 0x7f5088071e20 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 >> v1:172.21.15.32:6789/0 conn(0x7f5088071e20 legacy=0x7f50880ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 0x7f506c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f506c003fb0 0x7f506c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880b4d70 con 0x7f506c003fb0 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c00bb00 con 0x7f506c004d40 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c00bd00 con 0x7f5088072520 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f5087e00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f506c003fb0 0x7f506c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.694+0000 7f508d200640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 0x7f506c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.695+0000 7f508dc00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.695+0000 7f5087e00640 1 -- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 msgr2=0x7f506c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.695+0000 7f5087e00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5088072520 0x7f506c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.695+0000 7f5087e00640 1 -- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c004d40 msgr2=0x7f506c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.695+0000 7f5087e00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.695+0000 7f5087e00640 1 -- 172.21.15.32:0/1830128604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c003520 con 0x7f506c003fb0 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.696+0000 7f5087e00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f506c003fb0 0x7f506c004800 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5070009ac0 tx=0x7f506c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1ede4816ad751c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.696+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070039070 con 0x7f506c003fb0 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.696+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507002c600 con 0x7f506c003fb0 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.696+0000 7f5087400640 1 -- 172.21.15.32:0/1830128604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070009690 con 0x7f506c003fb0 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508fc00640 1 -- 172.21.15.32:0/1830128604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f506c003fb0 msgr2=0x7f506c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508fc00640 1 --2- 172.21.15.32:0/1830128604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f506c003fb0 0x7f506c004800 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5070009ac0 tx=0x7f506c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508dc00640 1 -- 172.21.15.32:0/1830128604 reap_dead start 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508fc00640 1 -- 172.21.15.32:0/1830128604 shutdown_connections 2024-03-06T07:29:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508fc00640 1 -- 172.21.15.32:0/1830128604 >> 172.21.15.32:0/1830128604 conn(0x7f508806b3d0 msgr2=0x7f508806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508fc00640 1 -- 172.21.15.32:0/1830128604 shutdown_connections 2024-03-06T07:29:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.697+0000 7f508fc00640 1 -- 172.21.15.32:0/1830128604 wait complete. 2024-03-06T07:29:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.700+0000 7f508fc00640 1 Processor -- start 2024-03-06T07:29:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.700+0000 7f508fc00640 1 -- start start 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c003fb0 0x7f508815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 0x7f50880cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50881f4620 0x7f50881f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880ad450 con 0x7f50880cb160 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880b4990 con 0x7f506c003fb0 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508815e390 con 0x7f50881f4620 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 0x7f50880cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 0x7f50880cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36196/0 (socket says 172.21.15.32:36196) 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 -- 172.21.15.32:0/2517082413 learned_addr learned my addr 172.21.15.32:0/2517082413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508dc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50881f4620 0x7f50881f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508d200640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c003fb0 0x7f508815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 -- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50881f4620 msgr2=0x7f50881f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50881f4620 0x7f50881f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 -- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c003fb0 msgr2=0x7f508815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c003fb0 0x7f508815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 -- 172.21.15.32:0/2517082413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508815e670 con 0x7f50880cb160 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f508d200640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c003fb0 0x7f508815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5087e00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 0x7f50880cb5e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5070000c00 tx=0x7f5070004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.701+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070042020 con 0x7f50880cb160 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.702+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507003a040 con 0x7f50880cb160 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.702+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070039070 con 0x7f50880cb160 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.702+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508815e950 con 0x7f50880cb160 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.702+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508815ef10 con 0x7f50880cb160 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.703+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508806d450 con 0x7f50880cb160 2024-03-06T07:29:37.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.705+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5070045020 con 0x7f50880cb160 2024-03-06T07:29:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.706+0000 7f5084c00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f506406b500 0x7f506406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.706+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50700aba00 con 0x7f50880cb160 2024-03-06T07:29:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.706+0000 7f508d200640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f506406b500 0x7f506406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.706+0000 7f508d200640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f506406b500 0x7f506406d9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5078007fd0 tx=0x7f5078009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.710+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f508806d450 con 0x7f50880cb160 2024-03-06T07:29:37.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.915+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f508804f330 con 0x7f50880cb160 2024-03-06T07:29:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.915+0000 7f5084c00640 1 -- 172.21.15.32:0/2517082413 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f507004c030 con 0x7f50880cb160 2024-03-06T07:29:37.917 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.918+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f506406b500 msgr2=0x7f506406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.918+0000 7f508fc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f506406b500 0x7f506406d9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5078007fd0 tx=0x7f5078009040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.918+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 msgr2=0x7f50880cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.918+0000 7f508fc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 0x7f50880cb5e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5070000c00 tx=0x7f5070004730 comp rx=0 tx=0).stop 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 shutdown_connections 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f506406b500 0x7f506406d9c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f50881f4620 0x7f50881f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50880cb160 0x7f50880cb5e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 --2- 172.21.15.32:0/2517082413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506c003fb0 0x7f508815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 >> 172.21.15.32:0/2517082413 conn(0x7f508806b3d0 msgr2=0x7f50880cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 shutdown_connections 2024-03-06T07:29:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:37.919+0000 7f508fc00640 1 -- 172.21.15.32:0/2517082413 wait complete. 2024-03-06T07:29:37.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.164+0000 7f5750a00640 1 Processor -- start 2024-03-06T07:29:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.164+0000 7f5750a00640 1 -- start start 2024-03-06T07:29:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.166+0000 7f5750a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.166+0000 7f5750a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f574c15a930 0x7f574c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.166+0000 7f5750a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c071640 con 0x7f574c0ce9f0 2024-03-06T07:29:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.166+0000 7f5750a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c071920 con 0x7f574c15a930 2024-03-06T07:29:38.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.166+0000 7f5750a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c071c00 con 0x7f574c15a550 2024-03-06T07:29:38.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.166+0000 7f5741400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f574c15a930 0x7f574c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5741400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f574c15a930 0x7f574c0ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39566/0 (socket says 172.21.15.32:39566) 2024-03-06T07:29:38.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5741400640 1 -- 172.21.15.32:0/4169458008 learned_addr learned my addr 172.21.15.32:0/4169458008 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5749e00640 1 --2- 172.21.15.32:0/4169458008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5741400640 1 -- 172.21.15.32:0/4169458008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 msgr2=0x7f574c0aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5741400640 1 --2- 172.21.15.32:0/4169458008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5741400640 1 -- 172.21.15.32:0/4169458008 >> v1:172.21.15.39:6789/0 conn(0x7f574c0ce9f0 legacy=0x7f574c0b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.167+0000 7f5741400640 1 -- 172.21.15.32:0/4169458008 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c071ee0 con 0x7f574c15a930 2024-03-06T07:29:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.168+0000 7f5741400640 1 --2- 172.21.15.32:0/4169458008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c15a930 0x7f574c0ce4b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5734009870 tx=0x7f573402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5222eb589c87abee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.168+0000 7f5749400640 1 -- 172.21.15.32:0/4169458008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734034070 con 0x7f574c15a930 2024-03-06T07:29:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5749400640 1 -- 172.21.15.32:0/4169458008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734030840 con 0x7f574c15a930 2024-03-06T07:29:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5749400640 1 -- 172.21.15.32:0/4169458008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734030c10 con 0x7f574c15a930 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 -- 172.21.15.32:0/4169458008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c15a930 msgr2=0x7f574c0ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 --2- 172.21.15.32:0/4169458008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c15a930 0x7f574c0ce4b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5734009870 tx=0x7f573402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 -- 172.21.15.32:0/4169458008 shutdown_connections 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 --2- 172.21.15.32:0/4169458008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c15a930 0x7f574c0ce4b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 --2- 172.21.15.32:0/4169458008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 -- 172.21.15.32:0/4169458008 >> 172.21.15.32:0/4169458008 conn(0x7f574c06afb0 msgr2=0x7f574c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 -- 172.21.15.32:0/4169458008 shutdown_connections 2024-03-06T07:29:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.169+0000 7f5750a00640 1 -- 172.21.15.32:0/4169458008 wait complete. 2024-03-06T07:29:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.171+0000 7f5750a00640 1 Processor -- start 2024-03-06T07:29:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.171+0000 7f5750a00640 1 -- start start 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.172+0000 7f5750a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 0x7f574c151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.172+0000 7f5750a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.172+0000 7f5750a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f574c15a930 0x7f574c149420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.172+0000 7f5750a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0b52b0 con 0x7f574c0ce9f0 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.172+0000 7f5750a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0b5130 con 0x7f574c15a930 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.172+0000 7f5750a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0ade50 con 0x7f574c15a550 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 0x7f574c151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 0x7f574c151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39576/0 (socket says 172.21.15.32:39576) 2024-03-06T07:29:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 -- 172.21.15.32:0/1054586579 learned_addr learned my addr 172.21.15.32:0/1054586579 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 -- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 msgr2=0x7f574c0afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 -- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f574c15a930 msgr2=0x7f574c149420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f574a800640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f574c15a930 0x7f574c149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f574c15a930 0x7f574c149420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f5749e00640 1 -- 172.21.15.32:0/1054586579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c152a90 con 0x7f574c0ce9f0 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.173+0000 7f574a800640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f574c15a930 0x7f574c149420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.174+0000 7f5749e00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 0x7f574c151cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f573c00d830 tx=0x7f573c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.174+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00b6c0 con 0x7f574c0ce9f0 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.174+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c152210 con 0x7f574c0ce9f0 2024-03-06T07:29:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.174+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c019070 con 0x7f574c0ce9f0 2024-03-06T07:29:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.174+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00ba80 con 0x7f574c0ce9f0 2024-03-06T07:29:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.174+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c167890 con 0x7f574c0ce9f0 2024-03-06T07:29:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.175+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c04f3a0 con 0x7f574c0ce9f0 2024-03-06T07:29:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.176+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573c00bc20 con 0x7f574c0ce9f0 2024-03-06T07:29:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.177+0000 7f5742a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.177+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f573c08a6a0 con 0x7f574c0ce9f0 2024-03-06T07:29:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.177+0000 7f5741400640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.178+0000 7f5741400640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572806b4d0 0x7f572806d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5734036000 tx=0x7f5734030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.182+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f573c059530 con 0x7f574c0ce9f0 2024-03-06T07:29:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.384+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f574c0ae8f0 con 0x7f574c0ce9f0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.387+0000 7f5742a00640 1 -- 172.21.15.32:0/1054586579 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f573c058e30 con 0x7f574c0ce9f0 2024-03-06T07:29:38.389 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:38.389 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.390+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572806b4d0 msgr2=0x7f572806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.390+0000 7f5750a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572806b4d0 0x7f572806d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5734036000 tx=0x7f5734030000 comp rx=0 tx=0).stop 2024-03-06T07:29:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.390+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 msgr2=0x7f574c151cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.390+0000 7f5750a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 0x7f574c151cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f573c00d830 tx=0x7f573c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 shutdown_connections 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f574c15a930 0x7f574c149420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f574c15a550 0x7f574c0afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 --2- 172.21.15.32:0/1054586579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f574c0ce9f0 0x7f574c151cd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 >> 172.21.15.32:0/1054586579 conn(0x7f574c06afb0 msgr2=0x7f574c0cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 shutdown_connections 2024-03-06T07:29:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.391+0000 7f5750a00640 1 -- 172.21.15.32:0/1054586579 wait complete. 2024-03-06T07:29:38.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:38.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:38.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.639+0000 7f2ed3200640 1 Processor -- start 2024-03-06T07:29:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.640+0000 7f2ed3200640 1 -- start start 2024-03-06T07:29:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.641+0000 7f2ed3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2ed415dee0 0x7f2ed40d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.641+0000 7f2ed3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ed40d4860 0x7f2ed40d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.641+0000 7f2ed3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40ceca0 con 0x7f2ed40cfa60 2024-03-06T07:29:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.641+0000 7f2ed3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed40cf5d0 con 0x7f2ed40d4860 2024-03-06T07:29:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.641+0000 7f2ed3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed40cef80 con 0x7f2ed415dee0 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed2800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2ed40cfa60 0x7f2ed40b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45862/0 (socket says 172.21.15.32:45862) 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed2800640 1 -- 172.21.15.32:0/3986047875 learned_addr learned my addr 172.21.15.32:0/3986047875 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed1400640 1 --2- 172.21.15.32:0/3986047875 >> v2:172.21.15.32:3300/0 conn(0x7f2ed40d4860 0x7f2ed40d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed1e00640 1 --2- 172.21.15.32:0/3986047875 >> v2:172.21.15.39:3300/0 conn(0x7f2ed415dee0 0x7f2ed40d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed0a00640 1 -- 172.21.15.32:0/3986047875 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279168688 0 0) 0x7f2ed40ceca0 con 0x7f2ed40cfa60 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed0a00640 1 -- 172.21.15.32:0/3986047875 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb8003520 con 0x7f2ed40cfa60 2024-03-06T07:29:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed1400640 1 -- 172.21.15.32:0/3986047875 >> v2:172.21.15.39:3300/0 conn(0x7f2ed415dee0 msgr2=0x7f2ed40d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed1400640 1 --2- 172.21.15.32:0/3986047875 >> v2:172.21.15.39:3300/0 conn(0x7f2ed415dee0 0x7f2ed40d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed1400640 1 -- 172.21.15.32:0/3986047875 >> v1:172.21.15.32:6789/0 conn(0x7f2ed40cfa60 legacy=0x7f2ed40b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.642+0000 7f2ed1400640 1 -- 172.21.15.32:0/3986047875 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40cf260 con 0x7f2ed40d4860 2024-03-06T07:29:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.644+0000 7f2ed1400640 1 --2- 172.21.15.32:0/3986047875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40d0630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2ed408c6f0 tx=0x7f2ec002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c90c995b9042171 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.644+0000 7f2ed0a00640 1 -- 172.21.15.32:0/3986047875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0039070 con 0x7f2ed40d4860 2024-03-06T07:29:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.644+0000 7f2ed0a00640 1 -- 172.21.15.32:0/3986047875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec002cc80 con 0x7f2ed40d4860 2024-03-06T07:29:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.644+0000 7f2ed0a00640 1 -- 172.21.15.32:0/3986047875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0034660 con 0x7f2ed40d4860 2024-03-06T07:29:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 -- 172.21.15.32:0/3986047875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 msgr2=0x7f2ed40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3986047875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40d0630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2ed408c6f0 tx=0x7f2ec002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 -- 172.21.15.32:0/3986047875 shutdown_connections 2024-03-06T07:29:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3986047875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40d0630 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3986047875 >> v2:172.21.15.39:3300/0 conn(0x7f2ed415dee0 0x7f2ed40d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 -- 172.21.15.32:0/3986047875 >> 172.21.15.32:0/3986047875 conn(0x7f2ed40d3cd0 msgr2=0x7f2ed40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 -- 172.21.15.32:0/3986047875 shutdown_connections 2024-03-06T07:29:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.645+0000 7f2ed3200640 1 -- 172.21.15.32:0/3986047875 wait complete. 2024-03-06T07:29:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.647+0000 7f2ed3200640 1 Processor -- start 2024-03-06T07:29:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.647+0000 7f2ed3200640 1 -- start start 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ed40cfa60 0x7f2ed40c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed415dee0 0x7f2ed414f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40d1de0 con 0x7f2ed40d4860 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed415e2c0 con 0x7f2ed415dee0 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40d1f60 con 0x7f2ed40cfa60 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39584/0 (socket says 172.21.15.32:39584) 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed1400640 1 -- 172.21.15.32:0/3923510525 learned_addr learned my addr 172.21.15.32:0/3923510525 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed1e00640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ed40cfa60 0x7f2ed40c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.648+0000 7f2ed2800640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed415dee0 0x7f2ed414f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2ed1400640 1 -- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ed40cfa60 msgr2=0x7f2ed40c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2ed1400640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ed40cfa60 0x7f2ed40c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2ed1400640 1 -- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed415dee0 msgr2=0x7f2ed414f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2ed1400640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed415dee0 0x7f2ed414f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2ed1400640 1 -- 172.21.15.32:0/3923510525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed409f4a0 con 0x7f2ed40d4860 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2ed1400640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40c9cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009e80 tx=0x7f2ec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0039050 con 0x7f2ed40d4860 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.649+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec0045070 con 0x7f2ed40d4860 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.650+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0009450 con 0x7f2ed40d4860 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.650+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed414cf90 con 0x7f2ed40d4860 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.650+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed409dd20 con 0x7f2ed40d4860 2024-03-06T07:29:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.651+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed404f330 con 0x7f2ed40d4860 2024-03-06T07:29:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.652+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ec003e850 con 0x7f2ed40d4860 2024-03-06T07:29:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.653+0000 7f2eca000640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.653+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ec00ad120 con 0x7f2ed40d4860 2024-03-06T07:29:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.653+0000 7f2ed1e00640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.654+0000 7f2ed1e00640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2eb406b4d0 0x7f2eb406d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2ed40d1b80 tx=0x7f2ec4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.657+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ed409f4a0 con 0x7f2ed40d4860 2024-03-06T07:29:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.859+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ed408d7d0 con 0x7f2ed40d4860 2024-03-06T07:29:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.860+0000 7f2eca000640 1 -- 172.21.15.32:0/3923510525 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ec007c0e0 con 0x7f2ed40d4860 2024-03-06T07:29:38.861 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.862+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2eb406b4d0 msgr2=0x7f2eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.862+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2eb406b4d0 0x7f2eb406d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2ed40d1b80 tx=0x7f2ec4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.862+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 msgr2=0x7f2ed40c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.862+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40c9cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009e80 tx=0x7f2ec0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 shutdown_connections 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed415dee0 0x7f2ed414f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ed40d4860 0x7f2ed40c9cc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 --2- 172.21.15.32:0/3923510525 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ed40cfa60 0x7f2ed40c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 >> 172.21.15.32:0/3923510525 conn(0x7f2ed40d3cd0 msgr2=0x7f2ed40b9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 shutdown_connections 2024-03-06T07:29:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:38.863+0000 7f2ed3200640 1 -- 172.21.15.32:0/3923510525 wait complete. 2024-03-06T07:29:38.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:39.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.108+0000 7f3097e00640 1 Processor -- start 2024-03-06T07:29:39.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.108+0000 7f3097e00640 1 -- start start 2024-03-06T07:29:39.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f3097e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f309815dee0 0x7f30980d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f3097e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30980d0d40 0x7f30980b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f3097e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30980ceca0 con 0x7f30980d4860 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f3097e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30980cf5d0 con 0x7f30980d0d40 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f3097e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30980cef80 con 0x7f309815dee0 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f3097400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30980d0d40 0x7f30980b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f308fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30980d4860 0x7f30980d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45864/0 (socket says 172.21.15.32:45864) 2024-03-06T07:29:39.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.110+0000 7f308fe00640 1 -- 172.21.15.32:0/1000824893 learned_addr learned my addr 172.21.15.32:0/1000824893 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.111+0000 7f3096a00640 1 --2- 172.21.15.32:0/1000824893 >> v2:172.21.15.39:3300/0 conn(0x7f309815dee0 0x7f30980d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.111+0000 7f3097400640 1 -- 172.21.15.32:0/1000824893 >> v2:172.21.15.39:3300/0 conn(0x7f309815dee0 msgr2=0x7f30980d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.111+0000 7f3097400640 1 --2- 172.21.15.32:0/1000824893 >> v2:172.21.15.39:3300/0 conn(0x7f309815dee0 0x7f30980d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.111+0000 7f3097400640 1 -- 172.21.15.32:0/1000824893 >> v1:172.21.15.32:6789/0 conn(0x7f30980d4860 legacy=0x7f30980d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.111+0000 7f3097400640 1 -- 172.21.15.32:0/1000824893 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30980cf260 con 0x7f30980d0d40 2024-03-06T07:29:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.112+0000 7f3097400640 1 --2- 172.21.15.32:0/1000824893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d0d40 0x7f30980b3510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f309808c6f0 tx=0x7f309002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d023c9d5faa43fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.112+0000 7f3096000640 1 -- 172.21.15.32:0/1000824893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090039070 con 0x7f30980d0d40 2024-03-06T07:29:39.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3096000640 1 -- 172.21.15.32:0/1000824893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309002cc70 con 0x7f30980d0d40 2024-03-06T07:29:39.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3096000640 1 -- 172.21.15.32:0/1000824893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090034660 con 0x7f30980d0d40 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 -- 172.21.15.32:0/1000824893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d0d40 msgr2=0x7f30980b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 --2- 172.21.15.32:0/1000824893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d0d40 0x7f30980b3510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f309808c6f0 tx=0x7f309002bec0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 -- 172.21.15.32:0/1000824893 shutdown_connections 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 --2- 172.21.15.32:0/1000824893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d0d40 0x7f30980b3510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 --2- 172.21.15.32:0/1000824893 >> v2:172.21.15.39:3300/0 conn(0x7f309815dee0 0x7f30980d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 -- 172.21.15.32:0/1000824893 >> 172.21.15.32:0/1000824893 conn(0x7f30980d3cd0 msgr2=0x7f30980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.113+0000 7f3097e00640 1 -- 172.21.15.32:0/1000824893 shutdown_connections 2024-03-06T07:29:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.114+0000 7f3097e00640 1 -- 172.21.15.32:0/1000824893 wait complete. 2024-03-06T07:29:39.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.115+0000 7f3097e00640 1 Processor -- start 2024-03-06T07:29:39.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.115+0000 7f3097e00640 1 -- start start 2024-03-06T07:29:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f3097e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30980d0d40 0x7f30981f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f3097e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 0x7f30980cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f3097e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309815dee0 0x7f309809f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f3097e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30980d1de0 con 0x7f30980d4860 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f3097e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309815e2c0 con 0x7f309815dee0 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f3097e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30980d1f60 con 0x7f30980d0d40 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.116+0000 7f308fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 0x7f30980cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 0x7f30980cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39594/0 (socket says 172.21.15.32:39594) 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 -- 172.21.15.32:0/2534076745 learned_addr learned my addr 172.21.15.32:0/2534076745 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f3096a00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30980d0d40 0x7f30981f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 -- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30980d0d40 msgr2=0x7f30981f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30980d0d40 0x7f30981f45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 -- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309815dee0 msgr2=0x7f309809f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309815dee0 0x7f309809f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 -- 172.21.15.32:0/2534076745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30981f4b10 con 0x7f30980d4860 2024-03-06T07:29:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308fe00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 0x7f30980cc700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f308000c400 tx=0x7f308000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080017070 con 0x7f30980d4860 2024-03-06T07:29:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.117+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080005ce0 con 0x7f30980d4860 2024-03-06T07:29:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.118+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30800072f0 con 0x7f30980d4860 2024-03-06T07:29:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.118+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309809dd20 con 0x7f30980d4860 2024-03-06T07:29:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.118+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309809e2e0 con 0x7f30980d4860 2024-03-06T07:29:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.119+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309804f330 con 0x7f30980d4860 2024-03-06T07:29:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.120+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f308001a070 con 0x7f30980d4860 2024-03-06T07:29:39.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.121+0000 7f308ea00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.121+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3080089690 con 0x7f30980d4860 2024-03-06T07:29:39.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.121+0000 7f3096a00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.122+0000 7f3096a00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307406b4d0 0x7f307406d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3088006fd0 tx=0x7f3088008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.125+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30981f4b10 con 0x7f30980d4860 2024-03-06T07:29:39.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.330+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f309808d7d0 con 0x7f30980d4860 2024-03-06T07:29:39.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.331+0000 7f308ea00640 1 -- 172.21.15.32:0/2534076745 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3080058470 con 0x7f30980d4860 2024-03-06T07:29:39.332 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307406b4d0 msgr2=0x7f307406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307406b4d0 0x7f307406d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3088006fd0 tx=0x7f3088008040 comp rx=0 tx=0).stop 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 msgr2=0x7f30980cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 0x7f30980cc700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f308000c400 tx=0x7f308000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 shutdown_connections 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309815dee0 0x7f309809f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30980d4860 0x7f30980cc700 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 --2- 172.21.15.32:0/2534076745 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30980d0d40 0x7f30981f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 >> 172.21.15.32:0/2534076745 conn(0x7f30980d3cd0 msgr2=0x7f30980b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 shutdown_connections 2024-03-06T07:29:39.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.334+0000 7f3097e00640 1 -- 172.21.15.32:0/2534076745 wait complete. 2024-03-06T07:29:39.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:39.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.573+0000 7fdee3200640 1 Processor -- start 2024-03-06T07:29:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.573+0000 7fdee3200640 1 -- start start 2024-03-06T07:29:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.575+0000 7fdee3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 0x7fdee415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.575+0000 7fdee3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee415fe40 con 0x7fdee40b2250 2024-03-06T07:29:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.575+0000 7fdee3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee4160120 con 0x7fdee415cc90 2024-03-06T07:29:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.575+0000 7fdee3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4160400 con 0x7fdee4145430 2024-03-06T07:29:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.575+0000 7fdee2800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdee40b2250 0x7fdee40b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45870/0 (socket says 172.21.15.32:45870) 2024-03-06T07:29:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.575+0000 7fdee2800640 1 -- 172.21.15.32:0/66387013 learned_addr learned my addr 172.21.15.32:0/66387013 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee1e00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 0x7fdee415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875621689 0 0) 0x7fdee415fe40 con 0x7fdee40b2250 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec4003520 con 0x7fdee40b2250 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3691803734 0 0) 0x7fdec4003520 con 0x7fdee40b2250 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 msgr2=0x7fdee415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 0x7fdee415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 >> v1:172.21.15.39:6789/0 conn(0x7fdee415cc90 legacy=0x7fdee40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee41606e0 con 0x7fdee40b2250 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fdedc001670 con 0x7fdee40b2250 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 >> v1:172.21.15.32:6789/0 conn(0x7fdee40b2250 legacy=0x7fdee40b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdec4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 0x7fdec4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec4004d40 0x7fdec40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4160400 con 0x7fdec40039e0 2024-03-06T07:29:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400ba70 con 0x7fdec4004d40 2024-03-06T07:29:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.576+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400bbf0 con 0x7fdee4145430 2024-03-06T07:29:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.577+0000 7fdee1e00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdec4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.577+0000 7fdee1e00640 1 -- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 msgr2=0x7fdec4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.577+0000 7fdee1e00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee4145430 0x7fdec4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.577+0000 7fdee1e00640 1 -- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec4004d40 msgr2=0x7fdec40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.577+0000 7fdee1e00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec4004d40 0x7fdec40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.577+0000 7fdee1e00640 1 -- 172.21.15.32:0/66387013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee41606e0 con 0x7fdec40039e0 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.578+0000 7fdee1e00640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdec4003e00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fded4009ac0 tx=0x7fdec4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22016c260ec15b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.578+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4039070 con 0x7fdec40039e0 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded402c710 con 0x7fdec40039e0 2024-03-06T07:29:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee0a00640 1 -- 172.21.15.32:0/66387013 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4009690 con 0x7fdec40039e0 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee3200640 1 -- 172.21.15.32:0/66387013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 msgr2=0x7fdec4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee3200640 1 --2- 172.21.15.32:0/66387013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdec4003e00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fded4009ac0 tx=0x7fdec4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee2800640 1 -- 172.21.15.32:0/66387013 reap_dead start 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee3200640 1 -- 172.21.15.32:0/66387013 shutdown_connections 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee3200640 1 -- 172.21.15.32:0/66387013 >> 172.21.15.32:0/66387013 conn(0x7fdee406b3a0 msgr2=0x7fdee406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee3200640 1 -- 172.21.15.32:0/66387013 shutdown_connections 2024-03-06T07:29:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.579+0000 7fdee3200640 1 -- 172.21.15.32:0/66387013 wait complete. 2024-03-06T07:29:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.581+0000 7fdee3200640 1 Processor -- start 2024-03-06T07:29:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.581+0000 7fdee3200640 1 -- start start 2024-03-06T07:29:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdee41366d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee40ae7b0 0x7fdee40abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdee40b0a90 0x7fdee40b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4145990 con 0x7fdec40039e0 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4145810 con 0x7fdee40b0a90 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee40cde00 con 0x7fdee40ae7b0 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdee41366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdee41366d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39618/0 (socket says 172.21.15.32:39618) 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.582+0000 7fdee1e00640 1 -- 172.21.15.32:0/1747619893 learned_addr learned my addr 172.21.15.32:0/1747619893 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee1e00640 1 -- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee40ae7b0 msgr2=0x7fdee40abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee1e00640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee40ae7b0 0x7fdee40abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee1e00640 1 -- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdee40b0a90 msgr2=0x7fdee40b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee2800640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdee40b0a90 0x7fdee40b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee1e00640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdee40b0a90 0x7fdee40b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee1e00640 1 -- 172.21.15.32:0/1747619893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee40ac940 con 0x7fdec40039e0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdee1e00640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdee41366d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fded4037040 tx=0x7fded4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4042020 con 0x7fdec40039e0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.583+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded403a040 con 0x7fdec40039e0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.584+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee40acc20 con 0x7fdec40039e0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.584+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4039070 con 0x7fdec40039e0 2024-03-06T07:29:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.584+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee40ad1e0 con 0x7fdec40039e0 2024-03-06T07:29:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.585+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee404f330 con 0x7fdec40039e0 2024-03-06T07:29:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.586+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fded4045020 con 0x7fdec40039e0 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.587+0000 7fdeda000640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdec006b4d0 0x7fdec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.587+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fded40ab6d0 con 0x7fdec40039e0 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.587+0000 7fdee1400640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdec006b4d0 0x7fdec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.588+0000 7fdee1400640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdec006b4d0 0x7fdec006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdecc0012e0 tx=0x7fdecc007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.592+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fded404c070 con 0x7fdec40039e0 2024-03-06T07:29:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.795+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdee406e240 con 0x7fdec40039e0 2024-03-06T07:29:39.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.796+0000 7fdeda000640 1 -- 172.21.15.32:0/1747619893 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fded407ee20 con 0x7fdec40039e0 2024-03-06T07:29:39.797 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdec006b4d0 msgr2=0x7fdec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdec006b4d0 0x7fdec006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdecc0012e0 tx=0x7fdecc007d30 comp rx=0 tx=0).stop 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 msgr2=0x7fdee41366d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdee41366d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fded4037040 tx=0x7fded4004730 comp rx=0 tx=0).stop 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 shutdown_connections 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdec006b4d0 0x7fdec006d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdee40b0a90 0x7fdee40b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdee40ae7b0 0x7fdee40abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 --2- 172.21.15.32:0/1747619893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec40039e0 0x7fdee41366d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 >> 172.21.15.32:0/1747619893 conn(0x7fdee406b3a0 msgr2=0x7fdee40b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.798+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 shutdown_connections 2024-03-06T07:29:39.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:39.799+0000 7fdee3200640 1 -- 172.21.15.32:0/1747619893 wait complete. 2024-03-06T07:29:39.828 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:29:39.828 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:29:39.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:40.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.048+0000 7fb087200640 1 Processor -- start 2024-03-06T07:29:40.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.049+0000 7fb087200640 1 -- start start 2024-03-06T07:29:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.050+0000 7fb087200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb08815e0b0 0x7fb0880cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.050+0000 7fb087200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0880d0e50 0x7fb0880b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.050+0000 7fb087200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880b3710 con 0x7fb0880d4a10 2024-03-06T07:29:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.050+0000 7fb087200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0880b09e0 con 0x7fb0880d0e50 2024-03-06T07:29:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.050+0000 7fb087200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0880b39f0 con 0x7fb08815e0b0 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0880d0e50 0x7fb0880b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0880d0e50 0x7fb0880b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39628/0 (socket says 172.21.15.32:39628) 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 -- 172.21.15.32:0/1505790785 learned_addr learned my addr 172.21.15.32:0/1505790785 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb085e00640 1 --2- 172.21.15.32:0/1505790785 >> v2:172.21.15.39:3300/0 conn(0x7fb08815e0b0 0x7fb0880cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 -- 172.21.15.32:0/1505790785 >> v2:172.21.15.39:3300/0 conn(0x7fb08815e0b0 msgr2=0x7fb0880cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 --2- 172.21.15.32:0/1505790785 >> v2:172.21.15.39:3300/0 conn(0x7fb08815e0b0 0x7fb0880cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 -- 172.21.15.32:0/1505790785 >> v1:172.21.15.39:6789/0 conn(0x7fb0880d4a10 legacy=0x7fb0880d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.051+0000 7fb086800640 1 -- 172.21.15.32:0/1505790785 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0880b3cd0 con 0x7fb0880d0e50 2024-03-06T07:29:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.052+0000 7fb086800640 1 --2- 172.21.15.32:0/1505790785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0880d0e50 0x7fb0880b44d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb0880b68c0 tx=0x7fb08002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab16a71a0c18866 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.052+0000 7fb085400640 1 -- 172.21.15.32:0/1505790785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08002ca80 con 0x7fb0880d0e50 2024-03-06T07:29:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb085400640 1 -- 172.21.15.32:0/1505790785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08002cc20 con 0x7fb0880d0e50 2024-03-06T07:29:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb085400640 1 -- 172.21.15.32:0/1505790785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0800343f0 con 0x7fb0880d0e50 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 -- 172.21.15.32:0/1505790785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0880d0e50 msgr2=0x7fb0880b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 --2- 172.21.15.32:0/1505790785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0880d0e50 0x7fb0880b44d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb0880b68c0 tx=0x7fb08002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 -- 172.21.15.32:0/1505790785 shutdown_connections 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 --2- 172.21.15.32:0/1505790785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0880d0e50 0x7fb0880b44d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 --2- 172.21.15.32:0/1505790785 >> v2:172.21.15.39:3300/0 conn(0x7fb08815e0b0 0x7fb0880cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 -- 172.21.15.32:0/1505790785 >> 172.21.15.32:0/1505790785 conn(0x7fb0880d3d30 msgr2=0x7fb0880d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 -- 172.21.15.32:0/1505790785 shutdown_connections 2024-03-06T07:29:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.053+0000 7fb087200640 1 -- 172.21.15.32:0/1505790785 wait complete. 2024-03-06T07:29:40.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.055+0000 7fb087200640 1 Processor -- start 2024-03-06T07:29:40.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.055+0000 7fb087200640 1 -- start start 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb087200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0880d0e50 0x7fb0880cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb087200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0880d4a10 0x7fb08815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb087200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 0x7fb0880cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb087200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880cfc60 con 0x7fb08815e0b0 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb087200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08815e490 con 0x7fb0880d0e50 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb087200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880cf850 con 0x7fb0880d4a10 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb086800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 0x7fb0880cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb086800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 0x7fb0880cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39638/0 (socket says 172.21.15.32:39638) 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.056+0000 7fb086800640 1 -- 172.21.15.32:0/3655411848 learned_addr learned my addr 172.21.15.32:0/3655411848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb085e00640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0880d0e50 0x7fb0880cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb07fe00640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0880d4a10 0x7fb08815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb086800640 1 -- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0880d4a10 msgr2=0x7fb08815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb086800640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0880d4a10 0x7fb08815a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb086800640 1 -- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0880d0e50 msgr2=0x7fb0880cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb086800640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0880d0e50 0x7fb0880cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb086800640 1 -- 172.21.15.32:0/3655411848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08815ac50 con 0x7fb08815e0b0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb085e00640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0880d0e50 0x7fb0880cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb07fe00640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0880d4a10 0x7fb08815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb086800640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 0x7fb0880cd740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb080036040 tx=0x7fb080004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.057+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080043020 con 0x7fb08815e0b0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.058+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08003e450 con 0x7fb08815e0b0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.058+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080009510 con 0x7fb08815e0b0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.058+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880cbdd0 con 0x7fb08815e0b0 2024-03-06T07:29:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.058+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0880cc390 con 0x7fb08815e0b0 2024-03-06T07:29:40.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.059+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08804f330 con 0x7fb08815e0b0 2024-03-06T07:29:40.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.060+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb080046020 con 0x7fb08815e0b0 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.061+0000 7fb07e000640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.061+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0800acb40 con 0x7fb08815e0b0 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.061+0000 7fb085e00640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.062+0000 7fb085e00640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb078006fd0 tx=0x7fb078008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.065+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb08004d030 con 0x7fb08815e0b0 2024-03-06T07:29:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.259+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0880d2af0 con 0x7fb08815e0b0 2024-03-06T07:29:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.262+0000 7fb07e000640 1 -- 172.21.15.32:0/3655411848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7fb08007b920 con 0x7fb08815e0b0 2024-03-06T07:29:40.264 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:40.264 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 msgr2=0x7fb06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb078006fd0 tx=0x7fb078008040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 msgr2=0x7fb0880cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 0x7fb0880cd740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb080036040 tx=0x7fb080004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 shutdown_connections 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815e0b0 0x7fb0880cd740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0880d4a10 0x7fb08815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 --2- 172.21.15.32:0/3655411848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0880d0e50 0x7fb0880cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 >> 172.21.15.32:0/3655411848 conn(0x7fb0880d3d30 msgr2=0x7fb0880b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 shutdown_connections 2024-03-06T07:29:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.265+0000 7fb087200640 1 -- 172.21.15.32:0/3655411848 wait complete. 2024-03-06T07:29:40.284 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:40.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:40.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.493+0000 7fbac0a00640 1 Processor -- start 2024-03-06T07:29:40.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.493+0000 7fbac0a00640 1 -- start start 2024-03-06T07:29:40.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.495+0000 7fbac0a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbabc165d50 0x7fbabc0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.495+0000 7fbac0a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.495+0000 7fbac0a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbabc166cb0 con 0x7fbabc162280 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.495+0000 7fbac0a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbabc0b4fd0 con 0x7fbabc165d50 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.495+0000 7fbac0a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc1661b0 con 0x7fbabc0b1b40 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.495+0000 7fbabaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbabc165d50 0x7fbabc0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbabc165d50 0x7fbabc0b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39648/0 (socket says 172.21.15.32:39648) 2024-03-06T07:29:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabaa00640 1 -- 172.21.15.32:0/1750821593 learned_addr learned my addr 172.21.15.32:0/1750821593 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabbe00640 1 --2- 172.21.15.32:0/1750821593 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabaa00640 1 -- 172.21.15.32:0/1750821593 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 msgr2=0x7fbabc0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabaa00640 1 --2- 172.21.15.32:0/1750821593 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc0b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabaa00640 1 -- 172.21.15.32:0/1750821593 >> v1:172.21.15.39:6789/0 conn(0x7fbabc162280 legacy=0x7fbabc0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.496+0000 7fbabaa00640 1 -- 172.21.15.32:0/1750821593 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc166410 con 0x7fbabc165d50 2024-03-06T07:29:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.497+0000 7fbabaa00640 1 --2- 172.21.15.32:0/1750821593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc0b4a90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40098a0 tx=0x7fbaa402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d9b7cbf9ff4cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.497+0000 7fbaba000640 1 -- 172.21.15.32:0/1750821593 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa40317c0 con 0x7fbabc165d50 2024-03-06T07:29:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbaba000640 1 -- 172.21.15.32:0/1750821593 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa4037070 con 0x7fbabc165d50 2024-03-06T07:29:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbaba000640 1 -- 172.21.15.32:0/1750821593 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4031b80 con 0x7fbabc165d50 2024-03-06T07:29:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 -- 172.21.15.32:0/1750821593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 msgr2=0x7fbabc0b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 --2- 172.21.15.32:0/1750821593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc0b4a90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40098a0 tx=0x7fbaa402b860 comp rx=0 tx=0).stop 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 -- 172.21.15.32:0/1750821593 shutdown_connections 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 --2- 172.21.15.32:0/1750821593 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 --2- 172.21.15.32:0/1750821593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc0b4a90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 -- 172.21.15.32:0/1750821593 >> 172.21.15.32:0/1750821593 conn(0x7fbabc06b3a0 msgr2=0x7fbabc06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.498+0000 7fbac0a00640 1 -- 172.21.15.32:0/1750821593 shutdown_connections 2024-03-06T07:29:40.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.499+0000 7fbac0a00640 1 -- 172.21.15.32:0/1750821593 wait complete. 2024-03-06T07:29:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.500+0000 7fbac0a00640 1 Processor -- start 2024-03-06T07:29:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.500+0000 7fbac0a00640 1 -- start start 2024-03-06T07:29:40.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.501+0000 7fbac0a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc14bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.501+0000 7fbac0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbabc162280 0x7fbabc15eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.501+0000 7fbac0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc1637e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.501+0000 7fbac0a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc071560 con 0x7fbabc165d50 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.501+0000 7fbac0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc1669d0 con 0x7fbabc162280 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.501+0000 7fbac0a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0716e0 con 0x7fbabc0b1b40 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc1637e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc1637e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39652/0 (socket says 172.21.15.32:39652) 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 -- 172.21.15.32:0/3378266209 learned_addr learned my addr 172.21.15.32:0/3378266209 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabaa00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbabc162280 0x7fbabc15eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabb400640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc14bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 -- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 msgr2=0x7fbabc14bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc14bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 -- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbabc162280 msgr2=0x7fbabc15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbabc162280 0x7fbabc15eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.502+0000 7fbabbe00640 1 -- 172.21.15.32:0/3378266209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc14c0f0 con 0x7fbabc165d50 2024-03-06T07:29:40.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.503+0000 7fbabbe00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc1637e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbab4007e40 tx=0x7fbab400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.503+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4014620 con 0x7fbabc165d50 2024-03-06T07:29:40.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.503+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4019070 con 0x7fbabc165d50 2024-03-06T07:29:40.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.503+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4014a20 con 0x7fbabc165d50 2024-03-06T07:29:40.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.503+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc15f8c0 con 0x7fbabc165d50 2024-03-06T07:29:40.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.503+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc14c8e0 con 0x7fbabc165d50 2024-03-06T07:29:40.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.504+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc14c3f0 con 0x7fbabc165d50 2024-03-06T07:29:40.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.505+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbab4014bc0 con 0x7fbabc165d50 2024-03-06T07:29:40.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.506+0000 7fbab3400640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fba9806b4d0 0x7fba9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.506+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbab408a9a0 con 0x7fbabc165d50 2024-03-06T07:29:40.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.506+0000 7fbabb400640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fba9806b4d0 0x7fba9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.507+0000 7fbabb400640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fba9806b4d0 0x7fba9806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbaac004ea0 tx=0x7fbaac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.511+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbab4059810 con 0x7fbabc165d50 2024-03-06T07:29:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.708+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbabc04f330 con 0x7fbabc165d50 2024-03-06T07:29:40.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.709+0000 7fbab3400640 1 -- 172.21.15.32:0/3378266209 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbab4059110 con 0x7fbabc165d50 2024-03-06T07:29:40.710 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.711+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fba9806b4d0 msgr2=0x7fba9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.711+0000 7fbac0a00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fba9806b4d0 0x7fba9806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbaac004ea0 tx=0x7fbaac009040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 msgr2=0x7fbabc1637e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc1637e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbab4007e40 tx=0x7fbab400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 shutdown_connections 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fba9806b4d0 0x7fba9806d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc165d50 0x7fbabc1637e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbabc162280 0x7fbabc15eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 --2- 172.21.15.32:0/3378266209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbabc0b1b40 0x7fbabc14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 >> 172.21.15.32:0/3378266209 conn(0x7fbabc06b3a0 msgr2=0x7fbabc0b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 shutdown_connections 2024-03-06T07:29:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.712+0000 7fbac0a00640 1 -- 172.21.15.32:0/3378266209 wait complete. 2024-03-06T07:29:40.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.938+0000 7f307e800640 1 Processor -- start 2024-03-06T07:29:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.938+0000 7f307e800640 1 -- start start 2024-03-06T07:29:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f30780b3c20 0x7f30780729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30780b4000 0x7f307815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307e800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3078071930 con 0x7f30780b1b40 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307e800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30780b4490 con 0x7f30780b4000 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307e800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3078071c10 con 0x7f30780b3c20 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30780b4000 0x7f307815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307de00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30780b1b40 0x7f307815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45880/0 (socket says 172.21.15.32:45880) 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.940+0000 7f307de00640 1 -- 172.21.15.32:0/1429964011 learned_addr learned my addr 172.21.15.32:0/1429964011 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.941+0000 7f307d400640 1 --2- 172.21.15.32:0/1429964011 >> v2:172.21.15.39:3300/0 conn(0x7f30780b3c20 0x7f30780729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.941+0000 7f307ca00640 1 -- 172.21.15.32:0/1429964011 >> v2:172.21.15.39:3300/0 conn(0x7f30780b3c20 msgr2=0x7f30780729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.941+0000 7f307ca00640 1 --2- 172.21.15.32:0/1429964011 >> v2:172.21.15.39:3300/0 conn(0x7f30780b3c20 0x7f30780729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.941+0000 7f307ca00640 1 -- 172.21.15.32:0/1429964011 >> v1:172.21.15.32:6789/0 conn(0x7f30780b1b40 legacy=0x7f307815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.941+0000 7f307ca00640 1 -- 172.21.15.32:0/1429964011 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078071ef0 con 0x7f30780b4000 2024-03-06T07:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.941+0000 7f3077e00640 1 -- 172.21.15.32:0/1429964011 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631215855 0 0) 0x7f3078071930 con 0x7f30780b1b40 2024-03-06T07:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.942+0000 7f307ca00640 1 --2- 172.21.15.32:0/1429964011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f307815b8a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f307806ead0 tx=0x7f306802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8721f7e771af7593 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.942+0000 7f3077e00640 1 -- 172.21.15.32:0/1429964011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068035020 con 0x7f30780b4000 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.942+0000 7f3077e00640 1 -- 172.21.15.32:0/1429964011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3068031830 con 0x7f30780b4000 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.942+0000 7f3077e00640 1 -- 172.21.15.32:0/1429964011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068031c00 con 0x7f30780b4000 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 -- 172.21.15.32:0/1429964011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 msgr2=0x7f307815b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 --2- 172.21.15.32:0/1429964011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f307815b8a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f307806ead0 tx=0x7f306802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 -- 172.21.15.32:0/1429964011 shutdown_connections 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 --2- 172.21.15.32:0/1429964011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f307815b8a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 --2- 172.21.15.32:0/1429964011 >> v2:172.21.15.39:3300/0 conn(0x7f30780b3c20 0x7f30780729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 -- 172.21.15.32:0/1429964011 >> 172.21.15.32:0/1429964011 conn(0x7f307806b3a0 msgr2=0x7f307806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 -- 172.21.15.32:0/1429964011 shutdown_connections 2024-03-06T07:29:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.943+0000 7f307e800640 1 -- 172.21.15.32:0/1429964011 wait complete. 2024-03-06T07:29:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.945+0000 7f307e800640 1 Processor -- start 2024-03-06T07:29:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.945+0000 7f307e800640 1 -- start start 2024-03-06T07:29:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30780b1b40 0x7f307814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307e800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30780b3c20 0x7f3078155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f3078159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307e800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307815bde0 con 0x7f30780b4000 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30780724b0 con 0x7f30780b1b40 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307e800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307815bf60 con 0x7f30780b3c20 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f3078159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f3078159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39680/0 (socket says 172.21.15.32:39680) 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307de00640 1 -- 172.21.15.32:0/3705107454 learned_addr learned my addr 172.21.15.32:0/3705107454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.946+0000 7f307d400640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30780b1b40 0x7f307814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307de00640 1 -- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30780b3c20 msgr2=0x7f3078155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307de00640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30780b3c20 0x7f3078155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307de00640 1 -- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30780b1b40 msgr2=0x7f307814e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307de00640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30780b1b40 0x7f307814e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307de00640 1 -- 172.21.15.32:0/3705107454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078156100 con 0x7f30780b4000 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307de00640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f3078159cd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f307000c7a0 tx=0x7f307000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070018070 con 0x7f30780b4000 2024-03-06T07:29:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307000a640 con 0x7f30780b4000 2024-03-06T07:29:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307000aa70 con 0x7f30780b4000 2024-03-06T07:29:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30781563e0 con 0x7f30780b4000 2024-03-06T07:29:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.947+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30781569a0 con 0x7f30780b4000 2024-03-06T07:29:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.948+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30780cbb50 con 0x7f30780b4000 2024-03-06T07:29:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.949+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f307000ac10 con 0x7f30780b4000 2024-03-06T07:29:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.950+0000 7f3075600640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f305c06b4d0 0x7f305c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.950+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30700899a0 con 0x7f30780b4000 2024-03-06T07:29:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.951+0000 7f307d400640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f305c06b4d0 0x7f305c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.951+0000 7f307d400640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f305c06b4d0 0x7f305c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f306c006fd0 tx=0x7f306c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:40.955+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30780cbb50 con 0x7f30780b4000 2024-03-06T07:29:41.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.152+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3078074ab0 con 0x7f30780b4000 2024-03-06T07:29:41.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.153+0000 7f3075600640 1 -- 172.21.15.32:0/3705107454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30700586f0 con 0x7f30780b4000 2024-03-06T07:29:41.154 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:41.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f305c06b4d0 msgr2=0x7f305c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f305c06b4d0 0x7f305c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f306c006fd0 tx=0x7f306c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 msgr2=0x7f3078159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f3078159cd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f307000c7a0 tx=0x7f307000cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 shutdown_connections 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f305c06b4d0 0x7f305c06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30780b4000 0x7f3078159cd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30780b3c20 0x7f3078155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 --2- 172.21.15.32:0/3705107454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30780b1b40 0x7f307814e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 >> 172.21.15.32:0/3705107454 conn(0x7f307806b3a0 msgr2=0x7f307815b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 shutdown_connections 2024-03-06T07:29:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.156+0000 7f307e800640 1 -- 172.21.15.32:0/3705107454 wait complete. 2024-03-06T07:29:41.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.394+0000 7fd813e00640 1 Processor -- start 2024-03-06T07:29:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.394+0000 7fd813e00640 1 -- start start 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.395+0000 7fd813e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8140b2250 0x7fd814166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.395+0000 7fd813e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8140b3fe0 con 0x7fd814145430 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.395+0000 7fd813e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81415fe40 con 0x7fd81415f810 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.395+0000 7fd813e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd814160120 con 0x7fd8140b2250 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd813400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8140b2250 0x7fd814166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd812a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd814145430 0x7fd81415cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45892/0 (socket says 172.21.15.32:45892) 2024-03-06T07:29:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd812a00640 1 -- 172.21.15.32:0/1130592482 learned_addr learned my addr 172.21.15.32:0/1130592482 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd813400640 1 -- 172.21.15.32:0/1130592482 >> v1:172.21.15.39:6789/0 conn(0x7fd81415f810 legacy=0x7fd8140b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd813400640 1 -- 172.21.15.32:0/1130592482 >> v1:172.21.15.32:6789/0 conn(0x7fd814145430 legacy=0x7fd81415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd813400640 1 -- 172.21.15.32:0/1130592482 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd814160400 con 0x7fd8140b2250 2024-03-06T07:29:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.396+0000 7fd812000640 1 -- 172.21.15.32:0/1130592482 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151701573 0 0) 0x7fd8140b3fe0 con 0x7fd814145430 2024-03-06T07:29:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.397+0000 7fd813400640 1 --2- 172.21.15.32:0/1130592482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8140b2250 0x7fd814166ba0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009d90 tx=0x7fd80c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bc46a3cc930fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.397+0000 7fd812000640 1 -- 172.21.15.32:0/1130592482 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c039070 con 0x7fd8140b2250 2024-03-06T07:29:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.397+0000 7fd812000640 1 -- 172.21.15.32:0/1130592482 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80c02cc60 con 0x7fd8140b2250 2024-03-06T07:29:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd812000640 1 -- 172.21.15.32:0/1130592482 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c034660 con 0x7fd8140b2250 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 -- 172.21.15.32:0/1130592482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8140b2250 msgr2=0x7fd814166ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 --2- 172.21.15.32:0/1130592482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8140b2250 0x7fd814166ba0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009d90 tx=0x7fd80c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 -- 172.21.15.32:0/1130592482 shutdown_connections 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 --2- 172.21.15.32:0/1130592482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8140b2250 0x7fd814166ba0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 -- 172.21.15.32:0/1130592482 >> 172.21.15.32:0/1130592482 conn(0x7fd81406b3a0 msgr2=0x7fd81406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 -- 172.21.15.32:0/1130592482 shutdown_connections 2024-03-06T07:29:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.398+0000 7fd813e00640 1 -- 172.21.15.32:0/1130592482 wait complete. 2024-03-06T07:29:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.400+0000 7fd813e00640 1 Processor -- start 2024-03-06T07:29:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.400+0000 7fd813e00640 1 -- start start 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd813e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd8140b2250 0x7fd81413a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd813e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 0x7fd8140ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd813e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd81415f810 0x7fd814071420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd813e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814145990 con 0x7fd814145430 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd813e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814145810 con 0x7fd81415f810 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd813e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8140cde00 con 0x7fd8140b2250 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd80be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 0x7fd8140ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd80be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 0x7fd8140ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39712/0 (socket says 172.21.15.32:39712) 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.401+0000 7fd80be00640 1 -- 172.21.15.32:0/4031443098 learned_addr learned my addr 172.21.15.32:0/4031443098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd813400640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd81415f810 0x7fd814071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd812a00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd8140b2250 0x7fd81413a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80be00640 1 -- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd8140b2250 msgr2=0x7fd81413a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80be00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd8140b2250 0x7fd81413a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80be00640 1 -- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd81415f810 msgr2=0x7fd814071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80be00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd81415f810 0x7fd814071420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80be00640 1 -- 172.21.15.32:0/4031443098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8140aed50 con 0x7fd814145430 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd813400640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd81415f810 0x7fd814071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80be00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 0x7fd8140ae290 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00b3f0 tx=0x7fd7fc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc015020 con 0x7fd814145430 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc011830 con 0x7fd814145430 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc011c20 con 0x7fd814145430 2024-03-06T07:29:41.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.402+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8140af030 con 0x7fd814145430 2024-03-06T07:29:41.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.403+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd814153840 con 0x7fd814145430 2024-03-06T07:29:41.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.404+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81413ace0 con 0x7fd814145430 2024-03-06T07:29:41.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.405+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7fc018020 con 0x7fd814145430 2024-03-06T07:29:41.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.406+0000 7fd80b400640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd7f006b4d0 0x7fd7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.406+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7fc089ea0 con 0x7fd814145430 2024-03-06T07:29:41.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.406+0000 7fd812a00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd7f006b4d0 0x7fd7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.407+0000 7fd812a00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd7f006b4d0 0x7fd7f006d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd804007fd0 tx=0x7fd804009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.411+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7fc01f030 con 0x7fd814145430 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.601+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd81404f330 con 0x7fd814145430 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.602+0000 7fd80b400640 1 -- 172.21.15.32:0/4031443098 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd7fc0172c0 con 0x7fd814145430 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:41.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd7f006b4d0 msgr2=0x7fd7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd7f006b4d0 0x7fd7f006d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd804007fd0 tx=0x7fd804009040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 msgr2=0x7fd8140ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 0x7fd8140ae290 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00b3f0 tx=0x7fd7fc00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 shutdown_connections 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd7f006b4d0 0x7fd7f006d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd81415f810 0x7fd814071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814145430 0x7fd8140ae290 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 --2- 172.21.15.32:0/4031443098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd8140b2250 0x7fd81413a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 >> 172.21.15.32:0/4031443098 conn(0x7fd81406b3a0 msgr2=0x7fd81415e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 shutdown_connections 2024-03-06T07:29:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.605+0000 7fd813e00640 1 -- 172.21.15.32:0/4031443098 wait complete. 2024-03-06T07:29:41.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:29:41.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.841+0000 7fe7b8e00640 1 Processor -- start 2024-03-06T07:29:41.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.842+0000 7fe7b8e00640 1 -- start start 2024-03-06T07:29:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b8e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7b415e0b0 0x7fe7b40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b8e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40b3710 con 0x7fe7b40d0e50 2024-03-06T07:29:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b8e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40b39f0 con 0x7fe7b40d4a10 2024-03-06T07:29:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b8e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b40b09e0 con 0x7fe7b415e0b0 2024-03-06T07:29:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7b415e0b0 0x7fe7b40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7b40d0e50 0x7fe7b40b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45896/0 (socket says 172.21.15.32:45896) 2024-03-06T07:29:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.843+0000 7fe7b3e00640 1 -- 172.21.15.32:0/2649980047 learned_addr learned my addr 172.21.15.32:0/2649980047 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.844+0000 7fe7b2000640 1 -- 172.21.15.32:0/2649980047 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112300187 0 0) 0x7fe7b40b3710 con 0x7fe7b40d0e50 2024-03-06T07:29:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.844+0000 7fe7b3400640 1 -- 172.21.15.32:0/2649980047 >> v1:172.21.15.39:6789/0 conn(0x7fe7b40d4a10 legacy=0x7fe7b40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.844+0000 7fe7b3400640 1 -- 172.21.15.32:0/2649980047 >> v1:172.21.15.32:6789/0 conn(0x7fe7b40d0e50 legacy=0x7fe7b40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.844+0000 7fe7b3400640 1 -- 172.21.15.32:0/2649980047 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40b3cd0 con 0x7fe7b415e0b0 2024-03-06T07:29:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.845+0000 7fe7b3400640 1 --2- 172.21.15.32:0/2649980047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b415e0b0 0x7fe7b40cf310 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009d90 tx=0x7fe7a402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb2d1c203095354 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.845+0000 7fe7b2000640 1 -- 172.21.15.32:0/2649980047 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4039070 con 0x7fe7b415e0b0 2024-03-06T07:29:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.845+0000 7fe7b2000640 1 -- 172.21.15.32:0/2649980047 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a402cd00 con 0x7fe7b415e0b0 2024-03-06T07:29:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.845+0000 7fe7b2000640 1 -- 172.21.15.32:0/2649980047 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4034660 con 0x7fe7b415e0b0 2024-03-06T07:29:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 -- 172.21.15.32:0/2649980047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b415e0b0 msgr2=0x7fe7b40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/2649980047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b415e0b0 0x7fe7b40cf310 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009d90 tx=0x7fe7a402bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 -- 172.21.15.32:0/2649980047 shutdown_connections 2024-03-06T07:29:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/2649980047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b415e0b0 0x7fe7b40cf310 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 -- 172.21.15.32:0/2649980047 >> 172.21.15.32:0/2649980047 conn(0x7fe7b40d3d30 msgr2=0x7fe7b40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 -- 172.21.15.32:0/2649980047 shutdown_connections 2024-03-06T07:29:41.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.846+0000 7fe7b8e00640 1 -- 172.21.15.32:0/2649980047 wait complete. 2024-03-06T07:29:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.848+0000 7fe7b8e00640 1 Processor -- start 2024-03-06T07:29:41.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.848+0000 7fe7b8e00640 1 -- start start 2024-03-06T07:29:41.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b40d0e50 0x7fe7b40cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 0x7fe7b414f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b8e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7b415e0b0 0x7fe7b415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b8e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b40cfc60 con 0x7fe7b40d4a10 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b415e490 con 0x7fe7b40d0e50 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b8e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b40cf850 con 0x7fe7b415e0b0 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 0x7fe7b414f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 0x7fe7b414f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39732/0 (socket says 172.21.15.32:39732) 2024-03-06T07:29:41.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.849+0000 7fe7b2a00640 1 -- 172.21.15.32:0/1434070418 learned_addr learned my addr 172.21.15.32:0/1434070418 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b3400640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b40d0e50 0x7fe7b40cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b3e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7b415e0b0 0x7fe7b415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b2a00640 1 -- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7b415e0b0 msgr2=0x7fe7b415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b2a00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7b415e0b0 0x7fe7b415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b2a00640 1 -- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b40d0e50 msgr2=0x7fe7b40cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b2a00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b40d0e50 0x7fe7b40cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b2a00640 1 -- 172.21.15.32:0/1434070418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b414f850 con 0x7fe7b40d4a10 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b3e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7b415e0b0 0x7fe7b415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b2a00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 0x7fe7b414f310 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c430 tx=0x7fe7a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0017070 con 0x7fe7b40d4a10 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a00052d0 con 0x7fe7b40d4a10 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0003830 con 0x7fe7b40d4a10 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.850+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b40cc8e0 con 0x7fe7b40d4a10 2024-03-06T07:29:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.851+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b40ccea0 con 0x7fe7b40d4a10 2024-03-06T07:29:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.852+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b40d0720 con 0x7fe7b40d4a10 2024-03-06T07:29:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.853+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7a001a070 con 0x7fe7b40d4a10 2024-03-06T07:29:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.854+0000 7fe7ab400640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe79406b500 0x7fe79406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.854+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7a0089610 con 0x7fe7b40d4a10 2024-03-06T07:29:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.854+0000 7fe7b3400640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe79406b500 0x7fe79406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.855+0000 7fe7b3400640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe79406b500 0x7fe79406d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4036000 tx=0x7fe7a4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:41.859+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7b40d0720 con 0x7fe7b40d4a10 2024-03-06T07:29:42.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.127+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fe7b40b3670 con 0x7fe7b40d4a10 2024-03-06T07:29:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.624+0000 7fe7ab400640 1 -- 172.21.15.32:0/1434070418 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v58) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe7b414f850 con 0x7fe7b40d4a10 2024-03-06T07:29:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.628+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe79406b500 msgr2=0x7fe79406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.628+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe79406b500 0x7fe79406d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4036000 tx=0x7fe7a4030000 comp rx=0 tx=0).stop 2024-03-06T07:29:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.628+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 msgr2=0x7fe7b414f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.628+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 0x7fe7b414f310 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c430 tx=0x7fe7a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 shutdown_connections 2024-03-06T07:29:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe79406b500 0x7fe79406d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7b415e0b0 0x7fe7b415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7b40d4a10 0x7fe7b414f310 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 --2- 172.21.15.32:0/1434070418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b40d0e50 0x7fe7b40cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 >> 172.21.15.32:0/1434070418 conn(0x7fe7b40d3d30 msgr2=0x7fe7b40b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 shutdown_connections 2024-03-06T07:29:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.629+0000 7fe7b8e00640 1 -- 172.21.15.32:0/1434070418 wait complete. 2024-03-06T07:29:42.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:29:42.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.877+0000 7f3dce400640 1 Processor -- start 2024-03-06T07:29:42.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.877+0000 7f3dce400640 1 -- start start 2024-03-06T07:29:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.878+0000 7f3dce400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3dc80d0d40 0x7f3dc80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.878+0000 7f3dce400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80ceca0 con 0x7f3dc80d4860 2024-03-06T07:29:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.878+0000 7f3dce400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80cef80 con 0x7f3dc815dee0 2024-03-06T07:29:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.878+0000 7f3dce400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc80cf5d0 con 0x7f3dc80d0d40 2024-03-06T07:29:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3dc80d4860 0x7f3dc80d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45908/0 (socket says 172.21.15.32:45908) 2024-03-06T07:29:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7e00640 1 -- 172.21.15.32:0/4178289035 learned_addr learned my addr 172.21.15.32:0/4178289035 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dcda00640 1 --2- 172.21.15.32:0/4178289035 >> v2:172.21.15.39:3300/0 conn(0x7f3dc80d0d40 0x7f3dc80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300996393 0 0) 0x7f3dc80ceca0 con 0x7f3dc80d4860 2024-03-06T07:29:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0003520 con 0x7f3dc80d4860 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587872582 0 0) 0x7f3db0003520 con 0x7f3dc80d4860 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 >> v2:172.21.15.39:3300/0 conn(0x7f3dc80d0d40 msgr2=0x7f3dc80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 --2- 172.21.15.32:0/4178289035 >> v2:172.21.15.39:3300/0 conn(0x7f3dc80d0d40 0x7f3dc80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 >> v1:172.21.15.39:6789/0 conn(0x7f3dc815dee0 legacy=0x7f3dc80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc80cf260 con 0x7f3dc80d4860 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.879+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3db8001670 con 0x7f3dc80d4860 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 >> v1:172.21.15.32:6789/0 conn(0x7f3dc80d4860 legacy=0x7f3dc80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db00039b0 0x7f3db0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db00051c0 0x7f3db000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80cf5d0 con 0x7f3db0004800 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db000c060 con 0x7f3db00051c0 2024-03-06T07:29:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db000c260 con 0x7f3db00039b0 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dcda00640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db00051c0 0x7f3db000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dcd000640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db00039b0 0x7f3db0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7e00640 1 -- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db00039b0 msgr2=0x7f3db0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db00039b0 0x7f3db0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7e00640 1 -- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db00051c0 msgr2=0x7f3db000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db00051c0 0x7f3db000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dc7e00640 1 -- 172.21.15.32:0/4178289035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db0003520 con 0x7f3db0004800 2024-03-06T07:29:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dcda00640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db00051c0 0x7f3db000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dcda00640 1 -- 172.21.15.32:0/4178289035 reap_dead start 2024-03-06T07:29:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.880+0000 7f3dcd000640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db00039b0 0x7f3db0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.881+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3db0004800 0x7f3db0004c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c240 tx=0x7f3db0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc46e0f351cdfdd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8007d10 con 0x7f3db0004800 2024-03-06T07:29:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8030070 con 0x7f3db0004800 2024-03-06T07:29:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dc7400640 1 -- 172.21.15.32:0/4178289035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8005d70 con 0x7f3db0004800 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dce400640 1 -- 172.21.15.32:0/4178289035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3db0004800 msgr2=0x7f3db0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dce400640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3db0004800 0x7f3db0004c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c240 tx=0x7f3db0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dce400640 1 -- 172.21.15.32:0/4178289035 shutdown_connections 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dce400640 1 --2- 172.21.15.32:0/4178289035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3db0004800 0x7f3db0004c80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.882+0000 7f3dce400640 1 -- 172.21.15.32:0/4178289035 >> 172.21.15.32:0/4178289035 conn(0x7f3dc80d3cd0 msgr2=0x7f3dc80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.883+0000 7f3dce400640 1 -- 172.21.15.32:0/4178289035 shutdown_connections 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.883+0000 7f3dce400640 1 -- 172.21.15.32:0/4178289035 wait complete. 2024-03-06T07:29:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.884+0000 7f3dce400640 1 Processor -- start 2024-03-06T07:29:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.885+0000 7f3dce400640 1 -- start start 2024-03-06T07:29:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dce400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db0004800 0x7f3dc81f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dce400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 0x7f3dc814dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dce400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc80c95c0 0x7f3dc80c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dce400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80cfa60 con 0x7f3dc8160100 2024-03-06T07:29:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dce400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc815e2c0 con 0x7f3dc80c95c0 2024-03-06T07:29:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dce400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc80cfbe0 con 0x7f3db0004800 2024-03-06T07:29:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 0x7f3dc814dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 0x7f3dc814dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39756/0 (socket says 172.21.15.32:39756) 2024-03-06T07:29:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 -- 172.21.15.32:0/436993426 learned_addr learned my addr 172.21.15.32:0/436993426 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dcda00640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc80c95c0 0x7f3dc80c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dcd000640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db0004800 0x7f3dc81f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 -- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db0004800 msgr2=0x7f3dc81f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db0004800 0x7f3dc81f6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 -- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc80c95c0 msgr2=0x7f3dc80c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc80c95c0 0x7f3dc80c9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.886+0000 7f3dc7e00640 1 -- 172.21.15.32:0/436993426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc814e0e0 con 0x7f3dc8160100 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.887+0000 7f3dc7e00640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 0x7f3dc814dba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c360 tx=0x7f3db802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.887+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8007d10 con 0x7f3dc8160100 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.887+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8004bf0 con 0x7f3dc8160100 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.887+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8007070 con 0x7f3dc8160100 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.887+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc8161740 con 0x7f3dc8160100 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.887+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc8160980 con 0x7f3dc8160100 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.888+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc804f330 con 0x7f3dc8160100 2024-03-06T07:29:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.889+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3db8048020 con 0x7f3dc8160100 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.890+0000 7f3dc4c00640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3dac06b4d0 0x7f3dac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.890+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3db80ab650 con 0x7f3dc8160100 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.891+0000 7f3dcd000640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3dac06b4d0 0x7f3dac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.891+0000 7f3dcd000640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3dac06b4d0 0x7f3dac06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8069ad0 tx=0x7f3dbc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:42.894+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3dc814e0e0 con 0x7f3dc8160100 2024-03-06T07:29:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.090+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3dc808d7d0 con 0x7f3dc8160100 2024-03-06T07:29:43.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.097+0000 7f3dc4c00640 1 -- 172.21.15.32:0/436993426 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3db807a430 con 0x7f3dc8160100 2024-03-06T07:29:43.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3dac06b4d0 msgr2=0x7f3dac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3dac06b4d0 0x7f3dac06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8069ad0 tx=0x7f3dbc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 msgr2=0x7f3dc814dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 0x7f3dc814dba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3db800c360 tx=0x7f3db802e070 comp rx=0 tx=0).stop 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 shutdown_connections 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3dac06b4d0 0x7f3dac06d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc80c95c0 0x7f3dc80c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc8160100 0x7f3dc814dba0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 --2- 172.21.15.32:0/436993426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3db0004800 0x7f3dc81f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 >> 172.21.15.32:0/436993426 conn(0x7f3dc80d3cd0 msgr2=0x7f3dc80b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 shutdown_connections 2024-03-06T07:29:43.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.100+0000 7f3dce400640 1 -- 172.21.15.32:0/436993426 wait complete. 2024-03-06T07:29:43.121 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:43.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:43.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:43.365 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:43.366 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:43.366 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:43.425 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:43.425 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:43.481 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:43.481 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:43.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:43.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.754+0000 7f9f7f200640 1 Processor -- start 2024-03-06T07:29:43.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.754+0000 7f9f7f200640 1 -- start start 2024-03-06T07:29:43.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.756+0000 7f9f7f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f800cd060 0x7f9f80161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.756+0000 7f9f7f200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f801627f0 con 0x7f9f80162000 2024-03-06T07:29:43.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.756+0000 7f9f7f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8006c860 con 0x7f9f8015efc0 2024-03-06T07:29:43.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.756+0000 7f9f7f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8015f3a0 con 0x7f9f800cd060 2024-03-06T07:29:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.756+0000 7f9f7e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9f80162000 0x7f9f80070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45916/0 (socket says 172.21.15.32:45916) 2024-03-06T07:29:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.756+0000 7f9f7e800640 1 -- 172.21.15.32:0/801118926 learned_addr learned my addr 172.21.15.32:0/801118926 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f75400640 1 --2- 172.21.15.32:0/801118926 >> v2:172.21.15.39:3300/0 conn(0x7f9f800cd060 0x7f9f80161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090240428 0 0) 0x7f9f801627f0 con 0x7f9f80162000 2024-03-06T07:29:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f64003520 con 0x7f9f80162000 2024-03-06T07:29:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751208069 0 0) 0x7f9f64003520 con 0x7f9f80162000 2024-03-06T07:29:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 >> v2:172.21.15.39:3300/0 conn(0x7f9f800cd060 msgr2=0x7f9f80161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 --2- 172.21.15.32:0/801118926 >> v2:172.21.15.39:3300/0 conn(0x7f9f800cd060 0x7f9f80161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 >> v1:172.21.15.39:6789/0 conn(0x7f9f8015efc0 legacy=0x7f9f800cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8006cb40 con 0x7f9f80162000 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f9f78001670 con 0x7f9f80162000 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.757+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 >> v1:172.21.15.32:6789/0 conn(0x7f9f80162000 legacy=0x7f9f80070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7d400640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7d400640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7d400640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015f3a0 con 0x7f9f640039b0 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400c060 con 0x7f9f640051c0 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400c260 con 0x7f9f64004800 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7de00640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f7e800640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.758+0000 7f9f75400640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7de00640 1 -- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f64004800 msgr2=0x7f9f64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7de00640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7de00640 1 -- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f640051c0 msgr2=0x7f9f6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7de00640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7de00640 1 -- 172.21.15.32:0/801118926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f64003520 con 0x7f9f640039b0 2024-03-06T07:29:43.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7e800640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:43.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.759+0000 7f9f7e800640 1 -- 172.21.15.32:0/801118926 reap_dead start 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.760+0000 7f9f7de00640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9f70001e50 tx=0x7f9f64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ca898a5eeea8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70001a70 con 0x7f9f640039b0 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70003b70 con 0x7f9f640039b0 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7d400640 1 -- 172.21.15.32:0/801118926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700050d0 con 0x7f9f640039b0 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 -- 172.21.15.32:0/801118926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 msgr2=0x7f9f64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9f70001e50 tx=0x7f9f64005830 comp rx=0 tx=0).stop 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 -- 172.21.15.32:0/801118926 shutdown_connections 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 --2- 172.21.15.32:0/801118926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 -- 172.21.15.32:0/801118926 >> 172.21.15.32:0/801118926 conn(0x7f9f8006b3d0 msgr2=0x7f9f8006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 -- 172.21.15.32:0/801118926 shutdown_connections 2024-03-06T07:29:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.761+0000 7f9f7f200640 1 -- 172.21.15.32:0/801118926 wait complete. 2024-03-06T07:29:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.763+0000 7f9f7f200640 1 Processor -- start 2024-03-06T07:29:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.763+0000 7f9f7f200640 1 -- start start 2024-03-06T07:29:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f801e7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f8015d710 0x7f9f801f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f801e8c10 0x7f9f8015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80162470 con 0x7f9f640039b0 2024-03-06T07:29:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8006d190 con 0x7f9f801e8c10 2024-03-06T07:29:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8006d620 con 0x7f9f8015d710 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f801e7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f801e7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39776/0 (socket says 172.21.15.32:39776) 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.764+0000 7f9f7de00640 1 -- 172.21.15.32:0/2998576699 learned_addr learned my addr 172.21.15.32:0/2998576699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f75400640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f8015d710 0x7f9f801f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7e800640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f801e8c10 0x7f9f8015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7de00640 1 -- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f8015d710 msgr2=0x7f9f801f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7de00640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f8015d710 0x7f9f801f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7de00640 1 -- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f801e8c10 msgr2=0x7f9f8015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7de00640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f801e8c10 0x7f9f8015a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7de00640 1 -- 172.21.15.32:0/2998576699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f801e7b70 con 0x7f9f640039b0 2024-03-06T07:29:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.765+0000 7f9f7de00640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f801e7500 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9f70005040 tx=0x7f9f700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.766+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70006b50 con 0x7f9f640039b0 2024-03-06T07:29:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.766+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70006cf0 con 0x7f9f640039b0 2024-03-06T07:29:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.766+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70007250 con 0x7f9f640039b0 2024-03-06T07:29:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.766+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f801e7e50 con 0x7f9f640039b0 2024-03-06T07:29:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.766+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f801e8410 con 0x7f9f640039b0 2024-03-06T07:29:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.767+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8004f330 con 0x7f9f640039b0 2024-03-06T07:29:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.768+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f70003a30 con 0x7f9f640039b0 2024-03-06T07:29:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.769+0000 7f9f76a00640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.769+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f700abe50 con 0x7f9f640039b0 2024-03-06T07:29:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.770+0000 7f9f75400640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.770+0000 7f9f75400640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f80159fc0 tx=0x7f9f68008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.774+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f70033020 con 0x7f9f640039b0 2024-03-06T07:29:43.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.976+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f801e87f0 con 0x7f9f640039b0 2024-03-06T07:29:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.978+0000 7f9f76a00640 1 -- 172.21.15.32:0/2998576699 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f9f7007acc0 con 0x7f9f640039b0 2024-03-06T07:29:43.980 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:43.980 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f5c06b500 msgr2=0x7f9f5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9f80159fc0 tx=0x7f9f68008040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 msgr2=0x7f9f801e7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f801e7500 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9f70005040 tx=0x7f9f700014a0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 shutdown_connections 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f801e8c10 0x7f9f8015a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f8015d710 0x7f9f801f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 --2- 172.21.15.32:0/2998576699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f640039b0 0x7f9f801e7500 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.981+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 >> 172.21.15.32:0/2998576699 conn(0x7f9f8006b3d0 msgr2=0x7f9f800b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.982+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 shutdown_connections 2024-03-06T07:29:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:43.982+0000 7f9f7f200640 1 -- 172.21.15.32:0/2998576699 wait complete. 2024-03-06T07:29:44.004 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:44.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:44.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.229+0000 7fe609200640 1 Processor -- start 2024-03-06T07:29:44.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.229+0000 7fe609200640 1 -- start start 2024-03-06T07:29:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.230+0000 7fe609200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe60415bf20 0x7fe6040acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.230+0000 7fe609200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe6040ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.230+0000 7fe609200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604072df0 con 0x7fe604072520 2024-03-06T07:29:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.230+0000 7fe609200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6040cdab0 con 0x7fe60415bf20 2024-03-06T07:29:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.230+0000 7fe609200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b4d70 con 0x7fe604071e20 2024-03-06T07:29:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.231+0000 7fe5f9400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe604072520 0x7fe6040cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45928/0 (socket says 172.21.15.32:45928) 2024-03-06T07:29:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.231+0000 7fe5f9400640 1 -- 172.21.15.32:0/1690866159 learned_addr learned my addr 172.21.15.32:0/1690866159 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.231+0000 7fe601e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe60415bf20 0x7fe6040acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.231+0000 7fe602800640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe6040ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe601400640 1 -- 172.21.15.32:0/1690866159 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175240390 0 0) 0x7fe604072df0 con 0x7fe604072520 2024-03-06T07:29:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe601400640 1 -- 172.21.15.32:0/1690866159 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e8003520 con 0x7fe604072520 2024-03-06T07:29:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe601e00640 1 -- 172.21.15.32:0/1690866159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 msgr2=0x7fe6040ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe601e00640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe6040ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe601e00640 1 -- 172.21.15.32:0/1690866159 >> v1:172.21.15.32:6789/0 conn(0x7fe604072520 legacy=0x7fe6040cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe601e00640 1 -- 172.21.15.32:0/1690866159 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040b5050 con 0x7fe60415bf20 2024-03-06T07:29:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.232+0000 7fe602800640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe6040ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.233+0000 7fe601e00640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe60415bf20 0x7fe6040acd40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009d90 tx=0x7fe5f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1a93264cfdaa59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.233+0000 7fe601400640 1 -- 172.21.15.32:0/1690866159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4039070 con 0x7fe60415bf20 2024-03-06T07:29:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.233+0000 7fe601400640 1 -- 172.21.15.32:0/1690866159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f402cc50 con 0x7fe60415bf20 2024-03-06T07:29:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.233+0000 7fe601400640 1 -- 172.21.15.32:0/1690866159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4034440 con 0x7fe60415bf20 2024-03-06T07:29:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 -- 172.21.15.32:0/1690866159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe60415bf20 msgr2=0x7fe6040acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe60415bf20 0x7fe6040acd40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009d90 tx=0x7fe5f402bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 -- 172.21.15.32:0/1690866159 shutdown_connections 2024-03-06T07:29:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe6040ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 --2- 172.21.15.32:0/1690866159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe60415bf20 0x7fe6040acd40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 -- 172.21.15.32:0/1690866159 >> 172.21.15.32:0/1690866159 conn(0x7fe60406b3d0 msgr2=0x7fe60406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 -- 172.21.15.32:0/1690866159 shutdown_connections 2024-03-06T07:29:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.234+0000 7fe609200640 1 -- 172.21.15.32:0/1690866159 wait complete. 2024-03-06T07:29:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.236+0000 7fe609200640 1 Processor -- start 2024-03-06T07:29:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.236+0000 7fe609200640 1 -- start start 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe609200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe60414d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe609200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 0x7fe6040afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe609200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe60415bf20 0x7fe60415cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe609200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b4b10 con 0x7fe604072520 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe609200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b4990 con 0x7fe60415bf20 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe609200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040ce6c0 con 0x7fe604071e20 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe5f9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 0x7fe6040afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe5f9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 0x7fe6040afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39794/0 (socket says 172.21.15.32:39794) 2024-03-06T07:29:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe5f9400640 1 -- 172.21.15.32:0/3324765838 learned_addr learned my addr 172.21.15.32:0/3324765838 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.237+0000 7fe602800640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe60415bf20 0x7fe60415cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe601e00640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe60414d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5f9400640 1 -- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 msgr2=0x7fe60414d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5f9400640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe60414d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5f9400640 1 -- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe60415bf20 msgr2=0x7fe60415cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5f9400640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe60415bf20 0x7fe60415cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5f9400640 1 -- 172.21.15.32:0/3324765838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040cb0e0 con 0x7fe604072520 2024-03-06T07:29:44.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5f9400640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 0x7fe6040afd70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00c390 tx=0x7fe5ec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec017070 con 0x7fe604072520 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec005ce0 con 0x7fe604072520 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0072f0 con 0x7fe604072520 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.238+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6040c9230 con 0x7fe604072520 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.239+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6040c97f0 con 0x7fe604072520 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.240+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60404f330 con 0x7fe604072520 2024-03-06T07:29:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.241+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5ec01a070 con 0x7fe604072520 2024-03-06T07:29:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.241+0000 7fe5faa00640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.242+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5ec089630 con 0x7fe604072520 2024-03-06T07:29:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.242+0000 7fe601e00640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.243+0000 7fe601e00640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe5e006b4d0 0x7fe5e006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4036000 tx=0x7fe5f4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.246+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5ec0584a0 con 0x7fe604072520 2024-03-06T07:29:44.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.446+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe60415f720 con 0x7fe604072520 2024-03-06T07:29:44.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.447+0000 7fe5faa00640 1 -- 172.21.15.32:0/3324765838 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5ec057da0 con 0x7fe604072520 2024-03-06T07:29:44.449 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:44.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.449+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe5e006b4d0 msgr2=0x7fe5e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe5e006b4d0 0x7fe5e006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4036000 tx=0x7fe5f4030000 comp rx=0 tx=0).stop 2024-03-06T07:29:44.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 msgr2=0x7fe6040afd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 0x7fe6040afd70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00c390 tx=0x7fe5ec00c850 comp rx=0 tx=0).stop 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 shutdown_connections 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe60415bf20 0x7fe60415cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe604072520 0x7fe6040afd70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 --2- 172.21.15.32:0/3324765838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe604071e20 0x7fe60414d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 >> 172.21.15.32:0/3324765838 conn(0x7fe60406b3d0 msgr2=0x7fe6040ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 shutdown_connections 2024-03-06T07:29:44.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.450+0000 7fe609200640 1 -- 172.21.15.32:0/3324765838 wait complete. 2024-03-06T07:29:44.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:44.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.699+0000 7f2f63200640 1 Processor -- start 2024-03-06T07:29:44.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.699+0000 7f2f63200640 1 -- start start 2024-03-06T07:29:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f63200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f64165d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f63200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2f640b4a90 0x7f2f640cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f63200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f64166cb0 con 0x7f2f640b1b40 2024-03-06T07:29:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f63200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f640cc970 con 0x7f2f640b4a90 2024-03-06T07:29:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f63200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64166410 con 0x7f2f64162280 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f62800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2f640b1b40 0x7f2f64071560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45934/0 (socket says 172.21.15.32:45934) 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f61400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2f640b4a90 0x7f2f640cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f62800640 1 -- 172.21.15.32:0/2216858449 learned_addr learned my addr 172.21.15.32:0/2216858449 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.701+0000 7f2f61e00640 1 --2- 172.21.15.32:0/2216858449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f64165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.702+0000 7f2f61400640 1 -- 172.21.15.32:0/2216858449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 msgr2=0x7f2f64165d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.702+0000 7f2f61400640 1 --2- 172.21.15.32:0/2216858449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f64165d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.702+0000 7f2f61400640 1 -- 172.21.15.32:0/2216858449 >> v1:172.21.15.32:6789/0 conn(0x7f2f640b1b40 legacy=0x7f2f64071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.702+0000 7f2f61400640 1 -- 172.21.15.32:0/2216858449 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f641666f0 con 0x7f2f640b4a90 2024-03-06T07:29:44.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.703+0000 7f2f61400640 1 --2- 172.21.15.32:0/2216858449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cc430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2f500097d0 tx=0x7f2f5002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac9579fa60f1eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.703+0000 7f2f60a00640 1 -- 172.21.15.32:0/2216858449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50031770 con 0x7f2f640b4a90 2024-03-06T07:29:44.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.703+0000 7f2f60a00640 1 -- 172.21.15.32:0/2216858449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50037070 con 0x7f2f640b4a90 2024-03-06T07:29:44.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.703+0000 7f2f60a00640 1 -- 172.21.15.32:0/2216858449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50031b30 con 0x7f2f640b4a90 2024-03-06T07:29:44.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 -- 172.21.15.32:0/2216858449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 msgr2=0x7f2f640cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 --2- 172.21.15.32:0/2216858449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cc430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2f500097d0 tx=0x7f2f5002b810 comp rx=0 tx=0).stop 2024-03-06T07:29:44.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 -- 172.21.15.32:0/2216858449 shutdown_connections 2024-03-06T07:29:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 --2- 172.21.15.32:0/2216858449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cc430 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 --2- 172.21.15.32:0/2216858449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f64165d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 -- 172.21.15.32:0/2216858449 >> 172.21.15.32:0/2216858449 conn(0x7f2f6406b3a0 msgr2=0x7f2f6406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 -- 172.21.15.32:0/2216858449 shutdown_connections 2024-03-06T07:29:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.704+0000 7f2f63200640 1 -- 172.21.15.32:0/2216858449 wait complete. 2024-03-06T07:29:44.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.706+0000 7f2f63200640 1 Processor -- start 2024-03-06T07:29:44.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.706+0000 7f2f63200640 1 -- start start 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f63200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f640b1b40 0x7f2f641f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f63200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f63200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f640d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f63200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64071c70 con 0x7f2f640b4a90 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f63200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f640b4ef0 con 0x7f2f640b1b40 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f63200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64071df0 con 0x7f2f64162280 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39806/0 (socket says 172.21.15.32:39806) 2024-03-06T07:29:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.707+0000 7f2f61400640 1 -- 172.21.15.32:0/608482639 learned_addr learned my addr 172.21.15.32:0/608482639 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f62800640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f640d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61400640 1 -- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 msgr2=0x7f2f640d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61e00640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f640b1b40 0x7f2f641f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61400640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f640d00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61400640 1 -- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f640b1b40 msgr2=0x7f2f641f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61400640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f640b1b40 0x7f2f641f8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61400640 1 -- 172.21.15.32:0/608482639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f641f8910 con 0x7f2f640b4a90 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f62800640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f640d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61e00640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f640b1b40 0x7f2f641f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.708+0000 7f2f61400640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cfb90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2f50034040 tx=0x7f2f500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.709+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50032810 con 0x7f2f640b4a90 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.709+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500329d0 con 0x7f2f640b4a90 2024-03-06T07:29:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.709+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50032dc0 con 0x7f2f640b4a90 2024-03-06T07:29:44.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.709+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f641f8bf0 con 0x7f2f640b4a90 2024-03-06T07:29:44.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.709+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f641f91b0 con 0x7f2f640b4a90 2024-03-06T07:29:44.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.710+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6404f330 con 0x7f2f640b4a90 2024-03-06T07:29:44.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.711+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f50031070 con 0x7f2f640b4a90 2024-03-06T07:29:44.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.712+0000 7f2f5a000640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f4406b4d0 0x7f2f4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.712+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f500abca0 con 0x7f2f640b4a90 2024-03-06T07:29:44.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.712+0000 7f2f61e00640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f4406b4d0 0x7f2f4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.713+0000 7f2f61e00640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f4406b4d0 0x7f2f4406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2f54006fd0 tx=0x7f2f54008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.716+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f50040070 con 0x7f2f640b4a90 2024-03-06T07:29:44.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.913+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f6406e490 con 0x7f2f640b4a90 2024-03-06T07:29:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.914+0000 7f2f5a000640 1 -- 172.21.15.32:0/608482639 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f5007aba0 con 0x7f2f640b4a90 2024-03-06T07:29:44.916 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:44.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f4406b4d0 msgr2=0x7f2f4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f4406b4d0 0x7f2f4406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2f54006fd0 tx=0x7f2f54008040 comp rx=0 tx=0).stop 2024-03-06T07:29:44.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 msgr2=0x7f2f640cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cfb90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2f50034040 tx=0x7f2f500046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 shutdown_connections 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f4406b4d0 0x7f2f4406d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2f64162280 0x7f2f640d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f640b4a90 0x7f2f640cfb90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 --2- 172.21.15.32:0/608482639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f640b1b40 0x7f2f641f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.917+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 >> 172.21.15.32:0/608482639 conn(0x7f2f6406b3a0 msgr2=0x7f2f6406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.918+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 shutdown_connections 2024-03-06T07:29:44.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:44.918+0000 7f2f63200640 1 -- 172.21.15.32:0/608482639 wait complete. 2024-03-06T07:29:44.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:45.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.153+0000 7efc0f200640 1 Processor -- start 2024-03-06T07:29:45.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.154+0000 7efc0f200640 1 -- start start 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc10154900 0x7efc100cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efc10072040 0x7efc100b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0f200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc100713f0 0x7efc10071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0f200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc10071d60 con 0x7efc100713f0 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc10154ce0 con 0x7efc10072040 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b2bf0 con 0x7efc10154900 2024-03-06T07:29:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc100713f0 0x7efc10071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc100713f0 0x7efc10071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39810/0 (socket says 172.21.15.32:39810) 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.155+0000 7efc0e800640 1 -- 172.21.15.32:0/986054943 learned_addr learned my addr 172.21.15.32:0/986054943 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc0de00640 1 --2- 172.21.15.32:0/986054943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc10154900 0x7efc100cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc07e00640 1 --2- 172.21.15.32:0/986054943 >> v2:172.21.15.39:3300/0 conn(0x7efc10072040 0x7efc100b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc0e800640 1 -- 172.21.15.32:0/986054943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc10154900 msgr2=0x7efc100cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc0e800640 1 --2- 172.21.15.32:0/986054943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc10154900 0x7efc100cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc0e800640 1 -- 172.21.15.32:0/986054943 >> v2:172.21.15.39:3300/0 conn(0x7efc10072040 msgr2=0x7efc100b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc0e800640 1 --2- 172.21.15.32:0/986054943 >> v2:172.21.15.39:3300/0 conn(0x7efc10072040 0x7efc100b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.156+0000 7efc0e800640 1 -- 172.21.15.32:0/986054943 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc100b2ed0 con 0x7efc100713f0 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.157+0000 7efc0e800640 1 --2- 172.21.15.32:0/986054943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc100713f0 0x7efc10071820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc1006eb10 tx=0x7efc0802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454b1514f37f24bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.157+0000 7efc0d400640 1 -- 172.21.15.32:0/986054943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0802c9e0 con 0x7efc100713f0 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.157+0000 7efc0d400640 1 -- 172.21.15.32:0/986054943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0802cb80 con 0x7efc100713f0 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.157+0000 7efc0d400640 1 -- 172.21.15.32:0/986054943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc080343f0 con 0x7efc100713f0 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 -- 172.21.15.32:0/986054943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc100713f0 msgr2=0x7efc10071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 --2- 172.21.15.32:0/986054943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc100713f0 0x7efc10071820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc1006eb10 tx=0x7efc0802be40 comp rx=0 tx=0).stop 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 -- 172.21.15.32:0/986054943 shutdown_connections 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 --2- 172.21.15.32:0/986054943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc100713f0 0x7efc10071820 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 --2- 172.21.15.32:0/986054943 >> v2:172.21.15.39:3300/0 conn(0x7efc10072040 0x7efc100b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 --2- 172.21.15.32:0/986054943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc10154900 0x7efc100cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 -- 172.21.15.32:0/986054943 >> 172.21.15.32:0/986054943 conn(0x7efc1006b3a0 msgr2=0x7efc1006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 -- 172.21.15.32:0/986054943 shutdown_connections 2024-03-06T07:29:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.158+0000 7efc0f200640 1 -- 172.21.15.32:0/986054943 wait complete. 2024-03-06T07:29:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.160+0000 7efc0f200640 1 Processor -- start 2024-03-06T07:29:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.160+0000 7efc0f200640 1 -- start start 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc100713f0 0x7efc101489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc10072040 0x7efc10155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 0x7efc101662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b3b70 con 0x7efc10154900 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100cf720 con 0x7efc10072040 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b3cf0 con 0x7efc100713f0 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 0x7efc101662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 0x7efc101662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39816/0 (socket says 172.21.15.32:39816) 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.161+0000 7efc0e800640 1 -- 172.21.15.32:0/851068127 learned_addr learned my addr 172.21.15.32:0/851068127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc07e00640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc10072040 0x7efc10155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0de00640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc100713f0 0x7efc101489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0e800640 1 -- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc100713f0 msgr2=0x7efc101489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0e800640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc100713f0 0x7efc101489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0e800640 1 -- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc10072040 msgr2=0x7efc10155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0e800640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc10072040 0x7efc10155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0e800640 1 -- 172.21.15.32:0/851068127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10144c20 con 0x7efc10154900 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc0e800640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 0x7efc101662e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efc08036040 tx=0x7efc08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08043020 con 0x7efc10154900 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0803e450 con 0x7efc10154900 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.162+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08009510 con 0x7efc10154900 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.163+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc10144eb0 con 0x7efc10154900 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.163+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc10145470 con 0x7efc10154900 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.164+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1004f330 con 0x7efc10154900 2024-03-06T07:29:45.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.165+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc08046020 con 0x7efc10154900 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.166+0000 7efc06000640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbec06b4d0 0x7efbec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.166+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc080aca80 con 0x7efc10154900 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.166+0000 7efc0de00640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbec06b4d0 0x7efbec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.167+0000 7efc0de00640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbec06b4d0 0x7efbec06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc00006fd0 tx=0x7efc00008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.170+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc080aee20 con 0x7efc10154900 2024-03-06T07:29:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.370+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc10070cc0 con 0x7efc10154900 2024-03-06T07:29:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.371+0000 7efc06000640 1 -- 172.21.15.32:0/851068127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7efc0804d030 con 0x7efc10154900 2024-03-06T07:29:45.372 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.373+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbec06b4d0 msgr2=0x7efbec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.373+0000 7efc0f200640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbec06b4d0 0x7efbec06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc00006fd0 tx=0x7efc00008040 comp rx=0 tx=0).stop 2024-03-06T07:29:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 msgr2=0x7efc101662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 0x7efc101662e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efc08036040 tx=0x7efc08004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 shutdown_connections 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbec06b4d0 0x7efbec06d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc10154900 0x7efc101662e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc10072040 0x7efc10155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 --2- 172.21.15.32:0/851068127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc100713f0 0x7efc101489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 >> 172.21.15.32:0/851068127 conn(0x7efc1006b3a0 msgr2=0x7efc1006b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 shutdown_connections 2024-03-06T07:29:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.374+0000 7efc0f200640 1 -- 172.21.15.32:0/851068127 wait complete. 2024-03-06T07:29:45.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:45.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.600+0000 7fa181600640 1 Processor -- start 2024-03-06T07:29:45.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.600+0000 7fa181600640 1 -- start start 2024-03-06T07:29:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.602+0000 7fa181600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa17c072da0 0x7fa17c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.602+0000 7fa181600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c0712c0 0x7fa17c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.602+0000 7fa181600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0ce090 con 0x7fa17c070ee0 2024-03-06T07:29:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.602+0000 7fa181600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17c0ceca0 con 0x7fa17c072da0 2024-03-06T07:29:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.602+0000 7fa181600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0ce370 con 0x7fa17c0712c0 2024-03-06T07:29:45.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.603+0000 7fa173e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa17c072da0 0x7fa17c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.603+0000 7fa173e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa17c072da0 0x7fa17c162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:38062/0 (socket says 172.21.15.32:38062) 2024-03-06T07:29:45.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.603+0000 7fa173e00640 1 -- 172.21.15.32:0/2168564840 learned_addr learned my addr 172.21.15.32:0/2168564840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:45.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.603+0000 7fa17b400640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c0712c0 0x7fa17c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.604+0000 7fa173e00640 1 -- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c0712c0 msgr2=0x7fa17c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.604+0000 7fa173e00640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c0712c0 0x7fa17c06e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.604+0000 7fa173e00640 1 -- 172.21.15.32:0/2168564840 >> v1:172.21.15.39:6789/0 conn(0x7fa17c070ee0 legacy=0x7fa17c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.604+0000 7fa173e00640 1 -- 172.21.15.32:0/2168564840 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0ce650 con 0x7fa17c072da0 2024-03-06T07:29:45.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.604+0000 7fa17b400640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c0712c0 0x7fa17c06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:45.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.605+0000 7fa173e00640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c072da0 0x7fa17c162c40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0cee20 tx=0x7fa16402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d71064260dffd213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa17a000640 1 -- 172.21.15.32:0/2168564840 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164034070 con 0x7fa17c072da0 2024-03-06T07:29:45.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa17a000640 1 -- 172.21.15.32:0/2168564840 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa164030840 con 0x7fa17c072da0 2024-03-06T07:29:45.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa17a000640 1 -- 172.21.15.32:0/2168564840 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164030c10 con 0x7fa17c072da0 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa181600640 1 -- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c072da0 msgr2=0x7fa17c162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa181600640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c072da0 0x7fa17c162c40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0cee20 tx=0x7fa16402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa181600640 1 -- 172.21.15.32:0/2168564840 shutdown_connections 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa181600640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c0712c0 0x7fa17c06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa181600640 1 --2- 172.21.15.32:0/2168564840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c072da0 0x7fa17c162c40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.606+0000 7fa181600640 1 -- 172.21.15.32:0/2168564840 >> 172.21.15.32:0/2168564840 conn(0x7fa17c06afb0 msgr2=0x7fa17c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.607+0000 7fa181600640 1 -- 172.21.15.32:0/2168564840 shutdown_connections 2024-03-06T07:29:45.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.607+0000 7fa181600640 1 -- 172.21.15.32:0/2168564840 wait complete. 2024-03-06T07:29:45.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.609+0000 7fa181600640 1 Processor -- start 2024-03-06T07:29:45.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.609+0000 7fa181600640 1 -- start start 2024-03-06T07:29:45.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa181600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 0x7fa17c14f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa181600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c0712c0 0x7fa17c152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa181600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c072da0 0x7fa17c09d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa181600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0cdda0 con 0x7fa17c070ee0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa181600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0cdc20 con 0x7fa17c0712c0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa181600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0cf130 con 0x7fa17c072da0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.610+0000 7fa17aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 0x7fa17c14f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 0x7fa17c14f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39818/0 (socket says 172.21.15.32:39818) 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 -- 172.21.15.32:0/633380658 learned_addr learned my addr 172.21.15.32:0/633380658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa173e00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c0712c0 0x7fa17c152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17b400640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c072da0 0x7fa17c09d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 -- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c072da0 msgr2=0x7fa17c09d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c072da0 0x7fa17c09d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 -- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c0712c0 msgr2=0x7fa17c152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c0712c0 0x7fa17c152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 -- 172.21.15.32:0/633380658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c14fad0 con 0x7fa17c070ee0 2024-03-06T07:29:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa173e00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c0712c0 0x7fa17c152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:45.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17b400640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c072da0 0x7fa17c09d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:45.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.611+0000 7fa17aa00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 0x7fa17c14f590 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00c400 tx=0x7fa16c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.612+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0052d0 con 0x7fa17c070ee0 2024-03-06T07:29:45.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.612+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c005490 con 0x7fa17c070ee0 2024-03-06T07:29:45.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.612+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c007260 con 0x7fa17c070ee0 2024-03-06T07:29:45.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.612+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c152d40 con 0x7fa17c070ee0 2024-03-06T07:29:45.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.612+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c0ad8f0 con 0x7fa17c070ee0 2024-03-06T07:29:45.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.613+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c04f3a0 con 0x7fa17c070ee0 2024-03-06T07:29:45.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.614+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa16c007400 con 0x7fa17c070ee0 2024-03-06T07:29:45.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.615+0000 7fa172a00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15806b4d0 0x7fa15806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.615+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa16c0897f0 con 0x7fa17c070ee0 2024-03-06T07:29:45.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.615+0000 7fa173e00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15806b4d0 0x7fa15806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.616+0000 7fa173e00640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15806b4d0 0x7fa15806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa164036000 tx=0x7fa164008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.619+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa16c0585d0 con 0x7fa17c070ee0 2024-03-06T07:29:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.832+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa17c0b52d0 con 0x7fa17c070ee0 2024-03-06T07:29:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.832+0000 7fa172a00640 1 -- 172.21.15.32:0/633380658 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa16c057ed0 con 0x7fa17c070ee0 2024-03-06T07:29:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15806b4d0 msgr2=0x7fa15806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15806b4d0 0x7fa15806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa164036000 tx=0x7fa164008030 comp rx=0 tx=0).stop 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 msgr2=0x7fa17c14f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 0x7fa17c14f590 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00c400 tx=0x7fa16c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 shutdown_connections 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15806b4d0 0x7fa15806d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa17c072da0 0x7fa17c09d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa17c0712c0 0x7fa17c152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 --2- 172.21.15.32:0/633380658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa17c070ee0 0x7fa17c14f590 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 >> 172.21.15.32:0/633380658 conn(0x7fa17c06afb0 msgr2=0x7fa17c0afa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.835+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 shutdown_connections 2024-03-06T07:29:45.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:45.836+0000 7fa181600640 1 -- 172.21.15.32:0/633380658 wait complete. 2024-03-06T07:29:45.854 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:29:45.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-06T07:29:45.958 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:29:45.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:45.958 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:46.024 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:46.024 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:46.083 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:46.083 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:46.134 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:46.134 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:46.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:46.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.415+0000 7ff7a5600640 1 Processor -- start 2024-03-06T07:29:46.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.415+0000 7ff7a5600640 1 -- start start 2024-03-06T07:29:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.417+0000 7ff7a5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7a00af1a0 0x7ff7a015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.417+0000 7ff7a5600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00cdc20 0x7ff7a00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.417+0000 7ff7a5600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a015cc40 con 0x7ff7a00af580 2024-03-06T07:29:46.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.417+0000 7ff7a5600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a015cf20 con 0x7ff7a00af1a0 2024-03-06T07:29:46.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.417+0000 7ff7a5600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a015d200 con 0x7ff7a00cdc20 2024-03-06T07:29:46.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7a00af1a0 0x7ff7a015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7a00af1a0 0x7ff7a015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:38088/0 (socket says 172.21.15.32:38088) 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 -- 172.21.15.32:0/2093260909 learned_addr learned my addr 172.21.15.32:0/2093260909 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79f400640 1 --2- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00cdc20 0x7ff7a00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 -- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00cdc20 msgr2=0x7ff7a00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 --2- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00cdc20 0x7ff7a00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 -- 172.21.15.32:0/2093260909 >> v1:172.21.15.39:6789/0 conn(0x7ff7a00af580 legacy=0x7ff7a00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.418+0000 7ff79ea00640 1 -- 172.21.15.32:0/2093260909 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a015d4e0 con 0x7ff7a00af1a0 2024-03-06T07:29:46.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.420+0000 7ff79ea00640 1 --2- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00af1a0 0x7ff7a015c600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff790009870 tx=0x7ff79002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42f9fbdac824e7f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.420+0000 7ff79e000640 1 -- 172.21.15.32:0/2093260909 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790035020 con 0x7ff7a00af1a0 2024-03-06T07:29:46.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.420+0000 7ff79e000640 1 -- 172.21.15.32:0/2093260909 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790031640 con 0x7ff7a00af1a0 2024-03-06T07:29:46.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.420+0000 7ff79e000640 1 -- 172.21.15.32:0/2093260909 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79002d040 con 0x7ff7a00af1a0 2024-03-06T07:29:46.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 -- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00af1a0 msgr2=0x7ff7a015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00af1a0 0x7ff7a015c600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff790009870 tx=0x7ff79002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 -- 172.21.15.32:0/2093260909 shutdown_connections 2024-03-06T07:29:46.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00cdc20 0x7ff7a00b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2093260909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00af1a0 0x7ff7a015c600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 -- 172.21.15.32:0/2093260909 >> 172.21.15.32:0/2093260909 conn(0x7ff7a006afb0 msgr2=0x7ff7a006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 -- 172.21.15.32:0/2093260909 shutdown_connections 2024-03-06T07:29:46.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.421+0000 7ff7a5600640 1 -- 172.21.15.32:0/2093260909 wait complete. 2024-03-06T07:29:46.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.423+0000 7ff7a5600640 1 Processor -- start 2024-03-06T07:29:46.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.423+0000 7ff7a5600640 1 -- start start 2024-03-06T07:29:46.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff7a5600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00af1a0 0x7ff7a0148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff7a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 0x7ff7a009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff7a5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00cdc20 0x7ff7a00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff7a5600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b3720 con 0x7ff7a00af580 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff7a5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b35a0 con 0x7ff7a00cdc20 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff7a5600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a0155e70 con 0x7ff7a00af1a0 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff797e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 0x7ff7a009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff797e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 0x7ff7a009bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39830/0 (socket says 172.21.15.32:39830) 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff797e00640 1 -- 172.21.15.32:0/2682493327 learned_addr learned my addr 172.21.15.32:0/2682493327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff79f400640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00cdc20 0x7ff7a00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.424+0000 7ff79ea00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00af1a0 0x7ff7a0148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff797e00640 1 -- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00af1a0 msgr2=0x7ff7a0148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff797e00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00af1a0 0x7ff7a0148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff797e00640 1 -- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00cdc20 msgr2=0x7ff7a00acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff797e00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00cdc20 0x7ff7a00acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff797e00640 1 -- 172.21.15.32:0/2682493327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a009c860 con 0x7ff7a00af580 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff797e00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 0x7ff7a009bba0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c440 tx=0x7ff788001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7880052d0 con 0x7ff7a00af580 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff788005470 con 0x7ff7a00af580 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788007260 con 0x7ff7a00af580 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a009cb40 con 0x7ff7a00af580 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.425+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a009d100 con 0x7ff7a00af580 2024-03-06T07:29:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.426+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a0069510 con 0x7ff7a00af580 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.427+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff788007400 con 0x7ff7a00af580 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.428+0000 7ff796a00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.428+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7880897e0 con 0x7ff7a00af580 2024-03-06T07:29:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.429+0000 7ff79ea00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.430+0000 7ff79ea00640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.434+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7880585c0 con 0x7ff7a00af580 2024-03-06T07:29:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.633+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7a004f3a0 con 0x7ff7a00af580 2024-03-06T07:29:46.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.636+0000 7ff796a00640 1 -- 172.21.15.32:0/2682493327 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16497 (secure 0 0 0) 0x7ff788057ec0 con 0x7ff7a00af580 2024-03-06T07:29:46.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:46.638 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:02.275114+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.638+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff77c06b4d0 msgr2=0x7ff77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.638+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 msgr2=0x7ff7a009bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 0x7ff7a009bba0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c440 tx=0x7ff788001e70 comp rx=0 tx=0).stop 2024-03-06T07:29:46.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 shutdown_connections 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7a00cdc20 0x7ff7a00acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7a00af580 0x7ff7a009bba0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 --2- 172.21.15.32:0/2682493327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7a00af1a0 0x7ff7a0148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 >> 172.21.15.32:0/2682493327 conn(0x7ff7a006afb0 msgr2=0x7ff7a0155620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 shutdown_connections 2024-03-06T07:29:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.639+0000 7ff7a5600640 1 -- 172.21.15.32:0/2682493327 wait complete. 2024-03-06T07:29:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:46.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:46.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.871+0000 7fea63000640 1 Processor -- start 2024-03-06T07:29:46.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.872+0000 7fea63000640 1 -- start start 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.873+0000 7fea63000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea5c147590 0x7fea5c071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.873+0000 7fea63000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c0b2250 0x7fea5c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.873+0000 7fea63000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c0719c0 con 0x7fea5c06d8d0 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.873+0000 7fea63000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c147970 con 0x7fea5c147590 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.873+0000 7fea63000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c0b31b0 con 0x7fea5c0b2250 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea5c147590 0x7fea5c071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fea5c06d8d0 0x7fea5c0b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45948/0 (socket says 172.21.15.32:45948) 2024-03-06T07:29:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61200640 1 -- 172.21.15.32:0/1053667089 learned_addr learned my addr 172.21.15.32:0/1053667089 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea62600640 1 --2- 172.21.15.32:0/1053667089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c0b2250 0x7fea5c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61c00640 1 -- 172.21.15.32:0/1053667089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c0b2250 msgr2=0x7fea5c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61c00640 1 --2- 172.21.15.32:0/1053667089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c0b2250 0x7fea5c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea5be00640 1 -- 172.21.15.32:0/1053667089 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840492780 0 0) 0x7fea5c0719c0 con 0x7fea5c06d8d0 2024-03-06T07:29:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61c00640 1 -- 172.21.15.32:0/1053667089 >> v1:172.21.15.32:6789/0 conn(0x7fea5c06d8d0 legacy=0x7fea5c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.874+0000 7fea61c00640 1 -- 172.21.15.32:0/1053667089 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c06de40 con 0x7fea5c147590 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.876+0000 7fea61c00640 1 --2- 172.21.15.32:0/1053667089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c147590 0x7fea5c071480 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0097a0 tx=0x7fea4c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd9021122ca4f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.876+0000 7fea5be00640 1 -- 172.21.15.32:0/1053667089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c035020 con 0x7fea5c147590 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.876+0000 7fea5be00640 1 -- 172.21.15.32:0/1053667089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c031810 con 0x7fea5c147590 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.876+0000 7fea5be00640 1 -- 172.21.15.32:0/1053667089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c031be0 con 0x7fea5c147590 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 -- 172.21.15.32:0/1053667089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c147590 msgr2=0x7fea5c071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 --2- 172.21.15.32:0/1053667089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c147590 0x7fea5c071480 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0097a0 tx=0x7fea4c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 -- 172.21.15.32:0/1053667089 shutdown_connections 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 --2- 172.21.15.32:0/1053667089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c0b2250 0x7fea5c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 --2- 172.21.15.32:0/1053667089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c147590 0x7fea5c071480 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 -- 172.21.15.32:0/1053667089 >> 172.21.15.32:0/1053667089 conn(0x7fea5c06b3a0 msgr2=0x7fea5c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 -- 172.21.15.32:0/1053667089 shutdown_connections 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.877+0000 7fea63000640 1 -- 172.21.15.32:0/1053667089 wait complete. 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.878+0000 7fea63000640 1 Processor -- start 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.879+0000 7fea63000640 1 -- start start 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.879+0000 7fea63000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 0x7fea5c15d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.879+0000 7fea63000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea5c0b2250 0x7fea5c140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea63000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c147590 0x7fea5c140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea63000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c0b2b50 con 0x7fea5c06d8d0 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea63000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c0b2e40 con 0x7fea5c0b2250 2024-03-06T07:29:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea63000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c0b3b80 con 0x7fea5c147590 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 0x7fea5c15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 0x7fea5c15d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39852/0 (socket says 172.21.15.32:39852) 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 -- 172.21.15.32:0/3777288682 learned_addr learned my addr 172.21.15.32:0/3777288682 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea62600640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c147590 0x7fea5c140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61200640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea5c0b2250 0x7fea5c140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 -- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c147590 msgr2=0x7fea5c140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c147590 0x7fea5c140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 -- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea5c0b2250 msgr2=0x7fea5c140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea5c0b2250 0x7fea5c140920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.880+0000 7fea61c00640 1 -- 172.21.15.32:0/3777288682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c143930 con 0x7fea5c06d8d0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.881+0000 7fea61c00640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 0x7fea5c15d750 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fea4c000c00 tx=0x7fea4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.881+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c041070 con 0x7fea5c06d8d0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.881+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c03ac80 con 0x7fea5c06d8d0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.881+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c035040 con 0x7fea5c06d8d0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.881+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c143c10 con 0x7fea5c06d8d0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.881+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c1441d0 con 0x7fea5c06d8d0 2024-03-06T07:29:46.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.883+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c04f330 con 0x7fea5c06d8d0 2024-03-06T07:29:46.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.883+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea4c038020 con 0x7fea5c06d8d0 2024-03-06T07:29:46.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.884+0000 7fea59600640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea3c06b4d0 0x7fea3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.884+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fea4c0abfd0 con 0x7fea5c06d8d0 2024-03-06T07:29:46.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.885+0000 7fea61200640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea3c06b4d0 0x7fea3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.885+0000 7fea61200640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea3c06b4d0 0x7fea3c06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea5c13e260 tx=0x7fea48009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:46.889+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea4c07add0 con 0x7fea5c06d8d0 2024-03-06T07:29:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.096+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea5c071e90 con 0x7fea5c06d8d0 2024-03-06T07:29:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.097+0000 7fea59600640 1 -- 172.21.15.32:0/3777288682 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea4c04de80 con 0x7fea5c06d8d0 2024-03-06T07:29:47.098 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.099+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea3c06b4d0 msgr2=0x7fea3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.099+0000 7fea63000640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea3c06b4d0 0x7fea3c06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea5c13e260 tx=0x7fea48009040 comp rx=0 tx=0).stop 2024-03-06T07:29:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.099+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 msgr2=0x7fea5c15d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.099+0000 7fea63000640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 0x7fea5c15d750 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fea4c000c00 tx=0x7fea4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.099+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 shutdown_connections 2024-03-06T07:29:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.099+0000 7fea63000640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea3c06b4d0 0x7fea3c06d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.100+0000 7fea63000640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea5c147590 0x7fea5c140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.100+0000 7fea63000640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea5c0b2250 0x7fea5c140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.100+0000 7fea63000640 1 --2- 172.21.15.32:0/3777288682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c06d8d0 0x7fea5c15d750 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.100+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 >> 172.21.15.32:0/3777288682 conn(0x7fea5c06b3a0 msgr2=0x7fea5c06b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.100+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 shutdown_connections 2024-03-06T07:29:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.100+0000 7fea63000640 1 -- 172.21.15.32:0/3777288682 wait complete. 2024-03-06T07:29:47.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:47.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.347+0000 7f8885a00640 1 Processor -- start 2024-03-06T07:29:47.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.347+0000 7f8885a00640 1 -- start start 2024-03-06T07:29:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.349+0000 7f8885a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 0x7f888006e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.349+0000 7f8885a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f88800ccad0 0x7f88800ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.349+0000 7f8885a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888006d090 con 0x7f888015e680 2024-03-06T07:29:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.349+0000 7f8885a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8880161c70 con 0x7f88800ccad0 2024-03-06T07:29:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.349+0000 7f8885a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888015eb80 con 0x7f888006da40 2024-03-06T07:29:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.349+0000 7f887ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 0x7f888006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f88800ccad0 0x7f88800ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 0x7f888006e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:39102/0 (socket says 172.21.15.32:39102) 2024-03-06T07:29:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887ea00640 1 -- 172.21.15.32:0/1409080774 learned_addr learned my addr 172.21.15.32:0/1409080774 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887ea00640 1 -- 172.21.15.32:0/1409080774 >> v2:172.21.15.39:3300/0 conn(0x7f88800ccad0 msgr2=0x7f88800ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887ea00640 1 --2- 172.21.15.32:0/1409080774 >> v2:172.21.15.39:3300/0 conn(0x7f88800ccad0 0x7f88800ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887ea00640 1 -- 172.21.15.32:0/1409080774 >> v1:172.21.15.39:6789/0 conn(0x7f888015e680 legacy=0x7f888006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.350+0000 7f887ea00640 1 -- 172.21.15.32:0/1409080774 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888015ee60 con 0x7f888006da40 2024-03-06T07:29:47.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.352+0000 7f887ea00640 1 --2- 172.21.15.32:0/1409080774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 0x7f888006e340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8870009870 tx=0x7f887002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c896d1fda910ca05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.352+0000 7f887e000640 1 -- 172.21.15.32:0/1409080774 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870034070 con 0x7f888006da40 2024-03-06T07:29:47.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f887e000640 1 -- 172.21.15.32:0/1409080774 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8870030840 con 0x7f888006da40 2024-03-06T07:29:47.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f887e000640 1 -- 172.21.15.32:0/1409080774 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870030c10 con 0x7f888006da40 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f8885a00640 1 -- 172.21.15.32:0/1409080774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 msgr2=0x7f888006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f8885a00640 1 --2- 172.21.15.32:0/1409080774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 0x7f888006e340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8870009870 tx=0x7f887002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f8885a00640 1 -- 172.21.15.32:0/1409080774 shutdown_connections 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f8885a00640 1 --2- 172.21.15.32:0/1409080774 >> v2:172.21.15.39:3300/0 conn(0x7f88800ccad0 0x7f88800ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f8885a00640 1 --2- 172.21.15.32:0/1409080774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f888006da40 0x7f888006e340 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.353+0000 7f8885a00640 1 -- 172.21.15.32:0/1409080774 >> 172.21.15.32:0/1409080774 conn(0x7f888006b3d0 msgr2=0x7f888006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.354+0000 7f8885a00640 1 -- 172.21.15.32:0/1409080774 shutdown_connections 2024-03-06T07:29:47.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.354+0000 7f8885a00640 1 -- 172.21.15.32:0/1409080774 wait complete. 2024-03-06T07:29:47.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.355+0000 7f8885a00640 1 Processor -- start 2024-03-06T07:29:47.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.355+0000 7f8885a00640 1 -- start start 2024-03-06T07:29:47.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.356+0000 7f8885a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888006da40 0x7f88800c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.356+0000 7f8885a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88800ccad0 0x7f888009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.356+0000 7f8885a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 0x7f8880155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.356+0000 7f8885a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888006e880 con 0x7f888015e680 2024-03-06T07:29:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.356+0000 7f8885a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888015f420 con 0x7f888006da40 2024-03-06T07:29:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.356+0000 7f8885a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888006ea00 con 0x7f88800ccad0 2024-03-06T07:29:47.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 0x7f8880155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 0x7f8880155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39854/0 (socket says 172.21.15.32:39854) 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 -- 172.21.15.32:0/1033995082 learned_addr learned my addr 172.21.15.32:0/1033995082 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887ea00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888006da40 0x7f88800c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f8877e00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88800ccad0 0x7f888009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 -- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88800ccad0 msgr2=0x7f888009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88800ccad0 0x7f888009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 -- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888006da40 msgr2=0x7f88800c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888006da40 0x7f88800c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.357+0000 7f887f400640 1 -- 172.21.15.32:0/1033995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88800b2c80 con 0x7f888015e680 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.358+0000 7f887f400640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 0x7f8880155600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f887800efc0 tx=0x7f887800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.358+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878018020 con 0x7f888015e680 2024-03-06T07:29:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.358+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8878014480 con 0x7f888015e680 2024-03-06T07:29:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.358+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878014870 con 0x7f888015e680 2024-03-06T07:29:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.358+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88800b2f60 con 0x7f888015e680 2024-03-06T07:29:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.358+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88800b3520 con 0x7f888015e680 2024-03-06T07:29:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.359+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888004f330 con 0x7f888015e680 2024-03-06T07:29:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.360+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8878005d10 con 0x7f888015e680 2024-03-06T07:29:47.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.361+0000 7f8876a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.361+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f887808adf0 con 0x7f888015e680 2024-03-06T07:29:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.361+0000 7f887ea00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.362+0000 7f887ea00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f885c06b4d0 0x7f885c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8870036000 tx=0x7f887003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.368+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8878022070 con 0x7f888015e680 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.573+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88800af000 con 0x7f888015e680 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.573+0000 7f8876a00640 1 -- 172.21.15.32:0/1033995082 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8878010040 con 0x7f888015e680 2024-03-06T07:29:47.575 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:47.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f885c06b4d0 msgr2=0x7f885c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f885c06b4d0 0x7f885c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8870036000 tx=0x7f887003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 msgr2=0x7f8880155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 0x7f8880155600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f887800efc0 tx=0x7f887800c490 comp rx=0 tx=0).stop 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 shutdown_connections 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f888015e680 0x7f8880155600 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88800ccad0 0x7f888009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 --2- 172.21.15.32:0/1033995082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f888006da40 0x7f88800c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 >> 172.21.15.32:0/1033995082 conn(0x7f888006b3d0 msgr2=0x7f888015cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.576+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 shutdown_connections 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.577+0000 7f8885a00640 1 -- 172.21.15.32:0/1033995082 wait complete. 2024-03-06T07:29:47.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.829+0000 7f702b200640 1 Processor -- start 2024-03-06T07:29:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.829+0000 7f702b200640 1 -- start start 2024-03-06T07:29:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.830+0000 7f702b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c15dee0 0x7f702c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.830+0000 7f702b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f702c0d4860 0x7f702c0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.830+0000 7f702b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702c0ceca0 con 0x7f702c0cfa60 2024-03-06T07:29:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.831+0000 7f702b200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702c0cf5d0 con 0x7f702c0d4860 2024-03-06T07:29:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.831+0000 7f702b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0cef80 con 0x7f702c15dee0 2024-03-06T07:29:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.831+0000 7f7029400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f702c0d4860 0x7f702c0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.831+0000 7f7029400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f702c0d4860 0x7f702c0d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39868/0 (socket says 172.21.15.32:39868) 2024-03-06T07:29:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.831+0000 7f7029400640 1 -- 172.21.15.32:0/4199377187 learned_addr learned my addr 172.21.15.32:0/4199377187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.831+0000 7f7029e00640 1 --2- 172.21.15.32:0/4199377187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c15dee0 0x7f702c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.832+0000 7f7029400640 1 -- 172.21.15.32:0/4199377187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c15dee0 msgr2=0x7f702c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.832+0000 7f7029400640 1 --2- 172.21.15.32:0/4199377187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c15dee0 0x7f702c0d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.832+0000 7f7029400640 1 -- 172.21.15.32:0/4199377187 >> v1:172.21.15.39:6789/0 conn(0x7f702c0cfa60 legacy=0x7f702c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.832+0000 7f7029400640 1 -- 172.21.15.32:0/4199377187 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c0cf260 con 0x7f702c0d4860 2024-03-06T07:29:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.833+0000 7f7029400640 1 --2- 172.21.15.32:0/4199377187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c0d0630 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7014009e50 tx=0x7f701402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2fecb6b7fd856b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.833+0000 7f7028a00640 1 -- 172.21.15.32:0/4199377187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701402cb00 con 0x7f702c0d4860 2024-03-06T07:29:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.833+0000 7f7028a00640 1 -- 172.21.15.32:0/4199377187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701402cca0 con 0x7f702c0d4860 2024-03-06T07:29:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.833+0000 7f7028a00640 1 -- 172.21.15.32:0/4199377187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014034610 con 0x7f702c0d4860 2024-03-06T07:29:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 -- 172.21.15.32:0/4199377187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 msgr2=0x7f702c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 --2- 172.21.15.32:0/4199377187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c0d0630 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7014009e50 tx=0x7f701402bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 -- 172.21.15.32:0/4199377187 shutdown_connections 2024-03-06T07:29:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 --2- 172.21.15.32:0/4199377187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c0d0630 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 --2- 172.21.15.32:0/4199377187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c15dee0 0x7f702c0d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 -- 172.21.15.32:0/4199377187 >> 172.21.15.32:0/4199377187 conn(0x7f702c0d3cd0 msgr2=0x7f702c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 -- 172.21.15.32:0/4199377187 shutdown_connections 2024-03-06T07:29:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.834+0000 7f702b200640 1 -- 172.21.15.32:0/4199377187 wait complete. 2024-03-06T07:29:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.836+0000 7f702b200640 1 Processor -- start 2024-03-06T07:29:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.836+0000 7f702b200640 1 -- start start 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f702b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c0cfa60 0x7f702c0cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f702b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c14f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f702b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702c15dee0 0x7f702c14f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f702b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0d1de0 con 0x7f702c0d4860 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f702b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c15e2c0 con 0x7f702c15dee0 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f702b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0d1f60 con 0x7f702c0cfa60 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f7029400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c14f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f7029400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c14f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39872/0 (socket says 172.21.15.32:39872) 2024-03-06T07:29:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f7029400640 1 -- 172.21.15.32:0/717997056 learned_addr learned my addr 172.21.15.32:0/717997056 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.837+0000 7f7029e00640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c0cfa60 0x7f702c0cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f702a800640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702c15dee0 0x7f702c14f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7029400640 1 -- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c0cfa60 msgr2=0x7f702c0cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7029400640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c0cfa60 0x7f702c0cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7029400640 1 -- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702c15dee0 msgr2=0x7f702c14f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7029400640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702c15dee0 0x7f702c14f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7029400640 1 -- 172.21.15.32:0/717997056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c14d250 con 0x7f702c0d4860 2024-03-06T07:29:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7029400640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c14f190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7014036040 tx=0x7f7014004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014043020 con 0x7f702c0d4860 2024-03-06T07:29:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701402ce50 con 0x7f702c0d4860 2024-03-06T07:29:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c14d530 con 0x7f702c0d4860 2024-03-06T07:29:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.838+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014009510 con 0x7f702c0d4860 2024-03-06T07:29:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.839+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c14daf0 con 0x7f702c0d4860 2024-03-06T07:29:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.840+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702c04f330 con 0x7f702c0d4860 2024-03-06T07:29:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.841+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7014046020 con 0x7f702c0d4860 2024-03-06T07:29:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.842+0000 7f7022000640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f700806b4d0 0x7f700806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.842+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f70140ac8c0 con 0x7f702c0d4860 2024-03-06T07:29:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.842+0000 7f7029e00640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f700806b4d0 0x7f700806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.843+0000 7f7029e00640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f700806b4d0 0x7f700806d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f701c009a30 tx=0x7f701c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:47.846+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f701404d030 con 0x7f702c0d4860 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.046+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f702c14dcc0 con 0x7f702c0d4860 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.047+0000 7f7022000640 1 -- 172.21.15.32:0/717997056 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f701407b630 con 0x7f702c0d4860 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f700806b4d0 msgr2=0x7f700806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f700806b4d0 0x7f700806d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f701c009a30 tx=0x7f701c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 msgr2=0x7f702c14f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c14f190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7014036040 tx=0x7f7014004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 shutdown_connections 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f700806b4d0 0x7f700806d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702c15dee0 0x7f702c14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702c0d4860 0x7f702c14f190 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 --2- 172.21.15.32:0/717997056 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f702c0cfa60 0x7f702c0cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.049+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 >> 172.21.15.32:0/717997056 conn(0x7f702c0d3cd0 msgr2=0x7f702c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.050+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 shutdown_connections 2024-03-06T07:29:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.050+0000 7f702b200640 1 -- 172.21.15.32:0/717997056 wait complete. 2024-03-06T07:29:48.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.275+0000 7fca56600640 1 Processor -- start 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.275+0000 7fca56600640 1 -- start start 2024-03-06T07:29:48.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca56600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca56600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca500cdc20 0x7fca500b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca56600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5015cc40 con 0x7fca500af1a0 2024-03-06T07:29:48.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca56600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5015cf20 con 0x7fca500cdc20 2024-03-06T07:29:48.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca56600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5015d200 con 0x7fca500af580 2024-03-06T07:29:48.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca4fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca500cdc20 0x7fca500b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca4f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fca500af1a0 0x7fca500b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34136/0 (socket says 172.21.15.32:34136) 2024-03-06T07:29:48.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.277+0000 7fca4f400640 1 -- 172.21.15.32:0/2693145637 learned_addr learned my addr 172.21.15.32:0/2693145637 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:48.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.278+0000 7fca46a00640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.278+0000 7fca4fe00640 1 -- 172.21.15.32:0/2693145637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 msgr2=0x7fca5015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.278+0000 7fca4fe00640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.278+0000 7fca4fe00640 1 -- 172.21.15.32:0/2693145637 >> v1:172.21.15.32:6789/0 conn(0x7fca500af1a0 legacy=0x7fca500b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.278+0000 7fca4fe00640 1 -- 172.21.15.32:0/2693145637 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5015d4e0 con 0x7fca500cdc20 2024-03-06T07:29:48.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.278+0000 7fca46a00640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.279+0000 7fca4fe00640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500cdc20 0x7fca500b2dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fca48009d90 tx=0x7fca4802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca89b513837ab261 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.279+0000 7fca4ea00640 1 -- 172.21.15.32:0/2693145637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48039070 con 0x7fca500cdc20 2024-03-06T07:29:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.279+0000 7fca4ea00640 1 -- 172.21.15.32:0/2693145637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4802cbd0 con 0x7fca500cdc20 2024-03-06T07:29:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.279+0000 7fca4ea00640 1 -- 172.21.15.32:0/2693145637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48034440 con 0x7fca500cdc20 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 -- 172.21.15.32:0/2693145637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500cdc20 msgr2=0x7fca500b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500cdc20 0x7fca500b2dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fca48009d90 tx=0x7fca4802be20 comp rx=0 tx=0).stop 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 -- 172.21.15.32:0/2693145637 shutdown_connections 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500cdc20 0x7fca500b2dd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 --2- 172.21.15.32:0/2693145637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 -- 172.21.15.32:0/2693145637 >> 172.21.15.32:0/2693145637 conn(0x7fca5006afb0 msgr2=0x7fca5006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 -- 172.21.15.32:0/2693145637 shutdown_connections 2024-03-06T07:29:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.280+0000 7fca56600640 1 -- 172.21.15.32:0/2693145637 wait complete. 2024-03-06T07:29:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.282+0000 7fca56600640 1 Processor -- start 2024-03-06T07:29:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.282+0000 7fca56600640 1 -- start start 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca56600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 0x7fca500cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca56600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5014f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca56600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca500cdc20 0x7fca500c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca56600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500b36a0 con 0x7fca500af1a0 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca56600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500b3520 con 0x7fca500cdc20 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca56600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500b3820 con 0x7fca500af580 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca4f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 0x7fca500cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca4f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 0x7fca500cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35724/0 (socket says 172.21.15.32:35724) 2024-03-06T07:29:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.283+0000 7fca4f400640 1 -- 172.21.15.32:0/922243653 learned_addr learned my addr 172.21.15.32:0/922243653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4fe00640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca500cdc20 0x7fca500c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4f400640 1 -- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 msgr2=0x7fca5014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4f400640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4f400640 1 -- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca500cdc20 msgr2=0x7fca500c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4f400640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca500cdc20 0x7fca500c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4f400640 1 -- 172.21.15.32:0/922243653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5009d500 con 0x7fca500af1a0 2024-03-06T07:29:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca4f400640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 0x7fca500cb4e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c400 tx=0x7fca4000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40017070 con 0x7fca500af1a0 2024-03-06T07:29:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca40005ce0 con 0x7fca500af1a0 2024-03-06T07:29:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca400072f0 con 0x7fca500af1a0 2024-03-06T07:29:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca500ca440 con 0x7fca500af1a0 2024-03-06T07:29:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.284+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca500cace0 con 0x7fca500af1a0 2024-03-06T07:29:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.285+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50069510 con 0x7fca500af1a0 2024-03-06T07:29:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.288+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca4001a070 con 0x7fca500af1a0 2024-03-06T07:29:48.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.288+0000 7fca47e00640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca2c06b4d0 0x7fca2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.289+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fca40089690 con 0x7fca500af1a0 2024-03-06T07:29:48.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.289+0000 7fca46a00640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca2c06b4d0 0x7fca2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.290+0000 7fca46a00640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca2c06b4d0 0x7fca2c06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fca38006fd0 tx=0x7fca38008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.293+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca40058490 con 0x7fca500af1a0 2024-03-06T07:29:48.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:48.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:48.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.508+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fca500ca800 con 0x7fca500af1a0 2024-03-06T07:29:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.509+0000 7fca47e00640 1 -- 172.21.15.32:0/922243653 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fca40057d90 con 0x7fca500af1a0 2024-03-06T07:29:48.510 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:29:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca2c06b4d0 msgr2=0x7fca2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca2c06b4d0 0x7fca2c06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fca38006fd0 tx=0x7fca38008040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 msgr2=0x7fca500cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 0x7fca500cb4e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c400 tx=0x7fca4000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 shutdown_connections 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca2c06b4d0 0x7fca2c06d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca500cdc20 0x7fca500c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca500af580 0x7fca5014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 --2- 172.21.15.32:0/922243653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca500af1a0 0x7fca500cb4e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 >> 172.21.15.32:0/922243653 conn(0x7fca5006afb0 msgr2=0x7fca50071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 shutdown_connections 2024-03-06T07:29:48.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:48.511+0000 7fca56600640 1 -- 172.21.15.32:0/922243653 wait complete. 2024-03-06T07:29:48.529 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-06T07:29:48.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:48.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-06T07:29:48.641 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:29:48.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:48.641 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:29:48.739 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:48.739 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:48.786 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:48.786 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:48.879 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:29:48.879 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:48.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:49.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.170+0000 7f1cffe00640 1 Processor -- start 2024-03-06T07:29:49.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.170+0000 7f1cffe00640 1 -- start start 2024-03-06T07:29:49.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.171+0000 7f1cffe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1d001552b0 0x7f1d0015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cffe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0015ed10 con 0x7f1d0015e810 2024-03-06T07:29:49.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cffe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0015eff0 con 0x7f1d000cd060 2024-03-06T07:29:49.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cffe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0015f2d0 con 0x7f1d001552b0 2024-03-06T07:29:49.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cf7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1d0015e810 0x7f1d000cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34150/0 (socket says 172.21.15.32:34150) 2024-03-06T07:29:49.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cf7e00640 1 -- 172.21.15.32:0/2920206318 learned_addr learned my addr 172.21.15.32:0/2920206318 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:49.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cfea00640 1 --2- 172.21.15.32:0/2920206318 >> v2:172.21.15.39:3300/0 conn(0x7f1d001552b0 0x7f1d0015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144796010 0 0) 0x7f1d0015ed10 con 0x7f1d0015e810 2024-03-06T07:29:49.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.172+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4003520 con 0x7f1d0015e810 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1907713194 0 0) 0x7f1ce4003520 con 0x7f1d0015e810 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 >> v2:172.21.15.39:3300/0 conn(0x7f1d001552b0 msgr2=0x7f1d0015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 --2- 172.21.15.32:0/2920206318 >> v2:172.21.15.39:3300/0 conn(0x7f1d001552b0 0x7f1d0015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 >> v1:172.21.15.39:6789/0 conn(0x7f1d000cd060 legacy=0x7f1d000b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0015f5b0 con 0x7f1d0015e810 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f1ce8001670 con 0x7f1d0015e810 2024-03-06T07:29:49.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 >> v1:172.21.15.32:6789/0 conn(0x7f1d0015e810 legacy=0x7f1d000cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce40039b0 0x7f1ce4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1ce4004800 0x7f1ce4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1ce400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0015f2d0 con 0x7f1ce40051c0 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce400e370 con 0x7f1ce40039b0 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce400e570 con 0x7f1ce4004800 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cff400640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1ce400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.173+0000 7f1cf7e00640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1ce4004800 0x7f1ce4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cfea00640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce40039b0 0x7f1ce4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cff400640 1 -- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1ce4004800 msgr2=0x7f1ce4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cff400640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1ce4004800 0x7f1ce4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cff400640 1 -- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce40039b0 msgr2=0x7f1ce4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cff400640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce40039b0 0x7f1ce4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cff400640 1 -- 172.21.15.32:0/2920206318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4003520 con 0x7f1ce40051c0 2024-03-06T07:29:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.174+0000 7f1cfea00640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce40039b0 0x7f1ce4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:49.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cff400640 1 -- 172.21.15.32:0/2920206318 reap_dead start 2024-03-06T07:29:49.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cff400640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1ce400dbd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8001e20 tx=0x7f1ce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a245315253727a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8001a70 con 0x7f1ce40051c0 2024-03-06T07:29:49.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf8003b70 con 0x7f1ce40051c0 2024-03-06T07:29:49.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cfe000640 1 -- 172.21.15.32:0/2920206318 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf80050d0 con 0x7f1ce40051c0 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cffe00640 1 -- 172.21.15.32:0/2920206318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 msgr2=0x7f1ce400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.175+0000 7f1cffe00640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1ce400dbd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8001e20 tx=0x7f1ce4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.176+0000 7f1cffe00640 1 -- 172.21.15.32:0/2920206318 shutdown_connections 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.176+0000 7f1cffe00640 1 --2- 172.21.15.32:0/2920206318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1ce400dbd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.176+0000 7f1cffe00640 1 -- 172.21.15.32:0/2920206318 >> 172.21.15.32:0/2920206318 conn(0x7f1d0006b3d0 msgr2=0x7f1d0006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.176+0000 7f1cffe00640 1 -- 172.21.15.32:0/2920206318 shutdown_connections 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.176+0000 7f1cffe00640 1 -- 172.21.15.32:0/2920206318 wait complete. 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.177+0000 7f1cffe00640 1 Processor -- start 2024-03-06T07:29:49.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.178+0000 7f1cffe00640 1 -- start start 2024-03-06T07:29:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.178+0000 7f1cffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1d001515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cffe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1d001510e0 0x7f1d00150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d000b3b00 0x7f1d000b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cffe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0006dd30 con 0x7f1ce40051c0 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cffe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d000b3760 con 0x7f1d000b3b00 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cffe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0006deb0 con 0x7f1d001510e0 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1d001515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1d001515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35740/0 (socket says 172.21.15.32:35740) 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 -- 172.21.15.32:0/3171195805 learned_addr learned my addr 172.21.15.32:0/3171195805 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cf7e00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1d001510e0 0x7f1d00150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 -- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1d001510e0 msgr2=0x7f1d00150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1d001510e0 0x7f1d00150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 -- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d000b3b00 msgr2=0x7f1d000b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cff400640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d000b3b00 0x7f1d000b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d000b3b00 0x7f1d000b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cfea00640 1 -- 172.21.15.32:0/3171195805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d00157060 con 0x7f1ce40051c0 2024-03-06T07:29:49.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cf7e00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1d001510e0 0x7f1d00150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.179+0000 7f1cff400640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d000b3b00 0x7f1d000b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.180+0000 7f1cfea00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1d001515d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00108b0 tx=0x7f1cf0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.180+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf000f040 con 0x7f1ce40051c0 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.180+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf000eb60 con 0x7f1ce40051c0 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.180+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf001c400 con 0x7f1ce40051c0 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.180+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d00157340 con 0x7f1ce40051c0 2024-03-06T07:29:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.180+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d00157900 con 0x7f1ce40051c0 2024-03-06T07:29:49.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.181+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d00067830 con 0x7f1ce40051c0 2024-03-06T07:29:49.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.182+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cf001c5a0 con 0x7f1ce40051c0 2024-03-06T07:29:49.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.183+0000 7f1cf6a00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1cdc06b4d0 0x7f1cdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.183+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1cf008a9d0 con 0x7f1ce40051c0 2024-03-06T07:29:49.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.183+0000 7f1cf7e00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1cdc06b4d0 0x7f1cdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.184+0000 7f1cf7e00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1cdc06b4d0 0x7f1cdc06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8001200 tx=0x7f1ce8000f30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.187+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d00067830 con 0x7f1ce40051c0 2024-03-06T07:29:49.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.421+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d0004f330 con 0x7f1ce40051c0 2024-03-06T07:29:49.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.423+0000 7f1cf6a00640 1 -- 172.21.15.32:0/3171195805 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16746 (secure 0 0 0) 0x7f1cf0018030 con 0x7f1ce40051c0 2024-03-06T07:29:49.425 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:49.425 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:47.656917+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:49.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1cdc06b4d0 msgr2=0x7f1cdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1cdc06b4d0 0x7f1cdc06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8001200 tx=0x7f1ce8000f30 comp rx=0 tx=0).stop 2024-03-06T07:29:49.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 msgr2=0x7f1d001515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1d001515d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00108b0 tx=0x7f1cf0010d70 comp rx=0 tx=0).stop 2024-03-06T07:29:49.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 shutdown_connections 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1cdc06b4d0 0x7f1cdc06d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d000b3b00 0x7f1d000b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1d001510e0 0x7f1d00150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 --2- 172.21.15.32:0/3171195805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ce40051c0 0x7f1d001515d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 >> 172.21.15.32:0/3171195805 conn(0x7f1d0006b3d0 msgr2=0x7f1d0006e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 shutdown_connections 2024-03-06T07:29:49.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.426+0000 7f1cffe00640 1 -- 172.21.15.32:0/3171195805 wait complete. 2024-03-06T07:29:49.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:49.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.649+0000 7f8b41600640 1 Processor -- start 2024-03-06T07:29:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.649+0000 7f8b41600640 1 -- start start 2024-03-06T07:29:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b41600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b3c0ce6e0 0x7f8b3c0c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b41600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0c75b0 0x7f8b3c06ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b41600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c06dc50 con 0x7f8b3c06afb0 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b41600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c06df30 con 0x7f8b3c0ce6e0 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b41600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c06e210 con 0x7f8b3c0c75b0 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b3a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b3c0ce6e0 0x7f8b3c0c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b3a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b3c0ce6e0 0x7f8b3c0c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35756/0 (socket says 172.21.15.32:35756) 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.651+0000 7f8b3a000640 1 -- 172.21.15.32:0/1316663403 learned_addr learned my addr 172.21.15.32:0/1316663403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.652+0000 7f8b3b400640 1 --2- 172.21.15.32:0/1316663403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0c75b0 0x7f8b3c06ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.652+0000 7f8b3a000640 1 -- 172.21.15.32:0/1316663403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0c75b0 msgr2=0x7f8b3c06ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.652+0000 7f8b3a000640 1 --2- 172.21.15.32:0/1316663403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0c75b0 0x7f8b3c06ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.652+0000 7f8b3a000640 1 -- 172.21.15.32:0/1316663403 >> v1:172.21.15.39:6789/0 conn(0x7f8b3c06afb0 legacy=0x7f8b3c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.652+0000 7f8b3a000640 1 -- 172.21.15.32:0/1316663403 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c06e4f0 con 0x7f8b3c0ce6e0 2024-03-06T07:29:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.653+0000 7f8b3a000640 1 --2- 172.21.15.32:0/1316663403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0c7070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009870 tx=0x7f8b2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec57ce9b401ece5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.653+0000 7f8b39600640 1 -- 172.21.15.32:0/1316663403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28035020 con 0x7f8b3c0ce6e0 2024-03-06T07:29:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.653+0000 7f8b39600640 1 -- 172.21.15.32:0/1316663403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28031640 con 0x7f8b3c0ce6e0 2024-03-06T07:29:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.653+0000 7f8b39600640 1 -- 172.21.15.32:0/1316663403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2802d040 con 0x7f8b3c0ce6e0 2024-03-06T07:29:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 -- 172.21.15.32:0/1316663403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0ce6e0 msgr2=0x7f8b3c0c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 --2- 172.21.15.32:0/1316663403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0c7070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009870 tx=0x7f8b2802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 -- 172.21.15.32:0/1316663403 shutdown_connections 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 --2- 172.21.15.32:0/1316663403 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0c75b0 0x7f8b3c06ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 --2- 172.21.15.32:0/1316663403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0c7070 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 -- 172.21.15.32:0/1316663403 >> 172.21.15.32:0/1316663403 conn(0x7f8b3c0b2760 msgr2=0x7f8b3c0b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 -- 172.21.15.32:0/1316663403 shutdown_connections 2024-03-06T07:29:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.654+0000 7f8b41600640 1 -- 172.21.15.32:0/1316663403 wait complete. 2024-03-06T07:29:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.656+0000 7f8b41600640 1 Processor -- start 2024-03-06T07:29:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.656+0000 7f8b41600640 1 -- start start 2024-03-06T07:29:49.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b41600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 0x7f8b3c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b41600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0c75b0 0x7f8b3c0b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b41600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b41600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15f230 con 0x7f8b3c06afb0 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b41600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15e2a0 con 0x7f8b3c0c75b0 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b41600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15f3b0 con 0x7f8b3c0ce6e0 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 0x7f8b3c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 0x7f8b3c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35770/0 (socket says 172.21.15.32:35770) 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.657+0000 7f8b3aa00640 1 -- 172.21.15.32:0/1196119679 learned_addr learned my addr 172.21.15.32:0/1196119679 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3a000640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0c75b0 0x7f8b3c0b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3b400640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3aa00640 1 -- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0ce6e0 msgr2=0x7f8b3c0b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3aa00640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3aa00640 1 -- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0c75b0 msgr2=0x7f8b3c0b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3aa00640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0c75b0 0x7f8b3c0b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3aa00640 1 -- 172.21.15.32:0/1196119679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c1f3ec0 con 0x7f8b3c06afb0 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3a000640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0c75b0 0x7f8b3c0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b3aa00640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 0x7f8b3c15d0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00c400 tx=0x7f8b2c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c017070 con 0x7f8b3c06afb0 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c0052d0 con 0x7f8b3c06afb0 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c003830 con 0x7f8b3c06afb0 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c1e9000 con 0x7f8b3c06afb0 2024-03-06T07:29:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.658+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c1f4460 con 0x7f8b3c06afb0 2024-03-06T07:29:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.660+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c04f330 con 0x7f8b3c06afb0 2024-03-06T07:29:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.660+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b2c01a070 con 0x7f8b3c06afb0 2024-03-06T07:29:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.661+0000 7f8b32a00640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.661+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8b2c089550 con 0x7f8b3c06afb0 2024-03-06T07:29:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.662+0000 7f8b3a000640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.662+0000 7f8b3a000640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b28036000 tx=0x7f8b28008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.666+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b2c0583e0 con 0x7f8b3c06afb0 2024-03-06T07:29:49.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.863+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b3c071ee0 con 0x7f8b3c06afb0 2024-03-06T07:29:49.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.864+0000 7f8b32a00640 1 -- 172.21.15.32:0/1196119679 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b2c087e20 con 0x7f8b3c06afb0 2024-03-06T07:29:49.865 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 msgr2=0x7f8b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b28036000 tx=0x7f8b28008030 comp rx=0 tx=0).stop 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 msgr2=0x7f8b3c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 0x7f8b3c15d0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00c400 tx=0x7f8b2c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 shutdown_connections 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c0ce6e0 0x7f8b3c0b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0c75b0 0x7f8b3c0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.866+0000 7f8b41600640 1 --2- 172.21.15.32:0/1196119679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c06afb0 0x7f8b3c15d0e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.867+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 >> 172.21.15.32:0/1196119679 conn(0x7f8b3c0b2760 msgr2=0x7f8b3c15f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.867+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 shutdown_connections 2024-03-06T07:29:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:49.867+0000 7f8b41600640 1 -- 172.21.15.32:0/1196119679 wait complete. 2024-03-06T07:29:49.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:50.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.099+0000 7f1079000640 1 Processor -- start 2024-03-06T07:29:50.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.099+0000 7f1079000640 1 -- start start 2024-03-06T07:29:50.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1079000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1074147590 0x7f1074071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1079000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1079000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10740719c0 con 0x7f107406d8d0 2024-03-06T07:29:50.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1079000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1074147970 con 0x7f1074147590 2024-03-06T07:29:50.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1079000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10740b31b0 con 0x7f10740b2250 2024-03-06T07:29:50.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1073400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1074147590 0x7f1074071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f106aa00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f107406d8d0 0x7f10740b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34156/0 (socket says 172.21.15.32:34156) 2024-03-06T07:29:50.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f106aa00640 1 -- 172.21.15.32:0/3191063625 learned_addr learned my addr 172.21.15.32:0/3191063625 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:50.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.101+0000 7f1073e00640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1072a00640 1 -- 172.21.15.32:0/3191063625 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201149622 0 0) 0x7f10740719c0 con 0x7f107406d8d0 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1072a00640 1 -- 172.21.15.32:0/3191063625 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1058003520 con 0x7f107406d8d0 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1073400640 1 -- 172.21.15.32:0/3191063625 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 msgr2=0x7f10740b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1073400640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1073400640 1 -- 172.21.15.32:0/3191063625 >> v1:172.21.15.32:6789/0 conn(0x7f107406d8d0 legacy=0x7f10740b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1073400640 1 -- 172.21.15.32:0/3191063625 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107406de40 con 0x7f1074147590 2024-03-06T07:29:50.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.102+0000 7f1073e00640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.103+0000 7f1073400640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074071480 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f10640097a0 tx=0x7f106402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1bf8fa847e8872 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.103+0000 7f1072a00640 1 -- 172.21.15.32:0/3191063625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064035020 con 0x7f1074147590 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.103+0000 7f1072a00640 1 -- 172.21.15.32:0/3191063625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064031810 con 0x7f1074147590 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.103+0000 7f1072a00640 1 -- 172.21.15.32:0/3191063625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064031be0 con 0x7f1074147590 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 -- 172.21.15.32:0/3191063625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 msgr2=0x7f1074071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074071480 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f10640097a0 tx=0x7f106402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 -- 172.21.15.32:0/3191063625 shutdown_connections 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 --2- 172.21.15.32:0/3191063625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074071480 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 -- 172.21.15.32:0/3191063625 >> 172.21.15.32:0/3191063625 conn(0x7f107406b3a0 msgr2=0x7f107406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 -- 172.21.15.32:0/3191063625 shutdown_connections 2024-03-06T07:29:50.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.104+0000 7f1079000640 1 -- 172.21.15.32:0/3191063625 wait complete. 2024-03-06T07:29:50.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.105+0000 7f1079000640 1 Processor -- start 2024-03-06T07:29:50.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 -- start start 2024-03-06T07:29:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f107406d8d0 0x7f1074150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10740b2b50 con 0x7f1074147590 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10740b2e40 con 0x7f107406d8d0 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.106+0000 7f1079000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10740b3b80 con 0x7f10740b2250 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074141800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35796/0 (socket says 172.21.15.32:35796) 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 -- 172.21.15.32:0/4250608346 learned_addr learned my addr 172.21.15.32:0/4250608346 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f106aa00640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073400640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f107406d8d0 0x7f1074150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 -- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 msgr2=0x7f10740cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 -- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f107406d8d0 msgr2=0x7f1074150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f107406d8d0 0x7f1074150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 -- 172.21.15.32:0/4250608346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10741f46a0 con 0x7f1074147590 2024-03-06T07:29:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.107+0000 7f1073e00640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074141800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f106c009f70 tx=0x7f106c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.108+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c01a070 con 0x7f1074147590 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.108+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c003bb0 con 0x7f1074147590 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.108+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c004030 con 0x7f1074147590 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.108+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1074141fa0 con 0x7f1074147590 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.108+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10741f3bc0 con 0x7f1074147590 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.109+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107404f330 con 0x7f1074147590 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.110+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f106c01d070 con 0x7f1074147590 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.110+0000 7f106be00640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f105006b4d0 0x7f105006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.110+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f106c08a5a0 con 0x7f1074147590 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.111+0000 7f1073400640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f105006b4d0 0x7f105006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.112+0000 7f1073400640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f105006b4d0 0x7f105006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1064036000 tx=0x7f1064008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.115+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f106c010070 con 0x7f1074147590 2024-03-06T07:29:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.307+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f107413f580 con 0x7f1074147590 2024-03-06T07:29:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.308+0000 7f106be00640 1 -- 172.21.15.32:0/4250608346 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f106c059300 con 0x7f1074147590 2024-03-06T07:29:50.309 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f105006b4d0 msgr2=0x7f105006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f105006b4d0 0x7f105006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1064036000 tx=0x7f1064008030 comp rx=0 tx=0).stop 2024-03-06T07:29:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 msgr2=0x7f1074141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074141800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f106c009f70 tx=0x7f106c00d560 comp rx=0 tx=0).stop 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 shutdown_connections 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f105006b4d0 0x7f105006d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1074147590 0x7f1074141800 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f10740b2250 0x7f10740cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 --2- 172.21.15.32:0/4250608346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f107406d8d0 0x7f1074150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 >> 172.21.15.32:0/4250608346 conn(0x7f107406b3a0 msgr2=0x7f107406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 shutdown_connections 2024-03-06T07:29:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.311+0000 7f1079000640 1 -- 172.21.15.32:0/4250608346 wait complete. 2024-03-06T07:29:50.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:50.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.540+0000 7fa92cc00640 1 Processor -- start 2024-03-06T07:29:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.540+0000 7fa92cc00640 1 -- start start 2024-03-06T07:29:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.542+0000 7fa92cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa928144560 0x7fa92806e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.542+0000 7fa92cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa928147510 0x7fa92806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.542+0000 7fa92cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa92806b4f0 0x7fa9280cd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.542+0000 7fa92cc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa928162ae0 con 0x7fa928147510 2024-03-06T07:29:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.542+0000 7fa92cc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa92806b920 con 0x7fa92806b4f0 2024-03-06T07:29:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.542+0000 7fa92cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa928161d20 con 0x7fa928144560 2024-03-06T07:29:50.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa928147510 0x7fa92806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa928147510 0x7fa92806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35810/0 (socket says 172.21.15.32:35810) 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 -- 172.21.15.32:0/2547628391 learned_addr learned my addr 172.21.15.32:0/2547628391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa926800640 1 --2- 172.21.15.32:0/2547628391 >> v2:172.21.15.39:3300/0 conn(0x7fa92806b4f0 0x7fa9280cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa925e00640 1 --2- 172.21.15.32:0/2547628391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa928144560 0x7fa92806e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 -- 172.21.15.32:0/2547628391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa928144560 msgr2=0x7fa92806e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 --2- 172.21.15.32:0/2547628391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa928144560 0x7fa92806e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 -- 172.21.15.32:0/2547628391 >> v2:172.21.15.39:3300/0 conn(0x7fa92806b4f0 msgr2=0x7fa9280cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 --2- 172.21.15.32:0/2547628391 >> v2:172.21.15.39:3300/0 conn(0x7fa92806b4f0 0x7fa9280cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.543+0000 7fa91d400640 1 -- 172.21.15.32:0/2547628391 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928162000 con 0x7fa928147510 2024-03-06T07:29:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.544+0000 7fa91d400640 1 --2- 172.21.15.32:0/2547628391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928147510 0x7fa92806afb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa92806fcf0 tx=0x7fa91002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d994aabcf85b814e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.545+0000 7fa925400640 1 -- 172.21.15.32:0/2547628391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9100317c0 con 0x7fa928147510 2024-03-06T07:29:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.545+0000 7fa925400640 1 -- 172.21.15.32:0/2547628391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910037070 con 0x7fa928147510 2024-03-06T07:29:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.545+0000 7fa925400640 1 -- 172.21.15.32:0/2547628391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910031b80 con 0x7fa928147510 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.545+0000 7fa92cc00640 1 -- 172.21.15.32:0/2547628391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928147510 msgr2=0x7fa92806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.545+0000 7fa92cc00640 1 --2- 172.21.15.32:0/2547628391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928147510 0x7fa92806afb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa92806fcf0 tx=0x7fa91002b860 comp rx=0 tx=0).stop 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 -- 172.21.15.32:0/2547628391 shutdown_connections 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 --2- 172.21.15.32:0/2547628391 >> v2:172.21.15.39:3300/0 conn(0x7fa92806b4f0 0x7fa9280cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 --2- 172.21.15.32:0/2547628391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928147510 0x7fa92806afb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 --2- 172.21.15.32:0/2547628391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa928144560 0x7fa92806e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 -- 172.21.15.32:0/2547628391 >> 172.21.15.32:0/2547628391 conn(0x7fa9280b2860 msgr2=0x7fa9280b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 -- 172.21.15.32:0/2547628391 shutdown_connections 2024-03-06T07:29:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.546+0000 7fa92cc00640 1 -- 172.21.15.32:0/2547628391 wait complete. 2024-03-06T07:29:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.547+0000 7fa92cc00640 1 Processor -- start 2024-03-06T07:29:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.548+0000 7fa92cc00640 1 -- start start 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa92cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa92806b4f0 0x7fa928160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa92cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 0x7fa9281f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa92cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa928147510 0x7fa9281f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa92cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92806efd0 con 0x7fa928144560 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa92cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281625c0 con 0x7fa928147510 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa92cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92806ebc0 con 0x7fa92806b4f0 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 0x7fa9281f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 0x7fa9281f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35816/0 (socket says 172.21.15.32:35816) 2024-03-06T07:29:50.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 -- 172.21.15.32:0/3562065209 learned_addr learned my addr 172.21.15.32:0/3562065209 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa925e00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa92806b4f0 0x7fa928160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa926800640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa928147510 0x7fa9281f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 -- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa92806b4f0 msgr2=0x7fa928160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa92806b4f0 0x7fa928160da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 -- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa928147510 msgr2=0x7fa9281f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.549+0000 7fa91d400640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa928147510 0x7fa9281f4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa91d400640 1 -- 172.21.15.32:0/3562065209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280b5f20 con 0x7fa928144560 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa91d400640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 0x7fa9281f4050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa91002b8a0 tx=0x7fa9100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910032650 con 0x7fa928144560 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910032810 con 0x7fa928144560 2024-03-06T07:29:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910032c00 con 0x7fa928144560 2024-03-06T07:29:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9280b6200 con 0x7fa928144560 2024-03-06T07:29:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.550+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9280b67c0 con 0x7fa928144560 2024-03-06T07:29:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.552+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92804f330 con 0x7fa928144560 2024-03-06T07:29:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.553+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa91004c060 con 0x7fa928144560 2024-03-06T07:29:50.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.554+0000 7fa91ea00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa90406b520 0x7fa90406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.554+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa9100abc70 con 0x7fa928144560 2024-03-06T07:29:50.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.554+0000 7fa925e00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa90406b520 0x7fa90406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.555+0000 7fa925e00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa90406b520 0x7fa90406d9e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9180099b0 tx=0x7fa918008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.559+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa910040070 con 0x7fa928144560 2024-03-06T07:29:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.759+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa928147bc0 con 0x7fa928144560 2024-03-06T07:29:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.760+0000 7fa91ea00640 1 -- 172.21.15.32:0/3562065209 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa91007aa50 con 0x7fa928144560 2024-03-06T07:29:50.762 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:50.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa90406b520 msgr2=0x7fa90406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa90406b520 0x7fa90406d9e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9180099b0 tx=0x7fa918008040 comp rx=0 tx=0).stop 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 msgr2=0x7fa9281f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 0x7fa9281f4050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa91002b8a0 tx=0x7fa9100046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 shutdown_connections 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa90406b520 0x7fa90406d9e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa928147510 0x7fa9281f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa928144560 0x7fa9281f4050 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 --2- 172.21.15.32:0/3562065209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa92806b4f0 0x7fa928160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 >> 172.21.15.32:0/3562065209 conn(0x7fa9280b2860 msgr2=0x7fa9280b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 shutdown_connections 2024-03-06T07:29:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:50.763+0000 7fa92cc00640 1 -- 172.21.15.32:0/3562065209 wait complete. 2024-03-06T07:29:50.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:51.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.004+0000 7f470a000640 1 Processor -- start 2024-03-06T07:29:51.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.004+0000 7f470a000640 1 -- start start 2024-03-06T07:29:51.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.005+0000 7f470a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f47040af580 0x7f470415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.005+0000 7f470a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040cdc20 0x7f47040b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f470a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470415cc40 con 0x7f47040af1a0 2024-03-06T07:29:51.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f470a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470415cf20 con 0x7f47040af580 2024-03-06T07:29:51.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f470a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470415d200 con 0x7f47040cdc20 2024-03-06T07:29:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f4703400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f47040af1a0 0x7f47040b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:52418/0 (socket says 172.21.15.32:52418) 2024-03-06T07:29:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f4703400640 1 -- 172.21.15.32:0/179362030 learned_addr learned my addr 172.21.15.32:0/179362030 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f46fbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f47040af580 0x7f470415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.006+0000 7f4703e00640 1 --2- 172.21.15.32:0/179362030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040cdc20 0x7f47040b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.007+0000 7f4702a00640 1 -- 172.21.15.32:0/179362030 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731589291 0 0) 0x7f470415cc40 con 0x7f47040af1a0 2024-03-06T07:29:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.007+0000 7f4703e00640 1 -- 172.21.15.32:0/179362030 >> v2:172.21.15.39:3300/0 conn(0x7f47040af580 msgr2=0x7f470415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.007+0000 7f4703e00640 1 --2- 172.21.15.32:0/179362030 >> v2:172.21.15.39:3300/0 conn(0x7f47040af580 0x7f470415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.007+0000 7f4703e00640 1 -- 172.21.15.32:0/179362030 >> v1:172.21.15.39:6789/0 conn(0x7f47040af1a0 legacy=0x7f47040b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.007+0000 7f4703e00640 1 -- 172.21.15.32:0/179362030 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470415d4e0 con 0x7f47040cdc20 2024-03-06T07:29:51.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.008+0000 7f4703e00640 1 --2- 172.21.15.32:0/179362030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040cdc20 0x7f47040b2dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009d90 tx=0x7f46fc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10628f242bbd1edd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.008+0000 7f4702a00640 1 -- 172.21.15.32:0/179362030 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc039070 con 0x7f47040cdc20 2024-03-06T07:29:51.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.008+0000 7f4702a00640 1 -- 172.21.15.32:0/179362030 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc02cbc0 con 0x7f47040cdc20 2024-03-06T07:29:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.008+0000 7f4702a00640 1 -- 172.21.15.32:0/179362030 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc034440 con 0x7f47040cdc20 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 -- 172.21.15.32:0/179362030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040cdc20 msgr2=0x7f47040b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 --2- 172.21.15.32:0/179362030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040cdc20 0x7f47040b2dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009d90 tx=0x7f46fc02be10 comp rx=0 tx=0).stop 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 -- 172.21.15.32:0/179362030 shutdown_connections 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 --2- 172.21.15.32:0/179362030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040cdc20 0x7f47040b2dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 --2- 172.21.15.32:0/179362030 >> v2:172.21.15.39:3300/0 conn(0x7f47040af580 0x7f470415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 -- 172.21.15.32:0/179362030 >> 172.21.15.32:0/179362030 conn(0x7f470406afb0 msgr2=0x7f470406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 -- 172.21.15.32:0/179362030 shutdown_connections 2024-03-06T07:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.009+0000 7f470a000640 1 -- 172.21.15.32:0/179362030 wait complete. 2024-03-06T07:29:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.012+0000 7f470a000640 1 Processor -- start 2024-03-06T07:29:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.012+0000 7f470a000640 1 -- start start 2024-03-06T07:29:51.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f470a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 0x7f47041f1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f470a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040af580 0x7f47041543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f470a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47040cdc20 0x7f470415e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f470a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47040b36e0 con 0x7f47040af1a0 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f470a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47040b3560 con 0x7f47040cdc20 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f470a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4704155e70 con 0x7f47040af580 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f4703400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 0x7f47041f1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f4703400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 0x7f47041f1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35828/0 (socket says 172.21.15.32:35828) 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f4703400640 1 -- 172.21.15.32:0/3676706526 learned_addr learned my addr 172.21.15.32:0/3676706526 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.013+0000 7f46fbe00640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040af580 0x7f47041543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703e00640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47040cdc20 0x7f470415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703400640 1 -- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040af580 msgr2=0x7f47041543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703400640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040af580 0x7f47041543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703400640 1 -- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47040cdc20 msgr2=0x7f470415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703400640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47040cdc20 0x7f470415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703400640 1 -- 172.21.15.32:0/3676706526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4704152c40 con 0x7f47040af1a0 2024-03-06T07:29:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703e00640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47040cdc20 0x7f470415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f4703400640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 0x7f47041f1430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f46f400c4d0 tx=0x7f46f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.014+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4018070 con 0x7f47040af1a0 2024-03-06T07:29:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.015+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4007500 con 0x7f47040af1a0 2024-03-06T07:29:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.015+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4704152f20 con 0x7f47040af1a0 2024-03-06T07:29:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.015+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4004a20 con 0x7f47040af1a0 2024-03-06T07:29:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.015+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47041534e0 con 0x7f47040af1a0 2024-03-06T07:29:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.016+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4704069510 con 0x7f47040af1a0 2024-03-06T07:29:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.017+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46f401b070 con 0x7f47040af1a0 2024-03-06T07:29:51.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.017+0000 7f46fb400640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46e006b4d0 0x7f46e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.018+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f46f40895b0 con 0x7f47040af1a0 2024-03-06T07:29:51.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.018+0000 7f46fbe00640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46e006b4d0 0x7f46e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.018+0000 7f46fbe00640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46e006b4d0 0x7f46e006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f47041f4ef0 tx=0x7f46ec009190 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.023+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46f4057cc0 con 0x7f47040af1a0 2024-03-06T07:29:51.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.238+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f470409b400 con 0x7f47040af1a0 2024-03-06T07:29:51.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.239+0000 7f46fb400640 1 -- 172.21.15.32:0/3676706526 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f470409b400 con 0x7f47040af1a0 2024-03-06T07:29:51.241 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:29:51.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.242+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46e006b4d0 msgr2=0x7f46e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.242+0000 7f470a000640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46e006b4d0 0x7f46e006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f47041f4ef0 tx=0x7f46ec009190 comp rx=0 tx=0).stop 2024-03-06T07:29:51.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.242+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 msgr2=0x7f47041f1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.242+0000 7f470a000640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 0x7f47041f1430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f46f400c4d0 tx=0x7f46f400c990 comp rx=0 tx=0).stop 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 shutdown_connections 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46e006b4d0 0x7f46e006d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47040cdc20 0x7f470415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f47040af580 0x7f47041543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 --2- 172.21.15.32:0/3676706526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47040af1a0 0x7f47041f1430 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 >> 172.21.15.32:0/3676706526 conn(0x7f470406afb0 msgr2=0x7f4704071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 shutdown_connections 2024-03-06T07:29:51.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.243+0000 7f470a000640 1 -- 172.21.15.32:0/3676706526 wait complete. 2024-03-06T07:29:51.261 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-06T07:29:51.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-06T07:29:51.360 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:29:51.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.360 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:29:51.414 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:51.414 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.468 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:51.469 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.509 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:29:51.509 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:51.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.823+0000 7fb7eb200640 1 Processor -- start 2024-03-06T07:29:51.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.823+0000 7fb7eb200640 1 -- start start 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7eb200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7ec145300 0x7fb7ec0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7eb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb7ec070f10 0x7fb7ec0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec0cce90 0x7fb7ec0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7eb200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec0cd300 con 0x7fb7ec145300 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7eb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec0714d0 con 0x7fb7ec070f10 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0717b0 con 0x7fb7ec0cce90 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7e9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7ec145300 0x7fb7ec0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7e9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7ec145300 0x7fb7ec0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35840/0 (socket says 172.21.15.32:35840) 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.825+0000 7fb7e9e00640 1 -- 172.21.15.32:0/3234810576 learned_addr learned my addr 172.21.15.32:0/3234810576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7ea800640 1 --2- 172.21.15.32:0/3234810576 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec0cce90 0x7fb7ec0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7e9400640 1 --2- 172.21.15.32:0/3234810576 >> v2:172.21.15.39:3300/0 conn(0x7fb7ec070f10 0x7fb7ec0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7e9e00640 1 -- 172.21.15.32:0/3234810576 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec0cce90 msgr2=0x7fb7ec0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7e9e00640 1 --2- 172.21.15.32:0/3234810576 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec0cce90 0x7fb7ec0cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7e9e00640 1 -- 172.21.15.32:0/3234810576 >> v2:172.21.15.39:3300/0 conn(0x7fb7ec070f10 msgr2=0x7fb7ec0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7e9e00640 1 --2- 172.21.15.32:0/3234810576 >> v2:172.21.15.39:3300/0 conn(0x7fb7ec070f10 0x7fb7ec0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.826+0000 7fb7e9e00640 1 -- 172.21.15.32:0/3234810576 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec071a90 con 0x7fb7ec145300 2024-03-06T07:29:51.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.827+0000 7fb7e9e00640 1 --2- 172.21.15.32:0/3234810576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b49a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0b5150 tx=0x7fb7dc02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1ab2427ef0f2c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.827+0000 7fb7e8a00640 1 -- 172.21.15.32:0/3234810576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0317d0 con 0x7fb7ec145300 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.827+0000 7fb7e8a00640 1 -- 172.21.15.32:0/3234810576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc037070 con 0x7fb7ec145300 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.827+0000 7fb7e8a00640 1 -- 172.21.15.32:0/3234810576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc031b90 con 0x7fb7ec145300 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 -- 172.21.15.32:0/3234810576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec145300 msgr2=0x7fb7ec0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 --2- 172.21.15.32:0/3234810576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b49a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0b5150 tx=0x7fb7dc02b870 comp rx=0 tx=0).stop 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 -- 172.21.15.32:0/3234810576 shutdown_connections 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 --2- 172.21.15.32:0/3234810576 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec0cce90 0x7fb7ec0cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 --2- 172.21.15.32:0/3234810576 >> v2:172.21.15.39:3300/0 conn(0x7fb7ec070f10 0x7fb7ec0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 --2- 172.21.15.32:0/3234810576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b49a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 -- 172.21.15.32:0/3234810576 >> 172.21.15.32:0/3234810576 conn(0x7fb7ec06b3d0 msgr2=0x7fb7ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 -- 172.21.15.32:0/3234810576 shutdown_connections 2024-03-06T07:29:51.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.828+0000 7fb7eb200640 1 -- 172.21.15.32:0/3234810576 wait complete. 2024-03-06T07:29:51.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.830+0000 7fb7eb200640 1 Processor -- start 2024-03-06T07:29:51.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.830+0000 7fb7eb200640 1 -- start start 2024-03-06T07:29:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec070f10 0x7fb7ec0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 0x7fb7ec0c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7eb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7eb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1456e0 con 0x7fb7ec0cce90 2024-03-06T07:29:51.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7eb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0b4ee0 con 0x7fb7ec145300 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0cde70 con 0x7fb7ec070f10 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7e9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 0x7fb7ec0c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7e9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 0x7fb7ec0c4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35852/0 (socket says 172.21.15.32:35852) 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7e9400640 1 -- 172.21.15.32:0/395783791 learned_addr learned my addr 172.21.15.32:0/395783791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.831+0000 7fb7e9e00640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec070f10 0x7fb7ec0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7ea800640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e9400640 1 -- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec070f10 msgr2=0x7fb7ec0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e9400640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec070f10 0x7fb7ec0cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e9400640 1 -- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7ec145300 msgr2=0x7fb7ec0b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e9400640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e9400640 1 -- 172.21.15.32:0/395783791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec1364e0 con 0x7fb7ec0cce90 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7ea800640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e9400640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 0x7fb7ec0c4b70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800d800 tx=0x7fb7d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800b6c0 con 0x7fb7ec0cce90 2024-03-06T07:29:51.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d8019070 con 0x7fb7ec0cce90 2024-03-06T07:29:51.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.832+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800ba80 con 0x7fb7ec0cce90 2024-03-06T07:29:51.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.833+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec1367c0 con 0x7fb7ec0cce90 2024-03-06T07:29:51.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.833+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec136d80 con 0x7fb7ec0cce90 2024-03-06T07:29:51.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.834+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec04f330 con 0x7fb7ec0cce90 2024-03-06T07:29:51.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.835+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7d800bc20 con 0x7fb7ec0cce90 2024-03-06T07:29:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.836+0000 7fb7e2000640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7cc06b520 0x7fb7cc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.836+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb7d808a770 con 0x7fb7ec0cce90 2024-03-06T07:29:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.836+0000 7fb7e9e00640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7cc06b520 0x7fb7cc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.837+0000 7fb7e9e00640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7cc06b520 0x7fb7cc06d9e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc036000 tx=0x7fb7dc03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:51.840+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7d80594c0 con 0x7fb7ec0cce90 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.041+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7ec0aeef0 con 0x7fb7ec0cce90 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.044+0000 7fb7e2000640 1 -- 172.21.15.32:0/395783791 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16746 (secure 0 0 0) 0x7fb7d8058dc0 con 0x7fb7ec0cce90 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:47.656917+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7cc06b520 msgr2=0x7fb7cc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7cc06b520 0x7fb7cc06d9e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc036000 tx=0x7fb7dc03d040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 msgr2=0x7fb7ec0c4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 0x7fb7ec0c4b70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800d800 tx=0x7fb7d800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 shutdown_connections 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7cc06b520 0x7fb7cc06d9e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7ec145300 0x7fb7ec0b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7ec0cce90 0x7fb7ec0c4b70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 --2- 172.21.15.32:0/395783791 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7ec070f10 0x7fb7ec0cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 >> 172.21.15.32:0/395783791 conn(0x7fb7ec06b3d0 msgr2=0x7fb7ec0ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 shutdown_connections 2024-03-06T07:29:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.046+0000 7fb7eb200640 1 -- 172.21.15.32:0/395783791 wait complete. 2024-03-06T07:29:52.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:52.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.294+0000 7ffb23600640 1 Processor -- start 2024-03-06T07:29:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.294+0000 7ffb23600640 1 -- start start 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.295+0000 7ffb23600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb1c072520 0x7ffb1c0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.295+0000 7ffb23600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c072df0 con 0x7ffb1c071e20 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb23600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0cdab0 con 0x7ffb1c15bf20 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb23600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c0b4d70 con 0x7ffb1c072520 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb21600640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffb1c071e20 0x7ffb1c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34162/0 (socket says 172.21.15.32:34162) 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb21600640 1 -- 172.21.15.32:0/3151366760 learned_addr learned my addr 172.21.15.32:0/3151366760 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb1be00640 1 --2- 172.21.15.32:0/3151366760 >> v2:172.21.15.39:3300/0 conn(0x7ffb1c072520 0x7ffb1c0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238378203 0 0) 0x7ffb1c072df0 con 0x7ffb1c071e20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.296+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003520 con 0x7ffb1c071e20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565365073 0 0) 0x7ffb1c0cdab0 con 0x7ffb1c15bf20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c072df0 con 0x7ffb1c15bf20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334376943 0 0) 0x7ffb04003520 con 0x7ffb1c071e20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 >> v2:172.21.15.39:3300/0 conn(0x7ffb1c072520 msgr2=0x7ffb1c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 --2- 172.21.15.32:0/3151366760 >> v2:172.21.15.39:3300/0 conn(0x7ffb1c072520 0x7ffb1c0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 >> v1:172.21.15.39:6789/0 conn(0x7ffb1c15bf20 legacy=0x7ffb1c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0b5050 con 0x7ffb1c071e20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ffb140019f0 con 0x7ffb1c071e20 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 >> v1:172.21.15.32:6789/0 conn(0x7ffb1c071e20 legacy=0x7ffb1c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb04003b40 0x7ffb04003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb04004800 0x7ffb04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb040051c0 0x7ffb0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0b4d70 con 0x7ffb040051c0 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400bfe0 con 0x7ffb04004800 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400c1e0 con 0x7ffb04003b40 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb21600640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb040051c0 0x7ffb0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb1be00640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb04004800 0x7ffb04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.297+0000 7ffb20c00640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb04003b40 0x7ffb04003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.298+0000 7ffb21600640 1 -- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb04003b40 msgr2=0x7ffb04003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.298+0000 7ffb21600640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb04003b40 0x7ffb04003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.298+0000 7ffb21600640 1 -- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb04004800 msgr2=0x7ffb04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.298+0000 7ffb21600640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb04004800 0x7ffb04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.298+0000 7ffb21600640 1 -- 172.21.15.32:0/3151366760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04003520 con 0x7ffb040051c0 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb21600640 1 -- 172.21.15.32:0/3151366760 reap_dead start 2024-03-06T07:29:52.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb21600640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb040051c0 0x7ffb0400b9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400a7f0 tx=0x7ffb04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b678ccb492f730 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1400a410 con 0x7ffb040051c0 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14034650 con 0x7ffb040051c0 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb1b400640 1 -- 172.21.15.32:0/3151366760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1402ce60 con 0x7ffb040051c0 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb23600640 1 -- 172.21.15.32:0/3151366760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb040051c0 msgr2=0x7ffb0400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.299+0000 7ffb23600640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb040051c0 0x7ffb0400b9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400a7f0 tx=0x7ffb04005830 comp rx=0 tx=0).stop 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.300+0000 7ffb23600640 1 -- 172.21.15.32:0/3151366760 shutdown_connections 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.300+0000 7ffb23600640 1 --2- 172.21.15.32:0/3151366760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb040051c0 0x7ffb0400b9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.300+0000 7ffb23600640 1 -- 172.21.15.32:0/3151366760 >> 172.21.15.32:0/3151366760 conn(0x7ffb1c06b3d0 msgr2=0x7ffb1c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.300+0000 7ffb23600640 1 -- 172.21.15.32:0/3151366760 shutdown_connections 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.300+0000 7ffb23600640 1 -- 172.21.15.32:0/3151366760 wait complete. 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.301+0000 7ffb23600640 1 Processor -- start 2024-03-06T07:29:52.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 -- start start 2024-03-06T07:29:52.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb040051c0 0x7ffb1c0c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb1c09d320 0x7ffb1c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 0x7ffb1c0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0b4b10 con 0x7ffb1c14e820 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0b4990 con 0x7ffb1c09d320 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.302+0000 7ffb23600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0acd40 con 0x7ffb040051c0 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 0x7ffb1c0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 0x7ffb1c0cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35870/0 (socket says 172.21.15.32:35870) 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 -- 172.21.15.32:0/1670102299 learned_addr learned my addr 172.21.15.32:0/1670102299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb20c00640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb040051c0 0x7ffb1c0c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb1be00640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb1c09d320 0x7ffb1c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 -- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb040051c0 msgr2=0x7ffb1c0c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb040051c0 0x7ffb1c0c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 -- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb1c09d320 msgr2=0x7ffb1c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb1c09d320 0x7ffb1c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb21600640 1 -- 172.21.15.32:0/1670102299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0b0af0 con 0x7ffb1c14e820 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.303+0000 7ffb1be00640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb1c09d320 0x7ffb1c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.304+0000 7ffb21600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 0x7ffb1c0cb010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffb1402cbf0 tx=0x7ffb1c0b5220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.304+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14039040 con 0x7ffb1c14e820 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.304+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14043070 con 0x7ffb1c14e820 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.304+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c0c9fd0 con 0x7ffb1c14e820 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.304+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb140019f0 con 0x7ffb1c14e820 2024-03-06T07:29:52.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.304+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c0b0d90 con 0x7ffb1c14e820 2024-03-06T07:29:52.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.305+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c04f330 con 0x7ffb1c14e820 2024-03-06T07:29:52.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.306+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb14034650 con 0x7ffb1c14e820 2024-03-06T07:29:52.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.307+0000 7ffb18c00640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffafc06b500 0x7ffafc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.307+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffb140abfe0 con 0x7ffb1c14e820 2024-03-06T07:29:52.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.307+0000 7ffb20c00640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffafc06b500 0x7ffafc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.308+0000 7ffb20c00640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffafc06b500 0x7ffafc06d9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffb10005460 tx=0x7ffb10009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.312+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb1407ad90 con 0x7ffb1c14e820 2024-03-06T07:29:52.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.513+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb1c15f720 con 0x7ffb1c14e820 2024-03-06T07:29:52.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.514+0000 7ffb18c00640 1 -- 172.21.15.32:0/1670102299 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb1407a690 con 0x7ffb1c14e820 2024-03-06T07:29:52.516 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.516+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffafc06b500 msgr2=0x7ffafc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffafc06b500 0x7ffafc06d9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffb10005460 tx=0x7ffb10009040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 msgr2=0x7ffb1c0cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 0x7ffb1c0cb010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffb1402cbf0 tx=0x7ffb1c0b5220 comp rx=0 tx=0).stop 2024-03-06T07:29:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 shutdown_connections 2024-03-06T07:29:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffafc06b500 0x7ffafc06d9c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb1c14e820 0x7ffb1c0cb010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb1c09d320 0x7ffb1c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 --2- 172.21.15.32:0/1670102299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb040051c0 0x7ffb1c0c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 >> 172.21.15.32:0/1670102299 conn(0x7ffb1c06b3d0 msgr2=0x7ffb1c0ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 shutdown_connections 2024-03-06T07:29:52.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.517+0000 7ffb23600640 1 -- 172.21.15.32:0/1670102299 wait complete. 2024-03-06T07:29:52.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:52.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.768+0000 7f21f6e00640 1 Processor -- start 2024-03-06T07:29:52.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.768+0000 7f21f6e00640 1 -- start start 2024-03-06T07:29:52.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.769+0000 7f21f6e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f21f00cf8a0 0x7f21f0160530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.769+0000 7f21f6e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf340 0x7f21f00cc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21f6e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f00cfda0 con 0x7f21f015fa40 2024-03-06T07:29:52.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21f6e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f015d3b0 con 0x7f21f00cf8a0 2024-03-06T07:29:52.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21f6e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00ce860 con 0x7f21f00cf340 2024-03-06T07:29:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21ef400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f21f00cf8a0 0x7f21f0160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21ef400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f21f00cf8a0 0x7f21f0160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:44564/0 (socket says 172.21.15.32:44564) 2024-03-06T07:29:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21ef400640 1 -- 172.21.15.32:0/3916249806 learned_addr learned my addr 172.21.15.32:0/3916249806 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.770+0000 7f21f4e00640 1 --2- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf340 0x7f21f00cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.771+0000 7f21ef400640 1 -- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf340 msgr2=0x7f21f00cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.771+0000 7f21ef400640 1 --2- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf340 0x7f21f00cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.771+0000 7f21ef400640 1 -- 172.21.15.32:0/3916249806 >> v1:172.21.15.39:6789/0 conn(0x7f21f015fa40 legacy=0x7f21f015fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.771+0000 7f21ef400640 1 -- 172.21.15.32:0/3916249806 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f00ceb40 con 0x7f21f00cf8a0 2024-03-06T07:29:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.772+0000 7f21ef400640 1 --2- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f00cf8a0 0x7f21f0160530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009870 tx=0x7f21d802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a06d584c5700b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21eea00640 1 -- 172.21.15.32:0/3916249806 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8034070 con 0x7f21f00cf8a0 2024-03-06T07:29:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21eea00640 1 -- 172.21.15.32:0/3916249806 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d8030840 con 0x7f21f00cf8a0 2024-03-06T07:29:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21eea00640 1 -- 172.21.15.32:0/3916249806 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8030c10 con 0x7f21f00cf8a0 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21f6e00640 1 -- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f00cf8a0 msgr2=0x7f21f0160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21f6e00640 1 --2- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f00cf8a0 0x7f21f0160530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009870 tx=0x7f21d802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21f6e00640 1 -- 172.21.15.32:0/3916249806 shutdown_connections 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21f6e00640 1 --2- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf340 0x7f21f00cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21f6e00640 1 --2- 172.21.15.32:0/3916249806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f00cf8a0 0x7f21f0160530 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.773+0000 7f21f6e00640 1 -- 172.21.15.32:0/3916249806 >> 172.21.15.32:0/3916249806 conn(0x7f21f00b3820 msgr2=0x7f21f00b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.774+0000 7f21f6e00640 1 -- 172.21.15.32:0/3916249806 shutdown_connections 2024-03-06T07:29:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.774+0000 7f21f6e00640 1 -- 172.21.15.32:0/3916249806 wait complete. 2024-03-06T07:29:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.775+0000 7f21f6e00640 1 Processor -- start 2024-03-06T07:29:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.775+0000 7f21f6e00640 1 -- start start 2024-03-06T07:29:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.776+0000 7f21f6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 0x7f21f01590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.776+0000 7f21f6e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf8a0 0x7f21f00afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.776+0000 7f21f6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f015fa40 0x7f21f00b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.776+0000 7f21f6e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00ccd90 con 0x7f21f00cf340 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.776+0000 7f21f6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00b4570 con 0x7f21f015fa40 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.776+0000 7f21f6e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00ccf10 con 0x7f21f00cf8a0 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 0x7f21f01590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 0x7f21f01590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35872/0 (socket says 172.21.15.32:35872) 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 -- 172.21.15.32:0/840959986 learned_addr learned my addr 172.21.15.32:0/840959986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21ef400640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf8a0 0x7f21f00afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21f4e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f015fa40 0x7f21f00b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 -- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf8a0 msgr2=0x7f21f00afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf8a0 0x7f21f00afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 -- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f015fa40 msgr2=0x7f21f00b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f015fa40 0x7f21f00b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.777+0000 7f21efe00640 1 -- 172.21.15.32:0/840959986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f006c310 con 0x7f21f00cf340 2024-03-06T07:29:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.778+0000 7f21efe00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 0x7f21f01590b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f21e000c730 tx=0x7f21e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.778+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0016020 con 0x7f21f00cf340 2024-03-06T07:29:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.778+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006d50 con 0x7f21f00cf340 2024-03-06T07:29:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.778+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f006c5f0 con 0x7f21f00cf340 2024-03-06T07:29:52.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.778+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003bb0 con 0x7f21f00cf340 2024-03-06T07:29:52.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.778+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f006cbb0 con 0x7f21f00cf340 2024-03-06T07:29:52.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.779+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f004f330 con 0x7f21f00cf340 2024-03-06T07:29:52.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.780+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21e000a850 con 0x7f21f00cf340 2024-03-06T07:29:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.782+0000 7f21e7e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21cc06b4d0 0x7f21cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.782+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f21e0089f40 con 0x7f21f00cf340 2024-03-06T07:29:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.782+0000 7f21ef400640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21cc06b4d0 0x7f21cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.783+0000 7f21ef400640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21cc06b4d0 0x7f21cc06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f21d8036000 tx=0x7f21d803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.786+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21e0020020 con 0x7f21f00cf340 2024-03-06T07:29:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.985+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21f0070bc0 con 0x7f21f00cf340 2024-03-06T07:29:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.986+0000 7f21e7e00640 1 -- 172.21.15.32:0/840959986 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f21e0019310 con 0x7f21f00cf340 2024-03-06T07:29:52.988 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.988+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21cc06b4d0 msgr2=0x7f21cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.988+0000 7f21f6e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21cc06b4d0 0x7f21cc06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f21d8036000 tx=0x7f21d803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.988+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 msgr2=0x7f21f01590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.988+0000 7f21f6e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 0x7f21f01590b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f21e000c730 tx=0x7f21e000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.988+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 shutdown_connections 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21cc06b4d0 0x7f21cc06d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21f015fa40 0x7f21f00b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21f00cf8a0 0x7f21f00afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 --2- 172.21.15.32:0/840959986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21f00cf340 0x7f21f01590b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 >> 172.21.15.32:0/840959986 conn(0x7f21f00b3820 msgr2=0x7f21f015ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 shutdown_connections 2024-03-06T07:29:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:52.989+0000 7f21f6e00640 1 -- 172.21.15.32:0/840959986 wait complete. 2024-03-06T07:29:53.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:53.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.231+0000 7fa592800640 1 Processor -- start 2024-03-06T07:29:53.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.231+0000 7fa592800640 1 -- start start 2024-03-06T07:29:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa592800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 0x7fa58c0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa592800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c072df0 con 0x7fa58c071e20 2024-03-06T07:29:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa592800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c0cdab0 con 0x7fa58c15bf20 2024-03-06T07:29:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa592800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0b4d70 con 0x7fa58c072520 2024-03-06T07:29:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa58be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa58c071e20 0x7fa58c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34176/0 (socket says 172.21.15.32:34176) 2024-03-06T07:29:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa58be00640 1 -- 172.21.15.32:0/3541472587 learned_addr learned my addr 172.21.15.32:0/3541472587 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.233+0000 7fa583e00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 0x7fa58c0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661794954 0 0) 0x7fa58c072df0 con 0x7fa58c071e20 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570003520 con 0x7fa58c071e20 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243672011 0 0) 0x7fa570003520 con 0x7fa58c071e20 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 msgr2=0x7fa58c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 0x7fa58c0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 >> v1:172.21.15.39:6789/0 conn(0x7fa58c15bf20 legacy=0x7fa58c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0b5050 con 0x7fa58c071e20 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa5840019f0 con 0x7fa58c071e20 2024-03-06T07:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 >> v1:172.21.15.32:6789/0 conn(0x7fa58c071e20 legacy=0x7fa58c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa570003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 0x7fa570004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa570004d40 0x7fa5700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0b4d70 con 0x7fa5700039e0 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57000baa0 con 0x7fa570004d40 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57000bca0 con 0x7fa58c072520 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.234+0000 7fa58b400640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa570003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58be00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa570004d40 0x7fa5700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58b400640 1 -- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 msgr2=0x7fa570004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58b400640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c072520 0x7fa570004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58b400640 1 -- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa570004d40 msgr2=0x7fa5700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58b400640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa570004d40 0x7fa5700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58b400640 1 -- 172.21.15.32:0/3541472587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570003520 con 0x7fa5700039e0 2024-03-06T07:29:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.235+0000 7fa58be00640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa570004d40 0x7fa5700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.236+0000 7fa58b400640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa570003e00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa57c001720 tx=0x7fa570005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a164d3433f5f427 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.236+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c001370 con 0x7fa5700039e0 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.236+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c0052d0 con 0x7fa5700039e0 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.236+0000 7fa58aa00640 1 -- 172.21.15.32:0/3541472587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c003830 con 0x7fa5700039e0 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa592800640 1 -- 172.21.15.32:0/3541472587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 msgr2=0x7fa570003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa592800640 1 --2- 172.21.15.32:0/3541472587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa570003e00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa57c001720 tx=0x7fa570005830 comp rx=0 tx=0).stop 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa58be00640 1 -- 172.21.15.32:0/3541472587 reap_dead start 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa592800640 1 -- 172.21.15.32:0/3541472587 shutdown_connections 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa592800640 1 -- 172.21.15.32:0/3541472587 >> 172.21.15.32:0/3541472587 conn(0x7fa58c06b3d0 msgr2=0x7fa58c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa592800640 1 -- 172.21.15.32:0/3541472587 shutdown_connections 2024-03-06T07:29:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.237+0000 7fa592800640 1 -- 172.21.15.32:0/3541472587 wait complete. 2024-03-06T07:29:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.239+0000 7fa592800640 1 Processor -- start 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.239+0000 7fa592800640 1 -- start start 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa592800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa58c0b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa592800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c15d520 0x7fa58c0b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa592800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa58c09bcd0 0x7fa58c1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa592800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0ad450 con 0x7fa5700039e0 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa592800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0b4990 con 0x7fa58c09bcd0 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa592800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0b1200 con 0x7fa58c15d520 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa58b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa58c0b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa58b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa58c0b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35884/0 (socket says 172.21.15.32:35884) 2024-03-06T07:29:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.240+0000 7fa58b400640 1 -- 172.21.15.32:0/1135036874 learned_addr learned my addr 172.21.15.32:0/1135036874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58be00640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa58c09bcd0 0x7fa58c1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa583e00640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c15d520 0x7fa58c0b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58b400640 1 -- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c15d520 msgr2=0x7fa58c0b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58b400640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c15d520 0x7fa58c0b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58b400640 1 -- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa58c09bcd0 msgr2=0x7fa58c1440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58b400640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa58c09bcd0 0x7fa58c1440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58b400640 1 -- 172.21.15.32:0/1135036874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0b14e0 con 0x7fa5700039e0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.241+0000 7fa58b400640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa58c0b22a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa57c008fd0 tx=0x7fa57c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.242+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c041070 con 0x7fa5700039e0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.242+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c005480 con 0x7fa5700039e0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.242+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c007a30 con 0x7fa5700039e0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.242+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c0b17c0 con 0x7fa5700039e0 2024-03-06T07:29:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.242+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c0b1d80 con 0x7fa5700039e0 2024-03-06T07:29:53.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.243+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c067830 con 0x7fa5700039e0 2024-03-06T07:29:53.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.244+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa57c044070 con 0x7fa5700039e0 2024-03-06T07:29:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.246+0000 7fa583400640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa56806b4d0 0x7fa56806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.246+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa57c0abf30 con 0x7fa5700039e0 2024-03-06T07:29:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.246+0000 7fa583e00640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa56806b4d0 0x7fa56806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.247+0000 7fa583e00640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa56806b4d0 0x7fa56806d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0cbe80 tx=0x7fa574008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.250+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa58c067830 con 0x7fa5700039e0 2024-03-06T07:29:53.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.456+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa58c04f330 con 0x7fa5700039e0 2024-03-06T07:29:53.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.457+0000 7fa583400640 1 -- 172.21.15.32:0/1135036874 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa57c07adb0 con 0x7fa5700039e0 2024-03-06T07:29:53.459 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:53.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.459+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa56806b4d0 msgr2=0x7fa56806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.459+0000 7fa592800640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa56806b4d0 0x7fa56806d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0cbe80 tx=0x7fa574008040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.459+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 msgr2=0x7fa58c0b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.459+0000 7fa592800640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa58c0b22a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa57c008fd0 tx=0x7fa57c001120 comp rx=0 tx=0).stop 2024-03-06T07:29:53.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 shutdown_connections 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa56806b4d0 0x7fa56806d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa58c09bcd0 0x7fa58c1440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa58c15d520 0x7fa58c0b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 --2- 172.21.15.32:0/1135036874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5700039e0 0x7fa58c0b22a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 >> 172.21.15.32:0/1135036874 conn(0x7fa58c06b3d0 msgr2=0x7fa58c0ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 shutdown_connections 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.460+0000 7fa592800640 1 -- 172.21.15.32:0/1135036874 wait complete. 2024-03-06T07:29:53.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:53.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.699+0000 7f66be400640 1 Processor -- start 2024-03-06T07:29:53.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.699+0000 7f66be400640 1 -- start start 2024-03-06T07:29:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.700+0000 7f66be400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f66b80b34e0 0x7f66b815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.701+0000 7f66be400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b815cc40 con 0x7f66b80af580 2024-03-06T07:29:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.701+0000 7f66be400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b815cf20 con 0x7f66b80af1a0 2024-03-06T07:29:53.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.701+0000 7f66be400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b815d200 con 0x7f66b80b34e0 2024-03-06T07:29:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.701+0000 7f66b6a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f66b80af580 0x7f66b80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34180/0 (socket says 172.21.15.32:34180) 2024-03-06T07:29:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.701+0000 7f66b6a00640 1 -- 172.21.15.32:0/180021012 learned_addr learned my addr 172.21.15.32:0/180021012 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.701+0000 7f66b7e00640 1 --2- 172.21.15.32:0/180021012 >> v2:172.21.15.39:3300/0 conn(0x7f66b80b34e0 0x7f66b815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334064039 0 0) 0x7f66b815cc40 con 0x7f66b80af580 2024-03-06T07:29:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f669c003520 con 0x7f66b80af580 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040205174 0 0) 0x7f669c003520 con 0x7f66b80af580 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 >> v2:172.21.15.39:3300/0 conn(0x7f66b80b34e0 msgr2=0x7f66b815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 --2- 172.21.15.32:0/180021012 >> v2:172.21.15.39:3300/0 conn(0x7f66b80b34e0 0x7f66b815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 >> v1:172.21.15.39:6789/0 conn(0x7f66b80af1a0 legacy=0x7f66b80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b815d4e0 con 0x7f66b80af580 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f66a4001670 con 0x7f66b80af580 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 >> v1:172.21.15.32:6789/0 conn(0x7f66b80af580 legacy=0x7f66b80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f669c003a70 0x7f669c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f669c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669c0052e0 0x7f669c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815d200 con 0x7f669c004800 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c00c1b0 con 0x7f669c0052e0 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.702+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669c00c3b0 con 0x7f669c003a70 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b6a00640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f669c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b7e00640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669c0052e0 0x7f669c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b7400640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f669c003a70 0x7f669c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b6a00640 1 -- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f669c003a70 msgr2=0x7f669c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b6a00640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f669c003a70 0x7f669c003e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b6a00640 1 -- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669c0052e0 msgr2=0x7f669c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b6a00640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669c0052e0 0x7f669c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b6a00640 1 -- 172.21.15.32:0/180021012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c003520 con 0x7f669c004800 2024-03-06T07:29:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.703+0000 7f66b7e00640 1 -- 172.21.15.32:0/180021012 reap_dead start 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.704+0000 7f66b6a00640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f669c004c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c160 tx=0x7f669c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0cc7c2a5ecf2fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.704+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4007c60 con 0x7f669c004800 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.704+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a4030070 con 0x7f669c004800 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.704+0000 7f66b6000640 1 -- 172.21.15.32:0/180021012 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4005d90 con 0x7f669c004800 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 -- 172.21.15.32:0/180021012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 msgr2=0x7f669c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f669c004c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c160 tx=0x7f669c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 -- 172.21.15.32:0/180021012 shutdown_connections 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 --2- 172.21.15.32:0/180021012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f669c004c80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 -- 172.21.15.32:0/180021012 >> 172.21.15.32:0/180021012 conn(0x7f66b806afb0 msgr2=0x7f66b806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 -- 172.21.15.32:0/180021012 shutdown_connections 2024-03-06T07:29:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.705+0000 7f66be400640 1 -- 172.21.15.32:0/180021012 wait complete. 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.707+0000 7f66be400640 1 Processor -- start 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.707+0000 7f66be400640 1 -- start start 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66be400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f66b809d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66be400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f66b814a500 0x7f66b80cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66be400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b80cccd0 0x7f66b806de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66be400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815c600 con 0x7f669c004800 2024-03-06T07:29:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66be400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80cde30 con 0x7f66b80cccd0 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66be400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815c780 con 0x7f66b814a500 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.708+0000 7f66b7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f66b809d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f66b809d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35904/0 (socket says 172.21.15.32:35904) 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 -- 172.21.15.32:0/420179739 learned_addr learned my addr 172.21.15.32:0/420179739 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b6a00640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f66b814a500 0x7f66b80cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7e00640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b80cccd0 0x7f66b806de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 -- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f66b814a500 msgr2=0x7f66b80cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f66b814a500 0x7f66b80cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 -- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b80cccd0 msgr2=0x7f66b806de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b80cccd0 0x7f66b806de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 -- 172.21.15.32:0/420179739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b809bff0 con 0x7f669c004800 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7e00640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b80cccd0 0x7f66b806de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.709+0000 7f66b7400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f66b809d050 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f66a800d910 tx=0x7f66a800ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.710+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8003da0 con 0x7f669c004800 2024-03-06T07:29:53.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.710+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a8017070 con 0x7f669c004800 2024-03-06T07:29:53.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.710+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a800b970 con 0x7f669c004800 2024-03-06T07:29:53.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.710+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b809c290 con 0x7f669c004800 2024-03-06T07:29:53.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.710+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b809c850 con 0x7f669c004800 2024-03-06T07:29:53.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.711+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b804f3a0 con 0x7f669c004800 2024-03-06T07:29:53.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.712+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66a800bb10 con 0x7f669c004800 2024-03-06T07:29:53.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.713+0000 7f66af400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f669806b4d0 0x7f669806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.713+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f66a808a380 con 0x7f669c004800 2024-03-06T07:29:53.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.713+0000 7f66b6a00640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f669806b4d0 0x7f669806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.714+0000 7f66b6a00640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f669806b4d0 0x7f669806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f66b80afa10 tx=0x7f66a4039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.719+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66a8059210 con 0x7f669c004800 2024-03-06T07:29:53.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.932+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f66b815f490 con 0x7f669c004800 2024-03-06T07:29:53.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.933+0000 7f66af400640 1 -- 172.21.15.32:0/420179739 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f66a8058b10 con 0x7f669c004800 2024-03-06T07:29:53.935 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:29:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f669806b4d0 msgr2=0x7f669806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f669806b4d0 0x7f669806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f66b80afa10 tx=0x7f66a4039040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 msgr2=0x7f66b809d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f66b809d050 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f66a800d910 tx=0x7f66a800ddd0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 shutdown_connections 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f669806b4d0 0x7f669806d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b80cccd0 0x7f66b806de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f66b814a500 0x7f66b80cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 --2- 172.21.15.32:0/420179739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669c004800 0x7f66b809d050 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 >> 172.21.15.32:0/420179739 conn(0x7f66b806afb0 msgr2=0x7f66b80b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 shutdown_connections 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:53.936+0000 7f66be400640 1 -- 172.21.15.32:0/420179739 wait complete. 2024-03-06T07:29:53.956 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-06T07:29:53.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:53.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-06T07:29:54.064 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-03-06T07:29:54.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:54.064 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:29:54.114 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:29:54.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.336+0000 7f1923e00640 1 Processor -- start 2024-03-06T07:29:54.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.336+0000 7f1923e00640 1 -- start start 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19240b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19240cd060 0x7f192415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19240714d0 con 0x7f1924070f10 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19240717b0 con 0x7f19240cd060 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1924071a90 con 0x7f1924145300 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19240cd060 0x7f192415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19240cd060 0x7f192415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35908/0 (socket says 172.21.15.32:35908) 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.338+0000 7f1923400640 1 -- 172.21.15.32:0/10531806 learned_addr learned my addr 172.21.15.32:0/10531806 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.339+0000 7f1922a00640 1 --2- 172.21.15.32:0/10531806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19240b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.339+0000 7f1923400640 1 -- 172.21.15.32:0/10531806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 msgr2=0x7f19240b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.339+0000 7f1923400640 1 --2- 172.21.15.32:0/10531806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19240b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.339+0000 7f1923400640 1 -- 172.21.15.32:0/10531806 >> v1:172.21.15.39:6789/0 conn(0x7f1924070f10 legacy=0x7f19240cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.339+0000 7f1923400640 1 -- 172.21.15.32:0/10531806 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924071d70 con 0x7f19240cd060 2024-03-06T07:29:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.340+0000 7f1923400640 1 --2- 172.21.15.32:0/10531806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f192415c570 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f191c009870 tx=0x7f191c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b00fe91d090b6be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.340+0000 7f1921600640 1 -- 172.21.15.32:0/10531806 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c034070 con 0x7f19240cd060 2024-03-06T07:29:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.340+0000 7f1921600640 1 -- 172.21.15.32:0/10531806 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c030840 con 0x7f19240cd060 2024-03-06T07:29:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.340+0000 7f1921600640 1 -- 172.21.15.32:0/10531806 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c030c10 con 0x7f19240cd060 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 -- 172.21.15.32:0/10531806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 msgr2=0x7f192415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 --2- 172.21.15.32:0/10531806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f192415c570 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f191c009870 tx=0x7f191c02b980 comp rx=0 tx=0).stop 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 -- 172.21.15.32:0/10531806 shutdown_connections 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 --2- 172.21.15.32:0/10531806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f192415c570 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 --2- 172.21.15.32:0/10531806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19240b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 -- 172.21.15.32:0/10531806 >> 172.21.15.32:0/10531806 conn(0x7f192406b3d0 msgr2=0x7f192406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 -- 172.21.15.32:0/10531806 shutdown_connections 2024-03-06T07:29:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.341+0000 7f1923e00640 1 -- 172.21.15.32:0/10531806 wait complete. 2024-03-06T07:29:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.343+0000 7f1923e00640 1 Processor -- start 2024-03-06T07:29:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.343+0000 7f1923e00640 1 -- start start 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1923e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1924070f10 0x7f192415b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1923e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f19241e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1923e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19241e6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1923e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192415cab0 con 0x7f19240cd060 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1923e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19241456e0 con 0x7f1924070f10 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1923e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192415cc30 con 0x7f1924145300 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1922000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f19241e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1922000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f19241e4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35910/0 (socket says 172.21.15.32:35910) 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.344+0000 7f1922000640 1 -- 172.21.15.32:0/2409576377 learned_addr learned my addr 172.21.15.32:0/2409576377 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922a00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1924070f10 0x7f192415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1923400640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19241e6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922000640 1 -- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 msgr2=0x7f19241e6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922000640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19241e6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922000640 1 -- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1924070f10 msgr2=0x7f192415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922000640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1924070f10 0x7f192415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922000640 1 -- 172.21.15.32:0/2409576377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19241e5530 con 0x7f19240cd060 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1923400640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19241e6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922a00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1924070f10 0x7f192415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1922000640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f19241e4e30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f191000c7c0 tx=0x7f191000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910016020 con 0x7f19240cd060 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910006d50 con 0x7f19240cd060 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910003bb0 con 0x7f19240cd060 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.345+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19241e5810 con 0x7f19240cd060 2024-03-06T07:29:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.346+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19241e77e0 con 0x7f19240cd060 2024-03-06T07:29:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.347+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19241e5d30 con 0x7f19240cd060 2024-03-06T07:29:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.348+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f191000a850 con 0x7f19240cd060 2024-03-06T07:29:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.348+0000 7f191b400640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f190406b4d0 0x7f190406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.349+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1910089fd0 con 0x7f19240cd060 2024-03-06T07:29:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.349+0000 7f1922a00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f190406b4d0 0x7f190406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.350+0000 7f1922a00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f190406b4d0 0x7f190406d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1914006fd0 tx=0x7f1914008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.353+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19241e5d30 con 0x7f19240cd060 2024-03-06T07:29:54.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.595+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f192404f330 con 0x7f19240cd060 2024-03-06T07:29:54.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.596+0000 7f191b400640 1 -- 172.21.15.32:0/2409576377 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f1910020020 con 0x7f19240cd060 2024-03-06T07:29:54.598 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:54.599 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:54.599 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:54.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f190406b4d0 msgr2=0x7f190406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f190406b4d0 0x7f190406d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1914006fd0 tx=0x7f1914008040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 msgr2=0x7f19241e4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f19241e4e30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f191000c7c0 tx=0x7f191000cc80 comp rx=0 tx=0).stop 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 shutdown_connections 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f190406b4d0 0x7f190406d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1924145300 0x7f19241e6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19240cd060 0x7f19241e4e30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 --2- 172.21.15.32:0/2409576377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1924070f10 0x7f192415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 >> 172.21.15.32:0/2409576377 conn(0x7f192406b3d0 msgr2=0x7f19240af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 shutdown_connections 2024-03-06T07:29:54.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.599+0000 7f1923e00640 1 -- 172.21.15.32:0/2409576377 wait complete. 2024-03-06T07:29:54.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.842+0000 7f2246800640 1 Processor -- start 2024-03-06T07:29:54.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.842+0000 7f2246800640 1 -- start start 2024-03-06T07:29:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.843+0000 7f2246800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2240070f10 0x7f22400cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.843+0000 7f2246800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f22400cce90 0x7f224015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.843+0000 7f2246800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400714d0 con 0x7f2240145300 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.843+0000 7f2246800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22400cd300 con 0x7f2240070f10 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.843+0000 7f2246800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22400717b0 con 0x7f22400cce90 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2245400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2240145300 0x7f22400b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34192/0 (socket says 172.21.15.32:34192) 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2245400640 1 -- 172.21.15.32:0/1061115963 learned_addr learned my addr 172.21.15.32:0/1061115963 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2244a00640 1 --2- 172.21.15.32:0/1061115963 >> v2:172.21.15.32:3300/0 conn(0x7f2240070f10 0x7f22400cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2244a00640 1 -- 172.21.15.32:0/1061115963 >> v2:172.21.15.39:3300/0 conn(0x7f22400cce90 msgr2=0x7f224015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2245e00640 1 --2- 172.21.15.32:0/1061115963 >> v2:172.21.15.39:3300/0 conn(0x7f22400cce90 0x7f224015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2244a00640 1 --2- 172.21.15.32:0/1061115963 >> v2:172.21.15.39:3300/0 conn(0x7f22400cce90 0x7f224015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2244a00640 1 -- 172.21.15.32:0/1061115963 >> v1:172.21.15.32:6789/0 conn(0x7f2240145300 legacy=0x7f22400b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2244a00640 1 -- 172.21.15.32:0/1061115963 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240071a90 con 0x7f2240070f10 2024-03-06T07:29:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f2245e00640 1 --2- 172.21.15.32:0/1061115963 >> v2:172.21.15.39:3300/0 conn(0x7f22400cce90 0x7f224015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.844+0000 7f223fe00640 1 -- 172.21.15.32:0/1061115963 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304670242 0 0) 0x7f22400714d0 con 0x7f2240145300 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.846+0000 7f2244a00640 1 --2- 172.21.15.32:0/1061115963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f22400cc950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f22400cc500 tx=0x7f222c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f126b4f685de81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.846+0000 7f223fe00640 1 -- 172.21.15.32:0/1061115963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0315a0 con 0x7f2240070f10 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.846+0000 7f223fe00640 1 -- 172.21.15.32:0/1061115963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c037070 con 0x7f2240070f10 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.846+0000 7f223fe00640 1 -- 172.21.15.32:0/1061115963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c031960 con 0x7f2240070f10 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 -- 172.21.15.32:0/1061115963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 msgr2=0x7f22400cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 --2- 172.21.15.32:0/1061115963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f22400cc950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f22400cc500 tx=0x7f222c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 -- 172.21.15.32:0/1061115963 shutdown_connections 2024-03-06T07:29:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 --2- 172.21.15.32:0/1061115963 >> v2:172.21.15.39:3300/0 conn(0x7f22400cce90 0x7f224015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 --2- 172.21.15.32:0/1061115963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f22400cc950 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 -- 172.21.15.32:0/1061115963 >> 172.21.15.32:0/1061115963 conn(0x7f224006b3d0 msgr2=0x7f224006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 -- 172.21.15.32:0/1061115963 shutdown_connections 2024-03-06T07:29:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.847+0000 7f2246800640 1 -- 172.21.15.32:0/1061115963 wait complete. 2024-03-06T07:29:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.849+0000 7f2246800640 1 Processor -- start 2024-03-06T07:29:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.849+0000 7f2246800640 1 -- start start 2024-03-06T07:29:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.850+0000 7f2246800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f224006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.850+0000 7f2246800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22400cce90 0x7f22400cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.850+0000 7f2246800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2240145300 0x7f22400c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2246800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224015cab0 con 0x7f2240070f10 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2246800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22401456e0 con 0x7f22400cce90 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2246800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224015cc30 con 0x7f2240145300 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f224006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f224006dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35926/0 (socket says 172.21.15.32:35926) 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 -- 172.21.15.32:0/1235395864 learned_addr learned my addr 172.21.15.32:0/1235395864 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2244a00640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22400cce90 0x7f22400cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 -- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2240145300 msgr2=0x7f22400c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2240145300 0x7f22400c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 -- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22400cce90 msgr2=0x7f22400cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22400cce90 0x7f22400cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.851+0000 7f2245400640 1 -- 172.21.15.32:0/1235395864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224006cc30 con 0x7f2240070f10 2024-03-06T07:29:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.852+0000 7f2245400640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f224006dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f223400c460 tx=0x7f223400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.852+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234018070 con 0x7f2240070f10 2024-03-06T07:29:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.852+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2234007500 con 0x7f2240070f10 2024-03-06T07:29:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.852+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234004a20 con 0x7f2240070f10 2024-03-06T07:29:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.852+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224006cf10 con 0x7f2240070f10 2024-03-06T07:29:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.852+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224006d4d0 con 0x7f2240070f10 2024-03-06T07:29:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.853+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224004f330 con 0x7f2240070f10 2024-03-06T07:29:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.854+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f223401b070 con 0x7f2240070f10 2024-03-06T07:29:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.856+0000 7f223d600640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f222006b4d0 0x7f222006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.856+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f22340895f0 con 0x7f2240070f10 2024-03-06T07:29:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.856+0000 7f2244a00640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f222006b4d0 0x7f222006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.857+0000 7f2244a00640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f222006b4d0 0x7f222006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f222c036000 tx=0x7f222c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:54.860+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2234058350 con 0x7f2240070f10 2024-03-06T07:29:55.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.062+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2240144810 con 0x7f2240070f10 2024-03-06T07:29:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.064+0000 7f223d600640 1 -- 172.21.15.32:0/1235395864 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16746 (secure 0 0 0) 0x7f2234087e20 con 0x7f2240070f10 2024-03-06T07:29:55.067 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:55.067 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:47.656917+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f222006b4d0 msgr2=0x7f222006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f222006b4d0 0x7f222006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f222c036000 tx=0x7f222c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 msgr2=0x7f224006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f224006dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f223400c460 tx=0x7f223400c920 comp rx=0 tx=0).stop 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 shutdown_connections 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f222006b4d0 0x7f222006d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2240145300 0x7f22400c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22400cce90 0x7f22400cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 --2- 172.21.15.32:0/1235395864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2240070f10 0x7f224006dcd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.067+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 >> 172.21.15.32:0/1235395864 conn(0x7f224006b3d0 msgr2=0x7f22400af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.068+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 shutdown_connections 2024-03-06T07:29:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.068+0000 7f2246800640 1 -- 172.21.15.32:0/1235395864 wait complete. 2024-03-06T07:29:55.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:58.495536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:59.791725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:55.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:55.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.315+0000 7f9ca3000640 1 Processor -- start 2024-03-06T07:29:55.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.315+0000 7f9ca3000640 1 -- start start 2024-03-06T07:29:55.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.316+0000 7f9ca3000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c9c0d4860 0x7f9c9c0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.316+0000 7f9ca3000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9c9c0cfa60 0x7f9c9c0d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.316+0000 7f9ca3000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0ceca0 con 0x7f9c9c15dee0 2024-03-06T07:29:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.316+0000 7f9ca3000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c0cf5d0 con 0x7f9c9c0d4860 2024-03-06T07:29:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.316+0000 7f9ca3000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c0cef80 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c9c0d4860 0x7f9c9c0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c9c0d4860 0x7f9c9c0d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35934/0 (socket says 172.21.15.32:35934) 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 -- 172.21.15.32:0/2311713016 learned_addr learned my addr 172.21.15.32:0/2311713016 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1c00640 1 --1- 172.21.15.32:0/2311713016 >> v1:172.21.15.32:6789/0 conn(0x7f9c9c15dee0 0x7f9c9c0d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34198/0 (socket says 172.21.15.32:34198) 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca2600640 1 --2- 172.21.15.32:0/2311713016 >> v2:172.21.15.39:3300/0 conn(0x7f9c9c0cfa60 0x7f9c9c0d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 -- 172.21.15.32:0/2311713016 >> v2:172.21.15.39:3300/0 conn(0x7f9c9c0cfa60 msgr2=0x7f9c9c0d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 --2- 172.21.15.32:0/2311713016 >> v2:172.21.15.39:3300/0 conn(0x7f9c9c0cfa60 0x7f9c9c0d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 -- 172.21.15.32:0/2311713016 >> v1:172.21.15.32:6789/0 conn(0x7f9c9c15dee0 legacy=0x7f9c9c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.317+0000 7f9ca1200640 1 -- 172.21.15.32:0/2311713016 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0cf260 con 0x7f9c9c0d4860 2024-03-06T07:29:55.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.318+0000 7f9ca1200640 1 --2- 172.21.15.32:0/2311713016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c0d0630 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c06b2d0 tx=0x7f9c8802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34497a3f36cc8dca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9c9be00640 1 -- 172.21.15.32:0/2311713016 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88034070 con 0x7f9c9c0d4860 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9c9be00640 1 -- 172.21.15.32:0/2311713016 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c88030860 con 0x7f9c9c0d4860 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9c9be00640 1 -- 172.21.15.32:0/2311713016 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88030c30 con 0x7f9c9c0d4860 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 -- 172.21.15.32:0/2311713016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0d4860 msgr2=0x7f9c9c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 --2- 172.21.15.32:0/2311713016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c0d0630 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c06b2d0 tx=0x7f9c8802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 -- 172.21.15.32:0/2311713016 shutdown_connections 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 --2- 172.21.15.32:0/2311713016 >> v2:172.21.15.39:3300/0 conn(0x7f9c9c0cfa60 0x7f9c9c0d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 --2- 172.21.15.32:0/2311713016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c0d0630 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 -- 172.21.15.32:0/2311713016 >> 172.21.15.32:0/2311713016 conn(0x7f9c9c0d3cd0 msgr2=0x7f9c9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.319+0000 7f9ca3000640 1 -- 172.21.15.32:0/2311713016 shutdown_connections 2024-03-06T07:29:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.320+0000 7f9ca3000640 1 -- 172.21.15.32:0/2311713016 wait complete. 2024-03-06T07:29:55.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.321+0000 7f9ca3000640 1 Processor -- start 2024-03-06T07:29:55.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.321+0000 7f9ca3000640 1 -- start start 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.322+0000 7f9ca3000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 0x7f9c9c15a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.322+0000 7f9ca3000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c1f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.322+0000 7f9ca3000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c9c15dee0 0x7f9c9c1f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.322+0000 7f9ca3000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0b3510 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.322+0000 7f9ca3000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c15e2c0 con 0x7f9c9c0d4860 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.322+0000 7f9ca3000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0b3690 con 0x7f9c9c15dee0 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 0x7f9c9c15a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 0x7f9c9c15a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35950/0 (socket says 172.21.15.32:35950) 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 -- 172.21.15.32:0/3209668530 learned_addr learned my addr 172.21.15.32:0/3209668530 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1200640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c1f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca2600640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c9c15dee0 0x7f9c9c1f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 -- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c9c15dee0 msgr2=0x7f9c9c1f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c9c15dee0 0x7f9c9c1f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 -- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c9c0d4860 msgr2=0x7f9c9c1f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c1f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 -- 172.21.15.32:0/3209668530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c1586d0 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.323+0000 7f9ca1c00640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 0x7f9c9c15a660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c760 tx=0x7f9c9000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.324+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90017070 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.324+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9000a640 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.324+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000aa10 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.324+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c0b9050 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.324+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c0b9610 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.325+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c04f330 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.326+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c9001a070 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.327+0000 7f9c99600640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c7c06b4d0 0x7f9c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.327+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9c90089860 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.328+0000 7f9ca1200640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c7c06b4d0 0x7f9c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.328+0000 7f9ca1200640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c7c06b4d0 0x7f9c7c06d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9c88036000 tx=0x7f9c8803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.332+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c90058650 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.529+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c9c08d7d0 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.530+0000 7f9c99600640 1 -- 172.21.15.32:0/3209668530 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c90057f50 con 0x7f9c9c0cfa60 2024-03-06T07:29:55.533 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:29:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c7c06b4d0 msgr2=0x7f9c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c7c06b4d0 0x7f9c7c06d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9c88036000 tx=0x7f9c8803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 msgr2=0x7f9c9c15a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 0x7f9c9c15a660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c760 tx=0x7f9c9000cc20 comp rx=0 tx=0).stop 2024-03-06T07:29:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 shutdown_connections 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c7c06b4d0 0x7f9c7c06d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9c9c15dee0 0x7f9c9c1f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c9c0d4860 0x7f9c9c1f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 --2- 172.21.15.32:0/3209668530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c9c0cfa60 0x7f9c9c15a660 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 >> 172.21.15.32:0/3209668530 conn(0x7f9c9c0d3cd0 msgr2=0x7f9c9c0d18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 shutdown_connections 2024-03-06T07:29:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.533+0000 7f9ca3000640 1 -- 172.21.15.32:0/3209668530 wait complete. 2024-03-06T07:29:55.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.774+0000 7f1b6f200640 1 Processor -- start 2024-03-06T07:29:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.774+0000 7f1b6f200640 1 -- start start 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.776+0000 7f1b6f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1b68072520 0x7f1b680cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.776+0000 7f1b6f200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b68072df0 con 0x7f1b68071e20 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.776+0000 7f1b6f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b680cdab0 con 0x7f1b6815bf20 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.776+0000 7f1b6f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b680b4d70 con 0x7f1b68072520 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.777+0000 7f1b6d200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1b68071e20 0x7f1b680ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34212/0 (socket says 172.21.15.32:34212) 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.777+0000 7f1b6d200640 1 -- 172.21.15.32:0/2507590773 learned_addr learned my addr 172.21.15.32:0/2507590773 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.777+0000 7f1b67400640 1 --2- 172.21.15.32:0/2507590773 >> v2:172.21.15.39:3300/0 conn(0x7f1b68072520 0x7f1b680cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.777+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199540758 0 0) 0x7f1b68072df0 con 0x7f1b68071e20 2024-03-06T07:29:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.777+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b4c003520 con 0x7f1b68071e20 2024-03-06T07:29:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.777+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3357422977 0 0) 0x7f1b4c003520 con 0x7f1b68071e20 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 >> v2:172.21.15.39:3300/0 conn(0x7f1b68072520 msgr2=0x7f1b680cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 --2- 172.21.15.32:0/2507590773 >> v2:172.21.15.39:3300/0 conn(0x7f1b68072520 0x7f1b680cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 >> v1:172.21.15.39:6789/0 conn(0x7f1b6815bf20 legacy=0x7f1b680acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b680b5050 con 0x7f1b68071e20 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f1b600019f0 con 0x7f1b68071e20 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 >> v1:172.21.15.32:6789/0 conn(0x7f1b68071e20 legacy=0x7f1b680ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b4c0039b0 0x7f1b4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c004800 0x7f1b4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c0051c0 0x7f1b4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680b4d70 con 0x7f1b4c004800 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c00c060 con 0x7f1b4c0039b0 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c00c260 con 0x7f1b4c0051c0 2024-03-06T07:29:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b67400640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c004800 0x7f1b4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b67e00640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b4c0039b0 0x7f1b4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b6d200640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c0051c0 0x7f1b4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b67400640 1 -- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c0051c0 msgr2=0x7f1b4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.778+0000 7f1b67400640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c0051c0 0x7f1b4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.779+0000 7f1b67400640 1 -- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b4c0039b0 msgr2=0x7f1b4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.779+0000 7f1b67400640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b4c0039b0 0x7f1b4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.779+0000 7f1b67400640 1 -- 172.21.15.32:0/2507590773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c003520 con 0x7f1b4c004800 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.779+0000 7f1b6d200640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c0051c0 0x7f1b4c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.779+0000 7f1b6d200640 1 -- 172.21.15.32:0/2507590773 reap_dead start 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.780+0000 7f1b67400640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c004800 0x7f1b4c004c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009ac0 tx=0x7f1b4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1452a6702b4affb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.780+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54039070 con 0x7f1b4c004800 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.780+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5402c6b0 con 0x7f1b4c004800 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.780+0000 7f1b66a00640 1 -- 172.21.15.32:0/2507590773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54009690 con 0x7f1b4c004800 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.780+0000 7f1b6f200640 1 -- 172.21.15.32:0/2507590773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c004800 msgr2=0x7f1b4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.780+0000 7f1b6f200640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c004800 0x7f1b4c004c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009ac0 tx=0x7f1b4c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.781+0000 7f1b6f200640 1 -- 172.21.15.32:0/2507590773 shutdown_connections 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.781+0000 7f1b6f200640 1 --2- 172.21.15.32:0/2507590773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b4c004800 0x7f1b4c004c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.781+0000 7f1b6f200640 1 -- 172.21.15.32:0/2507590773 >> 172.21.15.32:0/2507590773 conn(0x7f1b6806b3d0 msgr2=0x7f1b6806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.781+0000 7f1b6f200640 1 -- 172.21.15.32:0/2507590773 shutdown_connections 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.781+0000 7f1b6f200640 1 -- 172.21.15.32:0/2507590773 wait complete. 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.782+0000 7f1b6f200640 1 Processor -- start 2024-03-06T07:29:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.783+0000 7f1b6f200640 1 -- start start 2024-03-06T07:29:55.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c004800 0x7f1b681465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b68152f80 0x7f1b6815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 0x7f1b6815e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680b4b10 con 0x7f1b68153400 2024-03-06T07:29:55.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680b4990 con 0x7f1b68152f80 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680acd40 con 0x7f1b4c004800 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 0x7f1b6815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 0x7f1b6815e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35954/0 (socket says 172.21.15.32:35954) 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 -- 172.21.15.32:0/3600729159 learned_addr learned my addr 172.21.15.32:0/3600729159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b67400640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b68152f80 0x7f1b6815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b67e00640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c004800 0x7f1b681465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 -- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c004800 msgr2=0x7f1b681465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c004800 0x7f1b681465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 -- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b68152f80 msgr2=0x7f1b6815dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b68152f80 0x7f1b6815dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.784+0000 7f1b6d200640 1 -- 172.21.15.32:0/3600729159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68146b30 con 0x7f1b68153400 2024-03-06T07:29:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.785+0000 7f1b6d200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 0x7f1b6815e260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1b6806c030 tx=0x7f1b6000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.785+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60004d60 con 0x7f1b68153400 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.785+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6000bb00 con 0x7f1b68153400 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.785+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60007070 con 0x7f1b68153400 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.785+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b681539d0 con 0x7f1b68153400 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.785+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b68153f90 con 0x7f1b68153400 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.787+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b680ae0e0 con 0x7f1b68153400 2024-03-06T07:29:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.787+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b6001b020 con 0x7f1b68153400 2024-03-06T07:29:55.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.788+0000 7f1b5fe00640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b4806b500 0x7f1b4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.788+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1b6008a2f0 con 0x7f1b68153400 2024-03-06T07:29:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.788+0000 7f1b67e00640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b4806b500 0x7f1b4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.789+0000 7f1b67e00640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b4806b500 0x7f1b4806d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1b58007fd0 tx=0x7f1b58009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.793+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b680ae0e0 con 0x7f1b68153400 2024-03-06T07:29:55.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.993+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1b6804f330 con 0x7f1b68153400 2024-03-06T07:29:55.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.994+0000 7f1b5fe00640 1 -- 172.21.15.32:0/3600729159 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1b600590f0 con 0x7f1b68153400 2024-03-06T07:29:55.996 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.996+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b4806b500 msgr2=0x7f1b4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.996+0000 7f1b6f200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b4806b500 0x7f1b4806d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1b58007fd0 tx=0x7f1b58009040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.996+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 msgr2=0x7f1b6815e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.996+0000 7f1b6f200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 0x7f1b6815e260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1b6806c030 tx=0x7f1b6000b4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.996+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 shutdown_connections 2024-03-06T07:29:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b4806b500 0x7f1b4806d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b68153400 0x7f1b6815e260 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b68152f80 0x7f1b6815dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 --2- 172.21.15.32:0/3600729159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b4c004800 0x7f1b681465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 >> 172.21.15.32:0/3600729159 conn(0x7f1b6806b3d0 msgr2=0x7f1b680ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 shutdown_connections 2024-03-06T07:29:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:55.997+0000 7f1b6f200640 1 -- 172.21.15.32:0/3600729159 wait complete. 2024-03-06T07:29:56.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.238+0000 7f6284c00640 1 Processor -- start 2024-03-06T07:29:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.239+0000 7f6284c00640 1 -- start start 2024-03-06T07:29:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.240+0000 7f6284c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f628006da40 0x7f628006e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.240+0000 7f6284c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f628015e680 0x7f628006c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.240+0000 7f6284c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628015eb80 con 0x7f62800ccad0 2024-03-06T07:29:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.240+0000 7f6284c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280161c70 con 0x7f628015e680 2024-03-06T07:29:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.240+0000 7f6284c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628015ee60 con 0x7f628006da40 2024-03-06T07:29:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f628015e680 0x7f628006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f628015e680 0x7f628006c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35962/0 (socket says 172.21.15.32:35962) 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f627e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f62800ccad0 0x7f6280162100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34226/0 (socket says 172.21.15.32:34226) 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 -- 172.21.15.32:0/3349575338 learned_addr learned my addr 172.21.15.32:0/3349575338 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f627de00640 1 --2- 172.21.15.32:0/3349575338 >> v2:172.21.15.39:3300/0 conn(0x7f628006da40 0x7f628006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 -- 172.21.15.32:0/3349575338 >> v2:172.21.15.39:3300/0 conn(0x7f628006da40 msgr2=0x7f628006e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 --2- 172.21.15.32:0/3349575338 >> v2:172.21.15.39:3300/0 conn(0x7f628006da40 0x7f628006e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 -- 172.21.15.32:0/3349575338 >> v1:172.21.15.32:6789/0 conn(0x7f62800ccad0 legacy=0x7f6280162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.241+0000 7f6277e00640 1 -- 172.21.15.32:0/3349575338 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628015f140 con 0x7f628015e680 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.242+0000 7f6277e00640 1 --2- 172.21.15.32:0/3349575338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628006c980 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f62800cd7b0 tx=0x7f626802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06084775fb23327 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f627d400640 1 -- 172.21.15.32:0/3349575338 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268034070 con 0x7f628015e680 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f627d400640 1 -- 172.21.15.32:0/3349575338 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6268030860 con 0x7f628015e680 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f627d400640 1 -- 172.21.15.32:0/3349575338 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268030c30 con 0x7f628015e680 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f6284c00640 1 -- 172.21.15.32:0/3349575338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 msgr2=0x7f628006c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f6284c00640 1 --2- 172.21.15.32:0/3349575338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628006c980 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f62800cd7b0 tx=0x7f626802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f6284c00640 1 -- 172.21.15.32:0/3349575338 shutdown_connections 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f6284c00640 1 --2- 172.21.15.32:0/3349575338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628006c980 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f6284c00640 1 --2- 172.21.15.32:0/3349575338 >> v2:172.21.15.39:3300/0 conn(0x7f628006da40 0x7f628006e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.243+0000 7f6284c00640 1 -- 172.21.15.32:0/3349575338 >> 172.21.15.32:0/3349575338 conn(0x7f628006b3d0 msgr2=0x7f628006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.244+0000 7f6284c00640 1 -- 172.21.15.32:0/3349575338 shutdown_connections 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.244+0000 7f6284c00640 1 -- 172.21.15.32:0/3349575338 wait complete. 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.245+0000 7f6284c00640 1 Processor -- start 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.245+0000 7f6284c00640 1 -- start start 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.246+0000 7f6284c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f628006da40 0x7f62801f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.246+0000 7f6284c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62800ccad0 0x7f628009b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.246+0000 7f6284c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628014ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.246+0000 7f6284c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628006ea00 con 0x7f628015e680 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.246+0000 7f6284c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628006e880 con 0x7f62800ccad0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.246+0000 7f6284c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628006cec0 con 0x7f628006da40 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628014ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628014ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35966/0 (socket says 172.21.15.32:35966) 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 -- 172.21.15.32:0/326304754 learned_addr learned my addr 172.21.15.32:0/326304754 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f6277e00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62800ccad0 0x7f628009b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 -- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f628006da40 msgr2=0x7f62801f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627de00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f628006da40 0x7f62801f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f628006da40 0x7f62801f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 -- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62800ccad0 msgr2=0x7f628009b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62800ccad0 0x7f628009b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627e800640 1 -- 172.21.15.32:0/326304754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628014f2b0 con 0x7f628015e680 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f627de00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f628006da40 0x7f62801f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.247+0000 7f6277e00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62800ccad0 0x7f628009b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.248+0000 7f627e800640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628014ed70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f627800c780 tx=0x7f627800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.248+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278017070 con 0x7f628015e680 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.248+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627800a640 con 0x7f628015e680 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.248+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627800aa30 con 0x7f628015e680 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.248+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62801f3fa0 con 0x7f628015e680 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.248+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62801f4560 con 0x7f628015e680 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.249+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628004f330 con 0x7f628015e680 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.250+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f627801a070 con 0x7f628015e680 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.251+0000 7f6276000640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f625c06b4d0 0x7f625c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.251+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f62780897a0 con 0x7f628015e680 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.251+0000 7f627de00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f625c06b4d0 0x7f625c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.252+0000 7f627de00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f625c06b4d0 0x7f625c06d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6270006fd0 tx=0x7f6270008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.256+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6278057f40 con 0x7f628015e680 2024-03-06T07:29:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.457+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62800cf4b0 con 0x7f628015e680 2024-03-06T07:29:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.458+0000 7f6276000640 1 -- 172.21.15.32:0/326304754 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f627805bfe0 con 0x7f628015e680 2024-03-06T07:29:56.461 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:29:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f625c06b4d0 msgr2=0x7f625c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f625c06b4d0 0x7f625c06d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6270006fd0 tx=0x7f6270008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 msgr2=0x7f628014ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628014ed70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f627800c780 tx=0x7f627800cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 shutdown_connections 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f625c06b4d0 0x7f625c06d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f628015e680 0x7f628014ed70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62800ccad0 0x7f628009b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 --2- 172.21.15.32:0/326304754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f628006da40 0x7f62801f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 >> 172.21.15.32:0/326304754 conn(0x7f628006b3d0 msgr2=0x7f628006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.461+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 shutdown_connections 2024-03-06T07:29:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.462+0000 7f6284c00640 1 -- 172.21.15.32:0/326304754 wait complete. 2024-03-06T07:29:56.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:56.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.698+0000 7f94f8200640 1 Processor -- start 2024-03-06T07:29:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.698+0000 7f94f8200640 1 -- start start 2024-03-06T07:29:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.700+0000 7f94f8200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f94f0070ee0 0x7f94f0162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.700+0000 7f94f8200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f00cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.700+0000 7f94f8200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f00ce090 con 0x7f94f00712c0 2024-03-06T07:29:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.700+0000 7f94f8200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f00ceca0 con 0x7f94f0070ee0 2024-03-06T07:29:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.700+0000 7f94f8200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00ce370 con 0x7f94f0072da0 2024-03-06T07:29:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.701+0000 7f94f5800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f94f0070ee0 0x7f94f0162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.701+0000 7f94f4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f00cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.701+0000 7f94f4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f00cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:40588/0 (socket says 172.21.15.32:40588) 2024-03-06T07:29:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.701+0000 7f94f4e00640 1 -- 172.21.15.32:0/1286955700 learned_addr learned my addr 172.21.15.32:0/1286955700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.701+0000 7f94f5800640 1 --2- 172.21.15.32:0/1286955700 >> v2:172.21.15.39:3300/0 conn(0x7f94f0070ee0 0x7f94f0162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:44708/0 (socket says 172.21.15.32:44708) 2024-03-06T07:29:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.702+0000 7f94f4e00640 1 -- 172.21.15.32:0/1286955700 >> v2:172.21.15.39:3300/0 conn(0x7f94f0070ee0 msgr2=0x7f94f0162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.702+0000 7f94f4e00640 1 --2- 172.21.15.32:0/1286955700 >> v2:172.21.15.39:3300/0 conn(0x7f94f0070ee0 0x7f94f0162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.702+0000 7f94f4e00640 1 -- 172.21.15.32:0/1286955700 >> v1:172.21.15.39:6789/0 conn(0x7f94f00712c0 legacy=0x7f94f006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.702+0000 7f94f4e00640 1 -- 172.21.15.32:0/1286955700 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f00ce650 con 0x7f94f0072da0 2024-03-06T07:29:56.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.703+0000 7f94f4e00640 1 --2- 172.21.15.32:0/1286955700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f00cf130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009870 tx=0x7f94dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adbc9f8b3e2cfe94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.704+0000 7f94efe00640 1 -- 172.21.15.32:0/1286955700 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc034070 con 0x7f94f0072da0 2024-03-06T07:29:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.704+0000 7f94efe00640 1 -- 172.21.15.32:0/1286955700 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc030840 con 0x7f94f0072da0 2024-03-06T07:29:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.704+0000 7f94efe00640 1 -- 172.21.15.32:0/1286955700 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc030c10 con 0x7f94f0072da0 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.704+0000 7f94f8200640 1 -- 172.21.15.32:0/1286955700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 msgr2=0x7f94f00cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.704+0000 7f94f8200640 1 --2- 172.21.15.32:0/1286955700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f00cf130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009870 tx=0x7f94dc02b980 comp rx=0 tx=0).stop 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.705+0000 7f94f8200640 1 -- 172.21.15.32:0/1286955700 shutdown_connections 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.705+0000 7f94f8200640 1 --2- 172.21.15.32:0/1286955700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f00cf130 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.705+0000 7f94f8200640 1 --2- 172.21.15.32:0/1286955700 >> v2:172.21.15.39:3300/0 conn(0x7f94f0070ee0 0x7f94f0162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.705+0000 7f94f8200640 1 -- 172.21.15.32:0/1286955700 >> 172.21.15.32:0/1286955700 conn(0x7f94f006afb0 msgr2=0x7f94f006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.705+0000 7f94f8200640 1 -- 172.21.15.32:0/1286955700 shutdown_connections 2024-03-06T07:29:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.705+0000 7f94f8200640 1 -- 172.21.15.32:0/1286955700 wait complete. 2024-03-06T07:29:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.707+0000 7f94f8200640 1 Processor -- start 2024-03-06T07:29:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.707+0000 7f94f8200640 1 -- start start 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.708+0000 7f94f8200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f0070ee0 0x7f94f0155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.708+0000 7f94f8200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 0x7f94f009d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.708+0000 7f94f8200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f009d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.708+0000 7f94f8200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00cdda0 con 0x7f94f00712c0 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.708+0000 7f94f8200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00cdc20 con 0x7f94f0070ee0 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.708+0000 7f94f8200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f006e7d0 con 0x7f94f0072da0 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 0x7f94f009d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 0x7f94f009d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35976/0 (socket says 172.21.15.32:35976) 2024-03-06T07:29:56.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 -- 172.21.15.32:0/2276326814 learned_addr learned my addr 172.21.15.32:0/2276326814 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:56.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f5800640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f0070ee0 0x7f94f0155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f6200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f009d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 -- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 msgr2=0x7f94f009d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f009d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 -- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f0070ee0 msgr2=0x7f94f0155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f0070ee0 0x7f94f0155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f4e00640 1 -- 172.21.15.32:0/2276326814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f009ab30 con 0x7f94f00712c0 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.709+0000 7f94f5800640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f0070ee0 0x7f94f0155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.710+0000 7f94f4e00640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 0x7f94f009d270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f94dc02b9c0 tx=0x7f94dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.710+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc034040 con 0x7f94f00712c0 2024-03-06T07:29:56.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.710+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc03ada0 con 0x7f94f00712c0 2024-03-06T07:29:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.710+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f00ae5d0 con 0x7f94f00712c0 2024-03-06T07:29:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.710+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc044610 con 0x7f94f00712c0 2024-03-06T07:29:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.710+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f009b0b0 con 0x7f94f00712c0 2024-03-06T07:29:56.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.711+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f0069510 con 0x7f94f00712c0 2024-03-06T07:29:56.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.713+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94dc04b060 con 0x7f94f00712c0 2024-03-06T07:29:56.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.714+0000 7f94ed600640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b4d0 0x7f94d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.714+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f94dc0ac340 con 0x7f94f00712c0 2024-03-06T07:29:56.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.714+0000 7f94f5800640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b4d0 0x7f94d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.715+0000 7f94f5800640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b4d0 0x7f94d006d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009b90 tx=0x7f94e4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.718+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94dc03f070 con 0x7f94f00712c0 2024-03-06T07:29:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.966+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f004f3a0 con 0x7f94f00712c0 2024-03-06T07:29:56.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.968+0000 7f94ed600640 1 -- 172.21.15.32:0/2276326814 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f94dc07b0a0 con 0x7f94f00712c0 2024-03-06T07:29:56.970 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:56.970 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:56.971 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:56.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b4d0 msgr2=0x7f94d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b4d0 0x7f94d006d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009b90 tx=0x7f94e4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 msgr2=0x7f94f009d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 0x7f94f009d270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f94dc02b9c0 tx=0x7f94dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 shutdown_connections 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b4d0 0x7f94d006d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f0072da0 0x7f94f009d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f00712c0 0x7f94f009d270 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 --2- 172.21.15.32:0/2276326814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f0070ee0 0x7f94f0155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 >> 172.21.15.32:0/2276326814 conn(0x7f94f006afb0 msgr2=0x7f94f0164870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 shutdown_connections 2024-03-06T07:29:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:56.971+0000 7f94f8200640 1 -- 172.21.15.32:0/2276326814 wait complete. 2024-03-06T07:29:56.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.202+0000 7f54db000640 1 Processor -- start 2024-03-06T07:29:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.203+0000 7f54db000640 1 -- start start 2024-03-06T07:29:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.204+0000 7f54db000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 0x7f54d40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.204+0000 7f54db000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40b3710 con 0x7f54d40d0e50 2024-03-06T07:29:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.204+0000 7f54db000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40b39f0 con 0x7f54d40d4a10 2024-03-06T07:29:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.204+0000 7f54db000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40b09e0 con 0x7f54d415e0b0 2024-03-06T07:29:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54da600640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f54d40d0e50 0x7f54d40b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34234/0 (socket says 172.21.15.32:34234) 2024-03-06T07:29:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54da600640 1 -- 172.21.15.32:0/799964434 learned_addr learned my addr 172.21.15.32:0/799964434 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d9c00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 0x7f54d40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901026657 0 0) 0x7f54d40b3710 con 0x7f54d40d0e50 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54bc003520 con 0x7f54d40d0e50 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 629491479 0 0) 0x7f54bc003520 con 0x7f54d40d0e50 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 msgr2=0x7f54d40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 0x7f54d40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 >> v1:172.21.15.39:6789/0 conn(0x7f54d40d4a10 legacy=0x7f54d40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40b3cd0 con 0x7f54d40d0e50 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f54cc0019f0 con 0x7f54d40d0e50 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.205+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 >> v1:172.21.15.32:6789/0 conn(0x7f54d40d0e50 legacy=0x7f54d40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d3e00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54bc0039e0 0x7f54bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d3e00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 0x7f54bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d3e00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc004d40 0x7f54bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc00ba70 con 0x7f54bc004d40 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc00bbf0 con 0x7f54bc0039e0 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc00bdf0 con 0x7f54d415e0b0 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54da600640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc004d40 0x7f54bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d9c00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54bc0039e0 0x7f54bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d9200640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 0x7f54bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54da600640 1 -- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 msgr2=0x7f54bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54da600640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415e0b0 0x7f54bc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54da600640 1 -- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54bc0039e0 msgr2=0x7f54bc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54da600640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54bc0039e0 0x7f54bc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54da600640 1 -- 172.21.15.32:0/799964434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc003520 con 0x7f54bc004d40 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.206+0000 7f54d9c00640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54bc0039e0 0x7f54bc003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54da600640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc004d40 0x7f54bc0051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54cc002230 tx=0x7f54bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c929694a56dcac38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00a4a0 con 0x7f54bc004d40 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc034650 con 0x7f54bc004d40 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54d3e00640 1 -- 172.21.15.32:0/799964434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc034a40 con 0x7f54bc004d40 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54db000640 1 -- 172.21.15.32:0/799964434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc004d40 msgr2=0x7f54bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54db000640 1 --2- 172.21.15.32:0/799964434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc004d40 0x7f54bc0051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54cc002230 tx=0x7f54bc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.208+0000 7f54da600640 1 -- 172.21.15.32:0/799964434 reap_dead start 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.209+0000 7f54db000640 1 -- 172.21.15.32:0/799964434 shutdown_connections 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.209+0000 7f54db000640 1 -- 172.21.15.32:0/799964434 >> 172.21.15.32:0/799964434 conn(0x7f54d40d3d30 msgr2=0x7f54d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.209+0000 7f54db000640 1 -- 172.21.15.32:0/799964434 shutdown_connections 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.209+0000 7f54db000640 1 -- 172.21.15.32:0/799964434 wait complete. 2024-03-06T07:29:57.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.210+0000 7f54db000640 1 Processor -- start 2024-03-06T07:29:57.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 -- start start 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415c120 0x7f54d41611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d415c500 0x7f54d40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 0x7f54d415ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40cfc60 con 0x7f54d415e610 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d415e490 con 0x7f54d415c500 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.211+0000 7f54db000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40cf310 con 0x7f54d415c120 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 0x7f54d415ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 0x7f54d415ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35984/0 (socket says 172.21.15.32:35984) 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 -- 172.21.15.32:0/2451421465 learned_addr learned my addr 172.21.15.32:0/2451421465 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54d9200640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d415c500 0x7f54d40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54d9c00640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415c120 0x7f54d41611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 -- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415c120 msgr2=0x7f54d41611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415c120 0x7f54d41611f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 -- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d415c500 msgr2=0x7f54d40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d415c500 0x7f54d40cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.212+0000 7f54da600640 1 -- 172.21.15.32:0/2451421465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40cb570 con 0x7f54d415e610 2024-03-06T07:29:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.213+0000 7f54da600640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 0x7f54d415ea90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f54cc02cc40 tx=0x7f54cc001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.213+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc045070 con 0x7f54d415e610 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.213+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc03a040 con 0x7f54d415e610 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.213+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d40cb850 con 0x7f54d415e610 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.213+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc034c40 con 0x7f54d415e610 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.213+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d40cbe10 con 0x7f54d415e610 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.214+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d40d1ff0 con 0x7f54d415e610 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.215+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54cc048070 con 0x7f54d415e610 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.216+0000 7f54d1600640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54b406b4d0 0x7f54b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.216+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f54cc0abed0 con 0x7f54d415e610 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.217+0000 7f54d9c00640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54b406b4d0 0x7f54b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.217+0000 7f54d9c00640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54b406b4d0 0x7f54b406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54d415ca20 tx=0x7f54c8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.221+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54d40d1ff0 con 0x7f54d415e610 2024-03-06T07:29:57.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.427+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54d404f330 con 0x7f54d415e610 2024-03-06T07:29:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.430+0000 7f54d1600640 1 -- 172.21.15.32:0/2451421465 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16746 (secure 0 0 0) 0x7f54cc07ac30 con 0x7f54d415e610 2024-03-06T07:29:57.433 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:57.433 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:47.656917+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54b406b4d0 msgr2=0x7f54b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54b406b4d0 0x7f54b406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54d415ca20 tx=0x7f54c8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 msgr2=0x7f54d415ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 0x7f54d415ea90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f54cc02cc40 tx=0x7f54cc001dc0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 shutdown_connections 2024-03-06T07:29:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54b406b4d0 0x7f54b406d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d415e610 0x7f54d415ea90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d415c500 0x7f54d40cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 --2- 172.21.15.32:0/2451421465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d415c120 0x7f54d41611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 >> 172.21.15.32:0/2451421465 conn(0x7f54d40d3d30 msgr2=0x7f54d40d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 shutdown_connections 2024-03-06T07:29:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.433+0000 7f54db000640 1 -- 172.21.15.32:0/2451421465 wait complete. 2024-03-06T07:29:57.456 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:57.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:57.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.680+0000 7f3573000640 1 Processor -- start 2024-03-06T07:29:57.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.680+0000 7f3573000640 1 -- start start 2024-03-06T07:29:57.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3573000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f356c070f10 0x7f356c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3573000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f356c0cce90 0x7f356c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3573000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0714d0 con 0x7f356c145300 2024-03-06T07:29:57.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3573000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0cd300 con 0x7f356c070f10 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3573000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0717b0 con 0x7f356c0cce90 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3571200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f356c070f10 0x7f356c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3571200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f356c070f10 0x7f356c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35992/0 (socket says 172.21.15.32:35992) 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.682+0000 7f3571200640 1 -- 172.21.15.32:0/3965575484 learned_addr learned my addr 172.21.15.32:0/3965575484 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.683+0000 7f3571200640 1 -- 172.21.15.32:0/3965575484 >> v2:172.21.15.39:3300/0 conn(0x7f356c0cce90 msgr2=0x7f356c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.683+0000 7f3571200640 1 --2- 172.21.15.32:0/3965575484 >> v2:172.21.15.39:3300/0 conn(0x7f356c0cce90 0x7f356c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.683+0000 7f3571200640 1 -- 172.21.15.32:0/3965575484 >> v1:172.21.15.39:6789/0 conn(0x7f356c145300 legacy=0x7f356c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.683+0000 7f3571200640 1 -- 172.21.15.32:0/3965575484 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c071a90 con 0x7f356c070f10 2024-03-06T07:29:57.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.684+0000 7f3571200640 1 --2- 172.21.15.32:0/3965575484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c0cc950 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f356c0cc500 tx=0x7f355802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ca40a8d39049ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.684+0000 7f356be00640 1 -- 172.21.15.32:0/3965575484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355802e070 con 0x7f356c070f10 2024-03-06T07:29:57.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.684+0000 7f356be00640 1 -- 172.21.15.32:0/3965575484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558037070 con 0x7f356c070f10 2024-03-06T07:29:57.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.684+0000 7f356be00640 1 -- 172.21.15.32:0/3965575484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558031a00 con 0x7f356c070f10 2024-03-06T07:29:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 -- 172.21.15.32:0/3965575484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 msgr2=0x7f356c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 --2- 172.21.15.32:0/3965575484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c0cc950 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f356c0cc500 tx=0x7f355802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 -- 172.21.15.32:0/3965575484 shutdown_connections 2024-03-06T07:29:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 --2- 172.21.15.32:0/3965575484 >> v2:172.21.15.39:3300/0 conn(0x7f356c0cce90 0x7f356c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 --2- 172.21.15.32:0/3965575484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c0cc950 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 -- 172.21.15.32:0/3965575484 >> 172.21.15.32:0/3965575484 conn(0x7f356c06b3d0 msgr2=0x7f356c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 -- 172.21.15.32:0/3965575484 shutdown_connections 2024-03-06T07:29:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.685+0000 7f3573000640 1 -- 172.21.15.32:0/3965575484 wait complete. 2024-03-06T07:29:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.687+0000 7f3573000640 1 Processor -- start 2024-03-06T07:29:57.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.687+0000 7f3573000640 1 -- start start 2024-03-06T07:29:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3573000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3573000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f356c0cce90 0x7f356c15e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3573000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c145300 0x7f356c15e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3573000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c15cab0 con 0x7f356c070f10 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3573000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c1456e0 con 0x7f356c145300 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3573000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c15cc30 con 0x7f356c0cce90 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c1436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36000/0 (socket says 172.21.15.32:36000) 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 -- 172.21.15.32:0/4267710531 learned_addr learned my addr 172.21.15.32:0/4267710531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571200640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f356c0cce90 0x7f356c15e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 -- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f356c0cce90 msgr2=0x7f356c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f356c0cce90 0x7f356c15e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 -- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c145300 msgr2=0x7f356c15e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.688+0000 7f3571c00640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c145300 0x7f356c15e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3571c00640 1 -- 172.21.15.32:0/4267710531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c1e7770 con 0x7f356c070f10 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3571200640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f356c0cce90 0x7f356c15e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3571c00640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c1436c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f356000b3f0 tx=0x7f356000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560015020 con 0x7f356c070f10 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560011830 con 0x7f356c070f10 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560011c20 con 0x7f356c070f10 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c1e7a50 con 0x7f356c070f10 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.689+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c1f39a0 con 0x7f356c070f10 2024-03-06T07:29:57.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.690+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c04f330 con 0x7f356c070f10 2024-03-06T07:29:57.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.692+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3560018020 con 0x7f356c070f10 2024-03-06T07:29:57.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.692+0000 7f3569600640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.692+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3560089ed0 con 0x7f356c070f10 2024-03-06T07:29:57.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.693+0000 7f3571200640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.694+0000 7f3571200640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3558036000 tx=0x7f3558008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.697+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f356001f030 con 0x7f356c070f10 2024-03-06T07:29:57.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.944+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f356c074ab0 con 0x7f356c070f10 2024-03-06T07:29:57.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.946+0000 7f3569600640 1 -- 172.21.15.32:0/4267710531 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f3560058cc0 con 0x7f356c070f10 2024-03-06T07:29:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:57.948 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:57.949 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 msgr2=0x7f354c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3558036000 tx=0x7f3558008030 comp rx=0 tx=0).stop 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 msgr2=0x7f356c1436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c1436c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f356000b3f0 tx=0x7f356000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 shutdown_connections 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c145300 0x7f356c15e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f356c0cce90 0x7f356c15e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 --2- 172.21.15.32:0/4267710531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c070f10 0x7f356c1436c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 >> 172.21.15.32:0/4267710531 conn(0x7f356c06b3d0 msgr2=0x7f356c0cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.948+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 shutdown_connections 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:57.949+0000 7f3573000640 1 -- 172.21.15.32:0/4267710531 wait complete. 2024-03-06T07:29:57.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.191+0000 7fd642a00640 1 Processor -- start 2024-03-06T07:29:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.192+0000 7fd642a00640 1 -- start start 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.193+0000 7fd642a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd63c0d0e50 0x7fd63c168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.193+0000 7fd642a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63c0b3710 con 0x7fd63c0d4a10 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.193+0000 7fd642a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63c0b39f0 con 0x7fd63c15e0b0 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.193+0000 7fd642a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c0b09e0 con 0x7fd63c0d0e50 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.193+0000 7fd640c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd63c0d4a10 0x7fd63c0d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34154/0 (socket says 172.21.15.32:34154) 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.193+0000 7fd640c00640 1 -- 172.21.15.32:0/374250819 learned_addr learned my addr 172.21.15.32:0/374250819 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:29:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.194+0000 7fd642000640 1 --2- 172.21.15.32:0/374250819 >> v2:172.21.15.32:3300/0 conn(0x7fd63c0d0e50 0x7fd63c168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.194+0000 7fd63be00640 1 -- 172.21.15.32:0/374250819 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136890245 0 0) 0x7fd63c0b3710 con 0x7fd63c0d4a10 2024-03-06T07:29:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.194+0000 7fd642000640 1 -- 172.21.15.32:0/374250819 >> v1:172.21.15.39:6789/0 conn(0x7fd63c15e0b0 legacy=0x7fd63c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.194+0000 7fd642000640 1 -- 172.21.15.32:0/374250819 >> v1:172.21.15.32:6789/0 conn(0x7fd63c0d4a10 legacy=0x7fd63c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.194+0000 7fd642000640 1 -- 172.21.15.32:0/374250819 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c0b3cd0 con 0x7fd63c0d0e50 2024-03-06T07:29:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.195+0000 7fd642000640 1 --2- 172.21.15.32:0/374250819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c168c50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd634009d90 tx=0x7fd63402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a8da0314603f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd63be00640 1 -- 172.21.15.32:0/374250819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634039070 con 0x7fd63c0d0e50 2024-03-06T07:29:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd63be00640 1 -- 172.21.15.32:0/374250819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63402cd00 con 0x7fd63c0d0e50 2024-03-06T07:29:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd63be00640 1 -- 172.21.15.32:0/374250819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634034660 con 0x7fd63c0d0e50 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 -- 172.21.15.32:0/374250819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 msgr2=0x7fd63c168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 --2- 172.21.15.32:0/374250819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c168c50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd634009d90 tx=0x7fd63402bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 -- 172.21.15.32:0/374250819 shutdown_connections 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 --2- 172.21.15.32:0/374250819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c168c50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 -- 172.21.15.32:0/374250819 >> 172.21.15.32:0/374250819 conn(0x7fd63c0d3d30 msgr2=0x7fd63c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 -- 172.21.15.32:0/374250819 shutdown_connections 2024-03-06T07:29:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.196+0000 7fd642a00640 1 -- 172.21.15.32:0/374250819 wait complete. 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.198+0000 7fd642a00640 1 Processor -- start 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.198+0000 7fd642a00640 1 -- start start 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.199+0000 7fd642a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c1f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.199+0000 7fd642a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd63c0d4a10 0x7fd63c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.199+0000 7fd642a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd63c15e0b0 0x7fd63c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.199+0000 7fd642a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c15e490 con 0x7fd63c0d0e50 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.199+0000 7fd642a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c0cfa20 con 0x7fd63c0d4a10 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.199+0000 7fd642a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c0cfc60 con 0x7fd63c15e0b0 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c1f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c1f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53234/0 (socket says 172.21.15.32:53234) 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 -- 172.21.15.32:0/1920330242 learned_addr learned my addr 172.21.15.32:0/1920330242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd640c00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd63c0d4a10 0x7fd63c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd642000640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd63c15e0b0 0x7fd63c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 -- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd63c15e0b0 msgr2=0x7fd63c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd63c15e0b0 0x7fd63c0cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 -- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd63c0d4a10 msgr2=0x7fd63c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd63c0d4a10 0x7fd63c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 -- 172.21.15.32:0/1920330242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c0b7680 con 0x7fd63c0d0e50 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.200+0000 7fd641600640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c1f30c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c400 tx=0x7fd62c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.201+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c017070 con 0x7fd63c0d0e50 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.201+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c0052d0 con 0x7fd63c0d0e50 2024-03-06T07:29:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.201+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c003830 con 0x7fd63c0d0e50 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.201+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c0b7960 con 0x7fd63c0d0e50 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.201+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c0b7f20 con 0x7fd63c0d0e50 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.202+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c04f330 con 0x7fd63c0d0e50 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.203+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd62c01a070 con 0x7fd63c0d0e50 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.204+0000 7fd639600640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62006b4d0 0x7fd62006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.205+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd62c089680 con 0x7fd63c0d0e50 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.205+0000 7fd640c00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62006b4d0 0x7fd62006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.205+0000 7fd640c00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62006b4d0 0x7fd62006d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd63c1f6db0 tx=0x7fd630007b90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.208+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd62c0583e0 con 0x7fd63c0d0e50 2024-03-06T07:29:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.443+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd63c074920 con 0x7fd63c0d0e50 2024-03-06T07:29:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.444+0000 7fd639600640 1 -- 172.21.15.32:0/1920330242 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fd62c057ce0 con 0x7fd63c0d0e50 2024-03-06T07:29:58.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:58.447 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/1602299651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":1602299651},{"type":"v1","addr":"172.21.15.32:6837","nonce":1602299651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4208,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3012801654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3012801654},{"type":"v1","addr":"172.21.15.39:6835","nonce":3012801654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/1992127641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1992127641},{"type":"v1","addr":"172.21.15.32:6836","nonce":1992127641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:00.778364+0000","modified":"2024-03-06T07:29:03.836328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6833/3857082618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3857082618},{"type":"v1","addr":"172.21.15.39:6833","nonce":3857082618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258]},"id":1}]} 2024-03-06T07:29:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.446+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62006b4d0 msgr2=0x7fd62006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.446+0000 7fd642a00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62006b4d0 0x7fd62006d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd63c1f6db0 tx=0x7fd630007b90 comp rx=0 tx=0).stop 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 msgr2=0x7fd63c1f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c1f30c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c400 tx=0x7fd62c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 shutdown_connections 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62006b4d0 0x7fd62006d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd63c15e0b0 0x7fd63c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd63c0d4a10 0x7fd63c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 --2- 172.21.15.32:0/1920330242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd63c0d0e50 0x7fd63c1f30c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 >> 172.21.15.32:0/1920330242 conn(0x7fd63c0d3d30 msgr2=0x7fd63c0b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 shutdown_connections 2024-03-06T07:29:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.447+0000 7fd642a00640 1 -- 172.21.15.32:0/1920330242 wait complete. 2024-03-06T07:29:58.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.675+0000 7fb5a6000640 1 Processor -- start 2024-03-06T07:29:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.675+0000 7fb5a6000640 1 -- start start 2024-03-06T07:29:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb5a6000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb5a0145300 0x7fb5a00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb5a6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 0x7fb5a00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb5a6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00714d0 con 0x7fb5a00cce90 2024-03-06T07:29:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb5a6000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00cd300 con 0x7fb5a0145300 2024-03-06T07:29:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb5a6000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00717b0 con 0x7fb5a0070f10 2024-03-06T07:29:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb597e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 0x7fb5a00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb597e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 0x7fb5a00cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:39724/0 (socket says 172.21.15.32:39724) 2024-03-06T07:29:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb597e00640 1 -- 172.21.15.32:0/3328682646 learned_addr learned my addr 172.21.15.32:0/3328682646 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.677+0000 7fb5a4c00640 1 --2- 172.21.15.32:0/3328682646 >> v2:172.21.15.39:3300/0 conn(0x7fb5a0145300 0x7fb5a00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.678+0000 7fb597e00640 1 -- 172.21.15.32:0/3328682646 >> v2:172.21.15.39:3300/0 conn(0x7fb5a0145300 msgr2=0x7fb5a00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.678+0000 7fb597e00640 1 --2- 172.21.15.32:0/3328682646 >> v2:172.21.15.39:3300/0 conn(0x7fb5a0145300 0x7fb5a00b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.678+0000 7fb597e00640 1 -- 172.21.15.32:0/3328682646 >> v1:172.21.15.39:6789/0 conn(0x7fb5a00cce90 legacy=0x7fb5a00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.678+0000 7fb597e00640 1 -- 172.21.15.32:0/3328682646 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0071a90 con 0x7fb5a0070f10 2024-03-06T07:29:58.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.679+0000 7fb597e00640 1 --2- 172.21.15.32:0/3328682646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 0x7fb5a00cc950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb588009870 tx=0x7fb58802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52b7a2b8615ace47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.679+0000 7fb59fe00640 1 -- 172.21.15.32:0/3328682646 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588034070 con 0x7fb5a0070f10 2024-03-06T07:29:58.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.679+0000 7fb59fe00640 1 -- 172.21.15.32:0/3328682646 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb588030840 con 0x7fb5a0070f10 2024-03-06T07:29:58.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.679+0000 7fb59fe00640 1 -- 172.21.15.32:0/3328682646 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588030c10 con 0x7fb5a0070f10 2024-03-06T07:29:58.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 -- 172.21.15.32:0/3328682646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 msgr2=0x7fb5a00cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 --2- 172.21.15.32:0/3328682646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 0x7fb5a00cc950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb588009870 tx=0x7fb58802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 -- 172.21.15.32:0/3328682646 shutdown_connections 2024-03-06T07:29:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 --2- 172.21.15.32:0/3328682646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0070f10 0x7fb5a00cc950 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 --2- 172.21.15.32:0/3328682646 >> v2:172.21.15.39:3300/0 conn(0x7fb5a0145300 0x7fb5a00b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 -- 172.21.15.32:0/3328682646 >> 172.21.15.32:0/3328682646 conn(0x7fb5a006b3d0 msgr2=0x7fb5a006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 -- 172.21.15.32:0/3328682646 shutdown_connections 2024-03-06T07:29:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.680+0000 7fb5a6000640 1 -- 172.21.15.32:0/3328682646 wait complete. 2024-03-06T07:29:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.682+0000 7fb5a6000640 1 Processor -- start 2024-03-06T07:29:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.682+0000 7fb5a6000640 1 -- start start 2024-03-06T07:29:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 0x7fb5a00a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a00cce90 0x7fb5a00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0145300 0x7fb5a0140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a01456e0 con 0x7fb5a0070f10 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00b4ee0 con 0x7fb5a00cce90 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a6000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a015d580 con 0x7fb5a0145300 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 0x7fb5a00a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 0x7fb5a00a9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53246/0 (socket says 172.21.15.32:53246) 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a4c00640 1 -- 172.21.15.32:0/1893577856 learned_addr learned my addr 172.21.15.32:0/1893577856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.683+0000 7fb5a5600640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0145300 0x7fb5a0140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb597e00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a00cce90 0x7fb5a00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a4c00640 1 -- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0145300 msgr2=0x7fb5a0140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a4c00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0145300 0x7fb5a0140970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a4c00640 1 -- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a00cce90 msgr2=0x7fb5a00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a4c00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a00cce90 0x7fb5a00ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a4c00640 1 -- 172.21.15.32:0/1893577856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00aa4f0 con 0x7fb5a0070f10 2024-03-06T07:29:58.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb597e00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a00cce90 0x7fb5a00ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a4c00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 0x7fb5a00a9e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb59000efc0 tx=0x7fb59000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590018020 con 0x7fb5a0070f10 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb590014480 con 0x7fb5a0070f10 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590014850 con 0x7fb5a0070f10 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.684+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a00aa7d0 con 0x7fb5a0070f10 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.685+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a00aad90 con 0x7fb5a0070f10 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.686+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a004f330 con 0x7fb5a0070f10 2024-03-06T07:29:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.686+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb590005d10 con 0x7fb5a0070f10 2024-03-06T07:29:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.687+0000 7fb59d600640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb57c06b4d0 0x7fb57c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.687+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb59008ac40 con 0x7fb5a0070f10 2024-03-06T07:29:58.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.688+0000 7fb597e00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb57c06b4d0 0x7fb57c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.688+0000 7fb597e00640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb57c06b4d0 0x7fb57c06d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb588036000 tx=0x7fb58803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.692+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb590022070 con 0x7fb5a0070f10 2024-03-06T07:29:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.895+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5a0074ab0 con 0x7fb5a0070f10 2024-03-06T07:29:58.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.898+0000 7fb59d600640 1 -- 172.21.15.32:0/1893577856 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16746 (secure 0 0 0) 0x7fb590010040 con 0x7fb5a0070f10 2024-03-06T07:29:58.900 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:29:58.901 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:29:47.656917+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:58.495536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:59.791725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.900+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb57c06b4d0 msgr2=0x7fb57c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.900+0000 7fb5a6000640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb57c06b4d0 0x7fb57c06d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb588036000 tx=0x7fb58803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.900+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 msgr2=0x7fb5a00a9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.900+0000 7fb5a6000640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 0x7fb5a00a9e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb59000efc0 tx=0x7fb59000c490 comp rx=0 tx=0).stop 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 shutdown_connections 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb57c06b4d0 0x7fb57c06d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5a0145300 0x7fb5a0140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a00cce90 0x7fb5a00ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 --2- 172.21.15.32:0/1893577856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5a0070f10 0x7fb5a00a9e80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 >> 172.21.15.32:0/1893577856 conn(0x7fb5a006b3d0 msgr2=0x7fb5a00b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 shutdown_connections 2024-03-06T07:29:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:58.901+0000 7fb5a6000640 1 -- 172.21.15.32:0/1893577856 wait complete. 2024-03-06T07:29:58.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:59.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.137+0000 7f46ee400640 1 Processor -- start 2024-03-06T07:29:59.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.137+0000 7f46ee400640 1 -- start start 2024-03-06T07:29:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46ee400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46e80af1a0 0x7f46e815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46ee400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f46e80cdc20 0x7f46e80b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46ee400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e815cc40 con 0x7f46e80af580 2024-03-06T07:29:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46ee400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e815cf20 con 0x7f46e80af1a0 2024-03-06T07:29:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46ee400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e815d200 con 0x7f46e80cdc20 2024-03-06T07:29:59.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46e7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46e80af1a0 0x7f46e815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46e7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46e80af1a0 0x7f46e815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53260/0 (socket says 172.21.15.32:53260) 2024-03-06T07:29:59.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.139+0000 7f46e7400640 1 -- 172.21.15.32:0/80907131 learned_addr learned my addr 172.21.15.32:0/80907131 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:59.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.140+0000 7f46e7e00640 1 --2- 172.21.15.32:0/80907131 >> v2:172.21.15.39:3300/0 conn(0x7f46e80cdc20 0x7f46e80b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.140+0000 7f46e7400640 1 -- 172.21.15.32:0/80907131 >> v2:172.21.15.39:3300/0 conn(0x7f46e80cdc20 msgr2=0x7f46e80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.140+0000 7f46e7400640 1 --2- 172.21.15.32:0/80907131 >> v2:172.21.15.39:3300/0 conn(0x7f46e80cdc20 0x7f46e80b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.140+0000 7f46e7400640 1 -- 172.21.15.32:0/80907131 >> v1:172.21.15.39:6789/0 conn(0x7f46e80af580 legacy=0x7f46e80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.140+0000 7f46e7400640 1 -- 172.21.15.32:0/80907131 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e815d4e0 con 0x7f46e80af1a0 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.141+0000 7f46e7400640 1 --2- 172.21.15.32:0/80907131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80af1a0 0x7f46e815c600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f46e806c0c0 tx=0x7f46d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da1304fac525e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.141+0000 7f46e6a00640 1 -- 172.21.15.32:0/80907131 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d802e070 con 0x7f46e80af1a0 2024-03-06T07:29:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.141+0000 7f46e6a00640 1 -- 172.21.15.32:0/80907131 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d8037070 con 0x7f46e80af1a0 2024-03-06T07:29:59.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.141+0000 7f46e6a00640 1 -- 172.21.15.32:0/80907131 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8031a00 con 0x7f46e80af1a0 2024-03-06T07:29:59.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 -- 172.21.15.32:0/80907131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80af1a0 msgr2=0x7f46e815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 --2- 172.21.15.32:0/80907131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80af1a0 0x7f46e815c600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f46e806c0c0 tx=0x7f46d802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 -- 172.21.15.32:0/80907131 shutdown_connections 2024-03-06T07:29:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 --2- 172.21.15.32:0/80907131 >> v2:172.21.15.39:3300/0 conn(0x7f46e80cdc20 0x7f46e80b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 --2- 172.21.15.32:0/80907131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80af1a0 0x7f46e815c600 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 -- 172.21.15.32:0/80907131 >> 172.21.15.32:0/80907131 conn(0x7f46e806afb0 msgr2=0x7f46e806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 -- 172.21.15.32:0/80907131 shutdown_connections 2024-03-06T07:29:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.142+0000 7f46ee400640 1 -- 172.21.15.32:0/80907131 wait complete. 2024-03-06T07:29:59.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.144+0000 7f46ee400640 1 Processor -- start 2024-03-06T07:29:59.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.144+0000 7f46ee400640 1 -- start start 2024-03-06T07:29:59.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.145+0000 7f46ee400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46e80af1a0 0x7f46e806e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.145+0000 7f46ee400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46e80af580 0x7f46e815e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.145+0000 7f46ee400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 0x7f46e815ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.145+0000 7f46ee400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b36a0 con 0x7f46e80cdc20 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.145+0000 7f46ee400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b3520 con 0x7f46e80af580 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.145+0000 7f46ee400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80b3820 con 0x7f46e80af1a0 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 0x7f46e815ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 0x7f46e815ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53264/0 (socket says 172.21.15.32:53264) 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 -- 172.21.15.32:0/1889496859 learned_addr learned my addr 172.21.15.32:0/1889496859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46e80af1a0 0x7f46e806e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46dfe00640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46e80af580 0x7f46e815e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 -- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46e80af1a0 msgr2=0x7f46e806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46e80af1a0 0x7f46e806e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 -- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46e80af580 msgr2=0x7f46e815e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46e80af580 0x7f46e815e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 -- 172.21.15.32:0/1889496859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e8155a10 con 0x7f46e80cdc20 2024-03-06T07:29:59.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46e80af1a0 0x7f46e806e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.146+0000 7f46e7e00640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 0x7f46e815ed50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46e0007e60 tx=0x7f46e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.147+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0014420 con 0x7f46e80cdc20 2024-03-06T07:29:59.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.147+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e0019070 con 0x7f46e80cdc20 2024-03-06T07:29:59.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.147+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0014820 con 0x7f46e80cdc20 2024-03-06T07:29:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.147+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e8156e80 con 0x7f46e80cdc20 2024-03-06T07:29:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.147+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e8157120 con 0x7f46e80cdc20 2024-03-06T07:29:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.148+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e804f3a0 con 0x7f46e80cdc20 2024-03-06T07:29:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.149+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46e00149c0 con 0x7f46e80cdc20 2024-03-06T07:29:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.150+0000 7f46df400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46c406b4d0 0x7f46c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.150+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f46e008ac10 con 0x7f46e80cdc20 2024-03-06T07:29:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.150+0000 7f46e7400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46c406b4d0 0x7f46c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.151+0000 7f46e7400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46c406b4d0 0x7f46c406d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f46d8036000 tx=0x7f46d8008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.154+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46e00598e0 con 0x7f46e80cdc20 2024-03-06T07:29:59.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.395+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f46e806ba20 con 0x7f46e80cdc20 2024-03-06T07:29:59.398 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:29:59.396+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:59.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.400+0000 7f46df400640 1 -- 172.21.15.32:0/1889496859 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f46e00591e0 con 0x7f46e80cdc20 2024-03-06T07:29:59.402 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:59.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46c406b4d0 msgr2=0x7f46c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46c406b4d0 0x7f46c406d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f46d8036000 tx=0x7f46d8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 msgr2=0x7f46e815ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 0x7f46e815ed50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46e0007e60 tx=0x7f46e000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 shutdown_connections 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46c406b4d0 0x7f46c406d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e80cdc20 0x7f46e815ed50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46e80af580 0x7f46e815e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 --2- 172.21.15.32:0/1889496859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46e80af1a0 0x7f46e806e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.403+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 >> 172.21.15.32:0/1889496859 conn(0x7f46e806afb0 msgr2=0x7f46e80cd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.404+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 shutdown_connections 2024-03-06T07:29:59.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.404+0000 7f46ee400640 1 -- 172.21.15.32:0/1889496859 wait complete. 2024-03-06T07:29:59.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:59.442 INFO:tasks.ceph.mds.b.smithi039.stderr: -7835> 2024-03-06T07:29:04.371+0000 7f67d8200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:59.523 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:29:59.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.643+0000 7f2fc6a00640 1 Processor -- start 2024-03-06T07:29:59.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.643+0000 7f2fc6a00640 1 -- start start 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fc6a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fc015bf20 0x7f2fc00acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fc6a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fc6a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc0072df0 con 0x7f2fc0072520 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fc6a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc00cdab0 con 0x7f2fc015bf20 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fc6a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00b4d70 con 0x7f2fc0071e20 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fbfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fc015bf20 0x7f2fc00acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fbfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fc015bf20 0x7f2fc00acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53276/0 (socket says 172.21.15.32:53276) 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fbfe00640 1 -- 172.21.15.32:0/3906886734 learned_addr learned my addr 172.21.15.32:0/3906886734 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fc4a00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.645+0000 7f2fbfe00640 1 -- 172.21.15.32:0/3906886734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 msgr2=0x7f2fc00ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.646+0000 7f2fbfe00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.646+0000 7f2fbfe00640 1 -- 172.21.15.32:0/3906886734 >> v1:172.21.15.39:6789/0 conn(0x7f2fc0072520 legacy=0x7f2fc00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.646+0000 7f2fbfe00640 1 -- 172.21.15.32:0/3906886734 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00b5050 con 0x7f2fc015bf20 2024-03-06T07:29:59.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.646+0000 7f2fc4a00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.647+0000 7f2fbfe00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc015bf20 0x7f2fc00acd40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009870 tx=0x7f2fb002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7224e9ce6251fe85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.647+0000 7f2fbf400640 1 -- 172.21.15.32:0/3906886734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0035020 con 0x7f2fc015bf20 2024-03-06T07:29:59.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.647+0000 7f2fbf400640 1 -- 172.21.15.32:0/3906886734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0031640 con 0x7f2fc015bf20 2024-03-06T07:29:59.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.647+0000 7f2fbf400640 1 -- 172.21.15.32:0/3906886734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb002d040 con 0x7f2fc015bf20 2024-03-06T07:29:59.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 -- 172.21.15.32:0/3906886734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc015bf20 msgr2=0x7f2fc00acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc015bf20 0x7f2fc00acd40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009870 tx=0x7f2fb002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 -- 172.21.15.32:0/3906886734 shutdown_connections 2024-03-06T07:29:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/3906886734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc015bf20 0x7f2fc00acd40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 -- 172.21.15.32:0/3906886734 >> 172.21.15.32:0/3906886734 conn(0x7f2fc006b3d0 msgr2=0x7f2fc006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 -- 172.21.15.32:0/3906886734 shutdown_connections 2024-03-06T07:29:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.648+0000 7f2fc6a00640 1 -- 172.21.15.32:0/3906886734 wait complete. 2024-03-06T07:29:59.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.650+0000 7f2fc6a00640 1 Processor -- start 2024-03-06T07:29:59.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.650+0000 7f2fc6a00640 1 -- start start 2024-03-06T07:29:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fc6a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fc6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 0x7f2fc015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fc6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fc015bf20 0x7f2fc006c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fc6a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00b4b10 con 0x7f2fc0072520 2024-03-06T07:29:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fc6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00b4990 con 0x7f2fc015bf20 2024-03-06T07:29:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fc6a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00ce6c0 con 0x7f2fc0071e20 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 0x7f2fc015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 0x7f2fc015eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53282/0 (socket says 172.21.15.32:53282) 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fb7e00640 1 -- 172.21.15.32:0/288479693 learned_addr learned my addr 172.21.15.32:0/288479693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.651+0000 7f2fbfe00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fc4a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fc015bf20 0x7f2fc006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fb7e00640 1 -- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 msgr2=0x7f2fc00b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fb7e00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00b1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fb7e00640 1 -- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fc015bf20 msgr2=0x7f2fc006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fb7e00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fc015bf20 0x7f2fc006c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fb7e00640 1 -- 172.21.15.32:0/288479693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00b0a60 con 0x7f2fc0072520 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.652+0000 7f2fb7e00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 0x7f2fc015eca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800b3f0 tx=0x7f2fa800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.653+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8015020 con 0x7f2fc0072520 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.653+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa8011830 con 0x7f2fc0072520 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.653+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8011c20 con 0x7f2fc0072520 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.653+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc00ca7a0 con 0x7f2fc0072520 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.653+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc00cafe0 con 0x7f2fc0072520 2024-03-06T07:29:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.654+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc004f330 con 0x7f2fc0072520 2024-03-06T07:29:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.655+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fa8018020 con 0x7f2fc0072520 2024-03-06T07:29:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.656+0000 7f2fbcc00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f9c06b500 0x7f2f9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.656+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f2fa8089da0 con 0x7f2fc0072520 2024-03-06T07:29:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.657+0000 7f2fbfe00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f9c06b500 0x7f2f9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.657+0000 7f2fbfe00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f9c06b500 0x7f2f9c06d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0036000 tx=0x7f2fb003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.660+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fa801f030 con 0x7f2fc0072520 2024-03-06T07:29:59.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:29:59.903+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2fc015b740 con 0x7f2fc0072520 2024-03-06T07:30:00.001 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:29:59.999+0000 7fd1b8000640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.403+0000 7f2fbcc00640 1 -- 172.21.15.32:0/288479693 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fc00ca7a0 con 0x7f2fc0072520 2024-03-06T07:30:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f9c06b500 msgr2=0x7f2f9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f9c06b500 0x7f2f9c06d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0036000 tx=0x7f2fb003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 msgr2=0x7f2fc015eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 0x7f2fc015eca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800b3f0 tx=0x7f2fa800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 shutdown_connections 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f9c06b500 0x7f2f9c06d9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fc015bf20 0x7f2fc006c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc0072520 0x7f2fc015eca0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 --2- 172.21.15.32:0/288479693 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fc0071e20 0x7f2fc00b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 >> 172.21.15.32:0/288479693 conn(0x7f2fc006b3d0 msgr2=0x7f2fc00b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 shutdown_connections 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.407+0000 7f2fc6a00640 1 -- 172.21.15.32:0/288479693 wait complete. 2024-03-06T07:30:00.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:00.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.658+0000 7f3053e00640 1 Processor -- start 2024-03-06T07:30:00.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.659+0000 7f3053e00640 1 -- start start 2024-03-06T07:30:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.660+0000 7f3053e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 0x7f30540ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.660+0000 7f3053e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30540cf970 con 0x7f305415bd80 2024-03-06T07:30:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.660+0000 7f3053e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30540cd280 con 0x7f30540c5da0 2024-03-06T07:30:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.660+0000 7f3053e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30540c5630 con 0x7f30540b41d0 2024-03-06T07:30:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3052a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f305415bd80 0x7f30540ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34158/0 (socket says 172.21.15.32:34158) 2024-03-06T07:30:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3052a00640 1 -- 172.21.15.32:0/247450075 learned_addr learned my addr 172.21.15.32:0/247450075 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3052000640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 0x7f30540ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886003276 0 0) 0x7f30540cf970 con 0x7f305415bd80 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3038003520 con 0x7f305415bd80 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387246325 0 0) 0x7f3038003520 con 0x7f305415bd80 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 msgr2=0x7f30540ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3051600640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 0x7f30540ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.661+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 >> v1:172.21.15.39:6789/0 conn(0x7f30540c5da0 legacy=0x7f30540c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30540cd560 con 0x7f305415bd80 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f30440019f0 con 0x7f305415bd80 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 >> v1:172.21.15.32:6789/0 conn(0x7f305415bd80 legacy=0x7f30540ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 0x7f30380039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3038004060 0x7f3038004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3038004d40 0x7f30380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30540c5630 con 0x7f3038004060 2024-03-06T07:30:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038010010 con 0x7f3038004d40 2024-03-06T07:30:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3038010210 con 0x7f30540b41d0 2024-03-06T07:30:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3052000640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3038004060 0x7f3038004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3053400640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3038004d40 0x7f30380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3052a00640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 0x7f30380039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3052000640 1 -- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 msgr2=0x7f30380039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.662+0000 7f3052000640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30540b41d0 0x7f30380039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.663+0000 7f3052000640 1 -- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3038004d40 msgr2=0x7f30380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.663+0000 7f3052000640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3038004d40 0x7f30380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.663+0000 7f3052000640 1 -- 172.21.15.32:0/247450075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3038003520 con 0x7f3038004060 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.663+0000 7f3053400640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3038004d40 0x7f30380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.664+0000 7f3052000640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3038004060 0x7f3038004800 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f303c00ba30 tx=0x7f3038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7503905aa61a906 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.664+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c036020 con 0x7f3038004060 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.664+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303c032640 con 0x7f3038004060 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.664+0000 7f3051600640 1 -- 172.21.15.32:0/247450075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c00b600 con 0x7f3038004060 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053e00640 1 -- 172.21.15.32:0/247450075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3038004060 msgr2=0x7f3038004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053e00640 1 --2- 172.21.15.32:0/247450075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3038004060 0x7f3038004800 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f303c00ba30 tx=0x7f3038005830 comp rx=0 tx=0).stop 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053400640 1 -- 172.21.15.32:0/247450075 reap_dead start 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053e00640 1 -- 172.21.15.32:0/247450075 shutdown_connections 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053e00640 1 -- 172.21.15.32:0/247450075 >> 172.21.15.32:0/247450075 conn(0x7f30540b36b0 msgr2=0x7f30540b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053e00640 1 -- 172.21.15.32:0/247450075 shutdown_connections 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.665+0000 7f3053e00640 1 -- 172.21.15.32:0/247450075 wait complete. 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.667+0000 7f3053e00640 1 Processor -- start 2024-03-06T07:30:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.667+0000 7f3053e00640 1 -- start start 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f305415cc40 0x7f3054153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305415d020 0x7f3054153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 0x7f3054148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305415c160 con 0x7f3054148ac0 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30540c5910 con 0x7f305415cc40 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30540ce9a0 con 0x7f305415d020 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.668+0000 7f3053400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 0x7f3054148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 0x7f3054148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53300/0 (socket says 172.21.15.32:53300) 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 -- 172.21.15.32:0/692384907 learned_addr learned my addr 172.21.15.32:0/692384907 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 -- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305415d020 msgr2=0x7f3054153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305415d020 0x7f3054153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 -- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f305415cc40 msgr2=0x7f3054153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3052a00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f305415cc40 0x7f3054153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f305415cc40 0x7f3054153930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 -- 172.21.15.32:0/692384907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305415eea0 con 0x7f3054148ac0 2024-03-06T07:30:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.669+0000 7f3053400640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 0x7f3054148f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f305415f8b0 tx=0x7f304c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.670+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c005480 con 0x7f3054148ac0 2024-03-06T07:30:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.670+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c007030 con 0x7f3054148ac0 2024-03-06T07:30:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.670+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c007440 con 0x7f3054148ac0 2024-03-06T07:30:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.670+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30540b47a0 con 0x7f3054148ac0 2024-03-06T07:30:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.670+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3054158b00 con 0x7f3054148ac0 2024-03-06T07:30:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.671+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305404f330 con 0x7f3054148ac0 2024-03-06T07:30:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.672+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f304c01b020 con 0x7f3054148ac0 2024-03-06T07:30:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.673+0000 7f304aa00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f303006b6a0 0x7f303006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.673+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f304c089d50 con 0x7f3054148ac0 2024-03-06T07:30:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.673+0000 7f3052a00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f303006b6a0 0x7f303006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.674+0000 7f3052a00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f303006b6a0 0x7f303006db60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3044001b60 tx=0x7f3044009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.677+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f304c0589d0 con 0x7f3054148ac0 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:00.877+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f30540ce680 con 0x7f3054148ac0 2024-03-06T07:30:01.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.801+0000 7f304aa00640 1 -- 172.21.15.32:0/692384907 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f304c011e10 con 0x7f3054148ac0 2024-03-06T07:30:01.804 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f303006b6a0 msgr2=0x7f303006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f303006b6a0 0x7f303006db60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3044001b60 tx=0x7f3044009040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 msgr2=0x7f3054148f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 0x7f3054148f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f305415f8b0 tx=0x7f304c00d890 comp rx=0 tx=0).stop 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 shutdown_connections 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f303006b6a0 0x7f303006db60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3054148ac0 0x7f3054148f40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305415d020 0x7f3054153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 --2- 172.21.15.32:0/692384907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f305415cc40 0x7f3054153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.806+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 >> 172.21.15.32:0/692384907 conn(0x7f30540b36b0 msgr2=0x7f305406c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.807+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 shutdown_connections 2024-03-06T07:30:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:01.807+0000 7f3053e00640 1 -- 172.21.15.32:0/692384907 wait complete. 2024-03-06T07:30:01.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:02.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.047+0000 7f74a7e00640 1 Processor -- start 2024-03-06T07:30:02.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.048+0000 7f74a7e00640 1 -- start start 2024-03-06T07:30:02.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74a806e200 0x7f74a80720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a7e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a80b3ea0 con 0x7f74a8144320 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a7e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a80b4180 con 0x7f74a80725f0 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a7e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a80b4460 con 0x7f74a806e200 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74a806e200 0x7f74a80720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74a806e200 0x7f74a80720b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53314/0 (socket says 172.21.15.32:53314) 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a6000640 1 -- 172.21.15.32:0/1589920119 learned_addr learned my addr 172.21.15.32:0/1589920119 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.049+0000 7f74a6a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f74a8144320 0x7f74a806daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34160/0 (socket says 172.21.15.32:34160) 2024-03-06T07:30:02.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.050+0000 7f74a6000640 1 -- 172.21.15.32:0/1589920119 >> v1:172.21.15.39:6789/0 conn(0x7f74a80725f0 legacy=0x7f74a80ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.050+0000 7f74a6000640 1 -- 172.21.15.32:0/1589920119 >> v1:172.21.15.32:6789/0 conn(0x7f74a8144320 legacy=0x7f74a806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.050+0000 7f74a6000640 1 -- 172.21.15.32:0/1589920119 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80b4740 con 0x7f74a806e200 2024-03-06T07:30:02.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.051+0000 7f74a6000640 1 --2- 172.21.15.32:0/1589920119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a806e200 0x7f74a80720b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7490009870 tx=0x7f749002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9961194f060726b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.051+0000 7f74a5600640 1 -- 172.21.15.32:0/1589920119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490034070 con 0x7f74a806e200 2024-03-06T07:30:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.051+0000 7f74a5600640 1 -- 172.21.15.32:0/1589920119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490030860 con 0x7f74a806e200 2024-03-06T07:30:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.051+0000 7f74a5600640 1 -- 172.21.15.32:0/1589920119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490030c30 con 0x7f74a806e200 2024-03-06T07:30:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 -- 172.21.15.32:0/1589920119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a806e200 msgr2=0x7f74a80720b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 --2- 172.21.15.32:0/1589920119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a806e200 0x7f74a80720b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7490009870 tx=0x7f749002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 -- 172.21.15.32:0/1589920119 shutdown_connections 2024-03-06T07:30:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 --2- 172.21.15.32:0/1589920119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a806e200 0x7f74a80720b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 -- 172.21.15.32:0/1589920119 >> 172.21.15.32:0/1589920119 conn(0x7f74a80b17c0 msgr2=0x7f74a80b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 -- 172.21.15.32:0/1589920119 shutdown_connections 2024-03-06T07:30:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.052+0000 7f74a7e00640 1 -- 172.21.15.32:0/1589920119 wait complete. 2024-03-06T07:30:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.054+0000 7f74a7e00640 1 Processor -- start 2024-03-06T07:30:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.054+0000 7f74a7e00640 1 -- start start 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74a806e200 0x7f74a8160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 0x7f74a8144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74a8144320 0x7f74a8161960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a8072a50 con 0x7f74a80725f0 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80b4f40 con 0x7f74a806e200 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a7e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a8144700 con 0x7f74a8144320 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 0x7f74a8144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 0x7f74a8144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53318/0 (socket says 172.21.15.32:53318) 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.055+0000 7f74a6000640 1 -- 172.21.15.32:0/531993202 learned_addr learned my addr 172.21.15.32:0/531993202 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6a00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74a806e200 0x7f74a8160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a7400640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74a8144320 0x7f74a8161960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6000640 1 -- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74a8144320 msgr2=0x7f74a8161960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6000640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74a8144320 0x7f74a8161960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6000640 1 -- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74a806e200 msgr2=0x7f74a8160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6000640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74a806e200 0x7f74a8160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6000640 1 -- 172.21.15.32:0/531993202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8147080 con 0x7f74a80725f0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f74a6000640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 0x7f74a8144a00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7490000c00 tx=0x7f7490004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490048070 con 0x7f74a80725f0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490039d90 con 0x7f74a80725f0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.056+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490034040 con 0x7f74a80725f0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.057+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a8144f40 con 0x7f74a80725f0 2024-03-06T07:30:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.057+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a8147320 con 0x7f74a80725f0 2024-03-06T07:30:02.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.058+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a815e690 con 0x7f74a80725f0 2024-03-06T07:30:02.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.059+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7490037070 con 0x7f74a80725f0 2024-03-06T07:30:02.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.060+0000 7f749ea00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f748406b4d0 0x7f748406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.060+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f74900ac060 con 0x7f74a80725f0 2024-03-06T07:30:02.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.060+0000 7f74a6a00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f748406b4d0 0x7f748406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.061+0000 7f74a6a00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f748406b4d0 0x7f748406d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7498000f60 tx=0x7f7498007ce0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.065+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74a815e690 con 0x7f74a80725f0 2024-03-06T07:30:02.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.265+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f74a80b2380 con 0x7f74a80725f0 2024-03-06T07:30:02.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.805+0000 7f749ea00640 1 -- 172.21.15.32:0/531993202 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f749007afb0 con 0x7f74a80725f0 2024-03-06T07:30:02.807 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.808+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f748406b4d0 msgr2=0x7f748406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.808+0000 7f74a7e00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f748406b4d0 0x7f748406d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7498000f60 tx=0x7f7498007ce0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.808+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 msgr2=0x7f74a8144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.808+0000 7f74a7e00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 0x7f74a8144a00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7490000c00 tx=0x7f7490004730 comp rx=0 tx=0).stop 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 shutdown_connections 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f748406b4d0 0x7f748406d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74a8144320 0x7f74a8161960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74a80725f0 0x7f74a8144a00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 --2- 172.21.15.32:0/531993202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74a806e200 0x7f74a8160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 >> 172.21.15.32:0/531993202 conn(0x7f74a80b17c0 msgr2=0x7f74a806dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 shutdown_connections 2024-03-06T07:30:02.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:02.809+0000 7f74a7e00640 1 -- 172.21.15.32:0/531993202 wait complete. 2024-03-06T07:30:02.838 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:02.838 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:02.838 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:02.838 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:02.838 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:02.838 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:02.838 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:02.839 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:02.839 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:02.839 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:02.839 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:02.839 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:02.839 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:02.839 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:30:02.840 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:02.840 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:02.859 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:02.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:02.860 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:02.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:02.861 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:02.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:02.868 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:02.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:02.908 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:02.908 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:02.915 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:02.915 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:02.915 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:02.916 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:02.921 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:02.922 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:02.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:03.068 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:30:03.068 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:30:03.088 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:30:03.089 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:30:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.109+0000 7f89b5800640 1 Processor -- start 2024-03-06T07:30:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.109+0000 7f89b5800640 1 -- start start 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.110+0000 7f89b5800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89b00d0e50 0x7f89b0168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.110+0000 7f89b5800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b00b3710 con 0x7f89b015e0b0 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.110+0000 7f89b5800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b00b39f0 con 0x7f89b00d4a10 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.110+0000 7f89b5800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b00b09e0 con 0x7f89b00d0e50 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89afe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89b015e0b0 0x7f89b00cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34208/0 (socket says 172.21.15.32:34208) 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89afe00640 1 -- 172.21.15.32:0/1697814191 learned_addr learned my addr 172.21.15.32:0/1697814191 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89b4e00640 1 --2- 172.21.15.32:0/1697814191 >> v2:172.21.15.39:3300/0 conn(0x7f89b00d0e50 0x7f89b0168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89af400640 1 -- 172.21.15.32:0/1697814191 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202679347 0 0) 0x7f89b00b3710 con 0x7f89b015e0b0 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89af400640 1 -- 172.21.15.32:0/1697814191 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8994003520 con 0x7f89b015e0b0 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89b4e00640 1 -- 172.21.15.32:0/1697814191 >> v1:172.21.15.39:6789/0 conn(0x7f89b00d4a10 legacy=0x7f89b00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89af400640 1 -- 172.21.15.32:0/1697814191 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133730980 0 0) 0x7f8994003520 con 0x7f89b015e0b0 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89b4e00640 1 -- 172.21.15.32:0/1697814191 >> v1:172.21.15.32:6789/0 conn(0x7f89b015e0b0 legacy=0x7f89b00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.111+0000 7f89b4e00640 1 -- 172.21.15.32:0/1697814191 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00b3cd0 con 0x7f89b00d0e50 2024-03-06T07:30:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89b4e00640 1 --2- 172.21.15.32:0/1697814191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b00d0e50 0x7f89b0168c50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89a80097a0 tx=0x7f89a802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed9b554e2b915868 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89af400640 1 -- 172.21.15.32:0/1697814191 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8038070 con 0x7f89b00d0e50 2024-03-06T07:30:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89af400640 1 -- 172.21.15.32:0/1697814191 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a8031500 con 0x7f89b00d0e50 2024-03-06T07:30:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89af400640 1 -- 172.21.15.32:0/1697814191 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00019f0 con 0x7f89b00d0e50 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89b5800640 1 -- 172.21.15.32:0/1697814191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b00d0e50 msgr2=0x7f89b0168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89b5800640 1 --2- 172.21.15.32:0/1697814191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b00d0e50 0x7f89b0168c50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89a80097a0 tx=0x7f89a802b820 comp rx=0 tx=0).stop 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89b5800640 1 -- 172.21.15.32:0/1697814191 shutdown_connections 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.113+0000 7f89b5800640 1 --2- 172.21.15.32:0/1697814191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b00d0e50 0x7f89b0168c50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.114+0000 7f89b5800640 1 -- 172.21.15.32:0/1697814191 >> 172.21.15.32:0/1697814191 conn(0x7f89b00d3d30 msgr2=0x7f89b00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.114+0000 7f89b5800640 1 -- 172.21.15.32:0/1697814191 shutdown_connections 2024-03-06T07:30:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.114+0000 7f89b5800640 1 -- 172.21.15.32:0/1697814191 wait complete. 2024-03-06T07:30:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.115+0000 7f89b5800640 1 Processor -- start 2024-03-06T07:30:03.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.116+0000 7f89b5800640 1 -- start start 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.116+0000 7f89b5800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89b00d0e50 0x7f89b006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.116+0000 7f89b5800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 0x7f89b00cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.116+0000 7f89b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b015e0b0 0x7f89b0151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89b5800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b015e490 con 0x7f89b00d4a10 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89b5800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b00cfa20 con 0x7f89b015e0b0 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89b5800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b00cfc60 con 0x7f89b00d0e50 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 0x7f89b00cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89afe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89b00d0e50 0x7f89b006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 0x7f89b00cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53370/0 (socket says 172.21.15.32:53370) 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 -- 172.21.15.32:0/562214357 learned_addr learned my addr 172.21.15.32:0/562214357 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89afe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89b00d0e50 0x7f89b006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:39806/0 (socket says 172.21.15.32:39806) 2024-03-06T07:30:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89b4e00640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b015e0b0 0x7f89b0151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 -- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89b00d0e50 msgr2=0x7f89b006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89b00d0e50 0x7f89b006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 -- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b015e0b0 msgr2=0x7f89b0151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b015e0b0 0x7f89b0151d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.117+0000 7f89a7400640 1 -- 172.21.15.32:0/562214357 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b006c1d0 con 0x7f89b00d4a10 2024-03-06T07:30:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.118+0000 7f89a7400640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 0x7f89b00cb860 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f899800c440 tx=0x7f8998001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.118+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89980052d0 con 0x7f89b00d4a10 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.118+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8998005470 con 0x7f89b00d4a10 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.118+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998007260 con 0x7f89b00d4a10 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.118+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b015f7a0 con 0x7f89b00d4a10 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.118+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b006bee0 con 0x7f89b00d4a10 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.119+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b004f330 con 0x7f89b00d4a10 2024-03-06T07:30:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.120+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8998007400 con 0x7f89b00d4a10 2024-03-06T07:30:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.121+0000 7f89acc00640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898c06b500 0x7f898c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.121+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8998089380 con 0x7f89b00d4a10 2024-03-06T07:30:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.121+0000 7f89afe00640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898c06b500 0x7f898c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.122+0000 7f89afe00640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898c06b500 0x7f898c06d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89a0001560 tx=0x7f89a0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.126+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8998057e10 con 0x7f89b00d4a10 2024-03-06T07:30:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.357+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f89b00d07e0 con 0x7f89b00d4a10 2024-03-06T07:30:03.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.841+0000 7f89acc00640 1 -- 172.21.15.32:0/562214357 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f899805beb0 con 0x7f89b00d4a10 2024-03-06T07:30:03.844 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:30:03.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898c06b500 msgr2=0x7f898c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898c06b500 0x7f898c06d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89a0001560 tx=0x7f89a0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 msgr2=0x7f89b00cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 0x7f89b00cb860 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f899800c440 tx=0x7f8998001e70 comp rx=0 tx=0).stop 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 shutdown_connections 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f898c06b500 0x7f898c06d9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89b015e0b0 0x7f89b0151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b00d4a10 0x7f89b00cb860 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 --2- 172.21.15.32:0/562214357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89b00d0e50 0x7f89b006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.846+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 >> 172.21.15.32:0/562214357 conn(0x7f89b00d3d30 msgr2=0x7f89b00cf100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.847+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 shutdown_connections 2024-03-06T07:30:03.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:03.847+0000 7f89b5800640 1 -- 172.21.15.32:0/562214357 wait complete. 2024-03-06T07:30:03.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.070+0000 7fdf70200640 1 Processor -- start 2024-03-06T07:30:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.070+0000 7fdf70200640 1 -- start start 2024-03-06T07:30:04.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf70200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf680cf5c0 0x7fdf680cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf70200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf68071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf70200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680b4860 con 0x7fdf680b3c90 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf70200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf680ce160 con 0x7fdf680cf5c0 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf70200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680b4b40 con 0x7fdf68071050 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf6d800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdf680b3c90 0x7fdf680b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34214/0 (socket says 172.21.15.32:34214) 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf6d800640 1 -- 172.21.15.32:0/226859446 learned_addr learned my addr 172.21.15.32:0/226859446 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.072+0000 7fdf6ce00640 1 --2- 172.21.15.32:0/226859446 >> v2:172.21.15.32:3300/0 conn(0x7fdf680cf5c0 0x7fdf680cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.073+0000 7fdf6e200640 1 --2- 172.21.15.32:0/226859446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf68071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.073+0000 7fdf67e00640 1 -- 172.21.15.32:0/226859446 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343909519 0 0) 0x7fdf680b4860 con 0x7fdf680b3c90 2024-03-06T07:30:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.073+0000 7fdf6ce00640 1 -- 172.21.15.32:0/226859446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 msgr2=0x7fdf68071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.073+0000 7fdf6ce00640 1 --2- 172.21.15.32:0/226859446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf68071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.073+0000 7fdf6ce00640 1 -- 172.21.15.32:0/226859446 >> v1:172.21.15.32:6789/0 conn(0x7fdf680b3c90 legacy=0x7fdf680b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.073+0000 7fdf6ce00640 1 -- 172.21.15.32:0/226859446 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680b4e20 con 0x7fdf680cf5c0 2024-03-06T07:30:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.074+0000 7fdf6ce00640 1 --2- 172.21.15.32:0/226859446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf680cdc20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0097a0 tx=0x7fdf5c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3d72039b1e738f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf67e00640 1 -- 172.21.15.32:0/226859446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c034070 con 0x7fdf680cf5c0 2024-03-06T07:30:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf67e00640 1 -- 172.21.15.32:0/226859446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c030610 con 0x7fdf680cf5c0 2024-03-06T07:30:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf67e00640 1 -- 172.21.15.32:0/226859446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0309e0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 -- 172.21.15.32:0/226859446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 msgr2=0x7fdf680cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 --2- 172.21.15.32:0/226859446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf680cdc20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0097a0 tx=0x7fdf5c02b960 comp rx=0 tx=0).stop 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 -- 172.21.15.32:0/226859446 shutdown_connections 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 --2- 172.21.15.32:0/226859446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf680cdc20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 --2- 172.21.15.32:0/226859446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf68071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 -- 172.21.15.32:0/226859446 >> 172.21.15.32:0/226859446 conn(0x7fdf6806afb0 msgr2=0x7fdf6806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.075+0000 7fdf70200640 1 -- 172.21.15.32:0/226859446 shutdown_connections 2024-03-06T07:30:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.076+0000 7fdf70200640 1 -- 172.21.15.32:0/226859446 wait complete. 2024-03-06T07:30:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.077+0000 7fdf70200640 1 Processor -- start 2024-03-06T07:30:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.077+0000 7fdf70200640 1 -- start start 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.078+0000 7fdf70200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf6809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.078+0000 7fdf70200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf680b3c90 0x7fdf68147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.078+0000 7fdf70200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf681481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.078+0000 7fdf70200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680719d0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.078+0000 7fdf70200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6815b340 con 0x7fdf680b3c90 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.078+0000 7fdf70200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68071b50 con 0x7fdf68071050 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf681481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf681481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53396/0 (socket says 172.21.15.32:53396) 2024-03-06T07:30:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 -- 172.21.15.32:0/2693341374 learned_addr learned my addr 172.21.15.32:0/2693341374 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6ce00640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf680b3c90 0x7fdf68147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 -- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 msgr2=0x7fdf6809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf6809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 -- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf680b3c90 msgr2=0x7fdf68147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf680b3c90 0x7fdf68147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 -- 172.21.15.32:0/2693341374 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6809b0a0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6ce00640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf680b3c90 0x7fdf68147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.079+0000 7fdf6e200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf681481e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdf60007e40 tx=0x7fdf6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.080+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60014620 con 0x7fdf680cf5c0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.080+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf60019070 con 0x7fdf680cf5c0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.080+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60014a20 con 0x7fdf680cf5c0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.080+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6809b380 con 0x7fdf680cf5c0 2024-03-06T07:30:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.080+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6809b940 con 0x7fdf680cf5c0 2024-03-06T07:30:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.081+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6804f3a0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.082+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf60014bc0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.083+0000 7fdf65600640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.083+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdf6008a890 con 0x7fdf680cf5c0 2024-03-06T07:30:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.083+0000 7fdf6d800640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.084+0000 7fdf6d800640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf58007fd0 tx=0x7fdf58007920 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.087+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf600599c0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.360+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdf68074c70 con 0x7fdf680cf5c0 2024-03-06T07:30:04.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.361+0000 7fdf65600640 1 -- 172.21.15.32:0/2693341374 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fdf600592c0 con 0x7fdf680cf5c0 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.364 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:30:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.368 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.369 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:30:04.370 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:30:04.372 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:04.373 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:30:04.374 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAoG+hlZZ37NxAATnAwHkpml3ffROyDfMCVug==", 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAoG+hlaND7NxAAoHbvbOvgK6wN5ehyxn74OA==", 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:30:04.376 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAoG+hlmfz7NxAAagTnNb93uRkHuyW32gF4Ig==", 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAoG+hldSn8NxAAIO7uYDwDM/LgZhJDmL7MQQ==", 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAoG+hlWbD8NxAAcR4MkJlcwSt1dSr/RJSYuA==", 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAoG+hltPL8NxAAE1S+wn1POlJM/7HUuXA0wg==", 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:04.379 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:04.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:04.380 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:30:04.380 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf4c06b4d0 msgr2=0x7fdf4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf58007fd0 tx=0x7fdf58007920 comp rx=0 tx=0).stop 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 msgr2=0x7fdf681481e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf681481e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdf60007e40 tx=0x7fdf6000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 shutdown_connections 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf4c06b4d0 0x7fdf4c06d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf680cf5c0 0x7fdf681481e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf680b3c90 0x7fdf68147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 --2- 172.21.15.32:0/2693341374 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf68071050 0x7fdf6809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.364+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 >> 172.21.15.32:0/2693341374 conn(0x7fdf6806afb0 msgr2=0x7fdf68071f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.365+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 shutdown_connections 2024-03-06T07:30:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.365+0000 7fdf70200640 1 -- 172.21.15.32:0/2693341374 wait complete. 2024-03-06T07:30:04.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:30:04.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.598+0000 7f1ff2e00640 1 Processor -- start 2024-03-06T07:30:04.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.599+0000 7f1ff2e00640 1 -- start start 2024-03-06T07:30:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.600+0000 7f1ff2e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.600+0000 7f1ff2e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 0x7f1fec0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.600+0000 7f1ff2e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec15ed10 con 0x7f1fec1552b0 2024-03-06T07:30:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.600+0000 7f1ff2e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec15eff0 con 0x7f1fec15e810 2024-03-06T07:30:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.600+0000 7f1ff2e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec15f2d0 con 0x7f1fec0cd060 2024-03-06T07:30:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.601+0000 7f1ff2400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 0x7f1fec0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.601+0000 7f1ff1000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.601+0000 7f1ff2400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 0x7f1fec0b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:32944/0 (socket says 172.21.15.32:32944) 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.601+0000 7f1ff2400640 1 -- 172.21.15.32:0/907580877 learned_addr learned my addr 172.21.15.32:0/907580877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.602+0000 7f1ff1000640 1 -- 172.21.15.32:0/907580877 >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 msgr2=0x7f1fec0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.602+0000 7f1ff1000640 1 --2- 172.21.15.32:0/907580877 >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 0x7f1fec0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.602+0000 7f1ff1000640 1 -- 172.21.15.32:0/907580877 >> v1:172.21.15.39:6789/0 conn(0x7f1fec1552b0 legacy=0x7f1fec0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.602+0000 7f1ff1000640 1 -- 172.21.15.32:0/907580877 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec15f5b0 con 0x7f1fec0cd060 2024-03-06T07:30:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.602+0000 7f1ff2400640 1 --2- 172.21.15.32:0/907580877 >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 0x7f1fec0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.603+0000 7f1ff1000640 1 --2- 172.21.15.32:0/907580877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec15b1d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80097a0 tx=0x7f1fd802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4335f00e87701b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.604+0000 7f1febe00640 1 -- 172.21.15.32:0/907580877 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8031780 con 0x7f1fec0cd060 2024-03-06T07:30:04.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.604+0000 7f1febe00640 1 -- 172.21.15.32:0/907580877 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8037070 con 0x7f1fec0cd060 2024-03-06T07:30:04.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.604+0000 7f1febe00640 1 -- 172.21.15.32:0/907580877 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8031b40 con 0x7f1fec0cd060 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.604+0000 7f1ff2e00640 1 -- 172.21.15.32:0/907580877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 msgr2=0x7f1fec15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.604+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/907580877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec15b1d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80097a0 tx=0x7f1fd802b820 comp rx=0 tx=0).stop 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.605+0000 7f1ff2e00640 1 -- 172.21.15.32:0/907580877 shutdown_connections 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.605+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/907580877 >> v2:172.21.15.39:3300/0 conn(0x7f1fec15e810 0x7f1fec0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.605+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/907580877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec15b1d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.605+0000 7f1ff2e00640 1 -- 172.21.15.32:0/907580877 >> 172.21.15.32:0/907580877 conn(0x7f1fec06b3d0 msgr2=0x7f1fec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.605+0000 7f1ff2e00640 1 -- 172.21.15.32:0/907580877 shutdown_connections 2024-03-06T07:30:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.605+0000 7f1ff2e00640 1 -- 172.21.15.32:0/907580877 wait complete. 2024-03-06T07:30:04.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.607+0000 7f1ff2e00640 1 Processor -- start 2024-03-06T07:30:04.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.607+0000 7f1ff2e00640 1 -- start start 2024-03-06T07:30:04.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.608+0000 7f1ff2e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec1e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 0x7f1fec159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff2e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fec15e810 0x7f1fec0aa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff2e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec06dd30 con 0x7f1fec1552b0 2024-03-06T07:30:04.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff2e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0b3760 con 0x7f1fec15e810 2024-03-06T07:30:04.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff2e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec06deb0 con 0x7f1fec0cd060 2024-03-06T07:30:04.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 0x7f1fec159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 0x7f1fec159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53402/0 (socket says 172.21.15.32:53402) 2024-03-06T07:30:04.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff1000640 1 -- 172.21.15.32:0/2589026079 learned_addr learned my addr 172.21.15.32:0/2589026079 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:04.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff1a00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec1e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff2400640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fec15e810 0x7f1fec0aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.609+0000 7f1ff1000640 1 -- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 msgr2=0x7f1fec1e5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff1000640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec1e5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff1000640 1 -- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fec15e810 msgr2=0x7f1fec0aa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff1000640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fec15e810 0x7f1fec0aa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff1000640 1 -- 172.21.15.32:0/2589026079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0aa860 con 0x7f1fec1552b0 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff1a00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec1e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff1000640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 0x7f1fec159cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8034040 tx=0x7f1fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8032610 con 0x7f1fec1552b0 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd80327b0 con 0x7f1fec1552b0 2024-03-06T07:30:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8032ba0 con 0x7f1fec1552b0 2024-03-06T07:30:04.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.610+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec15a420 con 0x7f1fec1552b0 2024-03-06T07:30:04.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.611+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec15a9e0 con 0x7f1fec1552b0 2024-03-06T07:30:04.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.612+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec04f330 con 0x7f1fec1552b0 2024-03-06T07:30:04.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.613+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd804e440 con 0x7f1fec1552b0 2024-03-06T07:30:04.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.614+0000 7f1fe9600640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1fcc06b500 0x7f1fcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.614+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1fd80abc60 con 0x7f1fec1552b0 2024-03-06T07:30:04.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.614+0000 7f1ff1a00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1fcc06b500 0x7f1fcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.615+0000 7f1ff1a00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1fcc06b500 0x7f1fcc06d9c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0007fd0 tx=0x7f1fe0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.620+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fd8040070 con 0x7f1fec1552b0 2024-03-06T07:30:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.894+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1fec0cf4a0 con 0x7f1fec1552b0 2024-03-06T07:30:04.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.899+0000 7f1fe9600640 1 -- 172.21.15.32:0/2589026079 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1fd802f050 con 0x7f1fec1552b0 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.901+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1fcc06b500 msgr2=0x7f1fcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.901+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1fcc06b500 0x7f1fcc06d9c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0007fd0 tx=0x7f1fe0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 msgr2=0x7f1fec159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 0x7f1fec159cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8034040 tx=0x7f1fd80046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 shutdown_connections 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1fcc06b500 0x7f1fcc06d9c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fec15e810 0x7f1fec0aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec1552b0 0x7f1fec159cb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 --2- 172.21.15.32:0/2589026079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1fec0cd060 0x7f1fec1e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 >> 172.21.15.32:0/2589026079 conn(0x7f1fec06b3d0 msgr2=0x7f1fec0cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 shutdown_connections 2024-03-06T07:30:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:04.902+0000 7f1ff2e00640 1 -- 172.21.15.32:0/2589026079 wait complete. 2024-03-06T07:30:04.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:30:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.144+0000 7ff4e7200640 1 Processor -- start 2024-03-06T07:30:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.144+0000 7ff4e7200640 1 -- start start 2024-03-06T07:30:05.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.145+0000 7ff4e7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4e01552b0 0x7ff4e015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.145+0000 7ff4e7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e015ed10 con 0x7ff4e00cd060 2024-03-06T07:30:05.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e015eff0 con 0x7ff4e015e810 2024-03-06T07:30:05.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e015f2d0 con 0x7ff4e01552b0 2024-03-06T07:30:05.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff4e00cd060 0x7ff4e00b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34216/0 (socket says 172.21.15.32:34216) 2024-03-06T07:30:05.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e6800640 1 -- 172.21.15.32:0/717275899 learned_addr learned my addr 172.21.15.32:0/717275899 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e5e00640 1 --2- 172.21.15.32:0/717275899 >> v2:172.21.15.39:3300/0 conn(0x7ff4e01552b0 0x7ff4e015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061237309 0 0) 0x7ff4e015ed10 con 0x7ff4e00cd060 2024-03-06T07:30:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.146+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c8003520 con 0x7ff4e00cd060 2024-03-06T07:30:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1390915596 0 0) 0x7ff4c8003520 con 0x7ff4e00cd060 2024-03-06T07:30:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 >> v2:172.21.15.39:3300/0 conn(0x7ff4e01552b0 msgr2=0x7ff4e015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 --2- 172.21.15.32:0/717275899 >> v2:172.21.15.39:3300/0 conn(0x7ff4e01552b0 0x7ff4e015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 >> v1:172.21.15.39:6789/0 conn(0x7ff4e015e810 legacy=0x7ff4e00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e015f5b0 con 0x7ff4e00cd060 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff4dc0019f0 con 0x7ff4e00cd060 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 >> v1:172.21.15.32:6789/0 conn(0x7ff4e00cd060 legacy=0x7ff4e00b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8003a00 0x7ff4c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8004800 0x7ff4c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c8005210 0x7ff4c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e015f2d0 con 0x7ff4c8004800 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c800c160 con 0x7ff4c8005210 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c800c360 con 0x7ff4c8003a00 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.147+0000 7ff4e5400640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8004800 0x7ff4c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e6800640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c8005210 0x7ff4c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e5400640 1 -- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8003a00 msgr2=0x7ff4c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e5e00640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8003a00 0x7ff4c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e5400640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8003a00 0x7ff4c8003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e5400640 1 -- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c8005210 msgr2=0x7ff4c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e5400640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c8005210 0x7ff4c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e5400640 1 -- 172.21.15.32:0/717275899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c8003520 con 0x7ff4c8004800 2024-03-06T07:30:05.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e6800640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c8005210 0x7ff4c800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.148+0000 7ff4e6800640 1 -- 172.21.15.32:0/717275899 reap_dead start 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.149+0000 7ff4e5400640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8004800 0x7ff4c8004c80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0001720 tx=0x7ff4c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8bf0985092ecc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.149+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0001370 con 0x7ff4c8004800 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.149+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d00052d0 con 0x7ff4c8004800 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.149+0000 7ff4e4a00640 1 -- 172.21.15.32:0/717275899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0003830 con 0x7ff4c8004800 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 -- 172.21.15.32:0/717275899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8004800 msgr2=0x7ff4c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8004800 0x7ff4c8004c80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0001720 tx=0x7ff4c8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 -- 172.21.15.32:0/717275899 shutdown_connections 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 --2- 172.21.15.32:0/717275899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8004800 0x7ff4c8004c80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 -- 172.21.15.32:0/717275899 >> 172.21.15.32:0/717275899 conn(0x7ff4e006b3d0 msgr2=0x7ff4e006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 -- 172.21.15.32:0/717275899 shutdown_connections 2024-03-06T07:30:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.150+0000 7ff4e7200640 1 -- 172.21.15.32:0/717275899 wait complete. 2024-03-06T07:30:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.152+0000 7ff4e7200640 1 Processor -- start 2024-03-06T07:30:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.152+0000 7ff4e7200640 1 -- start start 2024-03-06T07:30:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8004800 0x7ff4e01558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 0x7ff4e0152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4e0156050 0x7ff4e01564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e006dd30 con 0x7ff4e0152aa0 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e00b3760 con 0x7ff4e0156050 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e006deb0 con 0x7ff4c8004800 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 0x7ff4e0152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 0x7ff4e0152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53422/0 (socket says 172.21.15.32:53422) 2024-03-06T07:30:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.153+0000 7ff4e5400640 1 -- 172.21.15.32:0/404054635 learned_addr learned my addr 172.21.15.32:0/404054635 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e6800640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4e0156050 0x7ff4e01564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5e00640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8004800 0x7ff4e01558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5400640 1 -- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8004800 msgr2=0x7ff4e01558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5400640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8004800 0x7ff4e01558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5400640 1 -- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4e0156050 msgr2=0x7ff4e01564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5400640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4e0156050 0x7ff4e01564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5400640 1 -- 172.21.15.32:0/404054635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e01f7950 con 0x7ff4e0152aa0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.154+0000 7ff4e5400640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 0x7ff4e0152f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0000c00 tx=0x7ff4d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.155+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0041020 con 0x7ff4e0152aa0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.155+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d0005480 con 0x7ff4e0152aa0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.155+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0001370 con 0x7ff4e0152aa0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.155+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e01f7bf0 con 0x7ff4e0152aa0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.155+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e01f81b0 con 0x7ff4e0152aa0 2024-03-06T07:30:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.156+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e004f330 con 0x7ff4e0152aa0 2024-03-06T07:30:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.157+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4d0044020 con 0x7ff4e0152aa0 2024-03-06T07:30:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.158+0000 7ff4da000640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff4c406b4d0 0x7ff4c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.158+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff4d00ab940 con 0x7ff4e0152aa0 2024-03-06T07:30:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.159+0000 7ff4e5e00640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff4c406b4d0 0x7ff4c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.160+0000 7ff4e5e00640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff4c406b4d0 0x7ff4c406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0069ad0 tx=0x7ff4d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.163+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4d004b030 con 0x7ff4e0152aa0 2024-03-06T07:30:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.437+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7ff4e015b4f0 con 0x7ff4e0152aa0 2024-03-06T07:30:05.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.441+0000 7ff4da000640 1 -- 172.21.15.32:0/404054635 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff4d00303c0 con 0x7ff4e0152aa0 2024-03-06T07:30:05.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff4c406b4d0 msgr2=0x7ff4c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff4c406b4d0 0x7ff4c406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0069ad0 tx=0x7ff4d4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 msgr2=0x7ff4e0152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 0x7ff4e0152f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0000c00 tx=0x7ff4d0001120 comp rx=0 tx=0).stop 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 shutdown_connections 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff4c406b4d0 0x7ff4c406d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4e0156050 0x7ff4e01564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e0152aa0 0x7ff4e0152f20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 --2- 172.21.15.32:0/404054635 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff4c8004800 0x7ff4e01558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 >> 172.21.15.32:0/404054635 conn(0x7ff4e006b3d0 msgr2=0x7ff4e0071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 shutdown_connections 2024-03-06T07:30:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.444+0000 7ff4e7200640 1 -- 172.21.15.32:0/404054635 wait complete. 2024-03-06T07:30:05.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:30:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.681+0000 7f65cda00640 1 Processor -- start 2024-03-06T07:30:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.681+0000 7f65cda00640 1 -- start start 2024-03-06T07:30:05.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.683+0000 7f65cda00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65c8168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.683+0000 7f65cda00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c80d1120 con 0x7f65c815dec0 2024-03-06T07:30:05.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.683+0000 7f65cda00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c80d1400 con 0x7f65c815e2a0 2024-03-06T07:30:05.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.683+0000 7f65cda00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b0ea0 con 0x7f65c80b2340 2024-03-06T07:30:05.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.683+0000 7f65c7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f65c815dec0 0x7f65c80ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34222/0 (socket says 172.21.15.32:34222) 2024-03-06T07:30:05.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.683+0000 7f65c7e00640 1 -- 172.21.15.32:0/148334054 learned_addr learned my addr 172.21.15.32:0/148334054 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65cd000640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65c8168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691642456 0 0) 0x7f65c80d1120 con 0x7f65c815dec0 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac003520 con 0x7f65c815dec0 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183018575 0 0) 0x7f65ac003520 con 0x7f65c815dec0 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 msgr2=0x7f65c8168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65c8168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 >> v1:172.21.15.39:6789/0 conn(0x7f65c815e2a0 legacy=0x7f65c80cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80b0b30 con 0x7f65c815dec0 2024-03-06T07:30:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f65b8001670 con 0x7f65c815dec0 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 >> v1:172.21.15.32:6789/0 conn(0x7f65c815dec0 legacy=0x7f65c80ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ac0039e0 0x7f65ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b0ea0 con 0x7f65ac004800 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac00bad0 con 0x7f65ac0039e0 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.684+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac00bcd0 con 0x7f65c80b2340 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65bfe00640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65cd000640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65c7e00640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ac0039e0 0x7f65ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65bfe00640 1 -- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 msgr2=0x7f65ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65bfe00640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65bfe00640 1 -- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ac0039e0 msgr2=0x7f65ac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65bfe00640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65ac0039e0 0x7f65ac003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65bfe00640 1 -- 172.21.15.32:0/148334054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac003520 con 0x7f65ac004800 2024-03-06T07:30:05.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.685+0000 7f65cd000640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b2340 0x7f65ac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65bfe00640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65ac004c80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f65b0001720 tx=0x7f65ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acf95d199ac3245 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0001370 con 0x7f65ac004800 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b00052d0 con 0x7f65ac004800 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65c7400640 1 -- 172.21.15.32:0/148334054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0003830 con 0x7f65ac004800 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65cda00640 1 -- 172.21.15.32:0/148334054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 msgr2=0x7f65ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65cda00640 1 --2- 172.21.15.32:0/148334054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65ac004c80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f65b0001720 tx=0x7f65ac005830 comp rx=0 tx=0).stop 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.687+0000 7f65cd000640 1 -- 172.21.15.32:0/148334054 reap_dead start 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.688+0000 7f65cda00640 1 -- 172.21.15.32:0/148334054 shutdown_connections 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.688+0000 7f65cda00640 1 -- 172.21.15.32:0/148334054 >> 172.21.15.32:0/148334054 conn(0x7f65c80d3d40 msgr2=0x7f65c80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.688+0000 7f65cda00640 1 -- 172.21.15.32:0/148334054 shutdown_connections 2024-03-06T07:30:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.688+0000 7f65cda00640 1 -- 172.21.15.32:0/148334054 wait complete. 2024-03-06T07:30:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.690+0000 7f65cda00640 1 Processor -- start 2024-03-06T07:30:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.690+0000 7f65cda00640 1 -- start start 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65cda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65c80b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65cda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65c815a580 0x7f65c8159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65cda00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b8c60 0x7f65c80b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65cda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80d0d40 con 0x7f65ac004800 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65cda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80cf630 con 0x7f65c815a580 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65cda00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80d0ec0 con 0x7f65c80b8c60 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.691+0000 7f65c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65c80b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65c80b7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53446/0 (socket says 172.21.15.32:53446) 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 -- 172.21.15.32:0/4224366851 learned_addr learned my addr 172.21.15.32:0/4224366851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65bfe00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65c815a580 0x7f65c8159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65cd000640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b8c60 0x7f65c80b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 -- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b8c60 msgr2=0x7f65c80b90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b8c60 0x7f65c80b90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 -- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65c815a580 msgr2=0x7f65c8159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65c815a580 0x7f65c8159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 -- 172.21.15.32:0/4224366851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c815fa70 con 0x7f65ac004800 2024-03-06T07:30:05.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65bfe00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65c815a580 0x7f65c8159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.692+0000 7f65c7e00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65c80b7220 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f65b800e760 tx=0x7f65b800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.693+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b800c6c0 con 0x7f65ac004800 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.693+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b8018070 con 0x7f65ac004800 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.693+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b800caa0 con 0x7f65ac004800 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.693+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c815fd50 con 0x7f65ac004800 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.693+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c8160310 con 0x7f65ac004800 2024-03-06T07:30:05.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.694+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c804f330 con 0x7f65ac004800 2024-03-06T07:30:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.695+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65b800cc40 con 0x7f65ac004800 2024-03-06T07:30:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.696+0000 7f65c4c00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65a406b500 0x7f65a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.696+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f65b8089980 con 0x7f65ac004800 2024-03-06T07:30:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.696+0000 7f65bfe00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65a406b500 0x7f65a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.697+0000 7f65bfe00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65a406b500 0x7f65a406d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f65b0033000 tx=0x7f65b0009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.703+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65b8058b40 con 0x7f65ac004800 2024-03-06T07:30:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.973+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f65c80ce060 con 0x7f65ac004800 2024-03-06T07:30:05.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.978+0000 7f65c4c00640 1 -- 172.21.15.32:0/4224366851 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f65b8058440 con 0x7f65ac004800 2024-03-06T07:30:05.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65a406b500 msgr2=0x7f65a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65a406b500 0x7f65a406d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f65b0033000 tx=0x7f65b0009030 comp rx=0 tx=0).stop 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 msgr2=0x7f65c80b7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65c80b7220 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f65b800e760 tx=0x7f65b800ec20 comp rx=0 tx=0).stop 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 shutdown_connections 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65a406b500 0x7f65a406d9c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65c80b8c60 0x7f65c80b90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65c815a580 0x7f65c8159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 --2- 172.21.15.32:0/4224366851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65ac004800 0x7f65c80b7220 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 >> 172.21.15.32:0/4224366851 conn(0x7f65c80d3d40 msgr2=0x7f65c815d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 shutdown_connections 2024-03-06T07:30:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:05.981+0000 7f65cda00640 1 -- 172.21.15.32:0/4224366851 wait complete. 2024-03-06T07:30:06.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:30:06.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.224+0000 7fdd61400640 1 Processor -- start 2024-03-06T07:30:06.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.224+0000 7fdd61400640 1 -- start start 2024-03-06T07:30:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.225+0000 7fdd61400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdd5c161b40 0x7fdd5c0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.225+0000 7fdd61400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdd5c0d2930 0x7fdd5c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.225+0000 7fdd61400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0cea50 con 0x7fdd5c0b5060 2024-03-06T07:30:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.225+0000 7fdd61400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cf900 con 0x7fdd5c0d2930 2024-03-06T07:30:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.225+0000 7fdd61400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cecf0 con 0x7fdd5c161b40 2024-03-06T07:30:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.225+0000 7fdd5b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdd5c0d2930 0x7fdd5c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdd5c0d2930 0x7fdd5c0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53458/0 (socket says 172.21.15.32:53458) 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5b200640 1 -- 172.21.15.32:0/2738982342 learned_addr learned my addr 172.21.15.32:0/2738982342 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5a800640 1 --2- 172.21.15.32:0/2738982342 >> v2:172.21.15.39:3300/0 conn(0x7fdd5c161b40 0x7fdd5c0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5b200640 1 -- 172.21.15.32:0/2738982342 >> v2:172.21.15.39:3300/0 conn(0x7fdd5c161b40 msgr2=0x7fdd5c0cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5b200640 1 --2- 172.21.15.32:0/2738982342 >> v2:172.21.15.39:3300/0 conn(0x7fdd5c161b40 0x7fdd5c0cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5b200640 1 -- 172.21.15.32:0/2738982342 >> v1:172.21.15.39:6789/0 conn(0x7fdd5c0b5060 legacy=0x7fdd5c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.226+0000 7fdd5b200640 1 -- 172.21.15.32:0/2738982342 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0cefd0 con 0x7fdd5c0d2930 2024-03-06T07:30:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.227+0000 7fdd5b200640 1 --2- 172.21.15.32:0/2738982342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c0d2930 0x7fdd5c0b3740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c08db00 tx=0x7fdd5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d7bf941969e803 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.227+0000 7fdd59400640 1 -- 172.21.15.32:0/2738982342 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5402ca80 con 0x7fdd5c0d2930 2024-03-06T07:30:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.227+0000 7fdd59400640 1 -- 172.21.15.32:0/2738982342 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5402cc20 con 0x7fdd5c0d2930 2024-03-06T07:30:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.227+0000 7fdd59400640 1 -- 172.21.15.32:0/2738982342 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd540343f0 con 0x7fdd5c0d2930 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 -- 172.21.15.32:0/2738982342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c0d2930 msgr2=0x7fdd5c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 --2- 172.21.15.32:0/2738982342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c0d2930 0x7fdd5c0b3740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c08db00 tx=0x7fdd5402bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 -- 172.21.15.32:0/2738982342 shutdown_connections 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 --2- 172.21.15.32:0/2738982342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c0d2930 0x7fdd5c0b3740 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 --2- 172.21.15.32:0/2738982342 >> v2:172.21.15.39:3300/0 conn(0x7fdd5c161b40 0x7fdd5c0cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 -- 172.21.15.32:0/2738982342 >> 172.21.15.32:0/2738982342 conn(0x7fdd5c09d220 msgr2=0x7fdd5c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 -- 172.21.15.32:0/2738982342 shutdown_connections 2024-03-06T07:30:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.228+0000 7fdd61400640 1 -- 172.21.15.32:0/2738982342 wait complete. 2024-03-06T07:30:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.230+0000 7fdd61400640 1 Processor -- start 2024-03-06T07:30:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.230+0000 7fdd61400640 1 -- start start 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd61400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5c0b5060 0x7fdd5c0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd61400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdd5c0d2930 0x7fdd5c0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 0x7fdd5c15fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd61400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0b2ff0 con 0x7fdd5c161b40 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd61400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cf590 con 0x7fdd5c0b5060 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd61400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0b3170 con 0x7fdd5c0d2930 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.231+0000 7fdd5b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 0x7fdd5c15fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 0x7fdd5c15fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53464/0 (socket says 172.21.15.32:53464) 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 -- 172.21.15.32:0/1603312846 learned_addr learned my addr 172.21.15.32:0/1603312846 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5a800640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5c0b5060 0x7fdd5c0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd59e00640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdd5c0d2930 0x7fdd5c0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 -- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdd5c0d2930 msgr2=0x7fdd5c0c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdd5c0d2930 0x7fdd5c0c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 -- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5c0b5060 msgr2=0x7fdd5c0b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5c0b5060 0x7fdd5c0b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 -- 172.21.15.32:0/1603312846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c15b960 con 0x7fdd5c161b40 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.232+0000 7fdd5b200640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 0x7fdd5c15fb00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009f40 tx=0x7fdd54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.233+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54043020 con 0x7fdd5c161b40 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.233+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5403e450 con 0x7fdd5c161b40 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.233+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54009510 con 0x7fdd5c161b40 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.233+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0b82f0 con 0x7fdd5c161b40 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.233+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c0b88b0 con 0x7fdd5c161b40 2024-03-06T07:30:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.234+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c04f3a0 con 0x7fdd5c161b40 2024-03-06T07:30:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.235+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd54046020 con 0x7fdd5c161b40 2024-03-06T07:30:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.236+0000 7fdd52a00640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd3c06b500 0x7fdd3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.236+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd540aca40 con 0x7fdd5c161b40 2024-03-06T07:30:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.237+0000 7fdd5a800640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd3c06b500 0x7fdd3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.237+0000 7fdd5a800640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd3c06b500 0x7fdd3c06d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006fd0 tx=0x7fdd4c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.241+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd5404d020 con 0x7fdd5c161b40 2024-03-06T07:30:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.515+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fdd5c0d43f0 con 0x7fdd5c161b40 2024-03-06T07:30:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.524+0000 7fdd52a00640 1 -- 172.21.15.32:0/1603312846 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdd5c15b960 con 0x7fdd5c161b40 2024-03-06T07:30:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.526+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd3c06b500 msgr2=0x7fdd3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.526+0000 7fdd61400640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd3c06b500 0x7fdd3c06d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006fd0 tx=0x7fdd4c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 msgr2=0x7fdd5c15fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 0x7fdd5c15fb00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009f40 tx=0x7fdd54004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 shutdown_connections 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd3c06b500 0x7fdd3c06d9c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd5c161b40 0x7fdd5c15fb00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdd5c0d2930 0x7fdd5c0c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 --2- 172.21.15.32:0/1603312846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5c0b5060 0x7fdd5c0b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 >> 172.21.15.32:0/1603312846 conn(0x7fdd5c09d220 msgr2=0x7fdd5c0b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 shutdown_connections 2024-03-06T07:30:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.527+0000 7fdd61400640 1 -- 172.21.15.32:0/1603312846 wait complete. 2024-03-06T07:30:06.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:30:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.762+0000 7fca17000640 1 Processor -- start 2024-03-06T07:30:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.762+0000 7fca17000640 1 -- start start 2024-03-06T07:30:06.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca17000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca10070f10 0x7fca100cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca17000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100714d0 con 0x7fca10145300 2024-03-06T07:30:06.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca17000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100717b0 con 0x7fca100cce90 2024-03-06T07:30:06.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca17000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100cd300 con 0x7fca10070f10 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca15200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca10070f10 0x7fca100cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca15200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca10070f10 0x7fca100cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53472/0 (socket says 172.21.15.32:53472) 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca15200640 1 -- 172.21.15.32:0/1484677967 learned_addr learned my addr 172.21.15.32:0/1484677967 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.764+0000 7fca15c00640 1 --1- 172.21.15.32:0/1484677967 >> v1:172.21.15.32:6789/0 conn(0x7fca10145300 0x7fca100b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34238/0 (socket says 172.21.15.32:34238) 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.765+0000 7fca15200640 1 -- 172.21.15.32:0/1484677967 >> v1:172.21.15.39:6789/0 conn(0x7fca100cce90 legacy=0x7fca1015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.765+0000 7fca15200640 1 -- 172.21.15.32:0/1484677967 >> v1:172.21.15.32:6789/0 conn(0x7fca10145300 legacy=0x7fca100b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.765+0000 7fca15200640 1 -- 172.21.15.32:0/1484677967 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10071a90 con 0x7fca10070f10 2024-03-06T07:30:06.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.766+0000 7fca15200640 1 --2- 172.21.15.32:0/1484677967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10070f10 0x7fca100cc950 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009870 tx=0x7fc9fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b140ba67dc1bdd12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.766+0000 7fca0fe00640 1 -- 172.21.15.32:0/1484677967 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc035020 con 0x7fca10070f10 2024-03-06T07:30:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.766+0000 7fca0fe00640 1 -- 172.21.15.32:0/1484677967 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc031870 con 0x7fca10070f10 2024-03-06T07:30:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.766+0000 7fca0fe00640 1 -- 172.21.15.32:0/1484677967 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc031c40 con 0x7fca10070f10 2024-03-06T07:30:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 -- 172.21.15.32:0/1484677967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10070f10 msgr2=0x7fca100cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 --2- 172.21.15.32:0/1484677967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10070f10 0x7fca100cc950 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009870 tx=0x7fc9fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 -- 172.21.15.32:0/1484677967 shutdown_connections 2024-03-06T07:30:06.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 --2- 172.21.15.32:0/1484677967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10070f10 0x7fca100cc950 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 -- 172.21.15.32:0/1484677967 >> 172.21.15.32:0/1484677967 conn(0x7fca1006b3d0 msgr2=0x7fca1006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 -- 172.21.15.32:0/1484677967 shutdown_connections 2024-03-06T07:30:06.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.767+0000 7fca17000640 1 -- 172.21.15.32:0/1484677967 wait complete. 2024-03-06T07:30:06.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.769+0000 7fca17000640 1 Processor -- start 2024-03-06T07:30:06.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.769+0000 7fca17000640 1 -- start start 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca17000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca10070f10 0x7fca101316c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca17000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 0x7fca10132df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca17000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca10145300 0x7fca101e7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca17000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1015cc80 con 0x7fca100cce90 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca17000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca101456e0 con 0x7fca10070f10 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca17000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1015ce00 con 0x7fca10145300 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 0x7fca10132df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 0x7fca10132df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53482/0 (socket says 172.21.15.32:53482) 2024-03-06T07:30:06.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 -- 172.21.15.32:0/2361569022 learned_addr learned my addr 172.21.15.32:0/2361569022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15c00640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca10070f10 0x7fca101316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 -- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca10145300 msgr2=0x7fca101e7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca10145300 0x7fca101e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 -- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca10070f10 msgr2=0x7fca101316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca10070f10 0x7fca101316c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15200640 1 -- 172.21.15.32:0/2361569022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10131d50 con 0x7fca100cce90 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.770+0000 7fca15c00640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca10070f10 0x7fca101316c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.771+0000 7fca15200640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 0x7fca10132df0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc000c00 tx=0x7fc9fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.771+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc040070 con 0x7fca100cce90 2024-03-06T07:30:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.771+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc03a420 con 0x7fca100cce90 2024-03-06T07:30:06.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.771+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc035040 con 0x7fca100cce90 2024-03-06T07:30:06.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.771+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca10132030 con 0x7fca100cce90 2024-03-06T07:30:06.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.771+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca101325f0 con 0x7fca100cce90 2024-03-06T07:30:06.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.772+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca101e60c0 con 0x7fca100cce90 2024-03-06T07:30:06.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.773+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9fc04b060 con 0x7fca100cce90 2024-03-06T07:30:06.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.775+0000 7fca0d600640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9f006b4d0 0x7fc9f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.775+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc9fc0abf20 con 0x7fca100cce90 2024-03-06T07:30:06.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.775+0000 7fca15c00640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9f006b4d0 0x7fc9f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.776+0000 7fca15c00640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9f006b4d0 0x7fc9f006d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca04000f60 tx=0x7fca04009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:06.780+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9fc07afc0 con 0x7fca100cce90 2024-03-06T07:30:07.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.047+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fca100b2a80 con 0x7fca100cce90 2024-03-06T07:30:07.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.051+0000 7fca0d600640 1 -- 172.21.15.32:0/2361569022 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc9fc07a8c0 con 0x7fca100cce90 2024-03-06T07:30:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9f006b4d0 msgr2=0x7fc9f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9f006b4d0 0x7fc9f006d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca04000f60 tx=0x7fca04009040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 msgr2=0x7fca10132df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 0x7fca10132df0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc000c00 tx=0x7fc9fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 shutdown_connections 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc9f006b4d0 0x7fc9f006d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca10145300 0x7fca101e7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca100cce90 0x7fca10132df0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 --2- 172.21.15.32:0/2361569022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca10070f10 0x7fca101316c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 >> 172.21.15.32:0/2361569022 conn(0x7fca1006b3d0 msgr2=0x7fca100ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 shutdown_connections 2024-03-06T07:30:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.054+0000 7fca17000640 1 -- 172.21.15.32:0/2361569022 wait complete. 2024-03-06T07:30:07.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.286+0000 7f1f37e00640 1 Processor -- start 2024-03-06T07:30:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.287+0000 7f1f37e00640 1 -- start start 2024-03-06T07:30:07.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f37e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f300d2b40 0x7f1f300d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f37e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f300b3940 con 0x7f1f300d1480 2024-03-06T07:30:07.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f37e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f300cf580 con 0x7f1f3015e2e0 2024-03-06T07:30:07.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f37e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300d4e70 con 0x7f1f300d2b40 2024-03-06T07:30:07.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f35e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1f300d1480 0x7f1f300b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34250/0 (socket says 172.21.15.32:34250) 2024-03-06T07:30:07.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f35e00640 1 -- 172.21.15.32:0/191865051 learned_addr learned my addr 172.21.15.32:0/191865051 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.289+0000 7f1f34a00640 1 --2- 172.21.15.32:0/191865051 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f300d2b40 0x7f1f300d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.290+0000 7f1f2fe00640 1 -- 172.21.15.32:0/191865051 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070740736 0 0) 0x7f1f300b3940 con 0x7f1f300d1480 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.290+0000 7f1f2fe00640 1 -- 172.21.15.32:0/191865051 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18003520 con 0x7f1f300d1480 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.290+0000 7f1f34a00640 1 -- 172.21.15.32:0/191865051 >> v1:172.21.15.39:6789/0 conn(0x7f1f3015e2e0 legacy=0x7f1f300d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.290+0000 7f1f2fe00640 1 -- 172.21.15.32:0/191865051 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911161436 0 0) 0x7f1f300cf580 con 0x7f1f3015e2e0 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.290+0000 7f1f34a00640 1 -- 172.21.15.32:0/191865051 >> v1:172.21.15.32:6789/0 conn(0x7f1f300d1480 legacy=0x7f1f300b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.290+0000 7f1f34a00640 1 -- 172.21.15.32:0/191865051 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300cf860 con 0x7f1f300d2b40 2024-03-06T07:30:07.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.292+0000 7f1f34a00640 1 --2- 172.21.15.32:0/191865051 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f300d2b40 0x7f1f300d0f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0097d0 tx=0x7f1f1c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6364114c6fc89229 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f2fe00640 1 -- 172.21.15.32:0/191865051 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c035020 con 0x7f1f300d2b40 2024-03-06T07:30:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f2fe00640 1 -- 172.21.15.32:0/191865051 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c031890 con 0x7f1f300d2b40 2024-03-06T07:30:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f2fe00640 1 -- 172.21.15.32:0/191865051 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c031c60 con 0x7f1f300d2b40 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f37e00640 1 -- 172.21.15.32:0/191865051 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f300d2b40 msgr2=0x7f1f300d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f37e00640 1 --2- 172.21.15.32:0/191865051 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f300d2b40 0x7f1f300d0f40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0097d0 tx=0x7f1f1c02b930 comp rx=0 tx=0).stop 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f37e00640 1 -- 172.21.15.32:0/191865051 shutdown_connections 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f37e00640 1 --2- 172.21.15.32:0/191865051 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f300d2b40 0x7f1f300d0f40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f37e00640 1 -- 172.21.15.32:0/191865051 >> 172.21.15.32:0/191865051 conn(0x7f1f300d3f60 msgr2=0x7f1f300d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.293+0000 7f1f37e00640 1 -- 172.21.15.32:0/191865051 shutdown_connections 2024-03-06T07:30:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.294+0000 7f1f37e00640 1 -- 172.21.15.32:0/191865051 wait complete. 2024-03-06T07:30:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.295+0000 7f1f37e00640 1 Processor -- start 2024-03-06T07:30:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.295+0000 7f1f37e00640 1 -- start start 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f37e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 0x7f1f300c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f37e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3015e2e0 0x7f1f3014f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f37e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f3014f870 0x7f1f3014e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f37e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300cfe20 con 0x7f1f300d2b40 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f37e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300b3c40 con 0x7f1f3015e2e0 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f37e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300b47e0 con 0x7f1f3014f870 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f35400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 0x7f1f300c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.296+0000 7f1f35400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 0x7f1f300c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53488/0 (socket says 172.21.15.32:53488) 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 -- 172.21.15.32:0/263642837 learned_addr learned my addr 172.21.15.32:0/263642837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f34a00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3015e2e0 0x7f1f3014f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f3014f870 0x7f1f3014e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 -- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f3014f870 msgr2=0x7f1f3014e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f3014f870 0x7f1f3014e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 -- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3015e2e0 msgr2=0x7f1f3014f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3015e2e0 0x7f1f3014f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 -- 172.21.15.32:0/263642837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300c82c0 con 0x7f1f300d2b40 2024-03-06T07:30:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f34a00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3015e2e0 0x7f1f3014f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.297+0000 7f1f35400640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 0x7f1f300c7a90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400c3c0 tx=0x7f1f2400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.298+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24017070 con 0x7f1f300d2b40 2024-03-06T07:30:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.298+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f240035f0 con 0x7f1f300d2b40 2024-03-06T07:30:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.298+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f240039e0 con 0x7f1f300d2b40 2024-03-06T07:30:07.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.298+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f300c85a0 con 0x7f1f300d2b40 2024-03-06T07:30:07.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.298+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3014ee10 con 0x7f1f300d2b40 2024-03-06T07:30:07.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.299+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3014f010 con 0x7f1f300d2b40 2024-03-06T07:30:07.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.300+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f2401a070 con 0x7f1f300d2b40 2024-03-06T07:30:07.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.301+0000 7f1f2d600640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f1006b4d0 0x7f1f1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.301+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1f240891a0 con 0x7f1f300d2b40 2024-03-06T07:30:07.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.301+0000 7f1f34a00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f1006b4d0 0x7f1f1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.302+0000 7f1f34a00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f1006b4d0 0x7f1f1006d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c036000 tx=0x7f1f1c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.305+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f3014f010 con 0x7f1f300d2b40 2024-03-06T07:30:07.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.581+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f1f300b5af0 con 0x7f1f300d2b40 2024-03-06T07:30:07.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.585+0000 7f1f2d600640 1 -- 172.21.15.32:0/263642837 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1f24058400 con 0x7f1f300d2b40 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f1006b4d0 msgr2=0x7f1f1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f1006b4d0 0x7f1f1006d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c036000 tx=0x7f1f1c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 msgr2=0x7f1f300c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 0x7f1f300c7a90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400c3c0 tx=0x7f1f2400c880 comp rx=0 tx=0).stop 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 shutdown_connections 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f1006b4d0 0x7f1f1006d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f3014f870 0x7f1f3014e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3015e2e0 0x7f1f3014f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 --2- 172.21.15.32:0/263642837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300d2b40 0x7f1f300c7a90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 >> 172.21.15.32:0/263642837 conn(0x7f1f300d3f60 msgr2=0x7f1f300b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 shutdown_connections 2024-03-06T07:30:07.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.588+0000 7f1f37e00640 1 -- 172.21.15.32:0/263642837 wait complete. 2024-03-06T07:30:07.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.831+0000 7f54d0800640 1 Processor -- start 2024-03-06T07:30:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.831+0000 7f54d0800640 1 -- start start 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.832+0000 7f54d0800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54c80af580 0x7f54c815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.832+0000 7f54d0800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c815cc40 con 0x7f54c80cdc20 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.832+0000 7f54d0800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c815cf20 con 0x7f54c80af1a0 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.832+0000 7f54d0800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c815d200 con 0x7f54c80af580 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.833+0000 7f54cd400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54c80af580 0x7f54c815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.833+0000 7f54ce800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f54c80cdc20 0x7f54c80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34266/0 (socket says 172.21.15.32:34266) 2024-03-06T07:30:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.833+0000 7f54ce800640 1 -- 172.21.15.32:0/2914041217 learned_addr learned my addr 172.21.15.32:0/2914041217 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.833+0000 7f54cd400640 1 -- 172.21.15.32:0/2914041217 >> v1:172.21.15.39:6789/0 conn(0x7f54c80af1a0 legacy=0x7f54c80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.833+0000 7f54cd400640 1 -- 172.21.15.32:0/2914041217 >> v1:172.21.15.32:6789/0 conn(0x7f54c80cdc20 legacy=0x7f54c80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.833+0000 7f54cd400640 1 -- 172.21.15.32:0/2914041217 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c815d4e0 con 0x7f54c80af580 2024-03-06T07:30:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.835+0000 7f54cd400640 1 --2- 172.21.15.32:0/2914041217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80af580 0x7f54c815c600 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f54b80097a0 tx=0x7f54b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6e57c778d2e332 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.835+0000 7f54cca00640 1 -- 172.21.15.32:0/2914041217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b80315a0 con 0x7f54c80af580 2024-03-06T07:30:07.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.835+0000 7f54cca00640 1 -- 172.21.15.32:0/2914041217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8037070 con 0x7f54c80af580 2024-03-06T07:30:07.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.835+0000 7f54cca00640 1 -- 172.21.15.32:0/2914041217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8031960 con 0x7f54c80af580 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.835+0000 7f54d0800640 1 -- 172.21.15.32:0/2914041217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80af580 msgr2=0x7f54c815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.835+0000 7f54d0800640 1 --2- 172.21.15.32:0/2914041217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80af580 0x7f54c815c600 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f54b80097a0 tx=0x7f54b802b840 comp rx=0 tx=0).stop 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.836+0000 7f54d0800640 1 -- 172.21.15.32:0/2914041217 shutdown_connections 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.836+0000 7f54d0800640 1 --2- 172.21.15.32:0/2914041217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80af580 0x7f54c815c600 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.836+0000 7f54d0800640 1 -- 172.21.15.32:0/2914041217 >> 172.21.15.32:0/2914041217 conn(0x7f54c806afb0 msgr2=0x7f54c806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.836+0000 7f54d0800640 1 -- 172.21.15.32:0/2914041217 shutdown_connections 2024-03-06T07:30:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.836+0000 7f54d0800640 1 -- 172.21.15.32:0/2914041217 wait complete. 2024-03-06T07:30:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.837+0000 7f54d0800640 1 Processor -- start 2024-03-06T07:30:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.838+0000 7f54d0800640 1 -- start start 2024-03-06T07:30:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54d0800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c80af1a0 0x7f54c8147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54d0800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54c80af580 0x7f54c809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54d0800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 0x7f54c806d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54d0800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8155e70 con 0x7f54c80cdc20 2024-03-06T07:30:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54d0800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80b3770 con 0x7f54c80af1a0 2024-03-06T07:30:07.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54d0800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8155ff0 con 0x7f54c80af580 2024-03-06T07:30:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 0x7f54c806d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 0x7f54c806d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53492/0 (socket says 172.21.15.32:53492) 2024-03-06T07:30:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 -- 172.21.15.32:0/2849306877 learned_addr learned my addr 172.21.15.32:0/2849306877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54cde00640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c80af1a0 0x7f54c8147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 -- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54c80af580 msgr2=0x7f54c809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54c80af580 0x7f54c809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 -- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c80af1a0 msgr2=0x7f54c8147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c80af1a0 0x7f54c8147700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54ce800640 1 -- 172.21.15.32:0/2849306877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c806cb10 con 0x7f54c80cdc20 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.839+0000 7f54cde00640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c80af1a0 0x7f54c8147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.840+0000 7f54ce800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 0x7f54c806d890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f54c400ca80 tx=0x7f54c400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.840+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4017070 con 0x7f54c80cdc20 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.840+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c400ac30 con 0x7f54c80cdc20 2024-03-06T07:30:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.840+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4012600 con 0x7f54c80cdc20 2024-03-06T07:30:07.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.840+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c806ea10 con 0x7f54c80cdc20 2024-03-06T07:30:07.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.840+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c806d090 con 0x7f54c80cdc20 2024-03-06T07:30:07.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.841+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c8069510 con 0x7f54c80cdc20 2024-03-06T07:30:07.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.842+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c401a070 con 0x7f54c80cdc20 2024-03-06T07:30:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.844+0000 7f54c2000640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b500 0x7f54ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.844+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54c4089a50 con 0x7f54c80cdc20 2024-03-06T07:30:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.844+0000 7f54cde00640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b500 0x7f54ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.845+0000 7f54cde00640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b500 0x7f54ac06d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f54bc003eb0 tx=0x7f54bc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:07.848+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54c400ae20 con 0x7f54c80cdc20 2024-03-06T07:30:08.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.048+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54c809dee0 con 0x7f54c80cdc20 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.050+0000 7f54c2000640 1 -- 172.21.15.32:0/2849306877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12783 (secure 0 0 0) 0x7f54c40585c0 con 0x7f54c80cdc20 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:03.852858+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:08.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b500 msgr2=0x7f54ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b500 0x7f54ac06d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f54bc003eb0 tx=0x7f54bc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 msgr2=0x7f54c806d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 0x7f54c806d890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f54c400ca80 tx=0x7f54c400cf40 comp rx=0 tx=0).stop 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 shutdown_connections 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b500 0x7f54ac06d9c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c80cdc20 0x7f54c806d890 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54c80af580 0x7f54c809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 --2- 172.21.15.32:0/2849306877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c80af1a0 0x7f54c8147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 >> 172.21.15.32:0/2849306877 conn(0x7f54c806afb0 msgr2=0x7f54c806b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 shutdown_connections 2024-03-06T07:30:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.053+0000 7f54d0800640 1 -- 172.21.15.32:0/2849306877 wait complete. 2024-03-06T07:30:08.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:08.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.295+0000 7f48fd600640 1 Processor -- start 2024-03-06T07:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.296+0000 7f48fd600640 1 -- start start 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fd600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f815dee0 0x7f48f80d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fd600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f48f80d4860 0x7f48f80d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fd600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48f80cfa60 0x7f48f80d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fd600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f80cf5d0 con 0x7f48f80cfa60 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fd600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f80ceca0 con 0x7f48f80d4860 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fd600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f80cef80 con 0x7f48f815dee0 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.297+0000 7f48fcc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48f80cfa60 0x7f48f80d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48f80cfa60 0x7f48f80d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49578/0 (socket says 172.21.15.32:49578) 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 -- 172.21.15.32:0/4042613344 learned_addr learned my addr 172.21.15.32:0/4042613344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48f7400640 1 --2- 172.21.15.32:0/4042613344 >> v2:172.21.15.39:3300/0 conn(0x7f48f80d4860 0x7f48f80d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48f7e00640 1 --2- 172.21.15.32:0/4042613344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f815dee0 0x7f48f80d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 -- 172.21.15.32:0/4042613344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f815dee0 msgr2=0x7f48f80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 --2- 172.21.15.32:0/4042613344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f815dee0 0x7f48f80d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 -- 172.21.15.32:0/4042613344 >> v2:172.21.15.39:3300/0 conn(0x7f48f80d4860 msgr2=0x7f48f80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 --2- 172.21.15.32:0/4042613344 >> v2:172.21.15.39:3300/0 conn(0x7f48f80d4860 0x7f48f80d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.298+0000 7f48fcc00640 1 -- 172.21.15.32:0/4042613344 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80cf260 con 0x7f48f80cfa60 2024-03-06T07:30:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.299+0000 7f48fcc00640 1 --2- 172.21.15.32:0/4042613344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80cfa60 0x7f48f80d0b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f48f808c9d0 tx=0x7f48f002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd26c3e783b1a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.300+0000 7f48f6a00640 1 -- 172.21.15.32:0/4042613344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0035020 con 0x7f48f80cfa60 2024-03-06T07:30:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.300+0000 7f48f6a00640 1 -- 172.21.15.32:0/4042613344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f002d040 con 0x7f48f80cfa60 2024-03-06T07:30:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.300+0000 7f48f6a00640 1 -- 172.21.15.32:0/4042613344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0031a80 con 0x7f48f80cfa60 2024-03-06T07:30:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.300+0000 7f48fd600640 1 -- 172.21.15.32:0/4042613344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80cfa60 msgr2=0x7f48f80d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.300+0000 7f48fd600640 1 --2- 172.21.15.32:0/4042613344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80cfa60 0x7f48f80d0b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f48f808c9d0 tx=0x7f48f002b900 comp rx=0 tx=0).stop 2024-03-06T07:30:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 -- 172.21.15.32:0/4042613344 shutdown_connections 2024-03-06T07:30:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 --2- 172.21.15.32:0/4042613344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80cfa60 0x7f48f80d0b70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 --2- 172.21.15.32:0/4042613344 >> v2:172.21.15.39:3300/0 conn(0x7f48f80d4860 0x7f48f80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 --2- 172.21.15.32:0/4042613344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f815dee0 0x7f48f80d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 -- 172.21.15.32:0/4042613344 >> 172.21.15.32:0/4042613344 conn(0x7f48f80d3cd0 msgr2=0x7f48f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 -- 172.21.15.32:0/4042613344 shutdown_connections 2024-03-06T07:30:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.301+0000 7f48fd600640 1 -- 172.21.15.32:0/4042613344 wait complete. 2024-03-06T07:30:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.303+0000 7f48fd600640 1 Processor -- start 2024-03-06T07:30:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.303+0000 7f48fd600640 1 -- start start 2024-03-06T07:30:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48fd600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f80cfa60 0x7f48f815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48fd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 0x7f48f80b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48fd600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48f815dee0 0x7f48f80a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48fd600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f80d1de0 con 0x7f48f80d4860 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48fd600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f815e2c0 con 0x7f48f815dee0 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48fd600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f80d1f60 con 0x7f48f80cfa60 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 0x7f48f80b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 0x7f48f80b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49580/0 (socket says 172.21.15.32:49580) 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 -- 172.21.15.32:0/2177169267 learned_addr learned my addr 172.21.15.32:0/2177169267 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 -- 172.21.15.32:0/2177169267 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f80cfa60 msgr2=0x7f48f815bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f80cfa60 0x7f48f815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 -- 172.21.15.32:0/2177169267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48f815dee0 msgr2=0x7f48f80a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48f815dee0 0x7f48f80a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.305+0000 7f48f7400640 1 -- 172.21.15.32:0/2177169267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f809fe40 con 0x7f48f80d4860 2024-03-06T07:30:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.306+0000 7f48f7400640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 0x7f48f80b4af0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007e10 tx=0x7f48e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.306+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8014620 con 0x7f48f80d4860 2024-03-06T07:30:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.306+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e8019070 con 0x7f48f80d4860 2024-03-06T07:30:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.306+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8014a00 con 0x7f48f80d4860 2024-03-06T07:30:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.306+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f815c2e0 con 0x7f48f80d4860 2024-03-06T07:30:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.306+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f80a03c0 con 0x7f48f80d4860 2024-03-06T07:30:08.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.307+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f804f330 con 0x7f48f80d4860 2024-03-06T07:30:08.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.308+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48e8014ba0 con 0x7f48f80d4860 2024-03-06T07:30:08.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.309+0000 7f48efe00640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48d406b4d0 0x7f48d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.309+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f48e808a740 con 0x7f48f80d4860 2024-03-06T07:30:08.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.309+0000 7f48f7e00640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48d406b4d0 0x7f48d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.310+0000 7f48f7e00640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48d406b4d0 0x7f48d406d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006fd0 tx=0x7f48e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.315+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48e8059900 con 0x7f48f80d4860 2024-03-06T07:30:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.518+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48f80c9f30 con 0x7f48f80d4860 2024-03-06T07:30:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.518+0000 7f48efe00640 1 -- 172.21.15.32:0/2177169267 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f48e8059200 con 0x7f48f80d4860 2024-03-06T07:30:08.521 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:30:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48d406b4d0 msgr2=0x7f48d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48d406b4d0 0x7f48d406d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006fd0 tx=0x7f48e0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 msgr2=0x7f48f80b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 0x7f48f80b4af0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007e10 tx=0x7f48e800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 shutdown_connections 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48d406b4d0 0x7f48d406d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48f815dee0 0x7f48f80a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f80d4860 0x7f48f80b4af0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 --2- 172.21.15.32:0/2177169267 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48f80cfa60 0x7f48f815bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 >> 172.21.15.32:0/2177169267 conn(0x7f48f80d3cd0 msgr2=0x7f48f806d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.521+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 shutdown_connections 2024-03-06T07:30:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.522+0000 7f48fd600640 1 -- 172.21.15.32:0/2177169267 wait complete. 2024-03-06T07:30:08.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:08.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.757+0000 7f59dec00640 1 Processor -- start 2024-03-06T07:30:08.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.757+0000 7f59dec00640 1 -- start start 2024-03-06T07:30:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.759+0000 7f59dec00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f59d80cd060 0x7f59d8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.759+0000 7f59dec00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59d8162000 0x7f59d8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.759+0000 7f59dec00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d81627f0 con 0x7f59d815efc0 2024-03-06T07:30:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.759+0000 7f59dec00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d815f3a0 con 0x7f59d8162000 2024-03-06T07:30:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.759+0000 7f59dec00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d806d190 con 0x7f59d80cd060 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59d8162000 0x7f59d8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59d8162000 0x7f59d8070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49586/0 (socket says 172.21.15.32:49586) 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 -- 172.21.15.32:0/678139917 learned_addr learned my addr 172.21.15.32:0/678139917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59dd800640 1 --1- 172.21.15.32:0/678139917 >> v1:172.21.15.32:6789/0 conn(0x7f59d815efc0 0x7f59d80cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53594/0 (socket says 172.21.15.32:53594) 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59dce00640 1 --2- 172.21.15.32:0/678139917 >> v2:172.21.15.39:3300/0 conn(0x7f59d80cd060 0x7f59d8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 -- 172.21.15.32:0/678139917 >> v2:172.21.15.39:3300/0 conn(0x7f59d80cd060 msgr2=0x7f59d8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 --2- 172.21.15.32:0/678139917 >> v2:172.21.15.39:3300/0 conn(0x7f59d80cd060 0x7f59d8161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 -- 172.21.15.32:0/678139917 >> v1:172.21.15.32:6789/0 conn(0x7f59d815efc0 legacy=0x7f59d80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.760+0000 7f59de200640 1 -- 172.21.15.32:0/678139917 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d806c860 con 0x7f59d8162000 2024-03-06T07:30:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.761+0000 7f59de200640 1 --2- 172.21.15.32:0/678139917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d8070c20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f59d80b51d0 tx=0x7f59d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226d61f5ecbf9813 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59d7e00640 1 -- 172.21.15.32:0/678139917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0034070 con 0x7f59d8162000 2024-03-06T07:30:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59d7e00640 1 -- 172.21.15.32:0/678139917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0030860 con 0x7f59d8162000 2024-03-06T07:30:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59d7e00640 1 -- 172.21.15.32:0/678139917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0030c30 con 0x7f59d8162000 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59dec00640 1 -- 172.21.15.32:0/678139917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 msgr2=0x7f59d8070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59dec00640 1 --2- 172.21.15.32:0/678139917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d8070c20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f59d80b51d0 tx=0x7f59d002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59dec00640 1 -- 172.21.15.32:0/678139917 shutdown_connections 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59dec00640 1 --2- 172.21.15.32:0/678139917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d8070c20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59dec00640 1 --2- 172.21.15.32:0/678139917 >> v2:172.21.15.39:3300/0 conn(0x7f59d80cd060 0x7f59d8161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.762+0000 7f59dec00640 1 -- 172.21.15.32:0/678139917 >> 172.21.15.32:0/678139917 conn(0x7f59d806b3d0 msgr2=0x7f59d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.763+0000 7f59dec00640 1 -- 172.21.15.32:0/678139917 shutdown_connections 2024-03-06T07:30:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.763+0000 7f59dec00640 1 -- 172.21.15.32:0/678139917 wait complete. 2024-03-06T07:30:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.764+0000 7f59dec00640 1 Processor -- start 2024-03-06T07:30:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.764+0000 7f59dec00640 1 -- start start 2024-03-06T07:30:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59dec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59d80cd060 0x7f59d8146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59dec00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59d815efc0 0x7f59d80b3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59dec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d80b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59dec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8162470 con 0x7f59d8162000 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59dec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d806ce20 con 0x7f59d80cd060 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59dec00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d806d620 con 0x7f59d815efc0 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.765+0000 7f59de200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d80b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d80b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49602/0 (socket says 172.21.15.32:49602) 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 -- 172.21.15.32:0/2601208568 learned_addr learned my addr 172.21.15.32:0/2601208568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59dd800640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59d80cd060 0x7f59d8146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 -- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59d815efc0 msgr2=0x7f59d80b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59d815efc0 0x7f59d80b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 -- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59d80cd060 msgr2=0x7f59d8146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59d80cd060 0x7f59d8146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 -- 172.21.15.32:0/2601208568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d81e8650 con 0x7f59d8162000 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59dd800640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59d80cd060 0x7f59d8146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.766+0000 7f59de200640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d80b3e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009960 tx=0x7f59d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.767+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0034040 con 0x7f59d8162000 2024-03-06T07:30:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.767+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0039d90 con 0x7f59d8162000 2024-03-06T07:30:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.767+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0043610 con 0x7f59d8162000 2024-03-06T07:30:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.767+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d81e8370 con 0x7f59d8162000 2024-03-06T07:30:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.767+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d815be00 con 0x7f59d8162000 2024-03-06T07:30:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.768+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d804f330 con 0x7f59d8162000 2024-03-06T07:30:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.769+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59d0037070 con 0x7f59d8162000 2024-03-06T07:30:08.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.770+0000 7f59d5600640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f59bc06b4d0 0x7f59bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.770+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f59d00ac010 con 0x7f59d8162000 2024-03-06T07:30:08.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.771+0000 7f59dd800640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f59bc06b4d0 0x7f59bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.771+0000 7f59dd800640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f59bc06b4d0 0x7f59bc06d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f59d80cd000 tx=0x7f59cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.774+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59d003e070 con 0x7f59d8162000 2024-03-06T07:30:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.981+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f59d80af2f0 con 0x7f59d8162000 2024-03-06T07:30:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.982+0000 7f59d5600640 1 -- 172.21.15.32:0/2601208568 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f59d007b2a0 con 0x7f59d8162000 2024-03-06T07:30:08.984 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:08.984 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:30:08.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:08.984 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:30:08.985 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:30:08.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:08.985 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.984+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f59bc06b4d0 msgr2=0x7f59bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.984+0000 7f59dec00640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f59bc06b4d0 0x7f59bc06d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f59d80cd000 tx=0x7f59cc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.984+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 msgr2=0x7f59d80b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.984+0000 7f59dec00640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d80b3e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009960 tx=0x7f59d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 shutdown_connections 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f59bc06b4d0 0x7f59bc06d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d8162000 0x7f59d80b3e40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59d815efc0 0x7f59d80b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 --2- 172.21.15.32:0/2601208568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59d80cd060 0x7f59d8146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 >> 172.21.15.32:0/2601208568 conn(0x7f59d806b3d0 msgr2=0x7f59d8071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 shutdown_connections 2024-03-06T07:30:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:08.985+0000 7f59dec00640 1 -- 172.21.15.32:0/2601208568 wait complete. 2024-03-06T07:30:09.006 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:09.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:09.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.227+0000 7fad9ae00640 1 Processor -- start 2024-03-06T07:30:09.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.227+0000 7fad9ae00640 1 -- start start 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad9ae00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fad94145570 0x7fad940cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad9ae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad9406e200 0x7fad9415c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad9ae00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94071c50 con 0x7fad94071450 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad9ae00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad94071f30 con 0x7fad94145570 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad9ae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad94072210 con 0x7fad9406e200 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad99a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fad94145570 0x7fad940cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad99a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fad94145570 0x7fad940cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49608/0 (socket says 172.21.15.32:49608) 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.229+0000 7fad99a00640 1 -- 172.21.15.32:0/698408662 learned_addr learned my addr 172.21.15.32:0/698408662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.230+0000 7fad99a00640 1 -- 172.21.15.32:0/698408662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad9406e200 msgr2=0x7fad9415c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.230+0000 7fad99a00640 1 --2- 172.21.15.32:0/698408662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad9406e200 0x7fad9415c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.230+0000 7fad99a00640 1 -- 172.21.15.32:0/698408662 >> v1:172.21.15.39:6789/0 conn(0x7fad94071450 legacy=0x7fad9406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.230+0000 7fad99a00640 1 -- 172.21.15.32:0/698408662 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad940724f0 con 0x7fad94145570 2024-03-06T07:30:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.231+0000 7fad99a00640 1 --2- 172.21.15.32:0/698408662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad940cf420 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fad88009870 tx=0x7fad8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711b13aa0ce56967 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.231+0000 7fad93e00640 1 -- 172.21.15.32:0/698408662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad880317c0 con 0x7fad94145570 2024-03-06T07:30:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.231+0000 7fad93e00640 1 -- 172.21.15.32:0/698408662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad88037070 con 0x7fad94145570 2024-03-06T07:30:09.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.231+0000 7fad93e00640 1 -- 172.21.15.32:0/698408662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88031b80 con 0x7fad94145570 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.232+0000 7fad9ae00640 1 -- 172.21.15.32:0/698408662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 msgr2=0x7fad940cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.232+0000 7fad9ae00640 1 --2- 172.21.15.32:0/698408662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad940cf420 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fad88009870 tx=0x7fad8802b860 comp rx=0 tx=0).stop 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.232+0000 7fad9ae00640 1 -- 172.21.15.32:0/698408662 shutdown_connections 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.232+0000 7fad9ae00640 1 --2- 172.21.15.32:0/698408662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad9406e200 0x7fad9415c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.232+0000 7fad9ae00640 1 --2- 172.21.15.32:0/698408662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad940cf420 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.232+0000 7fad9ae00640 1 -- 172.21.15.32:0/698408662 >> 172.21.15.32:0/698408662 conn(0x7fad940b17c0 msgr2=0x7fad940b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.233+0000 7fad9ae00640 1 -- 172.21.15.32:0/698408662 shutdown_connections 2024-03-06T07:30:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.233+0000 7fad9ae00640 1 -- 172.21.15.32:0/698408662 wait complete. 2024-03-06T07:30:09.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.235+0000 7fad9ae00640 1 Processor -- start 2024-03-06T07:30:09.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.235+0000 7fad9ae00640 1 -- start start 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9406e200 0x7fad940abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9ae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad94071450 0x7fad941fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad941fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94071a30 con 0x7fad94145570 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940718b0 con 0x7fad9406e200 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9ae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9415c840 con 0x7fad94071450 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad941fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad941fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49610/0 (socket says 172.21.15.32:49610) 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 -- 172.21.15.32:0/3356734738 learned_addr learned my addr 172.21.15.32:0/3356734738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad99000640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad94071450 0x7fad941fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad99a00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9406e200 0x7fad940abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 -- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad94071450 msgr2=0x7fad941fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad94071450 0x7fad941fb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 -- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9406e200 msgr2=0x7fad940abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9406e200 0x7fad940abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.237+0000 7fad9a400640 1 -- 172.21.15.32:0/3356734738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad940ac540 con 0x7fad94145570 2024-03-06T07:30:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad99a00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9406e200 0x7fad940abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad9a400640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad941fb8c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00b970 tx=0x7fad8c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c019070 con 0x7fad94145570 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c00cb60 con 0x7fad94145570 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c014440 con 0x7fad94145570 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad940ac820 con 0x7fad94145570 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.238+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad940acde0 con 0x7fad94145570 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.240+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9404f330 con 0x7fad94145570 2024-03-06T07:30:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.240+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad8c01c070 con 0x7fad94145570 2024-03-06T07:30:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.241+0000 7fad91600640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad7406b520 0x7fad7406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.241+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad8c089c90 con 0x7fad94145570 2024-03-06T07:30:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.242+0000 7fad99a00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad7406b520 0x7fad7406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.242+0000 7fad99a00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad7406b520 0x7fad7406d9e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fad88036000 tx=0x7fad88008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.246+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad8c058e40 con 0x7fad94145570 2024-03-06T07:30:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:09.448+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fad9406db90 con 0x7fad94145570 2024-03-06T07:30:10.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.008+0000 7fad91600640 1 -- 172.21.15.32:0/3356734738 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7fad8c058740 con 0x7fad94145570 2024-03-06T07:30:10.010 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:10.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad7406b520 msgr2=0x7fad7406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad7406b520 0x7fad7406d9e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fad88036000 tx=0x7fad88008030 comp rx=0 tx=0).stop 2024-03-06T07:30:10.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 msgr2=0x7fad941fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad941fb8c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00b970 tx=0x7fad8c00be30 comp rx=0 tx=0).stop 2024-03-06T07:30:10.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 shutdown_connections 2024-03-06T07:30:10.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad7406b520 0x7fad7406d9e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad94145570 0x7fad941fb8c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad94071450 0x7fad941fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 --2- 172.21.15.32:0/3356734738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9406e200 0x7fad940abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.013+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 >> 172.21.15.32:0/3356734738 conn(0x7fad940b17c0 msgr2=0x7fad940ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.014+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 shutdown_connections 2024-03-06T07:30:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.014+0000 7fad9ae00640 1 -- 172.21.15.32:0/3356734738 wait complete. 2024-03-06T07:30:10.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.269+0000 7ff441e00640 1 Processor -- start 2024-03-06T07:30:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.269+0000 7ff441e00640 1 -- start start 2024-03-06T07:30:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c15e220 0x7ff43c0d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff43c0d1920 0x7ff43c0d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff43c0d2520 0x7ff43c0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0b36b0 con 0x7ff43c0d2520 2024-03-06T07:30:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0cf4a0 con 0x7ff43c0d1920 2024-03-06T07:30:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0cf780 con 0x7ff43c15e220 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff43c0d2520 0x7ff43c0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff43c0d2520 0x7ff43c0d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49612/0 (socket says 172.21.15.32:49612) 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.271+0000 7ff441400640 1 -- 172.21.15.32:0/2597585062 learned_addr learned my addr 172.21.15.32:0/2597585062 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff440a00640 1 --2- 172.21.15.32:0/2597585062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c15e220 0x7ff43c0d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff43be00640 1 --2- 172.21.15.32:0/2597585062 >> v2:172.21.15.39:3300/0 conn(0x7ff43c0d1920 0x7ff43c0d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff441400640 1 -- 172.21.15.32:0/2597585062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c15e220 msgr2=0x7ff43c0d4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff441400640 1 --2- 172.21.15.32:0/2597585062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c15e220 0x7ff43c0d4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff441400640 1 -- 172.21.15.32:0/2597585062 >> v2:172.21.15.39:3300/0 conn(0x7ff43c0d1920 msgr2=0x7ff43c0d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff441400640 1 --2- 172.21.15.32:0/2597585062 >> v2:172.21.15.39:3300/0 conn(0x7ff43c0d1920 0x7ff43c0d1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.272+0000 7ff441400640 1 -- 172.21.15.32:0/2597585062 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c0cfa60 con 0x7ff43c0d2520 2024-03-06T07:30:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.273+0000 7ff441400640 1 --2- 172.21.15.32:0/2597585062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0d0720 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0b6940 tx=0x7ff43402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d8789ec24f00a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.274+0000 7ff43b400640 1 -- 172.21.15.32:0/2597585062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43402c9f0 con 0x7ff43c0d2520 2024-03-06T07:30:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.274+0000 7ff43b400640 1 -- 172.21.15.32:0/2597585062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43402cb90 con 0x7ff43c0d2520 2024-03-06T07:30:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.274+0000 7ff43b400640 1 -- 172.21.15.32:0/2597585062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4340343f0 con 0x7ff43c0d2520 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 -- 172.21.15.32:0/2597585062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 msgr2=0x7ff43c0d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 --2- 172.21.15.32:0/2597585062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0d0720 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0b6940 tx=0x7ff43402be50 comp rx=0 tx=0).stop 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 -- 172.21.15.32:0/2597585062 shutdown_connections 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 --2- 172.21.15.32:0/2597585062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0d0720 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 --2- 172.21.15.32:0/2597585062 >> v2:172.21.15.39:3300/0 conn(0x7ff43c0d1920 0x7ff43c0d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 --2- 172.21.15.32:0/2597585062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c15e220 0x7ff43c0d4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 -- 172.21.15.32:0/2597585062 >> 172.21.15.32:0/2597585062 conn(0x7ff43c0d3d50 msgr2=0x7ff43c0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 -- 172.21.15.32:0/2597585062 shutdown_connections 2024-03-06T07:30:10.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.275+0000 7ff441e00640 1 -- 172.21.15.32:0/2597585062 wait complete. 2024-03-06T07:30:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.277+0000 7ff441e00640 1 Processor -- start 2024-03-06T07:30:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.277+0000 7ff441e00640 1 -- start start 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff441e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c0d1920 0x7ff43c1f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff441e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff441e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff43c15e220 0x7ff43c0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff441e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0d0c60 con 0x7ff43c0d2520 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff441e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0d1d80 con 0x7ff43c15e220 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff441e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0d0de0 con 0x7ff43c0d1920 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.278+0000 7ff43be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49618/0 (socket says 172.21.15.32:49618) 2024-03-06T07:30:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 -- 172.21.15.32:0/312377568 learned_addr learned my addr 172.21.15.32:0/312377568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff440a00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c0d1920 0x7ff43c1f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff441400640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff43c15e220 0x7ff43c0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 -- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c0d1920 msgr2=0x7ff43c1f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c0d1920 0x7ff43c1f7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 -- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff43c15e220 msgr2=0x7ff43c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff43c15e220 0x7ff43c0cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 -- 172.21.15.32:0/312377568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c0b95a0 con 0x7ff43c0d2520 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.279+0000 7ff43be00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0b8b50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff42800d800 tx=0x7ff42800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.280+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42800b6c0 con 0x7ff43c0d2520 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.280+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff428019070 con 0x7ff43c0d2520 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.280+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c0b7b80 con 0x7ff43c0d2520 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.280+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42800ba80 con 0x7ff43c0d2520 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.280+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c1f80a0 con 0x7ff43c0d2520 2024-03-06T07:30:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.281+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c04f330 con 0x7ff43c0d2520 2024-03-06T07:30:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.282+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff42800bc20 con 0x7ff43c0d2520 2024-03-06T07:30:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.283+0000 7ff438c00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff41c06b500 0x7ff41c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.283+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff42808a490 con 0x7ff43c0d2520 2024-03-06T07:30:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.283+0000 7ff440a00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff41c06b500 0x7ff41c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.284+0000 7ff440a00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff41c06b500 0x7ff41c06d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff4300099b0 tx=0x7ff430008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.288+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff42801c360 con 0x7ff43c0d2520 2024-03-06T07:30:10.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:10.491+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff43c0d3b30 con 0x7ff43c0d2520 2024-03-06T07:30:11.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.015+0000 7ff438c00640 1 -- 172.21.15.32:0/312377568 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff4280594e0 con 0x7ff43c0d2520 2024-03-06T07:30:11.018 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff41c06b500 msgr2=0x7ff41c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff41c06b500 0x7ff41c06d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff4300099b0 tx=0x7ff430008040 comp rx=0 tx=0).stop 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 msgr2=0x7ff43c0b8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0b8b50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff42800d800 tx=0x7ff42800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 shutdown_connections 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff41c06b500 0x7ff41c06d9c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff43c15e220 0x7ff43c0cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0d2520 0x7ff43c0b8b50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 --2- 172.21.15.32:0/312377568 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff43c0d1920 0x7ff43c1f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.019+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 >> 172.21.15.32:0/312377568 conn(0x7ff43c0d3d50 msgr2=0x7ff43c06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.020+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 shutdown_connections 2024-03-06T07:30:11.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.021+0000 7ff441e00640 1 -- 172.21.15.32:0/312377568 wait complete. 2024-03-06T07:30:11.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:11.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.242+0000 7fc28aa00640 1 Processor -- start 2024-03-06T07:30:11.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.242+0000 7fc28aa00640 1 -- start start 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.244+0000 7fc28aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28409b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.244+0000 7fc28aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2840d1940 0x7fc2840d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.244+0000 7fc28aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2840b63e0 con 0x7fc28406f730 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.244+0000 7fc28aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2840d22e0 con 0x7fc2840d1940 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.244+0000 7fc28aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840b6ff0 con 0x7fc284161520 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.245+0000 7fc283400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2840d1940 0x7fc2840d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.245+0000 7fc283e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28409b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.245+0000 7fc283400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2840d1940 0x7fc2840d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:52768/0 (socket says 172.21.15.32:52768) 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.245+0000 7fc283400640 1 -- 172.21.15.32:0/2833649695 learned_addr learned my addr 172.21.15.32:0/2833649695 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.245+0000 7fc283e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28409b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:47040/0 (socket says 172.21.15.32:47040) 2024-03-06T07:30:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.245+0000 7fc288a00640 1 --1- 172.21.15.32:0/2833649695 >> v1:172.21.15.39:6789/0 conn(0x7fc28406f730 0x7fc28406fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:34510/0 (socket says 172.21.15.32:34510) 2024-03-06T07:30:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.246+0000 7fc283400640 1 -- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 msgr2=0x7fc28409b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.246+0000 7fc283400640 1 --2- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28409b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.246+0000 7fc283400640 1 -- 172.21.15.32:0/2833649695 >> v1:172.21.15.39:6789/0 conn(0x7fc28406f730 legacy=0x7fc28406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.246+0000 7fc283400640 1 -- 172.21.15.32:0/2833649695 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840b66c0 con 0x7fc2840d1940 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.247+0000 7fc283400640 1 --2- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2840d1da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0098a0 tx=0x7fc26c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad13ee313ad236f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.248+0000 7fc282a00640 1 -- 172.21.15.32:0/2833649695 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c034070 con 0x7fc2840d1940 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.248+0000 7fc282a00640 1 -- 172.21.15.32:0/2833649695 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26c030840 con 0x7fc2840d1940 2024-03-06T07:30:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.248+0000 7fc282a00640 1 -- 172.21.15.32:0/2833649695 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c030c10 con 0x7fc2840d1940 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.248+0000 7fc28aa00640 1 -- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 msgr2=0x7fc2840d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.248+0000 7fc28aa00640 1 --2- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2840d1da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0098a0 tx=0x7fc26c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.249+0000 7fc28aa00640 1 -- 172.21.15.32:0/2833649695 shutdown_connections 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.249+0000 7fc28aa00640 1 --2- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2840d1da0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.249+0000 7fc28aa00640 1 --2- 172.21.15.32:0/2833649695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28409b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.249+0000 7fc28aa00640 1 -- 172.21.15.32:0/2833649695 >> 172.21.15.32:0/2833649695 conn(0x7fc2840732d0 msgr2=0x7fc28409aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.249+0000 7fc28aa00640 1 -- 172.21.15.32:0/2833649695 shutdown_connections 2024-03-06T07:30:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.249+0000 7fc28aa00640 1 -- 172.21.15.32:0/2833649695 wait complete. 2024-03-06T07:30:11.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.251+0000 7fc28aa00640 1 Processor -- start 2024-03-06T07:30:11.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.251+0000 7fc28aa00640 1 -- start start 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.252+0000 7fc28aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 0x7fc28414e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.252+0000 7fc28aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2841554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.252+0000 7fc28aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.252+0000 7fc28aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840b6c80 con 0x7fc28406f730 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.252+0000 7fc28aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc284161900 con 0x7fc2840d1940 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.252+0000 7fc28aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28406f420 con 0x7fc284161520 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 0x7fc28414e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 0x7fc28414e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49622/0 (socket says 172.21.15.32:49622) 2024-03-06T07:30:11.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 -- 172.21.15.32:0/954025641 learned_addr learned my addr 172.21.15.32:0/954025641 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc288a00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283400640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2841554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 -- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 msgr2=0x7fc28415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28415a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 -- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 msgr2=0x7fc2841554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2841554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283e00640 1 -- 172.21.15.32:0/954025641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284159400 con 0x7fc28406f730 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.253+0000 7fc283400640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2841554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.254+0000 7fc283e00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 0x7fc28414e2f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc274009d10 tx=0x7fc27400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.254+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274018020 con 0x7fc28406f730 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.254+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc274014480 con 0x7fc28406f730 2024-03-06T07:30:11.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.254+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274014830 con 0x7fc28406f730 2024-03-06T07:30:11.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.254+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2841596e0 con 0x7fc28406f730 2024-03-06T07:30:11.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.254+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc284159ca0 con 0x7fc28406f730 2024-03-06T07:30:11.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.256+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28404f330 con 0x7fc28406f730 2024-03-06T07:30:11.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.257+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc274005d10 con 0x7fc28406f730 2024-03-06T07:30:11.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.258+0000 7fc27be00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc26006b4d0 0x7fc26006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.258+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc27408adb0 con 0x7fc28406f730 2024-03-06T07:30:11.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.258+0000 7fc283400640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc26006b4d0 0x7fc26006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.259+0000 7fc283400640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc26006b4d0 0x7fc26006d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc26c036000 tx=0x7fc26c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.263+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc27408f070 con 0x7fc28406f730 2024-03-06T07:30:11.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:11.507+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc2840b3bf0 con 0x7fc28406f730 2024-03-06T07:30:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.068+0000 7fc27be00640 1 -- 172.21.15.32:0/954025641 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc274022070 con 0x7fc28406f730 2024-03-06T07:30:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.076+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc26006b4d0 msgr2=0x7fc26006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.076+0000 7fc28aa00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc26006b4d0 0x7fc26006d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc26c036000 tx=0x7fc26c03b040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.076+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 msgr2=0x7fc28414e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.076+0000 7fc28aa00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 0x7fc28414e2f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc274009d10 tx=0x7fc27400c490 comp rx=0 tx=0).stop 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.077+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 shutdown_connections 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.077+0000 7fc28aa00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc26006b4d0 0x7fc26006d990 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc26c036000 tx=0x7fc26c03b040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.077+0000 7fc28aa00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc284161520 0x7fc28415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.077+0000 7fc28aa00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2840d1940 0x7fc2841554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.077+0000 7fc28aa00640 1 --2- 172.21.15.32:0/954025641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28406f730 0x7fc28414e2f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.077+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 >> 172.21.15.32:0/954025641 conn(0x7fc2840732d0 msgr2=0x7fc2840b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.079+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 shutdown_connections 2024-03-06T07:30:12.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.082+0000 7fc28aa00640 1 -- 172.21.15.32:0/954025641 wait complete. 2024-03-06T07:30:12.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.293+0000 7fac98e00640 1 Processor -- start 2024-03-06T07:30:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.293+0000 7fac98e00640 1 -- start start 2024-03-06T07:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac98e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac94070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac98e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fac940cd060 0x7fac9406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac98e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940cdad0 con 0x7fac940ce850 2024-03-06T07:30:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac98e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac94071500 con 0x7fac940cd060 2024-03-06T07:30:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac98e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac940cdd70 con 0x7fac9415c920 2024-03-06T07:30:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac8aa00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fac940ce850 0x7fac940cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53602/0 (socket says 172.21.15.32:53602) 2024-03-06T07:30:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.295+0000 7fac8aa00640 1 -- 172.21.15.32:0/260942703 learned_addr learned my addr 172.21.15.32:0/260942703 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac93400640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac94070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac93e00640 1 --2- 172.21.15.32:0/260942703 >> v2:172.21.15.39:3300/0 conn(0x7fac940cd060 0x7fac9406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405667745 0 0) 0x7fac940cdad0 con 0x7fac940ce850 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac78003520 con 0x7fac940ce850 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1614510289 0 0) 0x7fac78003520 con 0x7fac940ce850 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 msgr2=0x7fac94070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac94070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 >> v2:172.21.15.39:3300/0 conn(0x7fac940cd060 msgr2=0x7fac9406cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 --2- 172.21.15.32:0/260942703 >> v2:172.21.15.39:3300/0 conn(0x7fac940cd060 0x7fac9406cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940ce050 con 0x7fac940ce850 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fac7c001670 con 0x7fac940ce850 2024-03-06T07:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 >> v1:172.21.15.32:6789/0 conn(0x7fac940ce850 legacy=0x7fac940cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac78003f80 0x7fac78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94071500 con 0x7fac78004d40 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac940cdd70 con 0x7fac78003f80 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.296+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7800b9c0 con 0x7fac9415c920 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93e00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93400640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93e00640 1 -- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 msgr2=0x7fac780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93e00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93e00640 1 -- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac78003f80 msgr2=0x7fac78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93e00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac78003f80 0x7fac78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93e00640 1 -- 172.21.15.32:0/260942703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78003520 con 0x7fac78004d40 2024-03-06T07:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.297+0000 7fac93400640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac9415c920 0x7fac780039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.298+0000 7fac93e00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac780051c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00e170 tx=0x7fac78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53cfac797f75b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.298+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c039070 con 0x7fac78004d40 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c02cc20 con 0x7fac78004d40 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac92a00640 1 -- 172.21.15.32:0/260942703 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c009c10 con 0x7fac78004d40 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac98e00640 1 -- 172.21.15.32:0/260942703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 msgr2=0x7fac780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac98e00640 1 --2- 172.21.15.32:0/260942703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac780051c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00e170 tx=0x7fac78005830 comp rx=0 tx=0).stop 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac93e00640 1 -- 172.21.15.32:0/260942703 reap_dead start 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac98e00640 1 -- 172.21.15.32:0/260942703 shutdown_connections 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac98e00640 1 -- 172.21.15.32:0/260942703 >> 172.21.15.32:0/260942703 conn(0x7fac9406b3d0 msgr2=0x7fac9406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac98e00640 1 -- 172.21.15.32:0/260942703 shutdown_connections 2024-03-06T07:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.299+0000 7fac98e00640 1 -- 172.21.15.32:0/260942703 wait complete. 2024-03-06T07:30:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.301+0000 7fac98e00640 1 Processor -- start 2024-03-06T07:30:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.301+0000 7fac98e00640 1 -- start start 2024-03-06T07:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac98e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac941401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac98e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac94146b90 0x7fac941429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac98e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9415f970 0x7fac9415fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac98e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94070a40 con 0x7fac78004d40 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac98e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94163790 con 0x7fac9415f970 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac98e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94070bc0 con 0x7fac94146b90 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.302+0000 7fac93400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac941401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac941401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49646/0 (socket says 172.21.15.32:49646) 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 -- 172.21.15.32:0/4000975328 learned_addr learned my addr 172.21.15.32:0/4000975328 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac8aa00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac94146b90 0x7fac941429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9415f970 0x7fac9415fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 -- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac94146b90 msgr2=0x7fac941429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac94146b90 0x7fac941429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 -- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9415f970 msgr2=0x7fac9415fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9415f970 0x7fac9415fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 -- 172.21.15.32:0/4000975328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94143220 con 0x7fac78004d40 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.303+0000 7fac93400640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac941401c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac840108d0 tx=0x7fac84010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.304+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8400f040 con 0x7fac78004d40 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.304+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8400eb00 con 0x7fac78004d40 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.304+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8401c400 con 0x7fac78004d40 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.304+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac94143500 con 0x7fac78004d40 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.304+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9415f450 con 0x7fac78004d40 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.305+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9404f330 con 0x7fac78004d40 2024-03-06T07:30:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.306+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac8401c5a0 con 0x7fac78004d40 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.308+0000 7fac8be00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fac7006b4d0 0x7fac7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.308+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac8408a9b0 con 0x7fac78004d40 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.308+0000 7fac8aa00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fac7006b4d0 0x7fac7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.309+0000 7fac8aa00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fac7006b4d0 0x7fac7006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac7c007a50 tx=0x7fac7c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.313+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac84018030 con 0x7fac78004d40 2024-03-06T07:30:12.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.515+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac940cf3a0 con 0x7fac78004d40 2024-03-06T07:30:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.518+0000 7fac8be00640 1 -- 172.21.15.32:0/4000975328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16574 (secure 0 0 0) 0x7fac84059790 con 0x7fac78004d40 2024-03-06T07:30:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:12.521 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:11.509233+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fac7006b4d0 msgr2=0x7fac7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fac7006b4d0 0x7fac7006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac7c007a50 tx=0x7fac7c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 msgr2=0x7fac941401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac941401c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac840108d0 tx=0x7fac84010d90 comp rx=0 tx=0).stop 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 shutdown_connections 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fac7006b4d0 0x7fac7006d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9415f970 0x7fac9415fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fac94146b90 0x7fac941429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 --2- 172.21.15.32:0/4000975328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac78004d40 0x7fac941401c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 >> 172.21.15.32:0/4000975328 conn(0x7fac9406b3d0 msgr2=0x7fac94163910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 shutdown_connections 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.521+0000 7fac98e00640 1 -- 172.21.15.32:0/4000975328 wait complete. 2024-03-06T07:30:12.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.763+0000 7fb2a1600640 1 Processor -- start 2024-03-06T07:30:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.763+0000 7fb2a1600640 1 -- start start 2024-03-06T07:30:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.765+0000 7fb2a1600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 0x7fb29c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.765+0000 7fb2a1600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0714d0 con 0x7fb29c070f10 2024-03-06T07:30:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.765+0000 7fb2a1600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0717b0 con 0x7fb29c0cd060 2024-03-06T07:30:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.765+0000 7fb2a1600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c071a90 con 0x7fb29c145300 2024-03-06T07:30:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.765+0000 7fb29b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb29c070f10 0x7fb29c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53618/0 (socket says 172.21.15.32:53618) 2024-03-06T07:30:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.765+0000 7fb29b400640 1 -- 172.21.15.32:0/4224067413 learned_addr learned my addr 172.21.15.32:0/4224067413 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29be00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 0x7fb29c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346717278 0 0) 0x7fb29c0714d0 con 0x7fb29c070f10 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb280003520 con 0x7fb29c070f10 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126829404 0 0) 0x7fb280003520 con 0x7fb29c070f10 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 msgr2=0x7fb29c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 0x7fb29c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 >> v1:172.21.15.39:6789/0 conn(0x7fb29c0cd060 legacy=0x7fb29c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c071d70 con 0x7fb29c070f10 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb284001670 con 0x7fb29c070f10 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.766+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 >> v1:172.21.15.32:6789/0 conn(0x7fb29c070f10 legacy=0x7fb29c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29aa00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2800039e0 0x7fb280003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29aa00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 0x7fb280004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29aa00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb280004d40 0x7fb2800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c071a90 con 0x7fb280004d40 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28000baa0 con 0x7fb2800039e0 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28000bca0 con 0x7fb29c145300 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb2a0c00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb280004d40 0x7fb2800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29be00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2800039e0 0x7fb280003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.767+0000 7fb29b400640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 0x7fb280004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.768+0000 7fb2a0c00640 1 -- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 msgr2=0x7fb280004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.768+0000 7fb2a0c00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb29c145300 0x7fb280004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.768+0000 7fb2a0c00640 1 -- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2800039e0 msgr2=0x7fb280003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.768+0000 7fb2a0c00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2800039e0 0x7fb280003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.768+0000 7fb2a0c00640 1 -- 172.21.15.32:0/4224067413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb280003520 con 0x7fb280004d40 2024-03-06T07:30:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.769+0000 7fb2a0c00640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb280004d40 0x7fb2800051c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb294001e20 tx=0x7fb280005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce52187726435b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.769+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294001a70 con 0x7fb280004d40 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.769+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294003b70 con 0x7fb280004d40 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.769+0000 7fb29aa00640 1 -- 172.21.15.32:0/4224067413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940050d0 con 0x7fb280004d40 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a1600640 1 -- 172.21.15.32:0/4224067413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb280004d40 msgr2=0x7fb2800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a1600640 1 --2- 172.21.15.32:0/4224067413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb280004d40 0x7fb2800051c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb294001e20 tx=0x7fb280005830 comp rx=0 tx=0).stop 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a0c00640 1 -- 172.21.15.32:0/4224067413 reap_dead start 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a1600640 1 -- 172.21.15.32:0/4224067413 shutdown_connections 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a1600640 1 -- 172.21.15.32:0/4224067413 >> 172.21.15.32:0/4224067413 conn(0x7fb29c06b3d0 msgr2=0x7fb29c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a1600640 1 -- 172.21.15.32:0/4224067413 shutdown_connections 2024-03-06T07:30:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.770+0000 7fb2a1600640 1 -- 172.21.15.32:0/4224067413 wait complete. 2024-03-06T07:30:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.772+0000 7fb2a1600640 1 Processor -- start 2024-03-06T07:30:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.772+0000 7fb2a1600640 1 -- start start 2024-03-06T07:30:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a1600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb280004d40 0x7fb29c12fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 0x7fb29c1305c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb29c12e2b0 0x7fb29c12e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a1600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0b4b20 con 0x7fb29c130160 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0b49a0 con 0x7fb29c12e2b0 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a1600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0b4ca0 con 0x7fb280004d40 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb29b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 0x7fb29c1305c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb29b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 0x7fb29c1305c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49660/0 (socket says 172.21.15.32:49660) 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb29b400640 1 -- 172.21.15.32:0/3330713203 learned_addr learned my addr 172.21.15.32:0/3330713203 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb29be00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb280004d40 0x7fb29c12fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.773+0000 7fb2a0c00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb29c12e2b0 0x7fb29c12e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29b400640 1 -- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb280004d40 msgr2=0x7fb29c12fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29b400640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb280004d40 0x7fb29c12fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29b400640 1 -- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb29c12e2b0 msgr2=0x7fb29c12e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29b400640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb29c12e2b0 0x7fb29c12e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29b400640 1 -- 172.21.15.32:0/3330713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c13ff60 con 0x7fb29c130160 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb2a0c00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb29c12e2b0 0x7fb29c12e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29be00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb280004d40 0x7fb29c12fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb29b400640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 0x7fb29c1305c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb28400d870 tx=0x7fb28400dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284019070 con 0x7fb29c130160 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb284007560 con 0x7fb29c130160 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284004a00 con 0x7fb29c130160 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c140240 con 0x7fb29c130160 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.774+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c140800 con 0x7fb29c130160 2024-03-06T07:30:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.776+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c04f330 con 0x7fb29c130160 2024-03-06T07:30:12.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.777+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb28401c070 con 0x7fb29c130160 2024-03-06T07:30:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.777+0000 7fb293e00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb27806b4d0 0x7fb27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.778+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb284089c50 con 0x7fb29c130160 2024-03-06T07:30:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.778+0000 7fb29be00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb27806b4d0 0x7fb27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.779+0000 7fb29be00640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb27806b4d0 0x7fb27806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009a00 tx=0x7fb28c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.782+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb28404fe10 con 0x7fb29c130160 2024-03-06T07:30:12.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.983+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb29c13d600 con 0x7fb29c130160 2024-03-06T07:30:12.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.984+0000 7fb293e00640 1 -- 172.21.15.32:0/3330713203 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16574 (secure 0 0 0) 0x7fb28408f020 con 0x7fb29c130160 2024-03-06T07:30:12.987 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:12.987 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:11.509233+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb27806b4d0 msgr2=0x7fb27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb27806b4d0 0x7fb27806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009a00 tx=0x7fb28c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 msgr2=0x7fb29c1305c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 0x7fb29c1305c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb28400d870 tx=0x7fb28400dd30 comp rx=0 tx=0).stop 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 shutdown_connections 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb27806b4d0 0x7fb27806d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb29c12e2b0 0x7fb29c12e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb29c130160 0x7fb29c1305c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 --2- 172.21.15.32:0/3330713203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb280004d40 0x7fb29c12fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 >> 172.21.15.32:0/3330713203 conn(0x7fb29c06b3d0 msgr2=0x7fb29c0afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.987+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 shutdown_connections 2024-03-06T07:30:12.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:12.988+0000 7fb2a1600640 1 -- 172.21.15.32:0/3330713203 wait complete. 2024-03-06T07:30:13.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.234+0000 7f2cda800640 1 Processor -- start 2024-03-06T07:30:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.234+0000 7f2cda800640 1 -- start start 2024-03-06T07:30:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cda800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2cd406ca60 0x7f2cd4157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cda800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd40cf650 con 0x7f2cd415f0a0 2024-03-06T07:30:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cda800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd406bb00 con 0x7f2cd40afff0 2024-03-06T07:30:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cda800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd406bde0 con 0x7f2cd406ca60 2024-03-06T07:30:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cd8a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2cd415f0a0 0x7f2cd406c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53632/0 (socket says 172.21.15.32:53632) 2024-03-06T07:30:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cd8a00640 1 -- 172.21.15.32:0/2127690483 learned_addr learned my addr 172.21.15.32:0/2127690483 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028534395 0 0) 0x7f2cd40cf650 con 0x7f2cd415f0a0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cd9e00640 1 --2- 172.21.15.32:0/2127690483 >> v2:172.21.15.39:3300/0 conn(0x7f2cd406ca60 0x7f2cd4157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.237+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cbc003520 con 0x7f2cd415f0a0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 164329576 0 0) 0x7f2cbc003520 con 0x7f2cd415f0a0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 >> v2:172.21.15.39:3300/0 conn(0x7f2cd406ca60 msgr2=0x7f2cd4157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/2127690483 >> v2:172.21.15.39:3300/0 conn(0x7f2cd406ca60 0x7f2cd4157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 >> v1:172.21.15.39:6789/0 conn(0x7f2cd40afff0 legacy=0x7f2cd406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd415e5c0 con 0x7f2cd415f0a0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f2cc0001690 con 0x7f2cd415f0a0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 >> v1:172.21.15.32:6789/0 conn(0x7f2cd415f0a0 legacy=0x7f2cd406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cbc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc004800 0x7f2cbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cbc0051c0 0x7f2cbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd406bde0 con 0x7f2cbc0039b0 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc00c040 con 0x7f2cbc004800 2024-03-06T07:30:13.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc00c240 con 0x7f2cbc0051c0 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.238+0000 7f2cd9400640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cbc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd8a00640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc004800 0x7f2cbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9e00640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cbc0051c0 0x7f2cbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9400640 1 -- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cbc0051c0 msgr2=0x7f2cbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9400640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cbc0051c0 0x7f2cbc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9400640 1 -- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc004800 msgr2=0x7f2cbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9400640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc004800 0x7f2cbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9400640 1 -- 172.21.15.32:0/2127690483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc003520 con 0x7f2cbc0039b0 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9e00640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cbc0051c0 0x7f2cbc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:13.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.239+0000 7f2cd9e00640 1 -- 172.21.15.32:0/2127690483 reap_dead start 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.240+0000 7f2cd9400640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cbc003db0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8003db0 tx=0x7f2cbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5302e7f8fb8cd4b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.240+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8037070 con 0x7f2cbc0039b0 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.240+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc802be50 con 0x7f2cbc0039b0 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.240+0000 7f2cd3e00640 1 -- 172.21.15.32:0/2127690483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8003980 con 0x7f2cbc0039b0 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 -- 172.21.15.32:0/2127690483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 msgr2=0x7f2cbc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cbc003db0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8003db0 tx=0x7f2cbc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:13.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 -- 172.21.15.32:0/2127690483 shutdown_connections 2024-03-06T07:30:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 --2- 172.21.15.32:0/2127690483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cbc003db0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 -- 172.21.15.32:0/2127690483 >> 172.21.15.32:0/2127690483 conn(0x7f2cd40b2940 msgr2=0x7f2cd40b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 -- 172.21.15.32:0/2127690483 shutdown_connections 2024-03-06T07:30:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.241+0000 7f2cda800640 1 -- 172.21.15.32:0/2127690483 wait complete. 2024-03-06T07:30:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.244+0000 7f2cda800640 1 Processor -- start 2024-03-06T07:30:13.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.244+0000 7f2cda800640 1 -- start start 2024-03-06T07:30:13.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.245+0000 7f2cda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cd41f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.245+0000 7f2cda800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cd40cc330 0x7f2cd4160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.245+0000 7f2cda800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd4160bd0 0x7f2cd41f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.245+0000 7f2cda800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd40b03d0 con 0x7f2cbc0039b0 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.245+0000 7f2cda800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd415ec10 con 0x7f2cd4160bd0 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.245+0000 7f2cda800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd40b0550 con 0x7f2cd40cc330 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cd41f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cd41f7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49690/0 (socket says 172.21.15.32:49690) 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 -- 172.21.15.32:0/1354073323 learned_addr learned my addr 172.21.15.32:0/1354073323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 -- 172.21.15.32:0/1354073323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cd40cc330 msgr2=0x7f2cd4160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cd40cc330 0x7f2cd4160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 -- 172.21.15.32:0/1354073323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd4160bd0 msgr2=0x7f2cd41f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd4160bd0 0x7f2cd41f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 -- 172.21.15.32:0/1354073323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd41f4a60 con 0x7f2cbc0039b0 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.246+0000 7f2cd9400640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cd41f7d10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8003ed0 tx=0x7f2cc80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.247+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8042030 con 0x7f2cbc0039b0 2024-03-06T07:30:13.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.247+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc80073f0 con 0x7f2cbc0039b0 2024-03-06T07:30:13.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.247+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8037040 con 0x7f2cbc0039b0 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.247+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd41f4780 con 0x7f2cbc0039b0 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.247+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd41f7510 con 0x7f2cbc0039b0 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.248+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd404f330 con 0x7f2cbc0039b0 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.249+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cc8045070 con 0x7f2cbc0039b0 2024-03-06T07:30:13.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.250+0000 7f2cd1600640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2cb406b4d0 0x7f2cb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.250+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cc80abb30 con 0x7f2cbc0039b0 2024-03-06T07:30:13.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.250+0000 7f2cd8a00640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2cb406b4d0 0x7f2cb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.251+0000 7f2cd8a00640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2cb406b4d0 0x7f2cb406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0000f60 tx=0x7f2cc0000ed0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.254+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2cc807a910 con 0x7f2cbc0039b0 2024-03-06T07:30:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:13.602+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2cd4160ff0 con 0x7f2cbc0039b0 2024-03-06T07:30:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.077+0000 7f2cd1600640 1 -- 172.21.15.32:0/1354073323 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2cc807a210 con 0x7f2cbc0039b0 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.081+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2cb406b4d0 msgr2=0x7f2cb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.081+0000 7f2cda800640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2cb406b4d0 0x7f2cb406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0000f60 tx=0x7f2cc0000ed0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.081+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 msgr2=0x7f2cd41f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.081+0000 7f2cda800640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cd41f7d10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8003ed0 tx=0x7f2cc80036f0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 shutdown_connections 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2cb406b4d0 0x7f2cb406d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd4160bd0 0x7f2cd41f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2cd40cc330 0x7f2cd4160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 --2- 172.21.15.32:0/1354073323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cbc0039b0 0x7f2cd41f7d10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 >> 172.21.15.32:0/1354073323 conn(0x7f2cd40b2940 msgr2=0x7f2cd40ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 shutdown_connections 2024-03-06T07:30:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.082+0000 7f2cda800640 1 -- 172.21.15.32:0/1354073323 wait complete. 2024-03-06T07:30:14.106 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:14.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:14.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.327+0000 7f898a000640 1 Processor -- start 2024-03-06T07:30:14.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.327+0000 7f898a000640 1 -- start start 2024-03-06T07:30:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.328+0000 7f898a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8984161700 0x7f89840b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f898a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f898a000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89840b2c00 con 0x7f89840d4890 2024-03-06T07:30:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f898a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8984161ae0 con 0x7f8984161700 2024-03-06T07:30:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f898a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b2ea0 con 0x7f89840d24f0 2024-03-06T07:30:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f8983e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89840d4890 0x7f89840d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53638/0 (socket says 172.21.15.32:53638) 2024-03-06T07:30:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f8983e00640 1 -- 172.21.15.32:0/4221963739 learned_addr learned my addr 172.21.15.32:0/4221963739 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f8988c00640 1 --2- 172.21.15.32:0/4221963739 >> v2:172.21.15.39:3300/0 conn(0x7f8984161700 0x7f89840b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.329+0000 7f8989600640 1 --2- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.330+0000 7f8983400640 1 -- 172.21.15.32:0/4221963739 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320430338 0 0) 0x7f89840b2c00 con 0x7f89840d4890 2024-03-06T07:30:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.330+0000 7f8983400640 1 -- 172.21.15.32:0/4221963739 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f896c003520 con 0x7f89840d4890 2024-03-06T07:30:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.330+0000 7f8988c00640 1 -- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 msgr2=0x7f89840b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.330+0000 7f8988c00640 1 --2- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.330+0000 7f8988c00640 1 -- 172.21.15.32:0/4221963739 >> v1:172.21.15.32:6789/0 conn(0x7f89840d4890 legacy=0x7f89840d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.330+0000 7f8988c00640 1 -- 172.21.15.32:0/4221963739 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840b3180 con 0x7f8984161700 2024-03-06T07:30:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.332+0000 7f8988c00640 1 --2- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840b4fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f89780097a0 tx=0x7f897802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab4af3d4de733777 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.333+0000 7f8983400640 1 -- 172.21.15.32:0/4221963739 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978035020 con 0x7f8984161700 2024-03-06T07:30:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.333+0000 7f8983400640 1 -- 172.21.15.32:0/4221963739 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8978031810 con 0x7f8984161700 2024-03-06T07:30:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.333+0000 7f8983400640 1 -- 172.21.15.32:0/4221963739 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978031bc0 con 0x7f8984161700 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.333+0000 7f898a000640 1 -- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 msgr2=0x7f89840b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.333+0000 7f898a000640 1 --2- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840b4fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f89780097a0 tx=0x7f897802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.334+0000 7f898a000640 1 -- 172.21.15.32:0/4221963739 shutdown_connections 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.334+0000 7f898a000640 1 --2- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.334+0000 7f898a000640 1 --2- 172.21.15.32:0/4221963739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840b4fd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.334+0000 7f898a000640 1 -- 172.21.15.32:0/4221963739 >> 172.21.15.32:0/4221963739 conn(0x7f89840d3d70 msgr2=0x7f89840d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.334+0000 7f898a000640 1 -- 172.21.15.32:0/4221963739 shutdown_connections 2024-03-06T07:30:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.334+0000 7f898a000640 1 -- 172.21.15.32:0/4221963739 wait complete. 2024-03-06T07:30:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.336+0000 7f898a000640 1 Processor -- start 2024-03-06T07:30:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.337+0000 7f898a000640 1 -- start start 2024-03-06T07:30:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f898a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f898a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 0x7f89840c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f898a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f898a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b5510 con 0x7f89840d4890 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f898a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b0d00 con 0x7f8984161700 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f898a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b5690 con 0x7f89840d24f0 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.338+0000 7f8983e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 0x7f89840c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 0x7f89840c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49706/0 (socket says 172.21.15.32:49706) 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 -- 172.21.15.32:0/1555567719 learned_addr learned my addr 172.21.15.32:0/1555567719 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8988c00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8989600640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 -- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 msgr2=0x7f89840cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 -- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 msgr2=0x7f89840c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 -- 172.21.15.32:0/1555567719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898415f030 con 0x7f89840d4890 2024-03-06T07:30:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8988c00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8989600640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.339+0000 7f8983e00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 0x7f89840c9580 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f897000c430 tx=0x7f897000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.340+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8970017070 con 0x7f89840d4890 2024-03-06T07:30:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.340+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8970005ce0 con 0x7f89840d4890 2024-03-06T07:30:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.340+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89700072f0 con 0x7f89840d4890 2024-03-06T07:30:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.340+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898415f310 con 0x7f89840d4890 2024-03-06T07:30:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.340+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898415f8d0 con 0x7f89840d4890 2024-03-06T07:30:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.341+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8984068ac0 con 0x7f89840d4890 2024-03-06T07:30:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.343+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f897001a070 con 0x7f89840d4890 2024-03-06T07:30:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.344+0000 7f8980c00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f896406b4d0 0x7f896406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.344+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89700896f0 con 0x7f89840d4890 2024-03-06T07:30:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.344+0000 7f8988c00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f896406b4d0 0x7f896406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.345+0000 7f8988c00640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f896406b4d0 0x7f896406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8978036000 tx=0x7f8978008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.349+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8984068ac0 con 0x7f89840d4890 2024-03-06T07:30:14.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:14.592+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f89840cb100 con 0x7f89840d4890 2024-03-06T07:30:14.852 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-06T07:30:14.850+0000 7f0f42800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:15.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.080+0000 7f8980c00640 1 -- 172.21.15.32:0/1555567719 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8970058560 con 0x7f89840d4890 2024-03-06T07:30:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f896406b4d0 msgr2=0x7f896406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f896406b4d0 0x7f896406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8978036000 tx=0x7f8978008030 comp rx=0 tx=0).stop 2024-03-06T07:30:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 msgr2=0x7f89840c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 0x7f89840c9580 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f897000c430 tx=0x7f897000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 shutdown_connections 2024-03-06T07:30:15.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f896406b4d0 0x7f896406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984161700 0x7f89840c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89840d4890 0x7f89840c9580 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 --2- 172.21.15.32:0/1555567719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89840d24f0 0x7f89840cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 >> 172.21.15.32:0/1555567719 conn(0x7f89840d3d70 msgr2=0x7f898406c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 shutdown_connections 2024-03-06T07:30:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.084+0000 7f898a000640 1 -- 172.21.15.32:0/1555567719 wait complete. 2024-03-06T07:30:15.112 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:15.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:15.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.317+0000 7f5a73800640 1 Processor -- start 2024-03-06T07:30:15.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.317+0000 7f5a73800640 1 -- start start 2024-03-06T07:30:15.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a73800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a6c0af580 0x7f5a6c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a73800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a73800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c15cc40 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a73800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c15cf20 con 0x7f5a6c0af580 2024-03-06T07:30:15.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a73800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c15d200 con 0x7f5a6c0cdc20 2024-03-06T07:30:15.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a6be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a6c0af580 0x7f5a6c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a6be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a6c0af580 0x7f5a6c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49712/0 (socket says 172.21.15.32:49712) 2024-03-06T07:30:15.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.319+0000 7f5a6be00640 1 -- 172.21.15.32:0/2713535477 learned_addr learned my addr 172.21.15.32:0/2713535477 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:15.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.320+0000 7f5a71800640 1 --2- 172.21.15.32:0/2713535477 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.320+0000 7f5a6be00640 1 -- 172.21.15.32:0/2713535477 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 msgr2=0x7f5a6c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.320+0000 7f5a6be00640 1 --2- 172.21.15.32:0/2713535477 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.320+0000 7f5a6be00640 1 -- 172.21.15.32:0/2713535477 >> v1:172.21.15.39:6789/0 conn(0x7f5a6c0af1a0 legacy=0x7f5a6c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.320+0000 7f5a6be00640 1 -- 172.21.15.32:0/2713535477 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c15d4e0 con 0x7f5a6c0af580 2024-03-06T07:30:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.321+0000 7f5a6be00640 1 --2- 172.21.15.32:0/2713535477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af580 0x7f5a6c15c600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009870 tx=0x7f5a5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebf318ff9a2cde1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a6b400640 1 -- 172.21.15.32:0/2713535477 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58035020 con 0x7f5a6c0af580 2024-03-06T07:30:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a6b400640 1 -- 172.21.15.32:0/2713535477 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a58031640 con 0x7f5a6c0af580 2024-03-06T07:30:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a6b400640 1 -- 172.21.15.32:0/2713535477 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5802d040 con 0x7f5a6c0af580 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a73800640 1 -- 172.21.15.32:0/2713535477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af580 msgr2=0x7f5a6c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a73800640 1 --2- 172.21.15.32:0/2713535477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af580 0x7f5a6c15c600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009870 tx=0x7f5a5802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a73800640 1 -- 172.21.15.32:0/2713535477 shutdown_connections 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a73800640 1 --2- 172.21.15.32:0/2713535477 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a73800640 1 --2- 172.21.15.32:0/2713535477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af580 0x7f5a6c15c600 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.322+0000 7f5a73800640 1 -- 172.21.15.32:0/2713535477 >> 172.21.15.32:0/2713535477 conn(0x7f5a6c06afb0 msgr2=0x7f5a6c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.323+0000 7f5a73800640 1 -- 172.21.15.32:0/2713535477 shutdown_connections 2024-03-06T07:30:15.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.323+0000 7f5a73800640 1 -- 172.21.15.32:0/2713535477 wait complete. 2024-03-06T07:30:15.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.324+0000 7f5a73800640 1 Processor -- start 2024-03-06T07:30:15.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.324+0000 7f5a73800640 1 -- start start 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.325+0000 7f5a73800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 0x7f5a6c06cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.325+0000 7f5a73800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a6c0af580 0x7f5a6c15bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.325+0000 7f5a73800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c1455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.325+0000 7f5a73800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c0b3720 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.325+0000 7f5a73800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c0b35a0 con 0x7f5a6c0af580 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.325+0000 7f5a73800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c155e70 con 0x7f5a6c0cdc20 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 0x7f5a6c06cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 0x7f5a6c06cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49716/0 (socket says 172.21.15.32:49716) 2024-03-06T07:30:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 -- 172.21.15.32:0/1734659273 learned_addr learned my addr 172.21.15.32:0/1734659273 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:15.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 -- 172.21.15.32:0/1734659273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 msgr2=0x7f5a6c1455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c1455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 -- 172.21.15.32:0/1734659273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a6c0af580 msgr2=0x7f5a6c15bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a6c0af580 0x7f5a6c15bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 -- 172.21.15.32:0/1734659273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c06e610 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.326+0000 7f5a70e00640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 0x7f5a6c06cb10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c400 tx=0x7f5a6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.327+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a600052d0 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.327+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60005490 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.327+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60007260 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.327+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c06e8f0 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.327+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c145b20 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.328+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c069510 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.329+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a60007400 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.330+0000 7f5a68c00640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a4c06b4d0 0x7f5a4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.330+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a600897b0 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.330+0000 7f5a6be00640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a4c06b4d0 0x7f5a4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.331+0000 7f5a6be00640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a4c06b4d0 0x7f5a4c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5a58036000 tx=0x7f5a58030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.335+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a60058630 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.571+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a6c04f3a0 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.572+0000 7f5a68c00640 1 -- 172.21.15.32:0/1734659273 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f5a60057f30 con 0x7f5a6c0af1a0 2024-03-06T07:30:15.574 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:15.574 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:15.574 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:15.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a4c06b4d0 msgr2=0x7f5a4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a4c06b4d0 0x7f5a4c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5a58036000 tx=0x7f5a58030000 comp rx=0 tx=0).stop 2024-03-06T07:30:15.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 msgr2=0x7f5a6c06cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 0x7f5a6c06cb10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c400 tx=0x7f5a6000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 shutdown_connections 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a4c06b4d0 0x7f5a4c06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a6c0cdc20 0x7f5a6c1455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a6c0af580 0x7f5a6c15bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 --2- 172.21.15.32:0/1734659273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a6c0af1a0 0x7f5a6c06cb10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 >> 172.21.15.32:0/1734659273 conn(0x7f5a6c06afb0 msgr2=0x7f5a6c0cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 shutdown_connections 2024-03-06T07:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.575+0000 7f5a73800640 1 -- 172.21.15.32:0/1734659273 wait complete. 2024-03-06T07:30:15.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:15.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.807+0000 7ff2d8800640 1 Processor -- start 2024-03-06T07:30:15.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.807+0000 7ff2d8800640 1 -- start start 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d8800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2d00d2b40 0x7ff2d00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d8800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2d00d1480 0x7ff2d00b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d8800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d00b3940 con 0x7ff2d015e2e0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d8800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d00d4e70 con 0x7ff2d00d2b40 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d8800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d00b3c40 con 0x7ff2d00d1480 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d5400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2d00d2b40 0x7ff2d00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d5e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff2d015e2e0 0x7ff2d00d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53652/0 (socket says 172.21.15.32:53652) 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d5e00640 1 -- 172.21.15.32:0/2953432679 learned_addr learned my addr 172.21.15.32:0/2953432679 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.809+0000 7ff2d6800640 1 --2- 172.21.15.32:0/2953432679 >> v2:172.21.15.39:3300/0 conn(0x7ff2d00d1480 0x7ff2d00b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.810+0000 7ff2d5400640 1 -- 172.21.15.32:0/2953432679 >> v2:172.21.15.39:3300/0 conn(0x7ff2d00d1480 msgr2=0x7ff2d00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.810+0000 7ff2d5400640 1 --2- 172.21.15.32:0/2953432679 >> v2:172.21.15.39:3300/0 conn(0x7ff2d00d1480 0x7ff2d00b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.810+0000 7ff2d5400640 1 -- 172.21.15.32:0/2953432679 >> v1:172.21.15.32:6789/0 conn(0x7ff2d015e2e0 legacy=0x7ff2d00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.810+0000 7ff2d4a00640 1 -- 172.21.15.32:0/2953432679 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759743877 0 0) 0x7ff2d00b3940 con 0x7ff2d015e2e0 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.810+0000 7ff2d5400640 1 -- 172.21.15.32:0/2953432679 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00cf580 con 0x7ff2d00d2b40 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.811+0000 7ff2d5400640 1 --2- 172.21.15.32:0/2953432679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d00d0f40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff2d00ce540 tx=0x7ff2c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb41cf96ee43a7a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.811+0000 7ff2d4a00640 1 -- 172.21.15.32:0/2953432679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0035020 con 0x7ff2d00d2b40 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.811+0000 7ff2d4a00640 1 -- 172.21.15.32:0/2953432679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c0031830 con 0x7ff2d00d2b40 2024-03-06T07:30:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.811+0000 7ff2d4a00640 1 -- 172.21.15.32:0/2953432679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0031c00 con 0x7ff2d00d2b40 2024-03-06T07:30:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 -- 172.21.15.32:0/2953432679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 msgr2=0x7ff2d00d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 --2- 172.21.15.32:0/2953432679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d00d0f40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff2d00ce540 tx=0x7ff2c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 -- 172.21.15.32:0/2953432679 shutdown_connections 2024-03-06T07:30:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 --2- 172.21.15.32:0/2953432679 >> v2:172.21.15.39:3300/0 conn(0x7ff2d00d1480 0x7ff2d00b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 --2- 172.21.15.32:0/2953432679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d00d0f40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 -- 172.21.15.32:0/2953432679 >> 172.21.15.32:0/2953432679 conn(0x7ff2d00d3f60 msgr2=0x7ff2d00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 -- 172.21.15.32:0/2953432679 shutdown_connections 2024-03-06T07:30:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.812+0000 7ff2d8800640 1 -- 172.21.15.32:0/2953432679 wait complete. 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.815+0000 7ff2d8800640 1 Processor -- start 2024-03-06T07:30:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.815+0000 7ff2d8800640 1 -- start start 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.816+0000 7ff2d8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d0153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.816+0000 7ff2d8800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d015e2e0 0x7ff2d00c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.816+0000 7ff2d8800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2d00c7fd0 0x7ff2d00c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.816+0000 7ff2d8800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d00cfe20 con 0x7ff2d00d2b40 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d8800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d00cfb40 con 0x7ff2d015e2e0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d8800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d00b4610 con 0x7ff2d00c7fd0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d0153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d0153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49738/0 (socket says 172.21.15.32:49738) 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 -- 172.21.15.32:0/1609273072 learned_addr learned my addr 172.21.15.32:0/1609273072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d6800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2d00c7fd0 0x7ff2d00c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5400640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d015e2e0 0x7ff2d00c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 -- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2d00c7fd0 msgr2=0x7ff2d00c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2d00c7fd0 0x7ff2d00c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 -- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d015e2e0 msgr2=0x7ff2d00c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d015e2e0 0x7ff2d00c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.817+0000 7ff2d5e00640 1 -- 172.21.15.32:0/1609273072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00c9e20 con 0x7ff2d00d2b40 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.818+0000 7ff2d5e00640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d0153250 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400c430 tx=0x7ff2c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.818+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40074e0 con 0x7ff2d00d2b40 2024-03-06T07:30:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.818+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c40047e0 con 0x7ff2d00d2b40 2024-03-06T07:30:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.818+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4004bd0 con 0x7ff2d00d2b40 2024-03-06T07:30:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.818+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d00ca100 con 0x7ff2d00d2b40 2024-03-06T07:30:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.818+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d00ca6c0 con 0x7ff2d00d2b40 2024-03-06T07:30:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.819+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d004f3a0 con 0x7ff2d00d2b40 2024-03-06T07:30:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.820+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2c4011070 con 0x7ff2d00d2b40 2024-03-06T07:30:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.821+0000 7ff2ca000640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2b406b4d0 0x7ff2b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.821+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2c4089d70 con 0x7ff2d00d2b40 2024-03-06T07:30:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.821+0000 7ff2d5400640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2b406b4d0 0x7ff2b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.822+0000 7ff2d5400640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2b406b4d0 0x7ff2b406d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0036000 tx=0x7ff2c0008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:15.826+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2c4058c80 con 0x7ff2d00d2b40 2024-03-06T07:30:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.025+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2d00ca8c0 con 0x7ff2d00d2b40 2024-03-06T07:30:16.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.028+0000 7ff2ca000640 1 -- 172.21.15.32:0/1609273072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16561 (secure 0 0 0) 0x7ff2c4058580 con 0x7ff2d00d2b40 2024-03-06T07:30:16.031 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:16.031 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:13.040051+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:16.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2b406b4d0 msgr2=0x7ff2b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2b406b4d0 0x7ff2b406d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0036000 tx=0x7ff2c0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:16.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 msgr2=0x7ff2d0153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d0153250 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400c430 tx=0x7ff2c400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 shutdown_connections 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2b406b4d0 0x7ff2b406d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2d00c7fd0 0x7ff2d00c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d015e2e0 0x7ff2d00c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 --2- 172.21.15.32:0/1609273072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d00d2b40 0x7ff2d0153250 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 >> 172.21.15.32:0/1609273072 conn(0x7ff2d00d3f60 msgr2=0x7ff2d00b62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 shutdown_connections 2024-03-06T07:30:16.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.031+0000 7ff2d8800640 1 -- 172.21.15.32:0/1609273072 wait complete. 2024-03-06T07:30:16.051 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:16.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.254+0000 7f5445e00640 1 Processor -- start 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.254+0000 7f5445e00640 1 -- start start 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5445e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f544015e2a0 0x7f54400cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5445e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54400cf460 0x7f54400b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5445e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54400d1120 con 0x7f544015dec0 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5445e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54400b0ea0 con 0x7f544015e2a0 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5445e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400d1400 con 0x7f54400cf460 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5444a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f544015dec0 0x7f54400ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53660/0 (socket says 172.21.15.32:53660) 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f543fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f544015e2a0 0x7f54400cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.256+0000 7f5444a00640 1 -- 172.21.15.32:0/4122996150 learned_addr learned my addr 172.21.15.32:0/4122996150 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.257+0000 7f5445400640 1 --2- 172.21.15.32:0/4122996150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54400cf460 0x7f54400b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.257+0000 7f543fe00640 1 -- 172.21.15.32:0/4122996150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54400cf460 msgr2=0x7f54400b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.257+0000 7f543fe00640 1 --2- 172.21.15.32:0/4122996150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54400cf460 0x7f54400b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.257+0000 7f543fe00640 1 -- 172.21.15.32:0/4122996150 >> v1:172.21.15.32:6789/0 conn(0x7f544015dec0 legacy=0x7f54400ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.257+0000 7f543fe00640 1 -- 172.21.15.32:0/4122996150 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400b0b30 con 0x7f544015e2a0 2024-03-06T07:30:16.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.258+0000 7f543fe00640 1 --2- 172.21.15.32:0/4122996150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544015e2a0 0x7f54400cef20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f542c0097a0 tx=0x7f542c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3050d38553c80a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.258+0000 7f543f400640 1 -- 172.21.15.32:0/4122996150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c035020 con 0x7f544015e2a0 2024-03-06T07:30:16.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.258+0000 7f543f400640 1 -- 172.21.15.32:0/4122996150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542c031810 con 0x7f544015e2a0 2024-03-06T07:30:16.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.258+0000 7f543f400640 1 -- 172.21.15.32:0/4122996150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c031be0 con 0x7f544015e2a0 2024-03-06T07:30:16.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 -- 172.21.15.32:0/4122996150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544015e2a0 msgr2=0x7f54400cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 --2- 172.21.15.32:0/4122996150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544015e2a0 0x7f54400cef20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f542c0097a0 tx=0x7f542c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 -- 172.21.15.32:0/4122996150 shutdown_connections 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 --2- 172.21.15.32:0/4122996150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544015e2a0 0x7f54400cef20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 --2- 172.21.15.32:0/4122996150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54400cf460 0x7f54400b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 -- 172.21.15.32:0/4122996150 >> 172.21.15.32:0/4122996150 conn(0x7f54400d3d40 msgr2=0x7f54400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 -- 172.21.15.32:0/4122996150 shutdown_connections 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.259+0000 7f5445e00640 1 -- 172.21.15.32:0/4122996150 wait complete. 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.262+0000 7f5445e00640 1 Processor -- start 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.262+0000 7f5445e00640 1 -- start start 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 0x7f544015f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f544015dec0 0x7f54400cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f544015e2a0 0x7f54401f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400d0ec0 con 0x7f54400cf460 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400d0d40 con 0x7f544015dec0 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400b1870 con 0x7f544015e2a0 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5444a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 0x7f544015f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5444a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 0x7f544015f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49744/0 (socket says 172.21.15.32:49744) 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5444a00640 1 -- 172.21.15.32:0/38596738 learned_addr learned my addr 172.21.15.32:0/38596738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f543fe00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f544015dec0 0x7f54400cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.264+0000 7f5445400640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f544015e2a0 0x7f54401f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5444a00640 1 -- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f544015e2a0 msgr2=0x7f54401f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5444a00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f544015e2a0 0x7f54401f39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5444a00640 1 -- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f544015dec0 msgr2=0x7f54400cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5444a00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f544015dec0 0x7f54400cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5444a00640 1 -- 172.21.15.32:0/38596738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400cc2f0 con 0x7f54400cf460 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f543fe00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f544015dec0 0x7f54400cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5444a00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 0x7f544015f990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f543400c430 tx=0x7f543400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434017070 con 0x7f54400cf460 2024-03-06T07:30:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434005ce0 con 0x7f54400cf460 2024-03-06T07:30:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340072f0 con 0x7f54400cf460 2024-03-06T07:30:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54400c7ac0 con 0x7f54400cf460 2024-03-06T07:30:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.265+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54400c8080 con 0x7f54400cf460 2024-03-06T07:30:16.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.267+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544004f330 con 0x7f54400cf460 2024-03-06T07:30:16.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.268+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f543401a070 con 0x7f54400cf460 2024-03-06T07:30:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.268+0000 7f543cc00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f542006b4d0 0x7f542006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.269+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5434089640 con 0x7f54400cf460 2024-03-06T07:30:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.269+0000 7f543fe00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f542006b4d0 0x7f542006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.270+0000 7f543fe00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f542006b4d0 0x7f542006d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f542c036000 tx=0x7f542c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.273+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5434058420 con 0x7f54400cf460 2024-03-06T07:30:16.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.471+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54400748b0 con 0x7f542006b4d0 2024-03-06T07:30:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.488+0000 7f543cc00640 1 -- 172.21.15.32:0/38596738 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251953 (secure 0 0 0) 0x7f54400748b0 con 0x7f542006b4d0 2024-03-06T07:30:16.494 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:30:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f542006b4d0 msgr2=0x7f542006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f542006b4d0 0x7f542006d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f542c036000 tx=0x7f542c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 msgr2=0x7f544015f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 0x7f544015f990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f543400c430 tx=0x7f543400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 shutdown_connections 2024-03-06T07:30:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f542006b4d0 0x7f542006d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f544015e2a0 0x7f54401f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f544015dec0 0x7f54400cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 --2- 172.21.15.32:0/38596738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54400cf460 0x7f544015f990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 >> 172.21.15.32:0/38596738 conn(0x7f54400d3d40 msgr2=0x7f54400b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 shutdown_connections 2024-03-06T07:30:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.495+0000 7f5445e00640 1 -- 172.21.15.32:0/38596738 wait complete. 2024-03-06T07:30:16.523 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-06T07:30:16.208185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253584,"kb_used_data":4148,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721136,"statfs":{"total":773094113280,"available":772834443264,"internally_reserved":0,"allocated":4247552,"data_stored":2351624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001908"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187403+0000","last_change":"2024-03-06T07:30:12.092142+0000","last_active":"2024-03-06T07:30:13.187403+0000","last_peered":"2024-03-06T07:30:13.187403+0000","last_clean":"2024-03-06T07:30:13.187403+0000","last_became_active":"2024-03-06T07:30:12.091831+0000","last_became_peered":"2024-03-06T07:30:12.091831+0000","last_unstale":"2024-03-06T07:30:13.187403+0000","last_undegraded":"2024-03-06T07:30:13.187403+0000","last_fullsized":"2024-03-06T07:30:13.187403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:54.482169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184131+0000","last_change":"2024-03-06T07:30:12.088013+0000","last_active":"2024-03-06T07:30:13.184131+0000","last_peered":"2024-03-06T07:30:13.184131+0000","last_clean":"2024-03-06T07:30:13.184131+0000","last_became_active":"2024-03-06T07:30:12.087627+0000","last_became_peered":"2024-03-06T07:30:12.087627+0000","last_unstale":"2024-03-06T07:30:13.184131+0000","last_undegraded":"2024-03-06T07:30:13.184131+0000","last_fullsized":"2024-03-06T07:30:13.184131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:55.808069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054447+0000","last_change":"2024-03-06T07:30:12.083552+0000","last_active":"2024-03-06T07:30:13.054447+0000","last_peered":"2024-03-06T07:30:13.054447+0000","last_clean":"2024-03-06T07:30:13.054447+0000","last_became_active":"2024-03-06T07:30:12.083237+0000","last_became_peered":"2024-03-06T07:30:12.083237+0000","last_unstale":"2024-03-06T07:30:13.054447+0000","last_undegraded":"2024-03-06T07:30:13.054447+0000","last_fullsized":"2024-03-06T07:30:13.054447+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.006663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097932+0000","last_change":"2024-03-06T07:30:12.083371+0000","last_active":"2024-03-06T07:30:13.097932+0000","last_peered":"2024-03-06T07:30:13.097932+0000","last_clean":"2024-03-06T07:30:13.097932+0000","last_became_active":"2024-03-06T07:30:12.083044+0000","last_became_peered":"2024-03-06T07:30:12.083044+0000","last_unstale":"2024-03-06T07:30:13.097932+0000","last_undegraded":"2024-03-06T07:30:13.097932+0000","last_fullsized":"2024-03-06T07:30:13.097932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:41.441047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184326+0000","last_change":"2024-03-06T07:30:12.076729+0000","last_active":"2024-03-06T07:30:13.184326+0000","last_peered":"2024-03-06T07:30:13.184326+0000","last_clean":"2024-03-06T07:30:13.184326+0000","last_became_active":"2024-03-06T07:30:12.076442+0000","last_became_peered":"2024-03-06T07:30:12.076442+0000","last_unstale":"2024-03-06T07:30:13.184326+0000","last_undegraded":"2024-03-06T07:30:13.184326+0000","last_fullsized":"2024-03-06T07:30:13.184326+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:50.215492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182839+0000","last_change":"2024-03-06T07:30:12.091033+0000","last_active":"2024-03-06T07:30:13.182839+0000","last_peered":"2024-03-06T07:30:13.182839+0000","last_clean":"2024-03-06T07:30:13.182839+0000","last_became_active":"2024-03-06T07:30:12.090744+0000","last_became_peered":"2024-03-06T07:30:12.090744+0000","last_unstale":"2024-03-06T07:30:13.182839+0000","last_undegraded":"2024-03-06T07:30:13.182839+0000","last_fullsized":"2024-03-06T07:30:13.182839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:46.635895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052416+0000","last_change":"2024-03-06T07:30:12.086866+0000","last_active":"2024-03-06T07:30:13.052416+0000","last_peered":"2024-03-06T07:30:13.052416+0000","last_clean":"2024-03-06T07:30:13.052416+0000","last_became_active":"2024-03-06T07:30:12.085597+0000","last_became_peered":"2024-03-06T07:30:12.085597+0000","last_unstale":"2024-03-06T07:30:13.052416+0000","last_undegraded":"2024-03-06T07:30:13.052416+0000","last_fullsized":"2024-03-06T07:30:13.052416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:28.009850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.051969+0000","last_change":"2024-03-06T07:30:12.077390+0000","last_active":"2024-03-06T07:30:13.051969+0000","last_peered":"2024-03-06T07:30:13.051969+0000","last_clean":"2024-03-06T07:30:13.051969+0000","last_became_active":"2024-03-06T07:30:12.076937+0000","last_became_peered":"2024-03-06T07:30:12.076937+0000","last_unstale":"2024-03-06T07:30:13.051969+0000","last_undegraded":"2024-03-06T07:30:13.051969+0000","last_fullsized":"2024-03-06T07:30:13.051969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:38.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186817+0000","last_change":"2024-03-06T07:30:12.089583+0000","last_active":"2024-03-06T07:30:13.186817+0000","last_peered":"2024-03-06T07:30:13.186817+0000","last_clean":"2024-03-06T07:30:13.186817+0000","last_became_active":"2024-03-06T07:30:12.089307+0000","last_became_peered":"2024-03-06T07:30:12.089307+0000","last_unstale":"2024-03-06T07:30:13.186817+0000","last_undegraded":"2024-03-06T07:30:13.186817+0000","last_fullsized":"2024-03-06T07:30:13.186817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:39.071964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098255+0000","last_change":"2024-03-06T07:30:12.079454+0000","last_active":"2024-03-06T07:30:13.098255+0000","last_peered":"2024-03-06T07:30:13.098255+0000","last_clean":"2024-03-06T07:30:13.098255+0000","last_became_active":"2024-03-06T07:30:12.079148+0000","last_became_peered":"2024-03-06T07:30:12.079148+0000","last_unstale":"2024-03-06T07:30:13.098255+0000","last_undegraded":"2024-03-06T07:30:13.098255+0000","last_fullsized":"2024-03-06T07:30:13.098255+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:38.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053960+0000","last_change":"2024-03-06T07:30:12.075140+0000","last_active":"2024-03-06T07:30:13.053960+0000","last_peered":"2024-03-06T07:30:13.053960+0000","last_clean":"2024-03-06T07:30:13.053960+0000","last_became_active":"2024-03-06T07:30:12.074190+0000","last_became_peered":"2024-03-06T07:30:12.074190+0000","last_unstale":"2024-03-06T07:30:13.053960+0000","last_undegraded":"2024-03-06T07:30:13.053960+0000","last_fullsized":"2024-03-06T07:30:13.053960+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:39.121566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054857+0000","last_change":"2024-03-06T07:30:12.089961+0000","last_active":"2024-03-06T07:30:13.054857+0000","last_peered":"2024-03-06T07:30:13.054857+0000","last_clean":"2024-03-06T07:30:13.054857+0000","last_became_active":"2024-03-06T07:30:12.089476+0000","last_became_peered":"2024-03-06T07:30:12.089476+0000","last_unstale":"2024-03-06T07:30:13.054857+0000","last_undegraded":"2024-03-06T07:30:13.054857+0000","last_fullsized":"2024-03-06T07:30:13.054857+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:39.121566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184480+0000","last_change":"2024-03-06T07:30:12.081588+0000","last_active":"2024-03-06T07:30:13.184480+0000","last_peered":"2024-03-06T07:30:13.184480+0000","last_clean":"2024-03-06T07:30:13.184480+0000","last_became_active":"2024-03-06T07:30:12.080429+0000","last_became_peered":"2024-03-06T07:30:12.080429+0000","last_unstale":"2024-03-06T07:30:13.184480+0000","last_undegraded":"2024-03-06T07:30:13.184480+0000","last_fullsized":"2024-03-06T07:30:13.184480+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:48.145965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182608+0000","last_change":"2024-03-06T07:30:12.074044+0000","last_active":"2024-03-06T07:30:13.182608+0000","last_peered":"2024-03-06T07:30:13.182608+0000","last_clean":"2024-03-06T07:30:13.182608+0000","last_became_active":"2024-03-06T07:30:12.073709+0000","last_became_peered":"2024-03-06T07:30:12.073709+0000","last_unstale":"2024-03-06T07:30:13.182608+0000","last_undegraded":"2024-03-06T07:30:13.182608+0000","last_fullsized":"2024-03-06T07:30:13.182608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:39.121566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052589+0000","last_change":"2024-03-06T07:30:12.088894+0000","last_active":"2024-03-06T07:30:13.052589+0000","last_peered":"2024-03-06T07:30:13.052589+0000","last_clean":"2024-03-06T07:30:13.052589+0000","last_became_active":"2024-03-06T07:30:12.088557+0000","last_became_peered":"2024-03-06T07:30:12.088557+0000","last_unstale":"2024-03-06T07:30:13.052589+0000","last_undegraded":"2024-03-06T07:30:13.052589+0000","last_fullsized":"2024-03-06T07:30:13.052589+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:25.275202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053249+0000","last_change":"2024-03-06T07:30:12.076817+0000","last_active":"2024-03-06T07:30:13.053249+0000","last_peered":"2024-03-06T07:30:13.053249+0000","last_clean":"2024-03-06T07:30:13.053249+0000","last_became_active":"2024-03-06T07:30:12.076506+0000","last_became_peered":"2024-03-06T07:30:12.076506+0000","last_unstale":"2024-03-06T07:30:13.053249+0000","last_undegraded":"2024-03-06T07:30:13.053249+0000","last_fullsized":"2024-03-06T07:30:13.053249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:55.455402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.055207+0000","last_change":"2024-03-06T07:30:12.093804+0000","last_active":"2024-03-06T07:30:13.055207+0000","last_peered":"2024-03-06T07:30:13.055207+0000","last_clean":"2024-03-06T07:30:13.055207+0000","last_became_active":"2024-03-06T07:30:12.093480+0000","last_became_peered":"2024-03-06T07:30:12.093480+0000","last_unstale":"2024-03-06T07:30:13.055207+0000","last_undegraded":"2024-03-06T07:30:13.055207+0000","last_fullsized":"2024-03-06T07:30:13.055207+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:15.057951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183916+0000","last_change":"2024-03-06T07:30:12.084274+0000","last_active":"2024-03-06T07:30:13.183916+0000","last_peered":"2024-03-06T07:30:13.183916+0000","last_clean":"2024-03-06T07:30:13.183916+0000","last_became_active":"2024-03-06T07:30:12.083921+0000","last_became_peered":"2024-03-06T07:30:12.083921+0000","last_unstale":"2024-03-06T07:30:13.183916+0000","last_undegraded":"2024-03-06T07:30:13.183916+0000","last_fullsized":"2024-03-06T07:30:13.183916+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:54.264504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183172+0000","last_change":"2024-03-06T07:30:12.074907+0000","last_active":"2024-03-06T07:30:13.183172+0000","last_peered":"2024-03-06T07:30:13.183172+0000","last_clean":"2024-03-06T07:30:13.183172+0000","last_became_active":"2024-03-06T07:30:12.074559+0000","last_became_peered":"2024-03-06T07:30:12.074559+0000","last_unstale":"2024-03-06T07:30:13.183172+0000","last_undegraded":"2024-03-06T07:30:13.183172+0000","last_fullsized":"2024-03-06T07:30:13.183172+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:55.262416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184776+0000","last_change":"2024-03-06T07:30:12.096082+0000","last_active":"2024-03-06T07:30:13.184776+0000","last_peered":"2024-03-06T07:30:13.184776+0000","last_clean":"2024-03-06T07:30:13.184776+0000","last_became_active":"2024-03-06T07:30:12.095191+0000","last_became_peered":"2024-03-06T07:30:12.095191+0000","last_unstale":"2024-03-06T07:30:13.184776+0000","last_undegraded":"2024-03-06T07:30:13.184776+0000","last_fullsized":"2024-03-06T07:30:13.184776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:28.260506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184874+0000","last_change":"2024-03-06T07:30:12.098947+0000","last_active":"2024-03-06T07:30:13.184874+0000","last_peered":"2024-03-06T07:30:13.184874+0000","last_clean":"2024-03-06T07:30:13.184874+0000","last_became_active":"2024-03-06T07:30:12.098683+0000","last_became_peered":"2024-03-06T07:30:12.098683+0000","last_unstale":"2024-03-06T07:30:13.184874+0000","last_undegraded":"2024-03-06T07:30:13.184874+0000","last_fullsized":"2024-03-06T07:30:13.184874+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:54.482169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184993+0000","last_change":"2024-03-06T07:30:12.087934+0000","last_active":"2024-03-06T07:30:13.184993+0000","last_peered":"2024-03-06T07:30:13.184993+0000","last_clean":"2024-03-06T07:30:13.184993+0000","last_became_active":"2024-03-06T07:30:12.087621+0000","last_became_peered":"2024-03-06T07:30:12.087621+0000","last_unstale":"2024-03-06T07:30:13.184993+0000","last_undegraded":"2024-03-06T07:30:13.184993+0000","last_fullsized":"2024-03-06T07:30:13.184993+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:09.827964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094764+0000","last_change":"2024-03-06T07:30:12.064609+0000","last_active":"2024-03-06T07:30:13.094764+0000","last_peered":"2024-03-06T07:30:13.094764+0000","last_clean":"2024-03-06T07:30:13.094764+0000","last_became_active":"2024-03-06T07:30:12.064061+0000","last_became_peered":"2024-03-06T07:30:12.064061+0000","last_unstale":"2024-03-06T07:30:13.094764+0000","last_undegraded":"2024-03-06T07:30:13.094764+0000","last_fullsized":"2024-03-06T07:30:13.094764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:24.307467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052776+0000","last_change":"2024-03-06T07:30:12.084650+0000","last_active":"2024-03-06T07:30:13.052776+0000","last_peered":"2024-03-06T07:30:13.052776+0000","last_clean":"2024-03-06T07:30:13.052776+0000","last_became_active":"2024-03-06T07:30:12.084289+0000","last_became_peered":"2024-03-06T07:30:12.084289+0000","last_unstale":"2024-03-06T07:30:13.052776+0000","last_undegraded":"2024-03-06T07:30:13.052776+0000","last_fullsized":"2024-03-06T07:30:13.052776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:44.879274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098519+0000","last_change":"2024-03-06T07:30:12.065829+0000","last_active":"2024-03-06T07:30:13.098519+0000","last_peered":"2024-03-06T07:30:13.098519+0000","last_clean":"2024-03-06T07:30:13.098519+0000","last_became_active":"2024-03-06T07:30:12.064986+0000","last_became_peered":"2024-03-06T07:30:12.064986+0000","last_unstale":"2024-03-06T07:30:13.098519+0000","last_undegraded":"2024-03-06T07:30:13.098519+0000","last_fullsized":"2024-03-06T07:30:13.098519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:25.275202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184058+0000","last_change":"2024-03-06T07:30:12.077437+0000","last_active":"2024-03-06T07:30:13.184058+0000","last_peered":"2024-03-06T07:30:13.184058+0000","last_clean":"2024-03-06T07:30:13.184058+0000","last_became_active":"2024-03-06T07:30:12.076484+0000","last_became_peered":"2024-03-06T07:30:12.076484+0000","last_unstale":"2024-03-06T07:30:13.184058+0000","last_undegraded":"2024-03-06T07:30:13.184058+0000","last_fullsized":"2024-03-06T07:30:13.184058+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:56.958197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098528+0000","last_change":"2024-03-06T07:30:12.087007+0000","last_active":"2024-03-06T07:30:13.098528+0000","last_peered":"2024-03-06T07:30:13.098528+0000","last_clean":"2024-03-06T07:30:13.098528+0000","last_became_active":"2024-03-06T07:30:12.086716+0000","last_became_peered":"2024-03-06T07:30:12.086716+0000","last_unstale":"2024-03-06T07:30:13.098528+0000","last_undegraded":"2024-03-06T07:30:13.098528+0000","last_fullsized":"2024-03-06T07:30:13.098528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:44.879274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054092+0000","last_change":"2024-03-06T07:30:12.083729+0000","last_active":"2024-03-06T07:30:13.054092+0000","last_peered":"2024-03-06T07:30:13.054092+0000","last_clean":"2024-03-06T07:30:13.054092+0000","last_became_active":"2024-03-06T07:30:12.083179+0000","last_became_peered":"2024-03-06T07:30:12.083179+0000","last_unstale":"2024-03-06T07:30:13.054092+0000","last_undegraded":"2024-03-06T07:30:13.054092+0000","last_fullsized":"2024-03-06T07:30:13.054092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:54.174988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185049+0000","last_change":"2024-03-06T07:30:12.088538+0000","last_active":"2024-03-06T07:30:13.185049+0000","last_peered":"2024-03-06T07:30:13.185049+0000","last_clean":"2024-03-06T07:30:13.185049+0000","last_became_active":"2024-03-06T07:30:12.088232+0000","last_became_peered":"2024-03-06T07:30:12.088232+0000","last_unstale":"2024-03-06T07:30:13.185049+0000","last_undegraded":"2024-03-06T07:30:13.185049+0000","last_fullsized":"2024-03-06T07:30:13.185049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:26.622501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052933+0000","last_change":"2024-03-06T07:30:12.086424+0000","last_active":"2024-03-06T07:30:13.052933+0000","last_peered":"2024-03-06T07:30:13.052933+0000","last_clean":"2024-03-06T07:30:13.052933+0000","last_became_active":"2024-03-06T07:30:12.086152+0000","last_became_peered":"2024-03-06T07:30:12.086152+0000","last_unstale":"2024-03-06T07:30:13.052933+0000","last_undegraded":"2024-03-06T07:30:13.052933+0000","last_fullsized":"2024-03-06T07:30:13.052933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:54.174988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186017+0000","last_change":"2024-03-06T07:30:12.087396+0000","last_active":"2024-03-06T07:30:13.186017+0000","last_peered":"2024-03-06T07:30:13.186017+0000","last_clean":"2024-03-06T07:30:13.186017+0000","last_became_active":"2024-03-06T07:30:12.086007+0000","last_became_peered":"2024-03-06T07:30:12.086007+0000","last_unstale":"2024-03-06T07:30:13.186017+0000","last_undegraded":"2024-03-06T07:30:13.186017+0000","last_fullsized":"2024-03-06T07:30:13.186017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:49.101946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.051986+0000","last_change":"2024-03-06T07:30:12.083316+0000","last_active":"2024-03-06T07:30:13.051986+0000","last_peered":"2024-03-06T07:30:13.051986+0000","last_clean":"2024-03-06T07:30:13.051986+0000","last_became_active":"2024-03-06T07:30:12.082701+0000","last_became_peered":"2024-03-06T07:30:12.082701+0000","last_unstale":"2024-03-06T07:30:13.051986+0000","last_undegraded":"2024-03-06T07:30:13.051986+0000","last_fullsized":"2024-03-06T07:30:13.051986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:25.275202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.095161+0000","last_change":"2024-03-06T07:30:12.067352+0000","last_active":"2024-03-06T07:30:13.095161+0000","last_peered":"2024-03-06T07:30:13.095161+0000","last_clean":"2024-03-06T07:30:13.095161+0000","last_became_active":"2024-03-06T07:30:12.066935+0000","last_became_peered":"2024-03-06T07:30:12.066935+0000","last_unstale":"2024-03-06T07:30:13.095161+0000","last_undegraded":"2024-03-06T07:30:13.095161+0000","last_fullsized":"2024-03-06T07:30:13.095161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:26.622501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187458+0000","last_change":"2024-03-06T07:30:11.062503+0000","last_active":"2024-03-06T07:30:13.187458+0000","last_peered":"2024-03-06T07:30:13.187458+0000","last_clean":"2024-03-06T07:30:13.187458+0000","last_became_active":"2024-03-06T07:30:11.062029+0000","last_became_peered":"2024-03-06T07:30:11.062029+0000","last_unstale":"2024-03-06T07:30:13.187458+0000","last_undegraded":"2024-03-06T07:30:13.187458+0000","last_fullsized":"2024-03-06T07:30:13.187458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:47.131571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187314+0000","last_change":"2024-03-06T07:30:12.091817+0000","last_active":"2024-03-06T07:30:13.187314+0000","last_peered":"2024-03-06T07:30:13.187314+0000","last_clean":"2024-03-06T07:30:13.187314+0000","last_became_active":"2024-03-06T07:30:12.091464+0000","last_became_peered":"2024-03-06T07:30:12.091464+0000","last_unstale":"2024-03-06T07:30:13.187314+0000","last_undegraded":"2024-03-06T07:30:13.187314+0000","last_fullsized":"2024-03-06T07:30:13.187314+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:24.307467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.095085+0000","last_change":"2024-03-06T07:30:11.062351+0000","last_active":"2024-03-06T07:30:13.095085+0000","last_peered":"2024-03-06T07:30:13.095085+0000","last_clean":"2024-03-06T07:30:13.095085+0000","last_became_active":"2024-03-06T07:30:11.061981+0000","last_became_peered":"2024-03-06T07:30:11.061981+0000","last_unstale":"2024-03-06T07:30:13.095085+0000","last_undegraded":"2024-03-06T07:30:13.095085+0000","last_fullsized":"2024-03-06T07:30:13.095085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:27.246112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186969+0000","last_change":"2024-03-06T07:30:12.091354+0000","last_active":"2024-03-06T07:30:13.186969+0000","last_peered":"2024-03-06T07:30:13.186969+0000","last_clean":"2024-03-06T07:30:13.186969+0000","last_became_active":"2024-03-06T07:30:12.090978+0000","last_became_peered":"2024-03-06T07:30:12.090978+0000","last_unstale":"2024-03-06T07:30:13.186969+0000","last_undegraded":"2024-03-06T07:30:13.186969+0000","last_fullsized":"2024-03-06T07:30:13.186969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:07.689339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183499+0000","last_change":"2024-03-06T07:30:11.040466+0000","last_active":"2024-03-06T07:30:13.183499+0000","last_peered":"2024-03-06T07:30:13.183499+0000","last_clean":"2024-03-06T07:30:13.183499+0000","last_became_active":"2024-03-06T07:30:11.038262+0000","last_became_peered":"2024-03-06T07:30:11.038262+0000","last_unstale":"2024-03-06T07:30:13.183499+0000","last_undegraded":"2024-03-06T07:30:13.183499+0000","last_fullsized":"2024-03-06T07:30:13.183499+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:53.467774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098532+0000","last_change":"2024-03-06T07:30:12.084328+0000","last_active":"2024-03-06T07:30:13.098532+0000","last_peered":"2024-03-06T07:30:13.098532+0000","last_clean":"2024-03-06T07:30:13.098532+0000","last_became_active":"2024-03-06T07:30:12.084042+0000","last_became_peered":"2024-03-06T07:30:12.084042+0000","last_unstale":"2024-03-06T07:30:13.098532+0000","last_undegraded":"2024-03-06T07:30:13.098532+0000","last_fullsized":"2024-03-06T07:30:13.098532+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:28.009850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184711+0000","last_change":"2024-03-06T07:30:11.050325+0000","last_active":"2024-03-06T07:30:13.184711+0000","last_peered":"2024-03-06T07:30:13.184711+0000","last_clean":"2024-03-06T07:30:13.184711+0000","last_became_active":"2024-03-06T07:30:11.049996+0000","last_became_peered":"2024-03-06T07:30:11.049996+0000","last_unstale":"2024-03-06T07:30:13.184711+0000","last_undegraded":"2024-03-06T07:30:13.184711+0000","last_fullsized":"2024-03-06T07:30:13.184711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:59.879817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053371+0000","last_change":"2024-03-06T07:30:12.084268+0000","last_active":"2024-03-06T07:30:13.053371+0000","last_peered":"2024-03-06T07:30:13.053371+0000","last_clean":"2024-03-06T07:30:13.053371+0000","last_became_active":"2024-03-06T07:30:12.082657+0000","last_became_peered":"2024-03-06T07:30:12.082657+0000","last_unstale":"2024-03-06T07:30:13.053371+0000","last_undegraded":"2024-03-06T07:30:13.053371+0000","last_fullsized":"2024-03-06T07:30:13.053371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.826737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185461+0000","last_change":"2024-03-06T07:30:11.053917+0000","last_active":"2024-03-06T07:30:13.185461+0000","last_peered":"2024-03-06T07:30:13.185461+0000","last_clean":"2024-03-06T07:30:13.185461+0000","last_became_active":"2024-03-06T07:30:11.053573+0000","last_became_peered":"2024-03-06T07:30:11.053573+0000","last_unstale":"2024-03-06T07:30:13.185461+0000","last_undegraded":"2024-03-06T07:30:13.185461+0000","last_fullsized":"2024-03-06T07:30:13.185461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.623776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053886+0000","last_change":"2024-03-06T07:30:12.095550+0000","last_active":"2024-03-06T07:30:13.053886+0000","last_peered":"2024-03-06T07:30:13.053886+0000","last_clean":"2024-03-06T07:30:13.053886+0000","last_became_active":"2024-03-06T07:30:12.095172+0000","last_became_peered":"2024-03-06T07:30:12.095172+0000","last_unstale":"2024-03-06T07:30:13.053886+0000","last_undegraded":"2024-03-06T07:30:13.053886+0000","last_fullsized":"2024-03-06T07:30:13.053886+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:28.009850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184992+0000","last_change":"2024-03-06T07:30:11.053291+0000","last_active":"2024-03-06T07:30:13.184992+0000","last_peered":"2024-03-06T07:30:13.184992+0000","last_clean":"2024-03-06T07:30:13.184992+0000","last_became_active":"2024-03-06T07:30:11.052975+0000","last_became_peered":"2024-03-06T07:30:11.052975+0000","last_unstale":"2024-03-06T07:30:13.184992+0000","last_undegraded":"2024-03-06T07:30:13.184992+0000","last_fullsized":"2024-03-06T07:30:13.184992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:50.694170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184483+0000","last_change":"2024-03-06T07:30:12.093254+0000","last_active":"2024-03-06T07:30:13.184483+0000","last_peered":"2024-03-06T07:30:13.184483+0000","last_clean":"2024-03-06T07:30:13.184483+0000","last_became_active":"2024-03-06T07:30:12.092826+0000","last_became_peered":"2024-03-06T07:30:12.092826+0000","last_unstale":"2024-03-06T07:30:13.184483+0000","last_undegraded":"2024-03-06T07:30:13.184483+0000","last_fullsized":"2024-03-06T07:30:13.184483+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:24.307467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182137+0000","last_change":"2024-03-06T07:30:11.043582+0000","last_active":"2024-03-06T07:30:13.182137+0000","last_peered":"2024-03-06T07:30:13.182137+0000","last_clean":"2024-03-06T07:30:13.182137+0000","last_became_active":"2024-03-06T07:30:11.043153+0000","last_became_peered":"2024-03-06T07:30:11.043153+0000","last_unstale":"2024-03-06T07:30:13.182137+0000","last_undegraded":"2024-03-06T07:30:13.182137+0000","last_fullsized":"2024-03-06T07:30:13.182137+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:25.608107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097794+0000","last_change":"2024-03-06T07:30:12.092416+0000","last_active":"2024-03-06T07:30:13.097794+0000","last_peered":"2024-03-06T07:30:13.097794+0000","last_clean":"2024-03-06T07:30:13.097794+0000","last_became_active":"2024-03-06T07:30:12.092071+0000","last_became_peered":"2024-03-06T07:30:12.092071+0000","last_unstale":"2024-03-06T07:30:13.097794+0000","last_undegraded":"2024-03-06T07:30:13.097794+0000","last_fullsized":"2024-03-06T07:30:13.097794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:39.071964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184329+0000","last_change":"2024-03-06T07:30:11.057214+0000","last_active":"2024-03-06T07:30:13.184329+0000","last_peered":"2024-03-06T07:30:13.184329+0000","last_clean":"2024-03-06T07:30:13.184329+0000","last_became_active":"2024-03-06T07:30:11.056813+0000","last_became_peered":"2024-03-06T07:30:11.056813+0000","last_unstale":"2024-03-06T07:30:13.184329+0000","last_undegraded":"2024-03-06T07:30:13.184329+0000","last_fullsized":"2024-03-06T07:30:13.184329+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:53.660708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054464+0000","last_change":"2024-03-06T07:30:12.092025+0000","last_active":"2024-03-06T07:30:13.054464+0000","last_peered":"2024-03-06T07:30:13.054464+0000","last_clean":"2024-03-06T07:30:13.054464+0000","last_became_active":"2024-03-06T07:30:12.091570+0000","last_became_peered":"2024-03-06T07:30:12.091570+0000","last_unstale":"2024-03-06T07:30:13.054464+0000","last_undegraded":"2024-03-06T07:30:13.054464+0000","last_fullsized":"2024-03-06T07:30:13.054464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.826737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185040+0000","last_change":"2024-03-06T07:30:11.053376+0000","last_active":"2024-03-06T07:30:13.185040+0000","last_peered":"2024-03-06T07:30:13.185040+0000","last_clean":"2024-03-06T07:30:13.185040+0000","last_became_active":"2024-03-06T07:30:11.052938+0000","last_became_peered":"2024-03-06T07:30:11.052938+0000","last_unstale":"2024-03-06T07:30:13.185040+0000","last_undegraded":"2024-03-06T07:30:13.185040+0000","last_fullsized":"2024-03-06T07:30:13.185040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:09.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184743+0000","last_change":"2024-03-06T07:30:12.078292+0000","last_active":"2024-03-06T07:30:13.184743+0000","last_peered":"2024-03-06T07:30:13.184743+0000","last_clean":"2024-03-06T07:30:13.184743+0000","last_became_active":"2024-03-06T07:30:12.078006+0000","last_became_peered":"2024-03-06T07:30:12.078006+0000","last_unstale":"2024-03-06T07:30:13.184743+0000","last_undegraded":"2024-03-06T07:30:13.184743+0000","last_fullsized":"2024-03-06T07:30:13.184743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:11.516071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.095147+0000","last_change":"2024-03-06T07:30:11.051632+0000","last_active":"2024-03-06T07:30:13.095147+0000","last_peered":"2024-03-06T07:30:13.095147+0000","last_clean":"2024-03-06T07:30:13.095147+0000","last_became_active":"2024-03-06T07:30:11.051334+0000","last_became_peered":"2024-03-06T07:30:11.051334+0000","last_unstale":"2024-03-06T07:30:13.095147+0000","last_undegraded":"2024-03-06T07:30:13.095147+0000","last_fullsized":"2024-03-06T07:30:13.095147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:33.067615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182970+0000","last_change":"2024-03-06T07:30:12.072677+0000","last_active":"2024-03-06T07:30:13.182970+0000","last_peered":"2024-03-06T07:30:13.182970+0000","last_clean":"2024-03-06T07:30:13.182970+0000","last_became_active":"2024-03-06T07:30:12.072338+0000","last_became_peered":"2024-03-06T07:30:12.072338+0000","last_unstale":"2024-03-06T07:30:13.182970+0000","last_undegraded":"2024-03-06T07:30:13.182970+0000","last_fullsized":"2024-03-06T07:30:13.182970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:19.821191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054606+0000","last_change":"2024-03-06T07:30:11.062625+0000","last_active":"2024-03-06T07:30:13.054606+0000","last_peered":"2024-03-06T07:30:13.054606+0000","last_clean":"2024-03-06T07:30:13.054606+0000","last_became_active":"2024-03-06T07:30:11.062212+0000","last_became_peered":"2024-03-06T07:30:11.062212+0000","last_unstale":"2024-03-06T07:30:13.054606+0000","last_undegraded":"2024-03-06T07:30:13.054606+0000","last_fullsized":"2024-03-06T07:30:13.054606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:48.087552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054320+0000","last_change":"2024-03-06T07:30:12.088454+0000","last_active":"2024-03-06T07:30:13.054320+0000","last_peered":"2024-03-06T07:30:13.054320+0000","last_clean":"2024-03-06T07:30:13.054320+0000","last_became_active":"2024-03-06T07:30:12.088143+0000","last_became_peered":"2024-03-06T07:30:12.088143+0000","last_unstale":"2024-03-06T07:30:13.054320+0000","last_undegraded":"2024-03-06T07:30:13.054320+0000","last_fullsized":"2024-03-06T07:30:13.054320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:55.455402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183102+0000","last_change":"2024-03-06T07:30:11.049071+0000","last_active":"2024-03-06T07:30:13.183102+0000","last_peered":"2024-03-06T07:30:13.183102+0000","last_clean":"2024-03-06T07:30:13.183102+0000","last_became_active":"2024-03-06T07:30:11.048560+0000","last_became_peered":"2024-03-06T07:30:11.048560+0000","last_unstale":"2024-03-06T07:30:13.183102+0000","last_undegraded":"2024-03-06T07:30:13.183102+0000","last_fullsized":"2024-03-06T07:30:13.183102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:38.057570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097800+0000","last_change":"2024-03-06T07:30:12.087328+0000","last_active":"2024-03-06T07:30:13.097800+0000","last_peered":"2024-03-06T07:30:13.097800+0000","last_clean":"2024-03-06T07:30:13.097800+0000","last_became_active":"2024-03-06T07:30:12.085938+0000","last_became_peered":"2024-03-06T07:30:12.085938+0000","last_unstale":"2024-03-06T07:30:13.097800+0000","last_undegraded":"2024-03-06T07:30:13.097800+0000","last_fullsized":"2024-03-06T07:30:13.097800+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:07.689339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185800+0000","last_change":"2024-03-06T07:30:11.060502+0000","last_active":"2024-03-06T07:30:13.185800+0000","last_peered":"2024-03-06T07:30:13.185800+0000","last_clean":"2024-03-06T07:30:13.185800+0000","last_became_active":"2024-03-06T07:30:11.060208+0000","last_became_peered":"2024-03-06T07:30:11.060208+0000","last_unstale":"2024-03-06T07:30:13.185800+0000","last_undegraded":"2024-03-06T07:30:13.185800+0000","last_fullsized":"2024-03-06T07:30:13.185800+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:00.225758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182844+0000","last_change":"2024-03-06T07:30:12.080691+0000","last_active":"2024-03-06T07:30:13.182844+0000","last_peered":"2024-03-06T07:30:13.182844+0000","last_clean":"2024-03-06T07:30:13.182844+0000","last_became_active":"2024-03-06T07:30:12.080189+0000","last_became_peered":"2024-03-06T07:30:12.080189+0000","last_unstale":"2024-03-06T07:30:13.182844+0000","last_undegraded":"2024-03-06T07:30:13.182844+0000","last_fullsized":"2024-03-06T07:30:13.182844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:15.057951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097802+0000","last_change":"2024-03-06T07:30:11.060615+0000","last_active":"2024-03-06T07:30:13.097802+0000","last_peered":"2024-03-06T07:30:13.097802+0000","last_clean":"2024-03-06T07:30:13.097802+0000","last_became_active":"2024-03-06T07:30:11.060269+0000","last_became_peered":"2024-03-06T07:30:11.060269+0000","last_unstale":"2024-03-06T07:30:13.097802+0000","last_undegraded":"2024-03-06T07:30:13.097802+0000","last_fullsized":"2024-03-06T07:30:13.097802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:27.246112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054288+0000","last_change":"2024-03-06T07:30:12.083007+0000","last_active":"2024-03-06T07:30:13.054288+0000","last_peered":"2024-03-06T07:30:13.054288+0000","last_clean":"2024-03-06T07:30:13.054288+0000","last_became_active":"2024-03-06T07:30:12.082302+0000","last_became_peered":"2024-03-06T07:30:12.082302+0000","last_unstale":"2024-03-06T07:30:13.054288+0000","last_undegraded":"2024-03-06T07:30:13.054288+0000","last_fullsized":"2024-03-06T07:30:13.054288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:46.635895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182208+0000","last_change":"2024-03-06T07:30:11.054496+0000","last_active":"2024-03-06T07:30:13.182208+0000","last_peered":"2024-03-06T07:30:13.182208+0000","last_clean":"2024-03-06T07:30:13.182208+0000","last_became_active":"2024-03-06T07:30:11.053011+0000","last_became_peered":"2024-03-06T07:30:11.053011+0000","last_unstale":"2024-03-06T07:30:13.182208+0000","last_undegraded":"2024-03-06T07:30:13.182208+0000","last_fullsized":"2024-03-06T07:30:13.182208+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:08.813570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186037+0000","last_change":"2024-03-06T07:30:12.085561+0000","last_active":"2024-03-06T07:30:13.186037+0000","last_peered":"2024-03-06T07:30:13.186037+0000","last_clean":"2024-03-06T07:30:13.186037+0000","last_became_active":"2024-03-06T07:30:12.085245+0000","last_became_peered":"2024-03-06T07:30:12.085245+0000","last_unstale":"2024-03-06T07:30:13.186037+0000","last_undegraded":"2024-03-06T07:30:13.186037+0000","last_fullsized":"2024-03-06T07:30:13.186037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:10.325701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.181987+0000","last_change":"2024-03-06T07:30:11.040850+0000","last_active":"2024-03-06T07:30:13.181987+0000","last_peered":"2024-03-06T07:30:13.181987+0000","last_clean":"2024-03-06T07:30:13.181987+0000","last_became_active":"2024-03-06T07:30:11.040578+0000","last_became_peered":"2024-03-06T07:30:11.040578+0000","last_unstale":"2024-03-06T07:30:13.181987+0000","last_undegraded":"2024-03-06T07:30:13.181987+0000","last_fullsized":"2024-03-06T07:30:13.181987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:29.715109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187237+0000","last_change":"2024-03-06T07:30:12.093917+0000","last_active":"2024-03-06T07:30:13.187237+0000","last_peered":"2024-03-06T07:30:13.187237+0000","last_clean":"2024-03-06T07:30:13.187237+0000","last_became_active":"2024-03-06T07:30:12.093396+0000","last_became_peered":"2024-03-06T07:30:12.093396+0000","last_unstale":"2024-03-06T07:30:13.187237+0000","last_undegraded":"2024-03-06T07:30:13.187237+0000","last_fullsized":"2024-03-06T07:30:13.187237+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:41.441047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098579+0000","last_change":"2024-03-06T07:30:11.048996+0000","last_active":"2024-03-06T07:30:13.098579+0000","last_peered":"2024-03-06T07:30:13.098579+0000","last_clean":"2024-03-06T07:30:13.098579+0000","last_became_active":"2024-03-06T07:30:11.048751+0000","last_became_peered":"2024-03-06T07:30:11.048751+0000","last_unstale":"2024-03-06T07:30:13.098579+0000","last_undegraded":"2024-03-06T07:30:13.098579+0000","last_fullsized":"2024-03-06T07:30:13.098579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:47.131571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.055235+0000","last_change":"2024-03-06T07:30:12.086388+0000","last_active":"2024-03-06T07:30:13.055235+0000","last_peered":"2024-03-06T07:30:13.055235+0000","last_clean":"2024-03-06T07:30:13.055235+0000","last_became_active":"2024-03-06T07:30:12.085901+0000","last_became_peered":"2024-03-06T07:30:12.085901+0000","last_unstale":"2024-03-06T07:30:13.055235+0000","last_undegraded":"2024-03-06T07:30:13.055235+0000","last_fullsized":"2024-03-06T07:30:13.055235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:15.057951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185917+0000","last_change":"2024-03-06T07:30:11.056644+0000","last_active":"2024-03-06T07:30:13.185917+0000","last_peered":"2024-03-06T07:30:13.185917+0000","last_clean":"2024-03-06T07:30:13.185917+0000","last_became_active":"2024-03-06T07:30:11.055935+0000","last_became_peered":"2024-03-06T07:30:11.055935+0000","last_unstale":"2024-03-06T07:30:13.185917+0000","last_undegraded":"2024-03-06T07:30:13.185917+0000","last_fullsized":"2024-03-06T07:30:13.185917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:40.646294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097992+0000","last_change":"2024-03-06T07:30:12.083238+0000","last_active":"2024-03-06T07:30:13.097992+0000","last_peered":"2024-03-06T07:30:13.097992+0000","last_clean":"2024-03-06T07:30:13.097992+0000","last_became_active":"2024-03-06T07:30:12.082662+0000","last_became_peered":"2024-03-06T07:30:12.082662+0000","last_unstale":"2024-03-06T07:30:13.097992+0000","last_undegraded":"2024-03-06T07:30:13.097992+0000","last_fullsized":"2024-03-06T07:30:13.097992+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:16.327552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186138+0000","last_change":"2024-03-06T07:30:11.048542+0000","last_active":"2024-03-06T07:30:13.186138+0000","last_peered":"2024-03-06T07:30:13.186138+0000","last_clean":"2024-03-06T07:30:13.186138+0000","last_became_active":"2024-03-06T07:30:11.047908+0000","last_became_peered":"2024-03-06T07:30:11.047908+0000","last_unstale":"2024-03-06T07:30:13.186138+0000","last_undegraded":"2024-03-06T07:30:13.186138+0000","last_fullsized":"2024-03-06T07:30:13.186138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:49.201097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184243+0000","last_change":"2024-03-06T07:30:12.086286+0000","last_active":"2024-03-06T07:30:13.184243+0000","last_peered":"2024-03-06T07:30:13.184243+0000","last_clean":"2024-03-06T07:30:13.184243+0000","last_became_active":"2024-03-06T07:30:12.085976+0000","last_became_peered":"2024-03-06T07:30:12.085976+0000","last_unstale":"2024-03-06T07:30:13.184243+0000","last_undegraded":"2024-03-06T07:30:13.184243+0000","last_fullsized":"2024-03-06T07:30:13.184243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:19.635480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186887+0000","last_change":"2024-03-06T07:30:11.054038+0000","last_active":"2024-03-06T07:30:13.186887+0000","last_peered":"2024-03-06T07:30:13.186887+0000","last_clean":"2024-03-06T07:30:13.186887+0000","last_became_active":"2024-03-06T07:30:11.052117+0000","last_became_peered":"2024-03-06T07:30:11.052117+0000","last_unstale":"2024-03-06T07:30:13.186887+0000","last_undegraded":"2024-03-06T07:30:13.186887+0000","last_fullsized":"2024-03-06T07:30:13.186887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:18.621086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185516+0000","last_change":"2024-03-06T07:30:12.084150+0000","last_active":"2024-03-06T07:30:13.185516+0000","last_peered":"2024-03-06T07:30:13.185516+0000","last_clean":"2024-03-06T07:30:13.185516+0000","last_became_active":"2024-03-06T07:30:12.083829+0000","last_became_peered":"2024-03-06T07:30:12.083829+0000","last_unstale":"2024-03-06T07:30:13.185516+0000","last_undegraded":"2024-03-06T07:30:13.185516+0000","last_fullsized":"2024-03-06T07:30:13.185516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.788458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094982+0000","last_change":"2024-03-06T07:30:11.052430+0000","last_active":"2024-03-06T07:30:13.094982+0000","last_peered":"2024-03-06T07:30:13.094982+0000","last_clean":"2024-03-06T07:30:13.094982+0000","last_became_active":"2024-03-06T07:30:11.052050+0000","last_became_peered":"2024-03-06T07:30:11.052050+0000","last_unstale":"2024-03-06T07:30:13.094982+0000","last_undegraded":"2024-03-06T07:30:13.094982+0000","last_fullsized":"2024-03-06T07:30:13.094982+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:47.131571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184920+0000","last_change":"2024-03-06T07:30:12.092718+0000","last_active":"2024-03-06T07:30:13.184920+0000","last_peered":"2024-03-06T07:30:13.184920+0000","last_clean":"2024-03-06T07:30:13.184920+0000","last_became_active":"2024-03-06T07:30:12.092354+0000","last_became_peered":"2024-03-06T07:30:12.092354+0000","last_unstale":"2024-03-06T07:30:13.184920+0000","last_undegraded":"2024-03-06T07:30:13.184920+0000","last_fullsized":"2024-03-06T07:30:13.184920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:34.082010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186510+0000","last_change":"2024-03-06T07:30:11.060365+0000","last_active":"2024-03-06T07:30:13.186510+0000","last_peered":"2024-03-06T07:30:13.186510+0000","last_clean":"2024-03-06T07:30:13.186510+0000","last_became_active":"2024-03-06T07:30:11.059981+0000","last_became_peered":"2024-03-06T07:30:11.059981+0000","last_unstale":"2024-03-06T07:30:13.186510+0000","last_undegraded":"2024-03-06T07:30:13.186510+0000","last_fullsized":"2024-03-06T07:30:13.186510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:18.621086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185238+0000","last_change":"2024-03-06T07:30:12.089382+0000","last_active":"2024-03-06T07:30:13.185238+0000","last_peered":"2024-03-06T07:30:13.185238+0000","last_clean":"2024-03-06T07:30:13.185238+0000","last_became_active":"2024-03-06T07:30:12.089101+0000","last_became_peered":"2024-03-06T07:30:12.089101+0000","last_unstale":"2024-03-06T07:30:13.185238+0000","last_undegraded":"2024-03-06T07:30:13.185238+0000","last_fullsized":"2024-03-06T07:30:13.185238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:16.491261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184442+0000","last_change":"2024-03-06T07:30:11.049247+0000","last_active":"2024-03-06T07:30:13.184442+0000","last_peered":"2024-03-06T07:30:13.184442+0000","last_clean":"2024-03-06T07:30:13.184442+0000","last_became_active":"2024-03-06T07:30:11.048954+0000","last_became_peered":"2024-03-06T07:30:11.048954+0000","last_unstale":"2024-03-06T07:30:13.184442+0000","last_undegraded":"2024-03-06T07:30:13.184442+0000","last_fullsized":"2024-03-06T07:30:13.184442+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:17.991337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052189+0000","last_change":"2024-03-06T07:30:12.087098+0000","last_active":"2024-03-06T07:30:13.052189+0000","last_peered":"2024-03-06T07:30:13.052189+0000","last_clean":"2024-03-06T07:30:13.052189+0000","last_became_active":"2024-03-06T07:30:12.086608+0000","last_became_peered":"2024-03-06T07:30:12.086608+0000","last_unstale":"2024-03-06T07:30:13.052189+0000","last_undegraded":"2024-03-06T07:30:13.052189+0000","last_fullsized":"2024-03-06T07:30:13.052189+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:44.879274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053057+0000","last_change":"2024-03-06T07:30:11.062485+0000","last_active":"2024-03-06T07:30:13.053057+0000","last_peered":"2024-03-06T07:30:13.053057+0000","last_clean":"2024-03-06T07:30:13.053057+0000","last_became_active":"2024-03-06T07:30:11.062184+0000","last_became_peered":"2024-03-06T07:30:11.062184+0000","last_unstale":"2024-03-06T07:30:13.053057+0000","last_undegraded":"2024-03-06T07:30:13.053057+0000","last_fullsized":"2024-03-06T07:30:13.053057+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:11.774064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184629+0000","last_change":"2024-03-06T07:30:12.095920+0000","last_active":"2024-03-06T07:30:13.184629+0000","last_peered":"2024-03-06T07:30:13.184629+0000","last_clean":"2024-03-06T07:30:13.184629+0000","last_became_active":"2024-03-06T07:30:12.095580+0000","last_became_peered":"2024-03-06T07:30:12.095580+0000","last_unstale":"2024-03-06T07:30:13.184629+0000","last_undegraded":"2024-03-06T07:30:13.184629+0000","last_fullsized":"2024-03-06T07:30:13.184629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:11.554592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098683+0000","last_change":"2024-03-06T07:30:11.052325+0000","last_active":"2024-03-06T07:30:13.098683+0000","last_peered":"2024-03-06T07:30:13.098683+0000","last_clean":"2024-03-06T07:30:13.098683+0000","last_became_active":"2024-03-06T07:30:11.051994+0000","last_became_peered":"2024-03-06T07:30:11.051994+0000","last_unstale":"2024-03-06T07:30:13.098683+0000","last_undegraded":"2024-03-06T07:30:13.098683+0000","last_fullsized":"2024-03-06T07:30:13.098683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:10.540198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.055053+0000","last_change":"2024-03-06T07:30:12.086389+0000","last_active":"2024-03-06T07:30:13.055053+0000","last_peered":"2024-03-06T07:30:13.055053+0000","last_clean":"2024-03-06T07:30:13.055053+0000","last_became_active":"2024-03-06T07:30:12.085492+0000","last_became_peered":"2024-03-06T07:30:12.085492+0000","last_unstale":"2024-03-06T07:30:13.055053+0000","last_undegraded":"2024-03-06T07:30:13.055053+0000","last_fullsized":"2024-03-06T07:30:13.055053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:46.635895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.096789+0000","last_change":"2024-03-06T07:30:11.055929+0000","last_active":"2024-03-06T07:30:13.096789+0000","last_peered":"2024-03-06T07:30:13.096789+0000","last_clean":"2024-03-06T07:30:13.096789+0000","last_became_active":"2024-03-06T07:30:11.055599+0000","last_became_peered":"2024-03-06T07:30:11.055599+0000","last_unstale":"2024-03-06T07:30:13.096789+0000","last_undegraded":"2024-03-06T07:30:13.096789+0000","last_fullsized":"2024-03-06T07:30:13.096789+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:11.774064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094980+0000","last_change":"2024-03-06T07:30:12.069798+0000","last_active":"2024-03-06T07:30:13.094980+0000","last_peered":"2024-03-06T07:30:13.094980+0000","last_clean":"2024-03-06T07:30:13.094980+0000","last_became_active":"2024-03-06T07:30:12.069490+0000","last_became_peered":"2024-03-06T07:30:12.069490+0000","last_unstale":"2024-03-06T07:30:13.094980+0000","last_undegraded":"2024-03-06T07:30:13.094980+0000","last_fullsized":"2024-03-06T07:30:13.094980+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:09.827964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098141+0000","last_change":"2024-03-06T07:30:11.049364+0000","last_active":"2024-03-06T07:30:13.098141+0000","last_peered":"2024-03-06T07:30:13.098141+0000","last_clean":"2024-03-06T07:30:13.098141+0000","last_became_active":"2024-03-06T07:30:11.048543+0000","last_became_peered":"2024-03-06T07:30:11.048543+0000","last_unstale":"2024-03-06T07:30:13.098141+0000","last_undegraded":"2024-03-06T07:30:13.098141+0000","last_fullsized":"2024-03-06T07:30:13.098141+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:15.476867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186726+0000","last_change":"2024-03-06T07:30:12.087714+0000","last_active":"2024-03-06T07:30:13.186726+0000","last_peered":"2024-03-06T07:30:13.186726+0000","last_clean":"2024-03-06T07:30:13.186726+0000","last_became_active":"2024-03-06T07:30:12.086649+0000","last_became_peered":"2024-03-06T07:30:12.086649+0000","last_unstale":"2024-03-06T07:30:13.186726+0000","last_undegraded":"2024-03-06T07:30:13.186726+0000","last_fullsized":"2024-03-06T07:30:13.186726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:30.729503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094986+0000","last_change":"2024-03-06T07:30:11.054159+0000","last_active":"2024-03-06T07:30:13.094986+0000","last_peered":"2024-03-06T07:30:13.094986+0000","last_clean":"2024-03-06T07:30:13.094986+0000","last_became_active":"2024-03-06T07:30:11.053888+0000","last_became_peered":"2024-03-06T07:30:11.053888+0000","last_unstale":"2024-03-06T07:30:13.094986+0000","last_undegraded":"2024-03-06T07:30:13.094986+0000","last_fullsized":"2024-03-06T07:30:13.094986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:10.540198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094866+0000","last_change":"2024-03-06T07:30:12.090265+0000","last_active":"2024-03-06T07:30:13.094866+0000","last_peered":"2024-03-06T07:30:13.094866+0000","last_clean":"2024-03-06T07:30:13.094866+0000","last_became_active":"2024-03-06T07:30:12.089857+0000","last_became_peered":"2024-03-06T07:30:12.089857+0000","last_unstale":"2024-03-06T07:30:13.094866+0000","last_undegraded":"2024-03-06T07:30:13.094866+0000","last_fullsized":"2024-03-06T07:30:13.094866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:54.482169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098169+0000","last_change":"2024-03-06T07:30:11.057332+0000","last_active":"2024-03-06T07:30:13.098169+0000","last_peered":"2024-03-06T07:30:13.098169+0000","last_clean":"2024-03-06T07:30:13.098169+0000","last_became_active":"2024-03-06T07:30:11.056962+0000","last_became_peered":"2024-03-06T07:30:11.056962+0000","last_unstale":"2024-03-06T07:30:13.098169+0000","last_undegraded":"2024-03-06T07:30:13.098169+0000","last_fullsized":"2024-03-06T07:30:13.098169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:33.067615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182217+0000","last_change":"2024-03-06T07:30:12.073473+0000","last_active":"2024-03-06T07:30:13.182217+0000","last_peered":"2024-03-06T07:30:13.182217+0000","last_clean":"2024-03-06T07:30:13.182217+0000","last_became_active":"2024-03-06T07:30:12.073083+0000","last_became_peered":"2024-03-06T07:30:12.073083+0000","last_unstale":"2024-03-06T07:30:13.182217+0000","last_undegraded":"2024-03-06T07:30:13.182217+0000","last_fullsized":"2024-03-06T07:30:13.182217+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.826737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184717+0000","last_change":"2024-03-06T07:30:11.052839+0000","last_active":"2024-03-06T07:30:13.184717+0000","last_peered":"2024-03-06T07:30:13.184717+0000","last_clean":"2024-03-06T07:30:13.184717+0000","last_became_active":"2024-03-06T07:30:11.052473+0000","last_became_peered":"2024-03-06T07:30:11.052473+0000","last_unstale":"2024-03-06T07:30:13.184717+0000","last_undegraded":"2024-03-06T07:30:13.184717+0000","last_fullsized":"2024-03-06T07:30:13.184717+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:20.570693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184377+0000","last_change":"2024-03-06T07:30:12.092705+0000","last_active":"2024-03-06T07:30:13.184377+0000","last_peered":"2024-03-06T07:30:13.184377+0000","last_clean":"2024-03-06T07:30:13.184377+0000","last_became_active":"2024-03-06T07:30:12.092327+0000","last_became_peered":"2024-03-06T07:30:12.092327+0000","last_unstale":"2024-03-06T07:30:13.184377+0000","last_undegraded":"2024-03-06T07:30:13.184377+0000","last_fullsized":"2024-03-06T07:30:13.184377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:30.729503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182103+0000","last_change":"2024-03-06T07:30:11.053248+0000","last_active":"2024-03-06T07:30:13.182103+0000","last_peered":"2024-03-06T07:30:13.182103+0000","last_clean":"2024-03-06T07:30:13.182103+0000","last_became_active":"2024-03-06T07:30:11.052920+0000","last_became_peered":"2024-03-06T07:30:11.052920+0000","last_unstale":"2024-03-06T07:30:13.182103+0000","last_undegraded":"2024-03-06T07:30:13.182103+0000","last_fullsized":"2024-03-06T07:30:13.182103+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:23.293072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053298+0000","last_change":"2024-03-06T07:30:12.078425+0000","last_active":"2024-03-06T07:30:13.053298+0000","last_peered":"2024-03-06T07:30:13.053298+0000","last_clean":"2024-03-06T07:30:13.053298+0000","last_became_active":"2024-03-06T07:30:12.077931+0000","last_became_peered":"2024-03-06T07:30:12.077931+0000","last_unstale":"2024-03-06T07:30:13.053298+0000","last_undegraded":"2024-03-06T07:30:13.053298+0000","last_fullsized":"2024-03-06T07:30:13.053298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.006663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186492+0000","last_change":"2024-03-06T07:30:11.063511+0000","last_active":"2024-03-06T07:30:13.186492+0000","last_peered":"2024-03-06T07:30:13.186492+0000","last_clean":"2024-03-06T07:30:13.186492+0000","last_became_active":"2024-03-06T07:30:11.063201+0000","last_became_peered":"2024-03-06T07:30:11.063201+0000","last_unstale":"2024-03-06T07:30:13.186492+0000","last_undegraded":"2024-03-06T07:30:13.186492+0000","last_fullsized":"2024-03-06T07:30:13.186492+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:49.201097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053414+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:30:13.053414+0000","last_peered":"2024-03-06T07:30:13.053414+0000","last_clean":"2024-03-06T07:30:13.053414+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:30:13.053414+0000","last_undegraded":"2024-03-06T07:30:13.053414+0000","last_fullsized":"2024-03-06T07:30:13.053414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":970752,"data_stored":735429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":380928,"data_stored":145587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":380928,"data_stored":145643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":380928,"data_stored":145075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":146605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":376832,"data_stored":145041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":389120,"data_stored":146605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":978944,"data_stored":741639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:16.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:16.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.775+0000 7fe333200640 1 Processor -- start 2024-03-06T07:30:16.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.776+0000 7fe333200640 1 -- start start 2024-03-06T07:30:16.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe333200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3340d04b0 0x7fe334070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe333200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3340700c0 0x7fe3340cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe333200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3340cd0c0 con 0x7fe33415f450 2024-03-06T07:30:16.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe333200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3340d29e0 con 0x7fe3340700c0 2024-03-06T07:30:16.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe333200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340cd3a0 con 0x7fe3340d04b0 2024-03-06T07:30:16.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe332800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3340700c0 0x7fe3340cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe332800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3340700c0 0x7fe3340cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49748/0 (socket says 172.21.15.32:49748) 2024-03-06T07:30:16.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.777+0000 7fe332800640 1 -- 172.21.15.32:0/1582959476 learned_addr learned my addr 172.21.15.32:0/1582959476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.778+0000 7fe331400640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3340d04b0 0x7fe334070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.778+0000 7fe332800640 1 -- 172.21.15.32:0/1582959476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3340d04b0 msgr2=0x7fe334070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.778+0000 7fe332800640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3340d04b0 0x7fe334070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.778+0000 7fe332800640 1 -- 172.21.15.32:0/1582959476 >> v1:172.21.15.39:6789/0 conn(0x7fe33415f450 legacy=0x7fe3340cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.778+0000 7fe332800640 1 -- 172.21.15.32:0/1582959476 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340cd680 con 0x7fe3340700c0 2024-03-06T07:30:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.778+0000 7fe331400640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3340d04b0 0x7fe334070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.779+0000 7fe332800640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340cde80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009870 tx=0x7fe32c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4714750f83d2147d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.779+0000 7fe330a00640 1 -- 172.21.15.32:0/1582959476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c035020 con 0x7fe3340700c0 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.779+0000 7fe330a00640 1 -- 172.21.15.32:0/1582959476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c031640 con 0x7fe3340700c0 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.779+0000 7fe330a00640 1 -- 172.21.15.32:0/1582959476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c02d040 con 0x7fe3340700c0 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 -- 172.21.15.32:0/1582959476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 msgr2=0x7fe3340cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340cde80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009870 tx=0x7fe32c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 -- 172.21.15.32:0/1582959476 shutdown_connections 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340cde80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 --2- 172.21.15.32:0/1582959476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3340d04b0 0x7fe334070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 -- 172.21.15.32:0/1582959476 >> 172.21.15.32:0/1582959476 conn(0x7fe3340d1ae0 msgr2=0x7fe3340d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 -- 172.21.15.32:0/1582959476 shutdown_connections 2024-03-06T07:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.780+0000 7fe333200640 1 -- 172.21.15.32:0/1582959476 wait complete. 2024-03-06T07:30:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.782+0000 7fe333200640 1 Processor -- start 2024-03-06T07:30:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.782+0000 7fe333200640 1 -- start start 2024-03-06T07:30:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe333200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe333200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3340d04b0 0x7fe3340b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe333200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe33415f450 0x7fe3340c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe333200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33415f830 con 0x7fe3340700c0 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe333200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340ae8b0 con 0x7fe3340d04b0 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe333200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33415f9b0 con 0x7fe33415f450 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe331e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe331e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49750/0 (socket says 172.21.15.32:49750) 2024-03-06T07:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.783+0000 7fe331e00640 1 -- 172.21.15.32:0/753829117 learned_addr learned my addr 172.21.15.32:0/753829117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331400640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3340d04b0 0x7fe3340b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331e00640 1 -- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe33415f450 msgr2=0x7fe3340c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331e00640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe33415f450 0x7fe3340c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331e00640 1 -- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3340d04b0 msgr2=0x7fe3340b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331e00640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3340d04b0 0x7fe3340b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331e00640 1 -- 172.21.15.32:0/753829117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340afb40 con 0x7fe3340700c0 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331400640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3340d04b0 0x7fe3340b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe331e00640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340af440 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe32400ba50 tx=0x7fe32400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324019070 con 0x7fe3340700c0 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32400cc20 con 0x7fe3340700c0 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324014440 con 0x7fe3340700c0 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.784+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3340afe20 con 0x7fe3340700c0 2024-03-06T07:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.785+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3340b03e0 con 0x7fe3340700c0 2024-03-06T07:30:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.786+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33404f330 con 0x7fe3340700c0 2024-03-06T07:30:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.787+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe32401c070 con 0x7fe3340700c0 2024-03-06T07:30:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.788+0000 7fe32a000640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe31006b4d0 0x7fe31006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.788+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe324089ec0 con 0x7fe3340700c0 2024-03-06T07:30:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.788+0000 7fe331400640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe31006b4d0 0x7fe31006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.789+0000 7fe331400640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe31006b4d0 0x7fe31006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006fd0 tx=0x7fe31c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.796+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3240586e0 con 0x7fe3340700c0 2024-03-06T07:30:16.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:16.991+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3340b07e0 con 0x7fe31006b4d0 2024-03-06T07:30:17.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.009+0000 7fe32a000640 1 -- 172.21.15.32:0/753829117 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251953 (secure 0 0 0) 0x7fe3340b07e0 con 0x7fe31006b4d0 2024-03-06T07:30:17.015 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:17.016 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:30:17.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.014+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe31006b4d0 msgr2=0x7fe31006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.014+0000 7fe333200640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe31006b4d0 0x7fe31006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006fd0 tx=0x7fe31c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.014+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 msgr2=0x7fe3340af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.014+0000 7fe333200640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340af440 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe32400ba50 tx=0x7fe32400bf10 comp rx=0 tx=0).stop 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 shutdown_connections 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe31006b4d0 0x7fe31006d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe33415f450 0x7fe3340c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3340d04b0 0x7fe3340b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 --2- 172.21.15.32:0/753829117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3340700c0 0x7fe3340af440 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 >> 172.21.15.32:0/753829117 conn(0x7fe3340d1ae0 msgr2=0x7fe33415da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 shutdown_connections 2024-03-06T07:30:17.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.015+0000 7fe333200640 1 -- 172.21.15.32:0/753829117 wait complete. 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-06T07:30:16.208185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253584,"kb_used_data":4148,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721136,"statfs":{"total":773094113280,"available":772834443264,"internally_reserved":0,"allocated":4247552,"data_stored":2351624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001908"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187403+0000","last_change":"2024-03-06T07:30:12.092142+0000","last_active":"2024-03-06T07:30:13.187403+0000","last_peered":"2024-03-06T07:30:13.187403+0000","last_clean":"2024-03-06T07:30:13.187403+0000","last_became_active":"2024-03-06T07:30:12.091831+0000","last_became_peered":"2024-03-06T07:30:12.091831+0000","last_unstale":"2024-03-06T07:30:13.187403+0000","last_undegraded":"2024-03-06T07:30:13.187403+0000","last_fullsized":"2024-03-06T07:30:13.187403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:54.482169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184131+0000","last_change":"2024-03-06T07:30:12.088013+0000","last_active":"2024-03-06T07:30:13.184131+0000","last_peered":"2024-03-06T07:30:13.184131+0000","last_clean":"2024-03-06T07:30:13.184131+0000","last_became_active":"2024-03-06T07:30:12.087627+0000","last_became_peered":"2024-03-06T07:30:12.087627+0000","last_unstale":"2024-03-06T07:30:13.184131+0000","last_undegraded":"2024-03-06T07:30:13.184131+0000","last_fullsized":"2024-03-06T07:30:13.184131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:55.808069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054447+0000","last_change":"2024-03-06T07:30:12.083552+0000","last_active":"2024-03-06T07:30:13.054447+0000","last_peered":"2024-03-06T07:30:13.054447+0000","last_clean":"2024-03-06T07:30:13.054447+0000","last_became_active":"2024-03-06T07:30:12.083237+0000","last_became_peered":"2024-03-06T07:30:12.083237+0000","last_unstale":"2024-03-06T07:30:13.054447+0000","last_undegraded":"2024-03-06T07:30:13.054447+0000","last_fullsized":"2024-03-06T07:30:13.054447+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.006663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097932+0000","last_change":"2024-03-06T07:30:12.083371+0000","last_active":"2024-03-06T07:30:13.097932+0000","last_peered":"2024-03-06T07:30:13.097932+0000","last_clean":"2024-03-06T07:30:13.097932+0000","last_became_active":"2024-03-06T07:30:12.083044+0000","last_became_peered":"2024-03-06T07:30:12.083044+0000","last_unstale":"2024-03-06T07:30:13.097932+0000","last_undegraded":"2024-03-06T07:30:13.097932+0000","last_fullsized":"2024-03-06T07:30:13.097932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:41.441047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184326+0000","last_change":"2024-03-06T07:30:12.076729+0000","last_active":"2024-03-06T07:30:13.184326+0000","last_peered":"2024-03-06T07:30:13.184326+0000","last_clean":"2024-03-06T07:30:13.184326+0000","last_became_active":"2024-03-06T07:30:12.076442+0000","last_became_peered":"2024-03-06T07:30:12.076442+0000","last_unstale":"2024-03-06T07:30:13.184326+0000","last_undegraded":"2024-03-06T07:30:13.184326+0000","last_fullsized":"2024-03-06T07:30:13.184326+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:50.215492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182839+0000","last_change":"2024-03-06T07:30:12.091033+0000","last_active":"2024-03-06T07:30:13.182839+0000","last_peered":"2024-03-06T07:30:13.182839+0000","last_clean":"2024-03-06T07:30:13.182839+0000","last_became_active":"2024-03-06T07:30:12.090744+0000","last_became_peered":"2024-03-06T07:30:12.090744+0000","last_unstale":"2024-03-06T07:30:13.182839+0000","last_undegraded":"2024-03-06T07:30:13.182839+0000","last_fullsized":"2024-03-06T07:30:13.182839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:46.635895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052416+0000","last_change":"2024-03-06T07:30:12.086866+0000","last_active":"2024-03-06T07:30:13.052416+0000","last_peered":"2024-03-06T07:30:13.052416+0000","last_clean":"2024-03-06T07:30:13.052416+0000","last_became_active":"2024-03-06T07:30:12.085597+0000","last_became_peered":"2024-03-06T07:30:12.085597+0000","last_unstale":"2024-03-06T07:30:13.052416+0000","last_undegraded":"2024-03-06T07:30:13.052416+0000","last_fullsized":"2024-03-06T07:30:13.052416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:28.009850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.051969+0000","last_change":"2024-03-06T07:30:12.077390+0000","last_active":"2024-03-06T07:30:13.051969+0000","last_peered":"2024-03-06T07:30:13.051969+0000","last_clean":"2024-03-06T07:30:13.051969+0000","last_became_active":"2024-03-06T07:30:12.076937+0000","last_became_peered":"2024-03-06T07:30:12.076937+0000","last_unstale":"2024-03-06T07:30:13.051969+0000","last_undegraded":"2024-03-06T07:30:13.051969+0000","last_fullsized":"2024-03-06T07:30:13.051969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:38.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186817+0000","last_change":"2024-03-06T07:30:12.089583+0000","last_active":"2024-03-06T07:30:13.186817+0000","last_peered":"2024-03-06T07:30:13.186817+0000","last_clean":"2024-03-06T07:30:13.186817+0000","last_became_active":"2024-03-06T07:30:12.089307+0000","last_became_peered":"2024-03-06T07:30:12.089307+0000","last_unstale":"2024-03-06T07:30:13.186817+0000","last_undegraded":"2024-03-06T07:30:13.186817+0000","last_fullsized":"2024-03-06T07:30:13.186817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:39.071964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098255+0000","last_change":"2024-03-06T07:30:12.079454+0000","last_active":"2024-03-06T07:30:13.098255+0000","last_peered":"2024-03-06T07:30:13.098255+0000","last_clean":"2024-03-06T07:30:13.098255+0000","last_became_active":"2024-03-06T07:30:12.079148+0000","last_became_peered":"2024-03-06T07:30:12.079148+0000","last_unstale":"2024-03-06T07:30:13.098255+0000","last_undegraded":"2024-03-06T07:30:13.098255+0000","last_fullsized":"2024-03-06T07:30:13.098255+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:38.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053960+0000","last_change":"2024-03-06T07:30:12.075140+0000","last_active":"2024-03-06T07:30:13.053960+0000","last_peered":"2024-03-06T07:30:13.053960+0000","last_clean":"2024-03-06T07:30:13.053960+0000","last_became_active":"2024-03-06T07:30:12.074190+0000","last_became_peered":"2024-03-06T07:30:12.074190+0000","last_unstale":"2024-03-06T07:30:13.053960+0000","last_undegraded":"2024-03-06T07:30:13.053960+0000","last_fullsized":"2024-03-06T07:30:13.053960+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:39.121566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054857+0000","last_change":"2024-03-06T07:30:12.089961+0000","last_active":"2024-03-06T07:30:13.054857+0000","last_peered":"2024-03-06T07:30:13.054857+0000","last_clean":"2024-03-06T07:30:13.054857+0000","last_became_active":"2024-03-06T07:30:12.089476+0000","last_became_peered":"2024-03-06T07:30:12.089476+0000","last_unstale":"2024-03-06T07:30:13.054857+0000","last_undegraded":"2024-03-06T07:30:13.054857+0000","last_fullsized":"2024-03-06T07:30:13.054857+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:39.121566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184480+0000","last_change":"2024-03-06T07:30:12.081588+0000","last_active":"2024-03-06T07:30:13.184480+0000","last_peered":"2024-03-06T07:30:13.184480+0000","last_clean":"2024-03-06T07:30:13.184480+0000","last_became_active":"2024-03-06T07:30:12.080429+0000","last_became_peered":"2024-03-06T07:30:12.080429+0000","last_unstale":"2024-03-06T07:30:13.184480+0000","last_undegraded":"2024-03-06T07:30:13.184480+0000","last_fullsized":"2024-03-06T07:30:13.184480+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:48.145965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182608+0000","last_change":"2024-03-06T07:30:12.074044+0000","last_active":"2024-03-06T07:30:13.182608+0000","last_peered":"2024-03-06T07:30:13.182608+0000","last_clean":"2024-03-06T07:30:13.182608+0000","last_became_active":"2024-03-06T07:30:12.073709+0000","last_became_peered":"2024-03-06T07:30:12.073709+0000","last_unstale":"2024-03-06T07:30:13.182608+0000","last_undegraded":"2024-03-06T07:30:13.182608+0000","last_fullsized":"2024-03-06T07:30:13.182608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:39.121566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052589+0000","last_change":"2024-03-06T07:30:12.088894+0000","last_active":"2024-03-06T07:30:13.052589+0000","last_peered":"2024-03-06T07:30:13.052589+0000","last_clean":"2024-03-06T07:30:13.052589+0000","last_became_active":"2024-03-06T07:30:12.088557+0000","last_became_peered":"2024-03-06T07:30:12.088557+0000","last_unstale":"2024-03-06T07:30:13.052589+0000","last_undegraded":"2024-03-06T07:30:13.052589+0000","last_fullsized":"2024-03-06T07:30:13.052589+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:25.275202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053249+0000","last_change":"2024-03-06T07:30:12.076817+0000","last_active":"2024-03-06T07:30:13.053249+0000","last_peered":"2024-03-06T07:30:13.053249+0000","last_clean":"2024-03-06T07:30:13.053249+0000","last_became_active":"2024-03-06T07:30:12.076506+0000","last_became_peered":"2024-03-06T07:30:12.076506+0000","last_unstale":"2024-03-06T07:30:13.053249+0000","last_undegraded":"2024-03-06T07:30:13.053249+0000","last_fullsized":"2024-03-06T07:30:13.053249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:55.455402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.055207+0000","last_change":"2024-03-06T07:30:12.093804+0000","last_active":"2024-03-06T07:30:13.055207+0000","last_peered":"2024-03-06T07:30:13.055207+0000","last_clean":"2024-03-06T07:30:13.055207+0000","last_became_active":"2024-03-06T07:30:12.093480+0000","last_became_peered":"2024-03-06T07:30:12.093480+0000","last_unstale":"2024-03-06T07:30:13.055207+0000","last_undegraded":"2024-03-06T07:30:13.055207+0000","last_fullsized":"2024-03-06T07:30:13.055207+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:15.057951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183916+0000","last_change":"2024-03-06T07:30:12.084274+0000","last_active":"2024-03-06T07:30:13.183916+0000","last_peered":"2024-03-06T07:30:13.183916+0000","last_clean":"2024-03-06T07:30:13.183916+0000","last_became_active":"2024-03-06T07:30:12.083921+0000","last_became_peered":"2024-03-06T07:30:12.083921+0000","last_unstale":"2024-03-06T07:30:13.183916+0000","last_undegraded":"2024-03-06T07:30:13.183916+0000","last_fullsized":"2024-03-06T07:30:13.183916+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:54.264504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183172+0000","last_change":"2024-03-06T07:30:12.074907+0000","last_active":"2024-03-06T07:30:13.183172+0000","last_peered":"2024-03-06T07:30:13.183172+0000","last_clean":"2024-03-06T07:30:13.183172+0000","last_became_active":"2024-03-06T07:30:12.074559+0000","last_became_peered":"2024-03-06T07:30:12.074559+0000","last_unstale":"2024-03-06T07:30:13.183172+0000","last_undegraded":"2024-03-06T07:30:13.183172+0000","last_fullsized":"2024-03-06T07:30:13.183172+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:55.262416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184776+0000","last_change":"2024-03-06T07:30:12.096082+0000","last_active":"2024-03-06T07:30:13.184776+0000","last_peered":"2024-03-06T07:30:13.184776+0000","last_clean":"2024-03-06T07:30:13.184776+0000","last_became_active":"2024-03-06T07:30:12.095191+0000","last_became_peered":"2024-03-06T07:30:12.095191+0000","last_unstale":"2024-03-06T07:30:13.184776+0000","last_undegraded":"2024-03-06T07:30:13.184776+0000","last_fullsized":"2024-03-06T07:30:13.184776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:28.260506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184874+0000","last_change":"2024-03-06T07:30:12.098947+0000","last_active":"2024-03-06T07:30:13.184874+0000","last_peered":"2024-03-06T07:30:13.184874+0000","last_clean":"2024-03-06T07:30:13.184874+0000","last_became_active":"2024-03-06T07:30:12.098683+0000","last_became_peered":"2024-03-06T07:30:12.098683+0000","last_unstale":"2024-03-06T07:30:13.184874+0000","last_undegraded":"2024-03-06T07:30:13.184874+0000","last_fullsized":"2024-03-06T07:30:13.184874+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:54.482169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184993+0000","last_change":"2024-03-06T07:30:12.087934+0000","last_active":"2024-03-06T07:30:13.184993+0000","last_peered":"2024-03-06T07:30:13.184993+0000","last_clean":"2024-03-06T07:30:13.184993+0000","last_became_active":"2024-03-06T07:30:12.087621+0000","last_became_peered":"2024-03-06T07:30:12.087621+0000","last_unstale":"2024-03-06T07:30:13.184993+0000","last_undegraded":"2024-03-06T07:30:13.184993+0000","last_fullsized":"2024-03-06T07:30:13.184993+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:09.827964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094764+0000","last_change":"2024-03-06T07:30:12.064609+0000","last_active":"2024-03-06T07:30:13.094764+0000","last_peered":"2024-03-06T07:30:13.094764+0000","last_clean":"2024-03-06T07:30:13.094764+0000","last_became_active":"2024-03-06T07:30:12.064061+0000","last_became_peered":"2024-03-06T07:30:12.064061+0000","last_unstale":"2024-03-06T07:30:13.094764+0000","last_undegraded":"2024-03-06T07:30:13.094764+0000","last_fullsized":"2024-03-06T07:30:13.094764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:24.307467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052776+0000","last_change":"2024-03-06T07:30:12.084650+0000","last_active":"2024-03-06T07:30:13.052776+0000","last_peered":"2024-03-06T07:30:13.052776+0000","last_clean":"2024-03-06T07:30:13.052776+0000","last_became_active":"2024-03-06T07:30:12.084289+0000","last_became_peered":"2024-03-06T07:30:12.084289+0000","last_unstale":"2024-03-06T07:30:13.052776+0000","last_undegraded":"2024-03-06T07:30:13.052776+0000","last_fullsized":"2024-03-06T07:30:13.052776+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:44.879274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098519+0000","last_change":"2024-03-06T07:30:12.065829+0000","last_active":"2024-03-06T07:30:13.098519+0000","last_peered":"2024-03-06T07:30:13.098519+0000","last_clean":"2024-03-06T07:30:13.098519+0000","last_became_active":"2024-03-06T07:30:12.064986+0000","last_became_peered":"2024-03-06T07:30:12.064986+0000","last_unstale":"2024-03-06T07:30:13.098519+0000","last_undegraded":"2024-03-06T07:30:13.098519+0000","last_fullsized":"2024-03-06T07:30:13.098519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:25.275202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184058+0000","last_change":"2024-03-06T07:30:12.077437+0000","last_active":"2024-03-06T07:30:13.184058+0000","last_peered":"2024-03-06T07:30:13.184058+0000","last_clean":"2024-03-06T07:30:13.184058+0000","last_became_active":"2024-03-06T07:30:12.076484+0000","last_became_peered":"2024-03-06T07:30:12.076484+0000","last_unstale":"2024-03-06T07:30:13.184058+0000","last_undegraded":"2024-03-06T07:30:13.184058+0000","last_fullsized":"2024-03-06T07:30:13.184058+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:56.958197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098528+0000","last_change":"2024-03-06T07:30:12.087007+0000","last_active":"2024-03-06T07:30:13.098528+0000","last_peered":"2024-03-06T07:30:13.098528+0000","last_clean":"2024-03-06T07:30:13.098528+0000","last_became_active":"2024-03-06T07:30:12.086716+0000","last_became_peered":"2024-03-06T07:30:12.086716+0000","last_unstale":"2024-03-06T07:30:13.098528+0000","last_undegraded":"2024-03-06T07:30:13.098528+0000","last_fullsized":"2024-03-06T07:30:13.098528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:44.879274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054092+0000","last_change":"2024-03-06T07:30:12.083729+0000","last_active":"2024-03-06T07:30:13.054092+0000","last_peered":"2024-03-06T07:30:13.054092+0000","last_clean":"2024-03-06T07:30:13.054092+0000","last_became_active":"2024-03-06T07:30:12.083179+0000","last_became_peered":"2024-03-06T07:30:12.083179+0000","last_unstale":"2024-03-06T07:30:13.054092+0000","last_undegraded":"2024-03-06T07:30:13.054092+0000","last_fullsized":"2024-03-06T07:30:13.054092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:54.174988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185049+0000","last_change":"2024-03-06T07:30:12.088538+0000","last_active":"2024-03-06T07:30:13.185049+0000","last_peered":"2024-03-06T07:30:13.185049+0000","last_clean":"2024-03-06T07:30:13.185049+0000","last_became_active":"2024-03-06T07:30:12.088232+0000","last_became_peered":"2024-03-06T07:30:12.088232+0000","last_unstale":"2024-03-06T07:30:13.185049+0000","last_undegraded":"2024-03-06T07:30:13.185049+0000","last_fullsized":"2024-03-06T07:30:13.185049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:26.622501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052933+0000","last_change":"2024-03-06T07:30:12.086424+0000","last_active":"2024-03-06T07:30:13.052933+0000","last_peered":"2024-03-06T07:30:13.052933+0000","last_clean":"2024-03-06T07:30:13.052933+0000","last_became_active":"2024-03-06T07:30:12.086152+0000","last_became_peered":"2024-03-06T07:30:12.086152+0000","last_unstale":"2024-03-06T07:30:13.052933+0000","last_undegraded":"2024-03-06T07:30:13.052933+0000","last_fullsized":"2024-03-06T07:30:13.052933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:54.174988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186017+0000","last_change":"2024-03-06T07:30:12.087396+0000","last_active":"2024-03-06T07:30:13.186017+0000","last_peered":"2024-03-06T07:30:13.186017+0000","last_clean":"2024-03-06T07:30:13.186017+0000","last_became_active":"2024-03-06T07:30:12.086007+0000","last_became_peered":"2024-03-06T07:30:12.086007+0000","last_unstale":"2024-03-06T07:30:13.186017+0000","last_undegraded":"2024-03-06T07:30:13.186017+0000","last_fullsized":"2024-03-06T07:30:13.186017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:49.101946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.051986+0000","last_change":"2024-03-06T07:30:12.083316+0000","last_active":"2024-03-06T07:30:13.051986+0000","last_peered":"2024-03-06T07:30:13.051986+0000","last_clean":"2024-03-06T07:30:13.051986+0000","last_became_active":"2024-03-06T07:30:12.082701+0000","last_became_peered":"2024-03-06T07:30:12.082701+0000","last_unstale":"2024-03-06T07:30:13.051986+0000","last_undegraded":"2024-03-06T07:30:13.051986+0000","last_fullsized":"2024-03-06T07:30:13.051986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:25.275202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.095161+0000","last_change":"2024-03-06T07:30:12.067352+0000","last_active":"2024-03-06T07:30:13.095161+0000","last_peered":"2024-03-06T07:30:13.095161+0000","last_clean":"2024-03-06T07:30:13.095161+0000","last_became_active":"2024-03-06T07:30:12.066935+0000","last_became_peered":"2024-03-06T07:30:12.066935+0000","last_unstale":"2024-03-06T07:30:13.095161+0000","last_undegraded":"2024-03-06T07:30:13.095161+0000","last_fullsized":"2024-03-06T07:30:13.095161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:26.622501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187458+0000","last_change":"2024-03-06T07:30:11.062503+0000","last_active":"2024-03-06T07:30:13.187458+0000","last_peered":"2024-03-06T07:30:13.187458+0000","last_clean":"2024-03-06T07:30:13.187458+0000","last_became_active":"2024-03-06T07:30:11.062029+0000","last_became_peered":"2024-03-06T07:30:11.062029+0000","last_unstale":"2024-03-06T07:30:13.187458+0000","last_undegraded":"2024-03-06T07:30:13.187458+0000","last_fullsized":"2024-03-06T07:30:13.187458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:47.131571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187314+0000","last_change":"2024-03-06T07:30:12.091817+0000","last_active":"2024-03-06T07:30:13.187314+0000","last_peered":"2024-03-06T07:30:13.187314+0000","last_clean":"2024-03-06T07:30:13.187314+0000","last_became_active":"2024-03-06T07:30:12.091464+0000","last_became_peered":"2024-03-06T07:30:12.091464+0000","last_unstale":"2024-03-06T07:30:13.187314+0000","last_undegraded":"2024-03-06T07:30:13.187314+0000","last_fullsized":"2024-03-06T07:30:13.187314+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:24.307467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.095085+0000","last_change":"2024-03-06T07:30:11.062351+0000","last_active":"2024-03-06T07:30:13.095085+0000","last_peered":"2024-03-06T07:30:13.095085+0000","last_clean":"2024-03-06T07:30:13.095085+0000","last_became_active":"2024-03-06T07:30:11.061981+0000","last_became_peered":"2024-03-06T07:30:11.061981+0000","last_unstale":"2024-03-06T07:30:13.095085+0000","last_undegraded":"2024-03-06T07:30:13.095085+0000","last_fullsized":"2024-03-06T07:30:13.095085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:27.246112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186969+0000","last_change":"2024-03-06T07:30:12.091354+0000","last_active":"2024-03-06T07:30:13.186969+0000","last_peered":"2024-03-06T07:30:13.186969+0000","last_clean":"2024-03-06T07:30:13.186969+0000","last_became_active":"2024-03-06T07:30:12.090978+0000","last_became_peered":"2024-03-06T07:30:12.090978+0000","last_unstale":"2024-03-06T07:30:13.186969+0000","last_undegraded":"2024-03-06T07:30:13.186969+0000","last_fullsized":"2024-03-06T07:30:13.186969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:07.689339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183499+0000","last_change":"2024-03-06T07:30:11.040466+0000","last_active":"2024-03-06T07:30:13.183499+0000","last_peered":"2024-03-06T07:30:13.183499+0000","last_clean":"2024-03-06T07:30:13.183499+0000","last_became_active":"2024-03-06T07:30:11.038262+0000","last_became_peered":"2024-03-06T07:30:11.038262+0000","last_unstale":"2024-03-06T07:30:13.183499+0000","last_undegraded":"2024-03-06T07:30:13.183499+0000","last_fullsized":"2024-03-06T07:30:13.183499+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:53.467774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098532+0000","last_change":"2024-03-06T07:30:12.084328+0000","last_active":"2024-03-06T07:30:13.098532+0000","last_peered":"2024-03-06T07:30:13.098532+0000","last_clean":"2024-03-06T07:30:13.098532+0000","last_became_active":"2024-03-06T07:30:12.084042+0000","last_became_peered":"2024-03-06T07:30:12.084042+0000","last_unstale":"2024-03-06T07:30:13.098532+0000","last_undegraded":"2024-03-06T07:30:13.098532+0000","last_fullsized":"2024-03-06T07:30:13.098532+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:28.009850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184711+0000","last_change":"2024-03-06T07:30:11.050325+0000","last_active":"2024-03-06T07:30:13.184711+0000","last_peered":"2024-03-06T07:30:13.184711+0000","last_clean":"2024-03-06T07:30:13.184711+0000","last_became_active":"2024-03-06T07:30:11.049996+0000","last_became_peered":"2024-03-06T07:30:11.049996+0000","last_unstale":"2024-03-06T07:30:13.184711+0000","last_undegraded":"2024-03-06T07:30:13.184711+0000","last_fullsized":"2024-03-06T07:30:13.184711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:59.879817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053371+0000","last_change":"2024-03-06T07:30:12.084268+0000","last_active":"2024-03-06T07:30:13.053371+0000","last_peered":"2024-03-06T07:30:13.053371+0000","last_clean":"2024-03-06T07:30:13.053371+0000","last_became_active":"2024-03-06T07:30:12.082657+0000","last_became_peered":"2024-03-06T07:30:12.082657+0000","last_unstale":"2024-03-06T07:30:13.053371+0000","last_undegraded":"2024-03-06T07:30:13.053371+0000","last_fullsized":"2024-03-06T07:30:13.053371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.826737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185461+0000","last_change":"2024-03-06T07:30:11.053917+0000","last_active":"2024-03-06T07:30:13.185461+0000","last_peered":"2024-03-06T07:30:13.185461+0000","last_clean":"2024-03-06T07:30:13.185461+0000","last_became_active":"2024-03-06T07:30:11.053573+0000","last_became_peered":"2024-03-06T07:30:11.053573+0000","last_unstale":"2024-03-06T07:30:13.185461+0000","last_undegraded":"2024-03-06T07:30:13.185461+0000","last_fullsized":"2024-03-06T07:30:13.185461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.623776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053886+0000","last_change":"2024-03-06T07:30:12.095550+0000","last_active":"2024-03-06T07:30:13.053886+0000","last_peered":"2024-03-06T07:30:13.053886+0000","last_clean":"2024-03-06T07:30:13.053886+0000","last_became_active":"2024-03-06T07:30:12.095172+0000","last_became_peered":"2024-03-06T07:30:12.095172+0000","last_unstale":"2024-03-06T07:30:13.053886+0000","last_undegraded":"2024-03-06T07:30:13.053886+0000","last_fullsized":"2024-03-06T07:30:13.053886+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:28.009850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184992+0000","last_change":"2024-03-06T07:30:11.053291+0000","last_active":"2024-03-06T07:30:13.184992+0000","last_peered":"2024-03-06T07:30:13.184992+0000","last_clean":"2024-03-06T07:30:13.184992+0000","last_became_active":"2024-03-06T07:30:11.052975+0000","last_became_peered":"2024-03-06T07:30:11.052975+0000","last_unstale":"2024-03-06T07:30:13.184992+0000","last_undegraded":"2024-03-06T07:30:13.184992+0000","last_fullsized":"2024-03-06T07:30:13.184992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:50.694170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184483+0000","last_change":"2024-03-06T07:30:12.093254+0000","last_active":"2024-03-06T07:30:13.184483+0000","last_peered":"2024-03-06T07:30:13.184483+0000","last_clean":"2024-03-06T07:30:13.184483+0000","last_became_active":"2024-03-06T07:30:12.092826+0000","last_became_peered":"2024-03-06T07:30:12.092826+0000","last_unstale":"2024-03-06T07:30:13.184483+0000","last_undegraded":"2024-03-06T07:30:13.184483+0000","last_fullsized":"2024-03-06T07:30:13.184483+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:24.307467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182137+0000","last_change":"2024-03-06T07:30:11.043582+0000","last_active":"2024-03-06T07:30:13.182137+0000","last_peered":"2024-03-06T07:30:13.182137+0000","last_clean":"2024-03-06T07:30:13.182137+0000","last_became_active":"2024-03-06T07:30:11.043153+0000","last_became_peered":"2024-03-06T07:30:11.043153+0000","last_unstale":"2024-03-06T07:30:13.182137+0000","last_undegraded":"2024-03-06T07:30:13.182137+0000","last_fullsized":"2024-03-06T07:30:13.182137+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:25.608107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097794+0000","last_change":"2024-03-06T07:30:12.092416+0000","last_active":"2024-03-06T07:30:13.097794+0000","last_peered":"2024-03-06T07:30:13.097794+0000","last_clean":"2024-03-06T07:30:13.097794+0000","last_became_active":"2024-03-06T07:30:12.092071+0000","last_became_peered":"2024-03-06T07:30:12.092071+0000","last_unstale":"2024-03-06T07:30:13.097794+0000","last_undegraded":"2024-03-06T07:30:13.097794+0000","last_fullsized":"2024-03-06T07:30:13.097794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:39.071964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184329+0000","last_change":"2024-03-06T07:30:11.057214+0000","last_active":"2024-03-06T07:30:13.184329+0000","last_peered":"2024-03-06T07:30:13.184329+0000","last_clean":"2024-03-06T07:30:13.184329+0000","last_became_active":"2024-03-06T07:30:11.056813+0000","last_became_peered":"2024-03-06T07:30:11.056813+0000","last_unstale":"2024-03-06T07:30:13.184329+0000","last_undegraded":"2024-03-06T07:30:13.184329+0000","last_fullsized":"2024-03-06T07:30:13.184329+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:53.660708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054464+0000","last_change":"2024-03-06T07:30:12.092025+0000","last_active":"2024-03-06T07:30:13.054464+0000","last_peered":"2024-03-06T07:30:13.054464+0000","last_clean":"2024-03-06T07:30:13.054464+0000","last_became_active":"2024-03-06T07:30:12.091570+0000","last_became_peered":"2024-03-06T07:30:12.091570+0000","last_unstale":"2024-03-06T07:30:13.054464+0000","last_undegraded":"2024-03-06T07:30:13.054464+0000","last_fullsized":"2024-03-06T07:30:13.054464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.826737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185040+0000","last_change":"2024-03-06T07:30:11.053376+0000","last_active":"2024-03-06T07:30:13.185040+0000","last_peered":"2024-03-06T07:30:13.185040+0000","last_clean":"2024-03-06T07:30:13.185040+0000","last_became_active":"2024-03-06T07:30:11.052938+0000","last_became_peered":"2024-03-06T07:30:11.052938+0000","last_unstale":"2024-03-06T07:30:13.185040+0000","last_undegraded":"2024-03-06T07:30:13.185040+0000","last_fullsized":"2024-03-06T07:30:13.185040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:09.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184743+0000","last_change":"2024-03-06T07:30:12.078292+0000","last_active":"2024-03-06T07:30:13.184743+0000","last_peered":"2024-03-06T07:30:13.184743+0000","last_clean":"2024-03-06T07:30:13.184743+0000","last_became_active":"2024-03-06T07:30:12.078006+0000","last_became_peered":"2024-03-06T07:30:12.078006+0000","last_unstale":"2024-03-06T07:30:13.184743+0000","last_undegraded":"2024-03-06T07:30:13.184743+0000","last_fullsized":"2024-03-06T07:30:13.184743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:11.516071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.095147+0000","last_change":"2024-03-06T07:30:11.051632+0000","last_active":"2024-03-06T07:30:13.095147+0000","last_peered":"2024-03-06T07:30:13.095147+0000","last_clean":"2024-03-06T07:30:13.095147+0000","last_became_active":"2024-03-06T07:30:11.051334+0000","last_became_peered":"2024-03-06T07:30:11.051334+0000","last_unstale":"2024-03-06T07:30:13.095147+0000","last_undegraded":"2024-03-06T07:30:13.095147+0000","last_fullsized":"2024-03-06T07:30:13.095147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:33.067615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182970+0000","last_change":"2024-03-06T07:30:12.072677+0000","last_active":"2024-03-06T07:30:13.182970+0000","last_peered":"2024-03-06T07:30:13.182970+0000","last_clean":"2024-03-06T07:30:13.182970+0000","last_became_active":"2024-03-06T07:30:12.072338+0000","last_became_peered":"2024-03-06T07:30:12.072338+0000","last_unstale":"2024-03-06T07:30:13.182970+0000","last_undegraded":"2024-03-06T07:30:13.182970+0000","last_fullsized":"2024-03-06T07:30:13.182970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:19.821191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054606+0000","last_change":"2024-03-06T07:30:11.062625+0000","last_active":"2024-03-06T07:30:13.054606+0000","last_peered":"2024-03-06T07:30:13.054606+0000","last_clean":"2024-03-06T07:30:13.054606+0000","last_became_active":"2024-03-06T07:30:11.062212+0000","last_became_peered":"2024-03-06T07:30:11.062212+0000","last_unstale":"2024-03-06T07:30:13.054606+0000","last_undegraded":"2024-03-06T07:30:13.054606+0000","last_fullsized":"2024-03-06T07:30:13.054606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:48.087552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054320+0000","last_change":"2024-03-06T07:30:12.088454+0000","last_active":"2024-03-06T07:30:13.054320+0000","last_peered":"2024-03-06T07:30:13.054320+0000","last_clean":"2024-03-06T07:30:13.054320+0000","last_became_active":"2024-03-06T07:30:12.088143+0000","last_became_peered":"2024-03-06T07:30:12.088143+0000","last_unstale":"2024-03-06T07:30:13.054320+0000","last_undegraded":"2024-03-06T07:30:13.054320+0000","last_fullsized":"2024-03-06T07:30:13.054320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:55.455402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.183102+0000","last_change":"2024-03-06T07:30:11.049071+0000","last_active":"2024-03-06T07:30:13.183102+0000","last_peered":"2024-03-06T07:30:13.183102+0000","last_clean":"2024-03-06T07:30:13.183102+0000","last_became_active":"2024-03-06T07:30:11.048560+0000","last_became_peered":"2024-03-06T07:30:11.048560+0000","last_unstale":"2024-03-06T07:30:13.183102+0000","last_undegraded":"2024-03-06T07:30:13.183102+0000","last_fullsized":"2024-03-06T07:30:13.183102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:38.057570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097800+0000","last_change":"2024-03-06T07:30:12.087328+0000","last_active":"2024-03-06T07:30:13.097800+0000","last_peered":"2024-03-06T07:30:13.097800+0000","last_clean":"2024-03-06T07:30:13.097800+0000","last_became_active":"2024-03-06T07:30:12.085938+0000","last_became_peered":"2024-03-06T07:30:12.085938+0000","last_unstale":"2024-03-06T07:30:13.097800+0000","last_undegraded":"2024-03-06T07:30:13.097800+0000","last_fullsized":"2024-03-06T07:30:13.097800+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:07.689339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185800+0000","last_change":"2024-03-06T07:30:11.060502+0000","last_active":"2024-03-06T07:30:13.185800+0000","last_peered":"2024-03-06T07:30:13.185800+0000","last_clean":"2024-03-06T07:30:13.185800+0000","last_became_active":"2024-03-06T07:30:11.060208+0000","last_became_peered":"2024-03-06T07:30:11.060208+0000","last_unstale":"2024-03-06T07:30:13.185800+0000","last_undegraded":"2024-03-06T07:30:13.185800+0000","last_fullsized":"2024-03-06T07:30:13.185800+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:00.225758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182844+0000","last_change":"2024-03-06T07:30:12.080691+0000","last_active":"2024-03-06T07:30:13.182844+0000","last_peered":"2024-03-06T07:30:13.182844+0000","last_clean":"2024-03-06T07:30:13.182844+0000","last_became_active":"2024-03-06T07:30:12.080189+0000","last_became_peered":"2024-03-06T07:30:12.080189+0000","last_unstale":"2024-03-06T07:30:13.182844+0000","last_undegraded":"2024-03-06T07:30:13.182844+0000","last_fullsized":"2024-03-06T07:30:13.182844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:15.057951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097802+0000","last_change":"2024-03-06T07:30:11.060615+0000","last_active":"2024-03-06T07:30:13.097802+0000","last_peered":"2024-03-06T07:30:13.097802+0000","last_clean":"2024-03-06T07:30:13.097802+0000","last_became_active":"2024-03-06T07:30:11.060269+0000","last_became_peered":"2024-03-06T07:30:11.060269+0000","last_unstale":"2024-03-06T07:30:13.097802+0000","last_undegraded":"2024-03-06T07:30:13.097802+0000","last_fullsized":"2024-03-06T07:30:13.097802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:27.246112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.054288+0000","last_change":"2024-03-06T07:30:12.083007+0000","last_active":"2024-03-06T07:30:13.054288+0000","last_peered":"2024-03-06T07:30:13.054288+0000","last_clean":"2024-03-06T07:30:13.054288+0000","last_became_active":"2024-03-06T07:30:12.082302+0000","last_became_peered":"2024-03-06T07:30:12.082302+0000","last_unstale":"2024-03-06T07:30:13.054288+0000","last_undegraded":"2024-03-06T07:30:13.054288+0000","last_fullsized":"2024-03-06T07:30:13.054288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:46.635895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182208+0000","last_change":"2024-03-06T07:30:11.054496+0000","last_active":"2024-03-06T07:30:13.182208+0000","last_peered":"2024-03-06T07:30:13.182208+0000","last_clean":"2024-03-06T07:30:13.182208+0000","last_became_active":"2024-03-06T07:30:11.053011+0000","last_became_peered":"2024-03-06T07:30:11.053011+0000","last_unstale":"2024-03-06T07:30:13.182208+0000","last_undegraded":"2024-03-06T07:30:13.182208+0000","last_fullsized":"2024-03-06T07:30:13.182208+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:08.813570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186037+0000","last_change":"2024-03-06T07:30:12.085561+0000","last_active":"2024-03-06T07:30:13.186037+0000","last_peered":"2024-03-06T07:30:13.186037+0000","last_clean":"2024-03-06T07:30:13.186037+0000","last_became_active":"2024-03-06T07:30:12.085245+0000","last_became_peered":"2024-03-06T07:30:12.085245+0000","last_unstale":"2024-03-06T07:30:13.186037+0000","last_undegraded":"2024-03-06T07:30:13.186037+0000","last_fullsized":"2024-03-06T07:30:13.186037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:10.325701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.181987+0000","last_change":"2024-03-06T07:30:11.040850+0000","last_active":"2024-03-06T07:30:13.181987+0000","last_peered":"2024-03-06T07:30:13.181987+0000","last_clean":"2024-03-06T07:30:13.181987+0000","last_became_active":"2024-03-06T07:30:11.040578+0000","last_became_peered":"2024-03-06T07:30:11.040578+0000","last_unstale":"2024-03-06T07:30:13.181987+0000","last_undegraded":"2024-03-06T07:30:13.181987+0000","last_fullsized":"2024-03-06T07:30:13.181987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:29.715109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.187237+0000","last_change":"2024-03-06T07:30:12.093917+0000","last_active":"2024-03-06T07:30:13.187237+0000","last_peered":"2024-03-06T07:30:13.187237+0000","last_clean":"2024-03-06T07:30:13.187237+0000","last_became_active":"2024-03-06T07:30:12.093396+0000","last_became_peered":"2024-03-06T07:30:12.093396+0000","last_unstale":"2024-03-06T07:30:13.187237+0000","last_undegraded":"2024-03-06T07:30:13.187237+0000","last_fullsized":"2024-03-06T07:30:13.187237+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:41.441047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098579+0000","last_change":"2024-03-06T07:30:11.048996+0000","last_active":"2024-03-06T07:30:13.098579+0000","last_peered":"2024-03-06T07:30:13.098579+0000","last_clean":"2024-03-06T07:30:13.098579+0000","last_became_active":"2024-03-06T07:30:11.048751+0000","last_became_peered":"2024-03-06T07:30:11.048751+0000","last_unstale":"2024-03-06T07:30:13.098579+0000","last_undegraded":"2024-03-06T07:30:13.098579+0000","last_fullsized":"2024-03-06T07:30:13.098579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:47.131571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.055235+0000","last_change":"2024-03-06T07:30:12.086388+0000","last_active":"2024-03-06T07:30:13.055235+0000","last_peered":"2024-03-06T07:30:13.055235+0000","last_clean":"2024-03-06T07:30:13.055235+0000","last_became_active":"2024-03-06T07:30:12.085901+0000","last_became_peered":"2024-03-06T07:30:12.085901+0000","last_unstale":"2024-03-06T07:30:13.055235+0000","last_undegraded":"2024-03-06T07:30:13.055235+0000","last_fullsized":"2024-03-06T07:30:13.055235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:15.057951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185917+0000","last_change":"2024-03-06T07:30:11.056644+0000","last_active":"2024-03-06T07:30:13.185917+0000","last_peered":"2024-03-06T07:30:13.185917+0000","last_clean":"2024-03-06T07:30:13.185917+0000","last_became_active":"2024-03-06T07:30:11.055935+0000","last_became_peered":"2024-03-06T07:30:11.055935+0000","last_unstale":"2024-03-06T07:30:13.185917+0000","last_undegraded":"2024-03-06T07:30:13.185917+0000","last_fullsized":"2024-03-06T07:30:13.185917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:40.646294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.097992+0000","last_change":"2024-03-06T07:30:12.083238+0000","last_active":"2024-03-06T07:30:13.097992+0000","last_peered":"2024-03-06T07:30:13.097992+0000","last_clean":"2024-03-06T07:30:13.097992+0000","last_became_active":"2024-03-06T07:30:12.082662+0000","last_became_peered":"2024-03-06T07:30:12.082662+0000","last_unstale":"2024-03-06T07:30:13.097992+0000","last_undegraded":"2024-03-06T07:30:13.097992+0000","last_fullsized":"2024-03-06T07:30:13.097992+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:16.327552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186138+0000","last_change":"2024-03-06T07:30:11.048542+0000","last_active":"2024-03-06T07:30:13.186138+0000","last_peered":"2024-03-06T07:30:13.186138+0000","last_clean":"2024-03-06T07:30:13.186138+0000","last_became_active":"2024-03-06T07:30:11.047908+0000","last_became_peered":"2024-03-06T07:30:11.047908+0000","last_unstale":"2024-03-06T07:30:13.186138+0000","last_undegraded":"2024-03-06T07:30:13.186138+0000","last_fullsized":"2024-03-06T07:30:13.186138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:49.201097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184243+0000","last_change":"2024-03-06T07:30:12.086286+0000","last_active":"2024-03-06T07:30:13.184243+0000","last_peered":"2024-03-06T07:30:13.184243+0000","last_clean":"2024-03-06T07:30:13.184243+0000","last_became_active":"2024-03-06T07:30:12.085976+0000","last_became_peered":"2024-03-06T07:30:12.085976+0000","last_unstale":"2024-03-06T07:30:13.184243+0000","last_undegraded":"2024-03-06T07:30:13.184243+0000","last_fullsized":"2024-03-06T07:30:13.184243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:19.635480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186887+0000","last_change":"2024-03-06T07:30:11.054038+0000","last_active":"2024-03-06T07:30:13.186887+0000","last_peered":"2024-03-06T07:30:13.186887+0000","last_clean":"2024-03-06T07:30:13.186887+0000","last_became_active":"2024-03-06T07:30:11.052117+0000","last_became_peered":"2024-03-06T07:30:11.052117+0000","last_unstale":"2024-03-06T07:30:13.186887+0000","last_undegraded":"2024-03-06T07:30:13.186887+0000","last_fullsized":"2024-03-06T07:30:13.186887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:18.621086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185516+0000","last_change":"2024-03-06T07:30:12.084150+0000","last_active":"2024-03-06T07:30:13.185516+0000","last_peered":"2024-03-06T07:30:13.185516+0000","last_clean":"2024-03-06T07:30:13.185516+0000","last_became_active":"2024-03-06T07:30:12.083829+0000","last_became_peered":"2024-03-06T07:30:12.083829+0000","last_unstale":"2024-03-06T07:30:13.185516+0000","last_undegraded":"2024-03-06T07:30:13.185516+0000","last_fullsized":"2024-03-06T07:30:13.185516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:12.788458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094982+0000","last_change":"2024-03-06T07:30:11.052430+0000","last_active":"2024-03-06T07:30:13.094982+0000","last_peered":"2024-03-06T07:30:13.094982+0000","last_clean":"2024-03-06T07:30:13.094982+0000","last_became_active":"2024-03-06T07:30:11.052050+0000","last_became_peered":"2024-03-06T07:30:11.052050+0000","last_unstale":"2024-03-06T07:30:13.094982+0000","last_undegraded":"2024-03-06T07:30:13.094982+0000","last_fullsized":"2024-03-06T07:30:13.094982+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:47.131571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184920+0000","last_change":"2024-03-06T07:30:12.092718+0000","last_active":"2024-03-06T07:30:13.184920+0000","last_peered":"2024-03-06T07:30:13.184920+0000","last_clean":"2024-03-06T07:30:13.184920+0000","last_became_active":"2024-03-06T07:30:12.092354+0000","last_became_peered":"2024-03-06T07:30:12.092354+0000","last_unstale":"2024-03-06T07:30:13.184920+0000","last_undegraded":"2024-03-06T07:30:13.184920+0000","last_fullsized":"2024-03-06T07:30:13.184920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:34.082010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186510+0000","last_change":"2024-03-06T07:30:11.060365+0000","last_active":"2024-03-06T07:30:13.186510+0000","last_peered":"2024-03-06T07:30:13.186510+0000","last_clean":"2024-03-06T07:30:13.186510+0000","last_became_active":"2024-03-06T07:30:11.059981+0000","last_became_peered":"2024-03-06T07:30:11.059981+0000","last_unstale":"2024-03-06T07:30:13.186510+0000","last_undegraded":"2024-03-06T07:30:13.186510+0000","last_fullsized":"2024-03-06T07:30:13.186510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:18.621086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.185238+0000","last_change":"2024-03-06T07:30:12.089382+0000","last_active":"2024-03-06T07:30:13.185238+0000","last_peered":"2024-03-06T07:30:13.185238+0000","last_clean":"2024-03-06T07:30:13.185238+0000","last_became_active":"2024-03-06T07:30:12.089101+0000","last_became_peered":"2024-03-06T07:30:12.089101+0000","last_unstale":"2024-03-06T07:30:13.185238+0000","last_undegraded":"2024-03-06T07:30:13.185238+0000","last_fullsized":"2024-03-06T07:30:13.185238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:16.491261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184442+0000","last_change":"2024-03-06T07:30:11.049247+0000","last_active":"2024-03-06T07:30:13.184442+0000","last_peered":"2024-03-06T07:30:13.184442+0000","last_clean":"2024-03-06T07:30:13.184442+0000","last_became_active":"2024-03-06T07:30:11.048954+0000","last_became_peered":"2024-03-06T07:30:11.048954+0000","last_unstale":"2024-03-06T07:30:13.184442+0000","last_undegraded":"2024-03-06T07:30:13.184442+0000","last_fullsized":"2024-03-06T07:30:13.184442+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:17.991337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.052189+0000","last_change":"2024-03-06T07:30:12.087098+0000","last_active":"2024-03-06T07:30:13.052189+0000","last_peered":"2024-03-06T07:30:13.052189+0000","last_clean":"2024-03-06T07:30:13.052189+0000","last_became_active":"2024-03-06T07:30:12.086608+0000","last_became_peered":"2024-03-06T07:30:12.086608+0000","last_unstale":"2024-03-06T07:30:13.052189+0000","last_undegraded":"2024-03-06T07:30:13.052189+0000","last_fullsized":"2024-03-06T07:30:13.052189+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:44.879274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053057+0000","last_change":"2024-03-06T07:30:11.062485+0000","last_active":"2024-03-06T07:30:13.053057+0000","last_peered":"2024-03-06T07:30:13.053057+0000","last_clean":"2024-03-06T07:30:13.053057+0000","last_became_active":"2024-03-06T07:30:11.062184+0000","last_became_peered":"2024-03-06T07:30:11.062184+0000","last_unstale":"2024-03-06T07:30:13.053057+0000","last_undegraded":"2024-03-06T07:30:13.053057+0000","last_fullsized":"2024-03-06T07:30:13.053057+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:11.774064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184629+0000","last_change":"2024-03-06T07:30:12.095920+0000","last_active":"2024-03-06T07:30:13.184629+0000","last_peered":"2024-03-06T07:30:13.184629+0000","last_clean":"2024-03-06T07:30:13.184629+0000","last_became_active":"2024-03-06T07:30:12.095580+0000","last_became_peered":"2024-03-06T07:30:12.095580+0000","last_unstale":"2024-03-06T07:30:13.184629+0000","last_undegraded":"2024-03-06T07:30:13.184629+0000","last_fullsized":"2024-03-06T07:30:13.184629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:11.554592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098683+0000","last_change":"2024-03-06T07:30:11.052325+0000","last_active":"2024-03-06T07:30:13.098683+0000","last_peered":"2024-03-06T07:30:13.098683+0000","last_clean":"2024-03-06T07:30:13.098683+0000","last_became_active":"2024-03-06T07:30:11.051994+0000","last_became_peered":"2024-03-06T07:30:11.051994+0000","last_unstale":"2024-03-06T07:30:13.098683+0000","last_undegraded":"2024-03-06T07:30:13.098683+0000","last_fullsized":"2024-03-06T07:30:13.098683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:10.540198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.055053+0000","last_change":"2024-03-06T07:30:12.086389+0000","last_active":"2024-03-06T07:30:13.055053+0000","last_peered":"2024-03-06T07:30:13.055053+0000","last_clean":"2024-03-06T07:30:13.055053+0000","last_became_active":"2024-03-06T07:30:12.085492+0000","last_became_peered":"2024-03-06T07:30:12.085492+0000","last_unstale":"2024-03-06T07:30:13.055053+0000","last_undegraded":"2024-03-06T07:30:13.055053+0000","last_fullsized":"2024-03-06T07:30:13.055053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:46.635895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.096789+0000","last_change":"2024-03-06T07:30:11.055929+0000","last_active":"2024-03-06T07:30:13.096789+0000","last_peered":"2024-03-06T07:30:13.096789+0000","last_clean":"2024-03-06T07:30:13.096789+0000","last_became_active":"2024-03-06T07:30:11.055599+0000","last_became_peered":"2024-03-06T07:30:11.055599+0000","last_unstale":"2024-03-06T07:30:13.096789+0000","last_undegraded":"2024-03-06T07:30:13.096789+0000","last_fullsized":"2024-03-06T07:30:13.096789+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:11.774064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094980+0000","last_change":"2024-03-06T07:30:12.069798+0000","last_active":"2024-03-06T07:30:13.094980+0000","last_peered":"2024-03-06T07:30:13.094980+0000","last_clean":"2024-03-06T07:30:13.094980+0000","last_became_active":"2024-03-06T07:30:12.069490+0000","last_became_peered":"2024-03-06T07:30:12.069490+0000","last_unstale":"2024-03-06T07:30:13.094980+0000","last_undegraded":"2024-03-06T07:30:13.094980+0000","last_fullsized":"2024-03-06T07:30:13.094980+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:09.827964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098141+0000","last_change":"2024-03-06T07:30:11.049364+0000","last_active":"2024-03-06T07:30:13.098141+0000","last_peered":"2024-03-06T07:30:13.098141+0000","last_clean":"2024-03-06T07:30:13.098141+0000","last_became_active":"2024-03-06T07:30:11.048543+0000","last_became_peered":"2024-03-06T07:30:11.048543+0000","last_unstale":"2024-03-06T07:30:13.098141+0000","last_undegraded":"2024-03-06T07:30:13.098141+0000","last_fullsized":"2024-03-06T07:30:13.098141+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:15.476867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186726+0000","last_change":"2024-03-06T07:30:12.087714+0000","last_active":"2024-03-06T07:30:13.186726+0000","last_peered":"2024-03-06T07:30:13.186726+0000","last_clean":"2024-03-06T07:30:13.186726+0000","last_became_active":"2024-03-06T07:30:12.086649+0000","last_became_peered":"2024-03-06T07:30:12.086649+0000","last_unstale":"2024-03-06T07:30:13.186726+0000","last_undegraded":"2024-03-06T07:30:13.186726+0000","last_fullsized":"2024-03-06T07:30:13.186726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:30.729503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094986+0000","last_change":"2024-03-06T07:30:11.054159+0000","last_active":"2024-03-06T07:30:13.094986+0000","last_peered":"2024-03-06T07:30:13.094986+0000","last_clean":"2024-03-06T07:30:13.094986+0000","last_became_active":"2024-03-06T07:30:11.053888+0000","last_became_peered":"2024-03-06T07:30:11.053888+0000","last_unstale":"2024-03-06T07:30:13.094986+0000","last_undegraded":"2024-03-06T07:30:13.094986+0000","last_fullsized":"2024-03-06T07:30:13.094986+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:10.540198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.094866+0000","last_change":"2024-03-06T07:30:12.090265+0000","last_active":"2024-03-06T07:30:13.094866+0000","last_peered":"2024-03-06T07:30:13.094866+0000","last_clean":"2024-03-06T07:30:13.094866+0000","last_became_active":"2024-03-06T07:30:12.089857+0000","last_became_peered":"2024-03-06T07:30:12.089857+0000","last_unstale":"2024-03-06T07:30:13.094866+0000","last_undegraded":"2024-03-06T07:30:13.094866+0000","last_fullsized":"2024-03-06T07:30:13.094866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:54.482169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.098169+0000","last_change":"2024-03-06T07:30:11.057332+0000","last_active":"2024-03-06T07:30:13.098169+0000","last_peered":"2024-03-06T07:30:13.098169+0000","last_clean":"2024-03-06T07:30:13.098169+0000","last_became_active":"2024-03-06T07:30:11.056962+0000","last_became_peered":"2024-03-06T07:30:11.056962+0000","last_unstale":"2024-03-06T07:30:13.098169+0000","last_undegraded":"2024-03-06T07:30:13.098169+0000","last_fullsized":"2024-03-06T07:30:13.098169+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:33.067615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182217+0000","last_change":"2024-03-06T07:30:12.073473+0000","last_active":"2024-03-06T07:30:13.182217+0000","last_peered":"2024-03-06T07:30:13.182217+0000","last_clean":"2024-03-06T07:30:13.182217+0000","last_became_active":"2024-03-06T07:30:12.073083+0000","last_became_peered":"2024-03-06T07:30:12.073083+0000","last_unstale":"2024-03-06T07:30:13.182217+0000","last_undegraded":"2024-03-06T07:30:13.182217+0000","last_fullsized":"2024-03-06T07:30:13.182217+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.826737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184717+0000","last_change":"2024-03-06T07:30:11.052839+0000","last_active":"2024-03-06T07:30:13.184717+0000","last_peered":"2024-03-06T07:30:13.184717+0000","last_clean":"2024-03-06T07:30:13.184717+0000","last_became_active":"2024-03-06T07:30:11.052473+0000","last_became_peered":"2024-03-06T07:30:11.052473+0000","last_unstale":"2024-03-06T07:30:13.184717+0000","last_undegraded":"2024-03-06T07:30:13.184717+0000","last_fullsized":"2024-03-06T07:30:13.184717+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:20.570693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.184377+0000","last_change":"2024-03-06T07:30:12.092705+0000","last_active":"2024-03-06T07:30:13.184377+0000","last_peered":"2024-03-06T07:30:13.184377+0000","last_clean":"2024-03-06T07:30:13.184377+0000","last_became_active":"2024-03-06T07:30:12.092327+0000","last_became_peered":"2024-03-06T07:30:12.092327+0000","last_unstale":"2024-03-06T07:30:13.184377+0000","last_undegraded":"2024-03-06T07:30:13.184377+0000","last_fullsized":"2024-03-06T07:30:13.184377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:30.729503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.182103+0000","last_change":"2024-03-06T07:30:11.053248+0000","last_active":"2024-03-06T07:30:13.182103+0000","last_peered":"2024-03-06T07:30:13.182103+0000","last_clean":"2024-03-06T07:30:13.182103+0000","last_became_active":"2024-03-06T07:30:11.052920+0000","last_became_peered":"2024-03-06T07:30:11.052920+0000","last_unstale":"2024-03-06T07:30:13.182103+0000","last_undegraded":"2024-03-06T07:30:13.182103+0000","last_fullsized":"2024-03-06T07:30:13.182103+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:23.293072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053298+0000","last_change":"2024-03-06T07:30:12.078425+0000","last_active":"2024-03-06T07:30:13.053298+0000","last_peered":"2024-03-06T07:30:13.053298+0000","last_clean":"2024-03-06T07:30:13.053298+0000","last_became_active":"2024-03-06T07:30:12.077931+0000","last_became_peered":"2024-03-06T07:30:12.077931+0000","last_unstale":"2024-03-06T07:30:13.053298+0000","last_undegraded":"2024-03-06T07:30:13.053298+0000","last_fullsized":"2024-03-06T07:30:13.053298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:11.008939+0000","last_clean_scrub_stamp":"2024-03-06T07:30:11.008939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.006663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.186492+0000","last_change":"2024-03-06T07:30:11.063511+0000","last_active":"2024-03-06T07:30:13.186492+0000","last_peered":"2024-03-06T07:30:13.186492+0000","last_clean":"2024-03-06T07:30:13.186492+0000","last_became_active":"2024-03-06T07:30:11.063201+0000","last_became_peered":"2024-03-06T07:30:11.063201+0000","last_unstale":"2024-03-06T07:30:13.186492+0000","last_undegraded":"2024-03-06T07:30:13.186492+0000","last_fullsized":"2024-03-06T07:30:13.186492+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:09.994545+0000","last_clean_scrub_stamp":"2024-03-06T07:30:09.994545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:49.201097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:30:13.053414+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:30:13.053414+0000","last_peered":"2024-03-06T07:30:13.053414+0000","last_clean":"2024-03-06T07:30:13.053414+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:30:13.053414+0000","last_undegraded":"2024-03-06T07:30:13.053414+0000","last_fullsized":"2024-03-06T07:30:13.053414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":970752,"data_stored":735429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":380928,"data_stored":145587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":380928,"data_stored":145643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":380928,"data_stored":145075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":146605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":376832,"data_stored":145041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":389120,"data_stored":146605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":978944,"data_stored":741639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:17.040 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:17.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.298+0000 7f5703200640 1 Processor -- start 2024-03-06T07:30:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.299+0000 7f5703200640 1 -- start start 2024-03-06T07:30:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.300+0000 7f5703200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f56fc15ba20 0x7f56fc066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.300+0000 7f5703200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56fc0ce370 0x7f56fc0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.300+0000 7f5703200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56fc15be00 con 0x7f56fc0afab0 2024-03-06T07:30:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.300+0000 7f5703200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56fc06f2c0 con 0x7f56fc0ce370 2024-03-06T07:30:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.300+0000 7f5703200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56fc0cece0 con 0x7f56fc15ba20 2024-03-06T07:30:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56fc0ce370 0x7f56fc0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56fc0ce370 0x7f56fc0ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49752/0 (socket says 172.21.15.32:49752) 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 -- 172.21.15.32:0/131931576 learned_addr learned my addr 172.21.15.32:0/131931576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5701e00640 1 --2- 172.21.15.32:0/131931576 >> v2:172.21.15.39:3300/0 conn(0x7f56fc15ba20 0x7f56fc066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 -- 172.21.15.32:0/131931576 >> v2:172.21.15.39:3300/0 conn(0x7f56fc15ba20 msgr2=0x7f56fc066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 --2- 172.21.15.32:0/131931576 >> v2:172.21.15.39:3300/0 conn(0x7f56fc15ba20 0x7f56fc066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 -- 172.21.15.32:0/131931576 >> v1:172.21.15.39:6789/0 conn(0x7f56fc0afab0 legacy=0x7f56fc15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.301+0000 7f5702800640 1 -- 172.21.15.32:0/131931576 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc06d4b0 con 0x7f56fc0ce370 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.302+0000 7f5702800640 1 --2- 172.21.15.32:0/131931576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc0ce7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f56fc06f440 tx=0x7f56f802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965a7ee6451b07e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.303+0000 7f5700a00640 1 -- 172.21.15.32:0/131931576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f802c9f0 con 0x7f56fc0ce370 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.303+0000 7f5700a00640 1 -- 172.21.15.32:0/131931576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f802cb90 con 0x7f56fc0ce370 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.303+0000 7f5700a00640 1 -- 172.21.15.32:0/131931576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80343f0 con 0x7f56fc0ce370 2024-03-06T07:30:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 -- 172.21.15.32:0/131931576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 msgr2=0x7f56fc0ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 --2- 172.21.15.32:0/131931576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc0ce7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f56fc06f440 tx=0x7f56f802be50 comp rx=0 tx=0).stop 2024-03-06T07:30:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 -- 172.21.15.32:0/131931576 shutdown_connections 2024-03-06T07:30:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 --2- 172.21.15.32:0/131931576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc0ce7a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 --2- 172.21.15.32:0/131931576 >> v2:172.21.15.39:3300/0 conn(0x7f56fc15ba20 0x7f56fc066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 -- 172.21.15.32:0/131931576 >> 172.21.15.32:0/131931576 conn(0x7f56fc0ccf70 msgr2=0x7f56fc0cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 -- 172.21.15.32:0/131931576 shutdown_connections 2024-03-06T07:30:17.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.304+0000 7f5703200640 1 -- 172.21.15.32:0/131931576 wait complete. 2024-03-06T07:30:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.305+0000 7f5703200640 1 Processor -- start 2024-03-06T07:30:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.306+0000 7f5703200640 1 -- start start 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.306+0000 7f5703200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56fc0afab0 0x7f56fc0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.306+0000 7f5703200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5703200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56fc15ba20 0x7f56fc15d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5703200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc15f360 con 0x7f56fc0ce370 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5703200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc15e730 con 0x7f56fc0afab0 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5703200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc15e420 con 0x7f56fc15ba20 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49758/0 (socket says 172.21.15.32:49758) 2024-03-06T07:30:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 -- 172.21.15.32:0/2029976077 learned_addr learned my addr 172.21.15.32:0/2029976077 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701e00640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56fc0afab0 0x7f56fc0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5702800640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56fc15ba20 0x7f56fc15d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 -- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56fc15ba20 msgr2=0x7f56fc15d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56fc15ba20 0x7f56fc15d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 -- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56fc0afab0 msgr2=0x7f56fc0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56fc0afab0 0x7f56fc0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701400640 1 -- 172.21.15.32:0/2029976077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc1462f0 con 0x7f56fc0ce370 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.307+0000 7f5701e00640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56fc0afab0 0x7f56fc0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.308+0000 7f5701400640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc145db0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00c460 tx=0x7f56ec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.308+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec016020 con 0x7f56fc0ce370 2024-03-06T07:30:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.308+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec012870 con 0x7f56fc0ce370 2024-03-06T07:30:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.308+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec012c60 con 0x7f56fc0ce370 2024-03-06T07:30:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.308+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc0b0aa0 con 0x7f56fc0ce370 2024-03-06T07:30:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.308+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc0b1060 con 0x7f56fc0ce370 2024-03-06T07:30:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.309+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56fc04f330 con 0x7f56fc0ce370 2024-03-06T07:30:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.310+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56ec019020 con 0x7f56fc0ce370 2024-03-06T07:30:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.311+0000 7f56f6000640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56e006b4d0 0x7f56e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.311+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56ec089f50 con 0x7f56fc0ce370 2024-03-06T07:30:17.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.312+0000 7f5701e00640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56e006b4d0 0x7f56e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.312+0000 7f5701e00640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56e006b4d0 0x7f56e006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56f0006fd0 tx=0x7f56f0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.316+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56ec020030 con 0x7f56fc0ce370 2024-03-06T07:30:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.588+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f56fc0ce050 con 0x7f56fc0ce370 2024-03-06T07:30:17.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.592+0000 7f56f6000640 1 -- 172.21.15.32:0/2029976077 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f56fc1462f0 con 0x7f56fc0ce370 2024-03-06T07:30:17.595 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:30:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:30:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:30:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:30:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:17.595 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56e006b4d0 msgr2=0x7f56e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56e006b4d0 0x7f56e006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56f0006fd0 tx=0x7f56f0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 msgr2=0x7f56fc145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc145db0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00c460 tx=0x7f56ec00c920 comp rx=0 tx=0).stop 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 shutdown_connections 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56e006b4d0 0x7f56e006d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56fc15ba20 0x7f56fc15d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56fc0ce370 0x7f56fc145db0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 --2- 172.21.15.32:0/2029976077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56fc0afab0 0x7f56fc0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.596+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 >> 172.21.15.32:0/2029976077 conn(0x7f56fc0ccf70 msgr2=0x7f56fc0aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.597+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 shutdown_connections 2024-03-06T07:30:17.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.597+0000 7f5703200640 1 -- 172.21.15.32:0/2029976077 wait complete. 2024-03-06T07:30:17.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:17.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.822+0000 7ff5dfe00640 1 Processor -- start 2024-03-06T07:30:17.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.822+0000 7ff5dfe00640 1 -- start start 2024-03-06T07:30:17.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5dfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff5e015e0b0 0x7ff5e00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5dfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5dfe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e00b3710 con 0x7ff5e00d0c80 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5dfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e00b09e0 con 0x7ff5e015e0b0 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5dfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00b39f0 con 0x7ff5e00d4a10 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5df400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff5e00d0c80 0x7ff5e00d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53676/0 (socket says 172.21.15.32:53676) 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.824+0000 7ff5df400640 1 -- 172.21.15.32:0/594584060 learned_addr learned my addr 172.21.15.32:0/594584060 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.825+0000 7ff5dea00640 1 --2- 172.21.15.32:0/594584060 >> v2:172.21.15.39:3300/0 conn(0x7ff5e015e0b0 0x7ff5e00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.825+0000 7ff5de000640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.825+0000 7ff5dd600640 1 -- 172.21.15.32:0/594584060 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624933286 0 0) 0x7ff5e00b3710 con 0x7ff5e00d0c80 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.825+0000 7ff5dd600640 1 -- 172.21.15.32:0/594584060 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c4003520 con 0x7ff5e00d0c80 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.826+0000 7ff5dea00640 1 -- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 msgr2=0x7ff5e00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.826+0000 7ff5dea00640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e00d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.826+0000 7ff5dea00640 1 -- 172.21.15.32:0/594584060 >> v1:172.21.15.32:6789/0 conn(0x7ff5e00d0c80 legacy=0x7ff5e00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.826+0000 7ff5dd600640 1 -- 172.21.15.32:0/594584060 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744427715 0 0) 0x7ff5c4003520 con 0x7ff5e00d0c80 2024-03-06T07:30:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.826+0000 7ff5dea00640 1 -- 172.21.15.32:0/594584060 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e00b3cd0 con 0x7ff5e015e0b0 2024-03-06T07:30:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.826+0000 7ff5de000640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.827+0000 7ff5dea00640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00cf310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00097a0 tx=0x7ff5d002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29b635ac4c2335cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dd600640 1 -- 172.21.15.32:0/594584060 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0037070 con 0x7ff5e015e0b0 2024-03-06T07:30:17.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dd600640 1 -- 172.21.15.32:0/594584060 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0030510 con 0x7ff5e015e0b0 2024-03-06T07:30:17.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dd600640 1 -- 172.21.15.32:0/594584060 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80019f0 con 0x7ff5e015e0b0 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 -- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 msgr2=0x7ff5e00cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00cf310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00097a0 tx=0x7ff5d002b790 comp rx=0 tx=0).stop 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 -- 172.21.15.32:0/594584060 shutdown_connections 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/594584060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00cf310 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 -- 172.21.15.32:0/594584060 >> 172.21.15.32:0/594584060 conn(0x7ff5e00d3d30 msgr2=0x7ff5e00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 -- 172.21.15.32:0/594584060 shutdown_connections 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.828+0000 7ff5dfe00640 1 -- 172.21.15.32:0/594584060 wait complete. 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.830+0000 7ff5dfe00640 1 Processor -- start 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.830+0000 7ff5dfe00640 1 -- start start 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 0x7ff5e00cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e0152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00cfc60 con 0x7ff5e00d0c80 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e015e490 con 0x7ff5e015e0b0 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00b54e0 con 0x7ff5e00d4a10 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 0x7ff5e00cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 0x7ff5e00cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49772/0 (socket says 172.21.15.32:49772) 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.831+0000 7ff5dea00640 1 -- 172.21.15.32:0/2847865704 learned_addr learned my addr 172.21.15.32:0/2847865704 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:17.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5dea00640 1 -- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 msgr2=0x7ff5e0152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5de000640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e0152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5df400640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5dea00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e0152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5dea00640 1 -- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 msgr2=0x7ff5e00b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5dea00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00b92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5dea00640 1 -- 172.21.15.32:0/2847865704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e01525c0 con 0x7ff5e00d0c80 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.832+0000 7ff5dea00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 0x7ff5e00cd950 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000c00 tx=0x7ff5d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.833+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0040020 con 0x7ff5e00d0c80 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.833+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0038040 con 0x7ff5e00d0c80 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.833+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0037070 con 0x7ff5e00d0c80 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.833+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e00b8210 con 0x7ff5e00d0c80 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.833+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e00b87d0 con 0x7ff5e00d0c80 2024-03-06T07:30:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.834+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e004f330 con 0x7ff5e00d0c80 2024-03-06T07:30:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.835+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5d0043020 con 0x7ff5e00d0c80 2024-03-06T07:30:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.836+0000 7ff5d6a00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff5c006b4d0 0x7ff5c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.837+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5d00ab6a0 con 0x7ff5e00d0c80 2024-03-06T07:30:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.837+0000 7ff5de000640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff5c006b4d0 0x7ff5c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.837+0000 7ff5de000640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff5c006b4d0 0x7ff5c006d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009560 tx=0x7ff5cc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:17.844+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5d004a030 con 0x7ff5e00d0c80 2024-03-06T07:30:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.088+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5e00d2690 con 0x7ff5e00d0c80 2024-03-06T07:30:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.090+0000 7ff5d6a00640 1 -- 172.21.15.32:0/2847865704 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7ff5d007a530 con 0x7ff5e00d0c80 2024-03-06T07:30:18.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:18.092 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff5c006b4d0 msgr2=0x7ff5c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff5c006b4d0 0x7ff5c006d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009560 tx=0x7ff5cc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 msgr2=0x7ff5e00cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 0x7ff5e00cd950 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000c00 tx=0x7ff5d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 shutdown_connections 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff5c006b4d0 0x7ff5c006d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5e015e0b0 0x7ff5e00b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5e00d4a10 0x7ff5e0152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 --2- 172.21.15.32:0/2847865704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5e00d0c80 0x7ff5e00cd950 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 >> 172.21.15.32:0/2847865704 conn(0x7ff5e00d3d30 msgr2=0x7ff5e006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 shutdown_connections 2024-03-06T07:30:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.092+0000 7ff5dfe00640 1 -- 172.21.15.32:0/2847865704 wait complete. 2024-03-06T07:30:18.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:11.509859+0000', 'modified': '2024-03-06T07:30:14.594308+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5058}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5058': {'gid': 5058, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6837/2686928389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 2686928389}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 2686928389}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5058, 'qdb_cluster': [5058]} 2024-03-06T07:30:18.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:18.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:30:18.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:18.637 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.720+0000 7fa779c00640 1 Processor -- start 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.720+0000 7fa779c00640 1 -- start start 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa779c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa774161670 0x7fa774070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa779c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa77406e240 0x7fa7740711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa779c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740ce9a0 con 0x7fa77406cfe0 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa779c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7740730d0 con 0x7fa77406e240 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa779c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740cec80 con 0x7fa774161670 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa76be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa77406e240 0x7fa7740711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa76be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa77406e240 0x7fa7740711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:38020/0 (socket says 172.21.15.32:38020) 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.723+0000 7fa76be00640 1 -- 172.21.15.32:0/4232846354 learned_addr learned my addr 172.21.15.32:0/4232846354 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.724+0000 7fa772a00640 1 --2- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa774161670 0x7fa774070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.724+0000 7fa76be00640 1 -- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa774161670 msgr2=0x7fa774070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.724+0000 7fa76be00640 1 --2- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa774161670 0x7fa774070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.724+0000 7fa76be00640 1 -- 172.21.15.32:0/4232846354 >> v1:172.21.15.39:6789/0 conn(0x7fa77406cfe0 legacy=0x7fa77406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.724+0000 7fa76be00640 1 -- 172.21.15.32:0/4232846354 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7740cef60 con 0x7fa77406e240 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.725+0000 7fa76be00640 1 --2- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77406e240 0x7fa7740711b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009870 tx=0x7fa75c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ac450fb3232857f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.725+0000 7fa772000640 1 -- 172.21.15.32:0/4232846354 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c034070 con 0x7fa77406e240 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.726+0000 7fa772000640 1 -- 172.21.15.32:0/4232846354 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c030840 con 0x7fa77406e240 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.726+0000 7fa772000640 1 -- 172.21.15.32:0/4232846354 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c030c10 con 0x7fa77406e240 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 -- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77406e240 msgr2=0x7fa7740711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 --2- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77406e240 0x7fa7740711b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009870 tx=0x7fa75c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 -- 172.21.15.32:0/4232846354 shutdown_connections 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 --2- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77406e240 0x7fa7740711b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 --2- 172.21.15.32:0/4232846354 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa774161670 0x7fa774070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 -- 172.21.15.32:0/4232846354 >> 172.21.15.32:0/4232846354 conn(0x7fa77406afb0 msgr2=0x7fa77406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 -- 172.21.15.32:0/4232846354 shutdown_connections 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.727+0000 7fa779c00640 1 -- 172.21.15.32:0/4232846354 wait complete. 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.730+0000 7fa779c00640 1 Processor -- start 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.730+0000 7fa779c00640 1 -- start start 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.731+0000 7fa779c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 0x7fa77415da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.731+0000 7fa779c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa77406e240 0x7fa774157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.731+0000 7fa779c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa774161670 0x7fa77409ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.731+0000 7fa779c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740ce600 con 0x7fa77406cfe0 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.731+0000 7fa779c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740cf520 con 0x7fa774161670 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.732+0000 7fa779c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740ce780 con 0x7fa77406e240 2024-03-06T07:30:18.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.732+0000 7fa772a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 0x7fa77415da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.732+0000 7fa772a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 0x7fa77415da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34816/0 (socket says 172.21.15.32:34816) 2024-03-06T07:30:18.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.732+0000 7fa772a00640 1 -- 172.21.15.32:0/2300812038 learned_addr learned my addr 172.21.15.32:0/2300812038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:18.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.732+0000 7fa76be00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa77406e240 0x7fa774157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.732+0000 7fa773400640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa774161670 0x7fa77409ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.733+0000 7fa772a00640 1 -- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa77406e240 msgr2=0x7fa774157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.733+0000 7fa772a00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa77406e240 0x7fa774157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.733+0000 7fa772a00640 1 -- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa774161670 msgr2=0x7fa77409ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.733+0000 7fa772a00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa774161670 0x7fa77409ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.733+0000 7fa772a00640 1 -- 172.21.15.32:0/2300812038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77415e0a0 con 0x7fa77406cfe0 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.733+0000 7fa772a00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 0x7fa77415da80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d620 tx=0x7fa76400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.734+0000 7fa76aa00640 1 -- 172.21.15.32:0/2300812038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76400b620 con 0x7fa77406cfe0 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.734+0000 7fa76aa00640 1 -- 172.21.15.32:0/2300812038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa764019070 con 0x7fa77406cfe0 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.734+0000 7fa76aa00640 1 -- 172.21.15.32:0/2300812038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76400ba00 con 0x7fa77406cfe0 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.734+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77415e380 con 0x7fa77406cfe0 2024-03-06T07:30:18.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.734+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77415e940 con 0x7fa77406cfe0 2024-03-06T07:30:18.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.737+0000 7fa76aa00640 1 -- 172.21.15.32:0/2300812038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa76400bba0 con 0x7fa77406cfe0 2024-03-06T07:30:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.739+0000 7fa76aa00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa75006b4d0 0x7fa75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.739+0000 7fa76aa00640 1 -- 172.21.15.32:0/2300812038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa76408a560 con 0x7fa77406cfe0 2024-03-06T07:30:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.739+0000 7fa77c999740 1 Processor -- start 2024-03-06T07:30:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.739+0000 7fa76be00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa75006b4d0 0x7fa75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.739+0000 7fa77c999740 1 -- start start 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa76be00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa75006b4d0 0x7fa75006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa75c036000 tx=0x7fa75c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a8497750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa77c999740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 0x5607a84c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa77c999740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607a8401600 con 0x5607a8497c90 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607a83aa930 con 0x5607a84c4b90 2024-03-06T07:30:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607a83fd6a0 con 0x5607a84c47b0 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa773400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 0x5607a84c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa773400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 0x5607a84c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34830/0 (socket says 172.21.15.32:34830) 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.740+0000 7fa773400640 1 -- 172.21.15.32:0/2162394410 learned_addr learned my addr 172.21.15.32:0/2162394410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa772a00640 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a8497750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa76be00640 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa773400640 1 -- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 msgr2=0x5607a8497750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa773400640 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a8497750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa773400640 1 -- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 msgr2=0x5607a84c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa773400640 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa773400640 1 -- 172.21.15.32:0/2162394410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607a81162d0 con 0x5607a8497c90 2024-03-06T07:30:18.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.741+0000 7fa773400640 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 0x5607a84c7a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa76c001f60 tx=0x7fa76c001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16779493afd7581 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.742+0000 7fa763400640 1 -- 172.21.15.32:0/2162394410 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00fe50 con 0x5607a8497c90 2024-03-06T07:30:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.742+0000 7fa763400640 1 -- 172.21.15.32:0/2162394410 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c018070 con 0x5607a8497c90 2024-03-06T07:30:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.742+0000 7fa763400640 1 -- 172.21.15.32:0/2162394410 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c003b70 con 0x5607a8497c90 2024-03-06T07:30:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.742+0000 7fa77c999740 1 -- 172.21.15.32:0/2162394410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 msgr2=0x5607a84c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.742+0000 7fa77c999740 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 0x5607a84c7a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa76c001f60 tx=0x7fa76c001670 comp rx=0 tx=0).stop 2024-03-06T07:30:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 -- 172.21.15.32:0/2162394410 shutdown_connections 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a8497c90 0x5607a84c7a80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 --2- 172.21.15.32:0/2162394410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a8497750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 -- 172.21.15.32:0/2162394410 >> 172.21.15.32:0/2162394410 conn(0x5607a84999c0 msgr2=0x5607a84e2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 -- 172.21.15.32:0/2162394410 shutdown_connections 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 -- 172.21.15.32:0/2162394410 wait complete. 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.743+0000 7fa77c999740 1 -- start start 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.744+0000 7fa77c999740 1 Processor -- start 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 0x5607a84dbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84cad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607a849bf60 con 0x5607a84c47b0 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607a83bdd90 con 0x5607a84c4b90 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607a849b350 con 0x5607a84dc250 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa77c999740 10 client.0 _ll_register_callbacks cb 0x5607a83f9350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 0x5607a84dbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 0x5607a84dbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34842/0 (socket says 172.21.15.32:34842) 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 -- 172.21.15.32:0/3034213741 learned_addr learned my addr 172.21.15.32:0/3034213741 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa76be00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa773400640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 -- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 msgr2=0x5607a84cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 -- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 msgr2=0x5607a84cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.745+0000 7fa772a00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa772a00640 1 -- 172.21.15.32:0/3034213741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607a81e06b0 con 0x5607a84c47b0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa76be00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa772a00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 0x5607a84dbd10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d5f0 tx=0x7fa764058650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ee84d312ec09aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa757e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa757e00640 1 -- 172.21.15.32:0/3034213741 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764061350 con 0x5607a84c47b0 2024-03-06T07:30:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa757e00640 1 -- 172.21.15.32:0/3034213741 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa76408a0c0 con 0x5607a84c47b0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa76c018070 con 0x5607a84c47b0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa757e00640 1 -- 172.21.15.32:0/3034213741 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa76408a2c0 con 0x5607a84c47b0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa77c999740 10 client.5241 fetch_fsmap learned FSMap version 17 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.746+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5607a84ca860 con 0x5607a84c47b0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa757e00640 1 -- 172.21.15.32:0/3034213741 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c018070 con 0x5607a84c47b0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa757e00640 1 -- 172.21.15.32:0/3034213741 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa764013070 con 0x5607a84c47b0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa77c999740 10 client.5241 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa77c999740 10 client.5241 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa77c999740 20 client.5241 populate_metadata read hostname 'smithi032' 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa77c999740 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84c6220 0x5607a8500c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa77c999740 4 client.5241 mds_command: new command op to 5058 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 --> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607a840eb40 con 0x5607a84c6220 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.747+0000 7fa76be00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84c6220 0x5607a8500c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.748+0000 7fa757e00640 10 client.5241 ms_handle_connect on v2:172.21.15.32:6834/2686928389 2024-03-06T07:30:18.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.748+0000 7fa76be00640 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84c6220 0x5607a8500c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ccd29b80fa94118 server_cookie=d4357737b379885a in_seq=0 out_seq=0 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa757e00640 1 -- 172.21.15.32:0/3034213741 <== mds.0 v2:172.21.15.32:6834/2686928389 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5607a840eb40 con 0x5607a84c6220 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa757e00640 10 client.5241 handle_command_reply: tid=0 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 client.5241 shutdown 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 msgr2=0x5607a84dbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 0x5607a84dbd10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d5f0 tx=0x7fa764058650 comp rx=0 tx=0).stop 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 shutdown_connections 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84c6220 0x5607a8500c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 --2- 172.21.15.32:0/3034213741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c47b0 0x5607a84dbd10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.749+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 >> 172.21.15.32:0/3034213741 conn(0x5607a8497c90 msgr2=0x5607a8496eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.750+0000 7fa757e00640 0 client.5241 ms_handle_reset on v2:172.21.15.32:6834/2686928389 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.750+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 shutdown_connections 2024-03-06T07:30:18.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.750+0000 7fa77c999740 1 -- 172.21.15.32:0/3034213741 wait complete. 2024-03-06T07:30:18.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.750+0000 7fa77c999740 20 client.5241 trim_cache size 0 max 16384 2024-03-06T07:30:18.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.785+0000 7fa77c999740 1 Processor -- start 2024-03-06T07:30:18.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.785+0000 7fa77c999740 1 -- start start 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a84bea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84d9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa77c999740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 0x5607a8504c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa77c999740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77409d3a0 con 0x5607a84dc250 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77409c7a0 con 0x5607a84c4b90 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77409cda0 con 0x5607a84c47b0 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa773400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 0x5607a8504c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.787+0000 7fa773400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 0x5607a8504c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34844/0 (socket says 172.21.15.32:34844) 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 -- 172.21.15.32:0/4006541907 learned_addr learned my addr 172.21.15.32:0/4006541907 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa772a00640 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a84bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa76be00640 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84d9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 -- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 msgr2=0x5607a84bea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a84bea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 -- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 msgr2=0x5607a84d9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84d9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 -- 172.21.15.32:0/4006541907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607a83c6720 con 0x5607a84dc250 2024-03-06T07:30:18.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.788+0000 7fa773400640 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 0x5607a8504c60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa76c001b60 tx=0x7fa76c00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d99bb7959694b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.789+0000 7fa762a00640 1 -- 172.21.15.32:0/4006541907 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c006d50 con 0x5607a84dc250 2024-03-06T07:30:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.789+0000 7fa762a00640 1 -- 172.21.15.32:0/4006541907 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c021070 con 0x5607a84dc250 2024-03-06T07:30:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.789+0000 7fa762a00640 1 -- 172.21.15.32:0/4006541907 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c003bb0 con 0x5607a84dc250 2024-03-06T07:30:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 -- 172.21.15.32:0/4006541907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 msgr2=0x5607a8504c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 0x5607a8504c60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa76c001b60 tx=0x7fa76c00a000 comp rx=0 tx=0).stop 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 -- 172.21.15.32:0/4006541907 shutdown_connections 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84dc250 0x5607a8504c60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c4b90 0x5607a84d9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 --2- 172.21.15.32:0/4006541907 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84c47b0 0x5607a84bea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 -- 172.21.15.32:0/4006541907 >> 172.21.15.32:0/4006541907 conn(0x5607a8497c90 msgr2=0x5607a84929e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 -- 172.21.15.32:0/4006541907 shutdown_connections 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.790+0000 7fa77c999740 1 -- 172.21.15.32:0/4006541907 wait complete. 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.791+0000 7fa77c999740 1 -- start start 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.791+0000 7fa77c999740 1 Processor -- start 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c47b0 0x5607a84dfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 0x5607a85527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84fb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607a84c60a0 con 0x5607a84c4b90 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607a84c0430 con 0x5607a84c47b0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607a84c6610 con 0x5607a84dc250 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.792+0000 7fa77c999740 10 client.0 _ll_register_callbacks cb 0x7fa76400be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 0x5607a85527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 0x5607a85527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34848/0 (socket says 172.21.15.32:34848) 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 -- 172.21.15.32:0/3085832434 learned_addr learned my addr 172.21.15.32:0/3085832434 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa772a00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c47b0 0x5607a84dfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa773400640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 -- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 msgr2=0x5607a84fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84fb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 -- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c47b0 msgr2=0x5607a84dfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c47b0 0x5607a84dfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa76be00640 1 -- 172.21.15.32:0/3085832434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607a849bf40 con 0x5607a84c4b90 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa773400640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.793+0000 7fa772a00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c47b0 0x5607a84dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa76be00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 0x5607a85527c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009990 tx=0x7fa75c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8661077881b036 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa757e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa757e00640 1 -- 172.21.15.32:0/3085832434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c05a070 con 0x5607a84c4b90 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa757e00640 1 -- 172.21.15.32:0/3085832434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa75c044c00 con 0x5607a84c4b90 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa76c021070 con 0x5607a84c4b90 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa757e00640 1 -- 172.21.15.32:0/3085832434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa75c065050 con 0x5607a84c4b90 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa77c999740 10 client.5247 fetch_fsmap learned FSMap version 17 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5607a849c970 con 0x5607a84c4b90 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.794+0000 7fa757e00640 1 -- 172.21.15.32:0/3085832434 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c021070 con 0x5607a84c4b90 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa757e00640 1 -- 172.21.15.32:0/3085832434 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa75c068020 con 0x5607a84c4b90 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa77c999740 10 client.5247 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa77c999740 10 client.5247 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa77c999740 20 client.5247 populate_metadata read hostname 'smithi032' 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa77c999740 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84bef70 0x5607a84d7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa77c999740 4 client.5247 mds_command: new command op to 5058 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 --> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5607a8408e90 con 0x5607a84bef70 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.795+0000 7fa772a00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84bef70 0x5607a84d7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa772a00640 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84bef70 0x5607a84d7ed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b17361f556979b3 server_cookie=731efc03a1ec61d7 in_seq=0 out_seq=0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa757e00640 10 client.5247 ms_handle_connect on v2:172.21.15.32:6834/2686928389 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa757e00640 1 -- 172.21.15.32:0/3085832434 <== mds.0 v2:172.21.15.32:6834/2686928389 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5607a8408e90 con 0x5607a84bef70 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa757e00640 10 client.5247 handle_command_reply: tid=0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 client.5247 shutdown 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 msgr2=0x5607a85527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 0x5607a85527c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009990 tx=0x7fa75c02e000 comp rx=0 tx=0).stop 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 shutdown_connections 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:6834/2686928389,v1:172.21.15.32:6837/2686928389] conn(0x5607a84bef70 0x5607a84d7ed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5607a84dc250 0x5607a84fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5607a84c4b90 0x5607a85527c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 --2- 172.21.15.32:0/3085832434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5607a84c47b0 0x5607a84dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 >> 172.21.15.32:0/3085832434 conn(0x5607a8497c90 msgr2=0x5607a84929e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.796+0000 7fa757e00640 0 client.5247 ms_handle_reset on v2:172.21.15.32:6834/2686928389 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 shutdown_connections 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa77c999740 1 -- 172.21.15.32:0/3085832434 wait complete. 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa77c999740 20 client.5247 trim_cache size 0 max 16384 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5058, 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.7287963499999996, 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 17, 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 34, 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 34, 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 15.725319302999999 2024-03-06T07:30:18.803 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:30:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa75006b4d0 msgr2=0x7fa75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa779c00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa75006b4d0 0x7fa75006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa75c036000 tx=0x7fa75c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 msgr2=0x7fa77415da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.797+0000 7fa779c00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 0x7fa77415da80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d620 tx=0x7fa76400dae0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 shutdown_connections 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa75006b4d0 0x7fa75006d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa774161670 0x7fa77409ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa77406e240 0x7fa774157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 --2- 172.21.15.32:0/2300812038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77406cfe0 0x7fa77415da80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 >> 172.21.15.32:0/2300812038 conn(0x7fa77406afb0 msgr2=0x7fa7740b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 shutdown_connections 2024-03-06T07:30:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:18.798+0000 7fa779c00640 1 -- 172.21.15.32:0/2300812038 wait complete. 2024-03-06T07:30:18.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.067+0000 7f3c0a400640 1 Processor -- start 2024-03-06T07:30:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.067+0000 7f3c0a400640 1 -- start start 2024-03-06T07:30:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.068+0000 7f3c0a400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c0406b6c0 0x7f3c040b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.068+0000 7f3c0a400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c040b3da0 con 0x7f3c0415e1c0 2024-03-06T07:30:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.068+0000 7f3c0a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c040aff60 con 0x7f3c0406f6b0 2024-03-06T07:30:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.068+0000 7f3c0a400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c040b4010 con 0x7f3c0406b6c0 2024-03-06T07:30:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c0406b6c0 0x7f3c040b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c03400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c0415e1c0 0x7f3c0406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60396/0 (socket says 172.21.15.32:60396) 2024-03-06T07:30:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c03400640 1 -- 172.21.15.32:0/2932518195 learned_addr learned my addr 172.21.15.32:0/2932518195 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c0406b6c0 0x7f3c040b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34856/0 (socket says 172.21.15.32:34856) 2024-03-06T07:30:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c02a00640 1 -- 172.21.15.32:0/2932518195 >> v1:172.21.15.39:6789/0 conn(0x7f3c0406f6b0 legacy=0x7f3c0406da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c02a00640 1 -- 172.21.15.32:0/2932518195 >> v1:172.21.15.32:6789/0 conn(0x7f3c0415e1c0 legacy=0x7f3c0406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c02000640 1 -- 172.21.15.32:0/2932518195 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482056726 0 0) 0x7f3c040b3da0 con 0x7f3c0415e1c0 2024-03-06T07:30:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.069+0000 7f3c02a00640 1 -- 172.21.15.32:0/2932518195 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c04072870 con 0x7f3c0406b6c0 2024-03-06T07:30:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.071+0000 7f3c02a00640 1 --2- 172.21.15.32:0/2932518195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406b6c0 0x7f3c040b0310 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00097a0 tx=0x7f3bf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cba6701f09994c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.071+0000 7f3c02000640 1 -- 172.21.15.32:0/2932518195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0035020 con 0x7f3c0406b6c0 2024-03-06T07:30:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.071+0000 7f3c02000640 1 -- 172.21.15.32:0/2932518195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0031830 con 0x7f3c0406b6c0 2024-03-06T07:30:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.071+0000 7f3c02000640 1 -- 172.21.15.32:0/2932518195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0031c00 con 0x7f3c0406b6c0 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 -- 172.21.15.32:0/2932518195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406b6c0 msgr2=0x7f3c040b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2932518195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406b6c0 0x7f3c040b0310 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00097a0 tx=0x7f3bf002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 -- 172.21.15.32:0/2932518195 shutdown_connections 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2932518195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406b6c0 0x7f3c040b0310 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 -- 172.21.15.32:0/2932518195 >> 172.21.15.32:0/2932518195 conn(0x7f3c040b2a00 msgr2=0x7f3c040b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 -- 172.21.15.32:0/2932518195 shutdown_connections 2024-03-06T07:30:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.072+0000 7f3c0a400640 1 -- 172.21.15.32:0/2932518195 wait complete. 2024-03-06T07:30:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.074+0000 7f3c0a400640 1 Processor -- start 2024-03-06T07:30:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.074+0000 7f3c0a400640 1 -- start start 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c0a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c0406b6c0 0x7f3c0415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c0a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 0x7f3c0414b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c0a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c0415e1c0 0x7f3c04153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c0a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c04072b50 con 0x7f3c0406f6b0 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c0a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0406e150 con 0x7f3c0406b6c0 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c0a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0415ee10 con 0x7f3c0415e1c0 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.075+0000 7f3c02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 0x7f3c0414b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 0x7f3c0414b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34858/0 (socket says 172.21.15.32:34858) 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 -- 172.21.15.32:0/2286128704 learned_addr learned my addr 172.21.15.32:0/2286128704 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c03400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c0406b6c0 0x7f3c0415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c03e00640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c0415e1c0 0x7f3c04153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 -- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c0415e1c0 msgr2=0x7f3c04153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c0415e1c0 0x7f3c04153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 -- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c0406b6c0 msgr2=0x7f3c0415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c0406b6c0 0x7f3c0415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 -- 172.21.15.32:0/2286128704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0415f0f0 con 0x7f3c0406f6b0 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c03e00640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c0415e1c0 0x7f3c04153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.076+0000 7f3c02a00640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 0x7f3c0414b6f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3bf002bee0 tx=0x7f3bf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.077+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf003ba80 con 0x7f3c0406f6b0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.077+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf003bc20 con 0x7f3c0406f6b0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.077+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0415f3d0 con 0x7f3c0406f6b0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.077+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00453f0 con 0x7f3c0406f6b0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.077+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0415eae0 con 0x7f3c0406f6b0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.078+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0404f3a0 con 0x7f3c0406f6b0 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.079+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bf0038020 con 0x7f3c0406f6b0 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.081+0000 7f3bfb400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.081+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bf00ac070 con 0x7f3c0406f6b0 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.081+0000 7f3c03400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.082+0000 7f3c03400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000f60 tx=0x7f3bf4007c00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.086+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3bf0040070 con 0x7f3c0406f6b0 2024-03-06T07:30:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.376+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3c040b5b20 con 0x7f3c0406f6b0 2024-03-06T07:30:19.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.377+0000 7f3bfb400640 1 -- 172.21.15.32:0/2286128704 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3c040aff60 con 0x7f3c0406f6b0 2024-03-06T07:30:19.379 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 msgr2=0x7f3be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000f60 tx=0x7f3bf4007c00 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 msgr2=0x7f3c0414b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 0x7f3c0414b6f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3bf002bee0 tx=0x7f3bf00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 shutdown_connections 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c0415e1c0 0x7f3c04153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c0406f6b0 0x7f3c0414b6f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 --2- 172.21.15.32:0/2286128704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c0406b6c0 0x7f3c0415e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.379+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 >> 172.21.15.32:0/2286128704 conn(0x7f3c040b2a00 msgr2=0x7f3c0406e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.380+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 shutdown_connections 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:19.380+0000 7f3c0a400640 1 -- 172.21.15.32:0/2286128704 wait complete. 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:19.401 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:19.402 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:19.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.402 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:30:19.448 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:19.448 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:19.448 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:19.448 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42318sec preferred_lft 42318sec 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.449 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:30:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.451 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:19.586 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:30:19.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.591 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:30:19.637 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:30:19.637 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:19.637 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:30:19.637 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:30:19.637 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:19.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.638 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:30:19.638 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:30:19.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:30:19.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:30:19.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:30:19.638 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:30:19.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.810 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:30:19.858 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:19.858 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:19.859 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:19.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:30:19.931 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:19.931 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:19.970 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:19.970 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:19.970 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:20.058 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:20.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:20.196 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:20.196 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:20.228 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:20.228 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:20.228 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:30:20.228 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:30:20.228 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:30:20.230 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:20.230 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:20.234 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:20.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:20.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.317 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:20.385 INFO:teuthology.orchestra.run.smithi032.stdout:client5259 2024-03-06T07:30:20.386 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:30:20.386 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> subdir = "delete_me" 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> size = 1024 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> file_count = 1000 2024-03-06T07:30:20.441 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:30:20.442 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(0, file_count): 2024-03-06T07:30:20.442 DEBUG:teuthology.orchestra.run.smithi032:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:30:20.442 DEBUG:teuthology.orchestra.run.smithi032:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:30:20.442 DEBUG:teuthology.orchestra.run.smithi032:> f.write(size * '"'"'x'"'"') 2024-03-06T07:30:20.442 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:30:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:21.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.864+0000 7fd64b800640 1 Processor -- start 2024-03-06T07:30:21.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.864+0000 7fd64b800640 1 -- start start 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd64b800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6440cfd90 0x7fd6440d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd64b800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd6440d2760 0x7fd6440b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd64b800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6440cea50 con 0x7fd644161b40 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd64b800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6440b5770 con 0x7fd6440d2760 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd64b800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440cf900 con 0x7fd6440cfd90 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd648e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd644161b40 0x7fd6440b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60408/0 (socket says 172.21.15.32:60408) 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd648e00640 1 -- 172.21.15.32:0/1713604103 learned_addr learned my addr 172.21.15.32:0/1713604103 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd649800640 1 --2- 172.21.15.32:0/1713604103 >> v2:172.21.15.32:3300/0 conn(0x7fd6440d2760 0x7fd6440b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.866+0000 7fd643e00640 1 --2- 172.21.15.32:0/1713604103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6440cfd90 0x7fd6440d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.867+0000 7fd649800640 1 -- 172.21.15.32:0/1713604103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6440cfd90 msgr2=0x7fd6440d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.867+0000 7fd649800640 1 --2- 172.21.15.32:0/1713604103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6440cfd90 0x7fd6440d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.867+0000 7fd649800640 1 -- 172.21.15.32:0/1713604103 >> v1:172.21.15.32:6789/0 conn(0x7fd644161b40 legacy=0x7fd6440b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.867+0000 7fd649800640 1 -- 172.21.15.32:0/1713604103 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6440cecf0 con 0x7fd6440d2760 2024-03-06T07:30:21.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.868+0000 7fd649800640 1 --2- 172.21.15.32:0/1713604103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440b3740 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009d90 tx=0x7fd63c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105902f7c640161c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.868+0000 7fd643400640 1 -- 172.21.15.32:0/1713604103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c039070 con 0x7fd6440d2760 2024-03-06T07:30:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.868+0000 7fd643400640 1 -- 172.21.15.32:0/1713604103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63c02cc60 con 0x7fd6440d2760 2024-03-06T07:30:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.868+0000 7fd643400640 1 -- 172.21.15.32:0/1713604103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c034660 con 0x7fd6440d2760 2024-03-06T07:30:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 -- 172.21.15.32:0/1713604103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 msgr2=0x7fd6440b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 --2- 172.21.15.32:0/1713604103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440b3740 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009d90 tx=0x7fd63c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 -- 172.21.15.32:0/1713604103 shutdown_connections 2024-03-06T07:30:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 --2- 172.21.15.32:0/1713604103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440b3740 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 --2- 172.21.15.32:0/1713604103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6440cfd90 0x7fd6440d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 -- 172.21.15.32:0/1713604103 >> 172.21.15.32:0/1713604103 conn(0x7fd64409d220 msgr2=0x7fd6440d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 -- 172.21.15.32:0/1713604103 shutdown_connections 2024-03-06T07:30:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.869+0000 7fd64b800640 1 -- 172.21.15.32:0/1713604103 wait complete. 2024-03-06T07:30:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.871+0000 7fd64b800640 1 Processor -- start 2024-03-06T07:30:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.872+0000 7fd64b800640 1 -- start start 2024-03-06T07:30:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd64b800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6440cfd90 0x7fd6440ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd64b800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd64b800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd644161b40 0x7fd64406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd64b800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440cf590 con 0x7fd6440d2760 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd64b800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440cf2b0 con 0x7fd6440cfd90 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd64b800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440b2ff0 con 0x7fd644161b40 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd643e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd643e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34866/0 (socket says 172.21.15.32:34866) 2024-03-06T07:30:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd643e00640 1 -- 172.21.15.32:0/3878339762 learned_addr learned my addr 172.21.15.32:0/3878339762 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd648e00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6440cfd90 0x7fd6440ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.873+0000 7fd649800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd644161b40 0x7fd64406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd643e00640 1 -- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd644161b40 msgr2=0x7fd64406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd643e00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd644161b40 0x7fd64406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd643e00640 1 -- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6440cfd90 msgr2=0x7fd6440ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd643e00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6440cfd90 0x7fd6440ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd643e00640 1 -- 172.21.15.32:0/3878339762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6440c8b80 con 0x7fd6440d2760 2024-03-06T07:30:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd643e00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440c8410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd63000d5f0 tx=0x7fd63000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6300125f0 con 0x7fd6440d2760 2024-03-06T07:30:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630019070 con 0x7fd6440d2760 2024-03-06T07:30:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6300129d0 con 0x7fd6440d2760 2024-03-06T07:30:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.874+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6440c8e60 con 0x7fd6440d2760 2024-03-06T07:30:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.875+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6440c9420 con 0x7fd6440d2760 2024-03-06T07:30:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.876+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64404f3a0 con 0x7fd6440d2760 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.877+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd630012b70 con 0x7fd6440d2760 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.878+0000 7fd640c00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62406b4d0 0x7fd62406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.878+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd63008a690 con 0x7fd6440d2760 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.878+0000 7fd648e00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62406b4d0 0x7fd62406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.879+0000 7fd648e00640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62406b4d0 0x7fd62406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd638000f60 tx=0x7fd638009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:21.883+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd630059470 con 0x7fd6440d2760 2024-03-06T07:30:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.128+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6440d1df0 con 0x7fd6440d2760 2024-03-06T07:30:22.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.129+0000 7fd640c00640 1 -- 172.21.15.32:0/3878339762 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fd630058d70 con 0x7fd6440d2760 2024-03-06T07:30:22.132 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:22.132 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:22.132 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.132+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62406b4d0 msgr2=0x7fd62406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.132+0000 7fd64b800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62406b4d0 0x7fd62406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd638000f60 tx=0x7fd638009040 comp rx=0 tx=0).stop 2024-03-06T07:30:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.132+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 msgr2=0x7fd6440c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440c8410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd63000d5f0 tx=0x7fd63000dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 shutdown_connections 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd62406b4d0 0x7fd62406d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd644161b40 0x7fd64406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6440d2760 0x7fd6440c8410 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 --2- 172.21.15.32:0/3878339762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6440cfd90 0x7fd6440ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 >> 172.21.15.32:0/3878339762 conn(0x7fd64409d220 msgr2=0x7fd6440b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 shutdown_connections 2024-03-06T07:30:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22.133+0000 7fd64b800640 1 -- 172.21.15.32:0/3878339762 wait complete. 2024-03-06T07:30:22.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:30:22.759 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:30:22.759 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:22.759 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:30:22.760 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:30:22.760 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-03-06T07:30:22.760 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:30:22.760 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:30:22.760 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:30:22.869 INFO:teuthology.orchestra.run.smithi032.stdout:1099511627776 2024-03-06T07:30:22.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:22.882 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:30:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.027+0000 7f1ded241f40 1 Processor -- start 2024-03-06T07:30:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.028+0000 7f1ded241f40 1 -- start start 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1ded241f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f26ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1ded241f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 0x55699f282620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1ded241f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f268300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1ded241f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55699f1aca50 con 0x55699f1f9940 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1ded241f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55699f153fd0 con 0x55699f1ab4b0 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1ded241f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55699f1abdd0 con 0x55699f1ab0d0 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f268300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f268300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34872/0 (socket says 172.21.15.32:34872) 2024-03-06T07:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 -- 172.21.15.32:0/3947103881 learned_addr learned my addr 172.21.15.32:0/3947103881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1de9c00640 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f26ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 -- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 msgr2=0x55699f26ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f26ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 -- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 msgr2=0x55699f282620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 0x55699f282620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.029+0000 7f1dea600640 1 -- 172.21.15.32:0/3947103881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55699f2872a0 con 0x55699f1f9940 2024-03-06T07:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.030+0000 7f1de9c00640 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f26ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.030+0000 7f1dea600640 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f268300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1de4002880 tx=0x7f1de4002d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17684b9d22d7cdb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.030+0000 7f1de3400640 1 -- 172.21.15.32:0/3947103881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4016020 con 0x55699f1f9940 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.030+0000 7f1de3400640 1 -- 172.21.15.32:0/3947103881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400d9c0 con 0x55699f1f9940 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.030+0000 7f1de3400640 1 -- 172.21.15.32:0/3947103881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400ddb0 con 0x55699f1f9940 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 -- 172.21.15.32:0/3947103881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 msgr2=0x55699f268300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f268300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1de4002880 tx=0x7f1de4002d40 comp rx=0 tx=0).stop 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 -- 172.21.15.32:0/3947103881 shutdown_connections 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f268300 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 0x55699f282620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3947103881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f26ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 -- 172.21.15.32:0/3947103881 >> 172.21.15.32:0/3947103881 conn(0x55699f1dcc10 msgr2=0x55699f1dcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 -- 172.21.15.32:0/3947103881 shutdown_connections 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.031+0000 7f1ded241f40 1 -- 172.21.15.32:0/3947103881 wait complete. 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.032+0000 7f1ded241f40 1 Processor -- start 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.032+0000 7f1ded241f40 1 -- start start 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1ded241f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f266e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1ded241f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 0x55699f2802c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1ded241f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f27e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1ded241f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55699f1de8e0 con 0x55699f1f9940 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1ded241f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55699f287580 con 0x55699f1ab4b0 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1ded241f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55699f1debc0 con 0x55699f1ab0d0 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1dea600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f27e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1dea600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f27e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34880/0 (socket says 172.21.15.32:34880) 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.033+0000 7f1dea600640 1 -- 172.21.15.32:0/3164220798 learned_addr learned my addr 172.21.15.32:0/3164220798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1de9c00640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f266e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1de9200640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 0x55699f2802c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1dea600640 1 -- 172.21.15.32:0/3164220798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 msgr2=0x55699f266e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1dea600640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55699f1ab0d0 0x55699f266e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1dea600640 1 -- 172.21.15.32:0/3164220798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 msgr2=0x55699f2802c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1dea600640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55699f1ab4b0 0x55699f2802c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1dea600640 1 -- 172.21.15.32:0/3164220798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55699f268b80 con 0x55699f1f9940 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.034+0000 7f1dea600640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f27e9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55699f1deea0 tx=0x7f1de40033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.035+0000 7f1de2000640 1 -- 172.21.15.32:0/3164220798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4026480 con 0x55699f1f9940 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.035+0000 7f1de2000640 1 -- 172.21.15.32:0/3164220798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de4026620 con 0x55699f1f9940 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.035+0000 7f1de2000640 1 -- 172.21.15.32:0/3164220798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de40269f0 con 0x55699f1f9940 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.035+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55699f268e20 con 0x55699f1f9940 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.035+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55699f26d9c0 con 0x55699f1f9940 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.037+0000 7f1de2000640 1 -- 172.21.15.32:0/3164220798 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1de4019020 con 0x55699f1f9940 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.038+0000 7f1de2000640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1dcc069eb0 0x7f1dcc06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.038+0000 7f1de2000640 1 -- 172.21.15.32:0/3164220798 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1de408b060 con 0x55699f1f9940 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.038+0000 7f1de9c00640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1dcc069eb0 0x7f1dcc06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.038+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x55699f280800 0x55699f27f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.038+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v9 -- 0x55699f3208b0 con 0x55699f280800 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.038+0000 7f1de9200640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x55699f280800 0x55699f27f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.039+0000 7f1de9c00640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1dcc069eb0 0x7f1dcc06c370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55699f1dc730 tx=0x7f1ddc0035e0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.039+0000 7f1de9200640 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x55699f280800 0x55699f27f410 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1de9200640 1 -- 172.21.15.32:0/3164220798 <== osd.3 v2:172.21.15.32:6806/238384581 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f1dd80527f0 con 0x55699f280800 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:30:22.000000+0000, size 0 2024-03-06T07:30:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x55699f280800 msgr2=0x55699f27f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x55699f280800 0x55699f27f410 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1dcc069eb0 msgr2=0x7f1dcc06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1dcc069eb0 0x7f1dcc06c370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55699f1dc730 tx=0x7f1ddc0035e0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 msgr2=0x55699f27e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 --2- 172.21.15.32:0/3164220798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55699f1f9940 0x55699f27e9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55699f1deea0 tx=0x7f1de40033a0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1dea600640 1 -- 172.21.15.32:0/3164220798 reap_dead start 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 shutdown_connections 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 >> 172.21.15.32:0/3164220798 conn(0x55699f1dcc10 msgr2=0x55699f1dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 shutdown_connections 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:23.040+0000 7f1ded241f40 1 -- 172.21.15.32:0/3164220798 wait complete. 2024-03-06T07:30:23.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:23.047 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:30:25.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.972+0000 7fe08ea00640 1 Processor -- start 2024-03-06T07:30:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.972+0000 7fe08ea00640 1 -- start start 2024-03-06T07:30:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880af580 0x7fe08815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0880cdc20 0x7fe0880b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08815cc40 con 0x7fe0880af1a0 2024-03-06T07:30:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08815cf20 con 0x7fe0880cdc20 2024-03-06T07:30:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08815d200 con 0x7fe0880af580 2024-03-06T07:30:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0880cdc20 0x7fe0880b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0880cdc20 0x7fe0880b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34886/0 (socket says 172.21.15.32:34886) 2024-03-06T07:30:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ca00640 1 -- 172.21.15.32:0/1280681330 learned_addr learned my addr 172.21.15.32:0/1280681330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe07fe00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880af580 0x7fe08815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.974+0000 7fe08ca00640 1 -- 172.21.15.32:0/1280681330 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880af580 msgr2=0x7fe08815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.975+0000 7fe08ca00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880af580 0x7fe08815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.975+0000 7fe08ca00640 1 -- 172.21.15.32:0/1280681330 >> v1:172.21.15.39:6789/0 conn(0x7fe0880af1a0 legacy=0x7fe0880b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.975+0000 7fe08ca00640 1 -- 172.21.15.32:0/1280681330 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08815d4e0 con 0x7fe0880cdc20 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.975+0000 7fe07fe00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880af580 0x7fe08815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.976+0000 7fe08ca00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880cdc20 0x7fe0880b2dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe080009e50 tx=0x7fe08002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b53b8031d5e0dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.976+0000 7fe087400640 1 -- 172.21.15.32:0/1280681330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08002c9f0 con 0x7fe0880cdc20 2024-03-06T07:30:25.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.976+0000 7fe087400640 1 -- 172.21.15.32:0/1280681330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08002cb90 con 0x7fe0880cdc20 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.976+0000 7fe087400640 1 -- 172.21.15.32:0/1280681330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0800343f0 con 0x7fe0880cdc20 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 -- 172.21.15.32:0/1280681330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880cdc20 msgr2=0x7fe0880b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880cdc20 0x7fe0880b2dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe080009e50 tx=0x7fe08002be50 comp rx=0 tx=0).stop 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 -- 172.21.15.32:0/1280681330 shutdown_connections 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880cdc20 0x7fe0880b2dd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 --2- 172.21.15.32:0/1280681330 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880af580 0x7fe08815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 -- 172.21.15.32:0/1280681330 >> 172.21.15.32:0/1280681330 conn(0x7fe08806afb0 msgr2=0x7fe08806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 -- 172.21.15.32:0/1280681330 shutdown_connections 2024-03-06T07:30:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.977+0000 7fe08ea00640 1 -- 172.21.15.32:0/1280681330 wait complete. 2024-03-06T07:30:25.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.980+0000 7fe08ea00640 1 Processor -- start 2024-03-06T07:30:25.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.980+0000 7fe08ea00640 1 -- start start 2024-03-06T07:30:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 0x7fe0880ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0880af580 0x7fe0880cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880cdc20 0x7fe08809cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880b36a0 con 0x7fe0880af1a0 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880b3520 con 0x7fe0880af580 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880b3820 con 0x7fe0880cdc20 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 0x7fe0880ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 0x7fe0880ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34892/0 (socket says 172.21.15.32:34892) 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe087e00640 1 -- 172.21.15.32:0/870459225 learned_addr learned my addr 172.21.15.32:0/870459225 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe07fe00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0880af580 0x7fe0880cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.982+0000 7fe08ca00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880cdc20 0x7fe08809cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe087e00640 1 -- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880cdc20 msgr2=0x7fe08809cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe087e00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880cdc20 0x7fe08809cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe087e00640 1 -- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0880af580 msgr2=0x7fe0880cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe087e00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0880af580 0x7fe0880cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe087e00640 1 -- 172.21.15.32:0/870459225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0881f3830 con 0x7fe0880af1a0 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe08ca00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880cdc20 0x7fe08809cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe087e00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 0x7fe0880ceaa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe07800c430 tx=0x7fe07800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078011790 con 0x7fe0880af1a0 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe078017070 con 0x7fe0880af1a0 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078011b70 con 0x7fe0880af1a0 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.983+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08806cbb0 con 0x7fe0880af1a0 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.984+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08806d170 con 0x7fe0880af1a0 2024-03-06T07:30:25.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.985+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08804f3a0 con 0x7fe0880af1a0 2024-03-06T07:30:25.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.986+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe078011d10 con 0x7fe0880af1a0 2024-03-06T07:30:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.987+0000 7fe084c00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe06406b4d0 0x7fe06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.987+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe07808a080 con 0x7fe0880af1a0 2024-03-06T07:30:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.987+0000 7fe07fe00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe06406b4d0 0x7fe06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:25.988+0000 7fe07fe00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe06406b4d0 0x7fe06406d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe07000a990 tx=0x7fe070008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.001+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe078058ef0 con 0x7fe0880af1a0 2024-03-06T07:30:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.515+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe08809da60 con 0x7fe0880af1a0 2024-03-06T07:30:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.516+0000 7fe084c00640 1 -- 172.21.15.32:0/870459225 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fe0780587f0 con 0x7fe0880af1a0 2024-03-06T07:30:26.518 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:26.519 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:26.519 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe06406b4d0 msgr2=0x7fe06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe06406b4d0 0x7fe06406d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe07000a990 tx=0x7fe070008040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 msgr2=0x7fe0880ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 0x7fe0880ceaa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe07800c430 tx=0x7fe07800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 shutdown_connections 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe06406b4d0 0x7fe06406d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0880cdc20 0x7fe08809cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0880af580 0x7fe0880cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 --2- 172.21.15.32:0/870459225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0880af1a0 0x7fe0880ceaa0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.519+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 >> 172.21.15.32:0/870459225 conn(0x7fe08806afb0 msgr2=0x7fe08815e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.520+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 shutdown_connections 2024-03-06T07:30:26.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:26.520+0000 7fe08ea00640 1 -- 172.21.15.32:0/870459225 wait complete. 2024-03-06T07:30:26.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:30:27.004 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:30:27.005 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:27.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:27.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.212+0000 7fa7c0400640 1 Processor -- start 2024-03-06T07:30:27.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.212+0000 7fa7c0400640 1 -- start start 2024-03-06T07:30:27.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.215+0000 7fa7c0400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7b815e200 0x7fa7b80b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.215+0000 7fa7c0400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7b806afb0 0x7fa7b806b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.215+0000 7fa7c0400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b806bec0 con 0x7fa7b80afdf0 2024-03-06T07:30:27.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.215+0000 7fa7c0400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b80b0670 con 0x7fa7b806afb0 2024-03-06T07:30:27.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.215+0000 7fa7c0400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b80ae380 con 0x7fa7b815e200 2024-03-06T07:30:27.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.216+0000 7fa7be400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7b806afb0 0x7fa7b806b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.216+0000 7fa7bd000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa7b80afdf0 0x7fa7b80af140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:45710/0 (socket says 172.21.15.32:45710) 2024-03-06T07:30:27.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.216+0000 7fa7bd000640 1 -- 172.21.15.32:0/4054451139 learned_addr learned my addr 172.21.15.32:0/4054451139 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:27.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.217+0000 7fa7be400640 1 -- 172.21.15.32:0/4054451139 >> v2:172.21.15.39:3300/0 conn(0x7fa7b815e200 msgr2=0x7fa7b80b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.217+0000 7fa7be400640 1 --2- 172.21.15.32:0/4054451139 >> v2:172.21.15.39:3300/0 conn(0x7fa7b815e200 0x7fa7b80b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.217+0000 7fa7be400640 1 -- 172.21.15.32:0/4054451139 >> v1:172.21.15.39:6789/0 conn(0x7fa7b80afdf0 legacy=0x7fa7b80af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.217+0000 7fa7be400640 1 -- 172.21.15.32:0/4054451139 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80ae660 con 0x7fa7b806afb0 2024-03-06T07:30:27.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.219+0000 7fa7be400640 1 --2- 172.21.15.32:0/4054451139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b806afb0 0x7fa7b806b460 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8071cc0 tx=0x7fa7a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f9906c070caff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.220+0000 7fa7b7e00640 1 -- 172.21.15.32:0/4054451139 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40315a0 con 0x7fa7b806afb0 2024-03-06T07:30:27.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.220+0000 7fa7b7e00640 1 -- 172.21.15.32:0/4054451139 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4037070 con 0x7fa7b806afb0 2024-03-06T07:30:27.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.220+0000 7fa7b7e00640 1 -- 172.21.15.32:0/4054451139 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4031960 con 0x7fa7b806afb0 2024-03-06T07:30:27.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 -- 172.21.15.32:0/4054451139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b806afb0 msgr2=0x7fa7b806b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 --2- 172.21.15.32:0/4054451139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b806afb0 0x7fa7b806b460 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8071cc0 tx=0x7fa7a402b840 comp rx=0 tx=0).stop 2024-03-06T07:30:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 -- 172.21.15.32:0/4054451139 shutdown_connections 2024-03-06T07:30:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 --2- 172.21.15.32:0/4054451139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b806afb0 0x7fa7b806b460 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 --2- 172.21.15.32:0/4054451139 >> v2:172.21.15.39:3300/0 conn(0x7fa7b815e200 0x7fa7b80b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 -- 172.21.15.32:0/4054451139 >> 172.21.15.32:0/4054451139 conn(0x7fa7b80b2ae0 msgr2=0x7fa7b80b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 -- 172.21.15.32:0/4054451139 shutdown_connections 2024-03-06T07:30:27.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.221+0000 7fa7c0400640 1 -- 172.21.15.32:0/4054451139 wait complete. 2024-03-06T07:30:27.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.224+0000 7fa7c0400640 1 Processor -- start 2024-03-06T07:30:27.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.224+0000 7fa7c0400640 1 -- start start 2024-03-06T07:30:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7c0400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 0x7fa7b80b2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7c0400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b80afdf0 0x7fa7b814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7c0400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7b815e200 0x7fa7b815e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7c0400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b806dbb0 con 0x7fa7b80afdf0 2024-03-06T07:30:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7c0400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80aec20 con 0x7fa7b806afb0 2024-03-06T07:30:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7c0400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b806dd30 con 0x7fa7b815e200 2024-03-06T07:30:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.226+0000 7fa7bd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b80afdf0 0x7fa7b814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7be400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7b815e200 0x7fa7b815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 0x7fa7b80b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7be400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7b815e200 0x7fa7b815e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:55628/0 (socket says 172.21.15.32:55628) 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 0x7fa7b80b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:38126/0 (socket says 172.21.15.32:38126) 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7be400640 1 -- 172.21.15.32:0/1876975054 learned_addr learned my addr 172.21.15.32:0/1876975054 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7bda00640 1 -- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7b815e200 msgr2=0x7fa7b815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7bda00640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7b815e200 0x7fa7b815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.227+0000 7fa7bda00640 1 -- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b80afdf0 msgr2=0x7fa7b814a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.228+0000 7fa7bda00640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b80afdf0 0x7fa7b814a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.228+0000 7fa7bda00640 1 -- 172.21.15.32:0/1876975054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b815ec90 con 0x7fa7b806afb0 2024-03-06T07:30:27.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.228+0000 7fa7bd000640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b80afdf0 0x7fa7b814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:27.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.228+0000 7fa7bda00640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 0x7fa7b80b2500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00bd70 tx=0x7fa7ac00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.229+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac019070 con 0x7fa7b806afb0 2024-03-06T07:30:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.229+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b81611e0 con 0x7fa7b806afb0 2024-03-06T07:30:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.229+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac014440 con 0x7fa7b806afb0 2024-03-06T07:30:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.230+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac014830 con 0x7fa7b806afb0 2024-03-06T07:30:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.230+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b8159690 con 0x7fa7b806afb0 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.234+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7ac01c070 con 0x7fa7b806afb0 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.235+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794005290 con 0x7fa7b806afb0 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.235+0000 7fa7b5600640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa79806b5a0 0x7fa79806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.236+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7ac08a1f0 con 0x7fa7b806afb0 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.237+0000 7fa7bd000640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa79806b5a0 0x7fa79806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.238+0000 7fa7bd000640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa79806b5a0 0x7fa79806da60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0007c90 tx=0x7fa7b0007be0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.250+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7ac058970 con 0x7fa7b806afb0 2024-03-06T07:30:27.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.608+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa794005580 con 0x7fa7b806afb0 2024-03-06T07:30:27.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.610+0000 7fa7b5600640 1 -- 172.21.15.32:0/1876975054 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fa7ac05ca10 con 0x7fa7b806afb0 2024-03-06T07:30:27.611 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:27.612 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa79806b5a0 msgr2=0x7fa79806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa79806b5a0 0x7fa79806da60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0007c90 tx=0x7fa7b0007be0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 msgr2=0x7fa7b80b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 0x7fa7b80b2500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00bd70 tx=0x7fa7ac00be60 comp rx=0 tx=0).stop 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 shutdown_connections 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa79806b5a0 0x7fa79806da60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7b815e200 0x7fa7b815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7b80afdf0 0x7fa7b814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 --2- 172.21.15.32:0/1876975054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b806afb0 0x7fa7b80b2500 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.612+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 >> 172.21.15.32:0/1876975054 conn(0x7fa7b80b2ae0 msgr2=0x7fa7b815aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.613+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 shutdown_connections 2024-03-06T07:30:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:27.613+0000 7fa7c0400640 1 -- 172.21.15.32:0/1876975054 wait complete. 2024-03-06T07:30:27.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:30:27.854 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:30:27.855 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:30:27.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.059+0000 7f54d5a00640 1 Processor -- start 2024-03-06T07:30:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.059+0000 7f54d5a00640 1 -- start start 2024-03-06T07:30:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d01477e0 0x7f54d0162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54d006ef20 0x7f54d006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d0147bc0 con 0x7f54d006b4f0 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d0161f90 con 0x7f54d006ef20 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0162960 con 0x7f54d01477e0 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54c7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54d006ef20 0x7f54d006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f54d006b4f0 0x7f54d00d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42850/0 (socket says 172.21.15.32:42850) 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.061+0000 7f54d5000640 1 -- 172.21.15.32:0/3403907262 learned_addr learned my addr 172.21.15.32:0/3403907262 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54cfe00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d01477e0 0x7f54d0162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54c7e00640 1 -- 172.21.15.32:0/3403907262 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d01477e0 msgr2=0x7f54d0162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54c7e00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d01477e0 0x7f54d0162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54c7e00640 1 -- 172.21.15.32:0/3403907262 >> v1:172.21.15.32:6789/0 conn(0x7f54d006b4f0 legacy=0x7f54d00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54cf400640 1 -- 172.21.15.32:0/3403907262 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973349221 0 0) 0x7f54d0147bc0 con 0x7f54d006b4f0 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54c7e00640 1 -- 172.21.15.32:0/3403907262 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d00b3d70 con 0x7f54d006ef20 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.062+0000 7f54cfe00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d01477e0 0x7f54d0162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.063+0000 7f54c7e00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d006ef20 0x7f54d006afb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f54b80097a0 tx=0x7f54b802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78db4bb727248dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54cf400640 1 -- 172.21.15.32:0/3403907262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8031780 con 0x7f54d006ef20 2024-03-06T07:30:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54cf400640 1 -- 172.21.15.32:0/3403907262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8037070 con 0x7f54d006ef20 2024-03-06T07:30:28.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54cf400640 1 -- 172.21.15.32:0/3403907262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8031b40 con 0x7f54d006ef20 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54d5a00640 1 -- 172.21.15.32:0/3403907262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d006ef20 msgr2=0x7f54d006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54d5a00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d006ef20 0x7f54d006afb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f54b80097a0 tx=0x7f54b802b820 comp rx=0 tx=0).stop 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54d5a00640 1 -- 172.21.15.32:0/3403907262 shutdown_connections 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54d5a00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d006ef20 0x7f54d006afb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54d5a00640 1 --2- 172.21.15.32:0/3403907262 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d01477e0 0x7f54d0162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.064+0000 7f54d5a00640 1 -- 172.21.15.32:0/3403907262 >> 172.21.15.32:0/3403907262 conn(0x7f54d006c0c0 msgr2=0x7f54d006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.065+0000 7f54d5a00640 1 -- 172.21.15.32:0/3403907262 shutdown_connections 2024-03-06T07:30:28.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.065+0000 7f54d5a00640 1 -- 172.21.15.32:0/3403907262 wait complete. 2024-03-06T07:30:28.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.066+0000 7f54d5a00640 1 Processor -- start 2024-03-06T07:30:28.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.066+0000 7f54d5a00640 1 -- start start 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d006b4f0 0x7f54d013ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d006ef20 0x7f54d01f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 0x7f54d0145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0161c20 con 0x7f54d01477e0 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00b4330 con 0x7f54d006ef20 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d006f8f0 con 0x7f54d006b4f0 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 0x7f54d0145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 0x7f54d0145b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39856/0 (socket says 172.21.15.32:39856) 2024-03-06T07:30:28.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.067+0000 7f54d5000640 1 -- 172.21.15.32:0/2488305241 learned_addr learned my addr 172.21.15.32:0/2488305241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54cfe00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d006b4f0 0x7f54d013ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54c7e00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d006ef20 0x7f54d01f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5000640 1 -- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d006b4f0 msgr2=0x7f54d013ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5000640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d006b4f0 0x7f54d013ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5000640 1 -- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d006ef20 msgr2=0x7f54d01f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5000640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d006ef20 0x7f54d01f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5000640 1 -- 172.21.15.32:0/2488305241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d013f770 con 0x7f54d01477e0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5000640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 0x7f54d0145b20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f54c800c7d0 tx=0x7f54c800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8018070 con 0x7f54d01477e0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c800a640 con 0x7f54d01477e0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.068+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d013fa50 con 0x7f54d01477e0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.069+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c800aa70 con 0x7f54d01477e0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.069+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d0140010 con 0x7f54d01477e0 2024-03-06T07:30:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.070+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d013f4c0 con 0x7f54d01477e0 2024-03-06T07:30:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.071+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c801b070 con 0x7f54d01477e0 2024-03-06T07:30:28.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.072+0000 7f54ccc00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.072+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54c8089c30 con 0x7f54d01477e0 2024-03-06T07:30:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.073+0000 7f54cfe00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.074+0000 7f54cfe00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b4d0 0x7f54ac06d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f54c000a950 tx=0x7f54c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.077+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54c8058a00 con 0x7f54d01477e0 2024-03-06T07:30:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.326+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54d00724b0 con 0x7f54d01477e0 2024-03-06T07:30:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.327+0000 7f54ccc00640 1 -- 172.21.15.32:0/2488305241 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f54c8058300 con 0x7f54d01477e0 2024-03-06T07:30:28.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:28.329 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b4d0 msgr2=0x7f54ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b4d0 0x7f54ac06d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f54c000a950 tx=0x7f54c0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 msgr2=0x7f54d0145b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 0x7f54d0145b20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f54c800c7d0 tx=0x7f54c800cc90 comp rx=0 tx=0).stop 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 shutdown_connections 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f54ac06b4d0 0x7f54ac06d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54d01477e0 0x7f54d0145b20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54d006ef20 0x7f54d01f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 --2- 172.21.15.32:0/2488305241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f54d006b4f0 0x7f54d013ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 >> 172.21.15.32:0/2488305241 conn(0x7f54d006c0c0 msgr2=0x7f54d00b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 shutdown_connections 2024-03-06T07:30:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.330+0000 7f54d5a00640 1 -- 172.21.15.32:0/2488305241 wait complete. 2024-03-06T07:30:28.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:30:28.573 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:30:28.573 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:30:28.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.784+0000 7f8907200640 1 Processor -- start 2024-03-06T07:30:28.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.784+0000 7f8907200640 1 -- start start 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.786+0000 7f8907200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f89080cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.786+0000 7f8907200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89080cce90 0x7f890806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.786+0000 7f8907200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080cdad0 con 0x7f890815c920 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.786+0000 7f8907200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8908071500 con 0x7f89080cce90 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.786+0000 7f8907200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89080cdd70 con 0x7f89080ce850 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8906800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89080cce90 0x7f890806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8905e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f890815c920 0x7f8908070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42864/0 (socket says 172.21.15.32:42864) 2024-03-06T07:30:28.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8905e00640 1 -- 172.21.15.32:0/3760254450 learned_addr learned my addr 172.21.15.32:0/3760254450 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:28.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8906800640 1 -- 172.21.15.32:0/3760254450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 msgr2=0x7f89080cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8906800640 1 --2- 172.21.15.32:0/3760254450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f89080cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8906800640 1 -- 172.21.15.32:0/3760254450 >> v1:172.21.15.32:6789/0 conn(0x7f890815c920 legacy=0x7f8908070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.787+0000 7f8906800640 1 -- 172.21.15.32:0/3760254450 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89080ce050 con 0x7f89080cce90 2024-03-06T07:30:28.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.788+0000 7f8906800640 1 --2- 172.21.15.32:0/3760254450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f890806cac0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f89000097a0 tx=0x7f890002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e333910f1049fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8904a00640 1 -- 172.21.15.32:0/3760254450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900035020 con 0x7f89080cce90 2024-03-06T07:30:28.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8904a00640 1 -- 172.21.15.32:0/3760254450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8900031810 con 0x7f89080cce90 2024-03-06T07:30:28.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8904a00640 1 -- 172.21.15.32:0/3760254450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900031be0 con 0x7f89080cce90 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 -- 172.21.15.32:0/3760254450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 msgr2=0x7f890806cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 --2- 172.21.15.32:0/3760254450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f890806cac0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f89000097a0 tx=0x7f890002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 -- 172.21.15.32:0/3760254450 shutdown_connections 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 --2- 172.21.15.32:0/3760254450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f890806cac0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 --2- 172.21.15.32:0/3760254450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f89080cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 -- 172.21.15.32:0/3760254450 >> 172.21.15.32:0/3760254450 conn(0x7f890806b3d0 msgr2=0x7f890806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 -- 172.21.15.32:0/3760254450 shutdown_connections 2024-03-06T07:30:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.789+0000 7f8907200640 1 -- 172.21.15.32:0/3760254450 wait complete. 2024-03-06T07:30:28.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.792+0000 7f8907200640 1 Processor -- start 2024-03-06T07:30:28.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.792+0000 7f8907200640 1 -- start start 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8907200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f8908159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8907200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f890815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8907200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f890815c920 0x7f890815d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8907200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908071150 con 0x7f89080cce90 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8907200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908163790 con 0x7f890815c920 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8907200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8908159fe0 con 0x7f89080ce850 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8905e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f8908159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8905e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f8908159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39880/0 (socket says 172.21.15.32:39880) 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.794+0000 7f8905e00640 1 -- 172.21.15.32:0/2063176649 learned_addr learned my addr 172.21.15.32:0/2063176649 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8906800640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f890815c920 0x7f890815d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905400640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f890815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905e00640 1 -- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 msgr2=0x7f890815d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905e00640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f890815d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905e00640 1 -- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f890815c920 msgr2=0x7f890815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905e00640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f890815c920 0x7f890815d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905e00640 1 -- 172.21.15.32:0/2063176649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89081fc050 con 0x7f89080cce90 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8906800640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f890815c920 0x7f890815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905400640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f890815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8905e00640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f8908159aa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c3c0 tx=0x7f88f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8017070 con 0x7f89080cce90 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f8005ce0 con 0x7f89080cce90 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80072f0 con 0x7f89080cce90 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.795+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89081fc330 con 0x7f89080cce90 2024-03-06T07:30:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.796+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89081fc8f0 con 0x7f89080cce90 2024-03-06T07:30:28.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.797+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89080ccc80 con 0x7f89080cce90 2024-03-06T07:30:28.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.798+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88f801a070 con 0x7f89080cce90 2024-03-06T07:30:28.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.798+0000 7f88fe000640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f88e406b4d0 0x7f88e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.798+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88f8089640 con 0x7f89080cce90 2024-03-06T07:30:28.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.799+0000 7f8905400640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f88e406b4d0 0x7f88e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.799+0000 7f8905400640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f88e406b4d0 0x7f88e406d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f890815b7f0 tx=0x7f88f00073c0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:28.803+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89080ccc80 con 0x7f89080cce90 2024-03-06T07:30:29.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.044+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f890804f330 con 0x7f89080cce90 2024-03-06T07:30:29.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.044+0000 7f88fe000640 1 -- 172.21.15.32:0/2063176649 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f88f8058420 con 0x7f89080cce90 2024-03-06T07:30:29.046 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:29.046 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:29.046 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:29.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f88e406b4d0 msgr2=0x7f88e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f88e406b4d0 0x7f88e406d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f890815b7f0 tx=0x7f88f00073c0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 msgr2=0x7f8908159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f8908159aa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c3c0 tx=0x7f88f800c880 comp rx=0 tx=0).stop 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 shutdown_connections 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f88e406b4d0 0x7f88e406d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f890815c920 0x7f890815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89080ce850 0x7f890815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 --2- 172.21.15.32:0/2063176649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89080cce90 0x7f8908159aa0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 >> 172.21.15.32:0/2063176649 conn(0x7f890806b3d0 msgr2=0x7f890815aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 shutdown_connections 2024-03-06T07:30:29.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.047+0000 7f8907200640 1 -- 172.21.15.32:0/2063176649 wait complete. 2024-03-06T07:30:29.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:30:29.304 INFO:teuthology.orchestra.run.smithi032.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:30:29.304 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:30:29.304 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:30:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.410+0000 7fc68cb7bf40 1 Processor -- start 2024-03-06T07:30:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.410+0000 7fc68cb7bf40 1 -- start start 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc68cb7bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc68cb7bf40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc68cb7bf40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 0x5588d24d4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc68cb7bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d230cfd0 con 0x5588d24cfb10 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc68cb7bf40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d2310f30 con 0x5588d23b0140 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc68cb7bf40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588d2365a50 con 0x5588d24cfef0 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39904/0 (socket says 172.21.15.32:39904) 2024-03-06T07:30:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 -- 172.21.15.32:0/2909671000 learned_addr learned my addr 172.21.15.32:0/2909671000 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689400640 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 -- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 msgr2=0x5588d24d4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 0x5588d24d4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 -- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 msgr2=0x5588d23aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689e00640 1 -- 172.21.15.32:0/2909671000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588d2421ec0 con 0x5588d24cfb10 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.411+0000 7fc689400640 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.412+0000 7fc689e00640 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241e920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c3f0 tx=0x7fc67c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4fd6619fcb3db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.412+0000 7fc683e00640 1 -- 172.21.15.32:0/2909671000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c00e540 con 0x5588d24cfb10 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.412+0000 7fc683e00640 1 -- 172.21.15.32:0/2909671000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67c00e6e0 con 0x5588d24cfb10 2024-03-06T07:30:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.412+0000 7fc683e00640 1 -- 172.21.15.32:0/2909671000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c00e8d0 con 0x5588d24cfb10 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2909671000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 msgr2=0x5588d241e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241e920 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c3f0 tx=0x7fc67c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2909671000 shutdown_connections 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 0x5588d24d4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2909671000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241e920 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2909671000 >> 172.21.15.32:0/2909671000 conn(0x5588d2395c10 msgr2=0x5588d24269d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2909671000 shutdown_connections 2024-03-06T07:30:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2909671000 wait complete. 2024-03-06T07:30:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.413+0000 7fc68cb7bf40 1 Processor -- start 2024-03-06T07:30:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.414+0000 7fc68cb7bf40 1 -- start start 2024-03-06T07:30:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.414+0000 7fc68cb7bf40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc68cb7bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc68cb7bf40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 0x5588d243ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc68cb7bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d2423040 con 0x5588d24cfb10 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc68cb7bf40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d2398a90 con 0x5588d23b0140 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc68cb7bf40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588d2422d60 con 0x5588d24cfef0 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39920/0 (socket says 172.21.15.32:39920) 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 -- 172.21.15.32:0/2988657185 learned_addr learned my addr 172.21.15.32:0/2988657185 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689e00640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc68a800640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 0x5588d243ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 -- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 msgr2=0x5588d243ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5588d24cfef0 0x5588d243ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 -- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 msgr2=0x5588d23adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23adab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689400640 1 -- 172.21.15.32:0/2988657185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588d2426250 con 0x5588d24cfb10 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.415+0000 7fc689e00640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5588d23b0140 0x5588d23adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.416+0000 7fc689400640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241df40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc67400e490 tx=0x7fc67400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.416+0000 7fc682a00640 1 -- 172.21.15.32:0/2988657185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674017020 con 0x5588d24cfb10 2024-03-06T07:30:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.416+0000 7fc682a00640 1 -- 172.21.15.32:0/2988657185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc674013870 con 0x5588d24cfb10 2024-03-06T07:30:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.416+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5588d2426530 con 0x5588d24cfb10 2024-03-06T07:30:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.416+0000 7fc682a00640 1 -- 172.21.15.32:0/2988657185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674013c40 con 0x5588d24cfb10 2024-03-06T07:30:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.416+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588d241ff30 con 0x5588d24cfb10 2024-03-06T07:30:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.418+0000 7fc682a00640 1 -- 172.21.15.32:0/2988657185 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc67401a020 con 0x5588d24cfb10 2024-03-06T07:30:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.420+0000 7fc682a00640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc668069eb0 0x7fc66806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.420+0000 7fc682a00640 1 -- 172.21.15.32:0/2988657185 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc674089770 con 0x5588d24cfb10 2024-03-06T07:30:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.420+0000 7fc689e00640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc668069eb0 0x7fc66806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.420+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x5588d241e480 0x5588d23ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.420+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v9 -- 0x5588d243cf80 con 0x5588d241e480 2024-03-06T07:30:29.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.421+0000 7fc68a800640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x5588d241e480 0x5588d23ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.421+0000 7fc689e00640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc668069eb0 0x7fc66806c370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc67c000fd0 tx=0x7fc67c002000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.421+0000 7fc68a800640 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x5588d241e480 0x5588d23ab4a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68a800640 1 -- 172.21.15.32:0/2988657185 <== osd.3 v2:172.21.15.32:6806/238384581 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7fc684001470 con 0x5588d241e480 2024-03-06T07:30:29.423 INFO:teuthology.orchestra.run.smithi032.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x5588d241e480 msgr2=0x5588d23ab4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x5588d241e480 0x5588d23ab4a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc668069eb0 msgr2=0x7fc66806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc668069eb0 0x7fc66806c370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc67c000fd0 tx=0x7fc67c002000 comp rx=0 tx=0).stop 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 msgr2=0x5588d241df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 --2- 172.21.15.32:0/2988657185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588d24cfb10 0x5588d241df40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc67400e490 tx=0x7fc67400e950 comp rx=0 tx=0).stop 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68a800640 1 -- 172.21.15.32:0/2988657185 reap_dead start 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 shutdown_connections 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 >> 172.21.15.32:0/2988657185 conn(0x5588d2395c10 msgr2=0x5588d23aaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 shutdown_connections 2024-03-06T07:30:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.422+0000 7fc68cb7bf40 1 -- 172.21.15.32:0/2988657185 wait complete. 2024-03-06T07:30:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:30:29.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:30:29.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.642+0000 7f304fe00640 1 Processor -- start 2024-03-06T07:30:29.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.643+0000 7f304fe00640 1 -- start start 2024-03-06T07:30:29.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 0x7f30500728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f30500b4120 0x7f30500b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500d1a10 con 0x7f305006f690 2024-03-06T07:30:29.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3050072f70 con 0x7f30500b4120 2024-03-06T07:30:29.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30500d1450 con 0x7f305006b6c0 2024-03-06T07:30:29.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304ea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f305006f690 0x7f305006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42886/0 (socket says 172.21.15.32:42886) 2024-03-06T07:30:29.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.644+0000 7f304ea00640 1 -- 172.21.15.32:0/3529154323 learned_addr learned my addr 172.21.15.32:0/3529154323 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304f400640 1 --2- 172.21.15.32:0/3529154323 >> v2:172.21.15.39:3300/0 conn(0x7f30500b4120 0x7f30500b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f3047e00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 0x7f30500728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093535634 0 0) 0x7f30500d1a10 con 0x7f305006f690 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034003520 con 0x7f305006f690 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20916176 0 0) 0x7f3034003520 con 0x7f305006f690 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 msgr2=0x7f30500728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 0x7f30500728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 >> v2:172.21.15.39:3300/0 conn(0x7f30500b4120 msgr2=0x7f30500b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 --2- 172.21.15.32:0/3529154323 >> v2:172.21.15.39:3300/0 conn(0x7f30500b4120 0x7f30500b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500d1730 con 0x7f305006f690 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3040001670 con 0x7f305006f690 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.645+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 >> v1:172.21.15.32:6789/0 conn(0x7f305006f690 legacy=0x7f305006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304e000640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30340039b0 0x7f3034003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304e000640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034004800 0x7f3034004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304e000640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 0x7f30340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3050072f70 con 0x7f30340039b0 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500d1450 con 0x7f3034004800 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303400b9c0 con 0x7f305006b6c0 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304ea00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30340039b0 0x7f3034003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f3047e00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034004800 0x7f3034004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304ea00640 1 -- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 msgr2=0x7f30340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304ea00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f305006b6c0 0x7f30340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304ea00640 1 -- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034004800 msgr2=0x7f3034004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304ea00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034004800 0x7f3034004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.646+0000 7f304ea00640 1 -- 172.21.15.32:0/3529154323 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3034003520 con 0x7f30340039b0 2024-03-06T07:30:29.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304ea00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30340039b0 0x7f3034003dd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f304000c210 tx=0x7f3034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b582065000e707d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040007d70 con 0x7f30340039b0 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040030070 con 0x7f30340039b0 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304e000640 1 -- 172.21.15.32:0/3529154323 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304002cd60 con 0x7f30340039b0 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304fe00640 1 -- 172.21.15.32:0/3529154323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30340039b0 msgr2=0x7f3034003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304fe00640 1 --2- 172.21.15.32:0/3529154323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30340039b0 0x7f3034003dd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f304000c210 tx=0x7f3034005830 comp rx=0 tx=0).stop 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.648+0000 7f304f400640 1 -- 172.21.15.32:0/3529154323 reap_dead start 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.649+0000 7f304fe00640 1 -- 172.21.15.32:0/3529154323 shutdown_connections 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.649+0000 7f304fe00640 1 -- 172.21.15.32:0/3529154323 >> 172.21.15.32:0/3529154323 conn(0x7f305006c0c0 msgr2=0x7f305006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.649+0000 7f304fe00640 1 -- 172.21.15.32:0/3529154323 shutdown_connections 2024-03-06T07:30:29.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.649+0000 7f304fe00640 1 -- 172.21.15.32:0/3529154323 wait complete. 2024-03-06T07:30:29.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.651+0000 7f304fe00640 1 Processor -- start 2024-03-06T07:30:29.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.652+0000 7f304fe00640 1 -- start start 2024-03-06T07:30:29.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.653+0000 7f304fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30340039b0 0x7f305013e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.653+0000 7f304fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 0x7f3050161910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.653+0000 7f304fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3050161e50 0x7f30500cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.653+0000 7f304fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500728b0 con 0x7f3050141980 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.653+0000 7f304fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305006fa70 con 0x7f30340039b0 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.653+0000 7f304fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3050072a30 con 0x7f3050161e50 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 0x7f3050161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 0x7f3050161910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39952/0 (socket says 172.21.15.32:39952) 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 -- 172.21.15.32:0/2079416929 learned_addr learned my addr 172.21.15.32:0/2079416929 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f304ea00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30340039b0 0x7f305013e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f304f400640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3050161e50 0x7f30500cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 -- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3050161e50 msgr2=0x7f30500cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3050161e50 0x7f30500cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 -- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30340039b0 msgr2=0x7f305013e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30340039b0 0x7f305013e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f3047e00640 1 -- 172.21.15.32:0/2079416929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3050140350 con 0x7f3050141980 2024-03-06T07:30:29.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.654+0000 7f304f400640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3050161e50 0x7f30500cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.655+0000 7f3047e00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 0x7f3050161910 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3038010860 tx=0x7f3038010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.655+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303800f040 con 0x7f3050141980 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.655+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303800eb00 con 0x7f3050141980 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.655+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303801c400 con 0x7f3050141980 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.655+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3050140630 con 0x7f3050141980 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.655+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3050140bf0 con 0x7f3050141980 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.656+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305004f330 con 0x7f3050141980 2024-03-06T07:30:29.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.657+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f303801c5a0 con 0x7f3050141980 2024-03-06T07:30:29.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.658+0000 7f3046a00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f302c06b500 0x7f302c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.658+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f303808a720 con 0x7f3050141980 2024-03-06T07:30:29.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.658+0000 7f304ea00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f302c06b500 0x7f302c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.659+0000 7f304ea00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f302c06b500 0x7f302c06d9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3040034000 tx=0x7f3040007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.663+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3038018030 con 0x7f3050141980 2024-03-06T07:30:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.991+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f30500b5180 con 0x7f3050141980 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.991+0000 7f3046a00640 1 -- 172.21.15.32:0/2079416929 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1660 (secure 0 0 0) 0x7f3038059630 con 0x7f3050141980 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout: "total_bytes": 773094113280, 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout: "total_avail_bytes": 772825985024, 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_bytes": 268128256, 2024-03-06T07:30:29.993 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_bytes": 268128256, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_ratio": 0.00034682484692893922, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "num_osds": 8, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "num_per_pool_osds": 8, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "stats_by_class": { 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "ssd": { 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "total_bytes": 773094113280, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "total_avail_bytes": 772825985024, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_bytes": 268128256, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_bytes": 268128256, 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_ratio": 0.00034682484692893922 2024-03-06T07:30:29.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "name": ".mgr", 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 1, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 590368, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 2, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 1160, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 1187840, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 1.617983457435912e-06, 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367073624064 2024-03-06T07:30:29.995 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephfs_metadata", 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 4196898, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 24, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 8256, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 8454144, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 1.1515464393596631e-05, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367073624064 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:29.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephfs_data", 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5, 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 0, 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 1000, 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 0, 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 0, 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 0, 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367073624064 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:30:29.997 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:30:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f302c06b500 msgr2=0x7f302c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f302c06b500 0x7f302c06d9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3040034000 tx=0x7f3040007000 comp rx=0 tx=0).stop 2024-03-06T07:30:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 msgr2=0x7f3050161910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 0x7f3050161910 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3038010860 tx=0x7f3038010d20 comp rx=0 tx=0).stop 2024-03-06T07:30:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 shutdown_connections 2024-03-06T07:30:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f302c06b500 0x7f302c06d9c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3050161e50 0x7f30500cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3050141980 0x7f3050161910 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 --2- 172.21.15.32:0/2079416929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30340039b0 0x7f305013e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 >> 172.21.15.32:0/2079416929 conn(0x7f305006c0c0 msgr2=0x7f30500ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 shutdown_connections 2024-03-06T07:30:29.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:29.995+0000 7f304fe00640 1 -- 172.21.15.32:0/2079416929 wait complete. 2024-03-06T07:30:30.015 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:30:35.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:30:35.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.268+0000 7f9453000640 1 Processor -- start 2024-03-06T07:30:35.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.268+0000 7f9453000640 1 -- start start 2024-03-06T07:30:35.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.269+0000 7f9453000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f944c15dfb0 0x7f944c0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.269+0000 7f9453000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f944c15e390 0x7f944c0cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.269+0000 7f9453000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944c0ce840 con 0x7f944c0cfae0 2024-03-06T07:30:35.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.269+0000 7f9453000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c0cf110 con 0x7f944c15e390 2024-03-06T07:30:35.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.269+0000 7f9453000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c0d1480 con 0x7f944c15dfb0 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.270+0000 7f9452600640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f944c0cfae0 0x7f944c0d1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42894/0 (socket says 172.21.15.32:42894) 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.270+0000 7f9451200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f944c15e390 0x7f944c0cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.270+0000 7f9452600640 1 -- 172.21.15.32:0/4078353670 learned_addr learned my addr 172.21.15.32:0/4078353670 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.270+0000 7f9451c00640 1 --2- 172.21.15.32:0/4078353670 >> v2:172.21.15.39:3300/0 conn(0x7f944c15dfb0 0x7f944c0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.271+0000 7f9451200640 1 -- 172.21.15.32:0/4078353670 >> v2:172.21.15.39:3300/0 conn(0x7f944c15dfb0 msgr2=0x7f944c0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.271+0000 7f9451200640 1 --2- 172.21.15.32:0/4078353670 >> v2:172.21.15.39:3300/0 conn(0x7f944c15dfb0 0x7f944c0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.271+0000 7f9451200640 1 -- 172.21.15.32:0/4078353670 >> v1:172.21.15.32:6789/0 conn(0x7f944c0cfae0 legacy=0x7f944c0d1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.271+0000 7f9451200640 1 -- 172.21.15.32:0/4078353670 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c0ceac0 con 0x7f944c15e390 2024-03-06T07:30:35.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.271+0000 7f944be00640 1 -- 172.21.15.32:0/4078353670 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547985602 0 0) 0x7f9444001110 con 0x7f944c0cfae0 2024-03-06T07:30:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.272+0000 7f9451200640 1 --2- 172.21.15.32:0/4078353670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c15e390 0x7f944c0cf5a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f944c08c4f0 tx=0x7f943802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49746c944f3870b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.272+0000 7f944be00640 1 -- 172.21.15.32:0/4078353670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94380315a0 con 0x7f944c15e390 2024-03-06T07:30:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.272+0000 7f944be00640 1 -- 172.21.15.32:0/4078353670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9438037070 con 0x7f944c15e390 2024-03-06T07:30:35.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.272+0000 7f944be00640 1 -- 172.21.15.32:0/4078353670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438031960 con 0x7f944c15e390 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 -- 172.21.15.32:0/4078353670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c15e390 msgr2=0x7f944c0cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 --2- 172.21.15.32:0/4078353670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c15e390 0x7f944c0cf5a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f944c08c4f0 tx=0x7f943802b840 comp rx=0 tx=0).stop 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 -- 172.21.15.32:0/4078353670 shutdown_connections 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 --2- 172.21.15.32:0/4078353670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c15e390 0x7f944c0cf5a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 --2- 172.21.15.32:0/4078353670 >> v2:172.21.15.39:3300/0 conn(0x7f944c15dfb0 0x7f944c0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 -- 172.21.15.32:0/4078353670 >> 172.21.15.32:0/4078353670 conn(0x7f944c0d3d30 msgr2=0x7f944c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 -- 172.21.15.32:0/4078353670 shutdown_connections 2024-03-06T07:30:35.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.273+0000 7f9453000640 1 -- 172.21.15.32:0/4078353670 wait complete. 2024-03-06T07:30:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.275+0000 7f9453000640 1 Processor -- start 2024-03-06T07:30:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.275+0000 7f9453000640 1 -- start start 2024-03-06T07:30:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9453000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 0x7f944c14ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9453000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f944c15dfb0 0x7f944c1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9453000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f944c15f3a0 0x7f944c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9453000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0d1220 con 0x7f944c0cfae0 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9453000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0d10a0 con 0x7f944c15f3a0 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9453000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0d49c0 con 0x7f944c15dfb0 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9451c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 0x7f944c14ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9451c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 0x7f944c14ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39964/0 (socket says 172.21.15.32:39964) 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.277+0000 7f9451c00640 1 -- 172.21.15.32:0/3867196384 learned_addr learned my addr 172.21.15.32:0/3867196384 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9452600640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f944c15f3a0 0x7f944c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451c00640 1 -- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f944c15dfb0 msgr2=0x7f944c1574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451200640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f944c15dfb0 0x7f944c1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451c00640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f944c15dfb0 0x7f944c1574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451c00640 1 -- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f944c15f3a0 msgr2=0x7f944c1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451c00640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f944c15f3a0 0x7f944c1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451c00640 1 -- 172.21.15.32:0/3867196384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c15f8f0 con 0x7f944c0cfae0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9452600640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f944c15f3a0 0x7f944c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451200640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f944c15dfb0 0x7f944c1574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9451c00640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 0x7f944c14ba70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9440007ef0 tx=0x7f944000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440014620 con 0x7f944c0cfae0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9440019070 con 0x7f944c0cfae0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440014a00 con 0x7f944c0cfae0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.278+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c15fbd0 con 0x7f944c0cfae0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.279+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c160190 con 0x7f944c0cfae0 2024-03-06T07:30:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.280+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c04f330 con 0x7f944c0cfae0 2024-03-06T07:30:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.281+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9440014ba0 con 0x7f944c0cfae0 2024-03-06T07:30:35.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.282+0000 7f9449600640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f942c06b4d0 0x7f942c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.282+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f944008ab50 con 0x7f944c0cfae0 2024-03-06T07:30:35.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.282+0000 7f9451200640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f942c06b4d0 0x7f942c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.283+0000 7f9451200640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f942c06b4d0 0x7f942c06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9438036000 tx=0x7f9438008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.287+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94400599c0 con 0x7f944c0cfae0 2024-03-06T07:30:35.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.567+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f944c08db00 con 0x7f944c0cfae0 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.568+0000 7f9449600640 1 -- 172.21.15.32:0/3867196384 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1652 (secure 0 0 0) 0x7f94400592c0 con 0x7f944c0cfae0 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stdout: "total_bytes": 773094113280, 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stdout: "total_avail_bytes": 772831608832, 2024-03-06T07:30:35.569 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_bytes": 262504448, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_bytes": 262504448, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_ratio": 0.00033955043181777, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "num_osds": 8, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "num_per_pool_osds": 8, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "stats_by_class": { 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "ssd": { 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "total_bytes": 773094113280, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "total_avail_bytes": 772831608832, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_bytes": 262504448, 2024-03-06T07:30:35.570 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_bytes": 262504448, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_ratio": 0.00033955043181777 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "name": ".mgr", 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 1, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 590368, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 2, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 1160, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 1187840, 2024-03-06T07:30:35.571 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 1.6179424164874945e-06, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367082930176 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephfs_metadata", 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 1389134, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 23, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 2776, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 2842624, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 3.8718949326721486e-06, 2024-03-06T07:30:35.572 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367082930176 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephfs_data", 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 0, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 0, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 0, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 0, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 0, 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367082930176 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:35.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:35.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:30:35.574 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f942c06b4d0 msgr2=0x7f942c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f942c06b4d0 0x7f942c06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9438036000 tx=0x7f9438008030 comp rx=0 tx=0).stop 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 msgr2=0x7f944c14ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 0x7f944c14ba70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9440007ef0 tx=0x7f944000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 shutdown_connections 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f942c06b4d0 0x7f942c06d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f944c15f3a0 0x7f944c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f944c15dfb0 0x7f944c1574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 --2- 172.21.15.32:0/3867196384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f944c0cfae0 0x7f944c14ba70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.570+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 >> 172.21.15.32:0/3867196384 conn(0x7f944c0d3d30 msgr2=0x7f944c06c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.571+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 shutdown_connections 2024-03-06T07:30:35.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:35.571+0000 7f9453000640 1 -- 172.21.15.32:0/3867196384 wait complete. 2024-03-06T07:30:35.590 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:30:35.590 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:35.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:35.703 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:35.831 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:35.831 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:30:35.831 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:35.887 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:30:35.887 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:35.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.195+0000 7f1283200640 1 Processor -- start 2024-03-06T07:30:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.195+0000 7f1283200640 1 -- start start 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1283200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1284144c00 0x7f1284145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1283200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12841457c0 0x7f128406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1283200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f128406b4f0 0x7f128415f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1283200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12840d0910 con 0x7f128406b4f0 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1283200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12840d0bf0 con 0x7f1284144c00 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1283200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12840d0ed0 con 0x7f12841457c0 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1282800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f128406b4f0 0x7f128415f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1282800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f128406b4f0 0x7f128415f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39976/0 (socket says 172.21.15.32:39976) 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.197+0000 7f1282800640 1 -- 172.21.15.32:0/3453583620 learned_addr learned my addr 172.21.15.32:0/3453583620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1281400640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12841457c0 0x7f128406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1281e00640 1 --2- 172.21.15.32:0/3453583620 >> v2:172.21.15.39:3300/0 conn(0x7f1284144c00 0x7f1284145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1282800640 1 -- 172.21.15.32:0/3453583620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12841457c0 msgr2=0x7f128406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1282800640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12841457c0 0x7f128406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1282800640 1 -- 172.21.15.32:0/3453583620 >> v2:172.21.15.39:3300/0 conn(0x7f1284144c00 msgr2=0x7f1284145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1282800640 1 --2- 172.21.15.32:0/3453583620 >> v2:172.21.15.39:3300/0 conn(0x7f1284144c00 0x7f1284145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1282800640 1 -- 172.21.15.32:0/3453583620 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840d11b0 con 0x7f128406b4f0 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1281400640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12841457c0 0x7f128406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.198+0000 7f1281e00640 1 --2- 172.21.15.32:0/3453583620 >> v2:172.21.15.39:3300/0 conn(0x7f1284144c00 0x7f1284145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.199+0000 7f1282800640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128415f990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f128406bcf0 tx=0x7f127c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb65e464af573625 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.199+0000 7f1280a00640 1 -- 172.21.15.32:0/3453583620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c02ca80 con 0x7f128406b4f0 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.199+0000 7f1280a00640 1 -- 172.21.15.32:0/3453583620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127c02cc20 con 0x7f128406b4f0 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.199+0000 7f1280a00640 1 -- 172.21.15.32:0/3453583620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c0343f0 con 0x7f128406b4f0 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 -- 172.21.15.32:0/3453583620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 msgr2=0x7f128415f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128415f990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f128406bcf0 tx=0x7f127c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 -- 172.21.15.32:0/3453583620 shutdown_connections 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128415f990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 --2- 172.21.15.32:0/3453583620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12841457c0 0x7f128406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 --2- 172.21.15.32:0/3453583620 >> v2:172.21.15.39:3300/0 conn(0x7f1284144c00 0x7f1284145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 -- 172.21.15.32:0/3453583620 >> 172.21.15.32:0/3453583620 conn(0x7f128406c0c0 msgr2=0x7f128406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 -- 172.21.15.32:0/3453583620 shutdown_connections 2024-03-06T07:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.200+0000 7f1283200640 1 -- 172.21.15.32:0/3453583620 wait complete. 2024-03-06T07:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.202+0000 7f1283200640 1 Processor -- start 2024-03-06T07:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.202+0000 7f1283200640 1 -- start start 2024-03-06T07:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1283200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128413b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1283200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1284144c00 0x7f128415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1283200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12841457c0 0x7f12840b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1283200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12840d19b0 con 0x7f128406b4f0 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1283200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1284144fe0 con 0x7f12841457c0 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1283200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128406f820 con 0x7f1284144c00 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1281e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128413b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1281e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128413b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39986/0 (socket says 172.21.15.32:39986) 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1281e00640 1 -- 172.21.15.32:0/3925394046 learned_addr learned my addr 172.21.15.32:0/3925394046 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1281400640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1284144c00 0x7f128415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.203+0000 7f1281e00640 1 -- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1284144c00 msgr2=0x7f128415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1282800640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12841457c0 0x7f12840b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1281e00640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1284144c00 0x7f128415b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1281e00640 1 -- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12841457c0 msgr2=0x7f12840b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1281e00640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12841457c0 0x7f12840b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1281e00640 1 -- 172.21.15.32:0/3925394046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840ae7b0 con 0x7f128406b4f0 2024-03-06T07:30:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1281e00640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128413b640 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f127400c7c0 tx=0x7f127400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274015070 con 0x7f128406b4f0 2024-03-06T07:30:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1274007a30 con 0x7f128406b4f0 2024-03-06T07:30:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12840aea90 con 0x7f128406b4f0 2024-03-06T07:30:36.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.204+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274007e00 con 0x7f128406b4f0 2024-03-06T07:30:36.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.205+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12840af050 con 0x7f128406b4f0 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.206+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128404f330 con 0x7f128406b4f0 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.207+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1274018070 con 0x7f128406b4f0 2024-03-06T07:30:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.207+0000 7f127a000640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f126406b520 0x7f126406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.208+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f127408a180 con 0x7f128406b4f0 2024-03-06T07:30:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.208+0000 7f1281400640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f126406b520 0x7f126406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.209+0000 7f1281400640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f126406b520 0x7f126406d9e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f12840b5810 tx=0x7f1270008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.212+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1274058fe0 con 0x7f128406b4f0 2024-03-06T07:30:36.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.417+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f128415f220 con 0x7f128406b4f0 2024-03-06T07:30:36.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.421+0000 7f127a000640 1 -- 172.21.15.32:0/3925394046 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16561 (secure 0 0 0) 0x7f12740588e0 con 0x7f128406b4f0 2024-03-06T07:30:36.423 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:36.423 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:13.040051+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.423+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f126406b520 msgr2=0x7f126406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.423+0000 7f1283200640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f126406b520 0x7f126406d9e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f12840b5810 tx=0x7f1270008040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.423+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 msgr2=0x7f128413b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.423+0000 7f1283200640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128413b640 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f127400c7c0 tx=0x7f127400cc80 comp rx=0 tx=0).stop 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 shutdown_connections 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f126406b520 0x7f126406d9e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12841457c0 0x7f12840b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1284144c00 0x7f128415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 --2- 172.21.15.32:0/3925394046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f128406b4f0 0x7f128413b640 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 >> 172.21.15.32:0/3925394046 conn(0x7f128406c0c0 msgr2=0x7f12841422f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 shutdown_connections 2024-03-06T07:30:36.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.424+0000 7f1283200640 1 -- 172.21.15.32:0/3925394046 wait complete. 2024-03-06T07:30:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:09.502397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:10.547731+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:36.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:36.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.656+0000 7fe6d7c00640 1 Processor -- start 2024-03-06T07:30:36.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.656+0000 7fe6d7c00640 1 -- start start 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.657+0000 7fe6d7c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe6d015e280 0x7fe6d00d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.657+0000 7fe6d7c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d2990 0x7fe6d00cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d7c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d00d0080 con 0x7fe6d00d4a00 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d7c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d00cf500 con 0x7fe6d015e280 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d7c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00cf7e0 con 0x7fe6d00d2990 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe6d015e280 0x7fe6d00d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe6d015e280 0x7fe6d00d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39992/0 (socket says 172.21.15.32:39992) 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 -- 172.21.15.32:0/2496046853 learned_addr learned my addr 172.21.15.32:0/2496046853 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5c00640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d2990 0x7fe6d00cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 -- 172.21.15.32:0/2496046853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d2990 msgr2=0x7fe6d00cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d2990 0x7fe6d00cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 -- 172.21.15.32:0/2496046853 >> v1:172.21.15.39:6789/0 conn(0x7fe6d00d4a00 legacy=0x7fe6d00d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5200640 1 -- 172.21.15.32:0/2496046853 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d00cfac0 con 0x7fe6d015e280 2024-03-06T07:30:36.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.658+0000 7fe6d5c00640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d2990 0x7fe6d00cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:36.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.660+0000 7fe6d5200640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d00d1cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe6d008dc50 tx=0x7fe6c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b83da138cc82ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.660+0000 7fe6cf400640 1 -- 172.21.15.32:0/2496046853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40317c0 con 0x7fe6d015e280 2024-03-06T07:30:36.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.660+0000 7fe6cf400640 1 -- 172.21.15.32:0/2496046853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c4037070 con 0x7fe6d015e280 2024-03-06T07:30:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.660+0000 7fe6cf400640 1 -- 172.21.15.32:0/2496046853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4031b80 con 0x7fe6d015e280 2024-03-06T07:30:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 -- 172.21.15.32:0/2496046853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 msgr2=0x7fe6d00d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d00d1cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe6d008dc50 tx=0x7fe6c402b860 comp rx=0 tx=0).stop 2024-03-06T07:30:36.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 -- 172.21.15.32:0/2496046853 shutdown_connections 2024-03-06T07:30:36.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d2990 0x7fe6d00cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/2496046853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d00d1cc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 -- 172.21.15.32:0/2496046853 >> 172.21.15.32:0/2496046853 conn(0x7fe6d00d3ee0 msgr2=0x7fe6d00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 -- 172.21.15.32:0/2496046853 shutdown_connections 2024-03-06T07:30:36.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.661+0000 7fe6d7c00640 1 -- 172.21.15.32:0/2496046853 wait complete. 2024-03-06T07:30:36.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.664+0000 7fe6d7c00640 1 Processor -- start 2024-03-06T07:30:36.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.664+0000 7fe6d7c00640 1 -- start start 2024-03-06T07:30:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d7c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6d00d2990 0x7fe6d01514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d7c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d4a00 0x7fe6d015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d7c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d015b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d7c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d015e660 con 0x7fe6d015e280 2024-03-06T07:30:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d7c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00cb0e0 con 0x7fe6d00d2990 2024-03-06T07:30:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d7c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00cbb40 con 0x7fe6d00d4a00 2024-03-06T07:30:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d015b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d015b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40002/0 (socket says 172.21.15.32:40002) 2024-03-06T07:30:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d5c00640 1 -- 172.21.15.32:0/517294251 learned_addr learned my addr 172.21.15.32:0/517294251 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d4a00 0x7fe6d015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.666+0000 7fe6d5200640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6d00d2990 0x7fe6d01514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d5c00640 1 -- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d4a00 msgr2=0x7fe6d015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d5c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d4a00 0x7fe6d015a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d5c00640 1 -- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6d00d2990 msgr2=0x7fe6d01514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d5c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6d00d2990 0x7fe6d01514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d5c00640 1 -- 172.21.15.32:0/517294251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d01f3c80 con 0x7fe6d015e280 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d5c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d015b910 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800cef0 tx=0x7fe6c800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c801a070 con 0x7fe6d015e280 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c800d860 con 0x7fe6d015e280 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d0151a20 con 0x7fe6d015e280 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.667+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c800dc50 con 0x7fe6d015e280 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.668+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d01f44b0 con 0x7fe6d015e280 2024-03-06T07:30:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.669+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d004f330 con 0x7fe6d015e280 2024-03-06T07:30:36.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.670+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6c801d070 con 0x7fe6d015e280 2024-03-06T07:30:36.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.671+0000 7fe6ccc00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe6b006b4d0 0x7fe6b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.671+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6c808ab70 con 0x7fe6d015e280 2024-03-06T07:30:36.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.671+0000 7fe6d5200640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe6b006b4d0 0x7fe6b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.672+0000 7fe6d5200640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe6b006b4d0 0x7fe6b006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4036000 tx=0x7fe6c403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.676+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6d0151a20 con 0x7fe6d015e280 2024-03-06T07:30:36.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.878+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6d006cd70 con 0x7fe6d015e280 2024-03-06T07:30:36.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.879+0000 7fe6ccc00640 1 -- 172.21.15.32:0/517294251 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6c8059a10 con 0x7fe6d015e280 2024-03-06T07:30:36.880 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe6b006b4d0 msgr2=0x7fe6b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe6b006b4d0 0x7fe6b006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4036000 tx=0x7fe6c403d040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 msgr2=0x7fe6d015b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d015b910 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800cef0 tx=0x7fe6c800a710 comp rx=0 tx=0).stop 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 shutdown_connections 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe6b006b4d0 0x7fe6b006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d015e280 0x7fe6d015b910 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe6d00d4a00 0x7fe6d015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 --2- 172.21.15.32:0/517294251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6d00d2990 0x7fe6d01514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 >> 172.21.15.32:0/517294251 conn(0x7fe6d00d3ee0 msgr2=0x7fe6d00cbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.882+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 shutdown_connections 2024-03-06T07:30:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:36.883+0000 7fe6d7c00640 1 -- 172.21.15.32:0/517294251 wait complete. 2024-03-06T07:30:36.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.142+0000 7f0e58a00640 1 Processor -- start 2024-03-06T07:30:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.142+0000 7f0e58a00640 1 -- start start 2024-03-06T07:30:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.144+0000 7f0e58a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 0x7f0e54168dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.144+0000 7f0e58a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540b2c00 con 0x7f0e540d4890 2024-03-06T07:30:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.144+0000 7f0e58a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e54161ae0 con 0x7f0e54161700 2024-03-06T07:30:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.144+0000 7f0e58a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540b56e0 con 0x7f0e540d24f0 2024-03-06T07:30:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.144+0000 7f0e51e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0e540d4890 0x7f0e540d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42908/0 (socket says 172.21.15.32:42908) 2024-03-06T07:30:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.144+0000 7f0e51e00640 1 -- 172.21.15.32:0/3482855958 learned_addr learned my addr 172.21.15.32:0/3482855958 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e53200640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 0x7f0e54168dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011672216 0 0) 0x7f0e540b2c00 con 0x7f0e540d4890 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e38003520 con 0x7f0e540d4890 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3083536933 0 0) 0x7f0e38003520 con 0x7f0e540d4890 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 msgr2=0x7f0e54168dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 0x7f0e54168dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 >> v1:172.21.15.39:6789/0 conn(0x7f0e54161700 legacy=0x7f0e540b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e540b2ea0 con 0x7f0e540d4890 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f0e3c001670 con 0x7f0e540d4890 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 >> v1:172.21.15.32:6789/0 conn(0x7f0e540d4890 legacy=0x7f0e540d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e380039e0 0x7f0e38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.145+0000 7f0e51400640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 0x7f0e38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e51400640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e38004d40 0x7f0e380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540b56e0 con 0x7f0e380039e0 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3800baa0 con 0x7f0e38004d40 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3800bca0 con 0x7f0e540d24f0 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e52800640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e380039e0 0x7f0e38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e53200640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e38004d40 0x7f0e380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e51e00640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 0x7f0e38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e52800640 1 -- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 msgr2=0x7f0e38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.146+0000 7f0e52800640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540d24f0 0x7f0e38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.147+0000 7f0e52800640 1 -- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e38004d40 msgr2=0x7f0e380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.147+0000 7f0e52800640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e38004d40 0x7f0e380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.147+0000 7f0e52800640 1 -- 172.21.15.32:0/3482855958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38003520 con 0x7f0e380039e0 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.147+0000 7f0e53200640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e38004d40 0x7f0e380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.148+0000 7f0e52800640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e380039e0 0x7f0e38003e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0e44001e20 tx=0x7f0e38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc36ebcf2296ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.148+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44001a70 con 0x7f0e380039e0 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.148+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e44003b70 con 0x7f0e380039e0 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.148+0000 7f0e51400640 1 -- 172.21.15.32:0/3482855958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e440050d0 con 0x7f0e380039e0 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e58a00640 1 -- 172.21.15.32:0/3482855958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e380039e0 msgr2=0x7f0e38003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e58a00640 1 --2- 172.21.15.32:0/3482855958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e380039e0 0x7f0e38003e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0e44001e20 tx=0x7f0e38005830 comp rx=0 tx=0).stop 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e53200640 1 -- 172.21.15.32:0/3482855958 reap_dead start 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e58a00640 1 -- 172.21.15.32:0/3482855958 shutdown_connections 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e58a00640 1 -- 172.21.15.32:0/3482855958 >> 172.21.15.32:0/3482855958 conn(0x7f0e540d3d70 msgr2=0x7f0e540d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e58a00640 1 -- 172.21.15.32:0/3482855958 shutdown_connections 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.149+0000 7f0e58a00640 1 -- 172.21.15.32:0/3482855958 wait complete. 2024-03-06T07:30:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.151+0000 7f0e58a00640 1 Processor -- start 2024-03-06T07:30:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.151+0000 7f0e58a00640 1 -- start start 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.152+0000 7f0e58a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5414eb40 0x7f0e541f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.152+0000 7f0e58a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 0x7f0e541f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.152+0000 7f0e58a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540c78a0 0x7f0e540c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.152+0000 7f0e58a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540b0d00 con 0x7f0e541f7b00 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.152+0000 7f0e58a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540b0a20 con 0x7f0e5414eb40 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.152+0000 7f0e58a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540b2230 con 0x7f0e540c78a0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 0x7f0e541f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 0x7f0e541f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40008/0 (socket says 172.21.15.32:40008) 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 -- 172.21.15.32:0/2654756328 learned_addr learned my addr 172.21.15.32:0/2654756328 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 -- 172.21.15.32:0/2654756328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540c78a0 msgr2=0x7f0e540c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540c78a0 0x7f0e540c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 -- 172.21.15.32:0/2654756328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5414eb40 msgr2=0x7f0e541f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5414eb40 0x7f0e541f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 -- 172.21.15.32:0/2654756328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5414ef20 con 0x7f0e541f7b00 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.153+0000 7f0e51e00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 0x7f0e541f7f80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00eaa0 tx=0x7f0e3c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.154+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c00c820 con 0x7f0e541f7b00 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.154+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c018070 con 0x7f0e541f7b00 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.154+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c00cc00 con 0x7f0e541f7b00 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.154+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e540cb9f0 con 0x7f0e541f7b00 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.154+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e540cbfb0 con 0x7f0e541f7b00 2024-03-06T07:30:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.155+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5404f330 con 0x7f0e541f7b00 2024-03-06T07:30:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.156+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e3c01d460 con 0x7f0e541f7b00 2024-03-06T07:30:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.157+0000 7f0e4aa00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3006b4d0 0x7f0e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.157+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e3c08a010 con 0x7f0e541f7b00 2024-03-06T07:30:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.157+0000 7f0e52800640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3006b4d0 0x7f0e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.158+0000 7f0e52800640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3006b4d0 0x7f0e3006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e44034000 tx=0x7f0e44030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.165+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e3c058f20 con 0x7f0e541f7b00 2024-03-06T07:30:37.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.369+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e5406c0a0 con 0x7f0e541f7b00 2024-03-06T07:30:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.370+0000 7f0e4aa00640 1 -- 172.21.15.32:0/2654756328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e3c058820 con 0x7f0e541f7b00 2024-03-06T07:30:37.371 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3006b4d0 msgr2=0x7f0e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3006b4d0 0x7f0e3006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e44034000 tx=0x7f0e44030000 comp rx=0 tx=0).stop 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 msgr2=0x7f0e541f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 0x7f0e541f7f80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00eaa0 tx=0x7f0e3c00ef60 comp rx=0 tx=0).stop 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 shutdown_connections 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3006b4d0 0x7f0e3006d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e540c78a0 0x7f0e540c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e541f7b00 0x7f0e541f7f80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 --2- 172.21.15.32:0/2654756328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5414eb40 0x7f0e541f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.372+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 >> 172.21.15.32:0/2654756328 conn(0x7f0e540d3d70 msgr2=0x7f0e541602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.373+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 shutdown_connections 2024-03-06T07:30:37.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.373+0000 7f0e58a00640 1 -- 172.21.15.32:0/2654756328 wait complete. 2024-03-06T07:30:37.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.608+0000 7f06d8a00640 1 Processor -- start 2024-03-06T07:30:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.608+0000 7f06d8a00640 1 -- start start 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.609+0000 7f06d8a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f06d4161700 0x7f06d40b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.609+0000 7f06d8a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d40b2c00 con 0x7f06d40d4890 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.609+0000 7f06d8a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d4161ae0 con 0x7f06d40d24f0 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.609+0000 7f06d8a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d40b2ea0 con 0x7f06d4161700 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.610+0000 7f06d3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f06d4161700 0x7f06d40b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.610+0000 7f06caa00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f06d40d4890 0x7f06d40d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42910/0 (socket says 172.21.15.32:42910) 2024-03-06T07:30:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.610+0000 7f06caa00640 1 -- 172.21.15.32:0/3313547315 learned_addr learned my addr 172.21.15.32:0/3313547315 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:37.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.610+0000 7f06d3400640 1 -- 172.21.15.32:0/3313547315 >> v1:172.21.15.39:6789/0 conn(0x7f06d40d24f0 legacy=0x7f06d40b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.610+0000 7f06d3400640 1 -- 172.21.15.32:0/3313547315 >> v1:172.21.15.32:6789/0 conn(0x7f06d40d4890 legacy=0x7f06d40d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.610+0000 7f06d3400640 1 -- 172.21.15.32:0/3313547315 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d40b3180 con 0x7f06d4161700 2024-03-06T07:30:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.611+0000 7f06d3400640 1 --2- 172.21.15.32:0/3313547315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40b4fd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f06c40097a0 tx=0x7f06c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705c7b274dc28b6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.612+0000 7f06d2a00640 1 -- 172.21.15.32:0/3313547315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4035020 con 0x7f06d4161700 2024-03-06T07:30:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.612+0000 7f06d2a00640 1 -- 172.21.15.32:0/3313547315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4031830 con 0x7f06d4161700 2024-03-06T07:30:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.612+0000 7f06d2a00640 1 -- 172.21.15.32:0/3313547315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4031c00 con 0x7f06d4161700 2024-03-06T07:30:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.612+0000 7f06d8a00640 1 -- 172.21.15.32:0/3313547315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 msgr2=0x7f06d40b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.612+0000 7f06d8a00640 1 --2- 172.21.15.32:0/3313547315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40b4fd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f06c40097a0 tx=0x7f06c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.613+0000 7f06d8a00640 1 -- 172.21.15.32:0/3313547315 shutdown_connections 2024-03-06T07:30:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.613+0000 7f06d8a00640 1 --2- 172.21.15.32:0/3313547315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40b4fd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.613+0000 7f06d8a00640 1 -- 172.21.15.32:0/3313547315 >> 172.21.15.32:0/3313547315 conn(0x7f06d40d3d70 msgr2=0x7f06d40d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.613+0000 7f06d8a00640 1 -- 172.21.15.32:0/3313547315 shutdown_connections 2024-03-06T07:30:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.613+0000 7f06d8a00640 1 -- 172.21.15.32:0/3313547315 wait complete. 2024-03-06T07:30:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.614+0000 7f06d8a00640 1 Processor -- start 2024-03-06T07:30:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.615+0000 7f06d8a00640 1 -- start start 2024-03-06T07:30:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d40d24f0 0x7f06d415d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d8a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f06d40d4890 0x7f06d40b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d8a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40b5510 con 0x7f06d4161700 2024-03-06T07:30:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40b0d00 con 0x7f06d40d24f0 2024-03-06T07:30:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d8a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40b5690 con 0x7f06d40d4890 2024-03-06T07:30:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40034/0 (socket says 172.21.15.32:40034) 2024-03-06T07:30:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d3e00640 1 -- 172.21.15.32:0/2630629736 learned_addr learned my addr 172.21.15.32:0/2630629736 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06d3400640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d40d24f0 0x7f06d415d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.616+0000 7f06caa00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f06d40d4890 0x7f06d40b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d3e00640 1 -- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f06d40d4890 msgr2=0x7f06d40b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d3e00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f06d40d4890 0x7f06d40b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d3e00640 1 -- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d40d24f0 msgr2=0x7f06d415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d3e00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d40d24f0 0x7f06d415d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d3e00640 1 -- 172.21.15.32:0/2630629736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d415d8e0 con 0x7f06d4161700 2024-03-06T07:30:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06caa00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f06d40d4890 0x7f06d40b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d3e00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40c8240 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f06cc00c780 tx=0x7f06cc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc017070 con 0x7f06d4161700 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc003da0 con 0x7f06d4161700 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0052b0 con 0x7f06d4161700 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d40ccae0 con 0x7f06d4161700 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.617+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d40c7a40 con 0x7f06d4161700 2024-03-06T07:30:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.619+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d40cd000 con 0x7f06d4161700 2024-03-06T07:30:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.620+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06cc006b10 con 0x7f06d4161700 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.621+0000 7f06cbe00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f06b006b4d0 0x7f06b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.621+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06cc089690 con 0x7f06d4161700 2024-03-06T07:30:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.621+0000 7f06d3400640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f06b006b4d0 0x7f06b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.622+0000 7f06d3400640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f06b006b4d0 0x7f06b006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f06c4036000 tx=0x7f06c403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.626+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06d40cd000 con 0x7f06d4161700 2024-03-06T07:30:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.827+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06d40cfab0 con 0x7f06d4161700 2024-03-06T07:30:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.828+0000 7f06cbe00640 1 -- 172.21.15.32:0/2630629736 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f06d415d8e0 con 0x7f06d4161700 2024-03-06T07:30:37.829 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:30:37.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f06b006b4d0 msgr2=0x7f06b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f06b006b4d0 0x7f06b006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f06c4036000 tx=0x7f06c403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 msgr2=0x7f06d40c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40c8240 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f06cc00c780 tx=0x7f06cc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 shutdown_connections 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f06b006b4d0 0x7f06b006d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06d4161700 0x7f06d40c8240 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f06d40d4890 0x7f06d40b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 --2- 172.21.15.32:0/2630629736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d40d24f0 0x7f06d415d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.831+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 >> 172.21.15.32:0/2630629736 conn(0x7f06d40d3d70 msgr2=0x7f06d406cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.832+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 shutdown_connections 2024-03-06T07:30:37.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:37.832+0000 7f06d8a00640 1 -- 172.21.15.32:0/2630629736 wait complete. 2024-03-06T07:30:37.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.071+0000 7f20f1800640 1 Processor -- start 2024-03-06T07:30:38.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.072+0000 7f20f1800640 1 -- start start 2024-03-06T07:30:38.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.074+0000 7f20f1800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f20ec0af1a0 0x7f20ec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.074+0000 7f20f1800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.074+0000 7f20f1800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec15cc40 con 0x7f20ec0af580 2024-03-06T07:30:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.074+0000 7f20f1800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec15cf20 con 0x7f20ec0af1a0 2024-03-06T07:30:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.074+0000 7f20f1800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec15d200 con 0x7f20ec0cdc20 2024-03-06T07:30:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.075+0000 7f20eaa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f20ec0af1a0 0x7f20ec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.075+0000 7f20eb400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.075+0000 7f20e2000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f20ec0af580 0x7f20ec0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:43102/0 (socket says 172.21.15.32:43102) 2024-03-06T07:30:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.075+0000 7f20e2000640 1 -- 172.21.15.32:0/1083667695 learned_addr learned my addr 172.21.15.32:0/1083667695 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.076+0000 7f20eb400640 1 -- 172.21.15.32:0/1083667695 >> v2:172.21.15.39:3300/0 conn(0x7f20ec0af1a0 msgr2=0x7f20ec15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.076+0000 7f20ea000640 1 -- 172.21.15.32:0/1083667695 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166265428 0 0) 0x7f20ec15cc40 con 0x7f20ec0af580 2024-03-06T07:30:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.076+0000 7f20eb400640 1 --2- 172.21.15.32:0/1083667695 >> v2:172.21.15.39:3300/0 conn(0x7f20ec0af1a0 0x7f20ec15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.076+0000 7f20eb400640 1 -- 172.21.15.32:0/1083667695 >> v1:172.21.15.39:6789/0 conn(0x7f20ec0af580 legacy=0x7f20ec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.076+0000 7f20eb400640 1 -- 172.21.15.32:0/1083667695 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec15d4e0 con 0x7f20ec0cdc20 2024-03-06T07:30:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.078+0000 7f20eb400640 1 --2- 172.21.15.32:0/1083667695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec0b2dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20e40097a0 tx=0x7f20e402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e57e8df3193c1ec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.078+0000 7f20ea000640 1 -- 172.21.15.32:0/1083667695 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e40315a0 con 0x7f20ec0cdc20 2024-03-06T07:30:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.078+0000 7f20ea000640 1 -- 172.21.15.32:0/1083667695 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e4037070 con 0x7f20ec0cdc20 2024-03-06T07:30:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.078+0000 7f20ea000640 1 -- 172.21.15.32:0/1083667695 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e4031960 con 0x7f20ec0cdc20 2024-03-06T07:30:38.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 -- 172.21.15.32:0/1083667695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 msgr2=0x7f20ec0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 --2- 172.21.15.32:0/1083667695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec0b2dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20e40097a0 tx=0x7f20e402b840 comp rx=0 tx=0).stop 2024-03-06T07:30:38.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 -- 172.21.15.32:0/1083667695 shutdown_connections 2024-03-06T07:30:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 --2- 172.21.15.32:0/1083667695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec0b2dd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 --2- 172.21.15.32:0/1083667695 >> v2:172.21.15.39:3300/0 conn(0x7f20ec0af1a0 0x7f20ec15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 -- 172.21.15.32:0/1083667695 >> 172.21.15.32:0/1083667695 conn(0x7f20ec06afb0 msgr2=0x7f20ec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 -- 172.21.15.32:0/1083667695 shutdown_connections 2024-03-06T07:30:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.079+0000 7f20f1800640 1 -- 172.21.15.32:0/1083667695 wait complete. 2024-03-06T07:30:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.081+0000 7f20f1800640 1 Processor -- start 2024-03-06T07:30:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.081+0000 7f20f1800640 1 -- start start 2024-03-06T07:30:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20f1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20ec0af1a0 0x7f20ec14bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20f1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 0x7f20ec1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20f1800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec1f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20f1800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0b36e0 con 0x7f20ec0af580 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20f1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0b3560 con 0x7f20ec0af1a0 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20f1800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec155e70 con 0x7f20ec0cdc20 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20e2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 0x7f20ec1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20e2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 0x7f20ec1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41456/0 (socket says 172.21.15.32:41456) 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.082+0000 7f20e2000640 1 -- 172.21.15.32:0/155539471 learned_addr learned my addr 172.21.15.32:0/155539471 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20eaa00640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20ec0af1a0 0x7f20ec14bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20eb400640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec1f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e2000640 1 -- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 msgr2=0x7f20ec1f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e2000640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec1f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e2000640 1 -- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20ec0af1a0 msgr2=0x7f20ec14bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e2000640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20ec0af1a0 0x7f20ec14bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e2000640 1 -- 172.21.15.32:0/155539471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec1539b0 con 0x7f20ec0af580 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e2000640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 0x7f20ec1517c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f20d400c460 tx=0x7f20d400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.083+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4018070 con 0x7f20ec0af580 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.084+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4007520 con 0x7f20ec0af580 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.084+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec14c290 con 0x7f20ec0af580 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.084+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4004a20 con 0x7f20ec0af580 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.084+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec14c850 con 0x7f20ec0af580 2024-03-06T07:30:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.085+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec069510 con 0x7f20ec0af580 2024-03-06T07:30:38.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.086+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20d401b070 con 0x7f20ec0af580 2024-03-06T07:30:38.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.087+0000 7f20e3400640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f20c806b4d0 0x7f20c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.088+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20d40895b0 con 0x7f20ec0af580 2024-03-06T07:30:38.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.088+0000 7f20eaa00640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f20c806b4d0 0x7f20c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.089+0000 7f20eaa00640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f20c806b4d0 0x7f20c806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0074a0 tx=0x7f20dc00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.091+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20d4058420 con 0x7f20ec0af580 2024-03-06T07:30:38.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.304+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f20ec0b5060 con 0x7f20ec0af580 2024-03-06T07:30:38.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.305+0000 7f20e3400640 1 -- 172.21.15.32:0/155539471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f20d4057d20 con 0x7f20ec0af580 2024-03-06T07:30:38.306 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:30:38.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f20c806b4d0 msgr2=0x7f20c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f20c806b4d0 0x7f20c806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0074a0 tx=0x7f20dc00b040 comp rx=0 tx=0).stop 2024-03-06T07:30:38.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 msgr2=0x7f20ec1517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 0x7f20ec1517c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f20d400c460 tx=0x7f20d400c920 comp rx=0 tx=0).stop 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 shutdown_connections 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f20c806b4d0 0x7f20c806d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f20ec0cdc20 0x7f20ec1f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20ec0af580 0x7f20ec1517c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 --2- 172.21.15.32:0/155539471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20ec0af1a0 0x7f20ec14bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 >> 172.21.15.32:0/155539471 conn(0x7f20ec06afb0 msgr2=0x7f20ec071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 shutdown_connections 2024-03-06T07:30:38.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.308+0000 7f20f1800640 1 -- 172.21.15.32:0/155539471 wait complete. 2024-03-06T07:30:38.326 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:38.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:38.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:38.440 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:30:38.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:38.440 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:38.486 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:30:38.486 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:38.528 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:38.528 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:30:38.528 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:38.568 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:38.568 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:30:38.568 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:38.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:38.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:38.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.824+0000 7fec8b400640 1 Processor -- start 2024-03-06T07:30:38.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.825+0000 7fec8b400640 1 -- start start 2024-03-06T07:30:38.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.826+0000 7fec8b400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 0x7fec840b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.826+0000 7fec8b400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec840ae790 con 0x7fec8415b130 2024-03-06T07:30:38.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.826+0000 7fec8b400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec840b0760 con 0x7fec8415ad50 2024-03-06T07:30:38.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.826+0000 7fec8b400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406f050 con 0x7fec840723d0 2024-03-06T07:30:38.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.827+0000 7fec83e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fec8415b130 0x7fec84071cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55618/0 (socket says 172.21.15.32:55618) 2024-03-06T07:30:38.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.827+0000 7fec83e00640 1 -- 172.21.15.32:0/3025347679 learned_addr learned my addr 172.21.15.32:0/3025347679 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:38.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.827+0000 7fec89400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 0x7fec840b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.827+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604797325 0 0) 0x7fec840ae790 con 0x7fec8415b130 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.827+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6c005730 con 0x7fec8415b130 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309842967 0 0) 0x7fec6c005730 con 0x7fec8415b130 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 msgr2=0x7fec840b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 0x7fec840b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 >> v1:172.21.15.39:6789/0 conn(0x7fec8415ad50 legacy=0x7fec8406e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8406f330 con 0x7fec8415b130 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fec70001670 con 0x7fec8415b130 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 >> v1:172.21.15.32:6789/0 conn(0x7fec8415b130 legacy=0x7fec84071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 0x7fec6c005c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec6c006230 0x7fec6c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec6c006f50 0x7fec6c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406f050 con 0x7fec6c006f50 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c00de10 con 0x7fec6c006230 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c00e010 con 0x7fec840723d0 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec89400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec6c006f50 0x7fec6c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec88a00640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 0x7fec6c005c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.828+0000 7fec83e00640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec6c006230 0x7fec6c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.829+0000 7fec89400640 1 -- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 msgr2=0x7fec6c005c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.829+0000 7fec89400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec840723d0 0x7fec6c005c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.829+0000 7fec89400640 1 -- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec6c006230 msgr2=0x7fec6c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.829+0000 7fec89400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec6c006230 0x7fec6c006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.829+0000 7fec89400640 1 -- 172.21.15.32:0/3025347679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c005730 con 0x7fec6c006f50 2024-03-06T07:30:38.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.829+0000 7fec83e00640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec6c006230 0x7fec6c006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:38.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.830+0000 7fec89400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec6c006f50 0x7fec6c0073d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009f90 tx=0x7fec6c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41e461d4ea4c719 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.830+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c039070 con 0x7fec6c006f50 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.830+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c02cbd0 con 0x7fec6c006f50 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.830+0000 7fec83400640 1 -- 172.21.15.32:0/3025347679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c009c10 con 0x7fec6c006f50 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec8b400640 1 -- 172.21.15.32:0/3025347679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec6c006f50 msgr2=0x7fec6c0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec8b400640 1 --2- 172.21.15.32:0/3025347679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec6c006f50 0x7fec6c0073d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009f90 tx=0x7fec6c007a40 comp rx=0 tx=0).stop 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec89400640 1 -- 172.21.15.32:0/3025347679 reap_dead start 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec8b400640 1 -- 172.21.15.32:0/3025347679 shutdown_connections 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec8b400640 1 -- 172.21.15.32:0/3025347679 >> 172.21.15.32:0/3025347679 conn(0x7fec840b2a00 msgr2=0x7fec840b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec8b400640 1 -- 172.21.15.32:0/3025347679 shutdown_connections 2024-03-06T07:30:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.831+0000 7fec8b400640 1 -- 172.21.15.32:0/3025347679 wait complete. 2024-03-06T07:30:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.833+0000 7fec8b400640 1 Processor -- start 2024-03-06T07:30:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.833+0000 7fec8b400640 1 -- start start 2024-03-06T07:30:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec8b400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec6c006f50 0x7fec840aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 0x7fec84145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec8b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec841e55c0 0x7fec841e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec8b400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406eb60 con 0x7fec841454d0 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec8b400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406f8f0 con 0x7fec841e55c0 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec8b400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406ece0 con 0x7fec6c006f50 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 0x7fec84145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 0x7fec84145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41470/0 (socket says 172.21.15.32:41470) 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 -- 172.21.15.32:0/2548499887 learned_addr learned my addr 172.21.15.32:0/2548499887 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec88a00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec6c006f50 0x7fec840aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 -- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec6c006f50 msgr2=0x7fec840aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec6c006f50 0x7fec840aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 -- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec841e55c0 msgr2=0x7fec841e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec89400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec841e55c0 0x7fec841e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec841e55c0 0x7fec841e5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec83e00640 1 -- 172.21.15.32:0/2548499887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84145e70 con 0x7fec841454d0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.834+0000 7fec88a00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec6c006f50 0x7fec840aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.835+0000 7fec83e00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 0x7fec84145930 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fec70001200 tx=0x7fec7000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.835+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70019070 con 0x7fec841454d0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.835+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec700074e0 con 0x7fec841454d0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.835+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700049e0 con 0x7fec841454d0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.835+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec84146110 con 0x7fec841454d0 2024-03-06T07:30:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.835+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec841466d0 con 0x7fec841454d0 2024-03-06T07:30:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.837+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8404f3a0 con 0x7fec841454d0 2024-03-06T07:30:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.838+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec70004b80 con 0x7fec841454d0 2024-03-06T07:30:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.838+0000 7fec80c00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec6406b4d0 0x7fec6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.838+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec70089d20 con 0x7fec841454d0 2024-03-06T07:30:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.839+0000 7fec88a00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec6406b4d0 0x7fec6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.840+0000 7fec88a00640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec6406b4d0 0x7fec6406d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fec78001630 tx=0x7fec78009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:38.843+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec70058b10 con 0x7fec841454d0 2024-03-06T07:30:39.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.090+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec840b3ac0 con 0x7fec841454d0 2024-03-06T07:30:39.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.091+0000 7fec80c00640 1 -- 172.21.15.32:0/2548499887 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fec70058410 con 0x7fec841454d0 2024-03-06T07:30:39.093 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:39.093 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:39.093 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec6406b4d0 msgr2=0x7fec6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec6406b4d0 0x7fec6406d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fec78001630 tx=0x7fec78009040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 msgr2=0x7fec84145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 0x7fec84145930 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fec70001200 tx=0x7fec7000dd40 comp rx=0 tx=0).stop 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 shutdown_connections 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fec6406b4d0 0x7fec6406d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec841e55c0 0x7fec841e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec841454d0 0x7fec84145930 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 --2- 172.21.15.32:0/2548499887 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fec6c006f50 0x7fec840aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 >> 172.21.15.32:0/2548499887 conn(0x7fec840b2a00 msgr2=0x7fec840b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 shutdown_connections 2024-03-06T07:30:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.094+0000 7fec8b400640 1 -- 172.21.15.32:0/2548499887 wait complete. 2024-03-06T07:30:39.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:39.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.318+0000 7f7950800640 1 Processor -- start 2024-03-06T07:30:39.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.318+0000 7f7950800640 1 -- start start 2024-03-06T07:30:39.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.319+0000 7f7950800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f794806afb0 0x7f79480ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.319+0000 7f7950800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f79480b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.319+0000 7f7950800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79480af000 con 0x7f794815bfb0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.319+0000 7f7950800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79480b3de0 con 0x7f794806afb0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.319+0000 7f7950800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79480af2e0 con 0x7f794815cda0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.320+0000 7f794de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f794806afb0 0x7f79480ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.320+0000 7f794e800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f79480b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.320+0000 7f794e800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f79480b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:38448/0 (socket says 172.21.15.32:38448) 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.320+0000 7f794e800640 1 -- 172.21.15.32:0/254292729 learned_addr learned my addr 172.21.15.32:0/254292729 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.321+0000 7f794e800640 1 -- 172.21.15.32:0/254292729 >> v2:172.21.15.39:3300/0 conn(0x7f794806afb0 msgr2=0x7f79480ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.321+0000 7f794e800640 1 --2- 172.21.15.32:0/254292729 >> v2:172.21.15.39:3300/0 conn(0x7f794806afb0 0x7f79480ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.321+0000 7f794e800640 1 -- 172.21.15.32:0/254292729 >> v1:172.21.15.39:6789/0 conn(0x7f794815bfb0 legacy=0x7f794806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.321+0000 7f794e800640 1 -- 172.21.15.32:0/254292729 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480b4030 con 0x7f794815cda0 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.322+0000 7f794e800640 1 --2- 172.21.15.32:0/254292729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f79480b4830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7944009e50 tx=0x7f794402bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edb4bab031cb152 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.323+0000 7f794ca00640 1 -- 172.21.15.32:0/254292729 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794402caf0 con 0x7f794815cda0 2024-03-06T07:30:39.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.323+0000 7f794ca00640 1 -- 172.21.15.32:0/254292729 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794402cc90 con 0x7f794815cda0 2024-03-06T07:30:39.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.323+0000 7f794ca00640 1 -- 172.21.15.32:0/254292729 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944034610 con 0x7f794815cda0 2024-03-06T07:30:39.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.323+0000 7f7950800640 1 -- 172.21.15.32:0/254292729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 msgr2=0x7f79480b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.323+0000 7f7950800640 1 --2- 172.21.15.32:0/254292729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f79480b4830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7944009e50 tx=0x7f794402bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:39.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.324+0000 7f7950800640 1 -- 172.21.15.32:0/254292729 shutdown_connections 2024-03-06T07:30:39.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.324+0000 7f7950800640 1 --2- 172.21.15.32:0/254292729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f79480b4830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.324+0000 7f7950800640 1 --2- 172.21.15.32:0/254292729 >> v2:172.21.15.39:3300/0 conn(0x7f794806afb0 0x7f79480ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.324+0000 7f7950800640 1 -- 172.21.15.32:0/254292729 >> 172.21.15.32:0/254292729 conn(0x7f79480b2ae0 msgr2=0x7f79480b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.324+0000 7f7950800640 1 -- 172.21.15.32:0/254292729 shutdown_connections 2024-03-06T07:30:39.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.324+0000 7f7950800640 1 -- 172.21.15.32:0/254292729 wait complete. 2024-03-06T07:30:39.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.325+0000 7f7950800640 1 Processor -- start 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.326+0000 7f7950800640 1 -- start start 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.326+0000 7f7950800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f794806afb0 0x7f79480d0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f7950800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 0x7f7948146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f7950800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f7948157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f7950800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79480b4d70 con 0x7f794815bfb0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f7950800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794815d210 con 0x7f794806afb0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f7950800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79480b4ef0 con 0x7f794815cda0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 0x7f7948146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 0x7f7948146660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41480/0 (socket says 172.21.15.32:41480) 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 -- 172.21.15.32:0/1395590922 learned_addr learned my addr 172.21.15.32:0/1395590922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794de00640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f794806afb0 0x7f79480d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794e800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f7948157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 -- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 msgr2=0x7f7948157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f7948157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 -- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f794806afb0 msgr2=0x7f79480d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f794806afb0 0x7f79480d0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.327+0000 7f794d400640 1 -- 172.21.15.32:0/1395590922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7948146ba0 con 0x7f794815bfb0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.328+0000 7f794d400640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 0x7f7948146660 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f793800c400 tx=0x7f793800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.328+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380052d0 con 0x7f794815bfb0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.328+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7938005490 con 0x7f794815bfb0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.328+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938007260 con 0x7f794815bfb0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.328+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7948147670 con 0x7f794815bfb0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.328+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7948147c30 con 0x7f794815bfb0 2024-03-06T07:30:39.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.329+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794804f3a0 con 0x7f794815bfb0 2024-03-06T07:30:39.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.330+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7938007400 con 0x7f794815bfb0 2024-03-06T07:30:39.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.332+0000 7f793e000640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f792c06b4d0 0x7f792c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.332+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7938089760 con 0x7f794815bfb0 2024-03-06T07:30:39.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.332+0000 7f794de00640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f792c06b4d0 0x7f792c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.333+0000 7f794de00640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f792c06b4d0 0x7f792c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7940009a10 tx=0x7f7940008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.336+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7948146ba0 con 0x7f794815bfb0 2024-03-06T07:30:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.556+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f794815c830 con 0x7f794815bfb0 2024-03-06T07:30:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.558+0000 7f793e000640 1 -- 172.21.15.32:0/1395590922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16561 (secure 0 0 0) 0x7f79380585d0 con 0x7f794815bfb0 2024-03-06T07:30:39.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:13.040051+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.560+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f792c06b4d0 msgr2=0x7f792c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.560+0000 7f7950800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f792c06b4d0 0x7f792c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7940009a10 tx=0x7f7940008040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.560+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 msgr2=0x7f7948146660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.560+0000 7f7950800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 0x7f7948146660 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f793800c400 tx=0x7f793800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 shutdown_connections 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f792c06b4d0 0x7f792c06d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f794815cda0 0x7f7948157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f794815bfb0 0x7f7948146660 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 --2- 172.21.15.32:0/1395590922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f794806afb0 0x7f79480d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 >> 172.21.15.32:0/1395590922 conn(0x7f79480b2ae0 msgr2=0x7f7948071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 shutdown_connections 2024-03-06T07:30:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.561+0000 7f7950800640 1 -- 172.21.15.32:0/1395590922 wait complete. 2024-03-06T07:30:39.581 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:09.502397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:10.547731+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:39.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:39.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.795+0000 7f634d400640 1 Processor -- start 2024-03-06T07:30:39.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.795+0000 7f634d400640 1 -- start start 2024-03-06T07:30:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.797+0000 7f634d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6348162390 0x7f63480b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.797+0000 7f634d400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6348162770 con 0x7f634806afb0 2024-03-06T07:30:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.797+0000 7f634d400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634809ef10 con 0x7f634806bb10 2024-03-06T07:30:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.797+0000 7f634d400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6348072e20 con 0x7f6348162390 2024-03-06T07:30:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.798+0000 7f6345e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f634806afb0 0x7f634806b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55620/0 (socket says 172.21.15.32:55620) 2024-03-06T07:30:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.798+0000 7f6345e00640 1 -- 172.21.15.32:0/3736819378 learned_addr learned my addr 172.21.15.32:0/3736819378 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.798+0000 7f6346800640 1 --2- 172.21.15.32:0/3736819378 >> v2:172.21.15.39:3300/0 conn(0x7f6348162390 0x7f63480b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.798+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138674342 0 0) 0x7f6348162770 con 0x7f634806afb0 2024-03-06T07:30:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.798+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c003520 con 0x7f634806afb0 2024-03-06T07:30:39.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754947528 0 0) 0x7f632c003520 con 0x7f634806afb0 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 >> v2:172.21.15.39:3300/0 conn(0x7f6348162390 msgr2=0x7f63480b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 --2- 172.21.15.32:0/3736819378 >> v2:172.21.15.39:3300/0 conn(0x7f6348162390 0x7f63480b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 >> v1:172.21.15.39:6789/0 conn(0x7f634806bb10 legacy=0x7f63480addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348071da0 con 0x7f634806afb0 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6330001670 con 0x7f634806afb0 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 >> v1:172.21.15.32:6789/0 conn(0x7f634806afb0 legacy=0x7f634806b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f632c003a00 0x7f632c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f632c004800 0x7f632c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c0051c0 0x7f632c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c00c090 con 0x7f632c003a00 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c00c210 con 0x7f632c0051c0 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c00c410 con 0x7f632c004800 2024-03-06T07:30:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.799+0000 7f6346800640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f632c003a00 0x7f632c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6347200640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c0051c0 0x7f632c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6345e00640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f632c004800 0x7f632c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6346800640 1 -- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f632c004800 msgr2=0x7f632c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6346800640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f632c004800 0x7f632c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6346800640 1 -- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c0051c0 msgr2=0x7f632c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6346800640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c0051c0 0x7f632c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6346800640 1 -- 172.21.15.32:0/3736819378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c003520 con 0x7f632c003a00 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6347200640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c0051c0 0x7f632c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6347200640 1 -- 172.21.15.32:0/3736819378 reap_dead start 2024-03-06T07:30:39.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.800+0000 7f6345e00640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f632c004800 0x7f632c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:39.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.802+0000 7f6346800640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f632c003a00 0x7f632c003e00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6338009fc0 tx=0x7f632c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d3f8631b61322e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.802+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338039070 con 0x7f632c003a00 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.802+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633802cc20 con 0x7f632c003a00 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.802+0000 7f6345400640 1 -- 172.21.15.32:0/3736819378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338009c10 con 0x7f632c003a00 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.802+0000 7f634d400640 1 -- 172.21.15.32:0/3736819378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f632c003a00 msgr2=0x7f632c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.802+0000 7f634d400640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f632c003a00 0x7f632c003e00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6338009fc0 tx=0x7f632c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.803+0000 7f634d400640 1 -- 172.21.15.32:0/3736819378 shutdown_connections 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.803+0000 7f634d400640 1 --2- 172.21.15.32:0/3736819378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f632c003a00 0x7f632c003e00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.803+0000 7f634d400640 1 -- 172.21.15.32:0/3736819378 >> 172.21.15.32:0/3736819378 conn(0x7f63480b2ae0 msgr2=0x7f63480b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.803+0000 7f634d400640 1 -- 172.21.15.32:0/3736819378 shutdown_connections 2024-03-06T07:30:39.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.803+0000 7f634d400640 1 -- 172.21.15.32:0/3736819378 wait complete. 2024-03-06T07:30:39.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.804+0000 7f634d400640 1 Processor -- start 2024-03-06T07:30:39.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.805+0000 7f634d400640 1 -- start start 2024-03-06T07:30:39.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.805+0000 7f634d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c003a00 0x7f63481542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.805+0000 7f634d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6348157290 0x7f63481653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.805+0000 7f634d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 0x7f6348158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f634d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6348072620 con 0x7f6348157710 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f634d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6348072340 con 0x7f632c003a00 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f634d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6348072900 con 0x7f6348157290 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 0x7f6348158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 0x7f6348158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41488/0 (socket says 172.21.15.32:41488) 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 -- 172.21.15.32:0/2420254455 learned_addr learned my addr 172.21.15.32:0/2420254455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6345e00640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6348157290 0x7f63481653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6346800640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c003a00 0x7f63481542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 -- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6348157290 msgr2=0x7f63481653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6348157290 0x7f63481653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 -- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c003a00 msgr2=0x7f63481542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c003a00 0x7f63481542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.806+0000 7f6347200640 1 -- 172.21.15.32:0/2420254455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348155f10 con 0x7f6348157710 2024-03-06T07:30:39.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.807+0000 7f6347200640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 0x7f6348158c00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f63480b53e0 tx=0x7f634000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.807+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340005230 con 0x7f6348157710 2024-03-06T07:30:39.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.807+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63400053d0 con 0x7f6348157710 2024-03-06T07:30:39.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.807+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340007260 con 0x7f6348157710 2024-03-06T07:30:39.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.807+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63481561f0 con 0x7f6348157710 2024-03-06T07:30:39.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.807+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63481567b0 con 0x7f6348157710 2024-03-06T07:30:39.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.809+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634804f330 con 0x7f6348157710 2024-03-06T07:30:39.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.809+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6340007400 con 0x7f6348157710 2024-03-06T07:30:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.810+0000 7f633ea00640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f632406b4d0 0x7f632406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.810+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6340089490 con 0x7f6348157710 2024-03-06T07:30:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.811+0000 7f6346800640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f632406b4d0 0x7f632406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.811+0000 7f6346800640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f632406b4d0 0x7f632406d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6338036000 tx=0x7f633802e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:39.815+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6340058270 con 0x7f6348157710 2024-03-06T07:30:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.017+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63480b0300 con 0x7f6348157710 2024-03-06T07:30:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.018+0000 7f633ea00640 1 -- 172.21.15.32:0/2420254455 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6340057b70 con 0x7f6348157710 2024-03-06T07:30:40.019 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:30:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f632406b4d0 msgr2=0x7f632406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f632406b4d0 0x7f632406d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6338036000 tx=0x7f633802e000 comp rx=0 tx=0).stop 2024-03-06T07:30:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 msgr2=0x7f6348158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 0x7f6348158c00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f63480b53e0 tx=0x7f634000d890 comp rx=0 tx=0).stop 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 shutdown_connections 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f632406b4d0 0x7f632406d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6348157710 0x7f6348158c00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6348157290 0x7f63481653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 --2- 172.21.15.32:0/2420254455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632c003a00 0x7f63481542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 >> 172.21.15.32:0/2420254455 conn(0x7f63480b2ae0 msgr2=0x7f634806f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 shutdown_connections 2024-03-06T07:30:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.020+0000 7f634d400640 1 -- 172.21.15.32:0/2420254455 wait complete. 2024-03-06T07:30:40.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:40.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.257+0000 7f288a800640 1 Processor -- start 2024-03-06T07:30:40.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.257+0000 7f288a800640 1 -- start start 2024-03-06T07:30:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.259+0000 7f288a800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28840cd1e0 0x7f28840cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.259+0000 7f288a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2884070ce0 0x7f28840b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.259+0000 7f288a800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2884071c60 con 0x7f288415afa0 2024-03-06T07:30:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.259+0000 7f288a800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28840cfa10 con 0x7f28840cd1e0 2024-03-06T07:30:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.259+0000 7f288a800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2884071f40 con 0x7f2884070ce0 2024-03-06T07:30:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.259+0000 7f2882a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28840cd1e0 0x7f28840cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2882a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28840cd1e0 0x7f28840cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41492/0 (socket says 172.21.15.32:41492) 2024-03-06T07:30:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2882a00640 1 -- 172.21.15.32:0/2313893563 learned_addr learned my addr 172.21.15.32:0/2313893563 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2883e00640 1 --2- 172.21.15.32:0/2313893563 >> v2:172.21.15.39:3300/0 conn(0x7f2884070ce0 0x7f28840b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2882a00640 1 -- 172.21.15.32:0/2313893563 >> v2:172.21.15.39:3300/0 conn(0x7f2884070ce0 msgr2=0x7f28840b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2882a00640 1 --2- 172.21.15.32:0/2313893563 >> v2:172.21.15.39:3300/0 conn(0x7f2884070ce0 0x7f28840b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2882a00640 1 -- 172.21.15.32:0/2313893563 >> v1:172.21.15.39:6789/0 conn(0x7f288415afa0 legacy=0x7f28840ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.260+0000 7f2882a00640 1 -- 172.21.15.32:0/2313893563 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2884072220 con 0x7f28840cd1e0 2024-03-06T07:30:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.261+0000 7f2882a00640 1 --2- 172.21.15.32:0/2313893563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28840cd1e0 0x7f28840cf3a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f28840b52b0 tx=0x7f287002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2bb90f04599daa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.261+0000 7f2882000640 1 -- 172.21.15.32:0/2313893563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287002e070 con 0x7f28840cd1e0 2024-03-06T07:30:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f2882000640 1 -- 172.21.15.32:0/2313893563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2870037070 con 0x7f28840cd1e0 2024-03-06T07:30:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f2882000640 1 -- 172.21.15.32:0/2313893563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870031a00 con 0x7f28840cd1e0 2024-03-06T07:30:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 -- 172.21.15.32:0/2313893563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28840cd1e0 msgr2=0x7f28840cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 --2- 172.21.15.32:0/2313893563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28840cd1e0 0x7f28840cf3a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f28840b52b0 tx=0x7f287002b880 comp rx=0 tx=0).stop 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 -- 172.21.15.32:0/2313893563 shutdown_connections 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 --2- 172.21.15.32:0/2313893563 >> v2:172.21.15.39:3300/0 conn(0x7f2884070ce0 0x7f28840b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 --2- 172.21.15.32:0/2313893563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28840cd1e0 0x7f28840cf3a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 -- 172.21.15.32:0/2313893563 >> 172.21.15.32:0/2313893563 conn(0x7f288406b3d0 msgr2=0x7f288406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 -- 172.21.15.32:0/2313893563 shutdown_connections 2024-03-06T07:30:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.262+0000 7f288a800640 1 -- 172.21.15.32:0/2313893563 wait complete. 2024-03-06T07:30:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.264+0000 7f288a800640 1 Processor -- start 2024-03-06T07:30:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.264+0000 7f288a800640 1 -- start start 2024-03-06T07:30:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f288a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 0x7f28840b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f288a800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28840cd1e0 0x7f28840afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f288a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288415afa0 0x7f28840b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f288a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28840b4000 con 0x7f2884070ce0 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f288a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884071110 con 0x7f288415afa0 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f288a800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28840b4180 con 0x7f28840cd1e0 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f2883400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 0x7f28840b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f2883400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 0x7f28840b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41502/0 (socket says 172.21.15.32:41502) 2024-03-06T07:30:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.265+0000 7f2883400640 1 -- 172.21.15.32:0/3793679802 learned_addr learned my addr 172.21.15.32:0/3793679802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:40.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883e00640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288415afa0 0x7f28840b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2882a00640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28840cd1e0 0x7f28840afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883400640 1 -- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28840cd1e0 msgr2=0x7f28840afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883400640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28840cd1e0 0x7f28840afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883400640 1 -- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288415afa0 msgr2=0x7f28840b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883400640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288415afa0 0x7f28840b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883400640 1 -- 172.21.15.32:0/3793679802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28840b25f0 con 0x7f2884070ce0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883e00640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288415afa0 0x7f28840b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2882a00640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28840cd1e0 0x7f28840afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f2883400640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 0x7f28840b20b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f287400b410 tx=0x7f287400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.266+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874015020 con 0x7f2884070ce0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.267+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874011830 con 0x7f2884070ce0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.267+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874011c20 con 0x7f2884070ce0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.267+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28840b0820 con 0x7f2884070ce0 2024-03-06T07:30:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.267+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288414cdb0 con 0x7f2884070ce0 2024-03-06T07:30:40.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.268+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288404f330 con 0x7f2884070ce0 2024-03-06T07:30:40.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.269+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2874018020 con 0x7f2884070ce0 2024-03-06T07:30:40.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.270+0000 7f287b400640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f286406b4d0 0x7f286406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.271+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f287408a120 con 0x7f2884070ce0 2024-03-06T07:30:40.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.271+0000 7f2882a00640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f286406b4d0 0x7f286406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.271+0000 7f2882a00640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f286406b4d0 0x7f286406d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2870036000 tx=0x7f2870008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.274+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f287401f020 con 0x7f2884070ce0 2024-03-06T07:30:40.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.474+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f288406ba90 con 0x7f2884070ce0 2024-03-06T07:30:40.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.475+0000 7f287b400640 1 -- 172.21.15.32:0/3793679802 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f28740172c0 con 0x7f2884070ce0 2024-03-06T07:30:40.478 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f286406b4d0 msgr2=0x7f286406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f286406b4d0 0x7f286406d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2870036000 tx=0x7f2870008030 comp rx=0 tx=0).stop 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 msgr2=0x7f28840b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 0x7f28840b20b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f287400b410 tx=0x7f287400b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 shutdown_connections 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f286406b4d0 0x7f286406d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f288415afa0 0x7f28840b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28840cd1e0 0x7f28840afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 --2- 172.21.15.32:0/3793679802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2884070ce0 0x7f28840b20b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.478+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 >> 172.21.15.32:0/3793679802 conn(0x7f288406b3d0 msgr2=0x7f28840ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.479+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 shutdown_connections 2024-03-06T07:30:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.479+0000 7f288a800640 1 -- 172.21.15.32:0/3793679802 wait complete. 2024-03-06T07:30:40.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.720+0000 7fdcba400640 1 Processor -- start 2024-03-06T07:30:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.720+0000 7fdcba400640 1 -- start start 2024-03-06T07:30:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.722+0000 7fdcba400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdcb4145430 0x7fdcb415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.722+0000 7fdcba400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdcb415cc90 0x7fdcb40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.722+0000 7fdcba400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40b3fe0 con 0x7fdcb40b2080 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.722+0000 7fdcba400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb40b24d0 con 0x7fdcb415cc90 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.722+0000 7fdcba400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb415fe40 con 0x7fdcb4145430 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.722+0000 7fdcabe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdcb415cc90 0x7fdcb40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcabe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdcb415cc90 0x7fdcb40b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41510/0 (socket says 172.21.15.32:41510) 2024-03-06T07:30:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcabe00640 1 -- 172.21.15.32:0/3536393419 learned_addr learned my addr 172.21.15.32:0/3536393419 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcb9000640 1 --2- 172.21.15.32:0/3536393419 >> v2:172.21.15.39:3300/0 conn(0x7fdcb4145430 0x7fdcb415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcabe00640 1 -- 172.21.15.32:0/3536393419 >> v2:172.21.15.39:3300/0 conn(0x7fdcb4145430 msgr2=0x7fdcb415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcabe00640 1 --2- 172.21.15.32:0/3536393419 >> v2:172.21.15.39:3300/0 conn(0x7fdcb4145430 0x7fdcb415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcabe00640 1 -- 172.21.15.32:0/3536393419 >> v1:172.21.15.39:6789/0 conn(0x7fdcb40b2080 legacy=0x7fdcb406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.723+0000 7fdcabe00640 1 -- 172.21.15.32:0/3536393419 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb4160120 con 0x7fdcb415cc90 2024-03-06T07:30:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.725+0000 7fdcabe00640 1 --2- 172.21.15.32:0/3536393419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb415cc90 0x7fdcb40b1b40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40b2670 tx=0x7fdc9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac358cf82ea06585 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.725+0000 7fdcb3e00640 1 -- 172.21.15.32:0/3536393419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c035020 con 0x7fdcb415cc90 2024-03-06T07:30:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.725+0000 7fdcb3e00640 1 -- 172.21.15.32:0/3536393419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c031870 con 0x7fdcb415cc90 2024-03-06T07:30:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.725+0000 7fdcb3e00640 1 -- 172.21.15.32:0/3536393419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c031c40 con 0x7fdcb415cc90 2024-03-06T07:30:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 -- 172.21.15.32:0/3536393419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb415cc90 msgr2=0x7fdcb40b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 --2- 172.21.15.32:0/3536393419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb415cc90 0x7fdcb40b1b40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40b2670 tx=0x7fdc9c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 -- 172.21.15.32:0/3536393419 shutdown_connections 2024-03-06T07:30:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 --2- 172.21.15.32:0/3536393419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb415cc90 0x7fdcb40b1b40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 --2- 172.21.15.32:0/3536393419 >> v2:172.21.15.39:3300/0 conn(0x7fdcb4145430 0x7fdcb415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 -- 172.21.15.32:0/3536393419 >> 172.21.15.32:0/3536393419 conn(0x7fdcb406b3a0 msgr2=0x7fdcb406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 -- 172.21.15.32:0/3536393419 shutdown_connections 2024-03-06T07:30:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.726+0000 7fdcba400640 1 -- 172.21.15.32:0/3536393419 wait complete. 2024-03-06T07:30:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.728+0000 7fdcba400640 1 Processor -- start 2024-03-06T07:30:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.728+0000 7fdcba400640 1 -- start start 2024-03-06T07:30:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcba400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcb40b2080 0x7fdcb415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcba400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 0x7fdcb41f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcba400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdcb415cc90 0x7fdcb406c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcba400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb415d0d0 con 0x7fdcb4145430 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcba400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb41606e0 con 0x7fdcb40b2080 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcba400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4145810 con 0x7fdcb415cc90 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcabe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 0x7fdcb41f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcabe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 0x7fdcb41f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41516/0 (socket says 172.21.15.32:41516) 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcabe00640 1 -- 172.21.15.32:0/988339835 learned_addr learned my addr 172.21.15.32:0/988339835 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcb9000640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcb40b2080 0x7fdcb415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcabe00640 1 -- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdcb415cc90 msgr2=0x7fdcb406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.729+0000 7fdcabe00640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdcb415cc90 0x7fdcb406c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcabe00640 1 -- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcb40b2080 msgr2=0x7fdcb415c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcabe00640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcb40b2080 0x7fdcb415c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcabe00640 1 -- 172.21.15.32:0/988339835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb41f6400 con 0x7fdcb4145430 2024-03-06T07:30:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcabe00640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 0x7fdcb41f5b80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000c00 tx=0x7fdc9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c040070 con 0x7fdcb4145430 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c030d00 con 0x7fdcb4145430 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c035040 con 0x7fdcb4145430 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb41f66e0 con 0x7fdcb4145430 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.730+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb41f6ca0 con 0x7fdcb4145430 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.731+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb404f330 con 0x7fdcb4145430 2024-03-06T07:30:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.733+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc9c04b060 con 0x7fdcb4145430 2024-03-06T07:30:40.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.733+0000 7fdcb1600640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc9006b4d0 0x7fdc9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.734+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc9c0abf90 con 0x7fdcb4145430 2024-03-06T07:30:40.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.734+0000 7fdcb9000640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc9006b4d0 0x7fdc9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.734+0000 7fdcb9000640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc9006b4d0 0x7fdc9006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006fd0 tx=0x7fdca4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.738+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc9c07ad70 con 0x7fdcb4145430 2024-03-06T07:30:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.989+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdcb406e240 con 0x7fdcb4145430 2024-03-06T07:30:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.989+0000 7fdcb1600640 1 -- 172.21.15.32:0/988339835 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdc9c07a670 con 0x7fdcb4145430 2024-03-06T07:30:40.992 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:30:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc9006b4d0 msgr2=0x7fdc9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc9006b4d0 0x7fdc9006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006fd0 tx=0x7fdca4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 msgr2=0x7fdcb41f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 0x7fdcb41f5b80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000c00 tx=0x7fdc9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 shutdown_connections 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc9006b4d0 0x7fdc9006d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdcb415cc90 0x7fdcb406c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcb4145430 0x7fdcb41f5b80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 --2- 172.21.15.32:0/988339835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcb40b2080 0x7fdcb415c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 >> 172.21.15.32:0/988339835 conn(0x7fdcb406b3a0 msgr2=0x7fdcb40b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 shutdown_connections 2024-03-06T07:30:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:40.992+0000 7fdcba400640 1 -- 172.21.15.32:0/988339835 wait complete. 2024-03-06T07:30:41.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.218+0000 7f7d4a200640 1 Processor -- start 2024-03-06T07:30:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.218+0000 7f7d4a200640 1 -- start start 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d4a200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d440af580 0x7f7d4415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d4a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440cdc20 0x7f7d440b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d4a200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4415cc40 con 0x7f7d440af1a0 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d4a200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4415cf20 con 0x7f7d440af580 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d4a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4415d200 con 0x7f7d440cdc20 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d42a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d440af580 0x7f7d4415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d42a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d440af580 0x7f7d4415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41532/0 (socket says 172.21.15.32:41532) 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.220+0000 7f7d42a00640 1 -- 172.21.15.32:0/856508684 learned_addr learned my addr 172.21.15.32:0/856508684 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.221+0000 7f7d43e00640 1 --2- 172.21.15.32:0/856508684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440cdc20 0x7f7d440b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.221+0000 7f7d42a00640 1 -- 172.21.15.32:0/856508684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440cdc20 msgr2=0x7f7d440b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.221+0000 7f7d42a00640 1 --2- 172.21.15.32:0/856508684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440cdc20 0x7f7d440b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.221+0000 7f7d42a00640 1 -- 172.21.15.32:0/856508684 >> v1:172.21.15.39:6789/0 conn(0x7f7d440af1a0 legacy=0x7f7d440b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.221+0000 7f7d42a00640 1 -- 172.21.15.32:0/856508684 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4415d4e0 con 0x7f7d440af580 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.223+0000 7f7d42a00640 1 --2- 172.21.15.32:0/856508684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440af580 0x7f7d4415c600 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009870 tx=0x7f7d3002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16e65d1c807ae7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.223+0000 7f7d42000640 1 -- 172.21.15.32:0/856508684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300317c0 con 0x7f7d440af580 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.223+0000 7f7d42000640 1 -- 172.21.15.32:0/856508684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30037070 con 0x7f7d440af580 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.223+0000 7f7d42000640 1 -- 172.21.15.32:0/856508684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30031b80 con 0x7f7d440af580 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 -- 172.21.15.32:0/856508684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440af580 msgr2=0x7f7d4415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 --2- 172.21.15.32:0/856508684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440af580 0x7f7d4415c600 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009870 tx=0x7f7d3002b860 comp rx=0 tx=0).stop 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 -- 172.21.15.32:0/856508684 shutdown_connections 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 --2- 172.21.15.32:0/856508684 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440cdc20 0x7f7d440b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 --2- 172.21.15.32:0/856508684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440af580 0x7f7d4415c600 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 -- 172.21.15.32:0/856508684 >> 172.21.15.32:0/856508684 conn(0x7f7d4406afb0 msgr2=0x7f7d4406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 -- 172.21.15.32:0/856508684 shutdown_connections 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.224+0000 7f7d4a200640 1 -- 172.21.15.32:0/856508684 wait complete. 2024-03-06T07:30:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.226+0000 7f7d4a200640 1 Processor -- start 2024-03-06T07:30:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.226+0000 7f7d4a200640 1 -- start start 2024-03-06T07:30:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d4a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d440af1a0 0x7f7d440ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d4a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440af580 0x7f7d441f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d4a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 0x7f7d44071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d4a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440b3720 con 0x7f7d440cdc20 2024-03-06T07:30:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d4a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440b35a0 con 0x7f7d440af1a0 2024-03-06T07:30:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d4a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d44155e70 con 0x7f7d440af580 2024-03-06T07:30:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 0x7f7d44071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 0x7f7d44071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41542/0 (socket says 172.21.15.32:41542) 2024-03-06T07:30:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.227+0000 7f7d43e00640 1 -- 172.21.15.32:0/2821352670 learned_addr learned my addr 172.21.15.32:0/2821352670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:41.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d42a00640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440af580 0x7f7d441f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43400640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d440af1a0 0x7f7d440ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43e00640 1 -- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440af580 msgr2=0x7f7d441f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43e00640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440af580 0x7f7d441f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43e00640 1 -- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d440af1a0 msgr2=0x7f7d440ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43e00640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d440af1a0 0x7f7d440ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43e00640 1 -- 172.21.15.32:0/2821352670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440cf330 con 0x7f7d440cdc20 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43400640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d440af1a0 0x7f7d440ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d42a00640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440af580 0x7f7d441f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d43e00640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 0x7f7d44071620 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c007e60 tx=0x7f7d3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c014420 con 0x7f7d440cdc20 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c019070 con 0x7f7d440cdc20 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c014820 con 0x7f7d440cdc20 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.228+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d441f73c0 con 0x7f7d440cdc20 2024-03-06T07:30:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.229+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d441f7ea0 con 0x7f7d440cdc20 2024-03-06T07:30:41.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.230+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d44069510 con 0x7f7d440cdc20 2024-03-06T07:30:41.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.231+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d3c0149c0 con 0x7f7d440cdc20 2024-03-06T07:30:41.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.232+0000 7f7d3b400640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d2406b500 0x7f7d2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.232+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d3c08aad0 con 0x7f7d440cdc20 2024-03-06T07:30:41.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.232+0000 7f7d43400640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d2406b500 0x7f7d2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.233+0000 7f7d43400640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d2406b500 0x7f7d2406d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d440cfa70 tx=0x7f7d34009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.241+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d3c0598b0 con 0x7f7d440cdc20 2024-03-06T07:30:41.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.480+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d4404f3a0 con 0x7f7d440cdc20 2024-03-06T07:30:41.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.481+0000 7f7d3b400640 1 -- 172.21.15.32:0/2821352670 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f7d3c0591b0 con 0x7f7d440cdc20 2024-03-06T07:30:41.484 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:41.484 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:41.484 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:41.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d2406b500 msgr2=0x7f7d2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d2406b500 0x7f7d2406d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d440cfa70 tx=0x7f7d34009040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 msgr2=0x7f7d44071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 0x7f7d44071620 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c007e60 tx=0x7f7d3c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 shutdown_connections 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d2406b500 0x7f7d2406d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d440cdc20 0x7f7d44071620 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7d440af580 0x7f7d441f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 --2- 172.21.15.32:0/2821352670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d440af1a0 0x7f7d440ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 >> 172.21.15.32:0/2821352670 conn(0x7f7d4406afb0 msgr2=0x7f7d440cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 shutdown_connections 2024-03-06T07:30:41.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.484+0000 7f7d4a200640 1 -- 172.21.15.32:0/2821352670 wait complete. 2024-03-06T07:30:41.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.722+0000 7f5d01000640 1 Processor -- start 2024-03-06T07:30:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.723+0000 7f5d01000640 1 -- start start 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.724+0000 7f5d01000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5cfc15e2e0 0x7f5cfc0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.724+0000 7f5d01000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d0d70 0x7f5cfc0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.724+0000 7f5d01000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cfc0b3940 con 0x7f5cfc0d1b30 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.724+0000 7f5d01000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cfc0d4e70 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.724+0000 7f5d01000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0b3c40 con 0x7f5cfc0d0d70 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.724+0000 7f5cf9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5cfc15e2e0 0x7f5cfc0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cfa800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5cfc0d1b30 0x7f5cfc0d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55630/0 (socket says 172.21.15.32:55630) 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cfa800640 1 -- 172.21.15.32:0/2596164597 learned_addr learned my addr 172.21.15.32:0/2596164597 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cf1400640 1 --2- 172.21.15.32:0/2596164597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d0d70 0x7f5cfc0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cf9e00640 1 -- 172.21.15.32:0/2596164597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d0d70 msgr2=0x7f5cfc0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cf9e00640 1 --2- 172.21.15.32:0/2596164597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d0d70 0x7f5cfc0d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cf9e00640 1 -- 172.21.15.32:0/2596164597 >> v1:172.21.15.32:6789/0 conn(0x7f5cfc0d1b30 legacy=0x7f5cfc0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cf9e00640 1 -- 172.21.15.32:0/2596164597 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc0cf580 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.725+0000 7f5cf9400640 1 -- 172.21.15.32:0/2596164597 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728771399 0 0) 0x7f5cfc0b3940 con 0x7f5cfc0d1b30 2024-03-06T07:30:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.727+0000 7f5cf9e00640 1 --2- 172.21.15.32:0/2596164597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0d0830 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0097a0 tx=0x7f5cec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e5a89c0facf55e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.727+0000 7f5cf9400640 1 -- 172.21.15.32:0/2596164597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec035020 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.727+0000 7f5cf9400640 1 -- 172.21.15.32:0/2596164597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec031830 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.727+0000 7f5cf9400640 1 -- 172.21.15.32:0/2596164597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec031c00 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 -- 172.21.15.32:0/2596164597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 msgr2=0x7f5cfc0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 --2- 172.21.15.32:0/2596164597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0d0830 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0097a0 tx=0x7f5cec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 -- 172.21.15.32:0/2596164597 shutdown_connections 2024-03-06T07:30:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 --2- 172.21.15.32:0/2596164597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d0d70 0x7f5cfc0d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 --2- 172.21.15.32:0/2596164597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0d0830 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 -- 172.21.15.32:0/2596164597 >> 172.21.15.32:0/2596164597 conn(0x7f5cfc0d3f60 msgr2=0x7f5cfc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 -- 172.21.15.32:0/2596164597 shutdown_connections 2024-03-06T07:30:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.728+0000 7f5d01000640 1 -- 172.21.15.32:0/2596164597 wait complete. 2024-03-06T07:30:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.730+0000 7f5d01000640 1 Processor -- start 2024-03-06T07:30:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.730+0000 7f5d01000640 1 -- start start 2024-03-06T07:30:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5d01000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cfc0d0d70 0x7f5cfc0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5d01000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d1b30 0x7f5cfc0b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5d01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5d01000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0cfe20 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5d01000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0cfb40 con 0x7f5cfc0d0d70 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5d01000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0b50e0 con 0x7f5cfc0d1b30 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5cfa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5cfa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0b9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41558/0 (socket says 172.21.15.32:41558) 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.732+0000 7f5cfa800640 1 -- 172.21.15.32:0/497078840 learned_addr learned my addr 172.21.15.32:0/497078840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cf9e00640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cfc0d0d70 0x7f5cfc0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cf1400640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d1b30 0x7f5cfc0b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cfa800640 1 -- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d1b30 msgr2=0x7f5cfc0b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cfa800640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d1b30 0x7f5cfc0b8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cfa800640 1 -- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cfc0d0d70 msgr2=0x7f5cfc0b71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cfa800640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cfc0d0d70 0x7f5cfc0b71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cfa800640 1 -- 172.21.15.32:0/497078840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc15f280 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cf1400640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d1b30 0x7f5cfc0b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cfa800640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0b9390 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400c7a0 tx=0x7f5cf400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4018070 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf400a640 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.733+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cfc15f560 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.734+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf400aab0 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.734+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc15fb20 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.735+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc15fd20 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.737+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cf401b070 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.738+0000 7f5cf2a00640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cd806b4d0 0x7f5cd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.738+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cf4089780 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.739+0000 7f5cf9e00640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cd806b4d0 0x7f5cd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.739+0000 7f5cf9e00640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cd806b4d0 0x7f5cd806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5cec036000 tx=0x7f5cec03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.744+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cfc15fd20 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.947+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cfc04f3a0 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.948+0000 7f5cf2a00640 1 -- 172.21.15.32:0/497078840 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16561 (secure 0 0 0) 0x7f5cf4058690 con 0x7f5cfc15e2e0 2024-03-06T07:30:41.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:41.951 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:13.040051+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cd806b4d0 msgr2=0x7f5cd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cd806b4d0 0x7f5cd806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5cec036000 tx=0x7f5cec03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 msgr2=0x7f5cfc0b9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0b9390 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400c7a0 tx=0x7f5cf400cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 shutdown_connections 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cd806b4d0 0x7f5cd806d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cfc15e2e0 0x7f5cfc0b9390 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5cfc0d1b30 0x7f5cfc0b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 --2- 172.21.15.32:0/497078840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cfc0d0d70 0x7f5cfc0b71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.951+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 >> 172.21.15.32:0/497078840 conn(0x7f5cfc0d3f60 msgr2=0x7f5cfc0b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.952+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 shutdown_connections 2024-03-06T07:30:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:41.952+0000 7f5d01000640 1 -- 172.21.15.32:0/497078840 wait complete. 2024-03-06T07:30:41.982 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:30:41.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.196+0000 7fd08a000640 1 Processor -- start 2024-03-06T07:30:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.196+0000 7fd08a000640 1 -- start start 2024-03-06T07:30:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.199+0000 7fd08a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0840d2700 0x7fd0840c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.199+0000 7fd08a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd0840c4ff0 0x7fd0840af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.199+0000 7fd08a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840c5400 con 0x7fd084162410 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.199+0000 7fd08a000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0840cf450 con 0x7fd0840d2700 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.199+0000 7fd08a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0840cebc0 con 0x7fd0840c4ff0 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0840d2700 0x7fd0840c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0840d2700 0x7fd0840c4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41562/0 (socket says 172.21.15.32:41562) 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 -- 172.21.15.32:0/3149550427 learned_addr learned my addr 172.21.15.32:0/3149550427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd089600640 1 --2- 172.21.15.32:0/3149550427 >> v2:172.21.15.39:3300/0 conn(0x7fd0840c4ff0 0x7fd0840af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 -- 172.21.15.32:0/3149550427 >> v2:172.21.15.39:3300/0 conn(0x7fd0840c4ff0 msgr2=0x7fd0840af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 --2- 172.21.15.32:0/3149550427 >> v2:172.21.15.39:3300/0 conn(0x7fd0840c4ff0 0x7fd0840af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 -- 172.21.15.32:0/3149550427 >> v1:172.21.15.39:6789/0 conn(0x7fd084162410 legacy=0x7fd0840ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.200+0000 7fd083e00640 1 -- 172.21.15.32:0/3149550427 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0840cee90 con 0x7fd0840d2700 2024-03-06T07:30:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.202+0000 7fd083e00640 1 --2- 172.21.15.32:0/3149550427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0840d2700 0x7fd0840c4ab0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd08406e2a0 tx=0x7fd07002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d71d51e42f79cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.202+0000 7fd083400640 1 -- 172.21.15.32:0/3149550427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070035020 con 0x7fd0840d2700 2024-03-06T07:30:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.202+0000 7fd083400640 1 -- 172.21.15.32:0/3149550427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070031870 con 0x7fd0840d2700 2024-03-06T07:30:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.202+0000 7fd083400640 1 -- 172.21.15.32:0/3149550427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070031c40 con 0x7fd0840d2700 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 -- 172.21.15.32:0/3149550427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0840d2700 msgr2=0x7fd0840c4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 --2- 172.21.15.32:0/3149550427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0840d2700 0x7fd0840c4ab0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd08406e2a0 tx=0x7fd07002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 -- 172.21.15.32:0/3149550427 shutdown_connections 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 --2- 172.21.15.32:0/3149550427 >> v2:172.21.15.39:3300/0 conn(0x7fd0840c4ff0 0x7fd0840af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 --2- 172.21.15.32:0/3149550427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0840d2700 0x7fd0840c4ab0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 -- 172.21.15.32:0/3149550427 >> 172.21.15.32:0/3149550427 conn(0x7fd0840d1ae0 msgr2=0x7fd0840d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 -- 172.21.15.32:0/3149550427 shutdown_connections 2024-03-06T07:30:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.203+0000 7fd08a000640 1 -- 172.21.15.32:0/3149550427 wait complete. 2024-03-06T07:30:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.205+0000 7fd08a000640 1 Processor -- start 2024-03-06T07:30:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.205+0000 7fd08a000640 1 -- start start 2024-03-06T07:30:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd08a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0840c4ff0 0x7fd08415d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd08a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd0840d2700 0x7fd08409cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd08a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 0x7fd0840cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd08a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840aecc0 con 0x7fd084162410 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd08a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840ae9e0 con 0x7fd0840c4ff0 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd08a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840af720 con 0x7fd0840d2700 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd089600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 0x7fd0840cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd089600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 0x7fd0840cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41570/0 (socket says 172.21.15.32:41570) 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd089600640 1 -- 172.21.15.32:0/1779747799 learned_addr learned my addr 172.21.15.32:0/1779747799 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.206+0000 7fd088c00640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0840c4ff0 0x7fd08415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd083e00640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd0840d2700 0x7fd08409cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd089600640 1 -- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd0840d2700 msgr2=0x7fd08409cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd089600640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd0840d2700 0x7fd08409cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd089600640 1 -- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0840c4ff0 msgr2=0x7fd08415d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd089600640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0840c4ff0 0x7fd08415d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd089600640 1 -- 172.21.15.32:0/1779747799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0840c5760 con 0x7fd084162410 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd088c00640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0840c4ff0 0x7fd08415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd089600640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 0x7fd0840cacf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd07c007e40 tx=0x7fd07c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c014620 con 0x7fd084162410 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c019070 con 0x7fd084162410 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.207+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c014a20 con 0x7fd084162410 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.208+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0840c5a40 con 0x7fd084162410 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.208+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08409d250 con 0x7fd084162410 2024-03-06T07:30:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.209+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08404f330 con 0x7fd084162410 2024-03-06T07:30:42.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.210+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd07c014bc0 con 0x7fd084162410 2024-03-06T07:30:42.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.211+0000 7fd080c00640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06406b4d0 0x7fd06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.211+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd07c08ab50 con 0x7fd084162410 2024-03-06T07:30:42.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.211+0000 7fd088c00640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06406b4d0 0x7fd06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.212+0000 7fd088c00640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06406b4d0 0x7fd06406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd078007fd0 tx=0x7fd078009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.215+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd07c0599c0 con 0x7fd084162410 2024-03-06T07:30:42.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.591+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0840d0b30 con 0x7fd084162410 2024-03-06T07:30:42.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.592+0000 7fd080c00640 1 -- 172.21.15.32:0/1779747799 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fd07c0592c0 con 0x7fd084162410 2024-03-06T07:30:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:42.594 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:42.594 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06406b4d0 msgr2=0x7fd06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06406b4d0 0x7fd06406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd078007fd0 tx=0x7fd078009040 comp rx=0 tx=0).stop 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 msgr2=0x7fd0840cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 0x7fd0840cacf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd07c007e40 tx=0x7fd07c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 shutdown_connections 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06406b4d0 0x7fd06406d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd084162410 0x7fd0840cacf0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd0840d2700 0x7fd08409cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 --2- 172.21.15.32:0/1779747799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0840c4ff0 0x7fd08415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.594+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 >> 172.21.15.32:0/1779747799 conn(0x7fd0840d1ae0 msgr2=0x7fd0841612b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.595+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 shutdown_connections 2024-03-06T07:30:42.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.595+0000 7fd08a000640 1 -- 172.21.15.32:0/1779747799 wait complete. 2024-03-06T07:30:42.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.854+0000 7fa70fe00640 1 Processor -- start 2024-03-06T07:30:42.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.854+0000 7fa70fe00640 1 -- start start 2024-03-06T07:30:42.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71015e070 0x7fa7100d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100d18c0 con 0x7fa7100d0cd0 2024-03-06T07:30:42.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100d1ba0 con 0x7fa7100cf410 2024-03-06T07:30:42.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70fe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7100d1e80 con 0x7fa71015e070 2024-03-06T07:30:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71015e070 0x7fa7100d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7100d0cd0 0x7fa7100b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55644/0 (socket says 172.21.15.32:55644) 2024-03-06T07:30:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.856+0000 7fa70f400640 1 -- 172.21.15.32:0/3759575202 learned_addr learned my addr 172.21.15.32:0/3759575202 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.857+0000 7fa70e000640 1 -- 172.21.15.32:0/3759575202 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443170191 0 0) 0x7fa7100d18c0 con 0x7fa7100d0cd0 2024-03-06T07:30:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.857+0000 7fa70ea00640 1 -- 172.21.15.32:0/3759575202 >> v1:172.21.15.39:6789/0 conn(0x7fa7100cf410 legacy=0x7fa7100d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.857+0000 7fa70ea00640 1 -- 172.21.15.32:0/3759575202 >> v1:172.21.15.32:6789/0 conn(0x7fa7100d0cd0 legacy=0x7fa7100b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.857+0000 7fa70ea00640 1 -- 172.21.15.32:0/3759575202 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100d2160 con 0x7fa71015e070 2024-03-06T07:30:42.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.858+0000 7fa70ea00640 1 --2- 172.21.15.32:0/3759575202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100d2960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa7000097a0 tx=0x7fa70002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98a7857bd9f4112 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.858+0000 7fa70e000640 1 -- 172.21.15.32:0/3759575202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700034070 con 0x7fa71015e070 2024-03-06T07:30:42.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.858+0000 7fa70e000640 1 -- 172.21.15.32:0/3759575202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa700030610 con 0x7fa71015e070 2024-03-06T07:30:42.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.858+0000 7fa70e000640 1 -- 172.21.15.32:0/3759575202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000309e0 con 0x7fa71015e070 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 -- 172.21.15.32:0/3759575202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 msgr2=0x7fa7100d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 --2- 172.21.15.32:0/3759575202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100d2960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa7000097a0 tx=0x7fa70002b960 comp rx=0 tx=0).stop 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 -- 172.21.15.32:0/3759575202 shutdown_connections 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 --2- 172.21.15.32:0/3759575202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100d2960 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 -- 172.21.15.32:0/3759575202 >> 172.21.15.32:0/3759575202 conn(0x7fa71008d2a0 msgr2=0x7fa7100d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 -- 172.21.15.32:0/3759575202 shutdown_connections 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.859+0000 7fa70fe00640 1 -- 172.21.15.32:0/3759575202 wait complete. 2024-03-06T07:30:42.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.861+0000 7fa70fe00640 1 Processor -- start 2024-03-06T07:30:42.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.861+0000 7fa70fe00640 1 -- start start 2024-03-06T07:30:42.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7100cf410 0x7fa7101585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7100d0cd0 0x7fa7100c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100d4730 con 0x7fa71015e070 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100d45b0 con 0x7fa7100d0cd0 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100cf870 con 0x7fa7100cf410 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100b4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41580/0 (socket says 172.21.15.32:41580) 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.862+0000 7fa70f400640 1 -- 172.21.15.32:0/175597821 learned_addr learned my addr 172.21.15.32:0/175597821 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:42.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70ea00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7100cf410 0x7fa7101585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa707e00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7100d0cd0 0x7fa7100c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70f400640 1 -- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7100cf410 msgr2=0x7fa7101585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70f400640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7100cf410 0x7fa7101585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70f400640 1 -- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7100d0cd0 msgr2=0x7fa7100c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70f400640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7100d0cd0 0x7fa7100c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70f400640 1 -- 172.21.15.32:0/175597821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa710158b30 con 0x7fa71015e070 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70f400640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100b4490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa70800c770 tx=0x7fa70800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708018070 con 0x7fa71015e070 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70800a640 con 0x7fa71015e070 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7100b8980 con 0x7fa71015e070 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.863+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70800aa30 con 0x7fa71015e070 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.864+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7100b7bc0 con 0x7fa71015e070 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.865+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7100b3bc0 con 0x7fa71015e070 2024-03-06T07:30:42.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.866+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa70801b070 con 0x7fa71015e070 2024-03-06T07:30:42.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.866+0000 7fa707400640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.867+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7080897a0 con 0x7fa71015e070 2024-03-06T07:30:42.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.867+0000 7fa70ea00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.868+0000 7fa70ea00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa700036000 tx=0x7fa70003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:42.872+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7100b3bc0 con 0x7fa71015e070 2024-03-06T07:30:43.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.119+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7100d1370 con 0x7fa71015e070 2024-03-06T07:30:43.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.121+0000 7fa707400640 1 -- 172.21.15.32:0/175597821 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fa7080586b0 con 0x7fa71015e070 2024-03-06T07:30:43.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:43.123 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:11.509859+0000","modified":"2024-03-06T07:30:14.594308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5058":{"gid":5058,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/2686928389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2686928389},{"type":"v1","addr":"172.21.15.32:6837","nonce":2686928389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5058,"qdb_cluster":[5058]},"id":2}]} 2024-03-06T07:30:43.123 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-03-06T07:30:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 msgr2=0x7fa6ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa700036000 tx=0x7fa70003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 msgr2=0x7fa7100b4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100b4490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa70800c770 tx=0x7fa70800cc30 comp rx=0 tx=0).stop 2024-03-06T07:30:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 shutdown_connections 2024-03-06T07:30:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015e070 0x7fa7100b4490 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7100d0cd0 0x7fa7100c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 --2- 172.21.15.32:0/175597821 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7100cf410 0x7fa7101585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 >> 172.21.15.32:0/175597821 conn(0x7fa71008d2a0 msgr2=0x7fa7100b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 shutdown_connections 2024-03-06T07:30:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.124+0000 7fa70fe00640 1 -- 172.21.15.32:0/175597821 wait complete. 2024-03-06T07:30:43.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.367+0000 7f63daa00640 1 Processor -- start 2024-03-06T07:30:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.367+0000 7f63daa00640 1 -- start start 2024-03-06T07:30:43.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63daa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63d406c8b0 0x7f63d406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63daa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63d40ce370 0x7f63d40ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63daa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40cf1b0 con 0x7f63d4066d00 2024-03-06T07:30:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63daa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d415e470 con 0x7f63d406c8b0 2024-03-06T07:30:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63daa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d415ea00 con 0x7f63d40ce370 2024-03-06T07:30:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63d9600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63d406c8b0 0x7f63d406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63d8c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f63d4066d00 0x7f63d415ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55648/0 (socket says 172.21.15.32:55648) 2024-03-06T07:30:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.369+0000 7f63d8c00640 1 -- 172.21.15.32:0/1390806136 learned_addr learned my addr 172.21.15.32:0/1390806136 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63da000640 1 --2- 172.21.15.32:0/1390806136 >> v2:172.21.15.39:3300/0 conn(0x7f63d40ce370 0x7f63d40ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63d3e00640 1 -- 172.21.15.32:0/1390806136 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364322602 0 0) 0x7f63d40cf1b0 con 0x7f63d4066d00 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63d3e00640 1 -- 172.21.15.32:0/1390806136 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc003520 con 0x7f63d4066d00 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63d9600640 1 -- 172.21.15.32:0/1390806136 >> v2:172.21.15.39:3300/0 conn(0x7f63d40ce370 msgr2=0x7f63d40ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63d9600640 1 --2- 172.21.15.32:0/1390806136 >> v2:172.21.15.39:3300/0 conn(0x7f63d40ce370 0x7f63d40ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63d9600640 1 -- 172.21.15.32:0/1390806136 >> v1:172.21.15.32:6789/0 conn(0x7f63d4066d00 legacy=0x7f63d415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.370+0000 7f63d9600640 1 -- 172.21.15.32:0/1390806136 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d406d420 con 0x7f63d406c8b0 2024-03-06T07:30:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.371+0000 7f63d9600640 1 --2- 172.21.15.32:0/1390806136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d406c8b0 0x7f63d406ef10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63d4071370 tx=0x7f63c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5aaf82e409f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.371+0000 7f63d3e00640 1 -- 172.21.15.32:0/1390806136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8035020 con 0x7f63d406c8b0 2024-03-06T07:30:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63d3e00640 1 -- 172.21.15.32:0/1390806136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8031830 con 0x7f63d406c8b0 2024-03-06T07:30:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63d3e00640 1 -- 172.21.15.32:0/1390806136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8031be0 con 0x7f63d406c8b0 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 -- 172.21.15.32:0/1390806136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d406c8b0 msgr2=0x7f63d406ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 --2- 172.21.15.32:0/1390806136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d406c8b0 0x7f63d406ef10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63d4071370 tx=0x7f63c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 -- 172.21.15.32:0/1390806136 shutdown_connections 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 --2- 172.21.15.32:0/1390806136 >> v2:172.21.15.39:3300/0 conn(0x7f63d40ce370 0x7f63d40ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 --2- 172.21.15.32:0/1390806136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d406c8b0 0x7f63d406ef10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 -- 172.21.15.32:0/1390806136 >> 172.21.15.32:0/1390806136 conn(0x7f63d40ccf70 msgr2=0x7f63d40cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 -- 172.21.15.32:0/1390806136 shutdown_connections 2024-03-06T07:30:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.372+0000 7f63daa00640 1 -- 172.21.15.32:0/1390806136 wait complete. 2024-03-06T07:30:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.374+0000 7f63daa00640 1 Processor -- start 2024-03-06T07:30:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.374+0000 7f63daa00640 1 -- start start 2024-03-06T07:30:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.375+0000 7f63daa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 0x7f63d4151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.375+0000 7f63daa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f63d406c8b0 0x7f63d41e98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.375+0000 7f63daa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63d40ce370 0x7f63d4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.375+0000 7f63daa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40cee60 con 0x7f63d4066d00 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.375+0000 7f63daa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40cece0 con 0x7f63d40ce370 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.375+0000 7f63daa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d406afb0 con 0x7f63d406c8b0 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 0x7f63d4151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 0x7f63d4151de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41598/0 (socket says 172.21.15.32:41598) 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 -- 172.21.15.32:0/2654919507 learned_addr learned my addr 172.21.15.32:0/2654919507 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63da000640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63d40ce370 0x7f63d4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d8c00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f63d406c8b0 0x7f63d41e98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 -- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f63d406c8b0 msgr2=0x7f63d41e98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f63d406c8b0 0x7f63d41e98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 -- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63d40ce370 msgr2=0x7f63d4156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63d40ce370 0x7f63d4156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.376+0000 7f63d9600640 1 -- 172.21.15.32:0/2654919507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d41e8b10 con 0x7f63d4066d00 2024-03-06T07:30:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.377+0000 7f63d9600640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 0x7f63d4151de0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009890 tx=0x7f63c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.377+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8041070 con 0x7f63d4066d00 2024-03-06T07:30:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.377+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c803f070 con 0x7f63d4066d00 2024-03-06T07:30:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.377+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8035040 con 0x7f63d4066d00 2024-03-06T07:30:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.377+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d41e8df0 con 0x7f63d4066d00 2024-03-06T07:30:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.377+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d4157410 con 0x7f63d4066d00 2024-03-06T07:30:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.378+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d41567a0 con 0x7f63d4066d00 2024-03-06T07:30:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.380+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63c8038020 con 0x7f63d4066d00 2024-03-06T07:30:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.380+0000 7f63d1600640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f63b406b4d0 0x7f63b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.381+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c80abbc0 con 0x7f63d4066d00 2024-03-06T07:30:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.381+0000 7f63d8c00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f63b406b4d0 0x7f63b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.381+0000 7f63d8c00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f63b406b4d0 0x7f63b406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63d41581e0 tx=0x7f63c0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.387+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63c8031800 con 0x7f63d4066d00 2024-03-06T07:30:43.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.587+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63d404f330 con 0x7f63d4066d00 2024-03-06T07:30:43.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.589+0000 7f63d1600640 1 -- 172.21.15.32:0/2654919507 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16561 (secure 0 0 0) 0x7f63c807aa30 con 0x7f63d4066d00 2024-03-06T07:30:43.592 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:43.592 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:13.040051+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:09.502397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:10.547731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f63b406b4d0 msgr2=0x7f63b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f63b406b4d0 0x7f63b406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63d41581e0 tx=0x7f63c0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 msgr2=0x7f63d4151de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 0x7f63d4151de0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009890 tx=0x7f63c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 shutdown_connections 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f63b406b4d0 0x7f63b406d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63d40ce370 0x7f63d4156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f63d406c8b0 0x7f63d41e98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 --2- 172.21.15.32:0/2654919507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63d4066d00 0x7f63d4151de0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.592+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 >> 172.21.15.32:0/2654919507 conn(0x7f63d40ccf70 msgr2=0x7f63d4161f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.593+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 shutdown_connections 2024-03-06T07:30:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.593+0000 7f63daa00640 1 -- 172.21.15.32:0/2654919507 wait complete. 2024-03-06T07:30:43.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:30:43.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.837+0000 7ff8bca00640 1 Processor -- start 2024-03-06T07:30:43.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.838+0000 7ff8bca00640 1 -- start start 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.839+0000 7ff8bca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff8b8162420 0x7ff8b806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.839+0000 7ff8bca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b80d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.839+0000 7ff8bca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b8147bc0 con 0x7ff8b81477e0 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.839+0000 7ff8bca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b806b960 con 0x7ff8b8162420 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.839+0000 7ff8bca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b8161f90 con 0x7ff8b806b4f0 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.840+0000 7ff8b7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff8b81477e0 0x7ff8b806ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55656/0 (socket says 172.21.15.32:55656) 2024-03-06T07:30:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.840+0000 7ff8b7400640 1 -- 172.21.15.32:0/462632573 learned_addr learned my addr 172.21.15.32:0/462632573 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.840+0000 7ff8b7e00640 1 --2- 172.21.15.32:0/462632573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b80d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.840+0000 7ff8aea00640 1 --2- 172.21.15.32:0/462632573 >> v2:172.21.15.39:3300/0 conn(0x7ff8b8162420 0x7ff8b806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.840+0000 7ff8b6a00640 1 -- 172.21.15.32:0/462632573 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515238564 0 0) 0x7ff8b8147bc0 con 0x7ff8b81477e0 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.840+0000 7ff8b6a00640 1 -- 172.21.15.32:0/462632573 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c003520 con 0x7ff8b81477e0 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.841+0000 7ff8b6a00640 1 -- 172.21.15.32:0/462632573 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761227242 0 0) 0x7ff89c003520 con 0x7ff8b81477e0 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.841+0000 7ff8b7e00640 1 -- 172.21.15.32:0/462632573 >> v2:172.21.15.39:3300/0 conn(0x7ff8b8162420 msgr2=0x7ff8b806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.841+0000 7ff8b7e00640 1 --2- 172.21.15.32:0/462632573 >> v2:172.21.15.39:3300/0 conn(0x7ff8b8162420 0x7ff8b806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.841+0000 7ff8b7e00640 1 -- 172.21.15.32:0/462632573 >> v1:172.21.15.32:6789/0 conn(0x7ff8b81477e0 legacy=0x7ff8b806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.841+0000 7ff8b7e00640 1 -- 172.21.15.32:0/462632573 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80b3d70 con 0x7ff8b806b4f0 2024-03-06T07:30:43.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.842+0000 7ff8b7e00640 1 --2- 172.21.15.32:0/462632573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b80d0910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00097a0 tx=0x7ff8b002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e81cd639dbddc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8b6a00640 1 -- 172.21.15.32:0/462632573 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0037070 con 0x7ff8b806b4f0 2024-03-06T07:30:43.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8b6a00640 1 -- 172.21.15.32:0/462632573 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b0030510 con 0x7ff8b806b4f0 2024-03-06T07:30:43.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8b6a00640 1 -- 172.21.15.32:0/462632573 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80019f0 con 0x7ff8b806b4f0 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 -- 172.21.15.32:0/462632573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 msgr2=0x7ff8b80d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 --2- 172.21.15.32:0/462632573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b80d0910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00097a0 tx=0x7ff8b002b710 comp rx=0 tx=0).stop 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 -- 172.21.15.32:0/462632573 shutdown_connections 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 --2- 172.21.15.32:0/462632573 >> v2:172.21.15.39:3300/0 conn(0x7ff8b8162420 0x7ff8b806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 --2- 172.21.15.32:0/462632573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b80d0910 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 -- 172.21.15.32:0/462632573 >> 172.21.15.32:0/462632573 conn(0x7ff8b806c0c0 msgr2=0x7ff8b806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.843+0000 7ff8bca00640 1 -- 172.21.15.32:0/462632573 shutdown_connections 2024-03-06T07:30:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.844+0000 7ff8bca00640 1 -- 172.21.15.32:0/462632573 wait complete. 2024-03-06T07:30:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.845+0000 7ff8bca00640 1 Processor -- start 2024-03-06T07:30:43.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.845+0000 7ff8bca00640 1 -- start start 2024-03-06T07:30:43.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.846+0000 7ff8bca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b81340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.846+0000 7ff8bca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b81477e0 0x7ff8b80cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.846+0000 7ff8bca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 0x7ff8b8132060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.846+0000 7ff8bca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b8161c20 con 0x7ff8b8162420 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.846+0000 7ff8bca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b80b4330 con 0x7ff8b81477e0 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.846+0000 7ff8bca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b81628a0 con 0x7ff8b806b4f0 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 0x7ff8b8132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 0x7ff8b8132060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41614/0 (socket says 172.21.15.32:41614) 2024-03-06T07:30:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 -- 172.21.15.32:0/1243681512 learned_addr learned my addr 172.21.15.32:0/1243681512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7400640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8aea00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b81477e0 0x7ff8b80cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 -- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 msgr2=0x7ff8b81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b81340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 -- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b81477e0 msgr2=0x7ff8b80cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b81477e0 0x7ff8b80cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8b7e00640 1 -- 172.21.15.32:0/1243681512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80cdba0 con 0x7ff8b8162420 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.847+0000 7ff8aea00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b81477e0 0x7ff8b80cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.848+0000 7ff8b7e00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 0x7ff8b8132060 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00098c0 tx=0x7ff8b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.848+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0041020 con 0x7ff8b8162420 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.848+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b003c870 con 0x7ff8b8162420 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.848+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0037040 con 0x7ff8b8162420 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.848+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b81346b0 con 0x7ff8b8162420 2024-03-06T07:30:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.848+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b8134c70 con 0x7ff8b8162420 2024-03-06T07:30:43.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.850+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b804f330 con 0x7ff8b8162420 2024-03-06T07:30:43.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.850+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8b0044020 con 0x7ff8b8162420 2024-03-06T07:30:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.851+0000 7ff8afe00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89406b4d0 0x7ff89406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.851+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8b00abb00 con 0x7ff8b8162420 2024-03-06T07:30:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.851+0000 7ff8b7400640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89406b4d0 0x7ff89406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.852+0000 7ff8b7400640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89406b4d0 0x7ff89406d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8142630 tx=0x7ff8a8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:43.856+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8b004b030 con 0x7ff8b8162420 2024-03-06T07:30:44.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.127+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff8b80724b0 con 0x7ff8b8162420 2024-03-06T07:30:44.130 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:30:44.128+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:30:44.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.131+0000 7ff8afe00640 1 -- 172.21.15.32:0/1243681512 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff8b007aa10 con 0x7ff8b8162420 2024-03-06T07:30:44.133 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:30:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89406b4d0 msgr2=0x7ff89406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89406b4d0 0x7ff89406d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8142630 tx=0x7ff8a8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 msgr2=0x7ff8b8132060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 0x7ff8b8132060 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00098c0 tx=0x7ff8b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 shutdown_connections 2024-03-06T07:30:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89406b4d0 0x7ff89406d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8162420 0x7ff8b8132060 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b81477e0 0x7ff8b80cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 --2- 172.21.15.32:0/1243681512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b806b4f0 0x7ff8b81340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.134+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 >> 172.21.15.32:0/1243681512 conn(0x7ff8b806c0c0 msgr2=0x7ff8b806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.136+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 shutdown_connections 2024-03-06T07:30:44.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.137+0000 7ff8bca00640 1 -- 172.21.15.32:0/1243681512 wait complete. 2024-03-06T07:30:44.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:30:44.231 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:30:44.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.351+0000 7f3dcb600640 1 Processor -- start 2024-03-06T07:30:44.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.351+0000 7f3dcb600640 1 -- start start 2024-03-06T07:30:44.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.353+0000 7f3dcb600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3dc415afa0 0x7f3dc40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.353+0000 7f3dcb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc40cd1e0 0x7f3dc40b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.353+0000 7f3dcb600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc4071c60 con 0x7f3dc4070ce0 2024-03-06T07:30:44.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.353+0000 7f3dcb600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc40cfa10 con 0x7f3dc415afa0 2024-03-06T07:30:44.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.353+0000 7f3dcb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4071f40 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.353+0000 7f3dc8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3dc415afa0 0x7f3dc40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3dc415afa0 0x7f3dc40cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:35700/0 (socket says 172.21.15.32:35700) 2024-03-06T07:30:44.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc8c00640 1 -- 172.21.15.32:0/1593022489 learned_addr learned my addr 172.21.15.32:0/1593022489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:44.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc3e00640 1 --1- 172.21.15.32:0/1593022489 >> v1:172.21.15.39:6789/0 conn(0x7f3dc4070ce0 0x7f3dc40ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:43166/0 (socket says 172.21.15.32:43166) 2024-03-06T07:30:44.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc9600640 1 --2- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc40cd1e0 0x7f3dc40b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc8c00640 1 -- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc40cd1e0 msgr2=0x7f3dc40b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc8c00640 1 --2- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc40cd1e0 0x7f3dc40b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc8c00640 1 -- 172.21.15.32:0/1593022489 >> v1:172.21.15.39:6789/0 conn(0x7f3dc4070ce0 legacy=0x7f3dc40ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.354+0000 7f3dc8c00640 1 -- 172.21.15.32:0/1593022489 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc4072220 con 0x7f3dc415afa0 2024-03-06T07:30:44.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.356+0000 7f3dc8c00640 1 --2- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc40cf3a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5945b278f87ae425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dc3400640 1 -- 172.21.15.32:0/1593022489 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8035020 con 0x7f3dc415afa0 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dc3400640 1 -- 172.21.15.32:0/1593022489 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8031640 con 0x7f3dc415afa0 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dc3400640 1 -- 172.21.15.32:0/1593022489 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db802d040 con 0x7f3dc415afa0 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 -- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 msgr2=0x7f3dc40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 --2- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc40cf3a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 -- 172.21.15.32:0/1593022489 shutdown_connections 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 --2- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc40cd1e0 0x7f3dc40b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 --2- 172.21.15.32:0/1593022489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc40cf3a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 -- 172.21.15.32:0/1593022489 >> 172.21.15.32:0/1593022489 conn(0x7f3dc406b3d0 msgr2=0x7f3dc406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.358+0000 7f3dcb600640 1 -- 172.21.15.32:0/1593022489 shutdown_connections 2024-03-06T07:30:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.359+0000 7f3dcb600640 1 -- 172.21.15.32:0/1593022489 wait complete. 2024-03-06T07:30:44.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.361+0000 7f3dcb600640 1 Processor -- start 2024-03-06T07:30:44.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.361+0000 7f3dcb600640 1 -- start start 2024-03-06T07:30:44.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dcb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc4070ce0 0x7f3dc409d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dcb600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 0x7f3dc41f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dcb600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc4153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dcb600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40b4000 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dcb600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4071170 con 0x7f3dc415afa0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dcb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40b4180 con 0x7f3dc4070ce0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dc3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 0x7f3dc41f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dc3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 0x7f3dc41f7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41640/0 (socket says 172.21.15.32:41640) 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.362+0000 7f3dc3e00640 1 -- 172.21.15.32:0/663944939 learned_addr learned my addr 172.21.15.32:0/663944939 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc8c00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc4070ce0 0x7f3dc409d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc9600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc4153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc3e00640 1 -- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc4070ce0 msgr2=0x7f3dc409d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc3e00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc4070ce0 0x7f3dc409d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc3e00640 1 -- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 msgr2=0x7f3dc4153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc3e00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc4153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc3e00640 1 -- 172.21.15.32:0/663944939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc41f7f30 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc9600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc4153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc3e00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 0x7f3dc41f7570 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3db000c430 tx=0x7f3db000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db00074e0 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db0007680 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0004a00 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.363+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc41f8210 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.364+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc409d5c0 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.365+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc404f330 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.366+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3db001a020 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.367+0000 7f3dc0c00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da406b4d0 0x7f3da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.367+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3db0089a80 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.367+0000 7f3dc8c00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da406b4d0 0x7f3da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.368+0000 7f3dc8c00640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da406b4d0 0x7f3da406d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3db8036000 tx=0x7f3db803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.371+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3db008be20 con 0x7f3dc40cd1e0 2024-03-06T07:30:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:44.603+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3dc406bc30 con 0x7f3dc40cd1e0 2024-03-06T07:30:45.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.142+0000 7f3dc0c00640 1 -- 172.21.15.32:0/663944939 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3db0058890 con 0x7f3dc40cd1e0 2024-03-06T07:30:45.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.145+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da406b4d0 msgr2=0x7f3da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da406b4d0 0x7f3da406d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3db8036000 tx=0x7f3db803c040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 msgr2=0x7f3dc41f7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 0x7f3dc41f7570 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3db000c430 tx=0x7f3db000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 shutdown_connections 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da406b4d0 0x7f3da406d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dc415afa0 0x7f3dc4153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dc40cd1e0 0x7f3dc41f7570 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 --2- 172.21.15.32:0/663944939 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dc4070ce0 0x7f3dc409d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 >> 172.21.15.32:0/663944939 conn(0x7f3dc406b3d0 msgr2=0x7f3dc40b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 shutdown_connections 2024-03-06T07:30:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.146+0000 7f3dcb600640 1 -- 172.21.15.32:0/663944939 wait complete. 2024-03-06T07:30:45.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:45.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.377+0000 7fc94a000640 1 Processor -- start 2024-03-06T07:30:45.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.377+0000 7fc94a000640 1 -- start start 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.378+0000 7fc94a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440ae7b0 0x7fc9440aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.378+0000 7fc94a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc94415cdc0 0x7fc94406cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.378+0000 7fc94a000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9441531f0 con 0x7fc9440b3640 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.378+0000 7fc94a000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9440d04d0 con 0x7fc94415cdc0 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.378+0000 7fc94a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440b2b80 con 0x7fc9440ae7b0 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc94415cdc0 0x7fc94406cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc94415cdc0 0x7fc94406cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41644/0 (socket says 172.21.15.32:41644) 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 -- 172.21.15.32:0/2137832410 learned_addr learned my addr 172.21.15.32:0/2137832410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943400640 1 --2- 172.21.15.32:0/2137832410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440ae7b0 0x7fc9440aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 -- 172.21.15.32:0/2137832410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440ae7b0 msgr2=0x7fc9440aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 --2- 172.21.15.32:0/2137832410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440ae7b0 0x7fc9440aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 -- 172.21.15.32:0/2137832410 >> v1:172.21.15.32:6789/0 conn(0x7fc9440b3640 legacy=0x7fc9440aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.379+0000 7fc943e00640 1 -- 172.21.15.32:0/2137832410 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94406ebd0 con 0x7fc94415cdc0 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.380+0000 7fc943e00640 1 --2- 172.21.15.32:0/2137832410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc94415cdc0 0x7fc94406cc70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009e80 tx=0x7fc93c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56499baa9c24ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc942000640 1 -- 172.21.15.32:0/2137832410 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c02cae0 con 0x7fc94415cdc0 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc942000640 1 -- 172.21.15.32:0/2137832410 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c02cc80 con 0x7fc94415cdc0 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc942000640 1 -- 172.21.15.32:0/2137832410 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c034610 con 0x7fc94415cdc0 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 -- 172.21.15.32:0/2137832410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc94415cdc0 msgr2=0x7fc94406cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 --2- 172.21.15.32:0/2137832410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc94415cdc0 0x7fc94406cc70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009e80 tx=0x7fc93c02bf40 comp rx=0 tx=0).stop 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 -- 172.21.15.32:0/2137832410 shutdown_connections 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 --2- 172.21.15.32:0/2137832410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc94415cdc0 0x7fc94406cc70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 --2- 172.21.15.32:0/2137832410 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440ae7b0 0x7fc9440aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 -- 172.21.15.32:0/2137832410 >> 172.21.15.32:0/2137832410 conn(0x7fc94406afb0 msgr2=0x7fc94406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.381+0000 7fc94a000640 1 -- 172.21.15.32:0/2137832410 shutdown_connections 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.382+0000 7fc94a000640 1 -- 172.21.15.32:0/2137832410 wait complete. 2024-03-06T07:30:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.383+0000 7fc94a000640 1 Processor -- start 2024-03-06T07:30:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.383+0000 7fc94a000640 1 -- start start 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.384+0000 7fc94a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 0x7fc94409d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.384+0000 7fc94a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440b3640 0x7fc944155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.384+0000 7fc94a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc94415cdc0 0x7fc9441f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.384+0000 7fc94a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440b3120 con 0x7fc9440ae7b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.384+0000 7fc94a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440b2e40 con 0x7fc94415cdc0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.384+0000 7fc94a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94415d250 con 0x7fc9440b3640 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 0x7fc94409d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 0x7fc94409d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41646/0 (socket says 172.21.15.32:41646) 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 -- 172.21.15.32:0/1995648961 learned_addr learned my addr 172.21.15.32:0/1995648961 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc942a00640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440b3640 0x7fc944155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 -- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440b3640 msgr2=0x7fc944155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440b3640 0x7fc944155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 -- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc94415cdc0 msgr2=0x7fc9441f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc94415cdc0 0x7fc9441f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.385+0000 7fc943400640 1 -- 172.21.15.32:0/1995648961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944155a80 con 0x7fc9440ae7b0 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.386+0000 7fc943400640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 0x7fc94409d430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc93400d800 tx=0x7fc93400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.387+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93400b6c0 con 0x7fc9440ae7b0 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.387+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc934019070 con 0x7fc9440ae7b0 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.387+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93400ba80 con 0x7fc9440ae7b0 2024-03-06T07:30:45.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.387+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc944155d60 con 0x7fc9440ae7b0 2024-03-06T07:30:45.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.387+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94409ab40 con 0x7fc9440ae7b0 2024-03-06T07:30:45.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.388+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc944068010 con 0x7fc9440ae7b0 2024-03-06T07:30:45.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.389+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc93400bc20 con 0x7fc9440ae7b0 2024-03-06T07:30:45.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.390+0000 7fc93b400640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc92406b520 0x7fc92406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.390+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc93408a660 con 0x7fc9440ae7b0 2024-03-06T07:30:45.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.390+0000 7fc942a00640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc92406b520 0x7fc92406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.391+0000 7fc942a00640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc92406b520 0x7fc92406d9e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc94409da10 tx=0x7fc930009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.395+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9340594f0 con 0x7fc9440ae7b0 2024-03-06T07:30:45.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:45.597+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc9440b4670 con 0x7fc9440ae7b0 2024-03-06T07:30:46.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.411+0000 7fc93b400640 1 -- 172.21.15.32:0/1995648961 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc934058df0 con 0x7fc9440ae7b0 2024-03-06T07:30:46.413 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.414+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc92406b520 msgr2=0x7fc92406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.414+0000 7fc94a000640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc92406b520 0x7fc92406d9e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc94409da10 tx=0x7fc930009040 comp rx=0 tx=0).stop 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 msgr2=0x7fc94409d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 0x7fc94409d430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc93400d800 tx=0x7fc93400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 shutdown_connections 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc92406b520 0x7fc92406d9e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc94415cdc0 0x7fc9441f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9440b3640 0x7fc944155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 --2- 172.21.15.32:0/1995648961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440ae7b0 0x7fc94409d430 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 >> 172.21.15.32:0/1995648961 conn(0x7fc94406afb0 msgr2=0x7fc9440b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 shutdown_connections 2024-03-06T07:30:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.415+0000 7fc94a000640 1 -- 172.21.15.32:0/1995648961 wait complete. 2024-03-06T07:30:46.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.649+0000 7fbd9e400640 1 Processor -- start 2024-03-06T07:30:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.649+0000 7fbd9e400640 1 -- start start 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.651+0000 7fbd9e400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd98145570 0x7fbd980cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.651+0000 7fbd9e400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbd98071450 0x7fbd9806daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.651+0000 7fbd9e400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd98071c50 con 0x7fbd9806e030 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.651+0000 7fbd9e400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9806e4a0 con 0x7fbd98145570 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.651+0000 7fbd9e400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd98071f30 con 0x7fbd98071450 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd98145570 0x7fbd980cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd98145570 0x7fbd980cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41650/0 (socket says 172.21.15.32:41650) 2024-03-06T07:30:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 -- 172.21.15.32:0/2373744469 learned_addr learned my addr 172.21.15.32:0/2373744469 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd97e00640 1 --2- 172.21.15.32:0/2373744469 >> v2:172.21.15.39:3300/0 conn(0x7fbd98071450 0x7fbd9806daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 -- 172.21.15.32:0/2373744469 >> v2:172.21.15.39:3300/0 conn(0x7fbd98071450 msgr2=0x7fbd9806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 --2- 172.21.15.32:0/2373744469 >> v2:172.21.15.39:3300/0 conn(0x7fbd98071450 0x7fbd9806daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 -- 172.21.15.32:0/2373744469 >> v1:172.21.15.32:6789/0 conn(0x7fbd9806e030 legacy=0x7fbd980ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd9d000640 1 -- 172.21.15.32:0/2373744469 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd98072210 con 0x7fbd98145570 2024-03-06T07:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.652+0000 7fbd97400640 1 -- 172.21.15.32:0/2373744469 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865749399 0 0) 0x7fbd98071c50 con 0x7fbd9806e030 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.653+0000 7fbd9d000640 1 --2- 172.21.15.32:0/2373744469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98145570 0x7fbd980cf420 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbd9806c3b0 tx=0x7fbd8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64c6a1aec65d8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd97400640 1 -- 172.21.15.32:0/2373744469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c035020 con 0x7fbd98145570 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd97400640 1 -- 172.21.15.32:0/2373744469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c031870 con 0x7fbd98145570 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd97400640 1 -- 172.21.15.32:0/2373744469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c031c40 con 0x7fbd98145570 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 -- 172.21.15.32:0/2373744469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98145570 msgr2=0x7fbd980cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 --2- 172.21.15.32:0/2373744469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98145570 0x7fbd980cf420 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbd9806c3b0 tx=0x7fbd8c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 -- 172.21.15.32:0/2373744469 shutdown_connections 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 --2- 172.21.15.32:0/2373744469 >> v2:172.21.15.39:3300/0 conn(0x7fbd98071450 0x7fbd9806daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 --2- 172.21.15.32:0/2373744469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98145570 0x7fbd980cf420 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 -- 172.21.15.32:0/2373744469 >> 172.21.15.32:0/2373744469 conn(0x7fbd980b17c0 msgr2=0x7fbd980b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 -- 172.21.15.32:0/2373744469 shutdown_connections 2024-03-06T07:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.654+0000 7fbd9e400640 1 -- 172.21.15.32:0/2373744469 wait complete. 2024-03-06T07:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.656+0000 7fbd9e400640 1 Processor -- start 2024-03-06T07:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.656+0000 7fbd9e400640 1 -- start start 2024-03-06T07:30:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 0x7fbd9813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9e400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbd98071450 0x7fbd98141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd98145570 0x7fbd9815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980718b0 con 0x7fbd9806e030 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980727d0 con 0x7fbd98145570 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9e400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98071a30 con 0x7fbd98071450 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.657+0000 7fbd9d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 0x7fbd9813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 0x7fbd9813ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41666/0 (socket says 172.21.15.32:41666) 2024-03-06T07:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 -- 172.21.15.32:0/39053729 learned_addr learned my addr 172.21.15.32:0/39053729 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 -- 172.21.15.32:0/39053729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbd98071450 msgr2=0x7fbd98141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbd98071450 0x7fbd98141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 -- 172.21.15.32:0/39053729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd98145570 msgr2=0x7fbd9815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd98145570 0x7fbd9815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 -- 172.21.15.32:0/39053729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9813f340 con 0x7fbd9806e030 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.658+0000 7fbd9d000640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 0x7fbd9813ecf0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c02b950 tx=0x7fbd8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.659+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c040070 con 0x7fbd9806e030 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.659+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c030d00 con 0x7fbd9806e030 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.659+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c035040 con 0x7fbd9806e030 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.659+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd980ad920 con 0x7fbd9806e030 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.659+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd980adee0 con 0x7fbd9806e030 2024-03-06T07:30:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.660+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9804f330 con 0x7fbd9806e030 2024-03-06T07:30:46.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.661+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd8c04b060 con 0x7fbd9806e030 2024-03-06T07:30:46.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.662+0000 7fbd94c00640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7806b4d0 0x7fbd7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.662+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fbd8c0abed0 con 0x7fbd9806e030 2024-03-06T07:30:46.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.662+0000 7fbd97e00640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7806b4d0 0x7fbd7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.663+0000 7fbd97e00640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7806b4d0 0x7fbd7806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbd981f6d50 tx=0x7fbd84008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.667+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd8c07ae10 con 0x7fbd9806e030 2024-03-06T07:30:46.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:46.865+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbd980cf740 con 0x7fbd9806e030 2024-03-06T07:30:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.425+0000 7fbd94c00640 1 -- 172.21.15.32:0/39053729 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbd8c07a710 con 0x7fbd9806e030 2024-03-06T07:30:47.428 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.428+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7806b4d0 msgr2=0x7fbd7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.428+0000 7fbd9e400640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7806b4d0 0x7fbd7806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbd981f6d50 tx=0x7fbd84008040 comp rx=0 tx=0).stop 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.428+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 msgr2=0x7fbd9813ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.428+0000 7fbd9e400640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 0x7fbd9813ecf0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c02b950 tx=0x7fbd8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 shutdown_connections 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7806b4d0 0x7fbd7806d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd98145570 0x7fbd9815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbd98071450 0x7fbd98141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 --2- 172.21.15.32:0/39053729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd9806e030 0x7fbd9813ecf0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 >> 172.21.15.32:0/39053729 conn(0x7fbd980b17c0 msgr2=0x7fbd9815c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 shutdown_connections 2024-03-06T07:30:47.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.429+0000 7fbd9e400640 1 -- 172.21.15.32:0/39053729 wait complete. 2024-03-06T07:30:47.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:47.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.665+0000 7fe2fa400640 1 Processor -- start 2024-03-06T07:30:47.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.665+0000 7fe2fa400640 1 -- start start 2024-03-06T07:30:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.666+0000 7fe2fa400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2f415afa0 0x7fe2f40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.666+0000 7fe2fa400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f40ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2fa400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f4071c60 con 0x7fe2f40cd010 2024-03-06T07:30:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2fa400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f40cfa10 con 0x7fe2f415afa0 2024-03-06T07:30:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2fa400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4071f40 con 0x7fe2f4070ce0 2024-03-06T07:30:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2f3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2f415afa0 0x7fe2f40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2f3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2f415afa0 0x7fe2f40cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41676/0 (socket says 172.21.15.32:41676) 2024-03-06T07:30:47.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2f3400640 1 -- 172.21.15.32:0/2766182089 learned_addr learned my addr 172.21.15.32:0/2766182089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:47.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.667+0000 7fe2ebe00640 1 --2- 172.21.15.32:0/2766182089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f40ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.668+0000 7fe2f3400640 1 -- 172.21.15.32:0/2766182089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 msgr2=0x7fe2f40ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.668+0000 7fe2f3400640 1 --2- 172.21.15.32:0/2766182089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f40ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.668+0000 7fe2f3400640 1 -- 172.21.15.32:0/2766182089 >> v1:172.21.15.39:6789/0 conn(0x7fe2f40cd010 legacy=0x7fe2f406c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.668+0000 7fe2f3400640 1 -- 172.21.15.32:0/2766182089 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f4072220 con 0x7fe2f415afa0 2024-03-06T07:30:47.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.669+0000 7fe2f3400640 1 --2- 172.21.15.32:0/2766182089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f415afa0 0x7fe2f40cf3a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009870 tx=0x7fe2e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71a6744d07c9fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.669+0000 7fe2f2a00640 1 -- 172.21.15.32:0/2766182089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4035020 con 0x7fe2f415afa0 2024-03-06T07:30:47.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.669+0000 7fe2f2a00640 1 -- 172.21.15.32:0/2766182089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4031640 con 0x7fe2f415afa0 2024-03-06T07:30:47.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.669+0000 7fe2f2a00640 1 -- 172.21.15.32:0/2766182089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e402d040 con 0x7fe2f415afa0 2024-03-06T07:30:47.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 -- 172.21.15.32:0/2766182089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f415afa0 msgr2=0x7fe2f40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 --2- 172.21.15.32:0/2766182089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f415afa0 0x7fe2f40cf3a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009870 tx=0x7fe2e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 -- 172.21.15.32:0/2766182089 shutdown_connections 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 --2- 172.21.15.32:0/2766182089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f40ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 --2- 172.21.15.32:0/2766182089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f415afa0 0x7fe2f40cf3a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 -- 172.21.15.32:0/2766182089 >> 172.21.15.32:0/2766182089 conn(0x7fe2f406b3d0 msgr2=0x7fe2f406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 -- 172.21.15.32:0/2766182089 shutdown_connections 2024-03-06T07:30:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.670+0000 7fe2fa400640 1 -- 172.21.15.32:0/2766182089 wait complete. 2024-03-06T07:30:47.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.672+0000 7fe2fa400640 1 Processor -- start 2024-03-06T07:30:47.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.672+0000 7fe2fa400640 1 -- start start 2024-03-06T07:30:47.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2fa400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f409d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2fa400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 0x7fe2f413c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2fa400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f415afa0 0x7fe2f413ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2fa400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40b4ad0 con 0x7fe2f40cd010 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2fa400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4071140 con 0x7fe2f415afa0 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2fa400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40b4c50 con 0x7fe2f4070ce0 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 0x7fe2f413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 0x7fe2f413c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41688/0 (socket says 172.21.15.32:41688) 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2ebe00640 1 -- 172.21.15.32:0/590333053 learned_addr learned my addr 172.21.15.32:0/590333053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2f3400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f409d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2ebe00640 1 -- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 msgr2=0x7fe2f409d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2ebe00640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f409d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2f3e00640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f415afa0 0x7fe2f413ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.673+0000 7fe2ebe00640 1 -- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f415afa0 msgr2=0x7fe2f413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2ebe00640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f415afa0 0x7fe2f413ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2ebe00640 1 -- 172.21.15.32:0/590333053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f413d0e0 con 0x7fe2f40cd010 2024-03-06T07:30:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2f3e00640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f415afa0 0x7fe2f413ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2ebe00640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 0x7fe2f413c540 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00d830 tx=0x7fe2dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00b660 con 0x7fe2f40cd010 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc019070 con 0x7fe2f40cd010 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00ba20 con 0x7fe2f40cd010 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f413d3c0 con 0x7fe2f40cd010 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.674+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f409d130 con 0x7fe2f40cd010 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.675+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f404f330 con 0x7fe2f40cd010 2024-03-06T07:30:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.676+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2dc01e460 con 0x7fe2f40cd010 2024-03-06T07:30:47.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.678+0000 7fe2eb400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2d006b4d0 0x7fe2d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.678+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe2dc08a620 con 0x7fe2f40cd010 2024-03-06T07:30:47.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.678+0000 7fe2f3400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2d006b4d0 0x7fe2d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.679+0000 7fe2f3400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2d006b4d0 0x7fe2d006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4036000 tx=0x7fe2e4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.682+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2dc059700 con 0x7fe2f40cd010 2024-03-06T07:30:47.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.881+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe2f4154900 con 0x7fe2f40cd010 2024-03-06T07:30:47.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.887+0000 7fe2eb400640 1 -- 172.21.15.32:0/590333053 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe2dc059000 con 0x7fe2f40cd010 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2d006b4d0 msgr2=0x7fe2d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2d006b4d0 0x7fe2d006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4036000 tx=0x7fe2e4008030 comp rx=0 tx=0).stop 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 msgr2=0x7fe2f413c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 0x7fe2f413c540 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00d830 tx=0x7fe2dc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 shutdown_connections 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe2d006b4d0 0x7fe2d006d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f415afa0 0x7fe2f413ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2f40cd010 0x7fe2f413c540 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 --2- 172.21.15.32:0/590333053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe2f4070ce0 0x7fe2f409d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 >> 172.21.15.32:0/590333053 conn(0x7fe2f406b3d0 msgr2=0x7fe2f406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 shutdown_connections 2024-03-06T07:30:47.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:47.890+0000 7fe2fa400640 1 -- 172.21.15.32:0/590333053 wait complete. 2024-03-06T07:30:47.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:30:48.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.138+0000 7ff2f9000640 1 Processor -- start 2024-03-06T07:30:48.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.138+0000 7ff2f9000640 1 -- start start 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.139+0000 7ff2f9000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 0x7ff2f40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.139+0000 7ff2f9000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2f4070f10 0x7ff2f40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.139+0000 7ff2f9000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f40714d0 con 0x7ff2f40cce90 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.139+0000 7ff2f9000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f40cd300 con 0x7ff2f4070f10 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.139+0000 7ff2f9000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40717b0 con 0x7ff2f4145300 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff2f40cce90 0x7ff2f40cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59758/0 (socket says 172.21.15.32:59758) 2024-03-06T07:30:48.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f3e00640 1 -- 172.21.15.32:0/3118239936 learned_addr learned my addr 172.21.15.32:0/3118239936 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f3400640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 0x7ff2f40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144553880 0 0) 0x7ff2f40714d0 con 0x7ff2f40cce90 2024-03-06T07:30:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d8003520 con 0x7ff2f40cce90 2024-03-06T07:30:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509013026 0 0) 0x7ff2d8003520 con 0x7ff2f40cce90 2024-03-06T07:30:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 msgr2=0x7ff2f40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 0x7ff2f40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 >> v2:172.21.15.39:3300/0 conn(0x7ff2f4070f10 msgr2=0x7ff2f40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 --2- 172.21.15.32:0/3118239936 >> v2:172.21.15.39:3300/0 conn(0x7ff2f4070f10 0x7ff2f40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4071a90 con 0x7ff2f40cce90 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff2ec0019f0 con 0x7ff2f40cce90 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.140+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 >> v1:172.21.15.32:6789/0 conn(0x7ff2f40cce90 legacy=0x7ff2f40cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2000640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d80039b0 0x7ff2d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2000640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2000640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 0x7ff2d80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40cd300 con 0x7ff2d8004800 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40717b0 con 0x7ff2d80039b0 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d800bb50 con 0x7ff2f4145300 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f3400640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d80039b0 0x7ff2d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f3e00640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 0x7ff2d80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2a00640 1 -- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 msgr2=0x7ff2d80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f4145300 0x7ff2d80051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2a00640 1 -- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d80039b0 msgr2=0x7ff2d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2d80039b0 0x7ff2d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.141+0000 7ff2f2a00640 1 -- 172.21.15.32:0/3118239936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8003520 con 0x7ff2d8004800 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2d8004c80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009860 tx=0x7ff2d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d129f380f010b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0037070 con 0x7ff2d8004800 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e0032620 con 0x7ff2d8004800 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f2000640 1 -- 172.21.15.32:0/3118239936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0009460 con 0x7ff2d8004800 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f9000640 1 -- 172.21.15.32:0/3118239936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 msgr2=0x7ff2d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f9000640 1 --2- 172.21.15.32:0/3118239936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2d8004c80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009860 tx=0x7ff2d8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.143+0000 7ff2f3e00640 1 -- 172.21.15.32:0/3118239936 reap_dead start 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.144+0000 7ff2f9000640 1 -- 172.21.15.32:0/3118239936 shutdown_connections 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.144+0000 7ff2f9000640 1 -- 172.21.15.32:0/3118239936 >> 172.21.15.32:0/3118239936 conn(0x7ff2f406b3d0 msgr2=0x7ff2f406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.144+0000 7ff2f9000640 1 -- 172.21.15.32:0/3118239936 shutdown_connections 2024-03-06T07:30:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.144+0000 7ff2f9000640 1 -- 172.21.15.32:0/3118239936 wait complete. 2024-03-06T07:30:48.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.145+0000 7ff2f9000640 1 Processor -- start 2024-03-06T07:30:48.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.146+0000 7ff2f9000640 1 -- start start 2024-03-06T07:30:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2f413a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2f40a9470 0x7ff2f41f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f9000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f40a98f0 0x7ff2f41f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f9000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f415d580 con 0x7ff2d8004800 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f41456e0 con 0x7ff2f40a9470 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f9000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f415d700 con 0x7ff2f40a98f0 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2f413a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2f413a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54076/0 (socket says 172.21.15.32:54076) 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 -- 172.21.15.32:0/453207637 learned_addr learned my addr 172.21.15.32:0/453207637 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2f40a9470 0x7ff2f41f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 -- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f40a98f0 msgr2=0x7ff2f41f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f40a98f0 0x7ff2f41f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 -- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2f40a9470 msgr2=0x7ff2f41f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2f40a9470 0x7ff2f41f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.147+0000 7ff2f3400640 1 -- 172.21.15.32:0/453207637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f413efd0 con 0x7ff2d8004800 2024-03-06T07:30:48.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.148+0000 7ff2f3400640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2f413a9e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400f580 tx=0x7ff2e400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.148+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4019020 con 0x7ff2d8004800 2024-03-06T07:30:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.148+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e4014870 con 0x7ff2d8004800 2024-03-06T07:30:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.148+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4014c60 con 0x7ff2d8004800 2024-03-06T07:30:48.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.148+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f413f2b0 con 0x7ff2d8004800 2024-03-06T07:30:48.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.148+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f413f870 con 0x7ff2d8004800 2024-03-06T07:30:48.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.149+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f404f330 con 0x7ff2d8004800 2024-03-06T07:30:48.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.150+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2e401c020 con 0x7ff2d8004800 2024-03-06T07:30:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.151+0000 7ff2eb400640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2d406b500 0x7ff2d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.151+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff2e408a810 con 0x7ff2d8004800 2024-03-06T07:30:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.151+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2d406b500 0x7ff2d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.152+0000 7ff2f2a00640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2d406b500 0x7ff2d406d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0034000 tx=0x7ff2e0008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.156+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2e4023020 con 0x7ff2d8004800 2024-03-06T07:30:48.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:48.426+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7ff2f40cd9d0 con 0x7ff2d8004800 2024-03-06T07:30:48.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:48.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:49.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.431+0000 7ff2eb400640 1 -- 172.21.15.32:0/453207637 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v120) v1 ==== 132+0+0 (secure 0 0 0) 0x7ff2e4059900 con 0x7ff2d8004800 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.434+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2d406b500 msgr2=0x7ff2d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.434+0000 7ff2f9000640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2d406b500 0x7ff2d406d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0034000 tx=0x7ff2e0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.434+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 msgr2=0x7ff2f413a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.434+0000 7ff2f9000640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2f413a9e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400f580 tx=0x7ff2e400fa40 comp rx=0 tx=0).stop 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 shutdown_connections 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff2d406b500 0x7ff2d406d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff2f40a98f0 0x7ff2f41f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2f40a9470 0x7ff2f41f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 --2- 172.21.15.32:0/453207637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d8004800 0x7ff2f413a9e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 >> 172.21.15.32:0/453207637 conn(0x7ff2f406b3d0 msgr2=0x7ff2f40ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 shutdown_connections 2024-03-06T07:30:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.435+0000 7ff2f9000640 1 -- 172.21.15.32:0/453207637 wait complete. 2024-03-06T07:30:49.457 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:30:49.457 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:30:49.457 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:49.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.670+0000 7f5ea2c00640 1 Processor -- start 2024-03-06T07:30:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.671+0000 7f5ea2c00640 1 -- start start 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.672+0000 7f5ea2c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e9c15e070 0x7f5e9c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.672+0000 7f5ea2c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5e9c0d0cd0 0x7f5e9c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.672+0000 7f5ea2c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0d18c0 con 0x7f5e9c0cf410 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.672+0000 7f5ea2c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c0d1ba0 con 0x7f5e9c15e070 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.672+0000 7f5ea2c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c0d1e80 con 0x7f5e9c0d0cd0 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.672+0000 7f5ea1800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e9c15e070 0x7f5e9c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea0e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5e9c0cf410 0x7f5e9c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59766/0 (socket says 172.21.15.32:59766) 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea0e00640 1 -- 172.21.15.32:0/169774792 learned_addr learned my addr 172.21.15.32:0/169774792 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:49.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea2200640 1 --2- 172.21.15.32:0/169774792 >> v2:172.21.15.39:3300/0 conn(0x7f5e9c0d0cd0 0x7f5e9c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea1800640 1 -- 172.21.15.32:0/169774792 >> v2:172.21.15.39:3300/0 conn(0x7f5e9c0d0cd0 msgr2=0x7f5e9c0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea1800640 1 --2- 172.21.15.32:0/169774792 >> v2:172.21.15.39:3300/0 conn(0x7f5e9c0d0cd0 0x7f5e9c0cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea1800640 1 -- 172.21.15.32:0/169774792 >> v1:172.21.15.32:6789/0 conn(0x7f5e9c0cf410 legacy=0x7f5e9c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea1800640 1 -- 172.21.15.32:0/169774792 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0d2160 con 0x7f5e9c15e070 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5e9be00640 1 -- 172.21.15.32:0/169774792 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823665555 0 0) 0x7f5e9c0d18c0 con 0x7f5e9c0cf410 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.673+0000 7f5ea2200640 1 --2- 172.21.15.32:0/169774792 >> v2:172.21.15.39:3300/0 conn(0x7f5e9c0d0cd0 0x7f5e9c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.674+0000 7f5ea1800640 1 --2- 172.21.15.32:0/169774792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c15e070 0x7f5e9c0d2960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c08c1b0 tx=0x7f5e9002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9969cb9b7c04deb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.675+0000 7f5e9be00640 1 -- 172.21.15.32:0/169774792 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90039070 con 0x7f5e9c15e070 2024-03-06T07:30:49.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.675+0000 7f5e9be00640 1 -- 172.21.15.32:0/169774792 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9002cc80 con 0x7f5e9c15e070 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.675+0000 7f5e9be00640 1 -- 172.21.15.32:0/169774792 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90034660 con 0x7f5e9c15e070 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.675+0000 7f5ea2c00640 1 -- 172.21.15.32:0/169774792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c15e070 msgr2=0x7f5e9c0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.675+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/169774792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c15e070 0x7f5e9c0d2960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c08c1b0 tx=0x7f5e9002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.676+0000 7f5ea2c00640 1 -- 172.21.15.32:0/169774792 shutdown_connections 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.676+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/169774792 >> v2:172.21.15.39:3300/0 conn(0x7f5e9c0d0cd0 0x7f5e9c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.676+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/169774792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c15e070 0x7f5e9c0d2960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.676+0000 7f5ea2c00640 1 -- 172.21.15.32:0/169774792 >> 172.21.15.32:0/169774792 conn(0x7f5e9c08d2a0 msgr2=0x7f5e9c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.676+0000 7f5ea2c00640 1 -- 172.21.15.32:0/169774792 shutdown_connections 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.676+0000 7f5ea2c00640 1 -- 172.21.15.32:0/169774792 wait complete. 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.677+0000 7f5ea2c00640 1 Processor -- start 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 -- start start 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 0x7f5e9c1513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e9c0d0cd0 0x7f5e9c0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e9c15e070 0x7f5e9c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0d4730 con 0x7f5e9c0cf410 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0d45b0 con 0x7f5e9c15e070 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.678+0000 7f5ea2c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0b3630 con 0x7f5e9c0d0cd0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 0x7f5e9c1513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 0x7f5e9c1513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54102/0 (socket says 172.21.15.32:54102) 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 -- 172.21.15.32:0/1833310077 learned_addr learned my addr 172.21.15.32:0/1833310077 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea0e00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e9c0d0cd0 0x7f5e9c0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 -- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e9c0d0cd0 msgr2=0x7f5e9c0cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e9c0d0cd0 0x7f5e9c0cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 -- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e9c15e070 msgr2=0x7f5e9c0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e9c15e070 0x7f5e9c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.679+0000 7f5ea1800640 1 -- 172.21.15.32:0/1833310077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c151930 con 0x7f5e9c0cf410 2024-03-06T07:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.680+0000 7f5ea1800640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 0x7f5e9c1513f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e90037040 tx=0x7f5e90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.680+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90039050 con 0x7f5e9c0cf410 2024-03-06T07:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.680+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e90045070 con 0x7f5e9c0cf410 2024-03-06T07:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.680+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9003ea50 con 0x7f5e9c0cf410 2024-03-06T07:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.680+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c15e5f0 con 0x7f5e9c0cf410 2024-03-06T07:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.680+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c15ebb0 con 0x7f5e9c0cf410 2024-03-06T07:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.681+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c04f330 con 0x7f5e9c0cf410 2024-03-06T07:30:49.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.682+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e9002cb00 con 0x7f5e9c0cf410 2024-03-06T07:30:49.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.683+0000 7f5e99600640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.683+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5e900acd50 con 0x7f5e9c0cf410 2024-03-06T07:30:49.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.683+0000 7f5ea0e00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.684+0000 7f5ea0e00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5e88007fd0 tx=0x7f5e88009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.688+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e9c151930 con 0x7f5e9c0cf410 2024-03-06T07:30:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.886+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e9c0c75c0 con 0x7f5e9c0cf410 2024-03-06T07:30:49.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.887+0000 7f5e99600640 1 -- 172.21.15.32:0/1833310077 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+12912 (secure 0 0 0) 0x7f5e9007bfe0 con 0x7f5e9c0cf410 2024-03-06T07:30:49.890 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:49.890 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:47.412719+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6837/2686928389":"2024-03-07T07:30:44.128878+0000","172.21.15.32:6834/2686928389":"2024-03-07T07:30:44.128878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e7c06b4d0 msgr2=0x7f5e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5e88007fd0 tx=0x7f5e88009040 comp rx=0 tx=0).stop 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 msgr2=0x7f5e9c1513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 0x7f5e9c1513f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e90037040 tx=0x7f5e90004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 shutdown_connections 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e9c15e070 0x7f5e9c0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e9c0d0cd0 0x7f5e9c0cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 --2- 172.21.15.32:0/1833310077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e9c0cf410 0x7f5e9c1513f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.890+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 >> 172.21.15.32:0/1833310077 conn(0x7f5e9c08d2a0 msgr2=0x7f5e9c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.891+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 shutdown_connections 2024-03-06T07:30:49.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:49.891+0000 7f5ea2c00640 1 -- 172.21.15.32:0/1833310077 wait complete. 2024-03-06T07:30:49.913 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:49.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:50.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.128+0000 7f460aa00640 1 Processor -- start 2024-03-06T07:30:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.128+0000 7f460aa00640 1 -- start start 2024-03-06T07:30:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f460aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4604072520 0x7f46040cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f460aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4604071e20 0x7f46040ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f460aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4604072df0 con 0x7f460415bf20 2024-03-06T07:30:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f460aa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46040cdab0 con 0x7f4604071e20 2024-03-06T07:30:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f460aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46040b4d70 con 0x7f4604072520 2024-03-06T07:30:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f4608a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4604071e20 0x7f46040ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f4608a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4604071e20 0x7f46040ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54116/0 (socket says 172.21.15.32:54116) 2024-03-06T07:30:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.130+0000 7f4608a00640 1 -- 172.21.15.32:0/1420699692 learned_addr learned my addr 172.21.15.32:0/1420699692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:50.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.131+0000 7f4603400640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4604072520 0x7f46040cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.131+0000 7f4608a00640 1 -- 172.21.15.32:0/1420699692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4604072520 msgr2=0x7f46040cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.131+0000 7f4608a00640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4604072520 0x7f46040cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.131+0000 7f4608a00640 1 -- 172.21.15.32:0/1420699692 >> v1:172.21.15.39:6789/0 conn(0x7f460415bf20 legacy=0x7f46040acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.131+0000 7f4608a00640 1 -- 172.21.15.32:0/1420699692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46040b5050 con 0x7f4604071e20 2024-03-06T07:30:50.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.131+0000 7f4603400640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4604072520 0x7f46040cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.132+0000 7f4608a00640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604071e20 0x7f46040ce4f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f45fc009e50 tx=0x7f45fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a49c5597950870c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.132+0000 7f4602a00640 1 -- 172.21.15.32:0/1420699692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc02ca70 con 0x7f4604071e20 2024-03-06T07:30:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.132+0000 7f4602a00640 1 -- 172.21.15.32:0/1420699692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc02cc10 con 0x7f4604071e20 2024-03-06T07:30:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.132+0000 7f4602a00640 1 -- 172.21.15.32:0/1420699692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc0343f0 con 0x7f4604071e20 2024-03-06T07:30:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 -- 172.21.15.32:0/1420699692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604071e20 msgr2=0x7f46040ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604071e20 0x7f46040ce4f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f45fc009e50 tx=0x7f45fc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 -- 172.21.15.32:0/1420699692 shutdown_connections 2024-03-06T07:30:50.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604071e20 0x7f46040ce4f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 --2- 172.21.15.32:0/1420699692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4604072520 0x7f46040cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 -- 172.21.15.32:0/1420699692 >> 172.21.15.32:0/1420699692 conn(0x7f460406b3d0 msgr2=0x7f460406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 -- 172.21.15.32:0/1420699692 shutdown_connections 2024-03-06T07:30:50.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.133+0000 7f460aa00640 1 -- 172.21.15.32:0/1420699692 wait complete. 2024-03-06T07:30:50.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.135+0000 7f460aa00640 1 Processor -- start 2024-03-06T07:30:50.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.135+0000 7f460aa00640 1 -- start start 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f460aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4604071e20 0x7f460414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f460aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 0x7f460409b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f460aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f460415bf20 0x7f46040b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f460aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46040ad450 con 0x7f4604072520 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f460aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46040b4990 con 0x7f4604071e20 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f460aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460415d090 con 0x7f460415bf20 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f4603400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 0x7f460409b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f4603400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 0x7f460409b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54130/0 (socket says 172.21.15.32:54130) 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.136+0000 7f4603400640 1 -- 172.21.15.32:0/3636066239 learned_addr learned my addr 172.21.15.32:0/3636066239 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603e00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4604071e20 0x7f460414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603400640 1 -- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f460415bf20 msgr2=0x7f46040b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603400640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f460415bf20 0x7f46040b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603400640 1 -- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4604071e20 msgr2=0x7f460414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603400640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4604071e20 0x7f460414ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603400640 1 -- 172.21.15.32:0/3636066239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460414bf40 con 0x7f4604072520 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f4603400640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 0x7f460409b3f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f45f000c7f0 tx=0x7f45f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0015070 con 0x7f4604072520 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f0007820 con 0x7f4604072520 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460409b930 con 0x7f4604072520 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.137+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0007bd0 con 0x7f4604072520 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.138+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460409c4d0 con 0x7f4604072520 2024-03-06T07:30:50.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.139+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460404f330 con 0x7f4604072520 2024-03-06T07:30:50.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.140+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45f0018070 con 0x7f4604072520 2024-03-06T07:30:50.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.141+0000 7f45fbe00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f45e406b4d0 0x7f45e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.141+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f45f008a030 con 0x7f4604072520 2024-03-06T07:30:50.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.141+0000 7f4603e00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f45e406b4d0 0x7f45e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.142+0000 7f4603e00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f45e406b4d0 0x7f45e406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f45f400a750 tx=0x7f45f4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.146+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f460414bf40 con 0x7f4604072520 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.338+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4604156d60 con 0x7f4604072520 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.339+0000 7f45fbe00640 1 -- 172.21.15.32:0/3636066239 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45f00591b0 con 0x7f4604072520 2024-03-06T07:30:50.342 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f45e406b4d0 msgr2=0x7f45e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f45e406b4d0 0x7f45e406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f45f400a750 tx=0x7f45f4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 msgr2=0x7f460409b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 0x7f460409b3f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f45f000c7f0 tx=0x7f45f000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 shutdown_connections 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f45e406b4d0 0x7f45e406d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f460415bf20 0x7f46040b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4604072520 0x7f460409b3f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 --2- 172.21.15.32:0/3636066239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4604071e20 0x7f460414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 >> 172.21.15.32:0/3636066239 conn(0x7f460406b3d0 msgr2=0x7f460415b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 shutdown_connections 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.342+0000 7f460aa00640 1 -- 172.21.15.32:0/3636066239 wait complete. 2024-03-06T07:30:50.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:30:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.577+0000 7f8976600640 1 Processor -- start 2024-03-06T07:30:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.577+0000 7f8976600640 1 -- start start 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8976600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8970070f10 0x7f89700cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8976600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89700cce90 0x7f897015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8976600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700714d0 con 0x7f8970145300 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8976600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89700cd300 con 0x7f89700cce90 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8976600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89700717b0 con 0x7f8970070f10 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8975c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89700cce90 0x7f897015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8975c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89700cce90 0x7f897015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54140/0 (socket says 172.21.15.32:54140) 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8975c00640 1 -- 172.21.15.32:0/3239007935 learned_addr learned my addr 172.21.15.32:0/3239007935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.579+0000 7f8975200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8970145300 0x7f89700b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59780/0 (socket says 172.21.15.32:59780) 2024-03-06T07:30:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.580+0000 7f896fe00640 1 --2- 172.21.15.32:0/3239007935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8970070f10 0x7f89700cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.580+0000 7f8975c00640 1 -- 172.21.15.32:0/3239007935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8970070f10 msgr2=0x7f89700cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.580+0000 7f8975c00640 1 --2- 172.21.15.32:0/3239007935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8970070f10 0x7f89700cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.580+0000 7f8975c00640 1 -- 172.21.15.32:0/3239007935 >> v1:172.21.15.32:6789/0 conn(0x7f8970145300 legacy=0x7f89700b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.580+0000 7f8975c00640 1 -- 172.21.15.32:0/3239007935 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970071a90 con 0x7f89700cce90 2024-03-06T07:30:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.581+0000 7f8975c00640 1 --2- 172.21.15.32:0/3239007935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89700cce90 0x7f897015c570 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8968009e50 tx=0x7f896802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6804404b3e22a99e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.581+0000 7f896f400640 1 -- 172.21.15.32:0/3239007935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896802cb00 con 0x7f89700cce90 2024-03-06T07:30:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f896f400640 1 -- 172.21.15.32:0/3239007935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896802cca0 con 0x7f89700cce90 2024-03-06T07:30:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f896f400640 1 -- 172.21.15.32:0/3239007935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968034610 con 0x7f89700cce90 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 -- 172.21.15.32:0/3239007935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89700cce90 msgr2=0x7f897015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 --2- 172.21.15.32:0/3239007935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89700cce90 0x7f897015c570 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8968009e50 tx=0x7f896802bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 -- 172.21.15.32:0/3239007935 shutdown_connections 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 --2- 172.21.15.32:0/3239007935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89700cce90 0x7f897015c570 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 --2- 172.21.15.32:0/3239007935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8970070f10 0x7f89700cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 -- 172.21.15.32:0/3239007935 >> 172.21.15.32:0/3239007935 conn(0x7f897006b3d0 msgr2=0x7f897006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 -- 172.21.15.32:0/3239007935 shutdown_connections 2024-03-06T07:30:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.582+0000 7f8976600640 1 -- 172.21.15.32:0/3239007935 wait complete. 2024-03-06T07:30:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.584+0000 7f8976600640 1 Processor -- start 2024-03-06T07:30:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.584+0000 7f8976600640 1 -- start start 2024-03-06T07:30:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8976600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 0x7f8970141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8976600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89700cce90 0x7f897006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8976600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8970145300 0x7f897015ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8976600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897015cab0 con 0x7f8970070f10 2024-03-06T07:30:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8976600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89701456e0 con 0x7f8970145300 2024-03-06T07:30:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8976600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897015cc30 con 0x7f89700cce90 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8975200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 0x7f8970141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8975200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 0x7f8970141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54152/0 (socket says 172.21.15.32:54152) 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.585+0000 7f8975200640 1 -- 172.21.15.32:0/2257280789 learned_addr learned my addr 172.21.15.32:0/2257280789 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f896fe00640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89700cce90 0x7f897006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8975200640 1 -- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89700cce90 msgr2=0x7f897006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8975200640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89700cce90 0x7f897006dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8975200640 1 -- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8970145300 msgr2=0x7f897015ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8975200640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8970145300 0x7f897015ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8975200640 1 -- 172.21.15.32:0/2257280789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897013c930 con 0x7f8970070f10 2024-03-06T07:30:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8975200640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 0x7f8970141150 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f896400c3f0 tx=0x7f896400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89640074e0 con 0x7f8970070f10 2024-03-06T07:30:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8964007680 con 0x7f8970070f10 2024-03-06T07:30:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964004a00 con 0x7f8970070f10 2024-03-06T07:30:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.586+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897013cc10 con 0x7f8970070f10 2024-03-06T07:30:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.587+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897013d1d0 con 0x7f8970070f10 2024-03-06T07:30:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.588+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897004f330 con 0x7f8970070f10 2024-03-06T07:30:50.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.589+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f896401a020 con 0x7f8970070f10 2024-03-06T07:30:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.590+0000 7f896cc00640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f895406b4d0 0x7f895406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.590+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8964089670 con 0x7f8970070f10 2024-03-06T07:30:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.590+0000 7f896fe00640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f895406b4d0 0x7f895406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.591+0000 7f896fe00640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f895406b4d0 0x7f895406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f897006dbe0 tx=0x7f8960008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.594+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8964010300 con 0x7f8970070f10 2024-03-06T07:30:50.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:50.869+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f8970144820 con 0x7f8970070f10 2024-03-06T07:30:51.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.449+0000 7f896cc00640 1 -- 172.21.15.32:0/2257280789 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v122) v1 ==== 136+0+0 (secure 0 0 0) 0x7f89640587e0 con 0x7f8970070f10 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f895406b4d0 msgr2=0x7f895406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f895406b4d0 0x7f895406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f897006dbe0 tx=0x7f8960008040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 msgr2=0x7f8970141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 0x7f8970141150 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f896400c3f0 tx=0x7f896400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 shutdown_connections 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f895406b4d0 0x7f895406d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.452+0000 7f8976600640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8970145300 0x7f897015ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.453+0000 7f8976600640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89700cce90 0x7f897006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.453+0000 7f8976600640 1 --2- 172.21.15.32:0/2257280789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8970070f10 0x7f8970141150 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.453+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 >> 172.21.15.32:0/2257280789 conn(0x7f897006b3d0 msgr2=0x7f897015dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.453+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 shutdown_connections 2024-03-06T07:30:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.453+0000 7f8976600640 1 -- 172.21.15.32:0/2257280789 wait complete. 2024-03-06T07:30:51.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.703+0000 7fe69cc00640 1 Processor -- start 2024-03-06T07:30:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.704+0000 7fe69cc00640 1 -- start start 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.705+0000 7fe69cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe698145430 0x7fe69815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.705+0000 7fe69cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe69815cc90 0x7fe6980b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.705+0000 7fe69cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6980b3fe0 con 0x7fe6980b2080 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.705+0000 7fe69cc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6980b24d0 con 0x7fe698145430 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.705+0000 7fe69cc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69815fe40 con 0x7fe69815cc90 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe698145430 0x7fe69815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe698145430 0x7fe69815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54160/0 (socket says 172.21.15.32:54160) 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 -- 172.21.15.32:0/3194369854 learned_addr learned my addr 172.21.15.32:0/3194369854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe696a00640 1 --2- 172.21.15.32:0/3194369854 >> v2:172.21.15.39:3300/0 conn(0x7fe69815cc90 0x7fe6980b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 -- 172.21.15.32:0/3194369854 >> v2:172.21.15.39:3300/0 conn(0x7fe69815cc90 msgr2=0x7fe6980b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 --2- 172.21.15.32:0/3194369854 >> v2:172.21.15.39:3300/0 conn(0x7fe69815cc90 0x7fe6980b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 -- 172.21.15.32:0/3194369854 >> v1:172.21.15.39:6789/0 conn(0x7fe6980b2080 legacy=0x7fe69806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe697400640 1 -- 172.21.15.32:0/3194369854 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698160120 con 0x7fe698145430 2024-03-06T07:30:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.706+0000 7fe696a00640 1 --2- 172.21.15.32:0/3194369854 >> v2:172.21.15.39:3300/0 conn(0x7fe69815cc90 0x7fe6980b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.708+0000 7fe697400640 1 --2- 172.21.15.32:0/3194369854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe698145430 0x7fe69815f810 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe69806d8c0 tx=0x7fe68802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626b5262de2fe52b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.708+0000 7fe696000640 1 -- 172.21.15.32:0/3194369854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688035020 con 0x7fe698145430 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.708+0000 7fe696000640 1 -- 172.21.15.32:0/3194369854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688031870 con 0x7fe698145430 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.708+0000 7fe696000640 1 -- 172.21.15.32:0/3194369854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688031c40 con 0x7fe698145430 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 -- 172.21.15.32:0/3194369854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe698145430 msgr2=0x7fe69815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 --2- 172.21.15.32:0/3194369854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe698145430 0x7fe69815f810 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe69806d8c0 tx=0x7fe68802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 -- 172.21.15.32:0/3194369854 shutdown_connections 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 --2- 172.21.15.32:0/3194369854 >> v2:172.21.15.39:3300/0 conn(0x7fe69815cc90 0x7fe6980b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 --2- 172.21.15.32:0/3194369854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe698145430 0x7fe69815f810 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 -- 172.21.15.32:0/3194369854 >> 172.21.15.32:0/3194369854 conn(0x7fe69806b3a0 msgr2=0x7fe69806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 -- 172.21.15.32:0/3194369854 shutdown_connections 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.709+0000 7fe69cc00640 1 -- 172.21.15.32:0/3194369854 wait complete. 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.711+0000 7fe69cc00640 1 Processor -- start 2024-03-06T07:30:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.711+0000 7fe69cc00640 1 -- start start 2024-03-06T07:30:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe69cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 0x7fe6981f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe69cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe698145430 0x7fe69815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe69cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe69815cc90 0x7fe69815eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe69cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69815d0d0 con 0x7fe6980b2080 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe69cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6981606e0 con 0x7fe698145430 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe69cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe698145810 con 0x7fe69815cc90 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 0x7fe6981f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 0x7fe6981f6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54170/0 (socket says 172.21.15.32:54170) 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 -- 172.21.15.32:0/1019958713 learned_addr learned my addr 172.21.15.32:0/1019958713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697e00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe69815cc90 0x7fe69815eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe696a00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe698145430 0x7fe69815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 -- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe69815cc90 msgr2=0x7fe69815eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe69815cc90 0x7fe69815eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 -- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe698145430 msgr2=0x7fe69815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe698145430 0x7fe69815e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.712+0000 7fe697400640 1 -- 172.21.15.32:0/1019958713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698136bb0 con 0x7fe6980b2080 2024-03-06T07:30:51.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.713+0000 7fe697400640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 0x7fe6981f6ab0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe68802b950 tx=0x7fe6880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.713+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688040070 con 0x7fe6980b2080 2024-03-06T07:30:51.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.713+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68803a420 con 0x7fe6980b2080 2024-03-06T07:30:51.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.713+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688035040 con 0x7fe6980b2080 2024-03-06T07:30:51.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.713+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe698136e90 con 0x7fe6980b2080 2024-03-06T07:30:51.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.713+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe698137450 con 0x7fe6980b2080 2024-03-06T07:30:51.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.715+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69804f330 con 0x7fe6980b2080 2024-03-06T07:30:51.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.716+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe688038020 con 0x7fe6980b2080 2024-03-06T07:30:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.716+0000 7fe68f400640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe67406b4d0 0x7fe67406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.717+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe6880abfe0 con 0x7fe6980b2080 2024-03-06T07:30:51.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.717+0000 7fe696a00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe67406b4d0 0x7fe67406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.717+0000 7fe696a00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe67406b4d0 0x7fe67406d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe6980ae900 tx=0x7fe680008430 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.722+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe68807b0e0 con 0x7fe6980b2080 2024-03-06T07:30:51.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.926+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe69806df70 con 0x7fe6980b2080 2024-03-06T07:30:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.934+0000 7fe68f400640 1 -- 172.21.15.32:0/1019958713 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe68807a9e0 con 0x7fe6980b2080 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.936+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe67406b4d0 msgr2=0x7fe67406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.936+0000 7fe69cc00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe67406b4d0 0x7fe67406d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe6980ae900 tx=0x7fe680008430 comp rx=0 tx=0).stop 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.936+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 msgr2=0x7fe6981f6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.936+0000 7fe69cc00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 0x7fe6981f6ab0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe68802b950 tx=0x7fe6880046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 shutdown_connections 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe67406b4d0 0x7fe67406d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe69815cc90 0x7fe69815eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe698145430 0x7fe69815e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 --2- 172.21.15.32:0/1019958713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6980b2080 0x7fe6981f6ab0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 >> 172.21.15.32:0/1019958713 conn(0x7fe69806b3a0 msgr2=0x7fe69813fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 shutdown_connections 2024-03-06T07:30:51.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:51.937+0000 7fe69cc00640 1 -- 172.21.15.32:0/1019958713 wait complete. 2024-03-06T07:30:51.960 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:51.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:52.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:52.191 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:52.191 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:30:52.191 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:52.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:52.251 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:30:52.251 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:52.312 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:52.312 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:30:52.312 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:52.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:52.373 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:30:52.373 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:52.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:52.432 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:52.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.658+0000 7f22a5200640 1 Processor -- start 2024-03-06T07:30:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.658+0000 7f22a5200640 1 -- start start 2024-03-06T07:30:52.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.660+0000 7f22a5200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 0x7f22a00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.660+0000 7f22a5200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a00714d0 con 0x7f22a00cd060 2024-03-06T07:30:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.660+0000 7f22a5200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a00717b0 con 0x7f22a0070f10 2024-03-06T07:30:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.660+0000 7f22a5200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0071a90 con 0x7f22a0145300 2024-03-06T07:30:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.660+0000 7f229fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f22a00cd060 0x7f22a015c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59788/0 (socket says 172.21.15.32:59788) 2024-03-06T07:30:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.660+0000 7f229fe00640 1 -- 172.21.15.32:0/1276192617 learned_addr learned my addr 172.21.15.32:0/1276192617 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229f400640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 0x7f22a00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140539852 0 0) 0x7f22a00714d0 con 0x7f22a00cd060 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2284003520 con 0x7f22a00cd060 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865555171 0 0) 0x7f2284003520 con 0x7f22a00cd060 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 msgr2=0x7f22a00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 0x7f22a00b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 >> v1:172.21.15.39:6789/0 conn(0x7f22a0070f10 legacy=0x7f22a00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a0071d70 con 0x7f22a00cd060 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.661+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f22980019f0 con 0x7f22a00cd060 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 >> v1:172.21.15.32:6789/0 conn(0x7f22a00cd060 legacy=0x7f22a015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22840039e0 0x7f2284003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284004800 0x7f2284004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 0x7f22840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0071a90 con 0x7f2284004800 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228400ba70 con 0x7f22840039e0 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228400bc70 con 0x7f22a0145300 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229ea00640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284004800 0x7f2284004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229f400640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22840039e0 0x7f2284003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229ea00640 1 -- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 msgr2=0x7f22840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229ea00640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a0145300 0x7f22840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229ea00640 1 -- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22840039e0 msgr2=0x7f2284003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229ea00640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22840039e0 0x7f2284003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.662+0000 7f229ea00640 1 -- 172.21.15.32:0/1276192617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2284003520 con 0x7f2284004800 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.664+0000 7f229ea00640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284004800 0x7f2284004c80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f228c001720 tx=0x7f2284005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98f94f1dcbc539b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.664+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c001370 con 0x7f2284004800 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.664+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c0052d0 con 0x7f2284004800 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.664+0000 7f229e000640 1 -- 172.21.15.32:0/1276192617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c003830 con 0x7f2284004800 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f22a5200640 1 -- 172.21.15.32:0/1276192617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284004800 msgr2=0x7f2284004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f22a5200640 1 --2- 172.21.15.32:0/1276192617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284004800 0x7f2284004c80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f228c001720 tx=0x7f2284005830 comp rx=0 tx=0).stop 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f229fe00640 1 -- 172.21.15.32:0/1276192617 reap_dead start 2024-03-06T07:30:52.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f22a5200640 1 -- 172.21.15.32:0/1276192617 shutdown_connections 2024-03-06T07:30:52.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f22a5200640 1 -- 172.21.15.32:0/1276192617 >> 172.21.15.32:0/1276192617 conn(0x7f22a006b3d0 msgr2=0x7f22a006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f22a5200640 1 -- 172.21.15.32:0/1276192617 shutdown_connections 2024-03-06T07:30:52.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.665+0000 7f22a5200640 1 -- 172.21.15.32:0/1276192617 wait complete. 2024-03-06T07:30:52.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.667+0000 7f22a5200640 1 Processor -- start 2024-03-06T07:30:52.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.667+0000 7f22a5200640 1 -- start start 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f22a5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2284004800 0x7f22a00b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f22a5200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a013db10 0x7f22a01f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f22a5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 0x7f22a015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f22a5200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00b4b20 con 0x7f22a015e6d0 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f22a5200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00b49a0 con 0x7f2284004800 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f22a5200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00b4ca0 con 0x7f22a013db10 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f229fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 0x7f22a015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f229fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 0x7f22a015eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54178/0 (socket says 172.21.15.32:54178) 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.668+0000 7f229fe00640 1 -- 172.21.15.32:0/3856026626 learned_addr learned my addr 172.21.15.32:0/3856026626 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229ea00640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a013db10 0x7f22a01f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229f400640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2284004800 0x7f22a00b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229fe00640 1 -- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a013db10 msgr2=0x7f22a01f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229fe00640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a013db10 0x7f22a01f3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229fe00640 1 -- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2284004800 msgr2=0x7f22a00b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229fe00640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2284004800 0x7f22a00b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229fe00640 1 -- 172.21.15.32:0/3856026626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a013e2f0 con 0x7f22a015e6d0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f229fe00640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 0x7f22a015eb50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f22a00b40c0 tx=0x7f229800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229800a5e0 con 0x7f22a015e6d0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229800a780 con 0x7f22a015e6d0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a013e5d0 con 0x7f22a015e6d0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.669+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229800ab30 con 0x7f22a015e6d0 2024-03-06T07:30:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.670+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a013eb90 con 0x7f22a015e6d0 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.671+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a004f330 con 0x7f22a015e6d0 2024-03-06T07:30:52.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.672+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f229800acd0 con 0x7f22a015e6d0 2024-03-06T07:30:52.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.673+0000 7f2297400640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228006b4d0 0x7f228006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.673+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2298089810 con 0x7f22a015e6d0 2024-03-06T07:30:52.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.673+0000 7f229f400640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228006b4d0 0x7f228006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.674+0000 7f229f400640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228006b4d0 0x7f228006d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f22a015bb40 tx=0x7f2290008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.677+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2298058990 con 0x7f22a015e6d0 2024-03-06T07:30:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.918+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22a015c6f0 con 0x7f22a015e6d0 2024-03-06T07:30:52.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.920+0000 7f2297400640 1 -- 172.21.15.32:0/3856026626 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f2298058290 con 0x7f22a015e6d0 2024-03-06T07:30:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3063368539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3063368539},{"type":"v1","addr":"172.21.15.39:6835","nonce":3063368539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1938093632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1938093632},{"type":"v1","addr":"172.21.15.39:6834","nonce":1938093632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/3929455577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":3929455577},{"type":"v1","addr":"172.21.15.32:6836","nonce":3929455577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3326890409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3326890409},{"type":"v1","addr":"172.21.15.32:6837","nonce":3326890409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-03-06T07:30:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 19 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228006b4d0 msgr2=0x7f228006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228006b4d0 0x7f228006d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f22a015bb40 tx=0x7f2290008040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 msgr2=0x7f22a015eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 0x7f22a015eb50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f22a00b40c0 tx=0x7f229800ccd0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 shutdown_connections 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228006b4d0 0x7f228006d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22a015e6d0 0x7f22a015eb50 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22a013db10 0x7f22a01f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 --2- 172.21.15.32:0/3856026626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2284004800 0x7f22a00b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.922+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 >> 172.21.15.32:0/3856026626 conn(0x7f22a006b3d0 msgr2=0x7f22a00ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.923+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 shutdown_connections 2024-03-06T07:30:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:52.923+0000 7f22a5200640 1 -- 172.21.15.32:0/3856026626 wait complete. 2024-03-06T07:30:52.944 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:52.944 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:52.944 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:52.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:52.945 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:52.945 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:52.945 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:52.945 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:52.945 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:52.945 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:52.945 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:52.946 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:52.946 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:52.946 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:30:52.946 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:52.946 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:52.961 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:52.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:52.964 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:52.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:52.975 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:52.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:52.976 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:52.976 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:53.011 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:53.011 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:53.017 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:53.017 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:53.023 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:53.023 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:53.029 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:53.029 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:53.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:53.170 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:30:53.171 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:30:53.190 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:30:53.195 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:30:53.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.215+0000 7f5173400640 1 Processor -- start 2024-03-06T07:30:53.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.216+0000 7f5173400640 1 -- start start 2024-03-06T07:30:53.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5173400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f516c0af1a0 0x7f516c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5173400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 0x7f516c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5173400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c15cc40 con 0x7f516c0cdc20 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5173400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c15cf20 con 0x7f516c0af1a0 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5173400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c15d200 con 0x7f516c0af580 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5170a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f516c0af1a0 0x7f516c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f516be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 0x7f516c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5171400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f516c0cdc20 0x7f516c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59812/0 (socket says 172.21.15.32:59812) 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f516be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 0x7f516c0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:43868/0 (socket says 172.21.15.32:43868) 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.217+0000 7f5171400640 1 -- 172.21.15.32:0/663443081 learned_addr learned my addr 172.21.15.32:0/663443081 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:53.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.218+0000 7f5170a00640 1 -- 172.21.15.32:0/663443081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 msgr2=0x7f516c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.218+0000 7f5170a00640 1 --2- 172.21.15.32:0/663443081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 0x7f516c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.218+0000 7f5170a00640 1 -- 172.21.15.32:0/663443081 >> v1:172.21.15.32:6789/0 conn(0x7f516c0cdc20 legacy=0x7f516c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.218+0000 7f5170a00640 1 -- 172.21.15.32:0/663443081 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c15d4e0 con 0x7f516c0af1a0 2024-03-06T07:30:53.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.218+0000 7f516be00640 1 --2- 172.21.15.32:0/663443081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 0x7f516c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.219+0000 7f5170a00640 1 --2- 172.21.15.32:0/663443081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c15c600 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f51600097a0 tx=0x7f516002b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5d96c3cd375229 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f516b400640 1 -- 172.21.15.32:0/663443081 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160037070 con 0x7f516c0af1a0 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f516b400640 1 -- 172.21.15.32:0/663443081 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5160030710 con 0x7f516c0af1a0 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f516b400640 1 -- 172.21.15.32:0/663443081 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516002be60 con 0x7f516c0af1a0 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f5173400640 1 -- 172.21.15.32:0/663443081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 msgr2=0x7f516c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f5173400640 1 --2- 172.21.15.32:0/663443081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c15c600 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f51600097a0 tx=0x7f516002b660 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f5173400640 1 -- 172.21.15.32:0/663443081 shutdown_connections 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f5173400640 1 --2- 172.21.15.32:0/663443081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0af580 0x7f516c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f5173400640 1 --2- 172.21.15.32:0/663443081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c15c600 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.220+0000 7f5173400640 1 -- 172.21.15.32:0/663443081 >> 172.21.15.32:0/663443081 conn(0x7f516c06afb0 msgr2=0x7f516c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.221+0000 7f5173400640 1 -- 172.21.15.32:0/663443081 shutdown_connections 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.221+0000 7f5173400640 1 -- 172.21.15.32:0/663443081 wait complete. 2024-03-06T07:30:53.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.222+0000 7f5173400640 1 Processor -- start 2024-03-06T07:30:53.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.222+0000 7f5173400640 1 -- start start 2024-03-06T07:30:53.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.223+0000 7f5173400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c0cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.223+0000 7f5173400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f516c0af580 0x7f516c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.223+0000 7f5173400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0cdc20 0x7f516c15e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.223+0000 7f5173400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0b3660 con 0x7f516c0af1a0 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.223+0000 7f5173400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0b34e0 con 0x7f516c0af580 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.223+0000 7f5173400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0b37e0 con 0x7f516c0cdc20 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c0cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c0cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54240/0 (socket says 172.21.15.32:54240) 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 -- 172.21.15.32:0/1000426486 learned_addr learned my addr 172.21.15.32:0/1000426486 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f516be00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f516c0af580 0x7f516c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5171400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0cdc20 0x7f516c15e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 -- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0cdc20 msgr2=0x7f516c15e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0cdc20 0x7f516c15e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 -- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f516c0af580 msgr2=0x7f516c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f516c0af580 0x7f516c0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 -- 172.21.15.32:0/1000426486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c0b1350 con 0x7f516c0af1a0 2024-03-06T07:30:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.224+0000 7f5170a00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c0cb8c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5160035040 tx=0x7f51600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.225+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160041070 con 0x7f516c0af1a0 2024-03-06T07:30:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.225+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516003c670 con 0x7f516c0af1a0 2024-03-06T07:30:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.225+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160037040 con 0x7f516c0af1a0 2024-03-06T07:30:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.225+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c06eb20 con 0x7f516c0af1a0 2024-03-06T07:30:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.225+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c0afbc0 con 0x7f516c0af1a0 2024-03-06T07:30:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.226+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c069510 con 0x7f516c0af1a0 2024-03-06T07:30:53.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.227+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f516003a070 con 0x7f516c0af1a0 2024-03-06T07:30:53.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.228+0000 7f5168c00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f514c06b4d0 0x7f514c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.228+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f51600ab540 con 0x7f516c0af1a0 2024-03-06T07:30:53.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.228+0000 7f516be00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f514c06b4d0 0x7f514c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.229+0000 7f516be00640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f514c06b4d0 0x7f514c06d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5158009560 tx=0x7f5158009190 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.232+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f516007a060 con 0x7f516c0af1a0 2024-03-06T07:30:53.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:53.444+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f516c158250 con 0x7f516c0af1a0 2024-03-06T07:30:54.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.494+0000 7f5168c00640 1 -- 172.21.15.32:0/1000426486 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v42) v1 ==== 97+0+0 (secure 0 0 0) 0x7f516c158250 con 0x7f516c0af1a0 2024-03-06T07:30:54.496 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.497+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f514c06b4d0 msgr2=0x7f514c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.497+0000 7f5173400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f514c06b4d0 0x7f514c06d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5158009560 tx=0x7f5158009190 comp rx=0 tx=0).stop 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.497+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 msgr2=0x7f516c0cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.497+0000 7f5173400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c0cb8c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5160035040 tx=0x7f51600046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 shutdown_connections 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f514c06b4d0 0x7f514c06d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f516c0cdc20 0x7f516c15e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f516c0af580 0x7f516c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 --2- 172.21.15.32:0/1000426486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f516c0af1a0 0x7f516c0cb8c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 >> 172.21.15.32:0/1000426486 conn(0x7f516c06afb0 msgr2=0x7f516c0ad5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 shutdown_connections 2024-03-06T07:30:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.498+0000 7f5173400640 1 -- 172.21.15.32:0/1000426486 wait complete. 2024-03-06T07:30:54.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:54.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.730+0000 7f219ec00640 1 Processor -- start 2024-03-06T07:30:54.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.730+0000 7f219ec00640 1 -- start start 2024-03-06T07:30:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219ec00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21981552b0 0x7f219815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219ec00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21980cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219ec00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219815ed10 con 0x7f21980cce90 2024-03-06T07:30:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219ec00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219815eff0 con 0x7f21981552b0 2024-03-06T07:30:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219ec00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219815f2d0 con 0x7f219815e810 2024-03-06T07:30:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219d800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21981552b0 0x7f219815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219d800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21981552b0 0x7f219815b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54248/0 (socket says 172.21.15.32:54248) 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219d800640 1 -- 172.21.15.32:0/3405956627 learned_addr learned my addr 172.21.15.32:0/3405956627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.732+0000 7f219ce00640 1 --2- 172.21.15.32:0/3405956627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21980cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.733+0000 7f219d800640 1 -- 172.21.15.32:0/3405956627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 msgr2=0x7f21980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.733+0000 7f219d800640 1 --2- 172.21.15.32:0/3405956627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21980cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.733+0000 7f219d800640 1 -- 172.21.15.32:0/3405956627 >> v1:172.21.15.39:6789/0 conn(0x7f21980cce90 legacy=0x7f219806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.733+0000 7f219d800640 1 -- 172.21.15.32:0/3405956627 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219815f5b0 con 0x7f21981552b0 2024-03-06T07:30:54.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.734+0000 7f219d800640 1 --2- 172.21.15.32:0/3405956627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21981552b0 0x7f219815b1d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f218c009870 tx=0x7f218c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9059770b842b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.734+0000 7f2197e00640 1 -- 172.21.15.32:0/3405956627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c0317c0 con 0x7f21981552b0 2024-03-06T07:30:54.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.734+0000 7f2197e00640 1 -- 172.21.15.32:0/3405956627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c037070 con 0x7f21981552b0 2024-03-06T07:30:54.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.734+0000 7f2197e00640 1 -- 172.21.15.32:0/3405956627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c031b80 con 0x7f21981552b0 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 -- 172.21.15.32:0/3405956627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21981552b0 msgr2=0x7f219815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 --2- 172.21.15.32:0/3405956627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21981552b0 0x7f219815b1d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f218c009870 tx=0x7f218c02b860 comp rx=0 tx=0).stop 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 -- 172.21.15.32:0/3405956627 shutdown_connections 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 --2- 172.21.15.32:0/3405956627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21980cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 --2- 172.21.15.32:0/3405956627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21981552b0 0x7f219815b1d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 -- 172.21.15.32:0/3405956627 >> 172.21.15.32:0/3405956627 conn(0x7f219806b3d0 msgr2=0x7f219806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 -- 172.21.15.32:0/3405956627 shutdown_connections 2024-03-06T07:30:54.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.735+0000 7f219ec00640 1 -- 172.21.15.32:0/3405956627 wait complete. 2024-03-06T07:30:54.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.736+0000 7f219ec00640 1 Processor -- start 2024-03-06T07:30:54.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.737+0000 7f219ec00640 1 -- start start 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 0x7f219814a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21981552b0 0x7f2198159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ec00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21981450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219806dd30 con 0x7f21980cce90 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21980b3760 con 0x7f21981552b0 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ec00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219806deb0 con 0x7f219815e810 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 0x7f219814a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 0x7f219814a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54264/0 (socket says 172.21.15.32:54264) 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 -- 172.21.15.32:0/2657309919 learned_addr learned my addr 172.21.15.32:0/2657309919 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ce00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21981552b0 0x7f2198159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 -- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 msgr2=0x7f21981450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21981450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 -- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21981552b0 msgr2=0x7f2198159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21981552b0 0x7f2198159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219d800640 1 -- 172.21.15.32:0/2657309919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219815a150 con 0x7f21980cce90 2024-03-06T07:30:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.738+0000 7f219ce00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21981552b0 0x7f2198159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.739+0000 7f219d800640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 0x7f219814a770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f218c02b8a0 tx=0x7f218c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.739+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c032860 con 0x7f21980cce90 2024-03-06T07:30:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.739+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c032a20 con 0x7f21980cce90 2024-03-06T07:30:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.739+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c032e10 con 0x7f21980cce90 2024-03-06T07:30:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.739+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21981455f0 con 0x7f21980cce90 2024-03-06T07:30:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.739+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21980b14f0 con 0x7f21980cce90 2024-03-06T07:30:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.740+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219804f330 con 0x7f21980cce90 2024-03-06T07:30:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.741+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f218c04c060 con 0x7f21980cce90 2024-03-06T07:30:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.742+0000 7f2195600640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f217806b500 0x7f217806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.742+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f218c0ab7c0 con 0x7f21980cce90 2024-03-06T07:30:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.742+0000 7f219ce00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f217806b500 0x7f217806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.743+0000 7f219ce00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f217806b500 0x7f217806d9c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f219814b480 tx=0x7f2184008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:54.747+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f219815a150 con 0x7f21980cce90 2024-03-06T07:30:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.018+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f21980cf4a0 con 0x7f21980cce90 2024-03-06T07:30:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.019+0000 7f2195600640 1 -- 172.21.15.32:0/2657309919 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f218c040070 con 0x7f21980cce90 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:55.024 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.026 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:30:55.028 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:30:55.030 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:55.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:30:55.034 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:30:55.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:30:55.036 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:30:55.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f217806b500 msgr2=0x7f217806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f217806b500 0x7f217806d9c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f219814b480 tx=0x7f2184008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 msgr2=0x7f219814a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 0x7f219814a770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f218c02b8a0 tx=0x7f218c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 shutdown_connections 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f217806b500 0x7f217806d9c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f219815e810 0x7f21981450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21981552b0 0x7f2198159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 --2- 172.21.15.32:0/2657309919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21980cce90 0x7f219814a770 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.022+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 >> 172.21.15.32:0/2657309919 conn(0x7f219806b3d0 msgr2=0x7f2198071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.023+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 shutdown_connections 2024-03-06T07:30:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.023+0000 7f219ec00640 1 -- 172.21.15.32:0/2657309919 wait complete. 2024-03-06T07:30:55.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:55.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.235+0000 7fd52ae00640 1 Processor -- start 2024-03-06T07:30:55.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.235+0000 7fd52ae00640 1 -- start start 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.237+0000 7fd52ae00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd524070f10 0x7fd5240cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.237+0000 7fd52ae00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5240714d0 con 0x7fd5240cce90 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.237+0000 7fd52ae00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5240717b0 con 0x7fd524145300 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.237+0000 7fd52ae00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5240cd300 con 0x7fd524070f10 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.237+0000 7fd52a400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5240cce90 0x7fd52415c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59816/0 (socket says 172.21.15.32:59816) 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd52a400640 1 -- 172.21.15.32:0/1407923046 learned_addr learned my addr 172.21.15.32:0/1407923046 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd529000640 1 --2- 172.21.15.32:0/1407923046 >> v2:172.21.15.39:3300/0 conn(0x7fd524070f10 0x7fd5240cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656004627 0 0) 0x7fd5240714d0 con 0x7fd5240cce90 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c003520 con 0x7fd5240cce90 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757185592 0 0) 0x7fd50c003520 con 0x7fd5240cce90 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 >> v2:172.21.15.39:3300/0 conn(0x7fd524070f10 msgr2=0x7fd5240cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 --2- 172.21.15.32:0/1407923046 >> v2:172.21.15.39:3300/0 conn(0x7fd524070f10 0x7fd5240cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 >> v1:172.21.15.39:6789/0 conn(0x7fd524145300 legacy=0x7fd5240b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd524071a90 con 0x7fd5240cce90 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd51c0019f0 con 0x7fd5240cce90 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 >> v1:172.21.15.32:6789/0 conn(0x7fd5240cce90 legacy=0x7fd52415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.238+0000 7fd523e00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd50c0039b0 0x7fd50c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd523e00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd523e00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0051c0 0x7fd50c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240cd300 con 0x7fd50c0039b0 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c00c160 con 0x7fd50c004800 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c00c360 con 0x7fd50c0051c0 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529a00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd50c0039b0 0x7fd50c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529000640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529a00640 1 -- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0051c0 msgr2=0x7fd50c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529a00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0051c0 0x7fd50c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529a00640 1 -- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd50c004800 msgr2=0x7fd50c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529a00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd52a400640 1 -- 172.21.15.32:0/1407923046 reap_dead start 2024-03-06T07:30:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529a00640 1 -- 172.21.15.32:0/1407923046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c003520 con 0x7fd50c0039b0 2024-03-06T07:30:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.239+0000 7fd529000640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.240+0000 7fd529a00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd50c0039b0 0x7fd50c003dd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd518001750 tx=0x7fd50c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ee88b4c8eae80d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518001370 con 0x7fd50c0039b0 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5180052d0 con 0x7fd50c0039b0 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd523e00640 1 -- 172.21.15.32:0/1407923046 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518003830 con 0x7fd50c0039b0 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 -- 172.21.15.32:0/1407923046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd50c0039b0 msgr2=0x7fd50c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd50c0039b0 0x7fd50c003dd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd518001750 tx=0x7fd50c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 -- 172.21.15.32:0/1407923046 shutdown_connections 2024-03-06T07:30:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 --2- 172.21.15.32:0/1407923046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd50c0039b0 0x7fd50c003dd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 -- 172.21.15.32:0/1407923046 >> 172.21.15.32:0/1407923046 conn(0x7fd52406b3d0 msgr2=0x7fd52406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 -- 172.21.15.32:0/1407923046 shutdown_connections 2024-03-06T07:30:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.241+0000 7fd52ae00640 1 -- 172.21.15.32:0/1407923046 wait complete. 2024-03-06T07:30:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.243+0000 7fd52ae00640 1 Processor -- start 2024-03-06T07:30:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.243+0000 7fd52ae00640 1 -- start start 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd52ae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0039b0 0x7fd5241f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd52ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 0x7fd524142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd52ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5241f7270 0x7fd524131130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd52ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240cc950 con 0x7fd52415ef30 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd52ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5241456e0 con 0x7fd5241f7270 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd52ae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240ccad0 con 0x7fd50c0039b0 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd529000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 0x7fd524142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd529000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 0x7fd524142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54290/0 (socket says 172.21.15.32:54290) 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.244+0000 7fd529000640 1 -- 172.21.15.32:0/3241423270 learned_addr learned my addr 172.21.15.32:0/3241423270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529a00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0039b0 0x7fd5241f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd52a400640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5241f7270 0x7fd524131130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529000640 1 -- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0039b0 msgr2=0x7fd5241f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529000640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0039b0 0x7fd5241f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529000640 1 -- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5241f7270 msgr2=0x7fd524131130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529000640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5241f7270 0x7fd524131130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529000640 1 -- 172.21.15.32:0/3241423270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd524130c80 con 0x7fd52415ef30 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd529000640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 0x7fd524142dc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd5100107c0 tx=0x7fd510010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.245+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51000f040 con 0x7fd52415ef30 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.246+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51000eb60 con 0x7fd52415ef30 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.246+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5241f42d0 con 0x7fd52415ef30 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.246+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510010e70 con 0x7fd52415ef30 2024-03-06T07:30:55.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.246+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5241f4890 con 0x7fd52415ef30 2024-03-06T07:30:55.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.247+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5240b4bf0 con 0x7fd52415ef30 2024-03-06T07:30:55.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.248+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd51001c400 con 0x7fd52415ef30 2024-03-06T07:30:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.248+0000 7fd521600640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd50406b500 0x7fd50406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.249+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd51008a2e0 con 0x7fd52415ef30 2024-03-06T07:30:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.249+0000 7fd529a00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd50406b500 0x7fd50406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.250+0000 7fd529a00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd50406b500 0x7fd50406d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd518033000 tx=0x7fd518009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.253+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5240b4bf0 con 0x7fd52415ef30 2024-03-06T07:30:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.448+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd52404f330 con 0x7fd52415ef30 2024-03-06T07:30:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.450+0000 7fd521600640 1 -- 172.21.15.32:0/3241423270 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12783 (secure 0 0 0) 0x7fd510018030 con 0x7fd52415ef30 2024-03-06T07:30:55.453 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:55.453 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:54.491645+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd50406b500 msgr2=0x7fd50406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd50406b500 0x7fd50406d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd518033000 tx=0x7fd518009030 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 msgr2=0x7fd524142dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 0x7fd524142dc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd5100107c0 tx=0x7fd510010c80 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 shutdown_connections 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd50406b500 0x7fd50406d9c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5241f7270 0x7fd524131130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd52415ef30 0x7fd524142dc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 --2- 172.21.15.32:0/3241423270 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd50c0039b0 0x7fd5241f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.452+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 >> 172.21.15.32:0/3241423270 conn(0x7fd52406b3d0 msgr2=0x7fd5240cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.453+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 shutdown_connections 2024-03-06T07:30:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.453+0000 7fd52ae00640 1 -- 172.21.15.32:0/3241423270 wait complete. 2024-03-06T07:30:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:55.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.672+0000 7f3671400640 1 Processor -- start 2024-03-06T07:30:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.672+0000 7f3671400640 1 -- start start 2024-03-06T07:30:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3671400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f366c0b3c20 0x7f366c0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3671400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f366c0b4000 0x7f366c15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3671400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c071930 con 0x7f366c0b1b40 2024-03-06T07:30:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3671400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c0b4460 con 0x7f366c0b4000 2024-03-06T07:30:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3671400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c071c10 con 0x7f366c0b3c20 2024-03-06T07:30:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3670a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f366c0b1b40 0x7f366c15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59818/0 (socket says 172.21.15.32:59818) 2024-03-06T07:30:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.674+0000 7f3670a00640 1 -- 172.21.15.32:0/264265390 learned_addr learned my addr 172.21.15.32:0/264265390 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366be00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f366c0b3c20 0x7f366c0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366b400640 1 --2- 172.21.15.32:0/264265390 >> v2:172.21.15.39:3300/0 conn(0x7f366c0b4000 0x7f366c15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168682458 0 0) 0x7f366c071930 con 0x7f366c0b1b40 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3650003520 con 0x7f366c0b1b40 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2715726458 0 0) 0x7f3650003520 con 0x7f366c0b1b40 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f366c0b3c20 msgr2=0x7f366c0729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f366c0b3c20 0x7f366c0729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 >> v2:172.21.15.39:3300/0 conn(0x7f366c0b4000 msgr2=0x7f366c15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 --2- 172.21.15.32:0/264265390 >> v2:172.21.15.39:3300/0 conn(0x7f366c0b4000 0x7f366c15b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c071ef0 con 0x7f366c0b1b40 2024-03-06T07:30:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f36640019f0 con 0x7f366c0b1b40 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 >> v1:172.21.15.32:6789/0 conn(0x7f366c0b1b40 legacy=0x7f366c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3650004800 0x7f3650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36500051c0 0x7f365000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b4460 con 0x7f3650004800 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c071c10 con 0x7f36500039b0 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365000c030 con 0x7f36500051c0 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366b400640 1 --2- 172.21.15.32:0/264265390 >> v2:172.21.15.39:3300/0 conn(0x7f366c0b4000 0x7f366c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.675+0000 7f366be00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f366c0b3c20 0x7f366c0729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366b400640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3650004800 0x7f3650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f3670a00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36500051c0 0x7f365000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366be00640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366b400640 1 -- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36500051c0 msgr2=0x7f365000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366b400640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36500051c0 0x7f365000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366b400640 1 -- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36500039b0 msgr2=0x7f3650003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366b400640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.676+0000 7f366b400640 1 -- 172.21.15.32:0/264265390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3650003520 con 0x7f3650004800 2024-03-06T07:30:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.677+0000 7f366b400640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3650004800 0x7f3650004c80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f36580098a0 tx=0x7f3650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e0ae91ee4887d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800f040 con 0x7f3650004800 2024-03-06T07:30:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658037600 con 0x7f3650004800 2024-03-06T07:30:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f366aa00640 1 -- 172.21.15.32:0/264265390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36580379b0 con 0x7f3650004800 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3671400640 1 -- 172.21.15.32:0/264265390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3650004800 msgr2=0x7f3650004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3671400640 1 --2- 172.21.15.32:0/264265390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3650004800 0x7f3650004c80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f36580098a0 tx=0x7f3650005830 comp rx=0 tx=0).stop 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3670a00640 1 -- 172.21.15.32:0/264265390 reap_dead start 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3671400640 1 -- 172.21.15.32:0/264265390 shutdown_connections 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3671400640 1 -- 172.21.15.32:0/264265390 >> 172.21.15.32:0/264265390 conn(0x7f366c06b3a0 msgr2=0x7f366c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3671400640 1 -- 172.21.15.32:0/264265390 shutdown_connections 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.678+0000 7f3671400640 1 -- 172.21.15.32:0/264265390 wait complete. 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.680+0000 7f3671400640 1 Processor -- start 2024-03-06T07:30:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.680+0000 7f3671400640 1 -- start start 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f3671400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3650004800 0x7f366c15ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f3671400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 0x7f366c151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f3671400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f366c0ce2f0 0x7f366c0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f3671400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c15bde0 con 0x7f366c1f39e0 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f3671400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0724b0 con 0x7f366c0ce2f0 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f3671400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c15bf60 con 0x7f3650004800 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.681+0000 7f366b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 0x7f366c151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 0x7f366c151370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54320/0 (socket says 172.21.15.32:54320) 2024-03-06T07:30:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 -- 172.21.15.32:0/3483893184 learned_addr learned my addr 172.21.15.32:0/3483893184 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366be00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3650004800 0x7f366c15ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f3670a00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f366c0ce2f0 0x7f366c0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 -- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3650004800 msgr2=0x7f366c15ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3650004800 0x7f366c15ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 -- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f366c0ce2f0 msgr2=0x7f366c0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f366c0ce2f0 0x7f366c0ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 -- 172.21.15.32:0/3483893184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c157740 con 0x7f366c1f39e0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f3670a00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f366c0ce2f0 0x7f366c0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366be00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3650004800 0x7f366c15ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.682+0000 7f366b400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 0x7f366c151370 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3658009990 tx=0x7f3658004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.683+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800f000 con 0x7f366c1f39e0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.683+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365802f9e0 con 0x7f366c1f39e0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.683+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365802fdd0 con 0x7f366c1f39e0 2024-03-06T07:30:55.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.683+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c1f3f30 con 0x7f366c1f39e0 2024-03-06T07:30:55.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.683+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c1f44f0 con 0x7f366c1f39e0 2024-03-06T07:30:55.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.684+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c04f330 con 0x7f366c1f39e0 2024-03-06T07:30:55.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.685+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3658042440 con 0x7f366c1f39e0 2024-03-06T07:30:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.686+0000 7f3663e00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f364c06b500 0x7f364c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.686+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f36580ac720 con 0x7f366c1f39e0 2024-03-06T07:30:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.686+0000 7f366be00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f364c06b500 0x7f364c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.687+0000 7f366be00640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f364c06b500 0x7f364c06d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f365c006fd0 tx=0x7f365c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.690+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3658033030 con 0x7f366c1f39e0 2024-03-06T07:30:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.889+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f366c072b20 con 0x7f366c1f39e0 2024-03-06T07:30:55.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.890+0000 7f3663e00640 1 -- 172.21.15.32:0/3483893184 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f365807b7c0 con 0x7f366c1f39e0 2024-03-06T07:30:55.892 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.892+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f364c06b500 msgr2=0x7f364c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.892+0000 7f3671400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f364c06b500 0x7f364c06d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f365c006fd0 tx=0x7f365c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.892+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 msgr2=0x7f366c151370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.892+0000 7f3671400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 0x7f366c151370 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3658009990 tx=0x7f3658004730 comp rx=0 tx=0).stop 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 shutdown_connections 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f364c06b500 0x7f364c06d9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f366c0ce2f0 0x7f366c0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f366c1f39e0 0x7f366c151370 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 --2- 172.21.15.32:0/3483893184 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3650004800 0x7f366c15ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 >> 172.21.15.32:0/3483893184 conn(0x7f366c06b3a0 msgr2=0x7f366c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 shutdown_connections 2024-03-06T07:30:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:55.893+0000 7f3671400640 1 -- 172.21.15.32:0/3483893184 wait complete. 2024-03-06T07:30:55.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:56.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.125+0000 7ff734000640 1 Processor -- start 2024-03-06T07:30:56.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.125+0000 7ff734000640 1 -- start start 2024-03-06T07:30:56.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff734000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff72c0cf5c0 0x7ff72c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff734000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c071050 0x7ff72c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff734000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0b4860 con 0x7ff72c0b3c90 2024-03-06T07:30:56.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff734000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c0ce160 con 0x7ff72c0cf5c0 2024-03-06T07:30:56.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff734000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0b4b40 con 0x7ff72c071050 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff730c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff72c0cf5c0 0x7ff72c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff730c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff72c0cf5c0 0x7ff72c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54336/0 (socket says 172.21.15.32:54336) 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff730c00640 1 -- 172.21.15.32:0/2257268521 learned_addr learned my addr 172.21.15.32:0/2257268521 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.127+0000 7ff732000640 1 --2- 172.21.15.32:0/2257268521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c071050 0x7ff72c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.128+0000 7ff730c00640 1 -- 172.21.15.32:0/2257268521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c071050 msgr2=0x7ff72c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.128+0000 7ff730c00640 1 --2- 172.21.15.32:0/2257268521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c071050 0x7ff72c071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.128+0000 7ff730c00640 1 -- 172.21.15.32:0/2257268521 >> v1:172.21.15.39:6789/0 conn(0x7ff72c0b3c90 legacy=0x7ff72c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.128+0000 7ff730c00640 1 -- 172.21.15.32:0/2257268521 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c0b4e20 con 0x7ff72c0cf5c0 2024-03-06T07:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.129+0000 7ff730c00640 1 --2- 172.21.15.32:0/2257268521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c0cdc20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff718009870 tx=0x7ff71802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a0c2c42226ab78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.129+0000 7ff72be00640 1 -- 172.21.15.32:0/2257268521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718034070 con 0x7ff72c0cf5c0 2024-03-06T07:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.129+0000 7ff72be00640 1 -- 172.21.15.32:0/2257268521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff718030840 con 0x7ff72c0cf5c0 2024-03-06T07:30:56.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.129+0000 7ff72be00640 1 -- 172.21.15.32:0/2257268521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718030c10 con 0x7ff72c0cf5c0 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 -- 172.21.15.32:0/2257268521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0cf5c0 msgr2=0x7ff72c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 --2- 172.21.15.32:0/2257268521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c0cdc20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff718009870 tx=0x7ff71802b980 comp rx=0 tx=0).stop 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 -- 172.21.15.32:0/2257268521 shutdown_connections 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 --2- 172.21.15.32:0/2257268521 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c071050 0x7ff72c071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 --2- 172.21.15.32:0/2257268521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c0cdc20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 -- 172.21.15.32:0/2257268521 >> 172.21.15.32:0/2257268521 conn(0x7ff72c06afb0 msgr2=0x7ff72c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 -- 172.21.15.32:0/2257268521 shutdown_connections 2024-03-06T07:30:56.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.130+0000 7ff734000640 1 -- 172.21.15.32:0/2257268521 wait complete. 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.132+0000 7ff734000640 1 Processor -- start 2024-03-06T07:30:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.132+0000 7ff734000640 1 -- start start 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff734000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 0x7ff72c153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff734000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c0b3c90 0x7ff72c141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff734000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff734000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0719d0 con 0x7ff72c071050 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff734000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c15b340 con 0x7ff72c0cf5c0 2024-03-06T07:30:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff734000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c071b50 con 0x7ff72c0b3c90 2024-03-06T07:30:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 0x7ff72c153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 0x7ff72c153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54350/0 (socket says 172.21.15.32:54350) 2024-03-06T07:30:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 -- 172.21.15.32:0/3772766480 learned_addr learned my addr 172.21.15.32:0/3772766480 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff730c00640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c0b3c90 0x7ff72c141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff732000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 -- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c0b3c90 msgr2=0x7ff72c141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c0b3c90 0x7ff72c141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 -- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c0cf5c0 msgr2=0x7ff72c142110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c142110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.133+0000 7ff731600640 1 -- 172.21.15.32:0/3772766480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c09c8d0 con 0x7ff72c071050 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.134+0000 7ff731600640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 0x7ff72c153220 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff72000c430 tx=0x7ff72000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.134+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720011790 con 0x7ff72c071050 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.134+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff720017070 con 0x7ff72c071050 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.134+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720011bb0 con 0x7ff72c071050 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.134+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c09cbb0 con 0x7ff72c071050 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.134+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c09d170 con 0x7ff72c071050 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.135+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c09d550 con 0x7ff72c071050 2024-03-06T07:30:56.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.136+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff72001c460 con 0x7ff72c071050 2024-03-06T07:30:56.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.137+0000 7ff729600640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff70c06b4d0 0x7ff70c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.138+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff720089d70 con 0x7ff72c071050 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.138+0000 7ff730c00640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff70c06b4d0 0x7ff70c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.138+0000 7ff730c00640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff70c06b4d0 0x7ff70c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff718036000 tx=0x7ff718008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.142+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff72c09d550 con 0x7ff72c071050 2024-03-06T07:30:56.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.343+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff72c1595e0 con 0x7ff72c071050 2024-03-06T07:30:56.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.344+0000 7ff729600640 1 -- 172.21.15.32:0/3772766480 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff720058f30 con 0x7ff72c071050 2024-03-06T07:30:56.346 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:30:56.346 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:30:56.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:30:56.347 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:30:56.347 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:30:56.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:30:56.347 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.346+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff70c06b4d0 msgr2=0x7ff70c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.346+0000 7ff734000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff70c06b4d0 0x7ff70c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff718036000 tx=0x7ff718008030 comp rx=0 tx=0).stop 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.346+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 msgr2=0x7ff72c153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.346+0000 7ff734000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 0x7ff72c153220 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff72000c430 tx=0x7ff72000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 shutdown_connections 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff70c06b4d0 0x7ff70c06d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c0cf5c0 0x7ff72c142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff72c0b3c90 0x7ff72c141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 --2- 172.21.15.32:0/3772766480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c071050 0x7ff72c153220 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 >> 172.21.15.32:0/3772766480 conn(0x7ff72c06afb0 msgr2=0x7ff72c0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 shutdown_connections 2024-03-06T07:30:56.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.347+0000 7ff734000640 1 -- 172.21.15.32:0/3772766480 wait complete. 2024-03-06T07:30:56.369 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:56.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.600+0000 7f9d76200640 1 Processor -- start 2024-03-06T07:30:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.600+0000 7f9d76200640 1 -- start start 2024-03-06T07:30:56.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d76200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d70145570 0x7f9d700cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d76200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9d7006e200 0x7f9d7015c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d76200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d70071c50 con 0x7f9d70071450 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d76200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d70071f30 con 0x7f9d7006e200 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d76200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d70072210 con 0x7f9d70145570 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d6fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9d70071450 0x7f9d7006daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59820/0 (socket says 172.21.15.32:59820) 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d6fe00640 1 -- 172.21.15.32:0/2698672719 learned_addr learned my addr 172.21.15.32:0/2698672719 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d74e00640 1 --2- 172.21.15.32:0/2698672719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d70145570 0x7f9d700cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.602+0000 7f9d75800640 1 --2- 172.21.15.32:0/2698672719 >> v2:172.21.15.39:3300/0 conn(0x7f9d7006e200 0x7f9d7015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.603+0000 7f9d6f400640 1 -- 172.21.15.32:0/2698672719 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372138279 0 0) 0x7f9d70071c50 con 0x7f9d70071450 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.603+0000 7f9d6f400640 1 -- 172.21.15.32:0/2698672719 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d58003520 con 0x7f9d70071450 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.603+0000 7f9d74e00640 1 -- 172.21.15.32:0/2698672719 >> v2:172.21.15.39:3300/0 conn(0x7f9d7006e200 msgr2=0x7f9d7015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.603+0000 7f9d74e00640 1 --2- 172.21.15.32:0/2698672719 >> v2:172.21.15.39:3300/0 conn(0x7f9d7006e200 0x7f9d7015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.603+0000 7f9d74e00640 1 -- 172.21.15.32:0/2698672719 >> v1:172.21.15.32:6789/0 conn(0x7f9d70071450 legacy=0x7f9d7006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.603+0000 7f9d74e00640 1 -- 172.21.15.32:0/2698672719 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700724f0 con 0x7f9d70145570 2024-03-06T07:30:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.605+0000 7f9d74e00640 1 --2- 172.21.15.32:0/2698672719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d70145570 0x7f9d700cf420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d640097a0 tx=0x7f9d6402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5452171ee190e90d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d6f400640 1 -- 172.21.15.32:0/2698672719 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64031780 con 0x7f9d70145570 2024-03-06T07:30:56.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d6f400640 1 -- 172.21.15.32:0/2698672719 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64037070 con 0x7f9d70145570 2024-03-06T07:30:56.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d6f400640 1 -- 172.21.15.32:0/2698672719 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64031b20 con 0x7f9d70145570 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d76200640 1 -- 172.21.15.32:0/2698672719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d70145570 msgr2=0x7f9d700cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d76200640 1 --2- 172.21.15.32:0/2698672719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d70145570 0x7f9d700cf420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d640097a0 tx=0x7f9d6402b820 comp rx=0 tx=0).stop 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d76200640 1 -- 172.21.15.32:0/2698672719 shutdown_connections 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d76200640 1 --2- 172.21.15.32:0/2698672719 >> v2:172.21.15.39:3300/0 conn(0x7f9d7006e200 0x7f9d7015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d76200640 1 --2- 172.21.15.32:0/2698672719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d70145570 0x7f9d700cf420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.606+0000 7f9d76200640 1 -- 172.21.15.32:0/2698672719 >> 172.21.15.32:0/2698672719 conn(0x7f9d700b17c0 msgr2=0x7f9d700b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.607+0000 7f9d76200640 1 -- 172.21.15.32:0/2698672719 shutdown_connections 2024-03-06T07:30:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.607+0000 7f9d76200640 1 -- 172.21.15.32:0/2698672719 wait complete. 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.608+0000 7f9d76200640 1 Processor -- start 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.608+0000 7f9d76200640 1 -- start start 2024-03-06T07:30:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.609+0000 7f9d76200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d7006e200 0x7f9d700cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.609+0000 7f9d76200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d70071450 0x7f9d700c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.609+0000 7f9d76200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 0x7f9d700c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.609+0000 7f9d76200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d70071a30 con 0x7f9d70145570 2024-03-06T07:30:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.609+0000 7f9d76200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700718b0 con 0x7f9d70071450 2024-03-06T07:30:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.609+0000 7f9d76200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7015c840 con 0x7f9d7006e200 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 0x7f9d700c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 0x7f9d700c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54354/0 (socket says 172.21.15.32:54354) 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 -- 172.21.15.32:0/10440071 learned_addr learned my addr 172.21.15.32:0/10440071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 -- 172.21.15.32:0/10440071 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d7006e200 msgr2=0x7f9d700cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d7006e200 0x7f9d700cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 -- 172.21.15.32:0/10440071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d70071450 msgr2=0x7f9d700c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d70071450 0x7f9d700c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 -- 172.21.15.32:0/10440071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700afbe0 con 0x7f9d70145570 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.610+0000 7f9d75800640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 0x7f9d700c8110 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800efc0 tx=0x7f9d6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.611+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68018020 con 0x7f9d70145570 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.611+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d68014480 con 0x7f9d70145570 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.611+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68014870 con 0x7f9d70145570 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.611+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d700afec0 con 0x7f9d70145570 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.611+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d700b0480 con 0x7f9d70145570 2024-03-06T07:30:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.612+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d700b0860 con 0x7f9d70145570 2024-03-06T07:30:56.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.613+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d6801b020 con 0x7f9d70145570 2024-03-06T07:30:56.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.614+0000 7f9d6cc00640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d5006b500 0x7f9d5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.615+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9d6808adb0 con 0x7f9d70145570 2024-03-06T07:30:56.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.615+0000 7f9d74e00640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d5006b500 0x7f9d5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.615+0000 7f9d74e00640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d5006b500 0x7f9d5006d9c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d64036000 tx=0x7f9d64030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.619+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d700b0860 con 0x7f9d70145570 2024-03-06T07:30:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:56.818+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9d7015f470 con 0x7f9d70145570 2024-03-06T07:30:57.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.539+0000 7f9d6cc00640 1 -- 172.21.15.32:0/10440071 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9d6808f070 con 0x7f9d70145570 2024-03-06T07:30:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:57.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.542+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d5006b500 msgr2=0x7f9d5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.542+0000 7f9d76200640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d5006b500 0x7f9d5006d9c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d64036000 tx=0x7f9d64030000 comp rx=0 tx=0).stop 2024-03-06T07:30:57.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 msgr2=0x7f9d700c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 0x7f9d700c8110 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800efc0 tx=0x7f9d6800c490 comp rx=0 tx=0).stop 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 shutdown_connections 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d5006b500 0x7f9d5006d9c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d70145570 0x7f9d700c8110 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d70071450 0x7f9d700c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 --2- 172.21.15.32:0/10440071 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d7006e200 0x7f9d700cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 >> 172.21.15.32:0/10440071 conn(0x7f9d700b17c0 msgr2=0x7f9d700b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 shutdown_connections 2024-03-06T07:30:57.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.543+0000 7f9d76200640 1 -- 172.21.15.32:0/10440071 wait complete. 2024-03-06T07:30:57.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.779+0000 7f0b82a00640 1 Processor -- start 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.779+0000 7f0b82a00640 1 -- start start 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b82a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b7c144340 0x7f0b7c0ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b82a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0b2b80 con 0x7f0b7c0cee10 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b82a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0b2e60 con 0x7f0b7c0b4d10 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b82a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0cc780 con 0x7f0b7c144340 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b81600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b7c144340 0x7f0b7c0ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b80c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0b7c0cee10 0x7f0b7c0b4600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59836/0 (socket says 172.21.15.32:59836) 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.781+0000 7f0b80c00640 1 -- 172.21.15.32:0/266754560 learned_addr learned my addr 172.21.15.32:0/266754560 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.782+0000 7f0b81600640 1 -- 172.21.15.32:0/266754560 >> v1:172.21.15.39:6789/0 conn(0x7f0b7c0b4d10 legacy=0x7f0b7c070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.782+0000 7f0b81600640 1 -- 172.21.15.32:0/266754560 >> v1:172.21.15.32:6789/0 conn(0x7f0b7c0cee10 legacy=0x7f0b7c0b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.782+0000 7f0b81600640 1 -- 172.21.15.32:0/266754560 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0b3140 con 0x7f0b7c144340 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.783+0000 7f0b81600640 1 --2- 172.21.15.32:0/266754560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c144340 0x7f0b7c0ce8d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009d90 tx=0x7f0b7002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167c233805e82ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b7be00640 1 -- 172.21.15.32:0/266754560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7003b070 con 0x7f0b7c144340 2024-03-06T07:30:57.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b7be00640 1 -- 172.21.15.32:0/266754560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7002ede0 con 0x7f0b7c144340 2024-03-06T07:30:57.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b7be00640 1 -- 172.21.15.32:0/266754560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70036660 con 0x7f0b7c144340 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b82a00640 1 -- 172.21.15.32:0/266754560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c144340 msgr2=0x7f0b7c0ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b82a00640 1 --2- 172.21.15.32:0/266754560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c144340 0x7f0b7c0ce8d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009d90 tx=0x7f0b7002e030 comp rx=0 tx=0).stop 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b82a00640 1 -- 172.21.15.32:0/266754560 shutdown_connections 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b82a00640 1 --2- 172.21.15.32:0/266754560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c144340 0x7f0b7c0ce8d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.784+0000 7f0b82a00640 1 -- 172.21.15.32:0/266754560 >> 172.21.15.32:0/266754560 conn(0x7f0b7c06b190 msgr2=0x7f0b7c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.785+0000 7f0b82a00640 1 -- 172.21.15.32:0/266754560 shutdown_connections 2024-03-06T07:30:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.785+0000 7f0b82a00640 1 -- 172.21.15.32:0/266754560 wait complete. 2024-03-06T07:30:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.787+0000 7f0b82a00640 1 Processor -- start 2024-03-06T07:30:57.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.787+0000 7f0b82a00640 1 -- start start 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0b7c0b4d10 0x7f0b7c1393b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 0x7f0b7c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b7c144340 0x7f0b7c06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c144720 con 0x7f0b7c0cee10 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0b3790 con 0x7f0b7c144340 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0b3c20 con 0x7f0b7c0b4d10 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b80c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 0x7f0b7c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b80c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 0x7f0b7c15a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54374/0 (socket says 172.21.15.32:54374) 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b80c00640 1 -- 172.21.15.32:0/929699655 learned_addr learned my addr 172.21.15.32:0/929699655 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b81600640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0b7c0b4d10 0x7f0b7c1393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.788+0000 7f0b82000640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b7c144340 0x7f0b7c06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b80c00640 1 -- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0b7c0b4d10 msgr2=0x7f0b7c1393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b80c00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0b7c0b4d10 0x7f0b7c1393b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b80c00640 1 -- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b7c144340 msgr2=0x7f0b7c06d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b80c00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b7c144340 0x7f0b7c06d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b80c00640 1 -- 172.21.15.32:0/929699655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c15b030 con 0x7f0b7c0cee10 2024-03-06T07:30:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b82000640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b7c144340 0x7f0b7c06d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:57.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b80c00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 0x7f0b7c15a1b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0b6800c400 tx=0x7f0b6800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68017070 con 0x7f0b7c0cee10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b68005ce0 con 0x7f0b7c0cee10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680072f0 con 0x7f0b7c0cee10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c15a6f0 con 0x7f0b7c0cee10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.789+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c15ddc0 con 0x7f0b7c0cee10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.791+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c04f330 con 0x7f0b7c0cee10 2024-03-06T07:30:57.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.792+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b6801a070 con 0x7f0b7c0cee10 2024-03-06T07:30:57.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.793+0000 7f0b79600640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0b5c06b4d0 0x7f0b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.793+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f0b680894d0 con 0x7f0b7c0cee10 2024-03-06T07:30:57.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.793+0000 7f0b81600640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0b5c06b4d0 0x7f0b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.794+0000 7f0b81600640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0b5c06b4d0 0x7f0b5c06d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0b70038000 tx=0x7f0b7000d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:57.799+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b68058490 con 0x7f0b7c0cee10 2024-03-06T07:30:58.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.005+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0b7c0cd5d0 con 0x7f0b7c0cee10 2024-03-06T07:30:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.545+0000 7f0b79600640 1 -- 172.21.15.32:0/929699655 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0b68057d90 con 0x7f0b7c0cee10 2024-03-06T07:30:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:30:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0b5c06b4d0 msgr2=0x7f0b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0b5c06b4d0 0x7f0b5c06d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0b70038000 tx=0x7f0b7000d040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 msgr2=0x7f0b7c15a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 0x7f0b7c15a1b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0b6800c400 tx=0x7f0b6800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 shutdown_connections 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0b5c06b4d0 0x7f0b5c06d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b7c144340 0x7f0b7c06d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b7c0cee10 0x7f0b7c15a1b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 --2- 172.21.15.32:0/929699655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0b7c0b4d10 0x7f0b7c1393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 >> 172.21.15.32:0/929699655 conn(0x7f0b7c06b190 msgr2=0x7f0b7c141810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 shutdown_connections 2024-03-06T07:30:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.549+0000 7f0b82a00640 1 -- 172.21.15.32:0/929699655 wait complete. 2024-03-06T07:30:58.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:58.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.775+0000 7ffb6e400640 1 Processor -- start 2024-03-06T07:30:58.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.775+0000 7ffb6e400640 1 -- start start 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.776+0000 7ffb6e400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb680cfdf0 0x7ffb680d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.776+0000 7ffb6e400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680ccc50 con 0x7ffb680d07d0 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.776+0000 7ffb6e400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680ccf30 con 0x7ffb6815c0e0 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.776+0000 7ffb6e400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6815c4c0 con 0x7ffb680cfdf0 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.776+0000 7ffb66a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb680cfdf0 0x7ffb680d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.776+0000 7ffb66a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb680cfdf0 0x7ffb680d0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46954/0 (socket says 172.21.15.32:46954) 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.777+0000 7ffb66a00640 1 -- 172.21.15.32:0/3930223559 learned_addr learned my addr 172.21.15.32:0/3930223559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.777+0000 7ffb66a00640 1 -- 172.21.15.32:0/3930223559 >> v1:172.21.15.39:6789/0 conn(0x7ffb6815c0e0 legacy=0x7ffb680cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.777+0000 7ffb66a00640 1 -- 172.21.15.32:0/3930223559 >> v1:172.21.15.32:6789/0 conn(0x7ffb680d07d0 legacy=0x7ffb680b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.777+0000 7ffb66a00640 1 -- 172.21.15.32:0/3930223559 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680cd210 con 0x7ffb680cfdf0 2024-03-06T07:30:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.778+0000 7ffb66a00640 1 --2- 172.21.15.32:0/3930223559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb680cfdf0 0x7ffb680d0290 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009e50 tx=0x7ffb5802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4b6a19cf1b8de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb66000640 1 -- 172.21.15.32:0/3930223559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5802cbb0 con 0x7ffb680cfdf0 2024-03-06T07:30:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb66000640 1 -- 172.21.15.32:0/3930223559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5802cd50 con 0x7ffb680cfdf0 2024-03-06T07:30:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb66000640 1 -- 172.21.15.32:0/3930223559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58034610 con 0x7ffb680cfdf0 2024-03-06T07:30:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 -- 172.21.15.32:0/3930223559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb680cfdf0 msgr2=0x7ffb680d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3930223559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb680cfdf0 0x7ffb680d0290 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009e50 tx=0x7ffb5802c010 comp rx=0 tx=0).stop 2024-03-06T07:30:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 -- 172.21.15.32:0/3930223559 shutdown_connections 2024-03-06T07:30:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3930223559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb680cfdf0 0x7ffb680d0290 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 -- 172.21.15.32:0/3930223559 >> 172.21.15.32:0/3930223559 conn(0x7ffb680b3820 msgr2=0x7ffb680b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 -- 172.21.15.32:0/3930223559 shutdown_connections 2024-03-06T07:30:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.779+0000 7ffb6e400640 1 -- 172.21.15.32:0/3930223559 wait complete. 2024-03-06T07:30:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.781+0000 7ffb6e400640 1 Processor -- start 2024-03-06T07:30:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.781+0000 7ffb6e400640 1 -- start start 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.782+0000 7ffb6e400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb680cfdf0 0x7ffb680c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.782+0000 7ffb6e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb680d07d0 0x7ffb681568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.782+0000 7ffb6e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 0x7ffb681f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.782+0000 7ffb6e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cd580 con 0x7ffb6815c0e0 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.782+0000 7ffb6e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cf630 con 0x7ffb680d07d0 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.782+0000 7ffb6e400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cda10 con 0x7ffb680cfdf0 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 0x7ffb681f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 0x7ffb681f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46970/0 (socket says 172.21.15.32:46970) 2024-03-06T07:30:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 -- 172.21.15.32:0/3236300878 learned_addr learned my addr 172.21.15.32:0/3236300878 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb66a00640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb680d07d0 0x7ffb681568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 -- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb680cfdf0 msgr2=0x7ffb680c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb680cfdf0 0x7ffb680c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb680cfdf0 0x7ffb680c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 -- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb680d07d0 msgr2=0x7ffb681568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb680d07d0 0x7ffb681568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 -- 172.21.15.32:0/3236300878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680cb940 con 0x7ffb6815c0e0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.783+0000 7ffb67e00640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 0x7ffb681f7cc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c3c0 tx=0x7ffb6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.784+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb600052d0 con 0x7ffb6815c0e0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.784+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb60005490 con 0x7ffb6815c0e0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.784+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60007260 con 0x7ffb6815c0e0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.784+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb680cbc20 con 0x7ffb6815c0e0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.784+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb680b4e40 con 0x7ffb6815c0e0 2024-03-06T07:30:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.785+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6804f330 con 0x7ffb6815c0e0 2024-03-06T07:30:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.786+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb60007400 con 0x7ffb6815c0e0 2024-03-06T07:30:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.787+0000 7ffb5f400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.787+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ffb600895c0 con 0x7ffb6815c0e0 2024-03-06T07:30:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.787+0000 7ffb67400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.788+0000 7ffb67400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000a750 tx=0x7ffb50008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:58.793+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb600584a0 con 0x7ffb6815c0e0 2024-03-06T07:30:59.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.033+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ffb680b79d0 con 0x7ffb6815c0e0 2024-03-06T07:30:59.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.593+0000 7ffb5f400640 1 -- 172.21.15.32:0/3236300878 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7ffb60057da0 con 0x7ffb6815c0e0 2024-03-06T07:30:59.598 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:30:59.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.598+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 msgr2=0x7ffb4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.598+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000a750 tx=0x7ffb50008040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.598+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 msgr2=0x7ffb681f7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.598+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 0x7ffb681f7cc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c3c0 tx=0x7ffb6000c880 comp rx=0 tx=0).stop 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.599+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 shutdown_connections 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.599+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.599+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb6815c0e0 0x7ffb681f7cc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.599+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb680d07d0 0x7ffb681568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.599+0000 7ffb6e400640 1 --2- 172.21.15.32:0/3236300878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb680cfdf0 0x7ffb680c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.599+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 >> 172.21.15.32:0/3236300878 conn(0x7ffb680b3820 msgr2=0x7ffb680d1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.600+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 shutdown_connections 2024-03-06T07:30:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.600+0000 7ffb6e400640 1 -- 172.21.15.32:0/3236300878 wait complete. 2024-03-06T07:30:59.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:59.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.819+0000 7f07dd600640 1 Processor -- start 2024-03-06T07:30:59.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.819+0000 7f07dd600640 1 -- start start 2024-03-06T07:30:59.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.820+0000 7f07dd600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f07d80af1a0 0x7f07d815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.820+0000 7f07dd600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 0x7f07d80b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.820+0000 7f07dd600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d815cc40 con 0x7f07d80cdc20 2024-03-06T07:30:59.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.820+0000 7f07dd600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d815cf20 con 0x7f07d80af1a0 2024-03-06T07:30:59.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.820+0000 7f07dd600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d815d200 con 0x7f07d80af580 2024-03-06T07:30:59.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.820+0000 7f07d7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f07d80cdc20 0x7f07d80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38332/0 (socket says 172.21.15.32:38332) 2024-03-06T07:30:59.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d7400640 1 -- 172.21.15.32:0/2969422490 learned_addr learned my addr 172.21.15.32:0/2969422490 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6a00640 1 --2- 172.21.15.32:0/2969422490 >> v2:172.21.15.39:3300/0 conn(0x7f07d80af1a0 0x7f07d815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07ce000640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 0x7f07d80b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906690554 0 0) 0x7f07d815cc40 con 0x7f07d80cdc20 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07bc003520 con 0x7f07d80cdc20 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3630465971 0 0) 0x7f07bc003520 con 0x7f07d80cdc20 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 msgr2=0x7f07d80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 0x7f07d80b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 >> v2:172.21.15.39:3300/0 conn(0x7f07d80af1a0 msgr2=0x7f07d815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 --2- 172.21.15.32:0/2969422490 >> v2:172.21.15.39:3300/0 conn(0x7f07d80af1a0 0x7f07d815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d815d4e0 con 0x7f07d80cdc20 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f07d00019f0 con 0x7f07d80cdc20 2024-03-06T07:30:59.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.821+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 >> v1:172.21.15.32:6789/0 conn(0x7f07d80cdc20 legacy=0x7f07d80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6000640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07bc003a20 0x7f07bc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6000640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 0x7f07bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6000640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07bc004d40 0x7f07bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d815cf20 con 0x7f07bc004d40 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d815d200 con 0x7f07bc003a20 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc00ba40 con 0x7f07d80af580 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d7400640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07bc004d40 0x7f07bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6a00640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07bc003a20 0x7f07bc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07ce000640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 0x7f07bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d7400640 1 -- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 msgr2=0x7f07bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d7400640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07d80af580 0x7f07bc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d7400640 1 -- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07bc003a20 msgr2=0x7f07bc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d7400640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07bc003a20 0x7f07bc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d7400640 1 -- 172.21.15.32:0/2969422490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc003520 con 0x7f07bc004d40 2024-03-06T07:30:59.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.822+0000 7f07d6a00640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07bc003a20 0x7f07bc003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07d7400640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07bc004d40 0x7f07bc0051c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f07d000a8a0 tx=0x7f07bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5dc20cc166c964a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d000a4c0 con 0x7f07bc004d40 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0034650 con 0x7f07bc004d40 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07d6000640 1 -- 172.21.15.32:0/2969422490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0034a40 con 0x7f07bc004d40 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07dd600640 1 -- 172.21.15.32:0/2969422490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07bc004d40 msgr2=0x7f07bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07dd600640 1 --2- 172.21.15.32:0/2969422490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07bc004d40 0x7f07bc0051c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f07d000a8a0 tx=0x7f07bc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.824+0000 7f07d7400640 1 -- 172.21.15.32:0/2969422490 reap_dead start 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.825+0000 7f07dd600640 1 -- 172.21.15.32:0/2969422490 shutdown_connections 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.825+0000 7f07dd600640 1 -- 172.21.15.32:0/2969422490 >> 172.21.15.32:0/2969422490 conn(0x7f07d806afb0 msgr2=0x7f07d806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.825+0000 7f07dd600640 1 -- 172.21.15.32:0/2969422490 shutdown_connections 2024-03-06T07:30:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.825+0000 7f07dd600640 1 -- 172.21.15.32:0/2969422490 wait complete. 2024-03-06T07:30:59.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.826+0000 7f07dd600640 1 Processor -- start 2024-03-06T07:30:59.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.827+0000 7f07dd600640 1 -- start start 2024-03-06T07:30:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.827+0000 7f07dd600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07bc004d40 0x7f07d809d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.827+0000 7f07dd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 0x7f07d80c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07dd600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07d815b6d0 0x7f07d815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07dd600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b4c20 con 0x7f07d80c5700 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07dd600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b36f0 con 0x7f07d815b6d0 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07dd600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80b4da0 con 0x7f07bc004d40 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 0x7f07d80c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 0x7f07d80c6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46976/0 (socket says 172.21.15.32:46976) 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 -- 172.21.15.32:0/2052023710 learned_addr learned my addr 172.21.15.32:0/2052023710 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07d6a00640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07bc004d40 0x7f07d809d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07d7400640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07d815b6d0 0x7f07d815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 -- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07bc004d40 msgr2=0x7f07d809d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07bc004d40 0x7f07d809d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 -- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07d815b6d0 msgr2=0x7f07d815bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07d815b6d0 0x7f07d815bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.828+0000 7f07ce000640 1 -- 172.21.15.32:0/2052023710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80c5bf0 con 0x7f07d80c5700 2024-03-06T07:30:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.829+0000 7f07ce000640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 0x7f07d80c6cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f07c0010860 tx=0x7f07c0010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.829+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000f040 con 0x7f07d80c5700 2024-03-06T07:30:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.829+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000eb60 con 0x7f07d80c5700 2024-03-06T07:30:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.829+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c001c400 con 0x7f07d80c5700 2024-03-06T07:30:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.829+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80c5ed0 con 0x7f07d80c5700 2024-03-06T07:30:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.829+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80c6490 con 0x7f07d80c5700 2024-03-06T07:30:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.830+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d8069510 con 0x7f07d80c5700 2024-03-06T07:30:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.831+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07c001c5a0 con 0x7f07d80c5700 2024-03-06T07:30:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.832+0000 7f07cf400640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f07b406b500 0x7f07b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.832+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07c008a8a0 con 0x7f07d80c5700 2024-03-06T07:30:59.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.833+0000 7f07d6a00640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f07b406b500 0x7f07b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.833+0000 7f07d6a00640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f07b406b500 0x7f07b406d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009b20 tx=0x7f07c8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:30:59.838+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07c0018030 con 0x7f07d80c5700 2024-03-06T07:31:00.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.038+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07d804f3a0 con 0x7f07d80c5700 2024-03-06T07:31:00.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.041+0000 7f07cf400640 1 -- 172.21.15.32:0/2052023710 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f07c0059710 con 0x7f07d80c5700 2024-03-06T07:31:00.044 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:00.044 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:59.035908+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:00.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.043+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f07b406b500 msgr2=0x7f07b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.043+0000 7f07dd600640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f07b406b500 0x7f07b406d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009b20 tx=0x7f07c8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:00.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.043+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 msgr2=0x7f07d80c6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.043+0000 7f07dd600640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 0x7f07d80c6cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f07c0010860 tx=0x7f07c0010d20 comp rx=0 tx=0).stop 2024-03-06T07:31:00.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 shutdown_connections 2024-03-06T07:31:00.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f07b406b500 0x7f07b406d9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07d815b6d0 0x7f07d815bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d80c5700 0x7f07d80c6cc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 --2- 172.21.15.32:0/2052023710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07bc004d40 0x7f07d809d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 >> 172.21.15.32:0/2052023710 conn(0x7f07d806afb0 msgr2=0x7f07d815c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 shutdown_connections 2024-03-06T07:31:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.044+0000 7f07dd600640 1 -- 172.21.15.32:0/2052023710 wait complete. 2024-03-06T07:31:00.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:00.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.283+0000 7f99b7e00640 1 Processor -- start 2024-03-06T07:31:00.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.283+0000 7f99b7e00640 1 -- start start 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99b0071380 0x7f99b00ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f99b006e240 0x7f99b00ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b7e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00ce9a0 con 0x7f99b0161670 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b7e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00730d0 con 0x7f99b0071380 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00cec80 con 0x7f99b006e240 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b5400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f99b0161670 0x7f99b0070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38334/0 (socket says 172.21.15.32:38334) 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b5400640 1 -- 172.21.15.32:0/3839919399 learned_addr learned my addr 172.21.15.32:0/3839919399 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.285+0000 7f99b4a00640 1 --2- 172.21.15.32:0/3839919399 >> v2:172.21.15.32:3300/0 conn(0x7f99b0071380 0x7f99b00ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99b5e00640 1 --2- 172.21.15.32:0/3839919399 >> v2:172.21.15.39:3300/0 conn(0x7f99b006e240 0x7f99b00ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99afe00640 1 -- 172.21.15.32:0/3839919399 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459458670 0 0) 0x7f99b00ce9a0 con 0x7f99b0161670 2024-03-06T07:31:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99afe00640 1 -- 172.21.15.32:0/3839919399 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9998003520 con 0x7f99b0161670 2024-03-06T07:31:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99b4a00640 1 -- 172.21.15.32:0/3839919399 >> v2:172.21.15.39:3300/0 conn(0x7f99b006e240 msgr2=0x7f99b00ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99b4a00640 1 --2- 172.21.15.32:0/3839919399 >> v2:172.21.15.39:3300/0 conn(0x7f99b006e240 0x7f99b00ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99b4a00640 1 -- 172.21.15.32:0/3839919399 >> v1:172.21.15.32:6789/0 conn(0x7f99b0161670 legacy=0x7f99b0070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.286+0000 7f99b4a00640 1 -- 172.21.15.32:0/3839919399 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00cef60 con 0x7f99b0071380 2024-03-06T07:31:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.287+0000 7f99b4a00640 1 --2- 172.21.15.32:0/3839919399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0071380 0x7f99b00ad090 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f999c0097a0 tx=0x7f999c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771cf12c3dc93eca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99afe00640 1 -- 172.21.15.32:0/3839919399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c0315a0 con 0x7f99b0071380 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99afe00640 1 -- 172.21.15.32:0/3839919399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c037070 con 0x7f99b0071380 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99afe00640 1 -- 172.21.15.32:0/3839919399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c031960 con 0x7f99b0071380 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 -- 172.21.15.32:0/3839919399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0071380 msgr2=0x7f99b00ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3839919399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0071380 0x7f99b00ad090 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f999c0097a0 tx=0x7f999c02b840 comp rx=0 tx=0).stop 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 -- 172.21.15.32:0/3839919399 shutdown_connections 2024-03-06T07:31:00.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3839919399 >> v2:172.21.15.39:3300/0 conn(0x7f99b006e240 0x7f99b00ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3839919399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0071380 0x7f99b00ad090 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 -- 172.21.15.32:0/3839919399 >> 172.21.15.32:0/3839919399 conn(0x7f99b006afb0 msgr2=0x7f99b006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 -- 172.21.15.32:0/3839919399 shutdown_connections 2024-03-06T07:31:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.288+0000 7f99b7e00640 1 -- 172.21.15.32:0/3839919399 wait complete. 2024-03-06T07:31:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.290+0000 7f99b7e00640 1 Processor -- start 2024-03-06T07:31:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.290+0000 7f99b7e00640 1 -- start start 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.291+0000 7f99b7e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99b006e240 0x7f99b014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.291+0000 7f99b7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99b0071380 0x7f99b00b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.291+0000 7f99b7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 0x7f99b009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.291+0000 7f99b7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00ce600 con 0x7f99b0161670 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.291+0000 7f99b7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00cf520 con 0x7f99b0071380 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.291+0000 7f99b7e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00ce780 con 0x7f99b006e240 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 0x7f99b009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 0x7f99b009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46992/0 (socket says 172.21.15.32:46992) 2024-03-06T07:31:00.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 -- 172.21.15.32:0/3926650765 learned_addr learned my addr 172.21.15.32:0/3926650765 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:00.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 -- 172.21.15.32:0/3926650765 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99b006e240 msgr2=0x7f99b014b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99b006e240 0x7f99b014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 -- 172.21.15.32:0/3926650765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99b0071380 msgr2=0x7f99b00b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99b0071380 0x7f99b00b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.292+0000 7f99b5e00640 1 -- 172.21.15.32:0/3926650765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b014c000 con 0x7f99b0161670 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.293+0000 7f99b5e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 0x7f99b009aa10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99a8007e30 tx=0x7f99a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.293+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8014620 con 0x7f99b0161670 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.293+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a8019070 con 0x7f99b0161670 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.293+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8014a20 con 0x7f99b0161670 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.293+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b014c2e0 con 0x7f99b0161670 2024-03-06T07:31:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.293+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b014c8a0 con 0x7f99b0161670 2024-03-06T07:31:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.294+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0069510 con 0x7f99b0161670 2024-03-06T07:31:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.295+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99a8014bc0 con 0x7f99b0161670 2024-03-06T07:31:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.296+0000 7f99ad600640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f999006b4d0 0x7f999006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.296+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99a808ab00 con 0x7f99b0161670 2024-03-06T07:31:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.296+0000 7f99b5400640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f999006b4d0 0x7f999006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.297+0000 7f99b5400640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f999006b4d0 0x7f999006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f99a4000f60 tx=0x7f99a40078f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.301+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99a8059970 con 0x7f99b0161670 2024-03-06T07:31:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.500+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99b004f3a0 con 0x7f99b0161670 2024-03-06T07:31:00.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.503+0000 7f99ad600640 1 -- 172.21.15.32:0/3926650765 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f99a808ae20 con 0x7f99b0161670 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:00.506 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:30:59.035908+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f999006b4d0 msgr2=0x7f999006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f999006b4d0 0x7f999006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f99a4000f60 tx=0x7f99a40078f0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 msgr2=0x7f99b009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 0x7f99b009aa10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99a8007e30 tx=0x7f99a800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 shutdown_connections 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f999006b4d0 0x7f999006d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b0161670 0x7f99b009aa10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99b0071380 0x7f99b00b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 --2- 172.21.15.32:0/3926650765 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99b006e240 0x7f99b014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 >> 172.21.15.32:0/3926650765 conn(0x7f99b006afb0 msgr2=0x7f99b00cc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 shutdown_connections 2024-03-06T07:31:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.507+0000 7f99b7e00640 1 -- 172.21.15.32:0/3926650765 wait complete. 2024-03-06T07:31:00.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:31:00.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.742+0000 7f5d3b800640 1 Processor -- start 2024-03-06T07:31:00.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.742+0000 7f5d3b800640 1 -- start start 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.743+0000 7f5d3b800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5d3415bf20 0x7f5d340acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.743+0000 7f5d3b800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d34072520 0x7f5d340cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.743+0000 7f5d3b800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d34071e20 0x7f5d340ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.743+0000 7f5d3b800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d34072df0 con 0x7f5d34072520 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.743+0000 7f5d3b800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d340cdab0 con 0x7f5d3415bf20 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.743+0000 7f5d3b800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d340b4d70 con 0x7f5d34071e20 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d34072520 0x7f5d340cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d34072520 0x7f5d340cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46996/0 (socket says 172.21.15.32:46996) 2024-03-06T07:31:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 -- 172.21.15.32:0/116049362 learned_addr learned my addr 172.21.15.32:0/116049362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d38e00640 1 --2- 172.21.15.32:0/116049362 >> v2:172.21.15.39:3300/0 conn(0x7f5d3415bf20 0x7f5d340acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d39800640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d34071e20 0x7f5d340ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 -- 172.21.15.32:0/116049362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d34071e20 msgr2=0x7f5d340ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d34071e20 0x7f5d340ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 -- 172.21.15.32:0/116049362 >> v2:172.21.15.39:3300/0 conn(0x7f5d3415bf20 msgr2=0x7f5d340acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 --2- 172.21.15.32:0/116049362 >> v2:172.21.15.39:3300/0 conn(0x7f5d3415bf20 0x7f5d340acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d2be00640 1 -- 172.21.15.32:0/116049362 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340b5050 con 0x7f5d34072520 2024-03-06T07:31:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d38e00640 1 --2- 172.21.15.32:0/116049362 >> v2:172.21.15.39:3300/0 conn(0x7f5d3415bf20 0x7f5d340acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.744+0000 7f5d39800640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d34071e20 0x7f5d340ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.745+0000 7f5d2be00640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34072520 0x7f5d340cdfb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5d3406ec40 tx=0x7f5d1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5bb47ddde10504 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.746+0000 7f5d33e00640 1 -- 172.21.15.32:0/116049362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c035020 con 0x7f5d34072520 2024-03-06T07:31:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.746+0000 7f5d33e00640 1 -- 172.21.15.32:0/116049362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c031870 con 0x7f5d34072520 2024-03-06T07:31:00.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.746+0000 7f5d33e00640 1 -- 172.21.15.32:0/116049362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c031c40 con 0x7f5d34072520 2024-03-06T07:31:00.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.746+0000 7f5d3b800640 1 -- 172.21.15.32:0/116049362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34072520 msgr2=0x7f5d340cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.746+0000 7f5d3b800640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34072520 0x7f5d340cdfb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5d3406ec40 tx=0x7f5d1c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:00.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 -- 172.21.15.32:0/116049362 shutdown_connections 2024-03-06T07:31:00.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d34071e20 0x7f5d340ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 --2- 172.21.15.32:0/116049362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34072520 0x7f5d340cdfb0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 --2- 172.21.15.32:0/116049362 >> v2:172.21.15.39:3300/0 conn(0x7f5d3415bf20 0x7f5d340acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 -- 172.21.15.32:0/116049362 >> 172.21.15.32:0/116049362 conn(0x7f5d3406b3d0 msgr2=0x7f5d3406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 -- 172.21.15.32:0/116049362 shutdown_connections 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.747+0000 7f5d3b800640 1 -- 172.21.15.32:0/116049362 wait complete. 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.748+0000 7f5d3b800640 1 Processor -- start 2024-03-06T07:31:00.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.749+0000 7f5d3b800640 1 -- start start 2024-03-06T07:31:00.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.749+0000 7f5d3b800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 0x7f5d3414a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d3b800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d34072520 0x7f5d34153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d3b800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d3415bf20 0x7f5d3406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d3b800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d340b4b10 con 0x7f5d34071e20 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d3b800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d340b4990 con 0x7f5d34072520 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d3b800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d340ad280 con 0x7f5d3415bf20 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 0x7f5d3414a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 0x7f5d3414a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47006/0 (socket says 172.21.15.32:47006) 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 -- 172.21.15.32:0/543054600 learned_addr learned my addr 172.21.15.32:0/543054600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 -- 172.21.15.32:0/543054600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d3415bf20 msgr2=0x7f5d3406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d3415bf20 0x7f5d3406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 -- 172.21.15.32:0/543054600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d34072520 msgr2=0x7f5d34153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d34072520 0x7f5d34153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.750+0000 7f5d38e00640 1 -- 172.21.15.32:0/543054600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3414ac00 con 0x7f5d34071e20 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.751+0000 7f5d38e00640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 0x7f5d3414a6c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400c7f0 tx=0x7f5d2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.751+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24015070 con 0x7f5d34071e20 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.751+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d24007820 con 0x7f5d34071e20 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.751+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24007bd0 con 0x7f5d34071e20 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.751+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d34152160 con 0x7f5d34071e20 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.751+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d34152720 con 0x7f5d34071e20 2024-03-06T07:31:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.752+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3404f330 con 0x7f5d34071e20 2024-03-06T07:31:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.753+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d24018070 con 0x7f5d34071e20 2024-03-06T07:31:00.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.754+0000 7f5d31600640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5d1006b4d0 0x7f5d1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.754+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d2408a310 con 0x7f5d34071e20 2024-03-06T07:31:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.755+0000 7f5d2be00640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5d1006b4d0 0x7f5d1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.755+0000 7f5d2be00640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5d1006b4d0 0x7f5d1006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c036000 tx=0x7f5d1c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:00.760+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d240590f0 con 0x7f5d34071e20 2024-03-06T07:31:01.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.001+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5d3406b850 con 0x7f5d34071e20 2024-03-06T07:31:01.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.608+0000 7f5d31600640 1 -- 172.21.15.32:0/543054600 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5d240589f0 con 0x7f5d34071e20 2024-03-06T07:31:01.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5d1006b4d0 msgr2=0x7f5d1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5d1006b4d0 0x7f5d1006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c036000 tx=0x7f5d1c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:01.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 msgr2=0x7f5d3414a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 0x7f5d3414a6c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400c7f0 tx=0x7f5d2400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 shutdown_connections 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5d1006b4d0 0x7f5d1006d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5d3415bf20 0x7f5d3406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d34072520 0x7f5d34153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 --2- 172.21.15.32:0/543054600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d34071e20 0x7f5d3414a6c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 >> 172.21.15.32:0/543054600 conn(0x7f5d3406b3d0 msgr2=0x7f5d340cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.612+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 shutdown_connections 2024-03-06T07:31:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.613+0000 7f5d3b800640 1 -- 172.21.15.32:0/543054600 wait complete. 2024-03-06T07:31:01.637 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:31:01.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:31:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.899+0000 7f5a3cc00640 1 Processor -- start 2024-03-06T07:31:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.900+0000 7f5a3cc00640 1 -- start start 2024-03-06T07:31:01.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.901+0000 7f5a3cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5a380d4860 0x7f5a380d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.901+0000 7f5a3cc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a380ceca0 con 0x7f5a3815dee0 2024-03-06T07:31:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.901+0000 7f5a3cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a380cef80 con 0x7f5a380cfa60 2024-03-06T07:31:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.901+0000 7f5a3cc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a380cf5d0 con 0x7f5a380d4860 2024-03-06T07:31:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.901+0000 7f5a37400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a3815dee0 0x7f5a380d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38350/0 (socket says 172.21.15.32:38350) 2024-03-06T07:31:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a37400640 1 -- 172.21.15.32:0/3661032102 learned_addr learned my addr 172.21.15.32:0/3661032102 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36a00640 1 --2- 172.21.15.32:0/3661032102 >> v2:172.21.15.39:3300/0 conn(0x7f5a380d4860 0x7f5a380d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674051008 0 0) 0x7f5a380ceca0 con 0x7f5a3815dee0 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c003520 con 0x7f5a3815dee0 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007669948 0 0) 0x7f5a1c003520 con 0x7f5a3815dee0 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 >> v2:172.21.15.39:3300/0 conn(0x7f5a380d4860 msgr2=0x7f5a380d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36000640 1 --2- 172.21.15.32:0/3661032102 >> v2:172.21.15.39:3300/0 conn(0x7f5a380d4860 0x7f5a380d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.902+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 >> v1:172.21.15.39:6789/0 conn(0x7f5a380cfa60 legacy=0x7f5a380b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a380cf260 con 0x7f5a3815dee0 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f5a28001670 con 0x7f5a3815dee0 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 >> v1:172.21.15.32:6789/0 conn(0x7f5a3815dee0 legacy=0x7f5a380d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a1c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a1c004800 0x7f5a1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a1c0051c0 0x7f5a1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a380cf5d0 con 0x7f5a1c003a00 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c00c0c0 con 0x7f5a1c0051c0 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c00c2c0 con 0x7f5a1c004800 2024-03-06T07:31:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37400640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a1c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37400640 1 -- 172.21.15.32:0/3661032102 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a1c004800 msgr2=0x7f5a1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37400640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a1c004800 0x7f5a1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37400640 1 -- 172.21.15.32:0/3661032102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a1c0051c0 msgr2=0x7f5a1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37400640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a1c0051c0 0x7f5a1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37400640 1 -- 172.21.15.32:0/3661032102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c003520 con 0x7f5a1c003a00 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.903+0000 7f5a37e00640 1 -- 172.21.15.32:0/3661032102 reap_dead start 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.905+0000 7f5a37400640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a1c003e20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800c190 tx=0x7f5a1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74b97adea7b4fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.905+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28007c60 con 0x7f5a1c003a00 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.905+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28030070 con 0x7f5a1c003a00 2024-03-06T07:31:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.905+0000 7f5a36000640 1 -- 172.21.15.32:0/3661032102 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28005d50 con 0x7f5a1c003a00 2024-03-06T07:31:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.905+0000 7f5a3cc00640 1 -- 172.21.15.32:0/3661032102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 msgr2=0x7f5a1c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.905+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a1c003e20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800c190 tx=0x7f5a1c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.906+0000 7f5a3cc00640 1 -- 172.21.15.32:0/3661032102 shutdown_connections 2024-03-06T07:31:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.906+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/3661032102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a1c003e20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.906+0000 7f5a3cc00640 1 -- 172.21.15.32:0/3661032102 >> 172.21.15.32:0/3661032102 conn(0x7f5a380d3cd0 msgr2=0x7f5a380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.906+0000 7f5a3cc00640 1 -- 172.21.15.32:0/3661032102 shutdown_connections 2024-03-06T07:31:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.906+0000 7f5a3cc00640 1 -- 172.21.15.32:0/3661032102 wait complete. 2024-03-06T07:31:01.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.907+0000 7f5a3cc00640 1 Processor -- start 2024-03-06T07:31:01.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.908+0000 7f5a3cc00640 1 -- start start 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.908+0000 7f5a3cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a380c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a3cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a380cada0 0x7f5a38155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a3815aa20 0x7f5a3815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a3cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a380d18a0 con 0x7f5a1c003a00 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a3cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3815e2c0 con 0x7f5a3815aa20 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a3cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a380d1a20 con 0x7f5a380cada0 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a380c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a380c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47018/0 (socket says 172.21.15.32:47018) 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 -- 172.21.15.32:0/1431512767 learned_addr learned my addr 172.21.15.32:0/1431512767 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37e00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a3815aa20 0x7f5a3815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a36a00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a380cada0 0x7f5a38155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 -- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a380cada0 msgr2=0x7f5a38155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a380cada0 0x7f5a38155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 -- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a3815aa20 msgr2=0x7f5a3815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a3815aa20 0x7f5a3815aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37400640 1 -- 172.21.15.32:0/1431512767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a380cb340 con 0x7f5a1c003a00 2024-03-06T07:31:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.909+0000 7f5a37e00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a3815aa20 0x7f5a3815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.910+0000 7f5a37400640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a380c9590 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800c280 tx=0x7f5a2802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.910+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28007c60 con 0x7f5a1c003a00 2024-03-06T07:31:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.910+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28004bf0 con 0x7f5a1c003a00 2024-03-06T07:31:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.910+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28007070 con 0x7f5a1c003a00 2024-03-06T07:31:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.910+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a380cb620 con 0x7f5a1c003a00 2024-03-06T07:31:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.910+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a380cbbe0 con 0x7f5a1c003a00 2024-03-06T07:31:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.911+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3804f330 con 0x7f5a1c003a00 2024-03-06T07:31:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.912+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a28007230 con 0x7f5a1c003a00 2024-03-06T07:31:01.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.913+0000 7f5a2f400640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.913+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a280ab880 con 0x7f5a1c003a00 2024-03-06T07:31:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.914+0000 7f5a36a00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.915+0000 7f5a36a00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a1406b4d0 0x7f5a1406d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5a380d0dc0 tx=0x7f5a20008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:01.918+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a2807a120 con 0x7f5a1c003a00 2024-03-06T07:31:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.223+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5a3808d7d0 con 0x7f5a1c003a00 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.616+0000 7f5a2f400640 1 -- 172.21.15.32:0/1431512767 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5a2807e1c0 con 0x7f5a1c003a00 2024-03-06T07:31:02.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a1406b4d0 msgr2=0x7f5a1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a1406b4d0 0x7f5a1406d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5a380d0dc0 tx=0x7f5a20008040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 msgr2=0x7f5a380c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a380c9590 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800c280 tx=0x7f5a2802e070 comp rx=0 tx=0).stop 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 shutdown_connections 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a3815aa20 0x7f5a3815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5a380cada0 0x7f5a38155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 --2- 172.21.15.32:0/1431512767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1c003a00 0x7f5a380c9590 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 >> 172.21.15.32:0/1431512767 conn(0x7f5a380d3cd0 msgr2=0x7f5a38161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 shutdown_connections 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.620+0000 7f5a3cc00640 1 -- 172.21.15.32:0/1431512767 wait complete. 2024-03-06T07:31:02.647 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:31:02.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.872+0000 7fae1b200640 1 Processor -- start 2024-03-06T07:31:02.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.872+0000 7fae1b200640 1 -- start start 2024-03-06T07:31:02.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae1b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae1c0d2960 0x7fae1c0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae1b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fae1c0d0b00 0x7fae1c0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae1b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0d2dc0 con 0x7fae1c15e070 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae1b200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c0d0f70 con 0x7fae1c0d2960 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae1b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c0d18c0 con 0x7fae1c0d0b00 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae19400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae1c0d2960 0x7fae1c0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae19400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae1c0d2960 0x7fae1c0d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47022/0 (socket says 172.21.15.32:47022) 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae19400640 1 -- 172.21.15.32:0/3211757455 learned_addr learned my addr 172.21.15.32:0/3211757455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.874+0000 7fae1a800640 1 --2- 172.21.15.32:0/3211757455 >> v2:172.21.15.39:3300/0 conn(0x7fae1c0d0b00 0x7fae1c0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.875+0000 7fae19400640 1 -- 172.21.15.32:0/3211757455 >> v2:172.21.15.39:3300/0 conn(0x7fae1c0d0b00 msgr2=0x7fae1c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.875+0000 7fae19400640 1 --2- 172.21.15.32:0/3211757455 >> v2:172.21.15.39:3300/0 conn(0x7fae1c0d0b00 0x7fae1c0b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.875+0000 7fae19400640 1 -- 172.21.15.32:0/3211757455 >> v1:172.21.15.39:6789/0 conn(0x7fae1c15e070 legacy=0x7fae1c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.875+0000 7fae19400640 1 -- 172.21.15.32:0/3211757455 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c0d1ba0 con 0x7fae1c0d2960 2024-03-06T07:31:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.876+0000 7fae19400640 1 --2- 172.21.15.32:0/3211757455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d2960 0x7fae1c0d05c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fae1c08c1b0 tx=0x7fae0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e03a0d1c892aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.876+0000 7fae18a00640 1 -- 172.21.15.32:0/3211757455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08035020 con 0x7fae1c0d2960 2024-03-06T07:31:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.876+0000 7fae18a00640 1 -- 172.21.15.32:0/3211757455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae08031870 con 0x7fae1c0d2960 2024-03-06T07:31:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.876+0000 7fae18a00640 1 -- 172.21.15.32:0/3211757455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08031c40 con 0x7fae1c0d2960 2024-03-06T07:31:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 -- 172.21.15.32:0/3211757455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d2960 msgr2=0x7fae1c0d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 --2- 172.21.15.32:0/3211757455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d2960 0x7fae1c0d05c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fae1c08c1b0 tx=0x7fae0802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 -- 172.21.15.32:0/3211757455 shutdown_connections 2024-03-06T07:31:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 --2- 172.21.15.32:0/3211757455 >> v2:172.21.15.39:3300/0 conn(0x7fae1c0d0b00 0x7fae1c0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 --2- 172.21.15.32:0/3211757455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d2960 0x7fae1c0d05c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 -- 172.21.15.32:0/3211757455 >> 172.21.15.32:0/3211757455 conn(0x7fae1c08d2a0 msgr2=0x7fae1c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 -- 172.21.15.32:0/3211757455 shutdown_connections 2024-03-06T07:31:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.877+0000 7fae1b200640 1 -- 172.21.15.32:0/3211757455 wait complete. 2024-03-06T07:31:02.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.879+0000 7fae1b200640 1 Processor -- start 2024-03-06T07:31:02.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.879+0000 7fae1b200640 1 -- start start 2024-03-06T07:31:02.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae1b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 0x7fae1c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae1b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae1c0d2960 0x7fae1c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae1b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae1c15e070 0x7fae1c0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae1b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0d2440 con 0x7fae1c0d0b00 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae1b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0d2160 con 0x7fae1c15e070 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae1b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0d45b0 con 0x7fae1c0d2960 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae19e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 0x7fae1c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae19e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 0x7fae1c14dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47032/0 (socket says 172.21.15.32:47032) 2024-03-06T07:31:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.880+0000 7fae19e00640 1 -- 172.21.15.32:0/1235133044 learned_addr learned my addr 172.21.15.32:0/1235133044 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:02.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19400640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae1c0d2960 0x7fae1c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae1a800640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae1c15e070 0x7fae1c0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19e00640 1 -- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae1c0d2960 msgr2=0x7fae1c0b8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19e00640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae1c0d2960 0x7fae1c0b8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19e00640 1 -- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae1c15e070 msgr2=0x7fae1c0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19e00640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae1c15e070 0x7fae1c0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19e00640 1 -- 172.21.15.32:0/1235133044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c0b7a90 con 0x7fae1c0d0b00 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae19e00640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 0x7fae1c14dce0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00b410 tx=0x7fae0c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.881+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c015020 con 0x7fae1c0d0b00 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.882+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c011830 con 0x7fae1c0d0b00 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.882+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c011c20 con 0x7fae1c0d0b00 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.882+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c0b7d70 con 0x7fae1c0d0b00 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.882+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c0b8330 con 0x7fae1c0d0b00 2024-03-06T07:31:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.883+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c04f330 con 0x7fae1c0d0b00 2024-03-06T07:31:02.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.884+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae0c018020 con 0x7fae1c0d0b00 2024-03-06T07:31:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.885+0000 7fae12000640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadfc06b4d0 0x7fadfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.885+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae0c08a0d0 con 0x7fae1c0d0b00 2024-03-06T07:31:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.885+0000 7fae19400640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadfc06b4d0 0x7fadfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.886+0000 7fae19400640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadfc06b4d0 0x7fadfc06d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae08036000 tx=0x7fae0803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:02.891+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae0c01f020 con 0x7fae1c0d0b00 2024-03-06T07:31:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.131+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1c0c75c0 con 0x7fae1c0d0b00 2024-03-06T07:31:03.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.132+0000 7fae12000640 1 -- 172.21.15.32:0/1235133044 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4965 (secure 0 0 0) 0x7fae0c0172c0 con 0x7fae1c0d0b00 2024-03-06T07:31:03.135 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:03.135 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:59.036395+0000","modified":"2024-03-06T07:31:02.225064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/1942129463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1942129463},{"type":"v1","addr":"172.21.15.32:6836","nonce":1942129463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":3}]} 2024-03-06T07:31:03.135 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-06T07:31:03.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadfc06b4d0 msgr2=0x7fadfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadfc06b4d0 0x7fadfc06d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae08036000 tx=0x7fae0803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:03.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 msgr2=0x7fae1c14dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 0x7fae1c14dce0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00b410 tx=0x7fae0c00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 shutdown_connections 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadfc06b4d0 0x7fadfc06d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae1c15e070 0x7fae1c0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae1c0d2960 0x7fae1c0b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 --2- 172.21.15.32:0/1235133044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae1c0d0b00 0x7fae1c14dce0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 >> 172.21.15.32:0/1235133044 conn(0x7fae1c08d2a0 msgr2=0x7fae1c15d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 shutdown_connections 2024-03-06T07:31:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.135+0000 7fae1b200640 1 -- 172.21.15.32:0/1235133044 wait complete. 2024-03-06T07:31:03.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:03.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.379+0000 7f146b000640 1 Processor -- start 2024-03-06T07:31:03.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.379+0000 7f146b000640 1 -- start start 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f146b000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f146415e0b0 0x7f14640cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f146b000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640b3710 con 0x7f14640d4a10 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f146b000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640b39f0 con 0x7f14640d0e50 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f146b000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640b09e0 con 0x7f146415e0b0 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f1469200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f14640d4a10 0x7f14640d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38366/0 (socket says 172.21.15.32:38366) 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f1469200640 1 -- 172.21.15.32:0/608910305 learned_addr learned my addr 172.21.15.32:0/608910305 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.381+0000 7f1469c00640 1 --2- 172.21.15.32:0/608910305 >> v2:172.21.15.39:3300/0 conn(0x7f146415e0b0 0x7f14640cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448578822 0 0) 0x7f14640b3710 con 0x7f14640d4a10 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c003520 con 0x7f14640d4a10 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1663021229 0 0) 0x7f144c003520 con 0x7f14640d4a10 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 >> v2:172.21.15.39:3300/0 conn(0x7f146415e0b0 msgr2=0x7f14640cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 --2- 172.21.15.32:0/608910305 >> v2:172.21.15.39:3300/0 conn(0x7f146415e0b0 0x7f14640cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 >> v1:172.21.15.39:6789/0 conn(0x7f14640d0e50 legacy=0x7f14640b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640b3cd0 con 0x7f14640d4a10 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f1450001670 con 0x7f14640d4a10 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 >> v1:172.21.15.32:6789/0 conn(0x7f14640d4a10 legacy=0x7f14640d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c003a00 0x7f144c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f144c004800 0x7f144c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f144c0051c0 0x7f144c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c00c090 con 0x7f144c0051c0 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c00c240 con 0x7f144c003a00 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.382+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c00c440 con 0x7f144c004800 2024-03-06T07:31:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f146a600640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f144c0051c0 0x7f144c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f1469c00640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c003a00 0x7f144c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f1469200640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f144c004800 0x7f144c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f146a600640 1 -- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f144c004800 msgr2=0x7f144c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f146a600640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f144c004800 0x7f144c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f146a600640 1 -- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c003a00 msgr2=0x7f144c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f146a600640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c003a00 0x7f144c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.383+0000 7f146a600640 1 -- 172.21.15.32:0/608910305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c003520 con 0x7f144c0051c0 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.384+0000 7f146a600640 1 -- 172.21.15.32:0/608910305 reap_dead start 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.384+0000 7f146a600640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f144c0051c0 0x7f144c00b9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f145c001e20 tx=0x7f144c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf3e80b3a4170b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.384+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c001a70 con 0x7f144c0051c0 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c003b70 con 0x7f144c0051c0 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f1463e00640 1 -- 172.21.15.32:0/608910305 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0050d0 con 0x7f144c0051c0 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 -- 172.21.15.32:0/608910305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f144c0051c0 msgr2=0x7f144c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f144c0051c0 0x7f144c00b9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f145c001e20 tx=0x7f144c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 -- 172.21.15.32:0/608910305 shutdown_connections 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 --2- 172.21.15.32:0/608910305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f144c0051c0 0x7f144c00b9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 -- 172.21.15.32:0/608910305 >> 172.21.15.32:0/608910305 conn(0x7f14640d3d30 msgr2=0x7f14640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 -- 172.21.15.32:0/608910305 shutdown_connections 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.385+0000 7f146b000640 1 -- 172.21.15.32:0/608910305 wait complete. 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.388+0000 7f146b000640 1 Processor -- start 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.388+0000 7f146b000640 1 -- start start 2024-03-06T07:31:03.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f146b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c0051c0 0x7f14641f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f146b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 0x7f146406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f146b000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f146406cf30 0x7f146415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f146b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640cfc60 con 0x7f146414f050 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f146b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146415e490 con 0x7f144c0051c0 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f146b000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640d0740 con 0x7f146406cf30 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f1469200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 0x7f146406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f1469200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 0x7f146406c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47050/0 (socket says 172.21.15.32:47050) 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.390+0000 7f1469200640 1 -- 172.21.15.32:0/3149578649 learned_addr learned my addr 172.21.15.32:0/3149578649 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f146a600640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f146406cf30 0x7f146415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469c00640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c0051c0 0x7f14641f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469200640 1 -- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f146406cf30 msgr2=0x7f146415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469200640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f146406cf30 0x7f146415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469200640 1 -- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c0051c0 msgr2=0x7f14641f78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469200640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c0051c0 0x7f14641f78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469200640 1 -- 172.21.15.32:0/3149578649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464160010 con 0x7f146414f050 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.391+0000 7f1469200640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 0x7f146406c9f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f145000e700 tx=0x7f145000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.392+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145000c6c0 con 0x7f146414f050 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.392+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1450018070 con 0x7f146414f050 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.392+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145000cac0 con 0x7f146414f050 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.392+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14641602f0 con 0x7f146414f050 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.392+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14641608b0 con 0x7f146414f050 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.393+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146404f330 con 0x7f146414f050 2024-03-06T07:31:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.394+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f145000cc60 con 0x7f146414f050 2024-03-06T07:31:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.395+0000 7f1461600640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f144406b500 0x7f144406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.395+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1450089c90 con 0x7f146414f050 2024-03-06T07:31:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.396+0000 7f1469c00640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f144406b500 0x7f144406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.396+0000 7f1469c00640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f144406b500 0x7f144406d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1458009990 tx=0x7f1458008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.400+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1450058b00 con 0x7f146414f050 2024-03-06T07:31:03.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.623+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14640d2790 con 0x7f146414f050 2024-03-06T07:31:03.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.624+0000 7f1461600640 1 -- 172.21.15.32:0/3149578649 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1450058400 con 0x7f146414f050 2024-03-06T07:31:03.627 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:03.627 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:00.589676+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f144406b500 msgr2=0x7f144406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f144406b500 0x7f144406d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1458009990 tx=0x7f1458008040 comp rx=0 tx=0).stop 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 msgr2=0x7f146406c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 0x7f146406c9f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f145000e700 tx=0x7f145000ebc0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 shutdown_connections 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f144406b500 0x7f144406d9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f146406cf30 0x7f146415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f146414f050 0x7f146406c9f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 --2- 172.21.15.32:0/3149578649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f144c0051c0 0x7f14641f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.627+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 >> 172.21.15.32:0/3149578649 conn(0x7f14640d3d30 msgr2=0x7f14640b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.628+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 shutdown_connections 2024-03-06T07:31:03.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.628+0000 7f146b000640 1 -- 172.21.15.32:0/3149578649 wait complete. 2024-03-06T07:31:03.651 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:31:03.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.855+0000 7fd600e00640 1 Processor -- start 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.855+0000 7fd600e00640 1 -- start start 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.856+0000 7fd600e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5fc15e0b0 0x7fd5fc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.856+0000 7fd600e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd5fc0d4a10 0x7fd5fc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.856+0000 7fd600e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5fc0b3710 con 0x7fd5fc0d0c80 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd600e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc0d10f0 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd600e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc0b09e0 con 0x7fd5fc15e0b0 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd5faa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd5fc0d4a10 0x7fd5fc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd5fbe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5fc0d0c80 0x7fd5fc0d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38382/0 (socket says 172.21.15.32:38382) 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd5fbe00640 1 -- 172.21.15.32:0/233242637 learned_addr learned my addr 172.21.15.32:0/233242637 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd5fb400640 1 --2- 172.21.15.32:0/233242637 >> v2:172.21.15.39:3300/0 conn(0x7fd5fc15e0b0 0x7fd5fc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd5faa00640 1 -- 172.21.15.32:0/233242637 >> v2:172.21.15.39:3300/0 conn(0x7fd5fc15e0b0 msgr2=0x7fd5fc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.857+0000 7fd5faa00640 1 --2- 172.21.15.32:0/233242637 >> v2:172.21.15.39:3300/0 conn(0x7fd5fc15e0b0 0x7fd5fc0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.858+0000 7fd5faa00640 1 -- 172.21.15.32:0/233242637 >> v1:172.21.15.32:6789/0 conn(0x7fd5fc0d0c80 legacy=0x7fd5fc0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.858+0000 7fd5faa00640 1 -- 172.21.15.32:0/233242637 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc0b39f0 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.858+0000 7fd5fa000640 1 -- 172.21.15.32:0/233242637 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293204392 0 0) 0x7fd5fc0b3710 con 0x7fd5fc0d0c80 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.859+0000 7fd5faa00640 1 --2- 172.21.15.32:0/233242637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0d0740 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0b68c0 tx=0x7fd5e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909c7e79e1f8a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.859+0000 7fd5fa000640 1 -- 172.21.15.32:0/233242637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8035020 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.859+0000 7fd5fa000640 1 -- 172.21.15.32:0/233242637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e8031830 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.859+0000 7fd5fa000640 1 -- 172.21.15.32:0/233242637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8031c00 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 -- 172.21.15.32:0/233242637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 msgr2=0x7fd5fc0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 --2- 172.21.15.32:0/233242637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0d0740 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0b68c0 tx=0x7fd5e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 -- 172.21.15.32:0/233242637 shutdown_connections 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 --2- 172.21.15.32:0/233242637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0d0740 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 --2- 172.21.15.32:0/233242637 >> v2:172.21.15.39:3300/0 conn(0x7fd5fc15e0b0 0x7fd5fc0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 -- 172.21.15.32:0/233242637 >> 172.21.15.32:0/233242637 conn(0x7fd5fc0d3d30 msgr2=0x7fd5fc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 -- 172.21.15.32:0/233242637 shutdown_connections 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.860+0000 7fd600e00640 1 -- 172.21.15.32:0/233242637 wait complete. 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.862+0000 7fd600e00640 1 Processor -- start 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.862+0000 7fd600e00640 1 -- start start 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd600e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5fc0d0c80 0x7fd5fc15f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd600e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd600e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5fc15e0b0 0x7fd5fc0b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd600e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc15e490 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd600e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0b3fb0 con 0x7fd5fc0d0c80 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd600e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0cfc60 con 0x7fd5fc15e0b0 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47072/0 (socket says 172.21.15.32:47072) 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 -- 172.21.15.32:0/934682210 learned_addr learned my addr 172.21.15.32:0/934682210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5fb400640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5fc0d0c80 0x7fd5fc15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5fbe00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5fc15e0b0 0x7fd5fc0b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 -- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5fc15e0b0 msgr2=0x7fd5fc0b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5fc15e0b0 0x7fd5fc0b4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 -- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5fc0d0c80 msgr2=0x7fd5fc15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5fc0d0c80 0x7fd5fc15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5faa00640 1 -- 172.21.15.32:0/934682210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc1f4970 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.863+0000 7fd5fb400640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5fc0d0c80 0x7fd5fc15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.864+0000 7fd5faa00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0cc2d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8000c00 tx=0x7fd5e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.864+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8035070 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.864+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e8004230 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.864+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e803a6f0 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.864+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc15e7b0 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.864+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc0b4cb0 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.865+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc068a70 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.867+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5e8038020 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.868+0000 7fd5f3400640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5dc06b500 0x7fd5dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.868+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5e80abf90 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.869+0000 7fd5fb400640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5dc06b500 0x7fd5dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.869+0000 7fd5fb400640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5dc06b500 0x7fd5dc06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec006fd0 tx=0x7fd5ec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:03.874+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5fc068a70 con 0x7fd5fc0d4a10 2024-03-06T07:31:03.952 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-06T07:31:03.950+0000 7f18cb000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:04.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.071+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5fc04f330 con 0x7fd5dc06b500 2024-03-06T07:31:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.081+0000 7fd5f3400640 1 -- 172.21.15.32:0/934682210 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239859 (secure 0 0 0) 0x7fd5fc04f330 con 0x7fd5dc06b500 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:04.089 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5dc06b500 msgr2=0x7fd5dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5dc06b500 0x7fd5dc06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec006fd0 tx=0x7fd5ec008040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 msgr2=0x7fd5fc0cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0cc2d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8000c00 tx=0x7fd5e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 shutdown_connections 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5dc06b500 0x7fd5dc06d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5fc15e0b0 0x7fd5fc0b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5fc0d4a10 0x7fd5fc0cc2d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 --2- 172.21.15.32:0/934682210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5fc0d0c80 0x7fd5fc15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.087+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 >> 172.21.15.32:0/934682210 conn(0x7fd5fc0d3d30 msgr2=0x7fd5fc0b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.088+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 shutdown_connections 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.088+0000 7fd600e00640 1 -- 172.21.15.32:0/934682210 wait complete. 2024-03-06T07:31:04.111 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-03-06T07:31:02.221756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":254500,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720220,"statfs":{"total":773094113280,"available":772833505280,"internally_reserved":0,"allocated":5181440,"data_stored":2766773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.670968"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.548556+0000","last_change":"2024-03-06T07:30:57.548556+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.548556+0000","last_undegraded":"2024-03-06T07:30:57.548556+0000","last_fullsized":"2024-03-06T07:30:57.548556+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.556818+0000","last_change":"2024-03-06T07:30:57.556818+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.556818+0000","last_undegraded":"2024-03-06T07:30:57.556818+0000","last_fullsized":"2024-03-06T07:30:57.556818+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.560913+0000","last_change":"2024-03-06T07:30:57.560913+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.560913+0000","last_undegraded":"2024-03-06T07:30:57.560913+0000","last_fullsized":"2024-03-06T07:30:57.560913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.558996+0000","last_change":"2024-03-06T07:30:57.558996+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.558996+0000","last_undegraded":"2024-03-06T07:30:57.558996+0000","last_fullsized":"2024-03-06T07:30:57.558996+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.548727+0000","last_change":"2024-03-06T07:30:57.548727+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.548727+0000","last_undegraded":"2024-03-06T07:30:57.548727+0000","last_fullsized":"2024-03-06T07:30:57.548727+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.555794+0000","last_change":"2024-03-06T07:30:57.555794+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.555794+0000","last_undegraded":"2024-03-06T07:30:57.555794+0000","last_fullsized":"2024-03-06T07:30:57.555794+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.562053+0000","last_change":"2024-03-06T07:30:57.562053+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.562053+0000","last_undegraded":"2024-03-06T07:30:57.562053+0000","last_fullsized":"2024-03-06T07:30:57.562053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.552890+0000","last_change":"2024-03-06T07:30:57.552890+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.552890+0000","last_undegraded":"2024-03-06T07:30:57.552890+0000","last_fullsized":"2024-03-06T07:30:57.552890+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.554046+0000","last_change":"2024-03-06T07:30:57.554046+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.554046+0000","last_undegraded":"2024-03-06T07:30:57.554046+0000","last_fullsized":"2024-03-06T07:30:57.554046+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.552457+0000","last_change":"2024-03-06T07:30:57.552457+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.552457+0000","last_undegraded":"2024-03-06T07:30:57.552457+0000","last_fullsized":"2024-03-06T07:30:57.552457+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.562597+0000","last_change":"2024-03-06T07:30:57.562597+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.562597+0000","last_undegraded":"2024-03-06T07:30:57.562597+0000","last_fullsized":"2024-03-06T07:30:57.562597+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.559837+0000","last_change":"2024-03-06T07:30:57.559837+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.559837+0000","last_undegraded":"2024-03-06T07:30:57.559837+0000","last_fullsized":"2024-03-06T07:30:57.559837+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.547880+0000","last_change":"2024-03-06T07:30:57.547880+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.547880+0000","last_undegraded":"2024-03-06T07:30:57.547880+0000","last_fullsized":"2024-03-06T07:30:57.547880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.558789+0000","last_change":"2024-03-06T07:30:57.558789+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.558789+0000","last_undegraded":"2024-03-06T07:30:57.558789+0000","last_fullsized":"2024-03-06T07:30:57.558789+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.546826+0000","last_change":"2024-03-06T07:30:57.546826+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.546826+0000","last_undegraded":"2024-03-06T07:30:57.546826+0000","last_fullsized":"2024-03-06T07:30:57.546826+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.552513+0000","last_change":"2024-03-06T07:30:57.552513+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.552513+0000","last_undegraded":"2024-03-06T07:30:57.552513+0000","last_fullsized":"2024-03-06T07:30:57.552513+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.561575+0000","last_change":"2024-03-06T07:30:57.561575+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.561575+0000","last_undegraded":"2024-03-06T07:30:57.561575+0000","last_fullsized":"2024-03-06T07:30:57.561575+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.554192+0000","last_change":"2024-03-06T07:30:57.554192+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.554192+0000","last_undegraded":"2024-03-06T07:30:57.554192+0000","last_fullsized":"2024-03-06T07:30:57.554192+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.565826+0000","last_change":"2024-03-06T07:30:57.565826+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.565826+0000","last_undegraded":"2024-03-06T07:30:57.565826+0000","last_fullsized":"2024-03-06T07:30:57.565826+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.546545+0000","last_change":"2024-03-06T07:30:57.546545+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.546545+0000","last_undegraded":"2024-03-06T07:30:57.546545+0000","last_fullsized":"2024-03-06T07:30:57.546545+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.547243+0000","last_change":"2024-03-06T07:30:57.547243+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.547243+0000","last_undegraded":"2024-03-06T07:30:57.547243+0000","last_fullsized":"2024-03-06T07:30:57.547243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.562274+0000","last_change":"2024-03-06T07:30:57.562274+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.562274+0000","last_undegraded":"2024-03-06T07:30:57.562274+0000","last_fullsized":"2024-03-06T07:30:57.562274+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.551933+0000","last_change":"2024-03-06T07:30:57.551933+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.551933+0000","last_undegraded":"2024-03-06T07:30:57.551933+0000","last_fullsized":"2024-03-06T07:30:57.551933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.557172+0000","last_change":"2024-03-06T07:30:57.557172+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.557172+0000","last_undegraded":"2024-03-06T07:30:57.557172+0000","last_fullsized":"2024-03-06T07:30:57.557172+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.546545+0000","last_change":"2024-03-06T07:30:57.546545+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.546545+0000","last_undegraded":"2024-03-06T07:30:57.546545+0000","last_fullsized":"2024-03-06T07:30:57.546545+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.561015+0000","last_change":"2024-03-06T07:30:57.561015+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.561015+0000","last_undegraded":"2024-03-06T07:30:57.561015+0000","last_fullsized":"2024-03-06T07:30:57.561015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.558668+0000","last_change":"2024-03-06T07:30:57.558668+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.558668+0000","last_undegraded":"2024-03-06T07:30:57.558668+0000","last_fullsized":"2024-03-06T07:30:57.558668+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.557232+0000","last_change":"2024-03-06T07:30:57.557232+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.557232+0000","last_undegraded":"2024-03-06T07:30:57.557232+0000","last_fullsized":"2024-03-06T07:30:57.557232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.554053+0000","last_change":"2024-03-06T07:30:57.554053+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.554053+0000","last_undegraded":"2024-03-06T07:30:57.554053+0000","last_fullsized":"2024-03-06T07:30:57.554053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.559571+0000","last_change":"2024-03-06T07:30:57.559571+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.559571+0000","last_undegraded":"2024-03-06T07:30:57.559571+0000","last_fullsized":"2024-03-06T07:30:57.559571+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.551852+0000","last_change":"2024-03-06T07:30:57.551852+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.551852+0000","last_undegraded":"2024-03-06T07:30:57.551852+0000","last_fullsized":"2024-03-06T07:30:57.551852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:58.540115+0000","last_change":"2024-03-06T07:30:58.540115+0000","last_active":"2024-03-06T07:30:58.540115+0000","last_peered":"2024-03-06T07:30:58.540115+0000","last_clean":"2024-03-06T07:30:58.540115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:58.540115+0000","last_undegraded":"2024-03-06T07:30:58.540115+0000","last_fullsized":"2024-03-06T07:30:58.540115+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:30:57.564773+0000","last_change":"2024-03-06T07:30:57.564773+0000","last_active":"2024-03-06T07:30:57.515899+0000","last_peered":"2024-03-06T07:30:57.515899+0000","last_clean":"2024-03-06T07:30:57.515899+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:57.564773+0000","last_undegraded":"2024-03-06T07:30:57.564773+0000","last_fullsized":"2024-03-06T07:30:57.564773+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":211,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:30:57.559967+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:30:57.559967+0000","last_peered":"2024-03-06T07:30:57.559967+0000","last_clean":"2024-03-06T07:30:57.559967+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:30:57.559967+0000","last_undegraded":"2024-03-06T07:30:57.559967+0000","last_fullsized":"2024-03-06T07:30:57.559967+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:04.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.370+0000 7f85dfe00640 1 Processor -- start 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.370+0000 7f85dfe00640 1 -- start start 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.372+0000 7f85dfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f85d8161b40 0x7f85d80cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.372+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b5060 0x7f85d80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.372+0000 7f85dfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85d80b54a0 0x7f85d80d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.372+0000 7f85dfe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80cf900 con 0x7f85d80b54a0 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.372+0000 7f85dfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80cea50 con 0x7f85d8161b40 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.372+0000 7f85dfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80cecf0 con 0x7f85d80b5060 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85d80b54a0 0x7f85d80d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85d80b54a0 0x7f85d80d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47088/0 (socket says 172.21.15.32:47088) 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 -- 172.21.15.32:0/2583967981 learned_addr learned my addr 172.21.15.32:0/2583967981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dd400640 1 --2- 172.21.15.32:0/2583967981 >> v2:172.21.15.39:3300/0 conn(0x7f85d8161b40 0x7f85d80cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dca00640 1 --2- 172.21.15.32:0/2583967981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b5060 0x7f85d80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 -- 172.21.15.32:0/2583967981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b5060 msgr2=0x7f85d80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 --2- 172.21.15.32:0/2583967981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b5060 0x7f85d80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 -- 172.21.15.32:0/2583967981 >> v2:172.21.15.39:3300/0 conn(0x7f85d8161b40 msgr2=0x7f85d80cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 --2- 172.21.15.32:0/2583967981 >> v2:172.21.15.39:3300/0 conn(0x7f85d8161b40 0x7f85d80cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.373+0000 7f85dde00640 1 -- 172.21.15.32:0/2583967981 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80cefd0 con 0x7f85d80b54a0 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.374+0000 7f85dde00640 1 --2- 172.21.15.32:0/2583967981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d80b54a0 0x7f85d80d2760 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f85d808db00 tx=0x7f85d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83ae97886838435 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85d7e00640 1 -- 172.21.15.32:0/2583967981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d002ca70 con 0x7f85d80b54a0 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85d7e00640 1 -- 172.21.15.32:0/2583967981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d002cc10 con 0x7f85d80b54a0 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85d7e00640 1 -- 172.21.15.32:0/2583967981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00343f0 con 0x7f85d80b54a0 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 -- 172.21.15.32:0/2583967981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d80b54a0 msgr2=0x7f85d80d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 --2- 172.21.15.32:0/2583967981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d80b54a0 0x7f85d80d2760 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f85d808db00 tx=0x7f85d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 -- 172.21.15.32:0/2583967981 shutdown_connections 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 --2- 172.21.15.32:0/2583967981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d80b54a0 0x7f85d80d2760 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 --2- 172.21.15.32:0/2583967981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b5060 0x7f85d80d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 --2- 172.21.15.32:0/2583967981 >> v2:172.21.15.39:3300/0 conn(0x7f85d8161b40 0x7f85d80cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.375+0000 7f85dfe00640 1 -- 172.21.15.32:0/2583967981 >> 172.21.15.32:0/2583967981 conn(0x7f85d809d220 msgr2=0x7f85d80d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.376+0000 7f85dfe00640 1 -- 172.21.15.32:0/2583967981 shutdown_connections 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.376+0000 7f85dfe00640 1 -- 172.21.15.32:0/2583967981 wait complete. 2024-03-06T07:31:04.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.377+0000 7f85dfe00640 1 Processor -- start 2024-03-06T07:31:04.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.377+0000 7f85dfe00640 1 -- start start 2024-03-06T07:31:04.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.378+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85d80b5060 0x7f85d80b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.378+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b54a0 0x7f85d80b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.378+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 0x7f85d80b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.378+0000 7f85dfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80b2ff0 con 0x7f85d8161b40 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.378+0000 7f85dfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80cf590 con 0x7f85d80b5060 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.378+0000 7f85dfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80b3170 con 0x7f85d80b54a0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 0x7f85d80b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 0x7f85d80b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47104/0 (socket says 172.21.15.32:47104) 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 -- 172.21.15.32:0/3346506520 learned_addr learned my addr 172.21.15.32:0/3346506520 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dd400640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85d80b5060 0x7f85d80b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 -- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b54a0 msgr2=0x7f85d80b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b54a0 0x7f85d80b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 -- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85d80b5060 msgr2=0x7f85d80b95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85d80b5060 0x7f85d80b95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 -- 172.21.15.32:0/3346506520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80b8520 con 0x7f85d8161b40 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dd400640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85d80b5060 0x7f85d80b95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.379+0000 7f85dde00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 0x7f85d80b7df0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f85d0036040 tx=0x7f85d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.380+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0043020 con 0x7f85d8161b40 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.380+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d003e450 con 0x7f85d8161b40 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.380+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0009510 con 0x7f85d8161b40 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.380+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d80b8800 con 0x7f85d8161b40 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.380+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d80b8dc0 con 0x7f85d8161b40 2024-03-06T07:31:04.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.381+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d809d600 con 0x7f85d8161b40 2024-03-06T07:31:04.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.382+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85d0046020 con 0x7f85d8161b40 2024-03-06T07:31:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.383+0000 7f85d5600640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85b806b4d0 0x7f85b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.383+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85d00aca60 con 0x7f85d8161b40 2024-03-06T07:31:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.383+0000 7f85dd400640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85b806b4d0 0x7f85b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.384+0000 7f85dd400640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85b806b4d0 0x7f85b806d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006fd0 tx=0x7f85cc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.387+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85d809d600 con 0x7f85d8161b40 2024-03-06T07:31:04.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.585+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85d804f3a0 con 0x7f85b806b4d0 2024-03-06T07:31:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.596+0000 7f85d5600640 1 -- 172.21.15.32:0/3346506520 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7f85d804f3a0 con 0x7f85b806b4d0 2024-03-06T07:31:04.601 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:04.602 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85b806b4d0 msgr2=0x7f85b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85b806b4d0 0x7f85b806d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006fd0 tx=0x7f85cc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 msgr2=0x7f85d80b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 0x7f85d80b7df0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f85d0036040 tx=0x7f85d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 shutdown_connections 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85b806b4d0 0x7f85b806d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85d8161b40 0x7f85d80b7df0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f85d80b54a0 0x7f85d80b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 --2- 172.21.15.32:0/3346506520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85d80b5060 0x7f85d80b95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 >> 172.21.15.32:0/3346506520 conn(0x7f85d809d220 msgr2=0x7f85d80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 shutdown_connections 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.601+0000 7f85dfe00640 1 -- 172.21.15.32:0/3346506520 wait complete. 2024-03-06T07:31:04.624 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-06T07:31:04.222203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254888,"kb_used_data":5448,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719832,"statfs":{"total":773094113280,"available":772833107968,"internally_reserved":0,"allocated":5578752,"data_stored":2943780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001618"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613620+0000","last_change":"2024-03-06T07:30:59.619734+0000","last_active":"2024-03-06T07:31:00.613620+0000","last_peered":"2024-03-06T07:31:00.613620+0000","last_clean":"2024-03-06T07:31:00.613620+0000","last_became_active":"2024-03-06T07:30:59.619427+0000","last_became_peered":"2024-03-06T07:30:59.619427+0000","last_unstale":"2024-03-06T07:31:00.613620+0000","last_undegraded":"2024-03-06T07:31:00.613620+0000","last_fullsized":"2024-03-06T07:31:00.613620+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.122027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956213+0000","last_change":"2024-03-06T07:30:59.613730+0000","last_active":"2024-03-06T07:31:00.956213+0000","last_peered":"2024-03-06T07:31:00.956213+0000","last_clean":"2024-03-06T07:31:00.956213+0000","last_became_active":"2024-03-06T07:30:59.612583+0000","last_became_peered":"2024-03-06T07:30:59.612583+0000","last_unstale":"2024-03-06T07:31:00.956213+0000","last_undegraded":"2024-03-06T07:31:00.956213+0000","last_fullsized":"2024-03-06T07:31:00.956213+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:00.234855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618744+0000","last_change":"2024-03-06T07:30:59.612150+0000","last_active":"2024-03-06T07:31:00.618744+0000","last_peered":"2024-03-06T07:31:00.618744+0000","last_clean":"2024-03-06T07:31:00.618744+0000","last_became_active":"2024-03-06T07:30:59.611799+0000","last_became_peered":"2024-03-06T07:30:59.611799+0000","last_unstale":"2024-03-06T07:31:00.618744+0000","last_undegraded":"2024-03-06T07:31:00.618744+0000","last_fullsized":"2024-03-06T07:31:00.618744+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:00.163307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957733+0000","last_change":"2024-03-06T07:30:59.617847+0000","last_active":"2024-03-06T07:31:00.957733+0000","last_peered":"2024-03-06T07:31:00.957733+0000","last_clean":"2024-03-06T07:31:00.957733+0000","last_became_active":"2024-03-06T07:30:59.617319+0000","last_became_peered":"2024-03-06T07:30:59.617319+0000","last_unstale":"2024-03-06T07:31:00.957733+0000","last_undegraded":"2024-03-06T07:31:00.957733+0000","last_fullsized":"2024-03-06T07:31:00.957733+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:00.787988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955459+0000","last_change":"2024-03-06T07:30:59.621598+0000","last_active":"2024-03-06T07:31:00.955459+0000","last_peered":"2024-03-06T07:31:00.955459+0000","last_clean":"2024-03-06T07:31:00.955459+0000","last_became_active":"2024-03-06T07:30:59.621359+0000","last_became_peered":"2024-03-06T07:30:59.621359+0000","last_unstale":"2024-03-06T07:31:00.955459+0000","last_undegraded":"2024-03-06T07:31:00.955459+0000","last_fullsized":"2024-03-06T07:31:00.955459+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:00.787988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615326+0000","last_change":"2024-03-06T07:30:59.602927+0000","last_active":"2024-03-06T07:31:00.615326+0000","last_peered":"2024-03-06T07:31:00.615326+0000","last_clean":"2024-03-06T07:31:00.615326+0000","last_became_active":"2024-03-06T07:30:59.602628+0000","last_became_peered":"2024-03-06T07:30:59.602628+0000","last_unstale":"2024-03-06T07:31:00.615326+0000","last_undegraded":"2024-03-06T07:31:00.615326+0000","last_fullsized":"2024-03-06T07:31:00.615326+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:30.702595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613019+0000","last_change":"2024-03-06T07:30:59.610970+0000","last_active":"2024-03-06T07:31:00.613019+0000","last_peered":"2024-03-06T07:31:00.613019+0000","last_clean":"2024-03-06T07:31:00.613019+0000","last_became_active":"2024-03-06T07:30:59.610659+0000","last_became_peered":"2024-03-06T07:30:59.610659+0000","last_unstale":"2024-03-06T07:31:00.613019+0000","last_undegraded":"2024-03-06T07:31:00.613019+0000","last_fullsized":"2024-03-06T07:31:00.613019+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:29.840575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956956+0000","last_change":"2024-03-06T07:30:59.638513+0000","last_active":"2024-03-06T07:31:00.956956+0000","last_peered":"2024-03-06T07:31:00.956956+0000","last_clean":"2024-03-06T07:31:00.956956+0000","last_became_active":"2024-03-06T07:30:59.638195+0000","last_became_peered":"2024-03-06T07:30:59.638195+0000","last_unstale":"2024-03-06T07:31:00.956956+0000","last_undegraded":"2024-03-06T07:31:00.956956+0000","last_fullsized":"2024-03-06T07:31:00.956956+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:42.431941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957924+0000","last_change":"2024-03-06T07:30:59.632763+0000","last_active":"2024-03-06T07:31:00.957924+0000","last_peered":"2024-03-06T07:31:00.957924+0000","last_clean":"2024-03-06T07:31:00.957924+0000","last_became_active":"2024-03-06T07:30:59.632501+0000","last_became_peered":"2024-03-06T07:30:59.632501+0000","last_unstale":"2024-03-06T07:31:00.957924+0000","last_undegraded":"2024-03-06T07:31:00.957924+0000","last_fullsized":"2024-03-06T07:31:00.957924+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:00.234855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.960784+0000","last_change":"2024-03-06T07:30:59.616770+0000","last_active":"2024-03-06T07:31:00.960784+0000","last_peered":"2024-03-06T07:31:00.960784+0000","last_clean":"2024-03-06T07:31:00.960784+0000","last_became_active":"2024-03-06T07:30:59.616524+0000","last_became_peered":"2024-03-06T07:30:59.616524+0000","last_unstale":"2024-03-06T07:31:00.960784+0000","last_undegraded":"2024-03-06T07:31:00.960784+0000","last_fullsized":"2024-03-06T07:31:00.960784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:29.840575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.958050+0000","last_change":"2024-03-06T07:30:59.623470+0000","last_active":"2024-03-06T07:31:00.958050+0000","last_peered":"2024-03-06T07:31:00.958050+0000","last_clean":"2024-03-06T07:31:00.958050+0000","last_became_active":"2024-03-06T07:30:59.623176+0000","last_became_peered":"2024-03-06T07:30:59.623176+0000","last_unstale":"2024-03-06T07:31:00.958050+0000","last_undegraded":"2024-03-06T07:31:00.958050+0000","last_fullsized":"2024-03-06T07:31:00.958050+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:46.184113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955920+0000","last_change":"2024-03-06T07:30:59.611547+0000","last_active":"2024-03-06T07:31:00.955920+0000","last_peered":"2024-03-06T07:31:00.955920+0000","last_clean":"2024-03-06T07:31:00.955920+0000","last_became_active":"2024-03-06T07:30:59.609589+0000","last_became_peered":"2024-03-06T07:30:59.609589+0000","last_unstale":"2024-03-06T07:31:00.955920+0000","last_undegraded":"2024-03-06T07:31:00.955920+0000","last_fullsized":"2024-03-06T07:31:00.955920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:00.298752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955973+0000","last_change":"2024-03-06T07:30:59.614515+0000","last_active":"2024-03-06T07:31:00.955973+0000","last_peered":"2024-03-06T07:31:00.955973+0000","last_clean":"2024-03-06T07:31:00.955973+0000","last_became_active":"2024-03-06T07:30:59.614206+0000","last_became_peered":"2024-03-06T07:30:59.614206+0000","last_unstale":"2024-03-06T07:31:00.955973+0000","last_undegraded":"2024-03-06T07:31:00.955973+0000","last_fullsized":"2024-03-06T07:31:00.955973+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:41.795680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957391+0000","last_change":"2024-03-06T07:30:59.630771+0000","last_active":"2024-03-06T07:31:00.957391+0000","last_peered":"2024-03-06T07:31:00.957391+0000","last_clean":"2024-03-06T07:31:00.957391+0000","last_became_active":"2024-03-06T07:30:59.630450+0000","last_became_peered":"2024-03-06T07:30:59.630450+0000","last_unstale":"2024-03-06T07:31:00.957391+0000","last_undegraded":"2024-03-06T07:31:00.957391+0000","last_fullsized":"2024-03-06T07:31:00.957391+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:25.161032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614685+0000","last_change":"2024-03-06T07:30:59.605613+0000","last_active":"2024-03-06T07:31:00.614685+0000","last_peered":"2024-03-06T07:31:00.614685+0000","last_clean":"2024-03-06T07:31:00.614685+0000","last_became_active":"2024-03-06T07:30:59.605322+0000","last_became_peered":"2024-03-06T07:30:59.605322+0000","last_unstale":"2024-03-06T07:31:00.614685+0000","last_undegraded":"2024-03-06T07:31:00.614685+0000","last_fullsized":"2024-03-06T07:31:00.614685+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:43.926094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957539+0000","last_change":"2024-03-06T07:30:59.631503+0000","last_active":"2024-03-06T07:31:00.957539+0000","last_peered":"2024-03-06T07:31:00.957539+0000","last_clean":"2024-03-06T07:31:00.957539+0000","last_became_active":"2024-03-06T07:30:59.630987+0000","last_became_peered":"2024-03-06T07:30:59.630987+0000","last_unstale":"2024-03-06T07:31:00.957539+0000","last_undegraded":"2024-03-06T07:31:00.957539+0000","last_fullsized":"2024-03-06T07:31:00.957539+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:14.567520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618757+0000","last_change":"2024-03-06T07:30:59.611887+0000","last_active":"2024-03-06T07:31:00.618757+0000","last_peered":"2024-03-06T07:31:00.618757+0000","last_clean":"2024-03-06T07:31:00.618757+0000","last_became_active":"2024-03-06T07:30:59.611454+0000","last_became_peered":"2024-03-06T07:30:59.611454+0000","last_unstale":"2024-03-06T07:31:00.618757+0000","last_undegraded":"2024-03-06T07:31:00.618757+0000","last_fullsized":"2024-03-06T07:31:00.618757+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:42.431941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.619408+0000","last_change":"2024-03-06T07:30:59.622368+0000","last_active":"2024-03-06T07:31:00.619408+0000","last_peered":"2024-03-06T07:31:00.619408+0000","last_clean":"2024-03-06T07:31:00.619408+0000","last_became_active":"2024-03-06T07:30:59.622088+0000","last_became_peered":"2024-03-06T07:30:59.622088+0000","last_unstale":"2024-03-06T07:31:00.619408+0000","last_undegraded":"2024-03-06T07:31:00.619408+0000","last_fullsized":"2024-03-06T07:31:00.619408+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:56.249740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957701+0000","last_change":"2024-03-06T07:30:59.637731+0000","last_active":"2024-03-06T07:31:00.957701+0000","last_peered":"2024-03-06T07:31:00.957701+0000","last_clean":"2024-03-06T07:31:00.957701+0000","last_became_active":"2024-03-06T07:30:59.637366+0000","last_became_peered":"2024-03-06T07:30:59.637366+0000","last_unstale":"2024-03-06T07:31:00.957701+0000","last_undegraded":"2024-03-06T07:31:00.957701+0000","last_fullsized":"2024-03-06T07:31:00.957701+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:16.046756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956166+0000","last_change":"2024-03-06T07:30:59.626017+0000","last_active":"2024-03-06T07:31:00.956166+0000","last_peered":"2024-03-06T07:31:00.956166+0000","last_clean":"2024-03-06T07:31:00.956166+0000","last_became_active":"2024-03-06T07:30:59.625156+0000","last_became_peered":"2024-03-06T07:30:59.625156+0000","last_unstale":"2024-03-06T07:31:00.956166+0000","last_undegraded":"2024-03-06T07:31:00.956166+0000","last_fullsized":"2024-03-06T07:31:00.956166+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:43.339246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614866+0000","last_change":"2024-03-06T07:30:59.620943+0000","last_active":"2024-03-06T07:31:00.614866+0000","last_peered":"2024-03-06T07:31:00.614866+0000","last_clean":"2024-03-06T07:31:00.614866+0000","last_became_active":"2024-03-06T07:30:59.615965+0000","last_became_peered":"2024-03-06T07:30:59.615965+0000","last_unstale":"2024-03-06T07:31:00.614866+0000","last_undegraded":"2024-03-06T07:31:00.614866+0000","last_fullsized":"2024-03-06T07:31:00.614866+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:45.222498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957581+0000","last_change":"2024-03-06T07:30:59.634998+0000","last_active":"2024-03-06T07:31:00.957581+0000","last_peered":"2024-03-06T07:31:00.957581+0000","last_clean":"2024-03-06T07:31:00.957581+0000","last_became_active":"2024-03-06T07:30:59.634656+0000","last_became_peered":"2024-03-06T07:30:59.634656+0000","last_unstale":"2024-03-06T07:31:00.957581+0000","last_undegraded":"2024-03-06T07:31:00.957581+0000","last_fullsized":"2024-03-06T07:31:00.957581+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:56.249740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955752+0000","last_change":"2024-03-06T07:30:59.611906+0000","last_active":"2024-03-06T07:31:00.955752+0000","last_peered":"2024-03-06T07:31:00.955752+0000","last_clean":"2024-03-06T07:31:00.955752+0000","last_became_active":"2024-03-06T07:30:59.611526+0000","last_became_peered":"2024-03-06T07:30:59.611526+0000","last_unstale":"2024-03-06T07:31:00.955752+0000","last_undegraded":"2024-03-06T07:31:00.955752+0000","last_fullsized":"2024-03-06T07:31:00.955752+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:45.655431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957618+0000","last_change":"2024-03-06T07:30:59.610883+0000","last_active":"2024-03-06T07:31:00.957618+0000","last_peered":"2024-03-06T07:31:00.957618+0000","last_clean":"2024-03-06T07:31:00.957618+0000","last_became_active":"2024-03-06T07:30:59.610559+0000","last_became_peered":"2024-03-06T07:30:59.610559+0000","last_unstale":"2024-03-06T07:31:00.957618+0000","last_undegraded":"2024-03-06T07:31:00.957618+0000","last_fullsized":"2024-03-06T07:31:00.957618+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:59.674584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.958057+0000","last_change":"2024-03-06T07:30:59.624369+0000","last_active":"2024-03-06T07:31:00.958057+0000","last_peered":"2024-03-06T07:31:00.958057+0000","last_clean":"2024-03-06T07:31:00.958057+0000","last_became_active":"2024-03-06T07:30:59.617072+0000","last_became_peered":"2024-03-06T07:30:59.617072+0000","last_unstale":"2024-03-06T07:31:00.958057+0000","last_undegraded":"2024-03-06T07:31:00.958057+0000","last_fullsized":"2024-03-06T07:31:00.958057+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:44.999628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956103+0000","last_change":"2024-03-06T07:30:59.617221+0000","last_active":"2024-03-06T07:31:00.956103+0000","last_peered":"2024-03-06T07:31:00.956103+0000","last_clean":"2024-03-06T07:31:00.956103+0000","last_became_active":"2024-03-06T07:30:59.616920+0000","last_became_peered":"2024-03-06T07:30:59.616920+0000","last_unstale":"2024-03-06T07:31:00.956103+0000","last_undegraded":"2024-03-06T07:31:00.956103+0000","last_fullsized":"2024-03-06T07:31:00.956103+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:44.489373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618892+0000","last_change":"2024-03-06T07:30:59.608577+0000","last_active":"2024-03-06T07:31:00.618892+0000","last_peered":"2024-03-06T07:31:00.618892+0000","last_clean":"2024-03-06T07:31:00.618892+0000","last_became_active":"2024-03-06T07:30:59.608264+0000","last_became_peered":"2024-03-06T07:30:59.608264+0000","last_unstale":"2024-03-06T07:31:00.618892+0000","last_undegraded":"2024-03-06T07:31:00.618892+0000","last_fullsized":"2024-03-06T07:31:00.618892+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:23.309976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956317+0000","last_change":"2024-03-06T07:30:59.610223+0000","last_active":"2024-03-06T07:31:00.956317+0000","last_peered":"2024-03-06T07:31:00.956317+0000","last_clean":"2024-03-06T07:31:00.956317+0000","last_became_active":"2024-03-06T07:30:59.609880+0000","last_became_peered":"2024-03-06T07:30:59.609880+0000","last_unstale":"2024-03-06T07:31:00.956317+0000","last_undegraded":"2024-03-06T07:31:00.956317+0000","last_fullsized":"2024-03-06T07:31:00.956317+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.041444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957399+0000","last_change":"2024-03-06T07:30:59.616953+0000","last_active":"2024-03-06T07:31:00.957399+0000","last_peered":"2024-03-06T07:31:00.957399+0000","last_clean":"2024-03-06T07:31:00.957399+0000","last_became_active":"2024-03-06T07:30:59.616407+0000","last_became_peered":"2024-03-06T07:30:59.616407+0000","last_unstale":"2024-03-06T07:31:00.957399+0000","last_undegraded":"2024-03-06T07:31:00.957399+0000","last_fullsized":"2024-03-06T07:31:00.957399+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.366376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618565+0000","last_change":"2024-03-06T07:30:59.614841+0000","last_active":"2024-03-06T07:31:00.618565+0000","last_peered":"2024-03-06T07:31:00.618565+0000","last_clean":"2024-03-06T07:31:00.618565+0000","last_became_active":"2024-03-06T07:30:59.613185+0000","last_became_peered":"2024-03-06T07:30:59.613185+0000","last_unstale":"2024-03-06T07:31:00.618565+0000","last_undegraded":"2024-03-06T07:31:00.618565+0000","last_fullsized":"2024-03-06T07:31:00.618565+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:38.863533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614529+0000","last_change":"2024-03-06T07:30:59.592409+0000","last_active":"2024-03-06T07:31:00.614529+0000","last_peered":"2024-03-06T07:31:00.614529+0000","last_clean":"2024-03-06T07:31:00.614529+0000","last_became_active":"2024-03-06T07:30:59.591659+0000","last_became_peered":"2024-03-06T07:30:59.591659+0000","last_unstale":"2024-03-06T07:31:00.614529+0000","last_undegraded":"2024-03-06T07:31:00.614529+0000","last_fullsized":"2024-03-06T07:31:00.614529+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:29.181556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956555+0000","last_change":"2024-03-06T07:30:59.608723+0000","last_active":"2024-03-06T07:31:00.956555+0000","last_peered":"2024-03-06T07:31:00.956555+0000","last_clean":"2024-03-06T07:31:00.956555+0000","last_became_active":"2024-03-06T07:30:59.608107+0000","last_became_peered":"2024-03-06T07:30:59.608107+0000","last_unstale":"2024-03-06T07:31:00.956555+0000","last_undegraded":"2024-03-06T07:31:00.956555+0000","last_fullsized":"2024-03-06T07:31:00.956555+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:07.562444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.953183+0000","last_change":"2024-03-06T07:30:59.597909+0000","last_active":"2024-03-06T07:31:00.953183+0000","last_peered":"2024-03-06T07:31:00.953183+0000","last_clean":"2024-03-06T07:31:00.953183+0000","last_became_active":"2024-03-06T07:30:59.597605+0000","last_became_peered":"2024-03-06T07:30:59.597605+0000","last_unstale":"2024-03-06T07:31:00.953183+0000","last_undegraded":"2024-03-06T07:31:00.953183+0000","last_fullsized":"2024-03-06T07:31:00.953183+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:59.674584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615009+0000","last_change":"2024-03-06T07:30:58.587558+0000","last_active":"2024-03-06T07:31:00.615009+0000","last_peered":"2024-03-06T07:31:00.615009+0000","last_clean":"2024-03-06T07:31:00.615009+0000","last_became_active":"2024-03-06T07:30:58.587122+0000","last_became_peered":"2024-03-06T07:30:58.587122+0000","last_unstale":"2024-03-06T07:31:00.615009+0000","last_undegraded":"2024-03-06T07:31:00.615009+0000","last_fullsized":"2024-03-06T07:31:00.615009+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:59.274536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.958124+0000","last_change":"2024-03-06T07:30:59.622384+0000","last_active":"2024-03-06T07:31:00.958124+0000","last_peered":"2024-03-06T07:31:00.958124+0000","last_clean":"2024-03-06T07:31:00.958124+0000","last_became_active":"2024-03-06T07:30:59.622072+0000","last_became_peered":"2024-03-06T07:30:59.622072+0000","last_unstale":"2024-03-06T07:31:00.958124+0000","last_undegraded":"2024-03-06T07:31:00.958124+0000","last_fullsized":"2024-03-06T07:31:00.958124+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:04.993198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.619507+0000","last_change":"2024-03-06T07:30:58.587171+0000","last_active":"2024-03-06T07:31:00.619507+0000","last_peered":"2024-03-06T07:31:00.619507+0000","last_clean":"2024-03-06T07:31:00.619507+0000","last_became_active":"2024-03-06T07:30:58.586883+0000","last_became_peered":"2024-03-06T07:30:58.586883+0000","last_unstale":"2024-03-06T07:31:00.619507+0000","last_undegraded":"2024-03-06T07:31:00.619507+0000","last_fullsized":"2024-03-06T07:31:00.619507+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:44.631214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956345+0000","last_change":"2024-03-06T07:30:59.589917+0000","last_active":"2024-03-06T07:31:00.956345+0000","last_peered":"2024-03-06T07:31:00.956345+0000","last_clean":"2024-03-06T07:31:00.956345+0000","last_became_active":"2024-03-06T07:30:59.589078+0000","last_became_peered":"2024-03-06T07:30:59.589078+0000","last_unstale":"2024-03-06T07:31:00.956345+0000","last_undegraded":"2024-03-06T07:31:00.956345+0000","last_fullsized":"2024-03-06T07:31:00.956345+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:43.554296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.953450+0000","last_change":"2024-03-06T07:30:58.587910+0000","last_active":"2024-03-06T07:31:00.953450+0000","last_peered":"2024-03-06T07:31:00.953450+0000","last_clean":"2024-03-06T07:31:00.953450+0000","last_became_active":"2024-03-06T07:30:58.587087+0000","last_became_peered":"2024-03-06T07:30:58.587087+0000","last_unstale":"2024-03-06T07:31:00.953450+0000","last_undegraded":"2024-03-06T07:31:00.953450+0000","last_fullsized":"2024-03-06T07:31:00.953450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:19.523114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.953407+0000","last_change":"2024-03-06T07:30:59.617925+0000","last_active":"2024-03-06T07:31:00.953407+0000","last_peered":"2024-03-06T07:31:00.953407+0000","last_clean":"2024-03-06T07:31:00.953407+0000","last_became_active":"2024-03-06T07:30:59.617569+0000","last_became_peered":"2024-03-06T07:30:59.617569+0000","last_unstale":"2024-03-06T07:31:00.953407+0000","last_undegraded":"2024-03-06T07:31:00.953407+0000","last_fullsized":"2024-03-06T07:31:00.953407+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:16.046756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613202+0000","last_change":"2024-03-06T07:30:58.591529+0000","last_active":"2024-03-06T07:31:00.613202+0000","last_peered":"2024-03-06T07:31:00.613202+0000","last_clean":"2024-03-06T07:31:00.613202+0000","last_became_active":"2024-03-06T07:30:58.590513+0000","last_became_peered":"2024-03-06T07:30:58.590513+0000","last_unstale":"2024-03-06T07:31:00.613202+0000","last_undegraded":"2024-03-06T07:31:00.613202+0000","last_fullsized":"2024-03-06T07:31:00.613202+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:24.136815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618383+0000","last_change":"2024-03-06T07:30:59.617177+0000","last_active":"2024-03-06T07:31:00.618383+0000","last_peered":"2024-03-06T07:31:00.618383+0000","last_clean":"2024-03-06T07:31:00.618383+0000","last_became_active":"2024-03-06T07:30:59.616840+0000","last_became_peered":"2024-03-06T07:30:59.616840+0000","last_unstale":"2024-03-06T07:31:00.618383+0000","last_undegraded":"2024-03-06T07:31:00.618383+0000","last_fullsized":"2024-03-06T07:31:00.618383+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:59.265872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615040+0000","last_change":"2024-03-06T07:30:58.575068+0000","last_active":"2024-03-06T07:31:00.615040+0000","last_peered":"2024-03-06T07:31:00.615040+0000","last_clean":"2024-03-06T07:31:00.615040+0000","last_became_active":"2024-03-06T07:30:58.574795+0000","last_became_peered":"2024-03-06T07:30:58.574795+0000","last_unstale":"2024-03-06T07:31:00.615040+0000","last_undegraded":"2024-03-06T07:31:00.615040+0000","last_fullsized":"2024-03-06T07:31:00.615040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:59.210639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957951+0000","last_change":"2024-03-06T07:30:59.611584+0000","last_active":"2024-03-06T07:31:00.957951+0000","last_peered":"2024-03-06T07:31:00.957951+0000","last_clean":"2024-03-06T07:31:00.957951+0000","last_became_active":"2024-03-06T07:30:59.610807+0000","last_became_peered":"2024-03-06T07:30:59.610807+0000","last_unstale":"2024-03-06T07:31:00.957951+0000","last_undegraded":"2024-03-06T07:31:00.957951+0000","last_fullsized":"2024-03-06T07:31:00.957951+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.041444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618593+0000","last_change":"2024-03-06T07:30:58.583524+0000","last_active":"2024-03-06T07:31:00.618593+0000","last_peered":"2024-03-06T07:31:00.618593+0000","last_clean":"2024-03-06T07:31:00.618593+0000","last_became_active":"2024-03-06T07:30:58.583111+0000","last_became_peered":"2024-03-06T07:30:58.583111+0000","last_unstale":"2024-03-06T07:31:00.618593+0000","last_undegraded":"2024-03-06T07:31:00.618593+0000","last_fullsized":"2024-03-06T07:31:00.618593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:47.797331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956919+0000","last_change":"2024-03-06T07:30:59.621320+0000","last_active":"2024-03-06T07:31:00.956919+0000","last_peered":"2024-03-06T07:31:00.956919+0000","last_clean":"2024-03-06T07:31:00.956919+0000","last_became_active":"2024-03-06T07:30:59.620991+0000","last_became_peered":"2024-03-06T07:30:59.620991+0000","last_unstale":"2024-03-06T07:31:00.956919+0000","last_undegraded":"2024-03-06T07:31:00.956919+0000","last_fullsized":"2024-03-06T07:31:00.956919+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:00.163307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.954936+0000","last_change":"2024-03-06T07:30:58.584004+0000","last_active":"2024-03-06T07:31:00.954936+0000","last_peered":"2024-03-06T07:31:00.954936+0000","last_clean":"2024-03-06T07:31:00.954936+0000","last_became_active":"2024-03-06T07:30:58.583703+0000","last_became_peered":"2024-03-06T07:30:58.583703+0000","last_unstale":"2024-03-06T07:31:00.954936+0000","last_undegraded":"2024-03-06T07:31:00.954936+0000","last_fullsized":"2024-03-06T07:31:00.954936+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:43.975412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955784+0000","last_change":"2024-03-06T07:30:59.614538+0000","last_active":"2024-03-06T07:31:00.955784+0000","last_peered":"2024-03-06T07:31:00.955784+0000","last_clean":"2024-03-06T07:31:00.955784+0000","last_became_active":"2024-03-06T07:30:59.611942+0000","last_became_peered":"2024-03-06T07:30:59.611942+0000","last_unstale":"2024-03-06T07:31:00.955784+0000","last_undegraded":"2024-03-06T07:31:00.955784+0000","last_fullsized":"2024-03-06T07:31:00.955784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:48.138330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956974+0000","last_change":"2024-03-06T07:30:58.578570+0000","last_active":"2024-03-06T07:31:00.956974+0000","last_peered":"2024-03-06T07:31:00.956974+0000","last_clean":"2024-03-06T07:31:00.956974+0000","last_became_active":"2024-03-06T07:30:58.578277+0000","last_became_peered":"2024-03-06T07:30:58.578277+0000","last_unstale":"2024-03-06T07:31:00.956974+0000","last_undegraded":"2024-03-06T07:31:00.956974+0000","last_fullsized":"2024-03-06T07:31:00.956974+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:06.538228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.619394+0000","last_change":"2024-03-06T07:30:59.617264+0000","last_active":"2024-03-06T07:31:00.619394+0000","last_peered":"2024-03-06T07:31:00.619394+0000","last_clean":"2024-03-06T07:31:00.619394+0000","last_became_active":"2024-03-06T07:30:59.616906+0000","last_became_peered":"2024-03-06T07:30:59.616906+0000","last_unstale":"2024-03-06T07:31:00.619394+0000","last_undegraded":"2024-03-06T07:31:00.619394+0000","last_fullsized":"2024-03-06T07:31:00.619394+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:25.161032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.619274+0000","last_change":"2024-03-06T07:30:58.581154+0000","last_active":"2024-03-06T07:31:00.619274+0000","last_peered":"2024-03-06T07:31:00.619274+0000","last_clean":"2024-03-06T07:31:00.619274+0000","last_became_active":"2024-03-06T07:30:58.580036+0000","last_became_peered":"2024-03-06T07:30:58.580036+0000","last_unstale":"2024-03-06T07:31:00.619274+0000","last_undegraded":"2024-03-06T07:31:00.619274+0000","last_fullsized":"2024-03-06T07:31:00.619274+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:19.523114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614064+0000","last_change":"2024-03-06T07:30:59.614262+0000","last_active":"2024-03-06T07:31:00.614064+0000","last_peered":"2024-03-06T07:31:00.614064+0000","last_clean":"2024-03-06T07:31:00.614064+0000","last_became_active":"2024-03-06T07:30:59.613816+0000","last_became_peered":"2024-03-06T07:30:59.613816+0000","last_unstale":"2024-03-06T07:31:00.614064+0000","last_undegraded":"2024-03-06T07:31:00.614064+0000","last_fullsized":"2024-03-06T07:31:00.614064+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:42.321757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.961037+0000","last_change":"2024-03-06T07:30:58.586121+0000","last_active":"2024-03-06T07:31:00.961037+0000","last_peered":"2024-03-06T07:31:00.961037+0000","last_clean":"2024-03-06T07:31:00.961037+0000","last_became_active":"2024-03-06T07:30:58.585821+0000","last_became_peered":"2024-03-06T07:30:58.585821+0000","last_unstale":"2024-03-06T07:31:00.961037+0000","last_undegraded":"2024-03-06T07:31:00.961037+0000","last_fullsized":"2024-03-06T07:31:00.961037+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:42.315029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615136+0000","last_change":"2024-03-06T07:30:59.613693+0000","last_active":"2024-03-06T07:31:00.615136+0000","last_peered":"2024-03-06T07:31:00.615136+0000","last_clean":"2024-03-06T07:31:00.615136+0000","last_became_active":"2024-03-06T07:30:59.613341+0000","last_became_peered":"2024-03-06T07:30:59.613341+0000","last_unstale":"2024-03-06T07:31:00.615136+0000","last_undegraded":"2024-03-06T07:31:00.615136+0000","last_fullsized":"2024-03-06T07:31:00.615136+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:45.948254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":17,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613913+0000","last_change":"2024-03-06T07:30:58.591420+0000","last_active":"2024-03-06T07:31:00.613913+0000","last_peered":"2024-03-06T07:31:00.613913+0000","last_clean":"2024-03-06T07:31:00.613913+0000","last_became_active":"2024-03-06T07:30:58.591116+0000","last_became_peered":"2024-03-06T07:30:58.591116+0000","last_unstale":"2024-03-06T07:31:00.613913+0000","last_undegraded":"2024-03-06T07:31:00.613913+0000","last_fullsized":"2024-03-06T07:31:00.613913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:41.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613281+0000","last_change":"2024-03-06T07:30:59.614140+0000","last_active":"2024-03-06T07:31:00.613281+0000","last_peered":"2024-03-06T07:31:00.613281+0000","last_clean":"2024-03-06T07:31:00.613281+0000","last_became_active":"2024-03-06T07:30:59.613818+0000","last_became_peered":"2024-03-06T07:30:59.613818+0000","last_unstale":"2024-03-06T07:31:00.613281+0000","last_undegraded":"2024-03-06T07:31:00.613281+0000","last_fullsized":"2024-03-06T07:31:00.613281+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:44.061449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956206+0000","last_change":"2024-03-06T07:30:58.583972+0000","last_active":"2024-03-06T07:31:00.956206+0000","last_peered":"2024-03-06T07:31:00.956206+0000","last_clean":"2024-03-06T07:31:00.956206+0000","last_became_active":"2024-03-06T07:30:58.583702+0000","last_became_peered":"2024-03-06T07:30:58.583702+0000","last_unstale":"2024-03-06T07:31:00.956206+0000","last_undegraded":"2024-03-06T07:31:00.956206+0000","last_fullsized":"2024-03-06T07:31:00.956206+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:41.962362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956838+0000","last_change":"2024-03-06T07:30:59.631862+0000","last_active":"2024-03-06T07:31:00.956838+0000","last_peered":"2024-03-06T07:31:00.956838+0000","last_clean":"2024-03-06T07:31:00.956838+0000","last_became_active":"2024-03-06T07:30:59.630742+0000","last_became_peered":"2024-03-06T07:30:59.630742+0000","last_unstale":"2024-03-06T07:31:00.956838+0000","last_undegraded":"2024-03-06T07:31:00.956838+0000","last_fullsized":"2024-03-06T07:31:00.956838+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:58.953958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615125+0000","last_change":"2024-03-06T07:30:58.584388+0000","last_active":"2024-03-06T07:31:00.615125+0000","last_peered":"2024-03-06T07:31:00.615125+0000","last_clean":"2024-03-06T07:31:00.615125+0000","last_became_active":"2024-03-06T07:30:58.582897+0000","last_became_peered":"2024-03-06T07:30:58.582897+0000","last_unstale":"2024-03-06T07:31:00.615125+0000","last_undegraded":"2024-03-06T07:31:00.615125+0000","last_fullsized":"2024-03-06T07:31:00.615125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:45.159897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957647+0000","last_change":"2024-03-06T07:30:59.614738+0000","last_active":"2024-03-06T07:31:00.957647+0000","last_peered":"2024-03-06T07:31:00.957647+0000","last_clean":"2024-03-06T07:31:00.957647+0000","last_became_active":"2024-03-06T07:30:59.614474+0000","last_became_peered":"2024-03-06T07:30:59.614474+0000","last_unstale":"2024-03-06T07:31:00.957647+0000","last_undegraded":"2024-03-06T07:31:00.957647+0000","last_fullsized":"2024-03-06T07:31:00.957647+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:51.422603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613418+0000","last_change":"2024-03-06T07:30:58.582112+0000","last_active":"2024-03-06T07:31:00.613418+0000","last_peered":"2024-03-06T07:31:00.613418+0000","last_clean":"2024-03-06T07:31:00.613418+0000","last_became_active":"2024-03-06T07:30:58.581787+0000","last_became_peered":"2024-03-06T07:30:58.581787+0000","last_unstale":"2024-03-06T07:31:00.613418+0000","last_undegraded":"2024-03-06T07:31:00.613418+0000","last_fullsized":"2024-03-06T07:31:00.613418+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:37.839317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614560+0000","last_change":"2024-03-06T07:30:59.613140+0000","last_active":"2024-03-06T07:31:00.614560+0000","last_peered":"2024-03-06T07:31:00.614560+0000","last_clean":"2024-03-06T07:31:00.614560+0000","last_became_active":"2024-03-06T07:30:59.612784+0000","last_became_peered":"2024-03-06T07:30:59.612784+0000","last_unstale":"2024-03-06T07:31:00.614560+0000","last_undegraded":"2024-03-06T07:31:00.614560+0000","last_fullsized":"2024-03-06T07:31:00.614560+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:58.762985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614602+0000","last_change":"2024-03-06T07:30:58.580228+0000","last_active":"2024-03-06T07:31:00.614602+0000","last_peered":"2024-03-06T07:31:00.614602+0000","last_clean":"2024-03-06T07:31:00.614602+0000","last_became_active":"2024-03-06T07:30:58.579908+0000","last_became_peered":"2024-03-06T07:30:58.579908+0000","last_unstale":"2024-03-06T07:31:00.614602+0000","last_undegraded":"2024-03-06T07:31:00.614602+0000","last_fullsized":"2024-03-06T07:31:00.614602+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:59.763771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613511+0000","last_change":"2024-03-06T07:30:59.619129+0000","last_active":"2024-03-06T07:31:00.613511+0000","last_peered":"2024-03-06T07:31:00.613511+0000","last_clean":"2024-03-06T07:31:00.613511+0000","last_became_active":"2024-03-06T07:30:59.618296+0000","last_became_peered":"2024-03-06T07:30:59.618296+0000","last_unstale":"2024-03-06T07:31:00.613511+0000","last_undegraded":"2024-03-06T07:31:00.613511+0000","last_fullsized":"2024-03-06T07:31:00.613511+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:55.206422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957042+0000","last_change":"2024-03-06T07:30:58.589724+0000","last_active":"2024-03-06T07:31:00.957042+0000","last_peered":"2024-03-06T07:31:00.957042+0000","last_clean":"2024-03-06T07:31:00.957042+0000","last_became_active":"2024-03-06T07:30:58.589374+0000","last_became_peered":"2024-03-06T07:30:58.589374+0000","last_unstale":"2024-03-06T07:31:00.957042+0000","last_undegraded":"2024-03-06T07:31:00.957042+0000","last_fullsized":"2024-03-06T07:31:00.957042+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:47.797331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956529+0000","last_change":"2024-03-06T07:30:59.625592+0000","last_active":"2024-03-06T07:31:00.956529+0000","last_peered":"2024-03-06T07:31:00.956529+0000","last_clean":"2024-03-06T07:31:00.956529+0000","last_became_active":"2024-03-06T07:30:59.625183+0000","last_became_peered":"2024-03-06T07:30:59.625183+0000","last_unstale":"2024-03-06T07:31:00.956529+0000","last_undegraded":"2024-03-06T07:31:00.956529+0000","last_fullsized":"2024-03-06T07:31:00.956529+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:59.047248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956433+0000","last_change":"2024-03-06T07:30:58.581408+0000","last_active":"2024-03-06T07:31:00.956433+0000","last_peered":"2024-03-06T07:31:00.956433+0000","last_clean":"2024-03-06T07:31:00.956433+0000","last_became_active":"2024-03-06T07:30:58.580043+0000","last_became_peered":"2024-03-06T07:30:58.580043+0000","last_unstale":"2024-03-06T07:31:00.956433+0000","last_undegraded":"2024-03-06T07:31:00.956433+0000","last_fullsized":"2024-03-06T07:31:00.956433+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:22.513623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957973+0000","last_change":"2024-03-06T07:30:59.611972+0000","last_active":"2024-03-06T07:31:00.957973+0000","last_peered":"2024-03-06T07:31:00.957973+0000","last_clean":"2024-03-06T07:31:00.957973+0000","last_became_active":"2024-03-06T07:30:59.611679+0000","last_became_peered":"2024-03-06T07:30:59.611679+0000","last_unstale":"2024-03-06T07:31:00.957973+0000","last_undegraded":"2024-03-06T07:31:00.957973+0000","last_fullsized":"2024-03-06T07:31:00.957973+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:43.554296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.619581+0000","last_change":"2024-03-06T07:30:58.580593+0000","last_active":"2024-03-06T07:31:00.619581+0000","last_peered":"2024-03-06T07:31:00.619581+0000","last_clean":"2024-03-06T07:31:00.619581+0000","last_became_active":"2024-03-06T07:30:58.580197+0000","last_became_peered":"2024-03-06T07:30:58.580197+0000","last_unstale":"2024-03-06T07:31:00.619581+0000","last_undegraded":"2024-03-06T07:31:00.619581+0000","last_fullsized":"2024-03-06T07:31:00.619581+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:47.114114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957096+0000","last_change":"2024-03-06T07:30:59.613128+0000","last_active":"2024-03-06T07:31:00.957096+0000","last_peered":"2024-03-06T07:31:00.957096+0000","last_clean":"2024-03-06T07:31:00.957096+0000","last_became_active":"2024-03-06T07:30:59.612848+0000","last_became_peered":"2024-03-06T07:30:59.612848+0000","last_unstale":"2024-03-06T07:31:00.957096+0000","last_undegraded":"2024-03-06T07:31:00.957096+0000","last_fullsized":"2024-03-06T07:31:00.957096+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:31.611560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955992+0000","last_change":"2024-03-06T07:30:58.583828+0000","last_active":"2024-03-06T07:31:00.955992+0000","last_peered":"2024-03-06T07:31:00.955992+0000","last_clean":"2024-03-06T07:31:00.955992+0000","last_became_active":"2024-03-06T07:30:58.583498+0000","last_became_peered":"2024-03-06T07:30:58.583498+0000","last_unstale":"2024-03-06T07:31:00.955992+0000","last_undegraded":"2024-03-06T07:31:00.955992+0000","last_fullsized":"2024-03-06T07:31:00.955992+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:03.968982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956128+0000","last_change":"2024-03-06T07:30:59.618844+0000","last_active":"2024-03-06T07:31:00.956128+0000","last_peered":"2024-03-06T07:31:00.956128+0000","last_clean":"2024-03-06T07:31:00.956128+0000","last_became_active":"2024-03-06T07:30:59.618393+0000","last_became_peered":"2024-03-06T07:30:59.618393+0000","last_unstale":"2024-03-06T07:31:00.956128+0000","last_undegraded":"2024-03-06T07:31:00.956128+0000","last_fullsized":"2024-03-06T07:31:00.956128+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:20.547331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615154+0000","last_change":"2024-03-06T07:30:58.590504+0000","last_active":"2024-03-06T07:31:00.615154+0000","last_peered":"2024-03-06T07:31:00.615154+0000","last_clean":"2024-03-06T07:31:00.615154+0000","last_became_active":"2024-03-06T07:30:58.590154+0000","last_became_peered":"2024-03-06T07:30:58.590154+0000","last_unstale":"2024-03-06T07:31:00.615154+0000","last_undegraded":"2024-03-06T07:31:00.615154+0000","last_fullsized":"2024-03-06T07:31:00.615154+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:42.530080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614727+0000","last_change":"2024-03-06T07:30:59.612146+0000","last_active":"2024-03-06T07:31:00.614727+0000","last_peered":"2024-03-06T07:31:00.614727+0000","last_clean":"2024-03-06T07:31:00.614727+0000","last_became_active":"2024-03-06T07:30:59.611804+0000","last_became_peered":"2024-03-06T07:30:59.611804+0000","last_unstale":"2024-03-06T07:31:00.614727+0000","last_undegraded":"2024-03-06T07:31:00.614727+0000","last_fullsized":"2024-03-06T07:31:00.614727+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:46.688197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615265+0000","last_change":"2024-03-06T07:30:58.587722+0000","last_active":"2024-03-06T07:31:00.615265+0000","last_peered":"2024-03-06T07:31:00.615265+0000","last_clean":"2024-03-06T07:31:00.615265+0000","last_became_active":"2024-03-06T07:30:58.585157+0000","last_became_peered":"2024-03-06T07:30:58.585157+0000","last_unstale":"2024-03-06T07:31:00.615265+0000","last_undegraded":"2024-03-06T07:31:00.615265+0000","last_fullsized":"2024-03-06T07:31:00.615265+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:28.816358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.953031+0000","last_change":"2024-03-06T07:30:59.601587+0000","last_active":"2024-03-06T07:31:00.953031+0000","last_peered":"2024-03-06T07:31:00.953031+0000","last_clean":"2024-03-06T07:31:00.953031+0000","last_became_active":"2024-03-06T07:30:59.600957+0000","last_became_peered":"2024-03-06T07:30:59.600957+0000","last_unstale":"2024-03-06T07:31:00.953031+0000","last_undegraded":"2024-03-06T07:31:00.953031+0000","last_fullsized":"2024-03-06T07:31:00.953031+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:14.567520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.955090+0000","last_change":"2024-03-06T07:30:58.583005+0000","last_active":"2024-03-06T07:31:00.955090+0000","last_peered":"2024-03-06T07:31:00.955090+0000","last_clean":"2024-03-06T07:31:00.955090+0000","last_became_active":"2024-03-06T07:30:58.582678+0000","last_became_peered":"2024-03-06T07:30:58.582678+0000","last_unstale":"2024-03-06T07:31:00.955090+0000","last_undegraded":"2024-03-06T07:31:00.955090+0000","last_fullsized":"2024-03-06T07:31:00.955090+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:42.138180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956770+0000","last_change":"2024-03-06T07:30:59.618334+0000","last_active":"2024-03-06T07:31:00.956770+0000","last_peered":"2024-03-06T07:31:00.956770+0000","last_clean":"2024-03-06T07:31:00.956770+0000","last_became_active":"2024-03-06T07:30:59.609512+0000","last_became_peered":"2024-03-06T07:30:59.609512+0000","last_unstale":"2024-03-06T07:31:00.956770+0000","last_undegraded":"2024-03-06T07:31:00.956770+0000","last_fullsized":"2024-03-06T07:31:00.956770+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:48.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957322+0000","last_change":"2024-03-06T07:30:58.588508+0000","last_active":"2024-03-06T07:31:00.957322+0000","last_peered":"2024-03-06T07:31:00.957322+0000","last_clean":"2024-03-06T07:31:00.957322+0000","last_became_active":"2024-03-06T07:30:58.588194+0000","last_became_peered":"2024-03-06T07:30:58.588194+0000","last_unstale":"2024-03-06T07:31:00.957322+0000","last_undegraded":"2024-03-06T07:31:00.957322+0000","last_fullsized":"2024-03-06T07:31:00.957322+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:58.023031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957098+0000","last_change":"2024-03-06T07:30:59.630555+0000","last_active":"2024-03-06T07:31:00.957098+0000","last_peered":"2024-03-06T07:31:00.957098+0000","last_clean":"2024-03-06T07:31:00.957098+0000","last_became_active":"2024-03-06T07:30:59.630238+0000","last_became_peered":"2024-03-06T07:30:59.630238+0000","last_unstale":"2024-03-06T07:31:00.957098+0000","last_undegraded":"2024-03-06T07:31:00.957098+0000","last_fullsized":"2024-03-06T07:31:00.957098+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:23.309976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.612410+0000","last_change":"2024-03-06T07:30:58.581483+0000","last_active":"2024-03-06T07:31:00.612410+0000","last_peered":"2024-03-06T07:31:00.612410+0000","last_clean":"2024-03-06T07:31:00.612410+0000","last_became_active":"2024-03-06T07:30:58.581156+0000","last_became_peered":"2024-03-06T07:30:58.581156+0000","last_unstale":"2024-03-06T07:31:00.612410+0000","last_undegraded":"2024-03-06T07:31:00.612410+0000","last_fullsized":"2024-03-06T07:31:00.612410+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:58.241655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.958299+0000","last_change":"2024-03-06T07:30:59.631885+0000","last_active":"2024-03-06T07:31:00.958299+0000","last_peered":"2024-03-06T07:31:00.958299+0000","last_clean":"2024-03-06T07:31:00.958299+0000","last_became_active":"2024-03-06T07:30:59.630697+0000","last_became_peered":"2024-03-06T07:30:59.630697+0000","last_unstale":"2024-03-06T07:31:00.958299+0000","last_undegraded":"2024-03-06T07:31:00.958299+0000","last_fullsized":"2024-03-06T07:31:00.958299+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:43.162396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"46'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615491+0000","last_change":"2024-03-06T07:30:58.582590+0000","last_active":"2024-03-06T07:31:00.615491+0000","last_peered":"2024-03-06T07:31:00.615491+0000","last_clean":"2024-03-06T07:31:00.615491+0000","last_became_active":"2024-03-06T07:30:58.582299+0000","last_became_peered":"2024-03-06T07:30:58.582299+0000","last_unstale":"2024-03-06T07:31:00.615491+0000","last_undegraded":"2024-03-06T07:31:00.615491+0000","last_fullsized":"2024-03-06T07:31:00.615491+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:44.017228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957864+0000","last_change":"2024-03-06T07:30:59.617594+0000","last_active":"2024-03-06T07:31:00.957864+0000","last_peered":"2024-03-06T07:31:00.957864+0000","last_clean":"2024-03-06T07:31:00.957864+0000","last_became_active":"2024-03-06T07:30:59.616953+0000","last_became_peered":"2024-03-06T07:30:59.616953+0000","last_unstale":"2024-03-06T07:31:00.957864+0000","last_undegraded":"2024-03-06T07:31:00.957864+0000","last_fullsized":"2024-03-06T07:31:00.957864+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:00.298752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614199+0000","last_change":"2024-03-06T07:30:58.589017+0000","last_active":"2024-03-06T07:31:00.614199+0000","last_peered":"2024-03-06T07:31:00.614199+0000","last_clean":"2024-03-06T07:31:00.614199+0000","last_became_active":"2024-03-06T07:30:58.588455+0000","last_became_peered":"2024-03-06T07:30:58.588455+0000","last_unstale":"2024-03-06T07:31:00.614199+0000","last_undegraded":"2024-03-06T07:31:00.614199+0000","last_fullsized":"2024-03-06T07:31:00.614199+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:22.285760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613763+0000","last_change":"2024-03-06T07:30:59.609543+0000","last_active":"2024-03-06T07:31:00.613763+0000","last_peered":"2024-03-06T07:31:00.613763+0000","last_clean":"2024-03-06T07:31:00.613763+0000","last_became_active":"2024-03-06T07:30:59.608955+0000","last_became_peered":"2024-03-06T07:30:59.608955+0000","last_unstale":"2024-03-06T07:31:00.613763+0000","last_undegraded":"2024-03-06T07:31:00.613763+0000","last_fullsized":"2024-03-06T07:31:00.613763+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:46.348100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.953475+0000","last_change":"2024-03-06T07:30:58.589583+0000","last_active":"2024-03-06T07:31:00.953475+0000","last_peered":"2024-03-06T07:31:00.953475+0000","last_clean":"2024-03-06T07:31:00.953475+0000","last_became_active":"2024-03-06T07:30:58.588877+0000","last_became_peered":"2024-03-06T07:30:58.588877+0000","last_unstale":"2024-03-06T07:31:00.953475+0000","last_undegraded":"2024-03-06T07:31:00.953475+0000","last_fullsized":"2024-03-06T07:31:00.953475+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:44.631214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.614934+0000","last_change":"2024-03-06T07:30:59.597608+0000","last_active":"2024-03-06T07:31:00.614934+0000","last_peered":"2024-03-06T07:31:00.614934+0000","last_clean":"2024-03-06T07:31:00.614934+0000","last_became_active":"2024-03-06T07:30:59.597196+0000","last_became_peered":"2024-03-06T07:30:59.597196+0000","last_unstale":"2024-03-06T07:31:00.614934+0000","last_undegraded":"2024-03-06T07:31:00.614934+0000","last_fullsized":"2024-03-06T07:31:00.614934+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:50.379177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613749+0000","last_change":"2024-03-06T07:30:58.587826+0000","last_active":"2024-03-06T07:31:00.613749+0000","last_peered":"2024-03-06T07:31:00.613749+0000","last_clean":"2024-03-06T07:31:00.613749+0000","last_became_active":"2024-03-06T07:30:58.586896+0000","last_became_peered":"2024-03-06T07:30:58.586896+0000","last_unstale":"2024-03-06T07:31:00.613749+0000","last_undegraded":"2024-03-06T07:31:00.613749+0000","last_fullsized":"2024-03-06T07:31:00.613749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:59.139091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.615051+0000","last_change":"2024-03-06T07:30:59.605927+0000","last_active":"2024-03-06T07:31:00.615051+0000","last_peered":"2024-03-06T07:31:00.615051+0000","last_clean":"2024-03-06T07:31:00.615051+0000","last_became_active":"2024-03-06T07:30:59.605632+0000","last_became_peered":"2024-03-06T07:30:59.605632+0000","last_unstale":"2024-03-06T07:31:00.615051+0000","last_undegraded":"2024-03-06T07:31:00.615051+0000","last_fullsized":"2024-03-06T07:31:00.615051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:31.310477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618744+0000","last_change":"2024-03-06T07:30:58.574613+0000","last_active":"2024-03-06T07:31:00.618744+0000","last_peered":"2024-03-06T07:31:00.618744+0000","last_clean":"2024-03-06T07:31:00.618744+0000","last_became_active":"2024-03-06T07:30:58.574321+0000","last_became_peered":"2024-03-06T07:30:58.574321+0000","last_unstale":"2024-03-06T07:31:00.618744+0000","last_undegraded":"2024-03-06T07:31:00.618744+0000","last_fullsized":"2024-03-06T07:31:00.618744+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:59.802570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.954980+0000","last_change":"2024-03-06T07:30:59.600579+0000","last_active":"2024-03-06T07:31:00.954980+0000","last_peered":"2024-03-06T07:31:00.954980+0000","last_clean":"2024-03-06T07:31:00.954980+0000","last_became_active":"2024-03-06T07:30:59.600043+0000","last_became_peered":"2024-03-06T07:30:59.600043+0000","last_unstale":"2024-03-06T07:31:00.954980+0000","last_undegraded":"2024-03-06T07:31:00.954980+0000","last_fullsized":"2024-03-06T07:31:00.954980+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:51.422603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.952870+0000","last_change":"2024-03-06T07:30:58.587877+0000","last_active":"2024-03-06T07:31:00.952870+0000","last_peered":"2024-03-06T07:31:00.952870+0000","last_clean":"2024-03-06T07:31:00.952870+0000","last_became_active":"2024-03-06T07:30:58.587615+0000","last_became_peered":"2024-03-06T07:30:58.587615+0000","last_unstale":"2024-03-06T07:31:00.952870+0000","last_undegraded":"2024-03-06T07:31:00.952870+0000","last_fullsized":"2024-03-06T07:31:00.952870+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:47.114114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.618193+0000","last_change":"2024-03-06T07:30:59.600698+0000","last_active":"2024-03-06T07:31:00.618193+0000","last_peered":"2024-03-06T07:31:00.618193+0000","last_clean":"2024-03-06T07:31:00.618193+0000","last_became_active":"2024-03-06T07:30:59.599551+0000","last_became_peered":"2024-03-06T07:30:59.599551+0000","last_unstale":"2024-03-06T07:31:00.618193+0000","last_undegraded":"2024-03-06T07:31:00.618193+0000","last_fullsized":"2024-03-06T07:31:00.618193+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:43.162396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.957077+0000","last_change":"2024-03-06T07:30:58.581534+0000","last_active":"2024-03-06T07:31:00.957077+0000","last_peered":"2024-03-06T07:31:00.957077+0000","last_clean":"2024-03-06T07:31:00.957077+0000","last_became_active":"2024-03-06T07:30:58.581192+0000","last_became_peered":"2024-03-06T07:30:58.581192+0000","last_unstale":"2024-03-06T07:31:00.957077+0000","last_undegraded":"2024-03-06T07:31:00.957077+0000","last_fullsized":"2024-03-06T07:31:00.957077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:59.802570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.960973+0000","last_change":"2024-03-06T07:30:59.613144+0000","last_active":"2024-03-06T07:31:00.960973+0000","last_peered":"2024-03-06T07:31:00.960973+0000","last_clean":"2024-03-06T07:31:00.960973+0000","last_became_active":"2024-03-06T07:30:59.612873+0000","last_became_peered":"2024-03-06T07:30:59.612873+0000","last_unstale":"2024-03-06T07:31:00.960973+0000","last_undegraded":"2024-03-06T07:31:00.960973+0000","last_fullsized":"2024-03-06T07:31:00.960973+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:58.540115+0000","last_clean_scrub_stamp":"2024-03-06T07:30:58.540115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:44.061449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.956343+0000","last_change":"2024-03-06T07:30:58.582860+0000","last_active":"2024-03-06T07:31:00.956343+0000","last_peered":"2024-03-06T07:31:00.956343+0000","last_clean":"2024-03-06T07:31:00.956343+0000","last_became_active":"2024-03-06T07:30:58.582503+0000","last_became_peered":"2024-03-06T07:30:58.582503+0000","last_unstale":"2024-03-06T07:31:00.956343+0000","last_undegraded":"2024-03-06T07:31:00.956343+0000","last_fullsized":"2024-03-06T07:31:00.956343+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:57.515899+0000","last_clean_scrub_stamp":"2024-03-06T07:30:57.515899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:58.241655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":217,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:31:00.613909+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:31:00.613909+0000","last_peered":"2024-03-06T07:31:00.613909+0000","last_clean":"2024-03-06T07:31:00.613909+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:31:00.613909+0000","last_undegraded":"2024-03-06T07:31:00.613909+0000","last_fullsized":"2024-03-06T07:31:00.613909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":810236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":548864,"data_stored":220450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":219848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":552960,"data_stored":221390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":552960,"data_stored":220416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":557056,"data_stored":221412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":219848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":810180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:04.627 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:31:04.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:31:04.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.891+0000 7fc83a000640 1 Processor -- start 2024-03-06T07:31:04.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.892+0000 7fc83a000640 1 -- start start 2024-03-06T07:31:04.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.893+0000 7fc83a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc8340d26f0 0x7fc8340d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.893+0000 7fc83a000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8340d1920 0x7fc8340b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.893+0000 7fc83a000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8340cf4a0 con 0x7fc83415e220 2024-03-06T07:31:04.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.893+0000 7fc83a000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8340b36b0 con 0x7fc8340d1920 2024-03-06T07:31:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.893+0000 7fc83a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8340cf780 con 0x7fc8340d26f0 2024-03-06T07:31:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc838c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc83415e220 0x7fc8340d1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38390/0 (socket says 172.21.15.32:38390) 2024-03-06T07:31:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc838c00640 1 -- 172.21.15.32:0/1211986112 learned_addr learned my addr 172.21.15.32:0/1211986112 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc839600640 1 --2- 172.21.15.32:0/1211986112 >> v2:172.21.15.32:3300/0 conn(0x7fc8340d1920 0x7fc8340b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc82be00640 1 --2- 172.21.15.32:0/1211986112 >> v2:172.21.15.39:3300/0 conn(0x7fc8340d26f0 0x7fc8340d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc839600640 1 -- 172.21.15.32:0/1211986112 >> v2:172.21.15.39:3300/0 conn(0x7fc8340d26f0 msgr2=0x7fc8340d4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc839600640 1 --2- 172.21.15.32:0/1211986112 >> v2:172.21.15.39:3300/0 conn(0x7fc8340d26f0 0x7fc8340d4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc839600640 1 -- 172.21.15.32:0/1211986112 >> v1:172.21.15.32:6789/0 conn(0x7fc83415e220 legacy=0x7fc8340d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.894+0000 7fc839600640 1 -- 172.21.15.32:0/1211986112 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340cfa60 con 0x7fc8340d1920 2024-03-06T07:31:04.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.896+0000 7fc839600640 1 --2- 172.21.15.32:0/1211986112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc8340b3e20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc83408d0d0 tx=0x7fc82c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6987a94d6cdb4d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.896+0000 7fc833e00640 1 -- 172.21.15.32:0/1211986112 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c039070 con 0x7fc8340d1920 2024-03-06T07:31:04.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.896+0000 7fc833e00640 1 -- 172.21.15.32:0/1211986112 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c02cbe0 con 0x7fc8340d1920 2024-03-06T07:31:04.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.896+0000 7fc833e00640 1 -- 172.21.15.32:0/1211986112 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c034440 con 0x7fc8340d1920 2024-03-06T07:31:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 -- 172.21.15.32:0/1211986112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 msgr2=0x7fc8340b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 --2- 172.21.15.32:0/1211986112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc8340b3e20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc83408d0d0 tx=0x7fc82c02be30 comp rx=0 tx=0).stop 2024-03-06T07:31:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 -- 172.21.15.32:0/1211986112 shutdown_connections 2024-03-06T07:31:04.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 --2- 172.21.15.32:0/1211986112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc8340b3e20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 --2- 172.21.15.32:0/1211986112 >> v2:172.21.15.39:3300/0 conn(0x7fc8340d26f0 0x7fc8340d4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 -- 172.21.15.32:0/1211986112 >> 172.21.15.32:0/1211986112 conn(0x7fc8340d3d50 msgr2=0x7fc8340d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 -- 172.21.15.32:0/1211986112 shutdown_connections 2024-03-06T07:31:04.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.897+0000 7fc83a000640 1 -- 172.21.15.32:0/1211986112 wait complete. 2024-03-06T07:31:04.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.900+0000 7fc83a000640 1 Processor -- start 2024-03-06T07:31:04.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.900+0000 7fc83a000640 1 -- start start 2024-03-06T07:31:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc83a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc834157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc83a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8340d26f0 0x7fc834160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc83a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc83415e220 0x7fc8340b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc83a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340b4360 con 0x7fc8340d1920 2024-03-06T07:31:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc83a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340d1d50 con 0x7fc8340d26f0 2024-03-06T07:31:04.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc83a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340b44e0 con 0x7fc83415e220 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc838c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc834157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc838c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc834157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47126/0 (socket says 172.21.15.32:47126) 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.902+0000 7fc838c00640 1 -- 172.21.15.32:0/209621057 learned_addr learned my addr 172.21.15.32:0/209621057 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc82be00640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8340d26f0 0x7fc834160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc839600640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc83415e220 0x7fc8340b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc838c00640 1 -- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc83415e220 msgr2=0x7fc8340b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc838c00640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc83415e220 0x7fc8340b7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc838c00640 1 -- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8340d26f0 msgr2=0x7fc834160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc838c00640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8340d26f0 0x7fc834160d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc838c00640 1 -- 172.21.15.32:0/209621057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc834157cb0 con 0x7fc8340d1920 2024-03-06T07:31:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc838c00640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc834157770 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc82400c3c0 tx=0x7fc82400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824017070 con 0x7fc8340d1920 2024-03-06T07:31:04.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.903+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc824005ce0 con 0x7fc8340d1920 2024-03-06T07:31:04.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.904+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8240072f0 con 0x7fc8340d1920 2024-03-06T07:31:04.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.904+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83415fc70 con 0x7fc8340d1920 2024-03-06T07:31:04.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.904+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc834160230 con 0x7fc8340d1920 2024-03-06T07:31:04.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.905+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83404f330 con 0x7fc8340d1920 2024-03-06T07:31:04.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.906+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc82401a070 con 0x7fc8340d1920 2024-03-06T07:31:04.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.907+0000 7fc831600640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc81006b4d0 0x7fc81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.907+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc824089680 con 0x7fc8340d1920 2024-03-06T07:31:04.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.907+0000 7fc82be00640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc81006b4d0 0x7fc81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.908+0000 7fc82be00640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc81006b4d0 0x7fc81006d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8340cde20 tx=0x7fc81c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:04.912+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc824058460 con 0x7fc8340d1920 2024-03-06T07:31:05.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.189+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc8340d3b30 con 0x7fc8340d1920 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.194+0000 7fc831600640 1 -- 172.21.15.32:0/209621057 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc824057d60 con 0x7fc8340d1920 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:05.197 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:31:05.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc81006b4d0 msgr2=0x7fc81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc81006b4d0 0x7fc81006d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8340cde20 tx=0x7fc81c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 msgr2=0x7fc834157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc834157770 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc82400c3c0 tx=0x7fc82400c880 comp rx=0 tx=0).stop 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 shutdown_connections 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc81006b4d0 0x7fc81006d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc83415e220 0x7fc8340b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8340d26f0 0x7fc834160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 --2- 172.21.15.32:0/209621057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8340d1920 0x7fc834157770 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 >> 172.21.15.32:0/209621057 conn(0x7fc8340d3d50 msgr2=0x7fc83406ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 shutdown_connections 2024-03-06T07:31:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.198+0000 7fc83a000640 1 -- 172.21.15.32:0/209621057 wait complete. 2024-03-06T07:31:05.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.433+0000 7fae83e00640 1 Processor -- start 2024-03-06T07:31:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.434+0000 7fae83e00640 1 -- start start 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.435+0000 7fae83e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae840d4a10 0x7fae840d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.435+0000 7fae83e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae840d0c80 0x7fae840b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.435+0000 7fae83e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae840b3710 con 0x7fae8415e0b0 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.435+0000 7fae83e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae840b09e0 con 0x7fae840d4a10 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.435+0000 7fae83e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae840cfa20 con 0x7fae840d0c80 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae7be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae840d4a10 0x7fae840d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae82a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fae8415e0b0 0x7fae840cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38404/0 (socket says 172.21.15.32:38404) 2024-03-06T07:31:05.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae82a00640 1 -- 172.21.15.32:0/2589932478 learned_addr learned my addr 172.21.15.32:0/2589932478 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae83400640 1 --2- 172.21.15.32:0/2589932478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae840d0c80 0x7fae840b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae7be00640 1 -- 172.21.15.32:0/2589932478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae840d0c80 msgr2=0x7fae840b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae7be00640 1 --2- 172.21.15.32:0/2589932478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae840d0c80 0x7fae840b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae7be00640 1 -- 172.21.15.32:0/2589932478 >> v1:172.21.15.32:6789/0 conn(0x7fae8415e0b0 legacy=0x7fae840cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae82000640 1 -- 172.21.15.32:0/2589932478 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581288917 0 0) 0x7fae840b3710 con 0x7fae8415e0b0 2024-03-06T07:31:05.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.436+0000 7fae7be00640 1 -- 172.21.15.32:0/2589932478 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae840b39f0 con 0x7fae840d4a10 2024-03-06T07:31:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.437+0000 7fae7be00640 1 --2- 172.21.15.32:0/2589932478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840d0740 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0097a0 tx=0x7fae6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d3cc7f92e22692 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae82000640 1 -- 172.21.15.32:0/2589932478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c035020 con 0x7fae840d4a10 2024-03-06T07:31:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae82000640 1 -- 172.21.15.32:0/2589932478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c031830 con 0x7fae840d4a10 2024-03-06T07:31:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae82000640 1 -- 172.21.15.32:0/2589932478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c031c00 con 0x7fae840d4a10 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 -- 172.21.15.32:0/2589932478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 msgr2=0x7fae840d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 --2- 172.21.15.32:0/2589932478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840d0740 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0097a0 tx=0x7fae6c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 -- 172.21.15.32:0/2589932478 shutdown_connections 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 --2- 172.21.15.32:0/2589932478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840d0740 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 --2- 172.21.15.32:0/2589932478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae840d0c80 0x7fae840b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 -- 172.21.15.32:0/2589932478 >> 172.21.15.32:0/2589932478 conn(0x7fae840d3d30 msgr2=0x7fae840d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 -- 172.21.15.32:0/2589932478 shutdown_connections 2024-03-06T07:31:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.438+0000 7fae83e00640 1 -- 172.21.15.32:0/2589932478 wait complete. 2024-03-06T07:31:05.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.440+0000 7fae83e00640 1 Processor -- start 2024-03-06T07:31:05.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.440+0000 7fae83e00640 1 -- start start 2024-03-06T07:31:05.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae83e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae840d0c80 0x7fae841f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840b9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae83e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae8415e0b0 0x7fae84152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae83e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8415e490 con 0x7fae840d4a10 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae83e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae840b3fb0 con 0x7fae840d0c80 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae83e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae840cfc60 con 0x7fae8415e0b0 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.441+0000 7fae7be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840b9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47154/0 (socket says 172.21.15.32:47154) 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 -- 172.21.15.32:0/2080780397 learned_addr learned my addr 172.21.15.32:0/2080780397 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae82a00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae840d0c80 0x7fae841f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 -- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae8415e0b0 msgr2=0x7fae84152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae8415e0b0 0x7fae84152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 -- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae840d0c80 msgr2=0x7fae841f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae840d0c80 0x7fae841f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 -- 172.21.15.32:0/2080780397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae841f4330 con 0x7fae840d4a10 2024-03-06T07:31:05.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.442+0000 7fae7be00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840b9160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fae6c02bee0 tx=0x7fae6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.443+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c041070 con 0x7fae840d4a10 2024-03-06T07:31:05.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.443+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c03b920 con 0x7fae840d4a10 2024-03-06T07:31:05.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.443+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c035040 con 0x7fae840d4a10 2024-03-06T07:31:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.443+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae840cbfa0 con 0x7fae840d4a10 2024-03-06T07:31:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.443+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae840cc500 con 0x7fae840d4a10 2024-03-06T07:31:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.444+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8404f330 con 0x7fae840d4a10 2024-03-06T07:31:05.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.445+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae6c04c060 con 0x7fae840d4a10 2024-03-06T07:31:05.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.446+0000 7fae7b400640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae6006b4d0 0x7fae6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.446+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae6c0abc30 con 0x7fae840d4a10 2024-03-06T07:31:05.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.446+0000 7fae82a00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae6006b4d0 0x7fae6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.447+0000 7fae82a00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae6006b4d0 0x7fae6006d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fae74007fd0 tx=0x7fae74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.450+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae840cbfa0 con 0x7fae840d4a10 2024-03-06T07:31:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.709+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae840b5920 con 0x7fae840d4a10 2024-03-06T07:31:05.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.710+0000 7fae7b400640 1 -- 172.21.15.32:0/2080780397 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4965 (secure 0 0 0) 0x7fae6c07aa20 con 0x7fae840d4a10 2024-03-06T07:31:05.713 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:05.713 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:59.036395+0000","modified":"2024-03-06T07:31:02.225064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/1942129463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1942129463},{"type":"v1","addr":"172.21.15.32:6836","nonce":1942129463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":3}]} 2024-03-06T07:31:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.713+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae6006b4d0 msgr2=0x7fae6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.713+0000 7fae83e00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae6006b4d0 0x7fae6006d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fae74007fd0 tx=0x7fae74009040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.713+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 msgr2=0x7fae840b9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.713+0000 7fae83e00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840b9160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fae6c02bee0 tx=0x7fae6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 shutdown_connections 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fae6006b4d0 0x7fae6006d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae8415e0b0 0x7fae84152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae840d4a10 0x7fae840b9160 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 --2- 172.21.15.32:0/2080780397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae840d0c80 0x7fae841f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 >> 172.21.15.32:0/2080780397 conn(0x7fae840d3d30 msgr2=0x7fae840cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 shutdown_connections 2024-03-06T07:31:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:05.714+0000 7fae83e00640 1 -- 172.21.15.32:0/2080780397 wait complete. 2024-03-06T07:31:06.137 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:59.036395+0000', 'modified': '2024-03-06T07:31:02.225064+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5517}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5517': {'gid': 5517, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/1942129463', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1942129463}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 1942129463}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5517, 'qdb_cluster': [5517]} 2024-03-06T07:31:06.138 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:06.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:31:06.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.358+0000 7fc176600640 1 Processor -- start 2024-03-06T07:31:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.359+0000 7fc176600640 1 -- start start 2024-03-06T07:31:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc176600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc17015e2e0 0x7fc1700d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc176600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1700d0d70 0x7fc1700d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc176600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1700b3940 con 0x7fc1700d1b30 2024-03-06T07:31:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc176600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700d4e70 con 0x7fc17015e2e0 2024-03-06T07:31:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc176600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b3c40 con 0x7fc1700d0d70 2024-03-06T07:31:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc16fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1700d1b30 0x7fc1700d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38418/0 (socket says 172.21.15.32:38418) 2024-03-06T07:31:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc16fe00640 1 -- 172.21.15.32:0/4138225370 learned_addr learned my addr 172.21.15.32:0/4138225370 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.362+0000 7fc16f400640 1 --2- 172.21.15.32:0/4138225370 >> v2:172.21.15.39:3300/0 conn(0x7fc17015e2e0 0x7fc1700d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc167e00640 1 --2- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1700d0d70 0x7fc1700d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16ea00640 1 -- 172.21.15.32:0/4138225370 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165677132 0 0) 0x7fc1700b3940 con 0x7fc1700d1b30 2024-03-06T07:31:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16ea00640 1 -- 172.21.15.32:0/4138225370 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc154003520 con 0x7fc1700d1b30 2024-03-06T07:31:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16f400640 1 -- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1700d0d70 msgr2=0x7fc1700d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16f400640 1 --2- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1700d0d70 0x7fc1700d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16f400640 1 -- 172.21.15.32:0/4138225370 >> v1:172.21.15.32:6789/0 conn(0x7fc1700d1b30 legacy=0x7fc1700d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16ea00640 1 -- 172.21.15.32:0/4138225370 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707906728 0 0) 0x7fc154003520 con 0x7fc1700d1b30 2024-03-06T07:31:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.363+0000 7fc16f400640 1 -- 172.21.15.32:0/4138225370 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700cf580 con 0x7fc17015e2e0 2024-03-06T07:31:06.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.365+0000 7fc16f400640 1 --2- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17015e2e0 0x7fc1700d0830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc1600097a0 tx=0x7fc16002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1db53a045346396 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc16ea00640 1 -- 172.21.15.32:0/4138225370 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160037070 con 0x7fc17015e2e0 2024-03-06T07:31:06.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc16ea00640 1 -- 172.21.15.32:0/4138225370 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160030510 con 0x7fc17015e2e0 2024-03-06T07:31:06.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc16ea00640 1 -- 172.21.15.32:0/4138225370 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1680019f0 con 0x7fc17015e2e0 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc176600640 1 -- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17015e2e0 msgr2=0x7fc1700d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc176600640 1 --2- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17015e2e0 0x7fc1700d0830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc1600097a0 tx=0x7fc16002b790 comp rx=0 tx=0).stop 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc176600640 1 -- 172.21.15.32:0/4138225370 shutdown_connections 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc176600640 1 --2- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1700d0d70 0x7fc1700d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc176600640 1 --2- 172.21.15.32:0/4138225370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17015e2e0 0x7fc1700d0830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.366+0000 7fc176600640 1 -- 172.21.15.32:0/4138225370 >> 172.21.15.32:0/4138225370 conn(0x7fc1700d3f60 msgr2=0x7fc1700d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.367+0000 7fc176600640 1 -- 172.21.15.32:0/4138225370 shutdown_connections 2024-03-06T07:31:06.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.367+0000 7fc176600640 1 -- 172.21.15.32:0/4138225370 wait complete. 2024-03-06T07:31:06.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.368+0000 7fc176600640 1 Processor -- start 2024-03-06T07:31:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.369+0000 7fc176600640 1 -- start start 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc176600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1700d0d70 0x7fc1701f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc176600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 0x7fc1700cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc176600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc17015e2e0 0x7fc17014ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc176600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700cfe20 con 0x7fc1700d1b30 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc176600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700cfb40 con 0x7fc1700d0d70 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc176600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b50e0 con 0x7fc17015e2e0 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 0x7fc1700cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 0x7fc1700cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47170/0 (socket says 172.21.15.32:47170) 2024-03-06T07:31:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 -- 172.21.15.32:0/4133531150 learned_addr learned my addr 172.21.15.32:0/4133531150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 -- 172.21.15.32:0/4133531150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc17015e2e0 msgr2=0x7fc17014ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc17015e2e0 0x7fc17014ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 -- 172.21.15.32:0/4133531150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1700d0d70 msgr2=0x7fc1701f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1700d0d70 0x7fc1701f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.370+0000 7fc167e00640 1 -- 172.21.15.32:0/4133531150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17014b1d0 con 0x7fc1700d1b30 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.371+0000 7fc167e00640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 0x7fc1700cb030 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc15800e9b0 tx=0x7fc15800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.371+0000 7fc167400640 1 -- 172.21.15.32:0/4133531150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158017070 con 0x7fc1700d1b30 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.371+0000 7fc167400640 1 -- 172.21.15.32:0/4133531150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15800c9a0 con 0x7fc1700d1b30 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.371+0000 7fc167400640 1 -- 172.21.15.32:0/4133531150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15800cd90 con 0x7fc1700d1b30 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.371+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1700ca460 con 0x7fc1700d1b30 2024-03-06T07:31:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.371+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1700ccc50 con 0x7fc1700d1b30 2024-03-06T07:31:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.375+0000 7fc167400640 1 -- 172.21.15.32:0/4133531150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc15801a070 con 0x7fc1700d1b30 2024-03-06T07:31:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.376+0000 7fc17923b740 1 Processor -- start 2024-03-06T07:31:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.376+0000 7fc17923b740 1 -- start start 2024-03-06T07:31:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.376+0000 7fc167400640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc14c06b4d0 0x7fc14c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.376+0000 7fc167400640 1 -- 172.21.15.32:0/4133531150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc15808a5f0 con 0x7fc1700d1b30 2024-03-06T07:31:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.376+0000 7fc16fe00640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc14c06b4d0 0x7fc14c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x5648735b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e2b90 0x5648735e5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc17923b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 0x5648735e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc17923b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x56487351f600 con 0x5648735b5c90 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648734c8930 con 0x5648735e2b90 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x56487351b6a0 con 0x5648735e27b0 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 0x5648735e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 0x5648735e5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47174/0 (socket says 172.21.15.32:47174) 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc16fe00640 1 -- 172.21.15.32:0/3805063338 learned_addr learned my addr 172.21.15.32:0/3805063338 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc16fe00640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc14c06b4d0 0x7fc14c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc168002180 tx=0x7fc168001420 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc16f400640 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x5648735b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.377+0000 7fc167e00640 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e2b90 0x5648735e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc16fe00640 1 -- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 msgr2=0x5648735b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc16fe00640 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x5648735b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc16fe00640 1 -- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e2b90 msgr2=0x5648735e5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc16fe00640 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e2b90 0x5648735e5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc16fe00640 1 -- 172.21.15.32:0/3805063338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648732342d0 con 0x5648735b5c90 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc167e00640 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e2b90 0x5648735e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc16fe00640 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 0x5648735e5a80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc1680124e0 tx=0x7fc16800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76876abd8efcf93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc15fe00640 1 -- 172.21.15.32:0/3805063338 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168023070 con 0x5648735b5c90 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc15fe00640 1 -- 172.21.15.32:0/3805063338 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc168016030 con 0x5648735b5c90 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.378+0000 7fc15fe00640 1 -- 172.21.15.32:0/3805063338 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168018050 con 0x5648735b5c90 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 -- 172.21.15.32:0/3805063338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 msgr2=0x5648735e5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 0x5648735e5a80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc1680124e0 tx=0x7fc16800f000 comp rx=0 tx=0).stop 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 -- 172.21.15.32:0/3805063338 shutdown_connections 2024-03-06T07:31:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735b5c90 0x5648735e5a80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e2b90 0x5648735e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 --2- 172.21.15.32:0/3805063338 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x5648735b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 -- 172.21.15.32:0/3805063338 >> 172.21.15.32:0/3805063338 conn(0x5648735b79c0 msgr2=0x564873600540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 -- 172.21.15.32:0/3805063338 shutdown_connections 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.379+0000 7fc17923b740 1 -- 172.21.15.32:0/3805063338 wait complete. 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.380+0000 7fc17923b740 1 -- start start 2024-03-06T07:31:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.380+0000 7fc17923b740 1 Processor -- start 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 0x5648735d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x5648735f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648735b9f60 con 0x5648735e27b0 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648734dbd90 con 0x5648735f82e0 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648735b9350 con 0x5648735e2b90 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc17923b740 10 client.0 _ll_register_callbacks cb 0x564873517350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.381+0000 7fc16f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 0x5648735d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 0x5648735d58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47188/0 (socket says 172.21.15.32:47188) 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 -- 172.21.15.32:0/3807657169 learned_addr learned my addr 172.21.15.32:0/3807657169 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc167e00640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 -- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 msgr2=0x5648735f7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735f7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 -- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 msgr2=0x5648735f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16fe00640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x5648735f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x5648735f97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 -- 172.21.15.32:0/3807657169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648732fe6b0 con 0x5648735e27b0 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc167e00640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc16f400640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 0x5648735d58d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc1600098c0 tx=0x7fc1600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c1086ee91d90f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.382+0000 7fc15cc00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:31:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc15cc00640 1 -- 172.21.15.32:0/3807657169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160037040 con 0x5648735e27b0 2024-03-06T07:31:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc15cc00640 1 -- 172.21.15.32:0/3807657169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc160030d90 con 0x5648735e27b0 2024-03-06T07:31:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc168016030 con 0x5648735e27b0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc15cc00640 1 -- 172.21.15.32:0/3807657169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc160006e10 con 0x5648735e27b0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc17923b740 10 client.5673 fetch_fsmap learned FSMap version 25 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5648735e6ab0 con 0x5648735e27b0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc15cc00640 1 -- 172.21.15.32:0/3807657169 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc168016030 con 0x5648735e27b0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.383+0000 7fc15cc00640 1 -- 172.21.15.32:0/3807657169 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 25) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc16003c440 con 0x5648735e27b0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc17923b740 10 client.5673 fetch_fsmap finished waiting for FSMap version 25 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc17923b740 10 client.5673 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc17923b740 20 client.5673 populate_metadata read hostname 'smithi032' 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc17923b740 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x5648736143a0 0x5648735e7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc17923b740 4 client.5673 mds_command: new command op to 5517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 --> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56487352cb40 con 0x5648736143a0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc167e00640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x5648736143a0 0x5648735e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc167e00640 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x5648736143a0 0x5648735e7630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52ba8e9b60e826e8 server_cookie=292118bdd33accc3 in_seq=0 out_seq=0 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.384+0000 7fc15cc00640 10 client.5673 ms_handle_connect on v2:172.21.15.32:6834/1942129463 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.385+0000 7fc15cc00640 1 -- 172.21.15.32:0/3807657169 <== mds.0 v2:172.21.15.32:6834/1942129463 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56487352cb40 con 0x5648736143a0 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.385+0000 7fc15cc00640 10 client.5673 handle_command_reply: tid=0 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 client.5673 shutdown 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 msgr2=0x5648735d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 0x5648735d58d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc1600098c0 tx=0x7fc1600046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 shutdown_connections 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x5648736143a0 0x5648735e7630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x5648735f97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 --2- 172.21.15.32:0/3807657169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e27b0 0x5648735d58d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc15cc00640 0 client.5673 ms_handle_reset on v2:172.21.15.32:6834/1942129463 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 >> 172.21.15.32:0/3807657169 conn(0x5648735b5c90 msgr2=0x5648735e5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 shutdown_connections 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 1 -- 172.21.15.32:0/3807657169 wait complete. 2024-03-06T07:31:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.386+0000 7fc17923b740 20 client.5673 trim_cache size 0 max 16384 2024-03-06T07:31:06.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.404+0000 7fc17923b740 1 Processor -- start 2024-03-06T07:31:06.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 -- start start 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x56487360ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 0x5648735b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x564873603c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170160550 con 0x5648735e2b90 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17015f950 con 0x5648735f82e0 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.405+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17015ff50 con 0x5648735e27b0 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 0x5648735b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 0x5648735b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47190/0 (socket says 172.21.15.32:47190) 2024-03-06T07:31:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 -- 172.21.15.32:0/932909562 learned_addr learned my addr 172.21.15.32:0/932909562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 -- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 msgr2=0x56487360ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x56487360ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc16fe00640 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x564873603c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 -- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 msgr2=0x564873603c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x564873603c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 -- 172.21.15.32:0/932909562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648735b4c80 con 0x5648735e2b90 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.406+0000 7fc167e00640 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 0x5648735b09e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc15800c520 tx=0x7fc158055020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ffc75d00f7f2dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc15f400640 1 -- 172.21.15.32:0/932909562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158087070 con 0x5648735e2b90 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc15f400640 1 -- 172.21.15.32:0/932909562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158061470 con 0x5648735e2b90 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc15f400640 1 -- 172.21.15.32:0/932909562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158021040 con 0x5648735e2b90 2024-03-06T07:31:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 -- 172.21.15.32:0/932909562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 msgr2=0x5648735b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 0x5648735b09e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc15800c520 tx=0x7fc158055020 comp rx=0 tx=0).stop 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 -- 172.21.15.32:0/932909562 shutdown_connections 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735f82e0 0x564873603c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735e2b90 0x5648735b09e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 --2- 172.21.15.32:0/932909562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e27b0 0x56487360ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 -- 172.21.15.32:0/932909562 >> 172.21.15.32:0/932909562 conn(0x5648735b5c90 msgr2=0x5648735f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 -- 172.21.15.32:0/932909562 shutdown_connections 2024-03-06T07:31:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.407+0000 7fc17923b740 1 -- 172.21.15.32:0/932909562 wait complete. 2024-03-06T07:31:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.408+0000 7fc17923b740 1 -- start start 2024-03-06T07:31:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.408+0000 7fc17923b740 1 Processor -- start 2024-03-06T07:31:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e27b0 0x564873659810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735e80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 0x5648735e8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x564873614220 con 0x5648735f82e0 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564873610870 con 0x5648735e27b0 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648736147d0 con 0x5648735e2b90 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.409+0000 7fc17923b740 10 client.0 _ll_register_callbacks cb 0x7fc160035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 0x5648735e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 0x5648735e8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47200/0 (socket says 172.21.15.32:47200) 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 -- 172.21.15.32:0/753698350 learned_addr learned my addr 172.21.15.32:0/753698350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16f400640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e27b0 0x564873659810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 -- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 msgr2=0x5648735e80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735e80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 -- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e27b0 msgr2=0x564873659810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e27b0 0x564873659810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 -- 172.21.15.32:0/753698350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648735b9f40 con 0x5648735f82e0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16f400640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e27b0 0x564873659810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc16fe00640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 0x5648735e8630 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc16800cfd0 tx=0x7fc168010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d580b2a592a1bfd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.410+0000 7fc15cc00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.411+0000 7fc15cc00640 1 -- 172.21.15.32:0/753698350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16802e060 con 0x5648735f82e0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.411+0000 7fc15cc00640 1 -- 172.21.15.32:0/753698350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc16801d810 con 0x5648735f82e0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.411+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc158061470 con 0x5648735f82e0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.411+0000 7fc15cc00640 1 -- 172.21.15.32:0/753698350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc168018020 con 0x5648735f82e0 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.411+0000 7fc17923b740 10 client.5679 fetch_fsmap learned FSMap version 25 2024-03-06T07:31:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.411+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5648735e0930 con 0x5648735f82e0 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc15cc00640 1 -- 172.21.15.32:0/753698350 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc158061470 con 0x5648735f82e0 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc15cc00640 1 -- 172.21.15.32:0/753698350 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 25) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc16801da10 con 0x5648735f82e0 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc17923b740 10 client.5679 fetch_fsmap finished waiting for FSMap version 25 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc17923b740 10 client.5679 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc17923b740 20 client.5679 populate_metadata read hostname 'smithi032' 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc17923b740 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x564873623aa0 0x564873670490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc17923b740 4 client.5679 mds_command: new command op to 5517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 --> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564873526e90 con 0x564873623aa0 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.412+0000 7fc16f400640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x564873623aa0 0x564873670490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc16f400640 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x564873623aa0 0x564873670490 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d59048cd10f95dcf server_cookie=58ef4ee22a8c16cc in_seq=0 out_seq=0 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc15cc00640 10 client.5679 ms_handle_connect on v2:172.21.15.32:6834/1942129463 2024-03-06T07:31:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc15cc00640 1 -- 172.21.15.32:0/753698350 <== mds.0 v2:172.21.15.32:6834/1942129463 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x564873526e90 con 0x564873623aa0 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc15cc00640 10 client.5679 handle_command_reply: tid=0 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 client.5679 shutdown 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 msgr2=0x5648735e8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 0x5648735e8630 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc16800cfd0 tx=0x7fc168010000 comp rx=0 tx=0).stop 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 shutdown_connections 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:6834/1942129463,v1:172.21.15.32:6836/1942129463] conn(0x564873623aa0 0x564873670490 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5648735f82e0 0x5648735e8630 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5648735e2b90 0x5648735e80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 --2- 172.21.15.32:0/753698350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648735e27b0 0x564873659810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc15cc00640 0 client.5679 ms_handle_reset on v2:172.21.15.32:6834/1942129463 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 >> 172.21.15.32:0/753698350 conn(0x5648735b5c90 msgr2=0x5648735b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.413+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 shutdown_connections 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc17923b740 1 -- 172.21.15.32:0/753698350 wait complete. 2024-03-06T07:31:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc17923b740 20 client.5679 trim_cache size 0 max 16384 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5517, 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8217951499999998, 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 25, 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 46, 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 46, 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 13.240415693999999 2024-03-06T07:31:06.419 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:31:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc14c06b4d0 msgr2=0x7fc14c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc14c06b4d0 0x7fc14c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc168002180 tx=0x7fc168001420 comp rx=0 tx=0).stop 2024-03-06T07:31:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 msgr2=0x7fc1700cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 0x7fc1700cb030 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc15800e9b0 tx=0x7fc15800ee70 comp rx=0 tx=0).stop 2024-03-06T07:31:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 shutdown_connections 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc14c06b4d0 0x7fc14c06d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc17015e2e0 0x7fc17014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1700d1b30 0x7fc1700cb030 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 --2- 172.21.15.32:0/4133531150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1700d0d70 0x7fc1701f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.414+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 >> 172.21.15.32:0/4133531150 conn(0x7fc1700d3f60 msgr2=0x7fc17006d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.415+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 shutdown_connections 2024-03-06T07:31:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.415+0000 7fc176600640 1 -- 172.21.15.32:0/4133531150 wait complete. 2024-03-06T07:31:06.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:06.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.656+0000 7f323b200640 1 Processor -- start 2024-03-06T07:31:06.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.656+0000 7f323b200640 1 -- start start 2024-03-06T07:31:06.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.659+0000 7f323b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f323c06e710 0x7f323c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.659+0000 7f323b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c0ad880 0x7f323c06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.659+0000 7f323b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c06fb90 con 0x7f323c147870 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.659+0000 7f323b200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c147c50 con 0x7f323c06e710 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.659+0000 7f323b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0b3b60 con 0x7f323c0ad880 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f323c06e710 0x7f323c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f323c06e710 0x7f323c06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47216/0 (socket says 172.21.15.32:47216) 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 -- 172.21.15.32:0/1258471770 learned_addr learned my addr 172.21.15.32:0/1258471770 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f323a800640 1 --2- 172.21.15.32:0/1258471770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c0ad880 0x7f323c06b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 -- 172.21.15.32:0/1258471770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c0ad880 msgr2=0x7f323c06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 --2- 172.21.15.32:0/1258471770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c0ad880 0x7f323c06b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 -- 172.21.15.32:0/1258471770 >> v1:172.21.15.39:6789/0 conn(0x7f323c147870 legacy=0x7f323c0b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.660+0000 7f3233e00640 1 -- 172.21.15.32:0/1258471770 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c06ebf0 con 0x7f323c06e710 2024-03-06T07:31:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.662+0000 7f3233e00640 1 --2- 172.21.15.32:0/1258471770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c06afb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3224009870 tx=0x7f322402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1de8db1ecda2128 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.662+0000 7f3239400640 1 -- 172.21.15.32:0/1258471770 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224035020 con 0x7f323c06e710 2024-03-06T07:31:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.662+0000 7f3239400640 1 -- 172.21.15.32:0/1258471770 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3224031640 con 0x7f323c06e710 2024-03-06T07:31:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.662+0000 7f3239400640 1 -- 172.21.15.32:0/1258471770 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322402d040 con 0x7f323c06e710 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 -- 172.21.15.32:0/1258471770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 msgr2=0x7f323c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 --2- 172.21.15.32:0/1258471770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c06afb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3224009870 tx=0x7f322402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 -- 172.21.15.32:0/1258471770 shutdown_connections 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 --2- 172.21.15.32:0/1258471770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c0ad880 0x7f323c06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 --2- 172.21.15.32:0/1258471770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c06afb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 -- 172.21.15.32:0/1258471770 >> 172.21.15.32:0/1258471770 conn(0x7f323c06c0c0 msgr2=0x7f323c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 -- 172.21.15.32:0/1258471770 shutdown_connections 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.663+0000 7f323b200640 1 -- 172.21.15.32:0/1258471770 wait complete. 2024-03-06T07:31:06.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.666+0000 7f323b200640 1 Processor -- start 2024-03-06T07:31:06.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.666+0000 7f323b200640 1 -- start start 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.667+0000 7f323b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c0b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.667+0000 7f323b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f323c0ad880 0x7f323c15d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.667+0000 7f323b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c147870 0x7f323c0aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f323b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0b4370 con 0x7f323c06e710 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f323b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0b4090 con 0x7f323c0ad880 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f323b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0d0c60 con 0x7f323c147870 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c0b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c0b0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47230/0 (socket says 172.21.15.32:47230) 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 -- 172.21.15.32:0/3052292737 learned_addr learned my addr 172.21.15.32:0/3052292737 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3233e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f323c0ad880 0x7f323c15d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f323a800640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c147870 0x7f323c0aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 -- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c147870 msgr2=0x7f323c0aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c147870 0x7f323c0aeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 -- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f323c0ad880 msgr2=0x7f323c15d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f323c0ad880 0x7f323c15d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3239e00640 1 -- 172.21.15.32:0/3052292737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0b2400 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.668+0000 7f3233e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f323c0ad880 0x7f323c15d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.669+0000 7f3239e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c0b0d70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f322c00c3f0 tx=0x7f322c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.669+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c011790 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.669+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c017070 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.669+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c011b90 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.669+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c12df40 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.669+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c12e500 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.670+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c12dd60 con 0x7f323c06e710 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.671+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f322c01c460 con 0x7f323c06e710 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.672+0000 7f3232000640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f321806b4d0 0x7f321806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.672+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f322c08a0c0 con 0x7f323c06e710 2024-03-06T07:31:06.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.673+0000 7f3233e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f321806b4d0 0x7f321806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.673+0000 7f3233e00640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f321806b4d0 0x7f321806d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3224036000 tx=0x7f3224008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.677+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f323c12dd60 con 0x7f323c06e710 2024-03-06T07:31:06.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.915+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f323c04f330 con 0x7f323c06e710 2024-03-06T07:31:06.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.915+0000 7f3232000640 1 -- 172.21.15.32:0/3052292737 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f322c058ea0 con 0x7f323c06e710 2024-03-06T07:31:06.918 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:06.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f321806b4d0 msgr2=0x7f321806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f321806b4d0 0x7f321806d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3224036000 tx=0x7f3224008030 comp rx=0 tx=0).stop 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 msgr2=0x7f323c0b0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c0b0d70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f322c00c3f0 tx=0x7f322c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 shutdown_connections 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f321806b4d0 0x7f321806d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f323c147870 0x7f323c0aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f323c0ad880 0x7f323c15d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 --2- 172.21.15.32:0/3052292737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f323c06e710 0x7f323c0b0d70 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 >> 172.21.15.32:0/3052292737 conn(0x7f323c06c0c0 msgr2=0x7f323c144e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 shutdown_connections 2024-03-06T07:31:06.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:06.918+0000 7f323b200640 1 -- 172.21.15.32:0/3052292737 wait complete. 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:31:06.939 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:31:06.940 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:06.940 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:06.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:06.940 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42270sec preferred_lft 42270sec 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.992 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:06.995 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:07.093 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:31:07.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.098 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:31:07.152 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:31:07.152 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:07.152 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:31:07.152 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:31:07.152 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:07.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.154 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:31:07.154 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:31:07.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:31:07.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:31:07.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:31:07.154 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:31:07.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.347 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:31:07.401 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:07.402 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:07.402 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:07.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:31:07.498 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:07.498 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:07.551 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:07.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:07.552 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:07.603 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:31:07.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:07.745 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:07.745 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:07.778 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:07.778 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:07.778 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:31:07.778 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:31:07.778 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:31:07.781 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:31:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:31:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:31:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:31:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:31:07.785 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:31:07.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:07.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:07.863 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:31:07.926 INFO:teuthology.orchestra.run.smithi032.stdout:client5146 2024-03-06T07:31:07.926 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:07.926 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:07.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:31:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.237+0000 7f6eeee00640 1 Processor -- start 2024-03-06T07:31:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.237+0000 7f6eeee00640 1 -- start start 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.240+0000 7f6eeee00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6ee815c3f0 0x7f6ee815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.240+0000 7f6eeee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 0x7f6ee80b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6eeee00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee80af250 con 0x7f6ee815c010 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6eeee00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee815cf50 con 0x7f6ee815c3f0 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6eeee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee815d230 con 0x7f6ee80afcf0 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6ee7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6ee815c010 0x7f6ee806ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35010/0 (socket says 172.21.15.32:35010) 2024-03-06T07:31:08.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6ee7e00640 1 -- 172.21.15.32:0/863648407 learned_addr learned my addr 172.21.15.32:0/863648407 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6eece00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 0x7f6ee80b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6ee7400640 1 --2- 172.21.15.32:0/863648407 >> v2:172.21.15.39:3300/0 conn(0x7f6ee815c3f0 0x7f6ee815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.241+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912170942 0 0) 0x7f6ee80af250 con 0x7f6ee815c010 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc003520 con 0x7f6ee815c010 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2508502393 0 0) 0x7f6ecc003520 con 0x7f6ee815c010 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 msgr2=0x7f6ee80b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 0x7f6ee80b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 >> v2:172.21.15.39:3300/0 conn(0x7f6ee815c3f0 msgr2=0x7f6ee815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 --2- 172.21.15.32:0/863648407 >> v2:172.21.15.39:3300/0 conn(0x7f6ee815c3f0 0x7f6ee815c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee80af4f0 con 0x7f6ee815c010 2024-03-06T07:31:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6ed8001670 con 0x7f6ee815c010 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 >> v1:172.21.15.32:6789/0 conn(0x7f6ee815c010 legacy=0x7f6ee806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0039b0 0x7f6ecc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 0x7f6ecc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ecc004d40 0x7f6ecc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee815cf50 con 0x7f6ecc004d40 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee815d230 con 0x7f6ecc0039b0 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc00b9f0 con 0x7f6ee80afcf0 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.242+0000 7f6eece00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ecc004d40 0x7f6ecc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.243+0000 7f6ee7e00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0039b0 0x7f6ecc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.243+0000 7f6eece00640 1 -- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 msgr2=0x7f6ecc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.243+0000 7f6eece00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80afcf0 0x7f6ecc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.243+0000 7f6eece00640 1 -- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0039b0 msgr2=0x7f6ecc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.243+0000 7f6eece00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0039b0 0x7f6ecc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.243+0000 7f6eece00640 1 -- 172.21.15.32:0/863648407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc003520 con 0x7f6ecc004d40 2024-03-06T07:31:08.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.244+0000 7f6eece00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ecc004d40 0x7f6ecc0051c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009fc0 tx=0x7f6ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef115f7aff05e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.244+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0039070 con 0x7f6ecc004d40 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.244+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee002cc60 con 0x7f6ecc004d40 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.244+0000 7f6ee6a00640 1 -- 172.21.15.32:0/863648407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0009c10 con 0x7f6ecc004d40 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eeee00640 1 -- 172.21.15.32:0/863648407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ecc004d40 msgr2=0x7f6ecc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eeee00640 1 --2- 172.21.15.32:0/863648407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ecc004d40 0x7f6ecc0051c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009fc0 tx=0x7f6ecc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eece00640 1 -- 172.21.15.32:0/863648407 reap_dead start 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eeee00640 1 -- 172.21.15.32:0/863648407 shutdown_connections 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eeee00640 1 -- 172.21.15.32:0/863648407 >> 172.21.15.32:0/863648407 conn(0x7f6ee80732d0 msgr2=0x7f6ee809aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eeee00640 1 -- 172.21.15.32:0/863648407 shutdown_connections 2024-03-06T07:31:08.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.245+0000 7f6eeee00640 1 -- 172.21.15.32:0/863648407 wait complete. 2024-03-06T07:31:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.247+0000 7f6eeee00640 1 Processor -- start 2024-03-06T07:31:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.247+0000 7f6eeee00640 1 -- start start 2024-03-06T07:31:08.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6eeee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 0x7f6ee80b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6eeee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ee80b2ef0 0x7f6ee80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6eeee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80cf5c0 0x7f6ee80d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6eeee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80d1c10 con 0x7f6ee8150880 2024-03-06T07:31:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6eeee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80d1a90 con 0x7f6ee80b2ef0 2024-03-06T07:31:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6eeee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80b0180 con 0x7f6ee80cf5c0 2024-03-06T07:31:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6ee7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 0x7f6ee80b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6ee7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 0x7f6ee80b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40858/0 (socket says 172.21.15.32:40858) 2024-03-06T07:31:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.249+0000 7f6ee7e00640 1 -- 172.21.15.32:0/2268678145 learned_addr learned my addr 172.21.15.32:0/2268678145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6eece00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80cf5c0 0x7f6ee80d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7400640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ee80b2ef0 0x7f6ee80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7e00640 1 -- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80cf5c0 msgr2=0x7f6ee80d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7e00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80cf5c0 0x7f6ee80d04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7e00640 1 -- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ee80b2ef0 msgr2=0x7f6ee80cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7e00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ee80b2ef0 0x7f6ee80cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7e00640 1 -- 172.21.15.32:0/2268678145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee80b1910 con 0x7f6ee8150880 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6eece00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80cf5c0 0x7f6ee80d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6ee7e00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 0x7f6ee80b29b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800eb30 tx=0x7f6ed8005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.250+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800c930 con 0x7f6ee8150880 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.251+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed8018070 con 0x7f6ee8150880 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.251+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed800cd10 con 0x7f6ee8150880 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.251+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee80b1bf0 con 0x7f6ee8150880 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.251+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee80b21b0 con 0x7f6ee8150880 2024-03-06T07:31:08.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.252+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee804f330 con 0x7f6ee8150880 2024-03-06T07:31:08.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.253+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ed801d460 con 0x7f6ee8150880 2024-03-06T07:31:08.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.254+0000 7f6edfe00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ec806b4d0 0x7f6ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.254+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ed808a210 con 0x7f6ee8150880 2024-03-06T07:31:08.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.254+0000 7f6ee7400640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ec806b4d0 0x7f6ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.255+0000 7f6ee7400640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ec806b4d0 0x7f6ec806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40047b0 tx=0x7f6ed4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.263+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ed8059080 con 0x7f6ee8150880 2024-03-06T07:31:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.466+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f6ee80b6600 con 0x7f6ee8150880 2024-03-06T07:31:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.473+0000 7f6edfe00640 1 -- 172.21.15.32:0/2268678145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6ed8058980 con 0x7f6ee8150880 2024-03-06T07:31:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ec806b4d0 msgr2=0x7f6ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ec806b4d0 0x7f6ec806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40047b0 tx=0x7f6ed4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 msgr2=0x7f6ee80b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 0x7f6ee80b29b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800eb30 tx=0x7f6ed8005e10 comp rx=0 tx=0).stop 2024-03-06T07:31:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 shutdown_connections 2024-03-06T07:31:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ec806b4d0 0x7f6ec806d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ee80cf5c0 0x7f6ee80d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ee80b2ef0 0x7f6ee80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 --2- 172.21.15.32:0/2268678145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ee8150880 0x7f6ee80b29b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.476+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 >> 172.21.15.32:0/2268678145 conn(0x7f6ee80732d0 msgr2=0x7f6ee806f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.477+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 shutdown_connections 2024-03-06T07:31:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:08.477+0000 7f6eeee00640 1 -- 172.21.15.32:0/2268678145 wait complete. 2024-03-06T07:31:18.523 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:18.523 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:18.581 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:31:18.581 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> import uuid 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> n = 51 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = False 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> if 0 > 0: 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = True 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> dpath = os.path.dirname(path) 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:31:18.582 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(n): 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> f.write(f"{i}") 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> f.flush() 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(f.fileno()) 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> os.unlink(fpath) 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> if create_hard_links: 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> for j in range(0): 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:31:18.583 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:31:18.584 DEBUG:teuthology.orchestra.run.smithi032:> finally: 2024-03-06T07:31:18.584 DEBUG:teuthology.orchestra.run.smithi032:> os.close(dirfd) 2024-03-06T07:31:18.584 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:31:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last): 2024-03-06T07:31:18.811 INFO:teuthology.orchestra.run.smithi032.stderr: File "", line 33, in 2024-03-06T07:31:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:OSError: [Errno 28] No space left on device 2024-03-06T07:31:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:18 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:31:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:18.832 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:18.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:18.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:18.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:18.991 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:19.097 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:19.097 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:19.098 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:19.150 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:19.150 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:19.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.433+0000 7ff951400640 1 Processor -- start 2024-03-06T07:31:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.433+0000 7ff951400640 1 -- start start 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff951400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff94c15e030 0x7ff94c161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff951400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c0cfd50 0x7ff94c162260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff951400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff94c15e410 0x7ff94c0d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff951400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c0cea50 con 0x7ff94c15e410 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff951400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c0cecf0 con 0x7ff94c15e030 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff951400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0cef90 con 0x7ff94c0cfd50 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff94b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff94c15e410 0x7ff94c0d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff94b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff94c15e410 0x7ff94c0d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52188/0 (socket says 172.21.15.32:52188) 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff94b200640 1 -- 172.21.15.32:0/4218134394 learned_addr learned my addr 172.21.15.32:0/4218134394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff941e00640 1 --2- 172.21.15.32:0/4218134394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c0cfd50 0x7ff94c162260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.435+0000 7ff94a800640 1 --2- 172.21.15.32:0/4218134394 >> v2:172.21.15.39:3300/0 conn(0x7ff94c15e030 0x7ff94c161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.436+0000 7ff94b200640 1 -- 172.21.15.32:0/4218134394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c0cfd50 msgr2=0x7ff94c162260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.436+0000 7ff94b200640 1 --2- 172.21.15.32:0/4218134394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c0cfd50 0x7ff94c162260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.436+0000 7ff94b200640 1 -- 172.21.15.32:0/4218134394 >> v2:172.21.15.39:3300/0 conn(0x7ff94c15e030 msgr2=0x7ff94c161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.436+0000 7ff94b200640 1 --2- 172.21.15.32:0/4218134394 >> v2:172.21.15.39:3300/0 conn(0x7ff94c15e030 0x7ff94c161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.436+0000 7ff94b200640 1 -- 172.21.15.32:0/4218134394 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0cf270 con 0x7ff94c15e410 2024-03-06T07:31:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.437+0000 7ff94b200640 1 --2- 172.21.15.32:0/4218134394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c15e410 0x7ff94c0d2490 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff94c08cde0 tx=0x7ff94402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc1c3767709897f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.437+0000 7ff949e00640 1 -- 172.21.15.32:0/4218134394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944034070 con 0x7ff94c15e410 2024-03-06T07:31:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.437+0000 7ff949e00640 1 -- 172.21.15.32:0/4218134394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff944030850 con 0x7ff94c15e410 2024-03-06T07:31:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.437+0000 7ff949e00640 1 -- 172.21.15.32:0/4218134394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944030c20 con 0x7ff94c15e410 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 -- 172.21.15.32:0/4218134394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c15e410 msgr2=0x7ff94c0d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 --2- 172.21.15.32:0/4218134394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c15e410 0x7ff94c0d2490 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff94c08cde0 tx=0x7ff94402b990 comp rx=0 tx=0).stop 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 -- 172.21.15.32:0/4218134394 shutdown_connections 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 --2- 172.21.15.32:0/4218134394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c15e410 0x7ff94c0d2490 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 --2- 172.21.15.32:0/4218134394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c0cfd50 0x7ff94c162260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 --2- 172.21.15.32:0/4218134394 >> v2:172.21.15.39:3300/0 conn(0x7ff94c15e030 0x7ff94c161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 -- 172.21.15.32:0/4218134394 >> 172.21.15.32:0/4218134394 conn(0x7ff94c0d3eb0 msgr2=0x7ff94c0d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 -- 172.21.15.32:0/4218134394 shutdown_connections 2024-03-06T07:31:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.438+0000 7ff951400640 1 -- 172.21.15.32:0/4218134394 wait complete. 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.440+0000 7ff951400640 1 Processor -- start 2024-03-06T07:31:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.440+0000 7ff951400640 1 -- start start 2024-03-06T07:31:19.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff951400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff94c15e030 0x7ff94c0cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff951400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c15e410 0x7ff94c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff951400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 0x7ff94c0b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff951400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0d29d0 con 0x7ff94c0b3c30 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff951400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0cf830 con 0x7ff94c15e030 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff951400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0d2b50 con 0x7ff94c15e410 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff94b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 0x7ff94c0b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff94b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 0x7ff94c0b7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52200/0 (socket says 172.21.15.32:52200) 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.441+0000 7ff94b200640 1 -- 172.21.15.32:0/1756426758 learned_addr learned my addr 172.21.15.32:0/1756426758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94a800640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff94c15e030 0x7ff94c0cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff941e00640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c15e410 0x7ff94c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94b200640 1 -- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c15e410 msgr2=0x7ff94c0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94b200640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c15e410 0x7ff94c0b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94b200640 1 -- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff94c15e030 msgr2=0x7ff94c0cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94b200640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff94c15e030 0x7ff94c0cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94b200640 1 -- 172.21.15.32:0/1756426758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0b6be0 con 0x7ff94c0b3c30 2024-03-06T07:31:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff941e00640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c15e410 0x7ff94c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.442+0000 7ff94b200640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 0x7ff94c0b7c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff944009960 tx=0x7ff9440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.443+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944034040 con 0x7ff94c0b3c30 2024-03-06T07:31:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.443+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff944039d90 con 0x7ff94c0b3c30 2024-03-06T07:31:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.443+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c0b6ec0 con 0x7ff94c0b3c30 2024-03-06T07:31:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.443+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944043610 con 0x7ff94c0b3c30 2024-03-06T07:31:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.443+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c0b7480 con 0x7ff94c0b3c30 2024-03-06T07:31:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.444+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c04f3a0 con 0x7ff94c0b3c30 2024-03-06T07:31:19.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.446+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff944037070 con 0x7ff94c0b3c30 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.446+0000 7ff943400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.447+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9440ac260 con 0x7ff94c0b3c30 2024-03-06T07:31:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.447+0000 7ff94a800640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.448+0000 7ff94a800640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff92806b4d0 0x7ff92806d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0ca0a0 tx=0x7ff93c0039b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.452+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff94403e070 con 0x7ff94c0b3c30 2024-03-06T07:31:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.659+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff94c08e030 con 0x7ff94c0b3c30 2024-03-06T07:31:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.661+0000 7ff943400640 1 -- 172.21.15.32:0/1756426758 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16505 (secure 0 0 0) 0x7ff94407b170 con 0x7ff94c0b3c30 2024-03-06T07:31:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:19.662 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:00.589676+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.663+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff92806b4d0 msgr2=0x7ff92806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.663+0000 7ff951400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff92806b4d0 0x7ff92806d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0ca0a0 tx=0x7ff93c0039b0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.663+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 msgr2=0x7ff94c0b7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.663+0000 7ff951400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 0x7ff94c0b7c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff944009960 tx=0x7ff9440046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 shutdown_connections 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94c0b3c30 0x7ff94c0b7c80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff94c15e410 0x7ff94c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 --2- 172.21.15.32:0/1756426758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff94c15e030 0x7ff94c0cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 >> 172.21.15.32:0/1756426758 conn(0x7ff94c0d3eb0 msgr2=0x7ff94c0d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 shutdown_connections 2024-03-06T07:31:19.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.664+0000 7ff951400640 1 -- 172.21.15.32:0/1756426758 wait complete. 2024-03-06T07:31:19.691 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:56.874383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:58.058939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:19.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:19.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.905+0000 7fa527e00640 1 Processor -- start 2024-03-06T07:31:19.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.906+0000 7fa527e00640 1 -- start start 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa52815dee0 0x7fa5280d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa5280d4860 0x7fa5280d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5280ceca0 con 0x7fa5280cfa60 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5280cf5d0 con 0x7fa5280d4860 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280cef80 con 0x7fa52815dee0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa5280cfa60 0x7fa5280b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44782/0 (socket says 172.21.15.32:44782) 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa527400640 1 -- 172.21.15.32:0/2874766436 learned_addr learned my addr 172.21.15.32:0/2874766436 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa526000640 1 --2- 172.21.15.32:0/2874766436 >> v2:172.21.15.32:3300/0 conn(0x7fa5280d4860 0x7fa5280d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.907+0000 7fa526a00640 1 --2- 172.21.15.32:0/2874766436 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa52815dee0 0x7fa5280d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.908+0000 7fa525600640 1 -- 172.21.15.32:0/2874766436 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684430825 0 0) 0x7fa5280ceca0 con 0x7fa5280cfa60 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.908+0000 7fa525600640 1 -- 172.21.15.32:0/2874766436 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa50c003520 con 0x7fa5280cfa60 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.908+0000 7fa526000640 1 -- 172.21.15.32:0/2874766436 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa52815dee0 msgr2=0x7fa5280d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.908+0000 7fa526000640 1 --2- 172.21.15.32:0/2874766436 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa52815dee0 0x7fa5280d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.908+0000 7fa526000640 1 -- 172.21.15.32:0/2874766436 >> v1:172.21.15.32:6789/0 conn(0x7fa5280cfa60 legacy=0x7fa5280b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.908+0000 7fa526000640 1 -- 172.21.15.32:0/2874766436 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5280cf260 con 0x7fa5280d4860 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.909+0000 7fa526000640 1 --2- 172.21.15.32:0/2874766436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5280d4860 0x7fa5280d0630 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa5140097a0 tx=0x7fa51402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df6c6f8312876c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.909+0000 7fa525600640 1 -- 172.21.15.32:0/2874766436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514035020 con 0x7fa5280d4860 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.909+0000 7fa525600640 1 -- 172.21.15.32:0/2874766436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514031830 con 0x7fa5280d4860 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.909+0000 7fa525600640 1 -- 172.21.15.32:0/2874766436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514031be0 con 0x7fa5280d4860 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 -- 172.21.15.32:0/2874766436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5280d4860 msgr2=0x7fa5280d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 --2- 172.21.15.32:0/2874766436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5280d4860 0x7fa5280d0630 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa5140097a0 tx=0x7fa51402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 -- 172.21.15.32:0/2874766436 shutdown_connections 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 --2- 172.21.15.32:0/2874766436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5280d4860 0x7fa5280d0630 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 --2- 172.21.15.32:0/2874766436 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa52815dee0 0x7fa5280d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 -- 172.21.15.32:0/2874766436 >> 172.21.15.32:0/2874766436 conn(0x7fa5280d3cd0 msgr2=0x7fa5280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 -- 172.21.15.32:0/2874766436 shutdown_connections 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.910+0000 7fa527e00640 1 -- 172.21.15.32:0/2874766436 wait complete. 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.912+0000 7fa527e00640 1 Processor -- start 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.912+0000 7fa527e00640 1 -- start start 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.913+0000 7fa527e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa5280cfa60 0x7fa5280cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.913+0000 7fa527e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5280d4860 0x7fa5280a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.913+0000 7fa527e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 0x7fa528153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.913+0000 7fa527e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280d1de0 con 0x7fa52815dee0 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.913+0000 7fa527e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52815e2c0 con 0x7fa5280d4860 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.913+0000 7fa527e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280d1f60 con 0x7fa5280cfa60 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 0x7fa528153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa526a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa5280cfa60 0x7fa5280cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 0x7fa528153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52230/0 (socket says 172.21.15.32:52230) 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 -- 172.21.15.32:0/90158884 learned_addr learned my addr 172.21.15.32:0/90158884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa526000640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5280d4860 0x7fa5280a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 -- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa5280cfa60 msgr2=0x7fa5280cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa5280cfa60 0x7fa5280cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 -- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5280d4860 msgr2=0x7fa5280a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5280d4860 0x7fa5280a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 -- 172.21.15.32:0/90158884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5280a1170 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa526a00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa5280cfa60 0x7fa5280cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.914+0000 7fa527400640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 0x7fa528153030 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa52000c460 tx=0x7fa52000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.915+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520017070 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.915+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520005ce0 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.915+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5280cc3d0 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.915+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5200072b0 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.915+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5280cc990 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.916+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52804f330 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.918+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa52001a070 con 0x7fa52815dee0 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.919+0000 7fa51f400640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa50806b4d0 0x7fa50806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.919+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5200895a0 con 0x7fa52815dee0 2024-03-06T07:31:19.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.919+0000 7fa526a00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa50806b4d0 0x7fa50806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.920+0000 7fa526a00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa50806b4d0 0x7fa50806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa518009fd0 tx=0x7fa51800b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:19.924+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa520058410 con 0x7fa52815dee0 2024-03-06T07:31:20.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.125+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5280d2a20 con 0x7fa52815dee0 2024-03-06T07:31:20.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.126+0000 7fa51f400640 1 -- 172.21.15.32:0/90158884 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa520057d10 con 0x7fa52815dee0 2024-03-06T07:31:20.127 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.128+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa50806b4d0 msgr2=0x7fa50806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.128+0000 7fa527e00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa50806b4d0 0x7fa50806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa518009fd0 tx=0x7fa51800b040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.128+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 msgr2=0x7fa528153030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.128+0000 7fa527e00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 0x7fa528153030 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa52000c460 tx=0x7fa52000c920 comp rx=0 tx=0).stop 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 shutdown_connections 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa50806b4d0 0x7fa50806d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52815dee0 0x7fa528153030 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5280d4860 0x7fa5280a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 --2- 172.21.15.32:0/90158884 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa5280cfa60 0x7fa5280cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 >> 172.21.15.32:0/90158884 conn(0x7fa5280d3cd0 msgr2=0x7fa52815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 shutdown_connections 2024-03-06T07:31:20.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.129+0000 7fa527e00640 1 -- 172.21.15.32:0/90158884 wait complete. 2024-03-06T07:31:20.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:20.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.372+0000 7fbc07200640 1 Processor -- start 2024-03-06T07:31:20.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.373+0000 7fbc07200640 1 -- start start 2024-03-06T07:31:20.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.374+0000 7fbc07200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc080d4860 0x7fbc080d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.374+0000 7fbc07200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.374+0000 7fbc07200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc080ceca0 con 0x7fbc0815dee0 2024-03-06T07:31:20.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.374+0000 7fbc07200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc080cf5d0 con 0x7fbc080d4860 2024-03-06T07:31:20.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.374+0000 7fbc07200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080cef80 con 0x7fbc080cfa60 2024-03-06T07:31:20.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc080d4860 0x7fbc080d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc080d4860 0x7fbc080d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52244/0 (socket says 172.21.15.32:52244) 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 -- 172.21.15.32:0/400950358 learned_addr learned my addr 172.21.15.32:0/400950358 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbc06800640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 -- 172.21.15.32:0/400950358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 msgr2=0x7fbc080d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 -- 172.21.15.32:0/400950358 >> v1:172.21.15.39:6789/0 conn(0x7fbc0815dee0 legacy=0x7fbc080d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbbffe00640 1 -- 172.21.15.32:0/400950358 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc080cf260 con 0x7fbc080d4860 2024-03-06T07:31:20.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.375+0000 7fbc06800640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:20.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.378+0000 7fbbffe00640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc080d0630 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009870 tx=0x7fbbf002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0203e61e954b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.378+0000 7fbc05400640 1 -- 172.21.15.32:0/400950358 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0034070 con 0x7fbc080d4860 2024-03-06T07:31:20.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.378+0000 7fbc05400640 1 -- 172.21.15.32:0/400950358 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf0030840 con 0x7fbc080d4860 2024-03-06T07:31:20.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.378+0000 7fbc05400640 1 -- 172.21.15.32:0/400950358 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0030c10 con 0x7fbc080d4860 2024-03-06T07:31:20.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.379+0000 7fbc07200640 1 -- 172.21.15.32:0/400950358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 msgr2=0x7fbc080d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.379+0000 7fbc07200640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc080d0630 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009870 tx=0x7fbbf002b980 comp rx=0 tx=0).stop 2024-03-06T07:31:20.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.379+0000 7fbc07200640 1 -- 172.21.15.32:0/400950358 shutdown_connections 2024-03-06T07:31:20.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.379+0000 7fbc07200640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.379+0000 7fbc07200640 1 --2- 172.21.15.32:0/400950358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc080d0630 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.379+0000 7fbc07200640 1 -- 172.21.15.32:0/400950358 >> 172.21.15.32:0/400950358 conn(0x7fbc080d3cd0 msgr2=0x7fbc080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.380+0000 7fbc07200640 1 -- 172.21.15.32:0/400950358 shutdown_connections 2024-03-06T07:31:20.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.380+0000 7fbc07200640 1 -- 172.21.15.32:0/400950358 wait complete. 2024-03-06T07:31:20.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.382+0000 7fbc07200640 1 Processor -- start 2024-03-06T07:31:20.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.382+0000 7fbc07200640 1 -- start start 2024-03-06T07:31:20.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.383+0000 7fbc07200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.383+0000 7fbc07200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc0815f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.383+0000 7fbc07200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0815dee0 0x7fbc080c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.383+0000 7fbc07200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080b3510 con 0x7fbc080d4860 2024-03-06T07:31:20.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.383+0000 7fbc07200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0815e2c0 con 0x7fbc0815dee0 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.383+0000 7fbc07200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080b3690 con 0x7fbc080cfa60 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc0815f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc0815f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52248/0 (socket says 172.21.15.32:52248) 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 -- 172.21.15.32:0/4047471311 learned_addr learned my addr 172.21.15.32:0/4047471311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbc05e00640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbc06800640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0815dee0 0x7fbc080c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 -- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 msgr2=0x7fbc080b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 -- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0815dee0 msgr2=0x7fbc080c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0815dee0 0x7fbc080c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 -- 172.21.15.32:0/4047471311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08160e10 con 0x7fbc080d4860 2024-03-06T07:31:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.384+0000 7fbbffe00640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc0815f490 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbbf002b9c0 tx=0x7fbbf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.385+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0048070 con 0x7fbc080d4860 2024-03-06T07:31:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.385+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf0030640 con 0x7fbc080d4860 2024-03-06T07:31:20.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.385+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0034040 con 0x7fbc080d4860 2024-03-06T07:31:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.385+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0815fd70 con 0x7fbc080d4860 2024-03-06T07:31:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.385+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc08160330 con 0x7fbc080d4860 2024-03-06T07:31:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.386+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0804f330 con 0x7fbc080d4860 2024-03-06T07:31:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.387+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbf004b070 con 0x7fbc080d4860 2024-03-06T07:31:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.388+0000 7fbbfe000640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbbe406b4d0 0x7fbbe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.388+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbf00ac2f0 con 0x7fbc080d4860 2024-03-06T07:31:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.388+0000 7fbc05e00640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbbe406b4d0 0x7fbbe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.389+0000 7fbc05e00640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbbe406b4d0 0x7fbbe406d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8007fd0 tx=0x7fbbf8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.393+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbf007b1f0 con 0x7fbc080d4860 2024-03-06T07:31:20.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.593+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc0808d7d0 con 0x7fbc080d4860 2024-03-06T07:31:20.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.594+0000 7fbbfe000640 1 -- 172.21.15.32:0/4047471311 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbbf007aaf0 con 0x7fbc080d4860 2024-03-06T07:31:20.595 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.596+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbbe406b4d0 msgr2=0x7fbbe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.596+0000 7fbc07200640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbbe406b4d0 0x7fbbe406d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8007fd0 tx=0x7fbbf8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.596+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 msgr2=0x7fbc0815f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.596+0000 7fbc07200640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc0815f490 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbbf002b9c0 tx=0x7fbbf0004730 comp rx=0 tx=0).stop 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.596+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 shutdown_connections 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbbe406b4d0 0x7fbbe406d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0815dee0 0x7fbc080c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc080d4860 0x7fbc0815f490 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 --2- 172.21.15.32:0/4047471311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbc080cfa60 0x7fbc080b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 >> 172.21.15.32:0/4047471311 conn(0x7fbc080d3cd0 msgr2=0x7fbc080b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 shutdown_connections 2024-03-06T07:31:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.597+0000 7fbc07200640 1 -- 172.21.15.32:0/4047471311 wait complete. 2024-03-06T07:31:20.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:20.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.845+0000 7f1e5d000640 1 Processor -- start 2024-03-06T07:31:20.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.845+0000 7f1e5d000640 1 -- start start 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e5d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1e58145430 0x7f1e5815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e5d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e5815cc90 0x7f1e580b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e5d000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e580b2080 0x7f1e5806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e5d000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5815fe40 con 0x7f1e5815cc90 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e5d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e58160120 con 0x7f1e58145430 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e5d000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58160400 con 0x7f1e580b2080 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e56a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e5815cc90 0x7f1e580b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e56a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e5815cc90 0x7f1e580b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52250/0 (socket says 172.21.15.32:52250) 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e56a00640 1 -- 172.21.15.32:0/3450728031 learned_addr learned my addr 172.21.15.32:0/3450728031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:20.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e57400640 1 --2- 172.21.15.32:0/3450728031 >> v2:172.21.15.39:3300/0 conn(0x7f1e58145430 0x7f1e5815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.847+0000 7f1e57e00640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e580b2080 0x7f1e5806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.848+0000 7f1e56a00640 1 -- 172.21.15.32:0/3450728031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e580b2080 msgr2=0x7f1e5806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.848+0000 7f1e56a00640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e580b2080 0x7f1e5806c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.848+0000 7f1e56a00640 1 -- 172.21.15.32:0/3450728031 >> v2:172.21.15.39:3300/0 conn(0x7f1e58145430 msgr2=0x7f1e5815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.848+0000 7f1e56a00640 1 --2- 172.21.15.32:0/3450728031 >> v2:172.21.15.39:3300/0 conn(0x7f1e58145430 0x7f1e5815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.848+0000 7f1e56a00640 1 -- 172.21.15.32:0/3450728031 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e581606e0 con 0x7f1e5815cc90 2024-03-06T07:31:20.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.848+0000 7f1e57e00640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e580b2080 0x7f1e5806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:20.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.849+0000 7f1e56a00640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5815cc90 0x7f1e580b1b40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1e580b2670 tx=0x7f1e4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb256b277ed2675 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.849+0000 7f1e56000640 1 -- 172.21.15.32:0/3450728031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e44035020 con 0x7f1e5815cc90 2024-03-06T07:31:20.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.849+0000 7f1e56000640 1 -- 172.21.15.32:0/3450728031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e44031870 con 0x7f1e5815cc90 2024-03-06T07:31:20.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.849+0000 7f1e56000640 1 -- 172.21.15.32:0/3450728031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e44031c40 con 0x7f1e5815cc90 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 -- 172.21.15.32:0/3450728031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5815cc90 msgr2=0x7f1e580b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5815cc90 0x7f1e580b1b40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1e580b2670 tx=0x7f1e4402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 -- 172.21.15.32:0/3450728031 shutdown_connections 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e580b2080 0x7f1e5806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3450728031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5815cc90 0x7f1e580b1b40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3450728031 >> v2:172.21.15.39:3300/0 conn(0x7f1e58145430 0x7f1e5815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 -- 172.21.15.32:0/3450728031 >> 172.21.15.32:0/3450728031 conn(0x7f1e5806b3a0 msgr2=0x7f1e5806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 -- 172.21.15.32:0/3450728031 shutdown_connections 2024-03-06T07:31:20.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.850+0000 7f1e5d000640 1 -- 172.21.15.32:0/3450728031 wait complete. 2024-03-06T07:31:20.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.852+0000 7f1e5d000640 1 Processor -- start 2024-03-06T07:31:20.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.852+0000 7f1e5d000640 1 -- start start 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e5d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e580b2080 0x7f1e581f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e5d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 0x7f1e581f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e5d000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e5815cc90 0x7f1e580ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e5d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58145990 con 0x7f1e58145430 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e5d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58145810 con 0x7f1e580b2080 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e5d000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e580cde00 con 0x7f1e5815cc90 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e56a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 0x7f1e581f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e56a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 0x7f1e581f2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52260/0 (socket says 172.21.15.32:52260) 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.854+0000 7f1e56a00640 1 -- 172.21.15.32:0/3540698711 learned_addr learned my addr 172.21.15.32:0/3540698711 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e57400640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e580b2080 0x7f1e581f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e57e00640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e5815cc90 0x7f1e580ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e56a00640 1 -- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e5815cc90 msgr2=0x7f1e580ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e56a00640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e5815cc90 0x7f1e580ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e56a00640 1 -- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e580b2080 msgr2=0x7f1e581f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e56a00640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e580b2080 0x7f1e581f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e56a00640 1 -- 172.21.15.32:0/3540698711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e581f7b70 con 0x7f1e58145430 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.855+0000 7f1e56a00640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 0x7f1e581f2e70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1e4402b950 tx=0x7f1e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.856+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e44040070 con 0x7f1e58145430 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.856+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e44030c80 con 0x7f1e58145430 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.856+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e44035040 con 0x7f1e58145430 2024-03-06T07:31:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.856+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5813fc20 con 0x7f1e58145430 2024-03-06T07:31:20.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.856+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e58142ca0 con 0x7f1e58145430 2024-03-06T07:31:20.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.857+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e58142ac0 con 0x7f1e58145430 2024-03-06T07:31:20.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.858+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e4404b060 con 0x7f1e58145430 2024-03-06T07:31:20.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.859+0000 7f1e4f400640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.859+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e440ac060 con 0x7f1e58145430 2024-03-06T07:31:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.860+0000 7f1e57400640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.860+0000 7f1e57400640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e3806b4d0 0x7f1e3806d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1e48006fd0 tx=0x7f1e48008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:20.864+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e58142ac0 con 0x7f1e58145430 2024-03-06T07:31:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.065+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1e5806e240 con 0x7f1e58145430 2024-03-06T07:31:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.066+0000 7f1e4f400640 1 -- 172.21.15.32:0/3540698711 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1e581f7b70 con 0x7f1e58145430 2024-03-06T07:31:21.067 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.068+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e3806b4d0 msgr2=0x7f1e3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.068+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e3806b4d0 0x7f1e3806d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1e48006fd0 tx=0x7f1e48008040 comp rx=0 tx=0).stop 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.068+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 msgr2=0x7f1e581f2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.068+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 0x7f1e581f2e70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1e4402b950 tx=0x7f1e440046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 shutdown_connections 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1e5815cc90 0x7f1e580ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e58145430 0x7f1e581f2e70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 --2- 172.21.15.32:0/3540698711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e580b2080 0x7f1e581f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 >> 172.21.15.32:0/3540698711 conn(0x7f1e5806b3a0 msgr2=0x7f1e580ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 shutdown_connections 2024-03-06T07:31:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.069+0000 7f1e5d000640 1 -- 172.21.15.32:0/3540698711 wait complete. 2024-03-06T07:31:21.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:21.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.321+0000 7f469be00640 1 Processor -- start 2024-03-06T07:31:21.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.322+0000 7f469be00640 1 -- start start 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f469c15e0b0 0x7f469c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f469c0d4a10 0x7f469c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0b3710 con 0x7f469c0d0c80 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469c0d10f0 con 0x7f469c15e0b0 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469c0b09e0 con 0x7f469c0d4a10 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f469c15e0b0 0x7f469c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f469c15e0b0 0x7f469c0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52272/0 (socket says 172.21.15.32:52272) 2024-03-06T07:31:21.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.323+0000 7f469aa00640 1 -- 172.21.15.32:0/2136420367 learned_addr learned my addr 172.21.15.32:0/2136420367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:21.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.324+0000 7f469aa00640 1 -- 172.21.15.32:0/2136420367 >> v2:172.21.15.39:3300/0 conn(0x7f469c0d4a10 msgr2=0x7f469c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.324+0000 7f469aa00640 1 --2- 172.21.15.32:0/2136420367 >> v2:172.21.15.39:3300/0 conn(0x7f469c0d4a10 0x7f469c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.324+0000 7f469aa00640 1 -- 172.21.15.32:0/2136420367 >> v1:172.21.15.39:6789/0 conn(0x7f469c0d0c80 legacy=0x7f469c0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.324+0000 7f469aa00640 1 -- 172.21.15.32:0/2136420367 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0b39f0 con 0x7f469c15e0b0 2024-03-06T07:31:21.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.325+0000 7f469aa00640 1 --2- 172.21.15.32:0/2136420367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cf310 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f469c0b68c0 tx=0x7f468c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652e9c0a1953e8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.325+0000 7f4699600640 1 -- 172.21.15.32:0/2136420367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c035020 con 0x7f469c15e0b0 2024-03-06T07:31:21.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.325+0000 7f4699600640 1 -- 172.21.15.32:0/2136420367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c031640 con 0x7f469c15e0b0 2024-03-06T07:31:21.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.325+0000 7f4699600640 1 -- 172.21.15.32:0/2136420367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c02d040 con 0x7f469c15e0b0 2024-03-06T07:31:21.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 -- 172.21.15.32:0/2136420367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 msgr2=0x7f469c0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 --2- 172.21.15.32:0/2136420367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cf310 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f469c0b68c0 tx=0x7f468c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 -- 172.21.15.32:0/2136420367 shutdown_connections 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 --2- 172.21.15.32:0/2136420367 >> v2:172.21.15.39:3300/0 conn(0x7f469c0d4a10 0x7f469c0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 --2- 172.21.15.32:0/2136420367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cf310 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 -- 172.21.15.32:0/2136420367 >> 172.21.15.32:0/2136420367 conn(0x7f469c0d3d30 msgr2=0x7f469c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 -- 172.21.15.32:0/2136420367 shutdown_connections 2024-03-06T07:31:21.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.326+0000 7f469be00640 1 -- 172.21.15.32:0/2136420367 wait complete. 2024-03-06T07:31:21.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.328+0000 7f469be00640 1 Processor -- start 2024-03-06T07:31:21.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.328+0000 7f469be00640 1 -- start start 2024-03-06T07:31:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f469c0d0c80 0x7f469c0b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f469c0d4a10 0x7f469c15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c15e490 con 0x7f469c15e0b0 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0b3fb0 con 0x7f469c0d0c80 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0cfc60 con 0x7f469c0d4a10 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52288/0 (socket says 172.21.15.32:52288) 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469b400640 1 -- 172.21.15.32:0/3042954919 learned_addr learned my addr 172.21.15.32:0/3042954919 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469aa00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f469c0d0c80 0x7f469c0b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469b400640 1 -- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f469c0d4a10 msgr2=0x7f469c15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.329+0000 7f469a000640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f469c0d4a10 0x7f469c15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469b400640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f469c0d4a10 0x7f469c15c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469b400640 1 -- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f469c0d0c80 msgr2=0x7f469c0b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469b400640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f469c0d0c80 0x7f469c0b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469b400640 1 -- 172.21.15.32:0/3042954919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c15c800 con 0x7f469c15e0b0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469aa00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f469c0d0c80 0x7f469c0b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469a000640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f469c0d4a10 0x7f469c15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469b400640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cd390 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f469400c780 tx=0x7f469400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694017070 con 0x7f469c15e0b0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694003da0 con 0x7f469c15e0b0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46940052b0 con 0x7f469c15e0b0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c0ccfd0 con 0x7f469c15e0b0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.330+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c1e8f00 con 0x7f469c15e0b0 2024-03-06T07:31:21.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.331+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c04f330 con 0x7f469c15e0b0 2024-03-06T07:31:21.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.334+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f469401a070 con 0x7f469c15e0b0 2024-03-06T07:31:21.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.334+0000 7f4692a00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f467c06b4d0 0x7f467c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.335+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4694089990 con 0x7f469c15e0b0 2024-03-06T07:31:21.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.335+0000 7f469aa00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f467c06b4d0 0x7f467c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.336+0000 7f469aa00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f467c06b4d0 0x7f467c06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f468c036000 tx=0x7f468c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.339+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f469c15c800 con 0x7f469c15e0b0 2024-03-06T07:31:21.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.555+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f469c0d2810 con 0x7f469c15e0b0 2024-03-06T07:31:21.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.556+0000 7f4692a00640 1 -- 172.21.15.32:0/3042954919 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) v1 ==== 80+0+0 (secure 0 0 0) 0x7f46940586e0 con 0x7f469c15e0b0 2024-03-06T07:31:21.557 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.558+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f467c06b4d0 msgr2=0x7f467c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.558+0000 7f469be00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f467c06b4d0 0x7f467c06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f468c036000 tx=0x7f468c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.558+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 msgr2=0x7f469c0cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.558+0000 7f469be00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cd390 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f469400c780 tx=0x7f469400cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.558+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 shutdown_connections 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.558+0000 7f469be00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f467c06b4d0 0x7f467c06d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.559+0000 7f469be00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f469c15e0b0 0x7f469c0cd390 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.559+0000 7f469be00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f469c0d4a10 0x7f469c15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.559+0000 7f469be00640 1 --2- 172.21.15.32:0/3042954919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f469c0d0c80 0x7f469c0b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.559+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 >> 172.21.15.32:0/3042954919 conn(0x7f469c0d3d30 msgr2=0x7f469c0b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.559+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 shutdown_connections 2024-03-06T07:31:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:21.559+0000 7f469be00640 1 -- 172.21.15.32:0/3042954919 wait complete. 2024-03-06T07:31:21.578 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:31:21.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:21.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:21.752 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:31:21.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:21.753 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:21.800 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:31:21.800 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:21.843 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:21.844 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:21.844 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:21.883 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:21.883 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:31:21.883 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:21.939 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:21.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.147+0000 7f2aa5e00640 1 Processor -- start 2024-03-06T07:31:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.148+0000 7f2aa5e00640 1 -- start start 2024-03-06T07:31:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.149+0000 7f2aa5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2aa0070f10 0x7f2aa00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.149+0000 7f2aa5e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00714d0 con 0x7f2aa00cce90 2024-03-06T07:31:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.149+0000 7f2aa5e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00717b0 con 0x7f2aa0145300 2024-03-06T07:31:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.149+0000 7f2aa5e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00cd300 con 0x7f2aa0070f10 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.149+0000 7f2aa5400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2aa00cce90 0x7f2aa015c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44786/0 (socket says 172.21.15.32:44786) 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.149+0000 7f2aa5400640 1 -- 172.21.15.32:0/4189697050 learned_addr learned my addr 172.21.15.32:0/4189697050 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9fe00640 1 --2- 172.21.15.32:0/4189697050 >> v2:172.21.15.39:3300/0 conn(0x7f2aa0070f10 0x7f2aa00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044738519 0 0) 0x7f2aa00714d0 con 0x7f2aa00cce90 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a84003520 con 0x7f2aa00cce90 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1848625125 0 0) 0x7f2a84003520 con 0x7f2aa00cce90 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 >> v2:172.21.15.39:3300/0 conn(0x7f2aa0070f10 msgr2=0x7f2aa00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 --2- 172.21.15.32:0/4189697050 >> v2:172.21.15.39:3300/0 conn(0x7f2aa0070f10 0x7f2aa00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 >> v1:172.21.15.39:6789/0 conn(0x7f2aa0145300 legacy=0x7f2aa00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0071a90 con 0x7f2aa00cce90 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f2a98001670 con 0x7f2aa00cce90 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 >> v1:172.21.15.32:6789/0 conn(0x7f2aa00cce90 legacy=0x7f2aa015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.150+0000 7f2a9f400640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840039b0 0x7f2a84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9f400640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a84004800 0x7f2a84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9f400640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840051c0 0x7f2a8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00cd300 con 0x7f2a840039b0 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8400c0c0 con 0x7f2a84004800 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8400c2c0 con 0x7f2a840051c0 2024-03-06T07:31:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840039b0 0x7f2a84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa5400640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840051c0 0x7f2a8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9fe00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a84004800 0x7f2a84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa4a00640 1 -- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840051c0 msgr2=0x7f2a8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840051c0 0x7f2a8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa4a00640 1 -- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a84004800 msgr2=0x7f2a84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a84004800 0x7f2a84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa4a00640 1 -- 172.21.15.32:0/4189697050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84003520 con 0x7f2a840039b0 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2aa5400640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840051c0 0x7f2a8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.151+0000 7f2a9fe00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a84004800 0x7f2a84004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.152+0000 7f2aa5400640 1 -- 172.21.15.32:0/4189697050 reap_dead start 2024-03-06T07:31:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.153+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840039b0 0x7f2a84003d90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2a90001860 tx=0x7f2a84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec938cde63c0589f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90038070 con 0x7f2a840039b0 2024-03-06T07:31:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90033410 con 0x7f2a840039b0 2024-03-06T07:31:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2a9f400640 1 -- 172.21.15.32:0/4189697050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90001430 con 0x7f2a840039b0 2024-03-06T07:31:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2aa5e00640 1 -- 172.21.15.32:0/4189697050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840039b0 msgr2=0x7f2a84003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840039b0 0x7f2a84003d90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2a90001860 tx=0x7f2a84005830 comp rx=0 tx=0).stop 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2aa5e00640 1 -- 172.21.15.32:0/4189697050 shutdown_connections 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/4189697050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840039b0 0x7f2a84003d90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.154+0000 7f2aa5e00640 1 -- 172.21.15.32:0/4189697050 >> 172.21.15.32:0/4189697050 conn(0x7f2aa006b3d0 msgr2=0x7f2aa006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.155+0000 7f2aa5e00640 1 -- 172.21.15.32:0/4189697050 shutdown_connections 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.155+0000 7f2aa5e00640 1 -- 172.21.15.32:0/4189697050 wait complete. 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.156+0000 7f2aa5e00640 1 Processor -- start 2024-03-06T07:31:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.156+0000 7f2aa5e00640 1 -- start start 2024-03-06T07:31:22.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.157+0000 7f2aa5e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840039b0 0x7f2aa01f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.157+0000 7f2aa5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 0x7f2aa01f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.157+0000 7f2aa5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa01f36a0 0x7f2aa00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.157+0000 7f2aa5e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00cc950 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.157+0000 7f2aa5e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa01456e0 con 0x7f2aa01f36a0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.157+0000 7f2aa5e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00ccad0 con 0x7f2a840039b0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 0x7f2aa01f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 0x7f2aa01f1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52310/0 (socket says 172.21.15.32:52310) 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 -- 172.21.15.32:0/2357967692 learned_addr learned my addr 172.21.15.32:0/2357967692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840039b0 0x7f2aa01f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa01f36a0 0x7f2aa00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 -- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840039b0 msgr2=0x7f2aa01f48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840039b0 0x7f2aa01f48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 -- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa01f36a0 msgr2=0x7f2aa00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa01f36a0 0x7f2aa00cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2a9fe00640 1 -- 172.21.15.32:0/2357967692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa01f4df0 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.158+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa01f36a0 0x7f2aa00cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:22.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.159+0000 7f2a9fe00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 0x7f2aa01f1a70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00f3e0 tx=0x7f2a8c00f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.159+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c003990 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.159+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c003b30 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.159+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c004020 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.159+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa01f55b0 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.159+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa01f5b70 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.160+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa004f330 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.161+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a8c005d10 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.163+0000 7f2a9cc00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8006b4d0 0x7f2a8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.163+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a8c08abc0 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.163+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8006b4d0 0x7f2a8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.164+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8006b4d0 0x7f2a8006d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2a90035000 tx=0x7f2a90009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.167+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a8c024020 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.411+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa006e270 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.411+0000 7f2a9cc00640 1 -- 172.21.15.32:0/2357967692 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4965 (secure 0 0 0) 0x7f2aa01f4df0 con 0x7f2aa01f4ff0 2024-03-06T07:31:22.413 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:22.413 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:59.036395+0000","modified":"2024-03-06T07:31:02.225064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/1942129463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1942129463},{"type":"v1","addr":"172.21.15.32:6836","nonce":1942129463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":3}]} 2024-03-06T07:31:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-06T07:31:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8006b4d0 msgr2=0x7f2a8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8006b4d0 0x7f2a8006d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2a90035000 tx=0x7f2a90009030 comp rx=0 tx=0).stop 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 msgr2=0x7f2aa01f1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 0x7f2aa01f1a70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00f3e0 tx=0x7f2a8c00f8a0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 shutdown_connections 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8006b4d0 0x7f2a8006d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa01f36a0 0x7f2aa00cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa01f4ff0 0x7f2aa01f1a70 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2357967692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a840039b0 0x7f2aa01f48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 >> 172.21.15.32:0/2357967692 conn(0x7f2aa006b3d0 msgr2=0x7f2aa00b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 shutdown_connections 2024-03-06T07:31:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.415+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2357967692 wait complete. 2024-03-06T07:31:22.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.644+0000 7f3572200640 1 Processor -- start 2024-03-06T07:31:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.645+0000 7f3572200640 1 -- start start 2024-03-06T07:31:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.646+0000 7f3572200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f356c1621d0 0x7f356c168d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.646+0000 7f3572200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c1627f0 con 0x7f356c15efc0 2024-03-06T07:31:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.646+0000 7f3572200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c06c860 con 0x7f356c0cd060 2024-03-06T07:31:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.646+0000 7f3572200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c15f3a0 con 0x7f356c1621d0 2024-03-06T07:31:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.646+0000 7f3570e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f356c15efc0 0x7f356c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44792/0 (socket says 172.21.15.32:44792) 2024-03-06T07:31:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.646+0000 7f3570e00640 1 -- 172.21.15.32:0/3297658265 learned_addr learned my addr 172.21.15.32:0/3297658265 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f3571800640 1 --2- 172.21.15.32:0/3297658265 >> v2:172.21.15.39:3300/0 conn(0x7f356c1621d0 0x7f356c168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870672907 0 0) 0x7f356c1627f0 con 0x7f356c15efc0 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3554003520 con 0x7f356c15efc0 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 65299629 0 0) 0x7f3554003520 con 0x7f356c15efc0 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 >> v2:172.21.15.39:3300/0 conn(0x7f356c1621d0 msgr2=0x7f356c168d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 --2- 172.21.15.32:0/3297658265 >> v2:172.21.15.39:3300/0 conn(0x7f356c1621d0 0x7f356c168d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 >> v1:172.21.15.39:6789/0 conn(0x7f356c0cd060 legacy=0x7f356c161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c06cb40 con 0x7f356c15efc0 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3560001670 con 0x7f356c15efc0 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 >> v1:172.21.15.32:6789/0 conn(0x7f356c15efc0 legacy=0x7f356c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3554003a00 0x7f3554003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3554004800 0x7f3554004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35540051c0 0x7f355400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c15f3a0 con 0x7f35540051c0 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355400c0c0 con 0x7f3554003a00 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.647+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355400c2c0 con 0x7f3554004800 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3571800640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35540051c0 0x7f355400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3570e00640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3554003a00 0x7f3554003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f356be00640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3554004800 0x7f3554004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3571800640 1 -- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3554004800 msgr2=0x7f3554004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3571800640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3554004800 0x7f3554004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3571800640 1 -- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3554003a00 msgr2=0x7f3554003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3571800640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3554003a00 0x7f3554003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f3571800640 1 -- 172.21.15.32:0/3297658265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554003520 con 0x7f35540051c0 2024-03-06T07:31:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.648+0000 7f356be00640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3554004800 0x7f3554004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:22.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.649+0000 7f3571800640 1 -- 172.21.15.32:0/3297658265 reap_dead start 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.649+0000 7f3571800640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35540051c0 0x7f355400b9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3564009fc0 tx=0x7f3554005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab05f1937a91b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564009c10 con 0x7f35540051c0 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356402cc30 con 0x7f35540051c0 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f356b400640 1 -- 172.21.15.32:0/3297658265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564003990 con 0x7f35540051c0 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 -- 172.21.15.32:0/3297658265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35540051c0 msgr2=0x7f355400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35540051c0 0x7f355400b9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3564009fc0 tx=0x7f3554005830 comp rx=0 tx=0).stop 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 -- 172.21.15.32:0/3297658265 shutdown_connections 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 --2- 172.21.15.32:0/3297658265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35540051c0 0x7f355400b9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 -- 172.21.15.32:0/3297658265 >> 172.21.15.32:0/3297658265 conn(0x7f356c06b3d0 msgr2=0x7f356c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 -- 172.21.15.32:0/3297658265 shutdown_connections 2024-03-06T07:31:22.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.650+0000 7f3572200640 1 -- 172.21.15.32:0/3297658265 wait complete. 2024-03-06T07:31:22.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.652+0000 7f3572200640 1 Processor -- start 2024-03-06T07:31:22.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.652+0000 7f3572200640 1 -- start start 2024-03-06T07:31:22.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3572200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35540051c0 0x7f356c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3572200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c0b3900 0x7f356c0b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3572200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 0x7f356c1e9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3572200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c06d620 con 0x7f356c1e93c0 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3572200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c06d190 con 0x7f356c0b3900 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3572200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c1453b0 con 0x7f35540051c0 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3571800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 0x7f356c1e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3571800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 0x7f356c1e9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52318/0 (socket says 172.21.15.32:52318) 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.653+0000 7f3571800640 1 -- 172.21.15.32:0/1379517925 learned_addr learned my addr 172.21.15.32:0/1379517925 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3570e00640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35540051c0 0x7f356c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3571800640 1 -- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35540051c0 msgr2=0x7f356c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f356be00640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c0b3900 0x7f356c0b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3571800640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35540051c0 0x7f356c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3571800640 1 -- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c0b3900 msgr2=0x7f356c0b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3571800640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c0b3900 0x7f356c0b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3571800640 1 -- 172.21.15.32:0/1379517925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c145650 con 0x7f356c1e93c0 2024-03-06T07:31:22.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3571800640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 0x7f356c1e9840 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f356400e200 tx=0x7f3564009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.654+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564009c10 con 0x7f356c1e93c0 2024-03-06T07:31:22.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.655+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3564045070 con 0x7f356c1e93c0 2024-03-06T07:31:22.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.655+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564003990 con 0x7f356c1e93c0 2024-03-06T07:31:22.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.655+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c145930 con 0x7f356c1e93c0 2024-03-06T07:31:22.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.655+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c145ef0 con 0x7f356c1e93c0 2024-03-06T07:31:22.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.656+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c04f330 con 0x7f356c1e93c0 2024-03-06T07:31:22.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.657+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3564005d10 con 0x7f356c1e93c0 2024-03-06T07:31:22.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.658+0000 7f3568c00640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.658+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35640aca40 con 0x7f356c1e93c0 2024-03-06T07:31:22.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.659+0000 7f3570e00640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.659+0000 7f3570e00640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3560001050 tx=0x7f3560009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.665+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3564048360 con 0x7f356c1e93c0 2024-03-06T07:31:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.886+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f356c1460c0 con 0x7f356c1e93c0 2024-03-06T07:31:22.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.889+0000 7f3568c00640 1 -- 172.21.15.32:0/1379517925 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f356407b8b0 con 0x7f356c1e93c0 2024-03-06T07:31:22.891 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:22.891 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:00.589676+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 msgr2=0x7f354c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3560001050 tx=0x7f3560009040 comp rx=0 tx=0).stop 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 msgr2=0x7f356c1e9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 0x7f356c1e9840 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f356400e200 tx=0x7f3564009750 comp rx=0 tx=0).stop 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 shutdown_connections 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f354c06b4d0 0x7f354c06d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f356c1e93c0 0x7f356c1e9840 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f356c0b3900 0x7f356c0b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 --2- 172.21.15.32:0/1379517925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35540051c0 0x7f356c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 >> 172.21.15.32:0/1379517925 conn(0x7f356c06b3d0 msgr2=0x7f356c0b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 shutdown_connections 2024-03-06T07:31:22.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:22.891+0000 7f3572200640 1 -- 172.21.15.32:0/1379517925 wait complete. 2024-03-06T07:31:22.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:56.874383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:58.058939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:22.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:23.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.127+0000 7f7597a00640 1 Processor -- start 2024-03-06T07:31:23.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.127+0000 7f7597a00640 1 -- start start 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.129+0000 7f7597a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f75901615b0 0x7f75900d47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.129+0000 7f7597a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75900d1a80 0x7f75900d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.129+0000 7f7597a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75900cf670 con 0x7f75900d2810 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.129+0000 7f7597a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7590161990 con 0x7f75900d1a80 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.129+0000 7f7597a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75900d1ff0 con 0x7f75901615b0 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75900d1a80 0x7f75900d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75900d1a80 0x7f75900d22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52322/0 (socket says 172.21.15.32:52322) 2024-03-06T07:31:23.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 -- 172.21.15.32:0/3894874085 learned_addr learned my addr 172.21.15.32:0/3894874085 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:23.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f7595000640 1 --2- 172.21.15.32:0/3894874085 >> v2:172.21.15.39:3300/0 conn(0x7f75901615b0 0x7f75900d47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 -- 172.21.15.32:0/3894874085 >> v2:172.21.15.39:3300/0 conn(0x7f75901615b0 msgr2=0x7f75900d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 --2- 172.21.15.32:0/3894874085 >> v2:172.21.15.39:3300/0 conn(0x7f75901615b0 0x7f75900d47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 -- 172.21.15.32:0/3894874085 >> v1:172.21.15.39:6789/0 conn(0x7f75900d2810 legacy=0x7f75900d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f758fe00640 1 -- 172.21.15.32:0/3894874085 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75900cf910 con 0x7f75900d1a80 2024-03-06T07:31:23.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.130+0000 7f7595000640 1 --2- 172.21.15.32:0/3894874085 >> v2:172.21.15.39:3300/0 conn(0x7f75901615b0 0x7f75900d47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.131+0000 7f758fe00640 1 --2- 172.21.15.32:0/3894874085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75900d1a80 0x7f75900d22d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f75900d4750 tx=0x7f757c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b40bbbca0e563f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.132+0000 7f758f400640 1 -- 172.21.15.32:0/3894874085 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c035020 con 0x7f75900d1a80 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.132+0000 7f758f400640 1 -- 172.21.15.32:0/3894874085 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c031870 con 0x7f75900d1a80 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.132+0000 7f758f400640 1 -- 172.21.15.32:0/3894874085 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c031c40 con 0x7f75900d1a80 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.132+0000 7f7597a00640 1 -- 172.21.15.32:0/3894874085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75900d1a80 msgr2=0x7f75900d22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.132+0000 7f7597a00640 1 --2- 172.21.15.32:0/3894874085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75900d1a80 0x7f75900d22d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f75900d4750 tx=0x7f757c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.133+0000 7f7597a00640 1 -- 172.21.15.32:0/3894874085 shutdown_connections 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.133+0000 7f7597a00640 1 --2- 172.21.15.32:0/3894874085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75900d1a80 0x7f75900d22d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.133+0000 7f7597a00640 1 --2- 172.21.15.32:0/3894874085 >> v2:172.21.15.39:3300/0 conn(0x7f75901615b0 0x7f75900d47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.133+0000 7f7597a00640 1 -- 172.21.15.32:0/3894874085 >> 172.21.15.32:0/3894874085 conn(0x7f759009d180 msgr2=0x7f75900d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.133+0000 7f7597a00640 1 -- 172.21.15.32:0/3894874085 shutdown_connections 2024-03-06T07:31:23.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.133+0000 7f7597a00640 1 -- 172.21.15.32:0/3894874085 wait complete. 2024-03-06T07:31:23.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.134+0000 7f7597a00640 1 Processor -- start 2024-03-06T07:31:23.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.135+0000 7f7597a00640 1 -- start start 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.135+0000 7f7597a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75900d1a80 0x7f75901f3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7597a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75900d2810 0x7f75901f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7597a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 0x7f75901f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7597a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75900d2ca0 con 0x7f75901615b0 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7597a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75900d4d30 con 0x7f75900d2810 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7597a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75900d2e20 con 0x7f75900d1a80 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 0x7f75901f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 0x7f75901f7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52334/0 (socket says 172.21.15.32:52334) 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 -- 172.21.15.32:0/3947604025 learned_addr learned my addr 172.21.15.32:0/3947604025 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595000640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75900d1a80 0x7f75901f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f758fe00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75900d2810 0x7f75901f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 -- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75900d1a80 msgr2=0x7f75901f3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75900d1a80 0x7f75901f3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 -- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75900d2810 msgr2=0x7f75901f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.136+0000 7f7595a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75900d2810 0x7f75901f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f7595a00640 1 -- 172.21.15.32:0/3947604025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759015cba0 con 0x7f75901615b0 2024-03-06T07:31:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f7595a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 0x7f75901f7890 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f758800c780 tx=0x7f758800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588017070 con 0x7f75901615b0 2024-03-06T07:31:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588003da0 con 0x7f75901615b0 2024-03-06T07:31:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880052b0 con 0x7f75901615b0 2024-03-06T07:31:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75900b6d70 con 0x7f75901615b0 2024-03-06T07:31:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.137+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75900c96a0 con 0x7f75901615b0 2024-03-06T07:31:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.139+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759004f330 con 0x7f75901615b0 2024-03-06T07:31:23.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.139+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7588005450 con 0x7f75901615b0 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.141+0000 7f758cc00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.141+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7588089890 con 0x7f75901615b0 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.141+0000 7f7595000640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.142+0000 7f7595000640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f757006b4d0 0x7f757006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f758400a990 tx=0x7f7584008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.145+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7588058670 con 0x7f75901615b0 2024-03-06T07:31:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.400+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75900d02c0 con 0x7f75901615b0 2024-03-06T07:31:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.401+0000 7f758cc00640 1 -- 172.21.15.32:0/3947604025 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7588057f70 con 0x7f75901615b0 2024-03-06T07:31:23.402 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:31:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f757006b4d0 msgr2=0x7f757006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f757006b4d0 0x7f757006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f758400a990 tx=0x7f7584008040 comp rx=0 tx=0).stop 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 msgr2=0x7f75901f7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 0x7f75901f7890 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f758800c780 tx=0x7f758800cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 shutdown_connections 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f757006b4d0 0x7f757006d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75901615b0 0x7f75901f7890 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75900d2810 0x7f75901f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 --2- 172.21.15.32:0/3947604025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75900d1a80 0x7f75901f3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 >> 172.21.15.32:0/3947604025 conn(0x7f759009d180 msgr2=0x7f75900d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 shutdown_connections 2024-03-06T07:31:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.404+0000 7f7597a00640 1 -- 172.21.15.32:0/3947604025 wait complete. 2024-03-06T07:31:23.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.668+0000 7f53d3200640 1 Processor -- start 2024-03-06T07:31:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.669+0000 7f53d3200640 1 -- start start 2024-03-06T07:31:23.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.670+0000 7f53d3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d4071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.670+0000 7f53d3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53d406d8d0 0x7f53d40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.670+0000 7f53d3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f53d40b2080 0x7f53d40ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.670+0000 7f53d3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d40b24f0 con 0x7f53d406d8d0 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.670+0000 7f53d3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d4147970 con 0x7f53d40b2080 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.670+0000 7f53d3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40b31b0 con 0x7f53d4147590 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53d406d8d0 0x7f53d40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53d406d8d0 0x7f53d40b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52346/0 (socket says 172.21.15.32:52346) 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 -- 172.21.15.32:0/2556580007 learned_addr learned my addr 172.21.15.32:0/2556580007 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1e00640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d4071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 -- 172.21.15.32:0/2556580007 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 msgr2=0x7f53d4071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d4071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d2800640 1 --2- 172.21.15.32:0/2556580007 >> v2:172.21.15.39:3300/0 conn(0x7f53d40b2080 0x7f53d40ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 -- 172.21.15.32:0/2556580007 >> v2:172.21.15.39:3300/0 conn(0x7f53d40b2080 msgr2=0x7f53d40ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 --2- 172.21.15.32:0/2556580007 >> v2:172.21.15.39:3300/0 conn(0x7f53d40b2080 0x7f53d40ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1400640 1 -- 172.21.15.32:0/2556580007 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40719c0 con 0x7f53d406d8d0 2024-03-06T07:31:23.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.671+0000 7f53d1e00640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d4071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.673+0000 7f53d1400640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d406d8d0 0x7f53d40b1b40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f53d40b2720 tx=0x7f53c002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74156386805a597 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.674+0000 7f53d0a00640 1 -- 172.21.15.32:0/2556580007 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0035020 con 0x7f53d406d8d0 2024-03-06T07:31:23.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.674+0000 7f53d0a00640 1 -- 172.21.15.32:0/2556580007 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c002d040 con 0x7f53d406d8d0 2024-03-06T07:31:23.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.674+0000 7f53d0a00640 1 -- 172.21.15.32:0/2556580007 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0031a80 con 0x7f53d406d8d0 2024-03-06T07:31:23.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.674+0000 7f53d3200640 1 -- 172.21.15.32:0/2556580007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d406d8d0 msgr2=0x7f53d40b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.674+0000 7f53d3200640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d406d8d0 0x7f53d40b1b40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f53d40b2720 tx=0x7f53c002b900 comp rx=0 tx=0).stop 2024-03-06T07:31:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 -- 172.21.15.32:0/2556580007 shutdown_connections 2024-03-06T07:31:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 --2- 172.21.15.32:0/2556580007 >> v2:172.21.15.39:3300/0 conn(0x7f53d40b2080 0x7f53d40ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d406d8d0 0x7f53d40b1b40 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 --2- 172.21.15.32:0/2556580007 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d4071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 -- 172.21.15.32:0/2556580007 >> 172.21.15.32:0/2556580007 conn(0x7f53d406b3a0 msgr2=0x7f53d406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 -- 172.21.15.32:0/2556580007 shutdown_connections 2024-03-06T07:31:23.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.675+0000 7f53d3200640 1 -- 172.21.15.32:0/2556580007 wait complete. 2024-03-06T07:31:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.676+0000 7f53d3200640 1 Processor -- start 2024-03-06T07:31:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.677+0000 7f53d3200640 1 -- start start 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d406d8d0 0x7f53d4143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 0x7f53d40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40b2e40 con 0x7f53d40b2080 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d406e120 con 0x7f53d406d8d0 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40b2b50 con 0x7f53d4147590 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 0x7f53d40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 0x7f53d40c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52360/0 (socket says 172.21.15.32:52360) 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 -- 172.21.15.32:0/3425143570 learned_addr learned my addr 172.21.15.32:0/3425143570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1e00640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d406d8d0 0x7f53d4143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d2800640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 -- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 msgr2=0x7f53d415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d415ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 -- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d406d8d0 msgr2=0x7f53d4143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.678+0000 7f53d1400640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d406d8d0 0x7f53d4143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53d1400640 1 -- 172.21.15.32:0/3425143570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40c4b60 con 0x7f53d40b2080 2024-03-06T07:31:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53d2800640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d415ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53d1400640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 0x7f53d40c9270 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f53c0000c00 tx=0x7f53c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0040070 con 0x7f53d40b2080 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c0039bf0 con 0x7f53d40b2080 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d40c4e40 con 0x7f53d40b2080 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.679+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0035040 con 0x7f53d40b2080 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.680+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d40c5400 con 0x7f53d40b2080 2024-03-06T07:31:23.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.681+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d404f330 con 0x7f53d40b2080 2024-03-06T07:31:23.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.682+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53c0038020 con 0x7f53d40b2080 2024-03-06T07:31:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.684+0000 7f53ca000640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f53b406b4d0 0x7f53b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.685+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53c00abff0 con 0x7f53d40b2080 2024-03-06T07:31:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.685+0000 7f53d1e00640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f53b406b4d0 0x7f53b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.685+0000 7f53d1e00640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f53b406b4d0 0x7f53b406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53c400a8d0 tx=0x7f53c4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.689+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53c007ae80 con 0x7f53d40b2080 2024-03-06T07:31:23.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.886+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f53d406b8b0 con 0x7f53d40b2080 2024-03-06T07:31:23.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.886+0000 7f53ca000640 1 -- 172.21.15.32:0/3425143570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f53c007a780 con 0x7f53d40b2080 2024-03-06T07:31:23.888 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:31:23.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f53b406b4d0 msgr2=0x7f53b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f53b406b4d0 0x7f53b406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53c400a8d0 tx=0x7f53c4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 msgr2=0x7f53d40c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 0x7f53d40c9270 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f53c0000c00 tx=0x7f53c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 shutdown_connections 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f53b406b4d0 0x7f53b406d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f53d4147590 0x7f53d415ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d40b2080 0x7f53d40c9270 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 --2- 172.21.15.32:0/3425143570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d406d8d0 0x7f53d4143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 >> 172.21.15.32:0/3425143570 conn(0x7f53d406b3a0 msgr2=0x7f53d41467c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 shutdown_connections 2024-03-06T07:31:23.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:23.889+0000 7f53d3200640 1 -- 172.21.15.32:0/3425143570 wait complete. 2024-03-06T07:31:23.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.122+0000 7f6beb200640 1 Processor -- start 2024-03-06T07:31:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.122+0000 7f6beb200640 1 -- start start 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.123+0000 7f6beb200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be40d1a80 0x7f6be40d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.123+0000 7f6beb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40cb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6beb200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be40cf670 con 0x7f6be41615b0 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6beb200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be4161990 con 0x7f6be40d1a80 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6beb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be40d1ff0 con 0x7f6be40d2810 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be40d1a80 0x7f6be40d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6be41615b0 0x7f6be40d47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44800/0 (socket says 172.21.15.32:44800) 2024-03-06T07:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6be40d1a80 0x7f6be40d22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52362/0 (socket says 172.21.15.32:52362) 2024-03-06T07:31:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3e00640 1 -- 172.21.15.32:0/2443595237 learned_addr learned my addr 172.21.15.32:0/2443595237 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be9200640 1 --2- 172.21.15.32:0/2443595237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3400640 1 -- 172.21.15.32:0/2443595237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 msgr2=0x7f6be40cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3400640 1 --2- 172.21.15.32:0/2443595237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3400640 1 -- 172.21.15.32:0/2443595237 >> v1:172.21.15.32:6789/0 conn(0x7f6be41615b0 legacy=0x7f6be40d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.124+0000 7f6be3400640 1 -- 172.21.15.32:0/2443595237 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40cf910 con 0x7f6be40d1a80 2024-03-06T07:31:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.126+0000 7f6be3400640 1 --2- 172.21.15.32:0/2443595237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be40d22d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0097a0 tx=0x7f6bcc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c6d41fbe061bb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.126+0000 7f6be2a00640 1 -- 172.21.15.32:0/2443595237 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc035020 con 0x7f6be40d1a80 2024-03-06T07:31:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.126+0000 7f6be2a00640 1 -- 172.21.15.32:0/2443595237 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc031810 con 0x7f6be40d1a80 2024-03-06T07:31:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.126+0000 7f6be2a00640 1 -- 172.21.15.32:0/2443595237 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc031be0 con 0x7f6be40d1a80 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.126+0000 7f6beb200640 1 -- 172.21.15.32:0/2443595237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 msgr2=0x7f6be40d22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.126+0000 7f6beb200640 1 --2- 172.21.15.32:0/2443595237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be40d22d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0097a0 tx=0x7f6bcc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.127+0000 7f6beb200640 1 -- 172.21.15.32:0/2443595237 shutdown_connections 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.127+0000 7f6beb200640 1 --2- 172.21.15.32:0/2443595237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be40d22d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.127+0000 7f6beb200640 1 --2- 172.21.15.32:0/2443595237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.127+0000 7f6beb200640 1 -- 172.21.15.32:0/2443595237 >> 172.21.15.32:0/2443595237 conn(0x7f6be409d180 msgr2=0x7f6be40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.127+0000 7f6beb200640 1 -- 172.21.15.32:0/2443595237 shutdown_connections 2024-03-06T07:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.127+0000 7f6beb200640 1 -- 172.21.15.32:0/2443595237 wait complete. 2024-03-06T07:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.128+0000 7f6beb200640 1 Processor -- start 2024-03-06T07:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.129+0000 7f6beb200640 1 -- start start 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.129+0000 7f6beb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be4155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.129+0000 7f6beb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40b93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6beb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6be41615b0 0x7f6be40c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6beb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be40d2e20 con 0x7f6be40d1a80 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6beb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be40d2ca0 con 0x7f6be41615b0 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6beb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be40cb670 con 0x7f6be40d2810 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be4155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be4155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52378/0 (socket says 172.21.15.32:52378) 2024-03-06T07:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 -- 172.21.15.32:0/168354833 learned_addr learned my addr 172.21.15.32:0/168354833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 -- 172.21.15.32:0/168354833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 msgr2=0x7f6be40b93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40b93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 -- 172.21.15.32:0/168354833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6be41615b0 msgr2=0x7f6be40c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6be41615b0 0x7f6be40c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.130+0000 7f6be3e00640 1 -- 172.21.15.32:0/168354833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be4155a60 con 0x7f6be40d1a80 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.131+0000 7f6be3e00640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be4155130 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400c3c0 tx=0x7f6bd400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.131+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4017070 con 0x7f6be40d1a80 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.131+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4005ce0 con 0x7f6be40d1a80 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.131+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40072f0 con 0x7f6be40d1a80 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.131+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be40b98e0 con 0x7f6be40d1a80 2024-03-06T07:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.131+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be40b7660 con 0x7f6be40d1a80 2024-03-06T07:31:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.132+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be40b7470 con 0x7f6be40d1a80 2024-03-06T07:31:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.134+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bd401a070 con 0x7f6be40d1a80 2024-03-06T07:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.135+0000 7f6bdbe00640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6bc006b500 0x7f6bc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.135+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bd4089570 con 0x7f6be40d1a80 2024-03-06T07:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.135+0000 7f6be3400640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6bc006b500 0x7f6bc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.136+0000 7f6be3400640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6bc006b500 0x7f6bc006d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.140+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6be4155a60 con 0x7f6be40d1a80 2024-03-06T07:31:24.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.373+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6be404f330 con 0x7f6be40d1a80 2024-03-06T07:31:24.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.373+0000 7f6bdbe00640 1 -- 172.21.15.32:0/168354833 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6bd4087e20 con 0x7f6be40d1a80 2024-03-06T07:31:24.375 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:31:24.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6bc006b500 msgr2=0x7f6bc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6bc006b500 0x7f6bc006d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc030000 comp rx=0 tx=0).stop 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 msgr2=0x7f6be4155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be4155130 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400c3c0 tx=0x7f6bd400c880 comp rx=0 tx=0).stop 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 shutdown_connections 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6bc006b500 0x7f6bc006d9c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6be41615b0 0x7f6be40c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6be40d2810 0x7f6be40b93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 --2- 172.21.15.32:0/168354833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6be40d1a80 0x7f6be4155130 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 >> 172.21.15.32:0/168354833 conn(0x7f6be409d180 msgr2=0x7f6be41608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 shutdown_connections 2024-03-06T07:31:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.377+0000 7f6beb200640 1 -- 172.21.15.32:0/168354833 wait complete. 2024-03-06T07:31:24.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.614+0000 7fb27c200640 1 Processor -- start 2024-03-06T07:31:24.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.615+0000 7fb27c200640 1 -- start start 2024-03-06T07:31:24.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.616+0000 7fb27c200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb27415bc60 0x7fb2740af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.616+0000 7fb27c200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb2741600c0 0x7fb2740af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.616+0000 7fb27c200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740b0690 con 0x7fb27415c040 2024-03-06T07:31:24.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.616+0000 7fb27c200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2740afe10 con 0x7fb27415bc60 2024-03-06T07:31:24.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.616+0000 7fb27c200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2740b00d0 con 0x7fb2741600c0 2024-03-06T07:31:24.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb27415bc60 0x7fb2740af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb27415bc60 0x7fb2740af2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52390/0 (socket says 172.21.15.32:52390) 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 -- 172.21.15.32:0/352959675 learned_addr learned my addr 172.21.15.32:0/352959675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb27a200640 1 --2- 172.21.15.32:0/352959675 >> v2:172.21.15.39:3300/0 conn(0x7fb2741600c0 0x7fb2740af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 -- 172.21.15.32:0/352959675 >> v2:172.21.15.39:3300/0 conn(0x7fb2741600c0 msgr2=0x7fb2740af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 --2- 172.21.15.32:0/352959675 >> v2:172.21.15.39:3300/0 conn(0x7fb2741600c0 0x7fb2740af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 -- 172.21.15.32:0/352959675 >> v1:172.21.15.32:6789/0 conn(0x7fb27415c040 legacy=0x7fb27415f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb279800640 1 -- 172.21.15.32:0/352959675 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740b03b0 con 0x7fb27415bc60 2024-03-06T07:31:24.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.617+0000 7fb27a200640 1 --2- 172.21.15.32:0/352959675 >> v2:172.21.15.39:3300/0 conn(0x7fb2741600c0 0x7fb2740af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:24.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.619+0000 7fb279800640 1 --2- 172.21.15.32:0/352959675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2740af2b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb27415c440 tx=0x7fb26802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f5d79c6468c75d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb273e00640 1 -- 172.21.15.32:0/352959675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268035020 con 0x7fb27415bc60 2024-03-06T07:31:24.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb273e00640 1 -- 172.21.15.32:0/352959675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb268031870 con 0x7fb27415bc60 2024-03-06T07:31:24.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb273e00640 1 -- 172.21.15.32:0/352959675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268031c40 con 0x7fb27415bc60 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb27c200640 1 -- 172.21.15.32:0/352959675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 msgr2=0x7fb2740af2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb27c200640 1 --2- 172.21.15.32:0/352959675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2740af2b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb27415c440 tx=0x7fb26802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb27c200640 1 -- 172.21.15.32:0/352959675 shutdown_connections 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb27c200640 1 --2- 172.21.15.32:0/352959675 >> v2:172.21.15.39:3300/0 conn(0x7fb2741600c0 0x7fb2740af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb27c200640 1 --2- 172.21.15.32:0/352959675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2740af2b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.620+0000 7fb27c200640 1 -- 172.21.15.32:0/352959675 >> 172.21.15.32:0/352959675 conn(0x7fb27409aa10 msgr2=0x7fb27409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.621+0000 7fb27c200640 1 -- 172.21.15.32:0/352959675 shutdown_connections 2024-03-06T07:31:24.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.621+0000 7fb27c200640 1 -- 172.21.15.32:0/352959675 wait complete. 2024-03-06T07:31:24.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.622+0000 7fb27c200640 1 Processor -- start 2024-03-06T07:31:24.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.623+0000 7fb27c200640 1 -- start start 2024-03-06T07:31:24.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.623+0000 7fb27c200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2741f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb27c200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb27415c040 0x7fb2741580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb27c200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2741600c0 0x7fb274158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb27c200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27406f6f0 con 0x7fb27415bc60 2024-03-06T07:31:24.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb27c200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27406f570 con 0x7fb27415c040 2024-03-06T07:31:24.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb27c200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27406f870 con 0x7fb2741600c0 2024-03-06T07:31:24.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb279800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2741f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb279800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2741f4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52396/0 (socket says 172.21.15.32:52396) 2024-03-06T07:31:24.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb279800640 1 -- 172.21.15.32:0/798337149 learned_addr learned my addr 172.21.15.32:0/798337149 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:24.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb27a200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2741600c0 0x7fb274158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.624+0000 7fb278e00640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb27415c040 0x7fb2741580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb279800640 1 -- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2741600c0 msgr2=0x7fb274158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb279800640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2741600c0 0x7fb274158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb279800640 1 -- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb27415c040 msgr2=0x7fb2741580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb279800640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb27415c040 0x7fb2741580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb279800640 1 -- 172.21.15.32:0/798337149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2741585f0 con 0x7fb27415bc60 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb279800640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2741f4c70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb26802b950 tx=0x7fb2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268040070 con 0x7fb27415bc60 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26803a420 con 0x7fb27415bc60 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27415ad90 con 0x7fb27415bc60 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.625+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268035040 con 0x7fb27415bc60 2024-03-06T07:31:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.626+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27406c300 con 0x7fb27415bc60 2024-03-06T07:31:24.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.627+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27404f3a0 con 0x7fb27415bc60 2024-03-06T07:31:24.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.628+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb26804b060 con 0x7fb27415bc60 2024-03-06T07:31:24.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.628+0000 7fb271600640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25406b4d0 0x7fb25406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.629+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2680ac210 con 0x7fb27415bc60 2024-03-06T07:31:24.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.629+0000 7fb278e00640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25406b4d0 0x7fb25406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.630+0000 7fb278e00640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25406b4d0 0x7fb25406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb27406dfe0 tx=0x7fb260009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.633+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2741585f0 con 0x7fb27415bc60 2024-03-06T07:31:24.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.876+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb27406c0c0 con 0x7fb27415bc60 2024-03-06T07:31:24.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.877+0000 7fb271600640 1 -- 172.21.15.32:0/798337149 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4965 (secure 0 0 0) 0x7fb26807aff0 con 0x7fb27415bc60 2024-03-06T07:31:24.879 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:24.879 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:59.036395+0000","modified":"2024-03-06T07:31:02.225064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/1942129463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1942129463},{"type":"v1","addr":"172.21.15.32:6836","nonce":1942129463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":3}]} 2024-03-06T07:31:24.879 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-06T07:31:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25406b4d0 msgr2=0x7fb25406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25406b4d0 0x7fb25406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb27406dfe0 tx=0x7fb260009040 comp rx=0 tx=0).stop 2024-03-06T07:31:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 msgr2=0x7fb2741f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2741f4c70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb26802b950 tx=0x7fb2680046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 shutdown_connections 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb25406b4d0 0x7fb25406d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb2741600c0 0x7fb274158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb27415c040 0x7fb2741580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 --2- 172.21.15.32:0/798337149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb27415bc60 0x7fb2741f4c70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 >> 172.21.15.32:0/798337149 conn(0x7fb27409aa10 msgr2=0x7fb2740b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 shutdown_connections 2024-03-06T07:31:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:24.882+0000 7fb27c200640 1 -- 172.21.15.32:0/798337149 wait complete. 2024-03-06T07:31:24.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:25.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.110+0000 7f7527200640 1 Processor -- start 2024-03-06T07:31:25.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.110+0000 7f7527200640 1 -- start start 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.111+0000 7f7527200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f752015dd00 0x7f7520162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.111+0000 7f7527200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752006bfc0 0x7f752006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.111+0000 7f7527200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7520161a80 con 0x7f752015e0e0 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.111+0000 7f7527200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752006c960 con 0x7f752015dd00 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.111+0000 7f7527200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752015efa0 con 0x7f752006bfc0 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f751fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f752015dd00 0x7f7520162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f7525200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f752015e0e0 0x7f752015e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44818/0 (socket says 172.21.15.32:44818) 2024-03-06T07:31:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f7525200640 1 -- 172.21.15.32:0/2691193400 learned_addr learned my addr 172.21.15.32:0/2691193400 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f7517e00640 1 --2- 172.21.15.32:0/2691193400 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752006bfc0 0x7f752006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f751fe00640 1 -- 172.21.15.32:0/2691193400 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752006bfc0 msgr2=0x7f752006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f751fe00640 1 --2- 172.21.15.32:0/2691193400 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752006bfc0 0x7f752006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f751fe00640 1 -- 172.21.15.32:0/2691193400 >> v1:172.21.15.32:6789/0 conn(0x7f752015e0e0 legacy=0x7f752015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f751fe00640 1 -- 172.21.15.32:0/2691193400 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752015f280 con 0x7f752015dd00 2024-03-06T07:31:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.112+0000 7f751f400640 1 -- 172.21.15.32:0/2691193400 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633920122 0 0) 0x7f7520161a80 con 0x7f752015e0e0 2024-03-06T07:31:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.114+0000 7f751fe00640 1 --2- 172.21.15.32:0/2691193400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752015dd00 0x7f7520162520 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f75100097a0 tx=0x7f751002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52e165014c0bddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.114+0000 7f751f400640 1 -- 172.21.15.32:0/2691193400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510035020 con 0x7f752015dd00 2024-03-06T07:31:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.114+0000 7f751f400640 1 -- 172.21.15.32:0/2691193400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510031830 con 0x7f752015dd00 2024-03-06T07:31:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.114+0000 7f751f400640 1 -- 172.21.15.32:0/2691193400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510031c00 con 0x7f752015dd00 2024-03-06T07:31:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 -- 172.21.15.32:0/2691193400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752015dd00 msgr2=0x7f7520162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 --2- 172.21.15.32:0/2691193400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752015dd00 0x7f7520162520 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f75100097a0 tx=0x7f751002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 -- 172.21.15.32:0/2691193400 shutdown_connections 2024-03-06T07:31:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 --2- 172.21.15.32:0/2691193400 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752006bfc0 0x7f752006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 --2- 172.21.15.32:0/2691193400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752015dd00 0x7f7520162520 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 -- 172.21.15.32:0/2691193400 >> 172.21.15.32:0/2691193400 conn(0x7f75200714d0 msgr2=0x7f75200718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 -- 172.21.15.32:0/2691193400 shutdown_connections 2024-03-06T07:31:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.115+0000 7f7527200640 1 -- 172.21.15.32:0/2691193400 wait complete. 2024-03-06T07:31:25.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.116+0000 7f7527200640 1 Processor -- start 2024-03-06T07:31:25.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.117+0000 7f7527200640 1 -- start start 2024-03-06T07:31:25.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f7527200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 0x7f7520143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f7527200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752015dd00 0x7f75201f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f7527200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752015e0e0 0x7f75201f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f7527200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752015ec60 con 0x7f752006bfc0 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f7527200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7520162000 con 0x7f752015dd00 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f7527200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752006f280 con 0x7f752015e0e0 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 0x7f7520143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 0x7f7520143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52410/0 (socket says 172.21.15.32:52410) 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 -- 172.21.15.32:0/3403624770 learned_addr learned my addr 172.21.15.32:0/3403624770 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 -- 172.21.15.32:0/3403624770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752015e0e0 msgr2=0x7f75201f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752015e0e0 0x7f75201f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 -- 172.21.15.32:0/3403624770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752015dd00 msgr2=0x7f75201f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752015dd00 0x7f75201f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.118+0000 7f751fe00640 1 -- 172.21.15.32:0/3403624770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520160f00 con 0x7f752006bfc0 2024-03-06T07:31:25.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.119+0000 7f751fe00640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 0x7f7520143d20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f751002bee0 tx=0x7f75100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.119+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510041070 con 0x7f752006bfc0 2024-03-06T07:31:25.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.119+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751003b920 con 0x7f752006bfc0 2024-03-06T07:31:25.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.119+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510035040 con 0x7f752006bfc0 2024-03-06T07:31:25.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.119+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75201444d0 con 0x7f752006bfc0 2024-03-06T07:31:25.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.119+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7520144a90 con 0x7f752006bfc0 2024-03-06T07:31:25.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.120+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752004f3a0 con 0x7f752006bfc0 2024-03-06T07:31:25.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.122+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f751004c060 con 0x7f752006bfc0 2024-03-06T07:31:25.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.122+0000 7f751cc00640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74fc06b4d0 0x7f74fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.123+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75100abc20 con 0x7f752006bfc0 2024-03-06T07:31:25.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.123+0000 7f7517e00640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74fc06b4d0 0x7f74fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.124+0000 7f7517e00640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74fc06b4d0 0x7f74fc06d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f752015cb10 tx=0x7f7508008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.128+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f751007aa10 con 0x7f752006bfc0 2024-03-06T07:31:25.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.330+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75200724c0 con 0x7f752006bfc0 2024-03-06T07:31:25.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.332+0000 7f751cc00640 1 -- 172.21.15.32:0/3403624770 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f751007a310 con 0x7f752006bfc0 2024-03-06T07:31:25.333 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:25.333 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:00.589676+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74fc06b4d0 msgr2=0x7f74fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74fc06b4d0 0x7f74fc06d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f752015cb10 tx=0x7f7508008040 comp rx=0 tx=0).stop 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 msgr2=0x7f7520143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 0x7f7520143d20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f751002bee0 tx=0x7f75100046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 shutdown_connections 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74fc06b4d0 0x7f74fc06d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f752015e0e0 0x7f75201f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752015dd00 0x7f75201f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 --2- 172.21.15.32:0/3403624770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f752006bfc0 0x7f7520143d20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 >> 172.21.15.32:0/3403624770 conn(0x7f75200714d0 msgr2=0x7f75200af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.334+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 shutdown_connections 2024-03-06T07:31:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.335+0000 7f7527200640 1 -- 172.21.15.32:0/3403624770 wait complete. 2024-03-06T07:31:25.355 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:25.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:25.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.553+0000 7f2e0dc00640 1 Processor -- start 2024-03-06T07:31:25.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.554+0000 7f2e0dc00640 1 -- start start 2024-03-06T07:31:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e0dc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e080af1a0 0x7f2e0815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e0dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2e080af580 0x7f2e080b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e0dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080cdc20 0x7f2e080b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e0dc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0815cc40 con 0x7f2e080af1a0 2024-03-06T07:31:25.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e0dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0815cf20 con 0x7f2e080af580 2024-03-06T07:31:25.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e0dc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0815d200 con 0x7f2e080cdc20 2024-03-06T07:31:25.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e06a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e080af1a0 0x7f2e0815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e06a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e080af1a0 0x7f2e0815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52414/0 (socket says 172.21.15.32:52414) 2024-03-06T07:31:25.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.555+0000 7f2e06a00640 1 -- 172.21.15.32:0/152489733 learned_addr learned my addr 172.21.15.32:0/152489733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e07400640 1 --2- 172.21.15.32:0/152489733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080cdc20 0x7f2e080b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e06000640 1 --2- 172.21.15.32:0/152489733 >> v2:172.21.15.39:3300/0 conn(0x7f2e080af580 0x7f2e080b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e06a00640 1 -- 172.21.15.32:0/152489733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080cdc20 msgr2=0x7f2e080b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e06a00640 1 --2- 172.21.15.32:0/152489733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080cdc20 0x7f2e080b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e06a00640 1 -- 172.21.15.32:0/152489733 >> v2:172.21.15.39:3300/0 conn(0x7f2e080af580 msgr2=0x7f2e080b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e06a00640 1 --2- 172.21.15.32:0/152489733 >> v2:172.21.15.39:3300/0 conn(0x7f2e080af580 0x7f2e080b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.556+0000 7f2e06a00640 1 -- 172.21.15.32:0/152489733 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0815d4e0 con 0x7f2e080af1a0 2024-03-06T07:31:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.557+0000 7f2e06a00640 1 --2- 172.21.15.32:0/152489733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0815c600 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2e080b54c0 tx=0x7f2df802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8ab8b75aa21d35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.557+0000 7f2e05600640 1 -- 172.21.15.32:0/152489733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df802e070 con 0x7f2e080af1a0 2024-03-06T07:31:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.557+0000 7f2e05600640 1 -- 172.21.15.32:0/152489733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df8037070 con 0x7f2e080af1a0 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.557+0000 7f2e05600640 1 -- 172.21.15.32:0/152489733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8031a00 con 0x7f2e080af1a0 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 -- 172.21.15.32:0/152489733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 msgr2=0x7f2e0815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/152489733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0815c600 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2e080b54c0 tx=0x7f2df802b880 comp rx=0 tx=0).stop 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 -- 172.21.15.32:0/152489733 shutdown_connections 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/152489733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080cdc20 0x7f2e080b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/152489733 >> v2:172.21.15.39:3300/0 conn(0x7f2e080af580 0x7f2e080b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/152489733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0815c600 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 -- 172.21.15.32:0/152489733 >> 172.21.15.32:0/152489733 conn(0x7f2e0806afb0 msgr2=0x7f2e0806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 -- 172.21.15.32:0/152489733 shutdown_connections 2024-03-06T07:31:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.558+0000 7f2e0dc00640 1 -- 172.21.15.32:0/152489733 wait complete. 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.560+0000 7f2e0dc00640 1 Processor -- start 2024-03-06T07:31:25.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.560+0000 7f2e0dc00640 1 -- start start 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e0dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e0dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080af580 0x7f2e0809d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e0dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080cdc20 0x7f2e0814c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e0dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080b3490 con 0x7f2e080af1a0 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e0dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080b3310 con 0x7f2e080cdc20 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e0dc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080b3610 con 0x7f2e080af580 2024-03-06T07:31:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0809aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52428/0 (socket says 172.21.15.32:52428) 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.561+0000 7f2e06a00640 1 -- 172.21.15.32:0/3222985898 learned_addr learned my addr 172.21.15.32:0/3222985898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06000640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080af580 0x7f2e0809d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06a00640 1 -- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080af580 msgr2=0x7f2e0809d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06a00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080af580 0x7f2e0809d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e07400640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080cdc20 0x7f2e0814c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06a00640 1 -- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080cdc20 msgr2=0x7f2e0814c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06a00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080cdc20 0x7f2e0814c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06a00640 1 -- 172.21.15.32:0/3222985898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0814cbe0 con 0x7f2e080af1a0 2024-03-06T07:31:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2e06a00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0809aa10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2df8000c00 tx=0x7f2df80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8032670 con 0x7f2e080af1a0 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df8032850 con 0x7f2e080af1a0 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.562+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8032c40 con 0x7f2e080af1a0 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.563+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e080c93d0 con 0x7f2e080af1a0 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.563+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e080c9990 con 0x7f2e080af1a0 2024-03-06T07:31:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.564+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e08069510 con 0x7f2e080af1a0 2024-03-06T07:31:25.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.565+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2df804c060 con 0x7f2e080af1a0 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.566+0000 7f2dfea00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2de806b4d0 0x7f2de806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.566+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2df80abce0 con 0x7f2e080af1a0 2024-03-06T07:31:25.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.566+0000 7f2e06000640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2de806b4d0 0x7f2de806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.567+0000 7f2e06000640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2de806b4d0 0x7f2de806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e0809af50 tx=0x7f2df4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.570+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2df8040070 con 0x7f2e080af1a0 2024-03-06T07:31:25.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.899+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e080ad090 con 0x7f2e080af1a0 2024-03-06T07:31:25.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.900+0000 7f2dfea00640 1 -- 172.21.15.32:0/3222985898 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4965 (secure 0 0 0) 0x7f2df807aad0 con 0x7f2e080af1a0 2024-03-06T07:31:25.902 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:25.902 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:59.036395+0000","modified":"2024-03-06T07:31:02.225064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/1942129463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1942129463},{"type":"v1","addr":"172.21.15.32:6836","nonce":1942129463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":3}]} 2024-03-06T07:31:25.902 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2de806b4d0 msgr2=0x7f2de806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2de806b4d0 0x7f2de806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e0809af50 tx=0x7f2df4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 msgr2=0x7f2e0809aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0809aa10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2df8000c00 tx=0x7f2df80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 shutdown_connections 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2de806b4d0 0x7f2de806d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080cdc20 0x7f2e0814c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2e080af580 0x7f2e0809d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 --2- 172.21.15.32:0/3222985898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080af1a0 0x7f2e0809aa10 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 >> 172.21.15.32:0/3222985898 conn(0x7f2e0806afb0 msgr2=0x7f2e080cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 shutdown_connections 2024-03-06T07:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:25.904+0000 7f2e0dc00640 1 -- 172.21.15.32:0/3222985898 wait complete. 2024-03-06T07:31:25.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.111+0000 7f8787e00640 1 Processor -- start 2024-03-06T07:31:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.111+0000 7f8787e00640 1 -- start start 2024-03-06T07:31:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8787e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8788151b30 0x7f878806f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8787e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f87880b4650 0x7f87880b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8787e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878806e4b0 con 0x7f878815c4a0 2024-03-06T07:31:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8787e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87880b4fd0 con 0x7f8788151b30 2024-03-06T07:31:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8787e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878806e790 con 0x7f87880b4650 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8786a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8788151b30 0x7f878806f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8786a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8788151b30 0x7f878806f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52444/0 (socket says 172.21.15.32:52444) 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8786a00640 1 -- 172.21.15.32:0/3846860472 learned_addr learned my addr 172.21.15.32:0/3846860472 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8786a00640 1 -- 172.21.15.32:0/3846860472 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f87880b4650 msgr2=0x7f87880b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8786a00640 1 --2- 172.21.15.32:0/3846860472 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f87880b4650 0x7f87880b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.113+0000 7f8786a00640 1 -- 172.21.15.32:0/3846860472 >> v1:172.21.15.39:6789/0 conn(0x7f878815c4a0 legacy=0x7f878815c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.114+0000 7f8786a00640 1 -- 172.21.15.32:0/3846860472 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878806ea70 con 0x7f8788151b30 2024-03-06T07:31:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.115+0000 7f8786a00640 1 --2- 172.21.15.32:0/3846860472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8788151b30 0x7f878806f270 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8778009e50 tx=0x7f877802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8fd17977c68b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.115+0000 7f8786000640 1 -- 172.21.15.32:0/3846860472 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877802c9e0 con 0x7f8788151b30 2024-03-06T07:31:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.115+0000 7f8786000640 1 -- 172.21.15.32:0/3846860472 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877802cb80 con 0x7f8788151b30 2024-03-06T07:31:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.115+0000 7f8786000640 1 -- 172.21.15.32:0/3846860472 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780343f0 con 0x7f8788151b30 2024-03-06T07:31:26.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 -- 172.21.15.32:0/3846860472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8788151b30 msgr2=0x7f878806f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 --2- 172.21.15.32:0/3846860472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8788151b30 0x7f878806f270 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8778009e50 tx=0x7f877802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:26.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 -- 172.21.15.32:0/3846860472 shutdown_connections 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 --2- 172.21.15.32:0/3846860472 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f87880b4650 0x7f87880b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 --2- 172.21.15.32:0/3846860472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8788151b30 0x7f878806f270 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 -- 172.21.15.32:0/3846860472 >> 172.21.15.32:0/3846860472 conn(0x7f878806c0c0 msgr2=0x7f878806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 -- 172.21.15.32:0/3846860472 shutdown_connections 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.116+0000 7f8787e00640 1 -- 172.21.15.32:0/3846860472 wait complete. 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.117+0000 7f8787e00640 1 Processor -- start 2024-03-06T07:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 -- start start 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 0x7f87881436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8788151b30 0x7f8788152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f878815c4a0 0x7f8788152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878806f7b0 con 0x7f87880b4650 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8788151f10 con 0x7f8788151b30 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.118+0000 7f8787e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8788166c00 con 0x7f878815c4a0 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 0x7f87881436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 0x7f87881436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52456/0 (socket says 172.21.15.32:52456) 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 -- 172.21.15.32:0/2877746250 learned_addr learned my addr 172.21.15.32:0/2877746250 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 -- 172.21.15.32:0/2877746250 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f878815c4a0 msgr2=0x7f8788152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f878815c4a0 0x7f8788152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 -- 172.21.15.32:0/2877746250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8788151b30 msgr2=0x7f8788152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8788151b30 0x7f8788152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 -- 172.21.15.32:0/2877746250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788166ee0 con 0x7f87880b4650 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.119+0000 7f8786a00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 0x7f87881436a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8778036040 tx=0x7f8778004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.120+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778043020 con 0x7f87880b4650 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.120+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877803e450 con 0x7f87880b4650 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.120+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778009510 con 0x7f87880b4650 2024-03-06T07:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.120+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87881671c0 con 0x7f87880b4650 2024-03-06T07:31:26.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.120+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8788167ca0 con 0x7f87880b4650 2024-03-06T07:31:26.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.121+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878804f330 con 0x7f87880b4650 2024-03-06T07:31:26.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.122+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8778046020 con 0x7f87880b4650 2024-03-06T07:31:26.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.123+0000 7f877f400640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f876406b4d0 0x7f876406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.123+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87780acb40 con 0x7f87880b4650 2024-03-06T07:31:26.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.123+0000 7f877e000640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f876406b4d0 0x7f876406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.124+0000 7f877e000640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f876406b4d0 0x7f876406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f878814da10 tx=0x7f8770006c90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.128+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f877804d030 con 0x7f87880b4650 2024-03-06T07:31:26.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.366+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f878815aba0 con 0x7f87880b4650 2024-03-06T07:31:26.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.367+0000 7f877f400640 1 -- 172.21.15.32:0/2877746250 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4965 (secure 0 0 0) 0x7f877807bad0 con 0x7f87880b4650 2024-03-06T07:31:26.369 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:26.369 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:59.036395+0000","modified":"2024-03-06T07:31:02.225064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/1942129463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1942129463},{"type":"v1","addr":"172.21.15.32:6836","nonce":1942129463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":3}]} 2024-03-06T07:31:26.369 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-03-06T07:31:26.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f876406b4d0 msgr2=0x7f876406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f876406b4d0 0x7f876406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f878814da10 tx=0x7f8770006c90 comp rx=0 tx=0).stop 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 msgr2=0x7f87881436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 0x7f87881436a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8778036040 tx=0x7f8778004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 shutdown_connections 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f876406b4d0 0x7f876406d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f878815c4a0 0x7f8788152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8788151b30 0x7f8788152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 --2- 172.21.15.32:0/2877746250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87880b4650 0x7f87881436a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 >> 172.21.15.32:0/2877746250 conn(0x7f878806c0c0 msgr2=0x7f878815d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 shutdown_connections 2024-03-06T07:31:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.370+0000 7f8787e00640 1 -- 172.21.15.32:0/2877746250 wait complete. 2024-03-06T07:31:26.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.597+0000 7fe77ba00640 1 Processor -- start 2024-03-06T07:31:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.597+0000 7fe77ba00640 1 -- start start 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe77ba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740af1a0 0x7fe77415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe77ba00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7740af580 0x7fe7740b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe77ba00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7740cdc20 0x7fe7740b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe77ba00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77415cc40 con 0x7fe7740af580 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe77ba00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77415cf20 con 0x7fe7740cdc20 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe77ba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77415d200 con 0x7fe7740af1a0 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe76be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7740af580 0x7fe7740b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe76be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7740af580 0x7fe7740b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52462/0 (socket says 172.21.15.32:52462) 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe76be00640 1 -- 172.21.15.32:0/1918225597 learned_addr learned my addr 172.21.15.32:0/1918225597 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe779000640 1 --2- 172.21.15.32:0/1918225597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740af1a0 0x7fe77415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe76be00640 1 -- 172.21.15.32:0/1918225597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740af1a0 msgr2=0x7fe77415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe779a00640 1 --2- 172.21.15.32:0/1918225597 >> v2:172.21.15.39:3300/0 conn(0x7fe7740cdc20 0x7fe7740b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe76be00640 1 --2- 172.21.15.32:0/1918225597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740af1a0 0x7fe77415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.599+0000 7fe76be00640 1 -- 172.21.15.32:0/1918225597 >> v2:172.21.15.39:3300/0 conn(0x7fe7740cdc20 msgr2=0x7fe7740b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.600+0000 7fe76be00640 1 --2- 172.21.15.32:0/1918225597 >> v2:172.21.15.39:3300/0 conn(0x7fe7740cdc20 0x7fe7740b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.600+0000 7fe76be00640 1 -- 172.21.15.32:0/1918225597 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77415d4e0 con 0x7fe7740af580 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.600+0000 7fe779a00640 1 --2- 172.21.15.32:0/1918225597 >> v2:172.21.15.39:3300/0 conn(0x7fe7740cdc20 0x7fe7740b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.601+0000 7fe76be00640 1 --2- 172.21.15.32:0/1918225597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740b4c20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe7740cda30 tx=0x7fe75c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8f6e653af74e9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.601+0000 7fe773e00640 1 -- 172.21.15.32:0/1918225597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0317d0 con 0x7fe7740af580 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.601+0000 7fe773e00640 1 -- 172.21.15.32:0/1918225597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c037070 con 0x7fe7740af580 2024-03-06T07:31:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.601+0000 7fe773e00640 1 -- 172.21.15.32:0/1918225597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c031b90 con 0x7fe7740af580 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.601+0000 7fe77ba00640 1 -- 172.21.15.32:0/1918225597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 msgr2=0x7fe7740b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.601+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1918225597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740b4c20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe7740cda30 tx=0x7fe75c02b870 comp rx=0 tx=0).stop 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 -- 172.21.15.32:0/1918225597 shutdown_connections 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1918225597 >> v2:172.21.15.39:3300/0 conn(0x7fe7740cdc20 0x7fe7740b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1918225597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740b4c20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1918225597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740af1a0 0x7fe77415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 -- 172.21.15.32:0/1918225597 >> 172.21.15.32:0/1918225597 conn(0x7fe77406afb0 msgr2=0x7fe77406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 -- 172.21.15.32:0/1918225597 shutdown_connections 2024-03-06T07:31:26.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.602+0000 7fe77ba00640 1 -- 172.21.15.32:0/1918225597 wait complete. 2024-03-06T07:31:26.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.603+0000 7fe77ba00640 1 Processor -- start 2024-03-06T07:31:26.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.604+0000 7fe77ba00640 1 -- start start 2024-03-06T07:31:26.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.604+0000 7fe77ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7740af1a0 0x7fe774151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.604+0000 7fe77ba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740715b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.604+0000 7fe77ba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740cdc20 0x7fe774071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe77ba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740b3490 con 0x7fe7740af580 2024-03-06T07:31:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe77ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740b3310 con 0x7fe7740af1a0 2024-03-06T07:31:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe77ba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740b3610 con 0x7fe7740cdc20 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740715b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52470/0 (socket says 172.21.15.32:52470) 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 -- 172.21.15.32:0/1568335290 learned_addr learned my addr 172.21.15.32:0/1568335290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe779000640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7740af1a0 0x7fe774151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe779a00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740cdc20 0x7fe774071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 -- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740cdc20 msgr2=0x7fe774071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740cdc20 0x7fe774071af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 -- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7740af1a0 msgr2=0x7fe774151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7740af1a0 0x7fe774151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe76be00640 1 -- 172.21.15.32:0/1568335290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7741599a0 con 0x7fe7740af580 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.605+0000 7fe779000640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7740af1a0 0x7fe774151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.606+0000 7fe76be00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740715b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe75c000c00 tx=0x7fe75c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.606+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c032870 con 0x7fe7740af580 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.606+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c032a10 con 0x7fe7740af580 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.606+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c032e00 con 0x7fe7740af580 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.606+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe774159c80 con 0x7fe7740af580 2024-03-06T07:31:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.606+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7741535c0 con 0x7fe7740af580 2024-03-06T07:31:26.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.608+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe774069510 con 0x7fe7740af580 2024-03-06T07:31:26.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.608+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe75c0453f0 con 0x7fe7740af580 2024-03-06T07:31:26.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.609+0000 7fe771600640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe75006b4d0 0x7fe75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.609+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe75c0ac070 con 0x7fe7740af580 2024-03-06T07:31:26.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.610+0000 7fe779000640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe75006b4d0 0x7fe75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.610+0000 7fe779000640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe75006b4d0 0x7fe75006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe76400b800 tx=0x7fe764008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.614+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe75c040070 con 0x7fe7740af580 2024-03-06T07:31:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.809+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe77404f3a0 con 0x7fe7740af580 2024-03-06T07:31:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.811+0000 7fe771600640 1 -- 172.21.15.32:0/1568335290 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fe75c07af80 con 0x7fe7740af580 2024-03-06T07:31:26.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:26.813 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:00.589676+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:56.874383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:58.058939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe75006b4d0 msgr2=0x7fe75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe75006b4d0 0x7fe75006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe76400b800 tx=0x7fe764008040 comp rx=0 tx=0).stop 2024-03-06T07:31:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 msgr2=0x7fe7740715b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740715b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe75c000c00 tx=0x7fe75c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 shutdown_connections 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe75006b4d0 0x7fe75006d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7740cdc20 0x7fe774071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7740af580 0x7fe7740715b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 --2- 172.21.15.32:0/1568335290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7740af1a0 0x7fe774151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 >> 172.21.15.32:0/1568335290 conn(0x7fe77406afb0 msgr2=0x7fe7740acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 shutdown_connections 2024-03-06T07:31:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:26.814+0000 7fe77ba00640 1 -- 172.21.15.32:0/1568335290 wait complete. 2024-03-06T07:31:26.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.035+0000 7fe528e00640 1 Processor -- start 2024-03-06T07:31:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.035+0000 7fe528e00640 1 -- start start 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.036+0000 7fe528e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5240cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.036+0000 7fe528e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5240cc730 0x7fe5240d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.036+0000 7fe528e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5240b1370 con 0x7fe5240d05e0 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.036+0000 7fe528e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5240d0a50 con 0x7fe5240cc730 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.036+0000 7fe528e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240ccf80 con 0x7fe5240b0f90 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5240cc730 0x7fe5240d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5240cc730 0x7fe5240d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52482/0 (socket says 172.21.15.32:52482) 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 -- 172.21.15.32:0/3984488713 learned_addr learned my addr 172.21.15.32:0/3984488713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe523e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe5240d05e0 0x7fe52415d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44828/0 (socket says 172.21.15.32:44828) 2024-03-06T07:31:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe523400640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5240cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 -- 172.21.15.32:0/3984488713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 msgr2=0x7fe5240cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5240cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 -- 172.21.15.32:0/3984488713 >> v1:172.21.15.32:6789/0 conn(0x7fe5240d05e0 legacy=0x7fe52415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe522a00640 1 -- 172.21.15.32:0/3984488713 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240cd260 con 0x7fe5240cc730 2024-03-06T07:31:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.037+0000 7fe523400640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5240cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.038+0000 7fe522a00640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240cc730 0x7fe5240d00a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe510009870 tx=0x7fe51002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cf0631b406271c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe522000640 1 -- 172.21.15.32:0/3984488713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510035020 con 0x7fe5240cc730 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe522000640 1 -- 172.21.15.32:0/3984488713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510031640 con 0x7fe5240cc730 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe522000640 1 -- 172.21.15.32:0/3984488713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51002d040 con 0x7fe5240cc730 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 -- 172.21.15.32:0/3984488713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240cc730 msgr2=0x7fe5240d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240cc730 0x7fe5240d00a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe510009870 tx=0x7fe51002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 -- 172.21.15.32:0/3984488713 shutdown_connections 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240cc730 0x7fe5240d00a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 --2- 172.21.15.32:0/3984488713 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5240cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 -- 172.21.15.32:0/3984488713 >> 172.21.15.32:0/3984488713 conn(0x7fe5240d1ae0 msgr2=0x7fe5240d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 -- 172.21.15.32:0/3984488713 shutdown_connections 2024-03-06T07:31:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.039+0000 7fe528e00640 1 -- 172.21.15.32:0/3984488713 wait complete. 2024-03-06T07:31:27.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.041+0000 7fe528e00640 1 Processor -- start 2024-03-06T07:31:27.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.041+0000 7fe528e00640 1 -- start start 2024-03-06T07:31:27.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe528e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5241522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe528e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe5240cc730 0x7fe52415e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe528e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 0x7fe5241f7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe528e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240ccb70 con 0x7fe5240d05e0 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe528e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240cd820 con 0x7fe5240cc730 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe528e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240cccf0 con 0x7fe5240b0f90 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe523e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 0x7fe5241f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe523e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 0x7fe5241f7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52484/0 (socket says 172.21.15.32:52484) 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.042+0000 7fe523e00640 1 -- 172.21.15.32:0/3605953883 learned_addr learned my addr 172.21.15.32:0/3605953883 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523400640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5241522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe522a00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe5240cc730 0x7fe52415e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523e00640 1 -- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 msgr2=0x7fe5241522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5241522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523e00640 1 -- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe5240cc730 msgr2=0x7fe52415e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe5240cc730 0x7fe52415e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523e00640 1 -- 172.21.15.32:0/3605953883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5241f73d0 con 0x7fe5240d05e0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe523e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 0x7fe5241f7880 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00c740 tx=0x7fe51c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c017070 con 0x7fe5240d05e0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c00ce50 con 0x7fe5240d05e0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.043+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00a860 con 0x7fe5240d05e0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.044+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52415b1c0 con 0x7fe5240d05e0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.044+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52415d860 con 0x7fe5240d05e0 2024-03-06T07:31:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.045+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe524070c60 con 0x7fe5240d05e0 2024-03-06T07:31:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.046+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe51c01a070 con 0x7fe5240d05e0 2024-03-06T07:31:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.046+0000 7fe51b400640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.047+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe51c0898d0 con 0x7fe5240d05e0 2024-03-06T07:31:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.047+0000 7fe523400640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.048+0000 7fe523400640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50406b4d0 0x7fe50406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe524160e60 tx=0x7fe514008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.051+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe524070c60 con 0x7fe5240d05e0 2024-03-06T07:31:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.280+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe52404f330 con 0x7fe5240d05e0 2024-03-06T07:31:27.282 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:31:27.281+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:31:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.289+0000 7fe51b400640 1 -- 172.21.15.32:0/3605953883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe51c0586b0 con 0x7fe5240d05e0 2024-03-06T07:31:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50406b4d0 msgr2=0x7fe50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50406b4d0 0x7fe50406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe524160e60 tx=0x7fe514008040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 msgr2=0x7fe5241f7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 0x7fe5241f7880 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00c740 tx=0x7fe51c00cc00 comp rx=0 tx=0).stop 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 shutdown_connections 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5240d05e0 0x7fe5241f7880 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe5240cc730 0x7fe52415e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 --2- 172.21.15.32:0/3605953883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5240b0f90 0x7fe5241522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 >> 172.21.15.32:0/3605953883 conn(0x7fe5240d1ae0 msgr2=0x7fe52406c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 shutdown_connections 2024-03-06T07:31:27.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.292+0000 7fe528e00640 1 -- 172.21.15.32:0/3605953883 wait complete. 2024-03-06T07:31:27.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:27.406 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:31:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.501+0000 7fb7cce00640 1 Processor -- start 2024-03-06T07:31:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.502+0000 7fb7cce00640 1 -- start start 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.503+0000 7fb7cce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7c815f810 0x7fb7c80b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.503+0000 7fb7cce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c80b2080 0x7fb7c80b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.503+0000 7fb7cce00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c815fe40 con 0x7fb7c8145430 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.503+0000 7fb7cce00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c8160120 con 0x7fb7c815f810 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.503+0000 7fb7cce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8160400 con 0x7fb7c80b2080 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c6a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7c815f810 0x7fb7c80b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb7c8145430 0x7fb7c815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44854/0 (socket says 172.21.15.32:44854) 2024-03-06T07:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c7400640 1 -- 172.21.15.32:0/244632081 learned_addr learned my addr 172.21.15.32:0/244632081 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c7e00640 1 --2- 172.21.15.32:0/244632081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c80b2080 0x7fb7c80b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c6a00640 1 -- 172.21.15.32:0/244632081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c80b2080 msgr2=0x7fb7c80b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c6a00640 1 --2- 172.21.15.32:0/244632081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c80b2080 0x7fb7c80b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c6a00640 1 -- 172.21.15.32:0/244632081 >> v1:172.21.15.32:6789/0 conn(0x7fb7c8145430 legacy=0x7fb7c815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c6a00640 1 -- 172.21.15.32:0/244632081 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c81606e0 con 0x7fb7c815f810 2024-03-06T07:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.504+0000 7fb7c6000640 1 -- 172.21.15.32:0/244632081 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975939863 0 0) 0x7fb7c815fe40 con 0x7fb7c8145430 2024-03-06T07:31:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.505+0000 7fb7c6a00640 1 --2- 172.21.15.32:0/244632081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c80b1b40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097a0 tx=0x7fb7b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9a5d89c07fec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7c6000640 1 -- 172.21.15.32:0/244632081 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0035020 con 0x7fb7c815f810 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7c6000640 1 -- 172.21.15.32:0/244632081 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0031830 con 0x7fb7c815f810 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7c6000640 1 -- 172.21.15.32:0/244632081 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0031c00 con 0x7fb7c815f810 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 -- 172.21.15.32:0/244632081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 msgr2=0x7fb7c80b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 --2- 172.21.15.32:0/244632081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c80b1b40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097a0 tx=0x7fb7b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 -- 172.21.15.32:0/244632081 shutdown_connections 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 --2- 172.21.15.32:0/244632081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c80b1b40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 --2- 172.21.15.32:0/244632081 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c80b2080 0x7fb7c80b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 -- 172.21.15.32:0/244632081 >> 172.21.15.32:0/244632081 conn(0x7fb7c806b3a0 msgr2=0x7fb7c806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 -- 172.21.15.32:0/244632081 shutdown_connections 2024-03-06T07:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.506+0000 7fb7cce00640 1 -- 172.21.15.32:0/244632081 wait complete. 2024-03-06T07:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.508+0000 7fb7cce00640 1 Processor -- start 2024-03-06T07:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.508+0000 7fb7cce00640 1 -- start start 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7cce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c80b2080 0x7fb7c813fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7cce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c8145430 0x7fb7c80c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7cce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c81f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7cce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80cdf80 con 0x7fb7c815f810 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7cce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80cde00 con 0x7fb7c80b2080 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7cce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b3510 con 0x7fb7c8145430 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c81f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c81f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52532/0 (socket says 172.21.15.32:52532) 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.509+0000 7fb7c7e00640 1 -- 172.21.15.32:0/1393905099 learned_addr learned my addr 172.21.15.32:0/1393905099 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7400640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c80b2080 0x7fb7c813fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7e00640 1 -- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c8145430 msgr2=0x7fb7c80c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7e00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c8145430 0x7fb7c80c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7e00640 1 -- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c80b2080 msgr2=0x7fb7c813fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7e00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c80b2080 0x7fb7c813fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7e00640 1 -- 172.21.15.32:0/1393905099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80c6330 con 0x7fb7c815f810 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7c7e00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c81f3d10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0007e40 tx=0x7fb7c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0014620 con 0x7fb7c815f810 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c0019070 con 0x7fb7c815f810 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.510+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0014a20 con 0x7fb7c815f810 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.511+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c8140380 con 0x7fb7c815f810 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.511+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c8143440 con 0x7fb7c815f810 2024-03-06T07:31:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.512+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c804f330 con 0x7fb7c815f810 2024-03-06T07:31:27.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.513+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7c0014bc0 con 0x7fb7c815f810 2024-03-06T07:31:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.514+0000 7fb7bf400640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.514+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7c008abc0 con 0x7fb7c815f810 2024-03-06T07:31:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.514+0000 7fb7c7400640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.515+0000 7fb7c7400640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7a406b4d0 0x7fb7a406d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8000f60 tx=0x7fb7b8007b50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.518+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7c00599d0 con 0x7fb7c815f810 2024-03-06T07:31:27.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:27.750+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb7c806b9d0 con 0x7fb7c815f810 2024-03-06T07:31:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.293+0000 7fb7bf400640 1 -- 172.21.15.32:0/1393905099 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb7c00592d0 con 0x7fb7c815f810 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.296+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7a406b4d0 msgr2=0x7fb7a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.296+0000 7fb7cce00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7a406b4d0 0x7fb7a406d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8000f60 tx=0x7fb7b8007b50 comp rx=0 tx=0).stop 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.296+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 msgr2=0x7fb7c81f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.296+0000 7fb7cce00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c81f3d10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0007e40 tx=0x7fb7c000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 shutdown_connections 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c815f810 0x7fb7c81f3d10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7c8145430 0x7fb7c80c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 --2- 172.21.15.32:0/1393905099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c80b2080 0x7fb7c813fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 >> 172.21.15.32:0/1393905099 conn(0x7fb7c806b3a0 msgr2=0x7fb7c80ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 shutdown_connections 2024-03-06T07:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.297+0000 7fb7cce00640 1 -- 172.21.15.32:0/1393905099 wait complete. 2024-03-06T07:31:28.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:31:28.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.535+0000 7fb8ec400640 1 Processor -- start 2024-03-06T07:31:28.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.536+0000 7fb8ec400640 1 -- start start 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.537+0000 7fb8ec400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e40af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.537+0000 7fb8ec400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8e415faa0 0x7fb8e415ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.537+0000 7fb8ec400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e406efc0 con 0x7fb8e40b5870 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.537+0000 7fb8ec400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e406fbd0 con 0x7fb8e415faa0 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.537+0000 7fb8ec400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40af8f0 con 0x7fb8e4159a10 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8e415faa0 0x7fb8e415ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8e415faa0 0x7fb8e415ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45728/0 (socket says 172.21.15.32:45728) 2024-03-06T07:31:28.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 -- 172.21.15.32:0/243245297 learned_addr learned my addr 172.21.15.32:0/243245297 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 -- 172.21.15.32:0/243245297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 msgr2=0x7fb8e40af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 --2- 172.21.15.32:0/243245297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e40af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 -- 172.21.15.32:0/243245297 >> v1:172.21.15.39:6789/0 conn(0x7fb8e40b5870 legacy=0x7fb8e40b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.538+0000 7fb8e9000640 1 -- 172.21.15.32:0/243245297 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e406f2a0 con 0x7fb8e415faa0 2024-03-06T07:31:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.539+0000 7fb8e9000640 1 --2- 172.21.15.32:0/243245297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e415faa0 0x7fb8e415ff20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009870 tx=0x7fb8d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b0aea20672c868 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8e3e00640 1 -- 172.21.15.32:0/243245297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0035020 con 0x7fb8e415faa0 2024-03-06T07:31:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8e3e00640 1 -- 172.21.15.32:0/243245297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d0031640 con 0x7fb8e415faa0 2024-03-06T07:31:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8e3e00640 1 -- 172.21.15.32:0/243245297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d002d040 con 0x7fb8e415faa0 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 -- 172.21.15.32:0/243245297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e415faa0 msgr2=0x7fb8e415ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 --2- 172.21.15.32:0/243245297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e415faa0 0x7fb8e415ff20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009870 tx=0x7fb8d002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 -- 172.21.15.32:0/243245297 shutdown_connections 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 --2- 172.21.15.32:0/243245297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e415faa0 0x7fb8e415ff20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 --2- 172.21.15.32:0/243245297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e40af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 -- 172.21.15.32:0/243245297 >> 172.21.15.32:0/243245297 conn(0x7fb8e409aa10 msgr2=0x7fb8e409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 -- 172.21.15.32:0/243245297 shutdown_connections 2024-03-06T07:31:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.540+0000 7fb8ec400640 1 -- 172.21.15.32:0/243245297 wait complete. 2024-03-06T07:31:28.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.542+0000 7fb8ec400640 1 Processor -- start 2024-03-06T07:31:28.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.542+0000 7fb8ec400640 1 -- start start 2024-03-06T07:31:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.543+0000 7fb8ec400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 0x7fb8e409c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.543+0000 7fb8ec400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e4141790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.543+0000 7fb8ec400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8e415faa0 0x7fb8e414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.543+0000 7fb8ec400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4159df0 con 0x7fb8e40b5870 2024-03-06T07:31:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.543+0000 7fb8ec400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e406f860 con 0x7fb8e415faa0 2024-03-06T07:31:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.543+0000 7fb8ec400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4070060 con 0x7fb8e4159a10 2024-03-06T07:31:28.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 0x7fb8e409c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 0x7fb8e409c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45734/0 (socket says 172.21.15.32:45734) 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 -- 172.21.15.32:0/979377809 learned_addr learned my addr 172.21.15.32:0/979377809 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8ea400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8e415faa0 0x7fb8e414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9000640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e4141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 -- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 msgr2=0x7fb8e4141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e4141790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 -- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8e415faa0 msgr2=0x7fb8e414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8e415faa0 0x7fb8e414f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.544+0000 7fb8e9a00640 1 -- 172.21.15.32:0/979377809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e414fa20 con 0x7fb8e40b5870 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.545+0000 7fb8e9a00640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 0x7fb8e409c130 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400ba50 tx=0x7fb8d400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.545+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4019070 con 0x7fb8e40b5870 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.545+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d400cc00 con 0x7fb8e40b5870 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.545+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4014440 con 0x7fb8e40b5870 2024-03-06T07:31:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.545+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e414fd00 con 0x7fb8e40b5870 2024-03-06T07:31:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.545+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e4150b90 con 0x7fb8e40b5870 2024-03-06T07:31:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.546+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e4068010 con 0x7fb8e40b5870 2024-03-06T07:31:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.547+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8d401c070 con 0x7fb8e40b5870 2024-03-06T07:31:28.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.548+0000 7fb8e1600640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8c406b4d0 0x7fb8c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.548+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb8d408a0b0 con 0x7fb8e40b5870 2024-03-06T07:31:28.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.548+0000 7fb8e9000640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8c406b4d0 0x7fb8c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.549+0000 7fb8e9000640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8c406b4d0 0x7fb8c406d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0036000 tx=0x7fb8d003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.552+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8d4058e30 con 0x7fb8e40b5870 2024-03-06T07:31:28.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:28.757+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb8e40ce140 con 0x7fb8e40b5870 2024-03-06T07:31:29.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.606+0000 7fb8e1600640 1 -- 172.21.15.32:0/979377809 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb8d4058730 con 0x7fb8e40b5870 2024-03-06T07:31:29.608 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:31:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.610+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8c406b4d0 msgr2=0x7fb8c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.610+0000 7fb8ec400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8c406b4d0 0x7fb8c406d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0036000 tx=0x7fb8d003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.610+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 msgr2=0x7fb8e409c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.610+0000 7fb8ec400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 0x7fb8e409c130 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400ba50 tx=0x7fb8d400bf10 comp rx=0 tx=0).stop 2024-03-06T07:31:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 shutdown_connections 2024-03-06T07:31:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8c406b4d0 0x7fb8c406d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8e415faa0 0x7fb8e414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8e4159a10 0x7fb8e4141790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 --2- 172.21.15.32:0/979377809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e40b5870 0x7fb8e409c130 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 >> 172.21.15.32:0/979377809 conn(0x7fb8e409aa10 msgr2=0x7fb8e40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.611+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 shutdown_connections 2024-03-06T07:31:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.612+0000 7fb8ec400640 1 -- 172.21.15.32:0/979377809 wait complete. 2024-03-06T07:31:29.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:31:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.852+0000 7f4754600640 1 Processor -- start 2024-03-06T07:31:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.853+0000 7f4754600640 1 -- start start 2024-03-06T07:31:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.854+0000 7f4754600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0ae7b0 0x7f474c0aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.854+0000 7f4754600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f474c0b3640 0x7f474c15cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.854+0000 7f4754600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c1531f0 con 0x7f474c0aeb90 2024-03-06T07:31:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.854+0000 7f4754600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474c0d04d0 con 0x7f474c0b3640 2024-03-06T07:31:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.854+0000 7f4754600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c15d300 con 0x7f474c0ae7b0 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f474c0b3640 0x7f474c15cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f474c0b3640 0x7f474c15cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45740/0 (socket says 172.21.15.32:45740) 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 -- 172.21.15.32:0/3836064404 learned_addr learned my addr 172.21.15.32:0/3836064404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4752600640 1 --1- 172.21.15.32:0/3836064404 >> v1:172.21.15.32:6789/0 conn(0x7f474c0aeb90 0x7f474c06cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46632/0 (socket says 172.21.15.32:46632) 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751c00640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0ae7b0 0x7f474c0aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 -- 172.21.15.32:0/3836064404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0ae7b0 msgr2=0x7f474c0aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0ae7b0 0x7f474c0aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 -- 172.21.15.32:0/3836064404 >> v1:172.21.15.32:6789/0 conn(0x7f474c0aeb90 legacy=0x7f474c06cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751200640 1 -- 172.21.15.32:0/3836064404 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0b2b80 con 0x7f474c0b3640 2024-03-06T07:31:29.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.855+0000 7f4751c00640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0ae7b0 0x7f474c0aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.856+0000 7f4751200640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0b3640 0x7f474c15cdc0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f47380098a0 tx=0x7f473802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=badf8ce2e9b9af7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f474be00640 1 -- 172.21.15.32:0/3836064404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738034070 con 0x7f474c0b3640 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f474be00640 1 -- 172.21.15.32:0/3836064404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4738030840 con 0x7f474c0b3640 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f474be00640 1 -- 172.21.15.32:0/3836064404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738030c10 con 0x7f474c0b3640 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f4754600640 1 -- 172.21.15.32:0/3836064404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0b3640 msgr2=0x7f474c15cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f4754600640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0b3640 0x7f474c15cdc0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f47380098a0 tx=0x7f473802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f4754600640 1 -- 172.21.15.32:0/3836064404 shutdown_connections 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f4754600640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0b3640 0x7f474c15cdc0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f4754600640 1 --2- 172.21.15.32:0/3836064404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0ae7b0 0x7f474c0aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.857+0000 7f4754600640 1 -- 172.21.15.32:0/3836064404 >> 172.21.15.32:0/3836064404 conn(0x7f474c06afb0 msgr2=0x7f474c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.858+0000 7f4754600640 1 -- 172.21.15.32:0/3836064404 shutdown_connections 2024-03-06T07:31:29.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.858+0000 7f4754600640 1 -- 172.21.15.32:0/3836064404 wait complete. 2024-03-06T07:31:29.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.859+0000 7f4754600640 1 Processor -- start 2024-03-06T07:31:29.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.859+0000 7f4754600640 1 -- start start 2024-03-06T07:31:29.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.860+0000 7f4754600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 0x7f474c143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.860+0000 7f4754600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f474c0aeb90 0x7f474c09c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.860+0000 7f4754600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0b3640 0x7f474c151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.860+0000 7f4754600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0b2e40 con 0x7f474c0ae7b0 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.860+0000 7f4754600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c06eeb0 con 0x7f474c0aeb90 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.860+0000 7f4754600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0b3120 con 0x7f474c0b3640 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 0x7f474c143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 0x7f474c143640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45746/0 (socket says 172.21.15.32:45746) 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 -- 172.21.15.32:0/3460107860 learned_addr learned my addr 172.21.15.32:0/3460107860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751200640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f474c0aeb90 0x7f474c09c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4752600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0b3640 0x7f474c151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 -- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0b3640 msgr2=0x7f474c151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0b3640 0x7f474c151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 -- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f474c0aeb90 msgr2=0x7f474c09c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f474c0aeb90 0x7f474c09c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4751c00640 1 -- 172.21.15.32:0/3460107860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c09cd70 con 0x7f474c0ae7b0 2024-03-06T07:31:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.861+0000 7f4752600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0b3640 0x7f474c151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.862+0000 7f4751c00640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 0x7f474c143640 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f474000c7f0 tx=0x7f474000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.862+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740015070 con 0x7f474c0ae7b0 2024-03-06T07:31:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.862+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4740007a30 con 0x7f474c0ae7b0 2024-03-06T07:31:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.862+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740007e00 con 0x7f474c0ae7b0 2024-03-06T07:31:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.862+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c156bf0 con 0x7f474c0ae7b0 2024-03-06T07:31:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.862+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c1571b0 con 0x7f474c0ae7b0 2024-03-06T07:31:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.863+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c068010 con 0x7f474c0ae7b0 2024-03-06T07:31:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.864+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4740018070 con 0x7f474c0ae7b0 2024-03-06T07:31:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.865+0000 7f4749600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f472c06b4d0 0x7f472c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.865+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f474008a0b0 con 0x7f474c0ae7b0 2024-03-06T07:31:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.866+0000 7f4751200640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f472c06b4d0 0x7f472c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.866+0000 7f4751200640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f472c06b4d0 0x7f472c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4738036000 tx=0x7f473803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:29.870+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4740058ff0 con 0x7f474c0ae7b0 2024-03-06T07:31:30.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.070+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f474c0b4670 con 0x7f474c0ae7b0 2024-03-06T07:31:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.621+0000 7f4749600640 1 -- 172.21.15.32:0/3460107860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) v1 ==== 169+0+0 (secure 0 0 0) 0x7f47400588f0 con 0x7f474c0ae7b0 2024-03-06T07:31:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:31:30.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f472c06b4d0 msgr2=0x7f472c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f472c06b4d0 0x7f472c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4738036000 tx=0x7f473803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:30.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 msgr2=0x7f474c143640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 0x7f474c143640 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f474000c7f0 tx=0x7f474000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 shutdown_connections 2024-03-06T07:31:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f472c06b4d0 0x7f472c06d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f474c0b3640 0x7f474c151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f474c0aeb90 0x7f474c09c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 --2- 172.21.15.32:0/3460107860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f474c0ae7b0 0x7f474c143640 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.626+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 >> 172.21.15.32:0/3460107860 conn(0x7f474c06afb0 msgr2=0x7f474c0af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.628+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 shutdown_connections 2024-03-06T07:31:30.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.629+0000 7f4754600640 1 -- 172.21.15.32:0/3460107860 wait complete. 2024-03-06T07:31:30.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:31:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.855+0000 7fa716e00640 1 Processor -- start 2024-03-06T07:31:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.855+0000 7fa716e00640 1 -- start start 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa716e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71015bf20 0x7fa7100acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa716e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa710072520 0x7fa7100cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa716e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa710072df0 con 0x7fa710071e20 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa716e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7100cdab0 con 0x7fa71015bf20 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa716e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b4d70 con 0x7fa710072520 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa70fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71015bf20 0x7fa7100acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa70fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71015bf20 0x7fa7100acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45758/0 (socket says 172.21.15.32:45758) 2024-03-06T07:31:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.857+0000 7fa70fe00640 1 -- 172.21.15.32:0/1795889851 learned_addr learned my addr 172.21.15.32:0/1795889851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.858+0000 7fa707400640 1 --2- 172.21.15.32:0/1795889851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa710072520 0x7fa7100cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.858+0000 7fa70fe00640 1 -- 172.21.15.32:0/1795889851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa710072520 msgr2=0x7fa7100cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.858+0000 7fa70fe00640 1 --2- 172.21.15.32:0/1795889851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa710072520 0x7fa7100cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.858+0000 7fa70fe00640 1 -- 172.21.15.32:0/1795889851 >> v1:172.21.15.39:6789/0 conn(0x7fa710071e20 legacy=0x7fa7100ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.858+0000 7fa70fe00640 1 -- 172.21.15.32:0/1795889851 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100b5050 con 0x7fa71015bf20 2024-03-06T07:31:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.859+0000 7fa70fe00640 1 --2- 172.21.15.32:0/1795889851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015bf20 0x7fa7100acd40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa700009e50 tx=0x7fa70002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a814992f22f75d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.859+0000 7fa70f400640 1 -- 172.21.15.32:0/1795889851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70002ca70 con 0x7fa71015bf20 2024-03-06T07:31:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.859+0000 7fa70f400640 1 -- 172.21.15.32:0/1795889851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70002cc10 con 0x7fa71015bf20 2024-03-06T07:31:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.859+0000 7fa70f400640 1 -- 172.21.15.32:0/1795889851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000343f0 con 0x7fa71015bf20 2024-03-06T07:31:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 -- 172.21.15.32:0/1795889851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015bf20 msgr2=0x7fa7100acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 --2- 172.21.15.32:0/1795889851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015bf20 0x7fa7100acd40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa700009e50 tx=0x7fa70002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 -- 172.21.15.32:0/1795889851 shutdown_connections 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 --2- 172.21.15.32:0/1795889851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa710072520 0x7fa7100cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 --2- 172.21.15.32:0/1795889851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71015bf20 0x7fa7100acd40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 -- 172.21.15.32:0/1795889851 >> 172.21.15.32:0/1795889851 conn(0x7fa71006b3d0 msgr2=0x7fa71006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 -- 172.21.15.32:0/1795889851 shutdown_connections 2024-03-06T07:31:30.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.860+0000 7fa716e00640 1 -- 172.21.15.32:0/1795889851 wait complete. 2024-03-06T07:31:30.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.862+0000 7fa716e00640 1 Processor -- start 2024-03-06T07:31:30.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.862+0000 7fa716e00640 1 -- start start 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa716e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa710071e20 0x7fa71015d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa716e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 0x7fa7100b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa716e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa71015bf20 0x7fa7100b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa716e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b4b10 con 0x7fa710072520 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa716e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b4990 con 0x7fa710071e20 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa716e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100cd1e0 con 0x7fa71015bf20 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.863+0000 7fa707400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 0x7fa7100b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 0x7fa7100b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45764/0 (socket says 172.21.15.32:45764) 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 -- 172.21.15.32:0/338369087 learned_addr learned my addr 172.21.15.32:0/338369087 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa714e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa71015bf20 0x7fa7100b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa70fe00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa710071e20 0x7fa71015d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 -- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa71015bf20 msgr2=0x7fa7100b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa71015bf20 0x7fa7100b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 -- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa710071e20 msgr2=0x7fa71015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa710071e20 0x7fa71015d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 -- 172.21.15.32:0/338369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71015c6e0 con 0x7fa710072520 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.864+0000 7fa707400640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 0x7fa7100b1af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800d830 tx=0x7fa6f800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.865+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800b6c0 con 0x7fa710072520 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.865+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f8019070 con 0x7fa710072520 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.865+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800ba80 con 0x7fa710072520 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.865+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71015c9c0 con 0x7fa710072520 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.865+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71015cf80 con 0x7fa710072520 2024-03-06T07:31:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.866+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71004f330 con 0x7fa710072520 2024-03-06T07:31:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.867+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6f801e460 con 0x7fa710072520 2024-03-06T07:31:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.868+0000 7fa70cc00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.868+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa6f808a6e0 con 0x7fa710072520 2024-03-06T07:31:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.868+0000 7fa70fe00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.869+0000 7fa70fe00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa700035000 tx=0x7fa700030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:30.872+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6f8059730 con 0x7fa710072520 2024-03-06T07:31:31.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.073+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa71015f720 con 0x7fa710072520 2024-03-06T07:31:31.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.079+0000 7fa70cc00640 1 -- 172.21.15.32:0/338369087 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6f8059030 con 0x7fa710072520 2024-03-06T07:31:31.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 msgr2=0x7fa6ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa700035000 tx=0x7fa700030000 comp rx=0 tx=0).stop 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 msgr2=0x7fa7100b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 0x7fa7100b1af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800d830 tx=0x7fa6f800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 shutdown_connections 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6ec06b4d0 0x7fa6ec06d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa71015bf20 0x7fa7100b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa710072520 0x7fa7100b1af0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 --2- 172.21.15.32:0/338369087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa710071e20 0x7fa71015d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 >> 172.21.15.32:0/338369087 conn(0x7fa71006b3d0 msgr2=0x7fa71015a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 shutdown_connections 2024-03-06T07:31:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.082+0000 7fa716e00640 1 -- 172.21.15.32:0/338369087 wait complete. 2024-03-06T07:31:31.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:31:31.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.322+0000 7f22fc000640 1 Processor -- start 2024-03-06T07:31:31.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.322+0000 7f22fc000640 1 -- start start 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.323+0000 7f22fc000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.323+0000 7f22fc000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22f4071e20 0x7f22f40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.323+0000 7f22fc000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f4072df0 con 0x7f22f415bf20 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.323+0000 7f22fc000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f40cdab0 con 0x7f22f4071e20 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.323+0000 7f22fc000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40b4d70 con 0x7f22f4072520 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22f4071e20 0x7f22f40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22f4071e20 0x7f22f40ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45768/0 (socket says 172.21.15.32:45768) 2024-03-06T07:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 -- 172.21.15.32:0/374176562 learned_addr learned my addr 172.21.15.32:0/374176562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22f8c00640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 -- 172.21.15.32:0/374176562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 msgr2=0x7f22f40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 -- 172.21.15.32:0/374176562 >> v1:172.21.15.39:6789/0 conn(0x7f22f415bf20 legacy=0x7f22f40acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22fa000640 1 -- 172.21.15.32:0/374176562 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40b5050 con 0x7f22f4071e20 2024-03-06T07:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.324+0000 7f22f8c00640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.325+0000 7f22fa000640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f40ce4f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009e50 tx=0x7f22ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3749eba7cef5d293 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22f3e00640 1 -- 172.21.15.32:0/374176562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec02ca70 con 0x7f22f4071e20 2024-03-06T07:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22f3e00640 1 -- 172.21.15.32:0/374176562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec02cc10 con 0x7f22f4071e20 2024-03-06T07:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22f3e00640 1 -- 172.21.15.32:0/374176562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0343f0 con 0x7f22f4071e20 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22fc000640 1 -- 172.21.15.32:0/374176562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 msgr2=0x7f22f40ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22fc000640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f40ce4f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009e50 tx=0x7f22ec02bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22fc000640 1 -- 172.21.15.32:0/374176562 shutdown_connections 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22fc000640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f40ce4f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22fc000640 1 --2- 172.21.15.32:0/374176562 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.326+0000 7f22fc000640 1 -- 172.21.15.32:0/374176562 >> 172.21.15.32:0/374176562 conn(0x7f22f406b3d0 msgr2=0x7f22f406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.327+0000 7f22fc000640 1 -- 172.21.15.32:0/374176562 shutdown_connections 2024-03-06T07:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.327+0000 7f22fc000640 1 -- 172.21.15.32:0/374176562 wait complete. 2024-03-06T07:31:31.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.328+0000 7f22fc000640 1 Processor -- start 2024-03-06T07:31:31.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.328+0000 7f22fc000640 1 -- start start 2024-03-06T07:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.329+0000 7f22fc000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.329+0000 7f22fc000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.329+0000 7f22fc000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22f415bf20 0x7f22f406e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.329+0000 7f22fc000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40ad450 con 0x7f22f4071e20 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.329+0000 7f22fc000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40b4990 con 0x7f22f415bf20 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.329+0000 7f22fc000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f4157340 con 0x7f22f4072520 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f414b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45784/0 (socket says 172.21.15.32:45784) 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 -- 172.21.15.32:0/1857510311 learned_addr learned my addr 172.21.15.32:0/1857510311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 -- 172.21.15.32:0/1857510311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 msgr2=0x7f22f40b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 -- 172.21.15.32:0/1857510311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22f415bf20 msgr2=0x7f22f406e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22f415bf20 0x7f22f406e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 -- 172.21.15.32:0/1857510311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f4157620 con 0x7f22f4071e20 2024-03-06T07:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.330+0000 7f22f9600640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f414b5f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f22e800b3f0 tx=0x7f22e800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.331+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8015020 con 0x7f22f4071e20 2024-03-06T07:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.331+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e8011830 con 0x7f22f4071e20 2024-03-06T07:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.331+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8011c20 con 0x7f22f4071e20 2024-03-06T07:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.331+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f4157900 con 0x7f22f4071e20 2024-03-06T07:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.331+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f414bb30 con 0x7f22f4071e20 2024-03-06T07:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.332+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f404f330 con 0x7f22f4071e20 2024-03-06T07:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.333+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22e8018020 con 0x7f22f4071e20 2024-03-06T07:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.334+0000 7f22f1600640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f22d406b4d0 0x7f22d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.334+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f22e8089be0 con 0x7f22f4071e20 2024-03-06T07:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.334+0000 7f22f8c00640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f22d406b4d0 0x7f22d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.335+0000 7f22f8c00640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f22d406b4d0 0x7f22d406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006fd0 tx=0x7f22e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.338+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22e801f030 con 0x7f22f4071e20 2024-03-06T07:31:31.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.555+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f22f415f720 con 0x7f22f4071e20 2024-03-06T07:31:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.561+0000 7f22f1600640 1 -- 172.21.15.32:0/1857510311 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v15) v1 ==== 108+0+0 (secure 0 0 0) 0x7f22e80172c0 con 0x7f22f4071e20 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f22d406b4d0 msgr2=0x7f22d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f22d406b4d0 0x7f22d406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006fd0 tx=0x7f22e0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 msgr2=0x7f22f414b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f414b5f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f22e800b3f0 tx=0x7f22e800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 shutdown_connections 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f22d406b4d0 0x7f22d406d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22f415bf20 0x7f22f406e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22f4072520 0x7f22f40b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 --2- 172.21.15.32:0/1857510311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4071e20 0x7f22f414b5f0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.565+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 >> 172.21.15.32:0/1857510311 conn(0x7f22f406b3d0 msgr2=0x7f22f40ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.566+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 shutdown_connections 2024-03-06T07:31:31.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.566+0000 7f22fc000640 1 -- 172.21.15.32:0/1857510311 wait complete. 2024-03-06T07:31:31.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:31:31.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.775+0000 7f7af3000640 1 Processor -- start 2024-03-06T07:31:31.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.776+0000 7f7af3000640 1 -- start start 2024-03-06T07:31:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.777+0000 7f7af3000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7aec0af580 0x7f7aec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.777+0000 7f7af3000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0cdc20 0x7f7aec0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.777+0000 7f7af3000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec15cc40 con 0x7f7aec0af1a0 2024-03-06T07:31:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.777+0000 7f7af3000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aec15cf20 con 0x7f7aec0af580 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.777+0000 7f7af3000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec15d200 con 0x7f7aec0cdc20 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aeb400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7aec0af580 0x7f7aec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aebe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7aec0af1a0 0x7f7aec0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46648/0 (socket says 172.21.15.32:46648) 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aebe00640 1 -- 172.21.15.32:0/754898434 learned_addr learned my addr 172.21.15.32:0/754898434 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aeb400640 1 -- 172.21.15.32:0/754898434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0cdc20 msgr2=0x7f7aec0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aeb400640 1 --2- 172.21.15.32:0/754898434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0cdc20 0x7f7aec0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aeb400640 1 -- 172.21.15.32:0/754898434 >> v1:172.21.15.32:6789/0 conn(0x7f7aec0af1a0 legacy=0x7f7aec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.778+0000 7f7aeb400640 1 -- 172.21.15.32:0/754898434 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec15d4e0 con 0x7f7aec0af580 2024-03-06T07:31:31.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.779+0000 7f7aeb400640 1 --2- 172.21.15.32:0/754898434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0af580 0x7f7aec15c600 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40097a0 tx=0x7f7ad402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efabf9a24888820 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.780+0000 7f7aeaa00640 1 -- 172.21.15.32:0/754898434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4031780 con 0x7f7aec0af580 2024-03-06T07:31:31.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.780+0000 7f7aeaa00640 1 -- 172.21.15.32:0/754898434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad4037070 con 0x7f7aec0af580 2024-03-06T07:31:31.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.780+0000 7f7aeaa00640 1 -- 172.21.15.32:0/754898434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4031b40 con 0x7f7aec0af580 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.780+0000 7f7af3000640 1 -- 172.21.15.32:0/754898434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0af580 msgr2=0x7f7aec15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.780+0000 7f7af3000640 1 --2- 172.21.15.32:0/754898434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0af580 0x7f7aec15c600 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40097a0 tx=0x7f7ad402b820 comp rx=0 tx=0).stop 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.781+0000 7f7af3000640 1 -- 172.21.15.32:0/754898434 shutdown_connections 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.781+0000 7f7af3000640 1 --2- 172.21.15.32:0/754898434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0af580 0x7f7aec15c600 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.781+0000 7f7af3000640 1 --2- 172.21.15.32:0/754898434 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0cdc20 0x7f7aec0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.781+0000 7f7af3000640 1 -- 172.21.15.32:0/754898434 >> 172.21.15.32:0/754898434 conn(0x7f7aec06afb0 msgr2=0x7f7aec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.781+0000 7f7af3000640 1 -- 172.21.15.32:0/754898434 shutdown_connections 2024-03-06T07:31:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.781+0000 7f7af3000640 1 -- 172.21.15.32:0/754898434 wait complete. 2024-03-06T07:31:31.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.782+0000 7f7af3000640 1 Processor -- start 2024-03-06T07:31:31.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.783+0000 7f7af3000640 1 -- start start 2024-03-06T07:31:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af3000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0af1a0 0x7f7aec14c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af3000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aec0af580 0x7f7aec09b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af3000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 0x7f7aec15b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af3000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0b3720 con 0x7f7aec0cdc20 2024-03-06T07:31:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af3000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0b35a0 con 0x7f7aec0af580 2024-03-06T07:31:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af3000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec155e70 con 0x7f7aec0af1a0 2024-03-06T07:31:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 0x7f7aec15b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 0x7f7aec15b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45802/0 (socket says 172.21.15.32:45802) 2024-03-06T07:31:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7af1000640 1 -- 172.21.15.32:0/3536149934 learned_addr learned my addr 172.21.15.32:0/3536149934 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7aebe00640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0af1a0 0x7f7aec14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.784+0000 7f7aeb400640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aec0af580 0x7f7aec09b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af1000640 1 -- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0af1a0 msgr2=0x7f7aec14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af1000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0af1a0 0x7f7aec14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af1000640 1 -- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aec0af580 msgr2=0x7f7aec09b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af1000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aec0af580 0x7f7aec09b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af1000640 1 -- 172.21.15.32:0/3536149934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec14cd90 con 0x7f7aec0cdc20 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af1000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 0x7f7aec15b5a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400b970 tx=0x7f7ae400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4019070 con 0x7f7aec0cdc20 2024-03-06T07:31:31.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae400cb60 con 0x7f7aec0cdc20 2024-03-06T07:31:31.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec09bc70 con 0x7f7aec0cdc20 2024-03-06T07:31:31.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.785+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4014440 con 0x7f7aec0cdc20 2024-03-06T07:31:31.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.786+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec09c230 con 0x7f7aec0cdc20 2024-03-06T07:31:31.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.787+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aec04f3a0 con 0x7f7aec0cdc20 2024-03-06T07:31:31.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.788+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ae401c070 con 0x7f7aec0cdc20 2024-03-06T07:31:31.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.789+0000 7f7ae3e00640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ac806b4d0 0x7f7ac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.789+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7ae4089c20 con 0x7f7aec0cdc20 2024-03-06T07:31:31.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.789+0000 7f7aebe00640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ac806b4d0 0x7f7ac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.790+0000 7f7aebe00640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ac806b4d0 0x7f7ac806d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7aec15a680 tx=0x7f7adc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:31.793+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ae4058da0 con 0x7f7aec0cdc20 2024-03-06T07:31:32.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.066+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f7aec0b5110 con 0x7f7aec0cdc20 2024-03-06T07:31:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.635+0000 7f7ae3e00640 1 -- 172.21.15.32:0/3536149934 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v156) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7ae40586a0 con 0x7f7aec0cdc20 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.638+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ac806b4d0 msgr2=0x7f7ac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.638+0000 7f7af3000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ac806b4d0 0x7f7ac806d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7aec15a680 tx=0x7f7adc009040 comp rx=0 tx=0).stop 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.638+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 msgr2=0x7f7aec15b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.638+0000 7f7af3000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 0x7f7aec15b5a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400b970 tx=0x7f7ae400be30 comp rx=0 tx=0).stop 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 shutdown_connections 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7ac806b4d0 0x7f7ac806d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aec0cdc20 0x7f7aec15b5a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aec0af580 0x7f7aec09b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 --2- 172.21.15.32:0/3536149934 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aec0af1a0 0x7f7aec14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 >> 172.21.15.32:0/3536149934 conn(0x7f7aec06afb0 msgr2=0x7f7aec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 shutdown_connections 2024-03-06T07:31:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.639+0000 7f7af3000640 1 -- 172.21.15.32:0/3536149934 wait complete. 2024-03-06T07:31:32.663 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:31:32.663 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:31:32.663 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:31:32.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:32.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.884+0000 7fea96400640 1 Processor -- start 2024-03-06T07:31:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.885+0000 7fea96400640 1 -- start start 2024-03-06T07:31:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.886+0000 7fea96400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea900af1a0 0x7fea9015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.886+0000 7fea96400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea900b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.886+0000 7fea96400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9015cc40 con 0x7fea900af580 2024-03-06T07:31:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.886+0000 7fea96400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9015cf20 con 0x7fea900af1a0 2024-03-06T07:31:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.886+0000 7fea96400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9015d200 con 0x7fea900cdc20 2024-03-06T07:31:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea900af1a0 0x7fea9015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea900af1a0 0x7fea9015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45810/0 (socket says 172.21.15.32:45810) 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 -- 172.21.15.32:0/3929734478 learned_addr learned my addr 172.21.15.32:0/3929734478 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 -- 172.21.15.32:0/3929734478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 msgr2=0x7fea900b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8fe00640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea900b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea900b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 -- 172.21.15.32:0/3929734478 >> v1:172.21.15.39:6789/0 conn(0x7fea900af580 legacy=0x7fea900b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8f400640 1 -- 172.21.15.32:0/3929734478 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9015d4e0 con 0x7fea900af1a0 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.887+0000 7fea8fe00640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea900b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.888+0000 7fea8f400640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea9015c600 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fea80009870 tx=0x7fea8002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec551a31dd41b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.889+0000 7fea8e000640 1 -- 172.21.15.32:0/3929734478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800317c0 con 0x7fea900af1a0 2024-03-06T07:31:32.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.889+0000 7fea8e000640 1 -- 172.21.15.32:0/3929734478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80037070 con 0x7fea900af1a0 2024-03-06T07:31:32.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.889+0000 7fea8e000640 1 -- 172.21.15.32:0/3929734478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80031b80 con 0x7fea900af1a0 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 -- 172.21.15.32:0/3929734478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 msgr2=0x7fea9015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea9015c600 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fea80009870 tx=0x7fea8002b860 comp rx=0 tx=0).stop 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 -- 172.21.15.32:0/3929734478 shutdown_connections 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea900b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 --2- 172.21.15.32:0/3929734478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea9015c600 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 -- 172.21.15.32:0/3929734478 >> 172.21.15.32:0/3929734478 conn(0x7fea9006afb0 msgr2=0x7fea9006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 -- 172.21.15.32:0/3929734478 shutdown_connections 2024-03-06T07:31:32.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.890+0000 7fea96400640 1 -- 172.21.15.32:0/3929734478 wait complete. 2024-03-06T07:31:32.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.891+0000 7fea96400640 1 Processor -- start 2024-03-06T07:31:32.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.892+0000 7fea96400640 1 -- start start 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.892+0000 7fea96400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea90148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea96400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea900af580 0x7fea9006e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea96400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea90151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea96400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea900b3720 con 0x7fea900af1a0 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea96400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea900b35a0 con 0x7fea900af580 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea96400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea90155e70 con 0x7fea900cdc20 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea90148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea90148a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45818/0 (socket says 172.21.15.32:45818) 2024-03-06T07:31:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 -- 172.21.15.32:0/264794940 learned_addr learned my addr 172.21.15.32:0/264794940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 -- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 msgr2=0x7fea90151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea90151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8ea00640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea900af580 0x7fea9006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 -- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea900af580 msgr2=0x7fea9006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea900af580 0x7fea9006e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8f400640 1 -- 172.21.15.32:0/264794940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea90152e20 con 0x7fea900af1a0 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.893+0000 7fea8ea00640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea900af580 0x7fea9006e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.894+0000 7fea8f400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea90148a10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fea80000c00 tx=0x7fea800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.894+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80032860 con 0x7fea900af1a0 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.894+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80032a20 con 0x7fea900af1a0 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.894+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80032e10 con 0x7fea900af1a0 2024-03-06T07:31:32.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.894+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9015bc90 con 0x7fea900af1a0 2024-03-06T07:31:32.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.894+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9015bf70 con 0x7fea900af1a0 2024-03-06T07:31:32.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.896+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea90069510 con 0x7fea900af1a0 2024-03-06T07:31:32.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.896+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea8004c060 con 0x7fea900af1a0 2024-03-06T07:31:32.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.897+0000 7fea87400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b500 0x7fea6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.897+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fea800ab920 con 0x7fea900af1a0 2024-03-06T07:31:32.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.898+0000 7fea8ea00640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b500 0x7fea6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.898+0000 7fea8ea00640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b500 0x7fea6c06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea901f7580 tx=0x7fea78009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:32.902+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea80040070 con 0x7fea900af1a0 2024-03-06T07:31:33.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.102+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea9004f3a0 con 0x7fea900af1a0 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.103+0000 7fea87400640 1 -- 172.21.15.32:0/264794940 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+12912 (secure 0 0 0) 0x7fea8007a970 con 0x7fea900af1a0 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":49,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:30.607665+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6836/1942129463":"2024-03-07T07:31:27.281439+0000","172.21.15.32:6834/1942129463":"2024-03-07T07:31:27.281439+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:33.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b500 msgr2=0x7fea6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b500 0x7fea6c06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea901f7580 tx=0x7fea78009040 comp rx=0 tx=0).stop 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 msgr2=0x7fea90148a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea90148a10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fea80000c00 tx=0x7fea800046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 shutdown_connections 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b500 0x7fea6c06d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea900cdc20 0x7fea90151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea900af580 0x7fea9006e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 --2- 172.21.15.32:0/264794940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea900af1a0 0x7fea90148a10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 >> 172.21.15.32:0/264794940 conn(0x7fea9006afb0 msgr2=0x7fea9006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 shutdown_connections 2024-03-06T07:31:33.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.106+0000 7fea96400640 1 -- 172.21.15.32:0/264794940 wait complete. 2024-03-06T07:31:33.130 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:33.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.356+0000 7fea3c600640 1 Processor -- start 2024-03-06T07:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.357+0000 7fea3c600640 1 -- start start 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.358+0000 7fea3c600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fea3406da40 0x7fea3406e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.358+0000 7fea3c600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3415e680 0x7fea3406c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.358+0000 7fea3c600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea340ccad0 0x7fea340ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.358+0000 7fea3c600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea34161c70 con 0x7fea340ccad0 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.358+0000 7fea3c600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3415eb80 con 0x7fea3406da40 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.358+0000 7fea3c600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3415ee60 con 0x7fea3415e680 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea3a600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea340ccad0 0x7fea340ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea3a600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea340ccad0 0x7fea340ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45832/0 (socket says 172.21.15.32:45832) 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea3a600640 1 -- 172.21.15.32:0/3422349879 learned_addr learned my addr 172.21.15.32:0/3422349879 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea39c00640 1 --2- 172.21.15.32:0/3422349879 >> v2:172.21.15.39:3300/0 conn(0x7fea3406da40 0x7fea3406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea39200640 1 --2- 172.21.15.32:0/3422349879 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3415e680 0x7fea3406c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea3a600640 1 -- 172.21.15.32:0/3422349879 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3415e680 msgr2=0x7fea3406c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea3a600640 1 --2- 172.21.15.32:0/3422349879 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3415e680 0x7fea3406c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.359+0000 7fea3a600640 1 -- 172.21.15.32:0/3422349879 >> v2:172.21.15.39:3300/0 conn(0x7fea3406da40 msgr2=0x7fea3406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.360+0000 7fea3a600640 1 --2- 172.21.15.32:0/3422349879 >> v2:172.21.15.39:3300/0 conn(0x7fea3406da40 0x7fea3406e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.360+0000 7fea3a600640 1 -- 172.21.15.32:0/3422349879 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3415f140 con 0x7fea340ccad0 2024-03-06T07:31:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.361+0000 7fea3a600640 1 --2- 172.21.15.32:0/3422349879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea340ccad0 0x7fea340ccf80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fea340cd7b0 tx=0x7fea2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e15fc3bd2a1373 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.361+0000 7fea33e00640 1 -- 172.21.15.32:0/3422349879 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c02ca70 con 0x7fea340ccad0 2024-03-06T07:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea33e00640 1 -- 172.21.15.32:0/3422349879 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c02cc10 con 0x7fea340ccad0 2024-03-06T07:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea33e00640 1 -- 172.21.15.32:0/3422349879 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0343f0 con 0x7fea340ccad0 2024-03-06T07:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 -- 172.21.15.32:0/3422349879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea340ccad0 msgr2=0x7fea340ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 --2- 172.21.15.32:0/3422349879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea340ccad0 0x7fea340ccf80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fea340cd7b0 tx=0x7fea2c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 -- 172.21.15.32:0/3422349879 shutdown_connections 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 --2- 172.21.15.32:0/3422349879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea340ccad0 0x7fea340ccf80 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 --2- 172.21.15.32:0/3422349879 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3415e680 0x7fea3406c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 --2- 172.21.15.32:0/3422349879 >> v2:172.21.15.39:3300/0 conn(0x7fea3406da40 0x7fea3406e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 -- 172.21.15.32:0/3422349879 >> 172.21.15.32:0/3422349879 conn(0x7fea3406b3d0 msgr2=0x7fea3406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 -- 172.21.15.32:0/3422349879 shutdown_connections 2024-03-06T07:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.362+0000 7fea3c600640 1 -- 172.21.15.32:0/3422349879 wait complete. 2024-03-06T07:31:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.364+0000 7fea3c600640 1 Processor -- start 2024-03-06T07:31:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.364+0000 7fea3c600640 1 -- start start 2024-03-06T07:31:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.365+0000 7fea3c600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea3406da40 0x7fea341f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.365+0000 7fea3c600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea340ccad0 0x7fea3414d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.365+0000 7fea3c600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 0x7fea340c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.365+0000 7fea3c600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3406ea00 con 0x7fea3415e680 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.365+0000 7fea3c600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3406e880 con 0x7fea3406da40 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.365+0000 7fea3c600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3406cec0 con 0x7fea340ccad0 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 0x7fea340c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 0x7fea340c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45838/0 (socket says 172.21.15.32:45838) 2024-03-06T07:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 -- 172.21.15.32:0/2748641026 learned_addr learned my addr 172.21.15.32:0/2748641026 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea39c00640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea3406da40 0x7fea341f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 -- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea340ccad0 msgr2=0x7fea3414d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea340ccad0 0x7fea3414d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 -- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea3406da40 msgr2=0x7fea341f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea3406da40 0x7fea341f7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 -- 172.21.15.32:0/2748641026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea340c9750 con 0x7fea3415e680 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea39c00640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea3406da40 0x7fea341f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.366+0000 7fea3a600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 0x7fea340c8fe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009f40 tx=0x7fea2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.367+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c043020 con 0x7fea3415e680 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.367+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c03e450 con 0x7fea3415e680 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.367+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c009510 con 0x7fea3415e680 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.367+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea340c9a30 con 0x7fea3415e680 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.367+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea340c9ff0 con 0x7fea3415e680 2024-03-06T07:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.368+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3404f330 con 0x7fea3415e680 2024-03-06T07:31:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.369+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea2c046020 con 0x7fea3415e680 2024-03-06T07:31:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.370+0000 7fea31600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.371+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fea2c0ac690 con 0x7fea3415e680 2024-03-06T07:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.371+0000 7fea39c00640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.371+0000 7fea39c00640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fea28009a40 tx=0x7fea28008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.375+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea2c04d030 con 0x7fea3415e680 2024-03-06T07:31:33.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.581+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea3415fa70 con 0x7fea3415e680 2024-03-06T07:31:33.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.582+0000 7fea31600640 1 -- 172.21.15.32:0/2748641026 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea2c07b770 con 0x7fea3415e680 2024-03-06T07:31:33.584 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:31:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 msgr2=0x7fea1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fea28009a40 tx=0x7fea28008040 comp rx=0 tx=0).stop 2024-03-06T07:31:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 msgr2=0x7fea340c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 0x7fea340c8fe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009f40 tx=0x7fea2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 shutdown_connections 2024-03-06T07:31:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea3415e680 0x7fea340c8fe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea340ccad0 0x7fea3414d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 --2- 172.21.15.32:0/2748641026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea3406da40 0x7fea341f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 >> 172.21.15.32:0/2748641026 conn(0x7fea3406b3d0 msgr2=0x7fea3415db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 shutdown_connections 2024-03-06T07:31:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.585+0000 7fea3c600640 1 -- 172.21.15.32:0/2748641026 wait complete. 2024-03-06T07:31:33.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:31:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.829+0000 7f9bf4e00640 1 Processor -- start 2024-03-06T07:31:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.829+0000 7f9bf4e00640 1 -- start start 2024-03-06T07:31:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bf4e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9bf0145300 0x7f9bf00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bf4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9bf00cd060 0x7f9bf015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bf4e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00714d0 con 0x7f9bf0070f10 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bf4e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00717b0 con 0x7f9bf0145300 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bf4e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf0071a90 con 0x7f9bf00cd060 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bef400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9bf0145300 0x7f9bf00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bef400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9bf0145300 0x7f9bf00b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45844/0 (socket says 172.21.15.32:45844) 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9bef400640 1 -- 172.21.15.32:0/3876522097 learned_addr learned my addr 172.21.15.32:0/3876522097 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.831+0000 7f9befe00640 1 --2- 172.21.15.32:0/3876522097 >> v2:172.21.15.39:3300/0 conn(0x7f9bf00cd060 0x7f9bf015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.832+0000 7f9bef400640 1 -- 172.21.15.32:0/3876522097 >> v2:172.21.15.39:3300/0 conn(0x7f9bf00cd060 msgr2=0x7f9bf015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.832+0000 7f9bef400640 1 --2- 172.21.15.32:0/3876522097 >> v2:172.21.15.39:3300/0 conn(0x7f9bf00cd060 0x7f9bf015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.832+0000 7f9bef400640 1 -- 172.21.15.32:0/3876522097 >> v1:172.21.15.39:6789/0 conn(0x7f9bf0070f10 legacy=0x7f9bf00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.832+0000 7f9bef400640 1 -- 172.21.15.32:0/3876522097 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf0071d70 con 0x7f9bf0145300 2024-03-06T07:31:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.833+0000 7f9bef400640 1 --2- 172.21.15.32:0/3876522097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf0145300 0x7f9bf00b49a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00cc500 tx=0x7f9be002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bc54f57beaebef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.833+0000 7f9beea00640 1 -- 172.21.15.32:0/3876522097 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be002e070 con 0x7f9bf0145300 2024-03-06T07:31:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.833+0000 7f9beea00640 1 -- 172.21.15.32:0/3876522097 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be0037070 con 0x7f9bf0145300 2024-03-06T07:31:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.833+0000 7f9beea00640 1 -- 172.21.15.32:0/3876522097 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0031a00 con 0x7f9bf0145300 2024-03-06T07:31:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 -- 172.21.15.32:0/3876522097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf0145300 msgr2=0x7f9bf00b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/3876522097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf0145300 0x7f9bf00b49a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00cc500 tx=0x7f9be002b880 comp rx=0 tx=0).stop 2024-03-06T07:31:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 -- 172.21.15.32:0/3876522097 shutdown_connections 2024-03-06T07:31:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/3876522097 >> v2:172.21.15.39:3300/0 conn(0x7f9bf00cd060 0x7f9bf015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/3876522097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf0145300 0x7f9bf00b49a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 -- 172.21.15.32:0/3876522097 >> 172.21.15.32:0/3876522097 conn(0x7f9bf006b3d0 msgr2=0x7f9bf006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 -- 172.21.15.32:0/3876522097 shutdown_connections 2024-03-06T07:31:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.834+0000 7f9bf4e00640 1 -- 172.21.15.32:0/3876522097 wait complete. 2024-03-06T07:31:33.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.836+0000 7f9bf4e00640 1 Processor -- start 2024-03-06T07:31:33.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.836+0000 7f9bf4e00640 1 -- start start 2024-03-06T07:31:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bf4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bf0070f10 0x7f9bf013c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bf4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 0x7f9bf013fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bf4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf0145300 0x7f9bf013d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bf4e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf015cab0 con 0x7f9bf00cd060 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bf4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf01456e0 con 0x7f9bf0145300 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bf4e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf015cc30 con 0x7f9bf0070f10 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 0x7f9bf013fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 0x7f9bf013fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45858/0 (socket says 172.21.15.32:45858) 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 -- 172.21.15.32:0/2313762375 learned_addr learned my addr 172.21.15.32:0/2313762375 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bef400640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bf0070f10 0x7f9bf013c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 -- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bf0070f10 msgr2=0x7f9bf013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bf0070f10 0x7f9bf013c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 -- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf0145300 msgr2=0x7f9bf013d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9befe00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf0145300 0x7f9bf013d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf0145300 0x7f9bf013d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9be7e00640 1 -- 172.21.15.32:0/2313762375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf006d470 con 0x7f9bf00cd060 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.837+0000 7f9bef400640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bf0070f10 0x7f9bf013c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.838+0000 7f9be7e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 0x7f9bf013fa70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800c460 tx=0x7f9bd800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.838+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8016020 con 0x7f9bf00cd060 2024-03-06T07:31:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.838+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd8012870 con 0x7f9bf00cd060 2024-03-06T07:31:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.838+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8012c60 con 0x7f9bf00cd060 2024-03-06T07:31:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.838+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf006d750 con 0x7f9bf00cd060 2024-03-06T07:31:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.838+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf006dd10 con 0x7f9bf00cd060 2024-03-06T07:31:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.839+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf004f330 con 0x7f9bf00cd060 2024-03-06T07:31:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.840+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bd8019020 con 0x7f9bf00cd060 2024-03-06T07:31:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.841+0000 7f9be7400640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.841+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9bd8089bc0 con 0x7f9bf00cd060 2024-03-06T07:31:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.841+0000 7f9bef400640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.843+0000 7f9bef400640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9be0036000 tx=0x7f9be0008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:33.849+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bd8020030 con 0x7f9bf00cd060 2024-03-06T07:31:34.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.127+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f9bf006e7b0 con 0x7f9bf00cd060 2024-03-06T07:31:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.641+0000 7f9be7400640 1 -- 172.21.15.32:0/2313762375 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v158) v1 ==== 147+0+0 (secure 0 0 0) 0x7f9bd8058d30 con 0x7f9bf00cd060 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.644+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bcc06b500 msgr2=0x7f9bcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.644+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9be0036000 tx=0x7f9be0008030 comp rx=0 tx=0).stop 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.644+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 msgr2=0x7f9bf013fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.644+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 0x7f9bf013fa70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800c460 tx=0x7f9bd800c920 comp rx=0 tx=0).stop 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 shutdown_connections 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf0145300 0x7f9bf013d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf00cd060 0x7f9bf013fa70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 --2- 172.21.15.32:0/2313762375 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bf0070f10 0x7f9bf013c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 >> 172.21.15.32:0/2313762375 conn(0x7f9bf006b3d0 msgr2=0x7f9bf00b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 shutdown_connections 2024-03-06T07:31:34.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.645+0000 7f9bf4e00640 1 -- 172.21.15.32:0/2313762375 wait complete. 2024-03-06T07:31:34.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:31:34.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.878+0000 7f5788200640 1 Processor -- start 2024-03-06T07:31:34.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.878+0000 7f5788200640 1 -- start start 2024-03-06T07:31:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5788200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5780071e20 0x7f57800ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5788200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5780072df0 con 0x7f5780072520 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5788200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57800cdab0 con 0x7f578015bf20 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5788200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57800b4d70 con 0x7f5780071e20 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5784e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5780072520 0x7f57800cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46654/0 (socket says 172.21.15.32:46654) 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5784e00640 1 -- 172.21.15.32:0/3547465925 learned_addr learned my addr 172.21.15.32:0/3547465925 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f5786200640 1 --2- 172.21.15.32:0/3547465925 >> v2:172.21.15.39:3300/0 conn(0x7f5780071e20 0x7f57800ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.880+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373085493 0 0) 0x7f5780072df0 con 0x7f5780072520 2024-03-06T07:31:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5768003520 con 0x7f5780072520 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2452075357 0 0) 0x7f5768003520 con 0x7f5780072520 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 >> v2:172.21.15.39:3300/0 conn(0x7f5780071e20 msgr2=0x7f57800ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 --2- 172.21.15.32:0/3547465925 >> v2:172.21.15.39:3300/0 conn(0x7f5780071e20 0x7f57800ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 >> v1:172.21.15.39:6789/0 conn(0x7f578015bf20 legacy=0x7f57800acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57800b5050 con 0x7f5780072520 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f576c001670 con 0x7f5780072520 2024-03-06T07:31:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 >> v1:172.21.15.32:6789/0 conn(0x7f5780072520 legacy=0x7f57800cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768003a00 0x7f5768003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5768004800 0x7f5768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57680051c0 0x7f576800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57800b4d70 con 0x7f5768004800 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576800c0c0 con 0x7f57680051c0 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.881+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576800c2c0 con 0x7f5768003a00 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5784e00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5768004800 0x7f5768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5785800640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768003a00 0x7f5768003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5786200640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57680051c0 0x7f576800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5784e00640 1 -- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768003a00 msgr2=0x7f5768003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5784e00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768003a00 0x7f5768003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5784e00640 1 -- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57680051c0 msgr2=0x7f576800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5784e00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57680051c0 0x7f576800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5784e00640 1 -- 172.21.15.32:0/3547465925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5768003520 con 0x7f5768004800 2024-03-06T07:31:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.882+0000 7f5786200640 1 -- 172.21.15.32:0/3547465925 reap_dead start 2024-03-06T07:31:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.883+0000 7f5784e00640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5768004800 0x7f5768004c80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c110 tx=0x7f5768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f335c919d3783b20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.883+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c007c40 con 0x7f5768004800 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.883+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c030070 con 0x7f5768004800 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.883+0000 7f577fe00640 1 -- 172.21.15.32:0/3547465925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c005d70 con 0x7f5768004800 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 -- 172.21.15.32:0/3547465925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5768004800 msgr2=0x7f5768004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5768004800 0x7f5768004c80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c110 tx=0x7f5768005830 comp rx=0 tx=0).stop 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 -- 172.21.15.32:0/3547465925 shutdown_connections 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 --2- 172.21.15.32:0/3547465925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5768004800 0x7f5768004c80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 -- 172.21.15.32:0/3547465925 >> 172.21.15.32:0/3547465925 conn(0x7f578006b3d0 msgr2=0x7f578006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 -- 172.21.15.32:0/3547465925 shutdown_connections 2024-03-06T07:31:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.884+0000 7f5788200640 1 -- 172.21.15.32:0/3547465925 wait complete. 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.886+0000 7f5788200640 1 Processor -- start 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.886+0000 7f5788200640 1 -- start start 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5788200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768004800 0x7f57800b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5788200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 0x7f57800b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5788200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57800c9da0 0x7f57800ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5788200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57800b4b10 con 0x7f57800b08a0 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5788200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57800b4990 con 0x7f57800c9da0 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5788200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57800acd40 con 0x7f5768004800 2024-03-06T07:31:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5784e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 0x7f57800b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5784e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 0x7f57800b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45874/0 (socket says 172.21.15.32:45874) 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.887+0000 7f5784e00640 1 -- 172.21.15.32:0/3317833335 learned_addr learned my addr 172.21.15.32:0/3317833335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5785800640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768004800 0x7f57800b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5786200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57800c9da0 0x7f57800ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5784e00640 1 -- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768004800 msgr2=0x7f57800b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5784e00640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768004800 0x7f57800b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5784e00640 1 -- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57800c9da0 msgr2=0x7f57800ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5784e00640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57800c9da0 0x7f57800ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5784e00640 1 -- 172.21.15.32:0/3317833335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578009c730 con 0x7f57800b08a0 2024-03-06T07:31:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.888+0000 7f5784e00640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 0x7f57800b0d00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f576c02c4a0 tx=0x7f576c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.889+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c046070 con 0x7f57800b08a0 2024-03-06T07:31:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.889+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c004bf0 con 0x7f57800b08a0 2024-03-06T07:31:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.889+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c007c40 con 0x7f57800b08a0 2024-03-06T07:31:34.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.889+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57800c8fe0 con 0x7f57800b08a0 2024-03-06T07:31:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.889+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57800c95a0 con 0x7f57800b08a0 2024-03-06T07:31:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.890+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578004f330 con 0x7f57800b08a0 2024-03-06T07:31:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.891+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f576c007070 con 0x7f57800b08a0 2024-03-06T07:31:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.892+0000 7f577d600640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f576006b4d0 0x7f576006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.892+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f576c0ab480 con 0x7f57800b08a0 2024-03-06T07:31:34.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.893+0000 7f5785800640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f576006b4d0 0x7f576006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.893+0000 7f5785800640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f576006b4d0 0x7f576006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5774000c00 tx=0x7f5774009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:34.897+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f576c079f20 con 0x7f57800b08a0 2024-03-06T07:31:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.099+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f578015f720 con 0x7f57800b08a0 2024-03-06T07:31:35.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.104+0000 7f577d600640 1 -- 172.21.15.32:0/3317833335 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f576c07dfc0 con 0x7f57800b08a0 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.106+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f576006b4d0 msgr2=0x7f576006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.106+0000 7f5788200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f576006b4d0 0x7f576006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5774000c00 tx=0x7f5774009040 comp rx=0 tx=0).stop 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.106+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 msgr2=0x7f57800b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.106+0000 7f5788200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 0x7f57800b0d00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f576c02c4a0 tx=0x7f576c02e070 comp rx=0 tx=0).stop 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 shutdown_connections 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f576006b4d0 0x7f576006d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57800c9da0 0x7f57800ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57800b08a0 0x7f57800b0d00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 --2- 172.21.15.32:0/3317833335 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5768004800 0x7f57800b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 >> 172.21.15.32:0/3317833335 conn(0x7f578006b3d0 msgr2=0x7f57800ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 shutdown_connections 2024-03-06T07:31:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.107+0000 7f5788200640 1 -- 172.21.15.32:0/3317833335 wait complete. 2024-03-06T07:31:35.127 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:35.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:35.240 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:35.364 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:35.364 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:35.364 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:35.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:35.426 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:31:35.426 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:35.487 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:35.488 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:35.488 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:35.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:35.548 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:31:35.548 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:35.606 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:35.606 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:31:35.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.832+0000 7f5f8a800640 1 Processor -- start 2024-03-06T07:31:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.832+0000 7f5f8a800640 1 -- start start 2024-03-06T07:31:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f8a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f84070f10 0x7f5f840cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f8a800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f840714d0 con 0x7f5f840cce90 2024-03-06T07:31:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f8a800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f840717b0 con 0x7f5f84145300 2024-03-06T07:31:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f8a800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f840cd300 con 0x7f5f84070f10 2024-03-06T07:31:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f89e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f840cce90 0x7f5f8415c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46666/0 (socket says 172.21.15.32:46666) 2024-03-06T07:31:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f89e00640 1 -- 172.21.15.32:0/166464180 learned_addr learned my addr 172.21.15.32:0/166464180 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.834+0000 7f5f88a00640 1 --2- 172.21.15.32:0/166464180 >> v2:172.21.15.39:3300/0 conn(0x7f5f84070f10 0x7f5f840cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.835+0000 7f5f83e00640 1 -- 172.21.15.32:0/166464180 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799143212 0 0) 0x7f5f840714d0 con 0x7f5f840cce90 2024-03-06T07:31:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.835+0000 7f5f83e00640 1 -- 172.21.15.32:0/166464180 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c003520 con 0x7f5f840cce90 2024-03-06T07:31:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.835+0000 7f5f83e00640 1 -- 172.21.15.32:0/166464180 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2198568410 0 0) 0x7f5f6c003520 con 0x7f5f840cce90 2024-03-06T07:31:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.835+0000 7f5f88a00640 1 -- 172.21.15.32:0/166464180 >> v1:172.21.15.39:6789/0 conn(0x7f5f84145300 legacy=0x7f5f840b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.835+0000 7f5f88a00640 1 -- 172.21.15.32:0/166464180 >> v1:172.21.15.32:6789/0 conn(0x7f5f840cce90 legacy=0x7f5f8415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.835+0000 7f5f88a00640 1 -- 172.21.15.32:0/166464180 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f84071a90 con 0x7f5f84070f10 2024-03-06T07:31:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.837+0000 7f5f88a00640 1 --2- 172.21.15.32:0/166464180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f84070f10 0x7f5f840cc950 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5f700097a0 tx=0x7f5f7002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed97cc355b7d4f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.837+0000 7f5f83e00640 1 -- 172.21.15.32:0/166464180 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70037070 con 0x7f5f84070f10 2024-03-06T07:31:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.837+0000 7f5f83e00640 1 -- 172.21.15.32:0/166464180 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f70030510 con 0x7f5f84070f10 2024-03-06T07:31:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.837+0000 7f5f83e00640 1 -- 172.21.15.32:0/166464180 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0019f0 con 0x7f5f84070f10 2024-03-06T07:31:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 -- 172.21.15.32:0/166464180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f84070f10 msgr2=0x7f5f840cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 --2- 172.21.15.32:0/166464180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f84070f10 0x7f5f840cc950 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5f700097a0 tx=0x7f5f7002b790 comp rx=0 tx=0).stop 2024-03-06T07:31:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 -- 172.21.15.32:0/166464180 shutdown_connections 2024-03-06T07:31:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 --2- 172.21.15.32:0/166464180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f84070f10 0x7f5f840cc950 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 -- 172.21.15.32:0/166464180 >> 172.21.15.32:0/166464180 conn(0x7f5f8406b3d0 msgr2=0x7f5f8406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 -- 172.21.15.32:0/166464180 shutdown_connections 2024-03-06T07:31:35.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.838+0000 7f5f8a800640 1 -- 172.21.15.32:0/166464180 wait complete. 2024-03-06T07:31:35.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.839+0000 7f5f8a800640 1 Processor -- start 2024-03-06T07:31:35.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.840+0000 7f5f8a800640 1 -- start start 2024-03-06T07:31:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f8a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 0x7f5f841f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f8a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f840cce90 0x7f5f840ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f8a800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f84145300 0x7f5f84138e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f8a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8415cc80 con 0x7f5f84070f10 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f8a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f841456e0 con 0x7f5f840cce90 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f8a800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8415ce00 con 0x7f5f84145300 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 0x7f5f841f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 0x7f5f841f74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45886/0 (socket says 172.21.15.32:45886) 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 -- 172.21.15.32:0/1207710675 learned_addr learned my addr 172.21.15.32:0/1207710675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f88a00640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f840cce90 0x7f5f840ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 -- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f84145300 msgr2=0x7f5f84138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89e00640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f84145300 0x7f5f84138e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f84145300 0x7f5f84138e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 -- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f840cce90 msgr2=0x7f5f840ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f840cce90 0x7f5f840ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.841+0000 7f5f89400640 1 -- 172.21.15.32:0/1207710675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f840aed50 con 0x7f5f84070f10 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f88a00640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f840cce90 0x7f5f840ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f89e00640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f84145300 0x7f5f84138e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f89400640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 0x7f5f841f74f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c3c0 tx=0x7f5f7800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780052d0 con 0x7f5f84070f10 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f78005490 con 0x7f5f84070f10 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78007260 con 0x7f5f84070f10 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f840af030 con 0x7f5f84070f10 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.842+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8413d560 con 0x7f5f84070f10 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.843+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8404f330 con 0x7f5f84070f10 2024-03-06T07:31:35.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.844+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f78007400 con 0x7f5f84070f10 2024-03-06T07:31:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.846+0000 7f5f81600640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f6406b4d0 0x7f5f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.846+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5f780893f0 con 0x7f5f84070f10 2024-03-06T07:31:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.846+0000 7f5f88a00640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f6406b4d0 0x7f5f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.847+0000 7f5f88a00640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f6406b4d0 0x7f5f6406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5f70034000 tx=0x7f5f7002f000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:35.851+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f78058560 con 0x7f5f84070f10 2024-03-06T07:31:36.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.093+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f840729a0 con 0x7f5f84070f10 2024-03-06T07:31:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.094+0000 7f5f81600640 1 -- 172.21.15.32:0/1207710675 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3684 (secure 0 0 0) 0x7f5f78057e60 con 0x7f5f84070f10 2024-03-06T07:31:36.096 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:36.096 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/611281299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":611281299},{"type":"v1","addr":"172.21.15.39:6833","nonce":611281299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4787,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/34362170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":34362170},{"type":"v1","addr":"172.21.15.39:6835","nonce":34362170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/682646486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":682646486},{"type":"v1","addr":"172.21.15.32:6837","nonce":682646486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/908107004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":908107004},{"type":"v1","addr":"172.21.15.32:6836","nonce":908107004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[]} 2024-03-06T07:31:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.096+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f6406b4d0 msgr2=0x7f5f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.096+0000 7f5f8a800640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f6406b4d0 0x7f5f6406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5f70034000 tx=0x7f5f7002f000 comp rx=0 tx=0).stop 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.096+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 msgr2=0x7f5f841f74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.096+0000 7f5f8a800640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 0x7f5f841f74f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c3c0 tx=0x7f5f7800c880 comp rx=0 tx=0).stop 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 shutdown_connections 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f6406b4d0 0x7f5f6406d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f84145300 0x7f5f84138e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f840cce90 0x7f5f840ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 --2- 172.21.15.32:0/1207710675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f84070f10 0x7f5f841f74f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 >> 172.21.15.32:0/1207710675 conn(0x7f5f8406b3d0 msgr2=0x7f5f840b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 shutdown_connections 2024-03-06T07:31:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.097+0000 7f5f8a800640 1 -- 172.21.15.32:0/1207710675 wait complete. 2024-03-06T07:31:36.120 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:31:36.120 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:31:36.120 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:31:36.120 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:36.121 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:31:36.121 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:31:36.121 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:31:36.121 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:36.121 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:31:36.121 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:31:36.122 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:36.122 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:31:36.122 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:31:36.122 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:36.142 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:31:36.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:31:36.143 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:31:36.144 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:31:36.149 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:31:36.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:31:36.152 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:31:36.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:31:36.196 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:31:36.196 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:36.198 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:31:36.199 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:36.203 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:31:36.203 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:36.205 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:31:36.205 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:36.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:31:36.358 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:31:36.360 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:31:36.364 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:31:36.369 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:31:36.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.397+0000 7fcd97a00640 1 Processor -- start 2024-03-06T07:31:36.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.398+0000 7fcd97a00640 1 -- start start 2024-03-06T07:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd97a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd900b5160 0x7fcd900b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd97a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900af420 con 0x7fcd9015bf30 2024-03-06T07:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd97a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd90070230 con 0x7fcd900b5ce0 2024-03-06T07:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd97a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd900b6130 con 0x7fcd900b5160 2024-03-06T07:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd8fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd900b5160 0x7fcd900b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd95000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcd9015bf30 0x7fcd9006f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46686/0 (socket says 172.21.15.32:46686) 2024-03-06T07:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd95000640 1 -- 172.21.15.32:0/899163894 learned_addr learned my addr 172.21.15.32:0/899163894 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.399+0000 7fcd8fe00640 1 -- 172.21.15.32:0/899163894 >> v1:172.21.15.39:6789/0 conn(0x7fcd900b5ce0 legacy=0x7fcd900af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.400+0000 7fcd8fe00640 1 -- 172.21.15.32:0/899163894 >> v1:172.21.15.32:6789/0 conn(0x7fcd9015bf30 legacy=0x7fcd9006f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.400+0000 7fcd8fe00640 1 -- 172.21.15.32:0/899163894 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90070510 con 0x7fcd900b5160 2024-03-06T07:31:36.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.400+0000 7fcd8f400640 1 -- 172.21.15.32:0/899163894 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924211099 0 0) 0x7fcd84001110 con 0x7fcd9015bf30 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.401+0000 7fcd8fe00640 1 --2- 172.21.15.32:0/899163894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5160 0x7fcd900b57a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0097a0 tx=0x7fcd7c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e3f842dbf11902 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.401+0000 7fcd8f400640 1 -- 172.21.15.32:0/899163894 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c034070 con 0x7fcd900b5160 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.401+0000 7fcd8f400640 1 -- 172.21.15.32:0/899163894 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c030610 con 0x7fcd900b5160 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.401+0000 7fcd8f400640 1 -- 172.21.15.32:0/899163894 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c0309e0 con 0x7fcd900b5160 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.401+0000 7fcd97a00640 1 -- 172.21.15.32:0/899163894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5160 msgr2=0x7fcd900b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.401+0000 7fcd97a00640 1 --2- 172.21.15.32:0/899163894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5160 0x7fcd900b57a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0097a0 tx=0x7fcd7c02b960 comp rx=0 tx=0).stop 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.402+0000 7fcd97a00640 1 -- 172.21.15.32:0/899163894 shutdown_connections 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.402+0000 7fcd97a00640 1 --2- 172.21.15.32:0/899163894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5160 0x7fcd900b57a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.402+0000 7fcd97a00640 1 -- 172.21.15.32:0/899163894 >> 172.21.15.32:0/899163894 conn(0x7fcd9009aa10 msgr2=0x7fcd9009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.402+0000 7fcd97a00640 1 -- 172.21.15.32:0/899163894 shutdown_connections 2024-03-06T07:31:36.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.402+0000 7fcd97a00640 1 -- 172.21.15.32:0/899163894 wait complete. 2024-03-06T07:31:36.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.403+0000 7fcd97a00640 1 Processor -- start 2024-03-06T07:31:36.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 -- start start 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd900b5160 0x7fcd90155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 0x7fcd9014e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd9015bf30 0x7fcd90157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9015c310 con 0x7fcd900b5ce0 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd90070ad0 con 0x7fcd9015bf30 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.404+0000 7fcd97a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd900afff0 con 0x7fcd900b5160 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 0x7fcd9014e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 0x7fcd9014e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45930/0 (socket says 172.21.15.32:45930) 2024-03-06T07:31:36.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 -- 172.21.15.32:0/3361230114 learned_addr learned my addr 172.21.15.32:0/3361230114 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:36.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd95a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd9015bf30 0x7fcd90157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd95000640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd900b5160 0x7fcd90155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 -- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd900b5160 msgr2=0x7fcd90155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd900b5160 0x7fcd90155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 -- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd9015bf30 msgr2=0x7fcd90157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd9015bf30 0x7fcd90157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd8fe00640 1 -- 172.21.15.32:0/3361230114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd901564e0 con 0x7fcd900b5ce0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd95a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd9015bf30 0x7fcd90157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.405+0000 7fcd95000640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd900b5160 0x7fcd90155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.406+0000 7fcd8fe00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 0x7fcd9014e910 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02bf70 tx=0x7fcd7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.406+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c034040 con 0x7fcd900b5ce0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.406+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c039a10 con 0x7fcd900b5ce0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.406+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c039de0 con 0x7fcd900b5ce0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.406+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd901534d0 con 0x7fcd900b5ce0 2024-03-06T07:31:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.406+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd90153a90 con 0x7fcd900b5ce0 2024-03-06T07:31:36.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.407+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd900679a0 con 0x7fcd900b5ce0 2024-03-06T07:31:36.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.408+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd7c037070 con 0x7fcd900b5ce0 2024-03-06T07:31:36.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.409+0000 7fcd8cc00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7006b4d0 0x7fcd7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.409+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcd7c0abfe0 con 0x7fcd900b5ce0 2024-03-06T07:31:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.409+0000 7fcd95000640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7006b4d0 0x7fcd7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.410+0000 7fcd95000640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7006b4d0 0x7fcd7006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcd84000f60 tx=0x7fcd84009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.414+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd900679a0 con 0x7fcd900b5ce0 2024-03-06T07:31:36.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.629+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcd900ce240 con 0x7fcd900b5ce0 2024-03-06T07:31:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.685+0000 7fcd8cc00640 1 -- 172.21.15.32:0/3361230114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v51) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcd7c07b190 con 0x7fcd900b5ce0 2024-03-06T07:31:36.688 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7006b4d0 msgr2=0x7fcd7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7006b4d0 0x7fcd7006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcd84000f60 tx=0x7fcd84009040 comp rx=0 tx=0).stop 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 msgr2=0x7fcd9014e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 0x7fcd9014e910 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02bf70 tx=0x7fcd7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 shutdown_connections 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7006b4d0 0x7fcd7006d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd9015bf30 0x7fcd90157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd900b5ce0 0x7fcd9014e910 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 --2- 172.21.15.32:0/3361230114 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd900b5160 0x7fcd90155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 >> 172.21.15.32:0/3361230114 conn(0x7fcd9009aa10 msgr2=0x7fcd9006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.690+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 shutdown_connections 2024-03-06T07:31:36.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.691+0000 7fcd97a00640 1 -- 172.21.15.32:0/3361230114 wait complete. 2024-03-06T07:31:36.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:31:36.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.925+0000 7fe899600640 1 Processor -- start 2024-03-06T07:31:36.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.925+0000 7fe899600640 1 -- start start 2024-03-06T07:31:36.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe899600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe8940af1a0 0x7fe89415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe899600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8940af580 0x7fe8940b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe899600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89415cc40 con 0x7fe8940cdc20 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe899600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89415cf20 con 0x7fe8940af580 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe899600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89415d200 con 0x7fe8940af1a0 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe892000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8940af580 0x7fe8940b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe892000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8940af580 0x7fe8940b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45940/0 (socket says 172.21.15.32:45940) 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.927+0000 7fe892000640 1 -- 172.21.15.32:0/3189446623 learned_addr learned my addr 172.21.15.32:0/3189446623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:36.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.928+0000 7fe892a00640 1 --2- 172.21.15.32:0/3189446623 >> v2:172.21.15.39:3300/0 conn(0x7fe8940af1a0 0x7fe89415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.928+0000 7fe892000640 1 -- 172.21.15.32:0/3189446623 >> v2:172.21.15.39:3300/0 conn(0x7fe8940af1a0 msgr2=0x7fe89415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.928+0000 7fe892000640 1 --2- 172.21.15.32:0/3189446623 >> v2:172.21.15.39:3300/0 conn(0x7fe8940af1a0 0x7fe89415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.928+0000 7fe892000640 1 -- 172.21.15.32:0/3189446623 >> v1:172.21.15.39:6789/0 conn(0x7fe8940cdc20 legacy=0x7fe8940b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.928+0000 7fe892000640 1 -- 172.21.15.32:0/3189446623 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89415d4e0 con 0x7fe8940af580 2024-03-06T07:31:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.928+0000 7fe892a00640 1 --2- 172.21.15.32:0/3189446623 >> v2:172.21.15.39:3300/0 conn(0x7fe8940af1a0 0x7fe89415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.929+0000 7fe892000640 1 --2- 172.21.15.32:0/3189446623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af580 0x7fe8940b4c20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe894074960 tx=0x7fe88002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86b84eb341b66e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.929+0000 7fe891600640 1 -- 172.21.15.32:0/3189446623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88002e070 con 0x7fe8940af580 2024-03-06T07:31:36.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe891600640 1 -- 172.21.15.32:0/3189446623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880037070 con 0x7fe8940af580 2024-03-06T07:31:36.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe891600640 1 -- 172.21.15.32:0/3189446623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880031a00 con 0x7fe8940af580 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe899600640 1 -- 172.21.15.32:0/3189446623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af580 msgr2=0x7fe8940b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe899600640 1 --2- 172.21.15.32:0/3189446623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af580 0x7fe8940b4c20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe894074960 tx=0x7fe88002b880 comp rx=0 tx=0).stop 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe899600640 1 -- 172.21.15.32:0/3189446623 shutdown_connections 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe899600640 1 --2- 172.21.15.32:0/3189446623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af580 0x7fe8940b4c20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe899600640 1 --2- 172.21.15.32:0/3189446623 >> v2:172.21.15.39:3300/0 conn(0x7fe8940af1a0 0x7fe89415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.930+0000 7fe899600640 1 -- 172.21.15.32:0/3189446623 >> 172.21.15.32:0/3189446623 conn(0x7fe89406afb0 msgr2=0x7fe89406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.931+0000 7fe899600640 1 -- 172.21.15.32:0/3189446623 shutdown_connections 2024-03-06T07:31:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.931+0000 7fe899600640 1 -- 172.21.15.32:0/3189446623 wait complete. 2024-03-06T07:31:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.933+0000 7fe899600640 1 Processor -- start 2024-03-06T07:31:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.933+0000 7fe899600640 1 -- start start 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe899600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 0x7fe894147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe899600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8940af580 0x7fe89409c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe899600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8940cdc20 0x7fe89415d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe899600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940b3660 con 0x7fe8940af1a0 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe899600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940b34e0 con 0x7fe8940af580 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe899600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940b37e0 con 0x7fe8940cdc20 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.934+0000 7fe892a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 0x7fe894147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 0x7fe894147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45942/0 (socket says 172.21.15.32:45942) 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 -- 172.21.15.32:0/1946776439 learned_addr learned my addr 172.21.15.32:0/1946776439 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892000640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8940af580 0x7fe89409c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 -- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8940cdc20 msgr2=0x7fe89415d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8940cdc20 0x7fe89415d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 -- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8940af580 msgr2=0x7fe89409c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8940af580 0x7fe89409c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 -- 172.21.15.32:0/1946776439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89409cf70 con 0x7fe8940af1a0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892000640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8940af580 0x7fe89409c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.935+0000 7fe892a00640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 0x7fe894147650 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c7f0 tx=0x7fe88400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.936+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884015070 con 0x7fe8940af1a0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.936+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884007a30 con 0x7fe8940af1a0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.936+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884007e00 con 0x7fe8940af1a0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.936+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89409d250 con 0x7fe8940af1a0 2024-03-06T07:31:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.936+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89409d810 con 0x7fe8940af1a0 2024-03-06T07:31:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.937+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe894069510 con 0x7fe8940af1a0 2024-03-06T07:31:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.938+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe884018070 con 0x7fe8940af1a0 2024-03-06T07:31:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.939+0000 7fe88aa00640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.939+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe884089f90 con 0x7fe8940af1a0 2024-03-06T07:31:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.939+0000 7fe892000640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.940+0000 7fe892000640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe87406b4d0 0x7fe87406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe880036000 tx=0x7fe880008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:36.943+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe884059030 con 0x7fe8940af1a0 2024-03-06T07:31:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.214+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe89404f3a0 con 0x7fe8940af1a0 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.215+0000 7fe88aa00640 1 -- 172.21.15.32:0/1946776439 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe884058930 con 0x7fe8940af1a0 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:31:37.222 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:31:37.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:31:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe87406b4d0 msgr2=0x7fe87406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe87406b4d0 0x7fe87406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe880036000 tx=0x7fe880008030 comp rx=0 tx=0).stop 2024-03-06T07:31:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 msgr2=0x7fe894147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 0x7fe894147650 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c7f0 tx=0x7fe88400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 shutdown_connections 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8940cdc20 0x7fe89415d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8940af580 0x7fe89409c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 --2- 172.21.15.32:0/1946776439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940af1a0 0x7fe894147650 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.217+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 >> 172.21.15.32:0/1946776439 conn(0x7fe89406afb0 msgr2=0x7fe8940ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.218+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 shutdown_connections 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.218+0000 7fe899600640 1 -- 172.21.15.32:0/1946776439 wait complete. 2024-03-06T07:31:37.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.452+0000 7f5f2fe00640 1 Processor -- start 2024-03-06T07:31:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.453+0000 7f5f2fe00640 1 -- start start 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f2fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f3015ded0 0x7f5f3006c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f2fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 0x7f5f300af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f2fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3015e2b0 con 0x7f5f300b61b0 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f2fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f300b6970 con 0x7f5f3015ded0 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f2fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300b6c50 con 0x7f5f3006d1c0 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f27e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f300b61b0 0x7f5f3006cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46702/0 (socket says 172.21.15.32:46702) 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.454+0000 7f5f27e00640 1 -- 172.21.15.32:0/1479415185 learned_addr learned my addr 172.21.15.32:0/1479415185 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/1479415185 >> v2:172.21.15.39:3300/0 conn(0x7f5f3015ded0 0x7f5f3006c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655910914 0 0) 0x7f5f3015e2b0 con 0x7f5f300b61b0 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2f400640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 0x7f5f300af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f14003520 con 0x7f5f300b61b0 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820585808 0 0) 0x7f5f14003520 con 0x7f5f300b61b0 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 msgr2=0x7f5f300af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 0x7f5f300af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 >> v2:172.21.15.39:3300/0 conn(0x7f5f3015ded0 msgr2=0x7f5f3006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 --2- 172.21.15.32:0/1479415185 >> v2:172.21.15.39:3300/0 conn(0x7f5f3015ded0 0x7f5f3006c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300aeb30 con 0x7f5f300b61b0 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f5f18001670 con 0x7f5f300b61b0 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/1479415185 >> v2:172.21.15.39:3300/0 conn(0x7f5f3015ded0 0x7f5f3006c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 >> v1:172.21.15.32:6789/0 conn(0x7f5f300b61b0 legacy=0x7f5f3006cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140039b0 0x7f5f14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 0x7f5f14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f14004d40 0x7f5f140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300b6970 con 0x7f5f140039b0 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300b6c50 con 0x7f5f14004d40 2024-03-06T07:31:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1400baa0 con 0x7f5f3006d1c0 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.455+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140039b0 0x7f5f14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f27e00640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 0x7f5f14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f2f400640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f14004d40 0x7f5f140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f2ea00640 1 -- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 msgr2=0x7f5f14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f3006d1c0 0x7f5f14004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f2ea00640 1 -- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f14004d40 msgr2=0x7f5f140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f14004d40 0x7f5f140051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.456+0000 7f5f2ea00640 1 -- 172.21.15.32:0/1479415185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f14003520 con 0x7f5f140039b0 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.457+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140039b0 0x7f5f14003dd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5f200098a0 tx=0x7f5f14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b32fa1c72b93f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.457+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2000f040 con 0x7f5f140039b0 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.457+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f20004230 con 0x7f5f140039b0 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.457+0000 7f5f2e000640 1 -- 172.21.15.32:0/1479415185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20003610 con 0x7f5f140039b0 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2fe00640 1 -- 172.21.15.32:0/1479415185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140039b0 msgr2=0x7f5f14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/1479415185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f140039b0 0x7f5f14003dd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5f200098a0 tx=0x7f5f14005830 comp rx=0 tx=0).stop 2024-03-06T07:31:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2f400640 1 -- 172.21.15.32:0/1479415185 reap_dead start 2024-03-06T07:31:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2fe00640 1 -- 172.21.15.32:0/1479415185 shutdown_connections 2024-03-06T07:31:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2fe00640 1 -- 172.21.15.32:0/1479415185 >> 172.21.15.32:0/1479415185 conn(0x7f5f300731f0 msgr2=0x7f5f3009aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2fe00640 1 -- 172.21.15.32:0/1479415185 shutdown_connections 2024-03-06T07:31:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.458+0000 7f5f2fe00640 1 -- 172.21.15.32:0/1479415185 wait complete. 2024-03-06T07:31:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.460+0000 7f5f2fe00640 1 Processor -- start 2024-03-06T07:31:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.460+0000 7f5f2fe00640 1 -- start start 2024-03-06T07:31:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f2fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f140039b0 0x7f5f300b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f2fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 0x7f5f30155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f2fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f30158aa0 0x7f5f30158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f2fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300af330 con 0x7f5f301554e0 2024-03-06T07:31:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f2fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300b6610 con 0x7f5f140039b0 2024-03-06T07:31:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f2fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300af4b0 con 0x7f5f30158aa0 2024-03-06T07:31:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f27e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 0x7f5f30155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f27e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 0x7f5f30155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45964/0 (socket says 172.21.15.32:45964) 2024-03-06T07:31:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.461+0000 7f5f27e00640 1 -- 172.21.15.32:0/651027098 learned_addr learned my addr 172.21.15.32:0/651027098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f140039b0 0x7f5f300b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f2f400640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f30158aa0 0x7f5f30158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f27e00640 1 -- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f30158aa0 msgr2=0x7f5f30158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f27e00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f30158aa0 0x7f5f30158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f27e00640 1 -- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f140039b0 msgr2=0x7f5f300b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f27e00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f140039b0 0x7f5f300b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f27e00640 1 -- 172.21.15.32:0/651027098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30155ea0 con 0x7f5f301554e0 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f2f400640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f30158aa0 0x7f5f30158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f27e00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 0x7f5f30155960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800d7f0 tx=0x7f5f1800dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18019070 con 0x7f5f301554e0 2024-03-06T07:31:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f180074e0 con 0x7f5f301554e0 2024-03-06T07:31:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18004a20 con 0x7f5f301554e0 2024-03-06T07:31:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.462+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f300b1ed0 con 0x7f5f301554e0 2024-03-06T07:31:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.463+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f300b13f0 con 0x7f5f301554e0 2024-03-06T07:31:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.464+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3004f330 con 0x7f5f301554e0 2024-03-06T07:31:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.465+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f1801c070 con 0x7f5f301554e0 2024-03-06T07:31:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.466+0000 7f5f26a00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f0c06b550 0x7f5f0c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.466+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f180898c0 con 0x7f5f301554e0 2024-03-06T07:31:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.466+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f0c06b550 0x7f5f0c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.467+0000 7f5f2ea00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f0c06b550 0x7f5f0c06da10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f20035000 tx=0x7f5f20039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.470+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f18058420 con 0x7f5f301554e0 2024-03-06T07:31:37.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.662+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f3006ee80 con 0x7f5f301554e0 2024-03-06T07:31:37.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.663+0000 7f5f26a00640 1 -- 172.21.15.32:0/651027098 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12783 (secure 0 0 0) 0x7f5f1805c4c0 con 0x7f5f301554e0 2024-03-06T07:31:37.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:37.666 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":53,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:36.701187+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.665+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f0c06b550 msgr2=0x7f5f0c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.665+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f0c06b550 0x7f5f0c06da10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f20035000 tx=0x7f5f20039040 comp rx=0 tx=0).stop 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.665+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 msgr2=0x7f5f30155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.665+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 0x7f5f30155960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800d7f0 tx=0x7f5f1800dcb0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 shutdown_connections 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f0c06b550 0x7f5f0c06da10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f30158aa0 0x7f5f30158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f301554e0 0x7f5f30155960 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 --2- 172.21.15.32:0/651027098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f140039b0 0x7f5f300b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 >> 172.21.15.32:0/651027098 conn(0x7f5f300731f0 msgr2=0x7f5f300af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 shutdown_connections 2024-03-06T07:31:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.666+0000 7f5f2fe00640 1 -- 172.21.15.32:0/651027098 wait complete. 2024-03-06T07:31:37.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:37.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:37.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.886+0000 7fe11a600640 1 Processor -- start 2024-03-06T07:31:37.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.887+0000 7fe11a600640 1 -- start start 2024-03-06T07:31:37.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.888+0000 7fe11a600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe11415f810 0x7fe1140b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.888+0000 7fe11a600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1140b2080 0x7fe1140b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.888+0000 7fe11a600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11415fe40 con 0x7fe114145430 2024-03-06T07:31:37.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.888+0000 7fe11a600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1140b24d0 con 0x7fe1140b2080 2024-03-06T07:31:37.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.888+0000 7fe11a600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe114160120 con 0x7fe11415f810 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1140b2080 0x7fe1140b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1140b2080 0x7fe1140b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58972/0 (socket says 172.21.15.32:58972) 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 -- 172.21.15.32:0/4024614300 learned_addr learned my addr 172.21.15.32:0/4024614300 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe10be00640 1 --2- 172.21.15.32:0/4024614300 >> v2:172.21.15.39:3300/0 conn(0x7fe11415f810 0x7fe1140b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 -- 172.21.15.32:0/4024614300 >> v2:172.21.15.39:3300/0 conn(0x7fe11415f810 msgr2=0x7fe1140b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 --2- 172.21.15.32:0/4024614300 >> v2:172.21.15.39:3300/0 conn(0x7fe11415f810 0x7fe1140b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 -- 172.21.15.32:0/4024614300 >> v1:172.21.15.39:6789/0 conn(0x7fe114145430 legacy=0x7fe11415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.889+0000 7fe119c00640 1 -- 172.21.15.32:0/4024614300 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114160400 con 0x7fe1140b2080 2024-03-06T07:31:37.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.890+0000 7fe119c00640 1 --2- 172.21.15.32:0/4024614300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1140b2fd0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe11406d8c0 tx=0x7fe10c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8736525661d82518 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.891+0000 7fe113e00640 1 -- 172.21.15.32:0/4024614300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c02ca80 con 0x7fe1140b2080 2024-03-06T07:31:37.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.891+0000 7fe113e00640 1 -- 172.21.15.32:0/4024614300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c02cc20 con 0x7fe1140b2080 2024-03-06T07:31:37.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.891+0000 7fe113e00640 1 -- 172.21.15.32:0/4024614300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0343f0 con 0x7fe1140b2080 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.891+0000 7fe11a600640 1 -- 172.21.15.32:0/4024614300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 msgr2=0x7fe1140b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.891+0000 7fe11a600640 1 --2- 172.21.15.32:0/4024614300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1140b2fd0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe11406d8c0 tx=0x7fe10c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.892+0000 7fe11a600640 1 -- 172.21.15.32:0/4024614300 shutdown_connections 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.892+0000 7fe11a600640 1 --2- 172.21.15.32:0/4024614300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1140b2fd0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.892+0000 7fe11a600640 1 --2- 172.21.15.32:0/4024614300 >> v2:172.21.15.39:3300/0 conn(0x7fe11415f810 0x7fe1140b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.892+0000 7fe11a600640 1 -- 172.21.15.32:0/4024614300 >> 172.21.15.32:0/4024614300 conn(0x7fe11406b3a0 msgr2=0x7fe11406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.892+0000 7fe11a600640 1 -- 172.21.15.32:0/4024614300 shutdown_connections 2024-03-06T07:31:37.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.892+0000 7fe11a600640 1 -- 172.21.15.32:0/4024614300 wait complete. 2024-03-06T07:31:37.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.893+0000 7fe11a600640 1 Processor -- start 2024-03-06T07:31:37.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.894+0000 7fe11a600640 1 -- start start 2024-03-06T07:31:37.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.894+0000 7fe11a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1141f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.894+0000 7fe11a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe114145430 0x7fe11413c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe11a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe11415f810 0x7fe114143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe11a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114145990 con 0x7fe1140b2080 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe11a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114145810 con 0x7fe114145430 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe11a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140cde00 con 0x7fe11415f810 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1141f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1141f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58982/0 (socket says 172.21.15.32:58982) 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 -- 172.21.15.32:0/1182360639 learned_addr learned my addr 172.21.15.32:0/1182360639 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe10be00640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe114145430 0x7fe11413c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119c00640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe11415f810 0x7fe114143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 -- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe11415f810 msgr2=0x7fe114143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe11415f810 0x7fe114143ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 -- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe114145430 msgr2=0x7fe11413c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe114145430 0x7fe11413c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.895+0000 7fe119200640 1 -- 172.21.15.32:0/1182360639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11413e0c0 con 0x7fe1140b2080 2024-03-06T07:31:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.896+0000 7fe119200640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1141f3d70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c430 tx=0x7fe10400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.896+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104011790 con 0x7fe1140b2080 2024-03-06T07:31:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.896+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe104017070 con 0x7fe1140b2080 2024-03-06T07:31:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.896+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104011bb0 con 0x7fe1140b2080 2024-03-06T07:31:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.896+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11413e390 con 0x7fe1140b2080 2024-03-06T07:31:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.896+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11413ebd0 con 0x7fe1140b2080 2024-03-06T07:31:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.898+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11404f330 con 0x7fe1140b2080 2024-03-06T07:31:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.898+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe10401c460 con 0x7fe1140b2080 2024-03-06T07:31:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.899+0000 7fe111600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe0f006b500 0x7fe0f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.900+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe104089cb0 con 0x7fe1140b2080 2024-03-06T07:31:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.900+0000 7fe10be00640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe0f006b500 0x7fe0f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.900+0000 7fe10be00640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe0f006b500 0x7fe0f006d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc006fd0 tx=0x7fe0fc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:37.904+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe11413e390 con 0x7fe1140b2080 2024-03-06T07:31:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.102+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe11406ed30 con 0x7fe1140b2080 2024-03-06T07:31:38.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.103+0000 7fe111600640 1 -- 172.21.15.32:0/1182360639 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe104058f10 con 0x7fe1140b2080 2024-03-06T07:31:38.106 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe0f006b500 msgr2=0x7fe0f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe0f006b500 0x7fe0f006d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc006fd0 tx=0x7fe0fc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 msgr2=0x7fe1141f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1141f3d70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c430 tx=0x7fe10400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 shutdown_connections 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe0f006b500 0x7fe0f006d9c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe11415f810 0x7fe114143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe114145430 0x7fe11413c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 --2- 172.21.15.32:0/1182360639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1140b2080 0x7fe1141f3d70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.106+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 >> 172.21.15.32:0/1182360639 conn(0x7fe11406b3a0 msgr2=0x7fe1140ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.107+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 shutdown_connections 2024-03-06T07:31:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.107+0000 7fe11a600640 1 -- 172.21.15.32:0/1182360639 wait complete. 2024-03-06T07:31:38.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:31:38.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.342+0000 7f8a2d200640 1 Processor -- start 2024-03-06T07:31:38.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.342+0000 7f8a2d200640 1 -- start start 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.343+0000 7f8a2d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280b3c90 0x7f8a280cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.343+0000 7f8a2d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8a280cf5c0 0x7f8a280b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.343+0000 7f8a2d200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a280b45b0 con 0x7f8a2806cb10 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.343+0000 7f8a2d200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280ce160 con 0x7f8a280cf5c0 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.343+0000 7f8a2d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280b4860 con 0x7f8a280b3c90 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a26800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8a2806cb10 0x7f8a2806cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39852/0 (socket says 172.21.15.32:39852) 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a26800640 1 -- 172.21.15.32:0/173429583 learned_addr learned my addr 172.21.15.32:0/173429583 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:38.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a25400640 1 --2- 172.21.15.32:0/173429583 >> v2:172.21.15.32:3300/0 conn(0x7f8a280cf5c0 0x7f8a280b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a25e00640 1 --2- 172.21.15.32:0/173429583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280b3c90 0x7f8a280cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a24a00640 1 -- 172.21.15.32:0/173429583 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137162199 0 0) 0x7f8a280b45b0 con 0x7f8a2806cb10 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a24a00640 1 -- 172.21.15.32:0/173429583 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0c005730 con 0x7f8a2806cb10 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a25400640 1 -- 172.21.15.32:0/173429583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280b3c90 msgr2=0x7f8a280cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a25400640 1 --2- 172.21.15.32:0/173429583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280b3c90 0x7f8a280cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.344+0000 7f8a25400640 1 -- 172.21.15.32:0/173429583 >> v1:172.21.15.32:6789/0 conn(0x7f8a2806cb10 legacy=0x7f8a2806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.345+0000 7f8a25400640 1 -- 172.21.15.32:0/173429583 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280b4b40 con 0x7f8a280cf5c0 2024-03-06T07:31:38.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.346+0000 7f8a25400640 1 --2- 172.21.15.32:0/173429583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280cf5c0 0x7f8a280b4070 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8a140097a0 tx=0x7f8a1402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90467881c444677 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.346+0000 7f8a24a00640 1 -- 172.21.15.32:0/173429583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14034070 con 0x7f8a280cf5c0 2024-03-06T07:31:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.346+0000 7f8a24a00640 1 -- 172.21.15.32:0/173429583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14030800 con 0x7f8a280cf5c0 2024-03-06T07:31:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.346+0000 7f8a24a00640 1 -- 172.21.15.32:0/173429583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14030bb0 con 0x7f8a280cf5c0 2024-03-06T07:31:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 -- 172.21.15.32:0/173429583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280cf5c0 msgr2=0x7f8a280b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 --2- 172.21.15.32:0/173429583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280cf5c0 0x7f8a280b4070 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8a140097a0 tx=0x7f8a1402b940 comp rx=0 tx=0).stop 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 -- 172.21.15.32:0/173429583 shutdown_connections 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 --2- 172.21.15.32:0/173429583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280cf5c0 0x7f8a280b4070 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 --2- 172.21.15.32:0/173429583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280b3c90 0x7f8a280cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 -- 172.21.15.32:0/173429583 >> 172.21.15.32:0/173429583 conn(0x7f8a2806afb0 msgr2=0x7f8a2806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 -- 172.21.15.32:0/173429583 shutdown_connections 2024-03-06T07:31:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.347+0000 7f8a2d200640 1 -- 172.21.15.32:0/173429583 wait complete. 2024-03-06T07:31:38.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.349+0000 7f8a2d200640 1 Processor -- start 2024-03-06T07:31:38.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.349+0000 7f8a2d200640 1 -- start start 2024-03-06T07:31:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a2d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a2806cb10 0x7f8a280c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a2d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 0x7f8a281547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a2d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280cf5c0 0x7f8a28158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a2d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2815b340 con 0x7f8a280b3c90 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a2d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280b5100 con 0x7f8a2806cb10 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a2d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2806d690 con 0x7f8a280cf5c0 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a25400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 0x7f8a281547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a25400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 0x7f8a281547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59000/0 (socket says 172.21.15.32:59000) 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.350+0000 7f8a25400640 1 -- 172.21.15.32:0/1303061782 learned_addr learned my addr 172.21.15.32:0/1303061782 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25e00640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a2806cb10 0x7f8a280c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25400640 1 -- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280cf5c0 msgr2=0x7f8a28158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a26800640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280cf5c0 0x7f8a28158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25400640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280cf5c0 0x7f8a28158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25400640 1 -- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a2806cb10 msgr2=0x7f8a280c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25400640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a2806cb10 0x7f8a280c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25400640 1 -- 172.21.15.32:0/1303061782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28157cd0 con 0x7f8a280b3c90 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25e00640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a2806cb10 0x7f8a280c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.351+0000 7f8a25400640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 0x7f8a281547b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8a140098c0 tx=0x7f8a140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.352+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14034040 con 0x7f8a280b3c90 2024-03-06T07:31:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.352+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1403ada0 con 0x7f8a280b3c90 2024-03-06T07:31:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.352+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14006e10 con 0x7f8a280b3c90 2024-03-06T07:31:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.352+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a28157f40 con 0x7f8a280b3c90 2024-03-06T07:31:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.352+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a280c9690 con 0x7f8a280b3c90 2024-03-06T07:31:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.353+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2804f3a0 con 0x7f8a280b3c90 2024-03-06T07:31:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.354+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a14037070 con 0x7f8a280b3c90 2024-03-06T07:31:38.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.355+0000 7f8a1e000640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a0806b500 0x7f8a0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.355+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8a140abe20 con 0x7f8a280b3c90 2024-03-06T07:31:38.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.355+0000 7f8a25e00640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a0806b500 0x7f8a0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.356+0000 7f8a25e00640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a0806b500 0x7f8a0806d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.359+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a1403f070 con 0x7f8a280b3c90 2024-03-06T07:31:38.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.561+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8a280cdc20 con 0x7f8a280b3c90 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.562+0000 7f8a1e000640 1 -- 172.21.15.32:0/1303061782 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8a1407afe0 con 0x7f8a280b3c90 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:31:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.564+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a0806b500 msgr2=0x7f8a0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.564+0000 7f8a2d200640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a0806b500 0x7f8a0806d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18008040 comp rx=0 tx=0).stop 2024-03-06T07:31:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 msgr2=0x7f8a281547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 0x7f8a281547b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8a140098c0 tx=0x7f8a140046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 shutdown_connections 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a0806b500 0x7f8a0806d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a280cf5c0 0x7f8a28158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a280b3c90 0x7f8a281547b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 --2- 172.21.15.32:0/1303061782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a2806cb10 0x7f8a280c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 >> 172.21.15.32:0/1303061782 conn(0x7f8a2806afb0 msgr2=0x7f8a280b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 shutdown_connections 2024-03-06T07:31:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.565+0000 7f8a2d200640 1 -- 172.21.15.32:0/1303061782 wait complete. 2024-03-06T07:31:38.586 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:31:38.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:31:38.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.808+0000 7fb3e6a00640 1 Processor -- start 2024-03-06T07:31:38.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.808+0000 7fb3e6a00640 1 -- start start 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.809+0000 7fb3e6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb3e00b2ac0 0x7fb3e006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.809+0000 7fb3e6a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3e006e030 0x7fb3e00cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.809+0000 7fb3e6a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e006e410 con 0x7fb3e0144540 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3e6a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e00b3ff0 con 0x7fb3e006e030 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3e6a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e00b2f50 con 0x7fb3e00b2ac0 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3e5600640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb3e0144540 0x7fb3e015f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39862/0 (socket says 172.21.15.32:39862) 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3e5600640 1 -- 172.21.15.32:0/2461963724 learned_addr learned my addr 172.21.15.32:0/2461963724 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3e6000640 1 --2- 172.21.15.32:0/2461963724 >> v2:172.21.15.32:3300/0 conn(0x7fb3e006e030 0x7fb3e00cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3e4c00640 1 --2- 172.21.15.32:0/2461963724 >> v2:172.21.15.39:3300/0 conn(0x7fb3e00b2ac0 0x7fb3e006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3dfe00640 1 -- 172.21.15.32:0/2461963724 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296532753 0 0) 0x7fb3e006e410 con 0x7fb3e0144540 2024-03-06T07:31:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.810+0000 7fb3dfe00640 1 -- 172.21.15.32:0/2461963724 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c8003520 con 0x7fb3e0144540 2024-03-06T07:31:38.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.811+0000 7fb3e6000640 1 -- 172.21.15.32:0/2461963724 >> v2:172.21.15.39:3300/0 conn(0x7fb3e00b2ac0 msgr2=0x7fb3e006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.811+0000 7fb3e6000640 1 --2- 172.21.15.32:0/2461963724 >> v2:172.21.15.39:3300/0 conn(0x7fb3e00b2ac0 0x7fb3e006daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.811+0000 7fb3e6000640 1 -- 172.21.15.32:0/2461963724 >> v1:172.21.15.32:6789/0 conn(0x7fb3e0144540 legacy=0x7fb3e015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.811+0000 7fb3e6000640 1 -- 172.21.15.32:0/2461963724 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e00b3230 con 0x7fb3e006e030 2024-03-06T07:31:38.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.811+0000 7fb3e4c00640 1 --2- 172.21.15.32:0/2461963724 >> v2:172.21.15.39:3300/0 conn(0x7fb3e00b2ac0 0x7fb3e006daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:38.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.812+0000 7fb3e6000640 1 --2- 172.21.15.32:0/2461963724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e00cdee0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00cd0d0 tx=0x7fb3d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52178d29c80eb43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.812+0000 7fb3dfe00640 1 -- 172.21.15.32:0/2461963724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8035020 con 0x7fb3e006e030 2024-03-06T07:31:38.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.812+0000 7fb3dfe00640 1 -- 172.21.15.32:0/2461963724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d8031830 con 0x7fb3e006e030 2024-03-06T07:31:38.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.812+0000 7fb3dfe00640 1 -- 172.21.15.32:0/2461963724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8031be0 con 0x7fb3e006e030 2024-03-06T07:31:38.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2461963724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 msgr2=0x7fb3e00cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2461963724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e00cdee0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00cd0d0 tx=0x7fb3d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2461963724 shutdown_connections 2024-03-06T07:31:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2461963724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e00cdee0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2461963724 >> v2:172.21.15.39:3300/0 conn(0x7fb3e00b2ac0 0x7fb3e006daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2461963724 >> 172.21.15.32:0/2461963724 conn(0x7fb3e00b17c0 msgr2=0x7fb3e00b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2461963724 shutdown_connections 2024-03-06T07:31:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.813+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2461963724 wait complete. 2024-03-06T07:31:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.815+0000 7fb3e6a00640 1 Processor -- start 2024-03-06T07:31:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.815+0000 7fb3e6a00640 1 -- start start 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e014ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3e0144540 0x7fb3e014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e6a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3e0143690 0x7fb3e014b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e6a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e00b3ad0 con 0x7fb3e006e030 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e00b37f0 con 0x7fb3e0144540 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e6a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e00b4a30 con 0x7fb3e0143690 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e014ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e014ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59022/0 (socket says 172.21.15.32:59022) 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e5600640 1 -- 172.21.15.32:0/2676096510 learned_addr learned my addr 172.21.15.32:0/2676096510 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.816+0000 7fb3e4c00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3e0144540 0x7fb3e014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e5600640 1 -- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3e0143690 msgr2=0x7fb3e014b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e5600640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3e0143690 0x7fb3e014b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e5600640 1 -- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3e0144540 msgr2=0x7fb3e014f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e5600640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3e0144540 0x7fb3e014f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e5600640 1 -- 172.21.15.32:0/2676096510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e01403d0 con 0x7fb3e006e030 2024-03-06T07:31:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e5600640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e014ac80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400cad0 tx=0x7fb3d400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4017070 con 0x7fb3e006e030 2024-03-06T07:31:38.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d400abd0 con 0x7fb3e006e030 2024-03-06T07:31:38.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4012400 con 0x7fb3e006e030 2024-03-06T07:31:38.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.817+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e01406b0 con 0x7fb3e006e030 2024-03-06T07:31:38.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.818+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e0140c70 con 0x7fb3e006e030 2024-03-06T07:31:38.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.819+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e004f330 con 0x7fb3e006e030 2024-03-06T07:31:38.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.820+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3d401a070 con 0x7fb3e006e030 2024-03-06T07:31:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.820+0000 7fb3dd600640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3c006b500 0x7fb3c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.821+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb3d4089b10 con 0x7fb3e006e030 2024-03-06T07:31:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.821+0000 7fb3e4c00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3c006b500 0x7fb3c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.821+0000 7fb3e4c00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3c006b500 0x7fb3c006d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0157a60 tx=0x7fb3cc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:38.825+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3d4058d70 con 0x7fb3e006e030 2024-03-06T07:31:39.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.026+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb3e006e820 con 0x7fb3e006e030 2024-03-06T07:31:39.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.714+0000 7fb3dd600640 1 -- 172.21.15.32:0/2676096510 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v54) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb3d4058670 con 0x7fb3e006e030 2024-03-06T07:31:39.717 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:31:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.717+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3c006b500 msgr2=0x7fb3c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.717+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3c006b500 0x7fb3c006d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0157a60 tx=0x7fb3cc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.717+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 msgr2=0x7fb3e014ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.717+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e014ac80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb3d400cad0 tx=0x7fb3d400cf90 comp rx=0 tx=0).stop 2024-03-06T07:31:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 shutdown_connections 2024-03-06T07:31:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3c006b500 0x7fb3c006d9c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3e0143690 0x7fb3e014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3e0144540 0x7fb3e014f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 --2- 172.21.15.32:0/2676096510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e006e030 0x7fb3e014ac80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 >> 172.21.15.32:0/2676096510 conn(0x7fb3e00b17c0 msgr2=0x7fb3e0070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 shutdown_connections 2024-03-06T07:31:39.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.718+0000 7fb3e6a00640 1 -- 172.21.15.32:0/2676096510 wait complete. 2024-03-06T07:31:39.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:31:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.955+0000 7f62b6a00640 1 Processor -- start 2024-03-06T07:31:39.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.956+0000 7f62b6a00640 1 -- start start 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.957+0000 7f62b6a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62b00ad6f0 0x7f62b015c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.957+0000 7f62b6a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b015f960 0x7f62b00adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.957+0000 7f62b6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f62b00b3c30 0x7f62b00b4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.957+0000 7f62b6a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b006ce60 con 0x7f62b00ad6f0 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.957+0000 7f62b6a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b0156830 con 0x7f62b00b3c30 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.957+0000 7f62b6a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b00ae010 con 0x7f62b015f960 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62b00ad6f0 0x7f62b015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62b00ad6f0 0x7f62b015c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59036/0 (socket says 172.21.15.32:59036) 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 -- 172.21.15.32:0/703285774 learned_addr learned my addr 172.21.15.32:0/703285774 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62af400640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b015f960 0x7f62b00adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62b4a00640 1 --2- 172.21.15.32:0/703285774 >> v2:172.21.15.39:3300/0 conn(0x7f62b00b3c30 0x7f62b00b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 -- 172.21.15.32:0/703285774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b015f960 msgr2=0x7f62b00adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b015f960 0x7f62b00adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 -- 172.21.15.32:0/703285774 >> v2:172.21.15.39:3300/0 conn(0x7f62b00b3c30 msgr2=0x7f62b00b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 --2- 172.21.15.32:0/703285774 >> v2:172.21.15.39:3300/0 conn(0x7f62b00b3c30 0x7f62b00b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62afe00640 1 -- 172.21.15.32:0/703285774 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0160120 con 0x7f62b00ad6f0 2024-03-06T07:31:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62af400640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b015f960 0x7f62b00adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.958+0000 7f62b4a00640 1 --2- 172.21.15.32:0/703285774 >> v2:172.21.15.39:3300/0 conn(0x7f62b00b3c30 0x7f62b00b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.960+0000 7f62afe00640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00ad6f0 0x7f62b015c370 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f62b00af8a0 tx=0x7f62a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab1d6dff8609009 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.960+0000 7f62aea00640 1 -- 172.21.15.32:0/703285774 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a002ca80 con 0x7f62b00ad6f0 2024-03-06T07:31:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.960+0000 7f62aea00640 1 -- 172.21.15.32:0/703285774 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a002cc20 con 0x7f62b00ad6f0 2024-03-06T07:31:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.960+0000 7f62aea00640 1 -- 172.21.15.32:0/703285774 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a00343f0 con 0x7f62b00ad6f0 2024-03-06T07:31:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 -- 172.21.15.32:0/703285774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00ad6f0 msgr2=0x7f62b015c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00ad6f0 0x7f62b015c370 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f62b00af8a0 tx=0x7f62a002bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 -- 172.21.15.32:0/703285774 shutdown_connections 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 --2- 172.21.15.32:0/703285774 >> v2:172.21.15.39:3300/0 conn(0x7f62b00b3c30 0x7f62b00b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b015f960 0x7f62b00adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 --2- 172.21.15.32:0/703285774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00ad6f0 0x7f62b015c370 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 -- 172.21.15.32:0/703285774 >> 172.21.15.32:0/703285774 conn(0x7f62b006b3d0 msgr2=0x7f62b006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 -- 172.21.15.32:0/703285774 shutdown_connections 2024-03-06T07:31:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.961+0000 7f62b6a00640 1 -- 172.21.15.32:0/703285774 wait complete. 2024-03-06T07:31:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.963+0000 7f62b6a00640 1 Processor -- start 2024-03-06T07:31:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.963+0000 7f62b6a00640 1 -- start start 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62b6a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b00ad6f0 0x7f62b014a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62b6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 0x7f62b00716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62b6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b015f960 0x7f62b01f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62b6a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b006d0b0 con 0x7f62b00b3c30 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62b6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b01606e0 con 0x7f62b015f960 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62b6a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b015fdc0 con 0x7f62b00ad6f0 2024-03-06T07:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 0x7f62b00716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 0x7f62b00716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59048/0 (socket says 172.21.15.32:59048) 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.964+0000 7f62af400640 1 -- 172.21.15.32:0/2845032283 learned_addr learned my addr 172.21.15.32:0/2845032283 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62afe00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b00ad6f0 0x7f62b014a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62b4a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b015f960 0x7f62b01f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62af400640 1 -- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b00ad6f0 msgr2=0x7f62b014a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62af400640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b00ad6f0 0x7f62b014a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62af400640 1 -- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b015f960 msgr2=0x7f62b01f76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62af400640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b015f960 0x7f62b01f76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62af400640 1 -- 172.21.15.32:0/2845032283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b01f8170 con 0x7f62b00b3c30 2024-03-06T07:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62af400640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 0x7f62b00716d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f629800c7f0 tx=0x7f629800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.965+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298015070 con 0x7f62b00b3c30 2024-03-06T07:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.966+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6298007820 con 0x7f62b00b3c30 2024-03-06T07:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.966+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b01f8450 con 0x7f62b00b3c30 2024-03-06T07:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.966+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298007bd0 con 0x7f62b00b3c30 2024-03-06T07:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.966+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b00711b0 con 0x7f62b00b3c30 2024-03-06T07:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.967+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b0068010 con 0x7f62b00b3c30 2024-03-06T07:31:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.968+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6298018070 con 0x7f62b00b3c30 2024-03-06T07:31:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.969+0000 7f62a7e00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f628c06b4d0 0x7f628c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.969+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6298089f80 con 0x7f62b00b3c30 2024-03-06T07:31:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.969+0000 7f62afe00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f628c06b4d0 0x7f628c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.970+0000 7f62afe00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f628c06b4d0 0x7f628c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f62a0035000 tx=0x7f62a0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:39.975+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6298058f40 con 0x7f62b00b3c30 2024-03-06T07:31:40.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.175+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f62b00cc440 con 0x7f62b00b3c30 2024-03-06T07:31:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.720+0000 7f62a7e00640 1 -- 172.21.15.32:0/2845032283 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v55) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6298058840 con 0x7f62b00b3c30 2024-03-06T07:31:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:31:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f628c06b4d0 msgr2=0x7f628c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f628c06b4d0 0x7f628c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f62a0035000 tx=0x7f62a0030000 comp rx=0 tx=0).stop 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 msgr2=0x7f62b00716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 0x7f62b00716d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f629800c7f0 tx=0x7f629800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 shutdown_connections 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f628c06b4d0 0x7f628c06d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b015f960 0x7f62b01f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b00b3c30 0x7f62b00716d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 --2- 172.21.15.32:0/2845032283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62b00ad6f0 0x7f62b014a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 >> 172.21.15.32:0/2845032283 conn(0x7f62b006b3d0 msgr2=0x7f62b00b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 shutdown_connections 2024-03-06T07:31:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.724+0000 7f62b6a00640 1 -- 172.21.15.32:0/2845032283 wait complete. 2024-03-06T07:31:40.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:31:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.945+0000 7f7a03e00640 1 Processor -- start 2024-03-06T07:31:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.945+0000 7f7a03e00640 1 -- start start 2024-03-06T07:31:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a03e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a040b3c20 0x7f7a040729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a03e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a04071930 con 0x7f7a040b1b40 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a03e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a04071c10 con 0x7f7a040b4000 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a03e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a040b4490 con 0x7f7a040b3c20 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a040b3c20 0x7f7a040729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a03400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7a040b1b40 0x7f7a0415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39878/0 (socket says 172.21.15.32:39878) 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a03400640 1 -- 172.21.15.32:0/2223362954 learned_addr learned my addr 172.21.15.32:0/2223362954 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.947+0000 7f7a02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a040b3c20 0x7f7a040729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59062/0 (socket says 172.21.15.32:59062) 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.948+0000 7f7a02a00640 1 -- 172.21.15.32:0/2223362954 >> v1:172.21.15.39:6789/0 conn(0x7f7a040b4000 legacy=0x7f7a0415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.948+0000 7f7a02a00640 1 -- 172.21.15.32:0/2223362954 >> v1:172.21.15.32:6789/0 conn(0x7f7a040b1b40 legacy=0x7f7a0415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.948+0000 7f7a02a00640 1 -- 172.21.15.32:0/2223362954 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04071ef0 con 0x7f7a040b3c20 2024-03-06T07:31:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.949+0000 7f7a02a00640 1 --2- 172.21.15.32:0/2223362954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b3c20 0x7f7a040729d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009d90 tx=0x7f79f402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b6a8b5c923a8a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.949+0000 7f7a02000640 1 -- 172.21.15.32:0/2223362954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f403b070 con 0x7f7a040b3c20 2024-03-06T07:31:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.949+0000 7f7a02000640 1 -- 172.21.15.32:0/2223362954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f4036440 con 0x7f7a040b3c20 2024-03-06T07:31:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.949+0000 7f7a02000640 1 -- 172.21.15.32:0/2223362954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4036810 con 0x7f7a040b3c20 2024-03-06T07:31:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 -- 172.21.15.32:0/2223362954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b3c20 msgr2=0x7f7a040729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 --2- 172.21.15.32:0/2223362954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b3c20 0x7f7a040729d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009d90 tx=0x7f79f402e0c0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 -- 172.21.15.32:0/2223362954 shutdown_connections 2024-03-06T07:31:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 --2- 172.21.15.32:0/2223362954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b3c20 0x7f7a040729d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 -- 172.21.15.32:0/2223362954 >> 172.21.15.32:0/2223362954 conn(0x7f7a0406b3a0 msgr2=0x7f7a0406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 -- 172.21.15.32:0/2223362954 shutdown_connections 2024-03-06T07:31:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.950+0000 7f7a03e00640 1 -- 172.21.15.32:0/2223362954 wait complete. 2024-03-06T07:31:40.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.952+0000 7f7a03e00640 1 Processor -- start 2024-03-06T07:31:40.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.952+0000 7f7a03e00640 1 -- start start 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 0x7f7a041484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a040b3c20 0x7f7a04157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a040b4000 0x7f7a040aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a03e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040724b0 con 0x7f7a040b1b40 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a03e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0415bfb0 con 0x7f7a040b3c20 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a03e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040b1fa0 con 0x7f7a040b4000 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 0x7f7a041484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 0x7f7a041484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59070/0 (socket says 172.21.15.32:59070) 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.953+0000 7f7a02a00640 1 -- 172.21.15.32:0/1508521790 learned_addr learned my addr 172.21.15.32:0/1508521790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f7a02a00640 1 -- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a040b4000 msgr2=0x7f7a040aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f7a02a00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a040b4000 0x7f7a040aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f7a02a00640 1 -- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a040b3c20 msgr2=0x7f7a04157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f79fa000640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a040b3c20 0x7f7a04157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f7a02a00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a040b3c20 0x7f7a04157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f7a02a00640 1 -- 172.21.15.32:0/1508521790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04156a60 con 0x7f7a040b1b40 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f7a02a00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 0x7f7a041484f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79f4039040 tx=0x7f79f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f403b040 con 0x7f7a040b1b40 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f4044070 con 0x7f7a040b1b40 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.954+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f403e8a0 con 0x7f7a040b1b40 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.955+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a040adbf0 con 0x7f7a040b1b40 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.955+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a040ae1b0 con 0x7f7a040b1b40 2024-03-06T07:31:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.956+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0404f330 con 0x7f7a040b1b40 2024-03-06T07:31:40.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.957+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79f403ea40 con 0x7f7a040b1b40 2024-03-06T07:31:40.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.958+0000 7f79fbe00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f79e006b4d0 0x7f79e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.958+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f79f40ac710 con 0x7f7a040b1b40 2024-03-06T07:31:40.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.958+0000 7f79fa000640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f79e006b4d0 0x7f79e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.959+0000 7f79fa000640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f79e006b4d0 0x7f79e006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a040af7b0 tx=0x7f79ec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:40.963+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a04156a60 con 0x7f7a040b1b40 2024-03-06T07:31:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.204+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7a0406e3e0 con 0x7f7a040b1b40 2024-03-06T07:31:41.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.769+0000 7f79fbe00640 1 -- 172.21.15.32:0/1508521790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f79f407b710 con 0x7f7a040b1b40 2024-03-06T07:31:41.775 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:31:41.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.776+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f79e006b4d0 msgr2=0x7f79e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.776+0000 7f7a03e00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f79e006b4d0 0x7f79e006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a040af7b0 tx=0x7f79ec008040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.776+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 msgr2=0x7f7a041484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.776+0000 7f7a03e00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 0x7f7a041484f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79f4039040 tx=0x7f79f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.777+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 shutdown_connections 2024-03-06T07:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.777+0000 7f7a03e00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f79e006b4d0 0x7f79e006d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.777+0000 7f7a03e00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a040b4000 0x7f7a040aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.777+0000 7f7a03e00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a040b3c20 0x7f7a04157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.777+0000 7f7a03e00640 1 --2- 172.21.15.32:0/1508521790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a040b1b40 0x7f7a041484f0 secure :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79f4039040 tx=0x7f79f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.777+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 >> 172.21.15.32:0/1508521790 conn(0x7f7a0406b3a0 msgr2=0x7f7a0406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.779+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 shutdown_connections 2024-03-06T07:31:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.781+0000 7f7a03e00640 1 -- 172.21.15.32:0/1508521790 wait complete. 2024-03-06T07:31:41.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.991+0000 7f8edb600640 1 Processor -- start 2024-03-06T07:31:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.991+0000 7f8edb600640 1 -- start start 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8edb600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ed40af580 0x7f8ed415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8edb600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8ed40cdc20 0x7f8ed40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8edb600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed415cc40 con 0x7f8ed40af1a0 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8edb600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed415cf20 con 0x7f8ed40af580 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8edb600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed415d200 con 0x7f8ed40cdc20 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ed40af580 0x7f8ed415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ed40af580 0x7f8ed415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59086/0 (socket says 172.21.15.32:59086) 2024-03-06T07:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 -- 172.21.15.32:0/149448796 learned_addr learned my addr 172.21.15.32:0/149448796 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ed9600640 1 --2- 172.21.15.32:0/149448796 >> v2:172.21.15.39:3300/0 conn(0x7f8ed40cdc20 0x7f8ed40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 -- 172.21.15.32:0/149448796 >> v2:172.21.15.39:3300/0 conn(0x7f8ed40cdc20 msgr2=0x7f8ed40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 --2- 172.21.15.32:0/149448796 >> v2:172.21.15.39:3300/0 conn(0x7f8ed40cdc20 0x7f8ed40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 -- 172.21.15.32:0/149448796 >> v1:172.21.15.39:6789/0 conn(0x7f8ed40af1a0 legacy=0x7f8ed40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.993+0000 7f8ecbe00640 1 -- 172.21.15.32:0/149448796 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed415d4e0 con 0x7f8ed40af580 2024-03-06T07:31:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.994+0000 7f8ecbe00640 1 --2- 172.21.15.32:0/149448796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed415c600 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8ed406c0c0 tx=0x7f8ec002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac631c588c3c6e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8ed3e00640 1 -- 172.21.15.32:0/149448796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec002e070 con 0x7f8ed40af580 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8ed3e00640 1 -- 172.21.15.32:0/149448796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0037070 con 0x7f8ed40af580 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8ed3e00640 1 -- 172.21.15.32:0/149448796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0031a00 con 0x7f8ed40af580 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 -- 172.21.15.32:0/149448796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 msgr2=0x7f8ed415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 --2- 172.21.15.32:0/149448796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed415c600 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8ed406c0c0 tx=0x7f8ec002b880 comp rx=0 tx=0).stop 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 -- 172.21.15.32:0/149448796 shutdown_connections 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 --2- 172.21.15.32:0/149448796 >> v2:172.21.15.39:3300/0 conn(0x7f8ed40cdc20 0x7f8ed40b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 --2- 172.21.15.32:0/149448796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed415c600 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 -- 172.21.15.32:0/149448796 >> 172.21.15.32:0/149448796 conn(0x7f8ed406afb0 msgr2=0x7f8ed406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 -- 172.21.15.32:0/149448796 shutdown_connections 2024-03-06T07:31:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.995+0000 7f8edb600640 1 -- 172.21.15.32:0/149448796 wait complete. 2024-03-06T07:31:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.997+0000 7f8edb600640 1 Processor -- start 2024-03-06T07:31:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.997+0000 7f8edb600640 1 -- start start 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8edb600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed40af1a0 0x7f8ed41514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8edb600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8edb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40cdc20 0x7f8ed415e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8edb600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40b36a0 con 0x7f8ed40af580 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8edb600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40b3520 con 0x7f8ed40af1a0 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8edb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40b3820 con 0x7f8ed40cdc20 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.998+0000 7f8ecbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed406e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59102/0 (socket says 172.21.15.32:59102) 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 -- 172.21.15.32:0/3978840304 learned_addr learned my addr 172.21.15.32:0/3978840304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ed8c00640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed40af1a0 0x7f8ed41514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 -- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40cdc20 msgr2=0x7f8ed415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40cdc20 0x7f8ed415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 -- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed40af1a0 msgr2=0x7f8ed41514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed40af1a0 0x7f8ed41514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 -- 172.21.15.32:0/3978840304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed40c9630 con 0x7f8ed40af580 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:41.999+0000 7f8ecbe00640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed406e4a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8ec002b8c0 tx=0x7f8ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.000+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0032880 con 0x7f8ed40af580 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.000+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0032a20 con 0x7f8ed40af580 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.000+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0032e10 con 0x7f8ed40af580 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.000+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed40c9910 con 0x7f8ed40af580 2024-03-06T07:31:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.000+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed40c9ed0 con 0x7f8ed40af580 2024-03-06T07:31:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.001+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed4069510 con 0x7f8ed40af580 2024-03-06T07:31:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.002+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ec004c060 con 0x7f8ed40af580 2024-03-06T07:31:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.003+0000 7f8ed1600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b520 0x7f8eb406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.003+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ec00abd60 con 0x7f8ed40af580 2024-03-06T07:31:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.003+0000 7f8ed8c00640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b520 0x7f8eb406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.004+0000 7f8ed8c00640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b520 0x7f8eb406d9e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40015d0 tx=0x7f8ec4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.007+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ec0040070 con 0x7f8ed40af580 2024-03-06T07:31:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.197+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ed415c4f0 con 0x7f8ed40af580 2024-03-06T07:31:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.200+0000 7f8ed1600640 1 -- 172.21.15.32:0/3978840304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f8ec007ac50 con 0x7f8ed40af580 2024-03-06T07:31:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:42.203 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:41.206908+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.202+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b520 msgr2=0x7f8eb406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.202+0000 7f8edb600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b520 0x7f8eb406d9e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40015d0 tx=0x7f8ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.202+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 msgr2=0x7f8ed406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.202+0000 7f8edb600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed406e4a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8ec002b8c0 tx=0x7f8ec00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 shutdown_connections 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8eb406b520 0x7f8eb406d9e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ed40cdc20 0x7f8ed415e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed40af580 0x7f8ed406e4a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 --2- 172.21.15.32:0/3978840304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ed40af1a0 0x7f8ed41514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 >> 172.21.15.32:0/3978840304 conn(0x7f8ed406afb0 msgr2=0x7f8ed406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 shutdown_connections 2024-03-06T07:31:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.203+0000 7f8edb600640 1 -- 172.21.15.32:0/3978840304 wait complete. 2024-03-06T07:31:42.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:42.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.426+0000 7f6443600640 1 Processor -- start 2024-03-06T07:31:42.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.426+0000 7f6443600640 1 -- start start 2024-03-06T07:31:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.428+0000 7f6443600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f643c0b34e0 0x7f643c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.428+0000 7f6443600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c15cc40 con 0x7f643c0af580 2024-03-06T07:31:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.428+0000 7f6443600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c15cf20 con 0x7f643c0af1a0 2024-03-06T07:31:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.428+0000 7f6443600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c15d200 con 0x7f643c0b34e0 2024-03-06T07:31:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.428+0000 7f643be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f643c0af580 0x7f643c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39888/0 (socket says 172.21.15.32:39888) 2024-03-06T07:31:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.428+0000 7f643be00640 1 -- 172.21.15.32:0/3190345893 learned_addr learned my addr 172.21.15.32:0/3190345893 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f6441600640 1 --2- 172.21.15.32:0/3190345893 >> v2:172.21.15.39:3300/0 conn(0x7f643c0b34e0 0x7f643c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085350954 0 0) 0x7f643c15cc40 con 0x7f643c0af580 2024-03-06T07:31:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6424003520 con 0x7f643c0af580 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187596833 0 0) 0x7f6424003520 con 0x7f643c0af580 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 >> v2:172.21.15.39:3300/0 conn(0x7f643c0b34e0 msgr2=0x7f643c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 --2- 172.21.15.32:0/3190345893 >> v2:172.21.15.39:3300/0 conn(0x7f643c0b34e0 0x7f643c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 >> v1:172.21.15.39:6789/0 conn(0x7f643c0af1a0 legacy=0x7f643c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c15d4e0 con 0x7f643c0af580 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6428001670 con 0x7f643c0af580 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 >> v1:172.21.15.32:6789/0 conn(0x7f643c0af580 legacy=0x7f643c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424003a20 0x7f6424003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6424004800 0x7f6424004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424005210 0x7f642400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c15d200 con 0x7f6424004800 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642400c160 con 0x7f6424005210 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.429+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642400c360 con 0x7f6424003a20 2024-03-06T07:31:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f643be00640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6424004800 0x7f6424004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f643be00640 1 -- 172.21.15.32:0/3190345893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424003a20 msgr2=0x7f6424003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f643be00640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424003a20 0x7f6424003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f643be00640 1 -- 172.21.15.32:0/3190345893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424005210 msgr2=0x7f642400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f643be00640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424005210 0x7f642400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f643be00640 1 -- 172.21.15.32:0/3190345893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424003520 con 0x7f6424004800 2024-03-06T07:31:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.430+0000 7f6441600640 1 -- 172.21.15.32:0/3190345893 reap_dead start 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.431+0000 7f643be00640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6424004800 0x7f6424004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f643c0afa10 tx=0x7f6424005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536fd0fa5645bfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.431+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428007d50 con 0x7f6424004800 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.431+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6428030070 con 0x7f6424004800 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.431+0000 7f643b400640 1 -- 172.21.15.32:0/3190345893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428005d70 con 0x7f6424004800 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 -- 172.21.15.32:0/3190345893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6424004800 msgr2=0x7f6424004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6424004800 0x7f6424004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f643c0afa10 tx=0x7f6424005830 comp rx=0 tx=0).stop 2024-03-06T07:31:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 -- 172.21.15.32:0/3190345893 shutdown_connections 2024-03-06T07:31:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 --2- 172.21.15.32:0/3190345893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6424004800 0x7f6424004c80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 -- 172.21.15.32:0/3190345893 >> 172.21.15.32:0/3190345893 conn(0x7f643c06afb0 msgr2=0x7f643c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 -- 172.21.15.32:0/3190345893 shutdown_connections 2024-03-06T07:31:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.432+0000 7f6443600640 1 -- 172.21.15.32:0/3190345893 wait complete. 2024-03-06T07:31:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.434+0000 7f6443600640 1 Processor -- start 2024-03-06T07:31:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.434+0000 7f6443600640 1 -- start start 2024-03-06T07:31:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6443600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424004800 0x7f643c150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6443600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f643c15ae40 0x7f643c150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6443600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 0x7f643c15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6443600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c15c600 con 0x7f643c0b0b70 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6443600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c0cde30 con 0x7f643c15ae40 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6443600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c15c780 con 0x7f6424004800 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6441600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 0x7f643c15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6441600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 0x7f643c15e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59128/0 (socket says 172.21.15.32:59128) 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.435+0000 7f6441600640 1 -- 172.21.15.32:0/701372757 learned_addr learned my addr 172.21.15.32:0/701372757 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6440c00640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424004800 0x7f643c150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6441600640 1 -- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424004800 msgr2=0x7f643c150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6441600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424004800 0x7f643c150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f643be00640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f643c15ae40 0x7f643c150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6441600640 1 -- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f643c15ae40 msgr2=0x7f643c150c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6441600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f643c15ae40 0x7f643c150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6441600640 1 -- 172.21.15.32:0/701372757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c0b14e0 con 0x7f643c0b0b70 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6440c00640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424004800 0x7f643c150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6441600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 0x7f643c15e1a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f643c0cda30 tx=0x7f643400ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643400bb10 con 0x7f643c0b0b70 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643401a070 con 0x7f643c0b0b70 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434014400 con 0x7f643c0b0b70 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c0b17c0 con 0x7f643c0b0b70 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.436+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c09acf0 con 0x7f643c0b0b70 2024-03-06T07:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.438+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c04f3a0 con 0x7f643c0b0b70 2024-03-06T07:31:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.438+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6434013070 con 0x7f643c0b0b70 2024-03-06T07:31:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.439+0000 7f6438c00640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f641c06b4d0 0x7f641c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.439+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f643408a740 con 0x7f643c0b0b70 2024-03-06T07:31:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.439+0000 7f6440c00640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f641c06b4d0 0x7f641c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.440+0000 7f6440c00640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f641c06b4d0 0x7f641c06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f643c050c30 tx=0x7f6430009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.444+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6434024020 con 0x7f643c0b0b70 2024-03-06T07:31:42.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.639+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f643c0b3bb0 con 0x7f643c0b0b70 2024-03-06T07:31:42.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.641+0000 7f6438c00640 1 -- 172.21.15.32:0/701372757 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f6434059520 con 0x7f643c0b0b70 2024-03-06T07:31:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:41.206908+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:42.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f641c06b4d0 msgr2=0x7f641c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f641c06b4d0 0x7f641c06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f643c050c30 tx=0x7f6430009040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 msgr2=0x7f643c15e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 0x7f643c15e1a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f643c0cda30 tx=0x7f643400ef50 comp rx=0 tx=0).stop 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 shutdown_connections 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f641c06b4d0 0x7f641c06d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f643c0b0b70 0x7f643c15e1a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f643c15ae40 0x7f643c150c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 --2- 172.21.15.32:0/701372757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6424004800 0x7f643c150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 >> 172.21.15.32:0/701372757 conn(0x7f643c06afb0 msgr2=0x7f643c0b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 shutdown_connections 2024-03-06T07:31:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.644+0000 7f6443600640 1 -- 172.21.15.32:0/701372757 wait complete. 2024-03-06T07:31:42.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:31:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.879+0000 7f00f5000640 1 Processor -- start 2024-03-06T07:31:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.879+0000 7f00f5000640 1 -- start start 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.880+0000 7f00f5000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00f0145430 0x7f00f015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.880+0000 7f00f5000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.880+0000 7f00f5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f00b3fe0 con 0x7f00f00b2080 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.880+0000 7f00f5000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f015fe40 con 0x7f00f0145430 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.880+0000 7f00f5000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0160120 con 0x7f00f015cc90 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.880+0000 7f00ef400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00f0145430 0x7f00f015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00ef400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00f0145430 0x7f00f015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59134/0 (socket says 172.21.15.32:59134) 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00ef400640 1 -- 172.21.15.32:0/2455616512 learned_addr learned my addr 172.21.15.32:0/2455616512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00eea00640 1 --2- 172.21.15.32:0/2455616512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00ef400640 1 -- 172.21.15.32:0/2455616512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 msgr2=0x7f00f00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00ef400640 1 --2- 172.21.15.32:0/2455616512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f00b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00ef400640 1 -- 172.21.15.32:0/2455616512 >> v1:172.21.15.39:6789/0 conn(0x7f00f00b2080 legacy=0x7f00f006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.881+0000 7f00ef400640 1 -- 172.21.15.32:0/2455616512 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f0160400 con 0x7f00f0145430 2024-03-06T07:31:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.882+0000 7f00ef400640 1 --2- 172.21.15.32:0/2455616512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f015f810 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009870 tx=0x7f00e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc60aae2eb665d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.882+0000 7f00ee000640 1 -- 172.21.15.32:0/2455616512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e0035020 con 0x7f00f0145430 2024-03-06T07:31:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00ee000640 1 -- 172.21.15.32:0/2455616512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e0031640 con 0x7f00f0145430 2024-03-06T07:31:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00ee000640 1 -- 172.21.15.32:0/2455616512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e002d040 con 0x7f00f0145430 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 -- 172.21.15.32:0/2455616512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 msgr2=0x7f00f015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 --2- 172.21.15.32:0/2455616512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f015f810 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009870 tx=0x7f00e002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 -- 172.21.15.32:0/2455616512 shutdown_connections 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 --2- 172.21.15.32:0/2455616512 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f00b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 --2- 172.21.15.32:0/2455616512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f015f810 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 -- 172.21.15.32:0/2455616512 >> 172.21.15.32:0/2455616512 conn(0x7f00f006b3a0 msgr2=0x7f00f006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 -- 172.21.15.32:0/2455616512 shutdown_connections 2024-03-06T07:31:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.883+0000 7f00f5000640 1 -- 172.21.15.32:0/2455616512 wait complete. 2024-03-06T07:31:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.885+0000 7f00f5000640 1 Processor -- start 2024-03-06T07:31:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.885+0000 7f00f5000640 1 -- start start 2024-03-06T07:31:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.886+0000 7f00f5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f00b2080 0x7f00f01f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.886+0000 7f00f5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f01430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00f5000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f013ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00f5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0145810 con 0x7f00f0145430 2024-03-06T07:31:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00f5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f015d0d0 con 0x7f00f00b2080 2024-03-06T07:31:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00f5000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0145990 con 0x7f00f015cc90 2024-03-06T07:31:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f01430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f01430f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59148/0 (socket says 172.21.15.32:59148) 2024-03-06T07:31:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 -- 172.21.15.32:0/3829404792 learned_addr learned my addr 172.21.15.32:0/3829404792 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00efe00640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f013ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00ef400640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f00b2080 0x7f00f01f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 -- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 msgr2=0x7f00f013ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f013ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 -- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f00b2080 msgr2=0x7f00f01f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.887+0000 7f00eea00640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f00b2080 0x7f00f01f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.888+0000 7f00eea00640 1 -- 172.21.15.32:0/3829404792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f01f4d40 con 0x7f00f0145430 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.888+0000 7f00eea00640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f01430f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00d5f0 tx=0x7f00dc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.889+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0125f0 con 0x7f00f0145430 2024-03-06T07:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.889+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f0142cc0 con 0x7f00f0145430 2024-03-06T07:31:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.889+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f01f4410 con 0x7f00f0145430 2024-03-06T07:31:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.889+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00dc019070 con 0x7f00f0145430 2024-03-06T07:31:42.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.889+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0125f0 con 0x7f00f0145430 2024-03-06T07:31:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.891+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c8005290 con 0x7f00f0145430 2024-03-06T07:31:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.891+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00dc012790 con 0x7f00f0145430 2024-03-06T07:31:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.892+0000 7f00e7400640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00d006b6d0 0x7f00d006db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.892+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00dc08a1f0 con 0x7f00f0145430 2024-03-06T07:31:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.893+0000 7f00ef400640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00d006b6d0 0x7f00d006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.893+0000 7f00ef400640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00d006b6d0 0x7f00d006db90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f00e0036000 tx=0x7f00e003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:42.897+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00dc059070 con 0x7f00f0145430 2024-03-06T07:31:43.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.136+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f00c8005850 con 0x7f00f0145430 2024-03-06T07:31:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.782+0000 7f00e7400640 1 -- 172.21.15.32:0/3829404792 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f00dc058970 con 0x7f00f0145430 2024-03-06T07:31:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00d006b6d0 msgr2=0x7f00d006db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00d006b6d0 0x7f00d006db90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f00e0036000 tx=0x7f00e003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 msgr2=0x7f00f01430f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f01430f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00d5f0 tx=0x7f00dc00dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 shutdown_connections 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00d006b6d0 0x7f00d006db90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00f015cc90 0x7f00f013ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00f0145430 0x7f00f01430f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 --2- 172.21.15.32:0/3829404792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f00b2080 0x7f00f01f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 >> 172.21.15.32:0/3829404792 conn(0x7f00f006b3a0 msgr2=0x7f00f00b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 shutdown_connections 2024-03-06T07:31:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:43.786+0000 7f00f5000640 1 -- 172.21.15.32:0/3829404792 wait complete. 2024-03-06T07:31:43.809 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:31:43.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.027+0000 7f7c43000640 1 Processor -- start 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.027+0000 7f7c43000640 1 -- start start 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.028+0000 7f7c43000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7c3c06ed90 0x7f7c3c162600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.028+0000 7f7c43000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3c067030 con 0x7f7c3c161b10 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.028+0000 7f7c43000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3c06f3e0 con 0x7f7c3c0b3960 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.028+0000 7f7c43000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3c06f6c0 con 0x7f7c3c06ed90 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.029+0000 7f7c3be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7c3c161b10 0x7f7c3c161ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39892/0 (socket says 172.21.15.32:39892) 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.029+0000 7f7c3be00640 1 -- 172.21.15.32:0/1301185391 learned_addr learned my addr 172.21.15.32:0/1301185391 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.029+0000 7f7c33400640 1 --2- 172.21.15.32:0/1301185391 >> v2:172.21.15.39:3300/0 conn(0x7f7c3c06ed90 0x7f7c3c162600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.029+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677551015 0 0) 0x7f7c3c067030 con 0x7f7c3c161b10 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c20003520 con 0x7f7c3c161b10 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377648631 0 0) 0x7f7c20003520 con 0x7f7c3c161b10 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 >> v2:172.21.15.39:3300/0 conn(0x7f7c3c06ed90 msgr2=0x7f7c3c162600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 --2- 172.21.15.32:0/1301185391 >> v2:172.21.15.39:3300/0 conn(0x7f7c3c06ed90 0x7f7c3c162600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 >> v1:172.21.15.39:6789/0 conn(0x7f7c3c0b3960 legacy=0x7f7c3c0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c06f9a0 con 0x7f7c3c161b10 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f7c2c0019f0 con 0x7f7c3c161b10 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 >> v1:172.21.15.32:6789/0 conn(0x7f7c3c161b10 legacy=0x7f7c3c161ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200039b0 0x7f7c20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c20004800 0x7f7c20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c200051c0 0x7f7c2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c06f6c0 con 0x7f7c200039b0 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2000c160 con 0x7f7c200051c0 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2000c360 con 0x7f7c20004800 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.030+0000 7f7c3be00640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200039b0 0x7f7c20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c33400640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c20004800 0x7f7c20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c41000640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c200051c0 0x7f7c2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c3be00640 1 -- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c20004800 msgr2=0x7f7c20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c3be00640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c20004800 0x7f7c20004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c3be00640 1 -- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c200051c0 msgr2=0x7f7c2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c3be00640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c200051c0 0x7f7c2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c3be00640 1 -- 172.21.15.32:0/1301185391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c20003520 con 0x7f7c200039b0 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c41000640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c200051c0 0x7f7c2000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.031+0000 7f7c41000640 1 -- 172.21.15.32:0/1301185391 reap_dead start 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.032+0000 7f7c3be00640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200039b0 0x7f7c20003dd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00a850 tx=0x7f7c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6e3ea0d7eadb54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.032+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c00a4a0 con 0x7f7c200039b0 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.032+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c034650 con 0x7f7c200039b0 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.032+0000 7f7c3b400640 1 -- 172.21.15.32:0/1301185391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c034a80 con 0x7f7c200039b0 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 -- 172.21.15.32:0/1301185391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200039b0 msgr2=0x7f7c20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200039b0 0x7f7c20003dd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00a850 tx=0x7f7c20005830 comp rx=0 tx=0).stop 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 -- 172.21.15.32:0/1301185391 shutdown_connections 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 --2- 172.21.15.32:0/1301185391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c200039b0 0x7f7c20003dd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 -- 172.21.15.32:0/1301185391 >> 172.21.15.32:0/1301185391 conn(0x7f7c3c0722d0 msgr2=0x7f7c3c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 -- 172.21.15.32:0/1301185391 shutdown_connections 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.033+0000 7f7c43000640 1 -- 172.21.15.32:0/1301185391 wait complete. 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.035+0000 7f7c43000640 1 Processor -- start 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.035+0000 7f7c43000640 1 -- start start 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c43000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c200039b0 0x7f7c3c1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c43000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c151b70 0x7f7c3c13f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c43000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 0x7f7c3c14f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c43000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1551e0 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c43000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c0b3440 con 0x7f7c3c151b70 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c43000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c155360 con 0x7f7c200039b0 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c41000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 0x7f7c3c14f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c41000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 0x7f7c3c14f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59166/0 (socket says 172.21.15.32:59166) 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c41000640 1 -- 172.21.15.32:0/2722892145 learned_addr learned my addr 172.21.15.32:0/2722892145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c3be00640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c200039b0 0x7f7c3c1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.037+0000 7f7c33400640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c151b70 0x7f7c3c13f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c41000640 1 -- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c200039b0 msgr2=0x7f7c3c1511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c41000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c200039b0 0x7f7c3c1511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c41000640 1 -- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c151b70 msgr2=0x7f7c3c13f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c41000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c151b70 0x7f7c3c13f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c41000640 1 -- 172.21.15.32:0/2722892145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c13cad0 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c41000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 0x7f7c3c14f7c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c06d2b0 tx=0x7f7c3400d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34018070 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c340052f0 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34003830 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.038+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c13cd70 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.039+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c13d330 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.040+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c04f330 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.041+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c3401b070 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.042+0000 7f7c38c00640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7c1806b4d0 0x7f7c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.042+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c34089780 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.042+0000 7f7c3be00640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7c1806b4d0 0x7f7c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.043+0000 7f7c3be00640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7c1806b4d0 0x7f7c1806d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c034000 tx=0x7f7c2c02f000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.047+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c34058560 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.290+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7c3c160f70 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.792+0000 7f7c38c00640 1 -- 172.21.15.32:0/2722892145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7c34057e60 con 0x7f7c3c0cbec0 2024-03-06T07:31:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.796+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7c1806b4d0 msgr2=0x7f7c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.796+0000 7f7c43000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7c1806b4d0 0x7f7c1806d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c034000 tx=0x7f7c2c02f000 comp rx=0 tx=0).stop 2024-03-06T07:31:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.796+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 msgr2=0x7f7c3c14f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.796+0000 7f7c43000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 0x7f7c3c14f7c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c06d2b0 tx=0x7f7c3400d5e0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 shutdown_connections 2024-03-06T07:31:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7c1806b4d0 0x7f7c1806d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3c0cbec0 0x7f7c3c14f7c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c151b70 0x7f7c3c13f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 --2- 172.21.15.32:0/2722892145 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c200039b0 0x7f7c3c1511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 >> 172.21.15.32:0/2722892145 conn(0x7f7c3c0722d0 msgr2=0x7f7c3c06be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 shutdown_connections 2024-03-06T07:31:44.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:44.797+0000 7f7c43000640 1 -- 172.21.15.32:0/2722892145 wait complete. 2024-03-06T07:31:44.828 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:31:44.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:45.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.041+0000 7fc41e800640 1 Processor -- start 2024-03-06T07:31:45.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.042+0000 7fc41e800640 1 -- start start 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4180cd060 0x7fc41815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41e800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41815ed10 con 0x7fc4181552b0 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41e800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41815eff0 con 0x7fc41815e810 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41e800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41815f2d0 con 0x7fc4180cd060 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41d400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc4181552b0 0x7fc4180cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39908/0 (socket says 172.21.15.32:39908) 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41d400640 1 -- 172.21.15.32:0/1411612531 learned_addr learned my addr 172.21.15.32:0/1411612531 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc41ca00640 1 --2- 172.21.15.32:0/1411612531 >> v2:172.21.15.32:3300/0 conn(0x7fc4180cd060 0x7fc41815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.045+0000 7fc417e00640 1 -- 172.21.15.32:0/1411612531 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645765731 0 0) 0x7fc41815ed10 con 0x7fc4181552b0 2024-03-06T07:31:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.046+0000 7fc417e00640 1 -- 172.21.15.32:0/1411612531 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3fc003520 con 0x7fc4181552b0 2024-03-06T07:31:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.046+0000 7fc41ca00640 1 -- 172.21.15.32:0/1411612531 >> v1:172.21.15.39:6789/0 conn(0x7fc41815e810 legacy=0x7fc4180b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.046+0000 7fc41ca00640 1 -- 172.21.15.32:0/1411612531 >> v1:172.21.15.32:6789/0 conn(0x7fc4181552b0 legacy=0x7fc4180cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.046+0000 7fc41ca00640 1 -- 172.21.15.32:0/1411612531 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41815f5b0 con 0x7fc4180cd060 2024-03-06T07:31:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.047+0000 7fc41ca00640 1 --2- 172.21.15.32:0/1411612531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4180cd060 0x7fc41815b1d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc4040097a0 tx=0x7fc40402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca3824492ef7393 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.047+0000 7fc417e00640 1 -- 172.21.15.32:0/1411612531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404035020 con 0x7fc4180cd060 2024-03-06T07:31:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.047+0000 7fc417e00640 1 -- 172.21.15.32:0/1411612531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404031830 con 0x7fc4180cd060 2024-03-06T07:31:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.047+0000 7fc417e00640 1 -- 172.21.15.32:0/1411612531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404031be0 con 0x7fc4180cd060 2024-03-06T07:31:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 -- 172.21.15.32:0/1411612531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4180cd060 msgr2=0x7fc41815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 --2- 172.21.15.32:0/1411612531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4180cd060 0x7fc41815b1d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc4040097a0 tx=0x7fc40402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 -- 172.21.15.32:0/1411612531 shutdown_connections 2024-03-06T07:31:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 --2- 172.21.15.32:0/1411612531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4180cd060 0x7fc41815b1d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 -- 172.21.15.32:0/1411612531 >> 172.21.15.32:0/1411612531 conn(0x7fc41806b3d0 msgr2=0x7fc41806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 -- 172.21.15.32:0/1411612531 shutdown_connections 2024-03-06T07:31:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.048+0000 7fc41e800640 1 -- 172.21.15.32:0/1411612531 wait complete. 2024-03-06T07:31:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.050+0000 7fc41e800640 1 Processor -- start 2024-03-06T07:31:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.050+0000 7fc41e800640 1 -- start start 2024-03-06T07:31:45.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.051+0000 7fc41e800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc4180cd060 0x7fc41814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.051+0000 7fc41e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 0x7fc4180b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.051+0000 7fc41e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc41815e810 0x7fc4180b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.051+0000 7fc41e800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41806dd30 con 0x7fc4181552b0 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.051+0000 7fc41e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180b3760 con 0x7fc41815e810 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.051+0000 7fc41e800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41806deb0 con 0x7fc4180cd060 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 0x7fc4180b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 0x7fc4180b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59180/0 (socket says 172.21.15.32:59180) 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 -- 172.21.15.32:0/1599334388 learned_addr learned my addr 172.21.15.32:0/1599334388 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41d400640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc4180cd060 0x7fc41814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 -- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc4180cd060 msgr2=0x7fc41814c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc4180cd060 0x7fc41814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41de00640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc41815e810 0x7fc4180b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 -- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc41815e810 msgr2=0x7fc4180b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc41815e810 0x7fc4180b1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 -- 172.21.15.32:0/1599334388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41815a5d0 con 0x7fc4181552b0 2024-03-06T07:31:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.052+0000 7fc41ca00640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 0x7fc4180b17a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc404000c00 tx=0x7fc4040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.053+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404035070 con 0x7fc4181552b0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.053+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404004230 con 0x7fc4181552b0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.053+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40403a630 con 0x7fc4181552b0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.053+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41815ccb0 con 0x7fc4181552b0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.053+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41815d270 con 0x7fc4181552b0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.054+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41804f330 con 0x7fc4181552b0 2024-03-06T07:31:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.055+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc404038020 con 0x7fc4181552b0 2024-03-06T07:31:45.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.056+0000 7fc415600640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc3f806b520 0x7fc3f806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.056+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4040abe00 con 0x7fc4181552b0 2024-03-06T07:31:45.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.056+0000 7fc41d400640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc3f806b520 0x7fc3f806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.057+0000 7fc41d400640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc3f806b520 0x7fc3f806d9e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc408001560 tx=0x7fc408009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.061+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc41815a5d0 con 0x7fc4181552b0 2024-03-06T07:31:45.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.302+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4180cf4a0 con 0x7fc4181552b0 2024-03-06T07:31:45.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.303+0000 7fc415600640 1 -- 172.21.15.32:0/1599334388 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fc404040070 con 0x7fc4181552b0 2024-03-06T07:31:45.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:45.305 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:31:45.305 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc3f806b520 msgr2=0x7fc3f806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc3f806b520 0x7fc3f806d9e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc408001560 tx=0x7fc408009040 comp rx=0 tx=0).stop 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 msgr2=0x7fc4180b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 0x7fc4180b17a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc404000c00 tx=0x7fc4040046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 shutdown_connections 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc3f806b520 0x7fc3f806d9e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.305+0000 7fc41e800640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc41815e810 0x7fc4180b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.306+0000 7fc41e800640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4181552b0 0x7fc4180b17a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.306+0000 7fc41e800640 1 --2- 172.21.15.32:0/1599334388 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc4180cd060 0x7fc41814c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.306+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 >> 172.21.15.32:0/1599334388 conn(0x7fc41806b3d0 msgr2=0x7fc41806e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.306+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 shutdown_connections 2024-03-06T07:31:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.306+0000 7fc41e800640 1 -- 172.21.15.32:0/1599334388 wait complete. 2024-03-06T07:31:45.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:45.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.536+0000 7f0ea5a00640 1 Processor -- start 2024-03-06T07:31:45.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.536+0000 7f0ea5a00640 1 -- start start 2024-03-06T07:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0ea5a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 0x7f0ea00ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0ea5a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ea00ce9f0 0x7f0ea00ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0ea5a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea0071640 con 0x7f0ea015a550 2024-03-06T07:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0ea5a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea00af6f0 con 0x7f0ea00ce9f0 2024-03-06T07:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0ea5a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0071920 con 0x7f0ea015a930 2024-03-06T07:31:45.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0e9ea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0ea015a550 0x7f0ea00aefe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39910/0 (socket says 172.21.15.32:39910) 2024-03-06T07:31:45.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.538+0000 7f0e9ea00640 1 -- 172.21.15.32:0/3827487442 learned_addr learned my addr 172.21.15.32:0/3827487442 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9e000640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 0x7f0ea00ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9f400640 1 --2- 172.21.15.32:0/3827487442 >> v2:172.21.15.39:3300/0 conn(0x7f0ea00ce9f0 0x7f0ea00ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250845271 0 0) 0x7f0ea0071640 con 0x7f0ea015a550 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e84005730 con 0x7f0ea015a550 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127371630 0 0) 0x7f0e84005730 con 0x7f0ea015a550 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 msgr2=0x7f0ea00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 0x7f0ea00ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 >> v2:172.21.15.39:3300/0 conn(0x7f0ea00ce9f0 msgr2=0x7f0ea00ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 --2- 172.21.15.32:0/3827487442 >> v2:172.21.15.39:3300/0 conn(0x7f0ea00ce9f0 0x7f0ea00ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea0071c00 con 0x7f0ea015a550 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f0e900019f0 con 0x7f0ea015a550 2024-03-06T07:31:45.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 >> v1:172.21.15.32:6789/0 conn(0x7f0ea015a550 legacy=0x7f0ea00aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0e84006050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 0x7f0e84006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.539+0000 7f0e9d600640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e84006f50 0x7f0e840073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00af6f0 con 0x7f0e84005c30 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0071920 con 0x7f0e84006f50 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8400dc50 con 0x7f0ea015a930 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0e84006050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9f400640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e84006f50 0x7f0e840073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9ea00640 1 -- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 msgr2=0x7f0e84006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015a930 0x7f0e84006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9ea00640 1 -- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e84006f50 msgr2=0x7f0e840073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e84006f50 0x7f0e840073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9ea00640 1 -- 172.21.15.32:0/3827487442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84005730 con 0x7f0e84005c30 2024-03-06T07:31:45.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.540+0000 7f0e9f400640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e84006f50 0x7f0e840073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.541+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0e84006050 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000a880 tx=0x7f0e84007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619fdb68857d9217 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.542+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9000a4a0 con 0x7f0e84005c30 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.542+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e90034650 con 0x7f0e84005c30 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.542+0000 7f0e9d600640 1 -- 172.21.15.32:0/3827487442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90034a40 con 0x7f0e84005c30 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.542+0000 7f0ea5a00640 1 -- 172.21.15.32:0/3827487442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 msgr2=0x7f0e84006050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.542+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/3827487442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0e84006050 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000a880 tx=0x7f0e84007a40 comp rx=0 tx=0).stop 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.542+0000 7f0e9f400640 1 -- 172.21.15.32:0/3827487442 reap_dead start 2024-03-06T07:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.543+0000 7f0ea5a00640 1 -- 172.21.15.32:0/3827487442 shutdown_connections 2024-03-06T07:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.543+0000 7f0ea5a00640 1 -- 172.21.15.32:0/3827487442 >> 172.21.15.32:0/3827487442 conn(0x7f0ea006afb0 msgr2=0x7f0ea006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.543+0000 7f0ea5a00640 1 -- 172.21.15.32:0/3827487442 shutdown_connections 2024-03-06T07:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.543+0000 7f0ea5a00640 1 -- 172.21.15.32:0/3827487442 wait complete. 2024-03-06T07:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.544+0000 7f0ea5a00640 1 Processor -- start 2024-03-06T07:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.545+0000 7f0ea5a00640 1 -- start start 2024-03-06T07:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0ea5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0ea015c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0ea5a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015ad90 0x7f0ea00b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0ea5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ea015b210 0x7f0ea00b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0ea5a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00b52b0 con 0x7f0e84005c30 2024-03-06T07:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0ea5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00b5130 con 0x7f0ea015b210 2024-03-06T07:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0ea5a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00cdc20 con 0x7f0ea015ad90 2024-03-06T07:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0ea015c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0ea015c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59192/0 (socket says 172.21.15.32:59192) 2024-03-06T07:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9ea00640 1 -- 172.21.15.32:0/1690997914 learned_addr learned my addr 172.21.15.32:0/1690997914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9ea00640 1 -- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015ad90 msgr2=0x7f0ea00b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015ad90 0x7f0ea00b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9ea00640 1 -- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ea015b210 msgr2=0x7f0ea00b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.546+0000 7f0e9f400640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ea015b210 0x7f0ea00b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ea015b210 0x7f0ea00b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0e9ea00640 1 -- 172.21.15.32:0/1690997914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea00c92f0 con 0x7f0e84005c30 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0e9ea00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0ea015c190 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000a880 tx=0x7f0ea0071810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90046070 con 0x7f0e84005c30 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9003a040 con 0x7f0e84005c30 2024-03-06T07:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90004e30 con 0x7f0e84005c30 2024-03-06T07:31:45.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.547+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea00c95d0 con 0x7f0e84005c30 2024-03-06T07:31:45.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.548+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea00c9b90 con 0x7f0e84005c30 2024-03-06T07:31:45.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.549+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea0069510 con 0x7f0e84005c30 2024-03-06T07:31:45.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.551+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e90034650 con 0x7f0e84005c30 2024-03-06T07:31:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.551+0000 7f0e96a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e7c06b4d0 0x7f0e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.552+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e900ac0f0 con 0x7f0e84005c30 2024-03-06T07:31:45.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.552+0000 7f0e9e000640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e7c06b4d0 0x7f0e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.553+0000 7f0e9e000640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e7c06b4d0 0x7f0e7c06d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0e88006a90 tx=0x7f0e88006a20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.556+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e9007a9a0 con 0x7f0e84005c30 2024-03-06T07:31:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.755+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ea00af230 con 0x7f0e84005c30 2024-03-06T07:31:45.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.757+0000 7f0e96a00640 1 -- 172.21.15.32:0/1690997914 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f0e9007ea40 con 0x7f0e84005c30 2024-03-06T07:31:45.759 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:45.760 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:42.769932+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.759+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e7c06b4d0 msgr2=0x7f0e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.759+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e7c06b4d0 0x7f0e7c06d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0e88006a90 tx=0x7f0e88006a20 comp rx=0 tx=0).stop 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 msgr2=0x7f0ea015c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0ea015c190 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000a880 tx=0x7f0ea0071810 comp rx=0 tx=0).stop 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 shutdown_connections 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e7c06b4d0 0x7f0e7c06d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ea015b210 0x7f0ea00b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ea015ad90 0x7f0ea00b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 --2- 172.21.15.32:0/1690997914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e84005c30 0x7f0ea015c190 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 >> 172.21.15.32:0/1690997914 conn(0x7f0ea006afb0 msgr2=0x7f0ea006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 shutdown_connections 2024-03-06T07:31:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.760+0000 7f0ea5a00640 1 -- 172.21.15.32:0/1690997914 wait complete. 2024-03-06T07:31:45.782 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:31:45.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:45.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.991+0000 7fbbc3000640 1 Processor -- start 2024-03-06T07:31:45.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.991+0000 7fbbc3000640 1 -- start start 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc3000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbbbc0cd060 0x7fbbbc161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc3000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc162000 0x7fbbbc070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc3000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbbc1627f0 con 0x7fbbbc15efc0 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc3000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc15f3a0 con 0x7fbbbc0cd060 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc3000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06d190 con 0x7fbbbc162000 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbbbc0cd060 0x7fbbbc161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbbbc0cd060 0x7fbbbc161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59206/0 (socket says 172.21.15.32:59206) 2024-03-06T07:31:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc1200640 1 -- 172.21.15.32:0/3468501760 learned_addr learned my addr 172.21.15.32:0/3468501760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.993+0000 7fbbc2600640 1 --2- 172.21.15.32:0/3468501760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc162000 0x7fbbbc070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.994+0000 7fbbc1200640 1 -- 172.21.15.32:0/3468501760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc162000 msgr2=0x7fbbbc070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.994+0000 7fbbc1200640 1 --2- 172.21.15.32:0/3468501760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc162000 0x7fbbbc070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.994+0000 7fbbc1200640 1 -- 172.21.15.32:0/3468501760 >> v1:172.21.15.39:6789/0 conn(0x7fbbbc15efc0 legacy=0x7fbbbc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.994+0000 7fbbc1200640 1 -- 172.21.15.32:0/3468501760 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc06c860 con 0x7fbbbc0cd060 2024-03-06T07:31:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.995+0000 7fbbc1200640 1 --2- 172.21.15.32:0/3468501760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc161ac0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009870 tx=0x7fbba802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3c4da0392b16c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.995+0000 7fbbbbe00640 1 -- 172.21.15.32:0/3468501760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8035020 con 0x7fbbbc0cd060 2024-03-06T07:31:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.995+0000 7fbbbbe00640 1 -- 172.21.15.32:0/3468501760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba8031640 con 0x7fbbbc0cd060 2024-03-06T07:31:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.995+0000 7fbbbbe00640 1 -- 172.21.15.32:0/3468501760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba802d040 con 0x7fbbbc0cd060 2024-03-06T07:31:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 -- 172.21.15.32:0/3468501760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 msgr2=0x7fbbbc161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 --2- 172.21.15.32:0/3468501760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc161ac0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbba8009870 tx=0x7fbba802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 -- 172.21.15.32:0/3468501760 shutdown_connections 2024-03-06T07:31:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 --2- 172.21.15.32:0/3468501760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc162000 0x7fbbbc070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 --2- 172.21.15.32:0/3468501760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc161ac0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 -- 172.21.15.32:0/3468501760 >> 172.21.15.32:0/3468501760 conn(0x7fbbbc06b3d0 msgr2=0x7fbbbc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 -- 172.21.15.32:0/3468501760 shutdown_connections 2024-03-06T07:31:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.996+0000 7fbbc3000640 1 -- 172.21.15.32:0/3468501760 wait complete. 2024-03-06T07:31:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.998+0000 7fbbc3000640 1 Processor -- start 2024-03-06T07:31:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.998+0000 7fbbc3000640 1 -- start start 2024-03-06T07:31:46.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc3000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc15d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc3000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc15efc0 0x7fbbbc137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc3000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbbc162000 0x7fbbbc15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc3000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc1624b0 con 0x7fbbbc0cd060 2024-03-06T07:31:46.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc3000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06ce20 con 0x7fbbbc162000 2024-03-06T07:31:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc3000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06d620 con 0x7fbbbc15efc0 2024-03-06T07:31:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc15d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc15d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59214/0 (socket says 172.21.15.32:59214) 2024-03-06T07:31:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc1c00640 1 -- 172.21.15.32:0/2979910948 learned_addr learned my addr 172.21.15.32:0/2979910948 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:45.999+0000 7fbbc2600640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbbc162000 0x7fbbbc15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1200640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc15efc0 0x7fbbbc137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1c00640 1 -- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc15efc0 msgr2=0x7fbbbc137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1c00640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc15efc0 0x7fbbbc137890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1c00640 1 -- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbbc162000 msgr2=0x7fbbbc15bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1c00640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbbc162000 0x7fbbbc15bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1c00640 1 -- 172.21.15.32:0/2979910948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc15c130 con 0x7fbbbc0cd060 2024-03-06T07:31:46.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc2600640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbbc162000 0x7fbbbc15bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc1c00640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc15d480 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000c430 tx=0x7fbbb000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0011790 con 0x7fbbbc0cd060 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb0017070 con 0x7fbbbc0cd060 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0011bb0 con 0x7fbbbc0cd060 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.000+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc15c3e0 con 0x7fbbbc0cd060 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.001+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc15c9a0 con 0x7fbbbc0cd060 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.002+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc04f330 con 0x7fbbbc0cd060 2024-03-06T07:31:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.003+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbb001c460 con 0x7fbbbc0cd060 2024-03-06T07:31:46.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.004+0000 7fbbb9600640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.004+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbb008a100 con 0x7fbbbc0cd060 2024-03-06T07:31:46.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.004+0000 7fbbc1200640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.005+0000 7fbbc1200640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbba8036000 tx=0x7fbba8008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.008+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbb0058ee0 con 0x7fbbbc0cd060 2024-03-06T07:31:46.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.206+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbbbc0ae450 con 0x7fbb9c06b4d0 2024-03-06T07:31:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.225+0000 7fbbb9600640 1 -- 172.21.15.32:0/2979910948 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251950 (secure 0 0 0) 0x7fbbbc0ae450 con 0x7fbb9c06b4d0 2024-03-06T07:31:46.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:46.233 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:31:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.232+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb9c06b4d0 msgr2=0x7fbb9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.232+0000 7fbbc3000640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbba8036000 tx=0x7fbba8008030 comp rx=0 tx=0).stop 2024-03-06T07:31:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.232+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 msgr2=0x7fbbbc15d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.232+0000 7fbbc3000640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc15d480 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000c430 tx=0x7fbbb000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 shutdown_connections 2024-03-06T07:31:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb9c06b4d0 0x7fbb9c06d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbbc162000 0x7fbbbc15bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbbbc15efc0 0x7fbbbc137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 --2- 172.21.15.32:0/2979910948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc0cd060 0x7fbbbc15d480 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 >> 172.21.15.32:0/2979910948 conn(0x7fbbbc06b3d0 msgr2=0x7fbbbc0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 shutdown_connections 2024-03-06T07:31:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.233+0000 7fbbc3000640 1 -- 172.21.15.32:0/2979910948 wait complete. 2024-03-06T07:31:46.255 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-03-06T07:31:44.235003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255996,"kb_used_data":6532,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718724,"statfs":{"total":773094113280,"available":772831973376,"internally_reserved":0,"allocated":6688768,"data_stored":3430416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.517467"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890485+0000","last_change":"2024-03-06T07:31:41.795047+0000","last_active":"2024-03-06T07:31:42.890485+0000","last_peered":"2024-03-06T07:31:42.890485+0000","last_clean":"2024-03-06T07:31:42.890485+0000","last_became_active":"2024-03-06T07:31:41.794710+0000","last_became_peered":"2024-03-06T07:31:41.794710+0000","last_unstale":"2024-03-06T07:31:42.890485+0000","last_undegraded":"2024-03-06T07:31:42.890485+0000","last_fullsized":"2024-03-06T07:31:42.890485+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:42.409543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890824+0000","last_change":"2024-03-06T07:31:41.769224+0000","last_active":"2024-03-06T07:31:42.890824+0000","last_peered":"2024-03-06T07:31:42.890824+0000","last_clean":"2024-03-06T07:31:42.890824+0000","last_became_active":"2024-03-06T07:31:41.768448+0000","last_became_peered":"2024-03-06T07:31:41.768448+0000","last_unstale":"2024-03-06T07:31:42.890824+0000","last_undegraded":"2024-03-06T07:31:42.890824+0000","last_fullsized":"2024-03-06T07:31:42.890824+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:45.590263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894674+0000","last_change":"2024-03-06T07:31:41.786519+0000","last_active":"2024-03-06T07:31:42.894674+0000","last_peered":"2024-03-06T07:31:42.894674+0000","last_clean":"2024-03-06T07:31:42.894674+0000","last_became_active":"2024-03-06T07:31:41.784440+0000","last_became_peered":"2024-03-06T07:31:41.784440+0000","last_unstale":"2024-03-06T07:31:42.894674+0000","last_undegraded":"2024-03-06T07:31:42.894674+0000","last_fullsized":"2024-03-06T07:31:42.894674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:10.283128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893260+0000","last_change":"2024-03-06T07:31:41.787363+0000","last_active":"2024-03-06T07:31:42.893260+0000","last_peered":"2024-03-06T07:31:42.893260+0000","last_clean":"2024-03-06T07:31:42.893260+0000","last_became_active":"2024-03-06T07:31:41.787017+0000","last_became_peered":"2024-03-06T07:31:41.787017+0000","last_unstale":"2024-03-06T07:31:42.893260+0000","last_undegraded":"2024-03-06T07:31:42.893260+0000","last_fullsized":"2024-03-06T07:31:42.893260+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:42.018523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890823+0000","last_change":"2024-03-06T07:31:41.794850+0000","last_active":"2024-03-06T07:31:42.890823+0000","last_peered":"2024-03-06T07:31:42.890823+0000","last_clean":"2024-03-06T07:31:42.890823+0000","last_became_active":"2024-03-06T07:31:41.794576+0000","last_became_peered":"2024-03-06T07:31:41.794576+0000","last_unstale":"2024-03-06T07:31:42.890823+0000","last_undegraded":"2024-03-06T07:31:42.890823+0000","last_fullsized":"2024-03-06T07:31:42.890823+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:27.216132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811453+0000","last_change":"2024-03-06T07:31:41.798405+0000","last_active":"2024-03-06T07:31:42.811453+0000","last_peered":"2024-03-06T07:31:42.811453+0000","last_clean":"2024-03-06T07:31:42.811453+0000","last_became_active":"2024-03-06T07:31:41.790253+0000","last_became_peered":"2024-03-06T07:31:41.790253+0000","last_unstale":"2024-03-06T07:31:42.811453+0000","last_undegraded":"2024-03-06T07:31:42.811453+0000","last_fullsized":"2024-03-06T07:31:42.811453+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:44.511763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.787979+0000","last_change":"2024-03-06T07:31:41.794884+0000","last_active":"2024-03-06T07:31:42.787979+0000","last_peered":"2024-03-06T07:31:42.787979+0000","last_clean":"2024-03-06T07:31:42.787979+0000","last_became_active":"2024-03-06T07:31:41.794499+0000","last_became_peered":"2024-03-06T07:31:41.794499+0000","last_unstale":"2024-03-06T07:31:42.787979+0000","last_undegraded":"2024-03-06T07:31:42.787979+0000","last_fullsized":"2024-03-06T07:31:42.787979+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:15.108357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890405+0000","last_change":"2024-03-06T07:31:41.781988+0000","last_active":"2024-03-06T07:31:42.890405+0000","last_peered":"2024-03-06T07:31:42.890405+0000","last_clean":"2024-03-06T07:31:42.890405+0000","last_became_active":"2024-03-06T07:31:41.781141+0000","last_became_peered":"2024-03-06T07:31:41.781141+0000","last_unstale":"2024-03-06T07:31:42.890405+0000","last_undegraded":"2024-03-06T07:31:42.890405+0000","last_fullsized":"2024-03-06T07:31:42.890405+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:09.871916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788421+0000","last_change":"2024-03-06T07:31:41.790227+0000","last_active":"2024-03-06T07:31:42.788421+0000","last_peered":"2024-03-06T07:31:42.788421+0000","last_clean":"2024-03-06T07:31:42.788421+0000","last_became_active":"2024-03-06T07:31:41.789927+0000","last_became_peered":"2024-03-06T07:31:41.789927+0000","last_unstale":"2024-03-06T07:31:42.788421+0000","last_undegraded":"2024-03-06T07:31:42.788421+0000","last_fullsized":"2024-03-06T07:31:42.788421+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:41.729273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.784834+0000","last_change":"2024-03-06T07:31:41.776267+0000","last_active":"2024-03-06T07:31:42.784834+0000","last_peered":"2024-03-06T07:31:42.784834+0000","last_clean":"2024-03-06T07:31:42.784834+0000","last_became_active":"2024-03-06T07:31:41.775999+0000","last_became_peered":"2024-03-06T07:31:41.775999+0000","last_unstale":"2024-03-06T07:31:42.784834+0000","last_undegraded":"2024-03-06T07:31:42.784834+0000","last_fullsized":"2024-03-06T07:31:42.784834+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:28.862885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783961+0000","last_change":"2024-03-06T07:31:41.781701+0000","last_active":"2024-03-06T07:31:42.783961+0000","last_peered":"2024-03-06T07:31:42.783961+0000","last_clean":"2024-03-06T07:31:42.783961+0000","last_became_active":"2024-03-06T07:31:41.781404+0000","last_became_peered":"2024-03-06T07:31:41.781404+0000","last_unstale":"2024-03-06T07:31:42.783961+0000","last_undegraded":"2024-03-06T07:31:42.783961+0000","last_fullsized":"2024-03-06T07:31:42.783961+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:40.937673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893714+0000","last_change":"2024-03-06T07:31:41.783338+0000","last_active":"2024-03-06T07:31:42.893714+0000","last_peered":"2024-03-06T07:31:42.893714+0000","last_clean":"2024-03-06T07:31:42.893714+0000","last_became_active":"2024-03-06T07:31:41.781793+0000","last_became_peered":"2024-03-06T07:31:41.781793+0000","last_unstale":"2024-03-06T07:31:42.893714+0000","last_undegraded":"2024-03-06T07:31:42.893714+0000","last_fullsized":"2024-03-06T07:31:42.893714+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:16.534676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813499+0000","last_change":"2024-03-06T07:31:41.788894+0000","last_active":"2024-03-06T07:31:42.813499+0000","last_peered":"2024-03-06T07:31:42.813499+0000","last_clean":"2024-03-06T07:31:42.813499+0000","last_became_active":"2024-03-06T07:31:41.788604+0000","last_became_peered":"2024-03-06T07:31:41.788604+0000","last_unstale":"2024-03-06T07:31:42.813499+0000","last_undegraded":"2024-03-06T07:31:42.813499+0000","last_fullsized":"2024-03-06T07:31:42.813499+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:24.910268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788033+0000","last_change":"2024-03-06T07:31:41.786917+0000","last_active":"2024-03-06T07:31:42.788033+0000","last_peered":"2024-03-06T07:31:42.788033+0000","last_clean":"2024-03-06T07:31:42.788033+0000","last_became_active":"2024-03-06T07:31:41.786609+0000","last_became_peered":"2024-03-06T07:31:41.786609+0000","last_unstale":"2024-03-06T07:31:42.788033+0000","last_undegraded":"2024-03-06T07:31:42.788033+0000","last_fullsized":"2024-03-06T07:31:42.788033+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:12.014768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788362+0000","last_change":"2024-03-06T07:31:41.795565+0000","last_active":"2024-03-06T07:31:42.788362+0000","last_peered":"2024-03-06T07:31:42.788362+0000","last_clean":"2024-03-06T07:31:42.788362+0000","last_became_active":"2024-03-06T07:31:41.795207+0000","last_became_peered":"2024-03-06T07:31:41.795207+0000","last_unstale":"2024-03-06T07:31:42.788362+0000","last_undegraded":"2024-03-06T07:31:42.788362+0000","last_fullsized":"2024-03-06T07:31:42.788362+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:37.647144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811211+0000","last_change":"2024-03-06T07:31:41.790024+0000","last_active":"2024-03-06T07:31:42.811211+0000","last_peered":"2024-03-06T07:31:42.811211+0000","last_clean":"2024-03-06T07:31:42.811211+0000","last_became_active":"2024-03-06T07:31:41.789571+0000","last_became_peered":"2024-03-06T07:31:41.789571+0000","last_unstale":"2024-03-06T07:31:42.811211+0000","last_undegraded":"2024-03-06T07:31:42.811211+0000","last_fullsized":"2024-03-06T07:31:42.811211+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:34.540911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891174+0000","last_change":"2024-03-06T07:31:41.779510+0000","last_active":"2024-03-06T07:31:42.891174+0000","last_peered":"2024-03-06T07:31:42.891174+0000","last_clean":"2024-03-06T07:31:42.891174+0000","last_became_active":"2024-03-06T07:31:41.779129+0000","last_became_peered":"2024-03-06T07:31:41.779129+0000","last_unstale":"2024-03-06T07:31:42.891174+0000","last_undegraded":"2024-03-06T07:31:42.891174+0000","last_fullsized":"2024-03-06T07:31:42.891174+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:06.989000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895393+0000","last_change":"2024-03-06T07:31:41.791169+0000","last_active":"2024-03-06T07:31:42.895393+0000","last_peered":"2024-03-06T07:31:42.895393+0000","last_clean":"2024-03-06T07:31:42.895393+0000","last_became_active":"2024-03-06T07:31:41.790861+0000","last_became_peered":"2024-03-06T07:31:41.790861+0000","last_unstale":"2024-03-06T07:31:42.895393+0000","last_undegraded":"2024-03-06T07:31:42.895393+0000","last_fullsized":"2024-03-06T07:31:42.895393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:18.825154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890058+0000","last_change":"2024-03-06T07:31:41.797664+0000","last_active":"2024-03-06T07:31:42.890058+0000","last_peered":"2024-03-06T07:31:42.890058+0000","last_clean":"2024-03-06T07:31:42.890058+0000","last_became_active":"2024-03-06T07:31:41.797057+0000","last_became_peered":"2024-03-06T07:31:41.797057+0000","last_unstale":"2024-03-06T07:31:42.890058+0000","last_undegraded":"2024-03-06T07:31:42.890058+0000","last_fullsized":"2024-03-06T07:31:42.890058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:42.962676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890183+0000","last_change":"2024-03-06T07:31:41.790447+0000","last_active":"2024-03-06T07:31:42.890183+0000","last_peered":"2024-03-06T07:31:42.890183+0000","last_clean":"2024-03-06T07:31:42.890183+0000","last_became_active":"2024-03-06T07:31:41.790126+0000","last_became_peered":"2024-03-06T07:31:41.790126+0000","last_unstale":"2024-03-06T07:31:42.890183+0000","last_undegraded":"2024-03-06T07:31:42.890183+0000","last_fullsized":"2024-03-06T07:31:42.890183+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:19.134708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895206+0000","last_change":"2024-03-06T07:31:41.783119+0000","last_active":"2024-03-06T07:31:42.895206+0000","last_peered":"2024-03-06T07:31:42.895206+0000","last_clean":"2024-03-06T07:31:42.895206+0000","last_became_active":"2024-03-06T07:31:41.782828+0000","last_became_peered":"2024-03-06T07:31:41.782828+0000","last_unstale":"2024-03-06T07:31:42.895206+0000","last_undegraded":"2024-03-06T07:31:42.895206+0000","last_fullsized":"2024-03-06T07:31:42.895206+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:56.017501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891444+0000","last_change":"2024-03-06T07:31:41.790165+0000","last_active":"2024-03-06T07:31:42.891444+0000","last_peered":"2024-03-06T07:31:42.891444+0000","last_clean":"2024-03-06T07:31:42.891444+0000","last_became_active":"2024-03-06T07:31:41.786415+0000","last_became_peered":"2024-03-06T07:31:41.786415+0000","last_unstale":"2024-03-06T07:31:42.891444+0000","last_undegraded":"2024-03-06T07:31:42.891444+0000","last_fullsized":"2024-03-06T07:31:42.891444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:21.410115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893750+0000","last_change":"2024-03-06T07:31:41.782338+0000","last_active":"2024-03-06T07:31:42.893750+0000","last_peered":"2024-03-06T07:31:42.893750+0000","last_clean":"2024-03-06T07:31:42.893750+0000","last_became_active":"2024-03-06T07:31:41.782059+0000","last_became_peered":"2024-03-06T07:31:41.782059+0000","last_unstale":"2024-03-06T07:31:42.893750+0000","last_undegraded":"2024-03-06T07:31:42.893750+0000","last_fullsized":"2024-03-06T07:31:42.893750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:39.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813187+0000","last_change":"2024-03-06T07:31:41.788188+0000","last_active":"2024-03-06T07:31:42.813187+0000","last_peered":"2024-03-06T07:31:42.813187+0000","last_clean":"2024-03-06T07:31:42.813187+0000","last_became_active":"2024-03-06T07:31:41.787880+0000","last_became_peered":"2024-03-06T07:31:41.787880+0000","last_unstale":"2024-03-06T07:31:42.813187+0000","last_undegraded":"2024-03-06T07:31:42.813187+0000","last_fullsized":"2024-03-06T07:31:42.813187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:51.894909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890319+0000","last_change":"2024-03-06T07:31:41.802411+0000","last_active":"2024-03-06T07:31:42.890319+0000","last_peered":"2024-03-06T07:31:42.890319+0000","last_clean":"2024-03-06T07:31:42.890319+0000","last_became_active":"2024-03-06T07:31:41.802090+0000","last_became_peered":"2024-03-06T07:31:41.802090+0000","last_unstale":"2024-03-06T07:31:42.890319+0000","last_undegraded":"2024-03-06T07:31:42.890319+0000","last_fullsized":"2024-03-06T07:31:42.890319+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:42.473440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895168+0000","last_change":"2024-03-06T07:31:41.790330+0000","last_active":"2024-03-06T07:31:42.895168+0000","last_peered":"2024-03-06T07:31:42.895168+0000","last_clean":"2024-03-06T07:31:42.895168+0000","last_became_active":"2024-03-06T07:31:41.788392+0000","last_became_peered":"2024-03-06T07:31:41.788392+0000","last_unstale":"2024-03-06T07:31:42.895168+0000","last_undegraded":"2024-03-06T07:31:42.895168+0000","last_fullsized":"2024-03-06T07:31:42.895168+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:29.826508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811770+0000","last_change":"2024-03-06T07:31:41.799261+0000","last_active":"2024-03-06T07:31:42.811770+0000","last_peered":"2024-03-06T07:31:42.811770+0000","last_clean":"2024-03-06T07:31:42.811770+0000","last_became_active":"2024-03-06T07:31:41.798965+0000","last_became_peered":"2024-03-06T07:31:41.798965+0000","last_unstale":"2024-03-06T07:31:42.811770+0000","last_undegraded":"2024-03-06T07:31:42.811770+0000","last_fullsized":"2024-03-06T07:31:42.811770+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:44.734632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890896+0000","last_change":"2024-03-06T07:31:41.794429+0000","last_active":"2024-03-06T07:31:42.890896+0000","last_peered":"2024-03-06T07:31:42.890896+0000","last_clean":"2024-03-06T07:31:42.890896+0000","last_became_active":"2024-03-06T07:31:41.793852+0000","last_became_peered":"2024-03-06T07:31:41.793852+0000","last_unstale":"2024-03-06T07:31:42.890896+0000","last_undegraded":"2024-03-06T07:31:42.890896+0000","last_fullsized":"2024-03-06T07:31:42.890896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:25.728984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890826+0000","last_change":"2024-03-06T07:31:41.791402+0000","last_active":"2024-03-06T07:31:42.890826+0000","last_peered":"2024-03-06T07:31:42.890826+0000","last_clean":"2024-03-06T07:31:42.890826+0000","last_became_active":"2024-03-06T07:31:41.789373+0000","last_became_peered":"2024-03-06T07:31:41.789373+0000","last_unstale":"2024-03-06T07:31:42.890826+0000","last_undegraded":"2024-03-06T07:31:42.890826+0000","last_fullsized":"2024-03-06T07:31:42.890826+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.699678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893545+0000","last_change":"2024-03-06T07:31:41.778266+0000","last_active":"2024-03-06T07:31:42.893545+0000","last_peered":"2024-03-06T07:31:42.893545+0000","last_clean":"2024-03-06T07:31:42.893545+0000","last_became_active":"2024-03-06T07:31:41.777939+0000","last_became_peered":"2024-03-06T07:31:41.777939+0000","last_unstale":"2024-03-06T07:31:42.893545+0000","last_undegraded":"2024-03-06T07:31:42.893545+0000","last_fullsized":"2024-03-06T07:31:42.893545+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:34.540911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.889983+0000","last_change":"2024-03-06T07:31:41.783409+0000","last_active":"2024-03-06T07:31:42.889983+0000","last_peered":"2024-03-06T07:31:42.889983+0000","last_clean":"2024-03-06T07:31:42.889983+0000","last_became_active":"2024-03-06T07:31:41.783064+0000","last_became_peered":"2024-03-06T07:31:41.783064+0000","last_unstale":"2024-03-06T07:31:42.889983+0000","last_undegraded":"2024-03-06T07:31:42.889983+0000","last_fullsized":"2024-03-06T07:31:42.889983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:33.597292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811271+0000","last_change":"2024-03-06T07:31:41.807836+0000","last_active":"2024-03-06T07:31:42.811271+0000","last_peered":"2024-03-06T07:31:42.811271+0000","last_clean":"2024-03-06T07:31:42.811271+0000","last_became_active":"2024-03-06T07:31:41.807429+0000","last_became_peered":"2024-03-06T07:31:41.807429+0000","last_unstale":"2024-03-06T07:31:42.811271+0000","last_undegraded":"2024-03-06T07:31:42.811271+0000","last_fullsized":"2024-03-06T07:31:42.811271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:39.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812544+0000","last_change":"2024-03-06T07:31:41.792357+0000","last_active":"2024-03-06T07:31:42.812544+0000","last_peered":"2024-03-06T07:31:42.812544+0000","last_clean":"2024-03-06T07:31:42.812544+0000","last_became_active":"2024-03-06T07:31:41.791835+0000","last_became_peered":"2024-03-06T07:31:41.791835+0000","last_unstale":"2024-03-06T07:31:42.812544+0000","last_undegraded":"2024-03-06T07:31:42.812544+0000","last_fullsized":"2024-03-06T07:31:42.812544+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:56.017501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812044+0000","last_change":"2024-03-06T07:31:40.770850+0000","last_active":"2024-03-06T07:31:42.812044+0000","last_peered":"2024-03-06T07:31:42.812044+0000","last_clean":"2024-03-06T07:31:42.812044+0000","last_became_active":"2024-03-06T07:31:40.770586+0000","last_became_peered":"2024-03-06T07:31:40.770586+0000","last_unstale":"2024-03-06T07:31:42.812044+0000","last_undegraded":"2024-03-06T07:31:42.812044+0000","last_fullsized":"2024-03-06T07:31:42.812044+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.633372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783875+0000","last_change":"2024-03-06T07:31:41.778870+0000","last_active":"2024-03-06T07:31:42.783875+0000","last_peered":"2024-03-06T07:31:42.783875+0000","last_clean":"2024-03-06T07:31:42.783875+0000","last_became_active":"2024-03-06T07:31:41.777323+0000","last_became_peered":"2024-03-06T07:31:41.777323+0000","last_unstale":"2024-03-06T07:31:42.783875+0000","last_undegraded":"2024-03-06T07:31:42.783875+0000","last_fullsized":"2024-03-06T07:31:42.783875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:11.356244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890937+0000","last_change":"2024-03-06T07:31:40.758088+0000","last_active":"2024-03-06T07:31:42.890937+0000","last_peered":"2024-03-06T07:31:42.890937+0000","last_clean":"2024-03-06T07:31:42.890937+0000","last_became_active":"2024-03-06T07:31:40.757435+0000","last_became_peered":"2024-03-06T07:31:40.757435+0000","last_unstale":"2024-03-06T07:31:42.890937+0000","last_undegraded":"2024-03-06T07:31:42.890937+0000","last_fullsized":"2024-03-06T07:31:42.890937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:38.627915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811924+0000","last_change":"2024-03-06T07:31:41.807674+0000","last_active":"2024-03-06T07:31:42.811924+0000","last_peered":"2024-03-06T07:31:42.811924+0000","last_clean":"2024-03-06T07:31:42.811924+0000","last_became_active":"2024-03-06T07:31:41.807262+0000","last_became_peered":"2024-03-06T07:31:41.807262+0000","last_unstale":"2024-03-06T07:31:42.811924+0000","last_undegraded":"2024-03-06T07:31:42.811924+0000","last_fullsized":"2024-03-06T07:31:42.811924+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:04.178439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812018+0000","last_change":"2024-03-06T07:31:40.748552+0000","last_active":"2024-03-06T07:31:42.812018+0000","last_peered":"2024-03-06T07:31:42.812018+0000","last_clean":"2024-03-06T07:31:42.812018+0000","last_became_active":"2024-03-06T07:31:40.748253+0000","last_became_peered":"2024-03-06T07:31:40.748253+0000","last_unstale":"2024-03-06T07:31:42.812018+0000","last_undegraded":"2024-03-06T07:31:42.812018+0000","last_fullsized":"2024-03-06T07:31:42.812018+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:08.710620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812081+0000","last_change":"2024-03-06T07:31:41.800906+0000","last_active":"2024-03-06T07:31:42.812081+0000","last_peered":"2024-03-06T07:31:42.812081+0000","last_clean":"2024-03-06T07:31:42.812081+0000","last_became_active":"2024-03-06T07:31:41.800633+0000","last_became_peered":"2024-03-06T07:31:41.800633+0000","last_unstale":"2024-03-06T07:31:42.812081+0000","last_undegraded":"2024-03-06T07:31:42.812081+0000","last_fullsized":"2024-03-06T07:31:42.812081+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:31.417015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894451+0000","last_change":"2024-03-06T07:31:40.764233+0000","last_active":"2024-03-06T07:31:42.894451+0000","last_peered":"2024-03-06T07:31:42.894451+0000","last_clean":"2024-03-06T07:31:42.894451+0000","last_became_active":"2024-03-06T07:31:40.763332+0000","last_became_peered":"2024-03-06T07:31:40.763332+0000","last_unstale":"2024-03-06T07:31:42.894451+0000","last_undegraded":"2024-03-06T07:31:42.894451+0000","last_fullsized":"2024-03-06T07:31:42.894451+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:40.715501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891579+0000","last_change":"2024-03-06T07:31:41.789914+0000","last_active":"2024-03-06T07:31:42.891579+0000","last_peered":"2024-03-06T07:31:42.891579+0000","last_clean":"2024-03-06T07:31:42.891579+0000","last_became_active":"2024-03-06T07:31:41.785474+0000","last_became_peered":"2024-03-06T07:31:41.785474+0000","last_unstale":"2024-03-06T07:31:42.891579+0000","last_undegraded":"2024-03-06T07:31:42.891579+0000","last_fullsized":"2024-03-06T07:31:42.891579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:03.345699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788259+0000","last_change":"2024-03-06T07:31:40.759136+0000","last_active":"2024-03-06T07:31:42.788259+0000","last_peered":"2024-03-06T07:31:42.788259+0000","last_clean":"2024-03-06T07:31:42.788259+0000","last_became_active":"2024-03-06T07:31:40.758817+0000","last_became_peered":"2024-03-06T07:31:40.758817+0000","last_unstale":"2024-03-06T07:31:42.788259+0000","last_undegraded":"2024-03-06T07:31:42.788259+0000","last_fullsized":"2024-03-06T07:31:42.788259+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:39.923901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890602+0000","last_change":"2024-03-06T07:31:41.790980+0000","last_active":"2024-03-06T07:31:42.890602+0000","last_peered":"2024-03-06T07:31:42.890602+0000","last_clean":"2024-03-06T07:31:42.890602+0000","last_became_active":"2024-03-06T07:31:41.788054+0000","last_became_peered":"2024-03-06T07:31:41.788054+0000","last_unstale":"2024-03-06T07:31:42.890602+0000","last_undegraded":"2024-03-06T07:31:42.890602+0000","last_fullsized":"2024-03-06T07:31:42.890602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:59.635567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811533+0000","last_change":"2024-03-06T07:31:40.758941+0000","last_active":"2024-03-06T07:31:42.811533+0000","last_peered":"2024-03-06T07:31:42.811533+0000","last_clean":"2024-03-06T07:31:42.811533+0000","last_became_active":"2024-03-06T07:31:40.758617+0000","last_became_peered":"2024-03-06T07:31:40.758617+0000","last_unstale":"2024-03-06T07:31:42.811533+0000","last_undegraded":"2024-03-06T07:31:42.811533+0000","last_fullsized":"2024-03-06T07:31:42.811533+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:42.963353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813439+0000","last_change":"2024-03-06T07:31:41.778095+0000","last_active":"2024-03-06T07:31:42.813439+0000","last_peered":"2024-03-06T07:31:42.813439+0000","last_clean":"2024-03-06T07:31:42.813439+0000","last_became_active":"2024-03-06T07:31:41.777778+0000","last_became_peered":"2024-03-06T07:31:41.777778+0000","last_unstale":"2024-03-06T07:31:42.813439+0000","last_undegraded":"2024-03-06T07:31:42.813439+0000","last_fullsized":"2024-03-06T07:31:42.813439+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:50.530003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893345+0000","last_change":"2024-03-06T07:31:40.762429+0000","last_active":"2024-03-06T07:31:42.893345+0000","last_peered":"2024-03-06T07:31:42.893345+0000","last_clean":"2024-03-06T07:31:42.893345+0000","last_became_active":"2024-03-06T07:31:40.762145+0000","last_became_peered":"2024-03-06T07:31:40.762145+0000","last_unstale":"2024-03-06T07:31:42.893345+0000","last_undegraded":"2024-03-06T07:31:42.893345+0000","last_fullsized":"2024-03-06T07:31:42.893345+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:11.863510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890572+0000","last_change":"2024-03-06T07:31:41.774233+0000","last_active":"2024-03-06T07:31:42.890572+0000","last_peered":"2024-03-06T07:31:42.890572+0000","last_clean":"2024-03-06T07:31:42.890572+0000","last_became_active":"2024-03-06T07:31:41.770975+0000","last_became_peered":"2024-03-06T07:31:41.770975+0000","last_unstale":"2024-03-06T07:31:42.890572+0000","last_undegraded":"2024-03-06T07:31:42.890572+0000","last_fullsized":"2024-03-06T07:31:42.890572+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:06.436697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.787661+0000","last_change":"2024-03-06T07:31:40.753547+0000","last_active":"2024-03-06T07:31:42.787661+0000","last_peered":"2024-03-06T07:31:42.787661+0000","last_clean":"2024-03-06T07:31:42.787661+0000","last_became_active":"2024-03-06T07:31:40.753129+0000","last_became_peered":"2024-03-06T07:31:40.753129+0000","last_unstale":"2024-03-06T07:31:42.787661+0000","last_undegraded":"2024-03-06T07:31:42.787661+0000","last_fullsized":"2024-03-06T07:31:42.787661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:23.896496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788852+0000","last_change":"2024-03-06T07:31:41.795218+0000","last_active":"2024-03-06T07:31:42.788852+0000","last_peered":"2024-03-06T07:31:42.788852+0000","last_clean":"2024-03-06T07:31:42.788852+0000","last_became_active":"2024-03-06T07:31:41.794885+0000","last_became_peered":"2024-03-06T07:31:41.794885+0000","last_unstale":"2024-03-06T07:31:42.788852+0000","last_undegraded":"2024-03-06T07:31:42.788852+0000","last_fullsized":"2024-03-06T07:31:42.788852+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:09.724393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895065+0000","last_change":"2024-03-06T07:31:40.756304+0000","last_active":"2024-03-06T07:31:42.895065+0000","last_peered":"2024-03-06T07:31:42.895065+0000","last_clean":"2024-03-06T07:31:42.895065+0000","last_became_active":"2024-03-06T07:31:40.755877+0000","last_became_peered":"2024-03-06T07:31:40.755877+0000","last_unstale":"2024-03-06T07:31:42.895065+0000","last_undegraded":"2024-03-06T07:31:42.895065+0000","last_fullsized":"2024-03-06T07:31:42.895065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:40.715501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894800+0000","last_change":"2024-03-06T07:31:41.783826+0000","last_active":"2024-03-06T07:31:42.894800+0000","last_peered":"2024-03-06T07:31:42.894800+0000","last_clean":"2024-03-06T07:31:42.894800+0000","last_became_active":"2024-03-06T07:31:41.783401+0000","last_became_peered":"2024-03-06T07:31:41.783401+0000","last_unstale":"2024-03-06T07:31:42.894800+0000","last_undegraded":"2024-03-06T07:31:42.894800+0000","last_fullsized":"2024-03-06T07:31:42.894800+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:44.734632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890998+0000","last_change":"2024-03-06T07:31:40.753641+0000","last_active":"2024-03-06T07:31:42.890998+0000","last_peered":"2024-03-06T07:31:42.890998+0000","last_clean":"2024-03-06T07:31:42.890998+0000","last_became_active":"2024-03-06T07:31:40.753359+0000","last_became_peered":"2024-03-06T07:31:40.753359+0000","last_unstale":"2024-03-06T07:31:42.890998+0000","last_undegraded":"2024-03-06T07:31:42.890998+0000","last_fullsized":"2024-03-06T07:31:42.890998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:43.497991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783712+0000","last_change":"2024-03-06T07:31:41.788013+0000","last_active":"2024-03-06T07:31:42.783712+0000","last_peered":"2024-03-06T07:31:42.783712+0000","last_clean":"2024-03-06T07:31:42.783712+0000","last_became_active":"2024-03-06T07:31:41.783682+0000","last_became_peered":"2024-03-06T07:31:41.783682+0000","last_unstale":"2024-03-06T07:31:42.783712+0000","last_undegraded":"2024-03-06T07:31:42.783712+0000","last_fullsized":"2024-03-06T07:31:42.783712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:24.966333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895364+0000","last_change":"2024-03-06T07:31:40.749523+0000","last_active":"2024-03-06T07:31:42.895364+0000","last_peered":"2024-03-06T07:31:42.895364+0000","last_clean":"2024-03-06T07:31:42.895364+0000","last_became_active":"2024-03-06T07:31:40.748340+0000","last_became_peered":"2024-03-06T07:31:40.748340+0000","last_unstale":"2024-03-06T07:31:42.895364+0000","last_undegraded":"2024-03-06T07:31:42.895364+0000","last_fullsized":"2024-03-06T07:31:42.895364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:08.710620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895378+0000","last_change":"2024-03-06T07:31:41.786850+0000","last_active":"2024-03-06T07:31:42.895378+0000","last_peered":"2024-03-06T07:31:42.895378+0000","last_clean":"2024-03-06T07:31:42.895378+0000","last_became_active":"2024-03-06T07:31:41.786132+0000","last_became_peered":"2024-03-06T07:31:41.786132+0000","last_unstale":"2024-03-06T07:31:42.895378+0000","last_undegraded":"2024-03-06T07:31:42.895378+0000","last_fullsized":"2024-03-06T07:31:42.895378+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.201570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788696+0000","last_change":"2024-03-06T07:31:40.756763+0000","last_active":"2024-03-06T07:31:42.788696+0000","last_peered":"2024-03-06T07:31:42.788696+0000","last_clean":"2024-03-06T07:31:42.788696+0000","last_became_active":"2024-03-06T07:31:40.756276+0000","last_became_peered":"2024-03-06T07:31:40.756276+0000","last_unstale":"2024-03-06T07:31:42.788696+0000","last_undegraded":"2024-03-06T07:31:42.788696+0000","last_fullsized":"2024-03-06T07:31:42.788696+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:10.342472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892914+0000","last_change":"2024-03-06T07:31:41.796433+0000","last_active":"2024-03-06T07:31:42.892914+0000","last_peered":"2024-03-06T07:31:42.892914+0000","last_clean":"2024-03-06T07:31:42.892914+0000","last_became_active":"2024-03-06T07:31:41.795848+0000","last_became_peered":"2024-03-06T07:31:41.795848+0000","last_unstale":"2024-03-06T07:31:42.892914+0000","last_undegraded":"2024-03-06T07:31:42.892914+0000","last_fullsized":"2024-03-06T07:31:42.892914+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.778914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890695+0000","last_change":"2024-03-06T07:31:40.751197+0000","last_active":"2024-03-06T07:31:42.890695+0000","last_peered":"2024-03-06T07:31:42.890695+0000","last_clean":"2024-03-06T07:31:42.890695+0000","last_became_active":"2024-03-06T07:31:40.750896+0000","last_became_peered":"2024-03-06T07:31:40.750896+0000","last_unstale":"2024-03-06T07:31:42.890695+0000","last_undegraded":"2024-03-06T07:31:42.890695+0000","last_fullsized":"2024-03-06T07:31:42.890695+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:28.812735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894292+0000","last_change":"2024-03-06T07:31:41.778124+0000","last_active":"2024-03-06T07:31:42.894292+0000","last_peered":"2024-03-06T07:31:42.894292+0000","last_clean":"2024-03-06T07:31:42.894292+0000","last_became_active":"2024-03-06T07:31:41.770879+0000","last_became_peered":"2024-03-06T07:31:41.770879+0000","last_unstale":"2024-03-06T07:31:42.894292+0000","last_undegraded":"2024-03-06T07:31:42.894292+0000","last_fullsized":"2024-03-06T07:31:42.894292+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:18.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812952+0000","last_change":"2024-03-06T07:31:40.759869+0000","last_active":"2024-03-06T07:31:42.812952+0000","last_peered":"2024-03-06T07:31:42.812952+0000","last_clean":"2024-03-06T07:31:42.812952+0000","last_became_active":"2024-03-06T07:31:40.759506+0000","last_became_peered":"2024-03-06T07:31:40.759506+0000","last_unstale":"2024-03-06T07:31:42.812952+0000","last_undegraded":"2024-03-06T07:31:42.812952+0000","last_fullsized":"2024-03-06T07:31:42.812952+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:27.345029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.787760+0000","last_change":"2024-03-06T07:31:41.781383+0000","last_active":"2024-03-06T07:31:42.787760+0000","last_peered":"2024-03-06T07:31:42.787760+0000","last_clean":"2024-03-06T07:31:42.787760+0000","last_became_active":"2024-03-06T07:31:41.781039+0000","last_became_peered":"2024-03-06T07:31:41.781039+0000","last_unstale":"2024-03-06T07:31:42.787760+0000","last_undegraded":"2024-03-06T07:31:42.787760+0000","last_fullsized":"2024-03-06T07:31:42.787760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:12.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893621+0000","last_change":"2024-03-06T07:31:40.763977+0000","last_active":"2024-03-06T07:31:42.893621+0000","last_peered":"2024-03-06T07:31:42.893621+0000","last_clean":"2024-03-06T07:31:42.893621+0000","last_became_active":"2024-03-06T07:31:40.763409+0000","last_became_peered":"2024-03-06T07:31:40.763409+0000","last_unstale":"2024-03-06T07:31:42.893621+0000","last_undegraded":"2024-03-06T07:31:42.893621+0000","last_fullsized":"2024-03-06T07:31:42.893621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:14.094585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894020+0000","last_change":"2024-03-06T07:31:41.790503+0000","last_active":"2024-03-06T07:31:42.894020+0000","last_peered":"2024-03-06T07:31:42.894020+0000","last_clean":"2024-03-06T07:31:42.894020+0000","last_became_active":"2024-03-06T07:31:41.790234+0000","last_became_peered":"2024-03-06T07:31:41.790234+0000","last_unstale":"2024-03-06T07:31:42.894020+0000","last_undegraded":"2024-03-06T07:31:42.894020+0000","last_fullsized":"2024-03-06T07:31:42.894020+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:44.511763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892811+0000","last_change":"2024-03-06T07:31:40.765734+0000","last_active":"2024-03-06T07:31:42.892811+0000","last_peered":"2024-03-06T07:31:42.892811+0000","last_clean":"2024-03-06T07:31:42.892811+0000","last_became_active":"2024-03-06T07:31:40.764184+0000","last_became_peered":"2024-03-06T07:31:40.764184+0000","last_unstale":"2024-03-06T07:31:42.892811+0000","last_undegraded":"2024-03-06T07:31:42.892811+0000","last_fullsized":"2024-03-06T07:31:42.892811+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.633372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895136+0000","last_change":"2024-03-06T07:31:41.785800+0000","last_active":"2024-03-06T07:31:42.895136+0000","last_peered":"2024-03-06T07:31:42.895136+0000","last_clean":"2024-03-06T07:31:42.895136+0000","last_became_active":"2024-03-06T07:31:41.785475+0000","last_became_peered":"2024-03-06T07:31:41.785475+0000","last_unstale":"2024-03-06T07:31:42.895136+0000","last_undegraded":"2024-03-06T07:31:42.895136+0000","last_fullsized":"2024-03-06T07:31:42.895136+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:31.417015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891193+0000","last_change":"2024-03-06T07:31:40.753184+0000","last_active":"2024-03-06T07:31:42.891193+0000","last_peered":"2024-03-06T07:31:42.891193+0000","last_clean":"2024-03-06T07:31:42.891193+0000","last_became_active":"2024-03-06T07:31:40.752887+0000","last_became_peered":"2024-03-06T07:31:40.752887+0000","last_unstale":"2024-03-06T07:31:42.891193+0000","last_undegraded":"2024-03-06T07:31:42.891193+0000","last_fullsized":"2024-03-06T07:31:42.891193+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:17.778668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895998+0000","last_change":"2024-03-06T07:31:41.796442+0000","last_active":"2024-03-06T07:31:42.895998+0000","last_peered":"2024-03-06T07:31:42.895998+0000","last_clean":"2024-03-06T07:31:42.895998+0000","last_became_active":"2024-03-06T07:31:41.795853+0000","last_became_peered":"2024-03-06T07:31:41.795853+0000","last_unstale":"2024-03-06T07:31:42.895998+0000","last_undegraded":"2024-03-06T07:31:42.895998+0000","last_fullsized":"2024-03-06T07:31:42.895998+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:34.540911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812232+0000","last_change":"2024-03-06T07:31:40.752399+0000","last_active":"2024-03-06T07:31:42.812232+0000","last_peered":"2024-03-06T07:31:42.812232+0000","last_clean":"2024-03-06T07:31:42.812232+0000","last_became_active":"2024-03-06T07:31:40.752096+0000","last_became_peered":"2024-03-06T07:31:40.752096+0000","last_unstale":"2024-03-06T07:31:42.812232+0000","last_undegraded":"2024-03-06T07:31:42.812232+0000","last_fullsized":"2024-03-06T07:31:42.812232+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:40.715501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890337+0000","last_change":"2024-03-06T07:31:41.780583+0000","last_active":"2024-03-06T07:31:42.890337+0000","last_peered":"2024-03-06T07:31:42.890337+0000","last_clean":"2024-03-06T07:31:42.890337+0000","last_became_active":"2024-03-06T07:31:41.780299+0000","last_became_peered":"2024-03-06T07:31:41.780299+0000","last_unstale":"2024-03-06T07:31:42.890337+0000","last_undegraded":"2024-03-06T07:31:42.890337+0000","last_fullsized":"2024-03-06T07:31:42.890337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:14.927128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895173+0000","last_change":"2024-03-06T07:31:40.764368+0000","last_active":"2024-03-06T07:31:42.895173+0000","last_peered":"2024-03-06T07:31:42.895173+0000","last_clean":"2024-03-06T07:31:42.895173+0000","last_became_active":"2024-03-06T07:31:40.763979+0000","last_became_peered":"2024-03-06T07:31:40.763979+0000","last_unstale":"2024-03-06T07:31:42.895173+0000","last_undegraded":"2024-03-06T07:31:42.895173+0000","last_fullsized":"2024-03-06T07:31:42.895173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:11.000996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895085+0000","last_change":"2024-03-06T07:31:41.786484+0000","last_active":"2024-03-06T07:31:42.895085+0000","last_peered":"2024-03-06T07:31:42.895085+0000","last_clean":"2024-03-06T07:31:42.895085+0000","last_became_active":"2024-03-06T07:31:41.784913+0000","last_became_peered":"2024-03-06T07:31:41.784913+0000","last_unstale":"2024-03-06T07:31:42.895085+0000","last_undegraded":"2024-03-06T07:31:42.895085+0000","last_fullsized":"2024-03-06T07:31:42.895085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:58.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811898+0000","last_change":"2024-03-06T07:31:40.748126+0000","last_active":"2024-03-06T07:31:42.811898+0000","last_peered":"2024-03-06T07:31:42.811898+0000","last_clean":"2024-03-06T07:31:42.811898+0000","last_became_active":"2024-03-06T07:31:40.747842+0000","last_became_peered":"2024-03-06T07:31:40.747842+0000","last_unstale":"2024-03-06T07:31:42.811898+0000","last_undegraded":"2024-03-06T07:31:42.811898+0000","last_fullsized":"2024-03-06T07:31:42.811898+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:10.469237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894168+0000","last_change":"2024-03-06T07:31:41.797089+0000","last_active":"2024-03-06T07:31:42.894168+0000","last_peered":"2024-03-06T07:31:42.894168+0000","last_clean":"2024-03-06T07:31:42.894168+0000","last_became_active":"2024-03-06T07:31:41.796293+0000","last_became_peered":"2024-03-06T07:31:41.796293+0000","last_unstale":"2024-03-06T07:31:42.894168+0000","last_undegraded":"2024-03-06T07:31:42.894168+0000","last_fullsized":"2024-03-06T07:31:42.894168+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:23.360825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891235+0000","last_change":"2024-03-06T07:31:40.760412+0000","last_active":"2024-03-06T07:31:42.891235+0000","last_peered":"2024-03-06T07:31:42.891235+0000","last_clean":"2024-03-06T07:31:42.891235+0000","last_became_active":"2024-03-06T07:31:40.759875+0000","last_became_peered":"2024-03-06T07:31:40.759875+0000","last_unstale":"2024-03-06T07:31:42.891235+0000","last_undegraded":"2024-03-06T07:31:42.891235+0000","last_fullsized":"2024-03-06T07:31:42.891235+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:33.527138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811725+0000","last_change":"2024-03-06T07:31:41.785333+0000","last_active":"2024-03-06T07:31:42.811725+0000","last_peered":"2024-03-06T07:31:42.811725+0000","last_clean":"2024-03-06T07:31:42.811725+0000","last_became_active":"2024-03-06T07:31:41.784980+0000","last_became_peered":"2024-03-06T07:31:41.784980+0000","last_unstale":"2024-03-06T07:31:42.811725+0000","last_undegraded":"2024-03-06T07:31:42.811725+0000","last_fullsized":"2024-03-06T07:31:42.811725+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:10.283128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.784014+0000","last_change":"2024-03-06T07:31:40.758168+0000","last_active":"2024-03-06T07:31:42.784014+0000","last_peered":"2024-03-06T07:31:42.784014+0000","last_clean":"2024-03-06T07:31:42.784014+0000","last_became_active":"2024-03-06T07:31:40.757763+0000","last_became_peered":"2024-03-06T07:31:40.757763+0000","last_unstale":"2024-03-06T07:31:42.784014+0000","last_undegraded":"2024-03-06T07:31:42.784014+0000","last_fullsized":"2024-03-06T07:31:42.784014+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:27.345029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811457+0000","last_change":"2024-03-06T07:31:41.792651+0000","last_active":"2024-03-06T07:31:42.811457+0000","last_peered":"2024-03-06T07:31:42.811457+0000","last_clean":"2024-03-06T07:31:42.811457+0000","last_became_active":"2024-03-06T07:31:41.786755+0000","last_became_peered":"2024-03-06T07:31:41.786755+0000","last_unstale":"2024-03-06T07:31:42.811457+0000","last_undegraded":"2024-03-06T07:31:42.811457+0000","last_fullsized":"2024-03-06T07:31:42.811457+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:18.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892595+0000","last_change":"2024-03-06T07:31:40.763728+0000","last_active":"2024-03-06T07:31:42.892595+0000","last_peered":"2024-03-06T07:31:42.892595+0000","last_clean":"2024-03-06T07:31:42.892595+0000","last_became_active":"2024-03-06T07:31:40.763400+0000","last_became_peered":"2024-03-06T07:31:40.763400+0000","last_unstale":"2024-03-06T07:31:42.892595+0000","last_undegraded":"2024-03-06T07:31:42.892595+0000","last_fullsized":"2024-03-06T07:31:42.892595+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:27.109170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895554+0000","last_change":"2024-03-06T07:31:41.787928+0000","last_active":"2024-03-06T07:31:42.895554+0000","last_peered":"2024-03-06T07:31:42.895554+0000","last_clean":"2024-03-06T07:31:42.895554+0000","last_became_active":"2024-03-06T07:31:41.787610+0000","last_became_peered":"2024-03-06T07:31:41.787610+0000","last_unstale":"2024-03-06T07:31:42.895554+0000","last_undegraded":"2024-03-06T07:31:42.895554+0000","last_fullsized":"2024-03-06T07:31:42.895554+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:00.316298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783985+0000","last_change":"2024-03-06T07:31:40.750338+0000","last_active":"2024-03-06T07:31:42.783985+0000","last_peered":"2024-03-06T07:31:42.783985+0000","last_clean":"2024-03-06T07:31:42.783985+0000","last_became_active":"2024-03-06T07:31:40.749957+0000","last_became_peered":"2024-03-06T07:31:40.749957+0000","last_unstale":"2024-03-06T07:31:42.783985+0000","last_undegraded":"2024-03-06T07:31:42.783985+0000","last_fullsized":"2024-03-06T07:31:42.783985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:25.222365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.896156+0000","last_change":"2024-03-06T07:31:41.789450+0000","last_active":"2024-03-06T07:31:42.896156+0000","last_peered":"2024-03-06T07:31:42.896156+0000","last_clean":"2024-03-06T07:31:42.896156+0000","last_became_active":"2024-03-06T07:31:41.787795+0000","last_became_peered":"2024-03-06T07:31:41.787795+0000","last_unstale":"2024-03-06T07:31:42.896156+0000","last_undegraded":"2024-03-06T07:31:42.896156+0000","last_fullsized":"2024-03-06T07:31:42.896156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:39.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.815305+0000","last_change":"2024-03-06T07:31:40.760633+0000","last_active":"2024-03-06T07:31:42.815305+0000","last_peered":"2024-03-06T07:31:42.815305+0000","last_clean":"2024-03-06T07:31:42.815305+0000","last_became_active":"2024-03-06T07:31:40.760309+0000","last_became_peered":"2024-03-06T07:31:40.760309+0000","last_unstale":"2024-03-06T07:31:42.815305+0000","last_undegraded":"2024-03-06T07:31:42.815305+0000","last_fullsized":"2024-03-06T07:31:42.815305+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:29.515860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.784135+0000","last_change":"2024-03-06T07:31:41.783333+0000","last_active":"2024-03-06T07:31:42.784135+0000","last_peered":"2024-03-06T07:31:42.784135+0000","last_clean":"2024-03-06T07:31:42.784135+0000","last_became_active":"2024-03-06T07:31:41.782435+0000","last_became_peered":"2024-03-06T07:31:41.782435+0000","last_unstale":"2024-03-06T07:31:42.784135+0000","last_undegraded":"2024-03-06T07:31:42.784135+0000","last_fullsized":"2024-03-06T07:31:42.784135+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:26.100782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813687+0000","last_change":"2024-03-06T07:31:40.762392+0000","last_active":"2024-03-06T07:31:42.813687+0000","last_peered":"2024-03-06T07:31:42.813687+0000","last_clean":"2024-03-06T07:31:42.813687+0000","last_became_active":"2024-03-06T07:31:40.762033+0000","last_became_peered":"2024-03-06T07:31:40.762033+0000","last_unstale":"2024-03-06T07:31:42.813687+0000","last_undegraded":"2024-03-06T07:31:42.813687+0000","last_fullsized":"2024-03-06T07:31:42.813687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:25.222365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811949+0000","last_change":"2024-03-06T07:31:41.802844+0000","last_active":"2024-03-06T07:31:42.811949+0000","last_peered":"2024-03-06T07:31:42.811949+0000","last_clean":"2024-03-06T07:31:42.811949+0000","last_became_active":"2024-03-06T07:31:41.800880+0000","last_became_peered":"2024-03-06T07:31:41.800880+0000","last_unstale":"2024-03-06T07:31:42.811949+0000","last_undegraded":"2024-03-06T07:31:42.811949+0000","last_fullsized":"2024-03-06T07:31:42.811949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:58.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894746+0000","last_change":"2024-03-06T07:31:40.767571+0000","last_active":"2024-03-06T07:31:42.894746+0000","last_peered":"2024-03-06T07:31:42.894746+0000","last_clean":"2024-03-06T07:31:42.894746+0000","last_became_active":"2024-03-06T07:31:40.767311+0000","last_became_peered":"2024-03-06T07:31:40.767311+0000","last_unstale":"2024-03-06T07:31:42.894746+0000","last_undegraded":"2024-03-06T07:31:42.894746+0000","last_fullsized":"2024-03-06T07:31:42.894746+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.633372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891712+0000","last_change":"2024-03-06T07:31:41.769366+0000","last_active":"2024-03-06T07:31:42.891712+0000","last_peered":"2024-03-06T07:31:42.891712+0000","last_clean":"2024-03-06T07:31:42.891712+0000","last_became_active":"2024-03-06T07:31:41.768138+0000","last_became_peered":"2024-03-06T07:31:41.768138+0000","last_unstale":"2024-03-06T07:31:42.891712+0000","last_undegraded":"2024-03-06T07:31:42.891712+0000","last_fullsized":"2024-03-06T07:31:42.891712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:56.100684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783458+0000","last_change":"2024-03-06T07:31:40.759529+0000","last_active":"2024-03-06T07:31:42.783458+0000","last_peered":"2024-03-06T07:31:42.783458+0000","last_clean":"2024-03-06T07:31:42.783458+0000","last_became_active":"2024-03-06T07:31:40.759151+0000","last_became_peered":"2024-03-06T07:31:40.759151+0000","last_unstale":"2024-03-06T07:31:42.783458+0000","last_undegraded":"2024-03-06T07:31:42.783458+0000","last_fullsized":"2024-03-06T07:31:42.783458+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:11.001490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894811+0000","last_change":"2024-03-06T07:31:41.789384+0000","last_active":"2024-03-06T07:31:42.894811+0000","last_peered":"2024-03-06T07:31:42.894811+0000","last_clean":"2024-03-06T07:31:42.894811+0000","last_became_active":"2024-03-06T07:31:41.789102+0000","last_became_peered":"2024-03-06T07:31:41.789102+0000","last_unstale":"2024-03-06T07:31:42.894811+0000","last_undegraded":"2024-03-06T07:31:42.894811+0000","last_fullsized":"2024-03-06T07:31:42.894811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:04.178439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788135+0000","last_change":"2024-03-06T07:31:40.761170+0000","last_active":"2024-03-06T07:31:42.788135+0000","last_peered":"2024-03-06T07:31:42.788135+0000","last_clean":"2024-03-06T07:31:42.788135+0000","last_became_active":"2024-03-06T07:31:40.760817+0000","last_became_peered":"2024-03-06T07:31:40.760817+0000","last_unstale":"2024-03-06T07:31:42.788135+0000","last_undegraded":"2024-03-06T07:31:42.788135+0000","last_fullsized":"2024-03-06T07:31:42.788135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:23.952561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783408+0000","last_change":"2024-03-06T07:31:41.778561+0000","last_active":"2024-03-06T07:31:42.783408+0000","last_peered":"2024-03-06T07:31:42.783408+0000","last_clean":"2024-03-06T07:31:42.783408+0000","last_became_active":"2024-03-06T07:31:41.778262+0000","last_became_peered":"2024-03-06T07:31:41.778262+0000","last_unstale":"2024-03-06T07:31:42.783408+0000","last_undegraded":"2024-03-06T07:31:42.783408+0000","last_fullsized":"2024-03-06T07:31:42.783408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:24.910268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783436+0000","last_change":"2024-03-06T07:31:40.751811+0000","last_active":"2024-03-06T07:31:42.783436+0000","last_peered":"2024-03-06T07:31:42.783436+0000","last_clean":"2024-03-06T07:31:42.783436+0000","last_became_active":"2024-03-06T07:31:40.751538+0000","last_became_peered":"2024-03-06T07:31:40.751538+0000","last_unstale":"2024-03-06T07:31:42.783436+0000","last_undegraded":"2024-03-06T07:31:42.783436+0000","last_fullsized":"2024-03-06T07:31:42.783436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:24.715212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890719+0000","last_change":"2024-03-06T07:31:41.773608+0000","last_active":"2024-03-06T07:31:42.890719+0000","last_peered":"2024-03-06T07:31:42.890719+0000","last_clean":"2024-03-06T07:31:42.890719+0000","last_became_active":"2024-03-06T07:31:41.771480+0000","last_became_peered":"2024-03-06T07:31:41.771480+0000","last_unstale":"2024-03-06T07:31:42.890719+0000","last_undegraded":"2024-03-06T07:31:42.890719+0000","last_fullsized":"2024-03-06T07:31:42.890719+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890638+0000","last_change":"2024-03-06T07:31:40.763463+0000","last_active":"2024-03-06T07:31:42.890638+0000","last_peered":"2024-03-06T07:31:42.890638+0000","last_clean":"2024-03-06T07:31:42.890638+0000","last_became_active":"2024-03-06T07:31:40.763190+0000","last_became_peered":"2024-03-06T07:31:40.763190+0000","last_unstale":"2024-03-06T07:31:42.890638+0000","last_undegraded":"2024-03-06T07:31:42.890638+0000","last_fullsized":"2024-03-06T07:31:42.890638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:59.302526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890543+0000","last_change":"2024-03-06T07:31:41.796771+0000","last_active":"2024-03-06T07:31:42.890543+0000","last_peered":"2024-03-06T07:31:42.890543+0000","last_clean":"2024-03-06T07:31:42.890543+0000","last_became_active":"2024-03-06T07:31:41.791064+0000","last_became_peered":"2024-03-06T07:31:41.791064+0000","last_unstale":"2024-03-06T07:31:42.890543+0000","last_undegraded":"2024-03-06T07:31:42.890543+0000","last_fullsized":"2024-03-06T07:31:42.890543+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:49.902893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813108+0000","last_change":"2024-03-06T07:31:40.771153+0000","last_active":"2024-03-06T07:31:42.813108+0000","last_peered":"2024-03-06T07:31:42.813108+0000","last_clean":"2024-03-06T07:31:42.813108+0000","last_became_active":"2024-03-06T07:31:40.770825+0000","last_became_peered":"2024-03-06T07:31:40.770825+0000","last_unstale":"2024-03-06T07:31:42.813108+0000","last_undegraded":"2024-03-06T07:31:42.813108+0000","last_fullsized":"2024-03-06T07:31:42.813108+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:11.001490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892976+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:31:42.892976+0000","last_peered":"2024-03-06T07:31:42.892976+0000","last_clean":"2024-03-06T07:31:42.892976+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:31:42.892976+0000","last_undegraded":"2024-03-06T07:31:42.892976+0000","last_fullsized":"2024-03-06T07:31:42.892976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1286144,"data_stored":873498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":283622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":278282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":276726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":276726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":688128,"data_stored":283060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":875532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:46.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.522+0000 7f9b90a00640 1 Processor -- start 2024-03-06T07:31:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.522+0000 7f9b90a00640 1 -- start start 2024-03-06T07:31:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b90a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 0x7f9b8c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b90a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9b8c15ea80 0x7f9b8c06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b90a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c161ae0 con 0x7f9b8c1625a0 2024-03-06T07:31:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b90a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8c162110 con 0x7f9b8c15ea80 2024-03-06T07:31:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b90a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c15f080 con 0x7f9b8c147400 2024-03-06T07:31:46.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b8aa00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9b8c1625a0 0x7f9b8c15e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39926/0 (socket says 172.21.15.32:39926) 2024-03-06T07:31:46.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b8aa00640 1 -- 172.21.15.32:0/2118705315 learned_addr learned my addr 172.21.15.32:0/2118705315 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:46.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.524+0000 7f9b8b400640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 0x7f9b8c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8be00640 1 --2- 172.21.15.32:0/2118705315 >> v2:172.21.15.39:3300/0 conn(0x7f9b8c15ea80 0x7f9b8c06b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081631154 0 0) 0x7f9b8c161ae0 con 0x7f9b8c1625a0 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b70003520 con 0x7f9b8c1625a0 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255695639 0 0) 0x7f9b70003520 con 0x7f9b8c1625a0 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 msgr2=0x7f9b8c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 0x7f9b8c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 >> v2:172.21.15.39:3300/0 conn(0x7f9b8c15ea80 msgr2=0x7f9b8c06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 --2- 172.21.15.32:0/2118705315 >> v2:172.21.15.39:3300/0 conn(0x7f9b8c15ea80 0x7f9b8c06b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c161da0 con 0x7f9b8c1625a0 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f9b78001670 con 0x7f9b8c1625a0 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 >> v1:172.21.15.32:6789/0 conn(0x7f9b8c1625a0 legacy=0x7f9b8c15e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b700039b0 0x7f9b70003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 0x7f9b70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.525+0000 7f9b8a000640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c162110 con 0x7f9b70004d40 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c15f080 con 0x7f9b700039b0 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7000b9c0 con 0x7f9b8c147400 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8be00640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8b400640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b700039b0 0x7f9b70003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8aa00640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 0x7f9b70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8be00640 1 -- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 msgr2=0x7f9b70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8be00640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c147400 0x7f9b70004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8be00640 1 -- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b700039b0 msgr2=0x7f9b70003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8be00640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b700039b0 0x7f9b70003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8be00640 1 -- 172.21.15.32:0/2118705315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b70003520 con 0x7f9b70004d40 2024-03-06T07:31:46.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.526+0000 7f9b8b400640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b700039b0 0x7f9b70003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:46.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.527+0000 7f9b8be00640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b700051c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009fc0 tx=0x7f9b70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8254373cc4ca59c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84039070 con 0x7f9b70004d40 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8402cc60 con 0x7f9b70004d40 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b8a000640 1 -- 172.21.15.32:0/2118705315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84009c10 con 0x7f9b70004d40 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b90a00640 1 -- 172.21.15.32:0/2118705315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 msgr2=0x7f9b700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b90a00640 1 --2- 172.21.15.32:0/2118705315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b700051c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009fc0 tx=0x7f9b70005830 comp rx=0 tx=0).stop 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b8be00640 1 -- 172.21.15.32:0/2118705315 reap_dead start 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b90a00640 1 -- 172.21.15.32:0/2118705315 shutdown_connections 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b90a00640 1 -- 172.21.15.32:0/2118705315 >> 172.21.15.32:0/2118705315 conn(0x7f9b8c06c0c0 msgr2=0x7f9b8c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b90a00640 1 -- 172.21.15.32:0/2118705315 shutdown_connections 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.528+0000 7f9b90a00640 1 -- 172.21.15.32:0/2118705315 wait complete. 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.530+0000 7f9b90a00640 1 Processor -- start 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.530+0000 7f9b90a00640 1 -- start start 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.531+0000 7f9b90a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b8c1452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.531+0000 7f9b90a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b8c157d60 0x7f9b8c145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.531+0000 7f9b90a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c134410 0x7f9b8c134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.531+0000 7f9b90a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c06b130 con 0x7f9b70004d40 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.531+0000 7f9b90a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c06afb0 con 0x7f9b8c157d60 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.531+0000 7f9b90a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c06b2b0 con 0x7f9b8c134410 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b8c1452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b8c1452e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59238/0 (socket says 172.21.15.32:59238) 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 -- 172.21.15.32:0/1906250608 learned_addr learned my addr 172.21.15.32:0/1906250608 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8aa00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b8c157d60 0x7f9b8c145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 -- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c134410 msgr2=0x7f9b8c134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c134410 0x7f9b8c134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 -- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b8c157d60 msgr2=0x7f9b8c145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b8c157d60 0x7f9b8c145820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8b400640 1 -- 172.21.15.32:0/1906250608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c135190 con 0x7f9b70004d40 2024-03-06T07:31:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.532+0000 7f9b8aa00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b8c157d60 0x7f9b8c145820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.533+0000 7f9b8b400640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b8c1452e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c010910 tx=0x7f9b7c010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.533+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c00f040 con 0x7f9b70004d40 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.533+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7c00eba0 con 0x7f9b70004d40 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.533+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c01c400 con 0x7f9b70004d40 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.533+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c132570 con 0x7f9b70004d40 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.533+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c134dd0 con 0x7f9b70004d40 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.534+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c132360 con 0x7f9b70004d40 2024-03-06T07:31:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.536+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b7c01c5a0 con 0x7f9b70004d40 2024-03-06T07:31:46.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.537+0000 7f9b83400640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b6c06b4d0 0x7f9b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.537+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b7c08a910 con 0x7f9b70004d40 2024-03-06T07:31:46.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.537+0000 7f9b8aa00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b6c06b4d0 0x7f9b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.538+0000 7f9b8aa00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b6c06b4d0 0x7f9b6c06d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9b78001200 tx=0x7f9b78009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.546+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b7c018030 con 0x7f9b70004d40 2024-03-06T07:31:46.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.743+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9b8c04f330 con 0x7f9b6c06b4d0 2024-03-06T07:31:46.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.753+0000 7f9b83400640 1 -- 172.21.15.32:0/1906250608 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251950 (secure 0 0 0) 0x7f9b8c04f330 con 0x7f9b6c06b4d0 2024-03-06T07:31:46.759 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b6c06b4d0 msgr2=0x7f9b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b6c06b4d0 0x7f9b6c06d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9b78001200 tx=0x7f9b78009040 comp rx=0 tx=0).stop 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 msgr2=0x7f9b8c1452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b8c1452e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c010910 tx=0x7f9b7c010dd0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 shutdown_connections 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9b6c06b4d0 0x7f9b6c06d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b8c134410 0x7f9b8c134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b8c157d60 0x7f9b8c145820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1906250608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b70004d40 0x7f9b8c1452e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 >> 172.21.15.32:0/1906250608 conn(0x7f9b8c06c0c0 msgr2=0x7f9b8c06eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 shutdown_connections 2024-03-06T07:31:46.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:46.759+0000 7f9b90a00640 1 -- 172.21.15.32:0/1906250608 wait complete. 2024-03-06T07:31:46.779 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-06T07:31:46.235785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255996,"kb_used_data":6532,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718724,"statfs":{"total":773094113280,"available":772831973376,"internally_reserved":0,"allocated":6688768,"data_stored":3430416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001870"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890485+0000","last_change":"2024-03-06T07:31:41.795047+0000","last_active":"2024-03-06T07:31:42.890485+0000","last_peered":"2024-03-06T07:31:42.890485+0000","last_clean":"2024-03-06T07:31:42.890485+0000","last_became_active":"2024-03-06T07:31:41.794710+0000","last_became_peered":"2024-03-06T07:31:41.794710+0000","last_unstale":"2024-03-06T07:31:42.890485+0000","last_undegraded":"2024-03-06T07:31:42.890485+0000","last_fullsized":"2024-03-06T07:31:42.890485+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:42.409543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890824+0000","last_change":"2024-03-06T07:31:41.769224+0000","last_active":"2024-03-06T07:31:42.890824+0000","last_peered":"2024-03-06T07:31:42.890824+0000","last_clean":"2024-03-06T07:31:42.890824+0000","last_became_active":"2024-03-06T07:31:41.768448+0000","last_became_peered":"2024-03-06T07:31:41.768448+0000","last_unstale":"2024-03-06T07:31:42.890824+0000","last_undegraded":"2024-03-06T07:31:42.890824+0000","last_fullsized":"2024-03-06T07:31:42.890824+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:45.590263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894674+0000","last_change":"2024-03-06T07:31:41.786519+0000","last_active":"2024-03-06T07:31:42.894674+0000","last_peered":"2024-03-06T07:31:42.894674+0000","last_clean":"2024-03-06T07:31:42.894674+0000","last_became_active":"2024-03-06T07:31:41.784440+0000","last_became_peered":"2024-03-06T07:31:41.784440+0000","last_unstale":"2024-03-06T07:31:42.894674+0000","last_undegraded":"2024-03-06T07:31:42.894674+0000","last_fullsized":"2024-03-06T07:31:42.894674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:10.283128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893260+0000","last_change":"2024-03-06T07:31:41.787363+0000","last_active":"2024-03-06T07:31:42.893260+0000","last_peered":"2024-03-06T07:31:42.893260+0000","last_clean":"2024-03-06T07:31:42.893260+0000","last_became_active":"2024-03-06T07:31:41.787017+0000","last_became_peered":"2024-03-06T07:31:41.787017+0000","last_unstale":"2024-03-06T07:31:42.893260+0000","last_undegraded":"2024-03-06T07:31:42.893260+0000","last_fullsized":"2024-03-06T07:31:42.893260+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:42.018523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890823+0000","last_change":"2024-03-06T07:31:41.794850+0000","last_active":"2024-03-06T07:31:42.890823+0000","last_peered":"2024-03-06T07:31:42.890823+0000","last_clean":"2024-03-06T07:31:42.890823+0000","last_became_active":"2024-03-06T07:31:41.794576+0000","last_became_peered":"2024-03-06T07:31:41.794576+0000","last_unstale":"2024-03-06T07:31:42.890823+0000","last_undegraded":"2024-03-06T07:31:42.890823+0000","last_fullsized":"2024-03-06T07:31:42.890823+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:27.216132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811453+0000","last_change":"2024-03-06T07:31:41.798405+0000","last_active":"2024-03-06T07:31:42.811453+0000","last_peered":"2024-03-06T07:31:42.811453+0000","last_clean":"2024-03-06T07:31:42.811453+0000","last_became_active":"2024-03-06T07:31:41.790253+0000","last_became_peered":"2024-03-06T07:31:41.790253+0000","last_unstale":"2024-03-06T07:31:42.811453+0000","last_undegraded":"2024-03-06T07:31:42.811453+0000","last_fullsized":"2024-03-06T07:31:42.811453+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:44.511763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.787979+0000","last_change":"2024-03-06T07:31:41.794884+0000","last_active":"2024-03-06T07:31:42.787979+0000","last_peered":"2024-03-06T07:31:42.787979+0000","last_clean":"2024-03-06T07:31:42.787979+0000","last_became_active":"2024-03-06T07:31:41.794499+0000","last_became_peered":"2024-03-06T07:31:41.794499+0000","last_unstale":"2024-03-06T07:31:42.787979+0000","last_undegraded":"2024-03-06T07:31:42.787979+0000","last_fullsized":"2024-03-06T07:31:42.787979+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:15.108357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890405+0000","last_change":"2024-03-06T07:31:41.781988+0000","last_active":"2024-03-06T07:31:42.890405+0000","last_peered":"2024-03-06T07:31:42.890405+0000","last_clean":"2024-03-06T07:31:42.890405+0000","last_became_active":"2024-03-06T07:31:41.781141+0000","last_became_peered":"2024-03-06T07:31:41.781141+0000","last_unstale":"2024-03-06T07:31:42.890405+0000","last_undegraded":"2024-03-06T07:31:42.890405+0000","last_fullsized":"2024-03-06T07:31:42.890405+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:09.871916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788421+0000","last_change":"2024-03-06T07:31:41.790227+0000","last_active":"2024-03-06T07:31:42.788421+0000","last_peered":"2024-03-06T07:31:42.788421+0000","last_clean":"2024-03-06T07:31:42.788421+0000","last_became_active":"2024-03-06T07:31:41.789927+0000","last_became_peered":"2024-03-06T07:31:41.789927+0000","last_unstale":"2024-03-06T07:31:42.788421+0000","last_undegraded":"2024-03-06T07:31:42.788421+0000","last_fullsized":"2024-03-06T07:31:42.788421+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:41.729273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.784834+0000","last_change":"2024-03-06T07:31:41.776267+0000","last_active":"2024-03-06T07:31:42.784834+0000","last_peered":"2024-03-06T07:31:42.784834+0000","last_clean":"2024-03-06T07:31:42.784834+0000","last_became_active":"2024-03-06T07:31:41.775999+0000","last_became_peered":"2024-03-06T07:31:41.775999+0000","last_unstale":"2024-03-06T07:31:42.784834+0000","last_undegraded":"2024-03-06T07:31:42.784834+0000","last_fullsized":"2024-03-06T07:31:42.784834+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:28.862885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783961+0000","last_change":"2024-03-06T07:31:41.781701+0000","last_active":"2024-03-06T07:31:42.783961+0000","last_peered":"2024-03-06T07:31:42.783961+0000","last_clean":"2024-03-06T07:31:42.783961+0000","last_became_active":"2024-03-06T07:31:41.781404+0000","last_became_peered":"2024-03-06T07:31:41.781404+0000","last_unstale":"2024-03-06T07:31:42.783961+0000","last_undegraded":"2024-03-06T07:31:42.783961+0000","last_fullsized":"2024-03-06T07:31:42.783961+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:40.937673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893714+0000","last_change":"2024-03-06T07:31:41.783338+0000","last_active":"2024-03-06T07:31:42.893714+0000","last_peered":"2024-03-06T07:31:42.893714+0000","last_clean":"2024-03-06T07:31:42.893714+0000","last_became_active":"2024-03-06T07:31:41.781793+0000","last_became_peered":"2024-03-06T07:31:41.781793+0000","last_unstale":"2024-03-06T07:31:42.893714+0000","last_undegraded":"2024-03-06T07:31:42.893714+0000","last_fullsized":"2024-03-06T07:31:42.893714+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:16.534676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813499+0000","last_change":"2024-03-06T07:31:41.788894+0000","last_active":"2024-03-06T07:31:42.813499+0000","last_peered":"2024-03-06T07:31:42.813499+0000","last_clean":"2024-03-06T07:31:42.813499+0000","last_became_active":"2024-03-06T07:31:41.788604+0000","last_became_peered":"2024-03-06T07:31:41.788604+0000","last_unstale":"2024-03-06T07:31:42.813499+0000","last_undegraded":"2024-03-06T07:31:42.813499+0000","last_fullsized":"2024-03-06T07:31:42.813499+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:24.910268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788033+0000","last_change":"2024-03-06T07:31:41.786917+0000","last_active":"2024-03-06T07:31:42.788033+0000","last_peered":"2024-03-06T07:31:42.788033+0000","last_clean":"2024-03-06T07:31:42.788033+0000","last_became_active":"2024-03-06T07:31:41.786609+0000","last_became_peered":"2024-03-06T07:31:41.786609+0000","last_unstale":"2024-03-06T07:31:42.788033+0000","last_undegraded":"2024-03-06T07:31:42.788033+0000","last_fullsized":"2024-03-06T07:31:42.788033+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:12.014768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788362+0000","last_change":"2024-03-06T07:31:41.795565+0000","last_active":"2024-03-06T07:31:42.788362+0000","last_peered":"2024-03-06T07:31:42.788362+0000","last_clean":"2024-03-06T07:31:42.788362+0000","last_became_active":"2024-03-06T07:31:41.795207+0000","last_became_peered":"2024-03-06T07:31:41.795207+0000","last_unstale":"2024-03-06T07:31:42.788362+0000","last_undegraded":"2024-03-06T07:31:42.788362+0000","last_fullsized":"2024-03-06T07:31:42.788362+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:37.647144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811211+0000","last_change":"2024-03-06T07:31:41.790024+0000","last_active":"2024-03-06T07:31:42.811211+0000","last_peered":"2024-03-06T07:31:42.811211+0000","last_clean":"2024-03-06T07:31:42.811211+0000","last_became_active":"2024-03-06T07:31:41.789571+0000","last_became_peered":"2024-03-06T07:31:41.789571+0000","last_unstale":"2024-03-06T07:31:42.811211+0000","last_undegraded":"2024-03-06T07:31:42.811211+0000","last_fullsized":"2024-03-06T07:31:42.811211+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:34.540911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891174+0000","last_change":"2024-03-06T07:31:41.779510+0000","last_active":"2024-03-06T07:31:42.891174+0000","last_peered":"2024-03-06T07:31:42.891174+0000","last_clean":"2024-03-06T07:31:42.891174+0000","last_became_active":"2024-03-06T07:31:41.779129+0000","last_became_peered":"2024-03-06T07:31:41.779129+0000","last_unstale":"2024-03-06T07:31:42.891174+0000","last_undegraded":"2024-03-06T07:31:42.891174+0000","last_fullsized":"2024-03-06T07:31:42.891174+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:06.989000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895393+0000","last_change":"2024-03-06T07:31:41.791169+0000","last_active":"2024-03-06T07:31:42.895393+0000","last_peered":"2024-03-06T07:31:42.895393+0000","last_clean":"2024-03-06T07:31:42.895393+0000","last_became_active":"2024-03-06T07:31:41.790861+0000","last_became_peered":"2024-03-06T07:31:41.790861+0000","last_unstale":"2024-03-06T07:31:42.895393+0000","last_undegraded":"2024-03-06T07:31:42.895393+0000","last_fullsized":"2024-03-06T07:31:42.895393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:18.825154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890058+0000","last_change":"2024-03-06T07:31:41.797664+0000","last_active":"2024-03-06T07:31:42.890058+0000","last_peered":"2024-03-06T07:31:42.890058+0000","last_clean":"2024-03-06T07:31:42.890058+0000","last_became_active":"2024-03-06T07:31:41.797057+0000","last_became_peered":"2024-03-06T07:31:41.797057+0000","last_unstale":"2024-03-06T07:31:42.890058+0000","last_undegraded":"2024-03-06T07:31:42.890058+0000","last_fullsized":"2024-03-06T07:31:42.890058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:42.962676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890183+0000","last_change":"2024-03-06T07:31:41.790447+0000","last_active":"2024-03-06T07:31:42.890183+0000","last_peered":"2024-03-06T07:31:42.890183+0000","last_clean":"2024-03-06T07:31:42.890183+0000","last_became_active":"2024-03-06T07:31:41.790126+0000","last_became_peered":"2024-03-06T07:31:41.790126+0000","last_unstale":"2024-03-06T07:31:42.890183+0000","last_undegraded":"2024-03-06T07:31:42.890183+0000","last_fullsized":"2024-03-06T07:31:42.890183+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:19.134708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895206+0000","last_change":"2024-03-06T07:31:41.783119+0000","last_active":"2024-03-06T07:31:42.895206+0000","last_peered":"2024-03-06T07:31:42.895206+0000","last_clean":"2024-03-06T07:31:42.895206+0000","last_became_active":"2024-03-06T07:31:41.782828+0000","last_became_peered":"2024-03-06T07:31:41.782828+0000","last_unstale":"2024-03-06T07:31:42.895206+0000","last_undegraded":"2024-03-06T07:31:42.895206+0000","last_fullsized":"2024-03-06T07:31:42.895206+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:56.017501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891444+0000","last_change":"2024-03-06T07:31:41.790165+0000","last_active":"2024-03-06T07:31:42.891444+0000","last_peered":"2024-03-06T07:31:42.891444+0000","last_clean":"2024-03-06T07:31:42.891444+0000","last_became_active":"2024-03-06T07:31:41.786415+0000","last_became_peered":"2024-03-06T07:31:41.786415+0000","last_unstale":"2024-03-06T07:31:42.891444+0000","last_undegraded":"2024-03-06T07:31:42.891444+0000","last_fullsized":"2024-03-06T07:31:42.891444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:21.410115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893750+0000","last_change":"2024-03-06T07:31:41.782338+0000","last_active":"2024-03-06T07:31:42.893750+0000","last_peered":"2024-03-06T07:31:42.893750+0000","last_clean":"2024-03-06T07:31:42.893750+0000","last_became_active":"2024-03-06T07:31:41.782059+0000","last_became_peered":"2024-03-06T07:31:41.782059+0000","last_unstale":"2024-03-06T07:31:42.893750+0000","last_undegraded":"2024-03-06T07:31:42.893750+0000","last_fullsized":"2024-03-06T07:31:42.893750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:39.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813187+0000","last_change":"2024-03-06T07:31:41.788188+0000","last_active":"2024-03-06T07:31:42.813187+0000","last_peered":"2024-03-06T07:31:42.813187+0000","last_clean":"2024-03-06T07:31:42.813187+0000","last_became_active":"2024-03-06T07:31:41.787880+0000","last_became_peered":"2024-03-06T07:31:41.787880+0000","last_unstale":"2024-03-06T07:31:42.813187+0000","last_undegraded":"2024-03-06T07:31:42.813187+0000","last_fullsized":"2024-03-06T07:31:42.813187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:51.894909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890319+0000","last_change":"2024-03-06T07:31:41.802411+0000","last_active":"2024-03-06T07:31:42.890319+0000","last_peered":"2024-03-06T07:31:42.890319+0000","last_clean":"2024-03-06T07:31:42.890319+0000","last_became_active":"2024-03-06T07:31:41.802090+0000","last_became_peered":"2024-03-06T07:31:41.802090+0000","last_unstale":"2024-03-06T07:31:42.890319+0000","last_undegraded":"2024-03-06T07:31:42.890319+0000","last_fullsized":"2024-03-06T07:31:42.890319+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:42.473440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895168+0000","last_change":"2024-03-06T07:31:41.790330+0000","last_active":"2024-03-06T07:31:42.895168+0000","last_peered":"2024-03-06T07:31:42.895168+0000","last_clean":"2024-03-06T07:31:42.895168+0000","last_became_active":"2024-03-06T07:31:41.788392+0000","last_became_peered":"2024-03-06T07:31:41.788392+0000","last_unstale":"2024-03-06T07:31:42.895168+0000","last_undegraded":"2024-03-06T07:31:42.895168+0000","last_fullsized":"2024-03-06T07:31:42.895168+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:29.826508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811770+0000","last_change":"2024-03-06T07:31:41.799261+0000","last_active":"2024-03-06T07:31:42.811770+0000","last_peered":"2024-03-06T07:31:42.811770+0000","last_clean":"2024-03-06T07:31:42.811770+0000","last_became_active":"2024-03-06T07:31:41.798965+0000","last_became_peered":"2024-03-06T07:31:41.798965+0000","last_unstale":"2024-03-06T07:31:42.811770+0000","last_undegraded":"2024-03-06T07:31:42.811770+0000","last_fullsized":"2024-03-06T07:31:42.811770+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:44.734632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890896+0000","last_change":"2024-03-06T07:31:41.794429+0000","last_active":"2024-03-06T07:31:42.890896+0000","last_peered":"2024-03-06T07:31:42.890896+0000","last_clean":"2024-03-06T07:31:42.890896+0000","last_became_active":"2024-03-06T07:31:41.793852+0000","last_became_peered":"2024-03-06T07:31:41.793852+0000","last_unstale":"2024-03-06T07:31:42.890896+0000","last_undegraded":"2024-03-06T07:31:42.890896+0000","last_fullsized":"2024-03-06T07:31:42.890896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:25.728984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890826+0000","last_change":"2024-03-06T07:31:41.791402+0000","last_active":"2024-03-06T07:31:42.890826+0000","last_peered":"2024-03-06T07:31:42.890826+0000","last_clean":"2024-03-06T07:31:42.890826+0000","last_became_active":"2024-03-06T07:31:41.789373+0000","last_became_peered":"2024-03-06T07:31:41.789373+0000","last_unstale":"2024-03-06T07:31:42.890826+0000","last_undegraded":"2024-03-06T07:31:42.890826+0000","last_fullsized":"2024-03-06T07:31:42.890826+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.699678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893545+0000","last_change":"2024-03-06T07:31:41.778266+0000","last_active":"2024-03-06T07:31:42.893545+0000","last_peered":"2024-03-06T07:31:42.893545+0000","last_clean":"2024-03-06T07:31:42.893545+0000","last_became_active":"2024-03-06T07:31:41.777939+0000","last_became_peered":"2024-03-06T07:31:41.777939+0000","last_unstale":"2024-03-06T07:31:42.893545+0000","last_undegraded":"2024-03-06T07:31:42.893545+0000","last_fullsized":"2024-03-06T07:31:42.893545+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:34.540911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.889983+0000","last_change":"2024-03-06T07:31:41.783409+0000","last_active":"2024-03-06T07:31:42.889983+0000","last_peered":"2024-03-06T07:31:42.889983+0000","last_clean":"2024-03-06T07:31:42.889983+0000","last_became_active":"2024-03-06T07:31:41.783064+0000","last_became_peered":"2024-03-06T07:31:41.783064+0000","last_unstale":"2024-03-06T07:31:42.889983+0000","last_undegraded":"2024-03-06T07:31:42.889983+0000","last_fullsized":"2024-03-06T07:31:42.889983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:33.597292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811271+0000","last_change":"2024-03-06T07:31:41.807836+0000","last_active":"2024-03-06T07:31:42.811271+0000","last_peered":"2024-03-06T07:31:42.811271+0000","last_clean":"2024-03-06T07:31:42.811271+0000","last_became_active":"2024-03-06T07:31:41.807429+0000","last_became_peered":"2024-03-06T07:31:41.807429+0000","last_unstale":"2024-03-06T07:31:42.811271+0000","last_undegraded":"2024-03-06T07:31:42.811271+0000","last_fullsized":"2024-03-06T07:31:42.811271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:39.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812544+0000","last_change":"2024-03-06T07:31:41.792357+0000","last_active":"2024-03-06T07:31:42.812544+0000","last_peered":"2024-03-06T07:31:42.812544+0000","last_clean":"2024-03-06T07:31:42.812544+0000","last_became_active":"2024-03-06T07:31:41.791835+0000","last_became_peered":"2024-03-06T07:31:41.791835+0000","last_unstale":"2024-03-06T07:31:42.812544+0000","last_undegraded":"2024-03-06T07:31:42.812544+0000","last_fullsized":"2024-03-06T07:31:42.812544+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:56.017501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812044+0000","last_change":"2024-03-06T07:31:40.770850+0000","last_active":"2024-03-06T07:31:42.812044+0000","last_peered":"2024-03-06T07:31:42.812044+0000","last_clean":"2024-03-06T07:31:42.812044+0000","last_became_active":"2024-03-06T07:31:40.770586+0000","last_became_peered":"2024-03-06T07:31:40.770586+0000","last_unstale":"2024-03-06T07:31:42.812044+0000","last_undegraded":"2024-03-06T07:31:42.812044+0000","last_fullsized":"2024-03-06T07:31:42.812044+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.633372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783875+0000","last_change":"2024-03-06T07:31:41.778870+0000","last_active":"2024-03-06T07:31:42.783875+0000","last_peered":"2024-03-06T07:31:42.783875+0000","last_clean":"2024-03-06T07:31:42.783875+0000","last_became_active":"2024-03-06T07:31:41.777323+0000","last_became_peered":"2024-03-06T07:31:41.777323+0000","last_unstale":"2024-03-06T07:31:42.783875+0000","last_undegraded":"2024-03-06T07:31:42.783875+0000","last_fullsized":"2024-03-06T07:31:42.783875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:11.356244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890937+0000","last_change":"2024-03-06T07:31:40.758088+0000","last_active":"2024-03-06T07:31:42.890937+0000","last_peered":"2024-03-06T07:31:42.890937+0000","last_clean":"2024-03-06T07:31:42.890937+0000","last_became_active":"2024-03-06T07:31:40.757435+0000","last_became_peered":"2024-03-06T07:31:40.757435+0000","last_unstale":"2024-03-06T07:31:42.890937+0000","last_undegraded":"2024-03-06T07:31:42.890937+0000","last_fullsized":"2024-03-06T07:31:42.890937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:38.627915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811924+0000","last_change":"2024-03-06T07:31:41.807674+0000","last_active":"2024-03-06T07:31:42.811924+0000","last_peered":"2024-03-06T07:31:42.811924+0000","last_clean":"2024-03-06T07:31:42.811924+0000","last_became_active":"2024-03-06T07:31:41.807262+0000","last_became_peered":"2024-03-06T07:31:41.807262+0000","last_unstale":"2024-03-06T07:31:42.811924+0000","last_undegraded":"2024-03-06T07:31:42.811924+0000","last_fullsized":"2024-03-06T07:31:42.811924+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:04.178439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812018+0000","last_change":"2024-03-06T07:31:40.748552+0000","last_active":"2024-03-06T07:31:42.812018+0000","last_peered":"2024-03-06T07:31:42.812018+0000","last_clean":"2024-03-06T07:31:42.812018+0000","last_became_active":"2024-03-06T07:31:40.748253+0000","last_became_peered":"2024-03-06T07:31:40.748253+0000","last_unstale":"2024-03-06T07:31:42.812018+0000","last_undegraded":"2024-03-06T07:31:42.812018+0000","last_fullsized":"2024-03-06T07:31:42.812018+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:08.710620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812081+0000","last_change":"2024-03-06T07:31:41.800906+0000","last_active":"2024-03-06T07:31:42.812081+0000","last_peered":"2024-03-06T07:31:42.812081+0000","last_clean":"2024-03-06T07:31:42.812081+0000","last_became_active":"2024-03-06T07:31:41.800633+0000","last_became_peered":"2024-03-06T07:31:41.800633+0000","last_unstale":"2024-03-06T07:31:42.812081+0000","last_undegraded":"2024-03-06T07:31:42.812081+0000","last_fullsized":"2024-03-06T07:31:42.812081+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:31.417015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894451+0000","last_change":"2024-03-06T07:31:40.764233+0000","last_active":"2024-03-06T07:31:42.894451+0000","last_peered":"2024-03-06T07:31:42.894451+0000","last_clean":"2024-03-06T07:31:42.894451+0000","last_became_active":"2024-03-06T07:31:40.763332+0000","last_became_peered":"2024-03-06T07:31:40.763332+0000","last_unstale":"2024-03-06T07:31:42.894451+0000","last_undegraded":"2024-03-06T07:31:42.894451+0000","last_fullsized":"2024-03-06T07:31:42.894451+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:40.715501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891579+0000","last_change":"2024-03-06T07:31:41.789914+0000","last_active":"2024-03-06T07:31:42.891579+0000","last_peered":"2024-03-06T07:31:42.891579+0000","last_clean":"2024-03-06T07:31:42.891579+0000","last_became_active":"2024-03-06T07:31:41.785474+0000","last_became_peered":"2024-03-06T07:31:41.785474+0000","last_unstale":"2024-03-06T07:31:42.891579+0000","last_undegraded":"2024-03-06T07:31:42.891579+0000","last_fullsized":"2024-03-06T07:31:42.891579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:03.345699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788259+0000","last_change":"2024-03-06T07:31:40.759136+0000","last_active":"2024-03-06T07:31:42.788259+0000","last_peered":"2024-03-06T07:31:42.788259+0000","last_clean":"2024-03-06T07:31:42.788259+0000","last_became_active":"2024-03-06T07:31:40.758817+0000","last_became_peered":"2024-03-06T07:31:40.758817+0000","last_unstale":"2024-03-06T07:31:42.788259+0000","last_undegraded":"2024-03-06T07:31:42.788259+0000","last_fullsized":"2024-03-06T07:31:42.788259+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:39.923901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890602+0000","last_change":"2024-03-06T07:31:41.790980+0000","last_active":"2024-03-06T07:31:42.890602+0000","last_peered":"2024-03-06T07:31:42.890602+0000","last_clean":"2024-03-06T07:31:42.890602+0000","last_became_active":"2024-03-06T07:31:41.788054+0000","last_became_peered":"2024-03-06T07:31:41.788054+0000","last_unstale":"2024-03-06T07:31:42.890602+0000","last_undegraded":"2024-03-06T07:31:42.890602+0000","last_fullsized":"2024-03-06T07:31:42.890602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:59.635567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811533+0000","last_change":"2024-03-06T07:31:40.758941+0000","last_active":"2024-03-06T07:31:42.811533+0000","last_peered":"2024-03-06T07:31:42.811533+0000","last_clean":"2024-03-06T07:31:42.811533+0000","last_became_active":"2024-03-06T07:31:40.758617+0000","last_became_peered":"2024-03-06T07:31:40.758617+0000","last_unstale":"2024-03-06T07:31:42.811533+0000","last_undegraded":"2024-03-06T07:31:42.811533+0000","last_fullsized":"2024-03-06T07:31:42.811533+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:42.963353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813439+0000","last_change":"2024-03-06T07:31:41.778095+0000","last_active":"2024-03-06T07:31:42.813439+0000","last_peered":"2024-03-06T07:31:42.813439+0000","last_clean":"2024-03-06T07:31:42.813439+0000","last_became_active":"2024-03-06T07:31:41.777778+0000","last_became_peered":"2024-03-06T07:31:41.777778+0000","last_unstale":"2024-03-06T07:31:42.813439+0000","last_undegraded":"2024-03-06T07:31:42.813439+0000","last_fullsized":"2024-03-06T07:31:42.813439+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:50.530003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893345+0000","last_change":"2024-03-06T07:31:40.762429+0000","last_active":"2024-03-06T07:31:42.893345+0000","last_peered":"2024-03-06T07:31:42.893345+0000","last_clean":"2024-03-06T07:31:42.893345+0000","last_became_active":"2024-03-06T07:31:40.762145+0000","last_became_peered":"2024-03-06T07:31:40.762145+0000","last_unstale":"2024-03-06T07:31:42.893345+0000","last_undegraded":"2024-03-06T07:31:42.893345+0000","last_fullsized":"2024-03-06T07:31:42.893345+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:11.863510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890572+0000","last_change":"2024-03-06T07:31:41.774233+0000","last_active":"2024-03-06T07:31:42.890572+0000","last_peered":"2024-03-06T07:31:42.890572+0000","last_clean":"2024-03-06T07:31:42.890572+0000","last_became_active":"2024-03-06T07:31:41.770975+0000","last_became_peered":"2024-03-06T07:31:41.770975+0000","last_unstale":"2024-03-06T07:31:42.890572+0000","last_undegraded":"2024-03-06T07:31:42.890572+0000","last_fullsized":"2024-03-06T07:31:42.890572+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:06.436697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.787661+0000","last_change":"2024-03-06T07:31:40.753547+0000","last_active":"2024-03-06T07:31:42.787661+0000","last_peered":"2024-03-06T07:31:42.787661+0000","last_clean":"2024-03-06T07:31:42.787661+0000","last_became_active":"2024-03-06T07:31:40.753129+0000","last_became_peered":"2024-03-06T07:31:40.753129+0000","last_unstale":"2024-03-06T07:31:42.787661+0000","last_undegraded":"2024-03-06T07:31:42.787661+0000","last_fullsized":"2024-03-06T07:31:42.787661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:23.896496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788852+0000","last_change":"2024-03-06T07:31:41.795218+0000","last_active":"2024-03-06T07:31:42.788852+0000","last_peered":"2024-03-06T07:31:42.788852+0000","last_clean":"2024-03-06T07:31:42.788852+0000","last_became_active":"2024-03-06T07:31:41.794885+0000","last_became_peered":"2024-03-06T07:31:41.794885+0000","last_unstale":"2024-03-06T07:31:42.788852+0000","last_undegraded":"2024-03-06T07:31:42.788852+0000","last_fullsized":"2024-03-06T07:31:42.788852+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:09.724393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895065+0000","last_change":"2024-03-06T07:31:40.756304+0000","last_active":"2024-03-06T07:31:42.895065+0000","last_peered":"2024-03-06T07:31:42.895065+0000","last_clean":"2024-03-06T07:31:42.895065+0000","last_became_active":"2024-03-06T07:31:40.755877+0000","last_became_peered":"2024-03-06T07:31:40.755877+0000","last_unstale":"2024-03-06T07:31:42.895065+0000","last_undegraded":"2024-03-06T07:31:42.895065+0000","last_fullsized":"2024-03-06T07:31:42.895065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:40.715501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894800+0000","last_change":"2024-03-06T07:31:41.783826+0000","last_active":"2024-03-06T07:31:42.894800+0000","last_peered":"2024-03-06T07:31:42.894800+0000","last_clean":"2024-03-06T07:31:42.894800+0000","last_became_active":"2024-03-06T07:31:41.783401+0000","last_became_peered":"2024-03-06T07:31:41.783401+0000","last_unstale":"2024-03-06T07:31:42.894800+0000","last_undegraded":"2024-03-06T07:31:42.894800+0000","last_fullsized":"2024-03-06T07:31:42.894800+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:44.734632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890998+0000","last_change":"2024-03-06T07:31:40.753641+0000","last_active":"2024-03-06T07:31:42.890998+0000","last_peered":"2024-03-06T07:31:42.890998+0000","last_clean":"2024-03-06T07:31:42.890998+0000","last_became_active":"2024-03-06T07:31:40.753359+0000","last_became_peered":"2024-03-06T07:31:40.753359+0000","last_unstale":"2024-03-06T07:31:42.890998+0000","last_undegraded":"2024-03-06T07:31:42.890998+0000","last_fullsized":"2024-03-06T07:31:42.890998+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:43.497991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783712+0000","last_change":"2024-03-06T07:31:41.788013+0000","last_active":"2024-03-06T07:31:42.783712+0000","last_peered":"2024-03-06T07:31:42.783712+0000","last_clean":"2024-03-06T07:31:42.783712+0000","last_became_active":"2024-03-06T07:31:41.783682+0000","last_became_peered":"2024-03-06T07:31:41.783682+0000","last_unstale":"2024-03-06T07:31:42.783712+0000","last_undegraded":"2024-03-06T07:31:42.783712+0000","last_fullsized":"2024-03-06T07:31:42.783712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:24.966333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895364+0000","last_change":"2024-03-06T07:31:40.749523+0000","last_active":"2024-03-06T07:31:42.895364+0000","last_peered":"2024-03-06T07:31:42.895364+0000","last_clean":"2024-03-06T07:31:42.895364+0000","last_became_active":"2024-03-06T07:31:40.748340+0000","last_became_peered":"2024-03-06T07:31:40.748340+0000","last_unstale":"2024-03-06T07:31:42.895364+0000","last_undegraded":"2024-03-06T07:31:42.895364+0000","last_fullsized":"2024-03-06T07:31:42.895364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:08.710620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895378+0000","last_change":"2024-03-06T07:31:41.786850+0000","last_active":"2024-03-06T07:31:42.895378+0000","last_peered":"2024-03-06T07:31:42.895378+0000","last_clean":"2024-03-06T07:31:42.895378+0000","last_became_active":"2024-03-06T07:31:41.786132+0000","last_became_peered":"2024-03-06T07:31:41.786132+0000","last_unstale":"2024-03-06T07:31:42.895378+0000","last_undegraded":"2024-03-06T07:31:42.895378+0000","last_fullsized":"2024-03-06T07:31:42.895378+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.201570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788696+0000","last_change":"2024-03-06T07:31:40.756763+0000","last_active":"2024-03-06T07:31:42.788696+0000","last_peered":"2024-03-06T07:31:42.788696+0000","last_clean":"2024-03-06T07:31:42.788696+0000","last_became_active":"2024-03-06T07:31:40.756276+0000","last_became_peered":"2024-03-06T07:31:40.756276+0000","last_unstale":"2024-03-06T07:31:42.788696+0000","last_undegraded":"2024-03-06T07:31:42.788696+0000","last_fullsized":"2024-03-06T07:31:42.788696+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:10.342472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892914+0000","last_change":"2024-03-06T07:31:41.796433+0000","last_active":"2024-03-06T07:31:42.892914+0000","last_peered":"2024-03-06T07:31:42.892914+0000","last_clean":"2024-03-06T07:31:42.892914+0000","last_became_active":"2024-03-06T07:31:41.795848+0000","last_became_peered":"2024-03-06T07:31:41.795848+0000","last_unstale":"2024-03-06T07:31:42.892914+0000","last_undegraded":"2024-03-06T07:31:42.892914+0000","last_fullsized":"2024-03-06T07:31:42.892914+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.778914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890695+0000","last_change":"2024-03-06T07:31:40.751197+0000","last_active":"2024-03-06T07:31:42.890695+0000","last_peered":"2024-03-06T07:31:42.890695+0000","last_clean":"2024-03-06T07:31:42.890695+0000","last_became_active":"2024-03-06T07:31:40.750896+0000","last_became_peered":"2024-03-06T07:31:40.750896+0000","last_unstale":"2024-03-06T07:31:42.890695+0000","last_undegraded":"2024-03-06T07:31:42.890695+0000","last_fullsized":"2024-03-06T07:31:42.890695+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:28.812735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894292+0000","last_change":"2024-03-06T07:31:41.778124+0000","last_active":"2024-03-06T07:31:42.894292+0000","last_peered":"2024-03-06T07:31:42.894292+0000","last_clean":"2024-03-06T07:31:42.894292+0000","last_became_active":"2024-03-06T07:31:41.770879+0000","last_became_peered":"2024-03-06T07:31:41.770879+0000","last_unstale":"2024-03-06T07:31:42.894292+0000","last_undegraded":"2024-03-06T07:31:42.894292+0000","last_fullsized":"2024-03-06T07:31:42.894292+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:18.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812952+0000","last_change":"2024-03-06T07:31:40.759869+0000","last_active":"2024-03-06T07:31:42.812952+0000","last_peered":"2024-03-06T07:31:42.812952+0000","last_clean":"2024-03-06T07:31:42.812952+0000","last_became_active":"2024-03-06T07:31:40.759506+0000","last_became_peered":"2024-03-06T07:31:40.759506+0000","last_unstale":"2024-03-06T07:31:42.812952+0000","last_undegraded":"2024-03-06T07:31:42.812952+0000","last_fullsized":"2024-03-06T07:31:42.812952+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:27.345029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.787760+0000","last_change":"2024-03-06T07:31:41.781383+0000","last_active":"2024-03-06T07:31:42.787760+0000","last_peered":"2024-03-06T07:31:42.787760+0000","last_clean":"2024-03-06T07:31:42.787760+0000","last_became_active":"2024-03-06T07:31:41.781039+0000","last_became_peered":"2024-03-06T07:31:41.781039+0000","last_unstale":"2024-03-06T07:31:42.787760+0000","last_undegraded":"2024-03-06T07:31:42.787760+0000","last_fullsized":"2024-03-06T07:31:42.787760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:12.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.893621+0000","last_change":"2024-03-06T07:31:40.763977+0000","last_active":"2024-03-06T07:31:42.893621+0000","last_peered":"2024-03-06T07:31:42.893621+0000","last_clean":"2024-03-06T07:31:42.893621+0000","last_became_active":"2024-03-06T07:31:40.763409+0000","last_became_peered":"2024-03-06T07:31:40.763409+0000","last_unstale":"2024-03-06T07:31:42.893621+0000","last_undegraded":"2024-03-06T07:31:42.893621+0000","last_fullsized":"2024-03-06T07:31:42.893621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:14.094585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894020+0000","last_change":"2024-03-06T07:31:41.790503+0000","last_active":"2024-03-06T07:31:42.894020+0000","last_peered":"2024-03-06T07:31:42.894020+0000","last_clean":"2024-03-06T07:31:42.894020+0000","last_became_active":"2024-03-06T07:31:41.790234+0000","last_became_peered":"2024-03-06T07:31:41.790234+0000","last_unstale":"2024-03-06T07:31:42.894020+0000","last_undegraded":"2024-03-06T07:31:42.894020+0000","last_fullsized":"2024-03-06T07:31:42.894020+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:44.511763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892811+0000","last_change":"2024-03-06T07:31:40.765734+0000","last_active":"2024-03-06T07:31:42.892811+0000","last_peered":"2024-03-06T07:31:42.892811+0000","last_clean":"2024-03-06T07:31:42.892811+0000","last_became_active":"2024-03-06T07:31:40.764184+0000","last_became_peered":"2024-03-06T07:31:40.764184+0000","last_unstale":"2024-03-06T07:31:42.892811+0000","last_undegraded":"2024-03-06T07:31:42.892811+0000","last_fullsized":"2024-03-06T07:31:42.892811+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.633372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895136+0000","last_change":"2024-03-06T07:31:41.785800+0000","last_active":"2024-03-06T07:31:42.895136+0000","last_peered":"2024-03-06T07:31:42.895136+0000","last_clean":"2024-03-06T07:31:42.895136+0000","last_became_active":"2024-03-06T07:31:41.785475+0000","last_became_peered":"2024-03-06T07:31:41.785475+0000","last_unstale":"2024-03-06T07:31:42.895136+0000","last_undegraded":"2024-03-06T07:31:42.895136+0000","last_fullsized":"2024-03-06T07:31:42.895136+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:31.417015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891193+0000","last_change":"2024-03-06T07:31:40.753184+0000","last_active":"2024-03-06T07:31:42.891193+0000","last_peered":"2024-03-06T07:31:42.891193+0000","last_clean":"2024-03-06T07:31:42.891193+0000","last_became_active":"2024-03-06T07:31:40.752887+0000","last_became_peered":"2024-03-06T07:31:40.752887+0000","last_unstale":"2024-03-06T07:31:42.891193+0000","last_undegraded":"2024-03-06T07:31:42.891193+0000","last_fullsized":"2024-03-06T07:31:42.891193+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:17.778668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895998+0000","last_change":"2024-03-06T07:31:41.796442+0000","last_active":"2024-03-06T07:31:42.895998+0000","last_peered":"2024-03-06T07:31:42.895998+0000","last_clean":"2024-03-06T07:31:42.895998+0000","last_became_active":"2024-03-06T07:31:41.795853+0000","last_became_peered":"2024-03-06T07:31:41.795853+0000","last_unstale":"2024-03-06T07:31:42.895998+0000","last_undegraded":"2024-03-06T07:31:42.895998+0000","last_fullsized":"2024-03-06T07:31:42.895998+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:34.540911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.812232+0000","last_change":"2024-03-06T07:31:40.752399+0000","last_active":"2024-03-06T07:31:42.812232+0000","last_peered":"2024-03-06T07:31:42.812232+0000","last_clean":"2024-03-06T07:31:42.812232+0000","last_became_active":"2024-03-06T07:31:40.752096+0000","last_became_peered":"2024-03-06T07:31:40.752096+0000","last_unstale":"2024-03-06T07:31:42.812232+0000","last_undegraded":"2024-03-06T07:31:42.812232+0000","last_fullsized":"2024-03-06T07:31:42.812232+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:40.715501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890337+0000","last_change":"2024-03-06T07:31:41.780583+0000","last_active":"2024-03-06T07:31:42.890337+0000","last_peered":"2024-03-06T07:31:42.890337+0000","last_clean":"2024-03-06T07:31:42.890337+0000","last_became_active":"2024-03-06T07:31:41.780299+0000","last_became_peered":"2024-03-06T07:31:41.780299+0000","last_unstale":"2024-03-06T07:31:42.890337+0000","last_undegraded":"2024-03-06T07:31:42.890337+0000","last_fullsized":"2024-03-06T07:31:42.890337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:14.927128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895173+0000","last_change":"2024-03-06T07:31:40.764368+0000","last_active":"2024-03-06T07:31:42.895173+0000","last_peered":"2024-03-06T07:31:42.895173+0000","last_clean":"2024-03-06T07:31:42.895173+0000","last_became_active":"2024-03-06T07:31:40.763979+0000","last_became_peered":"2024-03-06T07:31:40.763979+0000","last_unstale":"2024-03-06T07:31:42.895173+0000","last_undegraded":"2024-03-06T07:31:42.895173+0000","last_fullsized":"2024-03-06T07:31:42.895173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:11.000996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895085+0000","last_change":"2024-03-06T07:31:41.786484+0000","last_active":"2024-03-06T07:31:42.895085+0000","last_peered":"2024-03-06T07:31:42.895085+0000","last_clean":"2024-03-06T07:31:42.895085+0000","last_became_active":"2024-03-06T07:31:41.784913+0000","last_became_peered":"2024-03-06T07:31:41.784913+0000","last_unstale":"2024-03-06T07:31:42.895085+0000","last_undegraded":"2024-03-06T07:31:42.895085+0000","last_fullsized":"2024-03-06T07:31:42.895085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:58.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811898+0000","last_change":"2024-03-06T07:31:40.748126+0000","last_active":"2024-03-06T07:31:42.811898+0000","last_peered":"2024-03-06T07:31:42.811898+0000","last_clean":"2024-03-06T07:31:42.811898+0000","last_became_active":"2024-03-06T07:31:40.747842+0000","last_became_peered":"2024-03-06T07:31:40.747842+0000","last_unstale":"2024-03-06T07:31:42.811898+0000","last_undegraded":"2024-03-06T07:31:42.811898+0000","last_fullsized":"2024-03-06T07:31:42.811898+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:10.469237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894168+0000","last_change":"2024-03-06T07:31:41.797089+0000","last_active":"2024-03-06T07:31:42.894168+0000","last_peered":"2024-03-06T07:31:42.894168+0000","last_clean":"2024-03-06T07:31:42.894168+0000","last_became_active":"2024-03-06T07:31:41.796293+0000","last_became_peered":"2024-03-06T07:31:41.796293+0000","last_unstale":"2024-03-06T07:31:42.894168+0000","last_undegraded":"2024-03-06T07:31:42.894168+0000","last_fullsized":"2024-03-06T07:31:42.894168+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:23.360825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891235+0000","last_change":"2024-03-06T07:31:40.760412+0000","last_active":"2024-03-06T07:31:42.891235+0000","last_peered":"2024-03-06T07:31:42.891235+0000","last_clean":"2024-03-06T07:31:42.891235+0000","last_became_active":"2024-03-06T07:31:40.759875+0000","last_became_peered":"2024-03-06T07:31:40.759875+0000","last_unstale":"2024-03-06T07:31:42.891235+0000","last_undegraded":"2024-03-06T07:31:42.891235+0000","last_fullsized":"2024-03-06T07:31:42.891235+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:33.527138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811725+0000","last_change":"2024-03-06T07:31:41.785333+0000","last_active":"2024-03-06T07:31:42.811725+0000","last_peered":"2024-03-06T07:31:42.811725+0000","last_clean":"2024-03-06T07:31:42.811725+0000","last_became_active":"2024-03-06T07:31:41.784980+0000","last_became_peered":"2024-03-06T07:31:41.784980+0000","last_unstale":"2024-03-06T07:31:42.811725+0000","last_undegraded":"2024-03-06T07:31:42.811725+0000","last_fullsized":"2024-03-06T07:31:42.811725+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:10.283128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.784014+0000","last_change":"2024-03-06T07:31:40.758168+0000","last_active":"2024-03-06T07:31:42.784014+0000","last_peered":"2024-03-06T07:31:42.784014+0000","last_clean":"2024-03-06T07:31:42.784014+0000","last_became_active":"2024-03-06T07:31:40.757763+0000","last_became_peered":"2024-03-06T07:31:40.757763+0000","last_unstale":"2024-03-06T07:31:42.784014+0000","last_undegraded":"2024-03-06T07:31:42.784014+0000","last_fullsized":"2024-03-06T07:31:42.784014+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:27.345029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811457+0000","last_change":"2024-03-06T07:31:41.792651+0000","last_active":"2024-03-06T07:31:42.811457+0000","last_peered":"2024-03-06T07:31:42.811457+0000","last_clean":"2024-03-06T07:31:42.811457+0000","last_became_active":"2024-03-06T07:31:41.786755+0000","last_became_peered":"2024-03-06T07:31:41.786755+0000","last_unstale":"2024-03-06T07:31:42.811457+0000","last_undegraded":"2024-03-06T07:31:42.811457+0000","last_fullsized":"2024-03-06T07:31:42.811457+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:18.792440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892595+0000","last_change":"2024-03-06T07:31:40.763728+0000","last_active":"2024-03-06T07:31:42.892595+0000","last_peered":"2024-03-06T07:31:42.892595+0000","last_clean":"2024-03-06T07:31:42.892595+0000","last_became_active":"2024-03-06T07:31:40.763400+0000","last_became_peered":"2024-03-06T07:31:40.763400+0000","last_unstale":"2024-03-06T07:31:42.892595+0000","last_undegraded":"2024-03-06T07:31:42.892595+0000","last_fullsized":"2024-03-06T07:31:42.892595+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:27.109170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.895554+0000","last_change":"2024-03-06T07:31:41.787928+0000","last_active":"2024-03-06T07:31:42.895554+0000","last_peered":"2024-03-06T07:31:42.895554+0000","last_clean":"2024-03-06T07:31:42.895554+0000","last_became_active":"2024-03-06T07:31:41.787610+0000","last_became_peered":"2024-03-06T07:31:41.787610+0000","last_unstale":"2024-03-06T07:31:42.895554+0000","last_undegraded":"2024-03-06T07:31:42.895554+0000","last_fullsized":"2024-03-06T07:31:42.895554+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:00.316298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783985+0000","last_change":"2024-03-06T07:31:40.750338+0000","last_active":"2024-03-06T07:31:42.783985+0000","last_peered":"2024-03-06T07:31:42.783985+0000","last_clean":"2024-03-06T07:31:42.783985+0000","last_became_active":"2024-03-06T07:31:40.749957+0000","last_became_peered":"2024-03-06T07:31:40.749957+0000","last_unstale":"2024-03-06T07:31:42.783985+0000","last_undegraded":"2024-03-06T07:31:42.783985+0000","last_fullsized":"2024-03-06T07:31:42.783985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:25.222365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.896156+0000","last_change":"2024-03-06T07:31:41.789450+0000","last_active":"2024-03-06T07:31:42.896156+0000","last_peered":"2024-03-06T07:31:42.896156+0000","last_clean":"2024-03-06T07:31:42.896156+0000","last_became_active":"2024-03-06T07:31:41.787795+0000","last_became_peered":"2024-03-06T07:31:41.787795+0000","last_unstale":"2024-03-06T07:31:42.896156+0000","last_undegraded":"2024-03-06T07:31:42.896156+0000","last_fullsized":"2024-03-06T07:31:42.896156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:39.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.815305+0000","last_change":"2024-03-06T07:31:40.760633+0000","last_active":"2024-03-06T07:31:42.815305+0000","last_peered":"2024-03-06T07:31:42.815305+0000","last_clean":"2024-03-06T07:31:42.815305+0000","last_became_active":"2024-03-06T07:31:40.760309+0000","last_became_peered":"2024-03-06T07:31:40.760309+0000","last_unstale":"2024-03-06T07:31:42.815305+0000","last_undegraded":"2024-03-06T07:31:42.815305+0000","last_fullsized":"2024-03-06T07:31:42.815305+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:29.515860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.784135+0000","last_change":"2024-03-06T07:31:41.783333+0000","last_active":"2024-03-06T07:31:42.784135+0000","last_peered":"2024-03-06T07:31:42.784135+0000","last_clean":"2024-03-06T07:31:42.784135+0000","last_became_active":"2024-03-06T07:31:41.782435+0000","last_became_peered":"2024-03-06T07:31:41.782435+0000","last_unstale":"2024-03-06T07:31:42.784135+0000","last_undegraded":"2024-03-06T07:31:42.784135+0000","last_fullsized":"2024-03-06T07:31:42.784135+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:26.100782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813687+0000","last_change":"2024-03-06T07:31:40.762392+0000","last_active":"2024-03-06T07:31:42.813687+0000","last_peered":"2024-03-06T07:31:42.813687+0000","last_clean":"2024-03-06T07:31:42.813687+0000","last_became_active":"2024-03-06T07:31:40.762033+0000","last_became_peered":"2024-03-06T07:31:40.762033+0000","last_unstale":"2024-03-06T07:31:42.813687+0000","last_undegraded":"2024-03-06T07:31:42.813687+0000","last_fullsized":"2024-03-06T07:31:42.813687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:25.222365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.811949+0000","last_change":"2024-03-06T07:31:41.802844+0000","last_active":"2024-03-06T07:31:42.811949+0000","last_peered":"2024-03-06T07:31:42.811949+0000","last_clean":"2024-03-06T07:31:42.811949+0000","last_became_active":"2024-03-06T07:31:41.800880+0000","last_became_peered":"2024-03-06T07:31:41.800880+0000","last_unstale":"2024-03-06T07:31:42.811949+0000","last_undegraded":"2024-03-06T07:31:42.811949+0000","last_fullsized":"2024-03-06T07:31:42.811949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:58.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894746+0000","last_change":"2024-03-06T07:31:40.767571+0000","last_active":"2024-03-06T07:31:42.894746+0000","last_peered":"2024-03-06T07:31:42.894746+0000","last_clean":"2024-03-06T07:31:42.894746+0000","last_became_active":"2024-03-06T07:31:40.767311+0000","last_became_peered":"2024-03-06T07:31:40.767311+0000","last_unstale":"2024-03-06T07:31:42.894746+0000","last_undegraded":"2024-03-06T07:31:42.894746+0000","last_fullsized":"2024-03-06T07:31:42.894746+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.633372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.891712+0000","last_change":"2024-03-06T07:31:41.769366+0000","last_active":"2024-03-06T07:31:42.891712+0000","last_peered":"2024-03-06T07:31:42.891712+0000","last_clean":"2024-03-06T07:31:42.891712+0000","last_became_active":"2024-03-06T07:31:41.768138+0000","last_became_peered":"2024-03-06T07:31:41.768138+0000","last_unstale":"2024-03-06T07:31:42.891712+0000","last_undegraded":"2024-03-06T07:31:42.891712+0000","last_fullsized":"2024-03-06T07:31:42.891712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:56.100684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783458+0000","last_change":"2024-03-06T07:31:40.759529+0000","last_active":"2024-03-06T07:31:42.783458+0000","last_peered":"2024-03-06T07:31:42.783458+0000","last_clean":"2024-03-06T07:31:42.783458+0000","last_became_active":"2024-03-06T07:31:40.759151+0000","last_became_peered":"2024-03-06T07:31:40.759151+0000","last_unstale":"2024-03-06T07:31:42.783458+0000","last_undegraded":"2024-03-06T07:31:42.783458+0000","last_fullsized":"2024-03-06T07:31:42.783458+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:11.001490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.894811+0000","last_change":"2024-03-06T07:31:41.789384+0000","last_active":"2024-03-06T07:31:42.894811+0000","last_peered":"2024-03-06T07:31:42.894811+0000","last_clean":"2024-03-06T07:31:42.894811+0000","last_became_active":"2024-03-06T07:31:41.789102+0000","last_became_peered":"2024-03-06T07:31:41.789102+0000","last_unstale":"2024-03-06T07:31:42.894811+0000","last_undegraded":"2024-03-06T07:31:42.894811+0000","last_fullsized":"2024-03-06T07:31:42.894811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:04.178439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.788135+0000","last_change":"2024-03-06T07:31:40.761170+0000","last_active":"2024-03-06T07:31:42.788135+0000","last_peered":"2024-03-06T07:31:42.788135+0000","last_clean":"2024-03-06T07:31:42.788135+0000","last_became_active":"2024-03-06T07:31:40.760817+0000","last_became_peered":"2024-03-06T07:31:40.760817+0000","last_unstale":"2024-03-06T07:31:42.788135+0000","last_undegraded":"2024-03-06T07:31:42.788135+0000","last_fullsized":"2024-03-06T07:31:42.788135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:23.952561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783408+0000","last_change":"2024-03-06T07:31:41.778561+0000","last_active":"2024-03-06T07:31:42.783408+0000","last_peered":"2024-03-06T07:31:42.783408+0000","last_clean":"2024-03-06T07:31:42.783408+0000","last_became_active":"2024-03-06T07:31:41.778262+0000","last_became_peered":"2024-03-06T07:31:41.778262+0000","last_unstale":"2024-03-06T07:31:42.783408+0000","last_undegraded":"2024-03-06T07:31:42.783408+0000","last_fullsized":"2024-03-06T07:31:42.783408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:24.910268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.783436+0000","last_change":"2024-03-06T07:31:40.751811+0000","last_active":"2024-03-06T07:31:42.783436+0000","last_peered":"2024-03-06T07:31:42.783436+0000","last_clean":"2024-03-06T07:31:42.783436+0000","last_became_active":"2024-03-06T07:31:40.751538+0000","last_became_peered":"2024-03-06T07:31:40.751538+0000","last_unstale":"2024-03-06T07:31:42.783436+0000","last_undegraded":"2024-03-06T07:31:42.783436+0000","last_fullsized":"2024-03-06T07:31:42.783436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:24.715212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890719+0000","last_change":"2024-03-06T07:31:41.773608+0000","last_active":"2024-03-06T07:31:42.890719+0000","last_peered":"2024-03-06T07:31:42.890719+0000","last_clean":"2024-03-06T07:31:42.890719+0000","last_became_active":"2024-03-06T07:31:41.771480+0000","last_became_peered":"2024-03-06T07:31:41.771480+0000","last_unstale":"2024-03-06T07:31:42.890719+0000","last_undegraded":"2024-03-06T07:31:42.890719+0000","last_fullsized":"2024-03-06T07:31:42.890719+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.685300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890638+0000","last_change":"2024-03-06T07:31:40.763463+0000","last_active":"2024-03-06T07:31:42.890638+0000","last_peered":"2024-03-06T07:31:42.890638+0000","last_clean":"2024-03-06T07:31:42.890638+0000","last_became_active":"2024-03-06T07:31:40.763190+0000","last_became_peered":"2024-03-06T07:31:40.763190+0000","last_unstale":"2024-03-06T07:31:42.890638+0000","last_undegraded":"2024-03-06T07:31:42.890638+0000","last_fullsized":"2024-03-06T07:31:42.890638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:59.302526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.890543+0000","last_change":"2024-03-06T07:31:41.796771+0000","last_active":"2024-03-06T07:31:42.890543+0000","last_peered":"2024-03-06T07:31:42.890543+0000","last_clean":"2024-03-06T07:31:42.890543+0000","last_became_active":"2024-03-06T07:31:41.791064+0000","last_became_peered":"2024-03-06T07:31:41.791064+0000","last_unstale":"2024-03-06T07:31:42.890543+0000","last_undegraded":"2024-03-06T07:31:42.890543+0000","last_fullsized":"2024-03-06T07:31:42.890543+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:40.714803+0000","last_clean_scrub_stamp":"2024-03-06T07:31:40.714803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:49.902893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.813108+0000","last_change":"2024-03-06T07:31:40.771153+0000","last_active":"2024-03-06T07:31:42.813108+0000","last_peered":"2024-03-06T07:31:42.813108+0000","last_clean":"2024-03-06T07:31:42.813108+0000","last_became_active":"2024-03-06T07:31:40.770825+0000","last_became_peered":"2024-03-06T07:31:40.770825+0000","last_unstale":"2024-03-06T07:31:42.813108+0000","last_undegraded":"2024-03-06T07:31:42.813108+0000","last_fullsized":"2024-03-06T07:31:42.813108+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:39.701031+0000","last_clean_scrub_stamp":"2024-03-06T07:31:39.701031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:11.001490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:31:42.892976+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:31:42.892976+0000","last_peered":"2024-03-06T07:31:42.892976+0000","last_clean":"2024-03-06T07:31:42.892976+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:31:42.892976+0000","last_undegraded":"2024-03-06T07:31:42.892976+0000","last_fullsized":"2024-03-06T07:31:42.892976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1286144,"data_stored":873498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":283622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":278282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":276726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":276726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":688128,"data_stored":283060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":875532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:46.782 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:31:46.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:31:47.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.048+0000 7f3dd1c00640 1 Processor -- start 2024-03-06T07:31:47.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.048+0000 7f3dd1c00640 1 -- start start 2024-03-06T07:31:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.050+0000 7f3dd1c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3dcc0d0240 0x7f3dcc0727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.050+0000 7f3dd1c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3dcc15e580 0x7f3dcc15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.050+0000 7f3dd1c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc15f110 con 0x7f3dcc06afb0 2024-03-06T07:31:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.050+0000 7f3dd1c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dcc071da0 con 0x7f3dcc0d0240 2024-03-06T07:31:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dd1c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dcc15f3f0 con 0x7f3dcc15e580 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dcaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3dcc0d0240 0x7f3dcc0727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dcaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3dcc0d0240 0x7f3dcc0727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59244/0 (socket says 172.21.15.32:59244) 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dcaa00640 1 -- 172.21.15.32:0/867482325 learned_addr learned my addr 172.21.15.32:0/867482325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dc2000640 1 --2- 172.21.15.32:0/867482325 >> v2:172.21.15.39:3300/0 conn(0x7f3dcc15e580 0x7f3dcc15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dcaa00640 1 -- 172.21.15.32:0/867482325 >> v2:172.21.15.39:3300/0 conn(0x7f3dcc15e580 msgr2=0x7f3dcc15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dcaa00640 1 --2- 172.21.15.32:0/867482325 >> v2:172.21.15.39:3300/0 conn(0x7f3dcc15e580 0x7f3dcc15ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.051+0000 7f3dcaa00640 1 -- 172.21.15.32:0/867482325 >> v1:172.21.15.39:6789/0 conn(0x7f3dcc06afb0 legacy=0x7f3dcc0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.052+0000 7f3dcaa00640 1 -- 172.21.15.32:0/867482325 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc071fd0 con 0x7f3dcc0d0240 2024-03-06T07:31:47.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.053+0000 7f3dcaa00640 1 --2- 172.21.15.32:0/867482325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc0727d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc06c070 tx=0x7f3dbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440775a87b5e8896 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.053+0000 7f3dca000640 1 -- 172.21.15.32:0/867482325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc035020 con 0x7f3dcc0d0240 2024-03-06T07:31:47.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.053+0000 7f3dca000640 1 -- 172.21.15.32:0/867482325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc031870 con 0x7f3dcc0d0240 2024-03-06T07:31:47.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.053+0000 7f3dca000640 1 -- 172.21.15.32:0/867482325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc031c40 con 0x7f3dcc0d0240 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 -- 172.21.15.32:0/867482325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 msgr2=0x7f3dcc0727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/867482325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc0727d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc06c070 tx=0x7f3dbc02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 -- 172.21.15.32:0/867482325 shutdown_connections 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/867482325 >> v2:172.21.15.39:3300/0 conn(0x7f3dcc15e580 0x7f3dcc15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/867482325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc0727d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 -- 172.21.15.32:0/867482325 >> 172.21.15.32:0/867482325 conn(0x7f3dcc0b2a00 msgr2=0x7f3dcc0b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 -- 172.21.15.32:0/867482325 shutdown_connections 2024-03-06T07:31:47.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.054+0000 7f3dd1c00640 1 -- 172.21.15.32:0/867482325 wait complete. 2024-03-06T07:31:47.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.055+0000 7f3dd1c00640 1 Processor -- start 2024-03-06T07:31:47.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.056+0000 7f3dd1c00640 1 -- start start 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.056+0000 7f3dd1c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dcc06afb0 0x7f3dcc14a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.056+0000 7f3dd1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc1e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dd1c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dcc15e580 0x7f3dcc15b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dd1c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc072d10 con 0x7f3dcc0d0240 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dd1c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0b00b0 con 0x7f3dcc06afb0 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dd1c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0d0620 con 0x7f3dcc15e580 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc1e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc1e7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59252/0 (socket says 172.21.15.32:59252) 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 -- 172.21.15.32:0/4084717740 learned_addr learned my addr 172.21.15.32:0/4084717740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dcaa00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dcc06afb0 0x7f3dcc14a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 -- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dcc15e580 msgr2=0x7f3dcc15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dcc15e580 0x7f3dcc15b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 -- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dcc06afb0 msgr2=0x7f3dcc14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dcc06afb0 0x7f3dcc14a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.057+0000 7f3dc2000640 1 -- 172.21.15.32:0/4084717740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc14aaf0 con 0x7f3dcc0d0240 2024-03-06T07:31:47.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.058+0000 7f3dc2000640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc1e7d30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3db400d620 tx=0x7f3db400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.058+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db400b620 con 0x7f3dcc0d0240 2024-03-06T07:31:47.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.058+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db4019070 con 0x7f3dcc0d0240 2024-03-06T07:31:47.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.058+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db400ba00 con 0x7f3dcc0d0240 2024-03-06T07:31:47.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.058+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dcc1e8760 con 0x7f3dcc0d0240 2024-03-06T07:31:47.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.058+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dcc1e8d20 con 0x7f3dcc0d0240 2024-03-06T07:31:47.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.059+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dcc068720 con 0x7f3dcc0d0240 2024-03-06T07:31:47.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.061+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3db400bba0 con 0x7f3dcc0d0240 2024-03-06T07:31:47.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.061+0000 7f3dc3400640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da806b4d0 0x7f3da806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.062+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3db408a570 con 0x7f3dcc0d0240 2024-03-06T07:31:47.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.062+0000 7f3dcaa00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da806b4d0 0x7f3da806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.063+0000 7f3dcaa00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da806b4d0 0x7f3da806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc036000 tx=0x7f3dbc03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.066+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3db4059480 con 0x7f3dcc0d0240 2024-03-06T07:31:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.493+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3dcc06bbb0 con 0x7f3dcc0d0240 2024-03-06T07:31:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.497+0000 7f3dc3400640 1 -- 172.21.15.32:0/4084717740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3db4058d80 con 0x7f3dcc0d0240 2024-03-06T07:31:47.499 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:31:47.499 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:31:47.499 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:31:47.499 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:31:47.499 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:31:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da806b4d0 msgr2=0x7f3da806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da806b4d0 0x7f3da806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc036000 tx=0x7f3dbc03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 msgr2=0x7f3dcc1e7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc1e7d30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3db400d620 tx=0x7f3db400dae0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 shutdown_connections 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3da806b4d0 0x7f3da806d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3dcc15e580 0x7f3dcc15b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dcc0d0240 0x7f3dcc1e7d30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 --2- 172.21.15.32:0/4084717740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dcc06afb0 0x7f3dcc14a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 >> 172.21.15.32:0/4084717740 conn(0x7f3dcc0b2a00 msgr2=0x7f3dcc160e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 shutdown_connections 2024-03-06T07:31:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.499+0000 7f3dd1c00640 1 -- 172.21.15.32:0/4084717740 wait complete. 2024-03-06T07:31:47.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.686 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-03-06T07:31:47.684+0000 7f13c2000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.727+0000 7fdc9be00640 1 Processor -- start 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.727+0000 7fdc9be00640 1 -- start start 2024-03-06T07:31:47.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdc9c15e0b0 0x7fdc9c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9be00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0b3710 con 0x7fdc9c0d0e50 2024-03-06T07:31:47.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0b39f0 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c0b09e0 con 0x7fdc9c15e0b0 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdc9c15e0b0 0x7fdc9c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdc9c0d0e50 0x7fdc9c0b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39928/0 (socket says 172.21.15.32:39928) 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.729+0000 7fdc9b400640 1 -- 172.21.15.32:0/469542314 learned_addr learned my addr 172.21.15.32:0/469542314 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.730+0000 7fdc9aa00640 1 -- 172.21.15.32:0/469542314 >> v1:172.21.15.39:6789/0 conn(0x7fdc9c0d4a10 legacy=0x7fdc9c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.730+0000 7fdc9aa00640 1 -- 172.21.15.32:0/469542314 >> v1:172.21.15.32:6789/0 conn(0x7fdc9c0d0e50 legacy=0x7fdc9c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.730+0000 7fdc9aa00640 1 -- 172.21.15.32:0/469542314 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0b3cd0 con 0x7fdc9c15e0b0 2024-03-06T07:31:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.732+0000 7fdc9aa00640 1 --2- 172.21.15.32:0/469542314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c0cf310 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009d90 tx=0x7fdc8c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad0dd02a179cf9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.732+0000 7fdc99600640 1 -- 172.21.15.32:0/469542314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c039070 con 0x7fdc9c15e0b0 2024-03-06T07:31:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.732+0000 7fdc99600640 1 -- 172.21.15.32:0/469542314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c02cc60 con 0x7fdc9c15e0b0 2024-03-06T07:31:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.732+0000 7fdc99600640 1 -- 172.21.15.32:0/469542314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c034660 con 0x7fdc9c15e0b0 2024-03-06T07:31:47.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 -- 172.21.15.32:0/469542314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c15e0b0 msgr2=0x7fdc9c0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 --2- 172.21.15.32:0/469542314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c0cf310 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009d90 tx=0x7fdc8c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 -- 172.21.15.32:0/469542314 shutdown_connections 2024-03-06T07:31:47.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 --2- 172.21.15.32:0/469542314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c0cf310 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 -- 172.21.15.32:0/469542314 >> 172.21.15.32:0/469542314 conn(0x7fdc9c0d3d30 msgr2=0x7fdc9c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 -- 172.21.15.32:0/469542314 shutdown_connections 2024-03-06T07:31:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.733+0000 7fdc9be00640 1 -- 172.21.15.32:0/469542314 wait complete. 2024-03-06T07:31:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.735+0000 7fdc9be00640 1 Processor -- start 2024-03-06T07:31:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.735+0000 7fdc9be00640 1 -- start start 2024-03-06T07:31:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc9c0d0e50 0x7fdc9c1f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 0x7fdc9c0cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0cfc60 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c15e490 con 0x7fdc9c15e0b0 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0cf850 con 0x7fdc9c0d0e50 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 0x7fdc9c0cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 0x7fdc9c0cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59262/0 (socket says 172.21.15.32:59262) 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.737+0000 7fdc9a000640 1 -- 172.21.15.32:0/4124327233 learned_addr learned my addr 172.21.15.32:0/4124327233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9aa00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc9c0d0e50 0x7fdc9c1f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9b400640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9a000640 1 -- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc9c0d0e50 msgr2=0x7fdc9c1f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9a000640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc9c0d0e50 0x7fdc9c1f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9a000640 1 -- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc9c15e0b0 msgr2=0x7fdc9c155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9a000640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9a000640 1 -- 172.21.15.32:0/4124327233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c1f75f0 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.738+0000 7fdc9a000640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 0x7fdc9c0cdaa0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800c400 tx=0x7fdc8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.739+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88011790 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.739+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88017070 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.739+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88011bb0 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.739+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c1f78d0 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.739+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c15fc40 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.740+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c15fe40 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.742+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc8801c460 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.743+0000 7fdc92a00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.743+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc8808a060 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.743+0000 7fdc9aa00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.744+0000 7fdc9aa00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.752+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc9c15fe40 con 0x7fdc9c0d4a10 2024-03-06T07:31:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.997+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc9c04f330 con 0x7fdc9c0d4a10 2024-03-06T07:31:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:47.998+0000 7fdc92a00640 1 -- 172.21.15.32:0/4124327233 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fdc88058ed0 con 0x7fdc9c0d4a10 2024-03-06T07:31:48.000 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:31:48.000 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:31:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:31:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc7c06b4d0 msgr2=0x7fdc7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c036000 tx=0x7fdc8c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 msgr2=0x7fdc9c0cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 0x7fdc9c0cdaa0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800c400 tx=0x7fdc8800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 shutdown_connections 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc9c15e0b0 0x7fdc9c155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc9c0d4a10 0x7fdc9c0cdaa0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 --2- 172.21.15.32:0/4124327233 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc9c0d0e50 0x7fdc9c1f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 >> 172.21.15.32:0/4124327233 conn(0x7fdc9c0d3d30 msgr2=0x7fdc9c0b5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 shutdown_connections 2024-03-06T07:31:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.001+0000 7fdc9be00640 1 -- 172.21.15.32:0/4124327233 wait complete. 2024-03-06T07:31:48.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:41.207788+0000', 'modified': '2024-03-06T07:31:44.292199+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5907}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5907': {'gid': 5907, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6837/436488985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 436488985}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 436488985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5907, 'qdb_cluster': [5907]} 2024-03-06T07:31:48.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:48.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:31:48.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.651+0000 7f9d2d800640 1 Processor -- start 2024-03-06T07:31:48.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.651+0000 7f9d2d800640 1 -- start start 2024-03-06T07:31:48.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d2d800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9d280ce850 0x7f9d280cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d2d800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280cce90 0x7f9d2806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d2d800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d280cdad0 con 0x7f9d2815c920 2024-03-06T07:31:48.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d2d800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d28071500 con 0x7f9d280ce850 2024-03-06T07:31:48.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d2d800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d280cdd70 con 0x7f9d280cce90 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d27e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9d2815c920 0x7f9d28070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53536/0 (socket says 172.21.15.32:53536) 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.653+0000 7f9d27e00640 1 -- 172.21.15.32:0/3971570913 learned_addr learned my addr 172.21.15.32:0/3971570913 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d27400640 1 --2- 172.21.15.32:0/3971570913 >> v2:172.21.15.32:3300/0 conn(0x7f9d280ce850 0x7f9d280cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/3971570913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280cce90 0x7f9d2806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d26a00640 1 -- 172.21.15.32:0/3971570913 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212121587 0 0) 0x7f9d280cdad0 con 0x7f9d2815c920 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d27400640 1 -- 172.21.15.32:0/3971570913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280cce90 msgr2=0x7f9d2806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d27400640 1 --2- 172.21.15.32:0/3971570913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280cce90 0x7f9d2806cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d27400640 1 -- 172.21.15.32:0/3971570913 >> v1:172.21.15.32:6789/0 conn(0x7f9d2815c920 legacy=0x7f9d28070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.654+0000 7f9d27400640 1 -- 172.21.15.32:0/3971570913 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d280ce050 con 0x7f9d280ce850 2024-03-06T07:31:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.655+0000 7f9d27400640 1 --2- 172.21.15.32:0/3971570913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d280ce850 0x7f9d280cc950 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9d100097a0 tx=0x7f9d1002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ed5002f28c6906 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d26a00640 1 -- 172.21.15.32:0/3971570913 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10035020 con 0x7f9d280ce850 2024-03-06T07:31:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d26a00640 1 -- 172.21.15.32:0/3971570913 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d10031810 con 0x7f9d280ce850 2024-03-06T07:31:48.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d26a00640 1 -- 172.21.15.32:0/3971570913 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10031be0 con 0x7f9d280ce850 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 -- 172.21.15.32:0/3971570913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d280ce850 msgr2=0x7f9d280cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 --2- 172.21.15.32:0/3971570913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d280ce850 0x7f9d280cc950 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9d100097a0 tx=0x7f9d1002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 -- 172.21.15.32:0/3971570913 shutdown_connections 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 --2- 172.21.15.32:0/3971570913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d280ce850 0x7f9d280cc950 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 --2- 172.21.15.32:0/3971570913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280cce90 0x7f9d2806cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 -- 172.21.15.32:0/3971570913 >> 172.21.15.32:0/3971570913 conn(0x7f9d2806b3d0 msgr2=0x7f9d2806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 -- 172.21.15.32:0/3971570913 shutdown_connections 2024-03-06T07:31:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.656+0000 7f9d2d800640 1 -- 172.21.15.32:0/3971570913 wait complete. 2024-03-06T07:31:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.658+0000 7f9d2d800640 1 Processor -- start 2024-03-06T07:31:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.658+0000 7f9d2d800640 1 -- start start 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.659+0000 7f9d2d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d280cce90 0x7f9d280b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.659+0000 7f9d2d800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280ce850 0x7f9d280b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.659+0000 7f9d2d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 0x7f9d281f8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.659+0000 7f9d2d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28071150 con 0x7f9d2815c920 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.659+0000 7f9d2d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28163790 con 0x7f9d280cce90 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.659+0000 7f9d2d800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d280b16e0 con 0x7f9d280ce850 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 0x7f9d281f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 0x7f9d281f8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46040/0 (socket says 172.21.15.32:46040) 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 -- 172.21.15.32:0/677243311 learned_addr learned my addr 172.21.15.32:0/677243311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d27e00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d280cce90 0x7f9d280b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d27400640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280ce850 0x7f9d280b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 -- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280ce850 msgr2=0x7f9d280b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280ce850 0x7f9d280b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 -- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d280cce90 msgr2=0x7f9d280b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d280cce90 0x7f9d280b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 -- 172.21.15.32:0/677243311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d280b1910 con 0x7f9d2815c920 2024-03-06T07:31:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.660+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 0x7f9d281f8460 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000d800 tx=0x7f9d2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.661+0000 7f9d1fe00640 1 -- 172.21.15.32:0/677243311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000b6c0 con 0x7f9d2815c920 2024-03-06T07:31:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.661+0000 7f9d1fe00640 1 -- 172.21.15.32:0/677243311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20019070 con 0x7f9d2815c920 2024-03-06T07:31:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.661+0000 7f9d1fe00640 1 -- 172.21.15.32:0/677243311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000bac0 con 0x7f9d2815c920 2024-03-06T07:31:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.661+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2815cf60 con 0x7f9d2815c920 2024-03-06T07:31:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.661+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2815d520 con 0x7f9d2815c920 2024-03-06T07:31:48.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.663+0000 7f9d1fe00640 1 -- 172.21.15.32:0/677243311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d2000bc60 con 0x7f9d2815c920 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.664+0000 7f9d1fe00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d0406b4d0 0x7f9d0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.664+0000 7f9d1fe00640 1 -- 172.21.15.32:0/677243311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d2008a710 con 0x7f9d2815c920 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.664+0000 7f9d27e00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d0406b4d0 0x7f9d0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.665+0000 7f9d31ba6740 1 Processor -- start 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.665+0000 7f9d31ba6740 1 -- start start 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.665+0000 7f9d27e00640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d0406b4d0 0x7f9d0406d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9d180012e0 tx=0x7f9d18009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 0x55ac7619e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac7619ee70 0x55ac761cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac76108600 con 0x55ac761cb990 2024-03-06T07:31:48.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac760b1930 con 0x55ac761cbd70 2024-03-06T07:31:48.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac761046a0 con 0x55ac7619ee70 2024-03-06T07:31:48.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d27e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 0x55ac7619e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d27e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 0x55ac7619e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46052/0 (socket says 172.21.15.32:46052) 2024-03-06T07:31:48.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.666+0000 7f9d27e00640 1 -- 172.21.15.32:0/2871600868 learned_addr learned my addr 172.21.15.32:0/2871600868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27400640 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27e00640 1 -- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac7619ee70 msgr2=0x55ac761cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac7619ee70 0x55ac761cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27e00640 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac7619ee70 0x55ac761cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27e00640 1 -- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 msgr2=0x55ac761ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27e00640 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27e00640 1 -- 172.21.15.32:0/2871600868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac75e1d2d0 con 0x55ac761cb990 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27400640 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d27e00640 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 0x55ac7619e930 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800d7d0 tx=0x7f9d18010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e0a3efdad6a14d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d1cc00640 1 -- 172.21.15.32:0/2871600868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18007850 con 0x55ac761cb990 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d1cc00640 1 -- 172.21.15.32:0/2871600868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d180079f0 con 0x55ac761cb990 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.667+0000 7f9d1cc00640 1 -- 172.21.15.32:0/2871600868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18020650 con 0x55ac761cb990 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2871600868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 msgr2=0x55ac7619e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 0x55ac7619e930 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800d7d0 tx=0x7f9d18010000 comp rx=0 tx=0).stop 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2871600868 shutdown_connections 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac7619ee70 0x55ac761cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2871600868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cb990 0x55ac7619e930 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2871600868 >> 172.21.15.32:0/2871600868 conn(0x55ac761a0ba0 msgr2=0x55ac761e9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2871600868 shutdown_connections 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.668+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2871600868 wait complete. 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.669+0000 7f9d31ba6740 1 -- start start 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.669+0000 7f9d31ba6740 1 Processor -- start 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cb990 0x55ac761b9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cbd70 0x55ac761d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac761e41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac761a3140 con 0x55ac761e3d50 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac760c4d90 con 0x55ac761cb990 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac761a2530 con 0x55ac761cbd70 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.670+0000 7f9d31ba6740 10 client.0 _ll_register_callbacks cb 0x55ac76100350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac761e41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac761e41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46066/0 (socket says 172.21.15.32:46066) 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 -- 172.21.15.32:0/3225955457 learned_addr learned my addr 172.21.15.32:0/3225955457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d27e00640 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cb990 0x55ac761b9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 -- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cbd70 msgr2=0x55ac761d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cbd70 0x55ac761d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 -- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cb990 msgr2=0x55ac761b9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cb990 0x55ac761b9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 -- 172.21.15.32:0/3225955457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac75ee76b0 con 0x55ac761e3d50 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac761e41d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9d20006fd0 tx=0x7f9d20059660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e1ee01ee1d520 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.671+0000 7f9d15600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d15600640 1 -- 172.21.15.32:0/3225955457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2005de60 con 0x55ac761e3d50 2024-03-06T07:31:48.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d15600640 1 -- 172.21.15.32:0/3225955457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d20055460 con 0x55ac761e3d50 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d180079f0 con 0x55ac761e3d50 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d15600640 1 -- 172.21.15.32:0/3225955457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d20070b80 con 0x55ac761e3d50 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 10 client.6048 fetch_fsmap learned FSMap version 34 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55ac761d24d0 con 0x55ac761e3d50 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d15600640 1 -- 172.21.15.32:0/3225955457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d180079f0 con 0x55ac761e3d50 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d15600640 1 -- 172.21.15.32:0/3225955457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9d20013070 con 0x55ac761e3d50 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 10 client.6048 fetch_fsmap finished waiting for FSMap version 34 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 10 client.6048 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 20 client.6048 populate_metadata read hostname 'smithi032' 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac761fd810 0x55ac761beaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 4 client.6048 mds_command: new command op to 5907 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.672+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 --> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac76115b40 con 0x55ac761fd810 2024-03-06T07:31:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.673+0000 7f9d27e00640 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac761fd810 0x55ac761beaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.673+0000 7f9d27e00640 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac761fd810 0x55ac761beaf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2d4d84efc39a80d server_cookie=8daa510b1ab66b09 in_seq=0 out_seq=0 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.673+0000 7f9d15600640 10 client.6048 ms_handle_connect on v2:172.21.15.32:6836/436488985 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.674+0000 7f9d15600640 1 -- 172.21.15.32:0/3225955457 <== mds.0 v2:172.21.15.32:6836/436488985 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ac76115b40 con 0x55ac761fd810 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.674+0000 7f9d15600640 10 client.6048 handle_command_reply: tid=0 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.674+0000 7f9d31ba6740 1 client.6048 shutdown 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.674+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 msgr2=0x55ac761e41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.674+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac761e41d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9d20006fd0 tx=0x7f9d20059660 comp rx=0 tx=0).stop 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 shutdown_connections 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac761fd810 0x55ac761beaf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac761e41d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cbd70 0x55ac761d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3225955457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cb990 0x55ac761b9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 >> 172.21.15.32:0/3225955457 conn(0x55ac7619ee70 msgr2=0x55ac761ce1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d15600640 0 client.6048 ms_handle_reset on v2:172.21.15.32:6836/436488985 2024-03-06T07:31:48.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 shutdown_connections 2024-03-06T07:31:48.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3225955457 wait complete. 2024-03-06T07:31:48.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.675+0000 7f9d31ba6740 20 client.6048 trim_cache size 0 max 16384 2024-03-06T07:31:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.698+0000 7f9d31ba6740 1 Processor -- start 2024-03-06T07:31:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.698+0000 7f9d31ba6740 1 -- start start 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.698+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 0x55ac761ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761e3d50 0x55ac761e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28203470 con 0x55ac761cbd70 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28159900 con 0x55ac761e3d50 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d281fcd90 con 0x55ac761cb990 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 0x55ac761ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 0x55ac761ecc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46072/0 (socket says 172.21.15.32:46072) 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 -- 172.21.15.32:0/3115761195 learned_addr learned my addr 172.21.15.32:0/3115761195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27e00640 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761e3d50 0x55ac761e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 -- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 msgr2=0x55ac761a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 -- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761e3d50 msgr2=0x55ac761e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761e3d50 0x55ac761e32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27400640 1 -- 172.21.15.32:0/3115761195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac761e42f0 con 0x55ac761cbd70 2024-03-06T07:31:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.699+0000 7f9d27e00640 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.700+0000 7f9d27400640 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 0x55ac761ecc30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9d1002bec0 tx=0x7f9d1002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0e5210e6232bca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.700+0000 7f9d17e00640 1 -- 172.21.15.32:0/3115761195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10035040 con 0x55ac761cbd70 2024-03-06T07:31:48.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.700+0000 7f9d17e00640 1 -- 172.21.15.32:0/3115761195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d10031910 con 0x55ac761cbd70 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.700+0000 7f9d17e00640 1 -- 172.21.15.32:0/3115761195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10031d00 con 0x55ac761cbd70 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3115761195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 msgr2=0x55ac761ecc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 0x55ac761ecc30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9d1002bec0 tx=0x7f9d1002e000 comp rx=0 tx=0).stop 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3115761195 shutdown_connections 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761e3d50 0x55ac761e32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761cbd70 0x55ac761ecc30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/3115761195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3115761195 >> 172.21.15.32:0/3115761195 conn(0x55ac7619ee70 msgr2=0x55ac761b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3115761195 shutdown_connections 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.701+0000 7f9d31ba6740 1 -- 172.21.15.32:0/3115761195 wait complete. 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.702+0000 7f9d31ba6740 1 -- start start 2024-03-06T07:31:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.702+0000 7f9d31ba6740 1 Processor -- start 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761b9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac76258f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac761fd690 con 0x55ac761e3d50 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac761bdd40 con 0x55ac761cbd70 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac761fdc60 con 0x55ac761cb990 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d31ba6740 10 client.0 _ll_register_callbacks cb 0x7f9d200046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d2ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac76258f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d2ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac76258f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46082/0 (socket says 172.21.15.32:46082) 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d2ce00640 1 -- 172.21.15.32:0/2404206970 learned_addr learned my addr 172.21.15.32:0/2404206970 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.704+0000 7f9d27e00640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d27400640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d2ce00640 1 -- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 msgr2=0x55ac761b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d2ce00640 1 -- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 msgr2=0x55ac761b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761b9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d2ce00640 1 -- 172.21.15.32:0/2404206970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac7625d8d0 con 0x55ac761e3d50 2024-03-06T07:31:48.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d2ce00640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac76258f20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000b4b0 tx=0x7f9d20020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aad528867571891 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d15600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d15600640 1 -- 172.21.15.32:0/2404206970 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2005de60 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d15600640 1 -- 172.21.15.32:0/2404206970 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d2005f0c0 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d10031910 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d15600640 1 -- 172.21.15.32:0/2404206970 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d20058240 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d31ba6740 10 client.6054 fetch_fsmap learned FSMap version 34 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.705+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55ac761b9790 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d15600640 1 -- 172.21.15.32:0/2404206970 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d10031910 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d15600640 1 -- 172.21.15.32:0/2404206970 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9d20057ec0 con 0x55ac761e3d50 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d31ba6740 10 client.6054 fetch_fsmap finished waiting for FSMap version 34 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d31ba6740 10 client.6054 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d31ba6740 20 client.6054 populate_metadata read hostname 'smithi032' 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac7625d140 0x55ac761a3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d31ba6740 4 client.6054 mds_command: new command op to 5907 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 --> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ac7610fe90 con 0x55ac7625d140 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.706+0000 7f9d27e00640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac7625d140 0x55ac761a3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d15600640 10 client.6054 ms_handle_connect on v2:172.21.15.32:6836/436488985 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d27e00640 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac7625d140 0x55ac761a3120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86216cc560549512 server_cookie=d03cf8dc64fc2b54 in_seq=0 out_seq=0 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d15600640 1 -- 172.21.15.32:0/2404206970 <== mds.0 v2:172.21.15.32:6836/436488985 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ac7610fe90 con 0x55ac7625d140 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d15600640 10 client.6054 handle_command_reply: tid=0 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 client.6054 shutdown 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 msgr2=0x55ac76258f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac76258f20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000b4b0 tx=0x7f9d20020070 comp rx=0 tx=0).stop 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 shutdown_connections 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:6836/436488985,v1:172.21.15.32:6837/436488985] conn(0x55ac7625d140 0x55ac761a3120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ac761e3d50 0x55ac76258f20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac761cbd70 0x55ac761b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 --2- 172.21.15.32:0/2404206970 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55ac761cb990 0x55ac761b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d15600640 0 client.6054 ms_handle_reset on v2:172.21.15.32:6836/436488985 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 >> 172.21.15.32:0/2404206970 conn(0x55ac7619ee70 msgr2=0x55ac761e0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 shutdown_connections 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.707+0000 7f9d31ba6740 1 -- 172.21.15.32:0/2404206970 wait complete. 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d31ba6740 20 client.6054 trim_cache size 0 max 16384 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5907, 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:31:48.713 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.9317235000000004, 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 34, 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 57, 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 57, 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.336001720000001 2024-03-06T07:31:48.714 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d0406b4d0 msgr2=0x7f9d0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d0406b4d0 0x7f9d0406d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9d180012e0 tx=0x7f9d18009040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 msgr2=0x7f9d281f8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 0x7f9d281f8460 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000d800 tx=0x7f9d2000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 shutdown_connections 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d0406b4d0 0x7f9d0406d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d2815c920 0x7f9d281f8460 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d280ce850 0x7f9d280b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 --2- 172.21.15.32:0/677243311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d280cce90 0x7f9d280b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.708+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 >> 172.21.15.32:0/677243311 conn(0x7f9d2806b3d0 msgr2=0x7f9d281628b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.709+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 shutdown_connections 2024-03-06T07:31:48.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.709+0000 7f9d2d800640 1 -- 172.21.15.32:0/677243311 wait complete. 2024-03-06T07:31:48.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:48.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.924+0000 7fa72ee00640 1 Processor -- start 2024-03-06T07:31:48.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.925+0000 7fa72ee00640 1 -- start start 2024-03-06T07:31:48.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa72ee00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7280d2bc0 0x7fa7280d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa72ee00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa72815e4a0 0x7fa7280c8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa72ee00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280d1160 con 0x7fa72815e0c0 2024-03-06T07:31:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa72ee00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7280d0f10 con 0x7fa72815e4a0 2024-03-06T07:31:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa72ee00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7280d21f0 con 0x7fa7280d2bc0 2024-03-06T07:31:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa72ce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa72815e4a0 0x7fa7280c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa727e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa72815e0c0 0x7fa7280d24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53540/0 (socket says 172.21.15.32:53540) 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.926+0000 7fa727e00640 1 -- 172.21.15.32:0/4029879052 learned_addr learned my addr 172.21.15.32:0/4029879052 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.927+0000 7fa71f400640 1 --2- 172.21.15.32:0/4029879052 >> v2:172.21.15.39:3300/0 conn(0x7fa7280d2bc0 0x7fa7280d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.927+0000 7fa72ce00640 1 -- 172.21.15.32:0/4029879052 >> v2:172.21.15.39:3300/0 conn(0x7fa7280d2bc0 msgr2=0x7fa7280d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.927+0000 7fa72ce00640 1 --2- 172.21.15.32:0/4029879052 >> v2:172.21.15.39:3300/0 conn(0x7fa7280d2bc0 0x7fa7280d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.927+0000 7fa72ce00640 1 -- 172.21.15.32:0/4029879052 >> v1:172.21.15.32:6789/0 conn(0x7fa72815e0c0 legacy=0x7fa7280d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.927+0000 7fa72ce00640 1 -- 172.21.15.32:0/4029879052 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280d1440 con 0x7fa72815e4a0 2024-03-06T07:31:48.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.927+0000 7fa727400640 1 -- 172.21.15.32:0/4029879052 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170102242 0 0) 0x7fa7280d1160 con 0x7fa72815e0c0 2024-03-06T07:31:48.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.928+0000 7fa72ce00640 1 --2- 172.21.15.32:0/4029879052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280c8220 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa72808cc00 tx=0x7fa72002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dc5244d7c98389 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa727400640 1 -- 172.21.15.32:0/4029879052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720034070 con 0x7fa72815e4a0 2024-03-06T07:31:48.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa727400640 1 -- 172.21.15.32:0/4029879052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720030610 con 0x7fa72815e4a0 2024-03-06T07:31:48.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa727400640 1 -- 172.21.15.32:0/4029879052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7200309e0 con 0x7fa72815e4a0 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 -- 172.21.15.32:0/4029879052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 msgr2=0x7fa7280c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 --2- 172.21.15.32:0/4029879052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280c8220 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa72808cc00 tx=0x7fa72002b960 comp rx=0 tx=0).stop 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 -- 172.21.15.32:0/4029879052 shutdown_connections 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 --2- 172.21.15.32:0/4029879052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280c8220 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 --2- 172.21.15.32:0/4029879052 >> v2:172.21.15.39:3300/0 conn(0x7fa7280d2bc0 0x7fa7280d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 -- 172.21.15.32:0/4029879052 >> 172.21.15.32:0/4029879052 conn(0x7fa7280d3ed0 msgr2=0x7fa7280d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 -- 172.21.15.32:0/4029879052 shutdown_connections 2024-03-06T07:31:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.929+0000 7fa72ee00640 1 -- 172.21.15.32:0/4029879052 wait complete. 2024-03-06T07:31:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.931+0000 7fa72ee00640 1 Processor -- start 2024-03-06T07:31:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.931+0000 7fa72ee00640 1 -- start start 2024-03-06T07:31:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7280d2bc0 0x7fa7280cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa72815e0c0 0x7fa7280cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280b8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280c7d00 con 0x7fa72815e4a0 2024-03-06T07:31:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280c7a20 con 0x7fa7280d2bc0 2024-03-06T07:31:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280d4b40 con 0x7fa72815e0c0 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.932+0000 7fa72ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280b8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280b8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46110/0 (socket says 172.21.15.32:46110) 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 -- 172.21.15.32:0/867132220 learned_addr learned my addr 172.21.15.32:0/867132220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa727e00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7280d2bc0 0x7fa7280cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa71f400640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa72815e0c0 0x7fa7280cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 -- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa72815e0c0 msgr2=0x7fa7280cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa72815e0c0 0x7fa7280cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 -- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7280d2bc0 msgr2=0x7fa7280cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7280d2bc0 0x7fa7280cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 -- 172.21.15.32:0/867132220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7281586c0 con 0x7fa72815e4a0 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa727e00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7280d2bc0 0x7fa7280cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa71f400640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa72815e0c0 0x7fa7280cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.933+0000 7fa72ce00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280b8dc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa720009890 tx=0x7fa7200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.934+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720034040 con 0x7fa72815e4a0 2024-03-06T07:31:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.934+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72003aa20 con 0x7fa72815e4a0 2024-03-06T07:31:48.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.934+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7280cb4a0 con 0x7fa72815e4a0 2024-03-06T07:31:48.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.934+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72003ae10 con 0x7fa72815e4a0 2024-03-06T07:31:48.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.934+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7280cba60 con 0x7fa72815e4a0 2024-03-06T07:31:48.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.935+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72804f3a0 con 0x7fa72815e4a0 2024-03-06T07:31:48.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.936+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa720037070 con 0x7fa72815e4a0 2024-03-06T07:31:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.937+0000 7fa724c00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa70406b4d0 0x7fa70406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.938+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7200ac150 con 0x7fa72815e4a0 2024-03-06T07:31:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.938+0000 7fa727e00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa70406b4d0 0x7fa70406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.939+0000 7fa727e00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa70406b4d0 0x7fa70406d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa7180012e0 tx=0x7fa718009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:48.942+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa72003f070 con 0x7fa72815e4a0 2024-03-06T07:31:49.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:49.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:49.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.218+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa7280d0bc0 con 0x7fa72815e4a0 2024-03-06T07:31:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.219+0000 7fa724c00640 1 -- 172.21.15.32:0/867132220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa72007af30 con 0x7fa72815e4a0 2024-03-06T07:31:49.221 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.222+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa70406b4d0 msgr2=0x7fa70406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.222+0000 7fa72ee00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa70406b4d0 0x7fa70406d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa7180012e0 tx=0x7fa718009040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.222+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 msgr2=0x7fa7280b8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.222+0000 7fa72ee00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280b8dc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa720009890 tx=0x7fa7200046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 shutdown_connections 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa70406b4d0 0x7fa70406d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72815e4a0 0x7fa7280b8dc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa72815e0c0 0x7fa7280cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 --2- 172.21.15.32:0/867132220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7280d2bc0 0x7fa7280cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 >> 172.21.15.32:0/867132220 conn(0x7fa7280d3ed0 msgr2=0x7fa728073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 shutdown_connections 2024-03-06T07:31:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:49.223+0000 7fa72ee00640 1 -- 172.21.15.32:0/867132220 wait complete. 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:49.243 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:49.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.243 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:31:49.291 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:49.291 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:49.292 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42228sec preferred_lft 42228sec 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:49.293 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:31:49.294 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.294 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:49.294 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:49.294 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:31:49.294 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:49.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.294 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:49.427 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:31:49.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.431 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:31:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:31:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:31:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:31:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:49.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.485 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:31:49.485 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:31:49.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:31:49.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:31:49.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:31:49.485 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:31:49.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.667 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:31:49.720 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:49.720 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:49.721 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:49.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:31:49.825 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:49.825 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:49.876 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:49.876 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:49.876 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:49.924 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:31:49.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:49.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:50.059 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:50.059 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:50.094 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:50.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:50.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:31:50.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:31:50.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:31:50.098 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:31:50.098 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:31:50.098 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:31:50.098 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:31:50.098 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:31:50.102 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:31:50.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:50.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:50.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:50.191 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:31:50.257 INFO:teuthology.orchestra.run.smithi032.stdout:client5314 2024-03-06T07:31:50.258 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:31:50.258 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:50.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.579+0000 7fb65d200640 1 Processor -- start 2024-03-06T07:31:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.579+0000 7fb65d200640 1 -- start start 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.580+0000 7fb65d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.580+0000 7fb65d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6580d4930 0x7fb65815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.580+0000 7fb65d200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6580cf780 con 0x7fb6580d0bb0 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.580+0000 7fb65d200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580b3770 con 0x7fb6580d4930 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.580+0000 7fb65d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580cfa60 con 0x7fb6580d07d0 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.581+0000 7fb656800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb6580d0bb0 0x7fb6580d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53542/0 (socket says 172.21.15.32:53542) 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.581+0000 7fb655400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6580d4930 0x7fb65815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.581+0000 7fb656800640 1 -- 172.21.15.32:0/716271524 learned_addr learned my addr 172.21.15.32:0/716271524 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:50.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.581+0000 7fb655e00640 1 --2- 172.21.15.32:0/716271524 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.582+0000 7fb654a00640 1 -- 172.21.15.32:0/716271524 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421472505 0 0) 0x7fb6580cf780 con 0x7fb6580d0bb0 2024-03-06T07:31:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.582+0000 7fb654a00640 1 -- 172.21.15.32:0/716271524 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb638003520 con 0x7fb6580d0bb0 2024-03-06T07:31:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.582+0000 7fb655400640 1 -- 172.21.15.32:0/716271524 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 msgr2=0x7fb6580b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.582+0000 7fb655400640 1 --2- 172.21.15.32:0/716271524 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.582+0000 7fb655400640 1 -- 172.21.15.32:0/716271524 >> v1:172.21.15.32:6789/0 conn(0x7fb6580d0bb0 legacy=0x7fb6580d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.582+0000 7fb655400640 1 -- 172.21.15.32:0/716271524 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580cfd40 con 0x7fb6580d4930 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.583+0000 7fb655400640 1 --2- 172.21.15.32:0/716271524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d4930 0x7fb65815dfb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb6400097a0 tx=0x7fb64002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9533a9d790dac361 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.584+0000 7fb654a00640 1 -- 172.21.15.32:0/716271524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640034070 con 0x7fb6580d4930 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.584+0000 7fb654a00640 1 -- 172.21.15.32:0/716271524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640030610 con 0x7fb6580d4930 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.584+0000 7fb654a00640 1 -- 172.21.15.32:0/716271524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6400309c0 con 0x7fb6580d4930 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.584+0000 7fb65d200640 1 -- 172.21.15.32:0/716271524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d4930 msgr2=0x7fb65815dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.584+0000 7fb65d200640 1 --2- 172.21.15.32:0/716271524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d4930 0x7fb65815dfb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb6400097a0 tx=0x7fb64002b960 comp rx=0 tx=0).stop 2024-03-06T07:31:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.585+0000 7fb65d200640 1 -- 172.21.15.32:0/716271524 shutdown_connections 2024-03-06T07:31:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.585+0000 7fb65d200640 1 --2- 172.21.15.32:0/716271524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d4930 0x7fb65815dfb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.585+0000 7fb65d200640 1 --2- 172.21.15.32:0/716271524 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.585+0000 7fb65d200640 1 -- 172.21.15.32:0/716271524 >> 172.21.15.32:0/716271524 conn(0x7fb6580d3e10 msgr2=0x7fb6580d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.585+0000 7fb65d200640 1 -- 172.21.15.32:0/716271524 shutdown_connections 2024-03-06T07:31:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.585+0000 7fb65d200640 1 -- 172.21.15.32:0/716271524 wait complete. 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.587+0000 7fb65d200640 1 Processor -- start 2024-03-06T07:31:50.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.588+0000 7fb65d200640 1 -- start start 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.589+0000 7fb65d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.589+0000 7fb65d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 0x7fb65815c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.589+0000 7fb65d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6580d4930 0x7fb6580b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.589+0000 7fb65d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b4d90 con 0x7fb6580d0bb0 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.589+0000 7fb65d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b4c10 con 0x7fb6580d4930 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.589+0000 7fb65d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65815e4f0 con 0x7fb6580d07d0 2024-03-06T07:31:50.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 0x7fb65815c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 0x7fb65815c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46116/0 (socket says 172.21.15.32:46116) 2024-03-06T07:31:50.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 -- 172.21.15.32:0/199013685 learned_addr learned my addr 172.21.15.32:0/199013685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 -- 172.21.15.32:0/199013685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 msgr2=0x7fb6580caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 -- 172.21.15.32:0/199013685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6580d4930 msgr2=0x7fb6580b69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6580d4930 0x7fb6580b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 -- 172.21.15.32:0/199013685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580ce170 con 0x7fb6580d0bb0 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.590+0000 7fb655400640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 0x7fb65815c630 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb64002bf70 tx=0x7fb6400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.591+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640034040 con 0x7fb6580d0bb0 2024-03-06T07:31:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.591+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64003aaa0 con 0x7fb6580d0bb0 2024-03-06T07:31:50.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.591+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640006e10 con 0x7fb6580d0bb0 2024-03-06T07:31:50.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.591+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6580b81a0 con 0x7fb6580d0bb0 2024-03-06T07:31:50.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.591+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6580b8700 con 0x7fb6580d0bb0 2024-03-06T07:31:50.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.592+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65804f330 con 0x7fb6580d0bb0 2024-03-06T07:31:50.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.594+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb640037070 con 0x7fb6580d0bb0 2024-03-06T07:31:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.595+0000 7fb64e000640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb63406b4d0 0x7fb63406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.596+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6400abf80 con 0x7fb6580d0bb0 2024-03-06T07:31:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.596+0000 7fb655e00640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb63406b4d0 0x7fb63406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.597+0000 7fb655e00640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb63406b4d0 0x7fb63406d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb648006a90 tx=0x7fb648006a20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.600+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6580b81a0 con 0x7fb6580d0bb0 2024-03-06T07:31:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.799+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fb6580b88d0 con 0x7fb6580d0bb0 2024-03-06T07:31:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.805+0000 7fb64e000640 1 -- 172.21.15.32:0/199013685 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb64003f070 con 0x7fb6580d0bb0 2024-03-06T07:31:50.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb63406b4d0 msgr2=0x7fb63406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb63406b4d0 0x7fb63406d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb648006a90 tx=0x7fb648006a20 comp rx=0 tx=0).stop 2024-03-06T07:31:50.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 msgr2=0x7fb65815c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 0x7fb65815c630 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb64002bf70 tx=0x7fb6400046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 shutdown_connections 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb63406b4d0 0x7fb63406d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6580d4930 0x7fb6580b69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6580d0bb0 0x7fb65815c630 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 --2- 172.21.15.32:0/199013685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb6580d07d0 0x7fb6580caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 >> 172.21.15.32:0/199013685 conn(0x7fb6580d3e10 msgr2=0x7fb65806caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 shutdown_connections 2024-03-06T07:31:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:50.808+0000 7fb65d200640 1 -- 172.21.15.32:0/199013685 wait complete. 2024-03-06T07:31:50.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:31:51.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.041+0000 7f89caa00640 1 Processor -- start 2024-03-06T07:31:51.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.042+0000 7f89caa00640 1 -- start start 2024-03-06T07:31:51.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.044+0000 7f89caa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89c415dee0 0x7f89c40d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.044+0000 7f89caa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89c40d4860 0x7f89c40d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.044+0000 7f89caa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c40ceca0 con 0x7f89c40cfa60 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89caa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c40cf5d0 con 0x7f89c415dee0 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89caa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c40cef80 con 0x7f89c40d4860 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c9600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89c415dee0 0x7f89c40d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89ca000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89c40cfa60 0x7f89c40b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53548/0 (socket says 172.21.15.32:53548) 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c9600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89c415dee0 0x7f89c40d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46118/0 (socket says 172.21.15.32:46118) 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89ca000640 1 -- 172.21.15.32:0/1397880191 learned_addr learned my addr 172.21.15.32:0/1397880191 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c8c00640 1 --2- 172.21.15.32:0/1397880191 >> v2:172.21.15.39:3300/0 conn(0x7f89c40d4860 0x7f89c40d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c9600640 1 -- 172.21.15.32:0/1397880191 >> v2:172.21.15.39:3300/0 conn(0x7f89c40d4860 msgr2=0x7f89c40d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c9600640 1 --2- 172.21.15.32:0/1397880191 >> v2:172.21.15.39:3300/0 conn(0x7f89c40d4860 0x7f89c40d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c9600640 1 -- 172.21.15.32:0/1397880191 >> v1:172.21.15.32:6789/0 conn(0x7f89c40cfa60 legacy=0x7f89c40b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.045+0000 7f89c9600640 1 -- 172.21.15.32:0/1397880191 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c40cf260 con 0x7f89c415dee0 2024-03-06T07:31:51.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.047+0000 7f89c9600640 1 --2- 172.21.15.32:0/1397880191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40d18a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f89c4162b70 tx=0x7f89b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6658d54b0c5291 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.047+0000 7f89c3e00640 1 -- 172.21.15.32:0/1397880191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8039070 con 0x7f89c415dee0 2024-03-06T07:31:51.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.047+0000 7f89c3e00640 1 -- 172.21.15.32:0/1397880191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b802cc80 con 0x7f89c415dee0 2024-03-06T07:31:51.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.047+0000 7f89c3e00640 1 -- 172.21.15.32:0/1397880191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8034660 con 0x7f89c415dee0 2024-03-06T07:31:51.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 -- 172.21.15.32:0/1397880191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 msgr2=0x7f89c40d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 --2- 172.21.15.32:0/1397880191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40d18a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f89c4162b70 tx=0x7f89b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 -- 172.21.15.32:0/1397880191 shutdown_connections 2024-03-06T07:31:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 --2- 172.21.15.32:0/1397880191 >> v2:172.21.15.39:3300/0 conn(0x7f89c40d4860 0x7f89c40d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 --2- 172.21.15.32:0/1397880191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40d18a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 -- 172.21.15.32:0/1397880191 >> 172.21.15.32:0/1397880191 conn(0x7f89c40d3cd0 msgr2=0x7f89c40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 -- 172.21.15.32:0/1397880191 shutdown_connections 2024-03-06T07:31:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.048+0000 7f89caa00640 1 -- 172.21.15.32:0/1397880191 wait complete. 2024-03-06T07:31:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.050+0000 7f89caa00640 1 Processor -- start 2024-03-06T07:31:51.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.050+0000 7f89caa00640 1 -- start start 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89caa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89c40cfa60 0x7f89c41f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c40d4860 0x7f89c40cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89caa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89caa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40d1de0 con 0x7f89c415dee0 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89caa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c415e2c0 con 0x7f89c40d4860 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89caa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40d1f60 con 0x7f89c40cfa60 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46122/0 (socket says 172.21.15.32:46122) 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 -- 172.21.15.32:0/3386533730 learned_addr learned my addr 172.21.15.32:0/3386533730 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89c9600640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89c40cfa60 0x7f89c41f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89c8c00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c40d4860 0x7f89c40cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 -- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89c40cfa60 msgr2=0x7f89c41f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89c40cfa60 0x7f89c41f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 -- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c40d4860 msgr2=0x7f89c40cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.051+0000 7f89ca000640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c40d4860 0x7f89c40cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89ca000640 1 -- 172.21.15.32:0/3386533730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c41f4840 con 0x7f89c415dee0 2024-03-06T07:31:51.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89ca000640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40ccce0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00c420 tx=0x7f89bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc017070 con 0x7f89c415dee0 2024-03-06T07:31:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc005ce0 con 0x7f89c415dee0 2024-03-06T07:31:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c40c7930 con 0x7f89c415dee0 2024-03-06T07:31:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc0072d0 con 0x7f89c415dee0 2024-03-06T07:31:51.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.052+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c41f4d40 con 0x7f89c415dee0 2024-03-06T07:31:51.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.054+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c415fe90 con 0x7f89c415dee0 2024-03-06T07:31:51.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.055+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89bc01a070 con 0x7f89c415dee0 2024-03-06T07:31:51.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.055+0000 7f89c1600640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f89a406b4d0 0x7f89a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.056+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89bc0894a0 con 0x7f89c415dee0 2024-03-06T07:31:51.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.056+0000 7f89c9600640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f89a406b4d0 0x7f89a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.057+0000 7f89c9600640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f89a406b4d0 0x7f89a406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f89b8036000 tx=0x7f89b8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.060+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89bc057cc0 con 0x7f89c415dee0 2024-03-06T07:31:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.263+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f89c404f330 con 0x7f89c415dee0 2024-03-06T07:31:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.268+0000 7f89c1600640 1 -- 172.21.15.32:0/3386533730 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f89bc05bd60 con 0x7f89c415dee0 2024-03-06T07:31:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.270+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f89a406b4d0 msgr2=0x7f89a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.270+0000 7f89caa00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f89a406b4d0 0x7f89a406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f89b8036000 tx=0x7f89b8030000 comp rx=0 tx=0).stop 2024-03-06T07:31:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.270+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 msgr2=0x7f89c40ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.270+0000 7f89caa00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40ccce0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00c420 tx=0x7f89bc00c8e0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 shutdown_connections 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f89a406b4d0 0x7f89a406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89c415dee0 0x7f89c40ccce0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c40d4860 0x7f89c40cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 --2- 172.21.15.32:0/3386533730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89c40cfa60 0x7f89c41f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 >> 172.21.15.32:0/3386533730 conn(0x7f89c40d3cd0 msgr2=0x7f89c40b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 shutdown_connections 2024-03-06T07:31:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:31:51.271+0000 7f89caa00640 1 -- 172.21.15.32:0/3386533730 wait complete. 2024-03-06T07:32:01.298 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:01.299 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:01.362 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:32:01.362 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:32:01.362 DEBUG:teuthology.orchestra.run.smithi032:> import uuid 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> n = 50 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = False 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> if 0 > 0: 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = True 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> dpath = os.path.dirname(path) 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> fnameprefix = os.path.basename(path) 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:01.363 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(n): 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> f.write(f"{i}") 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> f.flush() 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(f.fileno()) 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> os.unlink(fpath) 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:01.364 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> if create_hard_links: 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> for j in range(0): 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> finally: 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> os.close(dirfd) 2024-03-06T07:32:01.365 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:32:01.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:32:01.677 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:01.677 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:01.774 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:01.774 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:01.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:02.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.025+0000 7f35f4a00640 1 Processor -- start 2024-03-06T07:32:02.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.025+0000 7f35f4a00640 1 -- start start 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35f4a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f35f015dec0 0x7f35f00ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35f4a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35f015e2a0 0x7f35f00ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35f4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00cf700 con 0x7f35f00cf290 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35f4a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f00b0ea0 con 0x7f35f015e2a0 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35f4a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f00d1120 con 0x7f35f015dec0 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35e7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35f015e2a0 0x7f35f00ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35e7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35f015e2a0 0x7f35f00ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35274/0 (socket says 172.21.15.32:35274) 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.027+0000 7f35e7e00640 1 -- 172.21.15.32:0/2511625293 learned_addr learned my addr 172.21.15.32:0/2511625293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:02.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.028+0000 7f35ef400640 1 --2- 172.21.15.32:0/2511625293 >> v2:172.21.15.39:3300/0 conn(0x7f35f015dec0 0x7f35f00ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.028+0000 7f35e7e00640 1 -- 172.21.15.32:0/2511625293 >> v2:172.21.15.39:3300/0 conn(0x7f35f015dec0 msgr2=0x7f35f00ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.028+0000 7f35e7e00640 1 --2- 172.21.15.32:0/2511625293 >> v2:172.21.15.39:3300/0 conn(0x7f35f015dec0 0x7f35f00ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.028+0000 7f35e7e00640 1 -- 172.21.15.32:0/2511625293 >> v1:172.21.15.39:6789/0 conn(0x7f35f00cf290 legacy=0x7f35f00d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.028+0000 7f35e7e00640 1 -- 172.21.15.32:0/2511625293 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00d1400 con 0x7f35f015e2a0 2024-03-06T07:32:02.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.029+0000 7f35e7e00640 1 --2- 172.21.15.32:0/2511625293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00ced50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f35f008c780 tx=0x7f35d802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f5082129ff8c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.030+0000 7f35eea00640 1 -- 172.21.15.32:0/2511625293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8034070 con 0x7f35f015e2a0 2024-03-06T07:32:02.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.030+0000 7f35eea00640 1 -- 172.21.15.32:0/2511625293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d8030880 con 0x7f35f015e2a0 2024-03-06T07:32:02.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.030+0000 7f35eea00640 1 -- 172.21.15.32:0/2511625293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8030c50 con 0x7f35f015e2a0 2024-03-06T07:32:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.030+0000 7f35f4a00640 1 -- 172.21.15.32:0/2511625293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 msgr2=0x7f35f00ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.030+0000 7f35f4a00640 1 --2- 172.21.15.32:0/2511625293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00ced50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f35f008c780 tx=0x7f35d802b9c0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.031+0000 7f35f4a00640 1 -- 172.21.15.32:0/2511625293 shutdown_connections 2024-03-06T07:32:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.031+0000 7f35f4a00640 1 --2- 172.21.15.32:0/2511625293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00ced50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.031+0000 7f35f4a00640 1 --2- 172.21.15.32:0/2511625293 >> v2:172.21.15.39:3300/0 conn(0x7f35f015dec0 0x7f35f00ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.031+0000 7f35f4a00640 1 -- 172.21.15.32:0/2511625293 >> 172.21.15.32:0/2511625293 conn(0x7f35f00d3d40 msgr2=0x7f35f00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.031+0000 7f35f4a00640 1 -- 172.21.15.32:0/2511625293 shutdown_connections 2024-03-06T07:32:02.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.031+0000 7f35f4a00640 1 -- 172.21.15.32:0/2511625293 wait complete. 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.033+0000 7f35f4a00640 1 Processor -- start 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.033+0000 7f35f4a00640 1 -- start start 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35f4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35f00cf290 0x7f35f014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35f4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35f015dec0 0x7f35f00b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35f4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35f4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00d0ec0 con 0x7f35f015e2a0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35f4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00d0d40 con 0x7f35f015dec0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35f4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00b2340 con 0x7f35f00cf290 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35276/0 (socket says 172.21.15.32:35276) 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.034+0000 7f35efe00640 1 -- 172.21.15.32:0/942318378 learned_addr learned my addr 172.21.15.32:0/942318378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35ef400640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35f00cf290 0x7f35f014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35efe00640 1 -- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35f00cf290 msgr2=0x7f35f014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35efe00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35f00cf290 0x7f35f014e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35e7e00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35f015dec0 0x7f35f00b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35efe00640 1 -- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35f015dec0 msgr2=0x7f35f00b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35efe00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35f015dec0 0x7f35f00b6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35efe00640 1 -- 172.21.15.32:0/942318378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f014e590 con 0x7f35f015e2a0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35efe00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00b51a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c7d0 tx=0x7f35e800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.035+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8003b70 con 0x7f35f015e2a0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.036+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8003d30 con 0x7f35f015e2a0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.036+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00b9140 con 0x7f35f015e2a0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.036+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e800aa90 con 0x7f35f015e2a0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.036+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f00b9700 con 0x7f35f015e2a0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.037+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f004f330 con 0x7f35f015e2a0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.038+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35e800ac30 con 0x7f35f015e2a0 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.039+0000 7f35e7400640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f35cc06b4d0 0x7f35cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.039+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35e808a140 con 0x7f35f015e2a0 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.039+0000 7f35ef400640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f35cc06b4d0 0x7f35cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.040+0000 7f35ef400640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f35cc06b4d0 0x7f35cc06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f35e0006fd0 tx=0x7f35e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.044+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35e8058fb0 con 0x7f35f015e2a0 2024-03-06T07:32:02.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.245+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35f00d0570 con 0x7f35f015e2a0 2024-03-06T07:32:02.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.246+0000 7f35e7400640 1 -- 172.21.15.32:0/942318378 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f35e80588b0 con 0x7f35f015e2a0 2024-03-06T07:32:02.249 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:02.249 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:42.769932+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:02.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f35cc06b4d0 msgr2=0x7f35cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f35cc06b4d0 0x7f35cc06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f35e0006fd0 tx=0x7f35e0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 msgr2=0x7f35f00b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00b51a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c7d0 tx=0x7f35e800cc90 comp rx=0 tx=0).stop 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 shutdown_connections 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f35cc06b4d0 0x7f35cc06d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f015e2a0 0x7f35f00b51a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35f015dec0 0x7f35f00b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 --2- 172.21.15.32:0/942318378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f35f00cf290 0x7f35f014e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.249+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 >> 172.21.15.32:0/942318378 conn(0x7f35f00d3d40 msgr2=0x7f35f015e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.250+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 shutdown_connections 2024-03-06T07:32:02.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.250+0000 7f35f4a00640 1 -- 172.21.15.32:0/942318378 wait complete. 2024-03-06T07:32:02.271 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:39.080872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:40.231621+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:02.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:02.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.492+0000 7fbb71400640 1 Processor -- start 2024-03-06T07:32:02.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.493+0000 7fbb71400640 1 -- start start 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb71400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbb6c0d07d0 0x7fbb6c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb71400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c0cf780 con 0x7fbb6c0d4930 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb71400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c0cfa60 con 0x7fbb6c15e6c0 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb71400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c0b3770 con 0x7fbb6c0d07d0 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb61e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbb6c0d4930 0x7fbb6c15dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38986/0 (socket says 172.21.15.32:38986) 2024-03-06T07:32:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb61e00640 1 -- 172.21.15.32:0/958860231 learned_addr learned my addr 172.21.15.32:0/958860231 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.494+0000 7fbb6a800640 1 --2- 172.21.15.32:0/958860231 >> v2:172.21.15.39:3300/0 conn(0x7fbb6c0d07d0 0x7fbb6c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230438242 0 0) 0x7fbb6c0cf780 con 0x7fbb6c0d4930 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003520 con 0x7fbb6c0d4930 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1910918292 0 0) 0x7fbb50003520 con 0x7fbb6c0d4930 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 >> v2:172.21.15.39:3300/0 conn(0x7fbb6c0d07d0 msgr2=0x7fbb6c0b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 --2- 172.21.15.32:0/958860231 >> v2:172.21.15.39:3300/0 conn(0x7fbb6c0d07d0 0x7fbb6c0b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 >> v1:172.21.15.39:6789/0 conn(0x7fbb6c15e6c0 legacy=0x7fbb6c0b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0cfd40 con 0x7fbb6c0d4930 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fbb540019f0 con 0x7fbb6c0d4930 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 >> v1:172.21.15.32:6789/0 conn(0x7fbb6c0d4930 legacy=0x7fbb6c15dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500039b0 0x7fbb50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb50004800 0x7fbb50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb500051c0 0x7fbb5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0b3770 con 0x7fbb500051c0 2024-03-06T07:32:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000c070 con 0x7fbb500039b0 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.495+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000c270 con 0x7fbb50004800 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6b200640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb500051c0 0x7fbb5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb61e00640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb50004800 0x7fbb50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6a800640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500039b0 0x7fbb50003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6b200640 1 -- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb50004800 msgr2=0x7fbb50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6b200640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb50004800 0x7fbb50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6b200640 1 -- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500039b0 msgr2=0x7fbb50003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6b200640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500039b0 0x7fbb50003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.496+0000 7fbb6b200640 1 -- 172.21.15.32:0/958860231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb50003520 con 0x7fbb500051c0 2024-03-06T07:32:02.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.497+0000 7fbb6b200640 1 -- 172.21.15.32:0/958860231 reap_dead start 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.497+0000 7fbb6b200640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb500051c0 0x7fbb5000b9c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbb64001750 tx=0x7fbb50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f20e6b81594e57b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64001370 con 0x7fbb500051c0 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb640052f0 con 0x7fbb500051c0 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb69e00640 1 -- 172.21.15.32:0/958860231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64003830 con 0x7fbb500051c0 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 -- 172.21.15.32:0/958860231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb500051c0 msgr2=0x7fbb5000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb500051c0 0x7fbb5000b9c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbb64001750 tx=0x7fbb50005830 comp rx=0 tx=0).stop 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 -- 172.21.15.32:0/958860231 shutdown_connections 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 --2- 172.21.15.32:0/958860231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb500051c0 0x7fbb5000b9c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 -- 172.21.15.32:0/958860231 >> 172.21.15.32:0/958860231 conn(0x7fbb6c0d3e10 msgr2=0x7fbb6c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 -- 172.21.15.32:0/958860231 shutdown_connections 2024-03-06T07:32:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.498+0000 7fbb71400640 1 -- 172.21.15.32:0/958860231 wait complete. 2024-03-06T07:32:02.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.500+0000 7fbb71400640 1 Processor -- start 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.500+0000 7fbb71400640 1 -- start start 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb71400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb500051c0 0x7fbb6c152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb71400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 0x7fbb6c0c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb71400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6c0b60c0 0x7fbb6c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb71400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c15dfb0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb71400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0d0dc0 con 0x7fbb6c0b60c0 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb71400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c15e130 con 0x7fbb500051c0 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.501+0000 7fbb61e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 0x7fbb6c0c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 0x7fbb6c0c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35306/0 (socket says 172.21.15.32:35306) 2024-03-06T07:32:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 -- 172.21.15.32:0/1112274177 learned_addr learned my addr 172.21.15.32:0/1112274177 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb6a800640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb500051c0 0x7fbb6c152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb6b200640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6c0b60c0 0x7fbb6c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 -- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb500051c0 msgr2=0x7fbb6c152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb500051c0 0x7fbb6c152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 -- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6c0b60c0 msgr2=0x7fbb6c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6c0b60c0 0x7fbb6c0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 -- 172.21.15.32:0/1112274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c1527d0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.502+0000 7fbb61e00640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 0x7fbb6c0c8310 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fbb540013d0 tx=0x7fbb5400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.503+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5400edf0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.503+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb54018070 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.503+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb54013660 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.503+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c0cb130 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.503+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c0cb6f0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.504+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c0caec0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.505+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb54013800 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.506+0000 7fbb63400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb4806b500 0x7fbb4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.507+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb5408a7d0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.507+0000 7fbb6a800640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb4806b500 0x7fbb4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.508+0000 7fbb6a800640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb4806b500 0x7fbb4806d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c06bb80 tx=0x7fbb5c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.511+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb6c1527d0 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.711+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb6c04f330 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.712+0000 7fbb63400640 1 -- 172.21.15.32:0/1112274177 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb54059640 con 0x7fbb6c1f80d0 2024-03-06T07:32:02.714 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:32:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.714+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb4806b500 msgr2=0x7fbb4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.714+0000 7fbb71400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb4806b500 0x7fbb4806d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c06bb80 tx=0x7fbb5c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.714+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 msgr2=0x7fbb6c0c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.714+0000 7fbb71400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 0x7fbb6c0c8310 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fbb540013d0 tx=0x7fbb5400e530 comp rx=0 tx=0).stop 2024-03-06T07:32:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 shutdown_connections 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbb4806b500 0x7fbb4806d9c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6c0b60c0 0x7fbb6c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c1f80d0 0x7fbb6c0c8310 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 --2- 172.21.15.32:0/1112274177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbb500051c0 0x7fbb6c152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 >> 172.21.15.32:0/1112274177 conn(0x7fbb6c0d3e10 msgr2=0x7fbb6c15d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 shutdown_connections 2024-03-06T07:32:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.715+0000 7fbb71400640 1 -- 172.21.15.32:0/1112274177 wait complete. 2024-03-06T07:32:02.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.951+0000 7fd3b3c00640 1 Processor -- start 2024-03-06T07:32:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.951+0000 7fd3b3c00640 1 -- start start 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3b3c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd3ac0672d0 0x7fd3ac0ae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3b3c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd3ac0ae780 0x7fd3ac0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3b3c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac06e6f0 con 0x7fd3ac0aefd0 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3b3c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac06f020 con 0x7fd3ac0672d0 2024-03-06T07:32:02.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3b3c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ac06e9d0 con 0x7fd3ac0ae780 2024-03-06T07:32:02.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3abe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd3ac0672d0 0x7fd3ac0ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.953+0000 7fd3abe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd3ac0672d0 0x7fd3ac0ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35314/0 (socket says 172.21.15.32:35314) 2024-03-06T07:32:02.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.954+0000 7fd3abe00640 1 -- 172.21.15.32:0/2007980 learned_addr learned my addr 172.21.15.32:0/2007980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:02.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.954+0000 7fd3b1c00640 1 --2- 172.21.15.32:0/2007980 >> v2:172.21.15.39:3300/0 conn(0x7fd3ac0ae780 0x7fd3ac0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.954+0000 7fd3abe00640 1 -- 172.21.15.32:0/2007980 >> v2:172.21.15.39:3300/0 conn(0x7fd3ac0ae780 msgr2=0x7fd3ac0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.954+0000 7fd3abe00640 1 --2- 172.21.15.32:0/2007980 >> v2:172.21.15.39:3300/0 conn(0x7fd3ac0ae780 0x7fd3ac0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.954+0000 7fd3abe00640 1 -- 172.21.15.32:0/2007980 >> v1:172.21.15.39:6789/0 conn(0x7fd3ac0aefd0 legacy=0x7fd3ac0af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.954+0000 7fd3abe00640 1 -- 172.21.15.32:0/2007980 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac06ecb0 con 0x7fd3ac0672d0 2024-03-06T07:32:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.956+0000 7fd3abe00640 1 --2- 172.21.15.32:0/2007980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0672d0 0x7fd3ac0ae240 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0aec30 tx=0x7fd39802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90fec33247befc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.956+0000 7fd3ab400640 1 -- 172.21.15.32:0/2007980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398035020 con 0x7fd3ac0672d0 2024-03-06T07:32:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.956+0000 7fd3ab400640 1 -- 172.21.15.32:0/2007980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398031870 con 0x7fd3ac0672d0 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.956+0000 7fd3ab400640 1 -- 172.21.15.32:0/2007980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398031c40 con 0x7fd3ac0672d0 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 -- 172.21.15.32:0/2007980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0672d0 msgr2=0x7fd3ac0ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/2007980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0672d0 0x7fd3ac0ae240 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0aec30 tx=0x7fd39802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 -- 172.21.15.32:0/2007980 shutdown_connections 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/2007980 >> v2:172.21.15.39:3300/0 conn(0x7fd3ac0ae780 0x7fd3ac0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/2007980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0672d0 0x7fd3ac0ae240 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 -- 172.21.15.32:0/2007980 >> 172.21.15.32:0/2007980 conn(0x7fd3ac072280 msgr2=0x7fd3ac06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 -- 172.21.15.32:0/2007980 shutdown_connections 2024-03-06T07:32:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.957+0000 7fd3b3c00640 1 -- 172.21.15.32:0/2007980 wait complete. 2024-03-06T07:32:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.959+0000 7fd3b3c00640 1 Processor -- start 2024-03-06T07:32:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.959+0000 7fd3b3c00640 1 -- start start 2024-03-06T07:32:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b3c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd3ac0672d0 0x7fd3ac147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 0x7fd3ac15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b3c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3ac0aefd0 0x7fd3ac1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac15e340 con 0x7fd3ac0ae780 2024-03-06T07:32:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b3c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac15e1c0 con 0x7fd3ac0aefd0 2024-03-06T07:32:02.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b3c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac0b3980 con 0x7fd3ac0672d0 2024-03-06T07:32:02.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 0x7fd3ac15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 0x7fd3ac15c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35316/0 (socket says 172.21.15.32:35316) 2024-03-06T07:32:02.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3abe00640 1 -- 172.21.15.32:0/1758376712 learned_addr learned my addr 172.21.15.32:0/1758376712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:02.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.960+0000 7fd3b1200640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd3ac0672d0 0x7fd3ac147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3b1c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3ac0aefd0 0x7fd3ac1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3abe00640 1 -- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd3ac0672d0 msgr2=0x7fd3ac147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3abe00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd3ac0672d0 0x7fd3ac147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3abe00640 1 -- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3ac0aefd0 msgr2=0x7fd3ac1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3abe00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3ac0aefd0 0x7fd3ac1592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3abe00640 1 -- 172.21.15.32:0/1758376712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac1481a0 con 0x7fd3ac0ae780 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3b1c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3ac0aefd0 0x7fd3ac1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3abe00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 0x7fd3ac15c0e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd3980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.961+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398040070 con 0x7fd3ac0ae780 2024-03-06T07:32:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.962+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398030d00 con 0x7fd3ac0ae780 2024-03-06T07:32:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.962+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398035040 con 0x7fd3ac0ae780 2024-03-06T07:32:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.962+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac15cbc0 con 0x7fd3ac0ae780 2024-03-06T07:32:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.962+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac15d180 con 0x7fd3ac0ae780 2024-03-06T07:32:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.963+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac04f330 con 0x7fd3ac0ae780 2024-03-06T07:32:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.964+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd398038020 con 0x7fd3ac0ae780 2024-03-06T07:32:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.965+0000 7fd3a8c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd38c06b4d0 0x7fd38c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.965+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3980ac200 con 0x7fd3ac0ae780 2024-03-06T07:32:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.965+0000 7fd3b1200640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd38c06b4d0 0x7fd38c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.966+0000 7fd3b1200640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd38c06b4d0 0x7fd38c06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009f20 tx=0x7fd3a00094d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:02.969+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd39807af50 con 0x7fd3ac0ae780 2024-03-06T07:32:03.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.168+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3ac06bda0 con 0x7fd3ac0ae780 2024-03-06T07:32:03.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.169+0000 7fd3a8c00640 1 -- 172.21.15.32:0/1758376712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd39807a850 con 0x7fd3ac0ae780 2024-03-06T07:32:03.171 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.171+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd38c06b4d0 msgr2=0x7fd38c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.171+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd38c06b4d0 0x7fd38c06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009f20 tx=0x7fd3a00094d0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.171+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 msgr2=0x7fd3ac15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.171+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 0x7fd3ac15c0e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd3980046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 shutdown_connections 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd38c06b4d0 0x7fd38c06d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3ac0aefd0 0x7fd3ac1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3ac0ae780 0x7fd3ac15c0e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 --2- 172.21.15.32:0/1758376712 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd3ac0672d0 0x7fd3ac147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 >> 172.21.15.32:0/1758376712 conn(0x7fd3ac072280 msgr2=0x7fd3ac06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 shutdown_connections 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.172+0000 7fd3b3c00640 1 -- 172.21.15.32:0/1758376712 wait complete. 2024-03-06T07:32:03.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:03.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.399+0000 7f843f200640 1 Processor -- start 2024-03-06T07:32:03.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.399+0000 7f843f200640 1 -- start start 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f84400d4890 0x7f84400d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843f200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84400d2320 0x7f84400b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84400b2c00 con 0x7f8440161700 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843f200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84400b56e0 con 0x7f84400d2320 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84400d2750 con 0x7f84400d4890 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84400d2320 0x7f84400b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f84400d2320 0x7f84400b1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35332/0 (socket says 172.21.15.32:35332) 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.401+0000 7f843e800640 1 -- 172.21.15.32:0/1924046407 learned_addr learned my addr 172.21.15.32:0/1924046407 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.402+0000 7f843e800640 1 -- 172.21.15.32:0/1924046407 >> v2:172.21.15.39:3300/0 conn(0x7f84400d4890 msgr2=0x7f84400d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.402+0000 7f843d400640 1 --2- 172.21.15.32:0/1924046407 >> v2:172.21.15.39:3300/0 conn(0x7f84400d4890 0x7f84400d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.402+0000 7f843e800640 1 --2- 172.21.15.32:0/1924046407 >> v2:172.21.15.39:3300/0 conn(0x7f84400d4890 0x7f84400d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.402+0000 7f843e800640 1 -- 172.21.15.32:0/1924046407 >> v1:172.21.15.39:6789/0 conn(0x7f8440161700 legacy=0x7f84400b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.402+0000 7f843e800640 1 -- 172.21.15.32:0/1924046407 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440161ae0 con 0x7f84400d2320 2024-03-06T07:32:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.402+0000 7f843d400640 1 --2- 172.21.15.32:0/1924046407 >> v2:172.21.15.39:3300/0 conn(0x7f84400d4890 0x7f84400d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.403+0000 7f843e800640 1 --2- 172.21.15.32:0/1924046407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d2320 0x7f84400b1220 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f844008c350 tx=0x7f843802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f5dbd0b08e8e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.403+0000 7f843ca00640 1 -- 172.21.15.32:0/1924046407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843802ca80 con 0x7f84400d2320 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.403+0000 7f843ca00640 1 -- 172.21.15.32:0/1924046407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843802cc20 con 0x7f84400d2320 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.403+0000 7f843ca00640 1 -- 172.21.15.32:0/1924046407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84380343f0 con 0x7f84400d2320 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 -- 172.21.15.32:0/1924046407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d2320 msgr2=0x7f84400b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 --2- 172.21.15.32:0/1924046407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d2320 0x7f84400b1220 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f844008c350 tx=0x7f843802bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 -- 172.21.15.32:0/1924046407 shutdown_connections 2024-03-06T07:32:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 --2- 172.21.15.32:0/1924046407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d2320 0x7f84400b1220 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 --2- 172.21.15.32:0/1924046407 >> v2:172.21.15.39:3300/0 conn(0x7f84400d4890 0x7f84400d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 -- 172.21.15.32:0/1924046407 >> 172.21.15.32:0/1924046407 conn(0x7f84400d3d70 msgr2=0x7f84400d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 -- 172.21.15.32:0/1924046407 shutdown_connections 2024-03-06T07:32:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.404+0000 7f843f200640 1 -- 172.21.15.32:0/1924046407 wait complete. 2024-03-06T07:32:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.406+0000 7f843f200640 1 Processor -- start 2024-03-06T07:32:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.406+0000 7f843f200640 1 -- start start 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f84400d2320 0x7f84400c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 0x7f84401582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8440161700 0x7f84400b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400b0a20 con 0x7f84400d4890 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400b3180 con 0x7f8440161700 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84400b0d00 con 0x7f84400d2320 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 0x7f84401582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 0x7f84401582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35338/0 (socket says 172.21.15.32:35338) 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843d400640 1 -- 172.21.15.32:0/3652776320 learned_addr learned my addr 172.21.15.32:0/3652776320 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843de00640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f84400d2320 0x7f84400c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.407+0000 7f843e800640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8440161700 0x7f84400b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843d400640 1 -- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f84400d2320 msgr2=0x7f84400c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843d400640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f84400d2320 0x7f84400c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843d400640 1 -- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8440161700 msgr2=0x7f84400b7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843d400640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8440161700 0x7f84400b7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843d400640 1 -- 172.21.15.32:0/3652776320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84400b7b80 con 0x7f84400d4890 2024-03-06T07:32:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843de00640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f84400d2320 0x7f84400c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843d400640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 0x7f84401582c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f842c00c820 tx=0x7f842c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c015070 con 0x7f84400d4890 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c007a30 con 0x7f84400d4890 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84400b7e60 con 0x7f84400d4890 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c007e00 con 0x7f84400d4890 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.408+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84400c8750 con 0x7f84400d4890 2024-03-06T07:32:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.409+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844004f330 con 0x7f84400d4890 2024-03-06T07:32:03.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.411+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f842c018070 con 0x7f84400d4890 2024-03-06T07:32:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.412+0000 7f8436000640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f842006b4d0 0x7f842006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.412+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f842c08a1e0 con 0x7f84400d4890 2024-03-06T07:32:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.412+0000 7f843de00640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f842006b4d0 0x7f842006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.412+0000 7f843de00640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f842006b4d0 0x7f842006d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8430006fd0 tx=0x7f8430008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.416+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f842c059060 con 0x7f84400d4890 2024-03-06T07:32:03.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.618+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84401609d0 con 0x7f84400d4890 2024-03-06T07:32:03.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.618+0000 7f8436000640 1 -- 172.21.15.32:0/3652776320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f842c058960 con 0x7f84400d4890 2024-03-06T07:32:03.621 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:32:03.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f842006b4d0 msgr2=0x7f842006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f842006b4d0 0x7f842006d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8430006fd0 tx=0x7f8430008040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 msgr2=0x7f84401582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 0x7f84401582c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f842c00c820 tx=0x7f842c00cc20 comp rx=0 tx=0).stop 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 shutdown_connections 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f842006b4d0 0x7f842006d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8440161700 0x7f84400b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84400d4890 0x7f84401582c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 --2- 172.21.15.32:0/3652776320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f84400d2320 0x7f84400c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 >> 172.21.15.32:0/3652776320 conn(0x7f84400d3d70 msgr2=0x7f84400b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.621+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 shutdown_connections 2024-03-06T07:32:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.622+0000 7f843f200640 1 -- 172.21.15.32:0/3652776320 wait complete. 2024-03-06T07:32:03.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.850+0000 7f2ff6c00640 1 Processor -- start 2024-03-06T07:32:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.850+0000 7f2ff6c00640 1 -- start start 2024-03-06T07:32:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff6c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff6c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ff0070f10 0x7f2ff00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff6c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2ff00cce90 0x7f2ff00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff6c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff00cd300 con 0x7f2ff0070f10 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff6c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff00714d0 con 0x7f2ff00cce90 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff6c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00717b0 con 0x7f2ff0145300 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.851+0000 7f2ff4e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ff0070f10 0x7f2ff00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ff0070f10 0x7f2ff00cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35350/0 (socket says 172.21.15.32:35350) 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 -- 172.21.15.32:0/1514862906 learned_addr learned my addr 172.21.15.32:0/1514862906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff5800640 1 --2- 172.21.15.32:0/1514862906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 -- 172.21.15.32:0/1514862906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 msgr2=0x7f2ff00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 --2- 172.21.15.32:0/1514862906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff00b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 -- 172.21.15.32:0/1514862906 >> v2:172.21.15.39:3300/0 conn(0x7f2ff00cce90 msgr2=0x7f2ff00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 --2- 172.21.15.32:0/1514862906 >> v2:172.21.15.39:3300/0 conn(0x7f2ff00cce90 0x7f2ff00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.852+0000 7f2ff4e00640 1 -- 172.21.15.32:0/1514862906 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0071a90 con 0x7f2ff0070f10 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.853+0000 7f2ff4e00640 1 --2- 172.21.15.32:0/1514862906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0070f10 0x7f2ff00cc950 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0072420 tx=0x7f2fe002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6e9435e1ceda7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.853+0000 7f2fefe00640 1 -- 172.21.15.32:0/1514862906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0035020 con 0x7f2ff0070f10 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.853+0000 7f2fefe00640 1 -- 172.21.15.32:0/1514862906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe002d040 con 0x7f2ff0070f10 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.853+0000 7f2fefe00640 1 -- 172.21.15.32:0/1514862906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0031a80 con 0x7f2ff0070f10 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 -- 172.21.15.32:0/1514862906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0070f10 msgr2=0x7f2ff00cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/1514862906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0070f10 0x7f2ff00cc950 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0072420 tx=0x7f2fe002b900 comp rx=0 tx=0).stop 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 -- 172.21.15.32:0/1514862906 shutdown_connections 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/1514862906 >> v2:172.21.15.39:3300/0 conn(0x7f2ff00cce90 0x7f2ff00cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/1514862906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff0070f10 0x7f2ff00cc950 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/1514862906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff00b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 -- 172.21.15.32:0/1514862906 >> 172.21.15.32:0/1514862906 conn(0x7f2ff006b3d0 msgr2=0x7f2ff006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 -- 172.21.15.32:0/1514862906 shutdown_connections 2024-03-06T07:32:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.854+0000 7f2ff6c00640 1 -- 172.21.15.32:0/1514862906 wait complete. 2024-03-06T07:32:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.856+0000 7f2ff6c00640 1 Processor -- start 2024-03-06T07:32:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.856+0000 7f2ff6c00640 1 -- start start 2024-03-06T07:32:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff6c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ff0070f10 0x7f2ff006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff6c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 0x7f2ff01f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff6c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff01f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff6c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff01456e0 con 0x7f2ff00cce90 2024-03-06T07:32:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff6c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00b4ee0 con 0x7f2ff0070f10 2024-03-06T07:32:03.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff6c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00cde70 con 0x7f2ff0145300 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 0x7f2ff01f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 0x7f2ff01f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35364/0 (socket says 172.21.15.32:35364) 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.857+0000 7f2ff4e00640 1 -- 172.21.15.32:0/4085841833 learned_addr learned my addr 172.21.15.32:0/4085841833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff6200640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff01f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff4e00640 1 -- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 msgr2=0x7f2ff01f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff5800640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ff0070f10 0x7f2ff006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff4e00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff01f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff4e00640 1 -- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ff0070f10 msgr2=0x7f2ff006e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff4e00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ff0070f10 0x7f2ff006e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff4e00640 1 -- 172.21.15.32:0/4085841833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff01f72c0 con 0x7f2ff00cce90 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff6200640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff01f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff5800640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ff0070f10 0x7f2ff006e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.858+0000 7f2ff4e00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 0x7f2ff01f6d80 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0000c00 tx=0x7f2fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.861+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0040070 con 0x7f2ff00cce90 2024-03-06T07:32:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.861+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe0039bf0 con 0x7f2ff00cce90 2024-03-06T07:32:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.861+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0035040 con 0x7f2ff00cce90 2024-03-06T07:32:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.861+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff01f7560 con 0x7f2ff00cce90 2024-03-06T07:32:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.861+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff01f7b20 con 0x7f2ff00cce90 2024-03-06T07:32:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.862+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff01e4440 con 0x7f2ff00cce90 2024-03-06T07:32:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.863+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fe0038020 con 0x7f2ff00cce90 2024-03-06T07:32:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.864+0000 7f2fed600640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fd406b4d0 0x7f2fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.864+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fe00abf10 con 0x7f2ff00cce90 2024-03-06T07:32:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.864+0000 7f2ff5800640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fd406b4d0 0x7f2fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.865+0000 7f2ff5800640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fd406b4d0 0x7f2fd406d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4006fd0 tx=0x7f2fe4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:03.869+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fe007acf0 con 0x7f2ff00cce90 2024-03-06T07:32:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.076+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2ff004f330 con 0x7f2ff00cce90 2024-03-06T07:32:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.077+0000 7f2fed600640 1 -- 172.21.15.32:0/4085841833 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2fe007a5f0 con 0x7f2ff00cce90 2024-03-06T07:32:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:32:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fd406b4d0 msgr2=0x7f2fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fd406b4d0 0x7f2fd406d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4006fd0 tx=0x7f2fe4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 msgr2=0x7f2ff01f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 0x7f2ff01f6d80 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0000c00 tx=0x7f2fe00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 shutdown_connections 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fd406b4d0 0x7f2fd406d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ff0145300 0x7f2ff01f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ff00cce90 0x7f2ff01f6d80 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 --2- 172.21.15.32:0/4085841833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ff0070f10 0x7f2ff006e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 >> 172.21.15.32:0/4085841833 conn(0x7f2ff006b3d0 msgr2=0x7f2ff00726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 shutdown_connections 2024-03-06T07:32:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.080+0000 7f2ff6c00640 1 -- 172.21.15.32:0/4085841833 wait complete. 2024-03-06T07:32:04.099 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:32:04.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:04.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:04.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:32:04.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:04.275 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:04.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.529+0000 7fd1d6400640 1 Processor -- start 2024-03-06T07:32:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.530+0000 7fd1d6400640 1 -- start start 2024-03-06T07:32:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.531+0000 7fd1d6400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d0162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.531+0000 7fd1d6400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1d00712c0 0x7fd1d006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.531+0000 7fd1d6400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d00ce090 con 0x7fd1d0070ee0 2024-03-06T07:32:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.531+0000 7fd1d6400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d00ceca0 con 0x7fd1d00712c0 2024-03-06T07:32:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.531+0000 7fd1d6400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00ce370 con 0x7fd1d0072da0 2024-03-06T07:32:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1d00712c0 0x7fd1d006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1d00712c0 0x7fd1d006e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35378/0 (socket says 172.21.15.32:35378) 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 -- 172.21.15.32:0/3763312690 learned_addr learned my addr 172.21.15.32:0/3763312690 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cf400640 1 --1- 172.21.15.32:0/3763312690 >> v1:172.21.15.32:6789/0 conn(0x7fd1d0070ee0 0x7fd1d006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38988/0 (socket says 172.21.15.32:38988) 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cea00640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d0162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 -- 172.21.15.32:0/3763312690 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 msgr2=0x7fd1d0162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d0162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 -- 172.21.15.32:0/3763312690 >> v1:172.21.15.32:6789/0 conn(0x7fd1d0070ee0 legacy=0x7fd1d006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cfe00640 1 -- 172.21.15.32:0/3763312690 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00ce650 con 0x7fd1d00712c0 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1cea00640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d0162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.533+0000 7fd1ce000640 1 -- 172.21.15.32:0/3763312690 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004964336 0 0) 0x7fd1d00ce090 con 0x7fd1d0070ee0 2024-03-06T07:32:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.534+0000 7fd1cfe00640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00712c0 0x7fd1d006e7d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009870 tx=0x7fd1c002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c8d26291b8905 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.534+0000 7fd1ce000640 1 -- 172.21.15.32:0/3763312690 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0034070 con 0x7fd1d00712c0 2024-03-06T07:32:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.534+0000 7fd1ce000640 1 -- 172.21.15.32:0/3763312690 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0030840 con 0x7fd1d00712c0 2024-03-06T07:32:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.534+0000 7fd1ce000640 1 -- 172.21.15.32:0/3763312690 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0030c10 con 0x7fd1d00712c0 2024-03-06T07:32:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 -- 172.21.15.32:0/3763312690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00712c0 msgr2=0x7fd1d006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00712c0 0x7fd1d006e7d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009870 tx=0x7fd1c002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 -- 172.21.15.32:0/3763312690 shutdown_connections 2024-03-06T07:32:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00712c0 0x7fd1d006e7d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 --2- 172.21.15.32:0/3763312690 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d0162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 -- 172.21.15.32:0/3763312690 >> 172.21.15.32:0/3763312690 conn(0x7fd1d006afb0 msgr2=0x7fd1d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 -- 172.21.15.32:0/3763312690 shutdown_connections 2024-03-06T07:32:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.535+0000 7fd1d6400640 1 -- 172.21.15.32:0/3763312690 wait complete. 2024-03-06T07:32:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.537+0000 7fd1d6400640 1 Processor -- start 2024-03-06T07:32:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.537+0000 7fd1d6400640 1 -- start start 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 0x7fd1d015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1d00712c0 0x7fd1d00aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d01f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00cdda0 con 0x7fd1d0070ee0 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00cdc20 con 0x7fd1d00712c0 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00cf130 con 0x7fd1d0072da0 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 0x7fd1d015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 0x7fd1d015fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35382/0 (socket says 172.21.15.32:35382) 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 -- 172.21.15.32:0/770667342 learned_addr learned my addr 172.21.15.32:0/770667342 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cea00640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1d00712c0 0x7fd1d00aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cfe00640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d01f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 -- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 msgr2=0x7fd1d01f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d01f81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 -- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1d00712c0 msgr2=0x7fd1d00aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1d00712c0 0x7fd1d00aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 -- 172.21.15.32:0/770667342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00acdb0 con 0x7fd1d0070ee0 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cea00640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1d00712c0 0x7fd1d00aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1cf400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 0x7fd1d015fed0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c890 tx=0x7fd1c800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8018070 con 0x7fd1d0070ee0 2024-03-06T07:32:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c800a640 con 0x7fd1d0070ee0 2024-03-06T07:32:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800aa30 con 0x7fd1d0070ee0 2024-03-06T07:32:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d00ad090 con 0x7fd1d0070ee0 2024-03-06T07:32:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.539+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d00ad650 con 0x7fd1d0070ee0 2024-03-06T07:32:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.541+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d004f3a0 con 0x7fd1d0070ee0 2024-03-06T07:32:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.542+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1c801b070 con 0x7fd1d0070ee0 2024-03-06T07:32:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.543+0000 7fd1c7400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1ac06b4d0 0x7fd1ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.543+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1c8089990 con 0x7fd1d0070ee0 2024-03-06T07:32:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.543+0000 7fd1cea00640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1ac06b4d0 0x7fd1ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.544+0000 7fd1cea00640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1ac06b4d0 0x7fd1ac06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006fd0 tx=0x7fd1b8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.553+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1c8058770 con 0x7fd1d0070ee0 2024-03-06T07:32:04.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.797+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1d00cc600 con 0x7fd1d0070ee0 2024-03-06T07:32:04.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.798+0000 7fd1c7400640 1 -- 172.21.15.32:0/770667342 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fd1c8058070 con 0x7fd1d0070ee0 2024-03-06T07:32:04.800 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:04.800 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:32:04.800 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:32:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1ac06b4d0 msgr2=0x7fd1ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1ac06b4d0 0x7fd1ac06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006fd0 tx=0x7fd1b8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 msgr2=0x7fd1d015fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 0x7fd1d015fed0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c890 tx=0x7fd1c800cd50 comp rx=0 tx=0).stop 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 shutdown_connections 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1ac06b4d0 0x7fd1ac06d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1d0072da0 0x7fd1d01f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1d00712c0 0x7fd1d00aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 --2- 172.21.15.32:0/770667342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d0070ee0 0x7fd1d015fed0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 >> 172.21.15.32:0/770667342 conn(0x7fd1d006afb0 msgr2=0x7fd1d01646d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 shutdown_connections 2024-03-06T07:32:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:04.801+0000 7fd1d6400640 1 -- 172.21.15.32:0/770667342 wait complete. 2024-03-06T07:32:04.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:32:05.056 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:32:05.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.279+0000 7f0348e00640 1 Processor -- start 2024-03-06T07:32:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.279+0000 7f0348e00640 1 -- start start 2024-03-06T07:32:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.281+0000 7f0348e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0344070ee0 0x7f0344162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.281+0000 7f0348e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0344072da0 0x7f03440cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.281+0000 7f0348e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03440ce090 con 0x7f03440712c0 2024-03-06T07:32:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.281+0000 7f0348e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03440ceca0 con 0x7f0344072da0 2024-03-06T07:32:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.281+0000 7f0348e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03440ce370 con 0x7f0344070ee0 2024-03-06T07:32:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0344072da0 0x7f03440cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0344072da0 0x7f03440cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35390/0 (socket says 172.21.15.32:35390) 2024-03-06T07:32:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0342800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f03440712c0 0x7f034406e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38990/0 (socket says 172.21.15.32:38990) 2024-03-06T07:32:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 -- 172.21.15.32:0/2104295460 learned_addr learned my addr 172.21.15.32:0/2104295460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341e00640 1 --2- 172.21.15.32:0/2104295460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0344070ee0 0x7f0344162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 -- 172.21.15.32:0/2104295460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0344070ee0 msgr2=0x7f0344162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 --2- 172.21.15.32:0/2104295460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0344070ee0 0x7f0344162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 -- 172.21.15.32:0/2104295460 >> v1:172.21.15.32:6789/0 conn(0x7f03440712c0 legacy=0x7f034406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.282+0000 7f0341400640 1 -- 172.21.15.32:0/2104295460 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03440ce650 con 0x7f0344072da0 2024-03-06T07:32:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.283+0000 7f0341400640 1 --2- 172.21.15.32:0/2104295460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344072da0 0x7f03440cf130 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0330009870 tx=0x7f033002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79dcebe346ec9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0340a00640 1 -- 172.21.15.32:0/2104295460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330034070 con 0x7f0344072da0 2024-03-06T07:32:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0340a00640 1 -- 172.21.15.32:0/2104295460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0330030840 con 0x7f0344072da0 2024-03-06T07:32:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0340a00640 1 -- 172.21.15.32:0/2104295460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330030c10 con 0x7f0344072da0 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 -- 172.21.15.32:0/2104295460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344072da0 msgr2=0x7f03440cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 --2- 172.21.15.32:0/2104295460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344072da0 0x7f03440cf130 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0330009870 tx=0x7f033002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 -- 172.21.15.32:0/2104295460 shutdown_connections 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 --2- 172.21.15.32:0/2104295460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344072da0 0x7f03440cf130 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 --2- 172.21.15.32:0/2104295460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0344070ee0 0x7f0344162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 -- 172.21.15.32:0/2104295460 >> 172.21.15.32:0/2104295460 conn(0x7f034406afb0 msgr2=0x7f034406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 -- 172.21.15.32:0/2104295460 shutdown_connections 2024-03-06T07:32:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.284+0000 7f0348e00640 1 -- 172.21.15.32:0/2104295460 wait complete. 2024-03-06T07:32:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.286+0000 7f0348e00640 1 Processor -- start 2024-03-06T07:32:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.286+0000 7f0348e00640 1 -- start start 2024-03-06T07:32:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0348e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 0x7f034415fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0348e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03440712c0 0x7f034414fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0348e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0344072da0 0x7f03440ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0348e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03440cdda0 con 0x7f0344070ee0 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0348e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03440cdc20 con 0x7f0344072da0 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0348e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034406e7d0 con 0x7f03440712c0 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0341e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 0x7f034415fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0341e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 0x7f034415fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35400/0 (socket says 172.21.15.32:35400) 2024-03-06T07:32:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.287+0000 7f0341e00640 1 -- 172.21.15.32:0/3153303027 learned_addr learned my addr 172.21.15.32:0/3153303027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341400640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03440712c0 0x7f034414fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0342800640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0344072da0 0x7f03440ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341e00640 1 -- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03440712c0 msgr2=0x7f034414fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03440712c0 0x7f034414fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341e00640 1 -- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0344072da0 msgr2=0x7f03440ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0344072da0 0x7f03440ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341e00640 1 -- 172.21.15.32:0/3153303027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034415fff0 con 0x7f0344070ee0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.288+0000 7f0341e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 0x7f034415fab0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f033400d830 tx=0x7f033400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.289+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033400b6c0 con 0x7f0344070ee0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.289+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0334019070 con 0x7f0344070ee0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.289+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033400ba80 con 0x7f0344070ee0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.289+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0344150200 con 0x7f0344070ee0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.289+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034409c930 con 0x7f0344070ee0 2024-03-06T07:32:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.290+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0344069510 con 0x7f0344070ee0 2024-03-06T07:32:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.291+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f033401e460 con 0x7f0344070ee0 2024-03-06T07:32:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.292+0000 7f033a000640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f032406b4d0 0x7f032406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.292+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f033408a820 con 0x7f0344070ee0 2024-03-06T07:32:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.292+0000 7f0341400640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f032406b4d0 0x7f032406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.293+0000 7f0341400640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f032406b4d0 0x7f032406d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0330036000 tx=0x7f033003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.301+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0334059730 con 0x7f0344070ee0 2024-03-06T07:32:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.539+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03440b52d0 con 0x7f0344070ee0 2024-03-06T07:32:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.540+0000 7f033a000640 1 -- 172.21.15.32:0/3153303027 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f0334059030 con 0x7f0344070ee0 2024-03-06T07:32:05.541 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:05.542 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:32:05.542 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:32:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f032406b4d0 msgr2=0x7f032406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f032406b4d0 0x7f032406d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0330036000 tx=0x7f033003b040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 msgr2=0x7f034415fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 0x7f034415fab0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f033400d830 tx=0x7f033400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 shutdown_connections 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f032406b4d0 0x7f032406d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0344072da0 0x7f03440ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03440712c0 0x7f034414fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 --2- 172.21.15.32:0/3153303027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0344070ee0 0x7f034415fab0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 >> 172.21.15.32:0/3153303027 conn(0x7f034406afb0 msgr2=0x7f0344166030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 shutdown_connections 2024-03-06T07:32:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:05.543+0000 7f0348e00640 1 -- 172.21.15.32:0/3153303027 wait complete. 2024-03-06T07:32:05.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:32:05.830 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:32:05.830 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:32:05.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.025+0000 7fd090e00640 1 Processor -- start 2024-03-06T07:32:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.025+0000 7fd090e00640 1 -- start start 2024-03-06T07:32:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd090e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd08c15d870 0x7fd08c066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd090e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd08c06bfc0 0x7fd08c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd090e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c06ea30 0x7fd08c06eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd090e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08c06c960 con 0x7fd08c06bfc0 2024-03-06T07:32:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd090e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08c06f3e0 con 0x7fd08c15d870 2024-03-06T07:32:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd090e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c15dc50 con 0x7fd08c06ea30 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.027+0000 7fd083e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd08c06bfc0 0x7fd08c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd08c06bfc0 0x7fd08c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35416/0 (socket says 172.21.15.32:35416) 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 -- 172.21.15.32:0/2114045515 learned_addr learned my addr 172.21.15.32:0/2114045515 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd089e00640 1 --2- 172.21.15.32:0/2114045515 >> v2:172.21.15.39:3300/0 conn(0x7fd08c15d870 0x7fd08c066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd08a800640 1 --2- 172.21.15.32:0/2114045515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c06ea30 0x7fd08c06eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 -- 172.21.15.32:0/2114045515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c06ea30 msgr2=0x7fd08c06eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 --2- 172.21.15.32:0/2114045515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c06ea30 0x7fd08c06eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 -- 172.21.15.32:0/2114045515 >> v2:172.21.15.39:3300/0 conn(0x7fd08c15d870 msgr2=0x7fd08c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 --2- 172.21.15.32:0/2114045515 >> v2:172.21.15.39:3300/0 conn(0x7fd08c15d870 0x7fd08c066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.028+0000 7fd083e00640 1 -- 172.21.15.32:0/2114045515 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c0b3650 con 0x7fd08c06bfc0 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.029+0000 7fd083e00640 1 --2- 172.21.15.32:0/2114045515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06bfc0 0x7fd08c06c420 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd08c071110 tx=0x7fd07402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203ea595e0d80216 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.030+0000 7fd089400640 1 -- 172.21.15.32:0/2114045515 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07402e070 con 0x7fd08c06bfc0 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.030+0000 7fd089400640 1 -- 172.21.15.32:0/2114045515 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074037070 con 0x7fd08c06bfc0 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.030+0000 7fd089400640 1 -- 172.21.15.32:0/2114045515 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074031a00 con 0x7fd08c06bfc0 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.030+0000 7fd090e00640 1 -- 172.21.15.32:0/2114045515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06bfc0 msgr2=0x7fd08c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.030+0000 7fd090e00640 1 --2- 172.21.15.32:0/2114045515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06bfc0 0x7fd08c06c420 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd08c071110 tx=0x7fd07402b880 comp rx=0 tx=0).stop 2024-03-06T07:32:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 -- 172.21.15.32:0/2114045515 shutdown_connections 2024-03-06T07:32:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 --2- 172.21.15.32:0/2114045515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c06ea30 0x7fd08c06eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 --2- 172.21.15.32:0/2114045515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06bfc0 0x7fd08c06c420 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 --2- 172.21.15.32:0/2114045515 >> v2:172.21.15.39:3300/0 conn(0x7fd08c15d870 0x7fd08c066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 -- 172.21.15.32:0/2114045515 >> 172.21.15.32:0/2114045515 conn(0x7fd08c0714d0 msgr2=0x7fd08c0718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 -- 172.21.15.32:0/2114045515 shutdown_connections 2024-03-06T07:32:06.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.031+0000 7fd090e00640 1 -- 172.21.15.32:0/2114045515 wait complete. 2024-03-06T07:32:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.032+0000 7fd090e00640 1 Processor -- start 2024-03-06T07:32:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.033+0000 7fd090e00640 1 -- start start 2024-03-06T07:32:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.033+0000 7fd090e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd08c06bfc0 0x7fd08c15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.033+0000 7fd090e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 0x7fd08c15b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd090e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c15d870 0x7fd08c0c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd090e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0b3ef0 con 0x7fd08c06ea30 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd090e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0b3c10 con 0x7fd08c06bfc0 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd090e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0b41d0 con 0x7fd08c15d870 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 0x7fd08c15b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 0x7fd08c15b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35418/0 (socket says 172.21.15.32:35418) 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 -- 172.21.15.32:0/162347788 learned_addr learned my addr 172.21.15.32:0/162347788 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd089e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd08c06bfc0 0x7fd08c15c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd08a800640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c15d870 0x7fd08c0c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 -- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c15d870 msgr2=0x7fd08c0c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c15d870 0x7fd08c0c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 -- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd08c06bfc0 msgr2=0x7fd08c15c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.034+0000 7fd083e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd08c06bfc0 0x7fd08c15c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd083e00640 1 -- 172.21.15.32:0/162347788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c15b790 con 0x7fd08c06ea30 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd083e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 0x7fd08c15b250 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd074000c00 tx=0x7fd0740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074032880 con 0x7fd08c06ea30 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074032a40 con 0x7fd08c06ea30 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08c1f4950 con 0x7fd08c06ea30 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074032e30 con 0x7fd08c06ea30 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.035+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08c0c5dc0 con 0x7fd08c06ea30 2024-03-06T07:32:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.037+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08c04f3a0 con 0x7fd08c06ea30 2024-03-06T07:32:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.039+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0740453f0 con 0x7fd08c06ea30 2024-03-06T07:32:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.039+0000 7fd082000640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.040+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0740ac360 con 0x7fd08c06ea30 2024-03-06T07:32:06.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.040+0000 7fd089e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.041+0000 7fd089e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06806b4d0 0x7fd06806d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd07c006fd0 tx=0x7fd07c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.045+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd074040070 con 0x7fd08c06ea30 2024-03-06T07:32:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.290+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd08c0cdbe0 con 0x7fd08c06ea30 2024-03-06T07:32:06.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.291+0000 7fd082000640 1 -- 172.21.15.32:0/162347788 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd07407b2f0 con 0x7fd08c06ea30 2024-03-06T07:32:06.292 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06806b4d0 msgr2=0x7fd06806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06806b4d0 0x7fd06806d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd07c006fd0 tx=0x7fd07c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 msgr2=0x7fd08c15b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 0x7fd08c15b250 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd074000c00 tx=0x7fd0740046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 shutdown_connections 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd08c15d870 0x7fd08c0c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd08c06ea30 0x7fd08c15b250 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 --2- 172.21.15.32:0/162347788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd08c06bfc0 0x7fd08c15c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 >> 172.21.15.32:0/162347788 conn(0x7fd08c0714d0 msgr2=0x7fd08c0718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 shutdown_connections 2024-03-06T07:32:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:06.294+0000 7fd090e00640 1 -- 172.21.15.32:0/162347788 wait complete. 2024-03-06T07:32:06.314 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:06.314 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:06.315 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:06.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.315 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:32:06.368 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:06.368 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:06.368 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:06.368 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.368 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:06.368 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42211sec preferred_lft 42211sec 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.369 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.370 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:06.371 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:06.371 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:32:06.371 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:06.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.372 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:06.463 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:32:06.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.468 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:32:06.519 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:32:06.519 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:06.519 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:32:06.519 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:32:06.519 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:06.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.521 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:32:06.521 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:32:06.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:32:06.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:32:06.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:32:06.521 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:32:06.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.708 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:32:06.759 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:06.759 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:06.760 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:32:06.859 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:06.859 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:06.922 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:06.922 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:06.922 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:06.973 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:06.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:07.110 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:07.111 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:07.145 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:07.145 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:07.146 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:32:07.146 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:32:07.146 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:32:07.150 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:07.150 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:07.150 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:07.150 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:07.150 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:07.154 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:07.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:07.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:07.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:07.239 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:07.295 INFO:teuthology.orchestra.run.smithi032.stdout:client5347 2024-03-06T07:32:07.295 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:07.295 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:07.345 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:07.345 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> import uuid 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> n = 75 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = False 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> if 0 > 0: 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = True 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> dpath = os.path.dirname(path) 2024-03-06T07:32:07.396 DEBUG:teuthology.orchestra.run.smithi032:> fnameprefix = os.path.basename(path) 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(n): 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> f.write(f"{i}") 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> f.flush() 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(f.fileno()) 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> os.unlink(fpath) 2024-03-06T07:32:07.397 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> if create_hard_links: 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> for j in range(0): 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> finally: 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> os.close(dirfd) 2024-03-06T07:32:07.398 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:32:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:32:08.557 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:08.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:08.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:08.837 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:08.837 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:08.837 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:08.894 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:08.894 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:08.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:09.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.197+0000 7fa2eb200640 1 Processor -- start 2024-03-06T07:32:09.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.197+0000 7fa2eb200640 1 -- start start 2024-03-06T07:32:09.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.199+0000 7fa2eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 0x7fa2ec0afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.199+0000 7fa2eb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa2ec066d00 0x7fa2ec0cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.199+0000 7fa2eb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ec160220 con 0x7fa2ec06afb0 2024-03-06T07:32:09.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.199+0000 7fa2eb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ec0d0260 con 0x7fa2ec066d00 2024-03-06T07:32:09.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.199+0000 7fa2eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec067200 con 0x7fa2ec15fe40 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.200+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 0x7fa2ec0afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.200+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 0x7fa2ec0afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:40982/0 (socket says 172.21.15.32:40982) 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.200+0000 7fa2e9e00640 1 -- 172.21.15.32:0/2940134941 learned_addr learned my addr 172.21.15.32:0/2940134941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.200+0000 7fa2e1400640 1 --2- 172.21.15.32:0/2940134941 >> v2:172.21.15.39:3300/0 conn(0x7fa2ec066d00 0x7fa2ec0cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.201+0000 7fa2e9e00640 1 -- 172.21.15.32:0/2940134941 >> v2:172.21.15.39:3300/0 conn(0x7fa2ec066d00 msgr2=0x7fa2ec0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.201+0000 7fa2e9e00640 1 --2- 172.21.15.32:0/2940134941 >> v2:172.21.15.39:3300/0 conn(0x7fa2ec066d00 0x7fa2ec0cfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.201+0000 7fa2e9e00640 1 -- 172.21.15.32:0/2940134941 >> v1:172.21.15.39:6789/0 conn(0x7fa2ec06afb0 legacy=0x7fa2ec0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.201+0000 7fa2e9e00640 1 -- 172.21.15.32:0/2940134941 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec06d520 con 0x7fa2ec15fe40 2024-03-06T07:32:09.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.202+0000 7fa2e9e00640 1 --2- 172.21.15.32:0/2940134941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 0x7fa2ec0afbc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009e50 tx=0x7fa2dc02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10cc8dce07b1ba6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.202+0000 7fa2e9400640 1 -- 172.21.15.32:0/2940134941 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc02c9e0 con 0x7fa2ec15fe40 2024-03-06T07:32:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2e9400640 1 -- 172.21.15.32:0/2940134941 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc02cb80 con 0x7fa2ec15fe40 2024-03-06T07:32:09.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2e9400640 1 -- 172.21.15.32:0/2940134941 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc0343f0 con 0x7fa2ec15fe40 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 -- 172.21.15.32:0/2940134941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 msgr2=0x7fa2ec0afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2940134941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 0x7fa2ec0afbc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009e50 tx=0x7fa2dc02be40 comp rx=0 tx=0).stop 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 -- 172.21.15.32:0/2940134941 shutdown_connections 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2940134941 >> v2:172.21.15.39:3300/0 conn(0x7fa2ec066d00 0x7fa2ec0cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2940134941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec15fe40 0x7fa2ec0afbc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 -- 172.21.15.32:0/2940134941 >> 172.21.15.32:0/2940134941 conn(0x7fa2ec0ccf70 msgr2=0x7fa2ec0cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 -- 172.21.15.32:0/2940134941 shutdown_connections 2024-03-06T07:32:09.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.203+0000 7fa2eb200640 1 -- 172.21.15.32:0/2940134941 wait complete. 2024-03-06T07:32:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.205+0000 7fa2eb200640 1 Processor -- start 2024-03-06T07:32:09.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.205+0000 7fa2eb200640 1 -- start start 2024-03-06T07:32:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2eb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2ec066d00 0x7fa2ec15bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec06afb0 0x7fa2ec0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 0x7fa2ec0b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2eb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec165e70 con 0x7fa2ec15fe40 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2eb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec0cf490 con 0x7fa2ec066d00 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec165ff0 con 0x7fa2ec06afb0 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2ea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 0x7fa2ec0b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2ea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 0x7fa2ec0b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53704/0 (socket says 172.21.15.32:53704) 2024-03-06T07:32:09.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.206+0000 7fa2ea800640 1 -- 172.21.15.32:0/2738302649 learned_addr learned my addr 172.21.15.32:0/2738302649 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2e1400640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec06afb0 0x7fa2ec0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2e9e00640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2ec066d00 0x7fa2ec15bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2ea800640 1 -- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec06afb0 msgr2=0x7fa2ec0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2ea800640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec06afb0 0x7fa2ec0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2ea800640 1 -- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2ec066d00 msgr2=0x7fa2ec15bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2ea800640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2ec066d00 0x7fa2ec15bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2ea800640 1 -- 172.21.15.32:0/2738302649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec0b3e40 con 0x7fa2ec15fe40 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2ea800640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 0x7fa2ec0b3810 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4001eb0 tx=0x7fa2e400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.207+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4017070 con 0x7fa2ec15fe40 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.208+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e40052d0 con 0x7fa2ec15fe40 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.208+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4003810 con 0x7fa2ec15fe40 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.208+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ec0b4120 con 0x7fa2ec15fe40 2024-03-06T07:32:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.208+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec0b49c0 con 0x7fa2ec15fe40 2024-03-06T07:32:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.209+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec04f330 con 0x7fa2ec15fe40 2024-03-06T07:32:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.210+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2e401a070 con 0x7fa2ec15fe40 2024-03-06T07:32:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.211+0000 7fa2e2a00640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2c806b500 0x7fa2c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.212+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2e4089580 con 0x7fa2ec15fe40 2024-03-06T07:32:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.212+0000 7fa2e9e00640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2c806b500 0x7fa2c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.212+0000 7fa2e9e00640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2c806b500 0x7fa2c806d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc035000 tx=0x7fa2dc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.215+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2e40583f0 con 0x7fa2ec15fe40 2024-03-06T07:32:09.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.415+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2ec0b1f80 con 0x7fa2ec15fe40 2024-03-06T07:32:09.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.416+0000 7fa2e2a00640 1 -- 172.21.15.32:0/2738302649 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fa2e4057cf0 con 0x7fa2ec15fe40 2024-03-06T07:32:09.418 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:09.418 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:42.769932+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2c806b500 msgr2=0x7fa2c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2c806b500 0x7fa2c806d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc035000 tx=0x7fa2dc030000 comp rx=0 tx=0).stop 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 msgr2=0x7fa2ec0b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 0x7fa2ec0b3810 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4001eb0 tx=0x7fa2e400c890 comp rx=0 tx=0).stop 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 shutdown_connections 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2c806b500 0x7fa2c806d9c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2ec15fe40 0x7fa2ec0b3810 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2ec06afb0 0x7fa2ec0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 --2- 172.21.15.32:0/2738302649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2ec066d00 0x7fa2ec15bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.419+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 >> 172.21.15.32:0/2738302649 conn(0x7fa2ec0ccf70 msgr2=0x7fa2ec0b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.420+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 shutdown_connections 2024-03-06T07:32:09.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.420+0000 7fa2eb200640 1 -- 172.21.15.32:0/2738302649 wait complete. 2024-03-06T07:32:09.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:39.080872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:40.231621+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:09.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.652+0000 7f81ebe00640 1 Processor -- start 2024-03-06T07:32:09.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.652+0000 7f81ebe00640 1 -- start start 2024-03-06T07:32:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.654+0000 7f81ebe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f81ec0ca120 0x7f81ec0ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.654+0000 7f81ebe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec161bd0 con 0x7f81ec0cf960 2024-03-06T07:32:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.654+0000 7f81ebe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec0d1660 con 0x7f81ec0b4820 2024-03-06T07:32:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.654+0000 7f81ebe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ec0cf480 con 0x7f81ec0ca120 2024-03-06T07:32:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.654+0000 7f81ea000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f81ec0cf960 0x7f81ec0ced70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50832/0 (socket says 172.21.15.32:50832) 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.654+0000 7f81ea000640 1 -- 172.21.15.32:0/1561002029 learned_addr learned my addr 172.21.15.32:0/1561002029 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438108190 0 0) 0x7f81ec161bd0 con 0x7f81ec0cf960 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81eaa00640 1 --2- 172.21.15.32:0/1561002029 >> v2:172.21.15.39:3300/0 conn(0x7f81ec0ca120 0x7f81ec0ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d0003520 con 0x7f81ec0cf960 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547070520 0 0) 0x7f81d0003520 con 0x7f81ec0cf960 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 >> v2:172.21.15.39:3300/0 conn(0x7f81ec0ca120 msgr2=0x7f81ec0ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 --2- 172.21.15.32:0/1561002029 >> v2:172.21.15.39:3300/0 conn(0x7f81ec0ca120 0x7f81ec0ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 >> v1:172.21.15.39:6789/0 conn(0x7f81ec0b4820 legacy=0x7f81ec0c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec0d49e0 con 0x7f81ec0cf960 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f81dc001670 con 0x7f81ec0cf960 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81eaa00640 1 --2- 172.21.15.32:0/1561002029 >> v2:172.21.15.39:3300/0 conn(0x7f81ec0ca120 0x7f81ec0ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 >> v1:172.21.15.32:6789/0 conn(0x7f81ec0cf960 legacy=0x7f81ec0ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81d0003a00 0x7f81d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0005210 0x7f81d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0cf480 con 0x7f81d0004800 2024-03-06T07:32:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d000c150 con 0x7f81d0003a00 2024-03-06T07:32:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.655+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d000c350 con 0x7f81d0005210 2024-03-06T07:32:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81ea000640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81eb400640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0005210 0x7f81d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81ea000640 1 -- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0005210 msgr2=0x7f81d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81ea000640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0005210 0x7f81d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81ea000640 1 -- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81d0003a00 msgr2=0x7f81d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81ea000640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81d0003a00 0x7f81d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81ea000640 1 -- 172.21.15.32:0/1561002029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d0003520 con 0x7f81d0004800 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81eb400640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0005210 0x7f81d000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.656+0000 7f81eb400640 1 -- 172.21.15.32:0/1561002029 reap_dead start 2024-03-06T07:32:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.657+0000 7f81ea000640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c190 tx=0x7f81d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7dfa1ee9685aad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.657+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc007c60 con 0x7f81d0004800 2024-03-06T07:32:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc030070 con 0x7f81d0004800 2024-03-06T07:32:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81e9600640 1 -- 172.21.15.32:0/1561002029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc005d90 con 0x7f81d0004800 2024-03-06T07:32:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81ebe00640 1 -- 172.21.15.32:0/1561002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d0004800 msgr2=0x7f81d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81ebe00640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c190 tx=0x7f81d0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81ebe00640 1 -- 172.21.15.32:0/1561002029 shutdown_connections 2024-03-06T07:32:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81ebe00640 1 --2- 172.21.15.32:0/1561002029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.658+0000 7f81ebe00640 1 -- 172.21.15.32:0/1561002029 >> 172.21.15.32:0/1561002029 conn(0x7f81ec0d3d20 msgr2=0x7f81ec0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.659+0000 7f81ebe00640 1 -- 172.21.15.32:0/1561002029 shutdown_connections 2024-03-06T07:32:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.659+0000 7f81ebe00640 1 -- 172.21.15.32:0/1561002029 wait complete. 2024-03-06T07:32:09.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.661+0000 7f81ebe00640 1 Processor -- start 2024-03-06T07:32:09.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.661+0000 7f81ebe00640 1 -- start start 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.662+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0004800 0x7f81ec0cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81ec0cadf0 0x7f81ec1f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 0x7f81ec1f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81ebe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0ca500 con 0x7f81ec1f3d40 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81ebe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0d1f80 con 0x7f81ec0cadf0 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81ebe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0ca680 con 0x7f81d0004800 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 0x7f81ec1f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 0x7f81ec1f41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53722/0 (socket says 172.21.15.32:53722) 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 -- 172.21.15.32:0/2677431973 learned_addr learned my addr 172.21.15.32:0/2677431973 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eaa00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0004800 0x7f81ec0cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81ea000640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81ec0cadf0 0x7f81ec1f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 -- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0004800 msgr2=0x7f81ec0cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0004800 0x7f81ec0cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 -- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81ec0cadf0 msgr2=0x7f81ec1f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81ec0cadf0 0x7f81ec1f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.663+0000 7f81eb400640 1 -- 172.21.15.32:0/2677431973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec1f4700 con 0x7f81ec1f3d40 2024-03-06T07:32:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81ea000640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81ec0cadf0 0x7f81ec1f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81eb400640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 0x7f81ec1f41c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0d35b0 tx=0x7f81e400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4018070 con 0x7f81ec1f3d40 2024-03-06T07:32:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e4003b70 con 0x7f81ec1f3d40 2024-03-06T07:32:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4005070 con 0x7f81ec1f3d40 2024-03-06T07:32:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec0b3940 con 0x7f81ec1f3d40 2024-03-06T07:32:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.664+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec0cde20 con 0x7f81ec1f3d40 2024-03-06T07:32:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.665+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec04f330 con 0x7f81ec1f3d40 2024-03-06T07:32:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.668+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81e401b070 con 0x7f81ec1f3d40 2024-03-06T07:32:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.669+0000 7f81e2a00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81c806b500 0x7f81c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.670+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81e4089930 con 0x7f81ec1f3d40 2024-03-06T07:32:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.670+0000 7f81eaa00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81c806b500 0x7f81c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.671+0000 7f81eaa00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81c806b500 0x7f81c806d9c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f81ec1f1440 tx=0x7f81d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.675+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81e4058710 con 0x7f81ec1f3d40 2024-03-06T07:32:09.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.876+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81ec0b9ba0 con 0x7f81ec1f3d40 2024-03-06T07:32:09.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.877+0000 7f81e2a00640 1 -- 172.21.15.32:0/2677431973 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81e4058010 con 0x7f81ec1f3d40 2024-03-06T07:32:09.878 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:32:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.879+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81c806b500 msgr2=0x7f81c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.879+0000 7f81ebe00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81c806b500 0x7f81c806d9c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f81ec1f1440 tx=0x7f81d4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.879+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 msgr2=0x7f81ec1f41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.879+0000 7f81ebe00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 0x7f81ec1f41c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0d35b0 tx=0x7f81e400dc00 comp rx=0 tx=0).stop 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 shutdown_connections 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81c806b500 0x7f81c806d9c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81ec1f3d40 0x7f81ec1f41c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81ec0cadf0 0x7f81ec1f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 --2- 172.21.15.32:0/2677431973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f81d0004800 0x7f81ec0cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 >> 172.21.15.32:0/2677431973 conn(0x7f81ec0d3d20 msgr2=0x7f81ec0c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 shutdown_connections 2024-03-06T07:32:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:09.880+0000 7f81ebe00640 1 -- 172.21.15.32:0/2677431973 wait complete. 2024-03-06T07:32:09.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.167+0000 7fcdee600640 1 Processor -- start 2024-03-06T07:32:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.168+0000 7fcdee600640 1 -- start start 2024-03-06T07:32:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.169+0000 7fcdee600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcde80b2940 0x7fcde815c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.169+0000 7fcdee600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcde80b2d20 0x7fcde80b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.169+0000 7fcdee600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80af360 con 0x7fcde80b5060 2024-03-06T07:32:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.169+0000 7fcdee600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde80b1410 con 0x7fcde80b2d20 2024-03-06T07:32:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.169+0000 7fcdee600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde80af640 con 0x7fcde80b2940 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcddfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcde80b2d20 0x7fcde80b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcdedc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcde80b5060 0x7fcde80b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50838/0 (socket says 172.21.15.32:50838) 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcdedc00640 1 -- 172.21.15.32:0/1385865680 learned_addr learned my addr 172.21.15.32:0/1385865680 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcded200640 1 --2- 172.21.15.32:0/1385865680 >> v2:172.21.15.39:3300/0 conn(0x7fcde80b2940 0x7fcde815c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcddfe00640 1 -- 172.21.15.32:0/1385865680 >> v2:172.21.15.39:3300/0 conn(0x7fcde80b2940 msgr2=0x7fcde815c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcddfe00640 1 --2- 172.21.15.32:0/1385865680 >> v2:172.21.15.39:3300/0 conn(0x7fcde80b2940 0x7fcde815c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcddfe00640 1 -- 172.21.15.32:0/1385865680 >> v1:172.21.15.32:6789/0 conn(0x7fcde80b5060 legacy=0x7fcde80b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcddfe00640 1 -- 172.21.15.32:0/1385865680 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80af920 con 0x7fcde80b2d20 2024-03-06T07:32:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.170+0000 7fcde7e00640 1 -- 172.21.15.32:0/1385865680 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614960551 0 0) 0x7fcde80af360 con 0x7fcde80b5060 2024-03-06T07:32:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.172+0000 7fcddfe00640 1 --2- 172.21.15.32:0/1385865680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2d20 0x7fcde80b4b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fcde806e500 tx=0x7fcdd002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f4d22ece5438a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.172+0000 7fcde7e00640 1 -- 172.21.15.32:0/1385865680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00315a0 con 0x7fcde80b2d20 2024-03-06T07:32:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.172+0000 7fcde7e00640 1 -- 172.21.15.32:0/1385865680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0037070 con 0x7fcde80b2d20 2024-03-06T07:32:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.172+0000 7fcde7e00640 1 -- 172.21.15.32:0/1385865680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0031960 con 0x7fcde80b2d20 2024-03-06T07:32:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.172+0000 7fcdee600640 1 -- 172.21.15.32:0/1385865680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2d20 msgr2=0x7fcde80b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.172+0000 7fcdee600640 1 --2- 172.21.15.32:0/1385865680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2d20 0x7fcde80b4b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fcde806e500 tx=0x7fcdd002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.173+0000 7fcdee600640 1 -- 172.21.15.32:0/1385865680 shutdown_connections 2024-03-06T07:32:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.173+0000 7fcdee600640 1 --2- 172.21.15.32:0/1385865680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2d20 0x7fcde80b4b20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.173+0000 7fcdee600640 1 --2- 172.21.15.32:0/1385865680 >> v2:172.21.15.39:3300/0 conn(0x7fcde80b2940 0x7fcde815c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.173+0000 7fcdee600640 1 -- 172.21.15.32:0/1385865680 >> 172.21.15.32:0/1385865680 conn(0x7fcde809aa10 msgr2=0x7fcde809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.173+0000 7fcdee600640 1 -- 172.21.15.32:0/1385865680 shutdown_connections 2024-03-06T07:32:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.173+0000 7fcdee600640 1 -- 172.21.15.32:0/1385865680 wait complete. 2024-03-06T07:32:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.174+0000 7fcdee600640 1 Processor -- start 2024-03-06T07:32:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 -- start start 2024-03-06T07:32:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 0x7fcde81490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcde80b2d20 0x7fcde8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcde80b5060 0x7fcde8070230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80b3190 con 0x7fcde80b2940 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80afee0 con 0x7fcde80b5060 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.175+0000 7fcdee600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80aef50 con 0x7fcde80b2d20 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 0x7fcde81490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 0x7fcde81490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53748/0 (socket says 172.21.15.32:53748) 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 -- 172.21.15.32:0/2409061952 learned_addr learned my addr 172.21.15.32:0/2409061952 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcddfe00640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcde80b2d20 0x7fcde8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 -- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcde80b2d20 msgr2=0x7fcde8157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcdedc00640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcde80b5060 0x7fcde8070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcde80b2d20 0x7fcde8157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 -- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcde80b5060 msgr2=0x7fcde8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcde80b5060 0x7fcde8070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcded200640 1 -- 172.21.15.32:0/2409061952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde8157160 con 0x7fcde80b2940 2024-03-06T07:32:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.176+0000 7fcddfe00640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcde80b2d20 0x7fcde8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.177+0000 7fcded200640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 0x7fcde81490d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007e70 tx=0x7fcdd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.177+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8014620 con 0x7fcde80b2940 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.177+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd8019070 con 0x7fcde80b2940 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.177+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8014a00 con 0x7fcde80b2940 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.177+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde81573d0 con 0x7fcde80b2940 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.177+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde81598f0 con 0x7fcde80b2940 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.178+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde804f330 con 0x7fcde80b2940 2024-03-06T07:32:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.179+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdd8014ba0 con 0x7fcde80b2940 2024-03-06T07:32:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.180+0000 7fcde5600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdc406b500 0x7fcdc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.180+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdd808aae0 con 0x7fcde80b2940 2024-03-06T07:32:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.180+0000 7fcddfe00640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdc406b500 0x7fcdc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.181+0000 7fcddfe00640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdc406b500 0x7fcdc406d9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0036000 tx=0x7fcdd0008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.185+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdd80598c0 con 0x7fcde80b2940 2024-03-06T07:32:10.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.391+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcde806fa30 con 0x7fcde80b2940 2024-03-06T07:32:10.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.392+0000 7fcde5600640 1 -- 172.21.15.32:0/2409061952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcdd80591c0 con 0x7fcde80b2940 2024-03-06T07:32:10.394 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.395+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdc406b500 msgr2=0x7fcdc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.395+0000 7fcdee600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdc406b500 0x7fcdc406d9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0036000 tx=0x7fcdd0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.395+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 msgr2=0x7fcde81490d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.395+0000 7fcdee600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 0x7fcde81490d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007e70 tx=0x7fcdd800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 shutdown_connections 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdc406b500 0x7fcdc406d9c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcde80b5060 0x7fcde8070230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcde80b2d20 0x7fcde8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 --2- 172.21.15.32:0/2409061952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcde80b2940 0x7fcde81490d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 >> 172.21.15.32:0/2409061952 conn(0x7fcde809aa10 msgr2=0x7fcde806ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 shutdown_connections 2024-03-06T07:32:10.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.396+0000 7fcdee600640 1 -- 172.21.15.32:0/2409061952 wait complete. 2024-03-06T07:32:10.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.623+0000 7f7e33e00640 1 Processor -- start 2024-03-06T07:32:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.623+0000 7f7e33e00640 1 -- start start 2024-03-06T07:32:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.624+0000 7f7e33e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e3415dee0 0x7f7e340d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.624+0000 7f7e33e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e340d0d40 0x7f7e340b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.624+0000 7f7e33e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340ceca0 con 0x7f7e340d4860 2024-03-06T07:32:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.624+0000 7f7e33e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e340cf5d0 con 0x7f7e3415dee0 2024-03-06T07:32:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.624+0000 7f7e33e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e340cef80 con 0x7f7e340d0d40 2024-03-06T07:32:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.625+0000 7f7e32a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e3415dee0 0x7f7e340d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.625+0000 7f7e32a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e3415dee0 0x7f7e340d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53750/0 (socket says 172.21.15.32:53750) 2024-03-06T07:32:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.625+0000 7f7e32a00640 1 -- 172.21.15.32:0/2450038534 learned_addr learned my addr 172.21.15.32:0/2450038534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.625+0000 7f7e33400640 1 --2- 172.21.15.32:0/2450038534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e340d0d40 0x7f7e340b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.626+0000 7f7e32a00640 1 -- 172.21.15.32:0/2450038534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e340d0d40 msgr2=0x7f7e340b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.626+0000 7f7e32a00640 1 --2- 172.21.15.32:0/2450038534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e340d0d40 0x7f7e340b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.626+0000 7f7e32a00640 1 -- 172.21.15.32:0/2450038534 >> v1:172.21.15.39:6789/0 conn(0x7f7e340d4860 legacy=0x7f7e340d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.626+0000 7f7e32a00640 1 -- 172.21.15.32:0/2450038534 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340cf260 con 0x7f7e3415dee0 2024-03-06T07:32:10.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.627+0000 7f7e32a00640 1 --2- 172.21.15.32:0/2450038534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3415dee0 0x7f7e340d18a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009870 tx=0x7f7e2402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4441ccf5670f685 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.627+0000 7f7e32000640 1 -- 172.21.15.32:0/2450038534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24034070 con 0x7f7e3415dee0 2024-03-06T07:32:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.627+0000 7f7e32000640 1 -- 172.21.15.32:0/2450038534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24030840 con 0x7f7e3415dee0 2024-03-06T07:32:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.627+0000 7f7e32000640 1 -- 172.21.15.32:0/2450038534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24030c10 con 0x7f7e3415dee0 2024-03-06T07:32:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 -- 172.21.15.32:0/2450038534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3415dee0 msgr2=0x7f7e340d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2450038534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3415dee0 0x7f7e340d18a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009870 tx=0x7f7e2402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 -- 172.21.15.32:0/2450038534 shutdown_connections 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2450038534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e340d0d40 0x7f7e340b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2450038534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3415dee0 0x7f7e340d18a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 -- 172.21.15.32:0/2450038534 >> 172.21.15.32:0/2450038534 conn(0x7f7e340d3cd0 msgr2=0x7f7e340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 -- 172.21.15.32:0/2450038534 shutdown_connections 2024-03-06T07:32:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.628+0000 7f7e33e00640 1 -- 172.21.15.32:0/2450038534 wait complete. 2024-03-06T07:32:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.630+0000 7f7e33e00640 1 Processor -- start 2024-03-06T07:32:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.630+0000 7f7e33e00640 1 -- start start 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 0x7f7e340ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e33e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e340d4860 0x7f7e340c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e33e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e3415dee0 0x7f7e341606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e33e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e340d1de0 con 0x7f7e340d0d40 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e33e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3415e2c0 con 0x7f7e340d4860 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e33e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e340d1f60 con 0x7f7e3415dee0 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e32a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 0x7f7e340ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e32a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 0x7f7e340ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53754/0 (socket says 172.21.15.32:53754) 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.631+0000 7f7e32a00640 1 -- 172.21.15.32:0/2925100861 learned_addr learned my addr 172.21.15.32:0/2925100861 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e33400640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e3415dee0 0x7f7e341606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e2be00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e340d4860 0x7f7e340c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e32a00640 1 -- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e3415dee0 msgr2=0x7f7e341606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e32a00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e3415dee0 0x7f7e341606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e32a00640 1 -- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e340d4860 msgr2=0x7f7e340c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e32a00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e340d4860 0x7f7e340c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e32a00640 1 -- 172.21.15.32:0/2925100861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340c8980 con 0x7f7e340d0d40 2024-03-06T07:32:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e32a00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 0x7f7e340ca320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000c00 tx=0x7f7e240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24034040 con 0x7f7e340d0d40 2024-03-06T07:32:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.632+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24039d90 con 0x7f7e340d0d40 2024-03-06T07:32:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.633+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e340c8c60 con 0x7f7e340d0d40 2024-03-06T07:32:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.633+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24043630 con 0x7f7e340d0d40 2024-03-06T07:32:10.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.633+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e340c9220 con 0x7f7e340d0d40 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.634+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3404f330 con 0x7f7e340d0d40 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.635+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e2404a060 con 0x7f7e340d0d40 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.636+0000 7f7e2b400640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e1006b500 0x7f7e1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.636+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e240ac210 con 0x7f7e340d0d40 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.636+0000 7f7e2be00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e1006b500 0x7f7e1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.637+0000 7f7e2be00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e1006b500 0x7f7e1006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7e340c9ef0 tx=0x7f7e1c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.641+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e2403e070 con 0x7f7e340d0d40 2024-03-06T07:32:10.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.855+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e340c75a0 con 0x7f7e340d0d40 2024-03-06T07:32:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.857+0000 7f7e2b400640 1 -- 172.21.15.32:0/2925100861 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e2402d330 con 0x7f7e340d0d40 2024-03-06T07:32:10.858 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.859+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e1006b500 msgr2=0x7f7e1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.859+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e1006b500 0x7f7e1006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7e340c9ef0 tx=0x7f7e1c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.859+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 msgr2=0x7f7e340ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.859+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 0x7f7e340ca320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000c00 tx=0x7f7e240046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.859+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 shutdown_connections 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e1006b500 0x7f7e1006d9c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e3415dee0 0x7f7e341606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e340d4860 0x7f7e340c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 --2- 172.21.15.32:0/2925100861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e340d0d40 0x7f7e340ca320 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 >> 172.21.15.32:0/2925100861 conn(0x7f7e340d3cd0 msgr2=0x7f7e340cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 shutdown_connections 2024-03-06T07:32:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:10.860+0000 7f7e33e00640 1 -- 172.21.15.32:0/2925100861 wait complete. 2024-03-06T07:32:10.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.095+0000 7fdb08800640 1 Processor -- start 2024-03-06T07:32:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.096+0000 7fdb08800640 1 -- start start 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.097+0000 7fdb08800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb000712c0 0x7fdb00162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.097+0000 7fdb08800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb000ce090 con 0x7fdb00072da0 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.097+0000 7fdb08800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb000ce370 con 0x7fdb00070ee0 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.097+0000 7fdb08800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb000ceca0 con 0x7fdb000712c0 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.098+0000 7fdb06800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb000712c0 0x7fdb00162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.098+0000 7fdb06800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb000712c0 0x7fdb00162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53760/0 (socket says 172.21.15.32:53760) 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.098+0000 7fdb06800640 1 -- 172.21.15.32:0/490144981 learned_addr learned my addr 172.21.15.32:0/490144981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.098+0000 7fdb06800640 1 -- 172.21.15.32:0/490144981 >> v1:172.21.15.39:6789/0 conn(0x7fdb00070ee0 legacy=0x7fdb0006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.098+0000 7fdb06800640 1 -- 172.21.15.32:0/490144981 >> v1:172.21.15.32:6789/0 conn(0x7fdb00072da0 legacy=0x7fdb0006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.098+0000 7fdb06800640 1 -- 172.21.15.32:0/490144981 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb000ce650 con 0x7fdb000712c0 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.099+0000 7fdb06800640 1 --2- 172.21.15.32:0/490144981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb000712c0 0x7fdb00162c40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdb000cee20 tx=0x7fdafc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb0c97916342d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb04a00640 1 -- 172.21.15.32:0/490144981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc02cb00 con 0x7fdb000712c0 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb04a00640 1 -- 172.21.15.32:0/490144981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc02cca0 con 0x7fdb000712c0 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb04a00640 1 -- 172.21.15.32:0/490144981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc034610 con 0x7fdb000712c0 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb08800640 1 -- 172.21.15.32:0/490144981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb000712c0 msgr2=0x7fdb00162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb08800640 1 --2- 172.21.15.32:0/490144981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb000712c0 0x7fdb00162c40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdb000cee20 tx=0x7fdafc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb08800640 1 -- 172.21.15.32:0/490144981 shutdown_connections 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb08800640 1 --2- 172.21.15.32:0/490144981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb000712c0 0x7fdb00162c40 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.100+0000 7fdb08800640 1 -- 172.21.15.32:0/490144981 >> 172.21.15.32:0/490144981 conn(0x7fdb0006afb0 msgr2=0x7fdb0006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.101+0000 7fdb08800640 1 -- 172.21.15.32:0/490144981 shutdown_connections 2024-03-06T07:32:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.101+0000 7fdb08800640 1 -- 172.21.15.32:0/490144981 wait complete. 2024-03-06T07:32:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.102+0000 7fdb08800640 1 Processor -- start 2024-03-06T07:32:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.102+0000 7fdb08800640 1 -- start start 2024-03-06T07:32:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.103+0000 7fdb08800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb00070ee0 0x7fdb0009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.103+0000 7fdb08800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb000712c0 0x7fdb00151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.103+0000 7fdb08800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 0x7fdb000ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.103+0000 7fdb08800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000cdda0 con 0x7fdb00072da0 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.103+0000 7fdb08800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000cdc20 con 0x7fdb00070ee0 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.103+0000 7fdb08800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0006e7d0 con 0x7fdb000712c0 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 0x7fdb000ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 0x7fdb000ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53764/0 (socket says 172.21.15.32:53764) 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 -- 172.21.15.32:0/1004838213 learned_addr learned my addr 172.21.15.32:0/1004838213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb05e00640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb00070ee0 0x7fdb0009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 -- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb000712c0 msgr2=0x7fdb00151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb05400640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb000712c0 0x7fdb00151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb000712c0 0x7fdb00151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 -- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb00070ee0 msgr2=0x7fdb0009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb00070ee0 0x7fdb0009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb06800640 1 -- 172.21.15.32:0/1004838213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0009b260 con 0x7fdb00072da0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb05e00640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb00070ee0 0x7fdb0009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.104+0000 7fdb05400640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb000712c0 0x7fdb00151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.105+0000 7fdb06800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 0x7fdb000ade00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdafc02c570 tx=0x7fdafc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.105+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc043020 con 0x7fdb00072da0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.105+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc02ce50 con 0x7fdb00072da0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.105+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0009b540 con 0x7fdb00072da0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.105+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc03e6b0 con 0x7fdb00072da0 2024-03-06T07:32:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.105+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0009bb00 con 0x7fdb00072da0 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.106+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb00069510 con 0x7fdb00072da0 2024-03-06T07:32:11.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.108+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdafc046020 con 0x7fdb00072da0 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.109+0000 7fdafa000640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdae406b4d0 0x7fdae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.109+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdafc0acbc0 con 0x7fdb00072da0 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.109+0000 7fdb05e00640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdae406b4d0 0x7fdae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.110+0000 7fdb05e00640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdae406b4d0 0x7fdae406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4003eb0 tx=0x7fdaf4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.114+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdafc04d030 con 0x7fdb00072da0 2024-03-06T07:32:11.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.331+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdb000b52d0 con 0x7fdb00072da0 2024-03-06T07:32:11.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.332+0000 7fdafa000640 1 -- 172.21.15.32:0/1004838213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdafc07bb50 con 0x7fdb00072da0 2024-03-06T07:32:11.333 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.334+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdae406b4d0 msgr2=0x7fdae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.334+0000 7fdb08800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdae406b4d0 0x7fdae406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4003eb0 tx=0x7fdaf4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.334+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 msgr2=0x7fdb000ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.334+0000 7fdb08800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 0x7fdb000ade00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdafc02c570 tx=0x7fdafc004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.334+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 shutdown_connections 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdae406b4d0 0x7fdae406d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb00072da0 0x7fdb000ade00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb000712c0 0x7fdb00151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 --2- 172.21.15.32:0/1004838213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb00070ee0 0x7fdb0009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 >> 172.21.15.32:0/1004838213 conn(0x7fdb0006afb0 msgr2=0x7fdb00166030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 shutdown_connections 2024-03-06T07:32:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.335+0000 7fdb08800640 1 -- 172.21.15.32:0/1004838213 wait complete. 2024-03-06T07:32:11.356 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:32:11.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:11.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:11.563 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:11.563 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:11.602 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:32:11.602 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.643 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:11.644 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:11.644 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.680 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:11.680 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:32:11.681 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.737 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:11.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.949+0000 7fb8c8e00640 1 Processor -- start 2024-03-06T07:32:11.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.950+0000 7fb8c8e00640 1 -- start start 2024-03-06T07:32:11.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.951+0000 7fb8c8e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 0x7fb8c40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.951+0000 7fb8c8e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8c40d0e50 0x7fb8c40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.951+0000 7fb8c8e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c40b3710 con 0x7fb8c40d4a10 2024-03-06T07:32:11.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.951+0000 7fb8c8e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c40b09e0 con 0x7fb8c40d0e50 2024-03-06T07:32:11.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.951+0000 7fb8c8e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c40b39f0 con 0x7fb8c415e0b0 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.952+0000 7fb8c3400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 0x7fb8c40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.952+0000 7fb8c3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8c40d0e50 0x7fb8c40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.952+0000 7fb8c3400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 0x7fb8c40cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:41062/0 (socket says 172.21.15.32:41062) 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.952+0000 7fb8c3400640 1 -- 172.21.15.32:0/2829970620 learned_addr learned my addr 172.21.15.32:0/2829970620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.953+0000 7fb8c3400640 1 -- 172.21.15.32:0/2829970620 >> v2:172.21.15.39:3300/0 conn(0x7fb8c40d0e50 msgr2=0x7fb8c40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.953+0000 7fb8c3400640 1 --2- 172.21.15.32:0/2829970620 >> v2:172.21.15.39:3300/0 conn(0x7fb8c40d0e50 0x7fb8c40b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.953+0000 7fb8c3400640 1 -- 172.21.15.32:0/2829970620 >> v1:172.21.15.39:6789/0 conn(0x7fb8c40d4a10 legacy=0x7fb8c40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.953+0000 7fb8c3400640 1 -- 172.21.15.32:0/2829970620 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40b3cd0 con 0x7fb8c415e0b0 2024-03-06T07:32:11.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.954+0000 7fb8c3400640 1 --2- 172.21.15.32:0/2829970620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 0x7fb8c40cf310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009870 tx=0x7fb8b402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=701bc54b1dfe23eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c2a00640 1 -- 172.21.15.32:0/2829970620 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4034070 con 0x7fb8c415e0b0 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c2a00640 1 -- 172.21.15.32:0/2829970620 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b4030840 con 0x7fb8c415e0b0 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c2a00640 1 -- 172.21.15.32:0/2829970620 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4030c10 con 0x7fb8c415e0b0 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 -- 172.21.15.32:0/2829970620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 msgr2=0x7fb8c40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/2829970620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 0x7fb8c40cf310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009870 tx=0x7fb8b402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 -- 172.21.15.32:0/2829970620 shutdown_connections 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/2829970620 >> v2:172.21.15.39:3300/0 conn(0x7fb8c40d0e50 0x7fb8c40b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/2829970620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c415e0b0 0x7fb8c40cf310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 -- 172.21.15.32:0/2829970620 >> 172.21.15.32:0/2829970620 conn(0x7fb8c40d3d30 msgr2=0x7fb8c40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 -- 172.21.15.32:0/2829970620 shutdown_connections 2024-03-06T07:32:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.955+0000 7fb8c8e00640 1 -- 172.21.15.32:0/2829970620 wait complete. 2024-03-06T07:32:11.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.957+0000 7fb8c8e00640 1 Processor -- start 2024-03-06T07:32:11.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.957+0000 7fb8c8e00640 1 -- start start 2024-03-06T07:32:11.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.958+0000 7fb8c8e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c40d0e50 0x7fb8c41583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.958+0000 7fb8c8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8c40d4a10 0x7fb8c40cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.958+0000 7fb8c8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 0x7fb8c40cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.958+0000 7fb8c8e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c40cfc60 con 0x7fb8c415e0b0 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.958+0000 7fb8c8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c415e490 con 0x7fb8c40d4a10 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.958+0000 7fb8c8e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c40cf850 con 0x7fb8c40d0e50 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 0x7fb8c40cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 0x7fb8c40cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53774/0 (socket says 172.21.15.32:53774) 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 -- 172.21.15.32:0/3995955922 learned_addr learned my addr 172.21.15.32:0/3995955922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3400640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c40d0e50 0x7fb8c41583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8bbe00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8c40d4a10 0x7fb8c40cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 -- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c40d0e50 msgr2=0x7fb8c41583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c40d0e50 0x7fb8c41583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 -- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8c40d4a10 msgr2=0x7fb8c40cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8c40d4a10 0x7fb8c40cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.959+0000 7fb8c3e00640 1 -- 172.21.15.32:0/3995955922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c41f71c0 con 0x7fb8c415e0b0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.960+0000 7fb8c3e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 0x7fb8c40cceb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00efc0 tx=0x7fb8bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.960+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc018020 con 0x7fb8c415e0b0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.960+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8bc014480 con 0x7fb8c415e0b0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.960+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c41f7460 con 0x7fb8c415e0b0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.960+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc014870 con 0x7fb8c415e0b0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.960+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c41f7a20 con 0x7fb8c415e0b0 2024-03-06T07:32:11.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.961+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c404f330 con 0x7fb8c415e0b0 2024-03-06T07:32:11.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.962+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8bc005d10 con 0x7fb8c415e0b0 2024-03-06T07:32:11.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.963+0000 7fb8bb400640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8a006b4d0 0x7fb8a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.963+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8bc08adc0 con 0x7fb8c415e0b0 2024-03-06T07:32:11.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.964+0000 7fb8c3400640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8a006b4d0 0x7fb8a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.964+0000 7fb8c3400640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8a006b4d0 0x7fb8a006d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4036000 tx=0x7fb8b403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:11.968+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8bc022070 con 0x7fb8c415e0b0 2024-03-06T07:32:12.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.209+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8c40cf470 con 0x7fb8c415e0b0 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.210+0000 7fb8bb400640 1 -- 172.21.15.32:0/3995955922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fb8bc010040 con 0x7fb8c415e0b0 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8a006b4d0 msgr2=0x7fb8a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8a006b4d0 0x7fb8a006d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4036000 tx=0x7fb8b403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 msgr2=0x7fb8c40cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 0x7fb8c40cceb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00efc0 tx=0x7fb8bc00c490 comp rx=0 tx=0).stop 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 shutdown_connections 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb8a006b4d0 0x7fb8a006d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c415e0b0 0x7fb8c40cceb0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8c40d4a10 0x7fb8c40cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 --2- 172.21.15.32:0/3995955922 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8c40d0e50 0x7fb8c41583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 >> 172.21.15.32:0/3995955922 conn(0x7fb8c40d3d30 msgr2=0x7fb8c4074070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 shutdown_connections 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.213+0000 7fb8c8e00640 1 -- 172.21.15.32:0/3995955922 wait complete. 2024-03-06T07:32:12.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:12.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.452+0000 7fe01e600640 1 Processor -- start 2024-03-06T07:32:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.452+0000 7fe01e600640 1 -- start start 2024-03-06T07:32:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01e600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d4a10 0x7fe0180d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01e600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0180d0c80 0x7fe0180b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01e600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0180b3710 con 0x7fe01815e0b0 2024-03-06T07:32:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01e600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0180b09e0 con 0x7fe0180d0c80 2024-03-06T07:32:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01e600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180cfa20 con 0x7fe0180d4a10 2024-03-06T07:32:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01dc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0180d0c80 0x7fe0180b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01d200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe01815e0b0 0x7fe0180cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50844/0 (socket says 172.21.15.32:50844) 2024-03-06T07:32:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.454+0000 7fe01d200640 1 -- 172.21.15.32:0/172720735 learned_addr learned my addr 172.21.15.32:0/172720735 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.455+0000 7fe017e00640 1 --2- 172.21.15.32:0/172720735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d4a10 0x7fe0180d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.455+0000 7fe017400640 1 -- 172.21.15.32:0/172720735 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649515607 0 0) 0x7fe0180b3710 con 0x7fe01815e0b0 2024-03-06T07:32:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.455+0000 7fe01dc00640 1 -- 172.21.15.32:0/172720735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d4a10 msgr2=0x7fe0180d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.455+0000 7fe01dc00640 1 --2- 172.21.15.32:0/172720735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d4a10 0x7fe0180d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.455+0000 7fe01dc00640 1 -- 172.21.15.32:0/172720735 >> v1:172.21.15.32:6789/0 conn(0x7fe01815e0b0 legacy=0x7fe0180cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.455+0000 7fe01dc00640 1 -- 172.21.15.32:0/172720735 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180b39f0 con 0x7fe0180d0c80 2024-03-06T07:32:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.456+0000 7fe01dc00640 1 --2- 172.21.15.32:0/172720735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0180d0c80 0x7fe0180b44d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe010009d90 tx=0x7fe01002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f137c4c744461b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.456+0000 7fe017400640 1 -- 172.21.15.32:0/172720735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010039070 con 0x7fe0180d0c80 2024-03-06T07:32:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.456+0000 7fe017400640 1 -- 172.21.15.32:0/172720735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01002cc50 con 0x7fe0180d0c80 2024-03-06T07:32:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.456+0000 7fe017400640 1 -- 172.21.15.32:0/172720735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010034440 con 0x7fe0180d0c80 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 -- 172.21.15.32:0/172720735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0180d0c80 msgr2=0x7fe0180b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 --2- 172.21.15.32:0/172720735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0180d0c80 0x7fe0180b44d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe010009d90 tx=0x7fe01002bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 -- 172.21.15.32:0/172720735 shutdown_connections 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 --2- 172.21.15.32:0/172720735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0180d0c80 0x7fe0180b44d0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 --2- 172.21.15.32:0/172720735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d4a10 0x7fe0180d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 -- 172.21.15.32:0/172720735 >> 172.21.15.32:0/172720735 conn(0x7fe0180d3d30 msgr2=0x7fe0180d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 -- 172.21.15.32:0/172720735 shutdown_connections 2024-03-06T07:32:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.457+0000 7fe01e600640 1 -- 172.21.15.32:0/172720735 wait complete. 2024-03-06T07:32:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.459+0000 7fe01e600640 1 Processor -- start 2024-03-06T07:32:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.459+0000 7fe01e600640 1 -- start start 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01e600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d0c80 0x7fe0180cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0180d4a10 0x7fe01806c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 0x7fe01815b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01815e490 con 0x7fe01815e0b0 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01e600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180b3fb0 con 0x7fe0180d4a10 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01e600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180cfc60 con 0x7fe0180d0c80 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.460+0000 7fe01dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 0x7fe01815b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 0x7fe01815b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53794/0 (socket says 172.21.15.32:53794) 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 -- 172.21.15.32:0/2283601326 learned_addr learned my addr 172.21.15.32:0/2283601326 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01d200640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d0c80 0x7fe0180cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe017e00640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0180d4a10 0x7fe01806c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 -- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d0c80 msgr2=0x7fe0180cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d0c80 0x7fe0180cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 -- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0180d4a10 msgr2=0x7fe01806c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0180d4a10 0x7fe01806c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.461+0000 7fe01dc00640 1 -- 172.21.15.32:0/2283601326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01815b9a0 con 0x7fe01815e0b0 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.462+0000 7fe01dc00640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 0x7fe01815b460 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe010009d90 tx=0x7fe010004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.462+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010039050 con 0x7fe01815e0b0 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.462+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe010045070 con 0x7fe01815e0b0 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.462+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0181f38b0 con 0x7fe01815e0b0 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.462+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01003e610 con 0x7fe01815e0b0 2024-03-06T07:32:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.462+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0181f3e70 con 0x7fe01815e0b0 2024-03-06T07:32:12.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.464+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01804f330 con 0x7fe01815e0b0 2024-03-06T07:32:12.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.465+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe01003e7b0 con 0x7fe01815e0b0 2024-03-06T07:32:12.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.466+0000 7fe014c00640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdff806b4d0 0x7fdff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.466+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0100acdc0 con 0x7fe01815e0b0 2024-03-06T07:32:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.467+0000 7fe01d200640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdff806b4d0 0x7fdff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.467+0000 7fe01d200640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdff806b4d0 0x7fdff806d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe00c000f60 tx=0x7fe00c007b70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.471+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe010032e20 con 0x7fe01815e0b0 2024-03-06T07:32:12.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.672+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe018074920 con 0x7fe01815e0b0 2024-03-06T07:32:12.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.673+0000 7fe014c00640 1 -- 172.21.15.32:0/2283601326 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fe01007bcc0 con 0x7fe01815e0b0 2024-03-06T07:32:12.675 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:12.675 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:42.769932+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdff806b4d0 msgr2=0x7fdff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdff806b4d0 0x7fdff806d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe00c000f60 tx=0x7fe00c007b70 comp rx=0 tx=0).stop 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 msgr2=0x7fe01815b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 0x7fe01815b460 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe010009d90 tx=0x7fe010004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 shutdown_connections 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdff806b4d0 0x7fdff806d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01815e0b0 0x7fe01815b460 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0180d4a10 0x7fe01806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 --2- 172.21.15.32:0/2283601326 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe0180d0c80 0x7fe0180cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.676+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 >> 172.21.15.32:0/2283601326 conn(0x7fe0180d3d30 msgr2=0x7fe0180cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.677+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 shutdown_connections 2024-03-06T07:32:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.677+0000 7fe01e600640 1 -- 172.21.15.32:0/2283601326 wait complete. 2024-03-06T07:32:12.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:39.080872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:40.231621+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:12.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.910+0000 7f2db3200640 1 Processor -- start 2024-03-06T07:32:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.910+0000 7f2db3200640 1 -- start start 2024-03-06T07:32:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2db3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dac15bf20 0x7f2dac0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2db3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2dac072520 0x7f2dac0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2db3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac072df0 con 0x7f2dac071e20 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2db3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac0cdab0 con 0x7f2dac15bf20 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2db3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac0b4d70 con 0x7f2dac072520 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2dabe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dac15bf20 0x7f2dac0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2dabe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dac15bf20 0x7f2dac0acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53804/0 (socket says 172.21.15.32:53804) 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.912+0000 7f2dabe00640 1 -- 172.21.15.32:0/1198959593 learned_addr learned my addr 172.21.15.32:0/1198959593 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.913+0000 7f2dab400640 1 --2- 172.21.15.32:0/1198959593 >> v2:172.21.15.39:3300/0 conn(0x7f2dac072520 0x7f2dac0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.913+0000 7f2dabe00640 1 -- 172.21.15.32:0/1198959593 >> v2:172.21.15.39:3300/0 conn(0x7f2dac072520 msgr2=0x7f2dac0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.913+0000 7f2dabe00640 1 --2- 172.21.15.32:0/1198959593 >> v2:172.21.15.39:3300/0 conn(0x7f2dac072520 0x7f2dac0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.913+0000 7f2dabe00640 1 -- 172.21.15.32:0/1198959593 >> v1:172.21.15.39:6789/0 conn(0x7f2dac071e20 legacy=0x7f2dac0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.913+0000 7f2dabe00640 1 -- 172.21.15.32:0/1198959593 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0b5050 con 0x7f2dac15bf20 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.915+0000 7f2dabe00640 1 --2- 172.21.15.32:0/1198959593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac15bf20 0x7f2dac0acd40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2dac070b90 tx=0x7f2d9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82acffe14cf46b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.915+0000 7f2daaa00640 1 -- 172.21.15.32:0/1198959593 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c035020 con 0x7f2dac15bf20 2024-03-06T07:32:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.915+0000 7f2daaa00640 1 -- 172.21.15.32:0/1198959593 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c031870 con 0x7f2dac15bf20 2024-03-06T07:32:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.915+0000 7f2daaa00640 1 -- 172.21.15.32:0/1198959593 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c031c40 con 0x7f2dac15bf20 2024-03-06T07:32:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 -- 172.21.15.32:0/1198959593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac15bf20 msgr2=0x7f2dac0acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 --2- 172.21.15.32:0/1198959593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac15bf20 0x7f2dac0acd40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2dac070b90 tx=0x7f2d9c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 -- 172.21.15.32:0/1198959593 shutdown_connections 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 --2- 172.21.15.32:0/1198959593 >> v2:172.21.15.39:3300/0 conn(0x7f2dac072520 0x7f2dac0cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 --2- 172.21.15.32:0/1198959593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac15bf20 0x7f2dac0acd40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 -- 172.21.15.32:0/1198959593 >> 172.21.15.32:0/1198959593 conn(0x7f2dac06b3d0 msgr2=0x7f2dac06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 -- 172.21.15.32:0/1198959593 shutdown_connections 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.916+0000 7f2db3200640 1 -- 172.21.15.32:0/1198959593 wait complete. 2024-03-06T07:32:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.918+0000 7f2db3200640 1 Processor -- start 2024-03-06T07:32:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.918+0000 7f2db3200640 1 -- start start 2024-03-06T07:32:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2db3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dac071e20 0x7f2dac0b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2db3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 0x7f2dac14e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2db3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2dac15bf20 0x7f2dac14ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2db3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0b4b10 con 0x7f2dac072520 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2db3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0b4990 con 0x7f2dac071e20 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2db3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0cd1e0 con 0x7f2dac15bf20 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2dab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 0x7f2dac14e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2dab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 0x7f2dac14e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53806/0 (socket says 172.21.15.32:53806) 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2dab400640 1 -- 172.21.15.32:0/2745243572 learned_addr learned my addr 172.21.15.32:0/2745243572 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.919+0000 7f2dabe00640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dac071e20 0x7f2dac0b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2dab400640 1 -- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2dac15bf20 msgr2=0x7f2dac14ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2dab400640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2dac15bf20 0x7f2dac14ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2dab400640 1 -- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dac071e20 msgr2=0x7f2dac0b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2dab400640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dac071e20 0x7f2dac0b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2dab400640 1 -- 172.21.15.32:0/2745243572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0b1640 con 0x7f2dac072520 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2dab400640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 0x7f2dac14e4e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800d800 tx=0x7f2d9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800b6c0 con 0x7f2dac072520 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98019070 con 0x7f2dac072520 2024-03-06T07:32:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800ba80 con 0x7f2dac072520 2024-03-06T07:32:12.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac0afd90 con 0x7f2dac072520 2024-03-06T07:32:12.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.920+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac0b0350 con 0x7f2dac072520 2024-03-06T07:32:12.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.922+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac04f330 con 0x7f2dac072520 2024-03-06T07:32:12.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.923+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d9800bc20 con 0x7f2dac072520 2024-03-06T07:32:12.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.924+0000 7f2da3e00640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2d8c06b500 0x7f2d8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.925+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d9808a6d0 con 0x7f2dac072520 2024-03-06T07:32:12.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.925+0000 7f2dabe00640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2d8c06b500 0x7f2d8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.925+0000 7f2dabe00640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2d8c06b500 0x7f2d8c06d9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c036000 tx=0x7f2d9c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:12.929+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d9801c360 con 0x7f2dac072520 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.130+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2dac0b06f0 con 0x7f2dac072520 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.131+0000 7f2da3e00640 1 -- 172.21.15.32:0/2745243572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d98059540 con 0x7f2dac072520 2024-03-06T07:32:13.132 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2d8c06b500 msgr2=0x7f2d8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2d8c06b500 0x7f2d8c06d9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c036000 tx=0x7f2d9c03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 msgr2=0x7f2dac14e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 0x7f2dac14e4e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800d800 tx=0x7f2d9800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 shutdown_connections 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2d8c06b500 0x7f2d8c06d9c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2dac15bf20 0x7f2dac14ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dac072520 0x7f2dac14e4e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 --2- 172.21.15.32:0/2745243572 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dac071e20 0x7f2dac0b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 >> 172.21.15.32:0/2745243572 conn(0x7f2dac06b3d0 msgr2=0x7f2dac0cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 shutdown_connections 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.134+0000 7f2db3200640 1 -- 172.21.15.32:0/2745243572 wait complete. 2024-03-06T07:32:13.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.383+0000 7f83d5400640 1 Processor -- start 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.383+0000 7f83d5400640 1 -- start start 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.386+0000 7f83d5400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b3c20 0x7f83d00729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.386+0000 7f83d5400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83d00b4000 0x7f83d015b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.386+0000 7f83d5400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d0071930 con 0x7f83d00b1b40 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.386+0000 7f83d5400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d00b4460 con 0x7f83d00b4000 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.386+0000 7f83d5400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d0071c10 con 0x7f83d00b3c20 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83d00b4000 0x7f83d015b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83d00b4000 0x7f83d015b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53816/0 (socket says 172.21.15.32:53816) 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83d4a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f83d00b1b40 0x7f83d015c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50860/0 (socket says 172.21.15.32:50860) 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 -- 172.21.15.32:0/3719275628 learned_addr learned my addr 172.21.15.32:0/3719275628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cfe00640 1 --2- 172.21.15.32:0/3719275628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b3c20 0x7f83d00729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 -- 172.21.15.32:0/3719275628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b3c20 msgr2=0x7f83d00729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 --2- 172.21.15.32:0/3719275628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b3c20 0x7f83d00729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 -- 172.21.15.32:0/3719275628 >> v1:172.21.15.32:6789/0 conn(0x7f83d00b1b40 legacy=0x7f83d015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cea00640 1 -- 172.21.15.32:0/3719275628 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274510346 0 0) 0x7f83d0071930 con 0x7f83d00b1b40 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.387+0000 7f83cf400640 1 -- 172.21.15.32:0/3719275628 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0071ef0 con 0x7f83d00b4000 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.388+0000 7f83cf400640 1 --2- 172.21.15.32:0/3719275628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b4000 0x7f83d015b8a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009870 tx=0x7f83bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c624f1f62e822ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.389+0000 7f83cea00640 1 -- 172.21.15.32:0/3719275628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc034070 con 0x7f83d00b4000 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.389+0000 7f83cea00640 1 -- 172.21.15.32:0/3719275628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83bc030840 con 0x7f83d00b4000 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.389+0000 7f83cea00640 1 -- 172.21.15.32:0/3719275628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc030c10 con 0x7f83d00b4000 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 -- 172.21.15.32:0/3719275628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b4000 msgr2=0x7f83d015b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 --2- 172.21.15.32:0/3719275628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b4000 0x7f83d015b8a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009870 tx=0x7f83bc02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 -- 172.21.15.32:0/3719275628 shutdown_connections 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 --2- 172.21.15.32:0/3719275628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b4000 0x7f83d015b8a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 --2- 172.21.15.32:0/3719275628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b3c20 0x7f83d00729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 -- 172.21.15.32:0/3719275628 >> 172.21.15.32:0/3719275628 conn(0x7f83d006b3a0 msgr2=0x7f83d006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 -- 172.21.15.32:0/3719275628 shutdown_connections 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.390+0000 7f83d5400640 1 -- 172.21.15.32:0/3719275628 wait complete. 2024-03-06T07:32:13.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.392+0000 7f83d5400640 1 Processor -- start 2024-03-06T07:32:13.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.392+0000 7f83d5400640 1 -- start start 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.393+0000 7f83d5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83d00b1b40 0x7f83d00af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.393+0000 7f83d5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 0x7f83d0156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.393+0000 7f83d5400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b4000 0x7f83d00ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.393+0000 7f83d5400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d015bde0 con 0x7f83d00b3c20 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.393+0000 7f83d5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d00724b0 con 0x7f83d00b1b40 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.393+0000 7f83d5400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d015bf60 con 0x7f83d00b4000 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 0x7f83d0156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 0x7f83d0156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53828/0 (socket says 172.21.15.32:53828) 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 -- 172.21.15.32:0/4092393152 learned_addr learned my addr 172.21.15.32:0/4092393152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:13.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cfe00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83d00b1b40 0x7f83d00af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83d4a00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b4000 0x7f83d00ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 -- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b4000 msgr2=0x7f83d00ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b4000 0x7f83d00ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 -- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83d00b1b40 msgr2=0x7f83d00af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83d00b1b40 0x7f83d00af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cf400640 1 -- 172.21.15.32:0/4092393152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0156ee0 con 0x7f83d00b3c20 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83cfe00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83d00b1b40 0x7f83d00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.394+0000 7f83d4a00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b4000 0x7f83d00ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.395+0000 7f83cf400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 0x7f83d0156770 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0090b0 tx=0x7f83bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.395+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc034040 con 0x7f83d00b3c20 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.395+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83bc03ab90 con 0x7f83d00b3c20 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.395+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc004030 con 0x7f83d00b3c20 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.395+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d01571c0 con 0x7f83d00b3c20 2024-03-06T07:32:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.395+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d0157780 con 0x7f83d00b3c20 2024-03-06T07:32:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.396+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d004f330 con 0x7f83d00b3c20 2024-03-06T07:32:13.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.398+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83bc04b060 con 0x7f83d00b3c20 2024-03-06T07:32:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.400+0000 7f83c7e00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f83b006b4d0 0x7f83b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.400+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83bc0ac0f0 con 0x7f83d00b3c20 2024-03-06T07:32:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.400+0000 7f83cfe00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f83b006b4d0 0x7f83b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.401+0000 7f83cfe00640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f83b006b4d0 0x7f83b006d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009ac0 tx=0x7f83c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.405+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83bc03f070 con 0x7f83d00b3c20 2024-03-06T07:32:13.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.604+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83d0074ab0 con 0x7f83d00b3c20 2024-03-06T07:32:13.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.605+0000 7f83c7e00640 1 -- 172.21.15.32:0/4092393152 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f83bc07ae40 con 0x7f83d00b3c20 2024-03-06T07:32:13.607 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:32:13.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f83b006b4d0 msgr2=0x7f83b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f83b006b4d0 0x7f83b006d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009ac0 tx=0x7f83c0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 msgr2=0x7f83d0156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 0x7f83d0156770 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0090b0 tx=0x7f83bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 shutdown_connections 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f83b006b4d0 0x7f83b006d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83d00b4000 0x7f83d00ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d00b3c20 0x7f83d0156770 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 --2- 172.21.15.32:0/4092393152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83d00b1b40 0x7f83d00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 >> 172.21.15.32:0/4092393152 conn(0x7f83d006b3a0 msgr2=0x7f83d006d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 shutdown_connections 2024-03-06T07:32:13.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.608+0000 7f83d5400640 1 -- 172.21.15.32:0/4092393152 wait complete. 2024-03-06T07:32:13.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.860+0000 7f6e7da00640 1 Processor -- start 2024-03-06T07:32:13.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.860+0000 7f6e7da00640 1 -- start start 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e7da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d07d0 0x7f6e780b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e7da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e7815e6c0 0x7f6e780b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e7da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780cf780 con 0x7f6e780d4930 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e7da00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e780b3770 con 0x7f6e7815e6c0 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e7da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780cfa60 con 0x7f6e780d07d0 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e77400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e7815e6c0 0x7f6e780b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e77400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e7815e6c0 0x7f6e780b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53840/0 (socket says 172.21.15.32:53840) 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.862+0000 7f6e77400640 1 -- 172.21.15.32:0/135752316 learned_addr learned my addr 172.21.15.32:0/135752316 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.863+0000 7f6e76a00640 1 --2- 172.21.15.32:0/135752316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d07d0 0x7f6e780b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.863+0000 7f6e77400640 1 -- 172.21.15.32:0/135752316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d07d0 msgr2=0x7f6e780b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.863+0000 7f6e77400640 1 --2- 172.21.15.32:0/135752316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d07d0 0x7f6e780b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.863+0000 7f6e77400640 1 -- 172.21.15.32:0/135752316 >> v1:172.21.15.39:6789/0 conn(0x7f6e780d4930 legacy=0x7f6e7815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.863+0000 7f6e77400640 1 -- 172.21.15.32:0/135752316 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780cfd40 con 0x7f6e7815e6c0 2024-03-06T07:32:13.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.865+0000 7f6e77400640 1 --2- 172.21.15.32:0/135752316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e7815e6c0 0x7f6e780b3c00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009e50 tx=0x7f6e7002bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e827cd157ab89d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.865+0000 7f6e75600640 1 -- 172.21.15.32:0/135752316 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7002cb80 con 0x7f6e7815e6c0 2024-03-06T07:32:13.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.865+0000 7f6e75600640 1 -- 172.21.15.32:0/135752316 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7002cd20 con 0x7f6e7815e6c0 2024-03-06T07:32:13.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.865+0000 7f6e75600640 1 -- 172.21.15.32:0/135752316 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70034610 con 0x7f6e7815e6c0 2024-03-06T07:32:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 -- 172.21.15.32:0/135752316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e7815e6c0 msgr2=0x7f6e780b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 --2- 172.21.15.32:0/135752316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e7815e6c0 0x7f6e780b3c00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009e50 tx=0x7f6e7002bfe0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 -- 172.21.15.32:0/135752316 shutdown_connections 2024-03-06T07:32:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 --2- 172.21.15.32:0/135752316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e7815e6c0 0x7f6e780b3c00 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 --2- 172.21.15.32:0/135752316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d07d0 0x7f6e780b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 -- 172.21.15.32:0/135752316 >> 172.21.15.32:0/135752316 conn(0x7f6e780d3e10 msgr2=0x7f6e780d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 -- 172.21.15.32:0/135752316 shutdown_connections 2024-03-06T07:32:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.866+0000 7f6e7da00640 1 -- 172.21.15.32:0/135752316 wait complete. 2024-03-06T07:32:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.868+0000 7f6e7da00640 1 Processor -- start 2024-03-06T07:32:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.869+0000 7f6e7da00640 1 -- start start 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e7da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 0x7f6e780ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e7da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d4930 0x7f6e780cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e7da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e7815e6c0 0x7f6e781f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e7da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780d0d30 con 0x7f6e780d07d0 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e7da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780d0bb0 con 0x7f6e7815e6c0 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e7da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780d0eb0 con 0x7f6e780d4930 2024-03-06T07:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 0x7f6e780ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 0x7f6e780ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53846/0 (socket says 172.21.15.32:53846) 2024-03-06T07:32:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e76a00640 1 -- 172.21.15.32:0/4158108235 learned_addr learned my addr 172.21.15.32:0/4158108235 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e77400640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e7815e6c0 0x7f6e781f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e76000640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d4930 0x7f6e780cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.870+0000 7f6e76a00640 1 -- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d4930 msgr2=0x7f6e780cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e76a00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d4930 0x7f6e780cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e76a00640 1 -- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e7815e6c0 msgr2=0x7f6e781f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e76a00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e7815e6c0 0x7f6e781f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e76a00640 1 -- 172.21.15.32:0/4158108235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780ca6f0 con 0x7f6e780d07d0 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e77400640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e7815e6c0 0x7f6e781f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e76000640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d4930 0x7f6e780cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e76a00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 0x7f6e780ca1b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800d800 tx=0x7f6e6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6800b6c0 con 0x7f6e780d07d0 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68019070 con 0x7f6e780d07d0 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780c93f0 con 0x7f6e780d07d0 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6800bac0 con 0x7f6e780d07d0 2024-03-06T07:32:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.871+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e780c99b0 con 0x7f6e780d07d0 2024-03-06T07:32:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.873+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7804f330 con 0x7f6e780d07d0 2024-03-06T07:32:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.874+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e6800bc60 con 0x7f6e780d07d0 2024-03-06T07:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.874+0000 7f6e6ea00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e5806b4d0 0x7f6e5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.875+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e6808a650 con 0x7f6e780d07d0 2024-03-06T07:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.875+0000 7f6e76000640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e5806b4d0 0x7f6e5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.876+0000 7f6e76000640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e5806b4d0 0x7f6e5806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6e780cc160 tx=0x7f6e64009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:13.883+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e68059560 con 0x7f6e780d07d0 2024-03-06T07:32:14.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.149+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e780d2fb0 con 0x7f6e780d07d0 2024-03-06T07:32:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.149+0000 7f6e6ea00640 1 -- 172.21.15.32:0/4158108235 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e68058e60 con 0x7f6e780d07d0 2024-03-06T07:32:14.151 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:32:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e5806b4d0 msgr2=0x7f6e5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e5806b4d0 0x7f6e5806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6e780cc160 tx=0x7f6e64009040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 msgr2=0x7f6e780ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 0x7f6e780ca1b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800d800 tx=0x7f6e6800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 shutdown_connections 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e5806b4d0 0x7f6e5806d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e7815e6c0 0x7f6e781f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e780d4930 0x7f6e780cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 --2- 172.21.15.32:0/4158108235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e780d07d0 0x7f6e780ca1b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.152+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 >> 172.21.15.32:0/4158108235 conn(0x7f6e780d3e10 msgr2=0x7f6e7815d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.153+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 shutdown_connections 2024-03-06T07:32:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.153+0000 7f6e7da00640 1 -- 172.21.15.32:0/4158108235 wait complete. 2024-03-06T07:32:14.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.368+0000 7fad1ce00640 1 Processor -- start 2024-03-06T07:32:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.368+0000 7fad1ce00640 1 -- start start 2024-03-06T07:32:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad1ce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fad1806c690 0x7fad1815c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad1ce00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad180afb40 con 0x7fad1815c360 2024-03-06T07:32:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad1ce00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad180afe20 con 0x7fad1806ed90 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad1ce00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1806d310 con 0x7fad1806c690 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad17400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fad1815c360 0x7fad180ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50866/0 (socket says 172.21.15.32:50866) 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad17400640 1 -- 172.21.15.32:0/387059749 learned_addr learned my addr 172.21.15.32:0/387059749 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.370+0000 7fad17e00640 1 --2- 172.21.15.32:0/387059749 >> v2:172.21.15.32:3300/0 conn(0x7fad1806c690 0x7fad1815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.371+0000 7fad16000640 1 -- 172.21.15.32:0/387059749 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437065189 0 0) 0x7fad180afb40 con 0x7fad1815c360 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.371+0000 7fad16000640 1 -- 172.21.15.32:0/387059749 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facfc003520 con 0x7fad1815c360 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.371+0000 7fad17e00640 1 -- 172.21.15.32:0/387059749 >> v1:172.21.15.39:6789/0 conn(0x7fad1806ed90 legacy=0x7fad1806bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.371+0000 7fad17e00640 1 -- 172.21.15.32:0/387059749 >> v1:172.21.15.32:6789/0 conn(0x7fad1815c360 legacy=0x7fad180ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.371+0000 7fad17e00640 1 -- 172.21.15.32:0/387059749 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1806d560 con 0x7fad1806c690 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.373+0000 7fad17e00640 1 --2- 172.21.15.32:0/387059749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806c690 0x7fad1815c740 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fad10009d90 tx=0x7fad1002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecde283f26d3a450 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.373+0000 7fad16000640 1 -- 172.21.15.32:0/387059749 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10039070 con 0x7fad1806c690 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.373+0000 7fad16000640 1 -- 172.21.15.32:0/387059749 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1002cc60 con 0x7fad1806c690 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.373+0000 7fad16000640 1 -- 172.21.15.32:0/387059749 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10034640 con 0x7fad1806c690 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 -- 172.21.15.32:0/387059749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806c690 msgr2=0x7fad1815c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 --2- 172.21.15.32:0/387059749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806c690 0x7fad1815c740 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fad10009d90 tx=0x7fad1002beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 -- 172.21.15.32:0/387059749 shutdown_connections 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 --2- 172.21.15.32:0/387059749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806c690 0x7fad1815c740 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 -- 172.21.15.32:0/387059749 >> 172.21.15.32:0/387059749 conn(0x7fad180ccf70 msgr2=0x7fad180cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 -- 172.21.15.32:0/387059749 shutdown_connections 2024-03-06T07:32:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.374+0000 7fad1ce00640 1 -- 172.21.15.32:0/387059749 wait complete. 2024-03-06T07:32:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.376+0000 7fad1ce00640 1 Processor -- start 2024-03-06T07:32:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.376+0000 7fad1ce00640 1 -- start start 2024-03-06T07:32:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad1ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1806c690 0x7fad18142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad1ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 0x7fad18144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad1ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1815c360 0x7fad181f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad1ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad180cea80 con 0x7fad1806ed90 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad1ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1806d840 con 0x7fad1806c690 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad1ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1806f210 con 0x7fad1815c360 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad16a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 0x7fad18144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad16a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 0x7fad18144dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53860/0 (socket says 172.21.15.32:53860) 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad16a00640 1 -- 172.21.15.32:0/1136028529 learned_addr learned my addr 172.21.15.32:0/1136028529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.377+0000 7fad17400640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1806c690 0x7fad18142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad16a00640 1 -- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1815c360 msgr2=0x7fad181f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad17e00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1815c360 0x7fad181f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad16a00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1815c360 0x7fad181f40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad16a00640 1 -- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1806c690 msgr2=0x7fad18142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad16a00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1806c690 0x7fad18142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad16a00640 1 -- 172.21.15.32:0/1136028529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad181456c0 con 0x7fad1806ed90 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad17400640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1806c690 0x7fad18142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad16a00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 0x7fad18144dc0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fad0400c400 tx=0x7fad0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04017070 con 0x7fad1806ed90 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040052d0 con 0x7fad1806ed90 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad04003830 con 0x7fad1806ed90 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad181459a0 con 0x7fad1806ed90 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.378+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad181f38a0 con 0x7fad1806ed90 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.380+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1804f330 con 0x7fad1806ed90 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.380+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad0401a070 con 0x7fad1806ed90 2024-03-06T07:32:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.382+0000 7fad0f400640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facf806b4d0 0x7facf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.382+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad04089590 con 0x7fad1806ed90 2024-03-06T07:32:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.382+0000 7fad17400640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facf806b4d0 0x7facf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.383+0000 7fad17400640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facf806b4d0 0x7facf806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fad080011e0 tx=0x7fad08009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.386+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad04058400 con 0x7fad1806ed90 2024-03-06T07:32:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.627+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad1806caf0 con 0x7fad1806ed90 2024-03-06T07:32:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.627+0000 7fad0f400640 1 -- 172.21.15.32:0/1136028529 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fad04057d00 con 0x7fad1806ed90 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facf806b4d0 msgr2=0x7facf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facf806b4d0 0x7facf806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fad080011e0 tx=0x7fad08009040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 msgr2=0x7fad18144dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 0x7fad18144dc0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fad0400c400 tx=0x7fad0400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 shutdown_connections 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facf806b4d0 0x7facf806d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1815c360 0x7fad181f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad1806ed90 0x7fad18144dc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 --2- 172.21.15.32:0/1136028529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1806c690 0x7fad18142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 >> 172.21.15.32:0/1136028529 conn(0x7fad180ccf70 msgr2=0x7fad1806b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 shutdown_connections 2024-03-06T07:32:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.630+0000 7fad1ce00640 1 -- 172.21.15.32:0/1136028529 wait complete. 2024-03-06T07:32:14.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.868+0000 7f4c53200640 1 Processor -- start 2024-03-06T07:32:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.868+0000 7f4c53200640 1 -- start start 2024-03-06T07:32:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.870+0000 7f4c53200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.870+0000 7f4c53200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c540b0d30 0x7f4c540af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.870+0000 7f4c53200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540b6270 con 0x7f4c540b1280 2024-03-06T07:32:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.870+0000 7f4c53200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540aeb00 con 0x7f4c540b0d30 2024-03-06T07:32:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.870+0000 7f4c53200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540b6510 con 0x7f4c5415df00 2024-03-06T07:32:14.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.870+0000 7f4c52800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c540b0d30 0x7f4c540af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c52800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c540b0d30 0x7f4c540af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53866/0 (socket says 172.21.15.32:53866) 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c52800640 1 -- 172.21.15.32:0/3067862649 learned_addr learned my addr 172.21.15.32:0/3067862649 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c51e00640 1 --2- 172.21.15.32:0/3067862649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c52800640 1 -- 172.21.15.32:0/3067862649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 msgr2=0x7f4c5406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c52800640 1 --2- 172.21.15.32:0/3067862649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c52800640 1 -- 172.21.15.32:0/3067862649 >> v1:172.21.15.39:6789/0 conn(0x7f4c540b1280 legacy=0x7f4c5406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.871+0000 7f4c52800640 1 -- 172.21.15.32:0/3067862649 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540b67f0 con 0x7f4c540b0d30 2024-03-06T07:32:14.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.874+0000 7f4c52800640 1 --2- 172.21.15.32:0/3067862649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b0d30 0x7f4c540af270 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009e50 tx=0x7f4c4c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c3268b4722ef36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.874+0000 7f4c51400640 1 -- 172.21.15.32:0/3067862649 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c02ca70 con 0x7f4c540b0d30 2024-03-06T07:32:14.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.874+0000 7f4c51400640 1 -- 172.21.15.32:0/3067862649 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c02cc10 con 0x7f4c540b0d30 2024-03-06T07:32:14.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.874+0000 7f4c51400640 1 -- 172.21.15.32:0/3067862649 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0343f0 con 0x7f4c540b0d30 2024-03-06T07:32:14.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 -- 172.21.15.32:0/3067862649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b0d30 msgr2=0x7f4c540af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 --2- 172.21.15.32:0/3067862649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b0d30 0x7f4c540af270 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009e50 tx=0x7f4c4c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 -- 172.21.15.32:0/3067862649 shutdown_connections 2024-03-06T07:32:14.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 --2- 172.21.15.32:0/3067862649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b0d30 0x7f4c540af270 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 --2- 172.21.15.32:0/3067862649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 -- 172.21.15.32:0/3067862649 >> 172.21.15.32:0/3067862649 conn(0x7f4c5409aa10 msgr2=0x7f4c5409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 -- 172.21.15.32:0/3067862649 shutdown_connections 2024-03-06T07:32:14.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.875+0000 7f4c53200640 1 -- 172.21.15.32:0/3067862649 wait complete. 2024-03-06T07:32:14.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.877+0000 7f4c53200640 1 Processor -- start 2024-03-06T07:32:14.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.877+0000 7f4c53200640 1 -- start start 2024-03-06T07:32:14.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c53200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c540b0d30 0x7f4c5414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c53200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 0x7f4c540b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c53200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5414df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c53200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415e460 con 0x7f4c540b1280 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c53200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415e2e0 con 0x7f4c540b0d30 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c53200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540af7b0 con 0x7f4c5415df00 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c49400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 0x7f4c540b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c49400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 0x7f4c540b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53876/0 (socket says 172.21.15.32:53876) 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c49400640 1 -- 172.21.15.32:0/1156073575 learned_addr learned my addr 172.21.15.32:0/1156073575 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.878+0000 7f4c51e00640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c540b0d30 0x7f4c5414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c52800640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5414df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c49400640 1 -- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 msgr2=0x7f4c5414df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c49400640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5414df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c49400640 1 -- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c540b0d30 msgr2=0x7f4c5414c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c49400640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c540b0d30 0x7f4c5414c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c49400640 1 -- 172.21.15.32:0/1156073575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5414c6e0 con 0x7f4c540b1280 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c52800640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5414df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c49400640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 0x7f4c540b3d30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c480 tx=0x7f4c3c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0052d0 con 0x7f4c540b1280 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c005470 con 0x7f4c540b1280 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c012870 con 0x7f4c540b1280 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.879+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c540b2c90 con 0x7f4c540b1280 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.880+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c540b3250 con 0x7f4c540b1280 2024-03-06T07:32:14.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.881+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c540b4ec0 con 0x7f4c540b1280 2024-03-06T07:32:14.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.882+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c3c012a10 con 0x7f4c540b1280 2024-03-06T07:32:14.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.882+0000 7f4c4aa00640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3006b4d0 0x7f4c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.883+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c3c089e90 con 0x7f4c540b1280 2024-03-06T07:32:14.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.883+0000 7f4c51e00640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3006b4d0 0x7f4c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.884+0000 7f4c51e00640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3006b4d0 0x7f4c3006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4c54070bb0 tx=0x7f4c44008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:14.887+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c540b4ec0 con 0x7f4c540b1280 2024-03-06T07:32:15.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.203+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c5404f330 con 0x7f4c540b1280 2024-03-06T07:32:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.206+0000 7f4c4aa00640 1 -- 172.21.15.32:0/1156073575 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f4c3c08b090 con 0x7f4c540b1280 2024-03-06T07:32:15.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:15.208 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:42.769932+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:15.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3006b4d0 msgr2=0x7f4c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3006b4d0 0x7f4c3006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4c54070bb0 tx=0x7f4c44008040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 msgr2=0x7f4c540b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 0x7f4c540b3d30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c480 tx=0x7f4c3c00c940 comp rx=0 tx=0).stop 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 shutdown_connections 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c3006b4d0 0x7f4c3006d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4c5415df00 0x7f4c5414df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540b1280 0x7f4c540b3d30 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 --2- 172.21.15.32:0/1156073575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c540b0d30 0x7f4c5414c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 >> 172.21.15.32:0/1156073575 conn(0x7f4c5409aa10 msgr2=0x7f4c540b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 shutdown_connections 2024-03-06T07:32:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.210+0000 7f4c53200640 1 -- 172.21.15.32:0/1156073575 wait complete. 2024-03-06T07:32:15.233 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:15.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.443+0000 7fd5a7200640 1 Processor -- start 2024-03-06T07:32:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.443+0000 7fd5a7200640 1 -- start start 2024-03-06T07:32:15.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd5a80b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5a806e710 0x7fd5a806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a806fb90 con 0x7fd5a80ad880 2024-03-06T07:32:15.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a8147c50 con 0x7fd5a806e710 2024-03-06T07:32:15.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b3b60 con 0x7fd5a8147870 2024-03-06T07:32:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5a80ad880 0x7fd5a80d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50876/0 (socket says 172.21.15.32:50876) 2024-03-06T07:32:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.445+0000 7fd5a6800640 1 -- 172.21.15.32:0/1657791313 learned_addr learned my addr 172.21.15.32:0/1657791313 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5e00640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd5a80b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd59d400640 1 --2- 172.21.15.32:0/1657791313 >> v2:172.21.15.39:3300/0 conn(0x7fd5a806e710 0x7fd5a806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464088025 0 0) 0x7fd5a806fb90 con 0x7fd5a80ad880 2024-03-06T07:32:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd58c003520 con 0x7fd5a80ad880 2024-03-06T07:32:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2716434365 0 0) 0x7fd58c003520 con 0x7fd5a80ad880 2024-03-06T07:32:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 msgr2=0x7fd5a80b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd5a80b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 >> v2:172.21.15.39:3300/0 conn(0x7fd5a806e710 msgr2=0x7fd5a806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 --2- 172.21.15.32:0/1657791313 >> v2:172.21.15.39:3300/0 conn(0x7fd5a806e710 0x7fd5a806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a806ebf0 con 0x7fd5a80ad880 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd5a0001670 con 0x7fd5a80ad880 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 >> v1:172.21.15.32:6789/0 conn(0x7fd5a80ad880 legacy=0x7fd5a80d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.446+0000 7fd5a5400640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd58c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a5400640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd58c003f80 0x7fd58c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a5400640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd58c004d40 0x7fd58c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8147c50 con 0x7fd58c004d40 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b3b60 con 0x7fd58c003f80 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c00b9c0 con 0x7fd5a8147870 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a6800640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd58c004d40 0x7fd58c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd5a5e00640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd58c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.447+0000 7fd59d400640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd58c003f80 0x7fd58c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd5a6800640 1 -- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 msgr2=0x7fd58c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd5a6800640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd58c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd5a6800640 1 -- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd58c003f80 msgr2=0x7fd58c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd5a6800640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd58c003f80 0x7fd58c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd5a6800640 1 -- 172.21.15.32:0/1657791313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c003520 con 0x7fd58c004d40 2024-03-06T07:32:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd5a5e00640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a8147870 0x7fd58c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.448+0000 7fd59d400640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd58c003f80 0x7fd58c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.450+0000 7fd5a6800640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd58c004d40 0x7fd58c0051c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80add30 tx=0x7fd58c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4b0d9f69ac5865 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.450+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0007ca0 con 0x7fd58c004d40 2024-03-06T07:32:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.450+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0030070 con 0x7fd58c004d40 2024-03-06T07:32:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.450+0000 7fd5a5400640 1 -- 172.21.15.32:0/1657791313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0005d10 con 0x7fd58c004d40 2024-03-06T07:32:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a7200640 1 -- 172.21.15.32:0/1657791313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd58c004d40 msgr2=0x7fd58c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1657791313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd58c004d40 0x7fd58c0051c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80add30 tx=0x7fd58c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a6800640 1 -- 172.21.15.32:0/1657791313 reap_dead start 2024-03-06T07:32:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a7200640 1 -- 172.21.15.32:0/1657791313 shutdown_connections 2024-03-06T07:32:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a7200640 1 -- 172.21.15.32:0/1657791313 >> 172.21.15.32:0/1657791313 conn(0x7fd5a806c0c0 msgr2=0x7fd5a806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a7200640 1 -- 172.21.15.32:0/1657791313 shutdown_connections 2024-03-06T07:32:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.451+0000 7fd5a7200640 1 -- 172.21.15.32:0/1657791313 wait complete. 2024-03-06T07:32:15.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.453+0000 7fd5a7200640 1 Processor -- start 2024-03-06T07:32:15.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.453+0000 7fd5a7200640 1 -- start start 2024-03-06T07:32:15.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 0x7fd5a813d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a815dc70 0x7fd5a80b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a813d740 0x7fd5a80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b4370 con 0x7fd5a815d890 2024-03-06T07:32:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b4090 con 0x7fd5a813d740 2024-03-06T07:32:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a806b4f0 con 0x7fd5a815dc70 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.454+0000 7fd5a5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 0x7fd5a813d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 0x7fd5a813d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53898/0 (socket says 172.21.15.32:53898) 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 -- 172.21.15.32:0/1316445414 learned_addr learned my addr 172.21.15.32:0/1316445414 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 -- 172.21.15.32:0/1316445414 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a815dc70 msgr2=0x7fd5a80b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a815dc70 0x7fd5a80b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 -- 172.21.15.32:0/1316445414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a813d740 msgr2=0x7fd5a80b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a813d740 0x7fd5a80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 -- 172.21.15.32:0/1316445414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a813dcc0 con 0x7fd5a815d890 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.455+0000 7fd5a5e00640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 0x7fd5a813d200 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd5980101c0 tx=0x7fd598010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.456+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59801a030 con 0x7fd5a815d890 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.456+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598003990 con 0x7fd5a815d890 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.456+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598003d60 con 0x7fd5a815d890 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.456+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a8141120 con 0x7fd5a815d890 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.456+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a81416e0 con 0x7fd5a815d890 2024-03-06T07:32:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.457+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a804f330 con 0x7fd5a815d890 2024-03-06T07:32:15.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.458+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd59801d070 con 0x7fd5a815d890 2024-03-06T07:32:15.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.459+0000 7fd59ea00640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd58406b4d0 0x7fd58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.459+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd59808b0d0 con 0x7fd5a815d890 2024-03-06T07:32:15.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.459+0000 7fd59d400640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd58406b4d0 0x7fd58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.460+0000 7fd59d400640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd58406b4d0 0x7fd58406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8145d10 tx=0x7fd590008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.463+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd59801de40 con 0x7fd5a815d890 2024-03-06T07:32:15.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.703+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5a80724f0 con 0x7fd5a815d890 2024-03-06T07:32:15.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.704+0000 7fd59ea00640 1 -- 172.21.15.32:0/1316445414 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fd598059f50 con 0x7fd5a815d890 2024-03-06T07:32:15.706 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:15.706 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:32:15.706 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:32:15.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd58406b4d0 msgr2=0x7fd58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd58406b4d0 0x7fd58406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8145d10 tx=0x7fd590008040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 msgr2=0x7fd5a813d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 0x7fd5a813d200 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd5980101c0 tx=0x7fd598010680 comp rx=0 tx=0).stop 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 shutdown_connections 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd58406b4d0 0x7fd58406d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a813d740 0x7fd5a80b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5a815dc70 0x7fd5a80b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 --2- 172.21.15.32:0/1316445414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5a815d890 0x7fd5a813d200 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 >> 172.21.15.32:0/1316445414 conn(0x7fd5a806c0c0 msgr2=0x7fd5a8145df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 shutdown_connections 2024-03-06T07:32:15.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.707+0000 7fd5a7200640 1 -- 172.21.15.32:0/1316445414 wait complete. 2024-03-06T07:32:15.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.952+0000 7fd97da00640 1 Processor -- start 2024-03-06T07:32:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.952+0000 7fd97da00640 1 -- start start 2024-03-06T07:32:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd97da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd97815e200 0x7fd9780b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd97da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97806b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd97da00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97806bec0 con 0x7fd9780afdf0 2024-03-06T07:32:15.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd97da00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9780b0670 con 0x7fd97815e200 2024-03-06T07:32:15.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd97da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9780ae380 con 0x7fd97806afb0 2024-03-06T07:32:15.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd976a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd97815e200 0x7fd9780b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd976a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd97815e200 0x7fd9780b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53906/0 (socket says 172.21.15.32:53906) 2024-03-06T07:32:15.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.954+0000 7fd976a00640 1 -- 172.21.15.32:0/3924295574 learned_addr learned my addr 172.21.15.32:0/3924295574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:15.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.955+0000 7fd977400640 1 --2- 172.21.15.32:0/3924295574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97806b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.955+0000 7fd976a00640 1 -- 172.21.15.32:0/3924295574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 msgr2=0x7fd97806b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.955+0000 7fd976a00640 1 --2- 172.21.15.32:0/3924295574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97806b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.955+0000 7fd976a00640 1 -- 172.21.15.32:0/3924295574 >> v1:172.21.15.32:6789/0 conn(0x7fd9780afdf0 legacy=0x7fd9780af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.955+0000 7fd976a00640 1 -- 172.21.15.32:0/3924295574 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9780ae660 con 0x7fd97815e200 2024-03-06T07:32:15.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.955+0000 7fd975600640 1 -- 172.21.15.32:0/3924295574 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897398772 0 0) 0x7fd97806bec0 con 0x7fd9780afdf0 2024-03-06T07:32:15.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.956+0000 7fd976a00640 1 --2- 172.21.15.32:0/3924295574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9780b3de0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd968009870 tx=0x7fd96802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb40b3fabcd109d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.956+0000 7fd975600640 1 -- 172.21.15.32:0/3924295574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968034070 con 0x7fd97815e200 2024-03-06T07:32:15.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.956+0000 7fd975600640 1 -- 172.21.15.32:0/3924295574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd968030840 con 0x7fd97815e200 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.956+0000 7fd975600640 1 -- 172.21.15.32:0/3924295574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968030c10 con 0x7fd97815e200 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 -- 172.21.15.32:0/3924295574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 msgr2=0x7fd9780b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 --2- 172.21.15.32:0/3924295574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9780b3de0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd968009870 tx=0x7fd96802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 -- 172.21.15.32:0/3924295574 shutdown_connections 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 --2- 172.21.15.32:0/3924295574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 --2- 172.21.15.32:0/3924295574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9780b3de0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 -- 172.21.15.32:0/3924295574 >> 172.21.15.32:0/3924295574 conn(0x7fd9780b2ae0 msgr2=0x7fd9780b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 -- 172.21.15.32:0/3924295574 shutdown_connections 2024-03-06T07:32:15.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.957+0000 7fd97da00640 1 -- 172.21.15.32:0/3924295574 wait complete. 2024-03-06T07:32:15.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.959+0000 7fd97da00640 1 Processor -- start 2024-03-06T07:32:15.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.959+0000 7fd97da00640 1 -- start start 2024-03-06T07:32:15.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd97da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd97da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9780afdf0 0x7fd97815d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd97da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9781f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd97da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97806dbb0 con 0x7fd97815e200 2024-03-06T07:32:15.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd97da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9780aec20 con 0x7fd9780afdf0 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd97da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97806dd30 con 0x7fd97806afb0 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd977400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9781f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd977400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9781f3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53908/0 (socket says 172.21.15.32:53908) 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.960+0000 7fd977400640 1 -- 172.21.15.32:0/2725488151 learned_addr learned my addr 172.21.15.32:0/2725488151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd976a00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd976000640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9780afdf0 0x7fd97815d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd977400640 1 -- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 msgr2=0x7fd97815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd977400640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd977400640 1 -- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9780afdf0 msgr2=0x7fd97815d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd977400640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9780afdf0 0x7fd97815d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd977400640 1 -- 172.21.15.32:0/2725488151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97815b200 con 0x7fd97815e200 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd976000640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9780afdf0 0x7fd97815d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.961+0000 7fd977400640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9781f3f80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd970007e40 tx=0x7fd97000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.962+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970014620 con 0x7fd97815e200 2024-03-06T07:32:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.962+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd970019070 con 0x7fd97815e200 2024-03-06T07:32:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.962+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97815b4e0 con 0x7fd97815e200 2024-03-06T07:32:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.962+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970014a20 con 0x7fd97815e200 2024-03-06T07:32:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.962+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd978155b40 con 0x7fd97815e200 2024-03-06T07:32:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.963+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97815b6b0 con 0x7fd97815e200 2024-03-06T07:32:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.964+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd970014bc0 con 0x7fd97815e200 2024-03-06T07:32:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.966+0000 7fd96ea00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95806b520 0x7fd95806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.966+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd97008aaf0 con 0x7fd97815e200 2024-03-06T07:32:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.966+0000 7fd976a00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95806b520 0x7fd95806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.967+0000 7fd976a00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95806b520 0x7fd95806d9e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd968036000 tx=0x7fd968008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:15.973+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd970059950 con 0x7fd97815e200 2024-03-06T07:32:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.213+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd97804f3a0 con 0x7fd97815e200 2024-03-06T07:32:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.214+0000 7fd96ea00640 1 -- 172.21.15.32:0/2725488151 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fd970059250 con 0x7fd97815e200 2024-03-06T07:32:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:16.215 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:41.207788+0000","modified":"2024-03-06T07:31:44.292199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5907":{"gid":5907,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/436488985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":436488985},{"type":"v1","addr":"172.21.15.32:6837","nonce":436488985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5907,"qdb_cluster":[5907]},"id":4}]} 2024-03-06T07:32:16.215 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-03-06T07:32:16.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.216+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95806b520 msgr2=0x7fd95806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.216+0000 7fd97da00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95806b520 0x7fd95806d9e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd968036000 tx=0x7fd968008030 comp rx=0 tx=0).stop 2024-03-06T07:32:16.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.216+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 msgr2=0x7fd9781f3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.216+0000 7fd97da00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9781f3f80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd970007e40 tx=0x7fd97000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 shutdown_connections 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95806b520 0x7fd95806d9e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97815e200 0x7fd9781f3f80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9780afdf0 0x7fd97815d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 --2- 172.21.15.32:0/2725488151 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd97806afb0 0x7fd97815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 >> 172.21.15.32:0/2725488151 conn(0x7fd9780b2ae0 msgr2=0x7fd978161b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 shutdown_connections 2024-03-06T07:32:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.217+0000 7fd97da00640 1 -- 172.21.15.32:0/2725488151 wait complete. 2024-03-06T07:32:16.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.450+0000 7fc7ffa00640 1 Processor -- start 2024-03-06T07:32:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.450+0000 7fc7ffa00640 1 -- start start 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.451+0000 7fc7ffa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7f8145790 0x7fc7f80d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.451+0000 7fc7ffa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc7f80add50 0x7fc7f81603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.451+0000 7fc7ffa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f806b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.451+0000 7fc7ffa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80d1b70 con 0x7fc7f8145790 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.451+0000 7fc7ffa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f809ef10 con 0x7fc7f80add50 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.451+0000 7fc7ffa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80b5de0 con 0x7fc7f806afb0 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7f8145790 0x7fc7f80d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7f8145790 0x7fc7f80d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53910/0 (socket says 172.21.15.32:53910) 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 -- 172.21.15.32:0/1460114021 learned_addr learned my addr 172.21.15.32:0/1460114021 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fda00640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f806b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7f7e00640 1 --2- 172.21.15.32:0/1460114021 >> v2:172.21.15.39:3300/0 conn(0x7fc7f80add50 0x7fc7f81603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 -- 172.21.15.32:0/1460114021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 msgr2=0x7fc7f806b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f806b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 -- 172.21.15.32:0/1460114021 >> v2:172.21.15.39:3300/0 conn(0x7fc7f80add50 msgr2=0x7fc7f81603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1460114021 >> v2:172.21.15.39:3300/0 conn(0x7fc7f80add50 0x7fc7f81603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fd000640 1 -- 172.21.15.32:0/1460114021 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80b6080 con 0x7fc7f8145790 2024-03-06T07:32:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7fda00640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f806b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.452+0000 7fc7f7e00640 1 --2- 172.21.15.32:0/1460114021 >> v2:172.21.15.39:3300/0 conn(0x7fc7f80add50 0x7fc7f81603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.453+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f80d1630 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806c090 tx=0x7fc7e802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf541c3498ecca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.454+0000 7fc7f7400640 1 -- 172.21.15.32:0/1460114021 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8035020 con 0x7fc7f8145790 2024-03-06T07:32:16.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.454+0000 7fc7f7400640 1 -- 172.21.15.32:0/1460114021 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8031880 con 0x7fc7f8145790 2024-03-06T07:32:16.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.454+0000 7fc7f7400640 1 -- 172.21.15.32:0/1460114021 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8031c50 con 0x7fc7f8145790 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.454+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1460114021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 msgr2=0x7fc7f80d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.454+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f80d1630 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806c090 tx=0x7fc7e802b920 comp rx=0 tx=0).stop 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1460114021 shutdown_connections 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f806b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1460114021 >> v2:172.21.15.39:3300/0 conn(0x7fc7f80add50 0x7fc7f81603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1460114021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f80d1630 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1460114021 >> 172.21.15.32:0/1460114021 conn(0x7fc7f806c2a0 msgr2=0x7fc7f806c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1460114021 shutdown_connections 2024-03-06T07:32:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.455+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1460114021 wait complete. 2024-03-06T07:32:16.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.457+0000 7fc7ffa00640 1 Processor -- start 2024-03-06T07:32:16.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.457+0000 7fc7ffa00640 1 -- start start 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7ffa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7ffa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7f80add50 0x7fc7f809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7ffa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f809af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7ffa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8160930 con 0x7fc7f8145790 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7ffa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f815fed0 con 0x7fc7f80add50 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7ffa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8145b70 con 0x7fc7f806afb0 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7fda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f809af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7fda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f809af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53922/0 (socket says 172.21.15.32:53922) 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7fda00640 1 -- 172.21.15.32:0/1552925506 learned_addr learned my addr 172.21.15.32:0/1552925506 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7f7e00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7f80add50 0x7fc7f809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.458+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fda00640 1 -- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 msgr2=0x7fc7f815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fda00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f815b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fda00640 1 -- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7f80add50 msgr2=0x7fc7f809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fda00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7f80add50 0x7fc7f809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fda00640 1 -- 172.21.15.32:0/1552925506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f809e120 con 0x7fc7f8145790 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7f7e00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7f80add50 0x7fc7f809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7fda00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f809af50 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000cd10 tx=0x7fc7f000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f000f680 con 0x7fc7f8145790 2024-03-06T07:32:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f000f840 con 0x7fc7f8145790 2024-03-06T07:32:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f000fc30 con 0x7fc7f8145790 2024-03-06T07:32:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f809e400 con 0x7fc7f8145790 2024-03-06T07:32:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.459+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f81f4720 con 0x7fc7f8145790 2024-03-06T07:32:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.460+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f806b460 con 0x7fc7f8145790 2024-03-06T07:32:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.461+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7f00213f0 con 0x7fc7f8145790 2024-03-06T07:32:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.462+0000 7fc7f4c00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc7d406b500 0x7fc7d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.462+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7f008aff0 con 0x7fc7f8145790 2024-03-06T07:32:16.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.463+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc7d406b500 0x7fc7d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.463+0000 7fc7fd000640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc7d406b500 0x7fc7d406d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8036000 tx=0x7fc7e8008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.467+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7f806b460 con 0x7fc7f8145790 2024-03-06T07:32:16.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.672+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7f81f4920 con 0x7fc7f8145790 2024-03-06T07:32:16.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.675+0000 7fc7f4c00640 1 -- 172.21.15.32:0/1552925506 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fc7f81f4920 con 0x7fc7f8145790 2024-03-06T07:32:16.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:16.677 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:31:42.769932+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:39.080872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:40.231621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc7d406b500 msgr2=0x7fc7d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc7d406b500 0x7fc7d406d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8036000 tx=0x7fc7e8008030 comp rx=0 tx=0).stop 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 msgr2=0x7fc7f809af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f809af50 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000cd10 tx=0x7fc7f000a650 comp rx=0 tx=0).stop 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 shutdown_connections 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc7d406b500 0x7fc7d406d9c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7f8145790 0x7fc7f809af50 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7f80add50 0x7fc7f809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 --2- 172.21.15.32:0/1552925506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7f806afb0 0x7fc7f815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.678+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 >> 172.21.15.32:0/1552925506 conn(0x7fc7f806c2a0 msgr2=0x7fc7f815e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.679+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 shutdown_connections 2024-03-06T07:32:16.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.679+0000 7fc7ffa00640 1 -- 172.21.15.32:0/1552925506 wait complete. 2024-03-06T07:32:16.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:16.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.919+0000 7fb43f600640 1 Processor -- start 2024-03-06T07:32:16.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.919+0000 7fb43f600640 1 -- start start 2024-03-06T07:32:16.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.920+0000 7fb43f600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb43815e0c0 0x7fb4380d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.921+0000 7fb43f600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380d1160 con 0x7fb43815e4a0 2024-03-06T07:32:16.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.921+0000 7fb43f600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380d1440 con 0x7fb4380d2bc0 2024-03-06T07:32:16.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.921+0000 7fb43f600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4380d0f10 con 0x7fb43815e0c0 2024-03-06T07:32:16.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.921+0000 7fb437e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb43815e4a0 0x7fb4380d24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50884/0 (socket says 172.21.15.32:50884) 2024-03-06T07:32:16.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.921+0000 7fb437e00640 1 -- 172.21.15.32:0/1093498419 learned_addr learned my addr 172.21.15.32:0/1093498419 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:16.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.921+0000 7fb43cc00640 1 --2- 172.21.15.32:0/1093498419 >> v2:172.21.15.39:3300/0 conn(0x7fb43815e0c0 0x7fb4380d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.922+0000 7fb437400640 1 -- 172.21.15.32:0/1093498419 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219889583 0 0) 0x7fb4380d1160 con 0x7fb43815e4a0 2024-03-06T07:32:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.922+0000 7fb437400640 1 -- 172.21.15.32:0/1093498419 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb420003520 con 0x7fb43815e4a0 2024-03-06T07:32:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.922+0000 7fb43cc00640 1 -- 172.21.15.32:0/1093498419 >> v1:172.21.15.39:6789/0 conn(0x7fb4380d2bc0 legacy=0x7fb4380c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.922+0000 7fb43cc00640 1 -- 172.21.15.32:0/1093498419 >> v1:172.21.15.32:6789/0 conn(0x7fb43815e4a0 legacy=0x7fb4380d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.922+0000 7fb437400640 1 -- 172.21.15.32:0/1093498419 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1029235231 0 0) 0x7fb420003520 con 0x7fb43815e4a0 2024-03-06T07:32:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.922+0000 7fb43cc00640 1 -- 172.21.15.32:0/1093498419 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380d1720 con 0x7fb43815e0c0 2024-03-06T07:32:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.923+0000 7fb43cc00640 1 --2- 172.21.15.32:0/1093498419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb43815e0c0 0x7fb4380d1cb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0097a0 tx=0x7fb42c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b72c07ac7ac095e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.924+0000 7fb437400640 1 -- 172.21.15.32:0/1093498419 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424001670 con 0x7fb43815e0c0 2024-03-06T07:32:16.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.924+0000 7fb437400640 1 -- 172.21.15.32:0/1093498419 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c037070 con 0x7fb43815e0c0 2024-03-06T07:32:16.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.924+0000 7fb437400640 1 -- 172.21.15.32:0/1093498419 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c0317b0 con 0x7fb43815e0c0 2024-03-06T07:32:16.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 -- 172.21.15.32:0/1093498419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb43815e0c0 msgr2=0x7fb4380d1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 --2- 172.21.15.32:0/1093498419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb43815e0c0 0x7fb4380d1cb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0097a0 tx=0x7fb42c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 -- 172.21.15.32:0/1093498419 shutdown_connections 2024-03-06T07:32:16.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 --2- 172.21.15.32:0/1093498419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb43815e0c0 0x7fb4380d1cb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 -- 172.21.15.32:0/1093498419 >> 172.21.15.32:0/1093498419 conn(0x7fb4380d3ed0 msgr2=0x7fb4380d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 -- 172.21.15.32:0/1093498419 shutdown_connections 2024-03-06T07:32:16.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.925+0000 7fb43f600640 1 -- 172.21.15.32:0/1093498419 wait complete. 2024-03-06T07:32:16.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.927+0000 7fb43f600640 1 Processor -- start 2024-03-06T07:32:16.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.927+0000 7fb43f600640 1 -- start start 2024-03-06T07:32:16.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4380d2bc0 0x7fb43815c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43f600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb43815e0c0 0x7fb4380cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 0x7fb4380cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43f600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380c7d00 con 0x7fb43815e4a0 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380d21f0 con 0x7fb4380d2bc0 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43f600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380d4b40 con 0x7fb43815e0c0 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.928+0000 7fb43d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 0x7fb4380cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 0x7fb4380cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53924/0 (socket says 172.21.15.32:53924) 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 -- 172.21.15.32:0/1973574398 learned_addr learned my addr 172.21.15.32:0/1973574398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43cc00640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4380d2bc0 0x7fb43815c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb437e00640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb43815e0c0 0x7fb4380cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 -- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb43815e0c0 msgr2=0x7fb4380cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb43815e0c0 0x7fb4380cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 -- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4380d2bc0 msgr2=0x7fb43815c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4380d2bc0 0x7fb43815c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb43d600640 1 -- 172.21.15.32:0/1973574398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4381f41a0 con 0x7fb43815e4a0 2024-03-06T07:32:16.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.929+0000 7fb437e00640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb43815e0c0 0x7fb4380cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:16.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.930+0000 7fb43d600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 0x7fb4380cd6b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb43000c7a0 tx=0x7fb43000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.930+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430003b70 con 0x7fb43815e4a0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.930+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb430003d30 con 0x7fb43815e4a0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.930+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43000aa90 con 0x7fb43815e4a0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.930+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4381f4460 con 0x7fb43815e4a0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.930+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4381f4a20 con 0x7fb43815e4a0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.931+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43804f3a0 con 0x7fb43815e4a0 2024-03-06T07:32:16.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.933+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb43000ac30 con 0x7fb43815e4a0 2024-03-06T07:32:16.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.933+0000 7fb434c00640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb41806b500 0x7fb41806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.934+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb430089fc0 con 0x7fb43815e4a0 2024-03-06T07:32:16.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.934+0000 7fb43cc00640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb41806b500 0x7fb41806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.934+0000 7fb43cc00640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb41806b500 0x7fb41806d9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb42c036000 tx=0x7fb42c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:16.943+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb430058ed0 con 0x7fb43815e4a0 2024-03-06T07:32:17.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.185+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb43808da10 con 0x7fb43815e4a0 2024-03-06T07:32:17.187 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:32:17.186+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:17.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.190+0000 7fb434c00640 1 -- 172.21.15.32:0/1973574398 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb4300587d0 con 0x7fb43815e4a0 2024-03-06T07:32:17.191 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.193+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb41806b500 msgr2=0x7fb41806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.193+0000 7fb43f600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb41806b500 0x7fb41806d9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb42c036000 tx=0x7fb42c03d040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 msgr2=0x7fb4380cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 0x7fb4380cd6b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb43000c7a0 tx=0x7fb43000cc60 comp rx=0 tx=0).stop 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 shutdown_connections 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb41806b500 0x7fb41806d9c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb43815e4a0 0x7fb4380cd6b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb43815e0c0 0x7fb4380cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 --2- 172.21.15.32:0/1973574398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4380d2bc0 0x7fb43815c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 >> 172.21.15.32:0/1973574398 conn(0x7fb4380d3ed0 msgr2=0x7fb4380c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 shutdown_connections 2024-03-06T07:32:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.194+0000 7fb43f600640 1 -- 172.21.15.32:0/1973574398 wait complete. 2024-03-06T07:32:17.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:17.280 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:32:17.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.413+0000 7fe644200640 1 Processor -- start 2024-03-06T07:32:17.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.413+0000 7fe644200640 1 -- start start 2024-03-06T07:32:17.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe644200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe63c071e20 0x7fe63c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe644200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63c072df0 con 0x7fe63c072520 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe644200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63c0cdab0 con 0x7fe63c15bf20 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe644200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63c0b4d70 con 0x7fe63c071e20 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe642200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe63c071e20 0x7fe63c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe640e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe63c072520 0x7fe63c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50894/0 (socket says 172.21.15.32:50894) 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.414+0000 7fe640e00640 1 -- 172.21.15.32:0/3202633080 learned_addr learned my addr 172.21.15.32:0/3202633080 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.415+0000 7fe642200640 1 -- 172.21.15.32:0/3202633080 >> v1:172.21.15.39:6789/0 conn(0x7fe63c15bf20 legacy=0x7fe63c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.415+0000 7fe642200640 1 -- 172.21.15.32:0/3202633080 >> v1:172.21.15.32:6789/0 conn(0x7fe63c072520 legacy=0x7fe63c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.415+0000 7fe642200640 1 -- 172.21.15.32:0/3202633080 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0b5050 con 0x7fe63c071e20 2024-03-06T07:32:17.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.416+0000 7fe642200640 1 --2- 172.21.15.32:0/3202633080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c071e20 0x7fe63c0ccad0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe634009d90 tx=0x7fe63402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73422d002e7461 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.416+0000 7fe63be00640 1 -- 172.21.15.32:0/3202633080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634039070 con 0x7fe63c071e20 2024-03-06T07:32:17.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.416+0000 7fe63be00640 1 -- 172.21.15.32:0/3202633080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63402cc60 con 0x7fe63c071e20 2024-03-06T07:32:17.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.416+0000 7fe63be00640 1 -- 172.21.15.32:0/3202633080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634034660 con 0x7fe63c071e20 2024-03-06T07:32:17.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 -- 172.21.15.32:0/3202633080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c071e20 msgr2=0x7fe63c0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 --2- 172.21.15.32:0/3202633080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c071e20 0x7fe63c0ccad0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe634009d90 tx=0x7fe63402beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 -- 172.21.15.32:0/3202633080 shutdown_connections 2024-03-06T07:32:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 --2- 172.21.15.32:0/3202633080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c071e20 0x7fe63c0ccad0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 -- 172.21.15.32:0/3202633080 >> 172.21.15.32:0/3202633080 conn(0x7fe63c06b3d0 msgr2=0x7fe63c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 -- 172.21.15.32:0/3202633080 shutdown_connections 2024-03-06T07:32:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.417+0000 7fe644200640 1 -- 172.21.15.32:0/3202633080 wait complete. 2024-03-06T07:32:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.420+0000 7fe644200640 1 Processor -- start 2024-03-06T07:32:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.420+0000 7fe644200640 1 -- start start 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe644200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63c071e20 0x7fe63c145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe644200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe63c072520 0x7fe63c0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe644200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 0x7fe63c0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe644200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c0ad450 con 0x7fe63c15bf20 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe644200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c0b4990 con 0x7fe63c071e20 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe644200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c145f50 con 0x7fe63c072520 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe642200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 0x7fe63c0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe642200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 0x7fe63c0ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53964/0 (socket says 172.21.15.32:53964) 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.421+0000 7fe642200640 1 -- 172.21.15.32:0/40921721 learned_addr learned my addr 172.21.15.32:0/40921721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe641800640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63c071e20 0x7fe63c145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe640e00640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe63c072520 0x7fe63c0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe642200640 1 -- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe63c072520 msgr2=0x7fe63c0ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe642200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe63c072520 0x7fe63c0ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe642200640 1 -- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63c071e20 msgr2=0x7fe63c145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe642200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63c071e20 0x7fe63c145a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe642200640 1 -- 172.21.15.32:0/40921721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c1568f0 con 0x7fe63c15bf20 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.422+0000 7fe642200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 0x7fe63c0ca900 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe634009d90 tx=0x7fe634004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.423+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634039050 con 0x7fe63c15bf20 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.423+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63403a040 con 0x7fe63c15bf20 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.423+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63403e610 con 0x7fe63c15bf20 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.423+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63c09aab0 con 0x7fe63c15bf20 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.423+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63c09b070 con 0x7fe63c15bf20 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.424+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63c04f330 con 0x7fe63c15bf20 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.425+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe634047020 con 0x7fe63c15bf20 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.426+0000 7fe639600640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe61c06b4d0 0x7fe61c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.426+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe6340ac670 con 0x7fe63c15bf20 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.426+0000 7fe641800640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe61c06b4d0 0x7fe61c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.427+0000 7fe641800640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe61c06b4d0 0x7fe61c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe630004ea0 tx=0x7fe630009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.431+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe63404e020 con 0x7fe63c15bf20 2024-03-06T07:32:17.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:17.673+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe63c15f720 con 0x7fe63c15bf20 2024-03-06T07:32:18.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.193+0000 7fe639600640 1 -- 172.21.15.32:0/40921721 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe63407b540 con 0x7fe63c15bf20 2024-03-06T07:32:18.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe61c06b4d0 msgr2=0x7fe61c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe61c06b4d0 0x7fe61c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe630004ea0 tx=0x7fe630009040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 msgr2=0x7fe63c0ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 0x7fe63c0ca900 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe634009d90 tx=0x7fe634004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 shutdown_connections 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe61c06b4d0 0x7fe61c06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63c15bf20 0x7fe63c0ca900 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe63c072520 0x7fe63c0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 --2- 172.21.15.32:0/40921721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63c071e20 0x7fe63c145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.197+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 >> 172.21.15.32:0/40921721 conn(0x7fe63c06b3d0 msgr2=0x7fe63c0ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.198+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 shutdown_connections 2024-03-06T07:32:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.198+0000 7fe644200640 1 -- 172.21.15.32:0/40921721 wait complete. 2024-03-06T07:32:18.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.440+0000 7fba05a00640 1 Processor -- start 2024-03-06T07:32:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.440+0000 7fba05a00640 1 -- start start 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fba00159730 0x7fba0015cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba000ceea0 0x7fba00072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba00159b10 con 0x7fba000ccc20 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba00071690 con 0x7fba00159730 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba00071970 con 0x7fba000ceea0 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fba000ccc20 0x7fba000cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59680/0 (socket says 172.21.15.32:59680) 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fba05000640 1 -- 172.21.15.32:0/4137541446 learned_addr learned my addr 172.21.15.32:0/4137541446 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:18.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fb9ffe00640 1 --2- 172.21.15.32:0/4137541446 >> v2:172.21.15.32:3300/0 conn(0x7fba00159730 0x7fba0015cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.443+0000 7fb9ffe00640 1 -- 172.21.15.32:0/4137541446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba000ceea0 msgr2=0x7fba00072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.442+0000 7fb9ff400640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba000ceea0 0x7fba00072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.443+0000 7fb9ffe00640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba000ceea0 0x7fba00072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.443+0000 7fb9ffe00640 1 -- 172.21.15.32:0/4137541446 >> v1:172.21.15.32:6789/0 conn(0x7fba000ccc20 legacy=0x7fba000cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.443+0000 7fb9fea00640 1 -- 172.21.15.32:0/4137541446 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647772808 0 0) 0x7fba00159b10 con 0x7fba000ccc20 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.443+0000 7fb9ffe00640 1 -- 172.21.15.32:0/4137541446 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00071c50 con 0x7fba00159730 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.443+0000 7fb9ff400640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba000ceea0 0x7fba00072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.444+0000 7fb9ffe00640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00159730 0x7fba0015cc20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00097a0 tx=0x7fb9f002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c8270da1d9eaab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.444+0000 7fb9fea00640 1 -- 172.21.15.32:0/4137541446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0035020 con 0x7fba00159730 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.444+0000 7fb9fea00640 1 -- 172.21.15.32:0/4137541446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f0031810 con 0x7fba00159730 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.444+0000 7fb9fea00640 1 -- 172.21.15.32:0/4137541446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0031be0 con 0x7fba00159730 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.444+0000 7fba05a00640 1 -- 172.21.15.32:0/4137541446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00159730 msgr2=0x7fba0015cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.444+0000 7fba05a00640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00159730 0x7fba0015cc20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00097a0 tx=0x7fb9f002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.445+0000 7fba05a00640 1 -- 172.21.15.32:0/4137541446 shutdown_connections 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.445+0000 7fba05a00640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba000ceea0 0x7fba00072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.445+0000 7fba05a00640 1 --2- 172.21.15.32:0/4137541446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba00159730 0x7fba0015cc20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.445+0000 7fba05a00640 1 -- 172.21.15.32:0/4137541446 >> 172.21.15.32:0/4137541446 conn(0x7fba0006afb0 msgr2=0x7fba0006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.445+0000 7fba05a00640 1 -- 172.21.15.32:0/4137541446 shutdown_connections 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.445+0000 7fba05a00640 1 -- 172.21.15.32:0/4137541446 wait complete. 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.446+0000 7fba05a00640 1 Processor -- start 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.447+0000 7fba05a00640 1 -- start start 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.447+0000 7fba05a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 0x7fba00157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.447+0000 7fba05a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba000ceea0 0x7fba000b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.447+0000 7fba05a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba00159730 0x7fba000b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fba05a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000cf300 con 0x7fba000ccc20 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fba05a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000cf6d0 con 0x7fba000ceea0 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fba05a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba000cf480 con 0x7fba00159730 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 0x7fba00157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 0x7fba00157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46492/0 (socket says 172.21.15.32:46492) 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 -- 172.21.15.32:0/3720951337 learned_addr learned my addr 172.21.15.32:0/3720951337 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 -- 172.21.15.32:0/3720951337 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba00159730 msgr2=0x7fba000b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba00159730 0x7fba000b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 -- 172.21.15.32:0/3720951337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba000ceea0 msgr2=0x7fba000b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba000ceea0 0x7fba000b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 -- 172.21.15.32:0/3720951337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba001f7880 con 0x7fba000ccc20 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.448+0000 7fb9ffe00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 0x7fba00157ae0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00098c0 tx=0x7fb9f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.449+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0040070 con 0x7fba000ccc20 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.449+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f003e070 con 0x7fba000ccc20 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.449+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0035040 con 0x7fba000ccc20 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.449+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba001f7b60 con 0x7fba000ccc20 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.449+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba001f8120 con 0x7fba000ccc20 2024-03-06T07:32:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.450+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0004f330 con 0x7fba000ccc20 2024-03-06T07:32:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.451+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9f0038020 con 0x7fba000ccc20 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.452+0000 7fb9f7e00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9e006b4d0 0x7fb9e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.452+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9f00abc30 con 0x7fba000ccc20 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.452+0000 7fb9ff400640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9e006b4d0 0x7fb9e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.453+0000 7fb9ff400640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9e006b4d0 0x7fb9e006d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec006fd0 tx=0x7fb9ec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.456+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9f007aa40 con 0x7fba000ccc20 2024-03-06T07:32:18.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:18.652+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fba00072d40 con 0x7fba000ccc20 2024-03-06T07:32:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.202+0000 7fb9f7e00640 1 -- 172.21.15.32:0/3720951337 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb9f007a340 con 0x7fba000ccc20 2024-03-06T07:32:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:19.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9e006b4d0 msgr2=0x7fb9e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9e006b4d0 0x7fb9e006d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec006fd0 tx=0x7fb9ec008040 comp rx=0 tx=0).stop 2024-03-06T07:32:19.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 msgr2=0x7fba00157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 0x7fba00157ae0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00098c0 tx=0x7fb9f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 shutdown_connections 2024-03-06T07:32:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9e006b4d0 0x7fb9e006d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fba00159730 0x7fba000b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba000ceea0 0x7fba000b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 --2- 172.21.15.32:0/3720951337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba000ccc20 0x7fba00157ae0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.206+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 >> 172.21.15.32:0/3720951337 conn(0x7fba0006afb0 msgr2=0x7fba000725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.207+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 shutdown_connections 2024-03-06T07:32:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.207+0000 7fba05a00640 1 -- 172.21.15.32:0/3720951337 wait complete. 2024-03-06T07:32:19.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:19.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:19.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.452+0000 7fea3d200640 1 Processor -- start 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.452+0000 7fea3d200640 1 -- start start 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.453+0000 7fea3d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea380d0d90 0x7fea380ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.453+0000 7fea3d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea380b4900 0x7fea380cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.453+0000 7fea3d200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380d23c0 con 0x7fea3815e1f0 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.453+0000 7fea3d200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea380d4c30 con 0x7fea380b4900 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.453+0000 7fea3d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea380d26a0 con 0x7fea380d0d90 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea380b4900 0x7fea380cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea380b4900 0x7fea380cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46496/0 (socket says 172.21.15.32:46496) 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 -- 172.21.15.32:0/2598725910 learned_addr learned my addr 172.21.15.32:0/2598725910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea36a00640 1 --2- 172.21.15.32:0/2598725910 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea380d0d90 0x7fea380ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 -- 172.21.15.32:0/2598725910 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea380d0d90 msgr2=0x7fea380ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 --2- 172.21.15.32:0/2598725910 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea380d0d90 0x7fea380ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 -- 172.21.15.32:0/2598725910 >> v1:172.21.15.39:6789/0 conn(0x7fea3815e1f0 legacy=0x7fea380d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.454+0000 7fea37e00640 1 -- 172.21.15.32:0/2598725910 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea380d2980 con 0x7fea380b4900 2024-03-06T07:32:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.455+0000 7fea37e00640 1 --2- 172.21.15.32:0/2598725910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cee00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fea30009e50 tx=0x7fea3002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c0ae56daeb5b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.456+0000 7fea36000640 1 -- 172.21.15.32:0/2598725910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3002c9e0 con 0x7fea380b4900 2024-03-06T07:32:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.456+0000 7fea36000640 1 -- 172.21.15.32:0/2598725910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3002cb80 con 0x7fea380b4900 2024-03-06T07:32:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.456+0000 7fea36000640 1 -- 172.21.15.32:0/2598725910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea300343f0 con 0x7fea380b4900 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.456+0000 7fea3d200640 1 -- 172.21.15.32:0/2598725910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 msgr2=0x7fea380cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.456+0000 7fea3d200640 1 --2- 172.21.15.32:0/2598725910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cee00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fea30009e50 tx=0x7fea3002be40 comp rx=0 tx=0).stop 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.457+0000 7fea3d200640 1 -- 172.21.15.32:0/2598725910 shutdown_connections 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.457+0000 7fea3d200640 1 --2- 172.21.15.32:0/2598725910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cee00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.457+0000 7fea3d200640 1 --2- 172.21.15.32:0/2598725910 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea380d0d90 0x7fea380ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.457+0000 7fea3d200640 1 -- 172.21.15.32:0/2598725910 >> 172.21.15.32:0/2598725910 conn(0x7fea3809d000 msgr2=0x7fea380d3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.457+0000 7fea3d200640 1 -- 172.21.15.32:0/2598725910 shutdown_connections 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.457+0000 7fea3d200640 1 -- 172.21.15.32:0/2598725910 wait complete. 2024-03-06T07:32:19.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.458+0000 7fea3d200640 1 Processor -- start 2024-03-06T07:32:19.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.459+0000 7fea3d200640 1 -- start start 2024-03-06T07:32:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea3d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea3d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea380d0d90 0x7fea380b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea3d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3815e1f0 0x7fea38154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea3d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea380d4660 con 0x7fea380b4900 2024-03-06T07:32:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea3d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea380b2da0 con 0x7fea380d0d90 2024-03-06T07:32:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea3d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea380d47e0 con 0x7fea3815e1f0 2024-03-06T07:32:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea37400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea37400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46512/0 (socket says 172.21.15.32:46512) 2024-03-06T07:32:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea37400640 1 -- 172.21.15.32:0/2551320293 learned_addr learned my addr 172.21.15.32:0/2551320293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea36a00640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea380d0d90 0x7fea380b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea37400640 1 -- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3815e1f0 msgr2=0x7fea38154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.460+0000 7fea37e00640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3815e1f0 0x7fea38154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea37400640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3815e1f0 0x7fea38154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea37400640 1 -- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea380d0d90 msgr2=0x7fea380b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea37400640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea380d0d90 0x7fea380b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea37400640 1 -- 172.21.15.32:0/2551320293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38154560 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea37400640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cdeb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c460 tx=0x7fea2800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28016020 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28012870 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea380c7a30 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28012c60 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.461+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea380c7ff0 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.463+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3804f330 con 0x7fea380b4900 2024-03-06T07:32:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.463+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea28019020 con 0x7fea380b4900 2024-03-06T07:32:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.464+0000 7fea2f400640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.464+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fea28089f00 con 0x7fea380b4900 2024-03-06T07:32:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.465+0000 7fea36a00640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.465+0000 7fea36a00640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fea20006fd0 tx=0x7fea20008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.470+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea28020030 con 0x7fea380b4900 2024-03-06T07:32:19.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:19.681+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea380d0250 con 0x7fea380b4900 2024-03-06T07:32:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.206+0000 7fea2f400640 1 -- 172.21.15.32:0/2551320293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) v1 ==== 169+0+0 (secure 0 0 0) 0x7fea280182c0 con 0x7fea380b4900 2024-03-06T07:32:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 msgr2=0x7fea1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fea20006fd0 tx=0x7fea20008040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 msgr2=0x7fea380cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cdeb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c460 tx=0x7fea2800c920 comp rx=0 tx=0).stop 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 shutdown_connections 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea1406b4d0 0x7fea1406d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea3815e1f0 0x7fea38154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea380d0d90 0x7fea380b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 --2- 172.21.15.32:0/2551320293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea380b4900 0x7fea380cdeb0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.210+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 >> 172.21.15.32:0/2551320293 conn(0x7fea3809d000 msgr2=0x7fea3806d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.211+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 shutdown_connections 2024-03-06T07:32:20.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.212+0000 7fea3d200640 1 -- 172.21.15.32:0/2551320293 wait complete. 2024-03-06T07:32:20.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:20.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.455+0000 7f11eb200640 1 Processor -- start 2024-03-06T07:32:20.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.455+0000 7f11eb200640 1 -- start start 2024-03-06T07:32:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11eb200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f11e415cc90 0x7f11e40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11eb200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40b3fe0 con 0x7f11e40b2080 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11eb200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e415fe40 con 0x7f11e415cc90 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4160120 con 0x7f11e4145430 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11e9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f11e415cc90 0x7f11e40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11ea800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f11e40b2080 0x7f11e406c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59686/0 (socket says 172.21.15.32:59686) 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11e9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f11e415cc90 0x7f11e40b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46528/0 (socket says 172.21.15.32:46528) 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11ea800640 1 -- 172.21.15.32:0/3873488416 learned_addr learned my addr 172.21.15.32:0/3873488416 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.457+0000 7f11e9e00640 1 --2- 172.21.15.32:0/3873488416 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.458+0000 7f11e9400640 1 -- 172.21.15.32:0/3873488416 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 msgr2=0x7f11e415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.458+0000 7f11e9400640 1 --2- 172.21.15.32:0/3873488416 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.458+0000 7f11e9400640 1 -- 172.21.15.32:0/3873488416 >> v1:172.21.15.32:6789/0 conn(0x7f11e40b2080 legacy=0x7f11e406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.458+0000 7f11e9400640 1 -- 172.21.15.32:0/3873488416 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e4160400 con 0x7f11e415cc90 2024-03-06T07:32:20.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.459+0000 7f11e9400640 1 --2- 172.21.15.32:0/3873488416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e415cc90 0x7f11e40b1b40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f11d40097a0 tx=0x7f11d402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94db4fbac134214a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.459+0000 7f11e8a00640 1 -- 172.21.15.32:0/3873488416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4035020 con 0x7f11e415cc90 2024-03-06T07:32:20.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.459+0000 7f11e8a00640 1 -- 172.21.15.32:0/3873488416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d4031810 con 0x7f11e415cc90 2024-03-06T07:32:20.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.459+0000 7f11e8a00640 1 -- 172.21.15.32:0/3873488416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4031be0 con 0x7f11e415cc90 2024-03-06T07:32:20.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 -- 172.21.15.32:0/3873488416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e415cc90 msgr2=0x7f11e40b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 --2- 172.21.15.32:0/3873488416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e415cc90 0x7f11e40b1b40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f11d40097a0 tx=0x7f11d402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 -- 172.21.15.32:0/3873488416 shutdown_connections 2024-03-06T07:32:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 --2- 172.21.15.32:0/3873488416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e415cc90 0x7f11e40b1b40 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 --2- 172.21.15.32:0/3873488416 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 -- 172.21.15.32:0/3873488416 >> 172.21.15.32:0/3873488416 conn(0x7f11e406b3a0 msgr2=0x7f11e406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 -- 172.21.15.32:0/3873488416 shutdown_connections 2024-03-06T07:32:20.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.460+0000 7f11eb200640 1 -- 172.21.15.32:0/3873488416 wait complete. 2024-03-06T07:32:20.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.462+0000 7f11eb200640 1 Processor -- start 2024-03-06T07:32:20.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.462+0000 7f11eb200640 1 -- start start 2024-03-06T07:32:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 0x7f11e41513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e413a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11eb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11e415cc90 0x7f11e41397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11eb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4145810 con 0x7f11e40b2080 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11eb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e415d0d0 con 0x7f11e415cc90 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4145990 con 0x7f11e4145430 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 0x7f11e41513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 0x7f11e41513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46544/0 (socket says 172.21.15.32:46544) 2024-03-06T07:32:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 -- 172.21.15.32:0/4170536913 learned_addr learned my addr 172.21.15.32:0/4170536913 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 -- 172.21.15.32:0/4170536913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 msgr2=0x7f11e413a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e413a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 -- 172.21.15.32:0/4170536913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11e415cc90 msgr2=0x7f11e41397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11e415cc90 0x7f11e41397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.463+0000 7f11e9e00640 1 -- 172.21.15.32:0/4170536913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e413a6f0 con 0x7f11e40b2080 2024-03-06T07:32:20.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.464+0000 7f11e9e00640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 0x7f11e41513c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f11d800d800 tx=0x7f11d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.464+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d800b6c0 con 0x7f11e40b2080 2024-03-06T07:32:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.464+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8019070 con 0x7f11e40b2080 2024-03-06T07:32:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.464+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d800ba80 con 0x7f11e40b2080 2024-03-06T07:32:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.464+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e41428e0 con 0x7f11e40b2080 2024-03-06T07:32:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.464+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e4142ea0 con 0x7f11e40b2080 2024-03-06T07:32:20.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.465+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e404f330 con 0x7f11e40b2080 2024-03-06T07:32:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.466+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11d800bc20 con 0x7f11e40b2080 2024-03-06T07:32:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.467+0000 7f11de000640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f11c806b4d0 0x7f11c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.467+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f11d808a480 con 0x7f11e40b2080 2024-03-06T07:32:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.468+0000 7f11e9400640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f11c806b4d0 0x7f11c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.468+0000 7f11e9400640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f11c806b4d0 0x7f11c806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f11d4036000 tx=0x7f11d4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.472+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11d8059560 con 0x7f11e40b2080 2024-03-06T07:32:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.670+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f11e40ce240 con 0x7f11e40b2080 2024-03-06T07:32:20.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.678+0000 7f11de000640 1 -- 172.21.15.32:0/4170536913 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f11d8058e60 con 0x7f11e40b2080 2024-03-06T07:32:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f11c806b4d0 msgr2=0x7f11c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f11c806b4d0 0x7f11c806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f11d4036000 tx=0x7f11d4030000 comp rx=0 tx=0).stop 2024-03-06T07:32:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 msgr2=0x7f11e41513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 0x7f11e41513c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f11d800d800 tx=0x7f11d800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 shutdown_connections 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f11c806b4d0 0x7f11c806d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11e415cc90 0x7f11e41397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11e4145430 0x7f11e413a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 --2- 172.21.15.32:0/4170536913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e40b2080 0x7f11e41513c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 >> 172.21.15.32:0/4170536913 conn(0x7f11e406b3a0 msgr2=0x7f11e4144710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 shutdown_connections 2024-03-06T07:32:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.681+0000 7f11eb200640 1 -- 172.21.15.32:0/4170536913 wait complete. 2024-03-06T07:32:20.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:32:20.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.921+0000 7fee03400640 1 Processor -- start 2024-03-06T07:32:20.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.922+0000 7fee03400640 1 -- start start 2024-03-06T07:32:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.923+0000 7fee03400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fedfc0d1a80 0x7fedfc0d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.923+0000 7fee03400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedfc0cf670 con 0x7fedfc1615b0 2024-03-06T07:32:20.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.923+0000 7fee03400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedfc0d1ff0 con 0x7fedfc0d2810 2024-03-06T07:32:20.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.923+0000 7fee03400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedfc161990 con 0x7fedfc0d1a80 2024-03-06T07:32:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.924+0000 7fee00a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fedfc1615b0 0x7fedfc0d47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59702/0 (socket says 172.21.15.32:59702) 2024-03-06T07:32:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.924+0000 7fee00a00640 1 -- 172.21.15.32:0/2783103734 learned_addr learned my addr 172.21.15.32:0/2783103734 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.924+0000 7fedfbe00640 1 --2- 172.21.15.32:0/2783103734 >> v2:172.21.15.39:3300/0 conn(0x7fedfc0d1a80 0x7fedfc0d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.924+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770865134 0 0) 0x7fedfc0cf670 con 0x7fedfc1615b0 2024-03-06T07:32:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.924+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede0003520 con 0x7fedfc1615b0 2024-03-06T07:32:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4216568925 0 0) 0x7fede0003520 con 0x7fedfc1615b0 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 >> v2:172.21.15.39:3300/0 conn(0x7fedfc0d1a80 msgr2=0x7fedfc0d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 --2- 172.21.15.32:0/2783103734 >> v2:172.21.15.39:3300/0 conn(0x7fedfc0d1a80 0x7fedfc0d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 >> v1:172.21.15.39:6789/0 conn(0x7fedfc0d2810 legacy=0x7fedfc0cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc0cf910 con 0x7fedfc1615b0 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fedec001670 con 0x7fedfc1615b0 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 >> v1:172.21.15.32:6789/0 conn(0x7fedfc1615b0 legacy=0x7fedfc0d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede00039b0 0x7fede0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fede0004800 0x7fede0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fede000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc161990 con 0x7fede00051c0 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede000c040 con 0x7fede00039b0 2024-03-06T07:32:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede000c240 con 0x7fede0004800 2024-03-06T07:32:20.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.925+0000 7fee01400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fede000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fedfbe00640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fede0004800 0x7fede0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fee00a00640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede00039b0 0x7fede0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fee01400640 1 -- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fede0004800 msgr2=0x7fede0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fee01400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fede0004800 0x7fede0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fee01400640 1 -- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede00039b0 msgr2=0x7fede0003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fee01400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede00039b0 0x7fede0003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fee01400640 1 -- 172.21.15.32:0/2783103734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede0003520 con 0x7fede00051c0 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.926+0000 7fedfbe00640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fede0004800 0x7fede0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.927+0000 7fee01400640 1 -- 172.21.15.32:0/2783103734 reap_dead start 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.927+0000 7fee01400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fede000b9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fedf40044b0 tx=0x7fede0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb813da455ead32b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.927+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf4034070 con 0x7fede00051c0 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.927+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf4006d70 con 0x7fede00051c0 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.927+0000 7fedfb400640 1 -- 172.21.15.32:0/2783103734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf4004080 con 0x7fede00051c0 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 -- 172.21.15.32:0/2783103734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 msgr2=0x7fede000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fede000b9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fedf40044b0 tx=0x7fede0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 -- 172.21.15.32:0/2783103734 shutdown_connections 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 --2- 172.21.15.32:0/2783103734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fede000b9c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 -- 172.21.15.32:0/2783103734 >> 172.21.15.32:0/2783103734 conn(0x7fedfc09d180 msgr2=0x7fedfc0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 -- 172.21.15.32:0/2783103734 shutdown_connections 2024-03-06T07:32:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.928+0000 7fee03400640 1 -- 172.21.15.32:0/2783103734 wait complete. 2024-03-06T07:32:20.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.930+0000 7fee03400640 1 Processor -- start 2024-03-06T07:32:20.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.930+0000 7fee03400640 1 -- start start 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee03400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fedfc0c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee03400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedfc15df50 0x7fedfc0ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee03400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedfc06baf0 0x7fedfc06bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee03400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0d2e20 con 0x7fede00051c0 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee03400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0d2ca0 con 0x7fedfc06baf0 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee03400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0d22d0 con 0x7fedfc15df50 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fedfc0c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fedfc0c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46556/0 (socket says 172.21.15.32:46556) 2024-03-06T07:32:20.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 -- 172.21.15.32:0/2881993488 learned_addr learned my addr 172.21.15.32:0/2881993488 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fedfbe00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedfc15df50 0x7fedfc0ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee01400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedfc06baf0 0x7fedfc06bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 -- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedfc15df50 msgr2=0x7fedfc0ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedfc15df50 0x7fedfc0ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 -- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedfc06baf0 msgr2=0x7fedfc06bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedfc06baf0 0x7fedfc06bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.931+0000 7fee00a00640 1 -- 172.21.15.32:0/2881993488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc0c9c20 con 0x7fede00051c0 2024-03-06T07:32:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fee01400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedfc06baf0 0x7fedfc06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fee00a00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fedfc0c96e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fedec00e730 tx=0x7fedec00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00ee40 con 0x7fede00051c0 2024-03-06T07:32:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec018070 con 0x7fede00051c0 2024-03-06T07:32:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00cab0 con 0x7fede00051c0 2024-03-06T07:32:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc0c9ea0 con 0x7fede00051c0 2024-03-06T07:32:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.932+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc0b4170 con 0x7fede00051c0 2024-03-06T07:32:20.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.934+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc04f330 con 0x7fede00051c0 2024-03-06T07:32:20.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.934+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedec00cc50 con 0x7fede00051c0 2024-03-06T07:32:20.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.935+0000 7fedf8c00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feddc06b520 0x7feddc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.935+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fedec089c80 con 0x7fede00051c0 2024-03-06T07:32:20.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.936+0000 7fedfbe00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feddc06b520 0x7feddc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.936+0000 7fedfbe00640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feddc06b520 0x7feddc06d9e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fedfc06c690 tx=0x7fede8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:20.940+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedfc0c9c20 con 0x7fede00051c0 2024-03-06T07:32:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.141+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7fedfc0d02c0 con 0x7fede00051c0 2024-03-06T07:32:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.148+0000 7fedf8c00640 1 -- 172.21.15.32:0/2881993488 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7fedec058e70 con 0x7fede00051c0 2024-03-06T07:32:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feddc06b520 msgr2=0x7feddc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feddc06b520 0x7feddc06d9e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fedfc06c690 tx=0x7fede8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 msgr2=0x7fedfc0c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fedfc0c96e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fedec00e730 tx=0x7fedec00ebf0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 shutdown_connections 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feddc06b520 0x7feddc06d9e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedfc06baf0 0x7fedfc06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedfc15df50 0x7fedfc0ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 --2- 172.21.15.32:0/2881993488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fede00051c0 0x7fedfc0c96e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 >> 172.21.15.32:0/2881993488 conn(0x7fedfc09d180 msgr2=0x7fedfc06cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 shutdown_connections 2024-03-06T07:32:21.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.151+0000 7fee03400640 1 -- 172.21.15.32:0/2881993488 wait complete. 2024-03-06T07:32:21.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:32:21.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.396+0000 7f3a78a00640 1 Processor -- start 2024-03-06T07:32:21.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.396+0000 7f3a78a00640 1 -- start start 2024-03-06T07:32:21.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.398+0000 7f3a78a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3a74145430 0x7f3a7415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.398+0000 7f3a78a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a7415cc90 0x7f3a740b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.398+0000 7f3a78a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a740b3fe0 con 0x7f3a740b2080 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.398+0000 7f3a78a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a740b24d0 con 0x7f3a7415cc90 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.398+0000 7f3a78a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7415fe40 con 0x7f3a74145430 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.398+0000 7f3a6be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a7415cc90 0x7f3a740b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a73200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3a740b2080 0x7f3a7406c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59718/0 (socket says 172.21.15.32:59718) 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a6be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a7415cc90 0x7f3a740b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46560/0 (socket says 172.21.15.32:46560) 2024-03-06T07:32:21.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a73200640 1 -- 172.21.15.32:0/1826722432 learned_addr learned my addr 172.21.15.32:0/1826722432 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:21.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a72800640 1 --2- 172.21.15.32:0/1826722432 >> v2:172.21.15.39:3300/0 conn(0x7f3a74145430 0x7f3a7415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a6be00640 1 -- 172.21.15.32:0/1826722432 >> v2:172.21.15.39:3300/0 conn(0x7f3a74145430 msgr2=0x7f3a7415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a6be00640 1 --2- 172.21.15.32:0/1826722432 >> v2:172.21.15.39:3300/0 conn(0x7f3a74145430 0x7f3a7415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a6be00640 1 -- 172.21.15.32:0/1826722432 >> v1:172.21.15.32:6789/0 conn(0x7f3a740b2080 legacy=0x7f3a7406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.399+0000 7f3a6be00640 1 -- 172.21.15.32:0/1826722432 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74160120 con 0x7f3a7415cc90 2024-03-06T07:32:21.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.400+0000 7f3a6be00640 1 --2- 172.21.15.32:0/1826722432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7415cc90 0x7f3a740b1b40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3a740b2670 tx=0x7f3a5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389a2429bd10af80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.400+0000 7f3a71e00640 1 -- 172.21.15.32:0/1826722432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c035020 con 0x7f3a7415cc90 2024-03-06T07:32:21.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a71e00640 1 -- 172.21.15.32:0/1826722432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c031830 con 0x7f3a7415cc90 2024-03-06T07:32:21.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a71e00640 1 -- 172.21.15.32:0/1826722432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c031c00 con 0x7f3a7415cc90 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 -- 172.21.15.32:0/1826722432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7415cc90 msgr2=0x7f3a740b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 --2- 172.21.15.32:0/1826722432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7415cc90 0x7f3a740b1b40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3a740b2670 tx=0x7f3a5c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 -- 172.21.15.32:0/1826722432 shutdown_connections 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 --2- 172.21.15.32:0/1826722432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7415cc90 0x7f3a740b1b40 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 --2- 172.21.15.32:0/1826722432 >> v2:172.21.15.39:3300/0 conn(0x7f3a74145430 0x7f3a7415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 -- 172.21.15.32:0/1826722432 >> 172.21.15.32:0/1826722432 conn(0x7f3a7406b3a0 msgr2=0x7f3a7406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 -- 172.21.15.32:0/1826722432 shutdown_connections 2024-03-06T07:32:21.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.401+0000 7f3a78a00640 1 -- 172.21.15.32:0/1826722432 wait complete. 2024-03-06T07:32:21.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.403+0000 7f3a78a00640 1 Processor -- start 2024-03-06T07:32:21.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.404+0000 7f3a78a00640 1 -- start start 2024-03-06T07:32:21.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a78a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 0x7f3a741414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a78a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a74145430 0x7f3a74153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a78a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a7415cc90 0x7f3a7414ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a78a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7415d0d0 con 0x7f3a740b2080 2024-03-06T07:32:21.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a78a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a741606e0 con 0x7f3a74145430 2024-03-06T07:32:21.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a78a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a74145810 con 0x7f3a7415cc90 2024-03-06T07:32:21.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a72800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 0x7f3a741414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a72800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 0x7f3a741414f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46574/0 (socket says 172.21.15.32:46574) 2024-03-06T07:32:21.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.405+0000 7f3a72800640 1 -- 172.21.15.32:0/2606573065 learned_addr learned my addr 172.21.15.32:0/2606573065 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:21.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a6be00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a74145430 0x7f3a74153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a72800640 1 -- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a7415cc90 msgr2=0x7f3a7414ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a72800640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a7415cc90 0x7f3a7414ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a72800640 1 -- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a74145430 msgr2=0x7f3a74153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a72800640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a74145430 0x7f3a74153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a72800640 1 -- 172.21.15.32:0/2606573065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74141a30 con 0x7f3a740b2080 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a6be00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a74145430 0x7f3a74153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a72800640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 0x7f3a741414f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d800 tx=0x7f3a6400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400b6c0 con 0x7f3a740b2080 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64019070 con 0x7f3a740b2080 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a74071830 con 0x7f3a740b2080 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.406+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400ba80 con 0x7f3a740b2080 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.407+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a74071df0 con 0x7f3a740b2080 2024-03-06T07:32:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.408+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7404f330 con 0x7f3a740b2080 2024-03-06T07:32:21.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.409+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a6400bc20 con 0x7f3a740b2080 2024-03-06T07:32:21.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.410+0000 7f3a6aa00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a5006b4d0 0x7f3a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.410+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3a6408a320 con 0x7f3a740b2080 2024-03-06T07:32:21.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.410+0000 7f3a6be00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a5006b4d0 0x7f3a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.411+0000 7f3a6be00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a5006b4d0 0x7f3a5006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c036000 tx=0x7f3a5c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.414+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a64059530 con 0x7f3a740b2080 2024-03-06T07:32:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.616+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f3a74074ab0 con 0x7f3a740b2080 2024-03-06T07:32:21.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.621+0000 7f3a6aa00640 1 -- 172.21.15.32:0/2606573065 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v21) v1 ==== 108+0+0 (secure 0 0 0) 0x7f3a64058e30 con 0x7f3a740b2080 2024-03-06T07:32:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.624+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a5006b4d0 msgr2=0x7f3a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.624+0000 7f3a78a00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a5006b4d0 0x7f3a5006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c036000 tx=0x7f3a5c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.624+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 msgr2=0x7f3a741414f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.624+0000 7f3a78a00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 0x7f3a741414f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d800 tx=0x7f3a6400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 shutdown_connections 2024-03-06T07:32:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a5006b4d0 0x7f3a5006d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a7415cc90 0x7f3a7414ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a74145430 0x7f3a74153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 --2- 172.21.15.32:0/2606573065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a740b2080 0x7f3a741414f0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 >> 172.21.15.32:0/2606573065 conn(0x7f3a7406b3a0 msgr2=0x7f3a7406d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 shutdown_connections 2024-03-06T07:32:21.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.625+0000 7f3a78a00640 1 -- 172.21.15.32:0/2606573065 wait complete. 2024-03-06T07:32:21.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:32:21.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.873+0000 7f40fb200640 1 Processor -- start 2024-03-06T07:32:21.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.873+0000 7f40fb200640 1 -- start start 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40fb200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40fc145300 0x7f40fc0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40fb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc070f10 0x7f40fc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40fb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40fc0714d0 con 0x7f40fc0cce90 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40fb200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40fc0cd300 con 0x7f40fc145300 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40fb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc0717b0 con 0x7f40fc070f10 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40f9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40fc145300 0x7f40fc0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40f9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40fc145300 0x7f40fc0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46588/0 (socket says 172.21.15.32:46588) 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.875+0000 7f40f9e00640 1 -- 172.21.15.32:0/1656327312 learned_addr learned my addr 172.21.15.32:0/1656327312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.876+0000 7f40f9400640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc070f10 0x7f40fc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.876+0000 7f40f9e00640 1 -- 172.21.15.32:0/1656327312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc070f10 msgr2=0x7f40fc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.876+0000 7f40f9e00640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc070f10 0x7f40fc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.876+0000 7f40f9e00640 1 -- 172.21.15.32:0/1656327312 >> v1:172.21.15.39:6789/0 conn(0x7f40fc0cce90 legacy=0x7f40fc0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.876+0000 7f40f9e00640 1 -- 172.21.15.32:0/1656327312 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc071a90 con 0x7f40fc145300 2024-03-06T07:32:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.876+0000 7f40f9400640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc070f10 0x7f40fc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.877+0000 7f40f9e00640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc145300 0x7f40fc0b49a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009870 tx=0x7f40ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b01290aa68a87c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.877+0000 7f40f8a00640 1 -- 172.21.15.32:0/1656327312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec035020 con 0x7f40fc145300 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.877+0000 7f40f8a00640 1 -- 172.21.15.32:0/1656327312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec031640 con 0x7f40fc145300 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.877+0000 7f40f8a00640 1 -- 172.21.15.32:0/1656327312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec02d040 con 0x7f40fc145300 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 -- 172.21.15.32:0/1656327312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc145300 msgr2=0x7f40fc0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc145300 0x7f40fc0b49a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009870 tx=0x7f40ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 -- 172.21.15.32:0/1656327312 shutdown_connections 2024-03-06T07:32:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc070f10 0x7f40fc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 --2- 172.21.15.32:0/1656327312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc145300 0x7f40fc0b49a0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 -- 172.21.15.32:0/1656327312 >> 172.21.15.32:0/1656327312 conn(0x7f40fc06b3d0 msgr2=0x7f40fc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 -- 172.21.15.32:0/1656327312 shutdown_connections 2024-03-06T07:32:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.878+0000 7f40fb200640 1 -- 172.21.15.32:0/1656327312 wait complete. 2024-03-06T07:32:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.880+0000 7f40fb200640 1 Processor -- start 2024-03-06T07:32:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.880+0000 7f40fb200640 1 -- start start 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 0x7f40fc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc0cce90 0x7f40fc143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc145300 0x7f40fc06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1456e0 con 0x7f40fc070f10 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc0b4ee0 con 0x7f40fc145300 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc15d580 con 0x7f40fc0cce90 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40f9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 0x7f40fc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40f9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 0x7f40fc1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46600/0 (socket says 172.21.15.32:46600) 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40f9e00640 1 -- 172.21.15.32:0/1429620274 learned_addr learned my addr 172.21.15.32:0/1429620274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.881+0000 7f40fa800640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc145300 0x7f40fc06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f9e00640 1 -- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc0cce90 msgr2=0x7f40fc143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f9e00640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc0cce90 0x7f40fc143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f9e00640 1 -- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc145300 msgr2=0x7f40fc06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f9e00640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc145300 0x7f40fc06dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f9e00640 1 -- 172.21.15.32:0/1429620274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc141ac0 con 0x7f40fc070f10 2024-03-06T07:32:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40fa800640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc145300 0x7f40fc06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f9e00640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 0x7f40fc1414b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009870 tx=0x7f40ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec040070 con 0x7f40fc070f10 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec039c10 con 0x7f40fc070f10 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc141da0 con 0x7f40fc070f10 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec035040 con 0x7f40fc070f10 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.882+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc142360 con 0x7f40fc070f10 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.883+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40fc04f330 con 0x7f40fc070f10 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.884+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40ec04b060 con 0x7f40fc070f10 2024-03-06T07:32:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.885+0000 7f40f2000640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40dc06b4d0 0x7f40dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.885+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f40ec0ab8d0 con 0x7f40fc070f10 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.886+0000 7f40f9400640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40dc06b4d0 0x7f40dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.886+0000 7f40f9400640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40dc06b4d0 0x7f40dc06d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f40e8006fd0 tx=0x7f40e8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:21.890+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40ec02f300 con 0x7f40fc070f10 2024-03-06T07:32:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.162+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f40fc0adcc0 con 0x7f40fc070f10 2024-03-06T07:32:22.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.219+0000 7f40f2000640 1 -- 172.21.15.32:0/1429620274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v197) v1 ==== 144+0+0 (secure 0 0 0) 0x7f40ec07aa50 con 0x7f40fc070f10 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40dc06b4d0 msgr2=0x7f40dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40dc06b4d0 0x7f40dc06d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f40e8006fd0 tx=0x7f40e8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 msgr2=0x7f40fc1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 0x7f40fc1414b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009870 tx=0x7f40ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 shutdown_connections 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40dc06b4d0 0x7f40dc06d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc145300 0x7f40fc06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40fc0cce90 0x7f40fc143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 --2- 172.21.15.32:0/1429620274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40fc070f10 0x7f40fc1414b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 >> 172.21.15.32:0/1429620274 conn(0x7f40fc06b3d0 msgr2=0x7f40fc15e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 shutdown_connections 2024-03-06T07:32:22.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.224+0000 7f40fb200640 1 -- 172.21.15.32:0/1429620274 wait complete. 2024-03-06T07:32:22.248 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:32:22.248 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:32:22.248 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:22.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.466+0000 7fb966400640 1 Processor -- start 2024-03-06T07:32:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.467+0000 7fb966400640 1 -- start start 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.468+0000 7fb966400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb960161b40 0x7fb9600cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.468+0000 7fb966400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb9600b5060 0x7fb9600d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.468+0000 7fb966400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9600cea50 con 0x7fb9600b54d0 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.468+0000 7fb966400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9600cf900 con 0x7fb9600b5060 2024-03-06T07:32:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.468+0000 7fb966400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9600cecf0 con 0x7fb960161b40 2024-03-06T07:32:22.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.468+0000 7fb95ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb9600b5060 0x7fb9600d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb9600b5060 0x7fb9600d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46610/0 (socket says 172.21.15.32:46610) 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95ea00640 1 -- 172.21.15.32:0/2615697144 learned_addr learned my addr 172.21.15.32:0/2615697144 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95f400640 1 --2- 172.21.15.32:0/2615697144 >> v2:172.21.15.39:3300/0 conn(0x7fb960161b40 0x7fb9600cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95ea00640 1 -- 172.21.15.32:0/2615697144 >> v2:172.21.15.39:3300/0 conn(0x7fb960161b40 msgr2=0x7fb9600cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95ea00640 1 --2- 172.21.15.32:0/2615697144 >> v2:172.21.15.39:3300/0 conn(0x7fb960161b40 0x7fb9600cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95ea00640 1 -- 172.21.15.32:0/2615697144 >> v1:172.21.15.39:6789/0 conn(0x7fb9600b54d0 legacy=0x7fb9600b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95ea00640 1 -- 172.21.15.32:0/2615697144 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9600cefd0 con 0x7fb9600b5060 2024-03-06T07:32:22.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.469+0000 7fb95f400640 1 --2- 172.21.15.32:0/2615697144 >> v2:172.21.15.39:3300/0 conn(0x7fb960161b40 0x7fb9600cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.470+0000 7fb95ea00640 1 --2- 172.21.15.32:0/2615697144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600d2220 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb96008db00 tx=0x7fb94c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76dae2e45d94059 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.470+0000 7fb95e000640 1 -- 172.21.15.32:0/2615697144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c035020 con 0x7fb9600b5060 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb95e000640 1 -- 172.21.15.32:0/2615697144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c031870 con 0x7fb9600b5060 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb95e000640 1 -- 172.21.15.32:0/2615697144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c031c40 con 0x7fb9600b5060 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 -- 172.21.15.32:0/2615697144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 msgr2=0x7fb9600d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 --2- 172.21.15.32:0/2615697144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600d2220 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb96008db00 tx=0x7fb94c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 -- 172.21.15.32:0/2615697144 shutdown_connections 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 --2- 172.21.15.32:0/2615697144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600d2220 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 --2- 172.21.15.32:0/2615697144 >> v2:172.21.15.39:3300/0 conn(0x7fb960161b40 0x7fb9600cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 -- 172.21.15.32:0/2615697144 >> 172.21.15.32:0/2615697144 conn(0x7fb96009d220 msgr2=0x7fb9600d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 -- 172.21.15.32:0/2615697144 shutdown_connections 2024-03-06T07:32:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.471+0000 7fb966400640 1 -- 172.21.15.32:0/2615697144 wait complete. 2024-03-06T07:32:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.473+0000 7fb966400640 1 Processor -- start 2024-03-06T07:32:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.473+0000 7fb966400640 1 -- start start 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb966400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb966400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9600b54d0 0x7fb9600cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb966400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb960161b40 0x7fb960152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb966400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600b2ff0 con 0x7fb9600b5060 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb966400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600cf590 con 0x7fb9600b54d0 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb966400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600b3170 con 0x7fb960161b40 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb95f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb95f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46614/0 (socket says 172.21.15.32:46614) 2024-03-06T07:32:22.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.474+0000 7fb95f400640 1 -- 172.21.15.32:0/2077713993 learned_addr learned my addr 172.21.15.32:0/2077713993 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:22.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb95f400640 1 -- 172.21.15.32:0/2077713993 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb960161b40 msgr2=0x7fb960152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb95f400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb960161b40 0x7fb960152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb95f400640 1 -- 172.21.15.32:0/2077713993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9600b54d0 msgr2=0x7fb9600cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb95f400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9600b54d0 0x7fb9600cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb95f400640 1 -- 172.21.15.32:0/2077713993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb960152ad0 con 0x7fb9600b5060 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb95f400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600c8e70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c7f0 tx=0x7fb95000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950015070 con 0x7fb9600b5060 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb950007820 con 0x7fb9600b5060 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950007bd0 con 0x7fb9600b5060 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.475+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9600c9570 con 0x7fb9600b5060 2024-03-06T07:32:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.476+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9600c9b30 con 0x7fb9600b5060 2024-03-06T07:32:22.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.477+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9600b2de0 con 0x7fb9600b5060 2024-03-06T07:32:22.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.478+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb950018070 con 0x7fb9600b5060 2024-03-06T07:32:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.479+0000 7fb957400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb94006b520 0x7fb94006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.479+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb950089f10 con 0x7fb9600b5060 2024-03-06T07:32:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.479+0000 7fb95ea00640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb94006b520 0x7fb94006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.480+0000 7fb95ea00640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb94006b520 0x7fb94006d9e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb94c036000 tx=0x7fb94c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.483+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9600b2de0 con 0x7fb9600b5060 2024-03-06T07:32:22.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.685+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb96009d600 con 0x7fb9600b5060 2024-03-06T07:32:22.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.687+0000 7fb957400640 1 -- 172.21.15.32:0/2077713993 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+12910 (secure 0 0 0) 0x7fb96009d600 con 0x7fb9600b5060 2024-03-06T07:32:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:22.690 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":60,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:20.203783+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6837/436488985":"2024-03-07T07:32:17.185908+0000","172.21.15.32:6836/436488985":"2024-03-07T07:32:17.185908+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb94006b520 msgr2=0x7fb94006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb94006b520 0x7fb94006d9e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb94c036000 tx=0x7fb94c030000 comp rx=0 tx=0).stop 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 msgr2=0x7fb9600c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600c8e70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c7f0 tx=0x7fb95000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 shutdown_connections 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb94006b520 0x7fb94006d9e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb960161b40 0x7fb960152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9600b54d0 0x7fb9600cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 --2- 172.21.15.32:0/2077713993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9600b5060 0x7fb9600c8e70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.689+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 >> 172.21.15.32:0/2077713993 conn(0x7fb96009d220 msgr2=0x7fb9600d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.690+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 shutdown_connections 2024-03-06T07:32:22.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.690+0000 7fb966400640 1 -- 172.21.15.32:0/2077713993 wait complete. 2024-03-06T07:32:22.712 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:22.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.936+0000 7fc220a00640 1 Processor -- start 2024-03-06T07:32:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.936+0000 7fc220a00640 1 -- start start 2024-03-06T07:32:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc220a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc21c15afa0 0x7fc21c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc220a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc21c070ce0 0x7fc21c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc220a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21c071c60 con 0x7fc21c0cd010 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc220a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c0cd480 con 0x7fc21c070ce0 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc220a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c0cfa10 con 0x7fc21c15afa0 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc219400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc21c070ce0 0x7fc21c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc219400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc21c070ce0 0x7fc21c0ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46620/0 (socket says 172.21.15.32:46620) 2024-03-06T07:32:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.938+0000 7fc219400640 1 -- 172.21.15.32:0/2946386376 learned_addr learned my addr 172.21.15.32:0/2946386376 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.939+0000 7fc219e00640 1 --2- 172.21.15.32:0/2946386376 >> v2:172.21.15.39:3300/0 conn(0x7fc21c15afa0 0x7fc21c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.939+0000 7fc219400640 1 -- 172.21.15.32:0/2946386376 >> v2:172.21.15.39:3300/0 conn(0x7fc21c15afa0 msgr2=0x7fc21c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.939+0000 7fc219400640 1 --2- 172.21.15.32:0/2946386376 >> v2:172.21.15.39:3300/0 conn(0x7fc21c15afa0 0x7fc21c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.939+0000 7fc219400640 1 -- 172.21.15.32:0/2946386376 >> v1:172.21.15.39:6789/0 conn(0x7fc21c0cd010 legacy=0x7fc21c06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.939+0000 7fc219400640 1 -- 172.21.15.32:0/2946386376 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c071f40 con 0x7fc21c070ce0 2024-03-06T07:32:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.939+0000 7fc219e00640 1 --2- 172.21.15.32:0/2946386376 >> v2:172.21.15.39:3300/0 conn(0x7fc21c15afa0 0x7fc21c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.940+0000 7fc219400640 1 --2- 172.21.15.32:0/2946386376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c0ccad0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc21c06bfa0 tx=0x7fc20402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed9de0307997d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc218a00640 1 -- 172.21.15.32:0/2946386376 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20402e070 con 0x7fc21c070ce0 2024-03-06T07:32:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc218a00640 1 -- 172.21.15.32:0/2946386376 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc204037070 con 0x7fc21c070ce0 2024-03-06T07:32:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc218a00640 1 -- 172.21.15.32:0/2946386376 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204031a00 con 0x7fc21c070ce0 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 -- 172.21.15.32:0/2946386376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 msgr2=0x7fc21c0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 --2- 172.21.15.32:0/2946386376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c0ccad0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc21c06bfa0 tx=0x7fc20402b880 comp rx=0 tx=0).stop 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 -- 172.21.15.32:0/2946386376 shutdown_connections 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 --2- 172.21.15.32:0/2946386376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c0ccad0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 --2- 172.21.15.32:0/2946386376 >> v2:172.21.15.39:3300/0 conn(0x7fc21c15afa0 0x7fc21c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 -- 172.21.15.32:0/2946386376 >> 172.21.15.32:0/2946386376 conn(0x7fc21c06b3d0 msgr2=0x7fc21c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.941+0000 7fc220a00640 1 -- 172.21.15.32:0/2946386376 shutdown_connections 2024-03-06T07:32:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.942+0000 7fc220a00640 1 -- 172.21.15.32:0/2946386376 wait complete. 2024-03-06T07:32:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.943+0000 7fc220a00640 1 Processor -- start 2024-03-06T07:32:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.943+0000 7fc220a00640 1 -- start start 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.944+0000 7fc220a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.944+0000 7fc220a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc21c0cd010 0x7fc21c15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.944+0000 7fc220a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc21c15afa0 0x7fc21c1f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.944+0000 7fc220a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c071170 con 0x7fc21c070ce0 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.944+0000 7fc220a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c072500 con 0x7fc21c15afa0 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.944+0000 7fc220a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0b4ad0 con 0x7fc21c0cd010 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46634/0 (socket says 172.21.15.32:46634) 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 -- 172.21.15.32:0/1367684614 learned_addr learned my addr 172.21.15.32:0/1367684614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219400640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc21c0cd010 0x7fc21c15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc21a800640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc21c15afa0 0x7fc21c1f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 -- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc21c0cd010 msgr2=0x7fc21c15d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc21c0cd010 0x7fc21c15d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 -- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc21c15afa0 msgr2=0x7fc21c1f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc21c15afa0 0x7fc21c1f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.945+0000 7fc219e00640 1 -- 172.21.15.32:0/1367684614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c1f48a0 con 0x7fc21c070ce0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.946+0000 7fc219e00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c153b10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c7f0 tx=0x7fc20c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.946+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c015070 con 0x7fc21c070ce0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.946+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c007820 con 0x7fc21c070ce0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.946+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c007bd0 con 0x7fc21c070ce0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.946+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c1f4b80 con 0x7fc21c070ce0 2024-03-06T07:32:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.946+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c15cd90 con 0x7fc21c070ce0 2024-03-06T07:32:22.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.947+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c1558c0 con 0x7fc21c070ce0 2024-03-06T07:32:22.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.948+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc20c018070 con 0x7fc21c070ce0 2024-03-06T07:32:22.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.949+0000 7fc212000640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc1f806b4d0 0x7fc1f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.949+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc20c08a080 con 0x7fc21c070ce0 2024-03-06T07:32:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.950+0000 7fc219400640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc1f806b4d0 0x7fc1f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.950+0000 7fc219400640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc1f806b4d0 0x7fc1f806d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc204036000 tx=0x7fc20403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:22.954+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc21c1558c0 con 0x7fc21c070ce0 2024-03-06T07:32:23.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.154+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc21c04f330 con 0x7fc21c070ce0 2024-03-06T07:32:23.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.155+0000 7fc212000640 1 -- 172.21.15.32:0/1367684614 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc20c0591f0 con 0x7fc21c070ce0 2024-03-06T07:32:23.157 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc1f806b4d0 msgr2=0x7fc1f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc1f806b4d0 0x7fc1f806d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc204036000 tx=0x7fc20403d040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 msgr2=0x7fc21c153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c153b10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c7f0 tx=0x7fc20c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 shutdown_connections 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc1f806b4d0 0x7fc1f806d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc21c15afa0 0x7fc21c1f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc21c0cd010 0x7fc21c15d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 --2- 172.21.15.32:0/1367684614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc21c070ce0 0x7fc21c153b10 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.157+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 >> 172.21.15.32:0/1367684614 conn(0x7fc21c06b3d0 msgr2=0x7fc21c158980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.158+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 shutdown_connections 2024-03-06T07:32:23.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.158+0000 7fc220a00640 1 -- 172.21.15.32:0/1367684614 wait complete. 2024-03-06T07:32:23.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:32:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.389+0000 7fd240800640 1 Processor -- start 2024-03-06T07:32:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.389+0000 7fd240800640 1 -- start start 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd240800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd238161670 0x7fd238070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd240800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806e240 0x7fd2380711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd240800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd23806cfe0 0x7fd23806d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd240800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2380730d0 con 0x7fd23806cfe0 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd240800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2380ce9a0 con 0x7fd238161670 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd240800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380cec80 con 0x7fd23806e240 2024-03-06T07:32:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd23e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd23806cfe0 0x7fd23806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd23e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd23806cfe0 0x7fd23806d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46648/0 (socket says 172.21.15.32:46648) 2024-03-06T07:32:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd23e800640 1 -- 172.21.15.32:0/2993766906 learned_addr learned my addr 172.21.15.32:0/2993766906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd23de00640 1 --2- 172.21.15.32:0/2993766906 >> v2:172.21.15.39:3300/0 conn(0x7fd238161670 0x7fd238070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.391+0000 7fd23d400640 1 --2- 172.21.15.32:0/2993766906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806e240 0x7fd2380711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.392+0000 7fd23e800640 1 -- 172.21.15.32:0/2993766906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806e240 msgr2=0x7fd2380711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.392+0000 7fd23e800640 1 --2- 172.21.15.32:0/2993766906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806e240 0x7fd2380711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.392+0000 7fd23e800640 1 -- 172.21.15.32:0/2993766906 >> v2:172.21.15.39:3300/0 conn(0x7fd238161670 msgr2=0x7fd238070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.392+0000 7fd23e800640 1 --2- 172.21.15.32:0/2993766906 >> v2:172.21.15.39:3300/0 conn(0x7fd238161670 0x7fd238070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.392+0000 7fd23e800640 1 -- 172.21.15.32:0/2993766906 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380cef60 con 0x7fd23806cfe0 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.393+0000 7fd23e800640 1 --2- 172.21.15.32:0/2993766906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23806cfe0 0x7fd23806d400 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd238073610 tx=0x7fd23402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4a1a216cce8928 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.393+0000 7fd23ca00640 1 -- 172.21.15.32:0/2993766906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23402ca70 con 0x7fd23806cfe0 2024-03-06T07:32:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.393+0000 7fd23ca00640 1 -- 172.21.15.32:0/2993766906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23402cc10 con 0x7fd23806cfe0 2024-03-06T07:32:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.393+0000 7fd23ca00640 1 -- 172.21.15.32:0/2993766906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340343f0 con 0x7fd23806cfe0 2024-03-06T07:32:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 -- 172.21.15.32:0/2993766906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23806cfe0 msgr2=0x7fd23806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 --2- 172.21.15.32:0/2993766906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23806cfe0 0x7fd23806d400 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd238073610 tx=0x7fd23402bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 -- 172.21.15.32:0/2993766906 shutdown_connections 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 --2- 172.21.15.32:0/2993766906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23806cfe0 0x7fd23806d400 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 --2- 172.21.15.32:0/2993766906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806e240 0x7fd2380711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 --2- 172.21.15.32:0/2993766906 >> v2:172.21.15.39:3300/0 conn(0x7fd238161670 0x7fd238070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 -- 172.21.15.32:0/2993766906 >> 172.21.15.32:0/2993766906 conn(0x7fd23806afb0 msgr2=0x7fd23806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 -- 172.21.15.32:0/2993766906 shutdown_connections 2024-03-06T07:32:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.394+0000 7fd240800640 1 -- 172.21.15.32:0/2993766906 wait complete. 2024-03-06T07:32:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.396+0000 7fd240800640 1 Processor -- start 2024-03-06T07:32:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.396+0000 7fd240800640 1 -- start start 2024-03-06T07:32:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd240800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806cfe0 0x7fd2381507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd240800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23806e240 0x7fd23809b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd240800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 0x7fd23815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd240800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380ce600 con 0x7fd238161670 2024-03-06T07:32:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd240800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380cf520 con 0x7fd23806e240 2024-03-06T07:32:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd240800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380ce780 con 0x7fd23806cfe0 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd23e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 0x7fd23815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd23e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 0x7fd23815ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46650/0 (socket says 172.21.15.32:46650) 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.397+0000 7fd23e800640 1 -- 172.21.15.32:0/683204297 learned_addr learned my addr 172.21.15.32:0/683204297 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23d400640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23806e240 0x7fd23809b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23de00640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806cfe0 0x7fd2381507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23e800640 1 -- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806cfe0 msgr2=0x7fd2381507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23e800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806cfe0 0x7fd2381507e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23e800640 1 -- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23806e240 msgr2=0x7fd23809b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23e800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23806e240 0x7fd23809b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23e800640 1 -- 172.21.15.32:0/683204297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd238150d20 con 0x7fd238161670 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23d400640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23806e240 0x7fd23809b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd23e800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 0x7fd23815ab30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd234009f40 tx=0x7fd234004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234043020 con 0x7fd238161670 2024-03-06T07:32:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23403e450 con 0x7fd238161670 2024-03-06T07:32:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234009510 con 0x7fd238161670 2024-03-06T07:32:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.398+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23809cd10 con 0x7fd238161670 2024-03-06T07:32:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.399+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23809bf50 con 0x7fd238161670 2024-03-06T07:32:23.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.400+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238069510 con 0x7fd238161670 2024-03-06T07:32:23.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.401+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd234046020 con 0x7fd238161670 2024-03-06T07:32:23.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.401+0000 7fd232000640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd21c06b500 0x7fd21c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.402+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd2340ac6b0 con 0x7fd238161670 2024-03-06T07:32:23.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.402+0000 7fd23de00640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd21c06b500 0x7fd21c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.403+0000 7fd23de00640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd21c06b500 0x7fd21c06d9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd22c006fd0 tx=0x7fd22c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.406+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd23404d030 con 0x7fd238161670 2024-03-06T07:32:23.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:23.678+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fd2380aeb30 con 0x7fd238161670 2024-03-06T07:32:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.236+0000 7fd232000640 1 -- 172.21.15.32:0/683204297 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v199) v1 ==== 143+0+0 (secure 0 0 0) 0x7fd23407b820 con 0x7fd238161670 2024-03-06T07:32:24.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd21c06b500 msgr2=0x7fd21c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd21c06b500 0x7fd21c06d9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd22c006fd0 tx=0x7fd22c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 msgr2=0x7fd23815ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 0x7fd23815ab30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd234009f40 tx=0x7fd234004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 shutdown_connections 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd21c06b500 0x7fd21c06d9c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd238161670 0x7fd23815ab30 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23806e240 0x7fd23809b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 --2- 172.21.15.32:0/683204297 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd23806cfe0 0x7fd2381507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 >> 172.21.15.32:0/683204297 conn(0x7fd23806afb0 msgr2=0x7fd2380ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 shutdown_connections 2024-03-06T07:32:24.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.240+0000 7fd240800640 1 -- 172.21.15.32:0/683204297 wait complete. 2024-03-06T07:32:24.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.489+0000 7f097fc00640 1 Processor -- start 2024-03-06T07:32:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.490+0000 7f097fc00640 1 -- start start 2024-03-06T07:32:24.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.491+0000 7f097fc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f097815e280 0x7f09780d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.491+0000 7f097fc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f09780d4a00 0x7f09780d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.491+0000 7f097fc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780d0080 con 0x7f09780d27c0 2024-03-06T07:32:24.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.491+0000 7f097fc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09780cf500 con 0x7f097815e280 2024-03-06T07:32:24.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.491+0000 7f097fc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09780cf7e0 con 0x7f09780d4a00 2024-03-06T07:32:24.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f097815e280 0x7f09780d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097dc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f09780d27c0 0x7f09780d05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59722/0 (socket says 172.21.15.32:59722) 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097dc00640 1 -- 172.21.15.32:0/1321522852 learned_addr learned my addr 172.21.15.32:0/1321522852 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f097815e280 0x7f09780d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46658/0 (socket says 172.21.15.32:46658) 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f0977e00640 1 --2- 172.21.15.32:0/1321522852 >> v2:172.21.15.39:3300/0 conn(0x7f09780d4a00 0x7f09780d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097d200640 1 -- 172.21.15.32:0/1321522852 >> v2:172.21.15.39:3300/0 conn(0x7f09780d4a00 msgr2=0x7f09780d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097d200640 1 --2- 172.21.15.32:0/1321522852 >> v2:172.21.15.39:3300/0 conn(0x7f09780d4a00 0x7f09780d2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097d200640 1 -- 172.21.15.32:0/1321522852 >> v1:172.21.15.32:6789/0 conn(0x7f09780d27c0 legacy=0x7f09780d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.492+0000 7f097d200640 1 -- 172.21.15.32:0/1321522852 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780cfac0 con 0x7f097815e280 2024-03-06T07:32:24.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.493+0000 7f097d200640 1 --2- 172.21.15.32:0/1321522852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f09780d1cc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f097808e320 tx=0x7f096802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f75bf4218ea1bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f0977400640 1 -- 172.21.15.32:0/1321522852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09680315a0 con 0x7f097815e280 2024-03-06T07:32:24.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f0977400640 1 -- 172.21.15.32:0/1321522852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0968037070 con 0x7f097815e280 2024-03-06T07:32:24.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f0977400640 1 -- 172.21.15.32:0/1321522852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968031960 con 0x7f097815e280 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 -- 172.21.15.32:0/1321522852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 msgr2=0x7f09780d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 --2- 172.21.15.32:0/1321522852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f09780d1cc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f097808e320 tx=0x7f096802b840 comp rx=0 tx=0).stop 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 -- 172.21.15.32:0/1321522852 shutdown_connections 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 --2- 172.21.15.32:0/1321522852 >> v2:172.21.15.39:3300/0 conn(0x7f09780d4a00 0x7f09780d2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 --2- 172.21.15.32:0/1321522852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f09780d1cc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 -- 172.21.15.32:0/1321522852 >> 172.21.15.32:0/1321522852 conn(0x7f09780d3ee0 msgr2=0x7f09780d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 -- 172.21.15.32:0/1321522852 shutdown_connections 2024-03-06T07:32:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.494+0000 7f097fc00640 1 -- 172.21.15.32:0/1321522852 wait complete. 2024-03-06T07:32:24.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.496+0000 7f097fc00640 1 Processor -- start 2024-03-06T07:32:24.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.496+0000 7f097fc00640 1 -- start start 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09780d27c0 0x7f09780a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09780d4a00 0x7f0978155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f0978156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097815e660 con 0x7f097815e280 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09780cb0e0 con 0x7f09780d27c0 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09780cc610 con 0x7f09780d4a00 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.497+0000 7f097dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f0978156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f0978156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46660/0 (socket says 172.21.15.32:46660) 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 -- 172.21.15.32:0/229229381 learned_addr learned my addr 172.21.15.32:0/229229381 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097d200640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09780d27c0 0x7f09780a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f0977e00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09780d4a00 0x7f0978155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 -- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09780d4a00 msgr2=0x7f0978155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09780d4a00 0x7f0978155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 -- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09780d27c0 msgr2=0x7f09780a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09780d27c0 0x7f09780a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 -- 172.21.15.32:0/229229381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780c9340 con 0x7f097815e280 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.498+0000 7f097dc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f0978156320 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f097000c420 tx=0x7f097000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.499+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970017070 con 0x7f097815e280 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.499+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0970005ce0 con 0x7f097815e280 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.499+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700072d0 con 0x7f097815e280 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.499+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097809dcc0 con 0x7f097815e280 2024-03-06T07:32:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.499+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097809e280 con 0x7f097815e280 2024-03-06T07:32:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.500+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09780c96e0 con 0x7f097815e280 2024-03-06T07:32:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.501+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f097001a070 con 0x7f097815e280 2024-03-06T07:32:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.502+0000 7f0974c00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f095406b4d0 0x7f095406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.502+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0970089160 con 0x7f097815e280 2024-03-06T07:32:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.502+0000 7f097d200640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f095406b4d0 0x7f095406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.503+0000 7f097d200640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f095406b4d0 0x7f095406d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0968036000 tx=0x7f096803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.506+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0970057c80 con 0x7f097815e280 2024-03-06T07:32:24.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.706+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f097806d450 con 0x7f097815e280 2024-03-06T07:32:24.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.711+0000 7f0974c00640 1 -- 172.21.15.32:0/229229381 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f097005bd20 con 0x7f097815e280 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f095406b4d0 msgr2=0x7f095406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f095406b4d0 0x7f095406d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0968036000 tx=0x7f096803d040 comp rx=0 tx=0).stop 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 msgr2=0x7f0978156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f0978156320 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f097000c420 tx=0x7f097000c8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 shutdown_connections 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f095406b4d0 0x7f095406d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097815e280 0x7f0978156320 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09780d4a00 0x7f0978155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 --2- 172.21.15.32:0/229229381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09780d27c0 0x7f09780a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 >> 172.21.15.32:0/229229381 conn(0x7f09780d3ee0 msgr2=0x7f097806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 shutdown_connections 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:24.714+0000 7f097fc00640 1 -- 172.21.15.32:0/229229381 wait complete. 2024-03-06T07:32:24.735 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:24.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:24.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:24.978 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:24.978 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:24.978 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.039 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:25.039 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:32:25.039 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.100 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:25.100 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:25.100 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.158 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:25.158 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:32:25.158 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.217 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:25.217 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:25.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:25.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.446+0000 7fc77f000640 1 Processor -- start 2024-03-06T07:32:25.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.446+0000 7fc77f000640 1 -- start start 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.447+0000 7fc77f000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7780d2930 0x7fc7780b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.447+0000 7fc77f000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780cea50 con 0x7fc7780cfd90 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.447+0000 7fc77f000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780cecf0 con 0x7fc778161b40 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.447+0000 7fc77f000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7780b5770 con 0x7fc7780d2930 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.447+0000 7fc777400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7780cfd90 0x7fc7780d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59724/0 (socket says 172.21.15.32:59724) 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.447+0000 7fc777400640 1 -- 172.21.15.32:0/2290417261 learned_addr learned my addr 172.21.15.32:0/2290417261 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.448+0000 7fc77d000640 1 --2- 172.21.15.32:0/2290417261 >> v2:172.21.15.32:3300/0 conn(0x7fc7780d2930 0x7fc7780b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.448+0000 7fc77d000640 1 -- 172.21.15.32:0/2290417261 >> v1:172.21.15.39:6789/0 conn(0x7fc778161b40 legacy=0x7fc7780b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.448+0000 7fc77d000640 1 -- 172.21.15.32:0/2290417261 >> v1:172.21.15.32:6789/0 conn(0x7fc7780cfd90 legacy=0x7fc7780d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.448+0000 7fc77d000640 1 -- 172.21.15.32:0/2290417261 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7780cefd0 con 0x7fc7780d2930 2024-03-06T07:32:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.449+0000 7fc77d000640 1 --2- 172.21.15.32:0/2290417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7780d2930 0x7fc7780b3740 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc7700097a0 tx=0x7fc77002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1651cb6a512290 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc776a00640 1 -- 172.21.15.32:0/2290417261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770035020 con 0x7fc7780d2930 2024-03-06T07:32:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc776a00640 1 -- 172.21.15.32:0/2290417261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc770031830 con 0x7fc7780d2930 2024-03-06T07:32:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc776a00640 1 -- 172.21.15.32:0/2290417261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770031c00 con 0x7fc7780d2930 2024-03-06T07:32:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 -- 172.21.15.32:0/2290417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7780d2930 msgr2=0x7fc7780b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 --2- 172.21.15.32:0/2290417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7780d2930 0x7fc7780b3740 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc7700097a0 tx=0x7fc77002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 -- 172.21.15.32:0/2290417261 shutdown_connections 2024-03-06T07:32:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 --2- 172.21.15.32:0/2290417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7780d2930 0x7fc7780b3740 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 -- 172.21.15.32:0/2290417261 >> 172.21.15.32:0/2290417261 conn(0x7fc77809d220 msgr2=0x7fc7780d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 -- 172.21.15.32:0/2290417261 shutdown_connections 2024-03-06T07:32:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.450+0000 7fc77f000640 1 -- 172.21.15.32:0/2290417261 wait complete. 2024-03-06T07:32:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.452+0000 7fc77f000640 1 Processor -- start 2024-03-06T07:32:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.452+0000 7fc77f000640 1 -- start start 2024-03-06T07:32:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.453+0000 7fc77f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780cfd90 0x7fc778158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.453+0000 7fc77f000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7780d2930 0x7fc7780b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.453+0000 7fc77f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 0x7fc7780cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.453+0000 7fc77f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780cf590 con 0x7fc778161b40 2024-03-06T07:32:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.453+0000 7fc77f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780cf900 con 0x7fc7780cfd90 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.453+0000 7fc77f000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780b2ff0 con 0x7fc7780d2930 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 0x7fc7780cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 0x7fc7780cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46686/0 (socket says 172.21.15.32:46686) 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 -- 172.21.15.32:0/4134633117 learned_addr learned my addr 172.21.15.32:0/4134633117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc777400640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7780d2930 0x7fc7780b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc777e00640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780cfd90 0x7fc778158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 -- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7780d2930 msgr2=0x7fc7780b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7780d2930 0x7fc7780b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 -- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780cfd90 msgr2=0x7fc778158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780cfd90 0x7fc778158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 -- 172.21.15.32:0/4134633117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7780c8660 con 0x7fc778161b40 2024-03-06T07:32:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.454+0000 7fc77d000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 0x7fc7780cd380 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc77002bee0 tx=0x7fc7700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.455+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770041070 con 0x7fc778161b40 2024-03-06T07:32:25.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.455+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77003a8d0 con 0x7fc778161b40 2024-03-06T07:32:25.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.455+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770035040 con 0x7fc778161b40 2024-03-06T07:32:25.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.455+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7780c8940 con 0x7fc778161b40 2024-03-06T07:32:25.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.455+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc778158f60 con 0x7fc778161b40 2024-03-06T07:32:25.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.456+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77804f3a0 con 0x7fc778161b40 2024-03-06T07:32:25.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.457+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc770038020 con 0x7fc778161b40 2024-03-06T07:32:25.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.458+0000 7fc76fe00640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.458+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc7700aba70 con 0x7fc778161b40 2024-03-06T07:32:25.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.458+0000 7fc777e00640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.459+0000 7fc777e00640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc75406b4d0 0x7fc75406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc768005520 tx=0x7fc768009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.462+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc77007aac0 con 0x7fc778161b40 2024-03-06T07:32:25.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.702+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7780d1df0 con 0x7fc778161b40 2024-03-06T07:32:25.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.704+0000 7fc76fe00640 1 -- 172.21.15.32:0/4134633117 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fc77007a3c0 con 0x7fc778161b40 2024-03-06T07:32:25.706 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2239557977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2239557977},{"type":"v1","addr":"172.21.15.32:6835","nonce":2239557977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4919,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/379842450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":379842450},{"type":"v1","addr":"172.21.15.39:6835","nonce":379842450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5248,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2076973581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2076973581},{"type":"v1","addr":"172.21.15.39:6833","nonce":2076973581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/458034503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":458034503},{"type":"v1","addr":"172.21.15.32:6837","nonce":458034503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2024-03-06T07:32:25.706 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 36 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc75406b4d0 msgr2=0x7fc75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc75406b4d0 0x7fc75406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc768005520 tx=0x7fc768009040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 msgr2=0x7fc7780cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 0x7fc7780cd380 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc77002bee0 tx=0x7fc7700046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 shutdown_connections 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc778161b40 0x7fc7780cd380 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc7780d2930 0x7fc7780b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 --2- 172.21.15.32:0/4134633117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780cfd90 0x7fc778158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.706+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 >> 172.21.15.32:0/4134633117 conn(0x7fc77809d220 msgr2=0x7fc77806bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.707+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 shutdown_connections 2024-03-06T07:32:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:25.707+0000 7fc77f000640 1 -- 172.21.15.32:0/4134633117 wait complete. 2024-03-06T07:32:25.728 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:25.728 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:32:25.728 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:25.728 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:25.728 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:25.728 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:32:25.728 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:25.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:25.729 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:25.729 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:32:25.729 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:25.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:25.729 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:25.729 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:32:25.729 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:25.730 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:25.747 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:25.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:25.749 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:25.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:25.752 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:25.753 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:25.759 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:25.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:25.797 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:25.798 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:25.804 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:25.804 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:25.806 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:25.806 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:25.813 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:25.813 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:25.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:32:25.961 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:32:25.966 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:32:25.977 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:32:25.977 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:32:26.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.000+0000 7fee73a00640 1 Processor -- start 2024-03-06T07:32:26.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.000+0000 7fee73a00640 1 -- start start 2024-03-06T07:32:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee73a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fee6c071380 0x7fee6c0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee73a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c06e240 0x7fee6c0ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee73a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0ce9a0 con 0x7fee6c161670 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee73a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c0730d0 con 0x7fee6c071380 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee73a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0cec80 con 0x7fee6c06e240 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee71000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fee6c161670 0x7fee6c070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59748/0 (socket says 172.21.15.32:59748) 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee71000640 1 -- 172.21.15.32:0/1452230057 learned_addr learned my addr 172.21.15.32:0/1452230057 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee71a00640 1 --2- 172.21.15.32:0/1452230057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c06e240 0x7fee6c0ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee63e00640 1 --2- 172.21.15.32:0/1452230057 >> v2:172.21.15.39:3300/0 conn(0x7fee6c071380 0x7fee6c0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee6be00640 1 -- 172.21.15.32:0/1452230057 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211176972 0 0) 0x7fee6c0ce9a0 con 0x7fee6c161670 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee6be00640 1 -- 172.21.15.32:0/1452230057 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee50003520 con 0x7fee6c161670 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee71a00640 1 -- 172.21.15.32:0/1452230057 >> v2:172.21.15.39:3300/0 conn(0x7fee6c071380 msgr2=0x7fee6c0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.002+0000 7fee71a00640 1 --2- 172.21.15.32:0/1452230057 >> v2:172.21.15.39:3300/0 conn(0x7fee6c071380 0x7fee6c0ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.003+0000 7fee71a00640 1 -- 172.21.15.32:0/1452230057 >> v1:172.21.15.32:6789/0 conn(0x7fee6c161670 legacy=0x7fee6c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.003+0000 7fee71a00640 1 -- 172.21.15.32:0/1452230057 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c0cef60 con 0x7fee6c06e240 2024-03-06T07:32:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.004+0000 7fee71a00640 1 --2- 172.21.15.32:0/1452230057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c06e240 0x7fee6c0ad5d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fee640097a0 tx=0x7fee6402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e420e74fc7e4479e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.004+0000 7fee6be00640 1 -- 172.21.15.32:0/1452230057 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640315a0 con 0x7fee6c06e240 2024-03-06T07:32:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.004+0000 7fee6be00640 1 -- 172.21.15.32:0/1452230057 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee64037070 con 0x7fee6c06e240 2024-03-06T07:32:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.004+0000 7fee6be00640 1 -- 172.21.15.32:0/1452230057 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64031960 con 0x7fee6c06e240 2024-03-06T07:32:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 -- 172.21.15.32:0/1452230057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c06e240 msgr2=0x7fee6c0ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 --2- 172.21.15.32:0/1452230057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c06e240 0x7fee6c0ad5d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fee640097a0 tx=0x7fee6402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 -- 172.21.15.32:0/1452230057 shutdown_connections 2024-03-06T07:32:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 --2- 172.21.15.32:0/1452230057 >> v2:172.21.15.39:3300/0 conn(0x7fee6c071380 0x7fee6c0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 --2- 172.21.15.32:0/1452230057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c06e240 0x7fee6c0ad5d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 -- 172.21.15.32:0/1452230057 >> 172.21.15.32:0/1452230057 conn(0x7fee6c06afb0 msgr2=0x7fee6c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 -- 172.21.15.32:0/1452230057 shutdown_connections 2024-03-06T07:32:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.005+0000 7fee73a00640 1 -- 172.21.15.32:0/1452230057 wait complete. 2024-03-06T07:32:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.006+0000 7fee73a00640 1 Processor -- start 2024-03-06T07:32:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 -- start start 2024-03-06T07:32:26.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee6c06e240 0x7fee6c09d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c071380 0x7fee6c0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 0x7fee6c09b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0ce600 con 0x7fee6c161670 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0cf520 con 0x7fee6c06e240 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.007+0000 7fee73a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0ce780 con 0x7fee6c071380 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 0x7fee6c09b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 0x7fee6c09b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46714/0 (socket says 172.21.15.32:46714) 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 -- 172.21.15.32:0/3360260544 learned_addr learned my addr 172.21.15.32:0/3360260544 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee63e00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c071380 0x7fee6c0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71000640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee6c06e240 0x7fee6c09d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 -- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c071380 msgr2=0x7fee6c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c071380 0x7fee6c0cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 -- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee6c06e240 msgr2=0x7fee6c09d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee6c06e240 0x7fee6c09d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71a00640 1 -- 172.21.15.32:0/3360260544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c128020 con 0x7fee6c161670 2024-03-06T07:32:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.008+0000 7fee71000640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee6c06e240 0x7fee6c09d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.009+0000 7fee71a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 0x7fee6c09b870 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fee64034040 tx=0x7fee640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.009+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64032630 con 0x7fee6c161670 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.009+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee640327f0 con 0x7fee6c161670 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.009+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64032be0 con 0x7fee6c161670 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.009+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c09aab0 con 0x7fee6c161670 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.009+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c09b070 con 0x7fee6c161670 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.010+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c069510 con 0x7fee6c161670 2024-03-06T07:32:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.011+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee6404d3f0 con 0x7fee6c161670 2024-03-06T07:32:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.012+0000 7fee69600640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee4806b500 0x7fee4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.012+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fee6402f030 con 0x7fee6c161670 2024-03-06T07:32:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.012+0000 7fee71000640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee4806b500 0x7fee4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.013+0000 7fee71000640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee4806b500 0x7fee4806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fee5c005c00 tx=0x7fee5c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.017+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee64040070 con 0x7fee6c161670 2024-03-06T07:32:26.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.226+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fee6c04f3a0 con 0x7fee6c161670 2024-03-06T07:32:26.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.270+0000 7fee69600640 1 -- 172.21.15.32:0/3360260544 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v62) v1 ==== 97+0+0 (secure 0 0 0) 0x7fee6407aa40 con 0x7fee6c161670 2024-03-06T07:32:26.273 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.274+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee4806b500 msgr2=0x7fee4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.274+0000 7fee73a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee4806b500 0x7fee4806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fee5c005c00 tx=0x7fee5c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.274+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 msgr2=0x7fee6c09b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.274+0000 7fee73a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 0x7fee6c09b870 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fee64034040 tx=0x7fee640046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 shutdown_connections 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee4806b500 0x7fee4806d9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee6c161670 0x7fee6c09b870 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee6c071380 0x7fee6c0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 --2- 172.21.15.32:0/3360260544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee6c06e240 0x7fee6c09d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 >> 172.21.15.32:0/3360260544 conn(0x7fee6c06afb0 msgr2=0x7fee6c0adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 shutdown_connections 2024-03-06T07:32:26.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.275+0000 7fee73a00640 1 -- 172.21.15.32:0/3360260544 wait complete. 2024-03-06T07:32:26.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:32:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.498+0000 7fcafea00640 1 Processor -- start 2024-03-06T07:32:26.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.498+0000 7fcafea00640 1 -- start start 2024-03-06T07:32:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcaf815e070 0x7fcaf80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcaf80d0cd0 0x7fcaf80cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80d18c0 con 0x7fcaf80cf410 2024-03-06T07:32:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf80d1ba0 con 0x7fcaf815e070 2024-03-06T07:32:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafea00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf80d1e80 con 0x7fcaf80d0cd0 2024-03-06T07:32:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafd600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcaf815e070 0x7fcaf80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafd600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcaf815e070 0x7fcaf80d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46724/0 (socket says 172.21.15.32:46724) 2024-03-06T07:32:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.500+0000 7fcafd600640 1 -- 172.21.15.32:0/419594022 learned_addr learned my addr 172.21.15.32:0/419594022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.501+0000 7fcafe000640 1 --2- 172.21.15.32:0/419594022 >> v2:172.21.15.39:3300/0 conn(0x7fcaf80d0cd0 0x7fcaf80cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.501+0000 7fcafd600640 1 -- 172.21.15.32:0/419594022 >> v2:172.21.15.39:3300/0 conn(0x7fcaf80d0cd0 msgr2=0x7fcaf80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.501+0000 7fcafd600640 1 --2- 172.21.15.32:0/419594022 >> v2:172.21.15.39:3300/0 conn(0x7fcaf80d0cd0 0x7fcaf80cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.501+0000 7fcafd600640 1 -- 172.21.15.32:0/419594022 >> v1:172.21.15.39:6789/0 conn(0x7fcaf80cf410 legacy=0x7fcaf80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.501+0000 7fcafd600640 1 -- 172.21.15.32:0/419594022 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80d2160 con 0x7fcaf815e070 2024-03-06T07:32:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.502+0000 7fcafd600640 1 --2- 172.21.15.32:0/419594022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf815e070 0x7fcaf80d2960 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcaf808c1b0 tx=0x7fcaec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb26f88e8a789872 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.503+0000 7fcaf7e00640 1 -- 172.21.15.32:0/419594022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec035020 con 0x7fcaf815e070 2024-03-06T07:32:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.503+0000 7fcaf7e00640 1 -- 172.21.15.32:0/419594022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec031870 con 0x7fcaf815e070 2024-03-06T07:32:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.503+0000 7fcaf7e00640 1 -- 172.21.15.32:0/419594022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec031c40 con 0x7fcaf815e070 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.503+0000 7fcafea00640 1 -- 172.21.15.32:0/419594022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf815e070 msgr2=0x7fcaf80d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.503+0000 7fcafea00640 1 --2- 172.21.15.32:0/419594022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf815e070 0x7fcaf80d2960 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcaf808c1b0 tx=0x7fcaec02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.504+0000 7fcafea00640 1 -- 172.21.15.32:0/419594022 shutdown_connections 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.504+0000 7fcafea00640 1 --2- 172.21.15.32:0/419594022 >> v2:172.21.15.39:3300/0 conn(0x7fcaf80d0cd0 0x7fcaf80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.504+0000 7fcafea00640 1 --2- 172.21.15.32:0/419594022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf815e070 0x7fcaf80d2960 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.504+0000 7fcafea00640 1 -- 172.21.15.32:0/419594022 >> 172.21.15.32:0/419594022 conn(0x7fcaf808d2a0 msgr2=0x7fcaf80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.504+0000 7fcafea00640 1 -- 172.21.15.32:0/419594022 shutdown_connections 2024-03-06T07:32:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.504+0000 7fcafea00640 1 -- 172.21.15.32:0/419594022 wait complete. 2024-03-06T07:32:26.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.505+0000 7fcafea00640 1 Processor -- start 2024-03-06T07:32:26.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.506+0000 7fcafea00640 1 -- start start 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.506+0000 7fcafea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 0x7fcaf81f8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.506+0000 7fcafea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcaf80d0cd0 0x7fcaf815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.506+0000 7fcafea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf815e070 0x7fcaf80c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80d4730 con 0x7fcaf80cf410 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80d45b0 con 0x7fcaf815e070 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80b3630 con 0x7fcaf80d0cd0 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 0x7fcaf81f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 0x7fcaf81f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46726/0 (socket says 172.21.15.32:46726) 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 -- 172.21.15.32:0/3496890742 learned_addr learned my addr 172.21.15.32:0/3496890742 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 -- 172.21.15.32:0/3496890742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcaf80d0cd0 msgr2=0x7fcaf815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcaf80d0cd0 0x7fcaf815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 -- 172.21.15.32:0/3496890742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf815e070 msgr2=0x7fcaf80c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf815e070 0x7fcaf80c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 -- 172.21.15.32:0/3496890742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80ca050 con 0x7fcaf80cf410 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.507+0000 7fcafd600640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 0x7fcaf81f8210 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcaec000c00 tx=0x7fcaec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.508+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec035040 con 0x7fcaf80cf410 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.508+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec030d00 con 0x7fcaf80cf410 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.508+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf80ca330 con 0x7fcaf80cf410 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.508+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0456a0 con 0x7fcaf80cf410 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.508+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf80ca8f0 con 0x7fcaf80cf410 2024-03-06T07:32:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.509+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf80b9bb0 con 0x7fcaf80cf410 2024-03-06T07:32:26.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.510+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcaec038020 con 0x7fcaf80cf410 2024-03-06T07:32:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.511+0000 7fcaf5600640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcad806b4d0 0x7fcad806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.511+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcaec0ac190 con 0x7fcaf80cf410 2024-03-06T07:32:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.511+0000 7fcafcc00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcad806b4d0 0x7fcad806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.512+0000 7fcafcc00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcad806b4d0 0x7fcad806d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcae4003cc0 tx=0x7fcae40079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.515+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcaf80b9bb0 con 0x7fcaf80cf410 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.785+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcaf804f330 con 0x7fcaf80cf410 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.786+0000 7fcaf5600640 1 -- 172.21.15.32:0/3496890742 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcaec07b350 con 0x7fcaf80cf410 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:32:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:32:26.801 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcad806b4d0 msgr2=0x7fcad806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcad806b4d0 0x7fcad806d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcae4003cc0 tx=0x7fcae40079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 msgr2=0x7fcaf81f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 0x7fcaf81f8210 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcaec000c00 tx=0x7fcaec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 shutdown_connections 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcad806b4d0 0x7fcad806d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf815e070 0x7fcaf80c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcaf80d0cd0 0x7fcaf815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 --2- 172.21.15.32:0/3496890742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaf80cf410 0x7fcaf81f8210 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 >> 172.21.15.32:0/3496890742 conn(0x7fcaf808d2a0 msgr2=0x7fcaf806d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 shutdown_connections 2024-03-06T07:32:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:26.789+0000 7fcafea00640 1 -- 172.21.15.32:0/3496890742 wait complete. 2024-03-06T07:32:26.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.018+0000 7f7b9a600640 1 Processor -- start 2024-03-06T07:32:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.019+0000 7f7b9a600640 1 -- start start 2024-03-06T07:32:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b9a600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b9a600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b940d1120 con 0x7f7b9415dec0 2024-03-06T07:32:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b9a600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b940d1400 con 0x7f7b940cf460 2024-03-06T07:32:27.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b9a600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b940b0ea0 con 0x7f7b9415e2a0 2024-03-06T07:32:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b93e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b93e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46732/0 (socket says 172.21.15.32:46732) 2024-03-06T07:32:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b93e00640 1 -- 172.21.15.32:0/1511519884 learned_addr learned my addr 172.21.15.32:0/1511519884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.020+0000 7f7b99200640 1 --1- 172.21.15.32:0/1511519884 >> v1:172.21.15.32:6789/0 conn(0x7f7b9415dec0 0x7f7b940ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59758/0 (socket says 172.21.15.32:59758) 2024-03-06T07:32:27.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.021+0000 7f7b93e00640 1 -- 172.21.15.32:0/1511519884 >> v1:172.21.15.39:6789/0 conn(0x7f7b940cf460 legacy=0x7f7b940b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.021+0000 7f7b93e00640 1 -- 172.21.15.32:0/1511519884 >> v1:172.21.15.32:6789/0 conn(0x7f7b9415dec0 legacy=0x7f7b940ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.021+0000 7f7b93e00640 1 -- 172.21.15.32:0/1511519884 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940b0b30 con 0x7f7b9415e2a0 2024-03-06T07:32:27.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.022+0000 7f7b93e00640 1 --2- 172.21.15.32:0/1511519884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415e2a0 0x7f7b940cef20 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e50 tx=0x7f7b8402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5956a2692828513c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.022+0000 7f7b93400640 1 -- 172.21.15.32:0/1511519884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8402cb10 con 0x7f7b9415e2a0 2024-03-06T07:32:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.022+0000 7f7b93400640 1 -- 172.21.15.32:0/1511519884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8402ccb0 con 0x7f7b9415e2a0 2024-03-06T07:32:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.022+0000 7f7b93400640 1 -- 172.21.15.32:0/1511519884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8402ce60 con 0x7f7b9415e2a0 2024-03-06T07:32:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 -- 172.21.15.32:0/1511519884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415e2a0 msgr2=0x7f7b940cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 --2- 172.21.15.32:0/1511519884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415e2a0 0x7f7b940cef20 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e50 tx=0x7f7b8402bf70 comp rx=0 tx=0).stop 2024-03-06T07:32:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 -- 172.21.15.32:0/1511519884 shutdown_connections 2024-03-06T07:32:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 --2- 172.21.15.32:0/1511519884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 -- 172.21.15.32:0/1511519884 >> 172.21.15.32:0/1511519884 conn(0x7f7b940d3d40 msgr2=0x7f7b940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 -- 172.21.15.32:0/1511519884 shutdown_connections 2024-03-06T07:32:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.023+0000 7f7b9a600640 1 -- 172.21.15.32:0/1511519884 wait complete. 2024-03-06T07:32:27.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.025+0000 7f7b9a600640 1 Processor -- start 2024-03-06T07:32:27.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.025+0000 7f7b9a600640 1 -- start start 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b9a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b940cf460 0x7f7b940ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b9a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 0x7f7b9415b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b9a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b9415e2a0 0x7f7b940c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b9a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940d0ec0 con 0x7f7b9415dec0 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b9a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940d0d40 con 0x7f7b9415e2a0 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b9a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940b1a40 con 0x7f7b940cf460 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 0x7f7b9415b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 0x7f7b9415b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46734/0 (socket says 172.21.15.32:46734) 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 -- 172.21.15.32:0/806476189 learned_addr learned my addr 172.21.15.32:0/806476189 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b99200640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b940cf460 0x7f7b940ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 -- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b940cf460 msgr2=0x7f7b940ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b99c00640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b9415e2a0 0x7f7b940c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b940cf460 0x7f7b940ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 -- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b9415e2a0 msgr2=0x7f7b940c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b9415e2a0 0x7f7b940c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b93e00640 1 -- 172.21.15.32:0/806476189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940c8610 con 0x7f7b9415dec0 2024-03-06T07:32:27.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.026+0000 7f7b99c00640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b9415e2a0 0x7f7b940c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.027+0000 7f7b93e00640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 0x7f7b9415b4e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e50 tx=0x7f7b84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.027+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84043020 con 0x7f7b9415dec0 2024-03-06T07:32:27.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.027+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8403e450 con 0x7f7b9415dec0 2024-03-06T07:32:27.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.027+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8402ce60 con 0x7f7b9415dec0 2024-03-06T07:32:27.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.027+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b940c88f0 con 0x7f7b9415dec0 2024-03-06T07:32:27.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.027+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b940c8eb0 con 0x7f7b9415dec0 2024-03-06T07:32:27.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.028+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9404f330 con 0x7f7b9415dec0 2024-03-06T07:32:27.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.029+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b84046020 con 0x7f7b9415dec0 2024-03-06T07:32:27.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.030+0000 7f7b90c00640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.031+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b840ac700 con 0x7f7b9415dec0 2024-03-06T07:32:27.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.031+0000 7f7b99200640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.032+0000 7f7b99200640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b7006b4d0 0x7f7b7006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c000f60 tx=0x7f7b7c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.035+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b8404d030 con 0x7f7b9415dec0 2024-03-06T07:32:27.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.235+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b9406d790 con 0x7f7b9415dec0 2024-03-06T07:32:27.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.237+0000 7f7b90c00640 1 -- 172.21.15.32:0/806476189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12783 (secure 0 0 0) 0x7f7b8407b950 con 0x7f7b9415dec0 2024-03-06T07:32:27.239 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:27.239 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:26.284293+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b7006b4d0 msgr2=0x7f7b7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b7006b4d0 0x7f7b7006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c000f60 tx=0x7f7b7c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 msgr2=0x7f7b9415b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 0x7f7b9415b4e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e50 tx=0x7f7b84004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 shutdown_connections 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b9415e2a0 0x7f7b940c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.239+0000 7f7b9a600640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b9415dec0 0x7f7b9415b4e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.240+0000 7f7b9a600640 1 --2- 172.21.15.32:0/806476189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b940cf460 0x7f7b940ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.240+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 >> 172.21.15.32:0/806476189 conn(0x7f7b940d3d40 msgr2=0x7f7b940d0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.240+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 shutdown_connections 2024-03-06T07:32:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.240+0000 7f7b9a600640 1 -- 172.21.15.32:0/806476189 wait complete. 2024-03-06T07:32:27.260 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:27.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:27.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.468+0000 7f507ca00640 1 Processor -- start 2024-03-06T07:32:27.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.469+0000 7f507ca00640 1 -- start start 2024-03-06T07:32:27.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.470+0000 7f507ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5078145430 0x7f507815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.470+0000 7f507ca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507815fe40 con 0x7f507815cc90 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.470+0000 7f507ca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5078160120 con 0x7f50780b2250 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.470+0000 7f507ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5078160400 con 0x7f5078145430 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.470+0000 7f5077400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5078145430 0x7f507815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.471+0000 7f5076a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f507815cc90 0x7f50780b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59768/0 (socket says 172.21.15.32:59768) 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.471+0000 7f5077400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5078145430 0x7f507815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46740/0 (socket says 172.21.15.32:46740) 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.471+0000 7f5076a00640 1 -- 172.21.15.32:0/1169532372 learned_addr learned my addr 172.21.15.32:0/1169532372 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.471+0000 7f5077400640 1 -- 172.21.15.32:0/1169532372 >> v1:172.21.15.39:6789/0 conn(0x7f50780b2250 legacy=0x7f50780b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.471+0000 7f5077400640 1 -- 172.21.15.32:0/1169532372 >> v1:172.21.15.32:6789/0 conn(0x7f507815cc90 legacy=0x7f50780b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.471+0000 7f5077400640 1 -- 172.21.15.32:0/1169532372 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50781606e0 con 0x7f5078145430 2024-03-06T07:32:27.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.472+0000 7f5077400640 1 --2- 172.21.15.32:0/1169532372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5078145430 0x7f507815f810 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f506802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596fdec79957de81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.472+0000 7f5076000640 1 -- 172.21.15.32:0/1169532372 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068037070 con 0x7f5078145430 2024-03-06T07:32:27.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f5076000640 1 -- 172.21.15.32:0/1169532372 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068030710 con 0x7f5078145430 2024-03-06T07:32:27.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f5076000640 1 -- 172.21.15.32:0/1169532372 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068030ae0 con 0x7f5078145430 2024-03-06T07:32:27.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 -- 172.21.15.32:0/1169532372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5078145430 msgr2=0x7f507815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 --2- 172.21.15.32:0/1169532372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5078145430 0x7f507815f810 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f506802b710 comp rx=0 tx=0).stop 2024-03-06T07:32:27.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 -- 172.21.15.32:0/1169532372 shutdown_connections 2024-03-06T07:32:27.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 --2- 172.21.15.32:0/1169532372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5078145430 0x7f507815f810 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 -- 172.21.15.32:0/1169532372 >> 172.21.15.32:0/1169532372 conn(0x7f507806b3a0 msgr2=0x7f507806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 -- 172.21.15.32:0/1169532372 shutdown_connections 2024-03-06T07:32:27.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.473+0000 7f507ca00640 1 -- 172.21.15.32:0/1169532372 wait complete. 2024-03-06T07:32:27.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.475+0000 7f507ca00640 1 Processor -- start 2024-03-06T07:32:27.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.475+0000 7f507ca00640 1 -- start start 2024-03-06T07:32:27.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f507ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 0x7f507813ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f507ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5078145430 0x7f507813ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f507ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f507815cc90 0x7f50781e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f507ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5078145990 con 0x7f50780b2250 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f507ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5078145810 con 0x7f507815cc90 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f507ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50780cde00 con 0x7f5078145430 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f5077400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 0x7f507813ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f5077400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 0x7f507813ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46754/0 (socket says 172.21.15.32:46754) 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.476+0000 7f5077400640 1 -- 172.21.15.32:0/2066166094 learned_addr learned my addr 172.21.15.32:0/2066166094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077400640 1 -- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5078145430 msgr2=0x7f507813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5076a00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5078145430 0x7f507813ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077e00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f507815cc90 0x7f50781e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077400640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5078145430 0x7f507813ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077400640 1 -- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f507815cc90 msgr2=0x7f50781e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077400640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f507815cc90 0x7f50781e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077400640 1 -- 172.21.15.32:0/2066166094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507813b680 con 0x7f50780b2250 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077e00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f507815cc90 0x7f50781e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f5077400640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 0x7f507813ea50 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f50680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068042070 con 0x7f50780b2250 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068032a40 con 0x7f50780b2250 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068037040 con 0x7f50780b2250 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.477+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507813b960 con 0x7f50780b2250 2024-03-06T07:32:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.478+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507813bf20 con 0x7f50780b2250 2024-03-06T07:32:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.479+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507804f330 con 0x7f50780b2250 2024-03-06T07:32:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.480+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5068045070 con 0x7f50780b2250 2024-03-06T07:32:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.481+0000 7f506f400640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f505806b4d0 0x7f505806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.481+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f50680abec0 con 0x7f50780b2250 2024-03-06T07:32:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.481+0000 7f5076a00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f505806b4d0 0x7f505806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.482+0000 7f5076a00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f505806b4d0 0x7f505806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5064007fd0 tx=0x7f5064009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.486+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f506807b1a0 con 0x7f50780b2250 2024-03-06T07:32:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.681+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f507815f090 con 0x7f50780b2250 2024-03-06T07:32:27.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.682+0000 7f506f400640 1 -- 172.21.15.32:0/2066166094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f506807aaa0 con 0x7f50780b2250 2024-03-06T07:32:27.684 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:32:27.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f505806b4d0 msgr2=0x7f505806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f505806b4d0 0x7f505806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5064007fd0 tx=0x7f5064009040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 msgr2=0x7f507813ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 0x7f507813ea50 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f50680046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 shutdown_connections 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f505806b4d0 0x7f505806d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f507815cc90 0x7f50781e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5078145430 0x7f507813ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 --2- 172.21.15.32:0/2066166094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50780b2250 0x7f507813ea50 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 >> 172.21.15.32:0/2066166094 conn(0x7f507806b3a0 msgr2=0x7f507815d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.684+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 shutdown_connections 2024-03-06T07:32:27.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.685+0000 7f507ca00640 1 -- 172.21.15.32:0/2066166094 wait complete. 2024-03-06T07:32:27.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:32:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.923+0000 7f15ed000640 1 Processor -- start 2024-03-06T07:32:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.923+0000 7f15ed000640 1 -- start start 2024-03-06T07:32:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.924+0000 7f15ed000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f15e80b0f90 0x7f15e80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15ed000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15ed000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80b1370 con 0x7f15e80cc730 2024-03-06T07:32:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15ed000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e80ccf80 con 0x7f15e80b0f90 2024-03-06T07:32:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15ed000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80cd260 con 0x7f15e80d07b0 2024-03-06T07:32:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15e7e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15e7e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:53900/0 (socket says 172.21.15.32:53900) 2024-03-06T07:32:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.925+0000 7f15e7e00640 1 -- 172.21.15.32:0/2443371408 learned_addr learned my addr 172.21.15.32:0/2443371408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.926+0000 7f15e7400640 1 --2- 172.21.15.32:0/2443371408 >> v2:172.21.15.39:3300/0 conn(0x7f15e80b0f90 0x7f15e80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.926+0000 7f15e7e00640 1 -- 172.21.15.32:0/2443371408 >> v2:172.21.15.39:3300/0 conn(0x7f15e80b0f90 msgr2=0x7f15e80cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.926+0000 7f15e7e00640 1 --2- 172.21.15.32:0/2443371408 >> v2:172.21.15.39:3300/0 conn(0x7f15e80b0f90 0x7f15e80cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.926+0000 7f15e7e00640 1 -- 172.21.15.32:0/2443371408 >> v1:172.21.15.39:6789/0 conn(0x7f15e80cc730 legacy=0x7f15e80d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.926+0000 7f15e7e00640 1 -- 172.21.15.32:0/2443371408 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80cd540 con 0x7f15e80d07b0 2024-03-06T07:32:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.926+0000 7f15e7400640 1 --2- 172.21.15.32:0/2443371408 >> v2:172.21.15.39:3300/0 conn(0x7f15e80b0f90 0x7f15e80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:27.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.928+0000 7f15e7e00640 1 --2- 172.21.15.32:0/2443371408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815d750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009e50 tx=0x7f15e002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c95e84cd134aa94d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.928+0000 7f15e6000640 1 -- 172.21.15.32:0/2443371408 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e002ca70 con 0x7f15e80d07b0 2024-03-06T07:32:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15e6000640 1 -- 172.21.15.32:0/2443371408 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e002cc10 con 0x7f15e80d07b0 2024-03-06T07:32:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15e6000640 1 -- 172.21.15.32:0/2443371408 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e00343f0 con 0x7f15e80d07b0 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 -- 172.21.15.32:0/2443371408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 msgr2=0x7f15e815d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 --2- 172.21.15.32:0/2443371408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815d750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009e50 tx=0x7f15e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 -- 172.21.15.32:0/2443371408 shutdown_connections 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 --2- 172.21.15.32:0/2443371408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815d750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 --2- 172.21.15.32:0/2443371408 >> v2:172.21.15.39:3300/0 conn(0x7f15e80b0f90 0x7f15e80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 -- 172.21.15.32:0/2443371408 >> 172.21.15.32:0/2443371408 conn(0x7f15e80d1ae0 msgr2=0x7f15e80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 -- 172.21.15.32:0/2443371408 shutdown_connections 2024-03-06T07:32:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.929+0000 7f15ed000640 1 -- 172.21.15.32:0/2443371408 wait complete. 2024-03-06T07:32:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.931+0000 7f15ed000640 1 Processor -- start 2024-03-06T07:32:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.931+0000 7f15ed000640 1 -- start start 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.932+0000 7f15ed000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 0x7f15e8160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.932+0000 7f15ed000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15e80cc730 0x7f15e814cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.932+0000 7f15ed000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.932+0000 7f15ed000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80ccd20 con 0x7f15e80b0f90 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.932+0000 7f15ed000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80ccba0 con 0x7f15e80cc730 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.932+0000 7f15ed000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e815dc90 con 0x7f15e80d07b0 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 0x7f15e8160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 0x7f15e8160600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35494/0 (socket says 172.21.15.32:35494) 2024-03-06T07:32:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 -- 172.21.15.32:0/3687564489 learned_addr learned my addr 172.21.15.32:0/3687564489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e6a00640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15e80cc730 0x7f15e814cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7e00640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 -- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 msgr2=0x7f15e815dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 -- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15e80cc730 msgr2=0x7f15e814cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15e80cc730 0x7f15e814cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7400640 1 -- 172.21.15.32:0/3687564489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e814d450 con 0x7f15e80b0f90 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.933+0000 7f15e7e00640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.934+0000 7f15e7400640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 0x7f15e8160600 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f15d800ebe0 tx=0x7f15d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.934+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8017070 con 0x7f15e80b0f90 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.934+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d800cc30 con 0x7f15e80b0f90 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.934+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8013470 con 0x7f15e80b0f90 2024-03-06T07:32:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.934+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e8160be0 con 0x7f15e80b0f90 2024-03-06T07:32:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.934+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e80c6890 con 0x7f15e80b0f90 2024-03-06T07:32:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.935+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e804f330 con 0x7f15e80b0f90 2024-03-06T07:32:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.936+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15d801a070 con 0x7f15e80b0f90 2024-03-06T07:32:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.937+0000 7f15df400640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f15c806b4d0 0x7f15c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.937+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f15d8089ee0 con 0x7f15e80b0f90 2024-03-06T07:32:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.937+0000 7f15e6a00640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f15c806b4d0 0x7f15c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.938+0000 7f15e6a00640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f15c806b4d0 0x7f15c806d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f15e80cb5c0 tx=0x7f15d4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:27.941+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15d8059010 con 0x7f15e80b0f90 2024-03-06T07:32:28.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.142+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f15e80aea50 con 0x7f15e80b0f90 2024-03-06T07:32:28.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.143+0000 7f15df400640 1 -- 172.21.15.32:0/3687564489 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+65 (secure 0 0 0) 0x7f15d8058910 con 0x7f15e80b0f90 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:32:28.146 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.145+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f15c806b4d0 msgr2=0x7f15c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.145+0000 7f15ed000640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f15c806b4d0 0x7f15c806d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f15e80cb5c0 tx=0x7f15d4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.145+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 msgr2=0x7f15e8160600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.145+0000 7f15ed000640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 0x7f15e8160600 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f15d800ebe0 tx=0x7f15d800c520 comp rx=0 tx=0).stop 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 shutdown_connections 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f15c806b4d0 0x7f15c806d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15e80d07b0 0x7f15e815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15e80cc730 0x7f15e814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 --2- 172.21.15.32:0/3687564489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15e80b0f90 0x7f15e8160600 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 >> 172.21.15.32:0/3687564489 conn(0x7f15e80d1ae0 msgr2=0x7f15e80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 shutdown_connections 2024-03-06T07:32:28.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.146+0000 7f15ed000640 1 -- 172.21.15.32:0/3687564489 wait complete. 2024-03-06T07:32:28.167 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:32:28.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:32:28.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.393+0000 7f7c16c00640 1 Processor -- start 2024-03-06T07:32:28.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.393+0000 7f7c16c00640 1 -- start start 2024-03-06T07:32:28.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c16c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c1015e1f0 0x7f7c100d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c16c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c100d23c0 con 0x7f7c100d4660 2024-03-06T07:32:28.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c16c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c100d1050 con 0x7f7c1015e1f0 2024-03-06T07:32:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c16c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100d4c30 con 0x7f7c100d0bc0 2024-03-06T07:32:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c15800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c1015e1f0 0x7f7c100d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c14e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7c100d4660 0x7f7c100b38f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36982/0 (socket says 172.21.15.32:36982) 2024-03-06T07:32:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.395+0000 7f7c14e00640 1 -- 172.21.15.32:0/4235267294 learned_addr learned my addr 172.21.15.32:0/4235267294 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.396+0000 7f7c16200640 1 --2- 172.21.15.32:0/4235267294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.396+0000 7f7c15800640 1 -- 172.21.15.32:0/4235267294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 msgr2=0x7f7c100cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.396+0000 7f7c15800640 1 --2- 172.21.15.32:0/4235267294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.396+0000 7f7c15800640 1 -- 172.21.15.32:0/4235267294 >> v1:172.21.15.32:6789/0 conn(0x7f7c100d4660 legacy=0x7f7c100b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.396+0000 7f7c15800640 1 -- 172.21.15.32:0/4235267294 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100d26a0 con 0x7f7c1015e1f0 2024-03-06T07:32:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.396+0000 7f7c0fe00640 1 -- 172.21.15.32:0/4235267294 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156540850 0 0) 0x7f7c100d23c0 con 0x7f7c100d4660 2024-03-06T07:32:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.397+0000 7f7c15800640 1 --2- 172.21.15.32:0/4235267294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c1015e1f0 0x7f7c100d0680 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7c040097a0 tx=0x7f7c0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e9727d23137058 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.397+0000 7f7c0fe00640 1 -- 172.21.15.32:0/4235267294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040315a0 con 0x7f7c1015e1f0 2024-03-06T07:32:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.397+0000 7f7c0fe00640 1 -- 172.21.15.32:0/4235267294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c04037070 con 0x7f7c1015e1f0 2024-03-06T07:32:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.397+0000 7f7c0fe00640 1 -- 172.21.15.32:0/4235267294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04031960 con 0x7f7c1015e1f0 2024-03-06T07:32:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 -- 172.21.15.32:0/4235267294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c1015e1f0 msgr2=0x7f7c100d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 --2- 172.21.15.32:0/4235267294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c1015e1f0 0x7f7c100d0680 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7c040097a0 tx=0x7f7c0402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 -- 172.21.15.32:0/4235267294 shutdown_connections 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 --2- 172.21.15.32:0/4235267294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 --2- 172.21.15.32:0/4235267294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c1015e1f0 0x7f7c100d0680 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 -- 172.21.15.32:0/4235267294 >> 172.21.15.32:0/4235267294 conn(0x7f7c1009d000 msgr2=0x7f7c100d3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 -- 172.21.15.32:0/4235267294 shutdown_connections 2024-03-06T07:32:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.398+0000 7f7c16c00640 1 -- 172.21.15.32:0/4235267294 wait complete. 2024-03-06T07:32:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.400+0000 7f7c16c00640 1 Processor -- start 2024-03-06T07:32:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.400+0000 7f7c16c00640 1 -- start start 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 0x7f7c10160e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c1015e1f0 0x7f7c101f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c16c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b2da0 con 0x7f7c100d4660 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c16c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100d2c60 con 0x7f7c1015e1f0 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c16c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b4000 con 0x7f7c100d0bc0 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c14e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 0x7f7c10160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c14e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 0x7f7c10160e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35514/0 (socket says 172.21.15.32:35514) 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.401+0000 7f7c14e00640 1 -- 172.21.15.32:0/2095341159 learned_addr learned my addr 172.21.15.32:0/2095341159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c15800640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c16200640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c1015e1f0 0x7f7c101f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c14e00640 1 -- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 msgr2=0x7f7c100c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c14e00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c14e00640 1 -- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c1015e1f0 msgr2=0x7f7c101f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c14e00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c1015e1f0 0x7f7c101f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c14e00640 1 -- 172.21.15.32:0/2095341159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100c8b30 con 0x7f7c100d4660 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.402+0000 7f7c14e00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 0x7f7c10160e00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00c4d0 tx=0x7f7bfc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.403+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc018070 con 0x7f7c100d4660 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.403+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc007500 con 0x7f7c100d4660 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.403+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004a20 con 0x7f7c100d4660 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.403+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c100c8e10 con 0x7f7c100d4660 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.403+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c100c93d0 con 0x7f7c100d4660 2024-03-06T07:32:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.404+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1004f330 con 0x7f7c100d4660 2024-03-06T07:32:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.405+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bfc01b070 con 0x7f7c100d4660 2024-03-06T07:32:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.406+0000 7f7c0d600640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bf006b4d0 0x7f7bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.406+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7bfc0891f0 con 0x7f7c100d4660 2024-03-06T07:32:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.407+0000 7f7c15800640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bf006b4d0 0x7f7bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.407+0000 7f7c15800640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bf006b4d0 0x7f7bf006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7c04036000 tx=0x7f7c0403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.411+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bfc0583c0 con 0x7f7c100d4660 2024-03-06T07:32:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:28.609+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7c100d0250 con 0x7f7c100d4660 2024-03-06T07:32:29.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.301+0000 7f7c0d600640 1 -- 172.21.15.32:0/2095341159 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7bfc05e0c0 con 0x7f7c100d4660 2024-03-06T07:32:29.304 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.305+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bf006b4d0 msgr2=0x7f7bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.305+0000 7f7c16c00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bf006b4d0 0x7f7bf006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7c04036000 tx=0x7f7c0403d040 comp rx=0 tx=0).stop 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.305+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 msgr2=0x7f7c10160e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.305+0000 7f7c16c00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 0x7f7c10160e00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00c4d0 tx=0x7f7bfc00c990 comp rx=0 tx=0).stop 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 shutdown_connections 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bf006b4d0 0x7f7bf006d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c1015e1f0 0x7f7c101f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c100d4660 0x7f7c10160e00 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 --2- 172.21.15.32:0/2095341159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7c100d0bc0 0x7f7c100c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 >> 172.21.15.32:0/2095341159 conn(0x7f7c1009d000 msgr2=0x7f7c1006cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 shutdown_connections 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.306+0000 7f7c16c00640 1 -- 172.21.15.32:0/2095341159 wait complete. 2024-03-06T07:32:29.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:32:29.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.539+0000 7ffb53a00640 1 Processor -- start 2024-03-06T07:32:29.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.539+0000 7ffb53a00640 1 -- start start 2024-03-06T07:32:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb53a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb4c0cee80 0x7ffb4c0cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb53a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb4c15b600 0x7ffb4c15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb53a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c15f840 con 0x7ffb4c0ae710 2024-03-06T07:32:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb53a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c15f3b0 con 0x7ffb4c0cee80 2024-03-06T07:32:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb53a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c0cf7a0 con 0x7ffb4c15b600 2024-03-06T07:32:29.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb51000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb4c0cee80 0x7ffb4c0cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb51000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb4c0cee80 0x7ffb4c0cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35528/0 (socket says 172.21.15.32:35528) 2024-03-06T07:32:29.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.541+0000 7ffb51000640 1 -- 172.21.15.32:0/482361464 learned_addr learned my addr 172.21.15.32:0/482361464 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:29.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.542+0000 7ffb51000640 1 -- 172.21.15.32:0/482361464 >> v2:172.21.15.39:3300/0 conn(0x7ffb4c15b600 msgr2=0x7ffb4c15ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.542+0000 7ffb4be00640 1 --2- 172.21.15.32:0/482361464 >> v2:172.21.15.39:3300/0 conn(0x7ffb4c15b600 0x7ffb4c15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.542+0000 7ffb51000640 1 --2- 172.21.15.32:0/482361464 >> v2:172.21.15.39:3300/0 conn(0x7ffb4c15b600 0x7ffb4c15ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.542+0000 7ffb51000640 1 -- 172.21.15.32:0/482361464 >> v1:172.21.15.39:6789/0 conn(0x7ffb4c0ae710 legacy=0x7ffb4c0aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.542+0000 7ffb51000640 1 -- 172.21.15.32:0/482361464 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c15ea80 con 0x7ffb4c0cee80 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.542+0000 7ffb4be00640 1 --2- 172.21.15.32:0/482361464 >> v2:172.21.15.39:3300/0 conn(0x7ffb4c15b600 0x7ffb4c15ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.543+0000 7ffb51000640 1 --2- 172.21.15.32:0/482361464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c0cee80 0x7ffb4c0cf260 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c09e090 tx=0x7ffb4002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1163e897271e79e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.543+0000 7ffb4b400640 1 -- 172.21.15.32:0/482361464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4002e070 con 0x7ffb4c0cee80 2024-03-06T07:32:29.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.543+0000 7ffb4b400640 1 -- 172.21.15.32:0/482361464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb40037070 con 0x7ffb4c0cee80 2024-03-06T07:32:29.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.543+0000 7ffb4b400640 1 -- 172.21.15.32:0/482361464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40031a00 con 0x7ffb4c0cee80 2024-03-06T07:32:29.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 -- 172.21.15.32:0/482361464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c0cee80 msgr2=0x7ffb4c0cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 --2- 172.21.15.32:0/482361464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c0cee80 0x7ffb4c0cf260 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c09e090 tx=0x7ffb4002b880 comp rx=0 tx=0).stop 2024-03-06T07:32:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 -- 172.21.15.32:0/482361464 shutdown_connections 2024-03-06T07:32:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 --2- 172.21.15.32:0/482361464 >> v2:172.21.15.39:3300/0 conn(0x7ffb4c15b600 0x7ffb4c15ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 --2- 172.21.15.32:0/482361464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c0cee80 0x7ffb4c0cf260 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 -- 172.21.15.32:0/482361464 >> 172.21.15.32:0/482361464 conn(0x7ffb4c06b3d0 msgr2=0x7ffb4c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 -- 172.21.15.32:0/482361464 shutdown_connections 2024-03-06T07:32:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.544+0000 7ffb53a00640 1 -- 172.21.15.32:0/482361464 wait complete. 2024-03-06T07:32:29.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.546+0000 7ffb53a00640 1 Processor -- start 2024-03-06T07:32:29.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.546+0000 7ffb53a00640 1 -- start start 2024-03-06T07:32:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb53a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb4c0ae710 0x7ffb4c149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb53a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb4c0cee80 0x7ffb4c146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb53a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 0x7ffb4c0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb53a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0b3c90 con 0x7ffb4c15b600 2024-03-06T07:32:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb53a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c15f040 con 0x7ffb4c0ae710 2024-03-06T07:32:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb53a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c15bfa0 con 0x7ffb4c0cee80 2024-03-06T07:32:29.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb51a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 0x7ffb4c0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb51a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 0x7ffb4c0b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35540/0 (socket says 172.21.15.32:35540) 2024-03-06T07:32:29.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb51a00640 1 -- 172.21.15.32:0/3315360514 learned_addr learned my addr 172.21.15.32:0/3315360514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.547+0000 7ffb51000640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb4c0ae710 0x7ffb4c149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51a00640 1 -- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb4c0cee80 msgr2=0x7ffb4c146610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb4c0cee80 0x7ffb4c146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51a00640 1 -- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb4c0ae710 msgr2=0x7ffb4c149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb4c0ae710 0x7ffb4c149a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51a00640 1 -- 172.21.15.32:0/3315360514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c149f60 con 0x7ffb4c15b600 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51000640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb4c0ae710 0x7ffb4c149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb51a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 0x7ffb4c0b0d10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400b970 tx=0x7ffb4400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44019070 con 0x7ffb4c15b600 2024-03-06T07:32:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4400cb40 con 0x7ffb4c15b600 2024-03-06T07:32:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44014440 con 0x7ffb4c15b600 2024-03-06T07:32:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c09d0a0 con 0x7ffb4c15b600 2024-03-06T07:32:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.548+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c09d660 con 0x7ffb4c15b600 2024-03-06T07:32:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.550+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c0726c0 con 0x7ffb4c15b600 2024-03-06T07:32:29.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.551+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb4401c070 con 0x7ffb4c15b600 2024-03-06T07:32:29.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.552+0000 7ffb48c00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.552+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ffb44089d00 con 0x7ffb4c15b600 2024-03-06T07:32:29.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.552+0000 7ffb51000640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.553+0000 7ffb51000640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffb40036000 tx=0x7ffb40008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.557+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb4c0726c0 con 0x7ffb4c15b600 2024-03-06T07:32:29.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:29.758+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ffb4c0728d0 con 0x7ffb4c15b600 2024-03-06T07:32:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.308+0000 7ffb48c00640 1 -- 172.21.15.32:0/3315360514 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7ffb4c149f60 con 0x7ffb4c15b600 2024-03-06T07:32:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:32:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb2c06b4d0 msgr2=0x7ffb2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffb40036000 tx=0x7ffb40008030 comp rx=0 tx=0).stop 2024-03-06T07:32:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 msgr2=0x7ffb4c0b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 0x7ffb4c0b0d10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400b970 tx=0x7ffb4400be30 comp rx=0 tx=0).stop 2024-03-06T07:32:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 shutdown_connections 2024-03-06T07:32:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb2c06b4d0 0x7ffb2c06d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb4c15b600 0x7ffb4c0b0d10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb4c0cee80 0x7ffb4c146610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 --2- 172.21.15.32:0/3315360514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb4c0ae710 0x7ffb4c149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 >> 172.21.15.32:0/3315360514 conn(0x7ffb4c06b3d0 msgr2=0x7ffb4c071fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.312+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 shutdown_connections 2024-03-06T07:32:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.313+0000 7ffb53a00640 1 -- 172.21.15.32:0/3315360514 wait complete. 2024-03-06T07:32:30.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:32:30.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.535+0000 7fecfd600640 1 Processor -- start 2024-03-06T07:32:30.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.535+0000 7fecfd600640 1 -- start start 2024-03-06T07:32:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.536+0000 7fecfd600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80d22d0 0x7fecf80d26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecfd600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80d2c30 con 0x7fecf815e1a0 2024-03-06T07:32:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecfd600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80d1d70 con 0x7fecf80b4900 2024-03-06T07:32:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecfd600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80b77f0 con 0x7fecf80d22d0 2024-03-06T07:32:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecf7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fecf815e1a0 0x7fecf80cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36992/0 (socket says 172.21.15.32:36992) 2024-03-06T07:32:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecf7e00640 1 -- 172.21.15.32:0/2095461873 learned_addr learned my addr 172.21.15.32:0/2095461873 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345881726 0 0) 0x7fecf80d2c30 con 0x7fecf815e1a0 2024-03-06T07:32:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecdc003520 con 0x7fecf815e1a0 2024-03-06T07:32:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.537+0000 7fecef400640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80d22d0 0x7fecf80d26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776776821 0 0) 0x7fecdc003520 con 0x7fecf815e1a0 2024-03-06T07:32:30.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80d22d0 msgr2=0x7fecf80d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80d22d0 0x7fecf80d26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 >> v1:172.21.15.39:6789/0 conn(0x7fecf80b4900 legacy=0x7fecf80b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80b6be0 con 0x7fecf815e1a0 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fece8001670 con 0x7fecf815e1a0 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 >> v1:172.21.15.32:6789/0 conn(0x7fecf815e1a0 legacy=0x7fecf80cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecdc0039e0 0x7fecdc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecdc004800 0x7fecdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80b77f0 con 0x7fecdc0051c0 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc00c090 con 0x7fecdc004800 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc00c290 con 0x7fecdc0039e0 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecef400640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80d22d0 0x7fecf80d26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecfcc00640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecf7e00640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecdc0039e0 0x7fecdc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecfcc00640 1 -- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecdc0039e0 msgr2=0x7fecdc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecfcc00640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecdc0039e0 0x7fecdc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecfcc00640 1 -- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecdc004800 msgr2=0x7fecdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecfcc00640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecdc004800 0x7fecdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.538+0000 7fecfcc00640 1 -- 172.21.15.32:0/2095461873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc003520 con 0x7fecdc0051c0 2024-03-06T07:32:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.539+0000 7fecf7e00640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecdc0039e0 0x7fecdc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecfcc00640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecdc00b9c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fecf0001e50 tx=0x7fecdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba41aa5371324d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0001a70 con 0x7fecdc0051c0 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf0003b70 con 0x7fecdc0051c0 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecf7400640 1 -- 172.21.15.32:0/2095461873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf00050d0 con 0x7fecdc0051c0 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecfd600640 1 -- 172.21.15.32:0/2095461873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 msgr2=0x7fecdc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecfd600640 1 --2- 172.21.15.32:0/2095461873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecdc00b9c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fecf0001e50 tx=0x7fecdc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.540+0000 7fecfcc00640 1 -- 172.21.15.32:0/2095461873 reap_dead start 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.541+0000 7fecfd600640 1 -- 172.21.15.32:0/2095461873 shutdown_connections 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.541+0000 7fecfd600640 1 -- 172.21.15.32:0/2095461873 >> 172.21.15.32:0/2095461873 conn(0x7fecf808d2a0 msgr2=0x7fecf80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.541+0000 7fecfd600640 1 -- 172.21.15.32:0/2095461873 shutdown_connections 2024-03-06T07:32:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.541+0000 7fecfd600640 1 -- 172.21.15.32:0/2095461873 wait complete. 2024-03-06T07:32:30.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.542+0000 7fecfd600640 1 Processor -- start 2024-03-06T07:32:30.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.542+0000 7fecfd600640 1 -- start start 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.543+0000 7fecfd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecf80c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.543+0000 7fecfd600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecf814c310 0x7fecf8160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.543+0000 7fecfd600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80b5c90 0x7fecf80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.543+0000 7fecfd600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80b71a0 con 0x7fecdc0051c0 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.543+0000 7fecfd600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf815e580 con 0x7fecf814c310 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.543+0000 7fecfd600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80b7480 con 0x7fecf80b5c90 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecf80c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecf80c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35558/0 (socket says 172.21.15.32:35558) 2024-03-06T07:32:30.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 -- 172.21.15.32:0/1102269086 learned_addr learned my addr 172.21.15.32:0/1102269086 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:30.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecfcc00640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80b5c90 0x7fecf80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 -- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80b5c90 msgr2=0x7fecf80b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80b5c90 0x7fecf80b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 -- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecf814c310 msgr2=0x7fecf8160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecf814c310 0x7fecf8160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.544+0000 7fecf7e00640 1 -- 172.21.15.32:0/1102269086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80cc410 con 0x7fecdc0051c0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.545+0000 7fecf7e00640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecf80c80a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fece800d890 tx=0x7fece800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.545+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8019070 con 0x7fecdc0051c0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.545+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece80074e0 con 0x7fecdc0051c0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.545+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8007690 con 0x7fecdc0051c0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.545+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf80cc6f0 con 0x7fecdc0051c0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.545+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf80cccb0 con 0x7fecdc0051c0 2024-03-06T07:32:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.546+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf804f330 con 0x7fecdc0051c0 2024-03-06T07:32:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.547+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fece801c070 con 0x7fecdc0051c0 2024-03-06T07:32:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.549+0000 7fecf4c00640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fecd406b500 0x7fecd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.549+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fece8089c70 con 0x7fecdc0051c0 2024-03-06T07:32:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.549+0000 7fecef400640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fecd406b500 0x7fecd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.550+0000 7fecef400640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fecd406b500 0x7fecd406d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fece0006fd0 tx=0x7fece0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.554+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fece8058be0 con 0x7fecdc0051c0 2024-03-06T07:32:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:30.797+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fecf80d20b0 con 0x7fecdc0051c0 2024-03-06T07:32:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.332+0000 7fecf4c00640 1 -- 172.21.15.32:0/1102269086 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) v1 ==== 174+0+0 (secure 0 0 0) 0x7fece80584e0 con 0x7fecdc0051c0 2024-03-06T07:32:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.336+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fecd406b500 msgr2=0x7fecd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.336+0000 7fecfd600640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fecd406b500 0x7fecd406d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fece0006fd0 tx=0x7fece0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.336+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 msgr2=0x7fecf80c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.336+0000 7fecfd600640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecf80c80a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fece800d890 tx=0x7fece800dd50 comp rx=0 tx=0).stop 2024-03-06T07:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.337+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 shutdown_connections 2024-03-06T07:32:31.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.337+0000 7fecfd600640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fecd406b500 0x7fecd406d9c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.337+0000 7fecfd600640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fecf80b5c90 0x7fecf80b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.337+0000 7fecfd600640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecf814c310 0x7fecf8160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.337+0000 7fecfd600640 1 --2- 172.21.15.32:0/1102269086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc0051c0 0x7fecf80c80a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.337+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 >> 172.21.15.32:0/1102269086 conn(0x7fecf808d2a0 msgr2=0x7fecf80b4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.338+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 shutdown_connections 2024-03-06T07:32:31.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.338+0000 7fecfd600640 1 -- 172.21.15.32:0/1102269086 wait complete. 2024-03-06T07:32:31.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.566+0000 7f1b63400640 1 Processor -- start 2024-03-06T07:32:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.566+0000 7f1b63400640 1 -- start start 2024-03-06T07:32:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b63400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b63400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1b5c15e0e0 0x7f1b5c15e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b63400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5c161a80 con 0x7f1b5c15dd00 2024-03-06T07:32:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b63400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5c15efa0 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b63400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c15f280 con 0x7f1b5c06c6d0 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b60a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1b5c15dd00 0x7f1b5c06bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:44446/0 (socket says 172.21.15.32:44446) 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b60a00640 1 -- 172.21.15.32:0/3003091983 learned_addr learned my addr 172.21.15.32:0/3003091983 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.568+0000 7f1b53e00640 1 --2- 172.21.15.32:0/3003091983 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b61400640 1 --2- 172.21.15.32:0/3003091983 >> v2:172.21.15.39:3300/0 conn(0x7f1b5c15e0e0 0x7f1b5c15e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b5be00640 1 -- 172.21.15.32:0/3003091983 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432110956 0 0) 0x7f1b5c161a80 con 0x7f1b5c15dd00 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b5be00640 1 -- 172.21.15.32:0/3003091983 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b40003520 con 0x7f1b5c15dd00 2024-03-06T07:32:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b53e00640 1 -- 172.21.15.32:0/3003091983 >> v2:172.21.15.39:3300/0 conn(0x7f1b5c15e0e0 msgr2=0x7f1b5c15e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b53e00640 1 --2- 172.21.15.32:0/3003091983 >> v2:172.21.15.39:3300/0 conn(0x7f1b5c15e0e0 0x7f1b5c15e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b53e00640 1 -- 172.21.15.32:0/3003091983 >> v1:172.21.15.39:6789/0 conn(0x7f1b5c15dd00 legacy=0x7f1b5c06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.569+0000 7f1b53e00640 1 -- 172.21.15.32:0/3003091983 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c161d20 con 0x7f1b5c06c6d0 2024-03-06T07:32:31.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.571+0000 7f1b53e00640 1 --2- 172.21.15.32:0/3003091983 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c162520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1b440097a0 tx=0x7f1b4402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405ada7d181267be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.571+0000 7f1b5be00640 1 -- 172.21.15.32:0/3003091983 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44035020 con 0x7f1b5c06c6d0 2024-03-06T07:32:31.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.571+0000 7f1b5be00640 1 -- 172.21.15.32:0/3003091983 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44031830 con 0x7f1b5c06c6d0 2024-03-06T07:32:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.571+0000 7f1b5be00640 1 -- 172.21.15.32:0/3003091983 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44031c00 con 0x7f1b5c06c6d0 2024-03-06T07:32:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 -- 172.21.15.32:0/3003091983 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 msgr2=0x7f1b5c162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 --2- 172.21.15.32:0/3003091983 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c162520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1b440097a0 tx=0x7f1b4402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 -- 172.21.15.32:0/3003091983 shutdown_connections 2024-03-06T07:32:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 --2- 172.21.15.32:0/3003091983 >> v2:172.21.15.39:3300/0 conn(0x7f1b5c15e0e0 0x7f1b5c15e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 --2- 172.21.15.32:0/3003091983 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c162520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 -- 172.21.15.32:0/3003091983 >> 172.21.15.32:0/3003091983 conn(0x7f1b5c0714d0 msgr2=0x7f1b5c0718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 -- 172.21.15.32:0/3003091983 shutdown_connections 2024-03-06T07:32:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.572+0000 7f1b63400640 1 -- 172.21.15.32:0/3003091983 wait complete. 2024-03-06T07:32:31.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.574+0000 7f1b63400640 1 Processor -- start 2024-03-06T07:32:31.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.574+0000 7f1b63400640 1 -- start start 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b63400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c1569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b63400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b5c15dd00 0x7f1b5c158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b63400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 0x7f1b5c1412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b63400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c15ea70 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b63400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c06f280 con 0x7f1b5c15dd00 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b63400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c15ebf0 con 0x7f1b5c06c6d0 2024-03-06T07:32:31.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 0x7f1b5c1412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 0x7f1b5c1412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35568/0 (socket says 172.21.15.32:35568) 2024-03-06T07:32:31.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b61400640 1 -- 172.21.15.32:0/2376832750 learned_addr learned my addr 172.21.15.32:0/2376832750 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b60a00640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c1569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.575+0000 7f1b53e00640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b5c15dd00 0x7f1b5c158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b61400640 1 -- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 msgr2=0x7f1b5c1569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b61400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c1569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b61400640 1 -- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b5c15dd00 msgr2=0x7f1b5c158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b61400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b5c15dd00 0x7f1b5c158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b61400640 1 -- 172.21.15.32:0/2376832750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c159a40 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b61400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 0x7f1b5c1412d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009d30 tx=0x7f1b5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54018020 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b54014480 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54014870 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.576+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5c159200 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.577+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5c15f7a0 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.578+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5c04f3a0 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.579+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b54005d10 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.579+0000 7f1b59600640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b3806b4d0 0x7f1b3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.580+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b5408ac50 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.580+0000 7f1b60a00640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b3806b4d0 0x7f1b3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.581+0000 7f1b60a00640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b3806b4d0 0x7f1b3806d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005880 tx=0x7f1b4c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.585+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b54022070 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.785+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b5c0cfe10 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.787+0000 7f1b59600640 1 -- 172.21.15.32:0/2376832750 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f1b5c159200 con 0x7f1b5c15e0e0 2024-03-06T07:32:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:31.790 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:30.799113+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b3806b4d0 msgr2=0x7f1b3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b3806b4d0 0x7f1b3806d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005880 tx=0x7f1b4c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 msgr2=0x7f1b5c1412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 0x7f1b5c1412d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009d30 tx=0x7f1b5400c490 comp rx=0 tx=0).stop 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 shutdown_connections 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b3806b4d0 0x7f1b3806d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b5c15e0e0 0x7f1b5c1412d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b5c15dd00 0x7f1b5c158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 --2- 172.21.15.32:0/2376832750 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b5c06c6d0 0x7f1b5c1569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 >> 172.21.15.32:0/2376832750 conn(0x7f1b5c0714d0 msgr2=0x7f1b5c0aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 shutdown_connections 2024-03-06T07:32:31.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:31.790+0000 7f1b63400640 1 -- 172.21.15.32:0/2376832750 wait complete. 2024-03-06T07:32:31.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:32.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.026+0000 7f52cca00640 1 Processor -- start 2024-03-06T07:32:32.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.026+0000 7f52cca00640 1 -- start start 2024-03-06T07:32:32.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.028+0000 7f52cca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f52c81552b0 0x7f52c815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.028+0000 7f52cca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52c815e810 0x7f52c80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.028+0000 7f52cca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c815ed10 con 0x7f52c80cce90 2024-03-06T07:32:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.028+0000 7f52cca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c80cd300 con 0x7f52c815e810 2024-03-06T07:32:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.028+0000 7f52cca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c815eff0 con 0x7f52c81552b0 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52c815e810 0x7f52c80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52c815e810 0x7f52c80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35574/0 (socket says 172.21.15.32:35574) 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 -- 172.21.15.32:0/3038734738 learned_addr learned my addr 172.21.15.32:0/3038734738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52c6800640 1 --2- 172.21.15.32:0/3038734738 >> v2:172.21.15.39:3300/0 conn(0x7f52c81552b0 0x7f52c815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 -- 172.21.15.32:0/3038734738 >> v2:172.21.15.39:3300/0 conn(0x7f52c81552b0 msgr2=0x7f52c815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 --2- 172.21.15.32:0/3038734738 >> v2:172.21.15.39:3300/0 conn(0x7f52c81552b0 0x7f52c815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 -- 172.21.15.32:0/3038734738 >> v1:172.21.15.39:6789/0 conn(0x7f52c80cce90 legacy=0x7f52c806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52bde00640 1 -- 172.21.15.32:0/3038734738 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c815f2d0 con 0x7f52c815e810 2024-03-06T07:32:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.029+0000 7f52c6800640 1 --2- 172.21.15.32:0/3038734738 >> v2:172.21.15.39:3300/0 conn(0x7f52c81552b0 0x7f52c815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:32.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.030+0000 7f52bde00640 1 --2- 172.21.15.32:0/3038734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80cc950 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f52c80cd540 tx=0x7f52b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314bee2f1ad59c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.031+0000 7f52c5e00640 1 -- 172.21.15.32:0/3038734738 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b002ca80 con 0x7f52c815e810 2024-03-06T07:32:32.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.031+0000 7f52c5e00640 1 -- 172.21.15.32:0/3038734738 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b002cc20 con 0x7f52c815e810 2024-03-06T07:32:32.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.031+0000 7f52c5e00640 1 -- 172.21.15.32:0/3038734738 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00343f0 con 0x7f52c815e810 2024-03-06T07:32:32.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.031+0000 7f52cca00640 1 -- 172.21.15.32:0/3038734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 msgr2=0x7f52c80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.031+0000 7f52cca00640 1 --2- 172.21.15.32:0/3038734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80cc950 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f52c80cd540 tx=0x7f52b002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.032+0000 7f52cca00640 1 -- 172.21.15.32:0/3038734738 shutdown_connections 2024-03-06T07:32:32.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.032+0000 7f52cca00640 1 --2- 172.21.15.32:0/3038734738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80cc950 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.032+0000 7f52cca00640 1 --2- 172.21.15.32:0/3038734738 >> v2:172.21.15.39:3300/0 conn(0x7f52c81552b0 0x7f52c815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.032+0000 7f52cca00640 1 -- 172.21.15.32:0/3038734738 >> 172.21.15.32:0/3038734738 conn(0x7f52c806b3d0 msgr2=0x7f52c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.032+0000 7f52cca00640 1 -- 172.21.15.32:0/3038734738 shutdown_connections 2024-03-06T07:32:32.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.032+0000 7f52cca00640 1 -- 172.21.15.32:0/3038734738 wait complete. 2024-03-06T07:32:32.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.034+0000 7f52cca00640 1 Processor -- start 2024-03-06T07:32:32.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.034+0000 7f52cca00640 1 -- start start 2024-03-06T07:32:32.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.035+0000 7f52cca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52c80cce90 0x7f52c8144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.035+0000 7f52cca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c81552b0 0x7f52c80abc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.035+0000 7f52cca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.035+0000 7f52cca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b3760 con 0x7f52c815e810 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.035+0000 7f52cca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8155690 con 0x7f52c81552b0 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.035+0000 7f52cca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c806dd30 con 0x7f52c80cce90 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35590/0 (socket says 172.21.15.32:35590) 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 -- 172.21.15.32:0/415012751 learned_addr learned my addr 172.21.15.32:0/415012751 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c6800640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52c80cce90 0x7f52c8144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52bde00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c81552b0 0x7f52c80abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 -- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52c80cce90 msgr2=0x7f52c8144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52c80cce90 0x7f52c8144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 -- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c81552b0 msgr2=0x7f52c80abc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c81552b0 0x7f52c80abc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 -- 172.21.15.32:0/415012751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80b12f0 con 0x7f52c815e810 2024-03-06T07:32:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52bde00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c81552b0 0x7f52c80abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.036+0000 7f52c7200640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80ac170 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f52c000c390 tx=0x7f52c000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.037+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0017070 con 0x7f52c815e810 2024-03-06T07:32:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.037+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00052d0 con 0x7f52c815e810 2024-03-06T07:32:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.037+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0003810 con 0x7f52c815e810 2024-03-06T07:32:32.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.037+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c80b15d0 con 0x7f52c815e810 2024-03-06T07:32:32.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.037+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c80b1b90 con 0x7f52c815e810 2024-03-06T07:32:32.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.038+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c80adff0 con 0x7f52c815e810 2024-03-06T07:32:32.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.039+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52c001a070 con 0x7f52c815e810 2024-03-06T07:32:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.040+0000 7f52bf400640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f52a406b4d0 0x7f52a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.040+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52c00895b0 con 0x7f52c815e810 2024-03-06T07:32:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.041+0000 7f52c6800640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f52a406b4d0 0x7f52a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.041+0000 7f52c6800640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f52a406b4d0 0x7f52a406d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52b800a990 tx=0x7f52b8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.045+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52c80adff0 con 0x7f52c815e810 2024-03-06T07:32:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.246+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52c804f330 con 0x7f52c815e810 2024-03-06T07:32:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.248+0000 7f52bf400640 1 -- 172.21.15.32:0/415012751 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f52c0058390 con 0x7f52c815e810 2024-03-06T07:32:32.251 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:32.251 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:30.799113+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:32.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.250+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f52a406b4d0 msgr2=0x7f52a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.250+0000 7f52cca00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f52a406b4d0 0x7f52a406d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52b800a990 tx=0x7f52b8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:32.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.250+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 msgr2=0x7f52c80ac170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.250+0000 7f52cca00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80ac170 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f52c000c390 tx=0x7f52c000c850 comp rx=0 tx=0).stop 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 shutdown_connections 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f52a406b4d0 0x7f52a406d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52c815e810 0x7f52c80ac170 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c81552b0 0x7f52c80abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 --2- 172.21.15.32:0/415012751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52c80cce90 0x7f52c8144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 >> 172.21.15.32:0/415012751 conn(0x7f52c806b3d0 msgr2=0x7f52c80b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 shutdown_connections 2024-03-06T07:32:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.251+0000 7f52cca00640 1 -- 172.21.15.32:0/415012751 wait complete. 2024-03-06T07:32:32.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:32:32.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.497+0000 7fad21200640 1 Processor -- start 2024-03-06T07:32:32.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.497+0000 7fad21200640 1 -- start start 2024-03-06T07:32:32.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad21200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 0x7fad1c0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad21200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fad1c0d2320 0x7fad1c0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad21200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b2c00 con 0x7fad1c161700 2024-03-06T07:32:32.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad21200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c0b56e0 con 0x7fad1c0d2320 2024-03-06T07:32:32.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad21200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c161ae0 con 0x7fad1c0d4890 2024-03-06T07:32:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad1b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fad1c161700 0x7fad1c0b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37000/0 (socket says 172.21.15.32:37000) 2024-03-06T07:32:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad1b400640 1 -- 172.21.15.32:0/3170172752 learned_addr learned my addr 172.21.15.32:0/3170172752 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad1aa00640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 0x7fad1c0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad1be00640 1 --2- 172.21.15.32:0/3170172752 >> v2:172.21.15.39:3300/0 conn(0x7fad1c0d2320 0x7fad1c0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.499+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258754755 0 0) 0x7fad1c0b2c00 con 0x7fad1c161700 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad00003520 con 0x7fad1c161700 2024-03-06T07:32:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081350027 0 0) 0x7fad00003520 con 0x7fad1c161700 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 msgr2=0x7fad1c0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 0x7fad1c0d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 >> v2:172.21.15.39:3300/0 conn(0x7fad1c0d2320 msgr2=0x7fad1c0b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 --2- 172.21.15.32:0/3170172752 >> v2:172.21.15.39:3300/0 conn(0x7fad1c0d2320 0x7fad1c0b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0b2ea0 con 0x7fad1c161700 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fad0c001670 con 0x7fad1c161700 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 >> v1:172.21.15.32:6789/0 conn(0x7fad1c161700 legacy=0x7fad1c0b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 0x7fad00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad00004d40 0x7fad000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0b56e0 con 0x7fad000039b0 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c161ae0 con 0x7fad00004d40 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0000b9c0 con 0x7fad1c0d4890 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.500+0000 7fad1b400640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1be00640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad00004d40 0x7fad000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1aa00640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 0x7fad00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1b400640 1 -- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 msgr2=0x7fad00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1b400640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c0d4890 0x7fad00004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1b400640 1 -- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad00004d40 msgr2=0x7fad000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1b400640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad00004d40 0x7fad000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.501+0000 7fad1b400640 1 -- 172.21.15.32:0/3170172752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00003520 con 0x7fad000039b0 2024-03-06T07:32:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.502+0000 7fad1b400640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad00003dd0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c240 tx=0x7fad00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dabc9fcdc0736ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.502+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c007d50 con 0x7fad000039b0 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.502+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c030070 con 0x7fad000039b0 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.502+0000 7fad1a000640 1 -- 172.21.15.32:0/3170172752 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c02cd80 con 0x7fad000039b0 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad21200640 1 -- 172.21.15.32:0/3170172752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 msgr2=0x7fad00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad21200640 1 --2- 172.21.15.32:0/3170172752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad00003dd0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c240 tx=0x7fad00005830 comp rx=0 tx=0).stop 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad1be00640 1 -- 172.21.15.32:0/3170172752 reap_dead start 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad21200640 1 -- 172.21.15.32:0/3170172752 shutdown_connections 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad21200640 1 -- 172.21.15.32:0/3170172752 >> 172.21.15.32:0/3170172752 conn(0x7fad1c0d3d70 msgr2=0x7fad1c0d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad21200640 1 -- 172.21.15.32:0/3170172752 shutdown_connections 2024-03-06T07:32:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.503+0000 7fad21200640 1 -- 172.21.15.32:0/3170172752 wait complete. 2024-03-06T07:32:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.505+0000 7fad21200640 1 Processor -- start 2024-03-06T07:32:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.505+0000 7fad21200640 1 -- start start 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.506+0000 7fad21200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad1c0ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.506+0000 7fad21200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c1f3ba0 0x7fad1c1f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.506+0000 7fad21200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1c0ceef0 0x7fad1c0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.506+0000 7fad21200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0b0d00 con 0x7fad000039b0 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.506+0000 7fad21200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0b0a20 con 0x7fad1c0ceef0 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.506+0000 7fad21200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0d1de0 con 0x7fad1c1f3ba0 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad1c0ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad1c0ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35602/0 (socket says 172.21.15.32:35602) 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 -- 172.21.15.32:0/2957550191 learned_addr learned my addr 172.21.15.32:0/2957550191 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1aa00640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c1f3ba0 0x7fad1c1f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1be00640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1c0ceef0 0x7fad1c0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 -- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c1f3ba0 msgr2=0x7fad1c1f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c1f3ba0 0x7fad1c1f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 -- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1c0ceef0 msgr2=0x7fad1c0c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1c0ceef0 0x7fad1c0c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.507+0000 7fad1b400640 1 -- 172.21.15.32:0/2957550191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0ce710 con 0x7fad000039b0 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.508+0000 7fad1b400640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad1c0ccb00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c240 tx=0x7fad0c001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.508+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c001c90 con 0x7fad000039b0 2024-03-06T07:32:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.508+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c004bf0 con 0x7fad000039b0 2024-03-06T07:32:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.508+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c1f43d0 con 0x7fad000039b0 2024-03-06T07:32:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.508+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c007070 con 0x7fad000039b0 2024-03-06T07:32:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.508+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c1f4990 con 0x7fad000039b0 2024-03-06T07:32:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.509+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c04f330 con 0x7fad000039b0 2024-03-06T07:32:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.510+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad0c048020 con 0x7fad000039b0 2024-03-06T07:32:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.511+0000 7fad13400640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b500 0x7facfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.511+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad0c0ab830 con 0x7fad000039b0 2024-03-06T07:32:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.511+0000 7fad1aa00640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b500 0x7facfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.512+0000 7fad1aa00640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b500 0x7facfc06d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0b2650 tx=0x7fad08008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.516+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad0c07a610 con 0x7fad000039b0 2024-03-06T07:32:32.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:32.757+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fad1c160900 con 0x7fad000039b0 2024-03-06T07:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.385+0000 7fad13400640 1 -- 172.21.15.32:0/2957550191 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) v1 ==== 118+0+0 (secure 0 0 0) 0x7fad0c079f10 con 0x7fad000039b0 2024-03-06T07:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.389+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b500 msgr2=0x7facfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.389+0000 7fad21200640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b500 0x7facfc06d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0b2650 tx=0x7fad08008040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.389+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 msgr2=0x7fad1c0ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.389+0000 7fad21200640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad1c0ccb00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00c240 tx=0x7fad0c001ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 shutdown_connections 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b500 0x7facfc06d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad1c0ceef0 0x7fad1c0c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad1c1f3ba0 0x7fad1c1f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 --2- 172.21.15.32:0/2957550191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad000039b0 0x7fad1c0ccb00 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 >> 172.21.15.32:0/2957550191 conn(0x7fad1c0d3d70 msgr2=0x7fad1c15ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 shutdown_connections 2024-03-06T07:32:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.390+0000 7fad21200640 1 -- 172.21.15.32:0/2957550191 wait complete. 2024-03-06T07:32:33.413 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:32:33.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:32:33.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.643+0000 7f4935e00640 1 Processor -- start 2024-03-06T07:32:33.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.643+0000 7f4935e00640 1 -- start start 2024-03-06T07:32:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.644+0000 7f4935e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49300b2250 0x7f4930168950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493006de40 con 0x7f4930147590 2024-03-06T07:32:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493006e120 con 0x7f493006d8d0 2024-03-06T07:32:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4930147970 con 0x7f49300b2250 2024-03-06T07:32:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49300b2250 0x7f4930168950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49300b2250 0x7f4930168950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35610/0 (socket says 172.21.15.32:35610) 2024-03-06T07:32:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935400640 1 -- 172.21.15.32:0/2546823226 learned_addr learned my addr 172.21.15.32:0/2546823226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4934a00640 1 --1- 172.21.15.32:0/2546823226 >> v1:172.21.15.32:6789/0 conn(0x7f4930147590 0x7f4930071480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37008/0 (socket says 172.21.15.32:37008) 2024-03-06T07:32:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935400640 1 -- 172.21.15.32:0/2546823226 >> v1:172.21.15.39:6789/0 conn(0x7f493006d8d0 legacy=0x7f49300b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.645+0000 7f4935400640 1 -- 172.21.15.32:0/2546823226 >> v1:172.21.15.32:6789/0 conn(0x7f4930147590 legacy=0x7f4930071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.646+0000 7f4935400640 1 -- 172.21.15.32:0/2546823226 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300b2e40 con 0x7f49300b2250 2024-03-06T07:32:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.646+0000 7f492f400640 1 -- 172.21.15.32:0/2546823226 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242390718 0 0) 0x7f493006de40 con 0x7f4930147590 2024-03-06T07:32:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.647+0000 7f4935400640 1 --2- 172.21.15.32:0/2546823226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49300b2250 0x7f4930168950 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4928009e50 tx=0x7f492802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c581138524500e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.647+0000 7f492f400640 1 -- 172.21.15.32:0/2546823226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492802cb20 con 0x7f49300b2250 2024-03-06T07:32:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.647+0000 7f492f400640 1 -- 172.21.15.32:0/2546823226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492802ccc0 con 0x7f49300b2250 2024-03-06T07:32:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.647+0000 7f492f400640 1 -- 172.21.15.32:0/2546823226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928034610 con 0x7f49300b2250 2024-03-06T07:32:33.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.647+0000 7f4935e00640 1 -- 172.21.15.32:0/2546823226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49300b2250 msgr2=0x7f4930168950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.647+0000 7f4935e00640 1 --2- 172.21.15.32:0/2546823226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49300b2250 0x7f4930168950 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4928009e50 tx=0x7f492802bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:33.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.648+0000 7f4935e00640 1 -- 172.21.15.32:0/2546823226 shutdown_connections 2024-03-06T07:32:33.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.648+0000 7f4935e00640 1 --2- 172.21.15.32:0/2546823226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49300b2250 0x7f4930168950 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.648+0000 7f4935e00640 1 -- 172.21.15.32:0/2546823226 >> 172.21.15.32:0/2546823226 conn(0x7f493006b3a0 msgr2=0x7f493006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.648+0000 7f4935e00640 1 -- 172.21.15.32:0/2546823226 shutdown_connections 2024-03-06T07:32:33.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.648+0000 7f4935e00640 1 -- 172.21.15.32:0/2546823226 wait complete. 2024-03-06T07:32:33.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.649+0000 7f4935e00640 1 Processor -- start 2024-03-06T07:32:33.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 -- start start 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 0x7f4930160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49300b2250 0x7f49301608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4930147590 0x7f49300c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b4650 con 0x7f493006d8d0 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b31b0 con 0x7f4930147590 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.650+0000 7f4935e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b47d0 con 0x7f49300b2250 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 0x7f4930160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 0x7f4930160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35612/0 (socket says 172.21.15.32:35612) 2024-03-06T07:32:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 -- 172.21.15.32:0/2123234183 learned_addr learned my addr 172.21.15.32:0/2123234183 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 -- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49300b2250 msgr2=0x7f49301608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49300b2250 0x7f49301608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4935400640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4930147590 0x7f49300c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 -- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4930147590 msgr2=0x7f49300c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4930147590 0x7f49300c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.651+0000 7f4934a00640 1 -- 172.21.15.32:0/2123234183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300c7550 con 0x7f493006d8d0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.652+0000 7f4934a00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 0x7f4930160360 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f492400c460 tx=0x7f492400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.652+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924018070 con 0x7f493006d8d0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.652+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924007500 con 0x7f493006d8d0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.652+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924004a20 con 0x7f493006d8d0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.652+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49300c7830 con 0x7f493006d8d0 2024-03-06T07:32:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.652+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49300c7df0 con 0x7f493006d8d0 2024-03-06T07:32:33.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.654+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493006cea0 con 0x7f493006d8d0 2024-03-06T07:32:33.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.654+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4924004bc0 con 0x7f493006d8d0 2024-03-06T07:32:33.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.655+0000 7f492cc00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f491006b4d0 0x7f491006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.655+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4924089520 con 0x7f493006d8d0 2024-03-06T07:32:33.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.655+0000 7f492fe00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f491006b4d0 0x7f491006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.656+0000 7f492fe00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f491006b4d0 0x7f491006d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f49300c8990 tx=0x7f491c0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.660+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f493006cea0 con 0x7f493006d8d0 2024-03-06T07:32:33.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:33.902+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f493006b780 con 0x7f493006d8d0 2024-03-06T07:32:34.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.388+0000 7f492cc00640 1 -- 172.21.15.32:0/2123234183 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f493006b780 con 0x7f493006d8d0 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.392+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f491006b4d0 msgr2=0x7f491006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.392+0000 7f4935e00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f491006b4d0 0x7f491006d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f49300c8990 tx=0x7f491c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.392+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 msgr2=0x7f4930160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.392+0000 7f4935e00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 0x7f4930160360 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f492400c460 tx=0x7f492400c920 comp rx=0 tx=0).stop 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 shutdown_connections 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f491006b4d0 0x7f491006d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4930147590 0x7f49300c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f49300b2250 0x7f49301608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 --2- 172.21.15.32:0/2123234183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f493006d8d0 0x7f4930160360 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 >> 172.21.15.32:0/2123234183 conn(0x7f493006b3a0 msgr2=0x7f49300cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 shutdown_connections 2024-03-06T07:32:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.393+0000 7f4935e00640 1 -- 172.21.15.32:0/2123234183 wait complete. 2024-03-06T07:32:34.422 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:32:34.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:34.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.629+0000 7f55b7200640 1 Processor -- start 2024-03-06T07:32:34.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.630+0000 7f55b7200640 1 -- start start 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.631+0000 7f55b7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.631+0000 7f55b7200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55b80d4a10 0x7f55b80d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.631+0000 7f55b7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f55b80d0c80 0x7f55b80d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.631+0000 7f55b7200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b80b09e0 con 0x7f55b80d4a10 2024-03-06T07:32:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.631+0000 7f55b7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b80b3710 con 0x7f55b80d0c80 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.631+0000 7f55b7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80b39f0 con 0x7f55b815e0b0 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55b80d4a10 0x7f55b80d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55b80d4a10 0x7f55b80d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35622/0 (socket says 172.21.15.32:35622) 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 -- 172.21.15.32:0/177985191 learned_addr learned my addr 172.21.15.32:0/177985191 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55b5e00640 1 --2- 172.21.15.32:0/177985191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55b6800640 1 --2- 172.21.15.32:0/177985191 >> v2:172.21.15.39:3300/0 conn(0x7f55b80d0c80 0x7f55b80d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 -- 172.21.15.32:0/177985191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 msgr2=0x7f55b80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 --2- 172.21.15.32:0/177985191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b80cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 -- 172.21.15.32:0/177985191 >> v2:172.21.15.39:3300/0 conn(0x7f55b80d0c80 msgr2=0x7f55b80d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 --2- 172.21.15.32:0/177985191 >> v2:172.21.15.39:3300/0 conn(0x7f55b80d0c80 0x7f55b80d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55afe00640 1 -- 172.21.15.32:0/177985191 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80b3cd0 con 0x7f55b80d4a10 2024-03-06T07:32:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.632+0000 7f55b6800640 1 --2- 172.21.15.32:0/177985191 >> v2:172.21.15.39:3300/0 conn(0x7f55b80d0c80 0x7f55b80d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.634+0000 7f55afe00640 1 --2- 172.21.15.32:0/177985191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b80d0740 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f55b80d06c0 tx=0x7f55a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c58e4f58f19eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.634+0000 7f55b5400640 1 -- 172.21.15.32:0/177985191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0035020 con 0x7f55b80d4a10 2024-03-06T07:32:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.634+0000 7f55b5400640 1 -- 172.21.15.32:0/177985191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a0031870 con 0x7f55b80d4a10 2024-03-06T07:32:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.634+0000 7f55b5400640 1 -- 172.21.15.32:0/177985191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0031c40 con 0x7f55b80d4a10 2024-03-06T07:32:34.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.634+0000 7f55b7200640 1 -- 172.21.15.32:0/177985191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 msgr2=0x7f55b80d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.634+0000 7f55b7200640 1 --2- 172.21.15.32:0/177985191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b80d0740 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f55b80d06c0 tx=0x7f55a002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 -- 172.21.15.32:0/177985191 shutdown_connections 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 --2- 172.21.15.32:0/177985191 >> v2:172.21.15.39:3300/0 conn(0x7f55b80d0c80 0x7f55b80d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 --2- 172.21.15.32:0/177985191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b80d0740 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 --2- 172.21.15.32:0/177985191 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 -- 172.21.15.32:0/177985191 >> 172.21.15.32:0/177985191 conn(0x7f55b80d3d30 msgr2=0x7f55b80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 -- 172.21.15.32:0/177985191 shutdown_connections 2024-03-06T07:32:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.635+0000 7f55b7200640 1 -- 172.21.15.32:0/177985191 wait complete. 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.637+0000 7f55b7200640 1 Processor -- start 2024-03-06T07:32:34.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.637+0000 7f55b7200640 1 -- start start 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55b80d0c80 0x7f55b80ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b814b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b814b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80cfc60 con 0x7f55b80d4a10 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b815e490 con 0x7f55b80d0c80 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80cf850 con 0x7f55b815e0b0 2024-03-06T07:32:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b814b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b814b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35632/0 (socket says 172.21.15.32:35632) 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55afe00640 1 -- 172.21.15.32:0/3681736486 learned_addr learned my addr 172.21.15.32:0/3681736486 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.638+0000 7f55b5e00640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55b80d0c80 0x7f55b80ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55b6800640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b814b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55afe00640 1 -- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 msgr2=0x7f55b814b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55afe00640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b814b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55afe00640 1 -- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55b80d0c80 msgr2=0x7f55b80ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55afe00640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55b80d0c80 0x7f55b80ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55afe00640 1 -- 172.21.15.32:0/3681736486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80ca780 con 0x7f55b80d4a10 2024-03-06T07:32:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55afe00640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b814b080 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0035040 con 0x7f55b80d4a10 2024-03-06T07:32:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a0045480 con 0x7f55b80d4a10 2024-03-06T07:32:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0045870 con 0x7f55b80d4a10 2024-03-06T07:32:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.639+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b814bbf0 con 0x7f55b80d4a10 2024-03-06T07:32:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.640+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b814c1b0 con 0x7f55b80d4a10 2024-03-06T07:32:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.641+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b804f330 con 0x7f55b80d4a10 2024-03-06T07:32:34.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.642+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55a0038020 con 0x7f55b80d4a10 2024-03-06T07:32:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.643+0000 7f55ae000640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f559406b4d0 0x7f559406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.643+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55a00ac630 con 0x7f55b80d4a10 2024-03-06T07:32:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.643+0000 7f55b5e00640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f559406b4d0 0x7f559406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.644+0000 7f55b5e00640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f559406b4d0 0x7f559406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f55a8006fd0 tx=0x7f55a8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.647+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55a007b540 con 0x7f55b80d4a10 2024-03-06T07:32:34.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.929+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b806c2e0 con 0x7f55b80d4a10 2024-03-06T07:32:34.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.930+0000 7f55ae000640 1 -- 172.21.15.32:0/3681736486 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f55a007ae40 con 0x7f55b80d4a10 2024-03-06T07:32:34.932 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:34.932 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:32:34.933 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:32:34.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f559406b4d0 msgr2=0x7f559406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f559406b4d0 0x7f559406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f55a8006fd0 tx=0x7f55a8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 msgr2=0x7f55b814b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b814b080 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 shutdown_connections 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f559406b4d0 0x7f559406d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55b815e0b0 0x7f55b814b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b80d4a10 0x7f55b814b080 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 --2- 172.21.15.32:0/3681736486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55b80d0c80 0x7f55b80ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 >> 172.21.15.32:0/3681736486 conn(0x7f55b80d3d30 msgr2=0x7f55b8160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 shutdown_connections 2024-03-06T07:32:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:34.934+0000 7f55b7200640 1 -- 172.21.15.32:0/3681736486 wait complete. 2024-03-06T07:32:34.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.174+0000 7fa0fb000640 1 Processor -- start 2024-03-06T07:32:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.174+0000 7fa0fb000640 1 -- start start 2024-03-06T07:32:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0fb000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa0f40b01f0 0x7fa0f40adc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0fb000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f415e3f0 con 0x7fa0f406e640 2024-03-06T07:32:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0fb000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f406eb20 con 0x7fa0f415e010 2024-03-06T07:32:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0fb000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f406e440 con 0x7fa0f40b01f0 2024-03-06T07:32:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0f9000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0f406e640 0x7fa0f40ae160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37010/0 (socket says 172.21.15.32:37010) 2024-03-06T07:32:35.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0f9000640 1 -- 172.21.15.32:0/483212033 learned_addr learned my addr 172.21.15.32:0/483212033 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:35.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.177+0000 7fa0ebe00640 1 --2- 172.21.15.32:0/483212033 >> v2:172.21.15.32:3300/0 conn(0x7fa0f40b01f0 0x7fa0f40adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.178+0000 7fa0f3400640 1 -- 172.21.15.32:0/483212033 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099251042 0 0) 0x7fa0f415e3f0 con 0x7fa0f406e640 2024-03-06T07:32:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.178+0000 7fa0f3400640 1 -- 172.21.15.32:0/483212033 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d8003520 con 0x7fa0f406e640 2024-03-06T07:32:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.178+0000 7fa0ebe00640 1 -- 172.21.15.32:0/483212033 >> v1:172.21.15.39:6789/0 conn(0x7fa0f415e010 legacy=0x7fa0f40afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.178+0000 7fa0f3400640 1 -- 172.21.15.32:0/483212033 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898721139 0 0) 0x7fa0d8003520 con 0x7fa0f406e640 2024-03-06T07:32:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.178+0000 7fa0ebe00640 1 -- 172.21.15.32:0/483212033 >> v1:172.21.15.32:6789/0 conn(0x7fa0f406e640 legacy=0x7fa0f40ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.178+0000 7fa0ebe00640 1 -- 172.21.15.32:0/483212033 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f406de80 con 0x7fa0f40b01f0 2024-03-06T07:32:35.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.181+0000 7fa0ebe00640 1 --2- 172.21.15.32:0/483212033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40b01f0 0x7fa0f40adc20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0097a0 tx=0x7fa0dc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3738b23bf998f549 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.181+0000 7fa0f3400640 1 -- 172.21.15.32:0/483212033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc038070 con 0x7fa0f40b01f0 2024-03-06T07:32:35.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.181+0000 7fa0f3400640 1 -- 172.21.15.32:0/483212033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc031500 con 0x7fa0f40b01f0 2024-03-06T07:32:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.181+0000 7fa0f3400640 1 -- 172.21.15.32:0/483212033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec0019f0 con 0x7fa0f40b01f0 2024-03-06T07:32:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 -- 172.21.15.32:0/483212033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40b01f0 msgr2=0x7fa0f40adc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 --2- 172.21.15.32:0/483212033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40b01f0 0x7fa0f40adc20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0097a0 tx=0x7fa0dc02b820 comp rx=0 tx=0).stop 2024-03-06T07:32:35.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 -- 172.21.15.32:0/483212033 shutdown_connections 2024-03-06T07:32:35.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 --2- 172.21.15.32:0/483212033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40b01f0 0x7fa0f40adc20 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 -- 172.21.15.32:0/483212033 >> 172.21.15.32:0/483212033 conn(0x7fa0f40d0bb0 msgr2=0x7fa0f40d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 -- 172.21.15.32:0/483212033 shutdown_connections 2024-03-06T07:32:35.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.182+0000 7fa0fb000640 1 -- 172.21.15.32:0/483212033 wait complete. 2024-03-06T07:32:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.185+0000 7fa0fb000640 1 Processor -- start 2024-03-06T07:32:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.185+0000 7fa0fb000640 1 -- start start 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.187+0000 7fa0fb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 0x7fa0f40ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.187+0000 7fa0fb000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f415e010 0x7fa0f4157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.187+0000 7fa0fb000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa0f40b47f0 0x7fa0f40b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.187+0000 7fa0fb000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f406dc30 con 0x7fa0f406e640 2024-03-06T07:32:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.187+0000 7fa0fb000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40704d0 con 0x7fa0f415e010 2024-03-06T07:32:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.187+0000 7fa0fb000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4070650 con 0x7fa0f40b47f0 2024-03-06T07:32:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 0x7fa0f40ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f415e010 0x7fa0f4157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 0x7fa0f40ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35646/0 (socket says 172.21.15.32:35646) 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 -- 172.21.15.32:0/654998657 learned_addr learned my addr 172.21.15.32:0/654998657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f9000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa0f40b47f0 0x7fa0f40b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 -- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa0f40b47f0 msgr2=0x7fa0f40b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa0f40b47f0 0x7fa0f40b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 -- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f415e010 msgr2=0x7fa0f4157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f415e010 0x7fa0f4157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.188+0000 7fa0f3e00640 1 -- 172.21.15.32:0/654998657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f41e91c0 con 0x7fa0f406e640 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.189+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 0x7fa0f40ccf70 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa0e400c400 tx=0x7fa0e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.189+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4017070 con 0x7fa0f406e640 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.189+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e40052d0 con 0x7fa0f406e640 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.189+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f40cd4b0 con 0x7fa0f406e640 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.189+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4003830 con 0x7fa0f406e640 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.189+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f4157570 con 0x7fa0f406e640 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.190+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f404f3a0 con 0x7fa0f406e640 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.191+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0e401a070 con 0x7fa0f406e640 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.193+0000 7fa0f0c00640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa0d006b550 0x7fa0d006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.193+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0e40894e0 con 0x7fa0f406e640 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.193+0000 7fa0ebe00640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa0d006b550 0x7fa0d006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.194+0000 7fa0ebe00640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa0d006b550 0x7fa0d006da10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc035000 tx=0x7fa0dc008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.198+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0e4057cf0 con 0x7fa0f406e640 2024-03-06T07:32:35.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.398+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0f4160520 con 0x7fa0f406e640 2024-03-06T07:32:35.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.399+0000 7fa0f0c00640 1 -- 172.21.15.32:0/654998657 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16560 (secure 0 0 0) 0x7fa0e405bd90 con 0x7fa0f406e640 2024-03-06T07:32:35.402 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:35.402 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":68,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:32.333541+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:35.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa0d006b550 msgr2=0x7fa0d006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa0d006b550 0x7fa0d006da10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc035000 tx=0x7fa0dc008030 comp rx=0 tx=0).stop 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 msgr2=0x7fa0f40ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 0x7fa0f40ccf70 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa0e400c400 tx=0x7fa0e400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 shutdown_connections 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa0d006b550 0x7fa0d006da10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa0f40b47f0 0x7fa0f40b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f415e010 0x7fa0f4157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 --2- 172.21.15.32:0/654998657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f406e640 0x7fa0f40ccf70 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.402+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 >> 172.21.15.32:0/654998657 conn(0x7fa0f40d0bb0 msgr2=0x7fa0f415b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.403+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 shutdown_connections 2024-03-06T07:32:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.403+0000 7fa0fb000640 1 -- 172.21.15.32:0/654998657 wait complete. 2024-03-06T07:32:35.422 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:32:35.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.629+0000 7f2a46400640 1 Processor -- start 2024-03-06T07:32:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.630+0000 7f2a46400640 1 -- start start 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.631+0000 7f2a46400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a400b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.631+0000 7f2a46400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2a4009b120 0x7f2a400b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.631+0000 7f2a46400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a4015c210 0x7f2a4015fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.631+0000 7f2a46400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a400b65a0 con 0x7f2a4015c210 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.631+0000 7f2a46400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a400b27a0 con 0x7f2a4009b120 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.631+0000 7f2a46400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400b2a80 con 0x7f2a4015be30 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a4015c210 0x7f2a4015fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a4015c210 0x7f2a4015fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35656/0 (socket says 172.21.15.32:35656) 2024-03-06T07:32:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 -- 172.21.15.32:0/3388272972 learned_addr learned my addr 172.21.15.32:0/3388272972 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:35.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3f400640 1 --2- 172.21.15.32:0/3388272972 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a400b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a37e00640 1 --2- 172.21.15.32:0/3388272972 >> v2:172.21.15.39:3300/0 conn(0x7f2a4009b120 0x7f2a400b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 -- 172.21.15.32:0/3388272972 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 msgr2=0x7f2a400b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 --2- 172.21.15.32:0/3388272972 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a400b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 -- 172.21.15.32:0/3388272972 >> v2:172.21.15.39:3300/0 conn(0x7f2a4009b120 msgr2=0x7f2a400b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 --2- 172.21.15.32:0/3388272972 >> v2:172.21.15.39:3300/0 conn(0x7f2a4009b120 0x7f2a400b5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a3fe00640 1 -- 172.21.15.32:0/3388272972 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a400b2d60 con 0x7f2a4015c210 2024-03-06T07:32:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.632+0000 7f2a37e00640 1 --2- 172.21.15.32:0/3388272972 >> v2:172.21.15.39:3300/0 conn(0x7f2a4009b120 0x7f2a400b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.633+0000 7f2a3fe00640 1 --2- 172.21.15.32:0/3388272972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4015c210 0x7f2a4015fb80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009e50 tx=0x7f2a3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5fca394c2e4f1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.634+0000 7f2a3ea00640 1 -- 172.21.15.32:0/3388272972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3802ca70 con 0x7f2a4015c210 2024-03-06T07:32:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.634+0000 7f2a3ea00640 1 -- 172.21.15.32:0/3388272972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3802cc10 con 0x7f2a4015c210 2024-03-06T07:32:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.634+0000 7f2a3ea00640 1 -- 172.21.15.32:0/3388272972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a380343f0 con 0x7f2a4015c210 2024-03-06T07:32:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.634+0000 7f2a46400640 1 -- 172.21.15.32:0/3388272972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4015c210 msgr2=0x7f2a4015fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.634+0000 7f2a46400640 1 --2- 172.21.15.32:0/3388272972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4015c210 0x7f2a4015fb80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009e50 tx=0x7f2a3802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 -- 172.21.15.32:0/3388272972 shutdown_connections 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 --2- 172.21.15.32:0/3388272972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4015c210 0x7f2a4015fb80 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 --2- 172.21.15.32:0/3388272972 >> v2:172.21.15.39:3300/0 conn(0x7f2a4009b120 0x7f2a400b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 --2- 172.21.15.32:0/3388272972 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a400b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 -- 172.21.15.32:0/3388272972 >> 172.21.15.32:0/3388272972 conn(0x7f2a400b7450 msgr2=0x7f2a4009aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 -- 172.21.15.32:0/3388272972 shutdown_connections 2024-03-06T07:32:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.635+0000 7f2a46400640 1 -- 172.21.15.32:0/3388272972 wait complete. 2024-03-06T07:32:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.638+0000 7f2a46400640 1 Processor -- start 2024-03-06T07:32:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.638+0000 7f2a46400640 1 -- start start 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.639+0000 7f2a46400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 0x7f2a40151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.639+0000 7f2a46400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a4009eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a46400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a4015c210 0x7f2a4009f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a46400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400b1880 con 0x7f2a4009b120 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a46400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400b3320 con 0x7f2a4015c210 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a46400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400b1a00 con 0x7f2a4015be30 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 0x7f2a40151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 0x7f2a40151100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35670/0 (socket says 172.21.15.32:35670) 2024-03-06T07:32:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 -- 172.21.15.32:0/866607452 learned_addr learned my addr 172.21.15.32:0/866607452 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3fe00640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a4015c210 0x7f2a4009f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a37e00640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a4009eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 -- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 msgr2=0x7f2a4009eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a4009eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 -- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a4015c210 msgr2=0x7f2a4009f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a4015c210 0x7f2a4009f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.640+0000 7f2a3f400640 1 -- 172.21.15.32:0/866607452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4009c700 con 0x7f2a4009b120 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.641+0000 7f2a3f400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 0x7f2a40151100 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000c7f0 tx=0x7f2a3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.641+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30015070 con 0x7f2a4009b120 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.641+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30004240 con 0x7f2a4009b120 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.641+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30007620 con 0x7f2a4009b120 2024-03-06T07:32:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.641+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4009c9e0 con 0x7f2a4009b120 2024-03-06T07:32:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.641+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4006cda0 con 0x7f2a4009b120 2024-03-06T07:32:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.642+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4004f3a0 con 0x7f2a4009b120 2024-03-06T07:32:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.643+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a300079d0 con 0x7f2a4009b120 2024-03-06T07:32:35.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.644+0000 7f2a37400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a1c06b4d0 0x7f2a1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.644+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a3008a3a0 con 0x7f2a4009b120 2024-03-06T07:32:35.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.644+0000 7f2a37e00640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a1c06b4d0 0x7f2a1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.645+0000 7f2a37e00640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a1c06b4d0 0x7f2a1c06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2a4009ed40 tx=0x7f2a28008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.649+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a30059180 con 0x7f2a4009b120 2024-03-06T07:32:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.845+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a400d2c40 con 0x7f2a1c06b4d0 2024-03-06T07:32:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.857+0000 7f2a37400640 1 -- 172.21.15.32:0/866607452 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252071 (secure 0 0 0) 0x7f2a400d2c40 con 0x7f2a1c06b4d0 2024-03-06T07:32:35.863 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a1c06b4d0 msgr2=0x7f2a1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a1c06b4d0 0x7f2a1c06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2a4009ed40 tx=0x7f2a28008040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 msgr2=0x7f2a40151100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 0x7f2a40151100 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000c7f0 tx=0x7f2a3000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 shutdown_connections 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a1c06b4d0 0x7f2a1c06d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a4015c210 0x7f2a4009f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a4015be30 0x7f2a4009eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 --2- 172.21.15.32:0/866607452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a4009b120 0x7f2a40151100 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 >> 172.21.15.32:0/866607452 conn(0x7f2a400b7450 msgr2=0x7f2a4015c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 shutdown_connections 2024-03-06T07:32:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:35.864+0000 7f2a46400640 1 -- 172.21.15.32:0/866607452 wait complete. 2024-03-06T07:32:35.886 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-03-06T07:32:34.251206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265432,"kb_used_data":7652,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754709288,"statfs":{"total":773094113280,"available":772822310912,"internally_reserved":0,"allocated":7835648,"data_stored":3936016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.945404"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344800+0000","last_change":"2024-03-06T07:32:31.376227+0000","last_active":"2024-03-06T07:32:32.344800+0000","last_peered":"2024-03-06T07:32:32.344800+0000","last_clean":"2024-03-06T07:32:32.344800+0000","last_became_active":"2024-03-06T07:32:31.375565+0000","last_became_peered":"2024-03-06T07:32:31.375565+0000","last_unstale":"2024-03-06T07:32:32.344800+0000","last_undegraded":"2024-03-06T07:32:32.344800+0000","last_fullsized":"2024-03-06T07:32:32.344800+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:35.466204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428460+0000","last_change":"2024-03-06T07:32:31.398455+0000","last_active":"2024-03-06T07:32:32.428460+0000","last_peered":"2024-03-06T07:32:32.428460+0000","last_clean":"2024-03-06T07:32:32.428460+0000","last_became_active":"2024-03-06T07:32:31.398095+0000","last_became_peered":"2024-03-06T07:32:31.398095+0000","last_unstale":"2024-03-06T07:32:32.428460+0000","last_undegraded":"2024-03-06T07:32:32.428460+0000","last_fullsized":"2024-03-06T07:32:32.428460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:56.024432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868770+0000","last_change":"2024-03-06T07:32:31.365256+0000","last_active":"2024-03-06T07:32:32.868770+0000","last_peered":"2024-03-06T07:32:32.868770+0000","last_clean":"2024-03-06T07:32:32.868770+0000","last_became_active":"2024-03-06T07:32:31.364958+0000","last_became_peered":"2024-03-06T07:32:31.364958+0000","last_unstale":"2024-03-06T07:32:32.868770+0000","last_undegraded":"2024-03-06T07:32:32.868770+0000","last_fullsized":"2024-03-06T07:32:32.868770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:21.267960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022143+0000","last_change":"2024-03-06T07:32:31.396262+0000","last_active":"2024-03-06T07:32:33.022143+0000","last_peered":"2024-03-06T07:32:33.022143+0000","last_clean":"2024-03-06T07:32:33.022143+0000","last_became_active":"2024-03-06T07:32:31.395957+0000","last_became_peered":"2024-03-06T07:32:31.395957+0000","last_unstale":"2024-03-06T07:32:33.022143+0000","last_undegraded":"2024-03-06T07:32:33.022143+0000","last_fullsized":"2024-03-06T07:32:33.022143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:21.004750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379759+0000","last_change":"2024-03-06T07:32:31.375542+0000","last_active":"2024-03-06T07:32:32.379759+0000","last_peered":"2024-03-06T07:32:32.379759+0000","last_clean":"2024-03-06T07:32:32.379759+0000","last_became_active":"2024-03-06T07:32:31.375123+0000","last_became_peered":"2024-03-06T07:32:31.375123+0000","last_unstale":"2024-03-06T07:32:32.379759+0000","last_undegraded":"2024-03-06T07:32:32.379759+0000","last_fullsized":"2024-03-06T07:32:32.379759+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:06.122411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427620+0000","last_change":"2024-03-06T07:32:31.398861+0000","last_active":"2024-03-06T07:32:32.427620+0000","last_peered":"2024-03-06T07:32:32.427620+0000","last_clean":"2024-03-06T07:32:32.427620+0000","last_became_active":"2024-03-06T07:32:31.397983+0000","last_became_peered":"2024-03-06T07:32:31.397983+0000","last_unstale":"2024-03-06T07:32:32.427620+0000","last_undegraded":"2024-03-06T07:32:32.427620+0000","last_fullsized":"2024-03-06T07:32:32.427620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:56.197309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020809+0000","last_change":"2024-03-06T07:32:31.381519+0000","last_active":"2024-03-06T07:32:33.020809+0000","last_peered":"2024-03-06T07:32:33.020809+0000","last_clean":"2024-03-06T07:32:33.020809+0000","last_became_active":"2024-03-06T07:32:31.381144+0000","last_became_peered":"2024-03-06T07:32:31.381144+0000","last_unstale":"2024-03-06T07:32:33.020809+0000","last_undegraded":"2024-03-06T07:32:33.020809+0000","last_fullsized":"2024-03-06T07:32:33.020809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:30.599345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427442+0000","last_change":"2024-03-06T07:32:31.385922+0000","last_active":"2024-03-06T07:32:32.427442+0000","last_peered":"2024-03-06T07:32:32.427442+0000","last_clean":"2024-03-06T07:32:32.427442+0000","last_became_active":"2024-03-06T07:32:31.382343+0000","last_became_peered":"2024-03-06T07:32:31.382343+0000","last_unstale":"2024-03-06T07:32:32.427442+0000","last_undegraded":"2024-03-06T07:32:32.427442+0000","last_fullsized":"2024-03-06T07:32:32.427442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:14.700055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380271+0000","last_change":"2024-03-06T07:32:31.372688+0000","last_active":"2024-03-06T07:32:32.380271+0000","last_peered":"2024-03-06T07:32:32.380271+0000","last_clean":"2024-03-06T07:32:32.380271+0000","last_became_active":"2024-03-06T07:32:31.367541+0000","last_became_peered":"2024-03-06T07:32:31.367541+0000","last_unstale":"2024-03-06T07:32:32.380271+0000","last_undegraded":"2024-03-06T07:32:32.380271+0000","last_fullsized":"2024-03-06T07:32:32.380271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:49.904033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346551+0000","last_change":"2024-03-06T07:32:31.385665+0000","last_active":"2024-03-06T07:32:32.346551+0000","last_peered":"2024-03-06T07:32:32.346551+0000","last_clean":"2024-03-06T07:32:32.346551+0000","last_became_active":"2024-03-06T07:32:31.383642+0000","last_became_peered":"2024-03-06T07:32:31.383642+0000","last_unstale":"2024-03-06T07:32:32.346551+0000","last_undegraded":"2024-03-06T07:32:32.346551+0000","last_fullsized":"2024-03-06T07:32:32.346551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:35.177998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345034+0000","last_change":"2024-03-06T07:32:31.373392+0000","last_active":"2024-03-06T07:32:32.345034+0000","last_peered":"2024-03-06T07:32:32.345034+0000","last_clean":"2024-03-06T07:32:32.345034+0000","last_became_active":"2024-03-06T07:32:31.373035+0000","last_became_peered":"2024-03-06T07:32:31.373035+0000","last_unstale":"2024-03-06T07:32:32.345034+0000","last_undegraded":"2024-03-06T07:32:32.345034+0000","last_fullsized":"2024-03-06T07:32:32.345034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:13.161870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021452+0000","last_change":"2024-03-06T07:32:31.369325+0000","last_active":"2024-03-06T07:32:33.021452+0000","last_peered":"2024-03-06T07:32:33.021452+0000","last_clean":"2024-03-06T07:32:33.021452+0000","last_became_active":"2024-03-06T07:32:31.368709+0000","last_became_peered":"2024-03-06T07:32:31.368709+0000","last_unstale":"2024-03-06T07:32:33.021452+0000","last_undegraded":"2024-03-06T07:32:33.021452+0000","last_fullsized":"2024-03-06T07:32:33.021452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:59.870863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022564+0000","last_change":"2024-03-06T07:32:31.391584+0000","last_active":"2024-03-06T07:32:33.022564+0000","last_peered":"2024-03-06T07:32:33.022564+0000","last_clean":"2024-03-06T07:32:33.022564+0000","last_became_active":"2024-03-06T07:32:31.390388+0000","last_became_peered":"2024-03-06T07:32:31.390388+0000","last_unstale":"2024-03-06T07:32:33.022564+0000","last_undegraded":"2024-03-06T07:32:33.022564+0000","last_fullsized":"2024-03-06T07:32:33.022564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:22.386779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379621+0000","last_change":"2024-03-06T07:32:31.381169+0000","last_active":"2024-03-06T07:32:32.379621+0000","last_peered":"2024-03-06T07:32:32.379621+0000","last_clean":"2024-03-06T07:32:32.379621+0000","last_became_active":"2024-03-06T07:32:31.380457+0000","last_became_peered":"2024-03-06T07:32:31.380457+0000","last_unstale":"2024-03-06T07:32:32.379621+0000","last_undegraded":"2024-03-06T07:32:32.379621+0000","last_fullsized":"2024-03-06T07:32:32.379621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:51.366649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021099+0000","last_change":"2024-03-06T07:32:31.371980+0000","last_active":"2024-03-06T07:32:33.021099+0000","last_peered":"2024-03-06T07:32:33.021099+0000","last_clean":"2024-03-06T07:32:33.021099+0000","last_became_active":"2024-03-06T07:32:31.371665+0000","last_became_peered":"2024-03-06T07:32:31.371665+0000","last_unstale":"2024-03-06T07:32:33.021099+0000","last_undegraded":"2024-03-06T07:32:33.021099+0000","last_fullsized":"2024-03-06T07:32:33.021099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346137+0000","last_change":"2024-03-06T07:32:31.386413+0000","last_active":"2024-03-06T07:32:32.346137+0000","last_peered":"2024-03-06T07:32:32.346137+0000","last_clean":"2024-03-06T07:32:32.346137+0000","last_became_active":"2024-03-06T07:32:31.383714+0000","last_became_peered":"2024-03-06T07:32:31.383714+0000","last_unstale":"2024-03-06T07:32:32.346137+0000","last_undegraded":"2024-03-06T07:32:32.346137+0000","last_fullsized":"2024-03-06T07:32:32.346137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:49.223302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380439+0000","last_change":"2024-03-06T07:32:31.369605+0000","last_active":"2024-03-06T07:32:32.380439+0000","last_peered":"2024-03-06T07:32:32.380439+0000","last_clean":"2024-03-06T07:32:32.380439+0000","last_became_active":"2024-03-06T07:32:31.369101+0000","last_became_peered":"2024-03-06T07:32:31.369101+0000","last_unstale":"2024-03-06T07:32:32.380439+0000","last_undegraded":"2024-03-06T07:32:32.380439+0000","last_fullsized":"2024-03-06T07:32:32.380439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:24.128646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021707+0000","last_change":"2024-03-06T07:32:31.389516+0000","last_active":"2024-03-06T07:32:33.021707+0000","last_peered":"2024-03-06T07:32:33.021707+0000","last_clean":"2024-03-06T07:32:33.021707+0000","last_became_active":"2024-03-06T07:32:31.389085+0000","last_became_peered":"2024-03-06T07:32:31.389085+0000","last_unstale":"2024-03-06T07:32:33.021707+0000","last_undegraded":"2024-03-06T07:32:33.021707+0000","last_fullsized":"2024-03-06T07:32:33.021707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:53.766174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428267+0000","last_change":"2024-03-06T07:32:31.380364+0000","last_active":"2024-03-06T07:32:32.428267+0000","last_peered":"2024-03-06T07:32:32.428267+0000","last_clean":"2024-03-06T07:32:32.428267+0000","last_became_active":"2024-03-06T07:32:31.380071+0000","last_became_peered":"2024-03-06T07:32:31.380071+0000","last_unstale":"2024-03-06T07:32:32.428267+0000","last_undegraded":"2024-03-06T07:32:32.428267+0000","last_fullsized":"2024-03-06T07:32:32.428267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:59.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022541+0000","last_change":"2024-03-06T07:32:31.392008+0000","last_active":"2024-03-06T07:32:33.022541+0000","last_peered":"2024-03-06T07:32:33.022541+0000","last_clean":"2024-03-06T07:32:33.022541+0000","last_became_active":"2024-03-06T07:32:31.391504+0000","last_became_peered":"2024-03-06T07:32:31.391504+0000","last_unstale":"2024-03-06T07:32:33.022541+0000","last_undegraded":"2024-03-06T07:32:33.022541+0000","last_fullsized":"2024-03-06T07:32:33.022541+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:03.603511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346116+0000","last_change":"2024-03-06T07:32:31.383795+0000","last_active":"2024-03-06T07:32:32.346116+0000","last_peered":"2024-03-06T07:32:32.346116+0000","last_clean":"2024-03-06T07:32:32.346116+0000","last_became_active":"2024-03-06T07:32:31.383477+0000","last_became_peered":"2024-03-06T07:32:31.383477+0000","last_unstale":"2024-03-06T07:32:32.346116+0000","last_undegraded":"2024-03-06T07:32:32.346116+0000","last_fullsized":"2024-03-06T07:32:32.346116+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:56.024432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345932+0000","last_change":"2024-03-06T07:32:31.385058+0000","last_active":"2024-03-06T07:32:32.345932+0000","last_peered":"2024-03-06T07:32:32.345932+0000","last_clean":"2024-03-06T07:32:32.345932+0000","last_became_active":"2024-03-06T07:32:31.384664+0000","last_became_peered":"2024-03-06T07:32:31.384664+0000","last_unstale":"2024-03-06T07:32:32.345932+0000","last_undegraded":"2024-03-06T07:32:32.345932+0000","last_fullsized":"2024-03-06T07:32:32.345932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342819+0000","last_change":"2024-03-06T07:32:31.382364+0000","last_active":"2024-03-06T07:32:32.342819+0000","last_peered":"2024-03-06T07:32:32.342819+0000","last_clean":"2024-03-06T07:32:32.342819+0000","last_became_active":"2024-03-06T07:32:31.382036+0000","last_became_peered":"2024-03-06T07:32:31.382036+0000","last_unstale":"2024-03-06T07:32:32.342819+0000","last_undegraded":"2024-03-06T07:32:32.342819+0000","last_fullsized":"2024-03-06T07:32:32.342819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:22.386779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342679+0000","last_change":"2024-03-06T07:32:31.390221+0000","last_active":"2024-03-06T07:32:32.342679+0000","last_peered":"2024-03-06T07:32:32.342679+0000","last_clean":"2024-03-06T07:32:32.342679+0000","last_became_active":"2024-03-06T07:32:31.389885+0000","last_became_peered":"2024-03-06T07:32:31.389885+0000","last_unstale":"2024-03-06T07:32:32.342679+0000","last_undegraded":"2024-03-06T07:32:32.342679+0000","last_fullsized":"2024-03-06T07:32:32.342679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:03.603511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.347083+0000","last_change":"2024-03-06T07:32:31.393260+0000","last_active":"2024-03-06T07:32:32.347083+0000","last_peered":"2024-03-06T07:32:32.347083+0000","last_clean":"2024-03-06T07:32:32.347083+0000","last_became_active":"2024-03-06T07:32:31.392914+0000","last_became_peered":"2024-03-06T07:32:31.392914+0000","last_unstale":"2024-03-06T07:32:32.347083+0000","last_undegraded":"2024-03-06T07:32:32.347083+0000","last_fullsized":"2024-03-06T07:32:32.347083+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:10.997851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021450+0000","last_change":"2024-03-06T07:32:31.394397+0000","last_active":"2024-03-06T07:32:33.021450+0000","last_peered":"2024-03-06T07:32:33.021450+0000","last_clean":"2024-03-06T07:32:33.021450+0000","last_became_active":"2024-03-06T07:32:31.394100+0000","last_became_peered":"2024-03-06T07:32:31.394100+0000","last_unstale":"2024-03-06T07:32:33.021450+0000","last_undegraded":"2024-03-06T07:32:33.021450+0000","last_fullsized":"2024-03-06T07:32:33.021450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:38.416113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.869086+0000","last_change":"2024-03-06T07:32:31.378434+0000","last_active":"2024-03-06T07:32:32.869086+0000","last_peered":"2024-03-06T07:32:32.869086+0000","last_clean":"2024-03-06T07:32:32.869086+0000","last_became_active":"2024-03-06T07:32:31.375917+0000","last_became_peered":"2024-03-06T07:32:31.375917+0000","last_unstale":"2024-03-06T07:32:32.869086+0000","last_undegraded":"2024-03-06T07:32:32.869086+0000","last_fullsized":"2024-03-06T07:32:32.869086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:25.190301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346433+0000","last_change":"2024-03-06T07:32:31.380146+0000","last_active":"2024-03-06T07:32:32.346433+0000","last_peered":"2024-03-06T07:32:32.346433+0000","last_clean":"2024-03-06T07:32:32.346433+0000","last_became_active":"2024-03-06T07:32:31.379834+0000","last_became_peered":"2024-03-06T07:32:31.379834+0000","last_unstale":"2024-03-06T07:32:32.346433+0000","last_undegraded":"2024-03-06T07:32:32.346433+0000","last_fullsized":"2024-03-06T07:32:32.346433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.273035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868942+0000","last_change":"2024-03-06T07:32:31.387684+0000","last_active":"2024-03-06T07:32:32.868942+0000","last_peered":"2024-03-06T07:32:32.868942+0000","last_clean":"2024-03-06T07:32:32.868942+0000","last_became_active":"2024-03-06T07:32:31.387328+0000","last_became_peered":"2024-03-06T07:32:31.387328+0000","last_unstale":"2024-03-06T07:32:32.868942+0000","last_undegraded":"2024-03-06T07:32:32.868942+0000","last_fullsized":"2024-03-06T07:32:32.868942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:20.312459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021081+0000","last_change":"2024-03-06T07:32:31.395830+0000","last_active":"2024-03-06T07:32:33.021081+0000","last_peered":"2024-03-06T07:32:33.021081+0000","last_clean":"2024-03-06T07:32:33.021081+0000","last_became_active":"2024-03-06T07:32:31.395513+0000","last_became_peered":"2024-03-06T07:32:31.395513+0000","last_unstale":"2024-03-06T07:32:33.021081+0000","last_undegraded":"2024-03-06T07:32:33.021081+0000","last_fullsized":"2024-03-06T07:32:33.021081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:44.615322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021246+0000","last_change":"2024-03-06T07:32:31.380757+0000","last_active":"2024-03-06T07:32:33.021246+0000","last_peered":"2024-03-06T07:32:33.021246+0000","last_clean":"2024-03-06T07:32:33.021246+0000","last_became_active":"2024-03-06T07:32:31.380410+0000","last_became_peered":"2024-03-06T07:32:31.380410+0000","last_unstale":"2024-03-06T07:32:33.021246+0000","last_undegraded":"2024-03-06T07:32:33.021246+0000","last_fullsized":"2024-03-06T07:32:33.021246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:14.183626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428566+0000","last_change":"2024-03-06T07:32:31.382207+0000","last_active":"2024-03-06T07:32:32.428566+0000","last_peered":"2024-03-06T07:32:32.428566+0000","last_clean":"2024-03-06T07:32:32.428566+0000","last_became_active":"2024-03-06T07:32:31.381761+0000","last_became_peered":"2024-03-06T07:32:31.381761+0000","last_unstale":"2024-03-06T07:32:32.428566+0000","last_undegraded":"2024-03-06T07:32:32.428566+0000","last_fullsized":"2024-03-06T07:32:32.428566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:49.223302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344348+0000","last_change":"2024-03-06T07:32:31.392122+0000","last_active":"2024-03-06T07:32:32.344348+0000","last_peered":"2024-03-06T07:32:32.344348+0000","last_clean":"2024-03-06T07:32:32.344348+0000","last_became_active":"2024-03-06T07:32:31.389788+0000","last_became_peered":"2024-03-06T07:32:31.389788+0000","last_unstale":"2024-03-06T07:32:32.344348+0000","last_undegraded":"2024-03-06T07:32:32.344348+0000","last_fullsized":"2024-03-06T07:32:32.344348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:47.267079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345232+0000","last_change":"2024-03-06T07:32:30.336032+0000","last_active":"2024-03-06T07:32:32.345232+0000","last_peered":"2024-03-06T07:32:32.345232+0000","last_clean":"2024-03-06T07:32:32.345232+0000","last_became_active":"2024-03-06T07:32:30.335160+0000","last_became_peered":"2024-03-06T07:32:30.335160+0000","last_unstale":"2024-03-06T07:32:32.345232+0000","last_undegraded":"2024-03-06T07:32:32.345232+0000","last_fullsized":"2024-03-06T07:32:32.345232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.983848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380451+0000","last_change":"2024-03-06T07:32:31.369395+0000","last_active":"2024-03-06T07:32:32.380451+0000","last_peered":"2024-03-06T07:32:32.380451+0000","last_clean":"2024-03-06T07:32:32.380451+0000","last_became_active":"2024-03-06T07:32:31.367173+0000","last_became_peered":"2024-03-06T07:32:31.367173+0000","last_unstale":"2024-03-06T07:32:32.380451+0000","last_undegraded":"2024-03-06T07:32:32.380451+0000","last_fullsized":"2024-03-06T07:32:32.380451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:29.229422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.023167+0000","last_change":"2024-03-06T07:32:30.334239+0000","last_active":"2024-03-06T07:32:33.023167+0000","last_peered":"2024-03-06T07:32:33.023167+0000","last_clean":"2024-03-06T07:32:33.023167+0000","last_became_active":"2024-03-06T07:32:30.333895+0000","last_became_peered":"2024-03-06T07:32:30.333895+0000","last_unstale":"2024-03-06T07:32:33.023167+0000","last_undegraded":"2024-03-06T07:32:33.023167+0000","last_fullsized":"2024-03-06T07:32:33.023167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.983848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428590+0000","last_change":"2024-03-06T07:32:31.395284+0000","last_active":"2024-03-06T07:32:32.428590+0000","last_peered":"2024-03-06T07:32:32.428590+0000","last_clean":"2024-03-06T07:32:32.428590+0000","last_became_active":"2024-03-06T07:32:31.395003+0000","last_became_peered":"2024-03-06T07:32:31.395003+0000","last_unstale":"2024-03-06T07:32:32.428590+0000","last_undegraded":"2024-03-06T07:32:32.428590+0000","last_fullsized":"2024-03-06T07:32:32.428590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.273035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346707+0000","last_change":"2024-03-06T07:32:30.352461+0000","last_active":"2024-03-06T07:32:32.346707+0000","last_peered":"2024-03-06T07:32:32.346707+0000","last_clean":"2024-03-06T07:32:32.346707+0000","last_became_active":"2024-03-06T07:32:30.352126+0000","last_became_peered":"2024-03-06T07:32:30.352126+0000","last_unstale":"2024-03-06T07:32:32.346707+0000","last_undegraded":"2024-03-06T07:32:32.346707+0000","last_fullsized":"2024-03-06T07:32:32.346707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:44.591233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344064+0000","last_change":"2024-03-06T07:32:31.377035+0000","last_active":"2024-03-06T07:32:32.344064+0000","last_peered":"2024-03-06T07:32:32.344064+0000","last_clean":"2024-03-06T07:32:32.344064+0000","last_became_active":"2024-03-06T07:32:31.373591+0000","last_became_peered":"2024-03-06T07:32:31.373591+0000","last_unstale":"2024-03-06T07:32:32.344064+0000","last_undegraded":"2024-03-06T07:32:32.344064+0000","last_fullsized":"2024-03-06T07:32:32.344064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:51.068688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428740+0000","last_change":"2024-03-06T07:32:30.343983+0000","last_active":"2024-03-06T07:32:32.428740+0000","last_peered":"2024-03-06T07:32:32.428740+0000","last_clean":"2024-03-06T07:32:32.428740+0000","last_became_active":"2024-03-06T07:32:30.343601+0000","last_became_peered":"2024-03-06T07:32:30.343601+0000","last_unstale":"2024-03-06T07:32:32.428740+0000","last_undegraded":"2024-03-06T07:32:32.428740+0000","last_fullsized":"2024-03-06T07:32:32.428740+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:07.366173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021185+0000","last_change":"2024-03-06T07:32:31.381007+0000","last_active":"2024-03-06T07:32:33.021185+0000","last_peered":"2024-03-06T07:32:33.021185+0000","last_clean":"2024-03-06T07:32:33.021185+0000","last_became_active":"2024-03-06T07:32:31.380668+0000","last_became_peered":"2024-03-06T07:32:31.380668+0000","last_unstale":"2024-03-06T07:32:33.021185+0000","last_undegraded":"2024-03-06T07:32:33.021185+0000","last_fullsized":"2024-03-06T07:32:33.021185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:34.396284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.869093+0000","last_change":"2024-03-06T07:32:30.352023+0000","last_active":"2024-03-06T07:32:32.869093+0000","last_peered":"2024-03-06T07:32:32.869093+0000","last_clean":"2024-03-06T07:32:32.869093+0000","last_became_active":"2024-03-06T07:32:30.351701+0000","last_became_peered":"2024-03-06T07:32:30.351701+0000","last_unstale":"2024-03-06T07:32:32.869093+0000","last_undegraded":"2024-03-06T07:32:32.869093+0000","last_fullsized":"2024-03-06T07:32:32.869093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:08.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346844+0000","last_change":"2024-03-06T07:32:31.394436+0000","last_active":"2024-03-06T07:32:32.346844+0000","last_peered":"2024-03-06T07:32:32.346844+0000","last_clean":"2024-03-06T07:32:32.346844+0000","last_became_active":"2024-03-06T07:32:31.393394+0000","last_became_peered":"2024-03-06T07:32:31.393394+0000","last_unstale":"2024-03-06T07:32:32.346844+0000","last_undegraded":"2024-03-06T07:32:32.346844+0000","last_fullsized":"2024-03-06T07:32:32.346844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:10.287413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427456+0000","last_change":"2024-03-06T07:32:30.350472+0000","last_active":"2024-03-06T07:32:32.427456+0000","last_peered":"2024-03-06T07:32:32.427456+0000","last_clean":"2024-03-06T07:32:32.427456+0000","last_became_active":"2024-03-06T07:32:30.350124+0000","last_became_peered":"2024-03-06T07:32:30.350124+0000","last_unstale":"2024-03-06T07:32:32.427456+0000","last_undegraded":"2024-03-06T07:32:32.427456+0000","last_fullsized":"2024-03-06T07:32:32.427456+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:48.890030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379321+0000","last_change":"2024-03-06T07:32:31.387251+0000","last_active":"2024-03-06T07:32:32.379321+0000","last_peered":"2024-03-06T07:32:32.379321+0000","last_clean":"2024-03-06T07:32:32.379321+0000","last_became_active":"2024-03-06T07:32:31.386851+0000","last_became_peered":"2024-03-06T07:32:31.386851+0000","last_unstale":"2024-03-06T07:32:32.379321+0000","last_undegraded":"2024-03-06T07:32:32.379321+0000","last_fullsized":"2024-03-06T07:32:32.379321+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.084945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021006+0000","last_change":"2024-03-06T07:32:30.343876+0000","last_active":"2024-03-06T07:32:33.021006+0000","last_peered":"2024-03-06T07:32:33.021006+0000","last_clean":"2024-03-06T07:32:33.021006+0000","last_became_active":"2024-03-06T07:32:30.343537+0000","last_became_peered":"2024-03-06T07:32:30.343537+0000","last_unstale":"2024-03-06T07:32:33.021006+0000","last_undegraded":"2024-03-06T07:32:33.021006+0000","last_fullsized":"2024-03-06T07:32:33.021006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:30.592256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021531+0000","last_change":"2024-03-06T07:32:31.388097+0000","last_active":"2024-03-06T07:32:33.021531+0000","last_peered":"2024-03-06T07:32:33.021531+0000","last_clean":"2024-03-06T07:32:33.021531+0000","last_became_active":"2024-03-06T07:32:31.387730+0000","last_became_peered":"2024-03-06T07:32:31.387730+0000","last_unstale":"2024-03-06T07:32:33.021531+0000","last_undegraded":"2024-03-06T07:32:33.021531+0000","last_fullsized":"2024-03-06T07:32:33.021531+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:31.900032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379255+0000","last_change":"2024-03-06T07:32:30.339673+0000","last_active":"2024-03-06T07:32:32.379255+0000","last_peered":"2024-03-06T07:32:32.379255+0000","last_clean":"2024-03-06T07:32:32.379255+0000","last_became_active":"2024-03-06T07:32:30.339390+0000","last_became_peered":"2024-03-06T07:32:30.339390+0000","last_unstale":"2024-03-06T07:32:32.379255+0000","last_undegraded":"2024-03-06T07:32:32.379255+0000","last_fullsized":"2024-03-06T07:32:32.379255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.127597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021772+0000","last_change":"2024-03-06T07:32:31.393140+0000","last_active":"2024-03-06T07:32:33.021772+0000","last_peered":"2024-03-06T07:32:33.021772+0000","last_clean":"2024-03-06T07:32:33.021772+0000","last_became_active":"2024-03-06T07:32:31.392848+0000","last_became_peered":"2024-03-06T07:32:31.392848+0000","last_unstale":"2024-03-06T07:32:33.021772+0000","last_undegraded":"2024-03-06T07:32:33.021772+0000","last_fullsized":"2024-03-06T07:32:33.021772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346795+0000","last_change":"2024-03-06T07:32:30.345557+0000","last_active":"2024-03-06T07:32:32.346795+0000","last_peered":"2024-03-06T07:32:32.346795+0000","last_clean":"2024-03-06T07:32:32.346795+0000","last_became_active":"2024-03-06T07:32:30.345233+0000","last_became_peered":"2024-03-06T07:32:30.345233+0000","last_unstale":"2024-03-06T07:32:32.346795+0000","last_undegraded":"2024-03-06T07:32:32.346795+0000","last_fullsized":"2024-03-06T07:32:32.346795+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:07.398886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345138+0000","last_change":"2024-03-06T07:32:31.388519+0000","last_active":"2024-03-06T07:32:32.345138+0000","last_peered":"2024-03-06T07:32:32.345138+0000","last_clean":"2024-03-06T07:32:32.345138+0000","last_became_active":"2024-03-06T07:32:31.386548+0000","last_became_peered":"2024-03-06T07:32:31.386548+0000","last_unstale":"2024-03-06T07:32:32.345138+0000","last_undegraded":"2024-03-06T07:32:32.345138+0000","last_fullsized":"2024-03-06T07:32:32.345138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:09.678529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428543+0000","last_change":"2024-03-06T07:32:30.352241+0000","last_active":"2024-03-06T07:32:32.428543+0000","last_peered":"2024-03-06T07:32:32.428543+0000","last_clean":"2024-03-06T07:32:32.428543+0000","last_became_active":"2024-03-06T07:32:30.351713+0000","last_became_peered":"2024-03-06T07:32:30.351713+0000","last_unstale":"2024-03-06T07:32:32.428543+0000","last_undegraded":"2024-03-06T07:32:32.428543+0000","last_fullsized":"2024-03-06T07:32:32.428543+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:52.752172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428406+0000","last_change":"2024-03-06T07:32:31.398523+0000","last_active":"2024-03-06T07:32:32.428406+0000","last_peered":"2024-03-06T07:32:32.428406+0000","last_clean":"2024-03-06T07:32:32.428406+0000","last_became_active":"2024-03-06T07:32:31.398239+0000","last_became_peered":"2024-03-06T07:32:31.398239+0000","last_unstale":"2024-03-06T07:32:32.428406+0000","last_undegraded":"2024-03-06T07:32:32.428406+0000","last_fullsized":"2024-03-06T07:32:32.428406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868848+0000","last_change":"2024-03-06T07:32:30.347740+0000","last_active":"2024-03-06T07:32:32.868848+0000","last_peered":"2024-03-06T07:32:32.868848+0000","last_clean":"2024-03-06T07:32:32.868848+0000","last_became_active":"2024-03-06T07:32:30.347359+0000","last_became_peered":"2024-03-06T07:32:30.347359+0000","last_unstale":"2024-03-06T07:32:32.868848+0000","last_undegraded":"2024-03-06T07:32:32.868848+0000","last_fullsized":"2024-03-06T07:32:32.868848+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:54.044872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379649+0000","last_change":"2024-03-06T07:32:31.369356+0000","last_active":"2024-03-06T07:32:32.379649+0000","last_peered":"2024-03-06T07:32:32.379649+0000","last_clean":"2024-03-06T07:32:32.379649+0000","last_became_active":"2024-03-06T07:32:31.366886+0000","last_became_peered":"2024-03-06T07:32:31.366886+0000","last_unstale":"2024-03-06T07:32:32.379649+0000","last_undegraded":"2024-03-06T07:32:32.379649+0000","last_fullsized":"2024-03-06T07:32:32.379649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:31.606258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021886+0000","last_change":"2024-03-06T07:32:30.333935+0000","last_active":"2024-03-06T07:32:33.021886+0000","last_peered":"2024-03-06T07:32:33.021886+0000","last_clean":"2024-03-06T07:32:33.021886+0000","last_became_active":"2024-03-06T07:32:30.333488+0000","last_became_peered":"2024-03-06T07:32:30.333488+0000","last_unstale":"2024-03-06T07:32:33.021886+0000","last_undegraded":"2024-03-06T07:32:33.021886+0000","last_fullsized":"2024-03-06T07:32:33.021886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:18.400240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342679+0000","last_change":"2024-03-06T07:32:31.373123+0000","last_active":"2024-03-06T07:32:32.342679+0000","last_peered":"2024-03-06T07:32:32.342679+0000","last_clean":"2024-03-06T07:32:32.342679+0000","last_became_active":"2024-03-06T07:32:31.372748+0000","last_became_peered":"2024-03-06T07:32:31.372748+0000","last_unstale":"2024-03-06T07:32:32.342679+0000","last_undegraded":"2024-03-06T07:32:32.342679+0000","last_fullsized":"2024-03-06T07:32:32.342679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:29.118305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021987+0000","last_change":"2024-03-06T07:32:30.341457+0000","last_active":"2024-03-06T07:32:33.021987+0000","last_peered":"2024-03-06T07:32:33.021987+0000","last_clean":"2024-03-06T07:32:33.021987+0000","last_became_active":"2024-03-06T07:32:30.341045+0000","last_became_peered":"2024-03-06T07:32:30.341045+0000","last_unstale":"2024-03-06T07:32:33.021987+0000","last_undegraded":"2024-03-06T07:32:33.021987+0000","last_fullsized":"2024-03-06T07:32:33.021987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:11.934557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020894+0000","last_change":"2024-03-06T07:32:31.391789+0000","last_active":"2024-03-06T07:32:33.020894+0000","last_peered":"2024-03-06T07:32:33.020894+0000","last_clean":"2024-03-06T07:32:33.020894+0000","last_became_active":"2024-03-06T07:32:31.391321+0000","last_became_peered":"2024-03-06T07:32:31.391321+0000","last_unstale":"2024-03-06T07:32:33.020894+0000","last_undegraded":"2024-03-06T07:32:33.020894+0000","last_fullsized":"2024-03-06T07:32:33.020894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:46.759541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427849+0000","last_change":"2024-03-06T07:32:30.348110+0000","last_active":"2024-03-06T07:32:32.427849+0000","last_peered":"2024-03-06T07:32:32.427849+0000","last_clean":"2024-03-06T07:32:32.427849+0000","last_became_active":"2024-03-06T07:32:30.345283+0000","last_became_peered":"2024-03-06T07:32:30.345283+0000","last_unstale":"2024-03-06T07:32:32.427849+0000","last_undegraded":"2024-03-06T07:32:32.427849+0000","last_fullsized":"2024-03-06T07:32:32.427849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:33.085495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021033+0000","last_change":"2024-03-06T07:32:31.384028+0000","last_active":"2024-03-06T07:32:33.021033+0000","last_peered":"2024-03-06T07:32:33.021033+0000","last_clean":"2024-03-06T07:32:33.021033+0000","last_became_active":"2024-03-06T07:32:31.383689+0000","last_became_peered":"2024-03-06T07:32:31.383689+0000","last_unstale":"2024-03-06T07:32:33.021033+0000","last_undegraded":"2024-03-06T07:32:33.021033+0000","last_fullsized":"2024-03-06T07:32:33.021033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:38.416113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021383+0000","last_change":"2024-03-06T07:32:30.348848+0000","last_active":"2024-03-06T07:32:33.021383+0000","last_peered":"2024-03-06T07:32:33.021383+0000","last_clean":"2024-03-06T07:32:33.021383+0000","last_became_active":"2024-03-06T07:32:30.348513+0000","last_became_peered":"2024-03-06T07:32:30.348513+0000","last_unstale":"2024-03-06T07:32:33.021383+0000","last_undegraded":"2024-03-06T07:32:33.021383+0000","last_fullsized":"2024-03-06T07:32:33.021383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:58.445649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379451+0000","last_change":"2024-03-06T07:32:31.367305+0000","last_active":"2024-03-06T07:32:32.379451+0000","last_peered":"2024-03-06T07:32:32.379451+0000","last_clean":"2024-03-06T07:32:32.379451+0000","last_became_active":"2024-03-06T07:32:31.366969+0000","last_became_peered":"2024-03-06T07:32:31.366969+0000","last_unstale":"2024-03-06T07:32:32.379451+0000","last_undegraded":"2024-03-06T07:32:32.379451+0000","last_fullsized":"2024-03-06T07:32:32.379451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:09.456105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345534+0000","last_change":"2024-03-06T07:32:30.346654+0000","last_active":"2024-03-06T07:32:32.345534+0000","last_peered":"2024-03-06T07:32:32.345534+0000","last_clean":"2024-03-06T07:32:32.345534+0000","last_became_active":"2024-03-06T07:32:30.344988+0000","last_became_peered":"2024-03-06T07:32:30.344988+0000","last_unstale":"2024-03-06T07:32:32.345534+0000","last_undegraded":"2024-03-06T07:32:32.345534+0000","last_fullsized":"2024-03-06T07:32:32.345534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:19.990747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346856+0000","last_change":"2024-03-06T07:32:31.380008+0000","last_active":"2024-03-06T07:32:32.346856+0000","last_peered":"2024-03-06T07:32:32.346856+0000","last_clean":"2024-03-06T07:32:32.346856+0000","last_became_active":"2024-03-06T07:32:31.379684+0000","last_became_peered":"2024-03-06T07:32:31.379684+0000","last_unstale":"2024-03-06T07:32:32.346856+0000","last_undegraded":"2024-03-06T07:32:32.346856+0000","last_fullsized":"2024-03-06T07:32:32.346856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:56.576735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428674+0000","last_change":"2024-03-06T07:32:30.350922+0000","last_active":"2024-03-06T07:32:32.428674+0000","last_peered":"2024-03-06T07:32:32.428674+0000","last_clean":"2024-03-06T07:32:32.428674+0000","last_became_active":"2024-03-06T07:32:30.350616+0000","last_became_peered":"2024-03-06T07:32:30.350616+0000","last_unstale":"2024-03-06T07:32:32.428674+0000","last_undegraded":"2024-03-06T07:32:32.428674+0000","last_fullsized":"2024-03-06T07:32:32.428674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:47.004941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345038+0000","last_change":"2024-03-06T07:32:31.380454+0000","last_active":"2024-03-06T07:32:32.345038+0000","last_peered":"2024-03-06T07:32:32.345038+0000","last_clean":"2024-03-06T07:32:32.345038+0000","last_became_active":"2024-03-06T07:32:31.375982+0000","last_became_peered":"2024-03-06T07:32:31.375982+0000","last_unstale":"2024-03-06T07:32:32.345038+0000","last_undegraded":"2024-03-06T07:32:32.345038+0000","last_fullsized":"2024-03-06T07:32:32.345038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:55.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346884+0000","last_change":"2024-03-06T07:32:30.352788+0000","last_active":"2024-03-06T07:32:32.346884+0000","last_peered":"2024-03-06T07:32:32.346884+0000","last_clean":"2024-03-06T07:32:32.346884+0000","last_became_active":"2024-03-06T07:32:30.352354+0000","last_became_peered":"2024-03-06T07:32:30.352354+0000","last_unstale":"2024-03-06T07:32:32.346884+0000","last_undegraded":"2024-03-06T07:32:32.346884+0000","last_fullsized":"2024-03-06T07:32:32.346884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:47.004941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021675+0000","last_change":"2024-03-06T07:32:31.369095+0000","last_active":"2024-03-06T07:32:33.021675+0000","last_peered":"2024-03-06T07:32:33.021675+0000","last_clean":"2024-03-06T07:32:33.021675+0000","last_became_active":"2024-03-06T07:32:31.368344+0000","last_became_peered":"2024-03-06T07:32:31.368344+0000","last_unstale":"2024-03-06T07:32:33.021675+0000","last_undegraded":"2024-03-06T07:32:33.021675+0000","last_fullsized":"2024-03-06T07:32:33.021675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:34.322368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428386+0000","last_change":"2024-03-06T07:32:30.354128+0000","last_active":"2024-03-06T07:32:32.428386+0000","last_peered":"2024-03-06T07:32:32.428386+0000","last_clean":"2024-03-06T07:32:32.428386+0000","last_became_active":"2024-03-06T07:32:30.353264+0000","last_became_peered":"2024-03-06T07:32:30.353264+0000","last_unstale":"2024-03-06T07:32:32.428386+0000","last_undegraded":"2024-03-06T07:32:32.428386+0000","last_fullsized":"2024-03-06T07:32:32.428386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:33.308365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380100+0000","last_change":"2024-03-06T07:32:31.387217+0000","last_active":"2024-03-06T07:32:32.380100+0000","last_peered":"2024-03-06T07:32:32.380100+0000","last_clean":"2024-03-06T07:32:32.380100+0000","last_became_active":"2024-03-06T07:32:31.386934+0000","last_became_peered":"2024-03-06T07:32:31.386934+0000","last_unstale":"2024-03-06T07:32:32.380100+0000","last_undegraded":"2024-03-06T07:32:32.380100+0000","last_fullsized":"2024-03-06T07:32:32.380100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:19.414243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.343559+0000","last_change":"2024-03-06T07:32:30.349715+0000","last_active":"2024-03-06T07:32:32.343559+0000","last_peered":"2024-03-06T07:32:32.343559+0000","last_clean":"2024-03-06T07:32:32.343559+0000","last_became_active":"2024-03-06T07:32:30.349322+0000","last_became_peered":"2024-03-06T07:32:30.349322+0000","last_unstale":"2024-03-06T07:32:32.343559+0000","last_undegraded":"2024-03-06T07:32:32.343559+0000","last_fullsized":"2024-03-06T07:32:32.343559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:55.562732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.869137+0000","last_change":"2024-03-06T07:32:31.358476+0000","last_active":"2024-03-06T07:32:32.869137+0000","last_peered":"2024-03-06T07:32:32.869137+0000","last_clean":"2024-03-06T07:32:32.869137+0000","last_became_active":"2024-03-06T07:32:31.356609+0000","last_became_peered":"2024-03-06T07:32:31.356609+0000","last_unstale":"2024-03-06T07:32:32.869137+0000","last_undegraded":"2024-03-06T07:32:32.869137+0000","last_fullsized":"2024-03-06T07:32:32.869137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:25.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428000+0000","last_change":"2024-03-06T07:32:30.348348+0000","last_active":"2024-03-06T07:32:32.428000+0000","last_peered":"2024-03-06T07:32:32.428000+0000","last_clean":"2024-03-06T07:32:32.428000+0000","last_became_active":"2024-03-06T07:32:30.347784+0000","last_became_peered":"2024-03-06T07:32:30.347784+0000","last_unstale":"2024-03-06T07:32:32.428000+0000","last_undegraded":"2024-03-06T07:32:32.428000+0000","last_fullsized":"2024-03-06T07:32:32.428000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:23.114643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868701+0000","last_change":"2024-03-06T07:32:31.385742+0000","last_active":"2024-03-06T07:32:32.868701+0000","last_peered":"2024-03-06T07:32:32.868701+0000","last_clean":"2024-03-06T07:32:32.868701+0000","last_became_active":"2024-03-06T07:32:31.385086+0000","last_became_peered":"2024-03-06T07:32:31.385086+0000","last_unstale":"2024-03-06T07:32:32.868701+0000","last_undegraded":"2024-03-06T07:32:32.868701+0000","last_fullsized":"2024-03-06T07:32:32.868701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:03.855456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021395+0000","last_change":"2024-03-06T07:32:30.348334+0000","last_active":"2024-03-06T07:32:33.021395+0000","last_peered":"2024-03-06T07:32:33.021395+0000","last_clean":"2024-03-06T07:32:33.021395+0000","last_became_active":"2024-03-06T07:32:30.345552+0000","last_became_peered":"2024-03-06T07:32:30.345552+0000","last_unstale":"2024-03-06T07:32:33.021395+0000","last_undegraded":"2024-03-06T07:32:33.021395+0000","last_fullsized":"2024-03-06T07:32:33.021395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:38.476626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868636+0000","last_change":"2024-03-06T07:32:31.382597+0000","last_active":"2024-03-06T07:32:32.868636+0000","last_peered":"2024-03-06T07:32:32.868636+0000","last_clean":"2024-03-06T07:32:32.868636+0000","last_became_active":"2024-03-06T07:32:31.382158+0000","last_became_peered":"2024-03-06T07:32:31.382158+0000","last_unstale":"2024-03-06T07:32:32.868636+0000","last_undegraded":"2024-03-06T07:32:32.868636+0000","last_fullsized":"2024-03-06T07:32:32.868636+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:20.971174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020665+0000","last_change":"2024-03-06T07:32:30.336135+0000","last_active":"2024-03-06T07:32:33.020665+0000","last_peered":"2024-03-06T07:32:33.020665+0000","last_clean":"2024-03-06T07:32:33.020665+0000","last_became_active":"2024-03-06T07:32:30.335743+0000","last_became_peered":"2024-03-06T07:32:30.335743+0000","last_unstale":"2024-03-06T07:32:33.020665+0000","last_undegraded":"2024-03-06T07:32:33.020665+0000","last_fullsized":"2024-03-06T07:32:33.020665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:55.562732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022487+0000","last_change":"2024-03-06T07:32:31.365105+0000","last_active":"2024-03-06T07:32:33.022487+0000","last_peered":"2024-03-06T07:32:33.022487+0000","last_clean":"2024-03-06T07:32:33.022487+0000","last_became_active":"2024-03-06T07:32:31.364824+0000","last_became_peered":"2024-03-06T07:32:31.364824+0000","last_unstale":"2024-03-06T07:32:33.022487+0000","last_undegraded":"2024-03-06T07:32:33.022487+0000","last_fullsized":"2024-03-06T07:32:33.022487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:29.118305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428919+0000","last_change":"2024-03-06T07:32:30.350776+0000","last_active":"2024-03-06T07:32:32.428919+0000","last_peered":"2024-03-06T07:32:32.428919+0000","last_clean":"2024-03-06T07:32:32.428919+0000","last_became_active":"2024-03-06T07:32:30.350187+0000","last_became_peered":"2024-03-06T07:32:30.350187+0000","last_unstale":"2024-03-06T07:32:32.428919+0000","last_undegraded":"2024-03-06T07:32:32.428919+0000","last_fullsized":"2024-03-06T07:32:32.428919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:19.990747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021229+0000","last_change":"2024-03-06T07:32:31.387704+0000","last_active":"2024-03-06T07:32:33.021229+0000","last_peered":"2024-03-06T07:32:33.021229+0000","last_clean":"2024-03-06T07:32:33.021229+0000","last_became_active":"2024-03-06T07:32:31.387338+0000","last_became_peered":"2024-03-06T07:32:31.387338+0000","last_unstale":"2024-03-06T07:32:33.021229+0000","last_undegraded":"2024-03-06T07:32:33.021229+0000","last_fullsized":"2024-03-06T07:32:33.021229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:24.837198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021661+0000","last_change":"2024-03-06T07:32:30.349574+0000","last_active":"2024-03-06T07:32:33.021661+0000","last_peered":"2024-03-06T07:32:33.021661+0000","last_clean":"2024-03-06T07:32:33.021661+0000","last_became_active":"2024-03-06T07:32:30.348811+0000","last_became_peered":"2024-03-06T07:32:30.348811+0000","last_unstale":"2024-03-06T07:32:33.021661+0000","last_undegraded":"2024-03-06T07:32:33.021661+0000","last_fullsized":"2024-03-06T07:32:33.021661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:55.010429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.343937+0000","last_change":"2024-03-06T07:32:31.397499+0000","last_active":"2024-03-06T07:32:32.343937+0000","last_peered":"2024-03-06T07:32:32.343937+0000","last_clean":"2024-03-06T07:32:32.343937+0000","last_became_active":"2024-03-06T07:32:31.397170+0000","last_became_peered":"2024-03-06T07:32:31.397170+0000","last_unstale":"2024-03-06T07:32:32.343937+0000","last_undegraded":"2024-03-06T07:32:32.343937+0000","last_fullsized":"2024-03-06T07:32:32.343937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:07.228675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380092+0000","last_change":"2024-03-06T07:32:30.333850+0000","last_active":"2024-03-06T07:32:32.380092+0000","last_peered":"2024-03-06T07:32:32.380092+0000","last_clean":"2024-03-06T07:32:32.380092+0000","last_became_active":"2024-03-06T07:32:30.333532+0000","last_became_peered":"2024-03-06T07:32:30.333532+0000","last_unstale":"2024-03-06T07:32:32.380092+0000","last_undegraded":"2024-03-06T07:32:32.380092+0000","last_fullsized":"2024-03-06T07:32:32.380092+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:16.696675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344476+0000","last_change":"2024-03-06T07:32:31.389800+0000","last_active":"2024-03-06T07:32:32.344476+0000","last_peered":"2024-03-06T07:32:32.344476+0000","last_clean":"2024-03-06T07:32:32.344476+0000","last_became_active":"2024-03-06T07:32:31.388275+0000","last_became_peered":"2024-03-06T07:32:31.388275+0000","last_unstale":"2024-03-06T07:32:32.344476+0000","last_undegraded":"2024-03-06T07:32:32.344476+0000","last_fullsized":"2024-03-06T07:32:32.344476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:16.638971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022814+0000","last_change":"2024-03-06T07:32:30.345302+0000","last_active":"2024-03-06T07:32:33.022814+0000","last_peered":"2024-03-06T07:32:33.022814+0000","last_clean":"2024-03-06T07:32:33.022814+0000","last_became_active":"2024-03-06T07:32:30.344975+0000","last_became_peered":"2024-03-06T07:32:30.344975+0000","last_unstale":"2024-03-06T07:32:33.022814+0000","last_undegraded":"2024-03-06T07:32:33.022814+0000","last_fullsized":"2024-03-06T07:32:33.022814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:51.919431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379780+0000","last_change":"2024-03-06T07:32:31.368330+0000","last_active":"2024-03-06T07:32:32.379780+0000","last_peered":"2024-03-06T07:32:32.379780+0000","last_clean":"2024-03-06T07:32:32.379780+0000","last_became_active":"2024-03-06T07:32:31.368037+0000","last_became_peered":"2024-03-06T07:32:31.368037+0000","last_unstale":"2024-03-06T07:32:32.379780+0000","last_undegraded":"2024-03-06T07:32:32.379780+0000","last_fullsized":"2024-03-06T07:32:32.379780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:12.948560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379866+0000","last_change":"2024-03-06T07:32:30.345874+0000","last_active":"2024-03-06T07:32:32.379866+0000","last_peered":"2024-03-06T07:32:32.379866+0000","last_clean":"2024-03-06T07:32:32.379866+0000","last_became_active":"2024-03-06T07:32:30.345536+0000","last_became_peered":"2024-03-06T07:32:30.345536+0000","last_unstale":"2024-03-06T07:32:32.379866+0000","last_undegraded":"2024-03-06T07:32:32.379866+0000","last_fullsized":"2024-03-06T07:32:32.379866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:02.058897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428790+0000","last_change":"2024-03-06T07:32:31.389189+0000","last_active":"2024-03-06T07:32:32.428790+0000","last_peered":"2024-03-06T07:32:32.428790+0000","last_clean":"2024-03-06T07:32:32.428790+0000","last_became_active":"2024-03-06T07:32:31.388860+0000","last_became_peered":"2024-03-06T07:32:31.388860+0000","last_unstale":"2024-03-06T07:32:32.428790+0000","last_undegraded":"2024-03-06T07:32:32.428790+0000","last_fullsized":"2024-03-06T07:32:32.428790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:04.514863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020727+0000","last_change":"2024-03-06T07:32:30.343999+0000","last_active":"2024-03-06T07:32:33.020727+0000","last_peered":"2024-03-06T07:32:33.020727+0000","last_clean":"2024-03-06T07:32:33.020727+0000","last_became_active":"2024-03-06T07:32:30.343545+0000","last_became_peered":"2024-03-06T07:32:30.343545+0000","last_unstale":"2024-03-06T07:32:33.020727+0000","last_undegraded":"2024-03-06T07:32:33.020727+0000","last_fullsized":"2024-03-06T07:32:33.020727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:05.108408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022571+0000","last_change":"2024-03-06T07:32:31.353535+0000","last_active":"2024-03-06T07:32:33.022571+0000","last_peered":"2024-03-06T07:32:33.022571+0000","last_clean":"2024-03-06T07:32:33.022571+0000","last_became_active":"2024-03-06T07:32:31.353104+0000","last_became_peered":"2024-03-06T07:32:31.353104+0000","last_unstale":"2024-03-06T07:32:33.022571+0000","last_undegraded":"2024-03-06T07:32:33.022571+0000","last_fullsized":"2024-03-06T07:32:33.022571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:48.018944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345759+0000","last_change":"2024-03-06T07:32:30.343072+0000","last_active":"2024-03-06T07:32:32.345759+0000","last_peered":"2024-03-06T07:32:32.345759+0000","last_clean":"2024-03-06T07:32:32.345759+0000","last_became_active":"2024-03-06T07:32:30.342779+0000","last_became_peered":"2024-03-06T07:32:30.342779+0000","last_unstale":"2024-03-06T07:32:32.345759+0000","last_undegraded":"2024-03-06T07:32:32.345759+0000","last_fullsized":"2024-03-06T07:32:32.345759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:19.775303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427737+0000","last_change":"2024-03-06T07:32:31.379277+0000","last_active":"2024-03-06T07:32:32.427737+0000","last_peered":"2024-03-06T07:32:32.427737+0000","last_clean":"2024-03-06T07:32:32.427737+0000","last_became_active":"2024-03-06T07:32:31.378879+0000","last_became_peered":"2024-03-06T07:32:31.378879+0000","last_unstale":"2024-03-06T07:32:32.427737+0000","last_undegraded":"2024-03-06T07:32:32.427737+0000","last_fullsized":"2024-03-06T07:32:32.427737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:08.722443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427867+0000","last_change":"2024-03-06T07:32:30.343667+0000","last_active":"2024-03-06T07:32:32.427867+0000","last_peered":"2024-03-06T07:32:32.427867+0000","last_clean":"2024-03-06T07:32:32.427867+0000","last_became_active":"2024-03-06T07:32:30.343135+0000","last_became_peered":"2024-03-06T07:32:30.343135+0000","last_unstale":"2024-03-06T07:32:32.427867+0000","last_undegraded":"2024-03-06T07:32:32.427867+0000","last_fullsized":"2024-03-06T07:32:32.427867+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:11.934557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868693+0000","last_change":"2024-03-06T07:32:31.394532+0000","last_active":"2024-03-06T07:32:32.868693+0000","last_peered":"2024-03-06T07:32:32.868693+0000","last_clean":"2024-03-06T07:32:32.868693+0000","last_became_active":"2024-03-06T07:32:31.394216+0000","last_became_peered":"2024-03-06T07:32:31.394216+0000","last_unstale":"2024-03-06T07:32:32.868693+0000","last_undegraded":"2024-03-06T07:32:32.868693+0000","last_fullsized":"2024-03-06T07:32:32.868693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:10.877538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428356+0000","last_change":"2024-03-06T07:32:30.351683+0000","last_active":"2024-03-06T07:32:32.428356+0000","last_peered":"2024-03-06T07:32:32.428356+0000","last_clean":"2024-03-06T07:32:32.428356+0000","last_became_active":"2024-03-06T07:32:30.350553+0000","last_became_peered":"2024-03-06T07:32:30.350553+0000","last_unstale":"2024-03-06T07:32:32.428356+0000","last_undegraded":"2024-03-06T07:32:32.428356+0000","last_fullsized":"2024-03-06T07:32:32.428356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:28.215419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":257,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342345+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:32:32.342345+0000","last_peered":"2024-03-06T07:32:32.342345+0000","last_clean":"2024-03-06T07:32:32.342345+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:32:32.342345+0000","last_undegraded":"2024-03-06T07:32:32.342345+0000","last_fullsized":"2024-03-06T07:32:32.342345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":937154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":835584,"data_stored":346262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":819200,"data_stored":339960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":831488,"data_stored":346240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":339982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":347782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36156,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335684,"statfs":{"total":96636764160,"available":96599740416,"internally_reserved":0,"allocated":835584,"data_stored":342098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1421312,"data_stored":936538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:35.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.125+0000 7f25ef200640 1 Processor -- start 2024-03-06T07:32:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.126+0000 7f25ef200640 1 -- start start 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.128+0000 7f25ef200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25f015dec0 0x7f25f00ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.128+0000 7f25ef200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f00cf460 0x7f25f00b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ef200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00d1120 con 0x7f25f015e2a0 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ef200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f00b0ea0 con 0x7f25f015dec0 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ef200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00d1400 con 0x7f25f00cf460 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25e7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f25f015e2a0 0x7f25f00ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37020/0 (socket says 172.21.15.32:37020) 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25e7e00640 1 -- 172.21.15.32:0/3809145293 learned_addr learned my addr 172.21.15.32:0/3809145293 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ede00640 1 --2- 172.21.15.32:0/3809145293 >> v2:172.21.15.32:3300/0 conn(0x7f25f015dec0 0x7f25f00ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ee800640 1 --2- 172.21.15.32:0/3809145293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f00cf460 0x7f25f00b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ede00640 1 -- 172.21.15.32:0/3809145293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f00cf460 msgr2=0x7f25f00b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ede00640 1 --2- 172.21.15.32:0/3809145293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f00cf460 0x7f25f00b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ede00640 1 -- 172.21.15.32:0/3809145293 >> v1:172.21.15.32:6789/0 conn(0x7f25f015e2a0 legacy=0x7f25f00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ede00640 1 -- 172.21.15.32:0/3809145293 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f00b0b30 con 0x7f25f015dec0 2024-03-06T07:32:36.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.129+0000 7f25ed400640 1 -- 172.21.15.32:0/3809145293 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277182352 0 0) 0x7f25f00d1120 con 0x7f25f015e2a0 2024-03-06T07:32:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.131+0000 7f25ede00640 1 --2- 172.21.15.32:0/3809145293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f015dec0 0x7f25f00ce810 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f25e00097a0 tx=0x7f25e002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5367c91bb7ead23f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.131+0000 7f25ed400640 1 -- 172.21.15.32:0/3809145293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0035020 con 0x7f25f015dec0 2024-03-06T07:32:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.131+0000 7f25ed400640 1 -- 172.21.15.32:0/3809145293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0031810 con 0x7f25f015dec0 2024-03-06T07:32:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.131+0000 7f25ed400640 1 -- 172.21.15.32:0/3809145293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0031be0 con 0x7f25f015dec0 2024-03-06T07:32:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 -- 172.21.15.32:0/3809145293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f015dec0 msgr2=0x7f25f00ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 --2- 172.21.15.32:0/3809145293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f015dec0 0x7f25f00ce810 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f25e00097a0 tx=0x7f25e002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 -- 172.21.15.32:0/3809145293 shutdown_connections 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 --2- 172.21.15.32:0/3809145293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f00cf460 0x7f25f00b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 --2- 172.21.15.32:0/3809145293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f015dec0 0x7f25f00ce810 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 -- 172.21.15.32:0/3809145293 >> 172.21.15.32:0/3809145293 conn(0x7f25f00d3d40 msgr2=0x7f25f00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 -- 172.21.15.32:0/3809145293 shutdown_connections 2024-03-06T07:32:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.132+0000 7f25ef200640 1 -- 172.21.15.32:0/3809145293 wait complete. 2024-03-06T07:32:36.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.135+0000 7f25ef200640 1 Processor -- start 2024-03-06T07:32:36.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.135+0000 7f25ef200640 1 -- start start 2024-03-06T07:32:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.136+0000 7f25ef200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 0x7f25f014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ef200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f015dec0 0x7f25f014a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ef200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25f015e2a0 0x7f25f01605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ef200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00d0ec0 con 0x7f25f00cf460 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ef200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00d0d40 con 0x7f25f015e2a0 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ef200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f00b1870 con 0x7f25f015dec0 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 0x7f25f014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 0x7f25f014d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35682/0 (socket says 172.21.15.32:35682) 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 -- 172.21.15.32:0/2705767217 learned_addr learned my addr 172.21.15.32:0/2705767217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 -- 172.21.15.32:0/2705767217 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f015dec0 msgr2=0x7f25f014a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f015dec0 0x7f25f014a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 -- 172.21.15.32:0/2705767217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25f015e2a0 msgr2=0x7f25f01605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25f015e2a0 0x7f25f01605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.137+0000 7f25ede00640 1 -- 172.21.15.32:0/2705767217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f014ae70 con 0x7f25f00cf460 2024-03-06T07:32:36.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.138+0000 7f25ede00640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 0x7f25f014d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f25e0000c00 tx=0x7f25e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.138+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0035050 con 0x7f25f00cf460 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.138+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e003bcf0 con 0x7f25f00cf460 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.138+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0004250 con 0x7f25f00cf460 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.138+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f014aaa0 con 0x7f25f00cf460 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.138+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f0161130 con 0x7f25f00cf460 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.139+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f014b040 con 0x7f25f00cf460 2024-03-06T07:32:36.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.140+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25e0038020 con 0x7f25f00cf460 2024-03-06T07:32:36.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.142+0000 7f25e6000640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25cc06b500 0x7f25cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.142+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25e00ac270 con 0x7f25f00cf460 2024-03-06T07:32:36.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.142+0000 7f25e7e00640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25cc06b500 0x7f25cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.143+0000 7f25e7e00640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25cc06b500 0x7f25cc06d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f25d8007fd0 tx=0x7f25d8007960 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.146+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25e0040070 con 0x7f25f00cf460 2024-03-06T07:32:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.342+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25f00d44a0 con 0x7f25cc06b500 2024-03-06T07:32:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.355+0000 7f25e6000640 1 -- 172.21.15.32:0/2705767217 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252071 (secure 0 0 0) 0x7f25f00d44a0 con 0x7f25cc06b500 2024-03-06T07:32:36.360 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:32:36.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25cc06b500 msgr2=0x7f25cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25cc06b500 0x7f25cc06d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f25d8007fd0 tx=0x7f25d8007960 comp rx=0 tx=0).stop 2024-03-06T07:32:36.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 msgr2=0x7f25f014d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 0x7f25f014d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f25e0000c00 tx=0x7f25e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 shutdown_connections 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25cc06b500 0x7f25cc06d9c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25f015e2a0 0x7f25f01605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25f015dec0 0x7f25f014a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 --2- 172.21.15.32:0/2705767217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25f00cf460 0x7f25f014d9c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 >> 172.21.15.32:0/2705767217 conn(0x7f25f00d3d40 msgr2=0x7f25f006ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 shutdown_connections 2024-03-06T07:32:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.360+0000 7f25ef200640 1 -- 172.21.15.32:0/2705767217 wait complete. 2024-03-06T07:32:36.380 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-03-06T07:32:36.252130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265432,"kb_used_data":7652,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754709288,"statfs":{"total":773094113280,"available":772822310912,"internally_reserved":0,"allocated":7835648,"data_stored":3936016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001945"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344800+0000","last_change":"2024-03-06T07:32:31.376227+0000","last_active":"2024-03-06T07:32:32.344800+0000","last_peered":"2024-03-06T07:32:32.344800+0000","last_clean":"2024-03-06T07:32:32.344800+0000","last_became_active":"2024-03-06T07:32:31.375565+0000","last_became_peered":"2024-03-06T07:32:31.375565+0000","last_unstale":"2024-03-06T07:32:32.344800+0000","last_undegraded":"2024-03-06T07:32:32.344800+0000","last_fullsized":"2024-03-06T07:32:32.344800+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:35.466204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428460+0000","last_change":"2024-03-06T07:32:31.398455+0000","last_active":"2024-03-06T07:32:32.428460+0000","last_peered":"2024-03-06T07:32:32.428460+0000","last_clean":"2024-03-06T07:32:32.428460+0000","last_became_active":"2024-03-06T07:32:31.398095+0000","last_became_peered":"2024-03-06T07:32:31.398095+0000","last_unstale":"2024-03-06T07:32:32.428460+0000","last_undegraded":"2024-03-06T07:32:32.428460+0000","last_fullsized":"2024-03-06T07:32:32.428460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:56.024432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868770+0000","last_change":"2024-03-06T07:32:31.365256+0000","last_active":"2024-03-06T07:32:32.868770+0000","last_peered":"2024-03-06T07:32:32.868770+0000","last_clean":"2024-03-06T07:32:32.868770+0000","last_became_active":"2024-03-06T07:32:31.364958+0000","last_became_peered":"2024-03-06T07:32:31.364958+0000","last_unstale":"2024-03-06T07:32:32.868770+0000","last_undegraded":"2024-03-06T07:32:32.868770+0000","last_fullsized":"2024-03-06T07:32:32.868770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:21.267960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022143+0000","last_change":"2024-03-06T07:32:31.396262+0000","last_active":"2024-03-06T07:32:33.022143+0000","last_peered":"2024-03-06T07:32:33.022143+0000","last_clean":"2024-03-06T07:32:33.022143+0000","last_became_active":"2024-03-06T07:32:31.395957+0000","last_became_peered":"2024-03-06T07:32:31.395957+0000","last_unstale":"2024-03-06T07:32:33.022143+0000","last_undegraded":"2024-03-06T07:32:33.022143+0000","last_fullsized":"2024-03-06T07:32:33.022143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:21.004750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379759+0000","last_change":"2024-03-06T07:32:31.375542+0000","last_active":"2024-03-06T07:32:32.379759+0000","last_peered":"2024-03-06T07:32:32.379759+0000","last_clean":"2024-03-06T07:32:32.379759+0000","last_became_active":"2024-03-06T07:32:31.375123+0000","last_became_peered":"2024-03-06T07:32:31.375123+0000","last_unstale":"2024-03-06T07:32:32.379759+0000","last_undegraded":"2024-03-06T07:32:32.379759+0000","last_fullsized":"2024-03-06T07:32:32.379759+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:06.122411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427620+0000","last_change":"2024-03-06T07:32:31.398861+0000","last_active":"2024-03-06T07:32:32.427620+0000","last_peered":"2024-03-06T07:32:32.427620+0000","last_clean":"2024-03-06T07:32:32.427620+0000","last_became_active":"2024-03-06T07:32:31.397983+0000","last_became_peered":"2024-03-06T07:32:31.397983+0000","last_unstale":"2024-03-06T07:32:32.427620+0000","last_undegraded":"2024-03-06T07:32:32.427620+0000","last_fullsized":"2024-03-06T07:32:32.427620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:56.197309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020809+0000","last_change":"2024-03-06T07:32:31.381519+0000","last_active":"2024-03-06T07:32:33.020809+0000","last_peered":"2024-03-06T07:32:33.020809+0000","last_clean":"2024-03-06T07:32:33.020809+0000","last_became_active":"2024-03-06T07:32:31.381144+0000","last_became_peered":"2024-03-06T07:32:31.381144+0000","last_unstale":"2024-03-06T07:32:33.020809+0000","last_undegraded":"2024-03-06T07:32:33.020809+0000","last_fullsized":"2024-03-06T07:32:33.020809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:30.599345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427442+0000","last_change":"2024-03-06T07:32:31.385922+0000","last_active":"2024-03-06T07:32:32.427442+0000","last_peered":"2024-03-06T07:32:32.427442+0000","last_clean":"2024-03-06T07:32:32.427442+0000","last_became_active":"2024-03-06T07:32:31.382343+0000","last_became_peered":"2024-03-06T07:32:31.382343+0000","last_unstale":"2024-03-06T07:32:32.427442+0000","last_undegraded":"2024-03-06T07:32:32.427442+0000","last_fullsized":"2024-03-06T07:32:32.427442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:14.700055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380271+0000","last_change":"2024-03-06T07:32:31.372688+0000","last_active":"2024-03-06T07:32:32.380271+0000","last_peered":"2024-03-06T07:32:32.380271+0000","last_clean":"2024-03-06T07:32:32.380271+0000","last_became_active":"2024-03-06T07:32:31.367541+0000","last_became_peered":"2024-03-06T07:32:31.367541+0000","last_unstale":"2024-03-06T07:32:32.380271+0000","last_undegraded":"2024-03-06T07:32:32.380271+0000","last_fullsized":"2024-03-06T07:32:32.380271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:49.904033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346551+0000","last_change":"2024-03-06T07:32:31.385665+0000","last_active":"2024-03-06T07:32:32.346551+0000","last_peered":"2024-03-06T07:32:32.346551+0000","last_clean":"2024-03-06T07:32:32.346551+0000","last_became_active":"2024-03-06T07:32:31.383642+0000","last_became_peered":"2024-03-06T07:32:31.383642+0000","last_unstale":"2024-03-06T07:32:32.346551+0000","last_undegraded":"2024-03-06T07:32:32.346551+0000","last_fullsized":"2024-03-06T07:32:32.346551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:35.177998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345034+0000","last_change":"2024-03-06T07:32:31.373392+0000","last_active":"2024-03-06T07:32:32.345034+0000","last_peered":"2024-03-06T07:32:32.345034+0000","last_clean":"2024-03-06T07:32:32.345034+0000","last_became_active":"2024-03-06T07:32:31.373035+0000","last_became_peered":"2024-03-06T07:32:31.373035+0000","last_unstale":"2024-03-06T07:32:32.345034+0000","last_undegraded":"2024-03-06T07:32:32.345034+0000","last_fullsized":"2024-03-06T07:32:32.345034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:13.161870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021452+0000","last_change":"2024-03-06T07:32:31.369325+0000","last_active":"2024-03-06T07:32:33.021452+0000","last_peered":"2024-03-06T07:32:33.021452+0000","last_clean":"2024-03-06T07:32:33.021452+0000","last_became_active":"2024-03-06T07:32:31.368709+0000","last_became_peered":"2024-03-06T07:32:31.368709+0000","last_unstale":"2024-03-06T07:32:33.021452+0000","last_undegraded":"2024-03-06T07:32:33.021452+0000","last_fullsized":"2024-03-06T07:32:33.021452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:59.870863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022564+0000","last_change":"2024-03-06T07:32:31.391584+0000","last_active":"2024-03-06T07:32:33.022564+0000","last_peered":"2024-03-06T07:32:33.022564+0000","last_clean":"2024-03-06T07:32:33.022564+0000","last_became_active":"2024-03-06T07:32:31.390388+0000","last_became_peered":"2024-03-06T07:32:31.390388+0000","last_unstale":"2024-03-06T07:32:33.022564+0000","last_undegraded":"2024-03-06T07:32:33.022564+0000","last_fullsized":"2024-03-06T07:32:33.022564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:22.386779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379621+0000","last_change":"2024-03-06T07:32:31.381169+0000","last_active":"2024-03-06T07:32:32.379621+0000","last_peered":"2024-03-06T07:32:32.379621+0000","last_clean":"2024-03-06T07:32:32.379621+0000","last_became_active":"2024-03-06T07:32:31.380457+0000","last_became_peered":"2024-03-06T07:32:31.380457+0000","last_unstale":"2024-03-06T07:32:32.379621+0000","last_undegraded":"2024-03-06T07:32:32.379621+0000","last_fullsized":"2024-03-06T07:32:32.379621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:51.366649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021099+0000","last_change":"2024-03-06T07:32:31.371980+0000","last_active":"2024-03-06T07:32:33.021099+0000","last_peered":"2024-03-06T07:32:33.021099+0000","last_clean":"2024-03-06T07:32:33.021099+0000","last_became_active":"2024-03-06T07:32:31.371665+0000","last_became_peered":"2024-03-06T07:32:31.371665+0000","last_unstale":"2024-03-06T07:32:33.021099+0000","last_undegraded":"2024-03-06T07:32:33.021099+0000","last_fullsized":"2024-03-06T07:32:33.021099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346137+0000","last_change":"2024-03-06T07:32:31.386413+0000","last_active":"2024-03-06T07:32:32.346137+0000","last_peered":"2024-03-06T07:32:32.346137+0000","last_clean":"2024-03-06T07:32:32.346137+0000","last_became_active":"2024-03-06T07:32:31.383714+0000","last_became_peered":"2024-03-06T07:32:31.383714+0000","last_unstale":"2024-03-06T07:32:32.346137+0000","last_undegraded":"2024-03-06T07:32:32.346137+0000","last_fullsized":"2024-03-06T07:32:32.346137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:49.223302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380439+0000","last_change":"2024-03-06T07:32:31.369605+0000","last_active":"2024-03-06T07:32:32.380439+0000","last_peered":"2024-03-06T07:32:32.380439+0000","last_clean":"2024-03-06T07:32:32.380439+0000","last_became_active":"2024-03-06T07:32:31.369101+0000","last_became_peered":"2024-03-06T07:32:31.369101+0000","last_unstale":"2024-03-06T07:32:32.380439+0000","last_undegraded":"2024-03-06T07:32:32.380439+0000","last_fullsized":"2024-03-06T07:32:32.380439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:24.128646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021707+0000","last_change":"2024-03-06T07:32:31.389516+0000","last_active":"2024-03-06T07:32:33.021707+0000","last_peered":"2024-03-06T07:32:33.021707+0000","last_clean":"2024-03-06T07:32:33.021707+0000","last_became_active":"2024-03-06T07:32:31.389085+0000","last_became_peered":"2024-03-06T07:32:31.389085+0000","last_unstale":"2024-03-06T07:32:33.021707+0000","last_undegraded":"2024-03-06T07:32:33.021707+0000","last_fullsized":"2024-03-06T07:32:33.021707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:53.766174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428267+0000","last_change":"2024-03-06T07:32:31.380364+0000","last_active":"2024-03-06T07:32:32.428267+0000","last_peered":"2024-03-06T07:32:32.428267+0000","last_clean":"2024-03-06T07:32:32.428267+0000","last_became_active":"2024-03-06T07:32:31.380071+0000","last_became_peered":"2024-03-06T07:32:31.380071+0000","last_unstale":"2024-03-06T07:32:32.428267+0000","last_undegraded":"2024-03-06T07:32:32.428267+0000","last_fullsized":"2024-03-06T07:32:32.428267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:59.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022541+0000","last_change":"2024-03-06T07:32:31.392008+0000","last_active":"2024-03-06T07:32:33.022541+0000","last_peered":"2024-03-06T07:32:33.022541+0000","last_clean":"2024-03-06T07:32:33.022541+0000","last_became_active":"2024-03-06T07:32:31.391504+0000","last_became_peered":"2024-03-06T07:32:31.391504+0000","last_unstale":"2024-03-06T07:32:33.022541+0000","last_undegraded":"2024-03-06T07:32:33.022541+0000","last_fullsized":"2024-03-06T07:32:33.022541+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:03.603511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346116+0000","last_change":"2024-03-06T07:32:31.383795+0000","last_active":"2024-03-06T07:32:32.346116+0000","last_peered":"2024-03-06T07:32:32.346116+0000","last_clean":"2024-03-06T07:32:32.346116+0000","last_became_active":"2024-03-06T07:32:31.383477+0000","last_became_peered":"2024-03-06T07:32:31.383477+0000","last_unstale":"2024-03-06T07:32:32.346116+0000","last_undegraded":"2024-03-06T07:32:32.346116+0000","last_fullsized":"2024-03-06T07:32:32.346116+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:56.024432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345932+0000","last_change":"2024-03-06T07:32:31.385058+0000","last_active":"2024-03-06T07:32:32.345932+0000","last_peered":"2024-03-06T07:32:32.345932+0000","last_clean":"2024-03-06T07:32:32.345932+0000","last_became_active":"2024-03-06T07:32:31.384664+0000","last_became_peered":"2024-03-06T07:32:31.384664+0000","last_unstale":"2024-03-06T07:32:32.345932+0000","last_undegraded":"2024-03-06T07:32:32.345932+0000","last_fullsized":"2024-03-06T07:32:32.345932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342819+0000","last_change":"2024-03-06T07:32:31.382364+0000","last_active":"2024-03-06T07:32:32.342819+0000","last_peered":"2024-03-06T07:32:32.342819+0000","last_clean":"2024-03-06T07:32:32.342819+0000","last_became_active":"2024-03-06T07:32:31.382036+0000","last_became_peered":"2024-03-06T07:32:31.382036+0000","last_unstale":"2024-03-06T07:32:32.342819+0000","last_undegraded":"2024-03-06T07:32:32.342819+0000","last_fullsized":"2024-03-06T07:32:32.342819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:22.386779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342679+0000","last_change":"2024-03-06T07:32:31.390221+0000","last_active":"2024-03-06T07:32:32.342679+0000","last_peered":"2024-03-06T07:32:32.342679+0000","last_clean":"2024-03-06T07:32:32.342679+0000","last_became_active":"2024-03-06T07:32:31.389885+0000","last_became_peered":"2024-03-06T07:32:31.389885+0000","last_unstale":"2024-03-06T07:32:32.342679+0000","last_undegraded":"2024-03-06T07:32:32.342679+0000","last_fullsized":"2024-03-06T07:32:32.342679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:03.603511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.347083+0000","last_change":"2024-03-06T07:32:31.393260+0000","last_active":"2024-03-06T07:32:32.347083+0000","last_peered":"2024-03-06T07:32:32.347083+0000","last_clean":"2024-03-06T07:32:32.347083+0000","last_became_active":"2024-03-06T07:32:31.392914+0000","last_became_peered":"2024-03-06T07:32:31.392914+0000","last_unstale":"2024-03-06T07:32:32.347083+0000","last_undegraded":"2024-03-06T07:32:32.347083+0000","last_fullsized":"2024-03-06T07:32:32.347083+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:10.997851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021450+0000","last_change":"2024-03-06T07:32:31.394397+0000","last_active":"2024-03-06T07:32:33.021450+0000","last_peered":"2024-03-06T07:32:33.021450+0000","last_clean":"2024-03-06T07:32:33.021450+0000","last_became_active":"2024-03-06T07:32:31.394100+0000","last_became_peered":"2024-03-06T07:32:31.394100+0000","last_unstale":"2024-03-06T07:32:33.021450+0000","last_undegraded":"2024-03-06T07:32:33.021450+0000","last_fullsized":"2024-03-06T07:32:33.021450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:38.416113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.869086+0000","last_change":"2024-03-06T07:32:31.378434+0000","last_active":"2024-03-06T07:32:32.869086+0000","last_peered":"2024-03-06T07:32:32.869086+0000","last_clean":"2024-03-06T07:32:32.869086+0000","last_became_active":"2024-03-06T07:32:31.375917+0000","last_became_peered":"2024-03-06T07:32:31.375917+0000","last_unstale":"2024-03-06T07:32:32.869086+0000","last_undegraded":"2024-03-06T07:32:32.869086+0000","last_fullsized":"2024-03-06T07:32:32.869086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:25.190301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346433+0000","last_change":"2024-03-06T07:32:31.380146+0000","last_active":"2024-03-06T07:32:32.346433+0000","last_peered":"2024-03-06T07:32:32.346433+0000","last_clean":"2024-03-06T07:32:32.346433+0000","last_became_active":"2024-03-06T07:32:31.379834+0000","last_became_peered":"2024-03-06T07:32:31.379834+0000","last_unstale":"2024-03-06T07:32:32.346433+0000","last_undegraded":"2024-03-06T07:32:32.346433+0000","last_fullsized":"2024-03-06T07:32:32.346433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.273035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868942+0000","last_change":"2024-03-06T07:32:31.387684+0000","last_active":"2024-03-06T07:32:32.868942+0000","last_peered":"2024-03-06T07:32:32.868942+0000","last_clean":"2024-03-06T07:32:32.868942+0000","last_became_active":"2024-03-06T07:32:31.387328+0000","last_became_peered":"2024-03-06T07:32:31.387328+0000","last_unstale":"2024-03-06T07:32:32.868942+0000","last_undegraded":"2024-03-06T07:32:32.868942+0000","last_fullsized":"2024-03-06T07:32:32.868942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:20.312459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021081+0000","last_change":"2024-03-06T07:32:31.395830+0000","last_active":"2024-03-06T07:32:33.021081+0000","last_peered":"2024-03-06T07:32:33.021081+0000","last_clean":"2024-03-06T07:32:33.021081+0000","last_became_active":"2024-03-06T07:32:31.395513+0000","last_became_peered":"2024-03-06T07:32:31.395513+0000","last_unstale":"2024-03-06T07:32:33.021081+0000","last_undegraded":"2024-03-06T07:32:33.021081+0000","last_fullsized":"2024-03-06T07:32:33.021081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:44.615322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021246+0000","last_change":"2024-03-06T07:32:31.380757+0000","last_active":"2024-03-06T07:32:33.021246+0000","last_peered":"2024-03-06T07:32:33.021246+0000","last_clean":"2024-03-06T07:32:33.021246+0000","last_became_active":"2024-03-06T07:32:31.380410+0000","last_became_peered":"2024-03-06T07:32:31.380410+0000","last_unstale":"2024-03-06T07:32:33.021246+0000","last_undegraded":"2024-03-06T07:32:33.021246+0000","last_fullsized":"2024-03-06T07:32:33.021246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:14.183626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428566+0000","last_change":"2024-03-06T07:32:31.382207+0000","last_active":"2024-03-06T07:32:32.428566+0000","last_peered":"2024-03-06T07:32:32.428566+0000","last_clean":"2024-03-06T07:32:32.428566+0000","last_became_active":"2024-03-06T07:32:31.381761+0000","last_became_peered":"2024-03-06T07:32:31.381761+0000","last_unstale":"2024-03-06T07:32:32.428566+0000","last_undegraded":"2024-03-06T07:32:32.428566+0000","last_fullsized":"2024-03-06T07:32:32.428566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:49.223302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344348+0000","last_change":"2024-03-06T07:32:31.392122+0000","last_active":"2024-03-06T07:32:32.344348+0000","last_peered":"2024-03-06T07:32:32.344348+0000","last_clean":"2024-03-06T07:32:32.344348+0000","last_became_active":"2024-03-06T07:32:31.389788+0000","last_became_peered":"2024-03-06T07:32:31.389788+0000","last_unstale":"2024-03-06T07:32:32.344348+0000","last_undegraded":"2024-03-06T07:32:32.344348+0000","last_fullsized":"2024-03-06T07:32:32.344348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:47.267079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345232+0000","last_change":"2024-03-06T07:32:30.336032+0000","last_active":"2024-03-06T07:32:32.345232+0000","last_peered":"2024-03-06T07:32:32.345232+0000","last_clean":"2024-03-06T07:32:32.345232+0000","last_became_active":"2024-03-06T07:32:30.335160+0000","last_became_peered":"2024-03-06T07:32:30.335160+0000","last_unstale":"2024-03-06T07:32:32.345232+0000","last_undegraded":"2024-03-06T07:32:32.345232+0000","last_fullsized":"2024-03-06T07:32:32.345232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.983848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380451+0000","last_change":"2024-03-06T07:32:31.369395+0000","last_active":"2024-03-06T07:32:32.380451+0000","last_peered":"2024-03-06T07:32:32.380451+0000","last_clean":"2024-03-06T07:32:32.380451+0000","last_became_active":"2024-03-06T07:32:31.367173+0000","last_became_peered":"2024-03-06T07:32:31.367173+0000","last_unstale":"2024-03-06T07:32:32.380451+0000","last_undegraded":"2024-03-06T07:32:32.380451+0000","last_fullsized":"2024-03-06T07:32:32.380451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:29.229422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.023167+0000","last_change":"2024-03-06T07:32:30.334239+0000","last_active":"2024-03-06T07:32:33.023167+0000","last_peered":"2024-03-06T07:32:33.023167+0000","last_clean":"2024-03-06T07:32:33.023167+0000","last_became_active":"2024-03-06T07:32:30.333895+0000","last_became_peered":"2024-03-06T07:32:30.333895+0000","last_unstale":"2024-03-06T07:32:33.023167+0000","last_undegraded":"2024-03-06T07:32:33.023167+0000","last_fullsized":"2024-03-06T07:32:33.023167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.983848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428590+0000","last_change":"2024-03-06T07:32:31.395284+0000","last_active":"2024-03-06T07:32:32.428590+0000","last_peered":"2024-03-06T07:32:32.428590+0000","last_clean":"2024-03-06T07:32:32.428590+0000","last_became_active":"2024-03-06T07:32:31.395003+0000","last_became_peered":"2024-03-06T07:32:31.395003+0000","last_unstale":"2024-03-06T07:32:32.428590+0000","last_undegraded":"2024-03-06T07:32:32.428590+0000","last_fullsized":"2024-03-06T07:32:32.428590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.273035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346707+0000","last_change":"2024-03-06T07:32:30.352461+0000","last_active":"2024-03-06T07:32:32.346707+0000","last_peered":"2024-03-06T07:32:32.346707+0000","last_clean":"2024-03-06T07:32:32.346707+0000","last_became_active":"2024-03-06T07:32:30.352126+0000","last_became_peered":"2024-03-06T07:32:30.352126+0000","last_unstale":"2024-03-06T07:32:32.346707+0000","last_undegraded":"2024-03-06T07:32:32.346707+0000","last_fullsized":"2024-03-06T07:32:32.346707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:44.591233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344064+0000","last_change":"2024-03-06T07:32:31.377035+0000","last_active":"2024-03-06T07:32:32.344064+0000","last_peered":"2024-03-06T07:32:32.344064+0000","last_clean":"2024-03-06T07:32:32.344064+0000","last_became_active":"2024-03-06T07:32:31.373591+0000","last_became_peered":"2024-03-06T07:32:31.373591+0000","last_unstale":"2024-03-06T07:32:32.344064+0000","last_undegraded":"2024-03-06T07:32:32.344064+0000","last_fullsized":"2024-03-06T07:32:32.344064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:51.068688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428740+0000","last_change":"2024-03-06T07:32:30.343983+0000","last_active":"2024-03-06T07:32:32.428740+0000","last_peered":"2024-03-06T07:32:32.428740+0000","last_clean":"2024-03-06T07:32:32.428740+0000","last_became_active":"2024-03-06T07:32:30.343601+0000","last_became_peered":"2024-03-06T07:32:30.343601+0000","last_unstale":"2024-03-06T07:32:32.428740+0000","last_undegraded":"2024-03-06T07:32:32.428740+0000","last_fullsized":"2024-03-06T07:32:32.428740+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:07.366173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021185+0000","last_change":"2024-03-06T07:32:31.381007+0000","last_active":"2024-03-06T07:32:33.021185+0000","last_peered":"2024-03-06T07:32:33.021185+0000","last_clean":"2024-03-06T07:32:33.021185+0000","last_became_active":"2024-03-06T07:32:31.380668+0000","last_became_peered":"2024-03-06T07:32:31.380668+0000","last_unstale":"2024-03-06T07:32:33.021185+0000","last_undegraded":"2024-03-06T07:32:33.021185+0000","last_fullsized":"2024-03-06T07:32:33.021185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:34.396284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.869093+0000","last_change":"2024-03-06T07:32:30.352023+0000","last_active":"2024-03-06T07:32:32.869093+0000","last_peered":"2024-03-06T07:32:32.869093+0000","last_clean":"2024-03-06T07:32:32.869093+0000","last_became_active":"2024-03-06T07:32:30.351701+0000","last_became_peered":"2024-03-06T07:32:30.351701+0000","last_unstale":"2024-03-06T07:32:32.869093+0000","last_undegraded":"2024-03-06T07:32:32.869093+0000","last_fullsized":"2024-03-06T07:32:32.869093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:08.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346844+0000","last_change":"2024-03-06T07:32:31.394436+0000","last_active":"2024-03-06T07:32:32.346844+0000","last_peered":"2024-03-06T07:32:32.346844+0000","last_clean":"2024-03-06T07:32:32.346844+0000","last_became_active":"2024-03-06T07:32:31.393394+0000","last_became_peered":"2024-03-06T07:32:31.393394+0000","last_unstale":"2024-03-06T07:32:32.346844+0000","last_undegraded":"2024-03-06T07:32:32.346844+0000","last_fullsized":"2024-03-06T07:32:32.346844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:10.287413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427456+0000","last_change":"2024-03-06T07:32:30.350472+0000","last_active":"2024-03-06T07:32:32.427456+0000","last_peered":"2024-03-06T07:32:32.427456+0000","last_clean":"2024-03-06T07:32:32.427456+0000","last_became_active":"2024-03-06T07:32:30.350124+0000","last_became_peered":"2024-03-06T07:32:30.350124+0000","last_unstale":"2024-03-06T07:32:32.427456+0000","last_undegraded":"2024-03-06T07:32:32.427456+0000","last_fullsized":"2024-03-06T07:32:32.427456+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:48.890030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379321+0000","last_change":"2024-03-06T07:32:31.387251+0000","last_active":"2024-03-06T07:32:32.379321+0000","last_peered":"2024-03-06T07:32:32.379321+0000","last_clean":"2024-03-06T07:32:32.379321+0000","last_became_active":"2024-03-06T07:32:31.386851+0000","last_became_peered":"2024-03-06T07:32:31.386851+0000","last_unstale":"2024-03-06T07:32:32.379321+0000","last_undegraded":"2024-03-06T07:32:32.379321+0000","last_fullsized":"2024-03-06T07:32:32.379321+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:46.084945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021006+0000","last_change":"2024-03-06T07:32:30.343876+0000","last_active":"2024-03-06T07:32:33.021006+0000","last_peered":"2024-03-06T07:32:33.021006+0000","last_clean":"2024-03-06T07:32:33.021006+0000","last_became_active":"2024-03-06T07:32:30.343537+0000","last_became_peered":"2024-03-06T07:32:30.343537+0000","last_unstale":"2024-03-06T07:32:33.021006+0000","last_undegraded":"2024-03-06T07:32:33.021006+0000","last_fullsized":"2024-03-06T07:32:33.021006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:30.592256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021531+0000","last_change":"2024-03-06T07:32:31.388097+0000","last_active":"2024-03-06T07:32:33.021531+0000","last_peered":"2024-03-06T07:32:33.021531+0000","last_clean":"2024-03-06T07:32:33.021531+0000","last_became_active":"2024-03-06T07:32:31.387730+0000","last_became_peered":"2024-03-06T07:32:31.387730+0000","last_unstale":"2024-03-06T07:32:33.021531+0000","last_undegraded":"2024-03-06T07:32:33.021531+0000","last_fullsized":"2024-03-06T07:32:33.021531+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:31.900032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379255+0000","last_change":"2024-03-06T07:32:30.339673+0000","last_active":"2024-03-06T07:32:32.379255+0000","last_peered":"2024-03-06T07:32:32.379255+0000","last_clean":"2024-03-06T07:32:32.379255+0000","last_became_active":"2024-03-06T07:32:30.339390+0000","last_became_peered":"2024-03-06T07:32:30.339390+0000","last_unstale":"2024-03-06T07:32:32.379255+0000","last_undegraded":"2024-03-06T07:32:32.379255+0000","last_fullsized":"2024-03-06T07:32:32.379255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.127597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021772+0000","last_change":"2024-03-06T07:32:31.393140+0000","last_active":"2024-03-06T07:32:33.021772+0000","last_peered":"2024-03-06T07:32:33.021772+0000","last_clean":"2024-03-06T07:32:33.021772+0000","last_became_active":"2024-03-06T07:32:31.392848+0000","last_became_peered":"2024-03-06T07:32:31.392848+0000","last_unstale":"2024-03-06T07:32:33.021772+0000","last_undegraded":"2024-03-06T07:32:33.021772+0000","last_fullsized":"2024-03-06T07:32:33.021772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:48.300814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346795+0000","last_change":"2024-03-06T07:32:30.345557+0000","last_active":"2024-03-06T07:32:32.346795+0000","last_peered":"2024-03-06T07:32:32.346795+0000","last_clean":"2024-03-06T07:32:32.346795+0000","last_became_active":"2024-03-06T07:32:30.345233+0000","last_became_peered":"2024-03-06T07:32:30.345233+0000","last_unstale":"2024-03-06T07:32:32.346795+0000","last_undegraded":"2024-03-06T07:32:32.346795+0000","last_fullsized":"2024-03-06T07:32:32.346795+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:07.398886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345138+0000","last_change":"2024-03-06T07:32:31.388519+0000","last_active":"2024-03-06T07:32:32.345138+0000","last_peered":"2024-03-06T07:32:32.345138+0000","last_clean":"2024-03-06T07:32:32.345138+0000","last_became_active":"2024-03-06T07:32:31.386548+0000","last_became_peered":"2024-03-06T07:32:31.386548+0000","last_unstale":"2024-03-06T07:32:32.345138+0000","last_undegraded":"2024-03-06T07:32:32.345138+0000","last_fullsized":"2024-03-06T07:32:32.345138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:09.678529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428543+0000","last_change":"2024-03-06T07:32:30.352241+0000","last_active":"2024-03-06T07:32:32.428543+0000","last_peered":"2024-03-06T07:32:32.428543+0000","last_clean":"2024-03-06T07:32:32.428543+0000","last_became_active":"2024-03-06T07:32:30.351713+0000","last_became_peered":"2024-03-06T07:32:30.351713+0000","last_unstale":"2024-03-06T07:32:32.428543+0000","last_undegraded":"2024-03-06T07:32:32.428543+0000","last_fullsized":"2024-03-06T07:32:32.428543+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:52.752172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428406+0000","last_change":"2024-03-06T07:32:31.398523+0000","last_active":"2024-03-06T07:32:32.428406+0000","last_peered":"2024-03-06T07:32:32.428406+0000","last_clean":"2024-03-06T07:32:32.428406+0000","last_became_active":"2024-03-06T07:32:31.398239+0000","last_became_peered":"2024-03-06T07:32:31.398239+0000","last_unstale":"2024-03-06T07:32:32.428406+0000","last_undegraded":"2024-03-06T07:32:32.428406+0000","last_fullsized":"2024-03-06T07:32:32.428406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868848+0000","last_change":"2024-03-06T07:32:30.347740+0000","last_active":"2024-03-06T07:32:32.868848+0000","last_peered":"2024-03-06T07:32:32.868848+0000","last_clean":"2024-03-06T07:32:32.868848+0000","last_became_active":"2024-03-06T07:32:30.347359+0000","last_became_peered":"2024-03-06T07:32:30.347359+0000","last_unstale":"2024-03-06T07:32:32.868848+0000","last_undegraded":"2024-03-06T07:32:32.868848+0000","last_fullsized":"2024-03-06T07:32:32.868848+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:54.044872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379649+0000","last_change":"2024-03-06T07:32:31.369356+0000","last_active":"2024-03-06T07:32:32.379649+0000","last_peered":"2024-03-06T07:32:32.379649+0000","last_clean":"2024-03-06T07:32:32.379649+0000","last_became_active":"2024-03-06T07:32:31.366886+0000","last_became_peered":"2024-03-06T07:32:31.366886+0000","last_unstale":"2024-03-06T07:32:32.379649+0000","last_undegraded":"2024-03-06T07:32:32.379649+0000","last_fullsized":"2024-03-06T07:32:32.379649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:31.606258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021886+0000","last_change":"2024-03-06T07:32:30.333935+0000","last_active":"2024-03-06T07:32:33.021886+0000","last_peered":"2024-03-06T07:32:33.021886+0000","last_clean":"2024-03-06T07:32:33.021886+0000","last_became_active":"2024-03-06T07:32:30.333488+0000","last_became_peered":"2024-03-06T07:32:30.333488+0000","last_unstale":"2024-03-06T07:32:33.021886+0000","last_undegraded":"2024-03-06T07:32:33.021886+0000","last_fullsized":"2024-03-06T07:32:33.021886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:18.400240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342679+0000","last_change":"2024-03-06T07:32:31.373123+0000","last_active":"2024-03-06T07:32:32.342679+0000","last_peered":"2024-03-06T07:32:32.342679+0000","last_clean":"2024-03-06T07:32:32.342679+0000","last_became_active":"2024-03-06T07:32:31.372748+0000","last_became_peered":"2024-03-06T07:32:31.372748+0000","last_unstale":"2024-03-06T07:32:32.342679+0000","last_undegraded":"2024-03-06T07:32:32.342679+0000","last_fullsized":"2024-03-06T07:32:32.342679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:29.118305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021987+0000","last_change":"2024-03-06T07:32:30.341457+0000","last_active":"2024-03-06T07:32:33.021987+0000","last_peered":"2024-03-06T07:32:33.021987+0000","last_clean":"2024-03-06T07:32:33.021987+0000","last_became_active":"2024-03-06T07:32:30.341045+0000","last_became_peered":"2024-03-06T07:32:30.341045+0000","last_unstale":"2024-03-06T07:32:33.021987+0000","last_undegraded":"2024-03-06T07:32:33.021987+0000","last_fullsized":"2024-03-06T07:32:33.021987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:11.934557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020894+0000","last_change":"2024-03-06T07:32:31.391789+0000","last_active":"2024-03-06T07:32:33.020894+0000","last_peered":"2024-03-06T07:32:33.020894+0000","last_clean":"2024-03-06T07:32:33.020894+0000","last_became_active":"2024-03-06T07:32:31.391321+0000","last_became_peered":"2024-03-06T07:32:31.391321+0000","last_unstale":"2024-03-06T07:32:33.020894+0000","last_undegraded":"2024-03-06T07:32:33.020894+0000","last_fullsized":"2024-03-06T07:32:33.020894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:46.759541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427849+0000","last_change":"2024-03-06T07:32:30.348110+0000","last_active":"2024-03-06T07:32:32.427849+0000","last_peered":"2024-03-06T07:32:32.427849+0000","last_clean":"2024-03-06T07:32:32.427849+0000","last_became_active":"2024-03-06T07:32:30.345283+0000","last_became_peered":"2024-03-06T07:32:30.345283+0000","last_unstale":"2024-03-06T07:32:32.427849+0000","last_undegraded":"2024-03-06T07:32:32.427849+0000","last_fullsized":"2024-03-06T07:32:32.427849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:33.085495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021033+0000","last_change":"2024-03-06T07:32:31.384028+0000","last_active":"2024-03-06T07:32:33.021033+0000","last_peered":"2024-03-06T07:32:33.021033+0000","last_clean":"2024-03-06T07:32:33.021033+0000","last_became_active":"2024-03-06T07:32:31.383689+0000","last_became_peered":"2024-03-06T07:32:31.383689+0000","last_unstale":"2024-03-06T07:32:33.021033+0000","last_undegraded":"2024-03-06T07:32:33.021033+0000","last_fullsized":"2024-03-06T07:32:33.021033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:38.416113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021383+0000","last_change":"2024-03-06T07:32:30.348848+0000","last_active":"2024-03-06T07:32:33.021383+0000","last_peered":"2024-03-06T07:32:33.021383+0000","last_clean":"2024-03-06T07:32:33.021383+0000","last_became_active":"2024-03-06T07:32:30.348513+0000","last_became_peered":"2024-03-06T07:32:30.348513+0000","last_unstale":"2024-03-06T07:32:33.021383+0000","last_undegraded":"2024-03-06T07:32:33.021383+0000","last_fullsized":"2024-03-06T07:32:33.021383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:58.445649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379451+0000","last_change":"2024-03-06T07:32:31.367305+0000","last_active":"2024-03-06T07:32:32.379451+0000","last_peered":"2024-03-06T07:32:32.379451+0000","last_clean":"2024-03-06T07:32:32.379451+0000","last_became_active":"2024-03-06T07:32:31.366969+0000","last_became_peered":"2024-03-06T07:32:31.366969+0000","last_unstale":"2024-03-06T07:32:32.379451+0000","last_undegraded":"2024-03-06T07:32:32.379451+0000","last_fullsized":"2024-03-06T07:32:32.379451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:09.456105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345534+0000","last_change":"2024-03-06T07:32:30.346654+0000","last_active":"2024-03-06T07:32:32.345534+0000","last_peered":"2024-03-06T07:32:32.345534+0000","last_clean":"2024-03-06T07:32:32.345534+0000","last_became_active":"2024-03-06T07:32:30.344988+0000","last_became_peered":"2024-03-06T07:32:30.344988+0000","last_unstale":"2024-03-06T07:32:32.345534+0000","last_undegraded":"2024-03-06T07:32:32.345534+0000","last_fullsized":"2024-03-06T07:32:32.345534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:19.990747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346856+0000","last_change":"2024-03-06T07:32:31.380008+0000","last_active":"2024-03-06T07:32:32.346856+0000","last_peered":"2024-03-06T07:32:32.346856+0000","last_clean":"2024-03-06T07:32:32.346856+0000","last_became_active":"2024-03-06T07:32:31.379684+0000","last_became_peered":"2024-03-06T07:32:31.379684+0000","last_unstale":"2024-03-06T07:32:32.346856+0000","last_undegraded":"2024-03-06T07:32:32.346856+0000","last_fullsized":"2024-03-06T07:32:32.346856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:56.576735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428674+0000","last_change":"2024-03-06T07:32:30.350922+0000","last_active":"2024-03-06T07:32:32.428674+0000","last_peered":"2024-03-06T07:32:32.428674+0000","last_clean":"2024-03-06T07:32:32.428674+0000","last_became_active":"2024-03-06T07:32:30.350616+0000","last_became_peered":"2024-03-06T07:32:30.350616+0000","last_unstale":"2024-03-06T07:32:32.428674+0000","last_undegraded":"2024-03-06T07:32:32.428674+0000","last_fullsized":"2024-03-06T07:32:32.428674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:47.004941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345038+0000","last_change":"2024-03-06T07:32:31.380454+0000","last_active":"2024-03-06T07:32:32.345038+0000","last_peered":"2024-03-06T07:32:32.345038+0000","last_clean":"2024-03-06T07:32:32.345038+0000","last_became_active":"2024-03-06T07:32:31.375982+0000","last_became_peered":"2024-03-06T07:32:31.375982+0000","last_unstale":"2024-03-06T07:32:32.345038+0000","last_undegraded":"2024-03-06T07:32:32.345038+0000","last_fullsized":"2024-03-06T07:32:32.345038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:55.058875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.346884+0000","last_change":"2024-03-06T07:32:30.352788+0000","last_active":"2024-03-06T07:32:32.346884+0000","last_peered":"2024-03-06T07:32:32.346884+0000","last_clean":"2024-03-06T07:32:32.346884+0000","last_became_active":"2024-03-06T07:32:30.352354+0000","last_became_peered":"2024-03-06T07:32:30.352354+0000","last_unstale":"2024-03-06T07:32:32.346884+0000","last_undegraded":"2024-03-06T07:32:32.346884+0000","last_fullsized":"2024-03-06T07:32:32.346884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:47.004941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021675+0000","last_change":"2024-03-06T07:32:31.369095+0000","last_active":"2024-03-06T07:32:33.021675+0000","last_peered":"2024-03-06T07:32:33.021675+0000","last_clean":"2024-03-06T07:32:33.021675+0000","last_became_active":"2024-03-06T07:32:31.368344+0000","last_became_peered":"2024-03-06T07:32:31.368344+0000","last_unstale":"2024-03-06T07:32:33.021675+0000","last_undegraded":"2024-03-06T07:32:33.021675+0000","last_fullsized":"2024-03-06T07:32:33.021675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:34.322368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428386+0000","last_change":"2024-03-06T07:32:30.354128+0000","last_active":"2024-03-06T07:32:32.428386+0000","last_peered":"2024-03-06T07:32:32.428386+0000","last_clean":"2024-03-06T07:32:32.428386+0000","last_became_active":"2024-03-06T07:32:30.353264+0000","last_became_peered":"2024-03-06T07:32:30.353264+0000","last_unstale":"2024-03-06T07:32:32.428386+0000","last_undegraded":"2024-03-06T07:32:32.428386+0000","last_fullsized":"2024-03-06T07:32:32.428386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:33.308365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380100+0000","last_change":"2024-03-06T07:32:31.387217+0000","last_active":"2024-03-06T07:32:32.380100+0000","last_peered":"2024-03-06T07:32:32.380100+0000","last_clean":"2024-03-06T07:32:32.380100+0000","last_became_active":"2024-03-06T07:32:31.386934+0000","last_became_peered":"2024-03-06T07:32:31.386934+0000","last_unstale":"2024-03-06T07:32:32.380100+0000","last_undegraded":"2024-03-06T07:32:32.380100+0000","last_fullsized":"2024-03-06T07:32:32.380100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:19.414243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.343559+0000","last_change":"2024-03-06T07:32:30.349715+0000","last_active":"2024-03-06T07:32:32.343559+0000","last_peered":"2024-03-06T07:32:32.343559+0000","last_clean":"2024-03-06T07:32:32.343559+0000","last_became_active":"2024-03-06T07:32:30.349322+0000","last_became_peered":"2024-03-06T07:32:30.349322+0000","last_unstale":"2024-03-06T07:32:32.343559+0000","last_undegraded":"2024-03-06T07:32:32.343559+0000","last_fullsized":"2024-03-06T07:32:32.343559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:55.562732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.869137+0000","last_change":"2024-03-06T07:32:31.358476+0000","last_active":"2024-03-06T07:32:32.869137+0000","last_peered":"2024-03-06T07:32:32.869137+0000","last_clean":"2024-03-06T07:32:32.869137+0000","last_became_active":"2024-03-06T07:32:31.356609+0000","last_became_peered":"2024-03-06T07:32:31.356609+0000","last_unstale":"2024-03-06T07:32:32.869137+0000","last_undegraded":"2024-03-06T07:32:32.869137+0000","last_fullsized":"2024-03-06T07:32:32.869137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:25.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428000+0000","last_change":"2024-03-06T07:32:30.348348+0000","last_active":"2024-03-06T07:32:32.428000+0000","last_peered":"2024-03-06T07:32:32.428000+0000","last_clean":"2024-03-06T07:32:32.428000+0000","last_became_active":"2024-03-06T07:32:30.347784+0000","last_became_peered":"2024-03-06T07:32:30.347784+0000","last_unstale":"2024-03-06T07:32:32.428000+0000","last_undegraded":"2024-03-06T07:32:32.428000+0000","last_fullsized":"2024-03-06T07:32:32.428000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:23.114643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868701+0000","last_change":"2024-03-06T07:32:31.385742+0000","last_active":"2024-03-06T07:32:32.868701+0000","last_peered":"2024-03-06T07:32:32.868701+0000","last_clean":"2024-03-06T07:32:32.868701+0000","last_became_active":"2024-03-06T07:32:31.385086+0000","last_became_peered":"2024-03-06T07:32:31.385086+0000","last_unstale":"2024-03-06T07:32:32.868701+0000","last_undegraded":"2024-03-06T07:32:32.868701+0000","last_fullsized":"2024-03-06T07:32:32.868701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:03.855456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021395+0000","last_change":"2024-03-06T07:32:30.348334+0000","last_active":"2024-03-06T07:32:33.021395+0000","last_peered":"2024-03-06T07:32:33.021395+0000","last_clean":"2024-03-06T07:32:33.021395+0000","last_became_active":"2024-03-06T07:32:30.345552+0000","last_became_peered":"2024-03-06T07:32:30.345552+0000","last_unstale":"2024-03-06T07:32:33.021395+0000","last_undegraded":"2024-03-06T07:32:33.021395+0000","last_fullsized":"2024-03-06T07:32:33.021395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:38.476626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868636+0000","last_change":"2024-03-06T07:32:31.382597+0000","last_active":"2024-03-06T07:32:32.868636+0000","last_peered":"2024-03-06T07:32:32.868636+0000","last_clean":"2024-03-06T07:32:32.868636+0000","last_became_active":"2024-03-06T07:32:31.382158+0000","last_became_peered":"2024-03-06T07:32:31.382158+0000","last_unstale":"2024-03-06T07:32:32.868636+0000","last_undegraded":"2024-03-06T07:32:32.868636+0000","last_fullsized":"2024-03-06T07:32:32.868636+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:20.971174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020665+0000","last_change":"2024-03-06T07:32:30.336135+0000","last_active":"2024-03-06T07:32:33.020665+0000","last_peered":"2024-03-06T07:32:33.020665+0000","last_clean":"2024-03-06T07:32:33.020665+0000","last_became_active":"2024-03-06T07:32:30.335743+0000","last_became_peered":"2024-03-06T07:32:30.335743+0000","last_unstale":"2024-03-06T07:32:33.020665+0000","last_undegraded":"2024-03-06T07:32:33.020665+0000","last_fullsized":"2024-03-06T07:32:33.020665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:55.562732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022487+0000","last_change":"2024-03-06T07:32:31.365105+0000","last_active":"2024-03-06T07:32:33.022487+0000","last_peered":"2024-03-06T07:32:33.022487+0000","last_clean":"2024-03-06T07:32:33.022487+0000","last_became_active":"2024-03-06T07:32:31.364824+0000","last_became_peered":"2024-03-06T07:32:31.364824+0000","last_unstale":"2024-03-06T07:32:33.022487+0000","last_undegraded":"2024-03-06T07:32:33.022487+0000","last_fullsized":"2024-03-06T07:32:33.022487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:29.118305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428919+0000","last_change":"2024-03-06T07:32:30.350776+0000","last_active":"2024-03-06T07:32:32.428919+0000","last_peered":"2024-03-06T07:32:32.428919+0000","last_clean":"2024-03-06T07:32:32.428919+0000","last_became_active":"2024-03-06T07:32:30.350187+0000","last_became_peered":"2024-03-06T07:32:30.350187+0000","last_unstale":"2024-03-06T07:32:32.428919+0000","last_undegraded":"2024-03-06T07:32:32.428919+0000","last_fullsized":"2024-03-06T07:32:32.428919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:19.990747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021229+0000","last_change":"2024-03-06T07:32:31.387704+0000","last_active":"2024-03-06T07:32:33.021229+0000","last_peered":"2024-03-06T07:32:33.021229+0000","last_clean":"2024-03-06T07:32:33.021229+0000","last_became_active":"2024-03-06T07:32:31.387338+0000","last_became_peered":"2024-03-06T07:32:31.387338+0000","last_unstale":"2024-03-06T07:32:33.021229+0000","last_undegraded":"2024-03-06T07:32:33.021229+0000","last_fullsized":"2024-03-06T07:32:33.021229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:24.837198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.021661+0000","last_change":"2024-03-06T07:32:30.349574+0000","last_active":"2024-03-06T07:32:33.021661+0000","last_peered":"2024-03-06T07:32:33.021661+0000","last_clean":"2024-03-06T07:32:33.021661+0000","last_became_active":"2024-03-06T07:32:30.348811+0000","last_became_peered":"2024-03-06T07:32:30.348811+0000","last_unstale":"2024-03-06T07:32:33.021661+0000","last_undegraded":"2024-03-06T07:32:33.021661+0000","last_fullsized":"2024-03-06T07:32:33.021661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:55.010429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.343937+0000","last_change":"2024-03-06T07:32:31.397499+0000","last_active":"2024-03-06T07:32:32.343937+0000","last_peered":"2024-03-06T07:32:32.343937+0000","last_clean":"2024-03-06T07:32:32.343937+0000","last_became_active":"2024-03-06T07:32:31.397170+0000","last_became_peered":"2024-03-06T07:32:31.397170+0000","last_unstale":"2024-03-06T07:32:32.343937+0000","last_undegraded":"2024-03-06T07:32:32.343937+0000","last_fullsized":"2024-03-06T07:32:32.343937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:07.228675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.380092+0000","last_change":"2024-03-06T07:32:30.333850+0000","last_active":"2024-03-06T07:32:32.380092+0000","last_peered":"2024-03-06T07:32:32.380092+0000","last_clean":"2024-03-06T07:32:32.380092+0000","last_became_active":"2024-03-06T07:32:30.333532+0000","last_became_peered":"2024-03-06T07:32:30.333532+0000","last_unstale":"2024-03-06T07:32:32.380092+0000","last_undegraded":"2024-03-06T07:32:32.380092+0000","last_fullsized":"2024-03-06T07:32:32.380092+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:16.696675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.344476+0000","last_change":"2024-03-06T07:32:31.389800+0000","last_active":"2024-03-06T07:32:32.344476+0000","last_peered":"2024-03-06T07:32:32.344476+0000","last_clean":"2024-03-06T07:32:32.344476+0000","last_became_active":"2024-03-06T07:32:31.388275+0000","last_became_peered":"2024-03-06T07:32:31.388275+0000","last_unstale":"2024-03-06T07:32:32.344476+0000","last_undegraded":"2024-03-06T07:32:32.344476+0000","last_fullsized":"2024-03-06T07:32:32.344476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:16.638971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022814+0000","last_change":"2024-03-06T07:32:30.345302+0000","last_active":"2024-03-06T07:32:33.022814+0000","last_peered":"2024-03-06T07:32:33.022814+0000","last_clean":"2024-03-06T07:32:33.022814+0000","last_became_active":"2024-03-06T07:32:30.344975+0000","last_became_peered":"2024-03-06T07:32:30.344975+0000","last_unstale":"2024-03-06T07:32:33.022814+0000","last_undegraded":"2024-03-06T07:32:33.022814+0000","last_fullsized":"2024-03-06T07:32:33.022814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:51.919431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379780+0000","last_change":"2024-03-06T07:32:31.368330+0000","last_active":"2024-03-06T07:32:32.379780+0000","last_peered":"2024-03-06T07:32:32.379780+0000","last_clean":"2024-03-06T07:32:32.379780+0000","last_became_active":"2024-03-06T07:32:31.368037+0000","last_became_peered":"2024-03-06T07:32:31.368037+0000","last_unstale":"2024-03-06T07:32:32.379780+0000","last_undegraded":"2024-03-06T07:32:32.379780+0000","last_fullsized":"2024-03-06T07:32:32.379780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:12.948560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.379866+0000","last_change":"2024-03-06T07:32:30.345874+0000","last_active":"2024-03-06T07:32:32.379866+0000","last_peered":"2024-03-06T07:32:32.379866+0000","last_clean":"2024-03-06T07:32:32.379866+0000","last_became_active":"2024-03-06T07:32:30.345536+0000","last_became_peered":"2024-03-06T07:32:30.345536+0000","last_unstale":"2024-03-06T07:32:32.379866+0000","last_undegraded":"2024-03-06T07:32:32.379866+0000","last_fullsized":"2024-03-06T07:32:32.379866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:02.058897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428790+0000","last_change":"2024-03-06T07:32:31.389189+0000","last_active":"2024-03-06T07:32:32.428790+0000","last_peered":"2024-03-06T07:32:32.428790+0000","last_clean":"2024-03-06T07:32:32.428790+0000","last_became_active":"2024-03-06T07:32:31.388860+0000","last_became_peered":"2024-03-06T07:32:31.388860+0000","last_unstale":"2024-03-06T07:32:32.428790+0000","last_undegraded":"2024-03-06T07:32:32.428790+0000","last_fullsized":"2024-03-06T07:32:32.428790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:04.514863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.020727+0000","last_change":"2024-03-06T07:32:30.343999+0000","last_active":"2024-03-06T07:32:33.020727+0000","last_peered":"2024-03-06T07:32:33.020727+0000","last_clean":"2024-03-06T07:32:33.020727+0000","last_became_active":"2024-03-06T07:32:30.343545+0000","last_became_peered":"2024-03-06T07:32:30.343545+0000","last_unstale":"2024-03-06T07:32:33.020727+0000","last_undegraded":"2024-03-06T07:32:33.020727+0000","last_fullsized":"2024-03-06T07:32:33.020727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:05.108408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:33.022571+0000","last_change":"2024-03-06T07:32:31.353535+0000","last_active":"2024-03-06T07:32:33.022571+0000","last_peered":"2024-03-06T07:32:33.022571+0000","last_clean":"2024-03-06T07:32:33.022571+0000","last_became_active":"2024-03-06T07:32:31.353104+0000","last_became_peered":"2024-03-06T07:32:31.353104+0000","last_unstale":"2024-03-06T07:32:33.022571+0000","last_undegraded":"2024-03-06T07:32:33.022571+0000","last_fullsized":"2024-03-06T07:32:33.022571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:48.018944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.345759+0000","last_change":"2024-03-06T07:32:30.343072+0000","last_active":"2024-03-06T07:32:32.345759+0000","last_peered":"2024-03-06T07:32:32.345759+0000","last_clean":"2024-03-06T07:32:32.345759+0000","last_became_active":"2024-03-06T07:32:30.342779+0000","last_became_peered":"2024-03-06T07:32:30.342779+0000","last_unstale":"2024-03-06T07:32:32.345759+0000","last_undegraded":"2024-03-06T07:32:32.345759+0000","last_fullsized":"2024-03-06T07:32:32.345759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:19.775303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427737+0000","last_change":"2024-03-06T07:32:31.379277+0000","last_active":"2024-03-06T07:32:32.427737+0000","last_peered":"2024-03-06T07:32:32.427737+0000","last_clean":"2024-03-06T07:32:32.427737+0000","last_became_active":"2024-03-06T07:32:31.378879+0000","last_became_peered":"2024-03-06T07:32:31.378879+0000","last_unstale":"2024-03-06T07:32:32.427737+0000","last_undegraded":"2024-03-06T07:32:32.427737+0000","last_fullsized":"2024-03-06T07:32:32.427737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:08.722443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.427867+0000","last_change":"2024-03-06T07:32:30.343667+0000","last_active":"2024-03-06T07:32:32.427867+0000","last_peered":"2024-03-06T07:32:32.427867+0000","last_clean":"2024-03-06T07:32:32.427867+0000","last_became_active":"2024-03-06T07:32:30.343135+0000","last_became_peered":"2024-03-06T07:32:30.343135+0000","last_unstale":"2024-03-06T07:32:32.427867+0000","last_undegraded":"2024-03-06T07:32:32.427867+0000","last_fullsized":"2024-03-06T07:32:32.427867+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:11.934557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.868693+0000","last_change":"2024-03-06T07:32:31.394532+0000","last_active":"2024-03-06T07:32:32.868693+0000","last_peered":"2024-03-06T07:32:32.868693+0000","last_clean":"2024-03-06T07:32:32.868693+0000","last_became_active":"2024-03-06T07:32:31.394216+0000","last_became_peered":"2024-03-06T07:32:31.394216+0000","last_unstale":"2024-03-06T07:32:32.868693+0000","last_undegraded":"2024-03-06T07:32:32.868693+0000","last_fullsized":"2024-03-06T07:32:32.868693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:30.302539+0000","last_clean_scrub_stamp":"2024-03-06T07:32:30.302539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:10.877538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.428356+0000","last_change":"2024-03-06T07:32:30.351683+0000","last_active":"2024-03-06T07:32:32.428356+0000","last_peered":"2024-03-06T07:32:32.428356+0000","last_clean":"2024-03-06T07:32:32.428356+0000","last_became_active":"2024-03-06T07:32:30.350553+0000","last_became_peered":"2024-03-06T07:32:30.350553+0000","last_unstale":"2024-03-06T07:32:32.428356+0000","last_undegraded":"2024-03-06T07:32:32.428356+0000","last_fullsized":"2024-03-06T07:32:32.428356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:29.288536+0000","last_clean_scrub_stamp":"2024-03-06T07:32:29.288536+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:28.215419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":257,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:32:32.342345+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:32:32.342345+0000","last_peered":"2024-03-06T07:32:32.342345+0000","last_clean":"2024-03-06T07:32:32.342345+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:32:32.342345+0000","last_undegraded":"2024-03-06T07:32:32.342345+0000","last_fullsized":"2024-03-06T07:32:32.342345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":937154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":835584,"data_stored":346262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":819200,"data_stored":339960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":831488,"data_stored":346240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":339982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":347782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36156,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335684,"statfs":{"total":96636764160,"available":96599740416,"internally_reserved":0,"allocated":835584,"data_stored":342098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1421312,"data_stored":936538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:36.384 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:32:36.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.640+0000 7fc2cac00640 1 Processor -- start 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.640+0000 7fc2cac00640 1 -- start start 2024-03-06T07:32:36.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.642+0000 7fc2cac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2c415dde0 0x7fc2c40b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.642+0000 7fc2cac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 0x7fc2c406e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.642+0000 7fc2cac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40b3af0 con 0x7fc2c40b5110 2024-03-06T07:32:36.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.642+0000 7fc2cac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c40b3d90 con 0x7fc2c415dde0 2024-03-06T07:32:36.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.642+0000 7fc2cac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b4070 con 0x7fc2c415e1c0 2024-03-06T07:32:36.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.643+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 0x7fc2c406e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.643+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 0x7fc2c406e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:54110/0 (socket says 172.21.15.32:54110) 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.643+0000 7fc2c3400640 1 -- 172.21.15.32:0/1338098554 learned_addr learned my addr 172.21.15.32:0/1338098554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.643+0000 7fc2c3e00640 1 --2- 172.21.15.32:0/1338098554 >> v2:172.21.15.39:3300/0 conn(0x7fc2c415dde0 0x7fc2c40b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.644+0000 7fc2c3400640 1 -- 172.21.15.32:0/1338098554 >> v2:172.21.15.39:3300/0 conn(0x7fc2c415dde0 msgr2=0x7fc2c40b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.644+0000 7fc2c3400640 1 --2- 172.21.15.32:0/1338098554 >> v2:172.21.15.39:3300/0 conn(0x7fc2c415dde0 0x7fc2c40b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.644+0000 7fc2c3400640 1 -- 172.21.15.32:0/1338098554 >> v1:172.21.15.39:6789/0 conn(0x7fc2c40b5110 legacy=0x7fc2c406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.644+0000 7fc2c3400640 1 -- 172.21.15.32:0/1338098554 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c40b4350 con 0x7fc2c415e1c0 2024-03-06T07:32:36.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.646+0000 7fc2c3400640 1 --2- 172.21.15.32:0/1338098554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 0x7fc2c406e2d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009870 tx=0x7fc2b002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52fd1a5181a2abc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.646+0000 7fc2c2a00640 1 -- 172.21.15.32:0/1338098554 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0034070 con 0x7fc2c415e1c0 2024-03-06T07:32:36.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.646+0000 7fc2c2a00640 1 -- 172.21.15.32:0/1338098554 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b0030840 con 0x7fc2c415e1c0 2024-03-06T07:32:36.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.646+0000 7fc2c2a00640 1 -- 172.21.15.32:0/1338098554 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0030c10 con 0x7fc2c415e1c0 2024-03-06T07:32:36.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 -- 172.21.15.32:0/1338098554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 msgr2=0x7fc2c406e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 --2- 172.21.15.32:0/1338098554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 0x7fc2c406e2d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009870 tx=0x7fc2b002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 -- 172.21.15.32:0/1338098554 shutdown_connections 2024-03-06T07:32:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 --2- 172.21.15.32:0/1338098554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415e1c0 0x7fc2c406e2d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 --2- 172.21.15.32:0/1338098554 >> v2:172.21.15.39:3300/0 conn(0x7fc2c415dde0 0x7fc2c40b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 -- 172.21.15.32:0/1338098554 >> 172.21.15.32:0/1338098554 conn(0x7fc2c4071520 msgr2=0x7fc2c4071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 -- 172.21.15.32:0/1338098554 shutdown_connections 2024-03-06T07:32:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.647+0000 7fc2cac00640 1 -- 172.21.15.32:0/1338098554 wait complete. 2024-03-06T07:32:36.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.649+0000 7fc2cac00640 1 Processor -- start 2024-03-06T07:32:36.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.650+0000 7fc2cac00640 1 -- start start 2024-03-06T07:32:36.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2cac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c40b5110 0x7fc2c4160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2cac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415dde0 0x7fc2c4158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2cac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 0x7fc2c415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2cac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c406e990 con 0x7fc2c415e1c0 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2cac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c406e810 con 0x7fc2c40b5110 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2cac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c406c6d0 con 0x7fc2c415dde0 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2c8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 0x7fc2c415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2c8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 0x7fc2c415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35686/0 (socket says 172.21.15.32:35686) 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2c8c00640 1 -- 172.21.15.32:0/3872331597 learned_addr learned my addr 172.21.15.32:0/3872331597 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2c3400640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415dde0 0x7fc2c4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.651+0000 7fc2c3e00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c40b5110 0x7fc2c4160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2c8c00640 1 -- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415dde0 msgr2=0x7fc2c4158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2c8c00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415dde0 0x7fc2c4158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2c8c00640 1 -- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c40b5110 msgr2=0x7fc2c4160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2c8c00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c40b5110 0x7fc2c4160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2c8c00640 1 -- 172.21.15.32:0/3872331597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4159320 con 0x7fc2c415e1c0 2024-03-06T07:32:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2c8c00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 0x7fc2c415ef40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00c7a0 tx=0x7fc2bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc003b70 con 0x7fc2c415e1c0 2024-03-06T07:32:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc003d30 con 0x7fc2c415e1c0 2024-03-06T07:32:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00aa90 con 0x7fc2c415e1c0 2024-03-06T07:32:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.652+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c4159600 con 0x7fc2c415e1c0 2024-03-06T07:32:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.653+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c415ea20 con 0x7fc2c415e1c0 2024-03-06T07:32:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.654+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c4068010 con 0x7fc2c415e1c0 2024-03-06T07:32:36.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.655+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2bc00ac30 con 0x7fc2c415e1c0 2024-03-06T07:32:36.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.656+0000 7fc2bbe00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc2a406b500 0x7fc2a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.656+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2bc08a0a0 con 0x7fc2c415e1c0 2024-03-06T07:32:36.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.656+0000 7fc2c3e00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc2a406b500 0x7fc2a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.657+0000 7fc2c3e00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc2a406b500 0x7fc2a406d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009ad0 tx=0x7fc2b4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.664+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2c4068010 con 0x7fc2c415e1c0 2024-03-06T07:32:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.943+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc2c4068720 con 0x7fc2c415e1c0 2024-03-06T07:32:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.947+0000 7fc2bbe00640 1 -- 172.21.15.32:0/3872331597 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc2bc058fb0 con 0x7fc2c415e1c0 2024-03-06T07:32:36.949 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:32:36.949 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:32:36.950 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:32:36.950 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:32:36.950 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.951+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc2a406b500 msgr2=0x7fc2a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.951+0000 7fc2cac00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc2a406b500 0x7fc2a406d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009ad0 tx=0x7fc2b4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.951+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 msgr2=0x7fc2c415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.951+0000 7fc2cac00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 0x7fc2c415ef40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00c7a0 tx=0x7fc2bc00cc60 comp rx=0 tx=0).stop 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 shutdown_connections 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc2a406b500 0x7fc2a406d9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2c415e1c0 0x7fc2c415ef40 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2c415dde0 0x7fc2c4158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 --2- 172.21.15.32:0/3872331597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c40b5110 0x7fc2c4160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 >> 172.21.15.32:0/3872331597 conn(0x7fc2c4071520 msgr2=0x7fc2c40ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 shutdown_connections 2024-03-06T07:32:36.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:36.952+0000 7fc2cac00640 1 -- 172.21.15.32:0/3872331597 wait complete. 2024-03-06T07:32:36.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:37.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.187+0000 7fe292a00640 1 Processor -- start 2024-03-06T07:32:37.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.187+0000 7fe292a00640 1 -- start start 2024-03-06T07:32:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe292a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe28c06e760 0x7fe28c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe292a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c073360 con 0x7fe28c15b290 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe292a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c06d6c0 con 0x7fe28c0717a0 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe292a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c06e2d0 con 0x7fe28c06e760 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe28be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe28c15b290 0x7fe28c071090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37022/0 (socket says 172.21.15.32:37022) 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe28be00640 1 -- 172.21.15.32:0/3998565852 learned_addr learned my addr 172.21.15.32:0/3998565852 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe290a00640 1 --2- 172.21.15.32:0/3998565852 >> v2:172.21.15.39:3300/0 conn(0x7fe28c06e760 0x7fe28c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154395834 0 0) 0x7fe27c001110 con 0x7fe28c15b290 2024-03-06T07:32:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.191+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270003520 con 0x7fe28c15b290 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4077538524 0 0) 0x7fe270003520 con 0x7fe28c15b290 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 >> v2:172.21.15.39:3300/0 conn(0x7fe28c06e760 msgr2=0x7fe28c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 --2- 172.21.15.32:0/3998565852 >> v2:172.21.15.39:3300/0 conn(0x7fe28c06e760 0x7fe28c15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 >> v1:172.21.15.39:6789/0 conn(0x7fe28c0717a0 legacy=0x7fe28c161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c06d9a0 con 0x7fe28c15b290 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fe27c001840 con 0x7fe28c15b290 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 >> v1:172.21.15.32:6789/0 conn(0x7fe28c15b290 legacy=0x7fe28c071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe270003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe270004800 0x7fe270004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2700052e0 0x7fe270005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c06e2d0 con 0x7fe270003a70 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe270006270 con 0x7fe2700052e0 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe270006470 con 0x7fe270004800 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.192+0000 7fe28be00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe270003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe283e00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe270004800 0x7fe270004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe290a00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2700052e0 0x7fe270005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe28be00640 1 -- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe270004800 msgr2=0x7fe270004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe28be00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe270004800 0x7fe270004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe28be00640 1 -- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2700052e0 msgr2=0x7fe270005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe28be00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2700052e0 0x7fe270005990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe290a00640 1 -- 172.21.15.32:0/3998565852 reap_dead start 2024-03-06T07:32:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.193+0000 7fe28be00640 1 -- 172.21.15.32:0/3998565852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270003520 con 0x7fe270003a70 2024-03-06T07:32:37.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.195+0000 7fe28be00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe270003e90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00c320 tx=0x7fe270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2665b81926d0fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.195+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c007e10 con 0x7fe270003a70 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.195+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c030070 con 0x7fe270003a70 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.195+0000 7fe28b400640 1 -- 172.21.15.32:0/3998565852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c0058d0 con 0x7fe270003a70 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 -- 172.21.15.32:0/3998565852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 msgr2=0x7fe270003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe270003e90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00c320 tx=0x7fe270005830 comp rx=0 tx=0).stop 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 -- 172.21.15.32:0/3998565852 shutdown_connections 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 --2- 172.21.15.32:0/3998565852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe270003e90 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 -- 172.21.15.32:0/3998565852 >> 172.21.15.32:0/3998565852 conn(0x7fe28c0b1bb0 msgr2=0x7fe28c0b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 -- 172.21.15.32:0/3998565852 shutdown_connections 2024-03-06T07:32:37.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.196+0000 7fe292a00640 1 -- 172.21.15.32:0/3998565852 wait complete. 2024-03-06T07:32:37.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.198+0000 7fe292a00640 1 Processor -- start 2024-03-06T07:32:37.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.198+0000 7fe292a00640 1 -- start start 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe292a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe28c157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe292a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe28c1f7530 0x7fe28c0ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe292a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe28c0af8a0 0x7fe28c0afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe292a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c069060 con 0x7fe270003a70 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe292a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c06df60 con 0x7fe28c1f7530 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe292a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0691e0 con 0x7fe28c0af8a0 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe28be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe28c157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe28be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe28c157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35710/0 (socket says 172.21.15.32:35710) 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe28be00640 1 -- 172.21.15.32:0/3927387665 learned_addr learned my addr 172.21.15.32:0/3927387665 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.199+0000 7fe290a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe28c0af8a0 0x7fe28c0afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe283e00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe28c1f7530 0x7fe28c0ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe28be00640 1 -- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe28c0af8a0 msgr2=0x7fe28c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe28be00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe28c0af8a0 0x7fe28c0afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe28be00640 1 -- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe28c1f7530 msgr2=0x7fe28c0ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe28be00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe28c1f7530 0x7fe28c0ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe28be00640 1 -- 172.21.15.32:0/3927387665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c154320 con 0x7fe270003a70 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe283e00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe28c1f7530 0x7fe28c0ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe28be00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe28c157400 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00c440 tx=0x7fe27c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c046070 con 0x7fe270003a70 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c0049e0 con 0x7fe270003a70 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c004db0 con 0x7fe270003a70 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.200+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c1f79c0 con 0x7fe270003a70 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.201+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c1f7f80 con 0x7fe270003a70 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.202+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c04f3a0 con 0x7fe270003a70 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.203+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe27c007070 con 0x7fe270003a70 2024-03-06T07:32:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.204+0000 7fe288c00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe26806b500 0x7fe26806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.204+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe27c0ab560 con 0x7fe270003a70 2024-03-06T07:32:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.204+0000 7fe283e00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe26806b500 0x7fe26806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.205+0000 7fe283e00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe26806b500 0x7fe26806d9c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0af810 tx=0x7fe274009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.209+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe28c073360 con 0x7fe270003a70 2024-03-06T07:32:37.290 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-06T07:32:37.287+0000 7f83f3a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:37.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.448+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe28c0b0260 con 0x7fe270003a70 2024-03-06T07:32:37.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.449+0000 7fe288c00640 1 -- 172.21.15.32:0/3927387665 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe27c0800d0 con 0x7fe270003a70 2024-03-06T07:32:37.452 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:37.452 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:32:37.452 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:32:37.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe26806b500 msgr2=0x7fe26806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe26806b500 0x7fe26806d9c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0af810 tx=0x7fe274009040 comp rx=0 tx=0).stop 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 msgr2=0x7fe28c157400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe28c157400 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00c440 tx=0x7fe27c02e070 comp rx=0 tx=0).stop 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 shutdown_connections 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe26806b500 0x7fe26806d9c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe28c0af8a0 0x7fe28c0afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe28c1f7530 0x7fe28c0ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 --2- 172.21.15.32:0/3927387665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe270003a70 0x7fe28c157400 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 >> 172.21.15.32:0/3927387665 conn(0x7fe28c0b1bb0 msgr2=0x7fe28c0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 shutdown_connections 2024-03-06T07:32:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:37.452+0000 7fe292a00640 1 -- 172.21.15.32:0/3927387665 wait complete. 2024-03-06T07:32:37.874 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:30.799560+0000', 'modified': '2024-03-06T07:32:33.904744+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6360}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6360': {'gid': 6360, 'name': 'c', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6837/2323044149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 2323044149}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 2323044149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6360, 'qdb_cluster': [6360]} 2024-03-06T07:32:37.874 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:37.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.103+0000 7fa9b4400640 1 Processor -- start 2024-03-06T07:32:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.104+0000 7fa9b4400640 1 -- start start 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.105+0000 7fa9b4400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af580 0x7fa9ac15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.105+0000 7fa9b4400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa9ac0cdc20 0x7fa9ac0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.105+0000 7fa9b4400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac15cc40 con 0x7fa9ac0af1a0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.105+0000 7fa9b4400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac15cf20 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.105+0000 7fa9b4400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac15d200 con 0x7fa9ac0af580 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa9ac0cdc20 0x7fa9ac0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa9ac0cdc20 0x7fa9ac0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51022/0 (socket says 172.21.15.32:51022) 2024-03-06T07:32:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 -- 172.21.15.32:0/843385310 learned_addr learned my addr 172.21.15.32:0/843385310 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b1000640 1 --2- 172.21.15.32:0/843385310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af580 0x7fa9ac15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 -- 172.21.15.32:0/843385310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af580 msgr2=0x7fa9ac15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 --2- 172.21.15.32:0/843385310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af580 0x7fa9ac15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 -- 172.21.15.32:0/843385310 >> v1:172.21.15.39:6789/0 conn(0x7fa9ac0af1a0 legacy=0x7fa9ac0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.106+0000 7fa9b2400640 1 -- 172.21.15.32:0/843385310 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac15d4e0 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.108+0000 7fa9b2400640 1 --2- 172.21.15.32:0/843385310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac0b2dd0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009870 tx=0x7fa9a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f441e7ae54a2247 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.108+0000 7fa9abe00640 1 -- 172.21.15.32:0/843385310 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4035020 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.108+0000 7fa9abe00640 1 -- 172.21.15.32:0/843385310 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a4031640 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.108+0000 7fa9abe00640 1 -- 172.21.15.32:0/843385310 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a402d040 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.108+0000 7fa9b4400640 1 -- 172.21.15.32:0/843385310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 msgr2=0x7fa9ac0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.108+0000 7fa9b4400640 1 --2- 172.21.15.32:0/843385310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac0b2dd0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009870 tx=0x7fa9a402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.109+0000 7fa9b4400640 1 -- 172.21.15.32:0/843385310 shutdown_connections 2024-03-06T07:32:38.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.109+0000 7fa9b4400640 1 --2- 172.21.15.32:0/843385310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac0b2dd0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.109+0000 7fa9b4400640 1 --2- 172.21.15.32:0/843385310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af580 0x7fa9ac15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.109+0000 7fa9b4400640 1 -- 172.21.15.32:0/843385310 >> 172.21.15.32:0/843385310 conn(0x7fa9ac06afb0 msgr2=0x7fa9ac06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.109+0000 7fa9b4400640 1 -- 172.21.15.32:0/843385310 shutdown_connections 2024-03-06T07:32:38.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.109+0000 7fa9b4400640 1 -- 172.21.15.32:0/843385310 wait complete. 2024-03-06T07:32:38.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.111+0000 7fa9b4400640 1 Processor -- start 2024-03-06T07:32:38.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.112+0000 7fa9b4400640 1 -- start start 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.113+0000 7fa9b4400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af1a0 0x7fa9ac09d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.113+0000 7fa9b4400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9ac0af580 0x7fa9ac14bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.113+0000 7fa9b4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.113+0000 7fa9b4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0b36a0 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.113+0000 7fa9b4400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0b3520 con 0x7fa9ac0af580 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.113+0000 7fa9b4400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0b3820 con 0x7fa9ac0af1a0 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac1500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51026/0 (socket says 172.21.15.32:51026) 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 -- 172.21.15.32:0/732409514 learned_addr learned my addr 172.21.15.32:0/732409514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af1a0 0x7fa9ac09d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b1000640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9ac0af580 0x7fa9ac14bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 -- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af1a0 msgr2=0x7fa9ac09d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af1a0 0x7fa9ac09d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 -- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9ac0af580 msgr2=0x7fa9ac14bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9ac0af580 0x7fa9ac14bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.114+0000 7fa9b2400640 1 -- 172.21.15.32:0/732409514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac15dc00 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.115+0000 7fa9b2400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac1500a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa9a402b930 tx=0x7fa9a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.115+0000 7fa9a9600640 1 -- 172.21.15.32:0/732409514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4040070 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.115+0000 7fa9a9600640 1 -- 172.21.15.32:0/732409514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a40399f0 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.115+0000 7fa9a9600640 1 -- 172.21.15.32:0/732409514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4035040 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.115+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac15dee0 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.115+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac15e4a0 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.117+0000 7fa9a9600640 1 -- 172.21.15.32:0/732409514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9a4038020 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.118+0000 7fa9a9600640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.118+0000 7fa9a9600640 1 -- 172.21.15.32:0/732409514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9a40abb40 con 0x7fa9ac0cdc20 2024-03-06T07:32:38.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.118+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.119+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa98c06b4d0 0x7fa98c06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0b1790 tx=0x7fa99c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.120+0000 7fa9b718e740 1 Processor -- start 2024-03-06T07:32:38.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.120+0000 7fa9b718e740 1 -- start start 2024-03-06T07:32:38.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.121+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5570126da7b0 0x5570126ad750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.121+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126dd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.121+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126adc90 0x5570126dda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.121+0000 7fa9b718e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x557012617600 con 0x5570126dab90 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.121+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570125c0930 con 0x5570126da7b0 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.121+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570126136a0 con 0x5570126adc90 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126dd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126dd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51028/0 (socket says 172.21.15.32:51028) 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 -- 172.21.15.32:0/2831527832 learned_addr learned my addr 172.21.15.32:0/2831527832 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5570126da7b0 0x5570126ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b2400640 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126adc90 0x5570126dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 -- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126adc90 msgr2=0x5570126dda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126adc90 0x5570126dda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 -- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5570126da7b0 msgr2=0x5570126ad750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5570126da7b0 0x5570126ad750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.122+0000 7fa9b1000640 1 -- 172.21.15.32:0/2831527832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55701232c2d0 con 0x5570126dab90 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.123+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126dd540 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa99800b8d0 tx=0x7fa99800bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8365735ce5945e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.123+0000 7fa9a2000640 1 -- 172.21.15.32:0/2831527832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998018030 con 0x5570126dab90 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.123+0000 7fa9a2000640 1 -- 172.21.15.32:0/2831527832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998004020 con 0x5570126dab90 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.123+0000 7fa9a2000640 1 -- 172.21.15.32:0/2831527832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9980043d0 con 0x5570126dab90 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.123+0000 7fa9b718e740 1 -- 172.21.15.32:0/2831527832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 msgr2=0x5570126dd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.123+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126dd540 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa99800b8d0 tx=0x7fa99800bd90 comp rx=0 tx=0).stop 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 -- 172.21.15.32:0/2831527832 shutdown_connections 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126adc90 0x5570126dda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126dd540 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2831527832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5570126da7b0 0x5570126ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 -- 172.21.15.32:0/2831527832 >> 172.21.15.32:0/2831527832 conn(0x5570126af9c0 msgr2=0x5570126f8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 -- 172.21.15.32:0/2831527832 shutdown_connections 2024-03-06T07:32:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 -- 172.21.15.32:0/2831527832 wait complete. 2024-03-06T07:32:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.124+0000 7fa9b718e740 1 -- start start 2024-03-06T07:32:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.125+0000 7fa9b718e740 1 Processor -- start 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x55701270bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570126b1f60 con 0x5570126dab90 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570125d3d90 con 0x55701270c4a0 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570126b1350 con 0x5570126da7b0 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b718e740 10 client.0 _ll_register_callbacks cb 0x55701260f350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x55701270bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x55701270bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51042/0 (socket says 172.21.15.32:51042) 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 -- 172.21.15.32:0/2773160037 learned_addr learned my addr 172.21.15.32:0/2773160037 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b2400640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 -- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 msgr2=0x5570126cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 -- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 msgr2=0x5570126e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126e01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1000640 1 -- 172.21.15.32:0/2773160037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570123f66b0 con 0x5570126dab90 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.126+0000 7fa9b2400640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126e01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.127+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x55701270bf60 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa99800b870 tx=0x7fa998004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8a04389611d529 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.127+0000 7fa996a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.127+0000 7fa996a00640 1 -- 172.21.15.32:0/2773160037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99800fd80 con 0x5570126dab90 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.127+0000 7fa996a00640 1 -- 172.21.15.32:0/2773160037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa998028af0 con 0x5570126dab90 2024-03-06T07:32:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.127+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa998004020 con 0x5570126dab90 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa996a00640 1 -- 172.21.15.32:0/2773160037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa99802f050 con 0x5570126dab90 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 10 client.6495 fetch_fsmap learned FSMap version 43 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x5570127097c0 con 0x5570126dab90 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa996a00640 1 -- 172.21.15.32:0/2773160037 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998004020 con 0x5570126dab90 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa996a00640 1 -- 172.21.15.32:0/2773160037 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 43) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa998032020 con 0x5570126dab90 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 10 client.6495 fetch_fsmap finished waiting for FSMap version 43 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 10 client.6495 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 20 client.6495 populate_metadata read hostname 'smithi032' 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570127149a0 0x557012716d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 4 client.6495 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 --> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557012624b40 con 0x5570127149a0 2024-03-06T07:32:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.128+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570127149a0 0x557012716d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.129+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570127149a0 0x557012716d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=811b34443d9e37a1 server_cookie=5ff0840bcf1abf2 in_seq=0 out_seq=0 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.129+0000 7fa996a00640 10 client.6495 ms_handle_connect on v2:172.21.15.32:6836/2323044149 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa996a00640 1 -- 172.21.15.32:0/2773160037 <== mds.0 v2:172.21.15.32:6836/2323044149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557012624b40 con 0x5570127149a0 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa996a00640 10 client.6495 handle_command_reply: tid=0 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 client.6495 shutdown 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 msgr2=0x55701270bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x55701270bf60 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa99800b870 tx=0x7fa998004730 comp rx=0 tx=0).stop 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 shutdown_connections 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570127149a0 0x557012716d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126e01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa996a00640 0 client.6495 ms_handle_reset on v2:172.21.15.32:6836/2323044149 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x55701270bf60 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2773160037 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.130+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 >> 172.21.15.32:0/2773160037 conn(0x5570126adc90 msgr2=0x5570126fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.131+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 shutdown_connections 2024-03-06T07:32:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.131+0000 7fa9b718e740 1 -- 172.21.15.32:0/2773160037 wait complete. 2024-03-06T07:32:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.131+0000 7fa9b718e740 20 client.6495 trim_cache size 0 max 16384 2024-03-06T07:32:38.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.149+0000 7fa9b718e740 1 Processor -- start 2024-03-06T07:32:38.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.149+0000 7fa9b718e740 1 -- start start 2024-03-06T07:32:38.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b718e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac1ff580 con 0x5570126dab90 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0b0460 con 0x55701270c4a0 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0b0a60 con 0x5570126da7b0 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126f1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51044/0 (socket says 172.21.15.32:51044) 2024-03-06T07:32:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.150+0000 7fa9b1000640 1 -- 172.21.15.32:0/3698274660 learned_addr learned my addr 172.21.15.32:0/3698274660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.151+0000 7fa9b1000640 1 -- 172.21.15.32:0/3698274660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 msgr2=0x5570126f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.151+0000 7fa9b1000640 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.151+0000 7fa9b1000640 1 -- 172.21.15.32:0/3698274660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 msgr2=0x5570126f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.151+0000 7fa9b1000640 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.151+0000 7fa9b1000640 1 -- 172.21.15.32:0/3698274660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570126af6a0 con 0x5570126dab90 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.151+0000 7fa9b1000640 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126f1f20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa998006d30 tx=0x7fa998020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a195f468f16cd2a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9a1600640 1 -- 172.21.15.32:0/3698274660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998018070 con 0x5570126dab90 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9a1600640 1 -- 172.21.15.32:0/3698274660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99800fd80 con 0x5570126dab90 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9a1600640 1 -- 172.21.15.32:0/3698274660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99801e460 con 0x5570126dab90 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 -- 172.21.15.32:0/3698274660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 msgr2=0x5570126f1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126f1f20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa998006d30 tx=0x7fa998020040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 -- 172.21.15.32:0/3698274660 shutdown_connections 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270c4a0 0x5570126f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126dab90 0x5570126f1f20 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 --2- 172.21.15.32:0/3698274660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5570126da7b0 0x5570126f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 -- 172.21.15.32:0/3698274660 >> 172.21.15.32:0/3698274660 conn(0x5570126adc90 msgr2=0x5570126dff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 -- 172.21.15.32:0/3698274660 shutdown_connections 2024-03-06T07:32:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.152+0000 7fa9b718e740 1 -- 172.21.15.32:0/3698274660 wait complete. 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.153+0000 7fa9b718e740 1 -- start start 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.153+0000 7fa9b718e740 1 Processor -- start 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 0x55701270ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55701270c4a0 0x5570126b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270bc50 0x5570126f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570126c91d0 con 0x5570126da7b0 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570126df6c0 con 0x55701270bc50 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570126e0dc0 con 0x55701270c4a0 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b718e740 10 client.0 _ll_register_callbacks cb 0x7fa99800b6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.154+0000 7fa9b1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 0x55701270ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 0x55701270ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51052/0 (socket says 172.21.15.32:51052) 2024-03-06T07:32:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 -- 172.21.15.32:0/2030106082 learned_addr learned my addr 172.21.15.32:0/2030106082 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b2400640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270bc50 0x5570126f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 -- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55701270c4a0 msgr2=0x5570126b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55701270c4a0 0x5570126b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 -- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270bc50 msgr2=0x5570126f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270bc50 0x5570126f19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 -- 172.21.15.32:0/2030106082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557012712b90 con 0x5570126da7b0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b2400640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270bc50 0x5570126f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa9b1a00640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 0x55701270ef10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x55701270c8f0 tx=0x7fa99c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb7983d3df9ece0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa996a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.155+0000 7fa996a00640 1 -- 172.21.15.32:0/2030106082 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c01e040 con 0x5570126da7b0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa996a00640 1 -- 172.21.15.32:0/2030106082 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa99c010040 con 0x5570126da7b0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa99800fd80 con 0x5570126da7b0 2024-03-06T07:32:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa996a00640 1 -- 172.21.15.32:0/2030106082 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa99c003a30 con 0x5570126da7b0 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 10 client.6501 fetch_fsmap learned FSMap version 43 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55701270f5b0 con 0x5570126da7b0 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa996a00640 1 -- 172.21.15.32:0/2030106082 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99800fd80 con 0x5570126da7b0 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa996a00640 1 -- 172.21.15.32:0/2030106082 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 43) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa99c021070 con 0x5570126da7b0 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 10 client.6501 fetch_fsmap finished waiting for FSMap version 43 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 10 client.6501 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 20 client.6501 populate_metadata read hostname 'smithi032' 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570126ee0c0 0x5570126ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 4 client.6501 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.156+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 --> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55701261ee90 con 0x5570126ee0c0 2024-03-06T07:32:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.157+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570126ee0c0 0x5570126ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.157+0000 7fa9b1000640 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570126ee0c0 0x5570126ee4a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bfcebe103d0b066 server_cookie=b9b88bfa9efcb340 in_seq=0 out_seq=0 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.157+0000 7fa996a00640 10 client.6501 ms_handle_connect on v2:172.21.15.32:6836/2323044149 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.157+0000 7fa996a00640 1 -- 172.21.15.32:0/2030106082 <== mds.0 v2:172.21.15.32:6836/2323044149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55701261ee90 con 0x5570126ee0c0 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.157+0000 7fa996a00640 10 client.6501 handle_command_reply: tid=0 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.157+0000 7fa9b718e740 1 client.6501 shutdown 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 msgr2=0x55701270ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 0x55701270ef10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x55701270c8f0 tx=0x7fa99c000f40 comp rx=0 tx=0).stop 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 shutdown_connections 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:6836/2323044149,v1:172.21.15.32:6837/2323044149] conn(0x5570126ee0c0 0x5570126ee4a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55701270bc50 0x5570126f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55701270c4a0 0x5570126b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 --2- 172.21.15.32:0/2030106082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5570126da7b0 0x55701270ef10 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa996a00640 0 client.6501 ms_handle_reset on v2:172.21.15.32:6836/2323044149 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 >> 172.21.15.32:0/2030106082 conn(0x5570126adc90 msgr2=0x5570126ac7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 shutdown_connections 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 1 -- 172.21.15.32:0/2030106082 wait complete. 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.158+0000 7fa9b718e740 20 client.6501 trim_cache size 0 max 16384 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6360, 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:32:38.164 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8231178870000004, 2024-03-06T07:32:38.165 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 43, 2024-03-06T07:32:38.165 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 68, 2024-03-06T07:32:38.165 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 68, 2024-03-06T07:32:38.165 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.187304814999999 2024-03-06T07:32:38.165 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:32:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa98c06b4d0 msgr2=0x7fa98c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa98c06b4d0 0x7fa98c06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0b1790 tx=0x7fa99c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 msgr2=0x7fa9ac1500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac1500a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa9a402b930 tx=0x7fa9a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 shutdown_connections 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9ac0cdc20 0x7fa9ac1500a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9ac0af580 0x7fa9ac14bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 --2- 172.21.15.32:0/732409514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa9ac0af1a0 0x7fa9ac09d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 >> 172.21.15.32:0/732409514 conn(0x7fa9ac06afb0 msgr2=0x7fa9ac0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 shutdown_connections 2024-03-06T07:32:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.159+0000 7fa9b4400640 1 -- 172.21.15.32:0/732409514 wait complete. 2024-03-06T07:32:38.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:38.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.379+0000 7f619fa00640 1 Processor -- start 2024-03-06T07:32:38.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.379+0000 7f619fa00640 1 -- start start 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.380+0000 7f619fa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6198070250 0x7f619815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.380+0000 7f619fa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f61981457c0 0x7f619815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f619fa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61980706e0 con 0x7f619806afb0 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f619fa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619806de60 con 0x7f6198070250 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f619fa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619806e140 con 0x7f61981457c0 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f619d000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f619806afb0 0x7f619806b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33640/0 (socket says 172.21.15.32:33640) 2024-03-06T07:32:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f619d000640 1 -- 172.21.15.32:0/59731288 learned_addr learned my addr 172.21.15.32:0/59731288 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f618fe00640 1 --2- 172.21.15.32:0/59731288 >> v2:172.21.15.32:3300/0 conn(0x7f6198070250 0x7f619815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f619da00640 1 --2- 172.21.15.32:0/59731288 >> v2:172.21.15.39:3300/0 conn(0x7f61981457c0 0x7f619815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.381+0000 7f6197e00640 1 -- 172.21.15.32:0/59731288 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878300658 0 0) 0x7f61980706e0 con 0x7f619806afb0 2024-03-06T07:32:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.382+0000 7f6197e00640 1 -- 172.21.15.32:0/59731288 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c003520 con 0x7f619806afb0 2024-03-06T07:32:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.382+0000 7f618fe00640 1 -- 172.21.15.32:0/59731288 >> v2:172.21.15.39:3300/0 conn(0x7f61981457c0 msgr2=0x7f619815fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.382+0000 7f618fe00640 1 --2- 172.21.15.32:0/59731288 >> v2:172.21.15.39:3300/0 conn(0x7f61981457c0 0x7f619815fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.382+0000 7f618fe00640 1 -- 172.21.15.32:0/59731288 >> v1:172.21.15.32:6789/0 conn(0x7f619806afb0 legacy=0x7f619806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.382+0000 7f618fe00640 1 -- 172.21.15.32:0/59731288 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61980afe60 con 0x7f6198070250 2024-03-06T07:32:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.383+0000 7f618fe00640 1 --2- 172.21.15.32:0/59731288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6198070250 0x7f619815f9a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f61980b3d20 tx=0x7f618002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e19df4270cd2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.383+0000 7f6197e00640 1 -- 172.21.15.32:0/59731288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800315a0 con 0x7f6198070250 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.383+0000 7f6197e00640 1 -- 172.21.15.32:0/59731288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180037070 con 0x7f6198070250 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.383+0000 7f6197e00640 1 -- 172.21.15.32:0/59731288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180031940 con 0x7f6198070250 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 -- 172.21.15.32:0/59731288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6198070250 msgr2=0x7f619815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 --2- 172.21.15.32:0/59731288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6198070250 0x7f619815f9a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f61980b3d20 tx=0x7f618002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 -- 172.21.15.32:0/59731288 shutdown_connections 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 --2- 172.21.15.32:0/59731288 >> v2:172.21.15.39:3300/0 conn(0x7f61981457c0 0x7f619815fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 --2- 172.21.15.32:0/59731288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6198070250 0x7f619815f9a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 -- 172.21.15.32:0/59731288 >> 172.21.15.32:0/59731288 conn(0x7f61980b30c0 msgr2=0x7f619806c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 -- 172.21.15.32:0/59731288 shutdown_connections 2024-03-06T07:32:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.384+0000 7f619fa00640 1 -- 172.21.15.32:0/59731288 wait complete. 2024-03-06T07:32:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.387+0000 7f619fa00640 1 Processor -- start 2024-03-06T07:32:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.387+0000 7f619fa00640 1 -- start start 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619fa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f619806afb0 0x7f61981f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 0x7f619815df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619fa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f619815e770 0x7f619815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619fa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980ae220 con 0x7f61981457c0 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619fa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198144ef0 con 0x7f619806afb0 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619fa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980ae3a0 con 0x7f619815e770 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f618fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 0x7f619815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f618fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 0x7f619815df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51078/0 (socket says 172.21.15.32:51078) 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f618fe00640 1 -- 172.21.15.32:0/3502533951 learned_addr learned my addr 172.21.15.32:0/3502533951 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.389+0000 7f619d000640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f619806afb0 0x7f61981f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f619da00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f619815e770 0x7f619815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f618fe00640 1 -- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f619815e770 msgr2=0x7f619815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f618fe00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f619815e770 0x7f619815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f618fe00640 1 -- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f619806afb0 msgr2=0x7f61981f7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f618fe00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f619806afb0 0x7f61981f7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f618fe00640 1 -- 172.21.15.32:0/3502533951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619815ef10 con 0x7f61981457c0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f618fe00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 0x7f619815df50 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6180009890 tx=0x7f61800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180032630 con 0x7f61981457c0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61800327d0 con 0x7f61981457c0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180032b80 con 0x7f61981457c0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.390+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619815f1f0 con 0x7f61981457c0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.391+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61981f7660 con 0x7f61981457c0 2024-03-06T07:32:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.392+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619804f3a0 con 0x7f61981457c0 2024-03-06T07:32:38.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.393+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f618004c020 con 0x7f61981457c0 2024-03-06T07:32:38.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.395+0000 7f6195600640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f617406b500 0x7f617406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.395+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f618002f020 con 0x7f61981457c0 2024-03-06T07:32:38.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.395+0000 7f619d000640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f617406b500 0x7f617406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.396+0000 7f619d000640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f617406b500 0x7f617406d9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6188005880 tx=0x7f6188009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.399+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6180040070 con 0x7f61981457c0 2024-03-06T07:32:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.641+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f619806bea0 con 0x7f61981457c0 2024-03-06T07:32:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.642+0000 7f6195600640 1 -- 172.21.15.32:0/3502533951 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7f618007acd0 con 0x7f61981457c0 2024-03-06T07:32:38.644 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.644+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f617406b500 msgr2=0x7f617406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.644+0000 7f619fa00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f617406b500 0x7f617406d9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6188005880 tx=0x7f6188009040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.644+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 msgr2=0x7f619815df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.644+0000 7f619fa00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 0x7f619815df50 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6180009890 tx=0x7f61800046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 shutdown_connections 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f617406b500 0x7f617406d9c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f619815e770 0x7f619815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61981457c0 0x7f619815df50 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 --2- 172.21.15.32:0/3502533951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f619806afb0 0x7f61981f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 >> 172.21.15.32:0/3502533951 conn(0x7f61980b30c0 msgr2=0x7f61981424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 shutdown_connections 2024-03-06T07:32:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:38.645+0000 7f619fa00640 1 -- 172.21.15.32:0/3502533951 wait complete. 2024-03-06T07:32:38.666 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:38.666 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:38.666 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:32:38.666 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:38.667 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:38.667 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:38.667 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:38.667 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:38.667 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:38.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:38.667 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:32:38.713 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42179sec preferred_lft 42179sec 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:32:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:38.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:38.715 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:38.842 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:32:38.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:38.847 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:32:38.900 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:32:38.900 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:38.900 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:32:38.900 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:32:38.900 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:38.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:38.901 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:32:38.902 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:32:38.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:32:38.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:32:38.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:32:38.902 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:32:39.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:39.087 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:32:39.147 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:39.147 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:39.148 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:39.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:39.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:32:39.242 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:39.243 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:39.292 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:39.292 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:39.292 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:39.342 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:39.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:39.480 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:39.480 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:32:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:32:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:39.518 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:39.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:39.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:39.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:39.599 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:client5542 2024-03-06T07:32:39.655 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:39.655 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:39.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:32:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.960+0000 7f6cd2400640 1 Processor -- start 2024-03-06T07:32:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.961+0000 7f6cd2400640 1 -- start start 2024-03-06T07:32:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6cd2400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6ccc15dee0 0x7f6ccc0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6cd2400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6ccc0d4860 0x7f6ccc0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6cd2400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ccc0ceca0 con 0x7f6ccc0cfa60 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6cd2400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ccc0cf5d0 con 0x7f6ccc0d4860 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6cd2400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ccc0cef80 con 0x7f6ccc15dee0 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6ccbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6ccc0d4860 0x7f6ccc0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6ccbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6ccc0d4860 0x7f6ccc0d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51088/0 (socket says 172.21.15.32:51088) 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6ccbe00640 1 -- 172.21.15.32:0/1474517976 learned_addr learned my addr 172.21.15.32:0/1474517976 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.964+0000 7f6cd1000640 1 --2- 172.21.15.32:0/1474517976 >> v2:172.21.15.39:3300/0 conn(0x7f6ccc15dee0 0x7f6ccc0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.965+0000 7f6ccbe00640 1 -- 172.21.15.32:0/1474517976 >> v2:172.21.15.39:3300/0 conn(0x7f6ccc15dee0 msgr2=0x7f6ccc0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.965+0000 7f6ccbe00640 1 --2- 172.21.15.32:0/1474517976 >> v2:172.21.15.39:3300/0 conn(0x7f6ccc15dee0 0x7f6ccc0d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.965+0000 7f6ccbe00640 1 -- 172.21.15.32:0/1474517976 >> v1:172.21.15.39:6789/0 conn(0x7f6ccc0cfa60 legacy=0x7f6ccc0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.965+0000 7f6ccbe00640 1 -- 172.21.15.32:0/1474517976 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0cf260 con 0x7f6ccc0d4860 2024-03-06T07:32:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.966+0000 7f6ccbe00640 1 --2- 172.21.15.32:0/1474517976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc0d0630 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc08c6f0 tx=0x7f6cb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b181ad46ae0a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.966+0000 7f6ccb400640 1 -- 172.21.15.32:0/1474517976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8035020 con 0x7f6ccc0d4860 2024-03-06T07:32:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.966+0000 7f6ccb400640 1 -- 172.21.15.32:0/1474517976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb8031870 con 0x7f6ccc0d4860 2024-03-06T07:32:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.966+0000 7f6ccb400640 1 -- 172.21.15.32:0/1474517976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8031c40 con 0x7f6ccc0d4860 2024-03-06T07:32:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 -- 172.21.15.32:0/1474517976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc0d4860 msgr2=0x7f6ccc0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 --2- 172.21.15.32:0/1474517976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc0d0630 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc08c6f0 tx=0x7f6cb802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 -- 172.21.15.32:0/1474517976 shutdown_connections 2024-03-06T07:32:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 --2- 172.21.15.32:0/1474517976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc0d0630 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 --2- 172.21.15.32:0/1474517976 >> v2:172.21.15.39:3300/0 conn(0x7f6ccc15dee0 0x7f6ccc0d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 -- 172.21.15.32:0/1474517976 >> 172.21.15.32:0/1474517976 conn(0x7f6ccc0d3cd0 msgr2=0x7f6ccc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 -- 172.21.15.32:0/1474517976 shutdown_connections 2024-03-06T07:32:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.967+0000 7f6cd2400640 1 -- 172.21.15.32:0/1474517976 wait complete. 2024-03-06T07:32:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.970+0000 7f6cd2400640 1 Processor -- start 2024-03-06T07:32:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.970+0000 7f6cd2400640 1 -- start start 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd2400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ccc0cfa60 0x7f6ccc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 0x7f6ccc06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd2400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0d1de0 con 0x7f6ccc15dee0 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd2400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc15e2c0 con 0x7f6ccc0d4860 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd2400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0d1f60 con 0x7f6ccc0cfa60 2024-03-06T07:32:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 0x7f6ccc06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 0x7f6ccc06c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51100/0 (socket says 172.21.15.32:51100) 2024-03-06T07:32:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd1a00640 1 -- 172.21.15.32:0/8092666 learned_addr learned my addr 172.21.15.32:0/8092666 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6cd1000640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ccc0cfa60 0x7f6ccc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.972+0000 7f6ccbe00640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd1a00640 1 -- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ccc0cfa60 msgr2=0x7f6ccc157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd1a00640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ccc0cfa60 0x7f6ccc157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd1a00640 1 -- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ccc0d4860 msgr2=0x7f6ccc1f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd1a00640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc1f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd1a00640 1 -- 172.21.15.32:0/8092666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc158180 con 0x7f6ccc15dee0 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd1a00640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 0x7f6ccc06c920 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c7b0 tx=0x7f6cc400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4004b60 con 0x7f6ccc15dee0 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc4004d00 con 0x7f6ccc15dee0 2024-03-06T07:32:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc400aa90 con 0x7f6ccc15dee0 2024-03-06T07:32:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ccc1f4530 con 0x7f6ccc15dee0 2024-03-06T07:32:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.973+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ccc1f4df0 con 0x7f6ccc15dee0 2024-03-06T07:32:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.975+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ccc04f330 con 0x7f6ccc15dee0 2024-03-06T07:32:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.976+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6cc400ac30 con 0x7f6ccc15dee0 2024-03-06T07:32:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.977+0000 7f6cc8c00640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6cac06b4d0 0x7f6cac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.977+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cc408a510 con 0x7f6ccc15dee0 2024-03-06T07:32:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.977+0000 7f6cd1000640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6cac06b4d0 0x7f6cac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.978+0000 7f6cd1000640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6cac06b4d0 0x7f6cac06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0006fd0 tx=0x7f6cc0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:39.981+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6cc4020070 con 0x7f6ccc15dee0 2024-03-06T07:32:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.209+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f6ccc06d540 con 0x7f6ccc15dee0 2024-03-06T07:32:40.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.215+0000 7f6cc8c00640 1 -- 172.21.15.32:0/8092666 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6cc40592f0 con 0x7f6ccc15dee0 2024-03-06T07:32:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6cac06b4d0 msgr2=0x7f6cac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6cac06b4d0 0x7f6cac06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0006fd0 tx=0x7f6cc0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 msgr2=0x7f6ccc06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 0x7f6ccc06c920 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c7b0 tx=0x7f6cc400cc70 comp rx=0 tx=0).stop 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 shutdown_connections 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6cac06b4d0 0x7f6cac06d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ccc15dee0 0x7f6ccc06c920 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ccc0d4860 0x7f6ccc1f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 --2- 172.21.15.32:0/8092666 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ccc0cfa60 0x7f6ccc157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.217+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 >> 172.21.15.32:0/8092666 conn(0x7f6ccc0d3cd0 msgr2=0x7f6ccc0b72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.218+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 shutdown_connections 2024-03-06T07:32:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:40.218+0000 7f6cd2400640 1 -- 172.21.15.32:0/8092666 wait complete. 2024-03-06T07:32:49.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:49.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:50.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.467+0000 7fe53ca00640 1 Processor -- start 2024-03-06T07:32:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.467+0000 7fe53ca00640 1 -- start start 2024-03-06T07:32:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.469+0000 7fe53ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5381477e0 0x7fe538162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.469+0000 7fe53ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe53806b6c0 0x7fe5380d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.469+0000 7fe53ca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe538147bc0 con 0x7fe53806ef20 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.469+0000 7fe53ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe538161f90 con 0x7fe5381477e0 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.469+0000 7fe53ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe538162960 con 0x7fe53806b6c0 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.469+0000 7fe536800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5381477e0 0x7fe538162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe536800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5381477e0 0x7fe538162420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47662/0 (socket says 172.21.15.32:47662) 2024-03-06T07:32:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe536800640 1 -- 172.21.15.32:0/1083885980 learned_addr learned my addr 172.21.15.32:0/1083885980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe537200640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe53806b6c0 0x7fe5380d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe536800640 1 -- 172.21.15.32:0/1083885980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe53806b6c0 msgr2=0x7fe5380d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe536800640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe53806b6c0 0x7fe5380d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe536800640 1 -- 172.21.15.32:0/1083885980 >> v1:172.21.15.39:6789/0 conn(0x7fe53806ef20 legacy=0x7fe53806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe536800640 1 -- 172.21.15.32:0/1083885980 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380b3d70 con 0x7fe5381477e0 2024-03-06T07:32:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.470+0000 7fe537200640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe53806b6c0 0x7fe5380d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.472+0000 7fe536800640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5381477e0 0x7fe538162420 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe528009870 tx=0x7fe52802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4238706006083d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.472+0000 7fe535400640 1 -- 172.21.15.32:0/1083885980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280317c0 con 0x7fe5381477e0 2024-03-06T07:32:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.472+0000 7fe535400640 1 -- 172.21.15.32:0/1083885980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528037070 con 0x7fe5381477e0 2024-03-06T07:32:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.472+0000 7fe535400640 1 -- 172.21.15.32:0/1083885980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528031b80 con 0x7fe5381477e0 2024-03-06T07:32:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 -- 172.21.15.32:0/1083885980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5381477e0 msgr2=0x7fe538162420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5381477e0 0x7fe538162420 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe528009870 tx=0x7fe52802b860 comp rx=0 tx=0).stop 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 -- 172.21.15.32:0/1083885980 shutdown_connections 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe53806b6c0 0x7fe5380d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1083885980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5381477e0 0x7fe538162420 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 -- 172.21.15.32:0/1083885980 >> 172.21.15.32:0/1083885980 conn(0x7fe53806c0c0 msgr2=0x7fe53806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 -- 172.21.15.32:0/1083885980 shutdown_connections 2024-03-06T07:32:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.473+0000 7fe53ca00640 1 -- 172.21.15.32:0/1083885980 wait complete. 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.475+0000 7fe53ca00640 1 Processor -- start 2024-03-06T07:32:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.476+0000 7fe53ca00640 1 -- start start 2024-03-06T07:32:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.477+0000 7fe53ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe53806b6c0 0x7fe538160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.477+0000 7fe53ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 0x7fe5380cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.477+0000 7fe53ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5381477e0 0x7fe538141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.477+0000 7fe53ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe538161c20 con 0x7fe53806ef20 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.477+0000 7fe53ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380b4330 con 0x7fe53806b6c0 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.477+0000 7fe53ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53806f8f0 con 0x7fe5381477e0 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 0x7fe5380cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 0x7fe5380cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47664/0 (socket says 172.21.15.32:47664) 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 -- 172.21.15.32:0/1876392567 learned_addr learned my addr 172.21.15.32:0/1876392567 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe536800640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe53806b6c0 0x7fe538160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe537200640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5381477e0 0x7fe538141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 -- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5381477e0 msgr2=0x7fe538141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5381477e0 0x7fe538141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 -- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe53806b6c0 msgr2=0x7fe538160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe53806b6c0 0x7fe538160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe535e00640 1 -- 172.21.15.32:0/1876392567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538142480 con 0x7fe53806ef20 2024-03-06T07:32:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.478+0000 7fe537200640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5381477e0 0x7fe538141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.479+0000 7fe535e00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 0x7fe5380cd660 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe52400c460 tx=0x7fe52400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.479+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524016020 con 0x7fe53806ef20 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.479+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524012870 con 0x7fe53806ef20 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.479+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524012c60 con 0x7fe53806ef20 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.479+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5381f7a80 con 0x7fe53806ef20 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.479+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5381f8040 con 0x7fe53806ef20 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.480+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53804f330 con 0x7fe53806ef20 2024-03-06T07:32:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.481+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe524019020 con 0x7fe53806ef20 2024-03-06T07:32:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.482+0000 7fe52ea00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe51806b4d0 0x7fe51806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.482+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe52408a280 con 0x7fe53806ef20 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.482+0000 7fe536800640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe51806b4d0 0x7fe51806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.483+0000 7fe536800640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe51806b4d0 0x7fe51806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe528036000 tx=0x7fe52803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.492+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe524020020 con 0x7fe53806ef20 2024-03-06T07:32:50.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.732+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5380cdba0 con 0x7fe53806ef20 2024-03-06T07:32:50.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.733+0000 7fe52ea00640 1 -- 172.21.15.32:0/1876392567 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe5240182c0 con 0x7fe53806ef20 2024-03-06T07:32:50.735 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:50.736 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:32:50.736 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.735+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe51806b4d0 msgr2=0x7fe51806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.735+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe51806b4d0 0x7fe51806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe528036000 tx=0x7fe52803d040 comp rx=0 tx=0).stop 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.735+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 msgr2=0x7fe5380cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.735+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 0x7fe5380cd660 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe52400c460 tx=0x7fe52400c920 comp rx=0 tx=0).stop 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 shutdown_connections 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe51806b4d0 0x7fe51806d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe5381477e0 0x7fe538141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe53806ef20 0x7fe5380cd660 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 --2- 172.21.15.32:0/1876392567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe53806b6c0 0x7fe538160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 >> 172.21.15.32:0/1876392567 conn(0x7fe53806c0c0 msgr2=0x7fe53806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 shutdown_connections 2024-03-06T07:32:50.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:50.736+0000 7fe53ca00640 1 -- 172.21.15.32:0/1876392567 wait complete. 2024-03-06T07:32:50.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:32:50.989 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:32:50.989 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:32:50.989 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:32:50.989 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> import uuid 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> n = 100 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = False 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> if 0 > 0: 2024-03-06T07:32:51.040 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = True 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> dpath = os.path.dirname(path) 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> fnameprefix = os.path.basename(path) 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(n): 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> f.write(f"{i}") 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:51.041 DEBUG:teuthology.orchestra.run.smithi032:> f.flush() 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(f.fileno()) 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> os.unlink(fpath) 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> if create_hard_links: 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> for j in range(0): 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> finally: 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> os.close(dirfd) 2024-03-06T07:32:51.042 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:32:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last): 2024-03-06T07:32:51.266 INFO:teuthology.orchestra.run.smithi032.stderr: File "", line 26, in 2024-03-06T07:32:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:32:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:32:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:51.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.535+0000 7fe463200640 1 Processor -- start 2024-03-06T07:32:51.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.535+0000 7fe463200640 1 -- start start 2024-03-06T07:32:51.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe463200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4640d0e50 0x7fe464168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe463200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640b3710 con 0x7fe4640d4a10 2024-03-06T07:32:51.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe463200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640b39f0 con 0x7fe46415e0b0 2024-03-06T07:32:51.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe463200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640b09e0 con 0x7fe4640d0e50 2024-03-06T07:32:51.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe461400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe4640d4a10 0x7fe4640d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58064/0 (socket says 172.21.15.32:58064) 2024-03-06T07:32:51.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe461400640 1 -- 172.21.15.32:0/3632356037 learned_addr learned my addr 172.21.15.32:0/3632356037 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.537+0000 7fe462800640 1 --2- 172.21.15.32:0/3632356037 >> v2:172.21.15.32:3300/0 conn(0x7fe4640d0e50 0x7fe464168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.538+0000 7fe460a00640 1 -- 172.21.15.32:0/3632356037 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118614151 0 0) 0x7fe4640b3710 con 0x7fe4640d4a10 2024-03-06T07:32:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.538+0000 7fe460a00640 1 -- 172.21.15.32:0/3632356037 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe448003520 con 0x7fe4640d4a10 2024-03-06T07:32:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.538+0000 7fe462800640 1 -- 172.21.15.32:0/3632356037 >> v1:172.21.15.39:6789/0 conn(0x7fe46415e0b0 legacy=0x7fe4640cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.538+0000 7fe462800640 1 -- 172.21.15.32:0/3632356037 >> v1:172.21.15.32:6789/0 conn(0x7fe4640d4a10 legacy=0x7fe4640d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.538+0000 7fe462800640 1 -- 172.21.15.32:0/3632356037 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640b3cd0 con 0x7fe4640d0e50 2024-03-06T07:32:51.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.539+0000 7fe462800640 1 --2- 172.21.15.32:0/3632356037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe464168c50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009d90 tx=0x7fe45c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9720be77f0f0dd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.539+0000 7fe460a00640 1 -- 172.21.15.32:0/3632356037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c039070 con 0x7fe4640d0e50 2024-03-06T07:32:51.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.539+0000 7fe460a00640 1 -- 172.21.15.32:0/3632356037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c02cd00 con 0x7fe4640d0e50 2024-03-06T07:32:51.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.539+0000 7fe460a00640 1 -- 172.21.15.32:0/3632356037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c034660 con 0x7fe4640d0e50 2024-03-06T07:32:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 -- 172.21.15.32:0/3632356037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 msgr2=0x7fe464168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 --2- 172.21.15.32:0/3632356037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe464168c50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009d90 tx=0x7fe45c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 -- 172.21.15.32:0/3632356037 shutdown_connections 2024-03-06T07:32:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 --2- 172.21.15.32:0/3632356037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe464168c50 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 -- 172.21.15.32:0/3632356037 >> 172.21.15.32:0/3632356037 conn(0x7fe4640d3d30 msgr2=0x7fe4640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 -- 172.21.15.32:0/3632356037 shutdown_connections 2024-03-06T07:32:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.540+0000 7fe463200640 1 -- 172.21.15.32:0/3632356037 wait complete. 2024-03-06T07:32:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.542+0000 7fe463200640 1 Processor -- start 2024-03-06T07:32:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.542+0000 7fe463200640 1 -- start start 2024-03-06T07:32:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe463200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe4640b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe463200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe4640d4a10 0x7fe464161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe463200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46415e0b0 0x7fe464152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe463200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46415e490 con 0x7fe4640d0e50 2024-03-06T07:32:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe463200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640cfa20 con 0x7fe46415e0b0 2024-03-06T07:32:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe463200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640cfc60 con 0x7fe4640d4a10 2024-03-06T07:32:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe461e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe4640b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe461e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe4640b87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47676/0 (socket says 172.21.15.32:47676) 2024-03-06T07:32:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe461e00640 1 -- 172.21.15.32:0/2380283367 learned_addr learned my addr 172.21.15.32:0/2380283367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.543+0000 7fe462800640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46415e0b0 0x7fe464152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461400640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe4640d4a10 0x7fe464161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461e00640 1 -- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe4640d4a10 msgr2=0x7fe464161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461e00640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe4640d4a10 0x7fe464161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461e00640 1 -- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46415e0b0 msgr2=0x7fe464152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461e00640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46415e0b0 0x7fe464152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461e00640 1 -- 172.21.15.32:0/2380283367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640b8e60 con 0x7fe4640d0e50 2024-03-06T07:32:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe461e00640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe4640b87f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c400 tx=0x7fe45400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454017070 con 0x7fe4640d0e50 2024-03-06T07:32:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4540052d0 con 0x7fe4640d0e50 2024-03-06T07:32:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454003830 con 0x7fe4640d0e50 2024-03-06T07:32:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.544+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4640b9140 con 0x7fe4640d0e50 2024-03-06T07:32:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.545+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4640b9700 con 0x7fe4640d0e50 2024-03-06T07:32:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.546+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46404f330 con 0x7fe4640d0e50 2024-03-06T07:32:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.547+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe45401a070 con 0x7fe4640d0e50 2024-03-06T07:32:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.548+0000 7fe45a000640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe44406b4d0 0x7fe44406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.548+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe454089580 con 0x7fe4640d0e50 2024-03-06T07:32:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.548+0000 7fe461400640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe44406b4d0 0x7fe44406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.549+0000 7fe461400640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe44406b4d0 0x7fe44406d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe4641f7160 tx=0x7fe450009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.554+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4540583b0 con 0x7fe4640d0e50 2024-03-06T07:32:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.795+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4640b2e40 con 0x7fe4640d0e50 2024-03-06T07:32:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.796+0000 7fe45a000640 1 -- 172.21.15.32:0/2380283367 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe454057cb0 con 0x7fe4640d0e50 2024-03-06T07:32:51.798 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:51.799 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:32:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:32:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.799+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe44406b4d0 msgr2=0x7fe44406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.799+0000 7fe463200640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe44406b4d0 0x7fe44406d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe4641f7160 tx=0x7fe450009040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 msgr2=0x7fe4640b87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe4640b87f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c400 tx=0x7fe45400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 shutdown_connections 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe44406b4d0 0x7fe44406d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46415e0b0 0x7fe464152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe4640d4a10 0x7fe464161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 --2- 172.21.15.32:0/2380283367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4640d0e50 0x7fe4640b87f0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 >> 172.21.15.32:0/2380283367 conn(0x7fe4640d3d30 msgr2=0x7fe46415bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 shutdown_connections 2024-03-06T07:32:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:51.800+0000 7fe463200640 1 -- 172.21.15.32:0/2380283367 wait complete. 2024-03-06T07:32:51.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:32:52.059 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:32:52.059 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:32:52.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:52.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.279+0000 7ff037e00640 1 Processor -- start 2024-03-06T07:32:52.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.279+0000 7ff037e00640 1 -- start start 2024-03-06T07:32:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.280+0000 7ff037e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d4a10 0x7ff0380d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.280+0000 7ff037e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0380d0c80 0x7ff0380b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.280+0000 7ff037e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0380b3710 con 0x7ff03815e0b0 2024-03-06T07:32:52.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.280+0000 7ff037e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0380b09e0 con 0x7ff0380d0c80 2024-03-06T07:32:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.280+0000 7ff037e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380cfa20 con 0x7ff0380d4a10 2024-03-06T07:32:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.280+0000 7ff037400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0380d0c80 0x7ff0380b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff037400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0380d0c80 0x7ff0380b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47688/0 (socket says 172.21.15.32:47688) 2024-03-06T07:32:52.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff037400640 1 -- 172.21.15.32:0/1918349189 learned_addr learned my addr 172.21.15.32:0/1918349189 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:52.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff02e000640 1 --2- 172.21.15.32:0/1918349189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d4a10 0x7ff0380d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff037400640 1 -- 172.21.15.32:0/1918349189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d4a10 msgr2=0x7ff0380d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff037400640 1 --2- 172.21.15.32:0/1918349189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d4a10 0x7ff0380d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff037400640 1 -- 172.21.15.32:0/1918349189 >> v1:172.21.15.39:6789/0 conn(0x7ff03815e0b0 legacy=0x7ff0380cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.281+0000 7ff037400640 1 -- 172.21.15.32:0/1918349189 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0380b39f0 con 0x7ff0380d0c80 2024-03-06T07:32:52.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.284+0000 7ff037400640 1 --2- 172.21.15.32:0/1918349189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d0c80 0x7ff0380b44d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff030009e50 tx=0x7ff03002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fded1dd33cd1b03d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.284+0000 7ff036000640 1 -- 172.21.15.32:0/1918349189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03002cb00 con 0x7ff0380d0c80 2024-03-06T07:32:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.284+0000 7ff036000640 1 -- 172.21.15.32:0/1918349189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03002cca0 con 0x7ff0380d0c80 2024-03-06T07:32:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.284+0000 7ff036000640 1 -- 172.21.15.32:0/1918349189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030034610 con 0x7ff0380d0c80 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 -- 172.21.15.32:0/1918349189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d0c80 msgr2=0x7ff0380b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 --2- 172.21.15.32:0/1918349189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d0c80 0x7ff0380b44d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff030009e50 tx=0x7ff03002bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 -- 172.21.15.32:0/1918349189 shutdown_connections 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 --2- 172.21.15.32:0/1918349189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d0c80 0x7ff0380b44d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 --2- 172.21.15.32:0/1918349189 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d4a10 0x7ff0380d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 -- 172.21.15.32:0/1918349189 >> 172.21.15.32:0/1918349189 conn(0x7ff0380d3d30 msgr2=0x7ff0380d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 -- 172.21.15.32:0/1918349189 shutdown_connections 2024-03-06T07:32:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.285+0000 7ff037e00640 1 -- 172.21.15.32:0/1918349189 wait complete. 2024-03-06T07:32:52.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.287+0000 7ff037e00640 1 Processor -- start 2024-03-06T07:32:52.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.287+0000 7ff037e00640 1 -- start start 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.288+0000 7ff037e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d0c80 0x7ff03814eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.288+0000 7ff037e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 0x7ff038160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.288+0000 7ff037e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff03815e0b0 0x7ff03806c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.288+0000 7ff037e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03815e490 con 0x7ff0380d4a10 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.288+0000 7ff037e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380b3fb0 con 0x7ff03815e0b0 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.288+0000 7ff037e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380cfc60 con 0x7ff0380d0c80 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 0x7ff038160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 0x7ff038160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47696/0 (socket says 172.21.15.32:47696) 2024-03-06T07:32:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 -- 172.21.15.32:0/1509349997 learned_addr learned my addr 172.21.15.32:0/1509349997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff037400640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff03815e0b0 0x7ff03806c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff036a00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d0c80 0x7ff03814eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 -- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d0c80 msgr2=0x7ff03814eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d0c80 0x7ff03814eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 -- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff03815e0b0 msgr2=0x7ff03806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff03815e0b0 0x7ff03806c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff02e000640 1 -- 172.21.15.32:0/1509349997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03815fd00 con 0x7ff0380d4a10 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.289+0000 7ff037400640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff03815e0b0 0x7ff03806c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.290+0000 7ff02e000640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 0x7ff038160eb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff02000e980 tx=0x7ff02000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.290+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020017070 con 0x7ff0380d4a10 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.290+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02000c9a0 con 0x7ff0380d4a10 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.290+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02000cd70 con 0x7ff0380d4a10 2024-03-06T07:32:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.290+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03815ffe0 con 0x7ff0380d4a10 2024-03-06T07:32:52.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.290+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff038160990 con 0x7ff0380d4a10 2024-03-06T07:32:52.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.291+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03804f330 con 0x7ff0380d4a10 2024-03-06T07:32:52.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.292+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff02001a070 con 0x7ff0380d4a10 2024-03-06T07:32:52.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.293+0000 7ff02f400640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff01406b4d0 0x7ff01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.293+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff02008a590 con 0x7ff0380d4a10 2024-03-06T07:32:52.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.293+0000 7ff036a00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff01406b4d0 0x7ff01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.294+0000 7ff036a00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff01406b4d0 0x7ff01406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff0280015a0 tx=0x7ff028009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.298+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0200593c0 con 0x7ff0380d4a10 2024-03-06T07:32:52.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.544+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0380b1800 con 0x7ff0380d4a10 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.545+0000 7ff02f400640 1 -- 172.21.15.32:0/1509349997 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7ff020058cc0 con 0x7ff0380d4a10 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:52.547 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:32:52.548 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff01406b4d0 msgr2=0x7ff01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff01406b4d0 0x7ff01406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff0280015a0 tx=0x7ff028009040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 msgr2=0x7ff038160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 0x7ff038160eb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff02000e980 tx=0x7ff02000ee40 comp rx=0 tx=0).stop 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 shutdown_connections 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff01406b4d0 0x7ff01406d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff03815e0b0 0x7ff03806c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0380d4a10 0x7ff038160eb0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 --2- 172.21.15.32:0/1509349997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0380d0c80 0x7ff03814eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 >> 172.21.15.32:0/1509349997 conn(0x7ff0380d3d30 msgr2=0x7ff0380cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 shutdown_connections 2024-03-06T07:32:52.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:52.548+0000 7ff037e00640 1 -- 172.21.15.32:0/1509349997 wait complete. 2024-03-06T07:32:52.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:32:52.793 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:32:52.794 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:32:52.794 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:32:57.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.025+0000 7fe39be00640 1 Processor -- start 2024-03-06T07:32:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.025+0000 7fe39be00640 1 -- start start 2024-03-06T07:32:58.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe39be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3940af580 0x7fe39415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe39be00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39415cc40 con 0x7fe3940af1a0 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe39be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39415cf20 con 0x7fe3940cdc20 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe39be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39415d200 con 0x7fe3940af580 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe398a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3940af580 0x7fe39415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe399400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe3940af1a0 0x7fe3940b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52426/0 (socket says 172.21.15.32:52426) 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.027+0000 7fe399400640 1 -- 172.21.15.32:0/1448499122 learned_addr learned my addr 172.21.15.32:0/1448499122 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.028+0000 7fe398a00640 1 -- 172.21.15.32:0/1448499122 >> v1:172.21.15.39:6789/0 conn(0x7fe3940cdc20 legacy=0x7fe3940b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.028+0000 7fe393e00640 1 -- 172.21.15.32:0/1448499122 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652991742 0 0) 0x7fe39415cc40 con 0x7fe3940af1a0 2024-03-06T07:32:58.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.028+0000 7fe398a00640 1 -- 172.21.15.32:0/1448499122 >> v1:172.21.15.32:6789/0 conn(0x7fe3940af1a0 legacy=0x7fe3940b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.028+0000 7fe398a00640 1 -- 172.21.15.32:0/1448499122 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39415d4e0 con 0x7fe3940af580 2024-03-06T07:32:58.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.029+0000 7fe398a00640 1 --2- 172.21.15.32:0/1448499122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39415c600 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe3800097a0 tx=0x7fe38002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1380b7ba8139e111 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.029+0000 7fe393e00640 1 -- 172.21.15.32:0/1448499122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3800315a0 con 0x7fe3940af580 2024-03-06T07:32:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe393e00640 1 -- 172.21.15.32:0/1448499122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380037070 con 0x7fe3940af580 2024-03-06T07:32:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe393e00640 1 -- 172.21.15.32:0/1448499122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380031960 con 0x7fe3940af580 2024-03-06T07:32:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 -- 172.21.15.32:0/1448499122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 msgr2=0x7fe39415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 --2- 172.21.15.32:0/1448499122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39415c600 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe3800097a0 tx=0x7fe38002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:58.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 -- 172.21.15.32:0/1448499122 shutdown_connections 2024-03-06T07:32:58.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 --2- 172.21.15.32:0/1448499122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39415c600 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 -- 172.21.15.32:0/1448499122 >> 172.21.15.32:0/1448499122 conn(0x7fe39406afb0 msgr2=0x7fe39406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 -- 172.21.15.32:0/1448499122 shutdown_connections 2024-03-06T07:32:58.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.030+0000 7fe39be00640 1 -- 172.21.15.32:0/1448499122 wait complete. 2024-03-06T07:32:58.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.032+0000 7fe39be00640 1 Processor -- start 2024-03-06T07:32:58.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.032+0000 7fe39be00640 1 -- start start 2024-03-06T07:32:58.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe39be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3940af1a0 0x7fe39409d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe39be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe39be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3940cdc20 0x7fe39415eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe39be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394155e70 con 0x7fe3940af580 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe39be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3940b3730 con 0x7fe3940af1a0 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe39be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe394155ff0 con 0x7fe3940cdc20 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe398a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe398a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39409aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34682/0 (socket says 172.21.15.32:34682) 2024-03-06T07:32:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.033+0000 7fe398a00640 1 -- 172.21.15.32:0/524846747 learned_addr learned my addr 172.21.15.32:0/524846747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:32:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe399e00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3940cdc20 0x7fe39415eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe399400640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3940af1a0 0x7fe39409d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe398a00640 1 -- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3940cdc20 msgr2=0x7fe39415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe398a00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3940cdc20 0x7fe39415eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe398a00640 1 -- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3940af1a0 msgr2=0x7fe39409d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe398a00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3940af1a0 0x7fe39409d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.034+0000 7fe398a00640 1 -- 172.21.15.32:0/524846747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39406cc40 con 0x7fe3940af580 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.035+0000 7fe398a00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39409aa10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe380034040 tx=0x7fe3800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.035+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380032840 con 0x7fe3940af580 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.035+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3800329e0 con 0x7fe3940af580 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.035+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39406cf20 con 0x7fe3940af580 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.035+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380032dd0 con 0x7fe3940af580 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.035+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39406d4e0 con 0x7fe3940af580 2024-03-06T07:32:58.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.036+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe394069510 con 0x7fe3940af580 2024-03-06T07:32:58.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.038+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe38003b3f0 con 0x7fe3940af580 2024-03-06T07:32:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.039+0000 7fe391600640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe37406b4d0 0x7fe37406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.039+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe38002f030 con 0x7fe3940af580 2024-03-06T07:32:58.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.040+0000 7fe399400640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe37406b4d0 0x7fe37406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.040+0000 7fe399400640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe37406b4d0 0x7fe37406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe388007fd0 tx=0x7fe388009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.044+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe380040070 con 0x7fe3940af580 2024-03-06T07:32:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.286+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3940726a0 con 0x7fe3940af580 2024-03-06T07:32:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.286+0000 7fe391600640 1 -- 172.21.15.32:0/524846747 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe3940726a0 con 0x7fe3940af580 2024-03-06T07:32:58.288 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:32:58.288 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:32:58.288 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.289+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe37406b4d0 msgr2=0x7fe37406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.289+0000 7fe39be00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe37406b4d0 0x7fe37406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe388007fd0 tx=0x7fe388009040 comp rx=0 tx=0).stop 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.289+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 msgr2=0x7fe39409aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.289+0000 7fe39be00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39409aa10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe380034040 tx=0x7fe3800046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 shutdown_connections 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe37406b4d0 0x7fe37406d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3940cdc20 0x7fe39415eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3940af580 0x7fe39409aa10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 --2- 172.21.15.32:0/524846747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3940af1a0 0x7fe39409d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 >> 172.21.15.32:0/524846747 conn(0x7fe39406afb0 msgr2=0x7fe3940cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 shutdown_connections 2024-03-06T07:32:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:32:58.290+0000 7fe39be00640 1 -- 172.21.15.32:0/524846747 wait complete. 2024-03-06T07:32:58.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:32:58.553 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:32:58.554 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:32:58.554 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:33:03.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:03.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.786+0000 7fd2b3200640 1 Processor -- start 2024-03-06T07:33:03.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.786+0000 7fd2b3200640 1 -- start start 2024-03-06T07:33:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2b415e0b0 0x7fd2b40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2b40d4a10 0x7fd2b40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b40b3710 con 0x7fd2b40d0c80 2024-03-06T07:33:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b40d10f0 con 0x7fd2b415e0b0 2024-03-06T07:33:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b40b09e0 con 0x7fd2b40d4a10 2024-03-06T07:33:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b2800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd2b40d0c80 0x7fd2b40d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52434/0 (socket says 172.21.15.32:52434) 2024-03-06T07:33:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b2800640 1 -- 172.21.15.32:0/1788183762 learned_addr learned my addr 172.21.15.32:0/1788183762 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.788+0000 7fd2b1e00640 1 --2- 172.21.15.32:0/1788183762 >> v2:172.21.15.32:3300/0 conn(0x7fd2b415e0b0 0x7fd2b40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b1400640 1 --2- 172.21.15.32:0/1788183762 >> v2:172.21.15.39:3300/0 conn(0x7fd2b40d4a10 0x7fd2b40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b0a00640 1 -- 172.21.15.32:0/1788183762 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384864242 0 0) 0x7fd2b40b3710 con 0x7fd2b40d0c80 2024-03-06T07:33:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b0a00640 1 -- 172.21.15.32:0/1788183762 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd298003520 con 0x7fd2b40d0c80 2024-03-06T07:33:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b1e00640 1 -- 172.21.15.32:0/1788183762 >> v2:172.21.15.39:3300/0 conn(0x7fd2b40d4a10 msgr2=0x7fd2b40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b1e00640 1 --2- 172.21.15.32:0/1788183762 >> v2:172.21.15.39:3300/0 conn(0x7fd2b40d4a10 0x7fd2b40d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b1e00640 1 -- 172.21.15.32:0/1788183762 >> v1:172.21.15.32:6789/0 conn(0x7fd2b40d0c80 legacy=0x7fd2b40d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.789+0000 7fd2b1e00640 1 -- 172.21.15.32:0/1788183762 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40b39f0 con 0x7fd2b415e0b0 2024-03-06T07:33:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.790+0000 7fd2b1e00640 1 --2- 172.21.15.32:0/1788183762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b415e0b0 0x7fd2b40cf310 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd2b406b8c0 tx=0x7fd2a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a8e171da31466e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.791+0000 7fd2b0a00640 1 -- 172.21.15.32:0/1788183762 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4035020 con 0x7fd2b415e0b0 2024-03-06T07:33:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.791+0000 7fd2b0a00640 1 -- 172.21.15.32:0/1788183762 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4031830 con 0x7fd2b415e0b0 2024-03-06T07:33:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.791+0000 7fd2b0a00640 1 -- 172.21.15.32:0/1788183762 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4031be0 con 0x7fd2b415e0b0 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 -- 172.21.15.32:0/1788183762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b415e0b0 msgr2=0x7fd2b40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 --2- 172.21.15.32:0/1788183762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b415e0b0 0x7fd2b40cf310 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd2b406b8c0 tx=0x7fd2a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 -- 172.21.15.32:0/1788183762 shutdown_connections 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 --2- 172.21.15.32:0/1788183762 >> v2:172.21.15.39:3300/0 conn(0x7fd2b40d4a10 0x7fd2b40d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 --2- 172.21.15.32:0/1788183762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b415e0b0 0x7fd2b40cf310 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 -- 172.21.15.32:0/1788183762 >> 172.21.15.32:0/1788183762 conn(0x7fd2b40d3d30 msgr2=0x7fd2b40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 -- 172.21.15.32:0/1788183762 shutdown_connections 2024-03-06T07:33:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.792+0000 7fd2b3200640 1 -- 172.21.15.32:0/1788183762 wait complete. 2024-03-06T07:33:03.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.795+0000 7fd2b3200640 1 Processor -- start 2024-03-06T07:33:03.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.795+0000 7fd2b3200640 1 -- start start 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 0x7fd2b415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2b40d4a10 0x7fd2b40ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2b415e0b0 0x7fd2b41f7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b415e490 con 0x7fd2b40d0c80 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b3fb0 con 0x7fd2b40d4a10 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40cfc60 con 0x7fd2b415e0b0 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 0x7fd2b415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 0x7fd2b415ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34688/0 (socket says 172.21.15.32:34688) 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b1e00640 1 -- 172.21.15.32:0/2354296376 learned_addr learned my addr 172.21.15.32:0/2354296376 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.796+0000 7fd2b1e00640 1 -- 172.21.15.32:0/2354296376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2b415e0b0 msgr2=0x7fd2b41f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b1e00640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2b415e0b0 0x7fd2b41f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b1e00640 1 -- 172.21.15.32:0/2354296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2b40d4a10 msgr2=0x7fd2b40ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b1e00640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2b40d4a10 0x7fd2b40ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b1e00640 1 -- 172.21.15.32:0/2354296376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40cc230 con 0x7fd2b40d0c80 2024-03-06T07:33:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b1e00640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 0x7fd2b415ac70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd2a402b910 tx=0x7fd2a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4040070 con 0x7fd2b40d0c80 2024-03-06T07:33:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a403a910 con 0x7fd2b40d0c80 2024-03-06T07:33:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4035040 con 0x7fd2b40d0c80 2024-03-06T07:33:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b40cd7c0 con 0x7fd2b40d0c80 2024-03-06T07:33:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.797+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b40cdd80 con 0x7fd2b40d0c80 2024-03-06T07:33:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.798+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b404f330 con 0x7fd2b40d0c80 2024-03-06T07:33:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.799+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2a404b060 con 0x7fd2b40d0c80 2024-03-06T07:33:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.801+0000 7fd2aa000640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd29406b500 0x7fd29406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.801+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd2a40ab9c0 con 0x7fd2b40d0c80 2024-03-06T07:33:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.801+0000 7fd2b1400640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd29406b500 0x7fd29406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.802+0000 7fd2b1400640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd29406b500 0x7fd29406d9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40cadb0 tx=0x7fd2a0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:03.805+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2a40ade20 con 0x7fd2b40d0c80 2024-03-06T07:33:04.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.043+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2b40cea10 con 0x7fd2b40d0c80 2024-03-06T07:33:04.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.044+0000 7fd2aa000640 1 -- 172.21.15.32:0/2354296376 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fd2a407a890 con 0x7fd2b40d0c80 2024-03-06T07:33:04.046 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:04.046 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:33:04.046 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:33:04.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd29406b500 msgr2=0x7fd29406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd29406b500 0x7fd29406d9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40cadb0 tx=0x7fd2a0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:04.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 msgr2=0x7fd2b415ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 0x7fd2b415ac70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd2a402b910 tx=0x7fd2a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 shutdown_connections 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd29406b500 0x7fd29406d9c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2b415e0b0 0x7fd2b41f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2b40d4a10 0x7fd2b40ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 --2- 172.21.15.32:0/2354296376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2b40d0c80 0x7fd2b415ac70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 >> 172.21.15.32:0/2354296376 conn(0x7fd2b40d3d30 msgr2=0x7fd2b40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 shutdown_connections 2024-03-06T07:33:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.048+0000 7fd2b3200640 1 -- 172.21.15.32:0/2354296376 wait complete. 2024-03-06T07:33:04.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:33:04.305 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:33:04.305 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:33:04.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.526+0000 7fee4ce00640 1 Processor -- start 2024-03-06T07:33:04.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.526+0000 7fee4ce00640 1 -- start start 2024-03-06T07:33:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.527+0000 7fee4ce00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fee4815dec0 0x7fee480ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.527+0000 7fee4ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815e2a0 0x7fee480ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.527+0000 7fee4ce00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480cf720 con 0x7fee480cf290 2024-03-06T07:33:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.527+0000 7fee4ce00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee480b0ea0 con 0x7fee4815dec0 2024-03-06T07:33:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.527+0000 7fee4ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee480d1120 con 0x7fee4815e2a0 2024-03-06T07:33:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.528+0000 7fee3ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815e2a0 0x7fee480ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.528+0000 7fee47400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fee4815dec0 0x7fee480ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.528+0000 7fee3ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815e2a0 0x7fee480ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:35078/0 (socket says 172.21.15.32:35078) 2024-03-06T07:33:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.528+0000 7fee3ea00640 1 -- 172.21.15.32:0/96767130 learned_addr learned my addr 172.21.15.32:0/96767130 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.529+0000 7fee47400640 1 -- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815e2a0 msgr2=0x7fee480ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.529+0000 7fee47400640 1 --2- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815e2a0 0x7fee480ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.529+0000 7fee47400640 1 -- 172.21.15.32:0/96767130 >> v1:172.21.15.39:6789/0 conn(0x7fee480cf290 legacy=0x7fee480d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.529+0000 7fee47400640 1 -- 172.21.15.32:0/96767130 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee480d1400 con 0x7fee4815dec0 2024-03-06T07:33:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.530+0000 7fee47400640 1 --2- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4815dec0 0x7fee480ce810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fee38009d90 tx=0x7fee3802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc203aab5f86251 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.531+0000 7fee46a00640 1 -- 172.21.15.32:0/96767130 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38039070 con 0x7fee4815dec0 2024-03-06T07:33:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.531+0000 7fee46a00640 1 -- 172.21.15.32:0/96767130 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3802cc50 con 0x7fee4815dec0 2024-03-06T07:33:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.531+0000 7fee46a00640 1 -- 172.21.15.32:0/96767130 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38034440 con 0x7fee4815dec0 2024-03-06T07:33:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 -- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4815dec0 msgr2=0x7fee480ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 --2- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4815dec0 0x7fee480ce810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fee38009d90 tx=0x7fee3802bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 -- 172.21.15.32:0/96767130 shutdown_connections 2024-03-06T07:33:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 --2- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815e2a0 0x7fee480ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 --2- 172.21.15.32:0/96767130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4815dec0 0x7fee480ce810 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 -- 172.21.15.32:0/96767130 >> 172.21.15.32:0/96767130 conn(0x7fee480d3d40 msgr2=0x7fee480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 -- 172.21.15.32:0/96767130 shutdown_connections 2024-03-06T07:33:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.532+0000 7fee4ce00640 1 -- 172.21.15.32:0/96767130 wait complete. 2024-03-06T07:33:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.534+0000 7fee4ce00640 1 Processor -- start 2024-03-06T07:33:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.534+0000 7fee4ce00640 1 -- start start 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee4ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee480cf290 0x7fee48153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee4ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815dec0 0x7fee48160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee4ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 0x7fee48160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee4ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480d0ec0 con 0x7fee4815e2a0 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee4ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480d0d40 con 0x7fee480cf290 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee4ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee480b2340 con 0x7fee4815dec0 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 0x7fee48160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 0x7fee48160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34696/0 (socket says 172.21.15.32:34696) 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee47e00640 1 -- 172.21.15.32:0/553630327 learned_addr learned my addr 172.21.15.32:0/553630327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee47400640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee480cf290 0x7fee48153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.535+0000 7fee3ea00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815dec0 0x7fee48160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee47e00640 1 -- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815dec0 msgr2=0x7fee48160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee47e00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815dec0 0x7fee48160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee47e00640 1 -- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee480cf290 msgr2=0x7fee48153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee47e00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee480cf290 0x7fee48153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee47e00640 1 -- 172.21.15.32:0/553630327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee480b6c00 con 0x7fee4815e2a0 2024-03-06T07:33:04.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee47e00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 0x7fee48160680 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee40001eb0 tx=0x7fee4000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40017070 con 0x7fee4815e2a0 2024-03-06T07:33:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee400052d0 con 0x7fee4815e2a0 2024-03-06T07:33:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee480b6ee0 con 0x7fee4815e2a0 2024-03-06T07:33:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.536+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40003810 con 0x7fee4815e2a0 2024-03-06T07:33:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.537+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee480b74a0 con 0x7fee4815e2a0 2024-03-06T07:33:04.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.538+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4804f330 con 0x7fee4815e2a0 2024-03-06T07:33:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.540+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee4001a070 con 0x7fee4815e2a0 2024-03-06T07:33:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.541+0000 7fee3fe00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.541+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fee400895f0 con 0x7fee4815e2a0 2024-03-06T07:33:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.541+0000 7fee47400640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.542+0000 7fee47400640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee2406b4d0 0x7fee2406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fee38036000 tx=0x7fee38030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.546+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee480cf720 con 0x7fee4815e2a0 2024-03-06T07:33:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.787+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee480d2230 con 0x7fee4815e2a0 2024-03-06T07:33:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.788+0000 7fee3fe00640 1 -- 172.21.15.32:0/553630327 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fee40058420 con 0x7fee4815e2a0 2024-03-06T07:33:04.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:04.790 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:33:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:33:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee2406b4d0 msgr2=0x7fee2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee2406b4d0 0x7fee2406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fee38036000 tx=0x7fee38030000 comp rx=0 tx=0).stop 2024-03-06T07:33:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 msgr2=0x7fee48160680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 0x7fee48160680 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee40001eb0 tx=0x7fee4000c890 comp rx=0 tx=0).stop 2024-03-06T07:33:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 shutdown_connections 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4815e2a0 0x7fee48160680 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fee4815dec0 0x7fee48160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 --2- 172.21.15.32:0/553630327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee480cf290 0x7fee48153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 >> 172.21.15.32:0/553630327 conn(0x7fee480d3d40 msgr2=0x7fee4806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 shutdown_connections 2024-03-06T07:33:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:04.791+0000 7fee4ce00640 1 -- 172.21.15.32:0/553630327 wait complete. 2024-03-06T07:33:04.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:33:05.034 INFO:teuthology.orchestra.run.smithi032.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:33:05.034 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:33:05.034 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:05.034 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:05.081 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> import uuid 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> n = 10 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = False 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> if 0 > 0: 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> create_hard_links = True 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> dpath = os.path.dirname(path) 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> fnameprefix = os.path.basename(path) 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:33:05.082 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(n): 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> f.write(f"{i}") 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> f.flush() 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(f.fileno()) 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> os.unlink(fpath) 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> if True: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> if create_hard_links: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> for j in range(0): 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> if False: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> os.fsync(dirfd) 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> finally: 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> os.close(dirfd) 2024-03-06T07:33:05.083 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:33:05.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:33:05.383 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:05.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:05.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:05.657 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:05.658 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:05.658 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:05.714 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:05.714 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:05.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:06.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.020+0000 7f67efe00640 1 Processor -- start 2024-03-06T07:33:06.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.020+0000 7f67efe00640 1 -- start start 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.023+0000 7f67efe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f015e0b0 0x7f67f00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.023+0000 7f67efe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f67f00d4a10 0x7f67f00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.023+0000 7f67efe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67f00d0c80 0x7f67f00d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.023+0000 7f67efe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00b09e0 con 0x7f67f00d0c80 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.023+0000 7f67efe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00b3710 con 0x7f67f00d4a10 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.023+0000 7f67efe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00b39f0 con 0x7f67f015e0b0 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67f00d0c80 0x7f67f00d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67f00d0c80 0x7f67f00d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34708/0 (socket says 172.21.15.32:34708) 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 -- 172.21.15.32:0/4127329559 learned_addr learned my addr 172.21.15.32:0/4127329559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67eea00640 1 --2- 172.21.15.32:0/4127329559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f015e0b0 0x7f67f00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ee000640 1 --2- 172.21.15.32:0/4127329559 >> v2:172.21.15.39:3300/0 conn(0x7f67f00d4a10 0x7f67f00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 -- 172.21.15.32:0/4127329559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f015e0b0 msgr2=0x7f67f00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 --2- 172.21.15.32:0/4127329559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f015e0b0 0x7f67f00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 -- 172.21.15.32:0/4127329559 >> v2:172.21.15.39:3300/0 conn(0x7f67f00d4a10 msgr2=0x7f67f00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 --2- 172.21.15.32:0/4127329559 >> v2:172.21.15.39:3300/0 conn(0x7f67f00d4a10 0x7f67f00d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.024+0000 7f67ef400640 1 -- 172.21.15.32:0/4127329559 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f00b3cd0 con 0x7f67f00d0c80 2024-03-06T07:33:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.025+0000 7f67ef400640 1 --2- 172.21.15.32:0/4127329559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00d1900 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b6920 tx=0x7f67e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1567767ba189c9ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.026+0000 7f67ed600640 1 -- 172.21.15.32:0/4127329559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e802ca70 con 0x7f67f00d0c80 2024-03-06T07:33:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.026+0000 7f67ed600640 1 -- 172.21.15.32:0/4127329559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e802cc10 con 0x7f67f00d0c80 2024-03-06T07:33:06.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.026+0000 7f67ed600640 1 -- 172.21.15.32:0/4127329559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80343f0 con 0x7f67f00d0c80 2024-03-06T07:33:06.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 -- 172.21.15.32:0/4127329559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 msgr2=0x7f67f00d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 --2- 172.21.15.32:0/4127329559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00d1900 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b6920 tx=0x7f67e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 -- 172.21.15.32:0/4127329559 shutdown_connections 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 --2- 172.21.15.32:0/4127329559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00d1900 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 --2- 172.21.15.32:0/4127329559 >> v2:172.21.15.39:3300/0 conn(0x7f67f00d4a10 0x7f67f00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 --2- 172.21.15.32:0/4127329559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f015e0b0 0x7f67f00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 -- 172.21.15.32:0/4127329559 >> 172.21.15.32:0/4127329559 conn(0x7f67f00d3d30 msgr2=0x7f67f00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 -- 172.21.15.32:0/4127329559 shutdown_connections 2024-03-06T07:33:06.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.027+0000 7f67efe00640 1 -- 172.21.15.32:0/4127329559 wait complete. 2024-03-06T07:33:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.029+0000 7f67efe00640 1 Processor -- start 2024-03-06T07:33:06.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.029+0000 7f67efe00640 1 -- start start 2024-03-06T07:33:06.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67efe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f00d4a10 0x7f67f014f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67efe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f015e0b0 0x7f67f006ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67efe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00cfc60 con 0x7f67f00d0c80 2024-03-06T07:33:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67efe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f015e490 con 0x7f67f015e0b0 2024-03-06T07:33:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67efe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00cf850 con 0x7f67f00d4a10 2024-03-06T07:33:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67eea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67eea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34714/0 (socket says 172.21.15.32:34714) 2024-03-06T07:33:06.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.030+0000 7f67eea00640 1 -- 172.21.15.32:0/1931016379 learned_addr learned my addr 172.21.15.32:0/1931016379 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67ee000640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f00d4a10 0x7f67f014f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67ef400640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f015e0b0 0x7f67f006ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67eea00640 1 -- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f00d4a10 msgr2=0x7f67f014f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67eea00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f00d4a10 0x7f67f014f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67eea00640 1 -- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f015e0b0 msgr2=0x7f67f006ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67eea00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f015e0b0 0x7f67f006ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67eea00640 1 -- 172.21.15.32:0/1931016379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f014f6c0 con 0x7f67f00d0c80 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67ef400640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f015e0b0 0x7f67f006ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67eea00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00cb7a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f67e000e870 tx=0x7f67e000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.031+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0017070 con 0x7f67f00d0c80 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.032+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e000c940 con 0x7f67f00d0c80 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.032+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e000cd30 con 0x7f67f00d0c80 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.032+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f00cd5e0 con 0x7f67f00d0c80 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.032+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f00cb280 con 0x7f67f00d0c80 2024-03-06T07:33:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.033+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f004f330 con 0x7f67f00d0c80 2024-03-06T07:33:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.034+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67e001a070 con 0x7f67f00d0c80 2024-03-06T07:33:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.035+0000 7f67e6a00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67d006b500 0x7f67d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.035+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f67e008a010 con 0x7f67f00d0c80 2024-03-06T07:33:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.035+0000 7f67ee000640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67d006b500 0x7f67d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.036+0000 7f67ee000640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67d006b500 0x7f67d006d9c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f67f00ccf50 tx=0x7f67dc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.040+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67f014f6c0 con 0x7f67f00d0c80 2024-03-06T07:33:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.243+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67f00cd7b0 con 0x7f67f00d0c80 2024-03-06T07:33:06.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.246+0000 7f67e6a00640 1 -- 172.21.15.32:0/1931016379 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f67e0058e40 con 0x7f67f00d0c80 2024-03-06T07:33:06.248 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:06.248 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":70,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:48.336272+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67d006b500 msgr2=0x7f67d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67d006b500 0x7f67d006d9c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f67f00ccf50 tx=0x7f67dc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 msgr2=0x7f67f00cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00cb7a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f67e000e870 tx=0x7f67e000ed30 comp rx=0 tx=0).stop 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 shutdown_connections 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67d006b500 0x7f67d006d9c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f015e0b0 0x7f67f006ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f00d4a10 0x7f67f014f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 --2- 172.21.15.32:0/1931016379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f00d0c80 0x7f67f00cb7a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.248+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 >> 172.21.15.32:0/1931016379 conn(0x7f67f00d3d30 msgr2=0x7f67f00b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.249+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 shutdown_connections 2024-03-06T07:33:06.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.249+0000 7f67efe00640 1 -- 172.21.15.32:0/1931016379 wait complete. 2024-03-06T07:33:06.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:28.663107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:29.814796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:06.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.495+0000 7f05af200640 1 Processor -- start 2024-03-06T07:33:06.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.495+0000 7f05af200640 1 -- start start 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.498+0000 7f05af200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f05b015e0c0 0x7f05b0072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.498+0000 7f05af200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 0x7f05b00cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05af200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00b5800 con 0x7f05b006afb0 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05af200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b006dd30 con 0x7f05b015e0c0 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05af200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00b5aa0 con 0x7f05b006b440 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05ad400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f05b006afb0 0x7f05b00cd600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52446/0 (socket says 172.21.15.32:52446) 2024-03-06T07:33:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05ad400640 1 -- 172.21.15.32:0/3389775204 learned_addr learned my addr 172.21.15.32:0/3389775204 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05ade00640 1 --2- 172.21.15.32:0/3389775204 >> v2:172.21.15.39:3300/0 conn(0x7f05b015e0c0 0x7f05b0072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.499+0000 7f05ae800640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 0x7f05b00cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031194379 0 0) 0x7f05b00b5800 con 0x7f05b006afb0 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0594003520 con 0x7f05b006afb0 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2433783903 0 0) 0x7f0594003520 con 0x7f05b006afb0 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 msgr2=0x7f05b00cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 0x7f05b00cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 >> v2:172.21.15.39:3300/0 conn(0x7f05b015e0c0 msgr2=0x7f05b0072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 --2- 172.21.15.32:0/3389775204 >> v2:172.21.15.39:3300/0 conn(0x7f05b015e0c0 0x7f05b0072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00b5d80 con 0x7f05b006afb0 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f059c001670 con 0x7f05b006afb0 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05ade00640 1 --2- 172.21.15.32:0/3389775204 >> v2:172.21.15.39:3300/0 conn(0x7f05b015e0c0 0x7f05b0072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 >> v1:172.21.15.32:6789/0 conn(0x7f05b006afb0 legacy=0x7f05b00cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05940039b0 0x7f0594003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0594004800 0x7f0594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 0x7f05940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00b5aa0 con 0x7f05940039b0 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059400b9c0 con 0x7f0594004800 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.500+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059400bb40 con 0x7f05b006b440 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ade00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05940039b0 0x7f0594003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ad400640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0594004800 0x7f0594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ae800640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 0x7f05940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ade00640 1 -- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 msgr2=0x7f05940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ade00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05b006b440 0x7f05940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ade00640 1 -- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0594004800 msgr2=0x7f0594004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ade00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0594004800 0x7f0594004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.501+0000 7f05ade00640 1 -- 172.21.15.32:0/3389775204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594003520 con 0x7f05940039b0 2024-03-06T07:33:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.503+0000 7f05ade00640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05940039b0 0x7f0594003dd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009c90 tx=0x7f0594005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280f680376ae87e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.503+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0039070 con 0x7f05940039b0 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a002c980 con 0x7f05940039b0 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05aca00640 1 -- 172.21.15.32:0/3389775204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0009860 con 0x7f05940039b0 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05af200640 1 -- 172.21.15.32:0/3389775204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05940039b0 msgr2=0x7f0594003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05af200640 1 --2- 172.21.15.32:0/3389775204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05940039b0 0x7f0594003dd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009c90 tx=0x7f0594005830 comp rx=0 tx=0).stop 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05ae800640 1 -- 172.21.15.32:0/3389775204 reap_dead start 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05af200640 1 -- 172.21.15.32:0/3389775204 shutdown_connections 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05af200640 1 -- 172.21.15.32:0/3389775204 >> 172.21.15.32:0/3389775204 conn(0x7f05b00d0c00 msgr2=0x7f05b00d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05af200640 1 -- 172.21.15.32:0/3389775204 shutdown_connections 2024-03-06T07:33:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.504+0000 7f05af200640 1 -- 172.21.15.32:0/3389775204 wait complete. 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.506+0000 7f05af200640 1 Processor -- start 2024-03-06T07:33:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.506+0000 7f05af200640 1 -- start start 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05af200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05940039b0 0x7f05b0147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05af200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05b01606c0 0x7f05b0160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05af200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 0x7f05b015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05af200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b006b890 con 0x7f05b015bc10 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05af200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00cef60 con 0x7f05b01606c0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05af200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00727b0 con 0x7f05940039b0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.507+0000 7f05ae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 0x7f05b015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 0x7f05b015c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34728/0 (socket says 172.21.15.32:34728) 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 -- 172.21.15.32:0/1880113219 learned_addr learned my addr 172.21.15.32:0/1880113219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ad400640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05b01606c0 0x7f05b0160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ade00640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05940039b0 0x7f05b0147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 -- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05940039b0 msgr2=0x7f05b0147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05940039b0 0x7f05b0147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 -- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05b01606c0 msgr2=0x7f05b0160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05b01606c0 0x7f05b0160b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 -- 172.21.15.32:0/1880113219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b01481a0 con 0x7f05b015bc10 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.508+0000 7f05ae800640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 0x7f05b015c090 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f05a8007d10 tx=0x7f05a80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.509+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8011de0 con 0x7f05b015bc10 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.509+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a801b070 con 0x7f05b015bc10 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.509+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8003990 con 0x7f05b015bc10 2024-03-06T07:33:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.509+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b00add50 con 0x7f05b015bc10 2024-03-06T07:33:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.509+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b00ae310 con 0x7f05b015bc10 2024-03-06T07:33:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.510+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b004f330 con 0x7f05b015bc10 2024-03-06T07:33:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.511+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05a8003b30 con 0x7f05b015bc10 2024-03-06T07:33:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.512+0000 7f05a6000640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f059006b4d0 0x7f059006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.512+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f05a808aad0 con 0x7f05b015bc10 2024-03-06T07:33:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.512+0000 7f05ade00640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f059006b4d0 0x7f059006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.513+0000 7f05ade00640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f059006b4d0 0x7f059006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f05a0036000 tx=0x7f05a0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.521+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05a8003e20 con 0x7f05b015bc10 2024-03-06T07:33:06.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.723+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05b006bc40 con 0x7f05b015bc10 2024-03-06T07:33:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.724+0000 7f05a6000640 1 -- 172.21.15.32:0/1880113219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05a8059870 con 0x7f05b015bc10 2024-03-06T07:33:06.725 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f059006b4d0 msgr2=0x7f059006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f059006b4d0 0x7f059006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f05a0036000 tx=0x7f05a0030000 comp rx=0 tx=0).stop 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 msgr2=0x7f05b015c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 0x7f05b015c090 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f05a8007d10 tx=0x7f05a80115a0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 shutdown_connections 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f059006b4d0 0x7f059006d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.726+0000 7f05af200640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05b015bc10 0x7f05b015c090 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.727+0000 7f05af200640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05b01606c0 0x7f05b0160b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.727+0000 7f05af200640 1 --2- 172.21.15.32:0/1880113219 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05940039b0 0x7f05b0147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.727+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 >> 172.21.15.32:0/1880113219 conn(0x7f05b00d0c00 msgr2=0x7f05b00cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.727+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 shutdown_connections 2024-03-06T07:33:06.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.727+0000 7f05af200640 1 -- 172.21.15.32:0/1880113219 wait complete. 2024-03-06T07:33:06.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.965+0000 7f8aa5a00640 1 Processor -- start 2024-03-06T07:33:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.965+0000 7f8aa5a00640 1 -- start start 2024-03-06T07:33:06.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.967+0000 7f8aa5a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa00afca0 0x7f8aa00cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.967+0000 7f8aa5a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa00b4020 con 0x7f8aa00ceb00 2024-03-06T07:33:06.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.967+0000 7f8aa5a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa00b42c0 con 0x7f8aa00cdd60 2024-03-06T07:33:06.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.967+0000 7f8aa5a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00cff00 con 0x7f8aa00afca0 2024-03-06T07:33:06.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.967+0000 7f8a9f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8aa00ceb00 0x7f8aa00cd650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52460/0 (socket says 172.21.15.32:52460) 2024-03-06T07:33:06.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.967+0000 7f8a9f400640 1 -- 172.21.15.32:0/186893858 learned_addr learned my addr 172.21.15.32:0/186893858 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:06.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa00afca0 0x7f8aa00cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268650396 0 0) 0x7f8aa00b4020 con 0x7f8aa00ceb00 2024-03-06T07:33:06.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a84003520 con 0x7f8aa00ceb00 2024-03-06T07:33:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279595283 0 0) 0x7f8a84003520 con 0x7f8aa00ceb00 2024-03-06T07:33:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa00afca0 msgr2=0x7f8aa00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa00afca0 0x7f8aa00cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 >> v1:172.21.15.39:6789/0 conn(0x7f8aa00cdd60 legacy=0x7f8aa0072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa00b45a0 con 0x7f8aa00ceb00 2024-03-06T07:33:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f8a8c001670 con 0x7f8aa00ceb00 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.968+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 >> v1:172.21.15.32:6789/0 conn(0x7f8aa00ceb00 legacy=0x7f8aa00cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9ea00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a840039e0 0x7f8a84003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9ea00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a84004800 0x7f8a84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9ea00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a840051c0 0x7f8a8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00cff00 con 0x7f8a840039e0 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8400bfe0 con 0x7f8a84004800 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8400c1e0 con 0x7f8a840051c0 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa00afca0 0x7f8aa00cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a840039e0 0x7f8a84003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9f400640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a84004800 0x7f8a84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8aa5000640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a840051c0 0x7f8a8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.969+0000 7f8a9fe00640 1 -- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a840051c0 msgr2=0x7f8a8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.970+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a840051c0 0x7f8a8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.970+0000 7f8a9fe00640 1 -- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a84004800 msgr2=0x7f8a84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.970+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a84004800 0x7f8a84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.970+0000 7f8a9fe00640 1 -- 172.21.15.32:0/186893858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a84003520 con 0x7f8a840039e0 2024-03-06T07:33:06.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.971+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a840039e0 0x7f8a84003e00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8aa006d0d0 tx=0x7f8a84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5485f0fcfccd2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.972+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a900097d0 con 0x7f8a840039e0 2024-03-06T07:33:06.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.972+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a90004030 con 0x7f8a840039e0 2024-03-06T07:33:06.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.972+0000 7f8a9ea00640 1 -- 172.21.15.32:0/186893858 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90004400 con 0x7f8a840039e0 2024-03-06T07:33:06.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.972+0000 7f8aa5a00640 1 -- 172.21.15.32:0/186893858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a840039e0 msgr2=0x7f8a84003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.972+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/186893858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a840039e0 0x7f8a84003e00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8aa006d0d0 tx=0x7f8a84005830 comp rx=0 tx=0).stop 2024-03-06T07:33:06.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.973+0000 7f8aa5000640 1 -- 172.21.15.32:0/186893858 reap_dead start 2024-03-06T07:33:06.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.973+0000 7f8aa5a00640 1 -- 172.21.15.32:0/186893858 shutdown_connections 2024-03-06T07:33:06.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.973+0000 7f8aa5a00640 1 -- 172.21.15.32:0/186893858 >> 172.21.15.32:0/186893858 conn(0x7f8aa00d0c00 msgr2=0x7f8aa00d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.973+0000 7f8aa5a00640 1 -- 172.21.15.32:0/186893858 shutdown_connections 2024-03-06T07:33:06.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.973+0000 7f8aa5a00640 1 -- 172.21.15.32:0/186893858 wait complete. 2024-03-06T07:33:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.974+0000 7f8aa5a00640 1 Processor -- start 2024-03-06T07:33:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.975+0000 7f8aa5a00640 1 -- start start 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.975+0000 7f8aa5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a840039e0 0x7f8aa015c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa006eff0 0x7f8aa0159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 0x7f8aa015d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00b4b60 con 0x7f8aa006f470 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00b06d0 con 0x7f8a840039e0 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00b0080 con 0x7f8aa006eff0 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 0x7f8aa015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 0x7f8aa015d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34754/0 (socket says 172.21.15.32:34754) 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 -- 172.21.15.32:0/2173975397 learned_addr learned my addr 172.21.15.32:0/2173975397 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8a9f400640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa006eff0 0x7f8aa0159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a840039e0 0x7f8aa015c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 -- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa006eff0 msgr2=0x7f8aa0159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa006eff0 0x7f8aa0159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 -- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a840039e0 msgr2=0x7f8aa015c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a840039e0 0x7f8aa015c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.976+0000 7f8aa5000640 1 -- 172.21.15.32:0/2173975397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa015a310 con 0x7f8aa006f470 2024-03-06T07:33:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.977+0000 7f8aa5000640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 0x7f8aa015d130 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8a98000f80 tx=0x7f8a9800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.977+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98003970 con 0x7f8aa006f470 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.977+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a98018070 con 0x7f8aa006f470 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.977+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa015a5f0 con 0x7f8aa006f470 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.977+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98003d70 con 0x7f8aa006f470 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.977+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa015bae0 con 0x7f8aa006f470 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.978+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa004f330 con 0x7f8aa006f470 2024-03-06T07:33:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.979+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a980050d0 con 0x7f8aa006f470 2024-03-06T07:33:06.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.981+0000 7f8a97e00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a8006b4d0 0x7f8a8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.981+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a98089fa0 con 0x7f8aa006f470 2024-03-06T07:33:06.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.981+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a8006b4d0 0x7f8a8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.982+0000 7f8a9fe00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a8006b4d0 0x7f8a8006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8a90035000 tx=0x7f8a90030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:06.985+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a98058d40 con 0x7f8aa006f470 2024-03-06T07:33:07.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.187+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8aa00d1490 con 0x7f8aa006f470 2024-03-06T07:33:07.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.188+0000 7f8a97e00640 1 -- 172.21.15.32:0/2173975397 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a98058640 con 0x7f8aa006f470 2024-03-06T07:33:07.191 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.190+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a8006b4d0 msgr2=0x7f8a8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.190+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a8006b4d0 0x7f8a8006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8a90035000 tx=0x7f8a90030000 comp rx=0 tx=0).stop 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.190+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 msgr2=0x7f8aa015d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.190+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 0x7f8aa015d130 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8a98000f80 tx=0x7f8a9800ec10 comp rx=0 tx=0).stop 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 shutdown_connections 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a8006b4d0 0x7f8a8006d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8aa006f470 0x7f8aa015d130 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8aa006eff0 0x7f8aa0159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 --2- 172.21.15.32:0/2173975397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a840039e0 0x7f8aa015c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 >> 172.21.15.32:0/2173975397 conn(0x7f8aa00d0c00 msgr2=0x7f8aa015e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 shutdown_connections 2024-03-06T07:33:07.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.191+0000 7f8aa5a00640 1 -- 172.21.15.32:0/2173975397 wait complete. 2024-03-06T07:33:07.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.443+0000 7fbcbea00640 1 Processor -- start 2024-03-06T07:33:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.444+0000 7fbcbea00640 1 -- start start 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb80acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcb8071e20 0x7fbcb80ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbea00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb8072df0 con 0x7fbcb8072520 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb80cdab0 con 0x7fbcb8071e20 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80b4d70 con 0x7fbcb815bf20 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcb8071e20 0x7fbcb80ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcb8071e20 0x7fbcb80ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34768/0 (socket says 172.21.15.32:34768) 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcb7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbcb8072520 0x7fbcb80cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52476/0 (socket says 172.21.15.32:52476) 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.446+0000 7fbcbca00640 1 -- 172.21.15.32:0/899363116 learned_addr learned my addr 172.21.15.32:0/899363116 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.447+0000 7fbcb7e00640 1 --2- 172.21.15.32:0/899363116 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb80acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.447+0000 7fbcbca00640 1 -- 172.21.15.32:0/899363116 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 msgr2=0x7fbcb80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.447+0000 7fbcbca00640 1 --2- 172.21.15.32:0/899363116 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb80acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.447+0000 7fbcbca00640 1 -- 172.21.15.32:0/899363116 >> v1:172.21.15.32:6789/0 conn(0x7fbcb8072520 legacy=0x7fbcb80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.447+0000 7fbcbca00640 1 -- 172.21.15.32:0/899363116 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb80b5050 con 0x7fbcb8071e20 2024-03-06T07:33:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.448+0000 7fbcbca00640 1 --2- 172.21.15.32:0/899363116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb80ccad0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009870 tx=0x7fbcb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cbd813e07ebde1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcb6a00640 1 -- 172.21.15.32:0/899363116 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0034070 con 0x7fbcb8071e20 2024-03-06T07:33:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcb6a00640 1 -- 172.21.15.32:0/899363116 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb0030840 con 0x7fbcb8071e20 2024-03-06T07:33:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcb6a00640 1 -- 172.21.15.32:0/899363116 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0030c10 con 0x7fbcb8071e20 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcbea00640 1 -- 172.21.15.32:0/899363116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 msgr2=0x7fbcb80ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcbea00640 1 --2- 172.21.15.32:0/899363116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb80ccad0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009870 tx=0x7fbcb002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcbea00640 1 -- 172.21.15.32:0/899363116 shutdown_connections 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcbea00640 1 --2- 172.21.15.32:0/899363116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb80ccad0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcbea00640 1 --2- 172.21.15.32:0/899363116 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb80acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.449+0000 7fbcbea00640 1 -- 172.21.15.32:0/899363116 >> 172.21.15.32:0/899363116 conn(0x7fbcb806b3d0 msgr2=0x7fbcb806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.450+0000 7fbcbea00640 1 -- 172.21.15.32:0/899363116 shutdown_connections 2024-03-06T07:33:07.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.450+0000 7fbcbea00640 1 -- 172.21.15.32:0/899363116 wait complete. 2024-03-06T07:33:07.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.451+0000 7fbcbea00640 1 Processor -- start 2024-03-06T07:33:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.452+0000 7fbcbea00640 1 -- start start 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb81e4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8072520 0x7fbcb809d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80b4b10 con 0x7fbcb8071e20 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80b4990 con 0x7fbcb8072520 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80ce6c0 con 0x7fbcb815bf20 2024-03-06T07:33:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb81e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb81e4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34774/0 (socket says 172.21.15.32:34774) 2024-03-06T07:33:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 -- 172.21.15.32:0/3036644778 learned_addr learned my addr 172.21.15.32:0/3036644778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcbca00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7400640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8072520 0x7fbcb809d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 -- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 msgr2=0x7fbcb806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb806ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 -- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8072520 msgr2=0x7fbcb809d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8072520 0x7fbcb809d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.453+0000 7fbcb7e00640 1 -- 172.21.15.32:0/3036644778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb81f5500 con 0x7fbcb8071e20 2024-03-06T07:33:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.454+0000 7fbcb7e00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb81e4650 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbca800efc0 tx=0x7fbca800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.455+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8018020 con 0x7fbcb8071e20 2024-03-06T07:33:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.455+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8014480 con 0x7fbcb8071e20 2024-03-06T07:33:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.455+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb809d730 con 0x7fbcb8071e20 2024-03-06T07:33:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.455+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8014850 con 0x7fbcb8071e20 2024-03-06T07:33:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.456+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb81f5220 con 0x7fbcb8071e20 2024-03-06T07:33:07.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.457+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb804f330 con 0x7fbcb8071e20 2024-03-06T07:33:07.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.458+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbca8005d10 con 0x7fbcb8071e20 2024-03-06T07:33:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.458+0000 7fbcafe00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbc9806b4d0 0x7fbc9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.459+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbca808b140 con 0x7fbcb8071e20 2024-03-06T07:33:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.459+0000 7fbcb7400640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbc9806b4d0 0x7fbc9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.460+0000 7fbcb7400640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbc9806b4d0 0x7fbc9806d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8155230 tx=0x7fbca4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.463+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbca8022030 con 0x7fbcb8071e20 2024-03-06T07:33:07.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.664+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbcb815b140 con 0x7fbcb8071e20 2024-03-06T07:33:07.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.664+0000 7fbcafe00640 1 -- 172.21.15.32:0/3036644778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbca8010040 con 0x7fbcb8071e20 2024-03-06T07:33:07.667 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:33:07.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbc9806b4d0 msgr2=0x7fbc9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbc9806b4d0 0x7fbc9806d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8155230 tx=0x7fbca4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 msgr2=0x7fbcb81e4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb81e4650 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbca800efc0 tx=0x7fbca800c490 comp rx=0 tx=0).stop 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 shutdown_connections 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbc9806b4d0 0x7fbc9806d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbcb815bf20 0x7fbcb806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8072520 0x7fbcb809d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 --2- 172.21.15.32:0/3036644778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcb8071e20 0x7fbcb81e4650 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 >> 172.21.15.32:0/3036644778 conn(0x7fbcb806b3d0 msgr2=0x7fbcb80b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 shutdown_connections 2024-03-06T07:33:07.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.667+0000 7fbcbea00640 1 -- 172.21.15.32:0/3036644778 wait complete. 2024-03-06T07:33:07.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:33:07.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.959+0000 7fed6b200640 1 Processor -- start 2024-03-06T07:33:07.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.960+0000 7fed6b200640 1 -- start start 2024-03-06T07:33:07.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.961+0000 7fed6b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed6c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.961+0000 7fed6b200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c0b3710 con 0x7fed6c0d4a10 2024-03-06T07:33:07.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.961+0000 7fed6b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c0b39f0 con 0x7fed6c0d0e50 2024-03-06T07:33:07.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.961+0000 7fed6b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0b09e0 con 0x7fed6c15e0b0 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.962+0000 7fed63e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fed6c0d4a10 0x7fed6c0d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60194/0 (socket says 172.21.15.32:60194) 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.962+0000 7fed63e00640 1 -- 172.21.15.32:0/293369900 learned_addr learned my addr 172.21.15.32:0/293369900 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.962+0000 7fed69e00640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed6c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.962+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112992685 0 0) 0x7fed6c0b3710 con 0x7fed6c0d4a10 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.962+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed50003520 con 0x7fed6c0d4a10 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2929234403 0 0) 0x7fed50003520 con 0x7fed6c0d4a10 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 msgr2=0x7fed6c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed6c0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 >> v1:172.21.15.39:6789/0 conn(0x7fed6c0d0e50 legacy=0x7fed6c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c0b3cd0 con 0x7fed6c0d4a10 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fed54001670 con 0x7fed6c0d4a10 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69e00640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed6c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 >> v1:172.21.15.32:6789/0 conn(0x7fed6c0d4a10 legacy=0x7fed6c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed500039e0 0x7fed50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed50004d40 0x7fed500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5000ba70 con 0x7fed50004d40 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5000bbf0 con 0x7fed500039e0 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.963+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5000bdf0 con 0x7fed6c15e0b0 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed6a800640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed50004d40 0x7fed500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed69e00640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed500039e0 0x7fed50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed63e00640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed6a800640 1 -- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 msgr2=0x7fed50004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed6a800640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed50004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed6a800640 1 -- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed500039e0 msgr2=0x7fed50003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed6a800640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed500039e0 0x7fed50003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed6a800640 1 -- 172.21.15.32:0/293369900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed50003520 con 0x7fed50004d40 2024-03-06T07:33:07.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.964+0000 7fed63e00640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c15e0b0 0x7fed50004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:07.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.966+0000 7fed6a800640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed50004d40 0x7fed500051c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fed64001e20 tx=0x7fed50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2eb9115c359fdac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64001a70 con 0x7fed50004d40 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed64003b70 con 0x7fed50004d40 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed69400640 1 -- 172.21.15.32:0/293369900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed640050d0 con 0x7fed50004d40 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6b200640 1 -- 172.21.15.32:0/293369900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed50004d40 msgr2=0x7fed500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6b200640 1 --2- 172.21.15.32:0/293369900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed50004d40 0x7fed500051c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fed64001e20 tx=0x7fed50005830 comp rx=0 tx=0).stop 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6a800640 1 -- 172.21.15.32:0/293369900 reap_dead start 2024-03-06T07:33:07.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6b200640 1 -- 172.21.15.32:0/293369900 shutdown_connections 2024-03-06T07:33:07.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6b200640 1 -- 172.21.15.32:0/293369900 >> 172.21.15.32:0/293369900 conn(0x7fed6c0d3d30 msgr2=0x7fed6c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6b200640 1 -- 172.21.15.32:0/293369900 shutdown_connections 2024-03-06T07:33:07.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.967+0000 7fed6b200640 1 -- 172.21.15.32:0/293369900 wait complete. 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.969+0000 7fed6b200640 1 Processor -- start 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.969+0000 7fed6b200640 1 -- start start 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.970+0000 7fed6b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed50004d40 0x7fed6c0cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.970+0000 7fed6b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c158610 0x7fed6c14c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.970+0000 7fed6b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 0x7fed6c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.970+0000 7fed6b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0cfc60 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.970+0000 7fed6b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c15e490 con 0x7fed50004d40 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.970+0000 7fed6b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0cf310 con 0x7fed6c158610 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 0x7fed6c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 0x7fed6c0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42012/0 (socket says 172.21.15.32:42012) 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 -- 172.21.15.32:0/2220064446 learned_addr learned my addr 172.21.15.32:0/2220064446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed63e00640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c158610 0x7fed6c14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed69e00640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed50004d40 0x7fed6c0cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 -- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c158610 msgr2=0x7fed6c14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c158610 0x7fed6c14c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 -- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed50004d40 msgr2=0x7fed6c0cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed50004d40 0x7fed6c0cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed6a800640 1 -- 172.21.15.32:0/2220064446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c159620 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.971+0000 7fed69e00640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed50004d40 0x7fed6c0cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.972+0000 7fed6a800640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 0x7fed6c0cd480 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed64001f40 tx=0x7fed640014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.972+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64007db0 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.972+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed64003970 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.972+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64006d70 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.972+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c158d80 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.972+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c159340 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.974+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c04f330 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.976+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed64041030 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.976+0000 7fed62000640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fed4806b500 0x7fed4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.977+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed640ab560 con 0x7fed6c0ccfd0 2024-03-06T07:33:07.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.977+0000 7fed69e00640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fed4806b500 0x7fed4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.978+0000 7fed69e00640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fed4806b500 0x7fed4806d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009a30 tx=0x7fed5c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:07.982+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed64033020 con 0x7fed6c0ccfd0 2024-03-06T07:33:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.196+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fed6c0b11d0 con 0x7fed6c0ccfd0 2024-03-06T07:33:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.197+0000 7fed62000640 1 -- 172.21.15.32:0/2220064446 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) v1 ==== 80+0+0 (secure 0 0 0) 0x7fed6407a3a0 con 0x7fed6c0ccfd0 2024-03-06T07:33:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:33:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.199+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fed4806b500 msgr2=0x7fed4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.199+0000 7fed6b200640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fed4806b500 0x7fed4806d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009a30 tx=0x7fed5c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.199+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 msgr2=0x7fed6c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.199+0000 7fed6b200640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 0x7fed6c0cd480 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed64001f40 tx=0x7fed640014a0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 shutdown_connections 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fed4806b500 0x7fed4806d9c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed6c0ccfd0 0x7fed6c0cd480 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fed6c158610 0x7fed6c14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 --2- 172.21.15.32:0/2220064446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed50004d40 0x7fed6c0cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 >> 172.21.15.32:0/2220064446 conn(0x7fed6c0d3d30 msgr2=0x7fed6c0b13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 shutdown_connections 2024-03-06T07:33:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.200+0000 7fed6b200640 1 -- 172.21.15.32:0/2220064446 wait complete. 2024-03-06T07:33:08.221 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:33:08.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:08.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:08.339 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:33:08.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:08.339 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:33:08.398 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:33:08.398 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:08.439 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:08.439 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:08.439 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:08.478 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:08.478 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:33:08.478 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:08.525 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:08.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:08.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.721+0000 7fcd98e00640 1 Processor -- start 2024-03-06T07:33:08.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.721+0000 7fcd98e00640 1 -- start start 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd98e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd940cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd98e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcd940d0e50 0x7fcd940b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd98e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940b3710 con 0x7fcd940d4a10 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd98e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd940b09e0 con 0x7fcd940d0e50 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd98e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940b39f0 con 0x7fcd9415e0b0 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd92a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcd940d4a10 0x7fcd940d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60206/0 (socket says 172.21.15.32:60206) 2024-03-06T07:33:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd92a00640 1 -- 172.21.15.32:0/668903565 learned_addr learned my addr 172.21.15.32:0/668903565 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:08.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd93400640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd940cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.723+0000 7fcd93e00640 1 --2- 172.21.15.32:0/668903565 >> v2:172.21.15.39:3300/0 conn(0x7fcd940d0e50 0x7fcd940b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117975457 0 0) 0x7fcd940b3710 con 0x7fcd940d4a10 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd78003520 con 0x7fcd940d4a10 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1533720538 0 0) 0x7fcd78003520 con 0x7fcd940d4a10 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 msgr2=0x7fcd940cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd940cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 >> v2:172.21.15.39:3300/0 conn(0x7fcd940d0e50 msgr2=0x7fcd940b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 --2- 172.21.15.32:0/668903565 >> v2:172.21.15.39:3300/0 conn(0x7fcd940d0e50 0x7fcd940b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940b3cd0 con 0x7fcd940d4a10 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fcd80001670 con 0x7fcd940d4a10 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 >> v1:172.21.15.32:6789/0 conn(0x7fcd940d4a10 legacy=0x7fcd940d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd780039b0 0x7fcd78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd78004800 0x7fcd78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940b39f0 con 0x7fcd78004800 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7800b9c0 con 0x7fcd780039b0 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.724+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7800bb40 con 0x7fcd9415e0b0 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd92a00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd78004800 0x7fcd78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd93e00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd92a00640 1 -- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 msgr2=0x7fcd780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd92a00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd92a00640 1 -- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd780039b0 msgr2=0x7fcd78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd92a00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd780039b0 0x7fcd78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd92a00640 1 -- 172.21.15.32:0/668903565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd78003520 con 0x7fcd78004800 2024-03-06T07:33:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.725+0000 7fcd93e00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd9415e0b0 0x7fcd780051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:08.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.726+0000 7fcd92a00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd78004800 0x7fcd78004c80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fcd80007fd0 tx=0x7fcd78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def766e18d2cd887 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.726+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80007d50 con 0x7fcd78004800 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.726+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd80030070 con 0x7fcd78004800 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd92000640 1 -- 172.21.15.32:0/668903565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8002cd60 con 0x7fcd78004800 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd98e00640 1 -- 172.21.15.32:0/668903565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd78004800 msgr2=0x7fcd78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd98e00640 1 --2- 172.21.15.32:0/668903565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd78004800 0x7fcd78004c80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fcd80007fd0 tx=0x7fcd78005830 comp rx=0 tx=0).stop 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd93e00640 1 -- 172.21.15.32:0/668903565 reap_dead start 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd98e00640 1 -- 172.21.15.32:0/668903565 shutdown_connections 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd98e00640 1 -- 172.21.15.32:0/668903565 >> 172.21.15.32:0/668903565 conn(0x7fcd940d3d30 msgr2=0x7fcd940d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd98e00640 1 -- 172.21.15.32:0/668903565 shutdown_connections 2024-03-06T07:33:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.727+0000 7fcd98e00640 1 -- 172.21.15.32:0/668903565 wait complete. 2024-03-06T07:33:08.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.730+0000 7fcd98e00640 1 Processor -- start 2024-03-06T07:33:08.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.730+0000 7fcd98e00640 1 -- start start 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd98e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd78004800 0x7fcd940b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd98e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940b6920 0x7fcd9415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd98e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 0x7fcd940ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd98e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cfc60 con 0x7fcd940ca060 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd98e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9415e490 con 0x7fcd940b6920 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd98e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940cf310 con 0x7fcd78004800 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 0x7fcd940ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 0x7fcd940ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42036/0 (socket says 172.21.15.32:42036) 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.732+0000 7fcd93e00640 1 -- 172.21.15.32:0/4212203380 learned_addr learned my addr 172.21.15.32:0/4212203380 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93400640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd78004800 0x7fcd940b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd92a00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940b6920 0x7fcd9415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93e00640 1 -- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd78004800 msgr2=0x7fcd940b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd78004800 0x7fcd940b8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93e00640 1 -- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940b6920 msgr2=0x7fcd9415c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940b6920 0x7fcd9415c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93e00640 1 -- 172.21.15.32:0/4212203380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd94160f40 con 0x7fcd940ca060 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd93e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 0x7fcd940ca4e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcd9408e2e0 tx=0x7fcd8c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.733+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c01c070 con 0x7fcd940ca060 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.734+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c017400 con 0x7fcd940ca060 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.734+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0177d0 con 0x7fcd940ca060 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.734+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd940b6ee0 con 0x7fcd940ca060 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.734+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940b74a0 con 0x7fcd940ca060 2024-03-06T07:33:08.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.735+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404f330 con 0x7fcd940ca060 2024-03-06T07:33:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.736+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd8c005d10 con 0x7fcd940ca060 2024-03-06T07:33:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.737+0000 7fcd8b400640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b500 0x7fcd7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.737+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcd8c08a370 con 0x7fcd940ca060 2024-03-06T07:33:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.738+0000 7fcd93400640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b500 0x7fcd7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.738+0000 7fcd93400640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b500 0x7fcd7406d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcd840047b0 tx=0x7fcd84008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.742+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd8c012070 con 0x7fcd940ca060 2024-03-06T07:33:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.985+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9415d520 con 0x7fcd940ca060 2024-03-06T07:33:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.986+0000 7fcd8b400640 1 -- 172.21.15.32:0/4212203380 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fcd8c0591b0 con 0x7fcd940ca060 2024-03-06T07:33:08.988 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:08.989 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:33:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.988+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b500 msgr2=0x7fcd7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.988+0000 7fcd98e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b500 0x7fcd7406d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcd840047b0 tx=0x7fcd84008040 comp rx=0 tx=0).stop 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 msgr2=0x7fcd940ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 0x7fcd940ca4e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcd9408e2e0 tx=0x7fcd8c00f4a0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 shutdown_connections 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b500 0x7fcd7406d9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd940ca060 0x7fcd940ca4e0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940b6920 0x7fcd9415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 --2- 172.21.15.32:0/4212203380 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd78004800 0x7fcd940b8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 >> 172.21.15.32:0/4212203380 conn(0x7fcd940d3d30 msgr2=0x7fcd940ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 shutdown_connections 2024-03-06T07:33:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:08.989+0000 7fcd98e00640 1 -- 172.21.15.32:0/4212203380 wait complete. 2024-03-06T07:33:09.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.220+0000 7f9353200640 1 Processor -- start 2024-03-06T07:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.220+0000 7f9353200640 1 -- start start 2024-03-06T07:33:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f9353200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f934c0b3c90 0x7f934c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f9353200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f934c071050 0x7f934c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f9353200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c0b4860 con 0x7f934c0cf5c0 2024-03-06T07:33:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f9353200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c0ce160 con 0x7f934c071050 2024-03-06T07:33:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f9353200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c0b4b40 con 0x7f934c0b3c90 2024-03-06T07:33:09.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f934b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f934c0cf5c0 0x7f934c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60212/0 (socket says 172.21.15.32:60212) 2024-03-06T07:33:09.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f934b400640 1 -- 172.21.15.32:0/3183356417 learned_addr learned my addr 172.21.15.32:0/3183356417 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:09.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f9351200640 1 --2- 172.21.15.32:0/3183356417 >> v2:172.21.15.32:3300/0 conn(0x7f934c071050 0x7f934c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.222+0000 7f934be00640 1 --2- 172.21.15.32:0/3183356417 >> v2:172.21.15.39:3300/0 conn(0x7f934c0b3c90 0x7f934c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.223+0000 7f9351200640 1 -- 172.21.15.32:0/3183356417 >> v2:172.21.15.39:3300/0 conn(0x7f934c0b3c90 msgr2=0x7f934c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.223+0000 7f9351200640 1 --2- 172.21.15.32:0/3183356417 >> v2:172.21.15.39:3300/0 conn(0x7f934c0b3c90 0x7f934c0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.223+0000 7f9351200640 1 -- 172.21.15.32:0/3183356417 >> v1:172.21.15.32:6789/0 conn(0x7f934c0cf5c0 legacy=0x7f934c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.223+0000 7f934aa00640 1 -- 172.21.15.32:0/3183356417 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270000395 0 0) 0x7f934c0b4860 con 0x7f934c0cf5c0 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.223+0000 7f9351200640 1 -- 172.21.15.32:0/3183356417 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c0b4e20 con 0x7f934c071050 2024-03-06T07:33:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.225+0000 7f9351200640 1 --2- 172.21.15.32:0/3183356417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c071050 0x7f934c071490 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f934c070800 tx=0x7f934402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146733ca3555ee9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.225+0000 7f934aa00640 1 -- 172.21.15.32:0/3183356417 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9344034070 con 0x7f934c071050 2024-03-06T07:33:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.225+0000 7f934aa00640 1 -- 172.21.15.32:0/3183356417 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344030610 con 0x7f934c071050 2024-03-06T07:33:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.225+0000 7f934aa00640 1 -- 172.21.15.32:0/3183356417 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440309e0 con 0x7f934c071050 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 -- 172.21.15.32:0/3183356417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c071050 msgr2=0x7f934c071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 --2- 172.21.15.32:0/3183356417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c071050 0x7f934c071490 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f934c070800 tx=0x7f934402b960 comp rx=0 tx=0).stop 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 -- 172.21.15.32:0/3183356417 shutdown_connections 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 --2- 172.21.15.32:0/3183356417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c071050 0x7f934c071490 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 --2- 172.21.15.32:0/3183356417 >> v2:172.21.15.39:3300/0 conn(0x7f934c0b3c90 0x7f934c0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 -- 172.21.15.32:0/3183356417 >> 172.21.15.32:0/3183356417 conn(0x7f934c06afb0 msgr2=0x7f934c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 -- 172.21.15.32:0/3183356417 shutdown_connections 2024-03-06T07:33:09.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.226+0000 7f9353200640 1 -- 172.21.15.32:0/3183356417 wait complete. 2024-03-06T07:33:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.229+0000 7f9353200640 1 Processor -- start 2024-03-06T07:33:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.229+0000 7f9353200640 1 -- start start 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.230+0000 7f9353200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f934c071050 0x7f934c1445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.230+0000 7f9353200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 0x7f934c143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.230+0000 7f9353200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934c0cf5c0 0x7f934c09cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.230+0000 7f9353200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0719d0 con 0x7f934c0b3c90 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.230+0000 7f9353200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c15b340 con 0x7f934c0cf5c0 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.230+0000 7f9353200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c071b50 con 0x7f934c071050 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 0x7f934c143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 0x7f934c143da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42058/0 (socket says 172.21.15.32:42058) 2024-03-06T07:33:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 -- 172.21.15.32:0/3998819829 learned_addr learned my addr 172.21.15.32:0/3998819829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934be00640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f934c071050 0x7f934c1445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f9351200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934c0cf5c0 0x7f934c09cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 -- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f934c071050 msgr2=0x7f934c1445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f934c071050 0x7f934c1445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 -- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934c0cf5c0 msgr2=0x7f934c09cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934c0cf5c0 0x7f934c09cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934b400640 1 -- 172.21.15.32:0/3998819829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c1442e0 con 0x7f934c0b3c90 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.231+0000 7f934be00640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f934c071050 0x7f934c1445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.232+0000 7f934b400640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 0x7f934c143da0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f933400c7f0 tx=0x7f933400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.232+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334018070 con 0x7f934c0b3c90 2024-03-06T07:33:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.232+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933400a640 con 0x7f934c0b3c90 2024-03-06T07:33:09.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.232+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933400aa30 con 0x7f934c0b3c90 2024-03-06T07:33:09.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.232+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c151740 con 0x7f934c0b3c90 2024-03-06T07:33:09.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.232+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c151d00 con 0x7f934c0b3c90 2024-03-06T07:33:09.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.233+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c069510 con 0x7f934c0b3c90 2024-03-06T07:33:09.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.235+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f933401b070 con 0x7f934c0b3c90 2024-03-06T07:33:09.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.236+0000 7f9343e00640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.236+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9334089870 con 0x7f934c0b3c90 2024-03-06T07:33:09.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.236+0000 7f934be00640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.237+0000 7f934be00640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f932806b4d0 0x7f932806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f933c006fd0 tx=0x7f933c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.244+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93340586a0 con 0x7f934c0b3c90 2024-03-06T07:33:09.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.443+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f934c04f3a0 con 0x7f934c0b3c90 2024-03-06T07:33:09.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.445+0000 7f9343e00640 1 -- 172.21.15.32:0/3998819829 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f9334057fa0 con 0x7f934c0b3c90 2024-03-06T07:33:09.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:09.448 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":70,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:48.336272+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f932806b4d0 msgr2=0x7f932806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f932806b4d0 0x7f932806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f933c006fd0 tx=0x7f933c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 msgr2=0x7f934c143da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 0x7f934c143da0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f933400c7f0 tx=0x7f933400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 shutdown_connections 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f932806b4d0 0x7f932806d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934c0cf5c0 0x7f934c09cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934c0b3c90 0x7f934c143da0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 --2- 172.21.15.32:0/3998819829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f934c071050 0x7f934c1445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.447+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 >> 172.21.15.32:0/3998819829 conn(0x7f934c06afb0 msgr2=0x7f934c06d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.448+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 shutdown_connections 2024-03-06T07:33:09.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.448+0000 7f9353200640 1 -- 172.21.15.32:0/3998819829 wait complete. 2024-03-06T07:33:09.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:28.663107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:29.814796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:09.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:09.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.683+0000 7fd283200640 1 Processor -- start 2024-03-06T07:33:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.683+0000 7fd283200640 1 -- start start 2024-03-06T07:33:09.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd283200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd284161770 0x7fd2840d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd283200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2840b3200 0x7fd2840d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd283200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2840d3bf0 con 0x7fd2840b2cb0 2024-03-06T07:33:09.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd283200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2840d4900 con 0x7fd2840b3200 2024-03-06T07:33:09.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd283200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2840d19b0 con 0x7fd284161770 2024-03-06T07:33:09.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd279400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2840b3200 0x7fd2840d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd282800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd2840b2cb0 0x7fd2840d07b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60228/0 (socket says 172.21.15.32:60228) 2024-03-06T07:33:09.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.685+0000 7fd282800640 1 -- 172.21.15.32:0/2439386411 learned_addr learned my addr 172.21.15.32:0/2439386411 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:09.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.686+0000 7fd281e00640 1 --2- 172.21.15.32:0/2439386411 >> v2:172.21.15.39:3300/0 conn(0x7fd284161770 0x7fd2840d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.686+0000 7fd279400640 1 -- 172.21.15.32:0/2439386411 >> v2:172.21.15.39:3300/0 conn(0x7fd284161770 msgr2=0x7fd2840d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.686+0000 7fd279400640 1 --2- 172.21.15.32:0/2439386411 >> v2:172.21.15.39:3300/0 conn(0x7fd284161770 0x7fd2840d2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.686+0000 7fd279400640 1 -- 172.21.15.32:0/2439386411 >> v1:172.21.15.32:6789/0 conn(0x7fd2840b2cb0 legacy=0x7fd2840d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.686+0000 7fd279400640 1 -- 172.21.15.32:0/2439386411 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2840d4b50 con 0x7fd2840b3200 2024-03-06T07:33:09.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.686+0000 7fd281400640 1 -- 172.21.15.32:0/2439386411 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011937568 0 0) 0x7fd2840d3bf0 con 0x7fd2840b2cb0 2024-03-06T07:33:09.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.687+0000 7fd279400640 1 --2- 172.21.15.32:0/2439386411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840d25b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd28408d620 tx=0x7fd26c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa0abdd2a65cc8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.687+0000 7fd281400640 1 -- 172.21.15.32:0/2439386411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c035020 con 0x7fd2840b3200 2024-03-06T07:33:09.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.687+0000 7fd281400640 1 -- 172.21.15.32:0/2439386411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c031830 con 0x7fd2840b3200 2024-03-06T07:33:09.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.687+0000 7fd281400640 1 -- 172.21.15.32:0/2439386411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c031c00 con 0x7fd2840b3200 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 -- 172.21.15.32:0/2439386411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 msgr2=0x7fd2840d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 --2- 172.21.15.32:0/2439386411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840d25b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd28408d620 tx=0x7fd26c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 -- 172.21.15.32:0/2439386411 shutdown_connections 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 --2- 172.21.15.32:0/2439386411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840d25b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 --2- 172.21.15.32:0/2439386411 >> v2:172.21.15.39:3300/0 conn(0x7fd284161770 0x7fd2840d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 -- 172.21.15.32:0/2439386411 >> 172.21.15.32:0/2439386411 conn(0x7fd2840d3de0 msgr2=0x7fd2840d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 -- 172.21.15.32:0/2439386411 shutdown_connections 2024-03-06T07:33:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.688+0000 7fd283200640 1 -- 172.21.15.32:0/2439386411 wait complete. 2024-03-06T07:33:09.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.690+0000 7fd283200640 1 Processor -- start 2024-03-06T07:33:09.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.690+0000 7fd283200640 1 -- start start 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd283200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2840b2cb0 0x7fd2841501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd283200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd283200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd284161770 0x7fd2840ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd283200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd284161b50 con 0x7fd2840b3200 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd283200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840cf630 con 0x7fd284161770 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd283200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840cfac0 con 0x7fd2840b2cb0 2024-03-06T07:33:09.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd279400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd279400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42076/0 (socket says 172.21.15.32:42076) 2024-03-06T07:33:09.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd279400640 1 -- 172.21.15.32:0/3190704422 learned_addr learned my addr 172.21.15.32:0/3190704422 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd281e00640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2840b2cb0 0x7fd2841501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.691+0000 7fd282800640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd284161770 0x7fd2840ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd279400640 1 -- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2840b2cb0 msgr2=0x7fd2841501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd279400640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2840b2cb0 0x7fd2841501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd279400640 1 -- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd284161770 msgr2=0x7fd2840ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd279400640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd284161770 0x7fd2840ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd279400640 1 -- 172.21.15.32:0/3190704422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2840b7780 con 0x7fd2840b3200 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd279400640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840c9bd0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009890 tx=0x7fd26c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c041070 con 0x7fd2840b3200 2024-03-06T07:33:09.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c03aa30 con 0x7fd2840b3200 2024-03-06T07:33:09.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c035040 con 0x7fd2840b3200 2024-03-06T07:33:09.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.692+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2840b7a60 con 0x7fd2840b3200 2024-03-06T07:33:09.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.693+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2840b8020 con 0x7fd2840b3200 2024-03-06T07:33:09.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.694+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28404f330 con 0x7fd2840b3200 2024-03-06T07:33:09.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.695+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd26c038020 con 0x7fd2840b3200 2024-03-06T07:33:09.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.696+0000 7fd27aa00640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd26006b4d0 0x7fd26006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.696+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd26c0abd70 con 0x7fd2840b3200 2024-03-06T07:33:09.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.696+0000 7fd281e00640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd26006b4d0 0x7fd26006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.697+0000 7fd281e00640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd26006b4d0 0x7fd26006d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd2740099c0 tx=0x7fd274003990 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.700+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd26c07ab10 con 0x7fd2840b3200 2024-03-06T07:33:09.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.932+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2840d0380 con 0x7fd2840b3200 2024-03-06T07:33:09.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.933+0000 7fd27aa00640 1 -- 172.21.15.32:0/3190704422 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd26c07a410 con 0x7fd2840b3200 2024-03-06T07:33:09.935 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.935+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd26006b4d0 msgr2=0x7fd26006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.935+0000 7fd283200640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd26006b4d0 0x7fd26006d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd2740099c0 tx=0x7fd274003990 comp rx=0 tx=0).stop 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.935+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 msgr2=0x7fd2840c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.935+0000 7fd283200640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840c9bd0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009890 tx=0x7fd26c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 shutdown_connections 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd26006b4d0 0x7fd26006d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd284161770 0x7fd2840ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2840b3200 0x7fd2840c9bd0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 --2- 172.21.15.32:0/3190704422 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2840b2cb0 0x7fd2841501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 >> 172.21.15.32:0/3190704422 conn(0x7fd2840d3de0 msgr2=0x7fd28406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 shutdown_connections 2024-03-06T07:33:09.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:09.936+0000 7fd283200640 1 -- 172.21.15.32:0/3190704422 wait complete. 2024-03-06T07:33:09.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.163+0000 7ff86b200640 1 Processor -- start 2024-03-06T07:33:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.163+0000 7ff86b200640 1 -- start start 2024-03-06T07:33:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.164+0000 7ff86b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff86415c0e0 0x7ff8640cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.164+0000 7ff86b200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640ccc50 con 0x7ff8640cfdf0 2024-03-06T07:33:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.164+0000 7ff86b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640ccf30 con 0x7ff8640d09a0 2024-03-06T07:33:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.164+0000 7ff86b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86415c4c0 con 0x7ff86415c0e0 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.164+0000 7ff85b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff8640cfdf0 0x7ff8640d0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60234/0 (socket says 172.21.15.32:60234) 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.165+0000 7ff85b400640 1 -- 172.21.15.32:0/553546041 learned_addr learned my addr 172.21.15.32:0/553546041 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.165+0000 7ff863e00640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff86415c0e0 0x7ff8640cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.165+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201568272 0 0) 0x7ff8640ccc50 con 0x7ff8640cfdf0 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.165+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff848003520 con 0x7ff8640cfdf0 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2753001794 0 0) 0x7ff848003520 con 0x7ff8640cfdf0 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff86415c0e0 msgr2=0x7ff8640cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff86415c0e0 0x7ff8640cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 >> v1:172.21.15.39:6789/0 conn(0x7ff8640d09a0 legacy=0x7ff8640b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640cd210 con 0x7ff8640cfdf0 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff84c001670 con 0x7ff8640cfdf0 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 >> v1:172.21.15.32:6789/0 conn(0x7ff8640cfdf0 legacy=0x7ff8640d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8480039e0 0x7ff848003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8480043d0 0x7ff8480048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480050a0 0x7ff84800fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86415c4c0 con 0x7ff8480043d0 2024-03-06T07:33:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480103d0 con 0x7ff8480050a0 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480105d0 con 0x7ff8480039e0 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff863e00640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff86415c0e0 0x7ff8640cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff85b400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8480043d0 0x7ff8480048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.166+0000 7ff869200640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480050a0 0x7ff84800fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff863e00640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8480039e0 0x7ff848003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff85b400640 1 -- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8480039e0 msgr2=0x7ff848003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff85b400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8480039e0 0x7ff848003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff85b400640 1 -- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480050a0 msgr2=0x7ff84800fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff85b400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480050a0 0x7ff84800fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff85b400640 1 -- 172.21.15.32:0/553546041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848003520 con 0x7ff8480043d0 2024-03-06T07:33:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.167+0000 7ff863e00640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8480039e0 0x7ff848003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.168+0000 7ff85b400640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8480043d0 0x7ff8480048a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff8640d01f0 tx=0x7ff848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499f6a5a5b825fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c007d30 con 0x7ff8480043d0 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c030070 con 0x7ff8480043d0 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff863400640 1 -- 172.21.15.32:0/553546041 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c005d70 con 0x7ff8480043d0 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff86b200640 1 -- 172.21.15.32:0/553546041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8480043d0 msgr2=0x7ff8480048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff86b200640 1 --2- 172.21.15.32:0/553546041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8480043d0 0x7ff8480048a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff8640d01f0 tx=0x7ff848005830 comp rx=0 tx=0).stop 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff869200640 1 -- 172.21.15.32:0/553546041 reap_dead start 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff86b200640 1 -- 172.21.15.32:0/553546041 shutdown_connections 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff86b200640 1 -- 172.21.15.32:0/553546041 >> 172.21.15.32:0/553546041 conn(0x7ff8640b3820 msgr2=0x7ff8640b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.169+0000 7ff86b200640 1 -- 172.21.15.32:0/553546041 shutdown_connections 2024-03-06T07:33:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.170+0000 7ff86b200640 1 -- 172.21.15.32:0/553546041 wait complete. 2024-03-06T07:33:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.172+0000 7ff86b200640 1 Processor -- start 2024-03-06T07:33:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.173+0000 7ff86b200640 1 -- start start 2024-03-06T07:33:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.174+0000 7ff86b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480043d0 0x7ff864155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.174+0000 7ff86b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 0x7ff8640cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.174+0000 7ff86b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8640cba50 0x7ff86415e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.174+0000 7ff86b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640cda10 con 0x7ff8641f1670 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.174+0000 7ff86b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640cd580 con 0x7ff8480043d0 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.174+0000 7ff86b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640cdb90 con 0x7ff8640cba50 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 0x7ff8640cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 0x7ff8640cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42084/0 (socket says 172.21.15.32:42084) 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 -- 172.21.15.32:0/174194344 learned_addr learned my addr 172.21.15.32:0/174194344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff869200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8640cba50 0x7ff86415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff863e00640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480043d0 0x7ff864155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 -- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8640cba50 msgr2=0x7ff86415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8640cba50 0x7ff86415e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 -- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480043d0 msgr2=0x7ff864155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480043d0 0x7ff864155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 -- 172.21.15.32:0/174194344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640ca170 con 0x7ff8641f1670 2024-03-06T07:33:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.175+0000 7ff85b400640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 0x7ff8640cb510 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff84c001970 tx=0x7ff84c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.176+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c045070 con 0x7ff8641f1670 2024-03-06T07:33:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.176+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c000ed0 con 0x7ff8641f1670 2024-03-06T07:33:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.176+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8640ca450 con 0x7ff8641f1670 2024-03-06T07:33:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.176+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c004df0 con 0x7ff8641f1670 2024-03-06T07:33:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.176+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8640caa10 con 0x7ff8641f1670 2024-03-06T07:33:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.178+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8640c9f00 con 0x7ff8641f1670 2024-03-06T07:33:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.178+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff84c007070 con 0x7ff8641f1670 2024-03-06T07:33:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.179+0000 7ff860c00640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff84006b500 0x7ff84006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.179+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff84c0ab310 con 0x7ff8641f1670 2024-03-06T07:33:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.179+0000 7ff863e00640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff84006b500 0x7ff84006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.180+0000 7ff863e00640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff84006b500 0x7ff84006d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff854009fd0 tx=0x7ff85400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.183+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff84c079b70 con 0x7ff8641f1670 2024-03-06T07:33:10.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.454+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff86404f330 con 0x7ff8641f1670 2024-03-06T07:33:10.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.455+0000 7ff860c00640 1 -- 172.21.15.32:0/174194344 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff84c07dc10 con 0x7ff8641f1670 2024-03-06T07:33:10.457 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:33:10.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff84006b500 msgr2=0x7ff84006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff84006b500 0x7ff84006d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff854009fd0 tx=0x7ff85400b040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 msgr2=0x7ff8640cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 0x7ff8640cb510 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff84c001970 tx=0x7ff84c001a40 comp rx=0 tx=0).stop 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 shutdown_connections 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff84006b500 0x7ff84006d9c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8640cba50 0x7ff86415e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8641f1670 0x7ff8640cb510 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 --2- 172.21.15.32:0/174194344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8480043d0 0x7ff864155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 >> 172.21.15.32:0/174194344 conn(0x7ff8640b3820 msgr2=0x7ff8640cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 shutdown_connections 2024-03-06T07:33:10.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.458+0000 7ff86b200640 1 -- 172.21.15.32:0/174194344 wait complete. 2024-03-06T07:33:10.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.717+0000 7f4d52400640 1 Processor -- start 2024-03-06T07:33:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.718+0000 7f4d52400640 1 -- start start 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.719+0000 7f4d52400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 0x7f4d4c0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.719+0000 7f4d52400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4d4c072520 0x7f4d4c0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.719+0000 7f4d52400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c072df0 con 0x7f4d4c071e20 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.719+0000 7f4d52400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4c0cdab0 con 0x7f4d4c072520 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.719+0000 7f4d52400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0b4d70 con 0x7f4d4c15bf20 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.720+0000 7f4d4be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4d4c071e20 0x7f4d4c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60238/0 (socket says 172.21.15.32:60238) 2024-03-06T07:33:10.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.720+0000 7f4d4be00640 1 -- 172.21.15.32:0/858389456 learned_addr learned my addr 172.21.15.32:0/858389456 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.720+0000 7f4d4aa00640 1 --2- 172.21.15.32:0/858389456 >> v2:172.21.15.39:3300/0 conn(0x7f4d4c072520 0x7f4d4c0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.720+0000 7f4d4b400640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 0x7f4d4c0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.720+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561613997 0 0) 0x7f4d4c072df0 con 0x7f4d4c071e20 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003520 con 0x7f4d4c071e20 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 969094933 0 0) 0x7f4d30003520 con 0x7f4d4c071e20 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 msgr2=0x7f4d4c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 0x7f4d4c0acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 >> v2:172.21.15.39:3300/0 conn(0x7f4d4c072520 msgr2=0x7f4d4c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 --2- 172.21.15.32:0/858389456 >> v2:172.21.15.39:3300/0 conn(0x7f4d4c072520 0x7f4d4c0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c0b5050 con 0x7f4d4c071e20 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f4d44001670 con 0x7f4d4c071e20 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 >> v1:172.21.15.32:6789/0 conn(0x7f4d4c071e20 legacy=0x7f4d4c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d300039b0 0x7f4d30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 0x7f4d30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d30004d40 0x7f4d300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0cdab0 con 0x7f4d300039b0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0b4d70 con 0x7f4d30004d40 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.721+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000b9c0 con 0x7f4d4c15bf20 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4b400640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d300039b0 0x7f4d30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4aa00640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 0x7f4d30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4be00640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d30004d40 0x7f4d300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4b400640 1 -- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 msgr2=0x7f4d30004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4b400640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c15bf20 0x7f4d30004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4b400640 1 -- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d30004d40 msgr2=0x7f4d300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4b400640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d30004d40 0x7f4d300051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.722+0000 7f4d4b400640 1 -- 172.21.15.32:0/858389456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30003520 con 0x7f4d300039b0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.723+0000 7f4d4b400640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d300039b0 0x7f4d30003dd0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009fc0 tx=0x7f4d30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a49eb19ecd6ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.723+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c039070 con 0x7f4d300039b0 2024-03-06T07:33:10.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.724+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c02cbf0 con 0x7f4d300039b0 2024-03-06T07:33:10.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.724+0000 7f4d4a000640 1 -- 172.21.15.32:0/858389456 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c009c10 con 0x7f4d300039b0 2024-03-06T07:33:10.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.724+0000 7f4d52400640 1 -- 172.21.15.32:0/858389456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d300039b0 msgr2=0x7f4d30003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.724+0000 7f4d52400640 1 --2- 172.21.15.32:0/858389456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d300039b0 0x7f4d30003dd0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009fc0 tx=0x7f4d30005830 comp rx=0 tx=0).stop 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.725+0000 7f4d4be00640 1 -- 172.21.15.32:0/858389456 reap_dead start 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.725+0000 7f4d52400640 1 -- 172.21.15.32:0/858389456 shutdown_connections 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.725+0000 7f4d52400640 1 -- 172.21.15.32:0/858389456 >> 172.21.15.32:0/858389456 conn(0x7f4d4c06b3d0 msgr2=0x7f4d4c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.725+0000 7f4d52400640 1 -- 172.21.15.32:0/858389456 shutdown_connections 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.725+0000 7f4d52400640 1 -- 172.21.15.32:0/858389456 wait complete. 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.727+0000 7f4d52400640 1 Processor -- start 2024-03-06T07:33:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.727+0000 7f4d52400640 1 -- start start 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d52400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 0x7f4d4c15ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d52400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d4c0b1520 0x7f4d4c150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d52400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c1f7ca0 0x7f4d4c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d52400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0b4b10 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d52400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0b4990 con 0x7f4d4c0b1520 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d52400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0cd1e0 con 0x7f4d4c1f7ca0 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 0x7f4d4c15ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 0x7f4d4c15ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42104/0 (socket says 172.21.15.32:42104) 2024-03-06T07:33:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 -- 172.21.15.32:0/387539290 learned_addr learned my addr 172.21.15.32:0/387539290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:10.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 -- 172.21.15.32:0/387539290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c1f7ca0 msgr2=0x7f4d4c1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c1f7ca0 0x7f4d4c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 -- 172.21.15.32:0/387539290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d4c0b1520 msgr2=0x7f4d4c150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d4c0b1520 0x7f4d4c150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.729+0000 7f4d4b400640 1 -- 172.21.15.32:0/387539290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c0c9e40 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.730+0000 7f4d4b400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 0x7f4d4c15ece0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c02c660 tx=0x7f4d3c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.730+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c039050 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.730+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c03a040 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.730+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c009c10 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.730+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c0b2340 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.730+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c0b1aa0 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.732+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c04f330 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.733+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d3c047030 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.733+0000 7f4d43400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d2c06b550 0x7f4d2c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.734+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4d3c0ac350 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.734+0000 7f4d4aa00640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d2c06b550 0x7f4d2c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.735+0000 7f4d4aa00640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d2c06b550 0x7f4d2c06da10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c1f75b0 tx=0x7f4d38008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.739+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d3c04e060 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.937+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4d4c15f720 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.938+0000 7f4d43400640 1 -- 172.21.15.32:0/387539290 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d4c0b2340 con 0x7f4d4c1f78c0 2024-03-06T07:33:10.940 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:33:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d2c06b550 msgr2=0x7f4d2c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d2c06b550 0x7f4d2c06da10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c1f75b0 tx=0x7f4d38008040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 msgr2=0x7f4d4c15ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 0x7f4d4c15ece0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c02c660 tx=0x7f4d3c009750 comp rx=0 tx=0).stop 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 shutdown_connections 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4d2c06b550 0x7f4d2c06da10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4d4c1f7ca0 0x7f4d4c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d4c0b1520 0x7f4d4c150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 --2- 172.21.15.32:0/387539290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d4c1f78c0 0x7f4d4c15ece0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 >> 172.21.15.32:0/387539290 conn(0x7f4d4c06b3d0 msgr2=0x7f4d4c0ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 shutdown_connections 2024-03-06T07:33:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:10.940+0000 7f4d52400640 1 -- 172.21.15.32:0/387539290 wait complete. 2024-03-06T07:33:10.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:11.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.161+0000 7fdf2ca00640 1 Processor -- start 2024-03-06T07:33:11.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.161+0000 7fdf2ca00640 1 -- start start 2024-03-06T07:33:11.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf2ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf280b01f0 0x7fdf280adc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf2ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdf2806e640 0x7fdf280ae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf2ca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2815e3f0 con 0x7fdf2815e010 2024-03-06T07:33:11.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf2ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2806e440 con 0x7fdf2806e640 2024-03-06T07:33:11.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf2ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2806ee00 con 0x7fdf280b01f0 2024-03-06T07:33:11.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf25e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdf2815e010 0x7fdf280afae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60240/0 (socket says 172.21.15.32:60240) 2024-03-06T07:33:11.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.163+0000 7fdf25e00640 1 -- 172.21.15.32:0/2846581312 learned_addr learned my addr 172.21.15.32:0/2846581312 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf26800640 1 --2- 172.21.15.32:0/2846581312 >> v2:172.21.15.39:3300/0 conn(0x7fdf2806e640 0x7fdf280ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf280b01f0 0x7fdf280adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618062687 0 0) 0x7fdf2815e3f0 con 0x7fdf2815e010 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c003520 con 0x7fdf2815e010 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661201241 0 0) 0x7fdf0c003520 con 0x7fdf2815e010 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf280b01f0 msgr2=0x7fdf280adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf280b01f0 0x7fdf280adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 >> v2:172.21.15.39:3300/0 conn(0x7fdf2806e640 msgr2=0x7fdf280ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 --2- 172.21.15.32:0/2846581312 >> v2:172.21.15.39:3300/0 conn(0x7fdf2806e640 0x7fdf280ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2806eb20 con 0x7fdf2815e010 2024-03-06T07:33:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fdf18001670 con 0x7fdf2815e010 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 >> v1:172.21.15.32:6789/0 conn(0x7fdf2815e010 legacy=0x7fdf280afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c003a20 0x7fdf0c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0c004800 0x7fdf0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.164+0000 7fdf25400640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf0c0051c0 0x7fdf0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2806e440 con 0x7fdf0c004800 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c00bfe0 con 0x7fdf0c003a20 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c00c1e0 con 0x7fdf0c0051c0 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf280b01f0 0x7fdf280adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0c004800 0x7fdf0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf25e00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c003a20 0x7fdf0c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf26800640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf0c0051c0 0x7fdf0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 -- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf0c0051c0 msgr2=0x7fdf0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf0c0051c0 0x7fdf0c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 -- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c003a20 msgr2=0x7fdf0c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c003a20 0x7fdf0c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.165+0000 7fdf1fe00640 1 -- 172.21.15.32:0/2846581312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c003520 con 0x7fdf0c004800 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.166+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0c004800 0x7fdf0c004c80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006fd0 tx=0x7fdf0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117b0abf04116d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1000f040 con 0x7fdf0c004800 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10033600 con 0x7fdf0c004800 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf25400640 1 -- 172.21.15.32:0/2846581312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100339d0 con 0x7fdf0c004800 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf2ca00640 1 -- 172.21.15.32:0/2846581312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0c004800 msgr2=0x7fdf0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/2846581312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0c004800 0x7fdf0c004c80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006fd0 tx=0x7fdf0c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf26800640 1 -- 172.21.15.32:0/2846581312 reap_dead start 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf2ca00640 1 -- 172.21.15.32:0/2846581312 shutdown_connections 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.167+0000 7fdf2ca00640 1 -- 172.21.15.32:0/2846581312 >> 172.21.15.32:0/2846581312 conn(0x7fdf280d0bb0 msgr2=0x7fdf280d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.168+0000 7fdf2ca00640 1 -- 172.21.15.32:0/2846581312 shutdown_connections 2024-03-06T07:33:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.168+0000 7fdf2ca00640 1 -- 172.21.15.32:0/2846581312 wait complete. 2024-03-06T07:33:11.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.170+0000 7fdf2ca00640 1 Processor -- start 2024-03-06T07:33:11.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.170+0000 7fdf2ca00640 1 -- start start 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf2ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c004800 0x7fdf2814fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf2ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 0x7fdf28153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf2ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf281537a0 0x7fdf28157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf2ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2806e160 con 0x7fdf280b40d0 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf2ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2806dc30 con 0x7fdf0c004800 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf2ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280704d0 con 0x7fdf281537a0 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 0x7fdf28153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 0x7fdf28153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42126/0 (socket says 172.21.15.32:42126) 2024-03-06T07:33:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 -- 172.21.15.32:0/3977283997 learned_addr learned my addr 172.21.15.32:0/3977283997 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf25e00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c004800 0x7fdf2814fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf26800640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf281537a0 0x7fdf28157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 -- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf281537a0 msgr2=0x7fdf28157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf281537a0 0x7fdf28157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 -- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c004800 msgr2=0x7fdf2814fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.172+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c004800 0x7fdf2814fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf1fe00640 1 -- 172.21.15.32:0/3977283997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf28154270 con 0x7fdf280b40d0 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf1fe00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 0x7fdf28153260 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fdf28074f60 tx=0x7fdf100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10043070 con 0x7fdf280b40d0 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1000f000 con 0x7fdf280b40d0 2024-03-06T07:33:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf28153cb0 con 0x7fdf280b40d0 2024-03-06T07:33:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10032b90 con 0x7fdf280b40d0 2024-03-06T07:33:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.173+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf280b38d0 con 0x7fdf280b40d0 2024-03-06T07:33:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.175+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2804f3a0 con 0x7fdf280b40d0 2024-03-06T07:33:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.175+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf10046070 con 0x7fdf280b40d0 2024-03-06T07:33:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.176+0000 7fdf1e000640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf0406b4d0 0x7fdf0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.176+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf100abe70 con 0x7fdf280b40d0 2024-03-06T07:33:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.177+0000 7fdf25e00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf0406b4d0 0x7fdf0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.177+0000 7fdf25e00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf0406b4d0 0x7fdf0406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdf28153210 tx=0x7fdf18009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.181+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf1007aca0 con 0x7fdf280b40d0 2024-03-06T07:33:11.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.457+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf280b1910 con 0x7fdf280b40d0 2024-03-06T07:33:11.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.458+0000 7fdf1e000640 1 -- 172.21.15.32:0/3977283997 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fdf1007a5a0 con 0x7fdf280b40d0 2024-03-06T07:33:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:11.460 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:33:11.460 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:33:11.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf0406b4d0 msgr2=0x7fdf0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf0406b4d0 0x7fdf0406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdf28153210 tx=0x7fdf18009040 comp rx=0 tx=0).stop 2024-03-06T07:33:11.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 msgr2=0x7fdf28153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 0x7fdf28153260 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fdf28074f60 tx=0x7fdf100046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 shutdown_connections 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdf0406b4d0 0x7fdf0406d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdf281537a0 0x7fdf28157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf280b40d0 0x7fdf28153260 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 --2- 172.21.15.32:0/3977283997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf0c004800 0x7fdf2814fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 >> 172.21.15.32:0/3977283997 conn(0x7fdf280d0bb0 msgr2=0x7fdf2806b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 shutdown_connections 2024-03-06T07:33:11.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.461+0000 7fdf2ca00640 1 -- 172.21.15.32:0/3977283997 wait complete. 2024-03-06T07:33:11.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.701+0000 7f44ea800640 1 Processor -- start 2024-03-06T07:33:11.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.701+0000 7f44ea800640 1 -- start start 2024-03-06T07:33:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.703+0000 7f44ea800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 0x7f44e40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.703+0000 7f44ea800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40b3710 con 0x7f44e40d0c80 2024-03-06T07:33:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.703+0000 7f44ea800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40b39f0 con 0x7f44e415e0b0 2024-03-06T07:33:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.703+0000 7f44ea800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e40b09e0 con 0x7f44e40d4a10 2024-03-06T07:33:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.703+0000 7f44e9e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f44e40d0c80 0x7f44e40b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60250/0 (socket says 172.21.15.32:60250) 2024-03-06T07:33:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.703+0000 7f44e9e00640 1 -- 172.21.15.32:0/3567956998 learned_addr learned my addr 172.21.15.32:0/3567956998 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e8a00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 0x7f44e40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911581852 0 0) 0x7f44e40b3710 con 0x7f44e40d0c80 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc003520 con 0x7f44e40d0c80 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307996420 0 0) 0x7f44cc003520 con 0x7f44e40d0c80 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 msgr2=0x7f44e40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 0x7f44e40d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 >> v1:172.21.15.39:6789/0 conn(0x7f44e415e0b0 legacy=0x7f44e40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e40b3cd0 con 0x7f44e40d0c80 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f44dc0019f0 con 0x7f44e40d0c80 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 >> v1:172.21.15.32:6789/0 conn(0x7f44e40d0c80 legacy=0x7f44e40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 0x7f44cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc003f60 0x7f44cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44cc004d40 0x7f44cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00ba70 con 0x7f44cc004d40 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00bc20 con 0x7f44cc003f60 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.704+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00be20 con 0x7f44e40d4a10 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.705+0000 7f44e9e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44cc004d40 0x7f44cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.705+0000 7f44e9e00640 1 -- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 msgr2=0x7f44cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.705+0000 7f44e9e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40d4a10 0x7f44cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.705+0000 7f44e9e00640 1 -- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc003f60 msgr2=0x7f44cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.705+0000 7f44e9e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc003f60 0x7f44cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.705+0000 7f44e9e00640 1 -- 172.21.15.32:0/3567956998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc003520 con 0x7f44cc004d40 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.706+0000 7f44e9e00640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44cc004d40 0x7f44cc0051c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00a850 tx=0x7f44cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ef812b6e56f21b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.706+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc00a4a0 con 0x7f44cc004d40 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.706+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44dc034650 con 0x7f44cc004d40 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.706+0000 7f44e3e00640 1 -- 172.21.15.32:0/3567956998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc034a80 con 0x7f44cc004d40 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44ea800640 1 -- 172.21.15.32:0/3567956998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44cc004d40 msgr2=0x7f44cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44ea800640 1 --2- 172.21.15.32:0/3567956998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44cc004d40 0x7f44cc0051c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00a850 tx=0x7f44cc005830 comp rx=0 tx=0).stop 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44e9e00640 1 -- 172.21.15.32:0/3567956998 reap_dead start 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44ea800640 1 -- 172.21.15.32:0/3567956998 shutdown_connections 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44ea800640 1 -- 172.21.15.32:0/3567956998 >> 172.21.15.32:0/3567956998 conn(0x7f44e40d3d30 msgr2=0x7f44e40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44ea800640 1 -- 172.21.15.32:0/3567956998 shutdown_connections 2024-03-06T07:33:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.707+0000 7f44ea800640 1 -- 172.21.15.32:0/3567956998 wait complete. 2024-03-06T07:33:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.710+0000 7f44ea800640 1 Processor -- start 2024-03-06T07:33:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.710+0000 7f44ea800640 1 -- start start 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44ea800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc004d40 0x7f44e414d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44ea800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40b88a0 0x7f44e40b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44ea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 0x7f44e40cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44ea800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e415e490 con 0x7f44e40cb0a0 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44ea800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e40cfa20 con 0x7f44cc004d40 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44ea800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e40cfc60 con 0x7f44e40b88a0 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 0x7f44e40cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 0x7f44e40cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42142/0 (socket says 172.21.15.32:42142) 2024-03-06T07:33:11.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.712+0000 7f44e9e00640 1 -- 172.21.15.32:0/14969998 learned_addr learned my addr 172.21.15.32:0/14969998 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e8a00640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40b88a0 0x7f44e40b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9400640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc004d40 0x7f44e414d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9e00640 1 -- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40b88a0 msgr2=0x7f44e40b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9e00640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40b88a0 0x7f44e40b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9e00640 1 -- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc004d40 msgr2=0x7f44e414d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9e00640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc004d40 0x7f44e414d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9e00640 1 -- 172.21.15.32:0/14969998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e40cbe10 con 0x7f44e40cb0a0 2024-03-06T07:33:11.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e9e00640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 0x7f44e40cb550 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00a970 tx=0x7f44dc001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc034800 con 0x7f44e40cb0a0 2024-03-06T07:33:11.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44dc044070 con 0x7f44e40cb0a0 2024-03-06T07:33:11.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e40cc0f0 con 0x7f44e40cb0a0 2024-03-06T07:33:11.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.713+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc034ba0 con 0x7f44e40cb0a0 2024-03-06T07:33:11.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.714+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e40cc6b0 con 0x7f44e40cb0a0 2024-03-06T07:33:11.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.715+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e404f330 con 0x7f44e40cb0a0 2024-03-06T07:33:11.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.716+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44dc049460 con 0x7f44e40cb0a0 2024-03-06T07:33:11.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.717+0000 7f44e1600640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.717+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44dc0ac670 con 0x7f44e40cb0a0 2024-03-06T07:33:11.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.717+0000 7f44e9400640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.718+0000 7f44e9400640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f44c406b500 0x7f44c406d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f44e41f6fc0 tx=0x7f44d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.721+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44dc00a540 con 0x7f44e40cb0a0 2024-03-06T07:33:11.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.934+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44e40b09e0 con 0x7f44e40cb0a0 2024-03-06T07:33:11.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.936+0000 7f44e1600640 1 -- 172.21.15.32:0/14969998 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f44dc07b3a0 con 0x7f44e40cb0a0 2024-03-06T07:33:11.939 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:11.939 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":70,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:48.336272+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.938+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f44c406b500 msgr2=0x7f44c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.938+0000 7f44ea800640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f44c406b500 0x7f44c406d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f44e41f6fc0 tx=0x7f44d8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.938+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 msgr2=0x7f44e40cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.938+0000 7f44ea800640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 0x7f44e40cb550 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00a970 tx=0x7f44dc001dc0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 shutdown_connections 2024-03-06T07:33:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e40cb0a0 0x7f44e40cb550 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f44e40b88a0 0x7f44e40b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 --2- 172.21.15.32:0/14969998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44cc004d40 0x7f44e414d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 >> 172.21.15.32:0/14969998 conn(0x7f44e40d3d30 msgr2=0x7f44e40b5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 shutdown_connections 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:11.939+0000 7f44ea800640 1 -- 172.21.15.32:0/14969998 wait complete. 2024-03-06T07:33:11.961 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:11.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:12.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.197+0000 7fa28c600640 1 Processor -- start 2024-03-06T07:33:12.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.198+0000 7fa28c600640 1 -- start start 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa28c600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa28415dd00 0x7fa284162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa28c600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa28406c450 0x7fa28415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa28c600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28415ed30 con 0x7fa28406bfc0 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa28c600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa284161a80 con 0x7fa28406c450 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa28c600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28415efa0 con 0x7fa28415dd00 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa28a600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa28406c450 0x7fa28415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa289200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa28406bfc0 0x7fa28415e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60252/0 (socket says 172.21.15.32:60252) 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.200+0000 7fa289200640 1 -- 172.21.15.32:0/2040648594 learned_addr learned my addr 172.21.15.32:0/2040648594 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa289c00640 1 --2- 172.21.15.32:0/2040648594 >> v2:172.21.15.39:3300/0 conn(0x7fa28415dd00 0x7fa284162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa283e00640 1 -- 172.21.15.32:0/2040648594 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048081355 0 0) 0x7fa28415ed30 con 0x7fa28406bfc0 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa283e00640 1 -- 172.21.15.32:0/2040648594 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa26c003520 con 0x7fa28406bfc0 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa28a600640 1 -- 172.21.15.32:0/2040648594 >> v2:172.21.15.39:3300/0 conn(0x7fa28415dd00 msgr2=0x7fa284162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa28a600640 1 --2- 172.21.15.32:0/2040648594 >> v2:172.21.15.39:3300/0 conn(0x7fa28415dd00 0x7fa284162520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa28a600640 1 -- 172.21.15.32:0/2040648594 >> v1:172.21.15.32:6789/0 conn(0x7fa28406bfc0 legacy=0x7fa28415e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.201+0000 7fa28a600640 1 -- 172.21.15.32:0/2040648594 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28415f280 con 0x7fa28406c450 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.203+0000 7fa28a600640 1 --2- 172.21.15.32:0/2040648594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28406c450 0x7fa28415e7f0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa284071110 tx=0x7fa27c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4488a921be379e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.203+0000 7fa283e00640 1 -- 172.21.15.32:0/2040648594 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c039070 con 0x7fa28406c450 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.203+0000 7fa283e00640 1 -- 172.21.15.32:0/2040648594 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c02cbf0 con 0x7fa28406c450 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.203+0000 7fa283e00640 1 -- 172.21.15.32:0/2040648594 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c034440 con 0x7fa28406c450 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 -- 172.21.15.32:0/2040648594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28406c450 msgr2=0x7fa28415e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 --2- 172.21.15.32:0/2040648594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28406c450 0x7fa28415e7f0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa284071110 tx=0x7fa27c02be40 comp rx=0 tx=0).stop 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 -- 172.21.15.32:0/2040648594 shutdown_connections 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 --2- 172.21.15.32:0/2040648594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28406c450 0x7fa28415e7f0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 --2- 172.21.15.32:0/2040648594 >> v2:172.21.15.39:3300/0 conn(0x7fa28415dd00 0x7fa284162520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 -- 172.21.15.32:0/2040648594 >> 172.21.15.32:0/2040648594 conn(0x7fa2840714d0 msgr2=0x7fa2840718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 -- 172.21.15.32:0/2040648594 shutdown_connections 2024-03-06T07:33:12.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.204+0000 7fa28c600640 1 -- 172.21.15.32:0/2040648594 wait complete. 2024-03-06T07:33:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.207+0000 7fa28c600640 1 Processor -- start 2024-03-06T07:33:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.207+0000 7fa28c600640 1 -- start start 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28c600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa28406bfc0 0x7fa284144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28c600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa28406c450 0x7fa284156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28c600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 0x7fa2841568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28c600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28406c870 con 0x7fa28415dd00 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28c600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa284162000 con 0x7fa28406c450 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28c600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28406f280 con 0x7fa28406bfc0 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 0x7fa2841568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 0x7fa2841568c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42174/0 (socket says 172.21.15.32:42174) 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa28a600640 1 -- 172.21.15.32:0/2630746203 learned_addr learned my addr 172.21.15.32:0/2630746203 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa289c00640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa28406bfc0 0x7fa284144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.209+0000 7fa289200640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa28406c450 0x7fa284156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28a600640 1 -- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa28406bfc0 msgr2=0x7fa284144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28a600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa28406bfc0 0x7fa284144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28a600640 1 -- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa28406c450 msgr2=0x7fa284156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28a600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa28406c450 0x7fa284156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28a600640 1 -- 172.21.15.32:0/2630746203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28415b4c0 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28a600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 0x7fa2841568c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa27c02c450 tx=0x7fa27c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c039050 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c045070 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c009450 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.210+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa284153d30 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.211+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28415a720 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.212+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28404f3a0 con 0x7fa28415dd00 2024-03-06T07:33:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.213+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa27c03e850 con 0x7fa28415dd00 2024-03-06T07:33:12.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.214+0000 7fa281600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa26406b4d0 0x7fa26406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.214+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa27c0acf30 con 0x7fa28415dd00 2024-03-06T07:33:12.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.214+0000 7fa289c00640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa26406b4d0 0x7fa26406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.215+0000 7fa289c00640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa26406b4d0 0x7fa26406d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa278009a30 tx=0x7fa278008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.218+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa27c07bc40 con 0x7fa28415dd00 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.454+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa28406e410 con 0x7fa28415dd00 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.454+0000 7fa281600640 1 -- 172.21.15.32:0/2630746203 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fa27c07b540 con 0x7fa28415dd00 2024-03-06T07:33:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:12.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:33:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa26406b4d0 msgr2=0x7fa26406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa26406b4d0 0x7fa26406d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa278009a30 tx=0x7fa278008040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 msgr2=0x7fa2841568c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 0x7fa2841568c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa27c02c450 tx=0x7fa27c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 shutdown_connections 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa26406b4d0 0x7fa26406d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28415dd00 0x7fa2841568c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa28406c450 0x7fa284156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 --2- 172.21.15.32:0/2630746203 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa28406bfc0 0x7fa284144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 >> 172.21.15.32:0/2630746203 conn(0x7fa2840714d0 msgr2=0x7fa2840af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 shutdown_connections 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.457+0000 7fa28c600640 1 -- 172.21.15.32:0/2630746203 wait complete. 2024-03-06T07:33:12.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:12.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.669+0000 7f9e4cc00640 1 Processor -- start 2024-03-06T07:33:12.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.670+0000 7f9e4cc00640 1 -- start start 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.672+0000 7f9e4cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e480b3c90 0x7f9e480cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.672+0000 7f9e4cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e480cf5c0 0x7f9e480b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e4cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480b45b0 con 0x7f9e4806cb10 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e4cc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e480ce160 con 0x7f9e480cf5c0 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e4cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e480b4860 con 0x7f9e480b3c90 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e480cf5c0 0x7f9e480b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e480cf5c0 0x7f9e480b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42178/0 (socket says 172.21.15.32:42178) 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 -- 172.21.15.32:0/549962227 learned_addr learned my addr 172.21.15.32:0/549962227 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e45e00640 1 --2- 172.21.15.32:0/549962227 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e480b3c90 0x7f9e480cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 -- 172.21.15.32:0/549962227 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e480b3c90 msgr2=0x7f9e480cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 --2- 172.21.15.32:0/549962227 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e480b3c90 0x7f9e480cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 -- 172.21.15.32:0/549962227 >> v1:172.21.15.39:6789/0 conn(0x7f9e4806cb10 legacy=0x7f9e4806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.673+0000 7f9e3d400640 1 -- 172.21.15.32:0/549962227 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e480b4b40 con 0x7f9e480cf5c0 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.674+0000 7f9e3d400640 1 --2- 172.21.15.32:0/549962227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480cf5c0 0x7f9e480b4070 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009870 tx=0x7f9e3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ed98e64decddab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.675+0000 7f9e45400640 1 -- 172.21.15.32:0/549962227 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30034070 con 0x7f9e480cf5c0 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.675+0000 7f9e45400640 1 -- 172.21.15.32:0/549962227 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e30030840 con 0x7f9e480cf5c0 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.675+0000 7f9e45400640 1 -- 172.21.15.32:0/549962227 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30030c10 con 0x7f9e480cf5c0 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.675+0000 7f9e4cc00640 1 -- 172.21.15.32:0/549962227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480cf5c0 msgr2=0x7f9e480b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.675+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/549962227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480cf5c0 0x7f9e480b4070 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009870 tx=0x7f9e3002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.675+0000 7f9e4cc00640 1 -- 172.21.15.32:0/549962227 shutdown_connections 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.676+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/549962227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480cf5c0 0x7f9e480b4070 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.676+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/549962227 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e480b3c90 0x7f9e480cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.676+0000 7f9e4cc00640 1 -- 172.21.15.32:0/549962227 >> 172.21.15.32:0/549962227 conn(0x7f9e4806afb0 msgr2=0x7f9e4806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.676+0000 7f9e4cc00640 1 -- 172.21.15.32:0/549962227 shutdown_connections 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.676+0000 7f9e4cc00640 1 -- 172.21.15.32:0/549962227 wait complete. 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.677+0000 7f9e4cc00640 1 Processor -- start 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.677+0000 7f9e4cc00640 1 -- start start 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.678+0000 7f9e4cc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4806cb10 0x7f9e4809d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.678+0000 7f9e4cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 0x7f9e4809ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.678+0000 7f9e4cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e480cf5c0 0x7f9e4809b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.678+0000 7f9e4cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4815b340 con 0x7f9e480b3c90 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.678+0000 7f9e4cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e480b5100 con 0x7f9e480cf5c0 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.678+0000 7f9e4cc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4806d690 con 0x7f9e4806cb10 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 0x7f9e4809ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 0x7f9e4809ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42192/0 (socket says 172.21.15.32:42192) 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 -- 172.21.15.32:0/2563668283 learned_addr learned my addr 172.21.15.32:0/2563668283 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e45e00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4806cb10 0x7f9e4809d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 -- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4806cb10 msgr2=0x7f9e4809d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4806cb10 0x7f9e4809d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 -- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e480cf5c0 msgr2=0x7f9e4809b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e46800640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e480cf5c0 0x7f9e4809b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e480cf5c0 0x7f9e4809b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.679+0000 7f9e3d400640 1 -- 172.21.15.32:0/2563668283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48158d50 con 0x7f9e480b3c90 2024-03-06T07:33:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.680+0000 7f9e3d400640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 0x7f9e4809ac20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000c00 tx=0x7f9e300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.680+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30034040 con 0x7f9e480b3c90 2024-03-06T07:33:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.680+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3003ada0 con 0x7f9e480b3c90 2024-03-06T07:33:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.680+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e48159030 con 0x7f9e480b3c90 2024-03-06T07:33:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.680+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30044630 con 0x7f9e480b3c90 2024-03-06T07:33:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.680+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48152d90 con 0x7f9e480b3c90 2024-03-06T07:33:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.681+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e48069510 con 0x7f9e480b3c90 2024-03-06T07:33:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.682+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e3004b060 con 0x7f9e480b3c90 2024-03-06T07:33:12.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.683+0000 7f9e3ea00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2406b4d0 0x7f9e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.683+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e300ac280 con 0x7f9e480b3c90 2024-03-06T07:33:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.684+0000 7f9e45e00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2406b4d0 0x7f9e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.684+0000 7f9e45e00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2406b4d0 0x7f9e2406d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9e38006fd0 tx=0x7f9e38008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.692+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e3003f070 con 0x7f9e480b3c90 2024-03-06T07:33:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.947+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e48152b60 con 0x7f9e480b3c90 2024-03-06T07:33:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.948+0000 7f9e3ea00640 1 -- 172.21.15.32:0/2563668283 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f9e3007b150 con 0x7f9e480b3c90 2024-03-06T07:33:12.950 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:12.950 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:30.799560+0000","modified":"2024-03-06T07:32:33.904744+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6837/2323044149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2323044149},{"type":"v1","addr":"172.21.15.32:6837","nonce":2323044149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":5}]} 2024-03-06T07:33:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 43 2024-03-06T07:33:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2406b4d0 msgr2=0x7f9e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2406b4d0 0x7f9e2406d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9e38006fd0 tx=0x7f9e38008040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 msgr2=0x7f9e4809ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 0x7f9e4809ac20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000c00 tx=0x7f9e300046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 shutdown_connections 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e2406b4d0 0x7f9e2406d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e480cf5c0 0x7f9e4809b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e480b3c90 0x7f9e4809ac20 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 --2- 172.21.15.32:0/2563668283 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e4806cb10 0x7f9e4809d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 >> 172.21.15.32:0/2563668283 conn(0x7f9e4806afb0 msgr2=0x7f9e48156630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 shutdown_connections 2024-03-06T07:33:12.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:12.951+0000 7f9e4cc00640 1 -- 172.21.15.32:0/2563668283 wait complete. 2024-03-06T07:33:12.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:13.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.172+0000 7fbaeae00640 1 Processor -- start 2024-03-06T07:33:13.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.173+0000 7fbaeae00640 1 -- start start 2024-03-06T07:33:13.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.174+0000 7fbaeae00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae415e2a0 0x7fbae40cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.174+0000 7fbaeae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.174+0000 7fbaeae00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae40d1120 con 0x7fbae415dec0 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.174+0000 7fbaeae00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae40b0ea0 con 0x7fbae415e2a0 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.174+0000 7fbaeae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40d1400 con 0x7fbae40cf460 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae415e2a0 0x7fbae40cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae415e2a0 0x7fbae40cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42202/0 (socket says 172.21.15.32:42202) 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 -- 172.21.15.32:0/96212744 learned_addr learned my addr 172.21.15.32:0/96212744 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbaea400640 1 --2- 172.21.15.32:0/96212744 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 -- 172.21.15.32:0/96212744 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 msgr2=0x7fbae40b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 --2- 172.21.15.32:0/96212744 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 -- 172.21.15.32:0/96212744 >> v1:172.21.15.39:6789/0 conn(0x7fbae415dec0 legacy=0x7fbae40ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.175+0000 7fbae9000640 1 -- 172.21.15.32:0/96212744 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40b0b30 con 0x7fbae415e2a0 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.176+0000 7fbae9000640 1 --2- 172.21.15.32:0/96212744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415e2a0 0x7fbae40cef20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009870 tx=0x7fbad002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5906bef6afacfa51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbae3e00640 1 -- 172.21.15.32:0/96212744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0035020 con 0x7fbae415e2a0 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbae3e00640 1 -- 172.21.15.32:0/96212744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad0031640 con 0x7fbae415e2a0 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbae3e00640 1 -- 172.21.15.32:0/96212744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad002d040 con 0x7fbae415e2a0 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 -- 172.21.15.32:0/96212744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415e2a0 msgr2=0x7fbae40cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 --2- 172.21.15.32:0/96212744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415e2a0 0x7fbae40cef20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009870 tx=0x7fbad002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 -- 172.21.15.32:0/96212744 shutdown_connections 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 --2- 172.21.15.32:0/96212744 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 --2- 172.21.15.32:0/96212744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415e2a0 0x7fbae40cef20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 -- 172.21.15.32:0/96212744 >> 172.21.15.32:0/96212744 conn(0x7fbae40d3d40 msgr2=0x7fbae40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 -- 172.21.15.32:0/96212744 shutdown_connections 2024-03-06T07:33:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.177+0000 7fbaeae00640 1 -- 172.21.15.32:0/96212744 wait complete. 2024-03-06T07:33:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.179+0000 7fbaeae00640 1 Processor -- start 2024-03-06T07:33:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.179+0000 7fbaeae00640 1 -- start start 2024-03-06T07:33:13.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbaeae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbaeae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 0x7fbae40b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbaeae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbae415e2a0 0x7fbae40b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbaeae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40d0ec0 con 0x7fbae415dec0 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbaeae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40d0d40 con 0x7fbae415e2a0 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbaeae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40b1870 con 0x7fbae40cf460 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbae9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 0x7fbae40b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbae9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 0x7fbae40b79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42212/0 (socket says 172.21.15.32:42212) 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.180+0000 7fbae9000640 1 -- 172.21.15.32:0/3594421474 learned_addr learned my addr 172.21.15.32:0/3594421474 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9a00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbaea400640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbae415e2a0 0x7fbae40b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9000640 1 -- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 msgr2=0x7fbae40b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9000640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9000640 1 -- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbae415e2a0 msgr2=0x7fbae40b8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9000640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbae415e2a0 0x7fbae40b8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9000640 1 -- 172.21.15.32:0/3594421474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40b7f20 con 0x7fbae415dec0 2024-03-06T07:33:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.181+0000 7fbae9000640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 0x7fbae40b79e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbad002b930 tx=0x7fbad00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.182+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0040070 con 0x7fbae415dec0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.182+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad00399f0 con 0x7fbae415dec0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.182+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0035040 con 0x7fbae415dec0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.182+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae40b43e0 con 0x7fbae415dec0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.182+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae40b49a0 con 0x7fbae415dec0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.183+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae404f330 con 0x7fbae415dec0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.184+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbad004b060 con 0x7fbae415dec0 2024-03-06T07:33:13.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.185+0000 7fbae1600640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbac406b4d0 0x7fbac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.185+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbad00abc60 con 0x7fbae415dec0 2024-03-06T07:33:13.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.185+0000 7fbae9a00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbac406b4d0 0x7fbac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.186+0000 7fbae9a00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbac406b4d0 0x7fbac406d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbad8004ea0 tx=0x7fbad8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.190+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbad007a970 con 0x7fbae415dec0 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.441+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbae4161560 con 0x7fbae415dec0 2024-03-06T07:33:13.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.443+0000 7fbae1600640 1 -- 172.21.15.32:0/3594421474 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16657 (secure 0 0 0) 0x7fbad007a270 con 0x7fbae415dec0 2024-03-06T07:33:13.446 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:13.446 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":70,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:32:48.336272+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:28.663107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:29.814796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.8","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.446+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbac406b4d0 msgr2=0x7fbac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.446+0000 7fbaeae00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbac406b4d0 0x7fbac406d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbad8004ea0 tx=0x7fbad8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 msgr2=0x7fbae40b79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 0x7fbae40b79e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbad002b930 tx=0x7fbad00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 shutdown_connections 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbac406b4d0 0x7fbac406d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbae415e2a0 0x7fbae40b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae415dec0 0x7fbae40b79e0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 --2- 172.21.15.32:0/3594421474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbae40cf460 0x7fbae40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 >> 172.21.15.32:0/3594421474 conn(0x7fbae40d3d40 msgr2=0x7fbae40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 shutdown_connections 2024-03-06T07:33:13.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.447+0000 7fbaeae00640 1 -- 172.21.15.32:0/3594421474 wait complete. 2024-03-06T07:33:13.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:33:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.674+0000 7f5684a00640 1 Processor -- start 2024-03-06T07:33:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.675+0000 7f5684a00640 1 -- start start 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.676+0000 7f5684a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56800af580 0x7f568015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.676+0000 7f5684a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568015cc40 con 0x7f56800cdc20 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.676+0000 7f5684a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568015cf20 con 0x7f56800af1a0 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.676+0000 7f5684a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568015d200 con 0x7f56800af580 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56800af580 0x7f568015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f56800cdc20 0x7f56800b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60254/0 (socket says 172.21.15.32:60254) 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56800af580 0x7f568015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42218/0 (socket says 172.21.15.32:42218) 2024-03-06T07:33:13.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567e800640 1 -- 172.21.15.32:0/3846601778 learned_addr learned my addr 172.21.15.32:0/3846601778 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567d400640 1 -- 172.21.15.32:0/3846601778 >> v1:172.21.15.39:6789/0 conn(0x7f56800af1a0 legacy=0x7f56800b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567d400640 1 -- 172.21.15.32:0/3846601778 >> v1:172.21.15.32:6789/0 conn(0x7f56800cdc20 legacy=0x7f56800b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567ca00640 1 -- 172.21.15.32:0/3846601778 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729269506 0 0) 0x7f568015cc40 con 0x7f56800cdc20 2024-03-06T07:33:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.677+0000 7f567d400640 1 -- 172.21.15.32:0/3846601778 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568015d4e0 con 0x7f56800af580 2024-03-06T07:33:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.678+0000 7f567d400640 1 --2- 172.21.15.32:0/3846601778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568015c600 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f56680097a0 tx=0x7f566802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ce3c62ac848506 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.679+0000 7f567ca00640 1 -- 172.21.15.32:0/3846601778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680315a0 con 0x7f56800af580 2024-03-06T07:33:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.679+0000 7f567ca00640 1 -- 172.21.15.32:0/3846601778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668037070 con 0x7f56800af580 2024-03-06T07:33:13.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.679+0000 7f567ca00640 1 -- 172.21.15.32:0/3846601778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668031960 con 0x7f56800af580 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.679+0000 7f5684a00640 1 -- 172.21.15.32:0/3846601778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 msgr2=0x7f568015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.679+0000 7f5684a00640 1 --2- 172.21.15.32:0/3846601778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568015c600 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f56680097a0 tx=0x7f566802b840 comp rx=0 tx=0).stop 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.680+0000 7f5684a00640 1 -- 172.21.15.32:0/3846601778 shutdown_connections 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.680+0000 7f5684a00640 1 --2- 172.21.15.32:0/3846601778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568015c600 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.680+0000 7f5684a00640 1 -- 172.21.15.32:0/3846601778 >> 172.21.15.32:0/3846601778 conn(0x7f568006afb0 msgr2=0x7f568006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.680+0000 7f5684a00640 1 -- 172.21.15.32:0/3846601778 shutdown_connections 2024-03-06T07:33:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.680+0000 7f5684a00640 1 -- 172.21.15.32:0/3846601778 wait complete. 2024-03-06T07:33:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.681+0000 7f5684a00640 1 Processor -- start 2024-03-06T07:33:13.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.681+0000 7f5684a00640 1 -- start start 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.682+0000 7f5684a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56800af1a0 0x7f56800ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.682+0000 7f5684a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568014a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.682+0000 7f5684a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56800cdc20 0x7f568015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.682+0000 7f5684a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5680155e70 con 0x7f56800af580 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.682+0000 7f5684a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56800b3730 con 0x7f56800cdc20 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.682+0000 7f5684a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5680155ff0 con 0x7f56800af1a0 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568014a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568014a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42230/0 (socket says 172.21.15.32:42230) 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 -- 172.21.15.32:0/3863873671 learned_addr learned my addr 172.21.15.32:0/3863873671 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:13.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567de00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56800af1a0 0x7f56800ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 -- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56800af1a0 msgr2=0x7f56800ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56800af1a0 0x7f56800ad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 -- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56800cdc20 msgr2=0x7f568015e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567e800640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56800cdc20 0x7f568015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56800cdc20 0x7f568015e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567d400640 1 -- 172.21.15.32:0/3863873671 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568009aa10 con 0x7f56800af580 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.683+0000 7f567de00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56800af1a0 0x7f56800ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.684+0000 7f567d400640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568014a6a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f56680097a0 tx=0x7f56680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.684+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668004230 con 0x7f56800af580 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.684+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56680043f0 con 0x7f56800af580 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.684+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668032880 con 0x7f56800af580 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.684+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568006e600 con 0x7f56800af580 2024-03-06T07:33:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.684+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568015bff0 con 0x7f56800af580 2024-03-06T07:33:13.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.686+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5680069510 con 0x7f56800af580 2024-03-06T07:33:13.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.686+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5668032a20 con 0x7f56800af580 2024-03-06T07:33:13.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.687+0000 7f5676000640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f565c06b4d0 0x7f565c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.687+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f566802f030 con 0x7f56800af580 2024-03-06T07:33:13.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.688+0000 7f567de00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f565c06b4d0 0x7f565c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.688+0000 7f567de00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f565c06b4d0 0x7f565c06d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f56800ce8e0 tx=0x7f5670009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.692+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5668041070 con 0x7f56800af580 2024-03-06T07:33:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.968+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5680155900 con 0x7f56800af580 2024-03-06T07:33:13.970 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:33:13.969+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:13.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.971+0000 7f5676000640 1 -- 172.21.15.32:0/3863873671 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7f566807b390 con 0x7f56800af580 2024-03-06T07:33:13.974 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:33:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.974+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f565c06b4d0 msgr2=0x7f565c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.974+0000 7f5684a00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f565c06b4d0 0x7f565c06d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f56800ce8e0 tx=0x7f5670009040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.974+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 msgr2=0x7f568014a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.974+0000 7f5684a00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568014a6a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f56680097a0 tx=0x7f56680046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 shutdown_connections 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f565c06b4d0 0x7f565c06d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56800cdc20 0x7f568015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56800af580 0x7f568014a6a0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 --2- 172.21.15.32:0/3863873671 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56800af1a0 0x7f56800ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 >> 172.21.15.32:0/3863873671 conn(0x7f568006afb0 msgr2=0x7f56800cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 shutdown_connections 2024-03-06T07:33:13.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:13.975+0000 7f5684a00640 1 -- 172.21.15.32:0/3863873671 wait complete. 2024-03-06T07:33:13.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:33:14.098 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:33:14.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.206+0000 7f8934c00640 1 Processor -- start 2024-03-06T07:33:14.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.206+0000 7f8934c00640 1 -- start start 2024-03-06T07:33:14.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.208+0000 7f8934c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8930145430 0x7f893015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.208+0000 7f8934c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 0x7f89300b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.208+0000 7f8934c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300b3fe0 con 0x7f89300b2080 2024-03-06T07:33:14.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.208+0000 7f8934c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893015fe40 con 0x7f8930145430 2024-03-06T07:33:14.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.208+0000 7f8934c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8930160120 con 0x7f893015cc90 2024-03-06T07:33:14.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 0x7f89300b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 0x7f89300b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:43574/0 (socket says 172.21.15.32:43574) 2024-03-06T07:33:14.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 -- 172.21.15.32:0/2039635757 learned_addr learned my addr 172.21.15.32:0/2039635757 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:14.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f892f400640 1 --2- 172.21.15.32:0/2039635757 >> v2:172.21.15.39:3300/0 conn(0x7f8930145430 0x7f893015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 -- 172.21.15.32:0/2039635757 >> v2:172.21.15.39:3300/0 conn(0x7f8930145430 msgr2=0x7f893015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 --2- 172.21.15.32:0/2039635757 >> v2:172.21.15.39:3300/0 conn(0x7f8930145430 0x7f893015f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 -- 172.21.15.32:0/2039635757 >> v1:172.21.15.39:6789/0 conn(0x7f89300b2080 legacy=0x7f893006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.209+0000 7f8926a00640 1 -- 172.21.15.32:0/2039635757 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930160400 con 0x7f893015cc90 2024-03-06T07:33:14.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.211+0000 7f8926a00640 1 --2- 172.21.15.32:0/2039635757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 0x7f89300b1b40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8918009870 tx=0x7f891802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43bb04a44771962 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.212+0000 7f892ea00640 1 -- 172.21.15.32:0/2039635757 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918034070 con 0x7f893015cc90 2024-03-06T07:33:14.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.212+0000 7f892ea00640 1 -- 172.21.15.32:0/2039635757 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918030840 con 0x7f893015cc90 2024-03-06T07:33:14.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.212+0000 7f892ea00640 1 -- 172.21.15.32:0/2039635757 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918030c10 con 0x7f893015cc90 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.212+0000 7f8934c00640 1 -- 172.21.15.32:0/2039635757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 msgr2=0x7f89300b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.212+0000 7f8934c00640 1 --2- 172.21.15.32:0/2039635757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 0x7f89300b1b40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8918009870 tx=0x7f891802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.213+0000 7f8934c00640 1 -- 172.21.15.32:0/2039635757 shutdown_connections 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.213+0000 7f8934c00640 1 --2- 172.21.15.32:0/2039635757 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893015cc90 0x7f89300b1b40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.213+0000 7f8934c00640 1 --2- 172.21.15.32:0/2039635757 >> v2:172.21.15.39:3300/0 conn(0x7f8930145430 0x7f893015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.213+0000 7f8934c00640 1 -- 172.21.15.32:0/2039635757 >> 172.21.15.32:0/2039635757 conn(0x7f893006b3a0 msgr2=0x7f893006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.213+0000 7f8934c00640 1 -- 172.21.15.32:0/2039635757 shutdown_connections 2024-03-06T07:33:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.213+0000 7f8934c00640 1 -- 172.21.15.32:0/2039635757 wait complete. 2024-03-06T07:33:14.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.214+0000 7f8934c00640 1 Processor -- start 2024-03-06T07:33:14.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.215+0000 7f8934c00640 1 -- start start 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.215+0000 7f8934c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89300b2080 0x7f8930138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8934c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 0x7f8930071950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8934c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893015cc90 0x7f89300c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8934c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930145810 con 0x7f8930145430 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8934c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893015d0d0 con 0x7f893015cc90 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8934c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8930145990 con 0x7f89300b2080 2024-03-06T07:33:14.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8926a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 0x7f8930071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8926a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 0x7f8930071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42258/0 (socket says 172.21.15.32:42258) 2024-03-06T07:33:14.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8926a00640 1 -- 172.21.15.32:0/4282975041 learned_addr learned my addr 172.21.15.32:0/4282975041 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:14.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f892f400640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89300b2080 0x7f8930138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f892fe00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893015cc90 0x7f89300c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8926a00640 1 -- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89300b2080 msgr2=0x7f8930138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.216+0000 7f8926a00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89300b2080 0x7f8930138210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8926a00640 1 -- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893015cc90 msgr2=0x7f89300c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8926a00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893015cc90 0x7f89300c49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8926a00640 1 -- 172.21.15.32:0/4282975041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930072410 con 0x7f8930145430 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8926a00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 0x7f8930071950 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8918009870 tx=0x7f89180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918034040 con 0x7f8930145430 2024-03-06T07:33:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918039d90 con 0x7f8930145430 2024-03-06T07:33:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918043630 con 0x7f8930145430 2024-03-06T07:33:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89300726f0 con 0x7f8930145430 2024-03-06T07:33:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.217+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930138750 con 0x7f8930145430 2024-03-06T07:33:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.219+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893004f330 con 0x7f8930145430 2024-03-06T07:33:14.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.220+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f891804a060 con 0x7f8930145430 2024-03-06T07:33:14.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.221+0000 7f8927e00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.221+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f89180ac320 con 0x7f8930145430 2024-03-06T07:33:14.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.221+0000 7f892f400640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.222+0000 7f892f400640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f890c06b4d0 0x7f890c06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f89200074a0 tx=0x7f892000b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.226+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f891803e070 con 0x7f8930145430 2024-03-06T07:33:14.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.476+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f89300cc8c0 con 0x7f8930145430 2024-03-06T07:33:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.976+0000 7f8927e00640 1 -- 172.21.15.32:0/4282975041 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f891802d330 con 0x7f8930145430 2024-03-06T07:33:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.979+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f890c06b4d0 msgr2=0x7f890c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.979+0000 7f8934c00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f890c06b4d0 0x7f890c06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f89200074a0 tx=0x7f892000b040 comp rx=0 tx=0).stop 2024-03-06T07:33:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.979+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 msgr2=0x7f8930071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.979+0000 7f8934c00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 0x7f8930071950 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8918009870 tx=0x7f89180046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 shutdown_connections 2024-03-06T07:33:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893015cc90 0x7f89300c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8930145430 0x7f8930071950 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 --2- 172.21.15.32:0/4282975041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89300b2080 0x7f8930138210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 >> 172.21.15.32:0/4282975041 conn(0x7f893006b3a0 msgr2=0x7f89300ad9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 shutdown_connections 2024-03-06T07:33:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:14.980+0000 7f8934c00640 1 -- 172.21.15.32:0/4282975041 wait complete. 2024-03-06T07:33:15.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:33:15.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.222+0000 7fd1ece00640 1 Processor -- start 2024-03-06T07:33:15.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.222+0000 7fd1ece00640 1 -- start start 2024-03-06T07:33:15.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.224+0000 7fd1ece00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1e80ae7b0 0x7fd1e80aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.224+0000 7fd1ece00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.224+0000 7fd1ece00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1e80aeb90 0x7fd1e806cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.224+0000 7fd1ece00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e80d04d0 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.224+0000 7fd1ece00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e815d300 con 0x7fd1e80aeb90 2024-03-06T07:33:15.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.224+0000 7fd1ece00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81531f0 con 0x7fd1e80b3640 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1e80ae7b0 0x7fd1e80aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1e80ae7b0 0x7fd1e80aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42268/0 (socket says 172.21.15.32:42268) 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 -- 172.21.15.32:0/3745358433 learned_addr learned my addr 172.21.15.32:0/3745358433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5400640 1 --2- 172.21.15.32:0/3745358433 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e6800640 1 --2- 172.21.15.32:0/3745358433 >> v2:172.21.15.39:3300/0 conn(0x7fd1e80aeb90 0x7fd1e806cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 -- 172.21.15.32:0/3745358433 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 msgr2=0x7fd1e815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/3745358433 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e815cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 -- 172.21.15.32:0/3745358433 >> v2:172.21.15.39:3300/0 conn(0x7fd1e80aeb90 msgr2=0x7fd1e806cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/3745358433 >> v2:172.21.15.39:3300/0 conn(0x7fd1e80aeb90 0x7fd1e806cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.225+0000 7fd1e5e00640 1 -- 172.21.15.32:0/3745358433 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80b2b80 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.226+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/3745358433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e80aced0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8070a30 tx=0x7fd1d802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96824053a893fa36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.227+0000 7fd1e4a00640 1 -- 172.21.15.32:0/3745358433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8035020 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.227+0000 7fd1e4a00640 1 -- 172.21.15.32:0/3745358433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d802d040 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.227+0000 7fd1e4a00640 1 -- 172.21.15.32:0/3745358433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8031a80 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 -- 172.21.15.32:0/3745358433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 msgr2=0x7fd1e80aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 --2- 172.21.15.32:0/3745358433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e80aced0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8070a30 tx=0x7fd1d802b900 comp rx=0 tx=0).stop 2024-03-06T07:33:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 -- 172.21.15.32:0/3745358433 shutdown_connections 2024-03-06T07:33:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 --2- 172.21.15.32:0/3745358433 >> v2:172.21.15.39:3300/0 conn(0x7fd1e80aeb90 0x7fd1e806cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 --2- 172.21.15.32:0/3745358433 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 --2- 172.21.15.32:0/3745358433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e80aced0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 -- 172.21.15.32:0/3745358433 >> 172.21.15.32:0/3745358433 conn(0x7fd1e806afb0 msgr2=0x7fd1e806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 -- 172.21.15.32:0/3745358433 shutdown_connections 2024-03-06T07:33:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.228+0000 7fd1ece00640 1 -- 172.21.15.32:0/3745358433 wait complete. 2024-03-06T07:33:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.230+0000 7fd1ece00640 1 Processor -- start 2024-03-06T07:33:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.230+0000 7fd1ece00640 1 -- start start 2024-03-06T07:33:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1ece00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e8070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1ece00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80aeb90 0x7fd1e8137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1ece00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e814dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1ece00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80b2e40 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1ece00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e806eeb0 con 0x7fd1e80aeb90 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1ece00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80b3120 con 0x7fd1e80b3640 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e8070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e8070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42272/0 (socket says 172.21.15.32:42272) 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5e00640 1 -- 172.21.15.32:0/362430685 learned_addr learned my addr 172.21.15.32:0/362430685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e6800640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e814dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5e00640 1 -- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 msgr2=0x7fd1e814dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e814dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5e00640 1 -- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80aeb90 msgr2=0x7fd1e8137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.231+0000 7fd1e5400640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80aeb90 0x7fd1e8137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80aeb90 0x7fd1e8137ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1e5e00640 1 -- 172.21.15.32:0/362430685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e81380e0 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e8070c70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd1d802b940 tx=0x7fd1d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d803ac60 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d803ae00 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8044610 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e814e130 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.232+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80715e0 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.234+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e8068010 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.235+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1d8038020 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.235+0000 7fd1de000640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c806b500 0x7fd1c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.235+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd1d80ac310 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.236+0000 7fd1e5400640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c806b500 0x7fd1c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.237+0000 7fd1e5400640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c806b500 0x7fd1c806d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4006fd0 tx=0x7fd1d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.240+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1d803f070 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.443+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd1e815c150 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.994+0000 7fd1de000640 1 -- 172.21.15.32:0/362430685 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd1d807b210 con 0x7fd1e80ae7b0 2024-03-06T07:33:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:33:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c806b500 msgr2=0x7fd1c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c806b500 0x7fd1c806d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4006fd0 tx=0x7fd1d4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 msgr2=0x7fd1e8070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e8070c70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd1d802b940 tx=0x7fd1d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 shutdown_connections 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c806b500 0x7fd1c806d9c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80b3640 0x7fd1e814dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80aeb90 0x7fd1e8137ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 --2- 172.21.15.32:0/362430685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e80ae7b0 0x7fd1e8070c70 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 >> 172.21.15.32:0/362430685 conn(0x7fd1e806afb0 msgr2=0x7fd1e80af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 shutdown_connections 2024-03-06T07:33:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:15.998+0000 7fd1ece00640 1 -- 172.21.15.32:0/362430685 wait complete. 2024-03-06T07:33:16.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:33:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.233+0000 7f0e62400640 1 Processor -- start 2024-03-06T07:33:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.234+0000 7f0e62400640 1 -- start start 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.235+0000 7f0e62400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0e5c0ae7b0 0x7f0e5c0aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.235+0000 7f0e62400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 0x7f0e5c06cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.235+0000 7f0e62400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c1531f0 con 0x7f0e5c0b3640 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.235+0000 7f0e62400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c0d04d0 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.235+0000 7f0e62400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b2b80 con 0x7f0e5c15cdc0 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0e5c0ae7b0 0x7f0e5c0aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0e5c0ae7b0 0x7f0e5c0aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42288/0 (socket says 172.21.15.32:42288) 2024-03-06T07:33:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 -- 172.21.15.32:0/1822498866 learned_addr learned my addr 172.21.15.32:0/1822498866 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 -- 172.21.15.32:0/1822498866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 msgr2=0x7f0e5c06cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 --2- 172.21.15.32:0/1822498866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 0x7f0e5c06cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 -- 172.21.15.32:0/1822498866 >> v1:172.21.15.39:6789/0 conn(0x7f0e5c0b3640 legacy=0x7f0e5c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.236+0000 7f0e5b400640 1 -- 172.21.15.32:0/1822498866 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c06ebd0 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.237+0000 7f0e5b400640 1 --2- 172.21.15.32:0/1822498866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c0aced0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009e50 tx=0x7f0e4c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f892880f55ff1417 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e5a000640 1 -- 172.21.15.32:0/1822498866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c02ca70 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e5a000640 1 -- 172.21.15.32:0/1822498866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c02cc10 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e5a000640 1 -- 172.21.15.32:0/1822498866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0343f0 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e62400640 1 -- 172.21.15.32:0/1822498866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 msgr2=0x7f0e5c0aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e62400640 1 --2- 172.21.15.32:0/1822498866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c0aced0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009e50 tx=0x7f0e4c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e62400640 1 -- 172.21.15.32:0/1822498866 shutdown_connections 2024-03-06T07:33:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e62400640 1 --2- 172.21.15.32:0/1822498866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 0x7f0e5c06cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e62400640 1 --2- 172.21.15.32:0/1822498866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c0aced0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.238+0000 7f0e62400640 1 -- 172.21.15.32:0/1822498866 >> 172.21.15.32:0/1822498866 conn(0x7f0e5c06afb0 msgr2=0x7f0e5c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.239+0000 7f0e62400640 1 -- 172.21.15.32:0/1822498866 shutdown_connections 2024-03-06T07:33:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.239+0000 7f0e62400640 1 -- 172.21.15.32:0/1822498866 wait complete. 2024-03-06T07:33:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.240+0000 7f0e62400640 1 Processor -- start 2024-03-06T07:33:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.240+0000 7f0e62400640 1 -- start start 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.241+0000 7f0e62400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.241+0000 7f0e62400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5c0b3640 0x7f0e5c1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.241+0000 7f0e62400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 0x7f0e5c158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.241+0000 7f0e62400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b3120 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.241+0000 7f0e62400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b2e40 con 0x7f0e5c0b3640 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.241+0000 7f0e62400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c15d250 con 0x7f0e5c15cdc0 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42290/0 (socket says 172.21.15.32:42290) 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 -- 172.21.15.32:0/2814021251 learned_addr learned my addr 172.21.15.32:0/2814021251 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 -- 172.21.15.32:0/2814021251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 msgr2=0x7f0e5c158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 0x7f0e5c158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 -- 172.21.15.32:0/2814021251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5c0b3640 msgr2=0x7f0e5c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5c0b3640 0x7f0e5c1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 -- 172.21.15.32:0/2814021251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c157610 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.242+0000 7f0e5b400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c141620 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c036040 tx=0x7f0e4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.243+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c043020 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.243+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c03e450 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.243+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c155030 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.243+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c009510 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.243+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c1555f0 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.244+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c155d10 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.245+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e4c046020 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.246+0000 7f0e53400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.246+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f0e4c0acaa0 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.247+0000 7f0e5aa00640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.247+0000 7f0e5aa00640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c09b4f0 tx=0x7f0e480079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.251+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e4c04d030 con 0x7f0e5c0ae7b0 2024-03-06T07:33:16.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:16.454+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e5c068010 con 0x7f0e5c0ae7b0 2024-03-06T07:33:17.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.007+0000 7f0e53400640 1 -- 172.21.15.32:0/2814021251 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0e4c07b9a0 con 0x7f0e5c0ae7b0 2024-03-06T07:33:17.009 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:33:17.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.011+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3c06b4d0 msgr2=0x7f0e3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.011+0000 7f0e62400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c09b4f0 tx=0x7f0e480079b0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 msgr2=0x7f0e5c141620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c141620 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c036040 tx=0x7f0e4c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 shutdown_connections 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0e3c06b4d0 0x7f0e3c06d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0e5c15cdc0 0x7f0e5c158ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e5c0b3640 0x7f0e5c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 --2- 172.21.15.32:0/2814021251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e5c0ae7b0 0x7f0e5c141620 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 >> 172.21.15.32:0/2814021251 conn(0x7f0e5c06afb0 msgr2=0x7f0e5c0b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 shutdown_connections 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.012+0000 7f0e62400640 1 -- 172.21.15.32:0/2814021251 wait complete. 2024-03-06T07:33:17.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:33:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.265+0000 7f3ac3000640 1 Processor -- start 2024-03-06T07:33:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.265+0000 7f3ac3000640 1 -- start start 2024-03-06T07:33:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3ac3000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3abc0ce6e0 0x7f3abc0c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3ac3000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc06dc50 con 0x7f3abc06afb0 2024-03-06T07:33:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3ac3000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc06df30 con 0x7f3abc0c75b0 2024-03-06T07:33:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3ac3000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3abc06e210 con 0x7f3abc0ce6e0 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3abb400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3abc0ce6e0 0x7f3abc0c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3abbe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3abc06afb0 0x7f3abc06b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60264/0 (socket says 172.21.15.32:60264) 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.266+0000 7f3abbe00640 1 -- 172.21.15.32:0/928304962 learned_addr learned my addr 172.21.15.32:0/928304962 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.267+0000 7f3abb400640 1 -- 172.21.15.32:0/928304962 >> v1:172.21.15.39:6789/0 conn(0x7f3abc0c75b0 legacy=0x7f3abc06ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.267+0000 7f3abb400640 1 -- 172.21.15.32:0/928304962 >> v1:172.21.15.32:6789/0 conn(0x7f3abc06afb0 legacy=0x7f3abc06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.267+0000 7f3abb400640 1 -- 172.21.15.32:0/928304962 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc06e4f0 con 0x7f3abc0ce6e0 2024-03-06T07:33:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.268+0000 7f3abb400640 1 --2- 172.21.15.32:0/928304962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc0ce6e0 0x7f3abc0c7070 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40097a0 tx=0x7f3aa402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ded033f6f3752d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.268+0000 7f3abaa00640 1 -- 172.21.15.32:0/928304962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa40315a0 con 0x7f3abc0ce6e0 2024-03-06T07:33:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.268+0000 7f3abaa00640 1 -- 172.21.15.32:0/928304962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa4037070 con 0x7f3abc0ce6e0 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.268+0000 7f3abaa00640 1 -- 172.21.15.32:0/928304962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4031960 con 0x7f3abc0ce6e0 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 -- 172.21.15.32:0/928304962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc0ce6e0 msgr2=0x7f3abc0c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 --2- 172.21.15.32:0/928304962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc0ce6e0 0x7f3abc0c7070 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40097a0 tx=0x7f3aa402b840 comp rx=0 tx=0).stop 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 -- 172.21.15.32:0/928304962 shutdown_connections 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 --2- 172.21.15.32:0/928304962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc0ce6e0 0x7f3abc0c7070 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 -- 172.21.15.32:0/928304962 >> 172.21.15.32:0/928304962 conn(0x7f3abc0b2760 msgr2=0x7f3abc0b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 -- 172.21.15.32:0/928304962 shutdown_connections 2024-03-06T07:33:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.269+0000 7f3ac3000640 1 -- 172.21.15.32:0/928304962 wait complete. 2024-03-06T07:33:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.271+0000 7f3ac3000640 1 Processor -- start 2024-03-06T07:33:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.271+0000 7f3ac3000640 1 -- start start 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3ac3000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 0x7f3abc147a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3ac3000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3abc0c75b0 0x7f3abc0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3ac3000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3abc0ce6e0 0x7f3abc0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3ac3000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc15f230 con 0x7f3abc06afb0 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3ac3000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc15e2a0 con 0x7f3abc0c75b0 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3ac3000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc15f3b0 con 0x7f3abc0ce6e0 2024-03-06T07:33:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3abbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 0x7f3abc147a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3abbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 0x7f3abc147a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42302/0 (socket says 172.21.15.32:42302) 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.272+0000 7f3abbe00640 1 -- 172.21.15.32:0/3522822620 learned_addr learned my addr 172.21.15.32:0/3522822620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abb400640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3abc0c75b0 0x7f3abc0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abbe00640 1 -- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3abc0ce6e0 msgr2=0x7f3abc0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3ac1000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3abc0ce6e0 0x7f3abc0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abbe00640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3abc0ce6e0 0x7f3abc0ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abbe00640 1 -- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3abc0c75b0 msgr2=0x7f3abc0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abbe00640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3abc0c75b0 0x7f3abc0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abbe00640 1 -- 172.21.15.32:0/3522822620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc1483a0 con 0x7f3abc06afb0 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abb400640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3abc0c75b0 0x7f3abc0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3ac1000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3abc0ce6e0 0x7f3abc0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3abbe00640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 0x7f3abc147a60 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400ca70 tx=0x7f3ab400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4017070 con 0x7f3abc06afb0 2024-03-06T07:33:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab400abd0 con 0x7f3abc06afb0 2024-03-06T07:33:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.273+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc148680 con 0x7f3abc06afb0 2024-03-06T07:33:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.274+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4012400 con 0x7f3abc06afb0 2024-03-06T07:33:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.274+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc148c40 con 0x7f3abc06afb0 2024-03-06T07:33:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.275+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc04f330 con 0x7f3abc06afb0 2024-03-06T07:33:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.276+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ab401a070 con 0x7f3abc06afb0 2024-03-06T07:33:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.276+0000 7f3ab3e00640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a9806b4d0 0x7f3a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.277+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3ab4089ce0 con 0x7f3abc06afb0 2024-03-06T07:33:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.277+0000 7f3abb400640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a9806b4d0 0x7f3a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.277+0000 7f3abb400640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a9806b4d0 0x7f3a9806d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4036000 tx=0x7f3aa4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.281+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ab4058dc0 con 0x7f3abc06afb0 2024-03-06T07:33:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.482+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3abc071ee0 con 0x7f3abc06afb0 2024-03-06T07:33:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.488+0000 7f3ab3e00640 1 -- 172.21.15.32:0/3522822620 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3ab40586c0 con 0x7f3abc06afb0 2024-03-06T07:33:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a9806b4d0 msgr2=0x7f3a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a9806b4d0 0x7f3a9806d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4036000 tx=0x7f3aa4008030 comp rx=0 tx=0).stop 2024-03-06T07:33:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 msgr2=0x7f3abc147a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 0x7f3abc147a60 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400ca70 tx=0x7f3ab400cf30 comp rx=0 tx=0).stop 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 shutdown_connections 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a9806b4d0 0x7f3a9806d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3abc0ce6e0 0x7f3abc0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3abc0c75b0 0x7f3abc0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 --2- 172.21.15.32:0/3522822620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3abc06afb0 0x7f3abc147a60 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.491+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 >> 172.21.15.32:0/3522822620 conn(0x7f3abc0b2760 msgr2=0x7f3abc15f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.492+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 shutdown_connections 2024-03-06T07:33:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.492+0000 7f3ac3000640 1 -- 172.21.15.32:0/3522822620 wait complete. 2024-03-06T07:33:17.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:33:17.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.733+0000 7f9bbfe00640 1 Processor -- start 2024-03-06T07:33:17.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.734+0000 7f9bbfe00640 1 -- start start 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.735+0000 7f9bbfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 0x7f9bc006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.735+0000 7f9bbfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9bc006b4f0 0x7f9bc00cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.735+0000 7f9bbfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc0071ff0 con 0x7f9bc00d00f0 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.735+0000 7f9bbfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc006b950 con 0x7f9bc006b4f0 2024-03-06T07:33:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.735+0000 7f9bbfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc0071da0 con 0x7f9bc0145860 2024-03-06T07:33:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.736+0000 7f9bbea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 0x7f9bc006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.736+0000 7f9bbea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 0x7f9bc006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:43620/0 (socket says 172.21.15.32:43620) 2024-03-06T07:33:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.736+0000 7f9bbea00640 1 -- 172.21.15.32:0/425547309 learned_addr learned my addr 172.21.15.32:0/425547309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.736+0000 7f9bbe000640 1 --2- 172.21.15.32:0/425547309 >> v2:172.21.15.39:3300/0 conn(0x7f9bc006b4f0 0x7f9bc00cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.737+0000 7f9bbea00640 1 -- 172.21.15.32:0/425547309 >> v2:172.21.15.39:3300/0 conn(0x7f9bc006b4f0 msgr2=0x7f9bc00cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.737+0000 7f9bbea00640 1 --2- 172.21.15.32:0/425547309 >> v2:172.21.15.39:3300/0 conn(0x7f9bc006b4f0 0x7f9bc00cfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.737+0000 7f9bbea00640 1 -- 172.21.15.32:0/425547309 >> v1:172.21.15.39:6789/0 conn(0x7f9bc00d00f0 legacy=0x7f9bc00ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.737+0000 7f9bbea00640 1 -- 172.21.15.32:0/425547309 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00722d0 con 0x7f9bc0145860 2024-03-06T07:33:17.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.739+0000 7f9bbea00640 1 --2- 172.21.15.32:0/425547309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 0x7f9bc006afb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009870 tx=0x7f9bb002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dd972a446876c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.739+0000 7f9bbd600640 1 -- 172.21.15.32:0/425547309 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0034070 con 0x7f9bc0145860 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbd600640 1 -- 172.21.15.32:0/425547309 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb0030840 con 0x7f9bc0145860 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbd600640 1 -- 172.21.15.32:0/425547309 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0030c10 con 0x7f9bc0145860 2024-03-06T07:33:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 -- 172.21.15.32:0/425547309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 msgr2=0x7f9bc006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/425547309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 0x7f9bc006afb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009870 tx=0x7f9bb002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 -- 172.21.15.32:0/425547309 shutdown_connections 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/425547309 >> v2:172.21.15.39:3300/0 conn(0x7f9bc006b4f0 0x7f9bc00cfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/425547309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc0145860 0x7f9bc006afb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 -- 172.21.15.32:0/425547309 >> 172.21.15.32:0/425547309 conn(0x7f9bc00d0c00 msgr2=0x7f9bc00d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 -- 172.21.15.32:0/425547309 shutdown_connections 2024-03-06T07:33:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.740+0000 7f9bbfe00640 1 -- 172.21.15.32:0/425547309 wait complete. 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.742+0000 7f9bbfe00640 1 Processor -- start 2024-03-06T07:33:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.742+0000 7f9bbfe00640 1 -- start start 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.743+0000 7f9bbfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc006b4f0 0x7f9bc009ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.743+0000 7f9bbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bc00d00f0 0x7f9bc015e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.743+0000 7f9bbfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 0x7f9bc015eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.743+0000 7f9bbfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00d0700 con 0x7f9bc0145860 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.743+0000 7f9bbfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00d0580 con 0x7f9bc00d00f0 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.743+0000 7f9bbfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00afea0 con 0x7f9bc006b4f0 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 0x7f9bc015eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 0x7f9bc015eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42312/0 (socket says 172.21.15.32:42312) 2024-03-06T07:33:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 -- 172.21.15.32:0/4230802936 learned_addr learned my addr 172.21.15.32:0/4230802936 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:17.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbe000640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bc00d00f0 0x7f9bc015e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbea00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc006b4f0 0x7f9bc009ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 -- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc006b4f0 msgr2=0x7f9bc009ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc006b4f0 0x7f9bc009ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 -- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bc00d00f0 msgr2=0x7f9bc015e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bc00d00f0 0x7f9bc015e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.744+0000 7f9bbf400640 1 -- 172.21.15.32:0/4230802936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc01f2d60 con 0x7f9bc0145860 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.745+0000 7f9bbf400640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 0x7f9bc015eeb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800c7d0 tx=0x7f9bb800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.745+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8003b70 con 0x7f9bc0145860 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.745+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb8003d30 con 0x7f9bc0145860 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.745+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb800aa90 con 0x7f9bc0145860 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.745+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc015cbc0 con 0x7f9bc0145860 2024-03-06T07:33:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.745+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc015d180 con 0x7f9bc0145860 2024-03-06T07:33:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.746+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc004f330 con 0x7f9bc0145860 2024-03-06T07:33:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.747+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb800ac30 con 0x7f9bc0145860 2024-03-06T07:33:17.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.748+0000 7f9bb6a00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba006b4d0 0x7f9ba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.748+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9bb8089e40 con 0x7f9bc0145860 2024-03-06T07:33:17.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.748+0000 7f9bbea00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba006b4d0 0x7f9ba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.749+0000 7f9bbea00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba006b4d0 0x7f9ba006d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0036000 tx=0x7f9bb003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.753+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb8058fb0 con 0x7f9bc0145860 2024-03-06T07:33:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.952+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f9bc0144e00 con 0x7f9bc0145860 2024-03-06T07:33:17.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.958+0000 7f9bb6a00640 1 -- 172.21.15.32:0/4230802936 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v25) v1 ==== 108+0+0 (secure 0 0 0) 0x7f9bb80588b0 con 0x7f9bc0145860 2024-03-06T07:33:17.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.961+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba006b4d0 msgr2=0x7f9ba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.961+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba006b4d0 0x7f9ba006d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0036000 tx=0x7f9bb003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:17.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.961+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 msgr2=0x7f9bc015eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.961+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 0x7f9bc015eeb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800c7d0 tx=0x7f9bb800cc90 comp rx=0 tx=0).stop 2024-03-06T07:33:17.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 shutdown_connections 2024-03-06T07:33:17.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba006b4d0 0x7f9ba006d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bc0145860 0x7f9bc015eeb0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bc00d00f0 0x7f9bc015e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 --2- 172.21.15.32:0/4230802936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bc006b4f0 0x7f9bc009ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 >> 172.21.15.32:0/4230802936 conn(0x7f9bc00d0c00 msgr2=0x7f9bc00d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 shutdown_connections 2024-03-06T07:33:17.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:17.962+0000 7f9bbfe00640 1 -- 172.21.15.32:0/4230802936 wait complete. 2024-03-06T07:33:17.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:33:18.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.195+0000 7f4cd2800640 1 Processor -- start 2024-03-06T07:33:18.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.195+0000 7f4cd2800640 1 -- start start 2024-03-06T07:33:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.197+0000 7f4cd2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ccc15efc0 0x7f4ccc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.197+0000 7f4cd2800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ccc0cce90 0x7f4ccc161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.197+0000 7f4cd2800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc1627f0 con 0x7f4ccc162000 2024-03-06T07:33:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.197+0000 7f4cd2800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc15f3a0 con 0x7f4ccc0cce90 2024-03-06T07:33:18.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.197+0000 7f4cd2800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc06d190 con 0x7f4ccc15efc0 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd1e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ccc162000 0x7f4ccc0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53966/0 (socket says 172.21.15.32:53966) 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd1e00640 1 -- 172.21.15.32:0/3870570037 learned_addr learned my addr 172.21.15.32:0/3870570037 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd0a00640 1 --2- 172.21.15.32:0/3870570037 >> v2:172.21.15.32:3300/0 conn(0x7f4ccc0cce90 0x7f4ccc161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd1400640 1 --2- 172.21.15.32:0/3870570037 >> v2:172.21.15.39:3300/0 conn(0x7f4ccc15efc0 0x7f4ccc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4ccbe00640 1 -- 172.21.15.32:0/3870570037 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052286832 0 0) 0x7f4ccc1627f0 con 0x7f4ccc162000 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4ccbe00640 1 -- 172.21.15.32:0/3870570037 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb4003520 con 0x7f4ccc162000 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4ccbe00640 1 -- 172.21.15.32:0/3870570037 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3541820720 0 0) 0x7f4cb4003520 con 0x7f4ccc162000 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd0a00640 1 -- 172.21.15.32:0/3870570037 >> v2:172.21.15.39:3300/0 conn(0x7f4ccc15efc0 msgr2=0x7f4ccc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd0a00640 1 --2- 172.21.15.32:0/3870570037 >> v2:172.21.15.39:3300/0 conn(0x7f4ccc15efc0 0x7f4ccc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd0a00640 1 -- 172.21.15.32:0/3870570037 >> v1:172.21.15.32:6789/0 conn(0x7f4ccc162000 legacy=0x7f4ccc0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.198+0000 7f4cd0a00640 1 -- 172.21.15.32:0/3870570037 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc06c860 con 0x7f4ccc0cce90 2024-03-06T07:33:18.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.200+0000 7f4cd0a00640 1 --2- 172.21.15.32:0/3870570037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc0cce90 0x7f4ccc161ac0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0cd770 tx=0x7f4cb802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f05a4cdfb14381 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.200+0000 7f4ccbe00640 1 -- 172.21.15.32:0/3870570037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8038070 con 0x7f4ccc0cce90 2024-03-06T07:33:18.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.200+0000 7f4ccbe00640 1 -- 172.21.15.32:0/3870570037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8033440 con 0x7f4ccc0cce90 2024-03-06T07:33:18.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.200+0000 7f4ccbe00640 1 -- 172.21.15.32:0/3870570037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4001670 con 0x7f4ccc0cce90 2024-03-06T07:33:18.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 -- 172.21.15.32:0/3870570037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc0cce90 msgr2=0x7f4ccc161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 --2- 172.21.15.32:0/3870570037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc0cce90 0x7f4ccc161ac0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0cd770 tx=0x7f4cb802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 -- 172.21.15.32:0/3870570037 shutdown_connections 2024-03-06T07:33:18.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 --2- 172.21.15.32:0/3870570037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc0cce90 0x7f4ccc161ac0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 --2- 172.21.15.32:0/3870570037 >> v2:172.21.15.39:3300/0 conn(0x7f4ccc15efc0 0x7f4ccc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 -- 172.21.15.32:0/3870570037 >> 172.21.15.32:0/3870570037 conn(0x7f4ccc06b3d0 msgr2=0x7f4ccc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 -- 172.21.15.32:0/3870570037 shutdown_connections 2024-03-06T07:33:18.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.201+0000 7f4cd2800640 1 -- 172.21.15.32:0/3870570037 wait complete. 2024-03-06T07:33:18.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.203+0000 7f4cd2800640 1 Processor -- start 2024-03-06T07:33:18.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.203+0000 7f4cd2800640 1 -- start start 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd2800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ccc0cce90 0x7f4ccc157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ccc15efc0 0x7f4ccc144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 0x7f4ccc1447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd2800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc071c30 con 0x7f4ccc162000 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc06ce20 con 0x7f4ccc15efc0 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd2800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc162470 con 0x7f4ccc0cce90 2024-03-06T07:33:18.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 0x7f4ccc1447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 0x7f4ccc1447d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51846/0 (socket says 172.21.15.32:51846) 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd1e00640 1 -- 172.21.15.32:0/1709700919 learned_addr learned my addr 172.21.15.32:0/1709700919 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.204+0000 7f4cd0a00640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ccc15efc0 0x7f4ccc144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1400640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ccc0cce90 0x7f4ccc157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1e00640 1 -- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ccc0cce90 msgr2=0x7f4ccc157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1e00640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ccc0cce90 0x7f4ccc157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1e00640 1 -- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ccc15efc0 msgr2=0x7f4ccc144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1e00640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ccc15efc0 0x7f4ccc144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1e00640 1 -- 172.21.15.32:0/1709700919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc1f4010 con 0x7f4ccc162000 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1400640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ccc0cce90 0x7f4ccc157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd1e00640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 0x7f4ccc1447d0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c6d0 tx=0x7f4cc400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4005ad0 con 0x7f4ccc162000 2024-03-06T07:33:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4005c70 con 0x7f4ccc162000 2024-03-06T07:33:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc145000 con 0x7f4ccc162000 2024-03-06T07:33:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4007270 con 0x7f4ccc162000 2024-03-06T07:33:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.205+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc1f42d0 con 0x7f4ccc162000 2024-03-06T07:33:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.207+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc04f330 con 0x7f4ccc162000 2024-03-06T07:33:18.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.208+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cc4007410 con 0x7f4ccc162000 2024-03-06T07:33:18.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.209+0000 7f4cc9600640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cac06b4d0 0x7f4cac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.209+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4cc4089d70 con 0x7f4ccc162000 2024-03-06T07:33:18.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.209+0000 7f4cd1400640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cac06b4d0 0x7f4cac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.210+0000 7f4cd1400640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cac06b4d0 0x7f4cac06d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0007520 tx=0x7f4cc000b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.213+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cc4058e50 con 0x7f4ccc162000 2024-03-06T07:33:18.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:18.494+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f4ccc0af2f0 con 0x7f4ccc162000 2024-03-06T07:33:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.019+0000 7f4cc9600640 1 -- 172.21.15.32:0/1709700919 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v244) v1 ==== 140+0+0 (secure 0 0 0) 0x7f4cc4058750 con 0x7f4ccc162000 2024-03-06T07:33:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cac06b4d0 msgr2=0x7f4cac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cac06b4d0 0x7f4cac06d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0007520 tx=0x7f4cc000b040 comp rx=0 tx=0).stop 2024-03-06T07:33:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 msgr2=0x7f4ccc1447d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 0x7f4ccc1447d0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c6d0 tx=0x7f4cc400cb90 comp rx=0 tx=0).stop 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 shutdown_connections 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cac06b4d0 0x7f4cac06d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc162000 0x7f4ccc1447d0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ccc15efc0 0x7f4ccc144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 --2- 172.21.15.32:0/1709700919 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ccc0cce90 0x7f4ccc157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.023+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 >> 172.21.15.32:0/1709700919 conn(0x7f4ccc06b3d0 msgr2=0x7f4ccc15e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.024+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 shutdown_connections 2024-03-06T07:33:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.024+0000 7f4cd2800640 1 -- 172.21.15.32:0/1709700919 wait complete. 2024-03-06T07:33:19.049 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:33:19.049 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:33:19.049 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:33:19.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.271+0000 7fb0afe00640 1 Processor -- start 2024-03-06T07:33:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.272+0000 7fb0afe00640 1 -- start start 2024-03-06T07:33:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.273+0000 7fb0afe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0b00cd060 0x7fb0b01667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.273+0000 7fb0afe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b00714d0 con 0x7fb0b0070f10 2024-03-06T07:33:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.273+0000 7fb0afe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b00717b0 con 0x7fb0b0145300 2024-03-06T07:33:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.273+0000 7fb0afe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b0071a90 con 0x7fb0b00cd060 2024-03-06T07:33:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.274+0000 7fb0a6000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb0b0070f10 0x7fb0b00cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53976/0 (socket says 172.21.15.32:53976) 2024-03-06T07:33:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.274+0000 7fb0a6000640 1 -- 172.21.15.32:0/1249673282 learned_addr learned my addr 172.21.15.32:0/1249673282 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.274+0000 7fb0af400640 1 --2- 172.21.15.32:0/1249673282 >> v2:172.21.15.39:3300/0 conn(0x7fb0b00cd060 0x7fb0b01667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.274+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923791658 0 0) 0x7fb0b00714d0 con 0x7fb0b0070f10 2024-03-06T07:33:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.274+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb094003520 con 0x7fb0b0070f10 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2027268460 0 0) 0x7fb094003520 con 0x7fb0b0070f10 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 >> v2:172.21.15.39:3300/0 conn(0x7fb0b00cd060 msgr2=0x7fb0b01667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 --2- 172.21.15.32:0/1249673282 >> v2:172.21.15.39:3300/0 conn(0x7fb0b00cd060 0x7fb0b01667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 >> v1:172.21.15.39:6789/0 conn(0x7fb0b0145300 legacy=0x7fb0b00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0071d70 con 0x7fb0b0070f10 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb098001670 con 0x7fb0b0070f10 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 >> v1:172.21.15.32:6789/0 conn(0x7fb0b0070f10 legacy=0x7fb0b00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb094003a00 0x7fb094003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094004800 0x7fb094004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb094005210 0x7fb09400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0071a90 con 0x7fb094005210 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09400c160 con 0x7fb094003a00 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09400c360 con 0x7fb094004800 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.275+0000 7fb0af400640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb094005210 0x7fb09400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0aea00640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb094003a00 0x7fb094003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0a6000640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094004800 0x7fb094004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0af400640 1 -- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094004800 msgr2=0x7fb094004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0af400640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094004800 0x7fb094004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0af400640 1 -- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb094003a00 msgr2=0x7fb094003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0af400640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb094003a00 0x7fb094003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0af400640 1 -- 172.21.15.32:0/1249673282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094003520 con 0x7fb094005210 2024-03-06T07:33:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0aea00640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb094003a00 0x7fb094003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.276+0000 7fb0a6000640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094004800 0x7fb094004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.277+0000 7fb0af400640 1 -- 172.21.15.32:0/1249673282 reap_dead start 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.277+0000 7fb0af400640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb094005210 0x7fb09400b9c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0072090 tx=0x7fb094005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7df92c20ebc7be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.277+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8039070 con 0x7fb094005210 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.277+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a802c660 con 0x7fb094005210 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.277+0000 7fb0ae000640 1 -- 172.21.15.32:0/1249673282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8009670 con 0x7fb094005210 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 -- 172.21.15.32:0/1249673282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb094005210 msgr2=0x7fb09400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb094005210 0x7fb09400b9c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0072090 tx=0x7fb094005830 comp rx=0 tx=0).stop 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 -- 172.21.15.32:0/1249673282 shutdown_connections 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 --2- 172.21.15.32:0/1249673282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb094005210 0x7fb09400b9c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 -- 172.21.15.32:0/1249673282 >> 172.21.15.32:0/1249673282 conn(0x7fb0b006b3d0 msgr2=0x7fb0b006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 -- 172.21.15.32:0/1249673282 shutdown_connections 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.278+0000 7fb0afe00640 1 -- 172.21.15.32:0/1249673282 wait complete. 2024-03-06T07:33:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.280+0000 7fb0afe00640 1 Processor -- start 2024-03-06T07:33:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.280+0000 7fb0afe00640 1 -- start start 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094005210 0x7fb0b01e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0b013d0a0 0x7fb0b013b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 0x7fb0b01412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0afe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00ccad0 con 0x7fb0b014bc80 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0afe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00cc950 con 0x7fb0b013d0a0 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0afe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00ccc50 con 0x7fb094005210 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 0x7fb0b01412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 0x7fb0b01412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51868/0 (socket says 172.21.15.32:51868) 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.281+0000 7fb0af400640 1 -- 172.21.15.32:0/114684708 learned_addr learned my addr 172.21.15.32:0/114684708 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0aea00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094005210 0x7fb0b01e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0af400640 1 -- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094005210 msgr2=0x7fb0b01e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0af400640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094005210 0x7fb0b01e4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0a6000640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0b013d0a0 0x7fb0b013b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0af400640 1 -- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0b013d0a0 msgr2=0x7fb0b013b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0af400640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0b013d0a0 0x7fb0b013b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0af400640 1 -- 172.21.15.32:0/114684708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b013d5a0 con 0x7fb0b014bc80 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0a6000640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0b013d0a0 0x7fb0b013b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0af400640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 0x7fb0b01412a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009b60 tx=0x7fb0a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8046070 con 0x7fb0b014bc80 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a803e680 con 0x7fb0b014bc80 2024-03-06T07:33:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8039040 con 0x7fb0b014bc80 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.282+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b013d880 con 0x7fb0b014bc80 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.283+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b013de40 con 0x7fb0b014bc80 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.284+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b015f210 con 0x7fb0b014bc80 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.284+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0a8049070 con 0x7fb0b014bc80 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.285+0000 7fb0a7400640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb08c06b500 0x7fb08c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.285+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb0a80aab80 con 0x7fb0b014bc80 2024-03-06T07:33:19.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.286+0000 7fb0aea00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb08c06b500 0x7fb08c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.286+0000 7fb0aea00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb08c06b500 0x7fb08c06d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0001cc0 tx=0x7fb0a0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.290+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0b015f210 con 0x7fb0b014bc80 2024-03-06T07:33:19.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.494+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0b004f330 con 0x7fb0b014bc80 2024-03-06T07:33:19.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.495+0000 7fb0a7400640 1 -- 172.21.15.32:0/114684708 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+12913 (secure 0 0 0) 0x7fb0a80797c0 con 0x7fb0b014bc80 2024-03-06T07:33:19.498 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:19.498 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":73,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:16.996186+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6837/2323044149":"2024-03-07T07:33:13.969977+0000","172.21.15.32:6836/2323044149":"2024-03-07T07:33:13.969977+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:19.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb08c06b500 msgr2=0x7fb08c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb08c06b500 0x7fb08c06d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0001cc0 tx=0x7fb0a0009040 comp rx=0 tx=0).stop 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 msgr2=0x7fb0b01412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 0x7fb0b01412a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009b60 tx=0x7fb0a8004730 comp rx=0 tx=0).stop 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 shutdown_connections 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb08c06b500 0x7fb08c06d9c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b014bc80 0x7fb0b01412a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0b013d0a0 0x7fb0b013b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 --2- 172.21.15.32:0/114684708 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb094005210 0x7fb0b01e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 >> 172.21.15.32:0/114684708 conn(0x7fb0b006b3d0 msgr2=0x7fb0b00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 shutdown_connections 2024-03-06T07:33:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.498+0000 7fb0afe00640 1 -- 172.21.15.32:0/114684708 wait complete. 2024-03-06T07:33:19.518 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:19.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:19.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:19.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.730+0000 7f9f41000640 1 Processor -- start 2024-03-06T07:33:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.731+0000 7f9f41000640 1 -- start start 2024-03-06T07:33:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.732+0000 7f9f41000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f3c15b5f0 0x7f9f3c0aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.732+0000 7f9f41000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0cdb50 con 0x7f9f3c15ed90 2024-03-06T07:33:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.732+0000 7f9f41000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0cde30 con 0x7f9f3c0af4b0 2024-03-06T07:33:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.732+0000 7f9f41000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3c0cd1e0 con 0x7f9f3c15b5f0 2024-03-06T07:33:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f39e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9f3c15ed90 0x7f9f3c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53990/0 (socket says 172.21.15.32:53990) 2024-03-06T07:33:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f39e00640 1 -- 172.21.15.32:0/3997900394 learned_addr learned my addr 172.21.15.32:0/3997900394 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863911325 0 0) 0x7f9f3c0cdb50 con 0x7f9f3c15ed90 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20003520 con 0x7f9f3c15ed90 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f39400640 1 --2- 172.21.15.32:0/3997900394 >> v2:172.21.15.39:3300/0 conn(0x7f9f3c15b5f0 0x7f9f3c0aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145350398 0 0) 0x7f9f20003520 con 0x7f9f3c15ed90 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 >> v2:172.21.15.39:3300/0 conn(0x7f9f3c15b5f0 msgr2=0x7f9f3c0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f38a00640 1 --2- 172.21.15.32:0/3997900394 >> v2:172.21.15.39:3300/0 conn(0x7f9f3c15b5f0 0x7f9f3c0aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.733+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 >> v1:172.21.15.39:6789/0 conn(0x7f9f3c0af4b0 legacy=0x7f9f3c0cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0ce110 con 0x7f9f3c15ed90 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f9f24001670 con 0x7f9f3c15ed90 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 >> v1:172.21.15.32:6789/0 conn(0x7f9f3c15ed90 legacy=0x7f9f3c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f20003a00 0x7f9f20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f20005210 0x7f9f2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0cd1e0 con 0x7f9f20004800 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2000c160 con 0x7f9f20005210 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2000c360 con 0x7f9f20003a00 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f39400640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f3a800640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f20005210 0x7f9f2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.734+0000 7f9f39e00640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f20003a00 0x7f9f20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.735+0000 7f9f39400640 1 -- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f20003a00 msgr2=0x7f9f20003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.735+0000 7f9f39400640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f20003a00 0x7f9f20003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.735+0000 7f9f39400640 1 -- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f20005210 msgr2=0x7f9f2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.735+0000 7f9f39400640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f20005210 0x7f9f2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.735+0000 7f9f39400640 1 -- 172.21.15.32:0/3997900394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20003520 con 0x7f9f20004800 2024-03-06T07:33:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.735+0000 7f9f3a800640 1 -- 172.21.15.32:0/3997900394 reap_dead start 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.736+0000 7f9f39400640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c007360 tx=0x7f9f20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7b8f79cc235585 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.736+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c039070 con 0x7f9f20004800 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.736+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c02ccd0 con 0x7f9f20004800 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.736+0000 7f9f38a00640 1 -- 172.21.15.32:0/3997900394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c009be0 con 0x7f9f20004800 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 -- 172.21.15.32:0/3997900394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 msgr2=0x7f9f20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c007360 tx=0x7f9f20005830 comp rx=0 tx=0).stop 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 -- 172.21.15.32:0/3997900394 shutdown_connections 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 --2- 172.21.15.32:0/3997900394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f20004c80 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 -- 172.21.15.32:0/3997900394 >> 172.21.15.32:0/3997900394 conn(0x7f9f3c06b3d0 msgr2=0x7f9f3c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 -- 172.21.15.32:0/3997900394 shutdown_connections 2024-03-06T07:33:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.737+0000 7f9f41000640 1 -- 172.21.15.32:0/3997900394 wait complete. 2024-03-06T07:33:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.739+0000 7f9f41000640 1 Processor -- start 2024-03-06T07:33:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.739+0000 7f9f41000640 1 -- start start 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f41000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f3c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f41000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f3c14e810 0x7f9f3c143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f41000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c09b7c0 0x7f9f3c1430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f41000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0aef70 con 0x7f9f20004800 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f41000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0ce6d0 con 0x7f9f3c09b7c0 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f41000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0af0f0 con 0x7f9f3c14e810 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f3c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f3c09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51882/0 (socket says 172.21.15.32:51882) 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 -- 172.21.15.32:0/4247594784 learned_addr learned my addr 172.21.15.32:0/4247594784 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 -- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f3c14e810 msgr2=0x7f9f3c143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f3a800640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c09b7c0 0x7f9f3c1430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39400640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f3c14e810 0x7f9f3c143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f3c14e810 0x7f9f3c143ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 -- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c09b7c0 msgr2=0x7f9f3c1430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.740+0000 7f9f39e00640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c09b7c0 0x7f9f3c1430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f39e00640 1 -- 172.21.15.32:0/4247594784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0b3430 con 0x7f9f20004800 2024-03-06T07:33:19.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f39e00640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f3c09aa10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400e700 tx=0x7f9f2400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2400c6c0 con 0x7f9f20004800 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f24018070 con 0x7f9f20004800 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2400caa0 con 0x7f9f20004800 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c0b3710 con 0x7f9f20004800 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.741+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c0b3cd0 con 0x7f9f20004800 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.743+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c04f330 con 0x7f9f20004800 2024-03-06T07:33:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.744+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f2400cc40 con 0x7f9f20004800 2024-03-06T07:33:19.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.744+0000 7f9f32000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f1806b4d0 0x7f9f1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.745+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9f24089d50 con 0x7f9f20004800 2024-03-06T07:33:19.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.745+0000 7f9f39400640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f1806b4d0 0x7f9f1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.745+0000 7f9f39400640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f1806b4d0 0x7f9f1806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c036000 tx=0x7f9f2c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.749+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f2408d030 con 0x7f9f20004800 2024-03-06T07:33:19.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.948+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f3c06cc20 con 0x7f9f20004800 2024-03-06T07:33:19.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.949+0000 7f9f32000640 1 -- 172.21.15.32:0/4247594784 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f24058e40 con 0x7f9f20004800 2024-03-06T07:33:19.952 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:33:19.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f1806b4d0 msgr2=0x7f9f1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f1806b4d0 0x7f9f1806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c036000 tx=0x7f9f2c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 msgr2=0x7f9f3c09aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f3c09aa10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400e700 tx=0x7f9f2400ebc0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 shutdown_connections 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f1806b4d0 0x7f9f1806d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c09b7c0 0x7f9f3c1430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f3c14e810 0x7f9f3c143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 --2- 172.21.15.32:0/4247594784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f20004800 0x7f9f3c09aa10 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 >> 172.21.15.32:0/4247594784 conn(0x7f9f3c06b3d0 msgr2=0x7f9f3c06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 shutdown_connections 2024-03-06T07:33:19.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:19.952+0000 7f9f41000640 1 -- 172.21.15.32:0/4247594784 wait complete. 2024-03-06T07:33:19.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:33:20.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.196+0000 7f6ba2000640 1 Processor -- start 2024-03-06T07:33:20.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.197+0000 7f6ba2000640 1 -- start start 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.198+0000 7f6ba2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6b9c070f10 0x7f6b9c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.198+0000 7f6ba2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 0x7f6b9c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.198+0000 7f6ba2000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c0714d0 con 0x7f6b9c145300 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.198+0000 7f6ba2000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c0cd300 con 0x7f6b9c070f10 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.198+0000 7f6ba2000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0717b0 con 0x7f6b9c0cce90 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.199+0000 7f6ba0c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6b9c145300 0x7f6b9c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54000/0 (socket says 172.21.15.32:54000) 2024-03-06T07:33:20.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.199+0000 7f6ba0c00640 1 -- 172.21.15.32:0/2405532127 learned_addr learned my addr 172.21.15.32:0/2405532127 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:20.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.199+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81426740 0 0) 0x7f6b9c0714d0 con 0x7f6b9c145300 2024-03-06T07:33:20.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.199+0000 7f6ba1600640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 0x7f6b9c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.199+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b84003520 con 0x7f6b9c145300 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.199+0000 7f6b9be00640 1 --2- 172.21.15.32:0/2405532127 >> v2:172.21.15.39:3300/0 conn(0x7f6b9c070f10 0x7f6b9c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787144098 0 0) 0x7f6b84003520 con 0x7f6b9c145300 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 msgr2=0x7f6b9c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 0x7f6b9c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 >> v2:172.21.15.39:3300/0 conn(0x7f6b9c070f10 msgr2=0x7f6b9c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 --2- 172.21.15.32:0/2405532127 >> v2:172.21.15.39:3300/0 conn(0x7f6b9c070f10 0x7f6b9c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c071a90 con 0x7f6b9c145300 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6b90001670 con 0x7f6b9c145300 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 >> v1:172.21.15.32:6789/0 conn(0x7f6b9c145300 legacy=0x7f6b9c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 0x7f6b840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b84004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b84004d40 0x7f6b840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0cd300 con 0x7f6b84003f80 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0717b0 con 0x7f6b84004d40 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8400bb20 con 0x7f6b9c0cce90 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6b9be00640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b84004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.200+0000 7f6ba0c00640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 0x7f6b840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6b9be00640 1 -- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 msgr2=0x7f6b840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6b9be00640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c0cce90 0x7f6b840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6b9be00640 1 -- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b84004d40 msgr2=0x7f6b840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6ba1600640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b84004d40 0x7f6b840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6b9be00640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b84004d40 0x7f6b840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6b9be00640 1 -- 172.21.15.32:0/2405532127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b84003520 con 0x7f6b84003f80 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.201+0000 7f6ba1600640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b84004d40 0x7f6b840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.202+0000 7f6b9be00640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b84004800 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800ba30 tx=0x7f6b84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0456fd45c4e80de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.202+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88036020 con 0x7f6b84003f80 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.202+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b88032680 con 0x7f6b84003f80 2024-03-06T07:33:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.202+0000 7f6b9b400640 1 -- 172.21.15.32:0/2405532127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8800b600 con 0x7f6b84003f80 2024-03-06T07:33:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba2000640 1 -- 172.21.15.32:0/2405532127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 msgr2=0x7f6b84004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba2000640 1 --2- 172.21.15.32:0/2405532127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b84004800 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800ba30 tx=0x7f6b84005830 comp rx=0 tx=0).stop 2024-03-06T07:33:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba1600640 1 -- 172.21.15.32:0/2405532127 reap_dead start 2024-03-06T07:33:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba2000640 1 -- 172.21.15.32:0/2405532127 shutdown_connections 2024-03-06T07:33:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba2000640 1 -- 172.21.15.32:0/2405532127 >> 172.21.15.32:0/2405532127 conn(0x7f6b9c06b3d0 msgr2=0x7f6b9c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba2000640 1 -- 172.21.15.32:0/2405532127 shutdown_connections 2024-03-06T07:33:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.203+0000 7f6ba2000640 1 -- 172.21.15.32:0/2405532127 wait complete. 2024-03-06T07:33:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.205+0000 7f6ba2000640 1 Processor -- start 2024-03-06T07:33:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.205+0000 7f6ba2000640 1 -- start start 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b9c13a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b9c0ae920 0x7f6b9c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c1405d0 0x7f6b9c140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba2000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c15c570 con 0x7f6b84003f80 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba2000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c1456e0 con 0x7f6b9c0ae920 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba2000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c15c6f0 con 0x7f6b9c1405d0 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b9c13a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b9c13a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51910/0 (socket says 172.21.15.32:51910) 2024-03-06T07:33:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.206+0000 7f6ba0c00640 1 -- 172.21.15.32:0/3447833578 learned_addr learned my addr 172.21.15.32:0/3447833578 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba1600640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c1405d0 0x7f6b9c140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba0c00640 1 -- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c1405d0 msgr2=0x7f6b9c140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba0c00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c1405d0 0x7f6b9c140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6b9be00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b9c0ae920 0x7f6b9c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba0c00640 1 -- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b9c0ae920 msgr2=0x7f6b9c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba0c00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b9c0ae920 0x7f6b9c0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba0c00640 1 -- 172.21.15.32:0/3447833578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c141610 con 0x7f6b84003f80 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba1600640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c1405d0 0x7f6b9c140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba0c00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b9c13a9e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6b90000c00 tx=0x7f6b9000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000ee60 con 0x7f6b84003f80 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b90018070 con 0x7f6b84003f80 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000ca80 con 0x7f6b84003f80 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c141330 con 0x7f6b84003f80 2024-03-06T07:33:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.207+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c13b5f0 con 0x7f6b84003f80 2024-03-06T07:33:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.209+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9c04f330 con 0x7f6b84003f80 2024-03-06T07:33:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.209+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b9000cc20 con 0x7f6b84003f80 2024-03-06T07:33:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.210+0000 7f6b98c00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b7c06b4d0 0x7f6b7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.210+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6b90089db0 con 0x7f6b84003f80 2024-03-06T07:33:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.211+0000 7f6b9be00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b7c06b4d0 0x7f6b7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.211+0000 7f6b9be00640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b7c06b4d0 0x7f6b7c06d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800ba00 tx=0x7f6b88030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.215+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b9008e070 con 0x7f6b84003f80 2024-03-06T07:33:20.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:20.490+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f6b9c15e460 con 0x7f6b84003f80 2024-03-06T07:33:21.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.025+0000 7f6b98c00640 1 -- 172.21.15.32:0/3447833578 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v246) v1 ==== 135+0+0 (secure 0 0 0) 0x7f6b9c141610 con 0x7f6b84003f80 2024-03-06T07:33:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b7c06b4d0 msgr2=0x7f6b7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b7c06b4d0 0x7f6b7c06d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800ba00 tx=0x7f6b88030000 comp rx=0 tx=0).stop 2024-03-06T07:33:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 msgr2=0x7f6b9c13a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b9c13a9e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6b90000c00 tx=0x7f6b9000ec10 comp rx=0 tx=0).stop 2024-03-06T07:33:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 shutdown_connections 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b7c06b4d0 0x7f6b7c06d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b9c1405d0 0x7f6b9c140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b9c0ae920 0x7f6b9c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.028+0000 7f6ba2000640 1 --2- 172.21.15.32:0/3447833578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b84003f80 0x7f6b9c13a9e0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.029+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 >> 172.21.15.32:0/3447833578 conn(0x7f6b9c06b3d0 msgr2=0x7f6b9c15ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.029+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 shutdown_connections 2024-03-06T07:33:21.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.029+0000 7f6ba2000640 1 -- 172.21.15.32:0/3447833578 wait complete. 2024-03-06T07:33:21.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:33:21.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.283+0000 7f15a8a00640 1 Processor -- start 2024-03-06T07:33:21.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.283+0000 7f15a8a00640 1 -- start start 2024-03-06T07:33:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.285+0000 7f15a8a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.285+0000 7f15a8a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f15a40b2250 0x7f15a40b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.285+0000 7f15a8a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a415fe40 con 0x7f15a415cc90 2024-03-06T07:33:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.285+0000 7f15a8a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a4160120 con 0x7f15a40b2250 2024-03-06T07:33:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.285+0000 7f15a8a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4160400 con 0x7f15a4145430 2024-03-06T07:33:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.286+0000 7f15a3400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.286+0000 7f15a2a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f15a415cc90 0x7f15a40b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:55634/0 (socket says 172.21.15.32:55634) 2024-03-06T07:33:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.286+0000 7f15a2a00640 1 -- 172.21.15.32:0/1600238378 learned_addr learned my addr 172.21.15.32:0/1600238378 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.286+0000 7f15a3e00640 1 --2- 172.21.15.32:0/1600238378 >> v2:172.21.15.39:3300/0 conn(0x7f15a40b2250 0x7f15a40b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.286+0000 7f15a3e00640 1 -- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 msgr2=0x7f15a415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.286+0000 7f15a3e00640 1 --2- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.287+0000 7f15a3e00640 1 -- 172.21.15.32:0/1600238378 >> v1:172.21.15.39:6789/0 conn(0x7f15a415cc90 legacy=0x7f15a40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.287+0000 7f15a3e00640 1 -- 172.21.15.32:0/1600238378 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a41606e0 con 0x7f15a40b2250 2024-03-06T07:33:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.288+0000 7f15a3e00640 1 --2- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a40b2250 0x7f15a40b2fd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f159c009d90 tx=0x7f159c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21af1da2d8c95e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.289+0000 7f15a2000640 1 -- 172.21.15.32:0/1600238378 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c039070 con 0x7f15a40b2250 2024-03-06T07:33:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.289+0000 7f15a2000640 1 -- 172.21.15.32:0/1600238378 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c02cc50 con 0x7f15a40b2250 2024-03-06T07:33:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.289+0000 7f15a2000640 1 -- 172.21.15.32:0/1600238378 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c034440 con 0x7f15a40b2250 2024-03-06T07:33:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 -- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a40b2250 msgr2=0x7f15a40b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a40b2250 0x7f15a40b2fd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f159c009d90 tx=0x7f159c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 -- 172.21.15.32:0/1600238378 shutdown_connections 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a40b2250 0x7f15a40b2fd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1600238378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 -- 172.21.15.32:0/1600238378 >> 172.21.15.32:0/1600238378 conn(0x7f15a406b3a0 msgr2=0x7f15a406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 -- 172.21.15.32:0/1600238378 shutdown_connections 2024-03-06T07:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.290+0000 7f15a8a00640 1 -- 172.21.15.32:0/1600238378 wait complete. 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.293+0000 7f15a8a00640 1 Processor -- start 2024-03-06T07:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.293+0000 7f15a8a00640 1 -- start start 2024-03-06T07:33:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 0x7f15a41f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a8a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a40abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a415cc90 0x7f15a41440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a8a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4145990 con 0x7f15a40b2250 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4145810 con 0x7f15a415cc90 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a8a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cde00 con 0x7f15a4145430 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 0x7f15a41f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 0x7f15a41f4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51922/0 (socket says 172.21.15.32:51922) 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.294+0000 7f15a3400640 1 -- 172.21.15.32:0/1998730474 learned_addr learned my addr 172.21.15.32:0/1998730474 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3e00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a415cc90 0x7f15a41440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a2a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a40abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3400640 1 -- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 msgr2=0x7f15a40abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3400640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a40abdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3400640 1 -- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a415cc90 msgr2=0x7f15a41440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3400640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a415cc90 0x7f15a41440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3400640 1 -- 172.21.15.32:0/1998730474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40ac7b0 con 0x7f15a40b2250 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.295+0000 7f15a3400640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 0x7f15a41f4c00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f159400e930 tx=0x7f159400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.296+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594017070 con 0x7f15a40b2250 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.296+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400ca00 con 0x7f15a40b2250 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.296+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400cdf0 con 0x7f15a40b2250 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.296+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40aca90 con 0x7f15a40b2250 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.296+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40ad050 con 0x7f15a40b2250 2024-03-06T07:33:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.297+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a404f330 con 0x7f15a40b2250 2024-03-06T07:33:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.298+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f159401a070 con 0x7f15a40b2250 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.299+0000 7f159b400640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f158406b500 0x7f158406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.299+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f159408a310 con 0x7f15a40b2250 2024-03-06T07:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.299+0000 7f15a2a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f158406b500 0x7f158406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.300+0000 7f15a2a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f158406b500 0x7f158406d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ad9c0 tx=0x7f1590009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.306+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1594058e60 con 0x7f15a40b2250 2024-03-06T07:33:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.515+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f15a406e240 con 0x7f15a40b2250 2024-03-06T07:33:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.520+0000 7f159b400640 1 -- 172.21.15.32:0/1998730474 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f159405de20 con 0x7f15a40b2250 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f158406b500 msgr2=0x7f158406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f158406b500 0x7f158406d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ad9c0 tx=0x7f1590009040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 msgr2=0x7f15a41f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 0x7f15a41f4c00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f159400e930 tx=0x7f159400edf0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 shutdown_connections 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f158406b500 0x7f158406d9c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15a415cc90 0x7f15a41440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15a4145430 0x7f15a40abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 --2- 172.21.15.32:0/1998730474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15a40b2250 0x7f15a41f4c00 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.522+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 >> 172.21.15.32:0/1998730474 conn(0x7f15a406b3a0 msgr2=0x7f15a415dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.523+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 shutdown_connections 2024-03-06T07:33:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:21.523+0000 7f15a8a00640 1 -- 172.21.15.32:0/1998730474 wait complete. 2024-03-06T07:33:21.546 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:21.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:21.663 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:21.789 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:21.789 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:21.789 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:21.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:21.850 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:33:21.850 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:21.911 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:21.985 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:21.985 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:22.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:22.047 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:33:22.047 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:22.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:22.108 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:33:22.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:22.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.333+0000 7f050e600640 1 Processor -- start 2024-03-06T07:33:22.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.333+0000 7f050e600640 1 -- start start 2024-03-06T07:33:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f050e600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f05080b3c90 0x7f05080cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f050e600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05080cf5c0 0x7f05080b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f050e600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05080b45b0 con 0x7f050806cb10 2024-03-06T07:33:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f050e600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05080ce160 con 0x7f05080cf5c0 2024-03-06T07:33:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f050e600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05080b4860 con 0x7f05080b3c90 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f0506a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05080cf5c0 0x7f05080b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f0507e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f050806cb10 0x7f050806cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54014/0 (socket says 172.21.15.32:54014) 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.335+0000 7f0507e00640 1 -- 172.21.15.32:0/347975683 learned_addr learned my addr 172.21.15.32:0/347975683 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.336+0000 7f0507400640 1 --2- 172.21.15.32:0/347975683 >> v2:172.21.15.39:3300/0 conn(0x7f05080b3c90 0x7f05080cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.336+0000 7f0506a00640 1 -- 172.21.15.32:0/347975683 >> v2:172.21.15.39:3300/0 conn(0x7f05080b3c90 msgr2=0x7f05080cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.336+0000 7f0506a00640 1 --2- 172.21.15.32:0/347975683 >> v2:172.21.15.39:3300/0 conn(0x7f05080b3c90 0x7f05080cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.336+0000 7f0506a00640 1 -- 172.21.15.32:0/347975683 >> v1:172.21.15.32:6789/0 conn(0x7f050806cb10 legacy=0x7f050806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.336+0000 7f0506a00640 1 -- 172.21.15.32:0/347975683 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05080b4b40 con 0x7f05080cf5c0 2024-03-06T07:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.336+0000 7f0506000640 1 -- 172.21.15.32:0/347975683 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696105556 0 0) 0x7f05080b45b0 con 0x7f050806cb10 2024-03-06T07:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.337+0000 7f0506a00640 1 --2- 172.21.15.32:0/347975683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080b4070 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0508070800 tx=0x7f04f402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a940b15d0798e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.337+0000 7f0506000640 1 -- 172.21.15.32:0/347975683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4034070 con 0x7f05080cf5c0 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.337+0000 7f0506000640 1 -- 172.21.15.32:0/347975683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f4030610 con 0x7f05080cf5c0 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.337+0000 7f0506000640 1 -- 172.21.15.32:0/347975683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40309e0 con 0x7f05080cf5c0 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 -- 172.21.15.32:0/347975683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 msgr2=0x7f05080b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 --2- 172.21.15.32:0/347975683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080b4070 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0508070800 tx=0x7f04f402b960 comp rx=0 tx=0).stop 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 -- 172.21.15.32:0/347975683 shutdown_connections 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 --2- 172.21.15.32:0/347975683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080b4070 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 --2- 172.21.15.32:0/347975683 >> v2:172.21.15.39:3300/0 conn(0x7f05080b3c90 0x7f05080cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 -- 172.21.15.32:0/347975683 >> 172.21.15.32:0/347975683 conn(0x7f050806afb0 msgr2=0x7f050806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 -- 172.21.15.32:0/347975683 shutdown_connections 2024-03-06T07:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.338+0000 7f050e600640 1 -- 172.21.15.32:0/347975683 wait complete. 2024-03-06T07:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.340+0000 7f050e600640 1 Processor -- start 2024-03-06T07:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.340+0000 7f050e600640 1 -- start start 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.341+0000 7f050e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050806cb10 0x7f05081e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.341+0000 7f050e600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05080b3c90 0x7f050809d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.341+0000 7f050e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.341+0000 7f050e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050815b340 con 0x7f05080cf5c0 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.341+0000 7f050e600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05080b5100 con 0x7f050806cb10 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.341+0000 7f050e600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050806d690 con 0x7f05080b3c90 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51952/0 (socket says 172.21.15.32:51952) 2024-03-06T07:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 -- 172.21.15.32:0/2676586163 learned_addr learned my addr 172.21.15.32:0/2676586163 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507400640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050806cb10 0x7f05081e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0506a00640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05080b3c90 0x7f050809d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 -- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05080b3c90 msgr2=0x7f050809d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05080b3c90 0x7f050809d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 -- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050806cb10 msgr2=0x7f05081e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050806cb10 0x7f05081e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 -- 172.21.15.32:0/2676586163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05081e0840 con 0x7f05080cf5c0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0506a00640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05080b3c90 0x7f050809d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.342+0000 7f0507e00640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080cbf40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f050000c7a0 tx=0x7f050000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.343+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500018070 con 0x7f05080cf5c0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.343+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050000a640 con 0x7f05080cf5c0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.343+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050000aab0 con 0x7f05080cf5c0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.343+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05081e0b20 con 0x7f05080cf5c0 2024-03-06T07:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.343+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05081e10e0 con 0x7f05080cf5c0 2024-03-06T07:33:22.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.344+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0508069510 con 0x7f05080cf5c0 2024-03-06T07:33:22.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.345+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f050000ac50 con 0x7f05080cf5c0 2024-03-06T07:33:22.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.346+0000 7f04ff400640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f04e806b500 0x7f04e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.346+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0500089490 con 0x7f05080cf5c0 2024-03-06T07:33:22.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.346+0000 7f0507400640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f04e806b500 0x7f04e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.347+0000 7f0507400640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f04e806b500 0x7f04e806d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f04f8006fd0 tx=0x7f04f8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.352+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0500058600 con 0x7f05080cf5c0 2024-03-06T07:33:22.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.593+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f050804f3a0 con 0x7f05080cf5c0 2024-03-06T07:33:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.594+0000 7f04ff400640 1 -- 172.21.15.32:0/2676586163 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f0500057f00 con 0x7f05080cf5c0 2024-03-06T07:33:22.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:22.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5458,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3165963336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3165963336},{"type":"v1","addr":"172.21.15.39:6834","nonce":3165963336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1594629063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1594629063},{"type":"v1","addr":"172.21.15.39:6835","nonce":1594629063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1863016849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1863016849},{"type":"v1","addr":"172.21.15.32:6835","nonce":1863016849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/36813811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":36813811},{"type":"v1","addr":"172.21.15.32:6837","nonce":36813811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[]} 2024-03-06T07:33:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 45 2024-03-06T07:33:22.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f04e806b500 msgr2=0x7f04e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f04e806b500 0x7f04e806d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f04f8006fd0 tx=0x7f04f8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:22.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 msgr2=0x7f05080cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080cbf40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f050000c7a0 tx=0x7f050000cc60 comp rx=0 tx=0).stop 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 shutdown_connections 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f04e806b500 0x7f04e806d9c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05080cf5c0 0x7f05080cbf40 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f05080b3c90 0x7f050809d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 --2- 172.21.15.32:0/2676586163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050806cb10 0x7f05081e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 >> 172.21.15.32:0/2676586163 conn(0x7f050806afb0 msgr2=0x7f05080ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 shutdown_connections 2024-03-06T07:33:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:22.597+0000 7f050e600640 1 -- 172.21.15.32:0/2676586163 wait complete. 2024-03-06T07:33:22.621 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:33:22.621 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:33:22.621 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:22.621 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:22.621 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:33:22.621 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:33:22.622 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:22.622 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:22.622 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:33:22.622 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:33:22.622 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:33:22.622 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:22.622 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:33:22.622 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:33:22.623 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:33:22.623 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:22.641 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:22.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:33:22.643 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:33:22.643 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:33:22.645 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:22.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:33:22.656 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:33:22.656 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:22.662 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:33:22.662 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:22.698 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:33:22.699 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:22.820 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:33:22.822 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:33:22.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-06T07:33:22.891 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:33:28.650 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:33:28.651 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:33:28.660 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:33:28.661 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:28.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:33:28.850 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:33:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.888+0000 7f3edd000640 1 Processor -- start 2024-03-06T07:33:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.889+0000 7f3edd000640 1 -- start start 2024-03-06T07:33:28.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.890+0000 7f3edd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3ed815efc0 0x7f3ed80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.890+0000 7f3edd000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed81627f0 con 0x7f3ed81621d0 2024-03-06T07:33:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.890+0000 7f3edd000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed806c860 con 0x7f3ed80cce90 2024-03-06T07:33:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.890+0000 7f3edd000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed815f3a0 con 0x7f3ed815efc0 2024-03-06T07:33:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.891+0000 7f3ed7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3ed81621d0 0x7f3ed8070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59516/0 (socket says 172.21.15.32:59516) 2024-03-06T07:33:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.891+0000 7f3ed7e00640 1 -- 172.21.15.32:0/714435584 learned_addr learned my addr 172.21.15.32:0/714435584 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.891+0000 7f3ed7400640 1 --2- 172.21.15.32:0/714435584 >> v2:172.21.15.39:3300/0 conn(0x7f3ed815efc0 0x7f3ed80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.891+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113313893 0 0) 0x7f3ed81627f0 con 0x7f3ed81621d0 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.891+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc003520 con 0x7f3ed81621d0 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619638488 0 0) 0x7f3ed806c860 con 0x7f3ed80cce90 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed81627f0 con 0x7f3ed80cce90 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334864060 0 0) 0x7f3ebc003520 con 0x7f3ed81621d0 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 >> v2:172.21.15.39:3300/0 conn(0x7f3ed815efc0 msgr2=0x7f3ed80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 --2- 172.21.15.32:0/714435584 >> v2:172.21.15.39:3300/0 conn(0x7f3ed815efc0 0x7f3ed80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 >> v1:172.21.15.39:6789/0 conn(0x7f3ed80cce90 legacy=0x7f3ed8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed806cb40 con 0x7f3ed81621d0 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3ed00019f0 con 0x7f3ed81621d0 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 >> v1:172.21.15.32:6789/0 conn(0x7f3ed81621d0 legacy=0x7f3ed8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ebc003b40 0x7f3ebc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ebc004800 0x7f3ebc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ebc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed815f3a0 con 0x7f3ebc0051c0 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00c040 con 0x7f3ebc004800 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00c240 con 0x7f3ebc003b40 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed7e00640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ebc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.892+0000 7f3ed7400640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ebc003b40 0x7f3ebc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.893+0000 7f3ed6a00640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ebc004800 0x7f3ebc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.893+0000 7f3ed7e00640 1 -- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ebc003b40 msgr2=0x7f3ebc003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.893+0000 7f3ed7e00640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ebc003b40 0x7f3ebc003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.893+0000 7f3ed7e00640 1 -- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ebc004800 msgr2=0x7f3ebc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.893+0000 7f3ed7e00640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ebc004800 0x7f3ebc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.893+0000 7f3ed7e00640 1 -- 172.21.15.32:0/714435584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc003520 con 0x7f3ebc0051c0 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.894+0000 7f3ed7e00640 1 -- 172.21.15.32:0/714435584 reap_dead start 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.894+0000 7f3ed7e00640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ebc00b9c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000a820 tx=0x7f3ebc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3f83e440dc2648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.894+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000a450 con 0x7f3ebc0051c0 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.894+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed0007310 con 0x7f3ebc0051c0 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.894+0000 7f3ed6000640 1 -- 172.21.15.32:0/714435584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0007700 con 0x7f3ebc0051c0 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 -- 172.21.15.32:0/714435584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 msgr2=0x7f3ebc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ebc00b9c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000a820 tx=0x7f3ebc005830 comp rx=0 tx=0).stop 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 -- 172.21.15.32:0/714435584 shutdown_connections 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 --2- 172.21.15.32:0/714435584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ebc00b9c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 -- 172.21.15.32:0/714435584 >> 172.21.15.32:0/714435584 conn(0x7f3ed806b3d0 msgr2=0x7f3ed806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 -- 172.21.15.32:0/714435584 shutdown_connections 2024-03-06T07:33:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.895+0000 7f3edd000640 1 -- 172.21.15.32:0/714435584 wait complete. 2024-03-06T07:33:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.897+0000 7f3edd000640 1 Processor -- start 2024-03-06T07:33:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.897+0000 7f3edd000640 1 -- start start 2024-03-06T07:33:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3edd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ed8136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3edd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ed81f7b40 0x7f3ed815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3edd000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ed81f4220 0x7f3ed80af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3edd000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed806d620 con 0x7f3ebc0051c0 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3edd000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed806d190 con 0x7f3ed81f7b40 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3edd000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81375b0 con 0x7f3ed81f4220 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.898+0000 7f3ed7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ed8136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ed8136d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59494/0 (socket says 172.21.15.32:59494) 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 -- 172.21.15.32:0/3989788646 learned_addr learned my addr 172.21.15.32:0/3989788646 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 -- 172.21.15.32:0/3989788646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ed81f4220 msgr2=0x7f3ed80af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ed81f4220 0x7f3ed80af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 -- 172.21.15.32:0/3989788646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ed81f7b40 msgr2=0x7f3ed815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ed81f7b40 0x7f3ed815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 -- 172.21.15.32:0/3989788646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8137890 con 0x7f3ebc0051c0 2024-03-06T07:33:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ed7400640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ed8136d20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80118e0 tx=0x7f3ec8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.899+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec800faf0 con 0x7f3ebc0051c0 2024-03-06T07:33:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.900+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec800fc90 con 0x7f3ebc0051c0 2024-03-06T07:33:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.900+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8014440 con 0x7f3ebc0051c0 2024-03-06T07:33:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.900+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed8137b70 con 0x7f3ebc0051c0 2024-03-06T07:33:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.900+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed8137260 con 0x7f3ebc0051c0 2024-03-06T07:33:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.901+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed804f330 con 0x7f3ebc0051c0 2024-03-06T07:33:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.902+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ec8012040 con 0x7f3ebc0051c0 2024-03-06T07:33:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.903+0000 7f3ecf400640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3eb806b4d0 0x7f3eb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.903+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3ec808a680 con 0x7f3ebc0051c0 2024-03-06T07:33:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.903+0000 7f3ed6a00640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3eb806b4d0 0x7f3eb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.904+0000 7f3ed6a00640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3eb806b4d0 0x7f3eb806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4007850 tx=0x7f3ec4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:28.908+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ec8019030 con 0x7f3ebc0051c0 2024-03-06T07:33:29.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:29.125+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3ed80af2f0 con 0x7f3ebc0051c0 2024-03-06T07:33:29.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-06T07:33:30.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.330+0000 7f3ecf400640 1 -- 172.21.15.32:0/3989788646 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3ec801b300 con 0x7f3ebc0051c0 2024-03-06T07:33:30.333 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3eb806b4d0 msgr2=0x7f3eb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3eb806b4d0 0x7f3eb806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4007850 tx=0x7f3ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 msgr2=0x7f3ed8136d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ed8136d20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80118e0 tx=0x7f3ec8011da0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 shutdown_connections 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3eb806b4d0 0x7f3eb806d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ed81f4220 0x7f3ed80af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ed81f7b40 0x7f3ed815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 --2- 172.21.15.32:0/3989788646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ebc0051c0 0x7f3ed8136d20 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 >> 172.21.15.32:0/3989788646 conn(0x7f3ed806b3d0 msgr2=0x7f3ed80cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 shutdown_connections 2024-03-06T07:33:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.334+0000 7f3edd000640 1 -- 172.21.15.32:0/3989788646 wait complete. 2024-03-06T07:33:30.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:33:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.582+0000 7fa1dcc00640 1 Processor -- start 2024-03-06T07:33:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.582+0000 7fa1dcc00640 1 -- start start 2024-03-06T07:33:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1dcc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1d8145300 0x7fa1d80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1dcc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d8070f10 0x7fa1d80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1dcc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d80714d0 con 0x7fa1d80cce90 2024-03-06T07:33:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1dcc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d80cd300 con 0x7fa1d8145300 2024-03-06T07:33:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1dcc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80717b0 con 0x7fa1d8070f10 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1d7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1d8145300 0x7fa1d80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1d7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1d8145300 0x7fa1d80b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59504/0 (socket says 172.21.15.32:59504) 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.584+0000 7fa1d7400640 1 -- 172.21.15.32:0/2044191450 learned_addr learned my addr 172.21.15.32:0/2044191450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.585+0000 7fa1cfe00640 1 --2- 172.21.15.32:0/2044191450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d8070f10 0x7fa1d80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.585+0000 7fa1d7400640 1 -- 172.21.15.32:0/2044191450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d8070f10 msgr2=0x7fa1d80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.585+0000 7fa1d7400640 1 --2- 172.21.15.32:0/2044191450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d8070f10 0x7fa1d80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.585+0000 7fa1d7400640 1 -- 172.21.15.32:0/2044191450 >> v1:172.21.15.39:6789/0 conn(0x7fa1d80cce90 legacy=0x7fa1d80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.585+0000 7fa1d7400640 1 -- 172.21.15.32:0/2044191450 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8071a90 con 0x7fa1d8145300 2024-03-06T07:33:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.586+0000 7fa1d7400640 1 --2- 172.21.15.32:0/2044191450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8145300 0x7fa1d80b49a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009870 tx=0x7fa1c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbbfa097f4c9627 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.586+0000 7fa1d6a00640 1 -- 172.21.15.32:0/2044191450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8035020 con 0x7fa1d8145300 2024-03-06T07:33:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.586+0000 7fa1d6a00640 1 -- 172.21.15.32:0/2044191450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c8031640 con 0x7fa1d8145300 2024-03-06T07:33:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.586+0000 7fa1d6a00640 1 -- 172.21.15.32:0/2044191450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c802d040 con 0x7fa1d8145300 2024-03-06T07:33:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2044191450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8145300 msgr2=0x7fa1d80b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2044191450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8145300 0x7fa1d80b49a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009870 tx=0x7fa1c802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2044191450 shutdown_connections 2024-03-06T07:33:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2044191450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d8070f10 0x7fa1d80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2044191450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8145300 0x7fa1d80b49a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2044191450 >> 172.21.15.32:0/2044191450 conn(0x7fa1d806b3d0 msgr2=0x7fa1d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2044191450 shutdown_connections 2024-03-06T07:33:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.587+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2044191450 wait complete. 2024-03-06T07:33:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.589+0000 7fa1dcc00640 1 Processor -- start 2024-03-06T07:33:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.589+0000 7fa1dcc00640 1 -- start start 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1dcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 0x7fa1d815b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1dcc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d80cce90 0x7fa1d8143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1dcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d8145300 0x7fa1d806dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1dcc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d81456e0 con 0x7fa1d8070f10 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1dcc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80b4ee0 con 0x7fa1d8145300 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1dcc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d815d580 con 0x7fa1d80cce90 2024-03-06T07:33:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.590+0000 7fa1d7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 0x7fa1d815b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 0x7fa1d815b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59514/0 (socket says 172.21.15.32:59514) 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 -- 172.21.15.32:0/1919686343 learned_addr learned my addr 172.21.15.32:0/1919686343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1cfe00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d80cce90 0x7fa1d8143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7e00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d8145300 0x7fa1d806dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 -- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d80cce90 msgr2=0x7fa1d8143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d80cce90 0x7fa1d8143850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 -- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d8145300 msgr2=0x7fa1d806dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d8145300 0x7fa1d806dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 -- 172.21.15.32:0/1919686343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8143d90 con 0x7fa1d8070f10 2024-03-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7e00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d8145300 0x7fa1d806dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.591+0000 7fa1d7400640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 0x7fa1d815b530 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa1c802b930 tx=0x7fa1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.592+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8040070 con 0x7fa1d8070f10 2024-03-06T07:33:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.592+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c8039c10 con 0x7fa1d8070f10 2024-03-06T07:33:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.592+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8035040 con 0x7fa1d8070f10 2024-03-06T07:33:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.592+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d81f75e0 con 0x7fa1d8070f10 2024-03-06T07:33:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.592+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d81f7ba0 con 0x7fa1d8070f10 2024-03-06T07:33:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.593+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d804f330 con 0x7fa1d8070f10 2024-03-06T07:33:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.594+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1c804b060 con 0x7fa1d8070f10 2024-03-06T07:33:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.595+0000 7fa1cf400640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa1b406b4d0 0x7fa1b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.595+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa1c80ab9e0 con 0x7fa1d8070f10 2024-03-06T07:33:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.595+0000 7fa1cfe00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa1b406b4d0 0x7fa1b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.596+0000 7fa1cfe00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa1b406b4d0 0x7fa1b406d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8143780 tx=0x7fa1c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.599+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1c802f300 con 0x7fa1d8070f10 2024-03-06T07:33:30.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.873+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa1d80725a0 con 0x7fa1d8070f10 2024-03-06T07:33:30.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.874+0000 7fa1cf400640 1 -- 172.21.15.32:0/1919686343 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa1c807aba0 con 0x7fa1d8070f10 2024-03-06T07:33:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:30.876 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:33:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:33:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:33:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:33:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:33:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:33:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:33:30.892 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa1b406b4d0 msgr2=0x7fa1b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa1b406b4d0 0x7fa1b406d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8143780 tx=0x7fa1c0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 msgr2=0x7fa1d815b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 0x7fa1d815b530 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa1c802b930 tx=0x7fa1c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 shutdown_connections 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa1b406b4d0 0x7fa1b406d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d8145300 0x7fa1d806dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1d80cce90 0x7fa1d8143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/1919686343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1d8070f10 0x7fa1d815b530 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 >> 172.21.15.32:0/1919686343 conn(0x7fa1d806b3d0 msgr2=0x7fa1d80ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.878+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 shutdown_connections 2024-03-06T07:33:30.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:30.879+0000 7fa1dcc00640 1 -- 172.21.15.32:0/1919686343 wait complete. 2024-03-06T07:33:30.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:31.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.111+0000 7f625a200640 1 Processor -- start 2024-03-06T07:33:31.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.111+0000 7f625a200640 1 -- start start 2024-03-06T07:33:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.113+0000 7f625a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62540b34e0 0x7f625415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.113+0000 7f625a200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625415cc40 con 0x7f62540af580 2024-03-06T07:33:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.113+0000 7f625a200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625415cf20 con 0x7f62540af1a0 2024-03-06T07:33:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.113+0000 7f625a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625415d200 con 0x7f62540b34e0 2024-03-06T07:33:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.113+0000 7f6252a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f62540af580 0x7f62540b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59520/0 (socket says 172.21.15.32:59520) 2024-03-06T07:33:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.113+0000 7f6252a00640 1 -- 172.21.15.32:0/2924168590 learned_addr learned my addr 172.21.15.32:0/2924168590 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:31.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6253e00640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62540b34e0 0x7f625415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134447396 0 0) 0x7f625415cc40 con 0x7f62540af580 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6238003520 con 0x7f62540af580 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3753695296 0 0) 0x7f6238003520 con 0x7f62540af580 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62540b34e0 msgr2=0x7f625415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f62540b34e0 0x7f625415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 >> v1:172.21.15.39:6789/0 conn(0x7f62540af1a0 legacy=0x7f62540b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625415d4e0 con 0x7f62540af580 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6240001670 con 0x7f62540af580 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 >> v1:172.21.15.32:6789/0 conn(0x7f62540af580 legacy=0x7f62540b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6238003a50 0x7f6238003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6238004800 0x7f6238004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62380052e0 0x7f623800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625415d200 con 0x7f6238004800 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623800c190 con 0x7f62380052e0 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.114+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623800c390 con 0x7f6238003a50 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6252a00640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6238004800 0x7f6238004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6252a00640 1 -- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6238003a50 msgr2=0x7f6238003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6253400640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6238003a50 0x7f6238003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6252a00640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6238003a50 0x7f6238003e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6252a00640 1 -- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62380052e0 msgr2=0x7f623800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6253e00640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62380052e0 0x7f623800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6252a00640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62380052e0 0x7f623800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.115+0000 7f6252a00640 1 -- 172.21.15.32:0/2924168590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238003520 con 0x7f6238004800 2024-03-06T07:33:31.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.116+0000 7f6252a00640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6238004800 0x7f6238004c80 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6240001eb0 tx=0x7f6238005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e6810556a97c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.116+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6240007d30 con 0x7f6238004800 2024-03-06T07:33:31.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.116+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6240030070 con 0x7f6238004800 2024-03-06T07:33:31.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.116+0000 7f6252000640 1 -- 172.21.15.32:0/2924168590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6240005d70 con 0x7f6238004800 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f625a200640 1 -- 172.21.15.32:0/2924168590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6238004800 msgr2=0x7f6238004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f625a200640 1 --2- 172.21.15.32:0/2924168590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6238004800 0x7f6238004c80 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6240001eb0 tx=0x7f6238005830 comp rx=0 tx=0).stop 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f6253e00640 1 -- 172.21.15.32:0/2924168590 reap_dead start 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f625a200640 1 -- 172.21.15.32:0/2924168590 shutdown_connections 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f625a200640 1 -- 172.21.15.32:0/2924168590 >> 172.21.15.32:0/2924168590 conn(0x7f625406afb0 msgr2=0x7f625406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f625a200640 1 -- 172.21.15.32:0/2924168590 shutdown_connections 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.117+0000 7f625a200640 1 -- 172.21.15.32:0/2924168590 wait complete. 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.119+0000 7f625a200640 1 Processor -- start 2024-03-06T07:33:31.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.119+0000 7f625a200640 1 -- start start 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f625a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6238004800 0x7f62540cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f625a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f625406e380 0x7f62540ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f625a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 0x7f625414f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f625a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625415c600 con 0x7f625414f1e0 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f625a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62540cde70 con 0x7f6238004800 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f625a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625415c780 con 0x7f625406e380 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f6253e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 0x7f625414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f6253e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 0x7f625414f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59530/0 (socket says 172.21.15.32:59530) 2024-03-06T07:33:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.120+0000 7f6253e00640 1 -- 172.21.15.32:0/2289033080 learned_addr learned my addr 172.21.15.32:0/2289033080 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253400640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6238004800 0x7f62540cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6252a00640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f625406e380 0x7f62540ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253e00640 1 -- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f625406e380 msgr2=0x7f62540ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253e00640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f625406e380 0x7f62540ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253e00640 1 -- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6238004800 msgr2=0x7f62540cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253e00640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6238004800 0x7f62540cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253e00640 1 -- 172.21.15.32:0/2289033080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625406cbb0 con 0x7f625414f1e0 2024-03-06T07:33:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f6253e00640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 0x7f625414f660 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f62540b3970 tx=0x7f624c00e510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c01b070 con 0x7f625414f1e0 2024-03-06T07:33:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c00ee20 con 0x7f625414f1e0 2024-03-06T07:33:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c003bd0 con 0x7f625414f1e0 2024-03-06T07:33:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.121+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625406e800 con 0x7f625414f1e0 2024-03-06T07:33:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.122+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625406d100 con 0x7f625414f1e0 2024-03-06T07:33:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.123+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6254069510 con 0x7f625414f1e0 2024-03-06T07:33:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.124+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f624c01e070 con 0x7f625414f1e0 2024-03-06T07:33:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.125+0000 7f624b400640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f623406b4d0 0x7f623406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.125+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f624c08a2f0 con 0x7f625414f1e0 2024-03-06T07:33:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.125+0000 7f6253400640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f623406b4d0 0x7f623406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.125+0000 7f6253400640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f623406b4d0 0x7f623406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6244001710 tx=0x7f6244009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.129+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f625406e800 con 0x7f625414f1e0 2024-03-06T07:33:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.331+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625404f3a0 con 0x7f625414f1e0 2024-03-06T07:33:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.333+0000 7f624b400640 1 -- 172.21.15.32:0/2289033080 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12784 (secure 0 0 0) 0x7f624c010040 con 0x7f625414f1e0 2024-03-06T07:33:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:31.336 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":78,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:30.327900+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f623406b4d0 msgr2=0x7f623406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f623406b4d0 0x7f623406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6244001710 tx=0x7f6244009040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 msgr2=0x7f625414f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 0x7f625414f660 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f62540b3970 tx=0x7f624c00e510 comp rx=0 tx=0).stop 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 shutdown_connections 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f623406b4d0 0x7f623406d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f625414f1e0 0x7f625414f660 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f625406e380 0x7f62540ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 --2- 172.21.15.32:0/2289033080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6238004800 0x7f62540cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 >> 172.21.15.32:0/2289033080 conn(0x7f625406afb0 msgr2=0x7f62540b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 shutdown_connections 2024-03-06T07:33:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.336+0000 7f625a200640 1 -- 172.21.15.32:0/2289033080 wait complete. 2024-03-06T07:33:31.360 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:31.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.579+0000 7f3b92e00640 1 Processor -- start 2024-03-06T07:33:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.579+0000 7f3b92e00640 1 -- start start 2024-03-06T07:33:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b92e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b8c15f810 0x7f3b8c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b92e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3b8c0b2080 0x7f3b8c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b92e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c15fe40 con 0x7f3b8c145430 2024-03-06T07:33:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b92e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c0b24d0 con 0x7f3b8c15f810 2024-03-06T07:33:31.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b92e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c160120 con 0x7f3b8c0b2080 2024-03-06T07:33:31.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b91000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b8c15f810 0x7f3b8c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b91000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b8c15f810 0x7f3b8c0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59534/0 (socket says 172.21.15.32:59534) 2024-03-06T07:33:31.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.581+0000 7f3b91000640 1 -- 172.21.15.32:0/3513244011 learned_addr learned my addr 172.21.15.32:0/3513244011 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:31.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.582+0000 7f3b91000640 1 -- 172.21.15.32:0/3513244011 >> v2:172.21.15.39:3300/0 conn(0x7f3b8c0b2080 msgr2=0x7f3b8c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.582+0000 7f3b92400640 1 --2- 172.21.15.32:0/3513244011 >> v2:172.21.15.39:3300/0 conn(0x7f3b8c0b2080 0x7f3b8c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.582+0000 7f3b91000640 1 --2- 172.21.15.32:0/3513244011 >> v2:172.21.15.39:3300/0 conn(0x7f3b8c0b2080 0x7f3b8c0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.582+0000 7f3b91000640 1 -- 172.21.15.32:0/3513244011 >> v1:172.21.15.39:6789/0 conn(0x7f3b8c145430 legacy=0x7f3b8c15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.582+0000 7f3b91000640 1 -- 172.21.15.32:0/3513244011 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c160400 con 0x7f3b8c15f810 2024-03-06T07:33:31.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.582+0000 7f3b92400640 1 --2- 172.21.15.32:0/3513244011 >> v2:172.21.15.39:3300/0 conn(0x7f3b8c0b2080 0x7f3b8c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:31.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.583+0000 7f3b91000640 1 --2- 172.21.15.32:0/3513244011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b1b40 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0b4440 tx=0x7f3b7802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9f04ef4f815f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.583+0000 7f3b8be00640 1 -- 172.21.15.32:0/3513244011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78034070 con 0x7f3b8c15f810 2024-03-06T07:33:31.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b8be00640 1 -- 172.21.15.32:0/3513244011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b78030860 con 0x7f3b8c15f810 2024-03-06T07:33:31.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b8be00640 1 -- 172.21.15.32:0/3513244011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78030c30 con 0x7f3b8c15f810 2024-03-06T07:33:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 -- 172.21.15.32:0/3513244011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 msgr2=0x7f3b8c0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 --2- 172.21.15.32:0/3513244011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b1b40 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0b4440 tx=0x7f3b7802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 -- 172.21.15.32:0/3513244011 shutdown_connections 2024-03-06T07:33:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 --2- 172.21.15.32:0/3513244011 >> v2:172.21.15.39:3300/0 conn(0x7f3b8c0b2080 0x7f3b8c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 --2- 172.21.15.32:0/3513244011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b1b40 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 -- 172.21.15.32:0/3513244011 >> 172.21.15.32:0/3513244011 conn(0x7f3b8c06b3a0 msgr2=0x7f3b8c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 -- 172.21.15.32:0/3513244011 shutdown_connections 2024-03-06T07:33:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.584+0000 7f3b92e00640 1 -- 172.21.15.32:0/3513244011 wait complete. 2024-03-06T07:33:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.586+0000 7f3b92e00640 1 Processor -- start 2024-03-06T07:33:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.586+0000 7f3b92e00640 1 -- start start 2024-03-06T07:33:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b8c0b2080 0x7f3b8c15c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b8c145430 0x7f3b8c138580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c145990 con 0x7f3b8c15f810 2024-03-06T07:33:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c145810 con 0x7f3b8c0b2080 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0cde00 con 0x7f3b8c145430 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.587+0000 7f3b92400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59536/0 (socket says 172.21.15.32:59536) 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 -- 172.21.15.32:0/1319864700 learned_addr learned my addr 172.21.15.32:0/1319864700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 -- 172.21.15.32:0/1319864700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b8c145430 msgr2=0x7f3b8c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b8c145430 0x7f3b8c138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 -- 172.21.15.32:0/1319864700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b8c0b2080 msgr2=0x7f3b8c15c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b8c0b2080 0x7f3b8c15c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 -- 172.21.15.32:0/1319864700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c071990 con 0x7f3b8c15f810 2024-03-06T07:33:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.588+0000 7f3b92400640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b0720 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400cef0 tx=0x7f3b8400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.589+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8401a070 con 0x7f3b8c15f810 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.589+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8400d860 con 0x7f3b8c15f810 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.589+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c071c30 con 0x7f3b8c15f810 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.589+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8400dc50 con 0x7f3b8c15f810 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.589+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c0721f0 con 0x7f3b8c15f810 2024-03-06T07:33:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.590+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c04f330 con 0x7f3b8c15f810 2024-03-06T07:33:31.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.591+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b8401d070 con 0x7f3b8c15f810 2024-03-06T07:33:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.592+0000 7f3b89600640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.592+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b8408a740 con 0x7f3b8c15f810 2024-03-06T07:33:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.592+0000 7f3b91a00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.593+0000 7f3b91a00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3b80000f60 tx=0x7f3b800079d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.596+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b840599c0 con 0x7f3b8c15f810 2024-03-06T07:33:31.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.796+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b8c0cf0c0 con 0x7f3b8c15f810 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.797+0000 7f3b89600640 1 -- 172.21.15.32:0/1319864700 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b840592c0 con 0x7f3b8c15f810 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b6c06b4d0 msgr2=0x7f3b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3b80000f60 tx=0x7f3b800079d0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 msgr2=0x7f3b8c0b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b0720 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400cef0 tx=0x7f3b8400a710 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 shutdown_connections 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b6c06b4d0 0x7f3b6c06d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c15f810 0x7f3b8c0b0720 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b8c145430 0x7f3b8c138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 --2- 172.21.15.32:0/1319864700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b8c0b2080 0x7f3b8c15c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.799+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 >> 172.21.15.32:0/1319864700 conn(0x7f3b8c06b3a0 msgr2=0x7f3b8c15d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.800+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 shutdown_connections 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:31.800+0000 7f3b92e00640 1 -- 172.21.15.32:0/1319864700 wait complete. 2024-03-06T07:33:31.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:33:32.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.037+0000 7f2b28e00640 1 Processor -- start 2024-03-06T07:33:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.037+0000 7f2b28e00640 1 -- start start 2024-03-06T07:33:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b28e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b24070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b28e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2b2406e240 0x7f2b240711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b28e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b2406cfe0 0x7f2b2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b28e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b240730d0 con 0x7f2b2406cfe0 2024-03-06T07:33:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b28e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b240ce9a0 con 0x7f2b2406e240 2024-03-06T07:33:32.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b28e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240cec80 con 0x7f2b24161670 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b22800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b2406cfe0 0x7f2b2406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b22800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b2406cfe0 0x7f2b2406d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59552/0 (socket says 172.21.15.32:59552) 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.039+0000 7f2b22800640 1 -- 172.21.15.32:0/2436810798 learned_addr learned my addr 172.21.15.32:0/2436810798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b21e00640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b24070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b19400640 1 --2- 172.21.15.32:0/2436810798 >> v2:172.21.15.39:3300/0 conn(0x7f2b2406e240 0x7f2b240711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b22800640 1 -- 172.21.15.32:0/2436810798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 msgr2=0x7f2b24070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b22800640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b24070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b22800640 1 -- 172.21.15.32:0/2436810798 >> v2:172.21.15.39:3300/0 conn(0x7f2b2406e240 msgr2=0x7f2b240711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b22800640 1 --2- 172.21.15.32:0/2436810798 >> v2:172.21.15.39:3300/0 conn(0x7f2b2406e240 0x7f2b240711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b22800640 1 -- 172.21.15.32:0/2436810798 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240cef60 con 0x7f2b2406cfe0 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b19400640 1 --2- 172.21.15.32:0/2436810798 >> v2:172.21.15.39:3300/0 conn(0x7f2b2406e240 0x7f2b240711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.040+0000 7f2b21e00640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b24070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.041+0000 7f2b22800640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406cfe0 0x7f2b2406d420 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2b240748c0 tx=0x7f2b1c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7b80df9aa8ad3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.041+0000 7f2b21400640 1 -- 172.21.15.32:0/2436810798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c02ca80 con 0x7f2b2406cfe0 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b21400640 1 -- 172.21.15.32:0/2436810798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c02cc20 con 0x7f2b2406cfe0 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b21400640 1 -- 172.21.15.32:0/2436810798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c0343f0 con 0x7f2b2406cfe0 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 -- 172.21.15.32:0/2436810798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406cfe0 msgr2=0x7f2b2406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406cfe0 0x7f2b2406d420 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2b240748c0 tx=0x7f2b1c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 -- 172.21.15.32:0/2436810798 shutdown_connections 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406cfe0 0x7f2b2406d420 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 --2- 172.21.15.32:0/2436810798 >> v2:172.21.15.39:3300/0 conn(0x7f2b2406e240 0x7f2b240711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 --2- 172.21.15.32:0/2436810798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b24070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 -- 172.21.15.32:0/2436810798 >> 172.21.15.32:0/2436810798 conn(0x7f2b2406afb0 msgr2=0x7f2b2406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 -- 172.21.15.32:0/2436810798 shutdown_connections 2024-03-06T07:33:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.042+0000 7f2b28e00640 1 -- 172.21.15.32:0/2436810798 wait complete. 2024-03-06T07:33:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.044+0000 7f2b28e00640 1 Processor -- start 2024-03-06T07:33:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.044+0000 7f2b28e00640 1 -- start start 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b28e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b2406cfe0 0x7f2b2415e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b28e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 0x7f2b2414f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b28e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b2414fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b28e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240ce600 con 0x7f2b2406e240 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b28e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240cf520 con 0x7f2b2406cfe0 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b28e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240ce780 con 0x7f2b24161670 2024-03-06T07:33:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b19400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 0x7f2b2414f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b19400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 0x7f2b2414f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59568/0 (socket says 172.21.15.32:59568) 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.045+0000 7f2b19400640 1 -- 172.21.15.32:0/3473542134 learned_addr learned my addr 172.21.15.32:0/3473542134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b22800640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b2414fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b21e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b2406cfe0 0x7f2b2415e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b19400640 1 -- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 msgr2=0x7f2b2414fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b19400640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b2414fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b19400640 1 -- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b2406cfe0 msgr2=0x7f2b2415e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b19400640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b2406cfe0 0x7f2b2415e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b19400640 1 -- 172.21.15.32:0/3473542134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b241503e0 con 0x7f2b2406e240 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b22800640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b2414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b19400640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 0x7f2b2414f960 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00c7c0 tx=0x7f2b0c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.046+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c015070 con 0x7f2b2406e240 2024-03-06T07:33:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.047+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c007a30 con 0x7f2b2406e240 2024-03-06T07:33:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.047+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c007e00 con 0x7f2b2406e240 2024-03-06T07:33:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.047+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b241f7500 con 0x7f2b2406e240 2024-03-06T07:33:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.047+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b241f7ac0 con 0x7f2b2406e240 2024-03-06T07:33:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.048+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b24069510 con 0x7f2b2406e240 2024-03-06T07:33:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.049+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b0c018070 con 0x7f2b2406e240 2024-03-06T07:33:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.050+0000 7f2b1aa00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.050+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2b0c089e40 con 0x7f2b2406e240 2024-03-06T07:33:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.050+0000 7f2b21e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.051+0000 7f2b21e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b0006b4d0 0x7f2b0006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400a990 tx=0x7f2b14008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.055+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b241503e0 con 0x7f2b2406e240 2024-03-06T07:33:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.257+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2b240b1500 con 0x7f2b2406e240 2024-03-06T07:33:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.258+0000 7f2b1aa00640 1 -- 172.21.15.32:0/3473542134 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2b0c058f70 con 0x7f2b2406e240 2024-03-06T07:33:32.260 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:32.261 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:33:32.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:32.261 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:33:32.261 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:33:32.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:32.261 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:33:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b0006b4d0 msgr2=0x7f2b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b0006b4d0 0x7f2b0006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400a990 tx=0x7f2b14008040 comp rx=0 tx=0).stop 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 msgr2=0x7f2b2414f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 0x7f2b2414f960 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00c7c0 tx=0x7f2b0c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 shutdown_connections 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b0006b4d0 0x7f2b0006d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b24161670 0x7f2b2414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b2406e240 0x7f2b2414f960 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 --2- 172.21.15.32:0/3473542134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b2406cfe0 0x7f2b2415e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 >> 172.21.15.32:0/3473542134 conn(0x7f2b2406afb0 msgr2=0x7f2b240b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 shutdown_connections 2024-03-06T07:33:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.261+0000 7f2b28e00640 1 -- 172.21.15.32:0/3473542134 wait complete. 2024-03-06T07:33:32.283 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:33:32.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:33:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.501+0000 7fb567e00640 1 Processor -- start 2024-03-06T07:33:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.502+0000 7fb567e00640 1 -- start start 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.503+0000 7fb567e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5600ae7b0 0x7fb5600aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.503+0000 7fb567e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600b3640 0x7fb56015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.503+0000 7fb567e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5601531f0 con 0x7fb5600aeb90 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.503+0000 7fb567e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5600d04d0 con 0x7fb5600ae7b0 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.503+0000 7fb567e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56015d300 con 0x7fb5600b3640 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5600ae7b0 0x7fb5600aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5600ae7b0 0x7fb5600aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59576/0 (socket says 172.21.15.32:59576) 2024-03-06T07:33:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 -- 172.21.15.32:0/759788431 learned_addr learned my addr 172.21.15.32:0/759788431 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb564a00640 1 --2- 172.21.15.32:0/759788431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600b3640 0x7fb56015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 -- 172.21.15.32:0/759788431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600b3640 msgr2=0x7fb56015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 --2- 172.21.15.32:0/759788431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600b3640 0x7fb56015cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 -- 172.21.15.32:0/759788431 >> v1:172.21.15.39:6789/0 conn(0x7fb5600aeb90 legacy=0x7fb56006cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.504+0000 7fb565400640 1 -- 172.21.15.32:0/759788431 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600b2b80 con 0x7fb5600ae7b0 2024-03-06T07:33:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.506+0000 7fb565400640 1 --2- 172.21.15.32:0/759788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600ae7b0 0x7fb5600aced0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb5500098a0 tx=0x7fb55002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ecd05ffd180f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.506+0000 7fb55fe00640 1 -- 172.21.15.32:0/759788431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550035020 con 0x7fb5600ae7b0 2024-03-06T07:33:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.506+0000 7fb55fe00640 1 -- 172.21.15.32:0/759788431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb550031640 con 0x7fb5600ae7b0 2024-03-06T07:33:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.506+0000 7fb55fe00640 1 -- 172.21.15.32:0/759788431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55002d040 con 0x7fb5600ae7b0 2024-03-06T07:33:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 -- 172.21.15.32:0/759788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600ae7b0 msgr2=0x7fb5600aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 --2- 172.21.15.32:0/759788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600ae7b0 0x7fb5600aced0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb5500098a0 tx=0x7fb55002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 -- 172.21.15.32:0/759788431 shutdown_connections 2024-03-06T07:33:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 --2- 172.21.15.32:0/759788431 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600b3640 0x7fb56015cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 --2- 172.21.15.32:0/759788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600ae7b0 0x7fb5600aced0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 -- 172.21.15.32:0/759788431 >> 172.21.15.32:0/759788431 conn(0x7fb56006afb0 msgr2=0x7fb56006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 -- 172.21.15.32:0/759788431 shutdown_connections 2024-03-06T07:33:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.507+0000 7fb567e00640 1 -- 172.21.15.32:0/759788431 wait complete. 2024-03-06T07:33:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.509+0000 7fb567e00640 1 Processor -- start 2024-03-06T07:33:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.509+0000 7fb567e00640 1 -- start start 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.510+0000 7fb567e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600ae7b0 0x7fb5601f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb567e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 0x7fb56009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb567e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5600b3640 0x7fb56014e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb567e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600b2e40 con 0x7fb5600aeb90 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb567e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56006eeb0 con 0x7fb5600b3640 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb567e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600b3120 con 0x7fb5600ae7b0 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 0x7fb56009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 0x7fb56009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59580/0 (socket says 172.21.15.32:59580) 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 -- 172.21.15.32:0/1347200818 learned_addr learned my addr 172.21.15.32:0/1347200818 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb565400640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600ae7b0 0x7fb5601f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 -- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600ae7b0 msgr2=0x7fb5601f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600ae7b0 0x7fb5601f13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 -- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5600b3640 msgr2=0x7fb56014e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5600b3640 0x7fb56014e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.511+0000 7fb564a00640 1 -- 172.21.15.32:0/1347200818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56009c450 con 0x7fb5600aeb90 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.512+0000 7fb564a00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 0x7fb56009aa10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c7f0 tx=0x7fb54c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.512+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c015070 con 0x7fb5600aeb90 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.512+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c004220 con 0x7fb5600aeb90 2024-03-06T07:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.512+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c007620 con 0x7fb5600aeb90 2024-03-06T07:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.512+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56009b3b0 con 0x7fb5600aeb90 2024-03-06T07:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.513+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56009b970 con 0x7fb5600aeb90 2024-03-06T07:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.514+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb560068010 con 0x7fb5600aeb90 2024-03-06T07:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.514+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb54c0079d0 con 0x7fb5600aeb90 2024-03-06T07:33:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.515+0000 7fb55d600640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb54006b4d0 0x7fb54006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.515+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb54c08a2e0 con 0x7fb5600aeb90 2024-03-06T07:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.516+0000 7fb565400640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb54006b4d0 0x7fb54006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.516+0000 7fb565400640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb54006b4d0 0x7fb54006d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb550036000 tx=0x7fb55003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.520+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb54c059410 con 0x7fb5600aeb90 2024-03-06T07:33:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:32.718+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb5600b4cf0 con 0x7fb5600aeb90 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.371+0000 7fb55d600640 1 -- 172.21.15.32:0/1347200818 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb54c058d10 con 0x7fb5600aeb90 2024-03-06T07:33:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.374+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb54006b4d0 msgr2=0x7fb54006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.374+0000 7fb567e00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb54006b4d0 0x7fb54006d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb550036000 tx=0x7fb55003c040 comp rx=0 tx=0).stop 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.374+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 msgr2=0x7fb56009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.374+0000 7fb567e00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 0x7fb56009aa10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c7f0 tx=0x7fb54c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 shutdown_connections 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb54006b4d0 0x7fb54006d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5600b3640 0x7fb56014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5600aeb90 0x7fb56009aa10 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 --2- 172.21.15.32:0/1347200818 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb5600ae7b0 0x7fb5601f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 >> 172.21.15.32:0/1347200818 conn(0x7fb56006afb0 msgr2=0x7fb56006ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.375+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 shutdown_connections 2024-03-06T07:33:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.376+0000 7fb567e00640 1 -- 172.21.15.32:0/1347200818 wait complete. 2024-03-06T07:33:33.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:33:33.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.619+0000 7f0152a00640 1 Processor -- start 2024-03-06T07:33:33.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.619+0000 7f0152a00640 1 -- start start 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.621+0000 7f0152a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f014c0ad6f0 0x7f014c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.621+0000 7f0152a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f014c15f960 0x7f014c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.621+0000 7f0152a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c156830 con 0x7f014c0b3c30 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.621+0000 7f0152a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014c06ce60 con 0x7f014c0ad6f0 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.621+0000 7f0152a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014c0ae010 con 0x7f014c15f960 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f014c0ad6f0 0x7f014c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f014c0ad6f0 0x7f014c15c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59596/0 (socket says 172.21.15.32:59596) 2024-03-06T07:33:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 -- 172.21.15.32:0/545930692 learned_addr learned my addr 172.21.15.32:0/545930692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:33.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 -- 172.21.15.32:0/545930692 >> v2:172.21.15.39:3300/0 conn(0x7f014c15f960 msgr2=0x7f014c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 --2- 172.21.15.32:0/545930692 >> v2:172.21.15.39:3300/0 conn(0x7f014c15f960 0x7f014c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 -- 172.21.15.32:0/545930692 >> v1:172.21.15.39:6789/0 conn(0x7f014c0b3c30 legacy=0x7f014c0b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.622+0000 7f014be00640 1 -- 172.21.15.32:0/545930692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c160120 con 0x7f014c0ad6f0 2024-03-06T07:33:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.623+0000 7f014be00640 1 --2- 172.21.15.32:0/545930692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c15c370 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f014c0705e0 tx=0x7f013c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680a4fe7d9e04261 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f014b400640 1 -- 172.21.15.32:0/545930692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c035020 con 0x7f014c0ad6f0 2024-03-06T07:33:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f014b400640 1 -- 172.21.15.32:0/545930692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c031870 con 0x7f014c0ad6f0 2024-03-06T07:33:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f014b400640 1 -- 172.21.15.32:0/545930692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c031c40 con 0x7f014c0ad6f0 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 -- 172.21.15.32:0/545930692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 msgr2=0x7f014c15c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 --2- 172.21.15.32:0/545930692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c15c370 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f014c0705e0 tx=0x7f013c02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 -- 172.21.15.32:0/545930692 shutdown_connections 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 --2- 172.21.15.32:0/545930692 >> v2:172.21.15.39:3300/0 conn(0x7f014c15f960 0x7f014c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 --2- 172.21.15.32:0/545930692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c15c370 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 -- 172.21.15.32:0/545930692 >> 172.21.15.32:0/545930692 conn(0x7f014c06b3d0 msgr2=0x7f014c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.624+0000 7f0152a00640 1 -- 172.21.15.32:0/545930692 shutdown_connections 2024-03-06T07:33:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.625+0000 7f0152a00640 1 -- 172.21.15.32:0/545930692 wait complete. 2024-03-06T07:33:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.626+0000 7f0152a00640 1 Processor -- start 2024-03-06T07:33:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.626+0000 7f0152a00640 1 -- start start 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.627+0000 7f0152a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c09d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.627+0000 7f0152a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f014c0b3c30 0x7f014c0cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.627+0000 7f0152a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f014c15f960 0x7f014c1f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.627+0000 7f0152a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c06d0b0 con 0x7f014c0ad6f0 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.627+0000 7f0152a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c1606e0 con 0x7f014c0b3c30 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.627+0000 7f0152a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c15fdf0 con 0x7f014c15f960 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c09d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c09d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59600/0 (socket says 172.21.15.32:59600) 2024-03-06T07:33:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 -- 172.21.15.32:0/3003793636 learned_addr learned my addr 172.21.15.32:0/3003793636 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f0143e00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f014c0b3c30 0x7f014c0cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f0150a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f014c15f960 0x7f014c1f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 -- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f014c15f960 msgr2=0x7f014c1f4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f014c15f960 0x7f014c1f4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 -- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f014c0b3c30 msgr2=0x7f014c0cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f014c0b3c30 0x7f014c0cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f014be00640 1 -- 172.21.15.32:0/3003793636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c1f50d0 con 0x7f014c0ad6f0 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f0143e00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f014c0b3c30 0x7f014c0cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.628+0000 7f0150a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f014c15f960 0x7f014c1f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.629+0000 7f014be00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c09d2f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f013c000c00 tx=0x7f013c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.629+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c03ad60 con 0x7f014c0ad6f0 2024-03-06T07:33:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.629+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c0443f0 con 0x7f014c0ad6f0 2024-03-06T07:33:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.629+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c1f53b0 con 0x7f014c0ad6f0 2024-03-06T07:33:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.629+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c0447e0 con 0x7f014c0ad6f0 2024-03-06T07:33:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.629+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c1f5da0 con 0x7f014c0ad6f0 2024-03-06T07:33:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.631+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c068010 con 0x7f014c0ad6f0 2024-03-06T07:33:33.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.632+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f013c04b060 con 0x7f014c0ad6f0 2024-03-06T07:33:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.633+0000 7f0148c00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f012806b4d0 0x7f012806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.633+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f013c0ac230 con 0x7f014c0ad6f0 2024-03-06T07:33:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.633+0000 7f0143e00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f012806b4d0 0x7f012806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.634+0000 7f0143e00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f012806b4d0 0x7f012806d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0134006fd0 tx=0x7f0134008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.638+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f013c03f070 con 0x7f014c0ad6f0 2024-03-06T07:33:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:33.837+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f014c0cc440 con 0x7f014c0ad6f0 2024-03-06T07:33:34.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.389+0000 7f0148c00640 1 -- 172.21.15.32:0/3003793636 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7f013c02f320 con 0x7f014c0ad6f0 2024-03-06T07:33:34.391 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:33:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.393+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f012806b4d0 msgr2=0x7f012806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.393+0000 7f0152a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f012806b4d0 0x7f012806d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0134006fd0 tx=0x7f0134008040 comp rx=0 tx=0).stop 2024-03-06T07:33:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.393+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 msgr2=0x7f014c09d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.393+0000 7f0152a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c09d2f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f013c000c00 tx=0x7f013c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.394+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 shutdown_connections 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.394+0000 7f0152a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f012806b4d0 0x7f012806d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.394+0000 7f0152a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f014c15f960 0x7f014c1f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.394+0000 7f0152a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f014c0b3c30 0x7f014c0cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.394+0000 7f0152a00640 1 --2- 172.21.15.32:0/3003793636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f014c0ad6f0 0x7f014c09d2f0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.394+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 >> 172.21.15.32:0/3003793636 conn(0x7f014c06b3d0 msgr2=0x7f014c15cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.395+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 shutdown_connections 2024-03-06T07:33:34.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.395+0000 7f0152a00640 1 -- 172.21.15.32:0/3003793636 wait complete. 2024-03-06T07:33:34.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:33:34.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.616+0000 7f65ece00640 1 Processor -- start 2024-03-06T07:33:34.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.617+0000 7f65ece00640 1 -- start start 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.618+0000 7f65ece00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f65e8072520 0x7f65e80cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.618+0000 7f65ece00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e8072df0 con 0x7f65e8071e20 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.618+0000 7f65ece00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80cdab0 con 0x7f65e815bf20 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.618+0000 7f65ece00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e80b4d70 con 0x7f65e8072520 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.619+0000 7f65e6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f65e8071e20 0x7f65e80ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59532/0 (socket says 172.21.15.32:59532) 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.619+0000 7f65e6800640 1 -- 172.21.15.32:0/1734180560 learned_addr learned my addr 172.21.15.32:0/1734180560 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.619+0000 7f65dd400640 1 --2- 172.21.15.32:0/1734180560 >> v2:172.21.15.39:3300/0 conn(0x7f65e8072520 0x7f65e80cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.619+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348848295 0 0) 0x7f65e8072df0 con 0x7f65e8071e20 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.619+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc003520 con 0x7f65e8071e20 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3662916914 0 0) 0x7f65cc003520 con 0x7f65e8071e20 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 >> v2:172.21.15.39:3300/0 conn(0x7f65e8072520 msgr2=0x7f65e80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 --2- 172.21.15.32:0/1734180560 >> v2:172.21.15.39:3300/0 conn(0x7f65e8072520 0x7f65e80cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 >> v1:172.21.15.39:6789/0 conn(0x7f65e815bf20 legacy=0x7f65e80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e80b5050 con 0x7f65e8071e20 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f65e0001670 con 0x7f65e8071e20 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 >> v1:172.21.15.32:6789/0 conn(0x7f65e8071e20 legacy=0x7f65e80ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0039b0 0x7f65cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0043b0 0x7f65cc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65cc005080 0x7f65cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b4d70 con 0x7f65cc0039b0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc00bfe0 con 0x7f65cc005080 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc00c1e0 con 0x7f65cc0043b0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.620+0000 7f65e5e00640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0039b0 0x7f65cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e5e00640 1 -- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0043b0 msgr2=0x7f65cc004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e6800640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65cc005080 0x7f65cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e5e00640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0043b0 0x7f65cc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e5e00640 1 -- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65cc005080 msgr2=0x7f65cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e5e00640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65cc005080 0x7f65cc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e5e00640 1 -- 172.21.15.32:0/1734180560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc003520 con 0x7f65cc0039b0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.621+0000 7f65e6800640 1 -- 172.21.15.32:0/1734180560 reap_dead start 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.622+0000 7f65e5e00640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0039b0 0x7f65cc003d90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f65d8001f60 tx=0x7f65cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f705a03f4a3429 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.622+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8001b30 con 0x7f65cc0039b0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.622+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d802d710 con 0x7f65cc0039b0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.622+0000 7f65e5400640 1 -- 172.21.15.32:0/1734180560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d802db00 con 0x7f65cc0039b0 2024-03-06T07:33:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 -- 172.21.15.32:0/1734180560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0039b0 msgr2=0x7f65cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0039b0 0x7f65cc003d90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f65d8001f60 tx=0x7f65cc005830 comp rx=0 tx=0).stop 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 -- 172.21.15.32:0/1734180560 shutdown_connections 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 --2- 172.21.15.32:0/1734180560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0039b0 0x7f65cc003d90 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 -- 172.21.15.32:0/1734180560 >> 172.21.15.32:0/1734180560 conn(0x7f65e806b3d0 msgr2=0x7f65e806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 -- 172.21.15.32:0/1734180560 shutdown_connections 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.623+0000 7f65ece00640 1 -- 172.21.15.32:0/1734180560 wait complete. 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.625+0000 7f65ece00640 1 Processor -- start 2024-03-06T07:33:34.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.625+0000 7f65ece00640 1 -- start start 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65ece00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0039b0 0x7f65e806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65ece00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65e80ca4a0 0x7f65e81552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65ece00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 0x7f65e81f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65ece00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b4b10 con 0x7f65e8155830 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65ece00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b4990 con 0x7f65e80ca4a0 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65ece00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80acd40 con 0x7f65cc0039b0 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 0x7f65e81f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 0x7f65e81f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59610/0 (socket says 172.21.15.32:59610) 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65e6800640 1 -- 172.21.15.32:0/3256330798 learned_addr learned my addr 172.21.15.32:0/3256330798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.626+0000 7f65dd400640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65e80ca4a0 0x7f65e81552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e6800640 1 -- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0039b0 msgr2=0x7f65e806d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e5e00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0039b0 0x7f65e806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e6800640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0039b0 0x7f65e806d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e6800640 1 -- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65e80ca4a0 msgr2=0x7f65e81552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e6800640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65e80ca4a0 0x7f65e81552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e6800640 1 -- 172.21.15.32:0/3256330798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e806d940 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65dd400640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65e80ca4a0 0x7f65e81552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65e6800640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 0x7f65e81f4250 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f65e80cd670 tx=0x7f65e000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0007440 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0007600 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0004a20 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e806dc20 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.627+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e806e1e0 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.629+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e804f330 con 0x7f65e8155830 2024-03-06T07:33:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.630+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65e001b060 con 0x7f65e8155830 2024-03-06T07:33:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.631+0000 7f65dea00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65c406b4d0 0x7f65c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.631+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f65e0089a10 con 0x7f65e8155830 2024-03-06T07:33:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.631+0000 7f65e5e00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65c406b4d0 0x7f65c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.632+0000 7f65e5e00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65c406b4d0 0x7f65c406d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f65d8036000 tx=0x7f65d8031000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.636+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65e00588f0 con 0x7f65e8155830 2024-03-06T07:33:34.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:34.874+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f65e8152b20 con 0x7f65e8155830 2024-03-06T07:33:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.396+0000 7f65dea00640 1 -- 172.21.15.32:0/3256330798 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v49) v1 ==== 174+0+0 (secure 0 0 0) 0x7f65e00581f0 con 0x7f65e8155830 2024-03-06T07:33:35.399 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:33:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.402+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65c406b4d0 msgr2=0x7f65c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.402+0000 7f65ece00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65c406b4d0 0x7f65c406d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f65d8036000 tx=0x7f65d8031000 comp rx=0 tx=0).stop 2024-03-06T07:33:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.402+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 msgr2=0x7f65e81f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.402+0000 7f65ece00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 0x7f65e81f4250 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f65e80cd670 tx=0x7f65e000d9d0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.404+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 shutdown_connections 2024-03-06T07:33:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.404+0000 7f65ece00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f65c406b4d0 0x7f65c406d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.404+0000 7f65ece00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65e8155830 0x7f65e81f4250 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.404+0000 7f65ece00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65e80ca4a0 0x7f65e81552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.404+0000 7f65ece00640 1 --2- 172.21.15.32:0/3256330798 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f65cc0039b0 0x7f65e806d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.404+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 >> 172.21.15.32:0/3256330798 conn(0x7f65e806b3d0 msgr2=0x7f65e8159920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.405+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 shutdown_connections 2024-03-06T07:33:35.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.409+0000 7f65ece00640 1 -- 172.21.15.32:0/3256330798 wait complete. 2024-03-06T07:33:35.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:35.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.624+0000 7f6e4b200640 1 Processor -- start 2024-03-06T07:33:35.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.625+0000 7f6e4b200640 1 -- start start 2024-03-06T07:33:35.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.626+0000 7f6e4b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6e440cebc0 0x7f6e441652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.626+0000 7f6e4b200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e44071640 con 0x7f6e4415a930 2024-03-06T07:33:35.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.626+0000 7f6e4b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e44071920 con 0x7f6e4415a550 2024-03-06T07:33:35.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.626+0000 7f6e4b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e440af6f0 con 0x7f6e440cebc0 2024-03-06T07:33:35.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.626+0000 7f6e3b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6e4415a930 0x7f6e440ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59536/0 (socket says 172.21.15.32:59536) 2024-03-06T07:33:35.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.626+0000 7f6e3b400640 1 -- 172.21.15.32:0/1954192672 learned_addr learned my addr 172.21.15.32:0/1954192672 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077615195 0 0) 0x7f6e44071640 con 0x7f6e4415a930 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e28003520 con 0x7f6e4415a930 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e49200640 1 --2- 172.21.15.32:0/1954192672 >> v2:172.21.15.39:3300/0 conn(0x7f6e440cebc0 0x7f6e441652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 240899402 0 0) 0x7f6e28003520 con 0x7f6e4415a930 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 >> v2:172.21.15.39:3300/0 conn(0x7f6e440cebc0 msgr2=0x7f6e441652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 --2- 172.21.15.32:0/1954192672 >> v2:172.21.15.39:3300/0 conn(0x7f6e440cebc0 0x7f6e441652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 >> v1:172.21.15.39:6789/0 conn(0x7f6e4415a550 legacy=0x7f6e440aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44071c00 con 0x7f6e4415a930 2024-03-06T07:33:35.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6e2c001670 con 0x7f6e4415a930 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.627+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 >> v1:172.21.15.32:6789/0 conn(0x7f6e4415a930 legacy=0x7f6e440ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e28003a20 0x7f6e28003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e28004800 0x7f6e28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28005230 0x7f6e2800cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440af6f0 con 0x7f6e28004800 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2800d290 con 0x7f6e28005230 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2800d490 con 0x7f6e28003a20 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e3b400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e28004800 0x7f6e28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43e00640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e28003a20 0x7f6e28003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e3b400640 1 -- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e28003a20 msgr2=0x7f6e28003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e49200640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28005230 0x7f6e2800cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e3b400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e28003a20 0x7f6e28003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e3b400640 1 -- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28005230 msgr2=0x7f6e2800cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e3b400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28005230 0x7f6e2800cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e3b400640 1 -- 172.21.15.32:0/1954192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28003520 con 0x7f6e28004800 2024-03-06T07:33:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e49200640 1 -- 172.21.15.32:0/1954192672 reap_dead start 2024-03-06T07:33:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.628+0000 7f6e43e00640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e28003a20 0x7f6e28003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.629+0000 7f6e3b400640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e28004800 0x7f6e28004c80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c007cb0 tx=0x7f6e28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad579663ed4963e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c007cf0 con 0x7f6e28004800 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2c032070 con 0x7f6e28004800 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e43400640 1 -- 172.21.15.32:0/1954192672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c0058d0 con 0x7f6e28004800 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e4b200640 1 -- 172.21.15.32:0/1954192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e28004800 msgr2=0x7f6e28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e4b200640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e28004800 0x7f6e28004c80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c007cb0 tx=0x7f6e28005830 comp rx=0 tx=0).stop 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e4b200640 1 -- 172.21.15.32:0/1954192672 shutdown_connections 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e4b200640 1 --2- 172.21.15.32:0/1954192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e28004800 0x7f6e28004c80 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.630+0000 7f6e4b200640 1 -- 172.21.15.32:0/1954192672 >> 172.21.15.32:0/1954192672 conn(0x7f6e4406afb0 msgr2=0x7f6e4406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.631+0000 7f6e4b200640 1 -- 172.21.15.32:0/1954192672 shutdown_connections 2024-03-06T07:33:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.631+0000 7f6e4b200640 1 -- 172.21.15.32:0/1954192672 wait complete. 2024-03-06T07:33:35.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.632+0000 7f6e4b200640 1 Processor -- start 2024-03-06T07:33:35.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.632+0000 7f6e4b200640 1 -- start start 2024-03-06T07:33:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.633+0000 7f6e4b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28004800 0x7f6e441f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.633+0000 7f6e4b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 0x7f6e440cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.633+0000 7f6e4b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e440c6320 0x7f6e4406dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.633+0000 7f6e4b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440b52b0 con 0x7f6e4409d6f0 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.633+0000 7f6e4b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440b5130 con 0x7f6e28004800 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.633+0000 7f6e4b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440aefe0 con 0x7f6e440c6320 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 0x7f6e440cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 0x7f6e440cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59628/0 (socket says 172.21.15.32:59628) 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 -- 172.21.15.32:0/2724073632 learned_addr learned my addr 172.21.15.32:0/2724073632 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e43e00640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28004800 0x7f6e441f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e49200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e440c6320 0x7f6e4406dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 -- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e440c6320 msgr2=0x7f6e4406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e440c6320 0x7f6e4406dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 -- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28004800 msgr2=0x7f6e441f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28004800 0x7f6e441f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e3b400640 1 -- 172.21.15.32:0/2724073632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e440cd3e0 con 0x7f6e4409d6f0 2024-03-06T07:33:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e43e00640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28004800 0x7f6e441f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.634+0000 7f6e49200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e440c6320 0x7f6e4406dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.635+0000 7f6e3b400640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 0x7f6e440cc820 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c02e760 tx=0x7f6e44071810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.635+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c007cf0 con 0x7f6e4409d6f0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.635+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2c007070 con 0x7f6e4409d6f0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.635+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c001c30 con 0x7f6e4409d6f0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.635+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e440cd100 con 0x7f6e4409d6f0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.635+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e441f3480 con 0x7f6e4409d6f0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.636+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4404f3a0 con 0x7f6e4409d6f0 2024-03-06T07:33:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.637+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e2c04a020 con 0x7f6e4409d6f0 2024-03-06T07:33:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.638+0000 7f6e40c00640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e2006b500 0x7f6e2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.638+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e2c0ad870 con 0x7f6e4409d6f0 2024-03-06T07:33:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.638+0000 7f6e43e00640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e2006b500 0x7f6e2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.639+0000 7f6e43e00640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e2006b500 0x7f6e2006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6e44050c30 tx=0x7f6e34009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.643+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e2c07c5c0 con 0x7f6e4409d6f0 2024-03-06T07:33:35.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.847+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e440cf400 con 0x7f6e4409d6f0 2024-03-06T07:33:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.849+0000 7f6e40c00640 1 -- 172.21.15.32:0/2724073632 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f6e2c07bec0 con 0x7f6e4409d6f0 2024-03-06T07:33:35.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:34.876408+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e2006b500 msgr2=0x7f6e2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e2006b500 0x7f6e2006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6e44050c30 tx=0x7f6e34009040 comp rx=0 tx=0).stop 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 msgr2=0x7f6e440cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 0x7f6e440cc820 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c02e760 tx=0x7f6e44071810 comp rx=0 tx=0).stop 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 shutdown_connections 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6e2006b500 0x7f6e2006d9c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e440c6320 0x7f6e4406dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4409d6f0 0x7f6e440cc820 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 --2- 172.21.15.32:0/2724073632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e28004800 0x7f6e441f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.851+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 >> 172.21.15.32:0/2724073632 conn(0x7f6e4406afb0 msgr2=0x7f6e440ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.852+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 shutdown_connections 2024-03-06T07:33:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:35.852+0000 7f6e4b200640 1 -- 172.21.15.32:0/2724073632 wait complete. 2024-03-06T07:33:35.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:36.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.073+0000 7fa8f2a00640 1 Processor -- start 2024-03-06T07:33:36.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.074+0000 7fa8f2a00640 1 -- start start 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec15e0b0 0x7fa8ec0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8ec0d0e50 0x7fa8ec0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec0b3710 con 0x7fa8ec0d4a10 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b09e0 con 0x7fa8ec0d0e50 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b39f0 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8ec0d0e50 0x7fa8ec0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8ec0d0e50 0x7fa8ec0b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59644/0 (socket says 172.21.15.32:59644) 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.075+0000 7fa8f2000640 1 -- 172.21.15.32:0/1284051321 learned_addr learned my addr 172.21.15.32:0/1284051321 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.076+0000 7fa8f1600640 1 --2- 172.21.15.32:0/1284051321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec15e0b0 0x7fa8ec0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.076+0000 7fa8f2000640 1 -- 172.21.15.32:0/1284051321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec15e0b0 msgr2=0x7fa8ec0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.076+0000 7fa8f2000640 1 --2- 172.21.15.32:0/1284051321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec15e0b0 0x7fa8ec0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.076+0000 7fa8f2000640 1 -- 172.21.15.32:0/1284051321 >> v1:172.21.15.32:6789/0 conn(0x7fa8ec0d4a10 legacy=0x7fa8ec0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.076+0000 7fa8f2000640 1 -- 172.21.15.32:0/1284051321 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0b3cd0 con 0x7fa8ec0d0e50 2024-03-06T07:33:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.077+0000 7fa8f2000640 1 --2- 172.21.15.32:0/1284051321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec0d0e50 0x7fa8ec0b44d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009e50 tx=0x7fa8e402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc719c833bf6ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.077+0000 7fa8ebe00640 1 -- 172.21.15.32:0/1284051321 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e402cb00 con 0x7fa8ec0d0e50 2024-03-06T07:33:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.077+0000 7fa8ebe00640 1 -- 172.21.15.32:0/1284051321 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e402cca0 con 0x7fa8ec0d0e50 2024-03-06T07:33:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.077+0000 7fa8ebe00640 1 -- 172.21.15.32:0/1284051321 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4034610 con 0x7fa8ec0d0e50 2024-03-06T07:33:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 -- 172.21.15.32:0/1284051321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec0d0e50 msgr2=0x7fa8ec0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/1284051321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec0d0e50 0x7fa8ec0b44d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009e50 tx=0x7fa8e402bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 -- 172.21.15.32:0/1284051321 shutdown_connections 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/1284051321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec0d0e50 0x7fa8ec0b44d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/1284051321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec15e0b0 0x7fa8ec0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 -- 172.21.15.32:0/1284051321 >> 172.21.15.32:0/1284051321 conn(0x7fa8ec0d3d30 msgr2=0x7fa8ec0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 -- 172.21.15.32:0/1284051321 shutdown_connections 2024-03-06T07:33:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.078+0000 7fa8f2a00640 1 -- 172.21.15.32:0/1284051321 wait complete. 2024-03-06T07:33:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.080+0000 7fa8f2a00640 1 Processor -- start 2024-03-06T07:33:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.080+0000 7fa8f2a00640 1 -- start start 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec0d0e50 0x7fa8ec0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8ec0d4a10 0x7fa8ec0b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 0x7fa8ec14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0cfc60 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec15e490 con 0x7fa8ec0d4a10 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0cf850 con 0x7fa8ec0d0e50 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 0x7fa8ec14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 0x7fa8ec14fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59650/0 (socket says 172.21.15.32:59650) 2024-03-06T07:33:36.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f2000640 1 -- 172.21.15.32:0/4121197293 learned_addr learned my addr 172.21.15.32:0/4121197293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.081+0000 7fa8f1600640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec0d0e50 0x7fa8ec0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f0c00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8ec0d4a10 0x7fa8ec0b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f2000640 1 -- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec0d0e50 msgr2=0x7fa8ec0c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f2000640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec0d0e50 0x7fa8ec0c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f2000640 1 -- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8ec0d4a10 msgr2=0x7fa8ec0b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f2000640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8ec0d4a10 0x7fa8ec0b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f2000640 1 -- 172.21.15.32:0/4121197293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0b6a80 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.082+0000 7fa8f2000640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 0x7fa8ec14fd60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4036040 tx=0x7fa8e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.083+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4043020 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.083+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e402ce50 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.083+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec0b6d60 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.083+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4009510 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.083+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec0b7320 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.084+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec04f330 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.085+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8e4046020 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.086+0000 7fa8e9600640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b4d0 0x7fa8cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.086+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8e40ac880 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.086+0000 7fa8f1600640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b4d0 0x7fa8cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.087+0000 7fa8f1600640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b4d0 0x7fa8cc06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0006fd0 tx=0x7fa8e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.090+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8e404d030 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.468+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8ec0d3400 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.469+0000 7fa8e9600640 1 -- 172.21.15.32:0/4121197293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16552 (secure 0 0 0) 0x7fa8e407b6f0 con 0x7fa8ec15e0b0 2024-03-06T07:33:36.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:36.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:36.396956+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.472+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b4d0 msgr2=0x7fa8cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.472+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b4d0 0x7fa8cc06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0006fd0 tx=0x7fa8e0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.472+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 msgr2=0x7fa8ec14fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.472+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 0x7fa8ec14fd60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4036040 tx=0x7fa8e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 shutdown_connections 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b4d0 0x7fa8cc06d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8ec15e0b0 0x7fa8ec14fd60 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8ec0d4a10 0x7fa8ec0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 --2- 172.21.15.32:0/4121197293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8ec0d0e50 0x7fa8ec0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 >> 172.21.15.32:0/4121197293 conn(0x7fa8ec0d3d30 msgr2=0x7fa8ec0cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 shutdown_connections 2024-03-06T07:33:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.473+0000 7fa8f2a00640 1 -- 172.21.15.32:0/4121197293 wait complete. 2024-03-06T07:33:36.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:33:36.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.697+0000 7f36a3200640 1 Processor -- start 2024-03-06T07:33:36.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.697+0000 7f36a3200640 1 -- start start 2024-03-06T07:33:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.698+0000 7f36a3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36a415dee0 0x7f36a40d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.698+0000 7f36a3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a40ceca0 con 0x7f36a40d0d40 2024-03-06T07:33:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.698+0000 7f36a3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a40cef80 con 0x7f36a40d4860 2024-03-06T07:33:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.698+0000 7f36a3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a40cf5d0 con 0x7f36a415dee0 2024-03-06T07:33:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.699+0000 7f36a2800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f36a40d0d40 0x7f36a40b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59552/0 (socket says 172.21.15.32:59552) 2024-03-06T07:33:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.699+0000 7f36a2800640 1 -- 172.21.15.32:0/1956510080 learned_addr learned my addr 172.21.15.32:0/1956510080 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.699+0000 7f36a1e00640 1 --2- 172.21.15.32:0/1956510080 >> v2:172.21.15.39:3300/0 conn(0x7f36a415dee0 0x7f36a40d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.699+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686174076 0 0) 0x7f36a40ceca0 con 0x7f36a40d0d40 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.699+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003520 con 0x7f36a40d0d40 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153524033 0 0) 0x7f3688003520 con 0x7f36a40d0d40 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 >> v2:172.21.15.39:3300/0 conn(0x7f36a415dee0 msgr2=0x7f36a40d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 --2- 172.21.15.32:0/1956510080 >> v2:172.21.15.39:3300/0 conn(0x7f36a415dee0 0x7f36a40d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 >> v1:172.21.15.39:6789/0 conn(0x7f36a40d4860 legacy=0x7f36a40d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a40cf260 con 0x7f36a40d0d40 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f369c001670 con 0x7f36a40d0d40 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 >> v1:172.21.15.32:6789/0 conn(0x7f36a40d0d40 legacy=0x7f36a40b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36880039b0 0x7f3688003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3688004800 0x7f3688004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36880051c0 0x7f368800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40cf5d0 con 0x7f36880051c0 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368800c060 con 0x7f3688004800 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368800c260 con 0x7f36880039b0 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a2800640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36880051c0 0x7f368800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.700+0000 7f36a1400640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3688004800 0x7f3688004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.701+0000 7f36a1e00640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36880039b0 0x7f3688003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.701+0000 7f36a2800640 1 -- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36880039b0 msgr2=0x7f3688003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.701+0000 7f36a2800640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36880039b0 0x7f3688003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.701+0000 7f36a2800640 1 -- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3688004800 msgr2=0x7f3688004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.701+0000 7f36a2800640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3688004800 0x7f3688004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.701+0000 7f36a2800640 1 -- 172.21.15.32:0/1956510080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3688003520 con 0x7f36880051c0 2024-03-06T07:33:36.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.702+0000 7f36a2800640 1 -- 172.21.15.32:0/1956510080 reap_dead start 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.702+0000 7f36a2800640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36880051c0 0x7f368800b9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f369c00c240 tx=0x7f3688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e977cec3cb730d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.702+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c007d10 con 0x7f36880051c0 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.702+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369c030070 con 0x7f36880051c0 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.702+0000 7f36a0a00640 1 -- 172.21.15.32:0/1956510080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c02cd40 con 0x7f36880051c0 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 -- 172.21.15.32:0/1956510080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36880051c0 msgr2=0x7f368800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36880051c0 0x7f368800b9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f369c00c240 tx=0x7f3688005830 comp rx=0 tx=0).stop 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 -- 172.21.15.32:0/1956510080 shutdown_connections 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 --2- 172.21.15.32:0/1956510080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36880051c0 0x7f368800b9c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 -- 172.21.15.32:0/1956510080 >> 172.21.15.32:0/1956510080 conn(0x7f36a40d3cd0 msgr2=0x7f36a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 -- 172.21.15.32:0/1956510080 shutdown_connections 2024-03-06T07:33:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.703+0000 7f36a3200640 1 -- 172.21.15.32:0/1956510080 wait complete. 2024-03-06T07:33:36.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.704+0000 7f36a3200640 1 Processor -- start 2024-03-06T07:33:36.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 -- start start 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36880051c0 0x7f36a415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 0x7f36a40ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36a4160b30 0x7f36a4160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40cfa60 con 0x7f36a415a360 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a415e2c0 con 0x7f36880051c0 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.705+0000 7f36a3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40cfbe0 con 0x7f36a4160b30 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 0x7f36a40ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 0x7f36a40ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59664/0 (socket says 172.21.15.32:59664) 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 -- 172.21.15.32:0/2238000952 learned_addr learned my addr 172.21.15.32:0/2238000952 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1e00640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36880051c0 0x7f36a415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 -- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36a4160b30 msgr2=0x7f36a4160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36a4160b30 0x7f36a4160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 -- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36880051c0 msgr2=0x7f36a415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36880051c0 0x7f36a415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 -- 172.21.15.32:0/2238000952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a40cdd00 con 0x7f36a415a360 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.706+0000 7f36a1400640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 0x7f36a40ccd10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f369000e4e0 tx=0x7f369000e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.707+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36900052e0 con 0x7f36a415a360 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.707+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3690018070 con 0x7f36a415a360 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.707+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690003830 con 0x7f36a415a360 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.707+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a415a7c0 con 0x7f36a415a360 2024-03-06T07:33:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.707+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a40cd740 con 0x7f36a415a360 2024-03-06T07:33:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.708+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a404f330 con 0x7f36a415a360 2024-03-06T07:33:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.710+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36900039d0 con 0x7f36a415a360 2024-03-06T07:33:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.711+0000 7f369a000640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f368406b500 0x7f368406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.711+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3690089e20 con 0x7f36a415a360 2024-03-06T07:33:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.711+0000 7f36a1e00640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f368406b500 0x7f368406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.712+0000 7f36a1e00640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f368406b500 0x7f368406d9c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f36940074a0 tx=0x7f369400d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.716+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3690058c00 con 0x7f36a415a360 2024-03-06T07:33:36.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:36.962+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36a408d7d0 con 0x7f36a415a360 2024-03-06T07:33:37.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.738+0000 7f369a000640 1 -- 172.21.15.32:0/2238000952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3690058500 con 0x7f36a415a360 2024-03-06T07:33:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.741+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f368406b500 msgr2=0x7f368406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.741+0000 7f36a3200640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f368406b500 0x7f368406d9c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f36940074a0 tx=0x7f369400d040 comp rx=0 tx=0).stop 2024-03-06T07:33:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.741+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 msgr2=0x7f36a40ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.741+0000 7f36a3200640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 0x7f36a40ccd10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f369000e4e0 tx=0x7f369000e9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 shutdown_connections 2024-03-06T07:33:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f368406b500 0x7f368406d9c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f36a4160b30 0x7f36a4160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36a415a360 0x7f36a40ccd10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 --2- 172.21.15.32:0/2238000952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36880051c0 0x7f36a415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 >> 172.21.15.32:0/2238000952 conn(0x7f36a40d3cd0 msgr2=0x7f36a406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 shutdown_connections 2024-03-06T07:33:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.742+0000 7f36a3200640 1 -- 172.21.15.32:0/2238000952 wait complete. 2024-03-06T07:33:37.766 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:33:37.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:33:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.974+0000 7f8b41e00640 1 Processor -- start 2024-03-06T07:33:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.974+0000 7f8b41e00640 1 -- start start 2024-03-06T07:33:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.976+0000 7f8b41e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.976+0000 7f8b41e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8b3c0d4860 0x7f8b3c0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.976+0000 7f8b41e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c0ceca0 con 0x7f8b3c0cfa60 2024-03-06T07:33:37.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.976+0000 7f8b41e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0cf5d0 con 0x7f8b3c0d4860 2024-03-06T07:33:37.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.976+0000 7f8b41e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0cef80 con 0x7f8b3c15dee0 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.977+0000 7f8b40a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.977+0000 7f8b40a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c0d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:43200/0 (socket says 172.21.15.32:43200) 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.977+0000 7f8b40a00640 1 -- 172.21.15.32:0/732741261 learned_addr learned my addr 172.21.15.32:0/732741261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.977+0000 7f8b3be00640 1 --2- 172.21.15.32:0/732741261 >> v2:172.21.15.39:3300/0 conn(0x7f8b3c0d4860 0x7f8b3c0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.978+0000 7f8b40a00640 1 -- 172.21.15.32:0/732741261 >> v2:172.21.15.39:3300/0 conn(0x7f8b3c0d4860 msgr2=0x7f8b3c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.978+0000 7f8b40a00640 1 --2- 172.21.15.32:0/732741261 >> v2:172.21.15.39:3300/0 conn(0x7f8b3c0d4860 0x7f8b3c0d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.978+0000 7f8b40a00640 1 -- 172.21.15.32:0/732741261 >> v1:172.21.15.39:6789/0 conn(0x7f8b3c0cfa60 legacy=0x7f8b3c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.978+0000 7f8b40a00640 1 -- 172.21.15.32:0/732741261 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0cf260 con 0x7f8b3c15dee0 2024-03-06T07:33:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.979+0000 7f8b40a00640 1 --2- 172.21.15.32:0/732741261 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c0d18a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009870 tx=0x7f8b3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb44e12ae125eb4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.980+0000 7f8b3b400640 1 -- 172.21.15.32:0/732741261 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30034070 con 0x7f8b3c15dee0 2024-03-06T07:33:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.980+0000 7f8b3b400640 1 -- 172.21.15.32:0/732741261 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30030840 con 0x7f8b3c15dee0 2024-03-06T07:33:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.980+0000 7f8b3b400640 1 -- 172.21.15.32:0/732741261 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30030c10 con 0x7f8b3c15dee0 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.980+0000 7f8b41e00640 1 -- 172.21.15.32:0/732741261 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 msgr2=0x7f8b3c0d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.980+0000 7f8b41e00640 1 --2- 172.21.15.32:0/732741261 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c0d18a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009870 tx=0x7f8b3002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.981+0000 7f8b41e00640 1 -- 172.21.15.32:0/732741261 shutdown_connections 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.981+0000 7f8b41e00640 1 --2- 172.21.15.32:0/732741261 >> v2:172.21.15.39:3300/0 conn(0x7f8b3c0d4860 0x7f8b3c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.981+0000 7f8b41e00640 1 --2- 172.21.15.32:0/732741261 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c0d18a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.981+0000 7f8b41e00640 1 -- 172.21.15.32:0/732741261 >> 172.21.15.32:0/732741261 conn(0x7f8b3c0d3cd0 msgr2=0x7f8b3c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.981+0000 7f8b41e00640 1 -- 172.21.15.32:0/732741261 shutdown_connections 2024-03-06T07:33:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.981+0000 7f8b41e00640 1 -- 172.21.15.32:0/732741261 wait complete. 2024-03-06T07:33:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.983+0000 7f8b41e00640 1 Processor -- start 2024-03-06T07:33:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.983+0000 7f8b41e00640 1 -- start start 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.984+0000 7f8b41e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0cfa60 0x7f8b3c0c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.984+0000 7f8b41e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 0x7f8b3c0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.984+0000 7f8b41e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.984+0000 7f8b41e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0d1de0 con 0x7f8b3c0d4860 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.984+0000 7f8b41e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15e2c0 con 0x7f8b3c0cfa60 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.984+0000 7f8b41e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0d1f60 con 0x7f8b3c15dee0 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 0x7f8b3c0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 0x7f8b3c0c8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52180/0 (socket says 172.21.15.32:52180) 2024-03-06T07:33:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 -- 172.21.15.32:0/1820839826 learned_addr learned my addr 172.21.15.32:0/1820839826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b40a00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0cfa60 0x7f8b3c0c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b41400640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 -- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 msgr2=0x7f8b3c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 -- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0cfa60 msgr2=0x7f8b3c0c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0cfa60 0x7f8b3c0c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b3be00640 1 -- 172.21.15.32:0/1820839826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0c8eb0 con 0x7f8b3c0d4860 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b40a00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0cfa60 0x7f8b3c0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.985+0000 7f8b41400640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.986+0000 7f8b3be00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 0x7f8b3c0c8820 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800e980 tx=0x7f8b2800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.986+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28017070 con 0x7f8b3c0d4860 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.986+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2800ca00 con 0x7f8b3c0d4860 2024-03-06T07:33:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.986+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2800cdd0 con 0x7f8b3c0d4860 2024-03-06T07:33:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.986+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c0c9190 con 0x7f8b3c0d4860 2024-03-06T07:33:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.986+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c0c9750 con 0x7f8b3c0d4860 2024-03-06T07:33:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.987+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c04f330 con 0x7f8b3c0d4860 2024-03-06T07:33:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.988+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b2801a070 con 0x7f8b3c0d4860 2024-03-06T07:33:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.989+0000 7f8b38c00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.989+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b2808a600 con 0x7f8b3c0d4860 2024-03-06T07:33:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.989+0000 7f8b40a00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.990+0000 7f8b40a00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8b30036000 tx=0x7f8b30008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:37.994+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b28059470 con 0x7f8b3c0d4860 2024-03-06T07:33:38.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.346+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8b3c0ce220 con 0x7f8b3c0d4860 2024-03-06T07:33:38.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.753+0000 7f8b38c00640 1 -- 172.21.15.32:0/1820839826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8b28058d70 con 0x7f8b3c0d4860 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 msgr2=0x7f8b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8b30036000 tx=0x7f8b30008030 comp rx=0 tx=0).stop 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 msgr2=0x7f8b3c0c8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 0x7f8b3c0c8820 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800e980 tx=0x7f8b2800ee40 comp rx=0 tx=0).stop 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 shutdown_connections 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b3c15dee0 0x7f8b3c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b3c0d4860 0x7f8b3c0c8820 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 --2- 172.21.15.32:0/1820839826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b3c0cfa60 0x7f8b3c0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.757+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 >> 172.21.15.32:0/1820839826 conn(0x7f8b3c0d3cd0 msgr2=0x7f8b3c0b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.758+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 shutdown_connections 2024-03-06T07:33:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.758+0000 7f8b41e00640 1 -- 172.21.15.32:0/1820839826 wait complete. 2024-03-06T07:33:38.779 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:33:38.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.980+0000 7f90a6a00640 1 Processor -- start 2024-03-06T07:33:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.980+0000 7f90a6a00640 1 -- start start 2024-03-06T07:33:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.983+0000 7f90a6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90a00af1a0 0x7f90a015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f90a6a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a015cc40 con 0x7f90a00cdc20 2024-03-06T07:33:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f90a6a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a015cf20 con 0x7f90a00af580 2024-03-06T07:33:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f90a6a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a015d200 con 0x7f90a00af1a0 2024-03-06T07:33:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f90a4a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f90a00cdc20 0x7f90a00b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46836/0 (socket says 172.21.15.32:46836) 2024-03-06T07:33:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f90a4a00640 1 -- 172.21.15.32:0/3829133616 learned_addr learned my addr 172.21.15.32:0/3829133616 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273806690 0 0) 0x7f90a015cc40 con 0x7f90a00cdc20 2024-03-06T07:33:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9084003520 con 0x7f90a00cdc20 2024-03-06T07:33:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.984+0000 7f909fe00640 1 --2- 172.21.15.32:0/3829133616 >> v2:172.21.15.39:3300/0 conn(0x7f90a00af1a0 0x7f90a015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1352318448 0 0) 0x7f9084003520 con 0x7f90a00cdc20 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 >> v2:172.21.15.39:3300/0 conn(0x7f90a00af1a0 msgr2=0x7f90a015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 --2- 172.21.15.32:0/3829133616 >> v2:172.21.15.39:3300/0 conn(0x7f90a00af1a0 0x7f90a015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 >> v1:172.21.15.39:6789/0 conn(0x7f90a00af580 legacy=0x7f90a00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a015d4e0 con 0x7f90a00cdc20 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f90980019f0 con 0x7f90a00cdc20 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 >> v1:172.21.15.32:6789/0 conn(0x7f90a00cdc20 legacy=0x7f90a00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9084003a70 0x7f9084003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9084004800 0x7f9084004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90840053c0 0x7f908400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a015d200 con 0x7f90840053c0 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908400c250 con 0x7f9084003a70 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908400c450 con 0x7f9084004800 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f90a4a00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90840053c0 0x7f908400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.985+0000 7f909fe00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9084003a70 0x7f9084003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.986+0000 7f909f400640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9084004800 0x7f9084004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.986+0000 7f90a4a00640 1 -- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9084004800 msgr2=0x7f9084004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.986+0000 7f90a4a00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9084004800 0x7f9084004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.986+0000 7f90a4a00640 1 -- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9084003a70 msgr2=0x7f9084003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.986+0000 7f90a4a00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9084003a70 0x7f9084003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.986+0000 7f90a4a00640 1 -- 172.21.15.32:0/3829133616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084003520 con 0x7f90840053c0 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.987+0000 7f90a4a00640 1 -- 172.21.15.32:0/3829133616 reap_dead start 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.987+0000 7f90a4a00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90840053c0 0x7f908400b9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f909800a7a0 tx=0x7f9084005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ca7a9a7146d90b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.987+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909800a3f0 con 0x7f90840053c0 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.987+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9098034650 con 0x7f90840053c0 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.987+0000 7f909ea00640 1 -- 172.21.15.32:0/3829133616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098034a40 con 0x7f90840053c0 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 -- 172.21.15.32:0/3829133616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90840053c0 msgr2=0x7f908400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90840053c0 0x7f908400b9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f909800a7a0 tx=0x7f9084005830 comp rx=0 tx=0).stop 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 -- 172.21.15.32:0/3829133616 shutdown_connections 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 --2- 172.21.15.32:0/3829133616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90840053c0 0x7f908400b9c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 -- 172.21.15.32:0/3829133616 >> 172.21.15.32:0/3829133616 conn(0x7f90a006afb0 msgr2=0x7f90a006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 -- 172.21.15.32:0/3829133616 shutdown_connections 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.988+0000 7f90a6a00640 1 -- 172.21.15.32:0/3829133616 wait complete. 2024-03-06T07:33:38.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.990+0000 7f90a6a00640 1 Processor -- start 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.990+0000 7f90a6a00640 1 -- start start 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90840053c0 0x7f90a01efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a6a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90a0144640 0x7f90a015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 0x7f90a006cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a6a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a015c990 con 0x7f90a015e8f0 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a015c810 con 0x7f90840053c0 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a6a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a00b4c20 con 0x7f90a0144640 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 0x7f90a006cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 0x7f90a006cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52194/0 (socket says 172.21.15.32:52194) 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 -- 172.21.15.32:0/1289929147 learned_addr learned my addr 172.21.15.32:0/1289929147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 -- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90a0144640 msgr2=0x7f90a015e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f909f400640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90a0144640 0x7f90a015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f909fe00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90840053c0 0x7f90a01efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90a0144640 0x7f90a015e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 -- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90840053c0 msgr2=0x7f90a01efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90840053c0 0x7f90a01efe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f90a4a00640 1 -- 172.21.15.32:0/1289929147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a006d740 con 0x7f90a015e8f0 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.991+0000 7f909f400640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90a0144640 0x7f90a015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.992+0000 7f90a4a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 0x7f90a006cb10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9098002230 tx=0x7f9098001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.992+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098043070 con 0x7f90a015e8f0 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.992+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9098004b60 con 0x7f90a015e8f0 2024-03-06T07:33:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.992+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a0156e70 con 0x7f90a015e8f0 2024-03-06T07:33:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.992+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098039040 con 0x7f90a015e8f0 2024-03-06T07:33:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.992+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a006da20 con 0x7f90a015e8f0 2024-03-06T07:33:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.994+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a0069510 con 0x7f90a015e8f0 2024-03-06T07:33:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.995+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9098046070 con 0x7f90a015e8f0 2024-03-06T07:33:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.995+0000 7f9097e00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f908006b500 0x7f908006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.995+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90980ab6f0 con 0x7f90a015e8f0 2024-03-06T07:33:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.996+0000 7f909fe00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f908006b500 0x7f908006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:38.996+0000 7f909fe00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f908006b500 0x7f908006d9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9090006fd0 tx=0x7f9090008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.000+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f909807a560 con 0x7f90a015e8f0 2024-03-06T07:33:39.070 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-06T07:33:39.068+0000 7fa659000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:39.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.259+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a004f3a0 con 0x7f90a015e8f0 2024-03-06T07:33:39.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.260+0000 7f9097e00640 1 -- 172.21.15.32:0/1289929147 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f9098079e60 con 0x7f90a015e8f0 2024-03-06T07:33:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:39.263 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:39.263 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f908006b500 msgr2=0x7f908006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f908006b500 0x7f908006d9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9090006fd0 tx=0x7f9090008040 comp rx=0 tx=0).stop 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 msgr2=0x7f90a006cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 0x7f90a006cb10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9098002230 tx=0x7f9098001cf0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 shutdown_connections 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f908006b500 0x7f908006d9c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90a015e8f0 0x7f90a006cb10 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90a0144640 0x7f90a015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 --2- 172.21.15.32:0/1289929147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90840053c0 0x7f90a01efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.263+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 >> 172.21.15.32:0/1289929147 conn(0x7f90a006afb0 msgr2=0x7f90a00cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.264+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 shutdown_connections 2024-03-06T07:33:39.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.264+0000 7f90a6a00640 1 -- 172.21.15.32:0/1289929147 wait complete. 2024-03-06T07:33:39.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:39.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.499+0000 7f777da00640 1 Processor -- start 2024-03-06T07:33:39.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.499+0000 7f777da00640 1 -- start start 2024-03-06T07:33:39.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f777da00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f777815e0b0 0x7f77780cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f777da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f77780d4a10 0x7f77780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f777da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77780b3710 con 0x7f77780d0c80 2024-03-06T07:33:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f777da00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77780d10f0 con 0x7f77780d4a10 2024-03-06T07:33:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f777da00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77780b09e0 con 0x7f777815e0b0 2024-03-06T07:33:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f776fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f77780d4a10 0x7f77780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f776fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f77780d4a10 0x7f77780d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52196/0 (socket says 172.21.15.32:52196) 2024-03-06T07:33:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.501+0000 7f776fe00640 1 -- 172.21.15.32:0/1632125237 learned_addr learned my addr 172.21.15.32:0/1632125237 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.502+0000 7f7777e00640 1 --2- 172.21.15.32:0/1632125237 >> v2:172.21.15.39:3300/0 conn(0x7f777815e0b0 0x7f77780cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.502+0000 7f776fe00640 1 -- 172.21.15.32:0/1632125237 >> v2:172.21.15.39:3300/0 conn(0x7f777815e0b0 msgr2=0x7f77780cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.502+0000 7f776fe00640 1 --2- 172.21.15.32:0/1632125237 >> v2:172.21.15.39:3300/0 conn(0x7f777815e0b0 0x7f77780cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.502+0000 7f776fe00640 1 -- 172.21.15.32:0/1632125237 >> v1:172.21.15.39:6789/0 conn(0x7f77780d0c80 legacy=0x7f77780d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.502+0000 7f776fe00640 1 -- 172.21.15.32:0/1632125237 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780b39f0 con 0x7f77780d4a10 2024-03-06T07:33:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.503+0000 7f776fe00640 1 --2- 172.21.15.32:0/1632125237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780d4a10 0x7f77780d0740 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f777806b8c0 tx=0x7f776002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb7c14609056374 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.503+0000 7f7777400640 1 -- 172.21.15.32:0/1632125237 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760035020 con 0x7f77780d4a10 2024-03-06T07:33:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.503+0000 7f7777400640 1 -- 172.21.15.32:0/1632125237 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760031830 con 0x7f77780d4a10 2024-03-06T07:33:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.503+0000 7f7777400640 1 -- 172.21.15.32:0/1632125237 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760031c00 con 0x7f77780d4a10 2024-03-06T07:33:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 -- 172.21.15.32:0/1632125237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780d4a10 msgr2=0x7f77780d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 --2- 172.21.15.32:0/1632125237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780d4a10 0x7f77780d0740 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f777806b8c0 tx=0x7f776002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 -- 172.21.15.32:0/1632125237 shutdown_connections 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 --2- 172.21.15.32:0/1632125237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780d4a10 0x7f77780d0740 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 --2- 172.21.15.32:0/1632125237 >> v2:172.21.15.39:3300/0 conn(0x7f777815e0b0 0x7f77780cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 -- 172.21.15.32:0/1632125237 >> 172.21.15.32:0/1632125237 conn(0x7f77780d3d30 msgr2=0x7f77780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 -- 172.21.15.32:0/1632125237 shutdown_connections 2024-03-06T07:33:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.504+0000 7f777da00640 1 -- 172.21.15.32:0/1632125237 wait complete. 2024-03-06T07:33:39.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.506+0000 7f777da00640 1 Processor -- start 2024-03-06T07:33:39.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.506+0000 7f777da00640 1 -- start start 2024-03-06T07:33:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77780d0c80 0x7f77780cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f777815e0b0 0x7f777814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 0x7f77780b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777815e490 con 0x7f77780b0b70 2024-03-06T07:33:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b3fb0 con 0x7f777815e0b0 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780cfc60 con 0x7f77780d0c80 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 0x7f77780b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 0x7f77780b1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52208/0 (socket says 172.21.15.32:52208) 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.507+0000 7f777d000640 1 -- 172.21.15.32:0/2790040828 learned_addr learned my addr 172.21.15.32:0/2790040828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f7777e00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77780d0c80 0x7f77780cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f776fe00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f777815e0b0 0x7f777814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f777d000640 1 -- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77780d0c80 msgr2=0x7f77780cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f777d000640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77780d0c80 0x7f77780cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f777d000640 1 -- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f777815e0b0 msgr2=0x7f777814aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f777d000640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f777815e0b0 0x7f777814aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f777d000640 1 -- 172.21.15.32:0/2790040828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77781512e0 con 0x7f77780b0b70 2024-03-06T07:33:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f777d000640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 0x7f77780b1020 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f777000c7b0 tx=0x7f777000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.508+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770004b60 con 0x7f77780b0b70 2024-03-06T07:33:39.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.509+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7770004d00 con 0x7f77780b0b70 2024-03-06T07:33:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.509+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777000aa90 con 0x7f77780b0b70 2024-03-06T07:33:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.509+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777814d7f0 con 0x7f77780b0b70 2024-03-06T07:33:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.509+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777814ddb0 con 0x7f77780b0b70 2024-03-06T07:33:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.510+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777804f330 con 0x7f77780b0b70 2024-03-06T07:33:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.511+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f777000ac30 con 0x7f77780b0b70 2024-03-06T07:33:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.511+0000 7f7774c00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f775406b500 0x7f775406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.512+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f777008a4b0 con 0x7f77780b0b70 2024-03-06T07:33:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.512+0000 7f7777e00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f775406b500 0x7f775406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.513+0000 7f7777e00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f775406b500 0x7f775406d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7768006fd0 tx=0x7f7768008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.516+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7770020070 con 0x7f77780b0b70 2024-03-06T07:33:39.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.882+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77780b1960 con 0x7f77780b0b70 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.885+0000 7f7774c00640 1 -- 172.21.15.32:0/2790040828 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16552 (secure 0 0 0) 0x7f7770059290 con 0x7f77780b0b70 2024-03-06T07:33:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:39.888 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:36.396956+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f775406b500 msgr2=0x7f775406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f775406b500 0x7f775406d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7768006fd0 tx=0x7f7768008040 comp rx=0 tx=0).stop 2024-03-06T07:33:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 msgr2=0x7f77780b1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 0x7f77780b1020 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f777000c7b0 tx=0x7f777000cc70 comp rx=0 tx=0).stop 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 shutdown_connections 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f775406b500 0x7f775406d9c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77780b0b70 0x7f77780b1020 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f777815e0b0 0x7f777814aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 --2- 172.21.15.32:0/2790040828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77780d0c80 0x7f77780cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 >> 172.21.15.32:0/2790040828 conn(0x7f77780d3d30 msgr2=0x7f77780b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 shutdown_connections 2024-03-06T07:33:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:39.889+0000 7f777da00640 1 -- 172.21.15.32:0/2790040828 wait complete. 2024-03-06T07:33:39.910 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:33:39.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:40.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.127+0000 7fd548c00640 1 Processor -- start 2024-03-06T07:33:40.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.128+0000 7fd548c00640 1 -- start start 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd548c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd548c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd5440b50e0 0x7fd5440d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd548c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5440b3940 con 0x7fd5440d0d70 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd548c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5440d4e70 con 0x7fd5440b50e0 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd548c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5440b3c40 con 0x7fd54415e2e0 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd53be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5440d0d70 0x7fd5440d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46850/0 (socket says 172.21.15.32:46850) 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd53be00640 1 -- 172.21.15.32:0/3047778212 learned_addr learned my addr 172.21.15.32:0/3047778212 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd542800640 1 --2- 172.21.15.32:0/3047778212 >> v2:172.21.15.32:3300/0 conn(0x7fd5440b50e0 0x7fd5440d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd541e00640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd542800640 1 -- 172.21.15.32:0/3047778212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 msgr2=0x7fd5440d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd542800640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd542800640 1 -- 172.21.15.32:0/3047778212 >> v1:172.21.15.32:6789/0 conn(0x7fd5440d0d70 legacy=0x7fd5440d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd542800640 1 -- 172.21.15.32:0/3047778212 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5440cf580 con 0x7fd5440b50e0 2024-03-06T07:33:40.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd541400640 1 -- 172.21.15.32:0/3047778212 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895178673 0 0) 0x7fd5440b3940 con 0x7fd5440d0d70 2024-03-06T07:33:40.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.131+0000 7fd541e00640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.133+0000 7fd542800640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440d11d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009d90 tx=0x7fd53c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe92c46730f193af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.133+0000 7fd541400640 1 -- 172.21.15.32:0/3047778212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c039070 con 0x7fd5440b50e0 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.133+0000 7fd541400640 1 -- 172.21.15.32:0/3047778212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c02cc50 con 0x7fd5440b50e0 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.133+0000 7fd541400640 1 -- 172.21.15.32:0/3047778212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c034440 con 0x7fd5440b50e0 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.133+0000 7fd548c00640 1 -- 172.21.15.32:0/3047778212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 msgr2=0x7fd5440d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.133+0000 7fd548c00640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440d11d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009d90 tx=0x7fd53c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.134+0000 7fd548c00640 1 -- 172.21.15.32:0/3047778212 shutdown_connections 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.134+0000 7fd548c00640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440d11d0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.134+0000 7fd548c00640 1 --2- 172.21.15.32:0/3047778212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.134+0000 7fd548c00640 1 -- 172.21.15.32:0/3047778212 >> 172.21.15.32:0/3047778212 conn(0x7fd5440d3f60 msgr2=0x7fd5440d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.134+0000 7fd548c00640 1 -- 172.21.15.32:0/3047778212 shutdown_connections 2024-03-06T07:33:40.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.134+0000 7fd548c00640 1 -- 172.21.15.32:0/3047778212 wait complete. 2024-03-06T07:33:40.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.135+0000 7fd548c00640 1 Processor -- start 2024-03-06T07:33:40.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.136+0000 7fd548c00640 1 -- start start 2024-03-06T07:33:40.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd548c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd548c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5440d0d70 0x7fd5440b8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd548c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd548c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5440cfe20 con 0x7fd5440b50e0 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd548c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5440cfb40 con 0x7fd5440d0d70 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd548c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5440d2240 con 0x7fd54415e2e0 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52228/0 (socket says 172.21.15.32:52228) 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 -- 172.21.15.32:0/1744658038 learned_addr learned my addr 172.21.15.32:0/1744658038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 -- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 msgr2=0x7fd5440cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 -- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5440d0d70 msgr2=0x7fd5440b8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd53be00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5440d0d70 0x7fd5440b8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5440d0d70 0x7fd5440b8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.137+0000 7fd541e00640 1 -- 172.21.15.32:0/1744658038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5440cbaf0 con 0x7fd5440b50e0 2024-03-06T07:33:40.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.138+0000 7fd541e00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440ccb90 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd53400c7f0 tx=0x7fd53400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.138+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534015070 con 0x7fd5440b50e0 2024-03-06T07:33:40.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.138+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5440cbdd0 con 0x7fd5440b50e0 2024-03-06T07:33:40.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.138+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534007820 con 0x7fd5440b50e0 2024-03-06T07:33:40.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.138+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534007bd0 con 0x7fd5440b50e0 2024-03-06T07:33:40.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.138+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5440cc390 con 0x7fd5440b50e0 2024-03-06T07:33:40.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.140+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54404f3a0 con 0x7fd5440b50e0 2024-03-06T07:33:40.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.142+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd534018070 con 0x7fd5440b50e0 2024-03-06T07:33:40.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.143+0000 7fd53a000640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd52006b4d0 0x7fd52006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.144+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd53408a470 con 0x7fd5440b50e0 2024-03-06T07:33:40.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.144+0000 7fd53be00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd52006b4d0 0x7fd52006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.145+0000 7fd53be00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd52006b4d0 0x7fd52006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd5440ce320 tx=0x7fd52c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.148+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5340591c0 con 0x7fd5440b50e0 2024-03-06T07:33:40.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.346+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5440cc790 con 0x7fd52006b4d0 2024-03-06T07:33:40.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.355+0000 7fd53a000640 1 -- 172.21.15.32:0/1744658038 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252080 (secure 0 0 0) 0x7fd5440cc790 con 0x7fd52006b4d0 2024-03-06T07:33:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:33:40.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd52006b4d0 msgr2=0x7fd52006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd52006b4d0 0x7fd52006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd5440ce320 tx=0x7fd52c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:40.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 msgr2=0x7fd5440ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440ccb90 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd53400c7f0 tx=0x7fd53400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 shutdown_connections 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd52006b4d0 0x7fd52006d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd54415e2e0 0x7fd5440cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5440d0d70 0x7fd5440b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 --2- 172.21.15.32:0/1744658038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5440b50e0 0x7fd5440ccb90 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.360+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 >> 172.21.15.32:0/1744658038 conn(0x7fd5440d3f60 msgr2=0x7fd54406cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.361+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 shutdown_connections 2024-03-06T07:33:40.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.361+0000 7fd548c00640 1 -- 172.21.15.32:0/1744658038 wait complete. 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-03-06T07:33:40.272195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275200,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754699520,"statfs":{"total":773094113280,"available":772812308480,"internally_reserved":0,"allocated":9314304,"data_stored":4600300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001864"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744321+0000","last_change":"2024-03-06T07:33:35.433936+0000","last_active":"2024-03-06T07:33:36.744321+0000","last_peered":"2024-03-06T07:33:36.744321+0000","last_clean":"2024-03-06T07:33:36.744321+0000","last_became_active":"2024-03-06T07:33:35.433301+0000","last_became_peered":"2024-03-06T07:33:35.433301+0000","last_unstale":"2024-03-06T07:33:36.744321+0000","last_undegraded":"2024-03-06T07:33:36.744321+0000","last_fullsized":"2024-03-06T07:33:36.744321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.647203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415712+0000","last_change":"2024-03-06T07:33:35.434426+0000","last_active":"2024-03-06T07:33:36.415712+0000","last_peered":"2024-03-06T07:33:36.415712+0000","last_clean":"2024-03-06T07:33:36.415712+0000","last_became_active":"2024-03-06T07:33:35.434145+0000","last_became_peered":"2024-03-06T07:33:35.434145+0000","last_unstale":"2024-03-06T07:33:36.415712+0000","last_undegraded":"2024-03-06T07:33:36.415712+0000","last_fullsized":"2024-03-06T07:33:36.415712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:46.389191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416851+0000","last_change":"2024-03-06T07:33:35.423435+0000","last_active":"2024-03-06T07:33:36.416851+0000","last_peered":"2024-03-06T07:33:36.416851+0000","last_clean":"2024-03-06T07:33:36.416851+0000","last_became_active":"2024-03-06T07:33:35.423141+0000","last_became_peered":"2024-03-06T07:33:35.423141+0000","last_unstale":"2024-03-06T07:33:36.416851+0000","last_undegraded":"2024-03-06T07:33:36.416851+0000","last_fullsized":"2024-03-06T07:33:36.416851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:21.262143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740802+0000","last_change":"2024-03-06T07:33:35.411938+0000","last_active":"2024-03-06T07:33:36.740802+0000","last_peered":"2024-03-06T07:33:36.740802+0000","last_clean":"2024-03-06T07:33:36.740802+0000","last_became_active":"2024-03-06T07:33:35.411118+0000","last_became_peered":"2024-03-06T07:33:35.411118+0000","last_unstale":"2024-03-06T07:33:36.740802+0000","last_undegraded":"2024-03-06T07:33:36.740802+0000","last_fullsized":"2024-03-06T07:33:36.740802+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:48.685551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596739+0000","last_change":"2024-03-06T07:33:35.431875+0000","last_active":"2024-03-06T07:33:36.596739+0000","last_peered":"2024-03-06T07:33:36.596739+0000","last_clean":"2024-03-06T07:33:36.596739+0000","last_became_active":"2024-03-06T07:33:35.428073+0000","last_became_peered":"2024-03-06T07:33:35.428073+0000","last_unstale":"2024-03-06T07:33:36.596739+0000","last_undegraded":"2024-03-06T07:33:36.596739+0000","last_fullsized":"2024-03-06T07:33:36.596739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418765+0000","last_change":"2024-03-06T07:33:35.418942+0000","last_active":"2024-03-06T07:33:36.418765+0000","last_peered":"2024-03-06T07:33:36.418765+0000","last_clean":"2024-03-06T07:33:36.418765+0000","last_became_active":"2024-03-06T07:33:35.418647+0000","last_became_peered":"2024-03-06T07:33:35.418647+0000","last_unstale":"2024-03-06T07:33:36.418765+0000","last_undegraded":"2024-03-06T07:33:36.418765+0000","last_fullsized":"2024-03-06T07:33:36.418765+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:14.739091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740997+0000","last_change":"2024-03-06T07:33:35.419984+0000","last_active":"2024-03-06T07:33:36.740997+0000","last_peered":"2024-03-06T07:33:36.740997+0000","last_clean":"2024-03-06T07:33:36.740997+0000","last_became_active":"2024-03-06T07:33:35.419686+0000","last_became_peered":"2024-03-06T07:33:35.419686+0000","last_unstale":"2024-03-06T07:33:36.740997+0000","last_undegraded":"2024-03-06T07:33:36.740997+0000","last_fullsized":"2024-03-06T07:33:36.740997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:52.371043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418948+0000","last_change":"2024-03-06T07:33:35.425311+0000","last_active":"2024-03-06T07:33:36.418948+0000","last_peered":"2024-03-06T07:33:36.418948+0000","last_clean":"2024-03-06T07:33:36.418948+0000","last_became_active":"2024-03-06T07:33:35.425031+0000","last_became_peered":"2024-03-06T07:33:35.425031+0000","last_unstale":"2024-03-06T07:33:36.418948+0000","last_undegraded":"2024-03-06T07:33:36.418948+0000","last_fullsized":"2024-03-06T07:33:36.418948+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:10.216556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419802+0000","last_change":"2024-03-06T07:33:35.419640+0000","last_active":"2024-03-06T07:33:36.419802+0000","last_peered":"2024-03-06T07:33:36.419802+0000","last_clean":"2024-03-06T07:33:36.419802+0000","last_became_active":"2024-03-06T07:33:35.419308+0000","last_became_peered":"2024-03-06T07:33:35.419308+0000","last_unstale":"2024-03-06T07:33:36.419802+0000","last_undegraded":"2024-03-06T07:33:36.419802+0000","last_fullsized":"2024-03-06T07:33:36.419802+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:30.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744076+0000","last_change":"2024-03-06T07:33:35.424350+0000","last_active":"2024-03-06T07:33:36.744076+0000","last_peered":"2024-03-06T07:33:36.744076+0000","last_clean":"2024-03-06T07:33:36.744076+0000","last_became_active":"2024-03-06T07:33:35.422597+0000","last_became_peered":"2024-03-06T07:33:35.422597+0000","last_unstale":"2024-03-06T07:33:36.744076+0000","last_undegraded":"2024-03-06T07:33:36.744076+0000","last_fullsized":"2024-03-06T07:33:36.744076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415784+0000","last_change":"2024-03-06T07:33:35.418800+0000","last_active":"2024-03-06T07:33:36.415784+0000","last_peered":"2024-03-06T07:33:36.415784+0000","last_clean":"2024-03-06T07:33:36.415784+0000","last_became_active":"2024-03-06T07:33:35.418462+0000","last_became_peered":"2024-03-06T07:33:35.418462+0000","last_unstale":"2024-03-06T07:33:36.415784+0000","last_undegraded":"2024-03-06T07:33:36.415784+0000","last_fullsized":"2024-03-06T07:33:36.415784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:39.470631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743586+0000","last_change":"2024-03-06T07:33:35.431603+0000","last_active":"2024-03-06T07:33:36.743586+0000","last_peered":"2024-03-06T07:33:36.743586+0000","last_clean":"2024-03-06T07:33:36.743586+0000","last_became_active":"2024-03-06T07:33:35.431182+0000","last_became_peered":"2024-03-06T07:33:35.431182+0000","last_unstale":"2024-03-06T07:33:36.743586+0000","last_undegraded":"2024-03-06T07:33:36.743586+0000","last_fullsized":"2024-03-06T07:33:36.743586+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:06.823351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416560+0000","last_change":"2024-03-06T07:33:35.430473+0000","last_active":"2024-03-06T07:33:36.416560+0000","last_peered":"2024-03-06T07:33:36.416560+0000","last_clean":"2024-03-06T07:33:36.416560+0000","last_became_active":"2024-03-06T07:33:35.430134+0000","last_became_peered":"2024-03-06T07:33:35.430134+0000","last_unstale":"2024-03-06T07:33:36.416560+0000","last_undegraded":"2024-03-06T07:33:36.416560+0000","last_fullsized":"2024-03-06T07:33:36.416560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:30.593508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740863+0000","last_change":"2024-03-06T07:33:35.410474+0000","last_active":"2024-03-06T07:33:36.740863+0000","last_peered":"2024-03-06T07:33:36.740863+0000","last_clean":"2024-03-06T07:33:36.740863+0000","last_became_active":"2024-03-06T07:33:35.410096+0000","last_became_peered":"2024-03-06T07:33:35.410096+0000","last_unstale":"2024-03-06T07:33:36.740863+0000","last_undegraded":"2024-03-06T07:33:36.740863+0000","last_fullsized":"2024-03-06T07:33:36.740863+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.907427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745495+0000","last_change":"2024-03-06T07:33:35.422211+0000","last_active":"2024-03-06T07:33:36.745495+0000","last_peered":"2024-03-06T07:33:36.745495+0000","last_clean":"2024-03-06T07:33:36.745495+0000","last_became_active":"2024-03-06T07:33:35.421925+0000","last_became_peered":"2024-03-06T07:33:35.421925+0000","last_unstale":"2024-03-06T07:33:36.745495+0000","last_undegraded":"2024-03-06T07:33:36.745495+0000","last_fullsized":"2024-03-06T07:33:36.745495+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:57.891330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.595620+0000","last_change":"2024-03-06T07:33:35.432360+0000","last_active":"2024-03-06T07:33:36.595620+0000","last_peered":"2024-03-06T07:33:36.595620+0000","last_clean":"2024-03-06T07:33:36.595620+0000","last_became_active":"2024-03-06T07:33:35.432045+0000","last_became_peered":"2024-03-06T07:33:35.432045+0000","last_unstale":"2024-03-06T07:33:36.595620+0000","last_undegraded":"2024-03-06T07:33:36.595620+0000","last_fullsized":"2024-03-06T07:33:36.595620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:36.361275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596583+0000","last_change":"2024-03-06T07:33:35.423519+0000","last_active":"2024-03-06T07:33:36.596583+0000","last_peered":"2024-03-06T07:33:36.596583+0000","last_clean":"2024-03-06T07:33:36.596583+0000","last_became_active":"2024-03-06T07:33:35.422710+0000","last_became_peered":"2024-03-06T07:33:35.422710+0000","last_unstale":"2024-03-06T07:33:36.596583+0000","last_undegraded":"2024-03-06T07:33:36.596583+0000","last_fullsized":"2024-03-06T07:33:36.596583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:30.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741208+0000","last_change":"2024-03-06T07:33:35.434502+0000","last_active":"2024-03-06T07:33:36.741208+0000","last_peered":"2024-03-06T07:33:36.741208+0000","last_clean":"2024-03-06T07:33:36.741208+0000","last_became_active":"2024-03-06T07:33:35.434207+0000","last_became_peered":"2024-03-06T07:33:35.434207+0000","last_unstale":"2024-03-06T07:33:36.741208+0000","last_undegraded":"2024-03-06T07:33:36.741208+0000","last_fullsized":"2024-03-06T07:33:36.741208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:18.253855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418868+0000","last_change":"2024-03-06T07:33:35.416254+0000","last_active":"2024-03-06T07:33:36.418868+0000","last_peered":"2024-03-06T07:33:36.418868+0000","last_clean":"2024-03-06T07:33:36.418868+0000","last_became_active":"2024-03-06T07:33:35.415908+0000","last_became_peered":"2024-03-06T07:33:35.415908+0000","last_unstale":"2024-03-06T07:33:36.418868+0000","last_undegraded":"2024-03-06T07:33:36.418868+0000","last_fullsized":"2024-03-06T07:33:36.418868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:57.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740978+0000","last_change":"2024-03-06T07:33:35.415989+0000","last_active":"2024-03-06T07:33:36.740978+0000","last_peered":"2024-03-06T07:33:36.740978+0000","last_clean":"2024-03-06T07:33:36.740978+0000","last_became_active":"2024-03-06T07:33:35.415666+0000","last_became_peered":"2024-03-06T07:33:35.415666+0000","last_unstale":"2024-03-06T07:33:36.740978+0000","last_undegraded":"2024-03-06T07:33:36.740978+0000","last_fullsized":"2024-03-06T07:33:36.740978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:42.486342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740855+0000","last_change":"2024-03-06T07:33:35.408105+0000","last_active":"2024-03-06T07:33:36.740855+0000","last_peered":"2024-03-06T07:33:36.740855+0000","last_clean":"2024-03-06T07:33:36.740855+0000","last_became_active":"2024-03-06T07:33:35.407738+0000","last_became_peered":"2024-03-06T07:33:35.407738+0000","last_unstale":"2024-03-06T07:33:36.740855+0000","last_undegraded":"2024-03-06T07:33:36.740855+0000","last_fullsized":"2024-03-06T07:33:36.740855+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:38.466513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415370+0000","last_change":"2024-03-06T07:33:35.398830+0000","last_active":"2024-03-06T07:33:36.415370+0000","last_peered":"2024-03-06T07:33:36.415370+0000","last_clean":"2024-03-06T07:33:36.415370+0000","last_became_active":"2024-03-06T07:33:35.397897+0000","last_became_peered":"2024-03-06T07:33:35.397897+0000","last_unstale":"2024-03-06T07:33:36.415370+0000","last_undegraded":"2024-03-06T07:33:36.415370+0000","last_fullsized":"2024-03-06T07:33:36.415370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:20.709199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744116+0000","last_change":"2024-03-06T07:33:35.418391+0000","last_active":"2024-03-06T07:33:36.744116+0000","last_peered":"2024-03-06T07:33:36.744116+0000","last_clean":"2024-03-06T07:33:36.744116+0000","last_became_active":"2024-03-06T07:33:35.418084+0000","last_became_peered":"2024-03-06T07:33:35.418084+0000","last_unstale":"2024-03-06T07:33:36.744116+0000","last_undegraded":"2024-03-06T07:33:36.744116+0000","last_fullsized":"2024-03-06T07:33:36.744116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.623824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741350+0000","last_change":"2024-03-06T07:33:35.418948+0000","last_active":"2024-03-06T07:33:36.741350+0000","last_peered":"2024-03-06T07:33:36.741350+0000","last_clean":"2024-03-06T07:33:36.741350+0000","last_became_active":"2024-03-06T07:33:35.418641+0000","last_became_peered":"2024-03-06T07:33:35.418641+0000","last_unstale":"2024-03-06T07:33:36.741350+0000","last_undegraded":"2024-03-06T07:33:36.741350+0000","last_fullsized":"2024-03-06T07:33:36.741350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:07.673740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415759+0000","last_change":"2024-03-06T07:33:35.433657+0000","last_active":"2024-03-06T07:33:36.415759+0000","last_peered":"2024-03-06T07:33:36.415759+0000","last_clean":"2024-03-06T07:33:36.415759+0000","last_became_active":"2024-03-06T07:33:35.433321+0000","last_became_peered":"2024-03-06T07:33:35.433321+0000","last_unstale":"2024-03-06T07:33:36.415759+0000","last_undegraded":"2024-03-06T07:33:36.415759+0000","last_fullsized":"2024-03-06T07:33:36.415759+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:08.317174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417503+0000","last_change":"2024-03-06T07:33:35.426045+0000","last_active":"2024-03-06T07:33:36.417503+0000","last_peered":"2024-03-06T07:33:36.417503+0000","last_clean":"2024-03-06T07:33:36.417503+0000","last_became_active":"2024-03-06T07:33:35.425735+0000","last_became_peered":"2024-03-06T07:33:35.425735+0000","last_unstale":"2024-03-06T07:33:36.417503+0000","last_undegraded":"2024-03-06T07:33:36.417503+0000","last_fullsized":"2024-03-06T07:33:36.417503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:03.003568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596354+0000","last_change":"2024-03-06T07:33:35.425708+0000","last_active":"2024-03-06T07:33:36.596354+0000","last_peered":"2024-03-06T07:33:36.596354+0000","last_clean":"2024-03-06T07:33:36.596354+0000","last_became_active":"2024-03-06T07:33:35.425338+0000","last_became_peered":"2024-03-06T07:33:35.425338+0000","last_unstale":"2024-03-06T07:33:36.596354+0000","last_undegraded":"2024-03-06T07:33:36.596354+0000","last_fullsized":"2024-03-06T07:33:36.596354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:06.823351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743897+0000","last_change":"2024-03-06T07:33:35.431516+0000","last_active":"2024-03-06T07:33:36.743897+0000","last_peered":"2024-03-06T07:33:36.743897+0000","last_clean":"2024-03-06T07:33:36.743897+0000","last_became_active":"2024-03-06T07:33:35.430850+0000","last_became_peered":"2024-03-06T07:33:35.430850+0000","last_unstale":"2024-03-06T07:33:36.743897+0000","last_undegraded":"2024-03-06T07:33:36.743897+0000","last_fullsized":"2024-03-06T07:33:36.743897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:30.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415431+0000","last_change":"2024-03-06T07:33:35.420045+0000","last_active":"2024-03-06T07:33:36.415431+0000","last_peered":"2024-03-06T07:33:36.415431+0000","last_clean":"2024-03-06T07:33:36.415431+0000","last_became_active":"2024-03-06T07:33:35.419744+0000","last_became_peered":"2024-03-06T07:33:35.419744+0000","last_unstale":"2024-03-06T07:33:36.415431+0000","last_undegraded":"2024-03-06T07:33:36.415431+0000","last_fullsized":"2024-03-06T07:33:36.415431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:39.536433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415473+0000","last_change":"2024-03-06T07:33:35.414588+0000","last_active":"2024-03-06T07:33:36.415473+0000","last_peered":"2024-03-06T07:33:36.415473+0000","last_clean":"2024-03-06T07:33:36.415473+0000","last_became_active":"2024-03-06T07:33:35.413813+0000","last_became_peered":"2024-03-06T07:33:35.413813+0000","last_unstale":"2024-03-06T07:33:36.415473+0000","last_undegraded":"2024-03-06T07:33:36.415473+0000","last_fullsized":"2024-03-06T07:33:36.415473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:17.232099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596964+0000","last_change":"2024-03-06T07:33:35.421717+0000","last_active":"2024-03-06T07:33:36.596964+0000","last_peered":"2024-03-06T07:33:36.596964+0000","last_clean":"2024-03-06T07:33:36.596964+0000","last_became_active":"2024-03-06T07:33:35.421413+0000","last_became_peered":"2024-03-06T07:33:35.421413+0000","last_unstale":"2024-03-06T07:33:36.596964+0000","last_undegraded":"2024-03-06T07:33:36.596964+0000","last_fullsized":"2024-03-06T07:33:36.596964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:24.380160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743383+0000","last_change":"2024-03-06T07:33:35.431314+0000","last_active":"2024-03-06T07:33:36.743383+0000","last_peered":"2024-03-06T07:33:36.743383+0000","last_clean":"2024-03-06T07:33:36.743383+0000","last_became_active":"2024-03-06T07:33:35.430929+0000","last_became_peered":"2024-03-06T07:33:35.430929+0000","last_unstale":"2024-03-06T07:33:36.743383+0000","last_undegraded":"2024-03-06T07:33:36.743383+0000","last_fullsized":"2024-03-06T07:33:36.743383+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:10.216556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596799+0000","last_change":"2024-03-06T07:33:35.430067+0000","last_active":"2024-03-06T07:33:36.596799+0000","last_peered":"2024-03-06T07:33:36.596799+0000","last_clean":"2024-03-06T07:33:36.596799+0000","last_became_active":"2024-03-06T07:33:35.429736+0000","last_became_peered":"2024-03-06T07:33:35.429736+0000","last_unstale":"2024-03-06T07:33:36.596799+0000","last_undegraded":"2024-03-06T07:33:36.596799+0000","last_fullsized":"2024-03-06T07:33:36.596799+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:36.146350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744810+0000","last_change":"2024-03-06T07:33:34.418836+0000","last_active":"2024-03-06T07:33:36.744810+0000","last_peered":"2024-03-06T07:33:36.744810+0000","last_clean":"2024-03-06T07:33:36.744810+0000","last_became_active":"2024-03-06T07:33:34.418553+0000","last_became_peered":"2024-03-06T07:33:34.418553+0000","last_unstale":"2024-03-06T07:33:36.744810+0000","last_undegraded":"2024-03-06T07:33:36.744810+0000","last_fullsized":"2024-03-06T07:33:36.744810+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:58.543192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744583+0000","last_change":"2024-03-06T07:33:35.429003+0000","last_active":"2024-03-06T07:33:36.744583+0000","last_peered":"2024-03-06T07:33:36.744583+0000","last_clean":"2024-03-06T07:33:36.744583+0000","last_became_active":"2024-03-06T07:33:35.428592+0000","last_became_peered":"2024-03-06T07:33:35.428592+0000","last_unstale":"2024-03-06T07:33:36.744583+0000","last_undegraded":"2024-03-06T07:33:36.744583+0000","last_fullsized":"2024-03-06T07:33:36.744583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:15.522341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597197+0000","last_change":"2024-03-06T07:33:34.415694+0000","last_active":"2024-03-06T07:33:36.597197+0000","last_peered":"2024-03-06T07:33:36.597197+0000","last_clean":"2024-03-06T07:33:36.597197+0000","last_became_active":"2024-03-06T07:33:34.415343+0000","last_became_peered":"2024-03-06T07:33:34.415343+0000","last_unstale":"2024-03-06T07:33:36.597197+0000","last_undegraded":"2024-03-06T07:33:36.597197+0000","last_fullsized":"2024-03-06T07:33:36.597197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.269016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744118+0000","last_change":"2024-03-06T07:33:35.431346+0000","last_active":"2024-03-06T07:33:36.744118+0000","last_peered":"2024-03-06T07:33:36.744118+0000","last_clean":"2024-03-06T07:33:36.744118+0000","last_became_active":"2024-03-06T07:33:35.430823+0000","last_became_peered":"2024-03-06T07:33:35.430823+0000","last_unstale":"2024-03-06T07:33:36.744118+0000","last_undegraded":"2024-03-06T07:33:36.744118+0000","last_fullsized":"2024-03-06T07:33:36.744118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:36.146350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418889+0000","last_change":"2024-03-06T07:33:34.423715+0000","last_active":"2024-03-06T07:33:36.418889+0000","last_peered":"2024-03-06T07:33:36.418889+0000","last_clean":"2024-03-06T07:33:36.418889+0000","last_became_active":"2024-03-06T07:33:34.423391+0000","last_became_peered":"2024-03-06T07:33:34.423391+0000","last_unstale":"2024-03-06T07:33:36.418889+0000","last_undegraded":"2024-03-06T07:33:36.418889+0000","last_fullsized":"2024-03-06T07:33:36.418889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:35.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417128+0000","last_change":"2024-03-06T07:33:35.426012+0000","last_active":"2024-03-06T07:33:36.417128+0000","last_peered":"2024-03-06T07:33:36.417128+0000","last_clean":"2024-03-06T07:33:36.417128+0000","last_became_active":"2024-03-06T07:33:35.425256+0000","last_became_peered":"2024-03-06T07:33:35.425256+0000","last_unstale":"2024-03-06T07:33:36.417128+0000","last_undegraded":"2024-03-06T07:33:36.417128+0000","last_fullsized":"2024-03-06T07:33:36.417128+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:34.075715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419020+0000","last_change":"2024-03-06T07:33:34.425003+0000","last_active":"2024-03-06T07:33:36.419020+0000","last_peered":"2024-03-06T07:33:36.419020+0000","last_clean":"2024-03-06T07:33:36.419020+0000","last_became_active":"2024-03-06T07:33:34.422362+0000","last_became_peered":"2024-03-06T07:33:34.422362+0000","last_unstale":"2024-03-06T07:33:36.419020+0000","last_undegraded":"2024-03-06T07:33:36.419020+0000","last_fullsized":"2024-03-06T07:33:36.419020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:22.769198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415561+0000","last_change":"2024-03-06T07:33:35.414323+0000","last_active":"2024-03-06T07:33:36.415561+0000","last_peered":"2024-03-06T07:33:36.415561+0000","last_clean":"2024-03-06T07:33:36.415561+0000","last_became_active":"2024-03-06T07:33:35.413973+0000","last_became_peered":"2024-03-06T07:33:35.413973+0000","last_unstale":"2024-03-06T07:33:36.415561+0000","last_undegraded":"2024-03-06T07:33:36.415561+0000","last_fullsized":"2024-03-06T07:33:36.415561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:59.129104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597034+0000","last_change":"2024-03-06T07:33:34.414855+0000","last_active":"2024-03-06T07:33:36.597034+0000","last_peered":"2024-03-06T07:33:36.597034+0000","last_clean":"2024-03-06T07:33:36.597034+0000","last_became_active":"2024-03-06T07:33:34.414544+0000","last_became_peered":"2024-03-06T07:33:34.414544+0000","last_unstale":"2024-03-06T07:33:36.597034+0000","last_undegraded":"2024-03-06T07:33:36.597034+0000","last_fullsized":"2024-03-06T07:33:36.597034+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:04.291261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596297+0000","last_change":"2024-03-06T07:33:35.428151+0000","last_active":"2024-03-06T07:33:36.596297+0000","last_peered":"2024-03-06T07:33:36.596297+0000","last_clean":"2024-03-06T07:33:36.596297+0000","last_became_active":"2024-03-06T07:33:35.426950+0000","last_became_peered":"2024-03-06T07:33:35.426950+0000","last_unstale":"2024-03-06T07:33:36.596297+0000","last_undegraded":"2024-03-06T07:33:36.596297+0000","last_fullsized":"2024-03-06T07:33:36.596297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:10.216556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.595911+0000","last_change":"2024-03-06T07:33:34.416030+0000","last_active":"2024-03-06T07:33:36.595911+0000","last_peered":"2024-03-06T07:33:36.595911+0000","last_clean":"2024-03-06T07:33:36.595911+0000","last_became_active":"2024-03-06T07:33:34.415342+0000","last_became_peered":"2024-03-06T07:33:34.415342+0000","last_unstale":"2024-03-06T07:33:36.595911+0000","last_undegraded":"2024-03-06T07:33:36.595911+0000","last_fullsized":"2024-03-06T07:33:36.595911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:44.276115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415735+0000","last_change":"2024-03-06T07:33:35.413918+0000","last_active":"2024-03-06T07:33:36.415735+0000","last_peered":"2024-03-06T07:33:36.415735+0000","last_clean":"2024-03-06T07:33:36.415735+0000","last_became_active":"2024-03-06T07:33:35.413068+0000","last_became_peered":"2024-03-06T07:33:35.413068+0000","last_unstale":"2024-03-06T07:33:36.415735+0000","last_undegraded":"2024-03-06T07:33:36.415735+0000","last_fullsized":"2024-03-06T07:33:36.415735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:49.600775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418205+0000","last_change":"2024-03-06T07:33:34.423592+0000","last_active":"2024-03-06T07:33:36.418205+0000","last_peered":"2024-03-06T07:33:36.418205+0000","last_clean":"2024-03-06T07:33:36.418205+0000","last_became_active":"2024-03-06T07:33:34.423269+0000","last_became_peered":"2024-03-06T07:33:34.423269+0000","last_unstale":"2024-03-06T07:33:36.418205+0000","last_undegraded":"2024-03-06T07:33:36.418205+0000","last_fullsized":"2024-03-06T07:33:36.418205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.197881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745595+0000","last_change":"2024-03-06T07:33:35.421611+0000","last_active":"2024-03-06T07:33:36.745595+0000","last_peered":"2024-03-06T07:33:36.745595+0000","last_clean":"2024-03-06T07:33:36.745595+0000","last_became_active":"2024-03-06T07:33:35.421339+0000","last_became_peered":"2024-03-06T07:33:35.421339+0000","last_unstale":"2024-03-06T07:33:36.745595+0000","last_undegraded":"2024-03-06T07:33:36.745595+0000","last_fullsized":"2024-03-06T07:33:36.745595+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:07.286732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415126+0000","last_change":"2024-03-06T07:33:34.426732+0000","last_active":"2024-03-06T07:33:36.415126+0000","last_peered":"2024-03-06T07:33:36.415126+0000","last_clean":"2024-03-06T07:33:36.415126+0000","last_became_active":"2024-03-06T07:33:34.420142+0000","last_became_peered":"2024-03-06T07:33:34.420142+0000","last_unstale":"2024-03-06T07:33:36.415126+0000","last_undegraded":"2024-03-06T07:33:36.415126+0000","last_fullsized":"2024-03-06T07:33:36.415126+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:48.739973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745881+0000","last_change":"2024-03-06T07:33:35.424626+0000","last_active":"2024-03-06T07:33:36.745881+0000","last_peered":"2024-03-06T07:33:36.745881+0000","last_clean":"2024-03-06T07:33:36.745881+0000","last_became_active":"2024-03-06T07:33:35.424322+0000","last_became_peered":"2024-03-06T07:33:35.424322+0000","last_unstale":"2024-03-06T07:33:36.745881+0000","last_undegraded":"2024-03-06T07:33:36.745881+0000","last_fullsized":"2024-03-06T07:33:36.745881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:08.630175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744899+0000","last_change":"2024-03-06T07:33:34.422749+0000","last_active":"2024-03-06T07:33:36.744899+0000","last_peered":"2024-03-06T07:33:36.744899+0000","last_clean":"2024-03-06T07:33:36.744899+0000","last_became_active":"2024-03-06T07:33:34.420917+0000","last_became_peered":"2024-03-06T07:33:34.420917+0000","last_unstale":"2024-03-06T07:33:36.744899+0000","last_undegraded":"2024-03-06T07:33:36.744899+0000","last_fullsized":"2024-03-06T07:33:36.744899+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:24.022728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419851+0000","last_change":"2024-03-06T07:33:35.420287+0000","last_active":"2024-03-06T07:33:36.419851+0000","last_peered":"2024-03-06T07:33:36.419851+0000","last_clean":"2024-03-06T07:33:36.419851+0000","last_became_active":"2024-03-06T07:33:35.419979+0000","last_became_peered":"2024-03-06T07:33:35.419979+0000","last_unstale":"2024-03-06T07:33:36.419851+0000","last_undegraded":"2024-03-06T07:33:36.419851+0000","last_fullsized":"2024-03-06T07:33:36.419851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744455+0000","last_change":"2024-03-06T07:33:34.426256+0000","last_active":"2024-03-06T07:33:36.744455+0000","last_peered":"2024-03-06T07:33:36.744455+0000","last_clean":"2024-03-06T07:33:36.744455+0000","last_became_active":"2024-03-06T07:33:34.425970+0000","last_became_peered":"2024-03-06T07:33:34.425970+0000","last_unstale":"2024-03-06T07:33:36.744455+0000","last_undegraded":"2024-03-06T07:33:36.744455+0000","last_fullsized":"2024-03-06T07:33:36.744455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:24.319514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418894+0000","last_change":"2024-03-06T07:33:35.433575+0000","last_active":"2024-03-06T07:33:36.418894+0000","last_peered":"2024-03-06T07:33:36.418894+0000","last_clean":"2024-03-06T07:33:36.418894+0000","last_became_active":"2024-03-06T07:33:35.433276+0000","last_became_peered":"2024-03-06T07:33:35.433276+0000","last_unstale":"2024-03-06T07:33:36.418894+0000","last_undegraded":"2024-03-06T07:33:36.418894+0000","last_fullsized":"2024-03-06T07:33:36.418894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:30.593508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743670+0000","last_change":"2024-03-06T07:33:34.427477+0000","last_active":"2024-03-06T07:33:36.743670+0000","last_peered":"2024-03-06T07:33:36.743670+0000","last_clean":"2024-03-06T07:33:36.743670+0000","last_became_active":"2024-03-06T07:33:34.426590+0000","last_became_peered":"2024-03-06T07:33:34.426590+0000","last_unstale":"2024-03-06T07:33:36.743670+0000","last_undegraded":"2024-03-06T07:33:36.743670+0000","last_fullsized":"2024-03-06T07:33:36.743670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.269016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596418+0000","last_change":"2024-03-06T07:33:35.422229+0000","last_active":"2024-03-06T07:33:36.596418+0000","last_peered":"2024-03-06T07:33:36.596418+0000","last_clean":"2024-03-06T07:33:36.596418+0000","last_became_active":"2024-03-06T07:33:35.421940+0000","last_became_peered":"2024-03-06T07:33:35.421940+0000","last_unstale":"2024-03-06T07:33:36.596418+0000","last_undegraded":"2024-03-06T07:33:36.596418+0000","last_fullsized":"2024-03-06T07:33:36.596418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:13.287392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419620+0000","last_change":"2024-03-06T07:33:34.415820+0000","last_active":"2024-03-06T07:33:36.419620+0000","last_peered":"2024-03-06T07:33:36.419620+0000","last_clean":"2024-03-06T07:33:36.419620+0000","last_became_active":"2024-03-06T07:33:34.415298+0000","last_became_peered":"2024-03-06T07:33:34.415298+0000","last_unstale":"2024-03-06T07:33:36.419620+0000","last_undegraded":"2024-03-06T07:33:36.419620+0000","last_fullsized":"2024-03-06T07:33:36.419620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:13.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418447+0000","last_change":"2024-03-06T07:33:35.428922+0000","last_active":"2024-03-06T07:33:36.418447+0000","last_peered":"2024-03-06T07:33:36.418447+0000","last_clean":"2024-03-06T07:33:36.418447+0000","last_became_active":"2024-03-06T07:33:35.428523+0000","last_became_peered":"2024-03-06T07:33:35.428523+0000","last_unstale":"2024-03-06T07:33:36.418447+0000","last_undegraded":"2024-03-06T07:33:36.418447+0000","last_fullsized":"2024-03-06T07:33:36.418447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:20.620205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415553+0000","last_change":"2024-03-06T07:33:34.417859+0000","last_active":"2024-03-06T07:33:36.415553+0000","last_peered":"2024-03-06T07:33:36.415553+0000","last_clean":"2024-03-06T07:33:36.415553+0000","last_became_active":"2024-03-06T07:33:34.417563+0000","last_became_peered":"2024-03-06T07:33:34.417563+0000","last_unstale":"2024-03-06T07:33:36.415553+0000","last_undegraded":"2024-03-06T07:33:36.415553+0000","last_fullsized":"2024-03-06T07:33:36.415553+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:33.650899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416609+0000","last_change":"2024-03-06T07:33:35.414333+0000","last_active":"2024-03-06T07:33:36.416609+0000","last_peered":"2024-03-06T07:33:36.416609+0000","last_clean":"2024-03-06T07:33:36.416609+0000","last_became_active":"2024-03-06T07:33:35.413343+0000","last_became_peered":"2024-03-06T07:33:35.413343+0000","last_unstale":"2024-03-06T07:33:36.416609+0000","last_undegraded":"2024-03-06T07:33:36.416609+0000","last_fullsized":"2024-03-06T07:33:36.416609+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:00.522817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745557+0000","last_change":"2024-03-06T07:33:34.414094+0000","last_active":"2024-03-06T07:33:36.745557+0000","last_peered":"2024-03-06T07:33:36.745557+0000","last_clean":"2024-03-06T07:33:36.745557+0000","last_became_active":"2024-03-06T07:33:34.413811+0000","last_became_peered":"2024-03-06T07:33:34.413811+0000","last_unstale":"2024-03-06T07:33:36.745557+0000","last_undegraded":"2024-03-06T07:33:36.745557+0000","last_fullsized":"2024-03-06T07:33:36.745557+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:20.243469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415111+0000","last_change":"2024-03-06T07:33:35.425618+0000","last_active":"2024-03-06T07:33:36.415111+0000","last_peered":"2024-03-06T07:33:36.415111+0000","last_clean":"2024-03-06T07:33:36.415111+0000","last_became_active":"2024-03-06T07:33:35.425290+0000","last_became_peered":"2024-03-06T07:33:35.425290+0000","last_unstale":"2024-03-06T07:33:36.415111+0000","last_undegraded":"2024-03-06T07:33:36.415111+0000","last_fullsized":"2024-03-06T07:33:36.415111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:51.337308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419035+0000","last_change":"2024-03-06T07:33:34.420849+0000","last_active":"2024-03-06T07:33:36.419035+0000","last_peered":"2024-03-06T07:33:36.419035+0000","last_clean":"2024-03-06T07:33:36.419035+0000","last_became_active":"2024-03-06T07:33:34.420527+0000","last_became_peered":"2024-03-06T07:33:34.420527+0000","last_unstale":"2024-03-06T07:33:36.419035+0000","last_undegraded":"2024-03-06T07:33:36.419035+0000","last_fullsized":"2024-03-06T07:33:36.419035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:06.907010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415094+0000","last_change":"2024-03-06T07:33:35.421912+0000","last_active":"2024-03-06T07:33:36.415094+0000","last_peered":"2024-03-06T07:33:36.415094+0000","last_clean":"2024-03-06T07:33:36.415094+0000","last_became_active":"2024-03-06T07:33:35.421596+0000","last_became_peered":"2024-03-06T07:33:35.421596+0000","last_unstale":"2024-03-06T07:33:36.415094+0000","last_undegraded":"2024-03-06T07:33:36.415094+0000","last_fullsized":"2024-03-06T07:33:36.415094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:55.138916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418122+0000","last_change":"2024-03-06T07:33:34.421400+0000","last_active":"2024-03-06T07:33:36.418122+0000","last_peered":"2024-03-06T07:33:36.418122+0000","last_clean":"2024-03-06T07:33:36.418122+0000","last_became_active":"2024-03-06T07:33:34.420914+0000","last_became_peered":"2024-03-06T07:33:34.420914+0000","last_unstale":"2024-03-06T07:33:36.418122+0000","last_undegraded":"2024-03-06T07:33:36.418122+0000","last_fullsized":"2024-03-06T07:33:36.418122+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:05.804677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417604+0000","last_change":"2024-03-06T07:33:35.420920+0000","last_active":"2024-03-06T07:33:36.417604+0000","last_peered":"2024-03-06T07:33:36.417604+0000","last_clean":"2024-03-06T07:33:36.417604+0000","last_became_active":"2024-03-06T07:33:35.418326+0000","last_became_peered":"2024-03-06T07:33:35.418326+0000","last_unstale":"2024-03-06T07:33:36.417604+0000","last_undegraded":"2024-03-06T07:33:36.417604+0000","last_fullsized":"2024-03-06T07:33:36.417604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:12.227545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745123+0000","last_change":"2024-03-06T07:33:34.422775+0000","last_active":"2024-03-06T07:33:36.745123+0000","last_peered":"2024-03-06T07:33:36.745123+0000","last_clean":"2024-03-06T07:33:36.745123+0000","last_became_active":"2024-03-06T07:33:34.422386+0000","last_became_peered":"2024-03-06T07:33:34.422386+0000","last_unstale":"2024-03-06T07:33:36.745123+0000","last_undegraded":"2024-03-06T07:33:36.745123+0000","last_fullsized":"2024-03-06T07:33:36.745123+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:06.907010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744714+0000","last_change":"2024-03-06T07:33:35.429397+0000","last_active":"2024-03-06T07:33:36.744714+0000","last_peered":"2024-03-06T07:33:36.744714+0000","last_clean":"2024-03-06T07:33:36.744714+0000","last_became_active":"2024-03-06T07:33:35.429020+0000","last_became_peered":"2024-03-06T07:33:35.429020+0000","last_unstale":"2024-03-06T07:33:36.744714+0000","last_undegraded":"2024-03-06T07:33:36.744714+0000","last_fullsized":"2024-03-06T07:33:36.744714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:18.663607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418967+0000","last_change":"2024-03-06T07:33:34.426275+0000","last_active":"2024-03-06T07:33:36.418967+0000","last_peered":"2024-03-06T07:33:36.418967+0000","last_clean":"2024-03-06T07:33:36.418967+0000","last_became_active":"2024-03-06T07:33:34.425954+0000","last_became_peered":"2024-03-06T07:33:34.425954+0000","last_unstale":"2024-03-06T07:33:36.418967+0000","last_undegraded":"2024-03-06T07:33:36.418967+0000","last_fullsized":"2024-03-06T07:33:36.418967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:29.964030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596828+0000","last_change":"2024-03-06T07:33:35.430415+0000","last_active":"2024-03-06T07:33:36.596828+0000","last_peered":"2024-03-06T07:33:36.596828+0000","last_clean":"2024-03-06T07:33:36.596828+0000","last_became_active":"2024-03-06T07:33:35.430101+0000","last_became_peered":"2024-03-06T07:33:35.430101+0000","last_unstale":"2024-03-06T07:33:36.596828+0000","last_undegraded":"2024-03-06T07:33:36.596828+0000","last_fullsized":"2024-03-06T07:33:36.596828+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.647203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419150+0000","last_change":"2024-03-06T07:33:34.426447+0000","last_active":"2024-03-06T07:33:36.419150+0000","last_peered":"2024-03-06T07:33:36.419150+0000","last_clean":"2024-03-06T07:33:36.419150+0000","last_became_active":"2024-03-06T07:33:34.426131+0000","last_became_peered":"2024-03-06T07:33:34.426131+0000","last_unstale":"2024-03-06T07:33:36.419150+0000","last_undegraded":"2024-03-06T07:33:36.419150+0000","last_fullsized":"2024-03-06T07:33:36.419150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:58.605149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741010+0000","last_change":"2024-03-06T07:33:35.411593+0000","last_active":"2024-03-06T07:33:36.741010+0000","last_peered":"2024-03-06T07:33:36.741010+0000","last_clean":"2024-03-06T07:33:36.741010+0000","last_became_active":"2024-03-06T07:33:35.410483+0000","last_became_peered":"2024-03-06T07:33:35.410483+0000","last_unstale":"2024-03-06T07:33:36.741010+0000","last_undegraded":"2024-03-06T07:33:36.741010+0000","last_fullsized":"2024-03-06T07:33:36.741010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:35.970261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745635+0000","last_change":"2024-03-06T07:33:34.421867+0000","last_active":"2024-03-06T07:33:36.745635+0000","last_peered":"2024-03-06T07:33:36.745635+0000","last_clean":"2024-03-06T07:33:36.745635+0000","last_became_active":"2024-03-06T07:33:34.421601+0000","last_became_peered":"2024-03-06T07:33:34.421601+0000","last_unstale":"2024-03-06T07:33:36.745635+0000","last_undegraded":"2024-03-06T07:33:36.745635+0000","last_fullsized":"2024-03-06T07:33:36.745635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:33.057040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419424+0000","last_change":"2024-03-06T07:33:35.417503+0000","last_active":"2024-03-06T07:33:36.419424+0000","last_peered":"2024-03-06T07:33:36.419424+0000","last_clean":"2024-03-06T07:33:36.419424+0000","last_became_active":"2024-03-06T07:33:35.417120+0000","last_became_peered":"2024-03-06T07:33:35.417120+0000","last_unstale":"2024-03-06T07:33:36.419424+0000","last_undegraded":"2024-03-06T07:33:36.419424+0000","last_fullsized":"2024-03-06T07:33:36.419424+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:36.361275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741076+0000","last_change":"2024-03-06T07:33:34.428159+0000","last_active":"2024-03-06T07:33:36.741076+0000","last_peered":"2024-03-06T07:33:36.741076+0000","last_clean":"2024-03-06T07:33:36.741076+0000","last_became_active":"2024-03-06T07:33:34.427845+0000","last_became_peered":"2024-03-06T07:33:34.427845+0000","last_unstale":"2024-03-06T07:33:36.741076+0000","last_undegraded":"2024-03-06T07:33:36.741076+0000","last_fullsized":"2024-03-06T07:33:36.741076+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:58.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741051+0000","last_change":"2024-03-06T07:33:35.419019+0000","last_active":"2024-03-06T07:33:36.741051+0000","last_peered":"2024-03-06T07:33:36.741051+0000","last_clean":"2024-03-06T07:33:36.741051+0000","last_became_active":"2024-03-06T07:33:35.418635+0000","last_became_peered":"2024-03-06T07:33:35.418635+0000","last_unstale":"2024-03-06T07:33:36.741051+0000","last_undegraded":"2024-03-06T07:33:36.741051+0000","last_fullsized":"2024-03-06T07:33:36.741051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:33.188533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741275+0000","last_change":"2024-03-06T07:33:34.424418+0000","last_active":"2024-03-06T07:33:36.741275+0000","last_peered":"2024-03-06T07:33:36.741275+0000","last_clean":"2024-03-06T07:33:36.741275+0000","last_became_active":"2024-03-06T07:33:34.424027+0000","last_became_peered":"2024-03-06T07:33:34.424027+0000","last_unstale":"2024-03-06T07:33:36.741275+0000","last_undegraded":"2024-03-06T07:33:36.741275+0000","last_fullsized":"2024-03-06T07:33:36.741275+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:59.075986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419137+0000","last_change":"2024-03-06T07:33:35.419004+0000","last_active":"2024-03-06T07:33:36.419137+0000","last_peered":"2024-03-06T07:33:36.419137+0000","last_clean":"2024-03-06T07:33:36.419137+0000","last_became_active":"2024-03-06T07:33:35.418699+0000","last_became_peered":"2024-03-06T07:33:35.418699+0000","last_unstale":"2024-03-06T07:33:36.419137+0000","last_undegraded":"2024-03-06T07:33:36.419137+0000","last_fullsized":"2024-03-06T07:33:36.419137+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:06.823351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745410+0000","last_change":"2024-03-06T07:33:34.417603+0000","last_active":"2024-03-06T07:33:36.745410+0000","last_peered":"2024-03-06T07:33:36.745410+0000","last_clean":"2024-03-06T07:33:36.745410+0000","last_became_active":"2024-03-06T07:33:34.417330+0000","last_became_peered":"2024-03-06T07:33:34.417330+0000","last_unstale":"2024-03-06T07:33:36.745410+0000","last_undegraded":"2024-03-06T07:33:36.745410+0000","last_fullsized":"2024-03-06T07:33:36.745410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:21.409894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597025+0000","last_change":"2024-03-06T07:33:35.427378+0000","last_active":"2024-03-06T07:33:36.597025+0000","last_peered":"2024-03-06T07:33:36.597025+0000","last_clean":"2024-03-06T07:33:36.597025+0000","last_became_active":"2024-03-06T07:33:35.426877+0000","last_became_peered":"2024-03-06T07:33:35.426877+0000","last_unstale":"2024-03-06T07:33:36.597025+0000","last_undegraded":"2024-03-06T07:33:36.597025+0000","last_fullsized":"2024-03-06T07:33:36.597025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:15.522341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415812+0000","last_change":"2024-03-06T07:33:34.424870+0000","last_active":"2024-03-06T07:33:36.415812+0000","last_peered":"2024-03-06T07:33:36.415812+0000","last_clean":"2024-03-06T07:33:36.415812+0000","last_became_active":"2024-03-06T07:33:34.421161+0000","last_became_peered":"2024-03-06T07:33:34.421161+0000","last_unstale":"2024-03-06T07:33:36.415812+0000","last_undegraded":"2024-03-06T07:33:36.415812+0000","last_fullsized":"2024-03-06T07:33:36.415812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:47.666877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416825+0000","last_change":"2024-03-06T07:33:35.434446+0000","last_active":"2024-03-06T07:33:36.416825+0000","last_peered":"2024-03-06T07:33:36.416825+0000","last_clean":"2024-03-06T07:33:36.416825+0000","last_became_active":"2024-03-06T07:33:35.434115+0000","last_became_peered":"2024-03-06T07:33:35.434115+0000","last_unstale":"2024-03-06T07:33:36.416825+0000","last_undegraded":"2024-03-06T07:33:36.416825+0000","last_fullsized":"2024-03-06T07:33:36.416825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:22.428569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417176+0000","last_change":"2024-03-06T07:33:34.418898+0000","last_active":"2024-03-06T07:33:36.417176+0000","last_peered":"2024-03-06T07:33:36.417176+0000","last_clean":"2024-03-06T07:33:36.417176+0000","last_became_active":"2024-03-06T07:33:34.418627+0000","last_became_peered":"2024-03-06T07:33:34.418627+0000","last_unstale":"2024-03-06T07:33:36.417176+0000","last_undegraded":"2024-03-06T07:33:36.417176+0000","last_fullsized":"2024-03-06T07:33:36.417176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:49.811096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744386+0000","last_change":"2024-03-06T07:33:35.421129+0000","last_active":"2024-03-06T07:33:36.744386+0000","last_peered":"2024-03-06T07:33:36.744386+0000","last_clean":"2024-03-06T07:33:36.744386+0000","last_became_active":"2024-03-06T07:33:35.420533+0000","last_became_peered":"2024-03-06T07:33:35.420533+0000","last_unstale":"2024-03-06T07:33:36.744386+0000","last_undegraded":"2024-03-06T07:33:36.744386+0000","last_fullsized":"2024-03-06T07:33:36.744386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:24.380160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743987+0000","last_change":"2024-03-06T07:33:34.424152+0000","last_active":"2024-03-06T07:33:36.743987+0000","last_peered":"2024-03-06T07:33:36.743987+0000","last_clean":"2024-03-06T07:33:36.743987+0000","last_became_active":"2024-03-06T07:33:34.423736+0000","last_became_peered":"2024-03-06T07:33:34.423736+0000","last_unstale":"2024-03-06T07:33:36.743987+0000","last_undegraded":"2024-03-06T07:33:36.743987+0000","last_fullsized":"2024-03-06T07:33:36.743987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:44.276115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596769+0000","last_change":"2024-03-06T07:33:35.419310+0000","last_active":"2024-03-06T07:33:36.596769+0000","last_peered":"2024-03-06T07:33:36.596769+0000","last_clean":"2024-03-06T07:33:36.596769+0000","last_became_active":"2024-03-06T07:33:35.419026+0000","last_became_peered":"2024-03-06T07:33:35.419026+0000","last_unstale":"2024-03-06T07:33:36.596769+0000","last_undegraded":"2024-03-06T07:33:36.596769+0000","last_fullsized":"2024-03-06T07:33:36.596769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.623824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744484+0000","last_change":"2024-03-06T07:33:34.427936+0000","last_active":"2024-03-06T07:33:36.744484+0000","last_peered":"2024-03-06T07:33:36.744484+0000","last_clean":"2024-03-06T07:33:36.744484+0000","last_became_active":"2024-03-06T07:33:34.427268+0000","last_became_peered":"2024-03-06T07:33:34.427268+0000","last_unstale":"2024-03-06T07:33:36.744484+0000","last_undegraded":"2024-03-06T07:33:36.744484+0000","last_fullsized":"2024-03-06T07:33:36.744484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:04.291261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419173+0000","last_change":"2024-03-06T07:33:35.424420+0000","last_active":"2024-03-06T07:33:36.419173+0000","last_peered":"2024-03-06T07:33:36.419173+0000","last_clean":"2024-03-06T07:33:36.419173+0000","last_became_active":"2024-03-06T07:33:35.423970+0000","last_became_peered":"2024-03-06T07:33:35.423970+0000","last_unstale":"2024-03-06T07:33:36.419173+0000","last_undegraded":"2024-03-06T07:33:36.419173+0000","last_fullsized":"2024-03-06T07:33:36.419173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:13.287392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418644+0000","last_change":"2024-03-06T07:33:34.426244+0000","last_active":"2024-03-06T07:33:36.418644+0000","last_peered":"2024-03-06T07:33:36.418644+0000","last_clean":"2024-03-06T07:33:36.418644+0000","last_became_active":"2024-03-06T07:33:34.425587+0000","last_became_peered":"2024-03-06T07:33:34.425587+0000","last_unstale":"2024-03-06T07:33:36.418644+0000","last_undegraded":"2024-03-06T07:33:36.418644+0000","last_fullsized":"2024-03-06T07:33:36.418644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:12.268717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418620+0000","last_change":"2024-03-06T07:33:35.417892+0000","last_active":"2024-03-06T07:33:36.418620+0000","last_peered":"2024-03-06T07:33:36.418620+0000","last_clean":"2024-03-06T07:33:36.418620+0000","last_became_active":"2024-03-06T07:33:35.417259+0000","last_became_peered":"2024-03-06T07:33:35.417259+0000","last_unstale":"2024-03-06T07:33:36.418620+0000","last_undegraded":"2024-03-06T07:33:36.418620+0000","last_fullsized":"2024-03-06T07:33:36.418620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:49.600775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597222+0000","last_change":"2024-03-06T07:33:34.415291+0000","last_active":"2024-03-06T07:33:36.597222+0000","last_peered":"2024-03-06T07:33:36.597222+0000","last_clean":"2024-03-06T07:33:36.597222+0000","last_became_active":"2024-03-06T07:33:34.412160+0000","last_became_peered":"2024-03-06T07:33:34.412160+0000","last_unstale":"2024-03-06T07:33:36.597222+0000","last_undegraded":"2024-03-06T07:33:36.597222+0000","last_fullsized":"2024-03-06T07:33:36.597222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:58.543192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415755+0000","last_change":"2024-03-06T07:33:35.426401+0000","last_active":"2024-03-06T07:33:36.415755+0000","last_peered":"2024-03-06T07:33:36.415755+0000","last_clean":"2024-03-06T07:33:36.415755+0000","last_became_active":"2024-03-06T07:33:35.425716+0000","last_became_peered":"2024-03-06T07:33:35.425716+0000","last_unstale":"2024-03-06T07:33:36.415755+0000","last_undegraded":"2024-03-06T07:33:36.415755+0000","last_fullsized":"2024-03-06T07:33:36.415755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:57.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744556+0000","last_change":"2024-03-06T07:33:34.422229+0000","last_active":"2024-03-06T07:33:36.744556+0000","last_peered":"2024-03-06T07:33:36.744556+0000","last_clean":"2024-03-06T07:33:36.744556+0000","last_became_active":"2024-03-06T07:33:34.420817+0000","last_became_peered":"2024-03-06T07:33:34.420817+0000","last_unstale":"2024-03-06T07:33:36.744556+0000","last_undegraded":"2024-03-06T07:33:36.744556+0000","last_fullsized":"2024-03-06T07:33:36.744556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:38.451957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743593+0000","last_change":"2024-03-06T07:33:35.427030+0000","last_active":"2024-03-06T07:33:36.743593+0000","last_peered":"2024-03-06T07:33:36.743593+0000","last_clean":"2024-03-06T07:33:36.743593+0000","last_became_active":"2024-03-06T07:33:35.426399+0000","last_became_peered":"2024-03-06T07:33:35.426399+0000","last_unstale":"2024-03-06T07:33:36.743593+0000","last_undegraded":"2024-03-06T07:33:36.743593+0000","last_fullsized":"2024-03-06T07:33:36.743593+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:13.287392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.595851+0000","last_change":"2024-03-06T07:33:34.419681+0000","last_active":"2024-03-06T07:33:36.595851+0000","last_peered":"2024-03-06T07:33:36.595851+0000","last_clean":"2024-03-06T07:33:36.595851+0000","last_became_active":"2024-03-06T07:33:34.418719+0000","last_became_peered":"2024-03-06T07:33:34.418719+0000","last_unstale":"2024-03-06T07:33:36.595851+0000","last_undegraded":"2024-03-06T07:33:36.595851+0000","last_fullsized":"2024-03-06T07:33:36.595851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:43.327416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415634+0000","last_change":"2024-03-06T07:33:35.418209+0000","last_active":"2024-03-06T07:33:36.415634+0000","last_peered":"2024-03-06T07:33:36.415634+0000","last_clean":"2024-03-06T07:33:36.415634+0000","last_became_active":"2024-03-06T07:33:35.417960+0000","last_became_peered":"2024-03-06T07:33:35.417960+0000","last_unstale":"2024-03-06T07:33:36.415634+0000","last_undegraded":"2024-03-06T07:33:36.415634+0000","last_fullsized":"2024-03-06T07:33:36.415634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:13.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417118+0000","last_change":"2024-03-06T07:33:34.424965+0000","last_active":"2024-03-06T07:33:36.417118+0000","last_peered":"2024-03-06T07:33:36.417118+0000","last_clean":"2024-03-06T07:33:36.417118+0000","last_became_active":"2024-03-06T07:33:34.424153+0000","last_became_peered":"2024-03-06T07:33:34.424153+0000","last_unstale":"2024-03-06T07:33:36.417118+0000","last_undegraded":"2024-03-06T07:33:36.417118+0000","last_fullsized":"2024-03-06T07:33:36.417118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:29.964030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":286,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415966+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:33:36.415966+0000","last_peered":"2024-03-06T07:33:36.415966+0000","last_clean":"2024-03-06T07:33:36.415966+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:33:36.415966+0000","last_undegraded":"2024-03-06T07:33:36.415966+0000","last_fullsized":"2024-03-06T07:33:36.415966+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1017281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1019904,"data_stored":427537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":428477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36324,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335516,"statfs":{"total":96636764160,"available":96599568384,"internally_reserved":0,"allocated":1007616,"data_stored":426843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":428455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":427425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":427037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":1605632,"data_stored":1017245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:40.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:40.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.637+0000 7f9067200640 1 Processor -- start 2024-03-06T07:33:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.637+0000 7f9067200640 1 -- start start 2024-03-06T07:33:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9067200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90680cd760 0x7f906815d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9067200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f906815d970 0x7f906806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9067200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9068159cc0 con 0x7f90681598e0 2024-03-06T07:33:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9067200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906806e3f0 con 0x7f906815d970 2024-03-06T07:33:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9067200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90680729b0 con 0x7f90680cd760 2024-03-06T07:33:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9066800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f906815d970 0x7f906806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9066800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f906815d970 0x7f906806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52232/0 (socket says 172.21.15.32:52232) 2024-03-06T07:33:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.639+0000 7f9066800640 1 -- 172.21.15.32:0/2911920286 learned_addr learned my addr 172.21.15.32:0/2911920286 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.640+0000 7f905d400640 1 --2- 172.21.15.32:0/2911920286 >> v2:172.21.15.39:3300/0 conn(0x7f90680cd760 0x7f906815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.640+0000 7f9066800640 1 -- 172.21.15.32:0/2911920286 >> v2:172.21.15.39:3300/0 conn(0x7f90680cd760 msgr2=0x7f906815d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.640+0000 7f9066800640 1 --2- 172.21.15.32:0/2911920286 >> v2:172.21.15.39:3300/0 conn(0x7f90680cd760 0x7f906815d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.640+0000 7f9066800640 1 -- 172.21.15.32:0/2911920286 >> v1:172.21.15.39:6789/0 conn(0x7f90681598e0 legacy=0x7f906806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.640+0000 7f9066800640 1 -- 172.21.15.32:0/2911920286 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068072c50 con 0x7f906815d970 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.641+0000 7f9066800640 1 --2- 172.21.15.32:0/2911920286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f906806afb0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f90680ceeb0 tx=0x7f906002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4455041141a92c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.641+0000 7f9065400640 1 -- 172.21.15.32:0/2911920286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906002ca80 con 0x7f906815d970 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.641+0000 7f9065400640 1 -- 172.21.15.32:0/2911920286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906002cc20 con 0x7f906815d970 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.641+0000 7f9065400640 1 -- 172.21.15.32:0/2911920286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90600343f0 con 0x7f906815d970 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 -- 172.21.15.32:0/2911920286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 msgr2=0x7f906806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 --2- 172.21.15.32:0/2911920286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f906806afb0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f90680ceeb0 tx=0x7f906002bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 -- 172.21.15.32:0/2911920286 shutdown_connections 2024-03-06T07:33:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 --2- 172.21.15.32:0/2911920286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f906806afb0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 --2- 172.21.15.32:0/2911920286 >> v2:172.21.15.39:3300/0 conn(0x7f90680cd760 0x7f906815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 -- 172.21.15.32:0/2911920286 >> 172.21.15.32:0/2911920286 conn(0x7f906806c0c0 msgr2=0x7f906806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 -- 172.21.15.32:0/2911920286 shutdown_connections 2024-03-06T07:33:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.642+0000 7f9067200640 1 -- 172.21.15.32:0/2911920286 wait complete. 2024-03-06T07:33:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.644+0000 7f9067200640 1 Processor -- start 2024-03-06T07:33:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.644+0000 7f9067200640 1 -- start start 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9067200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90680cd760 0x7f90680ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9067200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90681598e0 0x7f90681541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9067200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f90681f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9067200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906806e080 con 0x7f906815d970 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9067200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906806dda0 con 0x7f90681598e0 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9067200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906806b4f0 con 0x7f90680cd760 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f90681f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f90681f71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52240/0 (socket says 172.21.15.32:52240) 2024-03-06T07:33:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 -- 172.21.15.32:0/3327166893 learned_addr learned my addr 172.21.15.32:0/3327166893 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 -- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90680cd760 msgr2=0x7f90680ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f905d400640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90681598e0 0x7f90681541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9065e00640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90680cd760 0x7f90680ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90680cd760 0x7f90680ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 -- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90681598e0 msgr2=0x7f90681541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.645+0000 7f9066800640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90681598e0 0x7f90681541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f9066800640 1 -- 172.21.15.32:0/3327166893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680cc740 con 0x7f906815d970 2024-03-06T07:33:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f9066800640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f90681f71e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9060009f40 tx=0x7f9060004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060043020 con 0x7f906815d970 2024-03-06T07:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906003e450 con 0x7f906815d970 2024-03-06T07:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060009510 con 0x7f906815d970 2024-03-06T07:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90680ca0a0 con 0x7f906815d970 2024-03-06T07:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.646+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90680cadc0 con 0x7f906815d970 2024-03-06T07:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.647+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9068159140 con 0x7f906815d970 2024-03-06T07:33:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.648+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9060046020 con 0x7f906815d970 2024-03-06T07:33:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.649+0000 7f905ea00640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f904406b4d0 0x7f904406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.649+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90600acb80 con 0x7f906815d970 2024-03-06T07:33:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.650+0000 7f9065e00640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f904406b4d0 0x7f904406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.650+0000 7f9065e00640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f904406b4d0 0x7f904406d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9058003d90 tx=0x7f9058009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.654+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9068159140 con 0x7f906815d970 2024-03-06T07:33:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.919+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90680c9ee0 con 0x7f904406b4d0 2024-03-06T07:33:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.938+0000 7f905ea00640 1 -- 172.21.15.32:0/3327166893 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252080 (secure 0 0 0) 0x7f905800d4c0 con 0x7f904406b4d0 2024-03-06T07:33:40.943 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:33:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f904406b4d0 msgr2=0x7f904406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f904406b4d0 0x7f904406d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9058003d90 tx=0x7f9058009040 comp rx=0 tx=0).stop 2024-03-06T07:33:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 msgr2=0x7f90681f71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f90681f71e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9060009f40 tx=0x7f9060004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 shutdown_connections 2024-03-06T07:33:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f904406b4d0 0x7f904406d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f906815d970 0x7f90681f71e0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90681598e0 0x7f90681541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.944+0000 7f9067200640 1 --2- 172.21.15.32:0/3327166893 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90680cd760 0x7f90680ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.945+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 >> 172.21.15.32:0/3327166893 conn(0x7f906806c0c0 msgr2=0x7f906806e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.945+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 shutdown_connections 2024-03-06T07:33:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:40.945+0000 7f9067200640 1 -- 172.21.15.32:0/3327166893 wait complete. 2024-03-06T07:33:40.965 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-03-06T07:33:40.272195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275200,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754699520,"statfs":{"total":773094113280,"available":772812308480,"internally_reserved":0,"allocated":9314304,"data_stored":4600300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001864"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744321+0000","last_change":"2024-03-06T07:33:35.433936+0000","last_active":"2024-03-06T07:33:36.744321+0000","last_peered":"2024-03-06T07:33:36.744321+0000","last_clean":"2024-03-06T07:33:36.744321+0000","last_became_active":"2024-03-06T07:33:35.433301+0000","last_became_peered":"2024-03-06T07:33:35.433301+0000","last_unstale":"2024-03-06T07:33:36.744321+0000","last_undegraded":"2024-03-06T07:33:36.744321+0000","last_fullsized":"2024-03-06T07:33:36.744321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.647203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415712+0000","last_change":"2024-03-06T07:33:35.434426+0000","last_active":"2024-03-06T07:33:36.415712+0000","last_peered":"2024-03-06T07:33:36.415712+0000","last_clean":"2024-03-06T07:33:36.415712+0000","last_became_active":"2024-03-06T07:33:35.434145+0000","last_became_peered":"2024-03-06T07:33:35.434145+0000","last_unstale":"2024-03-06T07:33:36.415712+0000","last_undegraded":"2024-03-06T07:33:36.415712+0000","last_fullsized":"2024-03-06T07:33:36.415712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:46.389191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416851+0000","last_change":"2024-03-06T07:33:35.423435+0000","last_active":"2024-03-06T07:33:36.416851+0000","last_peered":"2024-03-06T07:33:36.416851+0000","last_clean":"2024-03-06T07:33:36.416851+0000","last_became_active":"2024-03-06T07:33:35.423141+0000","last_became_peered":"2024-03-06T07:33:35.423141+0000","last_unstale":"2024-03-06T07:33:36.416851+0000","last_undegraded":"2024-03-06T07:33:36.416851+0000","last_fullsized":"2024-03-06T07:33:36.416851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:21.262143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740802+0000","last_change":"2024-03-06T07:33:35.411938+0000","last_active":"2024-03-06T07:33:36.740802+0000","last_peered":"2024-03-06T07:33:36.740802+0000","last_clean":"2024-03-06T07:33:36.740802+0000","last_became_active":"2024-03-06T07:33:35.411118+0000","last_became_peered":"2024-03-06T07:33:35.411118+0000","last_unstale":"2024-03-06T07:33:36.740802+0000","last_undegraded":"2024-03-06T07:33:36.740802+0000","last_fullsized":"2024-03-06T07:33:36.740802+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:48.685551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596739+0000","last_change":"2024-03-06T07:33:35.431875+0000","last_active":"2024-03-06T07:33:36.596739+0000","last_peered":"2024-03-06T07:33:36.596739+0000","last_clean":"2024-03-06T07:33:36.596739+0000","last_became_active":"2024-03-06T07:33:35.428073+0000","last_became_peered":"2024-03-06T07:33:35.428073+0000","last_unstale":"2024-03-06T07:33:36.596739+0000","last_undegraded":"2024-03-06T07:33:36.596739+0000","last_fullsized":"2024-03-06T07:33:36.596739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418765+0000","last_change":"2024-03-06T07:33:35.418942+0000","last_active":"2024-03-06T07:33:36.418765+0000","last_peered":"2024-03-06T07:33:36.418765+0000","last_clean":"2024-03-06T07:33:36.418765+0000","last_became_active":"2024-03-06T07:33:35.418647+0000","last_became_peered":"2024-03-06T07:33:35.418647+0000","last_unstale":"2024-03-06T07:33:36.418765+0000","last_undegraded":"2024-03-06T07:33:36.418765+0000","last_fullsized":"2024-03-06T07:33:36.418765+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:14.739091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740997+0000","last_change":"2024-03-06T07:33:35.419984+0000","last_active":"2024-03-06T07:33:36.740997+0000","last_peered":"2024-03-06T07:33:36.740997+0000","last_clean":"2024-03-06T07:33:36.740997+0000","last_became_active":"2024-03-06T07:33:35.419686+0000","last_became_peered":"2024-03-06T07:33:35.419686+0000","last_unstale":"2024-03-06T07:33:36.740997+0000","last_undegraded":"2024-03-06T07:33:36.740997+0000","last_fullsized":"2024-03-06T07:33:36.740997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:52.371043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418948+0000","last_change":"2024-03-06T07:33:35.425311+0000","last_active":"2024-03-06T07:33:36.418948+0000","last_peered":"2024-03-06T07:33:36.418948+0000","last_clean":"2024-03-06T07:33:36.418948+0000","last_became_active":"2024-03-06T07:33:35.425031+0000","last_became_peered":"2024-03-06T07:33:35.425031+0000","last_unstale":"2024-03-06T07:33:36.418948+0000","last_undegraded":"2024-03-06T07:33:36.418948+0000","last_fullsized":"2024-03-06T07:33:36.418948+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:10.216556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419802+0000","last_change":"2024-03-06T07:33:35.419640+0000","last_active":"2024-03-06T07:33:36.419802+0000","last_peered":"2024-03-06T07:33:36.419802+0000","last_clean":"2024-03-06T07:33:36.419802+0000","last_became_active":"2024-03-06T07:33:35.419308+0000","last_became_peered":"2024-03-06T07:33:35.419308+0000","last_unstale":"2024-03-06T07:33:36.419802+0000","last_undegraded":"2024-03-06T07:33:36.419802+0000","last_fullsized":"2024-03-06T07:33:36.419802+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:30.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744076+0000","last_change":"2024-03-06T07:33:35.424350+0000","last_active":"2024-03-06T07:33:36.744076+0000","last_peered":"2024-03-06T07:33:36.744076+0000","last_clean":"2024-03-06T07:33:36.744076+0000","last_became_active":"2024-03-06T07:33:35.422597+0000","last_became_peered":"2024-03-06T07:33:35.422597+0000","last_unstale":"2024-03-06T07:33:36.744076+0000","last_undegraded":"2024-03-06T07:33:36.744076+0000","last_fullsized":"2024-03-06T07:33:36.744076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415784+0000","last_change":"2024-03-06T07:33:35.418800+0000","last_active":"2024-03-06T07:33:36.415784+0000","last_peered":"2024-03-06T07:33:36.415784+0000","last_clean":"2024-03-06T07:33:36.415784+0000","last_became_active":"2024-03-06T07:33:35.418462+0000","last_became_peered":"2024-03-06T07:33:35.418462+0000","last_unstale":"2024-03-06T07:33:36.415784+0000","last_undegraded":"2024-03-06T07:33:36.415784+0000","last_fullsized":"2024-03-06T07:33:36.415784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:39.470631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743586+0000","last_change":"2024-03-06T07:33:35.431603+0000","last_active":"2024-03-06T07:33:36.743586+0000","last_peered":"2024-03-06T07:33:36.743586+0000","last_clean":"2024-03-06T07:33:36.743586+0000","last_became_active":"2024-03-06T07:33:35.431182+0000","last_became_peered":"2024-03-06T07:33:35.431182+0000","last_unstale":"2024-03-06T07:33:36.743586+0000","last_undegraded":"2024-03-06T07:33:36.743586+0000","last_fullsized":"2024-03-06T07:33:36.743586+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:06.823351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416560+0000","last_change":"2024-03-06T07:33:35.430473+0000","last_active":"2024-03-06T07:33:36.416560+0000","last_peered":"2024-03-06T07:33:36.416560+0000","last_clean":"2024-03-06T07:33:36.416560+0000","last_became_active":"2024-03-06T07:33:35.430134+0000","last_became_peered":"2024-03-06T07:33:35.430134+0000","last_unstale":"2024-03-06T07:33:36.416560+0000","last_undegraded":"2024-03-06T07:33:36.416560+0000","last_fullsized":"2024-03-06T07:33:36.416560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:30.593508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740863+0000","last_change":"2024-03-06T07:33:35.410474+0000","last_active":"2024-03-06T07:33:36.740863+0000","last_peered":"2024-03-06T07:33:36.740863+0000","last_clean":"2024-03-06T07:33:36.740863+0000","last_became_active":"2024-03-06T07:33:35.410096+0000","last_became_peered":"2024-03-06T07:33:35.410096+0000","last_unstale":"2024-03-06T07:33:36.740863+0000","last_undegraded":"2024-03-06T07:33:36.740863+0000","last_fullsized":"2024-03-06T07:33:36.740863+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.907427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745495+0000","last_change":"2024-03-06T07:33:35.422211+0000","last_active":"2024-03-06T07:33:36.745495+0000","last_peered":"2024-03-06T07:33:36.745495+0000","last_clean":"2024-03-06T07:33:36.745495+0000","last_became_active":"2024-03-06T07:33:35.421925+0000","last_became_peered":"2024-03-06T07:33:35.421925+0000","last_unstale":"2024-03-06T07:33:36.745495+0000","last_undegraded":"2024-03-06T07:33:36.745495+0000","last_fullsized":"2024-03-06T07:33:36.745495+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:57.891330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.595620+0000","last_change":"2024-03-06T07:33:35.432360+0000","last_active":"2024-03-06T07:33:36.595620+0000","last_peered":"2024-03-06T07:33:36.595620+0000","last_clean":"2024-03-06T07:33:36.595620+0000","last_became_active":"2024-03-06T07:33:35.432045+0000","last_became_peered":"2024-03-06T07:33:35.432045+0000","last_unstale":"2024-03-06T07:33:36.595620+0000","last_undegraded":"2024-03-06T07:33:36.595620+0000","last_fullsized":"2024-03-06T07:33:36.595620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:36.361275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596583+0000","last_change":"2024-03-06T07:33:35.423519+0000","last_active":"2024-03-06T07:33:36.596583+0000","last_peered":"2024-03-06T07:33:36.596583+0000","last_clean":"2024-03-06T07:33:36.596583+0000","last_became_active":"2024-03-06T07:33:35.422710+0000","last_became_peered":"2024-03-06T07:33:35.422710+0000","last_unstale":"2024-03-06T07:33:36.596583+0000","last_undegraded":"2024-03-06T07:33:36.596583+0000","last_fullsized":"2024-03-06T07:33:36.596583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:30.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741208+0000","last_change":"2024-03-06T07:33:35.434502+0000","last_active":"2024-03-06T07:33:36.741208+0000","last_peered":"2024-03-06T07:33:36.741208+0000","last_clean":"2024-03-06T07:33:36.741208+0000","last_became_active":"2024-03-06T07:33:35.434207+0000","last_became_peered":"2024-03-06T07:33:35.434207+0000","last_unstale":"2024-03-06T07:33:36.741208+0000","last_undegraded":"2024-03-06T07:33:36.741208+0000","last_fullsized":"2024-03-06T07:33:36.741208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:18.253855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418868+0000","last_change":"2024-03-06T07:33:35.416254+0000","last_active":"2024-03-06T07:33:36.418868+0000","last_peered":"2024-03-06T07:33:36.418868+0000","last_clean":"2024-03-06T07:33:36.418868+0000","last_became_active":"2024-03-06T07:33:35.415908+0000","last_became_peered":"2024-03-06T07:33:35.415908+0000","last_unstale":"2024-03-06T07:33:36.418868+0000","last_undegraded":"2024-03-06T07:33:36.418868+0000","last_fullsized":"2024-03-06T07:33:36.418868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:57.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740978+0000","last_change":"2024-03-06T07:33:35.415989+0000","last_active":"2024-03-06T07:33:36.740978+0000","last_peered":"2024-03-06T07:33:36.740978+0000","last_clean":"2024-03-06T07:33:36.740978+0000","last_became_active":"2024-03-06T07:33:35.415666+0000","last_became_peered":"2024-03-06T07:33:35.415666+0000","last_unstale":"2024-03-06T07:33:36.740978+0000","last_undegraded":"2024-03-06T07:33:36.740978+0000","last_fullsized":"2024-03-06T07:33:36.740978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:42.486342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.740855+0000","last_change":"2024-03-06T07:33:35.408105+0000","last_active":"2024-03-06T07:33:36.740855+0000","last_peered":"2024-03-06T07:33:36.740855+0000","last_clean":"2024-03-06T07:33:36.740855+0000","last_became_active":"2024-03-06T07:33:35.407738+0000","last_became_peered":"2024-03-06T07:33:35.407738+0000","last_unstale":"2024-03-06T07:33:36.740855+0000","last_undegraded":"2024-03-06T07:33:36.740855+0000","last_fullsized":"2024-03-06T07:33:36.740855+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:38.466513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415370+0000","last_change":"2024-03-06T07:33:35.398830+0000","last_active":"2024-03-06T07:33:36.415370+0000","last_peered":"2024-03-06T07:33:36.415370+0000","last_clean":"2024-03-06T07:33:36.415370+0000","last_became_active":"2024-03-06T07:33:35.397897+0000","last_became_peered":"2024-03-06T07:33:35.397897+0000","last_unstale":"2024-03-06T07:33:36.415370+0000","last_undegraded":"2024-03-06T07:33:36.415370+0000","last_fullsized":"2024-03-06T07:33:36.415370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:20.709199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744116+0000","last_change":"2024-03-06T07:33:35.418391+0000","last_active":"2024-03-06T07:33:36.744116+0000","last_peered":"2024-03-06T07:33:36.744116+0000","last_clean":"2024-03-06T07:33:36.744116+0000","last_became_active":"2024-03-06T07:33:35.418084+0000","last_became_peered":"2024-03-06T07:33:35.418084+0000","last_unstale":"2024-03-06T07:33:36.744116+0000","last_undegraded":"2024-03-06T07:33:36.744116+0000","last_fullsized":"2024-03-06T07:33:36.744116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.623824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741350+0000","last_change":"2024-03-06T07:33:35.418948+0000","last_active":"2024-03-06T07:33:36.741350+0000","last_peered":"2024-03-06T07:33:36.741350+0000","last_clean":"2024-03-06T07:33:36.741350+0000","last_became_active":"2024-03-06T07:33:35.418641+0000","last_became_peered":"2024-03-06T07:33:35.418641+0000","last_unstale":"2024-03-06T07:33:36.741350+0000","last_undegraded":"2024-03-06T07:33:36.741350+0000","last_fullsized":"2024-03-06T07:33:36.741350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:07.673740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415759+0000","last_change":"2024-03-06T07:33:35.433657+0000","last_active":"2024-03-06T07:33:36.415759+0000","last_peered":"2024-03-06T07:33:36.415759+0000","last_clean":"2024-03-06T07:33:36.415759+0000","last_became_active":"2024-03-06T07:33:35.433321+0000","last_became_peered":"2024-03-06T07:33:35.433321+0000","last_unstale":"2024-03-06T07:33:36.415759+0000","last_undegraded":"2024-03-06T07:33:36.415759+0000","last_fullsized":"2024-03-06T07:33:36.415759+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:08.317174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417503+0000","last_change":"2024-03-06T07:33:35.426045+0000","last_active":"2024-03-06T07:33:36.417503+0000","last_peered":"2024-03-06T07:33:36.417503+0000","last_clean":"2024-03-06T07:33:36.417503+0000","last_became_active":"2024-03-06T07:33:35.425735+0000","last_became_peered":"2024-03-06T07:33:35.425735+0000","last_unstale":"2024-03-06T07:33:36.417503+0000","last_undegraded":"2024-03-06T07:33:36.417503+0000","last_fullsized":"2024-03-06T07:33:36.417503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:03.003568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596354+0000","last_change":"2024-03-06T07:33:35.425708+0000","last_active":"2024-03-06T07:33:36.596354+0000","last_peered":"2024-03-06T07:33:36.596354+0000","last_clean":"2024-03-06T07:33:36.596354+0000","last_became_active":"2024-03-06T07:33:35.425338+0000","last_became_peered":"2024-03-06T07:33:35.425338+0000","last_unstale":"2024-03-06T07:33:36.596354+0000","last_undegraded":"2024-03-06T07:33:36.596354+0000","last_fullsized":"2024-03-06T07:33:36.596354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:06.823351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743897+0000","last_change":"2024-03-06T07:33:35.431516+0000","last_active":"2024-03-06T07:33:36.743897+0000","last_peered":"2024-03-06T07:33:36.743897+0000","last_clean":"2024-03-06T07:33:36.743897+0000","last_became_active":"2024-03-06T07:33:35.430850+0000","last_became_peered":"2024-03-06T07:33:35.430850+0000","last_unstale":"2024-03-06T07:33:36.743897+0000","last_undegraded":"2024-03-06T07:33:36.743897+0000","last_fullsized":"2024-03-06T07:33:36.743897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:30.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415431+0000","last_change":"2024-03-06T07:33:35.420045+0000","last_active":"2024-03-06T07:33:36.415431+0000","last_peered":"2024-03-06T07:33:36.415431+0000","last_clean":"2024-03-06T07:33:36.415431+0000","last_became_active":"2024-03-06T07:33:35.419744+0000","last_became_peered":"2024-03-06T07:33:35.419744+0000","last_unstale":"2024-03-06T07:33:36.415431+0000","last_undegraded":"2024-03-06T07:33:36.415431+0000","last_fullsized":"2024-03-06T07:33:36.415431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:39.536433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415473+0000","last_change":"2024-03-06T07:33:35.414588+0000","last_active":"2024-03-06T07:33:36.415473+0000","last_peered":"2024-03-06T07:33:36.415473+0000","last_clean":"2024-03-06T07:33:36.415473+0000","last_became_active":"2024-03-06T07:33:35.413813+0000","last_became_peered":"2024-03-06T07:33:35.413813+0000","last_unstale":"2024-03-06T07:33:36.415473+0000","last_undegraded":"2024-03-06T07:33:36.415473+0000","last_fullsized":"2024-03-06T07:33:36.415473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:17.232099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596964+0000","last_change":"2024-03-06T07:33:35.421717+0000","last_active":"2024-03-06T07:33:36.596964+0000","last_peered":"2024-03-06T07:33:36.596964+0000","last_clean":"2024-03-06T07:33:36.596964+0000","last_became_active":"2024-03-06T07:33:35.421413+0000","last_became_peered":"2024-03-06T07:33:35.421413+0000","last_unstale":"2024-03-06T07:33:36.596964+0000","last_undegraded":"2024-03-06T07:33:36.596964+0000","last_fullsized":"2024-03-06T07:33:36.596964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:24.380160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743383+0000","last_change":"2024-03-06T07:33:35.431314+0000","last_active":"2024-03-06T07:33:36.743383+0000","last_peered":"2024-03-06T07:33:36.743383+0000","last_clean":"2024-03-06T07:33:36.743383+0000","last_became_active":"2024-03-06T07:33:35.430929+0000","last_became_peered":"2024-03-06T07:33:35.430929+0000","last_unstale":"2024-03-06T07:33:36.743383+0000","last_undegraded":"2024-03-06T07:33:36.743383+0000","last_fullsized":"2024-03-06T07:33:36.743383+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:10.216556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596799+0000","last_change":"2024-03-06T07:33:35.430067+0000","last_active":"2024-03-06T07:33:36.596799+0000","last_peered":"2024-03-06T07:33:36.596799+0000","last_clean":"2024-03-06T07:33:36.596799+0000","last_became_active":"2024-03-06T07:33:35.429736+0000","last_became_peered":"2024-03-06T07:33:35.429736+0000","last_unstale":"2024-03-06T07:33:36.596799+0000","last_undegraded":"2024-03-06T07:33:36.596799+0000","last_fullsized":"2024-03-06T07:33:36.596799+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:36.146350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744810+0000","last_change":"2024-03-06T07:33:34.418836+0000","last_active":"2024-03-06T07:33:36.744810+0000","last_peered":"2024-03-06T07:33:36.744810+0000","last_clean":"2024-03-06T07:33:36.744810+0000","last_became_active":"2024-03-06T07:33:34.418553+0000","last_became_peered":"2024-03-06T07:33:34.418553+0000","last_unstale":"2024-03-06T07:33:36.744810+0000","last_undegraded":"2024-03-06T07:33:36.744810+0000","last_fullsized":"2024-03-06T07:33:36.744810+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:58.543192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744583+0000","last_change":"2024-03-06T07:33:35.429003+0000","last_active":"2024-03-06T07:33:36.744583+0000","last_peered":"2024-03-06T07:33:36.744583+0000","last_clean":"2024-03-06T07:33:36.744583+0000","last_became_active":"2024-03-06T07:33:35.428592+0000","last_became_peered":"2024-03-06T07:33:35.428592+0000","last_unstale":"2024-03-06T07:33:36.744583+0000","last_undegraded":"2024-03-06T07:33:36.744583+0000","last_fullsized":"2024-03-06T07:33:36.744583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:15.522341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597197+0000","last_change":"2024-03-06T07:33:34.415694+0000","last_active":"2024-03-06T07:33:36.597197+0000","last_peered":"2024-03-06T07:33:36.597197+0000","last_clean":"2024-03-06T07:33:36.597197+0000","last_became_active":"2024-03-06T07:33:34.415343+0000","last_became_peered":"2024-03-06T07:33:34.415343+0000","last_unstale":"2024-03-06T07:33:36.597197+0000","last_undegraded":"2024-03-06T07:33:36.597197+0000","last_fullsized":"2024-03-06T07:33:36.597197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.269016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744118+0000","last_change":"2024-03-06T07:33:35.431346+0000","last_active":"2024-03-06T07:33:36.744118+0000","last_peered":"2024-03-06T07:33:36.744118+0000","last_clean":"2024-03-06T07:33:36.744118+0000","last_became_active":"2024-03-06T07:33:35.430823+0000","last_became_peered":"2024-03-06T07:33:35.430823+0000","last_unstale":"2024-03-06T07:33:36.744118+0000","last_undegraded":"2024-03-06T07:33:36.744118+0000","last_fullsized":"2024-03-06T07:33:36.744118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:36.146350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418889+0000","last_change":"2024-03-06T07:33:34.423715+0000","last_active":"2024-03-06T07:33:36.418889+0000","last_peered":"2024-03-06T07:33:36.418889+0000","last_clean":"2024-03-06T07:33:36.418889+0000","last_became_active":"2024-03-06T07:33:34.423391+0000","last_became_peered":"2024-03-06T07:33:34.423391+0000","last_unstale":"2024-03-06T07:33:36.418889+0000","last_undegraded":"2024-03-06T07:33:36.418889+0000","last_fullsized":"2024-03-06T07:33:36.418889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:35.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417128+0000","last_change":"2024-03-06T07:33:35.426012+0000","last_active":"2024-03-06T07:33:36.417128+0000","last_peered":"2024-03-06T07:33:36.417128+0000","last_clean":"2024-03-06T07:33:36.417128+0000","last_became_active":"2024-03-06T07:33:35.425256+0000","last_became_peered":"2024-03-06T07:33:35.425256+0000","last_unstale":"2024-03-06T07:33:36.417128+0000","last_undegraded":"2024-03-06T07:33:36.417128+0000","last_fullsized":"2024-03-06T07:33:36.417128+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:34.075715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419020+0000","last_change":"2024-03-06T07:33:34.425003+0000","last_active":"2024-03-06T07:33:36.419020+0000","last_peered":"2024-03-06T07:33:36.419020+0000","last_clean":"2024-03-06T07:33:36.419020+0000","last_became_active":"2024-03-06T07:33:34.422362+0000","last_became_peered":"2024-03-06T07:33:34.422362+0000","last_unstale":"2024-03-06T07:33:36.419020+0000","last_undegraded":"2024-03-06T07:33:36.419020+0000","last_fullsized":"2024-03-06T07:33:36.419020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:22.769198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415561+0000","last_change":"2024-03-06T07:33:35.414323+0000","last_active":"2024-03-06T07:33:36.415561+0000","last_peered":"2024-03-06T07:33:36.415561+0000","last_clean":"2024-03-06T07:33:36.415561+0000","last_became_active":"2024-03-06T07:33:35.413973+0000","last_became_peered":"2024-03-06T07:33:35.413973+0000","last_unstale":"2024-03-06T07:33:36.415561+0000","last_undegraded":"2024-03-06T07:33:36.415561+0000","last_fullsized":"2024-03-06T07:33:36.415561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:59.129104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597034+0000","last_change":"2024-03-06T07:33:34.414855+0000","last_active":"2024-03-06T07:33:36.597034+0000","last_peered":"2024-03-06T07:33:36.597034+0000","last_clean":"2024-03-06T07:33:36.597034+0000","last_became_active":"2024-03-06T07:33:34.414544+0000","last_became_peered":"2024-03-06T07:33:34.414544+0000","last_unstale":"2024-03-06T07:33:36.597034+0000","last_undegraded":"2024-03-06T07:33:36.597034+0000","last_fullsized":"2024-03-06T07:33:36.597034+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:04.291261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596297+0000","last_change":"2024-03-06T07:33:35.428151+0000","last_active":"2024-03-06T07:33:36.596297+0000","last_peered":"2024-03-06T07:33:36.596297+0000","last_clean":"2024-03-06T07:33:36.596297+0000","last_became_active":"2024-03-06T07:33:35.426950+0000","last_became_peered":"2024-03-06T07:33:35.426950+0000","last_unstale":"2024-03-06T07:33:36.596297+0000","last_undegraded":"2024-03-06T07:33:36.596297+0000","last_fullsized":"2024-03-06T07:33:36.596297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:10.216556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.595911+0000","last_change":"2024-03-06T07:33:34.416030+0000","last_active":"2024-03-06T07:33:36.595911+0000","last_peered":"2024-03-06T07:33:36.595911+0000","last_clean":"2024-03-06T07:33:36.595911+0000","last_became_active":"2024-03-06T07:33:34.415342+0000","last_became_peered":"2024-03-06T07:33:34.415342+0000","last_unstale":"2024-03-06T07:33:36.595911+0000","last_undegraded":"2024-03-06T07:33:36.595911+0000","last_fullsized":"2024-03-06T07:33:36.595911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:44.276115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415735+0000","last_change":"2024-03-06T07:33:35.413918+0000","last_active":"2024-03-06T07:33:36.415735+0000","last_peered":"2024-03-06T07:33:36.415735+0000","last_clean":"2024-03-06T07:33:36.415735+0000","last_became_active":"2024-03-06T07:33:35.413068+0000","last_became_peered":"2024-03-06T07:33:35.413068+0000","last_unstale":"2024-03-06T07:33:36.415735+0000","last_undegraded":"2024-03-06T07:33:36.415735+0000","last_fullsized":"2024-03-06T07:33:36.415735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:49.600775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418205+0000","last_change":"2024-03-06T07:33:34.423592+0000","last_active":"2024-03-06T07:33:36.418205+0000","last_peered":"2024-03-06T07:33:36.418205+0000","last_clean":"2024-03-06T07:33:36.418205+0000","last_became_active":"2024-03-06T07:33:34.423269+0000","last_became_peered":"2024-03-06T07:33:34.423269+0000","last_unstale":"2024-03-06T07:33:36.418205+0000","last_undegraded":"2024-03-06T07:33:36.418205+0000","last_fullsized":"2024-03-06T07:33:36.418205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.197881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745595+0000","last_change":"2024-03-06T07:33:35.421611+0000","last_active":"2024-03-06T07:33:36.745595+0000","last_peered":"2024-03-06T07:33:36.745595+0000","last_clean":"2024-03-06T07:33:36.745595+0000","last_became_active":"2024-03-06T07:33:35.421339+0000","last_became_peered":"2024-03-06T07:33:35.421339+0000","last_unstale":"2024-03-06T07:33:36.745595+0000","last_undegraded":"2024-03-06T07:33:36.745595+0000","last_fullsized":"2024-03-06T07:33:36.745595+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:07.286732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415126+0000","last_change":"2024-03-06T07:33:34.426732+0000","last_active":"2024-03-06T07:33:36.415126+0000","last_peered":"2024-03-06T07:33:36.415126+0000","last_clean":"2024-03-06T07:33:36.415126+0000","last_became_active":"2024-03-06T07:33:34.420142+0000","last_became_peered":"2024-03-06T07:33:34.420142+0000","last_unstale":"2024-03-06T07:33:36.415126+0000","last_undegraded":"2024-03-06T07:33:36.415126+0000","last_fullsized":"2024-03-06T07:33:36.415126+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:48.739973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745881+0000","last_change":"2024-03-06T07:33:35.424626+0000","last_active":"2024-03-06T07:33:36.745881+0000","last_peered":"2024-03-06T07:33:36.745881+0000","last_clean":"2024-03-06T07:33:36.745881+0000","last_became_active":"2024-03-06T07:33:35.424322+0000","last_became_peered":"2024-03-06T07:33:35.424322+0000","last_unstale":"2024-03-06T07:33:36.745881+0000","last_undegraded":"2024-03-06T07:33:36.745881+0000","last_fullsized":"2024-03-06T07:33:36.745881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:08.630175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744899+0000","last_change":"2024-03-06T07:33:34.422749+0000","last_active":"2024-03-06T07:33:36.744899+0000","last_peered":"2024-03-06T07:33:36.744899+0000","last_clean":"2024-03-06T07:33:36.744899+0000","last_became_active":"2024-03-06T07:33:34.420917+0000","last_became_peered":"2024-03-06T07:33:34.420917+0000","last_unstale":"2024-03-06T07:33:36.744899+0000","last_undegraded":"2024-03-06T07:33:36.744899+0000","last_fullsized":"2024-03-06T07:33:36.744899+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:24.022728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419851+0000","last_change":"2024-03-06T07:33:35.420287+0000","last_active":"2024-03-06T07:33:36.419851+0000","last_peered":"2024-03-06T07:33:36.419851+0000","last_clean":"2024-03-06T07:33:36.419851+0000","last_became_active":"2024-03-06T07:33:35.419979+0000","last_became_peered":"2024-03-06T07:33:35.419979+0000","last_unstale":"2024-03-06T07:33:36.419851+0000","last_undegraded":"2024-03-06T07:33:36.419851+0000","last_fullsized":"2024-03-06T07:33:36.419851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744455+0000","last_change":"2024-03-06T07:33:34.426256+0000","last_active":"2024-03-06T07:33:36.744455+0000","last_peered":"2024-03-06T07:33:36.744455+0000","last_clean":"2024-03-06T07:33:36.744455+0000","last_became_active":"2024-03-06T07:33:34.425970+0000","last_became_peered":"2024-03-06T07:33:34.425970+0000","last_unstale":"2024-03-06T07:33:36.744455+0000","last_undegraded":"2024-03-06T07:33:36.744455+0000","last_fullsized":"2024-03-06T07:33:36.744455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:24.319514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418894+0000","last_change":"2024-03-06T07:33:35.433575+0000","last_active":"2024-03-06T07:33:36.418894+0000","last_peered":"2024-03-06T07:33:36.418894+0000","last_clean":"2024-03-06T07:33:36.418894+0000","last_became_active":"2024-03-06T07:33:35.433276+0000","last_became_peered":"2024-03-06T07:33:35.433276+0000","last_unstale":"2024-03-06T07:33:36.418894+0000","last_undegraded":"2024-03-06T07:33:36.418894+0000","last_fullsized":"2024-03-06T07:33:36.418894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:30.593508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743670+0000","last_change":"2024-03-06T07:33:34.427477+0000","last_active":"2024-03-06T07:33:36.743670+0000","last_peered":"2024-03-06T07:33:36.743670+0000","last_clean":"2024-03-06T07:33:36.743670+0000","last_became_active":"2024-03-06T07:33:34.426590+0000","last_became_peered":"2024-03-06T07:33:34.426590+0000","last_unstale":"2024-03-06T07:33:36.743670+0000","last_undegraded":"2024-03-06T07:33:36.743670+0000","last_fullsized":"2024-03-06T07:33:36.743670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.269016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596418+0000","last_change":"2024-03-06T07:33:35.422229+0000","last_active":"2024-03-06T07:33:36.596418+0000","last_peered":"2024-03-06T07:33:36.596418+0000","last_clean":"2024-03-06T07:33:36.596418+0000","last_became_active":"2024-03-06T07:33:35.421940+0000","last_became_peered":"2024-03-06T07:33:35.421940+0000","last_unstale":"2024-03-06T07:33:36.596418+0000","last_undegraded":"2024-03-06T07:33:36.596418+0000","last_fullsized":"2024-03-06T07:33:36.596418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:13.287392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419620+0000","last_change":"2024-03-06T07:33:34.415820+0000","last_active":"2024-03-06T07:33:36.419620+0000","last_peered":"2024-03-06T07:33:36.419620+0000","last_clean":"2024-03-06T07:33:36.419620+0000","last_became_active":"2024-03-06T07:33:34.415298+0000","last_became_peered":"2024-03-06T07:33:34.415298+0000","last_unstale":"2024-03-06T07:33:36.419620+0000","last_undegraded":"2024-03-06T07:33:36.419620+0000","last_fullsized":"2024-03-06T07:33:36.419620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:13.929092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418447+0000","last_change":"2024-03-06T07:33:35.428922+0000","last_active":"2024-03-06T07:33:36.418447+0000","last_peered":"2024-03-06T07:33:36.418447+0000","last_clean":"2024-03-06T07:33:36.418447+0000","last_became_active":"2024-03-06T07:33:35.428523+0000","last_became_peered":"2024-03-06T07:33:35.428523+0000","last_unstale":"2024-03-06T07:33:36.418447+0000","last_undegraded":"2024-03-06T07:33:36.418447+0000","last_fullsized":"2024-03-06T07:33:36.418447+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:20.620205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415553+0000","last_change":"2024-03-06T07:33:34.417859+0000","last_active":"2024-03-06T07:33:36.415553+0000","last_peered":"2024-03-06T07:33:36.415553+0000","last_clean":"2024-03-06T07:33:36.415553+0000","last_became_active":"2024-03-06T07:33:34.417563+0000","last_became_peered":"2024-03-06T07:33:34.417563+0000","last_unstale":"2024-03-06T07:33:36.415553+0000","last_undegraded":"2024-03-06T07:33:36.415553+0000","last_fullsized":"2024-03-06T07:33:36.415553+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:33.650899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416609+0000","last_change":"2024-03-06T07:33:35.414333+0000","last_active":"2024-03-06T07:33:36.416609+0000","last_peered":"2024-03-06T07:33:36.416609+0000","last_clean":"2024-03-06T07:33:36.416609+0000","last_became_active":"2024-03-06T07:33:35.413343+0000","last_became_peered":"2024-03-06T07:33:35.413343+0000","last_unstale":"2024-03-06T07:33:36.416609+0000","last_undegraded":"2024-03-06T07:33:36.416609+0000","last_fullsized":"2024-03-06T07:33:36.416609+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:00.522817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745557+0000","last_change":"2024-03-06T07:33:34.414094+0000","last_active":"2024-03-06T07:33:36.745557+0000","last_peered":"2024-03-06T07:33:36.745557+0000","last_clean":"2024-03-06T07:33:36.745557+0000","last_became_active":"2024-03-06T07:33:34.413811+0000","last_became_peered":"2024-03-06T07:33:34.413811+0000","last_unstale":"2024-03-06T07:33:36.745557+0000","last_undegraded":"2024-03-06T07:33:36.745557+0000","last_fullsized":"2024-03-06T07:33:36.745557+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:20.243469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415111+0000","last_change":"2024-03-06T07:33:35.425618+0000","last_active":"2024-03-06T07:33:36.415111+0000","last_peered":"2024-03-06T07:33:36.415111+0000","last_clean":"2024-03-06T07:33:36.415111+0000","last_became_active":"2024-03-06T07:33:35.425290+0000","last_became_peered":"2024-03-06T07:33:35.425290+0000","last_unstale":"2024-03-06T07:33:36.415111+0000","last_undegraded":"2024-03-06T07:33:36.415111+0000","last_fullsized":"2024-03-06T07:33:36.415111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:51.337308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419035+0000","last_change":"2024-03-06T07:33:34.420849+0000","last_active":"2024-03-06T07:33:36.419035+0000","last_peered":"2024-03-06T07:33:36.419035+0000","last_clean":"2024-03-06T07:33:36.419035+0000","last_became_active":"2024-03-06T07:33:34.420527+0000","last_became_peered":"2024-03-06T07:33:34.420527+0000","last_unstale":"2024-03-06T07:33:36.419035+0000","last_undegraded":"2024-03-06T07:33:36.419035+0000","last_fullsized":"2024-03-06T07:33:36.419035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:06.907010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415094+0000","last_change":"2024-03-06T07:33:35.421912+0000","last_active":"2024-03-06T07:33:36.415094+0000","last_peered":"2024-03-06T07:33:36.415094+0000","last_clean":"2024-03-06T07:33:36.415094+0000","last_became_active":"2024-03-06T07:33:35.421596+0000","last_became_peered":"2024-03-06T07:33:35.421596+0000","last_unstale":"2024-03-06T07:33:36.415094+0000","last_undegraded":"2024-03-06T07:33:36.415094+0000","last_fullsized":"2024-03-06T07:33:36.415094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:55.138916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418122+0000","last_change":"2024-03-06T07:33:34.421400+0000","last_active":"2024-03-06T07:33:36.418122+0000","last_peered":"2024-03-06T07:33:36.418122+0000","last_clean":"2024-03-06T07:33:36.418122+0000","last_became_active":"2024-03-06T07:33:34.420914+0000","last_became_peered":"2024-03-06T07:33:34.420914+0000","last_unstale":"2024-03-06T07:33:36.418122+0000","last_undegraded":"2024-03-06T07:33:36.418122+0000","last_fullsized":"2024-03-06T07:33:36.418122+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:05.804677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417604+0000","last_change":"2024-03-06T07:33:35.420920+0000","last_active":"2024-03-06T07:33:36.417604+0000","last_peered":"2024-03-06T07:33:36.417604+0000","last_clean":"2024-03-06T07:33:36.417604+0000","last_became_active":"2024-03-06T07:33:35.418326+0000","last_became_peered":"2024-03-06T07:33:35.418326+0000","last_unstale":"2024-03-06T07:33:36.417604+0000","last_undegraded":"2024-03-06T07:33:36.417604+0000","last_fullsized":"2024-03-06T07:33:36.417604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:12.227545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745123+0000","last_change":"2024-03-06T07:33:34.422775+0000","last_active":"2024-03-06T07:33:36.745123+0000","last_peered":"2024-03-06T07:33:36.745123+0000","last_clean":"2024-03-06T07:33:36.745123+0000","last_became_active":"2024-03-06T07:33:34.422386+0000","last_became_peered":"2024-03-06T07:33:34.422386+0000","last_unstale":"2024-03-06T07:33:36.745123+0000","last_undegraded":"2024-03-06T07:33:36.745123+0000","last_fullsized":"2024-03-06T07:33:36.745123+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:06.907010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744714+0000","last_change":"2024-03-06T07:33:35.429397+0000","last_active":"2024-03-06T07:33:36.744714+0000","last_peered":"2024-03-06T07:33:36.744714+0000","last_clean":"2024-03-06T07:33:36.744714+0000","last_became_active":"2024-03-06T07:33:35.429020+0000","last_became_peered":"2024-03-06T07:33:35.429020+0000","last_unstale":"2024-03-06T07:33:36.744714+0000","last_undegraded":"2024-03-06T07:33:36.744714+0000","last_fullsized":"2024-03-06T07:33:36.744714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:18.663607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418967+0000","last_change":"2024-03-06T07:33:34.426275+0000","last_active":"2024-03-06T07:33:36.418967+0000","last_peered":"2024-03-06T07:33:36.418967+0000","last_clean":"2024-03-06T07:33:36.418967+0000","last_became_active":"2024-03-06T07:33:34.425954+0000","last_became_peered":"2024-03-06T07:33:34.425954+0000","last_unstale":"2024-03-06T07:33:36.418967+0000","last_undegraded":"2024-03-06T07:33:36.418967+0000","last_fullsized":"2024-03-06T07:33:36.418967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:29.964030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596828+0000","last_change":"2024-03-06T07:33:35.430415+0000","last_active":"2024-03-06T07:33:36.596828+0000","last_peered":"2024-03-06T07:33:36.596828+0000","last_clean":"2024-03-06T07:33:36.596828+0000","last_became_active":"2024-03-06T07:33:35.430101+0000","last_became_peered":"2024-03-06T07:33:35.430101+0000","last_unstale":"2024-03-06T07:33:36.596828+0000","last_undegraded":"2024-03-06T07:33:36.596828+0000","last_fullsized":"2024-03-06T07:33:36.596828+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.647203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419150+0000","last_change":"2024-03-06T07:33:34.426447+0000","last_active":"2024-03-06T07:33:36.419150+0000","last_peered":"2024-03-06T07:33:36.419150+0000","last_clean":"2024-03-06T07:33:36.419150+0000","last_became_active":"2024-03-06T07:33:34.426131+0000","last_became_peered":"2024-03-06T07:33:34.426131+0000","last_unstale":"2024-03-06T07:33:36.419150+0000","last_undegraded":"2024-03-06T07:33:36.419150+0000","last_fullsized":"2024-03-06T07:33:36.419150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:58.605149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741010+0000","last_change":"2024-03-06T07:33:35.411593+0000","last_active":"2024-03-06T07:33:36.741010+0000","last_peered":"2024-03-06T07:33:36.741010+0000","last_clean":"2024-03-06T07:33:36.741010+0000","last_became_active":"2024-03-06T07:33:35.410483+0000","last_became_peered":"2024-03-06T07:33:35.410483+0000","last_unstale":"2024-03-06T07:33:36.741010+0000","last_undegraded":"2024-03-06T07:33:36.741010+0000","last_fullsized":"2024-03-06T07:33:36.741010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:35.970261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745635+0000","last_change":"2024-03-06T07:33:34.421867+0000","last_active":"2024-03-06T07:33:36.745635+0000","last_peered":"2024-03-06T07:33:36.745635+0000","last_clean":"2024-03-06T07:33:36.745635+0000","last_became_active":"2024-03-06T07:33:34.421601+0000","last_became_peered":"2024-03-06T07:33:34.421601+0000","last_unstale":"2024-03-06T07:33:36.745635+0000","last_undegraded":"2024-03-06T07:33:36.745635+0000","last_fullsized":"2024-03-06T07:33:36.745635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:33.057040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419424+0000","last_change":"2024-03-06T07:33:35.417503+0000","last_active":"2024-03-06T07:33:36.419424+0000","last_peered":"2024-03-06T07:33:36.419424+0000","last_clean":"2024-03-06T07:33:36.419424+0000","last_became_active":"2024-03-06T07:33:35.417120+0000","last_became_peered":"2024-03-06T07:33:35.417120+0000","last_unstale":"2024-03-06T07:33:36.419424+0000","last_undegraded":"2024-03-06T07:33:36.419424+0000","last_fullsized":"2024-03-06T07:33:36.419424+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:36.361275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741076+0000","last_change":"2024-03-06T07:33:34.428159+0000","last_active":"2024-03-06T07:33:36.741076+0000","last_peered":"2024-03-06T07:33:36.741076+0000","last_clean":"2024-03-06T07:33:36.741076+0000","last_became_active":"2024-03-06T07:33:34.427845+0000","last_became_peered":"2024-03-06T07:33:34.427845+0000","last_unstale":"2024-03-06T07:33:36.741076+0000","last_undegraded":"2024-03-06T07:33:36.741076+0000","last_fullsized":"2024-03-06T07:33:36.741076+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:58.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741051+0000","last_change":"2024-03-06T07:33:35.419019+0000","last_active":"2024-03-06T07:33:36.741051+0000","last_peered":"2024-03-06T07:33:36.741051+0000","last_clean":"2024-03-06T07:33:36.741051+0000","last_became_active":"2024-03-06T07:33:35.418635+0000","last_became_peered":"2024-03-06T07:33:35.418635+0000","last_unstale":"2024-03-06T07:33:36.741051+0000","last_undegraded":"2024-03-06T07:33:36.741051+0000","last_fullsized":"2024-03-06T07:33:36.741051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:33.188533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.741275+0000","last_change":"2024-03-06T07:33:34.424418+0000","last_active":"2024-03-06T07:33:36.741275+0000","last_peered":"2024-03-06T07:33:36.741275+0000","last_clean":"2024-03-06T07:33:36.741275+0000","last_became_active":"2024-03-06T07:33:34.424027+0000","last_became_peered":"2024-03-06T07:33:34.424027+0000","last_unstale":"2024-03-06T07:33:36.741275+0000","last_undegraded":"2024-03-06T07:33:36.741275+0000","last_fullsized":"2024-03-06T07:33:36.741275+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:59.075986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419137+0000","last_change":"2024-03-06T07:33:35.419004+0000","last_active":"2024-03-06T07:33:36.419137+0000","last_peered":"2024-03-06T07:33:36.419137+0000","last_clean":"2024-03-06T07:33:36.419137+0000","last_became_active":"2024-03-06T07:33:35.418699+0000","last_became_peered":"2024-03-06T07:33:35.418699+0000","last_unstale":"2024-03-06T07:33:36.419137+0000","last_undegraded":"2024-03-06T07:33:36.419137+0000","last_fullsized":"2024-03-06T07:33:36.419137+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:06.823351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.745410+0000","last_change":"2024-03-06T07:33:34.417603+0000","last_active":"2024-03-06T07:33:36.745410+0000","last_peered":"2024-03-06T07:33:36.745410+0000","last_clean":"2024-03-06T07:33:36.745410+0000","last_became_active":"2024-03-06T07:33:34.417330+0000","last_became_peered":"2024-03-06T07:33:34.417330+0000","last_unstale":"2024-03-06T07:33:36.745410+0000","last_undegraded":"2024-03-06T07:33:36.745410+0000","last_fullsized":"2024-03-06T07:33:36.745410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:21.409894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597025+0000","last_change":"2024-03-06T07:33:35.427378+0000","last_active":"2024-03-06T07:33:36.597025+0000","last_peered":"2024-03-06T07:33:36.597025+0000","last_clean":"2024-03-06T07:33:36.597025+0000","last_became_active":"2024-03-06T07:33:35.426877+0000","last_became_peered":"2024-03-06T07:33:35.426877+0000","last_unstale":"2024-03-06T07:33:36.597025+0000","last_undegraded":"2024-03-06T07:33:36.597025+0000","last_fullsized":"2024-03-06T07:33:36.597025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:15.522341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415812+0000","last_change":"2024-03-06T07:33:34.424870+0000","last_active":"2024-03-06T07:33:36.415812+0000","last_peered":"2024-03-06T07:33:36.415812+0000","last_clean":"2024-03-06T07:33:36.415812+0000","last_became_active":"2024-03-06T07:33:34.421161+0000","last_became_peered":"2024-03-06T07:33:34.421161+0000","last_unstale":"2024-03-06T07:33:36.415812+0000","last_undegraded":"2024-03-06T07:33:36.415812+0000","last_fullsized":"2024-03-06T07:33:36.415812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:47.666877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.416825+0000","last_change":"2024-03-06T07:33:35.434446+0000","last_active":"2024-03-06T07:33:36.416825+0000","last_peered":"2024-03-06T07:33:36.416825+0000","last_clean":"2024-03-06T07:33:36.416825+0000","last_became_active":"2024-03-06T07:33:35.434115+0000","last_became_peered":"2024-03-06T07:33:35.434115+0000","last_unstale":"2024-03-06T07:33:36.416825+0000","last_undegraded":"2024-03-06T07:33:36.416825+0000","last_fullsized":"2024-03-06T07:33:36.416825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:22.428569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417176+0000","last_change":"2024-03-06T07:33:34.418898+0000","last_active":"2024-03-06T07:33:36.417176+0000","last_peered":"2024-03-06T07:33:36.417176+0000","last_clean":"2024-03-06T07:33:36.417176+0000","last_became_active":"2024-03-06T07:33:34.418627+0000","last_became_peered":"2024-03-06T07:33:34.418627+0000","last_unstale":"2024-03-06T07:33:36.417176+0000","last_undegraded":"2024-03-06T07:33:36.417176+0000","last_fullsized":"2024-03-06T07:33:36.417176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:49.811096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744386+0000","last_change":"2024-03-06T07:33:35.421129+0000","last_active":"2024-03-06T07:33:36.744386+0000","last_peered":"2024-03-06T07:33:36.744386+0000","last_clean":"2024-03-06T07:33:36.744386+0000","last_became_active":"2024-03-06T07:33:35.420533+0000","last_became_peered":"2024-03-06T07:33:35.420533+0000","last_unstale":"2024-03-06T07:33:36.744386+0000","last_undegraded":"2024-03-06T07:33:36.744386+0000","last_fullsized":"2024-03-06T07:33:36.744386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:24.380160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743987+0000","last_change":"2024-03-06T07:33:34.424152+0000","last_active":"2024-03-06T07:33:36.743987+0000","last_peered":"2024-03-06T07:33:36.743987+0000","last_clean":"2024-03-06T07:33:36.743987+0000","last_became_active":"2024-03-06T07:33:34.423736+0000","last_became_peered":"2024-03-06T07:33:34.423736+0000","last_unstale":"2024-03-06T07:33:36.743987+0000","last_undegraded":"2024-03-06T07:33:36.743987+0000","last_fullsized":"2024-03-06T07:33:36.743987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:44.276115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.596769+0000","last_change":"2024-03-06T07:33:35.419310+0000","last_active":"2024-03-06T07:33:36.596769+0000","last_peered":"2024-03-06T07:33:36.596769+0000","last_clean":"2024-03-06T07:33:36.596769+0000","last_became_active":"2024-03-06T07:33:35.419026+0000","last_became_peered":"2024-03-06T07:33:35.419026+0000","last_unstale":"2024-03-06T07:33:36.596769+0000","last_undegraded":"2024-03-06T07:33:36.596769+0000","last_fullsized":"2024-03-06T07:33:36.596769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:59.623824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744484+0000","last_change":"2024-03-06T07:33:34.427936+0000","last_active":"2024-03-06T07:33:36.744484+0000","last_peered":"2024-03-06T07:33:36.744484+0000","last_clean":"2024-03-06T07:33:36.744484+0000","last_became_active":"2024-03-06T07:33:34.427268+0000","last_became_peered":"2024-03-06T07:33:34.427268+0000","last_unstale":"2024-03-06T07:33:36.744484+0000","last_undegraded":"2024-03-06T07:33:36.744484+0000","last_fullsized":"2024-03-06T07:33:36.744484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:04.291261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.419173+0000","last_change":"2024-03-06T07:33:35.424420+0000","last_active":"2024-03-06T07:33:36.419173+0000","last_peered":"2024-03-06T07:33:36.419173+0000","last_clean":"2024-03-06T07:33:36.419173+0000","last_became_active":"2024-03-06T07:33:35.423970+0000","last_became_peered":"2024-03-06T07:33:35.423970+0000","last_unstale":"2024-03-06T07:33:36.419173+0000","last_undegraded":"2024-03-06T07:33:36.419173+0000","last_fullsized":"2024-03-06T07:33:36.419173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:13.287392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418644+0000","last_change":"2024-03-06T07:33:34.426244+0000","last_active":"2024-03-06T07:33:36.418644+0000","last_peered":"2024-03-06T07:33:36.418644+0000","last_clean":"2024-03-06T07:33:36.418644+0000","last_became_active":"2024-03-06T07:33:34.425587+0000","last_became_peered":"2024-03-06T07:33:34.425587+0000","last_unstale":"2024-03-06T07:33:36.418644+0000","last_undegraded":"2024-03-06T07:33:36.418644+0000","last_fullsized":"2024-03-06T07:33:36.418644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:12.268717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.418620+0000","last_change":"2024-03-06T07:33:35.417892+0000","last_active":"2024-03-06T07:33:36.418620+0000","last_peered":"2024-03-06T07:33:36.418620+0000","last_clean":"2024-03-06T07:33:36.418620+0000","last_became_active":"2024-03-06T07:33:35.417259+0000","last_became_peered":"2024-03-06T07:33:35.417259+0000","last_unstale":"2024-03-06T07:33:36.418620+0000","last_undegraded":"2024-03-06T07:33:36.418620+0000","last_fullsized":"2024-03-06T07:33:36.418620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:49.600775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.597222+0000","last_change":"2024-03-06T07:33:34.415291+0000","last_active":"2024-03-06T07:33:36.597222+0000","last_peered":"2024-03-06T07:33:36.597222+0000","last_clean":"2024-03-06T07:33:36.597222+0000","last_became_active":"2024-03-06T07:33:34.412160+0000","last_became_peered":"2024-03-06T07:33:34.412160+0000","last_unstale":"2024-03-06T07:33:36.597222+0000","last_undegraded":"2024-03-06T07:33:36.597222+0000","last_fullsized":"2024-03-06T07:33:36.597222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:58.543192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415755+0000","last_change":"2024-03-06T07:33:35.426401+0000","last_active":"2024-03-06T07:33:36.415755+0000","last_peered":"2024-03-06T07:33:36.415755+0000","last_clean":"2024-03-06T07:33:36.415755+0000","last_became_active":"2024-03-06T07:33:35.425716+0000","last_became_peered":"2024-03-06T07:33:35.425716+0000","last_unstale":"2024-03-06T07:33:36.415755+0000","last_undegraded":"2024-03-06T07:33:36.415755+0000","last_fullsized":"2024-03-06T07:33:36.415755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:57.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.744556+0000","last_change":"2024-03-06T07:33:34.422229+0000","last_active":"2024-03-06T07:33:36.744556+0000","last_peered":"2024-03-06T07:33:36.744556+0000","last_clean":"2024-03-06T07:33:36.744556+0000","last_became_active":"2024-03-06T07:33:34.420817+0000","last_became_peered":"2024-03-06T07:33:34.420817+0000","last_unstale":"2024-03-06T07:33:36.744556+0000","last_undegraded":"2024-03-06T07:33:36.744556+0000","last_fullsized":"2024-03-06T07:33:36.744556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:38.451957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.743593+0000","last_change":"2024-03-06T07:33:35.427030+0000","last_active":"2024-03-06T07:33:36.743593+0000","last_peered":"2024-03-06T07:33:36.743593+0000","last_clean":"2024-03-06T07:33:36.743593+0000","last_became_active":"2024-03-06T07:33:35.426399+0000","last_became_peered":"2024-03-06T07:33:35.426399+0000","last_unstale":"2024-03-06T07:33:36.743593+0000","last_undegraded":"2024-03-06T07:33:36.743593+0000","last_fullsized":"2024-03-06T07:33:36.743593+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:13.287392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.595851+0000","last_change":"2024-03-06T07:33:34.419681+0000","last_active":"2024-03-06T07:33:36.595851+0000","last_peered":"2024-03-06T07:33:36.595851+0000","last_clean":"2024-03-06T07:33:36.595851+0000","last_became_active":"2024-03-06T07:33:34.418719+0000","last_became_peered":"2024-03-06T07:33:34.418719+0000","last_unstale":"2024-03-06T07:33:36.595851+0000","last_undegraded":"2024-03-06T07:33:36.595851+0000","last_fullsized":"2024-03-06T07:33:36.595851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:43.327416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415634+0000","last_change":"2024-03-06T07:33:35.418209+0000","last_active":"2024-03-06T07:33:36.415634+0000","last_peered":"2024-03-06T07:33:36.415634+0000","last_clean":"2024-03-06T07:33:36.415634+0000","last_became_active":"2024-03-06T07:33:35.417960+0000","last_became_peered":"2024-03-06T07:33:35.417960+0000","last_unstale":"2024-03-06T07:33:36.415634+0000","last_undegraded":"2024-03-06T07:33:36.415634+0000","last_fullsized":"2024-03-06T07:33:36.415634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:34.372767+0000","last_clean_scrub_stamp":"2024-03-06T07:33:34.372767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:13.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.417118+0000","last_change":"2024-03-06T07:33:34.424965+0000","last_active":"2024-03-06T07:33:36.417118+0000","last_peered":"2024-03-06T07:33:36.417118+0000","last_clean":"2024-03-06T07:33:36.417118+0000","last_became_active":"2024-03-06T07:33:34.424153+0000","last_became_peered":"2024-03-06T07:33:34.424153+0000","last_unstale":"2024-03-06T07:33:36.417118+0000","last_undegraded":"2024-03-06T07:33:36.417118+0000","last_fullsized":"2024-03-06T07:33:36.417118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:33.354093+0000","last_clean_scrub_stamp":"2024-03-06T07:33:33.354093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:29.964030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":286,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:33:36.415966+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:33:36.415966+0000","last_peered":"2024-03-06T07:33:36.415966+0000","last_clean":"2024-03-06T07:33:36.415966+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:33:36.415966+0000","last_undegraded":"2024-03-06T07:33:36.415966+0000","last_fullsized":"2024-03-06T07:33:36.415966+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1017281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1019904,"data_stored":427537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":428477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36324,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335516,"statfs":{"total":96636764160,"available":96599568384,"internally_reserved":0,"allocated":1007616,"data_stored":426843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":428455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":427425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":427037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":1605632,"data_stored":1017245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:40.968 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:33:40.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:33:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.216+0000 7f99eae00640 1 Processor -- start 2024-03-06T07:33:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.217+0000 7f99eae00640 1 -- start start 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.218+0000 7f99eae00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99e406f690 0x7f99e40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.218+0000 7f99eae00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f99e406b6c0 0x7f99e40b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99eae00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e40d1a10 con 0x7f99e40b4120 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99eae00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e4072f70 con 0x7f99e406f690 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99eae00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e40d1450 con 0x7f99e406b6c0 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99e406f690 0x7f99e40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f99e40b4120 0x7f99e406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46856/0 (socket says 172.21.15.32:46856) 2024-03-06T07:33:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9000640 1 -- 172.21.15.32:0/987449471 learned_addr learned my addr 172.21.15.32:0/987449471 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99ea400640 1 --2- 172.21.15.32:0/987449471 >> v2:172.21.15.39:3300/0 conn(0x7f99e406b6c0 0x7f99e40b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9a00640 1 -- 172.21.15.32:0/987449471 >> v2:172.21.15.39:3300/0 conn(0x7f99e406b6c0 msgr2=0x7f99e40b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9a00640 1 --2- 172.21.15.32:0/987449471 >> v2:172.21.15.39:3300/0 conn(0x7f99e406b6c0 0x7f99e40b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9a00640 1 -- 172.21.15.32:0/987449471 >> v1:172.21.15.32:6789/0 conn(0x7f99e40b4120 legacy=0x7f99e406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.219+0000 7f99e9a00640 1 -- 172.21.15.32:0/987449471 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40d1730 con 0x7f99e406f690 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.220+0000 7f99e9a00640 1 --2- 172.21.15.32:0/987449471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e406f690 0x7f99e40728b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f99e40b5560 tx=0x7f99d802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df0181663bcd724 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99e3e00640 1 -- 172.21.15.32:0/987449471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8039070 con 0x7f99e406f690 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99e3e00640 1 -- 172.21.15.32:0/987449471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d802cbe0 con 0x7f99e406f690 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99e3e00640 1 -- 172.21.15.32:0/987449471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8034440 con 0x7f99e406f690 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99eae00640 1 -- 172.21.15.32:0/987449471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e406f690 msgr2=0x7f99e40728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99eae00640 1 --2- 172.21.15.32:0/987449471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e406f690 0x7f99e40728b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f99e40b5560 tx=0x7f99d802be30 comp rx=0 tx=0).stop 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99eae00640 1 -- 172.21.15.32:0/987449471 shutdown_connections 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99eae00640 1 --2- 172.21.15.32:0/987449471 >> v2:172.21.15.39:3300/0 conn(0x7f99e406b6c0 0x7f99e40b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99eae00640 1 --2- 172.21.15.32:0/987449471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e406f690 0x7f99e40728b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.221+0000 7f99eae00640 1 -- 172.21.15.32:0/987449471 >> 172.21.15.32:0/987449471 conn(0x7f99e406c0c0 msgr2=0x7f99e406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.222+0000 7f99eae00640 1 -- 172.21.15.32:0/987449471 shutdown_connections 2024-03-06T07:33:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.222+0000 7f99eae00640 1 -- 172.21.15.32:0/987449471 wait complete. 2024-03-06T07:33:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.223+0000 7f99eae00640 1 Processor -- start 2024-03-06T07:33:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.223+0000 7f99eae00640 1 -- start start 2024-03-06T07:33:41.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.224+0000 7f99eae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99e406b6c0 0x7f99e41413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.224+0000 7f99eae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99e406f690 0x7f99e415fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.224+0000 7f99eae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 0x7f99e40cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99eae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e406dd00 con 0x7f99e40b4120 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99eae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e406fa70 con 0x7f99e406f690 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99eae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e406de80 con 0x7f99e406b6c0 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 0x7f99e40cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 0x7f99e40cc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52260/0 (socket says 172.21.15.32:52260) 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 -- 172.21.15.32:0/2523421008 learned_addr learned my addr 172.21.15.32:0/2523421008 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99e9a00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99e406b6c0 0x7f99e41413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99e9000640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99e406f690 0x7f99e415fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 -- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99e406b6c0 msgr2=0x7f99e41413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99e406b6c0 0x7f99e41413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 -- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99e406f690 msgr2=0x7f99e415fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99e406f690 0x7f99e415fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99ea400640 1 -- 172.21.15.32:0/2523421008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e4141cb0 con 0x7f99e40b4120 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99e9a00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99e406b6c0 0x7f99e41413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.225+0000 7f99e9000640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99e406f690 0x7f99e415fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.226+0000 7f99ea400640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 0x7f99e40cc530 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00d5d0 tx=0x7f99dc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.226+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc0125f0 con 0x7f99e40b4120 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.226+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc019070 con 0x7f99e40b4120 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.226+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc0129d0 con 0x7f99e40b4120 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.226+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e41419d0 con 0x7f99e40b4120 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.226+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e40cd1f0 con 0x7f99e40b4120 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.228+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e4067830 con 0x7f99e40b4120 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.229+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99dc012b70 con 0x7f99e40b4120 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.229+0000 7f99e1600640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f99c406b4d0 0x7f99c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.230+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99dc08a570 con 0x7f99e40b4120 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.230+0000 7f99e9a00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f99c406b4d0 0x7f99c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.230+0000 7f99e9a00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f99c406b4d0 0x7f99c406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f99d8036000 tx=0x7f99d8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.239+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99e4067830 con 0x7f99e40b4120 2024-03-06T07:33:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.512+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f99e40728b0 con 0x7f99e40b4120 2024-03-06T07:33:41.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.516+0000 7f99e1600640 1 -- 172.21.15.32:0/2523421008 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f99e40728b0 con 0x7f99e40b4120 2024-03-06T07:33:41.518 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:33:41.519 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:33:41.519 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:33:41.519 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:33:41.519 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:33:41.519 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:33:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f99c406b4d0 msgr2=0x7f99c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f99c406b4d0 0x7f99c406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f99d8036000 tx=0x7f99d8030000 comp rx=0 tx=0).stop 2024-03-06T07:33:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 msgr2=0x7f99e40cc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 0x7f99e40cc530 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00d5d0 tx=0x7f99dc00da90 comp rx=0 tx=0).stop 2024-03-06T07:33:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 shutdown_connections 2024-03-06T07:33:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f99c406b4d0 0x7f99c406d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e40b4120 0x7f99e40cc530 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99e406f690 0x7f99e415fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 --2- 172.21.15.32:0/2523421008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f99e406b6c0 0x7f99e41413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 >> 172.21.15.32:0/2523421008 conn(0x7f99e406c0c0 msgr2=0x7f99e406eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 shutdown_connections 2024-03-06T07:33:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.519+0000 7f99eae00640 1 -- 172.21.15.32:0/2523421008 wait complete. 2024-03-06T07:33:41.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.744+0000 7fa8e7200640 1 Processor -- start 2024-03-06T07:33:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.744+0000 7fa8e7200640 1 -- start start 2024-03-06T07:33:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.745+0000 7fa8e7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa8e81552b0 0x7fa8e815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.745+0000 7fa8e7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e815ed10 con 0x7fa8e80cd060 2024-03-06T07:33:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.745+0000 7fa8e7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e815eff0 con 0x7fa8e815e810 2024-03-06T07:33:41.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.745+0000 7fa8e7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e815f2d0 con 0x7fa8e81552b0 2024-03-06T07:33:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.745+0000 7fa8e6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa8e80cd060 0x7fa8e80b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46870/0 (socket says 172.21.15.32:46870) 2024-03-06T07:33:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.745+0000 7fa8e6800640 1 -- 172.21.15.32:0/31168574 learned_addr learned my addr 172.21.15.32:0/31168574 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/31168574 >> v2:172.21.15.39:3300/0 conn(0x7fa8e81552b0 0x7fa8e815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248665561 0 0) 0x7fa8e815ed10 con 0x7fa8e80cd060 2024-03-06T07:33:41.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8cc003520 con 0x7fa8e80cd060 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664666227 0 0) 0x7fa8cc003520 con 0x7fa8e80cd060 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 >> v2:172.21.15.39:3300/0 conn(0x7fa8e81552b0 msgr2=0x7fa8e815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 --2- 172.21.15.32:0/31168574 >> v2:172.21.15.39:3300/0 conn(0x7fa8e81552b0 0x7fa8e815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 >> v1:172.21.15.39:6789/0 conn(0x7fa8e815e810 legacy=0x7fa8e80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e815f5b0 con 0x7fa8e80cd060 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa8e00019f0 con 0x7fa8e80cd060 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 >> v1:172.21.15.32:6789/0 conn(0x7fa8e80cd060 legacy=0x7fa8e80b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.746+0000 7fa8e4a00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc0039b0 0x7fa8cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e4a00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8cc004800 0x7fa8cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e4a00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0051c0 0x7fa8cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e815f2d0 con 0x7fa8cc0039b0 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc00c0c0 con 0x7fa8cc0051c0 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc00c2c0 con 0x7fa8cc004800 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc0039b0 0x7fa8cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e6800640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0051c0 0x7fa8cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5400640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8cc004800 0x7fa8cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5e00640 1 -- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8cc004800 msgr2=0x7fa8cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8cc004800 0x7fa8cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5e00640 1 -- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0051c0 msgr2=0x7fa8cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0051c0 0x7fa8cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5e00640 1 -- 172.21.15.32:0/31168574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc003520 con 0x7fa8cc0039b0 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e6800640 1 -- 172.21.15.32:0/31168574 reap_dead start 2024-03-06T07:33:41.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.747+0000 7fa8e5400640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8cc004800 0x7fa8cc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.749+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc0039b0 0x7fa8cc003d90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009ac0 tx=0x7fa8cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e19db69eec3185b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.749+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8039070 con 0x7fa8cc0039b0 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.749+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d802c6b0 con 0x7fa8cc0039b0 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.749+0000 7fa8e4a00640 1 -- 172.21.15.32:0/31168574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8009690 con 0x7fa8cc0039b0 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.749+0000 7fa8e7200640 1 -- 172.21.15.32:0/31168574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc0039b0 msgr2=0x7fa8cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.749+0000 7fa8e7200640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc0039b0 0x7fa8cc003d90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009ac0 tx=0x7fa8cc005830 comp rx=0 tx=0).stop 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.750+0000 7fa8e7200640 1 -- 172.21.15.32:0/31168574 shutdown_connections 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.750+0000 7fa8e7200640 1 --2- 172.21.15.32:0/31168574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8cc0039b0 0x7fa8cc003d90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.750+0000 7fa8e7200640 1 -- 172.21.15.32:0/31168574 >> 172.21.15.32:0/31168574 conn(0x7fa8e806b3d0 msgr2=0x7fa8e806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.750+0000 7fa8e7200640 1 -- 172.21.15.32:0/31168574 shutdown_connections 2024-03-06T07:33:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.750+0000 7fa8e7200640 1 -- 172.21.15.32:0/31168574 wait complete. 2024-03-06T07:33:41.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.753+0000 7fa8e7200640 1 Processor -- start 2024-03-06T07:33:41.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.753+0000 7fa8e7200640 1 -- start start 2024-03-06T07:33:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0039b0 0x7fa8e814e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 0x7fa8e8153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8e814f980 0x7fa8e814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e806dd30 con 0x7fa8e81532f0 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e80b3720 con 0x7fa8cc0039b0 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e806deb0 con 0x7fa8e814f980 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 0x7fa8e8153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 0x7fa8e8153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52288/0 (socket says 172.21.15.32:52288) 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 -- 172.21.15.32:0/1334882551 learned_addr learned my addr 172.21.15.32:0/1334882551 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0039b0 0x7fa8e814e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e6800640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8e814f980 0x7fa8e814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 -- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8e814f980 msgr2=0x7fa8e814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8e814f980 0x7fa8e814fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 -- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0039b0 msgr2=0x7fa8e814e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0039b0 0x7fa8e814e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.755+0000 7fa8e5400640 1 -- 172.21.15.32:0/1334882551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e814ead0 con 0x7fa8e81532f0 2024-03-06T07:33:41.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.756+0000 7fa8e5400640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 0x7fa8e8153770 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400d5c0 tx=0x7fa8d400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.756+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4005280 con 0x7fa8e81532f0 2024-03-06T07:33:41.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.756+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d4005420 con 0x7fa8e81532f0 2024-03-06T07:33:41.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.756+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4007270 con 0x7fa8e81532f0 2024-03-06T07:33:41.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.756+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e8150340 con 0x7fa8e81532f0 2024-03-06T07:33:41.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.756+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e8160b10 con 0x7fa8e81532f0 2024-03-06T07:33:41.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.757+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e804f330 con 0x7fa8e81532f0 2024-03-06T07:33:41.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.760+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8d401a020 con 0x7fa8e81532f0 2024-03-06T07:33:41.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.761+0000 7fa8de000640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8c806b4d0 0x7fa8c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.761+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8d4089960 con 0x7fa8e81532f0 2024-03-06T07:33:41.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.761+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8c806b4d0 0x7fa8c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.762+0000 7fa8e5e00640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8c806b4d0 0x7fa8c806d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8036000 tx=0x7fa8d8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:41.768+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8e814ead0 con 0x7fa8e81532f0 2024-03-06T07:33:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.009+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8e80b0f20 con 0x7fa8e81532f0 2024-03-06T07:33:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.011+0000 7fa8de000640 1 -- 172.21.15.32:0/1334882551 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fa8d40588a0 con 0x7fa8e81532f0 2024-03-06T07:33:42.012 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:42.012 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8c806b4d0 msgr2=0x7fa8c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8c806b4d0 0x7fa8c806d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8036000 tx=0x7fa8d8030000 comp rx=0 tx=0).stop 2024-03-06T07:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 msgr2=0x7fa8e8153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 0x7fa8e8153770 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400d5c0 tx=0x7fa8d400da80 comp rx=0 tx=0).stop 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 shutdown_connections 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8c806b4d0 0x7fa8c806d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8e814f980 0x7fa8e814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81532f0 0x7fa8e8153770 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 --2- 172.21.15.32:0/1334882551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8cc0039b0 0x7fa8e814e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 >> 172.21.15.32:0/1334882551 conn(0x7fa8e806b3d0 msgr2=0x7fa8e806e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 shutdown_connections 2024-03-06T07:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.014+0000 7fa8e7200640 1 -- 172.21.15.32:0/1334882551 wait complete. 2024-03-06T07:33:42.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:34.876717+0000', 'modified': '2024-03-06T07:33:38.348392+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6756}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6756': {'gid': 6756, 'name': 'c', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6837/3892834285', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3892834285}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3892834285}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6756, 'qdb_cluster': [6756]} 2024-03-06T07:33:42.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:33:42.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:33:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.661+0000 7f9b03e00640 1 Processor -- start 2024-03-06T07:33:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.661+0000 7f9b03e00640 1 -- start start 2024-03-06T07:33:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b03e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b0415e0b0 0x7f9b040cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b03e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b040b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b03e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b040b3710 con 0x7f9b040d4a10 2024-03-06T07:33:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b03e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b040b09e0 con 0x7f9b0415e0b0 2024-03-06T07:33:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b03e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040b39f0 con 0x7f9b040d0e50 2024-03-06T07:33:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b0415e0b0 0x7f9b040cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b0415e0b0 0x7f9b040cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52302/0 (socket says 172.21.15.32:52302) 2024-03-06T07:33:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.663+0000 7f9b02a00640 1 -- 172.21.15.32:0/3156024592 learned_addr learned my addr 172.21.15.32:0/3156024592 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.664+0000 7f9b03400640 1 --2- 172.21.15.32:0/3156024592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b040b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.664+0000 7f9b02a00640 1 -- 172.21.15.32:0/3156024592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 msgr2=0x7f9b040b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.664+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3156024592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b040b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.664+0000 7f9b02a00640 1 -- 172.21.15.32:0/3156024592 >> v1:172.21.15.39:6789/0 conn(0x7f9b040d4a10 legacy=0x7f9b040d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.664+0000 7f9b02a00640 1 -- 172.21.15.32:0/3156024592 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040b3cd0 con 0x7f9b0415e0b0 2024-03-06T07:33:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.665+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3156024592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b040cf310 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f9af4009870 tx=0x7f9af402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45343f1c3d66dfad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.665+0000 7f9b02000640 1 -- 172.21.15.32:0/3156024592 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4034070 con 0x7f9b0415e0b0 2024-03-06T07:33:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.665+0000 7f9b02000640 1 -- 172.21.15.32:0/3156024592 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4030840 con 0x7f9b0415e0b0 2024-03-06T07:33:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.665+0000 7f9b02000640 1 -- 172.21.15.32:0/3156024592 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4030c10 con 0x7f9b0415e0b0 2024-03-06T07:33:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 -- 172.21.15.32:0/3156024592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 msgr2=0x7f9b040cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3156024592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b040cf310 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f9af4009870 tx=0x7f9af402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 -- 172.21.15.32:0/3156024592 shutdown_connections 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3156024592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b040b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3156024592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b040cf310 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 -- 172.21.15.32:0/3156024592 >> 172.21.15.32:0/3156024592 conn(0x7f9b040d3d30 msgr2=0x7f9b040d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 -- 172.21.15.32:0/3156024592 shutdown_connections 2024-03-06T07:33:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.666+0000 7f9b03e00640 1 -- 172.21.15.32:0/3156024592 wait complete. 2024-03-06T07:33:42.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.668+0000 7f9b03e00640 1 Processor -- start 2024-03-06T07:33:42.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.668+0000 7f9b03e00640 1 -- start start 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b041512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b040d4a10 0x7f9b040cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b041f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040cfc60 con 0x7f9b0415e0b0 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0415e490 con 0x7f9b040d4a10 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040cf850 con 0x7f9b040d0e50 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b041f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b041f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52314/0 (socket says 172.21.15.32:52314) 2024-03-06T07:33:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.669+0000 7f9b03400640 1 -- 172.21.15.32:0/3015863829 learned_addr learned my addr 172.21.15.32:0/3015863829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b041512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9afa000640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b040d4a10 0x7f9b040cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b03400640 1 -- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 msgr2=0x7f9b041512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b03400640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b041512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b03400640 1 -- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b040d4a10 msgr2=0x7f9b040cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b03400640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b040d4a10 0x7f9b040cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b03400640 1 -- 172.21.15.32:0/3015863829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040cc460 con 0x7f9b0415e0b0 2024-03-06T07:33:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b041512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.670+0000 7f9b03400640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b041f43b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f9afc007e30 tx=0x7f9afc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.671+0000 7f9afb400640 1 -- 172.21.15.32:0/3015863829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc014620 con 0x7f9b0415e0b0 2024-03-06T07:33:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.671+0000 7f9afb400640 1 -- 172.21.15.32:0/3015863829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc019070 con 0x7f9b0415e0b0 2024-03-06T07:33:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.671+0000 7f9afb400640 1 -- 172.21.15.32:0/3015863829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc014a00 con 0x7f9b0415e0b0 2024-03-06T07:33:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.671+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b040cc740 con 0x7f9b0415e0b0 2024-03-06T07:33:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.671+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b040ccd00 con 0x7f9b0415e0b0 2024-03-06T07:33:42.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.673+0000 7f9afb400640 1 -- 172.21.15.32:0/3015863829 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9afc014ba0 con 0x7f9b0415e0b0 2024-03-06T07:33:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.674+0000 7f9afb400640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ae006b4d0 0x7f9ae006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.674+0000 7f9afb400640 1 -- 172.21.15.32:0/3015863829 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9afc08ab00 con 0x7f9b0415e0b0 2024-03-06T07:33:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.674+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ae006b4d0 0x7f9ae006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.675+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ae006b4d0 0x7f9ae006d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9af4036000 tx=0x7f9af403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.675+0000 7f9b0c3df740 1 Processor -- start 2024-03-06T07:33:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.675+0000 7f9b0c3df740 1 -- start start 2024-03-06T07:33:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055aa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055da720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055aae70 0x5610055dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x561005514600 con 0x5610055d7990 2024-03-06T07:33:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610054bd930 con 0x5610055aae70 2024-03-06T07:33:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5610055106a0 con 0x5610055d7d70 2024-03-06T07:33:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055aa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52326/0 (socket says 172.21.15.32:52326) 2024-03-06T07:33:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.677+0000 7f9b02a00640 1 -- 172.21.15.32:0/4116080822 learned_addr learned my addr 172.21.15.32:0/4116080822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9afa000640 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055da720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b03400640 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055aae70 0x5610055dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b02a00640 1 -- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 msgr2=0x5610055da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b02a00640 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055da720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b02a00640 1 -- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055aae70 msgr2=0x5610055dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b02a00640 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055aae70 0x5610055dac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b02a00640 1 -- 172.21.15.32:0/4116080822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610052292d0 con 0x5610055d7990 2024-03-06T07:33:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.678+0000 7f9b02a00640 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055aa930 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f9af40309e0 tx=0x7f9af4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06bfdf3a22ebb39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.679+0000 7f9af3000640 1 -- 172.21.15.32:0/4116080822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4053070 con 0x5610055d7990 2024-03-06T07:33:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.679+0000 7f9af3000640 1 -- 172.21.15.32:0/4116080822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4039a10 con 0x5610055d7990 2024-03-06T07:33:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.679+0000 7f9af3000640 1 -- 172.21.15.32:0/4116080822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4034040 con 0x5610055d7990 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.679+0000 7f9b0c3df740 1 -- 172.21.15.32:0/4116080822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 msgr2=0x5610055aa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.679+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055aa930 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f9af40309e0 tx=0x7f9af4030000 comp rx=0 tx=0).stop 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 -- 172.21.15.32:0/4116080822 shutdown_connections 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055aae70 0x5610055dac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055da720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/4116080822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055aa930 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 -- 172.21.15.32:0/4116080822 >> 172.21.15.32:0/4116080822 conn(0x5610055acba0 msgr2=0x5610055f5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 -- 172.21.15.32:0/4116080822 shutdown_connections 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.680+0000 7f9b0c3df740 1 -- 172.21.15.32:0/4116080822 wait complete. 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.681+0000 7f9b0c3df740 1 -- start start 2024-03-06T07:33:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.681+0000 7f9b0c3df740 1 Processor -- start 2024-03-06T07:33:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055e1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610055af140 con 0x5610055d7990 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610054d0d90 con 0x5610055e06b0 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5610055ae530 con 0x5610055d7d70 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b0c3df740 10 client.0 _ll_register_callbacks cb 0x56100550c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055d5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52336/0 (socket says 172.21.15.32:52336) 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b02a00640 1 -- 172.21.15.32:0/3004050362 learned_addr learned my addr 172.21.15.32:0/3004050362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b02a00640 1 -- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 msgr2=0x5610055ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9afa000640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.682+0000 7f9b02a00640 1 -- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 msgr2=0x5610055e1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9b03400640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055e1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055e1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9b02a00640 1 -- 172.21.15.32:0/3004050362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610052f36b0 con 0x5610055d7990 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9afa000640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9b02a00640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055d5a70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9af4000c00 tx=0x7f9af4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652adf68aeb8bdb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9ae7400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9ae7400640 1 -- 172.21.15.32:0/3004050362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4053070 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9ae7400640 1 -- 172.21.15.32:0/3004050362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9af40439c0 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.683+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9af4039a10 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9ae7400640 1 -- 172.21.15.32:0/3004050362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9af4034070 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 10 client.6891 fetch_fsmap learned FSMap version 52 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x5610055dfbd0 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9ae7400640 1 -- 172.21.15.32:0/3004050362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4039a10 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9ae7400640 1 -- 172.21.15.32:0/3004050362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9af404e470 con 0x5610055d7990 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 10 client.6891 fetch_fsmap finished waiting for FSMap version 52 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 10 client.6891 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 20 client.6891 populate_metadata read hostname 'smithi032' 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055dbd80 0x5610055dc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 4 client.6891 mds_command: new command op to 6756 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:33:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 --> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561005521b40 con 0x5610055dbd80 2024-03-06T07:33:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.684+0000 7f9afa000640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055dbd80 0x5610055dc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.685+0000 7f9afa000640 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055dbd80 0x5610055dc1a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d8cf33f6b3997e2 server_cookie=cfcf6b4f35e1d77c in_seq=0 out_seq=0 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.685+0000 7f9ae7400640 10 client.6891 ms_handle_connect on v2:172.21.15.32:6836/3892834285 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9ae7400640 1 -- 172.21.15.32:0/3004050362 <== mds.0 v2:172.21.15.32:6836/3892834285 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561005521b40 con 0x5610055dbd80 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9ae7400640 10 client.6891 handle_command_reply: tid=0 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 client.6891 shutdown 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 msgr2=0x5610055d5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055d5a70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9af4000c00 tx=0x7f9af4004730 comp rx=0 tx=0).stop 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 shutdown_connections 2024-03-06T07:33:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055dbd80 0x5610055dc1a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055e1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7d70 0x5610055ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/3004050362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055d5a70 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9ae7400640 0 client.6891 ms_handle_reset on v2:172.21.15.32:6836/3892834285 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.686+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 >> 172.21.15.32:0/3004050362 conn(0x5610055aae70 msgr2=0x5610055da710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.687+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 shutdown_connections 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.687+0000 7f9b0c3df740 1 -- 172.21.15.32:0/3004050362 wait complete. 2024-03-06T07:33:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.687+0000 7f9b0c3df740 20 client.6891 trim_cache size 0 max 16384 2024-03-06T07:33:42.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.706+0000 7f9b0c3df740 1 Processor -- start 2024-03-06T07:33:42.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.706+0000 7f9b0c3df740 1 -- start start 2024-03-06T07:33:42.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055d7d70 0x561005603bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055e06b0 0x561005604100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041ff2c0 con 0x5610055d7990 2024-03-06T07:33:42.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041fe770 con 0x5610055d7d70 2024-03-06T07:33:42.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041fed40 con 0x5610055e06b0 2024-03-06T07:33:42.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b02a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055a99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52344/0 (socket says 172.21.15.32:52344) 2024-03-06T07:33:42.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.707+0000 7f9b02a00640 1 -- 172.21.15.32:0/176296640 learned_addr learned my addr 172.21.15.32:0/176296640 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9afa000640 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055d7d70 0x561005603bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b03400640 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055e06b0 0x561005604100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b02a00640 1 -- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055e06b0 msgr2=0x561005604100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b02a00640 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055e06b0 0x561005604100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b02a00640 1 -- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055d7d70 msgr2=0x561005603bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b02a00640 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055d7d70 0x561005603bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b02a00640 1 -- 172.21.15.32:0/176296640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610055e30f0 con 0x5610055d7990 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9afa000640 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055d7d70 0x561005603bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.708+0000 7f9b02a00640 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055a99b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9af4030730 tx=0x7f9af4050070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957e5d34e03b7535 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9af2600640 1 -- 172.21.15.32:0/176296640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4069070 con 0x5610055d7990 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9af2600640 1 -- 172.21.15.32:0/176296640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af404e470 con 0x5610055d7990 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9af2600640 1 -- 172.21.15.32:0/176296640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4053070 con 0x5610055d7990 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 -- 172.21.15.32:0/176296640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 msgr2=0x5610055a99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055a99b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9af4030730 tx=0x7f9af4050070 comp rx=0 tx=0).stop 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 -- 172.21.15.32:0/176296640 shutdown_connections 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055e06b0 0x561005604100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055d7d70 0x561005603bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/176296640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7990 0x5610055a99b0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 -- 172.21.15.32:0/176296640 >> 172.21.15.32:0/176296640 conn(0x5610055aae70 msgr2=0x5610055b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.709+0000 7f9b0c3df740 1 -- 172.21.15.32:0/176296640 shutdown_connections 2024-03-06T07:33:42.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.710+0000 7f9b0c3df740 1 -- 172.21.15.32:0/176296640 wait complete. 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.710+0000 7f9b0c3df740 1 -- start start 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.710+0000 7f9b0c3df740 1 Processor -- start 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7990 0x56100564e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 0x5610056693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055d5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610055df050 con 0x5610055d7d70 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610055dff70 con 0x5610055e06b0 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x561005605df0 con 0x5610055d7990 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.711+0000 7f9b0c3df740 10 client.0 _ll_register_callbacks cb 0x7f9af4051070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 0x5610056693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 0x5610056693c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52358/0 (socket says 172.21.15.32:52358) 2024-03-06T07:33:42.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 -- 172.21.15.32:0/1435923285 learned_addr learned my addr 172.21.15.32:0/1435923285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9b02a00640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7990 0x56100564e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9b03400640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055d5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 -- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7990 msgr2=0x56100564e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7990 0x56100564e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 -- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 msgr2=0x5610055d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055d5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 -- 172.21.15.32:0/1435923285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610055f8670 con 0x5610055d7d70 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9afa000640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 0x5610056693c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9b040d4e30 tx=0x7f9aec019e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44dcdf54ee49d8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.712+0000 7f9ae7400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9ae7400640 1 -- 172.21.15.32:0/1435923285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec010040 con 0x5610055d7d70 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9ae7400640 1 -- 172.21.15.32:0/1435923285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9aec01f900 con 0x5610055d7d70 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9af404e470 con 0x5610055d7d70 2024-03-06T07:33:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9ae7400640 1 -- 172.21.15.32:0/1435923285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aec01fb00 con 0x5610055d7d70 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9b0c3df740 10 client.6897 fetch_fsmap learned FSMap version 52 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x5610055d68d0 con 0x5610055d7d70 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.713+0000 7f9ae7400640 1 -- 172.21.15.32:0/1435923285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af404e470 con 0x5610055d7d70 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9ae7400640 1 -- 172.21.15.32:0/1435923285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9aec02c070 con 0x5610055d7d70 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b0c3df740 10 client.6897 fetch_fsmap finished waiting for FSMap version 52 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b0c3df740 10 client.6897 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b0c3df740 20 client.6897 populate_metadata read hostname 'smithi032' 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055ddba0 0x561005603bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b0c3df740 4 client.6897 mds_command: new command op to 6756 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 --> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56100551be90 con 0x5610055ddba0 2024-03-06T07:33:42.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b02a00640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055ddba0 0x561005603bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9b02a00640 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055ddba0 0x561005603bc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a317c5534edb3779 server_cookie=96d874fd5e71f00d in_seq=0 out_seq=0 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.714+0000 7f9ae7400640 10 client.6897 ms_handle_connect on v2:172.21.15.32:6836/3892834285 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9ae7400640 1 -- 172.21.15.32:0/1435923285 <== mds.0 v2:172.21.15.32:6836/3892834285 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56100551be90 con 0x5610055ddba0 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9ae7400640 10 client.6897 handle_command_reply: tid=0 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 client.6897 shutdown 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 msgr2=0x5610056693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 0x5610056693c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9b040d4e30 tx=0x7f9aec019e70 comp rx=0 tx=0).stop 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 shutdown_connections 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:6836/3892834285,v1:172.21.15.32:6837/3892834285] conn(0x5610055ddba0 0x561005603bc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610055e06b0 0x5610055d5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610055d7d70 0x5610056693c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 --2- 172.21.15.32:0/1435923285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5610055d7990 0x56100564e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9ae7400640 0 client.6897 ms_handle_reset on v2:172.21.15.32:6836/3892834285 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 >> 172.21.15.32:0/1435923285 conn(0x5610055aae70 msgr2=0x5610055b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 shutdown_connections 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 1 -- 172.21.15.32:0/1435923285 wait complete. 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.715+0000 7f9b0c3df740 20 client.6897 trim_cache size 0 max 16384 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6756, 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:33:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.3151767139999997, 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 52, 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 82, 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 82, 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 19.888736801 2024-03-06T07:33:42.721 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ae006b4d0 msgr2=0x7f9ae006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ae006b4d0 0x7f9ae006d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9af4036000 tx=0x7f9af403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 msgr2=0x7f9b041f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b041f43b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f9afc007e30 tx=0x7f9afc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 shutdown_connections 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ae006b4d0 0x7f9ae006d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b0415e0b0 0x7f9b041f43b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b040d4a10 0x7f9b040cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 --2- 172.21.15.32:0/3015863829 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9b040d0e50 0x7f9b041512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 >> 172.21.15.32:0/3015863829 conn(0x7f9b040d3d30 msgr2=0x7f9b040b8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 shutdown_connections 2024-03-06T07:33:42.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.716+0000 7f9b03e00640 1 -- 172.21.15.32:0/3015863829 wait complete. 2024-03-06T07:33:42.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:33:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.940+0000 7f7486800640 1 Processor -- start 2024-03-06T07:33:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.940+0000 7f7486800640 1 -- start start 2024-03-06T07:33:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.941+0000 7f7486800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74800d2960 0x7f74800d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.941+0000 7f7486800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74800d0b00 0x7f74800b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.941+0000 7f7486800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74800d2dc0 con 0x7f748015e070 2024-03-06T07:33:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.941+0000 7f7486800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74800d18c0 con 0x7f74800d2960 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.941+0000 7f7486800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74800d1ba0 con 0x7f74800d0b00 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.941+0000 7f7484a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74800d2960 0x7f74800d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7484a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74800d2960 0x7f74800d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52362/0 (socket says 172.21.15.32:52362) 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7484a00640 1 -- 172.21.15.32:0/2108146135 learned_addr learned my addr 172.21.15.32:0/2108146135 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7485e00640 1 --2- 172.21.15.32:0/2108146135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74800d0b00 0x7f74800b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7484a00640 1 -- 172.21.15.32:0/2108146135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74800d0b00 msgr2=0x7f74800b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7484a00640 1 --2- 172.21.15.32:0/2108146135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74800d0b00 0x7f74800b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7484a00640 1 -- 172.21.15.32:0/2108146135 >> v1:172.21.15.39:6789/0 conn(0x7f748015e070 legacy=0x7f74800cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.942+0000 7f7484a00640 1 -- 172.21.15.32:0/2108146135 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74800d1e80 con 0x7f74800d2960 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.943+0000 7f7484a00640 1 --2- 172.21.15.32:0/2108146135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800d05c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f746c009870 tx=0x7f746c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fecaeddf7ec7ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f747fe00640 1 -- 172.21.15.32:0/2108146135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c034070 con 0x7f74800d2960 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f747fe00640 1 -- 172.21.15.32:0/2108146135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c030840 con 0x7f74800d2960 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f747fe00640 1 -- 172.21.15.32:0/2108146135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c030c10 con 0x7f74800d2960 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 -- 172.21.15.32:0/2108146135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 msgr2=0x7f74800d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 --2- 172.21.15.32:0/2108146135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800d05c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f746c009870 tx=0x7f746c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 -- 172.21.15.32:0/2108146135 shutdown_connections 2024-03-06T07:33:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 --2- 172.21.15.32:0/2108146135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74800d0b00 0x7f74800b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 --2- 172.21.15.32:0/2108146135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800d05c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 -- 172.21.15.32:0/2108146135 >> 172.21.15.32:0/2108146135 conn(0x7f748008d2a0 msgr2=0x7f74800d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 -- 172.21.15.32:0/2108146135 shutdown_connections 2024-03-06T07:33:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.944+0000 7f7486800640 1 -- 172.21.15.32:0/2108146135 wait complete. 2024-03-06T07:33:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.946+0000 7f7486800640 1 Processor -- start 2024-03-06T07:33:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.946+0000 7f7486800640 1 -- start start 2024-03-06T07:33:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.947+0000 7f7486800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74800d0b00 0x7f748015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.947+0000 7f7486800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.947+0000 7f7486800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f748015e070 0x7f74800ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.947+0000 7f7486800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74800d45b0 con 0x7f74800d2960 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.947+0000 7f7486800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74800d2440 con 0x7f74800d0b00 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.947+0000 7f7486800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74800d4730 con 0x7f748015e070 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52378/0 (socket says 172.21.15.32:52378) 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 -- 172.21.15.32:0/2179343278 learned_addr learned my addr 172.21.15.32:0/2179343278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7485e00640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f748015e070 0x7f74800ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7485400640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74800d0b00 0x7f748015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 -- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f748015e070 msgr2=0x7f74800ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f748015e070 0x7f74800ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 -- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74800d0b00 msgr2=0x7f748015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74800d0b00 0x7f748015fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7484a00640 1 -- 172.21.15.32:0/2179343278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74800cc3c0 con 0x7f74800d2960 2024-03-06T07:33:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.948+0000 7f7485e00640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f748015e070 0x7f74800ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.949+0000 7f7484a00640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800cbe80 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f746c000c00 tx=0x7f746c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.949+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c048070 con 0x7f74800d2960 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.949+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c030640 con 0x7f74800d2960 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.949+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c034040 con 0x7f74800d2960 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.949+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748015ea10 con 0x7f74800d2960 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.949+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748015efd0 con 0x7f74800d2960 2024-03-06T07:33:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.950+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748006b5c0 con 0x7f74800d2960 2024-03-06T07:33:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.951+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f746c037070 con 0x7f74800d2960 2024-03-06T07:33:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.952+0000 7f747d600640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f746006b500 0x7f746006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.952+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f746c0abf10 con 0x7f74800d2960 2024-03-06T07:33:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.953+0000 7f7485400640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f746006b500 0x7f746006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.953+0000 7f7485400640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f746006b500 0x7f746006d9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7474001500 tx=0x7f7474009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:42.957+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f748006b5c0 con 0x7f74800d2960 2024-03-06T07:33:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.201+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f74800c75c0 con 0x7f74800d2960 2024-03-06T07:33:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.202+0000 7f747d600640 1 -- 172.21.15.32:0/2179343278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) v1 ==== 53+0+73 (secure 0 0 0) 0x7f746c07ae10 con 0x7f74800d2960 2024-03-06T07:33:43.203 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f746006b500 msgr2=0x7f746006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f746006b500 0x7f746006d9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7474001500 tx=0x7f7474009040 comp rx=0 tx=0).stop 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 msgr2=0x7f74800cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800cbe80 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f746c000c00 tx=0x7f746c004730 comp rx=0 tx=0).stop 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 shutdown_connections 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f746006b500 0x7f746006d9c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f748015e070 0x7f74800ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74800d2960 0x7f74800cbe80 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 --2- 172.21.15.32:0/2179343278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74800d0b00 0x7f748015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 >> 172.21.15.32:0/2179343278 conn(0x7f748008d2a0 msgr2=0x7f748015d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 shutdown_connections 2024-03-06T07:33:43.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:43.205+0000 7f7486800640 1 -- 172.21.15.32:0/2179343278 wait complete. 2024-03-06T07:33:43.226 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:33:43.226 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:33:43.226 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:33:43.226 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:33:43.226 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:43.227 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:33:43.227 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:33:43.227 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:33:43.227 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:43.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.227 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:33:43.278 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:33:43.278 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:33:43.278 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:33:43.278 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.278 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:33:43.278 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42114sec preferred_lft 42114sec 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:43.279 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:33:43.280 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:43.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.281 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:33:43.356 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:33:43.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.361 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:33:43.410 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:33:43.411 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:33:43.411 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:33:43.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:33:43.411 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:33:43.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.411 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:33:43.411 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:33:43.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:33:43.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:33:43.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:33:43.412 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.589 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:33:43.647 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:33:43.648 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:33:43.648 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:43.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:33:43.744 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:33:43.744 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:43.800 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:43.800 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:33:43.801 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:43.844 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:33:43.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:43.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:44.019 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:44.019 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:33:44.053 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:44.053 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:33:44.053 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:33:44.053 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:33:44.053 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:33:44.055 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:33:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:33:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:33:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:33:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:33:44.058 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:33:44.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:44.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:44.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:44.133 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:33:44.186 INFO:teuthology.orchestra.run.smithi032.stdout:client5303 2024-03-06T07:33:44.186 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:44.186 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:44.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:44.232 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:33:44.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:44.340 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:33:44.406 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:44.407 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:44.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:44.457 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:33:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:35+0 records in 2024-03-06T07:33:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:35+0 records out 2024-03-06T07:33:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.345971 s, 106 MB/s 2024-03-06T07:33:44.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:44.852 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:33:44.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.145+0000 7f8095400640 1 Processor -- start 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.145+0000 7f8095400640 1 -- start start 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f8095400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f809015e070 0x7f80900d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f8095400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900d18c0 con 0x7f80900d0cd0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f8095400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900d1ba0 con 0x7f80900cf410 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f8095400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80900d1e80 con 0x7f809015e070 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f8094a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80900d0cd0 0x7f80900b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46876/0 (socket says 172.21.15.32:46876) 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f808fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f809015e070 0x7f80900d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.147+0000 7f8094a00640 1 -- 172.21.15.32:0/1682018058 learned_addr learned my addr 172.21.15.32:0/1682018058 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.148+0000 7f808f400640 1 -- 172.21.15.32:0/1682018058 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573839766 0 0) 0x7f80900d18c0 con 0x7f80900d0cd0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.148+0000 7f808f400640 1 -- 172.21.15.32:0/1682018058 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8074003520 con 0x7f80900d0cd0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.148+0000 7f808fe00640 1 -- 172.21.15.32:0/1682018058 >> v1:172.21.15.39:6789/0 conn(0x7f80900cf410 legacy=0x7f80900d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.148+0000 7f808fe00640 1 -- 172.21.15.32:0/1682018058 >> v1:172.21.15.32:6789/0 conn(0x7f80900d0cd0 legacy=0x7f80900b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.148+0000 7f808fe00640 1 -- 172.21.15.32:0/1682018058 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900d2160 con 0x7f809015e070 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.150+0000 7f808fe00640 1 --2- 172.21.15.32:0/1682018058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809015e070 0x7f80900d2960 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f808002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83dab1c10663864 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.151+0000 7f808f400640 1 -- 172.21.15.32:0/1682018058 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080034070 con 0x7f809015e070 2024-03-06T07:33:45.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.151+0000 7f808f400640 1 -- 172.21.15.32:0/1682018058 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080030610 con 0x7f809015e070 2024-03-06T07:33:45.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.151+0000 7f808f400640 1 -- 172.21.15.32:0/1682018058 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80800309c0 con 0x7f809015e070 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.151+0000 7f8095400640 1 -- 172.21.15.32:0/1682018058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809015e070 msgr2=0x7f80900d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.151+0000 7f8095400640 1 --2- 172.21.15.32:0/1682018058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809015e070 0x7f80900d2960 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f808002b960 comp rx=0 tx=0).stop 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.152+0000 7f8095400640 1 -- 172.21.15.32:0/1682018058 shutdown_connections 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.152+0000 7f8095400640 1 --2- 172.21.15.32:0/1682018058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809015e070 0x7f80900d2960 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.152+0000 7f8095400640 1 -- 172.21.15.32:0/1682018058 >> 172.21.15.32:0/1682018058 conn(0x7f809008d2a0 msgr2=0x7f80900d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.152+0000 7f8095400640 1 -- 172.21.15.32:0/1682018058 shutdown_connections 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.152+0000 7f8095400640 1 -- 172.21.15.32:0/1682018058 wait complete. 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.153+0000 7f8095400640 1 Processor -- start 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.154+0000 7f8095400640 1 -- start start 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8095400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 0x7f80900b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8095400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80900d0cd0 0x7f80901548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8095400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f809015e070 0x7f8090154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8095400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80900d4730 con 0x7f80900cf410 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8095400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80900d45b0 con 0x7f80900d0cd0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8095400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80900cf870 con 0x7f809015e070 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 0x7f80900b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 0x7f80900b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52394/0 (socket says 172.21.15.32:52394) 2024-03-06T07:33:45.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 -- 172.21.15.32:0/208093034 learned_addr learned my addr 172.21.15.32:0/208093034 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8094a00640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f809015e070 0x7f8090154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8087400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80900d0cd0 0x7f80901548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 -- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f809015e070 msgr2=0x7f8090154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f809015e070 0x7f8090154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 -- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80900d0cd0 msgr2=0x7f80901548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80900d0cd0 0x7f80901548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f808fe00640 1 -- 172.21.15.32:0/208093034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809015e450 con 0x7f80900cf410 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.155+0000 7f8094a00640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f809015e070 0x7f8090154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f8087400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80900d0cd0 0x7f80901548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f808fe00640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 0x7f80900b8d20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f80800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080047070 con 0x7f80900cf410 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080030410 con 0x7f80900cf410 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080034040 con 0x7f80900cf410 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809015e730 con 0x7f80900cf410 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.156+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809015ecf0 con 0x7f80900cf410 2024-03-06T07:33:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.158+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809004f330 con 0x7f80900cf410 2024-03-06T07:33:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.158+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f808004a070 con 0x7f80900cf410 2024-03-06T07:33:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.159+0000 7f808cc00640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f806c06b500 0x7f806c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.159+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80800aba40 con 0x7f80900cf410 2024-03-06T07:33:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.160+0000 7f8087400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f806c06b500 0x7f806c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.160+0000 7f8087400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f806c06b500 0x7f806c06d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f80900b2730 tx=0x7f8078009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.164+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f808007a1d0 con 0x7f80900cf410 2024-03-06T07:33:45.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.404+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80900d1370 con 0x7f80900cf410 2024-03-06T07:33:45.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.404+0000 7f808cc00640 1 -- 172.21.15.32:0/208093034 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f808007e270 con 0x7f80900cf410 2024-03-06T07:33:45.406 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:45.406 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:45.406 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f806c06b500 msgr2=0x7f806c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f806c06b500 0x7f806c06d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f80900b2730 tx=0x7f8078009040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 msgr2=0x7f80900b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 0x7f80900b8d20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f80800046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 shutdown_connections 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f806c06b500 0x7f806c06d9c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f809015e070 0x7f8090154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80900d0cd0 0x7f80901548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 --2- 172.21.15.32:0/208093034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80900cf410 0x7f80900b8d20 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 >> 172.21.15.32:0/208093034 conn(0x7f809008d2a0 msgr2=0x7f809006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 shutdown_connections 2024-03-06T07:33:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.407+0000 7f8095400640 1 -- 172.21.15.32:0/208093034 wait complete. 2024-03-06T07:33:45.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:33:45.669 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:33:45.669 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:33:45.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:33:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.869+0000 7fcecd000640 1 Processor -- start 2024-03-06T07:33:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.869+0000 7fcecd000640 1 -- start start 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcecd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcec815ba20 0x7fcec8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcecd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcec80afab0 0x7fcec815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcecd000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcecd000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec806f2c0 con 0x7fcec80afab0 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcecd000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec815be00 con 0x7fcec815ba20 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcecd000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec806d4b0 con 0x7fcec815f190 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.871+0000 7fcebfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcec80afab0 0x7fcec815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcec80afab0 0x7fcec815ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52396/0 (socket says 172.21.15.32:52396) 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 -- 172.21.15.32:0/3789793648 learned_addr learned my addr 172.21.15.32:0/3789793648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcec7400640 1 --2- 172.21.15.32:0/3789793648 >> v2:172.21.15.39:3300/0 conn(0x7fcec815ba20 0x7fcec8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcec7e00640 1 --2- 172.21.15.32:0/3789793648 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 -- 172.21.15.32:0/3789793648 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 msgr2=0x7fcec80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 --2- 172.21.15.32:0/3789793648 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec80ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 -- 172.21.15.32:0/3789793648 >> v2:172.21.15.39:3300/0 conn(0x7fcec815ba20 msgr2=0x7fcec8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 --2- 172.21.15.32:0/3789793648 >> v2:172.21.15.39:3300/0 conn(0x7fcec815ba20 0x7fcec8066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcebfe00640 1 -- 172.21.15.32:0/3789793648 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec806d790 con 0x7fcec80afab0 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.872+0000 7fcec7400640 1 --2- 172.21.15.32:0/3789793648 >> v2:172.21.15.39:3300/0 conn(0x7fcec815ba20 0x7fcec8066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.873+0000 7fcebfe00640 1 --2- 172.21.15.32:0/3789793648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec815ec50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fcec804f960 tx=0x7fceb002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7321c77010d3b4b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcec6a00640 1 -- 172.21.15.32:0/3789793648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb00317d0 con 0x7fcec80afab0 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcec6a00640 1 -- 172.21.15.32:0/3789793648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb0037070 con 0x7fcec80afab0 2024-03-06T07:33:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcec6a00640 1 -- 172.21.15.32:0/3789793648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0031b90 con 0x7fcec80afab0 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 -- 172.21.15.32:0/3789793648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 msgr2=0x7fcec815ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 --2- 172.21.15.32:0/3789793648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec815ec50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fcec804f960 tx=0x7fceb002b870 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 -- 172.21.15.32:0/3789793648 shutdown_connections 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 --2- 172.21.15.32:0/3789793648 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec80ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 --2- 172.21.15.32:0/3789793648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec815ec50 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 --2- 172.21.15.32:0/3789793648 >> v2:172.21.15.39:3300/0 conn(0x7fcec815ba20 0x7fcec8066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.874+0000 7fcecd000640 1 -- 172.21.15.32:0/3789793648 >> 172.21.15.32:0/3789793648 conn(0x7fcec80ccf70 msgr2=0x7fcec80cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.875+0000 7fcecd000640 1 -- 172.21.15.32:0/3789793648 shutdown_connections 2024-03-06T07:33:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.875+0000 7fcecd000640 1 -- 172.21.15.32:0/3789793648 wait complete. 2024-03-06T07:33:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.876+0000 7fcecd000640 1 Processor -- start 2024-03-06T07:33:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.876+0000 7fcecd000640 1 -- start start 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.877+0000 7fcecd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec8149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.877+0000 7fcecd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec815ba20 0x7fcec8160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.877+0000 7fcecd000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec8160f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.877+0000 7fcecd000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec80ce8b0 con 0x7fcec80afab0 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.877+0000 7fcecd000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec815e420 con 0x7fcec815ba20 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.877+0000 7fcecd000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec80cea30 con 0x7fcec815f190 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec8149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec8149220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52404/0 (socket says 172.21.15.32:52404) 2024-03-06T07:33:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 -- 172.21.15.32:0/821340516 learned_addr learned my addr 172.21.15.32:0/821340516 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 -- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 msgr2=0x7fcec8160f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7e00640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec8160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcebfe00640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec815ba20 0x7fcec8160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec8160f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 -- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec815ba20 msgr2=0x7fcec8160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec815ba20 0x7fcec8160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcec7400640 1 -- 172.21.15.32:0/821340516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec81539d0 con 0x7fcec80afab0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.878+0000 7fcebfe00640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec815ba20 0x7fcec8160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.879+0000 7fcec7400640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec8149220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fceb800e730 tx=0x7fceb800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.879+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb800f040 con 0x7fcec80afab0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.879+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb8004230 con 0x7fcec80afab0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.879+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb800ee40 con 0x7fcec80afab0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.879+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec8153cb0 con 0x7fcec80afab0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.879+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec8154270 con 0x7fcec80afab0 2024-03-06T07:33:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.880+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec804f330 con 0x7fcec80afab0 2024-03-06T07:33:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.883+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceb800c6d0 con 0x7fcec80afab0 2024-03-06T07:33:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.884+0000 7fcebf400640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcea406b4d0 0x7fcea406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.884+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceb8089e90 con 0x7fcec80afab0 2024-03-06T07:33:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.884+0000 7fcebfe00640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcea406b4d0 0x7fcea406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.885+0000 7fcebfe00640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcea406b4d0 0x7fcea406d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fceb0036000 tx=0x7fceb0008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:45.889+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fceb8017020 con 0x7fcec80afab0 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.166+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fcec80ce050 con 0x7fcec80afab0 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.166+0000 7fcebf400640 1 -- 172.21.15.32:0/821340516 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1653 (secure 0 0 0) 0x7fceb8058da0 con 0x7fcec80afab0 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout: "total_bytes": 773094113280, 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout: "total_avail_bytes": 772812308480, 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_bytes": 281804800, 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_bytes": 281804800, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_ratio": 0.00036451552296057343, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "num_osds": 8, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "num_per_pool_osds": 8, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "stats_by_class": { 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "ssd": { 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "total_bytes": 773094113280, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "total_avail_bytes": 772812308480, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_bytes": 281804800, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_bytes": 281804800, 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: "total_used_raw_ratio": 0.00036451552296057343 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "name": ".mgr", 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 1, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 590368, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 2, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 1160, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 1187840, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 1.6180059674297809e-06, 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367068512256 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephfs_metadata", 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 2410, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 22, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 64, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 65536, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 8.9269427405724855e-08, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367068512256 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephfs_data", 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 13, 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "stats": { 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "stored": 0, 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "objects": 0, 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kb_used": 0, 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 0, 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "percent_used": 0, 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: "max_avail": 367068512256 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcea406b4d0 msgr2=0x7fcea406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcea406b4d0 0x7fcea406d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fceb0036000 tx=0x7fceb0008030 comp rx=0 tx=0).stop 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 msgr2=0x7fcec8149220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec8149220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fceb800e730 tx=0x7fceb800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 shutdown_connections 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcea406b4d0 0x7fcea406d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcec815f190 0x7fcec8160f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec815ba20 0x7fcec8160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 --2- 172.21.15.32:0/821340516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcec80afab0 0x7fcec8149220 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 >> 172.21.15.32:0/821340516 conn(0x7fcec80ccf70 msgr2=0x7fcec806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 shutdown_connections 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.169+0000 7fcecd000640 1 -- 172.21.15.32:0/821340516 wait complete. 2024-03-06T07:33:46.196 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:33:46.196 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:46.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:46.304 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:46.423 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:46.423 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:46.423 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:46.480 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:46.480 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:46.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:46.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.745+0000 7f4573200640 1 Processor -- start 2024-03-06T07:33:46.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.745+0000 7f4573200640 1 -- start start 2024-03-06T07:33:46.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.748+0000 7f4573200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f4574070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.748+0000 7f4573200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f45740cd060 0x7f457406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.748+0000 7f4573200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740cdad0 con 0x7f45740ce850 2024-03-06T07:33:46.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.748+0000 7f4573200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4574071500 con 0x7f45740cd060 2024-03-06T07:33:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.748+0000 7f4573200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740cdd70 con 0x7f457415c920 2024-03-06T07:33:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.749+0000 7f4571e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f4574070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.749+0000 7f4571400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f45740ce850 0x7f45740cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:38718/0 (socket says 172.21.15.32:38718) 2024-03-06T07:33:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.749+0000 7f4571400640 1 -- 172.21.15.32:0/435547658 learned_addr learned my addr 172.21.15.32:0/435547658 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:46.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.749+0000 7f4572800640 1 --2- 172.21.15.32:0/435547658 >> v2:172.21.15.39:3300/0 conn(0x7f45740cd060 0x7f457406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4570a00640 1 -- 172.21.15.32:0/435547658 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153040101 0 0) 0x7f45740cdad0 con 0x7f45740ce850 2024-03-06T07:33:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4570a00640 1 -- 172.21.15.32:0/435547658 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4558003520 con 0x7f45740ce850 2024-03-06T07:33:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4572800640 1 -- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 msgr2=0x7f4574070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4572800640 1 --2- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f4574070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4572800640 1 -- 172.21.15.32:0/435547658 >> v1:172.21.15.39:6789/0 conn(0x7f45740ce850 legacy=0x7f45740cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4572800640 1 -- 172.21.15.32:0/435547658 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740ce050 con 0x7f45740cd060 2024-03-06T07:33:46.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.750+0000 7f4571e00640 1 --2- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f4574070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:46.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.751+0000 7f4572800640 1 --2- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f457406cac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f456c009d90 tx=0x7f456c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3fb1bc885d16ddb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4570a00640 1 -- 172.21.15.32:0/435547658 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c039070 con 0x7f45740cd060 2024-03-06T07:33:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4570a00640 1 -- 172.21.15.32:0/435547658 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c02cc60 con 0x7f45740cd060 2024-03-06T07:33:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4570a00640 1 -- 172.21.15.32:0/435547658 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c034660 con 0x7f45740cd060 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4573200640 1 -- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 msgr2=0x7f457406cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4573200640 1 --2- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f457406cac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f456c009d90 tx=0x7f456c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4573200640 1 -- 172.21.15.32:0/435547658 shutdown_connections 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4573200640 1 --2- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f457406cac0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4573200640 1 --2- 172.21.15.32:0/435547658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f4574070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.752+0000 7f4573200640 1 -- 172.21.15.32:0/435547658 >> 172.21.15.32:0/435547658 conn(0x7f457406b3d0 msgr2=0x7f457406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.753+0000 7f4573200640 1 -- 172.21.15.32:0/435547658 shutdown_connections 2024-03-06T07:33:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.753+0000 7f4573200640 1 -- 172.21.15.32:0/435547658 wait complete. 2024-03-06T07:33:46.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.755+0000 7f4573200640 1 Processor -- start 2024-03-06T07:33:46.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.755+0000 7f4573200640 1 -- start start 2024-03-06T07:33:46.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4573200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f4574159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4573200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 0x7f45740b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4573200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f457415f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4573200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574070f80 con 0x7f45740ce850 2024-03-06T07:33:46.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4573200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574163790 con 0x7f45740cd060 2024-03-06T07:33:46.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4573200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4574071100 con 0x7f457415c920 2024-03-06T07:33:46.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4571400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 0x7f45740b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4571400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 0x7f45740b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52414/0 (socket says 172.21.15.32:52414) 2024-03-06T07:33:46.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.756+0000 7f4571400640 1 -- 172.21.15.32:0/1452462450 learned_addr learned my addr 172.21.15.32:0/1452462450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:46.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4572800640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f457415f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571e00640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f4574159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571400640 1 -- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 msgr2=0x7f457415f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571400640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f457415f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571400640 1 -- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 msgr2=0x7f4574159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571400640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f4574159b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571400640 1 -- 172.21.15.32:0/1452462450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457415d320 con 0x7f45740ce850 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4572800640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f457415f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.757+0000 7f4571400640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 0x7f45740b1d10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f456000c460 tx=0x7f456000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.758+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560017070 con 0x7f45740ce850 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.758+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4560005ce0 con 0x7f45740ce850 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.758+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560007290 con 0x7f45740ce850 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.758+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457415d600 con 0x7f45740ce850 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.758+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457415dbc0 con 0x7f45740ce850 2024-03-06T07:33:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.759+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457404f330 con 0x7f45740ce850 2024-03-06T07:33:46.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.760+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f456001a070 con 0x7f45740ce850 2024-03-06T07:33:46.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.762+0000 7f456a000640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f455406b4d0 0x7f455406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.762+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45600895e0 con 0x7f45740ce850 2024-03-06T07:33:46.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.762+0000 7f4571e00640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f455406b4d0 0x7f455406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.763+0000 7f4571e00640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f455406b4d0 0x7f455406d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4564009950 tx=0x7f4564008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.766+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45600583d0 con 0x7f45740ce850 2024-03-06T07:33:46.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.967+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45740cf3a0 con 0x7f45740ce850 2024-03-06T07:33:46.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.969+0000 7f456a000640 1 -- 172.21.15.32:0/1452462450 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16552 (secure 0 0 0) 0x7f456005e0d0 con 0x7f45740ce850 2024-03-06T07:33:46.971 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:46.971 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:36.396956+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.971+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f455406b4d0 msgr2=0x7f455406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f455406b4d0 0x7f455406d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4564009950 tx=0x7f4564008040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 msgr2=0x7f45740b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 0x7f45740b1d10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f456000c460 tx=0x7f456000c920 comp rx=0 tx=0).stop 2024-03-06T07:33:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 shutdown_connections 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f455406b4d0 0x7f455406d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f457415c920 0x7f457415f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45740ce850 0x7f45740b1d10 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 --2- 172.21.15.32:0/1452462450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45740cd060 0x7f4574159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 >> 172.21.15.32:0/1452462450 conn(0x7f457406b3d0 msgr2=0x7f4574163980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 shutdown_connections 2024-03-06T07:33:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:46.972+0000 7f4573200640 1 -- 172.21.15.32:0/1452462450 wait complete. 2024-03-06T07:33:46.994 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:32.774877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:33.889752+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:46.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:47.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.206+0000 7efe4d600640 1 Processor -- start 2024-03-06T07:33:47.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.207+0000 7efe4d600640 1 -- start start 2024-03-06T07:33:47.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.208+0000 7efe4d600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 0x7efe480b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.208+0000 7efe4d600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4815fe40 con 0x7efe48145430 2024-03-06T07:33:47.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.208+0000 7efe4d600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe48160120 con 0x7efe480b2080 2024-03-06T07:33:47.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.208+0000 7efe4d600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe48160400 con 0x7efe4815f810 2024-03-06T07:33:47.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.208+0000 7efe47e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efe48145430 0x7efe4815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46892/0 (socket says 172.21.15.32:46892) 2024-03-06T07:33:47.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.208+0000 7efe47e00640 1 -- 172.21.15.32:0/209549722 learned_addr learned my addr 172.21.15.32:0/209549722 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe47400640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 0x7efe480b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131255617 0 0) 0x7efe4815fe40 con 0x7efe48145430 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe2c003520 con 0x7efe48145430 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913608028 0 0) 0x7efe2c003520 con 0x7efe48145430 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 msgr2=0x7efe480b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 0x7efe480b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 >> v1:172.21.15.39:6789/0 conn(0x7efe480b2080 legacy=0x7efe480b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe481606e0 con 0x7efe48145430 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7efe38001670 con 0x7efe48145430 2024-03-06T07:33:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.209+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 >> v1:172.21.15.32:6789/0 conn(0x7efe48145430 legacy=0x7efe4815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe46a00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe2c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe46a00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe2c004800 0x7efe2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe46a00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 0x7efe2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe48160400 con 0x7efe2c0039e0 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c00baa0 con 0x7efe2c004800 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c00bca0 con 0x7efe4815f810 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47e00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe2c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe4cc00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 0x7efe2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47e00640 1 -- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 msgr2=0x7efe2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47e00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe4815f810 0x7efe2c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47e00640 1 -- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe2c004800 msgr2=0x7efe2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47400640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe2c004800 0x7efe2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47e00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe2c004800 0x7efe2c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.210+0000 7efe47e00640 1 -- 172.21.15.32:0/209549722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c003520 con 0x7efe2c0039e0 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.212+0000 7efe47e00640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe2c003e00 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c0e0 tx=0x7efe2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd073f74d99405e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.212+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38007c40 con 0x7efe2c0039e0 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.212+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe38030070 con 0x7efe2c0039e0 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.212+0000 7efe46a00640 1 -- 172.21.15.32:0/209549722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38005d30 con 0x7efe2c0039e0 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4d600640 1 -- 172.21.15.32:0/209549722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 msgr2=0x7efe2c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4d600640 1 --2- 172.21.15.32:0/209549722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe2c003e00 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c0e0 tx=0x7efe2c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4cc00640 1 -- 172.21.15.32:0/209549722 reap_dead start 2024-03-06T07:33:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4d600640 1 -- 172.21.15.32:0/209549722 shutdown_connections 2024-03-06T07:33:47.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4d600640 1 -- 172.21.15.32:0/209549722 >> 172.21.15.32:0/209549722 conn(0x7efe4806b3a0 msgr2=0x7efe4806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4d600640 1 -- 172.21.15.32:0/209549722 shutdown_connections 2024-03-06T07:33:47.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.213+0000 7efe4d600640 1 -- 172.21.15.32:0/209549722 wait complete. 2024-03-06T07:33:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.216+0000 7efe4d600640 1 Processor -- start 2024-03-06T07:33:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.216+0000 7efe4d600640 1 -- start start 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe4d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe481f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe4d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe481543f0 0x7efe48138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe4d600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe481392d0 0x7efe4813a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe4d600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe480cdf80 con 0x7efe2c0039e0 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe4d600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe480cde00 con 0x7efe481543f0 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe4d600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe480b1b40 con 0x7efe481392d0 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe481f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe481f1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52432/0 (socket says 172.21.15.32:52432) 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe47e00640 1 -- 172.21.15.32:0/3595011570 learned_addr learned my addr 172.21.15.32:0/3595011570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.218+0000 7efe47400640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe481543f0 0x7efe48138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe4cc00640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe481392d0 0x7efe4813a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47e00640 1 -- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe481392d0 msgr2=0x7efe4813a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47e00640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe481392d0 0x7efe4813a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47e00640 1 -- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe481543f0 msgr2=0x7efe48138d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47e00640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe481543f0 0x7efe48138d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47e00640 1 -- 172.21.15.32:0/3595011570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4813b030 con 0x7efe2c0039e0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47400640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe481543f0 0x7efe48138d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe4cc00640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe481392d0 0x7efe4813a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe47e00640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe481f1ce0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c200 tx=0x7efe48160010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38045070 con 0x7efe2c0039e0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe38000ed0 con 0x7efe2c0039e0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe481413e0 con 0x7efe2c0039e0 2024-03-06T07:33:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.219+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38007c40 con 0x7efe2c0039e0 2024-03-06T07:33:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.220+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe48154870 con 0x7efe2c0039e0 2024-03-06T07:33:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.221+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4804f330 con 0x7efe2c0039e0 2024-03-06T07:33:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.221+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe38048070 con 0x7efe2c0039e0 2024-03-06T07:33:47.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.222+0000 7efe3fe00640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efe2406b4d0 0x7efe2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.223+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe380ab2c0 con 0x7efe2c0039e0 2024-03-06T07:33:47.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.223+0000 7efe47400640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efe2406b4d0 0x7efe2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.224+0000 7efe47400640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efe2406b4d0 0x7efe2406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7efe480b4110 tx=0x7efe30008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.227+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe38079ae0 con 0x7efe2c0039e0 2024-03-06T07:33:47.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.428+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe481396b0 con 0x7efe2c0039e0 2024-03-06T07:33:47.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.429+0000 7efe3fe00640 1 -- 172.21.15.32:0/3595011570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe3807db80 con 0x7efe2c0039e0 2024-03-06T07:33:47.430 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efe2406b4d0 msgr2=0x7efe2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efe2406b4d0 0x7efe2406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7efe480b4110 tx=0x7efe30008040 comp rx=0 tx=0).stop 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 msgr2=0x7efe481f1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe481f1ce0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c200 tx=0x7efe48160010 comp rx=0 tx=0).stop 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 shutdown_connections 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efe2406b4d0 0x7efe2406d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efe481392d0 0x7efe4813a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe481543f0 0x7efe48138d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 --2- 172.21.15.32:0/3595011570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2c0039e0 0x7efe481f1ce0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.433+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 >> 172.21.15.32:0/3595011570 conn(0x7efe4806b3a0 msgr2=0x7efe480b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.434+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 shutdown_connections 2024-03-06T07:33:47.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.434+0000 7efe4d600640 1 -- 172.21.15.32:0/3595011570 wait complete. 2024-03-06T07:33:47.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:47.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.915+0000 7fb99be00640 1 Processor -- start 2024-03-06T07:33:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.915+0000 7fb99be00640 1 -- start start 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.917+0000 7fb99be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb99c15e0c0 0x7fb99c072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.917+0000 7fb99be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb99c06afb0 0x7fb99c06b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.917+0000 7fb99be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c0b5800 con 0x7fb99c0cd600 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.917+0000 7fb99be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c06dd30 con 0x7fb99c15e0c0 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.917+0000 7fb99be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c0b5aa0 con 0x7fb99c06afb0 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb99c15e0c0 0x7fb99c072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb99c15e0c0 0x7fb99c072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48264/0 (socket says 172.21.15.32:48264) 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 -- 172.21.15.32:0/3801366680 learned_addr learned my addr 172.21.15.32:0/3801366680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99a000640 1 --2- 172.21.15.32:0/3801366680 >> v2:172.21.15.39:3300/0 conn(0x7fb99c06afb0 0x7fb99c06b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 -- 172.21.15.32:0/3801366680 >> v2:172.21.15.39:3300/0 conn(0x7fb99c06afb0 msgr2=0x7fb99c06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 --2- 172.21.15.32:0/3801366680 >> v2:172.21.15.39:3300/0 conn(0x7fb99c06afb0 0x7fb99c06b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 -- 172.21.15.32:0/3801366680 >> v1:172.21.15.39:6789/0 conn(0x7fb99c0cd600 legacy=0x7fb99c0cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.918+0000 7fb99aa00640 1 -- 172.21.15.32:0/3801366680 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c0b5d80 con 0x7fb99c15e0c0 2024-03-06T07:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.920+0000 7fb99aa00640 1 --2- 172.21.15.32:0/3801366680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c072270 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb99c071b60 tx=0x7fb98c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ab9377721084e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.920+0000 7fb999600640 1 -- 172.21.15.32:0/3801366680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c034070 con 0x7fb99c15e0c0 2024-03-06T07:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.920+0000 7fb999600640 1 -- 172.21.15.32:0/3801366680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c030870 con 0x7fb99c15e0c0 2024-03-06T07:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.920+0000 7fb999600640 1 -- 172.21.15.32:0/3801366680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c030c40 con 0x7fb99c15e0c0 2024-03-06T07:33:47.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.920+0000 7fb99be00640 1 -- 172.21.15.32:0/3801366680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 msgr2=0x7fb99c072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.920+0000 7fb99be00640 1 --2- 172.21.15.32:0/3801366680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c072270 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb99c071b60 tx=0x7fb98c02b9b0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.921+0000 7fb99be00640 1 -- 172.21.15.32:0/3801366680 shutdown_connections 2024-03-06T07:33:47.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.921+0000 7fb99be00640 1 --2- 172.21.15.32:0/3801366680 >> v2:172.21.15.39:3300/0 conn(0x7fb99c06afb0 0x7fb99c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.921+0000 7fb99be00640 1 --2- 172.21.15.32:0/3801366680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c072270 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.921+0000 7fb99be00640 1 -- 172.21.15.32:0/3801366680 >> 172.21.15.32:0/3801366680 conn(0x7fb99c0d0c00 msgr2=0x7fb99c0d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.921+0000 7fb99be00640 1 -- 172.21.15.32:0/3801366680 shutdown_connections 2024-03-06T07:33:47.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.921+0000 7fb99be00640 1 -- 172.21.15.32:0/3801366680 wait complete. 2024-03-06T07:33:47.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.922+0000 7fb99be00640 1 Processor -- start 2024-03-06T07:33:47.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.923+0000 7fb99be00640 1 -- start start 2024-03-06T07:33:47.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.923+0000 7fb99be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb99c06afb0 0x7fb99c15f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.923+0000 7fb99be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99c0cd600 0x7fb99c1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.923+0000 7fb99be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.923+0000 7fb99be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0727b0 con 0x7fb99c15e0c0 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.923+0000 7fb99be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0cef60 con 0x7fb99c0cd600 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c072cc0 con 0x7fb99c06afb0 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c1f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48272/0 (socket says 172.21.15.32:48272) 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 -- 172.21.15.32:0/3674785256 learned_addr learned my addr 172.21.15.32:0/3674785256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99a000640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99c0cd600 0x7fb99c1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 -- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb99c06afb0 msgr2=0x7fb99c15f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb99c06afb0 0x7fb99c15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 -- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99c0cd600 msgr2=0x7fb99c1520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99c0cd600 0x7fb99c1520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 -- 172.21.15.32:0/3674785256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c160360 con 0x7fb99c15e0c0 2024-03-06T07:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.924+0000 7fb99b400640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c1f3990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fb99400c730 tx=0x7fb99400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.925+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994003b90 con 0x7fb99c15e0c0 2024-03-06T07:33:47.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.925+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb994003d30 con 0x7fb99c15e0c0 2024-03-06T07:33:47.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.925+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99400aa90 con 0x7fb99c15e0c0 2024-03-06T07:33:47.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.925+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c160640 con 0x7fb99c15e0c0 2024-03-06T07:33:47.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.925+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c160c00 con 0x7fb99c15e0c0 2024-03-06T07:33:47.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.926+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c161030 con 0x7fb99c15e0c0 2024-03-06T07:33:47.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.928+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb99400ac30 con 0x7fb99c15e0c0 2024-03-06T07:33:47.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.929+0000 7fb992a00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb97c06b4d0 0x7fb97c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.929+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb99408a1d0 con 0x7fb99c15e0c0 2024-03-06T07:33:47.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.929+0000 7fb99aa00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb97c06b4d0 0x7fb97c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.930+0000 7fb99aa00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb97c06b4d0 0x7fb97c06d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb98c036000 tx=0x7fb98c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:47.933+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb994058f70 con 0x7fb99c15e0c0 2024-03-06T07:33:48.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.133+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb99c0aff80 con 0x7fb99c15e0c0 2024-03-06T07:33:48.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.134+0000 7fb992a00640 1 -- 172.21.15.32:0/3674785256 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb994058870 con 0x7fb99c15e0c0 2024-03-06T07:33:48.135 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:33:48.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb97c06b4d0 msgr2=0x7fb97c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb97c06b4d0 0x7fb97c06d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb98c036000 tx=0x7fb98c03b040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 msgr2=0x7fb99c1f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c1f3990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fb99400c730 tx=0x7fb99400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 shutdown_connections 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb97c06b4d0 0x7fb97c06d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb99c15e0c0 0x7fb99c1f3990 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99c0cd600 0x7fb99c1520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 --2- 172.21.15.32:0/3674785256 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb99c06afb0 0x7fb99c15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 >> 172.21.15.32:0/3674785256 conn(0x7fb99c0d0c00 msgr2=0x7fb99c06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 shutdown_connections 2024-03-06T07:33:48.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.136+0000 7fb99be00640 1 -- 172.21.15.32:0/3674785256 wait complete. 2024-03-06T07:33:48.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:48.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.376+0000 7fb3df800640 1 Processor -- start 2024-03-06T07:33:48.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.376+0000 7fb3df800640 1 -- start start 2024-03-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.379+0000 7fb3df800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 0x7fb3d8070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.379+0000 7fb3df800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d8071c60 con 0x7fb3d80cd1e0 2024-03-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.379+0000 7fb3df800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d8071f40 con 0x7fb3d815afa0 2024-03-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.379+0000 7fb3df800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80cfa10 con 0x7fb3d80cf3a0 2024-03-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.379+0000 7fb3d7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb3d80cd1e0 0x7fb3d80b3ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50966/0 (socket says 172.21.15.32:50966) 2024-03-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.379+0000 7fb3d7e00640 1 -- 172.21.15.32:0/2304805896 learned_addr learned my addr 172.21.15.32:0/2304805896 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3dd800640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 0x7fb3d8070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543332495 0 0) 0x7fb3d8071c60 con 0x7fb3d80cd1e0 2024-03-06T07:33:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c0005730 con 0x7fb3d80cd1e0 2024-03-06T07:33:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727050555 0 0) 0x7fb3c0005730 con 0x7fb3d80cd1e0 2024-03-06T07:33:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 msgr2=0x7fb3d8070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 0x7fb3d8070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 >> v1:172.21.15.39:6789/0 conn(0x7fb3d815afa0 legacy=0x7fb3d80ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8072220 con 0x7fb3d80cd1e0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb3c4001670 con 0x7fb3d80cd1e0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 >> v1:172.21.15.32:6789/0 conn(0x7fb3d80cd1e0 legacy=0x7fb3d80b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3c0005bf0 0x7fb3c0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3c0006a10 0x7fb3c0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 0x7fb3c00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80cfa10 con 0x7fb3c0005bf0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000dc80 con 0x7fb3c0006a10 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.380+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000de80 con 0x7fb3d80cf3a0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3c0005bf0 0x7fb3c0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3d7e00640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3c0006a10 0x7fb3c0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dd800640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 0x7fb3c00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dce00640 1 -- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 msgr2=0x7fb3c00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3d80cf3a0 0x7fb3c00073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dce00640 1 -- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3c0006a10 msgr2=0x7fb3c0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3c0006a10 0x7fb3c0006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.381+0000 7fb3dce00640 1 -- 172.21.15.32:0/2304805896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c0005730 con 0x7fb3c0005bf0 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.382+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3c0005bf0 0x7fb3c0006010 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc001720 tx=0x7fb3c0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac47beb4836b847 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.383+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc001370 con 0x7fb3c0005bf0 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.383+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc0052d0 con 0x7fb3c0005bf0 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.383+0000 7fb3d7400640 1 -- 172.21.15.32:0/2304805896 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc003830 con 0x7fb3c0005bf0 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.383+0000 7fb3df800640 1 -- 172.21.15.32:0/2304805896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3c0005bf0 msgr2=0x7fb3c0006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.383+0000 7fb3df800640 1 --2- 172.21.15.32:0/2304805896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3c0005bf0 0x7fb3c0006010 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc001720 tx=0x7fb3c0007a40 comp rx=0 tx=0).stop 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.383+0000 7fb3dd800640 1 -- 172.21.15.32:0/2304805896 reap_dead start 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.384+0000 7fb3df800640 1 -- 172.21.15.32:0/2304805896 shutdown_connections 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.384+0000 7fb3df800640 1 -- 172.21.15.32:0/2304805896 >> 172.21.15.32:0/2304805896 conn(0x7fb3d806b3d0 msgr2=0x7fb3d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.384+0000 7fb3df800640 1 -- 172.21.15.32:0/2304805896 shutdown_connections 2024-03-06T07:33:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.384+0000 7fb3df800640 1 -- 172.21.15.32:0/2304805896 wait complete. 2024-03-06T07:33:48.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.385+0000 7fb3df800640 1 Processor -- start 2024-03-06T07:33:48.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 -- start start 2024-03-06T07:33:48.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3c0005bf0 0x7fb3d80b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3d814db30 0x7fb3d80b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 0x7fb3d814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8070e60 con 0x7fb3d80b06e0 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8070ce0 con 0x7fb3d814db30 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.386+0000 7fb3df800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8070fe0 con 0x7fb3c0005bf0 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 0x7fb3d814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 0x7fb3d814f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48302/0 (socket says 172.21.15.32:48302) 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 -- 172.21.15.32:0/2336546018 learned_addr learned my addr 172.21.15.32:0/2336546018 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3c0005bf0 0x7fb3d80b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3d7e00640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3d814db30 0x7fb3d80b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 -- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3c0005bf0 msgr2=0x7fb3d80b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3c0005bf0 0x7fb3d80b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 -- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3d814db30 msgr2=0x7fb3d80b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3d814db30 0x7fb3d80b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.387+0000 7fb3dd800640 1 -- 172.21.15.32:0/2336546018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d80b1380 con 0x7fb3d80b06e0 2024-03-06T07:33:48.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.388+0000 7fb3dd800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 0x7fb3d814f750 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb3d815e900 tx=0x7fb3d00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.388+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0003990 con 0x7fb3d80b06e0 2024-03-06T07:33:48.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.388+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d001b070 con 0x7fb3d80b06e0 2024-03-06T07:33:48.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.388+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0003b70 con 0x7fb3d80b06e0 2024-03-06T07:33:48.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.388+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d80b1660 con 0x7fb3d80b06e0 2024-03-06T07:33:48.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.388+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d80b1c20 con 0x7fb3d80b06e0 2024-03-06T07:33:48.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.390+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d81588c0 con 0x7fb3d80b06e0 2024-03-06T07:33:48.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.392+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3d0005d10 con 0x7fb3d80b06e0 2024-03-06T07:33:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.393+0000 7fb3d4c00640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3b806b4d0 0x7fb3b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.393+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb3d008a7a0 con 0x7fb3d80b06e0 2024-03-06T07:33:48.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.393+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3b806b4d0 0x7fb3b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.394+0000 7fb3dce00640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3b806b4d0 0x7fb3b806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc033000 tx=0x7fb3cc009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.397+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3d81588c0 con 0x7fb3d80b06e0 2024-03-06T07:33:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.592+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3d8072740 con 0x7fb3d80b06e0 2024-03-06T07:33:48.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.593+0000 7fb3d4c00640 1 -- 172.21.15.32:0/2336546018 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3d8072740 con 0x7fb3d80b06e0 2024-03-06T07:33:48.594 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:33:48.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3b806b4d0 msgr2=0x7fb3b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3b806b4d0 0x7fb3b806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc033000 tx=0x7fb3cc009030 comp rx=0 tx=0).stop 2024-03-06T07:33:48.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 msgr2=0x7fb3d814f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 0x7fb3d814f750 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb3d815e900 tx=0x7fb3d00115a0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 shutdown_connections 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb3b806b4d0 0x7fb3b806d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3d80b06e0 0x7fb3d814f750 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3d814db30 0x7fb3d80b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 --2- 172.21.15.32:0/2336546018 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb3c0005bf0 0x7fb3d80b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 >> 172.21.15.32:0/2336546018 conn(0x7fb3d806b3d0 msgr2=0x7fb3d8159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 shutdown_connections 2024-03-06T07:33:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.595+0000 7fb3df800640 1 -- 172.21.15.32:0/2336546018 wait complete. 2024-03-06T07:33:48.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:33:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.816+0000 7f31eb000640 1 Processor -- start 2024-03-06T07:33:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.816+0000 7f31eb000640 1 -- start start 2024-03-06T07:33:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.817+0000 7f31eb000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31e415dec0 0x7f31e40ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.817+0000 7f31eb000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415e2a0 0x7f31e40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.817+0000 7f31eb000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e40cf720 con 0x7f31e40cf290 2024-03-06T07:33:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.817+0000 7f31eb000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e40b0ea0 con 0x7f31e415dec0 2024-03-06T07:33:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.817+0000 7f31eb000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40d1120 con 0x7f31e415e2a0 2024-03-06T07:33:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31e415dec0 0x7f31e40ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31e415dec0 0x7f31e40ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48316/0 (socket says 172.21.15.32:48316) 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 -- 172.21.15.32:0/2408438938 learned_addr learned my addr 172.21.15.32:0/2408438938 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9200640 1 --2- 172.21.15.32:0/2408438938 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415e2a0 0x7f31e40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 -- 172.21.15.32:0/2408438938 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415e2a0 msgr2=0x7f31e40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 --2- 172.21.15.32:0/2408438938 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415e2a0 0x7f31e40ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 -- 172.21.15.32:0/2408438938 >> v1:172.21.15.39:6789/0 conn(0x7f31e40cf290 legacy=0x7f31e40d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.818+0000 7f31e9c00640 1 -- 172.21.15.32:0/2408438938 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e40d1400 con 0x7f31e415dec0 2024-03-06T07:33:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.820+0000 7f31e9c00640 1 --2- 172.21.15.32:0/2408438938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415dec0 0x7f31e40ce810 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009870 tx=0x7f31d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b768d06d26f7117a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.820+0000 7f31e3e00640 1 -- 172.21.15.32:0/2408438938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8035020 con 0x7f31e415dec0 2024-03-06T07:33:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31e3e00640 1 -- 172.21.15.32:0/2408438938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d8031640 con 0x7f31e415dec0 2024-03-06T07:33:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31e3e00640 1 -- 172.21.15.32:0/2408438938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d802d040 con 0x7f31e415dec0 2024-03-06T07:33:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 -- 172.21.15.32:0/2408438938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415dec0 msgr2=0x7f31e40ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 --2- 172.21.15.32:0/2408438938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415dec0 0x7f31e40ce810 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009870 tx=0x7f31d802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 -- 172.21.15.32:0/2408438938 shutdown_connections 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 --2- 172.21.15.32:0/2408438938 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415e2a0 0x7f31e40ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 --2- 172.21.15.32:0/2408438938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415dec0 0x7f31e40ce810 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 -- 172.21.15.32:0/2408438938 >> 172.21.15.32:0/2408438938 conn(0x7f31e40d3d40 msgr2=0x7f31e40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 -- 172.21.15.32:0/2408438938 shutdown_connections 2024-03-06T07:33:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.821+0000 7f31eb000640 1 -- 172.21.15.32:0/2408438938 wait complete. 2024-03-06T07:33:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.823+0000 7f31eb000640 1 Processor -- start 2024-03-06T07:33:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.823+0000 7f31eb000640 1 -- start start 2024-03-06T07:33:48.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31eb000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31e40cf290 0x7f31e40b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31eb000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415dec0 0x7f31e414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31eb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 0x7f31e40b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31eb000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40d0ec0 con 0x7f31e415e2a0 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31eb000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40d0d40 con 0x7f31e40cf290 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31eb000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40b2340 con 0x7f31e415dec0 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31ea600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 0x7f31e40b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31ea600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 0x7f31e40b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48322/0 (socket says 172.21.15.32:48322) 2024-03-06T07:33:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.824+0000 7f31ea600640 1 -- 172.21.15.32:0/3103793336 learned_addr learned my addr 172.21.15.32:0/3103793336 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31e9c00640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31e40cf290 0x7f31e40b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31e9200640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415dec0 0x7f31e414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31ea600640 1 -- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415dec0 msgr2=0x7f31e414dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31ea600640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415dec0 0x7f31e414dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31ea600640 1 -- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31e40cf290 msgr2=0x7f31e40b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31ea600640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31e40cf290 0x7f31e40b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31ea600640 1 -- 172.21.15.32:0/3103793336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e41f7740 con 0x7f31e415e2a0 2024-03-06T07:33:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.825+0000 7f31ea600640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 0x7f31e40b3340 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f31dc00c780 tx=0x7f31dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.827+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc004b60 con 0x7f31e415e2a0 2024-03-06T07:33:48.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.828+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e41f7a20 con 0x7f31e415e2a0 2024-03-06T07:33:48.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.828+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc004d00 con 0x7f31e415e2a0 2024-03-06T07:33:48.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.828+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc00aa90 con 0x7f31e415e2a0 2024-03-06T07:33:48.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.828+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e41f7fe0 con 0x7f31e415e2a0 2024-03-06T07:33:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.829+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e404f330 con 0x7f31e415e2a0 2024-03-06T07:33:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.831+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31dc00ac30 con 0x7f31e415e2a0 2024-03-06T07:33:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.832+0000 7f31e1600640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f31c406b500 0x7f31c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.832+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31dc08a580 con 0x7f31e415e2a0 2024-03-06T07:33:48.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.833+0000 7f31e9c00640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f31c406b500 0x7f31c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.833+0000 7f31e9c00640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f31c406b500 0x7f31c406d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f31d8036000 tx=0x7f31d803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:48.838+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31dc020070 con 0x7f31e415e2a0 2024-03-06T07:33:49.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.052+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f31e40cdf50 con 0x7f31e415e2a0 2024-03-06T07:33:49.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.053+0000 7f31e1600640 1 -- 172.21.15.32:0/3103793336 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v84) v1 ==== 80+0+0 (secure 0 0 0) 0x7f31dc059320 con 0x7f31e415e2a0 2024-03-06T07:33:49.054 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:33:49.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f31c406b500 msgr2=0x7f31c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f31c406b500 0x7f31c406d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f31d8036000 tx=0x7f31d803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 msgr2=0x7f31e40b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 0x7f31e40b3340 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f31dc00c780 tx=0x7f31dc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 shutdown_connections 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f31c406b500 0x7f31c406d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31e415e2a0 0x7f31e40b3340 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31e415dec0 0x7f31e414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 --2- 172.21.15.32:0/3103793336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31e40cf290 0x7f31e40b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.056+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 >> 172.21.15.32:0/3103793336 conn(0x7f31e40d3d40 msgr2=0x7f31e406ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.057+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 shutdown_connections 2024-03-06T07:33:49.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.057+0000 7f31eb000640 1 -- 172.21.15.32:0/3103793336 wait complete. 2024-03-06T07:33:49.083 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:33:49.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:49.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:49.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:33:49.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:49.178 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:33:49.227 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:33:49.227 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:49.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:49.287 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:33:49.287 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:49.343 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:49.343 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:33:49.343 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:49.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:49.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.615+0000 7f8e57200640 1 Processor -- start 2024-03-06T07:33:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.615+0000 7f8e57200640 1 -- start start 2024-03-06T07:33:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.617+0000 7f8e57200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8e58145300 0x7f8e580b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.617+0000 7f8e57200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580714d0 con 0x7f8e580cd060 2024-03-06T07:33:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.617+0000 7f8e57200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580717b0 con 0x7f8e58070f10 2024-03-06T07:33:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.617+0000 7f8e57200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e58071a90 con 0x7f8e58145300 2024-03-06T07:33:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.617+0000 7f8e56800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8e580cd060 0x7f8e5815c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50972/0 (socket says 172.21.15.32:50972) 2024-03-06T07:33:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.617+0000 7f8e56800640 1 -- 172.21.15.32:0/577757316 learned_addr learned my addr 172.21.15.32:0/577757316 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55e00640 1 --2- 172.21.15.32:0/577757316 >> v2:172.21.15.39:3300/0 conn(0x7f8e58145300 0x7f8e580b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988023008 0 0) 0x7f8e580714d0 con 0x7f8e580cd060 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38003520 con 0x7f8e580cd060 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165781394 0 0) 0x7f8e38003520 con 0x7f8e580cd060 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 >> v2:172.21.15.39:3300/0 conn(0x7f8e58145300 msgr2=0x7f8e580b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 --2- 172.21.15.32:0/577757316 >> v2:172.21.15.39:3300/0 conn(0x7f8e58145300 0x7f8e580b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 >> v1:172.21.15.39:6789/0 conn(0x7f8e58070f10 legacy=0x7f8e580cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58071d70 con 0x7f8e580cd060 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f8e500019f0 con 0x7f8e580cd060 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 >> v1:172.21.15.32:6789/0 conn(0x7f8e580cd060 legacy=0x7f8e5815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e38004800 0x7f8e38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e380052e0 0x7f8e3800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58071a90 con 0x7f8e380039b0 2024-03-06T07:33:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800c1b0 con 0x7f8e380052e0 2024-03-06T07:33:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.618+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800c3b0 con 0x7f8e38004800 2024-03-06T07:33:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e55e00640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e56800640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e380052e0 0x7f8e3800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e55e00640 1 -- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e38004800 msgr2=0x7f8e38004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e4d400640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e38004800 0x7f8e38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e55e00640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e38004800 0x7f8e38004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e55e00640 1 -- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e380052e0 msgr2=0x7f8e3800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e55e00640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e380052e0 0x7f8e3800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e55e00640 1 -- 172.21.15.32:0/577757316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38003520 con 0x7f8e380039b0 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e56800640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e380052e0 0x7f8e3800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e56800640 1 -- 172.21.15.32:0/577757316 reap_dead start 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.619+0000 7f8e4d400640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e38004800 0x7f8e38004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.620+0000 7f8e55e00640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e38003dd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800ba00 tx=0x7f8e38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec17f3bc80cd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48036020 con 0x7f8e380039b0 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e48032620 con 0x7f8e380039b0 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e55400640 1 -- 172.21.15.32:0/577757316 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800b600 con 0x7f8e380039b0 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 -- 172.21.15.32:0/577757316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 msgr2=0x7f8e38003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e38003dd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800ba00 tx=0x7f8e38005830 comp rx=0 tx=0).stop 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 -- 172.21.15.32:0/577757316 shutdown_connections 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 --2- 172.21.15.32:0/577757316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e38003dd0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 -- 172.21.15.32:0/577757316 >> 172.21.15.32:0/577757316 conn(0x7f8e5806b3d0 msgr2=0x7f8e5806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 -- 172.21.15.32:0/577757316 shutdown_connections 2024-03-06T07:33:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.621+0000 7f8e57200640 1 -- 172.21.15.32:0/577757316 wait complete. 2024-03-06T07:33:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.623+0000 7f8e57200640 1 Processor -- start 2024-03-06T07:33:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.623+0000 7f8e57200640 1 -- start start 2024-03-06T07:33:49.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.624+0000 7f8e57200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e5813c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.624+0000 7f8e57200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e5813ca40 0x7f8e581f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.624+0000 7f8e57200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e581f3ff0 0x7f8e581f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.624+0000 7f8e57200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580ccad0 con 0x7f8e380039b0 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.624+0000 7f8e57200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580cc950 con 0x7f8e5813ca40 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.624+0000 7f8e57200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580ccc50 con 0x7f8e581f3ff0 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e5813c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e5813c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48342/0 (socket says 172.21.15.32:48342) 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 -- 172.21.15.32:0/2327546806 learned_addr learned my addr 172.21.15.32:0/2327546806 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e56800640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e581f3ff0 0x7f8e581f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 -- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e581f3ff0 msgr2=0x7f8e581f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e4d400640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e5813ca40 0x7f8e581f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e581f3ff0 0x7f8e581f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 -- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e5813ca40 msgr2=0x7f8e581f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e5813ca40 0x7f8e581f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.625+0000 7f8e55e00640 1 -- 172.21.15.32:0/2327546806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5813cec0 con 0x7f8e380039b0 2024-03-06T07:33:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.626+0000 7f8e55e00640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e5813c1c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f8e48035040 tx=0x7f8e48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.626+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48041070 con 0x7f8e380039b0 2024-03-06T07:33:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.626+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5813d170 con 0x7f8e380039b0 2024-03-06T07:33:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.626+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4803b620 con 0x7f8e380039b0 2024-03-06T07:33:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.626+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48036040 con 0x7f8e380039b0 2024-03-06T07:33:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.626+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5813d730 con 0x7f8e380039b0 2024-03-06T07:33:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.627+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5804f330 con 0x7f8e380039b0 2024-03-06T07:33:49.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.628+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e4804c060 con 0x7f8e380039b0 2024-03-06T07:33:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.629+0000 7f8e4ea00640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8e3406b4d0 0x7f8e3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.630+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8e480ab920 con 0x7f8e380039b0 2024-03-06T07:33:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.630+0000 7f8e4d400640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8e3406b4d0 0x7f8e3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.630+0000 7f8e4d400640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8e3406b4d0 0x7f8e3406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c00 tx=0x7f8e40009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:49.633+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e4807a6c0 con 0x7f8e380039b0 2024-03-06T07:33:49.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:49.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.017+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e580b0200 con 0x7f8e380039b0 2024-03-06T07:33:50.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.019+0000 7f8e4ea00640 1 -- 172.21.15.32:0/2327546806 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f8e48079fc0 con 0x7f8e380039b0 2024-03-06T07:33:50.020 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:50.020 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:50.020 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8e3406b4d0 msgr2=0x7f8e3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8e3406b4d0 0x7f8e3406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c00 tx=0x7f8e40009040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 msgr2=0x7f8e5813c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e5813c1c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f8e48035040 tx=0x7f8e48004730 comp rx=0 tx=0).stop 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 shutdown_connections 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8e3406b4d0 0x7f8e3406d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8e581f3ff0 0x7f8e581f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e5813ca40 0x7f8e581f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 --2- 172.21.15.32:0/2327546806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e380039b0 0x7f8e5813c1c0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 >> 172.21.15.32:0/2327546806 conn(0x7f8e5806b3d0 msgr2=0x7f8e580aeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 shutdown_connections 2024-03-06T07:33:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.021+0000 7f8e57200640 1 -- 172.21.15.32:0/2327546806 wait complete. 2024-03-06T07:33:50.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:50.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.249+0000 7f0ecb200640 1 Processor -- start 2024-03-06T07:33:50.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.249+0000 7f0ecb200640 1 -- start start 2024-03-06T07:33:50.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.251+0000 7f0ecb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0ec4168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.251+0000 7f0ecb200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec40b3710 con 0x7f0ec415e0b0 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.251+0000 7f0ecb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec40b39f0 con 0x7f0ec40d4a10 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.251+0000 7f0ecb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40b09e0 con 0x7f0ec40d0e50 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.251+0000 7f0ec9e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0ec415e0b0 0x7f0ec40cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50986/0 (socket says 172.21.15.32:50986) 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.251+0000 7f0ec9e00640 1 -- 172.21.15.32:0/3361975327 learned_addr learned my addr 172.21.15.32:0/3361975327 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0eca800640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0ec4168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955990006 0 0) 0x7f0ec40b3710 con 0x7f0ec415e0b0 2024-03-06T07:33:50.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac003520 con 0x7f0ec415e0b0 2024-03-06T07:33:50.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 273493319 0 0) 0x7f0eac003520 con 0x7f0ec415e0b0 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 msgr2=0x7f0ec4168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0ec4168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 >> v1:172.21.15.39:6789/0 conn(0x7f0ec40d4a10 legacy=0x7f0ec40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec40b3cd0 con 0x7f0ec415e0b0 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f0eb8001690 con 0x7f0ec415e0b0 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 >> v1:172.21.15.32:6789/0 conn(0x7f0ec415e0b0 legacy=0x7f0ec40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0eac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.252+0000 7f0ec8a00640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eac003f20 0x7f0eac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0ec8a00640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0eac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac00b9f0 con 0x7f0eac004d40 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac00bba0 con 0x7f0eac003f20 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac00bda0 con 0x7f0ec40d0e50 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0eca800640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0eac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0ec9e00640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0eac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0eca800640 1 -- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 msgr2=0x7f0eac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0eca800640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0eac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0eca800640 1 -- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eac003f20 msgr2=0x7f0eac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0eca800640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eac003f20 0x7f0eac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0eca800640 1 -- 172.21.15.32:0/3361975327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac003520 con 0x7f0eac004d40 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.253+0000 7f0ec9e00640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40d0e50 0x7f0eac0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.255+0000 7f0eca800640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0eac0051c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009f90 tx=0x7f0eac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3b6468e38e7797 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.255+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0039070 con 0x7f0eac004d40 2024-03-06T07:33:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.255+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec002cbd0 con 0x7f0eac004d40 2024-03-06T07:33:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.255+0000 7f0ec8a00640 1 -- 172.21.15.32:0/3361975327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0009c10 con 0x7f0eac004d40 2024-03-06T07:33:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0ecb200640 1 -- 172.21.15.32:0/3361975327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 msgr2=0x7f0eac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0ecb200640 1 --2- 172.21.15.32:0/3361975327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0eac0051c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009f90 tx=0x7f0eac005830 comp rx=0 tx=0).stop 2024-03-06T07:33:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0eca800640 1 -- 172.21.15.32:0/3361975327 reap_dead start 2024-03-06T07:33:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0ecb200640 1 -- 172.21.15.32:0/3361975327 shutdown_connections 2024-03-06T07:33:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0ecb200640 1 -- 172.21.15.32:0/3361975327 >> 172.21.15.32:0/3361975327 conn(0x7f0ec40d3d30 msgr2=0x7f0ec40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0ecb200640 1 -- 172.21.15.32:0/3361975327 shutdown_connections 2024-03-06T07:33:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.256+0000 7f0ecb200640 1 -- 172.21.15.32:0/3361975327 wait complete. 2024-03-06T07:33:50.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.258+0000 7f0ecb200640 1 Processor -- start 2024-03-06T07:33:50.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.258+0000 7f0ecb200640 1 -- start start 2024-03-06T07:33:50.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ecb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0ec41516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ecb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec40b6920 0x7f0ec40ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ecb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40cd510 0x7f0ec415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ecb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec415e490 con 0x7f0eac004d40 2024-03-06T07:33:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ecb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40cfa20 con 0x7f0ec40b6920 2024-03-06T07:33:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ecb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40cfc60 con 0x7f0ec40cd510 2024-03-06T07:33:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ec9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0ec41516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ec9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0ec41516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48370/0 (socket says 172.21.15.32:48370) 2024-03-06T07:33:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.259+0000 7f0ec9e00640 1 -- 172.21.15.32:0/4086170105 learned_addr learned my addr 172.21.15.32:0/4086170105 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0eca800640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40cd510 0x7f0ec415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9400640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec40b6920 0x7f0ec40ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9e00640 1 -- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40cd510 msgr2=0x7f0ec415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9e00640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40cd510 0x7f0ec415ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9e00640 1 -- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec40b6920 msgr2=0x7f0ec40ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9e00640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec40b6920 0x7f0ec40ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9e00640 1 -- 172.21.15.32:0/4086170105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec40c8a50 con 0x7f0eac004d40 2024-03-06T07:33:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ec9e00640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0ec41516d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800d8d0 tx=0x7f0eb800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.260+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8019070 con 0x7f0eac004d40 2024-03-06T07:33:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.261+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb80074e0 con 0x7f0eac004d40 2024-03-06T07:33:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.261+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec415fc50 con 0x7f0eac004d40 2024-03-06T07:33:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.261+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb80047e0 con 0x7f0eac004d40 2024-03-06T07:33:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.261+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec40c8fd0 con 0x7f0eac004d40 2024-03-06T07:33:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.262+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec40cd920 con 0x7f0eac004d40 2024-03-06T07:33:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.263+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0eb801c070 con 0x7f0eac004d40 2024-03-06T07:33:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.264+0000 7f0ebe000640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ea806b4d0 0x7f0ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.264+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0eb8089e90 con 0x7f0eac004d40 2024-03-06T07:33:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.265+0000 7f0ec9400640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ea806b4d0 0x7f0ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.265+0000 7f0ec9400640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ea806b4d0 0x7f0ea806d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40d2dc0 tx=0x7f0eb4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.269+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ec40cd920 con 0x7f0eac004d40 2024-03-06T07:33:50.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.536+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ec404f330 con 0x7f0eac004d40 2024-03-06T07:33:50.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.538+0000 7f0ebe000640 1 -- 172.21.15.32:0/4086170105 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16649 (secure 0 0 0) 0x7f0eb8058cd0 con 0x7f0eac004d40 2024-03-06T07:33:50.540 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:50.540 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:48.823439+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.f","mappings":[{"from":6,"to":3}]},{"pgid":"13.3c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ea806b4d0 msgr2=0x7f0ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ea806b4d0 0x7f0ea806d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40d2dc0 tx=0x7f0eb4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 msgr2=0x7f0ec41516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0ec41516d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800d8d0 tx=0x7f0eb800dd90 comp rx=0 tx=0).stop 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 shutdown_connections 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ea806b4d0 0x7f0ea806d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ec40cd510 0x7f0ec415ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec40b6920 0x7f0ec40ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 --2- 172.21.15.32:0/4086170105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0eac004d40 0x7f0ec41516d0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 >> 172.21.15.32:0/4086170105 conn(0x7f0ec40d3d30 msgr2=0x7f0ec415d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 shutdown_connections 2024-03-06T07:33:50.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.541+0000 7f0ecb200640 1 -- 172.21.15.32:0/4086170105 wait complete. 2024-03-06T07:33:50.562 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:32.774877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:33.889752+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:50.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:50.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.784+0000 7f2a8be00640 1 Processor -- start 2024-03-06T07:33:50.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.784+0000 7f2a8be00640 1 -- start start 2024-03-06T07:33:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.786+0000 7f2a8be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2a8415aae0 0x7f2a840cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.786+0000 7f2a8be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a840cdd00 0x7f2a840c6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.786+0000 7f2a8be00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a840d00b0 con 0x7f2a8415aec0 2024-03-06T07:33:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.786+0000 7f2a8be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a840d0390 con 0x7f2a840cdd00 2024-03-06T07:33:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.786+0000 7f2a8be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a840d0670 con 0x7f2a8415aae0 2024-03-06T07:33:50.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a88a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a840cdd00 0x7f2a840c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a89e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2a8415aec0 0x7f2a84071bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50992/0 (socket says 172.21.15.32:50992) 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a89e00640 1 -- 172.21.15.32:0/2676589716 learned_addr learned my addr 172.21.15.32:0/2676589716 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a88a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a840cdd00 0x7f2a840c6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48386/0 (socket says 172.21.15.32:48386) 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a89400640 1 --2- 172.21.15.32:0/2676589716 >> v2:172.21.15.39:3300/0 conn(0x7f2a8415aae0 0x7f2a840cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a88a00640 1 -- 172.21.15.32:0/2676589716 >> v2:172.21.15.39:3300/0 conn(0x7f2a8415aae0 msgr2=0x7f2a840cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a88a00640 1 --2- 172.21.15.32:0/2676589716 >> v2:172.21.15.39:3300/0 conn(0x7f2a8415aae0 0x7f2a840cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a88a00640 1 -- 172.21.15.32:0/2676589716 >> v1:172.21.15.32:6789/0 conn(0x7f2a8415aec0 legacy=0x7f2a84071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.787+0000 7f2a88a00640 1 -- 172.21.15.32:0/2676589716 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8406dd80 con 0x7f2a840cdd00 2024-03-06T07:33:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.788+0000 7f2a88a00640 1 --2- 172.21.15.32:0/2676589716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840c6350 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2a840ce170 tx=0x7f2a7002e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf2d047dec5cea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a83e00640 1 -- 172.21.15.32:0/2676589716 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7003b070 con 0x7f2a840cdd00 2024-03-06T07:33:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a83e00640 1 -- 172.21.15.32:0/2676589716 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7002edc0 con 0x7f2a840cdd00 2024-03-06T07:33:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a83e00640 1 -- 172.21.15.32:0/2676589716 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70036660 con 0x7f2a840cdd00 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 -- 172.21.15.32:0/2676589716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 msgr2=0x7f2a840c6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2676589716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840c6350 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2a840ce170 tx=0x7f2a7002e010 comp rx=0 tx=0).stop 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 -- 172.21.15.32:0/2676589716 shutdown_connections 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2676589716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840c6350 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2676589716 >> v2:172.21.15.39:3300/0 conn(0x7f2a8415aae0 0x7f2a840cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 -- 172.21.15.32:0/2676589716 >> 172.21.15.32:0/2676589716 conn(0x7f2a840b2760 msgr2=0x7f2a840b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 -- 172.21.15.32:0/2676589716 shutdown_connections 2024-03-06T07:33:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.789+0000 7f2a8be00640 1 -- 172.21.15.32:0/2676589716 wait complete. 2024-03-06T07:33:50.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.792+0000 7f2a8be00640 1 Processor -- start 2024-03-06T07:33:50.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.792+0000 7f2a8be00640 1 -- start start 2024-03-06T07:33:50.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a8be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8415aae0 0x7f2a840cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a8be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8415aec0 0x7f2a840cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a8be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8406f430 con 0x7f2a840cdd00 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a8be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8406f2b0 con 0x7f2a8415aae0 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a8be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8406f5b0 con 0x7f2a8415aec0 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a89400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a89400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48398/0 (socket says 172.21.15.32:48398) 2024-03-06T07:33:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.794+0000 7f2a89400640 1 -- 172.21.15.32:0/2678322653 learned_addr learned my addr 172.21.15.32:0/2678322653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a88a00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8415aae0 0x7f2a840cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89e00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8415aec0 0x7f2a840cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89400640 1 -- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8415aec0 msgr2=0x7f2a840cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89400640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8415aec0 0x7f2a840cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89400640 1 -- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8415aae0 msgr2=0x7f2a840cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89400640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8415aae0 0x7f2a840cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89400640 1 -- 172.21.15.32:0/2678322653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8415ce40 con 0x7f2a840cdd00 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a88a00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8415aae0 0x7f2a840cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.795+0000 7f2a89400640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840ca040 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800d800 tx=0x7f2a7800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.796+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7800b6c0 con 0x7f2a840cdd00 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.796+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78019070 con 0x7f2a840cdd00 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.796+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7800ba80 con 0x7f2a840cdd00 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.796+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8415d120 con 0x7f2a840cdd00 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.796+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8415d400 con 0x7f2a840cdd00 2024-03-06T07:33:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.797+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8404f330 con 0x7f2a840cdd00 2024-03-06T07:33:50.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.798+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a7800bc20 con 0x7f2a840cdd00 2024-03-06T07:33:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.799+0000 7f2a81600640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a6406b4d0 0x7f2a6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.799+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a7808a700 con 0x7f2a840cdd00 2024-03-06T07:33:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.799+0000 7f2a88a00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a6406b4d0 0x7f2a6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.800+0000 7f2a88a00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a6406b4d0 0x7f2a6406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2a70038000 tx=0x7f2a70008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:50.804+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a78059530 con 0x7f2a840cdd00 2024-03-06T07:33:51.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.014+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2a8414ee20 con 0x7f2a840cdd00 2024-03-06T07:33:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.015+0000 7f2a81600640 1 -- 172.21.15.32:0/2678322653 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2a78058e30 con 0x7f2a840cdd00 2024-03-06T07:33:51.016 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a6406b4d0 msgr2=0x7f2a6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a6406b4d0 0x7f2a6406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2a70038000 tx=0x7f2a70008040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 msgr2=0x7f2a840ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840ca040 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800d800 tx=0x7f2a7800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 shutdown_connections 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a6406b4d0 0x7f2a6406d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8415aec0 0x7f2a840cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8415aae0 0x7f2a840cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 --2- 172.21.15.32:0/2678322653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a840cdd00 0x7f2a840ca040 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 >> 172.21.15.32:0/2678322653 conn(0x7f2a840b2760 msgr2=0x7f2a8406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.018+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 shutdown_connections 2024-03-06T07:33:51.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.019+0000 7f2a8be00640 1 -- 172.21.15.32:0/2678322653 wait complete. 2024-03-06T07:33:51.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.278+0000 7fd712a00640 1 Processor -- start 2024-03-06T07:33:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.278+0000 7fd712a00640 1 -- start start 2024-03-06T07:33:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd712a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd70c0af930 0x7fd70c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd712a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c0b3770 con 0x7fd70c15bef0 2024-03-06T07:33:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd712a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c0b3a50 con 0x7fd70c06f540 2024-03-06T07:33:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd712a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70c0b3520 con 0x7fd70c0af930 2024-03-06T07:33:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd710c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd70c0af930 0x7fd70c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd710c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd70c0af930 0x7fd70c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48402/0 (socket says 172.21.15.32:48402) 2024-03-06T07:33:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.280+0000 7fd710c00640 1 -- 172.21.15.32:0/4283027803 learned_addr learned my addr 172.21.15.32:0/4283027803 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.281+0000 7fd710c00640 1 -- 172.21.15.32:0/4283027803 >> v1:172.21.15.39:6789/0 conn(0x7fd70c06f540 legacy=0x7fd70c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.281+0000 7fd710c00640 1 -- 172.21.15.32:0/4283027803 >> v1:172.21.15.32:6789/0 conn(0x7fd70c15bef0 legacy=0x7fd70c15c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.281+0000 7fd710c00640 1 -- 172.21.15.32:0/4283027803 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c0b3d30 con 0x7fd70c0af930 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.281+0000 7fd70be00640 1 -- 172.21.15.32:0/4283027803 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891646806 0 0) 0x7fd70c0b3770 con 0x7fd70c15bef0 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.282+0000 7fd710c00640 1 --2- 172.21.15.32:0/4283027803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c0af930 0x7fd70c0cfa90 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009870 tx=0x7fd6f802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9badef450557e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.282+0000 7fd70be00640 1 -- 172.21.15.32:0/4283027803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f802f840 con 0x7fd70c0af930 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.282+0000 7fd70be00640 1 -- 172.21.15.32:0/4283027803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f802f9e0 con 0x7fd70c0af930 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.283+0000 7fd70be00640 1 -- 172.21.15.32:0/4283027803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f802fdb0 con 0x7fd70c0af930 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.283+0000 7fd712a00640 1 -- 172.21.15.32:0/4283027803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c0af930 msgr2=0x7fd70c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.283+0000 7fd712a00640 1 --2- 172.21.15.32:0/4283027803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c0af930 0x7fd70c0cfa90 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009870 tx=0x7fd6f802ba20 comp rx=0 tx=0).stop 2024-03-06T07:33:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.283+0000 7fd712a00640 1 -- 172.21.15.32:0/4283027803 shutdown_connections 2024-03-06T07:33:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.283+0000 7fd712a00640 1 --2- 172.21.15.32:0/4283027803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c0af930 0x7fd70c0cfa90 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.283+0000 7fd712a00640 1 -- 172.21.15.32:0/4283027803 >> 172.21.15.32:0/4283027803 conn(0x7fd70c0723d0 msgr2=0x7fd70c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.284+0000 7fd712a00640 1 -- 172.21.15.32:0/4283027803 shutdown_connections 2024-03-06T07:33:51.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.284+0000 7fd712a00640 1 -- 172.21.15.32:0/4283027803 wait complete. 2024-03-06T07:33:51.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.285+0000 7fd712a00640 1 Processor -- start 2024-03-06T07:33:51.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.286+0000 7fd712a00640 1 -- start start 2024-03-06T07:33:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd70c06f540 0x7fd70c1599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd70c0af930 0x7fd70c13ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 0x7fd70c155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c15d2b0 con 0x7fd70c15bef0 2024-03-06T07:33:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c0cffd0 con 0x7fd70c06f540 2024-03-06T07:33:51.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c15d430 con 0x7fd70c0af930 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 0x7fd70c155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 0x7fd70c155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48412/0 (socket says 172.21.15.32:48412) 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd712000640 1 -- 172.21.15.32:0/4270993722 learned_addr learned my addr 172.21.15.32:0/4270993722 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd710c00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd70c0af930 0x7fd70c13ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.287+0000 7fd711600640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd70c06f540 0x7fd70c1599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd712000640 1 -- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd70c0af930 msgr2=0x7fd70c13ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd712000640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd70c0af930 0x7fd70c13ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd712000640 1 -- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd70c06f540 msgr2=0x7fd70c1599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd712000640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd70c06f540 0x7fd70c1599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd712000640 1 -- 172.21.15.32:0/4270993722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c153fb0 con 0x7fd70c15bef0 2024-03-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd711600640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd70c06f540 0x7fd70c1599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd712000640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 0x7fd70c155050 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c780 tx=0x7fd70400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd704003b70 con 0x7fd70c15bef0 2024-03-06T07:33:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd704003d10 con 0x7fd70c15bef0 2024-03-06T07:33:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.288+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd704005270 con 0x7fd70c15bef0 2024-03-06T07:33:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.289+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c154290 con 0x7fd70c15bef0 2024-03-06T07:33:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.289+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c154850 con 0x7fd70c15bef0 2024-03-06T07:33:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.290+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70c0735d0 con 0x7fd70c15bef0 2024-03-06T07:33:51.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.291+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd704005410 con 0x7fd70c15bef0 2024-03-06T07:33:51.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.292+0000 7fd709600640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6ec06b4d0 0x7fd6ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.292+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd70408a1e0 con 0x7fd70c15bef0 2024-03-06T07:33:51.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.293+0000 7fd711600640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6ec06b4d0 0x7fd6ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.293+0000 7fd711600640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6ec06b4d0 0x7fd6ec06d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd700007fd0 tx=0x7fd700009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.297+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd70c0735d0 con 0x7fd70c15bef0 2024-03-06T07:33:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.505+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd70c15a1d0 con 0x7fd70c15bef0 2024-03-06T07:33:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.506+0000 7fd709600640 1 -- 172.21.15.32:0/4270993722 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd704058f80 con 0x7fd70c15bef0 2024-03-06T07:33:51.507 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:33:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6ec06b4d0 msgr2=0x7fd6ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6ec06b4d0 0x7fd6ec06d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd700007fd0 tx=0x7fd700009040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 msgr2=0x7fd70c155050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 0x7fd70c155050 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c780 tx=0x7fd70400cb80 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 shutdown_connections 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6ec06b4d0 0x7fd6ec06d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd70c15bef0 0x7fd70c155050 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd70c0af930 0x7fd70c13ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 --2- 172.21.15.32:0/4270993722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd70c06f540 0x7fd70c1599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 >> 172.21.15.32:0/4270993722 conn(0x7fd70c0723d0 msgr2=0x7fd70c06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 shutdown_connections 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.509+0000 7fd712a00640 1 -- 172.21.15.32:0/4270993722 wait complete. 2024-03-06T07:33:51.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:51.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.741+0000 7eff04e00640 1 Processor -- start 2024-03-06T07:33:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.741+0000 7eff04e00640 1 -- start start 2024-03-06T07:33:51.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7eff04e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7eff0015dee0 0x7eff000d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7eff04e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff000d4860 0x7eff000d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7eff04e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff000ceca0 con 0x7eff000cfa60 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7eff04e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff000cf5d0 con 0x7eff000d4860 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7eff04e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff000cef80 con 0x7eff0015dee0 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7efefea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff000d4860 0x7eff000d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7efefea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff000d4860 0x7eff000d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48428/0 (socket says 172.21.15.32:48428) 2024-03-06T07:33:51.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.743+0000 7efefea00640 1 -- 172.21.15.32:0/2562866142 learned_addr learned my addr 172.21.15.32:0/2562866142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.744+0000 7efeff400640 1 --2- 172.21.15.32:0/2562866142 >> v2:172.21.15.39:3300/0 conn(0x7eff0015dee0 0x7eff000d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.744+0000 7efefea00640 1 -- 172.21.15.32:0/2562866142 >> v2:172.21.15.39:3300/0 conn(0x7eff0015dee0 msgr2=0x7eff000d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.744+0000 7efefea00640 1 --2- 172.21.15.32:0/2562866142 >> v2:172.21.15.39:3300/0 conn(0x7eff0015dee0 0x7eff000d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.744+0000 7efefea00640 1 -- 172.21.15.32:0/2562866142 >> v1:172.21.15.39:6789/0 conn(0x7eff000cfa60 legacy=0x7eff000b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.744+0000 7efefea00640 1 -- 172.21.15.32:0/2562866142 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff000cf260 con 0x7eff000d4860 2024-03-06T07:33:51.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.744+0000 7efeff400640 1 --2- 172.21.15.32:0/2562866142 >> v2:172.21.15.39:3300/0 conn(0x7eff0015dee0 0x7eff000d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:51.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.746+0000 7efefea00640 1 --2- 172.21.15.32:0/2562866142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000d0630 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7eff0008c6f0 tx=0x7efeec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9ef9759f9ce3b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.746+0000 7efefe000640 1 -- 172.21.15.32:0/2562866142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec035020 con 0x7eff000d4860 2024-03-06T07:33:51.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.746+0000 7efefe000640 1 -- 172.21.15.32:0/2562866142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec031870 con 0x7eff000d4860 2024-03-06T07:33:51.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.746+0000 7efefe000640 1 -- 172.21.15.32:0/2562866142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec031c40 con 0x7eff000d4860 2024-03-06T07:33:51.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 -- 172.21.15.32:0/2562866142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 msgr2=0x7eff000d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 --2- 172.21.15.32:0/2562866142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000d0630 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7eff0008c6f0 tx=0x7efeec02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 -- 172.21.15.32:0/2562866142 shutdown_connections 2024-03-06T07:33:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 --2- 172.21.15.32:0/2562866142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000d0630 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 --2- 172.21.15.32:0/2562866142 >> v2:172.21.15.39:3300/0 conn(0x7eff0015dee0 0x7eff000d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 -- 172.21.15.32:0/2562866142 >> 172.21.15.32:0/2562866142 conn(0x7eff000d3cd0 msgr2=0x7eff000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 -- 172.21.15.32:0/2562866142 shutdown_connections 2024-03-06T07:33:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.747+0000 7eff04e00640 1 -- 172.21.15.32:0/2562866142 wait complete. 2024-03-06T07:33:51.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.749+0000 7eff04e00640 1 Processor -- start 2024-03-06T07:33:51.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.749+0000 7eff04e00640 1 -- start start 2024-03-06T07:33:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7eff04e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff000cfa60 0x7eff001504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7eff04e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7eff04e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff0015dee0 0x7eff000cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7eff04e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff000d1de0 con 0x7eff000d4860 2024-03-06T07:33:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7eff04e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0015e2c0 con 0x7eff000cfa60 2024-03-06T07:33:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7eff04e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff000d1f60 con 0x7eff0015dee0 2024-03-06T07:33:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7efefea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7efefea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48432/0 (socket says 172.21.15.32:48432) 2024-03-06T07:33:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.750+0000 7efefea00640 1 -- 172.21.15.32:0/3654664584 learned_addr learned my addr 172.21.15.32:0/3654664584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:51.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efeff400640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff000cfa60 0x7eff001504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efeffe00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff0015dee0 0x7eff000cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efefea00640 1 -- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff0015dee0 msgr2=0x7eff000cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efefea00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff0015dee0 0x7eff000cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efefea00640 1 -- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff000cfa60 msgr2=0x7eff001504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efefea00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff000cfa60 0x7eff001504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efefea00640 1 -- 172.21.15.32:0/3654664584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff000c8c50 con 0x7eff000d4860 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efeff400640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff000cfa60 0x7eff001504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efefea00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000cd680 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efeec000c00 tx=0x7efeec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec040070 con 0x7eff000d4860 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec03a420 con 0x7eff000d4860 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec035040 con 0x7eff000d4860 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.751+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff000c8f30 con 0x7eff000d4860 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.752+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff000c94f0 con 0x7eff000d4860 2024-03-06T07:33:51.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.753+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0004f330 con 0x7eff000d4860 2024-03-06T07:33:51.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.754+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efeec04b060 con 0x7eff000d4860 2024-03-06T07:33:51.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.755+0000 7efef7400640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efee006b500 0x7efee006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.755+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efeec0ac100 con 0x7eff000d4860 2024-03-06T07:33:51.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.756+0000 7efeff400640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efee006b500 0x7efee006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.756+0000 7efeff400640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efee006b500 0x7efee006d9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7efef0006fd0 tx=0x7efef0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:51.759+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efeec07afd0 con 0x7eff000d4860 2024-03-06T07:33:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.040+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff0006d490 con 0x7eff000d4860 2024-03-06T07:33:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.040+0000 7efef7400640 1 -- 172.21.15.32:0/3654664584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7efeec07a8d0 con 0x7eff000d4860 2024-03-06T07:33:52.042 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efee006b500 msgr2=0x7efee006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efee006b500 0x7efee006d9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7efef0006fd0 tx=0x7efef0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 msgr2=0x7eff000cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000cd680 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efeec000c00 tx=0x7efeec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 shutdown_connections 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efee006b500 0x7efee006d9c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7eff0015dee0 0x7eff000cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff000d4860 0x7eff000cd680 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 --2- 172.21.15.32:0/3654664584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff000cfa60 0x7eff001504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.043+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 >> 172.21.15.32:0/3654664584 conn(0x7eff000d3cd0 msgr2=0x7eff000b8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.044+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 shutdown_connections 2024-03-06T07:33:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.044+0000 7eff04e00640 1 -- 172.21.15.32:0/3654664584 wait complete. 2024-03-06T07:33:52.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.282+0000 7f2ddbe00640 1 Processor -- start 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.282+0000 7f2ddbe00640 1 -- start start 2024-03-06T07:33:52.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2ddbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ddc1552b0 0x7f2ddc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2ddbe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc15ed10 con 0x7f2ddc15e810 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2ddbe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc15eff0 con 0x7f2ddc0cd060 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2ddbe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc15f2d0 con 0x7f2ddc1552b0 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2ddaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ddc1552b0 0x7f2ddc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2ddaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ddc1552b0 0x7f2ddc15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48434/0 (socket says 172.21.15.32:48434) 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.285+0000 7f2dd2000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2ddc15e810 0x7f2ddc0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51016/0 (socket says 172.21.15.32:51016) 2024-03-06T07:33:52.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.286+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3738872085 learned_addr learned my addr 172.21.15.32:0/3738872085 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.286+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3738872085 >> v1:172.21.15.39:6789/0 conn(0x7f2ddc0cd060 legacy=0x7f2ddc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.286+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3738872085 >> v1:172.21.15.32:6789/0 conn(0x7f2ddc15e810 legacy=0x7f2ddc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.286+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3738872085 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc15f5b0 con 0x7f2ddc1552b0 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.286+0000 7f2dda000640 1 -- 172.21.15.32:0/3738872085 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541040608 0 0) 0x7f2ddc15ed10 con 0x7f2ddc15e810 2024-03-06T07:33:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.287+0000 7f2ddaa00640 1 --2- 172.21.15.32:0/3738872085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc1552b0 0x7f2ddc15b1d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00d150 tx=0x7f2dcc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd57898ec6b563de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.287+0000 7f2dda000640 1 -- 172.21.15.32:0/3738872085 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc02ea80 con 0x7f2ddc1552b0 2024-03-06T07:33:52.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.287+0000 7f2dda000640 1 -- 172.21.15.32:0/3738872085 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc02ec20 con 0x7f2ddc1552b0 2024-03-06T07:33:52.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.287+0000 7f2dda000640 1 -- 172.21.15.32:0/3738872085 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc009600 con 0x7f2ddc1552b0 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.288+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3738872085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc1552b0 msgr2=0x7f2ddc15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.288+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3738872085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc1552b0 0x7f2ddc15b1d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00d150 tx=0x7f2dcc02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.288+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3738872085 shutdown_connections 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.288+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3738872085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc1552b0 0x7f2ddc15b1d0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.288+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3738872085 >> 172.21.15.32:0/3738872085 conn(0x7f2ddc06b3d0 msgr2=0x7f2ddc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.288+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3738872085 shutdown_connections 2024-03-06T07:33:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.289+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3738872085 wait complete. 2024-03-06T07:33:52.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.291+0000 7f2ddbe00640 1 Processor -- start 2024-03-06T07:33:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.291+0000 7f2ddbe00640 1 -- start start 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 0x7f2ddc141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ddc1552b0 0x7f2ddc158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ddc15e810 0x7f2ddc1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc06dd30 con 0x7f2ddc0cd060 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0b3760 con 0x7f2ddc15e810 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc06deb0 con 0x7f2ddc1552b0 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 0x7f2ddc141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 0x7f2ddc141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48438/0 (socket says 172.21.15.32:48438) 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.293+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3804092437 learned_addr learned my addr 172.21.15.32:0/3804092437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2dd2000640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ddc1552b0 0x7f2ddc158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ddc1552b0 msgr2=0x7f2ddc158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddaa00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ddc1552b0 0x7f2ddc158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ddc15e810 msgr2=0x7f2ddc1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddb400640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ddc15e810 0x7f2ddc1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddaa00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ddc15e810 0x7f2ddc1594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddaa00640 1 -- 172.21.15.32:0/3804092437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc159ad0 con 0x7f2ddc0cd060 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddb400640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ddc15e810 0x7f2ddc1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.294+0000 7f2ddaa00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 0x7f2ddc141280 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00d150 tx=0x7f2dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.295+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc045020 con 0x7f2ddc0cd060 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.295+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc040660 con 0x7f2ddc0cd060 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.295+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00b720 con 0x7f2ddc0cd060 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.295+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc159db0 con 0x7f2ddc0cd060 2024-03-06T07:33:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.295+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc15a370 con 0x7f2ddc0cd060 2024-03-06T07:33:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.296+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc04f330 con 0x7f2ddc0cd060 2024-03-06T07:33:52.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.297+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dcc048020 con 0x7f2ddc0cd060 2024-03-06T07:33:52.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.298+0000 7f2dd3400640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2db806b520 0x7f2db806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.299+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2dcc0aedc0 con 0x7f2ddc0cd060 2024-03-06T07:33:52.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.299+0000 7f2dd2000640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2db806b520 0x7f2db806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.299+0000 7f2dd2000640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2db806b520 0x7f2db806d9e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0b4860 tx=0x7f2dc4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.303+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dcc04f030 con 0x7f2ddc0cd060 2024-03-06T07:33:52.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.694+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ddc0745e0 con 0x7f2ddc0cd060 2024-03-06T07:33:52.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.695+0000 7f2dd3400640 1 -- 172.21.15.32:0/3804092437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f2dcc07dc90 con 0x7f2ddc0cd060 2024-03-06T07:33:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:52.697 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:52.697 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2db806b520 msgr2=0x7f2db806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2db806b520 0x7f2db806d9e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0b4860 tx=0x7f2dc4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 msgr2=0x7f2ddc141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 0x7f2ddc141280 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00d150 tx=0x7f2dcc004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 shutdown_connections 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2db806b520 0x7f2db806d9e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ddc15e810 0x7f2ddc1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ddc1552b0 0x7f2ddc158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 --2- 172.21.15.32:0/3804092437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ddc0cd060 0x7f2ddc141280 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 >> 172.21.15.32:0/3804092437 conn(0x7f2ddc06b3d0 msgr2=0x7f2ddc1545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 shutdown_connections 2024-03-06T07:33:52.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.698+0000 7f2ddbe00640 1 -- 172.21.15.32:0/3804092437 wait complete. 2024-03-06T07:33:52.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:52.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.915+0000 7f9328400640 1 Processor -- start 2024-03-06T07:33:52.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.915+0000 7f9328400640 1 -- start start 2024-03-06T07:33:52.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9328400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93200af580 0x7f932015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9328400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f93200cdc20 0x7f93200b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9328400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932015cc40 con 0x7f93200af1a0 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9328400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932015cf20 con 0x7f93200af580 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9328400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932015d200 con 0x7f93200cdc20 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9325a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f93200af1a0 0x7f93200b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51020/0 (socket says 172.21.15.32:51020) 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9325a00640 1 -- 172.21.15.32:0/3191868223 learned_addr learned my addr 172.21.15.32:0/3191868223 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9325000640 1 --2- 172.21.15.32:0/3191868223 >> v2:172.21.15.32:3300/0 conn(0x7f93200af580 0x7f932015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.917+0000 7f9326400640 1 --2- 172.21.15.32:0/3191868223 >> v2:172.21.15.39:3300/0 conn(0x7f93200cdc20 0x7f93200b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.918+0000 7f931fe00640 1 -- 172.21.15.32:0/3191868223 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118805402 0 0) 0x7f932015cc40 con 0x7f93200af1a0 2024-03-06T07:33:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.918+0000 7f931fe00640 1 -- 172.21.15.32:0/3191868223 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9308003520 con 0x7f93200af1a0 2024-03-06T07:33:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.918+0000 7f9325000640 1 -- 172.21.15.32:0/3191868223 >> v2:172.21.15.39:3300/0 conn(0x7f93200cdc20 msgr2=0x7f93200b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.918+0000 7f9325000640 1 --2- 172.21.15.32:0/3191868223 >> v2:172.21.15.39:3300/0 conn(0x7f93200cdc20 0x7f93200b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.918+0000 7f9325000640 1 -- 172.21.15.32:0/3191868223 >> v1:172.21.15.32:6789/0 conn(0x7f93200af1a0 legacy=0x7f93200b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.918+0000 7f9325000640 1 -- 172.21.15.32:0/3191868223 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932015d4e0 con 0x7f93200af580 2024-03-06T07:33:52.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.919+0000 7f9325000640 1 --2- 172.21.15.32:0/3191868223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af580 0x7f932015c600 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f932006c0c0 tx=0x7f930c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78eaab3109ac0cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.919+0000 7f931fe00640 1 -- 172.21.15.32:0/3191868223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c0315a0 con 0x7f93200af580 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f931fe00640 1 -- 172.21.15.32:0/3191868223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c037070 con 0x7f93200af580 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f931fe00640 1 -- 172.21.15.32:0/3191868223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c031960 con 0x7f93200af580 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 -- 172.21.15.32:0/3191868223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af580 msgr2=0x7f932015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 --2- 172.21.15.32:0/3191868223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af580 0x7f932015c600 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f932006c0c0 tx=0x7f930c02b840 comp rx=0 tx=0).stop 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 -- 172.21.15.32:0/3191868223 shutdown_connections 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 --2- 172.21.15.32:0/3191868223 >> v2:172.21.15.39:3300/0 conn(0x7f93200cdc20 0x7f93200b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 --2- 172.21.15.32:0/3191868223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af580 0x7f932015c600 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 -- 172.21.15.32:0/3191868223 >> 172.21.15.32:0/3191868223 conn(0x7f932006afb0 msgr2=0x7f932006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 -- 172.21.15.32:0/3191868223 shutdown_connections 2024-03-06T07:33:52.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.920+0000 7f9328400640 1 -- 172.21.15.32:0/3191868223 wait complete. 2024-03-06T07:33:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.922+0000 7f9328400640 1 Processor -- start 2024-03-06T07:33:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.922+0000 7f9328400640 1 -- start start 2024-03-06T07:33:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9328400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 0x7f93201f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9328400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93200af580 0x7f9320156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9328400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93200cdc20 0x7f93200ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9328400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93200b36a0 con 0x7f93200af1a0 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9328400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93200b3520 con 0x7f93200cdc20 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9328400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93200b3820 con 0x7f93200af580 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9325a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 0x7f93201f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9325a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 0x7f93201f85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48456/0 (socket says 172.21.15.32:48456) 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.923+0000 7f9325a00640 1 -- 172.21.15.32:0/1524548755 learned_addr learned my addr 172.21.15.32:0/1524548755 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9326400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93200cdc20 0x7f93200ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325000640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93200af580 0x7f9320156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325a00640 1 -- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93200af580 msgr2=0x7f9320156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325a00640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93200af580 0x7f9320156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325a00640 1 -- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93200cdc20 msgr2=0x7f93200ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325a00640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93200cdc20 0x7f93200ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325a00640 1 -- 172.21.15.32:0/1524548755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9320157510 con 0x7f93200af1a0 2024-03-06T07:33:52.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325000640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93200af580 0x7f9320156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:52.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.924+0000 7f9325a00640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 0x7f93201f85e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f931400c430 tx=0x7f931400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.925+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314017070 con 0x7f93200af1a0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.925+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9314005ce0 con 0x7f93200af1a0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.925+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932006cf80 con 0x7f93200af1a0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.925+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93140072f0 con 0x7f93200af1a0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.925+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932006d540 con 0x7f93200af1a0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.926+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932004f3a0 con 0x7f93200af1a0 2024-03-06T07:33:52.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.927+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f931401a070 con 0x7f93200af1a0 2024-03-06T07:33:52.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.928+0000 7f931d600640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f930006b4d0 0x7f930006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.928+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f93140896b0 con 0x7f93200af1a0 2024-03-06T07:33:52.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.928+0000 7f9325000640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f930006b4d0 0x7f930006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.929+0000 7f9325000640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f930006b4d0 0x7f930006d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f930c036000 tx=0x7f930c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:52.933+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9320157510 con 0x7f93200af1a0 2024-03-06T07:33:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.154+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9320071e90 con 0x7f93200af1a0 2024-03-06T07:33:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.156+0000 7f931d600640 1 -- 172.21.15.32:0/1524548755 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16649 (secure 0 0 0) 0x7f9314058560 con 0x7f93200af1a0 2024-03-06T07:33:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:53.158 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:48.823439+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.f","mappings":[{"from":6,"to":3}]},{"pgid":"13.3c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:53.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f930006b4d0 msgr2=0x7f930006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f930006b4d0 0x7f930006d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f930c036000 tx=0x7f930c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 msgr2=0x7f93201f85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 0x7f93201f85e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f931400c430 tx=0x7f931400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 shutdown_connections 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f930006b4d0 0x7f930006d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93200cdc20 0x7f93200ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93200af580 0x7f9320156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 --2- 172.21.15.32:0/1524548755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93200af1a0 0x7f93201f85e0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 >> 172.21.15.32:0/1524548755 conn(0x7f932006afb0 msgr2=0x7f93200ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 shutdown_connections 2024-03-06T07:33:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.159+0000 7f9328400640 1 -- 172.21.15.32:0/1524548755 wait complete. 2024-03-06T07:33:53.188 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:53.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.410+0000 7f3c01a00640 1 Processor -- start 2024-03-06T07:33:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.411+0000 7f3c01a00640 1 -- start start 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.412+0000 7f3c01a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3bfc0add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.412+0000 7f3c01a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3bfc06b6c0 0x7f3bfc071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.412+0000 7f3c01a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0ae290 con 0x7f3bfc0b5b80 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.412+0000 7f3c01a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc160610 con 0x7f3bfc06b6c0 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.412+0000 7f3c01a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc15fa00 con 0x7f3bfc15f370 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.413+0000 7f3bfa000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3bfc0b5b80 0x7f3bfc06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51022/0 (socket says 172.21.15.32:51022) 2024-03-06T07:33:53.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.413+0000 7f3bfa000640 1 -- 172.21.15.32:0/2110898781 learned_addr learned my addr 172.21.15.32:0/2110898781 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.413+0000 7f3bfb400640 1 --2- 172.21.15.32:0/2110898781 >> v2:172.21.15.39:3300/0 conn(0x7f3bfc06b6c0 0x7f3bfc071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.413+0000 7f3bfaa00640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3bfc0add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.413+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292476869 0 0) 0x7f3bfc0ae290 con 0x7f3bfc0b5b80 2024-03-06T07:33:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be0003520 con 0x7f3bfc0b5b80 2024-03-06T07:33:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1687317891 0 0) 0x7f3be0003520 con 0x7f3bfc0b5b80 2024-03-06T07:33:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 msgr2=0x7f3bfc0add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3bfc0add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 >> v2:172.21.15.39:3300/0 conn(0x7f3bfc06b6c0 msgr2=0x7f3bfc071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 --2- 172.21.15.32:0/2110898781 >> v2:172.21.15.39:3300/0 conn(0x7f3bfc06b6c0 0x7f3bfc071dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc15fce0 con 0x7f3bfc0b5b80 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3bec0019f0 con 0x7f3bfc0b5b80 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 >> v1:172.21.15.32:6789/0 conn(0x7f3bfc0b5b80 legacy=0x7f3bfc06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0003a20 0x7f3be0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be0004800 0x7f3be0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3be00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc160610 con 0x7f3be0004800 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc15fa00 con 0x7f3be0003a20 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.414+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be000ba70 con 0x7f3bfc15f370 2024-03-06T07:33:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.415+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be0004800 0x7f3be0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.415+0000 7f3bfb400640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3be00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.415+0000 7f3bfaa00640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0003a20 0x7f3be0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.416+0000 7f3bfa000640 1 -- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 msgr2=0x7f3be00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.416+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3be00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.416+0000 7f3bfa000640 1 -- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0003a20 msgr2=0x7f3be0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.416+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0003a20 0x7f3be0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.416+0000 7f3bfa000640 1 -- 172.21.15.32:0/2110898781 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be0003520 con 0x7f3be0004800 2024-03-06T07:33:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.416+0000 7f3bfb400640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc15f370 0x7f3be00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.417+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be0004800 0x7f3be0004c80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3bec002230 tx=0x7f3be0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0424a96df8c6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.417+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00a410 con 0x7f3be0004800 2024-03-06T07:33:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.417+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec034630 con 0x7f3be0004800 2024-03-06T07:33:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.417+0000 7f3bf9600640 1 -- 172.21.15.32:0/2110898781 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec034a20 con 0x7f3be0004800 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3c01a00640 1 -- 172.21.15.32:0/2110898781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be0004800 msgr2=0x7f3be0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2110898781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be0004800 0x7f3be0004c80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3bec002230 tx=0x7f3be0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3bfb400640 1 -- 172.21.15.32:0/2110898781 reap_dead start 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3c01a00640 1 -- 172.21.15.32:0/2110898781 shutdown_connections 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3c01a00640 1 -- 172.21.15.32:0/2110898781 >> 172.21.15.32:0/2110898781 conn(0x7f3bfc0b2a00 msgr2=0x7f3bfc0b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3c01a00640 1 -- 172.21.15.32:0/2110898781 shutdown_connections 2024-03-06T07:33:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.418+0000 7f3c01a00640 1 -- 172.21.15.32:0/2110898781 wait complete. 2024-03-06T07:33:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.420+0000 7f3c01a00640 1 Processor -- start 2024-03-06T07:33:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.420+0000 7f3c01a00640 1 -- start start 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3c01a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0004800 0x7f3bfc1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3c01a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 0x7f3bfc1f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3c01a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc154710 0x7f3bfc1f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3c01a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0b3d00 con 0x7f3bfc153340 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3c01a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc1602a0 con 0x7f3be0004800 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3c01a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0b3e80 con 0x7f3bfc154710 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3bfa000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 0x7f3bfc1f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.421+0000 7f3bfa000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 0x7f3bfc1f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48468/0 (socket says 172.21.15.32:48468) 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 -- 172.21.15.32:0/2206318777 learned_addr learned my addr 172.21.15.32:0/2206318777 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfaa00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0004800 0x7f3bfc1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 -- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc154710 msgr2=0x7f3bfc1f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc154710 0x7f3bfc1f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 -- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0004800 msgr2=0x7f3bfc1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0004800 0x7f3bfc1541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 -- 172.21.15.32:0/2206318777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc09ece0 con 0x7f3bfc153340 2024-03-06T07:33:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bfa000640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 0x7f3bfc1f5920 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00a880 tx=0x7f3bfc074fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.422+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec039040 con 0x7f3bfc153340 2024-03-06T07:33:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.423+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec043070 con 0x7f3bfc153340 2024-03-06T07:33:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.423+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0349e0 con 0x7f3bfc153340 2024-03-06T07:33:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.423+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc09e440 con 0x7f3bfc153340 2024-03-06T07:33:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.423+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc09ea00 con 0x7f3bfc153340 2024-03-06T07:33:53.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.424+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc04f3a0 con 0x7f3bfc153340 2024-03-06T07:33:53.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.426+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bec034b80 con 0x7f3bfc153340 2024-03-06T07:33:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.427+0000 7f3bf2a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bd806b4d0 0x7f3bd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.427+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3bec0ac3b0 con 0x7f3bfc153340 2024-03-06T07:33:53.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.427+0000 7f3bfaa00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bd806b4d0 0x7f3bd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.428+0000 7f3bfaa00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bd806b4d0 0x7f3bd806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc15cb10 tx=0x7f3be4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.432+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3bec07b150 con 0x7f3bfc153340 2024-03-06T07:33:53.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.668+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bfc0cd780 con 0x7f3bfc153340 2024-03-06T07:33:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.670+0000 7f3bf2a00640 1 -- 172.21.15.32:0/2206318777 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f3bec0019f0 con 0x7f3bfc153340 2024-03-06T07:33:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:53.671 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bd806b4d0 msgr2=0x7f3bd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bd806b4d0 0x7f3bd806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc15cb10 tx=0x7f3be4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 msgr2=0x7f3bfc1f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 0x7f3bfc1f5920 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00a880 tx=0x7f3bfc074fa0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 shutdown_connections 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bd806b4d0 0x7f3bd806d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3bfc154710 0x7f3bfc1f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bfc153340 0x7f3bfc1f5920 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 --2- 172.21.15.32:0/2206318777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3be0004800 0x7f3bfc1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 >> 172.21.15.32:0/2206318777 conn(0x7f3bfc0b2a00 msgr2=0x7f3bfc0add50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 shutdown_connections 2024-03-06T07:33:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.672+0000 7f3c01a00640 1 -- 172.21.15.32:0/2206318777 wait complete. 2024-03-06T07:33:53.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:53.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.904+0000 7f3d83200640 1 Processor -- start 2024-03-06T07:33:53.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.904+0000 7f3d83200640 1 -- start start 2024-03-06T07:33:53.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.905+0000 7f3d83200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d840d4a10 0x7f3d840d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.905+0000 7f3d83200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d840d0c80 0x7f3d840b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.905+0000 7f3d83200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d840b3710 con 0x7f3d8415e0b0 2024-03-06T07:33:53.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.905+0000 7f3d83200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d840b09e0 con 0x7f3d840d0c80 2024-03-06T07:33:53.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.905+0000 7f3d83200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d840cfa20 con 0x7f3d840d4a10 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d82800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d840d0c80 0x7f3d840b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d82800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d840d0c80 0x7f3d840b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48476/0 (socket says 172.21.15.32:48476) 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d82800640 1 -- 172.21.15.32:0/1403422612 learned_addr learned my addr 172.21.15.32:0/1403422612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d79400640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d840d4a10 0x7f3d840d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d82800640 1 -- 172.21.15.32:0/1403422612 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d840d4a10 msgr2=0x7f3d840d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d82800640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d840d4a10 0x7f3d840d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.906+0000 7f3d82800640 1 -- 172.21.15.32:0/1403422612 >> v1:172.21.15.39:6789/0 conn(0x7f3d8415e0b0 legacy=0x7f3d840cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.907+0000 7f3d82800640 1 -- 172.21.15.32:0/1403422612 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d840b39f0 con 0x7f3d840d0c80 2024-03-06T07:33:53.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.907+0000 7f3d79400640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d840d4a10 0x7f3d840d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:53.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.908+0000 7f3d82800640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d840b44d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009e50 tx=0x7f3d7c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3a4145c440ce83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.908+0000 7f3d81400640 1 -- 172.21.15.32:0/1403422612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c02cb00 con 0x7f3d840d0c80 2024-03-06T07:33:53.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.908+0000 7f3d81400640 1 -- 172.21.15.32:0/1403422612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c02cca0 con 0x7f3d840d0c80 2024-03-06T07:33:53.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.908+0000 7f3d81400640 1 -- 172.21.15.32:0/1403422612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c034610 con 0x7f3d840d0c80 2024-03-06T07:33:53.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 -- 172.21.15.32:0/1403422612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 msgr2=0x7f3d840b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d840b44d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009e50 tx=0x7f3d7c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 -- 172.21.15.32:0/1403422612 shutdown_connections 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d840b44d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 --2- 172.21.15.32:0/1403422612 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d840d4a10 0x7f3d840d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 -- 172.21.15.32:0/1403422612 >> 172.21.15.32:0/1403422612 conn(0x7f3d840d3d30 msgr2=0x7f3d840d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 -- 172.21.15.32:0/1403422612 shutdown_connections 2024-03-06T07:33:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.909+0000 7f3d83200640 1 -- 172.21.15.32:0/1403422612 wait complete. 2024-03-06T07:33:53.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.911+0000 7f3d83200640 1 Processor -- start 2024-03-06T07:33:53.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.911+0000 7f3d83200640 1 -- start start 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d83200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d84157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d83200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d840d4a10 0x7f3d841f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d83200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d8415e0b0 0x7f3d8406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d83200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8415e490 con 0x7f3d840d0c80 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d83200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d840b3fb0 con 0x7f3d840d4a10 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d83200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d840cfc60 con 0x7f3d8415e0b0 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d81e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d84157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d81e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d84157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48482/0 (socket says 172.21.15.32:48482) 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d81e00640 1 -- 172.21.15.32:0/724948811 learned_addr learned my addr 172.21.15.32:0/724948811 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d82800640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d8415e0b0 0x7f3d8406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.913+0000 7f3d79400640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d840d4a10 0x7f3d841f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d81e00640 1 -- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d8415e0b0 msgr2=0x7f3d8406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d81e00640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d8415e0b0 0x7f3d8406c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d81e00640 1 -- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d840d4a10 msgr2=0x7f3d841f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d81e00640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d840d4a10 0x7f3d841f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d81e00640 1 -- 172.21.15.32:0/724948811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d840b8de0 con 0x7f3d840d0c80 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d79400640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d840d4a10 0x7f3d841f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d81e00640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d84157b90 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400c430 tx=0x7f3d7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d740052d0 con 0x7f3d840d0c80 2024-03-06T07:33:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d74005490 con 0x7f3d840d0c80 2024-03-06T07:33:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74007260 con 0x7f3d840d0c80 2024-03-06T07:33:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8415b330 con 0x7f3d840d0c80 2024-03-06T07:33:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.914+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8415b8f0 con 0x7f3d840d0c80 2024-03-06T07:33:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.916+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8404f330 con 0x7f3d840d0c80 2024-03-06T07:33:53.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.916+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d74007400 con 0x7f3d840d0c80 2024-03-06T07:33:53.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.918+0000 7f3d7aa00640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d6006b4d0 0x7f3d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.918+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d74089860 con 0x7f3d840d0c80 2024-03-06T07:33:53.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.918+0000 7f3d79400640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d6006b4d0 0x7f3d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.919+0000 7f3d79400640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d6006b4d0 0x7f3d6006d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c006fd0 tx=0x7f3d6c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:53.922+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d74058570 con 0x7f3d840d0c80 2024-03-06T07:33:54.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.160+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d840b2a20 con 0x7f3d840d0c80 2024-03-06T07:33:54.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.161+0000 7f3d7aa00640 1 -- 172.21.15.32:0/724948811 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f3d74057e70 con 0x7f3d840d0c80 2024-03-06T07:33:54.163 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:54.163 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:34.876717+0000","modified":"2024-03-06T07:33:38.348392+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6837/3892834285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3892834285},{"type":"v1","addr":"172.21.15.32:6837","nonce":3892834285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:33:54.163 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 52 2024-03-06T07:33:54.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.163+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d6006b4d0 msgr2=0x7f3d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d6006b4d0 0x7f3d6006d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c006fd0 tx=0x7f3d6c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:54.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 msgr2=0x7f3d84157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d84157b90 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400c430 tx=0x7f3d7400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 shutdown_connections 2024-03-06T07:33:54.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d6006b4d0 0x7f3d6006d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d8415e0b0 0x7f3d8406c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d840d4a10 0x7f3d841f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 --2- 172.21.15.32:0/724948811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d840d0c80 0x7f3d84157b90 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 >> 172.21.15.32:0/724948811 conn(0x7f3d840d3d30 msgr2=0x7f3d840b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 shutdown_connections 2024-03-06T07:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.164+0000 7f3d83200640 1 -- 172.21.15.32:0/724948811 wait complete. 2024-03-06T07:33:54.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.384+0000 7f0ad6000640 1 Processor -- start 2024-03-06T07:33:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.384+0000 7f0ad6000640 1 -- start start 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.386+0000 7f0ad6000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ad015bdb0 0x7f0ad00cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ad00d21c0 0x7f0ad00b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00d17f0 con 0x7f0ad00d1310 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad6000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad00b1490 con 0x7f0ad00d21c0 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad6000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad00b1770 con 0x7f0ad015bdb0 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ad00d21c0 0x7f0ad00b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ad00d21c0 0x7f0ad00b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48490/0 (socket says 172.21.15.32:48490) 2024-03-06T07:33:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 -- 172.21.15.32:0/2298586852 learned_addr learned my addr 172.21.15.32:0/2298586852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 -- 172.21.15.32:0/2298586852 >> v2:172.21.15.39:3300/0 conn(0x7f0ad015bdb0 msgr2=0x7f0ad00cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 --2- 172.21.15.32:0/2298586852 >> v2:172.21.15.39:3300/0 conn(0x7f0ad015bdb0 0x7f0ad00cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 -- 172.21.15.32:0/2298586852 >> v1:172.21.15.39:6789/0 conn(0x7f0ad00d1310 legacy=0x7f0ad00d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.387+0000 7f0ad5600640 1 -- 172.21.15.32:0/2298586852 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0160450 con 0x7f0ad00d21c0 2024-03-06T07:33:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.389+0000 7f0ad5600640 1 --2- 172.21.15.32:0/2298586852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d21c0 0x7f0ad00b1c00 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0074c40 tx=0x7f0ac802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12ab7b3f25dad7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.389+0000 7f0acfe00640 1 -- 172.21.15.32:0/2298586852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac802c9f0 con 0x7f0ad00d21c0 2024-03-06T07:33:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.389+0000 7f0acfe00640 1 -- 172.21.15.32:0/2298586852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac802cb90 con 0x7f0ad00d21c0 2024-03-06T07:33:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.389+0000 7f0acfe00640 1 -- 172.21.15.32:0/2298586852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac80343f0 con 0x7f0ad00d21c0 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 -- 172.21.15.32:0/2298586852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d21c0 msgr2=0x7f0ad00b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 --2- 172.21.15.32:0/2298586852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d21c0 0x7f0ad00b1c00 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0074c40 tx=0x7f0ac802be50 comp rx=0 tx=0).stop 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 -- 172.21.15.32:0/2298586852 shutdown_connections 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 --2- 172.21.15.32:0/2298586852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d21c0 0x7f0ad00b1c00 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 --2- 172.21.15.32:0/2298586852 >> v2:172.21.15.39:3300/0 conn(0x7f0ad015bdb0 0x7f0ad00cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 -- 172.21.15.32:0/2298586852 >> 172.21.15.32:0/2298586852 conn(0x7f0ad0073120 msgr2=0x7f0ad009aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 -- 172.21.15.32:0/2298586852 shutdown_connections 2024-03-06T07:33:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.390+0000 7f0ad6000640 1 -- 172.21.15.32:0/2298586852 wait complete. 2024-03-06T07:33:54.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.391+0000 7f0ad6000640 1 Processor -- start 2024-03-06T07:33:54.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 -- start start 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 0x7f0ad01534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ad00d21c0 0x7f0ad006fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad015bdb0 0x7f0ad00700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad009b2e0 con 0x7f0ad00d1310 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad015c190 con 0x7f0ad015bdb0 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.392+0000 7f0ad6000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad009b460 con 0x7f0ad00d21c0 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 0x7f0ad01534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 0x7f0ad01534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48494/0 (socket says 172.21.15.32:48494) 2024-03-06T07:33:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 -- 172.21.15.32:0/1212570902 learned_addr learned my addr 172.21.15.32:0/1212570902 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:54.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ac7e00640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ad00d21c0 0x7f0ad006fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 -- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ad00d21c0 msgr2=0x7f0ad006fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad5600640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad015bdb0 0x7f0ad00700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ad00d21c0 0x7f0ad006fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 -- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad015bdb0 msgr2=0x7f0ad00700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad015bdb0 0x7f0ad00700f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.393+0000 7f0ad4c00640 1 -- 172.21.15.32:0/1212570902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad01f4fa0 con 0x7f0ad00d1310 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.394+0000 7f0ad4c00640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 0x7f0ad01534e0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000b440 tx=0x7f0ac000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.394+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0015020 con 0x7f0ad00d1310 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.394+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac0011890 con 0x7f0ad00d1310 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.394+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0011c80 con 0x7f0ad00d1310 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.394+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad01f5240 con 0x7f0ad00d1310 2024-03-06T07:33:54.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.394+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad01f5800 con 0x7f0ad00d1310 2024-03-06T07:33:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.395+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad004f330 con 0x7f0ad00d1310 2024-03-06T07:33:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.396+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ac0018020 con 0x7f0ad00d1310 2024-03-06T07:33:54.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.397+0000 7f0acd600640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0aac06b4d0 0x7f0aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.397+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0ac008a3d0 con 0x7f0ad00d1310 2024-03-06T07:33:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.398+0000 7f0ac7e00640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0aac06b4d0 0x7f0aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.398+0000 7f0ac7e00640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0aac06b4d0 0x7f0aac06d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0ad015f420 tx=0x7f0ab8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.402+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ac001f020 con 0x7f0ad00d1310 2024-03-06T07:33:54.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.593+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ad00b6d80 con 0x7f0ad00d1310 2024-03-06T07:33:54.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.595+0000 7f0acd600640 1 -- 172.21.15.32:0/1212570902 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16649 (secure 0 0 0) 0x7f0ac000d040 con 0x7f0ad00d1310 2024-03-06T07:33:54.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:33:54.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:48.823439+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:32.774877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:33.889752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.f","mappings":[{"from":6,"to":3}]},{"pgid":"13.3c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:54.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0aac06b4d0 msgr2=0x7f0aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0aac06b4d0 0x7f0aac06d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0ad015f420 tx=0x7f0ab8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:54.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 msgr2=0x7f0ad01534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 0x7f0ad01534e0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000b440 tx=0x7f0ac000b900 comp rx=0 tx=0).stop 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 shutdown_connections 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0aac06b4d0 0x7f0aac06d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad015bdb0 0x7f0ad00700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ad00d21c0 0x7f0ad006fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 --2- 172.21.15.32:0/1212570902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad00d1310 0x7f0ad01534e0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 >> 172.21.15.32:0/1212570902 conn(0x7f0ad0073120 msgr2=0x7f0ad015f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 shutdown_connections 2024-03-06T07:33:54.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.598+0000 7f0ad6000640 1 -- 172.21.15.32:0/1212570902 wait complete. 2024-03-06T07:33:54.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:33:54.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.844+0000 7f90c2800640 1 Processor -- start 2024-03-06T07:33:54.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.845+0000 7f90c2800640 1 -- start start 2024-03-06T07:33:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.846+0000 7f90c2800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 0x7f90bc070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.846+0000 7f90c2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90bc0b6230 0x7f90bc0b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.846+0000 7f90c2800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc06f0a0 con 0x7f90bc161cc0 2024-03-06T07:33:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.846+0000 7f90c2800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc06fcb0 con 0x7f90bc0b6230 2024-03-06T07:33:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.846+0000 7f90c2800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc06f380 con 0x7f90bc15df70 2024-03-06T07:33:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.847+0000 7f90bb400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 0x7f90bc070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.847+0000 7f90baa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90bc0b6230 0x7f90bc0b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.847+0000 7f90bb400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 0x7f90bc070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:53698/0 (socket says 172.21.15.32:53698) 2024-03-06T07:33:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.847+0000 7f90bb400640 1 -- 172.21.15.32:0/703423476 learned_addr learned my addr 172.21.15.32:0/703423476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.848+0000 7f90bb400640 1 -- 172.21.15.32:0/703423476 >> v2:172.21.15.39:3300/0 conn(0x7f90bc0b6230 msgr2=0x7f90bc0b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.848+0000 7f90bb400640 1 --2- 172.21.15.32:0/703423476 >> v2:172.21.15.39:3300/0 conn(0x7f90bc0b6230 0x7f90bc0b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.848+0000 7f90bb400640 1 -- 172.21.15.32:0/703423476 >> v1:172.21.15.39:6789/0 conn(0x7f90bc161cc0 legacy=0x7f90bc06c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.848+0000 7f90bb400640 1 -- 172.21.15.32:0/703423476 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc06f660 con 0x7f90bc15df70 2024-03-06T07:33:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.848+0000 7f90baa00640 1 --2- 172.21.15.32:0/703423476 >> v2:172.21.15.39:3300/0 conn(0x7f90bc0b6230 0x7f90bc0b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.850+0000 7f90bb400640 1 --2- 172.21.15.32:0/703423476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 0x7f90bc070140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009870 tx=0x7f90ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=929da2418b9acd5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90ba000640 1 -- 172.21.15.32:0/703423476 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac034070 con 0x7f90bc15df70 2024-03-06T07:33:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90ba000640 1 -- 172.21.15.32:0/703423476 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac030840 con 0x7f90bc15df70 2024-03-06T07:33:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90ba000640 1 -- 172.21.15.32:0/703423476 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac030c10 con 0x7f90bc15df70 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90c2800640 1 -- 172.21.15.32:0/703423476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 msgr2=0x7f90bc070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90c2800640 1 --2- 172.21.15.32:0/703423476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 0x7f90bc070140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009870 tx=0x7f90ac02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90c2800640 1 -- 172.21.15.32:0/703423476 shutdown_connections 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90c2800640 1 --2- 172.21.15.32:0/703423476 >> v2:172.21.15.39:3300/0 conn(0x7f90bc0b6230 0x7f90bc0b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90c2800640 1 --2- 172.21.15.32:0/703423476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc15df70 0x7f90bc070140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.851+0000 7f90c2800640 1 -- 172.21.15.32:0/703423476 >> 172.21.15.32:0/703423476 conn(0x7f90bc09aa10 msgr2=0x7f90bc09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.852+0000 7f90c2800640 1 -- 172.21.15.32:0/703423476 shutdown_connections 2024-03-06T07:33:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.852+0000 7f90c2800640 1 -- 172.21.15.32:0/703423476 wait complete. 2024-03-06T07:33:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.854+0000 7f90c2800640 1 Processor -- start 2024-03-06T07:33:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.854+0000 7f90c2800640 1 -- start start 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90c2800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc0b6230 0x7f90bc145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc15df70 0x7f90bc1e88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90c2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 0x7f90bc0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90c2800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc070800 con 0x7f90bc161cc0 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90c2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc070680 con 0x7f90bc15df70 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90c2800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc15f360 con 0x7f90bc0b6230 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 0x7f90bc0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 0x7f90bc0b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48496/0 (socket says 172.21.15.32:48496) 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90bbe00640 1 -- 172.21.15.32:0/3102047895 learned_addr learned my addr 172.21.15.32:0/3102047895 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90bb400640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc0b6230 0x7f90bc145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.856+0000 7f90baa00640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc15df70 0x7f90bc1e88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90bbe00640 1 -- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc0b6230 msgr2=0x7f90bc145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90bbe00640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc0b6230 0x7f90bc145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90bbe00640 1 -- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc15df70 msgr2=0x7f90bc1e88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90bbe00640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc15df70 0x7f90bc1e88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90bbe00640 1 -- 172.21.15.32:0/3102047895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc153b80 con 0x7f90bc161cc0 2024-03-06T07:33:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90bbe00640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 0x7f90bc0b3120 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c6e0 tx=0x7f90b400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4003b70 con 0x7f90bc161cc0 2024-03-06T07:33:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b4003d30 con 0x7f90bc161cc0 2024-03-06T07:33:54.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4005310 con 0x7f90bc161cc0 2024-03-06T07:33:54.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.857+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc153e00 con 0x7f90bc161cc0 2024-03-06T07:33:54.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.858+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc1540c0 con 0x7f90bc161cc0 2024-03-06T07:33:54.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.859+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc04f3a0 con 0x7f90bc161cc0 2024-03-06T07:33:54.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.859+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90b40054b0 con 0x7f90bc161cc0 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.860+0000 7f90b3400640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f909c06b4d0 0x7f909c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.860+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90b408a120 con 0x7f90bc161cc0 2024-03-06T07:33:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.861+0000 7f90bb400640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f909c06b4d0 0x7f909c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.861+0000 7f90bb400640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f909c06b4d0 0x7f909c06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f90ac036000 tx=0x7f90ac03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:54.869+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90bc153b80 con 0x7f90bc161cc0 2024-03-06T07:33:55.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.111+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f90bc1607e0 con 0x7f90bc161cc0 2024-03-06T07:33:55.113 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:33:55.112+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:55.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.116+0000 7f90b3400640 1 -- 172.21.15.32:0/3102047895 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) v1 ==== 161+0+0 (secure 0 0 0) 0x7f90b4058ec0 con 0x7f90bc161cc0 2024-03-06T07:33:55.117 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:33:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.119+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f909c06b4d0 msgr2=0x7f909c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.119+0000 7f90c2800640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f909c06b4d0 0x7f909c06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f90ac036000 tx=0x7f90ac03b040 comp rx=0 tx=0).stop 2024-03-06T07:33:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.119+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 msgr2=0x7f90bc0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.119+0000 7f90c2800640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 0x7f90bc0b3120 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c6e0 tx=0x7f90b400cba0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.119+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 shutdown_connections 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f909c06b4d0 0x7f909c06d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90bc161cc0 0x7f90bc0b3120 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc15df70 0x7f90bc1e88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 --2- 172.21.15.32:0/3102047895 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90bc0b6230 0x7f90bc145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 >> 172.21.15.32:0/3102047895 conn(0x7f90bc09aa10 msgr2=0x7f90bc160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 shutdown_connections 2024-03-06T07:33:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.120+0000 7f90c2800640 1 -- 172.21.15.32:0/3102047895 wait complete. 2024-03-06T07:33:55.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:33:55.152 INFO:tasks.ceph.mds.c.smithi032.stderr: -4076> 2024-03-06T07:33:39.068+0000 7fa659000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:55.229 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:33:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.331+0000 7fa3d1200640 1 Processor -- start 2024-03-06T07:33:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.332+0000 7fa3d1200640 1 -- start start 2024-03-06T07:33:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.333+0000 7fa3d1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3cc15afa0 0x7fa3cc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.333+0000 7fa3d1200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa3cc0cd1e0 0x7fa3cc0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.333+0000 7fa3d1200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc071c60 con 0x7fa3cc070ce0 2024-03-06T07:33:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.333+0000 7fa3d1200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cfa10 con 0x7fa3cc15afa0 2024-03-06T07:33:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.333+0000 7fa3d1200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc071f40 con 0x7fa3cc0cd1e0 2024-03-06T07:33:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.333+0000 7fa3c9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3cc15afa0 0x7fa3cc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3cc15afa0 0x7fa3cc0cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48514/0 (socket says 172.21.15.32:48514) 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa3cc070ce0 0x7fa3cc0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51046/0 (socket says 172.21.15.32:51046) 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3876974918 learned_addr learned my addr 172.21.15.32:0/3876974918 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3ca800640 1 --2- 172.21.15.32:0/3876974918 >> v2:172.21.15.39:3300/0 conn(0x7fa3cc0cd1e0 0x7fa3cc0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3876974918 >> v2:172.21.15.39:3300/0 conn(0x7fa3cc0cd1e0 msgr2=0x7fa3cc0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9e00640 1 --2- 172.21.15.32:0/3876974918 >> v2:172.21.15.39:3300/0 conn(0x7fa3cc0cd1e0 0x7fa3cc0b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3876974918 >> v1:172.21.15.32:6789/0 conn(0x7fa3cc070ce0 legacy=0x7fa3cc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3876974918 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc072220 con 0x7fa3cc15afa0 2024-03-06T07:33:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.334+0000 7fa3c9400640 1 -- 172.21.15.32:0/3876974918 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332882355 0 0) 0x7fa3cc071c60 con 0x7fa3cc070ce0 2024-03-06T07:33:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.335+0000 7fa3c9e00640 1 --2- 172.21.15.32:0/3876974918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc15afa0 0x7fa3cc0cf3a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc070b90 tx=0x7fa3bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69590197ff8cf23e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.335+0000 7fa3c9400640 1 -- 172.21.15.32:0/3876974918 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc035020 con 0x7fa3cc15afa0 2024-03-06T07:33:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.335+0000 7fa3c9400640 1 -- 172.21.15.32:0/3876974918 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc031870 con 0x7fa3cc15afa0 2024-03-06T07:33:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.335+0000 7fa3c9400640 1 -- 172.21.15.32:0/3876974918 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc031c40 con 0x7fa3cc15afa0 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 -- 172.21.15.32:0/3876974918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc15afa0 msgr2=0x7fa3cc0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3876974918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc15afa0 0x7fa3cc0cf3a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc070b90 tx=0x7fa3bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 -- 172.21.15.32:0/3876974918 shutdown_connections 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3876974918 >> v2:172.21.15.39:3300/0 conn(0x7fa3cc0cd1e0 0x7fa3cc0b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3876974918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc15afa0 0x7fa3cc0cf3a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 -- 172.21.15.32:0/3876974918 >> 172.21.15.32:0/3876974918 conn(0x7fa3cc06b3d0 msgr2=0x7fa3cc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 -- 172.21.15.32:0/3876974918 shutdown_connections 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.336+0000 7fa3d1200640 1 -- 172.21.15.32:0/3876974918 wait complete. 2024-03-06T07:33:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.338+0000 7fa3d1200640 1 Processor -- start 2024-03-06T07:33:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.338+0000 7fa3d1200640 1 -- start start 2024-03-06T07:33:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3d1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 0x7fa3cc09b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3d1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3cc0cd1e0 0x7fa3cc1470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3d1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3cc15afa0 0x7fa3cc14a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3d1200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0b4000 con 0x7fa3cc070ce0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3d1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc071170 con 0x7fa3cc15afa0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3d1200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0b4180 con 0x7fa3cc0cd1e0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3c9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 0x7fa3cc09b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3c9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 0x7fa3cc09b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48522/0 (socket says 172.21.15.32:48522) 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.339+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3797213394 learned_addr learned my addr 172.21.15.32:0/3797213394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3797213394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3cc0cd1e0 msgr2=0x7fa3cc1470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c9e00640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3cc0cd1e0 0x7fa3cc1470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3797213394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3cc15afa0 msgr2=0x7fa3cc14a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c9e00640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3cc15afa0 0x7fa3cc14a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c9e00640 1 -- 172.21.15.32:0/3797213394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc09bd80 con 0x7fa3cc070ce0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c9e00640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 0x7fa3cc09b680 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc02b950 tx=0x7fa3bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc040070 con 0x7fa3cc070ce0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc030d00 con 0x7fa3cc070ce0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc035040 con 0x7fa3cc070ce0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc09c060 con 0x7fa3cc070ce0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.340+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc09c620 con 0x7fa3cc070ce0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.341+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc04f330 con 0x7fa3cc070ce0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.342+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3bc038020 con 0x7fa3cc070ce0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.344+0000 7fa3c2000640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa3a806b4d0 0x7fa3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.344+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa3bc0ac100 con 0x7fa3cc070ce0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.344+0000 7fa3c3e00640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa3a806b4d0 0x7fa3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.344+0000 7fa3c3e00640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa3a806b4d0 0x7fa3a806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4007fd0 tx=0x7fa3b4007920 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.348+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3bc07aee0 con 0x7fa3cc070ce0 2024-03-06T07:33:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:55.586+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3cc09dea0 con 0x7fa3cc070ce0 2024-03-06T07:33:56.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.119+0000 7fa3c2000640 1 -- 172.21.15.32:0/3797213394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3bc07a7e0 con 0x7fa3cc070ce0 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa3a806b4d0 msgr2=0x7fa3a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa3a806b4d0 0x7fa3a806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4007fd0 tx=0x7fa3b4007920 comp rx=0 tx=0).stop 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 msgr2=0x7fa3cc09b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 0x7fa3cc09b680 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc02b950 tx=0x7fa3bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 shutdown_connections 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa3a806b4d0 0x7fa3a806d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3cc15afa0 0x7fa3cc14a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3cc0cd1e0 0x7fa3cc1470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 --2- 172.21.15.32:0/3797213394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3cc070ce0 0x7fa3cc09b680 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 >> 172.21.15.32:0/3797213394 conn(0x7fa3cc06b3d0 msgr2=0x7fa3cc159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.123+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 shutdown_connections 2024-03-06T07:33:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.124+0000 7fa3d1200640 1 -- 172.21.15.32:0/3797213394 wait complete. 2024-03-06T07:33:56.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:33:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.417+0000 7fe8fde00640 1 Processor -- start 2024-03-06T07:33:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.418+0000 7fe8fde00640 1 -- start start 2024-03-06T07:33:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.419+0000 7fe8fde00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8f806afb0 0x7fe8f80afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.419+0000 7fe8fde00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.419+0000 7fe8fde00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f806e390 con 0x7fe8f806b390 2024-03-06T07:33:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.419+0000 7fe8fde00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f80aff60 con 0x7fe8f806afb0 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.419+0000 7fe8fde00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80ae120 con 0x7fe8f806d9c0 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8f806afb0 0x7fe8f80afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8f806afb0 0x7fe8f80afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48530/0 (socket says 172.21.15.32:48530) 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 -- 172.21.15.32:0/2460479223 learned_addr learned my addr 172.21.15.32:0/2460479223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 -- 172.21.15.32:0/2460479223 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 msgr2=0x7fe8f806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fd400640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f806de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 -- 172.21.15.32:0/2460479223 >> v1:172.21.15.39:6789/0 conn(0x7fe8f806b390 legacy=0x7fe8f80aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fca00640 1 -- 172.21.15.32:0/2460479223 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f80b3eb0 con 0x7fe8f806afb0 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.420+0000 7fe8fd400640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.421+0000 7fe8fca00640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f80afa20 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009870 tx=0x7fe8ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45c0005369e9ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.422+0000 7fe8f7400640 1 -- 172.21.15.32:0/2460479223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec035020 con 0x7fe8f806afb0 2024-03-06T07:33:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.422+0000 7fe8f7400640 1 -- 172.21.15.32:0/2460479223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec031640 con 0x7fe8f806afb0 2024-03-06T07:33:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.422+0000 7fe8f7400640 1 -- 172.21.15.32:0/2460479223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec02d040 con 0x7fe8f806afb0 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.422+0000 7fe8fde00640 1 -- 172.21.15.32:0/2460479223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 msgr2=0x7fe8f80afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.422+0000 7fe8fde00640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f80afa20 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009870 tx=0x7fe8ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.423+0000 7fe8fde00640 1 -- 172.21.15.32:0/2460479223 shutdown_connections 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.423+0000 7fe8fde00640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.423+0000 7fe8fde00640 1 --2- 172.21.15.32:0/2460479223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f80afa20 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.423+0000 7fe8fde00640 1 -- 172.21.15.32:0/2460479223 >> 172.21.15.32:0/2460479223 conn(0x7fe8f80b27f0 msgr2=0x7fe8f80b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.423+0000 7fe8fde00640 1 -- 172.21.15.32:0/2460479223 shutdown_connections 2024-03-06T07:33:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.423+0000 7fe8fde00640 1 -- 172.21.15.32:0/2460479223 wait complete. 2024-03-06T07:33:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.424+0000 7fe8fde00640 1 Processor -- start 2024-03-06T07:33:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.425+0000 7fe8fde00640 1 -- start start 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.425+0000 7fe8fde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f81e9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.425+0000 7fe8fde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f806b390 0x7fe8f81593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.425+0000 7fe8fde00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f8159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fde00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80ae960 con 0x7fe8f806afb0 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fde00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80ae680 con 0x7fe8f806b390 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fde00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80af590 con 0x7fe8f806d9c0 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f81e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f81e9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48540/0 (socket says 172.21.15.32:48540) 2024-03-06T07:33:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 -- 172.21.15.32:0/4207811372 learned_addr learned my addr 172.21.15.32:0/4207811372 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 -- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 msgr2=0x7fe8f8159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fd400640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f8159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8f7e00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f806b390 0x7fe8f81593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f8159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 -- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f806b390 msgr2=0x7fe8f81593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f806b390 0x7fe8f81593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.426+0000 7fe8fca00640 1 -- 172.21.15.32:0/4207811372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f8155ad0 con 0x7fe8f806afb0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.427+0000 7fe8fca00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f81e9860 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0097c0 tx=0x7fe8ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.427+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec040070 con 0x7fe8f806afb0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.427+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec039cb0 con 0x7fe8f806afb0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.427+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec035040 con 0x7fe8f806afb0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.427+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f8156a00 con 0x7fe8f806afb0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.427+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f81e9da0 con 0x7fe8f806afb0 2024-03-06T07:33:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.428+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f804f330 con 0x7fe8f806afb0 2024-03-06T07:33:56.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.429+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8ec04b060 con 0x7fe8f806afb0 2024-03-06T07:33:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.430+0000 7fe8f4c00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe8d806b4d0 0x7fe8d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.430+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe8ec0abd90 con 0x7fe8f806afb0 2024-03-06T07:33:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.430+0000 7fe8f7e00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe8d806b4d0 0x7fe8d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.431+0000 7fe8f7e00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe8d806b4d0 0x7fe8d806d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8156110 tx=0x7fe8e4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.435+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8ec07aae0 con 0x7fe8f806afb0 2024-03-06T07:33:56.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:56.638+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe8f80681f0 con 0x7fe8f806afb0 2024-03-06T07:33:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.123+0000 7fe8f4c00640 1 -- 172.21.15.32:0/4207811372 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe8ec039e60 con 0x7fe8f806afb0 2024-03-06T07:33:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:33:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe8d806b4d0 msgr2=0x7fe8d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe8d806b4d0 0x7fe8d806d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8156110 tx=0x7fe8e4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 msgr2=0x7fe8f81e9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f81e9860 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0097c0 tx=0x7fe8ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 shutdown_connections 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe8d806b4d0 0x7fe8d806d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe8f806d9c0 0x7fe8f8159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f806b390 0x7fe8f81593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 --2- 172.21.15.32:0/4207811372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f806afb0 0x7fe8f81e9860 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 >> 172.21.15.32:0/4207811372 conn(0x7fe8f80b27f0 msgr2=0x7fe8f815ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 shutdown_connections 2024-03-06T07:33:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.127+0000 7fe8fde00640 1 -- 172.21.15.32:0/4207811372 wait complete. 2024-03-06T07:33:57.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:33:57.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.387+0000 7fadfb200640 1 Processor -- start 2024-03-06T07:33:57.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.388+0000 7fadfb200640 1 -- start start 2024-03-06T07:33:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.389+0000 7fadfb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fadfc066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.389+0000 7fadfb200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc0d06b0 con 0x7fadfc0b5cb0 2024-03-06T07:33:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.389+0000 7fadfb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc0734e0 con 0x7fadfc0b58d0 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.389+0000 7fadfb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc161bb0 con 0x7fadfc15e630 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.390+0000 7fadfa800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fadfc0b5cb0 0x7fadfc06c060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51052/0 (socket says 172.21.15.32:51052) 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.390+0000 7fadfa800640 1 -- 172.21.15.32:0/2554847307 learned_addr learned my addr 172.21.15.32:0/2554847307 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.390+0000 7fadf9400640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fadfc066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.390+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407582854 0 0) 0x7fadfc0d06b0 con 0x7fadfc0b5cb0 2024-03-06T07:33:57.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.390+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade0003520 con 0x7fadfc0b5cb0 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215947166 0 0) 0x7fade0003520 con 0x7fadfc0b5cb0 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 msgr2=0x7fadfc066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fadfc066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 >> v1:172.21.15.39:6789/0 conn(0x7fadfc0b58d0 legacy=0x7fadfc0cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc15ead0 con 0x7fadfc0b5cb0 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fadf4001670 con 0x7fadfc0b5cb0 2024-03-06T07:33:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 >> v1:172.21.15.32:6789/0 conn(0x7fadfc0b5cb0 legacy=0x7fadfc06c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade00039e0 0x7fade0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade0004800 0x7fade0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fade00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade000ba70 con 0x7fade0004800 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade000bc20 con 0x7fade00039e0 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade000be20 con 0x7fadfc15e630 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf9400640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade0004800 0x7fade0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.391+0000 7fadf9e00640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade00039e0 0x7fade0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadf9400640 1 -- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 msgr2=0x7fade00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadfa800640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fade00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadf9400640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fade00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadf9400640 1 -- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade00039e0 msgr2=0x7fade0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadf9400640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade00039e0 0x7fade0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadf9400640 1 -- 172.21.15.32:0/2554847307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade0003520 con 0x7fade0004800 2024-03-06T07:33:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.392+0000 7fadfa800640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc15e630 0x7fade00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.393+0000 7fadf9400640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade0004800 0x7fade0004c80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fade4009a90 tx=0x7fade0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289586c83f3591e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.393+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4039070 con 0x7fade0004800 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.393+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade402c660 con 0x7fade0004800 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.393+0000 7fadf8a00640 1 -- 172.21.15.32:0/2554847307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4009690 con 0x7fade0004800 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfb200640 1 -- 172.21.15.32:0/2554847307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade0004800 msgr2=0x7fade0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfb200640 1 --2- 172.21.15.32:0/2554847307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade0004800 0x7fade0004c80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fade4009a90 tx=0x7fade0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfa800640 1 -- 172.21.15.32:0/2554847307 reap_dead start 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfb200640 1 -- 172.21.15.32:0/2554847307 shutdown_connections 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfb200640 1 -- 172.21.15.32:0/2554847307 >> 172.21.15.32:0/2554847307 conn(0x7fadfc0725f0 msgr2=0x7fadfc072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfb200640 1 -- 172.21.15.32:0/2554847307 shutdown_connections 2024-03-06T07:33:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.394+0000 7fadfb200640 1 -- 172.21.15.32:0/2554847307 wait complete. 2024-03-06T07:33:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.396+0000 7fadfb200640 1 Processor -- start 2024-03-06T07:33:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.396+0000 7fadfb200640 1 -- start start 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadfb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0004800 0x7fadfc0b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadfb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 0x7fadfc0b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadfb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc09a400 0x7fadfc09a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadfb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc15f350 con 0x7fadfc0afcd0 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadfb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc15f070 con 0x7fade0004800 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadfb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc06c770 con 0x7fadfc09a400 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadf9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 0x7fadfc0b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadf9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 0x7fadfc0b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48560/0 (socket says 172.21.15.32:48560) 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadf9400640 1 -- 172.21.15.32:0/1068553731 learned_addr learned my addr 172.21.15.32:0/1068553731 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.397+0000 7fadf9e00640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0004800 0x7fadfc0b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadfa800640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc09a400 0x7fadfc09a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf9400640 1 -- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc09a400 msgr2=0x7fadfc09a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf9400640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc09a400 0x7fadfc09a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf9400640 1 -- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0004800 msgr2=0x7fadfc0b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf9400640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0004800 0x7fadfc0b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf9400640 1 -- 172.21.15.32:0/1068553731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc0b0690 con 0x7fadfc0afcd0 2024-03-06T07:33:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf9400640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 0x7fadfc0b0150 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fade4000c00 tx=0x7fade4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.398+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4042020 con 0x7fadfc0afcd0 2024-03-06T07:33:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.399+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade403a040 con 0x7fadfc0afcd0 2024-03-06T07:33:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.399+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4039070 con 0x7fadfc0afcd0 2024-03-06T07:33:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.399+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc0b0900 con 0x7fadfc0afcd0 2024-03-06T07:33:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.399+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc0b0ec0 con 0x7fadfc0afcd0 2024-03-06T07:33:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.400+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc04f330 con 0x7fadfc0afcd0 2024-03-06T07:33:57.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.401+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fade4045020 con 0x7fadfc0afcd0 2024-03-06T07:33:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.402+0000 7fadf2000640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadd806b4d0 0x7fadd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.402+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fade40ab5f0 con 0x7fadfc0afcd0 2024-03-06T07:33:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.402+0000 7fadf9e00640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadd806b4d0 0x7fadd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.403+0000 7fadf9e00640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadd806b4d0 0x7fadd806d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadfc160d40 tx=0x7fadec009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.407+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fade404c070 con 0x7fadfc0afcd0 2024-03-06T07:33:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:57.606+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fadfc066e30 con 0x7fadfc0afcd0 2024-03-06T07:33:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.137+0000 7fadf2000640 1 -- 172.21.15.32:0/1068553731 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) v1 ==== 169+0+0 (secure 0 0 0) 0x7fade407a4f0 con 0x7fadfc0afcd0 2024-03-06T07:33:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:33:58.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadd806b4d0 msgr2=0x7fadd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadd806b4d0 0x7fadd806d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadfc160d40 tx=0x7fadec009040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 msgr2=0x7fadfc0b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 0x7fadfc0b0150 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fade4000c00 tx=0x7fade4004730 comp rx=0 tx=0).stop 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 shutdown_connections 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fadd806b4d0 0x7fadd806d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fadfc09a400 0x7fadfc09a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadfc0afcd0 0x7fadfc0b0150 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 --2- 172.21.15.32:0/1068553731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0004800 0x7fadfc0b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 >> 172.21.15.32:0/1068553731 conn(0x7fadfc0725f0 msgr2=0x7fadfc15c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 shutdown_connections 2024-03-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.141+0000 7fadfb200640 1 -- 172.21.15.32:0/1068553731 wait complete. 2024-03-06T07:33:58.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:33:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.391+0000 7f09f9600640 1 Processor -- start 2024-03-06T07:33:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.391+0000 7f09f9600640 1 -- start start 2024-03-06T07:33:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.393+0000 7f09f9600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f09f4165d50 0x7f09f40b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.393+0000 7f09f9600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09f40b1b40 0x7f09f40b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.393+0000 7f09f9600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f4166cb0 con 0x7f09f4162280 2024-03-06T07:33:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.393+0000 7f09f9600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f40b24f0 con 0x7f09f40b1b40 2024-03-06T07:33:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.393+0000 7f09f9600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f40b4fd0 con 0x7f09f4165d50 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09f40b1b40 0x7f09f40b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09f40b1b40 0x7f09f40b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58514/0 (socket says 172.21.15.32:58514) 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 -- 172.21.15.32:0/474112921 learned_addr learned my addr 172.21.15.32:0/474112921 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f3400640 1 --2- 172.21.15.32:0/474112921 >> v2:172.21.15.39:3300/0 conn(0x7f09f4165d50 0x7f09f40b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 -- 172.21.15.32:0/474112921 >> v2:172.21.15.39:3300/0 conn(0x7f09f4165d50 msgr2=0x7f09f40b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 --2- 172.21.15.32:0/474112921 >> v2:172.21.15.39:3300/0 conn(0x7f09f4165d50 0x7f09f40b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 -- 172.21.15.32:0/474112921 >> v1:172.21.15.39:6789/0 conn(0x7f09f4162280 legacy=0x7f09f40cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f8c00640 1 -- 172.21.15.32:0/474112921 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4166410 con 0x7f09f40b1b40 2024-03-06T07:33:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.394+0000 7f09f3400640 1 --2- 172.21.15.32:0/474112921 >> v2:172.21.15.39:3300/0 conn(0x7f09f4165d50 0x7f09f40b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.395+0000 7f09f8c00640 1 --2- 172.21.15.32:0/474112921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f40b1b40 0x7f09f40b1fb0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f09f4072f30 tx=0x7f09ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f5163517378f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f2a00640 1 -- 172.21.15.32:0/474112921 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec02e070 con 0x7f09f40b1b40 2024-03-06T07:33:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f2a00640 1 -- 172.21.15.32:0/474112921 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec037070 con 0x7f09f40b1b40 2024-03-06T07:33:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f2a00640 1 -- 172.21.15.32:0/474112921 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec031a00 con 0x7f09f40b1b40 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 -- 172.21.15.32:0/474112921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f40b1b40 msgr2=0x7f09f40b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 --2- 172.21.15.32:0/474112921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f40b1b40 0x7f09f40b1fb0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f09f4072f30 tx=0x7f09ec02b880 comp rx=0 tx=0).stop 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 -- 172.21.15.32:0/474112921 shutdown_connections 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 --2- 172.21.15.32:0/474112921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f40b1b40 0x7f09f40b1fb0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 --2- 172.21.15.32:0/474112921 >> v2:172.21.15.39:3300/0 conn(0x7f09f4165d50 0x7f09f40b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 -- 172.21.15.32:0/474112921 >> 172.21.15.32:0/474112921 conn(0x7f09f406b3a0 msgr2=0x7f09f406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 -- 172.21.15.32:0/474112921 shutdown_connections 2024-03-06T07:33:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.396+0000 7f09f9600640 1 -- 172.21.15.32:0/474112921 wait complete. 2024-03-06T07:33:58.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.398+0000 7f09f9600640 1 Processor -- start 2024-03-06T07:33:58.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.398+0000 7f09f9600640 1 -- start start 2024-03-06T07:33:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f9600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09f40b1b40 0x7f09f41559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 0x7f09f4147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09f4165d50 0x7f09f41f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41669d0 con 0x7f09f4162280 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f9600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41661b0 con 0x7f09f4165d50 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f9600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f4071560 con 0x7f09f40b1b40 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.399+0000 7f09f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 0x7f09f4147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 0x7f09f4147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58524/0 (socket says 172.21.15.32:58524) 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 -- 172.21.15.32:0/4070343531 learned_addr learned my addr 172.21.15.32:0/4070343531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f8c00640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09f4165d50 0x7f09f41f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3e00640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09f40b1b40 0x7f09f41559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 -- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09f40b1b40 msgr2=0x7f09f41559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09f40b1b40 0x7f09f41559d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 -- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09f4165d50 msgr2=0x7f09f41f77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09f4165d50 0x7f09f41f77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 -- 172.21.15.32:0/4070343531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f41ec330 con 0x7f09f4162280 2024-03-06T07:33:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.400+0000 7f09f3400640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 0x7f09f4147e30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00c7f0 tx=0x7f09dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.401+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc015070 con 0x7f09f4162280 2024-03-06T07:33:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.401+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc007820 con 0x7f09f4162280 2024-03-06T07:33:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.401+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc007bd0 con 0x7f09f4162280 2024-03-06T07:33:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.401+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f41560b0 con 0x7f09f4162280 2024-03-06T07:33:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.401+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f4156670 con 0x7f09f4162280 2024-03-06T07:33:58.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.402+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f404f330 con 0x7f09f4162280 2024-03-06T07:33:58.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.403+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09dc018070 con 0x7f09f4162280 2024-03-06T07:33:58.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.404+0000 7f09ebe00640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.404+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09dc08a0d0 con 0x7f09f4162280 2024-03-06T07:33:58.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.404+0000 7f09f3e00640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.405+0000 7f09f3e00640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09d006b4d0 0x7f09d006d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f09e4007fd0 tx=0x7f09e4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.408+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09dc0591b0 con 0x7f09f4162280 2024-03-06T07:33:58.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.608+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f09f41ec520 con 0x7f09f4162280 2024-03-06T07:33:58.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.615+0000 7f09ebe00640 1 -- 172.21.15.32:0/4070343531 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f09dc058ab0 con 0x7f09f4162280 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.617+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09d006b4d0 msgr2=0x7f09d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.617+0000 7f09f9600640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09d006b4d0 0x7f09d006d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f09e4007fd0 tx=0x7f09e4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 msgr2=0x7f09f4147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 0x7f09f4147e30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00c7f0 tx=0x7f09dc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 shutdown_connections 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09f4165d50 0x7f09f41f77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09f4162280 0x7f09f4147e30 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 --2- 172.21.15.32:0/4070343531 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f09f40b1b40 0x7f09f41559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 >> 172.21.15.32:0/4070343531 conn(0x7f09f406b3a0 msgr2=0x7f09f40b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 shutdown_connections 2024-03-06T07:33:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.618+0000 7f09f9600640 1 -- 172.21.15.32:0/4070343531 wait complete. 2024-03-06T07:33:58.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:33:58.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.864+0000 7f7da4e00640 1 Processor -- start 2024-03-06T07:33:58.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.864+0000 7f7da4e00640 1 -- start start 2024-03-06T07:33:58.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7da4e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7da0145790 0x7f7da00d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7da4e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da00b5de0 con 0x7f7da006afb0 2024-03-06T07:33:58.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7da4e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da00b6080 con 0x7f7da00add50 2024-03-06T07:33:58.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7da4e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da00d1b70 con 0x7f7da0145790 2024-03-06T07:33:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7d9de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7da0145790 0x7f7da00d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7d9e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7da006afb0 0x7f7da00af720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48330/0 (socket says 172.21.15.32:48330) 2024-03-06T07:33:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.866+0000 7f7d9e800640 1 -- 172.21.15.32:0/1113574392 learned_addr learned my addr 172.21.15.32:0/1113574392 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:33:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.867+0000 7f7d9de00640 1 -- 172.21.15.32:0/1113574392 >> v1:172.21.15.39:6789/0 conn(0x7f7da00add50 legacy=0x7f7da01603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.867+0000 7f7d9de00640 1 -- 172.21.15.32:0/1113574392 >> v1:172.21.15.32:6789/0 conn(0x7f7da006afb0 legacy=0x7f7da00af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.867+0000 7f7d9de00640 1 -- 172.21.15.32:0/1113574392 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da015fbf0 con 0x7f7da0145790 2024-03-06T07:33:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.868+0000 7f7d9de00640 1 --2- 172.21.15.32:0/1113574392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da0145790 0x7f7da00d1630 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7d900097a0 tx=0x7f7d9002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f0d246b33a99e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.868+0000 7f7d9ca00640 1 -- 172.21.15.32:0/1113574392 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90034070 con 0x7f7da0145790 2024-03-06T07:33:58.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7d9ca00640 1 -- 172.21.15.32:0/1113574392 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90030610 con 0x7f7da0145790 2024-03-06T07:33:58.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7d9ca00640 1 -- 172.21.15.32:0/1113574392 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d900309e0 con 0x7f7da0145790 2024-03-06T07:33:58.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 -- 172.21.15.32:0/1113574392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da0145790 msgr2=0x7f7da00d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 --2- 172.21.15.32:0/1113574392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da0145790 0x7f7da00d1630 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7d900097a0 tx=0x7f7d9002b960 comp rx=0 tx=0).stop 2024-03-06T07:33:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 -- 172.21.15.32:0/1113574392 shutdown_connections 2024-03-06T07:33:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 --2- 172.21.15.32:0/1113574392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da0145790 0x7f7da00d1630 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 -- 172.21.15.32:0/1113574392 >> 172.21.15.32:0/1113574392 conn(0x7f7da006c2a0 msgr2=0x7f7da006c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 -- 172.21.15.32:0/1113574392 shutdown_connections 2024-03-06T07:33:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.869+0000 7f7da4e00640 1 -- 172.21.15.32:0/1113574392 wait complete. 2024-03-06T07:33:58.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.871+0000 7f7da4e00640 1 Processor -- start 2024-03-06T07:33:58.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.871+0000 7f7da4e00640 1 -- start start 2024-03-06T07:33:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.872+0000 7f7da4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7da006afb0 0x7f7da013d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.872+0000 7f7da4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 0x7f7da0141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.872+0000 7f7da4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7da0145790 0x7f7da014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.872+0000 7f7da4e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da0145b70 con 0x7f7da00add50 2024-03-06T07:33:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.872+0000 7f7da4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da009ef10 con 0x7f7da006afb0 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.872+0000 7f7da4e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da006b440 con 0x7f7da0145790 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 0x7f7da0141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 0x7f7da0141ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58544/0 (socket says 172.21.15.32:58544) 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 -- 172.21.15.32:0/4222083803 learned_addr learned my addr 172.21.15.32:0/4222083803 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9de00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7da006afb0 0x7f7da013d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9e800640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7da0145790 0x7f7da014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 -- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7da0145790 msgr2=0x7f7da014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7da0145790 0x7f7da014d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 -- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7da006afb0 msgr2=0x7f7da013d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7da006afb0 0x7f7da013d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 -- 172.21.15.32:0/4222083803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da0140c40 con 0x7f7da00add50 2024-03-06T07:33:58.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.873+0000 7f7d9d400640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 0x7f7da0141ce0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c440 tx=0x7f7d8c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.874+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0052d0 con 0x7f7da00add50 2024-03-06T07:33:58.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.874+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c005470 con 0x7f7da00add50 2024-03-06T07:33:58.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.874+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c007260 con 0x7f7da00add50 2024-03-06T07:33:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.874+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da0140f20 con 0x7f7da00add50 2024-03-06T07:33:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.874+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da01417c0 con 0x7f7da00add50 2024-03-06T07:33:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.875+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da01410f0 con 0x7f7da00add50 2024-03-06T07:33:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.876+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d8c007400 con 0x7f7da00add50 2024-03-06T07:33:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.877+0000 7f7d96000640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.877+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7d8c0894e0 con 0x7f7da00add50 2024-03-06T07:33:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.877+0000 7f7d9de00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.878+0000 7f7d9de00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d8006b4d0 0x7f7d8006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7d90036000 tx=0x7f7d9003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:58.882+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d8c0585c0 con 0x7f7da00add50 2024-03-06T07:33:59.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:33:59.159+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f7da00b0330 con 0x7f7da00add50 2024-03-06T07:34:00.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.153+0000 7f7d96000640 1 -- 172.21.15.32:0/4222083803 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v282) v1 ==== 132+0+0 (secure 0 0 0) 0x7f7da00b0330 con 0x7f7da00add50 2024-03-06T07:34:00.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.156+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d8006b4d0 msgr2=0x7f7d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.156+0000 7f7da4e00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d8006b4d0 0x7f7d8006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7d90036000 tx=0x7f7d9003b040 comp rx=0 tx=0).stop 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 msgr2=0x7f7da0141ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 0x7f7da0141ce0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c440 tx=0x7f7d8c001e70 comp rx=0 tx=0).stop 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 shutdown_connections 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7da0145790 0x7f7da014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7da00add50 0x7f7da0141ce0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 --2- 172.21.15.32:0/4222083803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7da006afb0 0x7f7da013d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 >> 172.21.15.32:0/4222083803 conn(0x7f7da006c2a0 msgr2=0x7f7da015dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 shutdown_connections 2024-03-06T07:34:00.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.157+0000 7f7da4e00640 1 -- 172.21.15.32:0/4222083803 wait complete. 2024-03-06T07:34:00.179 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:34:00.180 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:00.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.401+0000 7f97cda00640 1 Processor -- start 2024-03-06T07:34:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.402+0000 7f97cda00640 1 -- start start 2024-03-06T07:34:00.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.403+0000 7f97cda00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f97c80d0cd0 0x7f97c80b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.403+0000 7f97cda00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c80d2dc0 con 0x7f97c815e070 2024-03-06T07:34:00.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.403+0000 7f97cda00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c80d18c0 con 0x7f97c80d2960 2024-03-06T07:34:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.403+0000 7f97cda00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c80d1ba0 con 0x7f97c80d0cd0 2024-03-06T07:34:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.403+0000 7f97c7e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f97c815e070 0x7f97c80cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48334/0 (socket says 172.21.15.32:48334) 2024-03-06T07:34:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.403+0000 7f97c7e00640 1 -- 172.21.15.32:0/1507891685 learned_addr learned my addr 172.21.15.32:0/1507891685 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97cd000640 1 --2- 172.21.15.32:0/1507891685 >> v2:172.21.15.39:3300/0 conn(0x7f97c80d0cd0 0x7f97c80b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443637464 0 0) 0x7f97c80d2dc0 con 0x7f97c815e070 2024-03-06T07:34:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac003520 con 0x7f97c815e070 2024-03-06T07:34:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1403211298 0 0) 0x7f97ac003520 con 0x7f97c815e070 2024-03-06T07:34:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 >> v2:172.21.15.39:3300/0 conn(0x7f97c80d0cd0 msgr2=0x7f97c80b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 --2- 172.21.15.32:0/1507891685 >> v2:172.21.15.39:3300/0 conn(0x7f97c80d0cd0 0x7f97c80b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 >> v1:172.21.15.39:6789/0 conn(0x7f97c80d2960 legacy=0x7f97c80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c80d1e80 con 0x7f97c815e070 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f97b80019f0 con 0x7f97c815e070 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 >> v1:172.21.15.32:6789/0 conn(0x7f97c815e070 legacy=0x7f97c80cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.404+0000 7f97c6a00640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac003a00 0x7f97ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c6a00640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ac004800 0x7f97ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c6a00640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97ac0051c0 0x7f97ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80d1ba0 con 0x7f97ac004800 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac00c0c0 con 0x7f97ac0051c0 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac00c2c0 con 0x7f97ac003a00 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7400640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ac004800 0x7f97ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7e00640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac003a00 0x7f97ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97cd000640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97ac0051c0 0x7f97ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7400640 1 -- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac003a00 msgr2=0x7f97ac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7400640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac003a00 0x7f97ac003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7400640 1 -- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97ac0051c0 msgr2=0x7f97ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7400640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97ac0051c0 0x7f97ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97c7400640 1 -- 172.21.15.32:0/1507891685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac003520 con 0x7f97ac004800 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97cd000640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97ac0051c0 0x7f97ac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.405+0000 7f97cd000640 1 -- 172.21.15.32:0/1507891685 reap_dead start 2024-03-06T07:34:00.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.407+0000 7f97c7400640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ac004800 0x7f97ac004c80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f97b4001750 tx=0x7f97ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96da4f3a51a83cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.407+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4001370 con 0x7f97ac004800 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.407+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b40052d0 con 0x7f97ac004800 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.407+0000 7f97c6a00640 1 -- 172.21.15.32:0/1507891685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4003830 con 0x7f97ac004800 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 -- 172.21.15.32:0/1507891685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ac004800 msgr2=0x7f97ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ac004800 0x7f97ac004c80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f97b4001750 tx=0x7f97ac005830 comp rx=0 tx=0).stop 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 -- 172.21.15.32:0/1507891685 shutdown_connections 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 --2- 172.21.15.32:0/1507891685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ac004800 0x7f97ac004c80 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 -- 172.21.15.32:0/1507891685 >> 172.21.15.32:0/1507891685 conn(0x7f97c808d2a0 msgr2=0x7f97c80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 -- 172.21.15.32:0/1507891685 shutdown_connections 2024-03-06T07:34:00.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.408+0000 7f97cda00640 1 -- 172.21.15.32:0/1507891685 wait complete. 2024-03-06T07:34:00.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.410+0000 7f97cda00640 1 Processor -- start 2024-03-06T07:34:00.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.410+0000 7f97cda00640 1 -- start start 2024-03-06T07:34:00.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97cda00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac004800 0x7f97c81500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97cda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 0x7f97c81e7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97cda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c815fd30 0x7f97c81601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97cda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80d45b0 con 0x7f97c81e7a10 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97cda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80d2440 con 0x7f97c815fd30 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97cda00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80d4730 con 0x7f97ac004800 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97c7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 0x7f97c81e7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97c7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 0x7f97c81e7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58564/0 (socket says 172.21.15.32:58564) 2024-03-06T07:34:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.411+0000 7f97c7400640 1 -- 172.21.15.32:0/2604512384 learned_addr learned my addr 172.21.15.32:0/2604512384 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7e00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac004800 0x7f97c81500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97cd000640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c815fd30 0x7f97c81601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7400640 1 -- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac004800 msgr2=0x7f97c81500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7400640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac004800 0x7f97c81500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7400640 1 -- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c815fd30 msgr2=0x7f97c81601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7400640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c815fd30 0x7f97c81601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7400640 1 -- 172.21.15.32:0/2604512384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c815e680 con 0x7f97c81e7a10 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7e00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac004800 0x7f97c81500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97cd000640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c815fd30 0x7f97c81601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97c7400640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 0x7f97c81e7e90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005040 tx=0x7f97b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4041020 con 0x7f97c81e7a10 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b4005480 con 0x7f97c81e7a10 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4007a30 con 0x7f97c81e7a10 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.412+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c815e960 con 0x7f97c81e7a10 2024-03-06T07:34:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.413+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c815ef20 con 0x7f97c81e7a10 2024-03-06T07:34:00.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.414+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c804f330 con 0x7f97c81e7a10 2024-03-06T07:34:00.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.414+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97b4044020 con 0x7f97c81e7a10 2024-03-06T07:34:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.416+0000 7f97bfe00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97a806b500 0x7f97a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.416+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f97b40ab9a0 con 0x7f97c81e7a10 2024-03-06T07:34:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.416+0000 7f97c7e00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97a806b500 0x7f97a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.417+0000 7f97c7e00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97a806b500 0x7f97a806d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f97b800e1c0 tx=0x7f97b8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.420+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97b404b030 con 0x7f97c81e7a10 2024-03-06T07:34:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.622+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97c80d1370 con 0x7f97c81e7a10 2024-03-06T07:34:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.625+0000 7f97bfe00640 1 -- 172.21.15.32:0/2604512384 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+12913 (secure 0 0 0) 0x7f97b4006e20 con 0x7f97c81e7a10 2024-03-06T07:34:00.627 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:00.627 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":87,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:33:58.134799+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6837/3892834285":"2024-03-07T07:33:55.112099+0000","172.21.15.32:6836/3892834285":"2024-03-07T07:33:55.112099+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.627+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97a806b500 msgr2=0x7f97a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.627+0000 7f97cda00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97a806b500 0x7f97a806d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f97b800e1c0 tx=0x7f97b8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.627+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 msgr2=0x7f97c81e7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.627+0000 7f97cda00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 0x7f97c81e7e90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005040 tx=0x7f97b4001120 comp rx=0 tx=0).stop 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 shutdown_connections 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97a806b500 0x7f97a806d9c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c815fd30 0x7f97c81601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c81e7a10 0x7f97c81e7e90 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 --2- 172.21.15.32:0/2604512384 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f97ac004800 0x7f97c81500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 >> 172.21.15.32:0/2604512384 conn(0x7f97c808d2a0 msgr2=0x7f97c80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 shutdown_connections 2024-03-06T07:34:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.628+0000 7f97cda00640 1 -- 172.21.15.32:0/2604512384 wait complete. 2024-03-06T07:34:00.649 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:00.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:00.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.876+0000 7efc0b400640 1 Processor -- start 2024-03-06T07:34:00.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.877+0000 7efc0b400640 1 -- start start 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc0b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc040d4930 0x7efc0415dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc0b400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d0bb0 0x7efc0415e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc0b400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc040cf780 con 0x7efc040d07d0 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc0b400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc040b3770 con 0x7efc040d4930 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc0b400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc040cfa60 con 0x7efc040d0bb0 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc08a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efc040d07d0 0x7efc040b4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48336/0 (socket says 172.21.15.32:48336) 2024-03-06T07:34:00.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.878+0000 7efc08a00640 1 -- 172.21.15.32:0/2146704480 learned_addr learned my addr 172.21.15.32:0/2146704480 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03e00640 1 --2- 172.21.15.32:0/2146704480 >> v2:172.21.15.32:3300/0 conn(0x7efc040d4930 0x7efc0415dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc09400640 1 --2- 172.21.15.32:0/2146704480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d0bb0 0x7efc0415e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03400640 1 -- 172.21.15.32:0/2146704480 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197018379 0 0) 0x7efc040cf780 con 0x7efc040d07d0 2024-03-06T07:34:00.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03400640 1 -- 172.21.15.32:0/2146704480 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe8003520 con 0x7efc040d07d0 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03e00640 1 -- 172.21.15.32:0/2146704480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d0bb0 msgr2=0x7efc0415e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03e00640 1 --2- 172.21.15.32:0/2146704480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d0bb0 0x7efc0415e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03e00640 1 -- 172.21.15.32:0/2146704480 >> v1:172.21.15.32:6789/0 conn(0x7efc040d07d0 legacy=0x7efc040b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.879+0000 7efc03e00640 1 -- 172.21.15.32:0/2146704480 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc040cfd40 con 0x7efc040d4930 2024-03-06T07:34:00.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.881+0000 7efc03e00640 1 --2- 172.21.15.32:0/2146704480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d4930 0x7efc0415dfb0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7efbec0097a0 tx=0x7efbec02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92888f30bcc1d904 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.881+0000 7efc03400640 1 -- 172.21.15.32:0/2146704480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec034070 con 0x7efc040d4930 2024-03-06T07:34:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.881+0000 7efc03400640 1 -- 172.21.15.32:0/2146704480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec030800 con 0x7efc040d4930 2024-03-06T07:34:00.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.881+0000 7efc03400640 1 -- 172.21.15.32:0/2146704480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec030bd0 con 0x7efc040d4930 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 -- 172.21.15.32:0/2146704480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d4930 msgr2=0x7efc0415dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 --2- 172.21.15.32:0/2146704480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d4930 0x7efc0415dfb0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7efbec0097a0 tx=0x7efbec02b940 comp rx=0 tx=0).stop 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 -- 172.21.15.32:0/2146704480 shutdown_connections 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 --2- 172.21.15.32:0/2146704480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d4930 0x7efc0415dfb0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 --2- 172.21.15.32:0/2146704480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d0bb0 0x7efc0415e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 -- 172.21.15.32:0/2146704480 >> 172.21.15.32:0/2146704480 conn(0x7efc040d3e10 msgr2=0x7efc040d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 -- 172.21.15.32:0/2146704480 shutdown_connections 2024-03-06T07:34:00.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.882+0000 7efc0b400640 1 -- 172.21.15.32:0/2146704480 wait complete. 2024-03-06T07:34:00.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.884+0000 7efc0b400640 1 Processor -- start 2024-03-06T07:34:00.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.884+0000 7efc0b400640 1 -- start start 2024-03-06T07:34:00.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc0b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 0x7efc040b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc0b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc040d0bb0 0x7efc040b93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc0b400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d4930 0x7efc041f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc0b400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc040b3fd0 con 0x7efc040d07d0 2024-03-06T07:34:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc0b400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc040b3e50 con 0x7efc040d0bb0 2024-03-06T07:34:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc0b400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc040b4150 con 0x7efc040d4930 2024-03-06T07:34:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc08a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 0x7efc040b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc08a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 0x7efc040b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58584/0 (socket says 172.21.15.32:58584) 2024-03-06T07:34:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.885+0000 7efc08a00640 1 -- 172.21.15.32:0/2373619 learned_addr learned my addr 172.21.15.32:0/2373619 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc09400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d4930 0x7efc041f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc08a00640 1 -- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d4930 msgr2=0x7efc041f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc08a00640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d4930 0x7efc041f4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc08a00640 1 -- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc040d0bb0 msgr2=0x7efc040b93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc08a00640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc040d0bb0 0x7efc040b93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc08a00640 1 -- 172.21.15.32:0/2373619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc040b6b50 con 0x7efc040d07d0 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc08a00640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 0x7efc040b7bf0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c430 tx=0x7efbf400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4017070 con 0x7efc040d07d0 2024-03-06T07:34:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf4005ce0 con 0x7efc040d07d0 2024-03-06T07:34:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40072f0 con 0x7efc040d07d0 2024-03-06T07:34:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc040b6e30 con 0x7efc040d07d0 2024-03-06T07:34:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.886+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc040b73f0 con 0x7efc040d07d0 2024-03-06T07:34:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.888+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0404f330 con 0x7efc040d07d0 2024-03-06T07:34:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.889+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efbf401a070 con 0x7efc040d07d0 2024-03-06T07:34:00.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.890+0000 7efc00c00640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe006b4d0 0x7efbe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.890+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efbf4089350 con 0x7efc040d07d0 2024-03-06T07:34:00.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.891+0000 7efc03e00640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe006b4d0 0x7efbe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.891+0000 7efc03e00640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe006b4d0 0x7efbe006d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efbec036000 tx=0x7efbec030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:00.895+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efbf4058560 con 0x7efc040d07d0 2024-03-06T07:34:01.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.097+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc04160fc0 con 0x7efc040d07d0 2024-03-06T07:34:01.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.099+0000 7efc00c00640 1 -- 172.21.15.32:0/2373619 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7efbf4057e60 con 0x7efc040d07d0 2024-03-06T07:34:01.101 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe006b4d0 msgr2=0x7efbe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe006b4d0 0x7efbe006d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efbec036000 tx=0x7efbec030000 comp rx=0 tx=0).stop 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 msgr2=0x7efc040b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 0x7efc040b7bf0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c430 tx=0x7efbf400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 shutdown_connections 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe006b4d0 0x7efbe006d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc040d4930 0x7efc041f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc040d0bb0 0x7efc040b93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 --2- 172.21.15.32:0/2373619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc040d07d0 0x7efc040b7bf0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.101+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 >> 172.21.15.32:0/2373619 conn(0x7efc040d3e10 msgr2=0x7efc040b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.102+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 shutdown_connections 2024-03-06T07:34:01.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.102+0000 7efc0b400640 1 -- 172.21.15.32:0/2373619 wait complete. 2024-03-06T07:34:01.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:34:01.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.349+0000 7fb18dc00640 1 Processor -- start 2024-03-06T07:34:01.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.350+0000 7fb18dc00640 1 -- start start 2024-03-06T07:34:01.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb18dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb1880ce850 0x7fb1880cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb18dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 0x7fb18806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb18dc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880cdad0 con 0x7fb18815c920 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb18dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb188071500 con 0x7fb1880ce850 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb18dc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880cdd70 con 0x7fb1880cce90 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb187e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb18815c920 0x7fb188070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48338/0 (socket says 172.21.15.32:48338) 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb187e00640 1 -- 172.21.15.32:0/3414762541 learned_addr learned my addr 172.21.15.32:0/3414762541 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb18d200640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 0x7fb18806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb17fe00640 1 --2- 172.21.15.32:0/3414762541 >> v2:172.21.15.39:3300/0 conn(0x7fb1880ce850 0x7fb1880cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.352+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271483378 0 0) 0x7fb1880cdad0 con 0x7fb18815c920 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb16c003520 con 0x7fb18815c920 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2644809592 0 0) 0x7fb16c003520 con 0x7fb18815c920 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 msgr2=0x7fb18806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 0x7fb18806cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 >> v2:172.21.15.39:3300/0 conn(0x7fb1880ce850 msgr2=0x7fb1880cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 --2- 172.21.15.32:0/3414762541 >> v2:172.21.15.39:3300/0 conn(0x7fb1880ce850 0x7fb1880cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880ce050 con 0x7fb18815c920 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb178001670 con 0x7fb18815c920 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb17fe00640 1 --2- 172.21.15.32:0/3414762541 >> v2:172.21.15.39:3300/0 conn(0x7fb1880ce850 0x7fb1880cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 >> v1:172.21.15.32:6789/0 conn(0x7fb18815c920 legacy=0x7fb188070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 0x7fb16c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb16c003f30 0x7fb16c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16c004d40 0x7fb16c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188071500 con 0x7fb16c004d40 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880cdd70 con 0x7fb16c003f30 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c00ba10 con 0x7fb1880cce90 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.353+0000 7fb18d200640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16c004d40 0x7fb16c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb187e00640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 0x7fb16c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb17fe00640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb16c003f30 0x7fb16c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb18d200640 1 -- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 msgr2=0x7fb16c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb18d200640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb1880cce90 0x7fb16c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb18d200640 1 -- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb16c003f30 msgr2=0x7fb16c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb18d200640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb16c003f30 0x7fb16c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb18d200640 1 -- 172.21.15.32:0/3414762541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16c003520 con 0x7fb16c004d40 2024-03-06T07:34:01.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.354+0000 7fb17fe00640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb16c003f30 0x7fb16c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:01.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.355+0000 7fb18d200640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16c004d40 0x7fb16c0051c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fb180009fc0 tx=0x7fb16c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27682b3b323c5bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.355+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180039070 con 0x7fb16c004d40 2024-03-06T07:34:01.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.355+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18002cc20 con 0x7fb16c004d40 2024-03-06T07:34:01.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.355+0000 7fb187400640 1 -- 172.21.15.32:0/3414762541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180009c10 con 0x7fb16c004d40 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18dc00640 1 -- 172.21.15.32:0/3414762541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16c004d40 msgr2=0x7fb16c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18dc00640 1 --2- 172.21.15.32:0/3414762541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16c004d40 0x7fb16c0051c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fb180009fc0 tx=0x7fb16c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18d200640 1 -- 172.21.15.32:0/3414762541 reap_dead start 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18dc00640 1 -- 172.21.15.32:0/3414762541 shutdown_connections 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18dc00640 1 -- 172.21.15.32:0/3414762541 >> 172.21.15.32:0/3414762541 conn(0x7fb18806b3d0 msgr2=0x7fb18806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18dc00640 1 -- 172.21.15.32:0/3414762541 shutdown_connections 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.356+0000 7fb18dc00640 1 -- 172.21.15.32:0/3414762541 wait complete. 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.358+0000 7fb18dc00640 1 Processor -- start 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.358+0000 7fb18dc00640 1 -- start start 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb16c004d40 0x7fb188151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1880b1210 0x7fb1880afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 0x7fb188154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188071150 con 0x7fb1881538e0 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188163790 con 0x7fb1880b1210 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18dc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880b0170 con 0x7fb16c004d40 2024-03-06T07:34:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 0x7fb188154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 0x7fb188154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58602/0 (socket says 172.21.15.32:58602) 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.359+0000 7fb18d200640 1 -- 172.21.15.32:0/1455468925 learned_addr learned my addr 172.21.15.32:0/1455468925 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb187e00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb16c004d40 0x7fb188151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb18d200640 1 -- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb16c004d40 msgr2=0x7fb188151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb17fe00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1880b1210 0x7fb1880afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb18d200640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb16c004d40 0x7fb188151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb18d200640 1 -- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1880b1210 msgr2=0x7fb1880afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb18d200640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1880b1210 0x7fb1880afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb18d200640 1 -- 172.21.15.32:0/1455468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880b0450 con 0x7fb1881538e0 2024-03-06T07:34:01.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.360+0000 7fb18d200640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 0x7fb188154090 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb18002c690 tx=0x7fb180009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.361+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180045070 con 0x7fb1881538e0 2024-03-06T07:34:01.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.361+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18003a040 con 0x7fb1881538e0 2024-03-06T07:34:01.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.361+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180039070 con 0x7fb1881538e0 2024-03-06T07:34:01.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.361+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1880b0730 con 0x7fb1881538e0 2024-03-06T07:34:01.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.361+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb188153d90 con 0x7fb1881538e0 2024-03-06T07:34:01.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.362+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18804f330 con 0x7fb1881538e0 2024-03-06T07:34:01.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.363+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb180003bb0 con 0x7fb1881538e0 2024-03-06T07:34:01.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.364+0000 7fb184c00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb16406b500 0x7fb16406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.364+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb1800ab810 con 0x7fb1881538e0 2024-03-06T07:34:01.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.364+0000 7fb187e00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb16406b500 0x7fb16406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.365+0000 7fb187e00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb16406b500 0x7fb16406d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb178000fb0 tx=0x7fb178009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.369+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb18007a2a0 con 0x7fb1881538e0 2024-03-06T07:34:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:01.645+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fb1880cf3a0 con 0x7fb1881538e0 2024-03-06T07:34:02.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.165+0000 7fb184c00640 1 -- 172.21.15.32:0/1455468925 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v284) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb18007e340 con 0x7fb1881538e0 2024-03-06T07:34:02.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.170+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb16406b500 msgr2=0x7fb16406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.170+0000 7fb18dc00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb16406b500 0x7fb16406d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb178000fb0 tx=0x7fb178009040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.170+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 msgr2=0x7fb188154090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.170+0000 7fb18dc00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 0x7fb188154090 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb18002c690 tx=0x7fb180009750 comp rx=0 tx=0).stop 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.170+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 shutdown_connections 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb16406b500 0x7fb16406d9c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1881538e0 0x7fb188154090 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1880b1210 0x7fb1880afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 --2- 172.21.15.32:0/1455468925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb16c004d40 0x7fb188151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 >> 172.21.15.32:0/1455468925 conn(0x7fb18806b3d0 msgr2=0x7fb18815a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 shutdown_connections 2024-03-06T07:34:02.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.171+0000 7fb18dc00640 1 -- 172.21.15.32:0/1455468925 wait complete. 2024-03-06T07:34:02.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.421+0000 7fd5e0800640 1 Processor -- start 2024-03-06T07:34:02.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.422+0000 7fd5e0800640 1 -- start start 2024-03-06T07:34:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.423+0000 7fd5e0800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5d8162bb0 0x7fd5d8163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.424+0000 7fd5e0800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d8163950 con 0x7fd5d80adb00 2024-03-06T07:34:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.424+0000 7fd5e0800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d80acd40 con 0x7fd5d80ccad0 2024-03-06T07:34:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.424+0000 7fd5e0800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d80ad020 con 0x7fd5d8162bb0 2024-03-06T07:34:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.424+0000 7fd5de800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5d80adb00 0x7fd5d80adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48350/0 (socket says 172.21.15.32:48350) 2024-03-06T07:34:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.424+0000 7fd5de800640 1 -- 172.21.15.32:0/414082171 learned_addr learned my addr 172.21.15.32:0/414082171 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.424+0000 7fd5dd400640 1 --2- 172.21.15.32:0/414082171 >> v2:172.21.15.39:3300/0 conn(0x7fd5d8162bb0 0x7fd5d8163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842176519 0 0) 0x7fd5d8163950 con 0x7fd5d80adb00 2024-03-06T07:34:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c0003520 con 0x7fd5d80adb00 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1596098501 0 0) 0x7fd5c0003520 con 0x7fd5d80adb00 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 >> v2:172.21.15.39:3300/0 conn(0x7fd5d8162bb0 msgr2=0x7fd5d8163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 --2- 172.21.15.32:0/414082171 >> v2:172.21.15.39:3300/0 conn(0x7fd5d8162bb0 0x7fd5d8163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 >> v1:172.21.15.39:6789/0 conn(0x7fd5d80ccad0 legacy=0x7fd5d80cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d80ad300 con 0x7fd5d80adb00 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd5d40019f0 con 0x7fd5d80adb00 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 >> v1:172.21.15.32:6789/0 conn(0x7fd5d80adb00 legacy=0x7fd5d80adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5c0003a00 0x7fd5c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5c0004800 0x7fd5c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c00051c0 0x7fd5c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80ad020 con 0x7fd5c0004800 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c000c0c0 con 0x7fd5c0003a00 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.425+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c000c2c0 con 0x7fd5c00051c0 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dd400640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5c0004800 0x7fd5c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dde00640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5c0003a00 0x7fd5c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5de800640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c00051c0 0x7fd5c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dd400640 1 -- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c00051c0 msgr2=0x7fd5c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dd400640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c00051c0 0x7fd5c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dd400640 1 -- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5c0003a00 msgr2=0x7fd5c0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dd400640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5c0003a00 0x7fd5c0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dd400640 1 -- 172.21.15.32:0/414082171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c0003520 con 0x7fd5c0004800 2024-03-06T07:34:02.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5dde00640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5c0003a00 0x7fd5c0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5de800640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c00051c0 0x7fd5c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.426+0000 7fd5de800640 1 -- 172.21.15.32:0/414082171 reap_dead start 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.427+0000 7fd5dd400640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5c0004800 0x7fd5c0004c80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009ac0 tx=0x7fd5c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe11c7f48aae114 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8039070 con 0x7fd5c0004800 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c802c6b0 con 0x7fd5c0004800 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5dca00640 1 -- 172.21.15.32:0/414082171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8009690 con 0x7fd5c0004800 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 -- 172.21.15.32:0/414082171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5c0004800 msgr2=0x7fd5c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5c0004800 0x7fd5c0004c80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009ac0 tx=0x7fd5c0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 -- 172.21.15.32:0/414082171 shutdown_connections 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 --2- 172.21.15.32:0/414082171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5c0004800 0x7fd5c0004c80 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 -- 172.21.15.32:0/414082171 >> 172.21.15.32:0/414082171 conn(0x7fd5d806b3d0 msgr2=0x7fd5d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 -- 172.21.15.32:0/414082171 shutdown_connections 2024-03-06T07:34:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.428+0000 7fd5e0800640 1 -- 172.21.15.32:0/414082171 wait complete. 2024-03-06T07:34:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.430+0000 7fd5e0800640 1 Processor -- start 2024-03-06T07:34:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.430+0000 7fd5e0800640 1 -- start start 2024-03-06T07:34:02.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.431+0000 7fd5e0800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c0004800 0x7fd5d814ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.431+0000 7fd5e0800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d8151900 0x7fd5d8151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.431+0000 7fd5e0800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 0x7fd5d8160b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.431+0000 7fd5e0800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d806c980 con 0x7fd5d81606b0 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.431+0000 7fd5e0800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d806e7d0 con 0x7fd5d8151900 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.431+0000 7fd5e0800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80cceb0 con 0x7fd5c0004800 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 0x7fd5d8160b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 0x7fd5d8160b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58624/0 (socket says 172.21.15.32:58624) 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 -- 172.21.15.32:0/814603616 learned_addr learned my addr 172.21.15.32:0/814603616 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5dde00640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c0004800 0x7fd5d814ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5dd400640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d8151900 0x7fd5d8151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 -- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c0004800 msgr2=0x7fd5d814ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c0004800 0x7fd5d814ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 -- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d8151900 msgr2=0x7fd5d8151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d8151900 0x7fd5d8151d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 -- 172.21.15.32:0/814603616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d81523c0 con 0x7fd5d81606b0 2024-03-06T07:34:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5dd400640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d8151900 0x7fd5d8151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.432+0000 7fd5de800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 0x7fd5d8160b30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80cd7b0 tx=0x7fd5d400b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.433+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400b900 con 0x7fd5d81606b0 2024-03-06T07:34:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.433+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d400baa0 con 0x7fd5d81606b0 2024-03-06T07:34:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.433+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400be70 con 0x7fd5d81606b0 2024-03-06T07:34:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.433+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d81526a0 con 0x7fd5d81606b0 2024-03-06T07:34:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.433+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d814fd60 con 0x7fd5d81606b0 2024-03-06T07:34:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.434+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d814f870 con 0x7fd5d81606b0 2024-03-06T07:34:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.435+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5d401b020 con 0x7fd5d81606b0 2024-03-06T07:34:02.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.436+0000 7fd5d2000640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5bc06b500 0x7fd5bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.436+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd5d4089fe0 con 0x7fd5d81606b0 2024-03-06T07:34:02.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.436+0000 7fd5dde00640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5bc06b500 0x7fd5bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.437+0000 7fd5dde00640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5bc06b500 0x7fd5bc06d9c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80cc5b0 tx=0x7fd5cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.441+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5d40591e0 con 0x7fd5d81606b0 2024-03-06T07:34:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.643+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd5d804f330 con 0x7fd5d81606b0 2024-03-06T07:34:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.649+0000 7fd5d2000640 1 -- 172.21.15.32:0/814603616 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd5d4058ae0 con 0x7fd5d81606b0 2024-03-06T07:34:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5bc06b500 msgr2=0x7fd5bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5bc06b500 0x7fd5bc06d9c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80cc5b0 tx=0x7fd5cc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 msgr2=0x7fd5d8160b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 0x7fd5d8160b30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80cd7b0 tx=0x7fd5d400b0d0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 shutdown_connections 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd5bc06b500 0x7fd5bc06d9c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5d81606b0 0x7fd5d8160b30 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d8151900 0x7fd5d8151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 --2- 172.21.15.32:0/814603616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5c0004800 0x7fd5d814ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 >> 172.21.15.32:0/814603616 conn(0x7fd5d806b3d0 msgr2=0x7fd5d806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 shutdown_connections 2024-03-06T07:34:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:02.652+0000 7fd5e0800640 1 -- 172.21.15.32:0/814603616 wait complete. 2024-03-06T07:34:02.674 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:34:02.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:02.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:02.928 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:34:02.928 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:34:02.928 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:02.988 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:02.988 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:34:02.988 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:03.048 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:03.048 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:34:03.049 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:03.107 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:03.107 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:34:03.107 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:03.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:03.167 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:34:03.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:03.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.398+0000 7f820ca00640 1 Processor -- start 2024-03-06T07:34:03.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.399+0000 7f820ca00640 1 -- start start 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.400+0000 7f820ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82080cf5c0 0x7f82080cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.400+0000 7f820ca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82080b4860 con 0x7f82080b3c90 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.400+0000 7f820ca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82080b4b40 con 0x7f8208071050 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.400+0000 7f820ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82080ce160 con 0x7f82080cf5c0 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8205e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82080b3c90 0x7f82080b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48366/0 (socket says 172.21.15.32:48366) 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8205e00640 1 -- 172.21.15.32:0/2142522886 learned_addr learned my addr 172.21.15.32:0/2142522886 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8205400640 1 --2- 172.21.15.32:0/2142522886 >> v2:172.21.15.32:3300/0 conn(0x7f82080cf5c0 0x7f82080cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8204a00640 1 -- 172.21.15.32:0/2142522886 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018232314 0 0) 0x7f82080b4860 con 0x7f82080b3c90 2024-03-06T07:34:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8204a00640 1 -- 172.21.15.32:0/2142522886 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ec005730 con 0x7f82080b3c90 2024-03-06T07:34:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8205400640 1 -- 172.21.15.32:0/2142522886 >> v1:172.21.15.39:6789/0 conn(0x7f8208071050 legacy=0x7f8208071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8205400640 1 -- 172.21.15.32:0/2142522886 >> v1:172.21.15.32:6789/0 conn(0x7f82080b3c90 legacy=0x7f82080b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.401+0000 7f8205400640 1 -- 172.21.15.32:0/2142522886 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080b4e20 con 0x7f82080cf5c0 2024-03-06T07:34:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.403+0000 7f8205400640 1 --2- 172.21.15.32:0/2142522886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82080cf5c0 0x7f82080cdc20 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f81f40097a0 tx=0x7f81f402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4d02794c894e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.403+0000 7f8204a00640 1 -- 172.21.15.32:0/2142522886 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f402f810 con 0x7f82080cf5c0 2024-03-06T07:34:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.403+0000 7f8204a00640 1 -- 172.21.15.32:0/2142522886 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f402f9b0 con 0x7f82080cf5c0 2024-03-06T07:34:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.403+0000 7f8204a00640 1 -- 172.21.15.32:0/2142522886 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f402fd80 con 0x7f82080cf5c0 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 -- 172.21.15.32:0/2142522886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82080cf5c0 msgr2=0x7f82080cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 --2- 172.21.15.32:0/2142522886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82080cf5c0 0x7f82080cdc20 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f81f40097a0 tx=0x7f81f402b9f0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 -- 172.21.15.32:0/2142522886 shutdown_connections 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 --2- 172.21.15.32:0/2142522886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82080cf5c0 0x7f82080cdc20 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 -- 172.21.15.32:0/2142522886 >> 172.21.15.32:0/2142522886 conn(0x7f820806afb0 msgr2=0x7f820806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 -- 172.21.15.32:0/2142522886 shutdown_connections 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.404+0000 7f820ca00640 1 -- 172.21.15.32:0/2142522886 wait complete. 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.406+0000 7f820ca00640 1 Processor -- start 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.406+0000 7f820ca00640 1 -- start start 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f820ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8208071050 0x7f82081f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f820ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82080b3c90 0x7f820809d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f820ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 0x7f82080ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f820ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208071ba0 con 0x7f8208151f40 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f820ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820815b340 con 0x7f82080b3c90 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f820ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8208071d20 con 0x7f8208071050 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 0x7f82080ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 0x7f82080ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58648/0 (socket says 172.21.15.32:58648) 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 -- 172.21.15.32:0/637856740 learned_addr learned my addr 172.21.15.32:0/637856740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 -- 172.21.15.32:0/637856740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8208071050 msgr2=0x7f82081f7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8208071050 0x7f82081f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 -- 172.21.15.32:0/637856740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82080b3c90 msgr2=0x7f820809d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82080b3c90 0x7f820809d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.407+0000 7f8206800640 1 -- 172.21.15.32:0/637856740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82081f7820 con 0x7f8208151f40 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.408+0000 7f8206800640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 0x7f82080ce350 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f820000c7a0 tx=0x7f820000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.408+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200003b70 con 0x7f8208151f40 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.408+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8200003d10 con 0x7f8208151f40 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.408+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000aa90 con 0x7f8208151f40 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.408+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82081f7b00 con 0x7f8208151f40 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.408+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820809aff0 con 0x7f8208151f40 2024-03-06T07:34:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.409+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82080cb3a0 con 0x7f8208151f40 2024-03-06T07:34:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.410+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f820000ac30 con 0x7f8208151f40 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.411+0000 7f81fe000640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81e806b4d0 0x7f81e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.411+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8200089e70 con 0x7f8208151f40 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.412+0000 7f8205e00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81e806b4d0 0x7f81e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.412+0000 7f8205e00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81e806b4d0 0x7f81e806d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f81f8001570 tx=0x7f81f8007930 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.416+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8200058fe0 con 0x7f8208151f40 2024-03-06T07:34:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.654+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8208153120 con 0x7f8208151f40 2024-03-06T07:34:03.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.655+0000 7f81fe000640 1 -- 172.21.15.32:0/637856740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f82000588e0 con 0x7f8208151f40 2024-03-06T07:34:03.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:03.658 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3391815278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3391815278},{"type":"v1","addr":"172.21.15.39:6833","nonce":3391815278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5695,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1677974397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1677974397},{"type":"v1","addr":"172.21.15.39:6835","nonce":1677974397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1658151472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1658151472},{"type":"v1","addr":"172.21.15.32:6835","nonce":1658151472}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2825192366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2825192366},{"type":"v1","addr":"172.21.15.32:6837","nonce":2825192366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[]} 2024-03-06T07:34:03.658 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 54 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81e806b4d0 msgr2=0x7f81e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81e806b4d0 0x7f81e806d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f81f8001570 tx=0x7f81f8007930 comp rx=0 tx=0).stop 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 msgr2=0x7f82080ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 0x7f82080ce350 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f820000c7a0 tx=0x7f820000cc60 comp rx=0 tx=0).stop 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 shutdown_connections 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f81e806b4d0 0x7f81e806d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8208151f40 0x7f82080ce350 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82080b3c90 0x7f820809d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 --2- 172.21.15.32:0/637856740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8208071050 0x7f82081f7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.658+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 >> 172.21.15.32:0/637856740 conn(0x7f820806afb0 msgr2=0x7f8208156780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.659+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 shutdown_connections 2024-03-06T07:34:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.659+0000 7f820ca00640 1 -- 172.21.15.32:0/637856740 wait complete. 2024-03-06T07:34:03.681 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:34:03.681 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:34:03.682 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:03.682 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:03.682 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:34:03.682 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:34:03.682 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:03.682 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:03.682 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:34:03.682 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:34:03.683 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:03.683 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:03.683 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:34:03.683 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:34:03.683 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:34:03.683 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:03.700 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:03.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:34:03.705 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:03.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:34:03.706 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:03.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:34:03.707 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:34:03.707 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:34:03.750 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:34:03.750 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:03.754 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:34:03.754 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:03.756 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:34:03.757 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:03.760 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:34:03.760 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:03.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:34:03.908 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:34:03.908 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:34:03.910 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-06T07:34:03.908+0000 7f7299c00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:34:03.920 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:34:03.921 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.939+0000 7fc59a400640 1 Processor -- start 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.940+0000 7fc59a400640 1 -- start start 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc59a400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc59415e070 0x7fc5940d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc59a400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5940d18c0 con 0x7fc5940d0cd0 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc59a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5940d1ba0 con 0x7fc5940cf410 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc59a400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5940d1e80 con 0x7fc59415e070 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc599a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc5940d0cd0 0x7fc5940b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48376/0 (socket says 172.21.15.32:48376) 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc599a00640 1 -- 172.21.15.32:0/2811638179 learned_addr learned my addr 172.21.15.32:0/2811638179 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.941+0000 7fc599000640 1 --2- 172.21.15.32:0/2811638179 >> v2:172.21.15.39:3300/0 conn(0x7fc59415e070 0x7fc5940d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278045630 0 0) 0x7fc5940d18c0 con 0x7fc5940d0cd0 2024-03-06T07:34:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc57c003520 con 0x7fc5940d0cd0 2024-03-06T07:34:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060549376 0 0) 0x7fc5940d1ba0 con 0x7fc5940cf410 2024-03-06T07:34:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5940d18c0 con 0x7fc5940cf410 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc599000640 1 -- 172.21.15.32:0/2811638179 >> v1:172.21.15.39:6789/0 conn(0x7fc5940cf410 legacy=0x7fc5940d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc599000640 1 -- 172.21.15.32:0/2811638179 >> v1:172.21.15.32:6789/0 conn(0x7fc5940d0cd0 legacy=0x7fc5940b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.942+0000 7fc599000640 1 -- 172.21.15.32:0/2811638179 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940d2160 con 0x7fc59415e070 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.943+0000 7fc599000640 1 --2- 172.21.15.32:0/2811638179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc59415e070 0x7fc5940d2960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc5880097a0 tx=0x7fc58802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf890c91ff0f29dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.943+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588035020 con 0x7fc59415e070 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.943+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc588031830 con 0x7fc59415e070 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.943+0000 7fc593400640 1 -- 172.21.15.32:0/2811638179 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588031c00 con 0x7fc59415e070 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 -- 172.21.15.32:0/2811638179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc59415e070 msgr2=0x7fc5940d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 --2- 172.21.15.32:0/2811638179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc59415e070 0x7fc5940d2960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc5880097a0 tx=0x7fc58802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 -- 172.21.15.32:0/2811638179 shutdown_connections 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 --2- 172.21.15.32:0/2811638179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc59415e070 0x7fc5940d2960 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 -- 172.21.15.32:0/2811638179 >> 172.21.15.32:0/2811638179 conn(0x7fc59408d2a0 msgr2=0x7fc5940d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 -- 172.21.15.32:0/2811638179 shutdown_connections 2024-03-06T07:34:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.944+0000 7fc59a400640 1 -- 172.21.15.32:0/2811638179 wait complete. 2024-03-06T07:34:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.945+0000 7fc59a400640 1 Processor -- start 2024-03-06T07:34:03.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.945+0000 7fc59a400640 1 -- start start 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.946+0000 7fc59a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 0x7fc594157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.946+0000 7fc59a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5940d0cd0 0x7fc5940c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.946+0000 7fc59a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc59415e070 0x7fc5940c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.946+0000 7fc59a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940d4730 con 0x7fc5940cf410 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.946+0000 7fc59a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940d45b0 con 0x7fc5940d0cd0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.946+0000 7fc59a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940cf870 con 0x7fc59415e070 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 0x7fc594157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 0x7fc594157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58696/0 (socket says 172.21.15.32:58696) 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 -- 172.21.15.32:0/3351528336 learned_addr learned my addr 172.21.15.32:0/3351528336 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599a00640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc59415e070 0x7fc5940c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc593e00640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5940d0cd0 0x7fc5940c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 -- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc59415e070 msgr2=0x7fc5940c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc59415e070 0x7fc5940c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 -- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5940d0cd0 msgr2=0x7fc5940c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5940d0cd0 0x7fc5940c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 -- 172.21.15.32:0/3351528336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940c7a30 con 0x7fc5940cf410 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.947+0000 7fc599000640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 0x7fc594157fa0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fc588000c00 tx=0x7fc5880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.948+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588040070 con 0x7fc5940cf410 2024-03-06T07:34:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.948+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58803a930 con 0x7fc5940cf410 2024-03-06T07:34:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.948+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5940c7d10 con 0x7fc5940cf410 2024-03-06T07:34:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.948+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588035040 con 0x7fc5940cf410 2024-03-06T07:34:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.948+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59415fd00 con 0x7fc5940cf410 2024-03-06T07:34:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.949+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59404f330 con 0x7fc5940cf410 2024-03-06T07:34:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.950+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc58804b060 con 0x7fc5940cf410 2024-03-06T07:34:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.951+0000 7fc590c00640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc57406b4d0 0x7fc57406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.951+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc5880ab6d0 con 0x7fc5940cf410 2024-03-06T07:34:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.951+0000 7fc593e00640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc57406b4d0 0x7fc57406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.952+0000 7fc593e00640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc57406b4d0 0x7fc57406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5941f7600 tx=0x7fc580009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:03.955+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc58807a0d0 con 0x7fc5940cf410 2024-03-06T07:34:04.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.166+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc5940b9b70 con 0x7fc5940cf410 2024-03-06T07:34:04.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.199+0000 7fc590c00640 1 -- 172.21.15.32:0/3351528336 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc58807e170 con 0x7fc5940cf410 2024-03-06T07:34:04.201 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:34:04.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.201+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc57406b4d0 msgr2=0x7fc57406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc57406b4d0 0x7fc57406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5941f7600 tx=0x7fc580009040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 msgr2=0x7fc594157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 0x7fc594157fa0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fc588000c00 tx=0x7fc5880046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 shutdown_connections 2024-03-06T07:34:04.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc57406b4d0 0x7fc57406d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc59415e070 0x7fc5940c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5940d0cd0 0x7fc5940c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 --2- 172.21.15.32:0/3351528336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5940cf410 0x7fc594157fa0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 >> 172.21.15.32:0/3351528336 conn(0x7fc59408d2a0 msgr2=0x7fc5940b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 shutdown_connections 2024-03-06T07:34:04.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.202+0000 7fc59a400640 1 -- 172.21.15.32:0/3351528336 wait complete. 2024-03-06T07:34:04.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:34:04.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.418+0000 7f94f9a00640 1 Processor -- start 2024-03-06T07:34:04.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.419+0000 7f94f9a00640 1 -- start start 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.420+0000 7f94f9a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.420+0000 7f94f9a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94f40b50e0 0x7f94f40d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.420+0000 7f94f9a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40b3940 con 0x7f94f40d0d70 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.420+0000 7f94f9a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40d4e70 con 0x7f94f40b50e0 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.420+0000 7f94f9a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40b3c40 con 0x7f94f415e2e0 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94f40b50e0 0x7f94f40d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94f40b50e0 0x7f94f40d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58706/0 (socket says 172.21.15.32:58706) 2024-03-06T07:34:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 -- 172.21.15.32:0/838161352 learned_addr learned my addr 172.21.15.32:0/838161352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f2a00640 1 --2- 172.21.15.32:0/838161352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 -- 172.21.15.32:0/838161352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 msgr2=0x7f94f40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 --2- 172.21.15.32:0/838161352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f40d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 -- 172.21.15.32:0/838161352 >> v1:172.21.15.39:6789/0 conn(0x7f94f40d0d70 legacy=0x7f94f40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.421+0000 7f94f3400640 1 -- 172.21.15.32:0/838161352 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40cf580 con 0x7f94f40b50e0 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.422+0000 7f94f3400640 1 --2- 172.21.15.32:0/838161352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f40d11d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009870 tx=0x7f94ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f53106f4f6c92fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f1600640 1 -- 172.21.15.32:0/838161352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec035020 con 0x7f94f40b50e0 2024-03-06T07:34:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f1600640 1 -- 172.21.15.32:0/838161352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec031640 con 0x7f94f40b50e0 2024-03-06T07:34:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f1600640 1 -- 172.21.15.32:0/838161352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec02d040 con 0x7f94f40b50e0 2024-03-06T07:34:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 -- 172.21.15.32:0/838161352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 msgr2=0x7f94f40d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 --2- 172.21.15.32:0/838161352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f40d11d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009870 tx=0x7f94ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 -- 172.21.15.32:0/838161352 shutdown_connections 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 --2- 172.21.15.32:0/838161352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f40d11d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 --2- 172.21.15.32:0/838161352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f40d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 -- 172.21.15.32:0/838161352 >> 172.21.15.32:0/838161352 conn(0x7f94f40d3f60 msgr2=0x7f94f40d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.423+0000 7f94f9a00640 1 -- 172.21.15.32:0/838161352 shutdown_connections 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.424+0000 7f94f9a00640 1 -- 172.21.15.32:0/838161352 wait complete. 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.425+0000 7f94f9a00640 1 Processor -- start 2024-03-06T07:34:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 -- start start 2024-03-06T07:34:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f415e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f40d0d70 0x7f94f4161390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f415acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40cfe20 con 0x7f94f40b50e0 2024-03-06T07:34:04.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40cfb40 con 0x7f94f40d0d70 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.426+0000 7f94f9a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40d2240 con 0x7f94f415e2e0 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f415e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f415e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58718/0 (socket says 172.21.15.32:58718) 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 -- 172.21.15.32:0/4119247340 learned_addr learned my addr 172.21.15.32:0/4119247340 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 -- 172.21.15.32:0/4119247340 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 msgr2=0x7f94f415acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f415acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 -- 172.21.15.32:0/4119247340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f40d0d70 msgr2=0x7f94f4161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f40d0d70 0x7f94f4161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.427+0000 7f94f2a00640 1 -- 172.21.15.32:0/4119247340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40b6f20 con 0x7f94f40b50e0 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.428+0000 7f94f2a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f415e9a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f94e400c7f0 tx=0x7f94e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.428+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4015070 con 0x7f94f40b50e0 2024-03-06T07:34:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.428+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e4004220 con 0x7f94f40b50e0 2024-03-06T07:34:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.428+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4007620 con 0x7f94f40b50e0 2024-03-06T07:34:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.428+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f40b7200 con 0x7f94f40b50e0 2024-03-06T07:34:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.428+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f40b77c0 con 0x7f94f40b50e0 2024-03-06T07:34:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.429+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f40b7bf0 con 0x7f94f40b50e0 2024-03-06T07:34:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.430+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94e40079d0 con 0x7f94f40b50e0 2024-03-06T07:34:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.431+0000 7f94eaa00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b500 0x7f94d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.431+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f94e4089ea0 con 0x7f94f40b50e0 2024-03-06T07:34:04.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.431+0000 7f94f2000640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b500 0x7f94d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.432+0000 7f94f2000640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b500 0x7f94d006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f94f415e770 tx=0x7f94dc007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.436+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94e4059070 con 0x7f94f40b50e0 2024-03-06T07:34:04.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.697+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f94f404f3a0 con 0x7f94f40b50e0 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.699+0000 7f94eaa00640 1 -- 172.21.15.32:0/4119247340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f94e4058970 con 0x7f94f40b50e0 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:34:04.701 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:34:04.702 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:34:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.704 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.710 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:34:04.711 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:34:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:34:04.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:34:04.714 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b500 msgr2=0x7f94d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b500 0x7f94d006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f94f415e770 tx=0x7f94dc007d30 comp rx=0 tx=0).stop 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 msgr2=0x7f94f415e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f415e9a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f94e400c7f0 tx=0x7f94e400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 shutdown_connections 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94d006b500 0x7f94d006d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94f415e2e0 0x7f94f415acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94f40d0d70 0x7f94f4161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 --2- 172.21.15.32:0/4119247340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40b50e0 0x7f94f415e9a0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 >> 172.21.15.32:0/4119247340 conn(0x7f94f40d3f60 msgr2=0x7f94f40b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 shutdown_connections 2024-03-06T07:34:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.701+0000 7f94f9a00640 1 -- 172.21.15.32:0/4119247340 wait complete. 2024-03-06T07:34:04.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.922+0000 7f0ce5000640 1 Processor -- start 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.923+0000 7f0ce5000640 1 -- start start 2024-03-06T07:34:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0ce5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ce00d2960 0x7f0ce00d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0ce5000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ce00d0b00 0x7f0ce00b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0ce5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00d2dc0 con 0x7f0ce015e070 2024-03-06T07:34:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0ce5000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00d0f70 con 0x7f0ce00d0b00 2024-03-06T07:34:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0ce5000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00d18c0 con 0x7f0ce00d2960 2024-03-06T07:34:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0cdfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ce00d0b00 0x7f0ce00b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0cdfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ce00d0b00 0x7f0ce00b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58722/0 (socket says 172.21.15.32:58722) 2024-03-06T07:34:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.924+0000 7f0cdfe00640 1 -- 172.21.15.32:0/75555873 learned_addr learned my addr 172.21.15.32:0/75555873 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.925+0000 7f0cdea00640 1 --2- 172.21.15.32:0/75555873 >> v2:172.21.15.39:3300/0 conn(0x7f0ce00d2960 0x7f0ce00d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.925+0000 7f0cdfe00640 1 -- 172.21.15.32:0/75555873 >> v2:172.21.15.39:3300/0 conn(0x7f0ce00d2960 msgr2=0x7f0ce00d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.925+0000 7f0cdfe00640 1 --2- 172.21.15.32:0/75555873 >> v2:172.21.15.39:3300/0 conn(0x7f0ce00d2960 0x7f0ce00d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.925+0000 7f0cdfe00640 1 -- 172.21.15.32:0/75555873 >> v1:172.21.15.39:6789/0 conn(0x7f0ce015e070 legacy=0x7f0ce00cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.925+0000 7f0cdfe00640 1 -- 172.21.15.32:0/75555873 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00d1ba0 con 0x7f0ce00d0b00 2024-03-06T07:34:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.926+0000 7f0cdfe00640 1 --2- 172.21.15.32:0/75555873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00b3630 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00b6880 tx=0x7f0cd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8121be859dcef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.926+0000 7f0cde000640 1 -- 172.21.15.32:0/75555873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8035020 con 0x7f0ce00d0b00 2024-03-06T07:34:04.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.926+0000 7f0cde000640 1 -- 172.21.15.32:0/75555873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd8031870 con 0x7f0ce00d0b00 2024-03-06T07:34:04.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0cde000640 1 -- 172.21.15.32:0/75555873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8031c40 con 0x7f0ce00d0b00 2024-03-06T07:34:04.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 -- 172.21.15.32:0/75555873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce00d0b00 msgr2=0x7f0ce00b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 --2- 172.21.15.32:0/75555873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00b3630 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00b6880 tx=0x7f0cd802b910 comp rx=0 tx=0).stop 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 -- 172.21.15.32:0/75555873 shutdown_connections 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 --2- 172.21.15.32:0/75555873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00b3630 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 --2- 172.21.15.32:0/75555873 >> v2:172.21.15.39:3300/0 conn(0x7f0ce00d2960 0x7f0ce00d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 -- 172.21.15.32:0/75555873 >> 172.21.15.32:0/75555873 conn(0x7f0ce008d2a0 msgr2=0x7f0ce00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 -- 172.21.15.32:0/75555873 shutdown_connections 2024-03-06T07:34:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.927+0000 7f0ce5000640 1 -- 172.21.15.32:0/75555873 wait complete. 2024-03-06T07:34:04.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.929+0000 7f0ce5000640 1 Processor -- start 2024-03-06T07:34:04.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.929+0000 7f0ce5000640 1 -- start start 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0ce5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0ce5000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ce00d2960 0x7f0ce00b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0ce5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 0x7f0ce0160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0ce5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00d2440 con 0x7f0ce015e070 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0ce5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00d2160 con 0x7f0ce00d0b00 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0ce5000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00d45b0 con 0x7f0ce00d2960 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 0x7f0ce0160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 0x7f0ce0160660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58732/0 (socket says 172.21.15.32:58732) 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.930+0000 7f0cdfe00640 1 -- 172.21.15.32:0/393886094 learned_addr learned my addr 172.21.15.32:0/393886094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdfe00640 1 -- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ce00d2960 msgr2=0x7f0ce00b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdfe00640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ce00d2960 0x7f0ce00b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdf400640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdfe00640 1 -- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce00d0b00 msgr2=0x7f0ce00cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdfe00640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdfe00640 1 -- 172.21.15.32:0/393886094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00b7cd0 con 0x7f0ce015e070 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdf400640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cdfe00640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 0x7f0ce0160660 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009960 tx=0x7f0cd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8040070 con 0x7f0ce015e070 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd8030c50 con 0x7f0ce015e070 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8035040 con 0x7f0ce015e070 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.931+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce00cae60 con 0x7f0ce015e070 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.932+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce00cb420 con 0x7f0ce015e070 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.933+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce004f330 con 0x7f0ce015e070 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.934+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cd8038020 con 0x7f0ce015e070 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.934+0000 7f0cd7400640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0cbc06b4d0 0x7f0cbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.935+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cd80abdc0 con 0x7f0ce015e070 2024-03-06T07:34:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.935+0000 7f0cdf400640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0cbc06b4d0 0x7f0cbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.935+0000 7f0cdf400640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0cbc06b4d0 0x7f0cbc06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0007e90 tx=0x7f0cd00073a0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:04.939+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cd807aef0 con 0x7f0ce015e070 2024-03-06T07:34:05.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.139+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ce00d1370 con 0x7f0ce015e070 2024-03-06T07:34:05.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.140+0000 7f0cd7400640 1 -- 172.21.15.32:0/393886094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+12784 (secure 0 0 0) 0x7f0cd807a7f0 con 0x7f0ce015e070 2024-03-06T07:34:05.143 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:05.143 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:04.210387+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0cbc06b4d0 msgr2=0x7f0cbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0cbc06b4d0 0x7f0cbc06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0007e90 tx=0x7f0cd00073a0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 msgr2=0x7f0ce0160660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 0x7f0ce0160660 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009960 tx=0x7f0cd80046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 shutdown_connections 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0cbc06b4d0 0x7f0cbc06d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ce015e070 0x7f0ce0160660 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ce00d2960 0x7f0ce00b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 --2- 172.21.15.32:0/393886094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce00d0b00 0x7f0ce00cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.143+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 >> 172.21.15.32:0/393886094 conn(0x7f0ce008d2a0 msgr2=0x7f0ce015d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.144+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 shutdown_connections 2024-03-06T07:34:05.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.144+0000 7f0ce5000640 1 -- 172.21.15.32:0/393886094 wait complete. 2024-03-06T07:34:05.165 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:05.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:05.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.384+0000 7f022fe00640 1 Processor -- start 2024-03-06T07:34:05.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.385+0000 7f022fe00640 1 -- start start 2024-03-06T07:34:05.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.386+0000 7f022fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0230145430 0x7f023015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.386+0000 7f022fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023015fe40 con 0x7f023015cc90 2024-03-06T07:34:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.386+0000 7f022fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0230160120 con 0x7f02300b2250 2024-03-06T07:34:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.386+0000 7f022fe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0230160400 con 0x7f0230145430 2024-03-06T07:34:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022e000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f023015cc90 0x7f02300b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48386/0 (socket says 172.21.15.32:48386) 2024-03-06T07:34:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0230145430 0x7f023015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022e000640 1 -- 172.21.15.32:0/3904202505 learned_addr learned my addr 172.21.15.32:0/3904202505 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:05.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022ea00640 1 -- 172.21.15.32:0/3904202505 >> v1:172.21.15.39:6789/0 conn(0x7f02300b2250 legacy=0x7f02300b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022ea00640 1 -- 172.21.15.32:0/3904202505 >> v1:172.21.15.32:6789/0 conn(0x7f023015cc90 legacy=0x7f02300b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022ea00640 1 -- 172.21.15.32:0/3904202505 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02301606e0 con 0x7f0230145430 2024-03-06T07:34:05.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.387+0000 7f022d600640 1 -- 172.21.15.32:0/3904202505 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431566780 0 0) 0x7f023015fe40 con 0x7f023015cc90 2024-03-06T07:34:05.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.388+0000 7f022ea00640 1 --2- 172.21.15.32:0/3904202505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0230145430 0x7f023015f810 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f02200097a0 tx=0x7f022002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e04b05bb39a7f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.389+0000 7f022d600640 1 -- 172.21.15.32:0/3904202505 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0220035020 con 0x7f0230145430 2024-03-06T07:34:05.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.389+0000 7f022d600640 1 -- 172.21.15.32:0/3904202505 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0220031830 con 0x7f0230145430 2024-03-06T07:34:05.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.389+0000 7f022d600640 1 -- 172.21.15.32:0/3904202505 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0220031c00 con 0x7f0230145430 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.389+0000 7f022fe00640 1 -- 172.21.15.32:0/3904202505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0230145430 msgr2=0x7f023015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.389+0000 7f022fe00640 1 --2- 172.21.15.32:0/3904202505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0230145430 0x7f023015f810 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f02200097a0 tx=0x7f022002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.390+0000 7f022fe00640 1 -- 172.21.15.32:0/3904202505 shutdown_connections 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.390+0000 7f022fe00640 1 --2- 172.21.15.32:0/3904202505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0230145430 0x7f023015f810 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.390+0000 7f022fe00640 1 -- 172.21.15.32:0/3904202505 >> 172.21.15.32:0/3904202505 conn(0x7f023006b3a0 msgr2=0x7f023006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.390+0000 7f022fe00640 1 -- 172.21.15.32:0/3904202505 shutdown_connections 2024-03-06T07:34:05.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.390+0000 7f022fe00640 1 -- 172.21.15.32:0/3904202505 wait complete. 2024-03-06T07:34:05.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.391+0000 7f022fe00640 1 Processor -- start 2024-03-06T07:34:05.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.392+0000 7f022fe00640 1 -- start start 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 0x7f02301406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0230145430 0x7f0230071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f023015cc90 0x7f0230072010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0230145990 con 0x7f02300b2250 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0230145810 con 0x7f0230145430 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02300cde00 con 0x7f023015cc90 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 0x7f02301406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 0x7f02301406b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58738/0 (socket says 172.21.15.32:58738) 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 -- 172.21.15.32:0/3040715662 learned_addr learned my addr 172.21.15.32:0/3040715662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022e000640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0230145430 0x7f0230071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 -- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f023015cc90 msgr2=0x7f0230072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f023015cc90 0x7f0230072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 -- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0230145430 msgr2=0x7f0230071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0230145430 0x7f0230071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022ea00640 1 -- 172.21.15.32:0/3040715662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02301f3980 con 0x7f02300b2250 2024-03-06T07:34:05.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.393+0000 7f022e000640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0230145430 0x7f0230071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:05.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.394+0000 7f022ea00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 0x7f02301406b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f02200097a0 tx=0x7f02200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.394+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0220004230 con 0x7f02300b2250 2024-03-06T07:34:05.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.394+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02200043d0 con 0x7f02300b2250 2024-03-06T07:34:05.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.394+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02301f3c60 con 0x7f02300b2250 2024-03-06T07:34:05.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.394+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022003cad0 con 0x7f02300b2250 2024-03-06T07:34:05.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.394+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02301f4220 con 0x7f02300b2250 2024-03-06T07:34:05.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.395+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023004f330 con 0x7f02300b2250 2024-03-06T07:34:05.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.396+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0220038020 con 0x7f02300b2250 2024-03-06T07:34:05.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.397+0000 7f0226a00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f020c06b4d0 0x7f020c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.397+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02200ac180 con 0x7f02300b2250 2024-03-06T07:34:05.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.397+0000 7f022e000640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f020c06b4d0 0x7f020c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.398+0000 7f022e000640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f020c06b4d0 0x7f020c06d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f02300d0510 tx=0x7f0218009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.402+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0220041070 con 0x7f02300b2250 2024-03-06T07:34:05.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.603+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f023006e240 con 0x7f02300b2250 2024-03-06T07:34:05.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.603+0000 7f0226a00640 1 -- 172.21.15.32:0/3040715662 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f022007b2b0 con 0x7f02300b2250 2024-03-06T07:34:05.606 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:34:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f020c06b4d0 msgr2=0x7f020c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f020c06b4d0 0x7f020c06d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f02300d0510 tx=0x7f0218009040 comp rx=0 tx=0).stop 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 msgr2=0x7f02301406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 0x7f02301406b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f02200097a0 tx=0x7f02200046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 shutdown_connections 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f020c06b4d0 0x7f020c06d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f023015cc90 0x7f0230072010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0230145430 0x7f0230071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 --2- 172.21.15.32:0/3040715662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02300b2250 0x7f02301406b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 >> 172.21.15.32:0/3040715662 conn(0x7f023006b3a0 msgr2=0x7f02300b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 shutdown_connections 2024-03-06T07:34:05.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.606+0000 7f022fe00640 1 -- 172.21.15.32:0/3040715662 wait complete. 2024-03-06T07:34:05.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.846+0000 7f0f31e00640 1 Processor -- start 2024-03-06T07:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.847+0000 7f0f31e00640 1 -- start start 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.848+0000 7f0f31e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 0x7f0f2c161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.848+0000 7f0f31e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c1627f0 con 0x7f0f2c15efc0 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.848+0000 7f0f31e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c06c860 con 0x7f0f2c162000 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.848+0000 7f0f31e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c15f3a0 con 0x7f0f2c0cd060 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.848+0000 7f0f30a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0f2c15efc0 0x7f0f2c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48394/0 (socket says 172.21.15.32:48394) 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.848+0000 7f0f30a00640 1 -- 172.21.15.32:0/3449864023 learned_addr learned my addr 172.21.15.32:0/3449864023 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078203964 0 0) 0x7f0f2c1627f0 con 0x7f0f2c15efc0 2024-03-06T07:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f14003520 con 0x7f0f2c15efc0 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2be00640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 0x7f0f2c161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377366093 0 0) 0x7f0f14003520 con 0x7f0f2c15efc0 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 msgr2=0x7f0f2c161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 0x7f0f2c161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 >> v1:172.21.15.39:6789/0 conn(0x7f0f2c162000 legacy=0x7f0f2c070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c06cb40 con 0x7f0f2c15efc0 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.849+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f0f20001670 con 0x7f0f2c15efc0 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 >> v1:172.21.15.32:6789/0 conn(0x7f0f2c15efc0 legacy=0x7f0f2c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f14004800 0x7f0f14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 0x7f0f14005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c15f3a0 con 0x7f0f140039e0 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1400bb50 con 0x7f0f14004800 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1400bd50 con 0x7f0f2c0cd060 2024-03-06T07:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f30a00640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f30a00640 1 -- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 msgr2=0x7f0f14005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f30a00640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c0cd060 0x7f0f14005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f30a00640 1 -- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f14004800 msgr2=0x7f0f14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f30a00640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f14004800 0x7f0f14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.850+0000 7f0f30a00640 1 -- 172.21.15.32:0/3449864023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14003520 con 0x7f0f140039e0 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.851+0000 7f0f30a00640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c190 tx=0x7f0f14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac38534e9b05614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20007c60 con 0x7f0f140039e0 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f20030070 con 0x7f0f140039e0 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f2b400640 1 -- 172.21.15.32:0/3449864023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20005d70 con 0x7f0f140039e0 2024-03-06T07:34:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31e00640 1 -- 172.21.15.32:0/3449864023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 msgr2=0x7f0f14003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31e00640 1 --2- 172.21.15.32:0/3449864023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c190 tx=0x7f0f14005830 comp rx=0 tx=0).stop 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31400640 1 -- 172.21.15.32:0/3449864023 reap_dead start 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31e00640 1 -- 172.21.15.32:0/3449864023 shutdown_connections 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31e00640 1 -- 172.21.15.32:0/3449864023 >> 172.21.15.32:0/3449864023 conn(0x7f0f2c06b3d0 msgr2=0x7f0f2c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31e00640 1 -- 172.21.15.32:0/3449864023 shutdown_connections 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.852+0000 7f0f31e00640 1 -- 172.21.15.32:0/3449864023 wait complete. 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.854+0000 7f0f31e00640 1 Processor -- start 2024-03-06T07:34:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.854+0000 7f0f31e00640 1 -- start start 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.855+0000 7f0f31e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f2c147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.855+0000 7f0f31e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f2c1e9960 0x7f0f2c15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.855+0000 7f0f31e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c15aa90 0x7f0f2c1f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.855+0000 7f0f31e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c1624b0 con 0x7f0f140039e0 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.855+0000 7f0f31e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c06d190 con 0x7f0f2c1e9960 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.855+0000 7f0f31e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c06d620 con 0x7f0f2c15aa90 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f2c147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f2c147160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58770/0 (socket says 172.21.15.32:58770) 2024-03-06T07:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 -- 172.21.15.32:0/981542664 learned_addr learned my addr 172.21.15.32:0/981542664 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f31400640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c15aa90 0x7f0f2c1f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f2be00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f2c1e9960 0x7f0f2c15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 -- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c15aa90 msgr2=0x7f0f2c1f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c15aa90 0x7f0f2c1f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 -- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f2c1e9960 msgr2=0x7f0f2c15a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f2c1e9960 0x7f0f2c15a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f30a00640 1 -- 172.21.15.32:0/981542664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c1e88c0 con 0x7f0f140039e0 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.856+0000 7f0f2be00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f2c1e9960 0x7f0f2c15a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.857+0000 7f0f30a00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f2c147160 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c190 tx=0x7f0f20001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.857+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20007c60 con 0x7f0f140039e0 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.857+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2002cc60 con 0x7f0f140039e0 2024-03-06T07:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.857+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c1e8ba0 con 0x7f0f140039e0 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.857+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20007290 con 0x7f0f140039e0 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.857+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c1e9160 con 0x7f0f140039e0 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.858+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c04f330 con 0x7f0f140039e0 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.859+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f20047020 con 0x7f0f140039e0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.860+0000 7f0f28c00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.860+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0f200ab1a0 con 0x7f0f140039e0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.860+0000 7f0f2be00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.861+0000 7f0f2be00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0f18006a90 tx=0x7f0f18008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:05.865+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f2007a360 con 0x7f0f140039e0 2024-03-06T07:34:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.069+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0f2c15e4f0 con 0x7f0f140039e0 2024-03-06T07:34:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.069+0000 7f0f28c00640 1 -- 172.21.15.32:0/981542664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v91) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0f20079c60 con 0x7f0f140039e0 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:34:06.072 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f0c06b4d0 msgr2=0x7f0f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0f18006a90 tx=0x7f0f18008040 comp rx=0 tx=0).stop 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 msgr2=0x7f0f2c147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f2c147160 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c190 tx=0x7f0f20001970 comp rx=0 tx=0).stop 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 shutdown_connections 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f2c15aa90 0x7f0f2c1f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f2c1e9960 0x7f0f2c15a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 --2- 172.21.15.32:0/981542664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f140039e0 0x7f0f2c147160 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.072+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 >> 172.21.15.32:0/981542664 conn(0x7f0f2c06b3d0 msgr2=0x7f0f2c0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.073+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 shutdown_connections 2024-03-06T07:34:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.073+0000 7f0f31e00640 1 -- 172.21.15.32:0/981542664 wait complete. 2024-03-06T07:34:06.094 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:34:06.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:34:06.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.320+0000 7f191ea00640 1 Processor -- start 2024-03-06T07:34:06.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.321+0000 7f191ea00640 1 -- start start 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191815e1f0 0x7f19180d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19180d4660 0x7f19180d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191ea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19180d23c0 con 0x7f19180b38f0 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191ea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19180d4c30 con 0x7f19180d4660 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19180d26a0 con 0x7f191815e1f0 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19180d4660 0x7f19180d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19180d4660 0x7f19180d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58786/0 (socket says 172.21.15.32:58786) 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.322+0000 7f191cc00640 1 -- 172.21.15.32:0/2687201851 learned_addr learned my addr 172.21.15.32:0/2687201851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.323+0000 7f191d600640 1 --2- 172.21.15.32:0/2687201851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191815e1f0 0x7f19180d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.323+0000 7f191cc00640 1 -- 172.21.15.32:0/2687201851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191815e1f0 msgr2=0x7f19180d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.323+0000 7f191cc00640 1 --2- 172.21.15.32:0/2687201851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191815e1f0 0x7f19180d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.323+0000 7f191cc00640 1 -- 172.21.15.32:0/2687201851 >> v1:172.21.15.39:6789/0 conn(0x7f19180b38f0 legacy=0x7f19180b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.323+0000 7f191cc00640 1 -- 172.21.15.32:0/2687201851 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19180d2980 con 0x7f19180d4660 2024-03-06T07:34:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.324+0000 7f191cc00640 1 --2- 172.21.15.32:0/2687201851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19180d4660 0x7f19180d0bc0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f1904009870 tx=0x7f190402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5d7c2d9591dc68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.324+0000 7f1917e00640 1 -- 172.21.15.32:0/2687201851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19040317c0 con 0x7f19180d4660 2024-03-06T07:34:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.324+0000 7f1917e00640 1 -- 172.21.15.32:0/2687201851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1904037070 con 0x7f19180d4660 2024-03-06T07:34:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.324+0000 7f1917e00640 1 -- 172.21.15.32:0/2687201851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1904031b80 con 0x7f19180d4660 2024-03-06T07:34:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 -- 172.21.15.32:0/2687201851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19180d4660 msgr2=0x7f19180d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 --2- 172.21.15.32:0/2687201851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19180d4660 0x7f19180d0bc0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f1904009870 tx=0x7f190402b860 comp rx=0 tx=0).stop 2024-03-06T07:34:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 -- 172.21.15.32:0/2687201851 shutdown_connections 2024-03-06T07:34:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 --2- 172.21.15.32:0/2687201851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19180d4660 0x7f19180d0bc0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 --2- 172.21.15.32:0/2687201851 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191815e1f0 0x7f19180d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 -- 172.21.15.32:0/2687201851 >> 172.21.15.32:0/2687201851 conn(0x7f191809d000 msgr2=0x7f19180d3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 -- 172.21.15.32:0/2687201851 shutdown_connections 2024-03-06T07:34:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.325+0000 7f191ea00640 1 -- 172.21.15.32:0/2687201851 wait complete. 2024-03-06T07:34:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.327+0000 7f191ea00640 1 Processor -- start 2024-03-06T07:34:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.327+0000 7f191ea00640 1 -- start start 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19180b38f0 0x7f19181585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19180d4660 0x7f1918160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 0x7f19180b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19180cf8d0 con 0x7f191815e1f0 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19180b2da0 con 0x7f19180b38f0 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19180cfa50 con 0x7f19180d4660 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 0x7f19180b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 0x7f19180b83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58796/0 (socket says 172.21.15.32:58796) 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191e000640 1 -- 172.21.15.32:0/2397206218 learned_addr learned my addr 172.21.15.32:0/2397206218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191d600640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19180b38f0 0x7f19181585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.328+0000 7f191cc00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19180d4660 0x7f1918160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191e000640 1 -- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19180d4660 msgr2=0x7f1918160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191e000640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19180d4660 0x7f1918160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191e000640 1 -- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19180b38f0 msgr2=0x7f19181585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191e000640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19180b38f0 0x7f19181585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191e000640 1 -- 172.21.15.32:0/2397206218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191815efe0 con 0x7f191815e1f0 2024-03-06T07:34:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191e000640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 0x7f19180b83f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f191000c780 tx=0x7f191000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910004b60 con 0x7f191815e1f0 2024-03-06T07:34:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910004d00 con 0x7f191815e1f0 2024-03-06T07:34:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191815f2c0 con 0x7f191815e1f0 2024-03-06T07:34:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.329+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191000aa90 con 0x7f191815e1f0 2024-03-06T07:34:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.330+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191815f880 con 0x7f191815e1f0 2024-03-06T07:34:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.331+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19180d0680 con 0x7f191815e1f0 2024-03-06T07:34:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.332+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f191000ac30 con 0x7f191815e1f0 2024-03-06T07:34:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.333+0000 7f1915600640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18f806b4d0 0x7f18f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.333+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f191008a2a0 con 0x7f191815e1f0 2024-03-06T07:34:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.333+0000 7f191d600640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18f806b4d0 0x7f18f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.334+0000 7f191d600640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18f806b4d0 0x7f18f806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1908006fd0 tx=0x7f1908008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.337+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19180d0680 con 0x7f191815e1f0 2024-03-06T07:34:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:06.538+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f191804f330 con 0x7f191815e1f0 2024-03-06T07:34:07.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.226+0000 7f1915600640 1 -- 172.21.15.32:0/2397206218 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v92) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1910020070 con 0x7f191815e1f0 2024-03-06T07:34:07.229 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.230+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18f806b4d0 msgr2=0x7f18f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.230+0000 7f191ea00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18f806b4d0 0x7f18f806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1908006fd0 tx=0x7f1908008040 comp rx=0 tx=0).stop 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.230+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 msgr2=0x7f19180b83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.230+0000 7f191ea00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 0x7f19180b83f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f191000c780 tx=0x7f191000cc40 comp rx=0 tx=0).stop 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 shutdown_connections 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18f806b4d0 0x7f18f806d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191815e1f0 0x7f19180b83f0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19180d4660 0x7f1918160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 --2- 172.21.15.32:0/2397206218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19180b38f0 0x7f19181585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 >> 172.21.15.32:0/2397206218 conn(0x7f191809d000 msgr2=0x7f191806b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.231+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 shutdown_connections 2024-03-06T07:34:07.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.232+0000 7f191ea00640 1 -- 172.21.15.32:0/2397206218 wait complete. 2024-03-06T07:34:07.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:34:07.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.479+0000 7fa18da00640 1 Processor -- start 2024-03-06T07:34:07.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.479+0000 7fa18da00640 1 -- start start 2024-03-06T07:34:07.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.480+0000 7fa18da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1880afab0 0x7fa188066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18da00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1880ce370 0x7fa1880ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18815be00 con 0x7fa18815ba20 2024-03-06T07:34:07.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18da00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18806f2c0 con 0x7fa1880ce370 2024-03-06T07:34:07.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1880cece0 con 0x7fa1880afab0 2024-03-06T07:34:07.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa187400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1880afab0 0x7fa188066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1880ce370 0x7fa1880ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1880ce370 0x7fa1880ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:54776/0 (socket says 172.21.15.32:54776) 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa18d000640 1 -- 172.21.15.32:0/2914484710 learned_addr learned my addr 172.21.15.32:0/2914484710 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.481+0000 7fa187e00640 1 --1- 172.21.15.32:0/2914484710 >> v1:172.21.15.39:6789/0 conn(0x7fa18815ba20 0x7fa18815ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:34900/0 (socket says 172.21.15.32:34900) 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.482+0000 7fa18d000640 1 -- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1880afab0 msgr2=0x7fa188066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.482+0000 7fa18d000640 1 --2- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1880afab0 0x7fa188066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.482+0000 7fa18d000640 1 -- 172.21.15.32:0/2914484710 >> v1:172.21.15.39:6789/0 conn(0x7fa18815ba20 legacy=0x7fa18815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.482+0000 7fa18d000640 1 -- 172.21.15.32:0/2914484710 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18806d4b0 con 0x7fa1880ce370 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.484+0000 7fa18d000640 1 --2- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa1880ce7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa18806f440 tx=0x7fa17802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5862cd421b997c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.484+0000 7fa186a00640 1 -- 172.21.15.32:0/2914484710 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178035020 con 0x7fa1880ce370 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.484+0000 7fa186a00640 1 -- 172.21.15.32:0/2914484710 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa178031640 con 0x7fa1880ce370 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.484+0000 7fa186a00640 1 -- 172.21.15.32:0/2914484710 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17802d040 con 0x7fa1880ce370 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 -- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 msgr2=0x7fa1880ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 --2- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa1880ce7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa18806f440 tx=0x7fa17802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 -- 172.21.15.32:0/2914484710 shutdown_connections 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 --2- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa1880ce7a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 --2- 172.21.15.32:0/2914484710 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa1880afab0 0x7fa188066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 -- 172.21.15.32:0/2914484710 >> 172.21.15.32:0/2914484710 conn(0x7fa1880ccf70 msgr2=0x7fa1880cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 -- 172.21.15.32:0/2914484710 shutdown_connections 2024-03-06T07:34:07.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.485+0000 7fa18da00640 1 -- 172.21.15.32:0/2914484710 wait complete. 2024-03-06T07:34:07.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.488+0000 7fa18da00640 1 Processor -- start 2024-03-06T07:34:07.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.488+0000 7fa18da00640 1 -- start start 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.489+0000 7fa18da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 0x7fa1881f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.489+0000 7fa18da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa18815da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.489+0000 7fa18da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18815ba20 0x7fa1881f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.489+0000 7fa18da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18815f360 con 0x7fa1880afab0 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.489+0000 7fa18da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18815e730 con 0x7fa1880ce370 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.489+0000 7fa18da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18815e420 con 0x7fa18815ba20 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 0x7fa1881f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 0x7fa1881f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58812/0 (socket says 172.21.15.32:58812) 2024-03-06T07:34:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 -- 172.21.15.32:0/91935337 learned_addr learned my addr 172.21.15.32:0/91935337 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:07.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187400640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa18815da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 -- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18815ba20 msgr2=0x7fa1881f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18815ba20 0x7fa1881f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 -- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 msgr2=0x7fa18815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa18815da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 -- 172.21.15.32:0/91935337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1881f44f0 con 0x7fa1880afab0 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.490+0000 7fa187e00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 0x7fa1881f3c10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa18000c790 tx=0x7fa18000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.491+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180017070 con 0x7fa1880afab0 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.491+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18000a640 con 0x7fa1880afab0 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.491+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18000a9f0 con 0x7fa1880afab0 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.491+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1881f47d0 con 0x7fa1880afab0 2024-03-06T07:34:07.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.491+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1881f74c0 con 0x7fa1880afab0 2024-03-06T07:34:07.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.493+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1881f76c0 con 0x7fa1880afab0 2024-03-06T07:34:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.493+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa18001a070 con 0x7fa1880afab0 2024-03-06T07:34:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.494+0000 7fa17fe00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa16406b520 0x7fa16406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.494+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa180089470 con 0x7fa1880afab0 2024-03-06T07:34:07.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.495+0000 7fa187400640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa16406b520 0x7fa16406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.496+0000 7fa187400640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa16406b520 0x7fa16406d9e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa188152f70 tx=0x7fa170008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.500+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1881f76c0 con 0x7fa1880afab0 2024-03-06T07:34:07.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:07.703+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa1881f49a0 con 0x7fa1880afab0 2024-03-06T07:34:08.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.232+0000 7fa17fe00640 1 -- 172.21.15.32:0/91935337 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v93) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa180057ff0 con 0x7fa1880afab0 2024-03-06T07:34:08.235 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa16406b520 msgr2=0x7fa16406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa16406b520 0x7fa16406d9e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa188152f70 tx=0x7fa170008040 comp rx=0 tx=0).stop 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 msgr2=0x7fa1881f3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 0x7fa1881f3c10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa18000c790 tx=0x7fa18000cc50 comp rx=0 tx=0).stop 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 shutdown_connections 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa16406b520 0x7fa16406d9e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18815ba20 0x7fa1881f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1880ce370 0x7fa18815da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 --2- 172.21.15.32:0/91935337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1880afab0 0x7fa1881f3c10 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.236+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 >> 172.21.15.32:0/91935337 conn(0x7fa1880ccf70 msgr2=0x7fa188156fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.237+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 shutdown_connections 2024-03-06T07:34:08.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.237+0000 7fa18da00640 1 -- 172.21.15.32:0/91935337 wait complete. 2024-03-06T07:34:08.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.463+0000 7f159c200640 1 Processor -- start 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.463+0000 7f159c200640 1 -- start start 2024-03-06T07:34:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.464+0000 7f159c200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15940ce550 0x7f159415f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.464+0000 7f159c200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f15940cffa0 0x7f15940ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.464+0000 7f159c200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940cee70 con 0x7f159415d940 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.464+0000 7f159c200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15940d0780 con 0x7f15940ce550 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.464+0000 7f159c200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15940d0a60 con 0x7f15940cffa0 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15940ce550 0x7f159415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15940ce550 0x7f159415f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46942/0 (socket says 172.21.15.32:46942) 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 -- 172.21.15.32:0/3771995986 learned_addr learned my addr 172.21.15.32:0/3771995986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1598e00640 1 --2- 172.21.15.32:0/3771995986 >> v2:172.21.15.39:3300/0 conn(0x7f15940cffa0 0x7f15940ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 -- 172.21.15.32:0/3771995986 >> v2:172.21.15.39:3300/0 conn(0x7f15940cffa0 msgr2=0x7f15940ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 --2- 172.21.15.32:0/3771995986 >> v2:172.21.15.39:3300/0 conn(0x7f15940cffa0 0x7f15940ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 -- 172.21.15.32:0/3771995986 >> v1:172.21.15.39:6789/0 conn(0x7f159415d940 legacy=0x7f159415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1599800640 1 -- 172.21.15.32:0/3771995986 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940d0d40 con 0x7f15940ce550 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.465+0000 7f1598e00640 1 --2- 172.21.15.32:0/3771995986 >> v2:172.21.15.39:3300/0 conn(0x7f15940cffa0 0x7f15940ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.467+0000 7f1599800640 1 --2- 172.21.15.32:0/3771995986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15940ce550 0x7f159415f230 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f1594071e10 tx=0x7f158802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400ea1e581148525 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.467+0000 7f1593e00640 1 -- 172.21.15.32:0/3771995986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1588035020 con 0x7f15940ce550 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.467+0000 7f1593e00640 1 -- 172.21.15.32:0/3771995986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1588031870 con 0x7f15940ce550 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.467+0000 7f1593e00640 1 -- 172.21.15.32:0/3771995986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1588031c40 con 0x7f15940ce550 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.467+0000 7f159c200640 1 -- 172.21.15.32:0/3771995986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15940ce550 msgr2=0x7f159415f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.467+0000 7f159c200640 1 --2- 172.21.15.32:0/3771995986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15940ce550 0x7f159415f230 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f1594071e10 tx=0x7f158802b910 comp rx=0 tx=0).stop 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.468+0000 7f159c200640 1 -- 172.21.15.32:0/3771995986 shutdown_connections 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.468+0000 7f159c200640 1 --2- 172.21.15.32:0/3771995986 >> v2:172.21.15.39:3300/0 conn(0x7f15940cffa0 0x7f15940ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.468+0000 7f159c200640 1 --2- 172.21.15.32:0/3771995986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15940ce550 0x7f159415f230 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.468+0000 7f159c200640 1 -- 172.21.15.32:0/3771995986 >> 172.21.15.32:0/3771995986 conn(0x7f15940b3820 msgr2=0x7f15940b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.468+0000 7f159c200640 1 -- 172.21.15.32:0/3771995986 shutdown_connections 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.468+0000 7f159c200640 1 -- 172.21.15.32:0/3771995986 wait complete. 2024-03-06T07:34:08.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.469+0000 7f159c200640 1 Processor -- start 2024-03-06T07:34:08.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.470+0000 7f159c200640 1 -- start start 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.470+0000 7f159c200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15940ce550 0x7f1594161ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.470+0000 7f159c200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15940cffa0 0x7f15940b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159c200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 0x7f15940b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159c200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15940d0400 con 0x7f159415d940 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159c200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15940d1300 con 0x7f15940ce550 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159c200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159415e4c0 con 0x7f15940cffa0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 0x7f15940b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 0x7f15940b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46954/0 (socket says 172.21.15.32:46954) 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159a200640 1 -- 172.21.15.32:0/2117037075 learned_addr learned my addr 172.21.15.32:0/2117037075 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f1599800640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15940ce550 0x7f1594161ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f1598e00640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15940cffa0 0x7f15940b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159a200640 1 -- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15940cffa0 msgr2=0x7f15940b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.471+0000 7f159a200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15940cffa0 0x7f15940b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f159a200640 1 -- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15940ce550 msgr2=0x7f1594161ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f159a200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15940ce550 0x7f1594161ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f159a200640 1 -- 172.21.15.32:0/2117037075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940b7580 con 0x7f159415d940 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f159a200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 0x7f15940b4e50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f158c00b970 tx=0x7f158c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c019070 con 0x7f159415d940 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c00cb40 con 0x7f159415d940 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c014440 con 0x7f159415d940 2024-03-06T07:34:08.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15940c92a0 con 0x7f159415d940 2024-03-06T07:34:08.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.472+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15940c9b40 con 0x7f159415d940 2024-03-06T07:34:08.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.474+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159404f330 con 0x7f159415d940 2024-03-06T07:34:08.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.475+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f158c01c070 con 0x7f159415d940 2024-03-06T07:34:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.476+0000 7f1591600640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f157806b4d0 0x7f157806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.477+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f158c089e80 con 0x7f159415d940 2024-03-06T07:34:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.477+0000 7f1599800640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f157806b4d0 0x7f157806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.477+0000 7f1599800640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f157806b4d0 0x7f157806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1588036000 tx=0x7f158803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.482+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f158c058c30 con 0x7f159415d940 2024-03-06T07:34:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:08.723+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f15940679a0 con 0x7f159415d940 2024-03-06T07:34:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:34:09.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.273+0000 7f1591600640 1 -- 172.21.15.32:0/2117037075 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v57) v1 ==== 174+0+0 (secure 0 0 0) 0x7f158c058530 con 0x7f159415d940 2024-03-06T07:34:09.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.279+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f157806b4d0 msgr2=0x7f157806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.279+0000 7f159c200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f157806b4d0 0x7f157806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1588036000 tx=0x7f158803c040 comp rx=0 tx=0).stop 2024-03-06T07:34:09.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.279+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 msgr2=0x7f15940b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.279+0000 7f159c200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 0x7f15940b4e50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f158c00b970 tx=0x7f158c00be30 comp rx=0 tx=0).stop 2024-03-06T07:34:09.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.281+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 shutdown_connections 2024-03-06T07:34:09.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.281+0000 7f159c200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f157806b4d0 0x7f157806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.281+0000 7f159c200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f159415d940 0x7f15940b4e50 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.281+0000 7f159c200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f15940cffa0 0x7f15940b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.281+0000 7f159c200640 1 --2- 172.21.15.32:0/2117037075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15940ce550 0x7f1594161ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.281+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 >> 172.21.15.32:0/2117037075 conn(0x7f15940b3820 msgr2=0x7f159406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.283+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 shutdown_connections 2024-03-06T07:34:09.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.283+0000 7f159c200640 1 -- 172.21.15.32:0/2117037075 wait complete. 2024-03-06T07:34:09.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.495+0000 7fe1f6c00640 1 Processor -- start 2024-03-06T07:34:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.495+0000 7fe1f6c00640 1 -- start start 2024-03-06T07:34:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1f6c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1f00af1a0 0x7fe1f015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1f6c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe1f00af580 0x7fe1f00b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1f6c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00cdc20 0x7fe1f00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1f6c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f015cc40 con 0x7fe1f00af1a0 2024-03-06T07:34:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1f6c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f015cf20 con 0x7fe1f00af580 2024-03-06T07:34:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1f6c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f015d200 con 0x7fe1f00cdc20 2024-03-06T07:34:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1efe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1f00af1a0 0x7fe1f015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1efe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1f00af1a0 0x7fe1f015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46966/0 (socket says 172.21.15.32:46966) 2024-03-06T07:34:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1efe00640 1 -- 172.21.15.32:0/3215130706 learned_addr learned my addr 172.21.15.32:0/3215130706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1efe00640 1 -- 172.21.15.32:0/3215130706 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00cdc20 msgr2=0x7fe1f00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.497+0000 7fe1efe00640 1 --2- 172.21.15.32:0/3215130706 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00cdc20 0x7fe1f00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.498+0000 7fe1efe00640 1 -- 172.21.15.32:0/3215130706 >> v2:172.21.15.39:3300/0 conn(0x7fe1f00af580 msgr2=0x7fe1f00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.498+0000 7fe1e7400640 1 --2- 172.21.15.32:0/3215130706 >> v2:172.21.15.39:3300/0 conn(0x7fe1f00af580 0x7fe1f00b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.498+0000 7fe1efe00640 1 --2- 172.21.15.32:0/3215130706 >> v2:172.21.15.39:3300/0 conn(0x7fe1f00af580 0x7fe1f00b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.498+0000 7fe1efe00640 1 -- 172.21.15.32:0/3215130706 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f015d4e0 con 0x7fe1f00af1a0 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.498+0000 7fe1e7400640 1 --2- 172.21.15.32:0/3215130706 >> v2:172.21.15.39:3300/0 conn(0x7fe1f00af580 0x7fe1f00b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.499+0000 7fe1efe00640 1 --2- 172.21.15.32:0/3215130706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af1a0 0x7fe1f015c600 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00b54c0 tx=0x7fe1e002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbe66f247b9a2d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.499+0000 7fe1ef400640 1 -- 172.21.15.32:0/3215130706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e002c9f0 con 0x7fe1f00af1a0 2024-03-06T07:34:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.499+0000 7fe1ef400640 1 -- 172.21.15.32:0/3215130706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e002cb90 con 0x7fe1f00af1a0 2024-03-06T07:34:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.499+0000 7fe1ef400640 1 -- 172.21.15.32:0/3215130706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e00343f0 con 0x7fe1f00af1a0 2024-03-06T07:34:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 -- 172.21.15.32:0/3215130706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af1a0 msgr2=0x7fe1f015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/3215130706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af1a0 0x7fe1f015c600 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00b54c0 tx=0x7fe1e002be50 comp rx=0 tx=0).stop 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 -- 172.21.15.32:0/3215130706 shutdown_connections 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/3215130706 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00cdc20 0x7fe1f00b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/3215130706 >> v2:172.21.15.39:3300/0 conn(0x7fe1f00af580 0x7fe1f00b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/3215130706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af1a0 0x7fe1f015c600 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 -- 172.21.15.32:0/3215130706 >> 172.21.15.32:0/3215130706 conn(0x7fe1f006afb0 msgr2=0x7fe1f006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 -- 172.21.15.32:0/3215130706 shutdown_connections 2024-03-06T07:34:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.500+0000 7fe1f6c00640 1 -- 172.21.15.32:0/3215130706 wait complete. 2024-03-06T07:34:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.502+0000 7fe1f6c00640 1 Processor -- start 2024-03-06T07:34:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.502+0000 7fe1f6c00640 1 -- start start 2024-03-06T07:34:09.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1f6c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00af1a0 0x7fe1f01467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1f6c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 0x7fe1f00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1f6c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f00cdc20 0x7fe1f006e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1f6c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00b3490 con 0x7fe1f00af580 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1f6c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00b3310 con 0x7fe1f00cdc20 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1f6c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00b3610 con 0x7fe1f00af1a0 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1e7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 0x7fe1f00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1e7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 0x7fe1f00c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46976/0 (socket says 172.21.15.32:46976) 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.503+0000 7fe1e7400640 1 -- 172.21.15.32:0/2016510303 learned_addr learned my addr 172.21.15.32:0/2016510303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1e7400640 1 -- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00af1a0 msgr2=0x7fe1f01467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1f4c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f00cdc20 0x7fe1f006e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1efe00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00af1a0 0x7fe1f01467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1e7400640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00af1a0 0x7fe1f01467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1e7400640 1 -- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f00cdc20 msgr2=0x7fe1f006e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1e7400640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f00cdc20 0x7fe1f006e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1e7400640 1 -- 172.21.15.32:0/2016510303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f0152710 con 0x7fe1f00af580 2024-03-06T07:34:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1f4c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f00cdc20 0x7fe1f006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1e7400640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 0x7fe1f00c9150 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800c7f0 tx=0x7fe1d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8015070 con 0x7fe1f00af580 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d8007a30 con 0x7fe1f00af580 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8007e00 con 0x7fe1f00af580 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f0159140 con 0x7fe1f00af580 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.504+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f0151e30 con 0x7fe1f00af580 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.505+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f0069510 con 0x7fe1f00af580 2024-03-06T07:34:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.506+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1d8018070 con 0x7fe1f00af580 2024-03-06T07:34:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.507+0000 7fe1ecc00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe1cc06b500 0x7fe1cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.507+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1d808a250 con 0x7fe1f00af580 2024-03-06T07:34:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.508+0000 7fe1efe00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe1cc06b500 0x7fe1cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.509+0000 7fe1efe00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe1cc06b500 0x7fe1cc06d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0035000 tx=0x7fe1e0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.512+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1d8059030 con 0x7fe1f00af580 2024-03-06T07:34:09.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.708+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1f00ae6c0 con 0x7fe1f00af580 2024-03-06T07:34:09.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.711+0000 7fe1ecc00640 1 -- 172.21.15.32:0/2016510303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16513 (secure 0 0 0) 0x7fe1d8058930 con 0x7fe1f00af580 2024-03-06T07:34:09.714 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:09.714 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:08.725633+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:09.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe1cc06b500 msgr2=0x7fe1cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe1cc06b500 0x7fe1cc06d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0035000 tx=0x7fe1e0030000 comp rx=0 tx=0).stop 2024-03-06T07:34:09.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 msgr2=0x7fe1f00c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 0x7fe1f00c9150 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800c7f0 tx=0x7fe1d800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 shutdown_connections 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe1cc06b500 0x7fe1cc06d9c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f00cdc20 0x7fe1f006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f00af580 0x7fe1f00c9150 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 --2- 172.21.15.32:0/2016510303 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1f00af1a0 0x7fe1f01467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 >> 172.21.15.32:0/2016510303 conn(0x7fe1f006afb0 msgr2=0x7fe1f00cc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 shutdown_connections 2024-03-06T07:34:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.714+0000 7fe1f6c00640 1 -- 172.21.15.32:0/2016510303 wait complete. 2024-03-06T07:34:09.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:09.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.951+0000 7feb2a400640 1 Processor -- start 2024-03-06T07:34:09.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.952+0000 7feb2a400640 1 -- start start 2024-03-06T07:34:09.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb2a400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feb240d4a10 0x7feb240d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb2a400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240b3710 con 0x7feb240d0c80 2024-03-06T07:34:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb2a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240b39f0 con 0x7feb2415e0b0 2024-03-06T07:34:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb2a400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb240d10f0 con 0x7feb240d4a10 2024-03-06T07:34:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb29a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feb240d0c80 0x7feb240b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44928/0 (socket says 172.21.15.32:44928) 2024-03-06T07:34:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb29a00640 1 -- 172.21.15.32:0/776856217 learned_addr learned my addr 172.21.15.32:0/776856217 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:09.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.953+0000 7feb23e00640 1 --2- 172.21.15.32:0/776856217 >> v2:172.21.15.32:3300/0 conn(0x7feb240d4a10 0x7feb240d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.954+0000 7feb23400640 1 -- 172.21.15.32:0/776856217 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546306862 0 0) 0x7feb240b3710 con 0x7feb240d0c80 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.954+0000 7feb23400640 1 -- 172.21.15.32:0/776856217 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb0c003520 con 0x7feb240d0c80 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.954+0000 7feb23e00640 1 -- 172.21.15.32:0/776856217 >> v1:172.21.15.39:6789/0 conn(0x7feb2415e0b0 legacy=0x7feb240cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.954+0000 7feb23e00640 1 -- 172.21.15.32:0/776856217 >> v1:172.21.15.32:6789/0 conn(0x7feb240d0c80 legacy=0x7feb240b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.954+0000 7feb23e00640 1 -- 172.21.15.32:0/776856217 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240b3cd0 con 0x7feb240d4a10 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.955+0000 7feb23e00640 1 --2- 172.21.15.32:0/776856217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d4a10 0x7feb240d0740 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7feb100097a0 tx=0x7feb1002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6158ad9b047a2be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb23400640 1 -- 172.21.15.32:0/776856217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10034070 con 0x7feb240d4a10 2024-03-06T07:34:09.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb23400640 1 -- 172.21.15.32:0/776856217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10030610 con 0x7feb240d4a10 2024-03-06T07:34:09.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb23400640 1 -- 172.21.15.32:0/776856217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb100309c0 con 0x7feb240d4a10 2024-03-06T07:34:09.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 -- 172.21.15.32:0/776856217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d4a10 msgr2=0x7feb240d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 --2- 172.21.15.32:0/776856217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d4a10 0x7feb240d0740 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7feb100097a0 tx=0x7feb1002b960 comp rx=0 tx=0).stop 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 -- 172.21.15.32:0/776856217 shutdown_connections 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 --2- 172.21.15.32:0/776856217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d4a10 0x7feb240d0740 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 -- 172.21.15.32:0/776856217 >> 172.21.15.32:0/776856217 conn(0x7feb240d3d30 msgr2=0x7feb240d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 -- 172.21.15.32:0/776856217 shutdown_connections 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.956+0000 7feb2a400640 1 -- 172.21.15.32:0/776856217 wait complete. 2024-03-06T07:34:09.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.958+0000 7feb2a400640 1 Processor -- start 2024-03-06T07:34:09.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.958+0000 7feb2a400640 1 -- start start 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb2a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 0x7feb241f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb2a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb240d4a10 0x7feb2415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb2a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb2415e0b0 0x7feb24156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb2a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb240cfa20 con 0x7feb240d0c80 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb2a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2415e490 con 0x7feb240d4a10 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb2a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb240cfc60 con 0x7feb2415e0b0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb29000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 0x7feb241f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb29000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 0x7feb241f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46994/0 (socket says 172.21.15.32:46994) 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.959+0000 7feb29000640 1 -- 172.21.15.32:0/1265826569 learned_addr learned my addr 172.21.15.32:0/1265826569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb23e00640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb240d4a10 0x7feb2415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb29000640 1 -- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb2415e0b0 msgr2=0x7feb24156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb29000640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb2415e0b0 0x7feb24156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb29000640 1 -- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb240d4a10 msgr2=0x7feb2415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb29000640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb240d4a10 0x7feb2415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb29000640 1 -- 172.21.15.32:0/1265826569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2415fab0 con 0x7feb240d0c80 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb23e00640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb240d4a10 0x7feb2415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb29000640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 0x7feb241f32d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7feb1800c450 tx=0x7feb1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb180052d0 con 0x7feb240d0c80 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18005470 con 0x7feb240d0c80 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18012870 con 0x7feb240d0c80 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.960+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2415fd90 con 0x7feb240d0c80 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.961+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb24160350 con 0x7feb240d0c80 2024-03-06T07:34:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.962+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2404f330 con 0x7feb240d0c80 2024-03-06T07:34:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.963+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb18012a10 con 0x7feb240d0c80 2024-03-06T07:34:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.963+0000 7feb20c00640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb0406b4d0 0x7feb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.964+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb18089dd0 con 0x7feb240d0c80 2024-03-06T07:34:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.964+0000 7feb23e00640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb0406b4d0 0x7feb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.964+0000 7feb23e00640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb0406b4d0 0x7feb0406d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feb10036000 tx=0x7feb10008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:09.968+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb18058cc0 con 0x7feb240d0c80 2024-03-06T07:34:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.169+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb240cf680 con 0x7feb240d0c80 2024-03-06T07:34:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.172+0000 7feb20c00640 1 -- 172.21.15.32:0/1265826569 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16513 (secure 0 0 0) 0x7feb180585c0 con 0x7feb240d0c80 2024-03-06T07:34:10.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:10.175 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:08.725633+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.174+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb0406b4d0 msgr2=0x7feb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.174+0000 7feb2a400640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb0406b4d0 0x7feb0406d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feb10036000 tx=0x7feb10008030 comp rx=0 tx=0).stop 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 msgr2=0x7feb241f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 0x7feb241f32d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7feb1800c450 tx=0x7feb1800c910 comp rx=0 tx=0).stop 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 shutdown_connections 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb0406b4d0 0x7feb0406d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb2415e0b0 0x7feb24156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb240d4a10 0x7feb2415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 --2- 172.21.15.32:0/1265826569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d0c80 0x7feb241f32d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 >> 172.21.15.32:0/1265826569 conn(0x7feb240d3d30 msgr2=0x7feb240b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 shutdown_connections 2024-03-06T07:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.175+0000 7feb2a400640 1 -- 172.21.15.32:0/1265826569 wait complete. 2024-03-06T07:34:10.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:34:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.418+0000 7fddc0600640 1 Processor -- start 2024-03-06T07:34:10.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.418+0000 7fddc0600640 1 -- start start 2024-03-06T07:34:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddc0600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fddb8072520 0x7fddb80cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddc0600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fddb8071e20 0x7fddb80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddc0600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb8072df0 con 0x7fddb815bf20 2024-03-06T07:34:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddc0600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80cdab0 con 0x7fddb8072520 2024-03-06T07:34:10.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddc0600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80b4d70 con 0x7fddb8071e20 2024-03-06T07:34:10.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddbdc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fddb815bf20 0x7fddb80acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44934/0 (socket says 172.21.15.32:44934) 2024-03-06T07:34:10.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.420+0000 7fddbdc00640 1 -- 172.21.15.32:0/3697599704 learned_addr learned my addr 172.21.15.32:0/3697599704 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:10.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddbd200640 1 --2- 172.21.15.32:0/3697599704 >> v2:172.21.15.32:3300/0 conn(0x7fddb8072520 0x7fddb80cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddbe600640 1 --2- 172.21.15.32:0/3697599704 >> v2:172.21.15.39:3300/0 conn(0x7fddb8071e20 0x7fddb80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddb7e00640 1 -- 172.21.15.32:0/3697599704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900852777 0 0) 0x7fddb8072df0 con 0x7fddb815bf20 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddb7e00640 1 -- 172.21.15.32:0/3697599704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda0003520 con 0x7fddb815bf20 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddbd200640 1 -- 172.21.15.32:0/3697599704 >> v2:172.21.15.39:3300/0 conn(0x7fddb8071e20 msgr2=0x7fddb80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddbd200640 1 --2- 172.21.15.32:0/3697599704 >> v2:172.21.15.39:3300/0 conn(0x7fddb8071e20 0x7fddb80ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddbd200640 1 -- 172.21.15.32:0/3697599704 >> v1:172.21.15.32:6789/0 conn(0x7fddb815bf20 legacy=0x7fddb80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.421+0000 7fddbd200640 1 -- 172.21.15.32:0/3697599704 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80b5050 con 0x7fddb8072520 2024-03-06T07:34:10.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.423+0000 7fddbd200640 1 --2- 172.21.15.32:0/3697599704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8072520 0x7fddb80cdfb0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fddb80af200 tx=0x7fdda402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bcb788b27bd1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.423+0000 7fddb7e00640 1 -- 172.21.15.32:0/3697599704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4035020 con 0x7fddb8072520 2024-03-06T07:34:10.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.423+0000 7fddb7e00640 1 -- 172.21.15.32:0/3697599704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda4031830 con 0x7fddb8072520 2024-03-06T07:34:10.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.423+0000 7fddb7e00640 1 -- 172.21.15.32:0/3697599704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031c00 con 0x7fddb8072520 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 -- 172.21.15.32:0/3697599704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8072520 msgr2=0x7fddb80cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 --2- 172.21.15.32:0/3697599704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8072520 0x7fddb80cdfb0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fddb80af200 tx=0x7fdda402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 -- 172.21.15.32:0/3697599704 shutdown_connections 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 --2- 172.21.15.32:0/3697599704 >> v2:172.21.15.39:3300/0 conn(0x7fddb8071e20 0x7fddb80ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 --2- 172.21.15.32:0/3697599704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8072520 0x7fddb80cdfb0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 -- 172.21.15.32:0/3697599704 >> 172.21.15.32:0/3697599704 conn(0x7fddb806b3d0 msgr2=0x7fddb806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 -- 172.21.15.32:0/3697599704 shutdown_connections 2024-03-06T07:34:10.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.424+0000 7fddc0600640 1 -- 172.21.15.32:0/3697599704 wait complete. 2024-03-06T07:34:10.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.426+0000 7fddc0600640 1 Processor -- start 2024-03-06T07:34:10.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.426+0000 7fddc0600640 1 -- start start 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddc0600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 0x7fddb809c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddc0600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddb8072520 0x7fddb809d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddc0600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fddb815bf20 0x7fddb815e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddc0600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80ad450 con 0x7fddb8071e20 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddc0600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80b4990 con 0x7fddb8072520 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddc0600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb809d880 con 0x7fddb815bf20 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddbdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 0x7fddb809c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddbdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 0x7fddb809c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47004/0 (socket says 172.21.15.32:47004) 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddbdc00640 1 -- 172.21.15.32:0/2385220797 learned_addr learned my addr 172.21.15.32:0/2385220797 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.427+0000 7fddbe600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fddb815bf20 0x7fddb815e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbdc00640 1 -- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fddb815bf20 msgr2=0x7fddb815e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbdc00640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fddb815bf20 0x7fddb815e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbd200640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddb8072520 0x7fddb809d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbdc00640 1 -- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddb8072520 msgr2=0x7fddb809d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbdc00640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddb8072520 0x7fddb809d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbdc00640 1 -- 172.21.15.32:0/2385220797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb815e980 con 0x7fddb8071e20 2024-03-06T07:34:10.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddbdc00640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 0x7fddb809c5a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c390 tx=0x7fddac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac017070 con 0x7fddb8071e20 2024-03-06T07:34:10.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac005ce0 con 0x7fddb8071e20 2024-03-06T07:34:10.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0072f0 con 0x7fddb8071e20 2024-03-06T07:34:10.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb815f1e0 con 0x7fddb8071e20 2024-03-06T07:34:10.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.428+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb815ef00 con 0x7fddb8071e20 2024-03-06T07:34:10.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.429+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb804f330 con 0x7fddb8071e20 2024-03-06T07:34:10.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.430+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddac01a070 con 0x7fddb8071e20 2024-03-06T07:34:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.431+0000 7fddb5600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd9806b4d0 0x7fdd9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.431+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddac0894c0 con 0x7fddb8071e20 2024-03-06T07:34:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.431+0000 7fddbd200640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd9806b4d0 0x7fdd9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.432+0000 7fddbd200640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd9806b4d0 0x7fdd9806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.435+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddac057d80 con 0x7fddb8071e20 2024-03-06T07:34:10.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:10.676+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fddb815f720 con 0x7fddb8071e20 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.296+0000 7fddb5600640 1 -- 172.21.15.32:0/2385220797 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7fddac05e0d0 con 0x7fddb8071e20 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd9806b4d0 msgr2=0x7fdd9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd9806b4d0 0x7fdd9806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda4008030 comp rx=0 tx=0).stop 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 msgr2=0x7fddb809c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 0x7fddb809c5a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c390 tx=0x7fddac00c850 comp rx=0 tx=0).stop 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 shutdown_connections 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdd9806b4d0 0x7fdd9806d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fddb815bf20 0x7fddb815e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddb8072520 0x7fddb809d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 --2- 172.21.15.32:0/2385220797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddb8071e20 0x7fddb809c5a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 >> 172.21.15.32:0/2385220797 conn(0x7fddb806b3d0 msgr2=0x7fddb80b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 shutdown_connections 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.300+0000 7fddc0600640 1 -- 172.21.15.32:0/2385220797 wait complete. 2024-03-06T07:34:11.327 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:34:11.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:34:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.548+0000 7fc5f9200640 1 Processor -- start 2024-03-06T07:34:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.548+0000 7fc5f9200640 1 -- start start 2024-03-06T07:34:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.550+0000 7fc5f9200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc5f40b2250 0x7fc5f4166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.550+0000 7fc5f9200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f40b3fe0 con 0x7fc5f415f810 2024-03-06T07:34:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.550+0000 7fc5f9200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f415fe40 con 0x7fc5f4145430 2024-03-06T07:34:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.550+0000 7fc5f9200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f4160120 con 0x7fc5f40b2250 2024-03-06T07:34:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.551+0000 7fc5f2a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc5f415f810 0x7fc5f40b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44940/0 (socket says 172.21.15.32:44940) 2024-03-06T07:34:11.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.551+0000 7fc5f2a00640 1 -- 172.21.15.32:0/3977514999 learned_addr learned my addr 172.21.15.32:0/3977514999 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.551+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673384934 0 0) 0x7fc5f40b3fe0 con 0x7fc5f415f810 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.551+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8003520 con 0x7fc5f415f810 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.551+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/3977514999 >> v2:172.21.15.39:3300/0 conn(0x7fc5f40b2250 0x7fc5f4166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293224145 0 0) 0x7fc5d8003520 con 0x7fc5f415f810 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 >> v2:172.21.15.39:3300/0 conn(0x7fc5f40b2250 msgr2=0x7fc5f4166ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 --2- 172.21.15.32:0/3977514999 >> v2:172.21.15.39:3300/0 conn(0x7fc5f40b2250 0x7fc5f4166ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 >> v1:172.21.15.39:6789/0 conn(0x7fc5f4145430 legacy=0x7fc5f415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f4160400 con 0x7fc5f415f810 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fc5e0001670 con 0x7fc5f415f810 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/3977514999 >> v2:172.21.15.39:3300/0 conn(0x7fc5f40b2250 0x7fc5f4166ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 >> v1:172.21.15.32:6789/0 conn(0x7fc5f415f810 legacy=0x7fc5f40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5d8003a00 0x7fc5d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8005210 0x7fc5d800fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4160120 con 0x7fc5d8004800 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8010570 con 0x7fc5d8003a00 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8010770 con 0x7fc5d8005210 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2a00640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f3400640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5d8003a00 0x7fc5d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8005210 0x7fc5d800fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2a00640 1 -- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8005210 msgr2=0x7fc5d800fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2a00640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8005210 0x7fc5d800fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2a00640 1 -- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5d8003a00 msgr2=0x7fc5d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2a00640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5d8003a00 0x7fc5d8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.552+0000 7fc5f2a00640 1 -- 172.21.15.32:0/3977514999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8003520 con 0x7fc5d8004800 2024-03-06T07:34:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.553+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8005210 0x7fc5d800fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:11.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.553+0000 7fc5f3e00640 1 -- 172.21.15.32:0/3977514999 reap_dead start 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f2a00640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000c110 tx=0x7fc5d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab8c194c4779122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0007c40 con 0x7fc5d8004800 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e0030070 con 0x7fc5d8004800 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f2000640 1 -- 172.21.15.32:0/3977514999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0005d70 con 0x7fc5d8004800 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f9200640 1 -- 172.21.15.32:0/3977514999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d8004800 msgr2=0x7fc5d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f9200640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000c110 tx=0x7fc5d8005830 comp rx=0 tx=0).stop 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f9200640 1 -- 172.21.15.32:0/3977514999 shutdown_connections 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f9200640 1 --2- 172.21.15.32:0/3977514999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.554+0000 7fc5f9200640 1 -- 172.21.15.32:0/3977514999 >> 172.21.15.32:0/3977514999 conn(0x7fc5f406b3a0 msgr2=0x7fc5f406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.555+0000 7fc5f9200640 1 -- 172.21.15.32:0/3977514999 shutdown_connections 2024-03-06T07:34:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.555+0000 7fc5f9200640 1 -- 172.21.15.32:0/3977514999 wait complete. 2024-03-06T07:34:11.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.556+0000 7fc5f9200640 1 Processor -- start 2024-03-06T07:34:11.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.556+0000 7fc5f9200640 1 -- start start 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.557+0000 7fc5f9200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8004800 0x7fc5f41406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.557+0000 7fc5f9200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5f413af50 0x7fc5f41ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.557+0000 7fc5f9200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 0x7fc5f4141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.557+0000 7fc5f9200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4145990 con 0x7fc5f41efa30 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.557+0000 7fc5f9200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4145810 con 0x7fc5f413af50 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.557+0000 7fc5f9200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f40cde00 con 0x7fc5d8004800 2024-03-06T07:34:11.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 0x7fc5f4141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 0x7fc5f4141980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47034/0 (socket says 172.21.15.32:47034) 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 -- 172.21.15.32:0/189314209 learned_addr learned my addr 172.21.15.32:0/189314209 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3400640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8004800 0x7fc5f41406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 -- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8004800 msgr2=0x7fc5f41406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8004800 0x7fc5f41406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 -- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5f413af50 msgr2=0x7fc5f41ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5f413af50 0x7fc5f41ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 -- 172.21.15.32:0/189314209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f41efeb0 con 0x7fc5f41efa30 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.558+0000 7fc5f3e00640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 0x7fc5f4141980 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc5f406df70 tx=0x7fc5ec004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.559+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec019070 con 0x7fc5f41efa30 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.559+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec007d60 con 0x7fc5f41efa30 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.559+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec014640 con 0x7fc5f41efa30 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.559+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f41f0190 con 0x7fc5f41efa30 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.559+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f4140c10 con 0x7fc5f41efa30 2024-03-06T07:34:11.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.560+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f404f330 con 0x7fc5f41efa30 2024-03-06T07:34:11.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.561+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5ec01c070 con 0x7fc5f41efa30 2024-03-06T07:34:11.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.562+0000 7fc5eb400640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc5d406b4d0 0x7fc5d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.562+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5ec089d70 con 0x7fc5f41efa30 2024-03-06T07:34:11.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.562+0000 7fc5f3400640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc5d406b4d0 0x7fc5d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.563+0000 7fc5f3400640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc5d406b4d0 0x7fc5d406d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4000c00 tx=0x7fc5e4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.567+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5ec058be0 con 0x7fc5f41efa30 2024-03-06T07:34:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:11.806+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc5f406e240 con 0x7fc5f41efa30 2024-03-06T07:34:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.306+0000 7fc5eb400640 1 -- 172.21.15.32:0/189314209 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc5ec0584e0 con 0x7fc5f41efa30 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc5d406b4d0 msgr2=0x7fc5d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc5d406b4d0 0x7fc5d406d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4000c00 tx=0x7fc5e4009040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 msgr2=0x7fc5f4141980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 0x7fc5f4141980 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc5f406df70 tx=0x7fc5ec004840 comp rx=0 tx=0).stop 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 shutdown_connections 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc5d406b4d0 0x7fc5d406d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5f41efa30 0x7fc5f4141980 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5f413af50 0x7fc5f41ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 --2- 172.21.15.32:0/189314209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5d8004800 0x7fc5f41406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 >> 172.21.15.32:0/189314209 conn(0x7fc5f406b3a0 msgr2=0x7fc5f40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 shutdown_connections 2024-03-06T07:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.310+0000 7fc5f9200640 1 -- 172.21.15.32:0/189314209 wait complete. 2024-03-06T07:34:12.334 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:34:12.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:12.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.553+0000 7ff543600640 1 Processor -- start 2024-03-06T07:34:12.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.553+0000 7ff543600640 1 -- start start 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.554+0000 7ff543600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff53c0af1a0 0x7ff53c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff543600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff53c0af580 0x7ff53c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff543600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0cdc20 0x7ff53c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff543600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c15cc40 con 0x7ff53c0af1a0 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff543600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c15cf20 con 0x7ff53c0af580 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff543600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c15d200 con 0x7ff53c0cdc20 2024-03-06T07:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff540c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff53c0af1a0 0x7ff53c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff540c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff53c0af1a0 0x7ff53c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47048/0 (socket says 172.21.15.32:47048) 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff540c00640 1 -- 172.21.15.32:0/2759559371 learned_addr learned my addr 172.21.15.32:0/2759559371 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff533e00640 1 --2- 172.21.15.32:0/2759559371 >> v2:172.21.15.39:3300/0 conn(0x7ff53c0af580 0x7ff53c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.555+0000 7ff541600640 1 --2- 172.21.15.32:0/2759559371 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0cdc20 0x7ff53c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.556+0000 7ff540c00640 1 -- 172.21.15.32:0/2759559371 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0cdc20 msgr2=0x7ff53c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.556+0000 7ff540c00640 1 --2- 172.21.15.32:0/2759559371 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0cdc20 0x7ff53c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.556+0000 7ff540c00640 1 -- 172.21.15.32:0/2759559371 >> v2:172.21.15.39:3300/0 conn(0x7ff53c0af580 msgr2=0x7ff53c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.556+0000 7ff540c00640 1 --2- 172.21.15.32:0/2759559371 >> v2:172.21.15.39:3300/0 conn(0x7ff53c0af580 0x7ff53c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.556+0000 7ff540c00640 1 -- 172.21.15.32:0/2759559371 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c15d4e0 con 0x7ff53c0af1a0 2024-03-06T07:34:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.557+0000 7ff540c00640 1 --2- 172.21.15.32:0/2759559371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c15c600 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0b54c0 tx=0x7ff52c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77292a060268fa35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.558+0000 7ff53be00640 1 -- 172.21.15.32:0/2759559371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0317d0 con 0x7ff53c0af1a0 2024-03-06T07:34:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.558+0000 7ff53be00640 1 -- 172.21.15.32:0/2759559371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c037070 con 0x7ff53c0af1a0 2024-03-06T07:34:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.558+0000 7ff53be00640 1 -- 172.21.15.32:0/2759559371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c031b90 con 0x7ff53c0af1a0 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.558+0000 7ff543600640 1 -- 172.21.15.32:0/2759559371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 msgr2=0x7ff53c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.558+0000 7ff543600640 1 --2- 172.21.15.32:0/2759559371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c15c600 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0b54c0 tx=0x7ff52c02b870 comp rx=0 tx=0).stop 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 -- 172.21.15.32:0/2759559371 shutdown_connections 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 --2- 172.21.15.32:0/2759559371 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0cdc20 0x7ff53c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 --2- 172.21.15.32:0/2759559371 >> v2:172.21.15.39:3300/0 conn(0x7ff53c0af580 0x7ff53c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 --2- 172.21.15.32:0/2759559371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c15c600 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 -- 172.21.15.32:0/2759559371 >> 172.21.15.32:0/2759559371 conn(0x7ff53c06afb0 msgr2=0x7ff53c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 -- 172.21.15.32:0/2759559371 shutdown_connections 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.559+0000 7ff543600640 1 -- 172.21.15.32:0/2759559371 wait complete. 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.560+0000 7ff543600640 1 Processor -- start 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.561+0000 7ff543600640 1 -- start start 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.561+0000 7ff543600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c09d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff543600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0af580 0x7ff53c148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff543600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53c0cdc20 0x7ff53c133a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff543600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0b3490 con 0x7ff53c0af1a0 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff543600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0b3310 con 0x7ff53c0cdc20 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff543600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0b3610 con 0x7ff53c0af580 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c09d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c09d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47052/0 (socket says 172.21.15.32:47052) 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 -- 172.21.15.32:0/363625927 learned_addr learned my addr 172.21.15.32:0/363625927 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff533e00640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0af580 0x7ff53c148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff541600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53c0cdc20 0x7ff53c133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 -- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0af580 msgr2=0x7ff53c148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0af580 0x7ff53c148670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 -- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53c0cdc20 msgr2=0x7ff53c133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53c0cdc20 0x7ff53c133a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.562+0000 7ff540c00640 1 -- 172.21.15.32:0/363625927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c151bf0 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.563+0000 7ff540c00640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c09d1d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ff52c02b8b0 tx=0x7ff52c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.563+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c032660 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.563+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c032820 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.563+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c131080 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.563+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c032c10 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.563+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c131640 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.564+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c069510 con 0x7ff53c0af1a0 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.565+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff52c04c060 con 0x7ff53c0af1a0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.566+0000 7ff539600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff51806b4d0 0x7ff51806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.566+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff52c0aba80 con 0x7ff53c0af1a0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.566+0000 7ff533e00640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff51806b4d0 0x7ff51806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.567+0000 7ff533e00640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff51806b4d0 0x7ff51806d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff53c09d710 tx=0x7ff524008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.571+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff52c040070 con 0x7ff53c0af1a0 2024-03-06T07:34:12.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.809+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff53c06ed60 con 0x7ff53c0af1a0 2024-03-06T07:34:12.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.810+0000 7ff539600640 1 -- 172.21.15.32:0/363625927 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4973 (secure 0 0 0) 0x7ff52c07a2b0 con 0x7ff53c0af1a0 2024-03-06T07:34:12.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:12.812 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5887,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2823770992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2823770992},{"type":"v1","addr":"172.21.15.39:6834","nonce":2823770992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/415659526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":415659526},{"type":"v1","addr":"172.21.15.32:6836","nonce":415659526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2586752485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":2586752485},{"type":"v1","addr":"172.21.15.32:6837","nonce":2586752485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:11.808218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/556051904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":556051904},{"type":"v1","addr":"172.21.15.39:6835","nonce":556051904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7122,"qdb_cluster":[7122]},"id":7}]} 2024-03-06T07:34:12.812 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 61 2024-03-06T07:34:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff51806b4d0 msgr2=0x7ff51806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff51806b4d0 0x7ff51806d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff53c09d710 tx=0x7ff524008040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 msgr2=0x7ff53c09d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c09d1d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ff52c02b8b0 tx=0x7ff52c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 shutdown_connections 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff51806b4d0 0x7ff51806d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53c0cdc20 0x7ff53c133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff53c0af580 0x7ff53c148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 --2- 172.21.15.32:0/363625927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff53c0af1a0 0x7ff53c09d1d0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 >> 172.21.15.32:0/363625927 conn(0x7ff53c06afb0 msgr2=0x7ff53c0cbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 shutdown_connections 2024-03-06T07:34:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:12.813+0000 7ff543600640 1 -- 172.21.15.32:0/363625927 wait complete. 2024-03-06T07:34:12.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:13.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.049+0000 7f395d000640 1 Processor -- start 2024-03-06T07:34:13.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.049+0000 7f395d000640 1 -- start start 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.050+0000 7f395d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39580d0e50 0x7f3958168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.050+0000 7f395d000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39580b3710 con 0x7f395815e0b0 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.050+0000 7f395d000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39580b39f0 con 0x7f39580d4a10 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.050+0000 7f395d000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39580b09e0 con 0x7f39580d0e50 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3957400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f395815e0b0 0x7f39580cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44946/0 (socket says 172.21.15.32:44946) 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3957400640 1 -- 172.21.15.32:0/4107146898 learned_addr learned my addr 172.21.15.32:0/4107146898 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3957e00640 1 --2- 172.21.15.32:0/4107146898 >> v2:172.21.15.32:3300/0 conn(0x7f39580d0e50 0x7f3958168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3956000640 1 -- 172.21.15.32:0/4107146898 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349576776 0 0) 0x7f39580b3710 con 0x7f395815e0b0 2024-03-06T07:34:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3956000640 1 -- 172.21.15.32:0/4107146898 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f393c003520 con 0x7f395815e0b0 2024-03-06T07:34:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3957e00640 1 -- 172.21.15.32:0/4107146898 >> v1:172.21.15.39:6789/0 conn(0x7f39580d4a10 legacy=0x7f39580d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3957e00640 1 -- 172.21.15.32:0/4107146898 >> v1:172.21.15.32:6789/0 conn(0x7f395815e0b0 legacy=0x7f39580cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.051+0000 7f3957e00640 1 -- 172.21.15.32:0/4107146898 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39580b3cd0 con 0x7f39580d0e50 2024-03-06T07:34:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f3957e00640 1 --2- 172.21.15.32:0/4107146898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39580d0e50 0x7f3958168c50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3950009d90 tx=0x7f395002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b83cab184fd30b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f3956000640 1 -- 172.21.15.32:0/4107146898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950039070 con 0x7f39580d0e50 2024-03-06T07:34:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f3956000640 1 -- 172.21.15.32:0/4107146898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395002cd00 con 0x7f39580d0e50 2024-03-06T07:34:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f3956000640 1 -- 172.21.15.32:0/4107146898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950034660 con 0x7f39580d0e50 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f395d000640 1 -- 172.21.15.32:0/4107146898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39580d0e50 msgr2=0x7f3958168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f395d000640 1 --2- 172.21.15.32:0/4107146898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39580d0e50 0x7f3958168c50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3950009d90 tx=0x7f395002bf50 comp rx=0 tx=0).stop 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f395d000640 1 -- 172.21.15.32:0/4107146898 shutdown_connections 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f395d000640 1 --2- 172.21.15.32:0/4107146898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39580d0e50 0x7f3958168c50 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.053+0000 7f395d000640 1 -- 172.21.15.32:0/4107146898 >> 172.21.15.32:0/4107146898 conn(0x7f39580d3d30 msgr2=0x7f39580d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.054+0000 7f395d000640 1 -- 172.21.15.32:0/4107146898 shutdown_connections 2024-03-06T07:34:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.054+0000 7f395d000640 1 -- 172.21.15.32:0/4107146898 wait complete. 2024-03-06T07:34:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.055+0000 7f395d000640 1 Processor -- start 2024-03-06T07:34:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.055+0000 7f395d000640 1 -- start start 2024-03-06T07:34:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.056+0000 7f395d000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39580d0e50 0x7f39580cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.056+0000 7f395d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39580d4a10 0x7f39580b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.056+0000 7f395d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 0x7f39581f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.056+0000 7f395d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395815e490 con 0x7f395815e0b0 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.056+0000 7f395d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39580cfa20 con 0x7f39580d4a10 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.056+0000 7f395d000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39580cfc60 con 0x7f39580d0e50 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 0x7f39581f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 0x7f39581f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47066/0 (socket says 172.21.15.32:47066) 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 -- 172.21.15.32:0/2358704570 learned_addr learned my addr 172.21.15.32:0/2358704570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3956a00640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39580d4a10 0x7f39580b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 -- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39580d0e50 msgr2=0x7f39580cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39580d0e50 0x7f39580cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 -- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39580d4a10 msgr2=0x7f39580b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39580d4a10 0x7f39580b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 -- 172.21.15.32:0/2358704570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39580b6a80 con 0x7f395815e0b0 2024-03-06T07:34:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.057+0000 7f3957e00640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 0x7f39581f72b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3950009d90 tx=0x7f3950004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.058+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950039050 con 0x7f395815e0b0 2024-03-06T07:34:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.058+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3950045070 con 0x7f395815e0b0 2024-03-06T07:34:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.058+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39580b6d60 con 0x7f395815e0b0 2024-03-06T07:34:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.058+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950009450 con 0x7f395815e0b0 2024-03-06T07:34:13.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.058+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39580b7320 con 0x7f395815e0b0 2024-03-06T07:34:13.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.059+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395804f330 con 0x7f395815e0b0 2024-03-06T07:34:13.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.060+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f395003e850 con 0x7f395815e0b0 2024-03-06T07:34:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.061+0000 7f394f400640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f393406b4d0 0x7f393406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.061+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39500ace50 con 0x7f395815e0b0 2024-03-06T07:34:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.061+0000 7f3957400640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f393406b4d0 0x7f393406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.062+0000 7f3957400640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f393406b4d0 0x7f393406d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3948001570 tx=0x7f3948007930 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.066+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3950032e20 con 0x7f395815e0b0 2024-03-06T07:34:13.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.271+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3958161640 con 0x7f395815e0b0 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.273+0000 7f394f400640 1 -- 172.21.15.32:0/2358704570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16500 (secure 0 0 0) 0x7f395007bd60 con 0x7f395815e0b0 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:13.275 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":95,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:10.274366+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f393406b4d0 msgr2=0x7f393406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f393406b4d0 0x7f393406d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3948001570 tx=0x7f3948007930 comp rx=0 tx=0).stop 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 msgr2=0x7f39581f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 0x7f39581f72b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3950009d90 tx=0x7f3950004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 shutdown_connections 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f393406b4d0 0x7f393406d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f395815e0b0 0x7f39581f72b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39580d4a10 0x7f39580b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 --2- 172.21.15.32:0/2358704570 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39580d0e50 0x7f39580cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 >> 172.21.15.32:0/2358704570 conn(0x7f39580d3d30 msgr2=0x7f39580b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 shutdown_connections 2024-03-06T07:34:13.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.276+0000 7f395d000640 1 -- 172.21.15.32:0/2358704570 wait complete. 2024-03-06T07:34:13.298 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:34:13.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:13.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.510+0000 7faa13e00640 1 Processor -- start 2024-03-06T07:34:13.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.511+0000 7faa13e00640 1 -- start start 2024-03-06T07:34:13.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.512+0000 7faa13e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faa0c0cf5c0 0x7faa0c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.512+0000 7faa13e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c071050 0x7faa0c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.512+0000 7faa13e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c0b4860 con 0x7faa0c0b3c90 2024-03-06T07:34:13.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.512+0000 7faa13e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0c0ce160 con 0x7faa0c0cf5c0 2024-03-06T07:34:13.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.512+0000 7faa13e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c0b4b40 con 0x7faa0c071050 2024-03-06T07:34:13.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.513+0000 7faa11e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c071050 0x7faa0c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.513+0000 7faa11400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7faa0c0b3c90 0x7faa0c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:57508/0 (socket says 172.21.15.32:57508) 2024-03-06T07:34:13.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.513+0000 7faa11400640 1 -- 172.21.15.32:0/735659302 learned_addr learned my addr 172.21.15.32:0/735659302 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:13.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.513+0000 7faa10a00640 1 --2- 172.21.15.32:0/735659302 >> v2:172.21.15.39:3300/0 conn(0x7faa0c0cf5c0 0x7faa0c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.514+0000 7faa0be00640 1 -- 172.21.15.32:0/735659302 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949631264 0 0) 0x7faa0c0b4860 con 0x7faa0c0b3c90 2024-03-06T07:34:13.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.514+0000 7faa0be00640 1 -- 172.21.15.32:0/735659302 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f4003520 con 0x7faa0c0b3c90 2024-03-06T07:34:13.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.514+0000 7faa11e00640 1 -- 172.21.15.32:0/735659302 >> v2:172.21.15.39:3300/0 conn(0x7faa0c0cf5c0 msgr2=0x7faa0c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.514+0000 7faa11e00640 1 --2- 172.21.15.32:0/735659302 >> v2:172.21.15.39:3300/0 conn(0x7faa0c0cf5c0 0x7faa0c0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.514+0000 7faa11e00640 1 -- 172.21.15.32:0/735659302 >> v1:172.21.15.39:6789/0 conn(0x7faa0c0b3c90 legacy=0x7faa0c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.514+0000 7faa11e00640 1 -- 172.21.15.32:0/735659302 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c0b4e20 con 0x7faa0c071050 2024-03-06T07:34:13.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.515+0000 7faa11e00640 1 --2- 172.21.15.32:0/735659302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c071050 0x7faa0c071490 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faa04009d90 tx=0x7faa0402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a66fcaafd10b1d24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa0be00640 1 -- 172.21.15.32:0/735659302 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04039070 con 0x7faa0c071050 2024-03-06T07:34:13.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa0be00640 1 -- 172.21.15.32:0/735659302 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0402cce0 con 0x7faa0c071050 2024-03-06T07:34:13.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa0be00640 1 -- 172.21.15.32:0/735659302 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04034640 con 0x7faa0c071050 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa13e00640 1 -- 172.21.15.32:0/735659302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c071050 msgr2=0x7faa0c071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa13e00640 1 --2- 172.21.15.32:0/735659302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c071050 0x7faa0c071490 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faa04009d90 tx=0x7faa0402bf30 comp rx=0 tx=0).stop 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa13e00640 1 -- 172.21.15.32:0/735659302 shutdown_connections 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa13e00640 1 --2- 172.21.15.32:0/735659302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c071050 0x7faa0c071490 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa13e00640 1 --2- 172.21.15.32:0/735659302 >> v2:172.21.15.39:3300/0 conn(0x7faa0c0cf5c0 0x7faa0c0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.516+0000 7faa13e00640 1 -- 172.21.15.32:0/735659302 >> 172.21.15.32:0/735659302 conn(0x7faa0c06afb0 msgr2=0x7faa0c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.517+0000 7faa13e00640 1 -- 172.21.15.32:0/735659302 shutdown_connections 2024-03-06T07:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.517+0000 7faa13e00640 1 -- 172.21.15.32:0/735659302 wait complete. 2024-03-06T07:34:13.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.519+0000 7faa13e00640 1 Processor -- start 2024-03-06T07:34:13.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.519+0000 7faa13e00640 1 -- start start 2024-03-06T07:34:13.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa0c071050 0x7faa0c154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 0x7faa0c0caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa13e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c0cf5c0 0x7faa0c0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa13e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c0719d0 con 0x7faa0c0b3c90 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa13e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c15b340 con 0x7faa0c071050 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa13e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c071b50 con 0x7faa0c0cf5c0 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa10a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 0x7faa0c0caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa10a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 0x7faa0c0caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47068/0 (socket says 172.21.15.32:47068) 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa10a00640 1 -- 172.21.15.32:0/339040278 learned_addr learned my addr 172.21.15.32:0/339040278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa11400640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa0c071050 0x7faa0c154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.520+0000 7faa11e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c0cf5c0 0x7faa0c0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa10a00640 1 -- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c0cf5c0 msgr2=0x7faa0c0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa10a00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c0cf5c0 0x7faa0c0caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa10a00640 1 -- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa0c071050 msgr2=0x7faa0c154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa10a00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa0c071050 0x7faa0c154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa10a00640 1 -- 172.21.15.32:0/339040278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c0c93f0 con 0x7faa0c0b3c90 2024-03-06T07:34:13.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa11e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c0cf5c0 0x7faa0c0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa10a00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 0x7faa0c0caab0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800e8a0 tx=0x7fa9f800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8017070 con 0x7faa0c0b3c90 2024-03-06T07:34:13.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f800c940 con 0x7faa0c0b3c90 2024-03-06T07:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f800cd30 con 0x7faa0c0b3c90 2024-03-06T07:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c0cb780 con 0x7faa0c0b3c90 2024-03-06T07:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.521+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c0cbd40 con 0x7faa0c0b3c90 2024-03-06T07:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.522+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0c069510 con 0x7faa0c0b3c90 2024-03-06T07:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.523+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9f801a070 con 0x7faa0c0b3c90 2024-03-06T07:34:13.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.524+0000 7faa09600640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.524+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9f808a4d0 con 0x7faa0c0b3c90 2024-03-06T07:34:13.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.525+0000 7faa11400640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.525+0000 7faa11400640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7faa00005880 tx=0x7faa00009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.529+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9f80592b0 con 0x7faa0c0b3c90 2024-03-06T07:34:13.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.726+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa0c06cc90 con 0x7fa9ec06b4d0 2024-03-06T07:34:13.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.744+0000 7faa09600640 1 -- 172.21.15.32:0/339040278 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246771 (secure 0 0 0) 0x7faa0c06cc90 con 0x7fa9ec06b4d0 2024-03-06T07:34:13.750 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:13.751 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa9ec06b4d0 msgr2=0x7fa9ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7faa00005880 tx=0x7faa00009040 comp rx=0 tx=0).stop 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 msgr2=0x7faa0c0caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 0x7faa0c0caab0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800e8a0 tx=0x7fa9f800ed60 comp rx=0 tx=0).stop 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 shutdown_connections 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.750+0000 7faa13e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faa0c0cf5c0 0x7faa0c0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.751+0000 7faa13e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa0c0b3c90 0x7faa0c0caab0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.751+0000 7faa13e00640 1 --2- 172.21.15.32:0/339040278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa0c071050 0x7faa0c154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.751+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 >> 172.21.15.32:0/339040278 conn(0x7faa0c06afb0 msgr2=0x7faa0c0ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.751+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 shutdown_connections 2024-03-06T07:34:13.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:13.751+0000 7faa13e00640 1 -- 172.21.15.32:0/339040278 wait complete. 2024-03-06T07:34:13.773 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-03-06T07:34:12.281550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276152,"kb_used_data":10068,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698568,"statfs":{"total":773094113280,"available":772811333632,"internally_reserved":0,"allocated":10309632,"data_stored":5049819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.048977"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248310+0000","last_change":"2024-03-06T07:34:08.248310+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248310+0000","last_undegraded":"2024-03-06T07:34:08.248310+0000","last_fullsized":"2024-03-06T07:34:08.248310+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.265079+0000","last_change":"2024-03-06T07:34:08.265079+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.265079+0000","last_undegraded":"2024-03-06T07:34:08.265079+0000","last_fullsized":"2024-03-06T07:34:08.265079+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248065+0000","last_change":"2024-03-06T07:34:08.248065+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248065+0000","last_undegraded":"2024-03-06T07:34:08.248065+0000","last_fullsized":"2024-03-06T07:34:08.248065+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249593+0000","last_change":"2024-03-06T07:34:08.249593+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249593+0000","last_undegraded":"2024-03-06T07:34:08.249593+0000","last_fullsized":"2024-03-06T07:34:08.249593+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259488+0000","last_change":"2024-03-06T07:34:08.259488+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259488+0000","last_undegraded":"2024-03-06T07:34:08.259488+0000","last_fullsized":"2024-03-06T07:34:08.259488+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253745+0000","last_change":"2024-03-06T07:34:08.253745+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253745+0000","last_undegraded":"2024-03-06T07:34:08.253745+0000","last_fullsized":"2024-03-06T07:34:08.253745+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.251376+0000","last_change":"2024-03-06T07:34:08.251376+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.251376+0000","last_undegraded":"2024-03-06T07:34:08.251376+0000","last_fullsized":"2024-03-06T07:34:08.251376+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.256039+0000","last_change":"2024-03-06T07:34:08.256039+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.256039+0000","last_undegraded":"2024-03-06T07:34:08.256039+0000","last_fullsized":"2024-03-06T07:34:08.256039+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.254185+0000","last_change":"2024-03-06T07:34:08.254185+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.254185+0000","last_undegraded":"2024-03-06T07:34:08.254185+0000","last_fullsized":"2024-03-06T07:34:08.254185+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258787+0000","last_change":"2024-03-06T07:34:08.258787+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258787+0000","last_undegraded":"2024-03-06T07:34:08.258787+0000","last_fullsized":"2024-03-06T07:34:08.258787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.255571+0000","last_change":"2024-03-06T07:34:08.255571+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.255571+0000","last_undegraded":"2024-03-06T07:34:08.255571+0000","last_fullsized":"2024-03-06T07:34:08.255571+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253834+0000","last_change":"2024-03-06T07:34:08.253834+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253834+0000","last_undegraded":"2024-03-06T07:34:08.253834+0000","last_fullsized":"2024-03-06T07:34:08.253834+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248210+0000","last_change":"2024-03-06T07:34:08.248210+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248210+0000","last_undegraded":"2024-03-06T07:34:08.248210+0000","last_fullsized":"2024-03-06T07:34:08.248210+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.265128+0000","last_change":"2024-03-06T07:34:08.265128+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.265128+0000","last_undegraded":"2024-03-06T07:34:08.265128+0000","last_fullsized":"2024-03-06T07:34:08.265128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249662+0000","last_change":"2024-03-06T07:34:08.249662+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249662+0000","last_undegraded":"2024-03-06T07:34:08.249662+0000","last_fullsized":"2024-03-06T07:34:08.249662+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.252716+0000","last_change":"2024-03-06T07:34:08.252716+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.252716+0000","last_undegraded":"2024-03-06T07:34:08.252716+0000","last_fullsized":"2024-03-06T07:34:08.252716+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.262163+0000","last_change":"2024-03-06T07:34:08.262163+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.262163+0000","last_undegraded":"2024-03-06T07:34:08.262163+0000","last_fullsized":"2024-03-06T07:34:08.262163+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248858+0000","last_change":"2024-03-06T07:34:08.248858+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248858+0000","last_undegraded":"2024-03-06T07:34:08.248858+0000","last_fullsized":"2024-03-06T07:34:08.248858+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258235+0000","last_change":"2024-03-06T07:34:08.258235+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258235+0000","last_undegraded":"2024-03-06T07:34:08.258235+0000","last_fullsized":"2024-03-06T07:34:08.258235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.250449+0000","last_change":"2024-03-06T07:34:08.250449+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.250449+0000","last_undegraded":"2024-03-06T07:34:08.250449+0000","last_fullsized":"2024-03-06T07:34:08.250449+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.256431+0000","last_change":"2024-03-06T07:34:08.256431+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.256431+0000","last_undegraded":"2024-03-06T07:34:08.256431+0000","last_fullsized":"2024-03-06T07:34:08.256431+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.255938+0000","last_change":"2024-03-06T07:34:08.255938+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.255938+0000","last_undegraded":"2024-03-06T07:34:08.255938+0000","last_fullsized":"2024-03-06T07:34:08.255938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.257872+0000","last_change":"2024-03-06T07:34:08.257872+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.257872+0000","last_undegraded":"2024-03-06T07:34:08.257872+0000","last_fullsized":"2024-03-06T07:34:08.257872+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.251370+0000","last_change":"2024-03-06T07:34:08.251370+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.251370+0000","last_undegraded":"2024-03-06T07:34:08.251370+0000","last_fullsized":"2024-03-06T07:34:08.251370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.261917+0000","last_change":"2024-03-06T07:34:08.261917+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.261917+0000","last_undegraded":"2024-03-06T07:34:08.261917+0000","last_fullsized":"2024-03-06T07:34:08.261917+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.264678+0000","last_change":"2024-03-06T07:34:08.264678+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.264678+0000","last_undegraded":"2024-03-06T07:34:08.264678+0000","last_fullsized":"2024-03-06T07:34:08.264678+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.252235+0000","last_change":"2024-03-06T07:34:08.252235+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.252235+0000","last_undegraded":"2024-03-06T07:34:08.252235+0000","last_fullsized":"2024-03-06T07:34:08.252235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.262084+0000","last_change":"2024-03-06T07:34:08.262084+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.262084+0000","last_undegraded":"2024-03-06T07:34:08.262084+0000","last_fullsized":"2024-03-06T07:34:08.262084+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258878+0000","last_change":"2024-03-06T07:34:08.258878+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258878+0000","last_undegraded":"2024-03-06T07:34:08.258878+0000","last_fullsized":"2024-03-06T07:34:08.258878+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.264700+0000","last_change":"2024-03-06T07:34:08.264700+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.264700+0000","last_undegraded":"2024-03-06T07:34:08.264700+0000","last_fullsized":"2024-03-06T07:34:08.264700+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248179+0000","last_change":"2024-03-06T07:34:08.248179+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248179+0000","last_undegraded":"2024-03-06T07:34:08.248179+0000","last_fullsized":"2024-03-06T07:34:08.248179+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249254+0000","last_change":"2024-03-06T07:34:08.249254+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249254+0000","last_undegraded":"2024-03-06T07:34:08.249254+0000","last_fullsized":"2024-03-06T07:34:08.249254+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258267+0000","last_change":"2024-03-06T07:34:08.258267+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258267+0000","last_undegraded":"2024-03-06T07:34:08.258267+0000","last_fullsized":"2024-03-06T07:34:08.258267+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.279958+0000","last_change":"2024-03-06T07:34:08.276064+0000","last_active":"2024-03-06T07:34:08.279958+0000","last_peered":"2024-03-06T07:34:08.279958+0000","last_clean":"2024-03-06T07:34:08.279958+0000","last_became_active":"2024-03-06T07:34:08.275801+0000","last_became_peered":"2024-03-06T07:34:08.275801+0000","last_unstale":"2024-03-06T07:34:08.279958+0000","last_undegraded":"2024-03-06T07:34:08.279958+0000","last_fullsized":"2024-03-06T07:34:08.279958+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:19.855458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249958+0000","last_change":"2024-03-06T07:34:08.249958+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249958+0000","last_undegraded":"2024-03-06T07:34:08.249958+0000","last_fullsized":"2024-03-06T07:34:08.249958+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281411+0000","last_change":"2024-03-06T07:34:08.277530+0000","last_active":"2024-03-06T07:34:08.281411+0000","last_peered":"2024-03-06T07:34:08.281411+0000","last_clean":"2024-03-06T07:34:08.281411+0000","last_became_active":"2024-03-06T07:34:08.277041+0000","last_became_peered":"2024-03-06T07:34:08.277041+0000","last_unstale":"2024-03-06T07:34:08.281411+0000","last_undegraded":"2024-03-06T07:34:08.281411+0000","last_fullsized":"2024-03-06T07:34:08.281411+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:24.201284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.247073+0000","last_change":"2024-03-06T07:34:08.247073+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.247073+0000","last_undegraded":"2024-03-06T07:34:08.247073+0000","last_fullsized":"2024-03-06T07:34:08.247073+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.276675+0000","last_change":"2024-03-06T07:34:08.270479+0000","last_active":"2024-03-06T07:34:08.276675+0000","last_peered":"2024-03-06T07:34:08.276675+0000","last_clean":"2024-03-06T07:34:08.276675+0000","last_became_active":"2024-03-06T07:34:08.270199+0000","last_became_peered":"2024-03-06T07:34:08.270199+0000","last_unstale":"2024-03-06T07:34:08.276675+0000","last_undegraded":"2024-03-06T07:34:08.276675+0000","last_fullsized":"2024-03-06T07:34:08.276675+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249279+0000","last_change":"2024-03-06T07:34:08.249279+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249279+0000","last_undegraded":"2024-03-06T07:34:08.249279+0000","last_fullsized":"2024-03-06T07:34:08.249279+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.280681+0000","last_change":"2024-03-06T07:34:08.277107+0000","last_active":"2024-03-06T07:34:08.280681+0000","last_peered":"2024-03-06T07:34:08.280681+0000","last_clean":"2024-03-06T07:34:08.280681+0000","last_became_active":"2024-03-06T07:34:08.276380+0000","last_became_peered":"2024-03-06T07:34:08.276380+0000","last_unstale":"2024-03-06T07:34:08.280681+0000","last_undegraded":"2024-03-06T07:34:08.280681+0000","last_fullsized":"2024-03-06T07:34:08.280681+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:24.201284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.266432+0000","last_change":"2024-03-06T07:34:08.266432+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.266432+0000","last_undegraded":"2024-03-06T07:34:08.266432+0000","last_fullsized":"2024-03-06T07:34:08.266432+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274418+0000","last_change":"2024-03-06T07:34:08.271587+0000","last_active":"2024-03-06T07:34:08.274418+0000","last_peered":"2024-03-06T07:34:08.274418+0000","last_clean":"2024-03-06T07:34:08.274418+0000","last_became_active":"2024-03-06T07:34:08.271319+0000","last_became_peered":"2024-03-06T07:34:08.271319+0000","last_unstale":"2024-03-06T07:34:08.274418+0000","last_undegraded":"2024-03-06T07:34:08.274418+0000","last_fullsized":"2024-03-06T07:34:08.274418+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:51.756006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.250555+0000","last_change":"2024-03-06T07:34:08.250555+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.250555+0000","last_undegraded":"2024-03-06T07:34:08.250555+0000","last_fullsized":"2024-03-06T07:34:08.250555+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278203+0000","last_change":"2024-03-06T07:34:08.275076+0000","last_active":"2024-03-06T07:34:08.278203+0000","last_peered":"2024-03-06T07:34:08.278203+0000","last_clean":"2024-03-06T07:34:08.278203+0000","last_became_active":"2024-03-06T07:34:08.274808+0000","last_became_peered":"2024-03-06T07:34:08.274808+0000","last_unstale":"2024-03-06T07:34:08.278203+0000","last_undegraded":"2024-03-06T07:34:08.278203+0000","last_fullsized":"2024-03-06T07:34:08.278203+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:49.865183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248225+0000","last_change":"2024-03-06T07:34:08.248225+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248225+0000","last_undegraded":"2024-03-06T07:34:08.248225+0000","last_fullsized":"2024-03-06T07:34:08.248225+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278806+0000","last_change":"2024-03-06T07:34:08.276324+0000","last_active":"2024-03-06T07:34:08.278806+0000","last_peered":"2024-03-06T07:34:08.278806+0000","last_clean":"2024-03-06T07:34:08.278806+0000","last_became_active":"2024-03-06T07:34:08.275963+0000","last_became_peered":"2024-03-06T07:34:08.275963+0000","last_unstale":"2024-03-06T07:34:08.278806+0000","last_undegraded":"2024-03-06T07:34:08.278806+0000","last_fullsized":"2024-03-06T07:34:08.278806+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:56.924294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.246810+0000","last_change":"2024-03-06T07:34:08.246810+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.246810+0000","last_undegraded":"2024-03-06T07:34:08.246810+0000","last_fullsized":"2024-03-06T07:34:08.246810+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.264410+0000","last_change":"2024-03-06T07:34:08.259184+0000","last_active":"2024-03-06T07:34:08.264410+0000","last_peered":"2024-03-06T07:34:08.264410+0000","last_clean":"2024-03-06T07:34:08.264410+0000","last_became_active":"2024-03-06T07:34:08.258879+0000","last_became_peered":"2024-03-06T07:34:08.258879+0000","last_unstale":"2024-03-06T07:34:08.264410+0000","last_undegraded":"2024-03-06T07:34:08.264410+0000","last_fullsized":"2024-03-06T07:34:08.264410+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:02.099209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.264165+0000","last_change":"2024-03-06T07:34:08.264165+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.264165+0000","last_undegraded":"2024-03-06T07:34:08.264165+0000","last_fullsized":"2024-03-06T07:34:08.264165+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274662+0000","last_change":"2024-03-06T07:34:08.272502+0000","last_active":"2024-03-06T07:34:08.274662+0000","last_peered":"2024-03-06T07:34:08.274662+0000","last_clean":"2024-03-06T07:34:08.274662+0000","last_became_active":"2024-03-06T07:34:08.272249+0000","last_became_peered":"2024-03-06T07:34:08.272249+0000","last_unstale":"2024-03-06T07:34:08.274662+0000","last_undegraded":"2024-03-06T07:34:08.274662+0000","last_fullsized":"2024-03-06T07:34:08.274662+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:45.980296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253063+0000","last_change":"2024-03-06T07:34:08.253063+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253063+0000","last_undegraded":"2024-03-06T07:34:08.253063+0000","last_fullsized":"2024-03-06T07:34:08.253063+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":8,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277747+0000","last_change":"2024-03-06T07:34:08.277747+0000","last_active":"2024-03-06T07:34:08.277747+0000","last_peered":"2024-03-06T07:34:08.277747+0000","last_clean":"2024-03-06T07:34:08.277747+0000","last_became_active":"2024-03-06T07:34:08.274892+0000","last_became_peered":"2024-03-06T07:34:08.274892+0000","last_unstale":"2024-03-06T07:34:08.277747+0000","last_undegraded":"2024-03-06T07:34:08.277747+0000","last_fullsized":"2024-03-06T07:34:08.277747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:44.005288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258295+0000","last_change":"2024-03-06T07:34:08.258295+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258295+0000","last_undegraded":"2024-03-06T07:34:08.258295+0000","last_fullsized":"2024-03-06T07:34:08.258295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281463+0000","last_change":"2024-03-06T07:34:08.277986+0000","last_active":"2024-03-06T07:34:08.281463+0000","last_peered":"2024-03-06T07:34:08.281463+0000","last_clean":"2024-03-06T07:34:08.281463+0000","last_became_active":"2024-03-06T07:34:08.276369+0000","last_became_peered":"2024-03-06T07:34:08.276369+0000","last_unstale":"2024-03-06T07:34:08.281463+0000","last_undegraded":"2024-03-06T07:34:08.281463+0000","last_fullsized":"2024-03-06T07:34:08.281463+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:16.465395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.251086+0000","last_change":"2024-03-06T07:34:08.251086+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.251086+0000","last_undegraded":"2024-03-06T07:34:08.251086+0000","last_fullsized":"2024-03-06T07:34:08.251086+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.272856+0000","last_change":"2024-03-06T07:34:08.265538+0000","last_active":"2024-03-06T07:34:08.272856+0000","last_peered":"2024-03-06T07:34:08.272856+0000","last_clean":"2024-03-06T07:34:08.272856+0000","last_became_active":"2024-03-06T07:34:08.264783+0000","last_became_peered":"2024-03-06T07:34:08.264783+0000","last_unstale":"2024-03-06T07:34:08.272856+0000","last_undegraded":"2024-03-06T07:34:08.272856+0000","last_fullsized":"2024-03-06T07:34:08.272856+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:06.914393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.247450+0000","last_change":"2024-03-06T07:34:08.247450+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.247450+0000","last_undegraded":"2024-03-06T07:34:08.247450+0000","last_fullsized":"2024-03-06T07:34:08.247450+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278419+0000","last_change":"2024-03-06T07:34:08.273221+0000","last_active":"2024-03-06T07:34:08.278419+0000","last_peered":"2024-03-06T07:34:08.278419+0000","last_clean":"2024-03-06T07:34:08.278419+0000","last_became_active":"2024-03-06T07:34:08.272960+0000","last_became_peered":"2024-03-06T07:34:08.272960+0000","last_unstale":"2024-03-06T07:34:08.278419+0000","last_undegraded":"2024-03-06T07:34:08.278419+0000","last_fullsized":"2024-03-06T07:34:08.278419+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:38.148614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.245117+0000","last_change":"2024-03-06T07:34:08.245117+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.245117+0000","last_undegraded":"2024-03-06T07:34:08.245117+0000","last_fullsized":"2024-03-06T07:34:08.245117+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274082+0000","last_change":"2024-03-06T07:34:08.269595+0000","last_active":"2024-03-06T07:34:08.274082+0000","last_peered":"2024-03-06T07:34:08.274082+0000","last_clean":"2024-03-06T07:34:08.274082+0000","last_became_active":"2024-03-06T07:34:08.269260+0000","last_became_peered":"2024-03-06T07:34:08.269260+0000","last_unstale":"2024-03-06T07:34:08.274082+0000","last_undegraded":"2024-03-06T07:34:08.274082+0000","last_fullsized":"2024-03-06T07:34:08.274082+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:12.309310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.246437+0000","last_change":"2024-03-06T07:34:08.246437+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.246437+0000","last_undegraded":"2024-03-06T07:34:08.246437+0000","last_fullsized":"2024-03-06T07:34:08.246437+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278224+0000","last_change":"2024-03-06T07:34:08.271511+0000","last_active":"2024-03-06T07:34:08.278224+0000","last_peered":"2024-03-06T07:34:08.278224+0000","last_clean":"2024-03-06T07:34:08.278224+0000","last_became_active":"2024-03-06T07:34:08.271119+0000","last_became_peered":"2024-03-06T07:34:08.271119+0000","last_unstale":"2024-03-06T07:34:08.278224+0000","last_undegraded":"2024-03-06T07:34:08.278224+0000","last_fullsized":"2024-03-06T07:34:08.278224+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:45.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.256734+0000","last_change":"2024-03-06T07:34:08.256734+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.256734+0000","last_undegraded":"2024-03-06T07:34:08.256734+0000","last_fullsized":"2024-03-06T07:34:08.256734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.272774+0000","last_change":"2024-03-06T07:34:08.269208+0000","last_active":"2024-03-06T07:34:08.272774+0000","last_peered":"2024-03-06T07:34:08.272774+0000","last_clean":"2024-03-06T07:34:08.272774+0000","last_became_active":"2024-03-06T07:34:08.268751+0000","last_became_peered":"2024-03-06T07:34:08.268751+0000","last_unstale":"2024-03-06T07:34:08.272774+0000","last_undegraded":"2024-03-06T07:34:08.272774+0000","last_fullsized":"2024-03-06T07:34:08.272774+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:02.270613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.260516+0000","last_change":"2024-03-06T07:34:08.260516+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.260516+0000","last_undegraded":"2024-03-06T07:34:08.260516+0000","last_fullsized":"2024-03-06T07:34:08.260516+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278072+0000","last_change":"2024-03-06T07:34:08.270938+0000","last_active":"2024-03-06T07:34:08.278072+0000","last_peered":"2024-03-06T07:34:08.278072+0000","last_clean":"2024-03-06T07:34:08.278072+0000","last_became_active":"2024-03-06T07:34:08.270180+0000","last_became_peered":"2024-03-06T07:34:08.270180+0000","last_unstale":"2024-03-06T07:34:08.278072+0000","last_undegraded":"2024-03-06T07:34:08.278072+0000","last_fullsized":"2024-03-06T07:34:08.278072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:35.842246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.265315+0000","last_change":"2024-03-06T07:34:08.265315+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.265315+0000","last_undegraded":"2024-03-06T07:34:08.265315+0000","last_fullsized":"2024-03-06T07:34:08.265315+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274510+0000","last_change":"2024-03-06T07:34:08.272524+0000","last_active":"2024-03-06T07:34:08.274510+0000","last_peered":"2024-03-06T07:34:08.274510+0000","last_clean":"2024-03-06T07:34:08.274510+0000","last_became_active":"2024-03-06T07:34:08.272078+0000","last_became_peered":"2024-03-06T07:34:08.272078+0000","last_unstale":"2024-03-06T07:34:08.274510+0000","last_undegraded":"2024-03-06T07:34:08.274510+0000","last_fullsized":"2024-03-06T07:34:08.274510+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:35.842246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249701+0000","last_change":"2024-03-06T07:34:08.249701+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249701+0000","last_undegraded":"2024-03-06T07:34:08.249701+0000","last_fullsized":"2024-03-06T07:34:08.249701+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277157+0000","last_change":"2024-03-06T07:34:08.275513+0000","last_active":"2024-03-06T07:34:08.277157+0000","last_peered":"2024-03-06T07:34:08.277157+0000","last_clean":"2024-03-06T07:34:08.277157+0000","last_became_active":"2024-03-06T07:34:08.275177+0000","last_became_peered":"2024-03-06T07:34:08.275177+0000","last_unstale":"2024-03-06T07:34:08.277157+0000","last_undegraded":"2024-03-06T07:34:08.277157+0000","last_fullsized":"2024-03-06T07:34:08.277157+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:45.286640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259886+0000","last_change":"2024-03-06T07:34:08.259886+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259886+0000","last_undegraded":"2024-03-06T07:34:08.259886+0000","last_fullsized":"2024-03-06T07:34:08.259886+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277630+0000","last_change":"2024-03-06T07:34:08.272959+0000","last_active":"2024-03-06T07:34:08.277630+0000","last_peered":"2024-03-06T07:34:08.277630+0000","last_clean":"2024-03-06T07:34:08.277630+0000","last_became_active":"2024-03-06T07:34:08.271962+0000","last_became_peered":"2024-03-06T07:34:08.271962+0000","last_unstale":"2024-03-06T07:34:08.277630+0000","last_undegraded":"2024-03-06T07:34:08.277630+0000","last_fullsized":"2024-03-06T07:34:08.277630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.298132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.255829+0000","last_change":"2024-03-06T07:34:08.255829+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.255829+0000","last_undegraded":"2024-03-06T07:34:08.255829+0000","last_fullsized":"2024-03-06T07:34:08.255829+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278927+0000","last_change":"2024-03-06T07:34:08.275800+0000","last_active":"2024-03-06T07:34:08.278927+0000","last_peered":"2024-03-06T07:34:08.278927+0000","last_clean":"2024-03-06T07:34:08.278927+0000","last_became_active":"2024-03-06T07:34:08.275476+0000","last_became_peered":"2024-03-06T07:34:08.275476+0000","last_unstale":"2024-03-06T07:34:08.278927+0000","last_undegraded":"2024-03-06T07:34:08.278927+0000","last_fullsized":"2024-03-06T07:34:08.278927+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:03.432186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259769+0000","last_change":"2024-03-06T07:34:08.259769+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259769+0000","last_undegraded":"2024-03-06T07:34:08.259769+0000","last_fullsized":"2024-03-06T07:34:08.259769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277934+0000","last_change":"2024-03-06T07:34:08.274027+0000","last_active":"2024-03-06T07:34:08.277934+0000","last_peered":"2024-03-06T07:34:08.277934+0000","last_clean":"2024-03-06T07:34:08.277934+0000","last_became_active":"2024-03-06T07:34:08.273695+0000","last_became_peered":"2024-03-06T07:34:08.273695+0000","last_unstale":"2024-03-06T07:34:08.277934+0000","last_undegraded":"2024-03-06T07:34:08.277934+0000","last_fullsized":"2024-03-06T07:34:08.277934+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:30.078168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259166+0000","last_change":"2024-03-06T07:34:08.259166+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259166+0000","last_undegraded":"2024-03-06T07:34:08.259166+0000","last_fullsized":"2024-03-06T07:34:08.259166+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.276327+0000","last_change":"2024-03-06T07:34:08.274029+0000","last_active":"2024-03-06T07:34:08.276327+0000","last_peered":"2024-03-06T07:34:08.276327+0000","last_clean":"2024-03-06T07:34:08.276327+0000","last_became_active":"2024-03-06T07:34:08.273759+0000","last_became_peered":"2024-03-06T07:34:08.273759+0000","last_unstale":"2024-03-06T07:34:08.276327+0000","last_undegraded":"2024-03-06T07:34:08.276327+0000","last_fullsized":"2024-03-06T07:34:08.276327+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:44.011669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.263721+0000","last_change":"2024-03-06T07:34:08.263721+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.263721+0000","last_undegraded":"2024-03-06T07:34:08.263721+0000","last_fullsized":"2024-03-06T07:34:08.263721+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.282257+0000","last_change":"2024-03-06T07:34:08.279992+0000","last_active":"2024-03-06T07:34:08.282257+0000","last_peered":"2024-03-06T07:34:08.282257+0000","last_clean":"2024-03-06T07:34:08.282257+0000","last_became_active":"2024-03-06T07:34:08.279058+0000","last_became_peered":"2024-03-06T07:34:08.279058+0000","last_unstale":"2024-03-06T07:34:08.282257+0000","last_undegraded":"2024-03-06T07:34:08.282257+0000","last_fullsized":"2024-03-06T07:34:08.282257+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:33.361496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.245275+0000","last_change":"2024-03-06T07:34:08.245275+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.245275+0000","last_undegraded":"2024-03-06T07:34:08.245275+0000","last_fullsized":"2024-03-06T07:34:08.245275+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277042+0000","last_change":"2024-03-06T07:34:08.271690+0000","last_active":"2024-03-06T07:34:08.277042+0000","last_peered":"2024-03-06T07:34:08.277042+0000","last_clean":"2024-03-06T07:34:08.277042+0000","last_became_active":"2024-03-06T07:34:08.271425+0000","last_became_peered":"2024-03-06T07:34:08.271425+0000","last_unstale":"2024-03-06T07:34:08.277042+0000","last_undegraded":"2024-03-06T07:34:08.277042+0000","last_fullsized":"2024-03-06T07:34:08.277042+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:41.468854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.245487+0000","last_change":"2024-03-06T07:34:08.245487+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.245487+0000","last_undegraded":"2024-03-06T07:34:08.245487+0000","last_fullsized":"2024-03-06T07:34:08.245487+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.280673+0000","last_change":"2024-03-06T07:34:08.276819+0000","last_active":"2024-03-06T07:34:08.280673+0000","last_peered":"2024-03-06T07:34:08.280673+0000","last_clean":"2024-03-06T07:34:08.280673+0000","last_became_active":"2024-03-06T07:34:08.276500+0000","last_became_peered":"2024-03-06T07:34:08.276500+0000","last_unstale":"2024-03-06T07:34:08.280673+0000","last_undegraded":"2024-03-06T07:34:08.280673+0000","last_fullsized":"2024-03-06T07:34:08.280673+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:22.315747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.257519+0000","last_change":"2024-03-06T07:34:08.257519+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.257519+0000","last_undegraded":"2024-03-06T07:34:08.257519+0000","last_fullsized":"2024-03-06T07:34:08.257519+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281676+0000","last_change":"2024-03-06T07:34:08.277478+0000","last_active":"2024-03-06T07:34:08.281676+0000","last_peered":"2024-03-06T07:34:08.281676+0000","last_clean":"2024-03-06T07:34:08.281676+0000","last_became_active":"2024-03-06T07:34:08.277186+0000","last_became_peered":"2024-03-06T07:34:08.277186+0000","last_unstale":"2024-03-06T07:34:08.281676+0000","last_undegraded":"2024-03-06T07:34:08.281676+0000","last_fullsized":"2024-03-06T07:34:08.281676+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:30.730009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253985+0000","last_change":"2024-03-06T07:34:08.253985+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253985+0000","last_undegraded":"2024-03-06T07:34:08.253985+0000","last_fullsized":"2024-03-06T07:34:08.253985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277608+0000","last_change":"2024-03-06T07:34:08.272451+0000","last_active":"2024-03-06T07:34:08.277608+0000","last_peered":"2024-03-06T07:34:08.277608+0000","last_clean":"2024-03-06T07:34:08.277608+0000","last_became_active":"2024-03-06T07:34:08.272175+0000","last_became_peered":"2024-03-06T07:34:08.272175+0000","last_unstale":"2024-03-06T07:34:08.277608+0000","last_undegraded":"2024-03-06T07:34:08.277608+0000","last_fullsized":"2024-03-06T07:34:08.277608+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:09.498839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248253+0000","last_change":"2024-03-06T07:34:08.248253+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248253+0000","last_undegraded":"2024-03-06T07:34:08.248253+0000","last_fullsized":"2024-03-06T07:34:08.248253+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277590+0000","last_change":"2024-03-06T07:34:08.274395+0000","last_active":"2024-03-06T07:34:08.277590+0000","last_peered":"2024-03-06T07:34:08.277590+0000","last_clean":"2024-03-06T07:34:08.277590+0000","last_became_active":"2024-03-06T07:34:08.274068+0000","last_became_peered":"2024-03-06T07:34:08.274068+0000","last_unstale":"2024-03-06T07:34:08.277590+0000","last_undegraded":"2024-03-06T07:34:08.277590+0000","last_fullsized":"2024-03-06T07:34:08.277590+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:45.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259318+0000","last_change":"2024-03-06T07:34:08.259318+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259318+0000","last_undegraded":"2024-03-06T07:34:08.259318+0000","last_fullsized":"2024-03-06T07:34:08.259318+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.265318+0000","last_change":"2024-03-06T07:34:08.257721+0000","last_active":"2024-03-06T07:34:08.265318+0000","last_peered":"2024-03-06T07:34:08.265318+0000","last_clean":"2024-03-06T07:34:08.265318+0000","last_became_active":"2024-03-06T07:34:08.257322+0000","last_became_peered":"2024-03-06T07:34:08.257322+0000","last_unstale":"2024-03-06T07:34:08.265318+0000","last_undegraded":"2024-03-06T07:34:08.265318+0000","last_fullsized":"2024-03-06T07:34:08.265318+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258236+0000","last_change":"2024-03-06T07:34:08.258236+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258236+0000","last_undegraded":"2024-03-06T07:34:08.258236+0000","last_fullsized":"2024-03-06T07:34:08.258236+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.280901+0000","last_change":"2024-03-06T07:34:08.279635+0000","last_active":"2024-03-06T07:34:08.280901+0000","last_peered":"2024-03-06T07:34:08.280901+0000","last_clean":"2024-03-06T07:34:08.280901+0000","last_became_active":"2024-03-06T07:34:08.279250+0000","last_became_peered":"2024-03-06T07:34:08.279250+0000","last_unstale":"2024-03-06T07:34:08.280901+0000","last_undegraded":"2024-03-06T07:34:08.280901+0000","last_fullsized":"2024-03-06T07:34:08.280901+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:06.126369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.257781+0000","last_change":"2024-03-06T07:34:08.257781+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.257781+0000","last_undegraded":"2024-03-06T07:34:08.257781+0000","last_fullsized":"2024-03-06T07:34:08.257781+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281358+0000","last_change":"2024-03-06T07:34:08.276832+0000","last_active":"2024-03-06T07:34:08.281358+0000","last_peered":"2024-03-06T07:34:08.281358+0000","last_clean":"2024-03-06T07:34:08.281358+0000","last_became_active":"2024-03-06T07:34:08.276488+0000","last_became_peered":"2024-03-06T07:34:08.276488+0000","last_unstale":"2024-03-06T07:34:08.281358+0000","last_undegraded":"2024-03-06T07:34:08.281358+0000","last_fullsized":"2024-03-06T07:34:08.281358+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:22.315747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":306,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.255371+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:34:08.255371+0000","last_peered":"2024-03-06T07:34:08.255371+0000","last_clean":"2024-03-06T07:34:08.255371+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:34:08.255371+0000","last_undegraded":"2024-03-06T07:34:08.255371+0000","last_fullsized":"2024-03-06T07:34:08.255371+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1744896,"data_stored":1079494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335388,"statfs":{"total":96636764160,"available":96599437312,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335388,"statfs":{"total":96636764160,"available":96599437312,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335388,"statfs":{"total":96636764160,"available":96599437312,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1732608,"data_stored":1073219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:13.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.035+0000 7f33c0a00640 1 Processor -- start 2024-03-06T07:34:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.035+0000 7f33c0a00640 1 -- start start 2024-03-06T07:34:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33c0a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f33bc070f10 0x7f33bc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33c0a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc0714d0 con 0x7f33bc0cce90 2024-03-06T07:34:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33c0a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc0717b0 con 0x7f33bc145300 2024-03-06T07:34:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33c0a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33bc0cd300 con 0x7f33bc070f10 2024-03-06T07:34:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33bbe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f33bc0cce90 0x7f33bc15c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44950/0 (socket says 172.21.15.32:44950) 2024-03-06T07:34:14.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33bbe00640 1 -- 172.21.15.32:0/3880325236 learned_addr learned my addr 172.21.15.32:0/3880325236 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:14.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.037+0000 7f33baa00640 1 --2- 172.21.15.32:0/3880325236 >> v2:172.21.15.39:3300/0 conn(0x7f33bc070f10 0x7f33bc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442172181 0 0) 0x7f33bc0714d0 con 0x7f33bc0cce90 2024-03-06T07:34:14.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a0003520 con 0x7f33bc0cce90 2024-03-06T07:34:14.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367960298 0 0) 0x7f33a0003520 con 0x7f33bc0cce90 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 >> v2:172.21.15.39:3300/0 conn(0x7f33bc070f10 msgr2=0x7f33bc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 --2- 172.21.15.32:0/3880325236 >> v2:172.21.15.39:3300/0 conn(0x7f33bc070f10 0x7f33bc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 >> v1:172.21.15.39:6789/0 conn(0x7f33bc145300 legacy=0x7f33bc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc071a90 con 0x7f33bc0cce90 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f33b40019f0 con 0x7f33bc0cce90 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 >> v1:172.21.15.32:6789/0 conn(0x7f33bc0cce90 legacy=0x7f33bc15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33a0003a00 0x7f33a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33a0005210 0x7f33a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0cd300 con 0x7f33a0003a00 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a000c160 con 0x7f33a0004800 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0010770 con 0x7f33a0005210 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.038+0000 7f33bb400640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33a0003a00 0x7f33a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bbe00640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33a0005210 0x7f33a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33baa00640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bb400640 1 -- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33a0005210 msgr2=0x7f33a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bb400640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33a0005210 0x7f33a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bb400640 1 -- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0004800 msgr2=0x7f33a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bb400640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bb400640 1 -- 172.21.15.32:0/3880325236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a0003520 con 0x7f33a0003a00 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bbe00640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33a0005210 0x7f33a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.039+0000 7f33bbe00640 1 -- 172.21.15.32:0/3880325236 reap_dead start 2024-03-06T07:34:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.040+0000 7f33bb400640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33a0003a00 0x7f33a0003e20 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f33ac001720 tx=0x7f33a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33709b44fe01e902 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.040+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac001370 con 0x7f33a0003a00 2024-03-06T07:34:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.040+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac0052d0 con 0x7f33a0003a00 2024-03-06T07:34:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.040+0000 7f33ba000640 1 -- 172.21.15.32:0/3880325236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac003830 con 0x7f33a0003a00 2024-03-06T07:34:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 -- 172.21.15.32:0/3880325236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33a0003a00 msgr2=0x7f33a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33a0003a00 0x7f33a0003e20 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f33ac001720 tx=0x7f33a0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 -- 172.21.15.32:0/3880325236 shutdown_connections 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 --2- 172.21.15.32:0/3880325236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33a0003a00 0x7f33a0003e20 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 -- 172.21.15.32:0/3880325236 >> 172.21.15.32:0/3880325236 conn(0x7f33bc06b3d0 msgr2=0x7f33bc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 -- 172.21.15.32:0/3880325236 shutdown_connections 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.041+0000 7f33c0a00640 1 -- 172.21.15.32:0/3880325236 wait complete. 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.043+0000 7f33c0a00640 1 Processor -- start 2024-03-06T07:34:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.043+0000 7f33c0a00640 1 -- start start 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33c0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0003a00 0x7f33bc0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33c0a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33bc15eb60 0x7f33bc15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33c0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 0x7f33bc15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33c0a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0cc950 con 0x7f33bc15efe0 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33c0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc1456e0 con 0x7f33a0003a00 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33c0a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0ccad0 con 0x7f33bc15eb60 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 0x7f33bc15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 0x7f33bc15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47080/0 (socket says 172.21.15.32:47080) 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33bbe00640 1 -- 172.21.15.32:0/4037400450 learned_addr learned my addr 172.21.15.32:0/4037400450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33bb400640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0003a00 0x7f33bc0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.044+0000 7f33baa00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33bc15eb60 0x7f33bc15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33bbe00640 1 -- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33bc15eb60 msgr2=0x7f33bc15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33bbe00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33bc15eb60 0x7f33bc15a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33bbe00640 1 -- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0003a00 msgr2=0x7f33bc0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33bbe00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0003a00 0x7f33bc0af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33bbe00640 1 -- 172.21.15.32:0/4037400450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc13c7e0 con 0x7f33bc15efe0 2024-03-06T07:34:14.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33bbe00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 0x7f33bc15ae40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f33bc072420 tx=0x7f33b400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b400bdb0 con 0x7f33bc15efe0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b4004b60 con 0x7f33bc15efe0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4007070 con 0x7f33bc15efe0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc13cac0 con 0x7f33bc15efe0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.045+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc13d080 con 0x7f33bc15efe0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.047+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc04f330 con 0x7f33bc15efe0 2024-03-06T07:34:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.047+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33b401b020 con 0x7f33bc15efe0 2024-03-06T07:34:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.048+0000 7f33b3400640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f339c06b4d0 0x7f339c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.049+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33b408a430 con 0x7f33bc15efe0 2024-03-06T07:34:14.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.049+0000 7f33bb400640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f339c06b4d0 0x7f339c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.049+0000 7f33bb400640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f339c06b4d0 0x7f339c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f33ac033000 tx=0x7f33ac009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.053+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33b4059210 con 0x7f33bc15efe0 2024-03-06T07:34:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.243+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33bc0cdc70 con 0x7f339c06b4d0 2024-03-06T07:34:14.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.251+0000 7f33b3400640 1 -- 172.21.15.32:0/4037400450 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246771 (secure 0 0 0) 0x7f33bc0cdc70 con 0x7f339c06b4d0 2024-03-06T07:34:14.256 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:14.258 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:34:14.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f339c06b4d0 msgr2=0x7f339c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f339c06b4d0 0x7f339c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f33ac033000 tx=0x7f33ac009030 comp rx=0 tx=0).stop 2024-03-06T07:34:14.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 msgr2=0x7f33bc15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 0x7f33bc15ae40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f33bc072420 tx=0x7f33b400b4a0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 shutdown_connections 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f339c06b4d0 0x7f339c06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33bc15efe0 0x7f33bc15ae40 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f33bc15eb60 0x7f33bc15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 --2- 172.21.15.32:0/4037400450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a0003a00 0x7f33bc0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 >> 172.21.15.32:0/4037400450 conn(0x7f33bc06b3d0 msgr2=0x7f33bc1444f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 shutdown_connections 2024-03-06T07:34:14.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.257+0000 7f33c0a00640 1 -- 172.21.15.32:0/4037400450 wait complete. 2024-03-06T07:34:14.277 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-03-06T07:34:12.281550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276152,"kb_used_data":10068,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698568,"statfs":{"total":773094113280,"available":772811333632,"internally_reserved":0,"allocated":10309632,"data_stored":5049819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.048977"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248310+0000","last_change":"2024-03-06T07:34:08.248310+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248310+0000","last_undegraded":"2024-03-06T07:34:08.248310+0000","last_fullsized":"2024-03-06T07:34:08.248310+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.265079+0000","last_change":"2024-03-06T07:34:08.265079+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.265079+0000","last_undegraded":"2024-03-06T07:34:08.265079+0000","last_fullsized":"2024-03-06T07:34:08.265079+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248065+0000","last_change":"2024-03-06T07:34:08.248065+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248065+0000","last_undegraded":"2024-03-06T07:34:08.248065+0000","last_fullsized":"2024-03-06T07:34:08.248065+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249593+0000","last_change":"2024-03-06T07:34:08.249593+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249593+0000","last_undegraded":"2024-03-06T07:34:08.249593+0000","last_fullsized":"2024-03-06T07:34:08.249593+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259488+0000","last_change":"2024-03-06T07:34:08.259488+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259488+0000","last_undegraded":"2024-03-06T07:34:08.259488+0000","last_fullsized":"2024-03-06T07:34:08.259488+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253745+0000","last_change":"2024-03-06T07:34:08.253745+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253745+0000","last_undegraded":"2024-03-06T07:34:08.253745+0000","last_fullsized":"2024-03-06T07:34:08.253745+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.251376+0000","last_change":"2024-03-06T07:34:08.251376+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.251376+0000","last_undegraded":"2024-03-06T07:34:08.251376+0000","last_fullsized":"2024-03-06T07:34:08.251376+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.256039+0000","last_change":"2024-03-06T07:34:08.256039+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.256039+0000","last_undegraded":"2024-03-06T07:34:08.256039+0000","last_fullsized":"2024-03-06T07:34:08.256039+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.254185+0000","last_change":"2024-03-06T07:34:08.254185+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.254185+0000","last_undegraded":"2024-03-06T07:34:08.254185+0000","last_fullsized":"2024-03-06T07:34:08.254185+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258787+0000","last_change":"2024-03-06T07:34:08.258787+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258787+0000","last_undegraded":"2024-03-06T07:34:08.258787+0000","last_fullsized":"2024-03-06T07:34:08.258787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.255571+0000","last_change":"2024-03-06T07:34:08.255571+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.255571+0000","last_undegraded":"2024-03-06T07:34:08.255571+0000","last_fullsized":"2024-03-06T07:34:08.255571+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253834+0000","last_change":"2024-03-06T07:34:08.253834+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253834+0000","last_undegraded":"2024-03-06T07:34:08.253834+0000","last_fullsized":"2024-03-06T07:34:08.253834+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248210+0000","last_change":"2024-03-06T07:34:08.248210+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248210+0000","last_undegraded":"2024-03-06T07:34:08.248210+0000","last_fullsized":"2024-03-06T07:34:08.248210+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.265128+0000","last_change":"2024-03-06T07:34:08.265128+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.265128+0000","last_undegraded":"2024-03-06T07:34:08.265128+0000","last_fullsized":"2024-03-06T07:34:08.265128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249662+0000","last_change":"2024-03-06T07:34:08.249662+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249662+0000","last_undegraded":"2024-03-06T07:34:08.249662+0000","last_fullsized":"2024-03-06T07:34:08.249662+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.252716+0000","last_change":"2024-03-06T07:34:08.252716+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.252716+0000","last_undegraded":"2024-03-06T07:34:08.252716+0000","last_fullsized":"2024-03-06T07:34:08.252716+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.262163+0000","last_change":"2024-03-06T07:34:08.262163+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.262163+0000","last_undegraded":"2024-03-06T07:34:08.262163+0000","last_fullsized":"2024-03-06T07:34:08.262163+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248858+0000","last_change":"2024-03-06T07:34:08.248858+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248858+0000","last_undegraded":"2024-03-06T07:34:08.248858+0000","last_fullsized":"2024-03-06T07:34:08.248858+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258235+0000","last_change":"2024-03-06T07:34:08.258235+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258235+0000","last_undegraded":"2024-03-06T07:34:08.258235+0000","last_fullsized":"2024-03-06T07:34:08.258235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.250449+0000","last_change":"2024-03-06T07:34:08.250449+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.250449+0000","last_undegraded":"2024-03-06T07:34:08.250449+0000","last_fullsized":"2024-03-06T07:34:08.250449+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.256431+0000","last_change":"2024-03-06T07:34:08.256431+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.256431+0000","last_undegraded":"2024-03-06T07:34:08.256431+0000","last_fullsized":"2024-03-06T07:34:08.256431+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.255938+0000","last_change":"2024-03-06T07:34:08.255938+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.255938+0000","last_undegraded":"2024-03-06T07:34:08.255938+0000","last_fullsized":"2024-03-06T07:34:08.255938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.257872+0000","last_change":"2024-03-06T07:34:08.257872+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.257872+0000","last_undegraded":"2024-03-06T07:34:08.257872+0000","last_fullsized":"2024-03-06T07:34:08.257872+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.251370+0000","last_change":"2024-03-06T07:34:08.251370+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.251370+0000","last_undegraded":"2024-03-06T07:34:08.251370+0000","last_fullsized":"2024-03-06T07:34:08.251370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.261917+0000","last_change":"2024-03-06T07:34:08.261917+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.261917+0000","last_undegraded":"2024-03-06T07:34:08.261917+0000","last_fullsized":"2024-03-06T07:34:08.261917+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.264678+0000","last_change":"2024-03-06T07:34:08.264678+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.264678+0000","last_undegraded":"2024-03-06T07:34:08.264678+0000","last_fullsized":"2024-03-06T07:34:08.264678+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.252235+0000","last_change":"2024-03-06T07:34:08.252235+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.252235+0000","last_undegraded":"2024-03-06T07:34:08.252235+0000","last_fullsized":"2024-03-06T07:34:08.252235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.262084+0000","last_change":"2024-03-06T07:34:08.262084+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.262084+0000","last_undegraded":"2024-03-06T07:34:08.262084+0000","last_fullsized":"2024-03-06T07:34:08.262084+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258878+0000","last_change":"2024-03-06T07:34:08.258878+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258878+0000","last_undegraded":"2024-03-06T07:34:08.258878+0000","last_fullsized":"2024-03-06T07:34:08.258878+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.264700+0000","last_change":"2024-03-06T07:34:08.264700+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.264700+0000","last_undegraded":"2024-03-06T07:34:08.264700+0000","last_fullsized":"2024-03-06T07:34:08.264700+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248179+0000","last_change":"2024-03-06T07:34:08.248179+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248179+0000","last_undegraded":"2024-03-06T07:34:08.248179+0000","last_fullsized":"2024-03-06T07:34:08.248179+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249254+0000","last_change":"2024-03-06T07:34:08.249254+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249254+0000","last_undegraded":"2024-03-06T07:34:08.249254+0000","last_fullsized":"2024-03-06T07:34:08.249254+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258267+0000","last_change":"2024-03-06T07:34:08.258267+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258267+0000","last_undegraded":"2024-03-06T07:34:08.258267+0000","last_fullsized":"2024-03-06T07:34:08.258267+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.279958+0000","last_change":"2024-03-06T07:34:08.276064+0000","last_active":"2024-03-06T07:34:08.279958+0000","last_peered":"2024-03-06T07:34:08.279958+0000","last_clean":"2024-03-06T07:34:08.279958+0000","last_became_active":"2024-03-06T07:34:08.275801+0000","last_became_peered":"2024-03-06T07:34:08.275801+0000","last_unstale":"2024-03-06T07:34:08.279958+0000","last_undegraded":"2024-03-06T07:34:08.279958+0000","last_fullsized":"2024-03-06T07:34:08.279958+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:19.855458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249958+0000","last_change":"2024-03-06T07:34:08.249958+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249958+0000","last_undegraded":"2024-03-06T07:34:08.249958+0000","last_fullsized":"2024-03-06T07:34:08.249958+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281411+0000","last_change":"2024-03-06T07:34:08.277530+0000","last_active":"2024-03-06T07:34:08.281411+0000","last_peered":"2024-03-06T07:34:08.281411+0000","last_clean":"2024-03-06T07:34:08.281411+0000","last_became_active":"2024-03-06T07:34:08.277041+0000","last_became_peered":"2024-03-06T07:34:08.277041+0000","last_unstale":"2024-03-06T07:34:08.281411+0000","last_undegraded":"2024-03-06T07:34:08.281411+0000","last_fullsized":"2024-03-06T07:34:08.281411+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:24.201284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.247073+0000","last_change":"2024-03-06T07:34:08.247073+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.247073+0000","last_undegraded":"2024-03-06T07:34:08.247073+0000","last_fullsized":"2024-03-06T07:34:08.247073+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.276675+0000","last_change":"2024-03-06T07:34:08.270479+0000","last_active":"2024-03-06T07:34:08.276675+0000","last_peered":"2024-03-06T07:34:08.276675+0000","last_clean":"2024-03-06T07:34:08.276675+0000","last_became_active":"2024-03-06T07:34:08.270199+0000","last_became_peered":"2024-03-06T07:34:08.270199+0000","last_unstale":"2024-03-06T07:34:08.276675+0000","last_undegraded":"2024-03-06T07:34:08.276675+0000","last_fullsized":"2024-03-06T07:34:08.276675+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249279+0000","last_change":"2024-03-06T07:34:08.249279+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249279+0000","last_undegraded":"2024-03-06T07:34:08.249279+0000","last_fullsized":"2024-03-06T07:34:08.249279+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.280681+0000","last_change":"2024-03-06T07:34:08.277107+0000","last_active":"2024-03-06T07:34:08.280681+0000","last_peered":"2024-03-06T07:34:08.280681+0000","last_clean":"2024-03-06T07:34:08.280681+0000","last_became_active":"2024-03-06T07:34:08.276380+0000","last_became_peered":"2024-03-06T07:34:08.276380+0000","last_unstale":"2024-03-06T07:34:08.280681+0000","last_undegraded":"2024-03-06T07:34:08.280681+0000","last_fullsized":"2024-03-06T07:34:08.280681+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:24.201284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.266432+0000","last_change":"2024-03-06T07:34:08.266432+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.266432+0000","last_undegraded":"2024-03-06T07:34:08.266432+0000","last_fullsized":"2024-03-06T07:34:08.266432+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274418+0000","last_change":"2024-03-06T07:34:08.271587+0000","last_active":"2024-03-06T07:34:08.274418+0000","last_peered":"2024-03-06T07:34:08.274418+0000","last_clean":"2024-03-06T07:34:08.274418+0000","last_became_active":"2024-03-06T07:34:08.271319+0000","last_became_peered":"2024-03-06T07:34:08.271319+0000","last_unstale":"2024-03-06T07:34:08.274418+0000","last_undegraded":"2024-03-06T07:34:08.274418+0000","last_fullsized":"2024-03-06T07:34:08.274418+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:51.756006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.250555+0000","last_change":"2024-03-06T07:34:08.250555+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.250555+0000","last_undegraded":"2024-03-06T07:34:08.250555+0000","last_fullsized":"2024-03-06T07:34:08.250555+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278203+0000","last_change":"2024-03-06T07:34:08.275076+0000","last_active":"2024-03-06T07:34:08.278203+0000","last_peered":"2024-03-06T07:34:08.278203+0000","last_clean":"2024-03-06T07:34:08.278203+0000","last_became_active":"2024-03-06T07:34:08.274808+0000","last_became_peered":"2024-03-06T07:34:08.274808+0000","last_unstale":"2024-03-06T07:34:08.278203+0000","last_undegraded":"2024-03-06T07:34:08.278203+0000","last_fullsized":"2024-03-06T07:34:08.278203+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:49.865183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248225+0000","last_change":"2024-03-06T07:34:08.248225+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248225+0000","last_undegraded":"2024-03-06T07:34:08.248225+0000","last_fullsized":"2024-03-06T07:34:08.248225+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278806+0000","last_change":"2024-03-06T07:34:08.276324+0000","last_active":"2024-03-06T07:34:08.278806+0000","last_peered":"2024-03-06T07:34:08.278806+0000","last_clean":"2024-03-06T07:34:08.278806+0000","last_became_active":"2024-03-06T07:34:08.275963+0000","last_became_peered":"2024-03-06T07:34:08.275963+0000","last_unstale":"2024-03-06T07:34:08.278806+0000","last_undegraded":"2024-03-06T07:34:08.278806+0000","last_fullsized":"2024-03-06T07:34:08.278806+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:56.924294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.246810+0000","last_change":"2024-03-06T07:34:08.246810+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.246810+0000","last_undegraded":"2024-03-06T07:34:08.246810+0000","last_fullsized":"2024-03-06T07:34:08.246810+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.264410+0000","last_change":"2024-03-06T07:34:08.259184+0000","last_active":"2024-03-06T07:34:08.264410+0000","last_peered":"2024-03-06T07:34:08.264410+0000","last_clean":"2024-03-06T07:34:08.264410+0000","last_became_active":"2024-03-06T07:34:08.258879+0000","last_became_peered":"2024-03-06T07:34:08.258879+0000","last_unstale":"2024-03-06T07:34:08.264410+0000","last_undegraded":"2024-03-06T07:34:08.264410+0000","last_fullsized":"2024-03-06T07:34:08.264410+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:02.099209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.264165+0000","last_change":"2024-03-06T07:34:08.264165+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.264165+0000","last_undegraded":"2024-03-06T07:34:08.264165+0000","last_fullsized":"2024-03-06T07:34:08.264165+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274662+0000","last_change":"2024-03-06T07:34:08.272502+0000","last_active":"2024-03-06T07:34:08.274662+0000","last_peered":"2024-03-06T07:34:08.274662+0000","last_clean":"2024-03-06T07:34:08.274662+0000","last_became_active":"2024-03-06T07:34:08.272249+0000","last_became_peered":"2024-03-06T07:34:08.272249+0000","last_unstale":"2024-03-06T07:34:08.274662+0000","last_undegraded":"2024-03-06T07:34:08.274662+0000","last_fullsized":"2024-03-06T07:34:08.274662+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:45.980296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253063+0000","last_change":"2024-03-06T07:34:08.253063+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253063+0000","last_undegraded":"2024-03-06T07:34:08.253063+0000","last_fullsized":"2024-03-06T07:34:08.253063+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":8,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277747+0000","last_change":"2024-03-06T07:34:08.277747+0000","last_active":"2024-03-06T07:34:08.277747+0000","last_peered":"2024-03-06T07:34:08.277747+0000","last_clean":"2024-03-06T07:34:08.277747+0000","last_became_active":"2024-03-06T07:34:08.274892+0000","last_became_peered":"2024-03-06T07:34:08.274892+0000","last_unstale":"2024-03-06T07:34:08.277747+0000","last_undegraded":"2024-03-06T07:34:08.277747+0000","last_fullsized":"2024-03-06T07:34:08.277747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:44.005288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258295+0000","last_change":"2024-03-06T07:34:08.258295+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258295+0000","last_undegraded":"2024-03-06T07:34:08.258295+0000","last_fullsized":"2024-03-06T07:34:08.258295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281463+0000","last_change":"2024-03-06T07:34:08.277986+0000","last_active":"2024-03-06T07:34:08.281463+0000","last_peered":"2024-03-06T07:34:08.281463+0000","last_clean":"2024-03-06T07:34:08.281463+0000","last_became_active":"2024-03-06T07:34:08.276369+0000","last_became_peered":"2024-03-06T07:34:08.276369+0000","last_unstale":"2024-03-06T07:34:08.281463+0000","last_undegraded":"2024-03-06T07:34:08.281463+0000","last_fullsized":"2024-03-06T07:34:08.281463+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:16.465395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.251086+0000","last_change":"2024-03-06T07:34:08.251086+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.251086+0000","last_undegraded":"2024-03-06T07:34:08.251086+0000","last_fullsized":"2024-03-06T07:34:08.251086+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.272856+0000","last_change":"2024-03-06T07:34:08.265538+0000","last_active":"2024-03-06T07:34:08.272856+0000","last_peered":"2024-03-06T07:34:08.272856+0000","last_clean":"2024-03-06T07:34:08.272856+0000","last_became_active":"2024-03-06T07:34:08.264783+0000","last_became_peered":"2024-03-06T07:34:08.264783+0000","last_unstale":"2024-03-06T07:34:08.272856+0000","last_undegraded":"2024-03-06T07:34:08.272856+0000","last_fullsized":"2024-03-06T07:34:08.272856+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:06.914393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.247450+0000","last_change":"2024-03-06T07:34:08.247450+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.247450+0000","last_undegraded":"2024-03-06T07:34:08.247450+0000","last_fullsized":"2024-03-06T07:34:08.247450+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278419+0000","last_change":"2024-03-06T07:34:08.273221+0000","last_active":"2024-03-06T07:34:08.278419+0000","last_peered":"2024-03-06T07:34:08.278419+0000","last_clean":"2024-03-06T07:34:08.278419+0000","last_became_active":"2024-03-06T07:34:08.272960+0000","last_became_peered":"2024-03-06T07:34:08.272960+0000","last_unstale":"2024-03-06T07:34:08.278419+0000","last_undegraded":"2024-03-06T07:34:08.278419+0000","last_fullsized":"2024-03-06T07:34:08.278419+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:38.148614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.245117+0000","last_change":"2024-03-06T07:34:08.245117+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.245117+0000","last_undegraded":"2024-03-06T07:34:08.245117+0000","last_fullsized":"2024-03-06T07:34:08.245117+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274082+0000","last_change":"2024-03-06T07:34:08.269595+0000","last_active":"2024-03-06T07:34:08.274082+0000","last_peered":"2024-03-06T07:34:08.274082+0000","last_clean":"2024-03-06T07:34:08.274082+0000","last_became_active":"2024-03-06T07:34:08.269260+0000","last_became_peered":"2024-03-06T07:34:08.269260+0000","last_unstale":"2024-03-06T07:34:08.274082+0000","last_undegraded":"2024-03-06T07:34:08.274082+0000","last_fullsized":"2024-03-06T07:34:08.274082+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:12.309310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.246437+0000","last_change":"2024-03-06T07:34:08.246437+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.246437+0000","last_undegraded":"2024-03-06T07:34:08.246437+0000","last_fullsized":"2024-03-06T07:34:08.246437+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278224+0000","last_change":"2024-03-06T07:34:08.271511+0000","last_active":"2024-03-06T07:34:08.278224+0000","last_peered":"2024-03-06T07:34:08.278224+0000","last_clean":"2024-03-06T07:34:08.278224+0000","last_became_active":"2024-03-06T07:34:08.271119+0000","last_became_peered":"2024-03-06T07:34:08.271119+0000","last_unstale":"2024-03-06T07:34:08.278224+0000","last_undegraded":"2024-03-06T07:34:08.278224+0000","last_fullsized":"2024-03-06T07:34:08.278224+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:45.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.256734+0000","last_change":"2024-03-06T07:34:08.256734+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.256734+0000","last_undegraded":"2024-03-06T07:34:08.256734+0000","last_fullsized":"2024-03-06T07:34:08.256734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.272774+0000","last_change":"2024-03-06T07:34:08.269208+0000","last_active":"2024-03-06T07:34:08.272774+0000","last_peered":"2024-03-06T07:34:08.272774+0000","last_clean":"2024-03-06T07:34:08.272774+0000","last_became_active":"2024-03-06T07:34:08.268751+0000","last_became_peered":"2024-03-06T07:34:08.268751+0000","last_unstale":"2024-03-06T07:34:08.272774+0000","last_undegraded":"2024-03-06T07:34:08.272774+0000","last_fullsized":"2024-03-06T07:34:08.272774+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:02.270613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.260516+0000","last_change":"2024-03-06T07:34:08.260516+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.260516+0000","last_undegraded":"2024-03-06T07:34:08.260516+0000","last_fullsized":"2024-03-06T07:34:08.260516+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278072+0000","last_change":"2024-03-06T07:34:08.270938+0000","last_active":"2024-03-06T07:34:08.278072+0000","last_peered":"2024-03-06T07:34:08.278072+0000","last_clean":"2024-03-06T07:34:08.278072+0000","last_became_active":"2024-03-06T07:34:08.270180+0000","last_became_peered":"2024-03-06T07:34:08.270180+0000","last_unstale":"2024-03-06T07:34:08.278072+0000","last_undegraded":"2024-03-06T07:34:08.278072+0000","last_fullsized":"2024-03-06T07:34:08.278072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:35.842246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.265315+0000","last_change":"2024-03-06T07:34:08.265315+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.265315+0000","last_undegraded":"2024-03-06T07:34:08.265315+0000","last_fullsized":"2024-03-06T07:34:08.265315+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.274510+0000","last_change":"2024-03-06T07:34:08.272524+0000","last_active":"2024-03-06T07:34:08.274510+0000","last_peered":"2024-03-06T07:34:08.274510+0000","last_clean":"2024-03-06T07:34:08.274510+0000","last_became_active":"2024-03-06T07:34:08.272078+0000","last_became_peered":"2024-03-06T07:34:08.272078+0000","last_unstale":"2024-03-06T07:34:08.274510+0000","last_undegraded":"2024-03-06T07:34:08.274510+0000","last_fullsized":"2024-03-06T07:34:08.274510+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:35.842246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.249701+0000","last_change":"2024-03-06T07:34:08.249701+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.249701+0000","last_undegraded":"2024-03-06T07:34:08.249701+0000","last_fullsized":"2024-03-06T07:34:08.249701+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277157+0000","last_change":"2024-03-06T07:34:08.275513+0000","last_active":"2024-03-06T07:34:08.277157+0000","last_peered":"2024-03-06T07:34:08.277157+0000","last_clean":"2024-03-06T07:34:08.277157+0000","last_became_active":"2024-03-06T07:34:08.275177+0000","last_became_peered":"2024-03-06T07:34:08.275177+0000","last_unstale":"2024-03-06T07:34:08.277157+0000","last_undegraded":"2024-03-06T07:34:08.277157+0000","last_fullsized":"2024-03-06T07:34:08.277157+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:45.286640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259886+0000","last_change":"2024-03-06T07:34:08.259886+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259886+0000","last_undegraded":"2024-03-06T07:34:08.259886+0000","last_fullsized":"2024-03-06T07:34:08.259886+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277630+0000","last_change":"2024-03-06T07:34:08.272959+0000","last_active":"2024-03-06T07:34:08.277630+0000","last_peered":"2024-03-06T07:34:08.277630+0000","last_clean":"2024-03-06T07:34:08.277630+0000","last_became_active":"2024-03-06T07:34:08.271962+0000","last_became_peered":"2024-03-06T07:34:08.271962+0000","last_unstale":"2024-03-06T07:34:08.277630+0000","last_undegraded":"2024-03-06T07:34:08.277630+0000","last_fullsized":"2024-03-06T07:34:08.277630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.298132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.255829+0000","last_change":"2024-03-06T07:34:08.255829+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.255829+0000","last_undegraded":"2024-03-06T07:34:08.255829+0000","last_fullsized":"2024-03-06T07:34:08.255829+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.278927+0000","last_change":"2024-03-06T07:34:08.275800+0000","last_active":"2024-03-06T07:34:08.278927+0000","last_peered":"2024-03-06T07:34:08.278927+0000","last_clean":"2024-03-06T07:34:08.278927+0000","last_became_active":"2024-03-06T07:34:08.275476+0000","last_became_peered":"2024-03-06T07:34:08.275476+0000","last_unstale":"2024-03-06T07:34:08.278927+0000","last_undegraded":"2024-03-06T07:34:08.278927+0000","last_fullsized":"2024-03-06T07:34:08.278927+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:03.432186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259769+0000","last_change":"2024-03-06T07:34:08.259769+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259769+0000","last_undegraded":"2024-03-06T07:34:08.259769+0000","last_fullsized":"2024-03-06T07:34:08.259769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277934+0000","last_change":"2024-03-06T07:34:08.274027+0000","last_active":"2024-03-06T07:34:08.277934+0000","last_peered":"2024-03-06T07:34:08.277934+0000","last_clean":"2024-03-06T07:34:08.277934+0000","last_became_active":"2024-03-06T07:34:08.273695+0000","last_became_peered":"2024-03-06T07:34:08.273695+0000","last_unstale":"2024-03-06T07:34:08.277934+0000","last_undegraded":"2024-03-06T07:34:08.277934+0000","last_fullsized":"2024-03-06T07:34:08.277934+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:30.078168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259166+0000","last_change":"2024-03-06T07:34:08.259166+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259166+0000","last_undegraded":"2024-03-06T07:34:08.259166+0000","last_fullsized":"2024-03-06T07:34:08.259166+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.276327+0000","last_change":"2024-03-06T07:34:08.274029+0000","last_active":"2024-03-06T07:34:08.276327+0000","last_peered":"2024-03-06T07:34:08.276327+0000","last_clean":"2024-03-06T07:34:08.276327+0000","last_became_active":"2024-03-06T07:34:08.273759+0000","last_became_peered":"2024-03-06T07:34:08.273759+0000","last_unstale":"2024-03-06T07:34:08.276327+0000","last_undegraded":"2024-03-06T07:34:08.276327+0000","last_fullsized":"2024-03-06T07:34:08.276327+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:44.011669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.263721+0000","last_change":"2024-03-06T07:34:08.263721+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.263721+0000","last_undegraded":"2024-03-06T07:34:08.263721+0000","last_fullsized":"2024-03-06T07:34:08.263721+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.282257+0000","last_change":"2024-03-06T07:34:08.279992+0000","last_active":"2024-03-06T07:34:08.282257+0000","last_peered":"2024-03-06T07:34:08.282257+0000","last_clean":"2024-03-06T07:34:08.282257+0000","last_became_active":"2024-03-06T07:34:08.279058+0000","last_became_peered":"2024-03-06T07:34:08.279058+0000","last_unstale":"2024-03-06T07:34:08.282257+0000","last_undegraded":"2024-03-06T07:34:08.282257+0000","last_fullsized":"2024-03-06T07:34:08.282257+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:33.361496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.245275+0000","last_change":"2024-03-06T07:34:08.245275+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.245275+0000","last_undegraded":"2024-03-06T07:34:08.245275+0000","last_fullsized":"2024-03-06T07:34:08.245275+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277042+0000","last_change":"2024-03-06T07:34:08.271690+0000","last_active":"2024-03-06T07:34:08.277042+0000","last_peered":"2024-03-06T07:34:08.277042+0000","last_clean":"2024-03-06T07:34:08.277042+0000","last_became_active":"2024-03-06T07:34:08.271425+0000","last_became_peered":"2024-03-06T07:34:08.271425+0000","last_unstale":"2024-03-06T07:34:08.277042+0000","last_undegraded":"2024-03-06T07:34:08.277042+0000","last_fullsized":"2024-03-06T07:34:08.277042+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:41.468854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.245487+0000","last_change":"2024-03-06T07:34:08.245487+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.245487+0000","last_undegraded":"2024-03-06T07:34:08.245487+0000","last_fullsized":"2024-03-06T07:34:08.245487+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.280673+0000","last_change":"2024-03-06T07:34:08.276819+0000","last_active":"2024-03-06T07:34:08.280673+0000","last_peered":"2024-03-06T07:34:08.280673+0000","last_clean":"2024-03-06T07:34:08.280673+0000","last_became_active":"2024-03-06T07:34:08.276500+0000","last_became_peered":"2024-03-06T07:34:08.276500+0000","last_unstale":"2024-03-06T07:34:08.280673+0000","last_undegraded":"2024-03-06T07:34:08.280673+0000","last_fullsized":"2024-03-06T07:34:08.280673+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:22.315747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.257519+0000","last_change":"2024-03-06T07:34:08.257519+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.257519+0000","last_undegraded":"2024-03-06T07:34:08.257519+0000","last_fullsized":"2024-03-06T07:34:08.257519+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281676+0000","last_change":"2024-03-06T07:34:08.277478+0000","last_active":"2024-03-06T07:34:08.281676+0000","last_peered":"2024-03-06T07:34:08.281676+0000","last_clean":"2024-03-06T07:34:08.281676+0000","last_became_active":"2024-03-06T07:34:08.277186+0000","last_became_peered":"2024-03-06T07:34:08.277186+0000","last_unstale":"2024-03-06T07:34:08.281676+0000","last_undegraded":"2024-03-06T07:34:08.281676+0000","last_fullsized":"2024-03-06T07:34:08.281676+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:30.730009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.253985+0000","last_change":"2024-03-06T07:34:08.253985+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.253985+0000","last_undegraded":"2024-03-06T07:34:08.253985+0000","last_fullsized":"2024-03-06T07:34:08.253985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277608+0000","last_change":"2024-03-06T07:34:08.272451+0000","last_active":"2024-03-06T07:34:08.277608+0000","last_peered":"2024-03-06T07:34:08.277608+0000","last_clean":"2024-03-06T07:34:08.277608+0000","last_became_active":"2024-03-06T07:34:08.272175+0000","last_became_peered":"2024-03-06T07:34:08.272175+0000","last_unstale":"2024-03-06T07:34:08.277608+0000","last_undegraded":"2024-03-06T07:34:08.277608+0000","last_fullsized":"2024-03-06T07:34:08.277608+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:09.498839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.248253+0000","last_change":"2024-03-06T07:34:08.248253+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.248253+0000","last_undegraded":"2024-03-06T07:34:08.248253+0000","last_fullsized":"2024-03-06T07:34:08.248253+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.277590+0000","last_change":"2024-03-06T07:34:08.274395+0000","last_active":"2024-03-06T07:34:08.277590+0000","last_peered":"2024-03-06T07:34:08.277590+0000","last_clean":"2024-03-06T07:34:08.277590+0000","last_became_active":"2024-03-06T07:34:08.274068+0000","last_became_peered":"2024-03-06T07:34:08.274068+0000","last_unstale":"2024-03-06T07:34:08.277590+0000","last_undegraded":"2024-03-06T07:34:08.277590+0000","last_fullsized":"2024-03-06T07:34:08.277590+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:45.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.259318+0000","last_change":"2024-03-06T07:34:08.259318+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.259318+0000","last_undegraded":"2024-03-06T07:34:08.259318+0000","last_fullsized":"2024-03-06T07:34:08.259318+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.265318+0000","last_change":"2024-03-06T07:34:08.257721+0000","last_active":"2024-03-06T07:34:08.265318+0000","last_peered":"2024-03-06T07:34:08.265318+0000","last_clean":"2024-03-06T07:34:08.265318+0000","last_became_active":"2024-03-06T07:34:08.257322+0000","last_became_peered":"2024-03-06T07:34:08.257322+0000","last_unstale":"2024-03-06T07:34:08.265318+0000","last_undegraded":"2024-03-06T07:34:08.265318+0000","last_fullsized":"2024-03-06T07:34:08.265318+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.258236+0000","last_change":"2024-03-06T07:34:08.258236+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.258236+0000","last_undegraded":"2024-03-06T07:34:08.258236+0000","last_fullsized":"2024-03-06T07:34:08.258236+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.280901+0000","last_change":"2024-03-06T07:34:08.279635+0000","last_active":"2024-03-06T07:34:08.280901+0000","last_peered":"2024-03-06T07:34:08.280901+0000","last_clean":"2024-03-06T07:34:08.280901+0000","last_became_active":"2024-03-06T07:34:08.279250+0000","last_became_peered":"2024-03-06T07:34:08.279250+0000","last_unstale":"2024-03-06T07:34:08.280901+0000","last_undegraded":"2024-03-06T07:34:08.280901+0000","last_fullsized":"2024-03-06T07:34:08.280901+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:06.126369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-03-06T07:34:08.257781+0000","last_change":"2024-03-06T07:34:08.257781+0000","last_active":"2024-03-06T07:34:08.227127+0000","last_peered":"2024-03-06T07:34:08.227127+0000","last_clean":"2024-03-06T07:34:08.227127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:34:08.257781+0000","last_undegraded":"2024-03-06T07:34:08.257781+0000","last_fullsized":"2024-03-06T07:34:08.257781+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.281358+0000","last_change":"2024-03-06T07:34:08.276832+0000","last_active":"2024-03-06T07:34:08.281358+0000","last_peered":"2024-03-06T07:34:08.281358+0000","last_clean":"2024-03-06T07:34:08.281358+0000","last_became_active":"2024-03-06T07:34:08.276488+0000","last_became_peered":"2024-03-06T07:34:08.276488+0000","last_unstale":"2024-03-06T07:34:08.281358+0000","last_undegraded":"2024-03-06T07:34:08.281358+0000","last_fullsized":"2024-03-06T07:34:08.281358+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:22.315747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":306,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.255371+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:34:08.255371+0000","last_peered":"2024-03-06T07:34:08.255371+0000","last_clean":"2024-03-06T07:34:08.255371+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:34:08.255371+0000","last_undegraded":"2024-03-06T07:34:08.255371+0000","last_fullsized":"2024-03-06T07:34:08.255371+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1744896,"data_stored":1079494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335388,"statfs":{"total":96636764160,"available":96599437312,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335388,"statfs":{"total":96636764160,"available":96599437312,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335388,"statfs":{"total":96636764160,"available":96599437312,"internally_reserved":0,"allocated":1138688,"data_stored":482851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1732608,"data_stored":1073219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:14.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:14.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.532+0000 7f585f000640 1 Processor -- start 2024-03-06T07:34:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.532+0000 7f585f000640 1 -- start start 2024-03-06T07:34:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585f000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5858145300 0x7f58580b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585f000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5858070f10 0x7f58580cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585f000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58580714d0 con 0x7f58580cce90 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585f000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58580cd300 con 0x7f5858145300 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585f000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580717b0 con 0x7f5858070f10 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585dc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5858145300 0x7f58580b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585dc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5858145300 0x7f58580b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47092/0 (socket says 172.21.15.32:47092) 2024-03-06T07:34:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.534+0000 7f585dc00640 1 -- 172.21.15.32:0/3653436277 learned_addr learned my addr 172.21.15.32:0/3653436277 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.535+0000 7f585d200640 1 --2- 172.21.15.32:0/3653436277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5858070f10 0x7f58580cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.535+0000 7f585dc00640 1 -- 172.21.15.32:0/3653436277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5858070f10 msgr2=0x7f58580cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.535+0000 7f585dc00640 1 --2- 172.21.15.32:0/3653436277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5858070f10 0x7f58580cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.535+0000 7f585dc00640 1 -- 172.21.15.32:0/3653436277 >> v1:172.21.15.32:6789/0 conn(0x7f58580cce90 legacy=0x7f58580cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.535+0000 7f585dc00640 1 -- 172.21.15.32:0/3653436277 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858071a90 con 0x7f5858145300 2024-03-06T07:34:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.536+0000 7f585dc00640 1 --2- 172.21.15.32:0/3653436277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f58580b49a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f584c009870 tx=0x7f584c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad43f3f8db7fafe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.536+0000 7f5857e00640 1 -- 172.21.15.32:0/3653436277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c034070 con 0x7f5858145300 2024-03-06T07:34:14.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.537+0000 7f5857e00640 1 -- 172.21.15.32:0/3653436277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c030840 con 0x7f5858145300 2024-03-06T07:34:14.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.537+0000 7f5857e00640 1 -- 172.21.15.32:0/3653436277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c030c10 con 0x7f5858145300 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.537+0000 7f585f000640 1 -- 172.21.15.32:0/3653436277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 msgr2=0x7f58580b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.537+0000 7f585f000640 1 --2- 172.21.15.32:0/3653436277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f58580b49a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f584c009870 tx=0x7f584c02b980 comp rx=0 tx=0).stop 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.537+0000 7f585f000640 1 -- 172.21.15.32:0/3653436277 shutdown_connections 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.537+0000 7f585f000640 1 --2- 172.21.15.32:0/3653436277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5858070f10 0x7f58580cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.538+0000 7f585f000640 1 --2- 172.21.15.32:0/3653436277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f58580b49a0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.538+0000 7f585f000640 1 -- 172.21.15.32:0/3653436277 >> 172.21.15.32:0/3653436277 conn(0x7f585806b3d0 msgr2=0x7f585806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.538+0000 7f585f000640 1 -- 172.21.15.32:0/3653436277 shutdown_connections 2024-03-06T07:34:14.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.538+0000 7f585f000640 1 -- 172.21.15.32:0/3653436277 wait complete. 2024-03-06T07:34:14.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.540+0000 7f585f000640 1 Processor -- start 2024-03-06T07:34:14.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.540+0000 7f585f000640 1 -- start start 2024-03-06T07:34:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5858070f10 0x7f58581e7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585f000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58580cce90 0x7f58581e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f5858142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58581456e0 con 0x7f5858145300 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580b4ee0 con 0x7f5858070f10 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585f000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585815d580 con 0x7f58580cce90 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f5858142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f5858142740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47104/0 (socket says 172.21.15.32:47104) 2024-03-06T07:34:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.541+0000 7f585e600640 1 -- 172.21.15.32:0/298906312 learned_addr learned my addr 172.21.15.32:0/298906312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585d200640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58580cce90 0x7f58581e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585dc00640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5858070f10 0x7f58581e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585e600640 1 -- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58580cce90 msgr2=0x7f58581e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585e600640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58580cce90 0x7f58581e5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585e600640 1 -- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5858070f10 msgr2=0x7f58581e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585e600640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5858070f10 0x7f58581e7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585e600640 1 -- 172.21.15.32:0/298906312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58581e6190 con 0x7f5858145300 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585d200640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58580cce90 0x7f58581e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585e600640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f5858142740 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f585000c7a0 tx=0x7f585000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850003b70 con 0x7f5858145300 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5850003d30 con 0x7f5858145300 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585000aa30 con 0x7f5858145300 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.542+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58581e6470 con 0x7f5858145300 2024-03-06T07:34:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.543+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58581e6a30 con 0x7f5858145300 2024-03-06T07:34:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.544+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585804f330 con 0x7f5858145300 2024-03-06T07:34:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.545+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f585000abd0 con 0x7f5858145300 2024-03-06T07:34:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.545+0000 7f5855600640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f583806b4d0 0x7f583806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.546+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f585008a0e0 con 0x7f5858145300 2024-03-06T07:34:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.546+0000 7f585dc00640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f583806b4d0 0x7f583806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.546+0000 7f585dc00640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f583806b4d0 0x7f583806d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f584c036000 tx=0x7f584c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.550+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5850058f50 con 0x7f5858145300 2024-03-06T07:34:14.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.747+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58580ce3e0 con 0x7f583806b4d0 2024-03-06T07:34:14.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.757+0000 7f5855600640 1 -- 172.21.15.32:0/298906312 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252086 (secure 0 0 0) 0x7f58580ce3e0 con 0x7f583806b4d0 2024-03-06T07:34:14.761 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:14.764 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f583806b4d0 msgr2=0x7f583806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f583806b4d0 0x7f583806d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f584c036000 tx=0x7f584c03b040 comp rx=0 tx=0).stop 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 msgr2=0x7f5858142740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f5858142740 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f585000c7a0 tx=0x7f585000cc60 comp rx=0 tx=0).stop 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 shutdown_connections 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f583806b4d0 0x7f583806d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5858145300 0x7f5858142740 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58580cce90 0x7f58581e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 --2- 172.21.15.32:0/298906312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5858070f10 0x7f58581e7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.763+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 >> 172.21.15.32:0/298906312 conn(0x7f585806b3d0 msgr2=0x7f58580cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.764+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 shutdown_connections 2024-03-06T07:34:14.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:14.764+0000 7f585f000640 1 -- 172.21.15.32:0/298906312 wait complete. 2024-03-06T07:34:14.785 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-06T07:34:14.282158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276504,"kb_used_data":10408,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698216,"statfs":{"total":773094113280,"available":772810973184,"internally_reserved":0,"allocated":10657792,"data_stored":5203164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.042259"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003602+0000","last_change":"2024-03-06T07:34:09.286619+0000","last_active":"2024-03-06T07:34:11.003602+0000","last_peered":"2024-03-06T07:34:11.003602+0000","last_clean":"2024-03-06T07:34:11.003602+0000","last_became_active":"2024-03-06T07:34:09.286339+0000","last_became_peered":"2024-03-06T07:34:09.286339+0000","last_unstale":"2024-03-06T07:34:11.003602+0000","last_undegraded":"2024-03-06T07:34:11.003602+0000","last_fullsized":"2024-03-06T07:34:11.003602+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:54.474565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300528+0000","last_change":"2024-03-06T07:34:09.310443+0000","last_active":"2024-03-06T07:34:10.300528+0000","last_peered":"2024-03-06T07:34:10.300528+0000","last_clean":"2024-03-06T07:34:10.300528+0000","last_became_active":"2024-03-06T07:34:09.310141+0000","last_became_peered":"2024-03-06T07:34:09.310141+0000","last_unstale":"2024-03-06T07:34:10.300528+0000","last_undegraded":"2024-03-06T07:34:10.300528+0000","last_fullsized":"2024-03-06T07:34:10.300528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:30:11.163295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004046+0000","last_change":"2024-03-06T07:34:09.286683+0000","last_active":"2024-03-06T07:34:11.004046+0000","last_peered":"2024-03-06T07:34:11.004046+0000","last_clean":"2024-03-06T07:34:11.004046+0000","last_became_active":"2024-03-06T07:34:09.286264+0000","last_became_peered":"2024-03-06T07:34:09.286264+0000","last_unstale":"2024-03-06T07:34:11.004046+0000","last_undegraded":"2024-03-06T07:34:11.004046+0000","last_fullsized":"2024-03-06T07:34:11.004046+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:23.455135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290827+0000","last_change":"2024-03-06T07:34:09.291689+0000","last_active":"2024-03-06T07:34:10.290827+0000","last_peered":"2024-03-06T07:34:10.290827+0000","last_clean":"2024-03-06T07:34:10.290827+0000","last_became_active":"2024-03-06T07:34:09.290808+0000","last_became_peered":"2024-03-06T07:34:09.290808+0000","last_unstale":"2024-03-06T07:34:10.290827+0000","last_undegraded":"2024-03-06T07:34:10.290827+0000","last_fullsized":"2024-03-06T07:34:10.290827+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:40.530510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004935+0000","last_change":"2024-03-06T07:34:09.303097+0000","last_active":"2024-03-06T07:34:11.004935+0000","last_peered":"2024-03-06T07:34:11.004935+0000","last_clean":"2024-03-06T07:34:11.004935+0000","last_became_active":"2024-03-06T07:34:09.302816+0000","last_became_peered":"2024-03-06T07:34:09.302816+0000","last_unstale":"2024-03-06T07:34:11.004935+0000","last_undegraded":"2024-03-06T07:34:11.004935+0000","last_fullsized":"2024-03-06T07:34:11.004935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:40.475415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006200+0000","last_change":"2024-03-06T07:34:09.282622+0000","last_active":"2024-03-06T07:34:11.006200+0000","last_peered":"2024-03-06T07:34:11.006200+0000","last_clean":"2024-03-06T07:34:11.006200+0000","last_became_active":"2024-03-06T07:34:09.282257+0000","last_became_peered":"2024-03-06T07:34:09.282257+0000","last_unstale":"2024-03-06T07:34:11.006200+0000","last_undegraded":"2024-03-06T07:34:11.006200+0000","last_fullsized":"2024-03-06T07:34:11.006200+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:43.980518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004120+0000","last_change":"2024-03-06T07:34:09.283961+0000","last_active":"2024-03-06T07:34:11.004120+0000","last_peered":"2024-03-06T07:34:11.004120+0000","last_clean":"2024-03-06T07:34:11.004120+0000","last_became_active":"2024-03-06T07:34:09.283671+0000","last_became_peered":"2024-03-06T07:34:09.283671+0000","last_unstale":"2024-03-06T07:34:11.004120+0000","last_undegraded":"2024-03-06T07:34:11.004120+0000","last_fullsized":"2024-03-06T07:34:11.004120+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:10.689868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300846+0000","last_change":"2024-03-06T07:34:09.294383+0000","last_active":"2024-03-06T07:34:10.300846+0000","last_peered":"2024-03-06T07:34:10.300846+0000","last_clean":"2024-03-06T07:34:10.300846+0000","last_became_active":"2024-03-06T07:34:09.294078+0000","last_became_peered":"2024-03-06T07:34:09.294078+0000","last_unstale":"2024-03-06T07:34:10.300846+0000","last_undegraded":"2024-03-06T07:34:10.300846+0000","last_fullsized":"2024-03-06T07:34:10.300846+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:52.440212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004429+0000","last_change":"2024-03-06T07:34:09.304643+0000","last_active":"2024-03-06T07:34:11.004429+0000","last_peered":"2024-03-06T07:34:11.004429+0000","last_clean":"2024-03-06T07:34:11.004429+0000","last_became_active":"2024-03-06T07:34:09.304161+0000","last_became_peered":"2024-03-06T07:34:09.304161+0000","last_unstale":"2024-03-06T07:34:11.004429+0000","last_undegraded":"2024-03-06T07:34:11.004429+0000","last_fullsized":"2024-03-06T07:34:11.004429+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:40.475415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007072+0000","last_change":"2024-03-06T07:34:09.307259+0000","last_active":"2024-03-06T07:34:11.007072+0000","last_peered":"2024-03-06T07:34:11.007072+0000","last_clean":"2024-03-06T07:34:11.007072+0000","last_became_active":"2024-03-06T07:34:09.306951+0000","last_became_peered":"2024-03-06T07:34:09.306951+0000","last_unstale":"2024-03-06T07:34:11.007072+0000","last_undegraded":"2024-03-06T07:34:11.007072+0000","last_fullsized":"2024-03-06T07:34:11.007072+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:46.651651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006768+0000","last_change":"2024-03-06T07:34:09.300094+0000","last_active":"2024-03-06T07:34:11.006768+0000","last_peered":"2024-03-06T07:34:11.006768+0000","last_clean":"2024-03-06T07:34:11.006768+0000","last_became_active":"2024-03-06T07:34:09.296249+0000","last_became_peered":"2024-03-06T07:34:09.296249+0000","last_unstale":"2024-03-06T07:34:11.006768+0000","last_undegraded":"2024-03-06T07:34:11.006768+0000","last_fullsized":"2024-03-06T07:34:11.006768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:07.616565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003277+0000","last_change":"2024-03-06T07:34:09.298489+0000","last_active":"2024-03-06T07:34:11.003277+0000","last_peered":"2024-03-06T07:34:11.003277+0000","last_clean":"2024-03-06T07:34:11.003277+0000","last_became_active":"2024-03-06T07:34:09.298216+0000","last_became_peered":"2024-03-06T07:34:09.298216+0000","last_unstale":"2024-03-06T07:34:11.003277+0000","last_undegraded":"2024-03-06T07:34:11.003277+0000","last_fullsized":"2024-03-06T07:34:11.003277+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:50.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290411+0000","last_change":"2024-03-06T07:34:09.298570+0000","last_active":"2024-03-06T07:34:10.290411+0000","last_peered":"2024-03-06T07:34:10.290411+0000","last_clean":"2024-03-06T07:34:10.290411+0000","last_became_active":"2024-03-06T07:34:09.298245+0000","last_became_peered":"2024-03-06T07:34:09.298245+0000","last_unstale":"2024-03-06T07:34:10.290411+0000","last_undegraded":"2024-03-06T07:34:10.290411+0000","last_fullsized":"2024-03-06T07:34:10.290411+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:00.105663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305124+0000","last_change":"2024-03-06T07:34:09.307111+0000","last_active":"2024-03-06T07:34:10.305124+0000","last_peered":"2024-03-06T07:34:10.305124+0000","last_clean":"2024-03-06T07:34:10.305124+0000","last_became_active":"2024-03-06T07:34:09.306758+0000","last_became_peered":"2024-03-06T07:34:09.306758+0000","last_unstale":"2024-03-06T07:34:10.305124+0000","last_undegraded":"2024-03-06T07:34:10.305124+0000","last_fullsized":"2024-03-06T07:34:10.305124+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:47.141751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289787+0000","last_change":"2024-03-06T07:34:09.282819+0000","last_active":"2024-03-06T07:34:10.289787+0000","last_peered":"2024-03-06T07:34:10.289787+0000","last_clean":"2024-03-06T07:34:10.289787+0000","last_became_active":"2024-03-06T07:34:09.282473+0000","last_became_peered":"2024-03-06T07:34:09.282473+0000","last_unstale":"2024-03-06T07:34:10.289787+0000","last_undegraded":"2024-03-06T07:34:10.289787+0000","last_fullsized":"2024-03-06T07:34:10.289787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.761768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004119+0000","last_change":"2024-03-06T07:34:09.298366+0000","last_active":"2024-03-06T07:34:11.004119+0000","last_peered":"2024-03-06T07:34:11.004119+0000","last_clean":"2024-03-06T07:34:11.004119+0000","last_became_active":"2024-03-06T07:34:09.298038+0000","last_became_peered":"2024-03-06T07:34:09.298038+0000","last_unstale":"2024-03-06T07:34:11.004119+0000","last_undegraded":"2024-03-06T07:34:11.004119+0000","last_fullsized":"2024-03-06T07:34:11.004119+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:15.349718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301431+0000","last_change":"2024-03-06T07:34:09.312585+0000","last_active":"2024-03-06T07:34:10.301431+0000","last_peered":"2024-03-06T07:34:10.301431+0000","last_clean":"2024-03-06T07:34:10.301431+0000","last_became_active":"2024-03-06T07:34:09.311067+0000","last_became_peered":"2024-03-06T07:34:09.311067+0000","last_unstale":"2024-03-06T07:34:10.301431+0000","last_undegraded":"2024-03-06T07:34:10.301431+0000","last_fullsized":"2024-03-06T07:34:10.301431+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.682188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290846+0000","last_change":"2024-03-06T07:34:09.292319+0000","last_active":"2024-03-06T07:34:10.290846+0000","last_peered":"2024-03-06T07:34:10.290846+0000","last_clean":"2024-03-06T07:34:10.290846+0000","last_became_active":"2024-03-06T07:34:09.292019+0000","last_became_peered":"2024-03-06T07:34:09.292019+0000","last_unstale":"2024-03-06T07:34:10.290846+0000","last_undegraded":"2024-03-06T07:34:10.290846+0000","last_fullsized":"2024-03-06T07:34:10.290846+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:07.499061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305752+0000","last_change":"2024-03-06T07:34:09.297981+0000","last_active":"2024-03-06T07:34:10.305752+0000","last_peered":"2024-03-06T07:34:10.305752+0000","last_clean":"2024-03-06T07:34:10.305752+0000","last_became_active":"2024-03-06T07:34:09.296988+0000","last_became_peered":"2024-03-06T07:34:09.296988+0000","last_unstale":"2024-03-06T07:34:10.305752+0000","last_undegraded":"2024-03-06T07:34:10.305752+0000","last_fullsized":"2024-03-06T07:34:10.305752+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:10.000710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003440+0000","last_change":"2024-03-06T07:34:09.296148+0000","last_active":"2024-03-06T07:34:11.003440+0000","last_peered":"2024-03-06T07:34:11.003440+0000","last_clean":"2024-03-06T07:34:11.003440+0000","last_became_active":"2024-03-06T07:34:09.295849+0000","last_became_peered":"2024-03-06T07:34:09.295849+0000","last_unstale":"2024-03-06T07:34:11.003440+0000","last_undegraded":"2024-03-06T07:34:11.003440+0000","last_fullsized":"2024-03-06T07:34:11.003440+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:19.254659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305347+0000","last_change":"2024-03-06T07:34:09.296885+0000","last_active":"2024-03-06T07:34:10.305347+0000","last_peered":"2024-03-06T07:34:10.305347+0000","last_clean":"2024-03-06T07:34:10.305347+0000","last_became_active":"2024-03-06T07:34:09.293552+0000","last_became_peered":"2024-03-06T07:34:09.293552+0000","last_unstale":"2024-03-06T07:34:10.305347+0000","last_undegraded":"2024-03-06T07:34:10.305347+0000","last_fullsized":"2024-03-06T07:34:10.305347+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:04.837064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300391+0000","last_change":"2024-03-06T07:34:09.306061+0000","last_active":"2024-03-06T07:34:10.300391+0000","last_peered":"2024-03-06T07:34:10.300391+0000","last_clean":"2024-03-06T07:34:10.300391+0000","last_became_active":"2024-03-06T07:34:09.305791+0000","last_became_peered":"2024-03-06T07:34:09.305791+0000","last_unstale":"2024-03-06T07:34:10.300391+0000","last_undegraded":"2024-03-06T07:34:10.300391+0000","last_fullsized":"2024-03-06T07:34:10.300391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:34.363051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300454+0000","last_change":"2024-03-06T07:34:09.298023+0000","last_active":"2024-03-06T07:34:10.300454+0000","last_peered":"2024-03-06T07:34:10.300454+0000","last_clean":"2024-03-06T07:34:10.300454+0000","last_became_active":"2024-03-06T07:34:09.297768+0000","last_became_peered":"2024-03-06T07:34:09.297768+0000","last_unstale":"2024-03-06T07:34:10.300454+0000","last_undegraded":"2024-03-06T07:34:10.300454+0000","last_fullsized":"2024-03-06T07:34:10.300454+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:09.786475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004542+0000","last_change":"2024-03-06T07:34:09.278562+0000","last_active":"2024-03-06T07:34:11.004542+0000","last_peered":"2024-03-06T07:34:11.004542+0000","last_clean":"2024-03-06T07:34:11.004542+0000","last_became_active":"2024-03-06T07:34:09.277675+0000","last_became_peered":"2024-03-06T07:34:09.277675+0000","last_unstale":"2024-03-06T07:34:11.004542+0000","last_undegraded":"2024-03-06T07:34:11.004542+0000","last_fullsized":"2024-03-06T07:34:11.004542+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.761768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007435+0000","last_change":"2024-03-06T07:34:09.301732+0000","last_active":"2024-03-06T07:34:11.007435+0000","last_peered":"2024-03-06T07:34:11.007435+0000","last_clean":"2024-03-06T07:34:11.007435+0000","last_became_active":"2024-03-06T07:34:09.301136+0000","last_became_peered":"2024-03-06T07:34:09.301136+0000","last_unstale":"2024-03-06T07:34:11.007435+0000","last_undegraded":"2024-03-06T07:34:11.007435+0000","last_fullsized":"2024-03-06T07:34:11.007435+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:44.539838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301126+0000","last_change":"2024-03-06T07:34:09.303091+0000","last_active":"2024-03-06T07:34:10.301126+0000","last_peered":"2024-03-06T07:34:10.301126+0000","last_clean":"2024-03-06T07:34:10.301126+0000","last_became_active":"2024-03-06T07:34:09.302483+0000","last_became_peered":"2024-03-06T07:34:09.302483+0000","last_unstale":"2024-03-06T07:34:10.301126+0000","last_undegraded":"2024-03-06T07:34:10.301126+0000","last_fullsized":"2024-03-06T07:34:10.301126+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:54.692369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006900+0000","last_change":"2024-03-06T07:34:09.291954+0000","last_active":"2024-03-06T07:34:11.006900+0000","last_peered":"2024-03-06T07:34:11.006900+0000","last_clean":"2024-03-06T07:34:11.006900+0000","last_became_active":"2024-03-06T07:34:09.291609+0000","last_became_peered":"2024-03-06T07:34:09.291609+0000","last_unstale":"2024-03-06T07:34:11.006900+0000","last_undegraded":"2024-03-06T07:34:11.006900+0000","last_fullsized":"2024-03-06T07:34:11.006900+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:00.105663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005831+0000","last_change":"2024-03-06T07:34:09.304043+0000","last_active":"2024-03-06T07:34:11.005831+0000","last_peered":"2024-03-06T07:34:11.005831+0000","last_clean":"2024-03-06T07:34:11.005831+0000","last_became_active":"2024-03-06T07:34:09.303776+0000","last_became_peered":"2024-03-06T07:34:09.303776+0000","last_unstale":"2024-03-06T07:34:11.005831+0000","last_undegraded":"2024-03-06T07:34:11.005831+0000","last_fullsized":"2024-03-06T07:34:11.005831+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:00.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300530+0000","last_change":"2024-03-06T07:34:09.301020+0000","last_active":"2024-03-06T07:34:10.300530+0000","last_peered":"2024-03-06T07:34:10.300530+0000","last_clean":"2024-03-06T07:34:10.300530+0000","last_became_active":"2024-03-06T07:34:09.300700+0000","last_became_peered":"2024-03-06T07:34:09.300700+0000","last_unstale":"2024-03-06T07:34:10.300530+0000","last_undegraded":"2024-03-06T07:34:10.300530+0000","last_fullsized":"2024-03-06T07:34:10.300530+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:16.617196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305617+0000","last_change":"2024-03-06T07:34:09.295913+0000","last_active":"2024-03-06T07:34:10.305617+0000","last_peered":"2024-03-06T07:34:10.305617+0000","last_clean":"2024-03-06T07:34:10.305617+0000","last_became_active":"2024-03-06T07:34:09.295591+0000","last_became_peered":"2024-03-06T07:34:09.295591+0000","last_unstale":"2024-03-06T07:34:10.305617+0000","last_undegraded":"2024-03-06T07:34:10.305617+0000","last_fullsized":"2024-03-06T07:34:10.305617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:33.478183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290025+0000","last_change":"2024-03-06T07:34:09.302182+0000","last_active":"2024-03-06T07:34:10.290025+0000","last_peered":"2024-03-06T07:34:10.290025+0000","last_clean":"2024-03-06T07:34:10.290025+0000","last_became_active":"2024-03-06T07:34:09.300296+0000","last_became_peered":"2024-03-06T07:34:09.300296+0000","last_unstale":"2024-03-06T07:34:10.290025+0000","last_undegraded":"2024-03-06T07:34:10.290025+0000","last_fullsized":"2024-03-06T07:34:10.290025+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:07.616565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003074+0000","last_change":"2024-03-06T07:34:09.305078+0000","last_active":"2024-03-06T07:34:11.003074+0000","last_peered":"2024-03-06T07:34:11.003074+0000","last_clean":"2024-03-06T07:34:11.003074+0000","last_became_active":"2024-03-06T07:34:09.302647+0000","last_became_peered":"2024-03-06T07:34:09.302647+0000","last_unstale":"2024-03-06T07:34:11.003074+0000","last_undegraded":"2024-03-06T07:34:11.003074+0000","last_fullsized":"2024-03-06T07:34:11.003074+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:58.234520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305931+0000","last_change":"2024-03-06T07:34:09.304151+0000","last_active":"2024-03-06T07:34:10.305931+0000","last_peered":"2024-03-06T07:34:10.305931+0000","last_clean":"2024-03-06T07:34:10.305931+0000","last_became_active":"2024-03-06T07:34:09.303784+0000","last_became_peered":"2024-03-06T07:34:09.303784+0000","last_unstale":"2024-03-06T07:34:10.305931+0000","last_undegraded":"2024-03-06T07:34:10.305931+0000","last_fullsized":"2024-03-06T07:34:10.305931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:40.501563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301349+0000","last_change":"2024-03-06T07:34:08.276064+0000","last_active":"2024-03-06T07:34:10.301349+0000","last_peered":"2024-03-06T07:34:10.301349+0000","last_clean":"2024-03-06T07:34:10.301349+0000","last_became_active":"2024-03-06T07:34:08.275801+0000","last_became_peered":"2024-03-06T07:34:08.275801+0000","last_unstale":"2024-03-06T07:34:10.301349+0000","last_undegraded":"2024-03-06T07:34:10.301349+0000","last_fullsized":"2024-03-06T07:34:10.301349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:45.635970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.291312+0000","last_change":"2024-03-06T07:34:09.305128+0000","last_active":"2024-03-06T07:34:10.291312+0000","last_peered":"2024-03-06T07:34:10.291312+0000","last_clean":"2024-03-06T07:34:10.291312+0000","last_became_active":"2024-03-06T07:34:09.302858+0000","last_became_peered":"2024-03-06T07:34:09.302858+0000","last_unstale":"2024-03-06T07:34:10.291312+0000","last_undegraded":"2024-03-06T07:34:10.291312+0000","last_fullsized":"2024-03-06T07:34:10.291312+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:34.363051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006429+0000","last_change":"2024-03-06T07:34:08.277530+0000","last_active":"2024-03-06T07:34:11.006429+0000","last_peered":"2024-03-06T07:34:11.006429+0000","last_clean":"2024-03-06T07:34:11.006429+0000","last_became_active":"2024-03-06T07:34:08.277041+0000","last_became_peered":"2024-03-06T07:34:08.277041+0000","last_unstale":"2024-03-06T07:34:11.006429+0000","last_undegraded":"2024-03-06T07:34:11.006429+0000","last_fullsized":"2024-03-06T07:34:11.006429+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:49.452341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003330+0000","last_change":"2024-03-06T07:34:09.301943+0000","last_active":"2024-03-06T07:34:11.003330+0000","last_peered":"2024-03-06T07:34:11.003330+0000","last_clean":"2024-03-06T07:34:11.003330+0000","last_became_active":"2024-03-06T07:34:09.301130+0000","last_became_peered":"2024-03-06T07:34:09.301130+0000","last_unstale":"2024-03-06T07:34:11.003330+0000","last_undegraded":"2024-03-06T07:34:11.003330+0000","last_fullsized":"2024-03-06T07:34:11.003330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:49.376701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300827+0000","last_change":"2024-03-06T07:34:08.270479+0000","last_active":"2024-03-06T07:34:10.300827+0000","last_peered":"2024-03-06T07:34:10.300827+0000","last_clean":"2024-03-06T07:34:10.300827+0000","last_became_active":"2024-03-06T07:34:08.270199+0000","last_became_peered":"2024-03-06T07:34:08.270199+0000","last_unstale":"2024-03-06T07:34:10.300827+0000","last_undegraded":"2024-03-06T07:34:10.300827+0000","last_fullsized":"2024-03-06T07:34:10.300827+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:28.296128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003817+0000","last_change":"2024-03-06T07:34:09.307500+0000","last_active":"2024-03-06T07:34:11.003817+0000","last_peered":"2024-03-06T07:34:11.003817+0000","last_clean":"2024-03-06T07:34:11.003817+0000","last_became_active":"2024-03-06T07:34:09.307175+0000","last_became_peered":"2024-03-06T07:34:09.307175+0000","last_unstale":"2024-03-06T07:34:11.003817+0000","last_undegraded":"2024-03-06T07:34:11.003817+0000","last_fullsized":"2024-03-06T07:34:11.003817+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:45.020969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300720+0000","last_change":"2024-03-06T07:34:08.277107+0000","last_active":"2024-03-06T07:34:10.300720+0000","last_peered":"2024-03-06T07:34:10.300720+0000","last_clean":"2024-03-06T07:34:10.300720+0000","last_became_active":"2024-03-06T07:34:08.276380+0000","last_became_peered":"2024-03-06T07:34:08.276380+0000","last_unstale":"2024-03-06T07:34:10.300720+0000","last_undegraded":"2024-03-06T07:34:10.300720+0000","last_fullsized":"2024-03-06T07:34:10.300720+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.730852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005854+0000","last_change":"2024-03-06T07:34:09.298716+0000","last_active":"2024-03-06T07:34:11.005854+0000","last_peered":"2024-03-06T07:34:11.005854+0000","last_clean":"2024-03-06T07:34:11.005854+0000","last_became_active":"2024-03-06T07:34:09.297049+0000","last_became_peered":"2024-03-06T07:34:09.297049+0000","last_unstale":"2024-03-06T07:34:11.005854+0000","last_undegraded":"2024-03-06T07:34:11.005854+0000","last_fullsized":"2024-03-06T07:34:11.005854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:31.745690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003669+0000","last_change":"2024-03-06T07:34:08.271587+0000","last_active":"2024-03-06T07:34:11.003669+0000","last_peered":"2024-03-06T07:34:11.003669+0000","last_clean":"2024-03-06T07:34:11.003669+0000","last_became_active":"2024-03-06T07:34:08.271319+0000","last_became_peered":"2024-03-06T07:34:08.271319+0000","last_unstale":"2024-03-06T07:34:11.003669+0000","last_undegraded":"2024-03-06T07:34:11.003669+0000","last_fullsized":"2024-03-06T07:34:11.003669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:47.577770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006186+0000","last_change":"2024-03-06T07:34:09.296669+0000","last_active":"2024-03-06T07:34:11.006186+0000","last_peered":"2024-03-06T07:34:11.006186+0000","last_clean":"2024-03-06T07:34:11.006186+0000","last_became_active":"2024-03-06T07:34:09.296332+0000","last_became_peered":"2024-03-06T07:34:09.296332+0000","last_unstale":"2024-03-06T07:34:11.006186+0000","last_undegraded":"2024-03-06T07:34:11.006186+0000","last_fullsized":"2024-03-06T07:34:11.006186+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:10.392795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300231+0000","last_change":"2024-03-06T07:34:08.275076+0000","last_active":"2024-03-06T07:34:10.300231+0000","last_peered":"2024-03-06T07:34:10.300231+0000","last_clean":"2024-03-06T07:34:10.300231+0000","last_became_active":"2024-03-06T07:34:08.274808+0000","last_became_peered":"2024-03-06T07:34:08.274808+0000","last_unstale":"2024-03-06T07:34:10.300231+0000","last_undegraded":"2024-03-06T07:34:10.300231+0000","last_fullsized":"2024-03-06T07:34:10.300231+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289734+0000","last_change":"2024-03-06T07:34:09.283444+0000","last_active":"2024-03-06T07:34:10.289734+0000","last_peered":"2024-03-06T07:34:10.289734+0000","last_clean":"2024-03-06T07:34:10.289734+0000","last_became_active":"2024-03-06T07:34:09.281506+0000","last_became_peered":"2024-03-06T07:34:09.281506+0000","last_unstale":"2024-03-06T07:34:10.289734+0000","last_undegraded":"2024-03-06T07:34:10.289734+0000","last_fullsized":"2024-03-06T07:34:10.289734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:10.392795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004607+0000","last_change":"2024-03-06T07:34:08.276324+0000","last_active":"2024-03-06T07:34:11.004607+0000","last_peered":"2024-03-06T07:34:11.004607+0000","last_clean":"2024-03-06T07:34:11.004607+0000","last_became_active":"2024-03-06T07:34:08.275963+0000","last_became_peered":"2024-03-06T07:34:08.275963+0000","last_unstale":"2024-03-06T07:34:11.004607+0000","last_undegraded":"2024-03-06T07:34:11.004607+0000","last_fullsized":"2024-03-06T07:34:11.004607+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002938+0000","last_change":"2024-03-06T07:34:09.307806+0000","last_active":"2024-03-06T07:34:11.002938+0000","last_peered":"2024-03-06T07:34:11.002938+0000","last_clean":"2024-03-06T07:34:11.002938+0000","last_became_active":"2024-03-06T07:34:09.307027+0000","last_became_peered":"2024-03-06T07:34:09.307027+0000","last_unstale":"2024-03-06T07:34:11.002938+0000","last_undegraded":"2024-03-06T07:34:11.002938+0000","last_fullsized":"2024-03-06T07:34:11.002938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:25.216965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002961+0000","last_change":"2024-03-06T07:34:08.259184+0000","last_active":"2024-03-06T07:34:11.002961+0000","last_peered":"2024-03-06T07:34:11.002961+0000","last_clean":"2024-03-06T07:34:11.002961+0000","last_became_active":"2024-03-06T07:34:08.258879+0000","last_became_peered":"2024-03-06T07:34:08.258879+0000","last_unstale":"2024-03-06T07:34:11.002961+0000","last_undegraded":"2024-03-06T07:34:11.002961+0000","last_fullsized":"2024-03-06T07:34:11.002961+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:09.754281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301209+0000","last_change":"2024-03-06T07:34:09.302869+0000","last_active":"2024-03-06T07:34:10.301209+0000","last_peered":"2024-03-06T07:34:10.301209+0000","last_clean":"2024-03-06T07:34:10.301209+0000","last_became_active":"2024-03-06T07:34:09.301409+0000","last_became_peered":"2024-03-06T07:34:09.301409+0000","last_unstale":"2024-03-06T07:34:10.301209+0000","last_undegraded":"2024-03-06T07:34:10.301209+0000","last_fullsized":"2024-03-06T07:34:10.301209+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:23.807287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003932+0000","last_change":"2024-03-06T07:34:08.272502+0000","last_active":"2024-03-06T07:34:11.003932+0000","last_peered":"2024-03-06T07:34:11.003932+0000","last_clean":"2024-03-06T07:34:11.003932+0000","last_became_active":"2024-03-06T07:34:08.272249+0000","last_became_peered":"2024-03-06T07:34:08.272249+0000","last_unstale":"2024-03-06T07:34:11.003932+0000","last_undegraded":"2024-03-06T07:34:11.003932+0000","last_fullsized":"2024-03-06T07:34:11.003932+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:43.055234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004514+0000","last_change":"2024-03-06T07:34:09.307573+0000","last_active":"2024-03-06T07:34:11.004514+0000","last_peered":"2024-03-06T07:34:11.004514+0000","last_clean":"2024-03-06T07:34:11.004514+0000","last_became_active":"2024-03-06T07:34:09.307259+0000","last_became_peered":"2024-03-06T07:34:09.307259+0000","last_unstale":"2024-03-06T07:34:11.004514+0000","last_undegraded":"2024-03-06T07:34:11.004514+0000","last_fullsized":"2024-03-06T07:34:11.004514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:00.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290824+0000","last_change":"2024-03-06T07:34:08.277747+0000","last_active":"2024-03-06T07:34:10.290824+0000","last_peered":"2024-03-06T07:34:10.290824+0000","last_clean":"2024-03-06T07:34:10.290824+0000","last_became_active":"2024-03-06T07:34:08.274892+0000","last_became_peered":"2024-03-06T07:34:08.274892+0000","last_unstale":"2024-03-06T07:34:10.290824+0000","last_undegraded":"2024-03-06T07:34:10.290824+0000","last_fullsized":"2024-03-06T07:34:10.290824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.388532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006168+0000","last_change":"2024-03-06T07:34:09.303069+0000","last_active":"2024-03-06T07:34:11.006168+0000","last_peered":"2024-03-06T07:34:11.006168+0000","last_clean":"2024-03-06T07:34:11.006168+0000","last_became_active":"2024-03-06T07:34:09.302690+0000","last_became_peered":"2024-03-06T07:34:09.302690+0000","last_unstale":"2024-03-06T07:34:11.006168+0000","last_undegraded":"2024-03-06T07:34:11.006168+0000","last_fullsized":"2024-03-06T07:34:11.006168+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:29.311809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007057+0000","last_change":"2024-03-06T07:34:08.277986+0000","last_active":"2024-03-06T07:34:11.007057+0000","last_peered":"2024-03-06T07:34:11.007057+0000","last_clean":"2024-03-06T07:34:11.007057+0000","last_became_active":"2024-03-06T07:34:08.276369+0000","last_became_peered":"2024-03-06T07:34:08.276369+0000","last_unstale":"2024-03-06T07:34:11.007057+0000","last_undegraded":"2024-03-06T07:34:11.007057+0000","last_fullsized":"2024-03-06T07:34:11.007057+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006945+0000","last_change":"2024-03-06T07:34:09.303409+0000","last_active":"2024-03-06T07:34:11.006945+0000","last_peered":"2024-03-06T07:34:11.006945+0000","last_clean":"2024-03-06T07:34:11.006945+0000","last_became_active":"2024-03-06T07:34:09.302220+0000","last_became_peered":"2024-03-06T07:34:09.302220+0000","last_unstale":"2024-03-06T07:34:11.006945+0000","last_undegraded":"2024-03-06T07:34:11.006945+0000","last_fullsized":"2024-03-06T07:34:11.006945+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:07.499061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003401+0000","last_change":"2024-03-06T07:34:08.265538+0000","last_active":"2024-03-06T07:34:11.003401+0000","last_peered":"2024-03-06T07:34:11.003401+0000","last_clean":"2024-03-06T07:34:11.003401+0000","last_became_active":"2024-03-06T07:34:08.264783+0000","last_became_peered":"2024-03-06T07:34:08.264783+0000","last_unstale":"2024-03-06T07:34:11.003401+0000","last_undegraded":"2024-03-06T07:34:11.003401+0000","last_fullsized":"2024-03-06T07:34:11.003401+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:56.924294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289949+0000","last_change":"2024-03-06T07:34:09.279232+0000","last_active":"2024-03-06T07:34:10.289949+0000","last_peered":"2024-03-06T07:34:10.289949+0000","last_clean":"2024-03-06T07:34:10.289949+0000","last_became_active":"2024-03-06T07:34:09.278568+0000","last_became_peered":"2024-03-06T07:34:09.278568+0000","last_unstale":"2024-03-06T07:34:10.289949+0000","last_undegraded":"2024-03-06T07:34:10.289949+0000","last_fullsized":"2024-03-06T07:34:10.289949+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:43.980518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305700+0000","last_change":"2024-03-06T07:34:08.273221+0000","last_active":"2024-03-06T07:34:10.305700+0000","last_peered":"2024-03-06T07:34:10.305700+0000","last_clean":"2024-03-06T07:34:10.305700+0000","last_became_active":"2024-03-06T07:34:08.272960+0000","last_became_peered":"2024-03-06T07:34:08.272960+0000","last_unstale":"2024-03-06T07:34:10.305700+0000","last_undegraded":"2024-03-06T07:34:10.305700+0000","last_fullsized":"2024-03-06T07:34:10.305700+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.133468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003387+0000","last_change":"2024-03-06T07:34:09.291512+0000","last_active":"2024-03-06T07:34:11.003387+0000","last_peered":"2024-03-06T07:34:11.003387+0000","last_clean":"2024-03-06T07:34:11.003387+0000","last_became_active":"2024-03-06T07:34:09.290557+0000","last_became_peered":"2024-03-06T07:34:09.290557+0000","last_unstale":"2024-03-06T07:34:11.003387+0000","last_undegraded":"2024-03-06T07:34:11.003387+0000","last_fullsized":"2024-03-06T07:34:11.003387+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:52.517967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306490+0000","last_change":"2024-03-06T07:34:08.269595+0000","last_active":"2024-03-06T07:34:10.306490+0000","last_peered":"2024-03-06T07:34:10.306490+0000","last_clean":"2024-03-06T07:34:10.306490+0000","last_became_active":"2024-03-06T07:34:08.269260+0000","last_became_peered":"2024-03-06T07:34:08.269260+0000","last_unstale":"2024-03-06T07:34:10.306490+0000","last_undegraded":"2024-03-06T07:34:10.306490+0000","last_fullsized":"2024-03-06T07:34:10.306490+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:32.400545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002962+0000","last_change":"2024-03-06T07:34:09.307736+0000","last_active":"2024-03-06T07:34:11.002962+0000","last_peered":"2024-03-06T07:34:11.002962+0000","last_clean":"2024-03-06T07:34:11.002962+0000","last_became_active":"2024-03-06T07:34:09.307136+0000","last_became_peered":"2024-03-06T07:34:09.307136+0000","last_unstale":"2024-03-06T07:34:11.002962+0000","last_undegraded":"2024-03-06T07:34:11.002962+0000","last_fullsized":"2024-03-06T07:34:11.002962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:17.481076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002822+0000","last_change":"2024-03-06T07:34:08.271511+0000","last_active":"2024-03-06T07:34:11.002822+0000","last_peered":"2024-03-06T07:34:11.002822+0000","last_clean":"2024-03-06T07:34:11.002822+0000","last_became_active":"2024-03-06T07:34:08.271119+0000","last_became_peered":"2024-03-06T07:34:08.271119+0000","last_unstale":"2024-03-06T07:34:11.002822+0000","last_undegraded":"2024-03-06T07:34:11.002822+0000","last_fullsized":"2024-03-06T07:34:11.002822+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:30.730009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.304931+0000","last_change":"2024-03-06T07:34:09.304652+0000","last_active":"2024-03-06T07:34:10.304931+0000","last_peered":"2024-03-06T07:34:10.304931+0000","last_clean":"2024-03-06T07:34:10.304931+0000","last_became_active":"2024-03-06T07:34:09.303092+0000","last_became_peered":"2024-03-06T07:34:09.303092+0000","last_unstale":"2024-03-06T07:34:10.304931+0000","last_undegraded":"2024-03-06T07:34:10.304931+0000","last_fullsized":"2024-03-06T07:34:10.304931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:10.215635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003249+0000","last_change":"2024-03-06T07:34:08.269208+0000","last_active":"2024-03-06T07:34:11.003249+0000","last_peered":"2024-03-06T07:34:11.003249+0000","last_clean":"2024-03-06T07:34:11.003249+0000","last_became_active":"2024-03-06T07:34:08.268751+0000","last_became_peered":"2024-03-06T07:34:08.268751+0000","last_unstale":"2024-03-06T07:34:11.003249+0000","last_undegraded":"2024-03-06T07:34:11.003249+0000","last_fullsized":"2024-03-06T07:34:11.003249+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:09.199953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007597+0000","last_change":"2024-03-06T07:34:09.298981+0000","last_active":"2024-03-06T07:34:11.007597+0000","last_peered":"2024-03-06T07:34:11.007597+0000","last_clean":"2024-03-06T07:34:11.007597+0000","last_became_active":"2024-03-06T07:34:09.296016+0000","last_became_peered":"2024-03-06T07:34:09.296016+0000","last_unstale":"2024-03-06T07:34:11.007597+0000","last_undegraded":"2024-03-06T07:34:11.007597+0000","last_fullsized":"2024-03-06T07:34:11.007597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:27.293917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003760+0000","last_change":"2024-03-06T07:34:08.270938+0000","last_active":"2024-03-06T07:34:11.003760+0000","last_peered":"2024-03-06T07:34:11.003760+0000","last_clean":"2024-03-06T07:34:11.003760+0000","last_became_active":"2024-03-06T07:34:08.270180+0000","last_became_peered":"2024-03-06T07:34:08.270180+0000","last_unstale":"2024-03-06T07:34:11.003760+0000","last_undegraded":"2024-03-06T07:34:11.003760+0000","last_fullsized":"2024-03-06T07:34:11.003760+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005724+0000","last_change":"2024-03-06T07:34:09.300024+0000","last_active":"2024-03-06T07:34:11.005724+0000","last_peered":"2024-03-06T07:34:11.005724+0000","last_clean":"2024-03-06T07:34:11.005724+0000","last_became_active":"2024-03-06T07:34:09.299739+0000","last_became_peered":"2024-03-06T07:34:09.299739+0000","last_unstale":"2024-03-06T07:34:11.005724+0000","last_undegraded":"2024-03-06T07:34:11.005724+0000","last_fullsized":"2024-03-06T07:34:11.005724+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:57.939975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006456+0000","last_change":"2024-03-06T07:34:08.272524+0000","last_active":"2024-03-06T07:34:11.006456+0000","last_peered":"2024-03-06T07:34:11.006456+0000","last_clean":"2024-03-06T07:34:11.006456+0000","last_became_active":"2024-03-06T07:34:08.272078+0000","last_became_peered":"2024-03-06T07:34:08.272078+0000","last_unstale":"2024-03-06T07:34:11.006456+0000","last_undegraded":"2024-03-06T07:34:11.006456+0000","last_fullsized":"2024-03-06T07:34:11.006456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.298132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007077+0000","last_change":"2024-03-06T07:34:09.297770+0000","last_active":"2024-03-06T07:34:11.007077+0000","last_peered":"2024-03-06T07:34:11.007077+0000","last_clean":"2024-03-06T07:34:11.007077+0000","last_became_active":"2024-03-06T07:34:09.297457+0000","last_became_peered":"2024-03-06T07:34:09.297457+0000","last_unstale":"2024-03-06T07:34:11.007077+0000","last_undegraded":"2024-03-06T07:34:11.007077+0000","last_fullsized":"2024-03-06T07:34:11.007077+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:40.530510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300777+0000","last_change":"2024-03-06T07:34:08.275513+0000","last_active":"2024-03-06T07:34:10.300777+0000","last_peered":"2024-03-06T07:34:10.300777+0000","last_clean":"2024-03-06T07:34:10.300777+0000","last_became_active":"2024-03-06T07:34:08.275177+0000","last_became_peered":"2024-03-06T07:34:08.275177+0000","last_unstale":"2024-03-06T07:34:10.300777+0000","last_undegraded":"2024-03-06T07:34:10.300777+0000","last_fullsized":"2024-03-06T07:34:10.300777+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:49.452341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306140+0000","last_change":"2024-03-06T07:34:09.293082+0000","last_active":"2024-03-06T07:34:10.306140+0000","last_peered":"2024-03-06T07:34:10.306140+0000","last_clean":"2024-03-06T07:34:10.306140+0000","last_became_active":"2024-03-06T07:34:09.292719+0000","last_became_peered":"2024-03-06T07:34:09.292719+0000","last_unstale":"2024-03-06T07:34:10.306140+0000","last_undegraded":"2024-03-06T07:34:10.306140+0000","last_fullsized":"2024-03-06T07:34:10.306140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:49.376701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300683+0000","last_change":"2024-03-06T07:34:08.272959+0000","last_active":"2024-03-06T07:34:10.300683+0000","last_peered":"2024-03-06T07:34:10.300683+0000","last_clean":"2024-03-06T07:34:10.300683+0000","last_became_active":"2024-03-06T07:34:08.271962+0000","last_became_peered":"2024-03-06T07:34:08.271962+0000","last_unstale":"2024-03-06T07:34:10.300683+0000","last_undegraded":"2024-03-06T07:34:10.300683+0000","last_fullsized":"2024-03-06T07:34:10.300683+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:39.459734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305569+0000","last_change":"2024-03-06T07:34:09.305597+0000","last_active":"2024-03-06T07:34:10.305569+0000","last_peered":"2024-03-06T07:34:10.305569+0000","last_clean":"2024-03-06T07:34:10.305569+0000","last_became_active":"2024-03-06T07:34:09.304907+0000","last_became_peered":"2024-03-06T07:34:09.304907+0000","last_unstale":"2024-03-06T07:34:10.305569+0000","last_undegraded":"2024-03-06T07:34:10.305569+0000","last_fullsized":"2024-03-06T07:34:10.305569+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:57.642232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005917+0000","last_change":"2024-03-06T07:34:08.275800+0000","last_active":"2024-03-06T07:34:11.005917+0000","last_peered":"2024-03-06T07:34:11.005917+0000","last_clean":"2024-03-06T07:34:11.005917+0000","last_became_active":"2024-03-06T07:34:08.275476+0000","last_became_peered":"2024-03-06T07:34:08.275476+0000","last_unstale":"2024-03-06T07:34:11.005917+0000","last_undegraded":"2024-03-06T07:34:11.005917+0000","last_fullsized":"2024-03-06T07:34:11.005917+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:54.100822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305023+0000","last_change":"2024-03-06T07:34:09.292697+0000","last_active":"2024-03-06T07:34:10.305023+0000","last_peered":"2024-03-06T07:34:10.305023+0000","last_clean":"2024-03-06T07:34:10.305023+0000","last_became_active":"2024-03-06T07:34:09.292397+0000","last_became_peered":"2024-03-06T07:34:09.292397+0000","last_unstale":"2024-03-06T07:34:10.305023+0000","last_undegraded":"2024-03-06T07:34:10.305023+0000","last_fullsized":"2024-03-06T07:34:10.305023+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:40.677711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005508+0000","last_change":"2024-03-06T07:34:08.274027+0000","last_active":"2024-03-06T07:34:11.005508+0000","last_peered":"2024-03-06T07:34:11.005508+0000","last_clean":"2024-03-06T07:34:11.005508+0000","last_became_active":"2024-03-06T07:34:08.273695+0000","last_became_peered":"2024-03-06T07:34:08.273695+0000","last_unstale":"2024-03-06T07:34:11.005508+0000","last_undegraded":"2024-03-06T07:34:11.005508+0000","last_fullsized":"2024-03-06T07:34:11.005508+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:55.267247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004182+0000","last_change":"2024-03-06T07:34:09.298873+0000","last_active":"2024-03-06T07:34:11.004182+0000","last_peered":"2024-03-06T07:34:11.004182+0000","last_clean":"2024-03-06T07:34:11.004182+0000","last_became_active":"2024-03-06T07:34:09.298515+0000","last_became_peered":"2024-03-06T07:34:09.298515+0000","last_unstale":"2024-03-06T07:34:11.004182+0000","last_undegraded":"2024-03-06T07:34:11.004182+0000","last_fullsized":"2024-03-06T07:34:11.004182+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:19.254659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300438+0000","last_change":"2024-03-06T07:34:08.274029+0000","last_active":"2024-03-06T07:34:10.300438+0000","last_peered":"2024-03-06T07:34:10.300438+0000","last_clean":"2024-03-06T07:34:10.300438+0000","last_became_active":"2024-03-06T07:34:08.273759+0000","last_became_peered":"2024-03-06T07:34:08.273759+0000","last_unstale":"2024-03-06T07:34:10.300438+0000","last_undegraded":"2024-03-06T07:34:10.300438+0000","last_fullsized":"2024-03-06T07:34:10.300438+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:44.005288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301343+0000","last_change":"2024-03-06T07:34:09.298874+0000","last_active":"2024-03-06T07:34:10.301343+0000","last_peered":"2024-03-06T07:34:10.301343+0000","last_clean":"2024-03-06T07:34:10.301343+0000","last_became_active":"2024-03-06T07:34:09.298602+0000","last_became_peered":"2024-03-06T07:34:09.298602+0000","last_unstale":"2024-03-06T07:34:10.301343+0000","last_undegraded":"2024-03-06T07:34:10.301343+0000","last_fullsized":"2024-03-06T07:34:10.301343+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:07.544494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005772+0000","last_change":"2024-03-06T07:34:08.279992+0000","last_active":"2024-03-06T07:34:11.005772+0000","last_peered":"2024-03-06T07:34:11.005772+0000","last_clean":"2024-03-06T07:34:11.005772+0000","last_became_active":"2024-03-06T07:34:08.279058+0000","last_became_peered":"2024-03-06T07:34:08.279058+0000","last_unstale":"2024-03-06T07:34:11.005772+0000","last_undegraded":"2024-03-06T07:34:11.005772+0000","last_fullsized":"2024-03-06T07:34:11.005772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:06.914393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004106+0000","last_change":"2024-03-06T07:34:09.290526+0000","last_active":"2024-03-06T07:34:11.004106+0000","last_peered":"2024-03-06T07:34:11.004106+0000","last_clean":"2024-03-06T07:34:11.004106+0000","last_became_active":"2024-03-06T07:34:09.290063+0000","last_became_peered":"2024-03-06T07:34:09.290063+0000","last_unstale":"2024-03-06T07:34:11.004106+0000","last_undegraded":"2024-03-06T07:34:11.004106+0000","last_fullsized":"2024-03-06T07:34:11.004106+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:31.093849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301495+0000","last_change":"2024-03-06T07:34:08.271690+0000","last_active":"2024-03-06T07:34:10.301495+0000","last_peered":"2024-03-06T07:34:10.301495+0000","last_clean":"2024-03-06T07:34:10.301495+0000","last_became_active":"2024-03-06T07:34:08.271425+0000","last_became_peered":"2024-03-06T07:34:08.271425+0000","last_unstale":"2024-03-06T07:34:10.301495+0000","last_undegraded":"2024-03-06T07:34:10.301495+0000","last_fullsized":"2024-03-06T07:34:10.301495+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:43.524157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003563+0000","last_change":"2024-03-06T07:34:09.297391+0000","last_active":"2024-03-06T07:34:11.003563+0000","last_peered":"2024-03-06T07:34:11.003563+0000","last_clean":"2024-03-06T07:34:11.003563+0000","last_became_active":"2024-03-06T07:34:09.297107+0000","last_became_peered":"2024-03-06T07:34:09.297107+0000","last_unstale":"2024-03-06T07:34:11.003563+0000","last_undegraded":"2024-03-06T07:34:11.003563+0000","last_fullsized":"2024-03-06T07:34:11.003563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:32.178946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300833+0000","last_change":"2024-03-06T07:34:08.276819+0000","last_active":"2024-03-06T07:34:10.300833+0000","last_peered":"2024-03-06T07:34:10.300833+0000","last_clean":"2024-03-06T07:34:10.300833+0000","last_became_active":"2024-03-06T07:34:08.276500+0000","last_became_peered":"2024-03-06T07:34:08.276500+0000","last_unstale":"2024-03-06T07:34:10.300833+0000","last_undegraded":"2024-03-06T07:34:10.300833+0000","last_fullsized":"2024-03-06T07:34:10.300833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:03.411533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006246+0000","last_change":"2024-03-06T07:34:09.300002+0000","last_active":"2024-03-06T07:34:11.006246+0000","last_peered":"2024-03-06T07:34:11.006246+0000","last_clean":"2024-03-06T07:34:11.006246+0000","last_became_active":"2024-03-06T07:34:09.299553+0000","last_became_peered":"2024-03-06T07:34:09.299553+0000","last_unstale":"2024-03-06T07:34:11.006246+0000","last_undegraded":"2024-03-06T07:34:11.006246+0000","last_fullsized":"2024-03-06T07:34:11.006246+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:04.427214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006365+0000","last_change":"2024-03-06T07:34:08.277478+0000","last_active":"2024-03-06T07:34:11.006365+0000","last_peered":"2024-03-06T07:34:11.006365+0000","last_clean":"2024-03-06T07:34:11.006365+0000","last_became_active":"2024-03-06T07:34:08.277186+0000","last_became_peered":"2024-03-06T07:34:08.277186+0000","last_unstale":"2024-03-06T07:34:11.006365+0000","last_undegraded":"2024-03-06T07:34:11.006365+0000","last_fullsized":"2024-03-06T07:34:11.006365+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:09.498839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306597+0000","last_change":"2024-03-06T07:34:09.306423+0000","last_active":"2024-03-06T07:34:10.306597+0000","last_peered":"2024-03-06T07:34:10.306597+0000","last_clean":"2024-03-06T07:34:10.306597+0000","last_became_active":"2024-03-06T07:34:09.302507+0000","last_became_peered":"2024-03-06T07:34:09.302507+0000","last_unstale":"2024-03-06T07:34:10.306597+0000","last_undegraded":"2024-03-06T07:34:10.306597+0000","last_fullsized":"2024-03-06T07:34:10.306597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:52.517967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300901+0000","last_change":"2024-03-06T07:34:08.272451+0000","last_active":"2024-03-06T07:34:10.300901+0000","last_peered":"2024-03-06T07:34:10.300901+0000","last_clean":"2024-03-06T07:34:10.300901+0000","last_became_active":"2024-03-06T07:34:08.272175+0000","last_became_peered":"2024-03-06T07:34:08.272175+0000","last_unstale":"2024-03-06T07:34:10.300901+0000","last_undegraded":"2024-03-06T07:34:10.300901+0000","last_fullsized":"2024-03-06T07:34:10.300901+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.388532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003157+0000","last_change":"2024-03-06T07:34:09.301236+0000","last_active":"2024-03-06T07:34:11.003157+0000","last_peered":"2024-03-06T07:34:11.003157+0000","last_clean":"2024-03-06T07:34:11.003157+0000","last_became_active":"2024-03-06T07:34:09.300337+0000","last_became_peered":"2024-03-06T07:34:09.300337+0000","last_unstale":"2024-03-06T07:34:11.003157+0000","last_undegraded":"2024-03-06T07:34:11.003157+0000","last_fullsized":"2024-03-06T07:34:11.003157+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:12.746533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004465+0000","last_change":"2024-03-06T07:34:08.274395+0000","last_active":"2024-03-06T07:34:11.004465+0000","last_peered":"2024-03-06T07:34:11.004465+0000","last_clean":"2024-03-06T07:34:11.004465+0000","last_became_active":"2024-03-06T07:34:08.274068+0000","last_became_peered":"2024-03-06T07:34:08.274068+0000","last_unstale":"2024-03-06T07:34:11.004465+0000","last_undegraded":"2024-03-06T07:34:11.004465+0000","last_fullsized":"2024-03-06T07:34:11.004465+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:41.468854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305022+0000","last_change":"2024-03-06T07:34:09.303120+0000","last_active":"2024-03-06T07:34:10.305022+0000","last_peered":"2024-03-06T07:34:10.305022+0000","last_clean":"2024-03-06T07:34:10.305022+0000","last_became_active":"2024-03-06T07:34:09.299011+0000","last_became_peered":"2024-03-06T07:34:09.299011+0000","last_unstale":"2024-03-06T07:34:10.305022+0000","last_undegraded":"2024-03-06T07:34:10.305022+0000","last_fullsized":"2024-03-06T07:34:10.305022+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:44.070915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290053+0000","last_change":"2024-03-06T07:34:08.257721+0000","last_active":"2024-03-06T07:34:10.290053+0000","last_peered":"2024-03-06T07:34:10.290053+0000","last_clean":"2024-03-06T07:34:10.290053+0000","last_became_active":"2024-03-06T07:34:08.257322+0000","last_became_peered":"2024-03-06T07:34:08.257322+0000","last_unstale":"2024-03-06T07:34:10.290053+0000","last_undegraded":"2024-03-06T07:34:10.290053+0000","last_fullsized":"2024-03-06T07:34:10.290053+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:39.459734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004391+0000","last_change":"2024-03-06T07:34:09.305318+0000","last_active":"2024-03-06T07:34:11.004391+0000","last_peered":"2024-03-06T07:34:11.004391+0000","last_clean":"2024-03-06T07:34:11.004391+0000","last_became_active":"2024-03-06T07:34:09.304946+0000","last_became_peered":"2024-03-06T07:34:09.304946+0000","last_unstale":"2024-03-06T07:34:11.004391+0000","last_undegraded":"2024-03-06T07:34:11.004391+0000","last_fullsized":"2024-03-06T07:34:11.004391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:45.020969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305330+0000","last_change":"2024-03-06T07:34:08.279635+0000","last_active":"2024-03-06T07:34:10.305330+0000","last_peered":"2024-03-06T07:34:10.305330+0000","last_clean":"2024-03-06T07:34:10.305330+0000","last_became_active":"2024-03-06T07:34:08.279250+0000","last_became_peered":"2024-03-06T07:34:08.279250+0000","last_unstale":"2024-03-06T07:34:10.305330+0000","last_undegraded":"2024-03-06T07:34:10.305330+0000","last_fullsized":"2024-03-06T07:34:10.305330+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:17.184769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305935+0000","last_change":"2024-03-06T07:34:09.300151+0000","last_active":"2024-03-06T07:34:10.305935+0000","last_peered":"2024-03-06T07:34:10.305935+0000","last_clean":"2024-03-06T07:34:10.305935+0000","last_became_active":"2024-03-06T07:34:09.299873+0000","last_became_peered":"2024-03-06T07:34:09.299873+0000","last_unstale":"2024-03-06T07:34:10.305935+0000","last_undegraded":"2024-03-06T07:34:10.305935+0000","last_fullsized":"2024-03-06T07:34:10.305935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:58.234520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006188+0000","last_change":"2024-03-06T07:34:08.276832+0000","last_active":"2024-03-06T07:34:11.006188+0000","last_peered":"2024-03-06T07:34:11.006188+0000","last_clean":"2024-03-06T07:34:11.006188+0000","last_became_active":"2024-03-06T07:34:08.276488+0000","last_became_peered":"2024-03-06T07:34:08.276488+0000","last_unstale":"2024-03-06T07:34:11.006188+0000","last_undegraded":"2024-03-06T07:34:11.006188+0000","last_fullsized":"2024-03-06T07:34:11.006188+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.730852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":310,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004681+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:34:11.004681+0000","last_peered":"2024-03-06T07:34:11.004681+0000","last_clean":"2024-03-06T07:34:11.004681+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:34:11.004681+0000","last_undegraded":"2024-03-06T07:34:11.004681+0000","last_fullsized":"2024-03-06T07:34:11.004681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1777664,"data_stored":1092681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":502853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1179648,"data_stored":502271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36512,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335328,"statfs":{"total":96636764160,"available":96599375872,"internally_reserved":0,"allocated":1196032,"data_stored":503937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335336,"statfs":{"total":96636764160,"available":96599384064,"internally_reserved":0,"allocated":1191936,"data_stored":503835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":502235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1175552,"data_stored":502201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1773568,"data_stored":1093151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:14.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:34:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.046+0000 7fa7d1800640 1 Processor -- start 2024-03-06T07:34:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.046+0000 7fa7d1800640 1 -- start start 2024-03-06T07:34:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7d1800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc158af0 0x7fa7cc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7d1800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7cc071380 0x7fa7cc0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7d1800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0b3d30 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7d1800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b4660 con 0x7fa7cc071380 2024-03-06T07:34:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7d1800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b4010 con 0x7fa7cc158af0 2024-03-06T07:34:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7cb400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7cc071380 0x7fa7cc0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7cb400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7cc071380 0x7fa7cc0ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47106/0 (socket says 172.21.15.32:47106) 2024-03-06T07:34:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7cb400640 1 -- 172.21.15.32:0/403305254 learned_addr learned my addr 172.21.15.32:0/403305254 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.048+0000 7fa7caa00640 1 --2- 172.21.15.32:0/403305254 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc158af0 0x7fa7cc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.049+0000 7fa7cb400640 1 -- 172.21.15.32:0/403305254 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc158af0 msgr2=0x7fa7cc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.049+0000 7fa7cb400640 1 --2- 172.21.15.32:0/403305254 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc158af0 0x7fa7cc0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.049+0000 7fa7cb400640 1 -- 172.21.15.32:0/403305254 >> v1:172.21.15.39:6789/0 conn(0x7fa7cc0ccad0 legacy=0x7fa7cc070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.049+0000 7fa7cb400640 1 -- 172.21.15.32:0/403305254 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0b42f0 con 0x7fa7cc071380 2024-03-06T07:34:15.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.050+0000 7fa7cb400640 1 --2- 172.21.15.32:0/403305254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc071380 0x7fa7cc0ccf50 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009e50 tx=0x7fa7c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7d25e892964ee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.050+0000 7fa7ca000640 1 -- 172.21.15.32:0/403305254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c402ca70 con 0x7fa7cc071380 2024-03-06T07:34:15.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.050+0000 7fa7ca000640 1 -- 172.21.15.32:0/403305254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c402cc10 con 0x7fa7cc071380 2024-03-06T07:34:15.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.050+0000 7fa7ca000640 1 -- 172.21.15.32:0/403305254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40343f0 con 0x7fa7cc071380 2024-03-06T07:34:15.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 -- 172.21.15.32:0/403305254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc071380 msgr2=0x7fa7cc0ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 --2- 172.21.15.32:0/403305254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc071380 0x7fa7cc0ccf50 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009e50 tx=0x7fa7c402bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 -- 172.21.15.32:0/403305254 shutdown_connections 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 --2- 172.21.15.32:0/403305254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc071380 0x7fa7cc0ccf50 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 --2- 172.21.15.32:0/403305254 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc158af0 0x7fa7cc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 -- 172.21.15.32:0/403305254 >> 172.21.15.32:0/403305254 conn(0x7fa7cc0b1d80 msgr2=0x7fa7cc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 -- 172.21.15.32:0/403305254 shutdown_connections 2024-03-06T07:34:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.051+0000 7fa7d1800640 1 -- 172.21.15.32:0/403305254 wait complete. 2024-03-06T07:34:15.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.053+0000 7fa7d1800640 1 Processor -- start 2024-03-06T07:34:15.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.053+0000 7fa7d1800640 1 -- start start 2024-03-06T07:34:15.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7d1800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc071380 0x7fa7cc140400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7d1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 0x7fa7cc0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7d1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7cc158af0 0x7fa7cc15d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7d1800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc15b750 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7d1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b5030 con 0x7fa7cc158af0 2024-03-06T07:34:15.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7d1800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc15b8d0 con 0x7fa7cc071380 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.054+0000 7fa7c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 0x7fa7cc0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7caa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc071380 0x7fa7cc140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 0x7fa7cc0ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47112/0 (socket says 172.21.15.32:47112) 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 -- 172.21.15.32:0/3465164816 learned_addr learned my addr 172.21.15.32:0/3465164816 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7cb400640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7cc158af0 0x7fa7cc15d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 -- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc071380 msgr2=0x7fa7cc140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc071380 0x7fa7cc140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 -- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7cc158af0 msgr2=0x7fa7cc15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7cc158af0 0x7fa7cc15d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 -- 172.21.15.32:0/3465164816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0adfc0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.055+0000 7fa7c2000640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 0x7fa7cc0ad750 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400c430 tx=0x7fa7b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.056+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4011790 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.056+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b4017070 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.056+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4011bb0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.056+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7cc0ae2a0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.056+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7cc09acf0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.057+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7cc04f3a0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.058+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7b401c460 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.059+0000 7fa7c3400640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7a806b500 0x7fa7a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.059+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7b408a000 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.059+0000 7fa7caa00640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7a806b500 0x7fa7a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.060+0000 7fa7caa00640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7a806b500 0x7fa7a806d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0079a0 tx=0x7fa7bc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.063+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7b4058df0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T07:34:15.218+0000 7fd020000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.336+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa7cc0adc90 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.338+0000 7fa7c3400640 1 -- 172.21.15.32:0/3465164816 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7fa7b40586f0 con 0x7fa7cc0ccad0 2024-03-06T07:34:15.340 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:15.340 INFO:teuthology.orchestra.run.smithi032.stdout:{"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":334,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":95,"num_osds":8,"num_up_osds":8,"osd_up_since":1709710132,"num_in_osds":8,"osd_in_since":1709710124,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":283140096,"bytes_avail":772810973184,"bytes_total":773094113280,"write_bytes_sec":2372,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":61,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"b","status":"up:active","gid":7122}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-06T07:33:28.268921+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7a806b500 msgr2=0x7fa7a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7a806b500 0x7fa7a806d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0079a0 tx=0x7fa7bc008040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 msgr2=0x7fa7cc0ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 0x7fa7cc0ad750 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400c430 tx=0x7fa7b400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 shutdown_connections 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7a806b500 0x7fa7a806d9c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7cc158af0 0x7fa7cc15d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7cc0ccad0 0x7fa7cc0ad750 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.340+0000 7fa7d1800640 1 --2- 172.21.15.32:0/3465164816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7cc071380 0x7fa7cc140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.341+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 >> 172.21.15.32:0/3465164816 conn(0x7fa7cc0b1d80 msgr2=0x7fa7cc06df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.341+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 shutdown_connections 2024-03-06T07:34:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.341+0000 7fa7d1800640 1 -- 172.21.15.32:0/3465164816 wait complete. 2024-03-06T07:34:15.360 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:34:15.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:15.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.575+0000 7ff455a00640 1 Processor -- start 2024-03-06T07:34:15.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.575+0000 7ff455a00640 1 -- start start 2024-03-06T07:34:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff455a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450145300 0x7ff4500b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff455a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4500cd060 0x7ff45015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff455a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4500714d0 con 0x7ff450070f10 2024-03-06T07:34:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff455a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4500717b0 con 0x7ff4500cd060 2024-03-06T07:34:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff455a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff450071a90 con 0x7ff450145300 2024-03-06T07:34:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff455000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4500cd060 0x7ff45015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.577+0000 7ff44fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450145300 0x7ff4500b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.578+0000 7ff455000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4500cd060 0x7ff45015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:42664/0 (socket says 172.21.15.32:42664) 2024-03-06T07:34:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.578+0000 7ff455000640 1 -- 172.21.15.32:0/4056908073 learned_addr learned my addr 172.21.15.32:0/4056908073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.578+0000 7ff455000640 1 -- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450145300 msgr2=0x7ff4500b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.578+0000 7ff455000640 1 --2- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450145300 0x7ff4500b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.578+0000 7ff455000640 1 -- 172.21.15.32:0/4056908073 >> v1:172.21.15.39:6789/0 conn(0x7ff450070f10 legacy=0x7ff4500cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.578+0000 7ff455000640 1 -- 172.21.15.32:0/4056908073 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450071d70 con 0x7ff4500cd060 2024-03-06T07:34:15.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.579+0000 7ff455000640 1 --2- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff45015c570 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff448009e50 tx=0x7ff44802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eca4ea14cb7b002 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.580+0000 7ff44f400640 1 -- 172.21.15.32:0/4056908073 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44802cb00 con 0x7ff4500cd060 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.580+0000 7ff44f400640 1 -- 172.21.15.32:0/4056908073 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44802cca0 con 0x7ff4500cd060 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.580+0000 7ff44f400640 1 -- 172.21.15.32:0/4056908073 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448034610 con 0x7ff4500cd060 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.580+0000 7ff455a00640 1 -- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 msgr2=0x7ff45015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.580+0000 7ff455a00640 1 --2- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff45015c570 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff448009e50 tx=0x7ff44802bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.581+0000 7ff455a00640 1 -- 172.21.15.32:0/4056908073 shutdown_connections 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.581+0000 7ff455a00640 1 --2- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff45015c570 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.581+0000 7ff455a00640 1 --2- 172.21.15.32:0/4056908073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450145300 0x7ff4500b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.581+0000 7ff455a00640 1 -- 172.21.15.32:0/4056908073 >> 172.21.15.32:0/4056908073 conn(0x7ff45006b3d0 msgr2=0x7ff45006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.581+0000 7ff455a00640 1 -- 172.21.15.32:0/4056908073 shutdown_connections 2024-03-06T07:34:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.581+0000 7ff455a00640 1 -- 172.21.15.32:0/4056908073 wait complete. 2024-03-06T07:34:15.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.582+0000 7ff455a00640 1 Processor -- start 2024-03-06T07:34:15.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.582+0000 7ff455a00640 1 -- start start 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.583+0000 7ff455a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450070f10 0x7ff450143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.583+0000 7ff455a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff4500ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.583+0000 7ff455a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 0x7ff4501fb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.583+0000 7ff455a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45015cab0 con 0x7ff450145300 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.583+0000 7ff455a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4501456e0 con 0x7ff4500cd060 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.583+0000 7ff455a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45015cc30 con 0x7ff450070f10 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 0x7ff4501fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 0x7ff4501fb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47124/0 (socket says 172.21.15.32:47124) 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 -- 172.21.15.32:0/3323300778 learned_addr learned my addr 172.21.15.32:0/3323300778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff44fe00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450070f10 0x7ff450143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff447e00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff4500ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 -- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450070f10 msgr2=0x7ff450143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450070f10 0x7ff450143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 -- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 msgr2=0x7ff4500ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff4500ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.584+0000 7ff455000640 1 -- 172.21.15.32:0/3323300778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4500aa060 con 0x7ff450145300 2024-03-06T07:34:15.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.585+0000 7ff455000640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 0x7ff4501fb770 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff448009e50 tx=0x7ff448004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.585+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448043020 con 0x7ff450145300 2024-03-06T07:34:15.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.585+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44802ce50 con 0x7ff450145300 2024-03-06T07:34:15.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.585+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44803e670 con 0x7ff450145300 2024-03-06T07:34:15.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.585+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4500aa340 con 0x7ff450145300 2024-03-06T07:34:15.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.585+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4500aa900 con 0x7ff450145300 2024-03-06T07:34:15.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.586+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45004f330 con 0x7ff450145300 2024-03-06T07:34:15.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.587+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff448005d10 con 0x7ff450145300 2024-03-06T07:34:15.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.588+0000 7ff44cc00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff42c06b500 0x7ff42c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.588+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4480aca30 con 0x7ff450145300 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.589+0000 7ff44fe00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff42c06b500 0x7ff42c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.589+0000 7ff44fe00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff42c06b500 0x7ff42c06d9c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff440009a30 tx=0x7ff440008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.592+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff44807b8a0 con 0x7ff450145300 2024-03-06T07:34:15.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.788+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4500b2d80 con 0x7ff42c06b500 2024-03-06T07:34:15.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.806+0000 7ff44cc00640 1 -- 172.21.15.32:0/3323300778 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252086 (secure 0 0 0) 0x7ff4500b2d80 con 0x7ff42c06b500 2024-03-06T07:34:15.810 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:34:15.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff42c06b500 msgr2=0x7ff42c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff42c06b500 0x7ff42c06d9c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff440009a30 tx=0x7ff440008040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 msgr2=0x7ff4501fb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 0x7ff4501fb770 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff448009e50 tx=0x7ff448004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 shutdown_connections 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff42c06b500 0x7ff42c06d9c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff450145300 0x7ff4501fb770 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4500cd060 0x7ff4500ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 --2- 172.21.15.32:0/3323300778 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff450070f10 0x7ff450143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 >> 172.21.15.32:0/3323300778 conn(0x7ff45006b3d0 msgr2=0x7ff4500aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 shutdown_connections 2024-03-06T07:34:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:15.813+0000 7ff455a00640 1 -- 172.21.15.32:0/3323300778 wait complete. 2024-03-06T07:34:15.835 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-06T07:34:14.282158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276504,"kb_used_data":10408,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698216,"statfs":{"total":773094113280,"available":772810973184,"internally_reserved":0,"allocated":10657792,"data_stored":5203164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.042259"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003602+0000","last_change":"2024-03-06T07:34:09.286619+0000","last_active":"2024-03-06T07:34:11.003602+0000","last_peered":"2024-03-06T07:34:11.003602+0000","last_clean":"2024-03-06T07:34:11.003602+0000","last_became_active":"2024-03-06T07:34:09.286339+0000","last_became_peered":"2024-03-06T07:34:09.286339+0000","last_unstale":"2024-03-06T07:34:11.003602+0000","last_undegraded":"2024-03-06T07:34:11.003602+0000","last_fullsized":"2024-03-06T07:34:11.003602+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:54.474565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300528+0000","last_change":"2024-03-06T07:34:09.310443+0000","last_active":"2024-03-06T07:34:10.300528+0000","last_peered":"2024-03-06T07:34:10.300528+0000","last_clean":"2024-03-06T07:34:10.300528+0000","last_became_active":"2024-03-06T07:34:09.310141+0000","last_became_peered":"2024-03-06T07:34:09.310141+0000","last_unstale":"2024-03-06T07:34:10.300528+0000","last_undegraded":"2024-03-06T07:34:10.300528+0000","last_fullsized":"2024-03-06T07:34:10.300528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:30:11.163295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004046+0000","last_change":"2024-03-06T07:34:09.286683+0000","last_active":"2024-03-06T07:34:11.004046+0000","last_peered":"2024-03-06T07:34:11.004046+0000","last_clean":"2024-03-06T07:34:11.004046+0000","last_became_active":"2024-03-06T07:34:09.286264+0000","last_became_peered":"2024-03-06T07:34:09.286264+0000","last_unstale":"2024-03-06T07:34:11.004046+0000","last_undegraded":"2024-03-06T07:34:11.004046+0000","last_fullsized":"2024-03-06T07:34:11.004046+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:23.455135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290827+0000","last_change":"2024-03-06T07:34:09.291689+0000","last_active":"2024-03-06T07:34:10.290827+0000","last_peered":"2024-03-06T07:34:10.290827+0000","last_clean":"2024-03-06T07:34:10.290827+0000","last_became_active":"2024-03-06T07:34:09.290808+0000","last_became_peered":"2024-03-06T07:34:09.290808+0000","last_unstale":"2024-03-06T07:34:10.290827+0000","last_undegraded":"2024-03-06T07:34:10.290827+0000","last_fullsized":"2024-03-06T07:34:10.290827+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:40.530510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004935+0000","last_change":"2024-03-06T07:34:09.303097+0000","last_active":"2024-03-06T07:34:11.004935+0000","last_peered":"2024-03-06T07:34:11.004935+0000","last_clean":"2024-03-06T07:34:11.004935+0000","last_became_active":"2024-03-06T07:34:09.302816+0000","last_became_peered":"2024-03-06T07:34:09.302816+0000","last_unstale":"2024-03-06T07:34:11.004935+0000","last_undegraded":"2024-03-06T07:34:11.004935+0000","last_fullsized":"2024-03-06T07:34:11.004935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:40.475415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006200+0000","last_change":"2024-03-06T07:34:09.282622+0000","last_active":"2024-03-06T07:34:11.006200+0000","last_peered":"2024-03-06T07:34:11.006200+0000","last_clean":"2024-03-06T07:34:11.006200+0000","last_became_active":"2024-03-06T07:34:09.282257+0000","last_became_peered":"2024-03-06T07:34:09.282257+0000","last_unstale":"2024-03-06T07:34:11.006200+0000","last_undegraded":"2024-03-06T07:34:11.006200+0000","last_fullsized":"2024-03-06T07:34:11.006200+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:43.980518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004120+0000","last_change":"2024-03-06T07:34:09.283961+0000","last_active":"2024-03-06T07:34:11.004120+0000","last_peered":"2024-03-06T07:34:11.004120+0000","last_clean":"2024-03-06T07:34:11.004120+0000","last_became_active":"2024-03-06T07:34:09.283671+0000","last_became_peered":"2024-03-06T07:34:09.283671+0000","last_unstale":"2024-03-06T07:34:11.004120+0000","last_undegraded":"2024-03-06T07:34:11.004120+0000","last_fullsized":"2024-03-06T07:34:11.004120+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:10.689868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300846+0000","last_change":"2024-03-06T07:34:09.294383+0000","last_active":"2024-03-06T07:34:10.300846+0000","last_peered":"2024-03-06T07:34:10.300846+0000","last_clean":"2024-03-06T07:34:10.300846+0000","last_became_active":"2024-03-06T07:34:09.294078+0000","last_became_peered":"2024-03-06T07:34:09.294078+0000","last_unstale":"2024-03-06T07:34:10.300846+0000","last_undegraded":"2024-03-06T07:34:10.300846+0000","last_fullsized":"2024-03-06T07:34:10.300846+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:52.440212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004429+0000","last_change":"2024-03-06T07:34:09.304643+0000","last_active":"2024-03-06T07:34:11.004429+0000","last_peered":"2024-03-06T07:34:11.004429+0000","last_clean":"2024-03-06T07:34:11.004429+0000","last_became_active":"2024-03-06T07:34:09.304161+0000","last_became_peered":"2024-03-06T07:34:09.304161+0000","last_unstale":"2024-03-06T07:34:11.004429+0000","last_undegraded":"2024-03-06T07:34:11.004429+0000","last_fullsized":"2024-03-06T07:34:11.004429+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:40.475415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007072+0000","last_change":"2024-03-06T07:34:09.307259+0000","last_active":"2024-03-06T07:34:11.007072+0000","last_peered":"2024-03-06T07:34:11.007072+0000","last_clean":"2024-03-06T07:34:11.007072+0000","last_became_active":"2024-03-06T07:34:09.306951+0000","last_became_peered":"2024-03-06T07:34:09.306951+0000","last_unstale":"2024-03-06T07:34:11.007072+0000","last_undegraded":"2024-03-06T07:34:11.007072+0000","last_fullsized":"2024-03-06T07:34:11.007072+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:46.651651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006768+0000","last_change":"2024-03-06T07:34:09.300094+0000","last_active":"2024-03-06T07:34:11.006768+0000","last_peered":"2024-03-06T07:34:11.006768+0000","last_clean":"2024-03-06T07:34:11.006768+0000","last_became_active":"2024-03-06T07:34:09.296249+0000","last_became_peered":"2024-03-06T07:34:09.296249+0000","last_unstale":"2024-03-06T07:34:11.006768+0000","last_undegraded":"2024-03-06T07:34:11.006768+0000","last_fullsized":"2024-03-06T07:34:11.006768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:07.616565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003277+0000","last_change":"2024-03-06T07:34:09.298489+0000","last_active":"2024-03-06T07:34:11.003277+0000","last_peered":"2024-03-06T07:34:11.003277+0000","last_clean":"2024-03-06T07:34:11.003277+0000","last_became_active":"2024-03-06T07:34:09.298216+0000","last_became_peered":"2024-03-06T07:34:09.298216+0000","last_unstale":"2024-03-06T07:34:11.003277+0000","last_undegraded":"2024-03-06T07:34:11.003277+0000","last_fullsized":"2024-03-06T07:34:11.003277+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:50.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290411+0000","last_change":"2024-03-06T07:34:09.298570+0000","last_active":"2024-03-06T07:34:10.290411+0000","last_peered":"2024-03-06T07:34:10.290411+0000","last_clean":"2024-03-06T07:34:10.290411+0000","last_became_active":"2024-03-06T07:34:09.298245+0000","last_became_peered":"2024-03-06T07:34:09.298245+0000","last_unstale":"2024-03-06T07:34:10.290411+0000","last_undegraded":"2024-03-06T07:34:10.290411+0000","last_fullsized":"2024-03-06T07:34:10.290411+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:00.105663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305124+0000","last_change":"2024-03-06T07:34:09.307111+0000","last_active":"2024-03-06T07:34:10.305124+0000","last_peered":"2024-03-06T07:34:10.305124+0000","last_clean":"2024-03-06T07:34:10.305124+0000","last_became_active":"2024-03-06T07:34:09.306758+0000","last_became_peered":"2024-03-06T07:34:09.306758+0000","last_unstale":"2024-03-06T07:34:10.305124+0000","last_undegraded":"2024-03-06T07:34:10.305124+0000","last_fullsized":"2024-03-06T07:34:10.305124+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:47.141751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289787+0000","last_change":"2024-03-06T07:34:09.282819+0000","last_active":"2024-03-06T07:34:10.289787+0000","last_peered":"2024-03-06T07:34:10.289787+0000","last_clean":"2024-03-06T07:34:10.289787+0000","last_became_active":"2024-03-06T07:34:09.282473+0000","last_became_peered":"2024-03-06T07:34:09.282473+0000","last_unstale":"2024-03-06T07:34:10.289787+0000","last_undegraded":"2024-03-06T07:34:10.289787+0000","last_fullsized":"2024-03-06T07:34:10.289787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.761768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004119+0000","last_change":"2024-03-06T07:34:09.298366+0000","last_active":"2024-03-06T07:34:11.004119+0000","last_peered":"2024-03-06T07:34:11.004119+0000","last_clean":"2024-03-06T07:34:11.004119+0000","last_became_active":"2024-03-06T07:34:09.298038+0000","last_became_peered":"2024-03-06T07:34:09.298038+0000","last_unstale":"2024-03-06T07:34:11.004119+0000","last_undegraded":"2024-03-06T07:34:11.004119+0000","last_fullsized":"2024-03-06T07:34:11.004119+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:15.349718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301431+0000","last_change":"2024-03-06T07:34:09.312585+0000","last_active":"2024-03-06T07:34:10.301431+0000","last_peered":"2024-03-06T07:34:10.301431+0000","last_clean":"2024-03-06T07:34:10.301431+0000","last_became_active":"2024-03-06T07:34:09.311067+0000","last_became_peered":"2024-03-06T07:34:09.311067+0000","last_unstale":"2024-03-06T07:34:10.301431+0000","last_undegraded":"2024-03-06T07:34:10.301431+0000","last_fullsized":"2024-03-06T07:34:10.301431+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.682188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290846+0000","last_change":"2024-03-06T07:34:09.292319+0000","last_active":"2024-03-06T07:34:10.290846+0000","last_peered":"2024-03-06T07:34:10.290846+0000","last_clean":"2024-03-06T07:34:10.290846+0000","last_became_active":"2024-03-06T07:34:09.292019+0000","last_became_peered":"2024-03-06T07:34:09.292019+0000","last_unstale":"2024-03-06T07:34:10.290846+0000","last_undegraded":"2024-03-06T07:34:10.290846+0000","last_fullsized":"2024-03-06T07:34:10.290846+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:07.499061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305752+0000","last_change":"2024-03-06T07:34:09.297981+0000","last_active":"2024-03-06T07:34:10.305752+0000","last_peered":"2024-03-06T07:34:10.305752+0000","last_clean":"2024-03-06T07:34:10.305752+0000","last_became_active":"2024-03-06T07:34:09.296988+0000","last_became_peered":"2024-03-06T07:34:09.296988+0000","last_unstale":"2024-03-06T07:34:10.305752+0000","last_undegraded":"2024-03-06T07:34:10.305752+0000","last_fullsized":"2024-03-06T07:34:10.305752+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:10.000710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003440+0000","last_change":"2024-03-06T07:34:09.296148+0000","last_active":"2024-03-06T07:34:11.003440+0000","last_peered":"2024-03-06T07:34:11.003440+0000","last_clean":"2024-03-06T07:34:11.003440+0000","last_became_active":"2024-03-06T07:34:09.295849+0000","last_became_peered":"2024-03-06T07:34:09.295849+0000","last_unstale":"2024-03-06T07:34:11.003440+0000","last_undegraded":"2024-03-06T07:34:11.003440+0000","last_fullsized":"2024-03-06T07:34:11.003440+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:19.254659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305347+0000","last_change":"2024-03-06T07:34:09.296885+0000","last_active":"2024-03-06T07:34:10.305347+0000","last_peered":"2024-03-06T07:34:10.305347+0000","last_clean":"2024-03-06T07:34:10.305347+0000","last_became_active":"2024-03-06T07:34:09.293552+0000","last_became_peered":"2024-03-06T07:34:09.293552+0000","last_unstale":"2024-03-06T07:34:10.305347+0000","last_undegraded":"2024-03-06T07:34:10.305347+0000","last_fullsized":"2024-03-06T07:34:10.305347+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:04.837064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300391+0000","last_change":"2024-03-06T07:34:09.306061+0000","last_active":"2024-03-06T07:34:10.300391+0000","last_peered":"2024-03-06T07:34:10.300391+0000","last_clean":"2024-03-06T07:34:10.300391+0000","last_became_active":"2024-03-06T07:34:09.305791+0000","last_became_peered":"2024-03-06T07:34:09.305791+0000","last_unstale":"2024-03-06T07:34:10.300391+0000","last_undegraded":"2024-03-06T07:34:10.300391+0000","last_fullsized":"2024-03-06T07:34:10.300391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:34.363051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300454+0000","last_change":"2024-03-06T07:34:09.298023+0000","last_active":"2024-03-06T07:34:10.300454+0000","last_peered":"2024-03-06T07:34:10.300454+0000","last_clean":"2024-03-06T07:34:10.300454+0000","last_became_active":"2024-03-06T07:34:09.297768+0000","last_became_peered":"2024-03-06T07:34:09.297768+0000","last_unstale":"2024-03-06T07:34:10.300454+0000","last_undegraded":"2024-03-06T07:34:10.300454+0000","last_fullsized":"2024-03-06T07:34:10.300454+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:09.786475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004542+0000","last_change":"2024-03-06T07:34:09.278562+0000","last_active":"2024-03-06T07:34:11.004542+0000","last_peered":"2024-03-06T07:34:11.004542+0000","last_clean":"2024-03-06T07:34:11.004542+0000","last_became_active":"2024-03-06T07:34:09.277675+0000","last_became_peered":"2024-03-06T07:34:09.277675+0000","last_unstale":"2024-03-06T07:34:11.004542+0000","last_undegraded":"2024-03-06T07:34:11.004542+0000","last_fullsized":"2024-03-06T07:34:11.004542+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.761768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007435+0000","last_change":"2024-03-06T07:34:09.301732+0000","last_active":"2024-03-06T07:34:11.007435+0000","last_peered":"2024-03-06T07:34:11.007435+0000","last_clean":"2024-03-06T07:34:11.007435+0000","last_became_active":"2024-03-06T07:34:09.301136+0000","last_became_peered":"2024-03-06T07:34:09.301136+0000","last_unstale":"2024-03-06T07:34:11.007435+0000","last_undegraded":"2024-03-06T07:34:11.007435+0000","last_fullsized":"2024-03-06T07:34:11.007435+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:44.539838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301126+0000","last_change":"2024-03-06T07:34:09.303091+0000","last_active":"2024-03-06T07:34:10.301126+0000","last_peered":"2024-03-06T07:34:10.301126+0000","last_clean":"2024-03-06T07:34:10.301126+0000","last_became_active":"2024-03-06T07:34:09.302483+0000","last_became_peered":"2024-03-06T07:34:09.302483+0000","last_unstale":"2024-03-06T07:34:10.301126+0000","last_undegraded":"2024-03-06T07:34:10.301126+0000","last_fullsized":"2024-03-06T07:34:10.301126+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:54.692369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006900+0000","last_change":"2024-03-06T07:34:09.291954+0000","last_active":"2024-03-06T07:34:11.006900+0000","last_peered":"2024-03-06T07:34:11.006900+0000","last_clean":"2024-03-06T07:34:11.006900+0000","last_became_active":"2024-03-06T07:34:09.291609+0000","last_became_peered":"2024-03-06T07:34:09.291609+0000","last_unstale":"2024-03-06T07:34:11.006900+0000","last_undegraded":"2024-03-06T07:34:11.006900+0000","last_fullsized":"2024-03-06T07:34:11.006900+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:00.105663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005831+0000","last_change":"2024-03-06T07:34:09.304043+0000","last_active":"2024-03-06T07:34:11.005831+0000","last_peered":"2024-03-06T07:34:11.005831+0000","last_clean":"2024-03-06T07:34:11.005831+0000","last_became_active":"2024-03-06T07:34:09.303776+0000","last_became_peered":"2024-03-06T07:34:09.303776+0000","last_unstale":"2024-03-06T07:34:11.005831+0000","last_undegraded":"2024-03-06T07:34:11.005831+0000","last_fullsized":"2024-03-06T07:34:11.005831+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:00.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300530+0000","last_change":"2024-03-06T07:34:09.301020+0000","last_active":"2024-03-06T07:34:10.300530+0000","last_peered":"2024-03-06T07:34:10.300530+0000","last_clean":"2024-03-06T07:34:10.300530+0000","last_became_active":"2024-03-06T07:34:09.300700+0000","last_became_peered":"2024-03-06T07:34:09.300700+0000","last_unstale":"2024-03-06T07:34:10.300530+0000","last_undegraded":"2024-03-06T07:34:10.300530+0000","last_fullsized":"2024-03-06T07:34:10.300530+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:16.617196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305617+0000","last_change":"2024-03-06T07:34:09.295913+0000","last_active":"2024-03-06T07:34:10.305617+0000","last_peered":"2024-03-06T07:34:10.305617+0000","last_clean":"2024-03-06T07:34:10.305617+0000","last_became_active":"2024-03-06T07:34:09.295591+0000","last_became_peered":"2024-03-06T07:34:09.295591+0000","last_unstale":"2024-03-06T07:34:10.305617+0000","last_undegraded":"2024-03-06T07:34:10.305617+0000","last_fullsized":"2024-03-06T07:34:10.305617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:33.478183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290025+0000","last_change":"2024-03-06T07:34:09.302182+0000","last_active":"2024-03-06T07:34:10.290025+0000","last_peered":"2024-03-06T07:34:10.290025+0000","last_clean":"2024-03-06T07:34:10.290025+0000","last_became_active":"2024-03-06T07:34:09.300296+0000","last_became_peered":"2024-03-06T07:34:09.300296+0000","last_unstale":"2024-03-06T07:34:10.290025+0000","last_undegraded":"2024-03-06T07:34:10.290025+0000","last_fullsized":"2024-03-06T07:34:10.290025+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:07.616565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003074+0000","last_change":"2024-03-06T07:34:09.305078+0000","last_active":"2024-03-06T07:34:11.003074+0000","last_peered":"2024-03-06T07:34:11.003074+0000","last_clean":"2024-03-06T07:34:11.003074+0000","last_became_active":"2024-03-06T07:34:09.302647+0000","last_became_peered":"2024-03-06T07:34:09.302647+0000","last_unstale":"2024-03-06T07:34:11.003074+0000","last_undegraded":"2024-03-06T07:34:11.003074+0000","last_fullsized":"2024-03-06T07:34:11.003074+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:58.234520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305931+0000","last_change":"2024-03-06T07:34:09.304151+0000","last_active":"2024-03-06T07:34:10.305931+0000","last_peered":"2024-03-06T07:34:10.305931+0000","last_clean":"2024-03-06T07:34:10.305931+0000","last_became_active":"2024-03-06T07:34:09.303784+0000","last_became_peered":"2024-03-06T07:34:09.303784+0000","last_unstale":"2024-03-06T07:34:10.305931+0000","last_undegraded":"2024-03-06T07:34:10.305931+0000","last_fullsized":"2024-03-06T07:34:10.305931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:40.501563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301349+0000","last_change":"2024-03-06T07:34:08.276064+0000","last_active":"2024-03-06T07:34:10.301349+0000","last_peered":"2024-03-06T07:34:10.301349+0000","last_clean":"2024-03-06T07:34:10.301349+0000","last_became_active":"2024-03-06T07:34:08.275801+0000","last_became_peered":"2024-03-06T07:34:08.275801+0000","last_unstale":"2024-03-06T07:34:10.301349+0000","last_undegraded":"2024-03-06T07:34:10.301349+0000","last_fullsized":"2024-03-06T07:34:10.301349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:45.635970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.291312+0000","last_change":"2024-03-06T07:34:09.305128+0000","last_active":"2024-03-06T07:34:10.291312+0000","last_peered":"2024-03-06T07:34:10.291312+0000","last_clean":"2024-03-06T07:34:10.291312+0000","last_became_active":"2024-03-06T07:34:09.302858+0000","last_became_peered":"2024-03-06T07:34:09.302858+0000","last_unstale":"2024-03-06T07:34:10.291312+0000","last_undegraded":"2024-03-06T07:34:10.291312+0000","last_fullsized":"2024-03-06T07:34:10.291312+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:34.363051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006429+0000","last_change":"2024-03-06T07:34:08.277530+0000","last_active":"2024-03-06T07:34:11.006429+0000","last_peered":"2024-03-06T07:34:11.006429+0000","last_clean":"2024-03-06T07:34:11.006429+0000","last_became_active":"2024-03-06T07:34:08.277041+0000","last_became_peered":"2024-03-06T07:34:08.277041+0000","last_unstale":"2024-03-06T07:34:11.006429+0000","last_undegraded":"2024-03-06T07:34:11.006429+0000","last_fullsized":"2024-03-06T07:34:11.006429+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:49.452341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003330+0000","last_change":"2024-03-06T07:34:09.301943+0000","last_active":"2024-03-06T07:34:11.003330+0000","last_peered":"2024-03-06T07:34:11.003330+0000","last_clean":"2024-03-06T07:34:11.003330+0000","last_became_active":"2024-03-06T07:34:09.301130+0000","last_became_peered":"2024-03-06T07:34:09.301130+0000","last_unstale":"2024-03-06T07:34:11.003330+0000","last_undegraded":"2024-03-06T07:34:11.003330+0000","last_fullsized":"2024-03-06T07:34:11.003330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:49.376701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300827+0000","last_change":"2024-03-06T07:34:08.270479+0000","last_active":"2024-03-06T07:34:10.300827+0000","last_peered":"2024-03-06T07:34:10.300827+0000","last_clean":"2024-03-06T07:34:10.300827+0000","last_became_active":"2024-03-06T07:34:08.270199+0000","last_became_peered":"2024-03-06T07:34:08.270199+0000","last_unstale":"2024-03-06T07:34:10.300827+0000","last_undegraded":"2024-03-06T07:34:10.300827+0000","last_fullsized":"2024-03-06T07:34:10.300827+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:28.296128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003817+0000","last_change":"2024-03-06T07:34:09.307500+0000","last_active":"2024-03-06T07:34:11.003817+0000","last_peered":"2024-03-06T07:34:11.003817+0000","last_clean":"2024-03-06T07:34:11.003817+0000","last_became_active":"2024-03-06T07:34:09.307175+0000","last_became_peered":"2024-03-06T07:34:09.307175+0000","last_unstale":"2024-03-06T07:34:11.003817+0000","last_undegraded":"2024-03-06T07:34:11.003817+0000","last_fullsized":"2024-03-06T07:34:11.003817+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:45.020969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300720+0000","last_change":"2024-03-06T07:34:08.277107+0000","last_active":"2024-03-06T07:34:10.300720+0000","last_peered":"2024-03-06T07:34:10.300720+0000","last_clean":"2024-03-06T07:34:10.300720+0000","last_became_active":"2024-03-06T07:34:08.276380+0000","last_became_peered":"2024-03-06T07:34:08.276380+0000","last_unstale":"2024-03-06T07:34:10.300720+0000","last_undegraded":"2024-03-06T07:34:10.300720+0000","last_fullsized":"2024-03-06T07:34:10.300720+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.730852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005854+0000","last_change":"2024-03-06T07:34:09.298716+0000","last_active":"2024-03-06T07:34:11.005854+0000","last_peered":"2024-03-06T07:34:11.005854+0000","last_clean":"2024-03-06T07:34:11.005854+0000","last_became_active":"2024-03-06T07:34:09.297049+0000","last_became_peered":"2024-03-06T07:34:09.297049+0000","last_unstale":"2024-03-06T07:34:11.005854+0000","last_undegraded":"2024-03-06T07:34:11.005854+0000","last_fullsized":"2024-03-06T07:34:11.005854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:31.745690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003669+0000","last_change":"2024-03-06T07:34:08.271587+0000","last_active":"2024-03-06T07:34:11.003669+0000","last_peered":"2024-03-06T07:34:11.003669+0000","last_clean":"2024-03-06T07:34:11.003669+0000","last_became_active":"2024-03-06T07:34:08.271319+0000","last_became_peered":"2024-03-06T07:34:08.271319+0000","last_unstale":"2024-03-06T07:34:11.003669+0000","last_undegraded":"2024-03-06T07:34:11.003669+0000","last_fullsized":"2024-03-06T07:34:11.003669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:47.577770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006186+0000","last_change":"2024-03-06T07:34:09.296669+0000","last_active":"2024-03-06T07:34:11.006186+0000","last_peered":"2024-03-06T07:34:11.006186+0000","last_clean":"2024-03-06T07:34:11.006186+0000","last_became_active":"2024-03-06T07:34:09.296332+0000","last_became_peered":"2024-03-06T07:34:09.296332+0000","last_unstale":"2024-03-06T07:34:11.006186+0000","last_undegraded":"2024-03-06T07:34:11.006186+0000","last_fullsized":"2024-03-06T07:34:11.006186+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:10.392795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300231+0000","last_change":"2024-03-06T07:34:08.275076+0000","last_active":"2024-03-06T07:34:10.300231+0000","last_peered":"2024-03-06T07:34:10.300231+0000","last_clean":"2024-03-06T07:34:10.300231+0000","last_became_active":"2024-03-06T07:34:08.274808+0000","last_became_peered":"2024-03-06T07:34:08.274808+0000","last_unstale":"2024-03-06T07:34:10.300231+0000","last_undegraded":"2024-03-06T07:34:10.300231+0000","last_fullsized":"2024-03-06T07:34:10.300231+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289734+0000","last_change":"2024-03-06T07:34:09.283444+0000","last_active":"2024-03-06T07:34:10.289734+0000","last_peered":"2024-03-06T07:34:10.289734+0000","last_clean":"2024-03-06T07:34:10.289734+0000","last_became_active":"2024-03-06T07:34:09.281506+0000","last_became_peered":"2024-03-06T07:34:09.281506+0000","last_unstale":"2024-03-06T07:34:10.289734+0000","last_undegraded":"2024-03-06T07:34:10.289734+0000","last_fullsized":"2024-03-06T07:34:10.289734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:10.392795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004607+0000","last_change":"2024-03-06T07:34:08.276324+0000","last_active":"2024-03-06T07:34:11.004607+0000","last_peered":"2024-03-06T07:34:11.004607+0000","last_clean":"2024-03-06T07:34:11.004607+0000","last_became_active":"2024-03-06T07:34:08.275963+0000","last_became_peered":"2024-03-06T07:34:08.275963+0000","last_unstale":"2024-03-06T07:34:11.004607+0000","last_undegraded":"2024-03-06T07:34:11.004607+0000","last_fullsized":"2024-03-06T07:34:11.004607+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002938+0000","last_change":"2024-03-06T07:34:09.307806+0000","last_active":"2024-03-06T07:34:11.002938+0000","last_peered":"2024-03-06T07:34:11.002938+0000","last_clean":"2024-03-06T07:34:11.002938+0000","last_became_active":"2024-03-06T07:34:09.307027+0000","last_became_peered":"2024-03-06T07:34:09.307027+0000","last_unstale":"2024-03-06T07:34:11.002938+0000","last_undegraded":"2024-03-06T07:34:11.002938+0000","last_fullsized":"2024-03-06T07:34:11.002938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:25.216965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002961+0000","last_change":"2024-03-06T07:34:08.259184+0000","last_active":"2024-03-06T07:34:11.002961+0000","last_peered":"2024-03-06T07:34:11.002961+0000","last_clean":"2024-03-06T07:34:11.002961+0000","last_became_active":"2024-03-06T07:34:08.258879+0000","last_became_peered":"2024-03-06T07:34:08.258879+0000","last_unstale":"2024-03-06T07:34:11.002961+0000","last_undegraded":"2024-03-06T07:34:11.002961+0000","last_fullsized":"2024-03-06T07:34:11.002961+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:09.754281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301209+0000","last_change":"2024-03-06T07:34:09.302869+0000","last_active":"2024-03-06T07:34:10.301209+0000","last_peered":"2024-03-06T07:34:10.301209+0000","last_clean":"2024-03-06T07:34:10.301209+0000","last_became_active":"2024-03-06T07:34:09.301409+0000","last_became_peered":"2024-03-06T07:34:09.301409+0000","last_unstale":"2024-03-06T07:34:10.301209+0000","last_undegraded":"2024-03-06T07:34:10.301209+0000","last_fullsized":"2024-03-06T07:34:10.301209+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:23.807287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003932+0000","last_change":"2024-03-06T07:34:08.272502+0000","last_active":"2024-03-06T07:34:11.003932+0000","last_peered":"2024-03-06T07:34:11.003932+0000","last_clean":"2024-03-06T07:34:11.003932+0000","last_became_active":"2024-03-06T07:34:08.272249+0000","last_became_peered":"2024-03-06T07:34:08.272249+0000","last_unstale":"2024-03-06T07:34:11.003932+0000","last_undegraded":"2024-03-06T07:34:11.003932+0000","last_fullsized":"2024-03-06T07:34:11.003932+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:43.055234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004514+0000","last_change":"2024-03-06T07:34:09.307573+0000","last_active":"2024-03-06T07:34:11.004514+0000","last_peered":"2024-03-06T07:34:11.004514+0000","last_clean":"2024-03-06T07:34:11.004514+0000","last_became_active":"2024-03-06T07:34:09.307259+0000","last_became_peered":"2024-03-06T07:34:09.307259+0000","last_unstale":"2024-03-06T07:34:11.004514+0000","last_undegraded":"2024-03-06T07:34:11.004514+0000","last_fullsized":"2024-03-06T07:34:11.004514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:00.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290824+0000","last_change":"2024-03-06T07:34:08.277747+0000","last_active":"2024-03-06T07:34:10.290824+0000","last_peered":"2024-03-06T07:34:10.290824+0000","last_clean":"2024-03-06T07:34:10.290824+0000","last_became_active":"2024-03-06T07:34:08.274892+0000","last_became_peered":"2024-03-06T07:34:08.274892+0000","last_unstale":"2024-03-06T07:34:10.290824+0000","last_undegraded":"2024-03-06T07:34:10.290824+0000","last_fullsized":"2024-03-06T07:34:10.290824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.388532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006168+0000","last_change":"2024-03-06T07:34:09.303069+0000","last_active":"2024-03-06T07:34:11.006168+0000","last_peered":"2024-03-06T07:34:11.006168+0000","last_clean":"2024-03-06T07:34:11.006168+0000","last_became_active":"2024-03-06T07:34:09.302690+0000","last_became_peered":"2024-03-06T07:34:09.302690+0000","last_unstale":"2024-03-06T07:34:11.006168+0000","last_undegraded":"2024-03-06T07:34:11.006168+0000","last_fullsized":"2024-03-06T07:34:11.006168+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:29.311809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007057+0000","last_change":"2024-03-06T07:34:08.277986+0000","last_active":"2024-03-06T07:34:11.007057+0000","last_peered":"2024-03-06T07:34:11.007057+0000","last_clean":"2024-03-06T07:34:11.007057+0000","last_became_active":"2024-03-06T07:34:08.276369+0000","last_became_peered":"2024-03-06T07:34:08.276369+0000","last_unstale":"2024-03-06T07:34:11.007057+0000","last_undegraded":"2024-03-06T07:34:11.007057+0000","last_fullsized":"2024-03-06T07:34:11.007057+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006945+0000","last_change":"2024-03-06T07:34:09.303409+0000","last_active":"2024-03-06T07:34:11.006945+0000","last_peered":"2024-03-06T07:34:11.006945+0000","last_clean":"2024-03-06T07:34:11.006945+0000","last_became_active":"2024-03-06T07:34:09.302220+0000","last_became_peered":"2024-03-06T07:34:09.302220+0000","last_unstale":"2024-03-06T07:34:11.006945+0000","last_undegraded":"2024-03-06T07:34:11.006945+0000","last_fullsized":"2024-03-06T07:34:11.006945+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:07.499061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003401+0000","last_change":"2024-03-06T07:34:08.265538+0000","last_active":"2024-03-06T07:34:11.003401+0000","last_peered":"2024-03-06T07:34:11.003401+0000","last_clean":"2024-03-06T07:34:11.003401+0000","last_became_active":"2024-03-06T07:34:08.264783+0000","last_became_peered":"2024-03-06T07:34:08.264783+0000","last_unstale":"2024-03-06T07:34:11.003401+0000","last_undegraded":"2024-03-06T07:34:11.003401+0000","last_fullsized":"2024-03-06T07:34:11.003401+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:56.924294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289949+0000","last_change":"2024-03-06T07:34:09.279232+0000","last_active":"2024-03-06T07:34:10.289949+0000","last_peered":"2024-03-06T07:34:10.289949+0000","last_clean":"2024-03-06T07:34:10.289949+0000","last_became_active":"2024-03-06T07:34:09.278568+0000","last_became_peered":"2024-03-06T07:34:09.278568+0000","last_unstale":"2024-03-06T07:34:10.289949+0000","last_undegraded":"2024-03-06T07:34:10.289949+0000","last_fullsized":"2024-03-06T07:34:10.289949+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:43.980518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305700+0000","last_change":"2024-03-06T07:34:08.273221+0000","last_active":"2024-03-06T07:34:10.305700+0000","last_peered":"2024-03-06T07:34:10.305700+0000","last_clean":"2024-03-06T07:34:10.305700+0000","last_became_active":"2024-03-06T07:34:08.272960+0000","last_became_peered":"2024-03-06T07:34:08.272960+0000","last_unstale":"2024-03-06T07:34:10.305700+0000","last_undegraded":"2024-03-06T07:34:10.305700+0000","last_fullsized":"2024-03-06T07:34:10.305700+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.133468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003387+0000","last_change":"2024-03-06T07:34:09.291512+0000","last_active":"2024-03-06T07:34:11.003387+0000","last_peered":"2024-03-06T07:34:11.003387+0000","last_clean":"2024-03-06T07:34:11.003387+0000","last_became_active":"2024-03-06T07:34:09.290557+0000","last_became_peered":"2024-03-06T07:34:09.290557+0000","last_unstale":"2024-03-06T07:34:11.003387+0000","last_undegraded":"2024-03-06T07:34:11.003387+0000","last_fullsized":"2024-03-06T07:34:11.003387+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:52.517967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306490+0000","last_change":"2024-03-06T07:34:08.269595+0000","last_active":"2024-03-06T07:34:10.306490+0000","last_peered":"2024-03-06T07:34:10.306490+0000","last_clean":"2024-03-06T07:34:10.306490+0000","last_became_active":"2024-03-06T07:34:08.269260+0000","last_became_peered":"2024-03-06T07:34:08.269260+0000","last_unstale":"2024-03-06T07:34:10.306490+0000","last_undegraded":"2024-03-06T07:34:10.306490+0000","last_fullsized":"2024-03-06T07:34:10.306490+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:32.400545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002962+0000","last_change":"2024-03-06T07:34:09.307736+0000","last_active":"2024-03-06T07:34:11.002962+0000","last_peered":"2024-03-06T07:34:11.002962+0000","last_clean":"2024-03-06T07:34:11.002962+0000","last_became_active":"2024-03-06T07:34:09.307136+0000","last_became_peered":"2024-03-06T07:34:09.307136+0000","last_unstale":"2024-03-06T07:34:11.002962+0000","last_undegraded":"2024-03-06T07:34:11.002962+0000","last_fullsized":"2024-03-06T07:34:11.002962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:17.481076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002822+0000","last_change":"2024-03-06T07:34:08.271511+0000","last_active":"2024-03-06T07:34:11.002822+0000","last_peered":"2024-03-06T07:34:11.002822+0000","last_clean":"2024-03-06T07:34:11.002822+0000","last_became_active":"2024-03-06T07:34:08.271119+0000","last_became_peered":"2024-03-06T07:34:08.271119+0000","last_unstale":"2024-03-06T07:34:11.002822+0000","last_undegraded":"2024-03-06T07:34:11.002822+0000","last_fullsized":"2024-03-06T07:34:11.002822+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:30.730009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.304931+0000","last_change":"2024-03-06T07:34:09.304652+0000","last_active":"2024-03-06T07:34:10.304931+0000","last_peered":"2024-03-06T07:34:10.304931+0000","last_clean":"2024-03-06T07:34:10.304931+0000","last_became_active":"2024-03-06T07:34:09.303092+0000","last_became_peered":"2024-03-06T07:34:09.303092+0000","last_unstale":"2024-03-06T07:34:10.304931+0000","last_undegraded":"2024-03-06T07:34:10.304931+0000","last_fullsized":"2024-03-06T07:34:10.304931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:10.215635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003249+0000","last_change":"2024-03-06T07:34:08.269208+0000","last_active":"2024-03-06T07:34:11.003249+0000","last_peered":"2024-03-06T07:34:11.003249+0000","last_clean":"2024-03-06T07:34:11.003249+0000","last_became_active":"2024-03-06T07:34:08.268751+0000","last_became_peered":"2024-03-06T07:34:08.268751+0000","last_unstale":"2024-03-06T07:34:11.003249+0000","last_undegraded":"2024-03-06T07:34:11.003249+0000","last_fullsized":"2024-03-06T07:34:11.003249+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:09.199953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007597+0000","last_change":"2024-03-06T07:34:09.298981+0000","last_active":"2024-03-06T07:34:11.007597+0000","last_peered":"2024-03-06T07:34:11.007597+0000","last_clean":"2024-03-06T07:34:11.007597+0000","last_became_active":"2024-03-06T07:34:09.296016+0000","last_became_peered":"2024-03-06T07:34:09.296016+0000","last_unstale":"2024-03-06T07:34:11.007597+0000","last_undegraded":"2024-03-06T07:34:11.007597+0000","last_fullsized":"2024-03-06T07:34:11.007597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:27.293917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003760+0000","last_change":"2024-03-06T07:34:08.270938+0000","last_active":"2024-03-06T07:34:11.003760+0000","last_peered":"2024-03-06T07:34:11.003760+0000","last_clean":"2024-03-06T07:34:11.003760+0000","last_became_active":"2024-03-06T07:34:08.270180+0000","last_became_peered":"2024-03-06T07:34:08.270180+0000","last_unstale":"2024-03-06T07:34:11.003760+0000","last_undegraded":"2024-03-06T07:34:11.003760+0000","last_fullsized":"2024-03-06T07:34:11.003760+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005724+0000","last_change":"2024-03-06T07:34:09.300024+0000","last_active":"2024-03-06T07:34:11.005724+0000","last_peered":"2024-03-06T07:34:11.005724+0000","last_clean":"2024-03-06T07:34:11.005724+0000","last_became_active":"2024-03-06T07:34:09.299739+0000","last_became_peered":"2024-03-06T07:34:09.299739+0000","last_unstale":"2024-03-06T07:34:11.005724+0000","last_undegraded":"2024-03-06T07:34:11.005724+0000","last_fullsized":"2024-03-06T07:34:11.005724+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:57.939975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006456+0000","last_change":"2024-03-06T07:34:08.272524+0000","last_active":"2024-03-06T07:34:11.006456+0000","last_peered":"2024-03-06T07:34:11.006456+0000","last_clean":"2024-03-06T07:34:11.006456+0000","last_became_active":"2024-03-06T07:34:08.272078+0000","last_became_peered":"2024-03-06T07:34:08.272078+0000","last_unstale":"2024-03-06T07:34:11.006456+0000","last_undegraded":"2024-03-06T07:34:11.006456+0000","last_fullsized":"2024-03-06T07:34:11.006456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.298132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007077+0000","last_change":"2024-03-06T07:34:09.297770+0000","last_active":"2024-03-06T07:34:11.007077+0000","last_peered":"2024-03-06T07:34:11.007077+0000","last_clean":"2024-03-06T07:34:11.007077+0000","last_became_active":"2024-03-06T07:34:09.297457+0000","last_became_peered":"2024-03-06T07:34:09.297457+0000","last_unstale":"2024-03-06T07:34:11.007077+0000","last_undegraded":"2024-03-06T07:34:11.007077+0000","last_fullsized":"2024-03-06T07:34:11.007077+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:40.530510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300777+0000","last_change":"2024-03-06T07:34:08.275513+0000","last_active":"2024-03-06T07:34:10.300777+0000","last_peered":"2024-03-06T07:34:10.300777+0000","last_clean":"2024-03-06T07:34:10.300777+0000","last_became_active":"2024-03-06T07:34:08.275177+0000","last_became_peered":"2024-03-06T07:34:08.275177+0000","last_unstale":"2024-03-06T07:34:10.300777+0000","last_undegraded":"2024-03-06T07:34:10.300777+0000","last_fullsized":"2024-03-06T07:34:10.300777+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:49.452341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306140+0000","last_change":"2024-03-06T07:34:09.293082+0000","last_active":"2024-03-06T07:34:10.306140+0000","last_peered":"2024-03-06T07:34:10.306140+0000","last_clean":"2024-03-06T07:34:10.306140+0000","last_became_active":"2024-03-06T07:34:09.292719+0000","last_became_peered":"2024-03-06T07:34:09.292719+0000","last_unstale":"2024-03-06T07:34:10.306140+0000","last_undegraded":"2024-03-06T07:34:10.306140+0000","last_fullsized":"2024-03-06T07:34:10.306140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:49.376701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300683+0000","last_change":"2024-03-06T07:34:08.272959+0000","last_active":"2024-03-06T07:34:10.300683+0000","last_peered":"2024-03-06T07:34:10.300683+0000","last_clean":"2024-03-06T07:34:10.300683+0000","last_became_active":"2024-03-06T07:34:08.271962+0000","last_became_peered":"2024-03-06T07:34:08.271962+0000","last_unstale":"2024-03-06T07:34:10.300683+0000","last_undegraded":"2024-03-06T07:34:10.300683+0000","last_fullsized":"2024-03-06T07:34:10.300683+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:39.459734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305569+0000","last_change":"2024-03-06T07:34:09.305597+0000","last_active":"2024-03-06T07:34:10.305569+0000","last_peered":"2024-03-06T07:34:10.305569+0000","last_clean":"2024-03-06T07:34:10.305569+0000","last_became_active":"2024-03-06T07:34:09.304907+0000","last_became_peered":"2024-03-06T07:34:09.304907+0000","last_unstale":"2024-03-06T07:34:10.305569+0000","last_undegraded":"2024-03-06T07:34:10.305569+0000","last_fullsized":"2024-03-06T07:34:10.305569+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:57.642232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005917+0000","last_change":"2024-03-06T07:34:08.275800+0000","last_active":"2024-03-06T07:34:11.005917+0000","last_peered":"2024-03-06T07:34:11.005917+0000","last_clean":"2024-03-06T07:34:11.005917+0000","last_became_active":"2024-03-06T07:34:08.275476+0000","last_became_peered":"2024-03-06T07:34:08.275476+0000","last_unstale":"2024-03-06T07:34:11.005917+0000","last_undegraded":"2024-03-06T07:34:11.005917+0000","last_fullsized":"2024-03-06T07:34:11.005917+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:54.100822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305023+0000","last_change":"2024-03-06T07:34:09.292697+0000","last_active":"2024-03-06T07:34:10.305023+0000","last_peered":"2024-03-06T07:34:10.305023+0000","last_clean":"2024-03-06T07:34:10.305023+0000","last_became_active":"2024-03-06T07:34:09.292397+0000","last_became_peered":"2024-03-06T07:34:09.292397+0000","last_unstale":"2024-03-06T07:34:10.305023+0000","last_undegraded":"2024-03-06T07:34:10.305023+0000","last_fullsized":"2024-03-06T07:34:10.305023+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:40.677711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005508+0000","last_change":"2024-03-06T07:34:08.274027+0000","last_active":"2024-03-06T07:34:11.005508+0000","last_peered":"2024-03-06T07:34:11.005508+0000","last_clean":"2024-03-06T07:34:11.005508+0000","last_became_active":"2024-03-06T07:34:08.273695+0000","last_became_peered":"2024-03-06T07:34:08.273695+0000","last_unstale":"2024-03-06T07:34:11.005508+0000","last_undegraded":"2024-03-06T07:34:11.005508+0000","last_fullsized":"2024-03-06T07:34:11.005508+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:55.267247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004182+0000","last_change":"2024-03-06T07:34:09.298873+0000","last_active":"2024-03-06T07:34:11.004182+0000","last_peered":"2024-03-06T07:34:11.004182+0000","last_clean":"2024-03-06T07:34:11.004182+0000","last_became_active":"2024-03-06T07:34:09.298515+0000","last_became_peered":"2024-03-06T07:34:09.298515+0000","last_unstale":"2024-03-06T07:34:11.004182+0000","last_undegraded":"2024-03-06T07:34:11.004182+0000","last_fullsized":"2024-03-06T07:34:11.004182+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:19.254659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300438+0000","last_change":"2024-03-06T07:34:08.274029+0000","last_active":"2024-03-06T07:34:10.300438+0000","last_peered":"2024-03-06T07:34:10.300438+0000","last_clean":"2024-03-06T07:34:10.300438+0000","last_became_active":"2024-03-06T07:34:08.273759+0000","last_became_peered":"2024-03-06T07:34:08.273759+0000","last_unstale":"2024-03-06T07:34:10.300438+0000","last_undegraded":"2024-03-06T07:34:10.300438+0000","last_fullsized":"2024-03-06T07:34:10.300438+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:44.005288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301343+0000","last_change":"2024-03-06T07:34:09.298874+0000","last_active":"2024-03-06T07:34:10.301343+0000","last_peered":"2024-03-06T07:34:10.301343+0000","last_clean":"2024-03-06T07:34:10.301343+0000","last_became_active":"2024-03-06T07:34:09.298602+0000","last_became_peered":"2024-03-06T07:34:09.298602+0000","last_unstale":"2024-03-06T07:34:10.301343+0000","last_undegraded":"2024-03-06T07:34:10.301343+0000","last_fullsized":"2024-03-06T07:34:10.301343+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:07.544494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005772+0000","last_change":"2024-03-06T07:34:08.279992+0000","last_active":"2024-03-06T07:34:11.005772+0000","last_peered":"2024-03-06T07:34:11.005772+0000","last_clean":"2024-03-06T07:34:11.005772+0000","last_became_active":"2024-03-06T07:34:08.279058+0000","last_became_peered":"2024-03-06T07:34:08.279058+0000","last_unstale":"2024-03-06T07:34:11.005772+0000","last_undegraded":"2024-03-06T07:34:11.005772+0000","last_fullsized":"2024-03-06T07:34:11.005772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:06.914393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004106+0000","last_change":"2024-03-06T07:34:09.290526+0000","last_active":"2024-03-06T07:34:11.004106+0000","last_peered":"2024-03-06T07:34:11.004106+0000","last_clean":"2024-03-06T07:34:11.004106+0000","last_became_active":"2024-03-06T07:34:09.290063+0000","last_became_peered":"2024-03-06T07:34:09.290063+0000","last_unstale":"2024-03-06T07:34:11.004106+0000","last_undegraded":"2024-03-06T07:34:11.004106+0000","last_fullsized":"2024-03-06T07:34:11.004106+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:31.093849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301495+0000","last_change":"2024-03-06T07:34:08.271690+0000","last_active":"2024-03-06T07:34:10.301495+0000","last_peered":"2024-03-06T07:34:10.301495+0000","last_clean":"2024-03-06T07:34:10.301495+0000","last_became_active":"2024-03-06T07:34:08.271425+0000","last_became_peered":"2024-03-06T07:34:08.271425+0000","last_unstale":"2024-03-06T07:34:10.301495+0000","last_undegraded":"2024-03-06T07:34:10.301495+0000","last_fullsized":"2024-03-06T07:34:10.301495+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:43.524157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003563+0000","last_change":"2024-03-06T07:34:09.297391+0000","last_active":"2024-03-06T07:34:11.003563+0000","last_peered":"2024-03-06T07:34:11.003563+0000","last_clean":"2024-03-06T07:34:11.003563+0000","last_became_active":"2024-03-06T07:34:09.297107+0000","last_became_peered":"2024-03-06T07:34:09.297107+0000","last_unstale":"2024-03-06T07:34:11.003563+0000","last_undegraded":"2024-03-06T07:34:11.003563+0000","last_fullsized":"2024-03-06T07:34:11.003563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:32.178946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300833+0000","last_change":"2024-03-06T07:34:08.276819+0000","last_active":"2024-03-06T07:34:10.300833+0000","last_peered":"2024-03-06T07:34:10.300833+0000","last_clean":"2024-03-06T07:34:10.300833+0000","last_became_active":"2024-03-06T07:34:08.276500+0000","last_became_peered":"2024-03-06T07:34:08.276500+0000","last_unstale":"2024-03-06T07:34:10.300833+0000","last_undegraded":"2024-03-06T07:34:10.300833+0000","last_fullsized":"2024-03-06T07:34:10.300833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:03.411533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006246+0000","last_change":"2024-03-06T07:34:09.300002+0000","last_active":"2024-03-06T07:34:11.006246+0000","last_peered":"2024-03-06T07:34:11.006246+0000","last_clean":"2024-03-06T07:34:11.006246+0000","last_became_active":"2024-03-06T07:34:09.299553+0000","last_became_peered":"2024-03-06T07:34:09.299553+0000","last_unstale":"2024-03-06T07:34:11.006246+0000","last_undegraded":"2024-03-06T07:34:11.006246+0000","last_fullsized":"2024-03-06T07:34:11.006246+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:04.427214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006365+0000","last_change":"2024-03-06T07:34:08.277478+0000","last_active":"2024-03-06T07:34:11.006365+0000","last_peered":"2024-03-06T07:34:11.006365+0000","last_clean":"2024-03-06T07:34:11.006365+0000","last_became_active":"2024-03-06T07:34:08.277186+0000","last_became_peered":"2024-03-06T07:34:08.277186+0000","last_unstale":"2024-03-06T07:34:11.006365+0000","last_undegraded":"2024-03-06T07:34:11.006365+0000","last_fullsized":"2024-03-06T07:34:11.006365+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:09.498839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306597+0000","last_change":"2024-03-06T07:34:09.306423+0000","last_active":"2024-03-06T07:34:10.306597+0000","last_peered":"2024-03-06T07:34:10.306597+0000","last_clean":"2024-03-06T07:34:10.306597+0000","last_became_active":"2024-03-06T07:34:09.302507+0000","last_became_peered":"2024-03-06T07:34:09.302507+0000","last_unstale":"2024-03-06T07:34:10.306597+0000","last_undegraded":"2024-03-06T07:34:10.306597+0000","last_fullsized":"2024-03-06T07:34:10.306597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:52.517967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300901+0000","last_change":"2024-03-06T07:34:08.272451+0000","last_active":"2024-03-06T07:34:10.300901+0000","last_peered":"2024-03-06T07:34:10.300901+0000","last_clean":"2024-03-06T07:34:10.300901+0000","last_became_active":"2024-03-06T07:34:08.272175+0000","last_became_peered":"2024-03-06T07:34:08.272175+0000","last_unstale":"2024-03-06T07:34:10.300901+0000","last_undegraded":"2024-03-06T07:34:10.300901+0000","last_fullsized":"2024-03-06T07:34:10.300901+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.388532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003157+0000","last_change":"2024-03-06T07:34:09.301236+0000","last_active":"2024-03-06T07:34:11.003157+0000","last_peered":"2024-03-06T07:34:11.003157+0000","last_clean":"2024-03-06T07:34:11.003157+0000","last_became_active":"2024-03-06T07:34:09.300337+0000","last_became_peered":"2024-03-06T07:34:09.300337+0000","last_unstale":"2024-03-06T07:34:11.003157+0000","last_undegraded":"2024-03-06T07:34:11.003157+0000","last_fullsized":"2024-03-06T07:34:11.003157+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:12.746533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004465+0000","last_change":"2024-03-06T07:34:08.274395+0000","last_active":"2024-03-06T07:34:11.004465+0000","last_peered":"2024-03-06T07:34:11.004465+0000","last_clean":"2024-03-06T07:34:11.004465+0000","last_became_active":"2024-03-06T07:34:08.274068+0000","last_became_peered":"2024-03-06T07:34:08.274068+0000","last_unstale":"2024-03-06T07:34:11.004465+0000","last_undegraded":"2024-03-06T07:34:11.004465+0000","last_fullsized":"2024-03-06T07:34:11.004465+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:41.468854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305022+0000","last_change":"2024-03-06T07:34:09.303120+0000","last_active":"2024-03-06T07:34:10.305022+0000","last_peered":"2024-03-06T07:34:10.305022+0000","last_clean":"2024-03-06T07:34:10.305022+0000","last_became_active":"2024-03-06T07:34:09.299011+0000","last_became_peered":"2024-03-06T07:34:09.299011+0000","last_unstale":"2024-03-06T07:34:10.305022+0000","last_undegraded":"2024-03-06T07:34:10.305022+0000","last_fullsized":"2024-03-06T07:34:10.305022+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:44.070915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290053+0000","last_change":"2024-03-06T07:34:08.257721+0000","last_active":"2024-03-06T07:34:10.290053+0000","last_peered":"2024-03-06T07:34:10.290053+0000","last_clean":"2024-03-06T07:34:10.290053+0000","last_became_active":"2024-03-06T07:34:08.257322+0000","last_became_peered":"2024-03-06T07:34:08.257322+0000","last_unstale":"2024-03-06T07:34:10.290053+0000","last_undegraded":"2024-03-06T07:34:10.290053+0000","last_fullsized":"2024-03-06T07:34:10.290053+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:39.459734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004391+0000","last_change":"2024-03-06T07:34:09.305318+0000","last_active":"2024-03-06T07:34:11.004391+0000","last_peered":"2024-03-06T07:34:11.004391+0000","last_clean":"2024-03-06T07:34:11.004391+0000","last_became_active":"2024-03-06T07:34:09.304946+0000","last_became_peered":"2024-03-06T07:34:09.304946+0000","last_unstale":"2024-03-06T07:34:11.004391+0000","last_undegraded":"2024-03-06T07:34:11.004391+0000","last_fullsized":"2024-03-06T07:34:11.004391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:45.020969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305330+0000","last_change":"2024-03-06T07:34:08.279635+0000","last_active":"2024-03-06T07:34:10.305330+0000","last_peered":"2024-03-06T07:34:10.305330+0000","last_clean":"2024-03-06T07:34:10.305330+0000","last_became_active":"2024-03-06T07:34:08.279250+0000","last_became_peered":"2024-03-06T07:34:08.279250+0000","last_unstale":"2024-03-06T07:34:10.305330+0000","last_undegraded":"2024-03-06T07:34:10.305330+0000","last_fullsized":"2024-03-06T07:34:10.305330+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:17.184769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305935+0000","last_change":"2024-03-06T07:34:09.300151+0000","last_active":"2024-03-06T07:34:10.305935+0000","last_peered":"2024-03-06T07:34:10.305935+0000","last_clean":"2024-03-06T07:34:10.305935+0000","last_became_active":"2024-03-06T07:34:09.299873+0000","last_became_peered":"2024-03-06T07:34:09.299873+0000","last_unstale":"2024-03-06T07:34:10.305935+0000","last_undegraded":"2024-03-06T07:34:10.305935+0000","last_fullsized":"2024-03-06T07:34:10.305935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:58.234520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006188+0000","last_change":"2024-03-06T07:34:08.276832+0000","last_active":"2024-03-06T07:34:11.006188+0000","last_peered":"2024-03-06T07:34:11.006188+0000","last_clean":"2024-03-06T07:34:11.006188+0000","last_became_active":"2024-03-06T07:34:08.276488+0000","last_became_peered":"2024-03-06T07:34:08.276488+0000","last_unstale":"2024-03-06T07:34:11.006188+0000","last_undegraded":"2024-03-06T07:34:11.006188+0000","last_fullsized":"2024-03-06T07:34:11.006188+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.730852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":310,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004681+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:34:11.004681+0000","last_peered":"2024-03-06T07:34:11.004681+0000","last_clean":"2024-03-06T07:34:11.004681+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:34:11.004681+0000","last_undegraded":"2024-03-06T07:34:11.004681+0000","last_fullsized":"2024-03-06T07:34:11.004681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1777664,"data_stored":1092681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":502853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1179648,"data_stored":502271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36512,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335328,"statfs":{"total":96636764160,"available":96599375872,"internally_reserved":0,"allocated":1196032,"data_stored":503937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335336,"statfs":{"total":96636764160,"available":96599384064,"internally_reserved":0,"allocated":1191936,"data_stored":503835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":502235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1175552,"data_stored":502201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1773568,"data_stored":1093151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:18.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.065+0000 7f269ea00640 1 Processor -- start 2024-03-06T07:34:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.065+0000 7f269ea00640 1 -- start start 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26981552b0 0x7f269815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269ea00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269815ed10 con 0x7f269815e810 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269ea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269815eff0 con 0x7f26980cd060 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269ea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269815f2d0 con 0x7f26981552b0 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269d600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26981552b0 0x7f269815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269cc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f269815e810 0x7f26980cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47022/0 (socket says 172.21.15.32:47022) 2024-03-06T07:34:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.067+0000 7f269cc00640 1 -- 172.21.15.32:0/3010726901 learned_addr learned my addr 172.21.15.32:0/3010726901 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.068+0000 7f269d600640 1 -- 172.21.15.32:0/3010726901 >> v1:172.21.15.39:6789/0 conn(0x7f26980cd060 legacy=0x7f26980b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.068+0000 7f269d600640 1 -- 172.21.15.32:0/3010726901 >> v1:172.21.15.32:6789/0 conn(0x7f269815e810 legacy=0x7f26980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.068+0000 7f269d600640 1 -- 172.21.15.32:0/3010726901 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269815f5b0 con 0x7f26981552b0 2024-03-06T07:34:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.068+0000 7f2697e00640 1 -- 172.21.15.32:0/3010726901 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880378095 0 0) 0x7f269815ed10 con 0x7f269815e810 2024-03-06T07:34:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.069+0000 7f269d600640 1 --2- 172.21.15.32:0/3010726901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f269815b1d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097a0 tx=0x7f268c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4769d4064c17ab95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.069+0000 7f2697e00640 1 -- 172.21.15.32:0/3010726901 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c0315a0 con 0x7f26981552b0 2024-03-06T07:34:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.069+0000 7f2697e00640 1 -- 172.21.15.32:0/3010726901 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c037070 con 0x7f26981552b0 2024-03-06T07:34:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.069+0000 7f2697e00640 1 -- 172.21.15.32:0/3010726901 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c031960 con 0x7f26981552b0 2024-03-06T07:34:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 -- 172.21.15.32:0/3010726901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 msgr2=0x7f269815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 --2- 172.21.15.32:0/3010726901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f269815b1d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097a0 tx=0x7f268c02b840 comp rx=0 tx=0).stop 2024-03-06T07:34:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 -- 172.21.15.32:0/3010726901 shutdown_connections 2024-03-06T07:34:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 --2- 172.21.15.32:0/3010726901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f269815b1d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 -- 172.21.15.32:0/3010726901 >> 172.21.15.32:0/3010726901 conn(0x7f269806b3d0 msgr2=0x7f269806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 -- 172.21.15.32:0/3010726901 shutdown_connections 2024-03-06T07:34:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.070+0000 7f269ea00640 1 -- 172.21.15.32:0/3010726901 wait complete. 2024-03-06T07:34:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.072+0000 7f269ea00640 1 Processor -- start 2024-03-06T07:34:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.072+0000 7f269ea00640 1 -- start start 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26980cd060 0x7f2698141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f2698158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f269815e810 0x7f26981594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269806dd30 con 0x7f26981552b0 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26980b3760 con 0x7f269815e810 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269806deb0 con 0x7f26980cd060 2024-03-06T07:34:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f2698158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f2698158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40248/0 (socket says 172.21.15.32:40248) 2024-03-06T07:34:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269cc00640 1 -- 172.21.15.32:0/1631444106 learned_addr learned my addr 172.21.15.32:0/1631444106 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269e000640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f269815e810 0x7f26981594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.073+0000 7f269d600640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26980cd060 0x7f2698141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269cc00640 1 -- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26980cd060 msgr2=0x7f2698141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269cc00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26980cd060 0x7f2698141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269cc00640 1 -- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f269815e810 msgr2=0x7f26981594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269cc00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f269815e810 0x7f26981594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269cc00640 1 -- 172.21.15.32:0/1631444106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2698159ad0 con 0x7f26981552b0 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269d600640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26980cd060 0x7f2698141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269cc00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f2698158fa0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f268400c430 tx=0x7f268400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26840074e0 con 0x7f26981552b0 2024-03-06T07:34:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26840047e0 con 0x7f26981552b0 2024-03-06T07:34:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684004bd0 con 0x7f26981552b0 2024-03-06T07:34:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698159db0 con 0x7f26981552b0 2024-03-06T07:34:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.074+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269815a370 con 0x7f26981552b0 2024-03-06T07:34:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.075+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269804f330 con 0x7f26981552b0 2024-03-06T07:34:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.076+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2684011070 con 0x7f26981552b0 2024-03-06T07:34:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.077+0000 7f2695600640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f267806b4d0 0x7f267806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.077+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2684089f10 con 0x7f26981552b0 2024-03-06T07:34:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.078+0000 7f269d600640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f267806b4d0 0x7f267806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.078+0000 7f269d600640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f267806b4d0 0x7f267806d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f268c036000 tx=0x7f268c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.082+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f268400e280 con 0x7f26981552b0 2024-03-06T07:34:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.280+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26980745e0 con 0x7f267806b4d0 2024-03-06T07:34:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.299+0000 7f2695600640 1 -- 172.21.15.32:0/1631444106 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252074 (secure 0 0 0) 0x7f26980745e0 con 0x7f267806b4d0 2024-03-06T07:34:19.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.305+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f267806b4d0 msgr2=0x7f267806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.305+0000 7f269ea00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f267806b4d0 0x7f267806d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f268c036000 tx=0x7f268c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.305+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 msgr2=0x7f2698158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.305+0000 7f269ea00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f2698158fa0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f268400c430 tx=0x7f268400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 shutdown_connections 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f267806b4d0 0x7f267806d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f269815e810 0x7f26981594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26981552b0 0x7f2698158fa0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 --2- 172.21.15.32:0/1631444106 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f26980cd060 0x7f2698141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 >> 172.21.15.32:0/1631444106 conn(0x7f269806b3d0 msgr2=0x7f26981545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 shutdown_connections 2024-03-06T07:34:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.306+0000 7f269ea00640 1 -- 172.21.15.32:0/1631444106 wait complete. 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-03-06T07:34:18.283854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276504,"kb_used_data":10408,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698216,"statfs":{"total":773094113280,"available":772810973184,"internally_reserved":0,"allocated":10657792,"data_stored":5203164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.024127"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003602+0000","last_change":"2024-03-06T07:34:09.286619+0000","last_active":"2024-03-06T07:34:11.003602+0000","last_peered":"2024-03-06T07:34:11.003602+0000","last_clean":"2024-03-06T07:34:11.003602+0000","last_became_active":"2024-03-06T07:34:09.286339+0000","last_became_peered":"2024-03-06T07:34:09.286339+0000","last_unstale":"2024-03-06T07:34:11.003602+0000","last_undegraded":"2024-03-06T07:34:11.003602+0000","last_fullsized":"2024-03-06T07:34:11.003602+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:54.474565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300528+0000","last_change":"2024-03-06T07:34:09.310443+0000","last_active":"2024-03-06T07:34:10.300528+0000","last_peered":"2024-03-06T07:34:10.300528+0000","last_clean":"2024-03-06T07:34:10.300528+0000","last_became_active":"2024-03-06T07:34:09.310141+0000","last_became_peered":"2024-03-06T07:34:09.310141+0000","last_unstale":"2024-03-06T07:34:10.300528+0000","last_undegraded":"2024-03-06T07:34:10.300528+0000","last_fullsized":"2024-03-06T07:34:10.300528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:30:11.163295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004046+0000","last_change":"2024-03-06T07:34:09.286683+0000","last_active":"2024-03-06T07:34:11.004046+0000","last_peered":"2024-03-06T07:34:11.004046+0000","last_clean":"2024-03-06T07:34:11.004046+0000","last_became_active":"2024-03-06T07:34:09.286264+0000","last_became_peered":"2024-03-06T07:34:09.286264+0000","last_unstale":"2024-03-06T07:34:11.004046+0000","last_undegraded":"2024-03-06T07:34:11.004046+0000","last_fullsized":"2024-03-06T07:34:11.004046+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:23.455135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290827+0000","last_change":"2024-03-06T07:34:09.291689+0000","last_active":"2024-03-06T07:34:10.290827+0000","last_peered":"2024-03-06T07:34:10.290827+0000","last_clean":"2024-03-06T07:34:10.290827+0000","last_became_active":"2024-03-06T07:34:09.290808+0000","last_became_peered":"2024-03-06T07:34:09.290808+0000","last_unstale":"2024-03-06T07:34:10.290827+0000","last_undegraded":"2024-03-06T07:34:10.290827+0000","last_fullsized":"2024-03-06T07:34:10.290827+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:40.530510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004935+0000","last_change":"2024-03-06T07:34:09.303097+0000","last_active":"2024-03-06T07:34:11.004935+0000","last_peered":"2024-03-06T07:34:11.004935+0000","last_clean":"2024-03-06T07:34:11.004935+0000","last_became_active":"2024-03-06T07:34:09.302816+0000","last_became_peered":"2024-03-06T07:34:09.302816+0000","last_unstale":"2024-03-06T07:34:11.004935+0000","last_undegraded":"2024-03-06T07:34:11.004935+0000","last_fullsized":"2024-03-06T07:34:11.004935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:40.475415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006200+0000","last_change":"2024-03-06T07:34:09.282622+0000","last_active":"2024-03-06T07:34:11.006200+0000","last_peered":"2024-03-06T07:34:11.006200+0000","last_clean":"2024-03-06T07:34:11.006200+0000","last_became_active":"2024-03-06T07:34:09.282257+0000","last_became_peered":"2024-03-06T07:34:09.282257+0000","last_unstale":"2024-03-06T07:34:11.006200+0000","last_undegraded":"2024-03-06T07:34:11.006200+0000","last_fullsized":"2024-03-06T07:34:11.006200+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:43.980518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004120+0000","last_change":"2024-03-06T07:34:09.283961+0000","last_active":"2024-03-06T07:34:11.004120+0000","last_peered":"2024-03-06T07:34:11.004120+0000","last_clean":"2024-03-06T07:34:11.004120+0000","last_became_active":"2024-03-06T07:34:09.283671+0000","last_became_peered":"2024-03-06T07:34:09.283671+0000","last_unstale":"2024-03-06T07:34:11.004120+0000","last_undegraded":"2024-03-06T07:34:11.004120+0000","last_fullsized":"2024-03-06T07:34:11.004120+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:10.689868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300846+0000","last_change":"2024-03-06T07:34:09.294383+0000","last_active":"2024-03-06T07:34:10.300846+0000","last_peered":"2024-03-06T07:34:10.300846+0000","last_clean":"2024-03-06T07:34:10.300846+0000","last_became_active":"2024-03-06T07:34:09.294078+0000","last_became_peered":"2024-03-06T07:34:09.294078+0000","last_unstale":"2024-03-06T07:34:10.300846+0000","last_undegraded":"2024-03-06T07:34:10.300846+0000","last_fullsized":"2024-03-06T07:34:10.300846+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:52.440212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004429+0000","last_change":"2024-03-06T07:34:09.304643+0000","last_active":"2024-03-06T07:34:11.004429+0000","last_peered":"2024-03-06T07:34:11.004429+0000","last_clean":"2024-03-06T07:34:11.004429+0000","last_became_active":"2024-03-06T07:34:09.304161+0000","last_became_peered":"2024-03-06T07:34:09.304161+0000","last_unstale":"2024-03-06T07:34:11.004429+0000","last_undegraded":"2024-03-06T07:34:11.004429+0000","last_fullsized":"2024-03-06T07:34:11.004429+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:40.475415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007072+0000","last_change":"2024-03-06T07:34:09.307259+0000","last_active":"2024-03-06T07:34:11.007072+0000","last_peered":"2024-03-06T07:34:11.007072+0000","last_clean":"2024-03-06T07:34:11.007072+0000","last_became_active":"2024-03-06T07:34:09.306951+0000","last_became_peered":"2024-03-06T07:34:09.306951+0000","last_unstale":"2024-03-06T07:34:11.007072+0000","last_undegraded":"2024-03-06T07:34:11.007072+0000","last_fullsized":"2024-03-06T07:34:11.007072+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:46.651651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006768+0000","last_change":"2024-03-06T07:34:09.300094+0000","last_active":"2024-03-06T07:34:11.006768+0000","last_peered":"2024-03-06T07:34:11.006768+0000","last_clean":"2024-03-06T07:34:11.006768+0000","last_became_active":"2024-03-06T07:34:09.296249+0000","last_became_peered":"2024-03-06T07:34:09.296249+0000","last_unstale":"2024-03-06T07:34:11.006768+0000","last_undegraded":"2024-03-06T07:34:11.006768+0000","last_fullsized":"2024-03-06T07:34:11.006768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:07.616565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003277+0000","last_change":"2024-03-06T07:34:09.298489+0000","last_active":"2024-03-06T07:34:11.003277+0000","last_peered":"2024-03-06T07:34:11.003277+0000","last_clean":"2024-03-06T07:34:11.003277+0000","last_became_active":"2024-03-06T07:34:09.298216+0000","last_became_peered":"2024-03-06T07:34:09.298216+0000","last_unstale":"2024-03-06T07:34:11.003277+0000","last_undegraded":"2024-03-06T07:34:11.003277+0000","last_fullsized":"2024-03-06T07:34:11.003277+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:50.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290411+0000","last_change":"2024-03-06T07:34:09.298570+0000","last_active":"2024-03-06T07:34:10.290411+0000","last_peered":"2024-03-06T07:34:10.290411+0000","last_clean":"2024-03-06T07:34:10.290411+0000","last_became_active":"2024-03-06T07:34:09.298245+0000","last_became_peered":"2024-03-06T07:34:09.298245+0000","last_unstale":"2024-03-06T07:34:10.290411+0000","last_undegraded":"2024-03-06T07:34:10.290411+0000","last_fullsized":"2024-03-06T07:34:10.290411+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:00.105663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305124+0000","last_change":"2024-03-06T07:34:09.307111+0000","last_active":"2024-03-06T07:34:10.305124+0000","last_peered":"2024-03-06T07:34:10.305124+0000","last_clean":"2024-03-06T07:34:10.305124+0000","last_became_active":"2024-03-06T07:34:09.306758+0000","last_became_peered":"2024-03-06T07:34:09.306758+0000","last_unstale":"2024-03-06T07:34:10.305124+0000","last_undegraded":"2024-03-06T07:34:10.305124+0000","last_fullsized":"2024-03-06T07:34:10.305124+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:47.141751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289787+0000","last_change":"2024-03-06T07:34:09.282819+0000","last_active":"2024-03-06T07:34:10.289787+0000","last_peered":"2024-03-06T07:34:10.289787+0000","last_clean":"2024-03-06T07:34:10.289787+0000","last_became_active":"2024-03-06T07:34:09.282473+0000","last_became_peered":"2024-03-06T07:34:09.282473+0000","last_unstale":"2024-03-06T07:34:10.289787+0000","last_undegraded":"2024-03-06T07:34:10.289787+0000","last_fullsized":"2024-03-06T07:34:10.289787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.761768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004119+0000","last_change":"2024-03-06T07:34:09.298366+0000","last_active":"2024-03-06T07:34:11.004119+0000","last_peered":"2024-03-06T07:34:11.004119+0000","last_clean":"2024-03-06T07:34:11.004119+0000","last_became_active":"2024-03-06T07:34:09.298038+0000","last_became_peered":"2024-03-06T07:34:09.298038+0000","last_unstale":"2024-03-06T07:34:11.004119+0000","last_undegraded":"2024-03-06T07:34:11.004119+0000","last_fullsized":"2024-03-06T07:34:11.004119+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:15.349718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301431+0000","last_change":"2024-03-06T07:34:09.312585+0000","last_active":"2024-03-06T07:34:10.301431+0000","last_peered":"2024-03-06T07:34:10.301431+0000","last_clean":"2024-03-06T07:34:10.301431+0000","last_became_active":"2024-03-06T07:34:09.311067+0000","last_became_peered":"2024-03-06T07:34:09.311067+0000","last_unstale":"2024-03-06T07:34:10.301431+0000","last_undegraded":"2024-03-06T07:34:10.301431+0000","last_fullsized":"2024-03-06T07:34:10.301431+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.682188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290846+0000","last_change":"2024-03-06T07:34:09.292319+0000","last_active":"2024-03-06T07:34:10.290846+0000","last_peered":"2024-03-06T07:34:10.290846+0000","last_clean":"2024-03-06T07:34:10.290846+0000","last_became_active":"2024-03-06T07:34:09.292019+0000","last_became_peered":"2024-03-06T07:34:09.292019+0000","last_unstale":"2024-03-06T07:34:10.290846+0000","last_undegraded":"2024-03-06T07:34:10.290846+0000","last_fullsized":"2024-03-06T07:34:10.290846+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:07.499061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305752+0000","last_change":"2024-03-06T07:34:09.297981+0000","last_active":"2024-03-06T07:34:10.305752+0000","last_peered":"2024-03-06T07:34:10.305752+0000","last_clean":"2024-03-06T07:34:10.305752+0000","last_became_active":"2024-03-06T07:34:09.296988+0000","last_became_peered":"2024-03-06T07:34:09.296988+0000","last_unstale":"2024-03-06T07:34:10.305752+0000","last_undegraded":"2024-03-06T07:34:10.305752+0000","last_fullsized":"2024-03-06T07:34:10.305752+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:10.000710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003440+0000","last_change":"2024-03-06T07:34:09.296148+0000","last_active":"2024-03-06T07:34:11.003440+0000","last_peered":"2024-03-06T07:34:11.003440+0000","last_clean":"2024-03-06T07:34:11.003440+0000","last_became_active":"2024-03-06T07:34:09.295849+0000","last_became_peered":"2024-03-06T07:34:09.295849+0000","last_unstale":"2024-03-06T07:34:11.003440+0000","last_undegraded":"2024-03-06T07:34:11.003440+0000","last_fullsized":"2024-03-06T07:34:11.003440+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:19.254659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305347+0000","last_change":"2024-03-06T07:34:09.296885+0000","last_active":"2024-03-06T07:34:10.305347+0000","last_peered":"2024-03-06T07:34:10.305347+0000","last_clean":"2024-03-06T07:34:10.305347+0000","last_became_active":"2024-03-06T07:34:09.293552+0000","last_became_peered":"2024-03-06T07:34:09.293552+0000","last_unstale":"2024-03-06T07:34:10.305347+0000","last_undegraded":"2024-03-06T07:34:10.305347+0000","last_fullsized":"2024-03-06T07:34:10.305347+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:04.837064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300391+0000","last_change":"2024-03-06T07:34:09.306061+0000","last_active":"2024-03-06T07:34:10.300391+0000","last_peered":"2024-03-06T07:34:10.300391+0000","last_clean":"2024-03-06T07:34:10.300391+0000","last_became_active":"2024-03-06T07:34:09.305791+0000","last_became_peered":"2024-03-06T07:34:09.305791+0000","last_unstale":"2024-03-06T07:34:10.300391+0000","last_undegraded":"2024-03-06T07:34:10.300391+0000","last_fullsized":"2024-03-06T07:34:10.300391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:34.363051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300454+0000","last_change":"2024-03-06T07:34:09.298023+0000","last_active":"2024-03-06T07:34:10.300454+0000","last_peered":"2024-03-06T07:34:10.300454+0000","last_clean":"2024-03-06T07:34:10.300454+0000","last_became_active":"2024-03-06T07:34:09.297768+0000","last_became_peered":"2024-03-06T07:34:09.297768+0000","last_unstale":"2024-03-06T07:34:10.300454+0000","last_undegraded":"2024-03-06T07:34:10.300454+0000","last_fullsized":"2024-03-06T07:34:10.300454+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:09.786475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004542+0000","last_change":"2024-03-06T07:34:09.278562+0000","last_active":"2024-03-06T07:34:11.004542+0000","last_peered":"2024-03-06T07:34:11.004542+0000","last_clean":"2024-03-06T07:34:11.004542+0000","last_became_active":"2024-03-06T07:34:09.277675+0000","last_became_peered":"2024-03-06T07:34:09.277675+0000","last_unstale":"2024-03-06T07:34:11.004542+0000","last_undegraded":"2024-03-06T07:34:11.004542+0000","last_fullsized":"2024-03-06T07:34:11.004542+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:29.761768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007435+0000","last_change":"2024-03-06T07:34:09.301732+0000","last_active":"2024-03-06T07:34:11.007435+0000","last_peered":"2024-03-06T07:34:11.007435+0000","last_clean":"2024-03-06T07:34:11.007435+0000","last_became_active":"2024-03-06T07:34:09.301136+0000","last_became_peered":"2024-03-06T07:34:09.301136+0000","last_unstale":"2024-03-06T07:34:11.007435+0000","last_undegraded":"2024-03-06T07:34:11.007435+0000","last_fullsized":"2024-03-06T07:34:11.007435+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:44.539838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301126+0000","last_change":"2024-03-06T07:34:09.303091+0000","last_active":"2024-03-06T07:34:10.301126+0000","last_peered":"2024-03-06T07:34:10.301126+0000","last_clean":"2024-03-06T07:34:10.301126+0000","last_became_active":"2024-03-06T07:34:09.302483+0000","last_became_peered":"2024-03-06T07:34:09.302483+0000","last_unstale":"2024-03-06T07:34:10.301126+0000","last_undegraded":"2024-03-06T07:34:10.301126+0000","last_fullsized":"2024-03-06T07:34:10.301126+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:54.692369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006900+0000","last_change":"2024-03-06T07:34:09.291954+0000","last_active":"2024-03-06T07:34:11.006900+0000","last_peered":"2024-03-06T07:34:11.006900+0000","last_clean":"2024-03-06T07:34:11.006900+0000","last_became_active":"2024-03-06T07:34:09.291609+0000","last_became_peered":"2024-03-06T07:34:09.291609+0000","last_unstale":"2024-03-06T07:34:11.006900+0000","last_undegraded":"2024-03-06T07:34:11.006900+0000","last_fullsized":"2024-03-06T07:34:11.006900+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:00.105663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005831+0000","last_change":"2024-03-06T07:34:09.304043+0000","last_active":"2024-03-06T07:34:11.005831+0000","last_peered":"2024-03-06T07:34:11.005831+0000","last_clean":"2024-03-06T07:34:11.005831+0000","last_became_active":"2024-03-06T07:34:09.303776+0000","last_became_peered":"2024-03-06T07:34:09.303776+0000","last_unstale":"2024-03-06T07:34:11.005831+0000","last_undegraded":"2024-03-06T07:34:11.005831+0000","last_fullsized":"2024-03-06T07:34:11.005831+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:00.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300530+0000","last_change":"2024-03-06T07:34:09.301020+0000","last_active":"2024-03-06T07:34:10.300530+0000","last_peered":"2024-03-06T07:34:10.300530+0000","last_clean":"2024-03-06T07:34:10.300530+0000","last_became_active":"2024-03-06T07:34:09.300700+0000","last_became_peered":"2024-03-06T07:34:09.300700+0000","last_unstale":"2024-03-06T07:34:10.300530+0000","last_undegraded":"2024-03-06T07:34:10.300530+0000","last_fullsized":"2024-03-06T07:34:10.300530+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:16.617196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305617+0000","last_change":"2024-03-06T07:34:09.295913+0000","last_active":"2024-03-06T07:34:10.305617+0000","last_peered":"2024-03-06T07:34:10.305617+0000","last_clean":"2024-03-06T07:34:10.305617+0000","last_became_active":"2024-03-06T07:34:09.295591+0000","last_became_peered":"2024-03-06T07:34:09.295591+0000","last_unstale":"2024-03-06T07:34:10.305617+0000","last_undegraded":"2024-03-06T07:34:10.305617+0000","last_fullsized":"2024-03-06T07:34:10.305617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:33.478183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290025+0000","last_change":"2024-03-06T07:34:09.302182+0000","last_active":"2024-03-06T07:34:10.290025+0000","last_peered":"2024-03-06T07:34:10.290025+0000","last_clean":"2024-03-06T07:34:10.290025+0000","last_became_active":"2024-03-06T07:34:09.300296+0000","last_became_peered":"2024-03-06T07:34:09.300296+0000","last_unstale":"2024-03-06T07:34:10.290025+0000","last_undegraded":"2024-03-06T07:34:10.290025+0000","last_fullsized":"2024-03-06T07:34:10.290025+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:07.616565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003074+0000","last_change":"2024-03-06T07:34:09.305078+0000","last_active":"2024-03-06T07:34:11.003074+0000","last_peered":"2024-03-06T07:34:11.003074+0000","last_clean":"2024-03-06T07:34:11.003074+0000","last_became_active":"2024-03-06T07:34:09.302647+0000","last_became_peered":"2024-03-06T07:34:09.302647+0000","last_unstale":"2024-03-06T07:34:11.003074+0000","last_undegraded":"2024-03-06T07:34:11.003074+0000","last_fullsized":"2024-03-06T07:34:11.003074+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:58.234520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305931+0000","last_change":"2024-03-06T07:34:09.304151+0000","last_active":"2024-03-06T07:34:10.305931+0000","last_peered":"2024-03-06T07:34:10.305931+0000","last_clean":"2024-03-06T07:34:10.305931+0000","last_became_active":"2024-03-06T07:34:09.303784+0000","last_became_peered":"2024-03-06T07:34:09.303784+0000","last_unstale":"2024-03-06T07:34:10.305931+0000","last_undegraded":"2024-03-06T07:34:10.305931+0000","last_fullsized":"2024-03-06T07:34:10.305931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:40.501563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301349+0000","last_change":"2024-03-06T07:34:08.276064+0000","last_active":"2024-03-06T07:34:10.301349+0000","last_peered":"2024-03-06T07:34:10.301349+0000","last_clean":"2024-03-06T07:34:10.301349+0000","last_became_active":"2024-03-06T07:34:08.275801+0000","last_became_peered":"2024-03-06T07:34:08.275801+0000","last_unstale":"2024-03-06T07:34:10.301349+0000","last_undegraded":"2024-03-06T07:34:10.301349+0000","last_fullsized":"2024-03-06T07:34:10.301349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:45.635970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.291312+0000","last_change":"2024-03-06T07:34:09.305128+0000","last_active":"2024-03-06T07:34:10.291312+0000","last_peered":"2024-03-06T07:34:10.291312+0000","last_clean":"2024-03-06T07:34:10.291312+0000","last_became_active":"2024-03-06T07:34:09.302858+0000","last_became_peered":"2024-03-06T07:34:09.302858+0000","last_unstale":"2024-03-06T07:34:10.291312+0000","last_undegraded":"2024-03-06T07:34:10.291312+0000","last_fullsized":"2024-03-06T07:34:10.291312+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:34.363051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006429+0000","last_change":"2024-03-06T07:34:08.277530+0000","last_active":"2024-03-06T07:34:11.006429+0000","last_peered":"2024-03-06T07:34:11.006429+0000","last_clean":"2024-03-06T07:34:11.006429+0000","last_became_active":"2024-03-06T07:34:08.277041+0000","last_became_peered":"2024-03-06T07:34:08.277041+0000","last_unstale":"2024-03-06T07:34:11.006429+0000","last_undegraded":"2024-03-06T07:34:11.006429+0000","last_fullsized":"2024-03-06T07:34:11.006429+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:49.452341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003330+0000","last_change":"2024-03-06T07:34:09.301943+0000","last_active":"2024-03-06T07:34:11.003330+0000","last_peered":"2024-03-06T07:34:11.003330+0000","last_clean":"2024-03-06T07:34:11.003330+0000","last_became_active":"2024-03-06T07:34:09.301130+0000","last_became_peered":"2024-03-06T07:34:09.301130+0000","last_unstale":"2024-03-06T07:34:11.003330+0000","last_undegraded":"2024-03-06T07:34:11.003330+0000","last_fullsized":"2024-03-06T07:34:11.003330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:49.376701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300827+0000","last_change":"2024-03-06T07:34:08.270479+0000","last_active":"2024-03-06T07:34:10.300827+0000","last_peered":"2024-03-06T07:34:10.300827+0000","last_clean":"2024-03-06T07:34:10.300827+0000","last_became_active":"2024-03-06T07:34:08.270199+0000","last_became_peered":"2024-03-06T07:34:08.270199+0000","last_unstale":"2024-03-06T07:34:10.300827+0000","last_undegraded":"2024-03-06T07:34:10.300827+0000","last_fullsized":"2024-03-06T07:34:10.300827+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:28.296128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003817+0000","last_change":"2024-03-06T07:34:09.307500+0000","last_active":"2024-03-06T07:34:11.003817+0000","last_peered":"2024-03-06T07:34:11.003817+0000","last_clean":"2024-03-06T07:34:11.003817+0000","last_became_active":"2024-03-06T07:34:09.307175+0000","last_became_peered":"2024-03-06T07:34:09.307175+0000","last_unstale":"2024-03-06T07:34:11.003817+0000","last_undegraded":"2024-03-06T07:34:11.003817+0000","last_fullsized":"2024-03-06T07:34:11.003817+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:45.020969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300720+0000","last_change":"2024-03-06T07:34:08.277107+0000","last_active":"2024-03-06T07:34:10.300720+0000","last_peered":"2024-03-06T07:34:10.300720+0000","last_clean":"2024-03-06T07:34:10.300720+0000","last_became_active":"2024-03-06T07:34:08.276380+0000","last_became_peered":"2024-03-06T07:34:08.276380+0000","last_unstale":"2024-03-06T07:34:10.300720+0000","last_undegraded":"2024-03-06T07:34:10.300720+0000","last_fullsized":"2024-03-06T07:34:10.300720+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.730852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005854+0000","last_change":"2024-03-06T07:34:09.298716+0000","last_active":"2024-03-06T07:34:11.005854+0000","last_peered":"2024-03-06T07:34:11.005854+0000","last_clean":"2024-03-06T07:34:11.005854+0000","last_became_active":"2024-03-06T07:34:09.297049+0000","last_became_peered":"2024-03-06T07:34:09.297049+0000","last_unstale":"2024-03-06T07:34:11.005854+0000","last_undegraded":"2024-03-06T07:34:11.005854+0000","last_fullsized":"2024-03-06T07:34:11.005854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:31.745690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003669+0000","last_change":"2024-03-06T07:34:08.271587+0000","last_active":"2024-03-06T07:34:11.003669+0000","last_peered":"2024-03-06T07:34:11.003669+0000","last_clean":"2024-03-06T07:34:11.003669+0000","last_became_active":"2024-03-06T07:34:08.271319+0000","last_became_peered":"2024-03-06T07:34:08.271319+0000","last_unstale":"2024-03-06T07:34:11.003669+0000","last_undegraded":"2024-03-06T07:34:11.003669+0000","last_fullsized":"2024-03-06T07:34:11.003669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:47.577770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006186+0000","last_change":"2024-03-06T07:34:09.296669+0000","last_active":"2024-03-06T07:34:11.006186+0000","last_peered":"2024-03-06T07:34:11.006186+0000","last_clean":"2024-03-06T07:34:11.006186+0000","last_became_active":"2024-03-06T07:34:09.296332+0000","last_became_peered":"2024-03-06T07:34:09.296332+0000","last_unstale":"2024-03-06T07:34:11.006186+0000","last_undegraded":"2024-03-06T07:34:11.006186+0000","last_fullsized":"2024-03-06T07:34:11.006186+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:10.392795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300231+0000","last_change":"2024-03-06T07:34:08.275076+0000","last_active":"2024-03-06T07:34:10.300231+0000","last_peered":"2024-03-06T07:34:10.300231+0000","last_clean":"2024-03-06T07:34:10.300231+0000","last_became_active":"2024-03-06T07:34:08.274808+0000","last_became_peered":"2024-03-06T07:34:08.274808+0000","last_unstale":"2024-03-06T07:34:10.300231+0000","last_undegraded":"2024-03-06T07:34:10.300231+0000","last_fullsized":"2024-03-06T07:34:10.300231+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289734+0000","last_change":"2024-03-06T07:34:09.283444+0000","last_active":"2024-03-06T07:34:10.289734+0000","last_peered":"2024-03-06T07:34:10.289734+0000","last_clean":"2024-03-06T07:34:10.289734+0000","last_became_active":"2024-03-06T07:34:09.281506+0000","last_became_peered":"2024-03-06T07:34:09.281506+0000","last_unstale":"2024-03-06T07:34:10.289734+0000","last_undegraded":"2024-03-06T07:34:10.289734+0000","last_fullsized":"2024-03-06T07:34:10.289734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:10.392795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004607+0000","last_change":"2024-03-06T07:34:08.276324+0000","last_active":"2024-03-06T07:34:11.004607+0000","last_peered":"2024-03-06T07:34:11.004607+0000","last_clean":"2024-03-06T07:34:11.004607+0000","last_became_active":"2024-03-06T07:34:08.275963+0000","last_became_peered":"2024-03-06T07:34:08.275963+0000","last_unstale":"2024-03-06T07:34:11.004607+0000","last_undegraded":"2024-03-06T07:34:11.004607+0000","last_fullsized":"2024-03-06T07:34:11.004607+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002938+0000","last_change":"2024-03-06T07:34:09.307806+0000","last_active":"2024-03-06T07:34:11.002938+0000","last_peered":"2024-03-06T07:34:11.002938+0000","last_clean":"2024-03-06T07:34:11.002938+0000","last_became_active":"2024-03-06T07:34:09.307027+0000","last_became_peered":"2024-03-06T07:34:09.307027+0000","last_unstale":"2024-03-06T07:34:11.002938+0000","last_undegraded":"2024-03-06T07:34:11.002938+0000","last_fullsized":"2024-03-06T07:34:11.002938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:25.216965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002961+0000","last_change":"2024-03-06T07:34:08.259184+0000","last_active":"2024-03-06T07:34:11.002961+0000","last_peered":"2024-03-06T07:34:11.002961+0000","last_clean":"2024-03-06T07:34:11.002961+0000","last_became_active":"2024-03-06T07:34:08.258879+0000","last_became_peered":"2024-03-06T07:34:08.258879+0000","last_unstale":"2024-03-06T07:34:11.002961+0000","last_undegraded":"2024-03-06T07:34:11.002961+0000","last_fullsized":"2024-03-06T07:34:11.002961+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:09.754281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301209+0000","last_change":"2024-03-06T07:34:09.302869+0000","last_active":"2024-03-06T07:34:10.301209+0000","last_peered":"2024-03-06T07:34:10.301209+0000","last_clean":"2024-03-06T07:34:10.301209+0000","last_became_active":"2024-03-06T07:34:09.301409+0000","last_became_peered":"2024-03-06T07:34:09.301409+0000","last_unstale":"2024-03-06T07:34:10.301209+0000","last_undegraded":"2024-03-06T07:34:10.301209+0000","last_fullsized":"2024-03-06T07:34:10.301209+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:23.807287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003932+0000","last_change":"2024-03-06T07:34:08.272502+0000","last_active":"2024-03-06T07:34:11.003932+0000","last_peered":"2024-03-06T07:34:11.003932+0000","last_clean":"2024-03-06T07:34:11.003932+0000","last_became_active":"2024-03-06T07:34:08.272249+0000","last_became_peered":"2024-03-06T07:34:08.272249+0000","last_unstale":"2024-03-06T07:34:11.003932+0000","last_undegraded":"2024-03-06T07:34:11.003932+0000","last_fullsized":"2024-03-06T07:34:11.003932+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:43.055234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004514+0000","last_change":"2024-03-06T07:34:09.307573+0000","last_active":"2024-03-06T07:34:11.004514+0000","last_peered":"2024-03-06T07:34:11.004514+0000","last_clean":"2024-03-06T07:34:11.004514+0000","last_became_active":"2024-03-06T07:34:09.307259+0000","last_became_peered":"2024-03-06T07:34:09.307259+0000","last_unstale":"2024-03-06T07:34:11.004514+0000","last_undegraded":"2024-03-06T07:34:11.004514+0000","last_fullsized":"2024-03-06T07:34:11.004514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:00.404213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290824+0000","last_change":"2024-03-06T07:34:08.277747+0000","last_active":"2024-03-06T07:34:10.290824+0000","last_peered":"2024-03-06T07:34:10.290824+0000","last_clean":"2024-03-06T07:34:10.290824+0000","last_became_active":"2024-03-06T07:34:08.274892+0000","last_became_peered":"2024-03-06T07:34:08.274892+0000","last_unstale":"2024-03-06T07:34:10.290824+0000","last_undegraded":"2024-03-06T07:34:10.290824+0000","last_fullsized":"2024-03-06T07:34:10.290824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.388532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006168+0000","last_change":"2024-03-06T07:34:09.303069+0000","last_active":"2024-03-06T07:34:11.006168+0000","last_peered":"2024-03-06T07:34:11.006168+0000","last_clean":"2024-03-06T07:34:11.006168+0000","last_became_active":"2024-03-06T07:34:09.302690+0000","last_became_peered":"2024-03-06T07:34:09.302690+0000","last_unstale":"2024-03-06T07:34:11.006168+0000","last_undegraded":"2024-03-06T07:34:11.006168+0000","last_fullsized":"2024-03-06T07:34:11.006168+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:29.311809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007057+0000","last_change":"2024-03-06T07:34:08.277986+0000","last_active":"2024-03-06T07:34:11.007057+0000","last_peered":"2024-03-06T07:34:11.007057+0000","last_clean":"2024-03-06T07:34:11.007057+0000","last_became_active":"2024-03-06T07:34:08.276369+0000","last_became_peered":"2024-03-06T07:34:08.276369+0000","last_unstale":"2024-03-06T07:34:11.007057+0000","last_undegraded":"2024-03-06T07:34:11.007057+0000","last_fullsized":"2024-03-06T07:34:11.007057+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:18.238978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006945+0000","last_change":"2024-03-06T07:34:09.303409+0000","last_active":"2024-03-06T07:34:11.006945+0000","last_peered":"2024-03-06T07:34:11.006945+0000","last_clean":"2024-03-06T07:34:11.006945+0000","last_became_active":"2024-03-06T07:34:09.302220+0000","last_became_peered":"2024-03-06T07:34:09.302220+0000","last_unstale":"2024-03-06T07:34:11.006945+0000","last_undegraded":"2024-03-06T07:34:11.006945+0000","last_fullsized":"2024-03-06T07:34:11.006945+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:07.499061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003401+0000","last_change":"2024-03-06T07:34:08.265538+0000","last_active":"2024-03-06T07:34:11.003401+0000","last_peered":"2024-03-06T07:34:11.003401+0000","last_clean":"2024-03-06T07:34:11.003401+0000","last_became_active":"2024-03-06T07:34:08.264783+0000","last_became_peered":"2024-03-06T07:34:08.264783+0000","last_unstale":"2024-03-06T07:34:11.003401+0000","last_undegraded":"2024-03-06T07:34:11.003401+0000","last_fullsized":"2024-03-06T07:34:11.003401+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:56.924294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.289949+0000","last_change":"2024-03-06T07:34:09.279232+0000","last_active":"2024-03-06T07:34:10.289949+0000","last_peered":"2024-03-06T07:34:10.289949+0000","last_clean":"2024-03-06T07:34:10.289949+0000","last_became_active":"2024-03-06T07:34:09.278568+0000","last_became_peered":"2024-03-06T07:34:09.278568+0000","last_unstale":"2024-03-06T07:34:10.289949+0000","last_undegraded":"2024-03-06T07:34:10.289949+0000","last_fullsized":"2024-03-06T07:34:10.289949+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:43.980518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305700+0000","last_change":"2024-03-06T07:34:08.273221+0000","last_active":"2024-03-06T07:34:10.305700+0000","last_peered":"2024-03-06T07:34:10.305700+0000","last_clean":"2024-03-06T07:34:10.305700+0000","last_became_active":"2024-03-06T07:34:08.272960+0000","last_became_peered":"2024-03-06T07:34:08.272960+0000","last_unstale":"2024-03-06T07:34:10.305700+0000","last_undegraded":"2024-03-06T07:34:10.305700+0000","last_fullsized":"2024-03-06T07:34:10.305700+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.133468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003387+0000","last_change":"2024-03-06T07:34:09.291512+0000","last_active":"2024-03-06T07:34:11.003387+0000","last_peered":"2024-03-06T07:34:11.003387+0000","last_clean":"2024-03-06T07:34:11.003387+0000","last_became_active":"2024-03-06T07:34:09.290557+0000","last_became_peered":"2024-03-06T07:34:09.290557+0000","last_unstale":"2024-03-06T07:34:11.003387+0000","last_undegraded":"2024-03-06T07:34:11.003387+0000","last_fullsized":"2024-03-06T07:34:11.003387+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:52.517967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306490+0000","last_change":"2024-03-06T07:34:08.269595+0000","last_active":"2024-03-06T07:34:10.306490+0000","last_peered":"2024-03-06T07:34:10.306490+0000","last_clean":"2024-03-06T07:34:10.306490+0000","last_became_active":"2024-03-06T07:34:08.269260+0000","last_became_peered":"2024-03-06T07:34:08.269260+0000","last_unstale":"2024-03-06T07:34:10.306490+0000","last_undegraded":"2024-03-06T07:34:10.306490+0000","last_fullsized":"2024-03-06T07:34:10.306490+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:32.400545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002962+0000","last_change":"2024-03-06T07:34:09.307736+0000","last_active":"2024-03-06T07:34:11.002962+0000","last_peered":"2024-03-06T07:34:11.002962+0000","last_clean":"2024-03-06T07:34:11.002962+0000","last_became_active":"2024-03-06T07:34:09.307136+0000","last_became_peered":"2024-03-06T07:34:09.307136+0000","last_unstale":"2024-03-06T07:34:11.002962+0000","last_undegraded":"2024-03-06T07:34:11.002962+0000","last_fullsized":"2024-03-06T07:34:11.002962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:17.481076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.002822+0000","last_change":"2024-03-06T07:34:08.271511+0000","last_active":"2024-03-06T07:34:11.002822+0000","last_peered":"2024-03-06T07:34:11.002822+0000","last_clean":"2024-03-06T07:34:11.002822+0000","last_became_active":"2024-03-06T07:34:08.271119+0000","last_became_peered":"2024-03-06T07:34:08.271119+0000","last_unstale":"2024-03-06T07:34:11.002822+0000","last_undegraded":"2024-03-06T07:34:11.002822+0000","last_fullsized":"2024-03-06T07:34:11.002822+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:30.730009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.304931+0000","last_change":"2024-03-06T07:34:09.304652+0000","last_active":"2024-03-06T07:34:10.304931+0000","last_peered":"2024-03-06T07:34:10.304931+0000","last_clean":"2024-03-06T07:34:10.304931+0000","last_became_active":"2024-03-06T07:34:09.303092+0000","last_became_peered":"2024-03-06T07:34:09.303092+0000","last_unstale":"2024-03-06T07:34:10.304931+0000","last_undegraded":"2024-03-06T07:34:10.304931+0000","last_fullsized":"2024-03-06T07:34:10.304931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:10.215635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003249+0000","last_change":"2024-03-06T07:34:08.269208+0000","last_active":"2024-03-06T07:34:11.003249+0000","last_peered":"2024-03-06T07:34:11.003249+0000","last_clean":"2024-03-06T07:34:11.003249+0000","last_became_active":"2024-03-06T07:34:08.268751+0000","last_became_peered":"2024-03-06T07:34:08.268751+0000","last_unstale":"2024-03-06T07:34:11.003249+0000","last_undegraded":"2024-03-06T07:34:11.003249+0000","last_fullsized":"2024-03-06T07:34:11.003249+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:09.199953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007597+0000","last_change":"2024-03-06T07:34:09.298981+0000","last_active":"2024-03-06T07:34:11.007597+0000","last_peered":"2024-03-06T07:34:11.007597+0000","last_clean":"2024-03-06T07:34:11.007597+0000","last_became_active":"2024-03-06T07:34:09.296016+0000","last_became_peered":"2024-03-06T07:34:09.296016+0000","last_unstale":"2024-03-06T07:34:11.007597+0000","last_undegraded":"2024-03-06T07:34:11.007597+0000","last_fullsized":"2024-03-06T07:34:11.007597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:27.293917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003760+0000","last_change":"2024-03-06T07:34:08.270938+0000","last_active":"2024-03-06T07:34:11.003760+0000","last_peered":"2024-03-06T07:34:11.003760+0000","last_clean":"2024-03-06T07:34:11.003760+0000","last_became_active":"2024-03-06T07:34:08.270180+0000","last_became_peered":"2024-03-06T07:34:08.270180+0000","last_unstale":"2024-03-06T07:34:11.003760+0000","last_undegraded":"2024-03-06T07:34:11.003760+0000","last_fullsized":"2024-03-06T07:34:11.003760+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005724+0000","last_change":"2024-03-06T07:34:09.300024+0000","last_active":"2024-03-06T07:34:11.005724+0000","last_peered":"2024-03-06T07:34:11.005724+0000","last_clean":"2024-03-06T07:34:11.005724+0000","last_became_active":"2024-03-06T07:34:09.299739+0000","last_became_peered":"2024-03-06T07:34:09.299739+0000","last_unstale":"2024-03-06T07:34:11.005724+0000","last_undegraded":"2024-03-06T07:34:11.005724+0000","last_fullsized":"2024-03-06T07:34:11.005724+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:57.939975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006456+0000","last_change":"2024-03-06T07:34:08.272524+0000","last_active":"2024-03-06T07:34:11.006456+0000","last_peered":"2024-03-06T07:34:11.006456+0000","last_clean":"2024-03-06T07:34:11.006456+0000","last_became_active":"2024-03-06T07:34:08.272078+0000","last_became_peered":"2024-03-06T07:34:08.272078+0000","last_unstale":"2024-03-06T07:34:11.006456+0000","last_undegraded":"2024-03-06T07:34:11.006456+0000","last_fullsized":"2024-03-06T07:34:11.006456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.298132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.007077+0000","last_change":"2024-03-06T07:34:09.297770+0000","last_active":"2024-03-06T07:34:11.007077+0000","last_peered":"2024-03-06T07:34:11.007077+0000","last_clean":"2024-03-06T07:34:11.007077+0000","last_became_active":"2024-03-06T07:34:09.297457+0000","last_became_peered":"2024-03-06T07:34:09.297457+0000","last_unstale":"2024-03-06T07:34:11.007077+0000","last_undegraded":"2024-03-06T07:34:11.007077+0000","last_fullsized":"2024-03-06T07:34:11.007077+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:40.530510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300777+0000","last_change":"2024-03-06T07:34:08.275513+0000","last_active":"2024-03-06T07:34:10.300777+0000","last_peered":"2024-03-06T07:34:10.300777+0000","last_clean":"2024-03-06T07:34:10.300777+0000","last_became_active":"2024-03-06T07:34:08.275177+0000","last_became_peered":"2024-03-06T07:34:08.275177+0000","last_unstale":"2024-03-06T07:34:10.300777+0000","last_undegraded":"2024-03-06T07:34:10.300777+0000","last_fullsized":"2024-03-06T07:34:10.300777+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:49.452341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306140+0000","last_change":"2024-03-06T07:34:09.293082+0000","last_active":"2024-03-06T07:34:10.306140+0000","last_peered":"2024-03-06T07:34:10.306140+0000","last_clean":"2024-03-06T07:34:10.306140+0000","last_became_active":"2024-03-06T07:34:09.292719+0000","last_became_peered":"2024-03-06T07:34:09.292719+0000","last_unstale":"2024-03-06T07:34:10.306140+0000","last_undegraded":"2024-03-06T07:34:10.306140+0000","last_fullsized":"2024-03-06T07:34:10.306140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:49.376701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300683+0000","last_change":"2024-03-06T07:34:08.272959+0000","last_active":"2024-03-06T07:34:10.300683+0000","last_peered":"2024-03-06T07:34:10.300683+0000","last_clean":"2024-03-06T07:34:10.300683+0000","last_became_active":"2024-03-06T07:34:08.271962+0000","last_became_peered":"2024-03-06T07:34:08.271962+0000","last_unstale":"2024-03-06T07:34:10.300683+0000","last_undegraded":"2024-03-06T07:34:10.300683+0000","last_fullsized":"2024-03-06T07:34:10.300683+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:39.459734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305569+0000","last_change":"2024-03-06T07:34:09.305597+0000","last_active":"2024-03-06T07:34:10.305569+0000","last_peered":"2024-03-06T07:34:10.305569+0000","last_clean":"2024-03-06T07:34:10.305569+0000","last_became_active":"2024-03-06T07:34:09.304907+0000","last_became_peered":"2024-03-06T07:34:09.304907+0000","last_unstale":"2024-03-06T07:34:10.305569+0000","last_undegraded":"2024-03-06T07:34:10.305569+0000","last_fullsized":"2024-03-06T07:34:10.305569+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:57.642232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005917+0000","last_change":"2024-03-06T07:34:08.275800+0000","last_active":"2024-03-06T07:34:11.005917+0000","last_peered":"2024-03-06T07:34:11.005917+0000","last_clean":"2024-03-06T07:34:11.005917+0000","last_became_active":"2024-03-06T07:34:08.275476+0000","last_became_peered":"2024-03-06T07:34:08.275476+0000","last_unstale":"2024-03-06T07:34:11.005917+0000","last_undegraded":"2024-03-06T07:34:11.005917+0000","last_fullsized":"2024-03-06T07:34:11.005917+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:54.100822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305023+0000","last_change":"2024-03-06T07:34:09.292697+0000","last_active":"2024-03-06T07:34:10.305023+0000","last_peered":"2024-03-06T07:34:10.305023+0000","last_clean":"2024-03-06T07:34:10.305023+0000","last_became_active":"2024-03-06T07:34:09.292397+0000","last_became_peered":"2024-03-06T07:34:09.292397+0000","last_unstale":"2024-03-06T07:34:10.305023+0000","last_undegraded":"2024-03-06T07:34:10.305023+0000","last_fullsized":"2024-03-06T07:34:10.305023+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:40.677711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005508+0000","last_change":"2024-03-06T07:34:08.274027+0000","last_active":"2024-03-06T07:34:11.005508+0000","last_peered":"2024-03-06T07:34:11.005508+0000","last_clean":"2024-03-06T07:34:11.005508+0000","last_became_active":"2024-03-06T07:34:08.273695+0000","last_became_peered":"2024-03-06T07:34:08.273695+0000","last_unstale":"2024-03-06T07:34:11.005508+0000","last_undegraded":"2024-03-06T07:34:11.005508+0000","last_fullsized":"2024-03-06T07:34:11.005508+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:55.267247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004182+0000","last_change":"2024-03-06T07:34:09.298873+0000","last_active":"2024-03-06T07:34:11.004182+0000","last_peered":"2024-03-06T07:34:11.004182+0000","last_clean":"2024-03-06T07:34:11.004182+0000","last_became_active":"2024-03-06T07:34:09.298515+0000","last_became_peered":"2024-03-06T07:34:09.298515+0000","last_unstale":"2024-03-06T07:34:11.004182+0000","last_undegraded":"2024-03-06T07:34:11.004182+0000","last_fullsized":"2024-03-06T07:34:11.004182+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:19.254659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300438+0000","last_change":"2024-03-06T07:34:08.274029+0000","last_active":"2024-03-06T07:34:10.300438+0000","last_peered":"2024-03-06T07:34:10.300438+0000","last_clean":"2024-03-06T07:34:10.300438+0000","last_became_active":"2024-03-06T07:34:08.273759+0000","last_became_peered":"2024-03-06T07:34:08.273759+0000","last_unstale":"2024-03-06T07:34:10.300438+0000","last_undegraded":"2024-03-06T07:34:10.300438+0000","last_fullsized":"2024-03-06T07:34:10.300438+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:44.005288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301343+0000","last_change":"2024-03-06T07:34:09.298874+0000","last_active":"2024-03-06T07:34:10.301343+0000","last_peered":"2024-03-06T07:34:10.301343+0000","last_clean":"2024-03-06T07:34:10.301343+0000","last_became_active":"2024-03-06T07:34:09.298602+0000","last_became_peered":"2024-03-06T07:34:09.298602+0000","last_unstale":"2024-03-06T07:34:10.301343+0000","last_undegraded":"2024-03-06T07:34:10.301343+0000","last_fullsized":"2024-03-06T07:34:10.301343+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:07.544494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.005772+0000","last_change":"2024-03-06T07:34:08.279992+0000","last_active":"2024-03-06T07:34:11.005772+0000","last_peered":"2024-03-06T07:34:11.005772+0000","last_clean":"2024-03-06T07:34:11.005772+0000","last_became_active":"2024-03-06T07:34:08.279058+0000","last_became_peered":"2024-03-06T07:34:08.279058+0000","last_unstale":"2024-03-06T07:34:11.005772+0000","last_undegraded":"2024-03-06T07:34:11.005772+0000","last_fullsized":"2024-03-06T07:34:11.005772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:06.914393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004106+0000","last_change":"2024-03-06T07:34:09.290526+0000","last_active":"2024-03-06T07:34:11.004106+0000","last_peered":"2024-03-06T07:34:11.004106+0000","last_clean":"2024-03-06T07:34:11.004106+0000","last_became_active":"2024-03-06T07:34:09.290063+0000","last_became_peered":"2024-03-06T07:34:09.290063+0000","last_unstale":"2024-03-06T07:34:11.004106+0000","last_undegraded":"2024-03-06T07:34:11.004106+0000","last_fullsized":"2024-03-06T07:34:11.004106+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:31.093849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.301495+0000","last_change":"2024-03-06T07:34:08.271690+0000","last_active":"2024-03-06T07:34:10.301495+0000","last_peered":"2024-03-06T07:34:10.301495+0000","last_clean":"2024-03-06T07:34:10.301495+0000","last_became_active":"2024-03-06T07:34:08.271425+0000","last_became_peered":"2024-03-06T07:34:08.271425+0000","last_unstale":"2024-03-06T07:34:10.301495+0000","last_undegraded":"2024-03-06T07:34:10.301495+0000","last_fullsized":"2024-03-06T07:34:10.301495+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:43.524157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003563+0000","last_change":"2024-03-06T07:34:09.297391+0000","last_active":"2024-03-06T07:34:11.003563+0000","last_peered":"2024-03-06T07:34:11.003563+0000","last_clean":"2024-03-06T07:34:11.003563+0000","last_became_active":"2024-03-06T07:34:09.297107+0000","last_became_peered":"2024-03-06T07:34:09.297107+0000","last_unstale":"2024-03-06T07:34:11.003563+0000","last_undegraded":"2024-03-06T07:34:11.003563+0000","last_fullsized":"2024-03-06T07:34:11.003563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:32.178946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300833+0000","last_change":"2024-03-06T07:34:08.276819+0000","last_active":"2024-03-06T07:34:10.300833+0000","last_peered":"2024-03-06T07:34:10.300833+0000","last_clean":"2024-03-06T07:34:10.300833+0000","last_became_active":"2024-03-06T07:34:08.276500+0000","last_became_peered":"2024-03-06T07:34:08.276500+0000","last_unstale":"2024-03-06T07:34:10.300833+0000","last_undegraded":"2024-03-06T07:34:10.300833+0000","last_fullsized":"2024-03-06T07:34:10.300833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:03.411533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006246+0000","last_change":"2024-03-06T07:34:09.300002+0000","last_active":"2024-03-06T07:34:11.006246+0000","last_peered":"2024-03-06T07:34:11.006246+0000","last_clean":"2024-03-06T07:34:11.006246+0000","last_became_active":"2024-03-06T07:34:09.299553+0000","last_became_peered":"2024-03-06T07:34:09.299553+0000","last_unstale":"2024-03-06T07:34:11.006246+0000","last_undegraded":"2024-03-06T07:34:11.006246+0000","last_fullsized":"2024-03-06T07:34:11.006246+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:04.427214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006365+0000","last_change":"2024-03-06T07:34:08.277478+0000","last_active":"2024-03-06T07:34:11.006365+0000","last_peered":"2024-03-06T07:34:11.006365+0000","last_clean":"2024-03-06T07:34:11.006365+0000","last_became_active":"2024-03-06T07:34:08.277186+0000","last_became_peered":"2024-03-06T07:34:08.277186+0000","last_unstale":"2024-03-06T07:34:11.006365+0000","last_undegraded":"2024-03-06T07:34:11.006365+0000","last_fullsized":"2024-03-06T07:34:11.006365+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:09.498839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.306597+0000","last_change":"2024-03-06T07:34:09.306423+0000","last_active":"2024-03-06T07:34:10.306597+0000","last_peered":"2024-03-06T07:34:10.306597+0000","last_clean":"2024-03-06T07:34:10.306597+0000","last_became_active":"2024-03-06T07:34:09.302507+0000","last_became_peered":"2024-03-06T07:34:09.302507+0000","last_unstale":"2024-03-06T07:34:10.306597+0000","last_undegraded":"2024-03-06T07:34:10.306597+0000","last_fullsized":"2024-03-06T07:34:10.306597+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:52.517967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.300901+0000","last_change":"2024-03-06T07:34:08.272451+0000","last_active":"2024-03-06T07:34:10.300901+0000","last_peered":"2024-03-06T07:34:10.300901+0000","last_clean":"2024-03-06T07:34:10.300901+0000","last_became_active":"2024-03-06T07:34:08.272175+0000","last_became_peered":"2024-03-06T07:34:08.272175+0000","last_unstale":"2024-03-06T07:34:10.300901+0000","last_undegraded":"2024-03-06T07:34:10.300901+0000","last_fullsized":"2024-03-06T07:34:10.300901+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.388532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.003157+0000","last_change":"2024-03-06T07:34:09.301236+0000","last_active":"2024-03-06T07:34:11.003157+0000","last_peered":"2024-03-06T07:34:11.003157+0000","last_clean":"2024-03-06T07:34:11.003157+0000","last_became_active":"2024-03-06T07:34:09.300337+0000","last_became_peered":"2024-03-06T07:34:09.300337+0000","last_unstale":"2024-03-06T07:34:11.003157+0000","last_undegraded":"2024-03-06T07:34:11.003157+0000","last_fullsized":"2024-03-06T07:34:11.003157+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:12.746533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004465+0000","last_change":"2024-03-06T07:34:08.274395+0000","last_active":"2024-03-06T07:34:11.004465+0000","last_peered":"2024-03-06T07:34:11.004465+0000","last_clean":"2024-03-06T07:34:11.004465+0000","last_became_active":"2024-03-06T07:34:08.274068+0000","last_became_peered":"2024-03-06T07:34:08.274068+0000","last_unstale":"2024-03-06T07:34:11.004465+0000","last_undegraded":"2024-03-06T07:34:11.004465+0000","last_fullsized":"2024-03-06T07:34:11.004465+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:41.468854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305022+0000","last_change":"2024-03-06T07:34:09.303120+0000","last_active":"2024-03-06T07:34:10.305022+0000","last_peered":"2024-03-06T07:34:10.305022+0000","last_clean":"2024-03-06T07:34:10.305022+0000","last_became_active":"2024-03-06T07:34:09.299011+0000","last_became_peered":"2024-03-06T07:34:09.299011+0000","last_unstale":"2024-03-06T07:34:10.305022+0000","last_undegraded":"2024-03-06T07:34:10.305022+0000","last_fullsized":"2024-03-06T07:34:10.305022+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:44.070915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.290053+0000","last_change":"2024-03-06T07:34:08.257721+0000","last_active":"2024-03-06T07:34:10.290053+0000","last_peered":"2024-03-06T07:34:10.290053+0000","last_clean":"2024-03-06T07:34:10.290053+0000","last_became_active":"2024-03-06T07:34:08.257322+0000","last_became_peered":"2024-03-06T07:34:08.257322+0000","last_unstale":"2024-03-06T07:34:10.290053+0000","last_undegraded":"2024-03-06T07:34:10.290053+0000","last_fullsized":"2024-03-06T07:34:10.290053+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:39.459734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004391+0000","last_change":"2024-03-06T07:34:09.305318+0000","last_active":"2024-03-06T07:34:11.004391+0000","last_peered":"2024-03-06T07:34:11.004391+0000","last_clean":"2024-03-06T07:34:11.004391+0000","last_became_active":"2024-03-06T07:34:09.304946+0000","last_became_peered":"2024-03-06T07:34:09.304946+0000","last_unstale":"2024-03-06T07:34:11.004391+0000","last_undegraded":"2024-03-06T07:34:11.004391+0000","last_fullsized":"2024-03-06T07:34:11.004391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:45.020969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305330+0000","last_change":"2024-03-06T07:34:08.279635+0000","last_active":"2024-03-06T07:34:10.305330+0000","last_peered":"2024-03-06T07:34:10.305330+0000","last_clean":"2024-03-06T07:34:10.305330+0000","last_became_active":"2024-03-06T07:34:08.279250+0000","last_became_peered":"2024-03-06T07:34:08.279250+0000","last_unstale":"2024-03-06T07:34:10.305330+0000","last_undegraded":"2024-03-06T07:34:10.305330+0000","last_fullsized":"2024-03-06T07:34:10.305330+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:17.184769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:10.305935+0000","last_change":"2024-03-06T07:34:09.300151+0000","last_active":"2024-03-06T07:34:10.305935+0000","last_peered":"2024-03-06T07:34:10.305935+0000","last_clean":"2024-03-06T07:34:10.305935+0000","last_became_active":"2024-03-06T07:34:09.299873+0000","last_became_peered":"2024-03-06T07:34:09.299873+0000","last_unstale":"2024-03-06T07:34:10.305935+0000","last_undegraded":"2024-03-06T07:34:10.305935+0000","last_fullsized":"2024-03-06T07:34:10.305935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:08.227127+0000","last_clean_scrub_stamp":"2024-03-06T07:34:08.227127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:58.234520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.006188+0000","last_change":"2024-03-06T07:34:08.276832+0000","last_active":"2024-03-06T07:34:11.006188+0000","last_peered":"2024-03-06T07:34:11.006188+0000","last_clean":"2024-03-06T07:34:11.006188+0000","last_became_active":"2024-03-06T07:34:08.276488+0000","last_became_peered":"2024-03-06T07:34:08.276488+0000","last_unstale":"2024-03-06T07:34:11.006188+0000","last_undegraded":"2024-03-06T07:34:11.006188+0000","last_fullsized":"2024-03-06T07:34:11.006188+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:07.211446+0000","last_clean_scrub_stamp":"2024-03-06T07:34:07.211446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.730852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":310,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:34:11.004681+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:34:11.004681+0000","last_peered":"2024-03-06T07:34:11.004681+0000","last_clean":"2024-03-06T07:34:11.004681+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:34:11.004681+0000","last_undegraded":"2024-03-06T07:34:11.004681+0000","last_fullsized":"2024-03-06T07:34:11.004681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1777664,"data_stored":1092681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":502853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1179648,"data_stored":502271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36512,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335328,"statfs":{"total":96636764160,"available":96599375872,"internally_reserved":0,"allocated":1196032,"data_stored":503937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335336,"statfs":{"total":96636764160,"available":96599384064,"internally_reserved":0,"allocated":1191936,"data_stored":503835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1179648,"data_stored":502235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1175552,"data_stored":502201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1773568,"data_stored":1093151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:19.334 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:34:19.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:34:19.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.600+0000 7f28fd400640 1 Processor -- start 2024-03-06T07:34:19.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.601+0000 7f28fd400640 1 -- start start 2024-03-06T07:34:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.602+0000 7f28fd400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f28f8165560 0x7f28f806c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.602+0000 7f28fd400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28f8072860 0x7f28f80b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.602+0000 7f28fd400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f806d060 con 0x7f28f80b3c40 2024-03-06T07:34:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.602+0000 7f28fd400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f8072cf0 con 0x7f28f8072860 2024-03-06T07:34:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.602+0000 7f28fd400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f80cdce0 con 0x7f28f8165560 2024-03-06T07:34:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28f8072860 0x7f28f80b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28f8072860 0x7f28f80b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40256/0 (socket says 172.21.15.32:40256) 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 -- 172.21.15.32:0/1659252675 learned_addr learned my addr 172.21.15.32:0/1659252675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7e00640 1 --2- 172.21.15.32:0/1659252675 >> v2:172.21.15.39:3300/0 conn(0x7f28f8165560 0x7f28f806c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 -- 172.21.15.32:0/1659252675 >> v2:172.21.15.39:3300/0 conn(0x7f28f8165560 msgr2=0x7f28f806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 --2- 172.21.15.32:0/1659252675 >> v2:172.21.15.39:3300/0 conn(0x7f28f8165560 0x7f28f806c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 -- 172.21.15.32:0/1659252675 >> v1:172.21.15.39:6789/0 conn(0x7f28f80b3c40 legacy=0x7f28f8165c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.603+0000 7f28f7400640 1 -- 172.21.15.32:0/1659252675 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f806d340 con 0x7f28f8072860 2024-03-06T07:34:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.604+0000 7f28f7400640 1 --2- 172.21.15.32:0/1659252675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8072860 0x7f28f80b3700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f28f8072f10 tx=0x7f28e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a8a986a58483ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28f6a00640 1 -- 172.21.15.32:0/1659252675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e002e070 con 0x7f28f8072860 2024-03-06T07:34:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28f6a00640 1 -- 172.21.15.32:0/1659252675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0037070 con 0x7f28f8072860 2024-03-06T07:34:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28f6a00640 1 -- 172.21.15.32:0/1659252675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0031a00 con 0x7f28f8072860 2024-03-06T07:34:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 -- 172.21.15.32:0/1659252675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8072860 msgr2=0x7f28f80b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 --2- 172.21.15.32:0/1659252675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8072860 0x7f28f80b3700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f28f8072f10 tx=0x7f28e002b880 comp rx=0 tx=0).stop 2024-03-06T07:34:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 -- 172.21.15.32:0/1659252675 shutdown_connections 2024-03-06T07:34:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 --2- 172.21.15.32:0/1659252675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8072860 0x7f28f80b3700 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 --2- 172.21.15.32:0/1659252675 >> v2:172.21.15.39:3300/0 conn(0x7f28f8165560 0x7f28f806c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 -- 172.21.15.32:0/1659252675 >> 172.21.15.32:0/1659252675 conn(0x7f28f806b190 msgr2=0x7f28f806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 -- 172.21.15.32:0/1659252675 shutdown_connections 2024-03-06T07:34:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.605+0000 7f28fd400640 1 -- 172.21.15.32:0/1659252675 wait complete. 2024-03-06T07:34:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.607+0000 7f28fd400640 1 Processor -- start 2024-03-06T07:34:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.607+0000 7f28fd400640 1 -- start start 2024-03-06T07:34:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.608+0000 7f28fd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28f8072860 0x7f28f8160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.608+0000 7f28fd400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28f80b3c40 0x7f28f8154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.608+0000 7f28fd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 0x7f28f8154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.608+0000 7f28fd400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f806d900 con 0x7f28f8165560 2024-03-06T07:34:19.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.608+0000 7f28fd400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f80b4070 con 0x7f28f8072860 2024-03-06T07:34:19.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.608+0000 7f28fd400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f806dbe0 con 0x7f28f80b3c40 2024-03-06T07:34:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 0x7f28f8154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 0x7f28f8154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40262/0 (socket says 172.21.15.32:40262) 2024-03-06T07:34:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 -- 172.21.15.32:0/3928885304 learned_addr learned my addr 172.21.15.32:0/3928885304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28f7e00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28f8072860 0x7f28f8160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28f7400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28f80b3c40 0x7f28f8154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 -- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28f80b3c40 msgr2=0x7f28f8154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28f80b3c40 0x7f28f8154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 -- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28f8072860 msgr2=0x7f28f8160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28f8072860 0x7f28f8160cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.609+0000 7f28fca00640 1 -- 172.21.15.32:0/3928885304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8155650 con 0x7f28f8165560 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.610+0000 7f28fca00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 0x7f28f8154fb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c7b0 tx=0x7f28f000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.610+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0017070 con 0x7f28f8165560 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.610+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0003da0 con 0x7f28f8165560 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.610+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f8155930 con 0x7f28f8165560 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.610+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f00052b0 con 0x7f28f8165560 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.610+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f8155ef0 con 0x7f28f8165560 2024-03-06T07:34:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.611+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f804f330 con 0x7f28f8165560 2024-03-06T07:34:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.612+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28f001a070 con 0x7f28f8165560 2024-03-06T07:34:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.613+0000 7f28efe00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f28d406b4d0 0x7f28d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.613+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28f00897b0 con 0x7f28f8165560 2024-03-06T07:34:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.613+0000 7f28f7e00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f28d406b4d0 0x7f28d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.614+0000 7f28f7e00640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f28d406b4d0 0x7f28d406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f28e8006fd0 tx=0x7f28e8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.617+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28f00586c0 con 0x7f28f8165560 2024-03-06T07:34:19.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:19.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:19.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.894+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f28f8071440 con 0x7f28f8165560 2024-03-06T07:34:19.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.898+0000 7f28efe00640 1 -- 172.21.15.32:0/3928885304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f28f0057fc0 con 0x7f28f8165560 2024-03-06T07:34:19.900 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:34:19.900 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:34:19.900 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:34:19.900 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:34:19.900 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:34:19.901 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f28d406b4d0 msgr2=0x7f28d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f28d406b4d0 0x7f28d406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f28e8006fd0 tx=0x7f28e8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 msgr2=0x7f28f8154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 0x7f28f8154fb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c7b0 tx=0x7f28f000cc70 comp rx=0 tx=0).stop 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 shutdown_connections 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f28d406b4d0 0x7f28d406d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28f8165560 0x7f28f8154fb0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28f80b3c40 0x7f28f8154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 --2- 172.21.15.32:0/3928885304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28f8072860 0x7f28f8160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.900+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 >> 172.21.15.32:0/3928885304 conn(0x7f28f806b190 msgr2=0x7f28f80cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.901+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 shutdown_connections 2024-03-06T07:34:19.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:19.901+0000 7f28fd400640 1 -- 172.21.15.32:0/3928885304 wait complete. 2024-03-06T07:34:19.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.175+0000 7f73f4e00640 1 Processor -- start 2024-03-06T07:34:20.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.175+0000 7f73f4e00640 1 -- start start 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.177+0000 7f73f4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 0x7f73f015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.177+0000 7f73f4e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f015cc40 con 0x7f73f00af1a0 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.177+0000 7f73f4e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f015cf20 con 0x7f73f00cdc20 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.177+0000 7f73f4e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f015d200 con 0x7f73f00af580 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.177+0000 7f73ede00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f73f00af1a0 0x7f73f00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47034/0 (socket says 172.21.15.32:47034) 2024-03-06T07:34:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.177+0000 7f73ede00640 1 -- 172.21.15.32:0/1762831953 learned_addr learned my addr 172.21.15.32:0/1762831953 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:20.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73e5400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 0x7f73f015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897216074 0 0) 0x7f73f015cc40 con 0x7f73f00af1a0 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d4003520 con 0x7f73f00af1a0 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3421080938 0 0) 0x7f73d4003520 con 0x7f73f00af1a0 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 msgr2=0x7f73f015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73ed400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 0x7f73f015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.178+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 >> v1:172.21.15.39:6789/0 conn(0x7f73f00cdc20 legacy=0x7f73f00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f015d4e0 con 0x7f73f00af1a0 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f73e00019f0 con 0x7f73f00af1a0 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 >> v1:172.21.15.32:6789/0 conn(0x7f73f00af1a0 legacy=0x7f73f00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 0x7f73d4003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d4003fd0 0x7f73d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4004d40 0x7f73d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f015d200 con 0x7f73d4003fd0 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d400bbb0 con 0x7f73d4004d40 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d400bdb0 con 0x7f73f00af580 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73e5400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d4003fd0 0x7f73d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73e5400640 1 -- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 msgr2=0x7f73d4003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73e5400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f00af580 0x7f73d4003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73e5400640 1 -- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4004d40 msgr2=0x7f73d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73e5400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4004d40 0x7f73d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.179+0000 7f73e5400640 1 -- 172.21.15.32:0/1762831953 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d4003520 con 0x7f73d4003fd0 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.180+0000 7f73e5400640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d4003fd0 0x7f73d4004800 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009ac0 tx=0x7f73d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c0891eb1d502dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8039070 con 0x7f73d4003fd0 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d802c710 con 0x7f73d4003fd0 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73ed400640 1 -- 172.21.15.32:0/1762831953 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8009690 con 0x7f73d4003fd0 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73f4e00640 1 -- 172.21.15.32:0/1762831953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d4003fd0 msgr2=0x7f73d4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73f4e00640 1 --2- 172.21.15.32:0/1762831953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d4003fd0 0x7f73d4004800 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009ac0 tx=0x7f73d4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73ee800640 1 -- 172.21.15.32:0/1762831953 reap_dead start 2024-03-06T07:34:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73f4e00640 1 -- 172.21.15.32:0/1762831953 shutdown_connections 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73f4e00640 1 -- 172.21.15.32:0/1762831953 >> 172.21.15.32:0/1762831953 conn(0x7f73f006afb0 msgr2=0x7f73f006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73f4e00640 1 -- 172.21.15.32:0/1762831953 shutdown_connections 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.181+0000 7f73f4e00640 1 -- 172.21.15.32:0/1762831953 wait complete. 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.183+0000 7f73f4e00640 1 Processor -- start 2024-03-06T07:34:20.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.183+0000 7f73f4e00640 1 -- start start 2024-03-06T07:34:20.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.184+0000 7f73f4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4003fd0 0x7f73f015e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.184+0000 7f73f4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f01548c0 0x7f73f0150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.184+0000 7f73f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 0x7f73f0150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.184+0000 7f73f4e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f015c600 con 0x7f73f0154d40 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.184+0000 7f73f4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f00b36f0 con 0x7f73d4003fd0 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.184+0000 7f73f4e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f015c780 con 0x7f73f01548c0 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 0x7f73f0150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 0x7f73f0150a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40280/0 (socket says 172.21.15.32:40280) 2024-03-06T07:34:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 -- 172.21.15.32:0/667983437 learned_addr learned my addr 172.21.15.32:0/667983437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ede00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4003fd0 0x7f73f015e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 -- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f01548c0 msgr2=0x7f73f0150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73e5400640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f01548c0 0x7f73f0150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f01548c0 0x7f73f0150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 -- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4003fd0 msgr2=0x7f73f015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4003fd0 0x7f73f015e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 -- 172.21.15.32:0/667983437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0155340 con 0x7f73f0154d40 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ede00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4003fd0 0x7f73f015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.185+0000 7f73ee800640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 0x7f73f0150a40 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f73f009e100 tx=0x7f73e800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.186+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8007340 con 0x7f73f0154d40 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.186+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e80074e0 con 0x7f73f0154d40 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.186+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8004260 con 0x7f73f0154d40 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.186+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f0155620 con 0x7f73f0154d40 2024-03-06T07:34:20.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.186+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f0157d70 con 0x7f73f0154d40 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.187+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f0069510 con 0x7f73f0154d40 2024-03-06T07:34:20.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.188+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73e8011070 con 0x7f73f0154d40 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.189+0000 7f73e6a00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f73cc06b4d0 0x7f73cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.189+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73e80893d0 con 0x7f73f0154d40 2024-03-06T07:34:20.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.189+0000 7f73ede00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f73cc06b4d0 0x7f73cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.190+0000 7f73ede00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f73cc06b4d0 0x7f73cc06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f73e00012e0 tx=0x7f73e0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.193+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73e80581b0 con 0x7f73f0154d40 2024-03-06T07:34:20.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.433+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73f004f3a0 con 0x7f73f0154d40 2024-03-06T07:34:20.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.434+0000 7f73e6a00640 1 -- 172.21.15.32:0/667983437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f73e8057ab0 con 0x7f73f0154d40 2024-03-06T07:34:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:20.436 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5887,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2823770992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2823770992},{"type":"v1","addr":"172.21.15.39:6834","nonce":2823770992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/415659526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":415659526},{"type":"v1","addr":"172.21.15.32:6836","nonce":415659526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2586752485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":2586752485},{"type":"v1","addr":"172.21.15.32:6837","nonce":2586752485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:11.808218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/556051904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":556051904},{"type":"v1","addr":"172.21.15.39:6835","nonce":556051904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7122,"qdb_cluster":[7122]},"id":7}]} 2024-03-06T07:34:20.436 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 61 2024-03-06T07:34:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.436+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f73cc06b4d0 msgr2=0x7f73cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.436+0000 7f73f4e00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f73cc06b4d0 0x7f73cc06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f73e00012e0 tx=0x7f73e0009040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.436+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 msgr2=0x7f73f0150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.436+0000 7f73f4e00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 0x7f73f0150a40 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f73f009e100 tx=0x7f73e800c820 comp rx=0 tx=0).stop 2024-03-06T07:34:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 shutdown_connections 2024-03-06T07:34:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f73cc06b4d0 0x7f73cc06d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73f0154d40 0x7f73f0150a40 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f73f01548c0 0x7f73f0150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 --2- 172.21.15.32:0/667983437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d4003fd0 0x7f73f015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 >> 172.21.15.32:0/667983437 conn(0x7f73f006afb0 msgr2=0x7f73f00cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 shutdown_connections 2024-03-06T07:34:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:20.437+0000 7f73f4e00640 1 -- 172.21.15.32:0/667983437 wait complete. 2024-03-06T07:34:20.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:08.726069+0000', 'modified': '2024-03-06T07:34:11.808218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7122}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7122': {'gid': 7122, 'name': 'b', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/556051904', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6833', 'nonce': 556051904}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 556051904}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7122, 'qdb_cluster': [7122]} 2024-03-06T07:34:20.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:20.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:34:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.093+0000 7f07a3200640 1 Processor -- start 2024-03-06T07:34:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.094+0000 7f07a3200640 1 -- start start 2024-03-06T07:34:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.095+0000 7f07a3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07a41552b0 0x7f07a415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.095+0000 7f07a3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f07a40cd060 0x7f07a40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.095+0000 7f07a3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a415ed10 con 0x7f07a415e810 2024-03-06T07:34:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.095+0000 7f07a3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a415eff0 con 0x7f07a41552b0 2024-03-06T07:34:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.095+0000 7f07a3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a415f2d0 con 0x7f07a40cd060 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07a41552b0 0x7f07a415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07a41552b0 0x7f07a415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40290/0 (socket says 172.21.15.32:40290) 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 -- 172.21.15.32:0/2803646699 learned_addr learned my addr 172.21.15.32:0/2803646699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a2800640 1 --2- 172.21.15.32:0/2803646699 >> v2:172.21.15.39:3300/0 conn(0x7f07a40cd060 0x7f07a40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 -- 172.21.15.32:0/2803646699 >> v2:172.21.15.39:3300/0 conn(0x7f07a40cd060 msgr2=0x7f07a40b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 --2- 172.21.15.32:0/2803646699 >> v2:172.21.15.39:3300/0 conn(0x7f07a40cd060 0x7f07a40b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 -- 172.21.15.32:0/2803646699 >> v1:172.21.15.39:6789/0 conn(0x7f07a415e810 legacy=0x7f07a40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a1e00640 1 -- 172.21.15.32:0/2803646699 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a415f5b0 con 0x7f07a41552b0 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.096+0000 7f07a2800640 1 --2- 172.21.15.32:0/2803646699 >> v2:172.21.15.39:3300/0 conn(0x7f07a40cd060 0x7f07a40b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.097+0000 7f07a1e00640 1 --2- 172.21.15.32:0/2803646699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a41552b0 0x7f07a415b1d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f07a40cf100 tx=0x7f079402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a402d3385ae96b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.098+0000 7f07a1400640 1 -- 172.21.15.32:0/2803646699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079402e070 con 0x7f07a41552b0 2024-03-06T07:34:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.098+0000 7f07a1400640 1 -- 172.21.15.32:0/2803646699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794037070 con 0x7f07a41552b0 2024-03-06T07:34:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.098+0000 7f07a1400640 1 -- 172.21.15.32:0/2803646699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794031a00 con 0x7f07a41552b0 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.098+0000 7f07a3200640 1 -- 172.21.15.32:0/2803646699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a41552b0 msgr2=0x7f07a415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.098+0000 7f07a3200640 1 --2- 172.21.15.32:0/2803646699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a41552b0 0x7f07a415b1d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f07a40cf100 tx=0x7f079402b880 comp rx=0 tx=0).stop 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.099+0000 7f07a3200640 1 -- 172.21.15.32:0/2803646699 shutdown_connections 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.099+0000 7f07a3200640 1 --2- 172.21.15.32:0/2803646699 >> v2:172.21.15.39:3300/0 conn(0x7f07a40cd060 0x7f07a40b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.099+0000 7f07a3200640 1 --2- 172.21.15.32:0/2803646699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a41552b0 0x7f07a415b1d0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.099+0000 7f07a3200640 1 -- 172.21.15.32:0/2803646699 >> 172.21.15.32:0/2803646699 conn(0x7f07a406b3d0 msgr2=0x7f07a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.099+0000 7f07a3200640 1 -- 172.21.15.32:0/2803646699 shutdown_connections 2024-03-06T07:34:21.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.099+0000 7f07a3200640 1 -- 172.21.15.32:0/2803646699 wait complete. 2024-03-06T07:34:21.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.101+0000 7f07a3200640 1 Processor -- start 2024-03-06T07:34:21.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.101+0000 7f07a3200640 1 -- start start 2024-03-06T07:34:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 0x7f07a40b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07a41552b0 0x7f07a4144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07a415e810 0x7f07a414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a406dd30 con 0x7f07a40cd060 2024-03-06T07:34:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b3760 con 0x7f07a41552b0 2024-03-06T07:34:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a406deb0 con 0x7f07a415e810 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 0x7f07a40b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 0x7f07a40b1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40304/0 (socket says 172.21.15.32:40304) 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.102+0000 7f07a1e00640 1 -- 172.21.15.32:0/2259411307 learned_addr learned my addr 172.21.15.32:0/2259411307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a2800640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07a415e810 0x7f07a414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f0799400640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07a41552b0 0x7f07a4144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a1e00640 1 -- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07a415e810 msgr2=0x7f07a414bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a1e00640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07a415e810 0x7f07a414bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a1e00640 1 -- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07a41552b0 msgr2=0x7f07a4144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a1e00640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07a41552b0 0x7f07a4144500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a1e00640 1 -- 172.21.15.32:0/2259411307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b2310 con 0x7f07a40cd060 2024-03-06T07:34:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f07a1e00640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 0x7f07a40b1dd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f0794000c00 tx=0x7f07940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f079aa00640 1 -- 172.21.15.32:0/2259411307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794032880 con 0x7f07a40cd060 2024-03-06T07:34:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.103+0000 7f079aa00640 1 -- 172.21.15.32:0/2259411307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794032a20 con 0x7f07a40cd060 2024-03-06T07:34:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.104+0000 7f079aa00640 1 -- 172.21.15.32:0/2259411307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794032e10 con 0x7f07a40cd060 2024-03-06T07:34:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.104+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4144a40 con 0x7f07a40cd060 2024-03-06T07:34:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.104+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a40affa0 con 0x7f07a40cd060 2024-03-06T07:34:21.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.106+0000 7f079aa00640 1 -- 172.21.15.32:0/2259411307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07940453f0 con 0x7f07a40cd060 2024-03-06T07:34:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.107+0000 7f079aa00640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f078006b500 0x7f078006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.107+0000 7f079aa00640 1 -- 172.21.15.32:0/2259411307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07940ac410 con 0x7f07a40cd060 2024-03-06T07:34:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.107+0000 7f0799400640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f078006b500 0x7f078006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.108+0000 7f0799400640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f078006b500 0x7f078006d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f07a41502c0 tx=0x7f078c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.108+0000 7f07aba33740 1 Processor -- start 2024-03-06T07:34:21.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.108+0000 7f07aba33740 1 -- start start 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.109+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c4186e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.109+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c4189e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.109+0000 7f07aba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 0x558c4189ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.109+0000 7f07aba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c417d8600 con 0x558c4186ee70 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.109+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c41781930 con 0x558c4189bd70 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.109+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c417d46a0 con 0x558c4189b990 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 0x558c4189ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 0x558c4189ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40316/0 (socket says 172.21.15.32:40316) 2024-03-06T07:34:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 -- 172.21.15.32:0/3667278735 learned_addr learned my addr 172.21.15.32:0/3667278735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a1e00640 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c4186e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f0799400640 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c4189e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 -- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 msgr2=0x558c4186e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c4186e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 -- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 msgr2=0x558c4189e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c4189e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 -- 172.21.15.32:0/3667278735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c414ed2d0 con 0x558c4186ee70 2024-03-06T07:34:21.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a1e00640 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c4186e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.110+0000 7f07a2800640 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 0x558c4189ec60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f079c00d430 tx=0x7f079c00d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827755654733af90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.111+0000 7f0792a00640 1 -- 172.21.15.32:0/3667278735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c01a070 con 0x558c4186ee70 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.111+0000 7f0792a00640 1 -- 172.21.15.32:0/3667278735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c00b6e0 con 0x558c4186ee70 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.111+0000 7f0792a00640 1 -- 172.21.15.32:0/3667278735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c00bab0 con 0x558c4186ee70 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.111+0000 7f07aba33740 1 -- 172.21.15.32:0/3667278735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 msgr2=0x558c4189ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.111+0000 7f07aba33740 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 0x558c4189ec60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f079c00d430 tx=0x7f079c00d8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 -- 172.21.15.32:0/3667278735 shutdown_connections 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4186ee70 0x558c4189ec60 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c4189e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 --2- 172.21.15.32:0/3667278735 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c4186e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 -- 172.21.15.32:0/3667278735 >> 172.21.15.32:0/3667278735 conn(0x558c41870ba0 msgr2=0x558c418b9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 -- 172.21.15.32:0/3667278735 shutdown_connections 2024-03-06T07:34:21.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 -- 172.21.15.32:0/3667278735 wait complete. 2024-03-06T07:34:21.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.112+0000 7f07aba33740 1 -- start start 2024-03-06T07:34:21.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.113+0000 7f07aba33740 1 Processor -- start 2024-03-06T07:34:21.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c418a0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c41873140 con 0x558c4189b990 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c41794d90 con 0x558c4189bd70 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c41872530 con 0x558c418a0990 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07aba33740 10 client.0 _ll_register_callbacks cb 0x558c417d0350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40326/0 (socket says 172.21.15.32:40326) 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 -- 172.21.15.32:0/3167186310 learned_addr learned my addr 172.21.15.32:0/3167186310 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 -- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 msgr2=0x558c418b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a2800640 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418b2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 -- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 msgr2=0x558c418a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c418a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.114+0000 7f07a1e00640 1 -- 172.21.15.32:0/3167186310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c415b76b0 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f07a1e00640 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418ccab0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f07940aa840 tx=0x7f0794009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1271478b4c95dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f0787400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f0787400640 1 -- 172.21.15.32:0/3167186310 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794083320 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f0787400640 1 -- 172.21.15.32:0/3167186310 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f079408ebd0 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f079c00b6e0 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f0787400640 1 -- 172.21.15.32:0/3167186310 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f079408edd0 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f07aba33740 10 client.7278 fetch_fsmap learned FSMap version 61 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x558c418b0950 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.115+0000 7f0787400640 1 -- 172.21.15.32:0/3167186310 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c00b6e0 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f0787400640 1 -- 172.21.15.32:0/3167186310 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0794082e90 con 0x558c4189b990 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f07aba33740 10 client.7278 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f07aba33740 10 client.7278 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f07aba33740 20 client.7278 populate_metadata read hostname 'smithi032' 2024-03-06T07:34:21.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f07aba33740 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418b17f0 0x558c418b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f07aba33740 4 client.7278 mds_command: new command op to 7122 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:21.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 --> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c417e5b40 con 0x558c418b17f0 2024-03-06T07:34:21.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.116+0000 7f0799400640 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418b17f0 0x558c418b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.117+0000 7f0799400640 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418b17f0 0x558c418b1bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=828774b30be1a62d server_cookie=14c655ecff4d7478 in_seq=0 out_seq=0 2024-03-06T07:34:21.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.117+0000 7f0787400640 10 client.7278 ms_handle_connect on v2:172.21.15.39:6833/556051904 2024-03-06T07:34:21.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.119+0000 7f0787400640 1 -- 172.21.15.32:0/3167186310 <== mds.0 v2:172.21.15.39:6833/556051904 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558c417e5b40 con 0x558c418b17f0 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.119+0000 7f0787400640 10 client.7278 handle_command_reply: tid=0 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 client.7278 shutdown 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 msgr2=0x558c418ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418ccab0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f07940aa840 tx=0x7f0794009550 comp rx=0 tx=0).stop 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 shutdown_connections 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418b17f0 0x558c418b1bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c418a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 --2- 172.21.15.32:0/3167186310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418ccab0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 >> 172.21.15.32:0/3167186310 conn(0x558c4186ee70 msgr2=0x558c4186e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f0787400640 0 client.7278 ms_handle_reset on v2:172.21.15.39:6833/556051904 2024-03-06T07:34:21.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 shutdown_connections 2024-03-06T07:34:21.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 1 -- 172.21.15.32:0/3167186310 wait complete. 2024-03-06T07:34:21.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.120+0000 7f07aba33740 20 client.7278 trim_cache size 0 max 16384 2024-03-06T07:34:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.138+0000 7f07aba33740 1 Processor -- start 2024-03-06T07:34:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 -- start start 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418b9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c418bce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418cb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4149120 con 0x558c4189b990 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a414fb70 con 0x558c4189bd70 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.139+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4148b50 con 0x558c418a0990 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418b9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418b9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40330/0 (socket says 172.21.15.32:40330) 2024-03-06T07:34:21.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 -- 172.21.15.32:0/1997183706 learned_addr learned my addr 172.21.15.32:0/1997183706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 -- 172.21.15.32:0/1997183706 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 msgr2=0x558c418cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 -- 172.21.15.32:0/1997183706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 msgr2=0x558c418bce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c418bce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 -- 172.21.15.32:0/1997183706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c417ec1a0 con 0x558c4189b990 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.140+0000 7f07a1e00640 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418b9e20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f079407a9c0 tx=0x7f0794050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bcc4ef0157e418 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f0792000640 1 -- 172.21.15.32:0/1997183706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794087030 con 0x558c4189b990 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f0792000640 1 -- 172.21.15.32:0/1997183706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794089070 con 0x558c4189b990 2024-03-06T07:34:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f0792000640 1 -- 172.21.15.32:0/1997183706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794083320 con 0x558c4189b990 2024-03-06T07:34:21.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 -- 172.21.15.32:0/1997183706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 msgr2=0x558c418b9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418b9e20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f079407a9c0 tx=0x7f0794050000 comp rx=0 tx=0).stop 2024-03-06T07:34:21.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 -- 172.21.15.32:0/1997183706 shutdown_connections 2024-03-06T07:34:21.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c418a0990 0x558c418cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c4189bd70 0x558c418bce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 --2- 172.21.15.32:0/1997183706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189b990 0x558c418b9e20 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.141+0000 7f07aba33740 1 -- 172.21.15.32:0/1997183706 >> 172.21.15.32:0/1997183706 conn(0x558c4186ee70 msgr2=0x558c418b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.142+0000 7f07aba33740 1 -- 172.21.15.32:0/1997183706 shutdown_connections 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.142+0000 7f07aba33740 1 -- 172.21.15.32:0/1997183706 wait complete. 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.142+0000 7f07aba33740 1 -- start start 2024-03-06T07:34:21.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.142+0000 7f07aba33740 1 Processor -- start 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.143+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c418ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07aba33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 0x558c418ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07aba33740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c418a0990 0x558c418ba970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07aba33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c41873140 con 0x558c4189bd70 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c41794d90 con 0x558c418a0990 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07aba33740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c4188ea00 con 0x558c4189b990 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07aba33740 10 client.0 _ll_register_callbacks cb 0x7f07940810b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 0x558c418ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 0x558c418ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40344/0 (socket says 172.21.15.32:40344) 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 -- 172.21.15.32:0/1834708285 learned_addr learned my addr 172.21.15.32:0/1834708285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07a1e00640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c418ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 -- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 msgr2=0x558c418ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c418ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 -- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c418a0990 msgr2=0x558c418ba970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f07a2800640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c418a0990 0x558c418ba970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c418a0990 0x558c418ba970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.144+0000 7f0799400640 1 -- 172.21.15.32:0/1834708285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c41899f00 con 0x558c4189bd70 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0799400640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 0x558c418ca6e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f07a4072380 tx=0x7f078c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945d4a812019f004 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0787400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0787400640 1 -- 172.21.15.32:0/1834708285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c013400 con 0x558c4189bd70 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0787400640 1 -- 172.21.15.32:0/1834708285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f078c0135a0 con 0x558c4189bd70 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0794089070 con 0x558c4189bd70 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0787400640 1 -- 172.21.15.32:0/1834708285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f078c031050 con 0x558c4189bd70 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f07aba33740 10 client.7284 fetch_fsmap learned FSMap version 61 2024-03-06T07:34:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x558c418bb7c0 con 0x558c4189bd70 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0787400640 1 -- 172.21.15.32:0/1834708285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794089070 con 0x558c4189bd70 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f0787400640 1 -- 172.21.15.32:0/1834708285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f078c034020 con 0x558c4189bd70 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f07aba33740 10 client.7284 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f07aba33740 10 client.7284 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.145+0000 7f07aba33740 20 client.7284 populate_metadata read hostname 'smithi032' 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.146+0000 7f07aba33740 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418bb300 0x558c418af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.146+0000 7f07aba33740 4 client.7284 mds_command: new command op to 7122 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.146+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 --> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558c417dfe90 con 0x558c418bb300 2024-03-06T07:34:21.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.146+0000 7f07a1e00640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418bb300 0x558c418af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.147+0000 7f07a1e00640 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418bb300 0x558c418af0b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f567e9f41cc54a64 server_cookie=ea537858f2ba1739 in_seq=0 out_seq=0 2024-03-06T07:34:21.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.147+0000 7f0787400640 10 client.7284 ms_handle_connect on v2:172.21.15.39:6833/556051904 2024-03-06T07:34:21.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f0787400640 1 -- 172.21.15.32:0/1834708285 <== mds.0 v2:172.21.15.39:6833/556051904 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x558c417dfe90 con 0x558c418bb300 2024-03-06T07:34:21.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f0787400640 10 client.7284 handle_command_reply: tid=0 2024-03-06T07:34:21.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 client.7284 shutdown 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 msgr2=0x558c418ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 0x558c418ca6e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f07a4072380 tx=0x7f078c00ced0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 shutdown_connections 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:6833/556051904,v1:172.21.15.39:6835/556051904] conn(0x558c418bb300 0x558c418af0b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558c418a0990 0x558c418ba970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c4189bd70 0x558c418ca6e0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 --2- 172.21.15.32:0/1834708285 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x558c4189b990 0x558c418ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f0787400640 0 client.7284 ms_handle_reset on v2:172.21.15.39:6833/556051904 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 >> 172.21.15.32:0/1834708285 conn(0x558c4186ee70 msgr2=0x558c418b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 shutdown_connections 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.148+0000 7f07aba33740 1 -- 172.21.15.32:0/1834708285 wait complete. 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07aba33740 20 client.7284 trim_cache size 0 max 16384 2024-03-06T07:34:21.153 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7122, 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 11.871427442, 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 61, 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 95, 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 95, 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 17.225611728000001 2024-03-06T07:34:21.154 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:34:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f078006b500 msgr2=0x7f078006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f078006b500 0x7f078006d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f07a41502c0 tx=0x7f078c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 msgr2=0x7f07a40b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 0x7f07a40b1dd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f0794000c00 tx=0x7f07940046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 shutdown_connections 2024-03-06T07:34:21.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f078006b500 0x7f078006d9c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f07a415e810 0x7f07a414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07a41552b0 0x7f07a4144500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 --2- 172.21.15.32:0/2259411307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a40cd060 0x7f07a40b1dd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 >> 172.21.15.32:0/2259411307 conn(0x7f07a406b3d0 msgr2=0x7f07a4070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.149+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 shutdown_connections 2024-03-06T07:34:21.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.150+0000 7f07a3200640 1 -- 172.21.15.32:0/2259411307 wait complete. 2024-03-06T07:34:21.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:34:21.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.378+0000 7effb1200640 1 Processor -- start 2024-03-06T07:34:21.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.378+0000 7effb1200640 1 -- start start 2024-03-06T07:34:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effb1200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7effac070f10 0x7effac0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effb1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac0cce90 0x7effac15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effb1200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac0714d0 con 0x7effac145300 2024-03-06T07:34:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effb1200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effac0cd300 con 0x7effac070f10 2024-03-06T07:34:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effb1200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac0717b0 con 0x7effac0cce90 2024-03-06T07:34:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effaaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7effac070f10 0x7effac0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effab400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7effac145300 0x7effac0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47038/0 (socket says 172.21.15.32:47038) 2024-03-06T07:34:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effab400640 1 -- 172.21.15.32:0/812354332 learned_addr learned my addr 172.21.15.32:0/812354332 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.380+0000 7effabe00640 1 --2- 172.21.15.32:0/812354332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac0cce90 0x7effac15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.381+0000 7effaaa00640 1 -- 172.21.15.32:0/812354332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac0cce90 msgr2=0x7effac15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.381+0000 7effaaa00640 1 --2- 172.21.15.32:0/812354332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac0cce90 0x7effac15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.381+0000 7effaaa00640 1 -- 172.21.15.32:0/812354332 >> v1:172.21.15.32:6789/0 conn(0x7effac145300 legacy=0x7effac0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.381+0000 7effaaa00640 1 -- 172.21.15.32:0/812354332 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac071a90 con 0x7effac070f10 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.381+0000 7effaa000640 1 -- 172.21.15.32:0/812354332 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728225352 0 0) 0x7effac0714d0 con 0x7effac145300 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.382+0000 7effaaa00640 1 --2- 172.21.15.32:0/812354332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac070f10 0x7effac0cc950 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7eff940097a0 tx=0x7eff9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9ba1ccfecc0d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.382+0000 7effaa000640 1 -- 172.21.15.32:0/812354332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff940315a0 con 0x7effac070f10 2024-03-06T07:34:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.382+0000 7effaa000640 1 -- 172.21.15.32:0/812354332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff94037070 con 0x7effac070f10 2024-03-06T07:34:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.382+0000 7effaa000640 1 -- 172.21.15.32:0/812354332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94031960 con 0x7effac070f10 2024-03-06T07:34:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 -- 172.21.15.32:0/812354332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac070f10 msgr2=0x7effac0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 --2- 172.21.15.32:0/812354332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac070f10 0x7effac0cc950 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7eff940097a0 tx=0x7eff9402b840 comp rx=0 tx=0).stop 2024-03-06T07:34:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 -- 172.21.15.32:0/812354332 shutdown_connections 2024-03-06T07:34:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 --2- 172.21.15.32:0/812354332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac070f10 0x7effac0cc950 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 --2- 172.21.15.32:0/812354332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac0cce90 0x7effac15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 -- 172.21.15.32:0/812354332 >> 172.21.15.32:0/812354332 conn(0x7effac06b3d0 msgr2=0x7effac06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 -- 172.21.15.32:0/812354332 shutdown_connections 2024-03-06T07:34:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.383+0000 7effb1200640 1 -- 172.21.15.32:0/812354332 wait complete. 2024-03-06T07:34:21.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.385+0000 7effb1200640 1 Processor -- start 2024-03-06T07:34:21.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.385+0000 7effb1200640 1 -- start start 2024-03-06T07:34:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effb1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac070f10 0x7effac1412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effb1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 0x7effac0cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effb1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effac145300 0x7effac14bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effb1200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac15cab0 con 0x7effac0cce90 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effb1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac1456e0 con 0x7effac145300 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effb1200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac15cc30 con 0x7effac070f10 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effaaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 0x7effac0cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effaaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 0x7effac0cbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40360/0 (socket says 172.21.15.32:40360) 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.386+0000 7effaaa00640 1 -- 172.21.15.32:0/2028345135 learned_addr learned my addr 172.21.15.32:0/2028345135 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effab400640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac070f10 0x7effac1412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effabe00640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effac145300 0x7effac14bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effaaa00640 1 -- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac070f10 msgr2=0x7effac1412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effaaa00640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac070f10 0x7effac1412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effaaa00640 1 -- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effac145300 msgr2=0x7effac14bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effaaa00640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effac145300 0x7effac14bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effaaa00640 1 -- 172.21.15.32:0/2028345135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac141970 con 0x7effac0cce90 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effabe00640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effac145300 0x7effac14bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effab400640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac070f10 0x7effac1412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.387+0000 7effaaa00640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 0x7effac0cbd30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7eff94034040 tx=0x7eff940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.388+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94032840 con 0x7effac0cce90 2024-03-06T07:34:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.388+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff94032a00 con 0x7effac0cce90 2024-03-06T07:34:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.388+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac141c50 con 0x7effac0cce90 2024-03-06T07:34:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.388+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94032df0 con 0x7effac0cce90 2024-03-06T07:34:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.388+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac142210 con 0x7effac0cce90 2024-03-06T07:34:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.389+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac04f330 con 0x7effac0cce90 2024-03-06T07:34:21.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.390+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff9404e430 con 0x7effac0cce90 2024-03-06T07:34:21.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.391+0000 7effa3400640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff8806b500 0x7eff8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.391+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff9402f030 con 0x7effac0cce90 2024-03-06T07:34:21.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.391+0000 7effab400640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff8806b500 0x7eff8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.392+0000 7effab400640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff8806b500 0x7eff8806d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7eff9c000f60 tx=0x7eff9c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.398+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff94040070 con 0x7effac0cce90 2024-03-06T07:34:21.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.640+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7effac0af1f0 con 0x7effac0cce90 2024-03-06T07:34:21.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.641+0000 7effa3400640 1 -- 172.21.15.32:0/2028345135 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) v1 ==== 53+0+73 (secure 0 0 0) 0x7eff9407afe0 con 0x7effac0cce90 2024-03-06T07:34:21.644 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:34:21.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff8806b500 msgr2=0x7eff8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff8806b500 0x7eff8806d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7eff9c000f60 tx=0x7eff9c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:21.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 msgr2=0x7effac0cbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 0x7effac0cbd30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7eff94034040 tx=0x7eff940046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 shutdown_connections 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff8806b500 0x7eff8806d9c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effac145300 0x7effac14bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effac0cce90 0x7effac0cbd30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 --2- 172.21.15.32:0/2028345135 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effac070f10 0x7effac1412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 >> 172.21.15.32:0/2028345135 conn(0x7effac06b3d0 msgr2=0x7effac0af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 shutdown_connections 2024-03-06T07:34:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:21.644+0000 7effb1200640 1 -- 172.21.15.32:0/2028345135 wait complete. 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:34:21.665 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:21.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:21.665 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42076sec preferred_lft 42076sec 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:34:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:34:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:21.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:21.721 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:34:21.811 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:34:21.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:21.816 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:34:21.873 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:34:21.874 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:34:21.874 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:34:21.874 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:34:21.874 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:34:21.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:21.874 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:34:21.874 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:34:21.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:34:21.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:34:21.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:34:21.875 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:34:22.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.070 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:34:22.123 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:22.123 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:22.124 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:22.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:34:22.224 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:22.225 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:22.281 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:22.281 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:22.281 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:22.335 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:34:22.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:22.483 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:22.483 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:22.521 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:22.521 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:22.521 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:34:22.521 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:34:22.521 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:34:22.527 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:34:22.527 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:34:22.527 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:34:22.527 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:34:22.527 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:34:22.532 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:34:22.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:22.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.633 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout:client5441 2024-03-06T07:34:22.715 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:34:22.716 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:22.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:34:22.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.955 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.956 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:34:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:34:22.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:34:22.959 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:34:22.960 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:34:22.961 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:34:22.962 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:34:22.963 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:34:22.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:34:22.965 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_files = 6 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.966 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:34:22.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.968 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.969 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:34:22.971 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:34:22.972 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:34:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:34:22.974 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:34:22.975 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:34:22.976 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:34:22.977 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:34:22.977 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:34:22.977 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:34:22.977 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_files = 6 2024-03-06T07:34:23.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:34:23.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:34:23.199 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:34:23.199 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:34:23.199 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:34:23.199 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:34:23.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:34:23.201 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:34:23.202 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:34:23.203 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:34:23.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:34:23.205 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:34:23.206 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:34:23.207 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:34:23.208 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:34:23.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_files = 6 2024-03-06T07:34:23.210 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:34:23.211 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:34:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:34:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:34:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:34:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:34:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:34:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:34:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:34:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:34:23.221 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_files = 6 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:34:23.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.487+0000 7f0138a00640 1 Processor -- start 2024-03-06T07:34:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.487+0000 7f0138a00640 1 -- start start 2024-03-06T07:34:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f0138a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f01341411d0 0x7f01340b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f0138a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01340b5b20 0x7f013415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f0138a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f013415f150 0x7f013406e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f0138a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013415f940 con 0x7f01340b5b20 2024-03-06T07:34:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f0138a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013415b850 con 0x7f01341411d0 2024-03-06T07:34:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f0138a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013415bb30 con 0x7f013415f150 2024-03-06T07:34:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f012be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01340b5b20 0x7f013415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f012be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01340b5b20 0x7f013415ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40362/0 (socket says 172.21.15.32:40362) 2024-03-06T07:34:23.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.489+0000 7f012be00640 1 -- 172.21.15.32:0/3313839286 learned_addr learned my addr 172.21.15.32:0/3313839286 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f0131e00640 1 --2- 172.21.15.32:0/3313839286 >> v2:172.21.15.39:3300/0 conn(0x7f01341411d0 0x7f01340b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f0132800640 1 --2- 172.21.15.32:0/3313839286 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f013415f150 0x7f013406e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f012be00640 1 -- 172.21.15.32:0/3313839286 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f013415f150 msgr2=0x7f013406e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f012be00640 1 --2- 172.21.15.32:0/3313839286 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f013415f150 0x7f013406e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f012be00640 1 -- 172.21.15.32:0/3313839286 >> v2:172.21.15.39:3300/0 conn(0x7f01341411d0 msgr2=0x7f01340b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f012be00640 1 --2- 172.21.15.32:0/3313839286 >> v2:172.21.15.39:3300/0 conn(0x7f01341411d0 0x7f01340b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f012be00640 1 -- 172.21.15.32:0/3313839286 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01341415b0 con 0x7f01340b5b20 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.490+0000 7f0131e00640 1 --2- 172.21.15.32:0/3313839286 >> v2:172.21.15.39:3300/0 conn(0x7f01341411d0 0x7f01340b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:23.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.491+0000 7f012be00640 1 --2- 172.21.15.32:0/3313839286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013415ec10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f011c009870 tx=0x7f011c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15f07d9867fe836 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.491+0000 7f0131400640 1 -- 172.21.15.32:0/3313839286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0317c0 con 0x7f01340b5b20 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.491+0000 7f0131400640 1 -- 172.21.15.32:0/3313839286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c037070 con 0x7f01340b5b20 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.491+0000 7f0131400640 1 -- 172.21.15.32:0/3313839286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c031b80 con 0x7f01340b5b20 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 -- 172.21.15.32:0/3313839286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 msgr2=0x7f013415ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 --2- 172.21.15.32:0/3313839286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013415ec10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f011c009870 tx=0x7f011c02b860 comp rx=0 tx=0).stop 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 -- 172.21.15.32:0/3313839286 shutdown_connections 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 --2- 172.21.15.32:0/3313839286 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f013415f150 0x7f013406e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 --2- 172.21.15.32:0/3313839286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013415ec10 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 --2- 172.21.15.32:0/3313839286 >> v2:172.21.15.39:3300/0 conn(0x7f01341411d0 0x7f01340b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 -- 172.21.15.32:0/3313839286 >> 172.21.15.32:0/3313839286 conn(0x7f01340b33b0 msgr2=0x7f01340b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 -- 172.21.15.32:0/3313839286 shutdown_connections 2024-03-06T07:34:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.492+0000 7f0138a00640 1 -- 172.21.15.32:0/3313839286 wait complete. 2024-03-06T07:34:23.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.494+0000 7f0138a00640 1 Processor -- start 2024-03-06T07:34:23.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.494+0000 7f0138a00640 1 -- start start 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0138a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0138a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f01341411d0 0x7f013413bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0138a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f013415f150 0x7f0134071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0138a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013415c330 con 0x7f01340b5b20 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0138a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013415f5e0 con 0x7f013415f150 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0138a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013415c4b0 con 0x7f01341411d0 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.495+0000 7f0131e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013406f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40376/0 (socket says 172.21.15.32:40376) 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 -- 172.21.15.32:0/68362199 learned_addr learned my addr 172.21.15.32:0/68362199 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f012be00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f01341411d0 0x7f013413bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0132800640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f013415f150 0x7f0134071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 -- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f01341411d0 msgr2=0x7f013413bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f01341411d0 0x7f013413bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 -- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f013415f150 msgr2=0x7f0134071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f013415f150 0x7f0134071dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 -- 172.21.15.32:0/68362199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013406e000 con 0x7f01340b5b20 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.496+0000 7f0131e00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013406f0a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f012400c820 tx=0x7f012400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.497+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124015070 con 0x7f01340b5b20 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.497+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124007a30 con 0x7f01340b5b20 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.497+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124007e20 con 0x7f01340b5b20 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.497+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013406e2e0 con 0x7f01340b5b20 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.497+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013406e8a0 con 0x7f01340b5b20 2024-03-06T07:34:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.498+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0134073fb0 con 0x7f01340b5b20 2024-03-06T07:34:23.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.499+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0124018070 con 0x7f01340b5b20 2024-03-06T07:34:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.500+0000 7f012a000640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f011006b4d0 0x7f011006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.500+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f012408a160 con 0x7f01340b5b20 2024-03-06T07:34:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.500+0000 7f012be00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f011006b4d0 0x7f011006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.501+0000 7f012be00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f011006b4d0 0x7f011006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f011c036000 tx=0x7f011c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.505+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0124058f50 con 0x7f01340b5b20 2024-03-06T07:34:23.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.712+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f01340704d0 con 0x7f01340b5b20 2024-03-06T07:34:23.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.713+0000 7f012a000640 1 -- 172.21.15.32:0/68362199 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0124058850 con 0x7f01340b5b20 2024-03-06T07:34:23.716 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f011006b4d0 msgr2=0x7f011006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f011006b4d0 0x7f011006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f011c036000 tx=0x7f011c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 msgr2=0x7f013406f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013406f0a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f012400c820 tx=0x7f012400cce0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 shutdown_connections 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f011006b4d0 0x7f011006d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f013415f150 0x7f0134071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f01341411d0 0x7f013413bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 --2- 172.21.15.32:0/68362199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01340b5b20 0x7f013406f0a0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 >> 172.21.15.32:0/68362199 conn(0x7f01340b33b0 msgr2=0x7f013415cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.716+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 shutdown_connections 2024-03-06T07:34:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:23.717+0000 7f0138a00640 1 -- 172.21.15.32:0/68362199 wait complete. 2024-03-06T07:34:23.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:34:23.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:34:23.908 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:34:23.908 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:34:23.908 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:34:23.908 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:34:23.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:34:23.910 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:34:23.911 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:34:23.912 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:34:23.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:34:23.914 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:34:23.915 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:34:23.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:34:23.917 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:34:23.918 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_files = 6 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:34:23.919 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:34:23.937 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:34:23.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:34:23.939 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:34:23.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:34:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:34:23.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:34:23.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.944 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:34:23.945 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:34:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_files = 6 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:34:23.947 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:34:23.989 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:23.989 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:23.989 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:23.989 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:23.990 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:23.990 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:23.990 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:34:23.990 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:24.012 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:34:24.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:34:24.013 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:24.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:34:24.021 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:24.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:34:24.027 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:24.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:34:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.206+0000 7f3270c00640 1 Processor -- start 2024-03-06T07:34:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.207+0000 7f3270c00640 1 -- start start 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.208+0000 7f3270c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f326c0b49c0 0x7f326c072af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.208+0000 7f3270c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0b3d40 con 0x7f326c1668f0 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.208+0000 7f3270c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c166db0 con 0x7f326c165c30 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.208+0000 7f3270c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c167020 con 0x7f326c0b49c0 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f3269400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f326c1668f0 0x7f326c06cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47050/0 (socket says 172.21.15.32:47050) 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f3269400640 1 -- 172.21.15.32:0/1079205851 learned_addr learned my addr 172.21.15.32:0/1079205851 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f326a800640 1 --2- 172.21.15.32:0/1079205851 >> v2:172.21.15.32:3300/0 conn(0x7f326c0b49c0 0x7f326c072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f3268a00640 1 -- 172.21.15.32:0/1079205851 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736552824 0 0) 0x7f326c0b3d40 con 0x7f326c1668f0 2024-03-06T07:34:24.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f3268a00640 1 -- 172.21.15.32:0/1079205851 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3250003520 con 0x7f326c1668f0 2024-03-06T07:34:24.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f326a800640 1 -- 172.21.15.32:0/1079205851 >> v1:172.21.15.39:6789/0 conn(0x7f326c165c30 legacy=0x7f326c1661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f326a800640 1 -- 172.21.15.32:0/1079205851 >> v1:172.21.15.32:6789/0 conn(0x7f326c1668f0 legacy=0x7f326c06cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.209+0000 7f326a800640 1 -- 172.21.15.32:0/1079205851 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c06ca00 con 0x7f326c0b49c0 2024-03-06T07:34:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.211+0000 7f326a800640 1 --2- 172.21.15.32:0/1079205851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c072af0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f326402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a508ce6d365599b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.211+0000 7f3268a00640 1 -- 172.21.15.32:0/1079205851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264039070 con 0x7f326c0b49c0 2024-03-06T07:34:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.211+0000 7f3268a00640 1 -- 172.21.15.32:0/1079205851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326402ccf0 con 0x7f326c0b49c0 2024-03-06T07:34:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.211+0000 7f3268a00640 1 -- 172.21.15.32:0/1079205851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264034640 con 0x7f326c0b49c0 2024-03-06T07:34:24.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 -- 172.21.15.32:0/1079205851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 msgr2=0x7f326c072af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 --2- 172.21.15.32:0/1079205851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c072af0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f326402bf40 comp rx=0 tx=0).stop 2024-03-06T07:34:24.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3c09600640 1 Processor -- start 2024-03-06T07:34:24.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 -- 172.21.15.32:0/1079205851 shutdown_connections 2024-03-06T07:34:24.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 --2- 172.21.15.32:0/1079205851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c072af0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3c09600640 1 -- start start 2024-03-06T07:34:24.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 -- 172.21.15.32:0/1079205851 >> 172.21.15.32:0/1079205851 conn(0x7f326c0b1990 msgr2=0x7f326c0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 -- 172.21.15.32:0/1079205851 shutdown_connections 2024-03-06T07:34:24.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.212+0000 7f3270c00640 1 -- 172.21.15.32:0/1079205851 wait complete. 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.214+0000 7f3c09600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c040d2e30 0x7f3c040d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.214+0000 7f3c09600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c040d22e0 con 0x7f3c040d09d0 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.214+0000 7f3c09600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c040cfbc0 con 0x7f3c040cf720 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.214+0000 7f3c09600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c04161e20 con 0x7f3c040d2e30 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c02a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c040d2e30 0x7f3c040d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c03400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c040d09d0 0x7f3c040b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47062/0 (socket says 172.21.15.32:47062) 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c03400640 1 -- 172.21.15.32:0/2566344597 learned_addr learned my addr 172.21.15.32:0/2566344597 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3270c00640 1 Processor -- start 2024-03-06T07:34:24.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c02a00640 1 -- 172.21.15.32:0/2566344597 >> v1:172.21.15.39:6789/0 conn(0x7f3c040cf720 legacy=0x7f3c040d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c02a00640 1 -- 172.21.15.32:0/2566344597 >> v1:172.21.15.32:6789/0 conn(0x7f3c040d09d0 legacy=0x7f3c040b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c02a00640 1 -- 172.21.15.32:0/2566344597 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040cfea0 con 0x7f3c040d2e30 2024-03-06T07:34:24.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3c01600640 1 -- 172.21.15.32:0/2566344597 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391199481 0 0) 0x7f3c040d22e0 con 0x7f3c040d09d0 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.215+0000 7f3270c00640 1 -- start start 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.216+0000 7f3270c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.216+0000 7f3270c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f326c165c30 0x7f326c15e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3270c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f326c1668f0 0x7f326c09b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3270c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c06d630 con 0x7f326c0b49c0 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3270c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0b3fb0 con 0x7f326c1668f0 2024-03-06T07:34:24.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3270c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c06d7b0 con 0x7f326c165c30 2024-03-06T07:34:24.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3c02a00640 1 --2- 172.21.15.32:0/2566344597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d2e30 0x7f3c040d4b20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40097a0 tx=0x7f3bf402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10760f86ac12a2ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3c01600640 1 -- 172.21.15.32:0/2566344597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4034070 con 0x7f3c040d2e30 2024-03-06T07:34:24.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3c01600640 1 -- 172.21.15.32:0/2566344597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4030610 con 0x7f3c040d2e30 2024-03-06T07:34:24.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3c01600640 1 -- 172.21.15.32:0/2566344597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40309e0 con 0x7f3c040d2e30 2024-03-06T07:34:24.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c14c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40400/0 (socket says 172.21.15.32:40400) 2024-03-06T07:34:24.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 -- 172.21.15.32:0/1010139309 learned_addr learned my addr 172.21.15.32:0/1010139309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:24.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f326a800640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f326c1668f0 0x7f326c09b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269400640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f326c165c30 0x7f326c15e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 -- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f326c165c30 msgr2=0x7f326c15e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f326c165c30 0x7f326c15e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 -- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f326c1668f0 msgr2=0x7f326c09b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f326c1668f0 0x7f326c09b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269e00640 1 -- 172.21.15.32:0/1010139309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c15d720 con 0x7f326c0b49c0 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f326a800640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f326c1668f0 0x7f326c09b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.217+0000 7f3269400640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f326c165c30 0x7f326c15e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3269e00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c14c000 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f325c00c400 tx=0x7f325c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c0052d0 con 0x7f326c0b49c0 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c005490 con 0x7f326c0b49c0 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c007260 con 0x7f326c0b49c0 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c15da00 con 0x7f326c0b49c0 2024-03-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3c09600640 1 -- 172.21.15.32:0/2566344597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d2e30 msgr2=0x7f3c040d4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3c09600640 1 --2- 172.21.15.32:0/2566344597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d2e30 0x7f3c040d4b20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40097a0 tx=0x7f3bf402b960 comp rx=0 tx=0).stop 2024-03-06T07:34:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c15dfc0 con 0x7f326c0b49c0 2024-03-06T07:34:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3c09600640 1 -- 172.21.15.32:0/2566344597 shutdown_connections 2024-03-06T07:34:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3c09600640 1 --2- 172.21.15.32:0/2566344597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d2e30 0x7f3c040d4b20 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.218+0000 7f3c09600640 1 -- 172.21.15.32:0/2566344597 >> 172.21.15.32:0/2566344597 conn(0x7f3c040d4000 msgr2=0x7f3c040d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.219+0000 7f3c09600640 1 -- 172.21.15.32:0/2566344597 shutdown_connections 2024-03-06T07:34:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.220+0000 7f3c09600640 1 -- 172.21.15.32:0/2566344597 wait complete. 2024-03-06T07:34:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.220+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c09e0f0 con 0x7f326c0b49c0 2024-03-06T07:34:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.221+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f325c007400 con 0x7f326c0b49c0 2024-03-06T07:34:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.222+0000 7f3262000640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f324c06b520 0x7f324c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.223+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f325c0896a0 con 0x7f326c0b49c0 2024-03-06T07:34:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.223+0000 7f3c09600640 1 Processor -- start 2024-03-06T07:34:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.223+0000 7f3c09600640 1 -- start start 2024-03-06T07:34:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.223+0000 7f3269400640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f324c06b520 0x7f324c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.224+0000 7f3269400640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f324c06b520 0x7f324c06d9e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f326c0cbd70 tx=0x7f3258009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c09600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c040cf720 0x7f3c04159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c09600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 0x7f3c040cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c09600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c040d2e30 0x7f3c040c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c09600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040b3e20 con 0x7f3c040d09d0 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c09600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040b38f0 con 0x7f3c040cf720 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c09600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040b50e0 con 0x7f3c040d2e30 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c02000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 0x7f3c040cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c02000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 0x7f3c040cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40416/0 (socket says 172.21.15.32:40416) 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c02000640 1 -- 172.21.15.32:0/1697436474 learned_addr learned my addr 172.21.15.32:0/1697436474 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.225+0000 7f3c02a00640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c040cf720 0x7f3c04159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02000640 1 -- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c040d2e30 msgr2=0x7f3c040c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02000640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c040d2e30 0x7f3c040c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02000640 1 -- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c040cf720 msgr2=0x7f3c04159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02000640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c040cf720 0x7f3c04159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02000640 1 -- 172.21.15.32:0/1697436474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0415a2d0 con 0x7f3c040d09d0 2024-03-06T07:34:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02a00640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c040cf720 0x7f3c04159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.226+0000 7f3c02000640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 0x7f3c040cd470 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c450 tx=0x7f3bf000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.228+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00052d0 con 0x7f3c040d09d0 2024-03-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.228+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0005470 con 0x7f3c040d09d0 2024-03-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.228+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0012870 con 0x7f3c040d09d0 2024-03-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.228+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f325c058600 con 0x7f326c0b49c0 2024-03-06T07:34:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.228+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c040c7d80 con 0x7f3c040d09d0 2024-03-06T07:34:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.228+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c040c8340 con 0x7f3c040d09d0 2024-03-06T07:34:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.230+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c040c79a0 con 0x7f3c040d09d0 2024-03-06T07:34:24.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.232+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bf0012a10 con 0x7f3c040d09d0 2024-03-06T07:34:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.233+0000 7f3bfaa00640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.233+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bf0089e50 con 0x7f3c040d09d0 2024-03-06T07:34:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.234+0000 7f3c02a00640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.235+0000 7f3c02a00640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4036000 tx=0x7f3bf403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.237+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3bf0058cc0 con 0x7f3c040d09d0 2024-03-06T07:34:24.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.254+0000 7f1a23a00640 1 Processor -- start 2024-03-06T07:34:24.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.254+0000 7f1a23a00640 1 -- start start 2024-03-06T07:34:24.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a23a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a1c1668f0 0x7f1a1c072af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a23a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c06cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a23a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0b3d40 con 0x7f1a1c165c30 2024-03-06T07:34:24.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a23a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c167020 con 0x7f1a1c1668f0 2024-03-06T07:34:24.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a23a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c06ca00 con 0x7f1a1c0b49c0 2024-03-06T07:34:24.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a21000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1a1c165c30 0x7f1a1c1661e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47074/0 (socket says 172.21.15.32:47074) 2024-03-06T07:34:24.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a21000640 1 -- 172.21.15.32:0/339001687 learned_addr learned my addr 172.21.15.32:0/339001687 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:24.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.256+0000 7f1a1be00640 1 --2- 172.21.15.32:0/339001687 >> v2:172.21.15.32:3300/0 conn(0x7f1a1c1668f0 0x7f1a1c072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a21a00640 1 --2- 172.21.15.32:0/339001687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c06cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a1b400640 1 -- 172.21.15.32:0/339001687 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298059247 0 0) 0x7f1a1c0b3d40 con 0x7f1a1c165c30 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a1b400640 1 -- 172.21.15.32:0/339001687 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04003520 con 0x7f1a1c165c30 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a1be00640 1 -- 172.21.15.32:0/339001687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 msgr2=0x7f1a1c06cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a1be00640 1 --2- 172.21.15.32:0/339001687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c06cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a1be00640 1 -- 172.21.15.32:0/339001687 >> v1:172.21.15.32:6789/0 conn(0x7f1a1c165c30 legacy=0x7f1a1c1661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.257+0000 7f1a1be00640 1 -- 172.21.15.32:0/339001687 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c166db0 con 0x7f1a1c1668f0 2024-03-06T07:34:24.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.259+0000 7f1a1be00640 1 --2- 172.21.15.32:0/339001687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c072af0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f1a080097a0 tx=0x7f1a0802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272b4b9a7a3495a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.259+0000 7f1a1b400640 1 -- 172.21.15.32:0/339001687 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08035020 con 0x7f1a1c1668f0 2024-03-06T07:34:24.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.259+0000 7f1a1b400640 1 -- 172.21.15.32:0/339001687 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08031810 con 0x7f1a1c1668f0 2024-03-06T07:34:24.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.259+0000 7f1a1b400640 1 -- 172.21.15.32:0/339001687 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08031be0 con 0x7f1a1c1668f0 2024-03-06T07:34:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 -- 172.21.15.32:0/339001687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c1668f0 msgr2=0x7f1a1c072af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 --2- 172.21.15.32:0/339001687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c072af0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f1a080097a0 tx=0x7f1a0802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 -- 172.21.15.32:0/339001687 shutdown_connections 2024-03-06T07:34:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 --2- 172.21.15.32:0/339001687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c072af0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 --2- 172.21.15.32:0/339001687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c06cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 -- 172.21.15.32:0/339001687 >> 172.21.15.32:0/339001687 conn(0x7f1a1c0b1990 msgr2=0x7f1a1c0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 -- 172.21.15.32:0/339001687 shutdown_connections 2024-03-06T07:34:24.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.260+0000 7f1a23a00640 1 -- 172.21.15.32:0/339001687 wait complete. 2024-03-06T07:34:24.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.262+0000 7f1a23a00640 1 Processor -- start 2024-03-06T07:34:24.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.262+0000 7f1a23a00640 1 -- start start 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.263+0000 7f1a23a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.263+0000 7f1a23a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 0x7f1a1c15bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.263+0000 7f1a23a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.263+0000 7f1a23a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c06d460 con 0x7f1a1c165c30 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.263+0000 7f1a23a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0b3fb0 con 0x7f1a1c1668f0 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.263+0000 7f1a23a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c06d5e0 con 0x7f1a1c0b49c0 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 0x7f1a1c15bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 0x7f1a1c15bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40448/0 (socket says 172.21.15.32:40448) 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 -- 172.21.15.32:0/3711469452 learned_addr learned my addr 172.21.15.32:0/3711469452 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a21000640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a21a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 -- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 msgr2=0x7f1a1c15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c15b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 -- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a1c1668f0 msgr2=0x7f1a1c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.264+0000 7f1a1be00640 1 -- 172.21.15.32:0/3711469452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c15c470 con 0x7f1a1c165c30 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.265+0000 7f1a1be00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 0x7f1a1c15bdf0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1a08034040 tx=0x7f1a080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.265+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08040070 con 0x7f1a1c165c30 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.265+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08039c70 con 0x7f1a1c165c30 2024-03-06T07:34:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.265+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c15c750 con 0x7f1a1c165c30 2024-03-06T07:34:24.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.265+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08035040 con 0x7f1a1c165c30 2024-03-06T07:34:24.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.265+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c09bca0 con 0x7f1a1c165c30 2024-03-06T07:34:24.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.267+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c09ba90 con 0x7f1a1c165c30 2024-03-06T07:34:24.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.269+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a08038020 con 0x7f1a1c165c30 2024-03-06T07:34:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.270+0000 7f1a18c00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19fc06b4d0 0x7f19fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.271+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a080abf50 con 0x7f1a1c165c30 2024-03-06T07:34:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.271+0000 7f56ed800640 1 Processor -- start 2024-03-06T07:34:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.271+0000 7f1a21000640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19fc06b4d0 0x7f19fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.272+0000 7f56ed800640 1 -- start start 2024-03-06T07:34:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.273+0000 7f1a21000640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19fc06b4d0 0x7f19fc06d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0cf720 tx=0x7f1a10009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.274+0000 7f56ed800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e815d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.274+0000 7f56ed800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f56e80af2e0 0x7f56e80cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.274+0000 7f56ed800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e80ad390 con 0x7f56e8072a50 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.274+0000 7f56ed800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e806e550 con 0x7f56e80af2e0 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.274+0000 7f56ed800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e815d580 con 0x7f56e806e170 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.275+0000 7f56e7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f56e8072a50 0x7f56e8070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47090/0 (socket says 172.21.15.32:47090) 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.275+0000 7f56e7400640 1 -- 172.21.15.32:0/1814824195 learned_addr learned my addr 172.21.15.32:0/1814824195 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.275+0000 7f56e6000640 1 --2- 172.21.15.32:0/1814824195 >> v2:172.21.15.39:3300/0 conn(0x7f56e80af2e0 0x7f56e80cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.275+0000 7f56e6a00640 1 --2- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e815d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.276+0000 7f56e6000640 1 -- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 msgr2=0x7f56e815d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.276+0000 7f56e6000640 1 --2- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e815d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.276+0000 7f56e6000640 1 -- 172.21.15.32:0/1814824195 >> v1:172.21.15.32:6789/0 conn(0x7f56e8072a50 legacy=0x7f56e8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.276+0000 7f56e6000640 1 -- 172.21.15.32:0/1814824195 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80ace10 con 0x7f56e80af2e0 2024-03-06T07:34:24.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.277+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a0802f2c0 con 0x7f1a1c165c30 2024-03-06T07:34:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.278+0000 7f56e6000640 1 --2- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e80cef40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f56d00097a0 tx=0x7f56d002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa47c69313709a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.278+0000 7f56e5600640 1 -- 172.21.15.32:0/1814824195 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0035020 con 0x7f56e80af2e0 2024-03-06T07:34:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.278+0000 7f56e5600640 1 -- 172.21.15.32:0/1814824195 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d0031810 con 0x7f56e80af2e0 2024-03-06T07:34:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.278+0000 7f56e5600640 1 -- 172.21.15.32:0/1814824195 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0031be0 con 0x7f56e80af2e0 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.279+0000 7f56ed800640 1 -- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 msgr2=0x7f56e80cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.279+0000 7f56ed800640 1 --2- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e80cef40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f56d00097a0 tx=0x7f56d002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.280+0000 7f56ed800640 1 -- 172.21.15.32:0/1814824195 shutdown_connections 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.280+0000 7f56ed800640 1 --2- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e80cef40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.280+0000 7f56ed800640 1 --2- 172.21.15.32:0/1814824195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e815d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.280+0000 7f56ed800640 1 -- 172.21.15.32:0/1814824195 >> 172.21.15.32:0/1814824195 conn(0x7f56e80b1990 msgr2=0x7f56e80b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.280+0000 7f56ed800640 1 -- 172.21.15.32:0/1814824195 shutdown_connections 2024-03-06T07:34:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.280+0000 7f56ed800640 1 -- 172.21.15.32:0/1814824195 wait complete. 2024-03-06T07:34:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.283+0000 7f56ed800640 1 Processor -- start 2024-03-06T07:34:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.283+0000 7f56ed800640 1 -- start start 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56ed800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e80ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56ed800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 0x7f56e8151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56ed800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e809afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56ed800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e809ddd0 con 0x7f56e8072a50 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56ed800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e815c840 con 0x7f56e80af2e0 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56ed800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e809df50 con 0x7f56e806e170 2024-03-06T07:34:24.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56e6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 0x7f56e8151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56e6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 0x7f56e8151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40456/0 (socket says 172.21.15.32:40456) 2024-03-06T07:34:24.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56e6000640 1 -- 172.21.15.32:0/646563432 learned_addr learned my addr 172.21.15.32:0/646563432 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.285+0000 7f56e6a00640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e80ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.286+0000 7f56e7400640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e809afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.286+0000 7f56e6000640 1 -- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 msgr2=0x7f56e80ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.286+0000 7f56e6000640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e80ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.286+0000 7f56e6000640 1 -- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 msgr2=0x7f56e809afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.286+0000 7f56e6000640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e809afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.286+0000 7f56e6000640 1 -- 172.21.15.32:0/646563432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e809aaf0 con 0x7f56e8072a50 2024-03-06T07:34:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.287+0000 7f56e6000640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 0x7f56e8151760 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f56d0034040 tx=0x7f56d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.287+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0040070 con 0x7f56e8072a50 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.287+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d0039cf0 con 0x7f56e8072a50 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.287+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0035040 con 0x7f56e8072a50 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.287+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e80ce6f0 con 0x7f56e8072a50 2024-03-06T07:34:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.287+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e80b3f80 con 0x7f56e8072a50 2024-03-06T07:34:24.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.289+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e809e0f0 con 0x7f56e8072a50 2024-03-06T07:34:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.292+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56d0038020 con 0x7f56e8072a50 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.295+0000 7f56dea00640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56c406b5a0 0x7f56c406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.295+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56d00abd80 con 0x7f56e8072a50 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.295+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56e809aaf0 con 0x7f56e8072a50 2024-03-06T07:34:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.295+0000 7f56e6a00640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56c406b5a0 0x7f56c406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.297+0000 7f56e6a00640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56c406b5a0 0x7f56c406da60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f56e00113e0 tx=0x7f56e000b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.522+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f326c04f410 con 0x7f326c0b49c0 2024-03-06T07:34:24.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.562+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f1a1c09e0f0 con 0x7f1a1c165c30 2024-03-06T07:34:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.568+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f3c040d4bb0 con 0x7f3c040d09d0 2024-03-06T07:34:24.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:24.599+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f56e80cea90 con 0x7f56e8072a50 2024-03-06T07:34:25.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.402+0000 7f3262000640 1 -- 172.21.15.32:0/1010139309 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5887 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7f325c057f00 con 0x7f326c0b49c0 2024-03-06T07:34:25.405 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 5887 2024-03-06T07:34:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.405+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f324c06b520 msgr2=0x7f324c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.405+0000 7f3270c00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f324c06b520 0x7f324c06d9e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f326c0cbd70 tx=0x7f3258009040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 msgr2=0x7f326c14c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c14c000 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f325c00c400 tx=0x7f325c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 shutdown_connections 2024-03-06T07:34:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f324c06b520 0x7f324c06d9e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f326c1668f0 0x7f326c09b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f326c165c30 0x7f326c15e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 --2- 172.21.15.32:0/1010139309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f326c0b49c0 0x7f326c14c000 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 >> 172.21.15.32:0/1010139309 conn(0x7f326c0b1990 msgr2=0x7f326c0b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 shutdown_connections 2024-03-06T07:34:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.406+0000 7f3270c00640 1 -- 172.21.15.32:0/1010139309 wait complete. 2024-03-06T07:34:25.420 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:34:25.418+0000 7fd1b1c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.422+0000 7f1a18c00640 1 -- 172.21.15.32:0/3711469452 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7110 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f1a0807adc0 con 0x7f1a1c165c30 2024-03-06T07:34:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.422+0000 7f3bfaa00640 1 -- 172.21.15.32:0/1697436474 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7122 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3bf00585c0 con 0x7f3c040d09d0 2024-03-06T07:34:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 7110 2024-03-06T07:34:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 7122 2024-03-06T07:34:25.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19fc06b4d0 msgr2=0x7f19fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19fc06b4d0 0x7f19fc06d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0cf720 tx=0x7f1a10009040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 msgr2=0x7f1a1c15bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 0x7f1a1c15bdf0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1a08034040 tx=0x7f1a080046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 shutdown_connections 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19fc06b4d0 0x7f19fc06d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a1c1668f0 0x7f1a1c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a1c165c30 0x7f1a1c15bdf0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 --2- 172.21.15.32:0/3711469452 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a1c0b49c0 0x7f1a1c15b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.425+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 >> 172.21.15.32:0/3711469452 conn(0x7f1a1c0b1990 msgr2=0x7f1a1c0b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 shutdown_connections 2024-03-06T07:34:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 msgr2=0x7f3be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4036000 tx=0x7f3bf403b040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 msgr2=0x7f3c040cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 0x7f3c040cd470 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c450 tx=0x7f3bf000c910 comp rx=0 tx=0).stop 2024-03-06T07:34:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f1a23a00640 1 -- 172.21.15.32:0/3711469452 wait complete. 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 shutdown_connections 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3be406b4d0 0x7f3be406d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c040d2e30 0x7f3c040c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c040d09d0 0x7f3c040cd470 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 --2- 172.21.15.32:0/1697436474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c040cf720 0x7f3c04159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 >> 172.21.15.32:0/1697436474 conn(0x7f3c040d4000 msgr2=0x7f3c040b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 shutdown_connections 2024-03-06T07:34:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.426+0000 7f3c09600640 1 -- 172.21.15.32:0/1697436474 wait complete. 2024-03-06T07:34:25.431 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:34:25.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:34:25.441 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:34:25.441 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:25.452 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:34:25.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:34:25.453 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:34:25.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:34:25.458 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:34:25.458 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:25.499 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:34:25.499 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:25.608 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:34:25.615 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:34:25.683 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:34:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.747+0000 7f56dea00640 1 -- 172.21.15.32:0/646563432 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7107 v64) v1 ==== 95+0+0 (secure 0 0 0) 0x7f56d007ab50 con 0x7f56e8072a50 2024-03-06T07:34:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 7107 2024-03-06T07:34:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.751+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56c406b5a0 msgr2=0x7f56c406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.751+0000 7f56ed800640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56c406b5a0 0x7f56c406da60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f56e00113e0 tx=0x7f56e000b040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.751+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 msgr2=0x7f56e8151760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.751+0000 7f56ed800640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 0x7f56e8151760 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f56d0034040 tx=0x7f56d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 shutdown_connections 2024-03-06T07:34:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56c406b5a0 0x7f56c406da60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56e80af2e0 0x7f56e809afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e8072a50 0x7f56e8151760 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 --2- 172.21.15.32:0/646563432 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56e806e170 0x7f56e80ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 >> 172.21.15.32:0/646563432 conn(0x7f56e80b1990 msgr2=0x7f56e80aea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 shutdown_connections 2024-03-06T07:34:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:25.752+0000 7f56ed800640 1 -- 172.21.15.32:0/646563432 wait complete. 2024-03-06T07:34:25.777 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:34:25.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:34:25.823 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:34:25.823 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:25.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:25.984 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:34:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.011+0000 7f9994e00640 1 Processor -- start 2024-03-06T07:34:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.011+0000 7f9994e00640 1 -- start start 2024-03-06T07:34:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f9994e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f99901606c0 0x7f999015fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f9994e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999015c430 con 0x7f999015c050 2024-03-06T07:34:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f9994e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900afd60 con 0x7f9990160060 2024-03-06T07:34:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f9994e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99901604b0 con 0x7f99901606c0 2024-03-06T07:34:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f998f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f999015c050 0x7f99900af360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47106/0 (socket says 172.21.15.32:47106) 2024-03-06T07:34:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f998f400640 1 -- 172.21.15.32:0/3273068955 learned_addr learned my addr 172.21.15.32:0/3273068955 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259698034 0 0) 0x7f999015c430 con 0x7f999015c050 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.013+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974003520 con 0x7f999015c050 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f9987e00640 1 --2- 172.21.15.32:0/3273068955 >> v2:172.21.15.39:3300/0 conn(0x7f99901606c0 0x7f999015fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2725425389 0 0) 0x7f9974003520 con 0x7f999015c050 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 >> v2:172.21.15.39:3300/0 conn(0x7f99901606c0 msgr2=0x7f999015fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 --2- 172.21.15.32:0/3273068955 >> v2:172.21.15.39:3300/0 conn(0x7f99901606c0 0x7f999015fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 >> v1:172.21.15.39:6789/0 conn(0x7f9990160060 legacy=0x7f99900b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900b0040 con 0x7f999015c050 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f9980001670 con 0x7f999015c050 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 >> v1:172.21.15.32:6789/0 conn(0x7f999015c050 legacy=0x7f99900af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9974003a00 0x7f9974003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974004800 0x7f9974004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9974005210 0x7f997400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99901604b0 con 0x7f9974003a00 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997400c140 con 0x7f9974004800 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997400c340 con 0x7f9974005210 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.014+0000 7f998f400640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9974003a00 0x7f9974003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f9987e00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974004800 0x7f9974004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998f400640 1 -- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9974005210 msgr2=0x7f997400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998fe00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9974005210 0x7f997400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998f400640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9974005210 0x7f997400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998f400640 1 -- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974004800 msgr2=0x7f9974004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998f400640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974004800 0x7f9974004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998f400640 1 -- 172.21.15.32:0/3273068955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974003520 con 0x7f9974003a00 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f9987e00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974004800 0x7f9974004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998fe00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9974005210 0x7f997400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.015+0000 7f998fe00640 1 -- 172.21.15.32:0/3273068955 reap_dead start 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.016+0000 7f998f400640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9974003a00 0x7f9974003e00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f998000c160 tx=0x7f9974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7377a1a99884868 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.016+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980007c60 con 0x7f9974003a00 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.016+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980030070 con 0x7f9974003a00 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.016+0000 7f998ea00640 1 -- 172.21.15.32:0/3273068955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980005d50 con 0x7f9974003a00 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 -- 172.21.15.32:0/3273068955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9974003a00 msgr2=0x7f9974003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9974003a00 0x7f9974003e00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f998000c160 tx=0x7f9974005830 comp rx=0 tx=0).stop 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 -- 172.21.15.32:0/3273068955 shutdown_connections 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 --2- 172.21.15.32:0/3273068955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9974003a00 0x7f9974003e00 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 -- 172.21.15.32:0/3273068955 >> 172.21.15.32:0/3273068955 conn(0x7f99900b7450 msgr2=0x7f999009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 -- 172.21.15.32:0/3273068955 shutdown_connections 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.017+0000 7f9994e00640 1 -- 172.21.15.32:0/3273068955 wait complete. 2024-03-06T07:34:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.019+0000 7f9994e00640 1 Processor -- start 2024-03-06T07:34:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.019+0000 7f9994e00640 1 -- start start 2024-03-06T07:34:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9994e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974003a00 0x7f99901457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9994e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9990153120 0x7f99901535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9994e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 0x7f9990157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9994e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b0b40 con 0x7f9990156b90 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9994e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b0600 con 0x7f9974003a00 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9994e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999015fb20 con 0x7f9990153120 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f998fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 0x7f9990157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f998fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 0x7f9990157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40484/0 (socket says 172.21.15.32:40484) 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f998fe00640 1 -- 172.21.15.32:0/91310458 learned_addr learned my addr 172.21.15.32:0/91310458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f998f400640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974003a00 0x7f99901457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.020+0000 7f9987e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9990153120 0x7f99901535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998fe00640 1 -- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9990153120 msgr2=0x7f99901535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998fe00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9990153120 0x7f99901535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998fe00640 1 -- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974003a00 msgr2=0x7f99901457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998fe00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974003a00 0x7f99901457f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998fe00640 1 -- 172.21.15.32:0/91310458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99901f5830 con 0x7f9990156b90 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998f400640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974003a00 0x7f99901457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f998fe00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 0x7f9990157040 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f999006e8e0 tx=0x7f998800d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988018070 con 0x7f9990156b90 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9988003d80 con 0x7f9990156b90 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99901f5b10 con 0x7f9990156b90 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988005310 con 0x7f9990156b90 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.021+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99901f60d0 con 0x7f9990156b90 2024-03-06T07:34:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.023+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9990153ae0 con 0x7f9990156b90 2024-03-06T07:34:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.024+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f998801b070 con 0x7f9990156b90 2024-03-06T07:34:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.025+0000 7f9987400640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f996c06b500 0x7f996c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.025+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9988089930 con 0x7f9990156b90 2024-03-06T07:34:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.025+0000 7f998f400640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f996c06b500 0x7f996c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.026+0000 7f998f400640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f996c06b500 0x7f996c06d9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9980034000 tx=0x7f9980039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.030+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9988058740 con 0x7f9990156b90 2024-03-06T07:34:26.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.277+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999015f180 con 0x7f9990156b90 2024-03-06T07:34:26.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.278+0000 7f9987400640 1 -- 172.21.15.32:0/91310458 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4182 (secure 0 0 0) 0x7f9988058040 con 0x7f9990156b90 2024-03-06T07:34:26.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:26.281 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:25.745135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:34:26.281 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2024-03-06T07:34:26.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f996c06b500 msgr2=0x7f996c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f996c06b500 0x7f996c06d9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9980034000 tx=0x7f9980039040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 msgr2=0x7f9990157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 0x7f9990157040 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f999006e8e0 tx=0x7f998800d920 comp rx=0 tx=0).stop 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 shutdown_connections 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f996c06b500 0x7f996c06d9c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9990156b90 0x7f9990157040 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9990153120 0x7f99901535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 --2- 172.21.15.32:0/91310458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9974003a00 0x7f99901457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 >> 172.21.15.32:0/91310458 conn(0x7f99900b7450 msgr2=0x7f99900b59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 shutdown_connections 2024-03-06T07:34:26.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:26.281+0000 7f9994e00640 1 -- 172.21.15.32:0/91310458 wait complete. 2024-03-06T07:34:26.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:08.726069+0000', 'modified': '2024-03-06T07:34:25.745135+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7344}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7344': {'gid': 7344, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/1229875104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1229875104}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1229875104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:26.705 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7344:up:replay 2024-03-06T07:34:27.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.930+0000 7f74d4400640 1 Processor -- start 2024-03-06T07:34:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.931+0000 7f74d4400640 1 -- start start 2024-03-06T07:34:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d4400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f74cc06e5f0 0x7f74cc06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d4400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74cc06c510 0x7f74cc0aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d4400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc06c920 con 0x7f74cc1587e0 2024-03-06T07:34:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d4400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc1623f0 con 0x7f74cc06c510 2024-03-06T07:34:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d4400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc158bc0 con 0x7f74cc06e5f0 2024-03-06T07:34:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d1a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f74cc1587e0 0x7f74cc161ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57510/0 (socket says 172.21.15.32:57510) 2024-03-06T07:34:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.932+0000 7f74d1a00640 1 -- 172.21.15.32:0/2563285236 learned_addr learned my addr 172.21.15.32:0/2563285236 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74d2400640 1 --2- 172.21.15.32:0/2563285236 >> v2:172.21.15.32:3300/0 conn(0x7f74cc06c510 0x7f74cc0aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74cbe00640 1 -- 172.21.15.32:0/2563285236 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516082136 0 0) 0x7f74c0001110 con 0x7f74cc1587e0 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74cbe00640 1 -- 172.21.15.32:0/2563285236 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b4003520 con 0x7f74cc1587e0 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74d1000640 1 --2- 172.21.15.32:0/2563285236 >> v2:172.21.15.39:3300/0 conn(0x7f74cc06e5f0 0x7f74cc06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74d2400640 1 -- 172.21.15.32:0/2563285236 >> v2:172.21.15.39:3300/0 conn(0x7f74cc06e5f0 msgr2=0x7f74cc06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74d2400640 1 --2- 172.21.15.32:0/2563285236 >> v2:172.21.15.39:3300/0 conn(0x7f74cc06e5f0 0x7f74cc06bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74d2400640 1 -- 172.21.15.32:0/2563285236 >> v1:172.21.15.32:6789/0 conn(0x7f74cc1587e0 legacy=0x7f74cc161ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.933+0000 7f74d2400640 1 -- 172.21.15.32:0/2563285236 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc162c10 con 0x7f74cc06c510 2024-03-06T07:34:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d2400640 1 --2- 172.21.15.32:0/2563285236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc0aedd0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0b32d0 tx=0x7f74c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7012e72e11c51880 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74cbe00640 1 -- 172.21.15.32:0/2563285236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4035020 con 0x7f74cc06c510 2024-03-06T07:34:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74cbe00640 1 -- 172.21.15.32:0/2563285236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4031830 con 0x7f74cc06c510 2024-03-06T07:34:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74cbe00640 1 -- 172.21.15.32:0/2563285236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4031c00 con 0x7f74cc06c510 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d4400640 1 -- 172.21.15.32:0/2563285236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 msgr2=0x7f74cc0aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d4400640 1 --2- 172.21.15.32:0/2563285236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc0aedd0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0b32d0 tx=0x7f74c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d4400640 1 -- 172.21.15.32:0/2563285236 shutdown_connections 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d4400640 1 --2- 172.21.15.32:0/2563285236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc0aedd0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d4400640 1 --2- 172.21.15.32:0/2563285236 >> v2:172.21.15.39:3300/0 conn(0x7f74cc06e5f0 0x7f74cc06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.935+0000 7f74d4400640 1 -- 172.21.15.32:0/2563285236 >> 172.21.15.32:0/2563285236 conn(0x7f74cc0b2980 msgr2=0x7f74cc0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.936+0000 7f74d4400640 1 -- 172.21.15.32:0/2563285236 shutdown_connections 2024-03-06T07:34:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.936+0000 7f74d4400640 1 -- 172.21.15.32:0/2563285236 wait complete. 2024-03-06T07:34:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.937+0000 7f74d4400640 1 Processor -- start 2024-03-06T07:34:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.937+0000 7f74d4400640 1 -- start start 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc14d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d4400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc06e5f0 0x7f74cc15e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d4400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74cc1587e0 0x7f74cc15ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0ae5d0 con 0x7f74cc06c510 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d4400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc162930 con 0x7f74cc06e5f0 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d4400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0ae8b0 con 0x7f74cc1587e0 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.938+0000 7f74d1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc14d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc14d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59258/0 (socket says 172.21.15.32:59258) 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 -- 172.21.15.32:0/625931255 learned_addr learned my addr 172.21.15.32:0/625931255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1000640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc06e5f0 0x7f74cc15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 -- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74cc1587e0 msgr2=0x7f74cc15ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74cc1587e0 0x7f74cc15ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 -- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc06e5f0 msgr2=0x7f74cc15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc06e5f0 0x7f74cc15e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 -- 172.21.15.32:0/625931255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc153960 con 0x7f74cc06c510 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74d1a00640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc14d8e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f74c000c3a0 tx=0x7f74c000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.939+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0017070 con 0x7f74cc06c510 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.940+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c0005ce0 con 0x7f74cc06c510 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.940+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc153c40 con 0x7f74cc06c510 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.940+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0007310 con 0x7f74cc06c510 2024-03-06T07:34:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.940+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc154200 con 0x7f74cc06c510 2024-03-06T07:34:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.941+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc0692a0 con 0x7f74cc06c510 2024-03-06T07:34:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.942+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74c001a070 con 0x7f74cc06c510 2024-03-06T07:34:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.943+0000 7f74c9600640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74b006b500 0x7f74b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.943+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f74c0089680 con 0x7f74cc06c510 2024-03-06T07:34:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.943+0000 7f74d1000640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74b006b500 0x7f74b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.944+0000 7f74d1000640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74b006b500 0x7f74b006d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f74cc157ea0 tx=0x7f74bc00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:27.947+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74cc06c920 con 0x7f74cc06c510 2024-03-06T07:34:28.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.190+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74cc071db0 con 0x7f74cc06c510 2024-03-06T07:34:28.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.192+0000 7f74c9600640 1 -- 172.21.15.32:0/625931255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f74c0058490 con 0x7f74cc06c510 2024-03-06T07:34:28.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:28.194 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:26.774201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:34:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 66 2024-03-06T07:34:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74b006b500 msgr2=0x7f74b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74b006b500 0x7f74b006d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f74cc157ea0 tx=0x7f74bc00b040 comp rx=0 tx=0).stop 2024-03-06T07:34:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 msgr2=0x7f74cc14d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc14d8e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f74c000c3a0 tx=0x7f74c000c860 comp rx=0 tx=0).stop 2024-03-06T07:34:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 shutdown_connections 2024-03-06T07:34:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f74b006b500 0x7f74b006d9c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f74cc1587e0 0x7f74cc15ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74cc06e5f0 0x7f74cc15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 --2- 172.21.15.32:0/625931255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74cc06c510 0x7f74cc14d8e0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 >> 172.21.15.32:0/625931255 conn(0x7f74cc0b2980 msgr2=0x7f74cc150b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 shutdown_connections 2024-03-06T07:34:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:28.195+0000 7f74d4400640 1 -- 172.21.15.32:0/625931255 wait complete. 2024-03-06T07:34:28.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:08.726069+0000', 'modified': '2024-03-06T07:34:26.774201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7344}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7344': {'gid': 7344, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.32:6835/1229875104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1229875104}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1229875104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:28.619 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7344:up:rejoin 2024-03-06T07:34:29.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:29.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.843+0000 7f1b82000640 1 Processor -- start 2024-03-06T07:34:29.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.843+0000 7f1b82000640 1 -- start start 2024-03-06T07:34:29.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.844+0000 7f1b82000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b7c1457e0 0x7f1b7c0cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b82000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c145bc0 con 0x7f1b7c15fba0 2024-03-06T07:34:29.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b82000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c0cfb20 con 0x7f1b7c0b5870 2024-03-06T07:34:29.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b82000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7c160040 con 0x7f1b7c1457e0 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b7b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b7c1457e0 0x7f1b7c0cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b7b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b7c1457e0 0x7f1b7c0cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59268/0 (socket says 172.21.15.32:59268) 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b7b400640 1 -- 172.21.15.32:0/4203606846 learned_addr learned my addr 172.21.15.32:0/4203606846 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.845+0000 7f1b73e00640 1 --1- 172.21.15.32:0/4203606846 >> v1:172.21.15.32:6789/0 conn(0x7f1b7c15fba0 0x7f1b7c0b5160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57512/0 (socket says 172.21.15.32:57512) 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.846+0000 7f1b7b400640 1 -- 172.21.15.32:0/4203606846 >> v1:172.21.15.39:6789/0 conn(0x7f1b7c0b5870 legacy=0x7f1b7c0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.846+0000 7f1b7b400640 1 -- 172.21.15.32:0/4203606846 >> v1:172.21.15.32:6789/0 conn(0x7f1b7c15fba0 legacy=0x7f1b7c0b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.846+0000 7f1b7b400640 1 -- 172.21.15.32:0/4203606846 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c160320 con 0x7f1b7c1457e0 2024-03-06T07:34:29.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.846+0000 7f1b7aa00640 1 -- 172.21.15.32:0/4203606846 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340581371 0 0) 0x7f1b7c145bc0 con 0x7f1b7c15fba0 2024-03-06T07:34:29.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.847+0000 7f1b7b400640 1 --2- 172.21.15.32:0/4203606846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c1457e0 0x7f1b7c0cf5e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009870 tx=0x7f1b6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6675af0fdbcdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b7aa00640 1 -- 172.21.15.32:0/4203606846 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c035020 con 0x7f1b7c1457e0 2024-03-06T07:34:29.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b7aa00640 1 -- 172.21.15.32:0/4203606846 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c031870 con 0x7f1b7c1457e0 2024-03-06T07:34:29.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b7aa00640 1 -- 172.21.15.32:0/4203606846 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c031c40 con 0x7f1b7c1457e0 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b82000640 1 -- 172.21.15.32:0/4203606846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c1457e0 msgr2=0x7f1b7c0cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b82000640 1 --2- 172.21.15.32:0/4203606846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c1457e0 0x7f1b7c0cf5e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009870 tx=0x7f1b6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b82000640 1 -- 172.21.15.32:0/4203606846 shutdown_connections 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.848+0000 7f1b82000640 1 --2- 172.21.15.32:0/4203606846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c1457e0 0x7f1b7c0cf5e0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.849+0000 7f1b82000640 1 -- 172.21.15.32:0/4203606846 >> 172.21.15.32:0/4203606846 conn(0x7f1b7c06cf40 msgr2=0x7f1b7c06d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.849+0000 7f1b82000640 1 -- 172.21.15.32:0/4203606846 shutdown_connections 2024-03-06T07:34:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.849+0000 7f1b82000640 1 -- 172.21.15.32:0/4203606846 wait complete. 2024-03-06T07:34:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.851+0000 7f1b82000640 1 Processor -- start 2024-03-06T07:34:29.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.851+0000 7f1b82000640 1 -- start start 2024-03-06T07:34:29.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.851+0000 7f1b82000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 0x7f1b7c140560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b82000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b7c1457e0 0x7f1b7c1f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b82000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b7c15fba0 0x7f1b7c0c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b82000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0cca70 con 0x7f1b7c0b5870 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b82000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c1608e0 con 0x7f1b7c15fba0 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b82000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0717c0 con 0x7f1b7c1457e0 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 0x7f1b7c140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 0x7f1b7c140560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59278/0 (socket says 172.21.15.32:59278) 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 -- 172.21.15.32:0/1973532298 learned_addr learned my addr 172.21.15.32:0/1973532298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 -- 172.21.15.32:0/1973532298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b7c1457e0 msgr2=0x7f1b7c1f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b7c1457e0 0x7f1b7c1f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 -- 172.21.15.32:0/1973532298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b7c15fba0 msgr2=0x7f1b7c0c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b7c15fba0 0x7f1b7c0c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.852+0000 7f1b7b400640 1 -- 172.21.15.32:0/1973532298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c142450 con 0x7f1b7c0b5870 2024-03-06T07:34:29.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.853+0000 7f1b7b400640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 0x7f1b7c140560 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c000c00 tx=0x7f1b6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.853+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c040070 con 0x7f1b7c0b5870 2024-03-06T07:34:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.853+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c03a420 con 0x7f1b7c0b5870 2024-03-06T07:34:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.853+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c035040 con 0x7f1b7c0b5870 2024-03-06T07:34:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.853+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c1426f0 con 0x7f1b7c0b5870 2024-03-06T07:34:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.853+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c142cb0 con 0x7f1b7c0b5870 2024-03-06T07:34:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.855+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c0692a0 con 0x7f1b7c0b5870 2024-03-06T07:34:29.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.855+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b6c04b060 con 0x7f1b7c0b5870 2024-03-06T07:34:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.856+0000 7f1b73400640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b5806b500 0x7f1b5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.856+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1b6c0ac110 con 0x7f1b7c0b5870 2024-03-06T07:34:29.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.856+0000 7f1b73e00640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b5806b500 0x7f1b5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.857+0000 7f1b73e00640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b5806b500 0x7f1b5806d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1b64007fd0 tx=0x7f1b64007960 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:29.861+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b6c07afc0 con 0x7f1b7c0b5870 2024-03-06T07:34:30.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.227+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7c0b5400 con 0x7f1b7c0b5870 2024-03-06T07:34:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.228+0000 7f1b73400640 1 -- 172.21.15.32:0/1973532298 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1b6c07a8c0 con 0x7f1b7c0b5870 2024-03-06T07:34:30.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:30.230 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b5806b500 msgr2=0x7f1b5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b5806b500 0x7f1b5806d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1b64007fd0 tx=0x7f1b64007960 comp rx=0 tx=0).stop 2024-03-06T07:34:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 msgr2=0x7f1b7c140560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 0x7f1b7c140560 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c000c00 tx=0x7f1b6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 shutdown_connections 2024-03-06T07:34:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b5806b500 0x7f1b5806d9c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b7c15fba0 0x7f1b7c0c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b7c1457e0 0x7f1b7c1f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 --2- 172.21.15.32:0/1973532298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b7c0b5870 0x7f1b7c140560 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 >> 172.21.15.32:0/1973532298 conn(0x7f1b7c06cf40 msgr2=0x7f1b7c144b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 shutdown_connections 2024-03-06T07:34:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.231+0000 7f1b82000640 1 -- 172.21.15.32:0/1973532298 wait complete. 2024-03-06T07:34:30.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:08.726069+0000', 'modified': '2024-03-06T07:34:28.763619+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7344}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7344': {'gid': 7344, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/1229875104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1229875104}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1229875104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7344, 'qdb_cluster': [7344]} 2024-03-06T07:34:30.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:30.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:34:30.760 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-03-06T07:34:30.759+0000 7f1023200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:30.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.878+0000 7efd76200640 1 Processor -- start 2024-03-06T07:34:30.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.878+0000 7efd76200640 1 -- start start 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.879+0000 7efd76200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700727a0 0x7efd70072b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.879+0000 7efd76200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd700b3ea0 0x7efd700b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.879+0000 7efd76200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd700ae880 con 0x7efd700ceed0 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.879+0000 7efd76200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700af490 con 0x7efd700b3ea0 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.879+0000 7efd76200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700aeb60 con 0x7efd700727a0 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.880+0000 7efd6fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd700b3ea0 0x7efd700b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.880+0000 7efd75800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efd700ceed0 0x7efd700cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57524/0 (socket says 172.21.15.32:57524) 2024-03-06T07:34:30.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.880+0000 7efd75800640 1 -- 172.21.15.32:0/831649719 learned_addr learned my addr 172.21.15.32:0/831649719 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:30.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.880+0000 7efd74e00640 1 --2- 172.21.15.32:0/831649719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700727a0 0x7efd70072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.881+0000 7efd6f400640 1 -- 172.21.15.32:0/831649719 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936598939 0 0) 0x7efd700ae880 con 0x7efd700ceed0 2024-03-06T07:34:30.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.881+0000 7efd6f400640 1 -- 172.21.15.32:0/831649719 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd54005730 con 0x7efd700ceed0 2024-03-06T07:34:30.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.881+0000 7efd6fe00640 1 -- 172.21.15.32:0/831649719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700727a0 msgr2=0x7efd70072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.881+0000 7efd6fe00640 1 --2- 172.21.15.32:0/831649719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700727a0 0x7efd70072b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.881+0000 7efd6fe00640 1 -- 172.21.15.32:0/831649719 >> v1:172.21.15.32:6789/0 conn(0x7efd700ceed0 legacy=0x7efd700cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.881+0000 7efd6fe00640 1 -- 172.21.15.32:0/831649719 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700aee40 con 0x7efd700b3ea0 2024-03-06T07:34:30.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.882+0000 7efd6fe00640 1 --2- 172.21.15.32:0/831649719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700b4320 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7efd580097a0 tx=0x7efd5802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e682f493a55151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.882+0000 7efd6f400640 1 -- 172.21.15.32:0/831649719 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58035020 con 0x7efd700b3ea0 2024-03-06T07:34:30.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.882+0000 7efd6f400640 1 -- 172.21.15.32:0/831649719 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58031810 con 0x7efd700b3ea0 2024-03-06T07:34:30.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.882+0000 7efd6f400640 1 -- 172.21.15.32:0/831649719 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58031be0 con 0x7efd700b3ea0 2024-03-06T07:34:30.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 -- 172.21.15.32:0/831649719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 msgr2=0x7efd700b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 --2- 172.21.15.32:0/831649719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700b4320 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7efd580097a0 tx=0x7efd5802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 -- 172.21.15.32:0/831649719 shutdown_connections 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 --2- 172.21.15.32:0/831649719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700b4320 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 --2- 172.21.15.32:0/831649719 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700727a0 0x7efd70072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 -- 172.21.15.32:0/831649719 >> 172.21.15.32:0/831649719 conn(0x7efd700b1990 msgr2=0x7efd700b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 -- 172.21.15.32:0/831649719 shutdown_connections 2024-03-06T07:34:30.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.883+0000 7efd76200640 1 -- 172.21.15.32:0/831649719 wait complete. 2024-03-06T07:34:30.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.885+0000 7efd76200640 1 Processor -- start 2024-03-06T07:34:30.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.885+0000 7efd76200640 1 -- start start 2024-03-06T07:34:30.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd76200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd700727a0 0x7efd70146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd76200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd76200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700ceed0 0x7efd700ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd76200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7015b770 con 0x7efd700b3ea0 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd76200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7015b5f0 con 0x7efd700727a0 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd76200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7015b8f0 con 0x7efd700ceed0 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59296/0 (socket says 172.21.15.32:59296) 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd6fe00640 1 -- 172.21.15.32:0/2363042125 learned_addr learned my addr 172.21.15.32:0/2363042125 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd74e00640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd700727a0 0x7efd70146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.886+0000 7efd75800640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700ceed0 0x7efd700ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6fe00640 1 -- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700ceed0 msgr2=0x7efd700ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6fe00640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700ceed0 0x7efd700ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6fe00640 1 -- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd700727a0 msgr2=0x7efd70146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6fe00640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd700727a0 0x7efd70146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6fe00640 1 -- 172.21.15.32:0/2363042125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd701f7010 con 0x7efd700b3ea0 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd75800640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700ceed0 0x7efd700ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6fe00640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700cdae0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7efd580098c0 tx=0x7efd580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6cc00640 1 -- 172.21.15.32:0/2363042125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58006e10 con 0x7efd700b3ea0 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6cc00640 1 -- 172.21.15.32:0/2363042125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5803bd70 con 0x7efd700b3ea0 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd701f72f0 con 0x7efd700b3ea0 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.887+0000 7efd6cc00640 1 -- 172.21.15.32:0/2363042125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58045630 con 0x7efd700b3ea0 2024-03-06T07:34:30.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.888+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd701f78b0 con 0x7efd700b3ea0 2024-03-06T07:34:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.890+0000 7efd6cc00640 1 -- 172.21.15.32:0/2363042125 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd5804c060 con 0x7efd700b3ea0 2024-03-06T07:34:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.891+0000 7efd7a5fd740 1 Processor -- start 2024-03-06T07:34:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.892+0000 7efd6cc00640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd4c06b4d0 0x7efd4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.892+0000 7efd7a5fd740 1 -- start start 2024-03-06T07:34:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.892+0000 7efd6cc00640 1 -- 172.21.15.32:0/2363042125 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efd580ac1c0 con 0x7efd700b3ea0 2024-03-06T07:34:30.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.892+0000 7efd75800640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd4c06b4d0 0x7efd4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd75800640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd4c06b4d0 0x7efd4c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7efd701f7bd0 tx=0x7efd68009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567aff5e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff8e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff5ee70 0x5567aff8ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567afec8600 con 0x5567aff8bd70 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567afe71930 con 0x5567aff8b990 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5567afec46a0 con 0x5567aff5ee70 2024-03-06T07:34:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff8e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff8e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59306/0 (socket says 172.21.15.32:59306) 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.893+0000 7efd6fe00640 1 -- 172.21.15.32:0/3661154212 learned_addr learned my addr 172.21.15.32:0/3661154212 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd74e00640 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567aff5e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd75800640 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff5ee70 0x5567aff8ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd6fe00640 1 -- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff5ee70 msgr2=0x5567aff8ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd6fe00640 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff5ee70 0x5567aff8ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd6fe00640 1 -- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 msgr2=0x5567aff5e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd6fe00640 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567aff5e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd6fe00640 1 -- 172.21.15.32:0/3661154212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567afbdd2d0 con 0x5567aff8bd70 2024-03-06T07:34:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd6fe00640 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff8e720 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7efd5802be40 tx=0x7efd5807a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7ca32943872e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd65600640 1 -- 172.21.15.32:0/3661154212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5807e990 con 0x5567aff8bd70 2024-03-06T07:34:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd65600640 1 -- 172.21.15.32:0/3661154212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58082f50 con 0x5567aff8bd70 2024-03-06T07:34:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.894+0000 7efd65600640 1 -- 172.21.15.32:0/3661154212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58078b40 con 0x5567aff8bd70 2024-03-06T07:34:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 -- 172.21.15.32:0/3661154212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 msgr2=0x5567aff8e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff8e720 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7efd5802be40 tx=0x7efd5807a4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 -- 172.21.15.32:0/3661154212 shutdown_connections 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff5ee70 0x5567aff8ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff8e720 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/3661154212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567aff5e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 -- 172.21.15.32:0/3661154212 >> 172.21.15.32:0/3661154212 conn(0x5567aff60ba0 msgr2=0x5567affa9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 -- 172.21.15.32:0/3661154212 shutdown_connections 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.895+0000 7efd7a5fd740 1 -- 172.21.15.32:0/3661154212 wait complete. 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.896+0000 7efd7a5fd740 1 -- start start 2024-03-06T07:34:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.896+0000 7efd7a5fd740 1 Processor -- start 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 0x5567affaa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8bd70 0x5567affa7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567aff90c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567afe71930 con 0x5567aff8b990 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567afe84d90 con 0x5567aff800c0 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5567afec8600 con 0x5567aff8bd70 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd7a5fd740 10 client.0 _ll_register_callbacks cb 0x5567afec0350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd74e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 0x5567affaa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd74e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 0x5567affaa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59322/0 (socket says 172.21.15.32:59322) 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.897+0000 7efd74e00640 1 -- 172.21.15.32:0/1497111828 learned_addr learned my addr 172.21.15.32:0/1497111828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd75800640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567aff90c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd6fe00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8bd70 0x5567affa7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd74e00640 1 -- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8bd70 msgr2=0x5567affa7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd74e00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8bd70 0x5567affa7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd74e00640 1 -- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 msgr2=0x5567aff90c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd74e00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567aff90c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd74e00640 1 -- 172.21.15.32:0/1497111828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567afca76b0 con 0x5567aff8b990 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd6fe00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8bd70 0x5567affa7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd74e00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 0x5567affaa2e0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7efd600043a0 tx=0x7efd60004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fc0d8d41cebd56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd5e000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.898+0000 7efd5e000640 1 -- 172.21.15.32:0/1497111828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60019020 con 0x5567aff8b990 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd58082f50 con 0x5567aff8b990 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd5e000640 1 -- 172.21.15.32:0/1497111828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efd60007b80 con 0x5567aff8b990 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd5e000640 1 -- 172.21.15.32:0/1497111828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd60023050 con 0x5567aff8b990 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 10 client.7389 fetch_fsmap learned FSMap version 67 2024-03-06T07:34:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x5567affa5480 con 0x5567aff8b990 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd5e000640 1 -- 172.21.15.32:0/1497111828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58082f50 con 0x5567aff8b990 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd5e000640 1 -- 172.21.15.32:0/1497111828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7efd60026020 con 0x5567aff8b990 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 10 client.7389 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 10 client.7389 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 20 client.7389 populate_metadata read hostname 'smithi032' 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567affa4490 0x5567affa6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 4 client.7389 mds_command: new command op to 7344 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.899+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 --> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5567afed5b40 con 0x5567affa4490 2024-03-06T07:34:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.900+0000 7efd6fe00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567affa4490 0x5567affa6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.900+0000 7efd6fe00640 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567affa4490 0x5567affa6290 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85cbfcf50f6ad29b server_cookie=cf7c9b7e3d99d1bd in_seq=0 out_seq=0 2024-03-06T07:34:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.900+0000 7efd5e000640 10 client.7389 ms_handle_connect on v2:172.21.15.32:6834/1229875104 2024-03-06T07:34:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.901+0000 7efd5e000640 1 -- 172.21.15.32:0/1497111828 <== mds.0 v2:172.21.15.32:6834/1229875104 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5567afed5b40 con 0x5567affa4490 2024-03-06T07:34:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.901+0000 7efd5e000640 10 client.7389 handle_command_reply: tid=0 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.901+0000 7efd7a5fd740 1 client.7389 shutdown 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 msgr2=0x5567affaa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 0x5567affaa2e0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7efd600043a0 tx=0x7efd60004860 comp rx=0 tx=0).stop 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 shutdown_connections 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567affa4490 0x5567affa6290 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567aff90c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8bd70 0x5567affa7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1497111828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8b990 0x5567affaa2e0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd5e000640 0 client.7389 ms_handle_reset on v2:172.21.15.32:6834/1229875104 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 >> 172.21.15.32:0/1497111828 conn(0x5567aff5ee70 msgr2=0x5567aff62810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 shutdown_connections 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1497111828 wait complete. 2024-03-06T07:34:30.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.902+0000 7efd7a5fd740 20 client.7389 trim_cache size 0 max 16384 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.921+0000 7efd7a5fd740 1 Processor -- start 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.921+0000 7efd7a5fd740 1 -- start start 2024-03-06T07:34:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567affa7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8b990 0x5567aff7f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701ff360 con 0x5567aff8bd70 2024-03-06T07:34:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7014ebe0 con 0x5567aff800c0 2024-03-06T07:34:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7014fd20 con 0x5567aff8b990 2024-03-06T07:34:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd75800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd75800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59334/0 (socket says 172.21.15.32:59334) 2024-03-06T07:34:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.922+0000 7efd75800640 1 -- 172.21.15.32:0/1340680999 learned_addr learned my addr 172.21.15.32:0/1340680999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd75800640 1 -- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8b990 msgr2=0x5567aff7f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd6fe00640 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8b990 0x5567aff7f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd75800640 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8b990 0x5567aff7f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd75800640 1 -- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 msgr2=0x5567affa7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd75800640 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567affa7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd75800640 1 -- 172.21.15.32:0/1340680999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567aff8b280 con 0x5567aff8bd70 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd75800640 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7fa40 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7efd68007ab0 tx=0x7efd68011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9121c406bb145dca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd64c00640 1 -- 172.21.15.32:0/1340680999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68025070 con 0x5567aff8bd70 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd64c00640 1 -- 172.21.15.32:0/1340680999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6800d700 con 0x5567aff8bd70 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.923+0000 7efd64c00640 1 -- 172.21.15.32:0/1340680999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800dad0 con 0x5567aff8bd70 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1340680999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 msgr2=0x5567aff7fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7fa40 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7efd68007ab0 tx=0x7efd68011000 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1340680999 shutdown_connections 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7fa40 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff8b990 0x5567aff7f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/1340680999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff800c0 0x5567affa7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1340680999 >> 172.21.15.32:0/1340680999 conn(0x5567aff5ee70 msgr2=0x5567aff5dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1340680999 shutdown_connections 2024-03-06T07:34:30.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.924+0000 7efd7a5fd740 1 -- 172.21.15.32:0/1340680999 wait complete. 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.925+0000 7efd7a5fd740 1 -- start start 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.925+0000 7efd7a5fd740 1 Processor -- start 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff800c0 0x5567affaaf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567affab450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567affa4310 con 0x5567aff8bd70 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567affa6ca0 con 0x5567aff8b990 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5567affa5d40 con 0x5567aff800c0 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd7a5fd740 10 client.0 _ll_register_callbacks cb 0x7efd60003cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd75800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd75800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59338/0 (socket says 172.21.15.32:59338) 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.926+0000 7efd75800640 1 -- 172.21.15.32:0/2979171492 learned_addr learned my addr 172.21.15.32:0/2979171492 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd6fe00640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567affab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd74e00640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff800c0 0x5567affaaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd75800640 1 -- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff800c0 msgr2=0x5567affaaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd75800640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff800c0 0x5567affaaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd75800640 1 -- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 msgr2=0x5567affab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd75800640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567affab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd75800640 1 -- 172.21.15.32:0/2979171492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567aff8fd10 con 0x5567aff8bd70 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd6fe00640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567affab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd75800640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7ec40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7efd680011e0 tx=0x7efd68000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7354d4543f2e5c79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd5e000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd5e000640 1 -- 172.21.15.32:0/2979171492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68025040 con 0x5567aff8bd70 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.927+0000 7efd5e000640 1 -- 172.21.15.32:0/2979171492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efd68029630 con 0x5567aff8bd70 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd6800d700 con 0x5567aff8bd70 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd5e000640 1 -- 172.21.15.32:0/2979171492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd68029830 con 0x5567aff8bd70 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 10 client.7395 fetch_fsmap learned FSMap version 67 2024-03-06T07:34:30.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x5567aff7f9b0 con 0x5567aff8bd70 2024-03-06T07:34:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd5e000640 1 -- 172.21.15.32:0/2979171492 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6800d700 con 0x5567aff8bd70 2024-03-06T07:34:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd5e000640 1 -- 172.21.15.32:0/2979171492 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7efd68028070 con 0x5567aff8bd70 2024-03-06T07:34:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 10 client.7395 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:34:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 10 client.7395 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:34:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 20 client.7395 populate_metadata read hostname 'smithi032' 2024-03-06T07:34:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567b0018fd0 0x5567b00193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 4 client.7395 mds_command: new command op to 7344 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.928+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 --> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5567afecfe90 con 0x5567b0018fd0 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.929+0000 7efd74e00640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567b0018fd0 0x5567b00193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.929+0000 7efd74e00640 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567b0018fd0 0x5567b00193b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daca9fa4ee4a197 server_cookie=808806ffc2e913bd in_seq=0 out_seq=0 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.929+0000 7efd5e000640 10 client.7395 ms_handle_connect on v2:172.21.15.32:6834/1229875104 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd5e000640 1 -- 172.21.15.32:0/2979171492 <== mds.0 v2:172.21.15.32:6834/1229875104 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5567afecfe90 con 0x5567b0018fd0 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd5e000640 10 client.7395 handle_command_reply: tid=0 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 client.7395 shutdown 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 msgr2=0x5567aff7ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7ec40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7efd680011e0 tx=0x7efd68000ed0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 shutdown_connections 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:6834/1229875104,v1:172.21.15.32:6835/1229875104] conn(0x5567b0018fd0 0x5567b00193b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567aff8bd70 0x5567aff7ec40 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567aff8b990 0x5567affab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 --2- 172.21.15.32:0/2979171492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5567aff800c0 0x5567affaaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd5e000640 0 client.7395 ms_handle_reset on v2:172.21.15.32:6834/1229875104 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 >> 172.21.15.32:0/2979171492 conn(0x5567aff5ee70 msgr2=0x5567affa7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 shutdown_connections 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 1 -- 172.21.15.32:0/2979171492 wait complete. 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.930+0000 7efd7a5fd740 20 client.7395 trim_cache size 0 max 16384 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7344, 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:34:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.1790112270000002, 2024-03-06T07:34:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 67, 2024-03-06T07:34:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 99, 2024-03-06T07:34:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 99, 2024-03-06T07:34:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 5.2440777020000002 2024-03-06T07:34:30.935 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:34:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd4c06b4d0 msgr2=0x7efd4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd4c06b4d0 0x7efd4c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7efd701f7bd0 tx=0x7efd68009040 comp rx=0 tx=0).stop 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 msgr2=0x7efd700cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700cdae0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7efd580098c0 tx=0x7efd580046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 shutdown_connections 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd4c06b4d0 0x7efd4c06d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd700ceed0 0x7efd700ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd700b3ea0 0x7efd700cdae0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 --2- 172.21.15.32:0/2363042125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd700727a0 0x7efd70146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 >> 172.21.15.32:0/2363042125 conn(0x7efd700b1990 msgr2=0x7efd700cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 shutdown_connections 2024-03-06T07:34:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:30.931+0000 7efd76200640 1 -- 172.21.15.32:0/2363042125 wait complete. 2024-03-06T07:34:30.956 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:34:30.956 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:34:30.956 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:34:30.956 DEBUG:teuthology.orchestra.run.smithi032:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:34:30.956 DEBUG:teuthology.orchestra.run.smithi032:> subdir = "delete_me" 2024-03-06T07:34:30.956 DEBUG:teuthology.orchestra.run.smithi032:> size_unit = 1024 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> file_multiplier = 200 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> for i in range(0, file_multiplier): 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> f.write(size * '"'"'x'"'"') 2024-03-06T07:34:30.957 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:34:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:34:35.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_ops 2024-03-06T07:34:35.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:35.449 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:34:35.726 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:34:35.727 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:34:35.727 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json config get mds_max_purge_files 2024-03-06T07:34:35.961 INFO:teuthology.orchestra.run.smithi039.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:34:35.961 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:34:35.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.222+0000 7f58d8c00640 1 Processor -- start 2024-03-06T07:34:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.223+0000 7f58d8c00640 1 -- start start 2024-03-06T07:34:36.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.224+0000 7f58d8c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d41457e0 0x7f58d40cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.224+0000 7f58d8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58d40b5870 0x7f58d40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.224+0000 7f58d8c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d4145bc0 con 0x7f58d415fba0 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.224+0000 7f58d8c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d40cfb20 con 0x7f58d40b5870 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.224+0000 7f58d8c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d4160040 con 0x7f58d41457e0 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58cbe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58d415fba0 0x7f58d40b5160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57532/0 (socket says 172.21.15.32:57532) 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58cbe00640 1 -- 172.21.15.32:0/3002547581 learned_addr learned my addr 172.21.15.32:0/3002547581 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d2800640 1 --2- 172.21.15.32:0/3002547581 >> v2:172.21.15.32:3300/0 conn(0x7f58d40b5870 0x7f58d40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d1e00640 1 --2- 172.21.15.32:0/3002547581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d41457e0 0x7f58d40cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d2800640 1 -- 172.21.15.32:0/3002547581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d41457e0 msgr2=0x7f58d40cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d2800640 1 --2- 172.21.15.32:0/3002547581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d41457e0 0x7f58d40cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d2800640 1 -- 172.21.15.32:0/3002547581 >> v1:172.21.15.32:6789/0 conn(0x7f58d415fba0 legacy=0x7f58d40b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d2800640 1 -- 172.21.15.32:0/3002547581 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4160320 con 0x7f58d40b5870 2024-03-06T07:34:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.225+0000 7f58d1400640 1 -- 172.21.15.32:0/3002547581 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723190084 0 0) 0x7f58d4145bc0 con 0x7f58d415fba0 2024-03-06T07:34:36.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.227+0000 7f58d2800640 1 --2- 172.21.15.32:0/3002547581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d40710b0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009d90 tx=0x7f58cc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63e3b0d0638877a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.227+0000 7f58d1400640 1 -- 172.21.15.32:0/3002547581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc039070 con 0x7f58d40b5870 2024-03-06T07:34:36.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.227+0000 7f58d1400640 1 -- 172.21.15.32:0/3002547581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc02cbc0 con 0x7f58d40b5870 2024-03-06T07:34:36.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.227+0000 7f58d1400640 1 -- 172.21.15.32:0/3002547581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc034440 con 0x7f58d40b5870 2024-03-06T07:34:36.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 -- 172.21.15.32:0/3002547581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 msgr2=0x7f58d40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 --2- 172.21.15.32:0/3002547581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d40710b0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009d90 tx=0x7f58cc02be10 comp rx=0 tx=0).stop 2024-03-06T07:34:36.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 -- 172.21.15.32:0/3002547581 shutdown_connections 2024-03-06T07:34:36.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 --2- 172.21.15.32:0/3002547581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d40710b0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 --2- 172.21.15.32:0/3002547581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d41457e0 0x7f58d40cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 -- 172.21.15.32:0/3002547581 >> 172.21.15.32:0/3002547581 conn(0x7f58d406cf40 msgr2=0x7f58d406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 -- 172.21.15.32:0/3002547581 shutdown_connections 2024-03-06T07:34:36.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.228+0000 7f58d8c00640 1 -- 172.21.15.32:0/3002547581 wait complete. 2024-03-06T07:34:36.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.231+0000 7f58d8c00640 1 Processor -- start 2024-03-06T07:34:36.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.231+0000 7f58d8c00640 1 -- start start 2024-03-06T07:34:36.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d41408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d41457e0 0x7f58d413eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d8c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d415fba0 0x7f58d415afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d8c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40cca70 con 0x7f58d40b5870 2024-03-06T07:34:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d8c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d41608e0 con 0x7f58d41457e0 2024-03-06T07:34:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d8c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40715f0 con 0x7f58d415fba0 2024-03-06T07:34:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d41408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d41408f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59364/0 (socket says 172.21.15.32:59364) 2024-03-06T07:34:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.233+0000 7f58d1e00640 1 -- 172.21.15.32:0/2749120013 learned_addr learned my addr 172.21.15.32:0/2749120013 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d2800640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d415fba0 0x7f58d415afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58cbe00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d41457e0 0x7f58d413eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d1e00640 1 -- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d415fba0 msgr2=0x7f58d415afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d1e00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d415fba0 0x7f58d415afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d1e00640 1 -- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d41457e0 msgr2=0x7f58d413eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d1e00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d41457e0 0x7f58d413eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d1e00640 1 -- 172.21.15.32:0/2749120013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d41f3cb0 con 0x7f58d40b5870 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58d1e00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d41408f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f58c4007f40 tx=0x7f58c400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.234+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c400b8c0 con 0x7f58d40b5870 2024-03-06T07:34:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.235+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c400ba60 con 0x7f58d40b5870 2024-03-06T07:34:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.235+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c400be10 con 0x7f58d40b5870 2024-03-06T07:34:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.235+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d41f3f90 con 0x7f58d40b5870 2024-03-06T07:34:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.235+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d41f4550 con 0x7f58d40b5870 2024-03-06T07:34:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.236+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d40692a0 con 0x7f58d40b5870 2024-03-06T07:34:36.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.237+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58c40125e0 con 0x7f58d40b5870 2024-03-06T07:34:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.238+0000 7f58ca000640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.239+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f58c408e630 con 0x7f58d40b5870 2024-03-06T07:34:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.239+0000 7f58cbe00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.240+0000 7f58cbe00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58b006b4d0 0x7f58b006d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f58d40cc2a0 tx=0x7f58bc007c00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.243+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58c405d440 con 0x7f58d40b5870 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.694+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58d40725f0 con 0x7f58d40b5870 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.696+0000 7f58ca000640 1 -- 172.21.15.32:0/2749120013 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f58c405cd40 con 0x7f58d40b5870 2024-03-06T07:34:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:36.700 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:36.701 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.702+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58b006b4d0 msgr2=0x7f58b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.703+0000 7f58d8c00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58b006b4d0 0x7f58b006d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f58d40cc2a0 tx=0x7f58bc007c00 comp rx=0 tx=0).stop 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.703+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 msgr2=0x7f58d41408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.703+0000 7f58d8c00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d41408f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f58c4007f40 tx=0x7f58c400dbe0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.704+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 shutdown_connections 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.704+0000 7f58d8c00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.704+0000 7f58d8c00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f58d415fba0 0x7f58d415afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.704+0000 7f58d8c00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d41457e0 0x7f58d413eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.704+0000 7f58d8c00640 1 --2- 172.21.15.32:0/2749120013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d40b5870 0x7f58d41408f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.704+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 >> 172.21.15.32:0/2749120013 conn(0x7f58d406cf40 msgr2=0x7f58d4071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.705+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 shutdown_connections 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:36.705+0000 7f58d8c00640 1 -- 172.21.15.32:0/2749120013 wait complete. 2024-03-06T07:34:36.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:36.978 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":34,"msgr_send_messages":31,"msgr_recv_bytes":111943,"msgr_send_bytes":4227410,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.029590725,"msgr_running_send_time":0.014352836,"msgr_running_recv_time":0.011219033,"msgr_running_fast_dispatch_time":0.003915881,"msgr_send_messages_queue_lat":{"avgcount":30,"sum":0.009724119,"avgtime":0.000324137},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":104992,"msgr_send_encrypted_bytes":6048},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6924,"msgr_send_messages":12059,"msgr_recv_bytes":2501171,"msgr_send_bytes":11723264,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.480722033,"msgr_running_send_time":1.014604739,"msgr_running_recv_time":0.957206218,"msgr_running_fast_dispatch_time":0.009419753,"msgr_send_messages_queue_lat":{"avgcount":12058,"sum":0.952178665,"avgtime":0.000078966},"msgr_handle_ack_lat":{"avgcount":14261,"sum":0.007754689,"avgtime":0.000000543},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92704},"AsyncMessenger::Worker-2":{"msgr_recv_messages":12,"msgr_send_messages":14,"msgr_recv_bytes":10350,"msgr_send_bytes":4419,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.005640185,"msgr_running_send_time":0.000632560,"msgr_running_recv_time":0.004204886,"msgr_running_fast_dispatch_time":0.000452279,"msgr_send_messages_queue_lat":{"avgcount":13,"sum":0.009776346,"avgtime":0.000752026},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1882,"complete_latency":{"avgcount":64,"sum":1.879295629,"avgtime":0.029363994}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000126573,"avgtime":0.000031643}},"mds":{"request":3204,"reply":3203,"reply_latency":{"avgcount":3203,"sum":655.206360354,"avgtime":0.204560212},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3206,"traverse_hit":3206,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2848,"root_rbytes":19239936,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3009,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3287,"handle_client_caps_dirty":2994,"handle_client_cap_release":0,"process_request_cap_release":3},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6494,"evex":0,"evtrm":0,"ev":6499,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17765827,"rdpos":4195674,"jlat":{"avgcount":8,"sum":0.808636167,"avgtime":0.101079520},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3211,"dn+":3211,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":108892,"heap":248164},"mds_server":{"dispatch_client_request":3204,"dispatch_server_request":0,"handle_client_request":3204,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3113,"average_load":3113,"avg_session_uptime":1160,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":23215858,"buffer_anon_items":11001,"buffer_meta_bytes":97416,"buffer_meta_items":1107,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13738115,"mds_co_items":131648,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":62,"op_send_bytes":13266197,"op_resend":0,"op_reply":62,"op_latency":{"avgcount":62,"sum":0.377992205,"avgtime":0.006096648},"op_inflight":0,"oplen_avg":{"avgcount":62,"sum":113},"op":62,"op_r":36,"op_w":26,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":10,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":0},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":102108,"max":104857600,"get_started":0,"get":6966,"get_sum":1993999,"get_or_fail_fail":0,"get_or_fail_success":6966,"take":0,"take_sum":0,"put":6585,"put_sum":1891891,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":13268118,"put":62,"put_sum":13268118,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":62,"put":62,"put_sum":62,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":599516,"max":3758096384,"get_started":0,"get":6495,"get_sum":13571375,"get_or_fail_fail":0,"get_or_fail_success":6495,"take":0,"take_sum":0,"put":10,"put_sum":12971859,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:36.979 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 111943, "msgr_recv_encrypted_bytes": 104992, "msgr_recv_messages": 34, "msgr_running_fast_dispatch_time": 0.003915881, "msgr_running_recv_time": 0.011219033, "msgr_running_send_time": 0.014352836, "msgr_running_total_time": 0.029590725, "msgr_send_bytes": 4227410, "msgr_send_encrypted_bytes": 6048, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 30, "avgtime": 0.000324137, "sum": 0.009724119 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 14261, "avgtime": 5.43e-07, "sum": 0.007754689 }, "msgr_recv_bytes": 2501171, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 6924, "msgr_running_fast_dispatch_time": 0.009419753, "msgr_running_recv_time": 0.957206218, "msgr_running_send_time": 1.014604739, "msgr_running_total_time": 1.480722033, "msgr_send_bytes": 11723264, "msgr_send_encrypted_bytes": 92704, "msgr_send_messages": 12059, "msgr_send_messages_queue_lat": { "avgcount": 12058, "avgtime": 7.8966e-05, "sum": 0.952178665 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 10350, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 12, "msgr_running_fast_dispatch_time": 0.000452279, "msgr_running_recv_time": 0.004204886, "msgr_running_send_time": 0.00063256, "msgr_running_total_time": 0.005640185, "msgr_send_bytes": 4419, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 14, "msgr_send_messages_queue_lat": { "avgcount": 13, "avgtime": 0.000752026, "sum": 0.009776346 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 64, "avgtime": 0.029363994, "sum": 1.879295629 }, "queue_len": 1882 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.1643e-05, "sum": 0.000126573 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3009, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3287, "handle_client_caps_dirty": 2994, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3, "q": 0, "reply": 3203, "reply_latency": { "avgcount": 3203, "avgtime": 0.204560212, "sum": 655.206360354 }, "request": 3204, "root_rbytes": 19239936, "root_rfiles": 2848, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3206, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3206, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6499, "evadd": 6494, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 8, "avgtime": 0.10107952, "sum": 0.808636167 }, "rdpos": 4195674, "replayed": 5, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17765827 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3211, "dn+": 3211, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 108892 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3204, "dispatch_server_request": 0, "handle_client_request": 3204, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3113, "avg_session_uptime": 1160, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3113 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 23215858, "buffer_anon_items": 11001, "buffer_meta_bytes": 97416, "buffer_meta_items": 1107, "mds_co_bytes": 13738115, "mds_co_items": 131648, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 0, "omap_rd": 48, "omap_wr": 11, "op": 62, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 62, "avgtime": 0.006096648, "sum": 0.377992205 }, "op_pg": 0, "op_r": 36, "op_reply": 62, "op_resend": 0, "op_rmw": 0, "op_send": 62, "op_send_bytes": 13266197, "op_w": 26, "oplen_avg": { "avgcount": 62, "sum": 113 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 10, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6966, "get_or_fail_fail": 0, "get_or_fail_success": 6966, "get_started": 0, "get_sum": 1993999, "max": 104857600, "put": 6585, "put_sum": 1891891, "take": 0, "take_sum": 0, "val": 102108, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 62, "put_sum": 13268118, "take": 62, "take_sum": 13268118, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 62, "put_sum": 62, "take": 62, "take_sum": 62, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 6495, "get_or_fail_fail": 0, "get_or_fail_success": 6495, "get_started": 0, "get_sum": 13571375, "max": 3758096384, "put": 10, "put_sum": 12971859, "take": 0, "take_sum": 0, "val": 599516, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:36.980 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/0, 0/0 2024-03-06T07:34:37.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.194+0000 7fc19f000640 1 Processor -- start 2024-03-06T07:34:38.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.195+0000 7fc19f000640 1 -- start start 2024-03-06T07:34:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.197+0000 7fc19f000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc19815c160 0x7fc198072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.197+0000 7fc19f000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19806b6d0 0x7fc198162030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.197+0000 7fc19f000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc198071e00 con 0x7fc19815bd80 2024-03-06T07:34:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.197+0000 7fc19f000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1980730e0 con 0x7fc19815c160 2024-03-06T07:34:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.197+0000 7fc19f000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc198162d80 con 0x7fc19806b6d0 2024-03-06T07:34:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.198+0000 7fc197e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc19815bd80 0x7fc19806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:33862/0 (socket says 172.21.15.32:33862) 2024-03-06T07:34:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.198+0000 7fc197e00640 1 -- 172.21.15.32:0/3110558962 learned_addr learned my addr 172.21.15.32:0/3110558962 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.198+0000 7fc19d000640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19806b6d0 0x7fc198162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.198+0000 7fc18fe00640 1 --2- 172.21.15.32:0/3110558962 >> v2:172.21.15.39:3300/0 conn(0x7fc19815c160 0x7fc198072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc197400640 1 -- 172.21.15.32:0/3110558962 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290161173 0 0) 0x7fc198071e00 con 0x7fc19815bd80 2024-03-06T07:34:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc197400640 1 -- 172.21.15.32:0/3110558962 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc17c003520 con 0x7fc19815bd80 2024-03-06T07:34:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc18fe00640 1 -- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19806b6d0 msgr2=0x7fc198162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc18fe00640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19806b6d0 0x7fc198162030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc18fe00640 1 -- 172.21.15.32:0/3110558962 >> v1:172.21.15.39:6789/0 conn(0x7fc19815bd80 legacy=0x7fc19806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc18fe00640 1 -- 172.21.15.32:0/3110558962 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1980720c0 con 0x7fc19815c160 2024-03-06T07:34:38.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.199+0000 7fc19d000640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19806b6d0 0x7fc198162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:38.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.200+0000 7fc18fe00640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19815c160 0x7fc198072ba0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc1800097a0 tx=0x7fc18002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4638f327f0e51031 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.201+0000 7fc197400640 1 -- 172.21.15.32:0/3110558962 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180035020 con 0x7fc19815c160 2024-03-06T07:34:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.201+0000 7fc197400640 1 -- 172.21.15.32:0/3110558962 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180031830 con 0x7fc19815c160 2024-03-06T07:34:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.201+0000 7fc197400640 1 -- 172.21.15.32:0/3110558962 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180031be0 con 0x7fc19815c160 2024-03-06T07:34:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 -- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19815c160 msgr2=0x7fc198072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19815c160 0x7fc198072ba0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc1800097a0 tx=0x7fc18002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 -- 172.21.15.32:0/3110558962 shutdown_connections 2024-03-06T07:34:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19806b6d0 0x7fc198162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 --2- 172.21.15.32:0/3110558962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19815c160 0x7fc198072ba0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 -- 172.21.15.32:0/3110558962 >> 172.21.15.32:0/3110558962 conn(0x7fc19806c0d0 msgr2=0x7fc19806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 -- 172.21.15.32:0/3110558962 shutdown_connections 2024-03-06T07:34:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.202+0000 7fc19f000640 1 -- 172.21.15.32:0/3110558962 wait complete. 2024-03-06T07:34:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.205+0000 7fc19f000640 1 Processor -- start 2024-03-06T07:34:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.205+0000 7fc19f000640 1 -- start start 2024-03-06T07:34:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.206+0000 7fc19f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19806b6d0 0x7fc1980b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19f000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19815bd80 0x7fc19815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 0x7fc1980b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc198162570 con 0x7fc19815c160 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc198072680 con 0x7fc19806b6d0 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19f000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1981626f0 con 0x7fc19815bd80 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 0x7fc1980b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 0x7fc1980b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55272/0 (socket says 172.21.15.32:55272) 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 -- 172.21.15.32:0/1005299069 learned_addr learned my addr 172.21.15.32:0/1005299069 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc18fe00640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19815bd80 0x7fc19815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc197e00640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19806b6d0 0x7fc1980b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 -- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19815bd80 msgr2=0x7fc19815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19815bd80 0x7fc19815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 -- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19806b6d0 msgr2=0x7fc1980b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.207+0000 7fc19d000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19806b6d0 0x7fc1980b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc19d000640 1 -- 172.21.15.32:0/1005299069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19815d9f0 con 0x7fc19815c160 2024-03-06T07:34:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc19d000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 0x7fc1980b0440 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc19000efc0 tx=0x7fc190009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190018030 con 0x7fc19815c160 2024-03-06T07:34:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19000cde0 con 0x7fc19815c160 2024-03-06T07:34:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19815dcd0 con 0x7fc19815c160 2024-03-06T07:34:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190003bb0 con 0x7fc19815c160 2024-03-06T07:34:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.208+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc198145a00 con 0x7fc19815c160 2024-03-06T07:34:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.210+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc198145510 con 0x7fc19815c160 2024-03-06T07:34:38.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.211+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc190005d10 con 0x7fc19815c160 2024-03-06T07:34:38.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.213+0000 7fc194c00640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc17406b500 0x7fc17406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.213+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc19008acf0 con 0x7fc19815c160 2024-03-06T07:34:38.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.213+0000 7fc197e00640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc17406b500 0x7fc17406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.214+0000 7fc197e00640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc17406b500 0x7fc17406d9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc1880011e0 tx=0x7fc188009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.217+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc190022070 con 0x7fc19815c160 2024-03-06T07:34:38.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.509+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1980d18d0 con 0x7fc19815c160 2024-03-06T07:34:38.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.510+0000 7fc194c00640 1 -- 172.21.15.32:0/1005299069 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc190010040 con 0x7fc19815c160 2024-03-06T07:34:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:38.512 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:38.512 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc17406b500 msgr2=0x7fc17406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc17406b500 0x7fc17406d9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc1880011e0 tx=0x7fc188009040 comp rx=0 tx=0).stop 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 msgr2=0x7fc1980b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 0x7fc1980b0440 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc19000efc0 tx=0x7fc190009f90 comp rx=0 tx=0).stop 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 shutdown_connections 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc17406b500 0x7fc17406d9c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc19815c160 0x7fc1980b0440 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc19815bd80 0x7fc19815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 --2- 172.21.15.32:0/1005299069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19806b6d0 0x7fc1980b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.513+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 >> 172.21.15.32:0/1005299069 conn(0x7fc19806c0d0 msgr2=0x7fc198156c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:38.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.514+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 shutdown_connections 2024-03-06T07:34:38.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:38.514+0000 7fc19f000640 1 -- 172.21.15.32:0/1005299069 wait complete. 2024-03-06T07:34:38.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:38.784 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":34,"msgr_send_messages":31,"msgr_recv_bytes":111943,"msgr_send_bytes":4227410,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.029676329,"msgr_running_send_time":0.014389451,"msgr_running_recv_time":0.011242179,"msgr_running_fast_dispatch_time":0.003915881,"msgr_send_messages_queue_lat":{"avgcount":30,"sum":0.009724119,"avgtime":0.000324137},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":104992,"msgr_send_encrypted_bytes":6048},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7779,"msgr_send_messages":13594,"msgr_recv_bytes":2794231,"msgr_send_bytes":18035636,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.732968369,"msgr_running_send_time":1.165715608,"msgr_running_recv_time":1.078598213,"msgr_running_fast_dispatch_time":0.011931717,"msgr_send_messages_queue_lat":{"avgcount":13593,"sum":1.031895996,"avgtime":0.000075913},"msgr_handle_ack_lat":{"avgcount":16596,"sum":0.008543234,"avgtime":0.000000514},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92704},"AsyncMessenger::Worker-2":{"msgr_recv_messages":13,"msgr_send_messages":15,"msgr_recv_bytes":10596,"msgr_send_bytes":4737,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.005805866,"msgr_running_send_time":0.000698246,"msgr_running_recv_time":0.004240639,"msgr_running_fast_dispatch_time":0.000503998,"msgr_send_messages_queue_lat":{"avgcount":14,"sum":0.009832015,"avgtime":0.000702286},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1477,"complete_latency":{"avgcount":70,"sum":3.514087150,"avgtime":0.050201245}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000126573,"avgtime":0.000031643}},"mds":{"request":4056,"reply":4056,"reply_latency":{"avgcount":4056,"sum":657.796968065,"avgtime":0.162178739},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4058,"traverse_hit":4058,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2848,"root_rbytes":19239936,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3298,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3668,"handle_client_caps_dirty":2996,"handle_client_cap_release":0,"process_request_cap_release":855},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":847,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":847,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7841,"evex":0,"evtrm":0,"ev":7846,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":21452868,"rdpos":4195674,"jlat":{"avgcount":11,"sum":3.041877755,"avgtime":0.276534341},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4058,"dn+":4058,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":109276,"heap":248164},"mds_server":{"dispatch_client_request":4057,"dispatch_server_request":0,"handle_client_request":4056,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":847,"sum":1.038770684,"avgtime":0.001226411},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3857,"average_load":3857,"avg_session_uptime":1160,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":23215858,"buffer_anon_items":11001,"buffer_meta_bytes":97416,"buffer_meta_items":1107,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13738115,"mds_co_items":131648,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":65,"op_send_bytes":17146108,"op_resend":0,"op_reply":65,"op_latency":{"avgcount":65,"sum":0.458990536,"avgtime":0.007061392},"op_inflight":0,"oplen_avg":{"avgcount":65,"sum":118},"op":65,"op_r":36,"op_w":29,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":9,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":13,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":7823,"get_sum":2210517,"get_or_fail_fail":0,"get_or_fail_success":7823,"take":0,"take_sum":0,"put":7822,"put_sum":2210265,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":65,"take_sum":17148029,"put":65,"put_sum":17148029,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":65,"take_sum":65,"put":65,"put_sum":65,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":484702,"max":3758096384,"get_started":0,"get":7841,"get_sum":17259733,"get_or_fail_fail":0,"get_or_fail_success":7841,"take":0,"take_sum":0,"put":11,"put_sum":16775031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:38.786 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 111943, "msgr_recv_encrypted_bytes": 104992, "msgr_recv_messages": 34, "msgr_running_fast_dispatch_time": 0.003915881, "msgr_running_recv_time": 0.011242179, "msgr_running_send_time": 0.014389451, "msgr_running_total_time": 0.029676329, "msgr_send_bytes": 4227410, "msgr_send_encrypted_bytes": 6048, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 30, "avgtime": 0.000324137, "sum": 0.009724119 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 16596, "avgtime": 5.14e-07, "sum": 0.008543234 }, "msgr_recv_bytes": 2794231, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 7779, "msgr_running_fast_dispatch_time": 0.011931717, "msgr_running_recv_time": 1.078598213, "msgr_running_send_time": 1.165715608, "msgr_running_total_time": 1.732968369, "msgr_send_bytes": 18035636, "msgr_send_encrypted_bytes": 92704, "msgr_send_messages": 13594, "msgr_send_messages_queue_lat": { "avgcount": 13593, "avgtime": 7.5913e-05, "sum": 1.031895996 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 10596, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 13, "msgr_running_fast_dispatch_time": 0.000503998, "msgr_running_recv_time": 0.004240639, "msgr_running_send_time": 0.000698246, "msgr_running_total_time": 0.005805866, "msgr_send_bytes": 4737, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 15, "msgr_send_messages_queue_lat": { "avgcount": 14, "avgtime": 0.000702286, "sum": 0.009832015 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 70, "avgtime": 0.050201245, "sum": 3.51408715 }, "queue_len": 1477 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.1643e-05, "sum": 0.000126573 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3298, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3668, "handle_client_caps_dirty": 2996, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 855, "q": 0, "reply": 4056, "reply_latency": { "avgcount": 4056, "avgtime": 0.162178739, "sum": 657.796968065 }, "request": 4056, "root_rbytes": 19239936, "root_rfiles": 2848, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4058, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4058, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 847, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 847, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7846, "evadd": 7841, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.276534341, "sum": 3.041877755 }, "rdpos": 4195674, "replayed": 5, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 21452868 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4058, "dn+": 4058, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 109276 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4057, "dispatch_server_request": 0, "handle_client_request": 4056, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 847, "avgtime": 0.001226411, "sum": 1.038770684 } }, "mds_sessions": { "average_load": 3857, "avg_session_uptime": 1160, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3857 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 23215858, "buffer_anon_items": 11001, "buffer_meta_bytes": 97416, "buffer_meta_items": 1107, "mds_co_bytes": 13738115, "mds_co_items": 131648, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 13, "op": 65, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 65, "avgtime": 0.007061392, "sum": 0.458990536 }, "op_pg": 0, "op_r": 36, "op_reply": 65, "op_resend": 0, "op_rmw": 0, "op_send": 65, "op_send_bytes": 17146108, "op_w": 29, "oplen_avg": { "avgcount": 65, "sum": 118 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 9, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7823, "get_or_fail_fail": 0, "get_or_fail_success": 7823, "get_started": 0, "get_sum": 2210517, "max": 104857600, "put": 7822, "put_sum": 2210265, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 65, "put_sum": 17148029, "take": 65, "take_sum": 17148029, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 65, "put_sum": 65, "take": 65, "take_sum": 65, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 7841, "get_or_fail_fail": 0, "get_or_fail_success": 7841, "get_started": 0, "get_sum": 17259733, "max": 3758096384, "put": 11, "put_sum": 16775031, "take": 0, "take_sum": 0, "val": 484702, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:38.786 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/847, 0/847 2024-03-06T07:34:39.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.019+0000 7ff5bda00640 1 Processor -- start 2024-03-06T07:34:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.020+0000 7ff5bda00640 1 -- start start 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5bda00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff5b815f4c0 0x7ff5b80cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5bda00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5b80d4990 0x7ff5b80d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5bda00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b80d07f0 0x7ff5b80d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5bda00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b80d11a0 con 0x7ff5b80d4990 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5bda00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b815f8a0 con 0x7ff5b815f4c0 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5bda00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80cfe90 con 0x7ff5b80d07f0 2024-03-06T07:34:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5b7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5b80d4990 0x7ff5b80d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5b7400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5b80d4990 0x7ff5b80d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55288/0 (socket says 172.21.15.32:55288) 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.021+0000 7ff5b7400640 1 -- 172.21.15.32:0/1963121569 learned_addr learned my addr 172.21.15.32:0/1963121569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5b7e00640 1 --2- 172.21.15.32:0/1963121569 >> v2:172.21.15.39:3300/0 conn(0x7ff5b815f4c0 0x7ff5b80cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5bd000640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b80d07f0 0x7ff5b80d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5b7400640 1 -- 172.21.15.32:0/1963121569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b80d07f0 msgr2=0x7ff5b80d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5b7400640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b80d07f0 0x7ff5b80d0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5b7400640 1 -- 172.21.15.32:0/1963121569 >> v2:172.21.15.39:3300/0 conn(0x7ff5b815f4c0 msgr2=0x7ff5b80cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5b7400640 1 --2- 172.21.15.32:0/1963121569 >> v2:172.21.15.39:3300/0 conn(0x7ff5b815f4c0 0x7ff5b80cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5b7400640 1 -- 172.21.15.32:0/1963121569 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80d1ae0 con 0x7ff5b80d4990 2024-03-06T07:34:40.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.022+0000 7ff5bd000640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b80d07f0 0x7ff5b80d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.024+0000 7ff5b7400640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b80d28a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff5b808caa0 tx=0x7ff5a002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c2937871b79e0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.024+0000 7ff5b6a00640 1 -- 172.21.15.32:0/1963121569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00317d0 con 0x7ff5b80d4990 2024-03-06T07:34:40.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.024+0000 7ff5b6a00640 1 -- 172.21.15.32:0/1963121569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a0037070 con 0x7ff5b80d4990 2024-03-06T07:34:40.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.024+0000 7ff5b6a00640 1 -- 172.21.15.32:0/1963121569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0031b90 con 0x7ff5b80d4990 2024-03-06T07:34:40.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 -- 172.21.15.32:0/1963121569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 msgr2=0x7ff5b80d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b80d28a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff5b808caa0 tx=0x7ff5a002b870 comp rx=0 tx=0).stop 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 -- 172.21.15.32:0/1963121569 shutdown_connections 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b80d07f0 0x7ff5b80d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 --2- 172.21.15.32:0/1963121569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b80d28a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 --2- 172.21.15.32:0/1963121569 >> v2:172.21.15.39:3300/0 conn(0x7ff5b815f4c0 0x7ff5b80cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 -- 172.21.15.32:0/1963121569 >> 172.21.15.32:0/1963121569 conn(0x7ff5b80d3e70 msgr2=0x7ff5b80d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 -- 172.21.15.32:0/1963121569 shutdown_connections 2024-03-06T07:34:40.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.025+0000 7ff5bda00640 1 -- 172.21.15.32:0/1963121569 wait complete. 2024-03-06T07:34:40.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.028+0000 7ff5bda00640 1 Processor -- start 2024-03-06T07:34:40.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.028+0000 7ff5bda00640 1 -- start start 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80d07f0 0x7ff5b814d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bda00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b815f4c0 0x7ff5b81e7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80d2380 con 0x7ff5b80d4990 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80d20a0 con 0x7ff5b80d07f0 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bda00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80cfb50 con 0x7ff5b815f4c0 2024-03-06T07:34:40.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5b7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5b7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b815cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55294/0 (socket says 172.21.15.32:55294) 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5b7400640 1 -- 172.21.15.32:0/779594332 learned_addr learned my addr 172.21.15.32:0/779594332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5b7e00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80d07f0 0x7ff5b814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.030+0000 7ff5bd000640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b815f4c0 0x7ff5b81e7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5b7400640 1 -- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b815f4c0 msgr2=0x7ff5b81e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5b7400640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b815f4c0 0x7ff5b81e7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5b7400640 1 -- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80d07f0 msgr2=0x7ff5b814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5b7400640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80d07f0 0x7ff5b814d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5b7400640 1 -- 172.21.15.32:0/779594332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b815d5a0 con 0x7ff5b80d4990 2024-03-06T07:34:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5b7400640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b815cae0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff5a002b8b0 tx=0x7ff5a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0032870 con 0x7ff5b80d4990 2024-03-06T07:34:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.031+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a0032a30 con 0x7ff5b80d4990 2024-03-06T07:34:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.032+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0032e20 con 0x7ff5b80d4990 2024-03-06T07:34:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.032+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b815d880 con 0x7ff5b80d4990 2024-03-06T07:34:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.032+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b81e5180 con 0x7ff5b80d4990 2024-03-06T07:34:40.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.033+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b81e4c90 con 0x7ff5b80d4990 2024-03-06T07:34:40.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.034+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5a004c060 con 0x7ff5b80d4990 2024-03-06T07:34:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.036+0000 7ff5afe00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff59406b500 0x7ff59406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.036+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff5a00abda0 con 0x7ff5b80d4990 2024-03-06T07:34:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.036+0000 7ff5b7e00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff59406b500 0x7ff59406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.037+0000 7ff5b7e00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff59406b500 0x7ff59406d9c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009a60 tx=0x7ff5a8003990 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:40.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.041+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5a0040070 con 0x7ff5b80d4990 2024-03-06T07:34:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.410+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5b804f330 con 0x7ff5b80d4990 2024-03-06T07:34:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.412+0000 7ff5afe00640 1 -- 172.21.15.32:0/779594332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff5a007ab20 con 0x7ff5b80d4990 2024-03-06T07:34:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:40.414 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff59406b500 msgr2=0x7ff59406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff59406b500 0x7ff59406d9c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009a60 tx=0x7ff5a8003990 comp rx=0 tx=0).stop 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 msgr2=0x7ff5b815cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b815cae0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff5a002b8b0 tx=0x7ff5a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 shutdown_connections 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff59406b500 0x7ff59406d9c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5b815f4c0 0x7ff5b81e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80d4990 0x7ff5b815cae0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.415+0000 7ff5bda00640 1 --2- 172.21.15.32:0/779594332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80d07f0 0x7ff5b814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.416+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 >> 172.21.15.32:0/779594332 conn(0x7ff5b80d3e70 msgr2=0x7ff5b80ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:40.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.416+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 shutdown_connections 2024-03-06T07:34:40.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:40.416+0000 7ff5bda00640 1 -- 172.21.15.32:0/779594332 wait complete. 2024-03-06T07:34:40.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:40.702 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":35,"msgr_send_messages":32,"msgr_recv_bytes":112215,"msgr_send_bytes":4228034,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.029841976,"msgr_running_send_time":0.014476803,"msgr_running_recv_time":0.011280726,"msgr_running_fast_dispatch_time":0.003943819,"msgr_send_messages_queue_lat":{"avgcount":31,"sum":0.009762708,"avgtime":0.000314926},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":105264,"msgr_send_encrypted_bytes":6672},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8545,"msgr_send_messages":15709,"msgr_recv_bytes":3059212,"msgr_send_bytes":18816458,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.982189079,"msgr_running_send_time":1.311937459,"msgr_running_recv_time":1.201435585,"msgr_running_fast_dispatch_time":0.012064141,"msgr_send_messages_queue_lat":{"avgcount":15708,"sum":1.121652684,"avgtime":0.000071406},"msgr_handle_ack_lat":{"avgcount":18929,"sum":0.009759032,"avgtime":0.000000515},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92704},"AsyncMessenger::Worker-2":{"msgr_recv_messages":13,"msgr_send_messages":15,"msgr_recv_bytes":10596,"msgr_send_bytes":4737,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.005805866,"msgr_running_send_time":0.000698246,"msgr_running_recv_time":0.004240639,"msgr_running_fast_dispatch_time":0.000503998,"msgr_send_messages_queue_lat":{"avgcount":14,"sum":0.009832015,"avgtime":0.000702286},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1478,"complete_latency":{"avgcount":70,"sum":3.514087150,"avgtime":0.050201245}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000126573,"avgtime":0.000031643}},"mds":{"request":4813,"reply":4813,"reply_latency":{"avgcount":4813,"sum":659.447959606,"avgtime":0.137013912},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4814,"inodes_top":0,"inodes_bottom":4814,"inodes_pin_tail":0,"inodes_pinned":4814,"inodes_expired":0,"inodes_with_caps":3095,"caps":3095,"subtrees":2,"traverse":4815,"traverse_hit":4815,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":24534016,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":4159,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3671,"handle_client_caps_dirty":2999,"handle_client_cap_release":1,"process_request_cap_release":1612},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1604,"num_strays_delayed":107,"num_strays_enqueuing":0,"strays_created":1604,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8734,"evex":0,"evtrm":0,"ev":8739,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":24141578,"rdpos":4195674,"jlat":{"avgcount":11,"sum":3.041877755,"avgtime":0.276534341},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4815,"dn+":4815,"dn-":0,"cap":3095,"cap+":3202,"cap-":107,"rss":109404,"heap":248164},"mds_server":{"dispatch_client_request":4814,"dispatch_server_request":0,"handle_client_request":4814,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1604,"sum":2.689762225,"avgtime":0.001676909},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4521,"average_load":4521,"avg_session_uptime":1160,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":23215858,"buffer_anon_items":11001,"buffer_meta_bytes":97416,"buffer_meta_items":1107,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13738115,"mds_co_items":131648,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":66,"op_send_bytes":17146150,"op_resend":0,"op_reply":66,"op_latency":{"avgcount":66,"sum":0.460990494,"avgtime":0.006984704},"op_inflight":0,"oplen_avg":{"avgcount":66,"sum":119},"op":66,"op_r":36,"op_w":30,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":9,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":14,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":8589,"get_sum":2406436,"get_or_fail_fail":0,"get_or_fail_success":8589,"take":0,"take_sum":0,"put":8588,"put_sum":2406183,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":17148071,"put":66,"put_sum":17148071,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":66,"put":66,"put_sum":66,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":3170058,"max":3758096384,"get_started":0,"get":8734,"get_sum":19945089,"get_or_fail_fail":0,"get_or_fail_success":8734,"take":0,"take_sum":0,"put":11,"put_sum":16775031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:40.703 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 112215, "msgr_recv_encrypted_bytes": 105264, "msgr_recv_messages": 35, "msgr_running_fast_dispatch_time": 0.003943819, "msgr_running_recv_time": 0.011280726, "msgr_running_send_time": 0.014476803, "msgr_running_total_time": 0.029841976, "msgr_send_bytes": 4228034, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 32, "msgr_send_messages_queue_lat": { "avgcount": 31, "avgtime": 0.000314926, "sum": 0.009762708 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18929, "avgtime": 5.15e-07, "sum": 0.009759032 }, "msgr_recv_bytes": 3059212, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 8545, "msgr_running_fast_dispatch_time": 0.012064141, "msgr_running_recv_time": 1.201435585, "msgr_running_send_time": 1.311937459, "msgr_running_total_time": 1.982189079, "msgr_send_bytes": 18816458, "msgr_send_encrypted_bytes": 92704, "msgr_send_messages": 15709, "msgr_send_messages_queue_lat": { "avgcount": 15708, "avgtime": 7.1406e-05, "sum": 1.121652684 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 10596, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 13, "msgr_running_fast_dispatch_time": 0.000503998, "msgr_running_recv_time": 0.004240639, "msgr_running_send_time": 0.000698246, "msgr_running_total_time": 0.005805866, "msgr_send_bytes": 4737, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 15, "msgr_send_messages_queue_lat": { "avgcount": 14, "avgtime": 0.000702286, "sum": 0.009832015 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 70, "avgtime": 0.050201245, "sum": 3.51408715 }, "queue_len": 1478 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.1643e-05, "sum": 0.000126573 }, "queue_len": 0 }, "mds": { "caps": 3095, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4159, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 3671, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4814, "inodes_bottom": 4814, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4814, "inodes_top": 0, "inodes_with_caps": 3095, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1612, "q": 0, "reply": 4813, "reply_latency": { "avgcount": 4813, "avgtime": 0.137013912, "sum": 659.447959606 }, "request": 4813, "root_rbytes": 24534016, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4815, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4815, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1604, "num_strays_delayed": 107, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1604, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8739, "evadd": 8734, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.276534341, "sum": 3.041877755 }, "rdpos": 4195674, "replayed": 5, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 24141578 }, "mds_mem": { "cap": 3095, "cap+": 3202, "cap-": 107, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4815, "dn+": 4815, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 109404 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4814, "dispatch_server_request": 0, "handle_client_request": 4814, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1604, "avgtime": 0.001676909, "sum": 2.689762225 } }, "mds_sessions": { "average_load": 4521, "avg_session_uptime": 1160, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4521 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 23215858, "buffer_anon_items": 11001, "buffer_meta_bytes": 97416, "buffer_meta_items": 1107, "mds_co_bytes": 13738115, "mds_co_items": 131648, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 14, "op": 66, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 66, "avgtime": 0.006984704, "sum": 0.460990494 }, "op_pg": 0, "op_r": 36, "op_reply": 66, "op_resend": 0, "op_rmw": 0, "op_send": 66, "op_send_bytes": 17146150, "op_w": 30, "oplen_avg": { "avgcount": 66, "sum": 119 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 9, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8589, "get_or_fail_fail": 0, "get_or_fail_success": 8589, "get_started": 0, "get_sum": 2406436, "max": 104857600, "put": 8588, "put_sum": 2406183, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 66, "put_sum": 17148071, "take": 66, "take_sum": 17148071, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 66, "put_sum": 66, "take": 66, "take_sum": 66, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 8734, "get_or_fail_fail": 0, "get_or_fail_success": 8734, "get_started": 0, "get_sum": 19945089, "max": 3758096384, "put": 11, "put_sum": 16775031, "take": 0, "take_sum": 0, "val": 3170058, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:40.703 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1604, 0/1604 2024-03-06T07:34:41.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.988+0000 7fca29000640 1 Processor -- start 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.988+0000 7fca29000640 1 -- start start 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.989+0000 7fca29000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca2415e140 0x7fca2406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.989+0000 7fca29000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fca240b3f40 0x7fca2415e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.989+0000 7fca29000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca240ae4e0 con 0x7fca2415ea60 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.989+0000 7fca29000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2415eef0 con 0x7fca2415e140 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.989+0000 7fca29000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2415f1d0 con 0x7fca240b3f40 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca21e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca2415e140 0x7fca2406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca21e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca2415e140 0x7fca2406fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55300/0 (socket says 172.21.15.32:55300) 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca21e00640 1 -- 172.21.15.32:0/2763649213 learned_addr learned my addr 172.21.15.32:0/2763649213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca1be00640 1 --2- 172.21.15.32:0/2763649213 >> v2:172.21.15.39:3300/0 conn(0x7fca240b3f40 0x7fca2415e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca21e00640 1 -- 172.21.15.32:0/2763649213 >> v2:172.21.15.39:3300/0 conn(0x7fca240b3f40 msgr2=0x7fca2415e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca21e00640 1 --2- 172.21.15.32:0/2763649213 >> v2:172.21.15.39:3300/0 conn(0x7fca240b3f40 0x7fca2415e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.990+0000 7fca21e00640 1 -- 172.21.15.32:0/2763649213 >> v1:172.21.15.39:6789/0 conn(0x7fca2415ea60 legacy=0x7fca2406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.991+0000 7fca21e00640 1 -- 172.21.15.32:0/2763649213 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2415f4b0 con 0x7fca2415e140 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.992+0000 7fca21e00640 1 --2- 172.21.15.32:0/2763649213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2415e140 0x7fca2406fc80 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fca24073710 tx=0x7fca1402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856dc1df3bf17bde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.993+0000 7fca21400640 1 -- 172.21.15.32:0/2763649213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca140317c0 con 0x7fca2415e140 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.993+0000 7fca21400640 1 -- 172.21.15.32:0/2763649213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca14037070 con 0x7fca2415e140 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.993+0000 7fca21400640 1 -- 172.21.15.32:0/2763649213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14031b80 con 0x7fca2415e140 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.993+0000 7fca29000640 1 -- 172.21.15.32:0/2763649213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2415e140 msgr2=0x7fca2406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.993+0000 7fca29000640 1 --2- 172.21.15.32:0/2763649213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2415e140 0x7fca2406fc80 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fca24073710 tx=0x7fca1402b860 comp rx=0 tx=0).stop 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.994+0000 7fca29000640 1 -- 172.21.15.32:0/2763649213 shutdown_connections 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.994+0000 7fca29000640 1 --2- 172.21.15.32:0/2763649213 >> v2:172.21.15.39:3300/0 conn(0x7fca240b3f40 0x7fca2415e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.994+0000 7fca29000640 1 --2- 172.21.15.32:0/2763649213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2415e140 0x7fca2406fc80 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.994+0000 7fca29000640 1 -- 172.21.15.32:0/2763649213 >> 172.21.15.32:0/2763649213 conn(0x7fca240b2c40 msgr2=0x7fca240b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.994+0000 7fca29000640 1 -- 172.21.15.32:0/2763649213 shutdown_connections 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.994+0000 7fca29000640 1 -- 172.21.15.32:0/2763649213 wait complete. 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.996+0000 7fca29000640 1 Processor -- start 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.996+0000 7fca29000640 1 -- start start 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca29000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 0x7fca241e4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca29000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca2415e140 0x7fca24159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca29000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca240d0560 0x7fca2414b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca29000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2406b850 con 0x7fca240b3f40 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca29000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2406b6d0 con 0x7fca240d0560 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca29000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca240aeca0 con 0x7fca2415e140 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca21e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 0x7fca241e4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca21e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 0x7fca241e4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55316/0 (socket says 172.21.15.32:55316) 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.998+0000 7fca21e00640 1 -- 172.21.15.32:0/2004095860 learned_addr learned my addr 172.21.15.32:0/2004095860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca1be00640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca2415e140 0x7fca24159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca22800640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca240d0560 0x7fca2414b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca21e00640 1 -- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca2415e140 msgr2=0x7fca24159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca21e00640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca2415e140 0x7fca24159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca21e00640 1 -- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca240d0560 msgr2=0x7fca2414b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca21e00640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca240d0560 0x7fca2414b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca21e00640 1 -- 172.21.15.32:0/2004095860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca241e4b70 con 0x7fca240b3f40 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:41.999+0000 7fca21e00640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 0x7fca241e4630 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fca14000c00 tx=0x7fca140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.000+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14032860 con 0x7fca240b3f40 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.000+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca14032a00 con 0x7fca240b3f40 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.000+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca240cf4c0 con 0x7fca240b3f40 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.000+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14032df0 con 0x7fca240b3f40 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.001+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca240cfa80 con 0x7fca240b3f40 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.002+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca240692a0 con 0x7fca240b3f40 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.003+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca1404c060 con 0x7fca240b3f40 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.004+0000 7fca1a000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca0006b4d0 0x7fca0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.004+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fca140abbc0 con 0x7fca240b3f40 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.004+0000 7fca1be00640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca0006b4d0 0x7fca0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.005+0000 7fca1be00640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca0006b4d0 0x7fca0006d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fca2415a270 tx=0x7fca0c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.009+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca14040070 con 0x7fca240b3f40 2024-03-06T07:34:42.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.362+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca240b11a0 con 0x7fca240b3f40 2024-03-06T07:34:42.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.363+0000 7fca1a000640 1 -- 172.21.15.32:0/2004095860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fca14031e90 con 0x7fca240b3f40 2024-03-06T07:34:42.364 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:42.364 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:42.365 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca0006b4d0 msgr2=0x7fca0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca0006b4d0 0x7fca0006d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fca2415a270 tx=0x7fca0c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 msgr2=0x7fca241e4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 0x7fca241e4630 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fca14000c00 tx=0x7fca140046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 shutdown_connections 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fca0006b4d0 0x7fca0006d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca240d0560 0x7fca2414b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fca2415e140 0x7fca24159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 --2- 172.21.15.32:0/2004095860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca240b3f40 0x7fca241e4630 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.365+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 >> 172.21.15.32:0/2004095860 conn(0x7fca240b2c40 msgr2=0x7fca240af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.366+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 shutdown_connections 2024-03-06T07:34:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:42.366+0000 7fca29000640 1 -- 172.21.15.32:0/2004095860 wait complete. 2024-03-06T07:34:42.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:42.642 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":152,"msgr_send_messages":149,"msgr_recv_bytes":141917,"msgr_send_bytes":4266160,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.049793112,"msgr_running_send_time":0.023650665,"msgr_running_recv_time":0.014558634,"msgr_running_fast_dispatch_time":0.009589769,"msgr_send_messages_queue_lat":{"avgcount":148,"sum":0.030666772,"avgtime":0.000207207},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":105264,"msgr_send_encrypted_bytes":6672},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10574,"msgr_send_messages":18352,"msgr_recv_bytes":3743619,"msgr_send_bytes":23814976,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.288894698,"msgr_running_send_time":1.519582024,"msgr_running_recv_time":1.502510243,"msgr_running_fast_dispatch_time":0.019927625,"msgr_send_messages_queue_lat":{"avgcount":18351,"sum":1.453423941,"avgtime":0.000079201},"msgr_handle_ack_lat":{"avgcount":22279,"sum":0.011217812,"avgtime":0.000000503},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94496},"AsyncMessenger::Worker-2":{"msgr_recv_messages":222,"msgr_send_messages":224,"msgr_recv_bytes":108414,"msgr_send_bytes":163801,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.039846165,"msgr_running_send_time":0.016434746,"msgr_running_recv_time":0.011043718,"msgr_running_fast_dispatch_time":0.009437282,"msgr_send_messages_queue_lat":{"avgcount":223,"sum":0.043088087,"avgtime":0.000193220},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1065,"complete_latency":{"avgcount":72,"sum":6.228786164,"avgtime":0.086510918}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":739,"sum":0.916545077,"avgtime":0.001240250}},"mds":{"request":5093,"reply":5093,"reply_latency":{"avgcount":5093,"sum":660.287409851,"avgtime":0.129646065},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5095,"inodes_top":0,"inodes_bottom":5095,"inodes_pin_tail":0,"inodes_pinned":5095,"inodes_expired":0,"inodes_with_caps":2538,"caps":2538,"subtrees":2,"traverse":5095,"traverse_hit":5095,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2031,"root_rbytes":15708160,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":54,"ceph_cap_op_grant":5334,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4413,"handle_client_caps_dirty":3000,"handle_client_cap_release":10,"process_request_cap_release":1892},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1440,"num_strays_delayed":31,"num_strays_enqueuing":0,"strays_created":1884,"strays_enqueued":444,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10186,"evex":0,"evtrm":0,"ev":10191,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26754861,"rdpos":4195674,"jlat":{"avgcount":12,"sum":3.663958183,"avgtime":0.305329848},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":5095,"dn+":5095,"dn-":0,"cap":2538,"cap+":3202,"cap-":664,"rss":109916,"heap":248164},"mds_server":{"dispatch_client_request":5094,"dispatch_server_request":0,"handle_client_request":5093,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1884,"sum":3.529212470,"avgtime":0.001873255},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4744,"average_load":4744,"avg_session_uptime":1165,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":19945524,"buffer_anon_items":9211,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16067356,"mds_co_items":205948,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":533,"op_send_bytes":21937264,"op_resend":0,"op_reply":533,"op_latency":{"avgcount":533,"sum":4.050916523,"avgtime":0.007600218},"op_inflight":0,"oplen_avg":{"avgcount":533,"sum":588},"op":533,"op_r":41,"op_w":492,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":13,"osdop_write":19,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":460,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":16,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":444,"pq_executed":444,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":195249,"max":104857600,"get_started":0,"get":10944,"get_sum":3006413,"get_or_fail_fail":0,"get_or_fail_success":10944,"take":0,"take_sum":0,"put":10087,"put_sum":2811164,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":533,"take_sum":21984029,"put":533,"put_sum":21984029,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":533,"take_sum":533,"put":533,"put_sum":533,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":444,"get_sum":44844,"get_or_fail_fail":0,"get_or_fail_success":444,"take":0,"take_sum":0,"put":5,"put_sum":44844,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":1086928,"max":3758096384,"get_started":0,"get":10187,"get_sum":22560410,"get_or_fail_fail":0,"get_or_fail_success":10187,"take":0,"take_sum":0,"put":14,"put_sum":21473482,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:42.643 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 141917, "msgr_recv_encrypted_bytes": 105264, "msgr_recv_messages": 152, "msgr_running_fast_dispatch_time": 0.009589769, "msgr_running_recv_time": 0.014558634, "msgr_running_send_time": 0.023650665, "msgr_running_total_time": 0.049793112, "msgr_send_bytes": 4266160, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 149, "msgr_send_messages_queue_lat": { "avgcount": 148, "avgtime": 0.000207207, "sum": 0.030666772 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22279, "avgtime": 5.03e-07, "sum": 0.011217812 }, "msgr_recv_bytes": 3743619, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 10574, "msgr_running_fast_dispatch_time": 0.019927625, "msgr_running_recv_time": 1.502510243, "msgr_running_send_time": 1.519582024, "msgr_running_total_time": 2.288894698, "msgr_send_bytes": 23814976, "msgr_send_encrypted_bytes": 94496, "msgr_send_messages": 18352, "msgr_send_messages_queue_lat": { "avgcount": 18351, "avgtime": 7.9201e-05, "sum": 1.453423941 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 108414, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 222, "msgr_running_fast_dispatch_time": 0.009437282, "msgr_running_recv_time": 0.011043718, "msgr_running_send_time": 0.016434746, "msgr_running_total_time": 0.039846165, "msgr_send_bytes": 163801, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 224, "msgr_send_messages_queue_lat": { "avgcount": 223, "avgtime": 0.00019322, "sum": 0.043088087 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 72, "avgtime": 0.086510918, "sum": 6.228786164 }, "queue_len": 1065 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 739, "avgtime": 0.00124025, "sum": 0.916545077 }, "queue_len": 0 }, "mds": { "caps": 2538, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5334, "ceph_cap_op_revoke": 54, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 10, "handle_client_caps": 4413, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5095, "inodes_bottom": 5095, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5095, "inodes_top": 0, "inodes_with_caps": 2538, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1892, "q": 0, "reply": 5093, "reply_latency": { "avgcount": 5093, "avgtime": 0.129646065, "sum": 660.287409851 }, "request": 5093, "root_rbytes": 15708160, "root_rfiles": 2031, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5095, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5095, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1440, "num_strays_delayed": 31, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1884, "strays_enqueued": 444, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10191, "evadd": 10186, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.305329848, "sum": 3.663958183 }, "rdpos": 4195674, "replayed": 5, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26754861 }, "mds_mem": { "cap": 2538, "cap+": 3202, "cap-": 664, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5095, "dn+": 5095, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 109916 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5094, "dispatch_server_request": 0, "handle_client_request": 5093, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1884, "avgtime": 0.001873255, "sum": 3.52921247 } }, "mds_sessions": { "average_load": 4744, "avg_session_uptime": 1165, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4744 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 19945524, "buffer_anon_items": 9211, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 16067356, "mds_co_items": 205948, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 16, "op": 533, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 533, "avgtime": 0.007600218, "sum": 4.050916523 }, "op_pg": 0, "op_r": 41, "op_reply": 533, "op_resend": 0, "op_rmw": 0, "op_send": 533, "op_send_bytes": 21937264, "op_w": 492, "oplen_avg": { "avgcount": 533, "sum": 588 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 460, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 13, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 19, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 444, "pq_executed_ops": 444, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10944, "get_or_fail_fail": 0, "get_or_fail_success": 10944, "get_started": 0, "get_sum": 3006413, "max": 104857600, "put": 10087, "put_sum": 2811164, "take": 0, "take_sum": 0, "val": 195249, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 533, "put_sum": 21984029, "take": 533, "take_sum": 21984029, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 533, "put_sum": 533, "take": 533, "take_sum": 533, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 444, "get_or_fail_fail": 0, "get_or_fail_success": 444, "get_started": 0, "get_sum": 44844, "max": 3758096384, "put": 5, "put_sum": 44844, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 10187, "get_or_fail_fail": 0, "get_or_fail_success": 10187, "get_started": 0, "get_sum": 22560410, "max": 3758096384, "put": 14, "put_sum": 21473482, "take": 0, "take_sum": 0, "val": 1086928, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:42.643 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1440, 444/1884 2024-03-06T07:34:43.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:43.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.866+0000 7ffa31800640 1 Processor -- start 2024-03-06T07:34:43.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.867+0000 7ffa31800640 1 -- start start 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa31800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa2c15e350 0x7ffa2c0cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa31800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffa2c0cfc30 0x7ffa2c0d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa31800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0d1250 con 0x7ffa2c0d0e50 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa31800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2c0b7f30 con 0x7ffa2c15e350 2024-03-06T07:34:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa31800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2c15e730 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa2aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa2c15e350 0x7ffa2c0cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa2aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa2c15e350 0x7ffa2c0cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55318/0 (socket says 172.21.15.32:55318) 2024-03-06T07:34:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa2aa00640 1 -- 172.21.15.32:0/2154067396 learned_addr learned my addr 172.21.15.32:0/2154067396 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.869+0000 7ffa23e00640 1 --2- 172.21.15.32:0/2154067396 >> v2:172.21.15.39:3300/0 conn(0x7ffa2c0cfc30 0x7ffa2c0d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.870+0000 7ffa2aa00640 1 -- 172.21.15.32:0/2154067396 >> v2:172.21.15.39:3300/0 conn(0x7ffa2c0cfc30 msgr2=0x7ffa2c0d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.870+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/2154067396 >> v2:172.21.15.39:3300/0 conn(0x7ffa2c0cfc30 0x7ffa2c0d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.870+0000 7ffa2aa00640 1 -- 172.21.15.32:0/2154067396 >> v1:172.21.15.39:6789/0 conn(0x7ffa2c0d0e50 legacy=0x7ffa2c161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.870+0000 7ffa2aa00640 1 -- 172.21.15.32:0/2154067396 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0d1e20 con 0x7ffa2c15e350 2024-03-06T07:34:43.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.871+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/2154067396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c15e350 0x7ffa2c0cf6f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0b6c60 tx=0x7ffa1c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72ed761ebc53803 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.872+0000 7ffa2a000640 1 -- 172.21.15.32:0/2154067396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c034070 con 0x7ffa2c15e350 2024-03-06T07:34:43.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.872+0000 7ffa2a000640 1 -- 172.21.15.32:0/2154067396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c030860 con 0x7ffa2c15e350 2024-03-06T07:34:43.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.872+0000 7ffa2a000640 1 -- 172.21.15.32:0/2154067396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c030c30 con 0x7ffa2c15e350 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.872+0000 7ffa31800640 1 -- 172.21.15.32:0/2154067396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c15e350 msgr2=0x7ffa2c0cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.872+0000 7ffa31800640 1 --2- 172.21.15.32:0/2154067396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c15e350 0x7ffa2c0cf6f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0b6c60 tx=0x7ffa1c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.873+0000 7ffa31800640 1 -- 172.21.15.32:0/2154067396 shutdown_connections 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.873+0000 7ffa31800640 1 --2- 172.21.15.32:0/2154067396 >> v2:172.21.15.39:3300/0 conn(0x7ffa2c0cfc30 0x7ffa2c0d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.873+0000 7ffa31800640 1 --2- 172.21.15.32:0/2154067396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c15e350 0x7ffa2c0cf6f0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.873+0000 7ffa31800640 1 -- 172.21.15.32:0/2154067396 >> 172.21.15.32:0/2154067396 conn(0x7ffa2c09d3a0 msgr2=0x7ffa2c0d40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.873+0000 7ffa31800640 1 -- 172.21.15.32:0/2154067396 shutdown_connections 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.873+0000 7ffa31800640 1 -- 172.21.15.32:0/2154067396 wait complete. 2024-03-06T07:34:43.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.875+0000 7ffa31800640 1 Processor -- start 2024-03-06T07:34:43.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.875+0000 7ffa31800640 1 -- start start 2024-03-06T07:34:43.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa31800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 0x7ffa2c0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa31800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa2c0d0e50 0x7ffa2c160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa31800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffa2c15e350 0x7ffa2c15f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa31800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0d26c0 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa31800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0d23e0 con 0x7ffa2c0d0e50 2024-03-06T07:34:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa31800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0d29a0 con 0x7ffa2c15e350 2024-03-06T07:34:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 0x7ffa2c0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 0x7ffa2c0cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55330/0 (socket says 172.21.15.32:55330) 2024-03-06T07:34:43.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.876+0000 7ffa2aa00640 1 -- 172.21.15.32:0/442213534 learned_addr learned my addr 172.21.15.32:0/442213534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa23e00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa2c0d0e50 0x7ffa2c160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa2aa00640 1 -- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffa2c15e350 msgr2=0x7ffa2c15f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffa2c15e350 0x7ffa2c15f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa2aa00640 1 -- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa2c0d0e50 msgr2=0x7ffa2c160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa2c0d0e50 0x7ffa2c160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa2aa00640 1 -- 172.21.15.32:0/442213534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0c78f0 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 0x7ffa2c0cb660 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c02b9e0 tx=0x7ffa1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c034040 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c039d90 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c043610 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.877+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c0c7bd0 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.878+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c0c8190 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.879+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c0692a0 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.880+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa1c04a060 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.881+0000 7ffa22a00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa0806b500 0x7ffa0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.881+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffa1c0ac2b0 con 0x7ffa2c0cfc30 2024-03-06T07:34:43.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.881+0000 7ffa23e00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa0806b500 0x7ffa0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.882+0000 7ffa23e00640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa0806b500 0x7ffa0806d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0d4660 tx=0x7ffa14008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:43.887+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa1c03e070 con 0x7ffa2c0cfc30 2024-03-06T07:34:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.260+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa2c0d1680 con 0x7ffa2c0cfc30 2024-03-06T07:34:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.261+0000 7ffa22a00640 1 -- 172.21.15.32:0/442213534 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ffa1c037360 con 0x7ffa2c0cfc30 2024-03-06T07:34:44.263 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:44.264 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa0806b500 msgr2=0x7ffa0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa0806b500 0x7ffa0806d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0d4660 tx=0x7ffa14008040 comp rx=0 tx=0).stop 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 msgr2=0x7ffa2c0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 0x7ffa2c0cb660 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c02b9e0 tx=0x7ffa1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 shutdown_connections 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa0806b500 0x7ffa0806d9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffa2c15e350 0x7ffa2c15f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa2c0d0e50 0x7ffa2c160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 --2- 172.21.15.32:0/442213534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c0cfc30 0x7ffa2c0cb660 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 >> 172.21.15.32:0/442213534 conn(0x7ffa2c09d3a0 msgr2=0x7ffa2c0b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.264+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 shutdown_connections 2024-03-06T07:34:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:44.265+0000 7ffa31800640 1 -- 172.21.15.32:0/442213534 wait complete. 2024-03-06T07:34:44.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:44.572 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":161,"msgr_send_messages":158,"msgr_recv_bytes":144221,"msgr_send_bytes":4269392,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.050679232,"msgr_running_send_time":0.024066590,"msgr_running_recv_time":0.014759951,"msgr_running_fast_dispatch_time":0.009786018,"msgr_send_messages_queue_lat":{"avgcount":157,"sum":0.031155459,"avgtime":0.000198442},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":105536,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11304,"msgr_send_messages":19885,"msgr_recv_bytes":3939702,"msgr_send_bytes":24406955,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.497043757,"msgr_running_send_time":1.643227307,"msgr_running_recv_time":1.611660161,"msgr_running_fast_dispatch_time":0.020638769,"msgr_send_messages_queue_lat":{"avgcount":19884,"sum":1.523953423,"avgtime":0.000076642},"msgr_handle_ack_lat":{"avgcount":24119,"sum":0.011833950,"avgtime":0.000000490},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94496},"AsyncMessenger::Worker-2":{"msgr_recv_messages":256,"msgr_send_messages":258,"msgr_recv_bytes":226717,"msgr_send_bytes":3653216,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.059186745,"msgr_running_send_time":0.030274703,"msgr_running_recv_time":0.012339967,"msgr_running_fast_dispatch_time":0.013376332,"msgr_send_messages_queue_lat":{"avgcount":257,"sum":0.059962951,"avgtime":0.000233318},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1365,"complete_latency":{"avgcount":81,"sum":9.303417929,"avgtime":0.114857011}},"finisher-PurgeQueue":{"queue_len":989,"complete_latency":{"avgcount":818,"sum":1.098140674,"avgtime":0.001342470}},"mds":{"request":5524,"reply":5523,"reply_latency":{"avgcount":5523,"sum":662.964493885,"avgtime":0.120037025},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4819,"inodes_top":0,"inodes_bottom":4819,"inodes_pin_tail":0,"inodes_pinned":4819,"inodes_expired":0,"inodes_with_caps":1928,"caps":1928,"subtrees":2,"traverse":5526,"traverse_hit":5526,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1687,"root_rbytes":13005824,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":145,"ceph_cap_op_grant":5930,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5039,"handle_client_caps_dirty":3000,"handle_client_cap_release":524,"process_request_cap_release":2323},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1259,"num_strays_delayed":222,"num_strays_enqueuing":475,"strays_created":2315,"strays_enqueued":1056,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11963,"evex":0,"evtrm":0,"ev":11968,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":29778020,"rdpos":4195674,"jlat":{"avgcount":17,"sum":6.753281511,"avgtime":0.397251853},"replayed":5},"mds_mem":{"ino":2733,"ino+":3214,"ino-":481,"dir":13,"dir+":13,"dir-":0,"dn":5044,"dn+":5525,"dn-":481,"cap":1318,"cap+":3202,"cap-":1884,"rss":109916,"heap":248164},"mds_server":{"dispatch_client_request":5525,"dispatch_server_request":0,"handle_client_request":5524,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2314,"sum":6.206296504,"avgtime":0.002682064},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5020,"average_load":5020,"avg_session_uptime":1165,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":19945524,"buffer_anon_items":9211,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16067356,"mds_co_items":205948,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":588,"op_send_bytes":25797123,"op_resend":0,"op_reply":588,"op_latency":{"avgcount":588,"sum":4.345910446,"avgtime":0.007391004},"op_inflight":0,"oplen_avg":{"avgcount":588,"sum":645},"op":588,"op_r":46,"op_w":542,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":18,"osdop_write":27,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":499,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":481,"pq_executed":481,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":11717,"get_sum":3253481,"get_or_fail_fail":0,"get_or_fail_success":11717,"take":0,"take_sum":0,"put":11716,"put_sum":3253229,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":588,"take_sum":25953675,"put":588,"put_sum":25953675,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":588,"take_sum":588,"put":588,"put_sum":588,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1531,"get_sum":154631,"get_or_fail_fail":0,"get_or_fail_success":1531,"take":0,"take_sum":0,"put":10,"put_sum":154631,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":417892,"max":3758096384,"get_started":0,"get":11963,"get_sum":25581531,"get_or_fail_fail":0,"get_or_fail_success":11963,"take":0,"take_sum":0,"put":17,"put_sum":25163639,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:44.573 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 144221, "msgr_recv_encrypted_bytes": 105536, "msgr_recv_messages": 161, "msgr_running_fast_dispatch_time": 0.009786018, "msgr_running_recv_time": 0.014759951, "msgr_running_send_time": 0.02406659, "msgr_running_total_time": 0.050679232, "msgr_send_bytes": 4269392, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 158, "msgr_send_messages_queue_lat": { "avgcount": 157, "avgtime": 0.000198442, "sum": 0.031155459 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 24119, "avgtime": 4.9e-07, "sum": 0.01183395 }, "msgr_recv_bytes": 3939702, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 11304, "msgr_running_fast_dispatch_time": 0.020638769, "msgr_running_recv_time": 1.611660161, "msgr_running_send_time": 1.643227307, "msgr_running_total_time": 2.497043757, "msgr_send_bytes": 24406955, "msgr_send_encrypted_bytes": 94496, "msgr_send_messages": 19885, "msgr_send_messages_queue_lat": { "avgcount": 19884, "avgtime": 7.6642e-05, "sum": 1.523953423 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 226717, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 256, "msgr_running_fast_dispatch_time": 0.013376332, "msgr_running_recv_time": 0.012339967, "msgr_running_send_time": 0.030274703, "msgr_running_total_time": 0.059186745, "msgr_send_bytes": 3653216, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 258, "msgr_send_messages_queue_lat": { "avgcount": 257, "avgtime": 0.000233318, "sum": 0.059962951 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 81, "avgtime": 0.114857011, "sum": 9.303417929 }, "queue_len": 1365 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 818, "avgtime": 0.00134247, "sum": 1.098140674 }, "queue_len": 989 }, "mds": { "caps": 1928, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5930, "ceph_cap_op_revoke": 145, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 524, "handle_client_caps": 5039, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4819, "inodes_bottom": 4819, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4819, "inodes_top": 0, "inodes_with_caps": 1928, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2323, "q": 0, "reply": 5523, "reply_latency": { "avgcount": 5523, "avgtime": 0.120037025, "sum": 662.964493885 }, "request": 5524, "root_rbytes": 13005824, "root_rfiles": 1687, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5526, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5526, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1259, "num_strays_delayed": 222, "num_strays_enqueuing": 475, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2315, "strays_enqueued": 1056, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11968, "evadd": 11963, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.397251853, "sum": 6.753281511 }, "rdpos": 4195674, "replayed": 5, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 29778020 }, "mds_mem": { "cap": 1318, "cap+": 3202, "cap-": 1884, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5044, "dn+": 5525, "dn-": 481, "heap": 248164, "ino": 2733, "ino+": 3214, "ino-": 481, "rss": 109916 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5525, "dispatch_server_request": 0, "handle_client_request": 5524, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2314, "avgtime": 0.002682064, "sum": 6.206296504 } }, "mds_sessions": { "average_load": 5020, "avg_session_uptime": 1165, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5020 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 19945524, "buffer_anon_items": 9211, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 16067356, "mds_co_items": 205948, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 18, "op": 588, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 588, "avgtime": 0.007391004, "sum": 4.345910446 }, "op_pg": 0, "op_r": 46, "op_reply": 588, "op_resend": 0, "op_rmw": 0, "op_send": 588, "op_send_bytes": 25797123, "op_w": 542, "oplen_avg": { "avgcount": 588, "sum": 645 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 499, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 18, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 27, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 481, "pq_executed_ops": 481, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11717, "get_or_fail_fail": 0, "get_or_fail_success": 11717, "get_started": 0, "get_sum": 3253481, "max": 104857600, "put": 11716, "put_sum": 3253229, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 588, "put_sum": 25953675, "take": 588, "take_sum": 25953675, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 588, "put_sum": 588, "take": 588, "take_sum": 588, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1531, "get_or_fail_fail": 0, "get_or_fail_success": 1531, "get_started": 0, "get_sum": 154631, "max": 3758096384, "put": 10, "put_sum": 154631, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 11963, "get_or_fail_fail": 0, "get_or_fail_success": 11963, "get_started": 0, "get_sum": 25581531, "max": 3758096384, "put": 17, "put_sum": 25163639, "take": 0, "take_sum": 0, "val": 417892, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:44.574 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1259, 481/2315 2024-03-06T07:34:45.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:45.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.809+0000 7fc960200640 1 Processor -- start 2024-03-06T07:34:45.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.810+0000 7fc960200640 1 -- start start 2024-03-06T07:34:45.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc960200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc95815e370 0x7fc9580d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc960200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc9580d1160 0x7fc9580cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc960200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9580d1e40 con 0x7fc9580cf8a0 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc960200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9580d2a50 con 0x7fc9580d1160 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc960200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9580d2120 con 0x7fc95815e370 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc95ce00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc9580cf8a0 0x7fc9580d0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36298/0 (socket says 172.21.15.32:36298) 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc95ce00640 1 -- 172.21.15.32:0/3399457986 learned_addr learned my addr 172.21.15.32:0/3399457986 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.811+0000 7fc95e200640 1 --2- 172.21.15.32:0/3399457986 >> v2:172.21.15.32:3300/0 conn(0x7fc9580d1160 0x7fc9580cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc95d800640 1 --2- 172.21.15.32:0/3399457986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc95815e370 0x7fc9580d2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc957e00640 1 -- 172.21.15.32:0/3399457986 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075319463 0 0) 0x7fc9580d1e40 con 0x7fc9580cf8a0 2024-03-06T07:34:45.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc957e00640 1 -- 172.21.15.32:0/3399457986 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc940003520 con 0x7fc9580cf8a0 2024-03-06T07:34:45.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc95e200640 1 -- 172.21.15.32:0/3399457986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc95815e370 msgr2=0x7fc9580d2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:45.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc95e200640 1 --2- 172.21.15.32:0/3399457986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc95815e370 0x7fc9580d2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc95e200640 1 -- 172.21.15.32:0/3399457986 >> v1:172.21.15.32:6789/0 conn(0x7fc9580cf8a0 legacy=0x7fc9580d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.812+0000 7fc95e200640 1 -- 172.21.15.32:0/3399457986 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9580d2400 con 0x7fc9580d1160 2024-03-06T07:34:45.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.813+0000 7fc95e200640 1 --2- 172.21.15.32:0/3399457986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580d1160 0x7fc9580cfd00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc950009d90 tx=0x7fc95002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c991132780c1eac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.814+0000 7fc957e00640 1 -- 172.21.15.32:0/3399457986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950039070 con 0x7fc9580d1160 2024-03-06T07:34:45.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.814+0000 7fc957e00640 1 -- 172.21.15.32:0/3399457986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95002cc60 con 0x7fc9580d1160 2024-03-06T07:34:45.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.814+0000 7fc957e00640 1 -- 172.21.15.32:0/3399457986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950034660 con 0x7fc9580d1160 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.814+0000 7fc960200640 1 -- 172.21.15.32:0/3399457986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580d1160 msgr2=0x7fc9580cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.814+0000 7fc960200640 1 --2- 172.21.15.32:0/3399457986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580d1160 0x7fc9580cfd00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc950009d90 tx=0x7fc95002beb0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.815+0000 7fc960200640 1 -- 172.21.15.32:0/3399457986 shutdown_connections 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.815+0000 7fc960200640 1 --2- 172.21.15.32:0/3399457986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580d1160 0x7fc9580cfd00 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.815+0000 7fc960200640 1 --2- 172.21.15.32:0/3399457986 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc95815e370 0x7fc9580d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.815+0000 7fc960200640 1 -- 172.21.15.32:0/3399457986 >> 172.21.15.32:0/3399457986 conn(0x7fc95809d3b0 msgr2=0x7fc9580d40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.815+0000 7fc960200640 1 -- 172.21.15.32:0/3399457986 shutdown_connections 2024-03-06T07:34:45.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.815+0000 7fc960200640 1 -- 172.21.15.32:0/3399457986 wait complete. 2024-03-06T07:34:45.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.817+0000 7fc960200640 1 Processor -- start 2024-03-06T07:34:45.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.817+0000 7fc960200640 1 -- start start 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc960200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9580cf8a0 0x7fc9580a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc960200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9580d1160 0x7fc95809e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc960200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 0x7fc958158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc960200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9580b6f20 con 0x7fc95815e370 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc960200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95815e750 con 0x7fc9580cf8a0 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc960200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9580b70a0 con 0x7fc9580d1160 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc95e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 0x7fc958158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc95e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 0x7fc958158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55348/0 (socket says 172.21.15.32:55348) 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc95e200640 1 -- 172.21.15.32:0/3721041314 learned_addr learned my addr 172.21.15.32:0/3721041314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.818+0000 7fc95ce00640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9580d1160 0x7fc95809e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95d800640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9580cf8a0 0x7fc9580a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95e200640 1 -- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9580d1160 msgr2=0x7fc95809e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95e200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9580d1160 0x7fc95809e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95e200640 1 -- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9580cf8a0 msgr2=0x7fc9580a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95e200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9580cf8a0 0x7fc9580a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95e200640 1 -- 172.21.15.32:0/3721041314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9580c80e0 con 0x7fc95815e370 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc95e200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 0x7fc958158cd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc950009d90 tx=0x7fc950004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950039050 con 0x7fc95815e370 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950045070 con 0x7fc95815e370 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.819+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950009450 con 0x7fc95815e370 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.820+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9580c83c0 con 0x7fc95815e370 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.820+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9580c8980 con 0x7fc95815e370 2024-03-06T07:34:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.821+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9580692a0 con 0x7fc95815e370 2024-03-06T07:34:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.822+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc95003e850 con 0x7fc95815e370 2024-03-06T07:34:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.824+0000 7fc955600640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc93806b4d0 0x7fc93806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.824+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc9500ad150 con 0x7fc95815e370 2024-03-06T07:34:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.824+0000 7fc95d800640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc93806b4d0 0x7fc93806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.825+0000 7fc95d800640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc93806b4d0 0x7fc93806d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc94c006fd0 tx=0x7fc94c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:45.832+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc950032e20 con 0x7fc95815e370 2024-03-06T07:34:46.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.281+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9580d1710 con 0x7fc95815e370 2024-03-06T07:34:46.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.282+0000 7fc955600640 1 -- 172.21.15.32:0/3721041314 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc95007c090 con 0x7fc95815e370 2024-03-06T07:34:46.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:46.285 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:46.285 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.285+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc93806b4d0 msgr2=0x7fc93806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.285+0000 7fc960200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc93806b4d0 0x7fc93806d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc94c006fd0 tx=0x7fc94c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.285+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 msgr2=0x7fc958158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.285+0000 7fc960200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 0x7fc958158cd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc950009d90 tx=0x7fc950004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 shutdown_connections 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc93806b4d0 0x7fc93806d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc95815e370 0x7fc958158cd0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc9580d1160 0x7fc95809e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 --2- 172.21.15.32:0/3721041314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9580cf8a0 0x7fc9580a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 >> 172.21.15.32:0/3721041314 conn(0x7fc95809d3b0 msgr2=0x7fc95815c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 shutdown_connections 2024-03-06T07:34:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:46.286+0000 7fc960200640 1 -- 172.21.15.32:0/3721041314 wait complete. 2024-03-06T07:34:46.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:46.626 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":388,"msgr_send_messages":385,"msgr_recv_bytes":201879,"msgr_send_bytes":4343394,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.077518853,"msgr_running_send_time":0.036041094,"msgr_running_recv_time":0.021223098,"msgr_running_fast_dispatch_time":0.016168291,"msgr_send_messages_queue_lat":{"avgcount":384,"sum":0.060736445,"avgtime":0.000158167},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":105536,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12333,"msgr_send_messages":21846,"msgr_recv_bytes":4275614,"msgr_send_bytes":27500461,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.800245040,"msgr_running_send_time":1.843420584,"msgr_running_recv_time":1.724770027,"msgr_running_fast_dispatch_time":0.031152969,"msgr_send_messages_queue_lat":{"avgcount":21845,"sum":1.793282133,"avgtime":0.000082091},"msgr_handle_ack_lat":{"avgcount":26034,"sum":0.012388966,"avgtime":0.000000475},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96288},"AsyncMessenger::Worker-2":{"msgr_recv_messages":715,"msgr_send_messages":721,"msgr_recv_bytes":396147,"msgr_send_bytes":3856998,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.128358090,"msgr_running_send_time":0.064361640,"msgr_running_recv_time":0.026100752,"msgr_running_fast_dispatch_time":0.029136746,"msgr_send_messages_queue_lat":{"avgcount":720,"sum":0.111565940,"avgtime":0.000154952},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1595,"complete_latency":{"avgcount":83,"sum":11.132688973,"avgtime":0.134128782}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2404,"sum":3.194367812,"avgtime":0.001328771}},"mds":{"request":6272,"reply":6271,"reply_latency":{"avgcount":6271,"sum":664.715443200,"avgtime":0.105998316},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5165,"inodes_top":0,"inodes_bottom":5165,"inodes_pin_tail":0,"inodes_pinned":5165,"inodes_expired":0,"inodes_with_caps":977,"caps":977,"subtrees":2,"traverse":6274,"traverse_hit":6273,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":963,"root_rbytes":7541760,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":145,"ceph_cap_op_grant":6397,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5039,"handle_client_caps_dirty":3000,"handle_client_cap_release":527,"process_request_cap_release":3071},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1007,"num_strays_delayed":170,"num_strays_enqueuing":0,"strays_created":3062,"strays_enqueued":2055,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":14188,"evex":0,"evtrm":0,"ev":14193,"evexg":0,"evexd":0,"segadd":15,"segex":0,"segtrm":0,"seg":17,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":33399174,"rdpos":4195674,"jlat":{"avgcount":19,"sum":7.870082574,"avgtime":0.414214872},"replayed":5},"mds_mem":{"ino":2106,"ino+":3214,"ino-":1108,"dir":13,"dir+":13,"dir-":0,"dn":5165,"dn+":6273,"dn-":1108,"cap":977,"cap+":3202,"cap-":2225,"rss":103044,"heap":248164},"mds_server":{"dispatch_client_request":6273,"dispatch_server_request":0,"handle_client_request":6272,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3062,"sum":7.957245819,"avgtime":0.002598708},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5642,"average_load":5642,"avg_session_uptime":1170,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18826691,"buffer_anon_items":7908,"buffer_meta_bytes":352,"buffer_meta_items":4,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13180332,"mds_co_items":196144,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":1558,"op_send_bytes":28221242,"op_resend":0,"op_reply":1552,"op_latency":{"avgcount":1552,"sum":7.884837533,"avgtime":0.005080436},"op_inflight":5,"oplen_avg":{"avgcount":1558,"sum":1617},"op":1558,"op_r":51,"op_w":1507,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":23,"osdop_write":34,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1456,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":20,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":1433,"pq_executed":1433,"pq_item_in_journal":622},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":13434,"get_sum":3662459,"get_or_fail_fail":0,"get_or_fail_success":13434,"take":0,"take_sum":0,"put":13433,"put_sum":3662207,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1558,"take_sum":28430718,"put":1552,"put_sum":28430718,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1558,"take_sum":1558,"put":1553,"put_sum":1553,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2055,"get_sum":207555,"get_or_fail_fail":0,"get_or_fail_success":2055,"take":0,"take_sum":0,"put":15,"put_sum":207555,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":1668412,"max":3758096384,"get_started":0,"get":14188,"get_sum":29202685,"get_or_fail_fail":0,"get_or_fail_success":14188,"take":0,"take_sum":0,"put":19,"put_sum":27534273,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:46.628 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 201879, "msgr_recv_encrypted_bytes": 105536, "msgr_recv_messages": 388, "msgr_running_fast_dispatch_time": 0.016168291, "msgr_running_recv_time": 0.021223098, "msgr_running_send_time": 0.036041094, "msgr_running_total_time": 0.077518853, "msgr_send_bytes": 4343394, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 385, "msgr_send_messages_queue_lat": { "avgcount": 384, "avgtime": 0.000158167, "sum": 0.060736445 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 26034, "avgtime": 4.75e-07, "sum": 0.012388966 }, "msgr_recv_bytes": 4275614, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 12333, "msgr_running_fast_dispatch_time": 0.031152969, "msgr_running_recv_time": 1.724770027, "msgr_running_send_time": 1.843420584, "msgr_running_total_time": 2.80024504, "msgr_send_bytes": 27500461, "msgr_send_encrypted_bytes": 96288, "msgr_send_messages": 21846, "msgr_send_messages_queue_lat": { "avgcount": 21845, "avgtime": 8.2091e-05, "sum": 1.793282133 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 396147, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 715, "msgr_running_fast_dispatch_time": 0.029136746, "msgr_running_recv_time": 0.026100752, "msgr_running_send_time": 0.06436164, "msgr_running_total_time": 0.12835809, "msgr_send_bytes": 3856998, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 721, "msgr_send_messages_queue_lat": { "avgcount": 720, "avgtime": 0.000154952, "sum": 0.11156594 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 83, "avgtime": 0.134128782, "sum": 11.132688973 }, "queue_len": 1595 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2404, "avgtime": 0.001328771, "sum": 3.194367812 }, "queue_len": 0 }, "mds": { "caps": 977, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6397, "ceph_cap_op_revoke": 145, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 527, "handle_client_caps": 5039, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5165, "inodes_bottom": 5165, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5165, "inodes_top": 0, "inodes_with_caps": 977, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3071, "q": 0, "reply": 6271, "reply_latency": { "avgcount": 6271, "avgtime": 0.105998316, "sum": 664.7154432 }, "request": 6272, "root_rbytes": 7541760, "root_rfiles": 963, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6274, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6273, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1007, "num_strays_delayed": 170, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3062, "strays_enqueued": 2055, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 14193, "evadd": 14188, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 19, "avgtime": 0.414214872, "sum": 7.870082574 }, "rdpos": 4195674, "replayed": 5, "seg": 17, "segadd": 15, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 33399174 }, "mds_mem": { "cap": 977, "cap+": 3202, "cap-": 2225, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5165, "dn+": 6273, "dn-": 1108, "heap": 248164, "ino": 2106, "ino+": 3214, "ino-": 1108, "rss": 103044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6273, "dispatch_server_request": 0, "handle_client_request": 6272, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3062, "avgtime": 0.002598708, "sum": 7.957245819 } }, "mds_sessions": { "average_load": 5642, "avg_session_uptime": 1170, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5642 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18826691, "buffer_anon_items": 7908, "buffer_meta_bytes": 352, "buffer_meta_items": 4, "mds_co_bytes": 13180332, "mds_co_items": 196144, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 20, "op": 1558, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 1552, "avgtime": 0.005080436, "sum": 7.884837533 }, "op_pg": 0, "op_r": 51, "op_reply": 1552, "op_resend": 0, "op_rmw": 0, "op_send": 1558, "op_send_bytes": 28221242, "op_w": 1507, "oplen_avg": { "avgcount": 1558, "sum": 1617 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1456, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 23, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 34, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1433, "pq_executed_ops": 1433, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 622 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13434, "get_or_fail_fail": 0, "get_or_fail_success": 13434, "get_started": 0, "get_sum": 3662459, "max": 104857600, "put": 13433, "put_sum": 3662207, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1552, "put_sum": 28430718, "take": 1558, "take_sum": 28430718, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1553, "put_sum": 1553, "take": 1558, "take_sum": 1558, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2055, "get_or_fail_fail": 0, "get_or_fail_success": 2055, "get_started": 0, "get_sum": 207555, "max": 3758096384, "put": 15, "put_sum": 207555, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 14188, "get_or_fail_fail": 0, "get_or_fail_success": 14188, "get_started": 0, "get_sum": 29202685, "max": 3758096384, "put": 19, "put_sum": 27534273, "take": 0, "take_sum": 0, "val": 1668412, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:46.628 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/1007, 1433/3062 2024-03-06T07:34:47.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:47.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.738+0000 7fb9cbe00640 1 Processor -- start 2024-03-06T07:34:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.739+0000 7fb9cbe00640 1 -- start start 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9cbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c40b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9cbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb9c40b5910 0x7fb9c4071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9cbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb9c4071ac0 0x7fb9c415e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9cbe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c40b5d50 con 0x7fb9c4071ac0 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9cbe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c4072290 con 0x7fb9c40b5910 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9cbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4072570 con 0x7fb9c4140790 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9c9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb9c4071ac0 0x7fb9c415e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9c9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb9c4071ac0 0x7fb9c415e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55352/0 (socket says 172.21.15.32:55352) 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.741+0000 7fb9c9e00640 1 -- 172.21.15.32:0/861682124 learned_addr learned my addr 172.21.15.32:0/861682124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c9e00640 1 -- 172.21.15.32:0/861682124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 msgr2=0x7fb9c40b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c9e00640 1 --2- 172.21.15.32:0/861682124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c40b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c8a00640 1 --2- 172.21.15.32:0/861682124 >> v2:172.21.15.39:3300/0 conn(0x7fb9c40b5910 0x7fb9c4071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c9e00640 1 -- 172.21.15.32:0/861682124 >> v2:172.21.15.39:3300/0 conn(0x7fb9c40b5910 msgr2=0x7fb9c4071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c9e00640 1 --2- 172.21.15.32:0/861682124 >> v2:172.21.15.39:3300/0 conn(0x7fb9c40b5910 0x7fb9c4071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c9e00640 1 -- 172.21.15.32:0/861682124 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c4072850 con 0x7fb9c4071ac0 2024-03-06T07:34:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.742+0000 7fb9c8a00640 1 --2- 172.21.15.32:0/861682124 >> v2:172.21.15.39:3300/0 conn(0x7fb9c40b5910 0x7fb9c4071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.743+0000 7fb9c9e00640 1 --2- 172.21.15.32:0/861682124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c415e090 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fb9c406beb0 tx=0x7fb9bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef78fa102124c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.743+0000 7fb9c3e00640 1 -- 172.21.15.32:0/861682124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc02ca70 con 0x7fb9c4071ac0 2024-03-06T07:34:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.743+0000 7fb9c3e00640 1 -- 172.21.15.32:0/861682124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc02cc10 con 0x7fb9c4071ac0 2024-03-06T07:34:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.743+0000 7fb9c3e00640 1 -- 172.21.15.32:0/861682124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc0343f0 con 0x7fb9c4071ac0 2024-03-06T07:34:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 -- 172.21.15.32:0/861682124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 msgr2=0x7fb9c415e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/861682124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c415e090 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fb9c406beb0 tx=0x7fb9bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 -- 172.21.15.32:0/861682124 shutdown_connections 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/861682124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c415e090 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/861682124 >> v2:172.21.15.39:3300/0 conn(0x7fb9c40b5910 0x7fb9c4071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/861682124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c40b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 -- 172.21.15.32:0/861682124 >> 172.21.15.32:0/861682124 conn(0x7fb9c406cf40 msgr2=0x7fb9c406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 -- 172.21.15.32:0/861682124 shutdown_connections 2024-03-06T07:34:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.744+0000 7fb9cbe00640 1 -- 172.21.15.32:0/861682124 wait complete. 2024-03-06T07:34:47.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.747+0000 7fb9cbe00640 1 Processor -- start 2024-03-06T07:34:47.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.747+0000 7fb9cbe00640 1 -- start start 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.748+0000 7fb9cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c41389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.748+0000 7fb9cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9c40b5910 0x7fb9c40b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.748+0000 7fb9cbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c414fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9cbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4140b70 con 0x7fb9c4071ac0 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9cbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4071f50 con 0x7fb9c40b5910 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9cbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4140cf0 con 0x7fb9c4140790 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c41389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c41389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55354/0 (socket says 172.21.15.32:55354) 2024-03-06T07:34:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 -- 172.21.15.32:0/1869364398 learned_addr learned my addr 172.21.15.32:0/1869364398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c8a00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9c40b5910 0x7fb9c40b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9e00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c414fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 -- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 msgr2=0x7fb9c414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c414fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 -- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9c40b5910 msgr2=0x7fb9c40b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9c40b5910 0x7fb9c40b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.749+0000 7fb9c9400640 1 -- 172.21.15.32:0/1869364398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c41f79a0 con 0x7fb9c4071ac0 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.750+0000 7fb9c9400640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c41389d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800b410 tx=0x7fb9b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.750+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8015020 con 0x7fb9c4071ac0 2024-03-06T07:34:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.750+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8011850 con 0x7fb9c4071ac0 2024-03-06T07:34:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.750+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8011c40 con 0x7fb9c4071ac0 2024-03-06T07:34:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.750+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c41f7c80 con 0x7fb9c4071ac0 2024-03-06T07:34:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.750+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c41f8240 con 0x7fb9c4071ac0 2024-03-06T07:34:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.752+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c4135050 con 0x7fb9c4071ac0 2024-03-06T07:34:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.753+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9b8018020 con 0x7fb9c4071ac0 2024-03-06T07:34:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.753+0000 7fb9c1600640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9a406b4d0 0x7fb9a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.754+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9b808a2a0 con 0x7fb9c4071ac0 2024-03-06T07:34:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.754+0000 7fb9c8a00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9a406b4d0 0x7fb9a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.755+0000 7fb9c8a00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9a406b4d0 0x7fb9a406d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4152cd0 tx=0x7fb9b0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.758+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9c4135050 con 0x7fb9c4071ac0 2024-03-06T07:34:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.803+0000 7f0c3a400640 1 Processor -- start 2024-03-06T07:34:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.803+0000 7f0c3a400640 1 -- start start 2024-03-06T07:34:47.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.804+0000 7f0c3a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.804+0000 7f0c3a400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0c34161470 0x7f0c34161900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.804+0000 7f0c3a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340cffc0 con 0x7f0c3406ef10 2024-03-06T07:34:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.804+0000 7f0c3a400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c340cf480 con 0x7f0c34161470 2024-03-06T07:34:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.804+0000 7f0c3a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3406ff60 con 0x7f0c340ce990 2024-03-06T07:34:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.805+0000 7f0c33400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.805+0000 7f0c33e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0c34161470 0x7f0c34161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.805+0000 7f0c33400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:49334/0 (socket says 172.21.15.32:49334) 2024-03-06T07:34:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.805+0000 7f0c33e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0c34161470 0x7f0c34161900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:35894/0 (socket says 172.21.15.32:35894) 2024-03-06T07:34:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.805+0000 7f0c33400640 1 -- 172.21.15.32:0/4079977657 learned_addr learned my addr 172.21.15.32:0/4079977657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.806+0000 7f0c33e00640 1 -- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 msgr2=0x7f0c340d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.806+0000 7f0c33e00640 1 --2- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.806+0000 7f0c33e00640 1 -- 172.21.15.32:0/4079977657 >> v1:172.21.15.39:6789/0 conn(0x7f0c3406ef10 legacy=0x7f0c340ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.806+0000 7f0c33e00640 1 -- 172.21.15.32:0/4079977657 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340d0260 con 0x7f0c34161470 2024-03-06T07:34:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.807+0000 7f0c33e00640 1 --2- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c34161470 0x7f0c34161900 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009d90 tx=0x7f0c2c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d90dadf2be00316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.807+0000 7f0c32000640 1 -- 172.21.15.32:0/4079977657 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c039070 con 0x7f0c34161470 2024-03-06T07:34:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.807+0000 7f0c32000640 1 -- 172.21.15.32:0/4079977657 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c02cc60 con 0x7f0c34161470 2024-03-06T07:34:47.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.807+0000 7f0c32000640 1 -- 172.21.15.32:0/4079977657 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c034660 con 0x7f0c34161470 2024-03-06T07:34:47.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 -- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c34161470 msgr2=0x7f0c34161900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c34161470 0x7f0c34161900 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009d90 tx=0x7f0c2c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 -- 172.21.15.32:0/4079977657 shutdown_connections 2024-03-06T07:34:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c34161470 0x7f0c34161900 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4079977657 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 -- 172.21.15.32:0/4079977657 >> 172.21.15.32:0/4079977657 conn(0x7f0c340b39d0 msgr2=0x7f0c340b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 -- 172.21.15.32:0/4079977657 shutdown_connections 2024-03-06T07:34:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.808+0000 7f0c3a400640 1 -- 172.21.15.32:0/4079977657 wait complete. 2024-03-06T07:34:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.810+0000 7f0c3a400640 1 Processor -- start 2024-03-06T07:34:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.810+0000 7f0c3a400640 1 -- start start 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c3a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c3406ef10 0x7f0c340b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c3a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c3a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 0x7f0c340c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c3a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340d1280 con 0x7f0c34161470 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c3a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340d0820 con 0x7f0c3406ef10 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c3a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3409be30 con 0x7f0c340ce990 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 0x7f0c340c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 0x7f0c340c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55362/0 (socket says 172.21.15.32:55362) 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c33e00640 1 -- 172.21.15.32:0/4292299906 learned_addr learned my addr 172.21.15.32:0/4292299906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c33400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c3406ef10 0x7f0c340b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.811+0000 7f0c32a00640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c33e00640 1 -- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 msgr2=0x7f0c340c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c33e00640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c33e00640 1 -- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c3406ef10 msgr2=0x7f0c340b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c33e00640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c3406ef10 0x7f0c340b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c33e00640 1 -- 172.21.15.32:0/4292299906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c341f3f10 con 0x7f0c34161470 2024-03-06T07:34:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c33e00640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 0x7f0c340c6d50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c037040 tx=0x7f0c2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c044020 con 0x7f0c34161470 2024-03-06T07:34:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.812+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c03f470 con 0x7f0c34161470 2024-03-06T07:34:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.813+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c039040 con 0x7f0c34161470 2024-03-06T07:34:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.813+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c341f41f0 con 0x7f0c34161470 2024-03-06T07:34:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.813+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c341f47b0 con 0x7f0c34161470 2024-03-06T07:34:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.814+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c340692a0 con 0x7f0c34161470 2024-03-06T07:34:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.815+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c2c047020 con 0x7f0c34161470 2024-03-06T07:34:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.815+0000 7f0c2b400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c1406b4d0 0x7f0c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.816+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0c2c0acdf0 con 0x7f0c34161470 2024-03-06T07:34:47.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.816+0000 7f0c33400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c1406b4d0 0x7f0c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.817+0000 7f0c33400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c1406b4d0 0x7f0c1406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009b60 tx=0x7f0c24008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:47.821+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c2c04e030 con 0x7f0c34161470 2024-03-06T07:34:48.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.072+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c406ef70 con 0x7fb9c4071ac0 2024-03-06T07:34:48.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.072+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c340b7e20 con 0x7f0c34161470 2024-03-06T07:34:48.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.073+0000 7f0c2b400640 1 -- 172.21.15.32:0/4292299906 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f0c2c07bc30 con 0x7f0c34161470 2024-03-06T07:34:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:48.076 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:48.076 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:48.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.073+0000 7fb9c1600640 1 -- 172.21.15.32:0/1869364398 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fb9b801f020 con 0x7fb9c4071ac0 2024-03-06T07:34:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:48.077 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:48.077 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.075+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c1406b4d0 msgr2=0x7f0c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c1406b4d0 0x7f0c1406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009b60 tx=0x7f0c24008040 comp rx=0 tx=0).stop 2024-03-06T07:34:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 msgr2=0x7f0c340c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 0x7f0c340c6d50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c037040 tx=0x7f0c2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 shutdown_connections 2024-03-06T07:34:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c1406b4d0 0x7f0c1406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c34161470 0x7f0c340c6d50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c340ce990 0x7f0c340c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 --2- 172.21.15.32:0/4292299906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c3406ef10 0x7f0c340b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9a406b4d0 msgr2=0x7fb9a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 >> 172.21.15.32:0/4292299906 conn(0x7f0c340b39d0 msgr2=0x7f0c340b6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9a406b4d0 0x7fb9a406d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4152cd0 tx=0x7fb9b0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 msgr2=0x7fb9c41389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c41389d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800b410 tx=0x7fb9b800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 shutdown_connections 2024-03-06T07:34:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb9a406b4d0 0x7fb9a406d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 shutdown_connections 2024-03-06T07:34:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb9c4140790 0x7fb9c414fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9c40b5910 0x7fb9c40b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 --2- 172.21.15.32:0/1869364398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9c4071ac0 0x7fb9c41389d0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 >> 172.21.15.32:0/1869364398 conn(0x7fb9c406cf40 msgr2=0x7fb9c415e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:48.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7f0c3a400640 1 -- 172.21.15.32:0/4292299906 wait complete. 2024-03-06T07:34:48.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 shutdown_connections 2024-03-06T07:34:48.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:48.076+0000 7fb9cbe00640 1 -- 172.21.15.32:0/1869364398 wait complete. 2024-03-06T07:34:48.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:34:48.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:48.512 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:34:48.512 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:48.527 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":834,"msgr_send_messages":831,"msgr_recv_bytes":318805,"msgr_send_bytes":4499230,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.130110170,"msgr_running_send_time":0.059125264,"msgr_running_recv_time":0.033334221,"msgr_running_fast_dispatch_time":0.029308515,"msgr_send_messages_queue_lat":{"avgcount":830,"sum":0.125779860,"avgtime":0.000151542},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":105808,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12967,"msgr_send_messages":24173,"msgr_recv_bytes":4473355,"msgr_send_bytes":31132004,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.012652252,"msgr_running_send_time":1.987464798,"msgr_running_recv_time":1.765493852,"msgr_running_fast_dispatch_time":0.047413778,"msgr_send_messages_queue_lat":{"avgcount":24172,"sum":2.132065905,"avgtime":0.000088203},"msgr_handle_ack_lat":{"avgcount":27170,"sum":0.012754033,"avgtime":0.000000469},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96288},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1615,"msgr_send_messages":1617,"msgr_recv_bytes":732070,"msgr_send_bytes":4269498,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.229577110,"msgr_running_send_time":0.112456102,"msgr_running_recv_time":0.047613382,"msgr_running_fast_dispatch_time":0.053275264,"msgr_send_messages_queue_lat":{"avgcount":1616,"sum":0.238380456,"avgtime":0.000147512},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":108,"sum":12.583652917,"avgtime":0.116515304}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5214,"sum":3.544806854,"avgtime":0.000679863}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":665.601454421,"avgtime":0.103837980},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3847,"inodes_top":0,"inodes_bottom":3847,"inodes_pin_tail":0,"inodes_pinned":3847,"inodes_expired":0,"inodes_with_caps":127,"caps":127,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":148,"ceph_cap_op_grant":7248,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5042,"handle_client_caps_dirty":3000,"handle_client_cap_release":532,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":126,"num_strays_delayed":126,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3075,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15440,"evex":15442,"evtrm":15442,"ev":15444,"evexg":0,"evexd":15441,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":34569750,"wrpos":34573944,"rdpos":4195674,"jlat":{"avgcount":24,"sum":8.483424523,"avgtime":0.353476021},"replayed":5},"mds_mem":{"ino":1159,"ino+":3214,"ino-":2055,"dir":13,"dir+":13,"dir-":0,"dn":4357,"dn+":6412,"dn-":2055,"cap":127,"cap+":3202,"cap-":3075,"rss":103044,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.585629125,"avgtime":0.585629125},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.257627915,"avgtime":0.002580508},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5718,"average_load":5718,"avg_session_uptime":1170,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18826691,"buffer_anon_items":7908,"buffer_meta_bytes":352,"buffer_meta_items":4,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13180332,"mds_co_items":196144,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3380,"op_send_bytes":31334317,"op_resend":0,"op_reply":3380,"op_latency":{"avgcount":3380,"sum":15.284685028,"avgtime":0.004522096},"op_inflight":0,"oplen_avg":{"avgcount":3380,"sum":3734},"op":3380,"op_r":57,"op_w":3323,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":137,"osdop_read":29,"osdop_write":45,"osdop_writefull":13,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3100,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":274,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":43,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3075,"pq_executed":3075,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15412,"get_sum":4134469,"get_or_fail_fail":0,"get_or_fail_success":15412,"take":0,"take_sum":0,"put":15412,"put_sum":4134469,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3380,"take_sum":31646813,"put":3380,"put_sum":31646813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3380,"take_sum":3380,"put":3380,"put_sum":3380,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3075,"get_sum":310575,"get_or_fail_fail":0,"get_or_fail_success":3075,"take":0,"take_sum":0,"put":21,"put_sum":310575,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":0,"max":3758096384,"get_started":0,"get":15440,"get_sum":30377455,"get_or_fail_fail":0,"get_or_fail_success":15440,"take":0,"take_sum":0,"put":24,"put_sum":30377455,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:48.529 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 318805, "msgr_recv_encrypted_bytes": 105808, "msgr_recv_messages": 834, "msgr_running_fast_dispatch_time": 0.029308515, "msgr_running_recv_time": 0.033334221, "msgr_running_send_time": 0.059125264, "msgr_running_total_time": 0.13011017, "msgr_send_bytes": 4499230, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 831, "msgr_send_messages_queue_lat": { "avgcount": 830, "avgtime": 0.000151542, "sum": 0.12577986 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27170, "avgtime": 4.69e-07, "sum": 0.012754033 }, "msgr_recv_bytes": 4473355, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 12967, "msgr_running_fast_dispatch_time": 0.047413778, "msgr_running_recv_time": 1.765493852, "msgr_running_send_time": 1.987464798, "msgr_running_total_time": 3.012652252, "msgr_send_bytes": 31132004, "msgr_send_encrypted_bytes": 96288, "msgr_send_messages": 24173, "msgr_send_messages_queue_lat": { "avgcount": 24172, "avgtime": 8.8203e-05, "sum": 2.132065905 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 732070, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1615, "msgr_running_fast_dispatch_time": 0.053275264, "msgr_running_recv_time": 0.047613382, "msgr_running_send_time": 0.112456102, "msgr_running_total_time": 0.22957711, "msgr_send_bytes": 4269498, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1617, "msgr_send_messages_queue_lat": { "avgcount": 1616, "avgtime": 0.000147512, "sum": 0.238380456 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 108, "avgtime": 0.116515304, "sum": 12.583652917 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5214, "avgtime": 0.000679863, "sum": 3.544806854 }, "queue_len": 0 }, "mds": { "caps": 127, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7248, "ceph_cap_op_revoke": 148, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 532, "handle_client_caps": 5042, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3847, "inodes_bottom": 3847, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3847, "inodes_top": 0, "inodes_with_caps": 127, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.10383798, "sum": 665.601454421 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 126, "num_strays_delayed": 126, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3075, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15444, "evadd": 15440, "evex": 15442, "evexd": 15441, "evexg": 0, "evlrg": 0, "evtrm": 15442, "expos": 34569750, "jlat": { "avgcount": 24, "avgtime": 0.353476021, "sum": 8.483424523 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 34573944 }, "mds_mem": { "cap": 127, "cap+": 3202, "cap-": 3075, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4357, "dn+": 6412, "dn-": 2055, "heap": 248164, "ino": 1159, "ino+": 3214, "ino-": 2055, "rss": 103044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.585629125, "sum": 0.585629125 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002580508, "sum": 8.257627915 } }, "mds_sessions": { "average_load": 5718, "avg_session_uptime": 1170, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5718 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18826691, "buffer_anon_items": 7908, "buffer_meta_bytes": 352, "buffer_meta_items": 4, "mds_co_bytes": 13180332, "mds_co_items": 196144, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 43, "op": 3380, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3380, "avgtime": 0.004522096, "sum": 15.284685028 }, "op_pg": 0, "op_r": 57, "op_reply": 3380, "op_resend": 0, "op_rmw": 0, "op_send": 3380, "op_send_bytes": 31334317, "op_w": 3323, "oplen_avg": { "avgcount": 3380, "sum": 3734 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 137, "osdop_delete": 3100, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 274, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 45, "osdop_writefull": 13, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3075, "pq_executed_ops": 3075, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15412, "get_or_fail_fail": 0, "get_or_fail_success": 15412, "get_started": 0, "get_sum": 4134469, "max": 104857600, "put": 15412, "put_sum": 4134469, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3380, "put_sum": 31646813, "take": 3380, "take_sum": 31646813, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3380, "put_sum": 3380, "take": 3380, "take_sum": 3380, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3075, "get_or_fail_fail": 0, "get_or_fail_success": 3075, "get_started": 0, "get_sum": 310575, "max": 3758096384, "put": 21, "put_sum": 310575, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 15440, "get_or_fail_fail": 0, "get_or_fail_success": 15440, "get_started": 0, "get_sum": 30377455, "max": 3758096384, "put": 24, "put_sum": 30377455, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:48.529 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/126, 3075/3201 2024-03-06T07:34:49.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:49.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.753+0000 7f55dbe00640 1 Processor -- start 2024-03-06T07:34:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.753+0000 7f55dbe00640 1 -- start start 2024-03-06T07:34:49.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55dbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f55dc06afc0 0x7f55dc15f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55dbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc06b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55dbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55dc0b56d0 0x7f55dc071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55dbe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc15ece0 con 0x7f55dc0b56d0 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55dbe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc06b8e0 con 0x7f55dc06afc0 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55dbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0b5c20 con 0x7f55dc06f900 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55db400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55dc0b56d0 0x7f55dc071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55db400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f55dc0b56d0 0x7f55dc071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33524/0 (socket says 172.21.15.32:33524) 2024-03-06T07:34:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.755+0000 7f55db400640 1 -- 172.21.15.32:0/756499866 learned_addr learned my addr 172.21.15.32:0/756499866 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55da000640 1 --2- 172.21.15.32:0/756499866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc06b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55daa00640 1 --2- 172.21.15.32:0/756499866 >> v2:172.21.15.39:3300/0 conn(0x7f55dc06afc0 0x7f55dc15f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55db400640 1 -- 172.21.15.32:0/756499866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 msgr2=0x7f55dc06b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55db400640 1 --2- 172.21.15.32:0/756499866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc06b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55db400640 1 -- 172.21.15.32:0/756499866 >> v2:172.21.15.39:3300/0 conn(0x7f55dc06afc0 msgr2=0x7f55dc15f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55db400640 1 --2- 172.21.15.32:0/756499866 >> v2:172.21.15.39:3300/0 conn(0x7f55dc06afc0 0x7f55dc15f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.756+0000 7f55db400640 1 -- 172.21.15.32:0/756499866 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc15e690 con 0x7f55dc0b56d0 2024-03-06T07:34:49.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.757+0000 7f55db400640 1 --2- 172.21.15.32:0/756499866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc071c90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f55dc075600 tx=0x7f55d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f3d3184783a980 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.757+0000 7f55d9600640 1 -- 172.21.15.32:0/756499866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d402ca70 con 0x7f55dc0b56d0 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.757+0000 7f55d9600640 1 -- 172.21.15.32:0/756499866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d402cc10 con 0x7f55dc0b56d0 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.757+0000 7f55d9600640 1 -- 172.21.15.32:0/756499866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d40343f0 con 0x7f55dc0b56d0 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 -- 172.21.15.32:0/756499866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 msgr2=0x7f55dc071c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 --2- 172.21.15.32:0/756499866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc071c90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f55dc075600 tx=0x7f55d402bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 -- 172.21.15.32:0/756499866 shutdown_connections 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 --2- 172.21.15.32:0/756499866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc071c90 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 --2- 172.21.15.32:0/756499866 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc06b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 --2- 172.21.15.32:0/756499866 >> v2:172.21.15.39:3300/0 conn(0x7f55dc06afc0 0x7f55dc15f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 -- 172.21.15.32:0/756499866 >> 172.21.15.32:0/756499866 conn(0x7f55dc0b29a0 msgr2=0x7f55dc0b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:49.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 -- 172.21.15.32:0/756499866 shutdown_connections 2024-03-06T07:34:49.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.758+0000 7f55dbe00640 1 -- 172.21.15.32:0/756499866 wait complete. 2024-03-06T07:34:49.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.760+0000 7f55dbe00640 1 Processor -- start 2024-03-06T07:34:49.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.760+0000 7f55dbe00640 1 -- start start 2024-03-06T07:34:49.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55dc06afc0 0x7f55dc0af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55dbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55dbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc0d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55dbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0b5f00 con 0x7f55dc0b56d0 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55dbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0d09d0 con 0x7f55dc06afc0 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55dbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0b3ca0 con 0x7f55dc06f900 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55db400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc0d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55db400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc0d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33530/0 (socket says 172.21.15.32:33530) 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55db400640 1 -- 172.21.15.32:0/2249832760 learned_addr learned my addr 172.21.15.32:0/2249832760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.761+0000 7f55daa00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55dc06afc0 0x7f55dc0af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55da000640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55db400640 1 -- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 msgr2=0x7f55dc154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55db400640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55db400640 1 -- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55dc06afc0 msgr2=0x7f55dc0af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55db400640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55dc06afc0 0x7f55dc0af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55db400640 1 -- 172.21.15.32:0/2249832760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc158b80 con 0x7f55dc0b56d0 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55daa00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55dc06afc0 0x7f55dc0af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55db400640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc0d0dc0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f55d4036040 tx=0x7f55d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4044020 con 0x7f55dc0b56d0 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4040690 con 0x7f55dc0b56d0 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc158e60 con 0x7f55dc0b56d0 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.762+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4009510 con 0x7f55dc0b56d0 2024-03-06T07:34:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.763+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc0afdd0 con 0x7f55dc0b56d0 2024-03-06T07:34:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.764+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc04f330 con 0x7f55dc0b56d0 2024-03-06T07:34:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.765+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55d4047020 con 0x7f55dc0b56d0 2024-03-06T07:34:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.766+0000 7f55d3400640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f55bc06b4d0 0x7f55bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.766+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f55d40aceb0 con 0x7f55dc0b56d0 2024-03-06T07:34:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.766+0000 7f55daa00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f55bc06b4d0 0x7f55bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.767+0000 7f55daa00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f55bc06b4d0 0x7f55bc06d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006fd0 tx=0x7f55cc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:49.770+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55d404e030 con 0x7f55dc0b56d0 2024-03-06T07:34:50.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.013+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55dc0ae9e0 con 0x7f55dc0b56d0 2024-03-06T07:34:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.015+0000 7f55d3400640 1 -- 172.21.15.32:0/2249832760 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f55d407bc30 con 0x7f55dc0b56d0 2024-03-06T07:34:50.017 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:50.017 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:50.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:50.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:50.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.017+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f55bc06b4d0 msgr2=0x7f55bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.017+0000 7f55dbe00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f55bc06b4d0 0x7f55bc06d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006fd0 tx=0x7f55cc008040 comp rx=0 tx=0).stop 2024-03-06T07:34:50.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.017+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 msgr2=0x7f55dc0d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.017+0000 7f55dbe00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc0d0dc0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f55d4036040 tx=0x7f55d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 shutdown_connections 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f55bc06b4d0 0x7f55bc06d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55dc0b56d0 0x7f55dc0d0dc0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f55dc06f900 0x7f55dc154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 --2- 172.21.15.32:0/2249832760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55dc06afc0 0x7f55dc0af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 >> 172.21.15.32:0/2249832760 conn(0x7f55dc0b29a0 msgr2=0x7f55dc0aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 shutdown_connections 2024-03-06T07:34:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:50.018+0000 7f55dbe00640 1 -- 172.21.15.32:0/2249832760 wait complete. 2024-03-06T07:34:50.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:50.287 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":834,"msgr_send_messages":831,"msgr_recv_bytes":318805,"msgr_send_bytes":4499230,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.130259184,"msgr_running_send_time":0.059200563,"msgr_running_recv_time":0.033365861,"msgr_running_fast_dispatch_time":0.029308515,"msgr_send_messages_queue_lat":{"avgcount":830,"sum":0.125779860,"avgtime":0.000151542},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":105808,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12970,"msgr_send_messages":24173,"msgr_recv_bytes":4477313,"msgr_send_bytes":31132004,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.013308283,"msgr_running_send_time":1.987723436,"msgr_running_recv_time":1.765760191,"msgr_running_fast_dispatch_time":0.047603830,"msgr_send_messages_queue_lat":{"avgcount":24172,"sum":2.132065905,"avgtime":0.000088203},"msgr_handle_ack_lat":{"avgcount":27173,"sum":0.012754530,"avgtime":0.000000469},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96288},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1615,"msgr_send_messages":1617,"msgr_recv_bytes":732070,"msgr_send_bytes":4269498,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.229577110,"msgr_running_send_time":0.112456102,"msgr_running_recv_time":0.047613382,"msgr_running_fast_dispatch_time":0.053275264,"msgr_send_messages_queue_lat":{"avgcount":1616,"sum":0.238380456,"avgtime":0.000147512},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":108,"sum":12.583652917,"avgtime":0.116515304}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5214,"sum":3.544806854,"avgtime":0.000679863}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":665.601454421,"avgtime":0.103837980},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3847,"inodes_top":0,"inodes_bottom":3847,"inodes_pin_tail":0,"inodes_pinned":3847,"inodes_expired":0,"inodes_with_caps":127,"caps":127,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":148,"ceph_cap_op_grant":7248,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5042,"handle_client_caps_dirty":3000,"handle_client_cap_release":533,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":126,"num_strays_delayed":126,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3075,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15440,"evex":15442,"evtrm":15442,"ev":15444,"evexg":0,"evexd":15441,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":34569750,"wrpos":34573944,"rdpos":4195674,"jlat":{"avgcount":24,"sum":8.483424523,"avgtime":0.353476021},"replayed":5},"mds_mem":{"ino":139,"ino+":3214,"ino-":3075,"dir":13,"dir+":13,"dir-":0,"dn":137,"dn+":6412,"dn-":6275,"cap":1,"cap+":3202,"cap-":3201,"rss":103300,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.585629125,"avgtime":0.585629125},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.257627915,"avgtime":0.002580508},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5718,"average_load":5718,"avg_session_uptime":1170,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18826691,"buffer_anon_items":7908,"buffer_meta_bytes":352,"buffer_meta_items":4,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13180332,"mds_co_items":196144,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3380,"op_send_bytes":31334317,"op_resend":0,"op_reply":3380,"op_latency":{"avgcount":3380,"sum":15.284685028,"avgtime":0.004522096},"op_inflight":0,"oplen_avg":{"avgcount":3380,"sum":3734},"op":3380,"op_r":57,"op_w":3323,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":137,"osdop_read":29,"osdop_write":45,"osdop_writefull":13,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3100,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":274,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":43,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3075,"pq_executed":3075,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15415,"get_sum":4138157,"get_or_fail_fail":0,"get_or_fail_success":15415,"take":0,"take_sum":0,"put":15415,"put_sum":4138157,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3380,"take_sum":31646813,"put":3380,"put_sum":31646813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3380,"take_sum":3380,"put":3380,"put_sum":3380,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3075,"get_sum":310575,"get_or_fail_fail":0,"get_or_fail_success":3075,"take":0,"take_sum":0,"put":21,"put_sum":310575,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":0,"max":3758096384,"get_started":0,"get":15440,"get_sum":30377455,"get_or_fail_fail":0,"get_or_fail_success":15440,"take":0,"take_sum":0,"put":24,"put_sum":30377455,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:50.288 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 318805, "msgr_recv_encrypted_bytes": 105808, "msgr_recv_messages": 834, "msgr_running_fast_dispatch_time": 0.029308515, "msgr_running_recv_time": 0.033365861, "msgr_running_send_time": 0.059200563, "msgr_running_total_time": 0.130259184, "msgr_send_bytes": 4499230, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 831, "msgr_send_messages_queue_lat": { "avgcount": 830, "avgtime": 0.000151542, "sum": 0.12577986 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27173, "avgtime": 4.69e-07, "sum": 0.01275453 }, "msgr_recv_bytes": 4477313, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 12970, "msgr_running_fast_dispatch_time": 0.04760383, "msgr_running_recv_time": 1.765760191, "msgr_running_send_time": 1.987723436, "msgr_running_total_time": 3.013308283, "msgr_send_bytes": 31132004, "msgr_send_encrypted_bytes": 96288, "msgr_send_messages": 24173, "msgr_send_messages_queue_lat": { "avgcount": 24172, "avgtime": 8.8203e-05, "sum": 2.132065905 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 732070, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1615, "msgr_running_fast_dispatch_time": 0.053275264, "msgr_running_recv_time": 0.047613382, "msgr_running_send_time": 0.112456102, "msgr_running_total_time": 0.22957711, "msgr_send_bytes": 4269498, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1617, "msgr_send_messages_queue_lat": { "avgcount": 1616, "avgtime": 0.000147512, "sum": 0.238380456 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 108, "avgtime": 0.116515304, "sum": 12.583652917 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5214, "avgtime": 0.000679863, "sum": 3.544806854 }, "queue_len": 0 }, "mds": { "caps": 127, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7248, "ceph_cap_op_revoke": 148, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 533, "handle_client_caps": 5042, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3847, "inodes_bottom": 3847, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3847, "inodes_top": 0, "inodes_with_caps": 127, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.10383798, "sum": 665.601454421 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 126, "num_strays_delayed": 126, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3075, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15444, "evadd": 15440, "evex": 15442, "evexd": 15441, "evexg": 0, "evlrg": 0, "evtrm": 15442, "expos": 34569750, "jlat": { "avgcount": 24, "avgtime": 0.353476021, "sum": 8.483424523 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 34573944 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 137, "dn+": 6412, "dn-": 6275, "heap": 248164, "ino": 139, "ino+": 3214, "ino-": 3075, "rss": 103300 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.585629125, "sum": 0.585629125 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002580508, "sum": 8.257627915 } }, "mds_sessions": { "average_load": 5718, "avg_session_uptime": 1170, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5718 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18826691, "buffer_anon_items": 7908, "buffer_meta_bytes": 352, "buffer_meta_items": 4, "mds_co_bytes": 13180332, "mds_co_items": 196144, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 43, "op": 3380, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3380, "avgtime": 0.004522096, "sum": 15.284685028 }, "op_pg": 0, "op_r": 57, "op_reply": 3380, "op_resend": 0, "op_rmw": 0, "op_send": 3380, "op_send_bytes": 31334317, "op_w": 3323, "oplen_avg": { "avgcount": 3380, "sum": 3734 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 137, "osdop_delete": 3100, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 274, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 45, "osdop_writefull": 13, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3075, "pq_executed_ops": 3075, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15415, "get_or_fail_fail": 0, "get_or_fail_success": 15415, "get_started": 0, "get_sum": 4138157, "max": 104857600, "put": 15415, "put_sum": 4138157, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3380, "put_sum": 31646813, "take": 3380, "take_sum": 31646813, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3380, "put_sum": 3380, "take": 3380, "take_sum": 3380, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3075, "get_or_fail_fail": 0, "get_or_fail_success": 3075, "get_started": 0, "get_sum": 310575, "max": 3758096384, "put": 21, "put_sum": 310575, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 15440, "get_or_fail_fail": 0, "get_or_fail_success": 15440, "get_started": 0, "get_sum": 30377455, "max": 3758096384, "put": 24, "put_sum": 30377455, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:50.289 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/126, 3075/3201 2024-03-06T07:34:51.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.487+0000 7fb8ac800640 1 Processor -- start 2024-03-06T07:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.488+0000 7fb8ac800640 1 -- start start 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.489+0000 7fb8ac800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8a4160380 0x7fb8a4162b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.489+0000 7fb8ac800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a40b3f40 con 0x7fb8a415ffa0 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.489+0000 7fb8ac800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a40b36a0 con 0x7fb8a40b4b20 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.489+0000 7fb8ac800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a4070e20 con 0x7fb8a4160380 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.489+0000 7fb8a9e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb8a415ffa0 0x7fb8a40b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55738/0 (socket says 172.21.15.32:55738) 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a9e00640 1 -- 172.21.15.32:0/2665156040 learned_addr learned my addr 172.21.15.32:0/2665156040 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a9400640 1 --2- 172.21.15.32:0/2665156040 >> v2:172.21.15.32:3300/0 conn(0x7fb8a4160380 0x7fb8a4162b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a8a00640 1 -- 172.21.15.32:0/2665156040 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220324297 0 0) 0x7fb8a40b3f40 con 0x7fb8a415ffa0 2024-03-06T07:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a8a00640 1 -- 172.21.15.32:0/2665156040 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c003520 con 0x7fb8a415ffa0 2024-03-06T07:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a9400640 1 -- 172.21.15.32:0/2665156040 >> v1:172.21.15.39:6789/0 conn(0x7fb8a40b4b20 legacy=0x7fb8a40712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a9400640 1 -- 172.21.15.32:0/2665156040 >> v1:172.21.15.32:6789/0 conn(0x7fb8a415ffa0 legacy=0x7fb8a40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.490+0000 7fb8a9400640 1 -- 172.21.15.32:0/2665156040 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40b3980 con 0x7fb8a4160380 2024-03-06T07:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.492+0000 7fb8a9400640 1 --2- 172.21.15.32:0/2665156040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a4160380 0x7fb8a4162b50 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4070fa0 tx=0x7fb89402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6596ba1f7f0d1a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.492+0000 7fb8a8a00640 1 -- 172.21.15.32:0/2665156040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894035020 con 0x7fb8a4160380 2024-03-06T07:34:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.492+0000 7fb8a8a00640 1 -- 172.21.15.32:0/2665156040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb894031830 con 0x7fb8a4160380 2024-03-06T07:34:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.492+0000 7fb8a8a00640 1 -- 172.21.15.32:0/2665156040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894031c00 con 0x7fb8a4160380 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.492+0000 7fb8ac800640 1 -- 172.21.15.32:0/2665156040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a4160380 msgr2=0x7fb8a4162b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.492+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2665156040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a4160380 0x7fb8a4162b50 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4070fa0 tx=0x7fb89402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.493+0000 7fb8ac800640 1 -- 172.21.15.32:0/2665156040 shutdown_connections 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.493+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2665156040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a4160380 0x7fb8a4162b50 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.493+0000 7fb8ac800640 1 -- 172.21.15.32:0/2665156040 >> 172.21.15.32:0/2665156040 conn(0x7fb8a406cf40 msgr2=0x7fb8a406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.493+0000 7fb8ac800640 1 -- 172.21.15.32:0/2665156040 shutdown_connections 2024-03-06T07:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.493+0000 7fb8ac800640 1 -- 172.21.15.32:0/2665156040 wait complete. 2024-03-06T07:34:51.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.494+0000 7fb8ac800640 1 Processor -- start 2024-03-06T07:34:51.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.495+0000 7fb8ac800640 1 -- start start 2024-03-06T07:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.495+0000 7fb8ac800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8a40b4b20 0x7fb8a415a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.495+0000 7fb8ac800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8a415ffa0 0x7fb8a4158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8ac800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 0x7fb8a4161590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8ac800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40b5150 con 0x7fb8a41589c0 2024-03-06T07:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8ac800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40b4fd0 con 0x7fb8a40b4b20 2024-03-06T07:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8ac800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40b52d0 con 0x7fb8a415ffa0 2024-03-06T07:34:51.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 0x7fb8a4161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 0x7fb8a4161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33540/0 (socket says 172.21.15.32:33540) 2024-03-06T07:34:51.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 -- 172.21.15.32:0/2656383592 learned_addr learned my addr 172.21.15.32:0/2656383592 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:51.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8a9e00640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8a40b4b20 0x7fb8a415a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8a9400640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8a415ffa0 0x7fb8a4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 -- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8a415ffa0 msgr2=0x7fb8a4158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8a415ffa0 0x7fb8a4158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 -- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8a40b4b20 msgr2=0x7fb8a415a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8a40b4b20 0x7fb8a415a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.496+0000 7fb8aa800640 1 -- 172.21.15.32:0/2656383592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a4160810 con 0x7fb8a41589c0 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.497+0000 7fb8aa800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 0x7fb8a4161590 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000b970 tx=0x7fb8a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.497+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0019070 con 0x7fb8a41589c0 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.497+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a000cb40 con 0x7fb8a41589c0 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.497+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0014440 con 0x7fb8a41589c0 2024-03-06T07:34:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.497+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a4160ab0 con 0x7fb8a41589c0 2024-03-06T07:34:51.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.497+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a41572c0 con 0x7fb8a41589c0 2024-03-06T07:34:51.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.498+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a40692a0 con 0x7fb8a41589c0 2024-03-06T07:34:51.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.500+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8a001c070 con 0x7fb8a41589c0 2024-03-06T07:34:51.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.500+0000 7fb89e000640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb88806b4d0 0x7fb88806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.501+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb8a0089ea0 con 0x7fb8a41589c0 2024-03-06T07:34:51.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.501+0000 7fb8a9e00640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb88806b4d0 0x7fb88806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.502+0000 7fb8a9e00640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb88806b4d0 0x7fb88806d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb8980011e0 tx=0x7fb898009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.505+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8a0058cb0 con 0x7fb8a41589c0 2024-03-06T07:34:51.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.745+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8a40b1130 con 0x7fb8a41589c0 2024-03-06T07:34:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.746+0000 7fb89e000640 1 -- 172.21.15.32:0/2656383592 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fb8a00585b0 con 0x7fb8a41589c0 2024-03-06T07:34:51.749 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:51.749 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:51.749 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb88806b4d0 msgr2=0x7fb88806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb88806b4d0 0x7fb88806d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb8980011e0 tx=0x7fb898009040 comp rx=0 tx=0).stop 2024-03-06T07:34:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 msgr2=0x7fb8a4161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 0x7fb8a4161590 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000b970 tx=0x7fb8a000be30 comp rx=0 tx=0).stop 2024-03-06T07:34:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 shutdown_connections 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb88806b4d0 0x7fb88806d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a41589c0 0x7fb8a4161590 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8a415ffa0 0x7fb8a4158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 --2- 172.21.15.32:0/2656383592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8a40b4b20 0x7fb8a415a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 >> 172.21.15.32:0/2656383592 conn(0x7fb8a406cf40 msgr2=0x7fb8a40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 shutdown_connections 2024-03-06T07:34:51.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:51.749+0000 7fb8ac800640 1 -- 172.21.15.32:0/2656383592 wait complete. 2024-03-06T07:34:51.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:51.973 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":873,"msgr_send_messages":870,"msgr_recv_bytes":328729,"msgr_send_bytes":4512242,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.135742969,"msgr_running_send_time":0.061742855,"msgr_running_recv_time":0.034628635,"msgr_running_fast_dispatch_time":0.030546864,"msgr_send_messages_queue_lat":{"avgcount":869,"sum":0.128269764,"avgtime":0.000147606},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":106080,"msgr_send_encrypted_bytes":8544},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13012,"msgr_send_messages":24214,"msgr_recv_bytes":4488389,"msgr_send_bytes":31147077,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.019603609,"msgr_running_send_time":1.990154374,"msgr_running_recv_time":1.767391304,"msgr_running_fast_dispatch_time":0.049265818,"msgr_send_messages_queue_lat":{"avgcount":24213,"sum":2.134113174,"avgtime":0.000088139},"msgr_handle_ack_lat":{"avgcount":27175,"sum":0.012754941,"avgtime":0.000000469},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":98080},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1671,"msgr_send_messages":1673,"msgr_recv_bytes":758964,"msgr_send_bytes":4300514,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.237411958,"msgr_running_send_time":0.115694533,"msgr_running_recv_time":0.049501840,"msgr_running_fast_dispatch_time":0.055388087,"msgr_send_messages_queue_lat":{"avgcount":1672,"sum":0.241292169,"avgtime":0.000144313},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":109,"sum":12.583677966,"avgtime":0.115446586}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5458,"sum":3.570290921,"avgtime":0.000654139}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":665.601454421,"avgtime":0.103837980},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":137,"inodes_top":0,"inodes_bottom":137,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":148,"ceph_cap_op_grant":7248,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5042,"handle_client_caps_dirty":3000,"handle_client_cap_release":533,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15566,"evex":15442,"evtrm":15442,"ev":129,"evexg":0,"evexd":15441,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":34569750,"wrpos":34650267,"rdpos":4195674,"jlat":{"avgcount":24,"sum":8.483424523,"avgtime":0.353476021},"replayed":5},"mds_mem":{"ino":139,"ino+":3214,"ino-":3075,"dir":13,"dir+":13,"dir-":0,"dn":137,"dn+":6412,"dn-":6275,"cap":1,"cap+":3202,"cap-":3201,"rss":103300,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.585629125,"avgtime":0.585629125},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.257627915,"avgtime":0.002580508},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5718,"average_load":5718,"avg_session_uptime":1175,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":246893,"buffer_anon_items":216,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":440209,"mds_co_items":8985,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3514,"op_send_bytes":31347302,"op_resend":0,"op_reply":3514,"op_latency":{"avgcount":3514,"sum":15.765675118,"avgtime":0.004486532},"op_inflight":0,"oplen_avg":{"avgcount":3514,"sum":3870},"op":3514,"op_r":60,"op_w":3454,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":137,"osdop_read":32,"osdop_write":48,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":274,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":45,"omap_rd":48,"omap_del":9},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15552,"get_sum":4173669,"get_or_fail_fail":0,"get_or_fail_success":15552,"take":0,"take_sum":0,"put":15552,"put_sum":4173669,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3514,"take_sum":31672524,"put":3514,"put_sum":31672524,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3514,"take_sum":3514,"put":3514,"put_sum":3514,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":24,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":76323,"max":3758096384,"get_started":0,"get":15566,"get_sum":30453778,"get_or_fail_fail":0,"get_or_fail_success":15566,"take":0,"take_sum":0,"put":24,"put_sum":30377455,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:51.975 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 328729, "msgr_recv_encrypted_bytes": 106080, "msgr_recv_messages": 873, "msgr_running_fast_dispatch_time": 0.030546864, "msgr_running_recv_time": 0.034628635, "msgr_running_send_time": 0.061742855, "msgr_running_total_time": 0.135742969, "msgr_send_bytes": 4512242, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 870, "msgr_send_messages_queue_lat": { "avgcount": 869, "avgtime": 0.000147606, "sum": 0.128269764 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27175, "avgtime": 4.69e-07, "sum": 0.012754941 }, "msgr_recv_bytes": 4488389, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 13012, "msgr_running_fast_dispatch_time": 0.049265818, "msgr_running_recv_time": 1.767391304, "msgr_running_send_time": 1.990154374, "msgr_running_total_time": 3.019603609, "msgr_send_bytes": 31147077, "msgr_send_encrypted_bytes": 98080, "msgr_send_messages": 24214, "msgr_send_messages_queue_lat": { "avgcount": 24213, "avgtime": 8.8139e-05, "sum": 2.134113174 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 758964, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1671, "msgr_running_fast_dispatch_time": 0.055388087, "msgr_running_recv_time": 0.04950184, "msgr_running_send_time": 0.115694533, "msgr_running_total_time": 0.237411958, "msgr_send_bytes": 4300514, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1673, "msgr_send_messages_queue_lat": { "avgcount": 1672, "avgtime": 0.000144313, "sum": 0.241292169 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 109, "avgtime": 0.115446586, "sum": 12.583677966 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5458, "avgtime": 0.000654139, "sum": 3.570290921 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7248, "ceph_cap_op_revoke": 148, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 533, "handle_client_caps": 5042, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 137, "inodes_bottom": 137, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.10383798, "sum": 665.601454421 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 129, "evadd": 15566, "evex": 15442, "evexd": 15441, "evexg": 0, "evlrg": 0, "evtrm": 15442, "expos": 34569750, "jlat": { "avgcount": 24, "avgtime": 0.353476021, "sum": 8.483424523 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 34650267 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 137, "dn+": 6412, "dn-": 6275, "heap": 248164, "ino": 139, "ino+": 3214, "ino-": 3075, "rss": 103300 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.585629125, "sum": 0.585629125 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002580508, "sum": 8.257627915 } }, "mds_sessions": { "average_load": 5718, "avg_session_uptime": 1175, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5718 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 246893, "buffer_anon_items": 216, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 440209, "mds_co_items": 8985, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 9, "omap_rd": 48, "omap_wr": 45, "op": 3514, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3514, "avgtime": 0.004486532, "sum": 15.765675118 }, "op_pg": 0, "op_r": 60, "op_reply": 3514, "op_resend": 0, "op_rmw": 0, "op_send": 3514, "op_send_bytes": 31347302, "op_w": 3454, "oplen_avg": { "avgcount": 3514, "sum": 3870 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 137, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 32, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 274, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15552, "get_or_fail_fail": 0, "get_or_fail_success": 15552, "get_started": 0, "get_sum": 4173669, "max": 104857600, "put": 15552, "put_sum": 4173669, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3514, "put_sum": 31672524, "take": 3514, "take_sum": 31672524, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3514, "put_sum": 3514, "take": 3514, "take_sum": 3514, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 24, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 15566, "get_or_fail_fail": 0, "get_or_fail_success": 15566, "get_started": 0, "get_sum": 30453778, "max": 3758096384, "put": 24, "put_sum": 30377455, "take": 0, "take_sum": 0, "val": 76323, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:51.975 INFO:tasks.cephfs.test_strays:Complete purge in 8 seconds 2024-03-06T07:34:51.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.184+0000 7f1025400640 1 Processor -- start 2024-03-06T07:34:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.184+0000 7f1025400640 1 -- start start 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1025400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1020156340 0x7f102006d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1025400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f102006dec0 0x7f10200cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1025400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10200cf310 0x7f10200ad3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1025400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1020160830 con 0x7f10200cf310 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1025400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102015fa70 con 0x7f1020156340 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1025400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102015fd50 con 0x7f102006dec0 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.187+0000 7f1024a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10200cf310 0x7f10200ad3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10200cf310 0x7f10200ad3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33550/0 (socket says 172.21.15.32:33550) 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 -- 172.21.15.32:0/1918966772 learned_addr learned my addr 172.21.15.32:0/1918966772 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1017e00640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f102006dec0 0x7f10200cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f101fe00640 1 --2- 172.21.15.32:0/1918966772 >> v2:172.21.15.39:3300/0 conn(0x7f1020156340 0x7f102006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 -- 172.21.15.32:0/1918966772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f102006dec0 msgr2=0x7f10200cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:52.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f102006dec0 0x7f10200cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 -- 172.21.15.32:0/1918966772 >> v2:172.21.15.39:3300/0 conn(0x7f1020156340 msgr2=0x7f102006d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:52.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 --2- 172.21.15.32:0/1918966772 >> v2:172.21.15.39:3300/0 conn(0x7f1020156340 0x7f102006d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1024a00640 1 -- 172.21.15.32:0/1918966772 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1020160030 con 0x7f10200cf310 2024-03-06T07:34:52.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.188+0000 7f1017e00640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f102006dec0 0x7f10200cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:52.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.189+0000 7f1024a00640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200ad3d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f1020074990 tx=0x7f101802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e8a415ef37035b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.190+0000 7f101f400640 1 -- 172.21.15.32:0/1918966772 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101802c9f0 con 0x7f10200cf310 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.190+0000 7f101f400640 1 -- 172.21.15.32:0/1918966772 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101802cb90 con 0x7f10200cf310 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.190+0000 7f101f400640 1 -- 172.21.15.32:0/1918966772 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10180343f0 con 0x7f10200cf310 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 -- 172.21.15.32:0/1918966772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 msgr2=0x7f10200ad3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200ad3d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f1020074990 tx=0x7f101802be50 comp rx=0 tx=0).stop 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 -- 172.21.15.32:0/1918966772 shutdown_connections 2024-03-06T07:34:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200ad3d0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 --2- 172.21.15.32:0/1918966772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f102006dec0 0x7f10200cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 --2- 172.21.15.32:0/1918966772 >> v2:172.21.15.39:3300/0 conn(0x7f1020156340 0x7f102006d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 -- 172.21.15.32:0/1918966772 >> 172.21.15.32:0/1918966772 conn(0x7f10200b1990 msgr2=0x7f10200b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 -- 172.21.15.32:0/1918966772 shutdown_connections 2024-03-06T07:34:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.191+0000 7f1025400640 1 -- 172.21.15.32:0/1918966772 wait complete. 2024-03-06T07:34:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.194+0000 7f1025400640 1 Processor -- start 2024-03-06T07:34:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.194+0000 7f1025400640 1 -- start start 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.195+0000 7f1025400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102006dec0 0x7f10200717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.195+0000 7f1025400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1025400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1020156340 0x7f10200ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1025400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10200cf720 con 0x7f10200cf310 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1025400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020156720 con 0x7f102006dec0 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1025400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10200cf8a0 con 0x7f1020156340 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33562/0 (socket says 172.21.15.32:33562) 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 -- 172.21.15.32:0/3965640240 learned_addr learned my addr 172.21.15.32:0/3965640240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f101fe00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102006dec0 0x7f10200717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1024a00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1020156340 0x7f10200ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 -- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1020156340 msgr2=0x7f10200ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1020156340 0x7f10200ad2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 -- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102006dec0 msgr2=0x7f10200717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102006dec0 0x7f10200717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.196+0000 7f1017e00640 1 -- 172.21.15.32:0/3965640240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10201f3d70 con 0x7f10200cf310 2024-03-06T07:34:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.197+0000 7f1017e00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200acd70 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f100800d7d0 tx=0x7f100800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.197+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008004030 con 0x7f10200cf310 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.197+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10080041d0 con 0x7f10200cf310 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.197+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10201f4010 con 0x7f10200cf310 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.197+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100800b6c0 con 0x7f10200cf310 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.197+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10201f45d0 con 0x7f10200cf310 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.198+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102004f330 con 0x7f10200cf310 2024-03-06T07:34:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.199+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f100800b860 con 0x7f10200cf310 2024-03-06T07:34:52.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.200+0000 7f101cc00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.200+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f100808a4e0 con 0x7f10200cf310 2024-03-06T07:34:52.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.200+0000 7f101fe00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.201+0000 7f101fe00640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1010006fd0 tx=0x7f1010008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:52.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.205+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10080592f0 con 0x7f10200cf310 2024-03-06T07:34:52.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.445+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10200b2790 con 0x7f10200cf310 2024-03-06T07:34:52.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.446+0000 7f101cc00640 1 -- 172.21.15.32:0/3965640240 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1008058bf0 con 0x7f10200cf310 2024-03-06T07:34:52.448 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:52.449 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:52.449 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:52.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ffc06b4d0 msgr2=0x7f0ffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1010006fd0 tx=0x7f1010008040 comp rx=0 tx=0).stop 2024-03-06T07:34:52.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 msgr2=0x7f10200acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200acd70 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f100800d7d0 tx=0x7f100800dc90 comp rx=0 tx=0).stop 2024-03-06T07:34:52.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 shutdown_connections 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1020156340 0x7f10200ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10200cf310 0x7f10200acd70 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 --2- 172.21.15.32:0/3965640240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102006dec0 0x7f10200717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 >> 172.21.15.32:0/3965640240 conn(0x7f10200b1990 msgr2=0x7f1020153930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 shutdown_connections 2024-03-06T07:34:52.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:52.449+0000 7f1025400640 1 -- 172.21.15.32:0/3965640240 wait complete. 2024-03-06T07:34:52.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:34:52.702 INFO:teuthology.orchestra.run.smithi032.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":873,"msgr_send_messages":870,"msgr_recv_bytes":328729,"msgr_send_bytes":4512242,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.135742969,"msgr_running_send_time":0.061742855,"msgr_running_recv_time":0.034628635,"msgr_running_fast_dispatch_time":0.030546864,"msgr_send_messages_queue_lat":{"avgcount":869,"sum":0.128269764,"avgtime":0.000147606},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000229,"avgtime":0.000000114},"msgr_recv_encrypted_bytes":106080,"msgr_send_encrypted_bytes":8544},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13012,"msgr_send_messages":24214,"msgr_recv_bytes":4488389,"msgr_send_bytes":31147077,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.019603609,"msgr_running_send_time":1.990154374,"msgr_running_recv_time":1.767391304,"msgr_running_fast_dispatch_time":0.049265818,"msgr_send_messages_queue_lat":{"avgcount":24213,"sum":2.134113174,"avgtime":0.000088139},"msgr_handle_ack_lat":{"avgcount":27175,"sum":0.012754941,"avgtime":0.000000469},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":98080},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1671,"msgr_send_messages":1673,"msgr_recv_bytes":758964,"msgr_send_bytes":4300514,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.237411958,"msgr_running_send_time":0.115694533,"msgr_running_recv_time":0.049501840,"msgr_running_fast_dispatch_time":0.055388087,"msgr_send_messages_queue_lat":{"avgcount":1672,"sum":0.241292169,"avgtime":0.000144313},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":109,"sum":12.583677966,"avgtime":0.115446586}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5458,"sum":3.570290921,"avgtime":0.000654139}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":665.601454421,"avgtime":0.103837980},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":137,"inodes_top":0,"inodes_bottom":137,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":148,"ceph_cap_op_grant":7248,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5042,"handle_client_caps_dirty":3000,"handle_client_cap_release":533,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15566,"evex":15442,"evtrm":15442,"ev":129,"evexg":0,"evexd":15441,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":34569750,"wrpos":34650267,"rdpos":4195674,"jlat":{"avgcount":24,"sum":8.483424523,"avgtime":0.353476021},"replayed":5},"mds_mem":{"ino":139,"ino+":3214,"ino-":3075,"dir":12,"dir+":13,"dir-":1,"dn":137,"dn+":6412,"dn-":6275,"cap":1,"cap+":3202,"cap-":3201,"rss":103300,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":654.875710611,"avgtime":0.204648659},"req_getattr_latency":{"avgcount":2,"sum":0.324833208,"avgtime":0.162416604},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005816535,"avgtime":0.005816535},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":1.551837027,"avgtime":0.258639504},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.585629125,"avgtime":0.585629125},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.257627915,"avgtime":0.002580508},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5718,"average_load":5718,"avg_session_uptime":1175,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":246893,"buffer_anon_items":216,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":440209,"mds_co_items":8985,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3514,"op_send_bytes":31347302,"op_resend":0,"op_reply":3514,"op_latency":{"avgcount":3514,"sum":15.765675118,"avgtime":0.004486532},"op_inflight":0,"oplen_avg":{"avgcount":3514,"sum":3870},"op":3514,"op_r":60,"op_w":3454,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":137,"osdop_read":32,"osdop_write":48,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":274,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":45,"omap_rd":48,"omap_del":9},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15552,"get_sum":4173669,"get_or_fail_fail":0,"get_or_fail_success":15552,"take":0,"take_sum":0,"put":15552,"put_sum":4173669,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3514,"take_sum":31672524,"put":3514,"put_sum":31672524,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3514,"take_sum":3514,"put":3514,"put_sum":3514,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":24,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55d5fa546820":{"val":76323,"max":3758096384,"get_started":0,"get":15566,"get_sum":30453778,"get_or_fail_fail":0,"get_or_fail_success":15566,"take":0,"take_sum":0,"put":24,"put_sum":30377455,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:34:52.704 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.14e-07, "sum": 2.29e-07 }, "msgr_recv_bytes": 328729, "msgr_recv_encrypted_bytes": 106080, "msgr_recv_messages": 873, "msgr_running_fast_dispatch_time": 0.030546864, "msgr_running_recv_time": 0.034628635, "msgr_running_send_time": 0.061742855, "msgr_running_total_time": 0.135742969, "msgr_send_bytes": 4512242, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 870, "msgr_send_messages_queue_lat": { "avgcount": 869, "avgtime": 0.000147606, "sum": 0.128269764 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27175, "avgtime": 4.69e-07, "sum": 0.012754941 }, "msgr_recv_bytes": 4488389, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 13012, "msgr_running_fast_dispatch_time": 0.049265818, "msgr_running_recv_time": 1.767391304, "msgr_running_send_time": 1.990154374, "msgr_running_total_time": 3.019603609, "msgr_send_bytes": 31147077, "msgr_send_encrypted_bytes": 98080, "msgr_send_messages": 24214, "msgr_send_messages_queue_lat": { "avgcount": 24213, "avgtime": 8.8139e-05, "sum": 2.134113174 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 758964, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1671, "msgr_running_fast_dispatch_time": 0.055388087, "msgr_running_recv_time": 0.04950184, "msgr_running_send_time": 0.115694533, "msgr_running_total_time": 0.237411958, "msgr_send_bytes": 4300514, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1673, "msgr_send_messages_queue_lat": { "avgcount": 1672, "avgtime": 0.000144313, "sum": 0.241292169 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 109, "avgtime": 0.115446586, "sum": 12.583677966 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5458, "avgtime": 0.000654139, "sum": 3.570290921 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7248, "ceph_cap_op_revoke": 148, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 533, "handle_client_caps": 5042, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 137, "inodes_bottom": 137, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.10383798, "sum": 665.601454421 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 129, "evadd": 15566, "evex": 15442, "evexd": 15441, "evexg": 0, "evlrg": 0, "evtrm": 15442, "expos": 34569750, "jlat": { "avgcount": 24, "avgtime": 0.353476021, "sum": 8.483424523 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 34650267 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 137, "dn+": 6412, "dn-": 6275, "heap": 248164, "ino": 139, "ino+": 3214, "ino-": 3075, "rss": 103300 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.204648659, "sum": 654.875710611 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.162416604, "sum": 0.324833208 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.005816535, "sum": 0.005816535 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.258639504, "sum": 1.551837027 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.585629125, "sum": 0.585629125 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002580508, "sum": 8.257627915 } }, "mds_sessions": { "average_load": 5718, "avg_session_uptime": 1175, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5718 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 246893, "buffer_anon_items": 216, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 440209, "mds_co_items": 8985, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 9, "omap_rd": 48, "omap_wr": 45, "op": 3514, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3514, "avgtime": 0.004486532, "sum": 15.765675118 }, "op_pg": 0, "op_r": 60, "op_reply": 3514, "op_resend": 0, "op_rmw": 0, "op_send": 3514, "op_send_bytes": 31347302, "op_w": 3454, "oplen_avg": { "avgcount": 3514, "sum": 3870 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 137, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 32, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 274, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15552, "get_or_fail_fail": 0, "get_or_fail_success": 15552, "get_started": 0, "get_sum": 4173669, "max": 104857600, "put": 15552, "put_sum": 4173669, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3514, "put_sum": 31672524, "take": 3514, "take_sum": 31672524, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3514, "put_sum": 3514, "take": 3514, "take_sum": 3514, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 24, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55d5fa546820": { "get": 15566, "get_or_fail_fail": 0, "get_or_fail_success": 15566, "get_started": 0, "get_sum": 30453778, "max": 3758096384, "put": 24, "put_sum": 30377455, "take": 0, "take_sum": 0, "val": 76323, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:34:52.704 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:34:52.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:52.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:52.939 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:34:52.940 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:34:52.940 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:52.995 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:34:52.995 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:53.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.262+0000 7f9be7600640 1 Processor -- start 2024-03-06T07:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.262+0000 7f9be7600640 1 -- start start 2024-03-06T07:34:53.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be7600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9be00b5870 0x7f9be0166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be7600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be0145bc0 con 0x7f9be015fba0 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be7600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be0160040 con 0x7f9be01457e0 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be7600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be0160320 con 0x7f9be00b5870 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be5600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9be00b5870 0x7f9be0166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be5600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9be00b5870 0x7f9be0166f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33568/0 (socket says 172.21.15.32:33568) 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.264+0000 7f9be5600640 1 -- 172.21.15.32:0/4186403302 learned_addr learned my addr 172.21.15.32:0/4186403302 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.265+0000 7f9be5600640 1 -- 172.21.15.32:0/4186403302 >> v1:172.21.15.39:6789/0 conn(0x7f9be01457e0 legacy=0x7f9be00cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.265+0000 7f9bdfe00640 1 -- 172.21.15.32:0/4186403302 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489769784 0 0) 0x7f9be0145bc0 con 0x7f9be015fba0 2024-03-06T07:34:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.265+0000 7f9be5600640 1 -- 172.21.15.32:0/4186403302 >> v1:172.21.15.32:6789/0 conn(0x7f9be015fba0 legacy=0x7f9be00b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.265+0000 7f9be5600640 1 -- 172.21.15.32:0/4186403302 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be0160600 con 0x7f9be00b5870 2024-03-06T07:34:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.267+0000 7f9be5600640 1 --2- 172.21.15.32:0/4186403302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be0166f30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009e50 tx=0x7f9bd802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9eb5bd08e07fdd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.267+0000 7f9bdfe00640 1 -- 172.21.15.32:0/4186403302 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd802cb20 con 0x7f9be00b5870 2024-03-06T07:34:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.267+0000 7f9bdfe00640 1 -- 172.21.15.32:0/4186403302 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd802ccc0 con 0x7f9be00b5870 2024-03-06T07:34:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.267+0000 7f9bdfe00640 1 -- 172.21.15.32:0/4186403302 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8034610 con 0x7f9be00b5870 2024-03-06T07:34:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.268+0000 7f9be7600640 1 -- 172.21.15.32:0/4186403302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 msgr2=0x7f9be0166f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.268+0000 7f9be7600640 1 --2- 172.21.15.32:0/4186403302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be0166f30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009e50 tx=0x7f9bd802bf80 comp rx=0 tx=0).stop 2024-03-06T07:34:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.268+0000 7f9be7600640 1 -- 172.21.15.32:0/4186403302 shutdown_connections 2024-03-06T07:34:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.268+0000 7f9be7600640 1 --2- 172.21.15.32:0/4186403302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be0166f30 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.268+0000 7f9be7600640 1 -- 172.21.15.32:0/4186403302 >> 172.21.15.32:0/4186403302 conn(0x7f9be006cf40 msgr2=0x7f9be006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.268+0000 7f9be7600640 1 -- 172.21.15.32:0/4186403302 shutdown_connections 2024-03-06T07:34:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.269+0000 7f9be7600640 1 -- 172.21.15.32:0/4186403302 wait complete. 2024-03-06T07:34:53.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.271+0000 7f9be7600640 1 Processor -- start 2024-03-06T07:34:53.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.271+0000 7f9be7600640 1 -- start start 2024-03-06T07:34:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be01f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be7600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be01457e0 0x7f9be0132340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be7600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9be015fba0 0x7f9be00cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be7600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00720c0 con 0x7f9be00b5870 2024-03-06T07:34:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be7600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00cca70 con 0x7f9be01457e0 2024-03-06T07:34:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be7600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be0072240 con 0x7f9be015fba0 2024-03-06T07:34:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be01f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be01f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33574/0 (socket says 172.21.15.32:33574) 2024-03-06T07:34:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.273+0000 7f9be4c00640 1 -- 172.21.15.32:0/1650672361 learned_addr learned my addr 172.21.15.32:0/1650672361 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be4c00640 1 -- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9be015fba0 msgr2=0x7f9be00cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be4c00640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9be015fba0 0x7f9be00cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be4c00640 1 -- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be01457e0 msgr2=0x7f9be0132340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9bd7e00640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be01457e0 0x7f9be0132340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be4c00640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be01457e0 0x7f9be0132340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be4c00640 1 -- 172.21.15.32:0/1650672361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be0132a80 con 0x7f9be00b5870 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be4c00640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be01f5bf0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000c450 tx=0x7f9bd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00052d0 con 0x7f9be00b5870 2024-03-06T07:34:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd0005470 con 0x7f9be00b5870 2024-03-06T07:34:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0012870 con 0x7f9be00b5870 2024-03-06T07:34:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be0132d60 con 0x7f9be00b5870 2024-03-06T07:34:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.274+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be0130490 con 0x7f9be00b5870 2024-03-06T07:34:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.276+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be0133280 con 0x7f9be00b5870 2024-03-06T07:34:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.276+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bd0012a10 con 0x7f9be00b5870 2024-03-06T07:34:53.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.277+0000 7f9bdd600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bbc06b4d0 0x7f9bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.277+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9bd0089f10 con 0x7f9be00b5870 2024-03-06T07:34:53.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.278+0000 7f9bd7e00640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bbc06b4d0 0x7f9bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.278+0000 7f9bd7e00640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bbc06b4d0 0x7f9bbc06d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8000f60 tx=0x7f9bc8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.282+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9be0133280 con 0x7f9be00b5870 2024-03-06T07:34:53.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.643+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9be00692a0 con 0x7f9be00b5870 2024-03-06T07:34:53.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.646+0000 7f9bdd600640 1 -- 172.21.15.32:0/1650672361 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f9bd0058c90 con 0x7f9be00b5870 2024-03-06T07:34:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:53.648 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:25.427369+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/556051904":"2024-03-07T07:34:25.418325+0000","172.21.15.39:6833/556051904":"2024-03-07T07:34:25.418325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.648+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bbc06b4d0 msgr2=0x7f9bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.648+0000 7f9be7600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bbc06b4d0 0x7f9bbc06d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8000f60 tx=0x7f9bc8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.648+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 msgr2=0x7f9be01f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.648+0000 7f9be7600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be01f5bf0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000c450 tx=0x7f9bd000c910 comp rx=0 tx=0).stop 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 shutdown_connections 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9bbc06b4d0 0x7f9bbc06d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9be015fba0 0x7f9be00cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be01457e0 0x7f9be0132340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 --2- 172.21.15.32:0/1650672361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9be00b5870 0x7f9be01f5bf0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 >> 172.21.15.32:0/1650672361 conn(0x7f9be006cf40 msgr2=0x7f9be0144aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 shutdown_connections 2024-03-06T07:34:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.649+0000 7f9be7600640 1 -- 172.21.15.32:0/1650672361 wait complete. 2024-03-06T07:34:53.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:06.595209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:07.760130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:53.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:53.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.886+0000 7f9717e00640 1 Processor -- start 2024-03-06T07:34:53.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.886+0000 7f9717e00640 1 -- start start 2024-03-06T07:34:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9717e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f971015e030 0x7f9710161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9717e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971006b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9717e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710162d70 con 0x7f971006be10 2024-03-06T07:34:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9717e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97101627b0 con 0x7f971015e030 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9717e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9710162a90 con 0x7f9710162330 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9715400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f971015e030 0x7f9710161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9715e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f971006be10 0x7f97100ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55756/0 (socket says 172.21.15.32:55756) 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.888+0000 7f9715e00640 1 -- 172.21.15.32:0/1262673230 learned_addr learned my addr 172.21.15.32:0/1262673230 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.889+0000 7f9714a00640 1 --2- 172.21.15.32:0/1262673230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971006b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.889+0000 7f9715400640 1 -- 172.21.15.32:0/1262673230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 msgr2=0x7f971006b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.889+0000 7f9715400640 1 --2- 172.21.15.32:0/1262673230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971006b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.889+0000 7f9715400640 1 -- 172.21.15.32:0/1262673230 >> v1:172.21.15.32:6789/0 conn(0x7f971006be10 legacy=0x7f97100ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.889+0000 7f970fe00640 1 -- 172.21.15.32:0/1262673230 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303781581 0 0) 0x7f9710162d70 con 0x7f971006be10 2024-03-06T07:34:53.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.889+0000 7f9715400640 1 -- 172.21.15.32:0/1262673230 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971006b0d0 con 0x7f971015e030 2024-03-06T07:34:53.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.890+0000 7f9715400640 1 --2- 172.21.15.32:0/1262673230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f9710161df0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f97040097a0 tx=0x7f970402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9499020343f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.890+0000 7f970fe00640 1 -- 172.21.15.32:0/1262673230 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704035020 con 0x7f971015e030 2024-03-06T07:34:53.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f970fe00640 1 -- 172.21.15.32:0/1262673230 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704031830 con 0x7f971015e030 2024-03-06T07:34:53.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f970fe00640 1 -- 172.21.15.32:0/1262673230 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704031c00 con 0x7f971015e030 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 -- 172.21.15.32:0/1262673230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 msgr2=0x7f9710161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 --2- 172.21.15.32:0/1262673230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f9710161df0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f97040097a0 tx=0x7f970402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 -- 172.21.15.32:0/1262673230 shutdown_connections 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 --2- 172.21.15.32:0/1262673230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971006b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 --2- 172.21.15.32:0/1262673230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f9710161df0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 -- 172.21.15.32:0/1262673230 >> 172.21.15.32:0/1262673230 conn(0x7f97100b2670 msgr2=0x7f97100b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 -- 172.21.15.32:0/1262673230 shutdown_connections 2024-03-06T07:34:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.891+0000 7f9717e00640 1 -- 172.21.15.32:0/1262673230 wait complete. 2024-03-06T07:34:53.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.893+0000 7f9717e00640 1 Processor -- start 2024-03-06T07:34:53.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.893+0000 7f9717e00640 1 -- start start 2024-03-06T07:34:53.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9717e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f97101f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9717e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9717e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f971014cc40 0x7f971014d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9717e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100ae140 con 0x7f971015e030 2024-03-06T07:34:53.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9717e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100adfc0 con 0x7f971014cc40 2024-03-06T07:34:53.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9717e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971015e410 con 0x7f9710162330 2024-03-06T07:34:53.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9715400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f97101f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9715400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f97101f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33590/0 (socket says 172.21.15.32:33590) 2024-03-06T07:34:53.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.894+0000 7f9715400640 1 -- 172.21.15.32:0/3242468368 learned_addr learned my addr 172.21.15.32:0/3242468368 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:53.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9714a00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f971014cc40 0x7f971014d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715400640 1 -- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 msgr2=0x7f971015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715400640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971015c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715400640 1 -- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f971014cc40 msgr2=0x7f971014d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715400640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f971014cc40 0x7f971014d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715400640 1 -- 172.21.15.32:0/3242468368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97100af470 con 0x7f971015e030 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9714a00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971015c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f971014cc40 0x7f971014d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f9715400640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f97101f3f70 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f9704000c00 tx=0x7f97040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704035070 con 0x7f971015e030 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704004230 con 0x7f971015e030 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.895+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970403a6f0 con 0x7f971015e030 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.896+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97100af710 con 0x7f971015e030 2024-03-06T07:34:53.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.896+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97100afcd0 con 0x7f971015e030 2024-03-06T07:34:53.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.897+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9710068070 con 0x7f971015e030 2024-03-06T07:34:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.899+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9704038020 con 0x7f971015e030 2024-03-06T07:34:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.900+0000 7f970d600640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f96f006b4d0 0x7f96f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.900+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f97040abff0 con 0x7f971015e030 2024-03-06T07:34:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.900+0000 7f9714a00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f96f006b4d0 0x7f96f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.900+0000 7f9714a00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f96f006b4d0 0x7f96f006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f96fc006fd0 tx=0x7f96fc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:53.909+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9704040070 con 0x7f971015e030 2024-03-06T07:34:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.170+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f971006c550 con 0x7f971015e030 2024-03-06T07:34:54.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.171+0000 7f970d600640 1 -- 172.21.15.32:0/3242468368 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9704031800 con 0x7f971015e030 2024-03-06T07:34:54.173 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:34:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.174+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f96f006b4d0 msgr2=0x7f96f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.174+0000 7f9717e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f96f006b4d0 0x7f96f006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f96fc006fd0 tx=0x7f96fc008040 comp rx=0 tx=0).stop 2024-03-06T07:34:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.174+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 msgr2=0x7f97101f3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.174+0000 7f9717e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f97101f3f70 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f9704000c00 tx=0x7f97040046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 shutdown_connections 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f96f006b4d0 0x7f96f006d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f971014cc40 0x7f971014d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9710162330 0x7f971015c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 --2- 172.21.15.32:0/3242468368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f971015e030 0x7f97101f3f70 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 >> 172.21.15.32:0/3242468368 conn(0x7f97100b2670 msgr2=0x7f97100ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 shutdown_connections 2024-03-06T07:34:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.175+0000 7f9717e00640 1 -- 172.21.15.32:0/3242468368 wait complete. 2024-03-06T07:34:54.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:54.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.402+0000 7fa17f200640 1 Processor -- start 2024-03-06T07:34:54.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.402+0000 7fa17f200640 1 -- start start 2024-03-06T07:34:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.404+0000 7fa17f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18006d500 0x7fa18006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.404+0000 7fa17f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa180072110 0x7fa1800725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.404+0000 7fa17f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18006d9e0 con 0x7fa180144d90 2024-03-06T07:34:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.404+0000 7fa17f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa180145170 con 0x7fa180072110 2024-03-06T07:34:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.404+0000 7fa17f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18006dcc0 con 0x7fa18006d500 2024-03-06T07:34:54.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa180072110 0x7fa1800725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa175400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18006d500 0x7fa18006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa180072110 0x7fa1800725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:35494/0 (socket says 172.21.15.32:35494) 2024-03-06T07:34:54.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 -- 172.21.15.32:0/865940914 learned_addr learned my addr 172.21.15.32:0/865940914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:54.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 -- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18006d500 msgr2=0x7fa18006ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:54.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 --2- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18006d500 0x7fa18006ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 -- 172.21.15.32:0/865940914 >> v1:172.21.15.39:6789/0 conn(0x7fa180144d90 legacy=0x7fa1800cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa17e800640 1 -- 172.21.15.32:0/865940914 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18006dfa0 con 0x7fa180072110 2024-03-06T07:34:54.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.405+0000 7fa175400640 1 --2- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18006d500 0x7fa18006ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:54.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.407+0000 7fa17e800640 1 --2- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180072110 0x7fa1800725c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa178009e50 tx=0x7fa17802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a17a7110556571b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.407+0000 7fa17d400640 1 -- 172.21.15.32:0/865940914 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17802cb00 con 0x7fa180072110 2024-03-06T07:34:54.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.407+0000 7fa17d400640 1 -- 172.21.15.32:0/865940914 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17802cca0 con 0x7fa180072110 2024-03-06T07:34:54.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.407+0000 7fa17d400640 1 -- 172.21.15.32:0/865940914 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178034610 con 0x7fa180072110 2024-03-06T07:34:54.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 -- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180072110 msgr2=0x7fa1800725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 --2- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180072110 0x7fa1800725c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa178009e50 tx=0x7fa17802bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:54.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 -- 172.21.15.32:0/865940914 shutdown_connections 2024-03-06T07:34:54.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 --2- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180072110 0x7fa1800725c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 --2- 172.21.15.32:0/865940914 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa18006d500 0x7fa18006ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 -- 172.21.15.32:0/865940914 >> 172.21.15.32:0/865940914 conn(0x7fa18006b3e0 msgr2=0x7fa18006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 -- 172.21.15.32:0/865940914 shutdown_connections 2024-03-06T07:34:54.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.408+0000 7fa17f200640 1 -- 172.21.15.32:0/865940914 wait complete. 2024-03-06T07:34:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.410+0000 7fa17f200640 1 Processor -- start 2024-03-06T07:34:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.410+0000 7fa17f200640 1 -- start start 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 0x7fa18015f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa180072110 0x7fa1800c4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180144d90 0x7fa1800c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa180071970 con 0x7fa18006d500 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18006e560 con 0x7fa180144d90 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18015c510 con 0x7fa180072110 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.411+0000 7fa17de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 0x7fa18015f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 0x7fa18015f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33598/0 (socket says 172.21.15.32:33598) 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 -- 172.21.15.32:0/1174254685 learned_addr learned my addr 172.21.15.32:0/1174254685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa175400640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa180072110 0x7fa1800c4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17e800640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180144d90 0x7fa1800c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 -- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa180072110 msgr2=0x7fa1800c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa180072110 0x7fa1800c4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 -- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180144d90 msgr2=0x7fa1800c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180144d90 0x7fa1800c5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17de00640 1 -- 172.21.15.32:0/1174254685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1801317d0 con 0x7fa18006d500 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa17e800640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180144d90 0x7fa1800c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.412+0000 7fa175400640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa180072110 0x7fa1800c4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.413+0000 7fa17de00640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 0x7fa18015f1a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c3c0 tx=0x7fa17000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.413+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1700052d0 con 0x7fa18006d500 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.413+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa170005490 con 0x7fa18006d500 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.413+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170007260 con 0x7fa18006d500 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.413+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa180131ab0 con 0x7fa18006d500 2024-03-06T07:34:54.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.413+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18015f6e0 con 0x7fa18006d500 2024-03-06T07:34:54.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.414+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18004f330 con 0x7fa18006d500 2024-03-06T07:34:54.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.415+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa170007400 con 0x7fa18006d500 2024-03-06T07:34:54.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.416+0000 7fa176a00640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15c06b4d0 0x7fa15c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.416+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa1700896f0 con 0x7fa18006d500 2024-03-06T07:34:54.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.417+0000 7fa175400640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15c06b4d0 0x7fa15c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.417+0000 7fa175400640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15c06b4d0 0x7fa15c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa1800b08f0 tx=0x7fa168008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.421+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa170058510 con 0x7fa18006d500 2024-03-06T07:34:54.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.620+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa180071120 con 0x7fa18006d500 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.621+0000 7fa176a00640 1 -- 172.21.15.32:0/1174254685 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa170057e10 con 0x7fa18006d500 2024-03-06T07:34:54.623 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:34:54.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15c06b4d0 msgr2=0x7fa15c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15c06b4d0 0x7fa15c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa1800b08f0 tx=0x7fa168008040 comp rx=0 tx=0).stop 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 msgr2=0x7fa18015f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 0x7fa18015f1a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c3c0 tx=0x7fa17000c880 comp rx=0 tx=0).stop 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 shutdown_connections 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa15c06b4d0 0x7fa15c06d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa180144d90 0x7fa1800c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa180072110 0x7fa1800c4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 --2- 172.21.15.32:0/1174254685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18006d500 0x7fa18015f1a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 >> 172.21.15.32:0/1174254685 conn(0x7fa18006b3e0 msgr2=0x7fa18006b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 shutdown_connections 2024-03-06T07:34:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.623+0000 7fa17f200640 1 -- 172.21.15.32:0/1174254685 wait complete. 2024-03-06T07:34:54.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.845+0000 7fda5e400640 1 Processor -- start 2024-03-06T07:34:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.845+0000 7fda5e400640 1 -- start start 2024-03-06T07:34:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.846+0000 7fda5e400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fda581617b0 0x7fda580b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.846+0000 7fda5e400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda580d19f0 0x7fda580cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5e400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda580d23e0 0x7fda580d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5e400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580d2d90 con 0x7fda580d23e0 2024-03-06T07:34:54.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5e400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580cea30 con 0x7fda581617b0 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5e400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d1f70 con 0x7fda580d19f0 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda580d23e0 0x7fda580d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda580d23e0 0x7fda580d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33610/0 (socket says 172.21.15.32:33610) 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5da00640 1 -- 172.21.15.32:0/1336618852 learned_addr learned my addr 172.21.15.32:0/1336618852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda5d000640 1 --2- 172.21.15.32:0/1336618852 >> v2:172.21.15.39:3300/0 conn(0x7fda581617b0 0x7fda580b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.847+0000 7fda57e00640 1 --2- 172.21.15.32:0/1336618852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda580d19f0 0x7fda580cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.848+0000 7fda5da00640 1 -- 172.21.15.32:0/1336618852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda580d19f0 msgr2=0x7fda580cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.848+0000 7fda5da00640 1 --2- 172.21.15.32:0/1336618852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda580d19f0 0x7fda580cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.848+0000 7fda5da00640 1 -- 172.21.15.32:0/1336618852 >> v2:172.21.15.39:3300/0 conn(0x7fda581617b0 msgr2=0x7fda580b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.848+0000 7fda5da00640 1 --2- 172.21.15.32:0/1336618852 >> v2:172.21.15.39:3300/0 conn(0x7fda581617b0 0x7fda580b62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.848+0000 7fda5da00640 1 -- 172.21.15.32:0/1336618852 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580cecd0 con 0x7fda580d23e0 2024-03-06T07:34:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.848+0000 7fda5d000640 1 --2- 172.21.15.32:0/1336618852 >> v2:172.21.15.39:3300/0 conn(0x7fda581617b0 0x7fda580b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.849+0000 7fda5da00640 1 --2- 172.21.15.32:0/1336618852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d23e0 0x7fda580d2850 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fda5808ccf0 tx=0x7fda5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c5558cc3d24d3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.849+0000 7fda57400640 1 -- 172.21.15.32:0/1336618852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50035020 con 0x7fda580d23e0 2024-03-06T07:34:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.849+0000 7fda57400640 1 -- 172.21.15.32:0/1336618852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda50031870 con 0x7fda580d23e0 2024-03-06T07:34:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.849+0000 7fda57400640 1 -- 172.21.15.32:0/1336618852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50031c40 con 0x7fda580d23e0 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 -- 172.21.15.32:0/1336618852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d23e0 msgr2=0x7fda580d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 --2- 172.21.15.32:0/1336618852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d23e0 0x7fda580d2850 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fda5808ccf0 tx=0x7fda5002b910 comp rx=0 tx=0).stop 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 -- 172.21.15.32:0/1336618852 shutdown_connections 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 --2- 172.21.15.32:0/1336618852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d23e0 0x7fda580d2850 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 --2- 172.21.15.32:0/1336618852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda580d19f0 0x7fda580cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 --2- 172.21.15.32:0/1336618852 >> v2:172.21.15.39:3300/0 conn(0x7fda581617b0 0x7fda580b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.850+0000 7fda5e400640 1 -- 172.21.15.32:0/1336618852 >> 172.21.15.32:0/1336618852 conn(0x7fda580d3eb0 msgr2=0x7fda580d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.851+0000 7fda5e400640 1 -- 172.21.15.32:0/1336618852 shutdown_connections 2024-03-06T07:34:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.851+0000 7fda5e400640 1 -- 172.21.15.32:0/1336618852 wait complete. 2024-03-06T07:34:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.852+0000 7fda5e400640 1 Processor -- start 2024-03-06T07:34:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.853+0000 7fda5e400640 1 -- start start 2024-03-06T07:34:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.853+0000 7fda5e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 0x7fda581511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda580d23e0 0x7fda580b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5e400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda581617b0 0x7fda580b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580b6810 con 0x7fda580d19f0 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580cf290 con 0x7fda580d23e0 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5e400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580cfcf0 con 0x7fda581617b0 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 0x7fda581511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 0x7fda581511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33620/0 (socket says 172.21.15.32:33620) 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 -- 172.21.15.32:0/1586863548 learned_addr learned my addr 172.21.15.32:0/1586863548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5da00640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda581617b0 0x7fda580b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 -- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda581617b0 msgr2=0x7fda580b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda581617b0 0x7fda580b8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 -- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda580d23e0 msgr2=0x7fda580b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda580d23e0 0x7fda580b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.854+0000 7fda5d000640 1 -- 172.21.15.32:0/1586863548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5809dc50 con 0x7fda580d19f0 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.855+0000 7fda5d000640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 0x7fda581511f0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fda48007ef0 tx=0x7fda4800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.855+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4801a070 con 0x7fda580d19f0 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.855+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48003b90 con 0x7fda580d19f0 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.855+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5809df30 con 0x7fda580d19f0 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.855+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48004030 con 0x7fda580d19f0 2024-03-06T07:34:54.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.855+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5815e550 con 0x7fda580d19f0 2024-03-06T07:34:54.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.857+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5804f330 con 0x7fda580d19f0 2024-03-06T07:34:54.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.858+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda4801d070 con 0x7fda580d19f0 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.859+0000 7fda54c00640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.859+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fda4808a4e0 con 0x7fda580d19f0 2024-03-06T07:34:54.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.859+0000 7fda57e00640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.860+0000 7fda57e00640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fda580d1480 tx=0x7fda44008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:54.863+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda48010070 con 0x7fda580d19f0 2024-03-06T07:34:55.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.164+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda5808d9d0 con 0x7fda580d19f0 2024-03-06T07:34:55.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.165+0000 7fda54c00640 1 -- 172.21.15.32:0/1586863548 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7fda48058c10 con 0x7fda580d19f0 2024-03-06T07:34:55.167 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 msgr2=0x7fda3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fda580d1480 tx=0x7fda44008040 comp rx=0 tx=0).stop 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 msgr2=0x7fda581511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 0x7fda581511f0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fda48007ef0 tx=0x7fda4800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 shutdown_connections 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda581617b0 0x7fda580b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda580d23e0 0x7fda580b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 --2- 172.21.15.32:0/1586863548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda580d19f0 0x7fda581511f0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 >> 172.21.15.32:0/1586863548 conn(0x7fda580d3eb0 msgr2=0x7fda580cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 shutdown_connections 2024-03-06T07:34:55.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.167+0000 7fda5e400640 1 -- 172.21.15.32:0/1586863548 wait complete. 2024-03-06T07:34:55.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.385+0000 7f108d600640 1 Processor -- start 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.385+0000 7f108d600640 1 -- start start 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f108d600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f108806e7a0 0x7f1088070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f108d600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 0x7f108806ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f108d600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10880b37d0 con 0x7f108815e0a0 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f108d600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108806e310 con 0x7f108806e7a0 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f108d600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880b4b00 con 0x7f108806d980 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f1087e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f108815e0a0 0x7f10880b4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55768/0 (socket says 172.21.15.32:55768) 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f1087e00640 1 -- 172.21.15.32:0/418852655 learned_addr learned my addr 172.21.15.32:0/418852655 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f1087400640 1 --2- 172.21.15.32:0/418852655 >> v2:172.21.15.39:3300/0 conn(0x7f108806e7a0 0x7f1088070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846680927 0 0) 0x7f10880b37d0 con 0x7f108815e0a0 2024-03-06T07:34:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c003520 con 0x7f108815e0a0 2024-03-06T07:34:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.387+0000 7f108cc00640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 0x7f108806ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3900098624 0 0) 0x7f106c003520 con 0x7f108815e0a0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 msgr2=0x7f108806ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 0x7f108806ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 >> v2:172.21.15.39:3300/0 conn(0x7f108806e7a0 msgr2=0x7f1088070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 --2- 172.21.15.32:0/418852655 >> v2:172.21.15.39:3300/0 conn(0x7f108806e7a0 0x7f1088070e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880b3ab0 con 0x7f108815e0a0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f1078001670 con 0x7f108815e0a0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 >> v1:172.21.15.32:6789/0 conn(0x7f108815e0a0 legacy=0x7f10880b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f106c0039b0 0x7f106c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f106c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 0x7f106c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108806e310 con 0x7f106c004800 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10880b4b00 con 0x7f106c0039b0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c00ba10 con 0x7f108806d980 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.388+0000 7f1087400640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f106c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.389+0000 7f1087e00640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f106c0039b0 0x7f106c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.389+0000 7f1087400640 1 -- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 msgr2=0x7f106c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.389+0000 7f1087400640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108806d980 0x7f106c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.389+0000 7f1087400640 1 -- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f106c0039b0 msgr2=0x7f106c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.389+0000 7f1087400640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f106c0039b0 0x7f106c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.389+0000 7f1087400640 1 -- 172.21.15.32:0/418852655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c003520 con 0x7f106c004800 2024-03-06T07:34:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f1087400640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f106c004c80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1074009ac0 tx=0x7f106c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9c97a230de69b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074039070 con 0x7f106c004800 2024-03-06T07:34:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107402c6b0 con 0x7f106c004800 2024-03-06T07:34:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f1086a00640 1 -- 172.21.15.32:0/418852655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074009690 con 0x7f106c004800 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f108d600640 1 -- 172.21.15.32:0/418852655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 msgr2=0x7f106c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f108d600640 1 --2- 172.21.15.32:0/418852655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f106c004c80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1074009ac0 tx=0x7f106c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.390+0000 7f108cc00640 1 -- 172.21.15.32:0/418852655 reap_dead start 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.391+0000 7f108d600640 1 -- 172.21.15.32:0/418852655 shutdown_connections 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.391+0000 7f108d600640 1 -- 172.21.15.32:0/418852655 >> 172.21.15.32:0/418852655 conn(0x7f10880b1990 msgr2=0x7f10880b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.391+0000 7f108d600640 1 -- 172.21.15.32:0/418852655 shutdown_connections 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.391+0000 7f108d600640 1 -- 172.21.15.32:0/418852655 wait complete. 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.392+0000 7f108d600640 1 Processor -- start 2024-03-06T07:34:55.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.393+0000 7f108d600640 1 -- start start 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.393+0000 7f108d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f108814d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.393+0000 7f108d600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108815ef00 0x7f1088160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.393+0000 7f108d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f108815c5b0 0x7f108815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f108d600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10880ad6c0 con 0x7f106c004800 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f108d600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108815e480 con 0x7f108815c5b0 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f108d600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880ad840 con 0x7f108815ef00 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f108814d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f108814d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33642/0 (socket says 172.21.15.32:33642) 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 -- 172.21.15.32:0/3292247476 learned_addr learned my addr 172.21.15.32:0/3292247476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 -- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108815ef00 msgr2=0x7f1088160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f108cc00640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f108815c5b0 0x7f108815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087400640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108815ef00 0x7f1088160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108815ef00 0x7f1088160ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 -- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f108815c5b0 msgr2=0x7f108815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f108815c5b0 0x7f108815ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.394+0000 7f1087e00640 1 -- 172.21.15.32:0/3292247476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108815f380 con 0x7f106c004800 2024-03-06T07:34:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.395+0000 7f1087e00640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f108814d350 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f107800e780 tx=0x7f107800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.395+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107800c870 con 0x7f106c004800 2024-03-06T07:34:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.395+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1078018070 con 0x7f106c004800 2024-03-06T07:34:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.395+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107800cc50 con 0x7f106c004800 2024-03-06T07:34:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.395+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108815f660 con 0x7f106c004800 2024-03-06T07:34:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.395+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108815fc20 con 0x7f106c004800 2024-03-06T07:34:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.396+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108804f330 con 0x7f106c004800 2024-03-06T07:34:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.397+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f107801d460 con 0x7f106c004800 2024-03-06T07:34:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.398+0000 7f107fe00640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f106806b4d0 0x7f106806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.398+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f107808a1f0 con 0x7f106c004800 2024-03-06T07:34:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.398+0000 7f1087400640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f106806b4d0 0x7f106806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.399+0000 7f1087400640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f106806b4d0 0x7f106806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1074036000 tx=0x7f1074030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:55.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.402+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1078059000 con 0x7f106c004800 2024-03-06T07:34:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.613+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f108806ebb0 con 0x7f106c004800 2024-03-06T07:34:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.614+0000 7f107fe00640 1 -- 172.21.15.32:0/3292247476 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v99) v1 ==== 80+0+120 (secure 0 0 0) 0x7f1078058900 con 0x7f106c004800 2024-03-06T07:34:55.616 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.39:6835/556051904 2024-03-07T07:34:25.418325+0000 2024-03-06T07:34:55.616 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.39:6833/556051904 2024-03-07T07:34:25.418325+0000 2024-03-06T07:34:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f106806b4d0 msgr2=0x7f106806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f106806b4d0 0x7f106806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1074036000 tx=0x7f1074030000 comp rx=0 tx=0).stop 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 msgr2=0x7f108814d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f108814d350 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f107800e780 tx=0x7f107800ec40 comp rx=0 tx=0).stop 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 shutdown_connections 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f106806b4d0 0x7f106806d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f108815c5b0 0x7f108815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f108815ef00 0x7f1088160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 --2- 172.21.15.32:0/3292247476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c004800 0x7f108814d350 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 >> 172.21.15.32:0/3292247476 conn(0x7f10880b1990 msgr2=0x7f10880afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:55.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.616+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 shutdown_connections 2024-03-06T07:34:55.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:55.617+0000 7f108d600640 1 -- 172.21.15.32:0/3292247476 wait complete. 2024-03-06T07:34:55.636 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:34:55.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:55.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:55.769 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:34:55.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:55.769 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:55.810 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:34:55.811 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:55.853 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:55.853 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:34:55.853 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:55.890 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:55.890 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:34:55.890 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:55.944 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:55.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:56.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.146+0000 7fa43e000640 1 Processor -- start 2024-03-06T07:34:56.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.146+0000 7fa43e000640 1 -- start start 2024-03-06T07:34:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa43e000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4381457e0 0x7fa4380cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa43e000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa43815fba0 0x7fa4380b5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa43e000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa4380b56a0 0x7fa43815c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa43e000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4380b5ab0 con 0x7fa4381457e0 2024-03-06T07:34:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa43e000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa438145bc0 con 0x7fa4380b56a0 2024-03-06T07:34:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa43e000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4380cfb20 con 0x7fa43815fba0 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.148+0000 7fa437400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4381457e0 0x7fa4380cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4381457e0 0x7fa4380cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33658/0 (socket says 172.21.15.32:33658) 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 -- 172.21.15.32:0/548668443 learned_addr learned my addr 172.21.15.32:0/548668443 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437e00640 1 --2- 172.21.15.32:0/548668443 >> v2:172.21.15.39:3300/0 conn(0x7fa4380b56a0 0x7fa43815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa436a00640 1 --2- 172.21.15.32:0/548668443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa43815fba0 0x7fa4380b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 -- 172.21.15.32:0/548668443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa43815fba0 msgr2=0x7fa4380b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 --2- 172.21.15.32:0/548668443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa43815fba0 0x7fa4380b5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 -- 172.21.15.32:0/548668443 >> v2:172.21.15.39:3300/0 conn(0x7fa4380b56a0 msgr2=0x7fa43815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 --2- 172.21.15.32:0/548668443 >> v2:172.21.15.39:3300/0 conn(0x7fa4380b56a0 0x7fa43815c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.149+0000 7fa437400640 1 -- 172.21.15.32:0/548668443 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa438160040 con 0x7fa4381457e0 2024-03-06T07:34:56.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.150+0000 7fa437400640 1 --2- 172.21.15.32:0/548668443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4381457e0 0x7fa4380cf5e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa43806f0e0 tx=0x7fa42802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56270eb9e3d4cebc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa436000640 1 -- 172.21.15.32:0/548668443 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42802e070 con 0x7fa4381457e0 2024-03-06T07:34:56.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa436000640 1 -- 172.21.15.32:0/548668443 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa428037070 con 0x7fa4381457e0 2024-03-06T07:34:56.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa436000640 1 -- 172.21.15.32:0/548668443 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428031a00 con 0x7fa4381457e0 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 -- 172.21.15.32:0/548668443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4381457e0 msgr2=0x7fa4380cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 --2- 172.21.15.32:0/548668443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4381457e0 0x7fa4380cf5e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa43806f0e0 tx=0x7fa42802b880 comp rx=0 tx=0).stop 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 -- 172.21.15.32:0/548668443 shutdown_connections 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 --2- 172.21.15.32:0/548668443 >> v2:172.21.15.39:3300/0 conn(0x7fa4380b56a0 0x7fa43815c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 --2- 172.21.15.32:0/548668443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa43815fba0 0x7fa4380b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 --2- 172.21.15.32:0/548668443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4381457e0 0x7fa4380cf5e0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.151+0000 7fa43e000640 1 -- 172.21.15.32:0/548668443 >> 172.21.15.32:0/548668443 conn(0x7fa43806cf40 msgr2=0x7fa43806d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.152+0000 7fa43e000640 1 -- 172.21.15.32:0/548668443 shutdown_connections 2024-03-06T07:34:56.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.152+0000 7fa43e000640 1 -- 172.21.15.32:0/548668443 wait complete. 2024-03-06T07:34:56.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.153+0000 7fa43e000640 1 Processor -- start 2024-03-06T07:34:56.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.153+0000 7fa43e000640 1 -- start start 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.154+0000 7fa43e000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4380b56a0 0x7fa4380723b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.154+0000 7fa43e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4381457e0 0x7fa4380728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.154+0000 7fa43e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 0x7fa4380b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.154+0000 7fa43e000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4381608e0 con 0x7fa43815fba0 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.154+0000 7fa43e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438160600 con 0x7fa4381457e0 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.154+0000 7fa43e000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4380cca70 con 0x7fa4380b56a0 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 0x7fa4380b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 0x7fa4380b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33668/0 (socket says 172.21.15.32:33668) 2024-03-06T07:34:56.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 -- 172.21.15.32:0/2400092777 learned_addr learned my addr 172.21.15.32:0/2400092777 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437400640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4380b56a0 0x7fa4380723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa436a00640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4381457e0 0x7fa4380728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 -- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4380b56a0 msgr2=0x7fa4380723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4380b56a0 0x7fa4380723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 -- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4381457e0 msgr2=0x7fa4380728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4381457e0 0x7fa4380728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.155+0000 7fa437e00640 1 -- 172.21.15.32:0/2400092777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43815b8d0 con 0x7fa43815fba0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.156+0000 7fa437e00640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 0x7fa4380b40b0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa430007e90 tx=0x7fa43000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.156+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa430014420 con 0x7fa43815fba0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.156+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa430019070 con 0x7fa43815fba0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.156+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa430014820 con 0x7fa43815fba0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.156+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43815bbb0 con 0x7fa43815fba0 2024-03-06T07:34:56.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.156+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43813f8f0 con 0x7fa43815fba0 2024-03-06T07:34:56.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.158+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4380692a0 con 0x7fa43815fba0 2024-03-06T07:34:56.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.158+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4300149c0 con 0x7fa43815fba0 2024-03-06T07:34:56.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.159+0000 7fa42f400640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa41806b4d0 0x7fa41806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.159+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa43008ab60 con 0x7fa43815fba0 2024-03-06T07:34:56.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.159+0000 7fa437400640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa41806b4d0 0x7fa41806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.160+0000 7fa437400640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa41806b4d0 0x7fa41806d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa428036000 tx=0x7fa42803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.169+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa430059970 con 0x7fa43815fba0 2024-03-06T07:34:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.461+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa43813fac0 con 0x7fa43815fba0 2024-03-06T07:34:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.462+0000 7fa42f400640 1 -- 172.21.15.32:0/2400092777 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fa430059270 con 0x7fa43815fba0 2024-03-06T07:34:56.465 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:56.465 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:56.465 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa41806b4d0 msgr2=0x7fa41806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa41806b4d0 0x7fa41806d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa428036000 tx=0x7fa42803d040 comp rx=0 tx=0).stop 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 msgr2=0x7fa4380b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 0x7fa4380b40b0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fa430007e90 tx=0x7fa43000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 shutdown_connections 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa41806b4d0 0x7fa41806d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43815fba0 0x7fa4380b40b0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4381457e0 0x7fa4380728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 --2- 172.21.15.32:0/2400092777 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa4380b56a0 0x7fa4380723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.465+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 >> 172.21.15.32:0/2400092777 conn(0x7fa43806cf40 msgr2=0x7fa43813ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.466+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 shutdown_connections 2024-03-06T07:34:56.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.466+0000 7fa43e000640 1 -- 172.21.15.32:0/2400092777 wait complete. 2024-03-06T07:34:56.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:56.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.687+0000 7f6aebe00640 1 Processor -- start 2024-03-06T07:34:56.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.687+0000 7f6aebe00640 1 -- start start 2024-03-06T07:34:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aebe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6aec15e1d0 0x7f6aec06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aebe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 0x7f6aec162090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aebe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec06d150 con 0x7f6aec1625d0 2024-03-06T07:34:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aebe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec0b4610 con 0x7f6aec15e1d0 2024-03-06T07:34:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aebe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec15e5b0 con 0x7f6aec161c50 2024-03-06T07:34:56.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aeb400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6aec1625d0 0x7f6aec0cfeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55778/0 (socket says 172.21.15.32:55778) 2024-03-06T07:34:56.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.689+0000 7f6aeb400640 1 -- 172.21.15.32:0/785674084 learned_addr learned my addr 172.21.15.32:0/785674084 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6ae2000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 0x7f6aec162090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aeaa00640 1 --2- 172.21.15.32:0/785674084 >> v2:172.21.15.39:3300/0 conn(0x7f6aec15e1d0 0x7f6aec06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094917671 0 0) 0x7f6aec06d150 con 0x7f6aec1625d0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad0003520 con 0x7f6aec1625d0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108621191 0 0) 0x7f6ad0003520 con 0x7f6aec1625d0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 msgr2=0x7f6aec162090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 0x7f6aec162090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 >> v2:172.21.15.39:3300/0 conn(0x7f6aec15e1d0 msgr2=0x7f6aec06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 --2- 172.21.15.32:0/785674084 >> v2:172.21.15.39:3300/0 conn(0x7f6aec15e1d0 0x7f6aec06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0b4bf0 con 0x7f6aec1625d0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6ae4001670 con 0x7f6aec1625d0 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.690+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 >> v1:172.21.15.32:6789/0 conn(0x7f6aec1625d0 legacy=0x7f6aec0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aea000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ad00039b0 0x7f6ad0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aea000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6ad0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aea000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 0x7f6ad00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0b4610 con 0x7f6ad0004800 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec15e5b0 con 0x7f6ad00039b0 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad000ba40 con 0x7f6aec161c50 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6ae2000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6ad0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6aeaa00640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ad00039b0 0x7f6ad0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6ae2000640 1 -- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 msgr2=0x7f6ad00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6ae2000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec161c50 0x7f6ad00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6ae2000640 1 -- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ad00039b0 msgr2=0x7f6ad0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6ae2000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ad00039b0 0x7f6ad0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.691+0000 7f6ae2000640 1 -- 172.21.15.32:0/785674084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0003520 con 0x7f6ad0004800 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.693+0000 7f6ae2000640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6ad0004c80 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009ac0 tx=0x7f6ad0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1aabcdc9a022042 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.693+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4039070 con 0x7f6ad0004800 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.693+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad402c710 con 0x7f6ad0004800 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.693+0000 7f6aea000640 1 -- 172.21.15.32:0/785674084 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4009690 con 0x7f6ad0004800 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aebe00640 1 -- 172.21.15.32:0/785674084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 msgr2=0x7f6ad0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aebe00640 1 --2- 172.21.15.32:0/785674084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6ad0004c80 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009ac0 tx=0x7f6ad0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aeb400640 1 -- 172.21.15.32:0/785674084 reap_dead start 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aebe00640 1 -- 172.21.15.32:0/785674084 shutdown_connections 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aebe00640 1 -- 172.21.15.32:0/785674084 >> 172.21.15.32:0/785674084 conn(0x7f6aec0b37c0 msgr2=0x7f6aec0b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aebe00640 1 -- 172.21.15.32:0/785674084 shutdown_connections 2024-03-06T07:34:56.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.694+0000 7f6aebe00640 1 -- 172.21.15.32:0/785674084 wait complete. 2024-03-06T07:34:56.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.696+0000 7f6aebe00640 1 Processor -- start 2024-03-06T07:34:56.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.696+0000 7f6aebe00640 1 -- start start 2024-03-06T07:34:56.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6aec0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6aec154e60 0x7f6aec1e92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aebe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec1e97e0 0x7f6aec06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aebe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec06d430 con 0x7f6ad0004800 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aebe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0b51b0 con 0x7f6aec154e60 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aebe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec162a20 con 0x7f6aec1e97e0 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aeaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6aec0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aeaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6aec0c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33690/0 (socket says 172.21.15.32:33690) 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aeaa00640 1 -- 172.21.15.32:0/1295106565 learned_addr learned my addr 172.21.15.32:0/1295106565 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6ae2000640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6aec154e60 0x7f6aec1e92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aeaa00640 1 -- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec1e97e0 msgr2=0x7f6aec06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.697+0000 7f6aeaa00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec1e97e0 0x7f6aec06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6aeaa00640 1 -- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6aec154e60 msgr2=0x7f6aec1e92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6aeaa00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6aec154e60 0x7f6aec1e92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6aeaa00640 1 -- 172.21.15.32:0/1295106565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0c5ec0 con 0x7f6ad0004800 2024-03-06T07:34:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6aeaa00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6aec0c5980 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00f3d0 tx=0x7f6adc00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc016620 con 0x7f6ad0004800 2024-03-06T07:34:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc01b070 con 0x7f6ad0004800 2024-03-06T07:34:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec15e7d0 con 0x7f6ad0004800 2024-03-06T07:34:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc016a20 con 0x7f6ad0004800 2024-03-06T07:34:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.698+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec15ed90 con 0x7f6ad0004800 2024-03-06T07:34:56.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.699+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec04f330 con 0x7f6ad0004800 2024-03-06T07:34:56.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.700+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6adc005d10 con 0x7f6ad0004800 2024-03-06T07:34:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.702+0000 7f6ae3400640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ac806b500 0x7f6ac806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.702+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6adc08aed0 con 0x7f6ad0004800 2024-03-06T07:34:56.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.702+0000 7f6ae2000640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ac806b500 0x7f6ac806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.703+0000 7f6ae2000640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ac806b500 0x7f6ac806d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4036000 tx=0x7f6ad4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.706+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6adc059c50 con 0x7f6ad0004800 2024-03-06T07:34:56.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.903+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aec0b3cf0 con 0x7f6ad0004800 2024-03-06T07:34:56.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.905+0000 7f6ae3400640 1 -- 172.21.15.32:0/1295106565 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f6adc059550 con 0x7f6ad0004800 2024-03-06T07:34:56.907 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:56.907 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:25.427369+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/556051904":"2024-03-07T07:34:25.418325+0000","172.21.15.39:6833/556051904":"2024-03-07T07:34:25.418325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:56.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ac806b500 msgr2=0x7f6ac806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ac806b500 0x7f6ac806d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4036000 tx=0x7f6ad4030000 comp rx=0 tx=0).stop 2024-03-06T07:34:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 msgr2=0x7f6aec0c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6aec0c5980 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00f3d0 tx=0x7f6adc00f890 comp rx=0 tx=0).stop 2024-03-06T07:34:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 shutdown_connections 2024-03-06T07:34:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6ac806b500 0x7f6ac806d9c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6aec1e97e0 0x7f6aec06fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6aec154e60 0x7f6aec1e92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 --2- 172.21.15.32:0/1295106565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad0004800 0x7f6aec0c5980 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.907+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 >> 172.21.15.32:0/1295106565 conn(0x7f6aec0b37c0 msgr2=0x7f6aec0cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.908+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 shutdown_connections 2024-03-06T07:34:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:56.908+0000 7f6aebe00640 1 -- 172.21.15.32:0/1295106565 wait complete. 2024-03-06T07:34:56.928 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:06.595209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:07.760130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:56.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.138+0000 7f1363200640 1 Processor -- start 2024-03-06T07:34:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.139+0000 7f1363200640 1 -- start start 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1363200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1364160830 0x7f13640d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1363200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d4a40 0x7f13640d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1363200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640d1030 con 0x7f13640d28a0 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1363200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13640d1310 con 0x7f1364160830 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1363200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640d15f0 con 0x7f13640d4a40 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1361e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1364160830 0x7f13640d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1361e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1364160830 0x7f13640d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33702/0 (socket says 172.21.15.32:33702) 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1361e00640 1 -- 172.21.15.32:0/2936130252 learned_addr learned my addr 172.21.15.32:0/2936130252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.140+0000 7f1362800640 1 --1- 172.21.15.32:0/2936130252 >> v1:172.21.15.32:6789/0 conn(0x7f13640d28a0 0x7f13640cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55784/0 (socket says 172.21.15.32:55784) 2024-03-06T07:34:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.141+0000 7f135be00640 1 --2- 172.21.15.32:0/2936130252 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d4a40 0x7f13640d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.141+0000 7f1361e00640 1 -- 172.21.15.32:0/2936130252 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d4a40 msgr2=0x7f13640d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.141+0000 7f1361e00640 1 --2- 172.21.15.32:0/2936130252 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d4a40 0x7f13640d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.141+0000 7f1361e00640 1 -- 172.21.15.32:0/2936130252 >> v1:172.21.15.32:6789/0 conn(0x7f13640d28a0 legacy=0x7f13640cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.141+0000 7f1361e00640 1 -- 172.21.15.32:0/2936130252 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640ceb30 con 0x7f1364160830 2024-03-06T07:34:57.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.141+0000 7f1361400640 1 -- 172.21.15.32:0/2936130252 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110849945 0 0) 0x7f13640d1030 con 0x7f13640d28a0 2024-03-06T07:34:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.142+0000 7f1361e00640 1 --2- 172.21.15.32:0/2936130252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1364160830 0x7f13640d1e20 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f1354009870 tx=0x7f135402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3620c8a66c4f2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1361400640 1 -- 172.21.15.32:0/2936130252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354035020 con 0x7f1364160830 2024-03-06T07:34:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1361400640 1 -- 172.21.15.32:0/2936130252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354031640 con 0x7f1364160830 2024-03-06T07:34:57.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1361400640 1 -- 172.21.15.32:0/2936130252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135402d040 con 0x7f1364160830 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1363200640 1 -- 172.21.15.32:0/2936130252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1364160830 msgr2=0x7f13640d1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1363200640 1 --2- 172.21.15.32:0/2936130252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1364160830 0x7f13640d1e20 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f1354009870 tx=0x7f135402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1363200640 1 -- 172.21.15.32:0/2936130252 shutdown_connections 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1363200640 1 --2- 172.21.15.32:0/2936130252 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d4a40 0x7f13640d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1363200640 1 --2- 172.21.15.32:0/2936130252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1364160830 0x7f13640d1e20 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.143+0000 7f1363200640 1 -- 172.21.15.32:0/2936130252 >> 172.21.15.32:0/2936130252 conn(0x7f13640d3f20 msgr2=0x7f13640d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.144+0000 7f1363200640 1 -- 172.21.15.32:0/2936130252 shutdown_connections 2024-03-06T07:34:57.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.144+0000 7f1363200640 1 -- 172.21.15.32:0/2936130252 wait complete. 2024-03-06T07:34:57.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.145+0000 7f1363200640 1 Processor -- start 2024-03-06T07:34:57.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.146+0000 7f1363200640 1 -- start start 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1363200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d28a0 0x7f136415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1363200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 0x7f13640b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1363200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1364160830 0x7f13640c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1363200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13640b3a00 con 0x7f13640d4a40 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1363200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13640b3880 con 0x7f1364160830 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1363200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1364161c20 con 0x7f13640d28a0 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 0x7f13640b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 0x7f13640b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33710/0 (socket says 172.21.15.32:33710) 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 -- 172.21.15.32:0/3345421883 learned_addr learned my addr 172.21.15.32:0/3345421883 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1361e00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d28a0 0x7f136415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 -- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d28a0 msgr2=0x7f136415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1362800640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1364160830 0x7f13640c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d28a0 0x7f136415a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 -- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1364160830 msgr2=0x7f13640c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1364160830 0x7f13640c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f135be00640 1 -- 172.21.15.32:0/3345421883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640c7e30 con 0x7f13640d4a40 2024-03-06T07:34:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.147+0000 7f1361e00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d28a0 0x7f136415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:57.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.148+0000 7f135be00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 0x7f13640b78e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f134c00d800 tx=0x7f134c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.148+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c00b6c0 con 0x7f13640d4a40 2024-03-06T07:34:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.148+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c019070 con 0x7f13640d4a40 2024-03-06T07:34:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.148+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13640c8110 con 0x7f13640d4a40 2024-03-06T07:34:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.148+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c00ba80 con 0x7f13640d4a40 2024-03-06T07:34:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.148+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13640c86d0 con 0x7f13640d4a40 2024-03-06T07:34:57.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.150+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13641f3c10 con 0x7f13640d4a40 2024-03-06T07:34:57.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.150+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f134c00bc20 con 0x7f13640d4a40 2024-03-06T07:34:57.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.151+0000 7f135a000640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f134006b4d0 0x7f134006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.151+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f134c08a7b0 con 0x7f13640d4a40 2024-03-06T07:34:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.152+0000 7f1361e00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f134006b4d0 0x7f134006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.153+0000 7f1361e00640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f134006b4d0 0x7f134006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1354036000 tx=0x7f135403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.156+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f134c059530 con 0x7f13640d4a40 2024-03-06T07:34:57.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.383+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13640c8910 con 0x7f13640d4a40 2024-03-06T07:34:57.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.384+0000 7f135a000640 1 -- 172.21.15.32:0/3345421883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13640c8910 con 0x7f13640d4a40 2024-03-06T07:34:57.386 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:34:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f134006b4d0 msgr2=0x7f134006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f134006b4d0 0x7f134006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1354036000 tx=0x7f135403c040 comp rx=0 tx=0).stop 2024-03-06T07:34:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 msgr2=0x7f13640b78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 0x7f13640b78e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f134c00d800 tx=0x7f134c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 shutdown_connections 2024-03-06T07:34:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f134006b4d0 0x7f134006d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1364160830 0x7f13640c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13640d4a40 0x7f13640b78e0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 --2- 172.21.15.32:0/3345421883 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f13640d28a0 0x7f136415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.386+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 >> 172.21.15.32:0/3345421883 conn(0x7f13640d3f20 msgr2=0x7f13640b5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.387+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 shutdown_connections 2024-03-06T07:34:57.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.387+0000 7f1363200640 1 -- 172.21.15.32:0/3345421883 wait complete. 2024-03-06T07:34:57.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:57.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.614+0000 7fcf35400640 1 Processor -- start 2024-03-06T07:34:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.614+0000 7fcf35400640 1 -- start start 2024-03-06T07:34:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf35400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcf300d2230 0x7fcf300d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf35400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf300c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf35400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300d1ac0 con 0x7fcf3015e330 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf35400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf300cf4b0 con 0x7fcf300d2230 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf35400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300c86d0 con 0x7fcf300c7d00 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf2de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcf300d2230 0x7fcf300d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf2de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcf300d2230 0x7fcf300d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33722/0 (socket says 172.21.15.32:33722) 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf2de00640 1 -- 172.21.15.32:0/2094613823 learned_addr learned my addr 172.21.15.32:0/2094613823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.616+0000 7fcf2f200640 1 --2- 172.21.15.32:0/2094613823 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf300c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.617+0000 7fcf2de00640 1 -- 172.21.15.32:0/2094613823 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 msgr2=0x7fcf300c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.617+0000 7fcf2de00640 1 --2- 172.21.15.32:0/2094613823 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf300c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.617+0000 7fcf2de00640 1 -- 172.21.15.32:0/2094613823 >> v1:172.21.15.32:6789/0 conn(0x7fcf3015e330 legacy=0x7fcf300cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.617+0000 7fcf2de00640 1 -- 172.21.15.32:0/2094613823 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300d17e0 con 0x7fcf300d2230 2024-03-06T07:34:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.617+0000 7fcf2d400640 1 -- 172.21.15.32:0/2094613823 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354797024 0 0) 0x7fcf300d1ac0 con 0x7fcf3015e330 2024-03-06T07:34:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.618+0000 7fcf2de00640 1 --2- 172.21.15.32:0/2094613823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300d26c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009870 tx=0x7fcf1c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f24340f2751ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf2d400640 1 -- 172.21.15.32:0/2094613823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c034070 con 0x7fcf300d2230 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf2d400640 1 -- 172.21.15.32:0/2094613823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c030840 con 0x7fcf300d2230 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf2d400640 1 -- 172.21.15.32:0/2094613823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c030c10 con 0x7fcf300d2230 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 -- 172.21.15.32:0/2094613823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 msgr2=0x7fcf300d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 --2- 172.21.15.32:0/2094613823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300d26c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009870 tx=0x7fcf1c02b980 comp rx=0 tx=0).stop 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 -- 172.21.15.32:0/2094613823 shutdown_connections 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 --2- 172.21.15.32:0/2094613823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300d26c0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 --2- 172.21.15.32:0/2094613823 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf300c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 -- 172.21.15.32:0/2094613823 >> 172.21.15.32:0/2094613823 conn(0x7fcf3009d400 msgr2=0x7fcf300d41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 -- 172.21.15.32:0/2094613823 shutdown_connections 2024-03-06T07:34:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.619+0000 7fcf35400640 1 -- 172.21.15.32:0/2094613823 wait complete. 2024-03-06T07:34:57.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.622+0000 7fcf35400640 1 Processor -- start 2024-03-06T07:34:57.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.622+0000 7fcf35400640 1 -- start start 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf35400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf30156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf35400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf35400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf3015e330 0x7fcf3015fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf35400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300d0050 con 0x7fcf300d2230 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf35400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300cf140 con 0x7fcf3015e330 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf35400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3015e710 con 0x7fcf300c7d00 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf2de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf2de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33732/0 (socket says 172.21.15.32:33732) 2024-03-06T07:34:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.624+0000 7fcf2de00640 1 -- 172.21.15.32:0/4152253786 learned_addr learned my addr 172.21.15.32:0/4152253786 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2e800640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf30156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2f200640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf3015e330 0x7fcf3015fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2de00640 1 -- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 msgr2=0x7fcf30156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2de00640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf30156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2de00640 1 -- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf3015e330 msgr2=0x7fcf3015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2de00640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf3015e330 0x7fcf3015fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2de00640 1 -- 172.21.15.32:0/4152253786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300cbd00 con 0x7fcf300d2230 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.625+0000 7fcf2de00640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300cb280 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c000c00 tx=0x7fcf1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.626+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c034040 con 0x7fcf300d2230 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.626+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c030640 con 0x7fcf300d2230 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.626+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf300cbfe0 con 0x7fcf300d2230 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.626+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c039d40 con 0x7fcf300d2230 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.626+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf300cc5a0 con 0x7fcf300d2230 2024-03-06T07:34:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.627+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf300692a0 con 0x7fcf300d2230 2024-03-06T07:34:57.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.628+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf1c04b060 con 0x7fcf300d2230 2024-03-06T07:34:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.629+0000 7fcf26a00640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcf1006b4d0 0x7fcf1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.629+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcf1c0abfa0 con 0x7fcf300d2230 2024-03-06T07:34:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.629+0000 7fcf2e800640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcf1006b4d0 0x7fcf1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.630+0000 7fcf2e800640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcf1006b4d0 0x7fcf1006d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcf20000f60 tx=0x7fcf20009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.634+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf1c04ce90 con 0x7fcf300d2230 2024-03-06T07:34:57.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.833+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf300c9d40 con 0x7fcf300d2230 2024-03-06T07:34:57.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.834+0000 7fcf26a00640 1 -- 172.21.15.32:0/4152253786 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf1c030810 con 0x7fcf300d2230 2024-03-06T07:34:57.837 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:34:57.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcf1006b4d0 msgr2=0x7fcf1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcf1006b4d0 0x7fcf1006d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcf20000f60 tx=0x7fcf20009040 comp rx=0 tx=0).stop 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 msgr2=0x7fcf300cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300cb280 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c000c00 tx=0x7fcf1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 shutdown_connections 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcf1006b4d0 0x7fcf1006d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf3015e330 0x7fcf3015fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf300d2230 0x7fcf300cb280 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 --2- 172.21.15.32:0/4152253786 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcf300c7d00 0x7fcf30156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 >> 172.21.15.32:0/4152253786 conn(0x7fcf3009d400 msgr2=0x7fcf300c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 shutdown_connections 2024-03-06T07:34:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:57.837+0000 7fcf35400640 1 -- 172.21.15.32:0/4152253786 wait complete. 2024-03-06T07:34:57.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.064+0000 7f00b1e00640 1 Processor -- start 2024-03-06T07:34:58.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.064+0000 7f00b1e00640 1 -- start start 2024-03-06T07:34:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00b1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00ac0705a0 0x7f00ac070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00b1e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac06f7e0 con 0x7f00ac15bdd0 2024-03-06T07:34:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00b1e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac06fac0 con 0x7f00ac0d1b90 2024-03-06T07:34:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00b1e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac0d09d0 con 0x7f00ac0705a0 2024-03-06T07:34:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00abe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00ac0705a0 0x7f00ac070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00b0a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f00ac15bdd0 0x7f00ac0d02c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38902/0 (socket says 172.21.15.32:38902) 2024-03-06T07:34:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.066+0000 7f00b0a00640 1 -- 172.21.15.32:0/2934872643 learned_addr learned my addr 172.21.15.32:0/2934872643 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.067+0000 7f00abe00640 1 -- 172.21.15.32:0/2934872643 >> v1:172.21.15.39:6789/0 conn(0x7f00ac0d1b90 legacy=0x7f00ac0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.067+0000 7f00abe00640 1 -- 172.21.15.32:0/2934872643 >> v1:172.21.15.32:6789/0 conn(0x7f00ac15bdd0 legacy=0x7f00ac0d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.067+0000 7f00abe00640 1 -- 172.21.15.32:0/2934872643 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac06fda0 con 0x7f00ac0705a0 2024-03-06T07:34:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.067+0000 7f00ab400640 1 -- 172.21.15.32:0/2934872643 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261023457 0 0) 0x7f00ac06f7e0 con 0x7f00ac15bdd0 2024-03-06T07:34:58.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.068+0000 7f00abe00640 1 --2- 172.21.15.32:0/2934872643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac0705a0 0x7f00ac070a30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f00980097a0 tx=0x7f009802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c981f06dbda4e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.068+0000 7f00ab400640 1 -- 172.21.15.32:0/2934872643 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00980315a0 con 0x7f00ac0705a0 2024-03-06T07:34:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.068+0000 7f00ab400640 1 -- 172.21.15.32:0/2934872643 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098037070 con 0x7f00ac0705a0 2024-03-06T07:34:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.068+0000 7f00ab400640 1 -- 172.21.15.32:0/2934872643 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098031960 con 0x7f00ac0705a0 2024-03-06T07:34:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 -- 172.21.15.32:0/2934872643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac0705a0 msgr2=0x7f00ac070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 --2- 172.21.15.32:0/2934872643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac0705a0 0x7f00ac070a30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f00980097a0 tx=0x7f009802b840 comp rx=0 tx=0).stop 2024-03-06T07:34:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 -- 172.21.15.32:0/2934872643 shutdown_connections 2024-03-06T07:34:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 --2- 172.21.15.32:0/2934872643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac0705a0 0x7f00ac070a30 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 -- 172.21.15.32:0/2934872643 >> 172.21.15.32:0/2934872643 conn(0x7f00ac09aa40 msgr2=0x7f00ac09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 -- 172.21.15.32:0/2934872643 shutdown_connections 2024-03-06T07:34:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.069+0000 7f00b1e00640 1 -- 172.21.15.32:0/2934872643 wait complete. 2024-03-06T07:34:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.071+0000 7f00b1e00640 1 Processor -- start 2024-03-06T07:34:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.071+0000 7f00b1e00640 1 -- start start 2024-03-06T07:34:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00ac0705a0 0x7f00ac1e7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac0d1b90 0x7f00ac15ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 0x7f00ac15b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac15c1b0 con 0x7f00ac15bdd0 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0d0c10 con 0x7f00ac0d1b90 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac15d090 con 0x7f00ac0705a0 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 0x7f00ac15b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 0x7f00ac15b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51270/0 (socket says 172.21.15.32:51270) 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.072+0000 7f00b1400640 1 -- 172.21.15.32:0/3951346700 learned_addr learned my addr 172.21.15.32:0/3951346700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b0a00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00ac0705a0 0x7f00ac1e7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00abe00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac0d1b90 0x7f00ac15ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b1400640 1 -- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00ac0705a0 msgr2=0x7f00ac1e7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b1400640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00ac0705a0 0x7f00ac1e7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b1400640 1 -- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac0d1b90 msgr2=0x7f00ac15ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b1400640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac0d1b90 0x7f00ac15ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b1400640 1 -- 172.21.15.32:0/3951346700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0c79b0 con 0x7f00ac15bdd0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00abe00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac0d1b90 0x7f00ac15ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00b1400640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 0x7f00ac15b2b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f00a400c780 tx=0x7f00a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.073+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4004b10 con 0x7f00ac15bdd0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.074+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a4004cb0 con 0x7f00ac15bdd0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.074+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400aa90 con 0x7f00ac15bdd0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.074+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac0c7c50 con 0x7f00ac15bdd0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.074+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0c8210 con 0x7f00ac15bdd0 2024-03-06T07:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.075+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac04f330 con 0x7f00ac15bdd0 2024-03-06T07:34:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.076+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00a400ac30 con 0x7f00ac15bdd0 2024-03-06T07:34:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.077+0000 7f00a8c00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.077+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00a408a4f0 con 0x7f00ac15bdd0 2024-03-06T07:34:58.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.077+0000 7f00b0a00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.078+0000 7f00b0a00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008c06b4d0 0x7f008c06d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f00a0000f60 tx=0x7f00a0007c00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.082+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00a4020070 con 0x7f00ac15bdd0 2024-03-06T07:34:58.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.300+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f00ac06ef30 con 0x7f00ac15bdd0 2024-03-06T07:34:58.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.301+0000 7f00a8c00640 1 -- 172.21.15.32:0/3951346700 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f00a4059300 con 0x7f00ac15bdd0 2024-03-06T07:34:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:34:58.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008c06b4d0 msgr2=0x7f008c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008c06b4d0 0x7f008c06d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f00a0000f60 tx=0x7f00a0007c00 comp rx=0 tx=0).stop 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 msgr2=0x7f00ac15b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 0x7f00ac15b2b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f00a400c780 tx=0x7f00a400cc40 comp rx=0 tx=0).stop 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 shutdown_connections 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ac15bdd0 0x7f00ac15b2b0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac0d1b90 0x7f00ac15ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 --2- 172.21.15.32:0/3951346700 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00ac0705a0 0x7f00ac1e7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 >> 172.21.15.32:0/3951346700 conn(0x7f00ac09aa40 msgr2=0x7f00ac0b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 shutdown_connections 2024-03-06T07:34:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.304+0000 7f00b1e00640 1 -- 172.21.15.32:0/3951346700 wait complete. 2024-03-06T07:34:58.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:58.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.539+0000 7fa8f4c00640 1 Processor -- start 2024-03-06T07:34:58.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.539+0000 7fa8f4c00640 1 -- start start 2024-03-06T07:34:58.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.541+0000 7fa8f4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 0x7fa8f0166c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.541+0000 7fa8f4c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f0160830 con 0x7fa8f0156340 2024-03-06T07:34:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.541+0000 7fa8f4c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f015fa70 con 0x7fa8f006e090 2024-03-06T07:34:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.541+0000 7fa8f4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f015fd50 con 0x7fa8f00cf4e0 2024-03-06T07:34:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.541+0000 7fa8ef400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa8f0156340 0x7fa8f006d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38916/0 (socket says 172.21.15.32:38916) 2024-03-06T07:34:58.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.541+0000 7fa8ef400640 1 -- 172.21.15.32:0/4060751110 learned_addr learned my addr 172.21.15.32:0/4060751110 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:58.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8efe00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 0x7fa8f0166c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128836965 0 0) 0x7fa8f0160830 con 0x7fa8f0156340 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4003520 con 0x7fa8f0156340 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1707114724 0 0) 0x7fa8d4003520 con 0x7fa8f0156340 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 msgr2=0x7fa8f0166c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 0x7fa8f0166c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 >> v1:172.21.15.39:6789/0 conn(0x7fa8f006e090 legacy=0x7fa8f00cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f0160030 con 0x7fa8f0156340 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa8e0001670 con 0x7fa8f0156340 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.542+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 >> v1:172.21.15.32:6789/0 conn(0x7fa8f0156340 legacy=0x7fa8f006d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8eea00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8d4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8eea00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d4004800 0x7fa8d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8eea00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 0x7fa8d4005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f015fd50 con 0x7fa8d40039e0 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d400bb80 con 0x7fa8d4004800 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d400bd80 con 0x7fa8f00cf4e0 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8ef400640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8d4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8e7e00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d4004800 0x7fa8d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8ef400640 1 -- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 msgr2=0x7fa8d4005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8efe00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 0x7fa8d4005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8ef400640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f00cf4e0 0x7fa8d4005270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8ef400640 1 -- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d4004800 msgr2=0x7fa8d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8ef400640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d4004800 0x7fa8d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8ef400640 1 -- 172.21.15.32:0/4060751110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d4003520 con 0x7fa8d40039e0 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.543+0000 7fa8e7e00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d4004800 0x7fa8d4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.544+0000 7fa8ef400640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8d4003e00 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c110 tx=0x7fa8d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205d6adfa8b1e79a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0007c40 con 0x7fa8d40039e0 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0030070 con 0x7fa8d40039e0 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8eea00640 1 -- 172.21.15.32:0/4060751110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0005d30 con 0x7fa8d40039e0 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8f4c00640 1 -- 172.21.15.32:0/4060751110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 msgr2=0x7fa8d4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/4060751110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8d4003e00 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c110 tx=0x7fa8d4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8efe00640 1 -- 172.21.15.32:0/4060751110 reap_dead start 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8f4c00640 1 -- 172.21.15.32:0/4060751110 shutdown_connections 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.545+0000 7fa8f4c00640 1 -- 172.21.15.32:0/4060751110 >> 172.21.15.32:0/4060751110 conn(0x7fa8f00b1990 msgr2=0x7fa8f00b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.546+0000 7fa8f4c00640 1 -- 172.21.15.32:0/4060751110 shutdown_connections 2024-03-06T07:34:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.546+0000 7fa8f4c00640 1 -- 172.21.15.32:0/4060751110 wait complete. 2024-03-06T07:34:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.547+0000 7fa8f4c00640 1 Processor -- start 2024-03-06T07:34:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.547+0000 7fa8f4c00640 1 -- start start 2024-03-06T07:34:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.548+0000 7fa8f4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8f00cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.548+0000 7fa8f4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8f0072660 0x7fa8f01463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.548+0000 7fa8f4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f015acb0 0x7fa8f015b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.548+0000 7fa8f4c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f0156720 con 0x7fa8d40039e0 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.548+0000 7fa8f4c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f015e4b0 con 0x7fa8f0072660 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.548+0000 7fa8f4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f0164c50 con 0x7fa8f015acb0 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8f00cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8f00cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51288/0 (socket says 172.21.15.32:51288) 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 -- 172.21.15.32:0/3871843280 learned_addr learned my addr 172.21.15.32:0/3871843280 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 -- 172.21.15.32:0/3871843280 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f015acb0 msgr2=0x7fa8f015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f015acb0 0x7fa8f015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 -- 172.21.15.32:0/3871843280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8f0072660 msgr2=0x7fa8f01463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8f0072660 0x7fa8f01463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 -- 172.21.15.32:0/3871843280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f00cd520 con 0x7fa8d40039e0 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.549+0000 7fa8ef400640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8f00cce60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c230 tx=0x7fa8f00af710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.550+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0004bf0 con 0x7fa8d40039e0 2024-03-06T07:34:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.550+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0000ed0 con 0x7fa8d40039e0 2024-03-06T07:34:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.550+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0007070 con 0x7fa8d40039e0 2024-03-06T07:34:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.550+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f00cd800 con 0x7fa8d40039e0 2024-03-06T07:34:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.550+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f00cddc0 con 0x7fa8d40039e0 2024-03-06T07:34:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.551+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f004f330 con 0x7fa8d40039e0 2024-03-06T07:34:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.552+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8e0047020 con 0x7fa8d40039e0 2024-03-06T07:34:58.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.553+0000 7fa8e7400640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.553+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa8e00ab3d0 con 0x7fa8d40039e0 2024-03-06T07:34:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.553+0000 7fa8e7e00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.554+0000 7fa8e7e00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00cc3b0 tx=0x7fa8d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.562+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8e007a150 con 0x7fa8d40039e0 2024-03-06T07:34:58.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.842+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8f00b2790 con 0x7fa8d40039e0 2024-03-06T07:34:58.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.843+0000 7fa8e7400640 1 -- 172.21.15.32:0/3871843280 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fa8e0079a50 con 0x7fa8d40039e0 2024-03-06T07:34:58.845 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:58.845 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:58.846 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:58.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b500 msgr2=0x7fa8cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00cc3b0 tx=0x7fa8d8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 msgr2=0x7fa8f00cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8f00cce60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c230 tx=0x7fa8f00af710 comp rx=0 tx=0).stop 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 shutdown_connections 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa8cc06b500 0x7fa8cc06d9c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8f015acb0 0x7fa8f015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8f0072660 0x7fa8f01463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 --2- 172.21.15.32:0/3871843280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8d40039e0 0x7fa8f00cce60 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 >> 172.21.15.32:0/3871843280 conn(0x7fa8f00b1990 msgr2=0x7fa8f015cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 shutdown_connections 2024-03-06T07:34:58.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:58.846+0000 7fa8f4c00640 1 -- 172.21.15.32:0/3871843280 wait complete. 2024-03-06T07:34:58.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.072+0000 7f3d63e00640 1 Processor -- start 2024-03-06T07:34:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.072+0000 7f3d63e00640 1 -- start start 2024-03-06T07:34:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d63e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d640b4b30 0x7f3d641623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d63e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3d640d1be0 0x7f3d640d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d63e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d640b50d0 con 0x7f3d6415f630 2024-03-06T07:34:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d63e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d640cf450 con 0x7f3d640b4b30 2024-03-06T07:34:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d63e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d640cf6a0 con 0x7f3d640d1be0 2024-03-06T07:34:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d5be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d640b4b30 0x7f3d641623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d62a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d6415f630 0x7f3d64161ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38922/0 (socket says 172.21.15.32:38922) 2024-03-06T07:34:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.073+0000 7f3d62a00640 1 -- 172.21.15.32:0/3636352625 learned_addr learned my addr 172.21.15.32:0/3636352625 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:34:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.074+0000 7f3d5be00640 1 -- 172.21.15.32:0/3636352625 >> v2:172.21.15.39:3300/0 conn(0x7f3d640d1be0 msgr2=0x7f3d640d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.074+0000 7f3d63400640 1 --2- 172.21.15.32:0/3636352625 >> v2:172.21.15.39:3300/0 conn(0x7f3d640d1be0 0x7f3d640d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.074+0000 7f3d5be00640 1 --2- 172.21.15.32:0/3636352625 >> v2:172.21.15.39:3300/0 conn(0x7f3d640d1be0 0x7f3d640d2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.074+0000 7f3d5be00640 1 -- 172.21.15.32:0/3636352625 >> v1:172.21.15.32:6789/0 conn(0x7f3d6415f630 legacy=0x7f3d64161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.074+0000 7f3d5be00640 1 -- 172.21.15.32:0/3636352625 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640cf980 con 0x7f3d640b4b30 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.074+0000 7f3d62000640 1 -- 172.21.15.32:0/3636352625 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035743397 0 0) 0x7f3d640b50d0 con 0x7f3d6415f630 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.075+0000 7f3d5be00640 1 --2- 172.21.15.32:0/3636352625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d640b4b30 0x7f3d641623b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821abebd4b582aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.075+0000 7f3d62000640 1 -- 172.21.15.32:0/3636352625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c038070 con 0x7f3d640b4b30 2024-03-06T07:34:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.075+0000 7f3d62000640 1 -- 172.21.15.32:0/3636352625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c031710 con 0x7f3d640b4b30 2024-03-06T07:34:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.075+0000 7f3d62000640 1 -- 172.21.15.32:0/3636352625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c031ae0 con 0x7f3d640b4b30 2024-03-06T07:34:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 -- 172.21.15.32:0/3636352625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d640b4b30 msgr2=0x7f3d641623b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3636352625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d640b4b30 0x7f3d641623b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c02b7b0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 -- 172.21.15.32:0/3636352625 shutdown_connections 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3636352625 >> v2:172.21.15.39:3300/0 conn(0x7f3d640d1be0 0x7f3d640d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3636352625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d640b4b30 0x7f3d641623b0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 -- 172.21.15.32:0/3636352625 >> 172.21.15.32:0/3636352625 conn(0x7f3d640d3e30 msgr2=0x7f3d640d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 -- 172.21.15.32:0/3636352625 shutdown_connections 2024-03-06T07:34:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.076+0000 7f3d63e00640 1 -- 172.21.15.32:0/3636352625 wait complete. 2024-03-06T07:34:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.079+0000 7f3d63e00640 1 Processor -- start 2024-03-06T07:34:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.079+0000 7f3d63e00640 1 -- start start 2024-03-06T07:34:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d640b4b30 0x7f3d641505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d640d1be0 0x7f3d6415e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 0x7f3d6415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d641628f0 con 0x7f3d6415f630 2024-03-06T07:34:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640d4c40 con 0x7f3d640b4b30 2024-03-06T07:34:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d64162a70 con 0x7f3d640d1be0 2024-03-06T07:34:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 0x7f3d6415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 0x7f3d6415bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51294/0 (socket says 172.21.15.32:51294) 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.081+0000 7f3d63400640 1 -- 172.21.15.32:0/3780894014 learned_addr learned my addr 172.21.15.32:0/3780894014 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d62a00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d640b4b30 0x7f3d641505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d5be00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d640d1be0 0x7f3d6415e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d63400640 1 -- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d640d1be0 msgr2=0x7f3d6415e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d63400640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d640d1be0 0x7f3d6415e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d63400640 1 -- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d640b4b30 msgr2=0x7f3d641505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d63400640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d640b4b30 0x7f3d641505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d63400640 1 -- 172.21.15.32:0/3780894014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640c96e0 con 0x7f3d6415f630 2024-03-06T07:34:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.082+0000 7f3d63400640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 0x7f3d6415bc00 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00ccb0 tx=0x7f3d5c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.083+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c00ce60 con 0x7f3d6415f630 2024-03-06T07:34:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.083+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c003990 con 0x7f3d6415f630 2024-03-06T07:34:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.083+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c003d60 con 0x7f3d6415f630 2024-03-06T07:34:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.083+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d640cc490 con 0x7f3d6415f630 2024-03-06T07:34:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.083+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d640cca50 con 0x7f3d6415f630 2024-03-06T07:34:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.084+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6404f330 con 0x7f3d6415f630 2024-03-06T07:34:59.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.085+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d5c003f00 con 0x7f3d6415f630 2024-03-06T07:34:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.086+0000 7f3d5aa00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d4006b4d0 0x7f3d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.086+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3d5c08a750 con 0x7f3d6415f630 2024-03-06T07:34:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.086+0000 7f3d62a00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d4006b4d0 0x7f3d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.087+0000 7f3d62a00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d4006b4d0 0x7f3d4006d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3d54000f60 tx=0x7f3d54007c00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.091+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d5c014030 con 0x7f3d6415f630 2024-03-06T07:34:59.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.289+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d640d42b0 con 0x7f3d6415f630 2024-03-06T07:34:59.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.291+0000 7f3d5aa00640 1 -- 172.21.15.32:0/3780894014 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f3d5c0594d0 con 0x7f3d6415f630 2024-03-06T07:34:59.293 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:59.294 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:25.427369+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/556051904":"2024-03-07T07:34:25.418325+0000","172.21.15.39:6833/556051904":"2024-03-07T07:34:25.418325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.293+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d4006b4d0 msgr2=0x7f3d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.293+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d4006b4d0 0x7f3d4006d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3d54000f60 tx=0x7f3d54007c00 comp rx=0 tx=0).stop 2024-03-06T07:34:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.293+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 msgr2=0x7f3d6415bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.293+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 0x7f3d6415bc00 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00ccb0 tx=0x7f3d5c00a5c0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 shutdown_connections 2024-03-06T07:34:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3d4006b4d0 0x7f3d4006d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6415f630 0x7f3d6415bc00 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d640d1be0 0x7f3d6415e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 --2- 172.21.15.32:0/3780894014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d640b4b30 0x7f3d641505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 >> 172.21.15.32:0/3780894014 conn(0x7f3d640d3e30 msgr2=0x7f3d6415fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 shutdown_connections 2024-03-06T07:34:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.294+0000 7f3d63e00640 1 -- 172.21.15.32:0/3780894014 wait complete. 2024-03-06T07:34:59.312 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:59.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:59.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.506+0000 7f3a67200640 1 Processor -- start 2024-03-06T07:34:59.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.506+0000 7f3a67200640 1 -- start start 2024-03-06T07:34:59.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.508+0000 7f3a67200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3a680b3190 0x7f3a680d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.508+0000 7f3a67200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.508+0000 7f3a67200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6806d4d0 con 0x7f3a680ceb50 2024-03-06T07:34:59.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.508+0000 7f3a67200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6806c450 con 0x7f3a680b3190 2024-03-06T07:34:59.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.508+0000 7f3a67200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6806c710 con 0x7f3a680d0050 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a65e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3a680b3190 0x7f3a680d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a65e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3a680b3190 0x7f3a680d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:35474/0 (socket says 172.21.15.32:35474) 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a5fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a65e00640 1 -- 172.21.15.32:0/2584793058 learned_addr learned my addr 172.21.15.32:0/2584793058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2584793058 >> v2:172.21.15.39:3300/0 conn(0x7f3a680b3190 msgr2=0x7f3a680d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a5fe00640 1 --2- 172.21.15.32:0/2584793058 >> v2:172.21.15.39:3300/0 conn(0x7f3a680b3190 0x7f3a680d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.509+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2584793058 >> v1:172.21.15.39:6789/0 conn(0x7f3a680ceb50 legacy=0x7f3a6815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.510+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2584793058 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6806c9f0 con 0x7f3a680d0050 2024-03-06T07:34:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.511+0000 7f3a5fe00640 1 --2- 172.21.15.32:0/2584793058 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680ce610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3a500097a0 tx=0x7f3a5002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20e31f0bba8a80db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.511+0000 7f3a65400640 1 -- 172.21.15.32:0/2584793058 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50035020 con 0x7f3a680d0050 2024-03-06T07:34:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.511+0000 7f3a65400640 1 -- 172.21.15.32:0/2584793058 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50031830 con 0x7f3a680d0050 2024-03-06T07:34:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.511+0000 7f3a65400640 1 -- 172.21.15.32:0/2584793058 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50031c00 con 0x7f3a680d0050 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.512+0000 7f3a67200640 1 -- 172.21.15.32:0/2584793058 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 msgr2=0x7f3a680ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.512+0000 7f3a67200640 1 --2- 172.21.15.32:0/2584793058 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680ce610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3a500097a0 tx=0x7f3a5002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.512+0000 7f3a67200640 1 -- 172.21.15.32:0/2584793058 shutdown_connections 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.512+0000 7f3a67200640 1 --2- 172.21.15.32:0/2584793058 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680ce610 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.512+0000 7f3a67200640 1 --2- 172.21.15.32:0/2584793058 >> v2:172.21.15.39:3300/0 conn(0x7f3a680b3190 0x7f3a680d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.512+0000 7f3a67200640 1 -- 172.21.15.32:0/2584793058 >> 172.21.15.32:0/2584793058 conn(0x7f3a680b37c0 msgr2=0x7f3a680b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.513+0000 7f3a67200640 1 -- 172.21.15.32:0/2584793058 shutdown_connections 2024-03-06T07:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.513+0000 7f3a67200640 1 -- 172.21.15.32:0/2584793058 wait complete. 2024-03-06T07:34:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.515+0000 7f3a67200640 1 Processor -- start 2024-03-06T07:34:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.516+0000 7f3a67200640 1 -- start start 2024-03-06T07:34:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.517+0000 7f3a67200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a680b3190 0x7f3a681f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.517+0000 7f3a67200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 0x7f3a681f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.517+0000 7f3a67200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.517+0000 7f3a67200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a680b3570 con 0x7f3a680ceb50 2024-03-06T07:34:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.517+0000 7f3a67200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6806cfb0 con 0x7f3a680b3190 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.517+0000 7f3a67200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6806c0d0 con 0x7f3a680d0050 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 0x7f3a681f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 0x7f3a681f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51298/0 (socket says 172.21.15.32:51298) 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2808672516 learned_addr learned my addr 172.21.15.32:0/2808672516 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a65e00640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a680b3190 0x7f3a681f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a66800640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 msgr2=0x7f3a680c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a680b3190 msgr2=0x7f3a681f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a680b3190 0x7f3a681f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.518+0000 7f3a5fe00640 1 -- 172.21.15.32:0/2808672516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a681f5ae0 con 0x7f3a680ceb50 2024-03-06T07:34:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.519+0000 7f3a5fe00640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 0x7f3a681f55a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f3a50034040 tx=0x7f3a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.519+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50040070 con 0x7f3a680ceb50 2024-03-06T07:34:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.519+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5003a910 con 0x7f3a680ceb50 2024-03-06T07:34:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.519+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a680c5ae0 con 0x7f3a680ceb50 2024-03-06T07:34:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.519+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50035040 con 0x7f3a680ceb50 2024-03-06T07:34:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.519+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a680c60a0 con 0x7f3a680ceb50 2024-03-06T07:34:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.521+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6804f330 con 0x7f3a680ceb50 2024-03-06T07:34:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.521+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a50038020 con 0x7f3a680ceb50 2024-03-06T07:34:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.522+0000 7f3a5e000640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a4406b4d0 0x7f3a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.522+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a500abb10 con 0x7f3a680ceb50 2024-03-06T07:34:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.522+0000 7f3a65e00640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a4406b4d0 0x7f3a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.523+0000 7f3a65e00640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a4406b4d0 0x7f3a4406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009ac0 tx=0x7f3a58008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.527+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a681f5ae0 con 0x7f3a680ceb50 2024-03-06T07:34:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.771+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a68161950 con 0x7f3a680ceb50 2024-03-06T07:34:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.772+0000 7f3a5e000640 1 -- 172.21.15.32:0/2808672516 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f3a50031800 con 0x7f3a680ceb50 2024-03-06T07:34:59.775 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:34:59.775 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:34:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:34:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a4406b4d0 msgr2=0x7f3a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a4406b4d0 0x7f3a4406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009ac0 tx=0x7f3a58008040 comp rx=0 tx=0).stop 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 msgr2=0x7f3a681f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 0x7f3a681f55a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f3a50034040 tx=0x7f3a500046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 shutdown_connections 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3a4406b4d0 0x7f3a4406d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3a680d0050 0x7f3a680c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a680ceb50 0x7f3a681f55a0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 --2- 172.21.15.32:0/2808672516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a680b3190 0x7f3a681f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 >> 172.21.15.32:0/2808672516 conn(0x7f3a680b37c0 msgr2=0x7f3a680705b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 shutdown_connections 2024-03-06T07:34:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.775+0000 7f3a67200640 1 -- 172.21.15.32:0/2808672516 wait complete. 2024-03-06T07:34:59.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:00.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:34:59.999+0000 7f4cab400640 1 Processor -- start 2024-03-06T07:35:00.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.000+0000 7f4cab400640 1 -- start start 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4cab400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ca4147770 0x7f4ca4071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4cab400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4147b50 0x7f4ca4161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4cab400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca40717f0 con 0x7f4ca4162180 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4cab400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca4072400 con 0x7f4ca4147770 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4cab400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca4071ad0 con 0x7f4ca4147b50 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4ca8a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ca4147770 0x7f4ca4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4ca8a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ca4147770 0x7f4ca4071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51304/0 (socket says 172.21.15.32:51304) 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.001+0000 7f4ca8a00640 1 -- 172.21.15.32:0/936207138 learned_addr learned my addr 172.21.15.32:0/936207138 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.002+0000 7f4ca3e00640 1 --2- 172.21.15.32:0/936207138 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4147b50 0x7f4ca4161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.002+0000 7f4ca8a00640 1 -- 172.21.15.32:0/936207138 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4147b50 msgr2=0x7f4ca4161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.002+0000 7f4ca8a00640 1 --2- 172.21.15.32:0/936207138 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4147b50 0x7f4ca4161c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.002+0000 7f4ca8a00640 1 -- 172.21.15.32:0/936207138 >> v1:172.21.15.39:6789/0 conn(0x7f4ca4162180 legacy=0x7f4ca40b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.002+0000 7f4ca8a00640 1 -- 172.21.15.32:0/936207138 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4071db0 con 0x7f4ca4147770 2024-03-06T07:35:00.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.003+0000 7f4ca8a00640 1 --2- 172.21.15.32:0/936207138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca4071130 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e50 tx=0x7f4c9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fdf53423e5470 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.003+0000 7f4ca3400640 1 -- 172.21.15.32:0/936207138 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c02ca70 con 0x7f4ca4147770 2024-03-06T07:35:00.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.003+0000 7f4ca3400640 1 -- 172.21.15.32:0/936207138 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c02cc10 con 0x7f4ca4147770 2024-03-06T07:35:00.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.003+0000 7f4ca3400640 1 -- 172.21.15.32:0/936207138 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0343f0 con 0x7f4ca4147770 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 -- 172.21.15.32:0/936207138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 msgr2=0x7f4ca4071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 --2- 172.21.15.32:0/936207138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca4071130 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e50 tx=0x7f4c9c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 -- 172.21.15.32:0/936207138 shutdown_connections 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 --2- 172.21.15.32:0/936207138 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4147b50 0x7f4ca4161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 --2- 172.21.15.32:0/936207138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca4071130 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 -- 172.21.15.32:0/936207138 >> 172.21.15.32:0/936207138 conn(0x7f4ca406cf40 msgr2=0x7f4ca406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 -- 172.21.15.32:0/936207138 shutdown_connections 2024-03-06T07:35:00.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.004+0000 7f4cab400640 1 -- 172.21.15.32:0/936207138 wait complete. 2024-03-06T07:35:00.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.006+0000 7f4cab400640 1 Processor -- start 2024-03-06T07:35:00.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.006+0000 7f4cab400640 1 -- start start 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.007+0000 7f4cab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca415db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.007+0000 7f4cab400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ca4162180 0x7f4ca4143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.007+0000 7f4cab400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4144040 0x7f4ca40c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.007+0000 7f4cab400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40aea40 con 0x7f4ca4147770 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.007+0000 7f4cab400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40ae8c0 con 0x7f4ca4162180 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.007+0000 7f4cab400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40aebc0 con 0x7f4ca4144040 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca415db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51308/0 (socket says 172.21.15.32:51308) 2024-03-06T07:35:00.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 -- 172.21.15.32:0/798334187 learned_addr learned my addr 172.21.15.32:0/798334187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:00.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca3e00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ca4162180 0x7f4ca4143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca9400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4144040 0x7f4ca40c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 -- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4144040 msgr2=0x7f4ca40c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4144040 0x7f4ca40c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 -- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ca4162180 msgr2=0x7f4ca4143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ca4162180 0x7f4ca4143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.008+0000 7f4ca8a00640 1 -- 172.21.15.32:0/798334187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4144580 con 0x7f4ca4147770 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.009+0000 7f4ca8a00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca415db10 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e50 tx=0x7f4c9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.009+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c043020 con 0x7f4ca4147770 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.009+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c03e450 con 0x7f4ca4147770 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.009+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca4144860 con 0x7f4ca4147770 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.009+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c009510 con 0x7f4ca4147770 2024-03-06T07:35:00.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.009+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca4142400 con 0x7f4ca4147770 2024-03-06T07:35:00.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.010+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca40692a0 con 0x7f4ca4147770 2024-03-06T07:35:00.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.012+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c9c046020 con 0x7f4ca4147770 2024-03-06T07:35:00.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.013+0000 7f4ca0c00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c8406b500 0x7f4c8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.014+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4c9c0acb60 con 0x7f4ca4147770 2024-03-06T07:35:00.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.014+0000 7f4ca3e00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c8406b500 0x7f4c8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.014+0000 7f4ca3e00640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c8406b500 0x7f4c8406d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4c90006fd0 tx=0x7f4c90008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.018+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c9c04d030 con 0x7f4ca4147770 2024-03-06T07:35:00.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.255+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ca40cf330 con 0x7f4ca4147770 2024-03-06T07:35:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.257+0000 7f4ca0c00640 1 -- 172.21.15.32:0/798334187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f4c9c07b970 con 0x7f4ca4147770 2024-03-06T07:35:00.259 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:00.259 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:08.726069+0000","modified":"2024-03-06T07:34:28.763619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7344":{"gid":7344,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1229875104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1229875104},{"type":"v1","addr":"172.21.15.32:6835","nonce":1229875104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7344,"qdb_cluster":[7344]},"id":7}]} 2024-03-06T07:35:00.260 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c8406b500 msgr2=0x7f4c8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c8406b500 0x7f4c8406d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4c90006fd0 tx=0x7f4c90008040 comp rx=0 tx=0).stop 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 msgr2=0x7f4ca415db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca415db10 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e50 tx=0x7f4c9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 shutdown_connections 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4c8406b500 0x7f4c8406d9c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ca4144040 0x7f4ca40c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ca4162180 0x7f4ca4143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 --2- 172.21.15.32:0/798334187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca4147770 0x7f4ca415db10 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.259+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 >> 172.21.15.32:0/798334187 conn(0x7f4ca406cf40 msgr2=0x7f4ca40b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.260+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 shutdown_connections 2024-03-06T07:35:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.260+0000 7f4cab400640 1 -- 172.21.15.32:0/798334187 wait complete. 2024-03-06T07:35:00.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.494+0000 7f9319c00640 1 Processor -- start 2024-03-06T07:35:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.494+0000 7f9319c00640 1 -- start start 2024-03-06T07:35:00.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f9319c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93140b3560 0x7f9314072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f9319c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f93140acf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f9319c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93140b3a60 con 0x7f931406cfb0 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f9319c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93140b3d40 con 0x7f93140b3560 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f9319c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93140b4020 con 0x7f931406e3a0 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f930b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93140b3560 0x7f9314072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f930b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93140b3560 0x7f9314072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51310/0 (socket says 172.21.15.32:51310) 2024-03-06T07:35:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.496+0000 7f930b400640 1 -- 172.21.15.32:0/1772375107 learned_addr learned my addr 172.21.15.32:0/1772375107 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.497+0000 7f9319200640 1 --2- 172.21.15.32:0/1772375107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f93140acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.497+0000 7f930b400640 1 -- 172.21.15.32:0/1772375107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 msgr2=0x7f93140acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.497+0000 7f930b400640 1 --2- 172.21.15.32:0/1772375107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f93140acf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.497+0000 7f930b400640 1 -- 172.21.15.32:0/1772375107 >> v1:172.21.15.39:6789/0 conn(0x7f931406cfb0 legacy=0x7f93140b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.497+0000 7f930b400640 1 -- 172.21.15.32:0/1772375107 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93140b4300 con 0x7f93140b3560 2024-03-06T07:35:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.499+0000 7f930b400640 1 --2- 172.21.15.32:0/1772375107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93140b3560 0x7f9314072dc0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009870 tx=0x7f92fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941d43f84a4032ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.499+0000 7f9313400640 1 -- 172.21.15.32:0/1772375107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc034070 con 0x7f93140b3560 2024-03-06T07:35:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.499+0000 7f9313400640 1 -- 172.21.15.32:0/1772375107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc030840 con 0x7f93140b3560 2024-03-06T07:35:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.499+0000 7f9313400640 1 -- 172.21.15.32:0/1772375107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc030c10 con 0x7f93140b3560 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.499+0000 7f9319c00640 1 -- 172.21.15.32:0/1772375107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93140b3560 msgr2=0x7f9314072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.499+0000 7f9319c00640 1 --2- 172.21.15.32:0/1772375107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93140b3560 0x7f9314072dc0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009870 tx=0x7f92fc02b980 comp rx=0 tx=0).stop 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.500+0000 7f9319c00640 1 -- 172.21.15.32:0/1772375107 shutdown_connections 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.500+0000 7f9319c00640 1 --2- 172.21.15.32:0/1772375107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f93140acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.500+0000 7f9319c00640 1 --2- 172.21.15.32:0/1772375107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93140b3560 0x7f9314072dc0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.500+0000 7f9319c00640 1 -- 172.21.15.32:0/1772375107 >> 172.21.15.32:0/1772375107 conn(0x7f93140b1c50 msgr2=0x7f93140b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.500+0000 7f9319c00640 1 -- 172.21.15.32:0/1772375107 shutdown_connections 2024-03-06T07:35:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.500+0000 7f9319c00640 1 -- 172.21.15.32:0/1772375107 wait complete. 2024-03-06T07:35:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.501+0000 7f9319c00640 1 Processor -- start 2024-03-06T07:35:00.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.502+0000 7f9319c00640 1 -- start start 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.502+0000 7f9319c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 0x7f931415c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9319c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f9314160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9319c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93140b3560 0x7f9314160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9319c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931406e850 con 0x7f931406cfb0 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9319c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93140b5210 con 0x7f93140b3560 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9319c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93140cfb40 con 0x7f931406e3a0 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 0x7f931415c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 0x7f931415c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51318/0 (socket says 172.21.15.32:51318) 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 -- 172.21.15.32:0/2714378200 learned_addr learned my addr 172.21.15.32:0/2714378200 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f930b400640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f9314160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9319200640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93140b3560 0x7f9314160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 -- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 msgr2=0x7f9314160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f9314160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 -- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93140b3560 msgr2=0x7f9314160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93140b3560 0x7f9314160970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.503+0000 7f9313e00640 1 -- 172.21.15.32:0/2714378200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93141f82d0 con 0x7f931406cfb0 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.504+0000 7f9313e00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 0x7f931415c890 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f930400c870 tx=0x7f930400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.504+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304003b70 con 0x7f931406cfb0 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.504+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9304003d30 con 0x7f931406cfb0 2024-03-06T07:35:00.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.504+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930400aa90 con 0x7f931406cfb0 2024-03-06T07:35:00.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.504+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931415f110 con 0x7f931406cfb0 2024-03-06T07:35:00.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.504+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93141f7fa0 con 0x7f931406cfb0 2024-03-06T07:35:00.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.505+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931415cdd0 con 0x7f931406cfb0 2024-03-06T07:35:00.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.507+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f930400ac30 con 0x7f931406cfb0 2024-03-06T07:35:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.508+0000 7f9310c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f92f006b4d0 0x7f92f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.508+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f930408a240 con 0x7f931406cfb0 2024-03-06T07:35:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.508+0000 7f930b400640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f92f006b4d0 0x7f92f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.509+0000 7f930b400640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f92f006b4d0 0x7f92f006d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f92fc036000 tx=0x7f92fc03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.512+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9304059050 con 0x7f931406cfb0 2024-03-06T07:35:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.712+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f931404f330 con 0x7f931406cfb0 2024-03-06T07:35:00.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.713+0000 7f9310c00640 1 -- 172.21.15.32:0/2714378200 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f9304058950 con 0x7f931406cfb0 2024-03-06T07:35:00.716 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:00.716 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:34:25.427369+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:06.595209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:07.760130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/556051904":"2024-03-07T07:34:25.418325+0000","172.21.15.39:6833/556051904":"2024-03-07T07:34:25.418325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:00.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f92f006b4d0 msgr2=0x7f92f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f92f006b4d0 0x7f92f006d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f92fc036000 tx=0x7f92fc03b040 comp rx=0 tx=0).stop 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 msgr2=0x7f931415c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 0x7f931415c890 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f930400c870 tx=0x7f930400cd30 comp rx=0 tx=0).stop 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 shutdown_connections 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f92f006b4d0 0x7f92f006d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93140b3560 0x7f9314160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f931406e3a0 0x7f9314160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 --2- 172.21.15.32:0/2714378200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f931406cfb0 0x7f931415c890 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 >> 172.21.15.32:0/2714378200 conn(0x7f93140b1c50 msgr2=0x7f93140d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 shutdown_connections 2024-03-06T07:35:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.716+0000 7f9319c00640 1 -- 172.21.15.32:0/2714378200 wait complete. 2024-03-06T07:35:00.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:35:00.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.954+0000 7fd2eb200640 1 Processor -- start 2024-03-06T07:35:00.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.954+0000 7fd2eb200640 1 -- start start 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.956+0000 7fd2eb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2ec15f630 0x7fd2ec161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.956+0000 7fd2eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec0d1be0 0x7fd2ec0d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.956+0000 7fd2eb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0b50d0 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.956+0000 7fd2eb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0cf450 con 0x7fd2ec15f630 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.956+0000 7fd2eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0cf6a0 con 0x7fd2ec0d1be0 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.957+0000 7fd2e9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2ec15f630 0x7fd2ec161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.957+0000 7fd2ea800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec0d1be0 0x7fd2ec0d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.957+0000 7fd2e9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2ec15f630 0x7fd2ec161ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:35516/0 (socket says 172.21.15.32:35516) 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.957+0000 7fd2e9e00640 1 -- 172.21.15.32:0/2563377940 learned_addr learned my addr 172.21.15.32:0/2563377940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.958+0000 7fd2ea800640 1 -- 172.21.15.32:0/2563377940 >> v2:172.21.15.39:3300/0 conn(0x7fd2ec15f630 msgr2=0x7fd2ec161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.958+0000 7fd2ea800640 1 --2- 172.21.15.32:0/2563377940 >> v2:172.21.15.39:3300/0 conn(0x7fd2ec15f630 0x7fd2ec161ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.958+0000 7fd2ea800640 1 -- 172.21.15.32:0/2563377940 >> v1:172.21.15.39:6789/0 conn(0x7fd2ec0b4b30 legacy=0x7fd2ec1621e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.958+0000 7fd2ea800640 1 -- 172.21.15.32:0/2563377940 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0cf980 con 0x7fd2ec0d1be0 2024-03-06T07:35:00.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.960+0000 7fd2ea800640 1 --2- 172.21.15.32:0/2563377940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec0d1be0 0x7fd2ec0d2070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009dc0 tx=0x7fd2e402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc4773573cb0903c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.960+0000 7fd2e9400640 1 -- 172.21.15.32:0/2563377940 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4039070 con 0x7fd2ec0d1be0 2024-03-06T07:35:00.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.960+0000 7fd2e9400640 1 -- 172.21.15.32:0/2563377940 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e402ca90 con 0x7fd2ec0d1be0 2024-03-06T07:35:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.960+0000 7fd2e9400640 1 -- 172.21.15.32:0/2563377940 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e402ce60 con 0x7fd2ec0d1be0 2024-03-06T07:35:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 -- 172.21.15.32:0/2563377940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec0d1be0 msgr2=0x7fd2ec0d2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 --2- 172.21.15.32:0/2563377940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec0d1be0 0x7fd2ec0d2070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009dc0 tx=0x7fd2e402bce0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 -- 172.21.15.32:0/2563377940 shutdown_connections 2024-03-06T07:35:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 --2- 172.21.15.32:0/2563377940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec0d1be0 0x7fd2ec0d2070 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 --2- 172.21.15.32:0/2563377940 >> v2:172.21.15.39:3300/0 conn(0x7fd2ec15f630 0x7fd2ec161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 -- 172.21.15.32:0/2563377940 >> 172.21.15.32:0/2563377940 conn(0x7fd2ec0d3e30 msgr2=0x7fd2ec0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 -- 172.21.15.32:0/2563377940 shutdown_connections 2024-03-06T07:35:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.961+0000 7fd2eb200640 1 -- 172.21.15.32:0/2563377940 wait complete. 2024-03-06T07:35:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.964+0000 7fd2eb200640 1 Processor -- start 2024-03-06T07:35:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.964+0000 7fd2eb200640 1 -- start start 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 0x7fd2ec0cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2eb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2ec0d1be0 0x7fd2ec1e7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2eb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec15f630 0x7fd2ec1e7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2eb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec1628f0 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2eb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0d4c40 con 0x7fd2ec0d1be0 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2eb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec162a70 con 0x7fd2ec15f630 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 0x7fd2ec0cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 0x7fd2ec0cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51322/0 (socket says 172.21.15.32:51322) 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2e9e00640 1 -- 172.21.15.32:0/4235059587 learned_addr learned my addr 172.21.15.32:0/4235059587 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.966+0000 7fd2e3e00640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2ec0d1be0 0x7fd2ec1e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2ea800640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec15f630 0x7fd2ec1e7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e9e00640 1 -- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec15f630 msgr2=0x7fd2ec1e7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e9e00640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec15f630 0x7fd2ec1e7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e9e00640 1 -- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2ec0d1be0 msgr2=0x7fd2ec1e7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e9e00640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2ec0d1be0 0x7fd2ec1e7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e9e00640 1 -- 172.21.15.32:0/4235059587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0c9b60 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2ea800640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec15f630 0x7fd2ec1e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e9e00640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 0x7fd2ec0cb620 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00e870 tx=0x7fd2dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc017070 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc00c940 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec0c9e40 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.967+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00cd30 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.968+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec143ec0 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.969+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec04f330 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.970+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2dc01a070 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.971+0000 7fd2e2000640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.972+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd2dc08a200 con 0x7fd2ec0b4b30 2024-03-06T07:35:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.972+0000 7fd2e3e00640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.973+0000 7fd2e3e00640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd2c806b4d0 0x7fd2c806d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4006fd0 tx=0x7fd2d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:00.976+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2dc058f80 con 0x7fd2ec0b4b30 2024-03-06T07:35:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.217+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd2ec08d950 con 0x7fd2ec0b4b30 2024-03-06T07:35:01.220 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:35:01.218+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:35:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.221+0000 7fd2e2000640 1 -- 172.21.15.32:0/4235059587 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd2dc058880 con 0x7fd2ec0b4b30 2024-03-06T07:35:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd2c806b4d0 msgr2=0x7fd2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd2c806b4d0 0x7fd2c806d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4006fd0 tx=0x7fd2d4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 msgr2=0x7fd2ec0cb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 0x7fd2ec0cb620 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00e870 tx=0x7fd2dc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 shutdown_connections 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd2ec15f630 0x7fd2ec1e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2ec0d1be0 0x7fd2ec1e7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 --2- 172.21.15.32:0/4235059587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0b4b30 0x7fd2ec0cb620 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 >> 172.21.15.32:0/4235059587 conn(0x7fd2ec0d3e30 msgr2=0x7fd2ec0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.226+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 shutdown_connections 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.227+0000 7fd2eb200640 1 -- 172.21.15.32:0/4235059587 wait complete. 2024-03-06T07:35:01.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:35:01.313 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:35:01.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.431+0000 7f3d15e00640 1 Processor -- start 2024-03-06T07:35:01.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.431+0000 7f3d15e00640 1 -- start start 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d15e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d1015b3f0 0x7f3d1015b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d15e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1013e300 con 0x7f3d10072700 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d15e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1015be80 con 0x7f3d1013df20 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d15e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d100ce780 con 0x7f3d1015b3f0 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0e000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d1015b3f0 0x7f3d1015b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d10072700 0x7f3d10072b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38942/0 (socket says 172.21.15.32:38942) 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0f400640 1 -- 172.21.15.32:0/4224171583 learned_addr learned my addr 172.21.15.32:0/4224171583 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0e000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d1015b3f0 0x7f3d1015b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51346/0 (socket says 172.21.15.32:51346) 2024-03-06T07:35:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0e000640 1 -- 172.21.15.32:0/4224171583 >> v1:172.21.15.39:6789/0 conn(0x7f3d1013df20 legacy=0x7f3d1013e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0e000640 1 -- 172.21.15.32:0/4224171583 >> v1:172.21.15.32:6789/0 conn(0x7f3d10072700 legacy=0x7f3d10072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.433+0000 7f3d0e000640 1 -- 172.21.15.32:0/4224171583 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1015c160 con 0x7f3d1015b3f0 2024-03-06T07:35:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.435+0000 7f3d0e000640 1 --2- 172.21.15.32:0/4224171583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1015b3f0 0x7f3d1015b830 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00ba40 tx=0x7f3cfc02e090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88476419ecde94d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.435+0000 7f3d0d600640 1 -- 172.21.15.32:0/4224171583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc03b070 con 0x7f3d1015b3f0 2024-03-06T07:35:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.435+0000 7f3d0d600640 1 -- 172.21.15.32:0/4224171583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc02ee40 con 0x7f3d1015b3f0 2024-03-06T07:35:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.435+0000 7f3d0d600640 1 -- 172.21.15.32:0/4224171583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc036660 con 0x7f3d1015b3f0 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.435+0000 7f3d15e00640 1 -- 172.21.15.32:0/4224171583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1015b3f0 msgr2=0x7f3d1015b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.435+0000 7f3d15e00640 1 --2- 172.21.15.32:0/4224171583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1015b3f0 0x7f3d1015b830 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00ba40 tx=0x7f3cfc02e090 comp rx=0 tx=0).stop 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.436+0000 7f3d15e00640 1 -- 172.21.15.32:0/4224171583 shutdown_connections 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.436+0000 7f3d15e00640 1 --2- 172.21.15.32:0/4224171583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1015b3f0 0x7f3d1015b830 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.436+0000 7f3d15e00640 1 -- 172.21.15.32:0/4224171583 >> 172.21.15.32:0/4224171583 conn(0x7f3d1006cfb0 msgr2=0x7f3d1006d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.436+0000 7f3d15e00640 1 -- 172.21.15.32:0/4224171583 shutdown_connections 2024-03-06T07:35:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.436+0000 7f3d15e00640 1 -- 172.21.15.32:0/4224171583 wait complete. 2024-03-06T07:35:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.437+0000 7f3d15e00640 1 Processor -- start 2024-03-06T07:35:01.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.438+0000 7f3d15e00640 1 -- start start 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d15e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 0x7f3d10158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d15e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d1013df20 0x7f3d10131490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d15e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d1015b3f0 0x7f3d101398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d15e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d100cec90 con 0x7f3d10072700 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d15e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1015c440 con 0x7f3d1015b3f0 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d15e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1013ed50 con 0x7f3d1013df20 2024-03-06T07:35:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 0x7f3d10158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 0x7f3d10158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51362/0 (socket says 172.21.15.32:51362) 2024-03-06T07:35:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 -- 172.21.15.32:0/2055994952 learned_addr learned my addr 172.21.15.32:0/2055994952 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 -- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d1013df20 msgr2=0x7f3d10131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d1013df20 0x7f3d10131490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0f400640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d1015b3f0 0x7f3d101398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 -- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d1015b3f0 msgr2=0x7f3d101398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d1015b3f0 0x7f3d101398b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.439+0000 7f3d0ea00640 1 -- 172.21.15.32:0/2055994952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d100aec30 con 0x7f3d10072700 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.440+0000 7f3d0ea00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 0x7f3d10158550 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c460 tx=0x7f3d0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.440+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00016020 con 0x7f3d10072700 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.440+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d00012870 con 0x7f3d10072700 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.440+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00012c60 con 0x7f3d10072700 2024-03-06T07:35:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.440+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d100aef10 con 0x7f3d10072700 2024-03-06T07:35:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.440+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1013f150 con 0x7f3d10072700 2024-03-06T07:35:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.441+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d100692a0 con 0x7f3d10072700 2024-03-06T07:35:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.442+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d00019020 con 0x7f3d10072700 2024-03-06T07:35:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.443+0000 7f3d07400640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3cf006b4d0 0x7f3cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.443+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3d0008a340 con 0x7f3d10072700 2024-03-06T07:35:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.443+0000 7f3d0e000640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3cf006b4d0 0x7f3cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.444+0000 7f3d0e000640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3cf006b4d0 0x7f3cf006d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc038000 tx=0x7f3cfc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.449+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d00020020 con 0x7f3d10072700 2024-03-06T07:35:01.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:01.690+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3d1015c840 con 0x7f3d10072700 2024-03-06T07:35:02.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.225+0000 7f3d07400640 1 -- 172.21.15.32:0/2055994952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3d000182c0 con 0x7f3d10072700 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.228+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3cf006b4d0 msgr2=0x7f3cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.228+0000 7f3d15e00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3cf006b4d0 0x7f3cf006d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc038000 tx=0x7f3cfc030000 comp rx=0 tx=0).stop 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 msgr2=0x7f3d10158550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 0x7f3d10158550 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c460 tx=0x7f3d0000c920 comp rx=0 tx=0).stop 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 shutdown_connections 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3cf006b4d0 0x7f3cf006d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d1015b3f0 0x7f3d101398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3d1013df20 0x7f3d10131490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 --2- 172.21.15.32:0/2055994952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d10072700 0x7f3d10158550 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 >> 172.21.15.32:0/2055994952 conn(0x7f3d1006cfb0 msgr2=0x7f3d100715f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 shutdown_connections 2024-03-06T07:35:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.229+0000 7f3d15e00640 1 -- 172.21.15.32:0/2055994952 wait complete. 2024-03-06T07:35:02.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:35:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.470+0000 7f25a9c00640 1 Processor -- start 2024-03-06T07:35:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.470+0000 7f25a9c00640 1 -- start start 2024-03-06T07:35:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a9c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 0x7f25a406ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a9c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a40726a0 con 0x7f25a4154e20 2024-03-06T07:35:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a9c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a4072940 con 0x7f25a40b41e0 2024-03-06T07:35:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a9c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406e7f0 con 0x7f25a4155200 2024-03-06T07:35:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f25a4154e20 0x7f25a40b3ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38948/0 (socket says 172.21.15.32:38948) 2024-03-06T07:35:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a3e00640 1 -- 172.21.15.32:0/89561569 learned_addr learned my addr 172.21.15.32:0/89561569 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:02.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.472+0000 7f25a3400640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 0x7f25a406ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307183223 0 0) 0x7f25a40726a0 con 0x7f25a4154e20 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588003520 con 0x7f25a4154e20 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547863615 0 0) 0x7f2588003520 con 0x7f25a4154e20 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 msgr2=0x7f25a406ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 0x7f25a406ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 >> v1:172.21.15.39:6789/0 conn(0x7f25a40b41e0 legacy=0x7f25a406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a406dbe0 con 0x7f25a4154e20 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f25940019f0 con 0x7f25a4154e20 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 >> v1:172.21.15.32:6789/0 conn(0x7f25a4154e20 legacy=0x7f25a40b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25880039e0 0x7f2588003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2588004800 0x7f2588004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 0x7f25880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258800ba70 con 0x7f25880039e0 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258800bbf0 con 0x7f2588004800 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.473+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258800bdf0 con 0x7f25a4155200 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3e00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25880039e0 0x7f2588003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3400640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2588004800 0x7f2588004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3e00640 1 -- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 msgr2=0x7f25880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3e00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a4155200 0x7f25880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3e00640 1 -- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2588004800 msgr2=0x7f2588004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3e00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2588004800 0x7f2588004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.474+0000 7f25a3e00640 1 -- 172.21.15.32:0/89561569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2588003520 con 0x7f25880039e0 2024-03-06T07:35:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.475+0000 7f25a3e00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25880039e0 0x7f2588003e00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f259400a810 tx=0x7f2588005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60b703079bb0e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.475+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259400a460 con 0x7f25880039e0 2024-03-06T07:35:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.475+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594034650 con 0x7f25880039e0 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.475+0000 7f25a2a00640 1 -- 172.21.15.32:0/89561569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594034a40 con 0x7f25880039e0 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9c00640 1 -- 172.21.15.32:0/89561569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25880039e0 msgr2=0x7f2588003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9c00640 1 --2- 172.21.15.32:0/89561569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25880039e0 0x7f2588003e00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f259400a810 tx=0x7f2588005830 comp rx=0 tx=0).stop 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9200640 1 -- 172.21.15.32:0/89561569 reap_dead start 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9c00640 1 -- 172.21.15.32:0/89561569 shutdown_connections 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9c00640 1 -- 172.21.15.32:0/89561569 >> 172.21.15.32:0/89561569 conn(0x7f25a406b3e0 msgr2=0x7f25a406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9c00640 1 -- 172.21.15.32:0/89561569 shutdown_connections 2024-03-06T07:35:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.476+0000 7f25a9c00640 1 -- 172.21.15.32:0/89561569 wait complete. 2024-03-06T07:35:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.478+0000 7f25a9c00640 1 Processor -- start 2024-03-06T07:35:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.478+0000 7f25a9c00640 1 -- start start 2024-03-06T07:35:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25880039e0 0x7f25a40b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a40afa00 0x7f25a415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 0x7f25a415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406e480 con 0x7f25a4154080 2024-03-06T07:35:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406e1a0 con 0x7f25880039e0 2024-03-06T07:35:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406ced0 con 0x7f25a40afa00 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.479+0000 7f25a9200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 0x7f25a415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 0x7f25a415b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51392/0 (socket says 172.21.15.32:51392) 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 -- 172.21.15.32:0/955485010 learned_addr learned my addr 172.21.15.32:0/955485010 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a3e00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25880039e0 0x7f25a40b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a3400640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a40afa00 0x7f25a415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 -- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a40afa00 msgr2=0x7f25a415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a40afa00 0x7f25a415ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 -- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25880039e0 msgr2=0x7f25a40b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25880039e0 0x7f25a40b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 -- 172.21.15.32:0/955485010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a40b28e0 con 0x7f25a4154080 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.480+0000 7f25a9200640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 0x7f25a415b350 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f25a40af9d0 tx=0x7f259c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.481+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c007480 con 0x7f25a4154080 2024-03-06T07:35:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.481+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c007620 con 0x7f25a4154080 2024-03-06T07:35:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.481+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c004260 con 0x7f25a4154080 2024-03-06T07:35:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.481+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a40aff70 con 0x7f25a4154080 2024-03-06T07:35:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.481+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a40b0530 con 0x7f25a4154080 2024-03-06T07:35:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.482+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a404f330 con 0x7f25a4154080 2024-03-06T07:35:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.483+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f259c011070 con 0x7f25a4154080 2024-03-06T07:35:02.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.484+0000 7f259be00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f258006b4d0 0x7f258006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.484+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f259c0892f0 con 0x7f25a4154080 2024-03-06T07:35:02.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.484+0000 7f25a3e00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f258006b4d0 0x7f258006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.485+0000 7f25a3e00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f258006b4d0 0x7f258006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2594034000 tx=0x7f2594009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.488+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f259c057f90 con 0x7f25a4154080 2024-03-06T07:35:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:02.689+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f25a4072520 con 0x7f25a4154080 2024-03-06T07:35:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.236+0000 7f259be00640 1 -- 172.21.15.32:0/955485010 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7f259c057890 con 0x7f25a4154080 2024-03-06T07:35:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:35:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.240+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f258006b4d0 msgr2=0x7f258006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.240+0000 7f25a9c00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f258006b4d0 0x7f258006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2594034000 tx=0x7f2594009030 comp rx=0 tx=0).stop 2024-03-06T07:35:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.240+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 msgr2=0x7f25a415b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.240+0000 7f25a9c00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 0x7f25a415b350 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f25a40af9d0 tx=0x7f259c00c820 comp rx=0 tx=0).stop 2024-03-06T07:35:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 shutdown_connections 2024-03-06T07:35:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f258006b4d0 0x7f258006d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25a4154080 0x7f25a415b350 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25a40afa00 0x7f25a415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 --2- 172.21.15.32:0/955485010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25880039e0 0x7f25a40b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 >> 172.21.15.32:0/955485010 conn(0x7f25a406b3e0 msgr2=0x7f25a40b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 shutdown_connections 2024-03-06T07:35:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.241+0000 7f25a9c00640 1 -- 172.21.15.32:0/955485010 wait complete. 2024-03-06T07:35:03.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:35:03.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.486+0000 7fb85a200640 1 Processor -- start 2024-03-06T07:35:03.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.487+0000 7fb85a200640 1 -- start start 2024-03-06T07:35:03.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.488+0000 7fb85a200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8541446f0 0x7fb85415be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.488+0000 7fb85a200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb85406e5d0 0x7fb8540ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.488+0000 7fb85a200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85415c3a0 con 0x7fb8540cc7a0 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.488+0000 7fb85a200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85415f1c0 con 0x7fb85406e5d0 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.488+0000 7fb85a200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8540cccd0 con 0x7fb8541446f0 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.488+0000 7fb853e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb85406e5d0 0x7fb8540ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb852a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb8540cc7a0 0x7fb85406dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38950/0 (socket says 172.21.15.32:38950) 2024-03-06T07:35:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb852a00640 1 -- 172.21.15.32:0/2600559802 learned_addr learned my addr 172.21.15.32:0/2600559802 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb853400640 1 --2- 172.21.15.32:0/2600559802 >> v2:172.21.15.39:3300/0 conn(0x7fb8541446f0 0x7fb85415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb853e00640 1 -- 172.21.15.32:0/2600559802 >> v2:172.21.15.39:3300/0 conn(0x7fb8541446f0 msgr2=0x7fb85415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb853e00640 1 --2- 172.21.15.32:0/2600559802 >> v2:172.21.15.39:3300/0 conn(0x7fb8541446f0 0x7fb85415be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb853e00640 1 -- 172.21.15.32:0/2600559802 >> v1:172.21.15.32:6789/0 conn(0x7fb8540cc7a0 legacy=0x7fb85406dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb853e00640 1 -- 172.21.15.32:0/2600559802 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85415eb70 con 0x7fb85406e5d0 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.489+0000 7fb852000640 1 -- 172.21.15.32:0/2600559802 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831727861 0 0) 0x7fb840001110 con 0x7fb8540cc7a0 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.490+0000 7fb853e00640 1 --2- 172.21.15.32:0/2600559802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb85406e5d0 0x7fb8540ce460 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb85406ed80 tx=0x7fb84c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a531d3103f8dc98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.490+0000 7fb852000640 1 -- 172.21.15.32:0/2600559802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c039070 con 0x7fb85406e5d0 2024-03-06T07:35:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.490+0000 7fb852000640 1 -- 172.21.15.32:0/2600559802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c02cc80 con 0x7fb85406e5d0 2024-03-06T07:35:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.490+0000 7fb852000640 1 -- 172.21.15.32:0/2600559802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c034660 con 0x7fb85406e5d0 2024-03-06T07:35:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 -- 172.21.15.32:0/2600559802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb85406e5d0 msgr2=0x7fb8540ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 --2- 172.21.15.32:0/2600559802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb85406e5d0 0x7fb8540ce460 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb85406ed80 tx=0x7fb84c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 -- 172.21.15.32:0/2600559802 shutdown_connections 2024-03-06T07:35:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 --2- 172.21.15.32:0/2600559802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb85406e5d0 0x7fb8540ce460 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 --2- 172.21.15.32:0/2600559802 >> v2:172.21.15.39:3300/0 conn(0x7fb8541446f0 0x7fb85415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 -- 172.21.15.32:0/2600559802 >> 172.21.15.32:0/2600559802 conn(0x7fb8540b1ae0 msgr2=0x7fb8540b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 -- 172.21.15.32:0/2600559802 shutdown_connections 2024-03-06T07:35:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.491+0000 7fb85a200640 1 -- 172.21.15.32:0/2600559802 wait complete. 2024-03-06T07:35:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.493+0000 7fb85a200640 1 Processor -- start 2024-03-06T07:35:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.493+0000 7fb85a200640 1 -- start start 2024-03-06T07:35:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb85a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85406e5d0 0x7fb8540cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb85a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 0x7fb8540adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb85a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8541446f0 0x7fb8540ae480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb85a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85415f650 con 0x7fb8540cc7a0 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb85a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb854144ad0 con 0x7fb85406e5d0 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb85a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85415f7d0 con 0x7fb8541446f0 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.494+0000 7fb852a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 0x7fb8540adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 0x7fb8540adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51414/0 (socket says 172.21.15.32:51414) 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 -- 172.21.15.32:0/1168295284 learned_addr learned my addr 172.21.15.32:0/1168295284 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb853400640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85406e5d0 0x7fb8540cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb853e00640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8541446f0 0x7fb8540ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 -- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8541446f0 msgr2=0x7fb8540ae480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8541446f0 0x7fb8540ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 -- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85406e5d0 msgr2=0x7fb8540cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85406e5d0 0x7fb8540cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 -- 172.21.15.32:0/1168295284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8540aad00 con 0x7fb8540cc7a0 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb853400640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85406e5d0 0x7fb8540cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.495+0000 7fb852a00640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 0x7fb8540adf40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c780 tx=0x7fb84000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.496+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840018070 con 0x7fb8540cc7a0 2024-03-06T07:35:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.496+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8400072d0 con 0x7fb8540cc7a0 2024-03-06T07:35:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.496+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8400047e0 con 0x7fb8540cc7a0 2024-03-06T07:35:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.496+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb854130500 con 0x7fb8540cc7a0 2024-03-06T07:35:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.496+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb854130ac0 con 0x7fb8540cc7a0 2024-03-06T07:35:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.497+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8540692a0 con 0x7fb8540cc7a0 2024-03-06T07:35:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.498+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb84001b070 con 0x7fb8540cc7a0 2024-03-06T07:35:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.499+0000 7fb84b400640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb83406b4d0 0x7fb83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.499+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5443+0+0 (secure 0 0 0) 0x7fb840089500 con 0x7fb8540cc7a0 2024-03-06T07:35:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.499+0000 7fb853400640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb83406b4d0 0x7fb83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.500+0000 7fb853400640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb83406b4d0 0x7fb83406d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb84400b820 tx=0x7fb844008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.504+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb85415c3a0 con 0x7fb8540cc7a0 2024-03-06T07:35:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:03.705+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb8540b3480 con 0x7fb8540cc7a0 2024-03-06T07:35:04.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.239+0000 7fb84b400640 1 -- 172.21.15.32:0/1168295284 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb8400583e0 con 0x7fb8540cc7a0 2024-03-06T07:35:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb83406b4d0 msgr2=0x7fb83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb83406b4d0 0x7fb83406d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb84400b820 tx=0x7fb844008040 comp rx=0 tx=0).stop 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 msgr2=0x7fb8540adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 0x7fb8540adf40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c780 tx=0x7fb84000cc40 comp rx=0 tx=0).stop 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 shutdown_connections 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb83406b4d0 0x7fb83406d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb8541446f0 0x7fb8540ae480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8540cc7a0 0x7fb8540adf40 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 --2- 172.21.15.32:0/1168295284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85406e5d0 0x7fb8540cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.242+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 >> 172.21.15.32:0/1168295284 conn(0x7fb8540b1ae0 msgr2=0x7fb8540ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.243+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 shutdown_connections 2024-03-06T07:35:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:04.243+0000 7fb85a200640 1 -- 172.21.15.32:0/1168295284 wait complete. 2024-03-06T07:35:04.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:35:04.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:35:04.439 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:35:04.440 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:35:04.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:35:04.442 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:35:04.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:35:04.445 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:35:04.446 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:35:04.447 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:35:04.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:35:04.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:35:04.450 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:35:04.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:35:04.452 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:35:04.453 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:35:04.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:35:04.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:35:04.456 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:35:04.457 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.458 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:35:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:35:04.462 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:35:04.463 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:35:04.464 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:35:04.465 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.465 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.465 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:35:04.465 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:35:04.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:35:04.501 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:35:04.666 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:35:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:35:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:35:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:35:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:35:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:35:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:35:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:35:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:35:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:35:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:35:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:35:04.678 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:35:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:35:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:35:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:35:04.681 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:35:04.681 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:35:04.681 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:35:04.681 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:35:04.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:35:04.683 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.684 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.685 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.686 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:35:04.687 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:35:04.688 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:35:04.689 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.690 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:35:04.691 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.692 INFO:teuthology.orchestra.run.smithi039.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:35:04.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:35:04.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:35:04.912 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-06T07:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-06T07:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-06T07:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-06T07:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-06T07:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-06T07:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-06T07:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-06T07:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-06T07:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-03-06T07:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-03-06T07:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.966 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-06T07:35:04.966 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.967 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-03-06T07:35:04.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-03-06T07:35:04.969 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-03-06T07:35:04.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.32,172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790]" 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = e91bbd07-5556-4949-b381-99a73b242ae4 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.971 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.972 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-03-06T07:35:04.973 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.974 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:35:04.975 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-03-06T07:35:04.976 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-03-06T07:35:04.977 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-03-06T07:35:04.978 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-03-06T07:35:04.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:35:05.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.196+0000 7f2582a00640 1 Processor -- start 2024-03-06T07:35:05.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.197+0000 7f2582a00640 1 -- start start 2024-03-06T07:35:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.198+0000 7f2582a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f257c15ed90 0x7f257c0cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.198+0000 7f2582a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c15b8d0 con 0x7f257c15a4e0 2024-03-06T07:35:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.198+0000 7f2582a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0cdb50 con 0x7f257c0cf130 2024-03-06T07:35:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.198+0000 7f2582a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0cd1e0 con 0x7f257c15ed90 2024-03-06T07:35:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.199+0000 7f257be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f257c15a4e0 0x7f257c0aef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38952/0 (socket says 172.21.15.32:38952) 2024-03-06T07:35:05.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.199+0000 7f257be00640 1 -- 172.21.15.32:0/3887133681 learned_addr learned my addr 172.21.15.32:0/3887133681 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.199+0000 7f2573e00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f257c15ed90 0x7f257c0cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.199+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740219096 0 0) 0x7f257c15b8d0 con 0x7f257c15a4e0 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.199+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2560003520 con 0x7f257c15a4e0 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045701538 0 0) 0x7f2560003520 con 0x7f257c15a4e0 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f257c15ed90 msgr2=0x7f257c0cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f257c15ed90 0x7f257c0cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 >> v1:172.21.15.39:6789/0 conn(0x7f257c0cf130 legacy=0x7f257c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0cde30 con 0x7f257c15a4e0 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f256c001670 con 0x7f257c15a4e0 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 >> v1:172.21.15.32:6789/0 conn(0x7f257c15a4e0 legacy=0x7f257c0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25600039e0 0x7f2560003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2560004800 0x7f2560004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600051c0 0x7f256000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0cd1e0 con 0x7f25600039e0 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256000e110 con 0x7f2560004800 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256000e290 con 0x7f25600051c0 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.200+0000 7f257be00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25600039e0 0x7f2560003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f2573e00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2560004800 0x7f2560004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f2580a00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600051c0 0x7f256000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f257be00640 1 -- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600051c0 msgr2=0x7f256000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f257be00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600051c0 0x7f256000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f257be00640 1 -- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2560004800 msgr2=0x7f2560004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f257be00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2560004800 0x7f2560004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.201+0000 7f257be00640 1 -- 172.21.15.32:0/3887133681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560003520 con 0x7f25600039e0 2024-03-06T07:35:05.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.202+0000 7f257be00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25600039e0 0x7f2560003e00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c240 tx=0x7f2560005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a064bb41e91159 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.202+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c007cf0 con 0x7f25600039e0 2024-03-06T07:35:05.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.202+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c030070 con 0x7f25600039e0 2024-03-06T07:35:05.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.202+0000 7f257b400640 1 -- 172.21.15.32:0/3887133681 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c005d10 con 0x7f25600039e0 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2582a00640 1 -- 172.21.15.32:0/3887133681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25600039e0 msgr2=0x7f2560003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2582a00640 1 --2- 172.21.15.32:0/3887133681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25600039e0 0x7f2560003e00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c240 tx=0x7f2560005830 comp rx=0 tx=0).stop 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2580a00640 1 -- 172.21.15.32:0/3887133681 reap_dead start 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2582a00640 1 -- 172.21.15.32:0/3887133681 shutdown_connections 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2582a00640 1 -- 172.21.15.32:0/3887133681 >> 172.21.15.32:0/3887133681 conn(0x7f257c06b3d0 msgr2=0x7f257c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2582a00640 1 -- 172.21.15.32:0/3887133681 shutdown_connections 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.203+0000 7f2582a00640 1 -- 172.21.15.32:0/3887133681 wait complete. 2024-03-06T07:35:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.205+0000 7f2582a00640 1 Processor -- start 2024-03-06T07:35:05.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.205+0000 7f2582a00640 1 -- start start 2024-03-06T07:35:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2582a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600039e0 0x7f257c1ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2582a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0b1f90 0x7f257c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2582a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 0x7f257c09af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2582a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ce6d0 con 0x7f257c0b23f0 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2582a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ce3f0 con 0x7f257c0b1f90 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2582a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0af680 con 0x7f25600039e0 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2580a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 0x7f257c09af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2580a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 0x7f257c09af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51442/0 (socket says 172.21.15.32:51442) 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f2580a00640 1 -- 172.21.15.32:0/815843307 learned_addr learned my addr 172.21.15.32:0/815843307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.206+0000 7f257be00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600039e0 0x7f257c1ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2573e00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0b1f90 0x7f257c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2580a00640 1 -- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600039e0 msgr2=0x7f257c1ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2580a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600039e0 0x7f257c1ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2580a00640 1 -- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0b1f90 msgr2=0x7f257c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2580a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0b1f90 0x7f257c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2580a00640 1 -- 172.21.15.32:0/815843307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0b0810 con 0x7f257c0b23f0 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2573e00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0b1f90 0x7f257c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2580a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 0x7f257c09af50 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f257c06c020 tx=0x7f257400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574003970 con 0x7f257c0b23f0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2574018070 con 0x7f257c0b23f0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574003d50 con 0x7f257c0b23f0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.207+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0b11d0 con 0x7f257c0b23f0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.208+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0b1790 con 0x7f257c0b23f0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.209+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c04f330 con 0x7f257c0b23f0 2024-03-06T07:35:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.210+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25740050d0 con 0x7f257c0b23f0 2024-03-06T07:35:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.211+0000 7f2578c00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f255806b4d0 0x7f255806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.211+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2574089d00 con 0x7f257c0b23f0 2024-03-06T07:35:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.211+0000 7f257be00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f255806b4d0 0x7f255806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.212+0000 7f257be00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f255806b4d0 0x7f255806d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f256c034000 tx=0x7f256c007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.216+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2574058d80 con 0x7f257c0b23f0 2024-03-06T07:35:05.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.427+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f257c06dda0 con 0x7f257c0b23f0 2024-03-06T07:35:05.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.435+0000 7f2578c00640 1 -- 172.21.15.32:0/815843307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2574058680 con 0x7f257c0b23f0 2024-03-06T07:35:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.437+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f255806b4d0 msgr2=0x7f255806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f255806b4d0 0x7f255806d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f256c034000 tx=0x7f256c007000 comp rx=0 tx=0).stop 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 msgr2=0x7f257c09af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 0x7f257c09af50 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f257c06c020 tx=0x7f257400ec10 comp rx=0 tx=0).stop 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 shutdown_connections 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f255806b4d0 0x7f255806d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f257c0b23f0 0x7f257c09af50 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0b1f90 0x7f257c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 --2- 172.21.15.32:0/815843307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f25600039e0 0x7f257c1ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 >> 172.21.15.32:0/815843307 conn(0x7f257c06b3d0 msgr2=0x7f257c159640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 shutdown_connections 2024-03-06T07:35:05.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.438+0000 7f2582a00640 1 -- 172.21.15.32:0/815843307 wait complete. 2024-03-06T07:35:05.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:35:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.678+0000 7ff75b200640 1 Processor -- start 2024-03-06T07:35:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.678+0000 7ff75b200640 1 -- start start 2024-03-06T07:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.680+0000 7ff75b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff75c070f10 0x7ff75c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.680+0000 7ff75b200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c0714d0 con 0x7ff75c145300 2024-03-06T07:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.680+0000 7ff75b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c0717b0 con 0x7ff75c0cce90 2024-03-06T07:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.680+0000 7ff75b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75c0cd300 con 0x7ff75c070f10 2024-03-06T07:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.680+0000 7ff759e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff75c145300 0x7ff75c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38968/0 (socket says 172.21.15.32:38968) 2024-03-06T07:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.681+0000 7ff759e00640 1 -- 172.21.15.32:0/1326159211 learned_addr learned my addr 172.21.15.32:0/1326159211 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.681+0000 7ff753e00640 1 --2- 172.21.15.32:0/1326159211 >> v2:172.21.15.39:3300/0 conn(0x7ff75c070f10 0x7ff75c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.681+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519300459 0 0) 0x7ff75c0714d0 con 0x7ff75c145300 2024-03-06T07:35:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.681+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff740003520 con 0x7ff75c145300 2024-03-06T07:35:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.681+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1979279038 0 0) 0x7ff740003520 con 0x7ff75c145300 2024-03-06T07:35:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 >> v2:172.21.15.39:3300/0 conn(0x7ff75c070f10 msgr2=0x7ff75c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 --2- 172.21.15.32:0/1326159211 >> v2:172.21.15.39:3300/0 conn(0x7ff75c070f10 0x7ff75c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 >> v1:172.21.15.39:6789/0 conn(0x7ff75c0cce90 legacy=0x7ff75c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c071a90 con 0x7ff75c145300 2024-03-06T07:35:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff74c001670 con 0x7ff75c145300 2024-03-06T07:35:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 >> v1:172.21.15.32:6789/0 conn(0x7ff75c145300 legacy=0x7ff75c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c0cd300 con 0x7ff7400039b0 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74000c160 con 0x7ff7400051c0 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74000c360 con 0x7ff740004800 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff759e00640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff75a800640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.682+0000 7ff753e00640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.683+0000 7ff759e00640 1 -- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff740004800 msgr2=0x7ff740004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.683+0000 7ff759e00640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.683+0000 7ff759e00640 1 -- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7400051c0 msgr2=0x7ff74000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.683+0000 7ff759e00640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.683+0000 7ff75a800640 1 -- 172.21.15.32:0/1326159211 reap_dead start 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.683+0000 7ff759e00640 1 -- 172.21.15.32:0/1326159211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740003520 con 0x7ff7400039b0 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.684+0000 7ff759e00640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c240 tx=0x7ff740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d66c642772aacb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c007d10 con 0x7ff7400039b0 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c030070 con 0x7ff7400039b0 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff759400640 1 -- 172.21.15.32:0/1326159211 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c02cdc0 con 0x7ff7400039b0 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 -- 172.21.15.32:0/1326159211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7400039b0 msgr2=0x7ff740003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c240 tx=0x7ff740005830 comp rx=0 tx=0).stop 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 -- 172.21.15.32:0/1326159211 shutdown_connections 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 --2- 172.21.15.32:0/1326159211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 -- 172.21.15.32:0/1326159211 >> 172.21.15.32:0/1326159211 conn(0x7ff75c06b3d0 msgr2=0x7ff75c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 -- 172.21.15.32:0/1326159211 shutdown_connections 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.685+0000 7ff75b200640 1 -- 172.21.15.32:0/1326159211 wait complete. 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.687+0000 7ff75b200640 1 Processor -- start 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.687+0000 7ff75b200640 1 -- start start 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7400039b0 0x7ff75c13bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff75c15ac20 0x7ff75c13d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 0x7ff75c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c0cc950 con 0x7ff75c13dc40 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c1456e0 con 0x7ff75c15ac20 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c0ccad0 con 0x7ff7400039b0 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 0x7ff75c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 0x7ff75c15c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51462/0 (socket says 172.21.15.32:51462) 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.688+0000 7ff75a800640 1 -- 172.21.15.32:0/757487082 learned_addr learned my addr 172.21.15.32:0/757487082 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:05.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff753e00640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff75c15ac20 0x7ff75c13d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff759e00640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7400039b0 0x7ff75c13bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff75a800640 1 -- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7400039b0 msgr2=0x7ff75c13bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff75a800640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7400039b0 0x7ff75c13bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff75a800640 1 -- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff75c15ac20 msgr2=0x7ff75c13d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff75a800640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff75c15ac20 0x7ff75c13d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff75a800640 1 -- 172.21.15.32:0/757487082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c15b080 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff75a800640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 0x7ff75c15c6d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7ff75c069170 tx=0x7ff75400ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.689+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754019070 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.690+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754006d60 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.690+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754003b90 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.690+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c13acf0 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.690+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c13b2b0 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.691+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75c04f330 con 0x7ff75c13dc40 2024-03-06T07:35:05.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.692+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff75401c070 con 0x7ff75c13dc40 2024-03-06T07:35:05.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.693+0000 7ff752000640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff73806b4d0 0x7ff73806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.693+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff754089590 con 0x7ff75c13dc40 2024-03-06T07:35:05.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.693+0000 7ff759e00640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff73806b4d0 0x7ff73806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.694+0000 7ff759e00640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff73806b4d0 0x7ff73806d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff74c034000 tx=0x7ff74c009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.697+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff754058620 con 0x7ff75c13dc40 2024-03-06T07:35:05.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:05.983+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7ff75c13e0f0 con 0x7ff75c13dc40 2024-03-06T07:35:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.251+0000 7ff752000640 1 -- 172.21.15.32:0/757487082 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v344) v1 ==== 134+0+0 (secure 0 0 0) 0x7ff754057f20 con 0x7ff75c13dc40 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff73806b4d0 msgr2=0x7ff73806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff73806b4d0 0x7ff73806d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff74c034000 tx=0x7ff74c009030 comp rx=0 tx=0).stop 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 msgr2=0x7ff75c15c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 0x7ff75c15c6d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7ff75c069170 tx=0x7ff75400ea00 comp rx=0 tx=0).stop 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 shutdown_connections 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff73806b4d0 0x7ff73806d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff75c13dc40 0x7ff75c15c6d0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff75c15ac20 0x7ff75c13d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 --2- 172.21.15.32:0/757487082 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7400039b0 0x7ff75c13bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 >> 172.21.15.32:0/757487082 conn(0x7ff75c06b3d0 msgr2=0x7ff75c072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 shutdown_connections 2024-03-06T07:35:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.255+0000 7ff75b200640 1 -- 172.21.15.32:0/757487082 wait complete. 2024-03-06T07:35:06.281 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:35:06.281 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:35:06.282 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:35:06.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.500+0000 7f590b200640 1 Processor -- start 2024-03-06T07:35:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.500+0000 7f590b200640 1 -- start start 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590b200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f590c15e810 0x7f590c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590c15ed10 con 0x7f590c1552b0 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590b200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590c15eff0 con 0x7f590c15e810 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c15f2d0 con 0x7f590c0cd060 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590a800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f590c15e810 0x7f590c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590a800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f590c15e810 0x7f590c0b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51478/0 (socket says 172.21.15.32:51478) 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f590a800640 1 -- 172.21.15.32:0/2397353885 learned_addr learned my addr 172.21.15.32:0/2397353885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:06.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.502+0000 7f5909400640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.503+0000 7f590a800640 1 -- 172.21.15.32:0/2397353885 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 msgr2=0x7f590c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.503+0000 7f590a800640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.503+0000 7f590a800640 1 -- 172.21.15.32:0/2397353885 >> v1:172.21.15.39:6789/0 conn(0x7f590c1552b0 legacy=0x7f590c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.503+0000 7f590a800640 1 -- 172.21.15.32:0/2397353885 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c15f5b0 con 0x7f590c15e810 2024-03-06T07:35:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.503+0000 7f5909400640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.504+0000 7f590a800640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c0b3cb0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5904009e50 tx=0x7f590402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c135a18a4bdc8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.504+0000 7f5908a00640 1 -- 172.21.15.32:0/2397353885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590402ebf0 con 0x7f590c15e810 2024-03-06T07:35:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.504+0000 7f5908a00640 1 -- 172.21.15.32:0/2397353885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590402ed90 con 0x7f590c15e810 2024-03-06T07:35:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.504+0000 7f5908a00640 1 -- 172.21.15.32:0/2397353885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904036610 con 0x7f590c15e810 2024-03-06T07:35:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 -- 172.21.15.32:0/2397353885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 msgr2=0x7f590c0b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c0b3cb0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5904009e50 tx=0x7f590402e050 comp rx=0 tx=0).stop 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 -- 172.21.15.32:0/2397353885 shutdown_connections 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c0b3cb0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 --2- 172.21.15.32:0/2397353885 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 -- 172.21.15.32:0/2397353885 >> 172.21.15.32:0/2397353885 conn(0x7f590c06b3d0 msgr2=0x7f590c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 -- 172.21.15.32:0/2397353885 shutdown_connections 2024-03-06T07:35:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.505+0000 7f590b200640 1 -- 172.21.15.32:0/2397353885 wait complete. 2024-03-06T07:35:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.507+0000 7f590b200640 1 Processor -- start 2024-03-06T07:35:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.507+0000 7f590b200640 1 -- start start 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c0b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f590c1552b0 0x7f590c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c147f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c06dd30 con 0x7f590c15e810 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0b3760 con 0x7f590c1552b0 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c06deb0 con 0x7f590c0cd060 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c147f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c147f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51488/0 (socket says 172.21.15.32:51488) 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f590a800640 1 -- 172.21.15.32:0/4191973774 learned_addr learned my addr 172.21.15.32:0/4191973774 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.508+0000 7f5909e00640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c0b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f5909400640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f590c1552b0 0x7f590c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590a800640 1 -- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 msgr2=0x7f590c0b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590a800640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c0b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590a800640 1 -- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f590c1552b0 msgr2=0x7f590c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590a800640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f590c1552b0 0x7f590c070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590a800640 1 -- 172.21.15.32:0/4191973774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c0b0890 con 0x7f590c15e810 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f5909e00640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c0b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f5909400640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f590c1552b0 0x7f590c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590a800640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c147f00 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f5904038040 tx=0x7f5904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904045020 con 0x7f590c15e810 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5904040450 con 0x7f590c15e810 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904009510 con 0x7f590c15e810 2024-03-06T07:35:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.509+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590c0b0b70 con 0x7f590c15e810 2024-03-06T07:35:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.510+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590c0b05a0 con 0x7f590c15e810 2024-03-06T07:35:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.511+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590c04f330 con 0x7f590c15e810 2024-03-06T07:35:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.512+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5904005d10 con 0x7f590c15e810 2024-03-06T07:35:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.512+0000 7f5902000640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58e806b4d0 0x7f58e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.513+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f59040aead0 con 0x7f590c15e810 2024-03-06T07:35:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.513+0000 7f5909e00640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58e806b4d0 0x7f58e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.514+0000 7f5909e00640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58e806b4d0 0x7f58e806d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f58f4003eb0 tx=0x7f58f4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.518+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f590407dbe0 con 0x7f590c15e810 2024-03-06T07:35:06.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.720+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f590c0cf4a0 con 0x7f590c15e810 2024-03-06T07:35:06.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.722+0000 7f5902000640 1 -- 172.21.15.32:0/4191973774 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+13042 (secure 0 0 0) 0x7f590407d4e0 con 0x7f590c15e810 2024-03-06T07:35:06.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:06.725 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":102,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:04.236706+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1229875104":"2024-03-07T07:35:01.219323+0000","172.21.15.32:6834/1229875104":"2024-03-07T07:35:01.219323+0000","172.21.15.39:6835/556051904":"2024-03-07T07:34:25.418325+0000","172.21.15.39:6833/556051904":"2024-03-07T07:34:25.418325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:06.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58e806b4d0 msgr2=0x7f58e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58e806b4d0 0x7f58e806d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f58f4003eb0 tx=0x7f58f4009040 comp rx=0 tx=0).stop 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 msgr2=0x7f590c147f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c147f00 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f5904038040 tx=0x7f5904004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 shutdown_connections 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f58e806b4d0 0x7f58e806d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f590c15e810 0x7f590c147f00 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f590c1552b0 0x7f590c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 --2- 172.21.15.32:0/4191973774 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f590c0cd060 0x7f590c0b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 >> 172.21.15.32:0/4191973774 conn(0x7f590c06b3d0 msgr2=0x7f590c0cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 shutdown_connections 2024-03-06T07:35:06.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.725+0000 7f590b200640 1 -- 172.21.15.32:0/4191973774 wait complete. 2024-03-06T07:35:06.748 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:06.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:06.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.972+0000 7f18dfe00640 1 Processor -- start 2024-03-06T07:35:06.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.973+0000 7f18dfe00640 1 -- start start 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.974+0000 7f18dfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18e015f810 0x7f18e00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.974+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e00b2080 0x7f18e00b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.974+0000 7f18dfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e015fe40 con 0x7f18e0145430 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.974+0000 7f18dfe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e0160120 con 0x7f18e015f810 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.974+0000 7f18dfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e0160400 con 0x7f18e00b2080 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18e015f810 0x7f18e00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18e015f810 0x7f18e00b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51496/0 (socket says 172.21.15.32:51496) 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 -- 172.21.15.32:0/4107162404 learned_addr learned my addr 172.21.15.32:0/4107162404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:06.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18df400640 1 --2- 172.21.15.32:0/4107162404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e00b2080 0x7f18e00b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 -- 172.21.15.32:0/4107162404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e00b2080 msgr2=0x7f18e00b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:06.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 --2- 172.21.15.32:0/4107162404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e00b2080 0x7f18e00b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 -- 172.21.15.32:0/4107162404 >> v1:172.21.15.39:6789/0 conn(0x7f18e0145430 legacy=0x7f18e015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.975+0000 7f18d6000640 1 -- 172.21.15.32:0/4107162404 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e01606e0 con 0x7f18e015f810 2024-03-06T07:35:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.977+0000 7f18d6000640 1 --2- 172.21.15.32:0/4107162404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e015f810 0x7f18e00b1b40 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009870 tx=0x7f18c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b723576005f85f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.977+0000 7f18de000640 1 -- 172.21.15.32:0/4107162404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8034070 con 0x7f18e015f810 2024-03-06T07:35:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.977+0000 7f18de000640 1 -- 172.21.15.32:0/4107162404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c8030840 con 0x7f18e015f810 2024-03-06T07:35:06.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.977+0000 7f18de000640 1 -- 172.21.15.32:0/4107162404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8030c10 con 0x7f18e015f810 2024-03-06T07:35:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 -- 172.21.15.32:0/4107162404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e015f810 msgr2=0x7f18e00b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 --2- 172.21.15.32:0/4107162404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e015f810 0x7f18e00b1b40 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009870 tx=0x7f18c802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 -- 172.21.15.32:0/4107162404 shutdown_connections 2024-03-06T07:35:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 --2- 172.21.15.32:0/4107162404 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e00b2080 0x7f18e00b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 --2- 172.21.15.32:0/4107162404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e015f810 0x7f18e00b1b40 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 -- 172.21.15.32:0/4107162404 >> 172.21.15.32:0/4107162404 conn(0x7f18e006b3a0 msgr2=0x7f18e006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 -- 172.21.15.32:0/4107162404 shutdown_connections 2024-03-06T07:35:06.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.978+0000 7f18dfe00640 1 -- 172.21.15.32:0/4107162404 wait complete. 2024-03-06T07:35:06.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.980+0000 7f18dfe00640 1 Processor -- start 2024-03-06T07:35:06.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.980+0000 7f18dfe00640 1 -- start start 2024-03-06T07:35:06.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18e00b2080 0x7f18e0138ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 0x7f18e013b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e015f810 0x7f18e015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18dfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00cdf80 con 0x7f18e0145430 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18dfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00cde00 con 0x7f18e00b2080 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18dfe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00b3510 con 0x7f18e015f810 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18d6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 0x7f18e013b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18d6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 0x7f18e013b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51510/0 (socket says 172.21.15.32:51510) 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.981+0000 7f18d6000640 1 -- 172.21.15.32:0/3204178913 learned_addr learned my addr 172.21.15.32:0/3204178913 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18df400640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e015f810 0x7f18e015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d6000640 1 -- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e015f810 msgr2=0x7f18e015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d6000640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e015f810 0x7f18e015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d6000640 1 -- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18e00b2080 msgr2=0x7f18e0138ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d6000640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18e00b2080 0x7f18e0138ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d6000640 1 -- 172.21.15.32:0/3204178913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0139ab0 con 0x7f18e0145430 2024-03-06T07:35:06.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18df400640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e015f810 0x7f18e015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:06.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d6000640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 0x7f18e013b6f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f18c802b9c0 tx=0x7f18c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8034040 con 0x7f18e0145430 2024-03-06T07:35:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c803ada0 con 0x7f18e0145430 2024-03-06T07:35:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8044610 con 0x7f18e0145430 2024-03-06T07:35:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.982+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e0139d90 con 0x7f18e0145430 2024-03-06T07:35:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.983+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e013f380 con 0x7f18e0145430 2024-03-06T07:35:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.984+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e004f330 con 0x7f18e0145430 2024-03-06T07:35:06.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.985+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18c804b060 con 0x7f18e0145430 2024-03-06T07:35:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.986+0000 7f18d7400640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18bc06b520 0x7f18bc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.986+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f18c80abfd0 con 0x7f18e0145430 2024-03-06T07:35:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.986+0000 7f18dea00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18bc06b520 0x7f18bc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.987+0000 7f18dea00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18bc06b520 0x7f18bc06d9e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f18d0000f60 tx=0x7f18d00079d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:06.990+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18c803f070 con 0x7f18e0145430 2024-03-06T07:35:07.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.197+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18e006e240 con 0x7f18e0145430 2024-03-06T07:35:07.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.198+0000 7f18d7400640 1 -- 172.21.15.32:0/3204178913 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f18c802d330 con 0x7f18e0145430 2024-03-06T07:35:07.201 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18bc06b520 msgr2=0x7f18bc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18bc06b520 0x7f18bc06d9e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f18d0000f60 tx=0x7f18d00079d0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 msgr2=0x7f18e013b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 0x7f18e013b6f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f18c802b9c0 tx=0x7f18c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 shutdown_connections 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18bc06b520 0x7f18bc06d9e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18e015f810 0x7f18e015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18e0145430 0x7f18e013b6f0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 --2- 172.21.15.32:0/3204178913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18e00b2080 0x7f18e0138ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.201+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 >> 172.21.15.32:0/3204178913 conn(0x7f18e006b3a0 msgr2=0x7f18e0140530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.202+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 shutdown_connections 2024-03-06T07:35:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.202+0000 7f18dfe00640 1 -- 172.21.15.32:0/3204178913 wait complete. 2024-03-06T07:35:07.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:35:07.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.438+0000 7f59b2000640 1 Processor -- start 2024-03-06T07:35:07.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.438+0000 7f59b2000640 1 -- start start 2024-03-06T07:35:07.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.440+0000 7f59b2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f59ac0af1a0 0x7f59ac15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.440+0000 7f59b2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.440+0000 7f59b2000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ac15cc40 con 0x7f59ac0cdc20 2024-03-06T07:35:07.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.440+0000 7f59b2000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ac15cf20 con 0x7f59ac0af1a0 2024-03-06T07:35:07.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.440+0000 7f59b2000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac15d200 con 0x7f59ac0af580 2024-03-06T07:35:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.441+0000 7f59ab400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f59ac0af1a0 0x7f59ac15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.441+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.441+0000 7f59ab400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f59ac0af1a0 0x7f59ac15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:35648/0 (socket says 172.21.15.32:35648) 2024-03-06T07:35:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.441+0000 7f59ab400640 1 -- 172.21.15.32:0/1619507723 learned_addr learned my addr 172.21.15.32:0/1619507723 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.441+0000 7f59abe00640 1 --1- 172.21.15.32:0/1619507723 >> v1:172.21.15.39:6789/0 conn(0x7f59ac0cdc20 0x7f59ac0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:43064/0 (socket says 172.21.15.32:43064) 2024-03-06T07:35:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.442+0000 7f59ab400640 1 -- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 msgr2=0x7f59ac0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.442+0000 7f59ab400640 1 --2- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.442+0000 7f59ab400640 1 -- 172.21.15.32:0/1619507723 >> v1:172.21.15.39:6789/0 conn(0x7f59ac0cdc20 legacy=0x7f59ac0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.442+0000 7f59ab400640 1 -- 172.21.15.32:0/1619507723 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac15d4e0 con 0x7f59ac0af1a0 2024-03-06T07:35:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.442+0000 7f59a3e00640 1 --2- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.443+0000 7f59ab400640 1 --2- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0af1a0 0x7f59ac15c600 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f599c009870 tx=0x7f599c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c75e77fdf8b9066a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.444+0000 7f59aaa00640 1 -- 172.21.15.32:0/1619507723 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c035020 con 0x7f59ac0af1a0 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.444+0000 7f59aaa00640 1 -- 172.21.15.32:0/1619507723 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c031640 con 0x7f59ac0af1a0 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.444+0000 7f59aaa00640 1 -- 172.21.15.32:0/1619507723 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c02d040 con 0x7f59ac0af1a0 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.444+0000 7f59b2000640 1 -- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0af1a0 msgr2=0x7f59ac15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.444+0000 7f59b2000640 1 --2- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0af1a0 0x7f59ac15c600 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f599c009870 tx=0x7f599c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.445+0000 7f59b2000640 1 -- 172.21.15.32:0/1619507723 shutdown_connections 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.445+0000 7f59b2000640 1 --2- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.445+0000 7f59b2000640 1 --2- 172.21.15.32:0/1619507723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0af1a0 0x7f59ac15c600 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.445+0000 7f59b2000640 1 -- 172.21.15.32:0/1619507723 >> 172.21.15.32:0/1619507723 conn(0x7f59ac06afb0 msgr2=0x7f59ac06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.445+0000 7f59b2000640 1 -- 172.21.15.32:0/1619507723 shutdown_connections 2024-03-06T07:35:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.445+0000 7f59b2000640 1 -- 172.21.15.32:0/1619507723 wait complete. 2024-03-06T07:35:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.446+0000 7f59b2000640 1 Processor -- start 2024-03-06T07:35:07.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 -- start start 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 0x7f59ac1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac15e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0cdc20 0x7f59ac1492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0b3660 con 0x7f59ac0af1a0 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0b34e0 con 0x7f59ac0cdc20 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.447+0000 7f59b2000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0b37e0 con 0x7f59ac0af580 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 0x7f59ac1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 0x7f59ac1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51512/0 (socket says 172.21.15.32:51512) 2024-03-06T07:35:07.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 -- 172.21.15.32:0/1075201669 learned_addr learned my addr 172.21.15.32:0/1075201669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59abe00640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0cdc20 0x7f59ac1492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59a3e00640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac15e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 -- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 msgr2=0x7f59ac15e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac15e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 -- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0cdc20 msgr2=0x7f59ac1492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0cdc20 0x7f59ac1492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.448+0000 7f59ab400640 1 -- 172.21.15.32:0/1075201669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac09aa10 con 0x7f59ac0af1a0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.449+0000 7f59ab400640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 0x7f59ac1581a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f599c000c00 tx=0x7f599c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.449+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c040070 con 0x7f59ac0af1a0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.449+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c0399f0 con 0x7f59ac0af1a0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.449+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c035040 con 0x7f59ac0af1a0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.449+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ac09acb0 con 0x7f59ac0af1a0 2024-03-06T07:35:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.449+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ac09b270 con 0x7f59ac0af1a0 2024-03-06T07:35:07.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.450+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ac04f3a0 con 0x7f59ac0af1a0 2024-03-06T07:35:07.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.451+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f599c04b060 con 0x7f59ac0af1a0 2024-03-06T07:35:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.452+0000 7f59a3400640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f598806b4d0 0x7f598806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.452+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f599c0ab970 con 0x7f59ac0af1a0 2024-03-06T07:35:07.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.452+0000 7f59a3e00640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f598806b4d0 0x7f598806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.453+0000 7f59a3e00640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f598806b4d0 0x7f598806d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f59ac15eca0 tx=0x7f5994008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.456+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f599c07a960 con 0x7f59ac0af1a0 2024-03-06T07:35:07.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:07.730+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f59ac0ceb30 con 0x7f59ac0af1a0 2024-03-06T07:35:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.273+0000 7f59a3400640 1 -- 172.21.15.32:0/1075201669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v346) v1 ==== 145+0+0 (secure 0 0 0) 0x7f599c07a260 con 0x7f59ac0af1a0 2024-03-06T07:35:08.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f598806b4d0 msgr2=0x7f598806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f598806b4d0 0x7f598806d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f59ac15eca0 tx=0x7f5994008040 comp rx=0 tx=0).stop 2024-03-06T07:35:08.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 msgr2=0x7f59ac1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 0x7f59ac1581a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f599c000c00 tx=0x7f599c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 shutdown_connections 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f598806b4d0 0x7f598806d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59ac0cdc20 0x7f59ac1492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f59ac0af580 0x7f59ac15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 --2- 172.21.15.32:0/1075201669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ac0af1a0 0x7f59ac1581a0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 >> 172.21.15.32:0/1075201669 conn(0x7f59ac06afb0 msgr2=0x7f59ac15bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 shutdown_connections 2024-03-06T07:35:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.277+0000 7f59b2000640 1 -- 172.21.15.32:0/1075201669 wait complete. 2024-03-06T07:35:08.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:35:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.515+0000 7fe370e00640 1 Processor -- start 2024-03-06T07:35:08.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.516+0000 7fe370e00640 1 -- start start 2024-03-06T07:35:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.517+0000 7fe370e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe36c15dee0 0x7fe36c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.517+0000 7fe370e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c0d0d40 0x7fe36c0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.517+0000 7fe370e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0ceca0 con 0x7fe36c0d4860 2024-03-06T07:35:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.517+0000 7fe370e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c0cf5d0 con 0x7fe36c15dee0 2024-03-06T07:35:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.517+0000 7fe370e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0cef80 con 0x7fe36c0d0d40 2024-03-06T07:35:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.518+0000 7fe36be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c0d0d40 0x7fe36c0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.518+0000 7fe36b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe36c15dee0 0x7fe36c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.518+0000 7fe36be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c0d0d40 0x7fe36c0b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:55174/0 (socket says 172.21.15.32:55174) 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.518+0000 7fe36be00640 1 -- 172.21.15.32:0/2386642210 learned_addr learned my addr 172.21.15.32:0/2386642210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.519+0000 7fe36b400640 1 -- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c0d0d40 msgr2=0x7fe36c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.519+0000 7fe36b400640 1 --2- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c0d0d40 0x7fe36c0b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.519+0000 7fe36b400640 1 -- 172.21.15.32:0/2386642210 >> v1:172.21.15.39:6789/0 conn(0x7fe36c0d4860 legacy=0x7fe36c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.519+0000 7fe36b400640 1 -- 172.21.15.32:0/2386642210 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0cf260 con 0x7fe36c15dee0 2024-03-06T07:35:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.520+0000 7fe36b400640 1 --2- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c15dee0 0x7fe36c0d18a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0097a0 tx=0x7fe35c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e4e5aab7b0caadf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.521+0000 7fe36a000640 1 -- 172.21.15.32:0/2386642210 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c035020 con 0x7fe36c15dee0 2024-03-06T07:35:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.521+0000 7fe36a000640 1 -- 172.21.15.32:0/2386642210 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c031810 con 0x7fe36c15dee0 2024-03-06T07:35:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.521+0000 7fe36a000640 1 -- 172.21.15.32:0/2386642210 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c031be0 con 0x7fe36c15dee0 2024-03-06T07:35:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.521+0000 7fe370e00640 1 -- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c15dee0 msgr2=0x7fe36c0d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.521+0000 7fe370e00640 1 --2- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c15dee0 0x7fe36c0d18a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0097a0 tx=0x7fe35c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.522+0000 7fe370e00640 1 -- 172.21.15.32:0/2386642210 shutdown_connections 2024-03-06T07:35:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.522+0000 7fe370e00640 1 --2- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c0d0d40 0x7fe36c0b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.522+0000 7fe370e00640 1 --2- 172.21.15.32:0/2386642210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c15dee0 0x7fe36c0d18a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.522+0000 7fe370e00640 1 -- 172.21.15.32:0/2386642210 >> 172.21.15.32:0/2386642210 conn(0x7fe36c0d3cd0 msgr2=0x7fe36c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.522+0000 7fe370e00640 1 -- 172.21.15.32:0/2386642210 shutdown_connections 2024-03-06T07:35:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.522+0000 7fe370e00640 1 -- 172.21.15.32:0/2386642210 wait complete. 2024-03-06T07:35:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.523+0000 7fe370e00640 1 Processor -- start 2024-03-06T07:35:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.523+0000 7fe370e00640 1 -- start start 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.524+0000 7fe370e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 0x7fe36c0ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.524+0000 7fe370e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c0d4860 0x7fe36c0c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe370e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c15dee0 0x7fe36c1606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe370e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0d1de0 con 0x7fe36c0d0d40 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe370e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c15e2c0 con 0x7fe36c0d4860 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe370e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0d1f60 con 0x7fe36c15dee0 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 0x7fe36c0ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 0x7fe36c0ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39606/0 (socket says 172.21.15.32:39606) 2024-03-06T07:35:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 -- 172.21.15.32:0/1256926169 learned_addr learned my addr 172.21.15.32:0/1256926169 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36be00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c15dee0 0x7fe36c1606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 -- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c15dee0 msgr2=0x7fe36c1606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36aa00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c0d4860 0x7fe36c0c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c15dee0 0x7fe36c1606f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 -- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c0d4860 msgr2=0x7fe36c0c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c0d4860 0x7fe36c0c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36b400640 1 -- 172.21.15.32:0/1256926169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0c8980 con 0x7fe36c0d0d40 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.525+0000 7fe36aa00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c0d4860 0x7fe36c0c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.526+0000 7fe36b400640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 0x7fe36c0ca320 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe35c034040 tx=0x7fe35c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.526+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c040070 con 0x7fe36c0d0d40 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.526+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c039cf0 con 0x7fe36c0d0d40 2024-03-06T07:35:08.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.526+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c035040 con 0x7fe36c0d0d40 2024-03-06T07:35:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.526+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c0c8c60 con 0x7fe36c0d0d40 2024-03-06T07:35:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.526+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c0c9220 con 0x7fe36c0d0d40 2024-03-06T07:35:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.528+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c04f330 con 0x7fe36c0d0d40 2024-03-06T07:35:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.528+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe35c038020 con 0x7fe36c0d0d40 2024-03-06T07:35:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.529+0000 7fe363400640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe34806b500 0x7fe34806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.529+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe35c0abb60 con 0x7fe36c0d0d40 2024-03-06T07:35:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.530+0000 7fe36aa00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe34806b500 0x7fe34806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.530+0000 7fe36aa00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe34806b500 0x7fe34806d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe36c0c9ef0 tx=0x7fe354009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.534+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe35c07abe0 con 0x7fe36c0d0d40 2024-03-06T07:35:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.735+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe36c0c75a0 con 0x7fe36c0d0d40 2024-03-06T07:35:08.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.742+0000 7fe363400640 1 -- 172.21.15.32:0/1256926169 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe35c07a4e0 con 0x7fe36c0d0d40 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.745+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe34806b500 msgr2=0x7fe34806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.745+0000 7fe370e00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe34806b500 0x7fe34806d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe36c0c9ef0 tx=0x7fe354009040 comp rx=0 tx=0).stop 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.745+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 msgr2=0x7fe36c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.745+0000 7fe370e00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 0x7fe36c0ca320 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe35c034040 tx=0x7fe35c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 shutdown_connections 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe34806b500 0x7fe34806d9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe36c15dee0 0x7fe36c1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe36c0d4860 0x7fe36c0c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 --2- 172.21.15.32:0/1256926169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c0d0d40 0x7fe36c0ca320 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 >> 172.21.15.32:0/1256926169 conn(0x7fe36c0d3cd0 msgr2=0x7fe36c0cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 shutdown_connections 2024-03-06T07:35:08.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:08.746+0000 7fe370e00640 1 -- 172.21.15.32:0/1256926169 wait complete. 2024-03-06T07:35:08.766 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:08.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:08.881 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:09.004 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:09.004 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:09.004 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:09.065 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:09.065 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:35:09.065 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:09.124 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:09.124 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:09.124 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:09.184 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:09.184 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:35:09.185 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:09.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:09.243 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:35:09.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:09.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.464+0000 7f8facc00640 1 Processor -- start 2024-03-06T07:35:09.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.464+0000 7f8facc00640 1 -- start start 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8facc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8fa8070f10 0x7f8fa80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8facc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8facc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80714d0 con 0x7f8fa8145300 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8facc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa80cd300 con 0x7f8fa8070f10 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8facc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa80717b0 con 0x7f8fa80cce90 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8fa7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8fa8145300 0x7f8fa80b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35822/0 (socket says 172.21.15.32:35822) 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8fa7400640 1 -- 172.21.15.32:0/3637779329 learned_addr learned my addr 172.21.15.32:0/3637779329 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8fa6a00640 1 --2- 172.21.15.32:0/3637779329 >> v2:172.21.15.32:3300/0 conn(0x7f8fa8070f10 0x7f8fa80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.466+0000 7f8fa7e00640 1 --2- 172.21.15.32:0/3637779329 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.467+0000 7f8fa6000640 1 -- 172.21.15.32:0/3637779329 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260647298 0 0) 0x7f8fa80714d0 con 0x7f8fa8145300 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.467+0000 7f8fa6000640 1 -- 172.21.15.32:0/3637779329 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003520 con 0x7f8fa8145300 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.467+0000 7f8fa6a00640 1 -- 172.21.15.32:0/3637779329 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 msgr2=0x7f8fa815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.467+0000 7f8fa6a00640 1 --2- 172.21.15.32:0/3637779329 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.467+0000 7f8fa6a00640 1 -- 172.21.15.32:0/3637779329 >> v1:172.21.15.32:6789/0 conn(0x7f8fa8145300 legacy=0x7f8fa80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.467+0000 7f8fa6a00640 1 -- 172.21.15.32:0/3637779329 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa8071a90 con 0x7f8fa8070f10 2024-03-06T07:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.468+0000 7f8fa6a00640 1 --2- 172.21.15.32:0/3637779329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8070f10 0x7f8fa80cc950 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8f940097a0 tx=0x7f8f9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7854cb0a2c9358b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8fa6000640 1 -- 172.21.15.32:0/3637779329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f940315a0 con 0x7f8fa8070f10 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8fa6000640 1 -- 172.21.15.32:0/3637779329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94037070 con 0x7f8fa8070f10 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8fa6000640 1 -- 172.21.15.32:0/3637779329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94031960 con 0x7f8fa8070f10 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 -- 172.21.15.32:0/3637779329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8070f10 msgr2=0x7f8fa80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 --2- 172.21.15.32:0/3637779329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8070f10 0x7f8fa80cc950 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8f940097a0 tx=0x7f8f9402b840 comp rx=0 tx=0).stop 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 -- 172.21.15.32:0/3637779329 shutdown_connections 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 --2- 172.21.15.32:0/3637779329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8070f10 0x7f8fa80cc950 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 --2- 172.21.15.32:0/3637779329 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 -- 172.21.15.32:0/3637779329 >> 172.21.15.32:0/3637779329 conn(0x7f8fa806b3d0 msgr2=0x7f8fa806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 -- 172.21.15.32:0/3637779329 shutdown_connections 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.469+0000 7f8facc00640 1 -- 172.21.15.32:0/3637779329 wait complete. 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.471+0000 7f8facc00640 1 Processor -- start 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.471+0000 7f8facc00640 1 -- start start 2024-03-06T07:35:09.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8facc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa8070f10 0x7f8fa80c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8facc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa81f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8facc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 0x7f8fa8142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8facc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa815cab0 con 0x7f8fa8145300 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8facc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa81456e0 con 0x7f8fa8070f10 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8facc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa815cc30 con 0x7f8fa80cce90 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8fa7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 0x7f8fa8142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8fa7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 0x7f8fa8142c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39626/0 (socket says 172.21.15.32:39626) 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.472+0000 7f8fa7e00640 1 -- 172.21.15.32:0/1972751942 learned_addr learned my addr 172.21.15.32:0/1972751942 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7400640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa8070f10 0x7f8fa80c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa6a00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa81f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7e00640 1 -- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 msgr2=0x7f8fa81f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7e00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa81f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7e00640 1 -- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa8070f10 msgr2=0x7f8fa80c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7e00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa8070f10 0x7f8fa80c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7e00640 1 -- 172.21.15.32:0/1972751942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80c94c0 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa6a00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa81f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8fa7e00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 0x7f8fa8142c90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000efc0 tx=0x7f8fa000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0018020 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa0014480 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0014870 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.473+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa81f4a80 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.474+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa806c860 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.475+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa80c8f60 con 0x7f8fa8145300 2024-03-06T07:35:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.476+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fa0005d10 con 0x7f8fa8145300 2024-03-06T07:35:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.476+0000 7f8f9f400640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.477+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8fa008aaf0 con 0x7f8fa8145300 2024-03-06T07:35:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.477+0000 7f8fa7400640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.477+0000 7f8fa7400640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8f8806b4d0 0x7f8f8806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8f98007fd0 tx=0x7f8f98009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.481+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fa80c8f60 con 0x7f8fa8145300 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.721+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa80b2cc0 con 0x7f8fa8145300 2024-03-06T07:35:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.722+0000 7f8f9f400640 1 -- 172.21.15.32:0/1972751942 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f8fa0022070 con 0x7f8fa8145300 2024-03-06T07:35:09.725 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:09.725 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5465,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2560036528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2560036528},{"type":"v1","addr":"172.21.15.39:6835","nonce":2560036528}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":5995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3437257449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3437257449},{"type":"v1","addr":"172.21.15.39:6833","nonce":3437257449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/3323445863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3323445863},{"type":"v1","addr":"172.21.15.32:6837","nonce":3323445863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7575,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2137892514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2137892514},{"type":"v1","addr":"172.21.15.32:6835","nonce":2137892514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[]} 2024-03-06T07:35:09.725 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 69 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8f8806b4d0 msgr2=0x7f8f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8f8806b4d0 0x7f8f8806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8f98007fd0 tx=0x7f8f98009040 comp rx=0 tx=0).stop 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 msgr2=0x7f8fa8142c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 0x7f8fa8142c90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000efc0 tx=0x7f8fa000c490 comp rx=0 tx=0).stop 2024-03-06T07:35:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 shutdown_connections 2024-03-06T07:35:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fa8145300 0x7f8fa8142c90 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8fa80cce90 0x7f8fa81f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 --2- 172.21.15.32:0/1972751942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa8070f10 0x7f8fa80c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.725+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 >> 172.21.15.32:0/1972751942 conn(0x7f8fa806b3d0 msgr2=0x7f8fa80b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.726+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 shutdown_connections 2024-03-06T07:35:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:09.726+0000 7f8facc00640 1 -- 172.21.15.32:0/1972751942 wait complete. 2024-03-06T07:35:09.748 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:35:09.748 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:35:09.748 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:35:09.748 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:09.749 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:35:09.749 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:35:09.749 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:35:09.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:09.749 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:35:09.749 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:35:09.749 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:35:09.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:09.750 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:35:09.750 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:35:09.750 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:35:09.750 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:09.771 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:35:09.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:35:09.772 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:35:09.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:35:09.774 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:35:09.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:35:09.775 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:35:09.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:35:09.820 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:35:09.821 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:09.822 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:35:09.822 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:09.827 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:35:09.827 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:09.828 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:35:09.829 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:09.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:35:09.976 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:35:09.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-06T07:35:09.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-06T07:35:09.977 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:35:09.994 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:35:09.994 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.015+0000 7fd56ba00640 1 Processor -- start 2024-03-06T07:35:10.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.015+0000 7fd56ba00640 1 -- start start 2024-03-06T07:35:10.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.016+0000 7fd56ba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd56415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.016+0000 7fd56ba00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56415ed30 con 0x7fd56415dd00 2024-03-06T07:35:10.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.016+0000 7fd56ba00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd564161a80 con 0x7fd56406c6d0 2024-03-06T07:35:10.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.016+0000 7fd56ba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd564162940 con 0x7fd564162520 2024-03-06T07:35:10.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.016+0000 7fd569000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd56415dd00 0x7fd56406bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35834/0 (socket says 172.21.15.32:35834) 2024-03-06T07:35:10.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.016+0000 7fd569000640 1 -- 172.21.15.32:0/1092319364 learned_addr learned my addr 172.21.15.32:0/1092319364 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:10.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673865635 0 0) 0x7fd56415ed30 con 0x7fd56415dd00 2024-03-06T07:35:10.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd569a00640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd56415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c003520 con 0x7fd56415dd00 2024-03-06T07:35:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434759565 0 0) 0x7fd54c003520 con 0x7fd56415dd00 2024-03-06T07:35:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 msgr2=0x7fd56415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd56415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 >> v1:172.21.15.39:6789/0 conn(0x7fd56406c6d0 legacy=0x7fd56415e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56415efa0 con 0x7fd56415dd00 2024-03-06T07:35:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd558001670 con 0x7fd56415dd00 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 >> v1:172.21.15.32:6789/0 conn(0x7fd56415dd00 legacy=0x7fd56406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd54c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd54c003f90 0x7fd54c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c004d40 0x7fd54c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564162940 con 0x7fd54c004d40 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c00bb00 con 0x7fd54c003f90 2024-03-06T07:35:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.017+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c00bd00 con 0x7fd564162520 2024-03-06T07:35:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569000640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd54c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569a00640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c004d40 0x7fd54c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd563e00640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd54c003f90 0x7fd54c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569000640 1 -- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd54c003f90 msgr2=0x7fd54c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569000640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd54c003f90 0x7fd54c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569000640 1 -- 172.21.15.32:0/1092319364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c004d40 msgr2=0x7fd54c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569000640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c004d40 0x7fd54c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.018+0000 7fd569000640 1 -- 172.21.15.32:0/1092319364 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c003520 con 0x7fd564162520 2024-03-06T07:35:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.019+0000 7fd569000640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd54c003a50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c190 tx=0x7fd54c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8837d2e8711247fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.019+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558007c60 con 0x7fd564162520 2024-03-06T07:35:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.019+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558030070 con 0x7fd564162520 2024-03-06T07:35:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.019+0000 7fd563400640 1 -- 172.21.15.32:0/1092319364 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558005d50 con 0x7fd564162520 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd56ba00640 1 -- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 msgr2=0x7fd54c003a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd56ba00640 1 --2- 172.21.15.32:0/1092319364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564162520 0x7fd54c003a50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c190 tx=0x7fd54c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd569a00640 1 -- 172.21.15.32:0/1092319364 reap_dead start 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd56ba00640 1 -- 172.21.15.32:0/1092319364 shutdown_connections 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd56ba00640 1 -- 172.21.15.32:0/1092319364 >> 172.21.15.32:0/1092319364 conn(0x7fd5640714d0 msgr2=0x7fd5640718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd56ba00640 1 -- 172.21.15.32:0/1092319364 shutdown_connections 2024-03-06T07:35:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.020+0000 7fd56ba00640 1 -- 172.21.15.32:0/1092319364 wait complete. 2024-03-06T07:35:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.022+0000 7fd56ba00640 1 Processor -- start 2024-03-06T07:35:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.022+0000 7fd56ba00640 1 -- start start 2024-03-06T07:35:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd56ba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 0x7fd564147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd56ba00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564154df0 0x7fd564155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd56ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5640afa40 0x7fd5640afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd56ba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564162000 con 0x7fd564162520 2024-03-06T07:35:10.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd56ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564161d20 con 0x7fd5640afa40 2024-03-06T07:35:10.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd56ba00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56406f280 con 0x7fd564154df0 2024-03-06T07:35:10.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd569000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 0x7fd564147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd569000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 0x7fd564147ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39682/0 (socket says 172.21.15.32:39682) 2024-03-06T07:35:10.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd569000640 1 -- 172.21.15.32:0/2015604809 learned_addr learned my addr 172.21.15.32:0/2015604809 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.023+0000 7fd563e00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564154df0 0x7fd564155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569a00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5640afa40 0x7fd5640afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569000640 1 -- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564154df0 msgr2=0x7fd564155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569000640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564154df0 0x7fd564155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569000640 1 -- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5640afa40 msgr2=0x7fd5640afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569000640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5640afa40 0x7fd5640afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569000640 1 -- 172.21.15.32:0/2015604809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd564148410 con 0x7fd564162520 2024-03-06T07:35:10.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd569000640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 0x7fd564147ed0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c1f0 tx=0x7fd56415f170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55802cba0 con 0x7fd564162520 2024-03-06T07:35:10.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558000ed0 con 0x7fd564162520 2024-03-06T07:35:10.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558007070 con 0x7fd564162520 2024-03-06T07:35:10.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.024+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd564156ca0 con 0x7fd564162520 2024-03-06T07:35:10.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.025+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd564157260 con 0x7fd564162520 2024-03-06T07:35:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.026+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd564157920 con 0x7fd564162520 2024-03-06T07:35:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.027+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd558047020 con 0x7fd564162520 2024-03-06T07:35:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.027+0000 7fd560c00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd54406b4d0 0x7fd54406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.027+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd5580ab150 con 0x7fd564162520 2024-03-06T07:35:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.028+0000 7fd563e00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd54406b4d0 0x7fd54406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.028+0000 7fd563e00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd54406b4d0 0x7fd54406d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd564158530 tx=0x7fd550009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.032+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd564157920 con 0x7fd564162520 2024-03-06T07:35:10.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.244+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd56404f3a0 con 0x7fd564162520 2024-03-06T07:35:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.313+0000 7fd560c00640 1 -- 172.21.15.32:0/2015604809 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd558079c10 con 0x7fd564162520 2024-03-06T07:35:10.316 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:35:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd54406b4d0 msgr2=0x7fd54406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd54406b4d0 0x7fd54406d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd564158530 tx=0x7fd550009040 comp rx=0 tx=0).stop 2024-03-06T07:35:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 msgr2=0x7fd564147ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 0x7fd564147ed0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c1f0 tx=0x7fd56415f170 comp rx=0 tx=0).stop 2024-03-06T07:35:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 shutdown_connections 2024-03-06T07:35:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd54406b4d0 0x7fd54406d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5640afa40 0x7fd5640afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd564154df0 0x7fd564155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 --2- 172.21.15.32:0/2015604809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd564162520 0x7fd564147ed0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.318+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 >> 172.21.15.32:0/2015604809 conn(0x7fd5640714d0 msgr2=0x7fd56406e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.319+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 shutdown_connections 2024-03-06T07:35:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.319+0000 7fd56ba00640 1 -- 172.21.15.32:0/2015604809 wait complete. 2024-03-06T07:35:10.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:35:10.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.558+0000 7fd51ee00640 1 Processor -- start 2024-03-06T07:35:10.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.559+0000 7fd51ee00640 1 -- start start 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.560+0000 7fd51ee00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd5180d2960 0x7fd5180d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.560+0000 7fd51ee00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180d2dc0 con 0x7fd5180d0b00 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.560+0000 7fd51ee00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180d18c0 con 0x7fd51815e070 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.560+0000 7fd51ee00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180d0f70 con 0x7fd5180d2960 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd5180d2960 0x7fd5180d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd5180d2960 0x7fd5180d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39694/0 (socket says 172.21.15.32:39694) 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51d000640 1 -- 172.21.15.32:0/1067191445 learned_addr learned my addr 172.21.15.32:0/1067191445 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51e400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5180d0b00 0x7fd5180b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35844/0 (socket says 172.21.15.32:35844) 2024-03-06T07:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51d000640 1 -- 172.21.15.32:0/1067191445 >> v1:172.21.15.39:6789/0 conn(0x7fd51815e070 legacy=0x7fd5180cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51d000640 1 -- 172.21.15.32:0/1067191445 >> v1:172.21.15.32:6789/0 conn(0x7fd5180d0b00 legacy=0x7fd5180b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.561+0000 7fd51d000640 1 -- 172.21.15.32:0/1067191445 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5180d1ba0 con 0x7fd5180d2960 2024-03-06T07:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.562+0000 7fd51d000640 1 --2- 172.21.15.32:0/1067191445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180d05c0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd508009870 tx=0x7fd50802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a63a605df03ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd517e00640 1 -- 172.21.15.32:0/1067191445 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508034070 con 0x7fd5180d2960 2024-03-06T07:35:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd517e00640 1 -- 172.21.15.32:0/1067191445 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508030860 con 0x7fd5180d2960 2024-03-06T07:35:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd517e00640 1 -- 172.21.15.32:0/1067191445 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508030c30 con 0x7fd5180d2960 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 -- 172.21.15.32:0/1067191445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 msgr2=0x7fd5180d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 --2- 172.21.15.32:0/1067191445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180d05c0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd508009870 tx=0x7fd50802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 -- 172.21.15.32:0/1067191445 shutdown_connections 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 --2- 172.21.15.32:0/1067191445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180d05c0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 -- 172.21.15.32:0/1067191445 >> 172.21.15.32:0/1067191445 conn(0x7fd51808d2a0 msgr2=0x7fd5180d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 -- 172.21.15.32:0/1067191445 shutdown_connections 2024-03-06T07:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.563+0000 7fd51ee00640 1 -- 172.21.15.32:0/1067191445 wait complete. 2024-03-06T07:35:10.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.565+0000 7fd51ee00640 1 Processor -- start 2024-03-06T07:35:10.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.565+0000 7fd51ee00640 1 -- start start 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5180d0b00 0x7fd51815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd51815e070 0x7fd518160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180d2440 con 0x7fd5180d2960 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180d2160 con 0x7fd51815e070 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180d45b0 con 0x7fd5180d0b00 2024-03-06T07:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.566+0000 7fd51d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39710/0 (socket says 172.21.15.32:39710) 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 -- 172.21.15.32:0/138584802 learned_addr learned my addr 172.21.15.32:0/138584802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 -- 172.21.15.32:0/138584802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5180d0b00 msgr2=0x7fd51815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5180d0b00 0x7fd51815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 -- 172.21.15.32:0/138584802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd51815e070 msgr2=0x7fd518160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd51815e070 0x7fd518160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 -- 172.21.15.32:0/138584802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5181f4950 con 0x7fd5180d2960 2024-03-06T07:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd51d000640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180c9dd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fd508009870 tx=0x7fd508004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.567+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508048070 con 0x7fd5180d2960 2024-03-06T07:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.568+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508039d90 con 0x7fd5180d2960 2024-03-06T07:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.568+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508034040 con 0x7fd5180d2960 2024-03-06T07:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.568+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5181f4bc0 con 0x7fd5180d2960 2024-03-06T07:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.568+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5180b0de0 con 0x7fd5180d2960 2024-03-06T07:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.569+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51804f330 con 0x7fd5180d2960 2024-03-06T07:35:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.570+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd508037070 con 0x7fd5180d2960 2024-03-06T07:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.571+0000 7fd515600640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.571+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd5080abe20 con 0x7fd5180d2960 2024-03-06T07:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.571+0000 7fd51da00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.572+0000 7fd51da00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd50c006fd0 tx=0x7fd50c006bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.575+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd50807af50 con 0x7fd5180d2960 2024-03-06T07:35:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.846+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd5180b83e0 con 0x7fd5180d2960 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.847+0000 7fd515600640 1 -- 172.21.15.32:0/138584802 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd50807a850 con 0x7fd5180d2960 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:35:10.849 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:35:10.850 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:35:10.851 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.853 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd4fc06b500 msgr2=0x7fd4fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd50c006fd0 tx=0x7fd50c006bc0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 msgr2=0x7fd5180c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180c9dd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fd508009870 tx=0x7fd508004730 comp rx=0 tx=0).stop 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 shutdown_connections 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd4fc06b500 0x7fd4fc06d9c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd51815e070 0x7fd518160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5180d2960 0x7fd5180c9dd0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 --2- 172.21.15.32:0/138584802 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd5180d0b00 0x7fd51815c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 >> 172.21.15.32:0/138584802 conn(0x7fd51808d2a0 msgr2=0x7fd51815cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 shutdown_connections 2024-03-06T07:35:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:10.850+0000 7fd51ee00640 1 -- 172.21.15.32:0/138584802 wait complete. 2024-03-06T07:35:10.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:11.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.089+0000 7ffb68800640 1 Processor -- start 2024-03-06T07:35:11.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.089+0000 7ffb68800640 1 -- start start 2024-03-06T07:35:11.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb68800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb600af1a0 0x7ffb6015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb68800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600af580 0x7ffb600b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb68800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6015cc40 con 0x7ffb600cdc20 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb68800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6015cf20 con 0x7ffb600af1a0 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb68800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6015d200 con 0x7ffb600af580 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb65e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb600af1a0 0x7ffb6015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb65e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb600af1a0 0x7ffb6015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39712/0 (socket says 172.21.15.32:39712) 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb65e00640 1 -- 172.21.15.32:0/719458142 learned_addr learned my addr 172.21.15.32:0/719458142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.091+0000 7ffb65400640 1 --2- 172.21.15.32:0/719458142 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600af580 0x7ffb600b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.092+0000 7ffb65e00640 1 -- 172.21.15.32:0/719458142 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600af580 msgr2=0x7ffb600b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.092+0000 7ffb65e00640 1 --2- 172.21.15.32:0/719458142 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600af580 0x7ffb600b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.092+0000 7ffb65e00640 1 -- 172.21.15.32:0/719458142 >> v1:172.21.15.39:6789/0 conn(0x7ffb600cdc20 legacy=0x7ffb600b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.092+0000 7ffb65e00640 1 -- 172.21.15.32:0/719458142 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6015d4e0 con 0x7ffb600af1a0 2024-03-06T07:35:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.093+0000 7ffb65e00640 1 --2- 172.21.15.32:0/719458142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb6015c600 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009870 tx=0x7ffb5402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc575b58c502132c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.093+0000 7ffb64a00640 1 -- 172.21.15.32:0/719458142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540317c0 con 0x7ffb600af1a0 2024-03-06T07:35:11.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.093+0000 7ffb64a00640 1 -- 172.21.15.32:0/719458142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb54037070 con 0x7ffb600af1a0 2024-03-06T07:35:11.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.093+0000 7ffb64a00640 1 -- 172.21.15.32:0/719458142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54031b80 con 0x7ffb600af1a0 2024-03-06T07:35:11.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 -- 172.21.15.32:0/719458142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 msgr2=0x7ffb6015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 --2- 172.21.15.32:0/719458142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb6015c600 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009870 tx=0x7ffb5402b860 comp rx=0 tx=0).stop 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 -- 172.21.15.32:0/719458142 shutdown_connections 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 --2- 172.21.15.32:0/719458142 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600af580 0x7ffb600b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 --2- 172.21.15.32:0/719458142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb6015c600 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 -- 172.21.15.32:0/719458142 >> 172.21.15.32:0/719458142 conn(0x7ffb6006afb0 msgr2=0x7ffb6006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 -- 172.21.15.32:0/719458142 shutdown_connections 2024-03-06T07:35:11.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.094+0000 7ffb68800640 1 -- 172.21.15.32:0/719458142 wait complete. 2024-03-06T07:35:11.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.096+0000 7ffb68800640 1 Processor -- start 2024-03-06T07:35:11.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.096+0000 7ffb68800640 1 -- start start 2024-03-06T07:35:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb68800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb60159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb68800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb600af580 0x7ffb60146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb68800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600cdc20 0x7ffb60146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb68800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb600b3660 con 0x7ffb600af1a0 2024-03-06T07:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb68800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb600b34e0 con 0x7ffb600af580 2024-03-06T07:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb68800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb600b37e0 con 0x7ffb600cdc20 2024-03-06T07:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb65e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb60159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb65e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb60159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39720/0 (socket says 172.21.15.32:39720) 2024-03-06T07:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.097+0000 7ffb65e00640 1 -- 172.21.15.32:0/2987798449 learned_addr learned my addr 172.21.15.32:0/2987798449 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65400640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb600af580 0x7ffb60146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb66800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600cdc20 0x7ffb60146750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65e00640 1 -- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600cdc20 msgr2=0x7ffb60146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65e00640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600cdc20 0x7ffb60146750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65e00640 1 -- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb600af580 msgr2=0x7ffb60146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65e00640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb600af580 0x7ffb60146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65e00640 1 -- 172.21.15.32:0/2987798449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6009ab40 con 0x7ffb600af1a0 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65400640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb600af580 0x7ffb60146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb65e00640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb60159300 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ffb5402b8a0 tx=0x7ffb540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.098+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54032650 con 0x7ffb600af1a0 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.099+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb54032810 con 0x7ffb600af1a0 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.099+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54032c00 con 0x7ffb600af1a0 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.099+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6009ae20 con 0x7ffb600af1a0 2024-03-06T07:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.099+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6009b3e0 con 0x7ffb600af1a0 2024-03-06T07:35:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.100+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6004f3a0 con 0x7ffb600af1a0 2024-03-06T07:35:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.101+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb5404c060 con 0x7ffb600af1a0 2024-03-06T07:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.102+0000 7ffb5a000640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.102+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffb540ab9f0 con 0x7ffb600af1a0 2024-03-06T07:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.102+0000 7ffb65400640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.103+0000 7ffb65400640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffb60150750 tx=0x7ffb50008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.107+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb54040070 con 0x7ffb600af1a0 2024-03-06T07:35:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.313+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb600cd6c0 con 0x7ffb600af1a0 2024-03-06T07:35:11.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.316+0000 7ffb5a000640 1 -- 172.21.15.32:0/2987798449 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+12785 (secure 0 0 0) 0x7ffb5407abb0 con 0x7ffb600af1a0 2024-03-06T07:35:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:11.319 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":106,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:10.333314+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.319+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 msgr2=0x7ffb4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.319+0000 7ffb68800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffb60150750 tx=0x7ffb50008040 comp rx=0 tx=0).stop 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.319+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 msgr2=0x7ffb60159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.319+0000 7ffb68800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb60159300 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ffb5402b8a0 tx=0x7ffb540046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 shutdown_connections 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffb4406b4d0 0x7ffb4406d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffb600cdc20 0x7ffb60146750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb600af580 0x7ffb60146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 --2- 172.21.15.32:0/2987798449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb600af1a0 0x7ffb60159300 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 >> 172.21.15.32:0/2987798449 conn(0x7ffb6006afb0 msgr2=0x7ffb6015c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 shutdown_connections 2024-03-06T07:35:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.320+0000 7ffb68800640 1 -- 172.21.15.32:0/2987798449 wait complete. 2024-03-06T07:35:11.339 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:11.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:11.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.536+0000 7f68ba200640 1 Processor -- start 2024-03-06T07:35:11.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.537+0000 7f68ba200640 1 -- start start 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.538+0000 7f68ba200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.538+0000 7f68ba200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68b40b2080 0x7f68b40b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.538+0000 7f68ba200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b406de40 con 0x7f68b4147590 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.538+0000 7f68ba200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b4147970 con 0x7f68b40b2080 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.538+0000 7f68ba200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b31b0 con 0x7f68b406d8d0 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68b40b2080 0x7f68b40b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68b40b2080 0x7f68b40b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39728/0 (socket says 172.21.15.32:39728) 2024-03-06T07:35:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 -- 172.21.15.32:0/2118128248 learned_addr learned my addr 172.21.15.32:0/2118128248 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 -- 172.21.15.32:0/2118128248 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 msgr2=0x7f68b40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b3e00640 1 --2- 172.21.15.32:0/2118128248 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 --2- 172.21.15.32:0/2118128248 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b40b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 -- 172.21.15.32:0/2118128248 >> v1:172.21.15.39:6789/0 conn(0x7f68b4147590 legacy=0x7f68b4071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.539+0000 7f68b9800640 1 -- 172.21.15.32:0/2118128248 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b406e120 con 0x7f68b40b2080 2024-03-06T07:35:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.540+0000 7f68b9800640 1 --2- 172.21.15.32:0/2118128248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b40b3640 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009e50 tx=0x7f68ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3f1ea8a2f11d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68b3400640 1 -- 172.21.15.32:0/2118128248 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac02ca70 con 0x7f68b40b2080 2024-03-06T07:35:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68b3400640 1 -- 172.21.15.32:0/2118128248 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac02cc10 con 0x7f68b40b2080 2024-03-06T07:35:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68b3400640 1 -- 172.21.15.32:0/2118128248 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0343f0 con 0x7f68b40b2080 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 -- 172.21.15.32:0/2118128248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 msgr2=0x7f68b40b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 --2- 172.21.15.32:0/2118128248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b40b3640 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009e50 tx=0x7f68ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 -- 172.21.15.32:0/2118128248 shutdown_connections 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 --2- 172.21.15.32:0/2118128248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b40b3640 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 --2- 172.21.15.32:0/2118128248 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b40b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 -- 172.21.15.32:0/2118128248 >> 172.21.15.32:0/2118128248 conn(0x7f68b406b3a0 msgr2=0x7f68b406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.541+0000 7f68ba200640 1 -- 172.21.15.32:0/2118128248 shutdown_connections 2024-03-06T07:35:11.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.542+0000 7f68ba200640 1 -- 172.21.15.32:0/2118128248 wait complete. 2024-03-06T07:35:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.543+0000 7f68ba200640 1 Processor -- start 2024-03-06T07:35:11.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.543+0000 7f68ba200640 1 -- start start 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.544+0000 7f68ba200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b415d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.544+0000 7f68ba200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b4141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.544+0000 7f68ba200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4147590 0x7f68b41e65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.544+0000 7f68ba200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b3b80 con 0x7f68b40b2080 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.544+0000 7f68ba200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b2b50 con 0x7f68b4147590 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.544+0000 7f68ba200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b3d00 con 0x7f68b406d8d0 2024-03-06T07:35:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b4141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b4141a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39740/0 (socket says 172.21.15.32:39740) 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 -- 172.21.15.32:0/1507462094 learned_addr learned my addr 172.21.15.32:0/1507462094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b8e00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b9800640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4147590 0x7f68b41e65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 -- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 msgr2=0x7f68b415d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b415d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 -- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4147590 msgr2=0x7f68b41e65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4147590 0x7f68b41e65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b3e00640 1 -- 172.21.15.32:0/1507462094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b413e940 con 0x7f68b40b2080 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.545+0000 7f68b9800640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4147590 0x7f68b41e65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.546+0000 7f68b3e00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b4141a30 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c7f0 tx=0x7f68a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.546+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0015070 con 0x7f68b40b2080 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.546+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0004240 con 0x7f68b40b2080 2024-03-06T07:35:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.546+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0007620 con 0x7f68b40b2080 2024-03-06T07:35:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.546+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4140990 con 0x7f68b40b2080 2024-03-06T07:35:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.546+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b4140f50 con 0x7f68b40b2080 2024-03-06T07:35:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.547+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b404f330 con 0x7f68b40b2080 2024-03-06T07:35:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.548+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68a00079d0 con 0x7f68b40b2080 2024-03-06T07:35:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.549+0000 7f68b0c00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f689406b4d0 0x7f689406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.549+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f68a008a3f0 con 0x7f68b40b2080 2024-03-06T07:35:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.549+0000 7f68b8e00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f689406b4d0 0x7f689406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.550+0000 7f68b8e00640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f689406b4d0 0x7f689406d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f68a8006fd0 tx=0x7f68a8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.554+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68a0059490 con 0x7f68b40b2080 2024-03-06T07:35:11.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.750+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68b40b1dd0 con 0x7f68b40b2080 2024-03-06T07:35:11.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.751+0000 7f68b0c00640 1 -- 172.21.15.32:0/1507462094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68a0058d90 con 0x7f68b40b2080 2024-03-06T07:35:11.753 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:35:11.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f689406b4d0 msgr2=0x7f689406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f689406b4d0 0x7f689406d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f68a8006fd0 tx=0x7f68a8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 msgr2=0x7f68b4141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b4141a30 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c7f0 tx=0x7f68a000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 shutdown_connections 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f689406b4d0 0x7f689406d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4147590 0x7f68b41e65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40b2080 0x7f68b4141a30 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 --2- 172.21.15.32:0/1507462094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f68b406d8d0 0x7f68b415d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 >> 172.21.15.32:0/1507462094 conn(0x7f68b406b3a0 msgr2=0x7f68b4143e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 shutdown_connections 2024-03-06T07:35:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.754+0000 7f68ba200640 1 -- 172.21.15.32:0/1507462094 wait complete. 2024-03-06T07:35:11.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:35:11.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.983+0000 7fda61800640 1 Processor -- start 2024-03-06T07:35:11.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.983+0000 7fda61800640 1 -- start start 2024-03-06T07:35:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.984+0000 7fda61800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fda5c1552b0 0x7fda5c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.984+0000 7fda61800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda5c15e810 0x7fda5c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda61800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c0cce90 0x7fda5c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda61800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c15ed10 con 0x7fda5c15e810 2024-03-06T07:35:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda61800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c15eff0 con 0x7fda5c1552b0 2024-03-06T07:35:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda61800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c15f2d0 con 0x7fda5c0cce90 2024-03-06T07:35:11.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda53e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda5c15e810 0x7fda5c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda53e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda5c15e810 0x7fda5c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39748/0 (socket says 172.21.15.32:39748) 2024-03-06T07:35:11.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda53e00640 1 -- 172.21.15.32:0/3349752942 learned_addr learned my addr 172.21.15.32:0/3349752942 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:11.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda5be00640 1 --2- 172.21.15.32:0/3349752942 >> v2:172.21.15.39:3300/0 conn(0x7fda5c1552b0 0x7fda5c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda60e00640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c0cce90 0x7fda5c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda53e00640 1 -- 172.21.15.32:0/3349752942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c0cce90 msgr2=0x7fda5c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.985+0000 7fda53e00640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c0cce90 0x7fda5c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.986+0000 7fda53e00640 1 -- 172.21.15.32:0/3349752942 >> v2:172.21.15.39:3300/0 conn(0x7fda5c1552b0 msgr2=0x7fda5c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.986+0000 7fda53e00640 1 --2- 172.21.15.32:0/3349752942 >> v2:172.21.15.39:3300/0 conn(0x7fda5c1552b0 0x7fda5c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.986+0000 7fda53e00640 1 -- 172.21.15.32:0/3349752942 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c15f5b0 con 0x7fda5c15e810 2024-03-06T07:35:11.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.986+0000 7fda60e00640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c0cce90 0x7fda5c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.987+0000 7fda53e00640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c15e810 0x7fda5c0cc950 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0cd540 tx=0x7fda4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b00ed8e880cfc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.987+0000 7fda5b400640 1 -- 172.21.15.32:0/3349752942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4402e070 con 0x7fda5c15e810 2024-03-06T07:35:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.987+0000 7fda5b400640 1 -- 172.21.15.32:0/3349752942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda44037070 con 0x7fda5c15e810 2024-03-06T07:35:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.987+0000 7fda5b400640 1 -- 172.21.15.32:0/3349752942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44031a00 con 0x7fda5c15e810 2024-03-06T07:35:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 -- 172.21.15.32:0/3349752942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c15e810 msgr2=0x7fda5c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c15e810 0x7fda5c0cc950 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0cd540 tx=0x7fda4402b880 comp rx=0 tx=0).stop 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 -- 172.21.15.32:0/3349752942 shutdown_connections 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c0cce90 0x7fda5c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 --2- 172.21.15.32:0/3349752942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c15e810 0x7fda5c0cc950 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 --2- 172.21.15.32:0/3349752942 >> v2:172.21.15.39:3300/0 conn(0x7fda5c1552b0 0x7fda5c15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 -- 172.21.15.32:0/3349752942 >> 172.21.15.32:0/3349752942 conn(0x7fda5c06b3d0 msgr2=0x7fda5c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 -- 172.21.15.32:0/3349752942 shutdown_connections 2024-03-06T07:35:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.988+0000 7fda61800640 1 -- 172.21.15.32:0/3349752942 wait complete. 2024-03-06T07:35:11.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.990+0000 7fda61800640 1 Processor -- start 2024-03-06T07:35:11.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.990+0000 7fda61800640 1 -- start start 2024-03-06T07:35:11.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.991+0000 7fda61800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 0x7fda5c0b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda61800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c1552b0 0x7fda5c149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda61800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda5c15e810 0x7fda5c146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda61800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c06dd30 con 0x7fda5c0cce90 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda61800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0b3760 con 0x7fda5c15e810 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda61800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c06deb0 con 0x7fda5c1552b0 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 0x7fda5c0b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 0x7fda5c0b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39758/0 (socket says 172.21.15.32:39758) 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 -- 172.21.15.32:0/3703192094 learned_addr learned my addr 172.21.15.32:0/3703192094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 -- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c1552b0 msgr2=0x7fda5c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda53e00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c1552b0 0x7fda5c149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda60e00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda5c15e810 0x7fda5c146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c1552b0 0x7fda5c149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 -- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda5c15e810 msgr2=0x7fda5c146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda5c15e810 0x7fda5c146760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda5be00640 1 -- 172.21.15.32:0/3703192094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c0af730 con 0x7fda5c0cce90 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.992+0000 7fda53e00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c1552b0 0x7fda5c149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.993+0000 7fda5be00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 0x7fda5c0b0ac0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00d5f0 tx=0x7fda4c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.993+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0125f0 con 0x7fda5c0cce90 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.993+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c019070 con 0x7fda5c0cce90 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.993+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0129d0 con 0x7fda5c0cce90 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.993+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c0afa10 con 0x7fda5c0cce90 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.993+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c0affd0 con 0x7fda5c0cce90 2024-03-06T07:35:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.994+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c159630 con 0x7fda5c0cce90 2024-03-06T07:35:11.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.995+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda4c012b70 con 0x7fda5c0cce90 2024-03-06T07:35:11.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.996+0000 7fda58c00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.996+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda4c08a360 con 0x7fda5c0cce90 2024-03-06T07:35:11.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.997+0000 7fda53e00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:11.997+0000 7fda53e00640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fda44036000 tx=0x7fda44008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.001+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda5c159630 con 0x7fda5c0cce90 2024-03-06T07:35:12.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.194+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fda5c04f330 con 0x7fda5c0cce90 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.195+0000 7fda58c00640 1 -- 172.21.15.32:0/3703192094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+65 (secure 0 0 0) 0x7fda4c059490 con 0x7fda5c0cce90 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:12.198 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:35:12.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 msgr2=0x7fda3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fda44036000 tx=0x7fda44008030 comp rx=0 tx=0).stop 2024-03-06T07:35:12.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 msgr2=0x7fda5c0b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 0x7fda5c0b0ac0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00d5f0 tx=0x7fda4c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 shutdown_connections 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda5c15e810 0x7fda5c146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fda5c1552b0 0x7fda5c149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 --2- 172.21.15.32:0/3703192094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5c0cce90 0x7fda5c0b0ac0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 >> 172.21.15.32:0/3703192094 conn(0x7fda5c06b3d0 msgr2=0x7fda5c0ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 shutdown_connections 2024-03-06T07:35:12.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.198+0000 7fda61800640 1 -- 172.21.15.32:0/3703192094 wait complete. 2024-03-06T07:35:12.218 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:35:12.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:35:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.441+0000 7f834aa00640 1 Processor -- start 2024-03-06T07:35:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.441+0000 7f834aa00640 1 -- start start 2024-03-06T07:35:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.443+0000 7f834aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83440b4af0 0x7f83440cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.443+0000 7f834aa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83440b2c20 con 0x7f834415de00 2024-03-06T07:35:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.443+0000 7f834aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83440b3ac0 con 0x7f834415e1e0 2024-03-06T07:35:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.443+0000 7f834aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b2f20 con 0x7f83440b4af0 2024-03-06T07:35:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.443+0000 7f8343e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f834415de00 0x7f8344072910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35850/0 (socket says 172.21.15.32:35850) 2024-03-06T07:35:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.443+0000 7f8343e00640 1 -- 172.21.15.32:0/160705824 learned_addr learned my addr 172.21.15.32:0/160705824 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:12.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8348a00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83440b4af0 0x7f83440cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902994221 0 0) 0x7f83440b2c20 con 0x7f834415de00 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8328003520 con 0x7f834415de00 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2011397359 0 0) 0x7f8328003520 con 0x7f834415de00 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83440b4af0 msgr2=0x7f83440cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f83440b4af0 0x7f83440cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 >> v1:172.21.15.39:6789/0 conn(0x7f834415e1e0 legacy=0x7f83440b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834406cda0 con 0x7f834415de00 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f8334001670 con 0x7f834415de00 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 >> v1:172.21.15.32:6789/0 conn(0x7f834415de00 legacy=0x7f8344072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83280039e0 0x7f8328003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8328004800 0x7f8328004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280051d0 0x7f832800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440b2f20 con 0x7f83280039e0 2024-03-06T07:35:12.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832800c090 con 0x7f83280051d0 2024-03-06T07:35:12.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.444+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832800c260 con 0x7f8328004800 2024-03-06T07:35:12.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f8343e00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83280039e0 0x7f8328003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f833be00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8328004800 0x7f8328004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f8343e00640 1 -- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8328004800 msgr2=0x7f8328004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f8343e00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8328004800 0x7f8328004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f8343e00640 1 -- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280051d0 msgr2=0x7f832800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f8343e00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280051d0 0x7f832800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f8343e00640 1 -- 172.21.15.32:0/160705824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328003520 con 0x7f83280039e0 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.445+0000 7f833be00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8328004800 0x7f8328004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.446+0000 7f8343e00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83280039e0 0x7f8328003e00 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f833400c110 tx=0x7f8328005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafddef6767e6a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.446+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334007c40 con 0x7f83280039e0 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.446+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8334030070 con 0x7f83280039e0 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.446+0000 7f8343400640 1 -- 172.21.15.32:0/160705824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334005d30 con 0x7f83280039e0 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f834aa00640 1 -- 172.21.15.32:0/160705824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83280039e0 msgr2=0x7f8328003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f834aa00640 1 --2- 172.21.15.32:0/160705824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83280039e0 0x7f8328003e00 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f833400c110 tx=0x7f8328005830 comp rx=0 tx=0).stop 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f8348a00640 1 -- 172.21.15.32:0/160705824 reap_dead start 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f834aa00640 1 -- 172.21.15.32:0/160705824 shutdown_connections 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f834aa00640 1 -- 172.21.15.32:0/160705824 >> 172.21.15.32:0/160705824 conn(0x7f834406afb0 msgr2=0x7f834406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f834aa00640 1 -- 172.21.15.32:0/160705824 shutdown_connections 2024-03-06T07:35:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.447+0000 7f834aa00640 1 -- 172.21.15.32:0/160705824 wait complete. 2024-03-06T07:35:12.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.449+0000 7f834aa00640 1 Processor -- start 2024-03-06T07:35:12.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.449+0000 7f834aa00640 1 -- start start 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f834aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280039e0 0x7f83441f7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f834aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8344148ae0 0x7f834414b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f834aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 0x7f8344160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f834aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834406d4e0 con 0x7f834414b8f0 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f834aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834406d360 con 0x7f83280039e0 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f834aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83440cf810 con 0x7f8344148ae0 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f8348a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 0x7f8344160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f8348a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 0x7f8344160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39788/0 (socket says 172.21.15.32:39788) 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.450+0000 7f8348a00640 1 -- 172.21.15.32:0/2505110507 learned_addr learned my addr 172.21.15.32:0/2505110507 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f833be00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8344148ae0 0x7f834414b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8343e00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280039e0 0x7f83441f7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8348a00640 1 -- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8344148ae0 msgr2=0x7f834414b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8348a00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8344148ae0 0x7f834414b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8348a00640 1 -- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280039e0 msgr2=0x7f83441f7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8348a00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280039e0 0x7f83441f7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8348a00640 1 -- 172.21.15.32:0/2505110507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834414be70 con 0x7f834414b8f0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8348a00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 0x7f8344160250 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f834409e090 tx=0x7f833c007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c019070 con 0x7f834414b8f0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c00ec50 con 0x7f834414b8f0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.451+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c003990 con 0x7f834414b8f0 2024-03-06T07:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.452+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834414c150 con 0x7f834414b8f0 2024-03-06T07:35:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.452+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8344148f60 con 0x7f834414b8f0 2024-03-06T07:35:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.453+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83441f7060 con 0x7f834414b8f0 2024-03-06T07:35:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.454+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f833c01c070 con 0x7f834414b8f0 2024-03-06T07:35:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.455+0000 7f8340c00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.455+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f833c089650 con 0x7f834414b8f0 2024-03-06T07:35:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.455+0000 7f8343e00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.456+0000 7f8343e00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f832006b500 0x7f832006d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8334034000 tx=0x7f8334009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.459+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f833c058810 con 0x7f834414b8f0 2024-03-06T07:35:12.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:12.658+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8344068010 con 0x7f834414b8f0 2024-03-06T07:35:13.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.343+0000 7f8340c00640 1 -- 172.21.15.32:0/2505110507 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v107) v1 ==== 138+0+0 (secure 0 0 0) 0x7f833c058110 con 0x7f834414b8f0 2024-03-06T07:35:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.347+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f832006b500 msgr2=0x7f832006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.347+0000 7f834aa00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f832006b500 0x7f832006d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8334034000 tx=0x7f8334009030 comp rx=0 tx=0).stop 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.347+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 msgr2=0x7f8344160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.347+0000 7f834aa00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 0x7f8344160250 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f834409e090 tx=0x7f833c007f40 comp rx=0 tx=0).stop 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.348+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 shutdown_connections 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.348+0000 7f834aa00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.348+0000 7f834aa00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f834414b8f0 0x7f8344160250 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.348+0000 7f834aa00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8344148ae0 0x7f834414b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.348+0000 7f834aa00640 1 --2- 172.21.15.32:0/2505110507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83280039e0 0x7f83441f7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.348+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 >> 172.21.15.32:0/2505110507 conn(0x7f834406afb0 msgr2=0x7f83440727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.349+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 shutdown_connections 2024-03-06T07:35:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.349+0000 7f834aa00640 1 -- 172.21.15.32:0/2505110507 wait complete. 2024-03-06T07:35:13.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:35:13.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.583+0000 7ff98da00640 1 Processor -- start 2024-03-06T07:35:13.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.583+0000 7ff98da00640 1 -- start start 2024-03-06T07:35:13.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.585+0000 7ff98da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff98815e1c0 0x7ff9880cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.585+0000 7ff98da00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff9880d22a0 0x7ff9880b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.585+0000 7ff98da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880ce800 con 0x7ff9880d0650 2024-03-06T07:35:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.585+0000 7ff98da00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9880d2cb0 con 0x7ff9880d22a0 2024-03-06T07:35:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.585+0000 7ff98da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880ceaa0 con 0x7ff98815e1c0 2024-03-06T07:35:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.586+0000 7ff987400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff98815e1c0 0x7ff9880cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.586+0000 7ff98d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff9880d22a0 0x7ff9880b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.586+0000 7ff98d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff9880d22a0 0x7ff9880b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:42716/0 (socket says 172.21.15.32:42716) 2024-03-06T07:35:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.586+0000 7ff98d000640 1 -- 172.21.15.32:0/572932122 learned_addr learned my addr 172.21.15.32:0/572932122 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:13.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.587+0000 7ff98d000640 1 -- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff98815e1c0 msgr2=0x7ff9880cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.587+0000 7ff98d000640 1 --2- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff98815e1c0 0x7ff9880cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.587+0000 7ff98d000640 1 -- 172.21.15.32:0/572932122 >> v1:172.21.15.39:6789/0 conn(0x7ff9880d0650 legacy=0x7ff9880d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.587+0000 7ff98d000640 1 -- 172.21.15.32:0/572932122 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880ced80 con 0x7ff9880d22a0 2024-03-06T07:35:13.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.588+0000 7ff98d000640 1 --2- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d22a0 0x7ff9880b3530 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff980009e50 tx=0x7ff98002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b27d1139981c46b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.588+0000 7ff986a00640 1 -- 172.21.15.32:0/572932122 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98002cb00 con 0x7ff9880d22a0 2024-03-06T07:35:13.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.589+0000 7ff986a00640 1 -- 172.21.15.32:0/572932122 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98002cca0 con 0x7ff9880d22a0 2024-03-06T07:35:13.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.589+0000 7ff986a00640 1 -- 172.21.15.32:0/572932122 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980034610 con 0x7ff9880d22a0 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.589+0000 7ff98da00640 1 -- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d22a0 msgr2=0x7ff9880b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.589+0000 7ff98da00640 1 --2- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d22a0 0x7ff9880b3530 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff980009e50 tx=0x7ff98002bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.590+0000 7ff98da00640 1 -- 172.21.15.32:0/572932122 shutdown_connections 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.590+0000 7ff98da00640 1 --2- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d22a0 0x7ff9880b3530 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.590+0000 7ff98da00640 1 --2- 172.21.15.32:0/572932122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff98815e1c0 0x7ff9880cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.590+0000 7ff98da00640 1 -- 172.21.15.32:0/572932122 >> 172.21.15.32:0/572932122 conn(0x7ff98808d2a0 msgr2=0x7ff9880d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.590+0000 7ff98da00640 1 -- 172.21.15.32:0/572932122 shutdown_connections 2024-03-06T07:35:13.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.590+0000 7ff98da00640 1 -- 172.21.15.32:0/572932122 wait complete. 2024-03-06T07:35:13.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.591+0000 7ff98da00640 1 Processor -- start 2024-03-06T07:35:13.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.592+0000 7ff98da00640 1 -- start start 2024-03-06T07:35:13.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.592+0000 7ff98da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d0650 0x7ff9880b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.592+0000 7ff98da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff9880d22a0 0x7ff98814b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 0x7ff98814b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880d4a30 con 0x7ff98815e1c0 2024-03-06T07:35:13.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880cf340 con 0x7ff9880d0650 2024-03-06T07:35:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880d4bb0 con 0x7ff9880d22a0 2024-03-06T07:35:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 0x7ff98814b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 0x7ff98814b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39804/0 (socket says 172.21.15.32:39804) 2024-03-06T07:35:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff98d000640 1 -- 172.21.15.32:0/1802267460 learned_addr learned my addr 172.21.15.32:0/1802267460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff987e00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d0650 0x7ff9880b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.593+0000 7ff987400640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff9880d22a0 0x7ff98814b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98d000640 1 -- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff9880d22a0 msgr2=0x7ff98814b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98d000640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff9880d22a0 0x7ff98814b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98d000640 1 -- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d0650 msgr2=0x7ff9880b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98d000640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d0650 0x7ff9880b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98d000640 1 -- 172.21.15.32:0/1802267460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880b42f0 con 0x7ff98815e1c0 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff987e00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d0650 0x7ff9880b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98d000640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 0x7ff98814b5f0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff980009e50 tx=0x7ff980004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980043020 con 0x7ff98815e1c0 2024-03-06T07:35:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98002ce50 con 0x7ff98815e1c0 2024-03-06T07:35:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98003e690 con 0x7ff98815e1c0 2024-03-06T07:35:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9880b45d0 con 0x7ff98815e1c0 2024-03-06T07:35:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.594+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9880b4b90 con 0x7ff98815e1c0 2024-03-06T07:35:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.596+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98804f330 con 0x7ff98815e1c0 2024-03-06T07:35:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.597+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff980046020 con 0x7ff98815e1c0 2024-03-06T07:35:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.598+0000 7ff97fe00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff96806b4d0 0x7ff96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.598+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff9800ac520 con 0x7ff98815e1c0 2024-03-06T07:35:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.598+0000 7ff987e00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff96806b4d0 0x7ff96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.599+0000 7ff987e00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff96806b4d0 0x7ff96806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff978001660 tx=0x7ff978009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.603+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff98004d030 con 0x7ff98815e1c0 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:13.806+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff98808d810 con 0x7ff98815e1c0 2024-03-06T07:35:14.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.349+0000 7ff97fe00640 1 -- 172.21.15.32:0/1802267460 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v108) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff98007b570 con 0x7ff98815e1c0 2024-03-06T07:35:14.352 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.352+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff96806b4d0 msgr2=0x7ff96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.352+0000 7ff98da00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff96806b4d0 0x7ff96806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff978001660 tx=0x7ff978009040 comp rx=0 tx=0).stop 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.352+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 msgr2=0x7ff98814b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.352+0000 7ff98da00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 0x7ff98814b5f0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff980009e50 tx=0x7ff980004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 shutdown_connections 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff96806b4d0 0x7ff96806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98815e1c0 0x7ff98814b5f0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff9880d22a0 0x7ff98814b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 --2- 172.21.15.32:0/1802267460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9880d0650 0x7ff9880b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 >> 172.21.15.32:0/1802267460 conn(0x7ff98808d2a0 msgr2=0x7ff98815cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 shutdown_connections 2024-03-06T07:35:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.353+0000 7ff98da00640 1 -- 172.21.15.32:0/1802267460 wait complete. 2024-03-06T07:35:14.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:35:14.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.582+0000 7f112ac00640 1 Processor -- start 2024-03-06T07:35:14.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.582+0000 7f112ac00640 1 -- start start 2024-03-06T07:35:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.584+0000 7f112ac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f112415c3f0 0x7f112415c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.584+0000 7f112ac00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11240af250 con 0x7f112415c010 2024-03-06T07:35:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.584+0000 7f112ac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112415cf50 con 0x7f11240afcf0 2024-03-06T07:35:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.584+0000 7f112ac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f112415d230 con 0x7f112415c3f0 2024-03-06T07:35:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.585+0000 7f1123e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f112415c010 0x7f112406ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35860/0 (socket says 172.21.15.32:35860) 2024-03-06T07:35:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.585+0000 7f1123e00640 1 -- 172.21.15.32:0/1194093249 learned_addr learned my addr 172.21.15.32:0/1194093249 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.585+0000 7f1123400640 1 --2- 172.21.15.32:0/1194093249 >> v2:172.21.15.39:3300/0 conn(0x7f112415c3f0 0x7f112415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.585+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647669584 0 0) 0x7f11240af250 con 0x7f112415c010 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.585+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1108003520 con 0x7f112415c010 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2970445472 0 0) 0x7f1108003520 con 0x7f112415c010 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 >> v2:172.21.15.39:3300/0 conn(0x7f112415c3f0 msgr2=0x7f112415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 --2- 172.21.15.32:0/1194093249 >> v2:172.21.15.39:3300/0 conn(0x7f112415c3f0 0x7f112415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 >> v1:172.21.15.39:6789/0 conn(0x7f11240afcf0 legacy=0x7f11240b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11240af4f0 con 0x7f112415c010 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f1114001670 con 0x7f112415c010 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 >> v1:172.21.15.32:6789/0 conn(0x7f112415c010 legacy=0x7f112406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080039b0 0x7f1108003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1108004800 0x7f1108004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11080051c0 0x7f110800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112415d230 con 0x7f11080051c0 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110800c060 con 0x7f11080039b0 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110800c260 con 0x7f1108004800 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.586+0000 7f1128c00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11080051c0 0x7f110800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1123e00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080039b0 0x7f1108003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1123400640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1108004800 0x7f1108004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1128c00640 1 -- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1108004800 msgr2=0x7f1108004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1128c00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1108004800 0x7f1108004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1128c00640 1 -- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080039b0 msgr2=0x7f1108003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1128c00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080039b0 0x7f1108003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.587+0000 7f1128c00640 1 -- 172.21.15.32:0/1194093249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1108003520 con 0x7f11080051c0 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.588+0000 7f1128c00640 1 -- 172.21.15.32:0/1194093249 reap_dead start 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.588+0000 7f1128c00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11080051c0 0x7f110800b9c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f111c001e50 tx=0x7f1108005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a364459f8a9f7365 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.588+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c001a70 con 0x7f11080051c0 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.588+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111c003bb0 con 0x7f11080051c0 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.588+0000 7f1122a00640 1 -- 172.21.15.32:0/1194093249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c0050d0 con 0x7f11080051c0 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 -- 172.21.15.32:0/1194093249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11080051c0 msgr2=0x7f110800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11080051c0 0x7f110800b9c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f111c001e50 tx=0x7f1108005830 comp rx=0 tx=0).stop 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 -- 172.21.15.32:0/1194093249 shutdown_connections 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 --2- 172.21.15.32:0/1194093249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11080051c0 0x7f110800b9c0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 -- 172.21.15.32:0/1194093249 >> 172.21.15.32:0/1194093249 conn(0x7f11240732d0 msgr2=0x7f112409aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:14.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 -- 172.21.15.32:0/1194093249 shutdown_connections 2024-03-06T07:35:14.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.589+0000 7f112ac00640 1 -- 172.21.15.32:0/1194093249 wait complete. 2024-03-06T07:35:14.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.591+0000 7f112ac00640 1 Processor -- start 2024-03-06T07:35:14.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.591+0000 7f112ac00640 1 -- start start 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f112ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080051c0 0x7f11241465b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f112ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 0x7f11241529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f112ac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11240b3930 0x7f11240b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f112ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112415c870 con 0x7f1124152570 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f112ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11240d1ca0 con 0x7f11080051c0 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f112ac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112415c9f0 con 0x7f11240b3930 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f1123400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 0x7f11241529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f1123400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 0x7f11241529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39816/0 (socket says 172.21.15.32:39816) 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f1123400640 1 -- 172.21.15.32:0/1782197749 learned_addr learned my addr 172.21.15.32:0/1782197749 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:14.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.592+0000 7f1123e00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080051c0 0x7f11241465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1128c00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11240b3930 0x7f11240b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123400640 1 -- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11240b3930 msgr2=0x7f11240b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123400640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11240b3930 0x7f11240b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123400640 1 -- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080051c0 msgr2=0x7f11241465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123400640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080051c0 0x7f11241465b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123400640 1 -- 172.21.15.32:0/1782197749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1124152ff0 con 0x7f1124152570 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123e00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080051c0 0x7f11241465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f1123400640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 0x7f11241529f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f11100115a0 tx=0x7f1110011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1110015930 con 0x7f1124152570 2024-03-06T07:35:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1110015ad0 con 0x7f1124152570 2024-03-06T07:35:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11241532d0 con 0x7f1124152570 2024-03-06T07:35:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111000f480 con 0x7f1124152570 2024-03-06T07:35:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.593+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1124155f40 con 0x7f1124152570 2024-03-06T07:35:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.595+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112404f330 con 0x7f1124152570 2024-03-06T07:35:14.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.596+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f111000f620 con 0x7f1124152570 2024-03-06T07:35:14.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.597+0000 7f111be00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f110406b4d0 0x7f110406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.597+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f111008ab20 con 0x7f1124152570 2024-03-06T07:35:14.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.597+0000 7f1123e00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f110406b4d0 0x7f110406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.598+0000 7f1123e00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f110406b4d0 0x7f110406d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f11140058b0 tx=0x7f1114000ed0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.602+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1110019030 con 0x7f1124152570 2024-03-06T07:35:14.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:14.854+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f112415abd0 con 0x7f1124152570 2024-03-06T07:35:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.403+0000 7f111be00640 1 -- 172.21.15.32:0/1782197749 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v72) v1 ==== 174+0+0 (secure 0 0 0) 0x7f11100599f0 con 0x7f1124152570 2024-03-06T07:35:15.409 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.412+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f110406b4d0 msgr2=0x7f110406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.412+0000 7f112ac00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f110406b4d0 0x7f110406d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f11140058b0 tx=0x7f1114000ed0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.412+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 msgr2=0x7f11241529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.412+0000 7f112ac00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 0x7f11241529f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f11100115a0 tx=0x7f1110011a60 comp rx=0 tx=0).stop 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.415+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 shutdown_connections 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.415+0000 7f112ac00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f110406b4d0 0x7f110406d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.415+0000 7f112ac00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f11240b3930 0x7f11240b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.415+0000 7f112ac00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1124152570 0x7f11241529f0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.415+0000 7f112ac00640 1 --2- 172.21.15.32:0/1782197749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11080051c0 0x7f11241465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.415+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 >> 172.21.15.32:0/1782197749 conn(0x7f11240732d0 msgr2=0x7f112415e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.416+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 shutdown_connections 2024-03-06T07:35:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.417+0000 7f112ac00640 1 -- 172.21.15.32:0/1782197749 wait complete. 2024-03-06T07:35:15.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.633+0000 7f30cc000640 1 Processor -- start 2024-03-06T07:35:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.633+0000 7f30cc000640 1 -- start start 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30cc000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c4161670 0x7f30c4070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30cc000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f30c406e240 0x7f30c40711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30cc000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30c406cfe0 0x7f30c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30cc000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c40730d0 con 0x7f30c406cfe0 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30cc000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c40ce9a0 con 0x7f30c406e240 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30cc000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40cec80 con 0x7f30c4161670 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30ca000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30c406cfe0 0x7f30c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30ca000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30c406cfe0 0x7f30c406d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39826/0 (socket says 172.21.15.32:39826) 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.635+0000 7f30ca000640 1 -- 172.21.15.32:0/21938214 learned_addr learned my addr 172.21.15.32:0/21938214 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30c9600640 1 --2- 172.21.15.32:0/21938214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c4161670 0x7f30c4070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30c8c00640 1 --2- 172.21.15.32:0/21938214 >> v2:172.21.15.39:3300/0 conn(0x7f30c406e240 0x7f30c40711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30ca000640 1 -- 172.21.15.32:0/21938214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c4161670 msgr2=0x7f30c4070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30ca000640 1 --2- 172.21.15.32:0/21938214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c4161670 0x7f30c4070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30ca000640 1 -- 172.21.15.32:0/21938214 >> v2:172.21.15.39:3300/0 conn(0x7f30c406e240 msgr2=0x7f30c40711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30ca000640 1 --2- 172.21.15.32:0/21938214 >> v2:172.21.15.39:3300/0 conn(0x7f30c406e240 0x7f30c40711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.636+0000 7f30ca000640 1 -- 172.21.15.32:0/21938214 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40cef60 con 0x7f30c406cfe0 2024-03-06T07:35:15.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.637+0000 7f30ca000640 1 --2- 172.21.15.32:0/21938214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c406cfe0 0x7f30c406d420 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f30c40748c0 tx=0x7f30bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b3ca4cfe813870 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.637+0000 7f30c3e00640 1 -- 172.21.15.32:0/21938214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc02ca70 con 0x7f30c406cfe0 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30c3e00640 1 -- 172.21.15.32:0/21938214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc02cc10 con 0x7f30c406cfe0 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30c3e00640 1 -- 172.21.15.32:0/21938214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0343f0 con 0x7f30c406cfe0 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 -- 172.21.15.32:0/21938214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c406cfe0 msgr2=0x7f30c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 --2- 172.21.15.32:0/21938214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c406cfe0 0x7f30c406d420 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f30c40748c0 tx=0x7f30bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 -- 172.21.15.32:0/21938214 shutdown_connections 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 --2- 172.21.15.32:0/21938214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c406cfe0 0x7f30c406d420 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 --2- 172.21.15.32:0/21938214 >> v2:172.21.15.39:3300/0 conn(0x7f30c406e240 0x7f30c40711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 --2- 172.21.15.32:0/21938214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c4161670 0x7f30c4070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 -- 172.21.15.32:0/21938214 >> 172.21.15.32:0/21938214 conn(0x7f30c406afb0 msgr2=0x7f30c406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 -- 172.21.15.32:0/21938214 shutdown_connections 2024-03-06T07:35:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.638+0000 7f30cc000640 1 -- 172.21.15.32:0/21938214 wait complete. 2024-03-06T07:35:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.640+0000 7f30cc000640 1 Processor -- start 2024-03-06T07:35:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.640+0000 7f30cc000640 1 -- start start 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.641+0000 7f30cc000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30c406cfe0 0x7f30c414d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.641+0000 7f30cc000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c406e240 0x7f30c40cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.641+0000 7f30cc000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 0x7f30c40cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.641+0000 7f30cc000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40ce600 con 0x7f30c4161670 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.641+0000 7f30cc000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40cf520 con 0x7f30c406cfe0 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.641+0000 7f30cc000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40ce780 con 0x7f30c406e240 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 0x7f30c40cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 0x7f30c40cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39840/0 (socket says 172.21.15.32:39840) 2024-03-06T07:35:15.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 -- 172.21.15.32:0/982271246 learned_addr learned my addr 172.21.15.32:0/982271246 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:15.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30c8c00640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c406e240 0x7f30c40cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 -- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c406e240 msgr2=0x7f30c40cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30c9600640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30c406cfe0 0x7f30c414d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c406e240 0x7f30c40cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 -- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30c406cfe0 msgr2=0x7f30c414d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30c406cfe0 0x7f30c414d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 -- 172.21.15.32:0/982271246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c409bfa0 con 0x7f30c4161670 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.642+0000 7f30ca000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 0x7f30c40cba30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009f40 tx=0x7f30bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.643+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc043020 con 0x7f30c4161670 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.643+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc03e450 con 0x7f30c4161670 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.643+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc009510 con 0x7f30c4161670 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.643+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c409b700 con 0x7f30c4161670 2024-03-06T07:35:15.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.643+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c409bcc0 con 0x7f30c4161670 2024-03-06T07:35:15.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.644+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c4069510 con 0x7f30c4161670 2024-03-06T07:35:15.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.645+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30bc046020 con 0x7f30c4161670 2024-03-06T07:35:15.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.646+0000 7f30c1600640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f30a406b4d0 0x7f30a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.646+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30bc0acab0 con 0x7f30c4161670 2024-03-06T07:35:15.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.646+0000 7f30c9600640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f30a406b4d0 0x7f30a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.647+0000 7f30c9600640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f30a406b4d0 0x7f30a406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f30b8006fd0 tx=0x7f30b8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.651+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30bc04d030 con 0x7f30c4161670 2024-03-06T07:35:15.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.843+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30c40b18f0 con 0x7f30c4161670 2024-03-06T07:35:15.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.845+0000 7f30c1600640 1 -- 172.21.15.32:0/982271246 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f30c40b18f0 con 0x7f30c4161670 2024-03-06T07:35:15.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:15.848 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":109,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:14.857129+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.847+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f30a406b4d0 msgr2=0x7f30a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.847+0000 7f30cc000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f30a406b4d0 0x7f30a406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f30b8006fd0 tx=0x7f30b8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.847+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 msgr2=0x7f30c40cba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.847+0000 7f30cc000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 0x7f30c40cba30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009f40 tx=0x7f30bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 shutdown_connections 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f30a406b4d0 0x7f30a406d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30c4161670 0x7f30c40cba30 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30c406e240 0x7f30c40cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 --2- 172.21.15.32:0/982271246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30c406cfe0 0x7f30c414d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 >> 172.21.15.32:0/982271246 conn(0x7f30c406afb0 msgr2=0x7f30c40ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 shutdown_connections 2024-03-06T07:35:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:15.848+0000 7f30cc000640 1 -- 172.21.15.32:0/982271246 wait complete. 2024-03-06T07:35:15.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:16.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.080+0000 7f77fca00640 1 Processor -- start 2024-03-06T07:35:16.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.080+0000 7f77fca00640 1 -- start start 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.081+0000 7f77fca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f77f80cd060 0x7f77f8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.081+0000 7f77fca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 0x7f77f8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.081+0000 7f77fca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f81627f0 con 0x7f77f815efc0 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.081+0000 7f77fca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f815f3a0 con 0x7f77f80cd060 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.081+0000 7f77fca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f806d190 con 0x7f77f8162000 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.082+0000 7f77f6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f77f815efc0 0x7f77f80cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35862/0 (socket says 172.21.15.32:35862) 2024-03-06T07:35:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.082+0000 7f77f6800640 1 -- 172.21.15.32:0/1097020097 learned_addr learned my addr 172.21.15.32:0/1097020097 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.082+0000 7f77f5e00640 1 --2- 172.21.15.32:0/1097020097 >> v2:172.21.15.39:3300/0 conn(0x7f77f80cd060 0x7f77f8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.082+0000 7f77f7200640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 0x7f77f8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.082+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368806926 0 0) 0x7f77f81627f0 con 0x7f77f815efc0 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.082+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d8003520 con 0x7f77f815efc0 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679465089 0 0) 0x7f77d8003520 con 0x7f77f815efc0 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 msgr2=0x7f77f8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 0x7f77f8070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 >> v2:172.21.15.39:3300/0 conn(0x7f77f80cd060 msgr2=0x7f77f8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 --2- 172.21.15.32:0/1097020097 >> v2:172.21.15.39:3300/0 conn(0x7f77f80cd060 0x7f77f8161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f806c860 con 0x7f77f815efc0 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f77e8001670 con 0x7f77f815efc0 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 >> v1:172.21.15.32:6789/0 conn(0x7f77f815efc0 legacy=0x7f77f80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039b0 0x7f77d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 0x7f77d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d8004d40 0x7f77d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f815f3a0 con 0x7f77d80039b0 2024-03-06T07:35:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f806d190 con 0x7f77d8004d40 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d800b9c0 con 0x7f77f8162000 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f6800640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039b0 0x7f77d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.083+0000 7f77f7200640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d8004d40 0x7f77d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.084+0000 7f77f5e00640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 0x7f77d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.084+0000 7f77f6800640 1 -- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 msgr2=0x7f77d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.084+0000 7f77f6800640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f8162000 0x7f77d8004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.084+0000 7f77f6800640 1 -- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d8004d40 msgr2=0x7f77d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.084+0000 7f77f6800640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d8004d40 0x7f77d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.084+0000 7f77f6800640 1 -- 172.21.15.32:0/1097020097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8003520 con 0x7f77d80039b0 2024-03-06T07:35:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.085+0000 7f77f6800640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039b0 0x7f77d8003dd0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f77e8001050 tx=0x7f77d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ab42351462e721 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.085+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8007d70 con 0x7f77d80039b0 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.085+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e8030070 con 0x7f77d80039b0 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.085+0000 7f77f5400640 1 -- 172.21.15.32:0/1097020097 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e802cd80 con 0x7f77d80039b0 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.085+0000 7f77fca00640 1 -- 172.21.15.32:0/1097020097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039b0 msgr2=0x7f77d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.085+0000 7f77fca00640 1 --2- 172.21.15.32:0/1097020097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039b0 0x7f77d8003dd0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f77e8001050 tx=0x7f77d8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.086+0000 7f77f7200640 1 -- 172.21.15.32:0/1097020097 reap_dead start 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.086+0000 7f77fca00640 1 -- 172.21.15.32:0/1097020097 shutdown_connections 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.086+0000 7f77fca00640 1 -- 172.21.15.32:0/1097020097 >> 172.21.15.32:0/1097020097 conn(0x7f77f806b3d0 msgr2=0x7f77f806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.086+0000 7f77fca00640 1 -- 172.21.15.32:0/1097020097 shutdown_connections 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.086+0000 7f77fca00640 1 -- 172.21.15.32:0/1097020097 wait complete. 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.087+0000 7f77fca00640 1 Processor -- start 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.088+0000 7f77fca00640 1 -- start start 2024-03-06T07:35:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77fca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d80039b0 0x7f77f814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 0x7f77f814dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77fca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f81e8120 0x7f77f80b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77fca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f81624b0 con 0x7f77f814e840 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77fca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f806ce20 con 0x7f77d80039b0 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77fca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f806d620 con 0x7f77f81e8120 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 0x7f77f814dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 0x7f77f814dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39856/0 (socket says 172.21.15.32:39856) 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 -- 172.21.15.32:0/2159463566 learned_addr learned my addr 172.21.15.32:0/2159463566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f7200640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f81e8120 0x7f77f80b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 -- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f81e8120 msgr2=0x7f77f80b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f81e8120 0x7f77f80b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 -- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d80039b0 msgr2=0x7f77f814c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d80039b0 0x7f77f814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.089+0000 7f77f5e00640 1 -- 172.21.15.32:0/2159463566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f814c6f0 con 0x7f77f814e840 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.090+0000 7f77f5e00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 0x7f77f814dfe0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f77e0010860 tx=0x7f77e0010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.090+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e000f040 con 0x7f77f814e840 2024-03-06T07:35:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.090+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e000eb60 con 0x7f77f814e840 2024-03-06T07:35:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.090+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e001c400 con 0x7f77f814e840 2024-03-06T07:35:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.090+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f81e86d0 con 0x7f77f814e840 2024-03-06T07:35:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.090+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f8154190 con 0x7f77f814e840 2024-03-06T07:35:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.091+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f804f330 con 0x7f77f814e840 2024-03-06T07:35:16.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.092+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77e001c5a0 con 0x7f77f814e840 2024-03-06T07:35:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.093+0000 7f77eea00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f77d406b500 0x7f77d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.093+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77e008a730 con 0x7f77f814e840 2024-03-06T07:35:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.094+0000 7f77f6800640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f77d406b500 0x7f77d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.094+0000 7f77f6800640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f77d406b500 0x7f77d406d9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f77e8034000 tx=0x7f77e802e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.098+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77e0018030 con 0x7f77f814e840 2024-03-06T07:35:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.308+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77f80af2a0 con 0x7f77f814e840 2024-03-06T07:35:16.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.311+0000 7f77eea00640 1 -- 172.21.15.32:0/2159463566 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f77e0058f50 con 0x7f77f814e840 2024-03-06T07:35:16.314 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:16.314 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":109,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:14.857129+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f77d406b500 msgr2=0x7f77d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f77d406b500 0x7f77d406d9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f77e8034000 tx=0x7f77e802e000 comp rx=0 tx=0).stop 2024-03-06T07:35:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 msgr2=0x7f77f814dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 0x7f77f814dfe0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f77e0010860 tx=0x7f77e0010d20 comp rx=0 tx=0).stop 2024-03-06T07:35:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 shutdown_connections 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f77d406b500 0x7f77d406d9c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f77f81e8120 0x7f77f80b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77f814e840 0x7f77f814dfe0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 --2- 172.21.15.32:0/2159463566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d80039b0 0x7f77f814c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.314+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 >> 172.21.15.32:0/2159463566 conn(0x7f77f806b3d0 msgr2=0x7f77f80af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.315+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 shutdown_connections 2024-03-06T07:35:16.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.315+0000 7f77fca00640 1 -- 172.21.15.32:0/2159463566 wait complete. 2024-03-06T07:35:16.330 INFO:tasks.ceph.mds.d.smithi039.stderr:2024-03-06T07:35:16.328+0000 7f8e2d400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:16.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:35:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.546+0000 7f6106c00640 1 Processor -- start 2024-03-06T07:35:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.546+0000 7f6106c00640 1 -- start start 2024-03-06T07:35:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f6106c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6100072520 0x7f61000cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f6106c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f61000ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f6106c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6100072df0 con 0x7f610015bf20 2024-03-06T07:35:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f6106c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61000cdab0 con 0x7f6100072520 2024-03-06T07:35:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f6106c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61000b4d70 con 0x7f6100071e20 2024-03-06T07:35:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f60ff400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6100072520 0x7f61000cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f60ff400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6100072520 0x7f61000cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39864/0 (socket says 172.21.15.32:39864) 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f60ffe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f610015bf20 0x7f61000acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35878/0 (socket says 172.21.15.32:35878) 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.548+0000 7f60ff400640 1 -- 172.21.15.32:0/470013581 learned_addr learned my addr 172.21.15.32:0/470013581 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.549+0000 7f6104c00640 1 --2- 172.21.15.32:0/470013581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f61000ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.549+0000 7f60ff400640 1 -- 172.21.15.32:0/470013581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 msgr2=0x7f61000ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.549+0000 7f60ff400640 1 --2- 172.21.15.32:0/470013581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f61000ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.549+0000 7f60ff400640 1 -- 172.21.15.32:0/470013581 >> v1:172.21.15.32:6789/0 conn(0x7f610015bf20 legacy=0x7f61000acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.549+0000 7f60ff400640 1 -- 172.21.15.32:0/470013581 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000b5050 con 0x7f6100072520 2024-03-06T07:35:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.549+0000 7f60fea00640 1 -- 172.21.15.32:0/470013581 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190946769 0 0) 0x7f6100072df0 con 0x7f610015bf20 2024-03-06T07:35:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.550+0000 7f60ff400640 1 --2- 172.21.15.32:0/470013581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100072520 0x7f61000cdfb0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f60ec009870 tx=0x7f60ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e9ccd56f672fca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.550+0000 7f60fea00640 1 -- 172.21.15.32:0/470013581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec034070 con 0x7f6100072520 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f60fea00640 1 -- 172.21.15.32:0/470013581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec030840 con 0x7f6100072520 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f60fea00640 1 -- 172.21.15.32:0/470013581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec030c10 con 0x7f6100072520 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 -- 172.21.15.32:0/470013581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100072520 msgr2=0x7f61000cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 --2- 172.21.15.32:0/470013581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100072520 0x7f61000cdfb0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f60ec009870 tx=0x7f60ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 -- 172.21.15.32:0/470013581 shutdown_connections 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 --2- 172.21.15.32:0/470013581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100072520 0x7f61000cdfb0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 --2- 172.21.15.32:0/470013581 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f61000ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 -- 172.21.15.32:0/470013581 >> 172.21.15.32:0/470013581 conn(0x7f610006b3d0 msgr2=0x7f610006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 -- 172.21.15.32:0/470013581 shutdown_connections 2024-03-06T07:35:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.551+0000 7f6106c00640 1 -- 172.21.15.32:0/470013581 wait complete. 2024-03-06T07:35:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.553+0000 7f6106c00640 1 Processor -- start 2024-03-06T07:35:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.553+0000 7f6106c00640 1 -- start start 2024-03-06T07:35:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.554+0000 7f6106c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f6100158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.554+0000 7f6106c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6100072520 0x7f610006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.554+0000 7f6106c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 0x7f610006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.554+0000 7f6106c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61000ad450 con 0x7f610015bf20 2024-03-06T07:35:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.554+0000 7f6106c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61000b4990 con 0x7f6100072520 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.554+0000 7f6106c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610006e1e0 con 0x7f6100071e20 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 0x7f610006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 0x7f610006d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39880/0 (socket says 172.21.15.32:39880) 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 -- 172.21.15.32:0/3332767980 learned_addr learned my addr 172.21.15.32:0/3332767980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f60ffe00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f6100158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f60ff400640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6100072520 0x7f610006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 -- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 msgr2=0x7f6100158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f6100158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 -- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6100072520 msgr2=0x7f610006d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6100072520 0x7f610006d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 -- 172.21.15.32:0/3332767980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000c9180 con 0x7f610015bf20 2024-03-06T07:35:16.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.555+0000 7f6104c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 0x7f610006d8d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009fd0 tx=0x7f60f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.556+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8014420 con 0x7f610015bf20 2024-03-06T07:35:16.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.556+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f8019070 con 0x7f610015bf20 2024-03-06T07:35:16.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.556+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8014800 con 0x7f610015bf20 2024-03-06T07:35:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.556+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6100147910 con 0x7f610015bf20 2024-03-06T07:35:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.556+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61000c9420 con 0x7f610015bf20 2024-03-06T07:35:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.557+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610004f330 con 0x7f610015bf20 2024-03-06T07:35:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.558+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60f80149a0 con 0x7f610015bf20 2024-03-06T07:35:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.559+0000 7f60f7e00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f60e006b520 0x7f60e006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.559+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60f808a9a0 con 0x7f610015bf20 2024-03-06T07:35:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.559+0000 7f60ffe00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f60e006b520 0x7f60e006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.560+0000 7f60ffe00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f60e006b520 0x7f60e006d9e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f60f0000f60 tx=0x7f60f0007b50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.564+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60f8059900 con 0x7f610015bf20 2024-03-06T07:35:16.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-06T07:35:16.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-06T07:35:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:16.809+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f610015f720 con 0x7f610015bf20 2024-03-06T07:35:17.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.769+0000 7f60f7e00640 1 -- 172.21.15.32:0/3332767980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v74) v1 ==== 118+0+0 (secure 0 0 0) 0x7f60f8059200 con 0x7f610015bf20 2024-03-06T07:35:17.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.773+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f60e006b520 msgr2=0x7f60e006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.773+0000 7f6106c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f60e006b520 0x7f60e006d9e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f60f0000f60 tx=0x7f60f0007b50 comp rx=0 tx=0).stop 2024-03-06T07:35:17.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 msgr2=0x7f610006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 0x7f610006d8d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009fd0 tx=0x7f60f800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 shutdown_connections 2024-03-06T07:35:17.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f60e006b520 0x7f60e006d9e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f610015bf20 0x7f610006d8d0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6100072520 0x7f610006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 --2- 172.21.15.32:0/3332767980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6100071e20 0x7f6100158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 >> 172.21.15.32:0/3332767980 conn(0x7f610006b3d0 msgr2=0x7f610015b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 shutdown_connections 2024-03-06T07:35:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:17.774+0000 7f6106c00640 1 -- 172.21.15.32:0/3332767980 wait complete. 2024-03-06T07:35:17.797 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:35:17.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:35:18.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.026+0000 7f40e4c00640 1 Processor -- start 2024-03-06T07:35:18.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.026+0000 7f40e4c00640 1 -- start start 2024-03-06T07:35:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40e4c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f40e0072520 0x7f40e00cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40e4c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e0072df0 con 0x7f40e015bf20 2024-03-06T07:35:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40e4c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e00cdab0 con 0x7f40e0071e20 2024-03-06T07:35:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40e4c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e00b4d70 con 0x7f40e0072520 2024-03-06T07:35:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dde00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f40e015bf20 0x7f40e00acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48758/0 (socket says 172.21.15.32:48758) 2024-03-06T07:35:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dde00640 1 -- 172.21.15.32:0/1204248586 learned_addr learned my addr 172.21.15.32:0/1204248586 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:18.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dd400640 1 --2- 172.21.15.32:0/1204248586 >> v2:172.21.15.39:3300/0 conn(0x7f40e0072520 0x7f40e00cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940877722 0 0) 0x7f40e0072df0 con 0x7f40e015bf20 2024-03-06T07:35:18.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c4003520 con 0x7f40e015bf20 2024-03-06T07:35:18.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77666450 0 0) 0x7f40c4003520 con 0x7f40e015bf20 2024-03-06T07:35:18.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 >> v2:172.21.15.39:3300/0 conn(0x7f40e0072520 msgr2=0x7f40e00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 --2- 172.21.15.32:0/1204248586 >> v2:172.21.15.39:3300/0 conn(0x7f40e0072520 0x7f40e00cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 >> v1:172.21.15.39:6789/0 conn(0x7f40e0071e20 legacy=0x7f40e00ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e00b5050 con 0x7f40e015bf20 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f40d0001670 con 0x7f40e015bf20 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 >> v1:172.21.15.32:6789/0 conn(0x7f40e015bf20 legacy=0x7f40e00acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c4003a00 0x7f40c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40c4004800 0x7f40c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40c40051c0 0x7f40c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00b4d70 con 0x7f40c40051c0 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c400c090 con 0x7f40c4003a00 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.028+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c400c290 con 0x7f40c4004800 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.029+0000 7f40de800640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40c40051c0 0x7f40c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.029+0000 7f40de800640 1 -- 172.21.15.32:0/1204248586 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40c4004800 msgr2=0x7f40c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.029+0000 7f40de800640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40c4004800 0x7f40c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.029+0000 7f40de800640 1 -- 172.21.15.32:0/1204248586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c4003a00 msgr2=0x7f40c4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.029+0000 7f40de800640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c4003a00 0x7f40c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.029+0000 7f40de800640 1 -- 172.21.15.32:0/1204248586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4003520 con 0x7f40c40051c0 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.030+0000 7f40de800640 1 -- 172.21.15.32:0/1204248586 reap_dead start 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.030+0000 7f40de800640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40c40051c0 0x7f40c400b9c0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f40d80044b0 tx=0x7f40c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7471ca29e38dbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.030+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8004080 con 0x7f40c40051c0 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.030+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d8006d70 con 0x7f40c40051c0 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.030+0000 7f40dca00640 1 -- 172.21.15.32:0/1204248586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8007260 con 0x7f40c40051c0 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 -- 172.21.15.32:0/1204248586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40c40051c0 msgr2=0x7f40c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40c40051c0 0x7f40c400b9c0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f40d80044b0 tx=0x7f40c4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 -- 172.21.15.32:0/1204248586 shutdown_connections 2024-03-06T07:35:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 --2- 172.21.15.32:0/1204248586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40c40051c0 0x7f40c400b9c0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 -- 172.21.15.32:0/1204248586 >> 172.21.15.32:0/1204248586 conn(0x7f40e006b3d0 msgr2=0x7f40e006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 -- 172.21.15.32:0/1204248586 shutdown_connections 2024-03-06T07:35:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.031+0000 7f40e4c00640 1 -- 172.21.15.32:0/1204248586 wait complete. 2024-03-06T07:35:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.033+0000 7f40e4c00640 1 Processor -- start 2024-03-06T07:35:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.033+0000 7f40e4c00640 1 -- start start 2024-03-06T07:35:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40e4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c40051c0 0x7f40e014f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40e4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40e01f5c00 0x7f40e014d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40e4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 0x7f40e01f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40e4c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00b4b10 con 0x7f40e014df10 2024-03-06T07:35:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40e4c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00b4990 con 0x7f40c40051c0 2024-03-06T07:35:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40e4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00acd40 con 0x7f40e01f5c00 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.034+0000 7f40de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 0x7f40e01f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 0x7f40e01f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50282/0 (socket says 172.21.15.32:50282) 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 -- 172.21.15.32:0/9822577 learned_addr learned my addr 172.21.15.32:0/9822577 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40dde00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c40051c0 0x7f40e014f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40dd400640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40e01f5c00 0x7f40e014d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 -- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40e01f5c00 msgr2=0x7f40e014d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40e01f5c00 0x7f40e014d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 -- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c40051c0 msgr2=0x7f40e014f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c40051c0 0x7f40e014f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 -- 172.21.15.32:0/9822577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e014e860 con 0x7f40e014df10 2024-03-06T07:35:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.035+0000 7f40de800640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 0x7f40e01f4ff0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f40d80045d0 tx=0x7f40d8003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.036+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8030be0 con 0x7f40e014df10 2024-03-06T07:35:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.036+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d8040070 con 0x7f40e014df10 2024-03-06T07:35:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.036+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8038420 con 0x7f40e014df10 2024-03-06T07:35:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.036+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e014eb40 con 0x7f40e014df10 2024-03-06T07:35:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.036+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e014f100 con 0x7f40e014df10 2024-03-06T07:35:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.037+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e004f330 con 0x7f40e014df10 2024-03-06T07:35:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.038+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40d80385c0 con 0x7f40e014df10 2024-03-06T07:35:18.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.039+0000 7f40d6000640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40c006b520 0x7f40c006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.039+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40d80ab330 con 0x7f40e014df10 2024-03-06T07:35:18.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.039+0000 7f40dde00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40c006b520 0x7f40c006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.040+0000 7f40dde00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40c006b520 0x7f40c006d9e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f40d0005a30 tx=0x7f40d00016e0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.043+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40d8034070 con 0x7f40e014df10 2024-03-06T07:35:18.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.286+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f40e00cf720 con 0x7f40e014df10 2024-03-06T07:35:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.773+0000 7f40d6000640 1 -- 172.21.15.32:0/9822577 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7f40d807a190 con 0x7f40e014df10 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40c006b520 msgr2=0x7f40c006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40c006b520 0x7f40c006d9e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f40d0005a30 tx=0x7f40d00016e0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 msgr2=0x7f40e01f4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 0x7f40e01f4ff0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f40d80045d0 tx=0x7f40d8003a70 comp rx=0 tx=0).stop 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 shutdown_connections 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f40c006b520 0x7f40c006d9e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e014df10 0x7f40e01f4ff0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f40e01f5c00 0x7f40e014d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 --2- 172.21.15.32:0/9822577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40c40051c0 0x7f40e014f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.777+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 >> 172.21.15.32:0/9822577 conn(0x7f40e006b3d0 msgr2=0x7f40e0159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.778+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 shutdown_connections 2024-03-06T07:35:18.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:18.778+0000 7f40e4c00640 1 -- 172.21.15.32:0/9822577 wait complete. 2024-03-06T07:35:18.801 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:35:18.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.025+0000 7f08b3e00640 1 Processor -- start 2024-03-06T07:35:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.025+0000 7f08b3e00640 1 -- start start 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.026+0000 7f08b3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b41552b0 0x7f08b415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.026+0000 7f08b3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f08b40cce90 0x7f08b406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b3e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b415ed10 con 0x7f08b41552b0 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b3e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b415eff0 con 0x7f08b40cce90 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b3e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b415f2d0 con 0x7f08b415e810 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b2a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b41552b0 0x7f08b415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b2a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b41552b0 0x7f08b415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50286/0 (socket says 172.21.15.32:50286) 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b2a00640 1 -- 172.21.15.32:0/684101623 learned_addr learned my addr 172.21.15.32:0/684101623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b2000640 1 --2- 172.21.15.32:0/684101623 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.027+0000 7f08b3400640 1 --2- 172.21.15.32:0/684101623 >> v2:172.21.15.39:3300/0 conn(0x7f08b40cce90 0x7f08b406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.028+0000 7f08b2a00640 1 -- 172.21.15.32:0/684101623 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 msgr2=0x7f08b40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.028+0000 7f08b2a00640 1 --2- 172.21.15.32:0/684101623 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b40cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.028+0000 7f08b2a00640 1 -- 172.21.15.32:0/684101623 >> v2:172.21.15.39:3300/0 conn(0x7f08b40cce90 msgr2=0x7f08b406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.028+0000 7f08b2a00640 1 --2- 172.21.15.32:0/684101623 >> v2:172.21.15.39:3300/0 conn(0x7f08b40cce90 0x7f08b406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.028+0000 7f08b2a00640 1 -- 172.21.15.32:0/684101623 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b415f5b0 con 0x7f08b41552b0 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.028+0000 7f08b3400640 1 --2- 172.21.15.32:0/684101623 >> v2:172.21.15.39:3300/0 conn(0x7f08b40cce90 0x7f08b406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.029+0000 7f08b2a00640 1 --2- 172.21.15.32:0/684101623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b415b1d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f08b406dc10 tx=0x7f08a402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e66553ec7359ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.029+0000 7f08b1600640 1 -- 172.21.15.32:0/684101623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a40317d0 con 0x7f08b41552b0 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.029+0000 7f08b1600640 1 -- 172.21.15.32:0/684101623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a4037070 con 0x7f08b41552b0 2024-03-06T07:35:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.029+0000 7f08b1600640 1 -- 172.21.15.32:0/684101623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a4031b90 con 0x7f08b41552b0 2024-03-06T07:35:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 -- 172.21.15.32:0/684101623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 msgr2=0x7f08b415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 --2- 172.21.15.32:0/684101623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b415b1d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f08b406dc10 tx=0x7f08a402b870 comp rx=0 tx=0).stop 2024-03-06T07:35:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 -- 172.21.15.32:0/684101623 shutdown_connections 2024-03-06T07:35:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 --2- 172.21.15.32:0/684101623 >> v2:172.21.15.39:3300/0 conn(0x7f08b40cce90 0x7f08b406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 --2- 172.21.15.32:0/684101623 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b40cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 --2- 172.21.15.32:0/684101623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b415b1d0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 -- 172.21.15.32:0/684101623 >> 172.21.15.32:0/684101623 conn(0x7f08b406b3d0 msgr2=0x7f08b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 -- 172.21.15.32:0/684101623 shutdown_connections 2024-03-06T07:35:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.030+0000 7f08b3e00640 1 -- 172.21.15.32:0/684101623 wait complete. 2024-03-06T07:35:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.032+0000 7f08b3e00640 1 Processor -- start 2024-03-06T07:35:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.032+0000 7f08b3e00640 1 -- start start 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.033+0000 7f08b3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08b40cce90 0x7f08b415a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b414b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b41666e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b406dd30 con 0x7f08b41552b0 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b3760 con 0x7f08b40cce90 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b3e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b406deb0 con 0x7f08b415e810 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b414b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b414b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50296/0 (socket says 172.21.15.32:50296) 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 -- 172.21.15.32:0/3661474511 learned_addr learned my addr 172.21.15.32:0/3661474511 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2a00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08b40cce90 0x7f08b415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b3400640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b41666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 -- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 msgr2=0x7f08b41666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b41666e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 -- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08b40cce90 msgr2=0x7f08b415a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08b40cce90 0x7f08b415a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2000640 1 -- 172.21.15.32:0/3661474511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4165c40 con 0x7f08b41552b0 2024-03-06T07:35:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.034+0000 7f08b2a00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08b40cce90 0x7f08b415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.035+0000 7f08b2000640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b414b4d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f08a000d5f0 tx=0x7f08a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.035+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a00125f0 con 0x7f08b41552b0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.035+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a0019070 con 0x7f08b41552b0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.035+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b4155c80 con 0x7f08b41552b0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.035+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a00129d0 con 0x7f08b41552b0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.035+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b41558a0 con 0x7f08b41552b0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.036+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b404f330 con 0x7f08b41552b0 2024-03-06T07:35:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.037+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08a0012b70 con 0x7f08b41552b0 2024-03-06T07:35:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.038+0000 7f08ab400640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f089406b4d0 0x7f089406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.038+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08a008a6b0 con 0x7f08b41552b0 2024-03-06T07:35:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.039+0000 7f08b2a00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f089406b4d0 0x7f089406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.039+0000 7f08b2a00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f089406b4d0 0x7f089406d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f08a4036000 tx=0x7f08a4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.043+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08b4155c80 con 0x7f08b41552b0 2024-03-06T07:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.282+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08b40cf4a0 con 0x7f08b41552b0 2024-03-06T07:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.283+0000 7f08ab400640 1 -- 172.21.15.32:0/3661474511 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f08a0059490 con 0x7f08b41552b0 2024-03-06T07:35:19.285 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:19.285 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3906463088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3906463088},{"type":"v1","addr":"172.21.15.39:6834","nonce":3906463088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7647,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/2563322789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2563322789},{"type":"v1","addr":"172.21.15.32:6836","nonce":2563322789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7650,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2154839965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":2154839965},{"type":"v1","addr":"172.21.15.32:6837","nonce":2154839965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:18.287992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7665":{"gid":7665,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/14889266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":14889266},{"type":"v1","addr":"172.21.15.39:6835","nonce":14889266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7665,"qdb_cluster":[7665]},"id":8}]} 2024-03-06T07:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-03-06T07:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f089406b4d0 msgr2=0x7f089406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f089406b4d0 0x7f089406d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f08a4036000 tx=0x7f08a4008030 comp rx=0 tx=0).stop 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 msgr2=0x7f08b414b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b414b4d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f08a000d5f0 tx=0x7f08a000dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 shutdown_connections 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f089406b4d0 0x7f089406d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08b415e810 0x7f08b41666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b41552b0 0x7f08b414b4d0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 --2- 172.21.15.32:0/3661474511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08b40cce90 0x7f08b415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 >> 172.21.15.32:0/3661474511 conn(0x7f08b406b3d0 msgr2=0x7f08b4070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 shutdown_connections 2024-03-06T07:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.286+0000 7f08b3e00640 1 -- 172.21.15.32:0/3661474511 wait complete. 2024-03-06T07:35:19.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:19.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.523+0000 7f6a8f600640 1 Processor -- start 2024-03-06T07:35:19.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.523+0000 7f6a8f600640 1 -- start start 2024-03-06T07:35:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.524+0000 7f6a8f600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6a880af1a0 0x7f6a8815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a8f600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a880b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a8f600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8815cc40 con 0x7f6a880af580 2024-03-06T07:35:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a8f600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8815cf20 con 0x7f6a880af1a0 2024-03-06T07:35:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a8f600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8815d200 con 0x7f6a880cdc20 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a87e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6a880af580 0x7f6a880b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48768/0 (socket says 172.21.15.32:48768) 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a87e00640 1 -- 172.21.15.32:0/3105624812 learned_addr learned my addr 172.21.15.32:0/3105624812 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a8cc00640 1 --2- 172.21.15.32:0/3105624812 >> v2:172.21.15.39:3300/0 conn(0x7f6a880af1a0 0x7f6a8815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.525+0000 7f6a8d600640 1 --2- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a880b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a87400640 1 -- 172.21.15.32:0/3105624812 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455526566 0 0) 0x7f6a8815cc40 con 0x7f6a880af580 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a87400640 1 -- 172.21.15.32:0/3105624812 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a70003520 con 0x7f6a880af580 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a8cc00640 1 -- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 msgr2=0x7f6a880b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a8cc00640 1 --2- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a880b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a87400640 1 -- 172.21.15.32:0/3105624812 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2936074472 0 0) 0x7f6a70003520 con 0x7f6a880af580 2024-03-06T07:35:19.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a8cc00640 1 -- 172.21.15.32:0/3105624812 >> v1:172.21.15.32:6789/0 conn(0x7f6a880af580 legacy=0x7f6a880b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.526+0000 7f6a8cc00640 1 -- 172.21.15.32:0/3105624812 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8815d4e0 con 0x7f6a880af1a0 2024-03-06T07:35:19.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.527+0000 7f6a8cc00640 1 --2- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a8815c600 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0097a0 tx=0x7f6a7c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35b035ee30895cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.528+0000 7f6a87400640 1 -- 172.21.15.32:0/3105624812 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c037070 con 0x7f6a880af1a0 2024-03-06T07:35:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.528+0000 7f6a87400640 1 -- 172.21.15.32:0/3105624812 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c030510 con 0x7f6a880af1a0 2024-03-06T07:35:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.528+0000 7f6a87400640 1 -- 172.21.15.32:0/3105624812 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74001670 con 0x7f6a880af1a0 2024-03-06T07:35:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.528+0000 7f6a8f600640 1 -- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 msgr2=0x7f6a8815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.528+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a8815c600 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0097a0 tx=0x7f6a7c02b710 comp rx=0 tx=0).stop 2024-03-06T07:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.529+0000 7f6a8f600640 1 -- 172.21.15.32:0/3105624812 shutdown_connections 2024-03-06T07:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.529+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a880b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.529+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3105624812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a8815c600 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.529+0000 7f6a8f600640 1 -- 172.21.15.32:0/3105624812 >> 172.21.15.32:0/3105624812 conn(0x7f6a8806afb0 msgr2=0x7f6a8806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.529+0000 7f6a8f600640 1 -- 172.21.15.32:0/3105624812 shutdown_connections 2024-03-06T07:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.529+0000 7f6a8f600640 1 -- 172.21.15.32:0/3105624812 wait complete. 2024-03-06T07:35:19.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.531+0000 7f6a8f600640 1 Processor -- start 2024-03-06T07:35:19.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.531+0000 7f6a8f600640 1 -- start start 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a881563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 0x7f6a8806d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8f600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a88143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8f600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a880b36e0 con 0x7f6a880af580 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a880b3560 con 0x7f6a880af1a0 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8f600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a88155e70 con 0x7f6a880cdc20 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 0x7f6a8806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 0x7f6a8806d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50298/0 (socket says 172.21.15.32:50298) 2024-03-06T07:35:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 -- 172.21.15.32:0/3052290513 learned_addr learned my addr 172.21.15.32:0/3052290513 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a8cc00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a881563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 -- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 msgr2=0x7f6a88143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a88143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 -- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 msgr2=0x7f6a881563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a881563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.532+0000 7f6a87e00640 1 -- 172.21.15.32:0/3052290513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8806da60 con 0x7f6a880af580 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.533+0000 7f6a87e00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 0x7f6a8806d520 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400c750 tx=0x7f6a7400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.533+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74017070 con 0x7f6a880af580 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.533+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a74005ad0 con 0x7f6a880af580 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.533+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74007070 con 0x7f6a880af580 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.533+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8806dd00 con 0x7f6a880af580 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.533+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8806e2c0 con 0x7f6a880af580 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.535+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8804f3a0 con 0x7f6a880af580 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.535+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a7401a070 con 0x7f6a880af580 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.536+0000 7f6a84c00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a6806b4d0 0x7f6a6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.536+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a74089530 con 0x7f6a880af580 2024-03-06T07:35:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.537+0000 7f6a8cc00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a6806b4d0 0x7f6a6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.537+0000 7f6a8cc00640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a6806b4d0 0x7f6a6806d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c034000 tx=0x7f6a7c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.541+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a74057d60 con 0x7f6a880af580 2024-03-06T07:35:19.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.742+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a8806b430 con 0x7f6a880af580 2024-03-06T07:35:19.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.745+0000 7f6a84c00640 1 -- 172.21.15.32:0/3052290513 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f6a7405be00 con 0x7f6a880af580 2024-03-06T07:35:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:19.747 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":110,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:16.400077+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.747+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a6806b4d0 msgr2=0x7f6a6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.747+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a6806b4d0 0x7f6a6806d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c034000 tx=0x7f6a7c008030 comp rx=0 tx=0).stop 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.747+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 msgr2=0x7f6a8806d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.747+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 0x7f6a8806d520 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400c750 tx=0x7f6a7400cc10 comp rx=0 tx=0).stop 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 shutdown_connections 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a6806b4d0 0x7f6a6806d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a880cdc20 0x7f6a88143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a880af580 0x7f6a8806d520 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 --2- 172.21.15.32:0/3052290513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a880af1a0 0x7f6a881563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 >> 172.21.15.32:0/3052290513 conn(0x7f6a8806afb0 msgr2=0x7f6a880acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 shutdown_connections 2024-03-06T07:35:19.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.748+0000 7f6a8f600640 1 -- 172.21.15.32:0/3052290513 wait complete. 2024-03-06T07:35:19.769 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:35:19.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.991+0000 7f3159e00640 1 Processor -- start 2024-03-06T07:35:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.991+0000 7f3159e00640 1 -- start start 2024-03-06T07:35:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.992+0000 7f3159e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 0x7f31540b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3159e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540b4370 con 0x7f315415b320 2024-03-06T07:35:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3159e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315415b7b0 con 0x7f31540cede0 2024-03-06T07:35:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3159e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cd060 con 0x7f31540b3f90 2024-03-06T07:35:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3159400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f315415b320 0x7f3154071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48774/0 (socket says 172.21.15.32:48774) 2024-03-06T07:35:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3159400640 1 -- 172.21.15.32:0/1952567613 learned_addr learned my addr 172.21.15.32:0/1952567613 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3158a00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 0x7f31540b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.993+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394952958 0 0) 0x7f31540b4370 con 0x7f315415b320 2024-03-06T07:35:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f313c003520 con 0x7f315415b320 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117895911 0 0) 0x7f313c003520 con 0x7f315415b320 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 msgr2=0x7f31540b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 0x7f31540b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 >> v1:172.21.15.39:6789/0 conn(0x7f31540cede0 legacy=0x7f31540cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540cf240 con 0x7f315415b320 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f314c0019f0 con 0x7f315415b320 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 >> v1:172.21.15.32:6789/0 conn(0x7f315415b320 legacy=0x7f3154071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c0039e0 0x7f313c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f313c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 0x7f313c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cd060 con 0x7f313c004390 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c00ba70 con 0x7f313c0039e0 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c00bc70 con 0x7f31540b3f90 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.994+0000 7f3153e00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f313c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.995+0000 7f3158a00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c0039e0 0x7f313c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.995+0000 7f3153e00640 1 -- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 msgr2=0x7f313c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.995+0000 7f3153e00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540b3f90 0x7f313c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.995+0000 7f3153e00640 1 -- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c0039e0 msgr2=0x7f313c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.995+0000 7f3153e00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c0039e0 0x7f313c003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.995+0000 7f3153e00640 1 -- 172.21.15.32:0/1952567613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c003520 con 0x7f313c004390 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.996+0000 7f3153e00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f313c004860 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f3140001860 tx=0x7f313c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239bbade7ba487a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.996+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140038070 con 0x7f313c004390 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.996+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3140033410 con 0x7f313c004390 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.996+0000 7f3153400640 1 -- 172.21.15.32:0/1952567613 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140001430 con 0x7f313c004390 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159e00640 1 -- 172.21.15.32:0/1952567613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 msgr2=0x7f313c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159e00640 1 --2- 172.21.15.32:0/1952567613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f313c004860 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f3140001860 tx=0x7f313c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159400640 1 -- 172.21.15.32:0/1952567613 reap_dead start 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159e00640 1 -- 172.21.15.32:0/1952567613 shutdown_connections 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159e00640 1 -- 172.21.15.32:0/1952567613 >> 172.21.15.32:0/1952567613 conn(0x7f315406b3d0 msgr2=0x7f315406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159e00640 1 -- 172.21.15.32:0/1952567613 shutdown_connections 2024-03-06T07:35:20.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.997+0000 7f3159e00640 1 -- 172.21.15.32:0/1952567613 wait complete. 2024-03-06T07:35:20.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.999+0000 7f3159e00640 1 Processor -- start 2024-03-06T07:35:20.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:19.999+0000 7f3159e00640 1 -- start start 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3159e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f315415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3159e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31540afbe0 0x7f31540714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3159e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540af280 0x7f31541550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3159e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b2f70 con 0x7f313c004390 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3159e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b3b00 con 0x7f31540afbe0 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3159e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b34b0 con 0x7f31540af280 2024-03-06T07:35:20.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3158a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f315415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3158a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f315415c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50320/0 (socket says 172.21.15.32:50320) 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.000+0000 7f3158a00640 1 -- 172.21.15.32:0/2183547212 learned_addr learned my addr 172.21.15.32:0/2183547212 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3159400640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540af280 0x7f31541550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3153e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31540afbe0 0x7f31540714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3158a00640 1 -- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540af280 msgr2=0x7f31541550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3158a00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540af280 0x7f31541550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3158a00640 1 -- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31540afbe0 msgr2=0x7f31540714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3158a00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31540afbe0 0x7f31540714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3158a00640 1 -- 172.21.15.32:0/2183547212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31541f4b10 con 0x7f313c004390 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3153e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31540afbe0 0x7f31540714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3159400640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540af280 0x7f31541550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:20.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3158a00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f315415c1c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f3148010860 tx=0x7f3148010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314800f040 con 0x7f313c004390 2024-03-06T07:35:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314800eb60 con 0x7f313c004390 2024-03-06T07:35:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314801c400 con 0x7f313c004390 2024-03-06T07:35:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.001+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31540b0260 con 0x7f313c004390 2024-03-06T07:35:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.002+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31540719f0 con 0x7f313c004390 2024-03-06T07:35:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.003+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315406b7e0 con 0x7f313c004390 2024-03-06T07:35:20.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.003+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f314801c5a0 con 0x7f313c004390 2024-03-06T07:35:20.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.004+0000 7f3150c00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f313406b500 0x7f313406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.004+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f314808a810 con 0x7f313c004390 2024-03-06T07:35:20.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.005+0000 7f3153e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f313406b500 0x7f313406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.005+0000 7f3153e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f313406b500 0x7f313406d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3140035000 tx=0x7f3140009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.009+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f315406b7e0 con 0x7f313c004390 2024-03-06T07:35:20.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:20.173 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.207+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31541f48c0 con 0x7f313406b500 2024-03-06T07:35:20.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.214+0000 7f3150c00640 1 -- 172.21.15.32:0/2183547212 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240997 (secure 0 0 0) 0x7f31541f48c0 con 0x7f313406b500 2024-03-06T07:35:20.218 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:20.219 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:35:20.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f313406b500 msgr2=0x7f313406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f313406b500 0x7f313406d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3140035000 tx=0x7f3140009030 comp rx=0 tx=0).stop 2024-03-06T07:35:20.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 msgr2=0x7f315415c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f315415c1c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f3148010860 tx=0x7f3148010d20 comp rx=0 tx=0).stop 2024-03-06T07:35:20.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 shutdown_connections 2024-03-06T07:35:20.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f313406b500 0x7f313406d9c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f31540af280 0x7f31541550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31540afbe0 0x7f31540714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 --2- 172.21.15.32:0/2183547212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f313c004390 0x7f315415c1c0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 >> 172.21.15.32:0/2183547212 conn(0x7f315406b3d0 msgr2=0x7f31541e6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 shutdown_connections 2024-03-06T07:35:20.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.219+0000 7f3159e00640 1 -- 172.21.15.32:0/2183547212 wait complete. 2024-03-06T07:35:20.238 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-03-06T07:35:18.301693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590402,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":310960,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754663760,"statfs":{"total":773094113280,"available":772775690240,"internally_reserved":0,"allocated":11837440,"data_stored":5736612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.955107"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742830+0000","last_change":"2024-03-06T07:35:15.400844+0000","last_active":"2024-03-06T07:35:17.742830+0000","last_peered":"2024-03-06T07:35:17.742830+0000","last_clean":"2024-03-06T07:35:17.742830+0000","last_became_active":"2024-03-06T07:35:15.399994+0000","last_became_peered":"2024-03-06T07:35:15.399994+0000","last_unstale":"2024-03-06T07:35:17.742830+0000","last_undegraded":"2024-03-06T07:35:17.742830+0000","last_fullsized":"2024-03-06T07:35:17.742830+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:06.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742567+0000","last_change":"2024-03-06T07:35:15.400857+0000","last_active":"2024-03-06T07:35:17.742567+0000","last_peered":"2024-03-06T07:35:17.742567+0000","last_clean":"2024-03-06T07:35:17.742567+0000","last_became_active":"2024-03-06T07:35:15.400225+0000","last_became_peered":"2024-03-06T07:35:15.400225+0000","last_unstale":"2024-03-06T07:35:17.742567+0000","last_undegraded":"2024-03-06T07:35:17.742567+0000","last_fullsized":"2024-03-06T07:35:17.742567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:13.733492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009059+0000","last_change":"2024-03-06T07:35:15.407566+0000","last_active":"2024-03-06T07:35:18.009059+0000","last_peered":"2024-03-06T07:35:18.009059+0000","last_clean":"2024-03-06T07:35:18.009059+0000","last_became_active":"2024-03-06T07:35:15.406814+0000","last_became_peered":"2024-03-06T07:35:15.406814+0000","last_unstale":"2024-03-06T07:35:18.009059+0000","last_undegraded":"2024-03-06T07:35:18.009059+0000","last_fullsized":"2024-03-06T07:35:18.009059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:32.686301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009848+0000","last_change":"2024-03-06T07:35:15.406700+0000","last_active":"2024-03-06T07:35:18.009848+0000","last_peered":"2024-03-06T07:35:18.009848+0000","last_clean":"2024-03-06T07:35:18.009848+0000","last_became_active":"2024-03-06T07:35:15.405222+0000","last_became_peered":"2024-03-06T07:35:15.405222+0000","last_unstale":"2024-03-06T07:35:18.009848+0000","last_undegraded":"2024-03-06T07:35:18.009848+0000","last_fullsized":"2024-03-06T07:35:18.009848+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:08.728833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743267+0000","last_change":"2024-03-06T07:35:15.427552+0000","last_active":"2024-03-06T07:35:17.743267+0000","last_peered":"2024-03-06T07:35:17.743267+0000","last_clean":"2024-03-06T07:35:17.743267+0000","last_became_active":"2024-03-06T07:35:15.426943+0000","last_became_peered":"2024-03-06T07:35:15.426943+0000","last_unstale":"2024-03-06T07:35:17.743267+0000","last_undegraded":"2024-03-06T07:35:17.743267+0000","last_fullsized":"2024-03-06T07:35:17.743267+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:15.903402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009776+0000","last_change":"2024-03-06T07:35:15.427094+0000","last_active":"2024-03-06T07:35:18.009776+0000","last_peered":"2024-03-06T07:35:18.009776+0000","last_clean":"2024-03-06T07:35:18.009776+0000","last_became_active":"2024-03-06T07:35:15.426715+0000","last_became_peered":"2024-03-06T07:35:15.426715+0000","last_unstale":"2024-03-06T07:35:18.009776+0000","last_undegraded":"2024-03-06T07:35:18.009776+0000","last_fullsized":"2024-03-06T07:35:18.009776+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:04.774836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742354+0000","last_change":"2024-03-06T07:35:15.400984+0000","last_active":"2024-03-06T07:35:17.742354+0000","last_peered":"2024-03-06T07:35:17.742354+0000","last_clean":"2024-03-06T07:35:17.742354+0000","last_became_active":"2024-03-06T07:35:15.400200+0000","last_became_peered":"2024-03-06T07:35:15.400200+0000","last_unstale":"2024-03-06T07:35:17.742354+0000","last_undegraded":"2024-03-06T07:35:17.742354+0000","last_fullsized":"2024-03-06T07:35:17.742354+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:50.097445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743089+0000","last_change":"2024-03-06T07:35:15.419944+0000","last_active":"2024-03-06T07:35:17.743089+0000","last_peered":"2024-03-06T07:35:17.743089+0000","last_clean":"2024-03-06T07:35:17.743089+0000","last_became_active":"2024-03-06T07:35:15.419679+0000","last_became_peered":"2024-03-06T07:35:15.419679+0000","last_unstale":"2024-03-06T07:35:17.743089+0000","last_undegraded":"2024-03-06T07:35:17.743089+0000","last_fullsized":"2024-03-06T07:35:17.743089+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:46.647437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743220+0000","last_change":"2024-03-06T07:35:15.402690+0000","last_active":"2024-03-06T07:35:17.743220+0000","last_peered":"2024-03-06T07:35:17.743220+0000","last_clean":"2024-03-06T07:35:17.743220+0000","last_became_active":"2024-03-06T07:35:15.401904+0000","last_became_peered":"2024-03-06T07:35:15.401904+0000","last_unstale":"2024-03-06T07:35:17.743220+0000","last_undegraded":"2024-03-06T07:35:17.743220+0000","last_fullsized":"2024-03-06T07:35:17.743220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:40.479978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009195+0000","last_change":"2024-03-06T07:35:14.380541+0000","last_active":"2024-03-06T07:35:18.009195+0000","last_peered":"2024-03-06T07:35:18.009195+0000","last_clean":"2024-03-06T07:35:18.009195+0000","last_became_active":"2024-03-06T07:35:14.380292+0000","last_became_peered":"2024-03-06T07:35:14.380292+0000","last_unstale":"2024-03-06T07:35:18.009195+0000","last_undegraded":"2024-03-06T07:35:18.009195+0000","last_fullsized":"2024-03-06T07:35:18.009195+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:47.863938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009470+0000","last_change":"2024-03-06T07:35:14.382538+0000","last_active":"2024-03-06T07:35:18.009470+0000","last_peered":"2024-03-06T07:35:18.009470+0000","last_clean":"2024-03-06T07:35:18.009470+0000","last_became_active":"2024-03-06T07:35:14.382216+0000","last_became_peered":"2024-03-06T07:35:14.382216+0000","last_unstale":"2024-03-06T07:35:18.009470+0000","last_undegraded":"2024-03-06T07:35:18.009470+0000","last_fullsized":"2024-03-06T07:35:18.009470+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:28.688041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009523+0000","last_change":"2024-03-06T07:35:15.418037+0000","last_active":"2024-03-06T07:35:18.009523+0000","last_peered":"2024-03-06T07:35:18.009523+0000","last_clean":"2024-03-06T07:35:18.009523+0000","last_became_active":"2024-03-06T07:35:15.417683+0000","last_became_peered":"2024-03-06T07:35:15.417683+0000","last_unstale":"2024-03-06T07:35:18.009523+0000","last_undegraded":"2024-03-06T07:35:18.009523+0000","last_fullsized":"2024-03-06T07:35:18.009523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:50.630525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009731+0000","last_change":"2024-03-06T07:35:15.429165+0000","last_active":"2024-03-06T07:35:18.009731+0000","last_peered":"2024-03-06T07:35:18.009731+0000","last_clean":"2024-03-06T07:35:18.009731+0000","last_became_active":"2024-03-06T07:35:15.428736+0000","last_became_peered":"2024-03-06T07:35:15.428736+0000","last_unstale":"2024-03-06T07:35:18.009731+0000","last_undegraded":"2024-03-06T07:35:18.009731+0000","last_fullsized":"2024-03-06T07:35:18.009731+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:00.975501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743000+0000","last_change":"2024-03-06T07:35:14.376041+0000","last_active":"2024-03-06T07:35:17.743000+0000","last_peered":"2024-03-06T07:35:17.743000+0000","last_clean":"2024-03-06T07:35:17.743000+0000","last_became_active":"2024-03-06T07:35:14.375640+0000","last_became_peered":"2024-03-06T07:35:14.375640+0000","last_unstale":"2024-03-06T07:35:17.743000+0000","last_undegraded":"2024-03-06T07:35:17.743000+0000","last_fullsized":"2024-03-06T07:35:17.743000+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:34.413706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009929+0000","last_change":"2024-03-06T07:35:15.405488+0000","last_active":"2024-03-06T07:35:18.009929+0000","last_peered":"2024-03-06T07:35:18.009929+0000","last_clean":"2024-03-06T07:35:18.009929+0000","last_became_active":"2024-03-06T07:35:15.404519+0000","last_became_peered":"2024-03-06T07:35:15.404519+0000","last_unstale":"2024-03-06T07:35:18.009929+0000","last_undegraded":"2024-03-06T07:35:18.009929+0000","last_fullsized":"2024-03-06T07:35:18.009929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:39.003490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009262+0000","last_change":"2024-03-06T07:35:15.418068+0000","last_active":"2024-03-06T07:35:18.009262+0000","last_peered":"2024-03-06T07:35:18.009262+0000","last_clean":"2024-03-06T07:35:18.009262+0000","last_became_active":"2024-03-06T07:35:15.417772+0000","last_became_peered":"2024-03-06T07:35:15.417772+0000","last_unstale":"2024-03-06T07:35:18.009262+0000","last_undegraded":"2024-03-06T07:35:18.009262+0000","last_fullsized":"2024-03-06T07:35:18.009262+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:34.226912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742993+0000","last_change":"2024-03-06T07:35:15.426171+0000","last_active":"2024-03-06T07:35:17.742993+0000","last_peered":"2024-03-06T07:35:17.742993+0000","last_clean":"2024-03-06T07:35:17.742993+0000","last_became_active":"2024-03-06T07:35:15.425884+0000","last_became_peered":"2024-03-06T07:35:15.425884+0000","last_unstale":"2024-03-06T07:35:17.742993+0000","last_undegraded":"2024-03-06T07:35:17.742993+0000","last_fullsized":"2024-03-06T07:35:17.742993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:13.615988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743151+0000","last_change":"2024-03-06T07:35:14.378202+0000","last_active":"2024-03-06T07:35:17.743151+0000","last_peered":"2024-03-06T07:35:17.743151+0000","last_clean":"2024-03-06T07:35:17.743151+0000","last_became_active":"2024-03-06T07:35:14.377306+0000","last_became_peered":"2024-03-06T07:35:14.377306+0000","last_unstale":"2024-03-06T07:35:17.743151+0000","last_undegraded":"2024-03-06T07:35:17.743151+0000","last_fullsized":"2024-03-06T07:35:17.743151+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:09.529110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742487+0000","last_change":"2024-03-06T07:35:14.380931+0000","last_active":"2024-03-06T07:35:17.742487+0000","last_peered":"2024-03-06T07:35:17.742487+0000","last_clean":"2024-03-06T07:35:17.742487+0000","last_became_active":"2024-03-06T07:35:14.378613+0000","last_became_peered":"2024-03-06T07:35:14.378613+0000","last_unstale":"2024-03-06T07:35:17.742487+0000","last_undegraded":"2024-03-06T07:35:17.742487+0000","last_fullsized":"2024-03-06T07:35:17.742487+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:05.506109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009643+0000","last_change":"2024-03-06T07:35:15.425805+0000","last_active":"2024-03-06T07:35:18.009643+0000","last_peered":"2024-03-06T07:35:18.009643+0000","last_clean":"2024-03-06T07:35:18.009643+0000","last_became_active":"2024-03-06T07:35:15.425475+0000","last_became_peered":"2024-03-06T07:35:15.425475+0000","last_unstale":"2024-03-06T07:35:18.009643+0000","last_undegraded":"2024-03-06T07:35:18.009643+0000","last_fullsized":"2024-03-06T07:35:18.009643+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:02.265621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009272+0000","last_change":"2024-03-06T07:35:14.390005+0000","last_active":"2024-03-06T07:35:18.009272+0000","last_peered":"2024-03-06T07:35:18.009272+0000","last_clean":"2024-03-06T07:35:18.009272+0000","last_became_active":"2024-03-06T07:35:14.389726+0000","last_became_peered":"2024-03-06T07:35:14.389726+0000","last_unstale":"2024-03-06T07:35:18.009272+0000","last_undegraded":"2024-03-06T07:35:18.009272+0000","last_fullsized":"2024-03-06T07:35:18.009272+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:47.664942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009586+0000","last_change":"2024-03-06T07:35:14.381592+0000","last_active":"2024-03-06T07:35:18.009586+0000","last_peered":"2024-03-06T07:35:18.009586+0000","last_clean":"2024-03-06T07:35:18.009586+0000","last_became_active":"2024-03-06T07:35:14.381330+0000","last_became_peered":"2024-03-06T07:35:14.381330+0000","last_unstale":"2024-03-06T07:35:18.009586+0000","last_undegraded":"2024-03-06T07:35:18.009586+0000","last_fullsized":"2024-03-06T07:35:18.009586+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:13.999882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:14.344054+0000","last_change":"2024-03-06T07:35:14.344054+0000","last_active":"2024-03-06T07:35:14.344054+0000","last_peered":"2024-03-06T07:35:14.344054+0000","last_clean":"2024-03-06T07:35:14.344054+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:14.344054+0000","last_undegraded":"2024-03-06T07:35:14.344054+0000","last_fullsized":"2024-03-06T07:35:14.344054+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009420+0000","last_change":"2024-03-06T07:35:15.406010+0000","last_active":"2024-03-06T07:35:18.009420+0000","last_peered":"2024-03-06T07:35:18.009420+0000","last_clean":"2024-03-06T07:35:18.009420+0000","last_became_active":"2024-03-06T07:35:15.404224+0000","last_became_peered":"2024-03-06T07:35:15.404224+0000","last_unstale":"2024-03-06T07:35:18.009420+0000","last_undegraded":"2024-03-06T07:35:18.009420+0000","last_fullsized":"2024-03-06T07:35:18.009420+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:59.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:35:13.329023+0000","last_change":"2024-03-06T07:35:13.329023+0000","last_active":"2024-03-06T07:35:13.329023+0000","last_peered":"2024-03-06T07:35:13.329023+0000","last_clean":"2024-03-06T07:35:13.329023+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:35:13.329023+0000","last_undegraded":"2024-03-06T07:35:13.329023+0000","last_fullsized":"2024-03-06T07:35:13.329023+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":331,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-06T07:35:10.373033+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:35:10.373033+0000","last_peered":"2024-03-06T07:35:10.373033+0000","last_clean":"2024-03-06T07:35:10.373033+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:35:10.373033+0000","last_undegraded":"2024-03-06T07:35:10.373033+0000","last_fullsized":"2024-03-06T07:35:10.373033+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":14,"acting":14,"num_store_stats":2},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37216,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334624,"statfs":{"total":96636764160,"available":96598654976,"internally_reserved":0,"allocated":1912832,"data_stored":1154910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":564542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":564542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40796,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331044,"statfs":{"total":96636764160,"available":96594989056,"internally_reserved":0,"allocated":1318912,"data_stored":564542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40848,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330992,"statfs":{"total":96636764160,"available":96594935808,"internally_reserved":0,"allocated":1372160,"data_stored":585091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":564542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":1363968,"data_stored":583533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":1912832,"data_stored":1154910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:20.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:20.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.504+0000 7f8009c00640 1 Processor -- start 2024-03-06T07:35:20.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.504+0000 7f8009c00640 1 -- start start 2024-03-06T07:35:20.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.505+0000 7f8009c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f80040d2930 0x7f80040b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.505+0000 7f8009c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80040cea50 con 0x7f8004161b40 2024-03-06T07:35:20.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.505+0000 7f8009c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80040cecf0 con 0x7f80040cfd90 2024-03-06T07:35:20.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.505+0000 7f8009c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040b5770 con 0x7f80040d2930 2024-03-06T07:35:20.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.505+0000 7f8002a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8004161b40 0x7f80040b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48788/0 (socket says 172.21.15.32:48788) 2024-03-06T07:35:20.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8002a00640 1 -- 172.21.15.32:0/648041188 learned_addr learned my addr 172.21.15.32:0/648041188 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821548024 0 0) 0x7f80040cea50 con 0x7f8004161b40 2024-03-06T07:35:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8003520 con 0x7f8004161b40 2024-03-06T07:35:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8003400640 1 --2- 172.21.15.32:0/648041188 >> v2:172.21.15.39:3300/0 conn(0x7f80040d2930 0x7f80040b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063597200 0 0) 0x7f7fe8003520 con 0x7f8004161b40 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 >> v2:172.21.15.39:3300/0 conn(0x7f80040d2930 msgr2=0x7f80040b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 --2- 172.21.15.32:0/648041188 >> v2:172.21.15.39:3300/0 conn(0x7f80040d2930 0x7f80040b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 >> v1:172.21.15.39:6789/0 conn(0x7f80040cfd90 legacy=0x7f80040d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.506+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80040cefd0 con 0x7f8004161b40 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f7ff4001670 con 0x7f8004161b40 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 >> v1:172.21.15.32:6789/0 conn(0x7f8004161b40 legacy=0x7f80040b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f7fe8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe8004800 0x7f7fe8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7fe8005230 0x7f7fe800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040b5770 con 0x7f7fe8003a20 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe800c190 con 0x7f7fe8004800 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe800c390 con 0x7f7fe8005230 2024-03-06T07:35:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8002a00640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f7fe8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8003400640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7fe8005230 0x7f7fe800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8002a00640 1 -- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7fe8005230 msgr2=0x7f7fe800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8002a00640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7fe8005230 0x7f7fe800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8002a00640 1 -- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe8004800 msgr2=0x7f7fe8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8002a00640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe8004800 0x7f7fe8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8002a00640 1 -- 172.21.15.32:0/648041188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8003520 con 0x7f7fe8003a20 2024-03-06T07:35:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.507+0000 7f8003400640 1 -- 172.21.15.32:0/648041188 reap_dead start 2024-03-06T07:35:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.509+0000 7f8002a00640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f7fe8003e40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400c210 tx=0x7f7fe8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0b7caa9c210d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.509+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4007d30 con 0x7f7fe8003a20 2024-03-06T07:35:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.509+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4030070 con 0x7f7fe8003a20 2024-03-06T07:35:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.509+0000 7f8001600640 1 -- 172.21.15.32:0/648041188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4005d70 con 0x7f7fe8003a20 2024-03-06T07:35:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.509+0000 7f8009c00640 1 -- 172.21.15.32:0/648041188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 msgr2=0x7f7fe8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.509+0000 7f8009c00640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f7fe8003e40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400c210 tx=0x7f7fe8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.510+0000 7f8009c00640 1 -- 172.21.15.32:0/648041188 shutdown_connections 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.510+0000 7f8009c00640 1 --2- 172.21.15.32:0/648041188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f7fe8003e40 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.510+0000 7f8009c00640 1 -- 172.21.15.32:0/648041188 >> 172.21.15.32:0/648041188 conn(0x7f800409d220 msgr2=0x7f80040d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.510+0000 7f8009c00640 1 -- 172.21.15.32:0/648041188 shutdown_connections 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.510+0000 7f8009c00640 1 -- 172.21.15.32:0/648041188 wait complete. 2024-03-06T07:35:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.511+0000 7f8009c00640 1 Processor -- start 2024-03-06T07:35:20.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 -- start start 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f8004159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8004159c20 0x7f80040b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f800415a0a0 0x7f80040b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040cf590 con 0x7f7fe8003a20 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040cf900 con 0x7f8004159c20 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.512+0000 7f8009c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80040b2ff0 con 0x7f800415a0a0 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f8004159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f8004159670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50334/0 (socket says 172.21.15.32:50334) 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 -- 172.21.15.32:0/691522002 learned_addr learned my addr 172.21.15.32:0/691522002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8003400640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f800415a0a0 0x7f80040b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002000640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8004159c20 0x7f80040b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 -- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f800415a0a0 msgr2=0x7f80040b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f800415a0a0 0x7f80040b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 -- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8004159c20 msgr2=0x7f80040b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8004159c20 0x7f80040b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002a00640 1 -- 172.21.15.32:0/691522002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800415a600 con 0x7f7fe8003a20 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8002000640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8004159c20 0x7f80040b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.513+0000 7f8003400640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f800415a0a0 0x7f80040b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.514+0000 7f8002a00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f8004159670 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400c330 tx=0x7f7ff402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.514+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4004bf0 con 0x7f7fe8003a20 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.514+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4004d90 con 0x7f7fe8003a20 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.514+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4007270 con 0x7f7fe8003a20 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.514+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800415a8e0 con 0x7f7fe8003a20 2024-03-06T07:35:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.514+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80040b6890 con 0x7f7fe8003a20 2024-03-06T07:35:20.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.515+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800404f3a0 con 0x7f7fe8003a20 2024-03-06T07:35:20.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.516+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ff4048020 con 0x7f7fe8003a20 2024-03-06T07:35:20.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.517+0000 7f7ffaa00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7fe006b500 0x7f7fe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.517+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ff40ab710 con 0x7f7fe8003a20 2024-03-06T07:35:20.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.517+0000 7f8002000640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7fe006b500 0x7f7fe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.518+0000 7f8002000640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7fe006b500 0x7f7fe006d9c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f80040ce380 tx=0x7f7fec009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.522+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ff407a580 con 0x7f7fe8003a20 2024-03-06T07:35:20.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.721+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80040d1df0 con 0x7f7fe006b500 2024-03-06T07:35:20.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.739+0000 7f7ffaa00640 1 -- 172.21.15.32:0/691522002 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f80040d1df0 con 0x7f7fe006b500 2024-03-06T07:35:20.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:35:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7fe006b500 msgr2=0x7f7fe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7fe006b500 0x7f7fe006d9c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f80040ce380 tx=0x7f7fec009040 comp rx=0 tx=0).stop 2024-03-06T07:35:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 msgr2=0x7f8004159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f8004159670 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400c330 tx=0x7f7ff402e070 comp rx=0 tx=0).stop 2024-03-06T07:35:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 shutdown_connections 2024-03-06T07:35:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7fe006b500 0x7f7fe006d9c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f800415a0a0 0x7f80040b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8004159c20 0x7f80040b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 --2- 172.21.15.32:0/691522002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe8003a20 0x7f8004159670 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.745+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 >> 172.21.15.32:0/691522002 conn(0x7f800409d220 msgr2=0x7f80040d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.746+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 shutdown_connections 2024-03-06T07:35:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:20.746+0000 7f8009c00640 1 -- 172.21.15.32:0/691522002 wait complete. 2024-03-06T07:35:20.767 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-03-06T07:35:20.302298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311236,"kb_used_data":11864,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754663484,"statfs":{"total":773094113280,"available":772775407616,"internally_reserved":0,"allocated":12148736,"data_stored":5878028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001880"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233139+0000","last_change":"2024-03-06T07:35:15.432427+0000","last_active":"2024-03-06T07:35:18.233139+0000","last_peered":"2024-03-06T07:35:18.233139+0000","last_clean":"2024-03-06T07:35:18.233139+0000","last_became_active":"2024-03-06T07:35:15.431799+0000","last_became_peered":"2024-03-06T07:35:15.431799+0000","last_unstale":"2024-03-06T07:35:18.233139+0000","last_undegraded":"2024-03-06T07:35:18.233139+0000","last_fullsized":"2024-03-06T07:35:18.233139+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:40.963319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410541+0000","last_change":"2024-03-06T07:35:15.418540+0000","last_active":"2024-03-06T07:35:16.410541+0000","last_peered":"2024-03-06T07:35:16.410541+0000","last_clean":"2024-03-06T07:35:16.410541+0000","last_became_active":"2024-03-06T07:35:15.418181+0000","last_became_peered":"2024-03-06T07:35:15.418181+0000","last_unstale":"2024-03-06T07:35:16.410541+0000","last_undegraded":"2024-03-06T07:35:16.410541+0000","last_fullsized":"2024-03-06T07:35:16.410541+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:57.070022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.411207+0000","last_change":"2024-03-06T07:35:15.422567+0000","last_active":"2024-03-06T07:35:16.411207+0000","last_peered":"2024-03-06T07:35:16.411207+0000","last_clean":"2024-03-06T07:35:16.411207+0000","last_became_active":"2024-03-06T07:35:15.422262+0000","last_became_peered":"2024-03-06T07:35:15.422262+0000","last_unstale":"2024-03-06T07:35:16.411207+0000","last_undegraded":"2024-03-06T07:35:16.411207+0000","last_fullsized":"2024-03-06T07:35:16.411207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:59.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233273+0000","last_change":"2024-03-06T07:35:15.434370+0000","last_active":"2024-03-06T07:35:18.233273+0000","last_peered":"2024-03-06T07:35:18.233273+0000","last_clean":"2024-03-06T07:35:18.233273+0000","last_became_active":"2024-03-06T07:35:15.430083+0000","last_became_peered":"2024-03-06T07:35:15.430083+0000","last_unstale":"2024-03-06T07:35:18.233273+0000","last_undegraded":"2024-03-06T07:35:18.233273+0000","last_fullsized":"2024-03-06T07:35:18.233273+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:07.815447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.230850+0000","last_change":"2024-03-06T07:35:15.422313+0000","last_active":"2024-03-06T07:35:18.230850+0000","last_peered":"2024-03-06T07:35:18.230850+0000","last_clean":"2024-03-06T07:35:18.230850+0000","last_became_active":"2024-03-06T07:35:15.422020+0000","last_became_peered":"2024-03-06T07:35:15.422020+0000","last_unstale":"2024-03-06T07:35:18.230850+0000","last_undegraded":"2024-03-06T07:35:18.230850+0000","last_fullsized":"2024-03-06T07:35:18.230850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:16.806795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742830+0000","last_change":"2024-03-06T07:35:15.400844+0000","last_active":"2024-03-06T07:35:17.742830+0000","last_peered":"2024-03-06T07:35:17.742830+0000","last_clean":"2024-03-06T07:35:17.742830+0000","last_became_active":"2024-03-06T07:35:15.399994+0000","last_became_peered":"2024-03-06T07:35:15.399994+0000","last_unstale":"2024-03-06T07:35:17.742830+0000","last_undegraded":"2024-03-06T07:35:17.742830+0000","last_fullsized":"2024-03-06T07:35:17.742830+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:06.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.426973+0000","last_change":"2024-03-06T07:35:15.432471+0000","last_active":"2024-03-06T07:35:16.426973+0000","last_peered":"2024-03-06T07:35:16.426973+0000","last_clean":"2024-03-06T07:35:16.426973+0000","last_became_active":"2024-03-06T07:35:15.432017+0000","last_became_peered":"2024-03-06T07:35:15.432017+0000","last_unstale":"2024-03-06T07:35:16.426973+0000","last_undegraded":"2024-03-06T07:35:16.426973+0000","last_fullsized":"2024-03-06T07:35:16.426973+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:58.557140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410502+0000","last_change":"2024-03-06T07:35:15.418779+0000","last_active":"2024-03-06T07:35:16.410502+0000","last_peered":"2024-03-06T07:35:16.410502+0000","last_clean":"2024-03-06T07:35:16.410502+0000","last_became_active":"2024-03-06T07:35:15.412573+0000","last_became_peered":"2024-03-06T07:35:15.412573+0000","last_unstale":"2024-03-06T07:35:16.410502+0000","last_undegraded":"2024-03-06T07:35:16.410502+0000","last_fullsized":"2024-03-06T07:35:16.410502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:58.365311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233628+0000","last_change":"2024-03-06T07:35:15.438354+0000","last_active":"2024-03-06T07:35:18.233628+0000","last_peered":"2024-03-06T07:35:18.233628+0000","last_clean":"2024-03-06T07:35:18.233628+0000","last_became_active":"2024-03-06T07:35:15.438052+0000","last_became_peered":"2024-03-06T07:35:15.438052+0000","last_unstale":"2024-03-06T07:35:18.233628+0000","last_undegraded":"2024-03-06T07:35:18.233628+0000","last_fullsized":"2024-03-06T07:35:18.233628+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:00.150466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.232594+0000","last_change":"2024-03-06T07:35:15.426327+0000","last_active":"2024-03-06T07:35:18.232594+0000","last_peered":"2024-03-06T07:35:18.232594+0000","last_clean":"2024-03-06T07:35:18.232594+0000","last_became_active":"2024-03-06T07:35:15.423371+0000","last_became_peered":"2024-03-06T07:35:15.423371+0000","last_unstale":"2024-03-06T07:35:18.232594+0000","last_undegraded":"2024-03-06T07:35:18.232594+0000","last_fullsized":"2024-03-06T07:35:18.232594+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:06.684036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.411350+0000","last_change":"2024-03-06T07:35:15.421983+0000","last_active":"2024-03-06T07:35:16.411350+0000","last_peered":"2024-03-06T07:35:16.411350+0000","last_clean":"2024-03-06T07:35:16.411350+0000","last_became_active":"2024-03-06T07:35:15.421250+0000","last_became_peered":"2024-03-06T07:35:15.421250+0000","last_unstale":"2024-03-06T07:35:16.411350+0000","last_undegraded":"2024-03-06T07:35:16.411350+0000","last_fullsized":"2024-03-06T07:35:16.411350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.951061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233704+0000","last_change":"2024-03-06T07:35:15.424732+0000","last_active":"2024-03-06T07:35:18.233704+0000","last_peered":"2024-03-06T07:35:18.233704+0000","last_clean":"2024-03-06T07:35:18.233704+0000","last_became_active":"2024-03-06T07:35:15.421992+0000","last_became_peered":"2024-03-06T07:35:15.421992+0000","last_unstale":"2024-03-06T07:35:18.233704+0000","last_undegraded":"2024-03-06T07:35:18.233704+0000","last_fullsized":"2024-03-06T07:35:18.233704+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:08.264225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.409932+0000","last_change":"2024-03-06T07:35:15.424303+0000","last_active":"2024-03-06T07:35:16.409932+0000","last_peered":"2024-03-06T07:35:16.409932+0000","last_clean":"2024-03-06T07:35:16.409932+0000","last_became_active":"2024-03-06T07:35:15.421489+0000","last_became_peered":"2024-03-06T07:35:15.421489+0000","last_unstale":"2024-03-06T07:35:16.409932+0000","last_undegraded":"2024-03-06T07:35:16.409932+0000","last_fullsized":"2024-03-06T07:35:16.409932+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:46.990345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408584+0000","last_change":"2024-03-06T07:35:15.417649+0000","last_active":"2024-03-06T07:35:16.408584+0000","last_peered":"2024-03-06T07:35:16.408584+0000","last_clean":"2024-03-06T07:35:16.408584+0000","last_became_active":"2024-03-06T07:35:15.415277+0000","last_became_peered":"2024-03-06T07:35:15.415277+0000","last_unstale":"2024-03-06T07:35:16.408584+0000","last_undegraded":"2024-03-06T07:35:16.408584+0000","last_fullsized":"2024-03-06T07:35:16.408584+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:33.531201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742567+0000","last_change":"2024-03-06T07:35:15.400857+0000","last_active":"2024-03-06T07:35:17.742567+0000","last_peered":"2024-03-06T07:35:17.742567+0000","last_clean":"2024-03-06T07:35:17.742567+0000","last_became_active":"2024-03-06T07:35:15.400225+0000","last_became_peered":"2024-03-06T07:35:15.400225+0000","last_unstale":"2024-03-06T07:35:17.742567+0000","last_undegraded":"2024-03-06T07:35:17.742567+0000","last_fullsized":"2024-03-06T07:35:17.742567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:13.733492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.230658+0000","last_change":"2024-03-06T07:35:15.431870+0000","last_active":"2024-03-06T07:35:18.230658+0000","last_peered":"2024-03-06T07:35:18.230658+0000","last_clean":"2024-03-06T07:35:18.230658+0000","last_became_active":"2024-03-06T07:35:15.431549+0000","last_became_peered":"2024-03-06T07:35:15.431549+0000","last_unstale":"2024-03-06T07:35:18.230658+0000","last_undegraded":"2024-03-06T07:35:18.230658+0000","last_fullsized":"2024-03-06T07:35:18.230658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:34.406673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.409568+0000","last_change":"2024-03-06T07:35:15.421667+0000","last_active":"2024-03-06T07:35:16.409568+0000","last_peered":"2024-03-06T07:35:16.409568+0000","last_clean":"2024-03-06T07:35:16.409568+0000","last_became_active":"2024-03-06T07:35:15.418447+0000","last_became_peered":"2024-03-06T07:35:15.418447+0000","last_unstale":"2024-03-06T07:35:16.409568+0000","last_undegraded":"2024-03-06T07:35:16.409568+0000","last_fullsized":"2024-03-06T07:35:16.409568+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:29.703072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.421855+0000","last_change":"2024-03-06T07:35:15.420029+0000","last_active":"2024-03-06T07:35:16.421855+0000","last_peered":"2024-03-06T07:35:16.421855+0000","last_clean":"2024-03-06T07:35:16.421855+0000","last_became_active":"2024-03-06T07:35:15.419708+0000","last_became_peered":"2024-03-06T07:35:15.419708+0000","last_unstale":"2024-03-06T07:35:16.421855+0000","last_undegraded":"2024-03-06T07:35:16.421855+0000","last_fullsized":"2024-03-06T07:35:16.421855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:30.431118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.411481+0000","last_change":"2024-03-06T07:35:15.420864+0000","last_active":"2024-03-06T07:35:16.411481+0000","last_peered":"2024-03-06T07:35:16.411481+0000","last_clean":"2024-03-06T07:35:16.411481+0000","last_became_active":"2024-03-06T07:35:15.420535+0000","last_became_peered":"2024-03-06T07:35:15.420535+0000","last_unstale":"2024-03-06T07:35:16.411481+0000","last_undegraded":"2024-03-06T07:35:16.411481+0000","last_fullsized":"2024-03-06T07:35:16.411481+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:02.265621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.409714+0000","last_change":"2024-03-06T07:35:15.429105+0000","last_active":"2024-03-06T07:35:16.409714+0000","last_peered":"2024-03-06T07:35:16.409714+0000","last_clean":"2024-03-06T07:35:16.409714+0000","last_became_active":"2024-03-06T07:35:15.428750+0000","last_became_peered":"2024-03-06T07:35:15.428750+0000","last_unstale":"2024-03-06T07:35:16.409714+0000","last_undegraded":"2024-03-06T07:35:16.409714+0000","last_fullsized":"2024-03-06T07:35:16.409714+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:48.679973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.409216+0000","last_change":"2024-03-06T07:35:15.425021+0000","last_active":"2024-03-06T07:35:16.409216+0000","last_peered":"2024-03-06T07:35:16.409216+0000","last_clean":"2024-03-06T07:35:16.409216+0000","last_became_active":"2024-03-06T07:35:15.423654+0000","last_became_peered":"2024-03-06T07:35:15.423654+0000","last_unstale":"2024-03-06T07:35:16.409216+0000","last_undegraded":"2024-03-06T07:35:16.409216+0000","last_fullsized":"2024-03-06T07:35:16.409216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:48.878969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009059+0000","last_change":"2024-03-06T07:35:15.407566+0000","last_active":"2024-03-06T07:35:18.009059+0000","last_peered":"2024-03-06T07:35:18.009059+0000","last_clean":"2024-03-06T07:35:18.009059+0000","last_became_active":"2024-03-06T07:35:15.406814+0000","last_became_peered":"2024-03-06T07:35:15.406814+0000","last_unstale":"2024-03-06T07:35:18.009059+0000","last_undegraded":"2024-03-06T07:35:18.009059+0000","last_fullsized":"2024-03-06T07:35:18.009059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:32.686301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408144+0000","last_change":"2024-03-06T07:35:15.431736+0000","last_active":"2024-03-06T07:35:16.408144+0000","last_peered":"2024-03-06T07:35:16.408144+0000","last_clean":"2024-03-06T07:35:16.408144+0000","last_became_active":"2024-03-06T07:35:15.431405+0000","last_became_peered":"2024-03-06T07:35:15.431405+0000","last_unstale":"2024-03-06T07:35:16.408144+0000","last_undegraded":"2024-03-06T07:35:16.408144+0000","last_fullsized":"2024-03-06T07:35:16.408144+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:16.987192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009848+0000","last_change":"2024-03-06T07:35:15.406700+0000","last_active":"2024-03-06T07:35:18.009848+0000","last_peered":"2024-03-06T07:35:18.009848+0000","last_clean":"2024-03-06T07:35:18.009848+0000","last_became_active":"2024-03-06T07:35:15.405222+0000","last_became_peered":"2024-03-06T07:35:15.405222+0000","last_unstale":"2024-03-06T07:35:18.009848+0000","last_undegraded":"2024-03-06T07:35:18.009848+0000","last_fullsized":"2024-03-06T07:35:18.009848+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:08.728833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410025+0000","last_change":"2024-03-06T07:35:15.419213+0000","last_active":"2024-03-06T07:35:16.410025+0000","last_peered":"2024-03-06T07:35:16.410025+0000","last_clean":"2024-03-06T07:35:16.410025+0000","last_became_active":"2024-03-06T07:35:15.418037+0000","last_became_peered":"2024-03-06T07:35:15.418037+0000","last_unstale":"2024-03-06T07:35:16.410025+0000","last_undegraded":"2024-03-06T07:35:16.410025+0000","last_fullsized":"2024-03-06T07:35:16.410025+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:15.014913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.232764+0000","last_change":"2024-03-06T07:35:15.440574+0000","last_active":"2024-03-06T07:35:18.232764+0000","last_peered":"2024-03-06T07:35:18.232764+0000","last_clean":"2024-03-06T07:35:18.232764+0000","last_became_active":"2024-03-06T07:35:15.436808+0000","last_became_peered":"2024-03-06T07:35:15.436808+0000","last_unstale":"2024-03-06T07:35:18.232764+0000","last_undegraded":"2024-03-06T07:35:18.232764+0000","last_fullsized":"2024-03-06T07:35:18.232764+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:11.574911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743267+0000","last_change":"2024-03-06T07:35:15.427552+0000","last_active":"2024-03-06T07:35:17.743267+0000","last_peered":"2024-03-06T07:35:17.743267+0000","last_clean":"2024-03-06T07:35:17.743267+0000","last_became_active":"2024-03-06T07:35:15.426943+0000","last_became_peered":"2024-03-06T07:35:15.426943+0000","last_unstale":"2024-03-06T07:35:17.743267+0000","last_undegraded":"2024-03-06T07:35:17.743267+0000","last_fullsized":"2024-03-06T07:35:17.743267+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:15.903402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009776+0000","last_change":"2024-03-06T07:35:15.427094+0000","last_active":"2024-03-06T07:35:18.009776+0000","last_peered":"2024-03-06T07:35:18.009776+0000","last_clean":"2024-03-06T07:35:18.009776+0000","last_became_active":"2024-03-06T07:35:15.426715+0000","last_became_peered":"2024-03-06T07:35:15.426715+0000","last_unstale":"2024-03-06T07:35:18.009776+0000","last_undegraded":"2024-03-06T07:35:18.009776+0000","last_fullsized":"2024-03-06T07:35:18.009776+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:04.774836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408674+0000","last_change":"2024-03-06T07:35:15.430954+0000","last_active":"2024-03-06T07:35:16.408674+0000","last_peered":"2024-03-06T07:35:16.408674+0000","last_clean":"2024-03-06T07:35:16.408674+0000","last_became_active":"2024-03-06T07:35:15.430154+0000","last_became_peered":"2024-03-06T07:35:15.430154+0000","last_unstale":"2024-03-06T07:35:16.408674+0000","last_undegraded":"2024-03-06T07:35:16.408674+0000","last_fullsized":"2024-03-06T07:35:16.408674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:30.431118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.427004+0000","last_change":"2024-03-06T07:35:15.425731+0000","last_active":"2024-03-06T07:35:16.427004+0000","last_peered":"2024-03-06T07:35:16.427004+0000","last_clean":"2024-03-06T07:35:16.427004+0000","last_became_active":"2024-03-06T07:35:15.425360+0000","last_became_peered":"2024-03-06T07:35:15.425360+0000","last_unstale":"2024-03-06T07:35:16.427004+0000","last_undegraded":"2024-03-06T07:35:16.427004+0000","last_fullsized":"2024-03-06T07:35:16.427004+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:00.809297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231542+0000","last_change":"2024-03-06T07:35:15.433464+0000","last_active":"2024-03-06T07:35:18.231542+0000","last_peered":"2024-03-06T07:35:18.231542+0000","last_clean":"2024-03-06T07:35:18.231542+0000","last_became_active":"2024-03-06T07:35:15.433157+0000","last_became_peered":"2024-03-06T07:35:15.433157+0000","last_unstale":"2024-03-06T07:35:18.231542+0000","last_undegraded":"2024-03-06T07:35:18.231542+0000","last_fullsized":"2024-03-06T07:35:18.231542+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:36.685493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742354+0000","last_change":"2024-03-06T07:35:15.400984+0000","last_active":"2024-03-06T07:35:17.742354+0000","last_peered":"2024-03-06T07:35:17.742354+0000","last_clean":"2024-03-06T07:35:17.742354+0000","last_became_active":"2024-03-06T07:35:15.400200+0000","last_became_peered":"2024-03-06T07:35:15.400200+0000","last_unstale":"2024-03-06T07:35:17.742354+0000","last_undegraded":"2024-03-06T07:35:17.742354+0000","last_fullsized":"2024-03-06T07:35:17.742354+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:50.097445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743089+0000","last_change":"2024-03-06T07:35:15.419944+0000","last_active":"2024-03-06T07:35:17.743089+0000","last_peered":"2024-03-06T07:35:17.743089+0000","last_clean":"2024-03-06T07:35:17.743089+0000","last_became_active":"2024-03-06T07:35:15.419679+0000","last_became_peered":"2024-03-06T07:35:15.419679+0000","last_unstale":"2024-03-06T07:35:17.743089+0000","last_undegraded":"2024-03-06T07:35:17.743089+0000","last_fullsized":"2024-03-06T07:35:17.743089+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:46.647437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.427413+0000","last_change":"2024-03-06T07:35:14.381910+0000","last_active":"2024-03-06T07:35:16.427413+0000","last_peered":"2024-03-06T07:35:16.427413+0000","last_clean":"2024-03-06T07:35:16.427413+0000","last_became_active":"2024-03-06T07:35:14.380249+0000","last_became_peered":"2024-03-06T07:35:14.380249+0000","last_unstale":"2024-03-06T07:35:16.427413+0000","last_undegraded":"2024-03-06T07:35:16.427413+0000","last_fullsized":"2024-03-06T07:35:16.427413+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:12.718461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.422701+0000","last_change":"2024-03-06T07:35:15.406093+0000","last_active":"2024-03-06T07:35:16.422701+0000","last_peered":"2024-03-06T07:35:16.422701+0000","last_clean":"2024-03-06T07:35:16.422701+0000","last_became_active":"2024-03-06T07:35:15.404254+0000","last_became_peered":"2024-03-06T07:35:15.404254+0000","last_unstale":"2024-03-06T07:35:16.422701+0000","last_undegraded":"2024-03-06T07:35:16.422701+0000","last_fullsized":"2024-03-06T07:35:16.422701+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:29.703072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408983+0000","last_change":"2024-03-06T07:35:14.384743+0000","last_active":"2024-03-06T07:35:16.408983+0000","last_peered":"2024-03-06T07:35:16.408983+0000","last_clean":"2024-03-06T07:35:16.408983+0000","last_became_active":"2024-03-06T07:35:14.381353+0000","last_became_peered":"2024-03-06T07:35:14.381353+0000","last_unstale":"2024-03-06T07:35:16.408983+0000","last_undegraded":"2024-03-06T07:35:16.408983+0000","last_fullsized":"2024-03-06T07:35:16.408983+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:15.972161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743220+0000","last_change":"2024-03-06T07:35:15.402690+0000","last_active":"2024-03-06T07:35:17.743220+0000","last_peered":"2024-03-06T07:35:17.743220+0000","last_clean":"2024-03-06T07:35:17.743220+0000","last_became_active":"2024-03-06T07:35:15.401904+0000","last_became_peered":"2024-03-06T07:35:15.401904+0000","last_unstale":"2024-03-06T07:35:17.743220+0000","last_undegraded":"2024-03-06T07:35:17.743220+0000","last_fullsized":"2024-03-06T07:35:17.743220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:40.479978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.232054+0000","last_change":"2024-03-06T07:35:14.383528+0000","last_active":"2024-03-06T07:35:18.232054+0000","last_peered":"2024-03-06T07:35:18.232054+0000","last_clean":"2024-03-06T07:35:18.232054+0000","last_became_active":"2024-03-06T07:35:14.382471+0000","last_became_peered":"2024-03-06T07:35:14.382471+0000","last_unstale":"2024-03-06T07:35:18.232054+0000","last_undegraded":"2024-03-06T07:35:18.232054+0000","last_fullsized":"2024-03-06T07:35:18.232054+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:05.506109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.411411+0000","last_change":"2024-03-06T07:35:15.421688+0000","last_active":"2024-03-06T07:35:16.411411+0000","last_peered":"2024-03-06T07:35:16.411411+0000","last_clean":"2024-03-06T07:35:16.411411+0000","last_became_active":"2024-03-06T07:35:15.421012+0000","last_became_peered":"2024-03-06T07:35:15.421012+0000","last_unstale":"2024-03-06T07:35:16.411411+0000","last_undegraded":"2024-03-06T07:35:16.411411+0000","last_fullsized":"2024-03-06T07:35:16.411411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:50.630525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231792+0000","last_change":"2024-03-06T07:35:14.383396+0000","last_active":"2024-03-06T07:35:18.231792+0000","last_peered":"2024-03-06T07:35:18.231792+0000","last_clean":"2024-03-06T07:35:18.231792+0000","last_became_active":"2024-03-06T07:35:14.381821+0000","last_became_peered":"2024-03-06T07:35:14.381821+0000","last_unstale":"2024-03-06T07:35:18.231792+0000","last_undegraded":"2024-03-06T07:35:18.231792+0000","last_fullsized":"2024-03-06T07:35:18.231792+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:45.577311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.232471+0000","last_change":"2024-03-06T07:35:15.430608+0000","last_active":"2024-03-06T07:35:18.232471+0000","last_peered":"2024-03-06T07:35:18.232471+0000","last_clean":"2024-03-06T07:35:18.232471+0000","last_became_active":"2024-03-06T07:35:15.430335+0000","last_became_peered":"2024-03-06T07:35:15.430335+0000","last_unstale":"2024-03-06T07:35:18.232471+0000","last_undegraded":"2024-03-06T07:35:18.232471+0000","last_fullsized":"2024-03-06T07:35:18.232471+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:37.356332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.427018+0000","last_change":"2024-03-06T07:35:14.381663+0000","last_active":"2024-03-06T07:35:16.427018+0000","last_peered":"2024-03-06T07:35:16.427018+0000","last_clean":"2024-03-06T07:35:16.427018+0000","last_became_active":"2024-03-06T07:35:14.380343+0000","last_became_peered":"2024-03-06T07:35:14.380343+0000","last_unstale":"2024-03-06T07:35:16.427018+0000","last_undegraded":"2024-03-06T07:35:16.427018+0000","last_fullsized":"2024-03-06T07:35:16.427018+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:34.863664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410166+0000","last_change":"2024-03-06T07:35:15.424865+0000","last_active":"2024-03-06T07:35:16.410166+0000","last_peered":"2024-03-06T07:35:16.410166+0000","last_clean":"2024-03-06T07:35:16.410166+0000","last_became_active":"2024-03-06T07:35:15.423157+0000","last_became_peered":"2024-03-06T07:35:15.423157+0000","last_unstale":"2024-03-06T07:35:16.410166+0000","last_undegraded":"2024-03-06T07:35:16.410166+0000","last_fullsized":"2024-03-06T07:35:16.410166+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:31.262420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009195+0000","last_change":"2024-03-06T07:35:14.380541+0000","last_active":"2024-03-06T07:35:18.009195+0000","last_peered":"2024-03-06T07:35:18.009195+0000","last_clean":"2024-03-06T07:35:18.009195+0000","last_became_active":"2024-03-06T07:35:14.380292+0000","last_became_peered":"2024-03-06T07:35:14.380292+0000","last_unstale":"2024-03-06T07:35:18.009195+0000","last_undegraded":"2024-03-06T07:35:18.009195+0000","last_fullsized":"2024-03-06T07:35:18.009195+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:47.863938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.230679+0000","last_change":"2024-03-06T07:35:15.432552+0000","last_active":"2024-03-06T07:35:18.230679+0000","last_peered":"2024-03-06T07:35:18.230679+0000","last_clean":"2024-03-06T07:35:18.230679+0000","last_became_active":"2024-03-06T07:35:15.432224+0000","last_became_peered":"2024-03-06T07:35:15.432224+0000","last_unstale":"2024-03-06T07:35:18.230679+0000","last_undegraded":"2024-03-06T07:35:18.230679+0000","last_fullsized":"2024-03-06T07:35:18.230679+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:54.513024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408301+0000","last_change":"2024-03-06T07:35:14.367332+0000","last_active":"2024-03-06T07:35:16.408301+0000","last_peered":"2024-03-06T07:35:16.408301+0000","last_clean":"2024-03-06T07:35:16.408301+0000","last_became_active":"2024-03-06T07:35:14.366557+0000","last_became_peered":"2024-03-06T07:35:14.366557+0000","last_unstale":"2024-03-06T07:35:16.408301+0000","last_undegraded":"2024-03-06T07:35:16.408301+0000","last_fullsized":"2024-03-06T07:35:16.408301+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:59.024999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.426681+0000","last_change":"2024-03-06T07:35:15.432268+0000","last_active":"2024-03-06T07:35:16.426681+0000","last_peered":"2024-03-06T07:35:16.426681+0000","last_clean":"2024-03-06T07:35:16.426681+0000","last_became_active":"2024-03-06T07:35:15.431550+0000","last_became_peered":"2024-03-06T07:35:15.431550+0000","last_unstale":"2024-03-06T07:35:16.426681+0000","last_undegraded":"2024-03-06T07:35:16.426681+0000","last_fullsized":"2024-03-06T07:35:16.426681+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:17.280222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231032+0000","last_change":"2024-03-06T07:35:14.394312+0000","last_active":"2024-03-06T07:35:18.231032+0000","last_peered":"2024-03-06T07:35:18.231032+0000","last_clean":"2024-03-06T07:35:18.231032+0000","last_became_active":"2024-03-06T07:35:14.393827+0000","last_became_peered":"2024-03-06T07:35:14.393827+0000","last_unstale":"2024-03-06T07:35:18.231032+0000","last_undegraded":"2024-03-06T07:35:18.231032+0000","last_fullsized":"2024-03-06T07:35:18.231032+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:42.520253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.427724+0000","last_change":"2024-03-06T07:35:15.427174+0000","last_active":"2024-03-06T07:35:16.427724+0000","last_peered":"2024-03-06T07:35:16.427724+0000","last_clean":"2024-03-06T07:35:16.427724+0000","last_became_active":"2024-03-06T07:35:15.426122+0000","last_became_peered":"2024-03-06T07:35:15.426122+0000","last_unstale":"2024-03-06T07:35:16.427724+0000","last_undegraded":"2024-03-06T07:35:16.427724+0000","last_fullsized":"2024-03-06T07:35:16.427724+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:13.661421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231645+0000","last_change":"2024-03-06T07:35:14.382199+0000","last_active":"2024-03-06T07:35:18.231645+0000","last_peered":"2024-03-06T07:35:18.231645+0000","last_clean":"2024-03-06T07:35:18.231645+0000","last_became_active":"2024-03-06T07:35:14.381918+0000","last_became_peered":"2024-03-06T07:35:14.381918+0000","last_unstale":"2024-03-06T07:35:18.231645+0000","last_undegraded":"2024-03-06T07:35:18.231645+0000","last_fullsized":"2024-03-06T07:35:18.231645+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:51.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.422813+0000","last_change":"2024-03-06T07:35:15.426656+0000","last_active":"2024-03-06T07:35:16.422813+0000","last_peered":"2024-03-06T07:35:16.422813+0000","last_clean":"2024-03-06T07:35:16.422813+0000","last_became_active":"2024-03-06T07:35:15.426357+0000","last_became_peered":"2024-03-06T07:35:15.426357+0000","last_unstale":"2024-03-06T07:35:16.422813+0000","last_undegraded":"2024-03-06T07:35:16.422813+0000","last_fullsized":"2024-03-06T07:35:16.422813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:48.679973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009470+0000","last_change":"2024-03-06T07:35:14.382538+0000","last_active":"2024-03-06T07:35:18.009470+0000","last_peered":"2024-03-06T07:35:18.009470+0000","last_clean":"2024-03-06T07:35:18.009470+0000","last_became_active":"2024-03-06T07:35:14.382216+0000","last_became_peered":"2024-03-06T07:35:14.382216+0000","last_unstale":"2024-03-06T07:35:18.009470+0000","last_undegraded":"2024-03-06T07:35:18.009470+0000","last_fullsized":"2024-03-06T07:35:18.009470+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:28.688041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009523+0000","last_change":"2024-03-06T07:35:15.418037+0000","last_active":"2024-03-06T07:35:18.009523+0000","last_peered":"2024-03-06T07:35:18.009523+0000","last_clean":"2024-03-06T07:35:18.009523+0000","last_became_active":"2024-03-06T07:35:15.417683+0000","last_became_peered":"2024-03-06T07:35:15.417683+0000","last_unstale":"2024-03-06T07:35:18.009523+0000","last_undegraded":"2024-03-06T07:35:18.009523+0000","last_fullsized":"2024-03-06T07:35:18.009523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:50.630525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"109'3","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233447+0000","last_change":"2024-03-06T07:35:14.388099+0000","last_active":"2024-03-06T07:35:18.233447+0000","last_peered":"2024-03-06T07:35:18.233447+0000","last_clean":"2024-03-06T07:35:18.233447+0000","last_became_active":"2024-03-06T07:35:14.386492+0000","last_became_peered":"2024-03-06T07:35:14.386492+0000","last_unstale":"2024-03-06T07:35:18.233447+0000","last_undegraded":"2024-03-06T07:35:18.233447+0000","last_fullsized":"2024-03-06T07:35:18.233447+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:45.975314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.427368+0000","last_change":"2024-03-06T07:35:15.433242+0000","last_active":"2024-03-06T07:35:16.427368+0000","last_peered":"2024-03-06T07:35:16.427368+0000","last_clean":"2024-03-06T07:35:16.427368+0000","last_became_active":"2024-03-06T07:35:15.432806+0000","last_became_peered":"2024-03-06T07:35:15.432806+0000","last_unstale":"2024-03-06T07:35:16.427368+0000","last_undegraded":"2024-03-06T07:35:16.427368+0000","last_fullsized":"2024-03-06T07:35:16.427368+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:29.924214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"109'6","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.234088+0000","last_change":"2024-03-06T07:35:14.392658+0000","last_active":"2024-03-06T07:35:18.234088+0000","last_peered":"2024-03-06T07:35:18.234088+0000","last_clean":"2024-03-06T07:35:18.234088+0000","last_became_active":"2024-03-06T07:35:14.392334+0000","last_became_peered":"2024-03-06T07:35:14.392334+0000","last_unstale":"2024-03-06T07:35:18.234088+0000","last_undegraded":"2024-03-06T07:35:18.234088+0000","last_fullsized":"2024-03-06T07:35:18.234088+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:31.671270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009731+0000","last_change":"2024-03-06T07:35:15.429165+0000","last_active":"2024-03-06T07:35:18.009731+0000","last_peered":"2024-03-06T07:35:18.009731+0000","last_clean":"2024-03-06T07:35:18.009731+0000","last_became_active":"2024-03-06T07:35:15.428736+0000","last_became_peered":"2024-03-06T07:35:15.428736+0000","last_unstale":"2024-03-06T07:35:18.009731+0000","last_undegraded":"2024-03-06T07:35:18.009731+0000","last_fullsized":"2024-03-06T07:35:18.009731+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:00.975501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743000+0000","last_change":"2024-03-06T07:35:14.376041+0000","last_active":"2024-03-06T07:35:17.743000+0000","last_peered":"2024-03-06T07:35:17.743000+0000","last_clean":"2024-03-06T07:35:17.743000+0000","last_became_active":"2024-03-06T07:35:14.375640+0000","last_became_peered":"2024-03-06T07:35:14.375640+0000","last_unstale":"2024-03-06T07:35:17.743000+0000","last_undegraded":"2024-03-06T07:35:17.743000+0000","last_fullsized":"2024-03-06T07:35:17.743000+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:34.413706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009929+0000","last_change":"2024-03-06T07:35:15.405488+0000","last_active":"2024-03-06T07:35:18.009929+0000","last_peered":"2024-03-06T07:35:18.009929+0000","last_clean":"2024-03-06T07:35:18.009929+0000","last_became_active":"2024-03-06T07:35:15.404519+0000","last_became_peered":"2024-03-06T07:35:15.404519+0000","last_unstale":"2024-03-06T07:35:18.009929+0000","last_undegraded":"2024-03-06T07:35:18.009929+0000","last_fullsized":"2024-03-06T07:35:18.009929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:39.003490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.421732+0000","last_change":"2024-03-06T07:35:14.386662+0000","last_active":"2024-03-06T07:35:16.421732+0000","last_peered":"2024-03-06T07:35:16.421732+0000","last_clean":"2024-03-06T07:35:16.421732+0000","last_became_active":"2024-03-06T07:35:14.385646+0000","last_became_peered":"2024-03-06T07:35:14.385646+0000","last_unstale":"2024-03-06T07:35:16.421732+0000","last_undegraded":"2024-03-06T07:35:16.421732+0000","last_fullsized":"2024-03-06T07:35:16.421732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:19.792255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009262+0000","last_change":"2024-03-06T07:35:15.418068+0000","last_active":"2024-03-06T07:35:18.009262+0000","last_peered":"2024-03-06T07:35:18.009262+0000","last_clean":"2024-03-06T07:35:18.009262+0000","last_became_active":"2024-03-06T07:35:15.417772+0000","last_became_peered":"2024-03-06T07:35:15.417772+0000","last_unstale":"2024-03-06T07:35:18.009262+0000","last_undegraded":"2024-03-06T07:35:18.009262+0000","last_fullsized":"2024-03-06T07:35:18.009262+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:34.226912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":22,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.230495+0000","last_change":"2024-03-06T07:35:14.390338+0000","last_active":"2024-03-06T07:35:18.230495+0000","last_peered":"2024-03-06T07:35:18.230495+0000","last_clean":"2024-03-06T07:35:18.230495+0000","last_became_active":"2024-03-06T07:35:14.390002+0000","last_became_peered":"2024-03-06T07:35:14.390002+0000","last_unstale":"2024-03-06T07:35:18.230495+0000","last_undegraded":"2024-03-06T07:35:18.230495+0000","last_fullsized":"2024-03-06T07:35:18.230495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:49.641734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231638+0000","last_change":"2024-03-06T07:35:15.425785+0000","last_active":"2024-03-06T07:35:18.231638+0000","last_peered":"2024-03-06T07:35:18.231638+0000","last_clean":"2024-03-06T07:35:18.231638+0000","last_became_active":"2024-03-06T07:35:15.424947+0000","last_became_peered":"2024-03-06T07:35:15.424947+0000","last_unstale":"2024-03-06T07:35:18.231638+0000","last_undegraded":"2024-03-06T07:35:18.231638+0000","last_fullsized":"2024-03-06T07:35:18.231638+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:21.466645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233910+0000","last_change":"2024-03-06T07:35:14.388645+0000","last_active":"2024-03-06T07:35:18.233910+0000","last_peered":"2024-03-06T07:35:18.233910+0000","last_clean":"2024-03-06T07:35:18.233910+0000","last_became_active":"2024-03-06T07:35:14.386018+0000","last_became_peered":"2024-03-06T07:35:14.386018+0000","last_unstale":"2024-03-06T07:35:18.233910+0000","last_undegraded":"2024-03-06T07:35:18.233910+0000","last_fullsized":"2024-03-06T07:35:18.233910+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:03.759805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742993+0000","last_change":"2024-03-06T07:35:15.426171+0000","last_active":"2024-03-06T07:35:17.742993+0000","last_peered":"2024-03-06T07:35:17.742993+0000","last_clean":"2024-03-06T07:35:17.742993+0000","last_became_active":"2024-03-06T07:35:15.425884+0000","last_became_peered":"2024-03-06T07:35:15.425884+0000","last_unstale":"2024-03-06T07:35:17.742993+0000","last_undegraded":"2024-03-06T07:35:17.742993+0000","last_fullsized":"2024-03-06T07:35:17.742993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:13.615988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231752+0000","last_change":"2024-03-06T07:35:14.385990+0000","last_active":"2024-03-06T07:35:18.231752+0000","last_peered":"2024-03-06T07:35:18.231752+0000","last_clean":"2024-03-06T07:35:18.231752+0000","last_became_active":"2024-03-06T07:35:14.385572+0000","last_became_peered":"2024-03-06T07:35:14.385572+0000","last_unstale":"2024-03-06T07:35:18.231752+0000","last_undegraded":"2024-03-06T07:35:18.231752+0000","last_fullsized":"2024-03-06T07:35:18.231752+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:09.529110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410748+0000","last_change":"2024-03-06T07:35:15.432874+0000","last_active":"2024-03-06T07:35:16.410748+0000","last_peered":"2024-03-06T07:35:16.410748+0000","last_clean":"2024-03-06T07:35:16.410748+0000","last_became_active":"2024-03-06T07:35:15.430564+0000","last_became_peered":"2024-03-06T07:35:15.430564+0000","last_unstale":"2024-03-06T07:35:16.410748+0000","last_undegraded":"2024-03-06T07:35:16.410748+0000","last_fullsized":"2024-03-06T07:35:16.410748+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:32.686301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.422104+0000","last_change":"2024-03-06T07:35:14.392030+0000","last_active":"2024-03-06T07:35:16.422104+0000","last_peered":"2024-03-06T07:35:16.422104+0000","last_clean":"2024-03-06T07:35:16.422104+0000","last_became_active":"2024-03-06T07:35:14.391707+0000","last_became_peered":"2024-03-06T07:35:14.391707+0000","last_unstale":"2024-03-06T07:35:16.422104+0000","last_undegraded":"2024-03-06T07:35:16.422104+0000","last_fullsized":"2024-03-06T07:35:16.422104+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:51.357012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.411603+0000","last_change":"2024-03-06T07:35:15.428291+0000","last_active":"2024-03-06T07:35:16.411603+0000","last_peered":"2024-03-06T07:35:16.411603+0000","last_clean":"2024-03-06T07:35:16.411603+0000","last_became_active":"2024-03-06T07:35:15.427968+0000","last_became_peered":"2024-03-06T07:35:15.427968+0000","last_unstale":"2024-03-06T07:35:16.411603+0000","last_undegraded":"2024-03-06T07:35:16.411603+0000","last_fullsized":"2024-03-06T07:35:16.411603+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:00.975501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.426806+0000","last_change":"2024-03-06T07:35:14.389869+0000","last_active":"2024-03-06T07:35:16.426806+0000","last_peered":"2024-03-06T07:35:16.426806+0000","last_clean":"2024-03-06T07:35:16.426806+0000","last_became_active":"2024-03-06T07:35:14.388816+0000","last_became_peered":"2024-03-06T07:35:14.388816+0000","last_unstale":"2024-03-06T07:35:16.426806+0000","last_undegraded":"2024-03-06T07:35:16.426806+0000","last_fullsized":"2024-03-06T07:35:16.426806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:49.082414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408830+0000","last_change":"2024-03-06T07:35:15.429802+0000","last_active":"2024-03-06T07:35:16.408830+0000","last_peered":"2024-03-06T07:35:16.408830+0000","last_clean":"2024-03-06T07:35:16.408830+0000","last_became_active":"2024-03-06T07:35:15.429481+0000","last_became_peered":"2024-03-06T07:35:15.429481+0000","last_unstale":"2024-03-06T07:35:16.408830+0000","last_undegraded":"2024-03-06T07:35:16.408830+0000","last_fullsized":"2024-03-06T07:35:16.408830+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:14.792832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.743151+0000","last_change":"2024-03-06T07:35:14.378202+0000","last_active":"2024-03-06T07:35:17.743151+0000","last_peered":"2024-03-06T07:35:17.743151+0000","last_clean":"2024-03-06T07:35:17.743151+0000","last_became_active":"2024-03-06T07:35:14.377306+0000","last_became_peered":"2024-03-06T07:35:14.377306+0000","last_unstale":"2024-03-06T07:35:17.743151+0000","last_undegraded":"2024-03-06T07:35:17.743151+0000","last_fullsized":"2024-03-06T07:35:17.743151+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:09.529110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.232200+0000","last_change":"2024-03-06T07:35:15.429386+0000","last_active":"2024-03-06T07:35:18.232200+0000","last_peered":"2024-03-06T07:35:18.232200+0000","last_clean":"2024-03-06T07:35:18.232200+0000","last_became_active":"2024-03-06T07:35:15.428481+0000","last_became_peered":"2024-03-06T07:35:15.428481+0000","last_unstale":"2024-03-06T07:35:18.232200+0000","last_undegraded":"2024-03-06T07:35:18.232200+0000","last_fullsized":"2024-03-06T07:35:18.232200+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:28.848943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.232394+0000","last_change":"2024-03-06T07:35:14.383997+0000","last_active":"2024-03-06T07:35:18.232394+0000","last_peered":"2024-03-06T07:35:18.232394+0000","last_clean":"2024-03-06T07:35:18.232394+0000","last_became_active":"2024-03-06T07:35:14.383712+0000","last_became_peered":"2024-03-06T07:35:14.383712+0000","last_unstale":"2024-03-06T07:35:18.232394+0000","last_undegraded":"2024-03-06T07:35:18.232394+0000","last_fullsized":"2024-03-06T07:35:18.232394+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:34.413706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410722+0000","last_change":"2024-03-06T07:35:15.424518+0000","last_active":"2024-03-06T07:35:16.410722+0000","last_peered":"2024-03-06T07:35:16.410722+0000","last_clean":"2024-03-06T07:35:16.410722+0000","last_became_active":"2024-03-06T07:35:15.421577+0000","last_became_peered":"2024-03-06T07:35:15.421577+0000","last_unstale":"2024-03-06T07:35:16.410722+0000","last_undegraded":"2024-03-06T07:35:16.410722+0000","last_fullsized":"2024-03-06T07:35:16.410722+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:18.629606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:17.742487+0000","last_change":"2024-03-06T07:35:14.380931+0000","last_active":"2024-03-06T07:35:17.742487+0000","last_peered":"2024-03-06T07:35:17.742487+0000","last_clean":"2024-03-06T07:35:17.742487+0000","last_became_active":"2024-03-06T07:35:14.378613+0000","last_became_peered":"2024-03-06T07:35:14.378613+0000","last_unstale":"2024-03-06T07:35:17.742487+0000","last_undegraded":"2024-03-06T07:35:17.742487+0000","last_fullsized":"2024-03-06T07:35:17.742487+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:05.506109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009643+0000","last_change":"2024-03-06T07:35:15.425805+0000","last_active":"2024-03-06T07:35:18.009643+0000","last_peered":"2024-03-06T07:35:18.009643+0000","last_clean":"2024-03-06T07:35:18.009643+0000","last_became_active":"2024-03-06T07:35:15.425475+0000","last_became_peered":"2024-03-06T07:35:15.425475+0000","last_unstale":"2024-03-06T07:35:18.009643+0000","last_undegraded":"2024-03-06T07:35:18.009643+0000","last_fullsized":"2024-03-06T07:35:18.009643+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:02.265621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231135+0000","last_change":"2024-03-06T07:35:14.394083+0000","last_active":"2024-03-06T07:35:18.231135+0000","last_peered":"2024-03-06T07:35:18.231135+0000","last_clean":"2024-03-06T07:35:18.231135+0000","last_became_active":"2024-03-06T07:35:14.393438+0000","last_became_peered":"2024-03-06T07:35:14.393438+0000","last_unstale":"2024-03-06T07:35:18.231135+0000","last_undegraded":"2024-03-06T07:35:18.231135+0000","last_fullsized":"2024-03-06T07:35:18.231135+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:32.516170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.422055+0000","last_change":"2024-03-06T07:35:15.427484+0000","last_active":"2024-03-06T07:35:16.422055+0000","last_peered":"2024-03-06T07:35:16.422055+0000","last_clean":"2024-03-06T07:35:16.422055+0000","last_became_active":"2024-03-06T07:35:15.425939+0000","last_became_peered":"2024-03-06T07:35:15.425939+0000","last_unstale":"2024-03-06T07:35:16.422055+0000","last_undegraded":"2024-03-06T07:35:16.422055+0000","last_fullsized":"2024-03-06T07:35:16.422055+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:48.878969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009272+0000","last_change":"2024-03-06T07:35:14.390005+0000","last_active":"2024-03-06T07:35:18.009272+0000","last_peered":"2024-03-06T07:35:18.009272+0000","last_clean":"2024-03-06T07:35:18.009272+0000","last_became_active":"2024-03-06T07:35:14.389726+0000","last_became_peered":"2024-03-06T07:35:14.389726+0000","last_unstale":"2024-03-06T07:35:18.009272+0000","last_undegraded":"2024-03-06T07:35:18.009272+0000","last_fullsized":"2024-03-06T07:35:18.009272+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:47.664942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233610+0000","last_change":"2024-03-06T07:35:15.429614+0000","last_active":"2024-03-06T07:35:18.233610+0000","last_peered":"2024-03-06T07:35:18.233610+0000","last_clean":"2024-03-06T07:35:18.233610+0000","last_became_active":"2024-03-06T07:35:15.428300+0000","last_became_peered":"2024-03-06T07:35:15.428300+0000","last_unstale":"2024-03-06T07:35:18.233610+0000","last_undegraded":"2024-03-06T07:35:18.233610+0000","last_fullsized":"2024-03-06T07:35:18.233610+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:40.766121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.424903+0000","last_change":"2024-03-06T07:35:14.382441+0000","last_active":"2024-03-06T07:35:16.424903+0000","last_peered":"2024-03-06T07:35:16.424903+0000","last_clean":"2024-03-06T07:35:16.424903+0000","last_became_active":"2024-03-06T07:35:14.382136+0000","last_became_peered":"2024-03-06T07:35:14.382136+0000","last_unstale":"2024-03-06T07:35:16.424903+0000","last_undegraded":"2024-03-06T07:35:16.424903+0000","last_fullsized":"2024-03-06T07:35:16.424903+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:46.540124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.234050+0000","last_change":"2024-03-06T07:35:15.422306+0000","last_active":"2024-03-06T07:35:18.234050+0000","last_peered":"2024-03-06T07:35:18.234050+0000","last_clean":"2024-03-06T07:35:18.234050+0000","last_became_active":"2024-03-06T07:35:15.421318+0000","last_became_peered":"2024-03-06T07:35:15.421318+0000","last_unstale":"2024-03-06T07:35:18.234050+0000","last_undegraded":"2024-03-06T07:35:18.234050+0000","last_fullsized":"2024-03-06T07:35:18.234050+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:13.412432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.233332+0000","last_change":"2024-03-06T07:35:14.383738+0000","last_active":"2024-03-06T07:35:18.233332+0000","last_peered":"2024-03-06T07:35:18.233332+0000","last_clean":"2024-03-06T07:35:18.233332+0000","last_became_active":"2024-03-06T07:35:14.383433+0000","last_became_peered":"2024-03-06T07:35:14.383433+0000","last_unstale":"2024-03-06T07:35:18.233332+0000","last_undegraded":"2024-03-06T07:35:18.233332+0000","last_fullsized":"2024-03-06T07:35:18.233332+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:57.350279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.421909+0000","last_change":"2024-03-06T07:35:15.425540+0000","last_active":"2024-03-06T07:35:16.421909+0000","last_peered":"2024-03-06T07:35:16.421909+0000","last_clean":"2024-03-06T07:35:16.421909+0000","last_became_active":"2024-03-06T07:35:15.425215+0000","last_became_peered":"2024-03-06T07:35:15.425215+0000","last_unstale":"2024-03-06T07:35:16.421909+0000","last_undegraded":"2024-03-06T07:35:16.421909+0000","last_fullsized":"2024-03-06T07:35:16.421909+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009586+0000","last_change":"2024-03-06T07:35:14.381592+0000","last_active":"2024-03-06T07:35:18.009586+0000","last_peered":"2024-03-06T07:35:18.009586+0000","last_clean":"2024-03-06T07:35:18.009586+0000","last_became_active":"2024-03-06T07:35:14.381330+0000","last_became_peered":"2024-03-06T07:35:14.381330+0000","last_unstale":"2024-03-06T07:35:18.009586+0000","last_undegraded":"2024-03-06T07:35:18.009586+0000","last_fullsized":"2024-03-06T07:35:18.009586+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:13.999882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408903+0000","last_change":"2024-03-06T07:35:15.431672+0000","last_active":"2024-03-06T07:35:16.408903+0000","last_peered":"2024-03-06T07:35:16.408903+0000","last_clean":"2024-03-06T07:35:16.408903+0000","last_became_active":"2024-03-06T07:35:15.430785+0000","last_became_peered":"2024-03-06T07:35:15.430785+0000","last_unstale":"2024-03-06T07:35:16.408903+0000","last_undegraded":"2024-03-06T07:35:16.408903+0000","last_fullsized":"2024-03-06T07:35:16.408903+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:57.000433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.426950+0000","last_change":"2024-03-06T07:35:14.390542+0000","last_active":"2024-03-06T07:35:16.426950+0000","last_peered":"2024-03-06T07:35:16.426950+0000","last_clean":"2024-03-06T07:35:16.426950+0000","last_became_active":"2024-03-06T07:35:14.389894+0000","last_became_peered":"2024-03-06T07:35:14.389894+0000","last_unstale":"2024-03-06T07:35:16.426950+0000","last_undegraded":"2024-03-06T07:35:16.426950+0000","last_fullsized":"2024-03-06T07:35:16.426950+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:15.494691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.408907+0000","last_change":"2024-03-06T07:35:15.422351+0000","last_active":"2024-03-06T07:35:16.408907+0000","last_peered":"2024-03-06T07:35:16.408907+0000","last_clean":"2024-03-06T07:35:16.408907+0000","last_became_active":"2024-03-06T07:35:15.421906+0000","last_became_peered":"2024-03-06T07:35:15.421906+0000","last_unstale":"2024-03-06T07:35:16.408907+0000","last_undegraded":"2024-03-06T07:35:16.408907+0000","last_fullsized":"2024-03-06T07:35:16.408907+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:16.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.409847+0000","last_change":"2024-03-06T07:35:14.377622+0000","last_active":"2024-03-06T07:35:16.409847+0000","last_peered":"2024-03-06T07:35:16.409847+0000","last_clean":"2024-03-06T07:35:16.409847+0000","last_became_active":"2024-03-06T07:35:14.376896+0000","last_became_peered":"2024-03-06T07:35:14.376896+0000","last_unstale":"2024-03-06T07:35:16.409847+0000","last_undegraded":"2024-03-06T07:35:16.409847+0000","last_fullsized":"2024-03-06T07:35:16.409847+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:38.163132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410884+0000","last_change":"2024-03-06T07:35:15.419644+0000","last_active":"2024-03-06T07:35:16.410884+0000","last_peered":"2024-03-06T07:35:16.410884+0000","last_clean":"2024-03-06T07:35:16.410884+0000","last_became_active":"2024-03-06T07:35:15.419264+0000","last_became_peered":"2024-03-06T07:35:15.419264+0000","last_unstale":"2024-03-06T07:35:16.410884+0000","last_undegraded":"2024-03-06T07:35:16.410884+0000","last_fullsized":"2024-03-06T07:35:16.410884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:34.226912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.230999+0000","last_change":"2024-03-06T07:35:14.384805+0000","last_active":"2024-03-06T07:35:18.230999+0000","last_peered":"2024-03-06T07:35:18.230999+0000","last_clean":"2024-03-06T07:35:18.230999+0000","last_became_active":"2024-03-06T07:35:14.382612+0000","last_became_peered":"2024-03-06T07:35:14.382612+0000","last_unstale":"2024-03-06T07:35:18.230999+0000","last_undegraded":"2024-03-06T07:35:18.230999+0000","last_fullsized":"2024-03-06T07:35:18.230999+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:50.122865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231739+0000","last_change":"2024-03-06T07:35:15.425207+0000","last_active":"2024-03-06T07:35:18.231739+0000","last_peered":"2024-03-06T07:35:18.231739+0000","last_clean":"2024-03-06T07:35:18.231739+0000","last_became_active":"2024-03-06T07:35:15.424930+0000","last_became_peered":"2024-03-06T07:35:15.424930+0000","last_unstale":"2024-03-06T07:35:18.231739+0000","last_undegraded":"2024-03-06T07:35:18.231739+0000","last_fullsized":"2024-03-06T07:35:18.231739+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:49.765691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.231276+0000","last_change":"2024-03-06T07:35:14.393175+0000","last_active":"2024-03-06T07:35:18.231276+0000","last_peered":"2024-03-06T07:35:18.231276+0000","last_clean":"2024-03-06T07:35:18.231276+0000","last_became_active":"2024-03-06T07:35:14.392390+0000","last_became_peered":"2024-03-06T07:35:14.392390+0000","last_unstale":"2024-03-06T07:35:18.231276+0000","last_undegraded":"2024-03-06T07:35:18.231276+0000","last_fullsized":"2024-03-06T07:35:18.231276+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:13.777801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:18.009420+0000","last_change":"2024-03-06T07:35:15.406010+0000","last_active":"2024-03-06T07:35:18.009420+0000","last_peered":"2024-03-06T07:35:18.009420+0000","last_clean":"2024-03-06T07:35:18.009420+0000","last_became_active":"2024-03-06T07:35:15.404224+0000","last_became_peered":"2024-03-06T07:35:15.404224+0000","last_unstale":"2024-03-06T07:35:18.009420+0000","last_undegraded":"2024-03-06T07:35:18.009420+0000","last_fullsized":"2024-03-06T07:35:18.009420+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:14.344054+0000","last_clean_scrub_stamp":"2024-03-06T07:35:14.344054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:59.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.427012+0000","last_change":"2024-03-06T07:35:14.390390+0000","last_active":"2024-03-06T07:35:16.427012+0000","last_peered":"2024-03-06T07:35:16.427012+0000","last_clean":"2024-03-06T07:35:16.427012+0000","last_became_active":"2024-03-06T07:35:14.389884+0000","last_became_peered":"2024-03-06T07:35:14.389884+0000","last_unstale":"2024-03-06T07:35:16.427012+0000","last_undegraded":"2024-03-06T07:35:16.427012+0000","last_fullsized":"2024-03-06T07:35:16.427012+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:13.329023+0000","last_clean_scrub_stamp":"2024-03-06T07:35:13.329023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:05.207559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":339,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:35:16.410907+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:35:16.410907+0000","last_peered":"2024-03-06T07:35:16.410907+0000","last_clean":"2024-03-06T07:35:16.410907+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:35:16.410907+0000","last_undegraded":"2024-03-06T07:35:16.410907+0000","last_fullsized":"2024-03-06T07:35:16.410907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1970176,"data_stored":1180693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574929,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":1372160,"data_stored":589779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36684,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335156,"statfs":{"total":96636764160,"available":96599199744,"internally_reserved":0,"allocated":1376256,"data_stored":590313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1368064,"data_stored":589709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40848,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330992,"statfs":{"total":96636764160,"available":96594935808,"internally_reserved":0,"allocated":1372160,"data_stored":585091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1355776,"data_stored":583409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":1363968,"data_stored":583533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574928,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41424,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330416,"statfs":{"total":96636764160,"available":96594345984,"internally_reserved":0,"allocated":1970176,"data_stored":1175501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:20.771 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:35:20.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:35:21.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.044+0000 7f7053e00640 1 Processor -- start 2024-03-06T07:35:21.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.044+0000 7f7053e00640 1 -- start start 2024-03-06T07:35:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70540cef80 0x7f70540ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70540cd260 0x7f705406d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70540b4ae0 con 0x7f7054145570 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540cf400 con 0x7f70540cd260 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705406c9f0 con 0x7f70540cef80 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70540cd260 0x7f705406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7052a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7054145570 0x7f70540b42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48794/0 (socket says 172.21.15.32:48794) 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7052a00640 1 -- 172.21.15.32:0/1325828925 learned_addr learned my addr 172.21.15.32:0/1325828925 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7053400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70540cd260 0x7f705406d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50340/0 (socket says 172.21.15.32:50340) 2024-03-06T07:35:21.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.046+0000 7f7052000640 1 --2- 172.21.15.32:0/1325828925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70540cef80 0x7f70540ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.047+0000 7f7053400640 1 -- 172.21.15.32:0/1325828925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70540cef80 msgr2=0x7f70540ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.047+0000 7f7053400640 1 --2- 172.21.15.32:0/1325828925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70540cef80 0x7f70540ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.047+0000 7f7053400640 1 -- 172.21.15.32:0/1325828925 >> v1:172.21.15.32:6789/0 conn(0x7f7054145570 legacy=0x7f70540b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.047+0000 7f7053400640 1 -- 172.21.15.32:0/1325828925 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540b4dc0 con 0x7f70540cd260 2024-03-06T07:35:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.047+0000 7f7051600640 1 -- 172.21.15.32:0/1325828925 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638837846 0 0) 0x7f70540b4ae0 con 0x7f7054145570 2024-03-06T07:35:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.048+0000 7f7053400640 1 --2- 172.21.15.32:0/1325828925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cd260 0x7f705406d440 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f704c009d90 tx=0x7f704c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163f4f84d3fe63eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.048+0000 7f7051600640 1 -- 172.21.15.32:0/1325828925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c039070 con 0x7f70540cd260 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7051600640 1 -- 172.21.15.32:0/1325828925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c02cc50 con 0x7f70540cd260 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7051600640 1 -- 172.21.15.32:0/1325828925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c034440 con 0x7f70540cd260 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 -- 172.21.15.32:0/1325828925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cd260 msgr2=0x7f705406d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 --2- 172.21.15.32:0/1325828925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cd260 0x7f705406d440 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f704c009d90 tx=0x7f704c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 -- 172.21.15.32:0/1325828925 shutdown_connections 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 --2- 172.21.15.32:0/1325828925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cd260 0x7f705406d440 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 --2- 172.21.15.32:0/1325828925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70540cef80 0x7f70540ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 -- 172.21.15.32:0/1325828925 >> 172.21.15.32:0/1325828925 conn(0x7f705406b190 msgr2=0x7f705406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.049+0000 7f7053e00640 1 -- 172.21.15.32:0/1325828925 shutdown_connections 2024-03-06T07:35:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.050+0000 7f7053e00640 1 -- 172.21.15.32:0/1325828925 wait complete. 2024-03-06T07:35:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.051+0000 7f7053e00640 1 Processor -- start 2024-03-06T07:35:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.051+0000 7f7053e00640 1 -- start start 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.052+0000 7f7053e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70540cd260 0x7f7054160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.052+0000 7f7053e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 0x7f70541f8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.052+0000 7f7053e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7054145570 0x7f70541fb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.052+0000 7f7053e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705415c110 con 0x7f70540cef80 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.052+0000 7f7053e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705406cfb0 con 0x7f70540cd260 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.052+0000 7f7053e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705406d980 con 0x7f7054145570 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 0x7f70541f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 0x7f70541f8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50354/0 (socket says 172.21.15.32:50354) 2024-03-06T07:35:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 -- 172.21.15.32:0/3304781699 learned_addr learned my addr 172.21.15.32:0/3304781699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052a00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70540cd260 0x7f7054160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7053400640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7054145570 0x7f70541fb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 -- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7054145570 msgr2=0x7f70541fb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7054145570 0x7f70541fb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 -- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70540cd260 msgr2=0x7f7054160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70540cd260 0x7f7054160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052000640 1 -- 172.21.15.32:0/3304781699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70541f7bb0 con 0x7f70540cef80 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.053+0000 7f7052a00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70540cd260 0x7f7054160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.054+0000 7f7052000640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 0x7f70541f8630 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f704000d800 tx=0x7f704000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.054+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704000b6c0 con 0x7f70540cef80 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.054+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7040019070 con 0x7f70540cef80 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.054+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704000ba80 con 0x7f70540cef80 2024-03-06T07:35:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.054+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70541f8b70 con 0x7f70540cef80 2024-03-06T07:35:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.054+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70541fbf20 con 0x7f70540cef80 2024-03-06T07:35:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.055+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705404f330 con 0x7f70540cef80 2024-03-06T07:35:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.056+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f704000bc20 con 0x7f70540cef80 2024-03-06T07:35:21.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.057+0000 7f704aa00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f703406b4d0 0x7f703406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.057+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f704008a640 con 0x7f70540cef80 2024-03-06T07:35:21.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.057+0000 7f7052a00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f703406b4d0 0x7f703406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.058+0000 7f7052a00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f703406b4d0 0x7f703406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7044000f60 tx=0x7f7044009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.062+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70541f8b70 con 0x7f70540cef80 2024-03-06T07:35:21.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.340+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f70540b3310 con 0x7f70540cef80 2024-03-06T07:35:21.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.345+0000 7f704aa00640 1 -- 172.21.15.32:0/3304781699 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f70541f7bb0 con 0x7f70540cef80 2024-03-06T07:35:21.347 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:35:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:35:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:35:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:35:21.347 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:21.347 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:35:21.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f703406b4d0 msgr2=0x7f703406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f703406b4d0 0x7f703406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7044000f60 tx=0x7f7044009040 comp rx=0 tx=0).stop 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 msgr2=0x7f70541f8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 0x7f70541f8630 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f704000d800 tx=0x7f704000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 shutdown_connections 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f703406b4d0 0x7f703406d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7054145570 0x7f70541fb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70540cef80 0x7f70541f8630 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 --2- 172.21.15.32:0/3304781699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70540cd260 0x7f7054160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 >> 172.21.15.32:0/3304781699 conn(0x7f705406b190 msgr2=0x7f705406df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 shutdown_connections 2024-03-06T07:35:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.348+0000 7f7053e00640 1 -- 172.21.15.32:0/3304781699 wait complete. 2024-03-06T07:35:21.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.590+0000 7fc5a8a00640 1 Processor -- start 2024-03-06T07:35:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.591+0000 7fc5a8a00640 1 -- start start 2024-03-06T07:35:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.592+0000 7fc5a8a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 0x7fc5a415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.592+0000 7fc5a8a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a415cc40 con 0x7fc5a40af580 2024-03-06T07:35:21.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.592+0000 7fc5a8a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a415cf20 con 0x7fc5a40cdc20 2024-03-06T07:35:21.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.592+0000 7fc5a8a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a415d200 con 0x7fc5a40af1a0 2024-03-06T07:35:21.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.593+0000 7fc5a1400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc5a40af580 0x7fc5a40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48798/0 (socket says 172.21.15.32:48798) 2024-03-06T07:35:21.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.593+0000 7fc5a1400640 1 -- 172.21.15.32:0/3368717029 learned_addr learned my addr 172.21.15.32:0/3368717029 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:21.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.593+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 0x7fc5a415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.593+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298487654 0 0) 0x7fc5a415cc40 con 0x7fc5a40af580 2024-03-06T07:35:21.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.593+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc588003520 con 0x7fc5a40af580 2024-03-06T07:35:21.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2959480189 0 0) 0x7fc588003520 con 0x7fc5a40af580 2024-03-06T07:35:21.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 msgr2=0x7fc5a415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 0x7fc5a415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 >> v1:172.21.15.39:6789/0 conn(0x7fc5a40cdc20 legacy=0x7fc5a40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a415d4e0 con 0x7fc5a40af580 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fc590001670 con 0x7fc5a40af580 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 >> v1:172.21.15.32:6789/0 conn(0x7fc5a40af580 legacy=0x7fc5a40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc588003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc588004800 0x7fc588004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 0x7fc588005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a415d200 con 0x7fc588003a50 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58800bb80 con 0x7fc588004800 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58800bd80 con 0x7fc5a40af1a0 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.594+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc588003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1400640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc588004800 0x7fc588004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 msgr2=0x7fc588005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a40af1a0 0x7fc588005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc588004800 msgr2=0x7fc588004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc588004800 0x7fc588004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3368717029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588003520 con 0x7fc588003a50 2024-03-06T07:35:21.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.595+0000 7fc5a1400640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc588004800 0x7fc588004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.596+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc588003e70 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fc594009a90 tx=0x7fc588005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66631a824f37e0d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.596+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594039070 con 0x7fc588003a50 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.596+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59402c6b0 con 0x7fc588003a50 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.596+0000 7fc5a0a00640 1 -- 172.21.15.32:0/3368717029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594009690 con 0x7fc588003a50 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3368717029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 msgr2=0x7fc588003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3368717029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc588003e70 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fc594009a90 tx=0x7fc588005830 comp rx=0 tx=0).stop 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a2800640 1 -- 172.21.15.32:0/3368717029 reap_dead start 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3368717029 shutdown_connections 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3368717029 >> 172.21.15.32:0/3368717029 conn(0x7fc5a406afb0 msgr2=0x7fc5a406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3368717029 shutdown_connections 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.597+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3368717029 wait complete. 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.599+0000 7fc5a8a00640 1 Processor -- start 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.599+0000 7fc5a8a00640 1 -- start start 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc5a409d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a406ce30 0x7fc5a406e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a8a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a414f660 0x7fc5a40ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a8a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a415c600 con 0x7fc588003a50 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a40b36f0 con 0x7fc5a406ce30 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a8a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a415c780 con 0x7fc5a414f660 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc5a409d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc5a409d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50364/0 (socket says 172.21.15.32:50364) 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3323892716 learned_addr learned my addr 172.21.15.32:0/3323892716 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a2800640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a414f660 0x7fc5a40ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a414f660 msgr2=0x7fc5a40ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1400640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a406ce30 0x7fc5a406e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a414f660 0x7fc5a40ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a406ce30 msgr2=0x7fc5a406e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a406ce30 0x7fc5a406e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.600+0000 7fc5a1e00640 1 -- 172.21.15.32:0/3323892716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a409ac40 con 0x7fc588003a50 2024-03-06T07:35:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.601+0000 7fc5a1e00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc5a409d530 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fc594009bb0 tx=0x7fc594004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.601+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594042020 con 0x7fc588003a50 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.601+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59403a040 con 0x7fc588003a50 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.601+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a406d4c0 con 0x7fc588003a50 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.601+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594039070 con 0x7fc588003a50 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.601+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a406da80 con 0x7fc588003a50 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.602+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a4069510 con 0x7fc588003a50 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.603+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc594045020 con 0x7fc588003a50 2024-03-06T07:35:21.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.604+0000 7fc59a000640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc58406b4d0 0x7fc58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.604+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5940ab6a0 con 0x7fc588003a50 2024-03-06T07:35:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.604+0000 7fc5a1400640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc58406b4d0 0x7fc58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.605+0000 7fc5a1400640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc58406b4d0 0x7fc58406d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc5a415b570 tx=0x7fc590001b60 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.608+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc59404c070 con 0x7fc588003a50 2024-03-06T07:35:21.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.851+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5a404f3a0 con 0x7fc588003a50 2024-03-06T07:35:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.852+0000 7fc59a000640 1 -- 172.21.15.32:0/3323892716 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fc59407a3f0 con 0x7fc588003a50 2024-03-06T07:35:21.853 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:21.853 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3906463088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3906463088},{"type":"v1","addr":"172.21.15.39:6834","nonce":3906463088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7647,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/2563322789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2563322789},{"type":"v1","addr":"172.21.15.32:6836","nonce":2563322789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7650,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2154839965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":2154839965},{"type":"v1","addr":"172.21.15.32:6837","nonce":2154839965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:18.287992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7665":{"gid":7665,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/14889266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":14889266},{"type":"v1","addr":"172.21.15.39:6835","nonce":14889266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7665,"qdb_cluster":[7665]},"id":8}]} 2024-03-06T07:35:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-03-06T07:35:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.854+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc58406b4d0 msgr2=0x7fc58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.854+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc58406b4d0 0x7fc58406d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc5a415b570 tx=0x7fc590001b60 comp rx=0 tx=0).stop 2024-03-06T07:35:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 msgr2=0x7fc5a409d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc5a409d530 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fc594009bb0 tx=0x7fc594004730 comp rx=0 tx=0).stop 2024-03-06T07:35:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 shutdown_connections 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc58406b4d0 0x7fc58406d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc5a414f660 0x7fc5a40ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a406ce30 0x7fc5a406e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 --2- 172.21.15.32:0/3323892716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc588003a50 0x7fc5a409d530 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 >> 172.21.15.32:0/3323892716 conn(0x7fc5a406afb0 msgr2=0x7fc5a406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 shutdown_connections 2024-03-06T07:35:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:21.855+0000 7fc5a8a00640 1 -- 172.21.15.32:0/3323892716 wait complete. 2024-03-06T07:35:22.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:14.858000+0000', 'modified': '2024-03-06T07:35:18.287992+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7665': {'gid': 7665, 'name': 'd', 'rank': 0, 'incarnation': 72, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/14889266', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6833', 'nonce': 14889266}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 14889266}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7665, 'qdb_cluster': [7665]} 2024-03-06T07:35:22.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:35:22.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:35:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.510+0000 7fef07e00640 1 Processor -- start 2024-03-06T07:35:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.510+0000 7fef07e00640 1 -- start start 2024-03-06T07:35:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.512+0000 7fef07e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fef08145300 0x7fef080b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.512+0000 7fef07e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080714d0 con 0x7fef08070f10 2024-03-06T07:35:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.512+0000 7fef07e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080717b0 con 0x7fef080cd060 2024-03-06T07:35:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.512+0000 7fef07e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef08071a90 con 0x7fef08145300 2024-03-06T07:35:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.512+0000 7feefe000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fef08070f10 0x7fef080cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48804/0 (socket says 172.21.15.32:48804) 2024-03-06T07:35:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.512+0000 7feefe000640 1 -- 172.21.15.32:0/4213006838 learned_addr learned my addr 172.21.15.32:0/4213006838 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06a00640 1 --2- 172.21.15.32:0/4213006838 >> v2:172.21.15.39:3300/0 conn(0x7fef08145300 0x7fef080b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445911800 0 0) 0x7fef080714d0 con 0x7fef08070f10 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec003520 con 0x7fef08070f10 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911822009 0 0) 0x7feeec003520 con 0x7fef08070f10 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 >> v2:172.21.15.39:3300/0 conn(0x7fef08145300 msgr2=0x7fef080b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 --2- 172.21.15.32:0/4213006838 >> v2:172.21.15.39:3300/0 conn(0x7fef08145300 0x7fef080b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 >> v1:172.21.15.39:6789/0 conn(0x7fef080cd060 legacy=0x7fef0815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef08071d70 con 0x7fef08070f10 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7feef0001670 con 0x7fef08070f10 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 >> v1:172.21.15.32:6789/0 conn(0x7fef08070f10 legacy=0x7fef080cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec0039b0 0x7feeec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feeec004800 0x7feeec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feeec0051c0 0x7feeec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef08071a90 con 0x7feeec004800 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec00c160 con 0x7feeec0039b0 2024-03-06T07:35:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.513+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeec00c360 con 0x7feeec0051c0 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7feefe000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feeec004800 0x7feeec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7fef06a00640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec0039b0 0x7feeec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7feefe000640 1 -- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feeec0051c0 msgr2=0x7feeec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7feefe000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feeec0051c0 0x7feeec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7feefe000640 1 -- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec0039b0 msgr2=0x7feeec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7feefe000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec0039b0 0x7feeec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7feefe000640 1 -- 172.21.15.32:0/4213006838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec003520 con 0x7feeec004800 2024-03-06T07:35:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.514+0000 7fef07400640 1 -- 172.21.15.32:0/4213006838 reap_dead start 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.515+0000 7feefe000640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feeec004800 0x7feeec004c80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7feef000c240 tx=0x7feeec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5f589f33eba786 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0007d10 con 0x7feeec004800 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0030070 con 0x7feeec004800 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef06000640 1 -- 172.21.15.32:0/4213006838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef002cda0 con 0x7feeec004800 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 -- 172.21.15.32:0/4213006838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feeec004800 msgr2=0x7feeec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feeec004800 0x7feeec004c80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7feef000c240 tx=0x7feeec005830 comp rx=0 tx=0).stop 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 -- 172.21.15.32:0/4213006838 shutdown_connections 2024-03-06T07:35:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 --2- 172.21.15.32:0/4213006838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feeec004800 0x7feeec004c80 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 -- 172.21.15.32:0/4213006838 >> 172.21.15.32:0/4213006838 conn(0x7fef0806b3d0 msgr2=0x7fef0806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 -- 172.21.15.32:0/4213006838 shutdown_connections 2024-03-06T07:35:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.516+0000 7fef07e00640 1 -- 172.21.15.32:0/4213006838 wait complete. 2024-03-06T07:35:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.518+0000 7fef07e00640 1 Processor -- start 2024-03-06T07:35:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.518+0000 7fef07e00640 1 -- start start 2024-03-06T07:35:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.519+0000 7fef07e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec004800 0x7fef080ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.519+0000 7fef07e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 0x7fef080aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.519+0000 7fef07e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fef0815b2c0 0x7fef0815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.519+0000 7fef07e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef080ccad0 con 0x7fef0806c7c0 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.519+0000 7fef07e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef080cc950 con 0x7feeec004800 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.519+0000 7fef07e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef080ccc50 con 0x7fef0815b2c0 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 0x7fef080aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 0x7fef080aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50388/0 (socket says 172.21.15.32:50388) 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 -- 172.21.15.32:0/2083171216 learned_addr learned my addr 172.21.15.32:0/2083171216 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7fef06a00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec004800 0x7fef080ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7fef07400640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fef0815b2c0 0x7fef0815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 -- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fef0815b2c0 msgr2=0x7fef0815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fef0815b2c0 0x7fef0815d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 -- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec004800 msgr2=0x7fef080ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec004800 0x7fef080ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 -- 172.21.15.32:0/2083171216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080af4f0 con 0x7fef0806c7c0 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7fef07400640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fef0815b2c0 0x7fef0815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.520+0000 7feefe000640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 0x7fef080aefb0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7feef0001970 tx=0x7feef002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.521+0000 7feeff400640 1 -- 172.21.15.32:0/2083171216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0007d10 con 0x7fef0806c7c0 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.521+0000 7feeff400640 1 -- 172.21.15.32:0/2083171216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0004bf0 con 0x7fef0806c7c0 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.521+0000 7feeff400640 1 -- 172.21.15.32:0/2083171216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0007070 con 0x7fef0806c7c0 2024-03-06T07:35:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.521+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef080ade60 con 0x7fef0806c7c0 2024-03-06T07:35:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.521+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef080ae420 con 0x7fef0806c7c0 2024-03-06T07:35:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.523+0000 7feeff400640 1 -- 172.21.15.32:0/2083171216 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feef0007210 con 0x7fef0806c7c0 2024-03-06T07:35:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.524+0000 7feeff400640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feee406b520 0x7feee406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.525+0000 7feeff400640 1 -- 172.21.15.32:0/2083171216 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feef00ab520 con 0x7fef0806c7c0 2024-03-06T07:35:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.525+0000 7fef06a00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feee406b520 0x7feee406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.525+0000 7fef10572740 1 Processor -- start 2024-03-06T07:35:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.525+0000 7fef10572740 1 -- start start 2024-03-06T07:35:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.525+0000 7fef06a00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feee406b520 0x7feee406d9e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7feef8009a90 tx=0x7feef8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a5f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a8f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef10572740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 0x559d61a8fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef10572740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d619c9600 con 0x559d61a5fe70 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d61972930 con 0x559d61a8c990 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d619c56a0 con 0x559d61a8cd70 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.526+0000 7fef07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 0x559d61a8fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 0x559d61a8fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50400/0 (socket says 172.21.15.32:50400) 2024-03-06T07:35:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 -- 172.21.15.32:0/23555616 learned_addr learned my addr 172.21.15.32:0/23555616 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7feefe000640 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a8f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef06a00640 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a5f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 -- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 msgr2=0x559d61a8f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a8f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 -- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 msgr2=0x559d61a5f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a5f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 -- 172.21.15.32:0/23555616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d616de2d0 con 0x559d61a5fe70 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7fef07400640 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 0x559d61a8fc60 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fef08072090 tx=0x7fef00002050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2608fd179848b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7feef7400640 1 -- 172.21.15.32:0/23555616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00017070 con 0x559d61a5fe70 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.527+0000 7feef7400640 1 -- 172.21.15.32:0/23555616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0000d7f0 con 0x559d61a5fe70 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7feef7400640 1 -- 172.21.15.32:0/23555616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0000dba0 con 0x559d61a5fe70 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 -- 172.21.15.32:0/23555616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 msgr2=0x559d61a8fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 0x559d61a8fc60 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fef08072090 tx=0x7fef00002050 comp rx=0 tx=0).stop 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 -- 172.21.15.32:0/23555616 shutdown_connections 2024-03-06T07:35:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a5fe70 0x559d61a8fc60 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a8f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 --2- 172.21.15.32:0/23555616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a5f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 -- 172.21.15.32:0/23555616 >> 172.21.15.32:0/23555616 conn(0x559d61a61ba0 msgr2=0x559d61aaa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 -- 172.21.15.32:0/23555616 shutdown_connections 2024-03-06T07:35:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.528+0000 7fef10572740 1 -- 172.21.15.32:0/23555616 wait complete. 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.529+0000 7fef10572740 1 -- start start 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.529+0000 7fef10572740 1 Processor -- start 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 0x559d61a7a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a92bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a93100 0x559d61aa4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d61a64140 con 0x559d61a8c990 2024-03-06T07:35:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d61985d90 con 0x559d61a93100 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d61a63530 con 0x559d61a8cd70 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.530+0000 7fef10572740 10 client.0 _ll_register_callbacks cb 0x559d619c1350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 0x559d61a7a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 0x559d61a7a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50404/0 (socket says 172.21.15.32:50404) 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 -- 172.21.15.32:0/560770813 learned_addr learned my addr 172.21.15.32:0/560770813 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7feefe000640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a92bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef07400640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a93100 0x559d61aa4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 -- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 msgr2=0x559d61a92bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a92bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 -- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a93100 msgr2=0x559d61aa4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a93100 0x559d61aa4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 -- 172.21.15.32:0/560770813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d617a86b0 con 0x559d61a8c990 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7fef06a00640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 0x559d61a7a960 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7feef8013de0 tx=0x7feef8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ca47a53b23c2e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.531+0000 7feeebe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:35:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7feeebe00640 1 -- 172.21.15.32:0/560770813 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8027070 con 0x559d61a8c990 2024-03-06T07:35:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7feeebe00640 1 -- 172.21.15.32:0/560770813 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feef8004220 con 0x559d61a8c990 2024-03-06T07:35:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef0000d7f0 con 0x559d61a8c990 2024-03-06T07:35:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7feeebe00640 1 -- 172.21.15.32:0/560770813 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7feef8031050 con 0x559d61a8c990 2024-03-06T07:35:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7fef10572740 10 client.7791 fetch_fsmap learned FSMap version 75 2024-03-06T07:35:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x559d61a7bef0 con 0x559d61a8c990 2024-03-06T07:35:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7feeebe00640 1 -- 172.21.15.32:0/560770813 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0000d7f0 con 0x559d61a8c990 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7feeebe00640 1 -- 172.21.15.32:0/560770813 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 75) v1 ==== 3314+0+0 (secure 0 0 0) 0x7feef8034020 con 0x559d61a8c990 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.532+0000 7fef10572740 10 client.7791 fetch_fsmap finished waiting for FSMap version 75 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.533+0000 7fef10572740 10 client.7791 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.533+0000 7fef10572740 20 client.7791 populate_metadata read hostname 'smithi032' 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.533+0000 7fef10572740 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61a7c8e0 0x559d61a7d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.533+0000 7fef10572740 4 client.7791 mds_command: new command op to 7665 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.533+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 --> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d619d6b40 con 0x559d61a7c8e0 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.533+0000 7feefe000640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61a7c8e0 0x559d61a7d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.534+0000 7feefe000640 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61a7c8e0 0x559d61a7d9a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7800c0fa92f9bb3f server_cookie=8aa248209dd3d4a4 in_seq=0 out_seq=0 2024-03-06T07:35:22.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.534+0000 7feeebe00640 10 client.7791 ms_handle_connect on v2:172.21.15.39:6833/14889266 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7feeebe00640 1 -- 172.21.15.32:0/560770813 <== mds.0 v2:172.21.15.39:6833/14889266 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d619d6b40 con 0x559d61a7c8e0 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7feeebe00640 10 client.7791 handle_command_reply: tid=0 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 client.7791 shutdown 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 msgr2=0x559d61a7a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 0x559d61a7a960 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7feef8013de0 tx=0x7feef8004730 comp rx=0 tx=0).stop 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 shutdown_connections 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61a7c8e0 0x559d61a7d9a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a93100 0x559d61aa4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a92bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 --2- 172.21.15.32:0/560770813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8c990 0x559d61a7a960 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7feeebe00640 0 client.7791 ms_handle_reset on v2:172.21.15.39:6833/14889266 2024-03-06T07:35:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.536+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 >> 172.21.15.32:0/560770813 conn(0x559d61a5fe70 msgr2=0x559d61a8f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.537+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 shutdown_connections 2024-03-06T07:35:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.537+0000 7fef10572740 1 -- 172.21.15.32:0/560770813 wait complete. 2024-03-06T07:35:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.537+0000 7fef10572740 20 client.7791 trim_cache size 0 max 16384 2024-03-06T07:35:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.555+0000 7fef10572740 1 Processor -- start 2024-03-06T07:35:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.555+0000 7fef10572740 1 -- start start 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a64eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a5e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef10572740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 0x559d61a5ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef10572740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef081ff860 con 0x559d61a93100 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef081f79d0 con 0x559d61a8c990 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef081ff430 con 0x559d61a8cd70 2024-03-06T07:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.556+0000 7fef07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 0x559d61a5ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 0x559d61a5ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50406/0 (socket says 172.21.15.32:50406) 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 -- 172.21.15.32:0/2841604237 learned_addr learned my addr 172.21.15.32:0/2841604237 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef06a00640 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a64eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7feefe000640 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a5e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 -- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 msgr2=0x559d61a5e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a5e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 -- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 msgr2=0x559d61a64eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a64eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 -- 172.21.15.32:0/2841604237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d61aab750 con 0x559d61a93100 2024-03-06T07:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7fef07400640 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 0x559d61a5ed00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fef00008000 tx=0x7fef0000b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf0c69fd3411fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.557+0000 7feef6a00640 1 -- 172.21.15.32:0/2841604237 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00021020 con 0x559d61a93100 2024-03-06T07:35:22.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7feef6a00640 1 -- 172.21.15.32:0/2841604237 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0000da10 con 0x559d61a93100 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7feef6a00640 1 -- 172.21.15.32:0/2841604237 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00017040 con 0x559d61a93100 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 -- 172.21.15.32:0/2841604237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 msgr2=0x559d61a5ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 0x559d61a5ed00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fef00008000 tx=0x7fef0000b000 comp rx=0 tx=0).stop 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 -- 172.21.15.32:0/2841604237 shutdown_connections 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a93100 0x559d61a5ed00 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a8cd70 0x559d61a5e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 --2- 172.21.15.32:0/2841604237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a64eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 -- 172.21.15.32:0/2841604237 >> 172.21.15.32:0/2841604237 conn(0x559d61a5fe70 msgr2=0x559d61a79e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 -- 172.21.15.32:0/2841604237 shutdown_connections 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.558+0000 7fef10572740 1 -- 172.21.15.32:0/2841604237 wait complete. 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.559+0000 7fef10572740 1 -- start start 2024-03-06T07:35:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.559+0000 7fef10572740 1 Processor -- start 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a78de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 0x559d61aa0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a93100 0x559d61afc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d61aa2820 con 0x559d61a8cd70 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d61aa2ab0 con 0x559d61a8c990 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d61a7cd10 con 0x559d61a93100 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7fef10572740 10 client.0 _ll_register_callbacks cb 0x7feef8006eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7feefe000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 0x559d61aa0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7feefe000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 0x559d61aa0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50410/0 (socket says 172.21.15.32:50410) 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.560+0000 7feefe000640 1 -- 172.21.15.32:0/2776001685 learned_addr learned my addr 172.21.15.32:0/2776001685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7fef06a00640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a78de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7fef07400640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a93100 0x559d61afc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feefe000640 1 -- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a93100 msgr2=0x559d61afc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feefe000640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a93100 0x559d61afc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feefe000640 1 -- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 msgr2=0x559d61a78de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feefe000640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a78de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feefe000640 1 -- 172.21.15.32:0/2776001685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d61aa6c00 con 0x559d61a8cd70 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7fef06a00640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a78de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feefe000640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 0x559d61aa0190 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7feef0079c10 tx=0x7feef0006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b122cf0003ff98e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feeebe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feeebe00640 1 -- 172.21.15.32:0/2776001685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0096060 con 0x559d61a8cd70 2024-03-06T07:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7feeebe00640 1 -- 172.21.15.32:0/2776001685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feef00762f0 con 0x559d61a8cd70 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.561+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef0000da10 con 0x559d61a8cd70 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7feeebe00640 1 -- 172.21.15.32:0/2776001685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7feef007ddd0 con 0x559d61a8cd70 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 10 client.7797 fetch_fsmap learned FSMap version 75 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x559d61a64d20 con 0x559d61a8cd70 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7feeebe00640 1 -- 172.21.15.32:0/2776001685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0000da10 con 0x559d61a8cd70 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7feeebe00640 1 -- 172.21.15.32:0/2776001685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 75) v1 ==== 3314+0+0 (secure 0 0 0) 0x7feef0048030 con 0x559d61a8cd70 2024-03-06T07:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 10 client.7797 fetch_fsmap finished waiting for FSMap version 75 2024-03-06T07:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 10 client.7797 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 20 client.7797 populate_metadata read hostname 'smithi032' 2024-03-06T07:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61aadbe0 0x559d61aadfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 4 client.7797 mds_command: new command op to 7665 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.562+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 --> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d619d0e90 con 0x559d61aadbe0 2024-03-06T07:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.563+0000 7fef06a00640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61aadbe0 0x559d61aadfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.564+0000 7fef06a00640 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61aadbe0 0x559d61aadfc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a7b90ca02faba9 server_cookie=c683fbd16c065b2d in_seq=0 out_seq=0 2024-03-06T07:35:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.564+0000 7feeebe00640 10 client.7797 ms_handle_connect on v2:172.21.15.39:6833/14889266 2024-03-06T07:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.564+0000 7feeebe00640 1 -- 172.21.15.32:0/2776001685 <== mds.0 v2:172.21.15.39:6833/14889266 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559d619d0e90 con 0x559d61aadbe0 2024-03-06T07:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.564+0000 7feeebe00640 10 client.7797 handle_command_reply: tid=0 2024-03-06T07:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.564+0000 7fef10572740 1 client.7797 shutdown 2024-03-06T07:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 msgr2=0x559d61aa0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 0x559d61aa0190 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7feef0079c10 tx=0x7feef0006040 comp rx=0 tx=0).stop 2024-03-06T07:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 shutdown_connections 2024-03-06T07:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:6833/14889266,v1:172.21.15.39:6835/14889266] conn(0x559d61aadbe0 0x559d61aadfc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x559d61a93100 0x559d61afc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d61a8cd70 0x559d61aa0190 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7feeebe00640 0 client.7797 ms_handle_reset on v2:172.21.15.39:6833/14889266 2024-03-06T07:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 --2- 172.21.15.32:0/2776001685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d61a8c990 0x559d61a78de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 >> 172.21.15.32:0/2776001685 conn(0x559d61a5fe70 msgr2=0x559d61aa3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 shutdown_connections 2024-03-06T07:35:22.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 1 -- 172.21.15.32:0/2776001685 wait complete. 2024-03-06T07:35:22.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.565+0000 7fef10572740 20 client.7797 trim_cache size 0 max 16384 2024-03-06T07:35:22.572 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:35:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:35:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:35:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7665, 2024-03-06T07:35:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:35:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:35:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:35:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1558810690000003, 2024-03-06T07:35:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 75, 2024-03-06T07:35:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 110, 2024-03-06T07:35:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 110, 2024-03-06T07:35:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.568494162 2024-03-06T07:35:22.574 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feee406b520 msgr2=0x7feee406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feee406b520 0x7feee406d9e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7feef8009a90 tx=0x7feef8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 msgr2=0x7fef080aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 0x7fef080aefb0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7feef0001970 tx=0x7feef002e070 comp rx=0 tx=0).stop 2024-03-06T07:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 shutdown_connections 2024-03-06T07:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feee406b520 0x7feee406d9e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fef0815b2c0 0x7fef0815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef0806c7c0 0x7fef080aefb0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 --2- 172.21.15.32:0/2083171216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feeec004800 0x7fef080ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 >> 172.21.15.32:0/2083171216 conn(0x7fef0806b3d0 msgr2=0x7fef080b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 shutdown_connections 2024-03-06T07:35:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.566+0000 7fef07e00640 1 -- 172.21.15.32:0/2083171216 wait complete. 2024-03-06T07:35:22.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:35:22.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.792+0000 7feb77200640 1 Processor -- start 2024-03-06T07:35:22.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.792+0000 7feb77200640 1 -- start start 2024-03-06T07:35:22.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.794+0000 7feb77200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7feb7815efc0 0x7feb780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.794+0000 7feb77200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb781627f0 con 0x7feb780cce90 2024-03-06T07:35:22.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.794+0000 7feb77200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7806c860 con 0x7feb781621d0 2024-03-06T07:35:22.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.795+0000 7feb77200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7815f3a0 con 0x7feb7815efc0 2024-03-06T07:35:22.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.795+0000 7feb6fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feb780cce90 0x7feb78161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48806/0 (socket says 172.21.15.32:48806) 2024-03-06T07:35:22.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.795+0000 7feb6fe00640 1 -- 172.21.15.32:0/3125051770 learned_addr learned my addr 172.21.15.32:0/3125051770 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:22.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.795+0000 7feb75e00640 1 --2- 172.21.15.32:0/3125051770 >> v2:172.21.15.39:3300/0 conn(0x7feb7815efc0 0x7feb780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.796+0000 7feb75e00640 1 -- 172.21.15.32:0/3125051770 >> v1:172.21.15.39:6789/0 conn(0x7feb781621d0 legacy=0x7feb78070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.796+0000 7feb75e00640 1 -- 172.21.15.32:0/3125051770 >> v1:172.21.15.32:6789/0 conn(0x7feb780cce90 legacy=0x7feb78161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.796+0000 7feb75e00640 1 -- 172.21.15.32:0/3125051770 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7806cb40 con 0x7feb7815efc0 2024-03-06T07:35:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.798+0000 7feb75e00640 1 --2- 172.21.15.32:0/3125051770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb780cc950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feb680097a0 tx=0x7feb6802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d305658d428a2c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.800+0000 7feb75400640 1 -- 172.21.15.32:0/3125051770 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68034070 con 0x7feb7815efc0 2024-03-06T07:35:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.800+0000 7feb75400640 1 -- 172.21.15.32:0/3125051770 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb68030610 con 0x7feb7815efc0 2024-03-06T07:35:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.800+0000 7feb75400640 1 -- 172.21.15.32:0/3125051770 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680309e0 con 0x7feb7815efc0 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.801+0000 7feb77200640 1 -- 172.21.15.32:0/3125051770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 msgr2=0x7feb780cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.801+0000 7feb77200640 1 --2- 172.21.15.32:0/3125051770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb780cc950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feb680097a0 tx=0x7feb6802b960 comp rx=0 tx=0).stop 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.801+0000 7feb77200640 1 -- 172.21.15.32:0/3125051770 shutdown_connections 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.801+0000 7feb77200640 1 --2- 172.21.15.32:0/3125051770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb780cc950 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.801+0000 7feb77200640 1 -- 172.21.15.32:0/3125051770 >> 172.21.15.32:0/3125051770 conn(0x7feb7806b3d0 msgr2=0x7feb7806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.802+0000 7feb77200640 1 -- 172.21.15.32:0/3125051770 shutdown_connections 2024-03-06T07:35:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.802+0000 7feb77200640 1 -- 172.21.15.32:0/3125051770 wait complete. 2024-03-06T07:35:22.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.805+0000 7feb77200640 1 Processor -- start 2024-03-06T07:35:22.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.805+0000 7feb77200640 1 -- start start 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.806+0000 7feb77200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb780cce90 0x7feb781f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.806+0000 7feb77200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb7815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.806+0000 7feb77200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 0x7feb7815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.806+0000 7feb77200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7806d620 con 0x7feb781621d0 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.806+0000 7feb77200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7806d190 con 0x7feb7815efc0 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.806+0000 7feb77200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb781e84e0 con 0x7feb780cce90 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 0x7feb7815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 0x7feb7815b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50416/0 (socket says 172.21.15.32:50416) 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 -- 172.21.15.32:0/76876122 learned_addr learned my addr 172.21.15.32:0/76876122 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb6fe00640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb7815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb75e00640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb780cce90 0x7feb781f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 -- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb780cce90 msgr2=0x7feb781f78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb780cce90 0x7feb781f78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 -- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 msgr2=0x7feb7815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb7815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.807+0000 7feb76800640 1 -- 172.21.15.32:0/76876122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb781e87c0 con 0x7feb781621d0 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.808+0000 7feb76800640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 0x7feb7815b1d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c6b0 tx=0x7feb7000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.808+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70003b70 con 0x7feb781621d0 2024-03-06T07:35:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.808+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb70003d30 con 0x7feb781621d0 2024-03-06T07:35:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.808+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70005310 con 0x7feb781621d0 2024-03-06T07:35:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.808+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb781e8aa0 con 0x7feb781621d0 2024-03-06T07:35:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.808+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781f7e00 con 0x7feb781621d0 2024-03-06T07:35:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.810+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7804f330 con 0x7feb781621d0 2024-03-06T07:35:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.810+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb700054b0 con 0x7feb781621d0 2024-03-06T07:35:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.811+0000 7feb6e000640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb5406b520 0x7feb5406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.811+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb70089f20 con 0x7feb781621d0 2024-03-06T07:35:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.812+0000 7feb75e00640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb5406b520 0x7feb5406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.812+0000 7feb75e00640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb5406b520 0x7feb5406d9e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feb68036000 tx=0x7feb6803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:22.817+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb70058f10 con 0x7feb781621d0 2024-03-06T07:35:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.058+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feb780af2f0 con 0x7feb781621d0 2024-03-06T07:35:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.059+0000 7feb6e000640 1 -- 172.21.15.32:0/76876122 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7feb70058810 con 0x7feb781621d0 2024-03-06T07:35:23.060 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:35:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb5406b520 msgr2=0x7feb5406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb5406b520 0x7feb5406d9e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feb68036000 tx=0x7feb6803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 msgr2=0x7feb7815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 0x7feb7815b1d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c6b0 tx=0x7feb7000cb70 comp rx=0 tx=0).stop 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 shutdown_connections 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feb5406b520 0x7feb5406d9e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb781621d0 0x7feb7815b1d0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb7815efc0 0x7feb7815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 --2- 172.21.15.32:0/76876122 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feb780cce90 0x7feb781f78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 >> 172.21.15.32:0/76876122 conn(0x7feb7806b3d0 msgr2=0x7feb780cdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 shutdown_connections 2024-03-06T07:35:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:23.062+0000 7feb77200640 1 -- 172.21.15.32:0/76876122 wait complete. 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:35:23.084 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:23.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.085 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42014sec preferred_lft 42014sec 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:35:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:35:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:23.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.140 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:35:23.229 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:35:23.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.233 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:35:23.289 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:35:23.289 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:35:23.289 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:35:23.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:35:23.289 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:35:23.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.290 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:35:23.290 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:35:23.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:35:23.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:35:23.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:35:23.290 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:35:23.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.491 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:35:23.544 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:35:23.544 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:35:23.545 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:23.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:35:23.643 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:23.643 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:23.699 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:23.699 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:23.699 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:23.755 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:35:23.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:23.901 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:23.901 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:23.939 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:23.939 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:23.939 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:35:23.939 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:35:23.939 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:35:23.942 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:35:23.942 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:35:23.942 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:35:23.943 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:35:23.943 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:35:23.949 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:35:23.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:23.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:24.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.046 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:35:24.129 INFO:teuthology.orchestra.run.smithi032.stdout:client6253 2024-03-06T07:35:24.129 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:24.129 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.187 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:35:24.256 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:24.256 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.311 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi032.stderr:8+0 records in 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi032.stderr:8+0 records out 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi032.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.141297 s, 59.4 MB/s 2024-03-06T07:35:24.510 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:35:24.511 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:35:24.511 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-03-06T07:35:24.511 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:35:24.511 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:35:24.511 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:35:24.662 INFO:teuthology.orchestra.run.smithi032.stdout:1099511627777 2024-03-06T07:35:24.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:35:24.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.676 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:35:24.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:24.778 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:35:24.846 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:35:24.846 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:35:24.846 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-03-06T07:35:24.846 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:35:24.846 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:35:24.846 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:35:24.994 INFO:teuthology.orchestra.run.smithi032.stdout:1099511627777 2024-03-06T07:35:24.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:35:25.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:25.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.267+0000 7f0533200640 1 Processor -- start 2024-03-06T07:35:25.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.268+0000 7f0533200640 1 -- start start 2024-03-06T07:35:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0533200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f05340d4a10 0x7f05340d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0533200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05340d0c80 0x7f05340b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0533200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340b3710 con 0x7f053415e0b0 2024-03-06T07:35:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0533200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340d10f0 con 0x7f05340d0c80 2024-03-06T07:35:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0533200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340b09e0 con 0x7f05340d4a10 2024-03-06T07:35:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0532800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05340d0c80 0x7f05340b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0532800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05340d0c80 0x7f05340b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50432/0 (socket says 172.21.15.32:50432) 2024-03-06T07:35:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.269+0000 7f0532800640 1 -- 172.21.15.32:0/2041477624 learned_addr learned my addr 172.21.15.32:0/2041477624 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.270+0000 7f0531400640 1 --2- 172.21.15.32:0/2041477624 >> v2:172.21.15.39:3300/0 conn(0x7f05340d4a10 0x7f05340d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.270+0000 7f0532800640 1 -- 172.21.15.32:0/2041477624 >> v2:172.21.15.39:3300/0 conn(0x7f05340d4a10 msgr2=0x7f05340d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.270+0000 7f0532800640 1 --2- 172.21.15.32:0/2041477624 >> v2:172.21.15.39:3300/0 conn(0x7f05340d4a10 0x7f05340d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.270+0000 7f0532800640 1 -- 172.21.15.32:0/2041477624 >> v1:172.21.15.39:6789/0 conn(0x7f053415e0b0 legacy=0x7f05340cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.270+0000 7f0532800640 1 -- 172.21.15.32:0/2041477624 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340b39f0 con 0x7f05340d0c80 2024-03-06T07:35:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.270+0000 7f0531400640 1 --2- 172.21.15.32:0/2041477624 >> v2:172.21.15.39:3300/0 conn(0x7f05340d4a10 0x7f05340d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.271+0000 7f0532800640 1 --2- 172.21.15.32:0/2041477624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d0c80 0x7f05340b44d0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f053408d6d0 tx=0x7f052c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25f3b822afd6d52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.271+0000 7f0530a00640 1 -- 172.21.15.32:0/2041477624 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c02ca40 con 0x7f05340d0c80 2024-03-06T07:35:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.271+0000 7f0530a00640 1 -- 172.21.15.32:0/2041477624 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c02cbe0 con 0x7f05340d0c80 2024-03-06T07:35:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.271+0000 7f0530a00640 1 -- 172.21.15.32:0/2041477624 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0343f0 con 0x7f05340d0c80 2024-03-06T07:35:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 -- 172.21.15.32:0/2041477624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d0c80 msgr2=0x7f05340b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 --2- 172.21.15.32:0/2041477624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d0c80 0x7f05340b44d0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f053408d6d0 tx=0x7f052c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 -- 172.21.15.32:0/2041477624 shutdown_connections 2024-03-06T07:35:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 --2- 172.21.15.32:0/2041477624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d0c80 0x7f05340b44d0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 --2- 172.21.15.32:0/2041477624 >> v2:172.21.15.39:3300/0 conn(0x7f05340d4a10 0x7f05340d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 -- 172.21.15.32:0/2041477624 >> 172.21.15.32:0/2041477624 conn(0x7f05340d3d30 msgr2=0x7f05340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 -- 172.21.15.32:0/2041477624 shutdown_connections 2024-03-06T07:35:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.272+0000 7f0533200640 1 -- 172.21.15.32:0/2041477624 wait complete. 2024-03-06T07:35:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.274+0000 7f0533200640 1 Processor -- start 2024-03-06T07:35:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.274+0000 7f0533200640 1 -- start start 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0533200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05340d0c80 0x7f05340b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0533200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 0x7f053415f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0533200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f053415e0b0 0x7f05341f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0533200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05340b3fb0 con 0x7f05340d4a10 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0533200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05340cfa20 con 0x7f05340d0c80 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0533200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053415e490 con 0x7f053415e0b0 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0531400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 0x7f053415f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0531400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 0x7f053415f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50446/0 (socket says 172.21.15.32:50446) 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0531400640 1 -- 172.21.15.32:0/1223324654 learned_addr learned my addr 172.21.15.32:0/1223324654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0531e00640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05340d0c80 0x7f05340b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.275+0000 7f0532800640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f053415e0b0 0x7f05341f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531400640 1 -- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f053415e0b0 msgr2=0x7f05341f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531400640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f053415e0b0 0x7f05341f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531400640 1 -- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05340d0c80 msgr2=0x7f05340b8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531400640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05340d0c80 0x7f05340b8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531400640 1 -- 172.21.15.32:0/1223324654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053415f8d0 con 0x7f05340d4a10 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531e00640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05340d0c80 0x7f05340b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0531400640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 0x7f053415f080 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f052000c7f0 tx=0x7f052000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520015070 con 0x7f05340d4a10 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0520007a30 con 0x7f05340d4a10 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520007e00 con 0x7f05340d4a10 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.276+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053415fbb0 con 0x7f05340d4a10 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.277+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0534160170 con 0x7f05340d4a10 2024-03-06T07:35:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.278+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053404f330 con 0x7f05340d4a10 2024-03-06T07:35:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.279+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0520018070 con 0x7f05340d4a10 2024-03-06T07:35:25.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.280+0000 7f052a000640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f051406b500 0x7f051406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.280+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f052008a250 con 0x7f05340d4a10 2024-03-06T07:35:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.281+0000 7f0531e00640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f051406b500 0x7f051406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.281+0000 7f0531e00640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f051406b500 0x7f051406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0524003eb0 tx=0x7f0524009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.285+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0520059030 con 0x7f05340d4a10 2024-03-06T07:35:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.525+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f053415f5c0 con 0x7f05340d4a10 2024-03-06T07:35:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.527+0000 7f052a000640 1 -- 172.21.15.32:0/1223324654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f0520058930 con 0x7f05340d4a10 2024-03-06T07:35:25.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:25.528 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3906463088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3906463088},{"type":"v1","addr":"172.21.15.39:6834","nonce":3906463088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7647,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/2563322789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2563322789},{"type":"v1","addr":"172.21.15.32:6836","nonce":2563322789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7650,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2154839965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":2154839965},{"type":"v1","addr":"172.21.15.32:6837","nonce":2154839965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:18.287992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7665":{"gid":7665,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/14889266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":14889266},{"type":"v1","addr":"172.21.15.39:6835","nonce":14889266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7665,"qdb_cluster":[7665]},"id":8}]} 2024-03-06T07:35:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-03-06T07:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.529+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f051406b500 msgr2=0x7f051406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.529+0000 7f0533200640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f051406b500 0x7f051406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0524003eb0 tx=0x7f0524009040 comp rx=0 tx=0).stop 2024-03-06T07:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.529+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 msgr2=0x7f053415f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.529+0000 7f0533200640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 0x7f053415f080 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f052000c7f0 tx=0x7f052000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.529+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 shutdown_connections 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f051406b500 0x7f051406d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f053415e0b0 0x7f05341f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05340d4a10 0x7f053415f080 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 --2- 172.21.15.32:0/1223324654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05340d0c80 0x7f05340b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 >> 172.21.15.32:0/1223324654 conn(0x7f05340d3d30 msgr2=0x7f05340b1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 shutdown_connections 2024-03-06T07:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.530+0000 7f0533200640 1 -- 172.21.15.32:0/1223324654 wait complete. 2024-03-06T07:35:25.552 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-06T07:35:25.817 INFO:teuthology.orchestra.run.smithi039.stdout:{"message":"","return_code":0} 2024-03-06T07:35:25.817 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:25.817 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.930+0000 7fb304846f40 1 Processor -- start 2024-03-06T07:35:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.930+0000 7fb304846f40 1 -- start start 2024-03-06T07:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.931+0000 7fb304846f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b715210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.931+0000 7fb304846f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b6f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.931+0000 7fb304846f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b788170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.931+0000 7fb304846f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55838b6c7a50 con 0x55838b713cf0 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.931+0000 7fb304846f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55838b66efd0 con 0x55838b7147e0 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.931+0000 7fb304846f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55838b6c6dd0 con 0x55838b6c60d0 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b6f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b6f91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50460/0 (socket says 172.21.15.32:50460) 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 -- 172.21.15.32:0/3482518140 learned_addr learned my addr 172.21.15.32:0/3482518140 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb301c00640 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b788170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb301200640 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b715210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 -- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 msgr2=0x55838b715210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b715210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 -- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 msgr2=0x55838b788170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b788170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.932+0000 7fb2fbe00640 1 -- 172.21.15.32:0/3482518140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55838b784290 con 0x55838b713cf0 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.933+0000 7fb2fbe00640 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b6f91c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00b2f0 tx=0x7fb2ec00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1246a5f96f9051f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.933+0000 7fb2faa00640 1 -- 172.21.15.32:0/3482518140 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec008070 con 0x55838b713cf0 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.933+0000 7fb2faa00640 1 -- 172.21.15.32:0/3482518140 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec00d500 con 0x55838b713cf0 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.933+0000 7fb2faa00640 1 -- 172.21.15.32:0/3482518140 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00d8d0 con 0x55838b713cf0 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.933+0000 7fb304846f40 1 -- 172.21.15.32:0/3482518140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 msgr2=0x55838b6f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.933+0000 7fb304846f40 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b6f91c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00b2f0 tx=0x7fb2ec00b7b0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 -- 172.21.15.32:0/3482518140 shutdown_connections 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b788170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b6f91c0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 --2- 172.21.15.32:0/3482518140 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b715210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 -- 172.21.15.32:0/3482518140 >> 172.21.15.32:0/3482518140 conn(0x55838b6f7c10 msgr2=0x55838b6f7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 -- 172.21.15.32:0/3482518140 shutdown_connections 2024-03-06T07:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 -- 172.21.15.32:0/3482518140 wait complete. 2024-03-06T07:35:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.934+0000 7fb304846f40 1 Processor -- start 2024-03-06T07:35:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.935+0000 7fb304846f40 1 -- start start 2024-03-06T07:35:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb304846f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b77da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb304846f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b788360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb304846f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b77a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb304846f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55838b715750 con 0x55838b713cf0 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb304846f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55838b782840 con 0x55838b7147e0 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb304846f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55838b79f860 con 0x55838b6c60d0 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b788360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b788360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50472/0 (socket says 172.21.15.32:50472) 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 -- 172.21.15.32:0/1912694964 learned_addr learned my addr 172.21.15.32:0/1912694964 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb301c00640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b77a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb301200640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b77da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 -- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 msgr2=0x55838b77da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b77da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 -- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 msgr2=0x55838b77a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55838b7147e0 0x55838b77a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.936+0000 7fb2fbe00640 1 -- 172.21.15.32:0/1912694964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55838b6fa020 con 0x55838b713cf0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb301200640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55838b6c60d0 0x55838b77da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb2fbe00640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b788360 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x55838b783010 tx=0x7fb2ec000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb2f9600640 1 -- 172.21.15.32:0/1912694964 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00fc30 con 0x55838b713cf0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb2f9600640 1 -- 172.21.15.32:0/1912694964 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec020030 con 0x55838b713cf0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55838b6fa300 con 0x55838b713cf0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb2f9600640 1 -- 172.21.15.32:0/1912694964 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00de30 con 0x55838b713cf0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.937+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55838b6fa8c0 con 0x55838b713cf0 2024-03-06T07:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.940+0000 7fb2f9600640 1 -- 172.21.15.32:0/1912694964 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2ec017040 con 0x55838b713cf0 2024-03-06T07:35:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.940+0000 7fb2f9600640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb2e0069eb0 0x7fb2e006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.941+0000 7fb2f9600640 1 -- 172.21.15.32:0/1912694964 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2ec08aee0 con 0x55838b713cf0 2024-03-06T07:35:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.941+0000 7fb301200640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb2e0069eb0 0x7fb2e006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.941+0000 7fb304846f40 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55838b7888a0 0x55838b79dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.941+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e110) v9 -- 0x55838b77ab00 con 0x55838b7888a0 2024-03-06T07:35:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.941+0000 7fb301c00640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55838b7888a0 0x55838b79dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.942+0000 7fb301200640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb2e0069eb0 0x7fb2e006c370 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4054000 tx=0x7fb2f404e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.942+0000 7fb301c00640 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55838b7888a0 0x55838b79dcf0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.942+0000 7fb301c00640 1 -- 172.21.15.32:0/1912694964 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fb2fc002a00 con 0x55838b7888a0 2024-03-06T07:35:25.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.942+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55838b7888a0 msgr2=0x55838b79dcf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.942+0000 7fb304846f40 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55838b7888a0 0x55838b79dcf0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb2e0069eb0 msgr2=0x7fb2e006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb2e0069eb0 0x7fb2e006c370 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4054000 tx=0x7fb2f404e000 comp rx=0 tx=0).stop 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 msgr2=0x55838b788360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 --2- 172.21.15.32:0/1912694964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55838b713cf0 0x55838b788360 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x55838b783010 tx=0x7fb2ec000ed0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb301c00640 1 -- 172.21.15.32:0/1912694964 reap_dead start 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 shutdown_connections 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 >> 172.21.15.32:0/1912694964 conn(0x55838b6f7c10 msgr2=0x55838b6f7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 shutdown_connections 2024-03-06T07:35:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:25.943+0000 7fb304846f40 1 -- 172.21.15.32:0/1912694964 wait complete. 2024-03-06T07:35:25.949 INFO:teuthology.orchestra.run.smithi032.stdout:Wfile_adir_1 2024-03-06T07:35:25.949 DEBUG:teuthology.orchestra.run.smithi032:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1099511627777, 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "ancestors": [ 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1099511627776, 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "file_a", 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 8 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:35:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1, 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "dir_1", 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 15 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 17, 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "old_pools": [] 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:35:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:26.098 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:26.098 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:26.156 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:26.156 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:26.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:26.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.434+0000 7f4c01400640 1 Processor -- start 2024-03-06T07:35:26.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.434+0000 7f4c01400640 1 -- start start 2024-03-06T07:35:26.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.435+0000 7f4c01400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4bfc15f450 0x7f4bfc070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.435+0000 7f4c01400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc0cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.435+0000 7f4c01400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4bfc0d02e0 0x7f4bfc06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.435+0000 7f4c01400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc0d29e0 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.435+0000 7f4c01400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc0cd0c0 con 0x7f4bfc15f450 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.435+0000 7f4c01400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc0cd3a0 con 0x7f4bfc0700c0 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4bfc0d02e0 0x7f4bfc06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4bfc0d02e0 0x7f4bfc06ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50478/0 (socket says 172.21.15.32:50478) 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 -- 172.21.15.32:0/2609070085 learned_addr learned my addr 172.21.15.32:0/2609070085 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4bfb400640 1 --2- 172.21.15.32:0/2609070085 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc0cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/2609070085 >> v2:172.21.15.39:3300/0 conn(0x7f4bfc15f450 0x7f4bfc070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 -- 172.21.15.32:0/2609070085 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 msgr2=0x7f4bfc0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 --2- 172.21.15.32:0/2609070085 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc0cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 -- 172.21.15.32:0/2609070085 >> v2:172.21.15.39:3300/0 conn(0x7f4bfc15f450 msgr2=0x7f4bfc070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 --2- 172.21.15.32:0/2609070085 >> v2:172.21.15.39:3300/0 conn(0x7f4bfc15f450 0x7f4bfc070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.436+0000 7f4c00a00640 1 -- 172.21.15.32:0/2609070085 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc0cd680 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.438+0000 7f4c00a00640 1 --2- 172.21.15.32:0/2609070085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc06ec60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0696b0 tx=0x7f4bf402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9232e77ecf2f4938 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.438+0000 7f4bfaa00640 1 -- 172.21.15.32:0/2609070085 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf402ca80 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.438+0000 7f4bfaa00640 1 -- 172.21.15.32:0/2609070085 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf402cc20 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.438+0000 7f4bfaa00640 1 -- 172.21.15.32:0/2609070085 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40343f0 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 -- 172.21.15.32:0/2609070085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 msgr2=0x7f4bfc06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 --2- 172.21.15.32:0/2609070085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc06ec60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0696b0 tx=0x7f4bf402bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 -- 172.21.15.32:0/2609070085 shutdown_connections 2024-03-06T07:35:26.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 --2- 172.21.15.32:0/2609070085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc06ec60 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 --2- 172.21.15.32:0/2609070085 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc0cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 --2- 172.21.15.32:0/2609070085 >> v2:172.21.15.39:3300/0 conn(0x7f4bfc15f450 0x7f4bfc070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 -- 172.21.15.32:0/2609070085 >> 172.21.15.32:0/2609070085 conn(0x7f4bfc0d1ae0 msgr2=0x7f4bfc0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 -- 172.21.15.32:0/2609070085 shutdown_connections 2024-03-06T07:35:26.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.439+0000 7f4c01400640 1 -- 172.21.15.32:0/2609070085 wait complete. 2024-03-06T07:35:26.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.440+0000 7f4c01400640 1 Processor -- start 2024-03-06T07:35:26.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.441+0000 7f4c01400640 1 -- start start 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c01400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c01400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc1552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c01400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bfc15f450 0x7f4bfc144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c01400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc15f830 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c01400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc0ae8b0 con 0x7f4bfc15f450 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c01400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc15f9b0 con 0x7f4bfc0700c0 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc1552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc1552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50486/0 (socket says 172.21.15.32:50486) 2024-03-06T07:35:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 -- 172.21.15.32:0/1920256662 learned_addr learned my addr 172.21.15.32:0/1920256662 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4c00a00640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bfc15f450 0x7f4bfc144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 -- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 msgr2=0x7f4bfc152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 -- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bfc15f450 msgr2=0x7f4bfc144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bfc15f450 0x7f4bfc144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.442+0000 7f4bfb400640 1 -- 172.21.15.32:0/1920256662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc147940 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.443+0000 7f4bfb400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc1552a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4be800d830 tx=0x7f4be800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.443+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8018030 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.443+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be800be10 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.443+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be80098b0 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.443+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc147c20 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.443+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc154aa0 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.444+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bfc04f330 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.445+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4be801b070 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.446+0000 7f4bf3e00640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.446+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4be808a4b0 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.447+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.447+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4bec006fd0 tx=0x7f4bec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.451+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4be8059290 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.651+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bfc0c9450 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.654+0000 7f4bf3e00640 1 -- 172.21.15.32:0/1920256662 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f4be8058b90 con 0x7f4bfc0d02e0 2024-03-06T07:35:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:26.656 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":110,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:16.400077+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4bdc06b4d0 msgr2=0x7f4bdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4bec006fd0 tx=0x7f4bec008040 comp rx=0 tx=0).stop 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 msgr2=0x7f4bfc1552a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc1552a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4be800d830 tx=0x7f4be800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 shutdown_connections 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4bfc15f450 0x7f4bfc144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bfc0d02e0 0x7f4bfc1552a0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 --2- 172.21.15.32:0/1920256662 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4bfc0700c0 0x7f4bfc152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 >> 172.21.15.32:0/1920256662 conn(0x7f4bfc0d1ae0 msgr2=0x7f4bfc15b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 shutdown_connections 2024-03-06T07:35:26.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.657+0000 7f4c01400640 1 -- 172.21.15.32:0/1920256662 wait complete. 2024-03-06T07:35:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:12.716010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:13.860496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:26.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.909+0000 7f6715c00640 1 Processor -- start 2024-03-06T07:35:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.909+0000 7f6715c00640 1 -- start start 2024-03-06T07:35:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.910+0000 7f6715c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f671015e680 0x7f671006c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.910+0000 7f6715c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.910+0000 7f6715c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671015eb80 con 0x7f671006da40 2024-03-06T07:35:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.910+0000 7f6715c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671006ea50 con 0x7f671015e680 2024-03-06T07:35:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.910+0000 7f6715c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6710161c70 con 0x7f67100ccad0 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.911+0000 7f670f400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.911+0000 7f6706000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f671015e680 0x7f671006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.911+0000 7f670ea00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f671006da40 0x7f671006e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:43744/0 (socket says 172.21.15.32:43744) 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.911+0000 7f670ea00640 1 -- 172.21.15.32:0/305577902 learned_addr learned my addr 172.21.15.32:0/305577902 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.912+0000 7f6706000640 1 -- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 msgr2=0x7f67100ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.912+0000 7f6706000640 1 --2- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.912+0000 7f6706000640 1 -- 172.21.15.32:0/305577902 >> v1:172.21.15.39:6789/0 conn(0x7f671006da40 legacy=0x7f671006e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.912+0000 7f6706000640 1 -- 172.21.15.32:0/305577902 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671015ee60 con 0x7f671015e680 2024-03-06T07:35:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.914+0000 7f6706000640 1 --2- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671015e680 0x7f671006c980 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f66f80097a0 tx=0x7f66f802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e2e5c26328f19b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.914+0000 7f670e000640 1 -- 172.21.15.32:0/305577902 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8035020 con 0x7f671015e680 2024-03-06T07:35:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f670e000640 1 -- 172.21.15.32:0/305577902 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8031810 con 0x7f671015e680 2024-03-06T07:35:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f670e000640 1 -- 172.21.15.32:0/305577902 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8031be0 con 0x7f671015e680 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 -- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671015e680 msgr2=0x7f671006c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 --2- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671015e680 0x7f671006c980 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f66f80097a0 tx=0x7f66f802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 -- 172.21.15.32:0/305577902 shutdown_connections 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 --2- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 --2- 172.21.15.32:0/305577902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671015e680 0x7f671006c980 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 -- 172.21.15.32:0/305577902 >> 172.21.15.32:0/305577902 conn(0x7f671006b3d0 msgr2=0x7f671006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.915+0000 7f6715c00640 1 -- 172.21.15.32:0/305577902 shutdown_connections 2024-03-06T07:35:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.916+0000 7f6715c00640 1 -- 172.21.15.32:0/305577902 wait complete. 2024-03-06T07:35:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.917+0000 7f6715c00640 1 Processor -- start 2024-03-06T07:35:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.917+0000 7f6715c00640 1 -- start start 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.918+0000 7f6715c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671006da40 0x7f671015aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.918+0000 7f6715c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.918+0000 7f6715c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 0x7f67100b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.918+0000 7f6715c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671006cec0 con 0x7f671015e680 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.918+0000 7f6715c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671015f420 con 0x7f671006da40 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.918+0000 7f6715c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671006d040 con 0x7f67100ccad0 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 0x7f67100b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 0x7f67100b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50494/0 (socket says 172.21.15.32:50494) 2024-03-06T07:35:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 -- 172.21.15.32:0/1256668025 learned_addr learned my addr 172.21.15.32:0/1256668025 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670ea00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671006da40 0x7f671015aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f6706000640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 -- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 msgr2=0x7f67100b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100b4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 -- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671006da40 msgr2=0x7f671015aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671006da40 0x7f671015aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 -- 172.21.15.32:0/1256668025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67101f74f0 con 0x7f671015e680 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.919+0000 7f670f400640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 0x7f67100b4cd0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6708009dd0 tx=0x7f670800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.920+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708018020 con 0x7f671015e680 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.920+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6708014480 con 0x7f671015e680 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.920+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708014870 con 0x7f671015e680 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.920+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67101f77d0 con 0x7f671015e680 2024-03-06T07:35:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.920+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67101f7d90 con 0x7f671015e680 2024-03-06T07:35:26.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.921+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671004f330 con 0x7f671015e680 2024-03-06T07:35:26.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.922+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f670801b020 con 0x7f671015e680 2024-03-06T07:35:26.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.923+0000 7f6707400640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f66ec06b500 0x7f66ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.923+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f670808ae20 con 0x7f671015e680 2024-03-06T07:35:26.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.923+0000 7f670ea00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f66ec06b500 0x7f66ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.924+0000 7f670ea00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f66ec06b500 0x7f66ec06d9c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f671015e4b0 tx=0x7f6700009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:26.928+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6708022030 con 0x7f671015e680 2024-03-06T07:35:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.129+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67100af550 con 0x7f671015e680 2024-03-06T07:35:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.130+0000 7f6707400640 1 -- 172.21.15.32:0/1256668025 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6708010040 con 0x7f671015e680 2024-03-06T07:35:27.131 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:35:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f66ec06b500 msgr2=0x7f66ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f66ec06b500 0x7f66ec06d9c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f671015e4b0 tx=0x7f6700009040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 msgr2=0x7f67100b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 0x7f67100b4cd0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6708009dd0 tx=0x7f670800c490 comp rx=0 tx=0).stop 2024-03-06T07:35:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 shutdown_connections 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f66ec06b500 0x7f66ec06d9c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e680 0x7f67100b4cd0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67100ccad0 0x7f67100b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 --2- 172.21.15.32:0/1256668025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f671006da40 0x7f671015aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 >> 172.21.15.32:0/1256668025 conn(0x7f671006b3d0 msgr2=0x7f67100b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 shutdown_connections 2024-03-06T07:35:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.133+0000 7f6715c00640 1 -- 172.21.15.32:0/1256668025 wait complete. 2024-03-06T07:35:27.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.378+0000 7fd476c00640 1 Processor -- start 2024-03-06T07:35:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.378+0000 7fd476c00640 1 -- start start 2024-03-06T07:35:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd476c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd47015bf20 0x7fd4700acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd476c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd470072df0 con 0x7fd470072520 2024-03-06T07:35:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd476c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4700cdab0 con 0x7fd470071e20 2024-03-06T07:35:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd476c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4700b4d70 con 0x7fd47015bf20 2024-03-06T07:35:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd46f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd470072520 0x7fd4700cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48810/0 (socket says 172.21.15.32:48810) 2024-03-06T07:35:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd46f400640 1 -- 172.21.15.32:0/2829042807 learned_addr learned my addr 172.21.15.32:0/2829042807 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.380+0000 7fd46fe00640 1 --2- 172.21.15.32:0/2829042807 >> v2:172.21.15.32:3300/0 conn(0x7fd47015bf20 0x7fd4700acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.381+0000 7fd46fe00640 1 -- 172.21.15.32:0/2829042807 >> v1:172.21.15.39:6789/0 conn(0x7fd470071e20 legacy=0x7fd4700ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.381+0000 7fd46ea00640 1 -- 172.21.15.32:0/2829042807 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53840542 0 0) 0x7fd470072df0 con 0x7fd470072520 2024-03-06T07:35:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.381+0000 7fd46fe00640 1 -- 172.21.15.32:0/2829042807 >> v1:172.21.15.32:6789/0 conn(0x7fd470072520 legacy=0x7fd4700cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.381+0000 7fd46fe00640 1 -- 172.21.15.32:0/2829042807 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4700b5050 con 0x7fd47015bf20 2024-03-06T07:35:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.382+0000 7fd46fe00640 1 --2- 172.21.15.32:0/2829042807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47015bf20 0x7fd4700acd40 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d90 tx=0x7fd46002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891e639d2fc47a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.382+0000 7fd46ea00640 1 -- 172.21.15.32:0/2829042807 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460039070 con 0x7fd47015bf20 2024-03-06T07:35:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.382+0000 7fd46ea00640 1 -- 172.21.15.32:0/2829042807 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46002cc60 con 0x7fd47015bf20 2024-03-06T07:35:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.382+0000 7fd46ea00640 1 -- 172.21.15.32:0/2829042807 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460034660 con 0x7fd47015bf20 2024-03-06T07:35:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 -- 172.21.15.32:0/2829042807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47015bf20 msgr2=0x7fd4700acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 --2- 172.21.15.32:0/2829042807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47015bf20 0x7fd4700acd40 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d90 tx=0x7fd46002beb0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 -- 172.21.15.32:0/2829042807 shutdown_connections 2024-03-06T07:35:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 --2- 172.21.15.32:0/2829042807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47015bf20 0x7fd4700acd40 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 -- 172.21.15.32:0/2829042807 >> 172.21.15.32:0/2829042807 conn(0x7fd47006b3d0 msgr2=0x7fd47006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 -- 172.21.15.32:0/2829042807 shutdown_connections 2024-03-06T07:35:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.383+0000 7fd476c00640 1 -- 172.21.15.32:0/2829042807 wait complete. 2024-03-06T07:35:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.385+0000 7fd476c00640 1 Processor -- start 2024-03-06T07:35:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.385+0000 7fd476c00640 1 -- start start 2024-03-06T07:35:27.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd476c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 0x7fd47006c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd476c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd470072520 0x7fd4700b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd476c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd47015bf20 0x7fd4700b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd476c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700b4b10 con 0x7fd470071e20 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd476c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700b4990 con 0x7fd47015bf20 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd476c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700ce6c0 con 0x7fd470072520 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd46fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 0x7fd47006c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd46fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 0x7fd47006c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50510/0 (socket says 172.21.15.32:50510) 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.386+0000 7fd46fe00640 1 -- 172.21.15.32:0/2904893339 learned_addr learned my addr 172.21.15.32:0/2904893339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46f400640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd470072520 0x7fd4700b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd474c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd47015bf20 0x7fd4700b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46fe00640 1 -- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd470072520 msgr2=0x7fd4700b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46fe00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd470072520 0x7fd4700b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46fe00640 1 -- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd47015bf20 msgr2=0x7fd4700b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46fe00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd47015bf20 0x7fd4700b2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46fe00640 1 -- 172.21.15.32:0/2904893339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4701557b0 con 0x7fd470071e20 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46f400640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd470072520 0x7fd4700b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd46fe00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 0x7fd47006c9c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fd460037040 tx=0x7fd460004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460039040 con 0x7fd470071e20 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460003b90 con 0x7fd470071e20 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.387+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460004030 con 0x7fd470071e20 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.388+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd470155a50 con 0x7fd470071e20 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.388+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4700c92c0 con 0x7fd470071e20 2024-03-06T07:35:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.389+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47004f330 con 0x7fd470071e20 2024-03-06T07:35:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.390+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd46002cb20 con 0x7fd470071e20 2024-03-06T07:35:27.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.391+0000 7fd467e00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.391+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4600ad520 con 0x7fd470071e20 2024-03-06T07:35:27.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.391+0000 7fd46f400640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.392+0000 7fd46f400640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd44c06b4d0 0x7fd44c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd4701f7540 tx=0x7fd458009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.395+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd46007c300 con 0x7fd470071e20 2024-03-06T07:35:27.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.596+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd47015f720 con 0x7fd470071e20 2024-03-06T07:35:27.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.597+0000 7fd467e00640 1 -- 172.21.15.32:0/2904893339 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd46007bc00 con 0x7fd470071e20 2024-03-06T07:35:27.598 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd44c06b4d0 msgr2=0x7fd44c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd44c06b4d0 0x7fd44c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd4701f7540 tx=0x7fd458009040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 msgr2=0x7fd47006c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 0x7fd47006c9c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fd460037040 tx=0x7fd460004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 shutdown_connections 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd47015bf20 0x7fd4700b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd470072520 0x7fd4700b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 --2- 172.21.15.32:0/2904893339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd470071e20 0x7fd47006c9c0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 >> 172.21.15.32:0/2904893339 conn(0x7fd47006b3d0 msgr2=0x7fd470159920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 shutdown_connections 2024-03-06T07:35:27.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.600+0000 7fd476c00640 1 -- 172.21.15.32:0/2904893339 wait complete. 2024-03-06T07:35:27.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.831+0000 7f7f66c00640 1 Processor -- start 2024-03-06T07:35:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.831+0000 7f7f66c00640 1 -- start start 2024-03-06T07:35:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.833+0000 7f7f66c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 0x7f7f6015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.833+0000 7f7f66c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7f6015cc90 0x7f7f600b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.833+0000 7f7f66c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f600b3fe0 con 0x7f7f600b2080 2024-03-06T07:35:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.833+0000 7f7f66c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6015fe40 con 0x7f7f6015cc90 2024-03-06T07:35:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.833+0000 7f7f66c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60160120 con 0x7f7f60145430 2024-03-06T07:35:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.834+0000 7f7f65800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 0x7f7f6015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.834+0000 7f7f65800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 0x7f7f6015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:43246/0 (socket says 172.21.15.32:43246) 2024-03-06T07:35:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.834+0000 7f7f65800640 1 -- 172.21.15.32:0/2187277936 learned_addr learned my addr 172.21.15.32:0/2187277936 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.834+0000 7f7f64e00640 1 --2- 172.21.15.32:0/2187277936 >> v2:172.21.15.39:3300/0 conn(0x7f7f6015cc90 0x7f7f600b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f5fe00640 1 -- 172.21.15.32:0/2187277936 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243712808 0 0) 0x7f7f600b3fe0 con 0x7f7f600b2080 2024-03-06T07:35:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f5fe00640 1 -- 172.21.15.32:0/2187277936 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f44003520 con 0x7f7f600b2080 2024-03-06T07:35:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f65800640 1 -- 172.21.15.32:0/2187277936 >> v2:172.21.15.39:3300/0 conn(0x7f7f6015cc90 msgr2=0x7f7f600b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f65800640 1 --2- 172.21.15.32:0/2187277936 >> v2:172.21.15.39:3300/0 conn(0x7f7f6015cc90 0x7f7f600b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f65800640 1 -- 172.21.15.32:0/2187277936 >> v1:172.21.15.39:6789/0 conn(0x7f7f600b2080 legacy=0x7f7f6006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f65800640 1 -- 172.21.15.32:0/2187277936 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f60160400 con 0x7f7f60145430 2024-03-06T07:35:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.835+0000 7f7f64e00640 1 --2- 172.21.15.32:0/2187277936 >> v2:172.21.15.39:3300/0 conn(0x7f7f6015cc90 0x7f7f600b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.837+0000 7f7f65800640 1 --2- 172.21.15.32:0/2187277936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 0x7f7f6015f810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009870 tx=0x7f7f5002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88fd31a9299c2de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.837+0000 7f7f5fe00640 1 -- 172.21.15.32:0/2187277936 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50034070 con 0x7f7f60145430 2024-03-06T07:35:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.838+0000 7f7f5fe00640 1 -- 172.21.15.32:0/2187277936 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f50030840 con 0x7f7f60145430 2024-03-06T07:35:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.838+0000 7f7f5fe00640 1 -- 172.21.15.32:0/2187277936 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50030bf0 con 0x7f7f60145430 2024-03-06T07:35:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.838+0000 7f7f66c00640 1 -- 172.21.15.32:0/2187277936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 msgr2=0x7f7f6015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.838+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2187277936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 0x7f7f6015f810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009870 tx=0x7f7f5002b980 comp rx=0 tx=0).stop 2024-03-06T07:35:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.839+0000 7f7f66c00640 1 -- 172.21.15.32:0/2187277936 shutdown_connections 2024-03-06T07:35:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.839+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2187277936 >> v2:172.21.15.39:3300/0 conn(0x7f7f6015cc90 0x7f7f600b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.839+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2187277936 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f60145430 0x7f7f6015f810 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.839+0000 7f7f66c00640 1 -- 172.21.15.32:0/2187277936 >> 172.21.15.32:0/2187277936 conn(0x7f7f6006b3a0 msgr2=0x7f7f6006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.839+0000 7f7f66c00640 1 -- 172.21.15.32:0/2187277936 shutdown_connections 2024-03-06T07:35:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.839+0000 7f7f66c00640 1 -- 172.21.15.32:0/2187277936 wait complete. 2024-03-06T07:35:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.841+0000 7f7f66c00640 1 Processor -- start 2024-03-06T07:35:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.841+0000 7f7f66c00640 1 -- start start 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.842+0000 7f7f66c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 0x7f7f601f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.842+0000 7f7f66c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f60145430 0x7f7f600aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.842+0000 7f7f66c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f6015cc90 0x7f7f6015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.842+0000 7f7f66c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60145810 con 0x7f7f600b2080 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.842+0000 7f7f66c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6015d0d0 con 0x7f7f60145430 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.842+0000 7f7f66c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60145990 con 0x7f7f6015cc90 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 0x7f7f601f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 0x7f7f601f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50516/0 (socket says 172.21.15.32:50516) 2024-03-06T07:35:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 -- 172.21.15.32:0/2780027762 learned_addr learned my addr 172.21.15.32:0/2780027762 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f66200640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f6015cc90 0x7f7f6015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f64e00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f60145430 0x7f7f600aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 -- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f6015cc90 msgr2=0x7f7f6015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f6015cc90 0x7f7f6015b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 -- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f60145430 msgr2=0x7f7f600aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f60145430 0x7f7f600aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f65800640 1 -- 172.21.15.32:0/2780027762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f60143790 con 0x7f7f600b2080 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.843+0000 7f7f64e00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f60145430 0x7f7f600aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.844+0000 7f7f65800640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 0x7f7f601f4050 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009870 tx=0x7f7f50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.844+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50048070 con 0x7f7f600b2080 2024-03-06T07:35:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.844+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f50030660 con 0x7f7f600b2080 2024-03-06T07:35:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.844+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50034040 con 0x7f7f600b2080 2024-03-06T07:35:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.844+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f60143a70 con 0x7f7f600b2080 2024-03-06T07:35:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.844+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f60144030 con 0x7f7f600b2080 2024-03-06T07:35:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.845+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6004f330 con 0x7f7f600b2080 2024-03-06T07:35:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.847+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f50037070 con 0x7f7f600b2080 2024-03-06T07:35:27.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.847+0000 7f7f5d600640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f4006b4d0 0x7f7f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.848+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f500abda0 con 0x7f7f600b2080 2024-03-06T07:35:27.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.848+0000 7f7f64e00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f4006b4d0 0x7f7f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.848+0000 7f7f64e00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f4006b4d0 0x7f7f4006d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7f600ced40 tx=0x7f7f4c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:27.852+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f50030810 con 0x7f7f600b2080 2024-03-06T07:35:28.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.055+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f6006e240 con 0x7f7f600b2080 2024-03-06T07:35:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.056+0000 7f7f5d600640 1 -- 172.21.15.32:0/2780027762 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f5007ad30 con 0x7f7f600b2080 2024-03-06T07:35:28.057 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f4006b4d0 msgr2=0x7f7f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f4006b4d0 0x7f7f4006d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7f600ced40 tx=0x7f7f4c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 msgr2=0x7f7f601f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 0x7f7f601f4050 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009870 tx=0x7f7f50004730 comp rx=0 tx=0).stop 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 shutdown_connections 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f4006b4d0 0x7f7f4006d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f6015cc90 0x7f7f6015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f60145430 0x7f7f600aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 --2- 172.21.15.32:0/2780027762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f600b2080 0x7f7f601f4050 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 >> 172.21.15.32:0/2780027762 conn(0x7f7f6006b3a0 msgr2=0x7f7f600ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 shutdown_connections 2024-03-06T07:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.059+0000 7f7f66c00640 1 -- 172.21.15.32:0/2780027762 wait complete. 2024-03-06T07:35:28.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:35:28.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.301+0000 7f8a47200640 1 Processor -- start 2024-03-06T07:35:28.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.302+0000 7f8a47200640 1 -- start start 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.303+0000 7f8a47200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8a401552b0 0x7f8a4015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.303+0000 7f8a47200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a400cd060 0x7f8a400b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.303+0000 7f8a47200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4015ed10 con 0x7f8a4015e810 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.303+0000 7f8a47200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4015eff0 con 0x7f8a401552b0 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.303+0000 7f8a47200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4015f2d0 con 0x7f8a400cd060 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8a401552b0 0x7f8a4015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8a401552b0 0x7f8a4015b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50310/0 (socket says 172.21.15.32:50310) 2024-03-06T07:35:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 -- 172.21.15.32:0/3227508273 learned_addr learned my addr 172.21.15.32:0/3227508273 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a46800640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a400cd060 0x7f8a400b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 -- 172.21.15.32:0/3227508273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a400cd060 msgr2=0x7f8a400b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a400cd060 0x7f8a400b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 -- 172.21.15.32:0/3227508273 >> v1:172.21.15.39:6789/0 conn(0x7f8a4015e810 legacy=0x7f8a400cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a45e00640 1 -- 172.21.15.32:0/3227508273 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4015f5b0 con 0x7f8a401552b0 2024-03-06T07:35:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.304+0000 7f8a46800640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a400cd060 0x7f8a400b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.306+0000 7f8a45e00640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a401552b0 0x7f8a4015b1d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009870 tx=0x7f8a3402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2366855fb3aa42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.306+0000 7f8a44a00640 1 -- 172.21.15.32:0/3227508273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a340317c0 con 0x7f8a401552b0 2024-03-06T07:35:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.306+0000 7f8a44a00640 1 -- 172.21.15.32:0/3227508273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a34037070 con 0x7f8a401552b0 2024-03-06T07:35:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.306+0000 7f8a44a00640 1 -- 172.21.15.32:0/3227508273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34031b80 con 0x7f8a401552b0 2024-03-06T07:35:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 -- 172.21.15.32:0/3227508273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a401552b0 msgr2=0x7f8a4015b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a401552b0 0x7f8a4015b1d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009870 tx=0x7f8a3402b860 comp rx=0 tx=0).stop 2024-03-06T07:35:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 -- 172.21.15.32:0/3227508273 shutdown_connections 2024-03-06T07:35:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a400cd060 0x7f8a400b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 --2- 172.21.15.32:0/3227508273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a401552b0 0x7f8a4015b1d0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 -- 172.21.15.32:0/3227508273 >> 172.21.15.32:0/3227508273 conn(0x7f8a4006b3d0 msgr2=0x7f8a4006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 -- 172.21.15.32:0/3227508273 shutdown_connections 2024-03-06T07:35:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.307+0000 7f8a47200640 1 -- 172.21.15.32:0/3227508273 wait complete. 2024-03-06T07:35:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.309+0000 7f8a47200640 1 Processor -- start 2024-03-06T07:35:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.309+0000 7f8a47200640 1 -- start start 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.310+0000 7f8a47200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a400cd060 0x7f8a4014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.310+0000 7f8a47200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a401552b0 0x7f8a4015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.310+0000 7f8a47200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 0x7f8a4015da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.310+0000 7f8a47200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4006dd30 con 0x7f8a4015e810 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.310+0000 7f8a47200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400b3760 con 0x7f8a400cd060 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.310+0000 7f8a47200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4006deb0 con 0x7f8a401552b0 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 0x7f8a4015da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 0x7f8a4015da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50324/0 (socket says 172.21.15.32:50324) 2024-03-06T07:35:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 -- 172.21.15.32:0/3188753324 learned_addr learned my addr 172.21.15.32:0/3188753324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a45e00640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a400cd060 0x7f8a4014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a45400640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a401552b0 0x7f8a4015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 -- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a401552b0 msgr2=0x7f8a4015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a401552b0 0x7f8a4015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 -- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a400cd060 msgr2=0x7f8a4014c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a400cd060 0x7f8a4014c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.311+0000 7f8a46800640 1 -- 172.21.15.32:0/3188753324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4015a920 con 0x7f8a4015e810 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.312+0000 7f8a46800640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 0x7f8a4015da30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00ce70 tx=0x7f8a3c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.312+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c003b90 con 0x7f8a4015e810 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.312+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c003d50 con 0x7f8a4015e810 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.312+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c004260 con 0x7f8a4015e810 2024-03-06T07:35:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.312+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4015e070 con 0x7f8a4015e810 2024-03-06T07:35:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.312+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4014d180 con 0x7f8a4015e810 2024-03-06T07:35:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.313+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4014cc90 con 0x7f8a4015e810 2024-03-06T07:35:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.314+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a3c004400 con 0x7f8a4015e810 2024-03-06T07:35:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.316+0000 7f8a3a000640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a2406b4d0 0x7f8a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.316+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a3c08ab70 con 0x7f8a4015e810 2024-03-06T07:35:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.316+0000 7f8a45e00640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a2406b4d0 0x7f8a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.317+0000 7f8a45e00640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a2406b4d0 0x7f8a2406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8a34036000 tx=0x7f8a3403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.320+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a3c0599f0 con 0x7f8a4015e810 2024-03-06T07:35:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.539+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8a4004f330 con 0x7f8a4015e810 2024-03-06T07:35:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.540+0000 7f8a3a000640 1 -- 172.21.15.32:0/3188753324 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8a3c0592f0 con 0x7f8a4015e810 2024-03-06T07:35:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a2406b4d0 msgr2=0x7f8a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a2406b4d0 0x7f8a2406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8a34036000 tx=0x7f8a3403d040 comp rx=0 tx=0).stop 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 msgr2=0x7f8a4015da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 0x7f8a4015da30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00ce70 tx=0x7f8a3c00a710 comp rx=0 tx=0).stop 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 shutdown_connections 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8a2406b4d0 0x7f8a2406d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.542+0000 7f8a47200640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a4015e810 0x7f8a4015da30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.543+0000 7f8a47200640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8a401552b0 0x7f8a4015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.543+0000 7f8a47200640 1 --2- 172.21.15.32:0/3188753324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a400cd060 0x7f8a4014c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.543+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 >> 172.21.15.32:0/3188753324 conn(0x7f8a4006b3d0 msgr2=0x7f8a400b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.543+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 shutdown_connections 2024-03-06T07:35:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.543+0000 7f8a47200640 1 -- 172.21.15.32:0/3188753324 wait complete. 2024-03-06T07:35:28.565 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:35:28.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:28.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:28.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:35:28.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:28.712 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:35:28.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:28.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.948+0000 7fb48da00640 1 Processor -- start 2024-03-06T07:35:28.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.949+0000 7fb48da00640 1 -- start start 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.950+0000 7fb48da00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb488161670 0x7fb488070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.950+0000 7fb48da00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48806de00 con 0x7fb4880ae0a0 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.950+0000 7fb48da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4880ce9a0 con 0x7fb48806e240 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.950+0000 7fb48da00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4880730d0 con 0x7fb488161670 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.950+0000 7fb487400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb4880ae0a0 0x7fb48806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48424/0 (socket says 172.21.15.32:48424) 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb487400640 1 -- 172.21.15.32:0/866797091 learned_addr learned my addr 172.21.15.32:0/866797091 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486a00640 1 --2- 172.21.15.32:0/866797091 >> v2:172.21.15.39:3300/0 conn(0x7fb488161670 0x7fb488070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583803336 0 0) 0x7fb48806de00 con 0x7fb4880ae0a0 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c003520 con 0x7fb4880ae0a0 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424259237 0 0) 0x7fb46c003520 con 0x7fb4880ae0a0 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 >> v2:172.21.15.39:3300/0 conn(0x7fb488161670 msgr2=0x7fb488070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 --2- 172.21.15.32:0/866797091 >> v2:172.21.15.39:3300/0 conn(0x7fb488161670 0x7fb488070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 >> v1:172.21.15.39:6789/0 conn(0x7fb48806e240 legacy=0x7fb48806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4880cec80 con 0x7fb4880ae0a0 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.951+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb4800019f0 con 0x7fb4880ae0a0 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 >> v1:172.21.15.32:6789/0 conn(0x7fb4880ae0a0 legacy=0x7fb48806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb46c003a20 0x7fb46c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb46c004800 0x7fb46c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c005230 0x7fb46c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880730d0 con 0x7fb46c004800 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c00c190 con 0x7fb46c005230 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c00c390 con 0x7fb46c003a20 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb47fe00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb46c004800 0x7fb46c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486a00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb46c003a20 0x7fb46c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb487400640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c005230 0x7fb46c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb47fe00640 1 -- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb46c003a20 msgr2=0x7fb46c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb47fe00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb46c003a20 0x7fb46c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb47fe00640 1 -- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c005230 msgr2=0x7fb46c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb47fe00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c005230 0x7fb46c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb47fe00640 1 -- 172.21.15.32:0/866797091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c003520 con 0x7fb46c004800 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb487400640 1 -- 172.21.15.32:0/866797091 reap_dead start 2024-03-06T07:35:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.952+0000 7fb486a00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb46c003a20 0x7fb46c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:28.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb47fe00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb46c004800 0x7fb46c004c80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fb470001750 tx=0x7fb46c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaec02122d50b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470001370 con 0x7fb46c004800 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4700052d0 con 0x7fb46c004800 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb486000640 1 -- 172.21.15.32:0/866797091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470003830 con 0x7fb46c004800 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb48da00640 1 -- 172.21.15.32:0/866797091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb46c004800 msgr2=0x7fb46c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb48da00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb46c004800 0x7fb46c004c80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fb470001750 tx=0x7fb46c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb48da00640 1 -- 172.21.15.32:0/866797091 shutdown_connections 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb48da00640 1 --2- 172.21.15.32:0/866797091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb46c004800 0x7fb46c004c80 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.954+0000 7fb48da00640 1 -- 172.21.15.32:0/866797091 >> 172.21.15.32:0/866797091 conn(0x7fb48806afb0 msgr2=0x7fb48806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.955+0000 7fb48da00640 1 -- 172.21.15.32:0/866797091 shutdown_connections 2024-03-06T07:35:28.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.955+0000 7fb48da00640 1 -- 172.21.15.32:0/866797091 wait complete. 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.956+0000 7fb48da00640 1 Processor -- start 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.956+0000 7fb48da00640 1 -- start start 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.957+0000 7fb48da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c004800 0x7fb488154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.957+0000 7fb48da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 0x7fb4880b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.957+0000 7fb48da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb48815a260 0x7fb4880b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.957+0000 7fb48da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880cf520 con 0x7fb488159e00 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.957+0000 7fb48da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880cf240 con 0x7fb46c004800 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.957+0000 7fb48da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4880ce600 con 0x7fb48815a260 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 0x7fb4880b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 0x7fb4880b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50336/0 (socket says 172.21.15.32:50336) 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 -- 172.21.15.32:0/1380833201 learned_addr learned my addr 172.21.15.32:0/1380833201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb486a00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c004800 0x7fb488154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb487400640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb48815a260 0x7fb4880b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 -- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb48815a260 msgr2=0x7fb4880b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb48815a260 0x7fb4880b0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 -- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c004800 msgr2=0x7fb488154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c004800 0x7fb488154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.958+0000 7fb47fe00640 1 -- 172.21.15.32:0/1380833201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4880af640 con 0x7fb488159e00 2024-03-06T07:35:28.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.959+0000 7fb47fe00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 0x7fb4880b06e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fb470005040 tx=0x7fb470001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.959+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470041020 con 0x7fb488159e00 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.959+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb470005480 con 0x7fb488159e00 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.959+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470007a30 con 0x7fb488159e00 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.959+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4880af920 con 0x7fb488159e00 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.959+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4880afee0 con 0x7fb488159e00 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.960+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb488068220 con 0x7fb488159e00 2024-03-06T07:35:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.961+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb470044020 con 0x7fb488159e00 2024-03-06T07:35:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.962+0000 7fb47ea00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb46406b4d0 0x7fb46406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.962+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4700abcd0 con 0x7fb488159e00 2024-03-06T07:35:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.962+0000 7fb486a00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb46406b4d0 0x7fb46406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.963+0000 7fb486a00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb46406b4d0 0x7fb46406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb488050c30 tx=0x7fb478008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:28.967+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb488068220 con 0x7fb488159e00 2024-03-06T07:35:29.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.211+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4880b52d0 con 0x7fb488159e00 2024-03-06T07:35:29.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.212+0000 7fb47ea00640 1 -- 172.21.15.32:0/1380833201 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fb47004b030 con 0x7fb488159e00 2024-03-06T07:35:29.214 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:29.214 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/3906463088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3906463088},{"type":"v1","addr":"172.21.15.39:6834","nonce":3906463088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71},{"gid":7647,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6836/2563322789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2563322789},{"type":"v1","addr":"172.21.15.32:6836","nonce":2563322789}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7650,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2154839965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":2154839965},{"type":"v1","addr":"172.21.15.32:6837","nonce":2154839965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:18.287992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7665":{"gid":7665,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/14889266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":14889266},{"type":"v1","addr":"172.21.15.39:6835","nonce":14889266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7665,"qdb_cluster":[7665]},"id":8}]} 2024-03-06T07:35:29.215 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb46406b4d0 msgr2=0x7fb46406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb46406b4d0 0x7fb46406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb488050c30 tx=0x7fb478008040 comp rx=0 tx=0).stop 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 msgr2=0x7fb4880b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 0x7fb4880b06e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fb470005040 tx=0x7fb470001120 comp rx=0 tx=0).stop 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 shutdown_connections 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb46406b4d0 0x7fb46406d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb48815a260 0x7fb4880b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb488159e00 0x7fb4880b06e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 --2- 172.21.15.32:0/1380833201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb46c004800 0x7fb488154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.215+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 >> 172.21.15.32:0/1380833201 conn(0x7fb48806afb0 msgr2=0x7fb4880cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.216+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 shutdown_connections 2024-03-06T07:35:29.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.216+0000 7fb48da00640 1 -- 172.21.15.32:0/1380833201 wait complete. 2024-03-06T07:35:29.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-06T07:35:29.485 INFO:teuthology.orchestra.run.smithi039.stdout:{"message":"","return_code":0} 2024-03-06T07:35:29.485 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:29.486 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:35:29.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:29.486 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:35:29.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:29.487 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:35:29.487 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:29.487 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:35:29.487 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:29.503 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:35:29.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:35:29.505 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:35:29.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:35:29.510 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:35:29.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:35:29.516 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:35:29.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:35:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.702+0000 7f14c9a00640 1 Processor -- start 2024-03-06T07:35:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.703+0000 7f14c9a00640 1 -- start start 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c9a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f14c40af1a0 0x7f14c415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c9a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c40cdc20 0x7f14c40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c9a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c415cc40 con 0x7f14c40af580 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c9a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c415cf20 con 0x7f14c40cdc20 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c9a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c415d200 con 0x7f14c40af1a0 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c40cdc20 0x7f14c40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c40cdc20 0x7f14c40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50342/0 (socket says 172.21.15.32:50342) 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.704+0000 7f14c3400640 1 -- 172.21.15.32:0/2153855121 learned_addr learned my addr 172.21.15.32:0/2153855121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.705+0000 7f14c2a00640 1 --2- 172.21.15.32:0/2153855121 >> v2:172.21.15.39:3300/0 conn(0x7f14c40af1a0 0x7f14c415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.705+0000 7f14c3400640 1 -- 172.21.15.32:0/2153855121 >> v2:172.21.15.39:3300/0 conn(0x7f14c40af1a0 msgr2=0x7f14c415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.705+0000 7f14c3400640 1 --2- 172.21.15.32:0/2153855121 >> v2:172.21.15.39:3300/0 conn(0x7f14c40af1a0 0x7f14c415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.705+0000 7f14c3400640 1 -- 172.21.15.32:0/2153855121 >> v1:172.21.15.39:6789/0 conn(0x7f14c40af580 legacy=0x7f14c40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.705+0000 7f14c3400640 1 -- 172.21.15.32:0/2153855121 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c415d4e0 con 0x7f14c40cdc20 2024-03-06T07:35:29.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.707+0000 7fe325000640 1 Processor -- start 2024-03-06T07:35:29.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.707+0000 7f14c3400640 1 --2- 172.21.15.32:0/2153855121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40cdc20 0x7f14c40b2dd0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f14c4074960 tx=0x7f14bc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78170e3e832ab0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.707+0000 7fe325000640 1 -- start start 2024-03-06T07:35:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.707+0000 7f14c1600640 1 -- 172.21.15.32:0/2153855121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc02c9f0 con 0x7f14c40cdc20 2024-03-06T07:35:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.707+0000 7f14c1600640 1 -- 172.21.15.32:0/2153855121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc02cb90 con 0x7f14c40cdc20 2024-03-06T07:35:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.707+0000 7f14c1600640 1 -- 172.21.15.32:0/2153855121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0343f0 con 0x7f14c40cdc20 2024-03-06T07:35:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7f14c9a00640 1 -- 172.21.15.32:0/2153855121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40cdc20 msgr2=0x7f14c40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2153855121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40cdc20 0x7f14c40b2dd0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f14c4074960 tx=0x7f14bc02be50 comp rx=0 tx=0).stop 2024-03-06T07:35:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7f14c9a00640 1 -- 172.21.15.32:0/2153855121 shutdown_connections 2024-03-06T07:35:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2153855121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40cdc20 0x7f14c40b2dd0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2153855121 >> v2:172.21.15.39:3300/0 conn(0x7f14c40af1a0 0x7f14c415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7f14c9a00640 1 -- 172.21.15.32:0/2153855121 >> 172.21.15.32:0/2153855121 conn(0x7f14c406afb0 msgr2=0x7f14c406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7fe325000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe32015e200 0x7fe3200b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7fe325000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3200afdf0 0x7fe3200af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7fe325000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32006bec0 con 0x7fe32006afb0 2024-03-06T07:35:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7fe325000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3200b0670 con 0x7fe32015e200 2024-03-06T07:35:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.708+0000 7fe325000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200ae380 con 0x7fe3200afdf0 2024-03-06T07:35:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7f14c9a00640 1 -- 172.21.15.32:0/2153855121 shutdown_connections 2024-03-06T07:35:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7f14c9a00640 1 -- 172.21.15.32:0/2153855121 wait complete. 2024-03-06T07:35:29.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7fe31de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe32015e200 0x7fe3200b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7fe31de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe32015e200 0x7fe3200b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:45360/0 (socket says 172.21.15.32:45360) 2024-03-06T07:35:29.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7fe31de00640 1 -- 172.21.15.32:0/3984591507 learned_addr learned my addr 172.21.15.32:0/3984591507 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:29.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7fe31d400640 1 --2- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3200afdf0 0x7fe3200af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.709+0000 7f00a7600640 1 Processor -- start 2024-03-06T07:35:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.710+0000 7fe31de00640 1 -- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3200afdf0 msgr2=0x7fe3200af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.710+0000 7fe31de00640 1 --2- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3200afdf0 0x7fe3200af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.710+0000 7fe31de00640 1 -- 172.21.15.32:0/3984591507 >> v1:172.21.15.39:6789/0 conn(0x7fe32006afb0 legacy=0x7fe32006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.710+0000 7fe31de00640 1 -- 172.21.15.32:0/3984591507 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200ae660 con 0x7fe32015e200 2024-03-06T07:35:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.710+0000 7f00a7600640 1 -- start start 2024-03-06T07:35:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.711+0000 7fdc4de00640 1 Processor -- start 2024-03-06T07:35:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.711+0000 7f14c9a00640 1 Processor -- start 2024-03-06T07:35:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.711+0000 7fdc4de00640 1 -- start start 2024-03-06T07:35:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.711+0000 7f14c9a00640 1 -- start start 2024-03-06T07:35:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.711+0000 7fe31de00640 1 --2- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe3200b3de0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe310009870 tx=0x7fe31002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d36edb52d8506ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f00a7600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00a0072da0 0x7f00a0162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f00a7600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00ce090 con 0x7f00a00712c0 2024-03-06T07:35:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f00a7600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00ce370 con 0x7f00a0070ee0 2024-03-06T07:35:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f00a7600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a00ceca0 con 0x7f00a0072da0 2024-03-06T07:35:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7fe31ca00640 1 -- 172.21.15.32:0/3984591507 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310034070 con 0x7fe32015e200 2024-03-06T07:35:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f14c9a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c40af1a0 0x7f14c406dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f14c9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 0x7f14c415e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f14c9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c40cdc20 0x7f14c415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f14c9a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b36a0 con 0x7f14c40af580 2024-03-06T07:35:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f14c9a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b3520 con 0x7f14c40cdc20 2024-03-06T07:35:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f14c9a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b3820 con 0x7f14c40af1a0 2024-03-06T07:35:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7fe31ca00640 1 -- 172.21.15.32:0/3984591507 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe310030840 con 0x7fe32015e200 2024-03-06T07:35:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7fe31ca00640 1 -- 172.21.15.32:0/3984591507 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310030c10 con 0x7fe32015e200 2024-03-06T07:35:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f00a4c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f00a0070ee0 0x7f00a006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:50544/0 (socket says 172.21.15.32:50544) 2024-03-06T07:35:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.712+0000 7f00a4c00640 1 -- 172.21.15.32:0/2247623228 learned_addr learned my addr 172.21.15.32:0/2247623228 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f14c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 0x7f14c415e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 -- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 msgr2=0x7fe3200b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 --2- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe3200b3de0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe310009870 tx=0x7fe31002b980 comp rx=0 tx=0).stop 2024-03-06T07:35:29.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-06T07:35:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f14c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 0x7f14c415e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50360/0 (socket says 172.21.15.32:50360) 2024-03-06T07:35:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f14c2000640 1 -- 172.21.15.32:0/2476943367 learned_addr learned my addr 172.21.15.32:0/2476943367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 -- 172.21.15.32:0/3984591507 shutdown_connections 2024-03-06T07:35:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f009fe00640 1 --2- 172.21.15.32:0/2247623228 >> v2:172.21.15.32:3300/0 conn(0x7f00a0072da0 0x7f00a0162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f14c2a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c40af1a0 0x7f14c406dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 --2- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe3200afdf0 0x7fe3200af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 --2- 172.21.15.32:0/3984591507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe3200b3de0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 -- 172.21.15.32:0/3984591507 >> 172.21.15.32:0/3984591507 conn(0x7fe3200b2ae0 msgr2=0x7fe3200b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327826744 0 0) 0x7f00a00ce090 con 0x7f00a00712c0 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0088003520 con 0x7f00a00712c0 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524699415 0 0) 0x7f00a00ce370 con 0x7f00a0070ee0 2024-03-06T07:35:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a00ce090 con 0x7f00a0070ee0 2024-03-06T07:35:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 -- 172.21.15.32:0/3984591507 shutdown_connections 2024-03-06T07:35:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.713+0000 7fe325000640 1 -- 172.21.15.32:0/3984591507 wait complete. 2024-03-06T07:35:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c3400640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c40cdc20 0x7f14c415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469283314 0 0) 0x7f0088003520 con 0x7f00a00712c0 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 >> v2:172.21.15.32:3300/0 conn(0x7f00a0072da0 msgr2=0x7f00a0162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 --2- 172.21.15.32:0/2247623228 >> v2:172.21.15.32:3300/0 conn(0x7f00a0072da0 0x7f00a0162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 >> v1:172.21.15.39:6789/0 conn(0x7f00a0070ee0 legacy=0x7f00a006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00ce650 con 0x7f00a00712c0 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c2000640 1 -- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c40af1a0 msgr2=0x7f14c406dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c2000640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c40af1a0 0x7f14c406dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c2000640 1 -- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c40cdc20 msgr2=0x7f14c415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c2000640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c40cdc20 0x7f14c415edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c2000640 1 -- 172.21.15.32:0/2476943367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40c93a0 con 0x7f14c40af580 2024-03-06T07:35:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f00980019f0 con 0x7f00a00712c0 2024-03-06T07:35:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 >> v1:172.21.15.32:6789/0 conn(0x7f00a00712c0 legacy=0x7f00a006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0088004c60 0x7f0088005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f0088003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7fdc4de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdc4815e080 0x7fdc4806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7fdc4de00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc480cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7fdc4de00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4806c920 con 0x7fdc4806de20 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7fdc4de00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4806d530 con 0x7fdc4815e080 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7fdc4de00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4806cc00 con 0x7fdc480d2340 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0088004280 0x7f008800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00ceca0 con 0x7f0088004c60 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008800c1e0 con 0x7f00880037a0 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008800c3e0 con 0x7f0088004280 2024-03-06T07:35:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7f00a4c00640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0088004c60 0x7f0088005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7fdc46a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdc4815e080 0x7fdc4806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.714+0000 7f14c2a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c40af1a0 0x7f14c406dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7f14c2000640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 0x7f14c415e880 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c430 tx=0x7f14b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7fdc46000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdc4806de20 0x7fdc480d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48436/0 (socket says 172.21.15.32:48436) 2024-03-06T07:35:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7fe325000640 1 Processor -- start 2024-03-06T07:35:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7fdc46000640 1 -- 172.21.15.32:0/2825844982 learned_addr learned my addr 172.21.15.32:0/2825844982 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7fdc47400640 1 --2- 172.21.15.32:0/2825844982 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc480cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7fe325000640 1 -- start start 2024-03-06T07:35:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7f009fe00640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f0088003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7f00a5600640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0088004280 0x7f008800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.715+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0011790 con 0x7f14c40af580 2024-03-06T07:35:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0017070 con 0x7f14c40af580 2024-03-06T07:35:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c40c9680 con 0x7f14c40af580 2024-03-06T07:35:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c40c9c40 con 0x7f14c40af580 2024-03-06T07:35:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0011b70 con 0x7f14c40af580 2024-03-06T07:35:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fe325000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe32006afb0 0x7fe32015ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fdc46a00640 1 -- 172.21.15.32:0/2825844982 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 msgr2=0x7fdc480cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fe325000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3200afdf0 0x7fe320154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fe325000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe32015f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fe325000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200af680 con 0x7fe3200afdf0 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fe325000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200aec20 con 0x7fe32015e200 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fe325000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32006e2c0 con 0x7fe32006afb0 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fdc46a00640 1 --2- 172.21.15.32:0/2825844982 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc480cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fdc46a00640 1 -- 172.21.15.32:0/2825844982 >> v1:172.21.15.32:6789/0 conn(0x7fdc4806de20 legacy=0x7fdc480d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fdc46a00640 1 -- 172.21.15.32:0/2825844982 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4806cee0 con 0x7fdc4815e080 2024-03-06T07:35:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.716+0000 7fdc45600640 1 -- 172.21.15.32:0/2825844982 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420193411 0 0) 0x7fdc4806c920 con 0x7fdc4806de20 2024-03-06T07:35:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7fe31d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3200afdf0 0x7fe320154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7fe31d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3200afdf0 0x7fe320154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50374/0 (socket says 172.21.15.32:50374) 2024-03-06T07:35:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7fe31d400640 1 -- 172.21.15.32:0/4141843267 learned_addr learned my addr 172.21.15.32:0/4141843267 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7fe31e800640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe32015f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f009fe00640 1 -- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0088004280 msgr2=0x7f008800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f009fe00640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0088004280 0x7f008800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f009fe00640 1 -- 172.21.15.32:0/2247623228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0088004c60 msgr2=0x7f0088005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f009fe00640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0088004c60 0x7f0088005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f009fe00640 1 -- 172.21.15.32:0/2247623228 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0088003520 con 0x7f00880037a0 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f00a4c00640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0088004c60 0x7f0088005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.717+0000 7f00a5600640 1 -- 172.21.15.32:0/2247623228 reap_dead start 2024-03-06T07:35:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7fe31e800640 1 -- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe32006afb0 msgr2=0x7fe32015ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:35:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7fe31e800640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe32006afb0 0x7fe32015ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7fe31e800640 1 -- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3200afdf0 msgr2=0x7fe320154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7fe31e800640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3200afdf0 0x7fe320154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7fe31e800640 1 -- 172.21.15.32:0/4141843267 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320155000 con 0x7fe32015e200 2024-03-06T07:35:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149c005290 con 0x7f14c40af580 2024-03-06T07:35:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.718+0000 7fdc46a00640 1 --2- 172.21.15.32:0/2825844982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc4815e080 0x7fdc4806c0c0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fdc4806d6b0 tx=0x7fdc3802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1421c37a98ca8736 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fe31e800640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe32015f8a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe32006c090 tx=0x7fe31800cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fdc45600640 1 -- 172.21.15.32:0/2825844982 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38031780 con 0x7fdc4815e080 2024-03-06T07:35:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fdc45600640 1 -- 172.21.15.32:0/2825844982 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc38037070 con 0x7fdc4815e080 2024-03-06T07:35:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fdc45600640 1 -- 172.21.15.32:0/2825844982 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38031b40 con 0x7fdc4815e080 2024-03-06T07:35:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7f009fe00640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f0088003c20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f008c00ba00 tx=0x7f0088005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5628e6e9d6322cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318003b90 con 0x7fe32015e200 2024-03-06T07:35:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32015b4a0 con 0x7fe32015e200 2024-03-06T07:35:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318003d30 con 0x7fe32015e200 2024-03-06T07:35:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318005310 con 0x7fe32015e200 2024-03-06T07:35:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c00b600 con 0x7f00880037a0 2024-03-06T07:35:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7f009f400640 1 -- 172.21.15.32:0/2247623228 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c038070 con 0x7f00880037a0 2024-03-06T07:35:29.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7fdc4de00640 1 -- 172.21.15.32:0/2825844982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc4815e080 msgr2=0x7fdc4806c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7fdc4de00640 1 --2- 172.21.15.32:0/2825844982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc4815e080 0x7fdc4806c0c0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fdc4806d6b0 tx=0x7fdc3802b820 comp rx=0 tx=0).stop 2024-03-06T07:35:29.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14b0011d10 con 0x7f14c40af580 2024-03-06T07:35:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7fdc4de00640 1 -- 172.21.15.32:0/2825844982 shutdown_connections 2024-03-06T07:35:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7fdc4de00640 1 --2- 172.21.15.32:0/2825844982 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc480cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7fdc4de00640 1 --2- 172.21.15.32:0/2825844982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc4815e080 0x7fdc4806c0c0 secure :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fdc4806d6b0 tx=0x7fdc3802b820 comp rx=0 tx=0).stop 2024-03-06T07:35:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.720+0000 7fdc4de00640 1 -- 172.21.15.32:0/2825844982 >> 172.21.15.32:0/2825844982 conn(0x7fdc4809aa10 msgr2=0x7fdc4809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.719+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3201542c0 con 0x7fe32015e200 2024-03-06T07:35:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 -- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 msgr2=0x7f0088003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f0088003c20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f008c00ba00 tx=0x7f0088005830 comp rx=0 tx=0).stop 2024-03-06T07:35:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f14baa00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a406b5a0 0x7f14a406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14b008a010 con 0x7f14c40af580 2024-03-06T07:35:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 -- 172.21.15.32:0/2247623228 shutdown_connections 2024-03-06T07:35:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32004f3a0 con 0x7fe32015e200 2024-03-06T07:35:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 --2- 172.21.15.32:0/2247623228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f0088003c20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 -- 172.21.15.32:0/2247623228 >> 172.21.15.32:0/2247623228 conn(0x7f00a006afb0 msgr2=0x7f00a006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7fdc4de00640 1 -- 172.21.15.32:0/2825844982 shutdown_connections 2024-03-06T07:35:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 -- 172.21.15.32:0/2247623228 shutdown_connections 2024-03-06T07:35:29.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7fdc4de00640 1 -- 172.21.15.32:0/2825844982 wait complete. 2024-03-06T07:35:29.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.721+0000 7f00a7600640 1 -- 172.21.15.32:0/2247623228 wait complete. 2024-03-06T07:35:29.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.722+0000 7f14c2a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a406b5a0 0x7f14a406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.723+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3180054b0 con 0x7fe32015e200 2024-03-06T07:35:29.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.723+0000 7f14c2a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a406b5a0 0x7f14a406da60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f14b4006fd0 tx=0x7f14b4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.723+0000 7fe316000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe30006b4d0 0x7fe30006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.724+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe31808aac0 con 0x7fe32015e200 2024-03-06T07:35:29.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.724+0000 7fe31de00640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe30006b4d0 0x7fe30006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.724+0000 7fdc4de00640 1 Processor -- start 2024-03-06T07:35:29.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.724+0000 7fdc4de00640 1 -- start start 2024-03-06T07:35:29.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.724+0000 7f00a7600640 1 Processor -- start 2024-03-06T07:35:29.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.725+0000 7fe31de00640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe30006b4d0 0x7fe30006d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe310036000 tx=0x7fe310008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.725+0000 7f00a7600640 1 -- start start 2024-03-06T07:35:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc4de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc4806de20 0x7fdc480b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc4de00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc481f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc4de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 0x7fdc4814d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc4de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4815e460 con 0x7fdc480b48e0 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc4de00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4806c600 con 0x7fdc4806de20 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc4de00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480d0290 con 0x7fdc480d2340 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc47400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 0x7fdc4814d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc47400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 0x7fdc4814d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50386/0 (socket says 172.21.15.32:50386) 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc47400640 1 -- 172.21.15.32:0/1783510742 learned_addr learned my addr 172.21.15.32:0/1783510742 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.726+0000 7fdc46a00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc4806de20 0x7fdc480b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a7600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f00a014fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc46000640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc481f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a7600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00a015f1e0 0x7f00a015f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 0x7f00a01eb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a7600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00cdda0 con 0x7f00a01eb000 2024-03-06T07:35:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a7600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00cdc20 con 0x7f00880037a0 2024-03-06T07:35:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a7600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0162c40 con 0x7f00a015f1e0 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc47400640 1 -- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 msgr2=0x7fdc481f7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc47400640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc481f7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc47400640 1 -- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc4806de20 msgr2=0x7fdc480b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc47400640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc4806de20 0x7fdc480b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc47400640 1 -- 172.21.15.32:0/1783510742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480b4d90 con 0x7fdc480b48e0 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc46000640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc481f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc46a00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc4806de20 0x7fdc480b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:29.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14b0058f00 con 0x7f14c40af580 2024-03-06T07:35:29.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 0x7f00a01eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 0x7f00a01eb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50396/0 (socket says 172.21.15.32:50396) 2024-03-06T07:35:29.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7f00a5600640 1 -- 172.21.15.32:0/2264139439 learned_addr learned my addr 172.21.15.32:0/2264139439 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:29.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.727+0000 7fdc47400640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 0x7fdc4814d230 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000ef90 tx=0x7fdc4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a4c00640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f00a014fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a5600640 1 -- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00a015f1e0 msgr2=0x7f00a015f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a5600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00a015f1e0 0x7f00a015f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a5600640 1 -- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 msgr2=0x7f00a014fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a5600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f00a014fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a5600640 1 -- 172.21.15.32:0/2264139439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a01ebbe0 con 0x7f00a01eb000 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40018020 con 0x7fdc480b48e0 2024-03-06T07:35:29.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc40014480 con 0x7fdc480b48e0 2024-03-06T07:35:29.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7f00a5600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 0x7f00a01eb480 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f00a0073180 tx=0x7f009800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40009e60 con 0x7fdc480b48e0 2024-03-06T07:35:29.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.728+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc480b78e0 con 0x7fdc480b48e0 2024-03-06T07:35:29.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.729+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48158700 con 0x7fdc480b48e0 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.729+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940015e0 con 0x7f00a01eb000 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.729+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098018070 con 0x7f00a01eb000 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.729+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009800dda0 con 0x7f00a01eb000 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.729+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a01ebec0 con 0x7f00a01eb000 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.729+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a01ec480 con 0x7f00a01eb000 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.731+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4804f3a0 con 0x7fdc480b48e0 2024-03-06T07:35:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.731+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0069510 con 0x7f00a01eb000 2024-03-06T07:35:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.731+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc40005d10 con 0x7fdc480b48e0 2024-03-06T07:35:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.732+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0098003970 con 0x7f00a01eb000 2024-03-06T07:35:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.732+0000 7fdc3f400640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc2806b520 0x7fdc2806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.732+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc4008ae20 con 0x7fdc480b48e0 2024-03-06T07:35:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.732+0000 7fdc46a00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc2806b520 0x7fdc2806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.733+0000 7f009cc00640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008006b520 0x7f008006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.733+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0098089e70 con 0x7f00a01eb000 2024-03-06T07:35:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.734+0000 7f00a4c00640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008006b520 0x7f008006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.734+0000 7fdc46a00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc2806b520 0x7fdc2806d9e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdc38036000 tx=0x7fdc3803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.734+0000 7f00a4c00640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008006b520 0x7f008006d9e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0094001b20 tx=0x7f0094009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.738+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc40022030 con 0x7fdc480b48e0 2024-03-06T07:35:29.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.740+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe318039030 con 0x7fe32015e200 2024-03-06T07:35:29.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:29.742+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f009801a310 con 0x7f00a01eb000 2024-03-06T07:35:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.007+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f149c005580 con 0x7f14c40af580 2024-03-06T07:35:30.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.011+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fdc480b7ba0 con 0x7fdc480b48e0 2024-03-06T07:35:30.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.018+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f00a004f3a0 con 0x7f00a01eb000 2024-03-06T07:35:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.019+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fe320073050 con 0x7fe32015e200 2024-03-06T07:35:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.860+0000 7f14baa00640 1 -- 172.21.15.32:0/2476943367 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6175 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7f14b0058800 con 0x7f14c40af580 2024-03-06T07:35:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 6175 2024-03-06T07:35:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.863+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a406b5a0 msgr2=0x7f14a406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.863+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a406b5a0 0x7f14a406da60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f14b4006fd0 tx=0x7f14b4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.863+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 msgr2=0x7f14c415e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.863+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 0x7f14c415e880 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c430 tx=0x7f14b000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 shutdown_connections 2024-03-06T07:35:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a406b5a0 0x7f14a406da60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c40cdc20 0x7f14c415edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40af580 0x7f14c415e880 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 --2- 172.21.15.32:0/2476943367 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c40af1a0 0x7f14c406dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 >> 172.21.15.32:0/2476943367 conn(0x7f14c406afb0 msgr2=0x7f14c415bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 shutdown_connections 2024-03-06T07:35:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.864+0000 7f14c9a00640 1 -- 172.21.15.32:0/2476943367 wait complete. 2024-03-06T07:35:30.885 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:35:30.884+0000 7fd1b1c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:35:30.890 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:35:30.890 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:35:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.890+0000 7fdc3f400640 1 -- 172.21.15.32:0/1783510742 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7650 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7fdc40010040 con 0x7fdc480b48e0 2024-03-06T07:35:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.890+0000 7f009cc00640 1 -- 172.21.15.32:0/2264139439 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7647 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7f0098058dd0 con 0x7f00a01eb000 2024-03-06T07:35:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 7650 2024-03-06T07:35:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 7647 2024-03-06T07:35:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc2806b520 msgr2=0x7fdc2806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc2806b520 0x7fdc2806d9e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdc38036000 tx=0x7fdc3803d040 comp rx=0 tx=0).stop 2024-03-06T07:35:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 msgr2=0x7fdc4814d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 0x7fdc4814d230 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000ef90 tx=0x7fdc4000c490 comp rx=0 tx=0).stop 2024-03-06T07:35:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008006b520 msgr2=0x7f008006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7f00a7600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008006b520 0x7f008006d9e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0094001b20 tx=0x7f0094009040 comp rx=0 tx=0).stop 2024-03-06T07:35:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 msgr2=0x7f00a01eb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7f00a7600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 0x7f00a01eb480 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f00a0073180 tx=0x7f009800d520 comp rx=0 tx=0).stop 2024-03-06T07:35:30.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 shutdown_connections 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdc2806b520 0x7fdc2806d9e0 secure :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fdc38036000 tx=0x7fdc3803d040 comp rx=0 tx=0).stop 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc480b48e0 0x7fdc4814d230 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdc480d2340 0x7fdc481f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 --2- 172.21.15.32:0/1783510742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc4806de20 0x7fdc480b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.894+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 >> 172.21.15.32:0/1783510742 conn(0x7fdc4809aa10 msgr2=0x7fdc480b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 shutdown_connections 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f008006b520 0x7f008006d9e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00a01eb000 0x7f00a01eb480 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00a015f1e0 0x7f00a015f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 --2- 172.21.15.32:0/2264139439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00880037a0 0x7f00a014fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 >> 172.21.15.32:0/2264139439 conn(0x7f00a006afb0 msgr2=0x7f00a0162120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 shutdown_connections 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 shutdown_connections 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7f00a7600640 1 -- 172.21.15.32:0/2264139439 wait complete. 2024-03-06T07:35:30.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.895+0000 7fdc4de00640 1 -- 172.21.15.32:0/1783510742 wait complete. 2024-03-06T07:35:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.899+0000 7fe316000640 1 -- 172.21.15.32:0/4141843267 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7665 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe3180598a0 con 0x7fe32015e200 2024-03-06T07:35:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 7665 2024-03-06T07:35:30.902 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:35:30.902 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe30006b4d0 msgr2=0x7fe30006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe30006b4d0 0x7fe30006d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe310036000 tx=0x7fe310008030 comp rx=0 tx=0).stop 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 msgr2=0x7fe32015f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe32015f8a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe32006c090 tx=0x7fe31800cb90 comp rx=0 tx=0).stop 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 shutdown_connections 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe30006b4d0 0x7fe30006d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe32015e200 0x7fe32015f8a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3200afdf0 0x7fe320154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 --2- 172.21.15.32:0/4141843267 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe32006afb0 0x7fe32015ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 >> 172.21.15.32:0/4141843267 conn(0x7fe3200b2ae0 msgr2=0x7fe320161b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 shutdown_connections 2024-03-06T07:35:30.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:30.902+0000 7fe325000640 1 -- 172.21.15.32:0/4141843267 wait complete. 2024-03-06T07:35:30.918 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:35:30.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:35:30.921 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:35:30.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:35:30.925 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:35:30.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:35:30.928 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:35:30.928 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:30.930 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:35:30.931 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:30.971 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:35:30.971 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:30.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:31.069 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:35:31.077 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:35:31.099 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:35:31.117 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:35:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.147+0000 7f606ae00640 1 Processor -- start 2024-03-06T07:35:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.147+0000 7f606ae00640 1 -- start start 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f606ae00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f606415efc0 0x7f60640cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f606ae00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60640cce90 0x7f6064161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f606ae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6064162000 0x7f60640ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f606ae00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606415f3a0 con 0x7f60640cce90 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f606ae00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606406d190 con 0x7f606415efc0 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f606ae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60641627f0 con 0x7f6064162000 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f6069000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60640cce90 0x7f6064161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f6069000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60640cce90 0x7f6064161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50424/0 (socket says 172.21.15.32:50424) 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.148+0000 7f6069000640 1 -- 172.21.15.32:0/908965101 learned_addr learned my addr 172.21.15.32:0/908965101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:31.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f606a400640 1 --2- 172.21.15.32:0/908965101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6064162000 0x7f60640ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069a00640 1 --2- 172.21.15.32:0/908965101 >> v2:172.21.15.39:3300/0 conn(0x7f606415efc0 0x7f60640cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069000640 1 -- 172.21.15.32:0/908965101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6064162000 msgr2=0x7f60640ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069000640 1 --2- 172.21.15.32:0/908965101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6064162000 0x7f60640ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069000640 1 -- 172.21.15.32:0/908965101 >> v2:172.21.15.39:3300/0 conn(0x7f606415efc0 msgr2=0x7f60640cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069000640 1 --2- 172.21.15.32:0/908965101 >> v2:172.21.15.39:3300/0 conn(0x7f606415efc0 0x7f60640cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069000640 1 -- 172.21.15.32:0/908965101 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606406c860 con 0x7f60640cce90 2024-03-06T07:35:31.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.149+0000 7f6069a00640 1 --2- 172.21.15.32:0/908965101 >> v2:172.21.15.39:3300/0 conn(0x7f606415efc0 0x7f60640cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.150+0000 7f6069000640 1 --2- 172.21.15.32:0/908965101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60640cce90 0x7f6064161ac0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f60640cd770 tx=0x7f605002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6395d621f5e4fa39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.150+0000 7f6063e00640 1 -- 172.21.15.32:0/908965101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050035020 con 0x7f60640cce90 2024-03-06T07:35:31.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.150+0000 7f6063e00640 1 -- 172.21.15.32:0/908965101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6050031870 con 0x7f60640cce90 2024-03-06T07:35:31.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.150+0000 7f6063e00640 1 -- 172.21.15.32:0/908965101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050031c40 con 0x7f60640cce90 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 -- 172.21.15.32:0/908965101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60640cce90 msgr2=0x7f6064161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 --2- 172.21.15.32:0/908965101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60640cce90 0x7f6064161ac0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f60640cd770 tx=0x7f605002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 -- 172.21.15.32:0/908965101 shutdown_connections 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 --2- 172.21.15.32:0/908965101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6064162000 0x7f60640ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 --2- 172.21.15.32:0/908965101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60640cce90 0x7f6064161ac0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 --2- 172.21.15.32:0/908965101 >> v2:172.21.15.39:3300/0 conn(0x7f606415efc0 0x7f60640cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 -- 172.21.15.32:0/908965101 >> 172.21.15.32:0/908965101 conn(0x7f606406b3d0 msgr2=0x7f606406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 -- 172.21.15.32:0/908965101 shutdown_connections 2024-03-06T07:35:31.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.151+0000 7f606ae00640 1 -- 172.21.15.32:0/908965101 wait complete. 2024-03-06T07:35:31.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.152+0000 7f606ae00640 1 Processor -- start 2024-03-06T07:35:31.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 -- start start 2024-03-06T07:35:31.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f60640cce90 0x7f6064157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 0x7f6064160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6064162000 0x7f6064160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064162490 con 0x7f606415efc0 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606406ce20 con 0x7f6064162000 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.153+0000 7f606ae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606406d620 con 0x7f60640cce90 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 0x7f6064160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 0x7f6064160570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50434/0 (socket says 172.21.15.32:50434) 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 -- 172.21.15.32:0/1012568588 learned_addr learned my addr 172.21.15.32:0/1012568588 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069a00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f60640cce90 0x7f6064157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 -- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f60640cce90 msgr2=0x7f6064157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f60640cce90 0x7f6064157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 -- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6064162000 msgr2=0x7f6064160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f606a400640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6064162000 0x7f6064160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6064162000 0x7f6064160ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069000640 1 -- 172.21.15.32:0/1012568588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064146420 con 0x7f606415efc0 2024-03-06T07:35:31.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.154+0000 7f6069a00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f60640cce90 0x7f6064157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.155+0000 7f6069000640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 0x7f6064160570 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f605002b950 tx=0x7f60500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.155+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050040070 con 0x7f606415efc0 2024-03-06T07:35:31.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.155+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6050030d00 con 0x7f606415efc0 2024-03-06T07:35:31.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.155+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6064146700 con 0x7f606415efc0 2024-03-06T07:35:31.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.155+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050035040 con 0x7f606415efc0 2024-03-06T07:35:31.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.155+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6064146cc0 con 0x7f606415efc0 2024-03-06T07:35:31.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.156+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606404f330 con 0x7f606415efc0 2024-03-06T07:35:31.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.157+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f605004b060 con 0x7f606415efc0 2024-03-06T07:35:31.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.158+0000 7f6061600640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f604406b4d0 0x7f604406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.158+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f60500ac1a0 con 0x7f606415efc0 2024-03-06T07:35:31.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.158+0000 7f6069a00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f604406b4d0 0x7f604406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.159+0000 7f6069a00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f604406b4d0 0x7f604406d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6058006fd0 tx=0x7f6058008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.162+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f605007ae90 con 0x7f606415efc0 2024-03-06T07:35:31.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.390+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60640ae3c0 con 0x7f606415efc0 2024-03-06T07:35:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.391+0000 7f6061600640 1 -- 172.21.15.32:0/1012568588 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+1802 (secure 0 0 0) 0x7f605007a790 con 0x7f606415efc0 2024-03-06T07:35:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:31.393 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:30.884919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:35:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 77 2024-03-06T07:35:31.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f604406b4d0 msgr2=0x7f604406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f604406b4d0 0x7f604406d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6058006fd0 tx=0x7f6058008040 comp rx=0 tx=0).stop 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 msgr2=0x7f6064160570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 0x7f6064160570 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f605002b950 tx=0x7f60500046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 shutdown_connections 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f604406b4d0 0x7f604406d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6064162000 0x7f6064160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606415efc0 0x7f6064160570 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 --2- 172.21.15.32:0/1012568588 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f60640cce90 0x7f6064157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 >> 172.21.15.32:0/1012568588 conn(0x7f606406b3d0 msgr2=0x7f60640ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 shutdown_connections 2024-03-06T07:35:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:31.394+0000 7f606ae00640 1 -- 172.21.15.32:0/1012568588 wait complete. 2024-03-06T07:35:31.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:14.858000+0000', 'modified': '2024-03-06T07:35:30.884919+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:35:31.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:35:32.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:33.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.033+0000 7fd1b4a00640 1 Processor -- start 2024-03-06T07:35:33.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.033+0000 7fd1b4a00640 1 -- start start 2024-03-06T07:35:33.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1b4a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1b0070f10 0x7fd1b00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1b4a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1b00cce90 0x7fd1b015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1b4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00714d0 con 0x7fd1b0145300 2024-03-06T07:35:33.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1b4a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b00cd300 con 0x7fd1b00cce90 2024-03-06T07:35:33.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1b4a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b00717b0 con 0x7fd1b0070f10 2024-03-06T07:35:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1af200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1b00cce90 0x7fd1b015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1af200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1b00cce90 0x7fd1b015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50440/0 (socket says 172.21.15.32:50440) 2024-03-06T07:35:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1af200640 1 -- 172.21.15.32:0/1090696351 learned_addr learned my addr 172.21.15.32:0/1090696351 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.035+0000 7fd1ade00640 1 --2- 172.21.15.32:0/1090696351 >> v2:172.21.15.39:3300/0 conn(0x7fd1b0070f10 0x7fd1b00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.036+0000 7fd1af200640 1 -- 172.21.15.32:0/1090696351 >> v2:172.21.15.39:3300/0 conn(0x7fd1b0070f10 msgr2=0x7fd1b00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.036+0000 7fd1af200640 1 --2- 172.21.15.32:0/1090696351 >> v2:172.21.15.39:3300/0 conn(0x7fd1b0070f10 0x7fd1b00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.036+0000 7fd1af200640 1 -- 172.21.15.32:0/1090696351 >> v1:172.21.15.39:6789/0 conn(0x7fd1b0145300 legacy=0x7fd1b00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.036+0000 7fd1af200640 1 -- 172.21.15.32:0/1090696351 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0071a90 con 0x7fd1b00cce90 2024-03-06T07:35:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.037+0000 7fd1af200640 1 --2- 172.21.15.32:0/1090696351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b00cce90 0x7fd1b015c570 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00b5150 tx=0x7fd1a802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b46019bf54e1a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.037+0000 7fd1ad400640 1 -- 172.21.15.32:0/1090696351 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a802c9f0 con 0x7fd1b00cce90 2024-03-06T07:35:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.037+0000 7fd1ad400640 1 -- 172.21.15.32:0/1090696351 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a802cb90 con 0x7fd1b00cce90 2024-03-06T07:35:33.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.037+0000 7fd1ad400640 1 -- 172.21.15.32:0/1090696351 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80343f0 con 0x7fd1b00cce90 2024-03-06T07:35:33.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 -- 172.21.15.32:0/1090696351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b00cce90 msgr2=0x7fd1b015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/1090696351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b00cce90 0x7fd1b015c570 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00b5150 tx=0x7fd1a802be50 comp rx=0 tx=0).stop 2024-03-06T07:35:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 -- 172.21.15.32:0/1090696351 shutdown_connections 2024-03-06T07:35:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/1090696351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b00cce90 0x7fd1b015c570 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/1090696351 >> v2:172.21.15.39:3300/0 conn(0x7fd1b0070f10 0x7fd1b00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 -- 172.21.15.32:0/1090696351 >> 172.21.15.32:0/1090696351 conn(0x7fd1b006b3d0 msgr2=0x7fd1b006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 -- 172.21.15.32:0/1090696351 shutdown_connections 2024-03-06T07:35:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.038+0000 7fd1b4a00640 1 -- 172.21.15.32:0/1090696351 wait complete. 2024-03-06T07:35:33.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.040+0000 7fd1b4a00640 1 Processor -- start 2024-03-06T07:35:33.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.040+0000 7fd1b4a00640 1 -- start start 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1b4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 0x7fd1b00abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1b4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1b00cce90 0x7fd1b015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1b4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1b0145300 0x7fd1b015aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1b4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b015cab0 con 0x7fd1b0070f10 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1b4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01456e0 con 0x7fd1b0145300 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1b4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b015cc30 con 0x7fd1b00cce90 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1ae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 0x7fd1b00abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1ae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 0x7fd1b00abda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50448/0 (socket says 172.21.15.32:50448) 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.041+0000 7fd1ae800640 1 -- 172.21.15.32:0/2832664790 learned_addr learned my addr 172.21.15.32:0/2832664790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ade00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1b00cce90 0x7fd1b015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1af200640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1b0145300 0x7fd1b015aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ae800640 1 -- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1b00cce90 msgr2=0x7fd1b015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ae800640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1b00cce90 0x7fd1b015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ae800640 1 -- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1b0145300 msgr2=0x7fd1b015aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ae800640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1b0145300 0x7fd1b015aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ae800640 1 -- 172.21.15.32:0/2832664790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b01e6da0 con 0x7fd1b0070f10 2024-03-06T07:35:33.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ade00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1b00cce90 0x7fd1b015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1ae800640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 0x7fd1b00abda0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000b3f0 tx=0x7fd1a000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0015020 con 0x7fd1b0070f10 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a0011830 con 0x7fd1b0070f10 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0011c20 con 0x7fd1b0070f10 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b01e7080 con 0x7fd1b0070f10 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.042+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b00a97f0 con 0x7fd1b0070f10 2024-03-06T07:35:33.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.044+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b0067830 con 0x7fd1b0070f10 2024-03-06T07:35:33.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.045+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1a0018020 con 0x7fd1b0070f10 2024-03-06T07:35:33.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.045+0000 7fd1a6a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.046+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd1a0089fe0 con 0x7fd1b0070f10 2024-03-06T07:35:33.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.046+0000 7fd1ade00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.046+0000 7fd1ade00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19006b4d0 0x7fd19006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd1b015bf00 tx=0x7fd19c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.050+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1b0067830 con 0x7fd1b0070f10 2024-03-06T07:35:33.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.288+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1b004f330 con 0x7fd1b0070f10 2024-03-06T07:35:33.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.289+0000 7fd1a6a00640 1 -- 172.21.15.32:0/2832664790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd1a001f030 con 0x7fd1b0070f10 2024-03-06T07:35:33.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:33.292 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:31.917887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:35:33.292 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 80 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19006b4d0 msgr2=0x7fd19006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19006b4d0 0x7fd19006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd1b015bf00 tx=0x7fd19c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 msgr2=0x7fd1b00abda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 0x7fd1b00abda0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000b3f0 tx=0x7fd1a000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 shutdown_connections 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1b0145300 0x7fd1b015aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1b00cce90 0x7fd1b015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 --2- 172.21.15.32:0/2832664790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1b0070f10 0x7fd1b00abda0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.292+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 >> 172.21.15.32:0/2832664790 conn(0x7fd1b006b3d0 msgr2=0x7fd1b015d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.293+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 shutdown_connections 2024-03-06T07:35:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:33.293+0000 7fd1b4a00640 1 -- 172.21.15.32:0/2832664790 wait complete. 2024-03-06T07:35:33.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:14.858000+0000', 'modified': '2024-03-06T07:35:31.917887+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7896}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7896': {'gid': 7896, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.32:6837/3465731454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3465731454}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3465731454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:35:33.716 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7896:up:reconnect 2024-03-06T07:35:34.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.946+0000 7f7e2ea00640 1 Processor -- start 2024-03-06T07:35:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.946+0000 7f7e2ea00640 1 -- start start 2024-03-06T07:35:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.948+0000 7f7e2ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e280cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.948+0000 7f7e2ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e280d4a10 0x7f7e280d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.948+0000 7f7e2ea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e280b3710 con 0x7f7e280d0c80 2024-03-06T07:35:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.948+0000 7f7e2ea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e280b09e0 con 0x7f7e280d4a10 2024-03-06T07:35:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.948+0000 7f7e2ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280b39f0 con 0x7f7e2815e0b0 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e280d4a10 0x7f7e280d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e280d4a10 0x7f7e280d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50454/0 (socket says 172.21.15.32:50454) 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 -- 172.21.15.32:0/2560569592 learned_addr learned my addr 172.21.15.32:0/2560569592 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2d600640 1 --2- 172.21.15.32:0/2560569592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e280cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 -- 172.21.15.32:0/2560569592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 msgr2=0x7f7e280cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 --2- 172.21.15.32:0/2560569592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e280cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 -- 172.21.15.32:0/2560569592 >> v1:172.21.15.39:6789/0 conn(0x7f7e280d0c80 legacy=0x7f7e280d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.949+0000 7f7e2cc00640 1 -- 172.21.15.32:0/2560569592 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280b3cd0 con 0x7f7e280d4a10 2024-03-06T07:35:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.950+0000 7f7e2cc00640 1 --2- 172.21.15.32:0/2560569592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e280d0740 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009870 tx=0x7f7e1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3127996f78f71e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.951+0000 7f7e27e00640 1 -- 172.21.15.32:0/2560569592 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035020 con 0x7f7e280d4a10 2024-03-06T07:35:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.951+0000 7f7e27e00640 1 -- 172.21.15.32:0/2560569592 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14031640 con 0x7f7e280d4a10 2024-03-06T07:35:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.951+0000 7f7e27e00640 1 -- 172.21.15.32:0/2560569592 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1402d040 con 0x7f7e280d4a10 2024-03-06T07:35:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.951+0000 7f7e2ea00640 1 -- 172.21.15.32:0/2560569592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 msgr2=0x7f7e280d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.951+0000 7f7e2ea00640 1 --2- 172.21.15.32:0/2560569592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e280d0740 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009870 tx=0x7f7e1402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.952+0000 7f7e2ea00640 1 -- 172.21.15.32:0/2560569592 shutdown_connections 2024-03-06T07:35:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.952+0000 7f7e2ea00640 1 --2- 172.21.15.32:0/2560569592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e280d0740 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.952+0000 7f7e2ea00640 1 --2- 172.21.15.32:0/2560569592 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e280cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.952+0000 7f7e2ea00640 1 -- 172.21.15.32:0/2560569592 >> 172.21.15.32:0/2560569592 conn(0x7f7e280d3d30 msgr2=0x7f7e280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.952+0000 7f7e2ea00640 1 -- 172.21.15.32:0/2560569592 shutdown_connections 2024-03-06T07:35:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.952+0000 7f7e2ea00640 1 -- 172.21.15.32:0/2560569592 wait complete. 2024-03-06T07:35:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.953+0000 7f7e2ea00640 1 Processor -- start 2024-03-06T07:35:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.954+0000 7f7e2ea00640 1 -- start start 2024-03-06T07:35:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e280d0c80 0x7f7e280cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e28160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e281605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280cfc60 con 0x7f7e280d4a10 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2815e490 con 0x7f7e280d0c80 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280b54e0 con 0x7f7e2815e0b0 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.955+0000 7f7e2cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e28160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e28160070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50458/0 (socket says 172.21.15.32:50458) 2024-03-06T07:35:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 -- 172.21.15.32:0/3446688664 learned_addr learned my addr 172.21.15.32:0/3446688664 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2d600640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e280d0c80 0x7f7e280cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 -- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 msgr2=0x7f7e281605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2e000640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e281605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e281605b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 -- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e280d0c80 msgr2=0x7f7e280cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e280d0c80 0x7f7e280cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 -- 172.21.15.32:0/3446688664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e281f3cc0 con 0x7f7e280d4a10 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2d600640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e280d0c80 0x7f7e280cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.956+0000 7f7e2cc00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e28160070 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7e14000c00 tx=0x7f7e140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.957+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14040070 con 0x7f7e280d4a10 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.957+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14039bf0 con 0x7f7e280d4a10 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.957+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035040 con 0x7f7e280d4a10 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.957+0000 7f7e2ea00640 1 -- 172.21.15.32:0/3446688664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e28160be0 con 0x7f7e280d4a10 2024-03-06T07:35:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.957+0000 7f7e2ea00640 1 -- 172.21.15.32:0/3446688664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e280b7750 con 0x7f7e280d4a10 2024-03-06T07:35:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.958+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2804f330 con 0x7f7e280d4a10 2024-03-06T07:35:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.959+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e1404b060 con 0x7f7e280d4a10 2024-03-06T07:35:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.960+0000 7f7e25600640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.960+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7e140abd60 con 0x7f7e280d4a10 2024-03-06T07:35:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.960+0000 7f7e2d600640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.961+0000 7f7e2d600640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e0806b4d0 0x7f7e0806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00b820 tx=0x7f7e1c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:34.964+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e14031610 con 0x7f7e280d4a10 2024-03-06T07:35:35.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.200+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e280d3300 con 0x7f7e280d4a10 2024-03-06T07:35:35.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.201+0000 7f7e25600640 1 -- 172.21.15.32:0/3446688664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f7e1407ab70 con 0x7f7e280d4a10 2024-03-06T07:35:35.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:35.204 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:35:35.204 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e0806b4d0 msgr2=0x7f7e0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e0806b4d0 0x7f7e0806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00b820 tx=0x7f7e1c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 msgr2=0x7f7e28160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e28160070 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7e14000c00 tx=0x7f7e140046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 shutdown_connections 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e2815e0b0 0x7f7e281605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e280d4a10 0x7f7e28160070 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 --2- 172.21.15.32:0/3446688664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e280d0c80 0x7f7e280cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 >> 172.21.15.32:0/3446688664 conn(0x7f7e280d3d30 msgr2=0x7f7e280d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 shutdown_connections 2024-03-06T07:35:35.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.204+0000 7f7e1aa00640 1 -- 172.21.15.32:0/3446688664 wait complete. 2024-03-06T07:35:35.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:14.858000+0000', 'modified': '2024-03-06T07:35:34.917265+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7896}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7896': {'gid': 7896, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6837/3465731454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 3465731454}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 3465731454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7896, 'qdb_cluster': [7896]} 2024-03-06T07:35:35.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:35:35.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:35:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.835+0000 7f129b000640 1 Processor -- start 2024-03-06T07:35:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.836+0000 7f129b000640 1 -- start start 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.837+0000 7f129b000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f12940b3c90 0x7f12940cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.837+0000 7f129b000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294071050 0x7f1294071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.837+0000 7f129b000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940b4860 con 0x7f12940cf5c0 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.837+0000 7f129b000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12940ce160 con 0x7f12940b3c90 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.837+0000 7f129b000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12940b4b40 con 0x7f1294071050 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.838+0000 7f128b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12940cf5c0 0x7f12940b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48478/0 (socket says 172.21.15.32:48478) 2024-03-06T07:35:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.838+0000 7f128b400640 1 -- 172.21.15.32:0/726222942 learned_addr learned my addr 172.21.15.32:0/726222942 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:35.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.838+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293603152 0 0) 0x7f12940b4860 con 0x7f12940cf5c0 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.838+0000 7f1293e00640 1 --2- 172.21.15.32:0/726222942 >> v2:172.21.15.39:3300/0 conn(0x7f12940b3c90 0x7f12940cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.838+0000 7f1299000640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294071050 0x7f1294071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.838+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003520 con 0x7f12940cf5c0 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2684927684 0 0) 0x7f1278003520 con 0x7f12940cf5c0 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294071050 msgr2=0x7f1294071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294071050 0x7f1294071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 >> v2:172.21.15.39:3300/0 conn(0x7f12940b3c90 msgr2=0x7f12940cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 --2- 172.21.15.32:0/726222942 >> v2:172.21.15.39:3300/0 conn(0x7f12940b3c90 0x7f12940cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940b4e20 con 0x7f12940cf5c0 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f127c001670 con 0x7f12940cf5c0 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 >> v1:172.21.15.32:6789/0 conn(0x7f12940cf5c0 legacy=0x7f12940b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1278003a20 0x7f1278003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1278004800 0x7f1278004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12780051c0 0x7f127800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12940ce160 con 0x7f1278003a20 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12940b4b40 con 0x7f12780051c0 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127800c010 con 0x7f1278004800 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1299000640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294071050 0x7f1294071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293e00640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1278003a20 0x7f1278003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f128b400640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1278004800 0x7f1278004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.839+0000 7f1293e00640 1 -- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1278004800 msgr2=0x7f1278004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.840+0000 7f1293e00640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1278004800 0x7f1278004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.840+0000 7f1293e00640 1 -- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12780051c0 msgr2=0x7f127800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.840+0000 7f1293e00640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12780051c0 0x7f127800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.840+0000 7f1293e00640 1 -- 172.21.15.32:0/726222942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278003520 con 0x7f1278003a20 2024-03-06T07:35:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.840+0000 7f128b400640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1278004800 0x7f1278004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f1293e00640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1278003a20 0x7f1278003e40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f1284009fc0 tx=0x7f1278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ac4f3b3c5c6aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284039070 con 0x7f1278003a20 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128402cc60 con 0x7f1278003a20 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f1293400640 1 -- 172.21.15.32:0/726222942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284009c10 con 0x7f1278003a20 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f129b000640 1 -- 172.21.15.32:0/726222942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1278003a20 msgr2=0x7f1278003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f129b000640 1 --2- 172.21.15.32:0/726222942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1278003a20 0x7f1278003e40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f1284009fc0 tx=0x7f1278005830 comp rx=0 tx=0).stop 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.841+0000 7f1299000640 1 -- 172.21.15.32:0/726222942 reap_dead start 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.842+0000 7f129b000640 1 -- 172.21.15.32:0/726222942 shutdown_connections 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.842+0000 7f129b000640 1 -- 172.21.15.32:0/726222942 >> 172.21.15.32:0/726222942 conn(0x7f129406afb0 msgr2=0x7f129406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.842+0000 7f129b000640 1 -- 172.21.15.32:0/726222942 shutdown_connections 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.842+0000 7f129b000640 1 -- 172.21.15.32:0/726222942 wait complete. 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.844+0000 7f129b000640 1 Processor -- start 2024-03-06T07:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.845+0000 7f129b000640 1 -- start start 2024-03-06T07:35:35.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f129b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1278003a20 0x7f12941419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f129b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 0x7f1294159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f129b000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294159d80 0x7f129413f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f129b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294071490 con 0x7f129414e150 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f129b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129415b340 con 0x7f1278003a20 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f129b000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1294071610 con 0x7f1294159d80 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f128b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 0x7f1294159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f128b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 0x7f1294159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50470/0 (socket says 172.21.15.32:50470) 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.846+0000 7f128b400640 1 -- 172.21.15.32:0/686614109 learned_addr learned my addr 172.21.15.32:0/686614109 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f1293e00640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1278003a20 0x7f12941419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f1299000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294159d80 0x7f129413f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f128b400640 1 -- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294159d80 msgr2=0x7f129413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f128b400640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294159d80 0x7f129413f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f128b400640 1 -- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1278003a20 msgr2=0x7f12941419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f128b400640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1278003a20 0x7f12941419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f128b400640 1 -- 172.21.15.32:0/686614109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1294158ac0 con 0x7f129414e150 2024-03-06T07:35:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.847+0000 7f128b400640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 0x7f1294159840 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f127c00d770 tx=0x7f127c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.848+0000 7f1290c00640 1 -- 172.21.15.32:0/686614109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c019070 con 0x7f129414e150 2024-03-06T07:35:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.848+0000 7f1290c00640 1 -- 172.21.15.32:0/686614109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127c0074e0 con 0x7f129414e150 2024-03-06T07:35:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.848+0000 7f1290c00640 1 -- 172.21.15.32:0/686614109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c0049e0 con 0x7f129414e150 2024-03-06T07:35:35.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.848+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1294158d60 con 0x7f129414e150 2024-03-06T07:35:35.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.848+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1294159320 con 0x7f129414e150 2024-03-06T07:35:35.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.850+0000 7f1290c00640 1 -- 172.21.15.32:0/686614109 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f127c01c070 con 0x7f129414e150 2024-03-06T07:35:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.851+0000 7f1290c00640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f127006b500 0x7f127006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.851+0000 7f1290c00640 1 -- 172.21.15.32:0/686614109 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f127c089c50 con 0x7f129414e150 2024-03-06T07:35:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.851+0000 7f1293e00640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f127006b500 0x7f127006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.852+0000 7f129de25740 1 Processor -- start 2024-03-06T07:35:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.852+0000 7f129de25740 1 -- start start 2024-03-06T07:35:35.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.852+0000 7f1293e00640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f127006b500 0x7f127006d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1284036000 tx=0x7f128402e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44e9ea750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea1a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f129de25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 0x55f44ea1aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f129de25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f44e954600 con 0x55f44e9eac90 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f44e8fd930 con 0x55f44ea17b90 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f44e9506a0 con 0x55f44ea177b0 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f1299000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 0x55f44ea1aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f1299000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 0x55f44ea1aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50484/0 (socket says 172.21.15.32:50484) 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f1299000640 1 -- 172.21.15.32:0/3252764899 learned_addr learned my addr 172.21.15.32:0/3252764899 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f1293e00640 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44e9ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.853+0000 7f128b400640 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea1a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1299000640 1 -- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 msgr2=0x55f44e9ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1299000640 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44e9ea750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1299000640 1 -- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 msgr2=0x55f44ea1a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1299000640 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea1a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1299000640 1 -- 172.21.15.32:0/3252764899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f44e6692d0 con 0x55f44e9eac90 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1299000640 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 0x55f44ea1aa80 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f12940cf9a0 tx=0x7f128c009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb024f6f1d548648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1288c00640 1 -- 172.21.15.32:0/3252764899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c013400 con 0x55f44e9eac90 2024-03-06T07:35:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1288c00640 1 -- 172.21.15.32:0/3252764899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c019070 con 0x55f44e9eac90 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.854+0000 7f1288c00640 1 -- 172.21.15.32:0/3252764899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0137e0 con 0x55f44e9eac90 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 -- 172.21.15.32:0/3252764899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 msgr2=0x55f44ea1aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 0x55f44ea1aa80 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f12940cf9a0 tx=0x7f128c009930 comp rx=0 tx=0).stop 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 -- 172.21.15.32:0/3252764899 shutdown_connections 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44e9eac90 0x55f44ea1aa80 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea1a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 --2- 172.21.15.32:0/3252764899 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44e9ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 -- 172.21.15.32:0/3252764899 >> 172.21.15.32:0/3252764899 conn(0x55f44e9ec9c0 msgr2=0x55f44ea35540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 -- 172.21.15.32:0/3252764899 shutdown_connections 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.855+0000 7f129de25740 1 -- 172.21.15.32:0/3252764899 wait complete. 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.856+0000 7f129de25740 1 -- start start 2024-03-06T07:35:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.856+0000 7f129de25740 1 Processor -- start 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea09550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea2fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f44e9eef60 con 0x55f44ea17b90 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f44e910d90 con 0x55f44ea177b0 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f44e9ee350 con 0x55f44ea08d00 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f129de25740 10 client.0 _ll_register_callbacks cb 0x55f44e94c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f128b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea2fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f128b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea2fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50500/0 (socket says 172.21.15.32:50500) 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f128b400640 1 -- 172.21.15.32:0/572892265 learned_addr learned my addr 172.21.15.32:0/572892265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.857+0000 7f1293e00640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea09550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f1299000640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f128b400640 1 -- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 msgr2=0x55f44ea1dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f128b400640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f128b400640 1 -- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 msgr2=0x55f44ea09550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f128b400640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea09550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f128b400640 1 -- 172.21.15.32:0/572892265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f44e7336b0 con 0x55f44ea17b90 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f1293e00640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea09550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f128b400640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea2fb70 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f127c088110 tx=0x7f127c058b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f397e3c1238ebad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f1281600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f1281600640 1 -- 172.21.15.32:0/572892265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c05c2b0 con 0x55f44ea17b90 2024-03-06T07:35:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f1281600640 1 -- 172.21.15.32:0/572892265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f127c004b80 con 0x55f44ea17b90 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f128c019070 con 0x55f44ea17b90 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.858+0000 7f1281600640 1 -- 172.21.15.32:0/572892265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f127c060a70 con 0x55f44ea17b90 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 10 client.7926 fetch_fsmap learned FSMap version 82 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55f44ea1d7f0 con 0x55f44ea17b90 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f1281600640 1 -- 172.21.15.32:0/572892265 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c019070 con 0x55f44ea17b90 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f1281600640 1 -- 172.21.15.32:0/572892265 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f127c0577b0 con 0x55f44ea17b90 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 10 client.7926 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 10 client.7926 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 20 client.7926 populate_metadata read hostname 'smithi032' 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea518c0 0x55f44ea53cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 4 client.7926 mds_command: new command op to 7896 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 --> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f44e961b40 con 0x55f44ea518c0 2024-03-06T07:35:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.859+0000 7f1293e00640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea518c0 0x55f44ea53cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.860+0000 7f1293e00640 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea518c0 0x55f44ea53cb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95431a0e7bbd5857 server_cookie=ecd60c927e525eeb in_seq=0 out_seq=0 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.860+0000 7f1281600640 10 client.7926 ms_handle_connect on v2:172.21.15.32:6836/3465731454 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f1281600640 1 -- 172.21.15.32:0/572892265 <== mds.0 v2:172.21.15.32:6836/3465731454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f44e961b40 con 0x55f44ea518c0 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f1281600640 10 client.7926 handle_command_reply: tid=0 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 client.7926 shutdown 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 msgr2=0x55f44ea2fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea2fb70 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f127c088110 tx=0x7f127c058b20 comp rx=0 tx=0).stop 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 shutdown_connections 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea518c0 0x55f44ea53cb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea2fb70 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 --2- 172.21.15.32:0/572892265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea09550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 >> 172.21.15.32:0/572892265 conn(0x55f44e9eac90 msgr2=0x55f44ea2f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f1281600640 0 client.7926 ms_handle_reset on v2:172.21.15.32:6836/3465731454 2024-03-06T07:35:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 shutdown_connections 2024-03-06T07:35:35.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 1 -- 172.21.15.32:0/572892265 wait complete. 2024-03-06T07:35:35.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.861+0000 7f129de25740 20 client.7926 trim_cache size 0 max 16384 2024-03-06T07:35:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.879+0000 7f129de25740 1 Processor -- start 2024-03-06T07:35:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.879+0000 7f129de25740 1 -- start start 2024-03-06T07:35:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea1a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f129de25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea36f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f129de25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941ffce0 con 0x55f44ea17b90 2024-03-06T07:35:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12940b02a0 con 0x55f44ea177b0 2024-03-06T07:35:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12940b08a0 con 0x55f44ea08d00 2024-03-06T07:35:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f1299000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea36f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f1299000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea36f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50502/0 (socket says 172.21.15.32:50502) 2024-03-06T07:35:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.880+0000 7f1299000640 1 -- 172.21.15.32:0/2028803877 learned_addr learned my addr 172.21.15.32:0/2028803877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f128b400640 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea1a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1293e00640 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1299000640 1 -- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 msgr2=0x55f44ea1dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1299000640 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1299000640 1 -- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 msgr2=0x55f44ea1a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1299000640 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea1a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1299000640 1 -- 172.21.15.32:0/2028803877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f44e9ec720 con 0x55f44ea17b90 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f128b400640 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea1a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1299000640 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea36f80 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f128c008010 tx=0x7f128c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa75f2bcc177607 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1283e00640 1 -- 172.21.15.32:0/2028803877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c023070 con 0x55f44ea17b90 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1283e00640 1 -- 172.21.15.32:0/2028803877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c019040 con 0x55f44ea17b90 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.881+0000 7f1283e00640 1 -- 172.21.15.32:0/2028803877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c003610 con 0x55f44ea17b90 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 -- 172.21.15.32:0/2028803877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 msgr2=0x55f44ea36f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea36f80 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f128c008010 tx=0x7f128c009f90 comp rx=0 tx=0).stop 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 -- 172.21.15.32:0/2028803877 shutdown_connections 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea17b90 0x55f44ea36f80 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea177b0 0x55f44ea1a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 --2- 172.21.15.32:0/2028803877 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea08d00 0x55f44ea1dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 -- 172.21.15.32:0/2028803877 >> 172.21.15.32:0/2028803877 conn(0x55f44e9eac90 msgr2=0x55f44ea4f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 -- 172.21.15.32:0/2028803877 shutdown_connections 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.882+0000 7f129de25740 1 -- 172.21.15.32:0/2028803877 wait complete. 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.883+0000 7f129de25740 1 -- start start 2024-03-06T07:35:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.883+0000 7f129de25740 1 Processor -- start 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 0x55f44ea9eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44ea377f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea44500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f44ea0c320 con 0x55f44ea08d00 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f44ea300b0 con 0x55f44ea17b90 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f44ea2e7f0 con 0x55f44ea177b0 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f129de25740 10 client.0 _ll_register_callbacks cb 0x7f127c054600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 0x55f44ea9eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 0x55f44ea9eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50518/0 (socket says 172.21.15.32:50518) 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 -- 172.21.15.32:0/3493510656 learned_addr learned my addr 172.21.15.32:0/3493510656 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 -- 172.21.15.32:0/3493510656 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 msgr2=0x55f44ea377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44ea377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 -- 172.21.15.32:0/3493510656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 msgr2=0x55f44ea44500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea44500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.884+0000 7f1293e00640 1 -- 172.21.15.32:0/3493510656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f44ea36f80 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f1293e00640 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 0x55f44ea9eff0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f128400e230 tx=0x7f1284004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ecab0ec5f4e930 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f1281600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f1281600640 1 -- 172.21.15.32:0/3493510656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284039040 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f1281600640 1 -- 172.21.15.32:0/3493510656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f128404f230 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f128c019040 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f1281600640 1 -- 172.21.15.32:0/3493510656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f128405a050 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f129de25740 10 client.7932 fetch_fsmap learned FSMap version 82 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55f44eaa5830 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.885+0000 7f1281600640 1 -- 172.21.15.32:0/3493510656 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c019040 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f1281600640 1 -- 172.21.15.32:0/3493510656 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f128405d020 con 0x55f44ea08d00 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f129de25740 10 client.7932 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f129de25740 10 client.7932 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f129de25740 20 client.7932 populate_metadata read hostname 'smithi032' 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f129de25740 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea16b50 0x55f44ea06030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f129de25740 4 client.7932 mds_command: new command op to 7896 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 --> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f44ea45730 con 0x55f44ea16b50 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f128b400640 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea16b50 0x55f44ea06030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f1281600640 10 client.7932 ms_handle_connect on v2:172.21.15.32:6836/3465731454 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.886+0000 7f128b400640 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea16b50 0x55f44ea06030 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53a907c3add6a56c server_cookie=8583301e398d972a in_seq=0 out_seq=0 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f1281600640 1 -- 172.21.15.32:0/3493510656 <== mds.0 v2:172.21.15.32:6836/3465731454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55f44ea45730 con 0x55f44ea16b50 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f1281600640 10 client.7932 handle_command_reply: tid=0 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 client.7932 shutdown 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 msgr2=0x55f44ea9eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 0x55f44ea9eff0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f128400e230 tx=0x7f1284004ab0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 shutdown_connections 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:6836/3465731454,v1:172.21.15.32:6837/3465731454] conn(0x55f44ea16b50 0x55f44ea06030 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f44ea17b90 0x55f44ea44500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55f44ea177b0 0x55f44ea377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 --2- 172.21.15.32:0/3493510656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f44ea08d00 0x55f44ea9eff0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 >> 172.21.15.32:0/3493510656 conn(0x55f44e9eac90 msgr2=0x55f44e9f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f1281600640 0 client.7932 ms_handle_reset on v2:172.21.15.32:6836/3465731454 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 shutdown_connections 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 1 -- 172.21.15.32:0/3493510656 wait complete. 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.887+0000 7f129de25740 20 client.7932 trim_cache size 0 max 16384 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7896, 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.9827576769999999, 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 82, 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 114, 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 114, 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 4.7679950809999996 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f127006b500 msgr2=0x7f127006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f127006b500 0x7f127006d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1284036000 tx=0x7f128402e000 comp rx=0 tx=0).stop 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 msgr2=0x7f1294159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 0x7f1294159840 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f127c00d770 tx=0x7f127c00dc30 comp rx=0 tx=0).stop 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 shutdown_connections 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f127006b500 0x7f127006d9c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1294159d80 0x7f129413f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f129414e150 0x7f1294159840 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 --2- 172.21.15.32:0/686614109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1278003a20 0x7f12941419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 >> 172.21.15.32:0/686614109 conn(0x7f129406afb0 msgr2=0x7f129406e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 shutdown_connections 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:35.888+0000 7f129b000640 1 -- 172.21.15.32:0/686614109 wait complete. 2024-03-06T07:35:35.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:35:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.098+0000 7f22afe00640 1 Processor -- start 2024-03-06T07:35:36.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.098+0000 7f22afe00640 1 -- start start 2024-03-06T07:35:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.100+0000 7f22afe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 0x7f22b0070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.100+0000 7f22afe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00cd0c0 con 0x7f22b015f450 2024-03-06T07:35:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.100+0000 7f22afe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00cd3a0 con 0x7f22b00700c0 2024-03-06T07:35:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.100+0000 7f22afe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00d29e0 con 0x7f22b00d04b0 2024-03-06T07:35:36.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.100+0000 7f22aea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f22b015f450 0x7f22b00cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48480/0 (socket says 172.21.15.32:48480) 2024-03-06T07:35:36.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.100+0000 7f22aea00640 1 -- 172.21.15.32:0/1356997183 learned_addr learned my addr 172.21.15.32:0/1356997183 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22a7e00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 0x7f22b0070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049084965 0 0) 0x7f22b00cd0c0 con 0x7f22b015f450 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2294003520 con 0x7f22b015f450 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892808103 0 0) 0x7f2294003520 con 0x7f22b015f450 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 msgr2=0x7f22b0070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 0x7f22b0070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 >> v1:172.21.15.39:6789/0 conn(0x7f22b00700c0 legacy=0x7f22b00cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b00cd680 con 0x7f22b015f450 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f22a0001670 con 0x7f22b015f450 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 >> v1:172.21.15.32:6789/0 conn(0x7f22b015f450 legacy=0x7f22b00cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22940039e0 0x7f2294003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2294004800 0x7f2294004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.101+0000 7f22ae000640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 0x7f22940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00d29e0 con 0x7f2294004800 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229400baa0 con 0x7f22940039e0 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229400bca0 con 0x7f22b00d04b0 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22a7e00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2294004800 0x7f2294004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22aea00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22940039e0 0x7f2294003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22a7e00640 1 -- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 msgr2=0x7f22940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22a7e00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b00d04b0 0x7f22940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22a7e00640 1 -- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22940039e0 msgr2=0x7f2294003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22a7e00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22940039e0 0x7f2294003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.102+0000 7f22a7e00640 1 -- 172.21.15.32:0/1356997183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2294003520 con 0x7f2294004800 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.103+0000 7f22a7e00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2294004800 0x7f2294004c80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f2298009ac0 tx=0x7f2294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13230f0578d27d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298039070 con 0x7f2294004800 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229802c710 con 0x7f2294004800 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22ae000640 1 -- 172.21.15.32:0/1356997183 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298009690 con 0x7f2294004800 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22afe00640 1 -- 172.21.15.32:0/1356997183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2294004800 msgr2=0x7f2294004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22afe00640 1 --2- 172.21.15.32:0/1356997183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2294004800 0x7f2294004c80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f2298009ac0 tx=0x7f2294005830 comp rx=0 tx=0).stop 2024-03-06T07:35:36.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22af400640 1 -- 172.21.15.32:0/1356997183 reap_dead start 2024-03-06T07:35:36.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22afe00640 1 -- 172.21.15.32:0/1356997183 shutdown_connections 2024-03-06T07:35:36.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.104+0000 7f22afe00640 1 -- 172.21.15.32:0/1356997183 >> 172.21.15.32:0/1356997183 conn(0x7f22b00d1ae0 msgr2=0x7f22b00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.105+0000 7f22afe00640 1 -- 172.21.15.32:0/1356997183 shutdown_connections 2024-03-06T07:35:36.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.105+0000 7f22afe00640 1 -- 172.21.15.32:0/1356997183 wait complete. 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.106+0000 7f22afe00640 1 Processor -- start 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.106+0000 7f22afe00640 1 -- start start 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.107+0000 7f22afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2294004800 0x7f22b015ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.107+0000 7f22afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 0x7f22b0145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.107+0000 7f22afe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b0158ba0 0x7f22b0159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.107+0000 7f22afe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b015f830 con 0x7f22b0145210 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.107+0000 7f22afe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00ae8b0 con 0x7f2294004800 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.107+0000 7f22afe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b015f9b0 con 0x7f22b0158ba0 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 0x7f22b0145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 0x7f22b0145690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50536/0 (socket says 172.21.15.32:50536) 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 -- 172.21.15.32:0/262882600 learned_addr learned my addr 172.21.15.32:0/262882600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22af400640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b0158ba0 0x7f22b0159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22aea00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2294004800 0x7f22b015ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 -- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b0158ba0 msgr2=0x7f22b0159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b0158ba0 0x7f22b0159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 -- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2294004800 msgr2=0x7f22b015ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2294004800 0x7f22b015ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22a7e00640 1 -- 172.21.15.32:0/262882600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b0145df0 con 0x7f22b0145210 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.108+0000 7f22aea00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2294004800 0x7f22b015ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.109+0000 7f22a7e00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 0x7f22b0145690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2298009be0 tx=0x7f2298004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.109+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298042020 con 0x7f22b0145210 2024-03-06T07:35:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.109+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229803a040 con 0x7f22b0145210 2024-03-06T07:35:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.109+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298039070 con 0x7f22b0145210 2024-03-06T07:35:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.109+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b01460d0 con 0x7f22b0145210 2024-03-06T07:35:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.109+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b0146690 con 0x7f22b0145210 2024-03-06T07:35:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.110+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b004f330 con 0x7f22b0145210 2024-03-06T07:35:36.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.111+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2298045020 con 0x7f22b0145210 2024-03-06T07:35:36.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.112+0000 7f22a6a00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228c06b4d0 0x7f228c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.112+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f22980ab820 con 0x7f22b0145210 2024-03-06T07:35:36.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.113+0000 7f22aea00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228c06b4d0 0x7f228c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.113+0000 7f22aea00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228c06b4d0 0x7f228c06d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f22a0001eb0 tx=0x7f22a0001dc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.117+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f229804c070 con 0x7f22b0145210 2024-03-06T07:35:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.359+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f22b00afa70 con 0x7f22b0145210 2024-03-06T07:35:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.360+0000 7f22a6a00640 1 -- 172.21.15.32:0/262882600 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) v1 ==== 53+0+73 (secure 0 0 0) 0x7f229807a640 con 0x7f22b0145210 2024-03-06T07:35:36.363 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:35:36.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228c06b4d0 msgr2=0x7f228c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228c06b4d0 0x7f228c06d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f22a0001eb0 tx=0x7f22a0001dc0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 msgr2=0x7f22b0145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 0x7f22b0145690 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2298009be0 tx=0x7f2298004730 comp rx=0 tx=0).stop 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 shutdown_connections 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f228c06b4d0 0x7f228c06d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22b0158ba0 0x7f22b0159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22b0145210 0x7f22b0145690 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 --2- 172.21.15.32:0/262882600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2294004800 0x7f22b015ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 >> 172.21.15.32:0/262882600 conn(0x7f22b00d1ae0 msgr2=0x7f22b015ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 shutdown_connections 2024-03-06T07:35:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:36.363+0000 7f22afe00640 1 -- 172.21.15.32:0/262882600 wait complete. 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:35:36.385 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:36.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:36.385 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:35:36.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42001sec preferred_lft 42001sec 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:35:36.434 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:36.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:36.436 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:35:36.540 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:35:36.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:36.546 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:35:36.600 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:35:36.600 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:35:36.601 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:35:36.601 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:35:36.601 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:35:36.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:36.601 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:35:36.601 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:35:36.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:35:36.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:35:36.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:35:36.602 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:35:36.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:36.782 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:35:36.834 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:35:36.834 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:35:36.835 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:36.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:36.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:35:36.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:36.931 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:36.980 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:36.980 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:36.981 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:37.028 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:35:37.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:37.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:37.164 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:37.164 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:35:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:35:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:35:37.198 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:35:37.198 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:35:37.198 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:35:37.198 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:35:37.198 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:35:37.203 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:35:37.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:37.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:37.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:37.278 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:35:37.333 INFO:teuthology.orchestra.run.smithi032.stdout:client6367 2024-03-06T07:35:37.333 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:35:37.333 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:37.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:37.386 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:35:37.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.674+0000 7fbd9fe00640 1 Processor -- start 2024-03-06T07:35:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.674+0000 7fbd9fe00640 1 -- start start 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.676+0000 7fbd9fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbda0070f10 0x7fbda00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.676+0000 7fbd9fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda00714d0 con 0x7fbda0145300 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.676+0000 7fbd9fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda00717b0 con 0x7fbda00cce90 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.676+0000 7fbd9fe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda00cd300 con 0x7fbda0070f10 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd97e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbda0070f10 0x7fbda00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd97e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbda0070f10 0x7fbda00cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50544/0 (socket says 172.21.15.32:50544) 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd97e00640 1 -- 172.21.15.32:0/1565749266 learned_addr learned my addr 172.21.15.32:0/1565749266 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd9ea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbda0145300 0x7fbda00b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48488/0 (socket says 172.21.15.32:48488) 2024-03-06T07:35:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd97e00640 1 -- 172.21.15.32:0/1565749266 >> v1:172.21.15.39:6789/0 conn(0x7fbda00cce90 legacy=0x7fbda015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd97e00640 1 -- 172.21.15.32:0/1565749266 >> v1:172.21.15.32:6789/0 conn(0x7fbda0145300 legacy=0x7fbda00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd9e000640 1 -- 172.21.15.32:0/1565749266 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133458626 0 0) 0x7fbda00714d0 con 0x7fbda0145300 2024-03-06T07:35:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.677+0000 7fbd97e00640 1 -- 172.21.15.32:0/1565749266 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0071a90 con 0x7fbda0070f10 2024-03-06T07:35:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.679+0000 7fbd97e00640 1 --2- 172.21.15.32:0/1565749266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0070f10 0x7fbda00cc950 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009870 tx=0x7fbd8802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33437f56dbad0c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.679+0000 7fbd9e000640 1 -- 172.21.15.32:0/1565749266 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88034070 con 0x7fbda0070f10 2024-03-06T07:35:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.679+0000 7fbd9e000640 1 -- 172.21.15.32:0/1565749266 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd88030860 con 0x7fbda0070f10 2024-03-06T07:35:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.679+0000 7fbd9e000640 1 -- 172.21.15.32:0/1565749266 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd88030c30 con 0x7fbda0070f10 2024-03-06T07:35:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 -- 172.21.15.32:0/1565749266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0070f10 msgr2=0x7fbda00cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/1565749266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0070f10 0x7fbda00cc950 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009870 tx=0x7fbd8802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 -- 172.21.15.32:0/1565749266 shutdown_connections 2024-03-06T07:35:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/1565749266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0070f10 0x7fbda00cc950 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 -- 172.21.15.32:0/1565749266 >> 172.21.15.32:0/1565749266 conn(0x7fbda006b3d0 msgr2=0x7fbda006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 -- 172.21.15.32:0/1565749266 shutdown_connections 2024-03-06T07:35:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.680+0000 7fbd9fe00640 1 -- 172.21.15.32:0/1565749266 wait complete. 2024-03-06T07:35:37.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.682+0000 7fbd9fe00640 1 Processor -- start 2024-03-06T07:35:37.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.682+0000 7fbd9fe00640 1 -- start start 2024-03-06T07:35:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.683+0000 7fbd9fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda0070f10 0x7fbda006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.683+0000 7fbd9fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbda00cce90 0x7fbda00aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.683+0000 7fbd9fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 0x7fbda013f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.683+0000 7fbd9fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda015cc80 con 0x7fbda0145300 2024-03-06T07:35:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.683+0000 7fbd9fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda01456e0 con 0x7fbda0070f10 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.683+0000 7fbd9fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda015ce00 con 0x7fbda00cce90 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 0x7fbda013f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 0x7fbda013f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50552/0 (socket says 172.21.15.32:50552) 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 -- 172.21.15.32:0/2365144314 learned_addr learned my addr 172.21.15.32:0/2365144314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9ea00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda0070f10 0x7fbda006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd97e00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbda00cce90 0x7fbda00aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 -- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbda00cce90 msgr2=0x7fbda00aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbda00cce90 0x7fbda00aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 -- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda0070f10 msgr2=0x7fbda006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda0070f10 0x7fbda006d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 -- 172.21.15.32:0/2365144314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00aa640 con 0x7fbda0145300 2024-03-06T07:35:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd97e00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbda00cce90 0x7fbda00aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.684+0000 7fbd9f400640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 0x7fbda013f730 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fbd9800c7d0 tx=0x7fbd9800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.685+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98003b70 con 0x7fbda0145300 2024-03-06T07:35:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.685+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd98003d30 con 0x7fbda0145300 2024-03-06T07:35:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.685+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9800aa90 con 0x7fbda0145300 2024-03-06T07:35:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.685+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda006dd20 con 0x7fbda0145300 2024-03-06T07:35:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.685+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda006e2e0 con 0x7fbda0145300 2024-03-06T07:35:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.686+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda004f330 con 0x7fbda0145300 2024-03-06T07:35:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.687+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd9800ac30 con 0x7fbda0145300 2024-03-06T07:35:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.688+0000 7fbd96a00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7c06b4d0 0x7fbd7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.688+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbd9808a1a0 con 0x7fbda0145300 2024-03-06T07:35:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.688+0000 7fbd9ea00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7c06b4d0 0x7fbd7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.689+0000 7fbd9ea00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7c06b4d0 0x7fbd7c06d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbd90000f60 tx=0x7fbd90007d00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.693+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd98058fb0 con 0x7fbda0145300 2024-03-06T07:35:37.909 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-03-06T07:35:37.907+0000 7f7d83800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:37.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.938+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbda00cc9c0 con 0x7fbda0145300 2024-03-06T07:35:37.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.942+0000 7fbd96a00640 1 -- 172.21.15.32:0/2365144314 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fbd980588b0 con 0x7fbda0145300 2024-03-06T07:35:37.944 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:37.944 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:35:37.945 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7c06b4d0 msgr2=0x7fbd7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7c06b4d0 0x7fbd7c06d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbd90000f60 tx=0x7fbd90007d00 comp rx=0 tx=0).stop 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 msgr2=0x7fbda013f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 0x7fbda013f730 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fbd9800c7d0 tx=0x7fbd9800cc90 comp rx=0 tx=0).stop 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 shutdown_connections 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbd7c06b4d0 0x7fbd7c06d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbda0145300 0x7fbda013f730 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbda00cce90 0x7fbda00aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 --2- 172.21.15.32:0/2365144314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda0070f10 0x7fbda006d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 >> 172.21.15.32:0/2365144314 conn(0x7fbda006b3d0 msgr2=0x7fbda00b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 shutdown_connections 2024-03-06T07:35:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:37.945+0000 7fbd9fe00640 1 -- 172.21.15.32:0/2365144314 wait complete. 2024-03-06T07:35:37.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:38.186 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:35:38.186 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:35:38.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.381+0000 7f8ad7200640 1 Processor -- start 2024-03-06T07:35:38.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.381+0000 7f8ad7200640 1 -- start start 2024-03-06T07:35:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8ad80b0f90 0x7f8ad80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 0x7f8ad80d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad80b1370 con 0x7f8ad80d05e0 2024-03-06T07:35:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad80d0a50 con 0x7f8ad80b0f90 2024-03-06T07:35:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80ccf80 con 0x7f8ad80cc730 2024-03-06T07:35:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8ad80d05e0 0x7f8ad815d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47176/0 (socket says 172.21.15.32:47176) 2024-03-06T07:35:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.383+0000 7f8ad6800640 1 -- 172.21.15.32:0/3564388963 learned_addr learned my addr 172.21.15.32:0/3564388963 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad5400640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 0x7f8ad80d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/3564388963 >> v2:172.21.15.39:3300/0 conn(0x7f8ad80b0f90 0x7f8ad80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541755782 0 0) 0x7f8ad80b1370 con 0x7f8ad80d05e0 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8abc003520 con 0x7f8ad80d05e0 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485584419 0 0) 0x7f8abc003520 con 0x7f8ad80d05e0 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 msgr2=0x7f8ad80d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 0x7f8ad80d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 >> v2:172.21.15.39:3300/0 conn(0x7f8ad80b0f90 msgr2=0x7f8ad80cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 --2- 172.21.15.32:0/3564388963 >> v2:172.21.15.39:3300/0 conn(0x7f8ad80b0f90 0x7f8ad80cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad80cd260 con 0x7f8ad80d05e0 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f8ad00019f0 con 0x7f8ad80d05e0 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.384+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 >> v1:172.21.15.32:6789/0 conn(0x7f8ad80d05e0 legacy=0x7f8ad815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad4a00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8abc0039b0 0x7f8abc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad4a00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 0x7f8abc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad4a00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc004d40 0x7f8abc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80d0a50 con 0x7f8abc0039b0 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80ccf80 con 0x7f8abc004d40 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc00b9c0 con 0x7f8ad80cc730 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8abc0039b0 0x7f8abc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad5400640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 0x7f8abc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad5e00640 1 -- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 msgr2=0x7f8abc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80cc730 0x7f8abc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad5e00640 1 -- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc004d40 msgr2=0x7f8abc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.385+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc004d40 0x7f8abc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.386+0000 7f8ad5e00640 1 -- 172.21.15.32:0/3564388963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc003520 con 0x7f8abc0039b0 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8abc0039b0 0x7f8abc003dd0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009ac0 tx=0x7f8abc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb875134f3af6c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8039070 con 0x7f8abc0039b0 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac802c710 con 0x7f8abc0039b0 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad4a00640 1 -- 172.21.15.32:0/3564388963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8009690 con 0x7f8abc0039b0 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad7200640 1 -- 172.21.15.32:0/3564388963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8abc0039b0 msgr2=0x7f8abc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad7200640 1 --2- 172.21.15.32:0/3564388963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8abc0039b0 0x7f8abc003dd0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009ac0 tx=0x7f8abc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad6800640 1 -- 172.21.15.32:0/3564388963 reap_dead start 2024-03-06T07:35:38.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad7200640 1 -- 172.21.15.32:0/3564388963 shutdown_connections 2024-03-06T07:35:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.387+0000 7f8ad7200640 1 -- 172.21.15.32:0/3564388963 >> 172.21.15.32:0/3564388963 conn(0x7f8ad80d1ae0 msgr2=0x7f8ad80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.388+0000 7f8ad7200640 1 -- 172.21.15.32:0/3564388963 shutdown_connections 2024-03-06T07:35:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.388+0000 7f8ad7200640 1 -- 172.21.15.32:0/3564388963 wait complete. 2024-03-06T07:35:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.390+0000 7f8ad7200640 1 Processor -- start 2024-03-06T07:35:38.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.390+0000 7f8ad7200640 1 -- start start 2024-03-06T07:35:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc0039b0 0x7f8ad815d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80c7d70 0x7f8ad80c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 0x7f8ad81fba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80ccb70 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80cd820 con 0x7f8abc0039b0 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80cccf0 con 0x7f8ad80c7d70 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 0x7f8ad81fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 0x7f8ad81fba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57540/0 (socket says 172.21.15.32:57540) 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.391+0000 7f8ad6800640 1 -- 172.21.15.32:0/1907884810 learned_addr learned my addr 172.21.15.32:0/1907884810 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:35:38.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc0039b0 0x7f8ad815d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad5400640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80c7d70 0x7f8ad80c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad6800640 1 -- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80c7d70 msgr2=0x7f8ad80c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad6800640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80c7d70 0x7f8ad80c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad6800640 1 -- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc0039b0 msgr2=0x7f8ad815d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad6800640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc0039b0 0x7f8ad815d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad6800640 1 -- 172.21.15.32:0/1907884810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad80c81f0 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.392+0000 7f8ad6800640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 0x7f8ad81fba20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00013d0 tx=0x7f8ad000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.393+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000a5b0 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.393+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad000a770 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.393+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000ab60 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.393+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad80c8470 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.393+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad80c9d80 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.394+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad80c8990 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.395+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ad000ad00 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.397+0000 7f8ace000640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ab806b500 0x7f8ab806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.397+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8ad0089630 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.397+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ab806b500 0x7f8ab806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.397+0000 7f8ad5e00640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ab806b500 0x7f8ab806d9c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8036000 tx=0x7f8ac8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.404+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ad001a070 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.643+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ad804f330 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.644+0000 7f8ace000640 1 -- 172.21.15.32:0/1907884810 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f8ad80c81f0 con 0x7f8ad81fb5a0 2024-03-06T07:35:38.646 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:35:38.646 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:35:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.646+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ab806b500 msgr2=0x7f8ab806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.646+0000 7f8ad7200640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ab806b500 0x7f8ab806d9c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8036000 tx=0x7f8ac8030000 comp rx=0 tx=0).stop 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.646+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 msgr2=0x7f8ad81fba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.646+0000 7f8ad7200640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 0x7f8ad81fba20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00013d0 tx=0x7f8ad000cf00 comp rx=0 tx=0).stop 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 shutdown_connections 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ab806b500 0x7f8ab806d9c0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad81fb5a0 0x7f8ad81fba20 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8ad80c7d70 0x7f8ad80c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 --2- 172.21.15.32:0/1907884810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8abc0039b0 0x7f8ad815d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 >> 172.21.15.32:0/1907884810 conn(0x7f8ad80d1ae0 msgr2=0x7f8ad806c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 shutdown_connections 2024-03-06T07:35:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:35:38.647+0000 7f8ad7200640 1 -- 172.21.15.32:0/1907884810 wait complete. 2024-03-06T07:35:38.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:35:38.891 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:35:38.891 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:35:50.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:50.323 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:08.892 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:36:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.993+0000 7f2be9837f40 1 Processor -- start 2024-03-06T07:36:08.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.993+0000 7f2be9837f40 1 -- start start 2024-03-06T07:36:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be9837f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b980e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be9837f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145b982830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be9837f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba0f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be9837f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145b8f8fd0 con 0x56145b99e930 2024-03-06T07:36:08.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be9837f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145b8fcf30 con 0x56145b9504a0 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be9837f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x56145b951a40 con 0x56145b9500c0 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba0f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba0f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36880/0 (socket says 172.21.15.32:36880) 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.994+0000 7f2be6e00640 1 -- 172.21.15.32:0/1691853751 learned_addr learned my addr 172.21.15.32:0/1691853751 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be5a00640 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145b982830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6400640 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b980e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6e00640 1 -- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 msgr2=0x56145b980e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6e00640 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b980e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6e00640 1 -- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 msgr2=0x56145b982830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6e00640 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145b982830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6e00640 1 -- 172.21.15.32:0/1691853751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145ba342c0 con 0x56145b99e930 2024-03-06T07:36:08.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2be6e00640 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba0f2a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2be0001600 tx=0x7f2be0001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1fc46262ae9111 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.995+0000 7f2bdfe00640 1 -- 172.21.15.32:0/1691853751 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0013620 con 0x56145b99e930 2024-03-06T07:36:08.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2bdfe00640 1 -- 172.21.15.32:0/1691853751 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be00137c0 con 0x56145b99e930 2024-03-06T07:36:08.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2bdfe00640 1 -- 172.21.15.32:0/1691853751 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0013bb0 con 0x56145b99e930 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 -- 172.21.15.32:0/1691853751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 msgr2=0x56145ba0f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba0f2a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2be0001600 tx=0x7f2be0001ac0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 -- 172.21.15.32:0/1691853751 shutdown_connections 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba0f2a0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145b982830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 --2- 172.21.15.32:0/1691853751 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b980e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.996+0000 7f2be9837f40 1 -- 172.21.15.32:0/1691853751 >> 172.21.15.32:0/1691853751 conn(0x56145b981c00 msgr2=0x56145ba0d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.997+0000 7f2be9837f40 1 -- 172.21.15.32:0/1691853751 shutdown_connections 2024-03-06T07:36:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.997+0000 7f2be9837f40 1 -- 172.21.15.32:0/1691853751 wait complete. 2024-03-06T07:36:09.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.998+0000 7f2be9837f40 1 Processor -- start 2024-03-06T07:36:09.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:08.998+0000 7f2be9837f40 1 -- start start 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be9837f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b99c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be9837f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145ba22e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be9837f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba2ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be9837f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ba127b0 con 0x56145b99e930 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be9837f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145b982d70 con 0x56145b9504a0 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be9837f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x56145b93f810 con 0x56145b9500c0 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba2ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba2ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36888/0 (socket says 172.21.15.32:36888) 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 learned_addr learned my addr 172.21.15.32:0/209591376 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b99c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 msgr2=0x56145b99c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x56145b9500c0 0x56145b99c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 msgr2=0x56145ba22e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.000+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145ba22e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145ba22e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145ba0f6e0 con 0x56145b99e930 2024-03-06T07:36:09.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56145b9504a0 0x56145ba22e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:09.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba2ab80 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2be0001fc0 tx=0x7f2be0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2bdea00640 1 -- 172.21.15.32:0/209591376 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0003450 con 0x56145b99e930 2024-03-06T07:36:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2bdea00640 1 -- 172.21.15.32:0/209591376 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be0021070 con 0x56145b99e930 2024-03-06T07:36:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2bdea00640 1 -- 172.21.15.32:0/209591376 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0003830 con 0x56145b99e930 2024-03-06T07:36:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56145ba23690 con 0x56145b99e930 2024-03-06T07:36:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.001+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56145b984940 con 0x56145b99e930 2024-03-06T07:36:09.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.003+0000 7f2bdea00640 1 -- 172.21.15.32:0/209591376 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2be001b070 con 0x56145b99e930 2024-03-06T07:36:09.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.004+0000 7f2bdea00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2bc8069ee0 0x7f2bc806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.004+0000 7f2bdea00640 1 -- 172.21.15.32:0/209591376 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2be008a300 con 0x56145b99e930 2024-03-06T07:36:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.005+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2bc8069ee0 0x7f2bc806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.005+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x56145ba0dc00 0x56145ba0a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.005+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56145ba2b570 con 0x56145ba0dc00 2024-03-06T07:36:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.005+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2bc8069ee0 0x7f2bc806c3a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x56145b9816c0 tx=0x7f2bd8005640 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.006+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x56145ba0dc00 0x56145ba0a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.006+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x56145ba0dc00 0x56145ba0a580 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.007+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 1 ==== osd_op_reply(1 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd4052970 con 0x56145ba0dc00 2024-03-06T07:36:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.007+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f2bd4006380 0x7f2bd40067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.007+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.008+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f2bd4006380 0x7f2bd40067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.008+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f2bd4006380 0x7f2bd40067e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.009+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 1 ==== osd_op_reply(2 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.009+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f2bd8018040 0x7f2bd8009940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.009+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.009+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f2bd8018040 0x7f2bd8009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.010+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f2bd8018040 0x7f2bd8009940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.010+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 1 ==== osd_op_reply(3 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd4052970 con 0x7f2bd8018040 2024-03-06T07:36:09.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.010+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f2bd4050bf0 0x7f2bd4060440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.010+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4050bf0 2024-03-06T07:36:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.011+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f2bd4050bf0 0x7f2bd4060440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.011+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f2bd4050bf0 0x7f2bd4060440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.012+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.2 v2:172.21.15.32:6800/1600044506 1 ==== osd_op_reply(4 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd4050bf0 2024-03-06T07:36:09.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.012+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.013+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 2 ==== osd_op_reply(5 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd4052970 con 0x56145ba0dc00 2024-03-06T07:36:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.013+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.013+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 2 ==== osd_op_reply(6 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd4052970 con 0x7f2bd8018040 2024-03-06T07:36:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.013+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f2bd406e400 0x7f2bd406e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.013+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406e400 2024-03-06T07:36:09.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.014+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f2bd406e400 0x7f2bd406e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.015+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f2bd406e400 0x7f2bd406e880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.016+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.5 v2:172.21.15.39:6816/969666024 1 ==== osd_op_reply(7 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd406e400 2024-03-06T07:36:09.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.016+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.017+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 3 ==== osd_op_reply(8 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd4052970 con 0x56145ba0dc00 2024-03-06T07:36:09.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.017+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f2bd406d8a0 0x7f2bd40513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.017+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.017+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f2bd406d8a0 0x7f2bd40513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.018+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f2bd406d8a0 0x7f2bd40513a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.018+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== osd_op_reply(9 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.018+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.019+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 3 ==== osd_op_reply(10 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.019+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.019+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 4 ==== osd_op_reply(11 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.019+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.019+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 2 ==== osd_op_reply(12 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.019+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.020+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 4 ==== osd_op_reply(13 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.020+0000 7f2be5a00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f2bd407f040 0x7f2bd407f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.020+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.021+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f2bd407f040 0x7f2bd407f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.022+0000 7f2be6e00640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f2bd407f040 0x7f2bd407f930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.023+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 1 ==== osd_op_reply(14 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.023+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.024+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 2 ==== osd_op_reply(15 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.024+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.024+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 3 ==== osd_op_reply(16 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.024+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.025+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 2 ==== osd_op_reply(17 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.025+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406e400 2024-03-06T07:36:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.026+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.5 v2:172.21.15.39:6816/969666024 2 ==== osd_op_reply(18 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd406e400 2024-03-06T07:36:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.026+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.026+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 5 ==== osd_op_reply(19 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.026+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.027+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 5 ==== osd_op_reply(20 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.027+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.028+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 6 ==== osd_op_reply(21 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.028+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.029+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 6 ==== osd_op_reply(22 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.029+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.029+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 4 ==== osd_op_reply(23 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.029+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.030+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 3 ==== osd_op_reply(24 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.030+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f2bd8027400 0x7f2bd8027880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.030+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8027400 2024-03-06T07:36:09.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.030+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f2bd8027400 0x7f2bd8027880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.032+0000 7f2be6400640 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f2bd8027400 0x7f2bd8027880 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.033+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.6 v2:172.21.15.39:6808/2944464285 1 ==== osd_op_reply(25 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd8027400 2024-03-06T07:36:09.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.033+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.033+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 5 ==== osd_op_reply(26 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.033+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406e400 2024-03-06T07:36:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.034+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.5 v2:172.21.15.39:6816/969666024 3 ==== osd_op_reply(27 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd406e400 2024-03-06T07:36:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.034+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.035+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 6 ==== osd_op_reply(28 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.035+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4050bf0 2024-03-06T07:36:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.035+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.2 v2:172.21.15.32:6800/1600044506 2 ==== osd_op_reply(29 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd4050bf0 2024-03-06T07:36:09.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.035+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8027400 2024-03-06T07:36:09.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.036+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.6 v2:172.21.15.39:6808/2944464285 2 ==== osd_op_reply(30 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd8027400 2024-03-06T07:36:09.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.036+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.037+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 4 ==== osd_op_reply(31 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.037+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.038+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 3 ==== osd_op_reply(32 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.038+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.039+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 7 ==== osd_op_reply(33 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.039+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.039+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 8 ==== osd_op_reply(34 [pgnls start_epoch 116 out=49b] v110'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.039+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.040+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 7 ==== osd_op_reply(35 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.040+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.040+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 5 ==== osd_op_reply(36 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.040+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.041+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 8 ==== osd_op_reply(37 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.041+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.042+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 4 ==== osd_op_reply(38 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.042+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406e400 2024-03-06T07:36:09.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.043+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.5 v2:172.21.15.39:6816/969666024 4 ==== osd_op_reply(39 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd406e400 2024-03-06T07:36:09.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.043+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406e400 2024-03-06T07:36:09.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.044+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.5 v2:172.21.15.39:6816/969666024 5 ==== osd_op_reply(40 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd406e400 2024-03-06T07:36:09.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.044+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8027400 2024-03-06T07:36:09.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.045+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.6 v2:172.21.15.39:6808/2944464285 3 ==== osd_op_reply(41 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd8027400 2024-03-06T07:36:09.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.045+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.045+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 6 ==== osd_op_reply(42 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.045+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406e400 2024-03-06T07:36:09.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.047+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.5 v2:172.21.15.39:6816/969666024 6 ==== osd_op_reply(43 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd406e400 2024-03-06T07:36:09.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.047+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.048+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 5 ==== osd_op_reply(44 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.048+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.049+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 7 ==== osd_op_reply(45 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.049+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.050+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 8 ==== osd_op_reply(46 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.050+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.051+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 9 ==== osd_op_reply(47 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.051+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.052+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 10 ==== osd_op_reply(48 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.052+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.053+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 6 ==== osd_op_reply(49 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.053+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.054+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 7 ==== osd_op_reply(50 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.054+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8018040 2024-03-06T07:36:09.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.055+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.3 v2:172.21.15.32:6806/238384581 9 ==== osd_op_reply(51 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd8018040 2024-03-06T07:36:09.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.055+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x56145ba0dc00 2024-03-06T07:36:09.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.056+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.7 v2:172.21.15.39:6823/3019464823 11 ==== osd_op_reply(52 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x56145ba0dc00 2024-03-06T07:36:09.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.056+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.057+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 9 ==== osd_op_reply(53 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.057+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.058+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 8 ==== osd_op_reply(54 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.058+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.059+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 9 ==== osd_op_reply(55 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.059+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4050bf0 2024-03-06T07:36:09.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.060+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.2 v2:172.21.15.32:6800/1600044506 3 ==== osd_op_reply(56 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd4050bf0 2024-03-06T07:36:09.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.060+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.060+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 10 ==== osd_op_reply(57 [pgnls start_epoch 116 out=81b] v110'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.060+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.061+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 10 ==== osd_op_reply(58 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.061+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8027400 2024-03-06T07:36:09.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.062+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.6 v2:172.21.15.39:6808/2944464285 4 ==== osd_op_reply(59 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd8027400 2024-03-06T07:36:09.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.063+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.063+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 11 ==== osd_op_reply(60 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.063+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd4006380 2024-03-06T07:36:09.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.064+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.1 v2:172.21.15.32:6817/50998742 7 ==== osd_op_reply(61 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd4006380 2024-03-06T07:36:09.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.064+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd407f040 2024-03-06T07:36:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.065+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 <== osd.4 v2:172.21.15.39:6800/89256438 11 ==== osd_op_reply(62 [pgnls start_epoch 116 out=81b] v110'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f2be000f030 con 0x7f2bd407f040 2024-03-06T07:36:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.065+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd8027400 2024-03-06T07:36:09.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.066+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 <== osd.6 v2:172.21.15.39:6808/2944464285 5 ==== osd_op_reply(63 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd8019790 con 0x7f2bd8027400 2024-03-06T07:36:09.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.066+0000 7f2be6400640 1 -- 172.21.15.32:0/209591376 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f2bd4050340 con 0x7f2bd406d8a0 2024-03-06T07:36:09.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be5a00640 1 -- 172.21.15.32:0/209591376 <== osd.0 v2:172.21.15.32:6816/3728617102 12 ==== osd_op_reply(64 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2bd407c040 con 0x7f2bd406d8a0 2024-03-06T07:36:09.069 INFO:teuthology.orchestra.run.smithi032.stdout:10000000001.00000000 2024-03-06T07:36:09.069 INFO:teuthology.orchestra.run.smithi032.stdout:10000000001.00000001 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f2bd406d8a0 msgr2=0x7f2bd40513a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f2bd406d8a0 0x7f2bd40513a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f2bd4006380 msgr2=0x7f2bd40067e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f2bd4006380 0x7f2bd40067e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f2bd4050bf0 msgr2=0x7f2bd4060440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f2bd4050bf0 0x7f2bd4060440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f2bd8018040 msgr2=0x7f2bd8009940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f2bd8018040 0x7f2bd8009940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 reap_dead start 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f2bd407f040 msgr2=0x7f2bd407f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f2bd407f040 0x7f2bd407f930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f2bd406e400 msgr2=0x7f2bd406e880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f2bd406e400 0x7f2bd406e880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f2bd8027400 msgr2=0x7f2bd8027880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f2bd8027400 0x7f2bd8027880 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x56145ba0dc00 msgr2=0x56145ba0a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x56145ba0dc00 0x56145ba0a580 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be6e00640 1 -- 172.21.15.32:0/209591376 reap_dead start 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2bc8069ee0 msgr2=0x7f2bc806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2bc8069ee0 0x7f2bc806c3a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x56145b9816c0 tx=0x7f2bd8005640 comp rx=0 tx=0).stop 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 msgr2=0x56145ba2ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.067+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba2ab80 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2be0001fc0 tx=0x7f2be0000f40 comp rx=0 tx=0).stop 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.068+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 shutdown_connections 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.068+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2bc8069ee0 0x7f2bc806c3a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.068+0000 7f2be9837f40 1 --2- 172.21.15.32:0/209591376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56145b99e930 0x56145ba2ab80 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.068+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 >> 172.21.15.32:0/209591376 conn(0x56145b981c00 msgr2=0x56145ba0d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.068+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 shutdown_connections 2024-03-06T07:36:09.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.068+0000 7f2be9837f40 1 -- 172.21.15.32:0/209591376 wait complete. 2024-03-06T07:36:09.077 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:36:09.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:09.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.288+0000 7fa859000640 1 Processor -- start 2024-03-06T07:36:09.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.288+0000 7fa859000640 1 -- start start 2024-03-06T07:36:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.289+0000 7fa859000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa85415e070 0x7fa8540d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa859000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8540cf410 0x7fa8540cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa859000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8540d05c0 0x7fa8540d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa859000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8540d18c0 con 0x7fa8540d05c0 2024-03-06T07:36:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa859000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8540d1ba0 con 0x7fa85415e070 2024-03-06T07:36:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa859000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540d1e80 con 0x7fa8540cf410 2024-03-06T07:36:09.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa853e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8540d05c0 0x7fa8540d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa853e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8540d05c0 0x7fa8540d0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36898/0 (socket says 172.21.15.32:36898) 2024-03-06T07:36:09.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa853e00640 1 -- 172.21.15.32:0/2114852842 learned_addr learned my addr 172.21.15.32:0/2114852842 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:09.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa853400640 1 --2- 172.21.15.32:0/2114852842 >> v2:172.21.15.39:3300/0 conn(0x7fa85415e070 0x7fa8540d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.290+0000 7fa852a00640 1 --2- 172.21.15.32:0/2114852842 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8540cf410 0x7fa8540cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.291+0000 7fa853e00640 1 -- 172.21.15.32:0/2114852842 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8540cf410 msgr2=0x7fa8540cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:09.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.291+0000 7fa853e00640 1 --2- 172.21.15.32:0/2114852842 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8540cf410 0x7fa8540cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.291+0000 7fa853e00640 1 -- 172.21.15.32:0/2114852842 >> v2:172.21.15.39:3300/0 conn(0x7fa85415e070 msgr2=0x7fa8540d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:09.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.291+0000 7fa853e00640 1 --2- 172.21.15.32:0/2114852842 >> v2:172.21.15.39:3300/0 conn(0x7fa85415e070 0x7fa8540d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.291+0000 7fa853e00640 1 -- 172.21.15.32:0/2114852842 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8540d2160 con 0x7fa8540d05c0 2024-03-06T07:36:09.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.294+0000 7fa853e00640 1 --2- 172.21.15.32:0/2114852842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540d0a50 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa85408d270 tx=0x7fa84c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6fe67619480620 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.294+0000 7fa852000640 1 -- 172.21.15.32:0/2114852842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c02ca70 con 0x7fa8540d05c0 2024-03-06T07:36:09.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.294+0000 7fa852000640 1 -- 172.21.15.32:0/2114852842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c02cc10 con 0x7fa8540d05c0 2024-03-06T07:36:09.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.294+0000 7fa852000640 1 -- 172.21.15.32:0/2114852842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c0343f0 con 0x7fa8540d05c0 2024-03-06T07:36:09.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 -- 172.21.15.32:0/2114852842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 msgr2=0x7fa8540d0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 --2- 172.21.15.32:0/2114852842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540d0a50 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa85408d270 tx=0x7fa84c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 -- 172.21.15.32:0/2114852842 shutdown_connections 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 --2- 172.21.15.32:0/2114852842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540d0a50 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 --2- 172.21.15.32:0/2114852842 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa8540cf410 0x7fa8540cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 --2- 172.21.15.32:0/2114852842 >> v2:172.21.15.39:3300/0 conn(0x7fa85415e070 0x7fa8540d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 -- 172.21.15.32:0/2114852842 >> 172.21.15.32:0/2114852842 conn(0x7fa85408d2a0 msgr2=0x7fa8540d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 -- 172.21.15.32:0/2114852842 shutdown_connections 2024-03-06T07:36:09.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.295+0000 7fa859000640 1 -- 172.21.15.32:0/2114852842 wait complete. 2024-03-06T07:36:09.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.297+0000 7fa859000640 1 Processor -- start 2024-03-06T07:36:09.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.297+0000 7fa859000640 1 -- start start 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa859000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8540cf410 0x7fa854154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa859000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa859000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa85415e070 0x7fa8540b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa859000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540d4730 con 0x7fa8540d05c0 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa859000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540d45b0 con 0x7fa8540cf410 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa859000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540b3630 con 0x7fa85415e070 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa852a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa852a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36904/0 (socket says 172.21.15.32:36904) 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.298+0000 7fa852a00640 1 -- 172.21.15.32:0/3357120514 learned_addr learned my addr 172.21.15.32:0/3357120514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa853400640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8540cf410 0x7fa854154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa853e00640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa85415e070 0x7fa8540b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa852a00640 1 -- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa85415e070 msgr2=0x7fa8540b8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa852a00640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa85415e070 0x7fa8540b8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa852a00640 1 -- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8540cf410 msgr2=0x7fa854154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa852a00640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8540cf410 0x7fa854154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa852a00640 1 -- 172.21.15.32:0/3357120514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854160360 con 0x7fa8540d05c0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.299+0000 7fa852a00640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540b8920 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa84000d800 tx=0x7fa84000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.300+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840018020 con 0x7fa8540d05c0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.300+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400096a0 con 0x7fa8540d05c0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.300+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840009a70 con 0x7fa8540d05c0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.300+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8540c9280 con 0x7fa8540d05c0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.300+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8540c9840 con 0x7fa8540d05c0 2024-03-06T07:36:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.301+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85404f330 con 0x7fa8540d05c0 2024-03-06T07:36:09.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.302+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa84001b020 con 0x7fa8540d05c0 2024-03-06T07:36:09.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.303+0000 7fa84b400640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa83406b4d0 0x7fa83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.303+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa84008a550 con 0x7fa8540d05c0 2024-03-06T07:36:09.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.303+0000 7fa853400640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa83406b4d0 0x7fa83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.304+0000 7fa853400640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa83406b4d0 0x7fa83406d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa85406c5c0 tx=0x7fa844008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.309+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa840022030 con 0x7fa8540d05c0 2024-03-06T07:36:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.675+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8540d1370 con 0x7fa8540d05c0 2024-03-06T07:36:09.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.677+0000 7fa84b400640 1 -- 172.21.15.32:0/3357120514 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fa8400592a0 con 0x7fa8540d05c0 2024-03-06T07:36:09.678 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:09.678 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:09.679 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.679+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa83406b4d0 msgr2=0x7fa83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.679+0000 7fa859000640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa83406b4d0 0x7fa83406d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa85406c5c0 tx=0x7fa844008040 comp rx=0 tx=0).stop 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.679+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 msgr2=0x7fa8540b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.679+0000 7fa859000640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540b8920 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa84000d800 tx=0x7fa84000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 shutdown_connections 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa83406b4d0 0x7fa83406d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa85415e070 0x7fa8540b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8540d05c0 0x7fa8540b8920 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 --2- 172.21.15.32:0/3357120514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8540cf410 0x7fa854154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 >> 172.21.15.32:0/3357120514 conn(0x7fa85408d2a0 msgr2=0x7fa85415d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 shutdown_connections 2024-03-06T07:36:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:09.680+0000 7fa859000640 1 -- 172.21.15.32:0/3357120514 wait complete. 2024-03-06T07:36:09.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:36:09.923 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:36:09.923 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:36:09.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.145+0000 7f7aa4a00640 1 Processor -- start 2024-03-06T07:36:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.145+0000 7f7aa4a00640 1 -- start start 2024-03-06T07:36:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.146+0000 7f7aa4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa015e0b0 0x7f7aa00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.146+0000 7f7aa4a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7aa00d4a10 0x7f7aa00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.146+0000 7f7aa4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa00b3710 con 0x7f7aa00d0c80 2024-03-06T07:36:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.146+0000 7f7aa4a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa00b09e0 con 0x7f7aa00d4a10 2024-03-06T07:36:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.146+0000 7f7aa4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa00b39f0 con 0x7f7aa015e0b0 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7aa00d4a10 0x7f7aa00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7aa00d4a10 0x7f7aa00d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36920/0 (socket says 172.21.15.32:36920) 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 -- 172.21.15.32:0/2448120376 learned_addr learned my addr 172.21.15.32:0/2448120376 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a9e800640 1 --2- 172.21.15.32:0/2448120376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa015e0b0 0x7f7aa00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 -- 172.21.15.32:0/2448120376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa015e0b0 msgr2=0x7f7aa00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 --2- 172.21.15.32:0/2448120376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa015e0b0 0x7f7aa00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 -- 172.21.15.32:0/2448120376 >> v1:172.21.15.39:6789/0 conn(0x7f7aa00d0c80 legacy=0x7f7aa00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.147+0000 7f7a97e00640 1 -- 172.21.15.32:0/2448120376 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa00b3cd0 con 0x7f7aa00d4a10 2024-03-06T07:36:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.148+0000 7f7a97e00640 1 --2- 172.21.15.32:0/2448120376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d4a10 0x7f7aa00d0740 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009870 tx=0x7f7a8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a30988c70292d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.149+0000 7f7a9de00640 1 -- 172.21.15.32:0/2448120376 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88035020 con 0x7f7aa00d4a10 2024-03-06T07:36:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.149+0000 7f7a9de00640 1 -- 172.21.15.32:0/2448120376 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88031640 con 0x7f7aa00d4a10 2024-03-06T07:36:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.149+0000 7f7a9de00640 1 -- 172.21.15.32:0/2448120376 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8802d040 con 0x7f7aa00d4a10 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 -- 172.21.15.32:0/2448120376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d4a10 msgr2=0x7f7aa00d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/2448120376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d4a10 0x7f7aa00d0740 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009870 tx=0x7f7a8802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 -- 172.21.15.32:0/2448120376 shutdown_connections 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/2448120376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d4a10 0x7f7aa00d0740 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/2448120376 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa015e0b0 0x7f7aa00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 -- 172.21.15.32:0/2448120376 >> 172.21.15.32:0/2448120376 conn(0x7f7aa00d3d30 msgr2=0x7f7aa00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 -- 172.21.15.32:0/2448120376 shutdown_connections 2024-03-06T07:36:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.150+0000 7f7aa4a00640 1 -- 172.21.15.32:0/2448120376 wait complete. 2024-03-06T07:36:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.151+0000 7f7aa4a00640 1 Processor -- start 2024-03-06T07:36:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.152+0000 7f7aa4a00640 1 -- start start 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7aa4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 0x7f7aa00cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7aa4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa00d4a10 0x7f7aa0160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7aa4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa015e0b0 0x7f7aa01605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7aa4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa00cfc60 con 0x7f7aa00d0c80 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7aa4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa015e490 con 0x7f7aa015e0b0 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7aa4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa00b54e0 con 0x7f7aa00d4a10 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7a9e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 0x7f7aa00cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7a9e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 0x7f7aa00cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36926/0 (socket says 172.21.15.32:36926) 2024-03-06T07:36:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7a9e800640 1 -- 172.21.15.32:0/1983529008 learned_addr learned my addr 172.21.15.32:0/1983529008 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7a97e00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa00d4a10 0x7f7aa0160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.153+0000 7f7a9f200640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa015e0b0 0x7f7aa01605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a9e800640 1 -- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa00d4a10 msgr2=0x7f7aa0160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a9e800640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa00d4a10 0x7f7aa0160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a9e800640 1 -- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa015e0b0 msgr2=0x7f7aa01605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a9e800640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa015e0b0 0x7f7aa01605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a9e800640 1 -- 172.21.15.32:0/1983529008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa01f3cc0 con 0x7f7aa00d0c80 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a9e800640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 0x7f7aa00cb070 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000d5f0 tx=0x7f7a9000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a900125f0 con 0x7f7aa00d0c80 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a90019070 con 0x7f7aa00d0c80 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa0160be0 con 0x7f7aa00d0c80 2024-03-06T07:36:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a900129d0 con 0x7f7aa00d0c80 2024-03-06T07:36:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.154+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa00b7750 con 0x7f7aa00d0c80 2024-03-06T07:36:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.156+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa004f330 con 0x7f7aa00d0c80 2024-03-06T07:36:10.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.157+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a90012b70 con 0x7f7aa00d0c80 2024-03-06T07:36:10.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.158+0000 7f7a96a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.158+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7a9008a610 con 0x7f7aa00d0c80 2024-03-06T07:36:10.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.159+0000 7f7a97e00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.159+0000 7f7a97e00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7a88036000 tx=0x7f7a8803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.162+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a90059490 con 0x7f7aa00d0c80 2024-03-06T07:36:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.516+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa00d3300 con 0x7f7aa00d0c80 2024-03-06T07:36:10.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.517+0000 7f7a96a00640 1 -- 172.21.15.32:0/1983529008 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f7a90058d90 con 0x7f7aa00d0c80 2024-03-06T07:36:10.518 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:10.519 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:10.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a7c06b4d0 msgr2=0x7f7a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7a88036000 tx=0x7f7a8803c040 comp rx=0 tx=0).stop 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 msgr2=0x7f7aa00cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 0x7f7aa00cb070 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000d5f0 tx=0x7f7a9000dab0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 shutdown_connections 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a7c06b4d0 0x7f7a7c06d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa015e0b0 0x7f7aa01605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7aa00d4a10 0x7f7aa0160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 --2- 172.21.15.32:0/1983529008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa00d0c80 0x7f7aa00cb070 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 >> 172.21.15.32:0/1983529008 conn(0x7f7aa00d3d30 msgr2=0x7f7aa00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 shutdown_connections 2024-03-06T07:36:10.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.520+0000 7f7aa4a00640 1 -- 172.21.15.32:0/1983529008 wait complete. 2024-03-06T07:36:10.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:36:10.781 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:36:10.781 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:36:10.782 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:36:10.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.869+0000 7ff575640f40 1 Processor -- start 2024-03-06T07:36:10.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.869+0000 7ff575640f40 1 -- start start 2024-03-06T07:36:10.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff575640f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff575640f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x56508268f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff575640f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650825ff600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff575640f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082578fd0 con 0x5650825d00d0 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff575640f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508257cf30 con 0x5650826b2060 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff575640f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650825d1a50 con 0x5650825d04b0 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.870+0000 7ff572200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36954/0 (socket says 172.21.15.32:36954) 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 -- 172.21.15.32:0/3922730553 learned_addr learned my addr 172.21.15.32:0/3922730553 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572c00640 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650825ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 -- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 msgr2=0x56508268f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x56508268f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 -- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 msgr2=0x5650825ff600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650825ff600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 -- 172.21.15.32:0/3922730553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650826b2740 con 0x5650825d00d0 2024-03-06T07:36:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.871+0000 7ff572200640 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b2e80 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ff56400a430 tx=0x7ff56400a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efd4c02531f0aba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff56be00640 1 -- 172.21.15.32:0/3922730553 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56400b920 con 0x5650825d00d0 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff56be00640 1 -- 172.21.15.32:0/3922730553 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56400f030 con 0x5650825d00d0 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff56be00640 1 -- 172.21.15.32:0/3922730553 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56400bd20 con 0x5650825d00d0 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 -- 172.21.15.32:0/3922730553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 msgr2=0x5650826b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b2e80 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ff56400a430 tx=0x7ff56400a8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 -- 172.21.15.32:0/3922730553 shutdown_connections 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650825ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x56508268f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 --2- 172.21.15.32:0/3922730553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b2e80 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 -- 172.21.15.32:0/3922730553 >> 172.21.15.32:0/3922730553 conn(0x565082601c10 msgr2=0x565082690650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 -- 172.21.15.32:0/3922730553 shutdown_connections 2024-03-06T07:36:10.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.872+0000 7ff575640f40 1 -- 172.21.15.32:0/3922730553 wait complete. 2024-03-06T07:36:10.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.873+0000 7ff575640f40 1 Processor -- start 2024-03-06T07:36:10.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.873+0000 7ff575640f40 1 -- start start 2024-03-06T07:36:10.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff575640f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff575640f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x5650826ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff575640f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650826abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff575640f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650826b3740 con 0x5650825d00d0 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff575640f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650826b3460 con 0x5650826b2060 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff575640f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650826b3a20 con 0x5650825d04b0 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff572200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff572200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36962/0 (socket says 172.21.15.32:36962) 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff572200640 1 -- 172.21.15.32:0/2875530624 learned_addr learned my addr 172.21.15.32:0/2875530624 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.875+0000 7ff571800640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x5650826ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572200640 1 -- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 msgr2=0x5650826ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572200640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x5650826ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572200640 1 -- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 msgr2=0x5650826abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572c00640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650826abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572200640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650826abb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572200640 1 -- 172.21.15.32:0/2875530624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508268df80 con 0x5650825d00d0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff571800640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5650825d04b0 0x5650826ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572c00640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5650826b2060 0x5650826abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff572200640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b15d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x56508268e1b0 tx=0x7ff564002f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff56aa00640 1 -- 172.21.15.32:0/2875530624 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564024070 con 0x5650825d00d0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff56aa00640 1 -- 172.21.15.32:0/2875530624 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff564014620 con 0x5650825d00d0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff56aa00640 1 -- 172.21.15.32:0/2875530624 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564014a10 con 0x5650825d00d0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.876+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5650826b3160 con 0x5650825d00d0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.877+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565082685310 con 0x5650825d00d0 2024-03-06T07:36:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.878+0000 7ff56aa00640 1 -- 172.21.15.32:0/2875530624 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff564027070 con 0x5650825d00d0 2024-03-06T07:36:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.879+0000 7ff56aa00640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff554069f00 0x7ff55406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.879+0000 7ff56aa00640 1 -- 172.21.15.32:0/2875530624 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff564089d10 con 0x5650825d00d0 2024-03-06T07:36:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.880+0000 7ff571800640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff554069f00 0x7ff55406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.880+0000 7ff575640f40 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x5650826a7800 0x5650826a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:10.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.880+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x5650826a8190 con 0x5650826a7800 2024-03-06T07:36:10.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.880+0000 7ff572c00640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x5650826a7800 0x5650826a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:10.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.880+0000 7ff571800640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff554069f00 0x7ff55406c3c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x5650826b2b90 tx=0x7ff560001420 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff572c00640 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x5650826a7800 0x5650826a7c50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff572c00640 1 -- 172.21.15.32:0/2875530624 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7ff56c052a00 con 0x5650826a7800 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x5650826a7800 msgr2=0x5650826a7c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff575640f40 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x5650826a7800 0x5650826a7c50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff554069f00 msgr2=0x7ff55406c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff575640f40 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff554069f00 0x7ff55406c3c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x5650826b2b90 tx=0x7ff560001420 comp rx=0 tx=0).stop 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 msgr2=0x5650826b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.881+0000 7ff575640f40 1 --2- 172.21.15.32:0/2875530624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5650825d00d0 0x5650826b15d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x56508268e1b0 tx=0x7ff564002f70 comp rx=0 tx=0).stop 2024-03-06T07:36:10.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.882+0000 7ff572c00640 1 -- 172.21.15.32:0/2875530624 reap_dead start 2024-03-06T07:36:10.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.882+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 shutdown_connections 2024-03-06T07:36:10.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.882+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 >> 172.21.15.32:0/2875530624 conn(0x565082601c10 msgr2=0x56508268f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:10.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.882+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 shutdown_connections 2024-03-06T07:36:10.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:10.882+0000 7ff575640f40 1 -- 172.21.15.32:0/2875530624 wait complete. 2024-03-06T07:36:10.888 INFO:teuthology.orchestra.run.smithi032.stdout:] 10000000001 stray0 2024-03-06T07:36:10.888 DEBUG:teuthology.orchestra.run.smithi032:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:36:10.979 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:36:10.979 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1099511627777, 2024-03-06T07:36:10.979 INFO:teuthology.orchestra.run.smithi032.stdout: "ancestors": [ 2024-03-06T07:36:10.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:10.979 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1536, 2024-03-06T07:36:10.979 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "10000000001", 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 13 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 256, 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "stray0", 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 5 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 17, 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: "old_pools": [] 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:36:10.980 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:10.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:11.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.201+0000 7fd655400640 1 Processor -- start 2024-03-06T07:36:11.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.202+0000 7fd655400640 1 -- start start 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.203+0000 7fd655400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd6500b5eb0 0x7fd650166e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.203+0000 7fd655400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6500af420 con 0x7fd65015bf30 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.203+0000 7fd655400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd650070230 con 0x7fd6500b5160 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.203+0000 7fd655400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd650070510 con 0x7fd6500b5eb0 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd65015bf30 0x7fd65006f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54666/0 (socket says 172.21.15.32:54666) 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64e800640 1 -- 172.21.15.32:0/3295658088 learned_addr learned my addr 172.21.15.32:0/3295658088 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:11.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64f200640 1 --2- 172.21.15.32:0/3295658088 >> v2:172.21.15.32:3300/0 conn(0x7fd6500b5eb0 0x7fd650166e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64d400640 1 -- 172.21.15.32:0/3295658088 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776676989 0 0) 0x7fd640001110 con 0x7fd65015bf30 2024-03-06T07:36:11.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64d400640 1 -- 172.21.15.32:0/3295658088 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd634003520 con 0x7fd65015bf30 2024-03-06T07:36:11.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64f200640 1 -- 172.21.15.32:0/3295658088 >> v1:172.21.15.39:6789/0 conn(0x7fd6500b5160 legacy=0x7fd6500b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64f200640 1 -- 172.21.15.32:0/3295658088 >> v1:172.21.15.32:6789/0 conn(0x7fd65015bf30 legacy=0x7fd65006f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.204+0000 7fd64f200640 1 -- 172.21.15.32:0/3295658088 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6500707f0 con 0x7fd6500b5eb0 2024-03-06T07:36:11.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.206+0000 7fd64f200640 1 --2- 172.21.15.32:0/3295658088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650166e40 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fd648009d90 tx=0x7fd64802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641ca71f19cedae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.206+0000 7fd64d400640 1 -- 172.21.15.32:0/3295658088 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648039070 con 0x7fd6500b5eb0 2024-03-06T07:36:11.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.206+0000 7fd64d400640 1 -- 172.21.15.32:0/3295658088 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64802cc60 con 0x7fd6500b5eb0 2024-03-06T07:36:11.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.206+0000 7fd64d400640 1 -- 172.21.15.32:0/3295658088 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648034660 con 0x7fd6500b5eb0 2024-03-06T07:36:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.206+0000 7fd655400640 1 -- 172.21.15.32:0/3295658088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 msgr2=0x7fd650166e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.206+0000 7fd655400640 1 --2- 172.21.15.32:0/3295658088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650166e40 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fd648009d90 tx=0x7fd64802beb0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.207+0000 7fd655400640 1 -- 172.21.15.32:0/3295658088 shutdown_connections 2024-03-06T07:36:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.207+0000 7fd655400640 1 --2- 172.21.15.32:0/3295658088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650166e40 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.207+0000 7fd655400640 1 -- 172.21.15.32:0/3295658088 >> 172.21.15.32:0/3295658088 conn(0x7fd65009aa10 msgr2=0x7fd65009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.207+0000 7fd655400640 1 -- 172.21.15.32:0/3295658088 shutdown_connections 2024-03-06T07:36:11.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.207+0000 7fd655400640 1 -- 172.21.15.32:0/3295658088 wait complete. 2024-03-06T07:36:11.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.208+0000 7fd655400640 1 Processor -- start 2024-03-06T07:36:11.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 -- start start 2024-03-06T07:36:11.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6500b5160 0x7fd650159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd65015bf30 0x7fd650146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6500b08f0 con 0x7fd6500b5eb0 2024-03-06T07:36:11.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65015c310 con 0x7fd65015bf30 2024-03-06T07:36:11.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.209+0000 7fd655400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6500b0a70 con 0x7fd6500b5160 2024-03-06T07:36:11.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36986/0 (socket says 172.21.15.32:36986) 2024-03-06T07:36:11.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 -- 172.21.15.32:0/2873809897 learned_addr learned my addr 172.21.15.32:0/2873809897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:11.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64e800640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6500b5160 0x7fd650159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 -- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6500b5160 msgr2=0x7fd650159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6500b5160 0x7fd650159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 -- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd65015bf30 msgr2=0x7fd650146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64f200640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd65015bf30 0x7fd650146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd65015bf30 0x7fd650146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.210+0000 7fd64de00640 1 -- 172.21.15.32:0/2873809897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65015a3a0 con 0x7fd6500b5eb0 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.211+0000 7fd64de00640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650149a50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00b420 tx=0x7fd63c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.211+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c011830 con 0x7fd6500b5eb0 2024-03-06T07:36:11.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.211+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63c016070 con 0x7fd6500b5eb0 2024-03-06T07:36:11.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.211+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c011c10 con 0x7fd6500b5eb0 2024-03-06T07:36:11.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.211+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65015a680 con 0x7fd6500b5eb0 2024-03-06T07:36:11.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.211+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65015ac40 con 0x7fd6500b5eb0 2024-03-06T07:36:11.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.212+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65004f3a0 con 0x7fd6500b5eb0 2024-03-06T07:36:11.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.213+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd63c01b460 con 0x7fd6500b5eb0 2024-03-06T07:36:11.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.214+0000 7fd646a00640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd63006b500 0x7fd63006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.214+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd63c089e30 con 0x7fd6500b5eb0 2024-03-06T07:36:11.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.214+0000 7fd64e800640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd63006b500 0x7fd63006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.215+0000 7fd64e800640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd63006b500 0x7fd63006d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd640005880 tx=0x7fd640009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.222+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd63c020020 con 0x7fd6500b5eb0 2024-03-06T07:36:11.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.469+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6500ce240 con 0x7fd6500b5eb0 2024-03-06T07:36:11.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.470+0000 7fd646a00640 1 -- 172.21.15.32:0/2873809897 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fd63c058b80 con 0x7fd6500b5eb0 2024-03-06T07:36:11.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:11.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:11.472 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd63006b500 msgr2=0x7fd63006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd63006b500 0x7fd63006d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd640005880 tx=0x7fd640009040 comp rx=0 tx=0).stop 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 msgr2=0x7fd650149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650149a50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00b420 tx=0x7fd63c00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 shutdown_connections 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd63006b500 0x7fd63006d9c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd65015bf30 0x7fd650146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6500b5eb0 0x7fd650149a50 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 --2- 172.21.15.32:0/2873809897 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6500b5160 0x7fd650159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.473+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 >> 172.21.15.32:0/2873809897 conn(0x7fd65009aa10 msgr2=0x7fd650156d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.474+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 shutdown_connections 2024-03-06T07:36:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:11.474+0000 7fd655400640 1 -- 172.21.15.32:0/2873809897 wait complete. 2024-03-06T07:36:11.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:36:11.737 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:36:11.737 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:36:11.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:11.738 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:36:11.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.017+0000 7f3989600640 1 Processor -- start 2024-03-06T07:36:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.017+0000 7f3989600640 1 -- start start 2024-03-06T07:36:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3989600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39840d4a10 0x7f39840d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3989600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39840d0c80 0x7f39840b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3989600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840b3710 con 0x7f398415e0b0 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3989600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840b09e0 con 0x7f39840d4a10 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3989600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39840cfa20 con 0x7f39840d0c80 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3983400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39840d4a10 0x7f39840d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3983e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f398415e0b0 0x7f39840cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54680/0 (socket says 172.21.15.32:54680) 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.021+0000 7f3983e00640 1 -- 172.21.15.32:0/3012494491 learned_addr learned my addr 172.21.15.32:0/3012494491 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.022+0000 7f3988c00640 1 --2- 172.21.15.32:0/3012494491 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39840d0c80 0x7f39840b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.022+0000 7f3983400640 1 -- 172.21.15.32:0/3012494491 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39840d0c80 msgr2=0x7f39840b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.022+0000 7f3983400640 1 --2- 172.21.15.32:0/3012494491 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39840d0c80 0x7f39840b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.022+0000 7f3983400640 1 -- 172.21.15.32:0/3012494491 >> v1:172.21.15.32:6789/0 conn(0x7f398415e0b0 legacy=0x7f39840cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.022+0000 7f3983400640 1 -- 172.21.15.32:0/3012494491 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39840b39f0 con 0x7f39840d4a10 2024-03-06T07:36:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.023+0000 7f3983400640 1 --2- 172.21.15.32:0/3012494491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d4a10 0x7f39840d0740 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f39700097a0 tx=0x7f397002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d858904fdbe45b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.023+0000 7f3982a00640 1 -- 172.21.15.32:0/3012494491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970035020 con 0x7f39840d4a10 2024-03-06T07:36:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.023+0000 7f3982a00640 1 -- 172.21.15.32:0/3012494491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970031830 con 0x7f39840d4a10 2024-03-06T07:36:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.023+0000 7f3982a00640 1 -- 172.21.15.32:0/3012494491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970031c00 con 0x7f39840d4a10 2024-03-06T07:36:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 -- 172.21.15.32:0/3012494491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d4a10 msgr2=0x7f39840d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 --2- 172.21.15.32:0/3012494491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d4a10 0x7f39840d0740 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f39700097a0 tx=0x7f397002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 -- 172.21.15.32:0/3012494491 shutdown_connections 2024-03-06T07:36:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 --2- 172.21.15.32:0/3012494491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d4a10 0x7f39840d0740 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 --2- 172.21.15.32:0/3012494491 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f39840d0c80 0x7f39840b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 -- 172.21.15.32:0/3012494491 >> 172.21.15.32:0/3012494491 conn(0x7f39840d3d30 msgr2=0x7f39840d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:12.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 -- 172.21.15.32:0/3012494491 shutdown_connections 2024-03-06T07:36:12.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.024+0000 7f3989600640 1 -- 172.21.15.32:0/3012494491 wait complete. 2024-03-06T07:36:12.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.026+0000 7f3989600640 1 Processor -- start 2024-03-06T07:36:12.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.026+0000 7f3989600640 1 -- start start 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3989600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 0x7f39840cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3989600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39840d4a10 0x7f398414f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3989600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f398415e0b0 0x7f398406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3989600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398415e490 con 0x7f39840d0c80 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3989600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b3fb0 con 0x7f39840d4a10 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3989600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39840cfc60 con 0x7f398415e0b0 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3983e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 0x7f39840cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3983e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 0x7f39840cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37018/0 (socket says 172.21.15.32:37018) 2024-03-06T07:36:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.027+0000 7f3983e00640 1 -- 172.21.15.32:0/319411981 learned_addr learned my addr 172.21.15.32:0/319411981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3988c00640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f398415e0b0 0x7f398406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983400640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39840d4a10 0x7f398414f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983e00640 1 -- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f398415e0b0 msgr2=0x7f398406c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983e00640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f398415e0b0 0x7f398406c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983e00640 1 -- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39840d4a10 msgr2=0x7f398414f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983e00640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39840d4a10 0x7f398414f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983e00640 1 -- 172.21.15.32:0/319411981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398414f650 con 0x7f39840d0c80 2024-03-06T07:36:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3983e00640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 0x7f39840cd080 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f397400c400 tx=0x7f397400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974017070 con 0x7f39840d0c80 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974005ce0 con 0x7f39840d0c80 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39740072f0 con 0x7f39840d0c80 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39840cbaf0 con 0x7f39840d0c80 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.028+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39840cc880 con 0x7f39840d0c80 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.030+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398404f330 con 0x7f39840d0c80 2024-03-06T07:36:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.030+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f397401a070 con 0x7f39840d0c80 2024-03-06T07:36:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.031+0000 7f397be00640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f396406b4d0 0x7f396406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.031+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f39740896b0 con 0x7f39840d0c80 2024-03-06T07:36:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.032+0000 7f3983400640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f396406b4d0 0x7f396406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.033+0000 7f3983400640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f396406b4d0 0x7f396406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3970036000 tx=0x7f3970008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.036+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3974058490 con 0x7f39840d0c80 2024-03-06T07:36:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.285+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39840b1b20 con 0x7f39840d0c80 2024-03-06T07:36:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.285+0000 7f397be00640 1 -- 172.21.15.32:0/319411981 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f3974057d90 con 0x7f39840d0c80 2024-03-06T07:36:12.287 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:12.287 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:12.287 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f396406b4d0 msgr2=0x7f396406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f396406b4d0 0x7f396406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3970036000 tx=0x7f3970008030 comp rx=0 tx=0).stop 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 msgr2=0x7f39840cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 0x7f39840cd080 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f397400c400 tx=0x7f397400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 shutdown_connections 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f396406b4d0 0x7f396406d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f398415e0b0 0x7f398406c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39840d4a10 0x7f398414f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 --2- 172.21.15.32:0/319411981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39840d0c80 0x7f39840cd080 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.289+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 >> 172.21.15.32:0/319411981 conn(0x7f39840d3d30 msgr2=0x7f39840b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.290+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 shutdown_connections 2024-03-06T07:36:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.290+0000 7f3989600640 1 -- 172.21.15.32:0/319411981 wait complete. 2024-03-06T07:36:12.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:36:12.548 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:36:12.548 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:36:12.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.758+0000 7f93a3e00640 1 Processor -- start 2024-03-06T07:36:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.758+0000 7f93a3e00640 1 -- start start 2024-03-06T07:36:12.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f93a4145300 0x7f93a40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93a40cd060 0x7f93a415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a40714d0 con 0x7f93a4070f10 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a40717b0 con 0x7f93a40cd060 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a4071a90 con 0x7f93a4145300 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93a40cd060 0x7f93a415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93a40cd060 0x7f93a415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37028/0 (socket says 172.21.15.32:37028) 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a3400640 1 -- 172.21.15.32:0/65796511 learned_addr learned my addr 172.21.15.32:0/65796511 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.760+0000 7f93a2a00640 1 --2- 172.21.15.32:0/65796511 >> v2:172.21.15.39:3300/0 conn(0x7f93a4145300 0x7f93a40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.761+0000 7f93a3400640 1 -- 172.21.15.32:0/65796511 >> v2:172.21.15.39:3300/0 conn(0x7f93a4145300 msgr2=0x7f93a40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:12.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.761+0000 7f93a3400640 1 --2- 172.21.15.32:0/65796511 >> v2:172.21.15.39:3300/0 conn(0x7f93a4145300 0x7f93a40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.761+0000 7f93a3400640 1 -- 172.21.15.32:0/65796511 >> v1:172.21.15.39:6789/0 conn(0x7f93a4070f10 legacy=0x7f93a40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:12.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.761+0000 7f93a3400640 1 -- 172.21.15.32:0/65796511 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4071d70 con 0x7f93a40cd060 2024-03-06T07:36:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.763+0000 7f93a3400640 1 --2- 172.21.15.32:0/65796511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a40cd060 0x7f93a415c570 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f93a40cf990 tx=0x7f939c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f5c3958a26325 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.763+0000 7f93a1600640 1 -- 172.21.15.32:0/65796511 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c02e070 con 0x7f93a40cd060 2024-03-06T07:36:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.763+0000 7f93a1600640 1 -- 172.21.15.32:0/65796511 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c037070 con 0x7f93a40cd060 2024-03-06T07:36:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.763+0000 7f93a1600640 1 -- 172.21.15.32:0/65796511 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c031a00 con 0x7f93a40cd060 2024-03-06T07:36:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 -- 172.21.15.32:0/65796511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a40cd060 msgr2=0x7f93a415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 --2- 172.21.15.32:0/65796511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a40cd060 0x7f93a415c570 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f93a40cf990 tx=0x7f939c02b880 comp rx=0 tx=0).stop 2024-03-06T07:36:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 -- 172.21.15.32:0/65796511 shutdown_connections 2024-03-06T07:36:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 --2- 172.21.15.32:0/65796511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a40cd060 0x7f93a415c570 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 --2- 172.21.15.32:0/65796511 >> v2:172.21.15.39:3300/0 conn(0x7f93a4145300 0x7f93a40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 -- 172.21.15.32:0/65796511 >> 172.21.15.32:0/65796511 conn(0x7f93a406b3d0 msgr2=0x7f93a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 -- 172.21.15.32:0/65796511 shutdown_connections 2024-03-06T07:36:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.764+0000 7f93a3e00640 1 -- 172.21.15.32:0/65796511 wait complete. 2024-03-06T07:36:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.765+0000 7f93a3e00640 1 Processor -- start 2024-03-06T07:36:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.766+0000 7f93a3e00640 1 -- start start 2024-03-06T07:36:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.766+0000 7f93a3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 0x7f93a41e9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93a40cd060 0x7f93a40cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a4145300 0x7f93a40cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a415cab0 con 0x7f93a4070f10 2024-03-06T07:36:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a41456e0 con 0x7f93a4145300 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a3e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a415cc30 con 0x7f93a40cd060 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 0x7f93a41e9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 0x7f93a41e9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37038/0 (socket says 172.21.15.32:37038) 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 -- 172.21.15.32:0/2920217235 learned_addr learned my addr 172.21.15.32:0/2920217235 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 -- 172.21.15.32:0/2920217235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93a40cd060 msgr2=0x7f93a40cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93a40cd060 0x7f93a40cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 -- 172.21.15.32:0/2920217235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a4145300 msgr2=0x7f93a40cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a4145300 0x7f93a40cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.767+0000 7f93a2a00640 1 -- 172.21.15.32:0/2920217235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a40cc2e0 con 0x7f93a4070f10 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.768+0000 7f93a2a00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 0x7f93a41e9450 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f9394007e40 tx=0x7f939400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.768+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394014620 con 0x7f93a4070f10 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.768+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394019070 con 0x7f93a4070f10 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.768+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394014a00 con 0x7f93a4070f10 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.768+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a412ff10 con 0x7f93a4070f10 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.768+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a41301b0 con 0x7f93a4070f10 2024-03-06T07:36:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.769+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a41308c0 con 0x7f93a4070f10 2024-03-06T07:36:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.770+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9394014ba0 con 0x7f93a4070f10 2024-03-06T07:36:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.771+0000 7f939aa00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.771+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f939408abd0 con 0x7f93a4070f10 2024-03-06T07:36:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.771+0000 7f93a2000640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.772+0000 7f93a2000640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f938406b4d0 0x7f938406d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f93a4133060 tx=0x7f93900079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:12.775+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9394059920 con 0x7f93a4070f10 2024-03-06T07:36:13.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.017+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a40ccde0 con 0x7f93a4070f10 2024-03-06T07:36:13.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.018+0000 7f939aa00640 1 -- 172.21.15.32:0/2920217235 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f9394059220 con 0x7f93a4070f10 2024-03-06T07:36:13.020 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:13.020 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:13.020 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:13.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f938406b4d0 msgr2=0x7f938406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f938406b4d0 0x7f938406d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f93a4133060 tx=0x7f93900079b0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 msgr2=0x7f93a41e9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 0x7f93a41e9450 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f9394007e40 tx=0x7f939400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 shutdown_connections 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a4145300 0x7f93a40cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93a40cd060 0x7f93a40cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 --2- 172.21.15.32:0/2920217235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93a4070f10 0x7f93a41e9450 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.021+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 >> 172.21.15.32:0/2920217235 conn(0x7f93a406b3d0 msgr2=0x7f93a4072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.022+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 shutdown_connections 2024-03-06T07:36:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.022+0000 7f93a3e00640 1 -- 172.21.15.32:0/2920217235 wait complete. 2024-03-06T07:36:13.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:36:13.265 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:36:13.265 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:36:13.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:13.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.477+0000 7fc370c00640 1 Processor -- start 2024-03-06T07:36:13.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.478+0000 7fc370c00640 1 -- start start 2024-03-06T07:36:13.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.479+0000 7fc370c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc36c0afdf0 0x7fc36c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.479+0000 7fc370c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c06bec0 con 0x7fc36c06afb0 2024-03-06T07:36:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.479+0000 7fc370c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c0b0670 con 0x7fc36c15e200 2024-03-06T07:36:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.479+0000 7fc370c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c0ae380 con 0x7fc36c0afdf0 2024-03-06T07:36:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.479+0000 7fc36a800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc36c06afb0 0x7fc36c06dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54682/0 (socket says 172.21.15.32:54682) 2024-03-06T07:36:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.479+0000 7fc36a800640 1 -- 172.21.15.32:0/3973449062 learned_addr learned my addr 172.21.15.32:0/3973449062 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc369400640 1 --2- 172.21.15.32:0/3973449062 >> v2:172.21.15.39:3300/0 conn(0x7fc36c0afdf0 0x7fc36c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130000735 0 0) 0x7fc36c06bec0 con 0x7fc36c06afb0 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc350003520 con 0x7fc36c06afb0 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289918820 0 0) 0x7fc350003520 con 0x7fc36c06afb0 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 >> v2:172.21.15.39:3300/0 conn(0x7fc36c0afdf0 msgr2=0x7fc36c0b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 --2- 172.21.15.32:0/3973449062 >> v2:172.21.15.39:3300/0 conn(0x7fc36c0afdf0 0x7fc36c0b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 >> v1:172.21.15.39:6789/0 conn(0x7fc36c15e200 legacy=0x7fc36c0af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0ae660 con 0x7fc36c06afb0 2024-03-06T07:36:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.480+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fc364001670 con 0x7fc36c06afb0 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 >> v1:172.21.15.32:6789/0 conn(0x7fc36c06afb0 legacy=0x7fc36c06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc350003a20 0x7fc350003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc350004800 0x7fc350004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc3500051c0 0x7fc35000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0ae380 con 0x7fc350004800 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35000c0f0 con 0x7fc350003a20 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35000c2f0 con 0x7fc3500051c0 2024-03-06T07:36:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369400640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc350004800 0x7fc350004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369e00640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc350003a20 0x7fc350003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc36a800640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc3500051c0 0x7fc35000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369400640 1 -- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc3500051c0 msgr2=0x7fc35000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369400640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc3500051c0 0x7fc35000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369400640 1 -- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc350003a20 msgr2=0x7fc350003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369400640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc350003a20 0x7fc350003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc369400640 1 -- 172.21.15.32:0/3973449062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350003520 con 0x7fc350004800 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.481+0000 7fc36a800640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc3500051c0 0x7fc35000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.482+0000 7fc36a800640 1 -- 172.21.15.32:0/3973449062 reap_dead start 2024-03-06T07:36:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.483+0000 7fc369400640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc350004800 0x7fc350004c80 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc354009f90 tx=0x7fc350005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d87beb45ffc51b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354039070 con 0x7fc350004800 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35402cbf0 con 0x7fc350004800 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc368a00640 1 -- 172.21.15.32:0/3973449062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354009c10 con 0x7fc350004800 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 -- 172.21.15.32:0/3973449062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc350004800 msgr2=0x7fc350004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc350004800 0x7fc350004c80 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc354009f90 tx=0x7fc350005830 comp rx=0 tx=0).stop 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 -- 172.21.15.32:0/3973449062 shutdown_connections 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 --2- 172.21.15.32:0/3973449062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc350004800 0x7fc350004c80 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 -- 172.21.15.32:0/3973449062 >> 172.21.15.32:0/3973449062 conn(0x7fc36c0b2ae0 msgr2=0x7fc36c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 -- 172.21.15.32:0/3973449062 shutdown_connections 2024-03-06T07:36:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.484+0000 7fc370c00640 1 -- 172.21.15.32:0/3973449062 wait complete. 2024-03-06T07:36:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.486+0000 7fc370c00640 1 Processor -- start 2024-03-06T07:36:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.486+0000 7fc370c00640 1 -- start start 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc370c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc350004800 0x7fc36c145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc370c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 0x7fc36c141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc370c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc36c142880 0x7fc36c142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc370c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0b3de0 con 0x7fc36c155c50 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc370c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0aec20 con 0x7fc36c142880 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc370c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0b3f60 con 0x7fc350004800 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc369400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 0x7fc36c141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc369400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 0x7fc36c141f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37056/0 (socket says 172.21.15.32:37056) 2024-03-06T07:36:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.487+0000 7fc369400640 1 -- 172.21.15.32:0/969044678 learned_addr learned my addr 172.21.15.32:0/969044678 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc36a800640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc36c142880 0x7fc36c142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369e00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc350004800 0x7fc36c145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369400640 1 -- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc350004800 msgr2=0x7fc36c145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369400640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc350004800 0x7fc36c145400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369400640 1 -- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc36c142880 msgr2=0x7fc36c142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369400640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc36c142880 0x7fc36c142d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369400640 1 -- 172.21.15.32:0/969044678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c15bc80 con 0x7fc36c155c50 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.488+0000 7fc369400640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 0x7fc36c141f80 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc35400e230 tx=0x7fc354009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.489+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354042020 con 0x7fc36c155c50 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.489+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35403a040 con 0x7fc36c155c50 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.489+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354039070 con 0x7fc36c155c50 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.489+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c15bf60 con 0x7fc36c155c50 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.489+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c15c520 con 0x7fc36c155c50 2024-03-06T07:36:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.490+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c04f3a0 con 0x7fc36c155c50 2024-03-06T07:36:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.491+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc354045020 con 0x7fc36c155c50 2024-03-06T07:36:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.493+0000 7fc362000640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc34806b4d0 0x7fc34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.493+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc3540ac5a0 con 0x7fc36c155c50 2024-03-06T07:36:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.493+0000 7fc369e00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc34806b4d0 0x7fc34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.494+0000 7fc369e00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc34806b4d0 0x7fc34806d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007fd0 tx=0x7fc35c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:13.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.496+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc35404c030 con 0x7fc36c155c50 2024-03-06T07:36:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.736+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc36c0d0020 con 0x7fc36c155c50 2024-03-06T07:36:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.737+0000 7fc362000640 1 -- 172.21.15.32:0/969044678 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fc35407b390 con 0x7fc36c155c50 2024-03-06T07:36:13.738 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:13.738 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc34806b4d0 msgr2=0x7fc34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc34806b4d0 0x7fc34806d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007fd0 tx=0x7fc35c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 msgr2=0x7fc36c141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 0x7fc36c141f80 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc35400e230 tx=0x7fc354009750 comp rx=0 tx=0).stop 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 shutdown_connections 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc34806b4d0 0x7fc34806d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.739+0000 7fc370c00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc36c142880 0x7fc36c142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.740+0000 7fc370c00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc36c155c50 0x7fc36c141f80 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.740+0000 7fc370c00640 1 --2- 172.21.15.32:0/969044678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc350004800 0x7fc36c145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.740+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 >> 172.21.15.32:0/969044678 conn(0x7fc36c0b2ae0 msgr2=0x7fc36c1609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.740+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 shutdown_connections 2024-03-06T07:36:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:13.740+0000 7fc370c00640 1 -- 172.21.15.32:0/969044678 wait complete. 2024-03-06T07:36:13.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:36:14.014 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:36:14.014 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:36:14.014 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:36:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.125+0000 7f2f54d74f40 1 Processor -- start 2024-03-06T07:36:14.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.125+0000 7f2f54d74f40 1 -- start start 2024-03-06T07:36:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f54d74f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb6232e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f54d74f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb62d7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f54d74f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f54d74f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb6200a50 con 0x556fb61ff4b0 2024-03-06T07:36:14.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f54d74f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb61a7fd0 con 0x556fb61ff0d0 2024-03-06T07:36:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f54d74f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fb61ffdd0 con 0x556fb624d940 2024-03-06T07:36:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f51600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb62d7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f51600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb62d7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37068/0 (socket says 172.21.15.32:37068) 2024-03-06T07:36:14.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f51600640 1 -- 172.21.15.32:0/3616920147 learned_addr learned my addr 172.21.15.32:0/3616920147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:14.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.126+0000 7f2f52a00640 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f51600640 1 -- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 msgr2=0x556fb62c0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:14.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f51600640 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62c0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f52000640 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb6232e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f51600640 1 -- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 msgr2=0x556fb6232e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f51600640 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb6232e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f51600640 1 -- 172.21.15.32:0/3616920147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fb6233930 con 0x556fb61ff4b0 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f51600640 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb62d7fd0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000a770 tx=0x7f2f4000a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86234b3d55b64bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f4be00640 1 -- 172.21.15.32:0/3616920147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40019070 con 0x556fb61ff4b0 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f4be00640 1 -- 172.21.15.32:0/3616920147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40007600 con 0x556fb61ff4b0 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.127+0000 7f2f4be00640 1 -- 172.21.15.32:0/3616920147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f400079d0 con 0x556fb61ff4b0 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 -- 172.21.15.32:0/3616920147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 msgr2=0x556fb62d7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb62d7fd0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000a770 tx=0x7f2f4000a860 comp rx=0 tx=0).stop 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 -- 172.21.15.32:0/3616920147 shutdown_connections 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62c0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb62d7fd0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/3616920147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb6232e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 -- 172.21.15.32:0/3616920147 >> 172.21.15.32:0/3616920147 conn(0x556fb6230c10 msgr2=0x556fb62342f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 -- 172.21.15.32:0/3616920147 shutdown_connections 2024-03-06T07:36:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.128+0000 7f2f54d74f40 1 -- 172.21.15.32:0/3616920147 wait complete. 2024-03-06T07:36:14.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.129+0000 7f2f54d74f40 1 Processor -- start 2024-03-06T07:36:14.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.129+0000 7f2f54d74f40 1 -- start start 2024-03-06T07:36:14.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f54d74f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb62d1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f54d74f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb6231d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f54d74f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62d4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f54d74f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb6233e70 con 0x556fb61ff4b0 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f54d74f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb6233bd0 con 0x556fb61ff0d0 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f54d74f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fb6232960 con 0x556fb624d940 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f51600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb6231d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f51600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb6231d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37076/0 (socket says 172.21.15.32:37076) 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.130+0000 7f2f51600640 1 -- 172.21.15.32:0/1130731857 learned_addr learned my addr 172.21.15.32:0/1130731857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f52000640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb62d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f52a00640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62d4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f51600640 1 -- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 msgr2=0x556fb62d4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f51600640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62d4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f51600640 1 -- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 msgr2=0x556fb62d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f51600640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb62d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f51600640 1 -- 172.21.15.32:0/1130731857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fb62da290 con 0x556fb61ff4b0 2024-03-06T07:36:14.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f52000640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556fb61ff0d0 0x556fb62d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f52a00640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x556fb624d940 0x556fb62d4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.131+0000 7f2f51600640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb6231d10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f2f40017040 tx=0x7f2f40003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.132+0000 7f2f4aa00640 1 -- 172.21.15.32:0/1130731857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40019040 con 0x556fb61ff4b0 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.132+0000 7f2f4aa00640 1 -- 172.21.15.32:0/1130731857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40022070 con 0x556fb61ff4b0 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.132+0000 7f2f4aa00640 1 -- 172.21.15.32:0/1130731857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40014770 con 0x556fb61ff4b0 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.132+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556fb62ba220 con 0x556fb61ff4b0 2024-03-06T07:36:14.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.132+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fb62da530 con 0x556fb61ff4b0 2024-03-06T07:36:14.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.134+0000 7f2f4aa00640 1 -- 172.21.15.32:0/1130731857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f40014910 con 0x556fb61ff4b0 2024-03-06T07:36:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.135+0000 7f2f4aa00640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f34069eb0 0x7f2f3406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.135+0000 7f2f4aa00640 1 -- 172.21.15.32:0/1130731857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2f40089db0 con 0x556fb61ff4b0 2024-03-06T07:36:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.135+0000 7f2f52000640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f34069eb0 0x7f2f3406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.135+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x556fb62d0880 0x556fb62d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.135+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x556fb62daa50 con 0x556fb62d0880 2024-03-06T07:36:14.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.135+0000 7f2f52a00640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x556fb62d0880 0x556fb62d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.136+0000 7f2f52000640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f34069eb0 0x7f2f3406c370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x556fb6231270 tx=0x7f2f44002450 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.136+0000 7f2f52a00640 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x556fb62d0880 0x556fb62d0c60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f52a00640 1 -- 172.21.15.32:0/1130731857 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f2f4c05a040 con 0x556fb62d0880 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x556fb62d0880 msgr2=0x556fb62d0c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x556fb62d0880 0x556fb62d0c60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f34069eb0 msgr2=0x7f2f3406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2f34069eb0 0x7f2f3406c370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x556fb6231270 tx=0x7f2f44002450 comp rx=0 tx=0).stop 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 msgr2=0x556fb6231d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 --2- 172.21.15.32:0/1130731857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556fb61ff4b0 0x556fb6231d10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f2f40017040 tx=0x7f2f40003020 comp rx=0 tx=0).stop 2024-03-06T07:36:14.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f52a00640 1 -- 172.21.15.32:0/1130731857 reap_dead start 2024-03-06T07:36:14.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 shutdown_connections 2024-03-06T07:36:14.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 >> 172.21.15.32:0/1130731857 conn(0x556fb6230c10 msgr2=0x556fb62341e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:14.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 shutdown_connections 2024-03-06T07:36:14.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.137+0000 7f2f54d74f40 1 -- 172.21.15.32:0/1130731857 wait complete. 2024-03-06T07:36:14.143 INFO:teuthology.orchestra.run.smithi032.stdout:W�file_cdir_2! 2024-03-06T07:36:14.143 DEBUG:teuthology.orchestra.run.smithi032:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1099511627777, 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout: "ancestors": [ 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1099511628278, 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "file_c", 2024-03-06T07:36:14.233 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 17 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1, 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "dir_2", 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 33 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 17, 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: "old_pools": [] 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:36:14.234 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:14.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:14.238 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:36:14.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:14.291 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:36:14.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.544+0000 7f4a4ec00640 1 Processor -- start 2024-03-06T07:36:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.544+0000 7f4a4ec00640 1 -- start start 2024-03-06T07:36:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a4ec00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a480af1a0 0x7f4a4815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a4ec00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a480b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a4ec00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4815cc40 con 0x7f4a480cdc20 2024-03-06T07:36:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a4ec00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4815cf20 con 0x7f4a480af1a0 2024-03-06T07:36:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a4ec00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4815d200 con 0x7f4a480af580 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a47e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a480af1a0 0x7f4a4815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a47e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a480af1a0 0x7f4a4815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37090/0 (socket says 172.21.15.32:37090) 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.546+0000 7f4a47e00640 1 -- 172.21.15.32:0/8654313 learned_addr learned my addr 172.21.15.32:0/8654313 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.547+0000 7f4a3fe00640 1 --2- 172.21.15.32:0/8654313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a480b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.547+0000 7f4a47e00640 1 -- 172.21.15.32:0/8654313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 msgr2=0x7f4a480b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.547+0000 7f4a47e00640 1 --2- 172.21.15.32:0/8654313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a480b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.547+0000 7f4a47e00640 1 -- 172.21.15.32:0/8654313 >> v1:172.21.15.39:6789/0 conn(0x7f4a480cdc20 legacy=0x7f4a480b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.547+0000 7f4a47e00640 1 -- 172.21.15.32:0/8654313 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4815d4e0 con 0x7f4a480af1a0 2024-03-06T07:36:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.548+0000 7f4a47e00640 1 --2- 172.21.15.32:0/8654313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480af1a0 0x7f4a4815c600 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009870 tx=0x7f4a3802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f4fc0137c978c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.549+0000 7f4a47400640 1 -- 172.21.15.32:0/8654313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380317c0 con 0x7f4a480af1a0 2024-03-06T07:36:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.549+0000 7f4a47400640 1 -- 172.21.15.32:0/8654313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38037070 con 0x7f4a480af1a0 2024-03-06T07:36:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.549+0000 7f4a47400640 1 -- 172.21.15.32:0/8654313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38031b80 con 0x7f4a480af1a0 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 -- 172.21.15.32:0/8654313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480af1a0 msgr2=0x7f4a4815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/8654313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480af1a0 0x7f4a4815c600 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009870 tx=0x7f4a3802b860 comp rx=0 tx=0).stop 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 -- 172.21.15.32:0/8654313 shutdown_connections 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/8654313 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a480b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/8654313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480af1a0 0x7f4a4815c600 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 -- 172.21.15.32:0/8654313 >> 172.21.15.32:0/8654313 conn(0x7f4a4806afb0 msgr2=0x7f4a4806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 -- 172.21.15.32:0/8654313 shutdown_connections 2024-03-06T07:36:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.550+0000 7f4a4ec00640 1 -- 172.21.15.32:0/8654313 wait complete. 2024-03-06T07:36:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.552+0000 7f4a4ec00640 1 Processor -- start 2024-03-06T07:36:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.552+0000 7f4a4ec00640 1 -- start start 2024-03-06T07:36:14.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a480af1a0 0x7f4a48159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4ec00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a48146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 0x7f4a48146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4ec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a480b3660 con 0x7f4a480cdc20 2024-03-06T07:36:14.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a480b34e0 con 0x7f4a480af1a0 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4ec00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a480b37e0 con 0x7f4a480af580 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 0x7f4a48146750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 0x7f4a48146750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37094/0 (socket says 172.21.15.32:37094) 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.554+0000 7f4a4cc00640 1 -- 172.21.15.32:0/2592305569 learned_addr learned my addr 172.21.15.32:0/2592305569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a47e00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a480af1a0 0x7f4a48159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a3fe00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a48146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a4cc00640 1 -- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 msgr2=0x7f4a48146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a4cc00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a48146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a4cc00640 1 -- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a480af1a0 msgr2=0x7f4a48159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a4cc00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a480af1a0 0x7f4a48159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a4cc00640 1 -- 172.21.15.32:0/2592305569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4809ab40 con 0x7f4a480cdc20 2024-03-06T07:36:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.555+0000 7f4a4cc00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 0x7f4a48146750 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000b970 tx=0x7f4a4000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.556+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40019070 con 0x7f4a480cdc20 2024-03-06T07:36:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.556+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4000cb40 con 0x7f4a480cdc20 2024-03-06T07:36:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.556+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40014440 con 0x7f4a480cdc20 2024-03-06T07:36:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.556+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4809ae20 con 0x7f4a480cdc20 2024-03-06T07:36:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.556+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4809b3e0 con 0x7f4a480cdc20 2024-03-06T07:36:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.557+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4804f3a0 con 0x7f4a480cdc20 2024-03-06T07:36:14.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.558+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a4001c070 con 0x7f4a480cdc20 2024-03-06T07:36:14.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.559+0000 7f4a44c00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a2406b4d0 0x7f4a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.559+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4a40089eb0 con 0x7f4a480cdc20 2024-03-06T07:36:14.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.559+0000 7f4a47e00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a2406b4d0 0x7f4a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.560+0000 7f4a47e00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a2406b4d0 0x7f4a2406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4a38036000 tx=0x7f4a3803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.563+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a40058d30 con 0x7f4a480cdc20 2024-03-06T07:36:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.805+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a480cd6c0 con 0x7f4a480cdc20 2024-03-06T07:36:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.805+0000 7f4a44c00640 1 -- 172.21.15.32:0/2592305569 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f4a40058630 con 0x7f4a480cdc20 2024-03-06T07:36:14.807 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:14.807 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:14.807 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a2406b4d0 msgr2=0x7f4a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a2406b4d0 0x7f4a2406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4a38036000 tx=0x7f4a3803d040 comp rx=0 tx=0).stop 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 msgr2=0x7f4a48146750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 0x7f4a48146750 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000b970 tx=0x7f4a4000be30 comp rx=0 tx=0).stop 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 shutdown_connections 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a2406b4d0 0x7f4a2406d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a480cdc20 0x7f4a48146750 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a480af580 0x7f4a48146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 --2- 172.21.15.32:0/2592305569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a480af1a0 0x7f4a48159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 >> 172.21.15.32:0/2592305569 conn(0x7f4a4806afb0 msgr2=0x7f4a4815c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.808+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 shutdown_connections 2024-03-06T07:36:14.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:14.809+0000 7f4a4ec00640 1 -- 172.21.15.32:0/2592305569 wait complete. 2024-03-06T07:36:14.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:36:15.071 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:36:15.072 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:36:15.072 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:36:20.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:20.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.303+0000 7f967ee00640 1 Processor -- start 2024-03-06T07:36:20.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.303+0000 7f967ee00640 1 -- start start 2024-03-06T07:36:20.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f967ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9678161670 0x7f9678070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f967ee00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f96780ae0a0 0x7f96780711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f967ee00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96780ce9a0 con 0x7f967806e240 2024-03-06T07:36:20.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f967ee00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96780730d0 con 0x7f96780ae0a0 2024-03-06T07:36:20.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f967ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96780cec80 con 0x7f9678161670 2024-03-06T07:36:20.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f967ce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f96780ae0a0 0x7f96780711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f966fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f967806e240 0x7f967806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37208/0 (socket says 172.21.15.32:37208) 2024-03-06T07:36:20.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.305+0000 7f966fe00640 1 -- 172.21.15.32:0/2290559681 learned_addr learned my addr 172.21.15.32:0/2290559681 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:20.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f9677e00640 1 --2- 172.21.15.32:0/2290559681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9678161670 0x7f9678070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f9677400640 1 -- 172.21.15.32:0/2290559681 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256974092 0 0) 0x7f96780ce9a0 con 0x7f967806e240 2024-03-06T07:36:20.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f9677400640 1 -- 172.21.15.32:0/2290559681 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c003520 con 0x7f967806e240 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f967ce00640 1 -- 172.21.15.32:0/2290559681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9678161670 msgr2=0x7f9678070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f967ce00640 1 --2- 172.21.15.32:0/2290559681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9678161670 0x7f9678070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f967ce00640 1 -- 172.21.15.32:0/2290559681 >> v1:172.21.15.32:6789/0 conn(0x7f967806e240 legacy=0x7f967806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.306+0000 7f967ce00640 1 -- 172.21.15.32:0/2290559681 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780cef60 con 0x7f96780ae0a0 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.307+0000 7f967ce00640 1 --2- 172.21.15.32:0/2290559681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96780ae0a0 0x7f96780711b0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f9670009d90 tx=0x7f967002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb46be99f615555 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.307+0000 7f9677400640 1 -- 172.21.15.32:0/2290559681 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670039070 con 0x7f96780ae0a0 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.307+0000 7f9677400640 1 -- 172.21.15.32:0/2290559681 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967002cc50 con 0x7f96780ae0a0 2024-03-06T07:36:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.307+0000 7f9677400640 1 -- 172.21.15.32:0/2290559681 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670034440 con 0x7f96780ae0a0 2024-03-06T07:36:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 -- 172.21.15.32:0/2290559681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96780ae0a0 msgr2=0x7f96780711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 --2- 172.21.15.32:0/2290559681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96780ae0a0 0x7f96780711b0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f9670009d90 tx=0x7f967002bea0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 -- 172.21.15.32:0/2290559681 shutdown_connections 2024-03-06T07:36:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 --2- 172.21.15.32:0/2290559681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96780ae0a0 0x7f96780711b0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 --2- 172.21.15.32:0/2290559681 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9678161670 0x7f9678070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 -- 172.21.15.32:0/2290559681 >> 172.21.15.32:0/2290559681 conn(0x7f967806afb0 msgr2=0x7f967806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 -- 172.21.15.32:0/2290559681 shutdown_connections 2024-03-06T07:36:20.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.308+0000 7f967ee00640 1 -- 172.21.15.32:0/2290559681 wait complete. 2024-03-06T07:36:20.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.310+0000 7f967ee00640 1 Processor -- start 2024-03-06T07:36:20.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.310+0000 7f967ee00640 1 -- start start 2024-03-06T07:36:20.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f967806e240 0x7f96781f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96780ae0a0 0x7f96780ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 0x7f96780b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96780ce600 con 0x7f9678161670 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96780cf520 con 0x7f96780ae0a0 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96780ce780 con 0x7f967806e240 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.311+0000 7f967ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 0x7f96780b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 0x7f96780b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60238/0 (socket says 172.21.15.32:60238) 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 -- 172.21.15.32:0/2556542816 learned_addr learned my addr 172.21.15.32:0/2556542816 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f966fe00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96780ae0a0 0x7f96780ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f9677e00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f967806e240 0x7f96781f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 -- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f967806e240 msgr2=0x7f96781f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f967806e240 0x7f96781f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 -- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96780ae0a0 msgr2=0x7f96780ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96780ae0a0 0x7f96780ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 -- 172.21.15.32:0/2556542816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780cb130 con 0x7f9678161670 2024-03-06T07:36:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.312+0000 7f967ce00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 0x7f96780b0170 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f9670009eb0 tx=0x7f9670004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.313+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670039050 con 0x7f9678161670 2024-03-06T07:36:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.313+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670045070 con 0x7f9678161670 2024-03-06T07:36:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.313+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96780cb410 con 0x7f9678161670 2024-03-06T07:36:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.313+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967003e610 con 0x7f9678161670 2024-03-06T07:36:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.313+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96780cb9d0 con 0x7f9678161670 2024-03-06T07:36:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.314+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9678069510 con 0x7f9678161670 2024-03-06T07:36:20.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.316+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f967003e7b0 con 0x7f9678161670 2024-03-06T07:36:20.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.317+0000 7f9674c00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.317+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f96700acf40 con 0x7f9678161670 2024-03-06T07:36:20.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.317+0000 7f9677e00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.318+0000 7f9677e00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f965406b4d0 0x7f965406d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9668006fd0 tx=0x7f9668008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.322+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f967007be40 con 0x7f9678161670 2024-03-06T07:36:20.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:20.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:20.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.566+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96780b52d0 con 0x7f9678161670 2024-03-06T07:36:20.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.567+0000 7f9674c00640 1 -- 172.21.15.32:0/2556542816 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f967007b740 con 0x7f9678161670 2024-03-06T07:36:20.568 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:20.568 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:20.569 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:20.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f965406b4d0 msgr2=0x7f965406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f965406b4d0 0x7f965406d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9668006fd0 tx=0x7f9668008040 comp rx=0 tx=0).stop 2024-03-06T07:36:20.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 msgr2=0x7f96780b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 0x7f96780b0170 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f9670009eb0 tx=0x7f9670004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:20.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 shutdown_connections 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f965406b4d0 0x7f965406d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9678161670 0x7f96780b0170 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96780ae0a0 0x7f96780ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 --2- 172.21.15.32:0/2556542816 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f967806e240 0x7f96781f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 >> 172.21.15.32:0/2556542816 conn(0x7f967806afb0 msgr2=0x7f96780b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 shutdown_connections 2024-03-06T07:36:20.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:20.570+0000 7f967ee00640 1 -- 172.21.15.32:0/2556542816 wait complete. 2024-03-06T07:36:20.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:36:20.857 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:36:20.857 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:36:20.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.080+0000 7f4a2be00640 1 Processor -- start 2024-03-06T07:36:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.080+0000 7f4a2be00640 1 -- start start 2024-03-06T07:36:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a2be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a240d2340 0x7f4a240d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a2be00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2406c920 con 0x7f4a2415e080 2024-03-06T07:36:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a2be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2406cc00 con 0x7f4a2406de20 2024-03-06T07:36:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a2be00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2406d530 con 0x7f4a240d2340 2024-03-06T07:36:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a29e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a240d2340 0x7f4a240d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a29400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4a2415e080 0x7f4a2406c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37224/0 (socket says 172.21.15.32:37224) 2024-03-06T07:36:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.082+0000 7f4a29400640 1 -- 172.21.15.32:0/76072457 learned_addr learned my addr 172.21.15.32:0/76072457 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.083+0000 7f4a29e00640 1 -- 172.21.15.32:0/76072457 >> v1:172.21.15.39:6789/0 conn(0x7f4a2406de20 legacy=0x7f4a240d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.083+0000 7f4a29e00640 1 -- 172.21.15.32:0/76072457 >> v1:172.21.15.32:6789/0 conn(0x7f4a2415e080 legacy=0x7f4a2406c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.083+0000 7f4a29e00640 1 -- 172.21.15.32:0/76072457 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2406cee0 con 0x7f4a240d2340 2024-03-06T07:36:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.083+0000 7f4a23e00640 1 -- 172.21.15.32:0/76072457 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117177745 0 0) 0x7f4a2406c920 con 0x7f4a2415e080 2024-03-06T07:36:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.084+0000 7f4a29e00640 1 --2- 172.21.15.32:0/76072457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a240d0d60 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009d90 tx=0x7f4a1c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab70fd8860a20fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.084+0000 7f4a23e00640 1 -- 172.21.15.32:0/76072457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c039070 con 0x7f4a240d2340 2024-03-06T07:36:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.084+0000 7f4a23e00640 1 -- 172.21.15.32:0/76072457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c02cc60 con 0x7f4a240d2340 2024-03-06T07:36:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.084+0000 7f4a23e00640 1 -- 172.21.15.32:0/76072457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c034660 con 0x7f4a240d2340 2024-03-06T07:36:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 -- 172.21.15.32:0/76072457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 msgr2=0x7f4a240d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 --2- 172.21.15.32:0/76072457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a240d0d60 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009d90 tx=0x7f4a1c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 -- 172.21.15.32:0/76072457 shutdown_connections 2024-03-06T07:36:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 --2- 172.21.15.32:0/76072457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a240d0d60 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 -- 172.21.15.32:0/76072457 >> 172.21.15.32:0/76072457 conn(0x7f4a2409aa10 msgr2=0x7f4a2409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 -- 172.21.15.32:0/76072457 shutdown_connections 2024-03-06T07:36:21.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.085+0000 7f4a2be00640 1 -- 172.21.15.32:0/76072457 wait complete. 2024-03-06T07:36:21.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.086+0000 7f4a2be00640 1 Processor -- start 2024-03-06T07:36:21.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.087+0000 7f4a2be00640 1 -- start start 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.087+0000 7f4a2be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2406de20 0x7f4a240b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.087+0000 7f4a2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a2414e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a2be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a2415e080 0x7f4a2415a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a2be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a24168c70 con 0x7f4a240d2340 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a2be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2415e460 con 0x7f4a2406de20 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a2be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a24168df0 con 0x7f4a2415e080 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a2414e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a2414e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60262/0 (socket says 172.21.15.32:60262) 2024-03-06T07:36:21.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 -- 172.21.15.32:0/2633413675 learned_addr learned my addr 172.21.15.32:0/2633413675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:21.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a29400640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2406de20 0x7f4a240b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a29e00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a2415e080 0x7f4a2415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 -- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a2415e080 msgr2=0x7f4a2415a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a2415e080 0x7f4a2415a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 -- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2406de20 msgr2=0x7f4a240b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2406de20 0x7f4a240b7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.088+0000 7f4a28a00640 1 -- 172.21.15.32:0/2633413675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240b5ff0 con 0x7f4a240d2340 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.089+0000 7f4a28a00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a2414e1b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000c480 tx=0x7f4a1000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.089+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a100052d0 con 0x7f4a240d2340 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.089+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10005470 con 0x7f4a240d2340 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.089+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a240b62d0 con 0x7f4a240d2340 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.089+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10012870 con 0x7f4a240d2340 2024-03-06T07:36:21.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.089+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a240b6890 con 0x7f4a240d2340 2024-03-06T07:36:21.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.091+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2404f3a0 con 0x7f4a240d2340 2024-03-06T07:36:21.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.091+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a10012a10 con 0x7f4a240d2340 2024-03-06T07:36:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.092+0000 7f4a21600640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a0406b500 0x7f4a0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.092+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4a10089f20 con 0x7f4a240d2340 2024-03-06T07:36:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.093+0000 7f4a29400640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a0406b500 0x7f4a0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.093+0000 7f4a29400640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a0406b500 0x7f4a0406d9c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4a18000f60 tx=0x7f4a18007d20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.097+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a10058d00 con 0x7f4a240d2340 2024-03-06T07:36:21.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.385+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a24155380 con 0x7f4a240d2340 2024-03-06T07:36:21.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.386+0000 7f4a21600640 1 -- 172.21.15.32:0/2633413675 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f4a10058600 con 0x7f4a240d2340 2024-03-06T07:36:21.388 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:21.388 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:21.388 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a0406b500 msgr2=0x7f4a0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a0406b500 0x7f4a0406d9c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4a18000f60 tx=0x7f4a18007d20 comp rx=0 tx=0).stop 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 msgr2=0x7f4a2414e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a2414e1b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000c480 tx=0x7f4a1000c940 comp rx=0 tx=0).stop 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 shutdown_connections 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4a0406b500 0x7f4a0406d9c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4a2415e080 0x7f4a2415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a240d2340 0x7f4a2414e1b0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 --2- 172.21.15.32:0/2633413675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2406de20 0x7f4a240b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 >> 172.21.15.32:0/2633413675 conn(0x7f4a2409aa10 msgr2=0x7f4a240b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 shutdown_connections 2024-03-06T07:36:21.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.388+0000 7f4a2be00640 1 -- 172.21.15.32:0/2633413675 wait complete. 2024-03-06T07:36:21.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:36:21.669 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:36:21.670 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:36:21.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.877+0000 7f4e49a00640 1 Processor -- start 2024-03-06T07:36:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.877+0000 7f4e49a00640 1 -- start start 2024-03-06T07:36:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.879+0000 7f4e49a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e4415c920 0x7f4e44070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.879+0000 7f4e49a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440cd060 0x7f4e4406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.879+0000 7f4e49a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e440cdad0 con 0x7f4e440ce850 2024-03-06T07:36:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.879+0000 7f4e49a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e44071500 con 0x7f4e4415c920 2024-03-06T07:36:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.879+0000 7f4e49a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440cdd70 con 0x7f4e440cd060 2024-03-06T07:36:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.880+0000 7f4e43e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e4415c920 0x7f4e44070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.880+0000 7f4e49000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440cd060 0x7f4e4406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.880+0000 7f4e43e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e4415c920 0x7f4e44070a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:58594/0 (socket says 172.21.15.32:58594) 2024-03-06T07:36:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.880+0000 7f4e43e00640 1 -- 172.21.15.32:0/3533928577 learned_addr learned my addr 172.21.15.32:0/3533928577 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.881+0000 7f4e49000640 1 -- 172.21.15.32:0/3533928577 >> v2:172.21.15.39:3300/0 conn(0x7f4e4415c920 msgr2=0x7f4e44070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.881+0000 7f4e49000640 1 --2- 172.21.15.32:0/3533928577 >> v2:172.21.15.39:3300/0 conn(0x7f4e4415c920 0x7f4e44070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.881+0000 7f4e49000640 1 -- 172.21.15.32:0/3533928577 >> v1:172.21.15.39:6789/0 conn(0x7f4e440ce850 legacy=0x7f4e440cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.881+0000 7f4e49000640 1 -- 172.21.15.32:0/3533928577 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e440ce050 con 0x7f4e440cd060 2024-03-06T07:36:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.883+0000 7f4e49000640 1 --2- 172.21.15.32:0/3533928577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440cd060 0x7f4e4406cac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0097a0 tx=0x7f4e3c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6b508dc33e0d1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.883+0000 7f4e42a00640 1 -- 172.21.15.32:0/3533928577 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c035020 con 0x7f4e440cd060 2024-03-06T07:36:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.883+0000 7f4e42a00640 1 -- 172.21.15.32:0/3533928577 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c031810 con 0x7f4e440cd060 2024-03-06T07:36:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.883+0000 7f4e42a00640 1 -- 172.21.15.32:0/3533928577 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c031be0 con 0x7f4e440cd060 2024-03-06T07:36:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 -- 172.21.15.32:0/3533928577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440cd060 msgr2=0x7f4e4406cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 --2- 172.21.15.32:0/3533928577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440cd060 0x7f4e4406cac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0097a0 tx=0x7f4e3c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 -- 172.21.15.32:0/3533928577 shutdown_connections 2024-03-06T07:36:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 --2- 172.21.15.32:0/3533928577 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440cd060 0x7f4e4406cac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 --2- 172.21.15.32:0/3533928577 >> v2:172.21.15.39:3300/0 conn(0x7f4e4415c920 0x7f4e44070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 -- 172.21.15.32:0/3533928577 >> 172.21.15.32:0/3533928577 conn(0x7f4e4406b3d0 msgr2=0x7f4e4406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 -- 172.21.15.32:0/3533928577 shutdown_connections 2024-03-06T07:36:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.884+0000 7f4e49a00640 1 -- 172.21.15.32:0/3533928577 wait complete. 2024-03-06T07:36:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.886+0000 7f4e49a00640 1 Processor -- start 2024-03-06T07:36:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.886+0000 7f4e49a00640 1 -- start start 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.887+0000 7f4e49a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 0x7f4e440b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.887+0000 7f4e49a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440ce850 0x7f4e4415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.887+0000 7f4e49a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4415c920 0x7f4e44146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.887+0000 7f4e49a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44070f80 con 0x7f4e440cd060 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.887+0000 7f4e49a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44163790 con 0x7f4e4415c920 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.887+0000 7f4e49a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44071100 con 0x7f4e440ce850 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 0x7f4e440b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 0x7f4e440b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60264/0 (socket says 172.21.15.32:60264) 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 -- 172.21.15.32:0/1512344878 learned_addr learned my addr 172.21.15.32:0/1512344878 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43400640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440ce850 0x7f4e4415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e49000640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4415c920 0x7f4e44146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 -- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440ce850 msgr2=0x7f4e4415d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440ce850 0x7f4e4415d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 -- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4415c920 msgr2=0x7f4e44146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4415c920 0x7f4e44146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.888+0000 7f4e43e00640 1 -- 172.21.15.32:0/1512344878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44146fe0 con 0x7f4e440cd060 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.889+0000 7f4e43e00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 0x7f4e440b44a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009cf0 tx=0x7f4e3400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.889+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3400cda0 con 0x7f4e440cd060 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.889+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e34019070 con 0x7f4e440cd060 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.889+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340146a0 con 0x7f4e440cd060 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.889+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e440b0270 con 0x7f4e440cd060 2024-03-06T07:36:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.889+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e440b0830 con 0x7f4e440cd060 2024-03-06T07:36:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.891+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4404f330 con 0x7f4e440cd060 2024-03-06T07:36:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.891+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e34005d10 con 0x7f4e440cd060 2024-03-06T07:36:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.892+0000 7f4e3be00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e2406b500 0x7f4e2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.892+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4e3408a730 con 0x7f4e440cd060 2024-03-06T07:36:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.893+0000 7f4e43400640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e2406b500 0x7f4e2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.893+0000 7f4e43400640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e2406b500 0x7f4e2406d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4e441582d0 tx=0x7f4e30009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:21.897+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e34010040 con 0x7f4e440cd060 2024-03-06T07:36:22.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.160+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e440cf3a0 con 0x7f4e440cd060 2024-03-06T07:36:22.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.161+0000 7f4e3be00640 1 -- 172.21.15.32:0/1512344878 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f4e34059510 con 0x7f4e440cd060 2024-03-06T07:36:22.164 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:22.164 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.163+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e2406b500 msgr2=0x7f4e2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.163+0000 7f4e49a00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e2406b500 0x7f4e2406d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4e441582d0 tx=0x7f4e30009040 comp rx=0 tx=0).stop 2024-03-06T07:36:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.163+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 msgr2=0x7f4e440b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.163+0000 7f4e49a00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 0x7f4e440b44a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009cf0 tx=0x7f4e3400c490 comp rx=0 tx=0).stop 2024-03-06T07:36:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 shutdown_connections 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e2406b500 0x7f4e2406d9c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4415c920 0x7f4e44146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e440ce850 0x7f4e4415d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 --2- 172.21.15.32:0/1512344878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e440cd060 0x7f4e440b44a0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 >> 172.21.15.32:0/1512344878 conn(0x7f4e4406b3d0 msgr2=0x7f4e4406d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 shutdown_connections 2024-03-06T07:36:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.164+0000 7f4e49a00640 1 -- 172.21.15.32:0/1512344878 wait complete. 2024-03-06T07:36:22.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:36:22.424 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-03-06T07:36:22.424 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:36:22.424 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:36:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.520+0000 7f8f1a639f40 1 Processor -- start 2024-03-06T07:36:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.520+0000 7f8f1a639f40 1 -- start start 2024-03-06T07:36:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f1a639f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f756150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f1a639f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f6d2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f1a639f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f779910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f1a639f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837f647fd0 con 0x55837f75e7a0 2024-03-06T07:36:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f1a639f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837f64bf30 con 0x55837f6edcd0 2024-03-06T07:36:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f1a639f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55837f6d3a00 con 0x55837f6a1540 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f17a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f779910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f17a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f779910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60270/0 (socket says 172.21.15.32:60270) 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f17a00640 1 -- 172.21.15.32:0/2501423070 learned_addr learned my addr 172.21.15.32:0/2501423070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f17000640 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f756150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.521+0000 7f8f16600640 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f6d2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f17a00640 1 -- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 msgr2=0x55837f756150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f17a00640 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f756150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f17a00640 1 -- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 msgr2=0x55837f6d2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f17a00640 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f6d2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f17a00640 1 -- 172.21.15.32:0/2501423070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55837f6a1170 con 0x55837f75e7a0 2024-03-06T07:36:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f17a00640 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f779910 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f8f1001f390 tx=0x7f8f10051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5655121c7f2bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f15200640 1 -- 172.21.15.32:0/2501423070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10050b00 con 0x55837f75e7a0 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f15200640 1 -- 172.21.15.32:0/2501423070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10067070 con 0x55837f75e7a0 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.522+0000 7f8f15200640 1 -- 172.21.15.32:0/2501423070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f100623f0 con 0x55837f75e7a0 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 -- 172.21.15.32:0/2501423070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 msgr2=0x55837f779910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f779910 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f8f1001f390 tx=0x7f8f10051000 comp rx=0 tx=0).stop 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 -- 172.21.15.32:0/2501423070 shutdown_connections 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f779910 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f6d2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/2501423070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f756150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 -- 172.21.15.32:0/2501423070 >> 172.21.15.32:0/2501423070 conn(0x55837f6eea90 msgr2=0x55837f777f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 -- 172.21.15.32:0/2501423070 shutdown_connections 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.523+0000 7f8f1a639f40 1 -- 172.21.15.32:0/2501423070 wait complete. 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.524+0000 7f8f1a639f40 1 Processor -- start 2024-03-06T07:36:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.524+0000 7f8f1a639f40 1 -- start start 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f1a639f40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f771ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f1a639f40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f761320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f1a639f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f761860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f1a639f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837f77a930 con 0x55837f75e7a0 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f1a639f40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837f77a490 con 0x55837f6edcd0 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f1a639f40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x55837f75d830 con 0x55837f6a1540 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.525+0000 7f8f17a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f761860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f761860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60286/0 (socket says 172.21.15.32:60286) 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 -- 172.21.15.32:0/1313097402 learned_addr learned my addr 172.21.15.32:0/1313097402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17000640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f771ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f16600640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f761320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 -- 172.21.15.32:0/1313097402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 msgr2=0x55837f771ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x55837f6a1540 0x55837f771ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 -- 172.21.15.32:0/1313097402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 msgr2=0x55837f761320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55837f6edcd0 0x55837f761320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 -- 172.21.15.32:0/1313097402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55837f772480 con 0x55837f75e7a0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.526+0000 7f8f17a00640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f761860 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8f10064040 tx=0x7f8f10003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.527+0000 7f8f0f400640 1 -- 172.21.15.32:0/1313097402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10058800 con 0x55837f75e7a0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.527+0000 7f8f0f400640 1 -- 172.21.15.32:0/1313097402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f100589c0 con 0x55837f75e7a0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.527+0000 7f8f0f400640 1 -- 172.21.15.32:0/1313097402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10058db0 con 0x55837f75e7a0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.527+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55837f806b30 con 0x55837f75e7a0 2024-03-06T07:36:22.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.527+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55837f7727f0 con 0x55837f75e7a0 2024-03-06T07:36:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.530+0000 7f8f0f400640 1 -- 172.21.15.32:0/1313097402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f10073020 con 0x55837f75e7a0 2024-03-06T07:36:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.530+0000 7f8f0f400640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ef8069eb0 0x7f8ef806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.531+0000 7f8f0f400640 1 -- 172.21.15.32:0/1313097402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8f100ce980 con 0x55837f75e7a0 2024-03-06T07:36:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.531+0000 7f8f17000640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ef8069eb0 0x7f8ef806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.531+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55837f6eae40 0x55837f75fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.531+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x55837f760190 con 0x55837f6eae40 2024-03-06T07:36:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.531+0000 7f8f16600640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55837f6eae40 0x55837f75fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.531+0000 7f8f17000640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ef8069eb0 0x7f8ef806c370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8f08001ed0 tx=0x7f8f08001c60 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f16600640 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55837f6eae40 0x55837f75fc50 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f16600640 1 -- 172.21.15.32:0/1313097402 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f8f04010350 con 0x55837f6eae40 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55837f6eae40 msgr2=0x55837f75fc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x55837f6eae40 0x55837f75fc50 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ef8069eb0 msgr2=0x7f8ef806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8ef8069eb0 0x7f8ef806c370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8f08001ed0 tx=0x7f8f08001c60 comp rx=0 tx=0).stop 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 msgr2=0x55837f761860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.532+0000 7f8f1a639f40 1 --2- 172.21.15.32:0/1313097402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55837f75e7a0 0x55837f761860 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8f10064040 tx=0x7f8f10003330 comp rx=0 tx=0).stop 2024-03-06T07:36:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.533+0000 7f8f17a00640 1 -- 172.21.15.32:0/1313097402 reap_dead start 2024-03-06T07:36:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.533+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 shutdown_connections 2024-03-06T07:36:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.533+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 >> 172.21.15.32:0/1313097402 conn(0x55837f6eea90 msgr2=0x55837f779780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.533+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 shutdown_connections 2024-03-06T07:36:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.533+0000 7f8f1a639f40 1 -- 172.21.15.32:0/1313097402 wait complete. 2024-03-06T07:36:22.540 INFO:teuthology.orchestra.run.smithi032.stdout:W�file_ddir_2, 2024-03-06T07:36:22.540 DEBUG:teuthology.orchestra.run.smithi032:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:36:22.628 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:36:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1099511627777, 2024-03-06T07:36:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "ancestors": [ 2024-03-06T07:36:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1099511628278, 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "file_d", 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 27 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "dirino": 1, 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "dir_2", 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "version": 44 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 17, 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "old_pools": [] 2024-03-06T07:36:22.629 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:36:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:22.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:22.634 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:36:22.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.894+0000 7ff5b1600640 1 Processor -- start 2024-03-06T07:36:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.895+0000 7ff5b1600640 1 -- start start 2024-03-06T07:36:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.896+0000 7ff5b1600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5ac158840 0x7ff5ac161c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.896+0000 7ff5b1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff5ac162150 0x7ff5ac0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.896+0000 7ff5b1600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0cff30 con 0x7ff5ac158460 2024-03-06T07:36:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.896+0000 7ff5b1600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac1626d0 con 0x7ff5ac158840 2024-03-06T07:36:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.896+0000 7ff5b1600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac1629b0 con 0x7ff5ac162150 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5ac158840 0x7ff5ac161c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5ac158840 0x7ff5ac161c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60294/0 (socket says 172.21.15.32:60294) 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 -- 172.21.15.32:0/3621510577 learned_addr learned my addr 172.21.15.32:0/3621510577 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5b0c00640 1 --2- 172.21.15.32:0/3621510577 >> v2:172.21.15.39:3300/0 conn(0x7ff5ac162150 0x7ff5ac0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 -- 172.21.15.32:0/3621510577 >> v2:172.21.15.39:3300/0 conn(0x7ff5ac162150 msgr2=0x7ff5ac0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 --2- 172.21.15.32:0/3621510577 >> v2:172.21.15.39:3300/0 conn(0x7ff5ac162150 0x7ff5ac0b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 -- 172.21.15.32:0/3621510577 >> v1:172.21.15.39:6789/0 conn(0x7ff5ac158460 legacy=0x7ff5ac0d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.897+0000 7ff5ab400640 1 -- 172.21.15.32:0/3621510577 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0d01d0 con 0x7ff5ac158840 2024-03-06T07:36:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.898+0000 7ff5ab400640 1 --2- 172.21.15.32:0/3621510577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158840 0x7ff5ac161c10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0b6360 tx=0x7ff59802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30f25fd62d42694 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.899+0000 7ff5aaa00640 1 -- 172.21.15.32:0/3621510577 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598034070 con 0x7ff5ac158840 2024-03-06T07:36:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.899+0000 7ff5aaa00640 1 -- 172.21.15.32:0/3621510577 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff598030860 con 0x7ff5ac158840 2024-03-06T07:36:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.899+0000 7ff5aaa00640 1 -- 172.21.15.32:0/3621510577 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598030c30 con 0x7ff5ac158840 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 -- 172.21.15.32:0/3621510577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158840 msgr2=0x7ff5ac161c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3621510577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158840 0x7ff5ac161c10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0b6360 tx=0x7ff59802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 -- 172.21.15.32:0/3621510577 shutdown_connections 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3621510577 >> v2:172.21.15.39:3300/0 conn(0x7ff5ac162150 0x7ff5ac0b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3621510577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158840 0x7ff5ac161c10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 -- 172.21.15.32:0/3621510577 >> 172.21.15.32:0/3621510577 conn(0x7ff5ac0b5100 msgr2=0x7ff5ac0b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 -- 172.21.15.32:0/3621510577 shutdown_connections 2024-03-06T07:36:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.900+0000 7ff5b1600640 1 -- 172.21.15.32:0/3621510577 wait complete. 2024-03-06T07:36:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.902+0000 7ff5b1600640 1 Processor -- start 2024-03-06T07:36:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.902+0000 7ff5b1600640 1 -- start start 2024-03-06T07:36:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 0x7ff5ac0998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b1600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5ac158840 0x7ff5ac0ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ac162150 0x7ff5ac0ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b1600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0b1850 con 0x7ff5ac158460 2024-03-06T07:36:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac15e8c0 con 0x7ff5ac162150 2024-03-06T07:36:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b1600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0b19d0 con 0x7ff5ac158840 2024-03-06T07:36:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 0x7ff5ac0998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 0x7ff5ac0998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60302/0 (socket says 172.21.15.32:60302) 2024-03-06T07:36:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5abe00640 1 -- 172.21.15.32:0/3798768070 learned_addr learned my addr 172.21.15.32:0/3798768070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5ab400640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5ac158840 0x7ff5ac0ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5b0c00640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ac162150 0x7ff5ac0ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5abe00640 1 -- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5ac158840 msgr2=0x7ff5ac0ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.903+0000 7ff5abe00640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5ac158840 0x7ff5ac0ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5abe00640 1 -- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ac162150 msgr2=0x7ff5ac0ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5abe00640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ac162150 0x7ff5ac0ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5abe00640 1 -- 172.21.15.32:0/3798768070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac1f7b10 con 0x7ff5ac158460 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5b0c00640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ac162150 0x7ff5ac0ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5abe00640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 0x7ff5ac0998d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00c4d0 tx=0x7ff59c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c0052d0 con 0x7ff5ac158460 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c005470 con 0x7ff5ac158460 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c012850 con 0x7ff5ac158460 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.904+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac1f7df0 con 0x7ff5ac158460 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.905+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac1f83b0 con 0x7ff5ac158460 2024-03-06T07:36:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.906+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac04f330 con 0x7ff5ac158460 2024-03-06T07:36:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.907+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff59c0129f0 con 0x7ff5ac158460 2024-03-06T07:36:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.908+0000 7ff5a3e00640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff58c06b4d0 0x7ff58c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.908+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff59c089e80 con 0x7ff5ac158460 2024-03-06T07:36:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.908+0000 7ff5ab400640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff58c06b4d0 0x7ff58c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.909+0000 7ff5ab400640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff58c06b4d0 0x7ff58c06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff598036000 tx=0x7ff59803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:22.913+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff59c058c70 con 0x7ff5ac158460 2024-03-06T07:36:23.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.156+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5ac071570 con 0x7ff5ac158460 2024-03-06T07:36:23.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.157+0000 7ff5a3e00640 1 -- 172.21.15.32:0/3798768070 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7ff59c058570 con 0x7ff5ac158460 2024-03-06T07:36:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:23.159 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:23.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff58c06b4d0 msgr2=0x7ff58c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff58c06b4d0 0x7ff58c06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff598036000 tx=0x7ff59803b040 comp rx=0 tx=0).stop 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 msgr2=0x7ff5ac0998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 0x7ff5ac0998d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00c4d0 tx=0x7ff59c00c990 comp rx=0 tx=0).stop 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 shutdown_connections 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff58c06b4d0 0x7ff58c06d990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ac162150 0x7ff5ac0ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff5ac158840 0x7ff5ac0ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 --2- 172.21.15.32:0/3798768070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac158460 0x7ff5ac0998d0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 >> 172.21.15.32:0/3798768070 conn(0x7ff5ac0b5100 msgr2=0x7ff5ac0afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 shutdown_connections 2024-03-06T07:36:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:23.160+0000 7ff5b1600640 1 -- 172.21.15.32:0/3798768070 wait complete. 2024-03-06T07:36:23.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:36:23.410 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:36:23.411 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:36:23.411 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:36:28.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:28.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.636+0000 7ff70ac00640 1 Processor -- start 2024-03-06T07:36:28.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.636+0000 7ff70ac00640 1 -- start start 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff70ac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7040d2960 0x7ff7040d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 0x7ff7040b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff70ac00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040d2dc0 con 0x7ff70415e070 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff70ac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7040d18c0 con 0x7ff7040d2960 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff70ac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d1ba0 con 0x7ff7040d0b00 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff709800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff70415e070 0x7ff7040cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57312/0 (socket says 172.21.15.32:57312) 2024-03-06T07:36:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff709800640 1 -- 172.21.15.32:0/3876605789 learned_addr learned my addr 172.21.15.32:0/3876605789 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:28.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff708e00640 1 --2- 172.21.15.32:0/3876605789 >> v2:172.21.15.39:3300/0 conn(0x7ff7040d2960 0x7ff7040d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.638+0000 7ff70a200640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 0x7ff7040b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909987323 0 0) 0x7ff7040d2dc0 con 0x7ff70415e070 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec003520 con 0x7ff70415e070 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3160697151 0 0) 0x7ff6ec003520 con 0x7ff70415e070 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 msgr2=0x7ff7040b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 0x7ff7040b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 >> v2:172.21.15.39:3300/0 conn(0x7ff7040d2960 msgr2=0x7ff7040d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 --2- 172.21.15.32:0/3876605789 >> v2:172.21.15.39:3300/0 conn(0x7ff7040d2960 0x7ff7040d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7040d1e80 con 0x7ff70415e070 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff6f8001670 con 0x7ff70415e070 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 >> v1:172.21.15.32:6789/0 conn(0x7ff70415e070 legacy=0x7ff7040cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec0039b0 0x7ff6ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 0x7ff6ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff6ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d18c0 con 0x7ff6ec004d40 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d1ba0 con 0x7ff6ec0039b0 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.639+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec00b9c0 con 0x7ff7040d0b00 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff70a200640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff6ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff709800640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec0039b0 0x7ff6ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff708e00640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 0x7ff6ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff70a200640 1 -- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 msgr2=0x7ff6ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff70a200640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7040d0b00 0x7ff6ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff70a200640 1 -- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec0039b0 msgr2=0x7ff6ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff70a200640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec0039b0 0x7ff6ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff70a200640 1 -- 172.21.15.32:0/3876605789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec003520 con 0x7ff6ec004d40 2024-03-06T07:36:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.640+0000 7ff709800640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec0039b0 0x7ff6ec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.641+0000 7ff70a200640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff6ec0051c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009fc0 tx=0x7ff6ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db935b2fe0455d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc039070 con 0x7ff6ec004d40 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc02cc20 con 0x7ff6ec004d40 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff703e00640 1 -- 172.21.15.32:0/3876605789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc009c10 con 0x7ff6ec004d40 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70ac00640 1 -- 172.21.15.32:0/3876605789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 msgr2=0x7ff6ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70ac00640 1 --2- 172.21.15.32:0/3876605789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff6ec0051c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009fc0 tx=0x7ff6ec005830 comp rx=0 tx=0).stop 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70a200640 1 -- 172.21.15.32:0/3876605789 reap_dead start 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70ac00640 1 -- 172.21.15.32:0/3876605789 shutdown_connections 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70ac00640 1 -- 172.21.15.32:0/3876605789 >> 172.21.15.32:0/3876605789 conn(0x7ff70408d2a0 msgr2=0x7ff7040d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70ac00640 1 -- 172.21.15.32:0/3876605789 shutdown_connections 2024-03-06T07:36:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.642+0000 7ff70ac00640 1 -- 172.21.15.32:0/3876605789 wait complete. 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.644+0000 7ff70ac00640 1 Processor -- start 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.644+0000 7ff70ac00640 1 -- start start 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.645+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff70414f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.645+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7041f3680 0x7ff7041e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.645+0000 7ff70ac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff70406b9f0 0x7ff70406be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.645+0000 7ff70ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d45b0 con 0x7ff6ec004d40 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.645+0000 7ff70ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d2440 con 0x7ff7041f3680 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.645+0000 7ff70ac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d4730 con 0x7ff70406b9f0 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff70414f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff70414f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32986/0 (socket says 172.21.15.32:32986) 2024-03-06T07:36:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 -- 172.21.15.32:0/2572802898 learned_addr learned my addr 172.21.15.32:0/2572802898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff70a200640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff70406b9f0 0x7ff70406be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff708e00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7041f3680 0x7ff7041e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 -- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff70406b9f0 msgr2=0x7ff70406be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff70406b9f0 0x7ff70406be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 -- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7041f3680 msgr2=0x7ff7041e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7041f3680 0x7ff7041e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff709800640 1 -- 172.21.15.32:0/2572802898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7041f3b00 con 0x7ff6ec004d40 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.646+0000 7ff708e00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7041f3680 0x7ff7041e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.647+0000 7ff709800640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff70414f590 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800e8c0 tx=0x7ff6f800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.647+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800c870 con 0x7ff6ec004d40 2024-03-06T07:36:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.647+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f8018070 con 0x7ff6ec004d40 2024-03-06T07:36:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.647+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800cc10 con 0x7ff6ec004d40 2024-03-06T07:36:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.647+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70414fb70 con 0x7ff6ec004d40 2024-03-06T07:36:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.647+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff704150130 con 0x7ff6ec004d40 2024-03-06T07:36:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.648+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70404f330 con 0x7ff6ec004d40 2024-03-06T07:36:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.649+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6f801d460 con 0x7ff6ec004d40 2024-03-06T07:36:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.650+0000 7ff701600640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff6e406b4d0 0x7ff6e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.650+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff6f808a290 con 0x7ff6ec004d40 2024-03-06T07:36:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.650+0000 7ff708e00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff6e406b4d0 0x7ff6e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.651+0000 7ff708e00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff6e406b4d0 0x7ff6e406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009a80 tx=0x7ff6f0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:28.655+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6f8058fe0 con 0x7ff6ec004d40 2024-03-06T07:36:29.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.035+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7040d1370 con 0x7ff6ec004d40 2024-03-06T07:36:29.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.037+0000 7ff701600640 1 -- 172.21.15.32:0/2572802898 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7ff6f80588e0 con 0x7ff6ec004d40 2024-03-06T07:36:29.039 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:29.039 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:29.040 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff6e406b4d0 msgr2=0x7ff6e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff6e406b4d0 0x7ff6e406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009a80 tx=0x7ff6f0008040 comp rx=0 tx=0).stop 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 msgr2=0x7ff70414f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff70414f590 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800e8c0 tx=0x7ff6f800ed80 comp rx=0 tx=0).stop 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 shutdown_connections 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff6e406b4d0 0x7ff6e406d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff70406b9f0 0x7ff70406be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7041f3680 0x7ff7041e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 --2- 172.21.15.32:0/2572802898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6ec004d40 0x7ff70414f590 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.040+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 >> 172.21.15.32:0/2572802898 conn(0x7ff70408d2a0 msgr2=0x7ff7040b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.041+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 shutdown_connections 2024-03-06T07:36:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.041+0000 7ff70ac00640 1 -- 172.21.15.32:0/2572802898 wait complete. 2024-03-06T07:36:29.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:36:29.274 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:36:29.274 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:36:29.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:29.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.476+0000 7f1f07e00640 1 Processor -- start 2024-03-06T07:36:29.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.477+0000 7f1f07e00640 1 -- start start 2024-03-06T07:36:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f07e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 0x7f1f0815e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f07e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f081625a0 0x7f1f0815e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f07e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0815edf0 con 0x7f1f08147400 2024-03-06T07:36:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f07e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f08162110 con 0x7f1f0806b6c0 2024-03-06T07:36:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f07e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f08161ae0 con 0x7f1f081625a0 2024-03-06T07:36:29.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f06000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 0x7f1f0815e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f07400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f081625a0 0x7f1f0815e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f06000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 0x7f1f0815e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:55538/0 (socket says 172.21.15.32:55538) 2024-03-06T07:36:29.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.480+0000 7f1f06000640 1 -- 172.21.15.32:0/1233282163 learned_addr learned my addr 172.21.15.32:0/1233282163 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.481+0000 7f1f07400640 1 -- 172.21.15.32:0/1233282163 >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 msgr2=0x7f1f0815e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.481+0000 7f1f07400640 1 --2- 172.21.15.32:0/1233282163 >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 0x7f1f0815e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.481+0000 7f1f07400640 1 -- 172.21.15.32:0/1233282163 >> v1:172.21.15.39:6789/0 conn(0x7f1f08147400 legacy=0x7f1f0806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.481+0000 7f1f07400640 1 -- 172.21.15.32:0/1233282163 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0815f080 con 0x7f1f081625a0 2024-03-06T07:36:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.481+0000 7f1f06000640 1 --2- 172.21.15.32:0/1233282163 >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 0x7f1f0815e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:29.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.483+0000 7f1f07400640 1 --2- 172.21.15.32:0/1233282163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f081625a0 0x7f1f0815e8b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009d90 tx=0x7f1f0002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e8f93430ddcdf7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.483+0000 7f1f05600640 1 -- 172.21.15.32:0/1233282163 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00039070 con 0x7f1f081625a0 2024-03-06T07:36:29.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.483+0000 7f1f05600640 1 -- 172.21.15.32:0/1233282163 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0002cc50 con 0x7f1f081625a0 2024-03-06T07:36:29.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.483+0000 7f1f05600640 1 -- 172.21.15.32:0/1233282163 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00034440 con 0x7f1f081625a0 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.484+0000 7f1f07e00640 1 -- 172.21.15.32:0/1233282163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f081625a0 msgr2=0x7f1f0815e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.484+0000 7f1f07e00640 1 --2- 172.21.15.32:0/1233282163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f081625a0 0x7f1f0815e8b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009d90 tx=0x7f1f0002bea0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.484+0000 7f1f07e00640 1 -- 172.21.15.32:0/1233282163 shutdown_connections 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.484+0000 7f1f07e00640 1 --2- 172.21.15.32:0/1233282163 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f081625a0 0x7f1f0815e8b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.484+0000 7f1f07e00640 1 --2- 172.21.15.32:0/1233282163 >> v2:172.21.15.39:3300/0 conn(0x7f1f0806b6c0 0x7f1f0815e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.484+0000 7f1f07e00640 1 -- 172.21.15.32:0/1233282163 >> 172.21.15.32:0/1233282163 conn(0x7f1f0806c0c0 msgr2=0x7f1f0806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.485+0000 7f1f07e00640 1 -- 172.21.15.32:0/1233282163 shutdown_connections 2024-03-06T07:36:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.485+0000 7f1f07e00640 1 -- 172.21.15.32:0/1233282163 wait complete. 2024-03-06T07:36:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.488+0000 7f1f07e00640 1 Processor -- start 2024-03-06T07:36:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.488+0000 7f1f07e00640 1 -- start start 2024-03-06T07:36:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.490+0000 7f1f07e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 0x7f1f0814ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.490+0000 7f1f07e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f08147400 0x7f1f080af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.490+0000 7f1f07e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f081625a0 0x7f1f080adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f07e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f08147960 con 0x7f1f0806b6c0 2024-03-06T07:36:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f07e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f081477e0 con 0x7f1f081625a0 2024-03-06T07:36:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f07e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0806e3b0 con 0x7f1f08147400 2024-03-06T07:36:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 0x7f1f0814ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 0x7f1f0814ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32994/0 (socket says 172.21.15.32:32994) 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f06a00640 1 -- 172.21.15.32:0/758009927 learned_addr learned my addr 172.21.15.32:0/758009927 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f07400640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f081625a0 0x7f1f080adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.491+0000 7f1f06a00640 1 -- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f08147400 msgr2=0x7f1f080af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f06a00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f08147400 0x7f1f080af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f06a00640 1 -- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f081625a0 msgr2=0x7f1f080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f06a00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f081625a0 0x7f1f080adfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f06a00640 1 -- 172.21.15.32:0/758009927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080aff30 con 0x7f1f0806b6c0 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f06a00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 0x7f1f0814ff70 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c480 tx=0x7f1ef800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef80052d0 con 0x7f1f0806b6c0 2024-03-06T07:36:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8005470 con 0x7f1f0806b6c0 2024-03-06T07:36:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8012870 con 0x7f1f0806b6c0 2024-03-06T07:36:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0814d450 con 0x7f1f0806b6c0 2024-03-06T07:36:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.492+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0814da10 con 0x7f1f0806b6c0 2024-03-06T07:36:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.493+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0804f330 con 0x7f1f0806b6c0 2024-03-06T07:36:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.494+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ef8012a10 con 0x7f1f0806b6c0 2024-03-06T07:36:29.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.495+0000 7f1efea00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1ee806b4d0 0x7f1ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.495+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1ef8089ea0 con 0x7f1f0806b6c0 2024-03-06T07:36:29.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.496+0000 7f1f06000640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1ee806b4d0 0x7f1ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.496+0000 7f1f06000640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1ee806b4d0 0x7f1ee806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009630 tx=0x7f1ef4006e90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.500+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f080aff30 con 0x7f1f0806b6c0 2024-03-06T07:36:29.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.739+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f0806e190 con 0x7f1f0806b6c0 2024-03-06T07:36:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.740+0000 7f1efea00640 1 -- 172.21.15.32:0/758009927 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f1ef8058d20 con 0x7f1f0806b6c0 2024-03-06T07:36:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:29.742 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1ee806b4d0 msgr2=0x7f1ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1ee806b4d0 0x7f1ee806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009630 tx=0x7f1ef4006e90 comp rx=0 tx=0).stop 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 msgr2=0x7f1f0814ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 0x7f1f0814ff70 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c480 tx=0x7f1ef800c940 comp rx=0 tx=0).stop 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 shutdown_connections 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1ee806b4d0 0x7f1ee806d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f081625a0 0x7f1f080adfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f08147400 0x7f1f080af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 --2- 172.21.15.32:0/758009927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f0806b6c0 0x7f1f0814ff70 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 >> 172.21.15.32:0/758009927 conn(0x7f1f0806c0c0 msgr2=0x7f1f080b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 shutdown_connections 2024-03-06T07:36:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:29.742+0000 7f1f07e00640 1 -- 172.21.15.32:0/758009927 wait complete. 2024-03-06T07:36:29.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:36:29.993 INFO:teuthology.orchestra.run.smithi032.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:36:29.993 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:36:29.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:30.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.217+0000 7f8556e00640 1 Processor -- start 2024-03-06T07:36:30.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.217+0000 7f8556e00640 1 -- start start 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8556e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85500cfd50 0x7f85500d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8556e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f855015e410 0x7f85500d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8556e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85500d2f10 con 0x7f855015e030 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8556e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85500cea50 con 0x7f85500cfd50 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8556e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85500cecf0 con 0x7f855015e410 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8547400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85500cfd50 0x7f85500d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8547400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85500cfd50 0x7f85500d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32998/0 (socket says 172.21.15.32:32998) 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f8547400640 1 -- 172.21.15.32:0/3405261024 learned_addr learned my addr 172.21.15.32:0/3405261024 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.219+0000 7f854fe00640 1 --1- 172.21.15.32:0/3405261024 >> v1:172.21.15.32:6789/0 conn(0x7f855015e030 0x7f8550161d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57328/0 (socket says 172.21.15.32:57328) 2024-03-06T07:36:30.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.220+0000 7f8554e00640 1 --2- 172.21.15.32:0/3405261024 >> v2:172.21.15.39:3300/0 conn(0x7f855015e410 0x7f85500d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.220+0000 7f8547400640 1 -- 172.21.15.32:0/3405261024 >> v2:172.21.15.39:3300/0 conn(0x7f855015e410 msgr2=0x7f85500d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.220+0000 7f8547400640 1 --2- 172.21.15.32:0/3405261024 >> v2:172.21.15.39:3300/0 conn(0x7f855015e410 0x7f85500d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.220+0000 7f8547400640 1 -- 172.21.15.32:0/3405261024 >> v1:172.21.15.32:6789/0 conn(0x7f855015e030 legacy=0x7f8550161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.220+0000 7f8547400640 1 -- 172.21.15.32:0/3405261024 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500cef90 con 0x7f85500cfd50 2024-03-06T07:36:30.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.221+0000 7f8547400640 1 --2- 172.21.15.32:0/3405261024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85500cfd50 0x7f85500d2490 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f85500d3e60 tx=0x7f853802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4f609483605347 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.221+0000 7f854f400640 1 -- 172.21.15.32:0/3405261024 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538034070 con 0x7f85500cfd50 2024-03-06T07:36:30.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f854f400640 1 -- 172.21.15.32:0/3405261024 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8538030870 con 0x7f85500cfd50 2024-03-06T07:36:30.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f854f400640 1 -- 172.21.15.32:0/3405261024 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538030c40 con 0x7f85500cfd50 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 -- 172.21.15.32:0/3405261024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85500cfd50 msgr2=0x7f85500d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 --2- 172.21.15.32:0/3405261024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85500cfd50 0x7f85500d2490 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f85500d3e60 tx=0x7f853802b9b0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 -- 172.21.15.32:0/3405261024 shutdown_connections 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 --2- 172.21.15.32:0/3405261024 >> v2:172.21.15.39:3300/0 conn(0x7f855015e410 0x7f85500d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 --2- 172.21.15.32:0/3405261024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85500cfd50 0x7f85500d2490 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 -- 172.21.15.32:0/3405261024 >> 172.21.15.32:0/3405261024 conn(0x7f85500d3eb0 msgr2=0x7f85500d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 -- 172.21.15.32:0/3405261024 shutdown_connections 2024-03-06T07:36:30.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.222+0000 7f8556e00640 1 -- 172.21.15.32:0/3405261024 wait complete. 2024-03-06T07:36:30.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.224+0000 7f8556e00640 1 Processor -- start 2024-03-06T07:36:30.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.224+0000 7f8556e00640 1 -- start start 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8556e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f855015e030 0x7f85500ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8556e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 0x7f85500c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8556e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8550159030 0x7f85501583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8556e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500cf830 con 0x7f855015e410 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8556e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500cf5d0 con 0x7f855015e030 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8556e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8550162430 con 0x7f8550159030 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.225+0000 7f8547400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 0x7f85500c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 0x7f85500c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33008/0 (socket says 172.21.15.32:33008) 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 -- 172.21.15.32:0/1999077089 learned_addr learned my addr 172.21.15.32:0/1999077089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f854fe00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f855015e030 0x7f85500ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8554e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8550159030 0x7f85501583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 -- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8550159030 msgr2=0x7f85501583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8550159030 0x7f85501583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 -- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f855015e030 msgr2=0x7f85500ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f855015e030 0x7f85500ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 -- 172.21.15.32:0/1999077089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855015f970 con 0x7f855015e410 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.226+0000 7f8547400640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 0x7f85500c8270 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8538000c00 tx=0x7f8538004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.227+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538048070 con 0x7f855015e410 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.227+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8538030670 con 0x7f855015e410 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.227+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855015fc30 con 0x7f855015e410 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.227+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538034040 con 0x7f855015e410 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.227+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85501601f0 con 0x7f855015e410 2024-03-06T07:36:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.228+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855004f3a0 con 0x7f855015e410 2024-03-06T07:36:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.229+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f853804b070 con 0x7f855015e410 2024-03-06T07:36:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.230+0000 7f854cc00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f852c06b4d0 0x7f852c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.230+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f85380ac500 con 0x7f855015e410 2024-03-06T07:36:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.230+0000 7f854fe00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f852c06b4d0 0x7f852c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.231+0000 7f854fe00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f852c06b4d0 0x7f852c06d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f85500b98a0 tx=0x7f8540007ce0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.238+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f853807b400 con 0x7f855015e410 2024-03-06T07:36:30.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.481+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f855008e030 con 0x7f855015e410 2024-03-06T07:36:30.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.483+0000 7f854cc00640 1 -- 172.21.15.32:0/1999077089 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f853807ad00 con 0x7f855015e410 2024-03-06T07:36:30.485 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:30.485 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:30.485 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:30.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f852c06b4d0 msgr2=0x7f852c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f852c06b4d0 0x7f852c06d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f85500b98a0 tx=0x7f8540007ce0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 msgr2=0x7f85500c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 0x7f85500c8270 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8538000c00 tx=0x7f8538004730 comp rx=0 tx=0).stop 2024-03-06T07:36:30.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 shutdown_connections 2024-03-06T07:36:30.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f852c06b4d0 0x7f852c06d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8550159030 0x7f85501583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f855015e410 0x7f85500c8270 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.485+0000 7f8556e00640 1 --2- 172.21.15.32:0/1999077089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f855015e030 0x7f85500ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.486+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 >> 172.21.15.32:0/1999077089 conn(0x7f85500d3eb0 msgr2=0x7f85500b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.486+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 shutdown_connections 2024-03-06T07:36:30.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.486+0000 7f8556e00640 1 -- 172.21.15.32:0/1999077089 wait complete. 2024-03-06T07:36:30.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2024-03-06T07:36:30.738 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:36:30.739 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:36:30.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:30.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.954+0000 7f6d5f200640 1 Processor -- start 2024-03-06T07:36:30.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.955+0000 7f6d5f200640 1 -- start start 2024-03-06T07:36:30.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.956+0000 7f6d5f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6d6015e0b0 0x7f6d600cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.956+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 0x7f6d600b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.956+0000 7f6d5f200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d600b3710 con 0x7f6d600d4a10 2024-03-06T07:36:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.956+0000 7f6d5f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d600b09e0 con 0x7f6d6015e0b0 2024-03-06T07:36:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.956+0000 7f6d5f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600b39f0 con 0x7f6d600d0e50 2024-03-06T07:36:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.956+0000 7f6d5d400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6d600d4a10 0x7f6d600d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57344/0 (socket says 172.21.15.32:57344) 2024-03-06T07:36:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5d400640 1 -- 172.21.15.32:0/3163987513 learned_addr learned my addr 172.21.15.32:0/3163987513 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5e800640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 0x7f6d600b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5de00640 1 --2- 172.21.15.32:0/3163987513 >> v2:172.21.15.39:3300/0 conn(0x7f6d6015e0b0 0x7f6d600cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551017119 0 0) 0x7f6d600b3710 con 0x7f6d600d4a10 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d40003520 con 0x7f6d600d4a10 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 344343507 0 0) 0x7f6d40003520 con 0x7f6d600d4a10 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 msgr2=0x7f6d600b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.957+0000 7f6d5ca00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 0x7f6d600b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 >> v2:172.21.15.39:3300/0 conn(0x7f6d6015e0b0 msgr2=0x7f6d600cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 --2- 172.21.15.32:0/3163987513 >> v2:172.21.15.39:3300/0 conn(0x7f6d6015e0b0 0x7f6d600cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d600b3cd0 con 0x7f6d600d4a10 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6d48001a10 con 0x7f6d600d4a10 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 >> v1:172.21.15.32:6789/0 conn(0x7f6d600d4a10 legacy=0x7f6d600d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d400039b0 0x7f6d40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d40004800 0x7f6d40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 0x7f6d400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600b39f0 con 0x7f6d400039b0 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4000b9c0 con 0x7f6d40004800 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4000bb40 con 0x7f6d600d0e50 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5de00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d400039b0 0x7f6d40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5e800640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 0x7f6d400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.958+0000 7f6d5d400640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d40004800 0x7f6d40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.959+0000 7f6d5de00640 1 -- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 msgr2=0x7f6d400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.959+0000 7f6d5de00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d600d0e50 0x7f6d400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.959+0000 7f6d5de00640 1 -- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d40004800 msgr2=0x7f6d40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.959+0000 7f6d5de00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d40004800 0x7f6d40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.959+0000 7f6d5de00640 1 -- 172.21.15.32:0/3163987513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d40003520 con 0x7f6d400039b0 2024-03-06T07:36:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.960+0000 7f6d5de00640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d400039b0 0x7f6d40003dd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009ac0 tx=0x7f6d40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d949abc402346207 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50039070 con 0x7f6d400039b0 2024-03-06T07:36:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5002c710 con 0x7f6d400039b0 2024-03-06T07:36:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5ca00640 1 -- 172.21.15.32:0/3163987513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50009690 con 0x7f6d400039b0 2024-03-06T07:36:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5f200640 1 -- 172.21.15.32:0/3163987513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d400039b0 msgr2=0x7f6d40003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5f200640 1 --2- 172.21.15.32:0/3163987513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d400039b0 0x7f6d40003dd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009ac0 tx=0x7f6d40005830 comp rx=0 tx=0).stop 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5e800640 1 -- 172.21.15.32:0/3163987513 reap_dead start 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5f200640 1 -- 172.21.15.32:0/3163987513 shutdown_connections 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.961+0000 7f6d5f200640 1 -- 172.21.15.32:0/3163987513 >> 172.21.15.32:0/3163987513 conn(0x7f6d600d3d30 msgr2=0x7f6d600d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.962+0000 7f6d5f200640 1 -- 172.21.15.32:0/3163987513 shutdown_connections 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.962+0000 7f6d5f200640 1 -- 172.21.15.32:0/3163987513 wait complete. 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.963+0000 7f6d5f200640 1 Processor -- start 2024-03-06T07:36:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.963+0000 7f6d5f200640 1 -- start start 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.964+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d400039b0 0x7f6d600ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.964+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 0x7f6d6015af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.964+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d601f4160 0x7f6d600cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.964+0000 7f6d5f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600cfc60 con 0x7f6d600b8080 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.964+0000 7f6d5f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6015e490 con 0x7f6d400039b0 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.964+0000 7f6d5f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600cf850 con 0x7f6d601f4160 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 0x7f6d6015af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 0x7f6d6015af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33026/0 (socket says 172.21.15.32:33026) 2024-03-06T07:36:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 -- 172.21.15.32:0/4015296943 learned_addr learned my addr 172.21.15.32:0/4015296943 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:30.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5e800640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d601f4160 0x7f6d600cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5de00640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d400039b0 0x7f6d600ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 -- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d601f4160 msgr2=0x7f6d600cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d601f4160 0x7f6d600cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 -- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d400039b0 msgr2=0x7f6d600ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d400039b0 0x7f6d600ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5d400640 1 -- 172.21.15.32:0/4015296943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6015b450 con 0x7f6d600b8080 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.965+0000 7f6d5e800640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d601f4160 0x7f6d600cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.966+0000 7f6d5d400640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 0x7f6d6015af10 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f6d480079e0 tx=0x7f6d4800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.966+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48007ad0 con 0x7f6d600b8080 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.966+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48018070 con 0x7f6d600b8080 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.966+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48013420 con 0x7f6d600b8080 2024-03-06T07:36:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.966+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d600b87f0 con 0x7f6d600b8080 2024-03-06T07:36:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.966+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d600b6cd0 con 0x7f6d600b8080 2024-03-06T07:36:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.967+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6004f330 con 0x7f6d600b8080 2024-03-06T07:36:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.968+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d480135e0 con 0x7f6d600b8080 2024-03-06T07:36:30.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.969+0000 7f6d56000640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.969+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6d4808a7c0 con 0x7f6d600b8080 2024-03-06T07:36:30.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.969+0000 7f6d5de00640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.970+0000 7f6d5de00640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6d50036000 tx=0x7f6d50030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:30.974+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d480595a0 con 0x7f6d600b8080 2024-03-06T07:36:31.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.219+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d6015d760 con 0x7f6d600b8080 2024-03-06T07:36:31.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.219+0000 7f6d56000640 1 -- 172.21.15.32:0/4015296943 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f6d48058ea0 con 0x7f6d600b8080 2024-03-06T07:36:31.222 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:31.222 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:31.222 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6d3c06b500 msgr2=0x7f6d3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6d50036000 tx=0x7f6d50030000 comp rx=0 tx=0).stop 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 msgr2=0x7f6d6015af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 0x7f6d6015af10 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f6d480079e0 tx=0x7f6d4800e530 comp rx=0 tx=0).stop 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 shutdown_connections 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6d601f4160 0x7f6d600cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600b8080 0x7f6d6015af10 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 --2- 172.21.15.32:0/4015296943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d400039b0 0x7f6d600ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.223+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 >> 172.21.15.32:0/4015296943 conn(0x7f6d600d3d30 msgr2=0x7f6d600b6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.224+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 shutdown_connections 2024-03-06T07:36:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.224+0000 7f6d5f200640 1 -- 172.21.15.32:0/4015296943 wait complete. 2024-03-06T07:36:31.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2024-03-06T07:36:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:36:31.504 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:36:31.504 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:36:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 Processor -- start 2024-03-06T07:36:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 -- start start 2024-03-06T07:36:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db450780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db3c3680 0x5582db444d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 0x5582db448590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582db320fd0 con 0x5582db4513c0 2024-03-06T07:36:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582db37cfa0 con 0x5582db3c3680 2024-03-06T07:36:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.605+0000 7f33b263bf40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5582db4446f0 con 0x5582db37cb90 2024-03-06T07:36:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 0x5582db448590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 0x5582db448590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33052/0 (socket says 172.21.15.32:33052) 2024-03-06T07:36:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 -- 172.21.15.32:0/4134938054 learned_addr learned my addr 172.21.15.32:0/4134938054 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33ae600640 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db3c3680 0x5582db444d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33af000640 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db450780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 -- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 msgr2=0x5582db450780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db450780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 -- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db3c3680 msgr2=0x5582db444d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db3c3680 0x5582db444d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.606+0000 7f33afa00640 1 -- 172.21.15.32:0/4134938054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582db3a9360 con 0x5582db4513c0 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.607+0000 7f33afa00640 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 0x5582db448590 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f33a800a560 tx=0x7f33a804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72af4514d72cb13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.607+0000 7f33ad200640 1 -- 172.21.15.32:0/4134938054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a80528d0 con 0x5582db4513c0 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.607+0000 7f33ad200640 1 -- 172.21.15.32:0/4134938054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8062070 con 0x5582db4513c0 2024-03-06T07:36:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.607+0000 7f33ad200640 1 -- 172.21.15.32:0/4134938054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8052c90 con 0x5582db4513c0 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 -- 172.21.15.32:0/4134938054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 msgr2=0x5582db448590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 0x5582db448590 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f33a800a560 tx=0x7f33a804c000 comp rx=0 tx=0).stop 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 -- 172.21.15.32:0/4134938054 shutdown_connections 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db4513c0 0x5582db448590 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db3c3680 0x5582db444d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 --2- 172.21.15.32:0/4134938054 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db450780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 -- 172.21.15.32:0/4134938054 >> 172.21.15.32:0/4134938054 conn(0x5582db37a5b0 msgr2=0x5582db44d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 -- 172.21.15.32:0/4134938054 shutdown_connections 2024-03-06T07:36:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.608+0000 7f33b263bf40 1 -- 172.21.15.32:0/4134938054 wait complete. 2024-03-06T07:36:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.609+0000 7f33b263bf40 1 Processor -- start 2024-03-06T07:36:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.610+0000 7f33b263bf40 1 -- start start 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.611+0000 7f33b263bf40 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db4356e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.611+0000 7f33b263bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 0x5582db447270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.611+0000 7f33b263bf40 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db4513c0 0x5582db44e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.611+0000 7f33b263bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582db3a9600 con 0x5582db3c3680 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.611+0000 7f33b263bf40 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582db3a87d0 con 0x5582db4513c0 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.611+0000 7f33b263bf40 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5582db3a8380 con 0x5582db37cb90 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 0x5582db447270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 0x5582db447270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33056/0 (socket says 172.21.15.32:33056) 2024-03-06T07:36:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 learned_addr learned my addr 172.21.15.32:0/461910565 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db4356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db4513c0 0x5582db44e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 msgr2=0x5582db4356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5582db37cb90 0x5582db4356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db4513c0 msgr2=0x5582db44e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db4513c0 0x5582db44e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582db435c20 con 0x5582db3c3680 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.612+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5582db4513c0 0x5582db44e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.613+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 0x5582db447270 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f339c009f10 tx=0x7f339c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.613+0000 7f33a7400640 1 -- 172.21.15.32:0/461910565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c0143f0 con 0x5582db3c3680 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.613+0000 7f33a7400640 1 -- 172.21.15.32:0/461910565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339c019070 con 0x5582db3c3680 2024-03-06T07:36:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.613+0000 7f33a7400640 1 -- 172.21.15.32:0/461910565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c0147b0 con 0x5582db3c3680 2024-03-06T07:36:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.613+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5582db448b10 con 0x5582db3c3680 2024-03-06T07:36:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.613+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582db4490d0 con 0x5582db3c3680 2024-03-06T07:36:31.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.615+0000 7f33a7400640 1 -- 172.21.15.32:0/461910565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f339c014950 con 0x5582db3c3680 2024-03-06T07:36:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.616+0000 7f33a7400640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3390069eb0 0x7f339006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.616+0000 7f33a7400640 1 -- 172.21.15.32:0/461910565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f339c088e30 con 0x5582db3c3680 2024-03-06T07:36:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.616+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3390069eb0 0x7f339006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.616+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x5582db3beca0 0x5582db3bf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.616+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.617+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3390069eb0 0x7f339006c370 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f33a0002e60 tx=0x7f33a0002440 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.617+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x5582db3beca0 0x5582db3bf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.618+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x5582db3beca0 0x5582db3bf080 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.619+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 1 ==== osd_op_reply(1 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a80527f0 con 0x5582db3beca0 2024-03-06T07:36:31.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.619+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f33a8052030 0x7f33a805daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.619+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.619+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f33a8052030 0x7f33a805daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.620+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f33a8052030 0x7f33a805daa0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.620+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 1 ==== osd_op_reply(2 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.620+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f33a0010860 0x7f33a001c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.620+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.621+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f33a0010860 0x7f33a001c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.621+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f33a0010860 0x7f33a001c320 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.622+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 1 ==== osd_op_reply(3 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c01bd80 con 0x7f33a0010860 2024-03-06T07:36:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.622+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f339c057770 0x7f339c0563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.622+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c057770 2024-03-06T07:36:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.622+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f339c057770 0x7f339c0563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.623+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f339c057770 0x7f339c0563d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.623+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.2 v2:172.21.15.32:6800/1600044506 1 ==== osd_op_reply(4 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a80527f0 con 0x7f339c057770 2024-03-06T07:36:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.623+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.624+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 2 ==== osd_op_reply(5 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a80527f0 con 0x5582db3beca0 2024-03-06T07:36:31.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.624+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.625+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 2 ==== osd_op_reply(6 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.625+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f339c05f790 0x7f339c058840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.625+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c05f790 2024-03-06T07:36:31.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.625+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f339c05f790 0x7f339c058840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.626+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f339c05f790 0x7f339c058840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.627+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.5 v2:172.21.15.39:6816/969666024 1 ==== osd_op_reply(7 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f339c05f790 2024-03-06T07:36:31.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.627+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.628+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 3 ==== osd_op_reply(8 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a80527f0 con 0x5582db3beca0 2024-03-06T07:36:31.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.628+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f33a80663f0 0x7f33a8066850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.628+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.629+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f33a80663f0 0x7f33a8066850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.629+0000 7f33ae600640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f33a80663f0 0x7f33a8066850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.629+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 1 ==== osd_op_reply(9 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.629+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.630+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 3 ==== osd_op_reply(10 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.630+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.630+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 4 ==== osd_op_reply(11 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.630+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.630+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 2 ==== osd_op_reply(12 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.631+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.632+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 4 ==== osd_op_reply(13 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a80527f0 con 0x5582db3beca0 2024-03-06T07:36:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.632+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f33a80758b0 0x7f33a80513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.632+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.632+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f33a80758b0 0x7f33a80513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.634+0000 7f33afa00640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f33a80758b0 0x7f33a80513b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.634+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 1 ==== osd_op_reply(14 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.634+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.635+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 2 ==== osd_op_reply(15 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.635+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.636+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 3 ==== osd_op_reply(16 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.636+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.637+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 2 ==== osd_op_reply(17 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.637+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c05f790 2024-03-06T07:36:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.638+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.5 v2:172.21.15.39:6816/969666024 2 ==== osd_op_reply(18 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f339c05f790 2024-03-06T07:36:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.638+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.638+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 5 ==== osd_op_reply(19 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.638+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.639+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 5 ==== osd_op_reply(20 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.639+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.640+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 6 ==== osd_op_reply(21 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.640+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.641+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 6 ==== osd_op_reply(22 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.641+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.641+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 4 ==== osd_op_reply(23 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.641+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.642+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 3 ==== osd_op_reply(24 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.642+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f33a0023440 0x7f33a00238c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.642+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0023440 2024-03-06T07:36:31.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.643+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f33a0023440 0x7f33a00238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.644+0000 7f33af000640 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f33a0023440 0x7f33a00238c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.645+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.6 v2:172.21.15.39:6808/2944464285 1 ==== osd_op_reply(25 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a0023440 2024-03-06T07:36:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.645+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.646+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 5 ==== osd_op_reply(26 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.646+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c05f790 2024-03-06T07:36:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.647+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.5 v2:172.21.15.39:6816/969666024 3 ==== osd_op_reply(27 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f339c05f790 2024-03-06T07:36:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.647+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.647+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 6 ==== osd_op_reply(28 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.647+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c057770 2024-03-06T07:36:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.648+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.2 v2:172.21.15.32:6800/1600044506 2 ==== osd_op_reply(29 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f339c057770 2024-03-06T07:36:31.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.648+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0023440 2024-03-06T07:36:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.649+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.6 v2:172.21.15.39:6808/2944464285 2 ==== osd_op_reply(30 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a0023440 2024-03-06T07:36:31.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.649+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.650+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 4 ==== osd_op_reply(31 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.650+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.651+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 3 ==== osd_op_reply(32 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.651+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.652+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 7 ==== osd_op_reply(33 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.652+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.652+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 8 ==== osd_op_reply(34 [pgnls start_epoch 116 out=49b] v110'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.652+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.653+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 7 ==== osd_op_reply(35 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.653+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.654+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 5 ==== osd_op_reply(36 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.654+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.655+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 8 ==== osd_op_reply(37 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.655+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.656+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 4 ==== osd_op_reply(38 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.656+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c05f790 2024-03-06T07:36:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.657+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.5 v2:172.21.15.39:6816/969666024 4 ==== osd_op_reply(39 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f339c05f790 2024-03-06T07:36:31.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.657+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c05f790 2024-03-06T07:36:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.658+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.5 v2:172.21.15.39:6816/969666024 5 ==== osd_op_reply(40 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f339c05f790 2024-03-06T07:36:31.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.658+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0023440 2024-03-06T07:36:31.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.659+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.6 v2:172.21.15.39:6808/2944464285 3 ==== osd_op_reply(41 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a0023440 2024-03-06T07:36:31.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.659+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.660+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 6 ==== osd_op_reply(42 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.660+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c05f790 2024-03-06T07:36:31.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.661+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.5 v2:172.21.15.39:6816/969666024 6 ==== osd_op_reply(43 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f339c05f790 2024-03-06T07:36:31.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.661+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.662+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 5 ==== osd_op_reply(44 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.662+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.663+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 7 ==== osd_op_reply(45 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.663+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.664+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 8 ==== osd_op_reply(46 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.664+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.665+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 9 ==== osd_op_reply(47 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.665+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.665+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 10 ==== osd_op_reply(48 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.666+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.667+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 6 ==== osd_op_reply(49 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.667+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.667+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 7 ==== osd_op_reply(50 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.668+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0010860 2024-03-06T07:36:31.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.668+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.3 v2:172.21.15.32:6806/238384581 9 ==== osd_op_reply(51 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a0010860 2024-03-06T07:36:31.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.668+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x5582db3beca0 2024-03-06T07:36:31.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.669+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.7 v2:172.21.15.39:6823/3019464823 11 ==== osd_op_reply(52 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x5582db3beca0 2024-03-06T07:36:31.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.669+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.670+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 9 ==== osd_op_reply(53 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.670+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.671+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 8 ==== osd_op_reply(54 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.671+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.672+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 9 ==== osd_op_reply(55 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.672+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f339c057770 2024-03-06T07:36:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.673+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.2 v2:172.21.15.32:6800/1600044506 3 ==== osd_op_reply(56 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f339c057770 2024-03-06T07:36:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.673+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.673+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 10 ==== osd_op_reply(57 [pgnls start_epoch 116 out=49b] v116'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.673+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.675+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 10 ==== osd_op_reply(58 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.675+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0023440 2024-03-06T07:36:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.676+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.6 v2:172.21.15.39:6808/2944464285 4 ==== osd_op_reply(59 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a0023440 2024-03-06T07:36:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.676+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.676+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 11 ==== osd_op_reply(60 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.676+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a8052030 2024-03-06T07:36:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.677+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.1 v2:172.21.15.32:6817/50998742 7 ==== osd_op_reply(61 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a8052030 2024-03-06T07:36:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.677+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80758b0 2024-03-06T07:36:31.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.679+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 <== osd.4 v2:172.21.15.39:6800/89256438 11 ==== osd_op_reply(62 [pgnls start_epoch 116 out=49b] v116'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a8087040 con 0x7f33a80758b0 2024-03-06T07:36:31.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.679+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a0023440 2024-03-06T07:36:31.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.680+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 <== osd.6 v2:172.21.15.39:6808/2944464285 5 ==== osd_op_reply(63 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f33a00114f0 con 0x7f33a0023440 2024-03-06T07:36:31.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.680+0000 7f33af000640 1 -- 172.21.15.32:0/461910565 --> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x5582db3bf5c0 con 0x7f33a80663f0 2024-03-06T07:36:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33ae600640 1 -- 172.21.15.32:0/461910565 <== osd.0 v2:172.21.15.32:6816/3728617102 12 ==== osd_op_reply(64 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f339c053b80 con 0x7f33a80663f0 2024-03-06T07:36:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f33a80663f0 msgr2=0x7f33a8066850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6816/3728617102,v1:172.21.15.32:6818/3728617102] conn(0x7f33a80663f0 0x7f33a8066850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f33a8052030 msgr2=0x7f33a805daa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6817/50998742,v1:172.21.15.32:6819/50998742] conn(0x7f33a8052030 0x7f33a805daa0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f339c057770 msgr2=0x7f339c0563d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6800/1600044506,v1:172.21.15.32:6801/1600044506] conn(0x7f339c057770 0x7f339c0563d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f33a0010860 msgr2=0x7f33a001c320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6806/238384581,v1:172.21.15.32:6808/238384581] conn(0x7f33a0010860 0x7f33a001c320 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 reap_dead start 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f33a80758b0 msgr2=0x7f33a80513b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6800/89256438,v1:172.21.15.39:6801/89256438] conn(0x7f33a80758b0 0x7f33a80513b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f339c05f790 msgr2=0x7f339c058840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6816/969666024,v1:172.21.15.39:6817/969666024] conn(0x7f339c05f790 0x7f339c058840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f33a0023440 msgr2=0x7f33a00238c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6808/2944464285,v1:172.21.15.39:6809/2944464285] conn(0x7f33a0023440 0x7f33a00238c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x5582db3beca0 msgr2=0x5582db3bf080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33afa00640 1 -- 172.21.15.32:0/461910565 reap_dead start 2024-03-06T07:36:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x5582db3beca0 0x5582db3bf080 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3390069eb0 msgr2=0x7f339006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3390069eb0 0x7f339006c370 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f33a0002e60 tx=0x7f33a0002440 comp rx=0 tx=0).stop 2024-03-06T07:36:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 msgr2=0x5582db447270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.681+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 0x5582db447270 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f339c009f10 tx=0x7f339c005f70 comp rx=0 tx=0).stop 2024-03-06T07:36:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 shutdown_connections 2024-03-06T07:36:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.39:6823/3019464823,v1:172.21.15.39:6825/3019464823] conn(0x5582db3beca0 0x5582db3bf080 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3390069eb0 0x7f339006c370 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 --2- 172.21.15.32:0/461910565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5582db3c3680 0x5582db447270 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 >> 172.21.15.32:0/461910565 conn(0x5582db37a5b0 msgr2=0x5582db3c2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 shutdown_connections 2024-03-06T07:36:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.682+0000 7f33b263bf40 1 -- 172.21.15.32:0/461910565 wait complete. 2024-03-06T07:36:31.692 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:36:31.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:31.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.899+0000 7f5e56e00640 1 Processor -- start 2024-03-06T07:36:31.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.899+0000 7f5e56e00640 1 -- start start 2024-03-06T07:36:31.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e56e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e500d4a10 0x7f5e500d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e56e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d0c80 0x7f5e500b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e56e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e500b3710 con 0x7f5e5015e0b0 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e56e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e500b09e0 con 0x7f5e500d4a10 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e56e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500cfa20 con 0x7f5e500d0c80 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e55000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e500d4a10 0x7f5e500d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e55000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e500d4a10 0x7f5e500d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33072/0 (socket says 172.21.15.32:33072) 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.901+0000 7f5e55000640 1 -- 172.21.15.32:0/885427783 learned_addr learned my addr 172.21.15.32:0/885427783 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.902+0000 7f5e56400640 1 --2- 172.21.15.32:0/885427783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d0c80 0x7f5e500b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.902+0000 7f5e55000640 1 -- 172.21.15.32:0/885427783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d0c80 msgr2=0x7f5e500b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.902+0000 7f5e55000640 1 --2- 172.21.15.32:0/885427783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d0c80 0x7f5e500b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.902+0000 7f5e55000640 1 -- 172.21.15.32:0/885427783 >> v1:172.21.15.39:6789/0 conn(0x7f5e5015e0b0 legacy=0x7f5e500cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.902+0000 7f5e55000640 1 -- 172.21.15.32:0/885427783 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500b39f0 con 0x7f5e500d4a10 2024-03-06T07:36:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.903+0000 7f5e55000640 1 --2- 172.21.15.32:0/885427783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e500d4a10 0x7f5e500d0740 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009870 tx=0x7f5e3c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbeac082b26741f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.903+0000 7f5e4fe00640 1 -- 172.21.15.32:0/885427783 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c034070 con 0x7f5e500d4a10 2024-03-06T07:36:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e4fe00640 1 -- 172.21.15.32:0/885427783 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c030840 con 0x7f5e500d4a10 2024-03-06T07:36:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e4fe00640 1 -- 172.21.15.32:0/885427783 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c030c10 con 0x7f5e500d4a10 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 -- 172.21.15.32:0/885427783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e500d4a10 msgr2=0x7f5e500d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 --2- 172.21.15.32:0/885427783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e500d4a10 0x7f5e500d0740 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009870 tx=0x7f5e3c02b980 comp rx=0 tx=0).stop 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 -- 172.21.15.32:0/885427783 shutdown_connections 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 --2- 172.21.15.32:0/885427783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d0c80 0x7f5e500b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 --2- 172.21.15.32:0/885427783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e500d4a10 0x7f5e500d0740 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 -- 172.21.15.32:0/885427783 >> 172.21.15.32:0/885427783 conn(0x7f5e500d3d30 msgr2=0x7f5e500d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 -- 172.21.15.32:0/885427783 shutdown_connections 2024-03-06T07:36:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.904+0000 7f5e56e00640 1 -- 172.21.15.32:0/885427783 wait complete. 2024-03-06T07:36:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.906+0000 7f5e56e00640 1 Processor -- start 2024-03-06T07:36:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.906+0000 7f5e56e00640 1 -- start start 2024-03-06T07:36:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.907+0000 7f5e56e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e500d0c80 0x7f5e500b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.907+0000 7f5e56e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d4a10 0x7f5e500b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.907+0000 7f5e56e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 0x7f5e5014fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.907+0000 7f5e56e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5015e490 con 0x7f5e5015e0b0 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.907+0000 7f5e56e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500b3fb0 con 0x7f5e500d0c80 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.907+0000 7f5e56e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500cfc60 con 0x7f5e500d4a10 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 0x7f5e5014fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 0x7f5e5014fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33078/0 (socket says 172.21.15.32:33078) 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 -- 172.21.15.32:0/1022464977 learned_addr learned my addr 172.21.15.32:0/1022464977 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e55000640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d4a10 0x7f5e500b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e55a00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e500d0c80 0x7f5e500b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 -- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d4a10 msgr2=0x7f5e500b86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d4a10 0x7f5e500b86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 -- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e500d0c80 msgr2=0x7f5e500b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e500d0c80 0x7f5e500b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.908+0000 7f5e56400640 1 -- 172.21.15.32:0/1022464977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500b72a0 con 0x7f5e5015e0b0 2024-03-06T07:36:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.909+0000 7f5e56400640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 0x7f5e5014fd60 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f5e48007e60 tx=0x7f5e4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.909+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48014420 con 0x7f5e5015e0b0 2024-03-06T07:36:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.909+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e48019070 con 0x7f5e5015e0b0 2024-03-06T07:36:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.909+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48014820 con 0x7f5e5015e0b0 2024-03-06T07:36:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.909+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e500b7580 con 0x7f5e5015e0b0 2024-03-06T07:36:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.909+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e500b7b40 con 0x7f5e5015e0b0 2024-03-06T07:36:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.910+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5004f330 con 0x7f5e5015e0b0 2024-03-06T07:36:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.911+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e480149c0 con 0x7f5e5015e0b0 2024-03-06T07:36:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.912+0000 7f5e4d600640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e3006b4d0 0x7f5e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.912+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5e4808ab90 con 0x7f5e5015e0b0 2024-03-06T07:36:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.913+0000 7f5e55a00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e3006b4d0 0x7f5e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.913+0000 7f5e55a00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e3006b4d0 0x7f5e3006d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001640 tx=0x7f5e44009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:31.921+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e480598e0 con 0x7f5e5015e0b0 2024-03-06T07:36:32.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.162+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e500d2af0 con 0x7f5e5015e0b0 2024-03-06T07:36:32.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.163+0000 7f5e4d600640 1 -- 172.21.15.32:0/1022464977 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f5e480591e0 con 0x7f5e5015e0b0 2024-03-06T07:36:32.166 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:32.166 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:32.166 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e3006b4d0 msgr2=0x7f5e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e3006b4d0 0x7f5e3006d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001640 tx=0x7f5e44009040 comp rx=0 tx=0).stop 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 msgr2=0x7f5e5014fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 0x7f5e5014fd60 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f5e48007e60 tx=0x7f5e4800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 shutdown_connections 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5e3006b4d0 0x7f5e3006d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e5015e0b0 0x7f5e5014fd60 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5e500d4a10 0x7f5e500b86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 --2- 172.21.15.32:0/1022464977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e500d0c80 0x7f5e500b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 >> 172.21.15.32:0/1022464977 conn(0x7f5e500d3d30 msgr2=0x7f5e500b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 shutdown_connections 2024-03-06T07:36:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.167+0000 7f5e56e00640 1 -- 172.21.15.32:0/1022464977 wait complete. 2024-03-06T07:36:32.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:36:32.418 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:36:32.418 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:36:32.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.626+0000 7f1f83200640 1 Processor -- start 2024-03-06T07:36:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.626+0000 7f1f83200640 1 -- start start 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.627+0000 7f1f83200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f840cd060 0x7f1f84161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.627+0000 7f1f83200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1f84162000 0x7f1f84070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.627+0000 7f1f83200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f841627f0 con 0x7f1f8415efc0 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.627+0000 7f1f83200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8415f3a0 con 0x7f1f840cd060 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.627+0000 7f1f83200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8406d190 con 0x7f1f84162000 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1f8415efc0 0x7f1f840cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57360/0 (socket says 172.21.15.32:57360) 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81e00640 1 -- 172.21.15.32:0/4078423845 learned_addr learned my addr 172.21.15.32:0/4078423845 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81400640 1 --2- 172.21.15.32:0/4078423845 >> v2:172.21.15.32:3300/0 conn(0x7f1f840cd060 0x7f1f84161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f82800640 1 --2- 172.21.15.32:0/4078423845 >> v2:172.21.15.39:3300/0 conn(0x7f1f84162000 0x7f1f84070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81400640 1 -- 172.21.15.32:0/4078423845 >> v2:172.21.15.39:3300/0 conn(0x7f1f84162000 msgr2=0x7f1f84070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81400640 1 --2- 172.21.15.32:0/4078423845 >> v2:172.21.15.39:3300/0 conn(0x7f1f84162000 0x7f1f84070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81400640 1 -- 172.21.15.32:0/4078423845 >> v1:172.21.15.32:6789/0 conn(0x7f1f8415efc0 legacy=0x7f1f840cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f81400640 1 -- 172.21.15.32:0/4078423845 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8406c860 con 0x7f1f840cd060 2024-03-06T07:36:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.628+0000 7f1f80a00640 1 -- 172.21.15.32:0/4078423845 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706429953 0 0) 0x7f1f841627f0 con 0x7f1f8415efc0 2024-03-06T07:36:32.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.629+0000 7f1f81400640 1 --2- 172.21.15.32:0/4078423845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840cd060 0x7f1f84161ac0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f1f840cf9e0 tx=0x7f1f7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efba15e985049a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f80a00640 1 -- 172.21.15.32:0/4078423845 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70035020 con 0x7f1f840cd060 2024-03-06T07:36:32.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f80a00640 1 -- 172.21.15.32:0/4078423845 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f70031830 con 0x7f1f840cd060 2024-03-06T07:36:32.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f80a00640 1 -- 172.21.15.32:0/4078423845 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70031c00 con 0x7f1f840cd060 2024-03-06T07:36:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f83200640 1 -- 172.21.15.32:0/4078423845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840cd060 msgr2=0x7f1f84161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f83200640 1 --2- 172.21.15.32:0/4078423845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840cd060 0x7f1f84161ac0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f1f840cf9e0 tx=0x7f1f7002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f83200640 1 -- 172.21.15.32:0/4078423845 shutdown_connections 2024-03-06T07:36:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f83200640 1 --2- 172.21.15.32:0/4078423845 >> v2:172.21.15.39:3300/0 conn(0x7f1f84162000 0x7f1f84070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f83200640 1 --2- 172.21.15.32:0/4078423845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f840cd060 0x7f1f84161ac0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.630+0000 7f1f83200640 1 -- 172.21.15.32:0/4078423845 >> 172.21.15.32:0/4078423845 conn(0x7f1f8406b3d0 msgr2=0x7f1f8406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.631+0000 7f1f83200640 1 -- 172.21.15.32:0/4078423845 shutdown_connections 2024-03-06T07:36:32.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.631+0000 7f1f83200640 1 -- 172.21.15.32:0/4078423845 wait complete. 2024-03-06T07:36:32.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.632+0000 7f1f83200640 1 Processor -- start 2024-03-06T07:36:32.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.632+0000 7f1f83200640 1 -- start start 2024-03-06T07:36:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.633+0000 7f1f83200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f840cd060 0x7f1f84147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.633+0000 7f1f83200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f8415efc0 0x7f1f840caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.633+0000 7f1f83200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 0x7f1f840cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.633+0000 7f1f83200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84162470 con 0x7f1f84162000 2024-03-06T07:36:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.633+0000 7f1f83200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8406ce20 con 0x7f1f840cd060 2024-03-06T07:36:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.633+0000 7f1f83200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8406d620 con 0x7f1f8415efc0 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 0x7f1f840cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 0x7f1f840cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33092/0 (socket says 172.21.15.32:33092) 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 -- 172.21.15.32:0/2920267206 learned_addr learned my addr 172.21.15.32:0/2920267206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f81e00640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f840cd060 0x7f1f84147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f81400640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f8415efc0 0x7f1f840caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 -- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f8415efc0 msgr2=0x7f1f840caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f8415efc0 0x7f1f840caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 -- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f840cd060 msgr2=0x7f1f84147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f840cd060 0x7f1f84147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f82800640 1 -- 172.21.15.32:0/2920267206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f841e8380 con 0x7f1f84162000 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.634+0000 7f1f81e00640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f840cd060 0x7f1f84147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.635+0000 7f1f82800640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 0x7f1f840cb490 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c007e60 tx=0x7f1f7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.635+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c014420 con 0x7f1f84162000 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.635+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c019070 con 0x7f1f84162000 2024-03-06T07:36:32.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.635+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c014820 con 0x7f1f84162000 2024-03-06T07:36:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.635+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f841e8660 con 0x7f1f84162000 2024-03-06T07:36:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.635+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f841e8c20 con 0x7f1f84162000 2024-03-06T07:36:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.636+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8404f330 con 0x7f1f84162000 2024-03-06T07:36:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.637+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f7c0149c0 con 0x7f1f84162000 2024-03-06T07:36:32.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.638+0000 7f1f7a000640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f6406b500 0x7f1f6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.638+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1f7c08ab60 con 0x7f1f84162000 2024-03-06T07:36:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.638+0000 7f1f81e00640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f6406b500 0x7f1f6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.639+0000 7f1f81e00640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f6406b500 0x7f1f6406d9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1f74000f60 tx=0x7f1f74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.643+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f7c0598b0 con 0x7f1f84162000 2024-03-06T07:36:32.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.882+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f8406ec60 con 0x7f1f84162000 2024-03-06T07:36:32.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.883+0000 7f1f7a000640 1 -- 172.21.15.32:0/2920267206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f1f7c0591b0 con 0x7f1f84162000 2024-03-06T07:36:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:32.886 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:32.886 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.885+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f6406b500 msgr2=0x7f1f6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.885+0000 7f1f83200640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f6406b500 0x7f1f6406d9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1f74000f60 tx=0x7f1f74009040 comp rx=0 tx=0).stop 2024-03-06T07:36:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.885+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 msgr2=0x7f1f840cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.885+0000 7f1f83200640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 0x7f1f840cb490 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c007e60 tx=0x7f1f7c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 shutdown_connections 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1f6406b500 0x7f1f6406d9c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84162000 0x7f1f840cb490 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1f8415efc0 0x7f1f840caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 --2- 172.21.15.32:0/2920267206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f840cd060 0x7f1f84147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 >> 172.21.15.32:0/2920267206 conn(0x7f1f8406b3d0 msgr2=0x7f1f840ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 shutdown_connections 2024-03-06T07:36:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:32.886+0000 7f1f83200640 1 -- 172.21.15.32:0/2920267206 wait complete. 2024-03-06T07:36:32.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:36:33.142 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:36:33.143 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:36:33.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:36:33.243 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:36:33.375 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:36:33.375 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:36:33.375 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:33.433 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:36:33.433 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:33.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.710+0000 7ff892000640 1 Processor -- start 2024-03-06T07:36:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.710+0000 7ff892000640 1 -- start start 2024-03-06T07:36:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.712+0000 7ff892000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 0x7ff88c0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.712+0000 7ff892000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0d2dc0 con 0x7ff88c15e070 2024-03-06T07:36:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.712+0000 7ff892000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0d18c0 con 0x7ff88c0d0b00 2024-03-06T07:36:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.712+0000 7ff892000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d1ba0 con 0x7ff88c0d2960 2024-03-06T07:36:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.712+0000 7ff890c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff88c15e070 0x7ff88c0cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57368/0 (socket says 172.21.15.32:57368) 2024-03-06T07:36:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.712+0000 7ff890c00640 1 -- 172.21.15.32:0/4080159825 learned_addr learned my addr 172.21.15.32:0/4080159825 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff883e00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 0x7ff88c0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148588293 0 0) 0x7ff88c0d2dc0 con 0x7ff88c15e070 2024-03-06T07:36:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870003520 con 0x7ff88c15e070 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97909506 0 0) 0x7ff870003520 con 0x7ff88c15e070 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 msgr2=0x7ff88c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 0x7ff88c0d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 >> v1:172.21.15.39:6789/0 conn(0x7ff88c0d0b00 legacy=0x7ff88c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0d1e80 con 0x7ff88c15e070 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff87c001670 con 0x7ff88c15e070 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 >> v1:172.21.15.32:6789/0 conn(0x7ff88c15e070 legacy=0x7ff88c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 0x7ff8700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870003f60 0x7ff870004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff870004d40 0x7ff8700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d1ba0 con 0x7ff870003f60 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87000baa0 con 0x7ff870004d40 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.713+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87000bca0 con 0x7ff88c0d2960 2024-03-06T07:36:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff883e00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870003f60 0x7ff870004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff890c00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 0x7ff8700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff883e00640 1 -- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 msgr2=0x7ff8700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff883e00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff88c0d2960 0x7ff8700039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff883e00640 1 -- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff870004d40 msgr2=0x7ff8700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff883e00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff870004d40 0x7ff8700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.714+0000 7ff883e00640 1 -- 172.21.15.32:0/4080159825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff870003520 con 0x7ff870003f60 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.715+0000 7ff883e00640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870003f60 0x7ff870004800 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff874009ac0 tx=0x7ff870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fd60ba0bbe7642 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874039070 con 0x7ff870003f60 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87402c6b0 con 0x7ff870003f60 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff88be00640 1 -- 172.21.15.32:0/4080159825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874009690 con 0x7ff870003f60 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff892000640 1 -- 172.21.15.32:0/4080159825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870003f60 msgr2=0x7ff870004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff892000640 1 --2- 172.21.15.32:0/4080159825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870003f60 0x7ff870004800 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff874009ac0 tx=0x7ff870005830 comp rx=0 tx=0).stop 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff891600640 1 -- 172.21.15.32:0/4080159825 reap_dead start 2024-03-06T07:36:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff892000640 1 -- 172.21.15.32:0/4080159825 shutdown_connections 2024-03-06T07:36:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff892000640 1 -- 172.21.15.32:0/4080159825 >> 172.21.15.32:0/4080159825 conn(0x7ff88c08d2a0 msgr2=0x7ff88c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff892000640 1 -- 172.21.15.32:0/4080159825 shutdown_connections 2024-03-06T07:36:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.716+0000 7ff892000640 1 -- 172.21.15.32:0/4080159825 wait complete. 2024-03-06T07:36:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.718+0000 7ff892000640 1 Processor -- start 2024-03-06T07:36:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.718+0000 7ff892000640 1 -- start start 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff892000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff870003f60 0x7ff88c0b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff892000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 0x7ff88c0cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff892000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88c151880 0x7ff88c151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff892000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d45b0 con 0x7ff88c0cadc0 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff892000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d2440 con 0x7ff88c151880 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff892000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0d4730 con 0x7ff870003f60 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.719+0000 7ff883e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 0x7ff88c0cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 0x7ff88c0cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33106/0 (socket says 172.21.15.32:33106) 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 -- 172.21.15.32:0/4052555345 learned_addr learned my addr 172.21.15.32:0/4052555345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff890c00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff870003f60 0x7ff88c0b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff891600640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88c151880 0x7ff88c151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 -- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff870003f60 msgr2=0x7ff88c0b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff870003f60 0x7ff88c0b8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 -- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88c151880 msgr2=0x7ff88c151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88c151880 0x7ff88c151d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 -- 172.21.15.32:0/4052555345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0cb680 con 0x7ff88c0cadc0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff890c00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff870003f60 0x7ff88c0b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.720+0000 7ff883e00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 0x7ff88c0cd110 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff8740099e0 tx=0x7ff874004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.721+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874042020 con 0x7ff88c0cadc0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.721+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87403a040 con 0x7ff88c0cadc0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.721+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874039070 con 0x7ff88c0cadc0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.721+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c0cb9c0 con 0x7ff88c0cadc0 2024-03-06T07:36:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.721+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c0cc200 con 0x7ff88c0cadc0 2024-03-06T07:36:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.722+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c04f330 con 0x7ff88c0cadc0 2024-03-06T07:36:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.723+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff874045020 con 0x7ff88c0cadc0 2024-03-06T07:36:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.724+0000 7ff889600640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff86806b550 0x7ff86806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.724+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff8740ab750 con 0x7ff88c0cadc0 2024-03-06T07:36:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.724+0000 7ff890c00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff86806b550 0x7ff86806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.725+0000 7ff890c00640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff86806b550 0x7ff86806da10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff87c001eb0 tx=0x7ff87c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.729+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff87404c070 con 0x7ff88c0cadc0 2024-03-06T07:36:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.939+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff88c0c75c0 con 0x7ff88c0cadc0 2024-03-06T07:36:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.943+0000 7ff889600640 1 -- 172.21.15.32:0/4052555345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16738 (secure 0 0 0) 0x7ff88c0cb9c0 con 0x7ff88c0cadc0 2024-03-06T07:36:33.946 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:33.946 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":116,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:49.005590+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.18","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/14889266":"2024-03-07T07:35:30.884623+0000","172.21.15.39:6833/14889266":"2024-03-07T07:35:30.884623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff86806b550 msgr2=0x7ff86806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff86806b550 0x7ff86806da10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff87c001eb0 tx=0x7ff87c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 msgr2=0x7ff88c0cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 0x7ff88c0cd110 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff8740099e0 tx=0x7ff874004730 comp rx=0 tx=0).stop 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 shutdown_connections 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff86806b550 0x7ff86806da10 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88c151880 0x7ff88c151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff88c0cadc0 0x7ff88c0cd110 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 --2- 172.21.15.32:0/4052555345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff870003f60 0x7ff88c0b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 >> 172.21.15.32:0/4052555345 conn(0x7ff88c08d2a0 msgr2=0x7ff88c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 shutdown_connections 2024-03-06T07:36:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:33.947+0000 7ff892000640 1 -- 172.21.15.32:0/4052555345 wait complete. 2024-03-06T07:36:33.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:12.716010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:13.860496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:36:33.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:36:34.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.188+0000 7efd77200640 1 Processor -- start 2024-03-06T07:36:34.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.188+0000 7efd77200640 1 -- start start 2024-03-06T07:36:34.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.191+0000 7efd77200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efd7806f540 0x7efd7806f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.191+0000 7efd77200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd780af930 0x7efd780cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.191+0000 7efd77200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7806fe60 con 0x7efd780b4530 2024-03-06T07:36:34.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.192+0000 7efd77200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd780b3520 con 0x7efd780af930 2024-03-06T07:36:34.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.192+0000 7efd77200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd780b3770 con 0x7efd7806f540 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.192+0000 7efd75400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd780af930 0x7efd780cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.192+0000 7efd76800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efd780b4530 0x7efd780b49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57372/0 (socket says 172.21.15.32:57372) 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.192+0000 7efd76800640 1 -- 172.21.15.32:0/692622337 learned_addr learned my addr 172.21.15.32:0/692622337 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.192+0000 7efd75e00640 1 --2- 172.21.15.32:0/692622337 >> v2:172.21.15.39:3300/0 conn(0x7efd7806f540 0x7efd7806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.193+0000 7efd75400640 1 -- 172.21.15.32:0/692622337 >> v2:172.21.15.39:3300/0 conn(0x7efd7806f540 msgr2=0x7efd7806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.193+0000 7efd75400640 1 --2- 172.21.15.32:0/692622337 >> v2:172.21.15.39:3300/0 conn(0x7efd7806f540 0x7efd7806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.193+0000 7efd75400640 1 -- 172.21.15.32:0/692622337 >> v1:172.21.15.32:6789/0 conn(0x7efd780b4530 legacy=0x7efd780b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.193+0000 7efd75400640 1 -- 172.21.15.32:0/692622337 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780b3a50 con 0x7efd780af930 2024-03-06T07:36:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.194+0000 7efd75400640 1 --2- 172.21.15.32:0/692622337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd780cfa90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7efd7804f9c0 tx=0x7efd6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e055ed34d37159 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.194+0000 7efd74a00640 1 -- 172.21.15.32:0/692622337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64035020 con 0x7efd780af930 2024-03-06T07:36:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.194+0000 7efd74a00640 1 -- 172.21.15.32:0/692622337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd64031830 con 0x7efd780af930 2024-03-06T07:36:34.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.194+0000 7efd74a00640 1 -- 172.21.15.32:0/692622337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64031c00 con 0x7efd780af930 2024-03-06T07:36:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 -- 172.21.15.32:0/692622337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 msgr2=0x7efd780cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 --2- 172.21.15.32:0/692622337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd780cfa90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7efd7804f9c0 tx=0x7efd6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 -- 172.21.15.32:0/692622337 shutdown_connections 2024-03-06T07:36:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 --2- 172.21.15.32:0/692622337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd780cfa90 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 --2- 172.21.15.32:0/692622337 >> v2:172.21.15.39:3300/0 conn(0x7efd7806f540 0x7efd7806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 -- 172.21.15.32:0/692622337 >> 172.21.15.32:0/692622337 conn(0x7efd780723d0 msgr2=0x7efd7806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 -- 172.21.15.32:0/692622337 shutdown_connections 2024-03-06T07:36:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.195+0000 7efd77200640 1 -- 172.21.15.32:0/692622337 wait complete. 2024-03-06T07:36:34.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.197+0000 7efd77200640 1 Processor -- start 2024-03-06T07:36:34.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.197+0000 7efd77200640 1 -- start start 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd77200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd7806f540 0x7efd780c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd77200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd781514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd77200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd780b4530 0x7efd781f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd77200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd780cffd0 con 0x7efd780af930 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd77200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd780b4010 con 0x7efd780b4530 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd77200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7815cf00 con 0x7efd7806f540 2024-03-06T07:36:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd75400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd781514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd75400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd781514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33116/0 (socket says 172.21.15.32:33116) 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd75400640 1 -- 172.21.15.32:0/3381706825 learned_addr learned my addr 172.21.15.32:0/3381706825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd75e00640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd7806f540 0x7efd780c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.198+0000 7efd76800640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd780b4530 0x7efd781f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd75400640 1 -- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd7806f540 msgr2=0x7efd780c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd75400640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd7806f540 0x7efd780c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd75400640 1 -- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd780b4530 msgr2=0x7efd781f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd75400640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd780b4530 0x7efd781f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd75400640 1 -- 172.21.15.32:0/3381706825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd781f8260 con 0x7efd780af930 2024-03-06T07:36:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd75400640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd781514e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7efd64000c00 tx=0x7efd640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6403b9c0 con 0x7efd780af930 2024-03-06T07:36:34.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6403bb80 con 0x7efd780af930 2024-03-06T07:36:34.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd640453f0 con 0x7efd780af930 2024-03-06T07:36:34.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.199+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd780cb370 con 0x7efd780af930 2024-03-06T07:36:34.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.200+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd780cb930 con 0x7efd780af930 2024-03-06T07:36:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.201+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7814e6c0 con 0x7efd780af930 2024-03-06T07:36:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.202+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd64038020 con 0x7efd780af930 2024-03-06T07:36:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.202+0000 7efd6e000640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd5806b4d0 0x7efd5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.203+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7efd640abfb0 con 0x7efd780af930 2024-03-06T07:36:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.203+0000 7efd75e00640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd5806b4d0 0x7efd5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.204+0000 7efd75e00640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd5806b4d0 0x7efd5806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efd68006fd0 tx=0x7efd68008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.208+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd7814e6c0 con 0x7efd780af930 2024-03-06T07:36:34.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.461+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd7804f330 con 0x7efd780af930 2024-03-06T07:36:34.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.463+0000 7efd6e000640 1 -- 172.21.15.32:0/3381706825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd781f8260 con 0x7efd780af930 2024-03-06T07:36:34.465 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:36:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.465+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd5806b4d0 msgr2=0x7efd5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.465+0000 7efd77200640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd5806b4d0 0x7efd5806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efd68006fd0 tx=0x7efd68008040 comp rx=0 tx=0).stop 2024-03-06T07:36:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.465+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 msgr2=0x7efd781514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.465+0000 7efd77200640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd781514e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7efd64000c00 tx=0x7efd640046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.465+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 shutdown_connections 2024-03-06T07:36:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.465+0000 7efd77200640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efd5806b4d0 0x7efd5806d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.466+0000 7efd77200640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd780b4530 0x7efd781f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.466+0000 7efd77200640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd780af930 0x7efd781514e0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.466+0000 7efd77200640 1 --2- 172.21.15.32:0/3381706825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efd7806f540 0x7efd780c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.466+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 >> 172.21.15.32:0/3381706825 conn(0x7efd780723d0 msgr2=0x7efd781580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.466+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 shutdown_connections 2024-03-06T07:36:34.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.466+0000 7efd77200640 1 -- 172.21.15.32:0/3381706825 wait complete. 2024-03-06T07:36:34.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:36:34.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.702+0000 7f19faa00640 1 Processor -- start 2024-03-06T07:36:34.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.702+0000 7f19faa00640 1 -- start start 2024-03-06T07:36:34.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.704+0000 7f19faa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19f40ce6e0 0x7f19f40c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.704+0000 7f19faa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f406dc50 con 0x7f19f406afb0 2024-03-06T07:36:34.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.704+0000 7f19faa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f406df30 con 0x7f19f40c75b0 2024-03-06T07:36:34.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.704+0000 7f19faa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f406e210 con 0x7f19f40ce6e0 2024-03-06T07:36:34.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.705+0000 7f19f3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19f40ce6e0 0x7f19f40c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.705+0000 7f19f3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19f40ce6e0 0x7f19f40c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33118/0 (socket says 172.21.15.32:33118) 2024-03-06T07:36:34.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.705+0000 7f19f3400640 1 -- 172.21.15.32:0/2927374518 learned_addr learned my addr 172.21.15.32:0/2927374518 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:34.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.705+0000 7f19f3400640 1 -- 172.21.15.32:0/2927374518 >> v1:172.21.15.39:6789/0 conn(0x7f19f40c75b0 legacy=0x7f19f406ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.705+0000 7f19f3400640 1 -- 172.21.15.32:0/2927374518 >> v1:172.21.15.32:6789/0 conn(0x7f19f406afb0 legacy=0x7f19f406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.705+0000 7f19f3400640 1 -- 172.21.15.32:0/2927374518 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f406e4f0 con 0x7f19f40ce6e0 2024-03-06T07:36:34.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.707+0000 7f19f3400640 1 --2- 172.21.15.32:0/2927374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f40ce6e0 0x7f19f40c7070 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009870 tx=0x7f19e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e7cc426888d2e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.707+0000 7f19f2a00640 1 -- 172.21.15.32:0/2927374518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0035020 con 0x7f19f40ce6e0 2024-03-06T07:36:34.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.707+0000 7f19f2a00640 1 -- 172.21.15.32:0/2927374518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0031870 con 0x7f19f40ce6e0 2024-03-06T07:36:34.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.707+0000 7f19f2a00640 1 -- 172.21.15.32:0/2927374518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0031c40 con 0x7f19f40ce6e0 2024-03-06T07:36:34.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 -- 172.21.15.32:0/2927374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f40ce6e0 msgr2=0x7f19f40c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 --2- 172.21.15.32:0/2927374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f40ce6e0 0x7f19f40c7070 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009870 tx=0x7f19e002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:34.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 -- 172.21.15.32:0/2927374518 shutdown_connections 2024-03-06T07:36:34.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 --2- 172.21.15.32:0/2927374518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f40ce6e0 0x7f19f40c7070 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 -- 172.21.15.32:0/2927374518 >> 172.21.15.32:0/2927374518 conn(0x7f19f40b2760 msgr2=0x7f19f40b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 -- 172.21.15.32:0/2927374518 shutdown_connections 2024-03-06T07:36:34.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.708+0000 7f19faa00640 1 -- 172.21.15.32:0/2927374518 wait complete. 2024-03-06T07:36:34.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.710+0000 7f19faa00640 1 Processor -- start 2024-03-06T07:36:34.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.710+0000 7f19faa00640 1 -- start start 2024-03-06T07:36:34.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19faa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 0x7f19f4161cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19faa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19f40c75b0 0x7f19f414f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19faa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19f40ce6e0 0x7f19f414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19faa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f415f230 con 0x7f19f406afb0 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19faa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f415e2a0 con 0x7f19f40c75b0 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19faa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f415f3b0 con 0x7f19f40ce6e0 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 0x7f19f4161cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 0x7f19f4161cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33120/0 (socket says 172.21.15.32:33120) 2024-03-06T07:36:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.711+0000 7f19f3e00640 1 -- 172.21.15.32:0/3322718886 learned_addr learned my addr 172.21.15.32:0/3322718886 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3400640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19f40c75b0 0x7f19f414f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f8a00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19f40ce6e0 0x7f19f414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3e00640 1 -- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19f40ce6e0 msgr2=0x7f19f414fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3e00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19f40ce6e0 0x7f19f414fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3e00640 1 -- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19f40c75b0 msgr2=0x7f19f414f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3e00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19f40c75b0 0x7f19f414f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3e00640 1 -- 172.21.15.32:0/3322718886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f40b4760 con 0x7f19f406afb0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f8a00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19f40ce6e0 0x7f19f414fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3400640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19f40c75b0 0x7f19f414f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.712+0000 7f19f3e00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 0x7f19f4161cc0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f19e400c3c0 tx=0x7f19e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.713+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e40052d0 con 0x7f19f406afb0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.713+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e4005490 con 0x7f19f406afb0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.713+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e4007260 con 0x7f19f406afb0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.713+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f40b4a40 con 0x7f19f406afb0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.713+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f40b5000 con 0x7f19f406afb0 2024-03-06T07:36:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.714+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f404f330 con 0x7f19f406afb0 2024-03-06T07:36:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.715+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19e4007400 con 0x7f19f406afb0 2024-03-06T07:36:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.716+0000 7f19ebe00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.716+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f19e4089840 con 0x7f19f406afb0 2024-03-06T07:36:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.716+0000 7f19f3400640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.716+0000 7f19f3400640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19d406b4d0 0x7f19d406d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.720+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19e4058590 con 0x7f19f406afb0 2024-03-06T07:36:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.915+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19f40b21c0 con 0x7f19f406afb0 2024-03-06T07:36:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.916+0000 7f19ebe00640 1 -- 172.21.15.32:0/3322718886 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f19e4057e90 con 0x7f19f406afb0 2024-03-06T07:36:34.919 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:36:34.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19d406b4d0 msgr2=0x7f19d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19d406b4d0 0x7f19d406d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003c040 comp rx=0 tx=0).stop 2024-03-06T07:36:34.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 msgr2=0x7f19f4161cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 0x7f19f4161cc0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f19e400c3c0 tx=0x7f19e400c880 comp rx=0 tx=0).stop 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 shutdown_connections 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19f40ce6e0 0x7f19f414fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19f40c75b0 0x7f19f414f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 --2- 172.21.15.32:0/3322718886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19f406afb0 0x7f19f4161cc0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 >> 172.21.15.32:0/3322718886 conn(0x7f19f40b2760 msgr2=0x7f19f41606d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 shutdown_connections 2024-03-06T07:36:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:34.919+0000 7f19faa00640 1 -- 172.21.15.32:0/3322718886 wait complete. 2024-03-06T07:36:34.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:36:35.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.210+0000 7f765f200640 1 Processor -- start 2024-03-06T07:36:35.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.211+0000 7f765f200640 1 -- start start 2024-03-06T07:36:35.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765f200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7660147590 0x7f7660071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f766006d8d0 0x7f76600b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600b2490 con 0x7f76600b2080 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765f200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7660147970 con 0x7f7660147590 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76600b31b0 con 0x7f766006d8d0 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7660147590 0x7f7660071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7660147590 0x7f7660071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33126/0 (socket says 172.21.15.32:33126) 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f765de00640 1 -- 172.21.15.32:0/2015920541 learned_addr learned my addr 172.21.15.32:0/2015920541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.212+0000 7f7655400640 1 --2- 172.21.15.32:0/2015920541 >> v2:172.21.15.39:3300/0 conn(0x7f766006d8d0 0x7f76600b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.213+0000 7f765de00640 1 -- 172.21.15.32:0/2015920541 >> v2:172.21.15.39:3300/0 conn(0x7f766006d8d0 msgr2=0x7f76600b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.213+0000 7f765de00640 1 --2- 172.21.15.32:0/2015920541 >> v2:172.21.15.39:3300/0 conn(0x7f766006d8d0 0x7f76600b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.213+0000 7f765de00640 1 -- 172.21.15.32:0/2015920541 >> v1:172.21.15.39:6789/0 conn(0x7f76600b2080 legacy=0x7f76600ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.213+0000 7f765de00640 1 -- 172.21.15.32:0/2015920541 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600719c0 con 0x7f7660147590 2024-03-06T07:36:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.214+0000 7f765de00640 1 --2- 172.21.15.32:0/2015920541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f7660071480 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f76600af690 tx=0x7f765002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a544b493effa98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.214+0000 7f765d400640 1 -- 172.21.15.32:0/2015920541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650034070 con 0x7f7660147590 2024-03-06T07:36:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.214+0000 7f765d400640 1 -- 172.21.15.32:0/2015920541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7650030860 con 0x7f7660147590 2024-03-06T07:36:35.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.214+0000 7f765d400640 1 -- 172.21.15.32:0/2015920541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650030c30 con 0x7f7660147590 2024-03-06T07:36:35.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 -- 172.21.15.32:0/2015920541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 msgr2=0x7f7660071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 --2- 172.21.15.32:0/2015920541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f7660071480 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f76600af690 tx=0x7f765002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 -- 172.21.15.32:0/2015920541 shutdown_connections 2024-03-06T07:36:35.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 --2- 172.21.15.32:0/2015920541 >> v2:172.21.15.39:3300/0 conn(0x7f766006d8d0 0x7f76600b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 --2- 172.21.15.32:0/2015920541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f7660071480 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 -- 172.21.15.32:0/2015920541 >> 172.21.15.32:0/2015920541 conn(0x7f766006b3a0 msgr2=0x7f766006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:35.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 -- 172.21.15.32:0/2015920541 shutdown_connections 2024-03-06T07:36:35.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.215+0000 7f765f200640 1 -- 172.21.15.32:0/2015920541 wait complete. 2024-03-06T07:36:35.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.217+0000 7f765f200640 1 Processor -- start 2024-03-06T07:36:35.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.217+0000 7f765f200640 1 -- start start 2024-03-06T07:36:35.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.218+0000 7f765f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f766006d8d0 0x7f7660143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.218+0000 7f765f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76600b2080 0x7f76601f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.218+0000 7f765f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f76601448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.218+0000 7f765f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600b2e40 con 0x7f7660147590 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.218+0000 7f765f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766006e120 con 0x7f766006d8d0 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.218+0000 7f765f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600b2b50 con 0x7f76600b2080 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f76601448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f76601448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33140/0 (socket says 172.21.15.32:33140) 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 -- 172.21.15.32:0/1267513889 learned_addr learned my addr 172.21.15.32:0/1267513889 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765de00640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f766006d8d0 0x7f7660143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f7655400640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76600b2080 0x7f76601f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 -- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76600b2080 msgr2=0x7f76601f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76600b2080 0x7f76601f7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 -- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f766006d8d0 msgr2=0x7f7660143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f766006d8d0 0x7f7660143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 -- 172.21.15.32:0/1267513889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600c85a0 con 0x7f7660147590 2024-03-06T07:36:35.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.219+0000 7f765e800640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f76601448f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f765800c7a0 tx=0x7f765800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.220+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658003b70 con 0x7f7660147590 2024-03-06T07:36:35.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.220+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658003d30 con 0x7f7660147590 2024-03-06T07:36:35.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.220+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76600c8880 con 0x7f7660147590 2024-03-06T07:36:35.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.220+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765800aa90 con 0x7f7660147590 2024-03-06T07:36:35.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.220+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76600c8e40 con 0x7f7660147590 2024-03-06T07:36:35.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.221+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766004f330 con 0x7f7660147590 2024-03-06T07:36:35.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.222+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f765800ac30 con 0x7f7660147590 2024-03-06T07:36:35.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.223+0000 7f7656a00640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f763c06b4d0 0x7f763c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.223+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f765808a1e0 con 0x7f7660147590 2024-03-06T07:36:35.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.223+0000 7f765de00640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f763c06b4d0 0x7f763c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.224+0000 7f765de00640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f763c06b4d0 0x7f763c06d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7650036000 tx=0x7f765003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.227+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76600b2490 con 0x7f7660147590 2024-03-06T07:36:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.432+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f76600c9640 con 0x7f7660147590 2024-03-06T07:36:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.432+0000 7f7656a00640 1 -- 172.21.15.32:0/1267513889 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7658058fc0 con 0x7f7660147590 2024-03-06T07:36:35.435 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:36:35.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f763c06b4d0 msgr2=0x7f763c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f763c06b4d0 0x7f763c06d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7650036000 tx=0x7f765003b040 comp rx=0 tx=0).stop 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 msgr2=0x7f76601448f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f76601448f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f765800c7a0 tx=0x7f765800cc60 comp rx=0 tx=0).stop 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 shutdown_connections 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f763c06b4d0 0x7f763c06d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7660147590 0x7f76601448f0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76600b2080 0x7f76601f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 --2- 172.21.15.32:0/1267513889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f766006d8d0 0x7f7660143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 >> 172.21.15.32:0/1267513889 conn(0x7f766006b3a0 msgr2=0x7f766006b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 shutdown_connections 2024-03-06T07:36:35.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.436+0000 7f765f200640 1 -- 172.21.15.32:0/1267513889 wait complete. 2024-03-06T07:36:35.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:36:35.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.669+0000 7fe52da00640 1 Processor -- start 2024-03-06T07:36:35.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.670+0000 7fe52da00640 1 -- start start 2024-03-06T07:36:35.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe52da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5280cd770 0x7fe5280cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe52da00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280ce090 con 0x7fe52815bb20 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe52da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52815c340 con 0x7fe52815f810 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe52da00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52815eae0 con 0x7fe5280cd770 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe526a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5280cd770 0x7fe5280cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe526a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5280cd770 0x7fe5280cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33144/0 (socket says 172.21.15.32:33144) 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.672+0000 7fe526a00640 1 -- 172.21.15.32:0/2570790070 learned_addr learned my addr 172.21.15.32:0/2570790070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.673+0000 7fe526a00640 1 -- 172.21.15.32:0/2570790070 >> v1:172.21.15.39:6789/0 conn(0x7fe52815f810 legacy=0x7fe52815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:35.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.673+0000 7fe526a00640 1 -- 172.21.15.32:0/2570790070 >> v1:172.21.15.32:6789/0 conn(0x7fe52815bb20 legacy=0x7fe5280cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.673+0000 7fe526a00640 1 -- 172.21.15.32:0/2570790070 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52815ed30 con 0x7fe5280cd770 2024-03-06T07:36:35.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.674+0000 7fe526a00640 1 --2- 172.21.15.32:0/2570790070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe5280cdb50 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe51802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9e9a561ad4a687 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.674+0000 7fe525600640 1 -- 172.21.15.32:0/2570790070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518034070 con 0x7fe5280cd770 2024-03-06T07:36:35.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.674+0000 7fe525600640 1 -- 172.21.15.32:0/2570790070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518030860 con 0x7fe5280cd770 2024-03-06T07:36:35.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.674+0000 7fe525600640 1 -- 172.21.15.32:0/2570790070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518030c30 con 0x7fe5280cd770 2024-03-06T07:36:35.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 -- 172.21.15.32:0/2570790070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 msgr2=0x7fe5280cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 --2- 172.21.15.32:0/2570790070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe5280cdb50 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe51802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 -- 172.21.15.32:0/2570790070 shutdown_connections 2024-03-06T07:36:35.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 --2- 172.21.15.32:0/2570790070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe5280cdb50 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 -- 172.21.15.32:0/2570790070 >> 172.21.15.32:0/2570790070 conn(0x7fe5280d0d80 msgr2=0x7fe5280d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:35.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 -- 172.21.15.32:0/2570790070 shutdown_connections 2024-03-06T07:36:35.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.675+0000 7fe52da00640 1 -- 172.21.15.32:0/2570790070 wait complete. 2024-03-06T07:36:35.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.677+0000 7fe52da00640 1 Processor -- start 2024-03-06T07:36:35.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.677+0000 7fe52da00640 1 -- start start 2024-03-06T07:36:35.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe52da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe528159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe52da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe52815bb20 0x7fe5281560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe52da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe52815f810 0x7fe52809e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe52da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52815bf90 con 0x7fe5280cd770 2024-03-06T07:36:35.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe52da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528070630 con 0x7fe52815f810 2024-03-06T07:36:35.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe52da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52809dc40 con 0x7fe52815bb20 2024-03-06T07:36:35.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe526a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe528159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe526a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe528159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33156/0 (socket says 172.21.15.32:33156) 2024-03-06T07:36:35.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.678+0000 7fe526a00640 1 -- 172.21.15.32:0/3970492770 learned_addr learned my addr 172.21.15.32:0/3970492770 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe527400640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe52815f810 0x7fe52809e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526000640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe52815bb20 0x7fe5281560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526a00640 1 -- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe52815bb20 msgr2=0x7fe5281560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526a00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe52815bb20 0x7fe5281560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526a00640 1 -- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe52815f810 msgr2=0x7fe52809e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526a00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe52815f810 0x7fe52809e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526a00640 1 -- 172.21.15.32:0/3970492770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52809d610 con 0x7fe5280cd770 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.679+0000 7fe526a00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe528159150 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe5180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.680+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518034040 con 0x7fe5280cd770 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.680+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5180433f0 con 0x7fe5280cd770 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.680+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe528149c20 con 0x7fe5280cd770 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.680+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180437e0 con 0x7fe5280cd770 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.680+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52809d8d0 con 0x7fe5280cd770 2024-03-06T07:36:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.681+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5280b5950 con 0x7fe5280cd770 2024-03-06T07:36:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.682+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe51804a060 con 0x7fe5280cd770 2024-03-06T07:36:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.683+0000 7fe51ea00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50806b4d0 0x7fe50806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.683+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe5180ac650 con 0x7fe5280cd770 2024-03-06T07:36:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.683+0000 7fe526000640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50806b4d0 0x7fe50806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.684+0000 7fe526000640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50806b4d0 0x7fe50806d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe52815abc0 tx=0x7fe514009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.688+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5280b5950 con 0x7fe5280cd770 2024-03-06T07:36:35.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.902+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe52804f3a0 con 0x7fe5280cd770 2024-03-06T07:36:35.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.903+0000 7fe51ea00640 1 -- 172.21.15.32:0/3970492770 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v116) v1 ==== 80+0+118 (secure 0 0 0) 0x7fe51803e070 con 0x7fe5280cd770 2024-03-06T07:36:35.905 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.39:6835/14889266 2024-03-07T07:35:30.884623+0000 2024-03-06T07:36:35.905 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.39:6833/14889266 2024-03-07T07:35:30.884623+0000 2024-03-06T07:36:35.905 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.905+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50806b4d0 msgr2=0x7fe50806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.905+0000 7fe52da00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50806b4d0 0x7fe50806d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe52815abc0 tx=0x7fe514009040 comp rx=0 tx=0).stop 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.905+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 msgr2=0x7fe528159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.905+0000 7fe52da00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe528159150 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe5180046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 shutdown_connections 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe50806b4d0 0x7fe50806d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe52815f810 0x7fe52809e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe52815bb20 0x7fe5281560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 --2- 172.21.15.32:0/3970492770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5280cd770 0x7fe528159150 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 >> 172.21.15.32:0/3970492770 conn(0x7fe5280d0d80 msgr2=0x7fe52806b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 shutdown_connections 2024-03-06T07:36:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:35.906+0000 7fe52da00640 1 -- 172.21.15.32:0/3970492770 wait complete. 2024-03-06T07:36:35.927 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:36:35.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:35.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:36:36.029 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-03-06T07:36:36.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:36.030 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:36:36.076 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:36:36.076 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:36.138 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:36:36.138 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:36:36.139 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:36.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:36:36.198 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:36:36.198 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:36.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:36:36.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.483+0000 7f9f94e00640 1 Processor -- start 2024-03-06T07:36:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.483+0000 7f9f94e00640 1 -- start start 2024-03-06T07:36:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.486+0000 7f9f94e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9f900cfd70 0x7f9f9006e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.486+0000 7f9f94e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9006d620 con 0x7f9f9006bfc0 2024-03-06T07:36:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.487+0000 7f9f94e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9006d900 con 0x7f9f900afa60 2024-03-06T07:36:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.487+0000 7f9f94e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9006dbe0 con 0x7f9f900cfd70 2024-03-06T07:36:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.487+0000 7f9f8de00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9f9006bfc0 0x7f9f9006c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57414/0 (socket says 172.21.15.32:57414) 2024-03-06T07:36:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.487+0000 7f9f8de00640 1 -- 172.21.15.32:0/932899176 learned_addr learned my addr 172.21.15.32:0/932899176 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.487+0000 7f9f8e800640 1 --2- 172.21.15.32:0/932899176 >> v2:172.21.15.32:3300/0 conn(0x7f9f900cfd70 0x7f9f9006e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.488+0000 7f9f8e800640 1 -- 172.21.15.32:0/932899176 >> v1:172.21.15.39:6789/0 conn(0x7f9f900afa60 legacy=0x7f9f9015c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.488+0000 7f9f8e800640 1 -- 172.21.15.32:0/932899176 >> v1:172.21.15.32:6789/0 conn(0x7f9f9006bfc0 legacy=0x7f9f9006c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.488+0000 7f9f8e800640 1 -- 172.21.15.32:0/932899176 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9006dec0 con 0x7f9f900cfd70 2024-03-06T07:36:36.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.489+0000 7f9f8e800640 1 --2- 172.21.15.32:0/932899176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f9006e6c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009d90 tx=0x7f9f8802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa9d4bffcf81947 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.489+0000 7f9f8ca00640 1 -- 172.21.15.32:0/932899176 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88039070 con 0x7f9f900cfd70 2024-03-06T07:36:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f8ca00640 1 -- 172.21.15.32:0/932899176 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8802cbd0 con 0x7f9f900cfd70 2024-03-06T07:36:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f8ca00640 1 -- 172.21.15.32:0/932899176 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88034440 con 0x7f9f900cfd70 2024-03-06T07:36:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 -- 172.21.15.32:0/932899176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 msgr2=0x7f9f9006e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 --2- 172.21.15.32:0/932899176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f9006e6c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009d90 tx=0x7f9f8802be20 comp rx=0 tx=0).stop 2024-03-06T07:36:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 -- 172.21.15.32:0/932899176 shutdown_connections 2024-03-06T07:36:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 --2- 172.21.15.32:0/932899176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f9006e6c0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 -- 172.21.15.32:0/932899176 >> 172.21.15.32:0/932899176 conn(0x7f9f90071490 msgr2=0x7f9f900718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 -- 172.21.15.32:0/932899176 shutdown_connections 2024-03-06T07:36:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.490+0000 7f9f94e00640 1 -- 172.21.15.32:0/932899176 wait complete. 2024-03-06T07:36:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.492+0000 7f9f94e00640 1 Processor -- start 2024-03-06T07:36:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.492+0000 7f9f94e00640 1 -- start start 2024-03-06T07:36:36.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f94e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f9006bfc0 0x7f9f901f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f94e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f900afa60 0x7f9f90154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f94e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f90143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f94e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9015c8a0 con 0x7f9f900cfd70 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f94e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9015c720 con 0x7f9f9006bfc0 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f94e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900671f0 con 0x7f9f900afa60 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.493+0000 7f9f8e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f90143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f90143eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33174/0 (socket says 172.21.15.32:33174) 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 -- 172.21.15.32:0/2411368973 learned_addr learned my addr 172.21.15.32:0/2411368973 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8de00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f9006bfc0 0x7f9f901f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8d400640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f900afa60 0x7f9f90154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 -- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f900afa60 msgr2=0x7f9f90154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f900afa60 0x7f9f90154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 -- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f9006bfc0 msgr2=0x7f9f901f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f9006bfc0 0x7f9f901f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 -- 172.21.15.32:0/2411368973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f90154550 con 0x7f9f900cfd70 2024-03-06T07:36:36.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8de00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f9006bfc0 0x7f9f901f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:36.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8d400640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f900afa60 0x7f9f90154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:36.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.494+0000 7f9f8e800640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f90143eb0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009d90 tx=0x7f9f88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.495+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88039050 con 0x7f9f900cfd70 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.495+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f88045070 con 0x7f9f900cfd70 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.495+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8803ea50 con 0x7f9f900cfd70 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.495+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f901f23f0 con 0x7f9f900cfd70 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.495+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f901f29b0 con 0x7f9f900cfd70 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.496+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9004f3a0 con 0x7f9f900cfd70 2024-03-06T07:36:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.497+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f8802ca70 con 0x7f9f900cfd70 2024-03-06T07:36:36.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.498+0000 7f9f86000640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f7006b4d0 0x7f9f7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.498+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9f880ad3d0 con 0x7f9f900cfd70 2024-03-06T07:36:36.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.498+0000 7f9f8de00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f7006b4d0 0x7f9f7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.499+0000 7f9f8de00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f7006b4d0 0x7f9f7006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9f80005880 tx=0x7f9f80009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.503+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f8807c240 con 0x7f9f900cfd70 2024-03-06T07:36:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.770+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f900b1a00 con 0x7f9f900cfd70 2024-03-06T07:36:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.771+0000 7f9f86000640 1 -- 172.21.15.32:0/2411368973 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f9f8807bb40 con 0x7f9f900cfd70 2024-03-06T07:36:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:36.774 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f7006b4d0 msgr2=0x7f9f7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f7006b4d0 0x7f9f7006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9f80005880 tx=0x7f9f80009040 comp rx=0 tx=0).stop 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 msgr2=0x7f9f90143eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f90143eb0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009d90 tx=0x7f9f88004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 shutdown_connections 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9f7006b4d0 0x7f9f7006d990 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f900cfd70 0x7f9f90143eb0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9f900afa60 0x7f9f90154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 --2- 172.21.15.32:0/2411368973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f9006bfc0 0x7f9f901f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.774+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 >> 172.21.15.32:0/2411368973 conn(0x7f9f90071490 msgr2=0x7f9f900ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.775+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 shutdown_connections 2024-03-06T07:36:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:36.775+0000 7f9f94e00640 1 -- 172.21.15.32:0/2411368973 wait complete. 2024-03-06T07:36:36.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.007+0000 7fc45a600640 1 Processor -- start 2024-03-06T07:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.007+0000 7fc45a600640 1 -- start start 2024-03-06T07:36:37.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc45a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 0x7fc4540cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc45a600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc4540d4a10 0x7fc4540d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc45a600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4540b3710 con 0x7fc4540d0c80 2024-03-06T07:36:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc45a600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4540b09e0 con 0x7fc4540d4a10 2024-03-06T07:36:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc45a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b39f0 con 0x7fc45415e0b0 2024-03-06T07:36:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc459c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc4540d0c80 0x7fc4540d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57424/0 (socket says 172.21.15.32:57424) 2024-03-06T07:36:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.009+0000 7fc459c00640 1 -- 172.21.15.32:0/2547978520 learned_addr learned my addr 172.21.15.32:0/2547978520 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453e00640 1 --2- 172.21.15.32:0/2547978520 >> v2:172.21.15.39:3300/0 conn(0x7fc4540d4a10 0x7fc4540d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc459200640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 0x7fc4540cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767127861 0 0) 0x7fc4540b3710 con 0x7fc4540d0c80 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c003520 con 0x7fc4540d0c80 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116777142 0 0) 0x7fc43c003520 con 0x7fc4540d0c80 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 msgr2=0x7fc4540cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 0x7fc4540cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 >> v2:172.21.15.39:3300/0 conn(0x7fc4540d4a10 msgr2=0x7fc4540d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 --2- 172.21.15.32:0/2547978520 >> v2:172.21.15.39:3300/0 conn(0x7fc4540d4a10 0x7fc4540d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540b3cd0 con 0x7fc4540d0c80 2024-03-06T07:36:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fc44c0019f0 con 0x7fc4540d0c80 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.010+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 >> v1:172.21.15.32:6789/0 conn(0x7fc4540d0c80 legacy=0x7fc4540d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453400640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc43c0039b0 0x7fc43c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453400640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc43c004800 0x7fc43c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453400640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 0x7fc43c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b39f0 con 0x7fc43c0039b0 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c00b9c0 con 0x7fc43c004800 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c00bb40 con 0x7fc45415e0b0 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc459200640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc43c0039b0 0x7fc43c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453e00640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc43c004800 0x7fc43c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc459200640 1 -- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 msgr2=0x7fc43c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc459200640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc45415e0b0 0x7fc43c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc459200640 1 -- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc43c004800 msgr2=0x7fc43c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc459200640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc43c004800 0x7fc43c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc459200640 1 -- 172.21.15.32:0/2547978520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c003520 con 0x7fc43c0039b0 2024-03-06T07:36:37.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.011+0000 7fc453e00640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc43c004800 0x7fc43c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.012+0000 7fc459200640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc43c0039b0 0x7fc43c003dd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fc448009ac0 tx=0x7fc43c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7289210347c79ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448039070 con 0x7fc43c0039b0 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44802c6d0 con 0x7fc43c0039b0 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc453400640 1 -- 172.21.15.32:0/2547978520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448009690 con 0x7fc43c0039b0 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc45a600640 1 -- 172.21.15.32:0/2547978520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc43c0039b0 msgr2=0x7fc43c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc45a600640 1 --2- 172.21.15.32:0/2547978520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc43c0039b0 0x7fc43c003dd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fc448009ac0 tx=0x7fc43c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc459c00640 1 -- 172.21.15.32:0/2547978520 reap_dead start 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc45a600640 1 -- 172.21.15.32:0/2547978520 shutdown_connections 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc45a600640 1 -- 172.21.15.32:0/2547978520 >> 172.21.15.32:0/2547978520 conn(0x7fc4540d3d30 msgr2=0x7fc4540d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc45a600640 1 -- 172.21.15.32:0/2547978520 shutdown_connections 2024-03-06T07:36:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.013+0000 7fc45a600640 1 -- 172.21.15.32:0/2547978520 wait complete. 2024-03-06T07:36:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.017+0000 7fc45a600640 1 Processor -- start 2024-03-06T07:36:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.017+0000 7fc45a600640 1 -- start start 2024-03-06T07:36:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.018+0000 7fc45a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc43c0039b0 0x7fc4540c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc45a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 0x7fc4540cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc45a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4541f70c0 0x7fc4541f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc45a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540cfc60 con 0x7fc4540cd3b0 2024-03-06T07:36:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc45a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45415e490 con 0x7fc4541f70c0 2024-03-06T07:36:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc45a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b54e0 con 0x7fc43c0039b0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 0x7fc4540cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 0x7fc4540cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33194/0 (socket says 172.21.15.32:33194) 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 -- 172.21.15.32:0/2183973649 learned_addr learned my addr 172.21.15.32:0/2183973649 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc459200640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc43c0039b0 0x7fc4540c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 -- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc43c0039b0 msgr2=0x7fc4540c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc43c0039b0 0x7fc4540c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 -- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4541f70c0 msgr2=0x7fc4541f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc459c00640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4541f70c0 0x7fc4541f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4541f70c0 0x7fc4541f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc453e00640 1 -- 172.21.15.32:0/2183973649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4541f7e40 con 0x7fc4540cd3b0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.019+0000 7fc459c00640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4541f70c0 0x7fc4541f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.020+0000 7fc453e00640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 0x7fc4540cd830 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fc4400109d0 tx=0x7fc440010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.020+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000f040 con 0x7fc4540cd3b0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.020+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44000ec20 con 0x7fc4540cd3b0 2024-03-06T07:36:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.020+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45414d560 con 0x7fc4540cd3b0 2024-03-06T07:36:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.020+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44001c400 con 0x7fc4540cd3b0 2024-03-06T07:36:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.020+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45414db20 con 0x7fc4540cd3b0 2024-03-06T07:36:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.021+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45404f330 con 0x7fc4540cd3b0 2024-03-06T07:36:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.022+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc44001c5a0 con 0x7fc4540cd3b0 2024-03-06T07:36:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.024+0000 7fc450c00640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc43406b500 0x7fc43406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.024+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc44008a940 con 0x7fc4540cd3b0 2024-03-06T07:36:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.024+0000 7fc459200640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc43406b500 0x7fc43406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.025+0000 7fc459200640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc43406b500 0x7fc43406d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc448036000 tx=0x7fc448030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.029+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc440018030 con 0x7fc4540cd3b0 2024-03-06T07:36:37.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.246+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4540d0690 con 0x7fc4540cd3b0 2024-03-06T07:36:37.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.248+0000 7fc450c00640 1 -- 172.21.15.32:0/2183973649 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16738 (secure 0 0 0) 0x7fc440059720 con 0x7fc4540cd3b0 2024-03-06T07:36:37.250 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:37.250 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":116,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:49.005590+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.18","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/14889266":"2024-03-07T07:35:30.884623+0000","172.21.15.39:6833/14889266":"2024-03-07T07:35:30.884623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc43406b500 msgr2=0x7fc43406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc43406b500 0x7fc43406d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc448036000 tx=0x7fc448030000 comp rx=0 tx=0).stop 2024-03-06T07:36:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 msgr2=0x7fc4540cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 0x7fc4540cd830 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fc4400109d0 tx=0x7fc440010e90 comp rx=0 tx=0).stop 2024-03-06T07:36:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 shutdown_connections 2024-03-06T07:36:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc43406b500 0x7fc43406d9c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4541f70c0 0x7fc4541f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4540cd3b0 0x7fc4540cd830 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 --2- 172.21.15.32:0/2183973649 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc43c0039b0 0x7fc4540c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.250+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 >> 172.21.15.32:0/2183973649 conn(0x7fc4540d3d30 msgr2=0x7fc4540ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.251+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 shutdown_connections 2024-03-06T07:36:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.251+0000 7fc45a600640 1 -- 172.21.15.32:0/2183973649 wait complete. 2024-03-06T07:36:37.272 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:12.716010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:13.860496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:36:37.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:36:37.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.492+0000 7f0649000640 1 Processor -- start 2024-03-06T07:36:37.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.493+0000 7f0649000640 1 -- start start 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.494+0000 7f0649000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f064415dee0 0x7f06440d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.494+0000 7f0649000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06440ceca0 con 0x7f06440d0d40 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.494+0000 7f0649000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06440cef80 con 0x7f06440d4860 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.494+0000 7f0649000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06440cf5d0 con 0x7f064415dee0 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.494+0000 7f0643400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f064415dee0 0x7f06440d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.494+0000 7f0643e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f06440d0d40 0x7f06440b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57428/0 (socket says 172.21.15.32:57428) 2024-03-06T07:36:37.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.495+0000 7f0643e00640 1 -- 172.21.15.32:0/442666842 learned_addr learned my addr 172.21.15.32:0/442666842 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:37.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.495+0000 7f0643400640 1 -- 172.21.15.32:0/442666842 >> v1:172.21.15.39:6789/0 conn(0x7f06440d4860 legacy=0x7f06440d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.495+0000 7f0643400640 1 -- 172.21.15.32:0/442666842 >> v1:172.21.15.32:6789/0 conn(0x7f06440d0d40 legacy=0x7f06440b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.495+0000 7f0643400640 1 -- 172.21.15.32:0/442666842 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06440cf260 con 0x7f064415dee0 2024-03-06T07:36:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.496+0000 7f0643400640 1 --2- 172.21.15.32:0/442666842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f064415dee0 0x7f06440d18a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f06340097a0 tx=0x7f063402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324200f34d9197bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.496+0000 7f0642a00640 1 -- 172.21.15.32:0/442666842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634035020 con 0x7f064415dee0 2024-03-06T07:36:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0642a00640 1 -- 172.21.15.32:0/442666842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634031830 con 0x7f064415dee0 2024-03-06T07:36:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0642a00640 1 -- 172.21.15.32:0/442666842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634031c00 con 0x7f064415dee0 2024-03-06T07:36:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 -- 172.21.15.32:0/442666842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f064415dee0 msgr2=0x7f06440d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 --2- 172.21.15.32:0/442666842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f064415dee0 0x7f06440d18a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f06340097a0 tx=0x7f063402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 -- 172.21.15.32:0/442666842 shutdown_connections 2024-03-06T07:36:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 --2- 172.21.15.32:0/442666842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f064415dee0 0x7f06440d18a0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 -- 172.21.15.32:0/442666842 >> 172.21.15.32:0/442666842 conn(0x7f06440d3cd0 msgr2=0x7f06440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 -- 172.21.15.32:0/442666842 shutdown_connections 2024-03-06T07:36:37.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.497+0000 7f0649000640 1 -- 172.21.15.32:0/442666842 wait complete. 2024-03-06T07:36:37.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.499+0000 7f0649000640 1 Processor -- start 2024-03-06T07:36:37.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.499+0000 7f0649000640 1 -- start start 2024-03-06T07:36:37.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f0649000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06440d0d40 0x7f06440cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f0649000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 0x7f06440b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f0649000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f064415dee0 0x7f06440b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f0649000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06440d1de0 con 0x7f06440d4860 2024-03-06T07:36:37.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f0649000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064415e2c0 con 0x7f06440d0d40 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f0649000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06440d1f60 con 0x7f064415dee0 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f063be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 0x7f06440b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f063be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 0x7f06440b7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33214/0 (socket says 172.21.15.32:33214) 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.501+0000 7f063be00640 1 -- 172.21.15.32:0/4238149097 learned_addr learned my addr 172.21.15.32:0/4238149097 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f0643400640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06440d0d40 0x7f06440cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f0643e00640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f064415dee0 0x7f06440b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063be00640 1 -- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f064415dee0 msgr2=0x7f06440b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063be00640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f064415dee0 0x7f06440b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063be00640 1 -- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06440d0d40 msgr2=0x7f06440cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063be00640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06440d0d40 0x7f06440cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063be00640 1 -- 172.21.15.32:0/4238149097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06440cd1f0 con 0x7f06440d4860 2024-03-06T07:36:37.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f0643400640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06440d0d40 0x7f06440cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f0643e00640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f064415dee0 0x7f06440b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063be00640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 0x7f06440b7640 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c430 tx=0x7f062c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.502+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c011790 con 0x7f06440d4860 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.503+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c017070 con 0x7f06440d4860 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.503+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c011bb0 con 0x7f06440d4860 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.503+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06440a1880 con 0x7f06440d4860 2024-03-06T07:36:37.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.503+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06441f7a90 con 0x7f06440d4860 2024-03-06T07:36:37.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.504+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064404f330 con 0x7f06440d4860 2024-03-06T07:36:37.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.505+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f062c01c460 con 0x7f06440d4860 2024-03-06T07:36:37.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.506+0000 7f063b400640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f062006b4d0 0x7f062006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.506+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f062c08a190 con 0x7f06440d4860 2024-03-06T07:36:37.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.507+0000 7f0643400640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f062006b4d0 0x7f062006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.507+0000 7f0643400640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f062006b4d0 0x7f062006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0634036000 tx=0x7f0634008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.511+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f062c058f70 con 0x7f06440d4860 2024-03-06T07:36:37.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.711+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06440cd4b0 con 0x7f06440d4860 2024-03-06T07:36:37.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.712+0000 7f063b400640 1 -- 172.21.15.32:0/4238149097 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f062c058870 con 0x7f06440d4860 2024-03-06T07:36:37.714 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f062006b4d0 msgr2=0x7f062006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f062006b4d0 0x7f062006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0634036000 tx=0x7f0634008030 comp rx=0 tx=0).stop 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 msgr2=0x7f06440b7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 0x7f06440b7640 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c430 tx=0x7f062c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 shutdown_connections 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f062006b4d0 0x7f062006d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f064415dee0 0x7f06440b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06440d4860 0x7f06440b7640 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 --2- 172.21.15.32:0/4238149097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06440d0d40 0x7f06440cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.714+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 >> 172.21.15.32:0/4238149097 conn(0x7f06440d3cd0 msgr2=0x7f06440c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.715+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 shutdown_connections 2024-03-06T07:36:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.715+0000 7f0649000640 1 -- 172.21.15.32:0/4238149097 wait complete. 2024-03-06T07:36:37.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:36:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.968+0000 7fde8ce00640 1 Processor -- start 2024-03-06T07:36:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.969+0000 7fde8ce00640 1 -- start start 2024-03-06T07:36:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.970+0000 7fde8ce00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fde8806ed40 0x7fde880b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.970+0000 7fde8ce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fde8806f830 0x7fde880ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.970+0000 7fde8ce00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8806dee0 con 0x7fde880cc710 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde8ce00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8806e180 con 0x7fde8806f830 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde8ce00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8806e460 con 0x7fde8806ed40 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fde8806f830 0x7fde880ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fde8806f830 0x7fde880ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44962/0 (socket says 172.21.15.32:44962) 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 -- 172.21.15.32:0/682152350 learned_addr learned my addr 172.21.15.32:0/682152350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87400640 1 --2- 172.21.15.32:0/682152350 >> v2:172.21.15.39:3300/0 conn(0x7fde8806ed40 0x7fde880b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 -- 172.21.15.32:0/682152350 >> v2:172.21.15.39:3300/0 conn(0x7fde8806ed40 msgr2=0x7fde880b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 --2- 172.21.15.32:0/682152350 >> v2:172.21.15.39:3300/0 conn(0x7fde8806ed40 0x7fde880b3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 -- 172.21.15.32:0/682152350 >> v1:172.21.15.39:6789/0 conn(0x7fde880cc710 legacy=0x7fde8806f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.971+0000 7fde87e00640 1 -- 172.21.15.32:0/682152350 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde880ce400 con 0x7fde8806f830 2024-03-06T07:36:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.973+0000 7fde87e00640 1 --2- 172.21.15.32:0/682152350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806f830 0x7fde880ccb70 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fde8804f9c0 tx=0x7fde8002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61aa045ebb634b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.973+0000 7fde86000640 1 -- 172.21.15.32:0/682152350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8002c960 con 0x7fde8806f830 2024-03-06T07:36:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.973+0000 7fde86000640 1 -- 172.21.15.32:0/682152350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8002cb00 con 0x7fde8806f830 2024-03-06T07:36:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.973+0000 7fde86000640 1 -- 172.21.15.32:0/682152350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde800343f0 con 0x7fde8806f830 2024-03-06T07:36:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 -- 172.21.15.32:0/682152350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806f830 msgr2=0x7fde880ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 --2- 172.21.15.32:0/682152350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806f830 0x7fde880ccb70 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fde8804f9c0 tx=0x7fde8002bdc0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 -- 172.21.15.32:0/682152350 shutdown_connections 2024-03-06T07:36:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 --2- 172.21.15.32:0/682152350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806f830 0x7fde880ccb70 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 --2- 172.21.15.32:0/682152350 >> v2:172.21.15.39:3300/0 conn(0x7fde8806ed40 0x7fde880b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 -- 172.21.15.32:0/682152350 >> 172.21.15.32:0/682152350 conn(0x7fde880723d0 msgr2=0x7fde8806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 -- 172.21.15.32:0/682152350 shutdown_connections 2024-03-06T07:36:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.974+0000 7fde8ce00640 1 -- 172.21.15.32:0/682152350 wait complete. 2024-03-06T07:36:37.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.977+0000 7fde8ce00640 1 Processor -- start 2024-03-06T07:36:37.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.977+0000 7fde8ce00640 1 -- start start 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde8ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde8806ed40 0x7fde881548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde8ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fde8806f830 0x7fde8814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde8ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 0x7fde881410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde8ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8806ea10 con 0x7fde880cc710 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde8ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde88066d00 con 0x7fde8806ed40 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde8ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde881566f0 con 0x7fde8806f830 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 0x7fde881410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 0x7fde881410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44976/0 (socket says 172.21.15.32:44976) 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde87e00640 1 -- 172.21.15.32:0/307690476 learned_addr learned my addr 172.21.15.32:0/307690476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde87400640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde8806ed40 0x7fde881548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.979+0000 7fde86a00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fde8806f830 0x7fde8814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde87e00640 1 -- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fde8806f830 msgr2=0x7fde8814cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde87e00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fde8806f830 0x7fde8814cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde87e00640 1 -- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde8806ed40 msgr2=0x7fde881548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde87e00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde8806ed40 0x7fde881548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde87e00640 1 -- 172.21.15.32:0/307690476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde881569d0 con 0x7fde880cc710 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde87e00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 0x7fde881410f0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fde80036040 tx=0x7fde80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80043020 con 0x7fde880cc710 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8003e450 con 0x7fde880cc710 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8003e840 con 0x7fde880cc710 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.980+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8814b300 con 0x7fde880cc710 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.981+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde881563a0 con 0x7fde880cc710 2024-03-06T07:36:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.982+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8804f330 con 0x7fde880cc710 2024-03-06T07:36:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.983+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde80046020 con 0x7fde880cc710 2024-03-06T07:36:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.984+0000 7fde7f400640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fde6806b500 0x7fde6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.984+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fde800acbb0 con 0x7fde880cc710 2024-03-06T07:36:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.984+0000 7fde87400640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fde6806b500 0x7fde6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.985+0000 7fde87400640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fde6806b500 0x7fde6806d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fde78006fd0 tx=0x7fde78008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:37.993+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde8004d030 con 0x7fde880cc710 2024-03-06T07:36:38.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.217+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fde880b12c0 con 0x7fde880cc710 2024-03-06T07:36:38.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.218+0000 7fde7f400640 1 -- 172.21.15.32:0/307690476 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fde8007b900 con 0x7fde880cc710 2024-03-06T07:36:38.220 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.220+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fde6806b500 msgr2=0x7fde6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.220+0000 7fde8ce00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fde6806b500 0x7fde6806d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fde78006fd0 tx=0x7fde78008040 comp rx=0 tx=0).stop 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.220+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 msgr2=0x7fde881410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.220+0000 7fde8ce00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 0x7fde881410f0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fde80036040 tx=0x7fde80004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 shutdown_connections 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fde6806b500 0x7fde6806d9c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde880cc710 0x7fde881410f0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fde8806f830 0x7fde8814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 --2- 172.21.15.32:0/307690476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde8806ed40 0x7fde881548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 >> 172.21.15.32:0/307690476 conn(0x7fde880723d0 msgr2=0x7fde880b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 shutdown_connections 2024-03-06T07:36:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.221+0000 7fde8ce00640 1 -- 172.21.15.32:0/307690476 wait complete. 2024-03-06T07:36:38.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:36:38.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.463+0000 7fb01f200640 1 Processor -- start 2024-03-06T07:36:38.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.463+0000 7fb01f200640 1 -- start start 2024-03-06T07:36:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.464+0000 7fb01f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb02015bb90 0x7fb0200b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.464+0000 7fb01f200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0200b4ff0 0x7fb0200b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.464+0000 7fb01f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02015d600 con 0x7fb0200b5b50 2024-03-06T07:36:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.464+0000 7fb01f200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02015f690 con 0x7fb0200b4ff0 2024-03-06T07:36:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.464+0000 7fb01f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0200b5f80 con 0x7fb02015bb90 2024-03-06T07:36:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0200b4ff0 0x7fb0200b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0200b4ff0 0x7fb0200b5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44984/0 (socket says 172.21.15.32:44984) 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 -- 172.21.15.32:0/3671210010 learned_addr learned my addr 172.21.15.32:0/3671210010 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01de00640 1 --2- 172.21.15.32:0/3671210010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb02015bb90 0x7fb0200b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 -- 172.21.15.32:0/3671210010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb02015bb90 msgr2=0x7fb0200b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 --2- 172.21.15.32:0/3671210010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb02015bb90 0x7fb0200b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 -- 172.21.15.32:0/3671210010 >> v1:172.21.15.39:6789/0 conn(0x7fb0200b5b50 legacy=0x7fb02015f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.465+0000 7fb01d400640 1 -- 172.21.15.32:0/3671210010 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02015d220 con 0x7fb0200b4ff0 2024-03-06T07:36:38.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.467+0000 7fb01d400640 1 --2- 172.21.15.32:0/3671210010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b4ff0 0x7fb0200b5610 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fb008009e50 tx=0x7fb00802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98071cc315c880da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.467+0000 7fb01ca00640 1 -- 172.21.15.32:0/3671210010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00802c9e0 con 0x7fb0200b4ff0 2024-03-06T07:36:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.467+0000 7fb01ca00640 1 -- 172.21.15.32:0/3671210010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00802cb80 con 0x7fb0200b4ff0 2024-03-06T07:36:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.467+0000 7fb01ca00640 1 -- 172.21.15.32:0/3671210010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0080343f0 con 0x7fb0200b4ff0 2024-03-06T07:36:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 -- 172.21.15.32:0/3671210010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b4ff0 msgr2=0x7fb0200b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 --2- 172.21.15.32:0/3671210010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b4ff0 0x7fb0200b5610 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fb008009e50 tx=0x7fb00802be40 comp rx=0 tx=0).stop 2024-03-06T07:36:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 -- 172.21.15.32:0/3671210010 shutdown_connections 2024-03-06T07:36:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 --2- 172.21.15.32:0/3671210010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b4ff0 0x7fb0200b5610 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 --2- 172.21.15.32:0/3671210010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb02015bb90 0x7fb0200b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 -- 172.21.15.32:0/3671210010 >> 172.21.15.32:0/3671210010 conn(0x7fb02009aa10 msgr2=0x7fb02009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 -- 172.21.15.32:0/3671210010 shutdown_connections 2024-03-06T07:36:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.468+0000 7fb01f200640 1 -- 172.21.15.32:0/3671210010 wait complete. 2024-03-06T07:36:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.470+0000 7fb01f200640 1 Processor -- start 2024-03-06T07:36:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.470+0000 7fb01f200640 1 -- start start 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0200b4ff0 0x7fb02014c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 0x7fb0200afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb02015bb90 0x7fb0200b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02015de80 con 0x7fb0200b5b50 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02015dba0 con 0x7fb02015bb90 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02015e160 con 0x7fb0200b4ff0 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 0x7fb0200afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 0x7fb0200afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44998/0 (socket says 172.21.15.32:44998) 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01d400640 1 -- 172.21.15.32:0/1417188378 learned_addr learned my addr 172.21.15.32:0/1417188378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01de00640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0200b4ff0 0x7fb02014c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01e800640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb02015bb90 0x7fb0200b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.471+0000 7fb01d400640 1 -- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0200b4ff0 msgr2=0x7fb02014c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01d400640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0200b4ff0 0x7fb02014c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01d400640 1 -- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb02015bb90 msgr2=0x7fb0200b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01d400640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb02015bb90 0x7fb0200b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01d400640 1 -- 172.21.15.32:0/1417188378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020153790 con 0x7fb0200b5b50 2024-03-06T07:36:38.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01e800640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb02015bb90 0x7fb0200b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01d400640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 0x7fb0200afa30 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fb008009e50 tx=0x7fb008004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008043020 con 0x7fb0200b5b50 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00803e450 con 0x7fb0200b5b50 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02014ca20 con 0x7fb0200b5b50 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008009510 con 0x7fb0200b5b50 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.472+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0200b1240 con 0x7fb0200b5b50 2024-03-06T07:36:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.473+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02004f330 con 0x7fb0200b5b50 2024-03-06T07:36:38.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.476+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb008005d10 con 0x7fb0200b5b50 2024-03-06T07:36:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.477+0000 7fb016000640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faffc06b4d0 0x7faffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.477+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb0080ace90 con 0x7fb0200b5b50 2024-03-06T07:36:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.477+0000 7fb01de00640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faffc06b4d0 0x7faffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.478+0000 7fb01de00640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faffc06b4d0 0x7faffc06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb010006fd0 tx=0x7fb010008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.485+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb020153790 con 0x7fb0200b5b50 2024-03-06T07:36:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.687+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0200b01a0 con 0x7fb0200b5b50 2024-03-06T07:36:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.688+0000 7fb016000640 1 -- 172.21.15.32:0/1417188378 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb00807bbe0 con 0x7fb0200b5b50 2024-03-06T07:36:38.690 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:36:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faffc06b4d0 msgr2=0x7faffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faffc06b4d0 0x7faffc06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb010006fd0 tx=0x7fb010008040 comp rx=0 tx=0).stop 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 msgr2=0x7fb0200afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 0x7fb0200afa30 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fb008009e50 tx=0x7fb008004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 shutdown_connections 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faffc06b4d0 0x7faffc06d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb02015bb90 0x7fb0200b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0200b5b50 0x7fb0200afa30 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 --2- 172.21.15.32:0/1417188378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0200b4ff0 0x7fb02014c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 >> 172.21.15.32:0/1417188378 conn(0x7fb02009aa10 msgr2=0x7fb020157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 shutdown_connections 2024-03-06T07:36:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.690+0000 7fb01f200640 1 -- 172.21.15.32:0/1417188378 wait complete. 2024-03-06T07:36:38.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.918+0000 7f773ca00640 1 Processor -- start 2024-03-06T07:36:38.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.918+0000 7f773ca00640 1 -- start start 2024-03-06T07:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f773ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f773815dd00 0x7f7738162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f773ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f773806bfc0 0x7f773806c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f773ca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7738161a80 con 0x7f773815e0e0 2024-03-06T07:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f773ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773815efa0 con 0x7f773815dd00 2024-03-06T07:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f773ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773815f280 con 0x7f773806bfc0 2024-03-06T07:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f7735e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f773815dd00 0x7f7738162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f7735e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f773815dd00 0x7f7738162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45008/0 (socket says 172.21.15.32:45008) 2024-03-06T07:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f7736800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f773815e0e0 0x7f773815e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45870/0 (socket says 172.21.15.32:45870) 2024-03-06T07:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.920+0000 7f7735e00640 1 -- 172.21.15.32:0/1014165767 learned_addr learned my addr 172.21.15.32:0/1014165767 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.921+0000 7f7735400640 1 --2- 172.21.15.32:0/1014165767 >> v2:172.21.15.39:3300/0 conn(0x7f773806bfc0 0x7f773806c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.921+0000 7f7735e00640 1 -- 172.21.15.32:0/1014165767 >> v2:172.21.15.39:3300/0 conn(0x7f773806bfc0 msgr2=0x7f773806c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.921+0000 7f7735e00640 1 --2- 172.21.15.32:0/1014165767 >> v2:172.21.15.39:3300/0 conn(0x7f773806bfc0 0x7f773806c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.921+0000 7f7735e00640 1 -- 172.21.15.32:0/1014165767 >> v1:172.21.15.32:6789/0 conn(0x7f773815e0e0 legacy=0x7f773815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.921+0000 7f7735e00640 1 -- 172.21.15.32:0/1014165767 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738161d20 con 0x7f773815dd00 2024-03-06T07:36:38.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.921+0000 7f7734a00640 1 -- 172.21.15.32:0/1014165767 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808363046 0 0) 0x7f7738161a80 con 0x7f773815e0e0 2024-03-06T07:36:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.922+0000 7f7735e00640 1 --2- 172.21.15.32:0/1014165767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815dd00 0x7f7738162520 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f77380721f0 tx=0x7f772802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62409e3b77a7dcba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.922+0000 7f7734a00640 1 -- 172.21.15.32:0/1014165767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728035020 con 0x7f773815dd00 2024-03-06T07:36:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.922+0000 7f7734a00640 1 -- 172.21.15.32:0/1014165767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7728031870 con 0x7f773815dd00 2024-03-06T07:36:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.922+0000 7f7734a00640 1 -- 172.21.15.32:0/1014165767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728031c40 con 0x7f773815dd00 2024-03-06T07:36:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 -- 172.21.15.32:0/1014165767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815dd00 msgr2=0x7f7738162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 --2- 172.21.15.32:0/1014165767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815dd00 0x7f7738162520 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f77380721f0 tx=0x7f772802b910 comp rx=0 tx=0).stop 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 -- 172.21.15.32:0/1014165767 shutdown_connections 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 --2- 172.21.15.32:0/1014165767 >> v2:172.21.15.39:3300/0 conn(0x7f773806bfc0 0x7f773806c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 --2- 172.21.15.32:0/1014165767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815dd00 0x7f7738162520 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 -- 172.21.15.32:0/1014165767 >> 172.21.15.32:0/1014165767 conn(0x7f77380714d0 msgr2=0x7f77380718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 -- 172.21.15.32:0/1014165767 shutdown_connections 2024-03-06T07:36:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.923+0000 7f773ca00640 1 -- 172.21.15.32:0/1014165767 wait complete. 2024-03-06T07:36:38.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.925+0000 7f773ca00640 1 Processor -- start 2024-03-06T07:36:38.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.925+0000 7f773ca00640 1 -- start start 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f773ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773806bfc0 0x7f77380c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f773ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f773815dd00 0x7f7738158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f773ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 0x7f7738148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f773ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773815ec10 con 0x7f773815e0e0 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f773ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773806f280 con 0x7f773806bfc0 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f773ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773806d250 con 0x7f773815dd00 2024-03-06T07:36:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f7736800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 0x7f7738148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f7736800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 0x7f7738148ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45018/0 (socket says 172.21.15.32:45018) 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.926+0000 7f7736800640 1 -- 172.21.15.32:0/2366836397 learned_addr learned my addr 172.21.15.32:0/2366836397 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7735e00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773806bfc0 0x7f77380c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7735400640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f773815dd00 0x7f7738158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7736800640 1 -- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f773815dd00 msgr2=0x7f7738158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7736800640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f773815dd00 0x7f7738158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7736800640 1 -- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773806bfc0 msgr2=0x7f77380c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7736800640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773806bfc0 0x7f77380c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7736800640 1 -- 172.21.15.32:0/2366836397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738158a10 con 0x7f773815e0e0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.927+0000 7f7736800640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 0x7f7738148ed0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f773000c700 tx=0x7f773000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.928+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7730018070 con 0x7f773815e0e0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.928+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773000ce10 con 0x7f773815e0e0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.928+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773000a860 con 0x7f773815e0e0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.928+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7738158cf0 con 0x7f773815e0e0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.928+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77381592b0 con 0x7f773815e0e0 2024-03-06T07:36:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.929+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773804f3a0 con 0x7f773815e0e0 2024-03-06T07:36:38.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.930+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f773000aa00 con 0x7f773815e0e0 2024-03-06T07:36:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.931+0000 7f772e000640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f771806b500 0x7f771806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.931+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f77300898b0 con 0x7f773815e0e0 2024-03-06T07:36:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.931+0000 7f7735e00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f771806b500 0x7f771806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.932+0000 7f7735e00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f771806b500 0x7f771806d9c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7728036000 tx=0x7f7728008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:38.937+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7730058600 con 0x7f773815e0e0 2024-03-06T07:36:39.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.174+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77380b0cc0 con 0x7f773815e0e0 2024-03-06T07:36:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.175+0000 7f772e000640 1 -- 172.21.15.32:0/2366836397 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f7730057f00 con 0x7f773815e0e0 2024-03-06T07:36:39.177 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:39.177 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.177+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f771806b500 msgr2=0x7f771806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.177+0000 7f773ca00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f771806b500 0x7f771806d9c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7728036000 tx=0x7f7728008030 comp rx=0 tx=0).stop 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.177+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 msgr2=0x7f7738148ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.177+0000 7f773ca00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 0x7f7738148ed0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f773000c700 tx=0x7f773000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 shutdown_connections 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f771806b500 0x7f771806d9c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f773815e0e0 0x7f7738148ed0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f773815dd00 0x7f7738158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 --2- 172.21.15.32:0/2366836397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773806bfc0 0x7f77380c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 >> 172.21.15.32:0/2366836397 conn(0x7f77380714d0 msgr2=0x7f77380b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 shutdown_connections 2024-03-06T07:36:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.178+0000 7f773ca00640 1 -- 172.21.15.32:0/2366836397 wait complete. 2024-03-06T07:36:39.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.392+0000 7fd1cd200640 1 Processor -- start 2024-03-06T07:36:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.393+0000 7fd1cd200640 1 -- start start 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.394+0000 7fd1cd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1c815e240 0x7fd1c80b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.394+0000 7fd1cd200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1c81628e0 0x7fd1c8161d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.394+0000 7fd1cd200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.394+0000 7fd1cd200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c80b7260 con 0x7fd1c81628e0 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.394+0000 7fd1cd200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c80b6ca0 con 0x7fd1c815e240 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.394+0000 7fd1cd200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80b6f80 con 0x7fd1c8162280 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1c81628e0 0x7fd1c8161d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1c81628e0 0x7fd1c8161d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45022/0 (socket says 172.21.15.32:45022) 2024-03-06T07:36:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 -- 172.21.15.32:0/253700495 learned_addr learned my addr 172.21.15.32:0/253700495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1c5e00640 1 --2- 172.21.15.32:0/253700495 >> v2:172.21.15.39:3300/0 conn(0x7fd1c815e240 0x7fd1c80b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1c6800640 1 --2- 172.21.15.32:0/253700495 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 -- 172.21.15.32:0/253700495 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 msgr2=0x7fd1c806c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 --2- 172.21.15.32:0/253700495 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 -- 172.21.15.32:0/253700495 >> v2:172.21.15.39:3300/0 conn(0x7fd1c815e240 msgr2=0x7fd1c80b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 --2- 172.21.15.32:0/253700495 >> v2:172.21.15.39:3300/0 conn(0x7fd1c815e240 0x7fd1c80b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.395+0000 7fd1bd400640 1 -- 172.21.15.32:0/253700495 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c815e950 con 0x7fd1c81628e0 2024-03-06T07:36:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.397+0000 7fd1bd400640 1 --2- 172.21.15.32:0/253700495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c8161d40 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd1c806ee60 tx=0x7fd1b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e058a5383652445 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.397+0000 7fd1c5400640 1 -- 172.21.15.32:0/253700495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0035020 con 0x7fd1c81628e0 2024-03-06T07:36:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.397+0000 7fd1c5400640 1 -- 172.21.15.32:0/253700495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b0031870 con 0x7fd1c81628e0 2024-03-06T07:36:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.397+0000 7fd1c5400640 1 -- 172.21.15.32:0/253700495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0031c40 con 0x7fd1c81628e0 2024-03-06T07:36:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.397+0000 7fd1cd200640 1 -- 172.21.15.32:0/253700495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 msgr2=0x7fd1c8161d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.397+0000 7fd1cd200640 1 --2- 172.21.15.32:0/253700495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c8161d40 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd1c806ee60 tx=0x7fd1b002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 -- 172.21.15.32:0/253700495 shutdown_connections 2024-03-06T07:36:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 --2- 172.21.15.32:0/253700495 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 --2- 172.21.15.32:0/253700495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c8161d40 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 --2- 172.21.15.32:0/253700495 >> v2:172.21.15.39:3300/0 conn(0x7fd1c815e240 0x7fd1c80b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 -- 172.21.15.32:0/253700495 >> 172.21.15.32:0/253700495 conn(0x7fd1c80b7450 msgr2=0x7fd1c809aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 -- 172.21.15.32:0/253700495 shutdown_connections 2024-03-06T07:36:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.398+0000 7fd1cd200640 1 -- 172.21.15.32:0/253700495 wait complete. 2024-03-06T07:36:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.399+0000 7fd1cd200640 1 Processor -- start 2024-03-06T07:36:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.400+0000 7fd1cd200640 1 -- start start 2024-03-06T07:36:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c815e240 0x7fd1c81e95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1cd200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1cd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c814a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1cd200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80b68e0 con 0x7fd1c81628e0 2024-03-06T07:36:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1cd200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c815e620 con 0x7fd1c815e240 2024-03-06T07:36:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1cd200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80b6a60 con 0x7fd1c8162280 2024-03-06T07:36:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1c6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c814a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1c6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c814a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45028/0 (socket says 172.21.15.32:45028) 2024-03-06T07:36:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1c6800640 1 -- 172.21.15.32:0/3146206393 learned_addr learned my addr 172.21.15.32:0/3146206393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1c5e00640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c815e240 0x7fd1c81e95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.401+0000 7fd1bd400640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1c6800640 1 -- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 msgr2=0x7fd1c806f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1c6800640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1c6800640 1 -- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c815e240 msgr2=0x7fd1c81e95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1c6800640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c815e240 0x7fd1c81e95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1c6800640 1 -- 172.21.15.32:0/3146206393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c81f5050 con 0x7fd1c81628e0 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1c6800640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c814a630 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0007e70 tx=0x7fd1c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0014620 con 0x7fd1c81628e0 2024-03-06T07:36:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0019070 con 0x7fd1c81628e0 2024-03-06T07:36:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0014a20 con 0x7fd1c81628e0 2024-03-06T07:36:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.402+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c81f4d70 con 0x7fd1c81628e0 2024-03-06T07:36:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.403+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c81e8dc0 con 0x7fd1c81628e0 2024-03-06T07:36:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.404+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c804f3a0 con 0x7fd1c81628e0 2024-03-06T07:36:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.406+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1c0014bc0 con 0x7fd1c81628e0 2024-03-06T07:36:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.407+0000 7fd1bea00640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1a406b4d0 0x7fd1a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.407+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd1c008aba0 con 0x7fd1c81628e0 2024-03-06T07:36:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.408+0000 7fd1c5e00640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1a406b4d0 0x7fd1a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.408+0000 7fd1c5e00640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1a406b4d0 0x7fd1a406d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006fd0 tx=0x7fd1b8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.412+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1c0059980 con 0x7fd1c81628e0 2024-03-06T07:36:39.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.612+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1c809acc0 con 0x7fd1c81628e0 2024-03-06T07:36:39.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.615+0000 7fd1bea00640 1 -- 172.21.15.32:0/3146206393 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16738 (secure 0 0 0) 0x7fd1c0059280 con 0x7fd1c81628e0 2024-03-06T07:36:39.618 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:39.618 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":116,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:49.005590+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.18","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/14889266":"2024-03-07T07:35:30.884623+0000","172.21.15.39:6833/14889266":"2024-03-07T07:35:30.884623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.617+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1a406b4d0 msgr2=0x7fd1a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.617+0000 7fd1cd200640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1a406b4d0 0x7fd1a406d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006fd0 tx=0x7fd1b8008040 comp rx=0 tx=0).stop 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.617+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 msgr2=0x7fd1c814a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.617+0000 7fd1cd200640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c814a630 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0007e70 tx=0x7fd1c000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 shutdown_connections 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1a406b4d0 0x7fd1a406d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c81628e0 0x7fd1c814a630 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1c8162280 0x7fd1c806f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 --2- 172.21.15.32:0/3146206393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c815e240 0x7fd1c81e95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 >> 172.21.15.32:0/3146206393 conn(0x7fd1c80b7450 msgr2=0x7fd1c8070830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 shutdown_connections 2024-03-06T07:36:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.618+0000 7fd1cd200640 1 -- 172.21.15.32:0/3146206393 wait complete. 2024-03-06T07:36:39.640 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:36:39.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:39.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.848+0000 7f9e3ee00640 1 Processor -- start 2024-03-06T07:36:39.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.848+0000 7f9e3ee00640 1 -- start start 2024-03-06T07:36:39.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.849+0000 7f9e3ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e380cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3ee00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e380d4a10 0x7f9e380d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3ee00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380b3710 con 0x7f9e380d0c80 2024-03-06T07:36:39.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3ee00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e380b09e0 con 0x7f9e380d4a10 2024-03-06T07:36:39.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380b39f0 con 0x7f9e3815e0b0 2024-03-06T07:36:39.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e380d4a10 0x7f9e380d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e380d4a10 0x7f9e380d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45044/0 (socket says 172.21.15.32:45044) 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3d000640 1 -- 172.21.15.32:0/2107496185 learned_addr learned my addr 172.21.15.32:0/2107496185 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.850+0000 7f9e3da00640 1 --2- 172.21.15.32:0/2107496185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e380cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.851+0000 7f9e3d000640 1 -- 172.21.15.32:0/2107496185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 msgr2=0x7f9e380cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.851+0000 7f9e3d000640 1 --2- 172.21.15.32:0/2107496185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e380cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.851+0000 7f9e3d000640 1 -- 172.21.15.32:0/2107496185 >> v1:172.21.15.39:6789/0 conn(0x7f9e380d0c80 legacy=0x7f9e380d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.851+0000 7f9e3d000640 1 -- 172.21.15.32:0/2107496185 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380b3cd0 con 0x7f9e380d4a10 2024-03-06T07:36:39.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.852+0000 7f9e3d000640 1 --2- 172.21.15.32:0/2107496185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e380d0740 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009870 tx=0x7f9e2402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6682eee8a2464bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e37e00640 1 -- 172.21.15.32:0/2107496185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24034070 con 0x7f9e380d4a10 2024-03-06T07:36:39.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e37e00640 1 -- 172.21.15.32:0/2107496185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e24030840 con 0x7f9e380d4a10 2024-03-06T07:36:39.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e37e00640 1 -- 172.21.15.32:0/2107496185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24030c10 con 0x7f9e380d4a10 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2107496185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 msgr2=0x7f9e380d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2107496185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e380d0740 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009870 tx=0x7f9e2402b980 comp rx=0 tx=0).stop 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2107496185 shutdown_connections 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2107496185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e380d0740 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2107496185 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e380cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2107496185 >> 172.21.15.32:0/2107496185 conn(0x7f9e380d3d30 msgr2=0x7f9e380d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.853+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2107496185 shutdown_connections 2024-03-06T07:36:39.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.854+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2107496185 wait complete. 2024-03-06T07:36:39.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.856+0000 7f9e3ee00640 1 Processor -- start 2024-03-06T07:36:39.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.857+0000 7f9e3ee00640 1 -- start start 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e380d0c80 0x7f9e380b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e3814adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e3806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380cfc60 con 0x7f9e380d4a10 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3815e490 con 0x7f9e380d0c80 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380b54e0 con 0x7f9e3815e0b0 2024-03-06T07:36:39.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.858+0000 7f9e3d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e3814adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e3814adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45054/0 (socket says 172.21.15.32:45054) 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 -- 172.21.15.32:0/2267065859 learned_addr learned my addr 172.21.15.32:0/2267065859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3da00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e380d0c80 0x7f9e380b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3e400640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e3806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 -- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 msgr2=0x7f9e3806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e3806bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 -- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e380d0c80 msgr2=0x7f9e380b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e380d0c80 0x7f9e380b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3d000640 1 -- 172.21.15.32:0/2267065859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380b8eb0 con 0x7f9e380d4a10 2024-03-06T07:36:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.859+0000 7f9e3da00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e380d0c80 0x7f9e380b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.860+0000 7f9e3d000640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e3814adf0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f9e24000c00 tx=0x7f9e24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.860+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24048070 con 0x7f9e380d4a10 2024-03-06T07:36:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.860+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e24030640 con 0x7f9e380d4a10 2024-03-06T07:36:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.860+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24034040 con 0x7f9e380d4a10 2024-03-06T07:36:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.860+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e380b9190 con 0x7f9e380d4a10 2024-03-06T07:36:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.860+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e380b9750 con 0x7f9e380d4a10 2024-03-06T07:36:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.862+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3804f330 con 0x7f9e380d4a10 2024-03-06T07:36:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.862+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e2404b070 con 0x7f9e380d4a10 2024-03-06T07:36:39.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.864+0000 7f9e35600640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e1806b4d0 0x7f9e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.864+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9e240ac320 con 0x7f9e380d4a10 2024-03-06T07:36:39.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.864+0000 7f9e3da00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e1806b4d0 0x7f9e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.865+0000 7f9e3da00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e1806b4d0 0x7f9e1806d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c006fd0 tx=0x7f9e2c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:39.868+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e2407b220 con 0x7f9e380d4a10 2024-03-06T07:36:40.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.159+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e380b7670 con 0x7f9e380d4a10 2024-03-06T07:36:40.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.160+0000 7f9e35600640 1 -- 172.21.15.32:0/2267065859 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f9e2407ab20 con 0x7f9e380d4a10 2024-03-06T07:36:40.163 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:40.163 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:40.163 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:40.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e1806b4d0 msgr2=0x7f9e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e1806b4d0 0x7f9e1806d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c006fd0 tx=0x7f9e2c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 msgr2=0x7f9e3814adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e3814adf0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f9e24000c00 tx=0x7f9e24004730 comp rx=0 tx=0).stop 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 shutdown_connections 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9e1806b4d0 0x7f9e1806d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9e3815e0b0 0x7f9e3806bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e380d4a10 0x7f9e3814adf0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 --2- 172.21.15.32:0/2267065859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e380d0c80 0x7f9e380b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 >> 172.21.15.32:0/2267065859 conn(0x7f9e380d3d30 msgr2=0x7f9e3806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 shutdown_connections 2024-03-06T07:36:40.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.163+0000 7f9e3ee00640 1 -- 172.21.15.32:0/2267065859 wait complete. 2024-03-06T07:36:40.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:40.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.400+0000 7f9cafe00640 1 Processor -- start 2024-03-06T07:36:40.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.400+0000 7f9cafe00640 1 -- start start 2024-03-06T07:36:40.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.401+0000 7f9cafe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9cb015efc0 0x7f9cb00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.401+0000 7f9cafe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9cb00cce90 0x7f9cb0161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.401+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb00ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.401+0000 7f9cafe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb015f3a0 con 0x7f9cb00cce90 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.401+0000 7f9cafe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb006d190 con 0x7f9cb015efc0 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.401+0000 7f9cafe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb01627f0 con 0x7f9cb0162000 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.402+0000 7f9ca7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9cb00cce90 0x7f9cb0161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.402+0000 7f9ca7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9cb00cce90 0x7f9cb0161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45064/0 (socket says 172.21.15.32:45064) 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.402+0000 7f9ca7e00640 1 -- 172.21.15.32:0/2056382948 learned_addr learned my addr 172.21.15.32:0/2056382948 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.402+0000 7f9caea00640 1 --2- 172.21.15.32:0/2056382948 >> v2:172.21.15.39:3300/0 conn(0x7f9cb015efc0 0x7f9cb00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.402+0000 7f9caf400640 1 --2- 172.21.15.32:0/2056382948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb00ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.403+0000 7f9ca7e00640 1 -- 172.21.15.32:0/2056382948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 msgr2=0x7f9cb00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:40.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.403+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2056382948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb00ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.403+0000 7f9ca7e00640 1 -- 172.21.15.32:0/2056382948 >> v2:172.21.15.39:3300/0 conn(0x7f9cb015efc0 msgr2=0x7f9cb00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:40.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.403+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2056382948 >> v2:172.21.15.39:3300/0 conn(0x7f9cb015efc0 0x7f9cb00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.403+0000 7f9ca7e00640 1 -- 172.21.15.32:0/2056382948 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb006c860 con 0x7f9cb00cce90 2024-03-06T07:36:40.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.404+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2056382948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0161ac0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f9cb00cd770 tx=0x7f9c9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a700eccb3f730c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.404+0000 7f9cae000640 1 -- 172.21.15.32:0/2056382948 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98035020 con 0x7f9cb00cce90 2024-03-06T07:36:40.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cae000640 1 -- 172.21.15.32:0/2056382948 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98031870 con 0x7f9cb00cce90 2024-03-06T07:36:40.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cae000640 1 -- 172.21.15.32:0/2056382948 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98031c40 con 0x7f9cb00cce90 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 -- 172.21.15.32:0/2056382948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 msgr2=0x7f9cb0161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2056382948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0161ac0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f9cb00cd770 tx=0x7f9c9802b910 comp rx=0 tx=0).stop 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 -- 172.21.15.32:0/2056382948 shutdown_connections 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2056382948 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb00ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2056382948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0161ac0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2056382948 >> v2:172.21.15.39:3300/0 conn(0x7f9cb015efc0 0x7f9cb00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 -- 172.21.15.32:0/2056382948 >> 172.21.15.32:0/2056382948 conn(0x7f9cb006b3d0 msgr2=0x7f9cb006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 -- 172.21.15.32:0/2056382948 shutdown_connections 2024-03-06T07:36:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.405+0000 7f9cafe00640 1 -- 172.21.15.32:0/2056382948 wait complete. 2024-03-06T07:36:40.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.407+0000 7f9cafe00640 1 Processor -- start 2024-03-06T07:36:40.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.407+0000 7f9cafe00640 1 -- start start 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.408+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.408+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb015efc0 0x7f9cb0146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.408+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb0160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.408+0000 7f9cafe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb0162490 con 0x7f9cb00cce90 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.408+0000 7f9cafe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb006ce20 con 0x7f9cb015efc0 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.408+0000 7f9cafe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb006d620 con 0x7f9cb0162000 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0157670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45080/0 (socket says 172.21.15.32:45080) 2024-03-06T07:36:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 -- 172.21.15.32:0/2299630740 learned_addr learned my addr 172.21.15.32:0/2299630740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caf400640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb0160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 -- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 msgr2=0x7f9cb0160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb015efc0 0x7f9cb0146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb0160570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 -- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb015efc0 msgr2=0x7f9cb0146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb015efc0 0x7f9cb0146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 -- 172.21.15.32:0/2299630740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb01473e0 con 0x7f9cb00cce90 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb015efc0 0x7f9cb0146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.409+0000 7f9caea00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0157670 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000e9e0 tx=0x7f9ca000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.410+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0017070 con 0x7f9cb00cce90 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.410+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca000c9d0 con 0x7f9cb00cce90 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.410+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000cdc0 con 0x7f9cb00cce90 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.410+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb01e7d00 con 0x7f9cb00cce90 2024-03-06T07:36:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.410+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb01e82c0 con 0x7f9cb00cce90 2024-03-06T07:36:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.411+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb004f330 con 0x7f9cb00cce90 2024-03-06T07:36:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.412+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ca001a070 con 0x7f9cb00cce90 2024-03-06T07:36:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.413+0000 7f9ca6a00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c8c06b4d0 0x7f9c8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.413+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9ca008a2d0 con 0x7f9cb00cce90 2024-03-06T07:36:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.413+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c8c06b4d0 0x7f9c8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.414+0000 7f9ca7e00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c8c06b4d0 0x7f9c8c06d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9c98036000 tx=0x7f9c98008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.417+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca00590c0 con 0x7f9cb00cce90 2024-03-06T07:36:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.663+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb00af2f0 con 0x7f9cb00cce90 2024-03-06T07:36:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.663+0000 7f9ca6a00640 1 -- 172.21.15.32:0/2299630740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f9ca00589c0 con 0x7f9cb00cce90 2024-03-06T07:36:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:40.666 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:14.858000+0000","modified":"2024-03-06T07:35:34.917265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7896":{"gid":7896,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6837/3465731454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":3465731454},{"type":"v1","addr":"172.21.15.32:6837","nonce":3465731454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7896,"qdb_cluster":[7896]},"id":8}]} 2024-03-06T07:36:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c8c06b4d0 msgr2=0x7f9c8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c8c06b4d0 0x7f9c8c06d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9c98036000 tx=0x7f9c98008030 comp rx=0 tx=0).stop 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 msgr2=0x7f9cb0157670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0157670 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000e9e0 tx=0x7f9ca000eea0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 shutdown_connections 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9c8c06b4d0 0x7f9c8c06d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9cb0162000 0x7f9cb0160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb015efc0 0x7f9cb0146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 --2- 172.21.15.32:0/2299630740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb00cce90 0x7f9cb0157670 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 >> 172.21.15.32:0/2299630740 conn(0x7f9cb006b3d0 msgr2=0x7f9cb0070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 shutdown_connections 2024-03-06T07:36:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.666+0000 7f9cafe00640 1 -- 172.21.15.32:0/2299630740 wait complete. 2024-03-06T07:36:40.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.898+0000 7f48d5400640 1 Processor -- start 2024-03-06T07:36:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.898+0000 7f48d5400640 1 -- start start 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48d5400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48d00b34e0 0x7f48d015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48d5400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d015cc40 con 0x7f48d00af1a0 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48d5400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d015cf20 con 0x7f48d00af580 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48d5400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d015d200 con 0x7f48d00b34e0 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48ce800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f48d00af1a0 0x7f48d00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45886/0 (socket says 172.21.15.32:45886) 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48ce800640 1 -- 172.21.15.32:0/3933956247 learned_addr learned my addr 172.21.15.32:0/3933956247 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48cf200640 1 --2- 172.21.15.32:0/3933956247 >> v2:172.21.15.32:3300/0 conn(0x7f48d00b34e0 0x7f48d015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48cf200640 1 -- 172.21.15.32:0/3933956247 >> v1:172.21.15.39:6789/0 conn(0x7f48d00af580 legacy=0x7f48d00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48cf200640 1 -- 172.21.15.32:0/3933956247 >> v1:172.21.15.32:6789/0 conn(0x7f48d00af1a0 legacy=0x7f48d00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.900+0000 7f48cf200640 1 -- 172.21.15.32:0/3933956247 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d015d4e0 con 0x7f48d00b34e0 2024-03-06T07:36:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.902+0000 7f48cf200640 1 --2- 172.21.15.32:0/3933956247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d015c600 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009d90 tx=0x7f48c802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7826cd6c165768b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.902+0000 7f48cd400640 1 -- 172.21.15.32:0/3933956247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8039070 con 0x7f48d00b34e0 2024-03-06T07:36:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.902+0000 7f48cd400640 1 -- 172.21.15.32:0/3933956247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c802cbd0 con 0x7f48d00b34e0 2024-03-06T07:36:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.902+0000 7f48cd400640 1 -- 172.21.15.32:0/3933956247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8034440 con 0x7f48d00b34e0 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.902+0000 7f48d5400640 1 -- 172.21.15.32:0/3933956247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 msgr2=0x7f48d015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.902+0000 7f48d5400640 1 --2- 172.21.15.32:0/3933956247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d015c600 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009d90 tx=0x7f48c802be20 comp rx=0 tx=0).stop 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.903+0000 7f48d5400640 1 -- 172.21.15.32:0/3933956247 shutdown_connections 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.903+0000 7f48d5400640 1 --2- 172.21.15.32:0/3933956247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d015c600 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.903+0000 7f48d5400640 1 -- 172.21.15.32:0/3933956247 >> 172.21.15.32:0/3933956247 conn(0x7f48d006afb0 msgr2=0x7f48d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.903+0000 7f48d5400640 1 -- 172.21.15.32:0/3933956247 shutdown_connections 2024-03-06T07:36:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.903+0000 7f48d5400640 1 -- 172.21.15.32:0/3933956247 wait complete. 2024-03-06T07:36:40.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.904+0000 7f48d5400640 1 Processor -- start 2024-03-06T07:36:40.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.905+0000 7f48d5400640 1 -- start start 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.905+0000 7f48d5400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48d00af1a0 0x7f48d0147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48d5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48d00af580 0x7f48d009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48d5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48d5400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0155e70 con 0x7f48d00b34e0 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48d5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d00cde30 con 0x7f48d00af580 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48d5400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0155ff0 con 0x7f48d00af1a0 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d00cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45092/0 (socket says 172.21.15.32:45092) 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 -- 172.21.15.32:0/486107541 learned_addr learned my addr 172.21.15.32:0/486107541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48ce800640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48d00af1a0 0x7f48d0147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cde00640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48d00af580 0x7f48d009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 -- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48d00af1a0 msgr2=0x7f48d0147770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48d00af1a0 0x7f48d0147770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 -- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48d00af580 msgr2=0x7f48d009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48d00af580 0x7f48d009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cf200640 1 -- 172.21.15.32:0/486107541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d0147cb0 con 0x7f48d00b34e0 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48ce800640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48d00af1a0 0x7f48d0147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.906+0000 7f48cde00640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48d00af580 0x7f48d009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:40.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.907+0000 7f48cf200640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d00cc4b0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f48c8037040 tx=0x7f48c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.907+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8039050 con 0x7f48d00b34e0 2024-03-06T07:36:40.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.907+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c8045070 con 0x7f48d00b34e0 2024-03-06T07:36:40.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.907+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c803e850 con 0x7f48d00b34e0 2024-03-06T07:36:40.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.907+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d009c6f0 con 0x7f48d00b34e0 2024-03-06T07:36:40.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.907+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d009ccb0 con 0x7f48d00b34e0 2024-03-06T07:36:40.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.909+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d004f3a0 con 0x7f48d00b34e0 2024-03-06T07:36:40.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.910+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48c803e9f0 con 0x7f48d00b34e0 2024-03-06T07:36:40.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.910+0000 7f48c6a00640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48b006b4d0 0x7f48b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.910+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f48c80ad180 con 0x7f48d00b34e0 2024-03-06T07:36:40.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.911+0000 7f48ce800640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48b006b4d0 0x7f48b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.911+0000 7f48ce800640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48b006b4d0 0x7f48b006d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f48c0000f60 tx=0x7f48c0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:40.920+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48c807bff0 con 0x7f48d00b34e0 2024-03-06T07:36:41.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.122+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48d01607f0 con 0x7f48d00b34e0 2024-03-06T07:36:41.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.124+0000 7f48c6a00640 1 -- 172.21.15.32:0/486107541 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16738 (secure 0 0 0) 0x7f48c807b8f0 con 0x7f48d00b34e0 2024-03-06T07:36:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:41.127 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":116,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:35:49.005590+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:12.716010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:13.860496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.18","mappings":[{"from":0,"to":5}]},{"pgid":"17.30","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6835/14889266":"2024-03-07T07:35:30.884623+0000","172.21.15.39:6833/14889266":"2024-03-07T07:35:30.884623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48b006b4d0 msgr2=0x7f48b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48b006b4d0 0x7f48b006d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f48c0000f60 tx=0x7f48c0009040 comp rx=0 tx=0).stop 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 msgr2=0x7f48d00cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d00cc4b0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f48c8037040 tx=0x7f48c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 shutdown_connections 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f48b006b4d0 0x7f48b006d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d00b34e0 0x7f48d00cc4b0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48d00af580 0x7f48d009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 --2- 172.21.15.32:0/486107541 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f48d00af1a0 0x7f48d0147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 >> 172.21.15.32:0/486107541 conn(0x7f48d006afb0 msgr2=0x7f48d00add20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 shutdown_connections 2024-03-06T07:36:41.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.128+0000 7f48d5400640 1 -- 172.21.15.32:0/486107541 wait complete. 2024-03-06T07:36:41.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:36:41.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.374+0000 7f287da00640 1 Processor -- start 2024-03-06T07:36:41.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.374+0000 7f287da00640 1 -- start start 2024-03-06T07:36:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.376+0000 7f287da00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28780af580 0x7f287815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.376+0000 7f287da00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287815cc40 con 0x7f28780cdc20 2024-03-06T07:36:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.376+0000 7f287da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287815cf20 con 0x7f28780af1a0 2024-03-06T07:36:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.376+0000 7f287da00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287815d200 con 0x7f28780af580 2024-03-06T07:36:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2876000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28780af580 0x7f287815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2877400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f28780cdc20 0x7f28780b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45888/0 (socket says 172.21.15.32:45888) 2024-03-06T07:36:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2877400640 1 -- 172.21.15.32:0/959526999 learned_addr learned my addr 172.21.15.32:0/959526999 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2876000640 1 -- 172.21.15.32:0/959526999 >> v1:172.21.15.39:6789/0 conn(0x7f28780af1a0 legacy=0x7f28780b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2876000640 1 -- 172.21.15.32:0/959526999 >> v1:172.21.15.32:6789/0 conn(0x7f28780cdc20 legacy=0x7f28780b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2876000640 1 -- 172.21.15.32:0/959526999 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287815d4e0 con 0x7f28780af580 2024-03-06T07:36:41.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.377+0000 7f2875600640 1 -- 172.21.15.32:0/959526999 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593639736 0 0) 0x7f287815cc40 con 0x7f28780cdc20 2024-03-06T07:36:41.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.378+0000 7f2876000640 1 --2- 172.21.15.32:0/959526999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780af580 0x7f287815c600 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f28680097a0 tx=0x7f286802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0e49eec9f70d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f2875600640 1 -- 172.21.15.32:0/959526999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28680315a0 con 0x7f28780af580 2024-03-06T07:36:41.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f2875600640 1 -- 172.21.15.32:0/959526999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2868037070 con 0x7f28780af580 2024-03-06T07:36:41.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f2875600640 1 -- 172.21.15.32:0/959526999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868031960 con 0x7f28780af580 2024-03-06T07:36:41.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 -- 172.21.15.32:0/959526999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780af580 msgr2=0x7f287815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 --2- 172.21.15.32:0/959526999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780af580 0x7f287815c600 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f28680097a0 tx=0x7f286802b840 comp rx=0 tx=0).stop 2024-03-06T07:36:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 -- 172.21.15.32:0/959526999 shutdown_connections 2024-03-06T07:36:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 --2- 172.21.15.32:0/959526999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780af580 0x7f287815c600 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 -- 172.21.15.32:0/959526999 >> 172.21.15.32:0/959526999 conn(0x7f287806afb0 msgr2=0x7f287806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 -- 172.21.15.32:0/959526999 shutdown_connections 2024-03-06T07:36:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.379+0000 7f287da00640 1 -- 172.21.15.32:0/959526999 wait complete. 2024-03-06T07:36:41.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.381+0000 7f287da00640 1 Processor -- start 2024-03-06T07:36:41.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.381+0000 7f287da00640 1 -- start start 2024-03-06T07:36:41.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f287da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28780af1a0 0x7f2878148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f287da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28780af580 0x7f28781f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f287da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 0x7f28781f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f287da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2878155e70 con 0x7f28780cdc20 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f287da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780b3730 con 0x7f28780af1a0 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f287da00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2878155ff0 con 0x7f28780af580 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.382+0000 7f2877400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 0x7f28781f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 0x7f28781f80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45108/0 (socket says 172.21.15.32:45108) 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 -- 172.21.15.32:0/1408220598 learned_addr learned my addr 172.21.15.32:0/1408220598 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2876a00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28780af1a0 0x7f2878148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2876000640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28780af580 0x7f28781f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 -- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28780af580 msgr2=0x7f28781f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28780af580 0x7f28781f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 -- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28780af1a0 msgr2=0x7f2878148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28780af1a0 0x7f2878148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 -- 172.21.15.32:0/1408220598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287809d840 con 0x7f28780cdc20 2024-03-06T07:36:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.383+0000 7f2877400640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 0x7f28781f80a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f287000c700 tx=0x7f287000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.384+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870018070 con 0x7f28780cdc20 2024-03-06T07:36:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.384+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287000ce10 con 0x7f28780cdc20 2024-03-06T07:36:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.384+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000a860 con 0x7f28780cdc20 2024-03-06T07:36:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.384+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287815e6a0 con 0x7f28780cdc20 2024-03-06T07:36:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.384+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287815ec60 con 0x7f28780cdc20 2024-03-06T07:36:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.385+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287804f3a0 con 0x7f28780cdc20 2024-03-06T07:36:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.386+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f287001b070 con 0x7f28780cdc20 2024-03-06T07:36:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.387+0000 7f286ea00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f285806b4d0 0x7f285806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.387+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2870089800 con 0x7f28780cdc20 2024-03-06T07:36:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.387+0000 7f2876a00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f285806b4d0 0x7f285806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.388+0000 7f2876a00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f285806b4d0 0x7f285806d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2864004ea0 tx=0x7f2864009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.392+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2870058680 con 0x7f28780cdc20 2024-03-06T07:36:41.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.631+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f28780ae240 con 0x7f28780cdc20 2024-03-06T07:36:41.634 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-06T07:36:41.632+0000 7fd1b4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:36:41.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.635+0000 7f286ea00640 1 -- 172.21.15.32:0/1408220598 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2870057f80 con 0x7f28780cdc20 2024-03-06T07:36:41.638 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:36:41.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.638+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f285806b4d0 msgr2=0x7f285806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.638+0000 7f287da00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f285806b4d0 0x7f285806d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2864004ea0 tx=0x7f2864009040 comp rx=0 tx=0).stop 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 msgr2=0x7f28781f80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 0x7f28781f80a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f287000c700 tx=0x7f287000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 shutdown_connections 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f285806b4d0 0x7f285806d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28780cdc20 0x7f28781f80a0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f28780af580 0x7f28781f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 --2- 172.21.15.32:0/1408220598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28780af1a0 0x7f2878148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 >> 172.21.15.32:0/1408220598 conn(0x7f287806afb0 msgr2=0x7f28780acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 shutdown_connections 2024-03-06T07:36:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.639+0000 7f287da00640 1 -- 172.21.15.32:0/1408220598 wait complete. 2024-03-06T07:36:41.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:36:41.668 INFO:tasks.ceph.mds.a.smithi032.stderr: -7497> 2024-03-06T07:35:37.907+0000 7f7d83800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:36:41.747 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:36:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.850+0000 7f94cac00640 1 Processor -- start 2024-03-06T07:36:41.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.850+0000 7f94cac00640 1 -- start start 2024-03-06T07:36:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.851+0000 7f94cac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f94c40729d0 0x7f94c40b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.851+0000 7f94cac00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4071930 con 0x7f94c40b3c20 2024-03-06T07:36:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.851+0000 7f94cac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4071c10 con 0x7f94c40b1b40 2024-03-06T07:36:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.851+0000 7f94cac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c415bfb0 con 0x7f94c40729d0 2024-03-06T07:36:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.851+0000 7f94c9800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f94c40b3c20 0x7f94c415b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45902/0 (socket says 172.21.15.32:45902) 2024-03-06T07:36:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c9800640 1 -- 172.21.15.32:0/3524958998 learned_addr learned my addr 172.21.15.32:0/3524958998 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041500202 0 0) 0x7f94c4071930 con 0x7f94c40b3c20 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac003520 con 0x7f94c40b3c20 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c8e00640 1 --2- 172.21.15.32:0/3524958998 >> v2:172.21.15.39:3300/0 conn(0x7f94c40729d0 0x7f94c40b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988553439 0 0) 0x7f94ac003520 con 0x7f94c40b3c20 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 >> v2:172.21.15.39:3300/0 conn(0x7f94c40729d0 msgr2=0x7f94c40b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 --2- 172.21.15.32:0/3524958998 >> v2:172.21.15.39:3300/0 conn(0x7f94c40729d0 0x7f94c40b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 >> v1:172.21.15.39:6789/0 conn(0x7f94c40b1b40 legacy=0x7f94c415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4071ef0 con 0x7f94c40b3c20 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f94b8001670 con 0x7f94c40b3c20 2024-03-06T07:36:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 >> v1:172.21.15.32:6789/0 conn(0x7f94c40b3c20 legacy=0x7f94c415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.852+0000 7f94c3e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94ac003a00 0x7f94ac003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c3e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94ac004800 0x7f94ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c3e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac005230 0x7f94ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c415bfb0 con 0x7f94ac004800 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac00c190 con 0x7f94ac005230 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac00c390 con 0x7f94ac003a00 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c8e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94ac004800 0x7f94ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c9800640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94ac003a00 0x7f94ac003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c8e00640 1 -- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94ac003a00 msgr2=0x7f94ac003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c8e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94ac003a00 0x7f94ac003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c8e00640 1 -- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac005230 msgr2=0x7f94ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c8e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac005230 0x7f94ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94c8e00640 1 -- 172.21.15.32:0/3524958998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac003520 con 0x7f94ac004800 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.853+0000 7f94ca200640 1 -- 172.21.15.32:0/3524958998 reap_dead start 2024-03-06T07:36:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.854+0000 7f94c8e00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94ac004800 0x7f94ac004c80 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f94b000ba30 tx=0x7f94ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1334eb2fa2e4a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0036020 con 0x7f94ac004800 2024-03-06T07:36:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0032660 con 0x7f94ac004800 2024-03-06T07:36:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94c3e00640 1 -- 172.21.15.32:0/3524958998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b000b600 con 0x7f94ac004800 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 -- 172.21.15.32:0/3524958998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94ac004800 msgr2=0x7f94ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94ac004800 0x7f94ac004c80 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f94b000ba30 tx=0x7f94ac005830 comp rx=0 tx=0).stop 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 -- 172.21.15.32:0/3524958998 shutdown_connections 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 --2- 172.21.15.32:0/3524958998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94ac004800 0x7f94ac004c80 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 -- 172.21.15.32:0/3524958998 >> 172.21.15.32:0/3524958998 conn(0x7f94c406b3a0 msgr2=0x7f94c406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 -- 172.21.15.32:0/3524958998 shutdown_connections 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.855+0000 7f94cac00640 1 -- 172.21.15.32:0/3524958998 wait complete. 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.857+0000 7f94cac00640 1 Processor -- start 2024-03-06T07:36:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.857+0000 7f94cac00640 1 -- start start 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94cac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac004800 0x7f94c4142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94cac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94c4151ae0 0x7f94c41f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94cac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 0x7f94c415a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94cac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40b4000 con 0x7f94c4159cd0 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94cac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40724b0 con 0x7f94ac004800 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94cac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40b4180 con 0x7f94c4151ae0 2024-03-06T07:36:41.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.858+0000 7f94ca200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 0x7f94c415a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 0x7f94c415a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45130/0 (socket says 172.21.15.32:45130) 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 -- 172.21.15.32:0/3773829754 learned_addr learned my addr 172.21.15.32:0/3773829754 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94c8e00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94c4151ae0 0x7f94c41f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 -- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94c4151ae0 msgr2=0x7f94c41f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94c9800640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac004800 0x7f94c4142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94c4151ae0 0x7f94c41f5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 -- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac004800 msgr2=0x7f94c4142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac004800 0x7f94c4142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 -- 172.21.15.32:0/3773829754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c41f5990 con 0x7f94c4159cd0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94c9800640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac004800 0x7f94c4142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94ca200640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 0x7f94c415a180 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f94c4072f30 tx=0x7f94bc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.859+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc017070 con 0x7f94c4159cd0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.860+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc007070 con 0x7f94c4159cd0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.860+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc007460 con 0x7f94c4159cd0 2024-03-06T07:36:41.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.860+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c41f5c70 con 0x7f94c4159cd0 2024-03-06T07:36:41.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.860+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c4153c50 con 0x7f94c4159cd0 2024-03-06T07:36:41.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.861+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c404f330 con 0x7f94c4159cd0 2024-03-06T07:36:41.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.862+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94bc01a070 con 0x7f94c4159cd0 2024-03-06T07:36:41.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.863+0000 7f94c1600640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94a406b4d0 0x7f94a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.863+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f94bc089550 con 0x7f94c4159cd0 2024-03-06T07:36:41.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.863+0000 7f94c9800640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94a406b4d0 0x7f94a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.864+0000 7f94c9800640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94a406b4d0 0x7f94a406d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f94b8000c00 tx=0x7f94b8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:41.867+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94bc0582d0 con 0x7f94c4159cd0 2024-03-06T07:36:42.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.095+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f94c40b2140 con 0x7f94c4159cd0 2024-03-06T07:36:42.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.639+0000 7f94c1600640 1 -- 172.21.15.32:0/3773829754 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) v1 ==== 104+0+0 (secure 0 0 0) 0x7f94bc057bd0 con 0x7f94c4159cd0 2024-03-06T07:36:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.644+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94a406b4d0 msgr2=0x7f94a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.644+0000 7f94cac00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94a406b4d0 0x7f94a406d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f94b8000c00 tx=0x7f94b8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.644+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 msgr2=0x7f94c415a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.644+0000 7f94cac00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 0x7f94c415a180 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f94c4072f30 tx=0x7f94bc00c960 comp rx=0 tx=0).stop 2024-03-06T07:36:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 shutdown_connections 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f94a406b4d0 0x7f94a406d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c4159cd0 0x7f94c415a180 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f94c4151ae0 0x7f94c41f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 --2- 172.21.15.32:0/3773829754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94ac004800 0x7f94c4142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 >> 172.21.15.32:0/3773829754 conn(0x7f94c406b3a0 msgr2=0x7f94c406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 shutdown_connections 2024-03-06T07:36:42.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.645+0000 7f94cac00640 1 -- 172.21.15.32:0/3773829754 wait complete. 2024-03-06T07:36:42.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:36:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.941+0000 7fbaeb200640 1 Processor -- start 2024-03-06T07:36:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.941+0000 7fbaeb200640 1 -- start start 2024-03-06T07:36:42.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbaeb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec15ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbaeb200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbaec0b3d50 0x7fbaec0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbaeb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbaec1618b0 0x7fbaec161d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbaeb200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaec0ad880 con 0x7fbaec0b3d50 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbaeb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaec0b46f0 con 0x7fbaec1618b0 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbaeb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec15f280 con 0x7fbaec0cf090 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.943+0000 7fbae9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbaec0b3d50 0x7fbaec0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbaec0b3d50 0x7fbaec0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45140/0 (socket says 172.21.15.32:45140) 2024-03-06T07:36:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 -- 172.21.15.32:0/2013560485 learned_addr learned my addr 172.21.15.32:0/2013560485 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9e00640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec15ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 -- 172.21.15.32:0/2013560485 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 msgr2=0x7fbaec15ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec15ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 -- 172.21.15.32:0/2013560485 >> v2:172.21.15.39:3300/0 conn(0x7fbaec1618b0 msgr2=0x7fbaec161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 --2- 172.21.15.32:0/2013560485 >> v2:172.21.15.39:3300/0 conn(0x7fbaec1618b0 0x7fbaec161d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9400640 1 -- 172.21.15.32:0/2013560485 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0cf470 con 0x7fbaec0b3d50 2024-03-06T07:36:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.944+0000 7fbae9e00640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.946+0000 7fbae9400640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec0b3d50 0x7fbaec0b41b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbaec15f430 tx=0x7fbad802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa3d9ab8bc31ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.946+0000 7fbae8a00640 1 -- 172.21.15.32:0/2013560485 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8038070 con 0x7fbaec0b3d50 2024-03-06T07:36:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.946+0000 7fbae8a00640 1 -- 172.21.15.32:0/2013560485 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad8031740 con 0x7fbaec0b3d50 2024-03-06T07:36:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.946+0000 7fbae8a00640 1 -- 172.21.15.32:0/2013560485 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad802f050 con 0x7fbaec0b3d50 2024-03-06T07:36:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 -- 172.21.15.32:0/2013560485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec0b3d50 msgr2=0x7fbaec0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec0b3d50 0x7fbaec0b41b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbaec15f430 tx=0x7fbad802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 -- 172.21.15.32:0/2013560485 shutdown_connections 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 --2- 172.21.15.32:0/2013560485 >> v2:172.21.15.39:3300/0 conn(0x7fbaec1618b0 0x7fbaec161d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec0b3d50 0x7fbaec0b41b0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 --2- 172.21.15.32:0/2013560485 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 -- 172.21.15.32:0/2013560485 >> 172.21.15.32:0/2013560485 conn(0x7fbaec06afb0 msgr2=0x7fbaec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 -- 172.21.15.32:0/2013560485 shutdown_connections 2024-03-06T07:36:42.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.947+0000 7fbaeb200640 1 -- 172.21.15.32:0/2013560485 wait complete. 2024-03-06T07:36:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.949+0000 7fbaeb200640 1 Processor -- start 2024-03-06T07:36:42.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.949+0000 7fbaeb200640 1 -- start start 2024-03-06T07:36:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaeb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaec0b3d50 0x7fbaec13cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaeb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaeb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 0x7fbaec1f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaeb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0b3340 con 0x7fbaec1618b0 2024-03-06T07:36:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaeb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0b3060 con 0x7fbaec0b3d50 2024-03-06T07:36:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaeb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0b3620 con 0x7fbaec0cf090 2024-03-06T07:36:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 0x7fbaec1f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 0x7fbaec1f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45142/0 (socket says 172.21.15.32:45142) 2024-03-06T07:36:42.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.950+0000 7fbaea800640 1 -- 172.21.15.32:0/1633797836 learned_addr learned my addr 172.21.15.32:0/1633797836 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae9e00640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaec0b3d50 0x7fbaec13cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae9400640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbaea800640 1 -- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 msgr2=0x7fbaec142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbaea800640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec142ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbaea800640 1 -- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaec0b3d50 msgr2=0x7fbaec13cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbaea800640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaec0b3d50 0x7fbaec13cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbaea800640 1 -- 172.21.15.32:0/1633797836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec13da50 con 0x7fbaec1618b0 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae9e00640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaec0b3d50 0x7fbaec13cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae9400640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec142ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbaea800640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 0x7fbaec1f7420 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b9a0 tx=0x7fbae400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4019070 con 0x7fbaec1618b0 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae400cbc0 con 0x7fbaec1618b0 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.951+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4014440 con 0x7fbaec1618b0 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.952+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec142670 con 0x7fbaec1618b0 2024-03-06T07:36:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.952+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec1e7960 con 0x7fbaec1618b0 2024-03-06T07:36:42.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.953+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaec158610 con 0x7fbaec1618b0 2024-03-06T07:36:42.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.954+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbae401c070 con 0x7fbaec1618b0 2024-03-06T07:36:42.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.955+0000 7fbae2000640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbacc06b4d0 0x7fbacc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.955+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbae4089fe0 con 0x7fbaec1618b0 2024-03-06T07:36:42.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.955+0000 7fbae9e00640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbacc06b4d0 0x7fbacc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.956+0000 7fbae9e00640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbacc06b4d0 0x7fbacc06d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fbadc006fd0 tx=0x7fbadc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:42.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:42.959+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaec158610 con 0x7fbaec1618b0 2024-03-06T07:36:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.160+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbaec04f330 con 0x7fbaec1618b0 2024-03-06T07:36:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.651+0000 7fbae2000640 1 -- 172.21.15.32:0/1633797836 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbae4058e00 con 0x7fbaec1618b0 2024-03-06T07:36:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:36:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbacc06b4d0 msgr2=0x7fbacc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbacc06b4d0 0x7fbacc06d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fbadc006fd0 tx=0x7fbadc008040 comp rx=0 tx=0).stop 2024-03-06T07:36:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 msgr2=0x7fbaec1f7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 0x7fbaec1f7420 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b9a0 tx=0x7fbae400be60 comp rx=0 tx=0).stop 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 shutdown_connections 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fbacc06b4d0 0x7fbacc06d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaec1618b0 0x7fbaec1f7420 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fbaec0cf090 0x7fbaec142ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 --2- 172.21.15.32:0/1633797836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaec0b3d50 0x7fbaec13cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.654+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 >> 172.21.15.32:0/1633797836 conn(0x7fbaec06afb0 msgr2=0x7fbaec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.655+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 shutdown_connections 2024-03-06T07:36:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.655+0000 7fbaeb200640 1 -- 172.21.15.32:0/1633797836 wait complete. 2024-03-06T07:36:43.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:36:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.887+0000 7f499ac00640 1 Processor -- start 2024-03-06T07:36:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.887+0000 7f499ac00640 1 -- start start 2024-03-06T07:36:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f499ac00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49940aced0 0x7f49940b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f499ac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499406cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f499ac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49941531f0 con 0x7f49940ae7b0 2024-03-06T07:36:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f499ac00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49940b3b80 con 0x7f49940aced0 2024-03-06T07:36:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f499ac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940d04d0 con 0x7f499415cdc0 2024-03-06T07:36:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f4993400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49940aced0 0x7f49940b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f4993e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f49940ae7b0 0x7f49940aeb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:55850/0 (socket says 172.21.15.32:55850) 2024-03-06T07:36:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.889+0000 7f4993e00640 1 -- 172.21.15.32:0/2497039209 learned_addr learned my addr 172.21.15.32:0/2497039209 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.890+0000 7f4998c00640 1 --2- 172.21.15.32:0/2497039209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499406cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.890+0000 7f4993400640 1 -- 172.21.15.32:0/2497039209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 msgr2=0x7f499406cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.890+0000 7f4993400640 1 --2- 172.21.15.32:0/2497039209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499406cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.890+0000 7f4993400640 1 -- 172.21.15.32:0/2497039209 >> v1:172.21.15.39:6789/0 conn(0x7f49940ae7b0 legacy=0x7f49940aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.890+0000 7f4993400640 1 -- 172.21.15.32:0/2497039209 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940b2b80 con 0x7f49940aced0 2024-03-06T07:36:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.891+0000 7f4993400640 1 --2- 172.21.15.32:0/2497039209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940aced0 0x7f49940b3640 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f49800097a0 tx=0x7f498002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1622f9de4c66ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.891+0000 7f4992a00640 1 -- 172.21.15.32:0/2497039209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980035020 con 0x7f49940aced0 2024-03-06T07:36:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.891+0000 7f4992a00640 1 -- 172.21.15.32:0/2497039209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4980031810 con 0x7f49940aced0 2024-03-06T07:36:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.891+0000 7f4992a00640 1 -- 172.21.15.32:0/2497039209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980031be0 con 0x7f49940aced0 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 -- 172.21.15.32:0/2497039209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940aced0 msgr2=0x7f49940b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 --2- 172.21.15.32:0/2497039209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940aced0 0x7f49940b3640 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f49800097a0 tx=0x7f498002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 -- 172.21.15.32:0/2497039209 shutdown_connections 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 --2- 172.21.15.32:0/2497039209 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499406cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 --2- 172.21.15.32:0/2497039209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940aced0 0x7f49940b3640 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 -- 172.21.15.32:0/2497039209 >> 172.21.15.32:0/2497039209 conn(0x7f499406afb0 msgr2=0x7f499406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 -- 172.21.15.32:0/2497039209 shutdown_connections 2024-03-06T07:36:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.892+0000 7f499ac00640 1 -- 172.21.15.32:0/2497039209 wait complete. 2024-03-06T07:36:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.894+0000 7f499ac00640 1 Processor -- start 2024-03-06T07:36:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.894+0000 7f499ac00640 1 -- start start 2024-03-06T07:36:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.895+0000 7f499ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49940aced0 0x7f499414a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.895+0000 7f499ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 0x7f49941546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.895+0000 7f499ac00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.895+0000 7f499ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940b2e40 con 0x7f49940ae7b0 2024-03-06T07:36:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.895+0000 7f499ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499406eeb0 con 0x7f49940aced0 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.895+0000 7f499ac00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940b3120 con 0x7f499415cdc0 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 0x7f49941546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 0x7f49941546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45168/0 (socket says 172.21.15.32:45168) 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 -- 172.21.15.32:0/2356411642 learned_addr learned my addr 172.21.15.32:0/2356411642 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 -- 172.21.15.32:0/2356411642 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 msgr2=0x7f499415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 -- 172.21.15.32:0/2356411642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49940aced0 msgr2=0x7f499414a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49940aced0 0x7f499414a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 -- 172.21.15.32:0/2356411642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49941550a0 con 0x7f49940ae7b0 2024-03-06T07:36:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.896+0000 7f4993400640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 0x7f49941546c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f4980009890 tx=0x7f49800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.897+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980041070 con 0x7f49940ae7b0 2024-03-06T07:36:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.897+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498003ae50 con 0x7f49940ae7b0 2024-03-06T07:36:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.897+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980035040 con 0x7f49940ae7b0 2024-03-06T07:36:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.897+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4994155380 con 0x7f49940ae7b0 2024-03-06T07:36:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.897+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4994155940 con 0x7f49940ae7b0 2024-03-06T07:36:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.898+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4994068010 con 0x7f49940ae7b0 2024-03-06T07:36:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.899+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f498004c060 con 0x7f49940ae7b0 2024-03-06T07:36:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.900+0000 7f498be00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f497406b520 0x7f497406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.900+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5501+0+0 (secure 0 0 0) 0x7f49800abd10 con 0x7f49940ae7b0 2024-03-06T07:36:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.900+0000 7f4993e00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f497406b520 0x7f497406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.901+0000 7f4993e00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f497406b520 0x7f497406d9e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4984000f60 tx=0x7f4984007a80 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:43.905+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49800b1020 con 0x7f49940ae7b0 2024-03-06T07:36:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.107+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f49940b4670 con 0x7f49940ae7b0 2024-03-06T07:36:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.663+0000 7f498be00640 1 -- 172.21.15.32:0/2356411642 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) v1 ==== 169+0+0 (secure 0 0 0) 0x7f498007ac40 con 0x7f49940ae7b0 2024-03-06T07:36:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.667+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f497406b520 msgr2=0x7f497406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.667+0000 7f499ac00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f497406b520 0x7f497406d9e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4984000f60 tx=0x7f4984007a80 comp rx=0 tx=0).stop 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.667+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 msgr2=0x7f49941546c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.667+0000 7f499ac00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 0x7f49941546c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f4980009890 tx=0x7f49800046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 shutdown_connections 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f497406b520 0x7f497406d9e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f499415cdc0 0x7f499415e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49940ae7b0 0x7f49941546c0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 --2- 172.21.15.32:0/2356411642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49940aced0 0x7f499414a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 >> 172.21.15.32:0/2356411642 conn(0x7f499406afb0 msgr2=0x7f4994070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 shutdown_connections 2024-03-06T07:36:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.668+0000 7f499ac00640 1 -- 172.21.15.32:0/2356411642 wait complete. 2024-03-06T07:36:44.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:36:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.892+0000 7f678d400640 1 Processor -- start 2024-03-06T07:36:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.892+0000 7f678d400640 1 -- start start 2024-03-06T07:36:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.893+0000 7f678d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b3c20 0x7f67880729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.893+0000 7f678d400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67880b1b40 0x7f67880b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.893+0000 7f678d400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6788071930 con 0x7f67880b4000 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.893+0000 7f678d400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67880b24e0 con 0x7f67880b1b40 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.893+0000 7f678d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678815bfb0 con 0x7f67880b3c20 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67880b1b40 0x7f67880b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67880b1b40 0x7f67880b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45182/0 (socket says 172.21.15.32:45182) 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 -- 172.21.15.32:0/898028271 learned_addr learned my addr 172.21.15.32:0/898028271 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f6787e00640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b3c20 0x7f67880729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 -- 172.21.15.32:0/898028271 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b3c20 msgr2=0x7f67880729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b3c20 0x7f67880729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 -- 172.21.15.32:0/898028271 >> v1:172.21.15.39:6789/0 conn(0x7f67880b4000 legacy=0x7f678815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f678ca00640 1 -- 172.21.15.32:0/898028271 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788071c10 con 0x7f67880b1b40 2024-03-06T07:36:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.894+0000 7f6787e00640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b3c20 0x7f67880729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.896+0000 7f678ca00640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b1b40 0x7f67880b1fa0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f6780009e50 tx=0x7f678002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f8fedf4e66c6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.896+0000 7f6786a00640 1 -- 172.21.15.32:0/898028271 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678002ca80 con 0x7f67880b1b40 2024-03-06T07:36:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.896+0000 7f6786a00640 1 -- 172.21.15.32:0/898028271 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678002cc20 con 0x7f67880b1b40 2024-03-06T07:36:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.896+0000 7f6786a00640 1 -- 172.21.15.32:0/898028271 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67800343f0 con 0x7f67880b1b40 2024-03-06T07:36:44.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 -- 172.21.15.32:0/898028271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b1b40 msgr2=0x7f67880b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b1b40 0x7f67880b1fa0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f6780009e50 tx=0x7f678002bee0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 -- 172.21.15.32:0/898028271 shutdown_connections 2024-03-06T07:36:44.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b1b40 0x7f67880b1fa0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 --2- 172.21.15.32:0/898028271 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b3c20 0x7f67880729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 -- 172.21.15.32:0/898028271 >> 172.21.15.32:0/898028271 conn(0x7f678806b3a0 msgr2=0x7f678806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:44.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 -- 172.21.15.32:0/898028271 shutdown_connections 2024-03-06T07:36:44.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.897+0000 7f678d400640 1 -- 172.21.15.32:0/898028271 wait complete. 2024-03-06T07:36:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.899+0000 7f678d400640 1 Processor -- start 2024-03-06T07:36:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.899+0000 7f678d400640 1 -- start start 2024-03-06T07:36:44.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f678d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b1b40 0x7f6788154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f678d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 0x7f6788145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f678d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67880b4000 0x7f678814f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f678d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67880721d0 con 0x7f67880b3c20 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f678d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67880b4460 con 0x7f67880b4000 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f678d400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880724b0 con 0x7f67880b1b40 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f6787400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 0x7f6788145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f6787400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 0x7f6788145570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45184/0 (socket says 172.21.15.32:45184) 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.900+0000 7f6787400640 1 -- 172.21.15.32:0/3254665411 learned_addr learned my addr 172.21.15.32:0/3254665411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f678ca00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67880b4000 0x7f678814f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787e00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b1b40 0x7f6788154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787400640 1 -- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b1b40 msgr2=0x7f6788154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b1b40 0x7f6788154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787400640 1 -- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67880b4000 msgr2=0x7f678814f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67880b4000 0x7f678814f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787400640 1 -- 172.21.15.32:0/3254665411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788150680 con 0x7f67880b3c20 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f678ca00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67880b4000 0x7f678814f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787e00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b1b40 0x7f6788154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f6787400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 0x7f6788145570 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f677000c430 tx=0x7f677000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770011790 con 0x7f67880b3c20 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770017070 con 0x7f67880b3c20 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6788150960 con 0x7f67880b3c20 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770011bb0 con 0x7f67880b3c20 2024-03-06T07:36:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.901+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6788150f20 con 0x7f67880b3c20 2024-03-06T07:36:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.903+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678804f330 con 0x7f67880b3c20 2024-03-06T07:36:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.904+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f677001c460 con 0x7f67880b3c20 2024-03-06T07:36:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.904+0000 7f677fe00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.905+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6770089dc0 con 0x7f67880b3c20 2024-03-06T07:36:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.905+0000 7f6787e00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.906+0000 7f6787e00640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f676406b4d0 0x7f676406d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6778006fd0 tx=0x7f6778008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:44.910+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6770058f70 con 0x7f67880b3c20 2024-03-06T07:36:45.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.111+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6788151120 con 0x7f67880b3c20 2024-03-06T07:36:45.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.116+0000 7f677fe00640 1 -- 172.21.15.32:0/3254665411 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6770058870 con 0x7f67880b3c20 2024-03-06T07:36:45.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.118+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f676406b4d0 msgr2=0x7f676406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.118+0000 7f678d400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f676406b4d0 0x7f676406d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6778006fd0 tx=0x7f6778008040 comp rx=0 tx=0).stop 2024-03-06T07:36:45.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.118+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 msgr2=0x7f6788145570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 0x7f6788145570 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f677000c430 tx=0x7f677000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 shutdown_connections 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67880b4000 0x7f678814f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67880b3c20 0x7f6788145570 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 --2- 172.21.15.32:0/3254665411 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67880b1b40 0x7f6788154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 >> 172.21.15.32:0/3254665411 conn(0x7f678806b3a0 msgr2=0x7f678815c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 shutdown_connections 2024-03-06T07:36:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.119+0000 7f678d400640 1 -- 172.21.15.32:0/3254665411 wait complete. 2024-03-06T07:36:45.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:36:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.363+0000 7f2bb4a00640 1 Processor -- start 2024-03-06T07:36:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.363+0000 7f2bb4a00640 1 -- start start 2024-03-06T07:36:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.364+0000 7f2bb4a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2bb00d2700 0x7f2bb00c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2bb4a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb00c5400 con 0x7f2bb00c4ff0 2024-03-06T07:36:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2bb4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb00cebc0 con 0x7f2bb0162410 2024-03-06T07:36:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2bb4a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb00cf450 con 0x7f2bb00d2700 2024-03-06T07:36:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2baf200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2bb00c4ff0 0x7f2bb00af1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45918/0 (socket says 172.21.15.32:45918) 2024-03-06T07:36:45.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2baf200640 1 -- 172.21.15.32:0/2850386080 learned_addr learned my addr 172.21.15.32:0/2850386080 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200231071 0 0) 0x7f2ba8001110 con 0x7f2bb00c4ff0 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2bade00640 1 --2- 172.21.15.32:0/2850386080 >> v2:172.21.15.39:3300/0 conn(0x7f2bb00d2700 0x7f2bb00c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.365+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b94003520 con 0x7f2bb00c4ff0 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 152185326 0 0) 0x7f2b94003520 con 0x7f2bb00c4ff0 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 >> v2:172.21.15.39:3300/0 conn(0x7f2bb00d2700 msgr2=0x7f2bb00c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 --2- 172.21.15.32:0/2850386080 >> v2:172.21.15.39:3300/0 conn(0x7f2bb00d2700 0x7f2bb00c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 >> v1:172.21.15.39:6789/0 conn(0x7f2bb0162410 legacy=0x7f2bb00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb00cee90 con 0x7f2bb00c4ff0 2024-03-06T07:36:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f2ba8001840 con 0x7f2bb00c4ff0 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 >> v1:172.21.15.32:6789/0 conn(0x7f2bb00c4ff0 legacy=0x7f2bb00af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2b94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b94004800 0x7f2b94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b940052e0 0x7f2b9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00cf450 con 0x7f2b940039b0 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400c1b0 con 0x7f2b940052e0 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400c3b0 con 0x7f2b94004800 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bade00640 1 --2- 172.21.15.32:0/2850386080 >> v2:172.21.15.39:3300/0 conn(0x7f2bb00d2700 0x7f2bb00c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bae800640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2b94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.366+0000 7f2bade00640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b94004800 0x7f2b94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2bae800640 1 -- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b94004800 msgr2=0x7f2b94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2bae800640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b94004800 0x7f2b94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2baf200640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b940052e0 0x7f2b9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2bae800640 1 -- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b940052e0 msgr2=0x7f2b9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2bae800640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b940052e0 0x7f2b9400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2bae800640 1 -- 172.21.15.32:0/2850386080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94003520 con 0x7f2b940039b0 2024-03-06T07:36:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.367+0000 7f2baf200640 1 -- 172.21.15.32:0/2850386080 reap_dead start 2024-03-06T07:36:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.368+0000 7f2bae800640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2b94003dd0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0001e20 tx=0x7f2b94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6942852f778d90d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.368+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0001a70 con 0x7f2b940039b0 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.368+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba0003b70 con 0x7f2b940039b0 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.368+0000 7f2bad400640 1 -- 172.21.15.32:0/2850386080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00050d0 con 0x7f2b940039b0 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 -- 172.21.15.32:0/2850386080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 msgr2=0x7f2b94003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2b94003dd0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0001e20 tx=0x7f2b94005830 comp rx=0 tx=0).stop 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 -- 172.21.15.32:0/2850386080 shutdown_connections 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/2850386080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2b94003dd0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 -- 172.21.15.32:0/2850386080 >> 172.21.15.32:0/2850386080 conn(0x7f2bb00d1ae0 msgr2=0x7f2bb00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 -- 172.21.15.32:0/2850386080 shutdown_connections 2024-03-06T07:36:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.369+0000 7f2bb4a00640 1 -- 172.21.15.32:0/2850386080 wait complete. 2024-03-06T07:36:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.371+0000 7f2bb4a00640 1 Processor -- start 2024-03-06T07:36:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.371+0000 7f2bb4a00640 1 -- start start 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bb4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2bb01e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bb4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2bb01e9c30 0x7f2bb01554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bb4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01ea0b0 0x7f2bb01559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bb4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00aecc0 con 0x7f2b940039b0 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bb4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00ae9e0 con 0x7f2bb01ea0b0 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bb4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb00c4ab0 con 0x7f2bb01e9c30 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2bb01e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2bb01e7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45192/0 (socket says 172.21.15.32:45192) 2024-03-06T07:36:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.372+0000 7f2bae800640 1 -- 172.21.15.32:0/625720668 learned_addr learned my addr 172.21.15.32:0/625720668 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bade00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2bb01e9c30 0x7f2bb01554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2baf200640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01ea0b0 0x7f2bb01559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bae800640 1 -- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2bb01e9c30 msgr2=0x7f2bb01554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bae800640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2bb01e9c30 0x7f2bb01554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bae800640 1 -- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01ea0b0 msgr2=0x7f2bb01559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bae800640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01ea0b0 0x7f2bb01559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bae800640 1 -- 172.21.15.32:0/625720668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb01e8760 con 0x7f2b940039b0 2024-03-06T07:36:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2baf200640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01ea0b0 0x7f2bb01559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:45.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.373+0000 7f2bae800640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2bb01e7f10 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0001e20 tx=0x7f2ba00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:45.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.374+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0003970 con 0x7f2b940039b0 2024-03-06T07:36:45.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.374+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba0006b50 con 0x7f2b940039b0 2024-03-06T07:36:45.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.374+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb01e8a40 con 0x7f2b940039b0 2024-03-06T07:36:45.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.374+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0007030 con 0x7f2b940039b0 2024-03-06T07:36:45.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.374+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb01e8450 con 0x7f2b940039b0 2024-03-06T07:36:45.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.375+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb01e8f60 con 0x7f2b940039b0 2024-03-06T07:36:45.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.376+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ba00071d0 con 0x7f2b940039b0 2024-03-06T07:36:45.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.377+0000 7f2ba6a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b8c06b500 0x7f2b8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.377+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2ba00abae0 con 0x7f2b940039b0 2024-03-06T07:36:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.377+0000 7f2bade00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b8c06b500 0x7f2b8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.378+0000 7f2bade00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b8c06b500 0x7f2b8c06d9c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00cafb0 tx=0x7f2b98008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:45.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.381+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bb01e8f60 con 0x7f2b940039b0 2024-03-06T07:36:45.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:45.663+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f2bb004f330 con 0x7f2b940039b0 2024-03-06T07:36:46.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.669+0000 7f2ba6a00640 1 -- 172.21.15.32:0/625720668 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v421) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2ba0033020 con 0x7f2b940039b0 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b8c06b500 msgr2=0x7f2b8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b8c06b500 0x7f2b8c06d9c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00cafb0 tx=0x7f2b98008040 comp rx=0 tx=0).stop 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 msgr2=0x7f2bb01e7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2bb01e7f10 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0001e20 tx=0x7f2ba00014a0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 shutdown_connections 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2b8c06b500 0x7f2b8c06d9c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb01ea0b0 0x7f2bb01559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2bb01e9c30 0x7f2bb01554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 --2- 172.21.15.32:0/625720668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b940039b0 0x7f2bb01e7f10 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 >> 172.21.15.32:0/625720668 conn(0x7f2bb00d1ae0 msgr2=0x7f2bb0157f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.673+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 shutdown_connections 2024-03-06T07:36:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.674+0000 7f2bb4a00640 1 -- 172.21.15.32:0/625720668 wait complete. 2024-03-06T07:36:46.696 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:36:46.697 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:36:46.697 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:36:46.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.914+0000 7fdb2b200640 1 Processor -- start 2024-03-06T07:36:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.914+0000 7fdb2b200640 1 -- start start 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.915+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 0x7fdb2c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.915+0000 7fdb2b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb2c0d04b0 0x7fdb2c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.915+0000 7fdb2b200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c0cd0c0 con 0x7fdb2c0700c0 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.915+0000 7fdb2b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2c0d29e0 con 0x7fdb2c0d04b0 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.915+0000 7fdb2b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0cd3a0 con 0x7fdb2c15f450 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.916+0000 7fdb29400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb2c0700c0 0x7fdb2c0cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45934/0 (socket says 172.21.15.32:45934) 2024-03-06T07:36:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.916+0000 7fdb29400640 1 -- 172.21.15.32:0/2193811989 learned_addr learned my addr 172.21.15.32:0/2193811989 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.916+0000 7fdb2a800640 1 --2- 172.21.15.32:0/2193811989 >> v2:172.21.15.39:3300/0 conn(0x7fdb2c0d04b0 0x7fdb2c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.916+0000 7fdb29e00640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 0x7fdb2c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.916+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897402334 0 0) 0x7fdb2c0cd0c0 con 0x7fdb2c0700c0 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.916+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb10003520 con 0x7fdb2c0700c0 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 500606726 0 0) 0x7fdb10003520 con 0x7fdb2c0700c0 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 msgr2=0x7fdb2c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 0x7fdb2c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 >> v2:172.21.15.39:3300/0 conn(0x7fdb2c0d04b0 msgr2=0x7fdb2c0cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 --2- 172.21.15.32:0/2193811989 >> v2:172.21.15.39:3300/0 conn(0x7fdb2c0d04b0 0x7fdb2c0cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c0cd680 con 0x7fdb2c0700c0 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fdb18001670 con 0x7fdb2c0700c0 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 >> v1:172.21.15.32:6789/0 conn(0x7fdb2c0700c0 legacy=0x7fdb2c0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb100039b0 0x7fdb10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb10004800 0x7fdb10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 0x7fdb100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0d29e0 con 0x7fdb10004800 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0cd3a0 con 0x7fdb100039b0 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1000b9c0 con 0x7fdb2c15f450 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.917+0000 7fdb29400640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb10004800 0x7fdb10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb2a800640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 0x7fdb100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb29400640 1 -- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 msgr2=0x7fdb100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb29e00640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb100039b0 0x7fdb10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb29400640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c15f450 0x7fdb100051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb29400640 1 -- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb100039b0 msgr2=0x7fdb10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:46.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb29400640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb100039b0 0x7fdb10003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.918+0000 7fdb29400640 1 -- 172.21.15.32:0/2193811989 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb10003520 con 0x7fdb10004800 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.919+0000 7fdb29400640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb10004800 0x7fdb10004c80 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800c210 tx=0x7fdb10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b87f1c506040fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.919+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18007d50 con 0x7fdb10004800 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.919+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb18030070 con 0x7fdb10004800 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.919+0000 7fdb28a00640 1 -- 172.21.15.32:0/2193811989 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1802cd90 con 0x7fdb10004800 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2b200640 1 -- 172.21.15.32:0/2193811989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb10004800 msgr2=0x7fdb10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2b200640 1 --2- 172.21.15.32:0/2193811989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb10004800 0x7fdb10004c80 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800c210 tx=0x7fdb10005830 comp rx=0 tx=0).stop 2024-03-06T07:36:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2a800640 1 -- 172.21.15.32:0/2193811989 reap_dead start 2024-03-06T07:36:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2b200640 1 -- 172.21.15.32:0/2193811989 shutdown_connections 2024-03-06T07:36:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2b200640 1 -- 172.21.15.32:0/2193811989 >> 172.21.15.32:0/2193811989 conn(0x7fdb2c0d1ae0 msgr2=0x7fdb2c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2b200640 1 -- 172.21.15.32:0/2193811989 shutdown_connections 2024-03-06T07:36:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.920+0000 7fdb2b200640 1 -- 172.21.15.32:0/2193811989 wait complete. 2024-03-06T07:36:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.922+0000 7fdb2b200640 1 Processor -- start 2024-03-06T07:36:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.922+0000 7fdb2b200640 1 -- start start 2024-03-06T07:36:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb10004800 0x7fdb2c1f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c0c8e20 0x7fdb2c153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 0x7fdb2c15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c15f830 con 0x7fdb2c155e00 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0ae8b0 con 0x7fdb10004800 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2b200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c15f9b0 con 0x7fdb2c0c8e20 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 0x7fdb2c15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 0x7fdb2c15d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45222/0 (socket says 172.21.15.32:45222) 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 -- 172.21.15.32:0/488806940 learned_addr learned my addr 172.21.15.32:0/488806940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb29400640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c0c8e20 0x7fdb2c153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 -- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c0c8e20 msgr2=0x7fdb2c153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c0c8e20 0x7fdb2c153920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb29e00640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb10004800 0x7fdb2c1f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 -- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb10004800 msgr2=0x7fdb2c1f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb10004800 0x7fdb2c1f7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.923+0000 7fdb2a800640 1 -- 172.21.15.32:0/488806940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c0c9880 con 0x7fdb2c155e00 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.924+0000 7fdb2a800640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 0x7fdb2c15d8e0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c071fe0 tx=0x7fdb2400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.924+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2401c070 con 0x7fdb2c155e00 2024-03-06T07:36:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.924+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb24017400 con 0x7fdb2c155e00 2024-03-06T07:36:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.924+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb240177f0 con 0x7fdb2c155e00 2024-03-06T07:36:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.924+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c0c9b60 con 0x7fdb2c155e00 2024-03-06T07:36:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.924+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c0ca120 con 0x7fdb2c155e00 2024-03-06T07:36:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.925+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c06be90 con 0x7fdb2c155e00 2024-03-06T07:36:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.926+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb2401f070 con 0x7fdb2c155e00 2024-03-06T07:36:46.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.927+0000 7fdb22000640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.927+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdb24089ff0 con 0x7fdb2c155e00 2024-03-06T07:36:46.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.928+0000 7fdb29e00640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.928+0000 7fdb29e00640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009830 tx=0x7fdb1c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:46.932+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb2c06be90 con 0x7fdb2c155e00 2024-03-06T07:36:47.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.136+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb2c0d1470 con 0x7fdb2c155e00 2024-03-06T07:36:47.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.138+0000 7fdb22000640 1 -- 172.21.15.32:0/488806940 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+13048 (secure 0 0 0) 0x7fdb2c0d1470 con 0x7fdb2c155e00 2024-03-06T07:36:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:47.141 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":119,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:36:44.652796+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6836/3465731454":"2024-03-07T07:36:41.633466+0000","172.21.15.32:6837/3465731454":"2024-03-07T07:36:41.633466+0000","172.21.15.39:6835/14889266":"2024-03-07T07:35:30.884623+0000","172.21.15.39:6833/14889266":"2024-03-07T07:35:30.884623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdb0c06b4d0 msgr2=0x7fdb0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009830 tx=0x7fdb1c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 msgr2=0x7fdb2c15d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 0x7fdb2c15d8e0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c071fe0 tx=0x7fdb2400f4a0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 shutdown_connections 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2c155e00 0x7fdb2c15d8e0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdb2c0c8e20 0x7fdb2c153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 --2- 172.21.15.32:0/488806940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb10004800 0x7fdb2c1f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.141+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 >> 172.21.15.32:0/488806940 conn(0x7fdb2c0d1ae0 msgr2=0x7fdb2c0b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.142+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 shutdown_connections 2024-03-06T07:36:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.142+0000 7fdb2b200640 1 -- 172.21.15.32:0/488806940 wait complete. 2024-03-06T07:36:47.162 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:36:47.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:36:47.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.371+0000 7f38d6400640 1 Processor -- start 2024-03-06T07:36:47.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.371+0000 7f38d6400640 1 -- start start 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.372+0000 7f38d6400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f38d0145430 0x7f38d015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.372+0000 7f38d6400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d015cc90 0x7f38d00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.372+0000 7f38d6400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38d00b2080 0x7f38d006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.372+0000 7f38d6400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d015fe40 con 0x7f38d00b2080 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.372+0000 7f38d6400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d0160120 con 0x7f38d0145430 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.372+0000 7f38d6400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0160400 con 0x7f38d015cc90 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38d00b2080 0x7f38d006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38d00b2080 0x7f38d006c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45234/0 (socket says 172.21.15.32:45234) 2024-03-06T07:36:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 -- 172.21.15.32:0/677796852 learned_addr learned my addr 172.21.15.32:0/677796852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5000640 1 --2- 172.21.15.32:0/677796852 >> v2:172.21.15.39:3300/0 conn(0x7f38d0145430 0x7f38d015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 -- 172.21.15.32:0/677796852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d015cc90 msgr2=0x7f38d00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38cfe00640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d015cc90 0x7f38d00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d015cc90 0x7f38d00b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 -- 172.21.15.32:0/677796852 >> v2:172.21.15.39:3300/0 conn(0x7f38d0145430 msgr2=0x7f38d015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 --2- 172.21.15.32:0/677796852 >> v2:172.21.15.39:3300/0 conn(0x7f38d0145430 0x7f38d015f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5a00640 1 -- 172.21.15.32:0/677796852 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d01606e0 con 0x7f38d00b2080 2024-03-06T07:36:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38d5000640 1 --2- 172.21.15.32:0/677796852 >> v2:172.21.15.39:3300/0 conn(0x7f38d0145430 0x7f38d015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.373+0000 7f38cfe00640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d015cc90 0x7f38d00b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.374+0000 7f38d5a00640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d00b2080 0x7f38d006c800 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f38d00b2670 tx=0x7f38c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7256005b13a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.375+0000 7f38cf400640 1 -- 172.21.15.32:0/677796852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c802ca70 con 0x7f38d00b2080 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.375+0000 7f38cf400640 1 -- 172.21.15.32:0/677796852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c802cc10 con 0x7f38d00b2080 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.375+0000 7f38cf400640 1 -- 172.21.15.32:0/677796852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c80343f0 con 0x7f38d00b2080 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.375+0000 7f38d6400640 1 -- 172.21.15.32:0/677796852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d00b2080 msgr2=0x7f38d006c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.375+0000 7f38d6400640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d00b2080 0x7f38d006c800 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f38d00b2670 tx=0x7f38c802bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 -- 172.21.15.32:0/677796852 shutdown_connections 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d00b2080 0x7f38d006c800 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 --2- 172.21.15.32:0/677796852 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d015cc90 0x7f38d00b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 --2- 172.21.15.32:0/677796852 >> v2:172.21.15.39:3300/0 conn(0x7f38d0145430 0x7f38d015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 -- 172.21.15.32:0/677796852 >> 172.21.15.32:0/677796852 conn(0x7f38d006b3a0 msgr2=0x7f38d006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 -- 172.21.15.32:0/677796852 shutdown_connections 2024-03-06T07:36:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.376+0000 7f38d6400640 1 -- 172.21.15.32:0/677796852 wait complete. 2024-03-06T07:36:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.377+0000 7f38d6400640 1 Processor -- start 2024-03-06T07:36:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.378+0000 7f38d6400640 1 -- start start 2024-03-06T07:36:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.378+0000 7f38d6400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d00b2080 0x7f38d01f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d6400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 0x7f38d00c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d015cc90 0x7f38d00cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d6400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0145990 con 0x7f38d0145430 2024-03-06T07:36:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0145810 con 0x7f38d015cc90 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d6400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00cde00 con 0x7f38d00b2080 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 0x7f38d00c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 0x7f38d00c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45242/0 (socket says 172.21.15.32:45242) 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 -- 172.21.15.32:0/1949836991 learned_addr learned my addr 172.21.15.32:0/1949836991 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d5000640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d00b2080 0x7f38d01f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d5a00640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d015cc90 0x7f38d00cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 -- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d00b2080 msgr2=0x7f38d01f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d00b2080 0x7f38d01f7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 -- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d015cc90 msgr2=0x7f38d00cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d015cc90 0x7f38d00cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38cfe00640 1 -- 172.21.15.32:0/1949836991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d01f3d20 con 0x7f38d0145430 2024-03-06T07:36:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.379+0000 7f38d5000640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d00b2080 0x7f38d01f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.380+0000 7f38cfe00640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 0x7f38d00c52b0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00c7f0 tx=0x7f38bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.380+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc015070 con 0x7f38d0145430 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.380+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc007a30 con 0x7f38d0145430 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.380+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc007e00 con 0x7f38d0145430 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.380+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d01f4060 con 0x7f38d0145430 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.380+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d00c57f0 con 0x7f38d0145430 2024-03-06T07:36:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.381+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d004f330 con 0x7f38d0145430 2024-03-06T07:36:47.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.382+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38bc018070 con 0x7f38d0145430 2024-03-06T07:36:47.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.383+0000 7f38ccc00640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f38b006b4d0 0x7f38b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.383+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f38bc089f90 con 0x7f38d0145430 2024-03-06T07:36:47.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.383+0000 7f38d5000640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f38b006b4d0 0x7f38b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.384+0000 7f38d5000640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f38b006b4d0 0x7f38b006d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f38c4006fd0 tx=0x7f38c4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.388+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38d01f4060 con 0x7f38d0145430 2024-03-06T07:36:47.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.587+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38d01f4240 con 0x7f38d0145430 2024-03-06T07:36:47.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.588+0000 7f38ccc00640 1 -- 172.21.15.32:0/1949836991 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38bc059010 con 0x7f38d0145430 2024-03-06T07:36:47.591 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:36:47.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.591+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f38b006b4d0 msgr2=0x7f38b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.591+0000 7f38d6400640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f38b006b4d0 0x7f38b006d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f38c4006fd0 tx=0x7f38c4008040 comp rx=0 tx=0).stop 2024-03-06T07:36:47.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.591+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 msgr2=0x7f38d00c52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.591+0000 7f38d6400640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 0x7f38d00c52b0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00c7f0 tx=0x7f38bc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 shutdown_connections 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f38b006b4d0 0x7f38b006d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d015cc90 0x7f38d00cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d0145430 0x7f38d00c52b0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 --2- 172.21.15.32:0/1949836991 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f38d00b2080 0x7f38d01f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 >> 172.21.15.32:0/1949836991 conn(0x7f38d006b3a0 msgr2=0x7f38d015bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 shutdown_connections 2024-03-06T07:36:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.592+0000 7f38d6400640 1 -- 172.21.15.32:0/1949836991 wait complete. 2024-03-06T07:36:47.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:36:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.830+0000 7f9496000640 1 Processor -- start 2024-03-06T07:36:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.830+0000 7f9496000640 1 -- start start 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f9496000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9490070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f9496000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f94900ae0a0 0x7f94900711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f9496000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900ce9a0 con 0x7f949006e240 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f9496000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94900730d0 con 0x7f94900ae0a0 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f9496000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94900cec80 con 0x7f9490161670 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f948ea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f949006e240 0x7f949006cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45950/0 (socket says 172.21.15.32:45950) 2024-03-06T07:36:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.832+0000 7f948ea00640 1 -- 172.21.15.32:0/3000067147 learned_addr learned my addr 172.21.15.32:0/3000067147 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948f400640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9490070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948fe00640 1 --2- 172.21.15.32:0/3000067147 >> v2:172.21.15.39:3300/0 conn(0x7f94900ae0a0 0x7f94900711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662896724 0 0) 0x7f94900ce9a0 con 0x7f949006e240 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9474003520 con 0x7f949006e240 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67157112 0 0) 0x7f9474003520 con 0x7f949006e240 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 msgr2=0x7f9490070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9490070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 >> v2:172.21.15.39:3300/0 conn(0x7f94900ae0a0 msgr2=0x7f94900711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 --2- 172.21.15.32:0/3000067147 >> v2:172.21.15.39:3300/0 conn(0x7f94900ae0a0 0x7f94900711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900cef60 con 0x7f949006e240 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f947c001670 con 0x7f949006e240 2024-03-06T07:36:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 >> v1:172.21.15.32:6789/0 conn(0x7f949006e240 legacy=0x7f949006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9474003a20 0x7f9474003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9474004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f94740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900730d0 con 0x7f9474004d40 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900cec80 con 0x7f9474003a20 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.833+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947400b9c0 con 0x7f9490161670 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948fe00640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f94740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948f400640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9474003a20 0x7f9474003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948ea00640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9474004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948fe00640 1 -- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 msgr2=0x7f9474004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948fe00640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9474004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948fe00640 1 -- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9474003a20 msgr2=0x7f9474003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948fe00640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9474003a20 0x7f9474003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948fe00640 1 -- 172.21.15.32:0/3000067147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9474003520 con 0x7f9474004d40 2024-03-06T07:36:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.834+0000 7f948ea00640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490161670 0x7f9474004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.835+0000 7f948fe00640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f94740051c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f9488009fc0 tx=0x7f9474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6d48dab7e294f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488039070 con 0x7f9474004d40 2024-03-06T07:36:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948802cc30 con 0x7f9474004d40 2024-03-06T07:36:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f948e000640 1 -- 172.21.15.32:0/3000067147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488009c10 con 0x7f9474004d40 2024-03-06T07:36:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f9496000640 1 -- 172.21.15.32:0/3000067147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 msgr2=0x7f94740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f9496000640 1 --2- 172.21.15.32:0/3000067147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f94740051c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f9488009fc0 tx=0x7f9474005830 comp rx=0 tx=0).stop 2024-03-06T07:36:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f948fe00640 1 -- 172.21.15.32:0/3000067147 reap_dead start 2024-03-06T07:36:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f9496000640 1 -- 172.21.15.32:0/3000067147 shutdown_connections 2024-03-06T07:36:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.836+0000 7f9496000640 1 -- 172.21.15.32:0/3000067147 >> 172.21.15.32:0/3000067147 conn(0x7f949006afb0 msgr2=0x7f949006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:47.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.837+0000 7f9496000640 1 -- 172.21.15.32:0/3000067147 shutdown_connections 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.837+0000 7f9496000640 1 -- 172.21.15.32:0/3000067147 wait complete. 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.838+0000 7f9496000640 1 Processor -- start 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.838+0000 7f9496000640 1 -- start start 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.839+0000 7f9496000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f9490157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.839+0000 7f9496000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490149720 0x7f94900b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.839+0000 7f9496000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900b3bd0 0x7f94900cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.839+0000 7f9496000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900ce600 con 0x7f9474004d40 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.839+0000 7f9496000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900cf520 con 0x7f94900b3bd0 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.839+0000 7f9496000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94900ce780 con 0x7f9490149720 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f9490157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f9490157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45256/0 (socket says 172.21.15.32:45256) 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 -- 172.21.15.32:0/4088831573 learned_addr learned my addr 172.21.15.32:0/4088831573 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 -- 172.21.15.32:0/4088831573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490149720 msgr2=0x7f94900b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490149720 0x7f94900b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 -- 172.21.15.32:0/4088831573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900b3bd0 msgr2=0x7f94900cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900b3bd0 0x7f94900cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 -- 172.21.15.32:0/4088831573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900cba10 con 0x7f9474004d40 2024-03-06T07:36:47.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.840+0000 7f948f400640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f9490157290 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9480011570 tx=0x7f9480011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.841+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480015930 con 0x7f9474004d40 2024-03-06T07:36:47.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.841+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9480015ad0 con 0x7f9474004d40 2024-03-06T07:36:47.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.841+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9490149b80 con 0x7f9474004d40 2024-03-06T07:36:47.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.841+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948001d400 con 0x7f9474004d40 2024-03-06T07:36:47.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.841+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94901567b0 con 0x7f9474004d40 2024-03-06T07:36:47.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.842+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949004f3a0 con 0x7f9474004d40 2024-03-06T07:36:47.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.843+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f948001d5a0 con 0x7f9474004d40 2024-03-06T07:36:47.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.844+0000 7f9487400640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f947006b4d0 0x7f947006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.844+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f948008a7d0 con 0x7f9474004d40 2024-03-06T07:36:47.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.844+0000 7f948ea00640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f947006b4d0 0x7f947006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.845+0000 7f948ea00640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f947006b4d0 0x7f947006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f947c0017e0 tx=0x7f947c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:47.848+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9480019030 con 0x7f9474004d40 2024-03-06T07:36:48.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.134+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f94900b3ff0 con 0x7f9474004d40 2024-03-06T07:36:48.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.681+0000 7f9487400640 1 -- 172.21.15.32:0/4088831573 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v423) v1 ==== 143+0+0 (secure 0 0 0) 0x7f9480059850 con 0x7f9474004d40 2024-03-06T07:36:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f947006b4d0 msgr2=0x7f947006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f947006b4d0 0x7f947006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f947c0017e0 tx=0x7f947c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 msgr2=0x7f9490157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f9490157290 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9480011570 tx=0x7f9480011a30 comp rx=0 tx=0).stop 2024-03-06T07:36:48.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 shutdown_connections 2024-03-06T07:36:48.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f947006b4d0 0x7f947006d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900b3bd0 0x7f94900cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9490149720 0x7f94900b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 --2- 172.21.15.32:0/4088831573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9474004d40 0x7f9490157290 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.685+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 >> 172.21.15.32:0/4088831573 conn(0x7f949006afb0 msgr2=0x7f94900cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.686+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 shutdown_connections 2024-03-06T07:36:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.686+0000 7f9496000640 1 -- 172.21.15.32:0/4088831573 wait complete. 2024-03-06T07:36:48.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:36:48.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.928+0000 7f9ceaa00640 1 Processor -- start 2024-03-06T07:36:48.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.929+0000 7f9ceaa00640 1 -- start start 2024-03-06T07:36:48.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9ceaa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ce415df70 0x7f9ce40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9ceaa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce40b6be0 con 0x7f9ce40d29e0 2024-03-06T07:36:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9ceaa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce40b6ec0 con 0x7f9ce415e350 2024-03-06T07:36:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9ceaa00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40b71a0 con 0x7f9ce415df70 2024-03-06T07:36:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9ce9600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ce415df70 0x7f9ce40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9cea000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9ce40d29e0 0x7f9ce40b4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39814/0 (socket says 172.21.15.32:39814) 2024-03-06T07:36:48.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.930+0000 7f9cea000640 1 -- 172.21.15.32:0/3780493051 learned_addr learned my addr 172.21.15.32:0/3780493051 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.931+0000 7f9ce9600640 1 -- 172.21.15.32:0/3780493051 >> v1:172.21.15.39:6789/0 conn(0x7f9ce415e350 legacy=0x7f9ce40d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.931+0000 7f9ce9600640 1 -- 172.21.15.32:0/3780493051 >> v1:172.21.15.32:6789/0 conn(0x7f9ce40d29e0 legacy=0x7f9ce40b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.931+0000 7f9ce9600640 1 -- 172.21.15.32:0/3780493051 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40b7480 con 0x7f9ce415df70 2024-03-06T07:36:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.931+0000 7f9ce3e00640 1 -- 172.21.15.32:0/3780493051 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943171509 0 0) 0x7f9ce40b6be0 con 0x7f9ce40d29e0 2024-03-06T07:36:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.932+0000 7f9ce9600640 1 --2- 172.21.15.32:0/3780493051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce415df70 0x7f9ce40cf3a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80097a0 tx=0x7f9cd802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484ac41a27f74e44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.932+0000 7f9ce3e00640 1 -- 172.21.15.32:0/3780493051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8034070 con 0x7f9ce415df70 2024-03-06T07:36:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.932+0000 7f9ce3e00640 1 -- 172.21.15.32:0/3780493051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd8030610 con 0x7f9ce415df70 2024-03-06T07:36:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.932+0000 7f9ce3e00640 1 -- 172.21.15.32:0/3780493051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80309e0 con 0x7f9ce415df70 2024-03-06T07:36:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 -- 172.21.15.32:0/3780493051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce415df70 msgr2=0x7f9ce40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/3780493051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce415df70 0x7f9ce40cf3a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80097a0 tx=0x7f9cd802b960 comp rx=0 tx=0).stop 2024-03-06T07:36:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 -- 172.21.15.32:0/3780493051 shutdown_connections 2024-03-06T07:36:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/3780493051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce415df70 0x7f9ce40cf3a0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 -- 172.21.15.32:0/3780493051 >> 172.21.15.32:0/3780493051 conn(0x7f9ce408d2a0 msgr2=0x7f9ce40d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 -- 172.21.15.32:0/3780493051 shutdown_connections 2024-03-06T07:36:48.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.933+0000 7f9ceaa00640 1 -- 172.21.15.32:0/3780493051 wait complete. 2024-03-06T07:36:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.935+0000 7f9ceaa00640 1 Processor -- start 2024-03-06T07:36:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.935+0000 7f9ceaa00640 1 -- start start 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ce40d29e0 0x7f9ce41565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ce415df70 0x7f9ce41f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 0x7f9ce40cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ceaa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40b8c90 con 0x7f9ce40ccf90 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ceaa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40d1d60 con 0x7f9ce415df70 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ceaa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40b8e10 con 0x7f9ce40d29e0 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9cea000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 0x7f9ce40cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9cea000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 0x7f9ce40cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34372/0 (socket says 172.21.15.32:34372) 2024-03-06T07:36:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9cea000640 1 -- 172.21.15.32:0/1743964187 learned_addr learned my addr 172.21.15.32:0/1743964187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ce8c00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ce415df70 0x7f9ce41f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.936+0000 7f9ce9600640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ce40d29e0 0x7f9ce41565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9cea000640 1 -- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ce40d29e0 msgr2=0x7f9ce41565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9cea000640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ce40d29e0 0x7f9ce41565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9cea000640 1 -- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ce415df70 msgr2=0x7f9ce41f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9cea000640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ce415df70 0x7f9ce41f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9cea000640 1 -- 172.21.15.32:0/1743964187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4155520 con 0x7f9ce40ccf90 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9ce9600640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ce40d29e0 0x7f9ce41565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9cea000640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 0x7f9ce40cd410 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c780 tx=0x7f9cdc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc018070 con 0x7f9ce40ccf90 2024-03-06T07:36:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc00a640 con 0x7f9ce40ccf90 2024-03-06T07:36:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc00aa70 con 0x7f9ce40ccf90 2024-03-06T07:36:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4155800 con 0x7f9ce40ccf90 2024-03-06T07:36:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.937+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4155dc0 con 0x7f9ce40ccf90 2024-03-06T07:36:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.939+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce404f330 con 0x7f9ce40ccf90 2024-03-06T07:36:48.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.940+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cdc00ac10 con 0x7f9ce40ccf90 2024-03-06T07:36:48.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.941+0000 7f9ce1600640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9cc406b4d0 0x7f9cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.941+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9cdc089610 con 0x7f9ce40ccf90 2024-03-06T07:36:48.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.941+0000 7f9ce9600640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9cc406b4d0 0x7f9cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.942+0000 7f9ce9600640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9cc406b4d0 0x7f9cc406d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8036000 tx=0x7f9cd803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:48.946+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cdc058690 con 0x7f9ce40ccf90 2024-03-06T07:36:49.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.148+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9ce40d20b0 con 0x7f9ce40ccf90 2024-03-06T07:36:49.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.155+0000 7f9ce1600640 1 -- 172.21.15.32:0/1743964187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9cdc057f90 con 0x7f9ce40ccf90 2024-03-06T07:36:49.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9cc406b4d0 msgr2=0x7f9cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9cc406b4d0 0x7f9cc406d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8036000 tx=0x7f9cd803b040 comp rx=0 tx=0).stop 2024-03-06T07:36:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 msgr2=0x7f9ce40cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 0x7f9ce40cd410 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c780 tx=0x7f9cdc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:36:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 shutdown_connections 2024-03-06T07:36:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9cc406b4d0 0x7f9cc406d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce40ccf90 0x7f9ce40cd410 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ce415df70 0x7f9ce41f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 --2- 172.21.15.32:0/1743964187 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ce40d29e0 0x7f9ce41565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.158+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 >> 172.21.15.32:0/1743964187 conn(0x7f9ce408d2a0 msgr2=0x7f9ce40cf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:49.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.159+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 shutdown_connections 2024-03-06T07:36:49.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.159+0000 7f9ceaa00640 1 -- 172.21.15.32:0/1743964187 wait complete. 2024-03-06T07:36:49.181 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:36:49.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:36:49.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:36:49.412 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:36:49.412 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:36:49.412 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:49.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:36:49.471 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:36:49.471 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:49.530 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:36:49.530 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:36:49.530 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:49.586 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:36:49.586 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-06T07:36:49.586 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:49.646 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:36:49.646 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:36:49.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.868+0000 7f9d9a400640 1 Processor -- start 2024-03-06T07:36:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.868+0000 7f9d9a400640 1 -- start start 2024-03-06T07:36:49.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.869+0000 7f9d9a400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9d9415df00 0x7f9d9406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d9a400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9d940b1280 0x7f9d9406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d9a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d94161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d9a400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940aeb00 con 0x7f9d940b1280 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d9a400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940b6270 con 0x7f9d9415df00 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d9a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940b6510 con 0x7f9d940b0d30 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9d940b1280 0x7f9d9406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9d940b1280 0x7f9d9406cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34376/0 (socket says 172.21.15.32:34376) 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 -- 172.21.15.32:0/469381930 learned_addr learned my addr 172.21.15.32:0/469381930 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d99a00640 1 --2- 172.21.15.32:0/469381930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d94161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 -- 172.21.15.32:0/469381930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 msgr2=0x7f9d94161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 --2- 172.21.15.32:0/469381930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d94161bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 -- 172.21.15.32:0/469381930 >> v2:172.21.15.39:3300/0 conn(0x7f9d9415df00 msgr2=0x7f9d9406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 --2- 172.21.15.32:0/469381930 >> v2:172.21.15.39:3300/0 conn(0x7f9d9415df00 0x7f9d9406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.870+0000 7f9d93e00640 1 -- 172.21.15.32:0/469381930 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940b67f0 con 0x7f9d940b1280 2024-03-06T07:36:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.872+0000 7f9d93e00640 1 --2- 172.21.15.32:0/469381930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d940b1280 0x7f9d9406cdc0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f9d94074d80 tx=0x7f9d8002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f1ed8e22810f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.872+0000 7f9d93400640 1 -- 172.21.15.32:0/469381930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d800317d0 con 0x7f9d940b1280 2024-03-06T07:36:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.872+0000 7f9d93400640 1 -- 172.21.15.32:0/469381930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d80037070 con 0x7f9d940b1280 2024-03-06T07:36:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.872+0000 7f9d93400640 1 -- 172.21.15.32:0/469381930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80031b90 con 0x7f9d940b1280 2024-03-06T07:36:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 -- 172.21.15.32:0/469381930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d940b1280 msgr2=0x7f9d9406cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 --2- 172.21.15.32:0/469381930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d940b1280 0x7f9d9406cdc0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f9d94074d80 tx=0x7f9d8002b870 comp rx=0 tx=0).stop 2024-03-06T07:36:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 -- 172.21.15.32:0/469381930 shutdown_connections 2024-03-06T07:36:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 --2- 172.21.15.32:0/469381930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d94161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 --2- 172.21.15.32:0/469381930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d940b1280 0x7f9d9406cdc0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 --2- 172.21.15.32:0/469381930 >> v2:172.21.15.39:3300/0 conn(0x7f9d9415df00 0x7f9d9406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 -- 172.21.15.32:0/469381930 >> 172.21.15.32:0/469381930 conn(0x7f9d9409aa10 msgr2=0x7f9d9409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:49.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 -- 172.21.15.32:0/469381930 shutdown_connections 2024-03-06T07:36:49.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.873+0000 7f9d9a400640 1 -- 172.21.15.32:0/469381930 wait complete. 2024-03-06T07:36:49.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.875+0000 7f9d9a400640 1 Processor -- start 2024-03-06T07:36:49.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.875+0000 7f9d9a400640 1 -- start start 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d9a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d941405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d9a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d940b1280 0x7f9d94155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d9a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 0x7f9d94156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d9a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9406d480 con 0x7f9d9415df00 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d9a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9406d300 con 0x7f9d940b1280 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d9a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9415e2e0 con 0x7f9d940b0d30 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d99a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 0x7f9d94156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d99a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 0x7f9d94156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34386/0 (socket says 172.21.15.32:34386) 2024-03-06T07:36:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d99a00640 1 -- 172.21.15.32:0/1347132272 learned_addr learned my addr 172.21.15.32:0/1347132272 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d93e00640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d940b1280 0x7f9d94155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.876+0000 7f9d99000640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d941405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d99a00640 1 -- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 msgr2=0x7f9d941405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d99a00640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d941405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d99a00640 1 -- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d940b1280 msgr2=0x7f9d94155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d99a00640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d940b1280 0x7f9d94155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d99a00640 1 -- 172.21.15.32:0/1347132272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d941525b0 con 0x7f9d9415df00 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d93e00640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d940b1280 0x7f9d94155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:49.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d99a00640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 0x7f9d94156290 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007e60 tx=0x7f9d8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c014420 con 0x7f9d9415df00 2024-03-06T07:36:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c019070 con 0x7f9d9415df00 2024-03-06T07:36:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c014820 con 0x7f9d9415df00 2024-03-06T07:36:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d94152890 con 0x7f9d9415df00 2024-03-06T07:36:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.877+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d94152e50 con 0x7f9d9415df00 2024-03-06T07:36:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.879+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9404f330 con 0x7f9d9415df00 2024-03-06T07:36:49.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.879+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d8c0149c0 con 0x7f9d9415df00 2024-03-06T07:36:49.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.880+0000 7f9d90c00640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d7406b4d0 0x7f9d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.881+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9d8c08a8f0 con 0x7f9d9415df00 2024-03-06T07:36:49.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.881+0000 7f9d99000640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d7406b4d0 0x7f9d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.881+0000 7f9d99000640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d7406b4d0 0x7f9d7406d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009890 tx=0x7f9d88008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:49.885+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d8c0598e0 con 0x7f9d9415df00 2024-03-06T07:36:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.124+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d9409b6e0 con 0x7f9d9415df00 2024-03-06T07:36:50.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.125+0000 7f9d90c00640 1 -- 172.21.15.32:0/1347132272 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f9d8c0591e0 con 0x7f9d9415df00 2024-03-06T07:36:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:50.128 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5708,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3359403288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3359403288},{"type":"v1","addr":"172.21.15.39:6835","nonce":3359403288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6328,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1979159785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1979159785},{"type":"v1","addr":"172.21.15.39:6833","nonce":1979159785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/848247975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":848247975},{"type":"v1","addr":"172.21.15.32:6835","nonce":848247975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":8241,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2735364994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2735364994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2735364994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[]} 2024-03-06T07:36:50.128 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 84 2024-03-06T07:36:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d7406b4d0 msgr2=0x7f9d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d7406b4d0 0x7f9d7406d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009890 tx=0x7f9d88008040 comp rx=0 tx=0).stop 2024-03-06T07:36:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 msgr2=0x7f9d94156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 0x7f9d94156290 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007e60 tx=0x7f9d8c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 shutdown_connections 2024-03-06T07:36:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9d7406b4d0 0x7f9d7406d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9415df00 0x7f9d94156290 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d940b1280 0x7f9d94155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 --2- 172.21.15.32:0/1347132272 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9d940b0d30 0x7f9d941405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.128+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 >> 172.21.15.32:0/1347132272 conn(0x7f9d9409aa10 msgr2=0x7f9d9409adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.129+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 shutdown_connections 2024-03-06T07:36:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.129+0000 7f9d9a400640 1 -- 172.21.15.32:0/1347132272 wait complete. 2024-03-06T07:36:50.151 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:36:50.151 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:36:50.151 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:36:50.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:50.152 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:36:50.152 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:36:50.152 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:36:50.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:50.152 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:36:50.152 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:36:50.153 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:36:50.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:50.153 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:36:50.153 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:36:50.153 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:36:50.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:50.176 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:36:50.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:36:50.177 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:36:50.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:36:50.178 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:36:50.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:36:50.179 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:36:50.180 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:36:50.223 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:36:50.224 DEBUG:teuthology.parallel:result is None 2024-03-06T07:36:50.226 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:36:50.227 DEBUG:teuthology.parallel:result is None 2024-03-06T07:36:50.229 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:36:50.229 DEBUG:teuthology.parallel:result is None 2024-03-06T07:36:50.233 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:36:50.233 DEBUG:teuthology.parallel:result is None 2024-03-06T07:36:50.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:36:50.386 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-03-06T07:36:50.388 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-03-06T07:36:50.390 INFO:tasks.ceph.mds.b.smithi039.stdout:starting mds.b at 2024-03-06T07:36:50.391 INFO:tasks.ceph.mds.d.smithi039.stdout:starting mds.d at 2024-03-06T07:36:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.424+0000 7fa316200640 1 Processor -- start 2024-03-06T07:36:50.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.424+0000 7fa316200640 1 -- start start 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.425+0000 7fa316200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3100d4a10 0x7fa3100d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.425+0000 7fa316200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100b3710 con 0x7fa31015e0b0 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.425+0000 7fa316200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100b39f0 con 0x7fa3100d0c80 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.425+0000 7fa316200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100b09e0 con 0x7fa3100d4a10 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa314e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa31015e0b0 0x7fa3100cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39826/0 (socket says 172.21.15.32:39826) 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa314e00640 1 -- 172.21.15.32:0/4218657320 learned_addr learned my addr 172.21.15.32:0/4218657320 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30fe00640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3100d4a10 0x7fa3100d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173397447 0 0) 0x7fa3100b3710 con 0x7fa31015e0b0 2024-03-06T07:36:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f8003520 con 0x7fa31015e0b0 2024-03-06T07:36:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297145220 0 0) 0x7fa3100b39f0 con 0x7fa3100d0c80 2024-03-06T07:36:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3100b3710 con 0x7fa3100d0c80 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558657254 0 0) 0x7fa2f8003520 con 0x7fa31015e0b0 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3100d4a10 msgr2=0x7fa3100d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3100d4a10 0x7fa3100d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.426+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 >> v1:172.21.15.39:6789/0 conn(0x7fa3100d0c80 legacy=0x7fa3100b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100b3cd0 con 0x7fa31015e0b0 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa304001670 con 0x7fa31015e0b0 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 >> v1:172.21.15.32:6789/0 conn(0x7fa31015e0b0 legacy=0x7fa3100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f800bfb0 con 0x7fa2f80051c0 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f800c130 con 0x7fa2f8003b70 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f800c330 con 0x7fa2f8004800 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30fe00640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3100d4a10 0x7fa3100d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa315800640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa314e00640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa30fe00640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa315800640 1 -- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2f8004800 msgr2=0x7fa2f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa315800640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa315800640 1 -- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2f8003b70 msgr2=0x7fa2f8003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa315800640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.427+0000 7fa315800640 1 -- 172.21.15.32:0/4218657320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8003520 con 0x7fa2f80051c0 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa315800640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fa308001fd0 tx=0x7fa2f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214447b10e6cf196 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308001bf0 con 0x7fa2f80051c0 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308003b70 con 0x7fa2f80051c0 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa30f400640 1 -- 172.21.15.32:0/4218657320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3080050d0 con 0x7fa2f80051c0 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa316200640 1 -- 172.21.15.32:0/4218657320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 msgr2=0x7fa2f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa316200640 1 --2- 172.21.15.32:0/4218657320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fa308001fd0 tx=0x7fa2f8005830 comp rx=0 tx=0).stop 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa315800640 1 -- 172.21.15.32:0/4218657320 reap_dead start 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa316200640 1 -- 172.21.15.32:0/4218657320 shutdown_connections 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa316200640 1 -- 172.21.15.32:0/4218657320 >> 172.21.15.32:0/4218657320 conn(0x7fa3100d3d30 msgr2=0x7fa3100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.429+0000 7fa316200640 1 -- 172.21.15.32:0/4218657320 shutdown_connections 2024-03-06T07:36:50.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.430+0000 7fa316200640 1 -- 172.21.15.32:0/4218657320 wait complete. 2024-03-06T07:36:50.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.431+0000 7fa316200640 1 Processor -- start 2024-03-06T07:36:50.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.431+0000 7fa316200640 1 -- start start 2024-03-06T07:36:50.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa316200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa31015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa316200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3101f39a0 0x7fa3101f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa316200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa31015ef00 0x7fa31015f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa316200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31015e490 con 0x7fa2f80051c0 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa316200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100cfa20 con 0x7fa31015ef00 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa316200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100cfc60 con 0x7fa3101f39a0 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.432+0000 7fa314e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa31015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa31015b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34420/0 (socket says 172.21.15.32:34420) 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 -- 172.21.15.32:0/3048031426 learned_addr learned my addr 172.21.15.32:0/3048031426 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 -- 172.21.15.32:0/3048031426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3101f39a0 msgr2=0x7fa3101f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3101f39a0 0x7fa3101f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 -- 172.21.15.32:0/3048031426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa31015ef00 msgr2=0x7fa31015f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa31015ef00 0x7fa31015f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 -- 172.21.15.32:0/3048031426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3101f4a20 con 0x7fa2f80051c0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa314e00640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa31015b5d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fa304007290 tx=0x7fa304007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30400f940 con 0x7fa2f80051c0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa304018070 con 0x7fa2f80051c0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.433+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30400fd40 con 0x7fa2f80051c0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.434+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3101f4ce0 con 0x7fa2f80051c0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.434+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3101f4740 con 0x7fa2f80051c0 2024-03-06T07:36:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.435+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31004f330 con 0x7fa2f80051c0 2024-03-06T07:36:50.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.436+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa30401d460 con 0x7fa2f80051c0 2024-03-06T07:36:50.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.437+0000 7fa30cc00640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.437+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa304089f60 con 0x7fa2f80051c0 2024-03-06T07:36:50.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.437+0000 7fa30fe00640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.438+0000 7fa30fe00640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2f006b4d0 0x7fa2f006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa3101f1590 tx=0x7fa2fc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.442+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa304059080 con 0x7fa2f80051c0 2024-03-06T07:36:50.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:50.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:50.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.656+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa31015f8c0 con 0x7fa2f80051c0 2024-03-06T07:36:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.720+0000 7fa30cc00640 1 -- 172.21.15.32:0/3048031426 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa304058980 con 0x7fa2f80051c0 2024-03-06T07:36:50.723 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-03-06T07:36:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2f006b4d0 msgr2=0x7fa2f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2f006b4d0 0x7fa2f006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa3101f1590 tx=0x7fa2fc008040 comp rx=0 tx=0).stop 2024-03-06T07:36:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 msgr2=0x7fa31015b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa31015b5d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fa304007290 tx=0x7fa304007750 comp rx=0 tx=0).stop 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 shutdown_connections 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa31015ef00 0x7fa31015f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa3101f39a0 0x7fa3101f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 --2- 172.21.15.32:0/3048031426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2f80051c0 0x7fa31015b5d0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.724+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 >> 172.21.15.32:0/3048031426 conn(0x7fa3100d3d30 msgr2=0x7fa3100b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.725+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 shutdown_connections 2024-03-06T07:36:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:50.725+0000 7fa316200640 1 -- 172.21.15.32:0/3048031426 wait complete. 2024-03-06T07:36:50.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:36:51.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.020+0000 7fad60400640 1 Processor -- start 2024-03-06T07:36:51.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.020+0000 7fad60400640 1 -- start start 2024-03-06T07:36:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.022+0000 7fad60400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fad580cf5c0 0x7fad580cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.022+0000 7fad60400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad58071050 0x7fad58071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.022+0000 7fad60400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580b4860 con 0x7fad580b3c90 2024-03-06T07:36:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.022+0000 7fad60400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad580ce160 con 0x7fad580cf5c0 2024-03-06T07:36:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.022+0000 7fad60400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad580b4b40 con 0x7fad58071050 2024-03-06T07:36:51.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.022+0000 7fad5d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fad580cf5c0 0x7fad580cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fad580cf5c0 0x7fad580cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:43004/0 (socket says 172.21.15.32:43004) 2024-03-06T07:36:51.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5d000640 1 -- 172.21.15.32:0/1677148266 learned_addr learned my addr 172.21.15.32:0/1677148266 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:51.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5e400640 1 --2- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad58071050 0x7fad58071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5d000640 1 -- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad58071050 msgr2=0x7fad58071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:51.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5d000640 1 --2- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad58071050 0x7fad58071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5d000640 1 -- 172.21.15.32:0/1677148266 >> v1:172.21.15.39:6789/0 conn(0x7fad580b3c90 legacy=0x7fad580b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.023+0000 7fad5d000640 1 -- 172.21.15.32:0/1677148266 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580b4e20 con 0x7fad580cf5c0 2024-03-06T07:36:51.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.025+0000 7fad5d000640 1 --2- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad580cf5c0 0x7fad580cdc20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fad4400bb10 tx=0x7fad4402e1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca0c404a2c73d321 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.025+0000 7fad57e00640 1 -- 172.21.15.32:0/1677148266 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4403b070 con 0x7fad580cf5c0 2024-03-06T07:36:51.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.025+0000 7fad57e00640 1 -- 172.21.15.32:0/1677148266 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44036440 con 0x7fad580cf5c0 2024-03-06T07:36:51.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.025+0000 7fad57e00640 1 -- 172.21.15.32:0/1677148266 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44036810 con 0x7fad580cf5c0 2024-03-06T07:36:51.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 -- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad580cf5c0 msgr2=0x7fad580cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 --2- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad580cf5c0 0x7fad580cdc20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fad4400bb10 tx=0x7fad4402e1e0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 -- 172.21.15.32:0/1677148266 shutdown_connections 2024-03-06T07:36:51.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 --2- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad58071050 0x7fad58071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 --2- 172.21.15.32:0/1677148266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad580cf5c0 0x7fad580cdc20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 -- 172.21.15.32:0/1677148266 >> 172.21.15.32:0/1677148266 conn(0x7fad5806afb0 msgr2=0x7fad5806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 -- 172.21.15.32:0/1677148266 shutdown_connections 2024-03-06T07:36:51.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.026+0000 7fad60400640 1 -- 172.21.15.32:0/1677148266 wait complete. 2024-03-06T07:36:51.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.029+0000 7fad60400640 1 Processor -- start 2024-03-06T07:36:51.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.029+0000 7fad60400640 1 -- start start 2024-03-06T07:36:51.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad60400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad58071050 0x7fad5814ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad60400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad580b3c90 0x7fad581f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad60400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 0x7fad580c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad60400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad580719d0 con 0x7fad580cf5c0 2024-03-06T07:36:51.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad60400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5815b340 con 0x7fad58071050 2024-03-06T07:36:51.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad60400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad58071b50 con 0x7fad580b3c90 2024-03-06T07:36:51.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad5e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 0x7fad580c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad5e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 0x7fad580c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34422/0 (socket says 172.21.15.32:34422) 2024-03-06T07:36:51.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.030+0000 7fad5e400640 1 -- 172.21.15.32:0/2885612594 learned_addr learned my addr 172.21.15.32:0/2885612594 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5d000640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad580b3c90 0x7fad581f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5da00640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad58071050 0x7fad5814ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5e400640 1 -- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad580b3c90 msgr2=0x7fad581f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5e400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad580b3c90 0x7fad581f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5e400640 1 -- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad58071050 msgr2=0x7fad5814ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5e400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad58071050 0x7fad5814ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5e400640 1 -- 172.21.15.32:0/2885612594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580cade0 con 0x7fad580cf5c0 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5da00640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad58071050 0x7fad5814ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad5e400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 0x7fad580c9150 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fad5000efc0 tx=0x7fad5000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50018020 con 0x7fad580cf5c0 2024-03-06T07:36:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad50014480 con 0x7fad580cf5c0 2024-03-06T07:36:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50014870 con 0x7fad580cf5c0 2024-03-06T07:36:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.031+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad580cb0c0 con 0x7fad580cf5c0 2024-03-06T07:36:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.032+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad580cb680 con 0x7fad580cf5c0 2024-03-06T07:36:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.033+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5804f3a0 con 0x7fad580cf5c0 2024-03-06T07:36:51.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.034+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad50005d10 con 0x7fad580cf5c0 2024-03-06T07:36:51.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.034+0000 7fad55600640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.035+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad5008aca0 con 0x7fad580cf5c0 2024-03-06T07:36:51.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.035+0000 7fad5da00640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.036+0000 7fad5da00640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fad4c001680 tx=0x7fad4c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.040+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad50022030 con 0x7fad580cf5c0 2024-03-06T07:36:51.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.311+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fad580715c0 con 0x7fad580cf5c0 2024-03-06T07:36:51.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.312+0000 7fad55600640 1 -- 172.21.15.32:0/2885612594 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fad50010040 con 0x7fad580cf5c0 2024-03-06T07:36:51.314 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:51.314 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:36:51.314 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-03-06T07:36:51.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.314 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlAQCNHBAAL48dy/6apleJgGSU6391xg==", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlGkH8AxAAzDh/lihzVj/M3HMfm1NXqA==", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:36:51.315 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhl0Ya8LRAAbeVSYJ2ALqZz2qJcJz6yVQ==", 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-03-06T07:36:51.316 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlFO+1FRAAiG7Ee77tj0CmM5wqpVu9Zw==", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8GuhlspKqBhAAszuV3+JCeIfOW5aZUgi+FQ==", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.317 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD/GuhlT/MmERAAmvfo933jHqLsD8JzgAV29g==", 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQACG+hlZFUxKxAAeq/9ghKbhVTBHVJp8eeqrg==", 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.318 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAGG+hl2dgzCRAA7lL3r2xYR9aV9lrYT1Elow==", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-03-06T07:36:51.319 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAQG+hlDxKUHxAA7z02RneQB1Z3YW6YJOYjcQ==", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQATG+hlrzPOOhAA8GSX7AawivVbdtBkX2DSQw==", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAXG+hlJgPvCxAAyRKu+sz06SdyftWb/IYNRg==", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAaG+hlXZ7eGBAAXu8oMqfrq5iZJ4hrYVERyQ==", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-03-06T07:36:51.321 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg==", 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6GuhlFsgHABAAkhVKLztvIvoH1HtykIe3Lg==", 2024-03-06T07:36:51.322 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD5GuhlwyoqMBAAgtkFHKQF+jVc/Drib6Ohmw==", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD6Guhl37siCxAAmBFpm1zAzJmOSS8spC3j8Q==", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD3GuhlxWQ9DBAAl/CYt48/8UDG+8R13cIoEA==", 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.324 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4Guhlq820DxAAiKCUSzCdlZPIRbRuy5KnIA==", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-03-06T07:36:51.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD4GuhlhmLEARAA2ySPiX2vXDUHB4YiM5L5Ew==", 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-03-06T07:36:51.326 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad3806b4d0 msgr2=0x7fad3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fad4c001680 tx=0x7fad4c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 msgr2=0x7fad580c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 0x7fad580c9150 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fad5000efc0 tx=0x7fad5000c490 comp rx=0 tx=0).stop 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 shutdown_connections 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad580cf5c0 0x7fad580c9150 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad580b3c90 0x7fad581f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 --2- 172.21.15.32:0/2885612594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad58071050 0x7fad5814ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 >> 172.21.15.32:0/2885612594 conn(0x7fad5806afb0 msgr2=0x7fad5815a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 shutdown_connections 2024-03-06T07:36:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.315+0000 7fad60400640 1 -- 172.21.15.32:0/2885612594 wait complete. 2024-03-06T07:36:51.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:51.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.556+0000 7faad3200640 1 Processor -- start 2024-03-06T07:36:51.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.556+0000 7faad3200640 1 -- start start 2024-03-06T07:36:51.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faad415e070 0x7faad40d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40d18c0 con 0x7faad40cf410 2024-03-06T07:36:51.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40d1ba0 con 0x7faad40d0cd0 2024-03-06T07:36:51.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40d1e80 con 0x7faad415e070 2024-03-06T07:36:51.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faad415e070 0x7faad40d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad1400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faad40cf410 0x7faad40d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39832/0 (socket says 172.21.15.32:39832) 2024-03-06T07:36:51.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.558+0000 7faad1400640 1 -- 172.21.15.32:0/1331429670 learned_addr learned my addr 172.21.15.32:0/1331429670 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:51.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.559+0000 7faad0a00640 1 -- 172.21.15.32:0/1331429670 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580626423 0 0) 0x7faad40d18c0 con 0x7faad40cf410 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.559+0000 7faad0a00640 1 -- 172.21.15.32:0/1331429670 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab8003520 con 0x7faad40cf410 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.559+0000 7faad1e00640 1 -- 172.21.15.32:0/1331429670 >> v1:172.21.15.39:6789/0 conn(0x7faad40d0cd0 legacy=0x7faad40b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.559+0000 7faad1e00640 1 -- 172.21.15.32:0/1331429670 >> v1:172.21.15.32:6789/0 conn(0x7faad40cf410 legacy=0x7faad40d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.559+0000 7faad1e00640 1 -- 172.21.15.32:0/1331429670 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40d2160 con 0x7faad415e070 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.560+0000 7faad1e00640 1 --2- 172.21.15.32:0/1331429670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad415e070 0x7faad40d2960 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7faac40097a0 tx=0x7faac402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129be95b297da8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.560+0000 7faad0a00640 1 -- 172.21.15.32:0/1331429670 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4034070 con 0x7faad415e070 2024-03-06T07:36:51.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.560+0000 7faad0a00640 1 -- 172.21.15.32:0/1331429670 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac4030610 con 0x7faad415e070 2024-03-06T07:36:51.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.560+0000 7faad0a00640 1 -- 172.21.15.32:0/1331429670 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac40309c0 con 0x7faad415e070 2024-03-06T07:36:51.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 -- 172.21.15.32:0/1331429670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad415e070 msgr2=0x7faad40d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 --2- 172.21.15.32:0/1331429670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad415e070 0x7faad40d2960 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7faac40097a0 tx=0x7faac402b960 comp rx=0 tx=0).stop 2024-03-06T07:36:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 -- 172.21.15.32:0/1331429670 shutdown_connections 2024-03-06T07:36:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 --2- 172.21.15.32:0/1331429670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad415e070 0x7faad40d2960 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 -- 172.21.15.32:0/1331429670 >> 172.21.15.32:0/1331429670 conn(0x7faad408d2a0 msgr2=0x7faad40d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 -- 172.21.15.32:0/1331429670 shutdown_connections 2024-03-06T07:36:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.561+0000 7faad3200640 1 -- 172.21.15.32:0/1331429670 wait complete. 2024-03-06T07:36:51.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.564+0000 7faad3200640 1 Processor -- start 2024-03-06T07:36:51.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.564+0000 7faad3200640 1 -- start start 2024-03-06T07:36:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faad40cf410 0x7faad4157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 0x7faad40b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faad415e070 0x7faad40cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad40d4730 con 0x7faad40d0cd0 2024-03-06T07:36:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad40d45b0 con 0x7faad40cf410 2024-03-06T07:36:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad40cf870 con 0x7faad415e070 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 0x7faad40b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 0x7faad40b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34446/0 (socket says 172.21.15.32:34446) 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.565+0000 7faad1400640 1 -- 172.21.15.32:0/481731146 learned_addr learned my addr 172.21.15.32:0/481731146 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad2800640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faad415e070 0x7faad40cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1e00640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faad40cf410 0x7faad4157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1400640 1 -- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faad415e070 msgr2=0x7faad40cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1400640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faad415e070 0x7faad40cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1400640 1 -- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faad40cf410 msgr2=0x7faad4157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1400640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faad40cf410 0x7faad4157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1400640 1 -- 172.21.15.32:0/481731146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40b73c0 con 0x7faad40d0cd0 2024-03-06T07:36:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.566+0000 7faad1400640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 0x7faad40b8220 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7faabc00c530 tx=0x7faabc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.567+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc017070 con 0x7faad40d0cd0 2024-03-06T07:36:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.567+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc005ce0 con 0x7faad40d0cd0 2024-03-06T07:36:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.567+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc0072b0 con 0x7faad40d0cd0 2024-03-06T07:36:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.567+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad40b76a0 con 0x7faad40d0cd0 2024-03-06T07:36:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.567+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad40b93e0 con 0x7faad40d0cd0 2024-03-06T07:36:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.568+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad404f330 con 0x7faad40d0cd0 2024-03-06T07:36:51.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.569+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faabc01a070 con 0x7faad40d0cd0 2024-03-06T07:36:51.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.570+0000 7faaca000640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faab006b4d0 0x7faab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.570+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faabc089420 con 0x7faad40d0cd0 2024-03-06T07:36:51.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.570+0000 7faad1e00640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faab006b4d0 0x7faab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.571+0000 7faad1e00640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faab006b4d0 0x7faab006d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7faac4036000 tx=0x7faac4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.574+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faabc058580 con 0x7faad40d0cd0 2024-03-06T07:36:51.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.781+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faad415e450 con 0x7faad40d0cd0 2024-03-06T07:36:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.782+0000 7faaca000640 1 -- 172.21.15.32:0/481731146 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+12793 (secure 0 0 0) 0x7faabc057e80 con 0x7faad40d0cd0 2024-03-06T07:36:51.784 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:51.785 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":123,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:36:50.732023+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:51.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faab006b4d0 msgr2=0x7faab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faab006b4d0 0x7faab006d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7faac4036000 tx=0x7faac4008030 comp rx=0 tx=0).stop 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 msgr2=0x7faad40b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 0x7faad40b8220 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7faabc00c530 tx=0x7faabc00c9f0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 shutdown_connections 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faab006b4d0 0x7faab006d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faad415e070 0x7faad40cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad40d0cd0 0x7faad40b8220 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 --2- 172.21.15.32:0/481731146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faad40cf410 0x7faad4157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 >> 172.21.15.32:0/481731146 conn(0x7faad408d2a0 msgr2=0x7faad406cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 shutdown_connections 2024-03-06T07:36:51.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:51.785+0000 7faad3200640 1 -- 172.21.15.32:0/481731146 wait complete. 2024-03-06T07:36:51.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:36:51.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:36:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.027+0000 7f3013e00640 1 Processor -- start 2024-03-06T07:36:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.028+0000 7f3013e00640 1 -- start start 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f3013e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3014147590 0x7f3014071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f3013e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30140b2250 0x7f30140b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f3013e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30140719c0 con 0x7f301406d8d0 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f3013e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3014147970 con 0x7f30140b2250 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f3013e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30140b31b0 con 0x7f3014147590 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f300a000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f301406d8d0 0x7f30140b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39846/0 (socket says 172.21.15.32:39846) 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f300a000640 1 -- 172.21.15.32:0/3084576521 learned_addr learned my addr 172.21.15.32:0/3084576521 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.029+0000 7f3013400640 1 --2- 172.21.15.32:0/3084576521 >> v2:172.21.15.32:3300/0 conn(0x7f30140b2250 0x7f30140b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3012a00640 1 --2- 172.21.15.32:0/3084576521 >> v2:172.21.15.39:3300/0 conn(0x7f3014147590 0x7f3014071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3012000640 1 -- 172.21.15.32:0/3084576521 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340799118 0 0) 0x7f30140719c0 con 0x7f301406d8d0 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3012000640 1 -- 172.21.15.32:0/3084576521 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff8003520 con 0x7f301406d8d0 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3013400640 1 -- 172.21.15.32:0/3084576521 >> v2:172.21.15.39:3300/0 conn(0x7f3014147590 msgr2=0x7f3014071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3013400640 1 --2- 172.21.15.32:0/3084576521 >> v2:172.21.15.39:3300/0 conn(0x7f3014147590 0x7f3014071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3013400640 1 -- 172.21.15.32:0/3084576521 >> v1:172.21.15.32:6789/0 conn(0x7f301406d8d0 legacy=0x7f30140b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.030+0000 7f3013400640 1 -- 172.21.15.32:0/3084576521 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301406de40 con 0x7f30140b2250 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.031+0000 7f3013400640 1 --2- 172.21.15.32:0/3084576521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30140b2250 0x7f30140b3640 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f30140b2790 tx=0x7f300c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93f746f84ef81f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.031+0000 7f3012000640 1 -- 172.21.15.32:0/3084576521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c039070 con 0x7f30140b2250 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.031+0000 7f3012000640 1 -- 172.21.15.32:0/3084576521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c02cc80 con 0x7f30140b2250 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3012000640 1 -- 172.21.15.32:0/3084576521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c034640 con 0x7f30140b2250 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 -- 172.21.15.32:0/3084576521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30140b2250 msgr2=0x7f30140b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 --2- 172.21.15.32:0/3084576521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30140b2250 0x7f30140b3640 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f30140b2790 tx=0x7f300c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 -- 172.21.15.32:0/3084576521 shutdown_connections 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 --2- 172.21.15.32:0/3084576521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30140b2250 0x7f30140b3640 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 --2- 172.21.15.32:0/3084576521 >> v2:172.21.15.39:3300/0 conn(0x7f3014147590 0x7f3014071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 -- 172.21.15.32:0/3084576521 >> 172.21.15.32:0/3084576521 conn(0x7f301406b3a0 msgr2=0x7f301406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 -- 172.21.15.32:0/3084576521 shutdown_connections 2024-03-06T07:36:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.032+0000 7f3013e00640 1 -- 172.21.15.32:0/3084576521 wait complete. 2024-03-06T07:36:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.034+0000 7f3013e00640 1 Processor -- start 2024-03-06T07:36:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.034+0000 7f3013e00640 1 -- start start 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f301406d8d0 0x7f301415f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30140b2250 0x7f301415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 0x7f3014131060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30140b2b50 con 0x7f3014147590 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30140b2e40 con 0x7f30140b2250 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30140b3b80 con 0x7f301406d8d0 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 0x7f3014131060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 0x7f3014131060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34464/0 (socket says 172.21.15.32:34464) 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.035+0000 7f3013400640 1 -- 172.21.15.32:0/4071101843 learned_addr learned my addr 172.21.15.32:0/4071101843 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3012a00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f301406d8d0 0x7f301415f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f300a000640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30140b2250 0x7f301415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3013400640 1 -- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f301406d8d0 msgr2=0x7f301415f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3013400640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f301406d8d0 0x7f301415f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3013400640 1 -- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30140b2250 msgr2=0x7f301415fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3013400640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30140b2250 0x7f301415fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3013400640 1 -- 172.21.15.32:0/4071101843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30140cefb0 con 0x7f3014147590 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3012a00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f301406d8d0 0x7f301415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.036+0000 7f3013400640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 0x7f3014131060 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f300c009e80 tx=0x7f300c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.037+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c039050 con 0x7f3014147590 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.037+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c045070 con 0x7f3014147590 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.037+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30141603b0 con 0x7f3014147590 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.037+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c03e830 con 0x7f3014147590 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.037+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3014160970 con 0x7f3014147590 2024-03-06T07:36:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.038+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301404f330 con 0x7f3014147590 2024-03-06T07:36:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.040+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f300c03e9d0 con 0x7f3014147590 2024-03-06T07:36:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.040+0000 7f300b400640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2ff006b4d0 0x7f2ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.040+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f300c0acf90 con 0x7f3014147590 2024-03-06T07:36:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.041+0000 7f3012a00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2ff006b4d0 0x7f2ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.041+0000 7f3012a00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2ff006b4d0 0x7f2ff006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3004006fd0 tx=0x7f3004008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.045+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f300c032e20 con 0x7f3014147590 2024-03-06T07:36:52.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.249+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30140b4c40 con 0x7f3014147590 2024-03-06T07:36:52.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.249+0000 7f300b400640 1 -- 172.21.15.32:0/4071101843 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7f300c07c270 con 0x7f3014147590 2024-03-06T07:36:52.252 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:36:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2ff006b4d0 msgr2=0x7f2ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2ff006b4d0 0x7f2ff006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3004006fd0 tx=0x7f3004008040 comp rx=0 tx=0).stop 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 msgr2=0x7f3014131060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 0x7f3014131060 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f300c009e80 tx=0x7f300c004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 shutdown_connections 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2ff006b4d0 0x7f2ff006d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3014147590 0x7f3014131060 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30140b2250 0x7f301415fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 --2- 172.21.15.32:0/4071101843 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f301406d8d0 0x7f301415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 >> 172.21.15.32:0/4071101843 conn(0x7f301406b3a0 msgr2=0x7f301406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 shutdown_connections 2024-03-06T07:36:52.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.252+0000 7f3013e00640 1 -- 172.21.15.32:0/4071101843 wait complete. 2024-03-06T07:36:52.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:36:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.500+0000 7f7b77200640 1 Processor -- start 2024-03-06T07:36:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.500+0000 7f7b77200640 1 -- start start 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.501+0000 7f7b77200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b7815e0b0 0x7f7b780cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.501+0000 7f7b77200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7b780d4a10 0x7f7b780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.501+0000 7f7b77200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d0c80 0x7f7b780d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.501+0000 7f7b77200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b780b09e0 con 0x7f7b7815e0b0 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.501+0000 7f7b77200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b780b3710 con 0x7f7b780d4a10 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.501+0000 7f7b77200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780b39f0 con 0x7f7b780d0c80 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b7815e0b0 0x7f7b780cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b7815e0b0 0x7f7b780cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34470/0 (socket says 172.21.15.32:34470) 2024-03-06T07:36:52.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 -- 172.21.15.32:0/3537860617 learned_addr learned my addr 172.21.15.32:0/3537860617 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:52.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b6fe00640 1 --2- 172.21.15.32:0/3537860617 >> v2:172.21.15.39:3300/0 conn(0x7f7b780d4a10 0x7f7b780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 -- 172.21.15.32:0/3537860617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d0c80 msgr2=0x7f7b780d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 --2- 172.21.15.32:0/3537860617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d0c80 0x7f7b780d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 -- 172.21.15.32:0/3537860617 >> v2:172.21.15.39:3300/0 conn(0x7f7b780d4a10 msgr2=0x7f7b780d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 --2- 172.21.15.32:0/3537860617 >> v2:172.21.15.39:3300/0 conn(0x7f7b780d4a10 0x7f7b780d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.502+0000 7f7b75e00640 1 -- 172.21.15.32:0/3537860617 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780b3cd0 con 0x7f7b7815e0b0 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.503+0000 7f7b75e00640 1 --2- 172.21.15.32:0/3537860617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cf310 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7b7806b8c0 tx=0x7f7b6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f47f2a35c2499e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b75400640 1 -- 172.21.15.32:0/3537860617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68035020 con 0x7f7b7815e0b0 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b75400640 1 -- 172.21.15.32:0/3537860617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68031870 con 0x7f7b7815e0b0 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b75400640 1 -- 172.21.15.32:0/3537860617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68031c40 con 0x7f7b7815e0b0 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b77200640 1 -- 172.21.15.32:0/3537860617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 msgr2=0x7f7b780cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b77200640 1 --2- 172.21.15.32:0/3537860617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cf310 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7b7806b8c0 tx=0x7f7b6802b910 comp rx=0 tx=0).stop 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b77200640 1 -- 172.21.15.32:0/3537860617 shutdown_connections 2024-03-06T07:36:52.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.504+0000 7f7b77200640 1 --2- 172.21.15.32:0/3537860617 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d0c80 0x7f7b780d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.505+0000 7f7b77200640 1 --2- 172.21.15.32:0/3537860617 >> v2:172.21.15.39:3300/0 conn(0x7f7b780d4a10 0x7f7b780d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.505+0000 7f7b77200640 1 --2- 172.21.15.32:0/3537860617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cf310 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.505+0000 7f7b77200640 1 -- 172.21.15.32:0/3537860617 >> 172.21.15.32:0/3537860617 conn(0x7f7b780d3d30 msgr2=0x7f7b780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.505+0000 7f7b77200640 1 -- 172.21.15.32:0/3537860617 shutdown_connections 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.505+0000 7f7b77200640 1 -- 172.21.15.32:0/3537860617 wait complete. 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.506+0000 7f7b77200640 1 Processor -- start 2024-03-06T07:36:52.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.506+0000 7f7b77200640 1 -- start start 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.507+0000 7f7b77200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b780d0c80 0x7f7b7815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.507+0000 7f7b77200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d4a10 0x7f7b780b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.507+0000 7f7b77200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.507+0000 7f7b77200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780cfc60 con 0x7f7b7815e0b0 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.507+0000 7f7b77200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7815e490 con 0x7f7b780d0c80 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.507+0000 7f7b77200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780cf850 con 0x7f7b780d4a10 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34474/0 (socket says 172.21.15.32:34474) 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 -- 172.21.15.32:0/3586514108 learned_addr learned my addr 172.21.15.32:0/3586514108 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b6fe00640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d4a10 0x7f7b780b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b75e00640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b780d0c80 0x7f7b7815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 -- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d4a10 msgr2=0x7f7b780b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d4a10 0x7f7b780b7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 -- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b780d0c80 msgr2=0x7f7b7815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b780d0c80 0x7f7b7815ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.508+0000 7f7b76800640 1 -- 172.21.15.32:0/3586514108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780b7e20 con 0x7f7b7815e0b0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.509+0000 7f7b76800640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cce70 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b9a0 tx=0x7f7b7000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.509+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70019070 con 0x7f7b7815e0b0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.509+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7000cbc0 con 0x7f7b7815e0b0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.509+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70014440 con 0x7f7b7815e0b0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.509+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780b8100 con 0x7f7b7815e0b0 2024-03-06T07:36:52.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.509+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b780cae00 con 0x7f7b7815e0b0 2024-03-06T07:36:52.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.510+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7804f330 con 0x7f7b7815e0b0 2024-03-06T07:36:52.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.511+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b7001c070 con 0x7f7b7815e0b0 2024-03-06T07:36:52.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.512+0000 7f7b6e000640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b5406b4d0 0x7f7b5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.512+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b70089cd0 con 0x7f7b7815e0b0 2024-03-06T07:36:52.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.513+0000 7f7b75e00640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b5406b4d0 0x7f7b5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.513+0000 7f7b75e00640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b5406b4d0 0x7f7b5406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7b68036000 tx=0x7f7b6803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.517+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b70058e00 con 0x7f7b7815e0b0 2024-03-06T07:36:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.723+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7b7806d640 con 0x7f7b7815e0b0 2024-03-06T07:36:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.724+0000 7f7b6e000640 1 -- 172.21.15.32:0/3586514108 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7b70058700 con 0x7f7b7815e0b0 2024-03-06T07:36:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-03-06T07:36:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-03-06T07:36:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-03-06T07:36:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-03-06T07:36:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-03-06T07:36:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-03-06T07:36:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b5406b4d0 msgr2=0x7f7b5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b5406b4d0 0x7f7b5406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7b68036000 tx=0x7f7b6803c040 comp rx=0 tx=0).stop 2024-03-06T07:36:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 msgr2=0x7f7b780cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cce70 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b9a0 tx=0x7f7b7000be60 comp rx=0 tx=0).stop 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 shutdown_connections 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7b5406b4d0 0x7f7b5406d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b7815e0b0 0x7f7b780cce70 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7b780d4a10 0x7f7b780b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 --2- 172.21.15.32:0/3586514108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b780d0c80 0x7f7b7815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 >> 172.21.15.32:0/3586514108 conn(0x7f7b780d3d30 msgr2=0x7f7b780b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 shutdown_connections 2024-03-06T07:36:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.727+0000 7f7b77200640 1 -- 172.21.15.32:0/3586514108 wait complete. 2024-03-06T07:36:52.752 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:36:52.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:36:52.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.970+0000 7f6569800640 1 Processor -- start 2024-03-06T07:36:52.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.970+0000 7f6569800640 1 -- start start 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.971+0000 7f6569800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6564144540 0x7f656415f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.971+0000 7f6569800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f656406e200 0x7f65640cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.971+0000 7f6569800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656415f5a0 con 0x7f65640b2ac0 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.971+0000 7f6569800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65640b3ff0 con 0x7f656406e200 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.971+0000 7f6569800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65640b2f50 con 0x7f6564144540 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6568e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f656406e200 0x7f65640cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6563400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f65640b2ac0 0x7f656406daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39860/0 (socket says 172.21.15.32:39860) 2024-03-06T07:36:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6568e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f656406e200 0x7f65640cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34482/0 (socket says 172.21.15.32:34482) 2024-03-06T07:36:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6563400640 1 -- 172.21.15.32:0/2242568469 learned_addr learned my addr 172.21.15.32:0/2242568469 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6563e00640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6564144540 0x7f656415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6568e00640 1 -- 172.21.15.32:0/2242568469 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6564144540 msgr2=0x7f656415f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6568e00640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6564144540 0x7f656415f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6568e00640 1 -- 172.21.15.32:0/2242568469 >> v1:172.21.15.32:6789/0 conn(0x7f65640b2ac0 legacy=0x7f656406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.973+0000 7f6568e00640 1 -- 172.21.15.32:0/2242568469 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640b3230 con 0x7f656406e200 2024-03-06T07:36:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.972+0000 7f6562a00640 1 -- 172.21.15.32:0/2242568469 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151696517 0 0) 0x7f656415f5a0 con 0x7f65640b2ac0 2024-03-06T07:36:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.973+0000 7f6563e00640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6564144540 0x7f656415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.974+0000 7f6568e00640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656406e200 0x7f65640cdee0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f655c00bfa0 tx=0x7f655c02e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc15fdb39f5d5bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.974+0000 7f6562a00640 1 -- 172.21.15.32:0/2242568469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c02ea40 con 0x7f656406e200 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.974+0000 7f6562a00640 1 -- 172.21.15.32:0/2242568469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c02ebe0 con 0x7f656406e200 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.974+0000 7f6562a00640 1 -- 172.21.15.32:0/2242568469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c009540 con 0x7f656406e200 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.974+0000 7f6569800640 1 -- 172.21.15.32:0/2242568469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656406e200 msgr2=0x7f65640cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.974+0000 7f6569800640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656406e200 0x7f65640cdee0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f655c00bfa0 tx=0x7f655c02e0a0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.975+0000 7f6569800640 1 -- 172.21.15.32:0/2242568469 shutdown_connections 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.975+0000 7f6569800640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656406e200 0x7f65640cdee0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.975+0000 7f6569800640 1 --2- 172.21.15.32:0/2242568469 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6564144540 0x7f656415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.975+0000 7f6569800640 1 -- 172.21.15.32:0/2242568469 >> 172.21.15.32:0/2242568469 conn(0x7f65640b17c0 msgr2=0x7f65640b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.975+0000 7f6569800640 1 -- 172.21.15.32:0/2242568469 shutdown_connections 2024-03-06T07:36:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.975+0000 7f6569800640 1 -- 172.21.15.32:0/2242568469 wait complete. 2024-03-06T07:36:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.976+0000 7f6569800640 1 Processor -- start 2024-03-06T07:36:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.977+0000 7f6569800640 1 -- start start 2024-03-06T07:36:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.977+0000 7f6569800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f656406e200 0x7f65641f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.977+0000 7f6569800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564144540 0x7f6564155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.977+0000 7f6569800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 0x7f6564070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.977+0000 7f6569800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65640b3ad0 con 0x7f656413c440 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6569800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65640b37f0 con 0x7f6564144540 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6569800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65640b4a30 con 0x7f656406e200 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 0x7f6564070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 0x7f6564070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34488/0 (socket says 172.21.15.32:34488) 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 -- 172.21.15.32:0/15277559 learned_addr learned my addr 172.21.15.32:0/15277559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6563e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f656406e200 0x7f65641f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6563400640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564144540 0x7f6564155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 -- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f656406e200 msgr2=0x7f65641f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f656406e200 0x7f65641f41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 -- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564144540 msgr2=0x7f6564155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564144540 0x7f6564155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6568e00640 1 -- 172.21.15.32:0/15277559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65641f46f0 con 0x7f656413c440 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6563e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f656406e200 0x7f65641f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.978+0000 7f6563400640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564144540 0x7f6564155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.979+0000 7f6568e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 0x7f6564070a80 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f655c038040 tx=0x7f655c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.979+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0092b0 con 0x7f656413c440 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.979+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c009e00 con 0x7f656413c440 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.979+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656413b3a0 con 0x7f656413c440 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.979+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c036840 con 0x7f656413c440 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.979+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656413b960 con 0x7f656413c440 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.980+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656404f330 con 0x7f656413c440 2024-03-06T07:36:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.981+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f655c047020 con 0x7f656413c440 2024-03-06T07:36:52.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.982+0000 7f655be00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f654406b670 0x7f654406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.982+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f655c0ac540 con 0x7f656413c440 2024-03-06T07:36:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.982+0000 7f6563e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f654406b670 0x7f654406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.983+0000 7f6563e00640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f654406b670 0x7f654406db30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6554006fd0 tx=0x7f6554008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:52.986+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f655c04e020 con 0x7f656413c440 2024-03-06T07:36:53.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.186+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f656415a880 con 0x7f656413c440 2024-03-06T07:36:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.763+0000 7f655be00640 1 -- 172.21.15.32:0/15277559 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v124) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65641f46f0 con 0x7f656413c440 2024-03-06T07:36:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.767+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f654406b670 msgr2=0x7f654406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.767+0000 7f6569800640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f654406b670 0x7f654406db30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6554006fd0 tx=0x7f6554008040 comp rx=0 tx=0).stop 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.767+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 msgr2=0x7f6564070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.767+0000 7f6569800640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 0x7f6564070a80 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f655c038040 tx=0x7f655c004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 shutdown_connections 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f654406b670 0x7f654406db30 secure :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6554006fd0 tx=0x7f6554008040 comp rx=0 tx=0).stop 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656413c440 0x7f6564070a80 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6564144540 0x7f6564155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 --2- 172.21.15.32:0/15277559 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f656406e200 0x7f65641f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 >> 172.21.15.32:0/15277559 conn(0x7f65640b17c0 msgr2=0x7f6564142cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 shutdown_connections 2024-03-06T07:36:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:53.768+0000 7f6569800640 1 -- 172.21.15.32:0/15277559 wait complete. 2024-03-06T07:36:53.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:36:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.004+0000 7faf6ae00640 1 Processor -- start 2024-03-06T07:36:54.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.004+0000 7faf6ae00640 1 -- start start 2024-03-06T07:36:54.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.006+0000 7faf6ae00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf640ad6f0 0x7faf6415c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.006+0000 7faf6ae00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf64156830 con 0x7faf6415f960 2024-03-06T07:36:54.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.006+0000 7faf6ae00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf64160120 con 0x7faf6406d5d0 2024-03-06T07:36:54.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.006+0000 7faf6ae00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6406ce60 con 0x7faf640ad6f0 2024-03-06T07:36:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.007+0000 7faf5be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faf6415f960 0x7faf640adad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39872/0 (socket says 172.21.15.32:39872) 2024-03-06T07:36:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.007+0000 7faf5be00640 1 -- 172.21.15.32:0/3480567594 learned_addr learned my addr 172.21.15.32:0/3480567594 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.007+0000 7faf63e00640 1 --2- 172.21.15.32:0/3480567594 >> v2:172.21.15.39:3300/0 conn(0x7faf640ad6f0 0x7faf6415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.007+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097298000 0 0) 0x7faf64156830 con 0x7faf6415f960 2024-03-06T07:36:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.007+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf48003520 con 0x7faf6415f960 2024-03-06T07:36:54.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738677079 0 0) 0x7faf48003520 con 0x7faf6415f960 2024-03-06T07:36:54.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 >> v2:172.21.15.39:3300/0 conn(0x7faf640ad6f0 msgr2=0x7faf6415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:54.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 --2- 172.21.15.32:0/3480567594 >> v2:172.21.15.39:3300/0 conn(0x7faf640ad6f0 0x7faf6415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 >> v1:172.21.15.39:6789/0 conn(0x7faf6406d5d0 legacy=0x7faf6406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf64160400 con 0x7faf6415f960 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7faf4c001670 con 0x7faf6415f960 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 >> v1:172.21.15.32:6789/0 conn(0x7faf6415f960 legacy=0x7faf640adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf480039b0 0x7faf48003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480043d0 0x7faf480048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480050a0 0x7faf4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6406ce60 con 0x7faf480043d0 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4800bfe0 con 0x7faf480039b0 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4800c1e0 con 0x7faf480050a0 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf5be00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480043d0 0x7faf480048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf63e00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf480039b0 0x7faf48003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.008+0000 7faf68e00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480050a0 0x7faf4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf5be00640 1 -- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480050a0 msgr2=0x7faf4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:54.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf5be00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480050a0 0x7faf4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf5be00640 1 -- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf480039b0 msgr2=0x7faf48003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf5be00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf480039b0 0x7faf48003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf5be00640 1 -- 172.21.15.32:0/3480567594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48003520 con 0x7faf480043d0 2024-03-06T07:36:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf63e00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf480039b0 0x7faf48003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf68e00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480050a0 0x7faf4800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.009+0000 7faf68e00640 1 -- 172.21.15.32:0/3480567594 reap_dead start 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.010+0000 7faf5be00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480043d0 0x7faf480048a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c240 tx=0x7faf48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c792d0f3d3bc49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.010+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c007d30 con 0x7faf480043d0 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.010+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c030070 con 0x7faf480043d0 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.010+0000 7faf63400640 1 -- 172.21.15.32:0/3480567594 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c02cd80 con 0x7faf480043d0 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 -- 172.21.15.32:0/3480567594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480043d0 msgr2=0x7faf480048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480043d0 0x7faf480048a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c240 tx=0x7faf48005830 comp rx=0 tx=0).stop 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 -- 172.21.15.32:0/3480567594 shutdown_connections 2024-03-06T07:36:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3480567594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480043d0 0x7faf480048a0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 -- 172.21.15.32:0/3480567594 >> 172.21.15.32:0/3480567594 conn(0x7faf6406b3d0 msgr2=0x7faf6406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 -- 172.21.15.32:0/3480567594 shutdown_connections 2024-03-06T07:36:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.011+0000 7faf6ae00640 1 -- 172.21.15.32:0/3480567594 wait complete. 2024-03-06T07:36:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.013+0000 7faf6ae00640 1 Processor -- start 2024-03-06T07:36:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.013+0000 7faf6ae00640 1 -- start start 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf6ae00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480043d0 0x7faf6414edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf6ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 0x7faf641442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf6ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6414e630 0x7faf64151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf6ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6415fdf0 con 0x7faf64150650 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf6ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6406d0b0 con 0x7faf6414e630 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf6ae00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6415c370 con 0x7faf480043d0 2024-03-06T07:36:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 0x7faf641442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 0x7faf641442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34514/0 (socket says 172.21.15.32:34514) 2024-03-06T07:36:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf5be00640 1 -- 172.21.15.32:0/3714949965 learned_addr learned my addr 172.21.15.32:0/3714949965 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf5be00640 1 -- 172.21.15.32:0/3714949965 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480043d0 msgr2=0x7faf6414edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf5be00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480043d0 0x7faf6414edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.014+0000 7faf5be00640 1 -- 172.21.15.32:0/3714949965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6414e630 msgr2=0x7faf64151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf5be00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6414e630 0x7faf64151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf5be00640 1 -- 172.21.15.32:0/3714949965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6414f310 con 0x7faf64150650 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf5be00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 0x7faf641442c0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c360 tx=0x7faf4c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c046070 con 0x7faf64150650 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c004bf0 con 0x7faf64150650 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6414f5b0 con 0x7faf64150650 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c007d30 con 0x7faf64150650 2024-03-06T07:36:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.015+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6414fb70 con 0x7faf64150650 2024-03-06T07:36:54.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.016+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf640af400 con 0x7faf64150650 2024-03-06T07:36:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.018+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf4c007070 con 0x7faf64150650 2024-03-06T07:36:54.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.019+0000 7faf60c00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf4006b4d0 0x7faf4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.019+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5227+0+0 (secure 0 0 0) 0x7faf4c0ab400 con 0x7faf64150650 2024-03-06T07:36:54.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.019+0000 7faf63e00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf4006b4d0 0x7faf4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.020+0000 7faf63e00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf4006b4d0 0x7faf4006d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7faf540099b0 tx=0x7faf54008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.024+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf640af400 con 0x7faf64150650 2024-03-06T07:36:54.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.224+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faf64068010 con 0x7faf64150650 2024-03-06T07:36:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.780+0000 7faf60c00640 1 -- 172.21.15.32:0/3714949965 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v125) v1 ==== 170+0+0 (secure 0 0 0) 0x7faf4c07a330 con 0x7faf64150650 2024-03-06T07:36:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-03-06T07:36:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf4006b4d0 msgr2=0x7faf4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf4006b4d0 0x7faf4006d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7faf540099b0 tx=0x7faf54008040 comp rx=0 tx=0).stop 2024-03-06T07:36:54.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 msgr2=0x7faf641442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 0x7faf641442c0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c360 tx=0x7faf4c02e070 comp rx=0 tx=0).stop 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 shutdown_connections 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7faf4006b4d0 0x7faf4006d990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf6414e630 0x7faf64151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf64150650 0x7faf641442c0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 --2- 172.21.15.32:0/3714949965 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faf480043d0 0x7faf6414edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.784+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 >> 172.21.15.32:0/3714949965 conn(0x7faf6406b3d0 msgr2=0x7faf640af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.785+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 shutdown_connections 2024-03-06T07:36:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:54.785+0000 7faf6ae00640 1 -- 172.21.15.32:0/3714949965 wait complete. 2024-03-06T07:36:54.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:36:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.011+0000 7f2fe9e00640 1 Processor -- start 2024-03-06T07:36:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.012+0000 7f2fe9e00640 1 -- start start 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.013+0000 7f2fe9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fe415fa40 0x7f2fe415fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.013+0000 7f2fe9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2fe40cf340 0x7f2fe40cc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.013+0000 7f2fe9e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe40cfda0 con 0x7f2fe40cf8a0 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.013+0000 7f2fe9e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe415d3b0 con 0x7f2fe415fa40 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.013+0000 7f2fe9e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe40ce860 con 0x7f2fe40cf340 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.014+0000 7f2fe2a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fe415fa40 0x7f2fe415fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.014+0000 7f2fe2a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fe415fa40 0x7f2fe415fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34520/0 (socket says 172.21.15.32:34520) 2024-03-06T07:36:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.014+0000 7f2fe2a00640 1 -- 172.21.15.32:0/1664882252 learned_addr learned my addr 172.21.15.32:0/1664882252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.014+0000 7f2fe3400640 1 --2- 172.21.15.32:0/1664882252 >> v2:172.21.15.39:3300/0 conn(0x7f2fe40cf340 0x7f2fe40cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.015+0000 7f2fe2a00640 1 -- 172.21.15.32:0/1664882252 >> v2:172.21.15.39:3300/0 conn(0x7f2fe40cf340 msgr2=0x7f2fe40cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.015+0000 7f2fe2a00640 1 --2- 172.21.15.32:0/1664882252 >> v2:172.21.15.39:3300/0 conn(0x7f2fe40cf340 0x7f2fe40cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.015+0000 7f2fe2a00640 1 -- 172.21.15.32:0/1664882252 >> v1:172.21.15.39:6789/0 conn(0x7f2fe40cf8a0 legacy=0x7f2fe4160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.015+0000 7f2fe2a00640 1 -- 172.21.15.32:0/1664882252 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40ceb40 con 0x7f2fe415fa40 2024-03-06T07:36:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.016+0000 7f2fe2a00640 1 --2- 172.21.15.32:0/1664882252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe415fe20 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40d1280 tx=0x7f2fd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715d61e70e193e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.016+0000 7f2fe1600640 1 -- 172.21.15.32:0/1664882252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4034070 con 0x7f2fe415fa40 2024-03-06T07:36:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.016+0000 7f2fe1600640 1 -- 172.21.15.32:0/1664882252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd4030860 con 0x7f2fe415fa40 2024-03-06T07:36:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.016+0000 7f2fe1600640 1 -- 172.21.15.32:0/1664882252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4030c30 con 0x7f2fe415fa40 2024-03-06T07:36:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1664882252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 msgr2=0x7f2fe415fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1664882252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe415fe20 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40d1280 tx=0x7f2fd402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1664882252 shutdown_connections 2024-03-06T07:36:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1664882252 >> v2:172.21.15.39:3300/0 conn(0x7f2fe40cf340 0x7f2fe40cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1664882252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe415fe20 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1664882252 >> 172.21.15.32:0/1664882252 conn(0x7f2fe40b3820 msgr2=0x7f2fe40b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1664882252 shutdown_connections 2024-03-06T07:36:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.017+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1664882252 wait complete. 2024-03-06T07:36:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.019+0000 7f2fe9e00640 1 Processor -- start 2024-03-06T07:36:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.019+0000 7f2fe9e00640 1 -- start start 2024-03-06T07:36:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe40cf340 0x7f2fe40c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fe40cf8a0 0x7f2fe409f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe40d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe9e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40ccd90 con 0x7f2fe415fa40 2024-03-06T07:36:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe9e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40b4570 con 0x7f2fe40cf340 2024-03-06T07:36:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe9e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40ccf10 con 0x7f2fe40cf8a0 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe40d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe40d1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34536/0 (socket says 172.21.15.32:34536) 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.020+0000 7f2fe3400640 1 -- 172.21.15.32:0/1034708540 learned_addr learned my addr 172.21.15.32:0/1034708540 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe2a00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe40cf340 0x7f2fe40c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe2000640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fe40cf8a0 0x7f2fe409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe3400640 1 -- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fe40cf8a0 msgr2=0x7f2fe409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe3400640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fe40cf8a0 0x7f2fe409f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe3400640 1 -- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe40cf340 msgr2=0x7f2fe40c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe3400640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe40cf340 0x7f2fe40c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe3400640 1 -- 172.21.15.32:0/1034708540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe409f9a0 con 0x7f2fe415fa40 2024-03-06T07:36:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fe3400640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe40d1f40 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00d800 tx=0x7f2fdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.021+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc00b6c0 con 0x7f2fe415fa40 2024-03-06T07:36:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.022+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc019070 con 0x7f2fe415fa40 2024-03-06T07:36:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.022+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe40c9a70 con 0x7f2fe415fa40 2024-03-06T07:36:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.022+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc00bac0 con 0x7f2fe415fa40 2024-03-06T07:36:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.022+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe40ca030 con 0x7f2fe415fa40 2024-03-06T07:36:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.023+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe404f330 con 0x7f2fe415fa40 2024-03-06T07:36:55.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.024+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fdc00bc60 con 0x7f2fe415fa40 2024-03-06T07:36:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.025+0000 7f2fdb400640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc006b500 0x7f2fc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.025+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2fdc08a640 con 0x7f2fe415fa40 2024-03-06T07:36:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.025+0000 7f2fe2a00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc006b500 0x7f2fc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.026+0000 7f2fe2a00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc006b500 0x7f2fc006d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4036000 tx=0x7f2fd403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.030+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fdc059510 con 0x7f2fe415fa40 2024-03-06T07:36:55.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.272+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2fe40679a0 con 0x7f2fe415fa40 2024-03-06T07:36:55.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.802+0000 7f2fdb400640 1 -- 172.21.15.32:0/1034708540 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v87) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2fdc058e10 con 0x7f2fe415fa40 2024-03-06T07:36:55.809 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:36:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.817+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc006b500 msgr2=0x7f2fc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.817+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc006b500 0x7f2fc006d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4036000 tx=0x7f2fd403b040 comp rx=0 tx=0).stop 2024-03-06T07:36:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.817+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 msgr2=0x7f2fe40d1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.817+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe40d1f40 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00d800 tx=0x7f2fdc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.818+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 shutdown_connections 2024-03-06T07:36:55.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.818+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc006b500 0x7f2fc006d9c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.818+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe415fa40 0x7f2fe40d1f40 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.818+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fe40cf8a0 0x7f2fe409f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.818+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/1034708540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe40cf340 0x7f2fe40c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.818+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 >> 172.21.15.32:0/1034708540 conn(0x7f2fe40b3820 msgr2=0x7f2fe4070440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:55.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.819+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 shutdown_connections 2024-03-06T07:36:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:55.823+0000 7f2fe9e00640 1 -- 172.21.15.32:0/1034708540 wait complete. 2024-03-06T07:36:55.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:56.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.033+0000 7f67dde00640 1 Processor -- start 2024-03-06T07:36:56.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.033+0000 7f67dde00640 1 -- start start 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dde00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f67d8145300 0x7f67d80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dde00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67d8070f10 0x7f67d80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dde00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dde00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80cd300 con 0x7f67d8070f10 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dde00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80714d0 con 0x7f67d8145300 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dde00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80717b0 con 0x7f67d80cce90 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67d8070f10 0x7f67d80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67d8070f10 0x7f67d80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34552/0 (socket says 172.21.15.32:34552) 2024-03-06T07:36:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 -- 172.21.15.32:0/542358995 learned_addr learned my addr 172.21.15.32:0/542358995 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dd400640 1 --2- 172.21.15.32:0/542358995 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67dca00640 1 --2- 172.21.15.32:0/542358995 >> v2:172.21.15.39:3300/0 conn(0x7f67d8145300 0x7f67d80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 -- 172.21.15.32:0/542358995 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 msgr2=0x7f67d80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 --2- 172.21.15.32:0/542358995 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 -- 172.21.15.32:0/542358995 >> v2:172.21.15.39:3300/0 conn(0x7f67d8145300 msgr2=0x7f67d80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 --2- 172.21.15.32:0/542358995 >> v2:172.21.15.39:3300/0 conn(0x7f67d8145300 0x7f67d80b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.034+0000 7f67d7e00640 1 -- 172.21.15.32:0/542358995 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d8071a90 con 0x7f67d8070f10 2024-03-06T07:36:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.036+0000 7f67d7e00640 1 --2- 172.21.15.32:0/542358995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8070f10 0x7f67d80cc950 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f67d80cf990 tx=0x7f67c402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f657a9c4913a6f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.036+0000 7f67d7400640 1 -- 172.21.15.32:0/542358995 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c40317d0 con 0x7f67d8070f10 2024-03-06T07:36:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.036+0000 7f67d7400640 1 -- 172.21.15.32:0/542358995 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c4037070 con 0x7f67d8070f10 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.036+0000 7f67d7400640 1 -- 172.21.15.32:0/542358995 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c4031b90 con 0x7f67d8070f10 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 -- 172.21.15.32:0/542358995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8070f10 msgr2=0x7f67d80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 --2- 172.21.15.32:0/542358995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8070f10 0x7f67d80cc950 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f67d80cf990 tx=0x7f67c402b870 comp rx=0 tx=0).stop 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 -- 172.21.15.32:0/542358995 shutdown_connections 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 --2- 172.21.15.32:0/542358995 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 --2- 172.21.15.32:0/542358995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8070f10 0x7f67d80cc950 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 --2- 172.21.15.32:0/542358995 >> v2:172.21.15.39:3300/0 conn(0x7f67d8145300 0x7f67d80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 -- 172.21.15.32:0/542358995 >> 172.21.15.32:0/542358995 conn(0x7f67d806b3d0 msgr2=0x7f67d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 -- 172.21.15.32:0/542358995 shutdown_connections 2024-03-06T07:36:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.038+0000 7f67dde00640 1 -- 172.21.15.32:0/542358995 wait complete. 2024-03-06T07:36:56.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.040+0000 7f67dde00640 1 Processor -- start 2024-03-06T07:36:56.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.040+0000 7f67dde00640 1 -- start start 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67d8070f10 0x7f67d813ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 0x7f67d80adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d81456e0 con 0x7f67d8145300 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80b4ee0 con 0x7f67d8070f10 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80cde70 con 0x7f67d80cce90 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 0x7f67d80adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 0x7f67d80adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34554/0 (socket says 172.21.15.32:34554) 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 -- 172.21.15.32:0/729028246 learned_addr learned my addr 172.21.15.32:0/729028246 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dca00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67d8070f10 0x7f67d813ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67d7e00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 -- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 msgr2=0x7f67d80ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 -- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67d8070f10 msgr2=0x7f67d813ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67d8070f10 0x7f67d813ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 -- 172.21.15.32:0/729028246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80acbf0 con 0x7f67d8145300 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dca00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67d8070f10 0x7f67d813ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dd400640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 0x7f67d80adc90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f67d0007e70 tx=0x7f67d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0014620 con 0x7f67d8145300 2024-03-06T07:36:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d0019070 con 0x7f67d8145300 2024-03-06T07:36:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0014a20 con 0x7f67d8145300 2024-03-06T07:36:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d80aced0 con 0x7f67d8145300 2024-03-06T07:36:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.041+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d80ad490 con 0x7f67d8145300 2024-03-06T07:36:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.043+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d80687b0 con 0x7f67d8145300 2024-03-06T07:36:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.044+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67d0014bc0 con 0x7f67d8145300 2024-03-06T07:36:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.044+0000 7f67d4c00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67b806b4d0 0x7f67b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.045+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67d008aac0 con 0x7f67d8145300 2024-03-06T07:36:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.045+0000 7f67dca00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67b806b4d0 0x7f67b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.045+0000 7f67dca00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67b806b4d0 0x7f67b806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f67cc006fd0 tx=0x7f67cc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.051+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67d80687b0 con 0x7f67d8145300 2024-03-06T07:36:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.250+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67d806ce90 con 0x7f67d8145300 2024-03-06T07:36:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.253+0000 7f67d4c00640 1 -- 172.21.15.32:0/729028246 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f67d0059930 con 0x7f67d8145300 2024-03-06T07:36:56.255 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:56.256 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":126,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:36:55.274529+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:36:53.245687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:36:54.277198+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67b806b4d0 msgr2=0x7f67b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67b806b4d0 0x7f67b806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f67cc006fd0 tx=0x7f67cc008040 comp rx=0 tx=0).stop 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 msgr2=0x7f67d80adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 0x7f67d80adc90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f67d0007e70 tx=0x7f67d000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 shutdown_connections 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67b806b4d0 0x7f67b806d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8145300 0x7f67d80adc90 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67d80cce90 0x7f67d80ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 --2- 172.21.15.32:0/729028246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67d8070f10 0x7f67d813ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.256+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 >> 172.21.15.32:0/729028246 conn(0x7f67d806b3d0 msgr2=0x7f67d8072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.257+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 shutdown_connections 2024-03-06T07:36:56.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.257+0000 7f67dde00640 1 -- 172.21.15.32:0/729028246 wait complete. 2024-03-06T07:36:56.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:56.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.502+0000 7f7ad2000640 1 Processor -- start 2024-03-06T07:36:56.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.503+0000 7f7ad2000640 1 -- start start 2024-03-06T07:36:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.504+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.504+0000 7f7ad2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 0x7f7acc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.504+0000 7f7ad2000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc15ed10 con 0x7f7acc0cce90 2024-03-06T07:36:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.504+0000 7f7ad2000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc15eff0 con 0x7f7acc15e810 2024-03-06T07:36:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.504+0000 7f7ad2000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc15f2d0 con 0x7f7acc1552b0 2024-03-06T07:36:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.505+0000 7f7ad0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.505+0000 7f7acbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 0x7f7acc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.505+0000 7f7ad0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:39374/0 (socket says 172.21.15.32:39374) 2024-03-06T07:36:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.505+0000 7f7acbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 0x7f7acc0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:43132/0 (socket says 172.21.15.32:43132) 2024-03-06T07:36:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.505+0000 7f7ad0c00640 1 -- 172.21.15.32:0/3736699318 learned_addr learned my addr 172.21.15.32:0/3736699318 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.506+0000 7f7ad0c00640 1 -- 172.21.15.32:0/3736699318 >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 msgr2=0x7f7acc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.506+0000 7f7ad0c00640 1 --2- 172.21.15.32:0/3736699318 >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 0x7f7acc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.506+0000 7f7ad0c00640 1 -- 172.21.15.32:0/3736699318 >> v1:172.21.15.39:6789/0 conn(0x7f7acc0cce90 legacy=0x7f7acc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.506+0000 7f7ad0c00640 1 -- 172.21.15.32:0/3736699318 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc15f5b0 con 0x7f7acc1552b0 2024-03-06T07:36:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.506+0000 7f7acbe00640 1 --2- 172.21.15.32:0/3736699318 >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 0x7f7acc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:56.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.507+0000 7f7ad0c00640 1 --2- 172.21.15.32:0/3736699318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc15b1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009870 tx=0x7f7abc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=940112ead5001c4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.508+0000 7f7acb400640 1 -- 172.21.15.32:0/3736699318 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc035020 con 0x7f7acc1552b0 2024-03-06T07:36:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.508+0000 7f7acb400640 1 -- 172.21.15.32:0/3736699318 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc031640 con 0x7f7acc1552b0 2024-03-06T07:36:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.508+0000 7f7acb400640 1 -- 172.21.15.32:0/3736699318 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc02d040 con 0x7f7acc1552b0 2024-03-06T07:36:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 -- 172.21.15.32:0/3736699318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 msgr2=0x7f7acc15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 --2- 172.21.15.32:0/3736699318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc15b1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009870 tx=0x7f7abc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 -- 172.21.15.32:0/3736699318 shutdown_connections 2024-03-06T07:36:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 --2- 172.21.15.32:0/3736699318 >> v2:172.21.15.39:3300/0 conn(0x7f7acc15e810 0x7f7acc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 --2- 172.21.15.32:0/3736699318 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc15b1d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 -- 172.21.15.32:0/3736699318 >> 172.21.15.32:0/3736699318 conn(0x7f7acc06b3d0 msgr2=0x7f7acc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 -- 172.21.15.32:0/3736699318 shutdown_connections 2024-03-06T07:36:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.509+0000 7f7ad2000640 1 -- 172.21.15.32:0/3736699318 wait complete. 2024-03-06T07:36:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.511+0000 7f7ad2000640 1 Processor -- start 2024-03-06T07:36:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.511+0000 7f7ad2000640 1 -- start start 2024-03-06T07:36:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.512+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 0x7f7acc145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.512+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc0b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.512+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7acc15e810 0x7f7acc0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.512+0000 7f7ad2000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc06dd30 con 0x7f7acc0cce90 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.512+0000 7f7ad2000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b3760 con 0x7f7acc15e810 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.512+0000 7f7ad2000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc06deb0 con 0x7f7acc1552b0 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 0x7f7acc145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 0x7f7acc145d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34570/0 (socket says 172.21.15.32:34570) 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 -- 172.21.15.32:0/1743096124 learned_addr learned my addr 172.21.15.32:0/1743096124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad1600640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7acc15e810 0x7f7acc0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7acbe00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc0b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 -- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 msgr2=0x7f7acc0b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc0b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 -- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7acc15e810 msgr2=0x7f7acc0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7acc15e810 0x7f7acc0cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 -- 172.21.15.32:0/1743096124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc146a10 con 0x7f7acc0cce90 2024-03-06T07:36:56.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.513+0000 7f7ad0c00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 0x7f7acc145d00 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f7abc02b930 tx=0x7f7abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.514+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc040070 con 0x7f7acc0cce90 2024-03-06T07:36:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.514+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc03e070 con 0x7f7acc0cce90 2024-03-06T07:36:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.514+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc146cf0 con 0x7f7acc0cce90 2024-03-06T07:36:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.514+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc035040 con 0x7f7acc0cce90 2024-03-06T07:36:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.514+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc147590 con 0x7f7acc0cce90 2024-03-06T07:36:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.515+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc1470e0 con 0x7f7acc0cce90 2024-03-06T07:36:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.516+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7abc038020 con 0x7f7acc0cce90 2024-03-06T07:36:56.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.517+0000 7f7ac8c00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7aac06b4d0 0x7f7aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.517+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7abc0abbc0 con 0x7f7acc0cce90 2024-03-06T07:36:56.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.517+0000 7f7acbe00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7aac06b4d0 0x7f7aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.518+0000 7f7acbe00640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7aac06b4d0 0x7f7aac06d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009bd0 tx=0x7f7ab800b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.522+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7abc07aa30 con 0x7f7acc0cce90 2024-03-06T07:36:56.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.724+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7acc04f330 con 0x7f7acc0cce90 2024-03-06T07:36:56.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.727+0000 7f7ac8c00640 1 -- 172.21.15.32:0/1743096124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f7abc043e40 con 0x7f7acc0cce90 2024-03-06T07:36:56.729 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:56.729 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":126,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:36:55.274529+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:36:53.245687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:36:54.277198+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7aac06b4d0 msgr2=0x7f7aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7aac06b4d0 0x7f7aac06d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009bd0 tx=0x7f7ab800b040 comp rx=0 tx=0).stop 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 msgr2=0x7f7acc145d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 0x7f7acc145d00 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f7abc02b930 tx=0x7f7abc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 shutdown_connections 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7aac06b4d0 0x7f7aac06d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7acc15e810 0x7f7acc0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7acc1552b0 0x7f7acc0b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 --2- 172.21.15.32:0/1743096124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc0cce90 0x7f7acc145d00 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 >> 172.21.15.32:0/1743096124 conn(0x7f7acc06b3d0 msgr2=0x7f7acc071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 shutdown_connections 2024-03-06T07:36:56.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.730+0000 7f7ad2000640 1 -- 172.21.15.32:0/1743096124 wait complete. 2024-03-06T07:36:56.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:36:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.977+0000 7fb08e800640 1 Processor -- start 2024-03-06T07:36:56.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.977+0000 7fb08e800640 1 -- start start 2024-03-06T07:36:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.978+0000 7fb08e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb08815bf20 0x7fb0880acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb08e800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb088071e20 0x7fb0880ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb08e800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088072df0 con 0x7fb088072520 2024-03-06T07:36:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb08e800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0880cdab0 con 0x7fb088071e20 2024-03-06T07:36:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb08e800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0880b4d70 con 0x7fb08815bf20 2024-03-06T07:36:56.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb087e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb088071e20 0x7fb0880ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb087e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb088071e20 0x7fb0880ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34582/0 (socket says 172.21.15.32:34582) 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb087e00640 1 -- 172.21.15.32:0/3467985221 learned_addr learned my addr 172.21.15.32:0/3467985221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb086a00640 1 --1- 172.21.15.32:0/3467985221 >> v1:172.21.15.32:6789/0 conn(0x7fb088072520 0x7fb0880cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39886/0 (socket says 172.21.15.32:39886) 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb087400640 1 --2- 172.21.15.32:0/3467985221 >> v2:172.21.15.39:3300/0 conn(0x7fb08815bf20 0x7fb0880acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb087e00640 1 -- 172.21.15.32:0/3467985221 >> v2:172.21.15.39:3300/0 conn(0x7fb08815bf20 msgr2=0x7fb0880acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.979+0000 7fb087e00640 1 --2- 172.21.15.32:0/3467985221 >> v2:172.21.15.39:3300/0 conn(0x7fb08815bf20 0x7fb0880acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.980+0000 7fb087e00640 1 -- 172.21.15.32:0/3467985221 >> v1:172.21.15.32:6789/0 conn(0x7fb088072520 legacy=0x7fb0880cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.980+0000 7fb087e00640 1 -- 172.21.15.32:0/3467985221 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0880b5050 con 0x7fb088071e20 2024-03-06T07:36:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.980+0000 7fb087400640 1 --2- 172.21.15.32:0/3467985221 >> v2:172.21.15.39:3300/0 conn(0x7fb08815bf20 0x7fb0880acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.981+0000 7fb087e00640 1 --2- 172.21.15.32:0/3467985221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb088071e20 0x7fb0880ccad0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fb08806ec40 tx=0x7fb08002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9db960e7759c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.981+0000 7fb086000640 1 -- 172.21.15.32:0/3467985221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080034070 con 0x7fb088071e20 2024-03-06T07:36:56.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.981+0000 7fb086000640 1 -- 172.21.15.32:0/3467985221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080030880 con 0x7fb088071e20 2024-03-06T07:36:56.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.981+0000 7fb086000640 1 -- 172.21.15.32:0/3467985221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080030c50 con 0x7fb088071e20 2024-03-06T07:36:56.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 -- 172.21.15.32:0/3467985221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb088071e20 msgr2=0x7fb0880ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 --2- 172.21.15.32:0/3467985221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb088071e20 0x7fb0880ccad0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fb08806ec40 tx=0x7fb08002b9c0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 -- 172.21.15.32:0/3467985221 shutdown_connections 2024-03-06T07:36:56.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 --2- 172.21.15.32:0/3467985221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb088071e20 0x7fb0880ccad0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 --2- 172.21.15.32:0/3467985221 >> v2:172.21.15.39:3300/0 conn(0x7fb08815bf20 0x7fb0880acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 -- 172.21.15.32:0/3467985221 >> 172.21.15.32:0/3467985221 conn(0x7fb08806b3d0 msgr2=0x7fb08806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 -- 172.21.15.32:0/3467985221 shutdown_connections 2024-03-06T07:36:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.982+0000 7fb08e800640 1 -- 172.21.15.32:0/3467985221 wait complete. 2024-03-06T07:36:56.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.984+0000 7fb08e800640 1 Processor -- start 2024-03-06T07:36:56.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.984+0000 7fb08e800640 1 -- start start 2024-03-06T07:36:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb08e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb088071e20 0x7fb08809d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb08e800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb088072520 0x7fb08814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb08e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 0x7fb088156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb08e800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880b4b10 con 0x7fb08815bf20 2024-03-06T07:36:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb08e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880b4990 con 0x7fb088071e20 2024-03-06T07:36:56.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb08e800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880ce6c0 con 0x7fb088072520 2024-03-06T07:36:56.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 0x7fb088156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 0x7fb088156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34592/0 (socket says 172.21.15.32:34592) 2024-03-06T07:36:56.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb087e00640 1 -- 172.21.15.32:0/2130478289 learned_addr learned my addr 172.21.15.32:0/2130478289 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.985+0000 7fb087400640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb088071e20 0x7fb08809d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087e00640 1 -- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb088072520 msgr2=0x7fb08814e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087e00640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb088072520 0x7fb08814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087e00640 1 -- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb088071e20 msgr2=0x7fb08809d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087e00640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb088071e20 0x7fb08809d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087e00640 1 -- 172.21.15.32:0/2130478289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08809aa10 con 0x7fb08815bf20 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087400640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb088071e20 0x7fb08809d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb087e00640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 0x7fb088156fa0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fb08002ba00 tx=0x7fb0800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080034040 con 0x7fb08815bf20 2024-03-06T07:36:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0800433f0 con 0x7fb08815bf20 2024-03-06T07:36:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0800437e0 con 0x7fb08815bf20 2024-03-06T07:36:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.986+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880b2690 con 0x7fb08815bf20 2024-03-06T07:36:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.987+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08809af90 con 0x7fb08815bf20 2024-03-06T07:36:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.988+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08804f330 con 0x7fb08815bf20 2024-03-06T07:36:56.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.989+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb080037070 con 0x7fb08815bf20 2024-03-06T07:36:56.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.989+0000 7fb07f400640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.990+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0800ac530 con 0x7fb08815bf20 2024-03-06T07:36:56.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.990+0000 7fb087400640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.991+0000 7fb087400640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb0880b0f90 tx=0x7fb078007550 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:56.994+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb08003e070 con 0x7fb08815bf20 2024-03-06T07:36:57.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.304+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb08815f720 con 0x7fb08815bf20 2024-03-06T07:36:57.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.821+0000 7fb07f400640 1 -- 172.21.15.32:0/2130478289 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v90) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb08007b3a0 con 0x7fb08815bf20 2024-03-06T07:36:57.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 msgr2=0x7fb06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb0880b0f90 tx=0x7fb078007550 comp rx=0 tx=0).stop 2024-03-06T07:36:57.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 msgr2=0x7fb088156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 0x7fb088156fa0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fb08002ba00 tx=0x7fb0800046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 shutdown_connections 2024-03-06T07:36:57.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb06406b4d0 0x7fb06406d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb08815bf20 0x7fb088156fa0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb088072520 0x7fb08814e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 --2- 172.21.15.32:0/2130478289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb088071e20 0x7fb08809d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 >> 172.21.15.32:0/2130478289 conn(0x7fb08806b3d0 msgr2=0x7fb0880ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:57.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 shutdown_connections 2024-03-06T07:36:57.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:57.826+0000 7fb08e800640 1 -- 172.21.15.32:0/2130478289 wait complete. 2024-03-06T07:36:57.852 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:36:57.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:36:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.074+0000 7f46d1000640 1 Processor -- start 2024-03-06T07:36:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.074+0000 7f46d1000640 1 -- start start 2024-03-06T07:36:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.076+0000 7f46d1000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 0x7f46cc0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.076+0000 7f46d1000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc166cb0 con 0x7f46cc162280 2024-03-06T07:36:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.076+0000 7f46d1000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc166410 con 0x7f46cc0b1b40 2024-03-06T07:36:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.076+0000 7f46d1000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0b4fd0 con 0x7f46cc165d50 2024-03-06T07:36:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.077+0000 7f46cb400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f46cc162280 0x7f46cc0cc430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39778/0 (socket says 172.21.15.32:39778) 2024-03-06T07:36:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.077+0000 7f46cb400640 1 -- 172.21.15.32:0/553002419 learned_addr learned my addr 172.21.15.32:0/553002419 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.077+0000 7f46c3e00640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 0x7f46cc0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.077+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907261135 0 0) 0x7f46cc166cb0 con 0x7f46cc162280 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.077+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b0003520 con 0x7f46cc162280 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523749529 0 0) 0x7f46b0003520 con 0x7f46cc162280 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 msgr2=0x7f46cc0b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 0x7f46cc0b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 >> v1:172.21.15.39:6789/0 conn(0x7f46cc0b1b40 legacy=0x7f46cc071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc1666f0 con 0x7f46cc162280 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f46bc001670 con 0x7f46cc162280 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 >> v1:172.21.15.32:6789/0 conn(0x7f46cc162280 legacy=0x7f46cc0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46b00039e0 0x7f46b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 0x7f46b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b0004d40 0x7f46b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0b4fd0 con 0x7f46b00039e0 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b000baf0 con 0x7f46b0004d40 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b000bcf0 con 0x7f46cc165d50 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46cb400640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46b00039e0 0x7f46b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46cb400640 1 -- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 msgr2=0x7f46b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46cb400640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc165d50 0x7f46b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46cb400640 1 -- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b0004d40 msgr2=0x7f46b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46cb400640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b0004d40 0x7f46b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.078+0000 7f46cb400640 1 -- 172.21.15.32:0/553002419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0003520 con 0x7f46b00039e0 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.079+0000 7f46cb400640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46b00039e0 0x7f46b0003e00 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c240 tx=0x7f46b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8019ed88cc0dd1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc007cf0 con 0x7f46b00039e0 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc030070 con 0x7f46b00039e0 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46caa00640 1 -- 172.21.15.32:0/553002419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc005d30 con 0x7f46b00039e0 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46d1000640 1 -- 172.21.15.32:0/553002419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46b00039e0 msgr2=0x7f46b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46d1000640 1 --2- 172.21.15.32:0/553002419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46b00039e0 0x7f46b0003e00 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c240 tx=0x7f46b0005830 comp rx=0 tx=0).stop 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46cbe00640 1 -- 172.21.15.32:0/553002419 reap_dead start 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46d1000640 1 -- 172.21.15.32:0/553002419 shutdown_connections 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.080+0000 7f46d1000640 1 -- 172.21.15.32:0/553002419 >> 172.21.15.32:0/553002419 conn(0x7f46cc06b3a0 msgr2=0x7f46cc06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.081+0000 7f46d1000640 1 -- 172.21.15.32:0/553002419 shutdown_connections 2024-03-06T07:36:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.081+0000 7f46d1000640 1 -- 172.21.15.32:0/553002419 wait complete. 2024-03-06T07:36:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.082+0000 7f46d1000640 1 Processor -- start 2024-03-06T07:36:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.082+0000 7f46d1000640 1 -- start start 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.083+0000 7f46d1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b00039e0 0x7f46cc152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.083+0000 7f46d1000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc1ecfc0 0x7f46cc15fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.083+0000 7f46d1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 0x7f46cc160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.083+0000 7f46d1000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0b4a90 con 0x7f46cc1ed440 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.083+0000 7f46d1000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc1661b0 con 0x7f46b00039e0 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.083+0000 7f46d1000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0b4c10 con 0x7f46cc1ecfc0 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 0x7f46cc160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 0x7f46cc160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33296/0 (socket says 172.21.15.32:33296) 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 -- 172.21.15.32:0/392302031 learned_addr learned my addr 172.21.15.32:0/392302031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cb400640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b00039e0 0x7f46cc152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46c3e00640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc1ecfc0 0x7f46cc15fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 -- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc1ecfc0 msgr2=0x7f46cc15fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc1ecfc0 0x7f46cc15fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 -- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b00039e0 msgr2=0x7f46cc152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b00039e0 0x7f46cc152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46cbe00640 1 -- 172.21.15.32:0/392302031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc1ed9c0 con 0x7f46cc1ed440 2024-03-06T07:36:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.084+0000 7f46c3e00640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc1ecfc0 0x7f46cc15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.085+0000 7f46cbe00640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 0x7f46cc160360 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f46cc06e1c0 tx=0x7f46c400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.085+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4017070 con 0x7f46cc1ed440 2024-03-06T07:36:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.085+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4005d60 con 0x7f46cc1ed440 2024-03-06T07:36:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.085+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400a7f0 con 0x7f46cc1ed440 2024-03-06T07:36:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.085+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc15e360 con 0x7f46cc1ed440 2024-03-06T07:36:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.085+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc15e920 con 0x7f46cc1ed440 2024-03-06T07:36:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.086+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc04f330 con 0x7f46cc1ed440 2024-03-06T07:36:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.087+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46c400a990 con 0x7f46cc1ed440 2024-03-06T07:36:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.089+0000 7f46c3400640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46a806b4d0 0x7f46a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.089+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46c4089310 con 0x7f46cc1ed440 2024-03-06T07:36:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.089+0000 7f46cb400640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46a806b4d0 0x7f46a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.090+0000 7f46cb400640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46a806b4d0 0x7f46a806d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f46bc034000 tx=0x7f46bc007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.094+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46c40580f0 con 0x7f46cc1ed440 2024-03-06T07:36:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.470+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f46cc06d620 con 0x7f46cc1ed440 2024-03-06T07:36:58.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.834+0000 7f46c3400640 1 -- 172.21.15.32:0/392302031 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v91) v1 ==== 122+0+0 (secure 0 0 0) 0x7f46c40579f0 con 0x7f46cc1ed440 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.837+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46a806b4d0 msgr2=0x7f46a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.837+0000 7f46d1000640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46a806b4d0 0x7f46a806d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f46bc034000 tx=0x7f46bc007000 comp rx=0 tx=0).stop 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.837+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 msgr2=0x7f46cc160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.837+0000 7f46d1000640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 0x7f46cc160360 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f46cc06e1c0 tx=0x7f46c400cb90 comp rx=0 tx=0).stop 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 shutdown_connections 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f46a806b4d0 0x7f46a806d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46cc1ed440 0x7f46cc160360 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f46cc1ecfc0 0x7f46cc15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 --2- 172.21.15.32:0/392302031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46b00039e0 0x7f46cc152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 >> 172.21.15.32:0/392302031 conn(0x7f46cc06b3a0 msgr2=0x7f46cc1615a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:58.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 shutdown_connections 2024-03-06T07:36:58.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:58.838+0000 7f46d1000640 1 -- 172.21.15.32:0/392302031 wait complete. 2024-03-06T07:36:58.859 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:36:58.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.064+0000 7f5ceca00640 1 Processor -- start 2024-03-06T07:36:59.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.064+0000 7f5ceca00640 1 -- start start 2024-03-06T07:36:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5ceca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ce80d4a10 0x7f5ce80d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5ceca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5ce80d0c80 0x7f5ce80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5ceca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80b3710 con 0x7f5ce815e0b0 2024-03-06T07:36:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5ceca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce80d10f0 con 0x7f5ce80d4a10 2024-03-06T07:36:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5ceca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce80b09e0 con 0x7f5ce80d0c80 2024-03-06T07:36:59.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5cdfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ce80d4a10 0x7f5ce80d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5cdfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ce80d4a10 0x7f5ce80d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33302/0 (socket says 172.21.15.32:33302) 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5cdfe00640 1 -- 172.21.15.32:0/1045857315 learned_addr learned my addr 172.21.15.32:0/1045857315 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.066+0000 7f5ce6800640 1 --1- 172.21.15.32:0/1045857315 >> v1:172.21.15.32:6789/0 conn(0x7f5ce815e0b0 0x7f5ce80cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39790/0 (socket says 172.21.15.32:39790) 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.067+0000 7f5ce7200640 1 --2- 172.21.15.32:0/1045857315 >> v2:172.21.15.39:3300/0 conn(0x7f5ce80d0c80 0x7f5ce80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.067+0000 7f5cdfe00640 1 -- 172.21.15.32:0/1045857315 >> v2:172.21.15.39:3300/0 conn(0x7f5ce80d0c80 msgr2=0x7f5ce80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.067+0000 7f5cdfe00640 1 --2- 172.21.15.32:0/1045857315 >> v2:172.21.15.39:3300/0 conn(0x7f5ce80d0c80 0x7f5ce80b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.067+0000 7f5cdfe00640 1 -- 172.21.15.32:0/1045857315 >> v1:172.21.15.32:6789/0 conn(0x7f5ce815e0b0 legacy=0x7f5ce80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.067+0000 7f5cdfe00640 1 -- 172.21.15.32:0/1045857315 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80b39f0 con 0x7f5ce80d4a10 2024-03-06T07:36:59.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.067+0000 7f5ce5e00640 1 -- 172.21.15.32:0/1045857315 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664952577 0 0) 0x7f5ce80b3710 con 0x7f5ce815e0b0 2024-03-06T07:36:59.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.068+0000 7f5cdfe00640 1 --2- 172.21.15.32:0/1045857315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80d0740 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f5ce808d6d0 tx=0x7f5cd002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc48717dbf07bade server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.068+0000 7f5ce5e00640 1 -- 172.21.15.32:0/1045857315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0034070 con 0x7f5ce80d4a10 2024-03-06T07:36:59.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.068+0000 7f5ce5e00640 1 -- 172.21.15.32:0/1045857315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd0030860 con 0x7f5ce80d4a10 2024-03-06T07:36:59.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.068+0000 7f5ce5e00640 1 -- 172.21.15.32:0/1045857315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0030c30 con 0x7f5ce80d4a10 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 -- 172.21.15.32:0/1045857315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce80d4a10 msgr2=0x7f5ce80d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1045857315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80d0740 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f5ce808d6d0 tx=0x7f5cd002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 -- 172.21.15.32:0/1045857315 shutdown_connections 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1045857315 >> v2:172.21.15.39:3300/0 conn(0x7f5ce80d0c80 0x7f5ce80b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1045857315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80d0740 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 -- 172.21.15.32:0/1045857315 >> 172.21.15.32:0/1045857315 conn(0x7f5ce80d3d30 msgr2=0x7f5ce80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 -- 172.21.15.32:0/1045857315 shutdown_connections 2024-03-06T07:36:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.069+0000 7f5ceca00640 1 -- 172.21.15.32:0/1045857315 wait complete. 2024-03-06T07:36:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.071+0000 7f5ceca00640 1 Processor -- start 2024-03-06T07:36:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.071+0000 7f5ceca00640 1 -- start start 2024-03-06T07:36:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ceca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ce80d0c80 0x7f5ce814f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ceca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ceca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 0x7f5ce80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ceca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80b3fb0 con 0x7f5ce815e0b0 2024-03-06T07:36:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ceca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80cfa20 con 0x7f5ce80d4a10 2024-03-06T07:36:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ceca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815e490 con 0x7f5ce80d0c80 2024-03-06T07:36:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ce7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 0x7f5ce80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ce7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 0x7f5ce80cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33316/0 (socket says 172.21.15.32:33316) 2024-03-06T07:36:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.072+0000 7f5ce7200640 1 -- 172.21.15.32:0/1807230192 learned_addr learned my addr 172.21.15.32:0/1807230192 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce6800640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ce80d0c80 0x7f5ce814f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce7200640 1 -- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ce80d0c80 msgr2=0x7f5ce814f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce7200640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ce80d0c80 0x7f5ce814f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5cdfe00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce7200640 1 -- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ce80d4a10 msgr2=0x7f5ce80cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce7200640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce7200640 1 -- 172.21.15.32:0/1807230192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8160430 con 0x7f5ce815e0b0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce6800640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ce80d0c80 0x7f5ce814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ce7200640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 0x7f5ce80cae00 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0007e90 tx=0x7f5ce000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0014420 con 0x7f5ce815e0b0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce0019070 con 0x7f5ce815e0b0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0014800 con 0x7f5ce815e0b0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce8160710 con 0x7f5ce815e0b0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.073+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce81611f0 con 0x7f5ce815e0b0 2024-03-06T07:36:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.075+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce804f330 con 0x7f5ce815e0b0 2024-03-06T07:36:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.076+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ce00149a0 con 0x7f5ce815e0b0 2024-03-06T07:36:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.077+0000 7f5cdea00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cc406b4d0 0x7f5cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.077+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ce008aac0 con 0x7f5ce815e0b0 2024-03-06T07:36:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.077+0000 7f5ce6800640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cc406b4d0 0x7f5cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.078+0000 7f5ce6800640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cc406b4d0 0x7f5cc406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8007fd0 tx=0x7f5cd8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.082+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ce0059930 con 0x7f5ce815e0b0 2024-03-06T07:36:59.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.323+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce80b29b0 con 0x7f5ce815e0b0 2024-03-06T07:36:59.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.324+0000 7f5cdea00640 1 -- 172.21.15.32:0/1807230192 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f5ce0059230 con 0x7f5ce815e0b0 2024-03-06T07:36:59.326 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:59.326 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/4261872692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4261872692},{"type":"v1","addr":"172.21.15.32:6837","nonce":4261872692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":5894,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/304553199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":304553199},{"type":"v1","addr":"172.21.15.39:6835","nonce":304553199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6574,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:36:58.471868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8313},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:36:59.326 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cc406b4d0 msgr2=0x7f5cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cc406b4d0 0x7f5cc406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8007fd0 tx=0x7f5cd8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 msgr2=0x7f5ce80cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 0x7f5ce80cae00 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0007e90 tx=0x7f5ce000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 shutdown_connections 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5cc406b4d0 0x7f5cc406d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ce815e0b0 0x7f5ce80cae00 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ce80d4a10 0x7f5ce80cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 --2- 172.21.15.32:0/1807230192 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ce80d0c80 0x7f5ce814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.327+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 >> 172.21.15.32:0/1807230192 conn(0x7f5ce80d3d30 msgr2=0x7f5ce80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.328+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 shutdown_connections 2024-03-06T07:36:59.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.328+0000 7f5ceca00640 1 -- 172.21.15.32:0/1807230192 wait complete. 2024-03-06T07:36:59.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.558+0000 7f187ca00640 1 Processor -- start 2024-03-06T07:36:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.558+0000 7f187ca00640 1 -- start start 2024-03-06T07:36:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f187ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f187815e0b0 0x7f18780cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f187ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d4a10 0x7f18780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f187ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18780d0c80 0x7f18780d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f187ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18780b09e0 con 0x7f18780d0c80 2024-03-06T07:36:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f187ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18780b3710 con 0x7f187815e0b0 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f187ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18780b39f0 con 0x7f18780d4a10 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f1877e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18780d0c80 0x7f18780d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f1877e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18780d0c80 0x7f18780d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33326/0 (socket says 172.21.15.32:33326) 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.560+0000 7f1877e00640 1 -- 172.21.15.32:0/4079691627 learned_addr learned my addr 172.21.15.32:0/4079691627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1876a00640 1 --2- 172.21.15.32:0/4079691627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d4a10 0x7f18780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877400640 1 --2- 172.21.15.32:0/4079691627 >> v2:172.21.15.39:3300/0 conn(0x7f187815e0b0 0x7f18780cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877e00640 1 -- 172.21.15.32:0/4079691627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d4a10 msgr2=0x7f18780d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877e00640 1 --2- 172.21.15.32:0/4079691627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d4a10 0x7f18780d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877e00640 1 -- 172.21.15.32:0/4079691627 >> v2:172.21.15.39:3300/0 conn(0x7f187815e0b0 msgr2=0x7f18780cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877e00640 1 --2- 172.21.15.32:0/4079691627 >> v2:172.21.15.39:3300/0 conn(0x7f187815e0b0 0x7f18780cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877e00640 1 -- 172.21.15.32:0/4079691627 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780b3cd0 con 0x7f18780d0c80 2024-03-06T07:36:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.561+0000 7f1877400640 1 --2- 172.21.15.32:0/4079691627 >> v2:172.21.15.39:3300/0 conn(0x7f187815e0b0 0x7f18780cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:59.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.562+0000 7f1877e00640 1 --2- 172.21.15.32:0/4079691627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d0c80 0x7f18780d1900 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f18780b68c0 tx=0x7f187002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3290ab70a58cd72f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.562+0000 7f1876000640 1 -- 172.21.15.32:0/4079691627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187002ca70 con 0x7f18780d0c80 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.562+0000 7f1876000640 1 -- 172.21.15.32:0/4079691627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187002cc10 con 0x7f18780d0c80 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.562+0000 7f1876000640 1 -- 172.21.15.32:0/4079691627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18700343f0 con 0x7f18780d0c80 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 -- 172.21.15.32:0/4079691627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d0c80 msgr2=0x7f18780d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 --2- 172.21.15.32:0/4079691627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d0c80 0x7f18780d1900 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f18780b68c0 tx=0x7f187002bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 -- 172.21.15.32:0/4079691627 shutdown_connections 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 --2- 172.21.15.32:0/4079691627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d0c80 0x7f18780d1900 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 --2- 172.21.15.32:0/4079691627 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d4a10 0x7f18780d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 --2- 172.21.15.32:0/4079691627 >> v2:172.21.15.39:3300/0 conn(0x7f187815e0b0 0x7f18780cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 -- 172.21.15.32:0/4079691627 >> 172.21.15.32:0/4079691627 conn(0x7f18780d3d30 msgr2=0x7f18780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.563+0000 7f187ca00640 1 -- 172.21.15.32:0/4079691627 shutdown_connections 2024-03-06T07:36:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.564+0000 7f187ca00640 1 -- 172.21.15.32:0/4079691627 wait complete. 2024-03-06T07:36:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.565+0000 7f187ca00640 1 Processor -- start 2024-03-06T07:36:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.565+0000 7f187ca00640 1 -- start start 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.566+0000 7f187ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d0c80 0x7f18780cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.566+0000 7f187ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 0x7f187814def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.566+0000 7f187ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f187815e0b0 0x7f187814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.566+0000 7f187ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18780cfc60 con 0x7f18780d4a10 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.566+0000 7f187ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187815e490 con 0x7f187815e0b0 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.566+0000 7f187ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18780cf850 con 0x7f18780d0c80 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 0x7f187814def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 0x7f187814def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33340/0 (socket says 172.21.15.32:33340) 2024-03-06T07:36:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 -- 172.21.15.32:0/2118443366 learned_addr learned my addr 172.21.15.32:0/2118443366 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1877400640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d0c80 0x7f18780cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 -- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d0c80 msgr2=0x7f18780cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d0c80 0x7f18780cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 -- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f187815e0b0 msgr2=0x7f187814f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1877e00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f187815e0b0 0x7f187814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f187815e0b0 0x7f187814f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.567+0000 7f1876a00640 1 -- 172.21.15.32:0/2118443366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187814e430 con 0x7f18780d4a10 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.568+0000 7f1876a00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 0x7f187814def0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f186400d7d0 tx=0x7f186400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.568+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186400b6c0 con 0x7f18780d4a10 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.568+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864019070 con 0x7f18780d4a10 2024-03-06T07:36:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.568+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187814e6d0 con 0x7f18780d4a10 2024-03-06T07:36:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.568+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186400ba80 con 0x7f18780d4a10 2024-03-06T07:36:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.568+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187814ef70 con 0x7f18780d4a10 2024-03-06T07:36:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.569+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187804f330 con 0x7f18780d4a10 2024-03-06T07:36:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.570+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f186400bc20 con 0x7f18780d4a10 2024-03-06T07:36:59.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.571+0000 7f186f400640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f185806b4d0 0x7f185806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.572+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186408a700 con 0x7f18780d4a10 2024-03-06T07:36:59.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.572+0000 7f1877400640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f185806b4d0 0x7f185806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.572+0000 7f1877400640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f185806b4d0 0x7f185806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f18781606f0 tx=0x7f1868008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.580+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f186401c360 con 0x7f18780d4a10 2024-03-06T07:36:59.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.783+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18780b1280 con 0x7f18780d4a10 2024-03-06T07:36:59.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.786+0000 7f186f400640 1 -- 172.21.15.32:0/2118443366 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16571 (secure 0 0 0) 0x7f18640594e0 con 0x7f18780d4a10 2024-03-06T07:36:59.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:36:59.788 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":127,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:36:56.797057+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:36:53.245687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:36:54.277198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.789+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f185806b4d0 msgr2=0x7f185806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.789+0000 7f187ca00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f185806b4d0 0x7f185806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f18781606f0 tx=0x7f1868008040 comp rx=0 tx=0).stop 2024-03-06T07:36:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.789+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 msgr2=0x7f187814def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.789+0000 7f187ca00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 0x7f187814def0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f186400d7d0 tx=0x7f186400dc90 comp rx=0 tx=0).stop 2024-03-06T07:36:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 shutdown_connections 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f185806b4d0 0x7f185806d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f187815e0b0 0x7f187814f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18780d4a10 0x7f187814def0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 --2- 172.21.15.32:0/2118443366 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18780d0c80 0x7f18780cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 >> 172.21.15.32:0/2118443366 conn(0x7f18780d3d30 msgr2=0x7f18780b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 shutdown_connections 2024-03-06T07:36:59.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:36:59.790+0000 7f187ca00640 1 -- 172.21.15.32:0/2118443366 wait complete. 2024-03-06T07:36:59.812 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:36:59.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:37:00.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.043+0000 7fb831000640 1 Processor -- start 2024-03-06T07:37:00.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.044+0000 7fb831000640 1 -- start start 2024-03-06T07:37:00.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.045+0000 7fb831000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb82c161b40 0x7fb82c0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.045+0000 7fb831000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0cea50 con 0x7fb82c0d2930 2024-03-06T07:37:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.045+0000 7fb831000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0cecf0 con 0x7fb82c0b5060 2024-03-06T07:37:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.045+0000 7fb831000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c0cf900 con 0x7fb82c161b40 2024-03-06T07:37:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.046+0000 7fb82a800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb82c0d2930 0x7fb82c0b3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39794/0 (socket says 172.21.15.32:39794) 2024-03-06T07:37:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.046+0000 7fb82a800640 1 -- 172.21.15.32:0/471626519 learned_addr learned my addr 172.21.15.32:0/471626519 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.046+0000 7fb829e00640 1 --2- 172.21.15.32:0/471626519 >> v2:172.21.15.39:3300/0 conn(0x7fb82c161b40 0x7fb82c0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.046+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707174240 0 0) 0x7fb82c0cea50 con 0x7fb82c0d2930 2024-03-06T07:37:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.046+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810003520 con 0x7fb82c0d2930 2024-03-06T07:37:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1460057651 0 0) 0x7fb810003520 con 0x7fb82c0d2930 2024-03-06T07:37:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 >> v2:172.21.15.39:3300/0 conn(0x7fb82c161b40 msgr2=0x7fb82c0cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 --2- 172.21.15.32:0/471626519 >> v2:172.21.15.39:3300/0 conn(0x7fb82c161b40 0x7fb82c0cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 >> v1:172.21.15.39:6789/0 conn(0x7fb82c0b5060 legacy=0x7fb82c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0cefd0 con 0x7fb82c0d2930 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb829e00640 1 --2- 172.21.15.32:0/471626519 >> v2:172.21.15.39:3300/0 conn(0x7fb82c161b40 0x7fb82c0cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fb824001670 con 0x7fb82c0d2930 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 >> v1:172.21.15.32:6789/0 conn(0x7fb82c0d2930 legacy=0x7fb82c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810003a70 0x7fb810003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb810004800 0x7fb810004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb810005210 0x7fb81000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0cf900 con 0x7fb810005210 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000c140 con 0x7fb810003a70 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000c340 con 0x7fb810004800 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb82a800640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb810005210 0x7fb81000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.047+0000 7fb829e00640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810003a70 0x7fb810003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb829400640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb810004800 0x7fb810004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb82a800640 1 -- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb810004800 msgr2=0x7fb810004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb82a800640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb810004800 0x7fb810004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb82a800640 1 -- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810003a70 msgr2=0x7fb810003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb82a800640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810003a70 0x7fb810003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb82a800640 1 -- 172.21.15.32:0/471626519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810003520 con 0x7fb810005210 2024-03-06T07:37:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.048+0000 7fb829400640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb810004800 0x7fb810004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.049+0000 7fb82a800640 1 -- 172.21.15.32:0/471626519 reap_dead start 2024-03-06T07:37:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.049+0000 7fb82a800640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb810005210 0x7fb81000b9c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0d19f0 tx=0x7fb810005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36ed53bb52a6ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.049+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824007c60 con 0x7fb810005210 2024-03-06T07:37:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.049+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824030070 con 0x7fb810005210 2024-03-06T07:37:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.049+0000 7fb828a00640 1 -- 172.21.15.32:0/471626519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824005d30 con 0x7fb810005210 2024-03-06T07:37:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 -- 172.21.15.32:0/471626519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb810005210 msgr2=0x7fb81000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb810005210 0x7fb81000b9c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0d19f0 tx=0x7fb810005830 comp rx=0 tx=0).stop 2024-03-06T07:37:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 -- 172.21.15.32:0/471626519 shutdown_connections 2024-03-06T07:37:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 --2- 172.21.15.32:0/471626519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb810005210 0x7fb81000b9c0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 -- 172.21.15.32:0/471626519 >> 172.21.15.32:0/471626519 conn(0x7fb82c09d220 msgr2=0x7fb82c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 -- 172.21.15.32:0/471626519 shutdown_connections 2024-03-06T07:37:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.050+0000 7fb831000640 1 -- 172.21.15.32:0/471626519 wait complete. 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.052+0000 7fb831000640 1 Processor -- start 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.052+0000 7fb831000640 1 -- start start 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb831000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810005210 0x7fb82c159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb831000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb82c154320 0x7fb82c1f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb831000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 0x7fb82c15c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb831000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0b2ff0 con 0x7fb82c1f46f0 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb831000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0cf590 con 0x7fb810005210 2024-03-06T07:37:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb831000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0b3170 con 0x7fb82c154320 2024-03-06T07:37:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb82a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 0x7fb82c15c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb82a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 0x7fb82c15c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33368/0 (socket says 172.21.15.32:33368) 2024-03-06T07:37:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb82a800640 1 -- 172.21.15.32:0/2767384311 learned_addr learned my addr 172.21.15.32:0/2767384311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.053+0000 7fb829e00640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810005210 0x7fb82c159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb829400640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb82c154320 0x7fb82c1f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb82a800640 1 -- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb82c154320 msgr2=0x7fb82c1f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb82a800640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb82c154320 0x7fb82c1f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb82a800640 1 -- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810005210 msgr2=0x7fb82c159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb82a800640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810005210 0x7fb82c159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb82a800640 1 -- 172.21.15.32:0/2767384311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c1f7d00 con 0x7fb82c1f46f0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb829e00640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810005210 0x7fb82c159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb829400640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb82c154320 0x7fb82c1f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb82a800640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 0x7fb82c15c990 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb82402c450 tx=0x7fb824001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824045070 con 0x7fb82c1f46f0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824000ed0 con 0x7fb82c1f46f0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c1f7fe0 con 0x7fb82c1f46f0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.054+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824007c60 con 0x7fb82c1f46f0 2024-03-06T07:37:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.055+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c1f3e80 con 0x7fb82c1f46f0 2024-03-06T07:37:00.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.056+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c04f3a0 con 0x7fb82c1f46f0 2024-03-06T07:37:00.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.057+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb824048070 con 0x7fb82c1f46f0 2024-03-06T07:37:00.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.057+0000 7fb822000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb80806b500 0x7fb80806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.058+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8240ab0b0 con 0x7fb82c1f46f0 2024-03-06T07:37:00.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.058+0000 7fb829e00640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb80806b500 0x7fb80806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.059+0000 7fb829e00640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb80806b500 0x7fb80806d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb81c007560 tx=0x7fb81c00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.062+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb824079a00 con 0x7fb82c1f46f0 2024-03-06T07:37:00.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.260+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb82c08bfa0 con 0x7fb80806b500 2024-03-06T07:37:00.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.278+0000 7fb822000640 1 -- 172.21.15.32:0/2767384311 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7fb82c08bfa0 con 0x7fb80806b500 2024-03-06T07:37:00.283 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:00.285 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb80806b500 msgr2=0x7fb80806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb80806b500 0x7fb80806d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb81c007560 tx=0x7fb81c00b040 comp rx=0 tx=0).stop 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 msgr2=0x7fb82c15c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 0x7fb82c15c990 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb82402c450 tx=0x7fb824001970 comp rx=0 tx=0).stop 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 shutdown_connections 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb80806b500 0x7fb80806d9c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb82c1f46f0 0x7fb82c15c990 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb82c154320 0x7fb82c1f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 --2- 172.21.15.32:0/2767384311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb810005210 0x7fb82c159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 >> 172.21.15.32:0/2767384311 conn(0x7fb82c09d220 msgr2=0x7fb82c160920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 shutdown_connections 2024-03-06T07:37:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.285+0000 7fb831000640 1 -- 172.21.15.32:0/2767384311 wait complete. 2024-03-06T07:37:00.307 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-03-06T07:36:58.335606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317176,"kb_used_data":13608,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754657544,"statfs":{"total":773094113280,"available":772769325056,"internally_reserved":0,"allocated":13934592,"data_stored":6701244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.568435"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852660+0000","last_change":"2024-03-06T07:36:55.831929+0000","last_active":"2024-03-06T07:36:56.852660+0000","last_peered":"2024-03-06T07:36:56.852660+0000","last_clean":"2024-03-06T07:36:56.852660+0000","last_became_active":"2024-03-06T07:36:55.827130+0000","last_became_peered":"2024-03-06T07:36:55.827130+0000","last_unstale":"2024-03-06T07:36:56.852660+0000","last_undegraded":"2024-03-06T07:36:56.852660+0000","last_fullsized":"2024-03-06T07:36:56.852660+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.915824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809803+0000","last_change":"2024-03-06T07:36:55.809249+0000","last_active":"2024-03-06T07:36:56.809803+0000","last_peered":"2024-03-06T07:36:56.809803+0000","last_clean":"2024-03-06T07:36:56.809803+0000","last_became_active":"2024-03-06T07:36:55.808725+0000","last_became_peered":"2024-03-06T07:36:55.808725+0000","last_unstale":"2024-03-06T07:36:56.809803+0000","last_undegraded":"2024-03-06T07:36:56.809803+0000","last_fullsized":"2024-03-06T07:36:56.809803+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:21.187745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818954+0000","last_change":"2024-03-06T07:36:55.817914+0000","last_active":"2024-03-06T07:36:56.818954+0000","last_peered":"2024-03-06T07:36:56.818954+0000","last_clean":"2024-03-06T07:36:56.818954+0000","last_became_active":"2024-03-06T07:36:55.817599+0000","last_became_peered":"2024-03-06T07:36:55.817599+0000","last_unstale":"2024-03-06T07:36:56.818954+0000","last_undegraded":"2024-03-06T07:36:56.818954+0000","last_fullsized":"2024-03-06T07:36:56.818954+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:26.782531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852184+0000","last_change":"2024-03-06T07:36:55.822838+0000","last_active":"2024-03-06T07:36:56.852184+0000","last_peered":"2024-03-06T07:36:56.852184+0000","last_clean":"2024-03-06T07:36:56.852184+0000","last_became_active":"2024-03-06T07:36:55.822521+0000","last_became_peered":"2024-03-06T07:36:55.822521+0000","last_unstale":"2024-03-06T07:36:56.852184+0000","last_undegraded":"2024-03-06T07:36:56.852184+0000","last_fullsized":"2024-03-06T07:36:56.852184+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:19.425114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815192+0000","last_change":"2024-03-06T07:36:55.809779+0000","last_active":"2024-03-06T07:36:56.815192+0000","last_peered":"2024-03-06T07:36:56.815192+0000","last_clean":"2024-03-06T07:36:56.815192+0000","last_became_active":"2024-03-06T07:36:55.808557+0000","last_became_peered":"2024-03-06T07:36:55.808557+0000","last_unstale":"2024-03-06T07:36:56.815192+0000","last_undegraded":"2024-03-06T07:36:56.815192+0000","last_fullsized":"2024-03-06T07:36:56.815192+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:09.742084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827418+0000","last_change":"2024-03-06T07:36:55.819339+0000","last_active":"2024-03-06T07:36:56.827418+0000","last_peered":"2024-03-06T07:36:56.827418+0000","last_clean":"2024-03-06T07:36:56.827418+0000","last_became_active":"2024-03-06T07:36:55.819027+0000","last_became_peered":"2024-03-06T07:36:55.819027+0000","last_unstale":"2024-03-06T07:36:56.827418+0000","last_undegraded":"2024-03-06T07:36:56.827418+0000","last_fullsized":"2024-03-06T07:36:56.827418+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:32.397732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.826922+0000","last_change":"2024-03-06T07:36:55.809551+0000","last_active":"2024-03-06T07:36:56.826922+0000","last_peered":"2024-03-06T07:36:56.826922+0000","last_clean":"2024-03-06T07:36:56.826922+0000","last_became_active":"2024-03-06T07:36:55.809152+0000","last_became_peered":"2024-03-06T07:36:55.809152+0000","last_unstale":"2024-03-06T07:36:56.826922+0000","last_undegraded":"2024-03-06T07:36:56.826922+0000","last_fullsized":"2024-03-06T07:36:56.826922+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:48.462774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818213+0000","last_change":"2024-03-06T07:36:55.813264+0000","last_active":"2024-03-06T07:36:56.818213+0000","last_peered":"2024-03-06T07:36:56.818213+0000","last_clean":"2024-03-06T07:36:56.818213+0000","last_became_active":"2024-03-06T07:36:55.812636+0000","last_became_peered":"2024-03-06T07:36:55.812636+0000","last_unstale":"2024-03-06T07:36:56.818213+0000","last_undegraded":"2024-03-06T07:36:56.818213+0000","last_fullsized":"2024-03-06T07:36:56.818213+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:24.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.819785+0000","last_change":"2024-03-06T07:36:55.809080+0000","last_active":"2024-03-06T07:36:56.819785+0000","last_peered":"2024-03-06T07:36:56.819785+0000","last_clean":"2024-03-06T07:36:56.819785+0000","last_became_active":"2024-03-06T07:36:55.808757+0000","last_became_peered":"2024-03-06T07:36:55.808757+0000","last_unstale":"2024-03-06T07:36:56.819785+0000","last_undegraded":"2024-03-06T07:36:56.819785+0000","last_fullsized":"2024-03-06T07:36:56.819785+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.535439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809542+0000","last_change":"2024-03-06T07:36:55.808992+0000","last_active":"2024-03-06T07:36:56.809542+0000","last_peered":"2024-03-06T07:36:56.809542+0000","last_clean":"2024-03-06T07:36:56.809542+0000","last_became_active":"2024-03-06T07:36:55.808342+0000","last_became_peered":"2024-03-06T07:36:55.808342+0000","last_unstale":"2024-03-06T07:36:56.809542+0000","last_undegraded":"2024-03-06T07:36:56.809542+0000","last_fullsized":"2024-03-06T07:36:56.809542+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:21.384944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853397+0000","last_change":"2024-03-06T07:36:55.816090+0000","last_active":"2024-03-06T07:36:56.853397+0000","last_peered":"2024-03-06T07:36:56.853397+0000","last_clean":"2024-03-06T07:36:56.853397+0000","last_became_active":"2024-03-06T07:36:55.815802+0000","last_became_peered":"2024-03-06T07:36:55.815802+0000","last_unstale":"2024-03-06T07:36:56.853397+0000","last_undegraded":"2024-03-06T07:36:56.853397+0000","last_fullsized":"2024-03-06T07:36:56.853397+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.258086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818310+0000","last_change":"2024-03-06T07:36:55.815050+0000","last_active":"2024-03-06T07:36:56.818310+0000","last_peered":"2024-03-06T07:36:56.818310+0000","last_clean":"2024-03-06T07:36:56.818310+0000","last_became_active":"2024-03-06T07:36:55.810164+0000","last_became_peered":"2024-03-06T07:36:55.810164+0000","last_unstale":"2024-03-06T07:36:56.818310+0000","last_undegraded":"2024-03-06T07:36:56.818310+0000","last_fullsized":"2024-03-06T07:36:56.818310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:38.893556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851452+0000","last_change":"2024-03-06T07:36:55.827159+0000","last_active":"2024-03-06T07:36:56.851452+0000","last_peered":"2024-03-06T07:36:56.851452+0000","last_clean":"2024-03-06T07:36:56.851452+0000","last_became_active":"2024-03-06T07:36:55.826871+0000","last_became_peered":"2024-03-06T07:36:55.826871+0000","last_unstale":"2024-03-06T07:36:56.851452+0000","last_undegraded":"2024-03-06T07:36:56.851452+0000","last_fullsized":"2024-03-06T07:36:56.851452+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:31.052149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808856+0000","last_change":"2024-03-06T07:36:55.813483+0000","last_active":"2024-03-06T07:36:56.808856+0000","last_peered":"2024-03-06T07:36:56.808856+0000","last_clean":"2024-03-06T07:36:56.808856+0000","last_became_active":"2024-03-06T07:36:55.813158+0000","last_became_peered":"2024-03-06T07:36:55.813158+0000","last_unstale":"2024-03-06T07:36:56.808856+0000","last_undegraded":"2024-03-06T07:36:56.808856+0000","last_fullsized":"2024-03-06T07:36:56.808856+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:17.777957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852351+0000","last_change":"2024-03-06T07:36:55.826408+0000","last_active":"2024-03-06T07:36:56.852351+0000","last_peered":"2024-03-06T07:36:56.852351+0000","last_clean":"2024-03-06T07:36:56.852351+0000","last_became_active":"2024-03-06T07:36:55.825070+0000","last_became_peered":"2024-03-06T07:36:55.825070+0000","last_unstale":"2024-03-06T07:36:56.852351+0000","last_undegraded":"2024-03-06T07:36:56.852351+0000","last_fullsized":"2024-03-06T07:36:56.852351+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:18.407562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851866+0000","last_change":"2024-03-06T07:36:55.811937+0000","last_active":"2024-03-06T07:36:56.851866+0000","last_peered":"2024-03-06T07:36:56.851866+0000","last_clean":"2024-03-06T07:36:56.851866+0000","last_became_active":"2024-03-06T07:36:55.811644+0000","last_became_peered":"2024-03-06T07:36:55.811644+0000","last_unstale":"2024-03-06T07:36:56.851866+0000","last_undegraded":"2024-03-06T07:36:56.851866+0000","last_fullsized":"2024-03-06T07:36:56.851866+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:41.726816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.828384+0000","last_change":"2024-03-06T07:36:55.809503+0000","last_active":"2024-03-06T07:36:56.828384+0000","last_peered":"2024-03-06T07:36:56.828384+0000","last_clean":"2024-03-06T07:36:56.828384+0000","last_became_active":"2024-03-06T07:36:55.807383+0000","last_became_peered":"2024-03-06T07:36:55.807383+0000","last_unstale":"2024-03-06T07:36:56.828384+0000","last_undegraded":"2024-03-06T07:36:56.828384+0000","last_fullsized":"2024-03-06T07:36:56.828384+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.535439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852212+0000","last_change":"2024-03-06T07:36:55.811702+0000","last_active":"2024-03-06T07:36:56.852212+0000","last_peered":"2024-03-06T07:36:56.852212+0000","last_clean":"2024-03-06T07:36:56.852212+0000","last_became_active":"2024-03-06T07:36:55.809871+0000","last_became_peered":"2024-03-06T07:36:55.809871+0000","last_unstale":"2024-03-06T07:36:56.852212+0000","last_undegraded":"2024-03-06T07:36:56.852212+0000","last_fullsized":"2024-03-06T07:36:56.852212+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:09.742084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853477+0000","last_change":"2024-03-06T07:36:55.815440+0000","last_active":"2024-03-06T07:36:56.853477+0000","last_peered":"2024-03-06T07:36:56.853477+0000","last_clean":"2024-03-06T07:36:56.853477+0000","last_became_active":"2024-03-06T07:36:55.815097+0000","last_became_peered":"2024-03-06T07:36:55.815097+0000","last_unstale":"2024-03-06T07:36:56.853477+0000","last_undegraded":"2024-03-06T07:36:56.853477+0000","last_fullsized":"2024-03-06T07:36:56.853477+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:34.152723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852828+0000","last_change":"2024-03-06T07:36:55.808351+0000","last_active":"2024-03-06T07:36:56.852828+0000","last_peered":"2024-03-06T07:36:56.852828+0000","last_clean":"2024-03-06T07:36:56.852828+0000","last_became_active":"2024-03-06T07:36:55.808024+0000","last_became_peered":"2024-03-06T07:36:55.808024+0000","last_unstale":"2024-03-06T07:36:56.852828+0000","last_undegraded":"2024-03-06T07:36:56.852828+0000","last_fullsized":"2024-03-06T07:36:56.852828+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:59.452571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815712+0000","last_change":"2024-03-06T07:36:55.810272+0000","last_active":"2024-03-06T07:36:56.815712+0000","last_peered":"2024-03-06T07:36:56.815712+0000","last_clean":"2024-03-06T07:36:56.815712+0000","last_became_active":"2024-03-06T07:36:55.808586+0000","last_became_peered":"2024-03-06T07:36:55.808586+0000","last_unstale":"2024-03-06T07:36:56.815712+0000","last_undegraded":"2024-03-06T07:36:56.815712+0000","last_fullsized":"2024-03-06T07:36:56.815712+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:15.217946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852097+0000","last_change":"2024-03-06T07:36:55.815854+0000","last_active":"2024-03-06T07:36:56.852097+0000","last_peered":"2024-03-06T07:36:56.852097+0000","last_clean":"2024-03-06T07:36:56.852097+0000","last_became_active":"2024-03-06T07:36:55.815528+0000","last_became_peered":"2024-03-06T07:36:55.815528+0000","last_unstale":"2024-03-06T07:36:56.852097+0000","last_undegraded":"2024-03-06T07:36:56.852097+0000","last_fullsized":"2024-03-06T07:36:56.852097+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:12.557995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.817993+0000","last_change":"2024-03-06T07:36:55.809642+0000","last_active":"2024-03-06T07:36:56.817993+0000","last_peered":"2024-03-06T07:36:56.817993+0000","last_clean":"2024-03-06T07:36:56.817993+0000","last_became_active":"2024-03-06T07:36:55.808529+0000","last_became_peered":"2024-03-06T07:36:55.808529+0000","last_unstale":"2024-03-06T07:36:56.817993+0000","last_undegraded":"2024-03-06T07:36:56.817993+0000","last_fullsized":"2024-03-06T07:36:56.817993+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:32.397732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851266+0000","last_change":"2024-03-06T07:36:55.813621+0000","last_active":"2024-03-06T07:36:56.851266+0000","last_peered":"2024-03-06T07:36:56.851266+0000","last_clean":"2024-03-06T07:36:56.851266+0000","last_became_active":"2024-03-06T07:36:55.813325+0000","last_became_peered":"2024-03-06T07:36:55.813325+0000","last_unstale":"2024-03-06T07:36:56.851266+0000","last_undegraded":"2024-03-06T07:36:56.851266+0000","last_fullsized":"2024-03-06T07:36:56.851266+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:46.783319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809590+0000","last_change":"2024-03-06T07:36:55.800483+0000","last_active":"2024-03-06T07:36:56.809590+0000","last_peered":"2024-03-06T07:36:56.809590+0000","last_clean":"2024-03-06T07:36:56.809590+0000","last_became_active":"2024-03-06T07:36:55.799956+0000","last_became_peered":"2024-03-06T07:36:55.799956+0000","last_unstale":"2024-03-06T07:36:56.809590+0000","last_undegraded":"2024-03-06T07:36:56.809590+0000","last_fullsized":"2024-03-06T07:36:56.809590+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:48.237071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852333+0000","last_change":"2024-03-06T07:36:55.812707+0000","last_active":"2024-03-06T07:36:56.852333+0000","last_peered":"2024-03-06T07:36:56.852333+0000","last_clean":"2024-03-06T07:36:56.852333+0000","last_became_active":"2024-03-06T07:36:55.812419+0000","last_became_peered":"2024-03-06T07:36:55.812419+0000","last_unstale":"2024-03-06T07:36:56.852333+0000","last_undegraded":"2024-03-06T07:36:56.852333+0000","last_fullsized":"2024-03-06T07:36:56.852333+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:39.932539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851266+0000","last_change":"2024-03-06T07:36:55.825666+0000","last_active":"2024-03-06T07:36:56.851266+0000","last_peered":"2024-03-06T07:36:56.851266+0000","last_clean":"2024-03-06T07:36:56.851266+0000","last_became_active":"2024-03-06T07:36:55.825372+0000","last_became_peered":"2024-03-06T07:36:55.825372+0000","last_unstale":"2024-03-06T07:36:56.851266+0000","last_undegraded":"2024-03-06T07:36:56.851266+0000","last_fullsized":"2024-03-06T07:36:56.851266+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:14.648537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853845+0000","last_change":"2024-03-06T07:36:55.821380+0000","last_active":"2024-03-06T07:36:56.853845+0000","last_peered":"2024-03-06T07:36:56.853845+0000","last_clean":"2024-03-06T07:36:56.853845+0000","last_became_active":"2024-03-06T07:36:55.821063+0000","last_became_peered":"2024-03-06T07:36:55.821063+0000","last_unstale":"2024-03-06T07:36:56.853845+0000","last_undegraded":"2024-03-06T07:36:56.853845+0000","last_fullsized":"2024-03-06T07:36:56.853845+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:14.828297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851600+0000","last_change":"2024-03-06T07:36:55.826961+0000","last_active":"2024-03-06T07:36:56.851600+0000","last_peered":"2024-03-06T07:36:56.851600+0000","last_clean":"2024-03-06T07:36:56.851600+0000","last_became_active":"2024-03-06T07:36:55.826723+0000","last_became_peered":"2024-03-06T07:36:55.826723+0000","last_unstale":"2024-03-06T07:36:56.851600+0000","last_undegraded":"2024-03-06T07:36:56.851600+0000","last_fullsized":"2024-03-06T07:36:56.851600+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:41.397125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851412+0000","last_change":"2024-03-06T07:36:55.823199+0000","last_active":"2024-03-06T07:36:56.851412+0000","last_peered":"2024-03-06T07:36:56.851412+0000","last_clean":"2024-03-06T07:36:56.851412+0000","last_became_active":"2024-03-06T07:36:55.822776+0000","last_became_peered":"2024-03-06T07:36:55.822776+0000","last_unstale":"2024-03-06T07:36:56.851412+0000","last_undegraded":"2024-03-06T07:36:56.851412+0000","last_fullsized":"2024-03-06T07:36:56.851412+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:39.681846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814991+0000","last_change":"2024-03-06T07:36:55.817551+0000","last_active":"2024-03-06T07:36:56.814991+0000","last_peered":"2024-03-06T07:36:56.814991+0000","last_clean":"2024-03-06T07:36:56.814991+0000","last_became_active":"2024-03-06T07:36:55.817248+0000","last_became_peered":"2024-03-06T07:36:55.817248+0000","last_unstale":"2024-03-06T07:36:56.814991+0000","last_undegraded":"2024-03-06T07:36:56.814991+0000","last_fullsized":"2024-03-06T07:36:56.814991+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:41.726816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.826858+0000","last_change":"2024-03-06T07:36:55.816048+0000","last_active":"2024-03-06T07:36:56.826858+0000","last_peered":"2024-03-06T07:36:56.826858+0000","last_clean":"2024-03-06T07:36:56.826858+0000","last_became_active":"2024-03-06T07:36:55.815752+0000","last_became_peered":"2024-03-06T07:36:55.815752+0000","last_unstale":"2024-03-06T07:36:56.826858+0000","last_undegraded":"2024-03-06T07:36:56.826858+0000","last_fullsized":"2024-03-06T07:36:56.826858+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:37.887549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852481+0000","last_change":"2024-03-06T07:36:55.812050+0000","last_active":"2024-03-06T07:36:56.852481+0000","last_peered":"2024-03-06T07:36:56.852481+0000","last_clean":"2024-03-06T07:36:56.852481+0000","last_became_active":"2024-03-06T07:36:55.810442+0000","last_became_peered":"2024-03-06T07:36:55.810442+0000","last_unstale":"2024-03-06T07:36:56.852481+0000","last_undegraded":"2024-03-06T07:36:56.852481+0000","last_fullsized":"2024-03-06T07:36:56.852481+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:15.217946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"126'4","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.816831+0000","last_change":"2024-03-06T07:36:54.822172+0000","last_active":"2024-03-06T07:36:56.816831+0000","last_peered":"2024-03-06T07:36:56.816831+0000","last_clean":"2024-03-06T07:36:56.816831+0000","last_became_active":"2024-03-06T07:36:54.820972+0000","last_became_peered":"2024-03-06T07:36:54.820972+0000","last_unstale":"2024-03-06T07:36:56.816831+0000","last_undegraded":"2024-03-06T07:36:56.816831+0000","last_fullsized":"2024-03-06T07:36:56.816831+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:39.396215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.819876+0000","last_change":"2024-03-06T07:36:55.817057+0000","last_active":"2024-03-06T07:36:56.819876+0000","last_peered":"2024-03-06T07:36:56.819876+0000","last_clean":"2024-03-06T07:36:56.819876+0000","last_became_active":"2024-03-06T07:36:55.816722+0000","last_became_peered":"2024-03-06T07:36:55.816722+0000","last_unstale":"2024-03-06T07:36:56.819876+0000","last_undegraded":"2024-03-06T07:36:56.819876+0000","last_fullsized":"2024-03-06T07:36:56.819876+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:02.275787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854123+0000","last_change":"2024-03-06T07:36:54.816389+0000","last_active":"2024-03-06T07:36:56.854123+0000","last_peered":"2024-03-06T07:36:56.854123+0000","last_clean":"2024-03-06T07:36:56.854123+0000","last_became_active":"2024-03-06T07:36:54.816113+0000","last_became_peered":"2024-03-06T07:36:54.816113+0000","last_unstale":"2024-03-06T07:36:56.854123+0000","last_undegraded":"2024-03-06T07:36:56.854123+0000","last_fullsized":"2024-03-06T07:36:56.854123+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:44.180475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815188+0000","last_change":"2024-03-06T07:36:55.815864+0000","last_active":"2024-03-06T07:36:56.815188+0000","last_peered":"2024-03-06T07:36:56.815188+0000","last_clean":"2024-03-06T07:36:56.815188+0000","last_became_active":"2024-03-06T07:36:55.814673+0000","last_became_peered":"2024-03-06T07:36:55.814673+0000","last_unstale":"2024-03-06T07:36:56.815188+0000","last_undegraded":"2024-03-06T07:36:56.815188+0000","last_fullsized":"2024-03-06T07:36:56.815188+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:57.807342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853533+0000","last_change":"2024-03-06T07:36:54.801673+0000","last_active":"2024-03-06T07:36:56.853533+0000","last_peered":"2024-03-06T07:36:56.853533+0000","last_clean":"2024-03-06T07:36:56.853533+0000","last_became_active":"2024-03-06T07:36:54.801420+0000","last_became_peered":"2024-03-06T07:36:54.801420+0000","last_unstale":"2024-03-06T07:36:56.853533+0000","last_undegraded":"2024-03-06T07:36:56.853533+0000","last_fullsized":"2024-03-06T07:36:56.853533+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:51.791721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815926+0000","last_change":"2024-03-06T07:36:55.806941+0000","last_active":"2024-03-06T07:36:56.815926+0000","last_peered":"2024-03-06T07:36:56.815926+0000","last_clean":"2024-03-06T07:36:56.815926+0000","last_became_active":"2024-03-06T07:36:55.806657+0000","last_became_peered":"2024-03-06T07:36:55.806657+0000","last_unstale":"2024-03-06T07:36:56.815926+0000","last_undegraded":"2024-03-06T07:36:56.815926+0000","last_fullsized":"2024-03-06T07:36:56.815926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:59.452571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809023+0000","last_change":"2024-03-06T07:36:54.805475+0000","last_active":"2024-03-06T07:36:56.809023+0000","last_peered":"2024-03-06T07:36:56.809023+0000","last_clean":"2024-03-06T07:36:56.809023+0000","last_became_active":"2024-03-06T07:36:54.804247+0000","last_became_peered":"2024-03-06T07:36:54.804247+0000","last_unstale":"2024-03-06T07:36:56.809023+0000","last_undegraded":"2024-03-06T07:36:56.809023+0000","last_fullsized":"2024-03-06T07:36:56.809023+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.381139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853060+0000","last_change":"2024-03-06T07:36:55.811868+0000","last_active":"2024-03-06T07:36:56.853060+0000","last_peered":"2024-03-06T07:36:56.853060+0000","last_clean":"2024-03-06T07:36:56.853060+0000","last_became_active":"2024-03-06T07:36:55.811575+0000","last_became_peered":"2024-03-06T07:36:55.811575+0000","last_unstale":"2024-03-06T07:36:56.853060+0000","last_undegraded":"2024-03-06T07:36:56.853060+0000","last_fullsized":"2024-03-06T07:36:56.853060+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:08.915022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808593+0000","last_change":"2024-03-06T07:36:54.801315+0000","last_active":"2024-03-06T07:36:56.808593+0000","last_peered":"2024-03-06T07:36:56.808593+0000","last_clean":"2024-03-06T07:36:56.808593+0000","last_became_active":"2024-03-06T07:36:54.800686+0000","last_became_peered":"2024-03-06T07:36:54.800686+0000","last_unstale":"2024-03-06T07:36:56.808593+0000","last_undegraded":"2024-03-06T07:36:56.808593+0000","last_fullsized":"2024-03-06T07:36:56.808593+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:38.665861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827521+0000","last_change":"2024-03-06T07:36:55.806356+0000","last_active":"2024-03-06T07:36:56.827521+0000","last_peered":"2024-03-06T07:36:56.827521+0000","last_clean":"2024-03-06T07:36:56.827521+0000","last_became_active":"2024-03-06T07:36:55.806024+0000","last_became_peered":"2024-03-06T07:36:55.806024+0000","last_unstale":"2024-03-06T07:36:56.827521+0000","last_undegraded":"2024-03-06T07:36:56.827521+0000","last_fullsized":"2024-03-06T07:36:56.827521+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:24.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853157+0000","last_change":"2024-03-06T07:36:54.801604+0000","last_active":"2024-03-06T07:36:56.853157+0000","last_peered":"2024-03-06T07:36:56.853157+0000","last_clean":"2024-03-06T07:36:56.853157+0000","last_became_active":"2024-03-06T07:36:54.801226+0000","last_became_peered":"2024-03-06T07:36:54.801226+0000","last_unstale":"2024-03-06T07:36:56.853157+0000","last_undegraded":"2024-03-06T07:36:56.853157+0000","last_fullsized":"2024-03-06T07:36:56.853157+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:19.565186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.817985+0000","last_change":"2024-03-06T07:36:55.813503+0000","last_active":"2024-03-06T07:36:56.817985+0000","last_peered":"2024-03-06T07:36:56.817985+0000","last_clean":"2024-03-06T07:36:56.817985+0000","last_became_active":"2024-03-06T07:36:55.812994+0000","last_became_peered":"2024-03-06T07:36:55.812994+0000","last_unstale":"2024-03-06T07:36:56.817985+0000","last_undegraded":"2024-03-06T07:36:56.817985+0000","last_fullsized":"2024-03-06T07:36:56.817985+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:48.462774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.826846+0000","last_change":"2024-03-06T07:36:54.820177+0000","last_active":"2024-03-06T07:36:56.826846+0000","last_peered":"2024-03-06T07:36:56.826846+0000","last_clean":"2024-03-06T07:36:56.826846+0000","last_became_active":"2024-03-06T07:36:54.819874+0000","last_became_peered":"2024-03-06T07:36:54.819874+0000","last_unstale":"2024-03-06T07:36:56.826846+0000","last_undegraded":"2024-03-06T07:36:56.826846+0000","last_fullsized":"2024-03-06T07:36:56.826846+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:20.368958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808331+0000","last_change":"2024-03-06T07:36:55.811642+0000","last_active":"2024-03-06T07:36:56.808331+0000","last_peered":"2024-03-06T07:36:56.808331+0000","last_clean":"2024-03-06T07:36:56.808331+0000","last_became_active":"2024-03-06T07:36:55.811365+0000","last_became_peered":"2024-03-06T07:36:55.811365+0000","last_unstale":"2024-03-06T07:36:56.808331+0000","last_undegraded":"2024-03-06T07:36:56.808331+0000","last_fullsized":"2024-03-06T07:36:56.808331+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:30.187315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808338+0000","last_change":"2024-03-06T07:36:54.811339+0000","last_active":"2024-03-06T07:36:56.808338+0000","last_peered":"2024-03-06T07:36:56.808338+0000","last_clean":"2024-03-06T07:36:56.808338+0000","last_became_active":"2024-03-06T07:36:54.811058+0000","last_became_peered":"2024-03-06T07:36:54.811058+0000","last_unstale":"2024-03-06T07:36:56.808338+0000","last_undegraded":"2024-03-06T07:36:56.808338+0000","last_fullsized":"2024-03-06T07:36:56.808338+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:12.091940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852043+0000","last_change":"2024-03-06T07:36:55.824144+0000","last_active":"2024-03-06T07:36:56.852043+0000","last_peered":"2024-03-06T07:36:56.852043+0000","last_clean":"2024-03-06T07:36:56.852043+0000","last_became_active":"2024-03-06T07:36:55.823770+0000","last_became_peered":"2024-03-06T07:36:55.823770+0000","last_unstale":"2024-03-06T07:36:56.852043+0000","last_undegraded":"2024-03-06T07:36:56.852043+0000","last_fullsized":"2024-03-06T07:36:56.852043+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:49.150457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854381+0000","last_change":"2024-03-06T07:36:54.821841+0000","last_active":"2024-03-06T07:36:56.854381+0000","last_peered":"2024-03-06T07:36:56.854381+0000","last_clean":"2024-03-06T07:36:56.854381+0000","last_became_active":"2024-03-06T07:36:54.821374+0000","last_became_peered":"2024-03-06T07:36:54.821374+0000","last_unstale":"2024-03-06T07:36:56.854381+0000","last_undegraded":"2024-03-06T07:36:56.854381+0000","last_fullsized":"2024-03-06T07:36:56.854381+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:48.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854069+0000","last_change":"2024-03-06T07:36:55.820122+0000","last_active":"2024-03-06T07:36:56.854069+0000","last_peered":"2024-03-06T07:36:56.854069+0000","last_clean":"2024-03-06T07:36:56.854069+0000","last_became_active":"2024-03-06T07:36:55.819833+0000","last_became_peered":"2024-03-06T07:36:55.819833+0000","last_unstale":"2024-03-06T07:36:56.854069+0000","last_undegraded":"2024-03-06T07:36:56.854069+0000","last_fullsized":"2024-03-06T07:36:56.854069+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.934649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853127+0000","last_change":"2024-03-06T07:36:54.821191+0000","last_active":"2024-03-06T07:36:56.853127+0000","last_peered":"2024-03-06T07:36:56.853127+0000","last_clean":"2024-03-06T07:36:56.853127+0000","last_became_active":"2024-03-06T07:36:54.820898+0000","last_became_peered":"2024-03-06T07:36:54.820898+0000","last_unstale":"2024-03-06T07:36:56.853127+0000","last_undegraded":"2024-03-06T07:36:56.853127+0000","last_fullsized":"2024-03-06T07:36:56.853127+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:00.979284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815258+0000","last_change":"2024-03-06T07:36:55.813910+0000","last_active":"2024-03-06T07:36:56.815258+0000","last_peered":"2024-03-06T07:36:56.815258+0000","last_clean":"2024-03-06T07:36:56.815258+0000","last_became_active":"2024-03-06T07:36:55.813195+0000","last_became_peered":"2024-03-06T07:36:55.813195+0000","last_unstale":"2024-03-06T07:36:56.815258+0000","last_undegraded":"2024-03-06T07:36:56.815258+0000","last_fullsized":"2024-03-06T07:36:56.815258+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:39.932539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"126'3","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853599+0000","last_change":"2024-03-06T07:36:54.815439+0000","last_active":"2024-03-06T07:36:56.853599+0000","last_peered":"2024-03-06T07:36:56.853599+0000","last_clean":"2024-03-06T07:36:56.853599+0000","last_became_active":"2024-03-06T07:36:54.814416+0000","last_became_peered":"2024-03-06T07:36:54.814416+0000","last_unstale":"2024-03-06T07:36:56.853599+0000","last_undegraded":"2024-03-06T07:36:56.853599+0000","last_fullsized":"2024-03-06T07:36:56.853599+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:41.671260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.816774+0000","last_change":"2024-03-06T07:36:55.798551+0000","last_active":"2024-03-06T07:36:56.816774+0000","last_peered":"2024-03-06T07:36:56.816774+0000","last_clean":"2024-03-06T07:36:56.816774+0000","last_became_active":"2024-03-06T07:36:55.797952+0000","last_became_peered":"2024-03-06T07:36:55.797952+0000","last_unstale":"2024-03-06T07:36:56.816774+0000","last_undegraded":"2024-03-06T07:36:56.816774+0000","last_fullsized":"2024-03-06T07:36:56.816774+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:23.054410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"126'6","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818621+0000","last_change":"2024-03-06T07:36:54.816635+0000","last_active":"2024-03-06T07:36:56.818621+0000","last_peered":"2024-03-06T07:36:56.818621+0000","last_clean":"2024-03-06T07:36:56.818621+0000","last_became_active":"2024-03-06T07:36:54.816198+0000","last_became_peered":"2024-03-06T07:36:54.816198+0000","last_unstale":"2024-03-06T07:36:56.818621+0000","last_undegraded":"2024-03-06T07:36:56.818621+0000","last_fullsized":"2024-03-06T07:36:56.818621+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:20.368958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851742+0000","last_change":"2024-03-06T07:36:55.817952+0000","last_active":"2024-03-06T07:36:56.851742+0000","last_peered":"2024-03-06T07:36:56.851742+0000","last_clean":"2024-03-06T07:36:56.851742+0000","last_became_active":"2024-03-06T07:36:55.817608+0000","last_became_peered":"2024-03-06T07:36:55.817608+0000","last_unstale":"2024-03-06T07:36:56.851742+0000","last_undegraded":"2024-03-06T07:36:56.851742+0000","last_fullsized":"2024-03-06T07:36:56.851742+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:45.196460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851642+0000","last_change":"2024-03-06T07:36:54.821988+0000","last_active":"2024-03-06T07:36:56.851642+0000","last_peered":"2024-03-06T07:36:56.851642+0000","last_clean":"2024-03-06T07:36:56.851642+0000","last_became_active":"2024-03-06T07:36:54.821659+0000","last_became_peered":"2024-03-06T07:36:54.821659+0000","last_unstale":"2024-03-06T07:36:56.851642+0000","last_undegraded":"2024-03-06T07:36:56.851642+0000","last_fullsized":"2024-03-06T07:36:56.851642+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:54.420552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808222+0000","last_change":"2024-03-06T07:36:55.801055+0000","last_active":"2024-03-06T07:36:56.808222+0000","last_peered":"2024-03-06T07:36:56.808222+0000","last_clean":"2024-03-06T07:36:56.808222+0000","last_became_active":"2024-03-06T07:36:55.800429+0000","last_became_peered":"2024-03-06T07:36:55.800429+0000","last_unstale":"2024-03-06T07:36:56.808222+0000","last_undegraded":"2024-03-06T07:36:56.808222+0000","last_fullsized":"2024-03-06T07:36:56.808222+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.934649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808464+0000","last_change":"2024-03-06T07:36:54.814157+0000","last_active":"2024-03-06T07:36:56.808464+0000","last_peered":"2024-03-06T07:36:56.808464+0000","last_clean":"2024-03-06T07:36:56.808464+0000","last_became_active":"2024-03-06T07:36:54.813869+0000","last_became_peered":"2024-03-06T07:36:54.813869+0000","last_unstale":"2024-03-06T07:36:56.808464+0000","last_undegraded":"2024-03-06T07:36:56.808464+0000","last_fullsized":"2024-03-06T07:36:56.808464+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:13.632551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.817739+0000","last_change":"2024-03-06T07:36:55.809823+0000","last_active":"2024-03-06T07:36:56.817739+0000","last_peered":"2024-03-06T07:36:56.817739+0000","last_clean":"2024-03-06T07:36:56.817739+0000","last_became_active":"2024-03-06T07:36:55.808996+0000","last_became_peered":"2024-03-06T07:36:55.808996+0000","last_unstale":"2024-03-06T07:36:56.817739+0000","last_undegraded":"2024-03-06T07:36:56.817739+0000","last_fullsized":"2024-03-06T07:36:56.817739+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:37.887549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"126'8","reported_seq":22,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818474+0000","last_change":"2024-03-06T07:36:54.811756+0000","last_active":"2024-03-06T07:36:56.818474+0000","last_peered":"2024-03-06T07:36:56.818474+0000","last_clean":"2024-03-06T07:36:56.818474+0000","last_became_active":"2024-03-06T07:36:54.811279+0000","last_became_peered":"2024-03-06T07:36:54.811279+0000","last_unstale":"2024-03-06T07:36:56.818474+0000","last_undegraded":"2024-03-06T07:36:56.818474+0000","last_fullsized":"2024-03-06T07:36:56.818474+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:50.980550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.819297+0000","last_change":"2024-03-06T07:36:55.819115+0000","last_active":"2024-03-06T07:36:56.819297+0000","last_peered":"2024-03-06T07:36:56.819297+0000","last_clean":"2024-03-06T07:36:56.819297+0000","last_became_active":"2024-03-06T07:36:55.818806+0000","last_became_peered":"2024-03-06T07:36:55.818806+0000","last_unstale":"2024-03-06T07:36:56.819297+0000","last_undegraded":"2024-03-06T07:36:56.819297+0000","last_fullsized":"2024-03-06T07:36:56.819297+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:49.178432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854399+0000","last_change":"2024-03-06T07:36:54.821213+0000","last_active":"2024-03-06T07:36:56.854399+0000","last_peered":"2024-03-06T07:36:56.854399+0000","last_clean":"2024-03-06T07:36:56.854399+0000","last_became_active":"2024-03-06T07:36:54.818860+0000","last_became_peered":"2024-03-06T07:36:54.818860+0000","last_unstale":"2024-03-06T07:36:56.854399+0000","last_undegraded":"2024-03-06T07:36:56.854399+0000","last_fullsized":"2024-03-06T07:36:56.854399+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:18.409129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854171+0000","last_change":"2024-03-06T07:36:55.810303+0000","last_active":"2024-03-06T07:36:56.854171+0000","last_peered":"2024-03-06T07:36:56.854171+0000","last_clean":"2024-03-06T07:36:56.854171+0000","last_became_active":"2024-03-06T07:36:55.810024+0000","last_became_peered":"2024-03-06T07:36:55.810024+0000","last_unstale":"2024-03-06T07:36:56.854171+0000","last_undegraded":"2024-03-06T07:36:56.854171+0000","last_fullsized":"2024-03-06T07:36:56.854171+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:17.107118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852970+0000","last_change":"2024-03-06T07:36:54.816875+0000","last_active":"2024-03-06T07:36:56.852970+0000","last_peered":"2024-03-06T07:36:56.852970+0000","last_clean":"2024-03-06T07:36:56.852970+0000","last_became_active":"2024-03-06T07:36:54.816612+0000","last_became_peered":"2024-03-06T07:36:54.816612+0000","last_unstale":"2024-03-06T07:36:56.852970+0000","last_undegraded":"2024-03-06T07:36:56.852970+0000","last_fullsized":"2024-03-06T07:36:56.852970+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:36.068638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853001+0000","last_change":"2024-03-06T07:36:55.817227+0000","last_active":"2024-03-06T07:36:56.853001+0000","last_peered":"2024-03-06T07:36:56.853001+0000","last_clean":"2024-03-06T07:36:56.853001+0000","last_became_active":"2024-03-06T07:36:55.815247+0000","last_became_peered":"2024-03-06T07:36:55.815247+0000","last_unstale":"2024-03-06T07:36:56.853001+0000","last_undegraded":"2024-03-06T07:36:56.853001+0000","last_fullsized":"2024-03-06T07:36:56.853001+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:23.054410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827883+0000","last_change":"2024-03-06T07:36:54.813308+0000","last_active":"2024-03-06T07:36:56.827883+0000","last_peered":"2024-03-06T07:36:56.827883+0000","last_clean":"2024-03-06T07:36:56.827883+0000","last_became_active":"2024-03-06T07:36:54.812993+0000","last_became_peered":"2024-03-06T07:36:54.812993+0000","last_unstale":"2024-03-06T07:36:56.827883+0000","last_undegraded":"2024-03-06T07:36:56.827883+0000","last_fullsized":"2024-03-06T07:36:56.827883+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:39.556105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809331+0000","last_change":"2024-03-06T07:36:55.812512+0000","last_active":"2024-03-06T07:36:56.809331+0000","last_peered":"2024-03-06T07:36:56.809331+0000","last_clean":"2024-03-06T07:36:56.809331+0000","last_became_active":"2024-03-06T07:36:55.811974+0000","last_became_peered":"2024-03-06T07:36:55.811974+0000","last_unstale":"2024-03-06T07:36:56.809331+0000","last_undegraded":"2024-03-06T07:36:56.809331+0000","last_fullsized":"2024-03-06T07:36:56.809331+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:51.996535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853848+0000","last_change":"2024-03-06T07:36:54.814933+0000","last_active":"2024-03-06T07:36:56.853848+0000","last_peered":"2024-03-06T07:36:56.853848+0000","last_clean":"2024-03-06T07:36:56.853848+0000","last_became_active":"2024-03-06T07:36:54.814215+0000","last_became_peered":"2024-03-06T07:36:54.814215+0000","last_unstale":"2024-03-06T07:36:56.853848+0000","last_undegraded":"2024-03-06T07:36:56.853848+0000","last_fullsized":"2024-03-06T07:36:56.853848+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:17.391576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851556+0000","last_change":"2024-03-06T07:36:55.825444+0000","last_active":"2024-03-06T07:36:56.851556+0000","last_peered":"2024-03-06T07:36:56.851556+0000","last_clean":"2024-03-06T07:36:56.851556+0000","last_became_active":"2024-03-06T07:36:55.825117+0000","last_became_peered":"2024-03-06T07:36:55.825117+0000","last_unstale":"2024-03-06T07:36:56.851556+0000","last_undegraded":"2024-03-06T07:36:56.851556+0000","last_fullsized":"2024-03-06T07:36:56.851556+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:42.687245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853838+0000","last_change":"2024-03-06T07:36:54.812962+0000","last_active":"2024-03-06T07:36:56.853838+0000","last_peered":"2024-03-06T07:36:56.853838+0000","last_clean":"2024-03-06T07:36:56.853838+0000","last_became_active":"2024-03-06T07:36:54.812032+0000","last_became_peered":"2024-03-06T07:36:54.812032+0000","last_unstale":"2024-03-06T07:36:56.853838+0000","last_undegraded":"2024-03-06T07:36:56.853838+0000","last_fullsized":"2024-03-06T07:36:56.853838+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.381139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827935+0000","last_change":"2024-03-06T07:36:55.805426+0000","last_active":"2024-03-06T07:36:56.827935+0000","last_peered":"2024-03-06T07:36:56.827935+0000","last_clean":"2024-03-06T07:36:56.827935+0000","last_became_active":"2024-03-06T07:36:55.805012+0000","last_became_peered":"2024-03-06T07:36:55.805012+0000","last_unstale":"2024-03-06T07:36:56.827935+0000","last_undegraded":"2024-03-06T07:36:56.827935+0000","last_fullsized":"2024-03-06T07:36:56.827935+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:38.893556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853247+0000","last_change":"2024-03-06T07:36:54.818382+0000","last_active":"2024-03-06T07:36:56.853247+0000","last_peered":"2024-03-06T07:36:56.853247+0000","last_clean":"2024-03-06T07:36:56.853247+0000","last_became_active":"2024-03-06T07:36:54.815856+0000","last_became_peered":"2024-03-06T07:36:54.815856+0000","last_unstale":"2024-03-06T07:36:56.853247+0000","last_undegraded":"2024-03-06T07:36:56.853247+0000","last_fullsized":"2024-03-06T07:36:56.853247+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:03.460516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808251+0000","last_change":"2024-03-06T07:36:55.806912+0000","last_active":"2024-03-06T07:36:56.808251+0000","last_peered":"2024-03-06T07:36:56.808251+0000","last_clean":"2024-03-06T07:36:56.808251+0000","last_became_active":"2024-03-06T07:36:55.806602+0000","last_became_peered":"2024-03-06T07:36:55.806602+0000","last_unstale":"2024-03-06T07:36:56.808251+0000","last_undegraded":"2024-03-06T07:36:56.808251+0000","last_fullsized":"2024-03-06T07:36:56.808251+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:17.107118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852061+0000","last_change":"2024-03-06T07:36:54.817853+0000","last_active":"2024-03-06T07:36:56.852061+0000","last_peered":"2024-03-06T07:36:56.852061+0000","last_clean":"2024-03-06T07:36:56.852061+0000","last_became_active":"2024-03-06T07:36:54.816265+0000","last_became_peered":"2024-03-06T07:36:54.816265+0000","last_unstale":"2024-03-06T07:36:56.852061+0000","last_undegraded":"2024-03-06T07:36:56.852061+0000","last_fullsized":"2024-03-06T07:36:56.852061+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:41.119076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814537+0000","last_change":"2024-03-06T07:36:55.819401+0000","last_active":"2024-03-06T07:36:56.814537+0000","last_peered":"2024-03-06T07:36:56.814537+0000","last_clean":"2024-03-06T07:36:56.814537+0000","last_became_active":"2024-03-06T07:36:55.819094+0000","last_became_peered":"2024-03-06T07:36:55.819094+0000","last_unstale":"2024-03-06T07:36:56.814537+0000","last_undegraded":"2024-03-06T07:36:56.814537+0000","last_fullsized":"2024-03-06T07:36:56.814537+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:58.738984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851301+0000","last_change":"2024-03-06T07:36:54.815755+0000","last_active":"2024-03-06T07:36:56.851301+0000","last_peered":"2024-03-06T07:36:56.851301+0000","last_clean":"2024-03-06T07:36:56.851301+0000","last_became_active":"2024-03-06T07:36:54.815407+0000","last_became_peered":"2024-03-06T07:36:54.815407+0000","last_unstale":"2024-03-06T07:36:56.851301+0000","last_undegraded":"2024-03-06T07:36:56.851301+0000","last_fullsized":"2024-03-06T07:36:56.851301+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.436585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.828384+0000","last_change":"2024-03-06T07:36:55.814206+0000","last_active":"2024-03-06T07:36:56.828384+0000","last_peered":"2024-03-06T07:36:56.828384+0000","last_clean":"2024-03-06T07:36:56.828384+0000","last_became_active":"2024-03-06T07:36:55.811889+0000","last_became_peered":"2024-03-06T07:36:55.811889+0000","last_unstale":"2024-03-06T07:36:56.828384+0000","last_undegraded":"2024-03-06T07:36:56.828384+0000","last_fullsized":"2024-03-06T07:36:56.828384+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:49.178432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852698+0000","last_change":"2024-03-06T07:36:54.817387+0000","last_active":"2024-03-06T07:36:56.852698+0000","last_peered":"2024-03-06T07:36:56.852698+0000","last_clean":"2024-03-06T07:36:56.852698+0000","last_became_active":"2024-03-06T07:36:54.817095+0000","last_became_peered":"2024-03-06T07:36:54.817095+0000","last_unstale":"2024-03-06T07:36:56.852698+0000","last_undegraded":"2024-03-06T07:36:56.852698+0000","last_fullsized":"2024-03-06T07:36:56.852698+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:28.085612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852580+0000","last_change":"2024-03-06T07:36:55.822657+0000","last_active":"2024-03-06T07:36:56.852580+0000","last_peered":"2024-03-06T07:36:56.852580+0000","last_clean":"2024-03-06T07:36:56.852580+0000","last_became_active":"2024-03-06T07:36:55.822354+0000","last_became_peered":"2024-03-06T07:36:55.822354+0000","last_unstale":"2024-03-06T07:36:56.852580+0000","last_undegraded":"2024-03-06T07:36:56.852580+0000","last_fullsized":"2024-03-06T07:36:56.852580+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:30.258955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"127'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.825701+0000","last_change":"2024-03-06T07:36:54.823648+0000","last_active":"2024-03-06T07:36:56.825701+0000","last_peered":"2024-03-06T07:36:56.825701+0000","last_clean":"2024-03-06T07:36:56.825701+0000","last_became_active":"2024-03-06T07:36:54.823243+0000","last_became_peered":"2024-03-06T07:36:54.823243+0000","last_unstale":"2024-03-06T07:36:56.825701+0000","last_undegraded":"2024-03-06T07:36:56.825701+0000","last_fullsized":"2024-03-06T07:36:56.825701+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:05.495029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809040+0000","last_change":"2024-03-06T07:36:55.803322+0000","last_active":"2024-03-06T07:36:56.809040+0000","last_peered":"2024-03-06T07:36:56.809040+0000","last_clean":"2024-03-06T07:36:56.809040+0000","last_became_active":"2024-03-06T07:36:55.803023+0000","last_became_peered":"2024-03-06T07:36:55.803023+0000","last_unstale":"2024-03-06T07:36:56.809040+0000","last_undegraded":"2024-03-06T07:36:56.809040+0000","last_fullsized":"2024-03-06T07:36:56.809040+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:47.105660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809663+0000","last_change":"2024-03-06T07:36:54.810145+0000","last_active":"2024-03-06T07:36:56.809663+0000","last_peered":"2024-03-06T07:36:56.809663+0000","last_clean":"2024-03-06T07:36:56.809663+0000","last_became_active":"2024-03-06T07:36:54.809790+0000","last_became_peered":"2024-03-06T07:36:54.809790+0000","last_unstale":"2024-03-06T07:36:56.809663+0000","last_undegraded":"2024-03-06T07:36:56.809663+0000","last_fullsized":"2024-03-06T07:36:56.809663+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:30.036164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.828142+0000","last_change":"2024-03-06T07:36:55.820896+0000","last_active":"2024-03-06T07:36:56.828142+0000","last_peered":"2024-03-06T07:36:56.828142+0000","last_clean":"2024-03-06T07:36:56.828142+0000","last_became_active":"2024-03-06T07:36:55.820210+0000","last_became_peered":"2024-03-06T07:36:55.820210+0000","last_unstale":"2024-03-06T07:36:56.828142+0000","last_undegraded":"2024-03-06T07:36:56.828142+0000","last_fullsized":"2024-03-06T07:36:56.828142+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:26.782531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852873+0000","last_change":"2024-03-06T07:36:54.802373+0000","last_active":"2024-03-06T07:36:56.852873+0000","last_peered":"2024-03-06T07:36:56.852873+0000","last_clean":"2024-03-06T07:36:56.852873+0000","last_became_active":"2024-03-06T07:36:54.802084+0000","last_became_peered":"2024-03-06T07:36:54.802084+0000","last_unstale":"2024-03-06T07:36:56.852873+0000","last_undegraded":"2024-03-06T07:36:56.852873+0000","last_fullsized":"2024-03-06T07:36:56.852873+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:07.899036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814902+0000","last_change":"2024-03-06T07:36:55.810310+0000","last_active":"2024-03-06T07:36:56.814902+0000","last_peered":"2024-03-06T07:36:56.814902+0000","last_clean":"2024-03-06T07:36:56.814902+0000","last_became_active":"2024-03-06T07:36:55.810021+0000","last_became_peered":"2024-03-06T07:36:55.810021+0000","last_unstale":"2024-03-06T07:36:56.814902+0000","last_undegraded":"2024-03-06T07:36:56.814902+0000","last_fullsized":"2024-03-06T07:36:56.814902+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:39.914436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808605+0000","last_change":"2024-03-06T07:36:54.814823+0000","last_active":"2024-03-06T07:36:56.808605+0000","last_peered":"2024-03-06T07:36:56.808605+0000","last_clean":"2024-03-06T07:36:56.808605+0000","last_became_active":"2024-03-06T07:36:54.813881+0000","last_became_peered":"2024-03-06T07:36:54.813881+0000","last_unstale":"2024-03-06T07:36:56.808605+0000","last_undegraded":"2024-03-06T07:36:56.808605+0000","last_fullsized":"2024-03-06T07:36:56.808605+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:41.671260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853850+0000","last_change":"2024-03-06T07:36:55.825766+0000","last_active":"2024-03-06T07:36:56.853850+0000","last_peered":"2024-03-06T07:36:56.853850+0000","last_clean":"2024-03-06T07:36:56.853850+0000","last_became_active":"2024-03-06T07:36:55.825443+0000","last_became_peered":"2024-03-06T07:36:55.825443+0000","last_unstale":"2024-03-06T07:36:56.853850+0000","last_undegraded":"2024-03-06T07:36:56.853850+0000","last_fullsized":"2024-03-06T07:36:56.853850+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:42.572203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815175+0000","last_change":"2024-03-06T07:36:54.817857+0000","last_active":"2024-03-06T07:36:56.815175+0000","last_peered":"2024-03-06T07:36:56.815175+0000","last_clean":"2024-03-06T07:36:56.815175+0000","last_became_active":"2024-03-06T07:36:54.816332+0000","last_became_peered":"2024-03-06T07:36:54.816332+0000","last_unstale":"2024-03-06T07:36:56.815175+0000","last_undegraded":"2024-03-06T07:36:56.815175+0000","last_fullsized":"2024-03-06T07:36:56.815175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:53.312572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851895+0000","last_change":"2024-03-06T07:36:55.803908+0000","last_active":"2024-03-06T07:36:56.851895+0000","last_peered":"2024-03-06T07:36:56.851895+0000","last_clean":"2024-03-06T07:36:56.851895+0000","last_became_active":"2024-03-06T07:36:55.803601+0000","last_became_peered":"2024-03-06T07:36:55.803601+0000","last_unstale":"2024-03-06T07:36:56.851895+0000","last_undegraded":"2024-03-06T07:36:56.851895+0000","last_fullsized":"2024-03-06T07:36:56.851895+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:57.807342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852167+0000","last_change":"2024-03-06T07:36:54.820560+0000","last_active":"2024-03-06T07:36:56.852167+0000","last_peered":"2024-03-06T07:36:56.852167+0000","last_clean":"2024-03-06T07:36:56.852167+0000","last_became_active":"2024-03-06T07:36:54.820191+0000","last_became_peered":"2024-03-06T07:36:54.820191+0000","last_unstale":"2024-03-06T07:36:56.852167+0000","last_undegraded":"2024-03-06T07:36:56.852167+0000","last_fullsized":"2024-03-06T07:36:56.852167+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:25.557647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808500+0000","last_change":"2024-03-06T07:36:55.804986+0000","last_active":"2024-03-06T07:36:56.808500+0000","last_peered":"2024-03-06T07:36:56.808500+0000","last_clean":"2024-03-06T07:36:56.808500+0000","last_became_active":"2024-03-06T07:36:55.801747+0000","last_became_peered":"2024-03-06T07:36:55.801747+0000","last_unstale":"2024-03-06T07:36:56.808500+0000","last_undegraded":"2024-03-06T07:36:56.808500+0000","last_fullsized":"2024-03-06T07:36:56.808500+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:09.270568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851419+0000","last_change":"2024-03-06T07:36:54.813927+0000","last_active":"2024-03-06T07:36:56.851419+0000","last_peered":"2024-03-06T07:36:56.851419+0000","last_clean":"2024-03-06T07:36:56.851419+0000","last_became_active":"2024-03-06T07:36:54.813612+0000","last_became_peered":"2024-03-06T07:36:54.813612+0000","last_unstale":"2024-03-06T07:36:56.851419+0000","last_undegraded":"2024-03-06T07:36:56.851419+0000","last_fullsized":"2024-03-06T07:36:56.851419+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:22.038424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814439+0000","last_change":"2024-03-06T07:36:55.816711+0000","last_active":"2024-03-06T07:36:56.814439+0000","last_peered":"2024-03-06T07:36:56.814439+0000","last_clean":"2024-03-06T07:36:56.814439+0000","last_became_active":"2024-03-06T07:36:55.816402+0000","last_became_peered":"2024-03-06T07:36:55.816402+0000","last_unstale":"2024-03-06T07:36:56.814439+0000","last_undegraded":"2024-03-06T07:36:56.814439+0000","last_fullsized":"2024-03-06T07:36:56.814439+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:47.806644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815419+0000","last_change":"2024-03-06T07:36:54.818441+0000","last_active":"2024-03-06T07:36:56.815419+0000","last_peered":"2024-03-06T07:36:56.815419+0000","last_clean":"2024-03-06T07:36:56.815419+0000","last_became_active":"2024-03-06T07:36:54.818125+0000","last_became_peered":"2024-03-06T07:36:54.818125+0000","last_unstale":"2024-03-06T07:36:56.815419+0000","last_undegraded":"2024-03-06T07:36:56.815419+0000","last_fullsized":"2024-03-06T07:36:56.815419+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:01.259801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":371,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814096+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:36:56.814096+0000","last_peered":"2024-03-06T07:36:56.814096+0000","last_clean":"2024-03-06T07:36:56.814096+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:36:56.814096+0000","last_undegraded":"2024-03-06T07:36:56.814096+0000","last_fullsized":"2024-03-06T07:36:56.814096+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37496,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334344,"statfs":{"total":96636764160,"available":96598368256,"internally_reserved":0,"allocated":2203648,"data_stored":1282079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1597440,"data_stored":691073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334940,"statfs":{"total":96636764160,"available":96598978560,"internally_reserved":0,"allocated":1593344,"data_stored":689573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1589248,"data_stored":689531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1593344,"data_stored":689621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574947,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41064,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330776,"statfs":{"total":96636764160,"available":96594714624,"internally_reserved":0,"allocated":1593344,"data_stored":690077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1585152,"data_stored":689461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574948,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41636,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330204,"statfs":{"total":96636764160,"available":96594128896,"internally_reserved":0,"allocated":2179072,"data_stored":1279829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:37:00.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:37:00.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.563+0000 7f8830e00640 1 Processor -- start 2024-03-06T07:37:00.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.563+0000 7f8830e00640 1 -- start start 2024-03-06T07:37:00.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f8830e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f8830e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f882c0cce90 0x7f882c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f8830e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c0714d0 con 0x7f882c145300 2024-03-06T07:37:00.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f8830e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882c0cd300 con 0x7f882c0cce90 2024-03-06T07:37:00.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f8830e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882c0717b0 con 0x7f882c070f10 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f882be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f882c0cce90 0x7f882c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f882be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f882c0cce90 0x7f882c15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33370/0 (socket says 172.21.15.32:33370) 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f882be00640 1 -- 172.21.15.32:0/711233759 learned_addr learned my addr 172.21.15.32:0/711233759 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.565+0000 7f882b400640 1 --1- 172.21.15.32:0/711233759 >> v1:172.21.15.32:6789/0 conn(0x7f882c145300 0x7f882c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39808/0 (socket says 172.21.15.32:39808) 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.566+0000 7f882aa00640 1 --2- 172.21.15.32:0/711233759 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.566+0000 7f882be00640 1 -- 172.21.15.32:0/711233759 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 msgr2=0x7f882c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:00.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.566+0000 7f882be00640 1 --2- 172.21.15.32:0/711233759 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.566+0000 7f882be00640 1 -- 172.21.15.32:0/711233759 >> v1:172.21.15.32:6789/0 conn(0x7f882c145300 legacy=0x7f882c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.566+0000 7f882be00640 1 -- 172.21.15.32:0/711233759 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c071a90 con 0x7f882c0cce90 2024-03-06T07:37:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.567+0000 7f882be00640 1 --2- 172.21.15.32:0/711233759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c15c570 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f8824009e50 tx=0x7f882402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600c217b78a54951 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.567+0000 7f882a000640 1 -- 172.21.15.32:0/711233759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882402cb00 con 0x7f882c0cce90 2024-03-06T07:37:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.567+0000 7f882a000640 1 -- 172.21.15.32:0/711233759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882402cca0 con 0x7f882c0cce90 2024-03-06T07:37:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.567+0000 7f882a000640 1 -- 172.21.15.32:0/711233759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824034610 con 0x7f882c0cce90 2024-03-06T07:37:00.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 -- 172.21.15.32:0/711233759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 msgr2=0x7f882c15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 --2- 172.21.15.32:0/711233759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c15c570 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f8824009e50 tx=0x7f882402bf60 comp rx=0 tx=0).stop 2024-03-06T07:37:00.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 -- 172.21.15.32:0/711233759 shutdown_connections 2024-03-06T07:37:00.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 --2- 172.21.15.32:0/711233759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c15c570 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 --2- 172.21.15.32:0/711233759 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 -- 172.21.15.32:0/711233759 >> 172.21.15.32:0/711233759 conn(0x7f882c06b3d0 msgr2=0x7f882c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 -- 172.21.15.32:0/711233759 shutdown_connections 2024-03-06T07:37:00.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.568+0000 7f8830e00640 1 -- 172.21.15.32:0/711233759 wait complete. 2024-03-06T07:37:00.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.570+0000 7f8830e00640 1 Processor -- start 2024-03-06T07:37:00.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.570+0000 7f8830e00640 1 -- start start 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.571+0000 7f8830e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c06e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.571+0000 7f8830e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c13b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.571+0000 7f8830e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f882c145300 0x7f882c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.571+0000 7f8830e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c15cab0 con 0x7f882c0cce90 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.571+0000 7f8830e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1456e0 con 0x7f882c145300 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.571+0000 7f8830e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882c15cc30 con 0x7f882c070f10 2024-03-06T07:37:00.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c13b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c13b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33386/0 (socket says 172.21.15.32:33386) 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 -- 172.21.15.32:0/2392230159 learned_addr learned my addr 172.21.15.32:0/2392230159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 -- 172.21.15.32:0/2392230159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 msgr2=0x7f882c06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 -- 172.21.15.32:0/2392230159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f882c145300 msgr2=0x7f882c153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f882c145300 0x7f882c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 -- 172.21.15.32:0/2392230159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c1402f0 con 0x7f882c0cce90 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.572+0000 7f882aa00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c13b050 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f881800d800 tx=0x7f881800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.573+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881800b6c0 con 0x7f882c0cce90 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.573+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8818019070 con 0x7f882c0cce90 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.573+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881800ba80 con 0x7f882c0cce90 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.573+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c13b710 con 0x7f882c0cce90 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.573+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c13bcd0 con 0x7f882c0cce90 2024-03-06T07:37:00.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.574+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882c04f330 con 0x7f882c0cce90 2024-03-06T07:37:00.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.575+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f881800bc20 con 0x7f882c0cce90 2024-03-06T07:37:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.577+0000 7f8823400640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f880c06b4d0 0x7f880c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.577+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f881808a750 con 0x7f882c0cce90 2024-03-06T07:37:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.577+0000 7f882b400640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f880c06b4d0 0x7f880c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.578+0000 7f882b400640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f880c06b4d0 0x7f880c06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f881c000f60 tx=0x7f881c007a00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.581+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8818059530 con 0x7f882c0cce90 2024-03-06T07:37:00.722 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-03-06T07:37:00.721+0000 7f882e800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:37:00.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.794+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 --> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f882c144820 con 0x7f880c06b4d0 2024-03-06T07:37:00.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.815+0000 7f8823400640 1 -- 172.21.15.32:0/2392230159 <== mgr.4098 v2:172.21.15.32:6832/1434773126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f882c144820 con 0x7f880c06b4d0 2024-03-06T07:37:00.819 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:00.821 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-03-06T07:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f880c06b4d0 msgr2=0x7f880c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f880c06b4d0 0x7f880c06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f881c000f60 tx=0x7f881c007a00 comp rx=0 tx=0).stop 2024-03-06T07:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 msgr2=0x7f882c13b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c13b050 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f881800d800 tx=0x7f881800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 shutdown_connections 2024-03-06T07:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f880c06b4d0 0x7f880c06d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f882c145300 0x7f882c153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f882c0cce90 0x7f882c13b050 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 --2- 172.21.15.32:0/2392230159 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f882c070f10 0x7f882c06e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.820+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 >> 172.21.15.32:0/2392230159 conn(0x7f882c06b3d0 msgr2=0x7f882c15dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.821+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 shutdown_connections 2024-03-06T07:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:00.821+0000 7f8830e00640 1 -- 172.21.15.32:0/2392230159 wait complete. 2024-03-06T07:37:00.845 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-03-06T07:37:00.336358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317176,"kb_used_data":13608,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754657544,"statfs":{"total":773094113280,"available":772769325056,"internally_reserved":0,"allocated":13934592,"data_stored":6701244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001919"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852660+0000","last_change":"2024-03-06T07:36:55.831929+0000","last_active":"2024-03-06T07:36:56.852660+0000","last_peered":"2024-03-06T07:36:56.852660+0000","last_clean":"2024-03-06T07:36:56.852660+0000","last_became_active":"2024-03-06T07:36:55.827130+0000","last_became_peered":"2024-03-06T07:36:55.827130+0000","last_unstale":"2024-03-06T07:36:56.852660+0000","last_undegraded":"2024-03-06T07:36:56.852660+0000","last_fullsized":"2024-03-06T07:36:56.852660+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:59.915824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809803+0000","last_change":"2024-03-06T07:36:55.809249+0000","last_active":"2024-03-06T07:36:56.809803+0000","last_peered":"2024-03-06T07:36:56.809803+0000","last_clean":"2024-03-06T07:36:56.809803+0000","last_became_active":"2024-03-06T07:36:55.808725+0000","last_became_peered":"2024-03-06T07:36:55.808725+0000","last_unstale":"2024-03-06T07:36:56.809803+0000","last_undegraded":"2024-03-06T07:36:56.809803+0000","last_fullsized":"2024-03-06T07:36:56.809803+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:21.187745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818954+0000","last_change":"2024-03-06T07:36:55.817914+0000","last_active":"2024-03-06T07:36:56.818954+0000","last_peered":"2024-03-06T07:36:56.818954+0000","last_clean":"2024-03-06T07:36:56.818954+0000","last_became_active":"2024-03-06T07:36:55.817599+0000","last_became_peered":"2024-03-06T07:36:55.817599+0000","last_unstale":"2024-03-06T07:36:56.818954+0000","last_undegraded":"2024-03-06T07:36:56.818954+0000","last_fullsized":"2024-03-06T07:36:56.818954+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:26.782531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852184+0000","last_change":"2024-03-06T07:36:55.822838+0000","last_active":"2024-03-06T07:36:56.852184+0000","last_peered":"2024-03-06T07:36:56.852184+0000","last_clean":"2024-03-06T07:36:56.852184+0000","last_became_active":"2024-03-06T07:36:55.822521+0000","last_became_peered":"2024-03-06T07:36:55.822521+0000","last_unstale":"2024-03-06T07:36:56.852184+0000","last_undegraded":"2024-03-06T07:36:56.852184+0000","last_fullsized":"2024-03-06T07:36:56.852184+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:19.425114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815192+0000","last_change":"2024-03-06T07:36:55.809779+0000","last_active":"2024-03-06T07:36:56.815192+0000","last_peered":"2024-03-06T07:36:56.815192+0000","last_clean":"2024-03-06T07:36:56.815192+0000","last_became_active":"2024-03-06T07:36:55.808557+0000","last_became_peered":"2024-03-06T07:36:55.808557+0000","last_unstale":"2024-03-06T07:36:56.815192+0000","last_undegraded":"2024-03-06T07:36:56.815192+0000","last_fullsized":"2024-03-06T07:36:56.815192+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:09.742084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827418+0000","last_change":"2024-03-06T07:36:55.819339+0000","last_active":"2024-03-06T07:36:56.827418+0000","last_peered":"2024-03-06T07:36:56.827418+0000","last_clean":"2024-03-06T07:36:56.827418+0000","last_became_active":"2024-03-06T07:36:55.819027+0000","last_became_peered":"2024-03-06T07:36:55.819027+0000","last_unstale":"2024-03-06T07:36:56.827418+0000","last_undegraded":"2024-03-06T07:36:56.827418+0000","last_fullsized":"2024-03-06T07:36:56.827418+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:32.397732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.826922+0000","last_change":"2024-03-06T07:36:55.809551+0000","last_active":"2024-03-06T07:36:56.826922+0000","last_peered":"2024-03-06T07:36:56.826922+0000","last_clean":"2024-03-06T07:36:56.826922+0000","last_became_active":"2024-03-06T07:36:55.809152+0000","last_became_peered":"2024-03-06T07:36:55.809152+0000","last_unstale":"2024-03-06T07:36:56.826922+0000","last_undegraded":"2024-03-06T07:36:56.826922+0000","last_fullsized":"2024-03-06T07:36:56.826922+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:48.462774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818213+0000","last_change":"2024-03-06T07:36:55.813264+0000","last_active":"2024-03-06T07:36:56.818213+0000","last_peered":"2024-03-06T07:36:56.818213+0000","last_clean":"2024-03-06T07:36:56.818213+0000","last_became_active":"2024-03-06T07:36:55.812636+0000","last_became_peered":"2024-03-06T07:36:55.812636+0000","last_unstale":"2024-03-06T07:36:56.818213+0000","last_undegraded":"2024-03-06T07:36:56.818213+0000","last_fullsized":"2024-03-06T07:36:56.818213+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:24.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.819785+0000","last_change":"2024-03-06T07:36:55.809080+0000","last_active":"2024-03-06T07:36:56.819785+0000","last_peered":"2024-03-06T07:36:56.819785+0000","last_clean":"2024-03-06T07:36:56.819785+0000","last_became_active":"2024-03-06T07:36:55.808757+0000","last_became_peered":"2024-03-06T07:36:55.808757+0000","last_unstale":"2024-03-06T07:36:56.819785+0000","last_undegraded":"2024-03-06T07:36:56.819785+0000","last_fullsized":"2024-03-06T07:36:56.819785+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.535439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809542+0000","last_change":"2024-03-06T07:36:55.808992+0000","last_active":"2024-03-06T07:36:56.809542+0000","last_peered":"2024-03-06T07:36:56.809542+0000","last_clean":"2024-03-06T07:36:56.809542+0000","last_became_active":"2024-03-06T07:36:55.808342+0000","last_became_peered":"2024-03-06T07:36:55.808342+0000","last_unstale":"2024-03-06T07:36:56.809542+0000","last_undegraded":"2024-03-06T07:36:56.809542+0000","last_fullsized":"2024-03-06T07:36:56.809542+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:21.384944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853397+0000","last_change":"2024-03-06T07:36:55.816090+0000","last_active":"2024-03-06T07:36:56.853397+0000","last_peered":"2024-03-06T07:36:56.853397+0000","last_clean":"2024-03-06T07:36:56.853397+0000","last_became_active":"2024-03-06T07:36:55.815802+0000","last_became_peered":"2024-03-06T07:36:55.815802+0000","last_unstale":"2024-03-06T07:36:56.853397+0000","last_undegraded":"2024-03-06T07:36:56.853397+0000","last_fullsized":"2024-03-06T07:36:56.853397+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.258086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818310+0000","last_change":"2024-03-06T07:36:55.815050+0000","last_active":"2024-03-06T07:36:56.818310+0000","last_peered":"2024-03-06T07:36:56.818310+0000","last_clean":"2024-03-06T07:36:56.818310+0000","last_became_active":"2024-03-06T07:36:55.810164+0000","last_became_peered":"2024-03-06T07:36:55.810164+0000","last_unstale":"2024-03-06T07:36:56.818310+0000","last_undegraded":"2024-03-06T07:36:56.818310+0000","last_fullsized":"2024-03-06T07:36:56.818310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:38.893556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851452+0000","last_change":"2024-03-06T07:36:55.827159+0000","last_active":"2024-03-06T07:36:56.851452+0000","last_peered":"2024-03-06T07:36:56.851452+0000","last_clean":"2024-03-06T07:36:56.851452+0000","last_became_active":"2024-03-06T07:36:55.826871+0000","last_became_peered":"2024-03-06T07:36:55.826871+0000","last_unstale":"2024-03-06T07:36:56.851452+0000","last_undegraded":"2024-03-06T07:36:56.851452+0000","last_fullsized":"2024-03-06T07:36:56.851452+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:31.052149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808856+0000","last_change":"2024-03-06T07:36:55.813483+0000","last_active":"2024-03-06T07:36:56.808856+0000","last_peered":"2024-03-06T07:36:56.808856+0000","last_clean":"2024-03-06T07:36:56.808856+0000","last_became_active":"2024-03-06T07:36:55.813158+0000","last_became_peered":"2024-03-06T07:36:55.813158+0000","last_unstale":"2024-03-06T07:36:56.808856+0000","last_undegraded":"2024-03-06T07:36:56.808856+0000","last_fullsized":"2024-03-06T07:36:56.808856+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:17.777957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852351+0000","last_change":"2024-03-06T07:36:55.826408+0000","last_active":"2024-03-06T07:36:56.852351+0000","last_peered":"2024-03-06T07:36:56.852351+0000","last_clean":"2024-03-06T07:36:56.852351+0000","last_became_active":"2024-03-06T07:36:55.825070+0000","last_became_peered":"2024-03-06T07:36:55.825070+0000","last_unstale":"2024-03-06T07:36:56.852351+0000","last_undegraded":"2024-03-06T07:36:56.852351+0000","last_fullsized":"2024-03-06T07:36:56.852351+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:18.407562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851866+0000","last_change":"2024-03-06T07:36:55.811937+0000","last_active":"2024-03-06T07:36:56.851866+0000","last_peered":"2024-03-06T07:36:56.851866+0000","last_clean":"2024-03-06T07:36:56.851866+0000","last_became_active":"2024-03-06T07:36:55.811644+0000","last_became_peered":"2024-03-06T07:36:55.811644+0000","last_unstale":"2024-03-06T07:36:56.851866+0000","last_undegraded":"2024-03-06T07:36:56.851866+0000","last_fullsized":"2024-03-06T07:36:56.851866+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:41.726816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.828384+0000","last_change":"2024-03-06T07:36:55.809503+0000","last_active":"2024-03-06T07:36:56.828384+0000","last_peered":"2024-03-06T07:36:56.828384+0000","last_clean":"2024-03-06T07:36:56.828384+0000","last_became_active":"2024-03-06T07:36:55.807383+0000","last_became_peered":"2024-03-06T07:36:55.807383+0000","last_unstale":"2024-03-06T07:36:56.828384+0000","last_undegraded":"2024-03-06T07:36:56.828384+0000","last_fullsized":"2024-03-06T07:36:56.828384+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.535439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852212+0000","last_change":"2024-03-06T07:36:55.811702+0000","last_active":"2024-03-06T07:36:56.852212+0000","last_peered":"2024-03-06T07:36:56.852212+0000","last_clean":"2024-03-06T07:36:56.852212+0000","last_became_active":"2024-03-06T07:36:55.809871+0000","last_became_peered":"2024-03-06T07:36:55.809871+0000","last_unstale":"2024-03-06T07:36:56.852212+0000","last_undegraded":"2024-03-06T07:36:56.852212+0000","last_fullsized":"2024-03-06T07:36:56.852212+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:09.742084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853477+0000","last_change":"2024-03-06T07:36:55.815440+0000","last_active":"2024-03-06T07:36:56.853477+0000","last_peered":"2024-03-06T07:36:56.853477+0000","last_clean":"2024-03-06T07:36:56.853477+0000","last_became_active":"2024-03-06T07:36:55.815097+0000","last_became_peered":"2024-03-06T07:36:55.815097+0000","last_unstale":"2024-03-06T07:36:56.853477+0000","last_undegraded":"2024-03-06T07:36:56.853477+0000","last_fullsized":"2024-03-06T07:36:56.853477+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:34.152723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852828+0000","last_change":"2024-03-06T07:36:55.808351+0000","last_active":"2024-03-06T07:36:56.852828+0000","last_peered":"2024-03-06T07:36:56.852828+0000","last_clean":"2024-03-06T07:36:56.852828+0000","last_became_active":"2024-03-06T07:36:55.808024+0000","last_became_peered":"2024-03-06T07:36:55.808024+0000","last_unstale":"2024-03-06T07:36:56.852828+0000","last_undegraded":"2024-03-06T07:36:56.852828+0000","last_fullsized":"2024-03-06T07:36:56.852828+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:59.452571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815712+0000","last_change":"2024-03-06T07:36:55.810272+0000","last_active":"2024-03-06T07:36:56.815712+0000","last_peered":"2024-03-06T07:36:56.815712+0000","last_clean":"2024-03-06T07:36:56.815712+0000","last_became_active":"2024-03-06T07:36:55.808586+0000","last_became_peered":"2024-03-06T07:36:55.808586+0000","last_unstale":"2024-03-06T07:36:56.815712+0000","last_undegraded":"2024-03-06T07:36:56.815712+0000","last_fullsized":"2024-03-06T07:36:56.815712+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:15.217946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852097+0000","last_change":"2024-03-06T07:36:55.815854+0000","last_active":"2024-03-06T07:36:56.852097+0000","last_peered":"2024-03-06T07:36:56.852097+0000","last_clean":"2024-03-06T07:36:56.852097+0000","last_became_active":"2024-03-06T07:36:55.815528+0000","last_became_peered":"2024-03-06T07:36:55.815528+0000","last_unstale":"2024-03-06T07:36:56.852097+0000","last_undegraded":"2024-03-06T07:36:56.852097+0000","last_fullsized":"2024-03-06T07:36:56.852097+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:12.557995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.817993+0000","last_change":"2024-03-06T07:36:55.809642+0000","last_active":"2024-03-06T07:36:56.817993+0000","last_peered":"2024-03-06T07:36:56.817993+0000","last_clean":"2024-03-06T07:36:56.817993+0000","last_became_active":"2024-03-06T07:36:55.808529+0000","last_became_peered":"2024-03-06T07:36:55.808529+0000","last_unstale":"2024-03-06T07:36:56.817993+0000","last_undegraded":"2024-03-06T07:36:56.817993+0000","last_fullsized":"2024-03-06T07:36:56.817993+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:32.397732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851266+0000","last_change":"2024-03-06T07:36:55.813621+0000","last_active":"2024-03-06T07:36:56.851266+0000","last_peered":"2024-03-06T07:36:56.851266+0000","last_clean":"2024-03-06T07:36:56.851266+0000","last_became_active":"2024-03-06T07:36:55.813325+0000","last_became_peered":"2024-03-06T07:36:55.813325+0000","last_unstale":"2024-03-06T07:36:56.851266+0000","last_undegraded":"2024-03-06T07:36:56.851266+0000","last_fullsized":"2024-03-06T07:36:56.851266+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:46.783319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809590+0000","last_change":"2024-03-06T07:36:55.800483+0000","last_active":"2024-03-06T07:36:56.809590+0000","last_peered":"2024-03-06T07:36:56.809590+0000","last_clean":"2024-03-06T07:36:56.809590+0000","last_became_active":"2024-03-06T07:36:55.799956+0000","last_became_peered":"2024-03-06T07:36:55.799956+0000","last_unstale":"2024-03-06T07:36:56.809590+0000","last_undegraded":"2024-03-06T07:36:56.809590+0000","last_fullsized":"2024-03-06T07:36:56.809590+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:48.237071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852333+0000","last_change":"2024-03-06T07:36:55.812707+0000","last_active":"2024-03-06T07:36:56.852333+0000","last_peered":"2024-03-06T07:36:56.852333+0000","last_clean":"2024-03-06T07:36:56.852333+0000","last_became_active":"2024-03-06T07:36:55.812419+0000","last_became_peered":"2024-03-06T07:36:55.812419+0000","last_unstale":"2024-03-06T07:36:56.852333+0000","last_undegraded":"2024-03-06T07:36:56.852333+0000","last_fullsized":"2024-03-06T07:36:56.852333+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:39.932539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851266+0000","last_change":"2024-03-06T07:36:55.825666+0000","last_active":"2024-03-06T07:36:56.851266+0000","last_peered":"2024-03-06T07:36:56.851266+0000","last_clean":"2024-03-06T07:36:56.851266+0000","last_became_active":"2024-03-06T07:36:55.825372+0000","last_became_peered":"2024-03-06T07:36:55.825372+0000","last_unstale":"2024-03-06T07:36:56.851266+0000","last_undegraded":"2024-03-06T07:36:56.851266+0000","last_fullsized":"2024-03-06T07:36:56.851266+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:14.648537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853845+0000","last_change":"2024-03-06T07:36:55.821380+0000","last_active":"2024-03-06T07:36:56.853845+0000","last_peered":"2024-03-06T07:36:56.853845+0000","last_clean":"2024-03-06T07:36:56.853845+0000","last_became_active":"2024-03-06T07:36:55.821063+0000","last_became_peered":"2024-03-06T07:36:55.821063+0000","last_unstale":"2024-03-06T07:36:56.853845+0000","last_undegraded":"2024-03-06T07:36:56.853845+0000","last_fullsized":"2024-03-06T07:36:56.853845+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:14.828297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851600+0000","last_change":"2024-03-06T07:36:55.826961+0000","last_active":"2024-03-06T07:36:56.851600+0000","last_peered":"2024-03-06T07:36:56.851600+0000","last_clean":"2024-03-06T07:36:56.851600+0000","last_became_active":"2024-03-06T07:36:55.826723+0000","last_became_peered":"2024-03-06T07:36:55.826723+0000","last_unstale":"2024-03-06T07:36:56.851600+0000","last_undegraded":"2024-03-06T07:36:56.851600+0000","last_fullsized":"2024-03-06T07:36:56.851600+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:41.397125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851412+0000","last_change":"2024-03-06T07:36:55.823199+0000","last_active":"2024-03-06T07:36:56.851412+0000","last_peered":"2024-03-06T07:36:56.851412+0000","last_clean":"2024-03-06T07:36:56.851412+0000","last_became_active":"2024-03-06T07:36:55.822776+0000","last_became_peered":"2024-03-06T07:36:55.822776+0000","last_unstale":"2024-03-06T07:36:56.851412+0000","last_undegraded":"2024-03-06T07:36:56.851412+0000","last_fullsized":"2024-03-06T07:36:56.851412+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:39.681846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814991+0000","last_change":"2024-03-06T07:36:55.817551+0000","last_active":"2024-03-06T07:36:56.814991+0000","last_peered":"2024-03-06T07:36:56.814991+0000","last_clean":"2024-03-06T07:36:56.814991+0000","last_became_active":"2024-03-06T07:36:55.817248+0000","last_became_peered":"2024-03-06T07:36:55.817248+0000","last_unstale":"2024-03-06T07:36:56.814991+0000","last_undegraded":"2024-03-06T07:36:56.814991+0000","last_fullsized":"2024-03-06T07:36:56.814991+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:41.726816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.826858+0000","last_change":"2024-03-06T07:36:55.816048+0000","last_active":"2024-03-06T07:36:56.826858+0000","last_peered":"2024-03-06T07:36:56.826858+0000","last_clean":"2024-03-06T07:36:56.826858+0000","last_became_active":"2024-03-06T07:36:55.815752+0000","last_became_peered":"2024-03-06T07:36:55.815752+0000","last_unstale":"2024-03-06T07:36:56.826858+0000","last_undegraded":"2024-03-06T07:36:56.826858+0000","last_fullsized":"2024-03-06T07:36:56.826858+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:37.887549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852481+0000","last_change":"2024-03-06T07:36:55.812050+0000","last_active":"2024-03-06T07:36:56.852481+0000","last_peered":"2024-03-06T07:36:56.852481+0000","last_clean":"2024-03-06T07:36:56.852481+0000","last_became_active":"2024-03-06T07:36:55.810442+0000","last_became_peered":"2024-03-06T07:36:55.810442+0000","last_unstale":"2024-03-06T07:36:56.852481+0000","last_undegraded":"2024-03-06T07:36:56.852481+0000","last_fullsized":"2024-03-06T07:36:56.852481+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:15.217946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"126'4","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.816831+0000","last_change":"2024-03-06T07:36:54.822172+0000","last_active":"2024-03-06T07:36:56.816831+0000","last_peered":"2024-03-06T07:36:56.816831+0000","last_clean":"2024-03-06T07:36:56.816831+0000","last_became_active":"2024-03-06T07:36:54.820972+0000","last_became_peered":"2024-03-06T07:36:54.820972+0000","last_unstale":"2024-03-06T07:36:56.816831+0000","last_undegraded":"2024-03-06T07:36:56.816831+0000","last_fullsized":"2024-03-06T07:36:56.816831+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:39.396215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.819876+0000","last_change":"2024-03-06T07:36:55.817057+0000","last_active":"2024-03-06T07:36:56.819876+0000","last_peered":"2024-03-06T07:36:56.819876+0000","last_clean":"2024-03-06T07:36:56.819876+0000","last_became_active":"2024-03-06T07:36:55.816722+0000","last_became_peered":"2024-03-06T07:36:55.816722+0000","last_unstale":"2024-03-06T07:36:56.819876+0000","last_undegraded":"2024-03-06T07:36:56.819876+0000","last_fullsized":"2024-03-06T07:36:56.819876+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:02.275787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854123+0000","last_change":"2024-03-06T07:36:54.816389+0000","last_active":"2024-03-06T07:36:56.854123+0000","last_peered":"2024-03-06T07:36:56.854123+0000","last_clean":"2024-03-06T07:36:56.854123+0000","last_became_active":"2024-03-06T07:36:54.816113+0000","last_became_peered":"2024-03-06T07:36:54.816113+0000","last_unstale":"2024-03-06T07:36:56.854123+0000","last_undegraded":"2024-03-06T07:36:56.854123+0000","last_fullsized":"2024-03-06T07:36:56.854123+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:44.180475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815188+0000","last_change":"2024-03-06T07:36:55.815864+0000","last_active":"2024-03-06T07:36:56.815188+0000","last_peered":"2024-03-06T07:36:56.815188+0000","last_clean":"2024-03-06T07:36:56.815188+0000","last_became_active":"2024-03-06T07:36:55.814673+0000","last_became_peered":"2024-03-06T07:36:55.814673+0000","last_unstale":"2024-03-06T07:36:56.815188+0000","last_undegraded":"2024-03-06T07:36:56.815188+0000","last_fullsized":"2024-03-06T07:36:56.815188+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:57.807342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853533+0000","last_change":"2024-03-06T07:36:54.801673+0000","last_active":"2024-03-06T07:36:56.853533+0000","last_peered":"2024-03-06T07:36:56.853533+0000","last_clean":"2024-03-06T07:36:56.853533+0000","last_became_active":"2024-03-06T07:36:54.801420+0000","last_became_peered":"2024-03-06T07:36:54.801420+0000","last_unstale":"2024-03-06T07:36:56.853533+0000","last_undegraded":"2024-03-06T07:36:56.853533+0000","last_fullsized":"2024-03-06T07:36:56.853533+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:51.791721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815926+0000","last_change":"2024-03-06T07:36:55.806941+0000","last_active":"2024-03-06T07:36:56.815926+0000","last_peered":"2024-03-06T07:36:56.815926+0000","last_clean":"2024-03-06T07:36:56.815926+0000","last_became_active":"2024-03-06T07:36:55.806657+0000","last_became_peered":"2024-03-06T07:36:55.806657+0000","last_unstale":"2024-03-06T07:36:56.815926+0000","last_undegraded":"2024-03-06T07:36:56.815926+0000","last_fullsized":"2024-03-06T07:36:56.815926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:59.452571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809023+0000","last_change":"2024-03-06T07:36:54.805475+0000","last_active":"2024-03-06T07:36:56.809023+0000","last_peered":"2024-03-06T07:36:56.809023+0000","last_clean":"2024-03-06T07:36:56.809023+0000","last_became_active":"2024-03-06T07:36:54.804247+0000","last_became_peered":"2024-03-06T07:36:54.804247+0000","last_unstale":"2024-03-06T07:36:56.809023+0000","last_undegraded":"2024-03-06T07:36:56.809023+0000","last_fullsized":"2024-03-06T07:36:56.809023+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.381139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853060+0000","last_change":"2024-03-06T07:36:55.811868+0000","last_active":"2024-03-06T07:36:56.853060+0000","last_peered":"2024-03-06T07:36:56.853060+0000","last_clean":"2024-03-06T07:36:56.853060+0000","last_became_active":"2024-03-06T07:36:55.811575+0000","last_became_peered":"2024-03-06T07:36:55.811575+0000","last_unstale":"2024-03-06T07:36:56.853060+0000","last_undegraded":"2024-03-06T07:36:56.853060+0000","last_fullsized":"2024-03-06T07:36:56.853060+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:08.915022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808593+0000","last_change":"2024-03-06T07:36:54.801315+0000","last_active":"2024-03-06T07:36:56.808593+0000","last_peered":"2024-03-06T07:36:56.808593+0000","last_clean":"2024-03-06T07:36:56.808593+0000","last_became_active":"2024-03-06T07:36:54.800686+0000","last_became_peered":"2024-03-06T07:36:54.800686+0000","last_unstale":"2024-03-06T07:36:56.808593+0000","last_undegraded":"2024-03-06T07:36:56.808593+0000","last_fullsized":"2024-03-06T07:36:56.808593+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:38.665861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827521+0000","last_change":"2024-03-06T07:36:55.806356+0000","last_active":"2024-03-06T07:36:56.827521+0000","last_peered":"2024-03-06T07:36:56.827521+0000","last_clean":"2024-03-06T07:36:56.827521+0000","last_became_active":"2024-03-06T07:36:55.806024+0000","last_became_peered":"2024-03-06T07:36:55.806024+0000","last_unstale":"2024-03-06T07:36:56.827521+0000","last_undegraded":"2024-03-06T07:36:56.827521+0000","last_fullsized":"2024-03-06T07:36:56.827521+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:24.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853157+0000","last_change":"2024-03-06T07:36:54.801604+0000","last_active":"2024-03-06T07:36:56.853157+0000","last_peered":"2024-03-06T07:36:56.853157+0000","last_clean":"2024-03-06T07:36:56.853157+0000","last_became_active":"2024-03-06T07:36:54.801226+0000","last_became_peered":"2024-03-06T07:36:54.801226+0000","last_unstale":"2024-03-06T07:36:56.853157+0000","last_undegraded":"2024-03-06T07:36:56.853157+0000","last_fullsized":"2024-03-06T07:36:56.853157+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:19.565186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.817985+0000","last_change":"2024-03-06T07:36:55.813503+0000","last_active":"2024-03-06T07:36:56.817985+0000","last_peered":"2024-03-06T07:36:56.817985+0000","last_clean":"2024-03-06T07:36:56.817985+0000","last_became_active":"2024-03-06T07:36:55.812994+0000","last_became_peered":"2024-03-06T07:36:55.812994+0000","last_unstale":"2024-03-06T07:36:56.817985+0000","last_undegraded":"2024-03-06T07:36:56.817985+0000","last_fullsized":"2024-03-06T07:36:56.817985+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:48.462774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.826846+0000","last_change":"2024-03-06T07:36:54.820177+0000","last_active":"2024-03-06T07:36:56.826846+0000","last_peered":"2024-03-06T07:36:56.826846+0000","last_clean":"2024-03-06T07:36:56.826846+0000","last_became_active":"2024-03-06T07:36:54.819874+0000","last_became_peered":"2024-03-06T07:36:54.819874+0000","last_unstale":"2024-03-06T07:36:56.826846+0000","last_undegraded":"2024-03-06T07:36:56.826846+0000","last_fullsized":"2024-03-06T07:36:56.826846+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:20.368958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808331+0000","last_change":"2024-03-06T07:36:55.811642+0000","last_active":"2024-03-06T07:36:56.808331+0000","last_peered":"2024-03-06T07:36:56.808331+0000","last_clean":"2024-03-06T07:36:56.808331+0000","last_became_active":"2024-03-06T07:36:55.811365+0000","last_became_peered":"2024-03-06T07:36:55.811365+0000","last_unstale":"2024-03-06T07:36:56.808331+0000","last_undegraded":"2024-03-06T07:36:56.808331+0000","last_fullsized":"2024-03-06T07:36:56.808331+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:36:30.187315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808338+0000","last_change":"2024-03-06T07:36:54.811339+0000","last_active":"2024-03-06T07:36:56.808338+0000","last_peered":"2024-03-06T07:36:56.808338+0000","last_clean":"2024-03-06T07:36:56.808338+0000","last_became_active":"2024-03-06T07:36:54.811058+0000","last_became_peered":"2024-03-06T07:36:54.811058+0000","last_unstale":"2024-03-06T07:36:56.808338+0000","last_undegraded":"2024-03-06T07:36:56.808338+0000","last_fullsized":"2024-03-06T07:36:56.808338+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:12.091940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852043+0000","last_change":"2024-03-06T07:36:55.824144+0000","last_active":"2024-03-06T07:36:56.852043+0000","last_peered":"2024-03-06T07:36:56.852043+0000","last_clean":"2024-03-06T07:36:56.852043+0000","last_became_active":"2024-03-06T07:36:55.823770+0000","last_became_peered":"2024-03-06T07:36:55.823770+0000","last_unstale":"2024-03-06T07:36:56.852043+0000","last_undegraded":"2024-03-06T07:36:56.852043+0000","last_fullsized":"2024-03-06T07:36:56.852043+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:49.150457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854381+0000","last_change":"2024-03-06T07:36:54.821841+0000","last_active":"2024-03-06T07:36:56.854381+0000","last_peered":"2024-03-06T07:36:56.854381+0000","last_clean":"2024-03-06T07:36:56.854381+0000","last_became_active":"2024-03-06T07:36:54.821374+0000","last_became_peered":"2024-03-06T07:36:54.821374+0000","last_unstale":"2024-03-06T07:36:56.854381+0000","last_undegraded":"2024-03-06T07:36:56.854381+0000","last_fullsized":"2024-03-06T07:36:56.854381+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:48.134472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854069+0000","last_change":"2024-03-06T07:36:55.820122+0000","last_active":"2024-03-06T07:36:56.854069+0000","last_peered":"2024-03-06T07:36:56.854069+0000","last_clean":"2024-03-06T07:36:56.854069+0000","last_became_active":"2024-03-06T07:36:55.819833+0000","last_became_peered":"2024-03-06T07:36:55.819833+0000","last_unstale":"2024-03-06T07:36:56.854069+0000","last_undegraded":"2024-03-06T07:36:56.854069+0000","last_fullsized":"2024-03-06T07:36:56.854069+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.934649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853127+0000","last_change":"2024-03-06T07:36:54.821191+0000","last_active":"2024-03-06T07:36:56.853127+0000","last_peered":"2024-03-06T07:36:56.853127+0000","last_clean":"2024-03-06T07:36:56.853127+0000","last_became_active":"2024-03-06T07:36:54.820898+0000","last_became_peered":"2024-03-06T07:36:54.820898+0000","last_unstale":"2024-03-06T07:36:56.853127+0000","last_undegraded":"2024-03-06T07:36:56.853127+0000","last_fullsized":"2024-03-06T07:36:56.853127+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:00.979284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815258+0000","last_change":"2024-03-06T07:36:55.813910+0000","last_active":"2024-03-06T07:36:56.815258+0000","last_peered":"2024-03-06T07:36:56.815258+0000","last_clean":"2024-03-06T07:36:56.815258+0000","last_became_active":"2024-03-06T07:36:55.813195+0000","last_became_peered":"2024-03-06T07:36:55.813195+0000","last_unstale":"2024-03-06T07:36:56.815258+0000","last_undegraded":"2024-03-06T07:36:56.815258+0000","last_fullsized":"2024-03-06T07:36:56.815258+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:39.932539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"126'3","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853599+0000","last_change":"2024-03-06T07:36:54.815439+0000","last_active":"2024-03-06T07:36:56.853599+0000","last_peered":"2024-03-06T07:36:56.853599+0000","last_clean":"2024-03-06T07:36:56.853599+0000","last_became_active":"2024-03-06T07:36:54.814416+0000","last_became_peered":"2024-03-06T07:36:54.814416+0000","last_unstale":"2024-03-06T07:36:56.853599+0000","last_undegraded":"2024-03-06T07:36:56.853599+0000","last_fullsized":"2024-03-06T07:36:56.853599+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:41.671260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.816774+0000","last_change":"2024-03-06T07:36:55.798551+0000","last_active":"2024-03-06T07:36:56.816774+0000","last_peered":"2024-03-06T07:36:56.816774+0000","last_clean":"2024-03-06T07:36:56.816774+0000","last_became_active":"2024-03-06T07:36:55.797952+0000","last_became_peered":"2024-03-06T07:36:55.797952+0000","last_unstale":"2024-03-06T07:36:56.816774+0000","last_undegraded":"2024-03-06T07:36:56.816774+0000","last_fullsized":"2024-03-06T07:36:56.816774+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:23.054410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"126'6","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818621+0000","last_change":"2024-03-06T07:36:54.816635+0000","last_active":"2024-03-06T07:36:56.818621+0000","last_peered":"2024-03-06T07:36:56.818621+0000","last_clean":"2024-03-06T07:36:56.818621+0000","last_became_active":"2024-03-06T07:36:54.816198+0000","last_became_peered":"2024-03-06T07:36:54.816198+0000","last_unstale":"2024-03-06T07:36:56.818621+0000","last_undegraded":"2024-03-06T07:36:56.818621+0000","last_fullsized":"2024-03-06T07:36:56.818621+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:20.368958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851742+0000","last_change":"2024-03-06T07:36:55.817952+0000","last_active":"2024-03-06T07:36:56.851742+0000","last_peered":"2024-03-06T07:36:56.851742+0000","last_clean":"2024-03-06T07:36:56.851742+0000","last_became_active":"2024-03-06T07:36:55.817608+0000","last_became_peered":"2024-03-06T07:36:55.817608+0000","last_unstale":"2024-03-06T07:36:56.851742+0000","last_undegraded":"2024-03-06T07:36:56.851742+0000","last_fullsized":"2024-03-06T07:36:56.851742+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:45.196460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851642+0000","last_change":"2024-03-06T07:36:54.821988+0000","last_active":"2024-03-06T07:36:56.851642+0000","last_peered":"2024-03-06T07:36:56.851642+0000","last_clean":"2024-03-06T07:36:56.851642+0000","last_became_active":"2024-03-06T07:36:54.821659+0000","last_became_peered":"2024-03-06T07:36:54.821659+0000","last_unstale":"2024-03-06T07:36:56.851642+0000","last_undegraded":"2024-03-06T07:36:56.851642+0000","last_fullsized":"2024-03-06T07:36:56.851642+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:54.420552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808222+0000","last_change":"2024-03-06T07:36:55.801055+0000","last_active":"2024-03-06T07:36:56.808222+0000","last_peered":"2024-03-06T07:36:56.808222+0000","last_clean":"2024-03-06T07:36:56.808222+0000","last_became_active":"2024-03-06T07:36:55.800429+0000","last_became_peered":"2024-03-06T07:36:55.800429+0000","last_unstale":"2024-03-06T07:36:56.808222+0000","last_undegraded":"2024-03-06T07:36:56.808222+0000","last_fullsized":"2024-03-06T07:36:56.808222+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.934649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808464+0000","last_change":"2024-03-06T07:36:54.814157+0000","last_active":"2024-03-06T07:36:56.808464+0000","last_peered":"2024-03-06T07:36:56.808464+0000","last_clean":"2024-03-06T07:36:56.808464+0000","last_became_active":"2024-03-06T07:36:54.813869+0000","last_became_peered":"2024-03-06T07:36:54.813869+0000","last_unstale":"2024-03-06T07:36:56.808464+0000","last_undegraded":"2024-03-06T07:36:56.808464+0000","last_fullsized":"2024-03-06T07:36:56.808464+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:13.632551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.817739+0000","last_change":"2024-03-06T07:36:55.809823+0000","last_active":"2024-03-06T07:36:56.817739+0000","last_peered":"2024-03-06T07:36:56.817739+0000","last_clean":"2024-03-06T07:36:56.817739+0000","last_became_active":"2024-03-06T07:36:55.808996+0000","last_became_peered":"2024-03-06T07:36:55.808996+0000","last_unstale":"2024-03-06T07:36:56.817739+0000","last_undegraded":"2024-03-06T07:36:56.817739+0000","last_fullsized":"2024-03-06T07:36:56.817739+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:37.887549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"126'8","reported_seq":22,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.818474+0000","last_change":"2024-03-06T07:36:54.811756+0000","last_active":"2024-03-06T07:36:56.818474+0000","last_peered":"2024-03-06T07:36:56.818474+0000","last_clean":"2024-03-06T07:36:56.818474+0000","last_became_active":"2024-03-06T07:36:54.811279+0000","last_became_peered":"2024-03-06T07:36:54.811279+0000","last_unstale":"2024-03-06T07:36:56.818474+0000","last_undegraded":"2024-03-06T07:36:56.818474+0000","last_fullsized":"2024-03-06T07:36:56.818474+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:50.980550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.819297+0000","last_change":"2024-03-06T07:36:55.819115+0000","last_active":"2024-03-06T07:36:56.819297+0000","last_peered":"2024-03-06T07:36:56.819297+0000","last_clean":"2024-03-06T07:36:56.819297+0000","last_became_active":"2024-03-06T07:36:55.818806+0000","last_became_peered":"2024-03-06T07:36:55.818806+0000","last_unstale":"2024-03-06T07:36:56.819297+0000","last_undegraded":"2024-03-06T07:36:56.819297+0000","last_fullsized":"2024-03-06T07:36:56.819297+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:49.178432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854399+0000","last_change":"2024-03-06T07:36:54.821213+0000","last_active":"2024-03-06T07:36:56.854399+0000","last_peered":"2024-03-06T07:36:56.854399+0000","last_clean":"2024-03-06T07:36:56.854399+0000","last_became_active":"2024-03-06T07:36:54.818860+0000","last_became_peered":"2024-03-06T07:36:54.818860+0000","last_unstale":"2024-03-06T07:36:56.854399+0000","last_undegraded":"2024-03-06T07:36:56.854399+0000","last_fullsized":"2024-03-06T07:36:56.854399+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:18.409129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.854171+0000","last_change":"2024-03-06T07:36:55.810303+0000","last_active":"2024-03-06T07:36:56.854171+0000","last_peered":"2024-03-06T07:36:56.854171+0000","last_clean":"2024-03-06T07:36:56.854171+0000","last_became_active":"2024-03-06T07:36:55.810024+0000","last_became_peered":"2024-03-06T07:36:55.810024+0000","last_unstale":"2024-03-06T07:36:56.854171+0000","last_undegraded":"2024-03-06T07:36:56.854171+0000","last_fullsized":"2024-03-06T07:36:56.854171+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:17.107118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852970+0000","last_change":"2024-03-06T07:36:54.816875+0000","last_active":"2024-03-06T07:36:56.852970+0000","last_peered":"2024-03-06T07:36:56.852970+0000","last_clean":"2024-03-06T07:36:56.852970+0000","last_became_active":"2024-03-06T07:36:54.816612+0000","last_became_peered":"2024-03-06T07:36:54.816612+0000","last_unstale":"2024-03-06T07:36:56.852970+0000","last_undegraded":"2024-03-06T07:36:56.852970+0000","last_fullsized":"2024-03-06T07:36:56.852970+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:36.068638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853001+0000","last_change":"2024-03-06T07:36:55.817227+0000","last_active":"2024-03-06T07:36:56.853001+0000","last_peered":"2024-03-06T07:36:56.853001+0000","last_clean":"2024-03-06T07:36:56.853001+0000","last_became_active":"2024-03-06T07:36:55.815247+0000","last_became_peered":"2024-03-06T07:36:55.815247+0000","last_unstale":"2024-03-06T07:36:56.853001+0000","last_undegraded":"2024-03-06T07:36:56.853001+0000","last_fullsized":"2024-03-06T07:36:56.853001+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:23.054410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827883+0000","last_change":"2024-03-06T07:36:54.813308+0000","last_active":"2024-03-06T07:36:56.827883+0000","last_peered":"2024-03-06T07:36:56.827883+0000","last_clean":"2024-03-06T07:36:56.827883+0000","last_became_active":"2024-03-06T07:36:54.812993+0000","last_became_peered":"2024-03-06T07:36:54.812993+0000","last_unstale":"2024-03-06T07:36:56.827883+0000","last_undegraded":"2024-03-06T07:36:56.827883+0000","last_fullsized":"2024-03-06T07:36:56.827883+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:39.556105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809331+0000","last_change":"2024-03-06T07:36:55.812512+0000","last_active":"2024-03-06T07:36:56.809331+0000","last_peered":"2024-03-06T07:36:56.809331+0000","last_clean":"2024-03-06T07:36:56.809331+0000","last_became_active":"2024-03-06T07:36:55.811974+0000","last_became_peered":"2024-03-06T07:36:55.811974+0000","last_unstale":"2024-03-06T07:36:56.809331+0000","last_undegraded":"2024-03-06T07:36:56.809331+0000","last_fullsized":"2024-03-06T07:36:56.809331+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:51.996535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853848+0000","last_change":"2024-03-06T07:36:54.814933+0000","last_active":"2024-03-06T07:36:56.853848+0000","last_peered":"2024-03-06T07:36:56.853848+0000","last_clean":"2024-03-06T07:36:56.853848+0000","last_became_active":"2024-03-06T07:36:54.814215+0000","last_became_peered":"2024-03-06T07:36:54.814215+0000","last_unstale":"2024-03-06T07:36:56.853848+0000","last_undegraded":"2024-03-06T07:36:56.853848+0000","last_fullsized":"2024-03-06T07:36:56.853848+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:17.391576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851556+0000","last_change":"2024-03-06T07:36:55.825444+0000","last_active":"2024-03-06T07:36:56.851556+0000","last_peered":"2024-03-06T07:36:56.851556+0000","last_clean":"2024-03-06T07:36:56.851556+0000","last_became_active":"2024-03-06T07:36:55.825117+0000","last_became_peered":"2024-03-06T07:36:55.825117+0000","last_unstale":"2024-03-06T07:36:56.851556+0000","last_undegraded":"2024-03-06T07:36:56.851556+0000","last_fullsized":"2024-03-06T07:36:56.851556+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:42.687245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853838+0000","last_change":"2024-03-06T07:36:54.812962+0000","last_active":"2024-03-06T07:36:56.853838+0000","last_peered":"2024-03-06T07:36:56.853838+0000","last_clean":"2024-03-06T07:36:56.853838+0000","last_became_active":"2024-03-06T07:36:54.812032+0000","last_became_peered":"2024-03-06T07:36:54.812032+0000","last_unstale":"2024-03-06T07:36:56.853838+0000","last_undegraded":"2024-03-06T07:36:56.853838+0000","last_fullsized":"2024-03-06T07:36:56.853838+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.381139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.827935+0000","last_change":"2024-03-06T07:36:55.805426+0000","last_active":"2024-03-06T07:36:56.827935+0000","last_peered":"2024-03-06T07:36:56.827935+0000","last_clean":"2024-03-06T07:36:56.827935+0000","last_became_active":"2024-03-06T07:36:55.805012+0000","last_became_peered":"2024-03-06T07:36:55.805012+0000","last_unstale":"2024-03-06T07:36:56.827935+0000","last_undegraded":"2024-03-06T07:36:56.827935+0000","last_fullsized":"2024-03-06T07:36:56.827935+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:38.893556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853247+0000","last_change":"2024-03-06T07:36:54.818382+0000","last_active":"2024-03-06T07:36:56.853247+0000","last_peered":"2024-03-06T07:36:56.853247+0000","last_clean":"2024-03-06T07:36:56.853247+0000","last_became_active":"2024-03-06T07:36:54.815856+0000","last_became_peered":"2024-03-06T07:36:54.815856+0000","last_unstale":"2024-03-06T07:36:56.853247+0000","last_undegraded":"2024-03-06T07:36:56.853247+0000","last_fullsized":"2024-03-06T07:36:56.853247+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:03.460516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808251+0000","last_change":"2024-03-06T07:36:55.806912+0000","last_active":"2024-03-06T07:36:56.808251+0000","last_peered":"2024-03-06T07:36:56.808251+0000","last_clean":"2024-03-06T07:36:56.808251+0000","last_became_active":"2024-03-06T07:36:55.806602+0000","last_became_peered":"2024-03-06T07:36:55.806602+0000","last_unstale":"2024-03-06T07:36:56.808251+0000","last_undegraded":"2024-03-06T07:36:56.808251+0000","last_fullsized":"2024-03-06T07:36:56.808251+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:17.107118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852061+0000","last_change":"2024-03-06T07:36:54.817853+0000","last_active":"2024-03-06T07:36:56.852061+0000","last_peered":"2024-03-06T07:36:56.852061+0000","last_clean":"2024-03-06T07:36:56.852061+0000","last_became_active":"2024-03-06T07:36:54.816265+0000","last_became_peered":"2024-03-06T07:36:54.816265+0000","last_unstale":"2024-03-06T07:36:56.852061+0000","last_undegraded":"2024-03-06T07:36:56.852061+0000","last_fullsized":"2024-03-06T07:36:56.852061+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:41.119076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814537+0000","last_change":"2024-03-06T07:36:55.819401+0000","last_active":"2024-03-06T07:36:56.814537+0000","last_peered":"2024-03-06T07:36:56.814537+0000","last_clean":"2024-03-06T07:36:56.814537+0000","last_became_active":"2024-03-06T07:36:55.819094+0000","last_became_peered":"2024-03-06T07:36:55.819094+0000","last_unstale":"2024-03-06T07:36:56.814537+0000","last_undegraded":"2024-03-06T07:36:56.814537+0000","last_fullsized":"2024-03-06T07:36:56.814537+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:58.738984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851301+0000","last_change":"2024-03-06T07:36:54.815755+0000","last_active":"2024-03-06T07:36:56.851301+0000","last_peered":"2024-03-06T07:36:56.851301+0000","last_clean":"2024-03-06T07:36:56.851301+0000","last_became_active":"2024-03-06T07:36:54.815407+0000","last_became_peered":"2024-03-06T07:36:54.815407+0000","last_unstale":"2024-03-06T07:36:56.851301+0000","last_undegraded":"2024-03-06T07:36:56.851301+0000","last_fullsized":"2024-03-06T07:36:56.851301+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:58.436585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.828384+0000","last_change":"2024-03-06T07:36:55.814206+0000","last_active":"2024-03-06T07:36:56.828384+0000","last_peered":"2024-03-06T07:36:56.828384+0000","last_clean":"2024-03-06T07:36:56.828384+0000","last_became_active":"2024-03-06T07:36:55.811889+0000","last_became_peered":"2024-03-06T07:36:55.811889+0000","last_unstale":"2024-03-06T07:36:56.828384+0000","last_undegraded":"2024-03-06T07:36:56.828384+0000","last_fullsized":"2024-03-06T07:36:56.828384+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:49.178432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852698+0000","last_change":"2024-03-06T07:36:54.817387+0000","last_active":"2024-03-06T07:36:56.852698+0000","last_peered":"2024-03-06T07:36:56.852698+0000","last_clean":"2024-03-06T07:36:56.852698+0000","last_became_active":"2024-03-06T07:36:54.817095+0000","last_became_peered":"2024-03-06T07:36:54.817095+0000","last_unstale":"2024-03-06T07:36:56.852698+0000","last_undegraded":"2024-03-06T07:36:56.852698+0000","last_fullsized":"2024-03-06T07:36:56.852698+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:28.085612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852580+0000","last_change":"2024-03-06T07:36:55.822657+0000","last_active":"2024-03-06T07:36:56.852580+0000","last_peered":"2024-03-06T07:36:56.852580+0000","last_clean":"2024-03-06T07:36:56.852580+0000","last_became_active":"2024-03-06T07:36:55.822354+0000","last_became_peered":"2024-03-06T07:36:55.822354+0000","last_unstale":"2024-03-06T07:36:56.852580+0000","last_undegraded":"2024-03-06T07:36:56.852580+0000","last_fullsized":"2024-03-06T07:36:56.852580+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:30.258955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"127'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.825701+0000","last_change":"2024-03-06T07:36:54.823648+0000","last_active":"2024-03-06T07:36:56.825701+0000","last_peered":"2024-03-06T07:36:56.825701+0000","last_clean":"2024-03-06T07:36:56.825701+0000","last_became_active":"2024-03-06T07:36:54.823243+0000","last_became_peered":"2024-03-06T07:36:54.823243+0000","last_unstale":"2024-03-06T07:36:56.825701+0000","last_undegraded":"2024-03-06T07:36:56.825701+0000","last_fullsized":"2024-03-06T07:36:56.825701+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:05.495029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809040+0000","last_change":"2024-03-06T07:36:55.803322+0000","last_active":"2024-03-06T07:36:56.809040+0000","last_peered":"2024-03-06T07:36:56.809040+0000","last_clean":"2024-03-06T07:36:56.809040+0000","last_became_active":"2024-03-06T07:36:55.803023+0000","last_became_peered":"2024-03-06T07:36:55.803023+0000","last_unstale":"2024-03-06T07:36:56.809040+0000","last_undegraded":"2024-03-06T07:36:56.809040+0000","last_fullsized":"2024-03-06T07:36:56.809040+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:47.105660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.809663+0000","last_change":"2024-03-06T07:36:54.810145+0000","last_active":"2024-03-06T07:36:56.809663+0000","last_peered":"2024-03-06T07:36:56.809663+0000","last_clean":"2024-03-06T07:36:56.809663+0000","last_became_active":"2024-03-06T07:36:54.809790+0000","last_became_peered":"2024-03-06T07:36:54.809790+0000","last_unstale":"2024-03-06T07:36:56.809663+0000","last_undegraded":"2024-03-06T07:36:56.809663+0000","last_fullsized":"2024-03-06T07:36:56.809663+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:30.036164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.828142+0000","last_change":"2024-03-06T07:36:55.820896+0000","last_active":"2024-03-06T07:36:56.828142+0000","last_peered":"2024-03-06T07:36:56.828142+0000","last_clean":"2024-03-06T07:36:56.828142+0000","last_became_active":"2024-03-06T07:36:55.820210+0000","last_became_peered":"2024-03-06T07:36:55.820210+0000","last_unstale":"2024-03-06T07:36:56.828142+0000","last_undegraded":"2024-03-06T07:36:56.828142+0000","last_fullsized":"2024-03-06T07:36:56.828142+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:26.782531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852873+0000","last_change":"2024-03-06T07:36:54.802373+0000","last_active":"2024-03-06T07:36:56.852873+0000","last_peered":"2024-03-06T07:36:56.852873+0000","last_clean":"2024-03-06T07:36:56.852873+0000","last_became_active":"2024-03-06T07:36:54.802084+0000","last_became_peered":"2024-03-06T07:36:54.802084+0000","last_unstale":"2024-03-06T07:36:56.852873+0000","last_undegraded":"2024-03-06T07:36:56.852873+0000","last_fullsized":"2024-03-06T07:36:56.852873+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:07.899036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814902+0000","last_change":"2024-03-06T07:36:55.810310+0000","last_active":"2024-03-06T07:36:56.814902+0000","last_peered":"2024-03-06T07:36:56.814902+0000","last_clean":"2024-03-06T07:36:56.814902+0000","last_became_active":"2024-03-06T07:36:55.810021+0000","last_became_peered":"2024-03-06T07:36:55.810021+0000","last_unstale":"2024-03-06T07:36:56.814902+0000","last_undegraded":"2024-03-06T07:36:56.814902+0000","last_fullsized":"2024-03-06T07:36:56.814902+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:39.914436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808605+0000","last_change":"2024-03-06T07:36:54.814823+0000","last_active":"2024-03-06T07:36:56.808605+0000","last_peered":"2024-03-06T07:36:56.808605+0000","last_clean":"2024-03-06T07:36:56.808605+0000","last_became_active":"2024-03-06T07:36:54.813881+0000","last_became_peered":"2024-03-06T07:36:54.813881+0000","last_unstale":"2024-03-06T07:36:56.808605+0000","last_undegraded":"2024-03-06T07:36:56.808605+0000","last_fullsized":"2024-03-06T07:36:56.808605+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:41.671260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.853850+0000","last_change":"2024-03-06T07:36:55.825766+0000","last_active":"2024-03-06T07:36:56.853850+0000","last_peered":"2024-03-06T07:36:56.853850+0000","last_clean":"2024-03-06T07:36:56.853850+0000","last_became_active":"2024-03-06T07:36:55.825443+0000","last_became_peered":"2024-03-06T07:36:55.825443+0000","last_unstale":"2024-03-06T07:36:56.853850+0000","last_undegraded":"2024-03-06T07:36:56.853850+0000","last_fullsized":"2024-03-06T07:36:56.853850+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:42.572203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815175+0000","last_change":"2024-03-06T07:36:54.817857+0000","last_active":"2024-03-06T07:36:56.815175+0000","last_peered":"2024-03-06T07:36:56.815175+0000","last_clean":"2024-03-06T07:36:56.815175+0000","last_became_active":"2024-03-06T07:36:54.816332+0000","last_became_peered":"2024-03-06T07:36:54.816332+0000","last_unstale":"2024-03-06T07:36:56.815175+0000","last_undegraded":"2024-03-06T07:36:56.815175+0000","last_fullsized":"2024-03-06T07:36:56.815175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:53.312572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851895+0000","last_change":"2024-03-06T07:36:55.803908+0000","last_active":"2024-03-06T07:36:56.851895+0000","last_peered":"2024-03-06T07:36:56.851895+0000","last_clean":"2024-03-06T07:36:56.851895+0000","last_became_active":"2024-03-06T07:36:55.803601+0000","last_became_peered":"2024-03-06T07:36:55.803601+0000","last_unstale":"2024-03-06T07:36:56.851895+0000","last_undegraded":"2024-03-06T07:36:56.851895+0000","last_fullsized":"2024-03-06T07:36:56.851895+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:57.807342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.852167+0000","last_change":"2024-03-06T07:36:54.820560+0000","last_active":"2024-03-06T07:36:56.852167+0000","last_peered":"2024-03-06T07:36:56.852167+0000","last_clean":"2024-03-06T07:36:56.852167+0000","last_became_active":"2024-03-06T07:36:54.820191+0000","last_became_peered":"2024-03-06T07:36:54.820191+0000","last_unstale":"2024-03-06T07:36:56.852167+0000","last_undegraded":"2024-03-06T07:36:56.852167+0000","last_fullsized":"2024-03-06T07:36:56.852167+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:25.557647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.808500+0000","last_change":"2024-03-06T07:36:55.804986+0000","last_active":"2024-03-06T07:36:56.808500+0000","last_peered":"2024-03-06T07:36:56.808500+0000","last_clean":"2024-03-06T07:36:56.808500+0000","last_became_active":"2024-03-06T07:36:55.801747+0000","last_became_peered":"2024-03-06T07:36:55.801747+0000","last_unstale":"2024-03-06T07:36:56.808500+0000","last_undegraded":"2024-03-06T07:36:56.808500+0000","last_fullsized":"2024-03-06T07:36:56.808500+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:09.270568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.851419+0000","last_change":"2024-03-06T07:36:54.813927+0000","last_active":"2024-03-06T07:36:56.851419+0000","last_peered":"2024-03-06T07:36:56.851419+0000","last_clean":"2024-03-06T07:36:56.851419+0000","last_became_active":"2024-03-06T07:36:54.813612+0000","last_became_peered":"2024-03-06T07:36:54.813612+0000","last_unstale":"2024-03-06T07:36:56.851419+0000","last_undegraded":"2024-03-06T07:36:56.851419+0000","last_fullsized":"2024-03-06T07:36:56.851419+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:22.038424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814439+0000","last_change":"2024-03-06T07:36:55.816711+0000","last_active":"2024-03-06T07:36:56.814439+0000","last_peered":"2024-03-06T07:36:56.814439+0000","last_clean":"2024-03-06T07:36:56.814439+0000","last_became_active":"2024-03-06T07:36:55.816402+0000","last_became_peered":"2024-03-06T07:36:55.816402+0000","last_unstale":"2024-03-06T07:36:56.814439+0000","last_undegraded":"2024-03-06T07:36:56.814439+0000","last_fullsized":"2024-03-06T07:36:56.814439+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:54.765678+0000","last_clean_scrub_stamp":"2024-03-06T07:36:54.765678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:47.806644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.815419+0000","last_change":"2024-03-06T07:36:54.818441+0000","last_active":"2024-03-06T07:36:56.815419+0000","last_peered":"2024-03-06T07:36:56.815419+0000","last_clean":"2024-03-06T07:36:56.815419+0000","last_became_active":"2024-03-06T07:36:54.818125+0000","last_became_peered":"2024-03-06T07:36:54.818125+0000","last_unstale":"2024-03-06T07:36:56.815419+0000","last_undegraded":"2024-03-06T07:36:56.815419+0000","last_fullsized":"2024-03-06T07:36:56.815419+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:36:53.749693+0000","last_clean_scrub_stamp":"2024-03-06T07:36:53.749693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:01.259801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":371,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:36:56.814096+0000","last_change":"2024-03-06T07:28:56.458889+0000","last_active":"2024-03-06T07:36:56.814096+0000","last_peered":"2024-03-06T07:36:56.814096+0000","last_clean":"2024-03-06T07:36:56.814096+0000","last_became_active":"2024-03-06T07:28:56.456395+0000","last_became_peered":"2024-03-06T07:28:56.456395+0000","last_unstale":"2024-03-06T07:36:56.814096+0000","last_undegraded":"2024-03-06T07:36:56.814096+0000","last_fullsized":"2024-03-06T07:36:56.814096+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:55.000107+0000","last_clean_scrub_stamp":"2024-03-06T07:28:55.000107+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.185691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37496,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334344,"statfs":{"total":96636764160,"available":96598368256,"internally_reserved":0,"allocated":2203648,"data_stored":1282079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1597440,"data_stored":691073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334940,"statfs":{"total":96636764160,"available":96598978560,"internally_reserved":0,"allocated":1593344,"data_stored":689573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1589248,"data_stored":689531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1593344,"data_stored":689621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574947,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41064,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330776,"statfs":{"total":96636764160,"available":96594714624,"internally_reserved":0,"allocated":1593344,"data_stored":690077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1585152,"data_stored":689461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574948,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41636,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330204,"statfs":{"total":96636764160,"available":96594128896,"internally_reserved":0,"allocated":2179072,"data_stored":1279829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:37:00.848 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:37:00.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:37:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.112+0000 7f00ed200640 1 Processor -- start 2024-03-06T07:37:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.112+0000 7f00ed200640 1 -- start start 2024-03-06T07:37:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00ed200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00e806b6c0 0x7f00e80728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00ed200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f00e80b4120 0x7f00e80b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00ed200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e80d1a10 con 0x7f00e806f690 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00ed200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e8072f70 con 0x7f00e806b6c0 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00ed200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e80d1450 con 0x7f00e80b4120 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00e6a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00e806b6c0 0x7f00e80728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00e6a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00e806b6c0 0x7f00e80728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33400/0 (socket says 172.21.15.32:33400) 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00e6a00640 1 -- 172.21.15.32:0/3345832916 learned_addr learned my addr 172.21.15.32:0/3345832916 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.114+0000 7f00e7e00640 1 --2- 172.21.15.32:0/3345832916 >> v2:172.21.15.39:3300/0 conn(0x7f00e80b4120 0x7f00e80b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.115+0000 7f00e6a00640 1 -- 172.21.15.32:0/3345832916 >> v2:172.21.15.39:3300/0 conn(0x7f00e80b4120 msgr2=0x7f00e80b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.115+0000 7f00e6a00640 1 --2- 172.21.15.32:0/3345832916 >> v2:172.21.15.39:3300/0 conn(0x7f00e80b4120 0x7f00e80b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.115+0000 7f00e6a00640 1 -- 172.21.15.32:0/3345832916 >> v1:172.21.15.39:6789/0 conn(0x7f00e806f690 legacy=0x7f00e806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.115+0000 7f00e6a00640 1 -- 172.21.15.32:0/3345832916 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e80d1730 con 0x7f00e806b6c0 2024-03-06T07:37:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.116+0000 7f00e6a00640 1 --2- 172.21.15.32:0/3345832916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806b6c0 0x7f00e80728b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f00e80720c0 tx=0x7f00d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd69430d9df5b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.116+0000 7f00e6000640 1 -- 172.21.15.32:0/3345832916 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d402e070 con 0x7f00e806b6c0 2024-03-06T07:37:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.116+0000 7f00e6000640 1 -- 172.21.15.32:0/3345832916 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d4037070 con 0x7f00e806b6c0 2024-03-06T07:37:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.116+0000 7f00e6000640 1 -- 172.21.15.32:0/3345832916 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4031a00 con 0x7f00e806b6c0 2024-03-06T07:37:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 -- 172.21.15.32:0/3345832916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806b6c0 msgr2=0x7f00e80728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 --2- 172.21.15.32:0/3345832916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806b6c0 0x7f00e80728b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f00e80720c0 tx=0x7f00d402b880 comp rx=0 tx=0).stop 2024-03-06T07:37:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 -- 172.21.15.32:0/3345832916 shutdown_connections 2024-03-06T07:37:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 --2- 172.21.15.32:0/3345832916 >> v2:172.21.15.39:3300/0 conn(0x7f00e80b4120 0x7f00e80b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 --2- 172.21.15.32:0/3345832916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806b6c0 0x7f00e80728b0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 -- 172.21.15.32:0/3345832916 >> 172.21.15.32:0/3345832916 conn(0x7f00e806c0c0 msgr2=0x7f00e806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 -- 172.21.15.32:0/3345832916 shutdown_connections 2024-03-06T07:37:01.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.117+0000 7f00ed200640 1 -- 172.21.15.32:0/3345832916 wait complete. 2024-03-06T07:37:01.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.119+0000 7f00ed200640 1 Processor -- start 2024-03-06T07:37:01.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.119+0000 7f00ed200640 1 -- start start 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00ed200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00e806b6c0 0x7f00e8160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00ed200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 0x7f00e813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00ed200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00e80b4120 0x7f00e813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00ed200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e806dd00 con 0x7f00e806f690 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00ed200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e806fa70 con 0x7f00e80b4120 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00ed200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e806de80 con 0x7f00e806b6c0 2024-03-06T07:37:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00e6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 0x7f00e813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00e6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 0x7f00e813d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33406/0 (socket says 172.21.15.32:33406) 2024-03-06T07:37:01.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00e6a00640 1 -- 172.21.15.32:0/1989542740 learned_addr learned my addr 172.21.15.32:0/1989542740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00e7e00640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00e80b4120 0x7f00e813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00e7400640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00e806b6c0 0x7f00e8160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.120+0000 7f00e6a00640 1 -- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00e806b6c0 msgr2=0x7f00e8160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e6a00640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00e806b6c0 0x7f00e8160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e6a00640 1 -- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00e80b4120 msgr2=0x7f00e813d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e6a00640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00e80b4120 0x7f00e813d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e6a00640 1 -- 172.21.15.32:0/1989542740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e813e270 con 0x7f00e806f690 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e7e00640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00e80b4120 0x7f00e813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e7400640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00e806b6c0 0x7f00e8160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00e6a00640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 0x7f00e813d4b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f00d402b8c0 tx=0x7f00d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4032670 con 0x7f00e806f690 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d4032830 con 0x7f00e806f690 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4032c20 con 0x7f00e806f690 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.121+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e813e550 con 0x7f00e806f690 2024-03-06T07:37:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.122+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e813eb10 con 0x7f00e806f690 2024-03-06T07:37:01.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.123+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e804f330 con 0x7f00e806f690 2024-03-06T07:37:01.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.124+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00d404c060 con 0x7f00e806f690 2024-03-06T07:37:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.125+0000 7f00df400640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00c806b4d0 0x7f00c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.125+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00d40abd40 con 0x7f00e806f690 2024-03-06T07:37:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.125+0000 7f00e7400640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00c806b4d0 0x7f00c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.126+0000 7f00e7400640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00c806b4d0 0x7f00c806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f00d8003eb0 tx=0x7f00d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.134+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00d4040070 con 0x7f00e806f690 2024-03-06T07:37:01.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.409+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f00e806e460 con 0x7f00e806f690 2024-03-06T07:37:01.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.415+0000 7f00df400640 1 -- 172.21.15.32:0/1989542740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f00d407ab20 con 0x7f00e806f690 2024-03-06T07:37:01.416 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-06T07:37:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQD5GuhlX7IGJBAAirqMd5fNXG/fx1uZEfkAEg== 2024-03-06T07:37:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-03-06T07:37:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-03-06T07:37:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:37:01.417 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00c806b4d0 msgr2=0x7f00c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00c806b4d0 0x7f00c806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f00d8003eb0 tx=0x7f00d8009040 comp rx=0 tx=0).stop 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 msgr2=0x7f00e813d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 0x7f00e813d4b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f00d402b8c0 tx=0x7f00d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 shutdown_connections 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f00c806b4d0 0x7f00c806d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00e80b4120 0x7f00e813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e806f690 0x7f00e813d4b0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 --2- 172.21.15.32:0/1989542740 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f00e806b6c0 0x7f00e8160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.417+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 >> 172.21.15.32:0/1989542740 conn(0x7f00e806c0c0 msgr2=0x7f00e80b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.418+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 shutdown_connections 2024-03-06T07:37:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.418+0000 7f00ed200640 1 -- 172.21.15.32:0/1989542740 wait complete. 2024-03-06T07:37:01.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.640+0000 7fcd99e00640 1 Processor -- start 2024-03-06T07:37:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.640+0000 7fcd99e00640 1 -- start start 2024-03-06T07:37:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd99e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcd9415dec0 0x7fcd940ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd99e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 0x7fcd940b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd99e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940d1120 con 0x7fcd9415e2a0 2024-03-06T07:37:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd99e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd940b0ea0 con 0x7fcd9415dec0 2024-03-06T07:37:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd99e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d1400 con 0x7fcd940cf460 2024-03-06T07:37:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd93e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcd9415e2a0 0x7fcd940ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39812/0 (socket says 172.21.15.32:39812) 2024-03-06T07:37:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd93e00640 1 -- 172.21.15.32:0/62388606 learned_addr learned my addr 172.21.15.32:0/62388606 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd98a00640 1 --2- 172.21.15.32:0/62388606 >> v2:172.21.15.39:3300/0 conn(0x7fcd9415dec0 0x7fcd940ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.642+0000 7fcd99400640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 0x7fcd940b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174125699 0 0) 0x7fcd940d1120 con 0x7fcd9415e2a0 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c003520 con 0x7fcd9415e2a0 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1954785568 0 0) 0x7fcd7c003520 con 0x7fcd9415e2a0 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 msgr2=0x7fcd940b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 0x7fcd940b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 >> v2:172.21.15.39:3300/0 conn(0x7fcd9415dec0 msgr2=0x7fcd940ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 --2- 172.21.15.32:0/62388606 >> v2:172.21.15.39:3300/0 conn(0x7fcd9415dec0 0x7fcd940ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940b0b30 con 0x7fcd9415e2a0 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fcd80001670 con 0x7fcd9415e2a0 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 >> v1:172.21.15.32:6789/0 conn(0x7fcd9415e2a0 legacy=0x7fcd940ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd7c0039b0 0x7fcd7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 0x7fcd7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940b0ea0 con 0x7fcd7c004800 2024-03-06T07:37:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d1400 con 0x7fcd7c0039b0 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c00b9c0 con 0x7fcd940cf460 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.643+0000 7fcd93e00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd98a00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd7c0039b0 0x7fcd7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd93e00640 1 -- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 msgr2=0x7fcd7c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd93e00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940cf460 0x7fcd7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd93e00640 1 -- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd7c0039b0 msgr2=0x7fcd7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd93e00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd7c0039b0 0x7fcd7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd93e00640 1 -- 172.21.15.32:0/62388606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c003520 con 0x7fcd7c004800 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.644+0000 7fcd98a00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd7c0039b0 0x7fcd7c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.645+0000 7fcd93e00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd7c004c80 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fcd80001050 tx=0x7fcd7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9d3490ed5ad6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.645+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80007d70 con 0x7fcd7c004800 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.645+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd80030070 con 0x7fcd7c004800 2024-03-06T07:37:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.645+0000 7fcd93400640 1 -- 172.21.15.32:0/62388606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8002cd80 con 0x7fcd7c004800 2024-03-06T07:37:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99e00640 1 -- 172.21.15.32:0/62388606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 msgr2=0x7fcd7c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99e00640 1 --2- 172.21.15.32:0/62388606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd7c004c80 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fcd80001050 tx=0x7fcd7c005830 comp rx=0 tx=0).stop 2024-03-06T07:37:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99400640 1 -- 172.21.15.32:0/62388606 reap_dead start 2024-03-06T07:37:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99e00640 1 -- 172.21.15.32:0/62388606 shutdown_connections 2024-03-06T07:37:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99e00640 1 -- 172.21.15.32:0/62388606 >> 172.21.15.32:0/62388606 conn(0x7fcd940d3d40 msgr2=0x7fcd940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99e00640 1 -- 172.21.15.32:0/62388606 shutdown_connections 2024-03-06T07:37:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.646+0000 7fcd99e00640 1 -- 172.21.15.32:0/62388606 wait complete. 2024-03-06T07:37:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.648+0000 7fcd99e00640 1 Processor -- start 2024-03-06T07:37:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.648+0000 7fcd99e00640 1 -- start start 2024-03-06T07:37:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd99e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd9414e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd99e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940cd150 0x7fcd940cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd99e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940c7900 0x7fcd940c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd99e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d0ec0 con 0x7fcd7c004800 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd99e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d0d40 con 0x7fcd940cd150 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd99e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940b1330 con 0x7fcd940c7900 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd98a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd9414e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd98a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd9414e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33418/0 (socket says 172.21.15.32:33418) 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.649+0000 7fcd98a00640 1 -- 172.21.15.32:0/2634357583 learned_addr learned my addr 172.21.15.32:0/2634357583 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd99400640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940c7900 0x7fcd940c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd93e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940cd150 0x7fcd940cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd98a00640 1 -- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940c7900 msgr2=0x7fcd940c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd98a00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940c7900 0x7fcd940c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd98a00640 1 -- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940cd150 msgr2=0x7fcd940cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd98a00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940cd150 0x7fcd940cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd98a00640 1 -- 172.21.15.32:0/2634357583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940c86c0 con 0x7fcd7c004800 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd98a00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd9414e0f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fcd880108e0 tx=0x7fcd88010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8800f040 con 0x7fcd7c004800 2024-03-06T07:37:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8800ec20 con 0x7fcd7c004800 2024-03-06T07:37:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd940c89a0 con 0x7fcd7c004800 2024-03-06T07:37:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.650+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8801c400 con 0x7fcd7c004800 2024-03-06T07:37:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.651+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940c8f60 con 0x7fcd7c004800 2024-03-06T07:37:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.652+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404f330 con 0x7fcd7c004800 2024-03-06T07:37:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.653+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd8801c5a0 con 0x7fcd7c004800 2024-03-06T07:37:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.653+0000 7fcd90c00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b4d0 0x7fcd7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.654+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd8808aaa0 con 0x7fcd7c004800 2024-03-06T07:37:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.654+0000 7fcd93e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b4d0 0x7fcd7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.655+0000 7fcd93e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b4d0 0x7fcd7406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcd80034000 tx=0x7fcd80007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.658+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd88018030 con 0x7fcd7c004800 2024-03-06T07:37:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.910+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9406c650 con 0x7fcd7c004800 2024-03-06T07:37:01.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.911+0000 7fcd90c00640 1 -- 172.21.15.32:0/2634357583 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fcd88059890 con 0x7fcd7c004800 2024-03-06T07:37:01.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:01.913 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/4261872692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4261872692},{"type":"v1","addr":"172.21.15.32:6837","nonce":4261872692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":5894,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/304553199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":304553199},{"type":"v1","addr":"172.21.15.39:6835","nonce":304553199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6574,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:36:58.471868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8313},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b4d0 msgr2=0x7fcd7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b4d0 0x7fcd7406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcd80034000 tx=0x7fcd80007000 comp rx=0 tx=0).stop 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 msgr2=0x7fcd9414e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd9414e0f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fcd880108e0 tx=0x7fcd88010da0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 shutdown_connections 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcd7406b4d0 0x7fcd7406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcd940c7900 0x7fcd940c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd940cd150 0x7fcd940cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 --2- 172.21.15.32:0/2634357583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd7c004800 0x7fcd9414e0f0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 >> 172.21.15.32:0/2634357583 conn(0x7fcd940d3d40 msgr2=0x7fcd9406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 shutdown_connections 2024-03-06T07:37:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:01.914+0000 7fcd99e00640 1 -- 172.21.15.32:0/2634357583 wait complete. 2024-03-06T07:37:02.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:36:58.471868+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8313}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:02.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:37:02.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:37:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.555+0000 7fb75dc00640 1 Processor -- start 2024-03-06T07:37:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.556+0000 7fb75dc00640 1 -- start start 2024-03-06T07:37:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.557+0000 7fb75dc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb758070ee0 0x7fb758162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.557+0000 7fb75dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb758072da0 0x7fb7580cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.557+0000 7fb75dc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7580ce090 con 0x7fb7580712c0 2024-03-06T07:37:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.557+0000 7fb75dc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580ceca0 con 0x7fb758070ee0 2024-03-06T07:37:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.557+0000 7fb75dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580ce370 con 0x7fb758072da0 2024-03-06T07:37:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb756a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb758070ee0 0x7fb758162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb756a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb758070ee0 0x7fb758162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33430/0 (socket says 172.21.15.32:33430) 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb756a00640 1 -- 172.21.15.32:0/3620074899 learned_addr learned my addr 172.21.15.32:0/3620074899 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb74fe00640 1 --2- 172.21.15.32:0/3620074899 >> v2:172.21.15.39:3300/0 conn(0x7fb758072da0 0x7fb7580cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb756a00640 1 -- 172.21.15.32:0/3620074899 >> v2:172.21.15.39:3300/0 conn(0x7fb758072da0 msgr2=0x7fb7580cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb756a00640 1 --2- 172.21.15.32:0/3620074899 >> v2:172.21.15.39:3300/0 conn(0x7fb758072da0 0x7fb7580cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.558+0000 7fb756a00640 1 -- 172.21.15.32:0/3620074899 >> v1:172.21.15.39:6789/0 conn(0x7fb7580712c0 legacy=0x7fb75806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.559+0000 7fb756a00640 1 -- 172.21.15.32:0/3620074899 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7580ce650 con 0x7fb758070ee0 2024-03-06T07:37:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.559+0000 7fb74fe00640 1 --2- 172.21.15.32:0/3620074899 >> v2:172.21.15.39:3300/0 conn(0x7fb758072da0 0x7fb7580cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.560+0000 7fb756a00640 1 --2- 172.21.15.32:0/3620074899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758070ee0 0x7fb758162c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fb75806c570 tx=0x7fb74802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584382030f0c7f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.560+0000 7fb756000640 1 -- 172.21.15.32:0/3620074899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb748034070 con 0x7fb758070ee0 2024-03-06T07:37:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.560+0000 7fb756000640 1 -- 172.21.15.32:0/3620074899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb748030860 con 0x7fb758070ee0 2024-03-06T07:37:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.560+0000 7fb756000640 1 -- 172.21.15.32:0/3620074899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb748030c30 con 0x7fb758070ee0 2024-03-06T07:37:02.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 -- 172.21.15.32:0/3620074899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758070ee0 msgr2=0x7fb758162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 --2- 172.21.15.32:0/3620074899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758070ee0 0x7fb758162c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fb75806c570 tx=0x7fb74802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 -- 172.21.15.32:0/3620074899 shutdown_connections 2024-03-06T07:37:02.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 --2- 172.21.15.32:0/3620074899 >> v2:172.21.15.39:3300/0 conn(0x7fb758072da0 0x7fb7580cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 --2- 172.21.15.32:0/3620074899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758070ee0 0x7fb758162c40 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 -- 172.21.15.32:0/3620074899 >> 172.21.15.32:0/3620074899 conn(0x7fb75806afb0 msgr2=0x7fb75806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 -- 172.21.15.32:0/3620074899 shutdown_connections 2024-03-06T07:37:02.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.561+0000 7fb75dc00640 1 -- 172.21.15.32:0/3620074899 wait complete. 2024-03-06T07:37:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.564+0000 7fb75dc00640 1 Processor -- start 2024-03-06T07:37:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.564+0000 7fb75dc00640 1 -- start start 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb75dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb758070ee0 0x7fb7581626b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb75dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7580712c0 0x7fb75814fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb75dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 0x7fb7580b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb75dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7580cdda0 con 0x7fb758072da0 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb75dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7580cdc20 con 0x7fb758070ee0 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb75dc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75806e7d0 con 0x7fb7580712c0 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 0x7fb7580b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 0x7fb7580b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33438/0 (socket says 172.21.15.32:33438) 2024-03-06T07:37:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb757400640 1 -- 172.21.15.32:0/2399574230 learned_addr learned my addr 172.21.15.32:0/2399574230 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.566+0000 7fb756a00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb758070ee0 0x7fb7581626b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb74fe00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7580712c0 0x7fb75814fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb757400640 1 -- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7580712c0 msgr2=0x7fb75814fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb757400640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7580712c0 0x7fb75814fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb757400640 1 -- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb758070ee0 msgr2=0x7fb7581626b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb757400640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb758070ee0 0x7fb7581626b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb757400640 1 -- 172.21.15.32:0/2399574230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758162bf0 con 0x7fb758072da0 2024-03-06T07:37:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.567+0000 7fb757400640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 0x7fb7580b3ba0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fb75000c7a0 tx=0x7fb75000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.568+0000 7fb74ea00640 1 -- 172.21.15.32:0/2399574230 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750003b70 con 0x7fb758072da0 2024-03-06T07:37:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.568+0000 7fb74ea00640 1 -- 172.21.15.32:0/2399574230 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750003d30 con 0x7fb758072da0 2024-03-06T07:37:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.568+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb758150290 con 0x7fb758072da0 2024-03-06T07:37:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.568+0000 7fb74ea00640 1 -- 172.21.15.32:0/2399574230 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75000aa90 con 0x7fb758072da0 2024-03-06T07:37:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.568+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75809cef0 con 0x7fb758072da0 2024-03-06T07:37:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.570+0000 7fb74ea00640 1 -- 172.21.15.32:0/2399574230 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb75000ac30 con 0x7fb758072da0 2024-03-06T07:37:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.571+0000 7fb74ea00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb73406b4d0 0x7fb73406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.571+0000 7fb74ea00640 1 -- 172.21.15.32:0/2399574230 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb75008a070 con 0x7fb758072da0 2024-03-06T07:37:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.571+0000 7fb756a00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb73406b4d0 0x7fb73406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.572+0000 7fb76097d740 1 Processor -- start 2024-03-06T07:37:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.572+0000 7fb756a00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb73406b4d0 0x7fb73406d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb748036000 tx=0x7fb748008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.572+0000 7fb76097d740 1 -- start start 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.573+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b9685750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.573+0000 7fb76097d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.573+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b9685c90 0x5600b96b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.573+0000 7fb76097d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600b95ef600 con 0x5600b96b2b90 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.573+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600b9598930 con 0x5600b9685c90 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.573+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600b95eb6a0 con 0x5600b96b27b0 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33454/0 (socket says 172.21.15.32:33454) 2024-03-06T07:37:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 -- 172.21.15.32:0/1854098448 learned_addr learned my addr 172.21.15.32:0/1854098448 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb757400640 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b9685c90 0x5600b96b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 -- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 msgr2=0x5600b9685750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b9685750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 -- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b9685c90 msgr2=0x5600b96b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b9685c90 0x5600b96b5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 -- 172.21.15.32:0/1854098448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600b93042d0 con 0x5600b96b2b90 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.574+0000 7fb74fe00640 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96b5540 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b970 tx=0x7fb74000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338d061810e2c31d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.575+0000 7fb747400640 1 -- 172.21.15.32:0/1854098448 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740018020 con 0x5600b96b2b90 2024-03-06T07:37:02.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.575+0000 7fb747400640 1 -- 172.21.15.32:0/1854098448 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74000fa80 con 0x5600b96b2b90 2024-03-06T07:37:02.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.575+0000 7fb747400640 1 -- 172.21.15.32:0/1854098448 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74000fe30 con 0x5600b96b2b90 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 -- 172.21.15.32:0/1854098448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 msgr2=0x5600b96b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96b5540 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b970 tx=0x7fb74000be30 comp rx=0 tx=0).stop 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 -- 172.21.15.32:0/1854098448 shutdown_connections 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b9685c90 0x5600b96b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96b5540 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 --2- 172.21.15.32:0/1854098448 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b9685750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 -- 172.21.15.32:0/1854098448 >> 172.21.15.32:0/1854098448 conn(0x5600b96879c0 msgr2=0x5600b96d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 -- 172.21.15.32:0/1854098448 shutdown_connections 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.576+0000 7fb76097d740 1 -- 172.21.15.32:0/1854098448 wait complete. 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.577+0000 7fb76097d740 1 -- start start 2024-03-06T07:37:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.577+0000 7fb76097d740 1 Processor -- start 2024-03-06T07:37:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b96abe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 0x5600b96b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600b9689f60 con 0x5600b96a7320 2024-03-06T07:37:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600b95abd90 con 0x5600b96b2b90 2024-03-06T07:37:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600b9689350 con 0x5600b96b27b0 2024-03-06T07:37:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb76097d740 10 client.0 _ll_register_callbacks cb 0x5600b95e7350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:37:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 0x5600b96b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 0x5600b96b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33466/0 (socket says 172.21.15.32:33466) 2024-03-06T07:37:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb757400640 1 -- 172.21.15.32:0/916429506 learned_addr learned my addr 172.21.15.32:0/916429506 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.578+0000 7fb756a00640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b96abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb74fe00640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb757400640 1 -- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 msgr2=0x5600b96abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb757400640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b96abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb757400640 1 -- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 msgr2=0x5600b96d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb757400640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb757400640 1 -- 172.21.15.32:0/916429506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600b93ce6b0 con 0x5600b96a7320 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb756a00640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b96abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb757400640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 0x5600b96b5220 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb75000a430 tx=0x7fb750058210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbddfd5666d14c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb73be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb73be00640 1 -- 172.21.15.32:0/916429506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750070070 con 0x5600b96a7320 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb73be00640 1 -- 172.21.15.32:0/916429506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb750054e60 con 0x5600b96a7320 2024-03-06T07:37:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb74000fa80 con 0x5600b96a7320 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb73be00640 1 -- 172.21.15.32:0/916429506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb75005c940 con 0x5600b96a7320 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb76097d740 10 client.8454 fetch_fsmap learned FSMap version 91 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.579+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x5600b96ac430 con 0x5600b96a7320 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb73be00640 1 -- 172.21.15.32:0/916429506 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74000fa80 con 0x5600b96a7320 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb73be00640 1 -- 172.21.15.32:0/916429506 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 91) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb750057c40 con 0x5600b96a7320 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb76097d740 10 client.8454 fetch_fsmap finished waiting for FSMap version 91 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb76097d740 10 client.8454 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb76097d740 20 client.8454 populate_metadata read hostname 'smithi032' 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb76097d740 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b96e4510 0x5600b96e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb76097d740 4 client.8454 mds_command: new command op to 8313 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 --> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600b95fcb40 con 0x5600b96e4510 2024-03-06T07:37:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.580+0000 7fb756a00640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b96e4510 0x5600b96e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.581+0000 7fb756a00640 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b96e4510 0x5600b96e48f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af178f8fab618eb server_cookie=abea34070a555c8e in_seq=0 out_seq=0 2024-03-06T07:37:02.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.581+0000 7fb73be00640 10 client.8454 ms_handle_connect on v2:172.21.15.32:6835/294605993 2024-03-06T07:37:02.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb73be00640 1 -- 172.21.15.32:0/916429506 <== mds.0 v2:172.21.15.32:6835/294605993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600b95fcb40 con 0x5600b96e4510 2024-03-06T07:37:02.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb73be00640 10 client.8454 handle_command_reply: tid=0 2024-03-06T07:37:02.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 client.8454 shutdown 2024-03-06T07:37:02.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 msgr2=0x5600b96b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 0x5600b96b5220 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb75000a430 tx=0x7fb750058210 comp rx=0 tx=0).stop 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 shutdown_connections 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b96e4510 0x5600b96e48f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96a7320 0x5600b96b5220 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 --2- 172.21.15.32:0/916429506 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96b27b0 0x5600b96abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb73be00640 0 client.8454 ms_handle_reset on v2:172.21.15.32:6835/294605993 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.582+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 >> 172.21.15.32:0/916429506 conn(0x5600b9685c90 msgr2=0x5600b96d3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.583+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 shutdown_connections 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.583+0000 7fb76097d740 1 -- 172.21.15.32:0/916429506 wait complete. 2024-03-06T07:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.583+0000 7fb76097d740 20 client.8454 trim_cache size 0 max 16384 2024-03-06T07:37:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.601+0000 7fb76097d740 1 Processor -- start 2024-03-06T07:37:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.601+0000 7fb76097d740 1 -- start start 2024-03-06T07:37:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.602+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.602+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b27b0 0x5600b96aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.602+0000 7fb76097d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.602+0000 7fb76097d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758203860 con 0x5600b96b2b90 2024-03-06T07:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.602+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758202d20 con 0x5600b96b27b0 2024-03-06T07:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.602+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb758203260 con 0x5600b96a7320 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33470/0 (socket says 172.21.15.32:33470) 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 -- 172.21.15.32:0/2850346348 learned_addr learned my addr 172.21.15.32:0/2850346348 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 -- 172.21.15.32:0/2850346348 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 msgr2=0x5600b96acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 -- 172.21.15.32:0/2850346348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b27b0 msgr2=0x5600b96aadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b27b0 0x5600b96aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 -- 172.21.15.32:0/2850346348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600b96ae1d0 con 0x5600b96b2b90 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb757400640 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96c5ed0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb750001980 tx=0x7fb750005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e9ea2c7279493 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb746a00640 1 -- 172.21.15.32:0/2850346348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750077030 con 0x5600b96b2b90 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb746a00640 1 -- 172.21.15.32:0/2850346348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7500884e0 con 0x5600b96b2b90 2024-03-06T07:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.603+0000 7fb746a00640 1 -- 172.21.15.32:0/2850346348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750070040 con 0x5600b96b2b90 2024-03-06T07:37:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 -- 172.21.15.32:0/2850346348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 msgr2=0x5600b96c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96c5ed0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb750001980 tx=0x7fb750005050 comp rx=0 tx=0).stop 2024-03-06T07:37:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 -- 172.21.15.32:0/2850346348 shutdown_connections 2024-03-06T07:37:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b2b90 0x5600b96c5ed0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b27b0 0x5600b96aadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 --2- 172.21.15.32:0/2850346348 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 -- 172.21.15.32:0/2850346348 >> 172.21.15.32:0/2850346348 conn(0x5600b9685c90 msgr2=0x5600b96abc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 -- 172.21.15.32:0/2850346348 shutdown_connections 2024-03-06T07:37:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.604+0000 7fb76097d740 1 -- 172.21.15.32:0/2850346348 wait complete. 2024-03-06T07:37:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.605+0000 7fb76097d740 1 -- start start 2024-03-06T07:37:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.605+0000 7fb76097d740 1 Processor -- start 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96b6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 0x5600b968b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96c6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600b96b4d00 con 0x5600b96b27b0 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600b96ab860 con 0x5600b96b2b90 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600b96adf00 con 0x5600b96a7320 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb76097d740 10 client.0 _ll_register_callbacks cb 0x7fb75000ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb74fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 0x5600b968b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb74fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 0x5600b968b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33486/0 (socket says 172.21.15.32:33486) 2024-03-06T07:37:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.606+0000 7fb74fe00640 1 -- 172.21.15.32:0/2393036321 learned_addr learned my addr 172.21.15.32:0/2393036321 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb757400640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb756a00640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb74fe00640 1 -- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 msgr2=0x5600b96b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb74fe00640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96b6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb74fe00640 1 -- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 msgr2=0x5600b96c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb74fe00640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb74fe00640 1 -- 172.21.15.32:0/2393036321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600b9729370 con 0x5600b96b27b0 2024-03-06T07:37:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb74fe00640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 0x5600b968b450 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb740017040 tx=0x7fb7400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93da990399b1289 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb73be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-03-06T07:37:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.607+0000 7fb73be00640 1 -- 172.21.15.32:0/2393036321 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740018070 con 0x5600b96b27b0 2024-03-06T07:37:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb73be00640 1 -- 172.21.15.32:0/2393036321 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb74000fa90 con 0x5600b96b27b0 2024-03-06T07:37:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7500884e0 con 0x5600b96b27b0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb73be00640 1 -- 172.21.15.32:0/2393036321 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb74002f050 con 0x5600b96b27b0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 10 client.8460 fetch_fsmap learned FSMap version 91 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x5600b9741250 con 0x5600b96b27b0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb73be00640 1 -- 172.21.15.32:0/2393036321 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7500884e0 con 0x5600b96b27b0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb73be00640 1 -- 172.21.15.32:0/2393036321 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 91) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb740032020 con 0x5600b96b27b0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 10 client.8460 fetch_fsmap finished waiting for FSMap version 91 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 10 client.8460 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 20 client.8460 populate_metadata read hostname 'smithi032' 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b9689ca0 0x5600b96b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 4 client.8460 mds_command: new command op to 8313 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.608+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 --> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5600b95f6e90 con 0x5600b9689ca0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.609+0000 7fb756a00640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b9689ca0 0x5600b96b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.609+0000 7fb756a00640 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b9689ca0 0x5600b96b3520 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f979124114524 server_cookie=ea40d95165f7487b in_seq=0 out_seq=0 2024-03-06T07:37:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.609+0000 7fb73be00640 10 client.8460 ms_handle_connect on v2:172.21.15.32:6835/294605993 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb73be00640 1 -- 172.21.15.32:0/2393036321 <== mds.0 v2:172.21.15.32:6835/294605993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5600b95f6e90 con 0x5600b9689ca0 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb73be00640 10 client.8460 handle_command_reply: tid=0 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 client.8460 shutdown 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 msgr2=0x5600b968b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 0x5600b968b450 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb740017040 tx=0x7fb7400046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 shutdown_connections 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:6835/294605993,v1:172.21.15.32:6836/294605993] conn(0x5600b9689ca0 0x5600b96b3520 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600b96b2b90 0x5600b96c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600b96b27b0 0x5600b968b450 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 --2- 172.21.15.32:0/2393036321 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x5600b96a7320 0x5600b96b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 >> 172.21.15.32:0/2393036321 conn(0x5600b9685c90 msgr2=0x5600b96b91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb73be00640 0 client.8460 ms_handle_reset on v2:172.21.15.32:6835/294605993 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 shutdown_connections 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 1 -- 172.21.15.32:0/2393036321 wait complete. 2024-03-06T07:37:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.610+0000 7fb76097d740 20 client.8460 trim_cache size 0 max 16384 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "e91bbd07-5556-4949-b381-99a73b242ae4", 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8313, 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8098504359999996, 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 91, 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 127, 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 127, 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.218717008 2024-03-06T07:37:02.614 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb73406b4d0 msgr2=0x7fb73406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb73406b4d0 0x7fb73406d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb748036000 tx=0x7fb748008030 comp rx=0 tx=0).stop 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 msgr2=0x7fb7580b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 0x7fb7580b3ba0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fb75000c7a0 tx=0x7fb75000cc60 comp rx=0 tx=0).stop 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 shutdown_connections 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb73406b4d0 0x7fb73406d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758072da0 0x7fb7580b3ba0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb7580712c0 0x7fb75814fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 --2- 172.21.15.32:0/2399574230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb758070ee0 0x7fb7581626b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 >> 172.21.15.32:0/2399574230 conn(0x7fb75806afb0 msgr2=0x7fb7581652c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 shutdown_connections 2024-03-06T07:37:02.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.611+0000 7fb75dc00640 1 -- 172.21.15.32:0/2399574230 wait complete. 2024-03-06T07:37:02.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:37:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.839+0000 7fe9e5800640 1 Processor -- start 2024-03-06T07:37:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.839+0000 7fe9e5800640 1 -- start start 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.840+0000 7fe9e5800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9e00b3c90 0x7fe9e00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.840+0000 7fe9e5800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe9e0071050 0x7fe9e0071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.840+0000 7fe9e5800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00b4860 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.840+0000 7fe9e5800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00ce160 con 0x7fe9e00b3c90 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.840+0000 7fe9e5800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00b4b40 con 0x7fe9e0071050 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9e00b3c90 0x7fe9e00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9e00b3c90 0x7fe9e00cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33498/0 (socket says 172.21.15.32:33498) 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 -- 172.21.15.32:0/2361629531 learned_addr learned my addr 172.21.15.32:0/2361629531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9de000640 1 --1- 172.21.15.32:0/2361629531 >> v1:172.21.15.32:6789/0 conn(0x7fe9e00cf5c0 0x7fe9e00b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39818/0 (socket says 172.21.15.32:39818) 2024-03-06T07:37:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9df400640 1 --2- 172.21.15.32:0/2361629531 >> v2:172.21.15.39:3300/0 conn(0x7fe9e0071050 0x7fe9e0071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 -- 172.21.15.32:0/2361629531 >> v2:172.21.15.39:3300/0 conn(0x7fe9e0071050 msgr2=0x7fe9e0071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 --2- 172.21.15.32:0/2361629531 >> v2:172.21.15.39:3300/0 conn(0x7fe9e0071050 0x7fe9e0071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 -- 172.21.15.32:0/2361629531 >> v1:172.21.15.32:6789/0 conn(0x7fe9e00cf5c0 legacy=0x7fe9e00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9dea00640 1 -- 172.21.15.32:0/2361629531 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00b4e20 con 0x7fe9e00b3c90 2024-03-06T07:37:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.841+0000 7fe9df400640 1 --2- 172.21.15.32:0/2361629531 >> v2:172.21.15.39:3300/0 conn(0x7fe9e0071050 0x7fe9e0071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.843+0000 7fe9dea00640 1 --2- 172.21.15.32:0/2361629531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00b3c90 0x7fe9e00cdc20 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0074860 tx=0x7fe9d002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659d3654d240a266 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9dd600640 1 -- 172.21.15.32:0/2361629531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d002f850 con 0x7fe9e00b3c90 2024-03-06T07:37:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9dd600640 1 -- 172.21.15.32:0/2361629531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d002f9f0 con 0x7fe9e00b3c90 2024-03-06T07:37:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9dd600640 1 -- 172.21.15.32:0/2361629531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d002fdc0 con 0x7fe9e00b3c90 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9e5800640 1 -- 172.21.15.32:0/2361629531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00b3c90 msgr2=0x7fe9e00cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2361629531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00b3c90 0x7fe9e00cdc20 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0074860 tx=0x7fe9d002ba30 comp rx=0 tx=0).stop 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9e5800640 1 -- 172.21.15.32:0/2361629531 shutdown_connections 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2361629531 >> v2:172.21.15.39:3300/0 conn(0x7fe9e0071050 0x7fe9e0071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2361629531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00b3c90 0x7fe9e00cdc20 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.846+0000 7fe9e5800640 1 -- 172.21.15.32:0/2361629531 >> 172.21.15.32:0/2361629531 conn(0x7fe9e006afb0 msgr2=0x7fe9e006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.847+0000 7fe9e5800640 1 -- 172.21.15.32:0/2361629531 shutdown_connections 2024-03-06T07:37:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.847+0000 7fe9e5800640 1 -- 172.21.15.32:0/2361629531 wait complete. 2024-03-06T07:37:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.848+0000 7fe9e5800640 1 Processor -- start 2024-03-06T07:37:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.848+0000 7fe9e5800640 1 -- start start 2024-03-06T07:37:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.849+0000 7fe9e5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9e0071050 0x7fe9e01441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.849+0000 7fe9e5800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe9e00b3c90 0x7fe9e0150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.849+0000 7fe9e5800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 0x7fe9e01f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.849+0000 7fe9e5800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00719d0 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.849+0000 7fe9e5800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e015b340 con 0x7fe9e0071050 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.849+0000 7fe9e5800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0071b50 con 0x7fe9e00b3c90 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 0x7fe9e01f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 0x7fe9e01f4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33502/0 (socket says 172.21.15.32:33502) 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 -- 172.21.15.32:0/2979253362 learned_addr learned my addr 172.21.15.32:0/2979253362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9dea00640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9e0071050 0x7fe9e01441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9de000640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe9e00b3c90 0x7fe9e0150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 -- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe9e00b3c90 msgr2=0x7fe9e0150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe9e00b3c90 0x7fe9e0150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 -- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9e0071050 msgr2=0x7fe9e01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9e0071050 0x7fe9e01441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9df400640 1 -- 172.21.15.32:0/2979253362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0144730 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.850+0000 7fe9dea00640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9e0071050 0x7fe9e01441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.851+0000 7fe9df400640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 0x7fe9e01f4fa0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800c7f0 tx=0x7fe9d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.851+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8015070 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.851+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d80077c0 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.851+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e0150670 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.851+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8007bf0 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.851+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e0152c90 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.853+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e0069510 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.853+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9d8018070 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.854+0000 7fe9d6a00640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe9bc06b500 0x7fe9bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.854+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9d808a2d0 con 0x7fe9e00cf5c0 2024-03-06T07:37:02.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.854+0000 7fe9dea00640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe9bc06b500 0x7fe9bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.855+0000 7fe9dea00640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe9bc06b500 0x7fe9bc06d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0036000 tx=0x7fe9d0039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:02.863+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9e0144730 con 0x7fe9e00cf5c0 2024-03-06T07:37:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.215+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe9e006b720 con 0x7fe9e00cf5c0 2024-03-06T07:37:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.216+0000 7fe9d6a00640 1 -- 172.21.15.32:0/2979253362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe9d8059140 con 0x7fe9e00cf5c0 2024-03-06T07:37:03.217 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe9bc06b500 msgr2=0x7fe9bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe9bc06b500 0x7fe9bc06d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0036000 tx=0x7fe9d0039040 comp rx=0 tx=0).stop 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 msgr2=0x7fe9e01f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 0x7fe9e01f4fa0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800c7f0 tx=0x7fe9d800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 shutdown_connections 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe9bc06b500 0x7fe9bc06d9c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e00cf5c0 0x7fe9e01f4fa0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe9e00b3c90 0x7fe9e0150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 --2- 172.21.15.32:0/2979253362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe9e0071050 0x7fe9e01441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.218+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 >> 172.21.15.32:0/2979253362 conn(0x7fe9e006afb0 msgr2=0x7fe9e009d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.219+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 shutdown_connections 2024-03-06T07:37:03.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:03.219+0000 7fe9e5800640 1 -- 172.21.15.32:0/2979253362 wait complete. 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:37:03.240 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:03.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.241 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-03-06T07:37:03.297 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41914sec preferred_lft 41914sec 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::9049:c6ff:fe8c:45c8/64 scope link proto kernel_ll 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.298 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 2e:96:e8:1e:c9:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2c96:e8ff:fe1e:c9fb/64 scope link proto kernel_ll 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 5e:5b:40:1a:14:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::5c5b:40ff:fe1a:14dd/64 scope link proto kernel_ll 2024-03-06T07:37:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:37:03.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.300 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:37:03.400 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-06T07:37:03.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.404 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-03-06T07:37:03.455 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-03-06T07:37:03.455 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:37:03.455 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:37:03.455 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:37:03.455 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:37:03.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.456 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-03-06T07:37:03.456 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-03-06T07:37:03.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:37:03.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:37:03.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:37:03.457 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:37:03.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.648 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-03-06T07:37:03.698 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:37:03.698 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:37:03.699 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:03.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-03-06T07:37:03.786 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:37:03.786 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:03.839 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:37:03.839 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:37:03.839 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:03.882 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:37:03.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:03.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:37:04.062 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:37:04.062 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:37:04.096 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:37:04.097 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:37:04.097 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.39:3300,172.21.15.39:3301" 2024-03-06T07:37:04.097 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with new device syntax: 0@e91bbd07-5556-4949-b381-99a73b242ae4.cephfs=/ 2024-03-06T07:37:04.097 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.32:3300/172.21.15.39:3300/172.21.15.39:3301" will pass to kernel 2024-03-06T07:37:04.100 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:37:04.100 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:37:04.100 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:37:04.100 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:37:04.101 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:37:04.104 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:37:04.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:04.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:37:04.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:04.195 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:37:04.264 INFO:teuthology.orchestra.run.smithi032.stdout:client5945 2024-03-06T07:37:04.264 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:37:04.264 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:04.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:37:04.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.583+0000 7f9692200640 1 Processor -- start 2024-03-06T07:37:04.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.584+0000 7f9692200640 1 -- start start 2024-03-06T07:37:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9692200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f968c15f0a0 0x7f968c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9692200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c06c890 0x7f968c157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9692200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c0cf650 con 0x7f968c0afff0 2024-03-06T07:37:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9692200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968c06bde0 con 0x7f968c15f0a0 2024-03-06T07:37:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9692200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968c15ec10 con 0x7f968c06c890 2024-03-06T07:37:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9690e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f968c0afff0 0x7f968c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39822/0 (socket says 172.21.15.32:39822) 2024-03-06T07:37:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f9690e00640 1 -- 172.21.15.32:0/1791236654 learned_addr learned my addr 172.21.15.32:0/1791236654 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.587+0000 7f968be00640 1 --2- 172.21.15.32:0/1791236654 >> v2:172.21.15.32:3300/0 conn(0x7f968c15f0a0 0x7f968c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.588+0000 7f9691800640 1 --2- 172.21.15.32:0/1791236654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c06c890 0x7f968c157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.588+0000 7f968be00640 1 -- 172.21.15.32:0/1791236654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c06c890 msgr2=0x7f968c157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.588+0000 7f968be00640 1 --2- 172.21.15.32:0/1791236654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c06c890 0x7f968c157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.588+0000 7f968be00640 1 -- 172.21.15.32:0/1791236654 >> v1:172.21.15.32:6789/0 conn(0x7f968c0afff0 legacy=0x7f968c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.588+0000 7f968be00640 1 -- 172.21.15.32:0/1791236654 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c06bb00 con 0x7f968c15f0a0 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.588+0000 7f968b400640 1 -- 172.21.15.32:0/1791236654 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835778602 0 0) 0x7f968c0cf650 con 0x7f968c0afff0 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.589+0000 7f968be00640 1 --2- 172.21.15.32:0/1791236654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c15f0a0 0x7f968c06c350 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f96780097a0 tx=0x7f967802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e6c4346d1c9e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.589+0000 7f968b400640 1 -- 172.21.15.32:0/1791236654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678035020 con 0x7f968c15f0a0 2024-03-06T07:37:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.589+0000 7f968b400640 1 -- 172.21.15.32:0/1791236654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678031830 con 0x7f968c15f0a0 2024-03-06T07:37:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.589+0000 7f968b400640 1 -- 172.21.15.32:0/1791236654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678031c00 con 0x7f968c15f0a0 2024-03-06T07:37:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 -- 172.21.15.32:0/1791236654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c15f0a0 msgr2=0x7f968c06c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 --2- 172.21.15.32:0/1791236654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c15f0a0 0x7f968c06c350 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f96780097a0 tx=0x7f967802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 -- 172.21.15.32:0/1791236654 shutdown_connections 2024-03-06T07:37:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 --2- 172.21.15.32:0/1791236654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c15f0a0 0x7f968c06c350 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 --2- 172.21.15.32:0/1791236654 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c06c890 0x7f968c157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 -- 172.21.15.32:0/1791236654 >> 172.21.15.32:0/1791236654 conn(0x7f968c0b2940 msgr2=0x7f968c0b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 -- 172.21.15.32:0/1791236654 shutdown_connections 2024-03-06T07:37:04.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.590+0000 7f9692200640 1 -- 172.21.15.32:0/1791236654 wait complete. 2024-03-06T07:37:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.593+0000 7f9692200640 1 Processor -- start 2024-03-06T07:37:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.593+0000 7f9692200640 1 -- start start 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9692200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 0x7f968c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9692200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f968c0afff0 0x7f968c160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9692200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c15f0a0 0x7f968c160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9692200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0b03d0 con 0x7f968c06c890 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9692200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c15e8a0 con 0x7f968c0afff0 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9692200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0b0550 con 0x7f968c15f0a0 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9690e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 0x7f968c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9690e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 0x7f968c0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33516/0 (socket says 172.21.15.32:33516) 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f9690e00640 1 -- 172.21.15.32:0/1062464837 learned_addr learned my addr 172.21.15.32:0/1062464837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.595+0000 7f968be00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f968c0afff0 0x7f968c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9691800640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c15f0a0 0x7f968c160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9690e00640 1 -- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c15f0a0 msgr2=0x7f968c160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9690e00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c15f0a0 0x7f968c160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9690e00640 1 -- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f968c0afff0 msgr2=0x7f968c160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9690e00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f968c0afff0 0x7f968c160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9690e00640 1 -- 172.21.15.32:0/1062464837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c1e7ac0 con 0x7f968c06c890 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9690e00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 0x7f968c0b4ed0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f968000c4d0 tx=0x7f968000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680018070 con 0x7f968c06c890 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9680007500 con 0x7f968c06c890 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680004a20 con 0x7f968c06c890 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.596+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c1e7da0 con 0x7f968c06c890 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.597+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c1e8360 con 0x7f968c06c890 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.598+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c04f330 con 0x7f968c06c890 2024-03-06T07:37:04.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.599+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9680004bc0 con 0x7f968c06c890 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.600+0000 7f9688c00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f966c06b4d0 0x7f966c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.600+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9680089620 con 0x7f968c06c890 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.600+0000 7f968be00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f966c06b4d0 0x7f966c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.601+0000 7f968be00640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f966c06b4d0 0x7f966c06d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9678036000 tx=0x7f967803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.604+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9680058400 con 0x7f968c06c890 2024-03-06T07:37:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.848+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f968c0b5410 con 0x7f968c06c890 2024-03-06T07:37:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.883+0000 7f9688c00640 1 -- 172.21.15.32:0/1062464837 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v92) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9680057d00 con 0x7f968c06c890 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f966c06b4d0 msgr2=0x7f966c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f966c06b4d0 0x7f966c06d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9678036000 tx=0x7f967803c040 comp rx=0 tx=0).stop 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 msgr2=0x7f968c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 0x7f968c0b4ed0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f968000c4d0 tx=0x7f968000c990 comp rx=0 tx=0).stop 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 shutdown_connections 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f966c06b4d0 0x7f966c06d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f968c15f0a0 0x7f968c160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f968c0afff0 0x7f968c160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 --2- 172.21.15.32:0/1062464837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968c06c890 0x7f968c0b4ed0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 >> 172.21.15.32:0/1062464837 conn(0x7f968c0b2940 msgr2=0x7f968c06f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.886+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 shutdown_connections 2024-03-06T07:37:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:04.887+0000 7f9692200640 1 -- 172.21.15.32:0/1062464837 wait complete. 2024-03-06T07:37:04.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:05.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.110+0000 7f80a7e00640 1 Processor -- start 2024-03-06T07:37:05.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.110+0000 7f80a7e00640 1 -- start start 2024-03-06T07:37:05.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80a006ca90 0x7f80a006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a7e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a006d7f0 con 0x7f80a006bf10 2024-03-06T07:37:05.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a7e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00b57f0 con 0x7f80a015bc80 2024-03-06T07:37:05.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a7e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a00b60e0 con 0x7f80a006ca90 2024-03-06T07:37:05.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a5e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80a006bf10 0x7f80a00b6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39828/0 (socket says 172.21.15.32:39828) 2024-03-06T07:37:05.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a5e00640 1 -- 172.21.15.32:0/3747543090 learned_addr learned my addr 172.21.15.32:0/3747543090 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:05.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.112+0000 7f80a4a00640 1 --2- 172.21.15.32:0/3747543090 >> v2:172.21.15.32:3300/0 conn(0x7f80a006ca90 0x7f80a006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.113+0000 7f809fe00640 1 -- 172.21.15.32:0/3747543090 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297891863 0 0) 0x7f8098001490 con 0x7f80a006bf10 2024-03-06T07:37:05.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.113+0000 7f809fe00640 1 -- 172.21.15.32:0/3747543090 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8088003520 con 0x7f80a006bf10 2024-03-06T07:37:05.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.113+0000 7f80a4a00640 1 -- 172.21.15.32:0/3747543090 >> v1:172.21.15.39:6789/0 conn(0x7f80a015bc80 legacy=0x7f80a006ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.113+0000 7f80a4a00640 1 -- 172.21.15.32:0/3747543090 >> v1:172.21.15.32:6789/0 conn(0x7f80a006bf10 legacy=0x7f80a00b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.113+0000 7f80a4a00640 1 -- 172.21.15.32:0/3747543090 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a006d510 con 0x7f80a006ca90 2024-03-06T07:37:05.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.114+0000 7f80a4a00640 1 --2- 172.21.15.32:0/3747543090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006ca90 0x7f80a006cef0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f808c0097a0 tx=0x7f808c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8a78cf27dc9b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.114+0000 7f809fe00640 1 -- 172.21.15.32:0/3747543090 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c034070 con 0x7f80a006ca90 2024-03-06T07:37:05.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.114+0000 7f809fe00640 1 -- 172.21.15.32:0/3747543090 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c030610 con 0x7f80a006ca90 2024-03-06T07:37:05.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.114+0000 7f809fe00640 1 -- 172.21.15.32:0/3747543090 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c0309c0 con 0x7f80a006ca90 2024-03-06T07:37:05.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 -- 172.21.15.32:0/3747543090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006ca90 msgr2=0x7f80a006cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 --2- 172.21.15.32:0/3747543090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006ca90 0x7f80a006cef0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f808c0097a0 tx=0x7f808c02b960 comp rx=0 tx=0).stop 2024-03-06T07:37:05.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 -- 172.21.15.32:0/3747543090 shutdown_connections 2024-03-06T07:37:05.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 --2- 172.21.15.32:0/3747543090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006ca90 0x7f80a006cef0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 -- 172.21.15.32:0/3747543090 >> 172.21.15.32:0/3747543090 conn(0x7f80a009aa10 msgr2=0x7f80a009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 -- 172.21.15.32:0/3747543090 shutdown_connections 2024-03-06T07:37:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.115+0000 7f80a7e00640 1 -- 172.21.15.32:0/3747543090 wait complete. 2024-03-06T07:37:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.117+0000 7f80a7e00640 1 Processor -- start 2024-03-06T07:37:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.117+0000 7f80a7e00640 1 -- start start 2024-03-06T07:37:05.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 0x7f80a014d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a7e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f80a006ca90 0x7f80a0150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a015bc80 0x7f80a01f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a015f630 con 0x7f80a006bf10 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00b5d70 con 0x7f80a015bc80 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a7e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a015c060 con 0x7f80a006ca90 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 0x7f80a014d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 0x7f80a014d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33534/0 (socket says 172.21.15.32:33534) 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.118+0000 7f80a5400640 1 -- 172.21.15.32:0/1915462079 learned_addr learned my addr 172.21.15.32:0/1915462079 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a4a00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f80a006ca90 0x7f80a0150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a015bc80 0x7f80a01f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5400640 1 -- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f80a006ca90 msgr2=0x7f80a0150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5400640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f80a006ca90 0x7f80a0150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5400640 1 -- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a015bc80 msgr2=0x7f80a01f78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5400640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a015bc80 0x7f80a01f78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5400640 1 -- 172.21.15.32:0/1915462079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a014f530 con 0x7f80a006bf10 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f80a5400640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 0x7f80a014d3a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f809400c3f0 tx=0x7f809400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80940052d0 con 0x7f80a006bf10 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8094005490 con 0x7f80a006bf10 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.119+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094007260 con 0x7f80a006bf10 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.120+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a014f810 con 0x7f80a006bf10 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.120+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a014fdd0 con 0x7f80a006bf10 2024-03-06T07:37:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.121+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a004f3a0 con 0x7f80a006bf10 2024-03-06T07:37:05.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.123+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8094007400 con 0x7f80a006bf10 2024-03-06T07:37:05.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.124+0000 7f809d600640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.124+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8094089780 con 0x7f80a006bf10 2024-03-06T07:37:05.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.124+0000 7f80a4a00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.125+0000 7f80a4a00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f808006b4d0 0x7f808006d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f808c036000 tx=0x7f808c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.129+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8094058560 con 0x7f80a006bf10 2024-03-06T07:37:05.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.367+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a009b7b0 con 0x7f80a006bf10 2024-03-06T07:37:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.368+0000 7f809d600640 1 -- 172.21.15.32:0/1915462079 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f8094057e60 con 0x7f80a006bf10 2024-03-06T07:37:05.370 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:05.370 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":93,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/4261872692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4261872692},{"type":"v1","addr":"172.21.15.32:6837","nonce":4261872692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":5894,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/304553199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":304553199},{"type":"v1","addr":"172.21.15.39:6835","nonce":304553199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:04.886061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:creating","state_seq":1,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 93 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f808006b4d0 msgr2=0x7f808006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f808006b4d0 0x7f808006d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f808c036000 tx=0x7f808c008030 comp rx=0 tx=0).stop 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 msgr2=0x7f80a014d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 0x7f80a014d3a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f809400c3f0 tx=0x7f809400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 shutdown_connections 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a015bc80 0x7f80a01f78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f80a006ca90 0x7f80a0150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 --2- 172.21.15.32:0/1915462079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a006bf10 0x7f80a014d3a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.371+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 >> 172.21.15.32:0/1915462079 conn(0x7f80a009aa10 msgr2=0x7f80a00b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.372+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 shutdown_connections 2024-03-06T07:37:05.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:05.372+0000 7f80a7e00640 1 -- 172.21.15.32:0/1915462079 wait complete. 2024-03-06T07:37:05.399 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:37:10.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:10.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.616+0000 7fe72fc00640 1 Processor -- start 2024-03-06T07:37:10.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.616+0000 7fe72fc00640 1 -- start start 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.617+0000 7fe72fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe7280d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe72fc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7280b50e0 0x7fe7280d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe72fc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7280b3940 con 0x7fe7280d0d70 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe72fc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7280d4e70 con 0x7fe7280b50e0 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe72fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280b3c40 con 0x7fe72815e2e0 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe71fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7280d0d70 0x7fe7280d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47730/0 (socket says 172.21.15.32:47730) 2024-03-06T07:37:10.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe71fe00640 1 -- 172.21.15.32:0/902890465 learned_addr learned my addr 172.21.15.32:0/902890465 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:10.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe72d200640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe7280d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.618+0000 7fe72dc00640 1 --2- 172.21.15.32:0/902890465 >> v2:172.21.15.39:3300/0 conn(0x7fe7280b50e0 0x7fe7280d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598432889 0 0) 0x7fe7280b3940 con 0x7fe7280d0d70 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c003520 con 0x7fe7280d0d70 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3844886620 0 0) 0x7fe70c003520 con 0x7fe7280d0d70 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 msgr2=0x7fe7280d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe7280d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 >> v2:172.21.15.39:3300/0 conn(0x7fe7280b50e0 msgr2=0x7fe7280d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 --2- 172.21.15.32:0/902890465 >> v2:172.21.15.39:3300/0 conn(0x7fe7280b50e0 0x7fe7280d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7280cf580 con 0x7fe7280d0d70 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fe710001670 con 0x7fe7280d0d70 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 >> v1:172.21.15.32:6789/0 conn(0x7fe7280d0d70 legacy=0x7fe7280d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe70c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c004d40 0x7fe70c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280b3c40 con 0x7fe70c003a20 2024-03-06T07:37:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c00b9c0 con 0x7fe70c004d40 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.619+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c00bb40 con 0x7fe72815e2e0 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72d200640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72dc00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c004d40 0x7fe70c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe71fe00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe70c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72d200640 1 -- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 msgr2=0x7fe70c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72d200640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe70c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72d200640 1 -- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c004d40 msgr2=0x7fe70c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72d200640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c004d40 0x7fe70c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe72d200640 1 -- 172.21.15.32:0/902890465 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c003520 con 0x7fe70c003a20 2024-03-06T07:37:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.620+0000 7fe71fe00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72815e2e0 0x7fe70c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:10.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.622+0000 7fe72d200640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ac0 tx=0x7fe70c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed613b435f9e09e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.622+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718039070 con 0x7fe70c003a20 2024-03-06T07:37:10.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.622+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71802c6b0 con 0x7fe70c003a20 2024-03-06T07:37:10.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.622+0000 7fe727e00640 1 -- 172.21.15.32:0/902890465 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718009690 con 0x7fe70c003a20 2024-03-06T07:37:10.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72fc00640 1 -- 172.21.15.32:0/902890465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe70c003a20 msgr2=0x7fe70c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72fc00640 1 --2- 172.21.15.32:0/902890465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ac0 tx=0x7fe70c005830 comp rx=0 tx=0).stop 2024-03-06T07:37:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72dc00640 1 -- 172.21.15.32:0/902890465 reap_dead start 2024-03-06T07:37:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72fc00640 1 -- 172.21.15.32:0/902890465 shutdown_connections 2024-03-06T07:37:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72fc00640 1 -- 172.21.15.32:0/902890465 >> 172.21.15.32:0/902890465 conn(0x7fe7280d3f60 msgr2=0x7fe7280d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72fc00640 1 -- 172.21.15.32:0/902890465 shutdown_connections 2024-03-06T07:37:10.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.623+0000 7fe72fc00640 1 -- 172.21.15.32:0/902890465 wait complete. 2024-03-06T07:37:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.625+0000 7fe72fc00640 1 Processor -- start 2024-03-06T07:37:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.625+0000 7fe72fc00640 1 -- start start 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe72fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c003a20 0x7fe7280c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe72fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 0x7fe72814dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe72fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72814aa80 0x7fe72815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe72fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280cfe20 con 0x7fe7280ca620 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe72fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280cfb40 con 0x7fe70c003a20 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe72fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280d0830 con 0x7fe72814aa80 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.626+0000 7fe71fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 0x7fe72814dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 0x7fe72814dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44900/0 (socket says 172.21.15.32:44900) 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 -- 172.21.15.32:0/433657980 learned_addr learned my addr 172.21.15.32:0/433657980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe72d200640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c003a20 0x7fe7280c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe72dc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72814aa80 0x7fe72815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 -- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72814aa80 msgr2=0x7fe72815b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72814aa80 0x7fe72815b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 -- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c003a20 msgr2=0x7fe7280c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c003a20 0x7fe7280c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 -- 172.21.15.32:0/433657980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7280c9860 con 0x7fe7280ca620 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.627+0000 7fe71fe00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 0x7fe72814dfa0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe71000d8e0 tx=0x7fe71000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.628+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710019070 con 0x7fe7280ca620 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.628+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7100074e0 con 0x7fe7280ca620 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.628+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7100049e0 con 0x7fe7280ca620 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.628+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7280c9b40 con 0x7fe7280ca620 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.628+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7280c9510 con 0x7fe7280ca620 2024-03-06T07:37:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.629+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72804f3a0 con 0x7fe7280ca620 2024-03-06T07:37:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.630+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe710004b80 con 0x7fe7280ca620 2024-03-06T07:37:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.631+0000 7fe725600640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.631+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe710089db0 con 0x7fe7280ca620 2024-03-06T07:37:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.632+0000 7fe72d200640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.633+0000 7fe72d200640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe70406b4d0 0x7fe70406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe718036000 tx=0x7fe718030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.639+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe71008e070 con 0x7fe7280ca620 2024-03-06T07:37:10.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.886+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7280ce150 con 0x7fe7280ca620 2024-03-06T07:37:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.887+0000 7fe725600640 1 -- 172.21.15.32:0/433657980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4992 (secure 0 0 0) 0x7fe710058b00 con 0x7fe7280ca620 2024-03-06T07:37:10.890 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:10.890 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/4261872692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4261872692},{"type":"v1","addr":"172.21.15.32:6837","nonce":4261872692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":5894,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/304553199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":304553199},{"type":"v1","addr":"172.21.15.39:6835","nonce":304553199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:05.894136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[6574,8313]},"id":9}]} 2024-03-06T07:37:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 94 2024-03-06T07:37:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe70406b4d0 msgr2=0x7fe70406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe70406b4d0 0x7fe70406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe718036000 tx=0x7fe718030000 comp rx=0 tx=0).stop 2024-03-06T07:37:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 msgr2=0x7fe72814dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 0x7fe72814dfa0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe71000d8e0 tx=0x7fe71000dda0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 shutdown_connections 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe72814aa80 0x7fe72815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7280ca620 0x7fe72814dfa0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 --2- 172.21.15.32:0/433657980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe70c003a20 0x7fe7280c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.890+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 >> 172.21.15.32:0/433657980 conn(0x7fe7280d3f60 msgr2=0x7fe7280b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.891+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 shutdown_connections 2024-03-06T07:37:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:10.891+0000 7fe72fc00640 1 -- 172.21.15.32:0/433657980 wait complete. 2024-03-06T07:37:10.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:11.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.133+0000 7f56e0e00640 1 Processor -- start 2024-03-06T07:37:11.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.134+0000 7f56e0e00640 1 -- start start 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.135+0000 7f56e0e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.135+0000 7f56e0e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f56dc06afb0 0x7f56dc06b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.135+0000 7f56e0e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc06bec0 con 0x7f56dc15e200 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.135+0000 7f56e0e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc0b0670 con 0x7f56dc06afb0 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.135+0000 7f56e0e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0ae380 con 0x7f56dc0afdf0 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.136+0000 7f56d9e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f56dc15e200 0x7f56dc0af140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:34672/0 (socket says 172.21.15.32:34672) 2024-03-06T07:37:11.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.136+0000 7f56d9e00640 1 -- 172.21.15.32:0/567725021 learned_addr learned my addr 172.21.15.32:0/567725021 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.136+0000 7f56d1400640 1 --2- 172.21.15.32:0/567725021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.136+0000 7f56da800640 1 --2- 172.21.15.32:0/567725021 >> v2:172.21.15.39:3300/0 conn(0x7f56dc06afb0 0x7f56dc06b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.137+0000 7f56d1400640 1 -- 172.21.15.32:0/567725021 >> v2:172.21.15.39:3300/0 conn(0x7f56dc06afb0 msgr2=0x7f56dc06b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.137+0000 7f56d1400640 1 --2- 172.21.15.32:0/567725021 >> v2:172.21.15.39:3300/0 conn(0x7f56dc06afb0 0x7f56dc06b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.137+0000 7f56d1400640 1 -- 172.21.15.32:0/567725021 >> v1:172.21.15.39:6789/0 conn(0x7f56dc15e200 legacy=0x7f56dc0af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.137+0000 7f56d1400640 1 -- 172.21.15.32:0/567725021 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0ae660 con 0x7f56dc0afdf0 2024-03-06T07:37:11.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.137+0000 7f56d9400640 1 -- 172.21.15.32:0/567725021 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263508209 0 0) 0x7f56dc06bec0 con 0x7f56dc15e200 2024-03-06T07:37:11.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.138+0000 7f56d1400640 1 --2- 172.21.15.32:0/567725021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc0b3de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f56c40097a0 tx=0x7f56c402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bfec767d7b1309d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56d9400640 1 -- 172.21.15.32:0/567725021 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4035020 con 0x7f56dc0afdf0 2024-03-06T07:37:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56d9400640 1 -- 172.21.15.32:0/567725021 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c4031830 con 0x7f56dc0afdf0 2024-03-06T07:37:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56d9400640 1 -- 172.21.15.32:0/567725021 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4031c00 con 0x7f56dc0afdf0 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 -- 172.21.15.32:0/567725021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 msgr2=0x7f56dc0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 --2- 172.21.15.32:0/567725021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc0b3de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f56c40097a0 tx=0x7f56c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 -- 172.21.15.32:0/567725021 shutdown_connections 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 --2- 172.21.15.32:0/567725021 >> v2:172.21.15.39:3300/0 conn(0x7f56dc06afb0 0x7f56dc06b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 --2- 172.21.15.32:0/567725021 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc0b3de0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 -- 172.21.15.32:0/567725021 >> 172.21.15.32:0/567725021 conn(0x7f56dc0b2ae0 msgr2=0x7f56dc0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.139+0000 7f56e0e00640 1 -- 172.21.15.32:0/567725021 shutdown_connections 2024-03-06T07:37:11.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.140+0000 7f56e0e00640 1 -- 172.21.15.32:0/567725021 wait complete. 2024-03-06T07:37:11.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.142+0000 7f56e0e00640 1 Processor -- start 2024-03-06T07:37:11.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.142+0000 7f56e0e00640 1 -- start start 2024-03-06T07:37:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.143+0000 7f56e0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 0x7f56dc15e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56e0e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc1f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56e0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56dc15e200 0x7f56dc1f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56e0e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc06dbb0 con 0x7f56dc06afb0 2024-03-06T07:37:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56e0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0aec20 con 0x7f56dc15e200 2024-03-06T07:37:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56e0e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc06dd30 con 0x7f56dc0afdf0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 0x7f56dc15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 0x7f56dc15e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44906/0 (socket says 172.21.15.32:44906) 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 -- 172.21.15.32:0/1359057358 learned_addr learned my addr 172.21.15.32:0/1359057358 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d1400640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc1f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56da800640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56dc15e200 0x7f56dc1f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 -- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 msgr2=0x7f56dc1f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc1f79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 -- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56dc15e200 msgr2=0x7f56dc1f7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56dc15e200 0x7f56dc1f7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.144+0000 7f56d9e00640 1 -- 172.21.15.32:0/1359057358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0b0890 con 0x7f56dc06afb0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.145+0000 7f56d9e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 0x7f56dc15e650 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00cb00 tx=0x7f56cc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.145+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00aca0 con 0x7f56dc06afb0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.145+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc00ae40 con 0x7f56dc06afb0 2024-03-06T07:37:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.145+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc1611b0 con 0x7f56dc06afb0 2024-03-06T07:37:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.145+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc012640 con 0x7f56dc06afb0 2024-03-06T07:37:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.145+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc0b0e10 con 0x7f56dc06afb0 2024-03-06T07:37:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.146+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc0b5ee0 con 0x7f56dc06afb0 2024-03-06T07:37:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.147+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56cc0127e0 con 0x7f56dc06afb0 2024-03-06T07:37:11.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.148+0000 7f56d2a00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.148+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56cc08a930 con 0x7f56dc06afb0 2024-03-06T07:37:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.149+0000 7f56d1400640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.149+0000 7f56d1400640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56b806b4d0 0x7f56b806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f56c4036000 tx=0x7f56c4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.153+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56dc0b5ee0 con 0x7f56dc06afb0 2024-03-06T07:37:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.393+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56dc1614f0 con 0x7f56dc06afb0 2024-03-06T07:37:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.394+0000 7f56d2a00640 1 -- 172.21.15.32:0/1359057358 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4992 (secure 0 0 0) 0x7f56cc059710 con 0x7f56dc06afb0 2024-03-06T07:37:11.396 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:11.396 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/4261872692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4261872692},{"type":"v1","addr":"172.21.15.32:6837","nonce":4261872692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":5894,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/304553199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":304553199},{"type":"v1","addr":"172.21.15.39:6835","nonce":304553199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:05.894136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[6574,8313]},"id":9}]} 2024-03-06T07:37:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 94 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.396+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56b806b4d0 msgr2=0x7f56b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.396+0000 7f56e0e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56b806b4d0 0x7f56b806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f56c4036000 tx=0x7f56c4008030 comp rx=0 tx=0).stop 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.396+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 msgr2=0x7f56dc15e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.396+0000 7f56e0e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 0x7f56dc15e650 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00cb00 tx=0x7f56cc00a060 comp rx=0 tx=0).stop 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 shutdown_connections 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56dc15e200 0x7f56dc1f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f56dc0afdf0 0x7f56dc1f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 --2- 172.21.15.32:0/1359057358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56dc06afb0 0x7f56dc15e650 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 >> 172.21.15.32:0/1359057358 conn(0x7f56dc0b2ae0 msgr2=0x7f56dc06e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 shutdown_connections 2024-03-06T07:37:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.397+0000 7f56e0e00640 1 -- 172.21.15.32:0/1359057358 wait complete. 2024-03-06T07:37:11.418 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and b 2024-03-06T07:37:11.419 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:37:11.419 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:37:11.436 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:37:11.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:37:11.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.699+0000 7ff552a00640 1 Processor -- start 2024-03-06T07:37:11.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.699+0000 7ff552a00640 1 -- start start 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff552a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c161670 0x7ff54c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff552a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff54c06e240 0x7ff54c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff552a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff54c06cfe0 0x7ff54c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff552a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c0730d0 con 0x7ff54c06e240 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff552a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c0ce9a0 con 0x7ff54c06cfe0 2024-03-06T07:37:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff552a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0cec80 con 0x7ff54c161670 2024-03-06T07:37:11.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff54b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff54c06e240 0x7ff54c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff54b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff54c06e240 0x7ff54c0711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44910/0 (socket says 172.21.15.32:44910) 2024-03-06T07:37:11.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff54b400640 1 -- 172.21.15.32:0/329517941 learned_addr learned my addr 172.21.15.32:0/329517941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.701+0000 7ff550a00640 1 --2- 172.21.15.32:0/329517941 >> v2:172.21.15.39:3300/0 conn(0x7ff54c06cfe0 0x7ff54c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.702+0000 7ff54be00640 1 --2- 172.21.15.32:0/329517941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c161670 0x7ff54c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.702+0000 7ff54b400640 1 -- 172.21.15.32:0/329517941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c161670 msgr2=0x7ff54c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.702+0000 7ff54b400640 1 --2- 172.21.15.32:0/329517941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c161670 0x7ff54c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.702+0000 7ff54b400640 1 -- 172.21.15.32:0/329517941 >> v2:172.21.15.39:3300/0 conn(0x7ff54c06cfe0 msgr2=0x7ff54c06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.702+0000 7ff54b400640 1 --2- 172.21.15.32:0/329517941 >> v2:172.21.15.39:3300/0 conn(0x7ff54c06cfe0 0x7ff54c06d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.702+0000 7ff54b400640 1 -- 172.21.15.32:0/329517941 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0cef60 con 0x7ff54c06e240 2024-03-06T07:37:11.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.703+0000 7ff54b400640 1 --2- 172.21.15.32:0/329517941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c06e240 0x7ff54c0711b0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7ff54c06bdc0 tx=0x7ff53802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660f183f39c6f024 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.704+0000 7ff54aa00640 1 -- 172.21.15.32:0/329517941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538035020 con 0x7ff54c06e240 2024-03-06T07:37:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.704+0000 7ff54aa00640 1 -- 172.21.15.32:0/329517941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53802d040 con 0x7ff54c06e240 2024-03-06T07:37:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.704+0000 7ff54aa00640 1 -- 172.21.15.32:0/329517941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538031a80 con 0x7ff54c06e240 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 -- 172.21.15.32:0/329517941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c06e240 msgr2=0x7ff54c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 --2- 172.21.15.32:0/329517941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c06e240 0x7ff54c0711b0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7ff54c06bdc0 tx=0x7ff53802b900 comp rx=0 tx=0).stop 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 -- 172.21.15.32:0/329517941 shutdown_connections 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 --2- 172.21.15.32:0/329517941 >> v2:172.21.15.39:3300/0 conn(0x7ff54c06cfe0 0x7ff54c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 --2- 172.21.15.32:0/329517941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c06e240 0x7ff54c0711b0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 --2- 172.21.15.32:0/329517941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c161670 0x7ff54c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 -- 172.21.15.32:0/329517941 >> 172.21.15.32:0/329517941 conn(0x7ff54c06afb0 msgr2=0x7ff54c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 -- 172.21.15.32:0/329517941 shutdown_connections 2024-03-06T07:37:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.705+0000 7ff552a00640 1 -- 172.21.15.32:0/329517941 wait complete. 2024-03-06T07:37:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.707+0000 7ff552a00640 1 Processor -- start 2024-03-06T07:37:11.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.707+0000 7ff552a00640 1 -- start start 2024-03-06T07:37:11.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.708+0000 7ff552a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c06cfe0 0x7ff54c0ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.708+0000 7ff552a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54c06e240 0x7ff54c1561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.708+0000 7ff552a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 0x7ff54c09c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.708+0000 7ff552a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0ce600 con 0x7ff54c161670 2024-03-06T07:37:11.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.708+0000 7ff552a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0cf520 con 0x7ff54c06e240 2024-03-06T07:37:11.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.708+0000 7ff552a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0ce780 con 0x7ff54c06cfe0 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 0x7ff54c09c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 0x7ff54c09c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44916/0 (socket says 172.21.15.32:44916) 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 -- 172.21.15.32:0/2625821955 learned_addr learned my addr 172.21.15.32:0/2625821955 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff54be00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c06cfe0 0x7ff54c0ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff54b400640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54c06e240 0x7ff54c1561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 -- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c06cfe0 msgr2=0x7ff54c0ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c06cfe0 0x7ff54c0ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 -- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54c06e240 msgr2=0x7ff54c1561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54c06e240 0x7ff54c1561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.709+0000 7ff550a00640 1 -- 172.21.15.32:0/2625821955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c09d530 con 0x7ff54c161670 2024-03-06T07:37:11.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.710+0000 7ff550a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 0x7ff54c09c7b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7ff54400ef00 tx=0x7ff54400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.710+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544018020 con 0x7ff54c161670 2024-03-06T07:37:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.710+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544014400 con 0x7ff54c161670 2024-03-06T07:37:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.710+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440147f0 con 0x7ff54c161670 2024-03-06T07:37:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.710+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c0c9c70 con 0x7ff54c161670 2024-03-06T07:37:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.710+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c09d810 con 0x7ff54c161670 2024-03-06T07:37:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.711+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c069510 con 0x7ff54c161670 2024-03-06T07:37:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.712+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff544005d10 con 0x7ff54c161670 2024-03-06T07:37:11.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.714+0000 7ff543e00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff52c06b4d0 0x7ff52c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.714+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff54408a910 con 0x7ff54c161670 2024-03-06T07:37:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.714+0000 7ff54be00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff52c06b4d0 0x7ff52c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.715+0000 7ff54be00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff52c06b4d0 0x7ff52c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff53c006fd0 tx=0x7ff53c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:11.718+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff544022070 con 0x7ff54c161670 2024-03-06T07:37:12.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.006+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7ff54c04f3a0 con 0x7ff54c161670 2024-03-06T07:37:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.954+0000 7ff543e00640 1 -- 172.21.15.32:0/2625821955 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 5888 v95) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff544059790 con 0x7ff54c161670 2024-03-06T07:37:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 5888 2024-03-06T07:37:12.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.958+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff52c06b4d0 msgr2=0x7ff52c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff52c06b4d0 0x7ff52c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff53c006fd0 tx=0x7ff53c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 msgr2=0x7ff54c09c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 0x7ff54c09c7b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7ff54400ef00 tx=0x7ff54400b480 comp rx=0 tx=0).stop 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 shutdown_connections 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff52c06b4d0 0x7ff52c06d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff54c161670 0x7ff54c09c7b0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff54c06e240 0x7ff54c1561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 --2- 172.21.15.32:0/2625821955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff54c06cfe0 0x7ff54c0ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 >> 172.21.15.32:0/2625821955 conn(0x7ff54c06afb0 msgr2=0x7ff54c06d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 shutdown_connections 2024-03-06T07:37:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:12.959+0000 7ff552a00640 1 -- 172.21.15.32:0/2625821955 wait complete. 2024-03-06T07:37:12.982 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:37:12.982 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:37:13.013 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:37:13.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:37:13.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.224+0000 7f75a3e00640 1 Processor -- start 2024-03-06T07:37:13.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.225+0000 7f75a3e00640 1 -- start start 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.226+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a4066d00 0x7f75a406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.226+0000 7f75a3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f75a40ce370 0x7f75a40ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.226+0000 7f75a3e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a40cf1b0 con 0x7f75a406c8b0 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.226+0000 7f75a3e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a415e470 con 0x7f75a40ce370 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.226+0000 7f75a3e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a415ea00 con 0x7f75a4066d00 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.227+0000 7f75a3400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f75a40ce370 0x7f75a40ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.227+0000 7f759be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a4066d00 0x7f75a406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.227+0000 7f75a3400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f75a40ce370 0x7f75a40ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:48532/0 (socket says 172.21.15.32:48532) 2024-03-06T07:37:13.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.227+0000 7f75a3400640 1 -- 172.21.15.32:0/1280493677 learned_addr learned my addr 172.21.15.32:0/1280493677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:13.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.228+0000 7f75a3400640 1 -- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a4066d00 msgr2=0x7f75a406ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.228+0000 7f75a3400640 1 --2- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a4066d00 0x7f75a406ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.228+0000 7f75a3400640 1 -- 172.21.15.32:0/1280493677 >> v1:172.21.15.39:6789/0 conn(0x7f75a406c8b0 legacy=0x7f75a415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.228+0000 7f75a3400640 1 -- 172.21.15.32:0/1280493677 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a406d420 con 0x7f75a40ce370 2024-03-06T07:37:13.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.229+0000 7f75a3400640 1 --2- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 0x7f75a40ce7e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f759c009e50 tx=0x7f759c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5db037ba419fbed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a2000640 1 -- 172.21.15.32:0/1280493677 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c02cb00 con 0x7f75a40ce370 2024-03-06T07:37:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a2000640 1 -- 172.21.15.32:0/1280493677 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c02cca0 con 0x7f75a40ce370 2024-03-06T07:37:13.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a2000640 1 -- 172.21.15.32:0/1280493677 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c034610 con 0x7f75a40ce370 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 -- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 msgr2=0x7f75a40ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 --2- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 0x7f75a40ce7e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f759c009e50 tx=0x7f759c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 -- 172.21.15.32:0/1280493677 shutdown_connections 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 --2- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 0x7f75a40ce7e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 --2- 172.21.15.32:0/1280493677 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a4066d00 0x7f75a406ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 -- 172.21.15.32:0/1280493677 >> 172.21.15.32:0/1280493677 conn(0x7f75a40ccf70 msgr2=0x7f75a40cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 -- 172.21.15.32:0/1280493677 shutdown_connections 2024-03-06T07:37:13.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.230+0000 7f75a3e00640 1 -- 172.21.15.32:0/1280493677 wait complete. 2024-03-06T07:37:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.232+0000 7f75a3e00640 1 Processor -- start 2024-03-06T07:37:13.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.232+0000 7f75a3e00640 1 -- start start 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 0x7f75a4155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a406c8b0 0x7f75a4158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 0x7f75a415b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40ceea0 con 0x7f75a4066d00 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40ced20 con 0x7f75a40ce370 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a3e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a406afb0 con 0x7f75a406c8b0 2024-03-06T07:37:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.233+0000 7f75a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 0x7f75a4155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 0x7f75a4155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44930/0 (socket says 172.21.15.32:44930) 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 -- 172.21.15.32:0/2386867607 learned_addr learned my addr 172.21.15.32:0/2386867607 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f759be00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a406c8b0 0x7f75a4158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 -- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a406c8b0 msgr2=0x7f75a4158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a406c8b0 0x7f75a4158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 -- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 msgr2=0x7f75a415b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 0x7f75a415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 -- 172.21.15.32:0/2386867607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a415ee30 con 0x7f75a4066d00 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.234+0000 7f75a2a00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 0x7f75a4155f90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f759400c450 tx=0x7f759400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.235+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940052d0 con 0x7f75a4066d00 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.235+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7594005470 con 0x7f75a4066d00 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.235+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7594012870 con 0x7f75a4066d00 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.235+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a415f0f0 con 0x7f75a4066d00 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.235+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a4159430 con 0x7f75a4066d00 2024-03-06T07:37:13.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.236+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a404f330 con 0x7f75a4066d00 2024-03-06T07:37:13.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.237+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7594012a10 con 0x7f75a4066d00 2024-03-06T07:37:13.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.238+0000 7f759b400640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f758006b4d0 0x7f758006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.238+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7594089e80 con 0x7f75a4066d00 2024-03-06T07:37:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.239+0000 7f759be00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f758006b4d0 0x7f758006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.239+0000 7f759be00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f758006b4d0 0x7f758006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f75a415ad70 tx=0x7f758c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.243+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7594058c60 con 0x7f75a4066d00 2024-03-06T07:37:13.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.520+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f75a40cdde0 con 0x7f75a4066d00 2024-03-06T07:37:13.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.957+0000 7f759b400640 1 -- 172.21.15.32:0/2386867607 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5894 v96) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7594058560 con 0x7f75a4066d00 2024-03-06T07:37:13.959 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 5894 2024-03-06T07:37:13.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.960+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f758006b4d0 msgr2=0x7f758006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.960+0000 7f75a3e00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f758006b4d0 0x7f758006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f75a415ad70 tx=0x7f758c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:13.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.960+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 msgr2=0x7f75a4155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.960+0000 7f75a3e00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 0x7f75a4155f90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f759400c450 tx=0x7f759400c910 comp rx=0 tx=0).stop 2024-03-06T07:37:13.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 shutdown_connections 2024-03-06T07:37:13.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f758006b4d0 0x7f758006d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75a40ce370 0x7f75a415b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f75a406c8b0 0x7f75a4158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 --2- 172.21.15.32:0/2386867607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75a4066d00 0x7f75a4155f90 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 >> 172.21.15.32:0/2386867607 conn(0x7f75a40ccf70 msgr2=0x7f75a40b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:13.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 shutdown_connections 2024-03-06T07:37:13.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:13.961+0000 7f75a3e00640 1 -- 172.21.15.32:0/2386867607 wait complete. 2024-03-06T07:37:13.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:13.985 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:37:13.985 DEBUG:teuthology.orchestra.run.smithi032:> mkdir pin 2024-03-06T07:37:13.985 DEBUG:teuthology.orchestra.run.smithi032:> touch pin/placeholder 2024-03-06T07:37:13.985 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-03-06T07:37:14.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:14.058 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:37:14.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:14.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.326+0000 7f2606000640 1 Processor -- start 2024-03-06T07:37:14.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.326+0000 7f2606000640 1 -- start start 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.328+0000 7f2606000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2600147870 0x7f26000b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.328+0000 7f2606000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.328+0000 7f2606000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26000ad880 0x7f260006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.328+0000 7f2606000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2600147c50 con 0x7f26000ad880 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.328+0000 7f2606000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26000b3b60 con 0x7f2600147870 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.328+0000 7f2606000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260006fb90 con 0x7f260006e710 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26000ad880 0x7f260006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f26000ad880 0x7f260006b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44940/0 (socket says 172.21.15.32:44940) 2024-03-06T07:37:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 -- 172.21.15.32:0/763888468 learned_addr learned my addr 172.21.15.32:0/763888468 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2604c00640 1 --2- 172.21.15.32:0/763888468 >> v2:172.21.15.39:3300/0 conn(0x7f2600147870 0x7f26000b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f25f7e00640 1 --2- 172.21.15.32:0/763888468 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 -- 172.21.15.32:0/763888468 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 msgr2=0x7f260006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 --2- 172.21.15.32:0/763888468 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 -- 172.21.15.32:0/763888468 >> v2:172.21.15.39:3300/0 conn(0x7f2600147870 msgr2=0x7f26000b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 --2- 172.21.15.32:0/763888468 >> v2:172.21.15.39:3300/0 conn(0x7f2600147870 0x7f26000b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.329+0000 7f2605600640 1 -- 172.21.15.32:0/763888468 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260006ebf0 con 0x7f26000ad880 2024-03-06T07:37:14.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.330+0000 7f2605600640 1 --2- 172.21.15.32:0/763888468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26000ad880 0x7f260006b4f0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f2600073420 tx=0x7f25f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc24bbefaf007cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.331+0000 7f25ffe00640 1 -- 172.21.15.32:0/763888468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f802ca70 con 0x7f26000ad880 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.331+0000 7f25ffe00640 1 -- 172.21.15.32:0/763888468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f802cc10 con 0x7f26000ad880 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.331+0000 7f25ffe00640 1 -- 172.21.15.32:0/763888468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f80343f0 con 0x7f26000ad880 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.331+0000 7f2606000640 1 -- 172.21.15.32:0/763888468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26000ad880 msgr2=0x7f260006b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.331+0000 7f2606000640 1 --2- 172.21.15.32:0/763888468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26000ad880 0x7f260006b4f0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f2600073420 tx=0x7f25f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 -- 172.21.15.32:0/763888468 shutdown_connections 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 --2- 172.21.15.32:0/763888468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26000ad880 0x7f260006b4f0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 --2- 172.21.15.32:0/763888468 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 --2- 172.21.15.32:0/763888468 >> v2:172.21.15.39:3300/0 conn(0x7f2600147870 0x7f26000b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 -- 172.21.15.32:0/763888468 >> 172.21.15.32:0/763888468 conn(0x7f260006c0c0 msgr2=0x7f260006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 -- 172.21.15.32:0/763888468 shutdown_connections 2024-03-06T07:37:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.332+0000 7f2606000640 1 -- 172.21.15.32:0/763888468 wait complete. 2024-03-06T07:37:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.334+0000 7f2606000640 1 Processor -- start 2024-03-06T07:37:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.334+0000 7f2606000640 1 -- start start 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2606000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260013b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2606000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26000ad880 0x7f260015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2606000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 0x7f26000b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2606000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26000b4370 con 0x7f2600147870 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2606000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26000b4090 con 0x7f26000ad880 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2606000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26000b5da0 con 0x7f260006e710 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2605600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 0x7f26000b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2605600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 0x7f26000b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44956/0 (socket says 172.21.15.32:44956) 2024-03-06T07:37:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2605600640 1 -- 172.21.15.32:0/2638409172 learned_addr learned my addr 172.21.15.32:0/2638409172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f2604c00640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260013b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.335+0000 7f25f7e00640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26000ad880 0x7f260015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2605600640 1 -- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 msgr2=0x7f260013b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2605600640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260013b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2605600640 1 -- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26000ad880 msgr2=0x7f260015e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2605600640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26000ad880 0x7f260015e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2605600640 1 -- 172.21.15.32:0/2638409172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260015ed90 con 0x7f2600147870 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f25f7e00640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26000ad880 0x7f260015e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2605600640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 0x7f26000b1040 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f25f8036040 tx=0x7f25f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8043020 con 0x7f2600147870 2024-03-06T07:37:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f803e450 con 0x7f2600147870 2024-03-06T07:37:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8009510 con 0x7f2600147870 2024-03-06T07:37:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.336+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260015f070 con 0x7f2600147870 2024-03-06T07:37:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.337+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260015fd60 con 0x7f2600147870 2024-03-06T07:37:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.338+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260004f330 con 0x7f2600147870 2024-03-06T07:37:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.339+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25f8046020 con 0x7f2600147870 2024-03-06T07:37:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.340+0000 7f25fd600640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25dc06b4d0 0x7f25dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.340+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25f80acaf0 con 0x7f2600147870 2024-03-06T07:37:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.340+0000 7f2604c00640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25dc06b4d0 0x7f25dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.341+0000 7f2604c00640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25dc06b4d0 0x7f25dc06d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f25f0006fd0 tx=0x7f25f0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.344+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25f804d030 con 0x7f2600147870 2024-03-06T07:37:14.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.590+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f260015ff30 con 0x7f2600147870 2024-03-06T07:37:14.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.591+0000 7f25fd600640 1 -- 172.21.15.32:0/2638409172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3414 (secure 0 0 0) 0x7f25f807b8d0 con 0x7f2600147870 2024-03-06T07:37:14.593 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:14.594 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:05.894136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[6574,8313]},"id":9}]} 2024-03-06T07:37:14.594 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 96 2024-03-06T07:37:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25dc06b4d0 msgr2=0x7f25dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25dc06b4d0 0x7f25dc06d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f25f0006fd0 tx=0x7f25f0008040 comp rx=0 tx=0).stop 2024-03-06T07:37:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 msgr2=0x7f26000b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 0x7f26000b1040 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f25f8036040 tx=0x7f25f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 shutdown_connections 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f25dc06b4d0 0x7f25dc06d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2600147870 0x7f26000b1040 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26000ad880 0x7f260015e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 --2- 172.21.15.32:0/2638409172 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f260006e710 0x7f260013b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 >> 172.21.15.32:0/2638409172 conn(0x7f260006c0c0 msgr2=0x7f2600145bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 shutdown_connections 2024-03-06T07:37:14.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:14.594+0000 7f2606000640 1 -- 172.21.15.32:0/2638409172 wait complete. 2024-03-06T07:37:14.616 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:37:14.862 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:37:14.862 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:14.862 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:37:16.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.089+0000 7f0a4bc00640 1 Processor -- start 2024-03-06T07:37:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.089+0000 7f0a4bc00640 1 -- start start 2024-03-06T07:37:17.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.091+0000 7f0a4bc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a4415a930 0x7f0a440ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.091+0000 7f0a4bc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0a440ce9f0 0x7f0a440ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.091+0000 7f0a4bc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a44071640 con 0x7f0a4415a550 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.091+0000 7f0a4bc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440af6f0 con 0x7f0a4415a930 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.091+0000 7f0a4bc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440cee30 con 0x7f0a440ce9f0 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a4415a930 0x7f0a440ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a4415a930 0x7f0a440ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44964/0 (socket says 172.21.15.32:44964) 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 -- 172.21.15.32:0/2550361554 learned_addr learned my addr 172.21.15.32:0/2550361554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a49c00640 1 --2- 172.21.15.32:0/2550361554 >> v2:172.21.15.39:3300/0 conn(0x7f0a440ce9f0 0x7f0a440ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 -- 172.21.15.32:0/2550361554 >> v2:172.21.15.39:3300/0 conn(0x7f0a440ce9f0 msgr2=0x7f0a440ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 --2- 172.21.15.32:0/2550361554 >> v2:172.21.15.39:3300/0 conn(0x7f0a440ce9f0 0x7f0a440ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 -- 172.21.15.32:0/2550361554 >> v1:172.21.15.39:6789/0 conn(0x7f0a4415a550 legacy=0x7f0a440aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.092+0000 7f0a3be00640 1 -- 172.21.15.32:0/2550361554 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a44071920 con 0x7f0a4415a930 2024-03-06T07:37:17.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.094+0000 7f0a3be00640 1 --2- 172.21.15.32:0/2550361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a930 0x7f0a440ce4b0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f0a440cfcc0 tx=0x7f0a2c02df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ab2ce5a975200f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.094+0000 7f0a43e00640 1 -- 172.21.15.32:0/2550361554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c034070 con 0x7f0a4415a930 2024-03-06T07:37:17.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.094+0000 7f0a43e00640 1 -- 172.21.15.32:0/2550361554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c02ed00 con 0x7f0a4415a930 2024-03-06T07:37:17.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.094+0000 7f0a43e00640 1 -- 172.21.15.32:0/2550361554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c038620 con 0x7f0a4415a930 2024-03-06T07:37:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 -- 172.21.15.32:0/2550361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a930 msgr2=0x7f0a440ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/2550361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a930 0x7f0a440ce4b0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f0a440cfcc0 tx=0x7f0a2c02df50 comp rx=0 tx=0).stop 2024-03-06T07:37:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 -- 172.21.15.32:0/2550361554 shutdown_connections 2024-03-06T07:37:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/2550361554 >> v2:172.21.15.39:3300/0 conn(0x7f0a440ce9f0 0x7f0a440ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/2550361554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a930 0x7f0a440ce4b0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 -- 172.21.15.32:0/2550361554 >> 172.21.15.32:0/2550361554 conn(0x7f0a4406afb0 msgr2=0x7f0a4406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 -- 172.21.15.32:0/2550361554 shutdown_connections 2024-03-06T07:37:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.095+0000 7f0a4bc00640 1 -- 172.21.15.32:0/2550361554 wait complete. 2024-03-06T07:37:17.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.098+0000 7f0a4bc00640 1 Processor -- start 2024-03-06T07:37:17.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.098+0000 7f0a4bc00640 1 -- start start 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a4bc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a440ce9f0 0x7f0a4414dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a4bc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 0x7f0a4409c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a4bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4415a930 0x7f0a44153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a4bc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440b5130 con 0x7f0a4415a550 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a4bc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44071ee0 con 0x7f0a4415a930 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a4bc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440b52b0 con 0x7f0a440ce9f0 2024-03-06T07:37:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 0x7f0a4409c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 0x7f0a4409c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44968/0 (socket says 172.21.15.32:44968) 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a3be00640 1 -- 172.21.15.32:0/3032970817 learned_addr learned my addr 172.21.15.32:0/3032970817 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a49c00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4415a930 0x7f0a44153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.100+0000 7f0a49200640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a440ce9f0 0x7f0a4414dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a3be00640 1 -- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a440ce9f0 msgr2=0x7f0a4414dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a3be00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a440ce9f0 0x7f0a4414dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a3be00640 1 -- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4415a930 msgr2=0x7f0a44153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a3be00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4415a930 0x7f0a44153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a3be00640 1 -- 172.21.15.32:0/3032970817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4414e0f0 con 0x7f0a4415a550 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a49c00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4415a930 0x7f0a44153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a3be00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 0x7f0a4409c930 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c03a040 tx=0x7f0a2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c02eb00 con 0x7f0a4415a550 2024-03-06T07:37:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c0415a0 con 0x7f0a4415a550 2024-03-06T07:37:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c041990 con 0x7f0a4415a550 2024-03-06T07:37:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.101+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4415ae40 con 0x7f0a4415a550 2024-03-06T07:37:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.102+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4415b400 con 0x7f0a4415a550 2024-03-06T07:37:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.103+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4404f3a0 con 0x7f0a4415a550 2024-03-06T07:37:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.103+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a2c049020 con 0x7f0a4415a550 2024-03-06T07:37:17.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.104+0000 7f0a41600640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a2006b500 0x7f0a2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.105+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a2c0ac580 con 0x7f0a4415a550 2024-03-06T07:37:17.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.105+0000 7f0a49200640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a2006b500 0x7f0a2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.106+0000 7f0a49200640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a2006b500 0x7f0a2006d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0a34003d90 tx=0x7f0a34009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.113+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a2c050020 con 0x7f0a4415a550 2024-03-06T07:37:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.354+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a4409d170 con 0x7f0a4415a550 2024-03-06T07:37:17.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.355+0000 7f0a41600640 1 -- 172.21.15.32:0/3032970817 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3414 (secure 0 0 0) 0x7f0a2c07b3f0 con 0x7f0a4415a550 2024-03-06T07:37:17.357 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:17.357 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:05.894136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[6574,8313]},"id":9}]} 2024-03-06T07:37:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 96 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a2006b500 msgr2=0x7f0a2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a2006b500 0x7f0a2006d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0a34003d90 tx=0x7f0a34009040 comp rx=0 tx=0).stop 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 msgr2=0x7f0a4409c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 0x7f0a4409c930 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c03a040 tx=0x7f0a2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 shutdown_connections 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a2006b500 0x7f0a2006d9c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a4415a930 0x7f0a44153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a4415a550 0x7f0a4409c930 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 --2- 172.21.15.32:0/3032970817 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a440ce9f0 0x7f0a4414dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.357+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 >> 172.21.15.32:0/3032970817 conn(0x7f0a4406afb0 msgr2=0x7f0a44159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.358+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 shutdown_connections 2024-03-06T07:37:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:17.358+0000 7f0a4bc00640 1 -- 172.21.15.32:0/3032970817 wait complete. 2024-03-06T07:37:17.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:37:17.623 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:37:17.624 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:17.624 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:37:19.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.855+0000 7ff962e00640 1 Processor -- start 2024-03-06T07:37:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.856+0000 7ff962e00640 1 -- start start 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.857+0000 7ff962e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 0x7ff95c0b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.857+0000 7ff962e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff95c0b6000 0x7ff95c162750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.858+0000 7ff962e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95c0b4690 con 0x7ff95c162c90 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.858+0000 7ff962e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c1639f0 con 0x7ff95c0b6000 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.858+0000 7ff962e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c163430 con 0x7ff95c0b3d70 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.858+0000 7ff95be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 0x7ff95c0b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.858+0000 7ff953e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff95c0b6000 0x7ff95c162750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff95be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 0x7ff95c0b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:51960/0 (socket says 172.21.15.32:51960) 2024-03-06T07:37:19.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff95be00640 1 -- 172.21.15.32:0/2997030781 learned_addr learned my addr 172.21.15.32:0/2997030781 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:19.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff95be00640 1 -- 172.21.15.32:0/2997030781 >> v2:172.21.15.39:3300/0 conn(0x7ff95c0b6000 msgr2=0x7ff95c162750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:19.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff95be00640 1 --2- 172.21.15.32:0/2997030781 >> v2:172.21.15.39:3300/0 conn(0x7ff95c0b6000 0x7ff95c162750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff95be00640 1 -- 172.21.15.32:0/2997030781 >> v1:172.21.15.39:6789/0 conn(0x7ff95c162c90 legacy=0x7ff95c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff95be00640 1 -- 172.21.15.32:0/2997030781 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c163710 con 0x7ff95c0b3d70 2024-03-06T07:37:19.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.859+0000 7ff953e00640 1 --2- 172.21.15.32:0/2997030781 >> v2:172.21.15.39:3300/0 conn(0x7ff95c0b6000 0x7ff95c162750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:19.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.861+0000 7ff95be00640 1 --2- 172.21.15.32:0/2997030781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 0x7ff95c0b4150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009870 tx=0x7ff94c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83c93cfe126743a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.861+0000 7ff95b400640 1 -- 172.21.15.32:0/2997030781 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c034070 con 0x7ff95c0b3d70 2024-03-06T07:37:19.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.861+0000 7ff95b400640 1 -- 172.21.15.32:0/2997030781 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c030840 con 0x7ff95c0b3d70 2024-03-06T07:37:19.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.861+0000 7ff95b400640 1 -- 172.21.15.32:0/2997030781 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c030c10 con 0x7ff95c0b3d70 2024-03-06T07:37:19.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 -- 172.21.15.32:0/2997030781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 msgr2=0x7ff95c0b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 --2- 172.21.15.32:0/2997030781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 0x7ff95c0b4150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009870 tx=0x7ff94c02b980 comp rx=0 tx=0).stop 2024-03-06T07:37:19.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 -- 172.21.15.32:0/2997030781 shutdown_connections 2024-03-06T07:37:19.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 --2- 172.21.15.32:0/2997030781 >> v2:172.21.15.39:3300/0 conn(0x7ff95c0b6000 0x7ff95c162750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 --2- 172.21.15.32:0/2997030781 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b3d70 0x7ff95c0b4150 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 -- 172.21.15.32:0/2997030781 >> 172.21.15.32:0/2997030781 conn(0x7ff95c06c2a0 msgr2=0x7ff95c06c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:19.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.862+0000 7ff962e00640 1 -- 172.21.15.32:0/2997030781 shutdown_connections 2024-03-06T07:37:19.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.863+0000 7ff962e00640 1 -- 172.21.15.32:0/2997030781 wait complete. 2024-03-06T07:37:19.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.865+0000 7ff962e00640 1 Processor -- start 2024-03-06T07:37:19.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.865+0000 7ff962e00640 1 -- start start 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff962e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 0x7ff95c14bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff962e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b6000 0x7ff95c1e9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff962e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff95c162c90 0x7ff95c1608c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff962e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c163120 con 0x7ff95c0b3d70 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff962e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0ae0d0 con 0x7ff95c162c90 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff962e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0af600 con 0x7ff95c0b6000 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff95be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 0x7ff95c14bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.866+0000 7ff95be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 0x7ff95c14bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50138/0 (socket says 172.21.15.32:50138) 2024-03-06T07:37:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff95be00640 1 -- 172.21.15.32:0/1849186344 learned_addr learned my addr 172.21.15.32:0/1849186344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff953e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b6000 0x7ff95c1e9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff960e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff95c162c90 0x7ff95c1608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff95be00640 1 -- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b6000 msgr2=0x7ff95c1e9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff95be00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b6000 0x7ff95c1e9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff95be00640 1 -- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff95c162c90 msgr2=0x7ff95c1608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff95be00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff95c162c90 0x7ff95c1608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff95be00640 1 -- 172.21.15.32:0/1849186344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c1ead20 con 0x7ff95c0b3d70 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.867+0000 7ff960e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff95c162c90 0x7ff95c1608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.868+0000 7ff95be00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 0x7ff95c14bf80 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02b9c0 tx=0x7ff94c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.868+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c048070 con 0x7ff95c0b3d70 2024-03-06T07:37:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.868+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c030640 con 0x7ff95c0b3d70 2024-03-06T07:37:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.868+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c034040 con 0x7ff95c0b3d70 2024-03-06T07:37:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.868+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c1eb000 con 0x7ff95c0b3d70 2024-03-06T07:37:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.868+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c1eb5c0 con 0x7ff95c0b3d70 2024-03-06T07:37:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.870+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c04f3a0 con 0x7ff95c0b3d70 2024-03-06T07:37:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.871+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff94c037070 con 0x7ff95c0b3d70 2024-03-06T07:37:19.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.872+0000 7ff958c00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff93806b4d0 0x7ff93806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.872+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff94c0ac140 con 0x7ff95c0b3d70 2024-03-06T07:37:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.872+0000 7ff953e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff93806b4d0 0x7ff93806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.873+0000 7ff953e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff93806b4d0 0x7ff93806d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff95c1616e0 tx=0x7ff944008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:19.877+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff94c07b0e0 con 0x7ff95c0b3d70 2024-03-06T07:37:20.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.120+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff95c072120 con 0x7ff95c0b3d70 2024-03-06T07:37:20.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.121+0000 7ff958c00640 1 -- 172.21.15.32:0/1849186344 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3414 (secure 0 0 0) 0x7ff94c07a9e0 con 0x7ff95c0b3d70 2024-03-06T07:37:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:20.123 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:05.894136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[6574,8313]},"id":9}]} 2024-03-06T07:37:20.124 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 96 2024-03-06T07:37:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff93806b4d0 msgr2=0x7ff93806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff93806b4d0 0x7ff93806d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff95c1616e0 tx=0x7ff944008040 comp rx=0 tx=0).stop 2024-03-06T07:37:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 msgr2=0x7ff95c14bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 0x7ff95c14bf80 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02b9c0 tx=0x7ff94c004730 comp rx=0 tx=0).stop 2024-03-06T07:37:20.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 shutdown_connections 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff93806b4d0 0x7ff93806d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff95c162c90 0x7ff95c1608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff95c0b6000 0x7ff95c1e9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 --2- 172.21.15.32:0/1849186344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff95c0b3d70 0x7ff95c14bf80 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 >> 172.21.15.32:0/1849186344 conn(0x7ff95c06c2a0 msgr2=0x7ff95c0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 shutdown_connections 2024-03-06T07:37:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:20.125+0000 7ff962e00640 1 -- 172.21.15.32:0/1849186344 wait complete. 2024-03-06T07:37:20.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:37:20.395 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:37:20.396 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:20.396 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:37:20.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:20.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:22.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:22.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.620+0000 7fe73d200640 1 Processor -- start 2024-03-06T07:37:22.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.620+0000 7fe73d200640 1 -- start start 2024-03-06T07:37:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe73d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7380d2930 0x7fe7380b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe73d200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380cea50 con 0x7fe7380cfd90 2024-03-06T07:37:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe73d200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380cecf0 con 0x7fe738161b40 2024-03-06T07:37:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe73d200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7380b5770 con 0x7fe7380d2930 2024-03-06T07:37:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe72d400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7380cfd90 0x7fe7380d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43894/0 (socket says 172.21.15.32:43894) 2024-03-06T07:37:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe72d400640 1 -- 172.21.15.32:0/632529728 learned_addr learned my addr 172.21.15.32:0/632529728 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.622+0000 7fe736800640 1 --2- 172.21.15.32:0/632529728 >> v2:172.21.15.32:3300/0 conn(0x7fe7380d2930 0x7fe7380b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.623+0000 7fe735400640 1 -- 172.21.15.32:0/632529728 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830509273 0 0) 0x7fe7380cea50 con 0x7fe7380cfd90 2024-03-06T07:37:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.623+0000 7fe735400640 1 -- 172.21.15.32:0/632529728 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c003520 con 0x7fe7380cfd90 2024-03-06T07:37:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.623+0000 7fe736800640 1 -- 172.21.15.32:0/632529728 >> v1:172.21.15.39:6789/0 conn(0x7fe738161b40 legacy=0x7fe7380b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.623+0000 7fe736800640 1 -- 172.21.15.32:0/632529728 >> v1:172.21.15.32:6789/0 conn(0x7fe7380cfd90 legacy=0x7fe7380d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.623+0000 7fe736800640 1 -- 172.21.15.32:0/632529728 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7380cefd0 con 0x7fe7380d2930 2024-03-06T07:37:22.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.624+0000 7fe736800640 1 --2- 172.21.15.32:0/632529728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380d2930 0x7fe7380b3740 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fe7300097a0 tx=0x7fe73002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc5227e31a8344 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:22.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.625+0000 7fe735400640 1 -- 172.21.15.32:0/632529728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730034070 con 0x7fe7380d2930 2024-03-06T07:37:22.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.625+0000 7fe735400640 1 -- 172.21.15.32:0/632529728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe730030610 con 0x7fe7380d2930 2024-03-06T07:37:22.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.625+0000 7fe735400640 1 -- 172.21.15.32:0/632529728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7300309c0 con 0x7fe7380d2930 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.625+0000 7fe73d200640 1 -- 172.21.15.32:0/632529728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380d2930 msgr2=0x7fe7380b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.625+0000 7fe73d200640 1 --2- 172.21.15.32:0/632529728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380d2930 0x7fe7380b3740 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fe7300097a0 tx=0x7fe73002b960 comp rx=0 tx=0).stop 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.626+0000 7fe73d200640 1 -- 172.21.15.32:0/632529728 shutdown_connections 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.626+0000 7fe73d200640 1 --2- 172.21.15.32:0/632529728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380d2930 0x7fe7380b3740 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.626+0000 7fe73d200640 1 -- 172.21.15.32:0/632529728 >> 172.21.15.32:0/632529728 conn(0x7fe73809d220 msgr2=0x7fe7380d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.626+0000 7fe73d200640 1 -- 172.21.15.32:0/632529728 shutdown_connections 2024-03-06T07:37:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.626+0000 7fe73d200640 1 -- 172.21.15.32:0/632529728 wait complete. 2024-03-06T07:37:22.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.628+0000 7fe73d200640 1 Processor -- start 2024-03-06T07:37:22.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.629+0000 7fe73d200640 1 -- start start 2024-03-06T07:37:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.630+0000 7fe73d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 0x7fe7380cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.630+0000 7fe73d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7380d2930 0x7fe738150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.630+0000 7fe73d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe738161b40 0x7fe73814da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.630+0000 7fe73d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7380cf590 con 0x7fe7380cfd90 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.630+0000 7fe73d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7380cf900 con 0x7fe738161b40 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.630+0000 7fe73d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7380b2ff0 con 0x7fe7380d2930 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 0x7fe7380cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 0x7fe7380cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50158/0 (socket says 172.21.15.32:50158) 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 -- 172.21.15.32:0/1507740424 learned_addr learned my addr 172.21.15.32:0/1507740424 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe72d400640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7380d2930 0x7fe738150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe736800640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe738161b40 0x7fe73814da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 -- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7380d2930 msgr2=0x7fe738150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7380d2930 0x7fe738150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 -- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe738161b40 msgr2=0x7fe73814da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe738161b40 0x7fe73814da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe735e00640 1 -- 172.21.15.32:0/1507740424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7380b8a20 con 0x7fe7380cfd90 2024-03-06T07:37:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.631+0000 7fe736800640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe738161b40 0x7fe73814da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.632+0000 7fe735e00640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 0x7fe7380cb7e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c870 tx=0x7fe72800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.632+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728003b70 con 0x7fe7380cfd90 2024-03-06T07:37:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.632+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe728003d30 con 0x7fe7380cfd90 2024-03-06T07:37:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.632+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72800aa30 con 0x7fe7380cfd90 2024-03-06T07:37:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.632+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7380b8cc0 con 0x7fe7380cfd90 2024-03-06T07:37:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.632+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7380b9280 con 0x7fe7380cfd90 2024-03-06T07:37:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.633+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73804f3a0 con 0x7fe7380cfd90 2024-03-06T07:37:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.634+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe72800abd0 con 0x7fe7380cfd90 2024-03-06T07:37:22.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.635+0000 7fe72ea00640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe71406b4d0 0x7fe71406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:22.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.635+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe72808a1a0 con 0x7fe7380cfd90 2024-03-06T07:37:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.635+0000 7fe72d400640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe71406b4d0 0x7fe71406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.636+0000 7fe72d400640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe71406b4d0 0x7fe71406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe7380b45c0 tx=0x7fe720009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:22.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.640+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe728059020 con 0x7fe7380cfd90 2024-03-06T07:37:22.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.873+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7380d1df0 con 0x7fe7380cfd90 2024-03-06T07:37:22.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.874+0000 7fe72ea00640 1 -- 172.21.15.32:0/1507740424 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fe728058920 con 0x7fe7380cfd90 2024-03-06T07:37:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:22.877 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:20.396357+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313,6574]},"id":9}]} 2024-03-06T07:37:22.877 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 97 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe71406b4d0 msgr2=0x7fe71406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe71406b4d0 0x7fe71406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe7380b45c0 tx=0x7fe720009040 comp rx=0 tx=0).stop 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 msgr2=0x7fe7380cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 0x7fe7380cb7e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c870 tx=0x7fe72800cd30 comp rx=0 tx=0).stop 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 shutdown_connections 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe71406b4d0 0x7fe71406d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe738161b40 0x7fe73814da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe7380d2930 0x7fe738150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 --2- 172.21.15.32:0/1507740424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7380cfd90 0x7fe7380cb7e0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 >> 172.21.15.32:0/1507740424 conn(0x7fe73809d220 msgr2=0x7fe73806d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 shutdown_connections 2024-03-06T07:37:22.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:22.878+0000 7fe73d200640 1 -- 172.21.15.32:0/1507740424 wait complete. 2024-03-06T07:37:22.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:37:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:37:23.135 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:37:23.135 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:37:23.135 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:37:23.135 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> import fcntl 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> import time 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:37:23.191 DEBUG:teuthology.orchestra.run.smithi032:> 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> while True: 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> if os.read(0, 4096) == b"": 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> break 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> except BlockingIOError: 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> pass 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> time.sleep(2) 2024-03-06T07:37:23.192 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-03-06T07:37:23.199 DEBUG:teuthology.orchestra.run.smithi032:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:37:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:37:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:37:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:24.245 DEBUG:teuthology.orchestra.run.smithi032:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:37:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:37:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-06T07:37:24.303 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 29h/41d Inode: 2199023255552 Links: 2 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run.smithi032.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-06 07:37:23.312717304 +0000 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-06 07:37:23.312717304 +0000 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-06 07:37:23.312717304 +0000 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-06 07:37:23.312717304 +0000 2024-03-06T07:37:24.304 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:37:24.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:37:24.304 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:37:24.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:37:24.607 INFO:teuthology.orchestra.run.smithi039.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:37:24.607 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:37:24.607 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:37:24.842 INFO:teuthology.orchestra.run.smithi039.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:37:24.842 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:37:24.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:37:25.072 INFO:teuthology.orchestra.run.smithi039.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:37:25.073 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:37:25.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:37:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:25.334 INFO:teuthology.orchestra.run.smithi039.stdout:{"message":"","return_code":0} 2024-03-06T07:37:25.334 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:37:25.335 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json cache drop 2024-03-06T07:37:25.567 INFO:teuthology.orchestra.run.smithi039.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":378,"bytes":65362}},"duration":0.013917097} 2024-03-06T07:37:25.568 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65362, "items": 378 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.013917097, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:37:25.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:37:25.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.789+0000 7efc80a00640 1 Processor -- start 2024-03-06T07:37:25.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.789+0000 7efc80a00640 1 -- start start 2024-03-06T07:37:25.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.791+0000 7efc80a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efc7c145860 0x7efc7c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.791+0000 7efc80a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 0x7efc7c0aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.791+0000 7efc80a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c071ff0 con 0x7efc7c06b4f0 2024-03-06T07:37:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.791+0000 7efc80a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c06b950 con 0x7efc7c145860 2024-03-06T07:37:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.791+0000 7efc80a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c071da0 con 0x7efc7c0d02c0 2024-03-06T07:37:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.792+0000 7efc79e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efc7c06b4f0 0x7efc7c0cfbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43898/0 (socket says 172.21.15.32:43898) 2024-03-06T07:37:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.792+0000 7efc79e00640 1 -- 172.21.15.32:0/2946557397 learned_addr learned my addr 172.21.15.32:0/2946557397 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.792+0000 7efc7b200640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 0x7efc7c0aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.792+0000 7efc7a800640 1 --2- 172.21.15.32:0/2946557397 >> v2:172.21.15.39:3300/0 conn(0x7efc7c145860 0x7efc7c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.792+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653658737 0 0) 0x7efc7c071ff0 con 0x7efc7c06b4f0 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.792+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60003520 con 0x7efc7c06b4f0 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294707922 0 0) 0x7efc60003520 con 0x7efc7c06b4f0 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 msgr2=0x7efc7c0aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 0x7efc7c0aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 >> v2:172.21.15.39:3300/0 conn(0x7efc7c145860 msgr2=0x7efc7c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:25.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 --2- 172.21.15.32:0/2946557397 >> v2:172.21.15.39:3300/0 conn(0x7efc7c145860 0x7efc7c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0722d0 con 0x7efc7c06b4f0 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7efc68001670 con 0x7efc7c06b4f0 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 >> v1:172.21.15.32:6789/0 conn(0x7efc7c06b4f0 legacy=0x7efc7c0cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 0x7efc600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60003f30 0x7efc60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc60004d40 0x7efc600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c071da0 con 0x7efc60003f30 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6000fdd0 con 0x7efc60004d40 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6000ff50 con 0x7efc7c0d02c0 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc79e00640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60003f30 0x7efc60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc7b200640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc60004d40 0x7efc600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.793+0000 7efc7a800640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 0x7efc600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.794+0000 7efc79e00640 1 -- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 msgr2=0x7efc600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.794+0000 7efc79e00640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc7c0d02c0 0x7efc600039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.794+0000 7efc79e00640 1 -- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc60004d40 msgr2=0x7efc600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:25.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.794+0000 7efc79e00640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc60004d40 0x7efc600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.794+0000 7efc79e00640 1 -- 172.21.15.32:0/2946557397 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60003520 con 0x7efc60003f30 2024-03-06T07:37:25.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.794+0000 7efc7b200640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc60004d40 0x7efc600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.795+0000 7efc79e00640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60003f30 0x7efc60004800 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c290 tx=0x7efc60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655854097e8ad772 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.795+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68007d50 con 0x7efc60003f30 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.795+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68030070 con 0x7efc60003f30 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.795+0000 7efc79400640 1 -- 172.21.15.32:0/2946557397 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6802cda0 con 0x7efc60003f30 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.795+0000 7efc80a00640 1 -- 172.21.15.32:0/2946557397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60003f30 msgr2=0x7efc60004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.795+0000 7efc80a00640 1 --2- 172.21.15.32:0/2946557397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc60003f30 0x7efc60004800 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c290 tx=0x7efc60005830 comp rx=0 tx=0).stop 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.796+0000 7efc7b200640 1 -- 172.21.15.32:0/2946557397 reap_dead start 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.796+0000 7efc80a00640 1 -- 172.21.15.32:0/2946557397 shutdown_connections 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.796+0000 7efc80a00640 1 -- 172.21.15.32:0/2946557397 >> 172.21.15.32:0/2946557397 conn(0x7efc7c0d0c00 msgr2=0x7efc7c0d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:25.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.796+0000 7efc80a00640 1 -- 172.21.15.32:0/2946557397 shutdown_connections 2024-03-06T07:37:25.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.796+0000 7efc80a00640 1 -- 172.21.15.32:0/2946557397 wait complete. 2024-03-06T07:37:25.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.799+0000 7efc80a00640 1 Processor -- start 2024-03-06T07:37:25.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.799+0000 7efc80a00640 1 -- start start 2024-03-06T07:37:25.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.800+0000 7efc80a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc60003f30 0x7efc7c15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.800+0000 7efc80a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 0x7efc7c1efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc80a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7c15cea0 0x7efc7c15e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc80a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0af010 con 0x7efc7c0b2130 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc80a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0aee90 con 0x7efc7c15cea0 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc80a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0af190 con 0x7efc60003f30 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 0x7efc7c1efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 0x7efc7c1efb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50168/0 (socket says 172.21.15.32:50168) 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 -- 172.21.15.32:0/3753519941 learned_addr learned my addr 172.21.15.32:0/3753519941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc7a800640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc60003f30 0x7efc7c15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc7b200640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7c15cea0 0x7efc7c15e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 -- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc60003f30 msgr2=0x7efc7c15b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc60003f30 0x7efc7c15b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 -- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7c15cea0 msgr2=0x7efc7c15e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7c15cea0 0x7efc7c15e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc79e00640 1 -- 172.21.15.32:0/3753519941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c15bc30 con 0x7efc7c0b2130 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.801+0000 7efc7b200640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7c15cea0 0x7efc7c15e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:25.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.802+0000 7efc79e00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 0x7efc7c1efb90 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c3b0 tx=0x7efc7c06d860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:25.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.802+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68046070 con 0x7efc7c0b2130 2024-03-06T07:37:25.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.802+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68004c10 con 0x7efc7c0b2130 2024-03-06T07:37:25.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.802+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68007d50 con 0x7efc7c0b2130 2024-03-06T07:37:25.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.802+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c0b25c0 con 0x7efc7c0b2130 2024-03-06T07:37:25.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.803+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c15ca10 con 0x7efc7c0b2130 2024-03-06T07:37:25.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.804+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c04f330 con 0x7efc7c0b2130 2024-03-06T07:37:25.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.805+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc68007070 con 0x7efc7c0b2130 2024-03-06T07:37:25.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.807+0000 7efc72a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efc5c06b4d0 0x7efc5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:25.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.807+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc680ab670 con 0x7efc7c0b2130 2024-03-06T07:37:25.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.807+0000 7efc7a800640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efc5c06b4d0 0x7efc5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:25.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.808+0000 7efc7a800640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efc5c06b4d0 0x7efc5c06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009a90 tx=0x7efc6c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:25.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:25.812+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc6807a440 con 0x7efc7c0b2130 2024-03-06T07:37:26.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:26.191+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7efc7c0d1a30 con 0x7efc7c0b2130 2024-03-06T07:37:27.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.031+0000 7efc72a00640 1 -- 172.21.15.32:0/3753519941 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7efc68079d40 con 0x7efc7c0b2130 2024-03-06T07:37:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efc5c06b4d0 msgr2=0x7efc5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efc5c06b4d0 0x7efc5c06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009a90 tx=0x7efc6c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 msgr2=0x7efc7c1efb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 0x7efc7c1efb90 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c3b0 tx=0x7efc7c06d860 comp rx=0 tx=0).stop 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 shutdown_connections 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efc5c06b4d0 0x7efc5c06d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7c15cea0 0x7efc7c15e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7c0b2130 0x7efc7c1efb90 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 --2- 172.21.15.32:0/3753519941 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc60003f30 0x7efc7c15b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 >> 172.21.15.32:0/3753519941 conn(0x7efc7c0d0c00 msgr2=0x7efc7c0ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 shutdown_connections 2024-03-06T07:37:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.034+0000 7efc80a00640 1 -- 172.21.15.32:0/3753519941 wait complete. 2024-03-06T07:37:27.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.275+0000 7f7010a00640 1 Processor -- start 2024-03-06T07:37:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.276+0000 7f7010a00640 1 -- start start 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.277+0000 7f7010a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 0x7f700c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.277+0000 7f7010a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f700c0cdc20 0x7f700c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.277+0000 7f7010a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700c15cc40 con 0x7f700c0af580 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.277+0000 7f7010a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700c15cf20 con 0x7f700c0cdc20 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.277+0000 7f7010a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15d200 con 0x7f700c0af1a0 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.278+0000 7f7003e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f700c0af580 0x7f700c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43908/0 (socket says 172.21.15.32:43908) 2024-03-06T07:37:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.278+0000 7f7003e00640 1 -- 172.21.15.32:0/3126086902 learned_addr learned my addr 172.21.15.32:0/3126086902 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.278+0000 7f700a800640 1 --2- 172.21.15.32:0/3126086902 >> v2:172.21.15.39:3300/0 conn(0x7f700c0cdc20 0x7f700c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.278+0000 7f7009e00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 0x7f700c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.278+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377326146 0 0) 0x7f700c15cc40 con 0x7f700c0af580 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.278+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff0003520 con 0x7f700c0af580 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332076770 0 0) 0x7f6ff0003520 con 0x7f700c0af580 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 msgr2=0x7f700c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 0x7f700c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 >> v2:172.21.15.39:3300/0 conn(0x7f700c0cdc20 msgr2=0x7f700c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 --2- 172.21.15.32:0/3126086902 >> v2:172.21.15.39:3300/0 conn(0x7f700c0cdc20 0x7f700c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c15d4e0 con 0x7f700c0af580 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6ff4001670 con 0x7f700c0af580 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 >> v1:172.21.15.32:6789/0 conn(0x7f700c0af580 legacy=0x7f700c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 0x7f6ff0003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff0003fa0 0x7f6ff0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0004d40 0x7f6ff00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15cf20 con 0x7f6ff0003fa0 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15d200 con 0x7f6ff0004d40 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000ba70 con 0x7f700c0af1a0 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7003e00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff0003fa0 0x7f6ff0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.279+0000 7f7009e00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 0x7f6ff0003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.280+0000 7f700a800640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0004d40 0x7f6ff00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.280+0000 7f7003e00640 1 -- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 msgr2=0x7f6ff0003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.280+0000 7f7003e00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f700c0af1a0 0x7f6ff0003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.280+0000 7f7003e00640 1 -- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0004d40 msgr2=0x7f6ff00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.280+0000 7f7003e00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0004d40 0x7f6ff00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.280+0000 7f7003e00640 1 -- 172.21.15.32:0/3126086902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0003520 con 0x7f6ff0003fa0 2024-03-06T07:37:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.281+0000 7f7003e00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff0003fa0 0x7f6ff0004800 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c210 tx=0x7f6ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84caf0e63c1bf84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4007d50 con 0x7f6ff0003fa0 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4030070 con 0x7f6ff0003fa0 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7009400640 1 -- 172.21.15.32:0/3126086902 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff402cdc0 con 0x7f6ff0003fa0 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7010a00640 1 -- 172.21.15.32:0/3126086902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff0003fa0 msgr2=0x7f6ff0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7010a00640 1 --2- 172.21.15.32:0/3126086902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff0003fa0 0x7f6ff0004800 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c210 tx=0x7f6ff0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f700a800640 1 -- 172.21.15.32:0/3126086902 reap_dead start 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7010a00640 1 -- 172.21.15.32:0/3126086902 shutdown_connections 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.282+0000 7f7010a00640 1 -- 172.21.15.32:0/3126086902 >> 172.21.15.32:0/3126086902 conn(0x7f700c06afb0 msgr2=0x7f700c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.283+0000 7f7010a00640 1 -- 172.21.15.32:0/3126086902 shutdown_connections 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.283+0000 7f7010a00640 1 -- 172.21.15.32:0/3126086902 wait complete. 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.284+0000 7f7010a00640 1 Processor -- start 2024-03-06T07:37:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.284+0000 7f7010a00640 1 -- start start 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.285+0000 7f7010a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ff0003fa0 0x7f700c159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.285+0000 7f7010a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700c155260 0x7f700c154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.285+0000 7f7010a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 0x7f700c0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.285+0000 7f7010a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15c990 con 0x7f700c0acdb0 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.285+0000 7f7010a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15c810 con 0x7f700c155260 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.285+0000 7f7010a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c0b3310 con 0x7f6ff0003fa0 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 0x7f700c0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 0x7f700c0ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50180/0 (socket says 172.21.15.32:50180) 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 -- 172.21.15.32:0/3807406853 learned_addr learned my addr 172.21.15.32:0/3807406853 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f7009e00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ff0003fa0 0x7f700c159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 -- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ff0003fa0 msgr2=0x7f700c159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ff0003fa0 0x7f700c159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 -- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700c155260 msgr2=0x7f700c154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f7003e00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700c155260 0x7f700c154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700c155260 0x7f700c154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f700a800640 1 -- 172.21.15.32:0/3807406853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c155ad0 con 0x7f700c0acdb0 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.286+0000 7f7003e00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700c155260 0x7f700c154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.287+0000 7f700a800640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 0x7f700c0ad230 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f700c0728b0 tx=0x7f70040115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.287+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004011e40 con 0x7f700c0acdb0 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.287+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700401b070 con 0x7f700c0acdb0 2024-03-06T07:37:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.287+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004003990 con 0x7f700c0acdb0 2024-03-06T07:37:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.287+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c0afa20 con 0x7f700c0acdb0 2024-03-06T07:37:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.287+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700c0affe0 con 0x7f700c0acdb0 2024-03-06T07:37:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.288+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700c069510 con 0x7f700c0acdb0 2024-03-06T07:37:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.289+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7004005d10 con 0x7f700c0acdb0 2024-03-06T07:37:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.290+0000 7f7002000640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6fe806b470 0x7f6fe806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.290+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f700408a7c0 con 0x7f700c0acdb0 2024-03-06T07:37:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.290+0000 7f7009e00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6fe806b470 0x7f6fe806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.291+0000 7f7009e00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6fe806b470 0x7f6fe806d930 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0098b0 tx=0x7f6ffc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.294+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7004014350 con 0x7f700c0acdb0 2024-03-06T07:37:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:27.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.532+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f700c0b4df0 con 0x7f700c0acdb0 2024-03-06T07:37:27.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.534+0000 7f7002000640 1 -- 172.21.15.32:0/3807406853 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f70040595a0 con 0x7f700c0acdb0 2024-03-06T07:37:27.536 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:27.536 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:27.537 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.536+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6fe806b470 msgr2=0x7f6fe806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.536+0000 7f7010a00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6fe806b470 0x7f6fe806d930 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0098b0 tx=0x7f6ffc008040 comp rx=0 tx=0).stop 2024-03-06T07:37:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.536+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 msgr2=0x7f700c0ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.536+0000 7f7010a00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 0x7f700c0ad230 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f700c0728b0 tx=0x7f70040115a0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 shutdown_connections 2024-03-06T07:37:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6fe806b470 0x7f6fe806d930 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f700c0acdb0 0x7f700c0ad230 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700c155260 0x7f700c154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 --2- 172.21.15.32:0/3807406853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ff0003fa0 0x7f700c159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:27.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 >> 172.21.15.32:0/3807406853 conn(0x7f700c06afb0 msgr2=0x7f700c0cef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:27.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 shutdown_connections 2024-03-06T07:37:27.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:27.537+0000 7f7010a00640 1 -- 172.21.15.32:0/3807406853 wait complete. 2024-03-06T07:37:27.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:27.759 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:28.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:28.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.986+0000 7f70d7a00640 1 Processor -- start 2024-03-06T07:37:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.986+0000 7f70d7a00640 1 -- start start 2024-03-06T07:37:28.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.988+0000 7f70d7a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d00ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.988+0000 7f70d7a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70d00d12e0 0x7f70d006e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.988+0000 7f70d7a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d015b110 con 0x7f70d00b52a0 2024-03-06T07:37:28.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.988+0000 7f70d7a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d00aeb30 con 0x7f70d00d12e0 2024-03-06T07:37:28.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.988+0000 7f70d7a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00aed60 con 0x7f70d015ad30 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70d00d12e0 0x7f70d006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70d00d12e0 0x7f70d006e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43362/0 (socket says 172.21.15.32:43362) 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 -- 172.21.15.32:0/1664370772 learned_addr learned my addr 172.21.15.32:0/1664370772 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5000640 1 --2- 172.21.15.32:0/1664370772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d00ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 -- 172.21.15.32:0/1664370772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 msgr2=0x7f70d00ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 --2- 172.21.15.32:0/1664370772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d00ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 -- 172.21.15.32:0/1664370772 >> v1:172.21.15.39:6789/0 conn(0x7f70d00b52a0 legacy=0x7f70d00d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.989+0000 7f70d5a00640 1 -- 172.21.15.32:0/1664370772 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d00af040 con 0x7f70d00d12e0 2024-03-06T07:37:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.991+0000 7f70d5a00640 1 --2- 172.21.15.32:0/1664370772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d006e160 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009e50 tx=0x7f70c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c499ff43a4aa9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.991+0000 7f70cf400640 1 -- 172.21.15.32:0/1664370772 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c802cb00 con 0x7f70d00d12e0 2024-03-06T07:37:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.991+0000 7f70cf400640 1 -- 172.21.15.32:0/1664370772 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c802cca0 con 0x7f70d00d12e0 2024-03-06T07:37:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.991+0000 7f70cf400640 1 -- 172.21.15.32:0/1664370772 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8034610 con 0x7f70d00d12e0 2024-03-06T07:37:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 -- 172.21.15.32:0/1664370772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 msgr2=0x7f70d006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 --2- 172.21.15.32:0/1664370772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d006e160 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009e50 tx=0x7f70c802bf60 comp rx=0 tx=0).stop 2024-03-06T07:37:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 -- 172.21.15.32:0/1664370772 shutdown_connections 2024-03-06T07:37:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 --2- 172.21.15.32:0/1664370772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d006e160 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 --2- 172.21.15.32:0/1664370772 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d00ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 -- 172.21.15.32:0/1664370772 >> 172.21.15.32:0/1664370772 conn(0x7f70d006c2a0 msgr2=0x7f70d006c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 -- 172.21.15.32:0/1664370772 shutdown_connections 2024-03-06T07:37:28.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.992+0000 7f70d7a00640 1 -- 172.21.15.32:0/1664370772 wait complete. 2024-03-06T07:37:28.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.994+0000 7f70d7a00640 1 Processor -- start 2024-03-06T07:37:28.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.995+0000 7f70d7a00640 1 -- start start 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70d7a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00b52a0 0x7f70d0143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70d7a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d00b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70d7a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d0154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70d7a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00af8e0 con 0x7f70d00d12e0 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70d7a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00af600 con 0x7f70d00b52a0 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70d7a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00b5700 con 0x7f70d015ad30 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d00b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d00b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43374/0 (socket says 172.21.15.32:43374) 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.996+0000 7f70cfe00640 1 -- 172.21.15.32:0/4022243242 learned_addr learned my addr 172.21.15.32:0/4022243242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70d5000640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00b52a0 0x7f70d0143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70d5a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d0154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70cfe00640 1 -- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 msgr2=0x7f70d0154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70cfe00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d0154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70cfe00640 1 -- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00b52a0 msgr2=0x7f70d0143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70cfe00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00b52a0 0x7f70d0143920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70cfe00640 1 -- 172.21.15.32:0/4022243242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d0143e60 con 0x7f70d00d12e0 2024-03-06T07:37:29.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70cfe00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d00b25b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00c450 tx=0x7f70bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0052d0 con 0x7f70d00d12e0 2024-03-06T07:37:29.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc005470 con 0x7f70d00d12e0 2024-03-06T07:37:29.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc012870 con 0x7f70d00d12e0 2024-03-06T07:37:29.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.997+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d00b3050 con 0x7f70d00d12e0 2024-03-06T07:37:29.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.998+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d015b790 con 0x7f70d00d12e0 2024-03-06T07:37:29.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:28.999+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d004f3a0 con 0x7f70d00d12e0 2024-03-06T07:37:29.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.001+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70bc012a10 con 0x7f70d00d12e0 2024-03-06T07:37:29.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.002+0000 7f70ccc00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f70b006b4d0 0x7f70b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.002+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70bc089db0 con 0x7f70d00d12e0 2024-03-06T07:37:29.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.003+0000 7f70d5000640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f70b006b4d0 0x7f70b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.003+0000 7f70d5000640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f70b006b4d0 0x7f70b006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f70c4006fd0 tx=0x7f70c4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.007+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70bc058cc0 con 0x7f70d00d12e0 2024-03-06T07:37:29.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.281+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70d006db20 con 0x7f70d00d12e0 2024-03-06T07:37:29.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.282+0000 7f70ccc00640 1 -- 172.21.15.32:0/4022243242 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f70bc0585c0 con 0x7f70d00d12e0 2024-03-06T07:37:29.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:29.284 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:29.284 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f70b006b4d0 msgr2=0x7f70b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f70b006b4d0 0x7f70b006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f70c4006fd0 tx=0x7f70c4008040 comp rx=0 tx=0).stop 2024-03-06T07:37:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 msgr2=0x7f70d00b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d00b25b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00c450 tx=0x7f70bc00c910 comp rx=0 tx=0).stop 2024-03-06T07:37:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 shutdown_connections 2024-03-06T07:37:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f70b006b4d0 0x7f70b006d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f70d015ad30 0x7f70d0154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70d00d12e0 0x7f70d00b25b0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.284+0000 7f70d7a00640 1 --2- 172.21.15.32:0/4022243242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00b52a0 0x7f70d0143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.285+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 >> 172.21.15.32:0/4022243242 conn(0x7f70d006c2a0 msgr2=0x7f70d00b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.285+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 shutdown_connections 2024-03-06T07:37:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:29.285+0000 7f70d7a00640 1 -- 172.21.15.32:0/4022243242 wait complete. 2024-03-06T07:37:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:29.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:29.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:30.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:30.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.731+0000 7f574c000640 1 Processor -- start 2024-03-06T07:37:30.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.732+0000 7f574c000640 1 -- start start 2024-03-06T07:37:30.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.733+0000 7f574c000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57440af1a0 0x7f574415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.733+0000 7f574c000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af580 0x7f57440b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.733+0000 7f574c000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f57440cdc20 0x7f57440b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.733+0000 7f574c000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574415cc40 con 0x7f57440af1a0 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.733+0000 7f574c000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574415cf20 con 0x7f57440cdc20 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.733+0000 7f574c000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574415d200 con 0x7f57440af580 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57440af1a0 0x7f574415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57440af1a0 0x7f574415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43386/0 (socket says 172.21.15.32:43386) 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 -- 172.21.15.32:0/2691152867 learned_addr learned my addr 172.21.15.32:0/2691152867 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f574a000640 1 --2- 172.21.15.32:0/2691152867 >> v2:172.21.15.39:3300/0 conn(0x7f57440cdc20 0x7f57440b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5748c00640 1 --2- 172.21.15.32:0/2691152867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af580 0x7f57440b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 -- 172.21.15.32:0/2691152867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af580 msgr2=0x7f57440b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 --2- 172.21.15.32:0/2691152867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af580 0x7f57440b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 -- 172.21.15.32:0/2691152867 >> v2:172.21.15.39:3300/0 conn(0x7f57440cdc20 msgr2=0x7f57440b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 --2- 172.21.15.32:0/2691152867 >> v2:172.21.15.39:3300/0 conn(0x7f57440cdc20 0x7f57440b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.734+0000 7f5749600640 1 -- 172.21.15.32:0/2691152867 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574415d4e0 con 0x7f57440af1a0 2024-03-06T07:37:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.736+0000 7f5749600640 1 --2- 172.21.15.32:0/2691152867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af1a0 0x7f574415c600 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f57440cda30 tx=0x7f573802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117faf07e57a03ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.736+0000 7f5743e00640 1 -- 172.21.15.32:0/2691152867 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573802e070 con 0x7f57440af1a0 2024-03-06T07:37:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.736+0000 7f5743e00640 1 -- 172.21.15.32:0/2691152867 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738037070 con 0x7f57440af1a0 2024-03-06T07:37:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.736+0000 7f5743e00640 1 -- 172.21.15.32:0/2691152867 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738031a00 con 0x7f57440af1a0 2024-03-06T07:37:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.736+0000 7f574c000640 1 -- 172.21.15.32:0/2691152867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af1a0 msgr2=0x7f574415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.736+0000 7f574c000640 1 --2- 172.21.15.32:0/2691152867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af1a0 0x7f574415c600 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f57440cda30 tx=0x7f573802b880 comp rx=0 tx=0).stop 2024-03-06T07:37:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 -- 172.21.15.32:0/2691152867 shutdown_connections 2024-03-06T07:37:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 --2- 172.21.15.32:0/2691152867 >> v2:172.21.15.39:3300/0 conn(0x7f57440cdc20 0x7f57440b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 --2- 172.21.15.32:0/2691152867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af580 0x7f57440b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 --2- 172.21.15.32:0/2691152867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af1a0 0x7f574415c600 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 -- 172.21.15.32:0/2691152867 >> 172.21.15.32:0/2691152867 conn(0x7f574406afb0 msgr2=0x7f574406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 -- 172.21.15.32:0/2691152867 shutdown_connections 2024-03-06T07:37:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.737+0000 7f574c000640 1 -- 172.21.15.32:0/2691152867 wait complete. 2024-03-06T07:37:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.738+0000 7f574c000640 1 Processor -- start 2024-03-06T07:37:30.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 -- start start 2024-03-06T07:37:30.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af1a0 0x7f57441676f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 0x7f57441538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57440cdc20 0x7f574414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440b3490 con 0x7f57440af580 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440b3310 con 0x7f57440cdc20 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.739+0000 7f574c000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57440b3610 con 0x7f57440af1a0 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 0x7f57441538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 0x7f57441538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43398/0 (socket says 172.21.15.32:43398) 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 -- 172.21.15.32:0/1839073110 learned_addr learned my addr 172.21.15.32:0/1839073110 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5749600640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af1a0 0x7f57441676f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f574a000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57440cdc20 0x7f574414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 -- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af1a0 msgr2=0x7f57441676f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af1a0 0x7f57441676f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 -- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57440cdc20 msgr2=0x7f574414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57440cdc20 0x7f574414c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f5748c00640 1 -- 172.21.15.32:0/1839073110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744153580 con 0x7f57440af580 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.740+0000 7f574a000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57440cdc20 0x7f574414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.741+0000 7f5748c00640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 0x7f57441538c0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f573000d800 tx=0x7f573000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.741+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730018020 con 0x7f57440af580 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.741+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57300096a0 con 0x7f57440af580 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.741+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5744153e00 con 0x7f57440af580 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.741+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730009a70 con 0x7f57440af580 2024-03-06T07:37:30.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.741+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57441532a0 con 0x7f57440af580 2024-03-06T07:37:30.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.742+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574404f3a0 con 0x7f57440af580 2024-03-06T07:37:30.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.743+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573001b020 con 0x7f57440af580 2024-03-06T07:37:30.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.744+0000 7f5741600640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572406b500 0x7f572406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.744+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f573008a340 con 0x7f57440af580 2024-03-06T07:37:30.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.745+0000 7f5749600640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572406b500 0x7f572406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.745+0000 7f5749600640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572406b500 0x7f572406d9c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5738036000 tx=0x7f573803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.749+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5730022030 con 0x7f57440af580 2024-03-06T07:37:30.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.992+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57440adbd0 con 0x7f57440af580 2024-03-06T07:37:30.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.993+0000 7f5741600640 1 -- 172.21.15.32:0/1839073110 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f5730059250 con 0x7f57440af580 2024-03-06T07:37:30.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:30.995 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:30.996 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572406b500 msgr2=0x7f572406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572406b500 0x7f572406d9c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5738036000 tx=0x7f573803d040 comp rx=0 tx=0).stop 2024-03-06T07:37:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 msgr2=0x7f57441538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 0x7f57441538c0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f573000d800 tx=0x7f573000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 shutdown_connections 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f572406b500 0x7f572406d9c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57440cdc20 0x7f574414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57440af580 0x7f57441538c0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.995+0000 7f574c000640 1 --2- 172.21.15.32:0/1839073110 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f57440af1a0 0x7f57441676f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.996+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 >> 172.21.15.32:0/1839073110 conn(0x7f574406afb0 msgr2=0x7f57440ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.996+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 shutdown_connections 2024-03-06T07:37:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:30.996+0000 7f574c000640 1 -- 172.21.15.32:0/1839073110 wait complete. 2024-03-06T07:37:31.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:31.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:32.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:32.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.450+0000 7fd9de400640 1 Processor -- start 2024-03-06T07:37:32.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.451+0000 7fd9de400640 1 -- start start 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9de400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9d81552b0 0x7fd9d815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9de400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd9d815e810 0x7fd9d80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9de400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9de400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d815ed10 con 0x7fd9d81552b0 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9de400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d815eff0 con 0x7fd9d815e810 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9de400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d815f2d0 con 0x7fd9d80cce90 2024-03-06T07:37:32.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9dd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9d81552b0 0x7fd9d815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9dd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9d81552b0 0x7fd9d815b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43408/0 (socket says 172.21.15.32:43408) 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.454+0000 7fd9dd000640 1 -- 172.21.15.32:0/3143291651 learned_addr learned my addr 172.21.15.32:0/3143291651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9d7e00640 1 --2- 172.21.15.32:0/3143291651 >> v2:172.21.15.39:3300/0 conn(0x7fd9d815e810 0x7fd9d80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9dda00640 1 --2- 172.21.15.32:0/3143291651 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9dd000640 1 -- 172.21.15.32:0/3143291651 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 msgr2=0x7fd9d806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9dd000640 1 --2- 172.21.15.32:0/3143291651 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9dd000640 1 -- 172.21.15.32:0/3143291651 >> v2:172.21.15.39:3300/0 conn(0x7fd9d815e810 msgr2=0x7fd9d80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9dd000640 1 --2- 172.21.15.32:0/3143291651 >> v2:172.21.15.39:3300/0 conn(0x7fd9d815e810 0x7fd9d80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.455+0000 7fd9dd000640 1 -- 172.21.15.32:0/3143291651 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d815f5b0 con 0x7fd9d81552b0 2024-03-06T07:37:32.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.456+0000 7fd9dd000640 1 --2- 172.21.15.32:0/3143291651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d815b1d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80cf1d0 tx=0x7fd9cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bda5792082193f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:32.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.457+0000 7fd9d7400640 1 -- 172.21.15.32:0/3143291651 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc02e070 con 0x7fd9d81552b0 2024-03-06T07:37:32.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.457+0000 7fd9d7400640 1 -- 172.21.15.32:0/3143291651 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc037070 con 0x7fd9d81552b0 2024-03-06T07:37:32.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.457+0000 7fd9d7400640 1 -- 172.21.15.32:0/3143291651 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc031a00 con 0x7fd9d81552b0 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.457+0000 7fd9de400640 1 -- 172.21.15.32:0/3143291651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 msgr2=0x7fd9d815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.457+0000 7fd9de400640 1 --2- 172.21.15.32:0/3143291651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d815b1d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80cf1d0 tx=0x7fd9cc02b880 comp rx=0 tx=0).stop 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 -- 172.21.15.32:0/3143291651 shutdown_connections 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 --2- 172.21.15.32:0/3143291651 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 --2- 172.21.15.32:0/3143291651 >> v2:172.21.15.39:3300/0 conn(0x7fd9d815e810 0x7fd9d80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 --2- 172.21.15.32:0/3143291651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d815b1d0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 -- 172.21.15.32:0/3143291651 >> 172.21.15.32:0/3143291651 conn(0x7fd9d806b3d0 msgr2=0x7fd9d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:32.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 -- 172.21.15.32:0/3143291651 shutdown_connections 2024-03-06T07:37:32.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.458+0000 7fd9de400640 1 -- 172.21.15.32:0/3143291651 wait complete. 2024-03-06T07:37:32.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.460+0000 7fd9de400640 1 Processor -- start 2024-03-06T07:37:32.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.460+0000 7fd9de400640 1 -- start start 2024-03-06T07:37:32.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9de400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9de400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d81443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9de400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9d815e810 0x7fd9d8150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9de400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d806dd30 con 0x7fd9d81552b0 2024-03-06T07:37:32.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9de400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80b3760 con 0x7fd9d815e810 2024-03-06T07:37:32.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9de400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d806deb0 con 0x7fd9d80cce90 2024-03-06T07:37:32.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d81443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d81443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43418/0 (socket says 172.21.15.32:43418) 2024-03-06T07:37:32.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9d7e00640 1 -- 172.21.15.32:0/3919726825 learned_addr learned my addr 172.21.15.32:0/3919726825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:32.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.461+0000 7fd9dd000640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9dda00640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9d815e810 0x7fd9d8150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d7e00640 1 -- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 msgr2=0x7fd9d814edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d7e00640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d814edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d7e00640 1 -- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9d815e810 msgr2=0x7fd9d8150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d7e00640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9d815e810 0x7fd9d8150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d7e00640 1 -- 172.21.15.32:0/3919726825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d81515f0 con 0x7fd9d81552b0 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9dda00640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9d815e810 0x7fd9d8150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d7e00640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d81443f0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400d5f0 tx=0x7fd9c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40125f0 con 0x7fd9d81552b0 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c4019070 con 0x7fd9d81552b0 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40129d0 con 0x7fd9d81552b0 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d81518d0 con 0x7fd9d81552b0 2024-03-06T07:37:32.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.462+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d8151e90 con 0x7fd9d81552b0 2024-03-06T07:37:32.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.464+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d804f330 con 0x7fd9d81552b0 2024-03-06T07:37:32.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.465+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9c4012b70 con 0x7fd9d81552b0 2024-03-06T07:37:32.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.466+0000 7fd9d4c00640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd9b806b500 0x7fd9b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:32.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.466+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9c408a5d0 con 0x7fd9d81552b0 2024-03-06T07:37:32.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.466+0000 7fd9dd000640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd9b806b500 0x7fd9b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:32.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.467+0000 7fd9dd000640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd9b806b500 0x7fd9b806d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc036000 tx=0x7fd9cc03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:32.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.470+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9c4059440 con 0x7fd9d81552b0 2024-03-06T07:37:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.715+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d80cf4a0 con 0x7fd9d81552b0 2024-03-06T07:37:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.716+0000 7fd9d4c00640 1 -- 172.21.15.32:0/3919726825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7fd9c4058d40 con 0x7fd9d81552b0 2024-03-06T07:37:32.719 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:32.719 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd9b806b500 msgr2=0x7fd9b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd9b806b500 0x7fd9b806d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc036000 tx=0x7fd9cc03d040 comp rx=0 tx=0).stop 2024-03-06T07:37:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 msgr2=0x7fd9d81443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d81443f0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400d5f0 tx=0x7fd9c400dab0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 shutdown_connections 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd9b806b500 0x7fd9b806d9c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9d815e810 0x7fd9d8150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d81552b0 0x7fd9d81443f0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 --2- 172.21.15.32:0/3919726825 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9d80cce90 0x7fd9d814edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 >> 172.21.15.32:0/3919726825 conn(0x7fd9d806b3d0 msgr2=0x7fd9d815c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 shutdown_connections 2024-03-06T07:37:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:32.720+0000 7fd9de400640 1 -- 172.21.15.32:0/3919726825 wait complete. 2024-03-06T07:37:32.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:32.950 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:33.331 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:33.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:34.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.170+0000 7f1a67a00640 1 Processor -- start 2024-03-06T07:37:34.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.171+0000 7f1a67a00640 1 -- start start 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.172+0000 7f1a67a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a6015e360 0x7f1a600ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.172+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600d0be0 0x7f1a600cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.172+0000 7f1a67a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6006d350 con 0x7f1a600b1060 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.172+0000 7f1a67a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6006d630 con 0x7f1a6015e360 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.172+0000 7f1a67a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6006d910 con 0x7f1a600d0be0 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1a600b1060 0x7f1a600cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55778/0 (socket says 172.21.15.32:55778) 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65a00640 1 -- 172.21.15.32:0/2966789483 learned_addr learned my addr 172.21.15.32:0/2966789483 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:34.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65000640 1 --2- 172.21.15.32:0/2966789483 >> v2:172.21.15.32:3300/0 conn(0x7f1a6015e360 0x7f1a600ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a5fe00640 1 --2- 172.21.15.32:0/2966789483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600d0be0 0x7f1a600cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65000640 1 -- 172.21.15.32:0/2966789483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600d0be0 msgr2=0x7f1a600cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65000640 1 --2- 172.21.15.32:0/2966789483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600d0be0 0x7f1a600cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65000640 1 -- 172.21.15.32:0/2966789483 >> v1:172.21.15.32:6789/0 conn(0x7f1a600b1060 legacy=0x7f1a600cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.173+0000 7f1a65000640 1 -- 172.21.15.32:0/2966789483 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6006f000 con 0x7f1a6015e360 2024-03-06T07:37:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.175+0000 7f1a65000640 1 --2- 172.21.15.32:0/2966789483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6015e360 0x7f1a600ce550 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f1a540097a0 tx=0x7f1a5402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9086b07cb37d9e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.175+0000 7f1a5f400640 1 -- 172.21.15.32:0/2966789483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54031780 con 0x7f1a6015e360 2024-03-06T07:37:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.175+0000 7f1a5f400640 1 -- 172.21.15.32:0/2966789483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54037070 con 0x7f1a6015e360 2024-03-06T07:37:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.175+0000 7f1a5f400640 1 -- 172.21.15.32:0/2966789483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54031b40 con 0x7f1a6015e360 2024-03-06T07:37:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 -- 172.21.15.32:0/2966789483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6015e360 msgr2=0x7f1a600ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 --2- 172.21.15.32:0/2966789483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6015e360 0x7f1a600ce550 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f1a540097a0 tx=0x7f1a5402b820 comp rx=0 tx=0).stop 2024-03-06T07:37:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 -- 172.21.15.32:0/2966789483 shutdown_connections 2024-03-06T07:37:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 --2- 172.21.15.32:0/2966789483 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600d0be0 0x7f1a600cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 --2- 172.21.15.32:0/2966789483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6015e360 0x7f1a600ce550 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 -- 172.21.15.32:0/2966789483 >> 172.21.15.32:0/2966789483 conn(0x7f1a600b38a0 msgr2=0x7f1a600b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 -- 172.21.15.32:0/2966789483 shutdown_connections 2024-03-06T07:37:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.176+0000 7f1a67a00640 1 -- 172.21.15.32:0/2966789483 wait complete. 2024-03-06T07:37:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.178+0000 7f1a67a00640 1 Processor -- start 2024-03-06T07:37:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.178+0000 7f1a67a00640 1 -- start start 2024-03-06T07:37:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600b1060 0x7f1a601f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 0x7f1a601f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a6015e360 0x7f1a600b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a67a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6015e8c0 con 0x7f1a600d0be0 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a67a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6015e740 con 0x7f1a6015e360 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a67a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a60070510 con 0x7f1a600b1060 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 0x7f1a601f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 0x7f1a601f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43436/0 (socket says 172.21.15.32:43436) 2024-03-06T07:37:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.179+0000 7f1a5fe00640 1 -- 172.21.15.32:0/3454391832 learned_addr learned my addr 172.21.15.32:0/3454391832 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a65a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a6015e360 0x7f1a600b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a65000640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600b1060 0x7f1a601f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5fe00640 1 -- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600b1060 msgr2=0x7f1a601f4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5fe00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600b1060 0x7f1a601f4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5fe00640 1 -- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a6015e360 msgr2=0x7f1a600b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5fe00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a6015e360 0x7f1a600b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5fe00640 1 -- 172.21.15.32:0/3454391832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6009dff0 con 0x7f1a600d0be0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5fe00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 0x7f1a601f5620 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d800 tx=0x7f1a4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.180+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00b6c0 con 0x7f1a600d0be0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.181+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c019070 con 0x7f1a600d0be0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.181+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00ba80 con 0x7f1a600d0be0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.181+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6009e2d0 con 0x7f1a600d0be0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.181+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6009e890 con 0x7f1a600d0be0 2024-03-06T07:37:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.182+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6004f3a0 con 0x7f1a600d0be0 2024-03-06T07:37:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.183+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a4c00bc20 con 0x7f1a600d0be0 2024-03-06T07:37:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.184+0000 7f1a5cc00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a4006b520 0x7f1a4006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.184+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a4c08a6d0 con 0x7f1a600d0be0 2024-03-06T07:37:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.184+0000 7f1a65000640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a4006b520 0x7f1a4006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.185+0000 7f1a65000640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a4006b520 0x7f1a4006d9e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1a54036000 tx=0x7f1a5403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:34.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.188+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a4c059530 con 0x7f1a600d0be0 2024-03-06T07:37:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.428+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a600707d0 con 0x7f1a600d0be0 2024-03-06T07:37:34.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.429+0000 7f1a5cc00640 1 -- 172.21.15.32:0/3454391832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f1a4c058e30 con 0x7f1a600d0be0 2024-03-06T07:37:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:34.432 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:34.432 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:34.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a4006b520 msgr2=0x7f1a4006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a4006b520 0x7f1a4006d9e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1a54036000 tx=0x7f1a5403d040 comp rx=0 tx=0).stop 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 msgr2=0x7f1a601f5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 0x7f1a601f5620 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d800 tx=0x7f1a4c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 shutdown_connections 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a4006b520 0x7f1a4006d9e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a6015e360 0x7f1a600b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a600d0be0 0x7f1a601f5620 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 --2- 172.21.15.32:0/3454391832 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a600b1060 0x7f1a601f4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 >> 172.21.15.32:0/3454391832 conn(0x7f1a600b38a0 msgr2=0x7f1a600d1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 shutdown_connections 2024-03-06T07:37:34.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:34.432+0000 7f1a67a00640 1 -- 172.21.15.32:0/3454391832 wait complete. 2024-03-06T07:37:34.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:34.655 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:35.333 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:35.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.880+0000 7f893aa00640 1 Processor -- start 2024-03-06T07:37:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.880+0000 7f893aa00640 1 -- start start 2024-03-06T07:37:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f893aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 0x7f893415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f893aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89340af580 0x7f89340b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f893aa00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893415cc40 con 0x7f89340cdc20 2024-03-06T07:37:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f893aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893415cf20 con 0x7f89340af580 2024-03-06T07:37:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f893aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893415d200 con 0x7f89340af1a0 2024-03-06T07:37:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f8938a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89340cdc20 0x7f89340b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55790/0 (socket says 172.21.15.32:55790) 2024-03-06T07:37:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.882+0000 7f8938a00640 1 -- 172.21.15.32:0/3527421093 learned_addr learned my addr 172.21.15.32:0/3527421093 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210938792 0 0) 0x7f893415cc40 con 0x7f89340cdc20 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f892b400640 1 --2- 172.21.15.32:0/3527421093 >> v2:172.21.15.39:3300/0 conn(0x7f89340af580 0x7f89340b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933e00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 0x7f893415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918003520 con 0x7f89340cdc20 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943353073 0 0) 0x7f8918003520 con 0x7f89340cdc20 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 msgr2=0x7f893415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 0x7f893415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 >> v2:172.21.15.39:3300/0 conn(0x7f89340af580 msgr2=0x7f89340b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 --2- 172.21.15.32:0/3527421093 >> v2:172.21.15.39:3300/0 conn(0x7f89340af580 0x7f89340b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893415d4e0 con 0x7f89340cdc20 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f892c001670 con 0x7f89340cdc20 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 >> v1:172.21.15.32:6789/0 conn(0x7f89340cdc20 legacy=0x7f89340b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 0x7f8918003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918003fa0 0x7f8918004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8918004d40 0x7f89180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893415cf20 con 0x7f8918004d40 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893415d200 con 0x7f8918003fa0 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.883+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891800bb20 con 0x7f89340af1a0 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8938a00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8918004d40 0x7f89180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8933e00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 0x7f8918003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f892b400640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918003fa0 0x7f8918004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8938a00640 1 -- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 msgr2=0x7f8918003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8938a00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f89340af1a0 0x7f8918003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8938a00640 1 -- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918003fa0 msgr2=0x7f8918004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8938a00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918003fa0 0x7f8918004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f8938a00640 1 -- 172.21.15.32:0/3527421093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918003520 con 0x7f8918004d40 2024-03-06T07:37:35.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.884+0000 7f892b400640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918003fa0 0x7f8918004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.885+0000 7f8938a00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8918004d40 0x7f89180051c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f892c00c210 tx=0x7f8918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6db89c1b318fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c007d50 con 0x7f8918004d40 2024-03-06T07:37:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c030070 con 0x7f8918004d40 2024-03-06T07:37:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f8933400640 1 -- 172.21.15.32:0/3527421093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c02cdc0 con 0x7f8918004d40 2024-03-06T07:37:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f893aa00640 1 -- 172.21.15.32:0/3527421093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8918004d40 msgr2=0x7f89180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f893aa00640 1 --2- 172.21.15.32:0/3527421093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8918004d40 0x7f89180051c0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f892c00c210 tx=0x7f8918005830 comp rx=0 tx=0).stop 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f8938a00640 1 -- 172.21.15.32:0/3527421093 reap_dead start 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f893aa00640 1 -- 172.21.15.32:0/3527421093 shutdown_connections 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f893aa00640 1 -- 172.21.15.32:0/3527421093 >> 172.21.15.32:0/3527421093 conn(0x7f893406afb0 msgr2=0x7f893406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f893aa00640 1 -- 172.21.15.32:0/3527421093 shutdown_connections 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.886+0000 7f893aa00640 1 -- 172.21.15.32:0/3527421093 wait complete. 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.888+0000 7f893aa00640 1 Processor -- start 2024-03-06T07:37:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.888+0000 7f893aa00640 1 -- start start 2024-03-06T07:37:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.889+0000 7f893aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918004d40 0x7f8934152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.889+0000 7f893aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893414c630 0x7f89340cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.889+0000 7f893aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 0x7f89340cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.889+0000 7f893aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89340b3660 con 0x7f893414ca90 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.889+0000 7f893aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89340b34e0 con 0x7f8918004d40 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.889+0000 7f893aa00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89340b37e0 con 0x7f893414c630 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 0x7f89340cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 0x7f89340cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43448/0 (socket says 172.21.15.32:43448) 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 -- 172.21.15.32:0/867875698 learned_addr learned my addr 172.21.15.32:0/867875698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8933e00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918004d40 0x7f8934152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f892b400640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893414c630 0x7f89340cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 -- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893414c630 msgr2=0x7f89340cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893414c630 0x7f89340cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 -- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918004d40 msgr2=0x7f8934152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918004d40 0x7f8934152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 -- 172.21.15.32:0/867875698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934153470 con 0x7f893414ca90 2024-03-06T07:37:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8933e00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918004d40 0x7f8934152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.890+0000 7f8938a00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 0x7f89340cccd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f892c02c590 tx=0x7f892c001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.891+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c007d50 con 0x7f893414ca90 2024-03-06T07:37:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.891+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c004bf0 con 0x7f893414ca90 2024-03-06T07:37:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.891+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c001c90 con 0x7f893414ca90 2024-03-06T07:37:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.891+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893409ad40 con 0x7f893414ca90 2024-03-06T07:37:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.891+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893409b300 con 0x7f893414ca90 2024-03-06T07:37:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.892+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893404f3a0 con 0x7f893414ca90 2024-03-06T07:37:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.894+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f892c007070 con 0x7f893414ca90 2024-03-06T07:37:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.895+0000 7f8930c00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f891006b4d0 0x7f891006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.895+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f892c0aadf0 con 0x7f893414ca90 2024-03-06T07:37:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.896+0000 7f8933e00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f891006b4d0 0x7f891006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.896+0000 7f8933e00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f891006b4d0 0x7f891006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8924004310 tx=0x7f892400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:35.900+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f892c048030 con 0x7f893414ca90 2024-03-06T07:37:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.170+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89340ad530 con 0x7f893414ca90 2024-03-06T07:37:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.171+0000 7f8930c00640 1 -- 172.21.15.32:0/867875698 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f892c079bd0 con 0x7f893414ca90 2024-03-06T07:37:36.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:36.174 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:36.174 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:36.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f891006b4d0 msgr2=0x7f891006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f891006b4d0 0x7f891006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8924004310 tx=0x7f892400b040 comp rx=0 tx=0).stop 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 msgr2=0x7f89340cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 0x7f89340cccd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f892c02c590 tx=0x7f892c001ab0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 shutdown_connections 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f891006b4d0 0x7f891006d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f893414ca90 0x7f89340cccd0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f893414c630 0x7f89340cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 --2- 172.21.15.32:0/867875698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8918004d40 0x7f8934152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 >> 172.21.15.32:0/867875698 conn(0x7f893406afb0 msgr2=0x7f89340ad810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 shutdown_connections 2024-03-06T07:37:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:36.174+0000 7f893aa00640 1 -- 172.21.15.32:0/867875698 wait complete. 2024-03-06T07:37:36.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:36.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:37.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:37.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.627+0000 7ff74ec00640 1 Processor -- start 2024-03-06T07:37:37.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.627+0000 7ff74ec00640 1 -- start start 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.629+0000 7ff74ec00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff74815c360 0x7ff74806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.629+0000 7ff74ec00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480afb40 con 0x7ff74806c690 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.629+0000 7ff74ec00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480afe20 con 0x7ff7480ce370 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.629+0000 7ff74ec00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74806d310 con 0x7ff74815c360 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.630+0000 7ff74e200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff74806c690 0x7ff74815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55800/0 (socket says 172.21.15.32:55800) 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.630+0000 7ff74e200640 1 -- 172.21.15.32:0/2879566030 learned_addr learned my addr 172.21.15.32:0/2879566030 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.630+0000 7ff74d800640 1 --2- 172.21.15.32:0/2879566030 >> v2:172.21.15.39:3300/0 conn(0x7ff74815c360 0x7ff74806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.630+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303134602 0 0) 0x7ff7480afb40 con 0x7ff74806c690 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.630+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff730003520 con 0x7ff74806c690 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903191144 0 0) 0x7ff730003520 con 0x7ff74806c690 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 >> v2:172.21.15.39:3300/0 conn(0x7ff74815c360 msgr2=0x7ff74806ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 --2- 172.21.15.32:0/2879566030 >> v2:172.21.15.39:3300/0 conn(0x7ff74815c360 0x7ff74806ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 >> v1:172.21.15.39:6789/0 conn(0x7ff7480ce370 legacy=0x7ff74806bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74806d560 con 0x7ff74806c690 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff7400019f0 con 0x7ff74806c690 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 >> v1:172.21.15.32:6789/0 conn(0x7ff74806c690 legacy=0x7ff74815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff730003a00 0x7ff730003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff730005210 0x7ff73000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74806d310 con 0x7ff730004800 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000c160 con 0x7ff730003a00 2024-03-06T07:37:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000c360 con 0x7ff730005210 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff74ce00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.631+0000 7ff74d800640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff730003a00 0x7ff730003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74ce00640 1 -- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff730005210 msgr2=0x7ff73000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74e200640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff730005210 0x7ff73000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74ce00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff730005210 0x7ff73000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74ce00640 1 -- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff730003a00 msgr2=0x7ff730003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74ce00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff730003a00 0x7ff730003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74ce00640 1 -- 172.21.15.32:0/2879566030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730003520 con 0x7ff730004800 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74d800640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff730003a00 0x7ff730003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74e200640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff730005210 0x7ff73000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.632+0000 7ff74e200640 1 -- 172.21.15.32:0/2879566030 reap_dead start 2024-03-06T07:37:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.633+0000 7ff74ce00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff730004c80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff734001720 tx=0x7ff730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fe9d10a7fc30c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.633+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734001370 con 0x7ff730004800 2024-03-06T07:37:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.633+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7340052d0 con 0x7ff730004800 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.633+0000 7ff747e00640 1 -- 172.21.15.32:0/2879566030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734003830 con 0x7ff730004800 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 -- 172.21.15.32:0/2879566030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 msgr2=0x7ff730004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff730004c80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff734001720 tx=0x7ff730005830 comp rx=0 tx=0).stop 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 -- 172.21.15.32:0/2879566030 shutdown_connections 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 --2- 172.21.15.32:0/2879566030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 -- 172.21.15.32:0/2879566030 >> 172.21.15.32:0/2879566030 conn(0x7ff7480ccf70 msgr2=0x7ff7480cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 -- 172.21.15.32:0/2879566030 shutdown_connections 2024-03-06T07:37:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.634+0000 7ff74ec00640 1 -- 172.21.15.32:0/2879566030 wait complete. 2024-03-06T07:37:37.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.637+0000 7ff74ec00640 1 Processor -- start 2024-03-06T07:37:37.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.637+0000 7ff74ec00640 1 -- start start 2024-03-06T07:37:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff7481e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c5490 0x7ff7481f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74ec00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff74814da20 0x7ff74814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74ec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74815bfa0 con 0x7ff730004800 2024-03-06T07:37:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74806d840 con 0x7ff7480c5490 2024-03-06T07:37:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74ec00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74815c120 con 0x7ff74814da20 2024-03-06T07:37:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff7481e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff7481e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43462/0 (socket says 172.21.15.32:43462) 2024-03-06T07:37:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.639+0000 7ff74d800640 1 -- 172.21.15.32:0/1316851480 learned_addr learned my addr 172.21.15.32:0/1316851480 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74e200640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff74814da20 0x7ff74814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74ce00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c5490 0x7ff7481f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74d800640 1 -- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff74814da20 msgr2=0x7ff74814dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74d800640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff74814da20 0x7ff74814dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74d800640 1 -- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c5490 msgr2=0x7ff7481f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74d800640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c5490 0x7ff7481f7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74d800640 1 -- 172.21.15.32:0/1316851480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7481e6480 con 0x7ff730004800 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74e200640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff74814da20 0x7ff74814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74d800640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff7481e5e30 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7ff73c011730 tx=0x7ff73c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c00f970 con 0x7ff730004800 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c00fb10 con 0x7ff730004800 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c015460 con 0x7ff730004800 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.640+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7481e6760 con 0x7ff730004800 2024-03-06T07:37:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.641+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7481e6d20 con 0x7ff730004800 2024-03-06T07:37:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.642+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff748072270 con 0x7ff730004800 2024-03-06T07:37:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.643+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff73c015600 con 0x7ff730004800 2024-03-06T07:37:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.643+0000 7ff745600640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff72806b500 0x7ff72806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.644+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff73c08ad00 con 0x7ff730004800 2024-03-06T07:37:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.644+0000 7ff74ce00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff72806b500 0x7ff72806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.645+0000 7ff74ce00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff72806b500 0x7ff72806d9c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff734033000 tx=0x7ff734009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:37.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.652+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff748072270 con 0x7ff730004800 2024-03-06T07:37:37.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.894+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff74804f330 con 0x7ff730004800 2024-03-06T07:37:37.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.895+0000 7ff745600640 1 -- 172.21.15.32:0/1316851480 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7ff73c019030 con 0x7ff730004800 2024-03-06T07:37:37.897 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:37.897 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:37.898 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.897+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff72806b500 msgr2=0x7ff72806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.897+0000 7ff74ec00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff72806b500 0x7ff72806d9c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff734033000 tx=0x7ff734009030 comp rx=0 tx=0).stop 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.897+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 msgr2=0x7ff7481e5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.897+0000 7ff74ec00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff7481e5e30 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7ff73c011730 tx=0x7ff73c011bf0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 shutdown_connections 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff72806b500 0x7ff72806d9c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff74814da20 0x7ff74814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c5490 0x7ff7481f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 --2- 172.21.15.32:0/1316851480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff730004800 0x7ff7481e5e30 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 >> 172.21.15.32:0/1316851480 conn(0x7ff7480ccf70 msgr2=0x7ff74815e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 shutdown_connections 2024-03-06T07:37:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:37.898+0000 7ff74ec00640 1 -- 172.21.15.32:0/1316851480 wait complete. 2024-03-06T07:37:38.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:38.120 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:39.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:39.337 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.343+0000 7fd67f400640 1 Processor -- start 2024-03-06T07:37:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.344+0000 7fd67f400640 1 -- start start 2024-03-06T07:37:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.345+0000 7fd67f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd67815e2e0 0x7fd6780d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.345+0000 7fd67f400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6780b3940 con 0x7fd6780d0d70 2024-03-06T07:37:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.345+0000 7fd67f400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6780cf580 con 0x7fd6780b50e0 2024-03-06T07:37:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.345+0000 7fd67f400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6780d4e70 con 0x7fd67815e2e0 2024-03-06T07:37:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.346+0000 7fd677e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd6780d0d70 0x7fd6780d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35022/0 (socket says 172.21.15.32:35022) 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.346+0000 7fd677e00640 1 -- 172.21.15.32:0/483758320 learned_addr learned my addr 172.21.15.32:0/483758320 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.346+0000 7fd67ca00640 1 --2- 172.21.15.32:0/483758320 >> v2:172.21.15.39:3300/0 conn(0x7fd67815e2e0 0x7fd6780d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.346+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304978030 0 0) 0x7fd6780b3940 con 0x7fd6780d0d70 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.346+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd660003520 con 0x7fd6780d0d70 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898411321 0 0) 0x7fd660003520 con 0x7fd6780d0d70 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 >> v2:172.21.15.39:3300/0 conn(0x7fd67815e2e0 msgr2=0x7fd6780d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 --2- 172.21.15.32:0/483758320 >> v2:172.21.15.39:3300/0 conn(0x7fd67815e2e0 0x7fd6780d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 >> v1:172.21.15.39:6789/0 conn(0x7fd6780b50e0 legacy=0x7fd6780d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780cf860 con 0x7fd6780d0d70 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd664001670 con 0x7fd6780d0d70 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 >> v1:172.21.15.32:6789/0 conn(0x7fd6780d0d70 legacy=0x7fd6780d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd660003a70 0x7fd660003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd660004800 0x7fd660004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660005210 0x7fd66000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66000c110 con 0x7fd660003a70 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66000c2c0 con 0x7fd660005210 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66000c4c0 con 0x7fd660004800 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67ca00640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd660003a70 0x7fd660003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67ca00640 1 -- 172.21.15.32:0/483758320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd660004800 msgr2=0x7fd660004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67ca00640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd660004800 0x7fd660004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67ca00640 1 -- 172.21.15.32:0/483758320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660005210 msgr2=0x7fd66000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67ca00640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660005210 0x7fd66000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67ca00640 1 -- 172.21.15.32:0/483758320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd660003520 con 0x7fd660003a70 2024-03-06T07:37:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.347+0000 7fd67d400640 1 -- 172.21.15.32:0/483758320 reap_dead start 2024-03-06T07:37:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.349+0000 7fd67ca00640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd660003a70 0x7fd660003e90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009ac0 tx=0x7fd660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961d402ddc4e7cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.349+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c039070 con 0x7fd660003a70 2024-03-06T07:37:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.349+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c02c710 con 0x7fd660003a70 2024-03-06T07:37:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.349+0000 7fd677400640 1 -- 172.21.15.32:0/483758320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c009690 con 0x7fd660003a70 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 -- 172.21.15.32:0/483758320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd660003a70 msgr2=0x7fd660003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd660003a70 0x7fd660003e90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009ac0 tx=0x7fd660005830 comp rx=0 tx=0).stop 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 -- 172.21.15.32:0/483758320 shutdown_connections 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 --2- 172.21.15.32:0/483758320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd660003a70 0x7fd660003e90 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 -- 172.21.15.32:0/483758320 >> 172.21.15.32:0/483758320 conn(0x7fd6780d3f60 msgr2=0x7fd6780d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 -- 172.21.15.32:0/483758320 shutdown_connections 2024-03-06T07:37:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.350+0000 7fd67f400640 1 -- 172.21.15.32:0/483758320 wait complete. 2024-03-06T07:37:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.352+0000 7fd67f400640 1 Processor -- start 2024-03-06T07:37:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.352+0000 7fd67f400640 1 -- start start 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660003a70 0x7fd6781f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67f400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6781f3e90 0x7fd6780b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 0x7fd6780b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67f400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780cfe20 con 0x7fd6781f4310 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67f400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780b3c40 con 0x7fd660003a70 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67f400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780d0830 con 0x7fd6781f3e90 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 0x7fd6780b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 0x7fd6780b8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53116/0 (socket says 172.21.15.32:53116) 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.353+0000 7fd67d400640 1 -- 172.21.15.32:0/1582517955 learned_addr learned my addr 172.21.15.32:0/1582517955 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd677e00640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6781f3e90 0x7fd6780b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67ca00640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660003a70 0x7fd6781f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67d400640 1 -- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6781f3e90 msgr2=0x7fd6780b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67d400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6781f3e90 0x7fd6780b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67d400640 1 -- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660003a70 msgr2=0x7fd6781f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67d400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660003a70 0x7fd6781f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67d400640 1 -- 172.21.15.32:0/1582517955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780b8250 con 0x7fd6781f4310 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd67d400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 0x7fd6780b8fa0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd6780b5570 tx=0x7fd67000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.354+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670017070 con 0x7fd6781f4310 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.355+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6700070d0 con 0x7fd6781f4310 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.355+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700074c0 con 0x7fd6781f4310 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.355+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6780cb0d0 con 0x7fd6781f4310 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.355+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6780cb690 con 0x7fd6781f4310 2024-03-06T07:37:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.356+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67804f3a0 con 0x7fd6781f4310 2024-03-06T07:37:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.357+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd67001a070 con 0x7fd6781f4310 2024-03-06T07:37:39.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.358+0000 7fd674c00640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd65806b4d0 0x7fd65806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:39.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.358+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6700894c0 con 0x7fd6781f4310 2024-03-06T07:37:39.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.358+0000 7fd67ca00640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd65806b4d0 0x7fd65806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:39.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.359+0000 7fd67ca00640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd65806b4d0 0x7fd65806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd66c036000 tx=0x7fd66c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:39.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.367+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd670058340 con 0x7fd6781f4310 2024-03-06T07:37:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.601+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6780b68d0 con 0x7fd6781f4310 2024-03-06T07:37:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.601+0000 7fd674c00640 1 -- 172.21.15.32:0/1582517955 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7fd670057c40 con 0x7fd6781f4310 2024-03-06T07:37:39.604 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:39.604 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:39.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.603+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd65806b4d0 msgr2=0x7fd65806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.603+0000 7fd67f400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd65806b4d0 0x7fd65806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd66c036000 tx=0x7fd66c030000 comp rx=0 tx=0).stop 2024-03-06T07:37:39.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 msgr2=0x7fd6780b8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:39.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 0x7fd6780b8fa0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd6780b5570 tx=0x7fd67000c980 comp rx=0 tx=0).stop 2024-03-06T07:37:39.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 shutdown_connections 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd65806b4d0 0x7fd65806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6781f4310 0x7fd6780b8fa0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6781f3e90 0x7fd6780b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 --2- 172.21.15.32:0/1582517955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd660003a70 0x7fd6781f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 >> 172.21.15.32:0/1582517955 conn(0x7fd6780d3f60 msgr2=0x7fd6780b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 shutdown_connections 2024-03-06T07:37:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:39.604+0000 7fd67f400640 1 -- 172.21.15.32:0/1582517955 wait complete. 2024-03-06T07:37:39.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:39.828 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:40.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:41.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.049+0000 7f4ce7200640 1 Processor -- start 2024-03-06T07:37:41.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.049+0000 7f4ce7200640 1 -- start start 2024-03-06T07:37:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.050+0000 7f4ce7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ce815e0b0 0x7f4ce80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d0e50 0x7f4ce80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce80b3710 con 0x7f4ce80d4a10 2024-03-06T07:37:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce80b09e0 con 0x7f4ce815e0b0 2024-03-06T07:37:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80b39f0 con 0x7f4ce80d0e50 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ce815e0b0 0x7f4ce80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce6800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d0e50 0x7f4ce80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ce815e0b0 0x7f4ce80cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:39088/0 (socket says 172.21.15.32:39088) 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.051+0000 7f4ce5e00640 1 -- 172.21.15.32:0/3580570406 learned_addr learned my addr 172.21.15.32:0/3580570406 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.052+0000 7f4ce5e00640 1 -- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d0e50 msgr2=0x7f4ce80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.052+0000 7f4ce5e00640 1 --2- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d0e50 0x7f4ce80b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.052+0000 7f4ce5e00640 1 -- 172.21.15.32:0/3580570406 >> v1:172.21.15.39:6789/0 conn(0x7f4ce80d4a10 legacy=0x7f4ce80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.052+0000 7f4ce5e00640 1 -- 172.21.15.32:0/3580570406 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80b3cd0 con 0x7f4ce815e0b0 2024-03-06T07:37:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.053+0000 7f4ce5e00640 1 --2- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce815e0b0 0x7f4ce80cf310 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009870 tx=0x7f4cd802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba9c4bd66bff7c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:41.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce5400640 1 -- 172.21.15.32:0/3580570406 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8034070 con 0x7f4ce815e0b0 2024-03-06T07:37:41.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce5400640 1 -- 172.21.15.32:0/3580570406 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd8030840 con 0x7f4ce815e0b0 2024-03-06T07:37:41.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce5400640 1 -- 172.21.15.32:0/3580570406 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8030c10 con 0x7f4ce815e0b0 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 -- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce815e0b0 msgr2=0x7f4ce80cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 --2- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce815e0b0 0x7f4ce80cf310 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009870 tx=0x7f4cd802b980 comp rx=0 tx=0).stop 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 -- 172.21.15.32:0/3580570406 shutdown_connections 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 --2- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d0e50 0x7f4ce80b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 --2- 172.21.15.32:0/3580570406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce815e0b0 0x7f4ce80cf310 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 -- 172.21.15.32:0/3580570406 >> 172.21.15.32:0/3580570406 conn(0x7f4ce80d3d30 msgr2=0x7f4ce80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.054+0000 7f4ce7200640 1 -- 172.21.15.32:0/3580570406 shutdown_connections 2024-03-06T07:37:41.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.055+0000 7f4ce7200640 1 -- 172.21.15.32:0/3580570406 wait complete. 2024-03-06T07:37:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.056+0000 7f4ce7200640 1 Processor -- start 2024-03-06T07:37:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.056+0000 7f4ce7200640 1 -- start start 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce80d0e50 0x7f4ce814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d4a10 0x7f4ce806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 0x7f4ce806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80cfc60 con 0x7f4ce815e0b0 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce815e490 con 0x7f4ce80d0e50 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80cf850 con 0x7f4ce80d4a10 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.057+0000 7f4ce6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 0x7f4ce806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 0x7f4ce806c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53122/0 (socket says 172.21.15.32:53122) 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 -- 172.21.15.32:0/526583703 learned_addr learned my addr 172.21.15.32:0/526583703 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4cdd400640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d4a10 0x7f4ce806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 -- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d4a10 msgr2=0x7f4ce806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce5e00640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce80d0e50 0x7f4ce814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d4a10 0x7f4ce806c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 -- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce80d0e50 msgr2=0x7f4ce814c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce80d0e50 0x7f4ce814c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 -- 172.21.15.32:0/526583703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce8160290 con 0x7f4ce815e0b0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4ce6800640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 0x7f4ce806c7f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000efc0 tx=0x7f4ce000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.058+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0018020 con 0x7f4ce815e0b0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.059+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce0014480 con 0x7f4ce815e0b0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.059+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0014870 con 0x7f4ce815e0b0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.059+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce8160570 con 0x7f4ce815e0b0 2024-03-06T07:37:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.059+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce8160b30 con 0x7f4ce815e0b0 2024-03-06T07:37:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.060+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce8160d30 con 0x7f4ce815e0b0 2024-03-06T07:37:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.061+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ce0005d10 con 0x7f4ce815e0b0 2024-03-06T07:37:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.062+0000 7f4cdea00640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cc406b500 0x7f4cc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.062+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ce008b110 con 0x7f4ce815e0b0 2024-03-06T07:37:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.062+0000 7f4ce5e00640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cc406b500 0x7f4cc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.063+0000 7f4ce5e00640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cc406b500 0x7f4cc406d9c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8036000 tx=0x7f4cd803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.067+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ce8160d30 con 0x7f4ce815e0b0 2024-03-06T07:37:41.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.308+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ce804f330 con 0x7f4ce815e0b0 2024-03-06T07:37:41.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.309+0000 7f4cdea00640 1 -- 172.21.15.32:0/526583703 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f4ce0022030 con 0x7f4ce815e0b0 2024-03-06T07:37:41.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:41.312 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:41.312 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:41.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cc406b500 msgr2=0x7f4cc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cc406b500 0x7f4cc406d9c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8036000 tx=0x7f4cd803b040 comp rx=0 tx=0).stop 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 msgr2=0x7f4ce806c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 0x7f4ce806c7f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000efc0 tx=0x7f4ce000c490 comp rx=0 tx=0).stop 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 shutdown_connections 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4cc406b500 0x7f4cc406d9c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ce815e0b0 0x7f4ce806c7f0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ce80d4a10 0x7f4ce806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 --2- 172.21.15.32:0/526583703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ce80d0e50 0x7f4ce814c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 >> 172.21.15.32:0/526583703 conn(0x7f4ce80d3d30 msgr2=0x7f4ce80b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 shutdown_connections 2024-03-06T07:37:41.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:41.312+0000 7f4ce7200640 1 -- 172.21.15.32:0/526583703 wait complete. 2024-03-06T07:37:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:41.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:41.534 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:42.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.764+0000 7f938a600640 1 Processor -- start 2024-03-06T07:37:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.764+0000 7f938a600640 1 -- start start 2024-03-06T07:37:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f938a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840af1a0 0x7f938415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f938a600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f93840af580 0x7f93840b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f938a600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f938415cc40 con 0x7f93840cdc20 2024-03-06T07:37:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f938a600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938415cf20 con 0x7f93840af580 2024-03-06T07:37:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f938a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938415d200 con 0x7f93840af1a0 2024-03-06T07:37:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f9383400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840af1a0 0x7f938415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f9383e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f93840cdc20 0x7f93840b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:58318/0 (socket says 172.21.15.32:58318) 2024-03-06T07:37:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.766+0000 7f9383e00640 1 -- 172.21.15.32:0/2158865600 learned_addr learned my addr 172.21.15.32:0/2158865600 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9382a00640 1 --2- 172.21.15.32:0/2158865600 >> v2:172.21.15.39:3300/0 conn(0x7f93840af580 0x7f93840b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9382000640 1 -- 172.21.15.32:0/2158865600 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742560598 0 0) 0x7f938415cc40 con 0x7f93840cdc20 2024-03-06T07:37:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9382000640 1 -- 172.21.15.32:0/2158865600 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9368003520 con 0x7f93840cdc20 2024-03-06T07:37:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9383400640 1 -- 172.21.15.32:0/2158865600 >> v2:172.21.15.39:3300/0 conn(0x7f93840af580 msgr2=0x7f93840b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9383400640 1 --2- 172.21.15.32:0/2158865600 >> v2:172.21.15.39:3300/0 conn(0x7f93840af580 0x7f93840b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9383400640 1 -- 172.21.15.32:0/2158865600 >> v1:172.21.15.39:6789/0 conn(0x7f93840cdc20 legacy=0x7f93840b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.767+0000 7f9383400640 1 -- 172.21.15.32:0/2158865600 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938415d4e0 con 0x7f93840af1a0 2024-03-06T07:37:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.769+0000 7f9383400640 1 --2- 172.21.15.32:0/2158865600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840af1a0 0x7f938415c600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9374009d90 tx=0x7f937402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9c8ab7603d6523 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.769+0000 7f9382000640 1 -- 172.21.15.32:0/2158865600 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374039070 con 0x7f93840af1a0 2024-03-06T07:37:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.769+0000 7f9382000640 1 -- 172.21.15.32:0/2158865600 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937402cbc0 con 0x7f93840af1a0 2024-03-06T07:37:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.769+0000 7f9382000640 1 -- 172.21.15.32:0/2158865600 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374034440 con 0x7f93840af1a0 2024-03-06T07:37:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 -- 172.21.15.32:0/2158865600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840af1a0 msgr2=0x7f938415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 --2- 172.21.15.32:0/2158865600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840af1a0 0x7f938415c600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9374009d90 tx=0x7f937402be10 comp rx=0 tx=0).stop 2024-03-06T07:37:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 -- 172.21.15.32:0/2158865600 shutdown_connections 2024-03-06T07:37:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 --2- 172.21.15.32:0/2158865600 >> v2:172.21.15.39:3300/0 conn(0x7f93840af580 0x7f93840b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 --2- 172.21.15.32:0/2158865600 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840af1a0 0x7f938415c600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 -- 172.21.15.32:0/2158865600 >> 172.21.15.32:0/2158865600 conn(0x7f938406afb0 msgr2=0x7f938406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 -- 172.21.15.32:0/2158865600 shutdown_connections 2024-03-06T07:37:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.770+0000 7f938a600640 1 -- 172.21.15.32:0/2158865600 wait complete. 2024-03-06T07:37:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.772+0000 7f938a600640 1 Processor -- start 2024-03-06T07:37:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.772+0000 7f938a600640 1 -- start start 2024-03-06T07:37:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f938a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 0x7f93840cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f938a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93840af580 0x7f93840b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f938a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840cdc20 0x7f93841f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f938a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93840b3660 con 0x7f93840af1a0 2024-03-06T07:37:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f938a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93840b34e0 con 0x7f93840af580 2024-03-06T07:37:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f938a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93840b37e0 con 0x7f93840cdc20 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f9383400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 0x7f93840cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f9383400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 0x7f93840cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53136/0 (socket says 172.21.15.32:53136) 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.773+0000 7f9383400640 1 -- 172.21.15.32:0/1990276022 learned_addr learned my addr 172.21.15.32:0/1990276022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9382a00640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93840af580 0x7f93840b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383e00640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840cdc20 0x7f93841f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383400640 1 -- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840cdc20 msgr2=0x7f93841f4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383400640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840cdc20 0x7f93841f4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383400640 1 -- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93840af580 msgr2=0x7f93840b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383400640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93840af580 0x7f93840b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383400640 1 -- 172.21.15.32:0/1990276022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93840ad7c0 con 0x7f93840af1a0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.774+0000 7f9383400640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 0x7f93840cac10 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f9374037040 tx=0x7f9374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.775+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374039050 con 0x7f93840af1a0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.775+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9374045070 con 0x7f93840af1a0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.775+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374009450 con 0x7f93840af1a0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.775+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93840adaa0 con 0x7f93840af1a0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.775+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93840cb150 con 0x7f93840af1a0 2024-03-06T07:37:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.776+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9384069510 con 0x7f93840af1a0 2024-03-06T07:37:42.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.777+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f937403e850 con 0x7f93840af1a0 2024-03-06T07:37:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.778+0000 7f937b400640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f936406b4d0 0x7f936406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.778+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93740acef0 con 0x7f93840af1a0 2024-03-06T07:37:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.778+0000 7f9382a00640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f936406b4d0 0x7f936406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.779+0000 7f9382a00640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f936406b4d0 0x7f936406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f93840ae2a0 tx=0x7f9370008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:42.783+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9374032e20 con 0x7f93840af1a0 2024-03-06T07:37:43.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.135+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f938404f3a0 con 0x7f93840af1a0 2024-03-06T07:37:43.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.136+0000 7f937b400640 1 -- 172.21.15.32:0/1990276022 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f937407bcd0 con 0x7f93840af1a0 2024-03-06T07:37:43.139 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:43.139 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:43.139 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:43.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.138+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f936406b4d0 msgr2=0x7f936406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.138+0000 7f938a600640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f936406b4d0 0x7f936406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f93840ae2a0 tx=0x7f9370008040 comp rx=0 tx=0).stop 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.138+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 msgr2=0x7f93840cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.138+0000 7f938a600640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 0x7f93840cac10 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f9374037040 tx=0x7f9374004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 shutdown_connections 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f936406b4d0 0x7f936406d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f93840cdc20 0x7f93841f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93840af580 0x7f93840b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 --2- 172.21.15.32:0/1990276022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93840af1a0 0x7f93840cac10 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 >> 172.21.15.32:0/1990276022 conn(0x7f938406afb0 msgr2=0x7f938406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 shutdown_connections 2024-03-06T07:37:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:43.139+0000 7f938a600640 1 -- 172.21.15.32:0/1990276022 wait complete. 2024-03-06T07:37:43.341 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:43.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:43.363 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:44.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.592+0000 7f9ac6800640 1 Processor -- start 2024-03-06T07:37:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.593+0000 7f9ac6800640 1 -- start start 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.594+0000 7f9ac6800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9ac00af580 0x7f9ac015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.594+0000 7f9ac6800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ac00cdc20 0x7f9ac00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.594+0000 7f9ac6800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac015cc40 con 0x7f9ac00af1a0 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.594+0000 7f9ac6800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac015cf20 con 0x7f9ac00cdc20 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.594+0000 7f9ac6800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac015d200 con 0x7f9ac00af580 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ac00cdc20 0x7f9ac00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ac00cdc20 0x7f9ac00b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53146/0 (socket says 172.21.15.32:53146) 2024-03-06T07:37:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 -- 172.21.15.32:0/2273011614 learned_addr learned my addr 172.21.15.32:0/2273011614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abea00640 1 --2- 172.21.15.32:0/2273011614 >> v2:172.21.15.39:3300/0 conn(0x7f9ac00af580 0x7f9ac015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 -- 172.21.15.32:0/2273011614 >> v2:172.21.15.39:3300/0 conn(0x7f9ac00af580 msgr2=0x7f9ac015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 --2- 172.21.15.32:0/2273011614 >> v2:172.21.15.39:3300/0 conn(0x7f9ac00af580 0x7f9ac015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 -- 172.21.15.32:0/2273011614 >> v1:172.21.15.39:6789/0 conn(0x7f9ac00af1a0 legacy=0x7f9ac00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.595+0000 7f9abfe00640 1 -- 172.21.15.32:0/2273011614 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac015d4e0 con 0x7f9ac00cdc20 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.596+0000 7f9abfe00640 1 --2- 172.21.15.32:0/2273011614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00cdc20 0x7f9ac00b2dd0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00b54c0 tx=0x7f9ab802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f877b49f141e9fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9abe000640 1 -- 172.21.15.32:0/2273011614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802c9f0 con 0x7f9ac00cdc20 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9abe000640 1 -- 172.21.15.32:0/2273011614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab802cb90 con 0x7f9ac00cdc20 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9abe000640 1 -- 172.21.15.32:0/2273011614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80343f0 con 0x7f9ac00cdc20 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9ac6800640 1 -- 172.21.15.32:0/2273011614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00cdc20 msgr2=0x7f9ac00b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9ac6800640 1 --2- 172.21.15.32:0/2273011614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00cdc20 0x7f9ac00b2dd0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00b54c0 tx=0x7f9ab802be50 comp rx=0 tx=0).stop 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9ac6800640 1 -- 172.21.15.32:0/2273011614 shutdown_connections 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9ac6800640 1 --2- 172.21.15.32:0/2273011614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00cdc20 0x7f9ac00b2dd0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9ac6800640 1 --2- 172.21.15.32:0/2273011614 >> v2:172.21.15.39:3300/0 conn(0x7f9ac00af580 0x7f9ac015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.597+0000 7f9ac6800640 1 -- 172.21.15.32:0/2273011614 >> 172.21.15.32:0/2273011614 conn(0x7f9ac006afb0 msgr2=0x7f9ac006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:44.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.598+0000 7f9ac6800640 1 -- 172.21.15.32:0/2273011614 shutdown_connections 2024-03-06T07:37:44.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.598+0000 7f9ac6800640 1 -- 172.21.15.32:0/2273011614 wait complete. 2024-03-06T07:37:44.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.600+0000 7f9ac6800640 1 Processor -- start 2024-03-06T07:37:44.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.601+0000 7f9ac6800640 1 -- start start 2024-03-06T07:37:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.602+0000 7f9ac6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 0x7f9ac0156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.602+0000 7f9ac6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ac00af580 0x7f9ac015e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.602+0000 7f9ac6800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ac00cdc20 0x7f9ac0154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.602+0000 7f9ac6800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b36a0 con 0x7f9ac00af1a0 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.602+0000 7f9ac6800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b3520 con 0x7f9ac00af580 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.602+0000 7f9ac6800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00b3820 con 0x7f9ac00cdc20 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 0x7f9ac0156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 0x7f9ac0156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53152/0 (socket says 172.21.15.32:53152) 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 -- 172.21.15.32:0/3469136714 learned_addr learned my addr 172.21.15.32:0/3469136714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 -- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ac00cdc20 msgr2=0x7f9ac0154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ac00cdc20 0x7f9ac0154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 -- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ac00af580 msgr2=0x7f9ac015e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abea00640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ac00af580 0x7f9ac015e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ac00af580 0x7f9ac015e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.603+0000 7f9abf400640 1 -- 172.21.15.32:0/3469136714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac01573d0 con 0x7f9ac00af1a0 2024-03-06T07:37:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.604+0000 7f9abf400640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 0x7f9ac0156df0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c430 tx=0x7f9ab000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.604+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0011790 con 0x7f9ac00af1a0 2024-03-06T07:37:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.604+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab0017070 con 0x7f9ac00af1a0 2024-03-06T07:37:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.604+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0153b80 con 0x7f9ac00af1a0 2024-03-06T07:37:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.604+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0011b70 con 0x7f9ac00af1a0 2024-03-06T07:37:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.604+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0154140 con 0x7f9ac00af1a0 2024-03-06T07:37:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.605+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac0154340 con 0x7f9ac00af1a0 2024-03-06T07:37:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.606+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ab0011d10 con 0x7f9ac00af1a0 2024-03-06T07:37:44.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.607+0000 7f9ab7400640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9aa006b4d0 0x7f9aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.607+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ab0089fc0 con 0x7f9ac00af1a0 2024-03-06T07:37:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.608+0000 7f9abea00640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9aa006b4d0 0x7f9aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.608+0000 7f9abea00640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9aa006b4d0 0x7f9aa006d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0157760 tx=0x7f9aac008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.612+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ac0154340 con 0x7f9ac00af1a0 2024-03-06T07:37:44.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.849+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ac00b5140 con 0x7f9ac00af1a0 2024-03-06T07:37:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.850+0000 7f9ab7400640 1 -- 172.21.15.32:0/3469136714 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f9ab0058ed0 con 0x7f9ac00af1a0 2024-03-06T07:37:44.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:44.852 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:44.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9aa006b4d0 msgr2=0x7f9aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9aa006b4d0 0x7f9aa006d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0157760 tx=0x7f9aac008040 comp rx=0 tx=0).stop 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 msgr2=0x7f9ac0156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 0x7f9ac0156df0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c430 tx=0x7f9ab000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 shutdown_connections 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9aa006b4d0 0x7f9aa006d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ac00cdc20 0x7f9ac0154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ac00af580 0x7f9ac015e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 --2- 172.21.15.32:0/3469136714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac00af1a0 0x7f9ac0156df0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 >> 172.21.15.32:0/3469136714 conn(0x7f9ac006afb0 msgr2=0x7f9ac006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 shutdown_connections 2024-03-06T07:37:44.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:44.854+0000 7f9ac6800640 1 -- 172.21.15.32:0/3469136714 wait complete. 2024-03-06T07:37:45.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:45.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:46.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:46.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.285+0000 7ff0fd400640 1 Processor -- start 2024-03-06T07:37:46.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.286+0000 7ff0fd400640 1 -- start start 2024-03-06T07:37:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.287+0000 7ff0fd400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0f80b34e0 0x7ff0f815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.287+0000 7ff0fd400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f815cc40 con 0x7ff0f80af1a0 2024-03-06T07:37:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.287+0000 7ff0fd400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f815cf20 con 0x7ff0f80af580 2024-03-06T07:37:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.287+0000 7ff0fd400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f815d200 con 0x7ff0f80b34e0 2024-03-06T07:37:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.287+0000 7ff0f6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0f80af1a0 0x7ff0f80b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35028/0 (socket says 172.21.15.32:35028) 2024-03-06T07:37:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.287+0000 7ff0f6800640 1 -- 172.21.15.32:0/1474882530 learned_addr learned my addr 172.21.15.32:0/1474882530 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512980063 0 0) 0x7ff0f815cc40 con 0x7ff0f80af1a0 2024-03-06T07:37:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0dc003520 con 0x7ff0f80af1a0 2024-03-06T07:37:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f7200640 1 --2- 172.21.15.32:0/1474882530 >> v2:172.21.15.39:3300/0 conn(0x7ff0f80b34e0 0x7ff0f815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971106499 0 0) 0x7ff0dc003520 con 0x7ff0f80af1a0 2024-03-06T07:37:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 >> v2:172.21.15.39:3300/0 conn(0x7ff0f80b34e0 msgr2=0x7ff0f815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 --2- 172.21.15.32:0/1474882530 >> v2:172.21.15.39:3300/0 conn(0x7ff0f80b34e0 0x7ff0f815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 >> v1:172.21.15.39:6789/0 conn(0x7ff0f80af580 legacy=0x7ff0f80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f815d4e0 con 0x7ff0f80af1a0 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff0e8001670 con 0x7ff0f80af1a0 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 >> v1:172.21.15.32:6789/0 conn(0x7ff0f80af1a0 legacy=0x7ff0f80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0dc003a20 0x7ff0dc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.288+0000 7ff0f5e00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0f5e00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0dc0052e0 0x7ff0dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f815d200 con 0x7ff0dc004800 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc00c200 con 0x7ff0dc0052e0 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc00c400 con 0x7ff0dc003a20 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0efe00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0f6800640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0dc003a20 0x7ff0dc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0f7200640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0dc0052e0 0x7ff0dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.289+0000 7ff0efe00640 1 -- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0dc003a20 msgr2=0x7ff0dc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.290+0000 7ff0efe00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0dc003a20 0x7ff0dc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.290+0000 7ff0efe00640 1 -- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0dc0052e0 msgr2=0x7ff0dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.290+0000 7ff0efe00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0dc0052e0 0x7ff0dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.290+0000 7ff0efe00640 1 -- 172.21.15.32:0/1474882530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc003520 con 0x7ff0dc004800 2024-03-06T07:37:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.290+0000 7ff0f7200640 1 -- 172.21.15.32:0/1474882530 reap_dead start 2024-03-06T07:37:46.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.291+0000 7ff0efe00640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0dc004c80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0003d80 tx=0x7ff0dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7518405cc1f3f23d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:46.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.292+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0037070 con 0x7ff0dc004800 2024-03-06T07:37:46.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.292+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e0032600 con 0x7ff0dc004800 2024-03-06T07:37:46.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.292+0000 7ff0f5e00640 1 -- 172.21.15.32:0/1474882530 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0003980 con 0x7ff0dc004800 2024-03-06T07:37:46.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.292+0000 7ff0fd400640 1 -- 172.21.15.32:0/1474882530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 msgr2=0x7ff0dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.292+0000 7ff0fd400640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0dc004c80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0003d80 tx=0x7ff0dc005830 comp rx=0 tx=0).stop 2024-03-06T07:37:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.293+0000 7ff0fd400640 1 -- 172.21.15.32:0/1474882530 shutdown_connections 2024-03-06T07:37:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.293+0000 7ff0fd400640 1 --2- 172.21.15.32:0/1474882530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0dc004c80 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.293+0000 7ff0fd400640 1 -- 172.21.15.32:0/1474882530 >> 172.21.15.32:0/1474882530 conn(0x7ff0f806afb0 msgr2=0x7ff0f806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.293+0000 7ff0fd400640 1 -- 172.21.15.32:0/1474882530 shutdown_connections 2024-03-06T07:37:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.293+0000 7ff0fd400640 1 -- 172.21.15.32:0/1474882530 wait complete. 2024-03-06T07:37:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.294+0000 7ff0fd400640 1 Processor -- start 2024-03-06T07:37:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.295+0000 7ff0fd400640 1 -- start start 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.295+0000 7ff0fd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0f81f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.295+0000 7ff0fd400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0f806e100 0x7ff0f806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0fd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f81588b0 0x7ff0f8158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0fd400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f815c600 con 0x7ff0dc004800 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0fd400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f80cde30 con 0x7ff0f81588b0 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0fd400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f815c780 con 0x7ff0f806e100 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0f81f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0f81f31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53172/0 (socket says 172.21.15.32:53172) 2024-03-06T07:37:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 -- 172.21.15.32:0/3751318554 learned_addr learned my addr 172.21.15.32:0/3751318554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f7200640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f81588b0 0x7ff0f8158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0efe00640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0f806e100 0x7ff0f806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 -- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0f806e100 msgr2=0x7ff0f806e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0f806e100 0x7ff0f806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 -- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f81588b0 msgr2=0x7ff0f8158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f81588b0 0x7ff0f8158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.296+0000 7ff0f6800640 1 -- 172.21.15.32:0/3751318554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f80c91f0 con 0x7ff0dc004800 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.297+0000 7ff0f6800640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0f81f31e0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80017e0 tx=0x7ff0e800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.297+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800ee60 con 0x7ff0dc004800 2024-03-06T07:37:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.297+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e8018070 con 0x7ff0dc004800 2024-03-06T07:37:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.297+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800ca70 con 0x7ff0dc004800 2024-03-06T07:37:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.297+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f80c94d0 con 0x7ff0dc004800 2024-03-06T07:37:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.297+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f80c9a90 con 0x7ff0dc004800 2024-03-06T07:37:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.299+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f806eac0 con 0x7ff0dc004800 2024-03-06T07:37:46.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.300+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0e800cc10 con 0x7ff0dc004800 2024-03-06T07:37:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.301+0000 7ff0eea00640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff0d406b4d0 0x7ff0d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.301+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0e808a020 con 0x7ff0dc004800 2024-03-06T07:37:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.301+0000 7ff0efe00640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff0d406b4d0 0x7ff0d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:46.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.302+0000 7ff0efe00640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff0d406b4d0 0x7ff0d406d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0034000 tx=0x7ff0e0009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:46.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.306+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0e8058f30 con 0x7ff0dc004800 2024-03-06T07:37:46.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.545+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0f8069510 con 0x7ff0dc004800 2024-03-06T07:37:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.546+0000 7ff0eea00640 1 -- 172.21.15.32:0/3751318554 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7ff0e8058830 con 0x7ff0dc004800 2024-03-06T07:37:46.547 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:46.548 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:46.548 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:46.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff0d406b4d0 msgr2=0x7ff0d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff0d406b4d0 0x7ff0d406d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0034000 tx=0x7ff0e0009030 comp rx=0 tx=0).stop 2024-03-06T07:37:46.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 msgr2=0x7ff0f81f31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:46.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0f81f31e0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80017e0 tx=0x7ff0e800ec10 comp rx=0 tx=0).stop 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 shutdown_connections 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff0d406b4d0 0x7ff0d406d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f81588b0 0x7ff0f8158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff0f806e100 0x7ff0f806e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 --2- 172.21.15.32:0/3751318554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0dc004800 0x7ff0f81f31e0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.549+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 >> 172.21.15.32:0/3751318554 conn(0x7ff0f806afb0 msgr2=0x7ff0f80b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.550+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 shutdown_connections 2024-03-06T07:37:46.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:46.550+0000 7ff0fd400640 1 -- 172.21.15.32:0/3751318554 wait complete. 2024-03-06T07:37:46.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:46.770 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:47.345 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:47.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.993+0000 7f51fca00640 1 Processor -- start 2024-03-06T07:37:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.993+0000 7f51fca00640 1 -- start start 2024-03-06T07:37:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.995+0000 7f51fca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f51f815e0b0 0x7f51f80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.995+0000 7f51fca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80b3710 con 0x7f51f80d0e50 2024-03-06T07:37:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.995+0000 7f51fca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80b39f0 con 0x7f51f80d4a10 2024-03-06T07:37:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.995+0000 7f51fca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f80b09e0 con 0x7f51f815e0b0 2024-03-06T07:37:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.995+0000 7f51f7200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f51f80d0e50 0x7f51f80b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50456/0 (socket says 172.21.15.32:50456) 2024-03-06T07:37:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.995+0000 7f51f7200640 1 -- 172.21.15.32:0/3432165454 learned_addr learned my addr 172.21.15.32:0/3432165454 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.996+0000 7f51f6800640 1 --2- 172.21.15.32:0/3432165454 >> v2:172.21.15.39:3300/0 conn(0x7f51f815e0b0 0x7f51f80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.996+0000 7f51f5e00640 1 -- 172.21.15.32:0/3432165454 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205471429 0 0) 0x7f51f80b3710 con 0x7f51f80d0e50 2024-03-06T07:37:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.996+0000 7f51f5e00640 1 -- 172.21.15.32:0/3432165454 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51dc003520 con 0x7f51f80d0e50 2024-03-06T07:37:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.996+0000 7f51f6800640 1 -- 172.21.15.32:0/3432165454 >> v1:172.21.15.39:6789/0 conn(0x7f51f80d4a10 legacy=0x7f51f80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.996+0000 7f51f6800640 1 -- 172.21.15.32:0/3432165454 >> v1:172.21.15.32:6789/0 conn(0x7f51f80d0e50 legacy=0x7f51f80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.996+0000 7f51f6800640 1 -- 172.21.15.32:0/3432165454 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f80b3cd0 con 0x7f51f815e0b0 2024-03-06T07:37:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.998+0000 7f51f6800640 1 --2- 172.21.15.32:0/3432165454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f80cf310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f51e80097a0 tx=0x7f51e802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bace66d01e493694 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.998+0000 7f51f5e00640 1 -- 172.21.15.32:0/3432165454 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8034070 con 0x7f51f815e0b0 2024-03-06T07:37:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.998+0000 7f51f5e00640 1 -- 172.21.15.32:0/3432165454 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e8030610 con 0x7f51f815e0b0 2024-03-06T07:37:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.998+0000 7f51f5e00640 1 -- 172.21.15.32:0/3432165454 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e80309c0 con 0x7f51f815e0b0 2024-03-06T07:37:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.999+0000 7f51fca00640 1 -- 172.21.15.32:0/3432165454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 msgr2=0x7f51f80cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:47.999+0000 7f51fca00640 1 --2- 172.21.15.32:0/3432165454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f80cf310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f51e80097a0 tx=0x7f51e802b960 comp rx=0 tx=0).stop 2024-03-06T07:37:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.000+0000 7f51fca00640 1 -- 172.21.15.32:0/3432165454 shutdown_connections 2024-03-06T07:37:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.000+0000 7f51fca00640 1 --2- 172.21.15.32:0/3432165454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f80cf310 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.000+0000 7f51fca00640 1 -- 172.21.15.32:0/3432165454 >> 172.21.15.32:0/3432165454 conn(0x7f51f80d3d30 msgr2=0x7f51f80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:48.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.000+0000 7f51fca00640 1 -- 172.21.15.32:0/3432165454 shutdown_connections 2024-03-06T07:37:48.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.000+0000 7f51fca00640 1 -- 172.21.15.32:0/3432165454 wait complete. 2024-03-06T07:37:48.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.004+0000 7f51fca00640 1 Processor -- start 2024-03-06T07:37:48.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.004+0000 7f51fca00640 1 -- start start 2024-03-06T07:37:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.006+0000 7f51fca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f51f80d0e50 0x7f51f815c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.006+0000 7f51fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 0x7f51f81f7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.006+0000 7f51fca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f81f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51fca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80cfc60 con 0x7f51f80d4a10 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51fca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f815e490 con 0x7f51f815e0b0 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51fca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80cf850 con 0x7f51f80d0e50 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 0x7f51f81f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 0x7f51f81f7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53464/0 (socket says 172.21.15.32:53464) 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51ede00640 1 -- 172.21.15.32:0/777931673 learned_addr learned my addr 172.21.15.32:0/777931673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51f7200640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f81f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.007+0000 7f51f6800640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f51f80d0e50 0x7f51f815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.008+0000 7f51ede00640 1 -- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f51f80d0e50 msgr2=0x7f51f815c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.008+0000 7f51ede00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f51f80d0e50 0x7f51f815c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.008+0000 7f51ede00640 1 -- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 msgr2=0x7f51f81f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.008+0000 7f51ede00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f81f75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.008+0000 7f51ede00640 1 -- 172.21.15.32:0/777931673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f815e790 con 0x7f51f80d4a10 2024-03-06T07:37:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.008+0000 7f51ede00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 0x7f51f81f7090 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c440 tx=0x7f51e0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.009+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e00052d0 con 0x7f51f80d4a10 2024-03-06T07:37:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.009+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e0005470 con 0x7f51f80d4a10 2024-03-06T07:37:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.009+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f80cbdd0 con 0x7f51f80d4a10 2024-03-06T07:37:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.009+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e0007260 con 0x7f51f80d4a10 2024-03-06T07:37:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.009+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81614c0 con 0x7f51f80d4a10 2024-03-06T07:37:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.010+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f804f330 con 0x7f51f80d4a10 2024-03-06T07:37:48.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.011+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51e0007400 con 0x7f51f80d4a10 2024-03-06T07:37:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.012+0000 7f51ef400640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f51d406b500 0x7f51d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.012+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51e0089670 con 0x7f51f80d4a10 2024-03-06T07:37:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.012+0000 7f51f6800640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f51d406b500 0x7f51d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.013+0000 7f51f6800640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f51d406b500 0x7f51d406d9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f51e8036000 tx=0x7f51e8008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.017+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51e0057e90 con 0x7f51f80d4a10 2024-03-06T07:37:48.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.278+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f80b30f0 con 0x7f51f80d4a10 2024-03-06T07:37:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.280+0000 7f51ef400640 1 -- 172.21.15.32:0/777931673 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f51e005bf30 con 0x7f51f80d4a10 2024-03-06T07:37:48.281 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:48.281 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f51d406b500 msgr2=0x7f51d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f51d406b500 0x7f51d406d9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f51e8036000 tx=0x7f51e8008030 comp rx=0 tx=0).stop 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 msgr2=0x7f51f81f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 0x7f51f81f7090 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c440 tx=0x7f51e0001e70 comp rx=0 tx=0).stop 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 shutdown_connections 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f51d406b500 0x7f51d406d9c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51f815e0b0 0x7f51f81f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f80d4a10 0x7f51f81f7090 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 --2- 172.21.15.32:0/777931673 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f51f80d0e50 0x7f51f815c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.282+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 >> 172.21.15.32:0/777931673 conn(0x7f51f80d3d30 msgr2=0x7f51f80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.283+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 shutdown_connections 2024-03-06T07:37:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:48.283+0000 7f51fca00640 1 -- 172.21.15.32:0/777931673 wait complete. 2024-03-06T07:37:48.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:48.504 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:49.347 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:49.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.724+0000 7f8875a00640 1 Processor -- start 2024-03-06T07:37:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.725+0000 7f8875a00640 1 -- start start 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.726+0000 7f8875a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88700b4450 0x7f88700ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.726+0000 7f8875a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700cf090 con 0x7f8870161c80 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.726+0000 7f8875a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700d0650 con 0x7f887015e000 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.726+0000 7f8875a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887006ef40 con 0x7f88700b4450 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f8866000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88700b4450 0x7f88700ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f8866000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88700b4450 0x7f88700ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53476/0 (socket says 172.21.15.32:53476) 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f886f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8870161c80 0x7f88701620b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50468/0 (socket says 172.21.15.32:50468) 2024-03-06T07:37:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f8866000640 1 -- 172.21.15.32:0/157098231 learned_addr learned my addr 172.21.15.32:0/157098231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f8866000640 1 -- 172.21.15.32:0/157098231 >> v1:172.21.15.39:6789/0 conn(0x7f887015e000 legacy=0x7f88700ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f8866000640 1 -- 172.21.15.32:0/157098231 >> v1:172.21.15.32:6789/0 conn(0x7f8870161c80 legacy=0x7f88701620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f8866000640 1 -- 172.21.15.32:0/157098231 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700d0930 con 0x7f88700b4450 2024-03-06T07:37:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.727+0000 7f886e000640 1 -- 172.21.15.32:0/157098231 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50904503 0 0) 0x7f88700cf090 con 0x7f8870161c80 2024-03-06T07:37:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.728+0000 7f8866000640 1 --2- 172.21.15.32:0/157098231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4450 0x7f88700ceb50 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8858009870 tx=0x7f885802bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7602936d6e62a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f886e000640 1 -- 172.21.15.32:0/157098231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885802d870 con 0x7f88700b4450 2024-03-06T07:37:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f886e000640 1 -- 172.21.15.32:0/157098231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885802da10 con 0x7f88700b4450 2024-03-06T07:37:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f886e000640 1 -- 172.21.15.32:0/157098231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885802dde0 con 0x7f88700b4450 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f8875a00640 1 -- 172.21.15.32:0/157098231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4450 msgr2=0x7f88700ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f8875a00640 1 --2- 172.21.15.32:0/157098231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4450 0x7f88700ceb50 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8858009870 tx=0x7f885802bab0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f8875a00640 1 -- 172.21.15.32:0/157098231 shutdown_connections 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.729+0000 7f8875a00640 1 --2- 172.21.15.32:0/157098231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4450 0x7f88700ceb50 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.730+0000 7f8875a00640 1 -- 172.21.15.32:0/157098231 >> 172.21.15.32:0/157098231 conn(0x7f88700b38a0 msgr2=0x7f88700b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.730+0000 7f8875a00640 1 -- 172.21.15.32:0/157098231 shutdown_connections 2024-03-06T07:37:49.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.730+0000 7f8875a00640 1 -- 172.21.15.32:0/157098231 wait complete. 2024-03-06T07:37:49.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.731+0000 7f8875a00640 1 Processor -- start 2024-03-06T07:37:49.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.731+0000 7f8875a00640 1 -- start start 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.732+0000 7f8875a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88700b4450 0x7f88701544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.732+0000 7f8875a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 0x7f88700c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.732+0000 7f8875a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8870161c80 0x7f88700b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.732+0000 7f8875a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700d0280 con 0x7f887015e000 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.732+0000 7f8875a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887006f190 con 0x7f88700b4450 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.732+0000 7f8875a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88700d0400 con 0x7f8870161c80 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 0x7f88700c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 0x7f88700c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53492/0 (socket says 172.21.15.32:53492) 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 -- 172.21.15.32:0/2018235453 learned_addr learned my addr 172.21.15.32:0/2018235453 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f886ea00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88700b4450 0x7f88701544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f886f400640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8870161c80 0x7f88700b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 -- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8870161c80 msgr2=0x7f88700b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8870161c80 0x7f88700b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 -- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88700b4450 msgr2=0x7f88701544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88700b4450 0x7f88701544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f8866000640 1 -- 172.21.15.32:0/2018235453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887015e480 con 0x7f887015e000 2024-03-06T07:37:49.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.733+0000 7f886ea00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88700b4450 0x7f88701544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:49.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.734+0000 7f8866000640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 0x7f88700c97b0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f885802c0c0 tx=0x7f8858004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:49.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.734+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858047070 con 0x7f887015e000 2024-03-06T07:37:49.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.734+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858037ac0 con 0x7f887015e000 2024-03-06T07:37:49.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.734+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885802e440 con 0x7f887015e000 2024-03-06T07:37:49.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.734+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700ca0e0 con 0x7f887015e000 2024-03-06T07:37:49.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.734+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700ca6a0 con 0x7f887015e000 2024-03-06T07:37:49.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.736+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887004f3a0 con 0x7f887015e000 2024-03-06T07:37:49.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.736+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8858041390 con 0x7f887015e000 2024-03-06T07:37:49.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.738+0000 7f8867400640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f884c06b4d0 0x7f884c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:49.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.738+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f88580abf40 con 0x7f887015e000 2024-03-06T07:37:49.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.738+0000 7f886ea00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f884c06b4d0 0x7f884c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:49.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.739+0000 7f886ea00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f884c06b4d0 0x7f884c06d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8860000f60 tx=0x7f8860009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:49.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:49.741+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f885807ad70 con 0x7f887015e000 2024-03-06T07:37:50.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.093+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88700b14d0 con 0x7f887015e000 2024-03-06T07:37:50.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.094+0000 7f8867400640 1 -- 172.21.15.32:0/2018235453 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f885807a670 con 0x7f887015e000 2024-03-06T07:37:50.095 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:50.095 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:50.096 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.096+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f884c06b4d0 msgr2=0x7f884c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.096+0000 7f8875a00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f884c06b4d0 0x7f884c06d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8860000f60 tx=0x7f8860009040 comp rx=0 tx=0).stop 2024-03-06T07:37:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.096+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 msgr2=0x7f88700c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.096+0000 7f8875a00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 0x7f88700c97b0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f885802c0c0 tx=0x7f8858004730 comp rx=0 tx=0).stop 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 shutdown_connections 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f884c06b4d0 0x7f884c06d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8870161c80 0x7f88700b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e000 0x7f88700c97b0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 --2- 172.21.15.32:0/2018235453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88700b4450 0x7f88701544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 >> 172.21.15.32:0/2018235453 conn(0x7f88700b38a0 msgr2=0x7f88700cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 shutdown_connections 2024-03-06T07:37:50.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:50.097+0000 7f8875a00640 1 -- 172.21.15.32:0/2018235453 wait complete. 2024-03-06T07:37:50.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:50.319 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:50.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:50.921 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:51.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.543+0000 7f4f51e00640 1 Processor -- start 2024-03-06T07:37:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.543+0000 7f4f51e00640 1 -- start start 2024-03-06T07:37:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f51e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4f4c0d04b0 0x7f4f4c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f51e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0700c0 0x7f4f4c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f51e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0cd0c0 con 0x7f4f4c15f450 2024-03-06T07:37:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f51e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4c0d29e0 con 0x7f4f4c0d04b0 2024-03-06T07:37:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f51e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0cd3a0 con 0x7f4f4c0700c0 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f4be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4f4c0d04b0 0x7f4f4c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f4be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4f4c0d04b0 0x7f4f4c070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53502/0 (socket says 172.21.15.32:53502) 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f4be00640 1 -- 172.21.15.32:0/2371319443 learned_addr learned my addr 172.21.15.32:0/2371319443 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.546+0000 7f4f51400640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0700c0 0x7f4f4c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.547+0000 7f4f4be00640 1 -- 172.21.15.32:0/2371319443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0700c0 msgr2=0x7f4f4c0cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.547+0000 7f4f4be00640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0700c0 0x7f4f4c0cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.547+0000 7f4f4be00640 1 -- 172.21.15.32:0/2371319443 >> v1:172.21.15.39:6789/0 conn(0x7f4f4c15f450 legacy=0x7f4f4c0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.547+0000 7f4f4be00640 1 -- 172.21.15.32:0/2371319443 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0cd680 con 0x7f4f4c0d04b0 2024-03-06T07:37:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.547+0000 7f4f51400640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0700c0 0x7f4f4c0cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.548+0000 7f4f4be00640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c0d04b0 0x7f4f4c070990 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009870 tx=0x7f4f3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab150128b3accbe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.548+0000 7f4f4b400640 1 -- 172.21.15.32:0/2371319443 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38034070 con 0x7f4f4c0d04b0 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.548+0000 7f4f4b400640 1 -- 172.21.15.32:0/2371319443 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f38030840 con 0x7f4f4c0d04b0 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.548+0000 7f4f4b400640 1 -- 172.21.15.32:0/2371319443 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38030c10 con 0x7f4f4c0d04b0 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 -- 172.21.15.32:0/2371319443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c0d04b0 msgr2=0x7f4f4c070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c0d04b0 0x7f4f4c070990 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009870 tx=0x7f4f3802b980 comp rx=0 tx=0).stop 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 -- 172.21.15.32:0/2371319443 shutdown_connections 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0700c0 0x7f4f4c0cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 --2- 172.21.15.32:0/2371319443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c0d04b0 0x7f4f4c070990 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 -- 172.21.15.32:0/2371319443 >> 172.21.15.32:0/2371319443 conn(0x7f4f4c0d1ae0 msgr2=0x7f4f4c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 -- 172.21.15.32:0/2371319443 shutdown_connections 2024-03-06T07:37:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.549+0000 7f4f51e00640 1 -- 172.21.15.32:0/2371319443 wait complete. 2024-03-06T07:37:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.551+0000 7f4f51e00640 1 Processor -- start 2024-03-06T07:37:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.551+0000 7f4f51e00640 1 -- start start 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f4c0700c0 0x7f4f4c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0d04b0 0x7f4f4c1f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 0x7f4f4c1f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c15f830 con 0x7f4f4c15f450 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0ae8b0 con 0x7f4f4c0700c0 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c15f9b0 con 0x7f4f4c0d04b0 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 0x7f4f4c1f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 0x7f4f4c1f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53516/0 (socket says 172.21.15.32:53516) 2024-03-06T07:37:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f51400640 1 -- 172.21.15.32:0/1049144039 learned_addr learned my addr 172.21.15.32:0/1049144039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f50a00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f4c0700c0 0x7f4f4c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.553+0000 7f4f4be00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0d04b0 0x7f4f4c1f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f51400640 1 -- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0d04b0 msgr2=0x7f4f4c1f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f51400640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0d04b0 0x7f4f4c1f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f51400640 1 -- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f4c0700c0 msgr2=0x7f4f4c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f51400640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f4c0700c0 0x7f4f4c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f51400640 1 -- 172.21.15.32:0/1049144039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c1f8250 con 0x7f4f4c15f450 2024-03-06T07:37:51.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f51400640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 0x7f4f4c1f3d10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400cef0 tx=0x7f4f4400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:51.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4401a070 con 0x7f4f4c15f450 2024-03-06T07:37:51.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.554+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400d860 con 0x7f4f4c15f450 2024-03-06T07:37:51.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.555+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c0c9070 con 0x7f4f4c15f450 2024-03-06T07:37:51.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.555+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400dc50 con 0x7f4f4c15f450 2024-03-06T07:37:51.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.555+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c0c9630 con 0x7f4f4c15f450 2024-03-06T07:37:51.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.556+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c15d750 con 0x7f4f4c15f450 2024-03-06T07:37:51.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.557+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f4401d070 con 0x7f4f4c15f450 2024-03-06T07:37:51.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.558+0000 7f4f48c00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4f2c06b4d0 0x7f4f2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:51.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.558+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4f4408ab10 con 0x7f4f4c15f450 2024-03-06T07:37:51.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.558+0000 7f4f50a00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4f2c06b4d0 0x7f4f2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:51.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.559+0000 7f4f50a00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4f2c06b4d0 0x7f4f2c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4f40000f60 tx=0x7f4f40009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.563+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f4c15d750 con 0x7f4f4c15f450 2024-03-06T07:37:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.803+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f4c04f330 con 0x7f4f4c15f450 2024-03-06T07:37:51.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.804+0000 7f4f48c00640 1 -- 172.21.15.32:0/1049144039 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f4f4c1f8250 con 0x7f4f4c15f450 2024-03-06T07:37:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:51.805 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4f2c06b4d0 msgr2=0x7f4f2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4f2c06b4d0 0x7f4f2c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4f40000f60 tx=0x7f4f40009040 comp rx=0 tx=0).stop 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 msgr2=0x7f4f4c1f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 0x7f4f4c1f3d10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400cef0 tx=0x7f4f4400a710 comp rx=0 tx=0).stop 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 shutdown_connections 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4f2c06b4d0 0x7f4f2c06d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c15f450 0x7f4f4c1f3d10 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4f4c0d04b0 0x7f4f4c1f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 --2- 172.21.15.32:0/1049144039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f4c0700c0 0x7f4f4c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 >> 172.21.15.32:0/1049144039 conn(0x7f4f4c0d1ae0 msgr2=0x7f4f4c0aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:51.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 shutdown_connections 2024-03-06T07:37:51.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:51.807+0000 7f4f51e00640 1 -- 172.21.15.32:0/1049144039 wait complete. 2024-03-06T07:37:52.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:52.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:53.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:53.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.259+0000 7f90ef200640 1 Processor -- start 2024-03-06T07:37:53.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.259+0000 7f90ef200640 1 -- start start 2024-03-06T07:37:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.260+0000 7f90ef200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 0x7f90f0070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.260+0000 7f90ef200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90f00700c0 0x7f90f00cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.260+0000 7f90ef200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00cd0c0 con 0x7f90f015f450 2024-03-06T07:37:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.260+0000 7f90ef200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f00d29e0 con 0x7f90f00700c0 2024-03-06T07:37:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.260+0000 7f90ef200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00cd3a0 con 0x7f90f00d04b0 2024-03-06T07:37:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.261+0000 7f90ede00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f90f015f450 0x7f90f00cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50474/0 (socket says 172.21.15.32:50474) 2024-03-06T07:37:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.261+0000 7f90ede00640 1 -- 172.21.15.32:0/1292599053 learned_addr learned my addr 172.21.15.32:0/1292599053 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.261+0000 7f90ee800640 1 --2- 172.21.15.32:0/1292599053 >> v2:172.21.15.39:3300/0 conn(0x7f90f00700c0 0x7f90f00cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.261+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468990939 0 0) 0x7f90f00cd0c0 con 0x7f90f015f450 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.261+0000 7f90ed400640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 0x7f90f0070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.261+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d4003520 con 0x7f90f015f450 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620835164 0 0) 0x7f90d4003520 con 0x7f90f015f450 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 msgr2=0x7f90f0070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 0x7f90f0070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 >> v2:172.21.15.39:3300/0 conn(0x7f90f00700c0 msgr2=0x7f90f00cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 --2- 172.21.15.32:0/1292599053 >> v2:172.21.15.39:3300/0 conn(0x7f90f00700c0 0x7f90f00cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00cd680 con 0x7f90f015f450 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f90e00019f0 con 0x7f90f015f450 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 >> v1:172.21.15.32:6789/0 conn(0x7f90f015f450 legacy=0x7f90f00cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 0x7f90d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90d4003f30 0x7f90d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00d29e0 con 0x7f90d4004d40 2024-03-06T07:37:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00cd3a0 con 0x7f90d4003f30 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d400b9c0 con 0x7f90f00d04b0 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.262+0000 7f90ee800640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ede00640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 0x7f90d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ed400640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90d4003f30 0x7f90d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ee800640 1 -- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 msgr2=0x7f90d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ee800640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00d04b0 0x7f90d40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ee800640 1 -- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90d4003f30 msgr2=0x7f90d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ee800640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90d4003f30 0x7f90d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ee800640 1 -- 172.21.15.32:0/1292599053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4003520 con 0x7f90d4004d40 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.263+0000 7f90ed400640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90d4003f30 0x7f90d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.264+0000 7f90ee800640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90d40051c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009ac0 tx=0x7f90d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a68939fa5c7f62e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.264+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8039070 con 0x7f90d4004d40 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.264+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e802c6d0 con 0x7f90d4004d40 2024-03-06T07:37:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.264+0000 7f90eca00640 1 -- 172.21.15.32:0/1292599053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8009690 con 0x7f90d4004d40 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ef200640 1 -- 172.21.15.32:0/1292599053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 msgr2=0x7f90d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ef200640 1 --2- 172.21.15.32:0/1292599053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90d40051c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009ac0 tx=0x7f90d4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ee800640 1 -- 172.21.15.32:0/1292599053 reap_dead start 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ef200640 1 -- 172.21.15.32:0/1292599053 shutdown_connections 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ef200640 1 -- 172.21.15.32:0/1292599053 >> 172.21.15.32:0/1292599053 conn(0x7f90f00d1ae0 msgr2=0x7f90f00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ef200640 1 -- 172.21.15.32:0/1292599053 shutdown_connections 2024-03-06T07:37:53.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.265+0000 7f90ef200640 1 -- 172.21.15.32:0/1292599053 wait complete. 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.267+0000 7f90ef200640 1 Processor -- start 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.267+0000 7f90ef200640 1 -- start start 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ef200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90f00c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ef200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00af440 0x7f90f00c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ef200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0151b30 0x7f90f015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ef200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f015f830 con 0x7f90d4004d40 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ef200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00ae8b0 con 0x7f90f0151b30 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ef200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f015f9b0 con 0x7f90f00af440 2024-03-06T07:37:53.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90f00c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90f00c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53542/0 (socket says 172.21.15.32:53542) 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.268+0000 7f90ede00640 1 -- 172.21.15.32:0/858669599 learned_addr learned my addr 172.21.15.32:0/858669599 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ede00640 1 -- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00af440 msgr2=0x7f90f00c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ed400640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00af440 0x7f90f00c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ee800640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0151b30 0x7f90f015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ede00640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00af440 0x7f90f00c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ede00640 1 -- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0151b30 msgr2=0x7f90f015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ede00640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0151b30 0x7f90f015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ede00640 1 -- 172.21.15.32:0/858669599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00af8a0 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ee800640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0151b30 0x7f90f015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ede00640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90f00c6ab0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f90e0001b00 tx=0x7f90e000b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0019070 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e0004d60 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e00072f0 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f00c9ad0 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.269+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f00c7420 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.271+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f004f330 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.271+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90e0007490 con 0x7f90d4004d40 2024-03-06T07:37:53.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.272+0000 7f90e6000640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f90cc06b4a0 0x7f90cc06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.272+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90e008a570 con 0x7f90d4004d40 2024-03-06T07:37:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.273+0000 7f90ed400640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f90cc06b4a0 0x7f90cc06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:53.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.273+0000 7f90ed400640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f90cc06b4a0 0x7f90cc06d960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f90f015b8e0 tx=0x7f90d8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:53.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.280+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90e0059310 con 0x7f90d4004d40 2024-03-06T07:37:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:53.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.664+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f00b7970 con 0x7f90d4004d40 2024-03-06T07:37:53.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.664+0000 7f90e6000640 1 -- 172.21.15.32:0/858669599 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f90e0058c10 con 0x7f90d4004d40 2024-03-06T07:37:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:53.666 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:53.666 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f90cc06b4a0 msgr2=0x7f90cc06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f90cc06b4a0 0x7f90cc06d960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f90f015b8e0 tx=0x7f90d8008040 comp rx=0 tx=0).stop 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 msgr2=0x7f90f00c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90f00c6ab0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f90e0001b00 tx=0x7f90e000b560 comp rx=0 tx=0).stop 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 shutdown_connections 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f90cc06b4a0 0x7f90cc06d960 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0151b30 0x7f90f015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90f00af440 0x7f90f00c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 --2- 172.21.15.32:0/858669599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d4004d40 0x7f90f00c6ab0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 >> 172.21.15.32:0/858669599 conn(0x7f90f00d1ae0 msgr2=0x7f90f015cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 shutdown_connections 2024-03-06T07:37:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:53.667+0000 7f90ef200640 1 -- 172.21.15.32:0/858669599 wait complete. 2024-03-06T07:37:53.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:53.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:54.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:55.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.114+0000 7f1924200640 1 Processor -- start 2024-03-06T07:37:55.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.114+0000 7f1924200640 1 -- start start 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.116+0000 7f1924200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.116+0000 7f1924200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f191c0cf5c0 0x7f191c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1924200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c0b45b0 con 0x7f191c06cb10 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1924200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191c0ce160 con 0x7f191c0cf5c0 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1924200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c0b4860 con 0x7f191c0b3c90 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1920e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f191c0cf5c0 0x7f191c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1922200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f191c06cb10 0x7f191c06cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50490/0 (socket says 172.21.15.32:50490) 2024-03-06T07:37:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1922200640 1 -- 172.21.15.32:0/288610692 learned_addr learned my addr 172.21.15.32:0/288610692 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1920e00640 1 -- 172.21.15.32:0/288610692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 msgr2=0x7f191c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1920e00640 1 --2- 172.21.15.32:0/288610692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1920e00640 1 -- 172.21.15.32:0/288610692 >> v1:172.21.15.32:6789/0 conn(0x7f191c06cb10 legacy=0x7f191c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f1920e00640 1 -- 172.21.15.32:0/288610692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0b4b40 con 0x7f191c0cf5c0 2024-03-06T07:37:55.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.117+0000 7f191be00640 1 -- 172.21.15.32:0/288610692 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082641788 0 0) 0x7f191c0b45b0 con 0x7f191c06cb10 2024-03-06T07:37:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.119+0000 7f1920e00640 1 --2- 172.21.15.32:0/288610692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0b4070 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f19080097a0 tx=0x7f190802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd53b3038ecf9a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.119+0000 7f191be00640 1 -- 172.21.15.32:0/288610692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908034070 con 0x7f191c0cf5c0 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.119+0000 7f191be00640 1 -- 172.21.15.32:0/288610692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1908030610 con 0x7f191c0cf5c0 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.119+0000 7f191be00640 1 -- 172.21.15.32:0/288610692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19080309e0 con 0x7f191c0cf5c0 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.119+0000 7f1924200640 1 -- 172.21.15.32:0/288610692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 msgr2=0x7f191c0b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.119+0000 7f1924200640 1 --2- 172.21.15.32:0/288610692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0b4070 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f19080097a0 tx=0x7f190802b960 comp rx=0 tx=0).stop 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.120+0000 7f1924200640 1 -- 172.21.15.32:0/288610692 shutdown_connections 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.120+0000 7f1924200640 1 --2- 172.21.15.32:0/288610692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0b4070 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.120+0000 7f1924200640 1 --2- 172.21.15.32:0/288610692 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.120+0000 7f1924200640 1 -- 172.21.15.32:0/288610692 >> 172.21.15.32:0/288610692 conn(0x7f191c06afb0 msgr2=0x7f191c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:55.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.120+0000 7f1924200640 1 -- 172.21.15.32:0/288610692 shutdown_connections 2024-03-06T07:37:55.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.120+0000 7f1924200640 1 -- 172.21.15.32:0/288610692 wait complete. 2024-03-06T07:37:55.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.121+0000 7f1924200640 1 Processor -- start 2024-03-06T07:37:55.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.122+0000 7f1924200640 1 -- start start 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.122+0000 7f1924200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f191c06cb10 0x7f191c154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1924200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1924200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1924200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c15b340 con 0x7f191c0cf5c0 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1924200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c0b5100 con 0x7f191c06cb10 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1924200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c06d690 con 0x7f191c0b3c90 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53556/0 (socket says 172.21.15.32:53556) 2024-03-06T07:37:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 -- 172.21.15.32:0/2867070605 learned_addr learned my addr 172.21.15.32:0/2867070605 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1921800640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f191c06cb10 0x7f191c154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1920e00640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 -- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 msgr2=0x7f191c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 -- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f191c06cb10 msgr2=0x7f191c154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f191c06cb10 0x7f191c154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.123+0000 7f1922200640 1 -- 172.21.15.32:0/2867070605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c151c60 con 0x7f191c0cf5c0 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.124+0000 7f1922200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0ce4b0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f191400c7a0 tx=0x7f191400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.124+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1914018070 con 0x7f191c0cf5c0 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.124+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191400a640 con 0x7f191c0cf5c0 2024-03-06T07:37:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.124+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c0cbb20 con 0x7f191c0cf5c0 2024-03-06T07:37:55.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.124+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191400aab0 con 0x7f191c0cf5c0 2024-03-06T07:37:55.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.124+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c0cc0e0 con 0x7f191c0cf5c0 2024-03-06T07:37:55.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.126+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c069510 con 0x7f191c0cf5c0 2024-03-06T07:37:55.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.128+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f191401b070 con 0x7f191c0cf5c0 2024-03-06T07:37:55.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.129+0000 7f1919600640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18fc06b520 0x7f18fc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.129+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1914089990 con 0x7f191c0cf5c0 2024-03-06T07:37:55.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.129+0000 7f1921800640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18fc06b520 0x7f18fc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.130+0000 7f1921800640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18fc06b520 0x7f18fc06d9e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1910009890 tx=0x7f1910008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.133+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19140587b0 con 0x7f191c0cf5c0 2024-03-06T07:37:55.353 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:55.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.374+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f191c04f3a0 con 0x7f191c0cf5c0 2024-03-06T07:37:55.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.376+0000 7f1919600640 1 -- 172.21.15.32:0/2867070605 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3412 (secure 0 0 0) 0x7f19140580b0 con 0x7f191c0cf5c0 2024-03-06T07:37:55.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:55.378 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:27.037791+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:55.378 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-03-06T07:37:55.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.378+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18fc06b520 msgr2=0x7f18fc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.378+0000 7f1924200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18fc06b520 0x7f18fc06d9e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1910009890 tx=0x7f1910008040 comp rx=0 tx=0).stop 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.378+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 msgr2=0x7f191c0ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.378+0000 7f1924200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0ce4b0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f191400c7a0 tx=0x7f191400cc60 comp rx=0 tx=0).stop 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 shutdown_connections 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f18fc06b520 0x7f18fc06d9e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191c0cf5c0 0x7f191c0ce4b0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f191c0b3c90 0x7f191c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 --2- 172.21.15.32:0/2867070605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f191c06cb10 0x7f191c154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 >> 172.21.15.32:0/2867070605 conn(0x7f191c06afb0 msgr2=0x7f191c06cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 shutdown_connections 2024-03-06T07:37:55.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:55.379+0000 7f1924200640 1 -- 172.21.15.32:0/2867070605 wait complete. 2024-03-06T07:37:55.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:27.037791+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:55.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:56.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.823+0000 7f33a0c00640 1 Processor -- start 2024-03-06T07:37:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.823+0000 7f33a0c00640 1 -- start start 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f33a0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d4a10 0x7f339c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f33a0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f339c0d0c80 0x7f339c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f33a0c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c0b3710 con 0x7f339c15e0b0 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f33a0c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0b09e0 con 0x7f339c0d0c80 2024-03-06T07:37:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f33a0c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0cfa20 con 0x7f339c0d4a10 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f339aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d4a10 0x7f339c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f339be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f339c0d0c80 0x7f339c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f339aa00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d4a10 0x7f339c0d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:57696/0 (socket says 172.21.15.32:57696) 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.825+0000 7f339aa00640 1 -- 172.21.15.32:0/1716886213 learned_addr learned my addr 172.21.15.32:0/1716886213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.826+0000 7f339be00640 1 -- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d4a10 msgr2=0x7f339c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:56.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.826+0000 7f339be00640 1 --2- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d4a10 0x7f339c0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.826+0000 7f339be00640 1 -- 172.21.15.32:0/1716886213 >> v1:172.21.15.39:6789/0 conn(0x7f339c15e0b0 legacy=0x7f339c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.826+0000 7f339be00640 1 -- 172.21.15.32:0/1716886213 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0b39f0 con 0x7f339c0d0c80 2024-03-06T07:37:56.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.828+0000 7f339be00640 1 --2- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d0c80 0x7f339c0b44d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f33940097a0 tx=0x7f339402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68985ff435274c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f339a000640 1 -- 172.21.15.32:0/1716886213 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394035020 con 0x7f339c0d0c80 2024-03-06T07:37:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f339a000640 1 -- 172.21.15.32:0/1716886213 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3394031810 con 0x7f339c0d0c80 2024-03-06T07:37:56.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f339a000640 1 -- 172.21.15.32:0/1716886213 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394031be0 con 0x7f339c0d0c80 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f33a0c00640 1 -- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d0c80 msgr2=0x7f339c0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d0c80 0x7f339c0b44d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f33940097a0 tx=0x7f339402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f33a0c00640 1 -- 172.21.15.32:0/1716886213 shutdown_connections 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d0c80 0x7f339c0b44d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1716886213 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d4a10 0x7f339c0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.833+0000 7f33a0c00640 1 -- 172.21.15.32:0/1716886213 >> 172.21.15.32:0/1716886213 conn(0x7f339c0d3d30 msgr2=0x7f339c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.834+0000 7f33a0c00640 1 -- 172.21.15.32:0/1716886213 shutdown_connections 2024-03-06T07:37:56.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.834+0000 7f33a0c00640 1 -- 172.21.15.32:0/1716886213 wait complete. 2024-03-06T07:37:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.835+0000 7f33a0c00640 1 Processor -- start 2024-03-06T07:37:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.836+0000 7f33a0c00640 1 -- start start 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f33a0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d0c80 0x7f339c15d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f33a0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d4a10 0x7f339c0b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f33a0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 0x7f339c153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f33a0c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c15e490 con 0x7f339c15e0b0 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f33a0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0b3fb0 con 0x7f339c0d4a10 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f33a0c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0cfc60 con 0x7f339c0d0c80 2024-03-06T07:37:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 0x7f339c153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 0x7f339c153b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53570/0 (socket says 172.21.15.32:53570) 2024-03-06T07:37:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339be00640 1 -- 172.21.15.32:0/1363361783 learned_addr learned my addr 172.21.15.32:0/1363361783 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339b400640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d0c80 0x7f339c15d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339be00640 1 -- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d0c80 msgr2=0x7f339c15d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339be00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d0c80 0x7f339c15d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339be00640 1 -- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d4a10 msgr2=0x7f339c0b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.837+0000 7f339aa00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d4a10 0x7f339c0b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f339be00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d4a10 0x7f339c0b6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f339be00640 1 -- 172.21.15.32:0/1363361783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0b6e90 con 0x7f339c15e0b0 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f339be00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 0x7f339c153b50 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f33940098c0 tx=0x7f33940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394041070 con 0x7f339c15e0b0 2024-03-06T07:37:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339403ac80 con 0x7f339c15e0b0 2024-03-06T07:37:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394035040 con 0x7f339c15e0b0 2024-03-06T07:37:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.838+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c154090 con 0x7f339c15e0b0 2024-03-06T07:37:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.839+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c0b7410 con 0x7f339c15e0b0 2024-03-06T07:37:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.840+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c04f330 con 0x7f339c15e0b0 2024-03-06T07:37:56.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.841+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3394038020 con 0x7f339c15e0b0 2024-03-06T07:37:56.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.842+0000 7f3393400640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f337c06b4d0 0x7f337c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:56.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.842+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f33940abec0 con 0x7f339c15e0b0 2024-03-06T07:37:56.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.842+0000 7f339b400640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f337c06b4d0 0x7f337c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:56.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.843+0000 7f339b400640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f337c06b4d0 0x7f337c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f338c001ca0 tx=0x7f338c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:56.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:56.847+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f339407ac60 con 0x7f339c15e0b0 2024-03-06T07:37:57.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.086+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f339c0cab70 con 0x7f339c15e0b0 2024-03-06T07:37:57.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.087+0000 7f3393400640 1 -- 172.21.15.32:0/1363361783 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f339407a560 con 0x7f339c15e0b0 2024-03-06T07:37:57.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:57.088 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:57.089 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:37:57.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.089+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f337c06b4d0 msgr2=0x7f337c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.089+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f337c06b4d0 0x7f337c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f338c001ca0 tx=0x7f338c009040 comp rx=0 tx=0).stop 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 msgr2=0x7f339c153b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 0x7f339c153b50 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f33940098c0 tx=0x7f33940046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 shutdown_connections 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f337c06b4d0 0x7f337c06d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f339c15e0b0 0x7f339c153b50 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f339c0d4a10 0x7f339c0b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 --2- 172.21.15.32:0/1363361783 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f339c0d0c80 0x7f339c15d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 >> 172.21.15.32:0/1363361783 conn(0x7f339c0d3d30 msgr2=0x7f339c0b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 shutdown_connections 2024-03-06T07:37:57.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:57.090+0000 7f33a0c00640 1 -- 172.21.15.32:0/1363361783 wait complete. 2024-03-06T07:37:57.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:57.311 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:58.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:58.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.544+0000 7f8d96000640 1 Processor -- start 2024-03-06T07:37:58.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.544+0000 7f8d96000640 1 -- start start 2024-03-06T07:37:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.545+0000 7f8d96000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 0x7f8d900cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.545+0000 7f8d96000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d900b4860 con 0x7f8d900b3c90 2024-03-06T07:37:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.546+0000 7f8d96000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d900b4b40 con 0x7f8d90071050 2024-03-06T07:37:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.546+0000 7f8d96000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900ce160 con 0x7f8d900cf5c0 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.546+0000 7f8d8f400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8d900b3c90 0x7f8d900b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42666/0 (socket says 172.21.15.32:42666) 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.546+0000 7f8d8f400640 1 -- 172.21.15.32:0/169109716 learned_addr learned my addr 172.21.15.32:0/169109716 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.546+0000 7f8d8ea00640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 0x7f8d900cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.546+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022061267 0 0) 0x7f8d900b4860 con 0x7f8d900b3c90 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d74003520 con 0x7f8d900b3c90 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036763069 0 0) 0x7f8d74003520 con 0x7f8d900b3c90 2024-03-06T07:37:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 msgr2=0x7f8d900cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 0x7f8d900cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 >> v1:172.21.15.39:6789/0 conn(0x7f8d90071050 legacy=0x7f8d900714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900b4e20 con 0x7f8d900b3c90 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f8d80001670 con 0x7f8d900b3c90 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 >> v1:172.21.15.32:6789/0 conn(0x7f8d900b3c90 legacy=0x7f8d900b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 0x7f8d74003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d74003fd0 0x7f8d74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d74004d40 0x7f8d740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900ce160 con 0x7f8d74004d40 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7400bbb0 con 0x7f8d74003fd0 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7400bdb0 con 0x7f8d900cf5c0 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.547+0000 7f8d8fe00640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d74004d40 0x7f8d740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8f400640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 0x7f8d74003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8ea00640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d74003fd0 0x7f8d74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8fe00640 1 -- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 msgr2=0x7f8d74003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8fe00640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d900cf5c0 0x7f8d74003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8fe00640 1 -- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d74003fd0 msgr2=0x7f8d74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8fe00640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d74003fd0 0x7f8d74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.548+0000 7f8d8fe00640 1 -- 172.21.15.32:0/169109716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d74003520 con 0x7f8d74004d40 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.549+0000 7f8d8fe00640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d74004d40 0x7f8d740051c0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f8d88001330 tx=0x7f8d74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94669a9166ed116 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.549+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88001370 con 0x7f8d74004d40 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.549+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8802f040 con 0x7f8d74004d40 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.549+0000 7f8d8e000640 1 -- 172.21.15.32:0/169109716 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d880035f0 con 0x7f8d74004d40 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d96000640 1 -- 172.21.15.32:0/169109716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d74004d40 msgr2=0x7f8d740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d96000640 1 --2- 172.21.15.32:0/169109716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d74004d40 0x7f8d740051c0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f8d88001330 tx=0x7f8d74005830 comp rx=0 tx=0).stop 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d8fe00640 1 -- 172.21.15.32:0/169109716 reap_dead start 2024-03-06T07:37:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d96000640 1 -- 172.21.15.32:0/169109716 shutdown_connections 2024-03-06T07:37:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d96000640 1 -- 172.21.15.32:0/169109716 >> 172.21.15.32:0/169109716 conn(0x7f8d9006afb0 msgr2=0x7f8d9006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d96000640 1 -- 172.21.15.32:0/169109716 shutdown_connections 2024-03-06T07:37:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.550+0000 7f8d96000640 1 -- 172.21.15.32:0/169109716 wait complete. 2024-03-06T07:37:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.552+0000 7f8d96000640 1 Processor -- start 2024-03-06T07:37:58.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.552+0000 7f8d96000640 1 -- start start 2024-03-06T07:37:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d96000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d74004d40 0x7f8d90154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d96000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 0x7f8d90155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d96000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d9009d5d0 0x7f8d901558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d96000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900cdc20 con 0x7f8d9009d150 2024-03-06T07:37:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d96000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9015b340 con 0x7f8d9009d5d0 2024-03-06T07:37:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d96000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900cdda0 con 0x7f8d74004d40 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d8ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 0x7f8d90155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d8ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 0x7f8d90155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59790/0 (socket says 172.21.15.32:59790) 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.553+0000 7f8d8ea00640 1 -- 172.21.15.32:0/1334921756 learned_addr learned my addr 172.21.15.32:0/1334921756 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8f400640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d74004d40 0x7f8d90154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8fe00640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d9009d5d0 0x7f8d901558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8ea00640 1 -- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d74004d40 msgr2=0x7f8d90154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8ea00640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d74004d40 0x7f8d90154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8ea00640 1 -- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d9009d5d0 msgr2=0x7f8d901558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8ea00640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d9009d5d0 0x7f8d901558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8ea00640 1 -- 172.21.15.32:0/1334921756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9009c0b0 con 0x7f8d9009d150 2024-03-06T07:37:58.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d8ea00640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 0x7f8d90155370 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8d78007f30 tx=0x7f8d78011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7801a020 con 0x7f8d9009d150 2024-03-06T07:37:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d78016470 con 0x7f8d9009d150 2024-03-06T07:37:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.554+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78016860 con 0x7f8d9009d150 2024-03-06T07:37:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.555+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9009c390 con 0x7f8d9009d150 2024-03-06T07:37:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.555+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9009c950 con 0x7f8d9009d150 2024-03-06T07:37:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.556+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d90069510 con 0x7f8d9009d150 2024-03-06T07:37:58.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.557+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d78005d10 con 0x7f8d9009d150 2024-03-06T07:37:58.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.558+0000 7f8d87400640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:58.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.558+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8d7808b240 con 0x7f8d9009d150 2024-03-06T07:37:58.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.558+0000 7f8d8f400640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:58.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.559+0000 7f8d8f400640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8d80000fb0 tx=0x7f8d80009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:58.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.567+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d78024030 con 0x7f8d9009d150 2024-03-06T07:37:58.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.848+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d9004f3a0 con 0x7f8d9009d150 2024-03-06T07:37:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.849+0000 7f8d87400640 1 -- 172.21.15.32:0/1334921756 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f8d78059fe0 con 0x7f8d9009d150 2024-03-06T07:37:58.851 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:37:58.851 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:37:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.851+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d6c06b4d0 msgr2=0x7f8d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.851+0000 7f8d96000640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8d80000fb0 tx=0x7f8d80009040 comp rx=0 tx=0).stop 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.851+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 msgr2=0x7f8d90155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.851+0000 7f8d96000640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 0x7f8d90155370 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8d78007f30 tx=0x7f8d78011660 comp rx=0 tx=0).stop 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 shutdown_connections 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d9009d5d0 0x7f8d901558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9009d150 0x7f8d90155370 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 --2- 172.21.15.32:0/1334921756 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d74004d40 0x7f8d90154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 >> 172.21.15.32:0/1334921756 conn(0x7f8d9006afb0 msgr2=0x7f8d900b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:58.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 shutdown_connections 2024-03-06T07:37:58.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:37:58.852+0000 7f8d96000640 1 -- 172.21.15.32:0/1334921756 wait complete. 2024-03-06T07:37:59.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:37:59.075 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:37:59.357 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:00.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.304+0000 7effc4a00640 1 Processor -- start 2024-03-06T07:38:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.304+0000 7effc4a00640 1 -- start start 2024-03-06T07:38:00.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.305+0000 7effc4a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7effc00d0e50 0x7effc0168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effc4a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00b3710 con 0x7effc015e0b0 2024-03-06T07:38:00.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effc4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00b39f0 con 0x7effc00d4a10 2024-03-06T07:38:00.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effc4a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc00b09e0 con 0x7effc00d0e50 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effbe800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7effc015e0b0 0x7effc00cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42680/0 (socket says 172.21.15.32:42680) 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effbe800640 1 -- 172.21.15.32:0/1252061924 learned_addr learned my addr 172.21.15.32:0/1252061924 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effbf200640 1 --2- 172.21.15.32:0/1252061924 >> v2:172.21.15.39:3300/0 conn(0x7effc00d0e50 0x7effc0168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.306+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814677323 0 0) 0x7effc00b3710 con 0x7effc015e0b0 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa4003520 con 0x7effc015e0b0 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1820944953 0 0) 0x7effa4003520 con 0x7effc015e0b0 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 >> v2:172.21.15.39:3300/0 conn(0x7effc00d0e50 msgr2=0x7effc0168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 --2- 172.21.15.32:0/1252061924 >> v2:172.21.15.39:3300/0 conn(0x7effc00d0e50 0x7effc0168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 >> v1:172.21.15.39:6789/0 conn(0x7effc00d4a10 legacy=0x7effc00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00b3cd0 con 0x7effc015e0b0 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7effb0001670 con 0x7effc015e0b0 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 >> v1:172.21.15.32:6789/0 conn(0x7effc015e0b0 legacy=0x7effc00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effa40051c0 0x7effa400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c090 con 0x7effa4003a00 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c240 con 0x7effa40051c0 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c440 con 0x7effa4004800 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.307+0000 7effbe800640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effb7e00640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbf200640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effa40051c0 0x7effa400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbe800640 1 -- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4004800 msgr2=0x7effa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbe800640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4004800 0x7effa4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbe800640 1 -- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effa40051c0 msgr2=0x7effa400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbe800640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effa40051c0 0x7effa400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbe800640 1 -- 172.21.15.32:0/1252061924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4003520 con 0x7effa4003a00 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.308+0000 7effbf200640 1 -- 172.21.15.32:0/1252061924 reap_dead start 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.310+0000 7effbe800640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa4003a00 0x7effa4003e20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7effb000c110 tx=0x7effa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cce8e78469ea099 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.310+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0007c40 con 0x7effa4003a00 2024-03-06T07:38:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.310+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0030070 con 0x7effa4003a00 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.310+0000 7effbde00640 1 -- 172.21.15.32:0/1252061924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0005d70 con 0x7effa4003a00 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 -- 172.21.15.32:0/1252061924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa4003a00 msgr2=0x7effa4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa4003a00 0x7effa4003e20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7effb000c110 tx=0x7effa4005830 comp rx=0 tx=0).stop 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 -- 172.21.15.32:0/1252061924 shutdown_connections 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 --2- 172.21.15.32:0/1252061924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa4003a00 0x7effa4003e20 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 -- 172.21.15.32:0/1252061924 >> 172.21.15.32:0/1252061924 conn(0x7effc00d3d30 msgr2=0x7effc00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 -- 172.21.15.32:0/1252061924 shutdown_connections 2024-03-06T07:38:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.311+0000 7effc4a00640 1 -- 172.21.15.32:0/1252061924 wait complete. 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.313+0000 7effc4a00640 1 Processor -- start 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.313+0000 7effc4a00640 1 -- start start 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effc4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4003a00 0x7effc015b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effc4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 0x7effc00b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effc4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc0160440 0x7effc01608c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effc4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00cfc60 con 0x7effc00b72d0 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effc4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc015e490 con 0x7effc0160440 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effc4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00d0740 con 0x7effa4003a00 2024-03-06T07:38:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 0x7effc00b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 0x7effc00b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59804/0 (socket says 172.21.15.32:59804) 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effb7e00640 1 -- 172.21.15.32:0/2631391687 learned_addr learned my addr 172.21.15.32:0/2631391687 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.314+0000 7effbe800640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4003a00 0x7effc015b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effbf200640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc0160440 0x7effc01608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb7e00640 1 -- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4003a00 msgr2=0x7effc015b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb7e00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4003a00 0x7effc015b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb7e00640 1 -- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc0160440 msgr2=0x7effc01608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb7e00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc0160440 0x7effc01608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb7e00640 1 -- 172.21.15.32:0/2631391687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc015b650 con 0x7effc00b72d0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb7e00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 0x7effc00b7750 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7effa800e580 tx=0x7effa800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.315+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa80052e0 con 0x7effc00b72d0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.316+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa8018070 con 0x7effc00b72d0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.316+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8003810 con 0x7effc00b72d0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.316+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc015f3a0 con 0x7effc00b72d0 2024-03-06T07:38:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.316+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc015f960 con 0x7effc00b72d0 2024-03-06T07:38:00.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.317+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc004f330 con 0x7effc00b72d0 2024-03-06T07:38:00.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.318+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effa80039b0 con 0x7effc00b72d0 2024-03-06T07:38:00.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.319+0000 7effb6a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff9c06b4d0 0x7eff9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:00.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.319+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7effa8089ff0 con 0x7effc00b72d0 2024-03-06T07:38:00.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.319+0000 7effbe800640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff9c06b4d0 0x7eff9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:00.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.320+0000 7effbe800640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff9c06b4d0 0x7eff9c06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7effb0034000 tx=0x7effb0009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:00.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.323+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effa8058e40 con 0x7effc00b72d0 2024-03-06T07:38:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.623+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effc0074920 con 0x7effc00b72d0 2024-03-06T07:38:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.624+0000 7effb6a00640 1 -- 172.21.15.32:0/2631391687 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7effa8058740 con 0x7effc00b72d0 2024-03-06T07:38:00.627 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:00.627 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:00.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff9c06b4d0 msgr2=0x7eff9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff9c06b4d0 0x7eff9c06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7effb0034000 tx=0x7effb0009030 comp rx=0 tx=0).stop 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 msgr2=0x7effc00b7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 0x7effc00b7750 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7effa800e580 tx=0x7effa800ea40 comp rx=0 tx=0).stop 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 shutdown_connections 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7eff9c06b4d0 0x7eff9c06d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7effc0160440 0x7effc01608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc00b72d0 0x7effc00b7750 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 --2- 172.21.15.32:0/2631391687 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7effa4003a00 0x7effc015b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 >> 172.21.15.32:0/2631391687 conn(0x7effc00d3d30 msgr2=0x7effc006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 shutdown_connections 2024-03-06T07:38:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:00.627+0000 7effc4a00640 1 -- 172.21.15.32:0/2631391687 wait complete. 2024-03-06T07:38:00.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:00.853 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:01.360 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:01.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.102+0000 7efcc5e00640 1 Processor -- start 2024-03-06T07:38:02.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.102+0000 7efcc5e00640 1 -- start start 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.104+0000 7efcc5e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcc0145300 0x7efcc00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.104+0000 7efcc5e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc00714d0 con 0x7efcc0070f10 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.104+0000 7efcc5e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc00717b0 con 0x7efcc00cd060 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.104+0000 7efcc5e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc0071a90 con 0x7efcc0145300 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.105+0000 7efcc4a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcc0145300 0x7efcc00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.105+0000 7efcc4a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcc0145300 0x7efcc00b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59814/0 (socket says 172.21.15.32:59814) 2024-03-06T07:38:02.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.105+0000 7efcc4a00640 1 -- 172.21.15.32:0/2092780714 learned_addr learned my addr 172.21.15.32:0/2092780714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:02.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.105+0000 7efcc4a00640 1 -- 172.21.15.32:0/2092780714 >> v1:172.21.15.39:6789/0 conn(0x7efcc00cd060 legacy=0x7efcc015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:02.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.105+0000 7efcc4a00640 1 -- 172.21.15.32:0/2092780714 >> v1:172.21.15.32:6789/0 conn(0x7efcc0070f10 legacy=0x7efcc00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.105+0000 7efcc4a00640 1 -- 172.21.15.32:0/2092780714 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0071d70 con 0x7efcc0145300 2024-03-06T07:38:02.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.106+0000 7efcc4a00640 1 --2- 172.21.15.32:0/2092780714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0145300 0x7efcc00b49a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009870 tx=0x7efcb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345c255a49c0d4ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcbf400640 1 -- 172.21.15.32:0/2092780714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4035020 con 0x7efcc0145300 2024-03-06T07:38:02.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcbf400640 1 -- 172.21.15.32:0/2092780714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb4031870 con 0x7efcc0145300 2024-03-06T07:38:02.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcbf400640 1 -- 172.21.15.32:0/2092780714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4031c40 con 0x7efcc0145300 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcc5e00640 1 -- 172.21.15.32:0/2092780714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0145300 msgr2=0x7efcc00b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcc5e00640 1 --2- 172.21.15.32:0/2092780714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0145300 0x7efcc00b49a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009870 tx=0x7efcb402b910 comp rx=0 tx=0).stop 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcc5e00640 1 -- 172.21.15.32:0/2092780714 shutdown_connections 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcc5e00640 1 --2- 172.21.15.32:0/2092780714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc0145300 0x7efcc00b49a0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcc5e00640 1 -- 172.21.15.32:0/2092780714 >> 172.21.15.32:0/2092780714 conn(0x7efcc006b3d0 msgr2=0x7efcc006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.107+0000 7efcc5e00640 1 -- 172.21.15.32:0/2092780714 shutdown_connections 2024-03-06T07:38:02.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.108+0000 7efcc5e00640 1 -- 172.21.15.32:0/2092780714 wait complete. 2024-03-06T07:38:02.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.109+0000 7efcc5e00640 1 Processor -- start 2024-03-06T07:38:02.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.109+0000 7efcc5e00640 1 -- start start 2024-03-06T07:38:02.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcc5e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcc0070f10 0x7efcc01320a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcc5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 0x7efcc013bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcc5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc0145300 0x7efcc0139f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcc5e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc015cc80 con 0x7efcc00cd060 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcc5e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc01456e0 con 0x7efcc0145300 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcc5e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc015ce00 con 0x7efcc0070f10 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.110+0000 7efcbfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 0x7efcc013bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 0x7efcc013bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59816/0 (socket says 172.21.15.32:59816) 2024-03-06T07:38:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 -- 172.21.15.32:0/1389008332 learned_addr learned my addr 172.21.15.32:0/1389008332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcc4a00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcc0070f10 0x7efcc01320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcc5400640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc0145300 0x7efcc0139f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 -- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcc0070f10 msgr2=0x7efcc01320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcc0070f10 0x7efcc01320a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 -- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc0145300 msgr2=0x7efcc0139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc0145300 0x7efcc0139f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 -- 172.21.15.32:0/1389008332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0135d30 con 0x7efcc00cd060 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.111+0000 7efcbfe00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 0x7efcc013bad0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c430 tx=0x7efcac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.112+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0052d0 con 0x7efcc00cd060 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.112+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac005490 con 0x7efcc00cd060 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.112+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc0136010 con 0x7efcc00cd060 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.112+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac007260 con 0x7efcc00cd060 2024-03-06T07:38:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.112+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc01365d0 con 0x7efcc00cd060 2024-03-06T07:38:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.113+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc004f330 con 0x7efcc00cd060 2024-03-06T07:38:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.114+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcac007400 con 0x7efcc00cd060 2024-03-06T07:38:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.115+0000 7efcbcc00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efca006b4d0 0x7efca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.116+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcac0897d0 con 0x7efcc00cd060 2024-03-06T07:38:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.116+0000 7efcc4a00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efca006b4d0 0x7efca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.116+0000 7efcc4a00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efca006b4d0 0x7efca006d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7efcb4036000 tx=0x7efcb403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.120+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcac058570 con 0x7efcc00cd060 2024-03-06T07:38:02.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.362+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcc0139d00 con 0x7efcc00cd060 2024-03-06T07:38:02.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.363+0000 7efcbcc00640 1 -- 172.21.15.32:0/1389008332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7efcac057e70 con 0x7efcc00cd060 2024-03-06T07:38:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:02.366 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:02.366 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efca006b4d0 msgr2=0x7efca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efca006b4d0 0x7efca006d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7efcb4036000 tx=0x7efcb403c040 comp rx=0 tx=0).stop 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 msgr2=0x7efcc013bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 0x7efcc013bad0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c430 tx=0x7efcac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 shutdown_connections 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efca006b4d0 0x7efca006d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc0145300 0x7efcc0139f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcc00cd060 0x7efcc013bad0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 --2- 172.21.15.32:0/1389008332 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcc0070f10 0x7efcc01320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.366+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 >> 172.21.15.32:0/1389008332 conn(0x7efcc006b3d0 msgr2=0x7efcc00b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.367+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 shutdown_connections 2024-03-06T07:38:02.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:02.367+0000 7efcc5e00640 1 -- 172.21.15.32:0/1389008332 wait complete. 2024-03-06T07:38:02.588 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:02.588 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:03.362 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:03.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:03.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.811+0000 7f423fe00640 1 Processor -- start 2024-03-06T07:38:03.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.811+0000 7f423fe00640 1 -- start start 2024-03-06T07:38:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f423fe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42400700c0 0x7f4240070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f423fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42400cd0c0 con 0x7f42400d04b0 2024-03-06T07:38:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f423fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42400cd3a0 con 0x7f424015f450 2024-03-06T07:38:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f423fe00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400d29e0 con 0x7f42400700c0 2024-03-06T07:38:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f423f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42400700c0 0x7f4240070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f4236000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f42400d04b0 0x7f42400cde80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42704/0 (socket says 172.21.15.32:42704) 2024-03-06T07:38:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.813+0000 7f4236000640 1 -- 172.21.15.32:0/554902066 learned_addr learned my addr 172.21.15.32:0/554902066 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.814+0000 7f423e000640 1 -- 172.21.15.32:0/554902066 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085601539 0 0) 0x7f42400cd0c0 con 0x7f42400d04b0 2024-03-06T07:38:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.814+0000 7f423e000640 1 -- 172.21.15.32:0/554902066 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4224003520 con 0x7f42400d04b0 2024-03-06T07:38:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.814+0000 7f423f400640 1 -- 172.21.15.32:0/554902066 >> v1:172.21.15.39:6789/0 conn(0x7f424015f450 legacy=0x7f42400cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.814+0000 7f423f400640 1 -- 172.21.15.32:0/554902066 >> v1:172.21.15.32:6789/0 conn(0x7f42400d04b0 legacy=0x7f42400cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.814+0000 7f423f400640 1 -- 172.21.15.32:0/554902066 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400cd680 con 0x7f42400700c0 2024-03-06T07:38:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.815+0000 7f423f400640 1 --2- 172.21.15.32:0/554902066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f4240070990 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f4238009d90 tx=0x7f423802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd8f4a325e6c208 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423e000640 1 -- 172.21.15.32:0/554902066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238039070 con 0x7f42400700c0 2024-03-06T07:38:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423e000640 1 -- 172.21.15.32:0/554902066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423802cd00 con 0x7f42400700c0 2024-03-06T07:38:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423e000640 1 -- 172.21.15.32:0/554902066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238034640 con 0x7f42400700c0 2024-03-06T07:38:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 -- 172.21.15.32:0/554902066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 msgr2=0x7f4240070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 --2- 172.21.15.32:0/554902066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f4240070990 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f4238009d90 tx=0x7f423802bf50 comp rx=0 tx=0).stop 2024-03-06T07:38:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 -- 172.21.15.32:0/554902066 shutdown_connections 2024-03-06T07:38:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 --2- 172.21.15.32:0/554902066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f4240070990 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 -- 172.21.15.32:0/554902066 >> 172.21.15.32:0/554902066 conn(0x7f42400d1ae0 msgr2=0x7f42400d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 -- 172.21.15.32:0/554902066 shutdown_connections 2024-03-06T07:38:03.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.816+0000 7f423fe00640 1 -- 172.21.15.32:0/554902066 wait complete. 2024-03-06T07:38:03.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.819+0000 7f423fe00640 1 Processor -- start 2024-03-06T07:38:03.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.819+0000 7f423fe00640 1 -- start start 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.820+0000 7f423fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f424014f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.820+0000 7f423fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f42400d04b0 0x7f42401f51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.820+0000 7f423fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f424015f450 0x7f42401f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.820+0000 7f423fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424015f830 con 0x7f42400700c0 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.820+0000 7f423fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400ae8b0 con 0x7f424015f450 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.820+0000 7f423fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424015f9b0 con 0x7f42400d04b0 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f424014f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f424014f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59844/0 (socket says 172.21.15.32:59844) 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 -- 172.21.15.32:0/2313660959 learned_addr learned my addr 172.21.15.32:0/2313660959 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f4236000640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f42400d04b0 0x7f42401f51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423f400640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f424015f450 0x7f42401f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 -- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f42400d04b0 msgr2=0x7f42401f51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f42400d04b0 0x7f42401f51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 -- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f424015f450 msgr2=0x7f42401f5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f424015f450 0x7f42401f5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423ea00640 1 -- 172.21.15.32:0/2313660959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400c7530 con 0x7f42400700c0 2024-03-06T07:38:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.821+0000 7f423f400640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f424015f450 0x7f42401f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.822+0000 7f423ea00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f424014f5c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f423000c400 tx=0x7f423000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.822+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230017070 con 0x7f42400700c0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.822+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42300052d0 con 0x7f42400700c0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.822+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230003830 con 0x7f42400700c0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.822+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42400c7810 con 0x7f42400700c0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.822+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4240150440 con 0x7f42400700c0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.823+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240068120 con 0x7f42400700c0 2024-03-06T07:38:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.824+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f423001a070 con 0x7f42400700c0 2024-03-06T07:38:03.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.825+0000 7f4237400640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f421c06b500 0x7f421c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.825+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4230089560 con 0x7f42400700c0 2024-03-06T07:38:03.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.826+0000 7f4236000640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f421c06b500 0x7f421c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.826+0000 7f4236000640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f421c06b500 0x7f421c06d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f42400af8d0 tx=0x7f4228009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:03.830+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4240068120 con 0x7f42400700c0 2024-03-06T07:38:04.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.072+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f424004f330 con 0x7f42400700c0 2024-03-06T07:38:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.072+0000 7f4237400640 1 -- 172.21.15.32:0/2313660959 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f4230058390 con 0x7f42400700c0 2024-03-06T07:38:04.075 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:04.075 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f421c06b500 msgr2=0x7f421c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f421c06b500 0x7f421c06d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f42400af8d0 tx=0x7f4228009040 comp rx=0 tx=0).stop 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 msgr2=0x7f424014f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f424014f5c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f423000c400 tx=0x7f423000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 shutdown_connections 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f421c06b500 0x7f421c06d9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f424015f450 0x7f42401f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f42400d04b0 0x7f42401f51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 --2- 172.21.15.32:0/2313660959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400700c0 0x7f424014f5c0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 >> 172.21.15.32:0/2313660959 conn(0x7f42400d1ae0 msgr2=0x7f42400aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 shutdown_connections 2024-03-06T07:38:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:04.075+0000 7f423fe00640 1 -- 172.21.15.32:0/2313660959 wait complete. 2024-03-06T07:38:04.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:04.299 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:05.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:05.364 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:05.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.528+0000 7f1859800640 1 Processor -- start 2024-03-06T07:38:05.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.528+0000 7f1859800640 1 -- start start 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1859800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18540d2960 0x7f18540d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1859800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18540d0b00 0x7f18540b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1859800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18540d2dc0 con 0x7f185415e070 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1859800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18540d0f70 con 0x7f18540d0b00 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1859800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18540d18c0 con 0x7f18540d2960 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1858e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18540d0b00 0x7f18540b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:05.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1858e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18540d0b00 0x7f18540b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59852/0 (socket says 172.21.15.32:59852) 2024-03-06T07:38:05.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.530+0000 7f1858e00640 1 -- 172.21.15.32:0/3900380220 learned_addr learned my addr 172.21.15.32:0/3900380220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:05.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.531+0000 7f184b400640 1 --2- 172.21.15.32:0/3900380220 >> v2:172.21.15.39:3300/0 conn(0x7f18540d2960 0x7f18540d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:05.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.531+0000 7f1858e00640 1 -- 172.21.15.32:0/3900380220 >> v2:172.21.15.39:3300/0 conn(0x7f18540d2960 msgr2=0x7f18540d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:05.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.531+0000 7f1858e00640 1 --2- 172.21.15.32:0/3900380220 >> v2:172.21.15.39:3300/0 conn(0x7f18540d2960 0x7f18540d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.531+0000 7f1858e00640 1 -- 172.21.15.32:0/3900380220 >> v1:172.21.15.39:6789/0 conn(0x7f185415e070 legacy=0x7f18540cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:05.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.531+0000 7f1858e00640 1 -- 172.21.15.32:0/3900380220 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18540d1ba0 con 0x7f18540d0b00 2024-03-06T07:38:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.532+0000 7f1858e00640 1 --2- 172.21.15.32:0/3900380220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d0b00 0x7f18540b3630 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f18540b6880 tx=0x7f184c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b0d4f7e21bf3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.532+0000 7f1853400640 1 -- 172.21.15.32:0/3900380220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c02ca80 con 0x7f18540d0b00 2024-03-06T07:38:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.532+0000 7f1853400640 1 -- 172.21.15.32:0/3900380220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c02cc20 con 0x7f18540d0b00 2024-03-06T07:38:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.532+0000 7f1853400640 1 -- 172.21.15.32:0/3900380220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c0343f0 con 0x7f18540d0b00 2024-03-06T07:38:05.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 -- 172.21.15.32:0/3900380220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d0b00 msgr2=0x7f18540b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 --2- 172.21.15.32:0/3900380220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d0b00 0x7f18540b3630 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f18540b6880 tx=0x7f184c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 -- 172.21.15.32:0/3900380220 shutdown_connections 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 --2- 172.21.15.32:0/3900380220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d0b00 0x7f18540b3630 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 --2- 172.21.15.32:0/3900380220 >> v2:172.21.15.39:3300/0 conn(0x7f18540d2960 0x7f18540d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 -- 172.21.15.32:0/3900380220 >> 172.21.15.32:0/3900380220 conn(0x7f185408d2a0 msgr2=0x7f18540d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 -- 172.21.15.32:0/3900380220 shutdown_connections 2024-03-06T07:38:05.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.533+0000 7f1859800640 1 -- 172.21.15.32:0/3900380220 wait complete. 2024-03-06T07:38:05.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.535+0000 7f1859800640 1 Processor -- start 2024-03-06T07:38:05.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.535+0000 7f1859800640 1 -- start start 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1859800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18540d0b00 0x7f1854160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1859800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 0x7f18540c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1859800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f185415e070 0x7f18540c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1859800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540d2440 con 0x7f18540d2960 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1859800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540d2160 con 0x7f185415e070 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1859800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18540d45b0 con 0x7f18540d0b00 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f184b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 0x7f18540c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f184b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 0x7f18540c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59864/0 (socket says 172.21.15.32:59864) 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f184b400640 1 -- 172.21.15.32:0/3279201790 learned_addr learned my addr 172.21.15.32:0/3279201790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.536+0000 7f1853e00640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18540d0b00 0x7f1854160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f1858e00640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f185415e070 0x7f18540c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f184b400640 1 -- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18540d0b00 msgr2=0x7f1854160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f184b400640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18540d0b00 0x7f1854160a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f184b400640 1 -- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f185415e070 msgr2=0x7f18540c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f184b400640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f185415e070 0x7f18540c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f184b400640 1 -- 172.21.15.32:0/3279201790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18540c96c0 con 0x7f18540d2960 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f1858e00640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f185415e070 0x7f18540c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f184b400640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 0x7f18540c8a00 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f183c00c7f0 tx=0x7f183c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c015070 con 0x7f18540d2960 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c004240 con 0x7f18540d2960 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c007620 con 0x7f18540d2960 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.537+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18540c99a0 con 0x7f18540d2960 2024-03-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.538+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185415e580 con 0x7f18540d2960 2024-03-06T07:38:05.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.539+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185406d0b0 con 0x7f18540d2960 2024-03-06T07:38:05.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.540+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f183c0077c0 con 0x7f18540d2960 2024-03-06T07:38:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.541+0000 7f1850c00640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f183006b4d0 0x7f183006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.541+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f183c08a4c0 con 0x7f18540d2960 2024-03-06T07:38:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.541+0000 7f1853e00640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f183006b4d0 0x7f183006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.542+0000 7f1853e00640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f183006b4d0 0x7f183006d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1844006fd0 tx=0x7f1844008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.546+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f185406d0b0 con 0x7f18540d2960 2024-03-06T07:38:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.970+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f185404f330 con 0x7f18540d2960 2024-03-06T07:38:05.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.972+0000 7f1850c00640 1 -- 172.21.15.32:0/3279201790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f183c0592f0 con 0x7f18540d2960 2024-03-06T07:38:05.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:05.974 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f183006b4d0 msgr2=0x7f183006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f183006b4d0 0x7f183006d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1844006fd0 tx=0x7f1844008040 comp rx=0 tx=0).stop 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 msgr2=0x7f18540c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 0x7f18540c8a00 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f183c00c7f0 tx=0x7f183c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 shutdown_connections 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f183006b4d0 0x7f183006d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f185415e070 0x7f18540c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18540d2960 0x7f18540c8a00 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.974+0000 7f1859800640 1 --2- 172.21.15.32:0/3279201790 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f18540d0b00 0x7f1854160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.975+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 >> 172.21.15.32:0/3279201790 conn(0x7f185408d2a0 msgr2=0x7f185406d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.975+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 shutdown_connections 2024-03-06T07:38:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:05.975+0000 7f1859800640 1 -- 172.21.15.32:0/3279201790 wait complete. 2024-03-06T07:38:06.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:06.196 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:07.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:07.366 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:07.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.438+0000 7fe408800640 1 Processor -- start 2024-03-06T07:38:07.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.438+0000 7fe408800640 1 -- start start 2024-03-06T07:38:07.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.440+0000 7fe408800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400070ee0 0x7fe400162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.440+0000 7fe408800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe4000712c0 0x7fe40006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.440+0000 7fe408800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4000ce090 con 0x7fe400072da0 2024-03-06T07:38:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.440+0000 7fe408800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4000ceca0 con 0x7fe4000712c0 2024-03-06T07:38:07.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.440+0000 7fe408800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000ce370 con 0x7fe400070ee0 2024-03-06T07:38:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.441+0000 7fe405400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe400072da0 0x7fe40006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:35038/0 (socket says 172.21.15.32:35038) 2024-03-06T07:38:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.441+0000 7fe405400640 1 -- 172.21.15.32:0/984904510 learned_addr learned my addr 172.21.15.32:0/984904510 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.441+0000 7fe405e00640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400070ee0 0x7fe400162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.441+0000 7fe406800640 1 --2- 172.21.15.32:0/984904510 >> v2:172.21.15.39:3300/0 conn(0x7fe4000712c0 0x7fe40006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe404a00640 1 -- 172.21.15.32:0/984904510 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644306367 0 0) 0x7fe4000ce090 con 0x7fe400072da0 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe404a00640 1 -- 172.21.15.32:0/984904510 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e8003520 con 0x7fe400072da0 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe406800640 1 -- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400070ee0 msgr2=0x7fe400162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe406800640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400070ee0 0x7fe400162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe406800640 1 -- 172.21.15.32:0/984904510 >> v1:172.21.15.39:6789/0 conn(0x7fe400072da0 legacy=0x7fe40006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe406800640 1 -- 172.21.15.32:0/984904510 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4000ce650 con 0x7fe4000712c0 2024-03-06T07:38:07.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.442+0000 7fe405e00640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400070ee0 0x7fe400162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:38:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.443+0000 7fe406800640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4000712c0 0x7fe40006e7d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009d90 tx=0x7fe3fc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7973376210a6ed26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.444+0000 7fe404a00640 1 -- 172.21.15.32:0/984904510 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc039070 con 0x7fe4000712c0 2024-03-06T07:38:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.444+0000 7fe404a00640 1 -- 172.21.15.32:0/984904510 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc02cc50 con 0x7fe4000712c0 2024-03-06T07:38:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.444+0000 7fe404a00640 1 -- 172.21.15.32:0/984904510 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc034440 con 0x7fe4000712c0 2024-03-06T07:38:07.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 -- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4000712c0 msgr2=0x7fe40006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4000712c0 0x7fe40006e7d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009d90 tx=0x7fe3fc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 -- 172.21.15.32:0/984904510 shutdown_connections 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4000712c0 0x7fe40006e7d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 --2- 172.21.15.32:0/984904510 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400070ee0 0x7fe400162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 -- 172.21.15.32:0/984904510 >> 172.21.15.32:0/984904510 conn(0x7fe40006afb0 msgr2=0x7fe40006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 -- 172.21.15.32:0/984904510 shutdown_connections 2024-03-06T07:38:07.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.445+0000 7fe408800640 1 -- 172.21.15.32:0/984904510 wait complete. 2024-03-06T07:38:07.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.449+0000 7fe408800640 1 Processor -- start 2024-03-06T07:38:07.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.450+0000 7fe408800640 1 -- start start 2024-03-06T07:38:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe408800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400070ee0 0x7fe40015dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe408800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 0x7fe40009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe408800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400072da0 0x7fe4000b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe408800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000cdda0 con 0x7fe4000712c0 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe408800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000cdc20 con 0x7fe400070ee0 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe408800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000cf130 con 0x7fe400072da0 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe405400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 0x7fe40009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.452+0000 7fe405400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 0x7fe40009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59866/0 (socket says 172.21.15.32:59866) 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 -- 172.21.15.32:0/1690117177 learned_addr learned my addr 172.21.15.32:0/1690117177 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405e00640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400070ee0 0x7fe40015dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe406800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400072da0 0x7fe4000b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 -- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400072da0 msgr2=0x7fe4000b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400072da0 0x7fe4000b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 -- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400070ee0 msgr2=0x7fe40015dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400070ee0 0x7fe40015dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 -- 172.21.15.32:0/1690117177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4001ec070 con 0x7fe4000712c0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405e00640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400070ee0 0x7fe40015dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.453+0000 7fe405400640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 0x7fe40009aa10 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c440 tx=0x7fe3f000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.454+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0017070 con 0x7fe4000712c0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.454+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f0005ce0 con 0x7fe4000712c0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.454+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f00072b0 con 0x7fe4000712c0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.454+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4001ec310 con 0x7fe4000712c0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.454+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4001ec8d0 con 0x7fe4000712c0 2024-03-06T07:38:07.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.455+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40004f3a0 con 0x7fe4000712c0 2024-03-06T07:38:07.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.457+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3f001a070 con 0x7fe4000712c0 2024-03-06T07:38:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.458+0000 7fe3fa000640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe3e406b4d0 0x7fe3e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.458+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe3f0089670 con 0x7fe4000712c0 2024-03-06T07:38:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.459+0000 7fe405e00640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe3e406b4d0 0x7fe3e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.459+0000 7fe405e00640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe3e406b4d0 0x7fe3e406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009ba0 tx=0x7fe3f4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:07.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.466+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3f00584d0 con 0x7fe4000712c0 2024-03-06T07:38:07.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.705+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4000b52d0 con 0x7fe4000712c0 2024-03-06T07:38:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.706+0000 7fe3fa000640 1 -- 172.21.15.32:0/1690117177 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fe3f0057dd0 con 0x7fe4000712c0 2024-03-06T07:38:07.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:07.709 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:07.709 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.708+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe3e406b4d0 msgr2=0x7fe3e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.708+0000 7fe408800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe3e406b4d0 0x7fe3e406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009ba0 tx=0x7fe3f4008040 comp rx=0 tx=0).stop 2024-03-06T07:38:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.708+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 msgr2=0x7fe40009aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.708+0000 7fe408800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 0x7fe40009aa10 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c440 tx=0x7fe3f000c900 comp rx=0 tx=0).stop 2024-03-06T07:38:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 shutdown_connections 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe3e406b4d0 0x7fe3e406d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe400072da0 0x7fe4000b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4000712c0 0x7fe40009aa10 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 --2- 172.21.15.32:0/1690117177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400070ee0 0x7fe40015dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 >> 172.21.15.32:0/1690117177 conn(0x7fe40006afb0 msgr2=0x7fe4000b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 shutdown_connections 2024-03-06T07:38:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:07.709+0000 7fe408800640 1 -- 172.21.15.32:0/1690117177 wait complete. 2024-03-06T07:38:07.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:07.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:08.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:09.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.216+0000 7fd923e00640 1 Processor -- start 2024-03-06T07:38:09.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.217+0000 7fd923e00640 1 -- start start 2024-03-06T07:38:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd92415e0b0 0x7fd9240cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9240d4a10 0x7fd9240d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240b3710 con 0x7fd9240d0c80 2024-03-06T07:38:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9240b09e0 con 0x7fd9240d4a10 2024-03-06T07:38:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240b39f0 con 0x7fd92415e0b0 2024-03-06T07:38:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd922000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9240d4a10 0x7fd9240d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd9240d0c80 0x7fd9240d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52698/0 (socket says 172.21.15.32:52698) 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.219+0000 7fd923400640 1 -- 172.21.15.32:0/1344896039 learned_addr learned my addr 172.21.15.32:0/1344896039 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.220+0000 7fd922a00640 1 --2- 172.21.15.32:0/1344896039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd92415e0b0 0x7fd9240cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.220+0000 7fd922000640 1 -- 172.21.15.32:0/1344896039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd92415e0b0 msgr2=0x7fd9240cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.220+0000 7fd922000640 1 --2- 172.21.15.32:0/1344896039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd92415e0b0 0x7fd9240cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.220+0000 7fd922000640 1 -- 172.21.15.32:0/1344896039 >> v1:172.21.15.32:6789/0 conn(0x7fd9240d0c80 legacy=0x7fd9240d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.220+0000 7fd922000640 1 -- 172.21.15.32:0/1344896039 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240b3cd0 con 0x7fd9240d4a10 2024-03-06T07:38:09.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.220+0000 7fd921600640 1 -- 172.21.15.32:0/1344896039 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561741923 0 0) 0x7fd9240b3710 con 0x7fd9240d0c80 2024-03-06T07:38:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.221+0000 7fd922000640 1 --2- 172.21.15.32:0/1344896039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d4a10 0x7fd9240d0740 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd9100097a0 tx=0x7fd91002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57eac9ba8f42e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.221+0000 7fd921600640 1 -- 172.21.15.32:0/1344896039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910035020 con 0x7fd9240d4a10 2024-03-06T07:38:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.221+0000 7fd921600640 1 -- 172.21.15.32:0/1344896039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910031830 con 0x7fd9240d4a10 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.221+0000 7fd921600640 1 -- 172.21.15.32:0/1344896039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910031c00 con 0x7fd9240d4a10 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 -- 172.21.15.32:0/1344896039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d4a10 msgr2=0x7fd9240d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 --2- 172.21.15.32:0/1344896039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d4a10 0x7fd9240d0740 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd9100097a0 tx=0x7fd91002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 -- 172.21.15.32:0/1344896039 shutdown_connections 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 --2- 172.21.15.32:0/1344896039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d4a10 0x7fd9240d0740 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 --2- 172.21.15.32:0/1344896039 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd92415e0b0 0x7fd9240cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 -- 172.21.15.32:0/1344896039 >> 172.21.15.32:0/1344896039 conn(0x7fd9240d3d30 msgr2=0x7fd9240d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 -- 172.21.15.32:0/1344896039 shutdown_connections 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.222+0000 7fd923e00640 1 -- 172.21.15.32:0/1344896039 wait complete. 2024-03-06T07:38:09.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.224+0000 7fd923e00640 1 Processor -- start 2024-03-06T07:38:09.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.224+0000 7fd923e00640 1 -- start start 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd923e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 0x7fd92415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd923e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9240d4a10 0x7fd924160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd923e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd92415e0b0 0x7fd924160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd923e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240cfc60 con 0x7fd9240d0c80 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd923e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92415e490 con 0x7fd92415e0b0 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd923e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240b54e0 con 0x7fd9240d4a10 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd922a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 0x7fd92415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd922a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 0x7fd92415b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50526/0 (socket says 172.21.15.32:50526) 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.225+0000 7fd922a00640 1 -- 172.21.15.32:0/668652867 learned_addr learned my addr 172.21.15.32:0/668652867 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922000640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9240d4a10 0x7fd924160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922a00640 1 -- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9240d4a10 msgr2=0x7fd924160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922a00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9240d4a10 0x7fd924160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd923400640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd92415e0b0 0x7fd924160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922a00640 1 -- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd92415e0b0 msgr2=0x7fd924160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922a00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd92415e0b0 0x7fd924160aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922a00640 1 -- 172.21.15.32:0/668652867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9241f4b90 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922000640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9240d4a10 0x7fd924160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd922a00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 0x7fd92415b960 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd91400d620 tx=0x7fd91400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400b620 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd914019070 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400ba00 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.226+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9240cb050 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.227+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92415bea0 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.228+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92404f330 con 0x7fd9240d0c80 2024-03-06T07:38:09.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.229+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd91400bba0 con 0x7fd9240d0c80 2024-03-06T07:38:09.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.230+0000 7fd91b400640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd90406b500 0x7fd90406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.230+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd91408a6e0 con 0x7fd9240d0c80 2024-03-06T07:38:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.230+0000 7fd922000640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd90406b500 0x7fd90406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.231+0000 7fd922000640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd90406b500 0x7fd90406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd910036000 tx=0x7fd910008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.234+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd914059480 con 0x7fd9240d0c80 2024-03-06T07:38:09.368 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.476+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9240d1af0 con 0x7fd9240d0c80 2024-03-06T07:38:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.477+0000 7fd91b400640 1 -- 172.21.15.32:0/668652867 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fd914058d80 con 0x7fd9240d0c80 2024-03-06T07:38:09.480 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:09.480 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd90406b500 msgr2=0x7fd90406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd90406b500 0x7fd90406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd910036000 tx=0x7fd910008030 comp rx=0 tx=0).stop 2024-03-06T07:38:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 msgr2=0x7fd92415b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 0x7fd92415b960 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd91400d620 tx=0x7fd91400dae0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 shutdown_connections 2024-03-06T07:38:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd90406b500 0x7fd90406d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd92415e0b0 0x7fd924160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9240d4a10 0x7fd924160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 --2- 172.21.15.32:0/668652867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0c80 0x7fd92415b960 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 >> 172.21.15.32:0/668652867 conn(0x7fd9240d3d30 msgr2=0x7fd9240d22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 shutdown_connections 2024-03-06T07:38:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:09.480+0000 7fd923e00640 1 -- 172.21.15.32:0/668652867 wait complete. 2024-03-06T07:38:09.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:09.703 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:10.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:10.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.918+0000 7fabaa000640 1 Processor -- start 2024-03-06T07:38:10.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.918+0000 7fabaa000640 1 -- start start 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.920+0000 7fabaa000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faba40b34e0 0x7faba415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.920+0000 7fabaa000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba415cc40 con 0x7faba40af1a0 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.920+0000 7fabaa000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba415cf20 con 0x7faba40af580 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.920+0000 7fabaa000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba415d200 con 0x7faba40b34e0 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.920+0000 7faba3400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faba40af1a0 0x7faba40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52702/0 (socket says 172.21.15.32:52702) 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.920+0000 7faba3400640 1 -- 172.21.15.32:0/3594415784 learned_addr learned my addr 172.21.15.32:0/3594415784 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba3e00640 1 --2- 172.21.15.32:0/3594415784 >> v2:172.21.15.39:3300/0 conn(0x7faba40b34e0 0x7faba415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625933198 0 0) 0x7faba415cc40 con 0x7faba40af1a0 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab88003520 con 0x7faba40af1a0 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861187588 0 0) 0x7fab88003520 con 0x7faba40af1a0 2024-03-06T07:38:10.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 >> v2:172.21.15.39:3300/0 conn(0x7faba40b34e0 msgr2=0x7faba415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 --2- 172.21.15.32:0/3594415784 >> v2:172.21.15.39:3300/0 conn(0x7faba40b34e0 0x7faba415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 >> v1:172.21.15.39:6789/0 conn(0x7faba40af580 legacy=0x7faba40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba415d4e0 con 0x7faba40af1a0 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fab940019f0 con 0x7faba40af1a0 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 >> v1:172.21.15.32:6789/0 conn(0x7faba40af1a0 legacy=0x7faba40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab88003a20 0x7fab88003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab88004800 0x7fab88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab880052e0 0x7fab8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba415d200 con 0x7fab88004800 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8800c230 con 0x7fab880052e0 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.921+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8800c430 con 0x7fab88003a20 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba2a00640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab88004800 0x7fab88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba3400640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab88003a20 0x7fab88003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba3e00640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab880052e0 0x7fab8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba2a00640 1 -- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab88003a20 msgr2=0x7fab88003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba2a00640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab88003a20 0x7fab88003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba2a00640 1 -- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab880052e0 msgr2=0x7fab8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba2a00640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab880052e0 0x7fab8800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba2a00640 1 -- 172.21.15.32:0/3594415784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88003520 con 0x7fab88004800 2024-03-06T07:38:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba3e00640 1 -- 172.21.15.32:0/3594415784 reap_dead start 2024-03-06T07:38:10.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.922+0000 7faba3400640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab88003a20 0x7fab88003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.923+0000 7faba2a00640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab88004800 0x7fab88004c80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7faba40afa10 tx=0x7fab88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dd58f98c862b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.923+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90001390 con 0x7fab88004800 2024-03-06T07:38:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.923+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab900052d0 con 0x7fab88004800 2024-03-06T07:38:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.923+0000 7faba2000640 1 -- 172.21.15.32:0/3594415784 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90003830 con 0x7fab88004800 2024-03-06T07:38:10.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 -- 172.21.15.32:0/3594415784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab88004800 msgr2=0x7fab88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab88004800 0x7fab88004c80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7faba40afa10 tx=0x7fab88005830 comp rx=0 tx=0).stop 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 -- 172.21.15.32:0/3594415784 shutdown_connections 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 --2- 172.21.15.32:0/3594415784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab88004800 0x7fab88004c80 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 -- 172.21.15.32:0/3594415784 >> 172.21.15.32:0/3594415784 conn(0x7faba406afb0 msgr2=0x7faba406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 -- 172.21.15.32:0/3594415784 shutdown_connections 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.924+0000 7fabaa000640 1 -- 172.21.15.32:0/3594415784 wait complete. 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.926+0000 7fabaa000640 1 Processor -- start 2024-03-06T07:38:10.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.926+0000 7fabaa000640 1 -- start start 2024-03-06T07:38:10.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7fabaa000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab88004800 0x7faba41fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7fabaa000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 0x7faba4153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7fabaa000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faba41f8c50 0x7faba406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7fabaa000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba415c600 con 0x7faba415abc0 2024-03-06T07:38:10.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7fabaa000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40cde30 con 0x7fab88004800 2024-03-06T07:38:10.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7fabaa000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba415c780 con 0x7faba41f8c50 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7faba2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 0x7faba4153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.927+0000 7faba2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 0x7faba4153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50550/0 (socket says 172.21.15.32:50550) 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 -- 172.21.15.32:0/2151846608 learned_addr learned my addr 172.21.15.32:0/2151846608 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba3400640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab88004800 0x7faba41fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba3e00640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faba41f8c50 0x7faba406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 -- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faba41f8c50 msgr2=0x7faba406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faba41f8c50 0x7faba406ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 -- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab88004800 msgr2=0x7faba41fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab88004800 0x7faba41fbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 -- 172.21.15.32:0/2151846608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba4154140 con 0x7faba415abc0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.928+0000 7faba2a00640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 0x7faba4153c00 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fab90001830 tx=0x7fab90001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.929+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90041070 con 0x7faba415abc0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.929+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab90005480 con 0x7faba415abc0 2024-03-06T07:38:10.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.929+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba41fc210 con 0x7faba415abc0 2024-03-06T07:38:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.929+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90007240 con 0x7faba415abc0 2024-03-06T07:38:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.929+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba41fc7d0 con 0x7faba415abc0 2024-03-06T07:38:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.930+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba4069510 con 0x7faba415abc0 2024-03-06T07:38:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.931+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab9003b040 con 0x7faba415abc0 2024-03-06T07:38:10.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.932+0000 7fab9b400640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab8406b4d0 0x7fab8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:10.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.932+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fab900abb00 con 0x7faba415abc0 2024-03-06T07:38:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.932+0000 7faba3400640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab8406b4d0 0x7fab8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:10.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.933+0000 7faba3400640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab8406b4d0 0x7fab8406d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fab94002230 tx=0x7fab940014f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:10.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:10.937+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab9007a8b0 con 0x7faba415abc0 2024-03-06T07:38:11.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.182+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faba41fc9a0 con 0x7faba415abc0 2024-03-06T07:38:11.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.183+0000 7fab9b400640 1 -- 172.21.15.32:0/2151846608 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fab9007a1b0 con 0x7faba415abc0 2024-03-06T07:38:11.185 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:11.186 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:11.186 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab8406b4d0 msgr2=0x7fab8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab8406b4d0 0x7fab8406d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fab94002230 tx=0x7fab940014f0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 msgr2=0x7faba4153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 0x7faba4153c00 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fab90001830 tx=0x7fab90001120 comp rx=0 tx=0).stop 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 shutdown_connections 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab8406b4d0 0x7fab8406d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7faba41f8c50 0x7faba406ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faba415abc0 0x7faba4153c00 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 --2- 172.21.15.32:0/2151846608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab88004800 0x7faba41fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 >> 172.21.15.32:0/2151846608 conn(0x7faba406afb0 msgr2=0x7faba40ced80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 shutdown_connections 2024-03-06T07:38:11.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:11.186+0000 7fabaa000640 1 -- 172.21.15.32:0/2151846608 wait complete. 2024-03-06T07:38:11.370 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:11.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:11.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:12.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:12.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.643+0000 7fc11a000640 1 Processor -- start 2024-03-06T07:38:12.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.644+0000 7fc11a000640 1 -- start start 2024-03-06T07:38:12.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.645+0000 7fc11a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 0x7fc11415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:12.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.645+0000 7fc11a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc1140af580 0x7fc1140b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:12.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.645+0000 7fc11a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11415cc40 con 0x7fc1140cdc20 2024-03-06T07:38:12.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.645+0000 7fc11a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11415cf20 con 0x7fc1140af580 2024-03-06T07:38:12.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.645+0000 7fc11a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11415d200 con 0x7fc1140af1a0 2024-03-06T07:38:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.646+0000 7fc113400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 0x7fc11415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.646+0000 7fc113400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 0x7fc11415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:53316/0 (socket says 172.21.15.32:53316) 2024-03-06T07:38:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.646+0000 7fc113400640 1 -- 172.21.15.32:0/2092246212 learned_addr learned my addr 172.21.15.32:0/2092246212 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.646+0000 7fc10be00640 1 --2- 172.21.15.32:0/2092246212 >> v2:172.21.15.39:3300/0 conn(0x7fc1140af580 0x7fc1140b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.647+0000 7fc113400640 1 -- 172.21.15.32:0/2092246212 >> v2:172.21.15.39:3300/0 conn(0x7fc1140af580 msgr2=0x7fc1140b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.647+0000 7fc113400640 1 --2- 172.21.15.32:0/2092246212 >> v2:172.21.15.39:3300/0 conn(0x7fc1140af580 0x7fc1140b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.647+0000 7fc113400640 1 -- 172.21.15.32:0/2092246212 >> v1:172.21.15.39:6789/0 conn(0x7fc1140cdc20 legacy=0x7fc1140b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.647+0000 7fc113400640 1 -- 172.21.15.32:0/2092246212 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11415d4e0 con 0x7fc1140af1a0 2024-03-06T07:38:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.648+0000 7fc113400640 1 --2- 172.21.15.32:0/2092246212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 0x7fc11415c600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc104009870 tx=0x7fc10402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=867568b172856d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc112a00640 1 -- 172.21.15.32:0/2092246212 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1040317c0 con 0x7fc1140af1a0 2024-03-06T07:38:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc112a00640 1 -- 172.21.15.32:0/2092246212 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104037070 con 0x7fc1140af1a0 2024-03-06T07:38:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc112a00640 1 -- 172.21.15.32:0/2092246212 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc104031b80 con 0x7fc1140af1a0 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc11a000640 1 -- 172.21.15.32:0/2092246212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 msgr2=0x7fc11415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc11a000640 1 --2- 172.21.15.32:0/2092246212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 0x7fc11415c600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc104009870 tx=0x7fc10402b860 comp rx=0 tx=0).stop 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc11a000640 1 -- 172.21.15.32:0/2092246212 shutdown_connections 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc11a000640 1 --2- 172.21.15.32:0/2092246212 >> v2:172.21.15.39:3300/0 conn(0x7fc1140af580 0x7fc1140b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.649+0000 7fc11a000640 1 --2- 172.21.15.32:0/2092246212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af1a0 0x7fc11415c600 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.650+0000 7fc11a000640 1 -- 172.21.15.32:0/2092246212 >> 172.21.15.32:0/2092246212 conn(0x7fc11406afb0 msgr2=0x7fc11406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.650+0000 7fc11a000640 1 -- 172.21.15.32:0/2092246212 shutdown_connections 2024-03-06T07:38:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.650+0000 7fc11a000640 1 -- 172.21.15.32:0/2092246212 wait complete. 2024-03-06T07:38:12.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.652+0000 7fc11a000640 1 Processor -- start 2024-03-06T07:38:12.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.652+0000 7fc11a000640 1 -- start start 2024-03-06T07:38:12.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.653+0000 7fc11a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 0x7fc11415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:12.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.653+0000 7fc11a000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af580 0x7fc1141f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.653+0000 7fc11a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1140cdc20 0x7fc1140ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.653+0000 7fc11a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1140b3660 con 0x7fc1140af1a0 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.653+0000 7fc11a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1140b34e0 con 0x7fc1140cdc20 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.653+0000 7fc11a000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1140b37e0 con 0x7fc1140af580 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 0x7fc11415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 0x7fc11415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50552/0 (socket says 172.21.15.32:50552) 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 -- 172.21.15.32:0/3475459951 learned_addr learned my addr 172.21.15.32:0/3475459951 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc10be00640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af580 0x7fc1141f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113e00640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1140cdc20 0x7fc1140ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 -- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af580 msgr2=0x7fc1141f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af580 0x7fc1141f4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 -- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1140cdc20 msgr2=0x7fc1140ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1140cdc20 0x7fc1140ad720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113400640 1 -- 172.21.15.32:0/3475459951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11406d010 con 0x7fc1140af1a0 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.654+0000 7fc113e00640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1140cdc20 0x7fc1140ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.655+0000 7fc113400640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 0x7fc11415ad50 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc104000c00 tx=0x7fc1040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.655+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc104032650 con 0x7fc1140af1a0 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.655+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104032830 con 0x7fc1140af1a0 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.655+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc104032c20 con 0x7fc1140af1a0 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.655+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11406d2f0 con 0x7fc1140af1a0 2024-03-06T07:38:12.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.655+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11415b350 con 0x7fc1140af1a0 2024-03-06T07:38:12.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.657+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc114069510 con 0x7fc1140af1a0 2024-03-06T07:38:12.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.657+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc10404c060 con 0x7fc1140af1a0 2024-03-06T07:38:12.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.658+0000 7fc10b400640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc0f006b4d0 0x7fc0f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:12.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.658+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc1040abca0 con 0x7fc1140af1a0 2024-03-06T07:38:12.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.659+0000 7fc10be00640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc0f006b4d0 0x7fc0f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:12.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.660+0000 7fc10be00640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc0f006b4d0 0x7fc0f006d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc11414fde0 tx=0x7fc0fc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:12.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.663+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc104040070 con 0x7fc1140af1a0 2024-03-06T07:38:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.905+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1140b3020 con 0x7fc1140af1a0 2024-03-06T07:38:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.905+0000 7fc10b400640 1 -- 172.21.15.32:0/3475459951 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fc10407a490 con 0x7fc1140af1a0 2024-03-06T07:38:12.908 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:12.908 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc0f006b4d0 msgr2=0x7fc0f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc0f006b4d0 0x7fc0f006d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc11414fde0 tx=0x7fc0fc008040 comp rx=0 tx=0).stop 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 msgr2=0x7fc11415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 0x7fc11415ad50 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc104000c00 tx=0x7fc1040046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 shutdown_connections 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc0f006b4d0 0x7fc0f006d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1140cdc20 0x7fc1140ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc1140af580 0x7fc1141f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 --2- 172.21.15.32:0/3475459951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1140af1a0 0x7fc11415ad50 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 >> 172.21.15.32:0/3475459951 conn(0x7fc11406afb0 msgr2=0x7fc114071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 shutdown_connections 2024-03-06T07:38:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:12.908+0000 7fc11a000640 1 -- 172.21.15.32:0/3475459951 wait complete. 2024-03-06T07:38:13.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:13.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:14.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.357+0000 7f7e66600640 1 Processor -- start 2024-03-06T07:38:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.357+0000 7f7e66600640 1 -- start start 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e66600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7e600d2960 0x7f7e600d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e66600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e600d0b00 0x7f7e600b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e66600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e600d2dc0 con 0x7f7e6015e070 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e66600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e600d0f70 con 0x7f7e600d0b00 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e66600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e600d18c0 con 0x7f7e600d2960 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e65c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e600d0b00 0x7f7e600b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e65200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7e6015e070 0x7f7e600cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52708/0 (socket says 172.21.15.32:52708) 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e65200640 1 -- 172.21.15.32:0/2323728241 learned_addr learned my addr 172.21.15.32:0/2323728241 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.358+0000 7f7e65c00640 1 --2- 172.21.15.32:0/2323728241 >> v2:172.21.15.32:3300/0 conn(0x7f7e600d0b00 0x7f7e600b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50556/0 (socket says 172.21.15.32:50556) 2024-03-06T07:38:14.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.359+0000 7f7e5fe00640 1 --2- 172.21.15.32:0/2323728241 >> v2:172.21.15.39:3300/0 conn(0x7f7e600d2960 0x7f7e600d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.359+0000 7f7e65c00640 1 -- 172.21.15.32:0/2323728241 >> v2:172.21.15.39:3300/0 conn(0x7f7e600d2960 msgr2=0x7f7e600d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.359+0000 7f7e65c00640 1 --2- 172.21.15.32:0/2323728241 >> v2:172.21.15.39:3300/0 conn(0x7f7e600d2960 0x7f7e600d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.359+0000 7f7e65c00640 1 -- 172.21.15.32:0/2323728241 >> v1:172.21.15.32:6789/0 conn(0x7f7e6015e070 legacy=0x7f7e600cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.359+0000 7f7e65c00640 1 -- 172.21.15.32:0/2323728241 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e600d1ba0 con 0x7f7e600d0b00 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.359+0000 7f7e5fe00640 1 --2- 172.21.15.32:0/2323728241 >> v2:172.21.15.39:3300/0 conn(0x7f7e600d2960 0x7f7e600d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.360+0000 7f7e65c00640 1 --2- 172.21.15.32:0/2323728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d0b00 0x7f7e600b3630 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f7e600b6880 tx=0x7f7e5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2967556ed896eb35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.360+0000 7f7e5f400640 1 -- 172.21.15.32:0/2323728241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58035020 con 0x7f7e600d0b00 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e5f400640 1 -- 172.21.15.32:0/2323728241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e58031830 con 0x7f7e600d0b00 2024-03-06T07:38:14.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e5f400640 1 -- 172.21.15.32:0/2323728241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58031c00 con 0x7f7e600d0b00 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 -- 172.21.15.32:0/2323728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d0b00 msgr2=0x7f7e600b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 --2- 172.21.15.32:0/2323728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d0b00 0x7f7e600b3630 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f7e600b6880 tx=0x7f7e5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 -- 172.21.15.32:0/2323728241 shutdown_connections 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 --2- 172.21.15.32:0/2323728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d0b00 0x7f7e600b3630 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 --2- 172.21.15.32:0/2323728241 >> v2:172.21.15.39:3300/0 conn(0x7f7e600d2960 0x7f7e600d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 -- 172.21.15.32:0/2323728241 >> 172.21.15.32:0/2323728241 conn(0x7f7e6008d2a0 msgr2=0x7f7e600d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 -- 172.21.15.32:0/2323728241 shutdown_connections 2024-03-06T07:38:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.361+0000 7f7e66600640 1 -- 172.21.15.32:0/2323728241 wait complete. 2024-03-06T07:38:14.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.363+0000 7f7e66600640 1 Processor -- start 2024-03-06T07:38:14.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.363+0000 7f7e66600640 1 -- start start 2024-03-06T07:38:14.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.364+0000 7f7e66600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e600d0b00 0x7f7e6015e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.364+0000 7f7e66600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 0x7f7e601500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.364+0000 7f7e66600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e6015e070 0x7f7e600cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.364+0000 7f7e66600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e600d2440 con 0x7f7e600d2960 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.364+0000 7f7e66600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e600d2160 con 0x7f7e6015e070 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.364+0000 7f7e66600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e600d45b0 con 0x7f7e600d0b00 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 0x7f7e601500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 0x7f7e601500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50558/0 (socket says 172.21.15.32:50558) 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 -- 172.21.15.32:0/2137751364 learned_addr learned my addr 172.21.15.32:0/2137751364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e65200640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e600d0b00 0x7f7e6015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e65c00640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e6015e070 0x7f7e600cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 -- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e600d0b00 msgr2=0x7f7e6015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e600d0b00 0x7f7e6015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 -- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e6015e070 msgr2=0x7f7e600cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e6015e070 0x7f7e600cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e5fe00640 1 -- 172.21.15.32:0/2137751364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6015e990 con 0x7f7e600d2960 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.365+0000 7f7e65c00640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e6015e070 0x7f7e600cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.366+0000 7f7e5fe00640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 0x7f7e601500e0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00c7f0 tx=0x7f7e4c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.366+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c015070 con 0x7f7e600d2960 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.366+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c007a30 con 0x7f7e600d2960 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.366+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6015ec30 con 0x7f7e600d2960 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.366+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c007e00 con 0x7f7e600d2960 2024-03-06T07:38:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.366+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6015f1f0 con 0x7f7e600d2960 2024-03-06T07:38:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.368+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6004f330 con 0x7f7e600d2960 2024-03-06T07:38:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.368+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e4c018070 con 0x7f7e600d2960 2024-03-06T07:38:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.369+0000 7f7e5cc00640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e4006b4d0 0x7f7e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.369+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e4c08a230 con 0x7f7e600d2960 2024-03-06T07:38:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.369+0000 7f7e65200640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e4006b4d0 0x7f7e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.370+0000 7f7e65200640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e4006b4d0 0x7f7e4006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7e500012e0 tx=0x7f7e50009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.374+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e4c058fd0 con 0x7f7e600d2960 2024-03-06T07:38:14.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.721+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e600d1370 con 0x7f7e600d2960 2024-03-06T07:38:14.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.722+0000 7f7e5cc00640 1 -- 172.21.15.32:0/2137751364 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f7e4c0588d0 con 0x7f7e600d2960 2024-03-06T07:38:14.725 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:14.725 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:14.725 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.725+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e4006b4d0 msgr2=0x7f7e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.725+0000 7f7e66600640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e4006b4d0 0x7f7e4006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7e500012e0 tx=0x7f7e50009040 comp rx=0 tx=0).stop 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.725+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 msgr2=0x7f7e601500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.725+0000 7f7e66600640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 0x7f7e601500e0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00c7f0 tx=0x7f7e4c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 shutdown_connections 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7e4006b4d0 0x7f7e4006d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e6015e070 0x7f7e600cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e600d2960 0x7f7e601500e0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 --2- 172.21.15.32:0/2137751364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7e600d0b00 0x7f7e6015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 >> 172.21.15.32:0/2137751364 conn(0x7f7e6008d2a0 msgr2=0x7f7e6006cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 shutdown_connections 2024-03-06T07:38:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:14.726+0000 7f7e66600640 1 -- 172.21.15.32:0/2137751364 wait complete. 2024-03-06T07:38:14.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:14.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:15.374 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:15.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:16.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.179+0000 7f912d000640 1 Processor -- start 2024-03-06T07:38:16.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.179+0000 7f912d000640 1 -- start start 2024-03-06T07:38:16.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f912d000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f91280b34e0 0x7f912815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f912d000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912815cc40 con 0x7f91280af1a0 2024-03-06T07:38:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f912d000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912815cf20 con 0x7f91280af580 2024-03-06T07:38:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f912d000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912815d200 con 0x7f91280b34e0 2024-03-06T07:38:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f9125e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f91280af1a0 0x7f91280b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52724/0 (socket says 172.21.15.32:52724) 2024-03-06T07:38:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f9125e00640 1 -- 172.21.15.32:0/1221933552 learned_addr learned my addr 172.21.15.32:0/1221933552 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.181+0000 7f9126800640 1 --2- 172.21.15.32:0/1221933552 >> v2:172.21.15.32:3300/0 conn(0x7f91280b34e0 0x7f912815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.182+0000 7f9125400640 1 -- 172.21.15.32:0/1221933552 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301024778 0 0) 0x7f912815cc40 con 0x7f91280af1a0 2024-03-06T07:38:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.182+0000 7f9126800640 1 -- 172.21.15.32:0/1221933552 >> v1:172.21.15.39:6789/0 conn(0x7f91280af580 legacy=0x7f91280b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.182+0000 7f9126800640 1 -- 172.21.15.32:0/1221933552 >> v1:172.21.15.32:6789/0 conn(0x7f91280af1a0 legacy=0x7f91280b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.182+0000 7f9126800640 1 -- 172.21.15.32:0/1221933552 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912815d4e0 con 0x7f91280b34e0 2024-03-06T07:38:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.183+0000 7f9126800640 1 --2- 172.21.15.32:0/1221933552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280b34e0 0x7f912815c600 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f9120009d90 tx=0x7f912002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b66b01af7cf23c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.183+0000 7f9125400640 1 -- 172.21.15.32:0/1221933552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120039070 con 0x7f91280b34e0 2024-03-06T07:38:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.183+0000 7f9125400640 1 -- 172.21.15.32:0/1221933552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912002cbd0 con 0x7f91280b34e0 2024-03-06T07:38:16.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.183+0000 7f9125400640 1 -- 172.21.15.32:0/1221933552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120034440 con 0x7f91280b34e0 2024-03-06T07:38:16.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 -- 172.21.15.32:0/1221933552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280b34e0 msgr2=0x7f912815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 --2- 172.21.15.32:0/1221933552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280b34e0 0x7f912815c600 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f9120009d90 tx=0x7f912002be20 comp rx=0 tx=0).stop 2024-03-06T07:38:16.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 -- 172.21.15.32:0/1221933552 shutdown_connections 2024-03-06T07:38:16.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 --2- 172.21.15.32:0/1221933552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280b34e0 0x7f912815c600 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 -- 172.21.15.32:0/1221933552 >> 172.21.15.32:0/1221933552 conn(0x7f912806afb0 msgr2=0x7f912806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:16.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 -- 172.21.15.32:0/1221933552 shutdown_connections 2024-03-06T07:38:16.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.184+0000 7f912d000640 1 -- 172.21.15.32:0/1221933552 wait complete. 2024-03-06T07:38:16.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.186+0000 7f912d000640 1 Processor -- start 2024-03-06T07:38:16.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.186+0000 7f912d000640 1 -- start start 2024-03-06T07:38:16.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f912d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 0x7f91281f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f912d000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f91280af580 0x7f912806e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f912d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91280b34e0 0x7f9128148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f912d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128155e70 con 0x7f91280af1a0 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f912d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280cde30 con 0x7f91280b34e0 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f912d000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9128155ff0 con 0x7f91280af580 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f9125e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 0x7f91281f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f9125e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 0x7f91281f7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50576/0 (socket says 172.21.15.32:50576) 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.187+0000 7f9125e00640 1 -- 172.21.15.32:0/3755241833 learned_addr learned my addr 172.21.15.32:0/3755241833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f911fe00640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f91280af580 0x7f912806e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9126800640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91280b34e0 0x7f9128148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9125e00640 1 -- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f91280af580 msgr2=0x7f912806e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9125e00640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f91280af580 0x7f912806e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9125e00640 1 -- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91280b34e0 msgr2=0x7f9128148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9125e00640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91280b34e0 0x7f9128148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9125e00640 1 -- 172.21.15.32:0/3755241833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912815ef20 con 0x7f91280af1a0 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9126800640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91280b34e0 0x7f9128148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f9125e00640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 0x7f91281f7bb0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f911800c4d0 tx=0x7f911800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.188+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118018070 con 0x7f91280af1a0 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.189+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9118007500 con 0x7f91280af1a0 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.189+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118004a20 con 0x7f91280af1a0 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.189+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912815f200 con 0x7f91280af1a0 2024-03-06T07:38:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.189+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91281f73b0 con 0x7f91280af1a0 2024-03-06T07:38:16.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.190+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912804f3a0 con 0x7f91280af1a0 2024-03-06T07:38:16.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.192+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9118004bc0 con 0x7f91280af1a0 2024-03-06T07:38:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.193+0000 7f911e000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f910406b4d0 0x7f910406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.193+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f91180896e0 con 0x7f91280af1a0 2024-03-06T07:38:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.193+0000 7f911fe00640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f910406b4d0 0x7f910406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.194+0000 7f911fe00640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f910406b4d0 0x7f910406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f912814e2a0 tx=0x7f9110009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.201+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9118058400 con 0x7f91280af1a0 2024-03-06T07:38:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.444+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91280cc9e0 con 0x7f91280af1a0 2024-03-06T07:38:16.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.445+0000 7f911e000640 1 -- 172.21.15.32:0/3755241833 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f9118006e20 con 0x7f91280af1a0 2024-03-06T07:38:16.448 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:16.448 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f910406b4d0 msgr2=0x7f910406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f910406b4d0 0x7f910406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f912814e2a0 tx=0x7f9110009040 comp rx=0 tx=0).stop 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 msgr2=0x7f91281f7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 0x7f91281f7bb0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f911800c4d0 tx=0x7f911800c990 comp rx=0 tx=0).stop 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 shutdown_connections 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f910406b4d0 0x7f910406d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91280b34e0 0x7f9128148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f91280af580 0x7f912806e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 --2- 172.21.15.32:0/3755241833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91280af1a0 0x7f91281f7bb0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 >> 172.21.15.32:0/3755241833 conn(0x7f912806afb0 msgr2=0x7f912806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.448+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 shutdown_connections 2024-03-06T07:38:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:16.449+0000 7f912d000640 1 -- 172.21.15.32:0/3755241833 wait complete. 2024-03-06T07:38:16.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:16.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:17.376 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:17.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.889+0000 7f7f48a00640 1 Processor -- start 2024-03-06T07:38:17.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.889+0000 7f7f48a00640 1 -- start start 2024-03-06T07:38:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f48a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f440d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f48a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7f440cf410 0x7f7f440cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f48a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f440d18c0 con 0x7f7f440d05c0 2024-03-06T07:38:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f48a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f440d1ba0 con 0x7f7f440cf410 2024-03-06T07:38:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f48a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440d1e80 con 0x7f7f4415e070 2024-03-06T07:38:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f3aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7f440cf410 0x7f7f440cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f3aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7f440cf410 0x7f7f440cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43152/0 (socket says 172.21.15.32:43152) 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.891+0000 7f7f3aa00640 1 -- 172.21.15.32:0/887318205 learned_addr learned my addr 172.21.15.32:0/887318205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.892+0000 7f7f43400640 1 --2- 172.21.15.32:0/887318205 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f440d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.892+0000 7f7f3aa00640 1 -- 172.21.15.32:0/887318205 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 msgr2=0x7f7f440d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.892+0000 7f7f3aa00640 1 --2- 172.21.15.32:0/887318205 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f440d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.892+0000 7f7f3aa00640 1 -- 172.21.15.32:0/887318205 >> v1:172.21.15.32:6789/0 conn(0x7f7f440d05c0 legacy=0x7f7f440b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.892+0000 7f7f3aa00640 1 -- 172.21.15.32:0/887318205 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f440d2160 con 0x7f7f440cf410 2024-03-06T07:38:17.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.893+0000 7f7f3aa00640 1 --2- 172.21.15.32:0/887318205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440cf870 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009870 tx=0x7f7f2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52617aeb32152407 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:17.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.893+0000 7f7f42a00640 1 -- 172.21.15.32:0/887318205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c034070 con 0x7f7f440cf410 2024-03-06T07:38:17.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.893+0000 7f7f42a00640 1 -- 172.21.15.32:0/887318205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c030840 con 0x7f7f440cf410 2024-03-06T07:38:17.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.893+0000 7f7f42a00640 1 -- 172.21.15.32:0/887318205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c030c10 con 0x7f7f440cf410 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 -- 172.21.15.32:0/887318205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 msgr2=0x7f7f440cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 --2- 172.21.15.32:0/887318205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440cf870 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009870 tx=0x7f7f2c02b980 comp rx=0 tx=0).stop 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 -- 172.21.15.32:0/887318205 shutdown_connections 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 --2- 172.21.15.32:0/887318205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440cf870 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 --2- 172.21.15.32:0/887318205 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f440d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 -- 172.21.15.32:0/887318205 >> 172.21.15.32:0/887318205 conn(0x7f7f4408d2a0 msgr2=0x7f7f440d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 -- 172.21.15.32:0/887318205 shutdown_connections 2024-03-06T07:38:17.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.894+0000 7f7f48a00640 1 -- 172.21.15.32:0/887318205 wait complete. 2024-03-06T07:38:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.896+0000 7f7f48a00640 1 Processor -- start 2024-03-06T07:38:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.896+0000 7f7f48a00640 1 -- start start 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f48a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f48a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f440d05c0 0x7f7f44158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f48a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f441586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f48a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440d4730 con 0x7f7f440cf410 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f48a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440d45b0 con 0x7f7f440d05c0 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f48a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440d0a50 con 0x7f7f4415e070 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f43400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f43400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43168/0 (socket says 172.21.15.32:43168) 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.897+0000 7f7f43400640 1 -- 172.21.15.32:0/763804862 learned_addr learned my addr 172.21.15.32:0/763804862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f3aa00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f440d05c0 0x7f7f44158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f43400640 1 -- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 msgr2=0x7f7f441586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f43400640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f441586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f43400640 1 -- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f440d05c0 msgr2=0x7f7f44158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f43400640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f440d05c0 0x7f7f44158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f43400640 1 -- 172.21.15.32:0/763804862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f441f7dc0 con 0x7f7f440cf410 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.898+0000 7f7f43400640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440b1ef0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400d5f0 tx=0x7f7f3400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.899+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f340125f0 con 0x7f7f440cf410 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.899+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f34019070 con 0x7f7f440cf410 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.899+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f340129d0 con 0x7f7f440cf410 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.899+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f441f80a0 con 0x7f7f440cf410 2024-03-06T07:38:17.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.899+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4415e8d0 con 0x7f7f440cf410 2024-03-06T07:38:17.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.900+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4404f330 con 0x7f7f440cf410 2024-03-06T07:38:17.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.901+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f34012b70 con 0x7f7f440cf410 2024-03-06T07:38:17.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.902+0000 7f7f3be00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f2006b4d0 0x7f7f2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:17.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.902+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f3408a6d0 con 0x7f7f440cf410 2024-03-06T07:38:17.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.903+0000 7f7f3aa00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f2006b4d0 0x7f7f2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:17.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.903+0000 7f7f3aa00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f2006b4d0 0x7f7f2006d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c036000 tx=0x7f7f2c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:17.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:17.907+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f34059470 con 0x7f7f440cf410 2024-03-06T07:38:18.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.147+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f440d1370 con 0x7f7f440cf410 2024-03-06T07:38:18.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.148+0000 7f7f3be00640 1 -- 172.21.15.32:0/763804862 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f7f34058d70 con 0x7f7f440cf410 2024-03-06T07:38:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:18.151 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:18.151 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f2006b4d0 msgr2=0x7f7f2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f2006b4d0 0x7f7f2006d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c036000 tx=0x7f7f2c03b040 comp rx=0 tx=0).stop 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 msgr2=0x7f7f440b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440b1ef0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400d5f0 tx=0x7f7f3400dab0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 shutdown_connections 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7f2006b4d0 0x7f7f2006d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7f4415e070 0x7f7f441586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f440d05c0 0x7f7f44158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 --2- 172.21.15.32:0/763804862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440cf410 0x7f7f440b1ef0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 >> 172.21.15.32:0/763804862 conn(0x7f7f4408d2a0 msgr2=0x7f7f440b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 shutdown_connections 2024-03-06T07:38:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:18.151+0000 7f7f48a00640 1 -- 172.21.15.32:0/763804862 wait complete. 2024-03-06T07:38:18.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:18.372 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:19.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:19.381 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:19.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.593+0000 7f8b8f200640 1 Processor -- start 2024-03-06T07:38:19.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.593+0000 7f8b8f200640 1 -- start start 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.595+0000 7f8b8f200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b900b1280 0x7f8b9006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.595+0000 7f8b8f200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b900b6270 con 0x7f8b900b0d30 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.595+0000 7f8b8f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b900b6510 con 0x7f8b9015df00 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.595+0000 7f8b8f200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b900aeb00 con 0x7f8b900b1280 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.595+0000 7f8b85400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b900b1280 0x7f8b9006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.596+0000 7f8b85400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b900b1280 0x7f8b9006cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43174/0 (socket says 172.21.15.32:43174) 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.596+0000 7f8b8e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8b900b0d30 0x7f8b900af270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35232/0 (socket says 172.21.15.32:35232) 2024-03-06T07:38:19.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.596+0000 7f8b85400640 1 -- 172.21.15.32:0/2341687114 learned_addr learned my addr 172.21.15.32:0/2341687114 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:19.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.596+0000 7f8b85400640 1 -- 172.21.15.32:0/2341687114 >> v1:172.21.15.39:6789/0 conn(0x7f8b9015df00 legacy=0x7f8b9006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:19.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.596+0000 7f8b85400640 1 -- 172.21.15.32:0/2341687114 >> v1:172.21.15.32:6789/0 conn(0x7f8b900b0d30 legacy=0x7f8b900af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.596+0000 7f8b85400640 1 -- 172.21.15.32:0/2341687114 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b900b67f0 con 0x7f8b900b1280 2024-03-06T07:38:19.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.597+0000 7f8b85400640 1 --2- 172.21.15.32:0/2341687114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b1280 0x7f8b9006cf90 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009870 tx=0x7f8b7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b92c05570d9b3ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:19.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.597+0000 7f8b8d400640 1 -- 172.21.15.32:0/2341687114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78035020 con 0x7f8b900b1280 2024-03-06T07:38:19.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8d400640 1 -- 172.21.15.32:0/2341687114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78031870 con 0x7f8b900b1280 2024-03-06T07:38:19.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8d400640 1 -- 172.21.15.32:0/2341687114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78031c40 con 0x7f8b900b1280 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 -- 172.21.15.32:0/2341687114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b1280 msgr2=0x7f8b9006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2341687114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b1280 0x7f8b9006cf90 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009870 tx=0x7f8b7802b910 comp rx=0 tx=0).stop 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 -- 172.21.15.32:0/2341687114 shutdown_connections 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2341687114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b1280 0x7f8b9006cf90 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 -- 172.21.15.32:0/2341687114 >> 172.21.15.32:0/2341687114 conn(0x7f8b9009aa10 msgr2=0x7f8b9009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 -- 172.21.15.32:0/2341687114 shutdown_connections 2024-03-06T07:38:19.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.598+0000 7f8b8f200640 1 -- 172.21.15.32:0/2341687114 wait complete. 2024-03-06T07:38:19.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.600+0000 7f8b8f200640 1 Processor -- start 2024-03-06T07:38:19.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.600+0000 7f8b8f200640 1 -- start start 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 0x7f8b901588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b900b1280 0x7f8b900b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b9015df00 0x7f8b9006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9015e460 con 0x7f8b900b0d30 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9015e2e0 con 0x7f8b9015df00 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900af980 con 0x7f8b900b1280 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.601+0000 7f8b8de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 0x7f8b901588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 0x7f8b901588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43178/0 (socket says 172.21.15.32:43178) 2024-03-06T07:38:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 -- 172.21.15.32:0/2071656534 learned_addr learned my addr 172.21.15.32:0/2071656534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8e800640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b9015df00 0x7f8b9006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 -- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b900b1280 msgr2=0x7f8b900b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b85400640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b900b1280 0x7f8b900b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b900b1280 0x7f8b900b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 -- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b9015df00 msgr2=0x7f8b9006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b9015df00 0x7f8b9006f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 -- 172.21.15.32:0/2071656534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b900b1e60 con 0x7f8b900b0d30 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8e800640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b9015df00 0x7f8b9006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.602+0000 7f8b8de00640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 0x7f8b901588f0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000b3f0 tx=0x7f8b8000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.603+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80015020 con 0x7f8b900b0d30 2024-03-06T07:38:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.603+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b80011830 con 0x7f8b900b0d30 2024-03-06T07:38:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.603+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80011c20 con 0x7f8b900b0d30 2024-03-06T07:38:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.603+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b90158ed0 con 0x7f8b900b0d30 2024-03-06T07:38:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.603+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b90159490 con 0x7f8b900b0d30 2024-03-06T07:38:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.604+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9004f330 con 0x7f8b900b0d30 2024-03-06T07:38:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.605+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b80018020 con 0x7f8b900b0d30 2024-03-06T07:38:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.606+0000 7f8b86a00640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b6c06b4d0 0x7f8b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.606+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8b80089ee0 con 0x7f8b900b0d30 2024-03-06T07:38:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.606+0000 7f8b85400640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b6c06b4d0 0x7f8b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.607+0000 7f8b85400640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b6c06b4d0 0x7f8b6c06d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8b78036000 tx=0x7f8b78008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:19.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.610+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b8001f030 con 0x7f8b900b0d30 2024-03-06T07:38:19.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.853+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b90157b10 con 0x7f8b900b0d30 2024-03-06T07:38:19.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.854+0000 7f8b86a00640 1 -- 172.21.15.32:0/2071656534 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f8b80058c80 con 0x7f8b900b0d30 2024-03-06T07:38:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:19.857 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:19.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.857+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b6c06b4d0 msgr2=0x7f8b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.857+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b6c06b4d0 0x7f8b6c06d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8b78036000 tx=0x7f8b78008030 comp rx=0 tx=0).stop 2024-03-06T07:38:19.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.857+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 msgr2=0x7f8b901588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.857+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 0x7f8b901588f0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000b3f0 tx=0x7f8b8000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 shutdown_connections 2024-03-06T07:38:19.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8b6c06b4d0 0x7f8b6c06d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b9015df00 0x7f8b9006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8b900b1280 0x7f8b900b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 --2- 172.21.15.32:0/2071656534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900b0d30 0x7f8b901588f0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 >> 172.21.15.32:0/2071656534 conn(0x7f8b9009aa10 msgr2=0x7f8b900702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 shutdown_connections 2024-03-06T07:38:19.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:19.858+0000 7f8b8f200640 1 -- 172.21.15.32:0/2071656534 wait complete. 2024-03-06T07:38:20.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:20.080 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:21.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:21.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:21.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:21.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.370+0000 7f24ce400640 1 Processor -- start 2024-03-06T07:38:21.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.371+0000 7f24ce400640 1 -- start start 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.372+0000 7f24ce400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24c8145300 0x7f24c80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.372+0000 7f24ce400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f24c8070f10 0x7f24c80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.372+0000 7f24ce400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c80714d0 con 0x7f24c80cce90 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.372+0000 7f24ce400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c80cd300 con 0x7f24c8145300 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.372+0000 7f24ce400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c80717b0 con 0x7f24c8070f10 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24c8145300 0x7f24c80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24c8145300 0x7f24c80b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43186/0 (socket says 172.21.15.32:43186) 2024-03-06T07:38:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 -- 172.21.15.32:0/2596992870 learned_addr learned my addr 172.21.15.32:0/2596992870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 -- 172.21.15.32:0/2596992870 >> v2:172.21.15.39:3300/0 conn(0x7f24c8070f10 msgr2=0x7f24c80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24c7e00640 1 --2- 172.21.15.32:0/2596992870 >> v2:172.21.15.39:3300/0 conn(0x7f24c8070f10 0x7f24c80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 --2- 172.21.15.32:0/2596992870 >> v2:172.21.15.39:3300/0 conn(0x7f24c8070f10 0x7f24c80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 -- 172.21.15.32:0/2596992870 >> v1:172.21.15.39:6789/0 conn(0x7f24c80cce90 legacy=0x7f24c80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24cd000640 1 -- 172.21.15.32:0/2596992870 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c8071a90 con 0x7f24c8145300 2024-03-06T07:38:21.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.373+0000 7f24c7e00640 1 --2- 172.21.15.32:0/2596992870 >> v2:172.21.15.39:3300/0 conn(0x7f24c8070f10 0x7f24c80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:21.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.374+0000 7f24cd000640 1 --2- 172.21.15.32:0/2596992870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8145300 0x7f24c80b49a0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f24c80b5150 tx=0x7f24bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bff96470204361 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24c7400640 1 -- 172.21.15.32:0/2596992870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc02e070 con 0x7f24c8145300 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24c7400640 1 -- 172.21.15.32:0/2596992870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc037070 con 0x7f24c8145300 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24c7400640 1 -- 172.21.15.32:0/2596992870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc031a00 con 0x7f24c8145300 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 -- 172.21.15.32:0/2596992870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8145300 msgr2=0x7f24c80b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 --2- 172.21.15.32:0/2596992870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8145300 0x7f24c80b49a0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f24c80b5150 tx=0x7f24bc02b880 comp rx=0 tx=0).stop 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 -- 172.21.15.32:0/2596992870 shutdown_connections 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 --2- 172.21.15.32:0/2596992870 >> v2:172.21.15.39:3300/0 conn(0x7f24c8070f10 0x7f24c80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 --2- 172.21.15.32:0/2596992870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8145300 0x7f24c80b49a0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 -- 172.21.15.32:0/2596992870 >> 172.21.15.32:0/2596992870 conn(0x7f24c806b3d0 msgr2=0x7f24c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 -- 172.21.15.32:0/2596992870 shutdown_connections 2024-03-06T07:38:21.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.375+0000 7f24ce400640 1 -- 172.21.15.32:0/2596992870 wait complete. 2024-03-06T07:38:21.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.377+0000 7f24ce400640 1 Processor -- start 2024-03-06T07:38:21.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.377+0000 7f24ce400640 1 -- start start 2024-03-06T07:38:21.380 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:21.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24ce400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 0x7f24c815e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24ce400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f24c80cce90 0x7f24c815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24ce400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24c8145300 0x7f24c815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24ce400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c81456e0 con 0x7f24c8070f10 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24ce400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80b4ee0 con 0x7f24c8145300 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24ce400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c815d580 con 0x7f24c80cce90 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24cd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 0x7f24c815e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24cd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 0x7f24c815e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43194/0 (socket says 172.21.15.32:43194) 2024-03-06T07:38:21.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.378+0000 7f24cd000640 1 -- 172.21.15.32:0/2164101049 learned_addr learned my addr 172.21.15.32:0/2164101049 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24c7e00640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f24c80cce90 0x7f24c815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cd000640 1 -- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f24c80cce90 msgr2=0x7f24c815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cda00640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24c8145300 0x7f24c815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cd000640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f24c80cce90 0x7f24c815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cd000640 1 -- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24c8145300 msgr2=0x7f24c815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cd000640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24c8145300 0x7f24c815b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cd000640 1 -- 172.21.15.32:0/2164101049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c815b910 con 0x7f24c8070f10 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24c7e00640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f24c80cce90 0x7f24c815ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.379+0000 7f24cd000640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 0x7f24c815e940 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f24bc000c00 tx=0x7f24bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.380+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc032880 con 0x7f24c8070f10 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.380+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc032a20 con 0x7f24c8070f10 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.380+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c813f000 con 0x7f24c8070f10 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.380+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc032e10 con 0x7f24c8070f10 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.380+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c81f7670 con 0x7f24c8070f10 2024-03-06T07:38:21.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.381+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c804f330 con 0x7f24c8070f10 2024-03-06T07:38:21.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.382+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24bc0453f0 con 0x7f24c8070f10 2024-03-06T07:38:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.383+0000 7f24c4c00640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f24ac06b520 0x7f24ac06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.383+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f24bc0ac320 con 0x7f24c8070f10 2024-03-06T07:38:21.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.383+0000 7f24c7e00640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f24ac06b520 0x7f24ac06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.384+0000 7f24c7e00640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f24ac06b520 0x7f24ac06d9e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f24b8006fd0 tx=0x7f24b8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.388+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24bc040070 con 0x7f24c8070f10 2024-03-06T07:38:21.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.725+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24c80ae180 con 0x7f24c8070f10 2024-03-06T07:38:21.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.726+0000 7f24c4c00640 1 -- 172.21.15.32:0/2164101049 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f24bc07b1d0 con 0x7f24c8070f10 2024-03-06T07:38:21.728 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:21.728 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:21.729 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:21.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f24ac06b520 msgr2=0x7f24ac06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f24ac06b520 0x7f24ac06d9e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f24b8006fd0 tx=0x7f24b8008040 comp rx=0 tx=0).stop 2024-03-06T07:38:21.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 msgr2=0x7f24c815e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 0x7f24c815e940 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f24bc000c00 tx=0x7f24bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 shutdown_connections 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f24ac06b520 0x7f24ac06d9e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24c8145300 0x7f24c815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f24c80cce90 0x7f24c815ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 --2- 172.21.15.32:0/2164101049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c8070f10 0x7f24c815e940 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 >> 172.21.15.32:0/2164101049 conn(0x7f24c806b3d0 msgr2=0x7f24c80cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 shutdown_connections 2024-03-06T07:38:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:21.729+0000 7f24ce400640 1 -- 172.21.15.32:0/2164101049 wait complete. 2024-03-06T07:38:21.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:21.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:22.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.215+0000 7f1a31a00640 1 Processor -- start 2024-03-06T07:38:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.215+0000 7f1a31a00640 1 -- start start 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a31a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1a2c0b3260 0x7f1a2c0d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a31a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a2c0d0ff0 0x7f1a2c0b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a31a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0d15b0 con 0x7f1a2c15e240 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a31a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0d1850 con 0x7f1a2c0d0ff0 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a31a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0d2640 con 0x7f1a2c0b3260 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a2b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a2c0d0ff0 0x7f1a2c0b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a2b400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a2c0d0ff0 0x7f1a2c0b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43202/0 (socket says 172.21.15.32:43202) 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.218+0000 7f1a2b400640 1 -- 172.21.15.32:0/2021738008 learned_addr learned my addr 172.21.15.32:0/2021738008 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.219+0000 7f1a2a000640 1 --2- 172.21.15.32:0/2021738008 >> v2:172.21.15.39:3300/0 conn(0x7f1a2c0b3260 0x7f1a2c0d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.219+0000 7f1a2b400640 1 -- 172.21.15.32:0/2021738008 >> v2:172.21.15.39:3300/0 conn(0x7f1a2c0b3260 msgr2=0x7f1a2c0d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.219+0000 7f1a2b400640 1 --2- 172.21.15.32:0/2021738008 >> v2:172.21.15.39:3300/0 conn(0x7f1a2c0b3260 0x7f1a2c0d4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.219+0000 7f1a2b400640 1 -- 172.21.15.32:0/2021738008 >> v1:172.21.15.39:6789/0 conn(0x7f1a2c15e240 legacy=0x7f1a2c0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.219+0000 7f1a2b400640 1 -- 172.21.15.32:0/2021738008 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0d2920 con 0x7f1a2c0d0ff0 2024-03-06T07:38:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.220+0000 7f1a2b400640 1 --2- 172.21.15.32:0/2021738008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b3920 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c06b9e0 tx=0x7f1a2402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2c90a63ce76335 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.220+0000 7f1a29600640 1 -- 172.21.15.32:0/2021738008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2402c9f0 con 0x7f1a2c0d0ff0 2024-03-06T07:38:23.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a29600640 1 -- 172.21.15.32:0/2021738008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2402cb90 con 0x7f1a2c0d0ff0 2024-03-06T07:38:23.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a29600640 1 -- 172.21.15.32:0/2021738008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240343f0 con 0x7f1a2c0d0ff0 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 -- 172.21.15.32:0/2021738008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c0d0ff0 msgr2=0x7f1a2c0b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 --2- 172.21.15.32:0/2021738008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b3920 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c06b9e0 tx=0x7f1a2402be50 comp rx=0 tx=0).stop 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 -- 172.21.15.32:0/2021738008 shutdown_connections 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 --2- 172.21.15.32:0/2021738008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b3920 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 --2- 172.21.15.32:0/2021738008 >> v2:172.21.15.39:3300/0 conn(0x7f1a2c0b3260 0x7f1a2c0d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 -- 172.21.15.32:0/2021738008 >> 172.21.15.32:0/2021738008 conn(0x7f1a2c0d3f30 msgr2=0x7f1a2c0d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 -- 172.21.15.32:0/2021738008 shutdown_connections 2024-03-06T07:38:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.221+0000 7f1a31a00640 1 -- 172.21.15.32:0/2021738008 wait complete. 2024-03-06T07:38:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.224+0000 7f1a31a00640 1 Processor -- start 2024-03-06T07:38:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.224+0000 7f1a31a00640 1 -- start start 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a31a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a2c0b3260 0x7f1a2c1544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a31a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a31a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 0x7f1a2c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a31a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0b3e60 con 0x7f1a2c15e240 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a31a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c15e620 con 0x7f1a2c0b3260 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a31a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0b3fe0 con 0x7f1a2c0d0ff0 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a2b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 0x7f1a2c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a2b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 0x7f1a2c0d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43210/0 (socket says 172.21.15.32:43210) 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.226+0000 7f1a2b400640 1 -- 172.21.15.32:0/1421566955 learned_addr learned my addr 172.21.15.32:0/1421566955 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2aa00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a2c0b3260 0x7f1a2c1544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2a000640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2b400640 1 -- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a2c0d0ff0 msgr2=0x7f1a2c0b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2b400640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2b400640 1 -- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a2c0b3260 msgr2=0x7f1a2c1544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2b400640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a2c0b3260 0x7f1a2c1544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2b400640 1 -- 172.21.15.32:0/1421566955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c154d10 con 0x7f1a2c15e240 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2a000640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.227+0000 7f1a2b400640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 0x7f1a2c0d0060 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f1a24036040 tx=0x7f1a24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.228+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24043020 con 0x7f1a2c15e240 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.228+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2403e450 con 0x7f1a2c15e240 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.228+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24009510 con 0x7f1a2c15e240 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.228+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c156050 con 0x7f1a2c15e240 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.228+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c155290 con 0x7f1a2c15e240 2024-03-06T07:38:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.229+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c04f3a0 con 0x7f1a2c15e240 2024-03-06T07:38:23.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.230+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a24046020 con 0x7f1a2c15e240 2024-03-06T07:38:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.231+0000 7f1a22a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.232+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1a240acaa0 con 0x7f1a2c15e240 2024-03-06T07:38:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.232+0000 7f1a2aa00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.233+0000 7f1a2aa00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c006fd0 tx=0x7f1a1c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.236+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a2404d030 con 0x7f1a2c15e240 2024-03-06T07:38:23.382 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.476+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a2c0d39c0 con 0x7f1a2c15e240 2024-03-06T07:38:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.477+0000 7f1a22a00640 1 -- 172.21.15.32:0/1421566955 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f1a2407b9f0 con 0x7f1a2c15e240 2024-03-06T07:38:23.479 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:23.479 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.479+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a0c06b4d0 msgr2=0x7f1a0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.479+0000 7f1a31a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c006fd0 tx=0x7f1a1c008040 comp rx=0 tx=0).stop 2024-03-06T07:38:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.479+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 msgr2=0x7f1a2c0d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.479+0000 7f1a31a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 0x7f1a2c0d0060 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f1a24036040 tx=0x7f1a24004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 shutdown_connections 2024-03-06T07:38:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a2c15e240 0x7f1a2c0d0060 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1a2c0d0ff0 0x7f1a2c0b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 --2- 172.21.15.32:0/1421566955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a2c0b3260 0x7f1a2c1544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 >> 172.21.15.32:0/1421566955 conn(0x7f1a2c0d3f30 msgr2=0x7f1a2c156dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 shutdown_connections 2024-03-06T07:38:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:23.480+0000 7f1a31a00640 1 -- 172.21.15.32:0/1421566955 wait complete. 2024-03-06T07:38:23.707 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:23.707 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:24.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:24.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.932+0000 7f0a17200640 1 Processor -- start 2024-03-06T07:38:24.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.933+0000 7f0a17200640 1 -- start start 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.934+0000 7f0a17200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 0x7f0a10168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.934+0000 7f0a17200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a100b3710 con 0x7f0a100d4a10 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.934+0000 7f0a17200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a100b39f0 con 0x7f0a1015e0b0 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.934+0000 7f0a17200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a100b09e0 con 0x7f0a100d0e50 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.935+0000 7f0a15400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0a100d4a10 0x7f0a100d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35234/0 (socket says 172.21.15.32:35234) 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.935+0000 7f0a15400640 1 -- 172.21.15.32:0/1617647956 learned_addr learned my addr 172.21.15.32:0/1617647956 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:24.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.935+0000 7f0a16800640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 0x7f0a10168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.935+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822222578 0 0) 0x7f0a100b3710 con 0x7f0a100d4a10 2024-03-06T07:38:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.935+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09fc003520 con 0x7f0a100d4a10 2024-03-06T07:38:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 363788281 0 0) 0x7f09fc003520 con 0x7f0a100d4a10 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 msgr2=0x7f0a10168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 0x7f0a10168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 >> v1:172.21.15.39:6789/0 conn(0x7f0a1015e0b0 legacy=0x7f0a100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100b3cd0 con 0x7f0a100d4a10 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f0a00001690 con 0x7f0a100d4a10 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 >> v1:172.21.15.32:6789/0 conn(0x7f0a100d4a10 legacy=0x7f0a100d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09fc0039e0 0x7f09fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 0x7f09fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc004d40 0x7f09fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00b9c0 con 0x7f09fc0039e0 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00bb70 con 0x7f09fc004d40 2024-03-06T07:38:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00bd70 con 0x7f0a100d0e50 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.936+0000 7f0a15e00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09fc0039e0 0x7f09fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.937+0000 7f0a16800640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc004d40 0x7f09fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.937+0000 7f0a15e00640 1 -- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 msgr2=0x7f09fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.937+0000 7f0a15e00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a100d0e50 0x7f09fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.937+0000 7f0a15e00640 1 -- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc004d40 msgr2=0x7f09fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.937+0000 7f0a15e00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc004d40 0x7f09fc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:24.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.937+0000 7f0a15e00640 1 -- 172.21.15.32:0/1617647956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc003520 con 0x7f09fc0039e0 2024-03-06T07:38:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.938+0000 7f0a15e00640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09fc0039e0 0x7f09fc003dc0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f0a08001f60 tx=0x7f09fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ac459d756f8730 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.938+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08001b30 con 0x7f09fc0039e0 2024-03-06T07:38:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.938+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0802d690 con 0x7f09fc0039e0 2024-03-06T07:38:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.938+0000 7f0a14a00640 1 -- 172.21.15.32:0/1617647956 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0802da80 con 0x7f09fc0039e0 2024-03-06T07:38:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a17200640 1 -- 172.21.15.32:0/1617647956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09fc0039e0 msgr2=0x7f09fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a17200640 1 --2- 172.21.15.32:0/1617647956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09fc0039e0 0x7f09fc003dc0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f0a08001f60 tx=0x7f09fc005830 comp rx=0 tx=0).stop 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a16800640 1 -- 172.21.15.32:0/1617647956 reap_dead start 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a17200640 1 -- 172.21.15.32:0/1617647956 shutdown_connections 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a17200640 1 -- 172.21.15.32:0/1617647956 >> 172.21.15.32:0/1617647956 conn(0x7f0a100d3d30 msgr2=0x7f0a100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a17200640 1 -- 172.21.15.32:0/1617647956 shutdown_connections 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.939+0000 7f0a17200640 1 -- 172.21.15.32:0/1617647956 wait complete. 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.941+0000 7f0a17200640 1 Processor -- start 2024-03-06T07:38:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.941+0000 7f0a17200640 1 -- start start 2024-03-06T07:38:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a17200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc0039e0 0x7f0a100cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a17200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a10161220 0x7f0a10150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a17200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 0x7f0a100b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a17200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1015e490 con 0x7f0a100b8a50 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a17200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a100cfa20 con 0x7f09fc0039e0 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a17200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a100cfc60 con 0x7f0a10161220 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a16800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 0x7f0a100b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a16800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 0x7f0a100b8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43228/0 (socket says 172.21.15.32:43228) 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.942+0000 7f0a16800640 1 -- 172.21.15.32:0/3927011299 learned_addr learned my addr 172.21.15.32:0/3927011299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a15e00640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc0039e0 0x7f0a100cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a15400640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a10161220 0x7f0a10150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a16800640 1 -- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a10161220 msgr2=0x7f0a10150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a16800640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a10161220 0x7f0a10150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a16800640 1 -- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc0039e0 msgr2=0x7f0a100cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a16800640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc0039e0 0x7f0a100cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a16800640 1 -- 172.21.15.32:0/3927011299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100cdba0 con 0x7f0a100b8a50 2024-03-06T07:38:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a16800640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 0x7f0a100b8ed0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c010580 tx=0x7f0a0c010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.943+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c00f040 con 0x7f0a100b8a50 2024-03-06T07:38:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.944+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c01c070 con 0x7f0a100b8a50 2024-03-06T07:38:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.944+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c014b00 con 0x7f0a100b8a50 2024-03-06T07:38:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.944+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a100cde80 con 0x7f0a100b8a50 2024-03-06T07:38:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.944+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1015e8e0 con 0x7f0a100b8a50 2024-03-06T07:38:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.945+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a100b83b0 con 0x7f0a100b8a50 2024-03-06T07:38:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.946+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a0c014ca0 con 0x7f0a100b8a50 2024-03-06T07:38:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.947+0000 7f0a06000640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.947+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0a0c089840 con 0x7f0a100b8a50 2024-03-06T07:38:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.947+0000 7f0a15e00640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.948+0000 7f0a15e00640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09f406b4d0 0x7f09f406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0a08036000 tx=0x7f0a08031000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:24.956+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a0c025020 con 0x7f0a100b8a50 2024-03-06T07:38:25.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.205+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a1004f330 con 0x7f0a100b8a50 2024-03-06T07:38:25.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.206+0000 7f0a06000640 1 -- 172.21.15.32:0/3927011299 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f0a0c0585f0 con 0x7f0a100b8a50 2024-03-06T07:38:25.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:25.209 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:25.209 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09f406b4d0 msgr2=0x7f09f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09f406b4d0 0x7f09f406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0a08036000 tx=0x7f0a08031000 comp rx=0 tx=0).stop 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 msgr2=0x7f0a100b8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 0x7f0a100b8ed0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c010580 tx=0x7f0a0c010a40 comp rx=0 tx=0).stop 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 shutdown_connections 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a100b8a50 0x7f0a100b8ed0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0a10161220 0x7f0a10150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 --2- 172.21.15.32:0/3927011299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09fc0039e0 0x7f0a100cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.209+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 >> 172.21.15.32:0/3927011299 conn(0x7f0a100d3d30 msgr2=0x7f0a100b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.210+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 shutdown_connections 2024-03-06T07:38:25.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:25.210+0000 7f0a17200640 1 -- 172.21.15.32:0/3927011299 wait complete. 2024-03-06T07:38:25.384 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:25.433 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:25.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:26.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:26.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.657+0000 7fe461a00640 1 Processor -- start 2024-03-06T07:38:26.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.657+0000 7fe461a00640 1 -- start start 2024-03-06T07:38:26.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe461a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe45c0b0f90 0x7fe45c0cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe461a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c15d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe461a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c0b1370 con 0x7fe45c0cc730 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe461a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c0ccf80 con 0x7fe45c0b0f90 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe461a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0cd260 con 0x7fe45c0d07b0 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe45be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe45c0b0f90 0x7fe45c0cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe45be00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe45c0b0f90 0x7fe45c0cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43236/0 (socket says 172.21.15.32:43236) 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.659+0000 7fe45be00640 1 -- 172.21.15.32:0/2311545990 learned_addr learned my addr 172.21.15.32:0/2311545990 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.660+0000 7fe461000640 1 --2- 172.21.15.32:0/2311545990 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.660+0000 7fe45be00640 1 -- 172.21.15.32:0/2311545990 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 msgr2=0x7fe45c15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:26.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.660+0000 7fe45be00640 1 --2- 172.21.15.32:0/2311545990 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c15d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.660+0000 7fe45be00640 1 -- 172.21.15.32:0/2311545990 >> v1:172.21.15.39:6789/0 conn(0x7fe45c0cc730 legacy=0x7fe45c0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:26.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.660+0000 7fe45be00640 1 -- 172.21.15.32:0/2311545990 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0cd540 con 0x7fe45c0b0f90 2024-03-06T07:38:26.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.662+0000 7fe45be00640 1 --2- 172.21.15.32:0/2311545990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0b0f90 0x7fe45c0cf080 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009e50 tx=0x7fe44c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d73cab984374f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:26.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.662+0000 7fe45aa00640 1 -- 172.21.15.32:0/2311545990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c02c9e0 con 0x7fe45c0b0f90 2024-03-06T07:38:26.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.662+0000 7fe45aa00640 1 -- 172.21.15.32:0/2311545990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c02cb80 con 0x7fe45c0b0f90 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.662+0000 7fe45aa00640 1 -- 172.21.15.32:0/2311545990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0343f0 con 0x7fe45c0b0f90 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 -- 172.21.15.32:0/2311545990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0b0f90 msgr2=0x7fe45c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 --2- 172.21.15.32:0/2311545990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0b0f90 0x7fe45c0cf080 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009e50 tx=0x7fe44c02be40 comp rx=0 tx=0).stop 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 -- 172.21.15.32:0/2311545990 shutdown_connections 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 --2- 172.21.15.32:0/2311545990 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 --2- 172.21.15.32:0/2311545990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0b0f90 0x7fe45c0cf080 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 -- 172.21.15.32:0/2311545990 >> 172.21.15.32:0/2311545990 conn(0x7fe45c0d1ae0 msgr2=0x7fe45c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 -- 172.21.15.32:0/2311545990 shutdown_connections 2024-03-06T07:38:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.663+0000 7fe461a00640 1 -- 172.21.15.32:0/2311545990 wait complete. 2024-03-06T07:38:26.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.665+0000 7fe461a00640 1 Processor -- start 2024-03-06T07:38:26.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.665+0000 7fe461a00640 1 -- start start 2024-03-06T07:38:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe461a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe45c0b0f90 0x7fe45c14ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe461a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 0x7fe45c15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe461a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe461a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0ccd20 con 0x7fe45c0cc730 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe461a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0ccba0 con 0x7fe45c0b0f90 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe461a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c15dc90 con 0x7fe45c0d07b0 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe45b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 0x7fe45c15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe45b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 0x7fe45c15ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43238/0 (socket says 172.21.15.32:43238) 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.666+0000 7fe45b400640 1 -- 172.21.15.32:0/2070415935 learned_addr learned my addr 172.21.15.32:0/2070415935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45be00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe45c0b0f90 0x7fe45c14ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe461000640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45b400640 1 -- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 msgr2=0x7fe45c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45b400640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45b400640 1 -- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe45c0b0f90 msgr2=0x7fe45c14ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45b400640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe45c0b0f90 0x7fe45c14ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45b400640 1 -- 172.21.15.32:0/2070415935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c15c830 con 0x7fe45c0cc730 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe461000640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe45b400640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 0x7fe45c15ba90 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c460 tx=0x7fe44400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444016020 con 0x7fe45c0cc730 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe444012870 con 0x7fe45c0cc730 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444012c60 con 0x7fe45c0cc730 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.667+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c15c550 con 0x7fe45c0cc730 2024-03-06T07:38:26.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.668+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c15e2b0 con 0x7fe45c0cc730 2024-03-06T07:38:26.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.669+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45c04f330 con 0x7fe45c0cc730 2024-03-06T07:38:26.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.670+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe444019020 con 0x7fe45c0cc730 2024-03-06T07:38:26.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.671+0000 7fe453e00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.671+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe444089fe0 con 0x7fe45c0cc730 2024-03-06T07:38:26.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.671+0000 7fe45be00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.672+0000 7fe45be00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe44c035000 tx=0x7fe44c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:26.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.680+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe444020030 con 0x7fe45c0cc730 2024-03-06T07:38:26.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.930+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe45c0d0080 con 0x7fe45c0cc730 2024-03-06T07:38:26.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.931+0000 7fe453e00640 1 -- 172.21.15.32:0/2070415935 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fe4440182c0 con 0x7fe45c0cc730 2024-03-06T07:38:26.933 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:26.933 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:26.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 msgr2=0x7fe43806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe44c035000 tx=0x7fe44c030000 comp rx=0 tx=0).stop 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 msgr2=0x7fe45c15ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 0x7fe45c15ba90 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c460 tx=0x7fe44400c920 comp rx=0 tx=0).stop 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 shutdown_connections 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe43806b4d0 0x7fe43806d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe45c0d07b0 0x7fe45c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe45c0cc730 0x7fe45c15ba90 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 --2- 172.21.15.32:0/2070415935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe45c0b0f90 0x7fe45c14ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 >> 172.21.15.32:0/2070415935 conn(0x7fe45c0d1ae0 msgr2=0x7fe45c15d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 shutdown_connections 2024-03-06T07:38:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:26.934+0000 7fe461a00640 1 -- 172.21.15.32:0/2070415935 wait complete. 2024-03-06T07:38:27.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:27.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:28.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:28.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.394+0000 7f43f2400640 1 Processor -- start 2024-03-06T07:38:28.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.394+0000 7f43f2400640 1 -- start start 2024-03-06T07:38:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43f2400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f43ec15df70 0x7f43ec070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43f2400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f43ec0b6230 0x7f43ec0b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43f2400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec162150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43f2400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec06fcb0 con 0x7f43ec15df70 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43f2400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec06f0a0 con 0x7f43ec0b6230 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43f2400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec06f380 con 0x7f43ec161cc0 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43eb400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f43ec15df70 0x7f43ec070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43eb400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f43ec15df70 0x7f43ec070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45440/0 (socket says 172.21.15.32:45440) 2024-03-06T07:38:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.396+0000 7f43eb400640 1 -- 172.21.15.32:0/3860456293 learned_addr learned my addr 172.21.15.32:0/3860456293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.397+0000 7f43eb400640 1 -- 172.21.15.32:0/3860456293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 msgr2=0x7f43ec162150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.397+0000 7f43eb400640 1 --2- 172.21.15.32:0/3860456293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec162150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.397+0000 7f43eb400640 1 -- 172.21.15.32:0/3860456293 >> v2:172.21.15.39:3300/0 conn(0x7f43ec0b6230 msgr2=0x7f43ec0b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.397+0000 7f43eb400640 1 --2- 172.21.15.32:0/3860456293 >> v2:172.21.15.39:3300/0 conn(0x7f43ec0b6230 0x7f43ec0b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.397+0000 7f43eb400640 1 -- 172.21.15.32:0/3860456293 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec06f660 con 0x7f43ec15df70 2024-03-06T07:38:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.398+0000 7f43eb400640 1 --2- 172.21.15.32:0/3860456293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec15df70 0x7f43ec070140 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0b17e0 tx=0x7f43dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f06ddda9b83a432 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.398+0000 7f43ea000640 1 -- 172.21.15.32:0/3860456293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc02ca80 con 0x7f43ec15df70 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.398+0000 7f43ea000640 1 -- 172.21.15.32:0/3860456293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc02cc20 con 0x7f43ec15df70 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.398+0000 7f43ea000640 1 -- 172.21.15.32:0/3860456293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0343f0 con 0x7f43ec15df70 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 -- 172.21.15.32:0/3860456293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec15df70 msgr2=0x7f43ec070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 --2- 172.21.15.32:0/3860456293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec15df70 0x7f43ec070140 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0b17e0 tx=0x7f43dc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 -- 172.21.15.32:0/3860456293 shutdown_connections 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 --2- 172.21.15.32:0/3860456293 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec162150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 --2- 172.21.15.32:0/3860456293 >> v2:172.21.15.39:3300/0 conn(0x7f43ec0b6230 0x7f43ec0b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 --2- 172.21.15.32:0/3860456293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec15df70 0x7f43ec070140 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 -- 172.21.15.32:0/3860456293 >> 172.21.15.32:0/3860456293 conn(0x7f43ec09aa10 msgr2=0x7f43ec09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 -- 172.21.15.32:0/3860456293 shutdown_connections 2024-03-06T07:38:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.399+0000 7f43f2400640 1 -- 172.21.15.32:0/3860456293 wait complete. 2024-03-06T07:38:28.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.401+0000 7f43f2400640 1 Processor -- start 2024-03-06T07:38:28.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.401+0000 7f43f2400640 1 -- start start 2024-03-06T07:38:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43f2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 0x7f43ec09ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43f2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43ec15df70 0x7f43ec160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43f2400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43f2400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec070800 con 0x7f43ec0b6230 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43f2400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec070680 con 0x7f43ec15df70 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43f2400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec06c0c0 con 0x7f43ec161cc0 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43eb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 0x7f43ec09ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43eb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 0x7f43ec09ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45450/0 (socket says 172.21.15.32:45450) 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43eb400640 1 -- 172.21.15.32:0/101503010 learned_addr learned my addr 172.21.15.32:0/101503010 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43eaa00640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43ec15df70 0x7f43ec160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.402+0000 7f43ebe00640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43eb400640 1 -- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 msgr2=0x7f43ec1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43eb400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43eb400640 1 -- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43ec15df70 msgr2=0x7f43ec160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43eb400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43ec15df70 0x7f43ec160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43eb400640 1 -- 172.21.15.32:0/101503010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec09c000 con 0x7f43ec0b6230 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43eb400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 0x7f43ec09ded0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0093e0 tx=0x7f43dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc043020 con 0x7f43ec0b6230 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc03e450 con 0x7f43ec0b6230 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc03e840 con 0x7f43ec0b6230 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.403+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec1f42c0 con 0x7f43ec0b6230 2024-03-06T07:38:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.404+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec1f4880 con 0x7f43ec0b6230 2024-03-06T07:38:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.405+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec09b670 con 0x7f43ec0b6230 2024-03-06T07:38:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.406+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43dc046020 con 0x7f43ec0b6230 2024-03-06T07:38:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.406+0000 7f43e3400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f43c806b4d0 0x7f43c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.407+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f43dc0acc80 con 0x7f43ec0b6230 2024-03-06T07:38:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.407+0000 7f43eaa00640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f43c806b4d0 0x7f43c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.407+0000 7f43eaa00640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f43c806b4d0 0x7f43c806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f43ec15d960 tx=0x7f43d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.411+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43dc04d030 con 0x7f43ec0b6230 2024-03-06T07:38:28.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.680+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43ec04f380 con 0x7f43ec0b6230 2024-03-06T07:38:28.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.681+0000 7f43e3400640 1 -- 172.21.15.32:0/101503010 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f43dc07bab0 con 0x7f43ec0b6230 2024-03-06T07:38:28.684 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:28.684 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f43c806b4d0 msgr2=0x7f43c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f43c806b4d0 0x7f43c806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f43ec15d960 tx=0x7f43d4008040 comp rx=0 tx=0).stop 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 msgr2=0x7f43ec09ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 0x7f43ec09ded0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0093e0 tx=0x7f43dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 shutdown_connections 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f43c806b4d0 0x7f43c806d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f43ec161cc0 0x7f43ec1608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43ec15df70 0x7f43ec160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 --2- 172.21.15.32:0/101503010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43ec0b6230 0x7f43ec09ded0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 >> 172.21.15.32:0/101503010 conn(0x7f43ec09aa10 msgr2=0x7f43ec0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 shutdown_connections 2024-03-06T07:38:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:28.684+0000 7f43f2400640 1 -- 172.21.15.32:0/101503010 wait complete. 2024-03-06T07:38:28.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:28.908 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:29.388 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:29.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.132+0000 7fd1e7200640 1 Processor -- start 2024-03-06T07:38:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.133+0000 7fd1e7200640 1 -- start start 2024-03-06T07:38:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1e7200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1e815e2a0 0x7fd1e80cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1e7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80d1120 con 0x7fd1e80cf460 2024-03-06T07:38:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1e7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80d1400 con 0x7fd1e815dec0 2024-03-06T07:38:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1e7200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e80b0ea0 con 0x7fd1e815e2a0 2024-03-06T07:38:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1dfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1e815e2a0 0x7fd1e80cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1e6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd1e80cf460 0x7fd1e80b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40948/0 (socket says 172.21.15.32:40948) 2024-03-06T07:38:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.134+0000 7fd1e6800640 1 -- 172.21.15.32:0/989260258 learned_addr learned my addr 172.21.15.32:0/989260258 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:30.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.135+0000 7fd1e5400640 1 -- 172.21.15.32:0/989260258 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014189242 0 0) 0x7fd1e80d1120 con 0x7fd1e80cf460 2024-03-06T07:38:30.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.135+0000 7fd1e5400640 1 -- 172.21.15.32:0/989260258 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc003520 con 0x7fd1e80cf460 2024-03-06T07:38:30.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.135+0000 7fd1dfe00640 1 -- 172.21.15.32:0/989260258 >> v1:172.21.15.39:6789/0 conn(0x7fd1e815dec0 legacy=0x7fd1e80ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.135+0000 7fd1dfe00640 1 -- 172.21.15.32:0/989260258 >> v1:172.21.15.32:6789/0 conn(0x7fd1e80cf460 legacy=0x7fd1e80b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.135+0000 7fd1dfe00640 1 -- 172.21.15.32:0/989260258 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80b0b30 con 0x7fd1e815e2a0 2024-03-06T07:38:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.139+0000 7fd1dfe00640 1 --2- 172.21.15.32:0/989260258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815e2a0 0x7fd1e80cef20 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00097a0 tx=0x7fd1d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e81fcfb9609dedb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.139+0000 7fd1e5400640 1 -- 172.21.15.32:0/989260258 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0034070 con 0x7fd1e815e2a0 2024-03-06T07:38:30.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.139+0000 7fd1e5400640 1 -- 172.21.15.32:0/989260258 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d0030610 con 0x7fd1e815e2a0 2024-03-06T07:38:30.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.139+0000 7fd1e5400640 1 -- 172.21.15.32:0/989260258 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00309c0 con 0x7fd1e815e2a0 2024-03-06T07:38:30.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 -- 172.21.15.32:0/989260258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815e2a0 msgr2=0x7fd1e80cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 --2- 172.21.15.32:0/989260258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815e2a0 0x7fd1e80cef20 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00097a0 tx=0x7fd1d002b960 comp rx=0 tx=0).stop 2024-03-06T07:38:30.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 -- 172.21.15.32:0/989260258 shutdown_connections 2024-03-06T07:38:30.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 --2- 172.21.15.32:0/989260258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815e2a0 0x7fd1e80cef20 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 -- 172.21.15.32:0/989260258 >> 172.21.15.32:0/989260258 conn(0x7fd1e80d3d40 msgr2=0x7fd1e80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 -- 172.21.15.32:0/989260258 shutdown_connections 2024-03-06T07:38:30.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.140+0000 7fd1e7200640 1 -- 172.21.15.32:0/989260258 wait complete. 2024-03-06T07:38:30.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.142+0000 7fd1e7200640 1 Processor -- start 2024-03-06T07:38:30.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.142+0000 7fd1e7200640 1 -- start start 2024-03-06T07:38:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1e7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80cf460 0x7fd1e80c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1e7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 0x7fd1e814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1e7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80c88c0 0x7fd1e80b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1e7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80d0ec0 con 0x7fd1e815dec0 2024-03-06T07:38:30.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1e7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80d0d40 con 0x7fd1e80cf460 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1e7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80b1a40 con 0x7fd1e80c88c0 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.143+0000 7fd1dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 0x7fd1e814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 0x7fd1e814db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45464/0 (socket says 172.21.15.32:45464) 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 -- 172.21.15.32:0/4169134726 learned_addr learned my addr 172.21.15.32:0/4169134726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80cf460 0x7fd1e80c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1e6800640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80c88c0 0x7fd1e80b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 -- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80c88c0 msgr2=0x7fd1e80b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80c88c0 0x7fd1e80b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 -- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80cf460 msgr2=0x7fd1e80c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80cf460 0x7fd1e80c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 -- 172.21.15.32:0/4169134726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80c8d50 con 0x7fd1e815dec0 2024-03-06T07:38:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.144+0000 7fd1dfe00640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 0x7fd1e814db00 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd1d002bef0 tx=0x7fd1d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.145+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0034040 con 0x7fd1e815dec0 2024-03-06T07:38:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.145+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d0030410 con 0x7fd1e815dec0 2024-03-06T07:38:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.145+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e80c9030 con 0x7fd1e815dec0 2024-03-06T07:38:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.145+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0006e10 con 0x7fd1e815dec0 2024-03-06T07:38:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.145+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80c95f0 con 0x7fd1e815dec0 2024-03-06T07:38:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.146+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e804f330 con 0x7fd1e815dec0 2024-03-06T07:38:30.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.147+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1d004c060 con 0x7fd1e815dec0 2024-03-06T07:38:30.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.148+0000 7fd1de000640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c406b4d0 0x7fd1c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.148+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1d00abde0 con 0x7fd1e815dec0 2024-03-06T07:38:30.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.149+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c406b4d0 0x7fd1c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.149+0000 7fd1e5e00640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c406b4d0 0x7fd1c406d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8000f60 tx=0x7fd1d8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.153+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1d00305e0 con 0x7fd1e815dec0 2024-03-06T07:38:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.392+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1e80cf900 con 0x7fd1e815dec0 2024-03-06T07:38:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.393+0000 7fd1de000640 1 -- 172.21.15.32:0/4169134726 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fd1d007aaf0 con 0x7fd1e815dec0 2024-03-06T07:38:30.396 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:30.396 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:30.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c406b4d0 msgr2=0x7fd1c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c406b4d0 0x7fd1c406d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8000f60 tx=0x7fd1d8009040 comp rx=0 tx=0).stop 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 msgr2=0x7fd1e814db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 0x7fd1e814db00 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd1d002bef0 tx=0x7fd1d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 shutdown_connections 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd1c406b4d0 0x7fd1c406d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1e80c88c0 0x7fd1e80b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1e815dec0 0x7fd1e814db00 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 --2- 172.21.15.32:0/4169134726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e80cf460 0x7fd1e80c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 >> 172.21.15.32:0/4169134726 conn(0x7fd1e80d3d40 msgr2=0x7fd1e80cdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 shutdown_connections 2024-03-06T07:38:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:30.396+0000 7fd1e7200640 1 -- 172.21.15.32:0/4169134726 wait complete. 2024-03-06T07:38:30.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:30.621 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:31.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.858+0000 7f0dc4400640 1 Processor -- start 2024-03-06T07:38:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.858+0000 7f0dc4400640 1 -- start start 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc4400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc15e010 0x7f0dbc0afae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc4400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dbc06e640 0x7f0dbc0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc4400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc15e3f0 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc4400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dbc06e440 con 0x7f0dbc06e640 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc4400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc06ee00 con 0x7f0dbc15e010 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dbc06e640 0x7f0dbc0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc1000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dbc06e640 0x7f0dbc0b0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45470/0 (socket says 172.21.15.32:45470) 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc1000640 1 -- 172.21.15.32:0/4237673345 learned_addr learned my addr 172.21.15.32:0/4237673345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:31.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc2400640 1 --1- 172.21.15.32:0/4237673345 >> v1:172.21.15.32:6789/0 conn(0x7f0dbc0d16d0 0x7f0dbc161b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40952/0 (socket says 172.21.15.32:40952) 2024-03-06T07:38:31.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.860+0000 7f0dc1a00640 1 --2- 172.21.15.32:0/4237673345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc15e010 0x7f0dbc0afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.861+0000 7f0dc1000640 1 -- 172.21.15.32:0/4237673345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc15e010 msgr2=0x7f0dbc0afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:31.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.861+0000 7f0dc1000640 1 --2- 172.21.15.32:0/4237673345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc15e010 0x7f0dbc0afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.861+0000 7f0dc1000640 1 -- 172.21.15.32:0/4237673345 >> v1:172.21.15.32:6789/0 conn(0x7f0dbc0d16d0 legacy=0x7f0dbc161b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.861+0000 7f0dc1000640 1 -- 172.21.15.32:0/4237673345 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc06eb20 con 0x7f0dbc06e640 2024-03-06T07:38:31.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.862+0000 7f0dc1000640 1 --2- 172.21.15.32:0/4237673345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc06e640 0x7f0dbc0b0020 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009870 tx=0x7f0da802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0eb8c110cc6e8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:31.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.862+0000 7f0dbbe00640 1 -- 172.21.15.32:0/4237673345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8034070 con 0x7f0dbc06e640 2024-03-06T07:38:31.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.862+0000 7f0dbbe00640 1 -- 172.21.15.32:0/4237673345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8030840 con 0x7f0dbc06e640 2024-03-06T07:38:31.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.862+0000 7f0dbbe00640 1 -- 172.21.15.32:0/4237673345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8030c10 con 0x7f0dbc06e640 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 -- 172.21.15.32:0/4237673345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc06e640 msgr2=0x7f0dbc0b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 --2- 172.21.15.32:0/4237673345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc06e640 0x7f0dbc0b0020 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009870 tx=0x7f0da802b980 comp rx=0 tx=0).stop 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 -- 172.21.15.32:0/4237673345 shutdown_connections 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 --2- 172.21.15.32:0/4237673345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc06e640 0x7f0dbc0b0020 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 --2- 172.21.15.32:0/4237673345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc15e010 0x7f0dbc0afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 -- 172.21.15.32:0/4237673345 >> 172.21.15.32:0/4237673345 conn(0x7f0dbc0d0bb0 msgr2=0x7f0dbc0d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 -- 172.21.15.32:0/4237673345 shutdown_connections 2024-03-06T07:38:31.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.863+0000 7f0dc4400640 1 -- 172.21.15.32:0/4237673345 wait complete. 2024-03-06T07:38:31.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.866+0000 7f0dc4400640 1 Processor -- start 2024-03-06T07:38:31.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.866+0000 7f0dc4400640 1 -- start start 2024-03-06T07:38:31.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc4400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc06e640 0x7f0dbc0b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 0x7f0dbc1f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc4400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dbc15e010 0x7f0dbc0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc06e160 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc4400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc06dc30 con 0x7f0dbc15e010 2024-03-06T07:38:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc4400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc0704d0 con 0x7f0dbc06e640 2024-03-06T07:38:31.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.868+0000 7f0dc1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 0x7f0dbc1f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 0x7f0dbc1f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45474/0 (socket says 172.21.15.32:45474) 2024-03-06T07:38:31.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 -- 172.21.15.32:0/2493629294 learned_addr learned my addr 172.21.15.32:0/2493629294 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc2400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dbc15e010 0x7f0dbc0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1a00640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc06e640 0x7f0dbc0b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 -- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc06e640 msgr2=0x7f0dbc0b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc06e640 0x7f0dbc0b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 -- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dbc15e010 msgr2=0x7f0dbc0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dbc15e010 0x7f0dbc0ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1000640 1 -- 172.21.15.32:0/2493629294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0b4480 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.869+0000 7f0dc1a00640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc06e640 0x7f0dbc0b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.870+0000 7f0dc1000640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 0x7f0dbc1f4600 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0da8000c00 tx=0x7f0da8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.870+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8048070 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.870+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8030640 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.870+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8034040 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.870+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc0b4760 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.870+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc0b4d20 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.871+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dbc04f3a0 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.873+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da8037070 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.873+0000 7f0db9600640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0d9c06b4d0 0x7f0d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.873+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0da80abe60 con 0x7f0dbc0d16d0 2024-03-06T07:38:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.874+0000 7f0dc1a00640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0d9c06b4d0 0x7f0d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.874+0000 7f0dc1a00640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0d9c06b4d0 0x7f0d9c06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0db0006fd0 tx=0x7f0db0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:31.878+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da807a760 con 0x7f0dbc0d16d0 2024-03-06T07:38:32.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.121+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dbc0d11a0 con 0x7f0dbc0d16d0 2024-03-06T07:38:32.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.122+0000 7f0db9600640 1 -- 172.21.15.32:0/2493629294 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f0da807e800 con 0x7f0dbc0d16d0 2024-03-06T07:38:32.125 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:32.125 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0d9c06b4d0 msgr2=0x7f0d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0d9c06b4d0 0x7f0d9c06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0db0006fd0 tx=0x7f0db0008040 comp rx=0 tx=0).stop 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 msgr2=0x7f0dbc1f4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 0x7f0dbc1f4600 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0da8000c00 tx=0x7f0da8004730 comp rx=0 tx=0).stop 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 shutdown_connections 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0d9c06b4d0 0x7f0d9c06d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dbc15e010 0x7f0dbc0ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dbc0d16d0 0x7f0dbc1f4600 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 --2- 172.21.15.32:0/2493629294 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dbc06e640 0x7f0dbc0b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 >> 172.21.15.32:0/2493629294 conn(0x7f0dbc0d0bb0 msgr2=0x7f0dbc15b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.125+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 shutdown_connections 2024-03-06T07:38:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:32.126+0000 7f0dc4400640 1 -- 172.21.15.32:0/2493629294 wait complete. 2024-03-06T07:38:32.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:32.348 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:33.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.577+0000 7f6042a00640 1 Processor -- start 2024-03-06T07:38:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.577+0000 7f6042a00640 1 -- start start 2024-03-06T07:38:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c15e070 0x7f603c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f603c0d0cd0 0x7f603c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c0d18c0 con 0x7f603c0cf410 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c0d1ba0 con 0x7f603c0d0cd0 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0d1e80 con 0x7f603c15e070 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f603c0d0cd0 0x7f603c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f603c0d0cd0 0x7f603c0cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45482/0 (socket says 172.21.15.32:45482) 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6042000640 1 -- 172.21.15.32:0/2596327731 learned_addr learned my addr 172.21.15.32:0/2596327731 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.581+0000 7f6041600640 1 --2- 172.21.15.32:0/2596327731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c15e070 0x7f603c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.582+0000 7f6042000640 1 -- 172.21.15.32:0/2596327731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c15e070 msgr2=0x7f603c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.582+0000 7f6042000640 1 --2- 172.21.15.32:0/2596327731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c15e070 0x7f603c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.582+0000 7f6042000640 1 -- 172.21.15.32:0/2596327731 >> v1:172.21.15.39:6789/0 conn(0x7f603c0cf410 legacy=0x7f603c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.582+0000 7f6042000640 1 -- 172.21.15.32:0/2596327731 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0d2160 con 0x7f603c0d0cd0 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.583+0000 7f6042000640 1 --2- 172.21.15.32:0/2596327731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c0cf870 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f6034009e50 tx=0x7f603402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fb05fde2a10fac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.583+0000 7f603be00640 1 -- 172.21.15.32:0/2596327731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603402ca70 con 0x7f603c0d0cd0 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.583+0000 7f603be00640 1 -- 172.21.15.32:0/2596327731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603402cc10 con 0x7f603c0d0cd0 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.583+0000 7f603be00640 1 -- 172.21.15.32:0/2596327731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60340343f0 con 0x7f603c0d0cd0 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 -- 172.21.15.32:0/2596327731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 msgr2=0x7f603c0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 --2- 172.21.15.32:0/2596327731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c0cf870 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f6034009e50 tx=0x7f603402bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 -- 172.21.15.32:0/2596327731 shutdown_connections 2024-03-06T07:38:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 --2- 172.21.15.32:0/2596327731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c0cf870 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 --2- 172.21.15.32:0/2596327731 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c15e070 0x7f603c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 -- 172.21.15.32:0/2596327731 >> 172.21.15.32:0/2596327731 conn(0x7f603c08d2a0 msgr2=0x7f603c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 -- 172.21.15.32:0/2596327731 shutdown_connections 2024-03-06T07:38:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.584+0000 7f6042a00640 1 -- 172.21.15.32:0/2596327731 wait complete. 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.586+0000 7f6042a00640 1 Processor -- start 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.586+0000 7f6042a00640 1 -- start start 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6042a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c0cf410 0x7f603c1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6042a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c15f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6042a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f603c15e070 0x7f603c0cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6042a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0d4730 con 0x7f603c0d0cd0 2024-03-06T07:38:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6042a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0d45b0 con 0x7f603c15e070 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6042a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0b3630 con 0x7f603c0cf410 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6040c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c15f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6040c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c15f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45490/0 (socket says 172.21.15.32:45490) 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.587+0000 7f6040c00640 1 -- 172.21.15.32:0/2421674476 learned_addr learned my addr 172.21.15.32:0/2421674476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6041600640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c0cf410 0x7f603c1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6042000640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f603c15e070 0x7f603c0cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6040c00640 1 -- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c0cf410 msgr2=0x7f603c1555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6040c00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c0cf410 0x7f603c1555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6040c00640 1 -- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f603c15e070 msgr2=0x7f603c0cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6040c00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f603c15e070 0x7f603c0cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6040c00640 1 -- 172.21.15.32:0/2421674476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0b6ab0 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6042000640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f603c15e070 0x7f603c0cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6040c00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c15f330 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f602800c460 tx=0x7f602800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028016020 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028012870 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.588+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028012c60 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.589+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c0b6d90 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.589+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c0cc390 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.590+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c04f330 con 0x7f603c0d0cd0 2024-03-06T07:38:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.591+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6028019020 con 0x7f603c0d0cd0 2024-03-06T07:38:33.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.592+0000 7f6039600640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f601c06b4d0 0x7f601c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.592+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6028089f10 con 0x7f603c0d0cd0 2024-03-06T07:38:33.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.592+0000 7f6041600640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f601c06b4d0 0x7f601c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.593+0000 7f6041600640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f601c06b4d0 0x7f601c06d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6030006fd0 tx=0x7f6030008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.601+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6028020030 con 0x7f603c0d0cd0 2024-03-06T07:38:33.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.837+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603c0cf140 con 0x7f603c0d0cd0 2024-03-06T07:38:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.838+0000 7f6039600640 1 -- 172.21.15.32:0/2421674476 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f6028058d40 con 0x7f603c0d0cd0 2024-03-06T07:38:33.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:33.839 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f601c06b4d0 msgr2=0x7f601c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f601c06b4d0 0x7f601c06d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6030006fd0 tx=0x7f6030008040 comp rx=0 tx=0).stop 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 msgr2=0x7f603c15f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c15f330 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f602800c460 tx=0x7f602800c920 comp rx=0 tx=0).stop 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 shutdown_connections 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f601c06b4d0 0x7f601c06d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f603c15e070 0x7f603c0cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f603c0d0cd0 0x7f603c15f330 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 --2- 172.21.15.32:0/2421674476 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f603c0cf410 0x7f603c1555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 >> 172.21.15.32:0/2421674476 conn(0x7f603c08d2a0 msgr2=0x7f603c15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 shutdown_connections 2024-03-06T07:38:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:33.840+0000 7f6042a00640 1 -- 172.21.15.32:0/2421674476 wait complete. 2024-03-06T07:38:34.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:34.063 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:35.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:35.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.290+0000 7f8859200640 1 Processor -- start 2024-03-06T07:38:35.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.290+0000 7f8859200640 1 -- start start 2024-03-06T07:38:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.292+0000 7f8859200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8854152bc0 0x7f88540cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.292+0000 7f8859200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f88540af940 0x7f885406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.292+0000 7f8859200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885415c5e0 con 0x7f885406c500 2024-03-06T07:38:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.292+0000 7f8859200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885415cf10 con 0x7f88540af940 2024-03-06T07:38:35.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.292+0000 7f8859200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885415c8c0 con 0x7f8854152bc0 2024-03-06T07:38:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.293+0000 7f8852800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f885406c500 0x7f88540ae5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:55084/0 (socket says 172.21.15.32:55084) 2024-03-06T07:38:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.293+0000 7f8852800640 1 -- 172.21.15.32:0/2933346634 learned_addr learned my addr 172.21.15.32:0/2933346634 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.293+0000 7f8849400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f88540af940 0x7f885406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.293+0000 7f8851e00640 1 --2- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8854152bc0 0x7f88540cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.294+0000 7f8851400640 1 -- 172.21.15.32:0/2933346634 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219861698 0 0) 0x7f885415c5e0 con 0x7f885406c500 2024-03-06T07:38:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.294+0000 7f8849400640 1 -- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8854152bc0 msgr2=0x7f88540cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.294+0000 7f8849400640 1 --2- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8854152bc0 0x7f88540cfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.294+0000 7f8849400640 1 -- 172.21.15.32:0/2933346634 >> v1:172.21.15.39:6789/0 conn(0x7f885406c500 legacy=0x7f88540ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.294+0000 7f8849400640 1 -- 172.21.15.32:0/2933346634 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885415cba0 con 0x7f88540af940 2024-03-06T07:38:35.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.295+0000 7f8849400640 1 --2- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88540af940 0x7f885406bfc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f883c0097a0 tx=0x7f883c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7785caba036b8a9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.296+0000 7f8851400640 1 -- 172.21.15.32:0/2933346634 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c035020 con 0x7f88540af940 2024-03-06T07:38:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.296+0000 7f8851400640 1 -- 172.21.15.32:0/2933346634 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c031830 con 0x7f88540af940 2024-03-06T07:38:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.296+0000 7f8851400640 1 -- 172.21.15.32:0/2933346634 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c031c00 con 0x7f88540af940 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.296+0000 7f8859200640 1 -- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88540af940 msgr2=0x7f885406bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.296+0000 7f8859200640 1 --2- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88540af940 0x7f885406bfc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f883c0097a0 tx=0x7f883c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.297+0000 7f8859200640 1 -- 172.21.15.32:0/2933346634 shutdown_connections 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.297+0000 7f8859200640 1 --2- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88540af940 0x7f885406bfc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.297+0000 7f8859200640 1 --2- 172.21.15.32:0/2933346634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8854152bc0 0x7f88540cfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.297+0000 7f8859200640 1 -- 172.21.15.32:0/2933346634 >> 172.21.15.32:0/2933346634 conn(0x7f88540714d0 msgr2=0x7f88540718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.297+0000 7f8859200640 1 -- 172.21.15.32:0/2933346634 shutdown_connections 2024-03-06T07:38:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.297+0000 7f8859200640 1 -- 172.21.15.32:0/2933346634 wait complete. 2024-03-06T07:38:35.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.298+0000 7f8859200640 1 Processor -- start 2024-03-06T07:38:35.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.299+0000 7f8859200640 1 -- start start 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.299+0000 7f8859200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 0x7f88540b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8859200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88540af940 0x7f88540b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8859200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8854152bc0 0x7f8854148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8859200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88540aff00 con 0x7f885406c500 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8859200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88540afd80 con 0x7f8854152bc0 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8859200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8854152fa0 con 0x7f88540af940 2024-03-06T07:38:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 0x7f88540b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 0x7f88540b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45502/0 (socket says 172.21.15.32:45502) 2024-03-06T07:38:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 -- 172.21.15.32:0/2012692523 learned_addr learned my addr 172.21.15.32:0/2012692523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8849400640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88540af940 0x7f88540b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8852800640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8854152bc0 0x7f8854148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 -- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88540af940 msgr2=0x7f88540b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88540af940 0x7f88540b5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 -- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8854152bc0 msgr2=0x7f8854148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8854152bc0 0x7f8854148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8851e00640 1 -- 172.21.15.32:0/2012692523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88540b42c0 con 0x7f885406c500 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.300+0000 7f8852800640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8854152bc0 0x7f8854148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.301+0000 7f8851e00640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 0x7f88540b3ce0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f8844009d10 tx=0x7f884400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.301+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8844018020 con 0x7f885406c500 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.301+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8844014480 con 0x7f885406c500 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.301+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8844014830 con 0x7f885406c500 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.301+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88540b45a0 con 0x7f885406c500 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.301+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88540b4b60 con 0x7f885406c500 2024-03-06T07:38:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.302+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885404f3a0 con 0x7f885406c500 2024-03-06T07:38:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.303+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8844005d10 con 0x7f885406c500 2024-03-06T07:38:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.304+0000 7f884aa00640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f883006b4d0 0x7f883006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.304+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f884408ad30 con 0x7f885406c500 2024-03-06T07:38:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.304+0000 7f8849400640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f883006b4d0 0x7f883006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.305+0000 7f8849400640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f883006b4d0 0x7f883006d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f883c036000 tx=0x7f883c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.309+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8844022070 con 0x7f885406c500 2024-03-06T07:38:35.394 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:35.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.634+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8854072100 con 0x7f885406c500 2024-03-06T07:38:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.636+0000 7f884aa00640 1 -- 172.21.15.32:0/2012692523 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f8844010040 con 0x7f885406c500 2024-03-06T07:38:35.637 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:35.637 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:35.637 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.638+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f883006b4d0 msgr2=0x7f883006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.638+0000 7f8859200640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f883006b4d0 0x7f883006d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f883c036000 tx=0x7f883c008030 comp rx=0 tx=0).stop 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.638+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 msgr2=0x7f88540b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.638+0000 7f8859200640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 0x7f88540b3ce0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f8844009d10 tx=0x7f884400c490 comp rx=0 tx=0).stop 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 shutdown_connections 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f883006b4d0 0x7f883006d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8854152bc0 0x7f8854148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f88540af940 0x7f88540b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 --2- 172.21.15.32:0/2012692523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885406c500 0x7f88540b3ce0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 >> 172.21.15.32:0/2012692523 conn(0x7f88540714d0 msgr2=0x7f885415e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 shutdown_connections 2024-03-06T07:38:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:35.639+0000 7f8859200640 1 -- 172.21.15.32:0/2012692523 wait complete. 2024-03-06T07:38:35.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:35.863 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:36.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.149+0000 7f82f7200640 1 Processor -- start 2024-03-06T07:38:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.150+0000 7f82f7200640 1 -- start start 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.151+0000 7f82f7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f82f815e0b0 0x7f82f80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.151+0000 7f82f7200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82f80d0e50 0x7f82f80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.151+0000 7f82f7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f80b3710 con 0x7f82f80d4a10 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.151+0000 7f82f7200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f80b09e0 con 0x7f82f80d0e50 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.151+0000 7f82f7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f80b39f0 con 0x7f82f815e0b0 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.151+0000 7f82f6800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82f80d0e50 0x7f82f80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f6800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82f80d0e50 0x7f82f80b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45514/0 (socket says 172.21.15.32:45514) 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82efe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82f80d4a10 0x7f82f80d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40968/0 (socket says 172.21.15.32:40968) 2024-03-06T07:38:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f6800640 1 -- 172.21.15.32:0/2798423605 learned_addr learned my addr 172.21.15.32:0/2798423605 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f5e00640 1 --2- 172.21.15.32:0/2798423605 >> v2:172.21.15.39:3300/0 conn(0x7f82f815e0b0 0x7f82f80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f6800640 1 -- 172.21.15.32:0/2798423605 >> v2:172.21.15.39:3300/0 conn(0x7f82f815e0b0 msgr2=0x7f82f80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f6800640 1 --2- 172.21.15.32:0/2798423605 >> v2:172.21.15.39:3300/0 conn(0x7f82f815e0b0 0x7f82f80cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f6800640 1 -- 172.21.15.32:0/2798423605 >> v1:172.21.15.32:6789/0 conn(0x7f82f80d4a10 legacy=0x7f82f80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f6800640 1 -- 172.21.15.32:0/2798423605 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f80b3cd0 con 0x7f82f80d0e50 2024-03-06T07:38:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.152+0000 7f82f5e00640 1 --2- 172.21.15.32:0/2798423605 >> v2:172.21.15.39:3300/0 conn(0x7f82f815e0b0 0x7f82f80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.153+0000 7f82f6800640 1 --2- 172.21.15.32:0/2798423605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d0e50 0x7f82f80b44d0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f82f80b68c0 tx=0x7f82f002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211bc38a4716c926 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.153+0000 7f82f5400640 1 -- 172.21.15.32:0/2798423605 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f002cb10 con 0x7f82f80d0e50 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f5400640 1 -- 172.21.15.32:0/2798423605 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f002ccb0 con 0x7f82f80d0e50 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f5400640 1 -- 172.21.15.32:0/2798423605 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f002ce60 con 0x7f82f80d0e50 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 -- 172.21.15.32:0/2798423605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d0e50 msgr2=0x7f82f80b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 --2- 172.21.15.32:0/2798423605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d0e50 0x7f82f80b44d0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f82f80b68c0 tx=0x7f82f002bf70 comp rx=0 tx=0).stop 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 -- 172.21.15.32:0/2798423605 shutdown_connections 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 --2- 172.21.15.32:0/2798423605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d0e50 0x7f82f80b44d0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 --2- 172.21.15.32:0/2798423605 >> v2:172.21.15.39:3300/0 conn(0x7f82f815e0b0 0x7f82f80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 -- 172.21.15.32:0/2798423605 >> 172.21.15.32:0/2798423605 conn(0x7f82f80d3d30 msgr2=0x7f82f80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 -- 172.21.15.32:0/2798423605 shutdown_connections 2024-03-06T07:38:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.154+0000 7f82f7200640 1 -- 172.21.15.32:0/2798423605 wait complete. 2024-03-06T07:38:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.156+0000 7f82f7200640 1 Processor -- start 2024-03-06T07:38:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.156+0000 7f82f7200640 1 -- start start 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82f7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82f80d0e50 0x7f82f8161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82f7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 0x7f82f815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82f7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f82f815e0b0 0x7f82f81f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82f7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80cfc60 con 0x7f82f80d4a10 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82f7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f815e490 con 0x7f82f80d0e50 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82f7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80cf850 con 0x7f82f815e0b0 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.157+0000 7f82efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 0x7f82f815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 0x7f82f815ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45518/0 (socket says 172.21.15.32:45518) 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 -- 172.21.15.32:0/773376658 learned_addr learned my addr 172.21.15.32:0/773376658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82f5e00640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82f80d0e50 0x7f82f8161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82f6800640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f82f815e0b0 0x7f82f81f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 -- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f82f815e0b0 msgr2=0x7f82f81f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f82f815e0b0 0x7f82f81f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 -- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82f80d0e50 msgr2=0x7f82f8161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82f80d0e50 0x7f82f8161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 -- 172.21.15.32:0/773376658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f80cc360 con 0x7f82f80d4a10 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.158+0000 7f82efe00640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 0x7f82f815ef00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c440 tx=0x7f82e0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.159+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00074e0 con 0x7f82f80d4a10 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.159+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e0007680 con 0x7f82f80d4a10 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.159+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0004a20 con 0x7f82f80d4a10 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.159+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f80cc640 con 0x7f82f80d4a10 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.159+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f80ccee0 con 0x7f82f80d4a10 2024-03-06T07:38:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.160+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f80cc810 con 0x7f82f80d4a10 2024-03-06T07:38:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.161+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82e001a020 con 0x7f82f80d4a10 2024-03-06T07:38:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.162+0000 7f82ee000640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f82d406b4d0 0x7f82d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.162+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f82e0089af0 con 0x7f82f80d4a10 2024-03-06T07:38:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.162+0000 7f82f5e00640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f82d406b4d0 0x7f82d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.163+0000 7f82f5e00640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f82d406b4d0 0x7f82d406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f82e800a950 tx=0x7f82e8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.167+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82e0010300 con 0x7f82f80d4a10 2024-03-06T07:38:37.396 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.556+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f80cd0e0 con 0x7f82f80d4a10 2024-03-06T07:38:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.557+0000 7f82ee000640 1 -- 172.21.15.32:0/773376658 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f82f80cd0e0 con 0x7f82f80d4a10 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.559+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f82d406b4d0 msgr2=0x7f82d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.559+0000 7f82f7200640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f82d406b4d0 0x7f82d406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f82e800a950 tx=0x7f82e8008040 comp rx=0 tx=0).stop 2024-03-06T07:38:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.559+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 msgr2=0x7f82f815ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.559+0000 7f82f7200640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 0x7f82f815ef00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c440 tx=0x7f82e0001e70 comp rx=0 tx=0).stop 2024-03-06T07:38:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 shutdown_connections 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f82d406b4d0 0x7f82d406d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f82f815e0b0 0x7f82f81f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f80d4a10 0x7f82f815ef00 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 --2- 172.21.15.32:0/773376658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82f80d0e50 0x7f82f8161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 >> 172.21.15.32:0/773376658 conn(0x7f82f80d3d30 msgr2=0x7f82f80b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 shutdown_connections 2024-03-06T07:38:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:37.560+0000 7f82f7200640 1 -- 172.21.15.32:0/773376658 wait complete. 2024-03-06T07:38:37.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:37.782 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:38.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:38.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.992+0000 7fe16dc00640 1 Processor -- start 2024-03-06T07:38:38.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.992+0000 7fe16dc00640 1 -- start start 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe16dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe1680af580 0x7fe16815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe16dc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1680cdc20 0x7fe1680b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe16dc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16815cc40 con 0x7fe1680af1a0 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe16dc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16815cf20 con 0x7fe1680cdc20 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe16dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16815d200 con 0x7fe1680af580 2024-03-06T07:38:38.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe166a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe1680af1a0 0x7fe1680b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49128/0 (socket says 172.21.15.32:49128) 2024-03-06T07:38:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe166a00640 1 -- 172.21.15.32:0/3893942278 learned_addr learned my addr 172.21.15.32:0/3893942278 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe167400640 1 --2- 172.21.15.32:0/3893942278 >> v2:172.21.15.32:3300/0 conn(0x7fe1680cdc20 0x7fe1680b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.994+0000 7fe15e000640 1 --2- 172.21.15.32:0/3893942278 >> v2:172.21.15.39:3300/0 conn(0x7fe1680af580 0x7fe16815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe166000640 1 -- 172.21.15.32:0/3893942278 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797856730 0 0) 0x7fe16815cc40 con 0x7fe1680af1a0 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe166000640 1 -- 172.21.15.32:0/3893942278 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c003520 con 0x7fe1680af1a0 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe167400640 1 -- 172.21.15.32:0/3893942278 >> v2:172.21.15.39:3300/0 conn(0x7fe1680af580 msgr2=0x7fe16815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe167400640 1 --2- 172.21.15.32:0/3893942278 >> v2:172.21.15.39:3300/0 conn(0x7fe1680af580 0x7fe16815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe167400640 1 -- 172.21.15.32:0/3893942278 >> v1:172.21.15.32:6789/0 conn(0x7fe1680af1a0 legacy=0x7fe1680b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe166000640 1 -- 172.21.15.32:0/3893942278 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3870714202 0 0) 0x7fe14c003520 con 0x7fe1680af1a0 2024-03-06T07:38:38.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.995+0000 7fe167400640 1 -- 172.21.15.32:0/3893942278 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16815d4e0 con 0x7fe1680cdc20 2024-03-06T07:38:38.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.996+0000 7fe167400640 1 --2- 172.21.15.32:0/3893942278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680cdc20 0x7fe1680b2dd0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe1680b54c0 tx=0x7fe16002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b64020ef7f79944 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.996+0000 7fe166000640 1 -- 172.21.15.32:0/3893942278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160037070 con 0x7fe1680cdc20 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.996+0000 7fe166000640 1 -- 172.21.15.32:0/3893942278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160030510 con 0x7fe1680cdc20 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe166000640 1 -- 172.21.15.32:0/3893942278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580019f0 con 0x7fe1680cdc20 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 -- 172.21.15.32:0/3893942278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680cdc20 msgr2=0x7fe1680b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 --2- 172.21.15.32:0/3893942278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680cdc20 0x7fe1680b2dd0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe1680b54c0 tx=0x7fe16002b710 comp rx=0 tx=0).stop 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 -- 172.21.15.32:0/3893942278 shutdown_connections 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 --2- 172.21.15.32:0/3893942278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680cdc20 0x7fe1680b2dd0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 --2- 172.21.15.32:0/3893942278 >> v2:172.21.15.39:3300/0 conn(0x7fe1680af580 0x7fe16815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 -- 172.21.15.32:0/3893942278 >> 172.21.15.32:0/3893942278 conn(0x7fe16806afb0 msgr2=0x7fe16806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:38.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 -- 172.21.15.32:0/3893942278 shutdown_connections 2024-03-06T07:38:39.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:38.997+0000 7fe16dc00640 1 -- 172.21.15.32:0/3893942278 wait complete. 2024-03-06T07:38:39.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.000+0000 7fe16dc00640 1 Processor -- start 2024-03-06T07:38:39.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.000+0000 7fe16dc00640 1 -- start start 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe16dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680af1a0 0x7fe16809d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe16dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 0x7fe16815d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe16dc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1680cdc20 0x7fe1680c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe16dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680b36a0 con 0x7fe1680af580 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe16dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680b3520 con 0x7fe1680af1a0 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe16dc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680b3820 con 0x7fe1680cdc20 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe15e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 0x7fe16815d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe15e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 0x7fe16815d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46222/0 (socket says 172.21.15.32:46222) 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.002+0000 7fe15e000640 1 -- 172.21.15.32:0/779833730 learned_addr learned my addr 172.21.15.32:0/779833730 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe166a00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680af1a0 0x7fe16809d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe167400640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1680cdc20 0x7fe1680c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15e000640 1 -- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1680cdc20 msgr2=0x7fe1680c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15e000640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1680cdc20 0x7fe1680c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15e000640 1 -- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680af1a0 msgr2=0x7fe16809d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15e000640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680af1a0 0x7fe16809d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15e000640 1 -- 172.21.15.32:0/779833730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16815e220 con 0x7fe1680af580 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15e000640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 0x7fe16815d760 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fe15000e980 tx=0x7fe15000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.003+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150017070 con 0x7fe1680af580 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.004+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15000ca00 con 0x7fe1680af580 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.004+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15000cdd0 con 0x7fe1680af580 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.004+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16815e500 con 0x7fe1680af580 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.004+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16815eac0 con 0x7fe1680af580 2024-03-06T07:38:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.005+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe168069510 con 0x7fe1680af580 2024-03-06T07:38:39.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.006+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe15001a070 con 0x7fe1680af580 2024-03-06T07:38:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.007+0000 7fe15f400640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe14406b500 0x7fe14406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.008+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe15008a1e0 con 0x7fe1680af580 2024-03-06T07:38:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.008+0000 7fe166a00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe14406b500 0x7fe14406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.009+0000 7fe166a00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe14406b500 0x7fe14406d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe158001560 tx=0x7fe158009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.012+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1500590b0 con 0x7fe1680af580 2024-03-06T07:38:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.249+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe16809dc50 con 0x7fe1680af580 2024-03-06T07:38:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.250+0000 7fe15f400640 1 -- 172.21.15.32:0/779833730 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fe1500589b0 con 0x7fe1680af580 2024-03-06T07:38:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:39.252 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:39.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe14406b500 msgr2=0x7fe14406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe14406b500 0x7fe14406d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe158001560 tx=0x7fe158009040 comp rx=0 tx=0).stop 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 msgr2=0x7fe16815d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 0x7fe16815d760 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fe15000e980 tx=0x7fe15000ee40 comp rx=0 tx=0).stop 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 shutdown_connections 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fe14406b500 0x7fe14406d9c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fe1680cdc20 0x7fe1680c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680af580 0x7fe16815d760 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 --2- 172.21.15.32:0/779833730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680af1a0 0x7fe16809d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 >> 172.21.15.32:0/779833730 conn(0x7fe16806afb0 msgr2=0x7fe168159d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 shutdown_connections 2024-03-06T07:38:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:39.253+0000 7fe16dc00640 1 -- 172.21.15.32:0/779833730 wait complete. 2024-03-06T07:38:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:39.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:39.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:40.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:40.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.688+0000 7f21d9e00640 1 Processor -- start 2024-03-06T07:38:40.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.689+0000 7f21d9e00640 1 -- start start 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21d415e0b0 0x7f21d40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d40b3710 con 0x7f21d40d0c80 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d40b09e0 con 0x7f21d415e0b0 2024-03-06T07:38:40.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40b39f0 con 0x7f21d40d4a10 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d8a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21d415e0b0 0x7f21d40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f21d40d0c80 0x7f21d40d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49144/0 (socket says 172.21.15.32:49144) 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.690+0000 7f21d9400640 1 -- 172.21.15.32:0/2114373200 learned_addr learned my addr 172.21.15.32:0/2114373200 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.691+0000 7f21d3e00640 1 --2- 172.21.15.32:0/2114373200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.691+0000 7f21d8a00640 1 -- 172.21.15.32:0/2114373200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 msgr2=0x7f21d40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.691+0000 7f21d8a00640 1 --2- 172.21.15.32:0/2114373200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.691+0000 7f21d8a00640 1 -- 172.21.15.32:0/2114373200 >> v1:172.21.15.32:6789/0 conn(0x7f21d40d0c80 legacy=0x7f21d40d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.691+0000 7f21d3400640 1 -- 172.21.15.32:0/2114373200 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174054860 0 0) 0x7f21d40b3710 con 0x7f21d40d0c80 2024-03-06T07:38:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.691+0000 7f21d8a00640 1 -- 172.21.15.32:0/2114373200 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d40b3cd0 con 0x7f21d415e0b0 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.692+0000 7f21d8a00640 1 --2- 172.21.15.32:0/2114373200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d40cf310 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009d90 tx=0x7f21c402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1a32eabba3770f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.692+0000 7f21d3400640 1 -- 172.21.15.32:0/2114373200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4039070 con 0x7f21d415e0b0 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.692+0000 7f21d3400640 1 -- 172.21.15.32:0/2114373200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c402cc50 con 0x7f21d415e0b0 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.692+0000 7f21d3400640 1 -- 172.21.15.32:0/2114373200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4034440 con 0x7f21d415e0b0 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 -- 172.21.15.32:0/2114373200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 msgr2=0x7f21d40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 --2- 172.21.15.32:0/2114373200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d40cf310 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009d90 tx=0x7f21c402bea0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 -- 172.21.15.32:0/2114373200 shutdown_connections 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 --2- 172.21.15.32:0/2114373200 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 --2- 172.21.15.32:0/2114373200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d40cf310 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 -- 172.21.15.32:0/2114373200 >> 172.21.15.32:0/2114373200 conn(0x7f21d40d3d30 msgr2=0x7f21d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 -- 172.21.15.32:0/2114373200 shutdown_connections 2024-03-06T07:38:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.693+0000 7f21d9e00640 1 -- 172.21.15.32:0/2114373200 wait complete. 2024-03-06T07:38:40.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.695+0000 7f21d9e00640 1 Processor -- start 2024-03-06T07:38:40.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.695+0000 7f21d9e00640 1 -- start start 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21d40d0c80 0x7f21d40b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d4151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40cfc60 con 0x7f21d415e0b0 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d415e490 con 0x7f21d40d0c80 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40b54e0 con 0x7f21d40d4a10 2024-03-06T07:38:40.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.696+0000 7f21d9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d4151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d4151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46232/0 (socket says 172.21.15.32:46232) 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 -- 172.21.15.32:0/1106668733 learned_addr learned my addr 172.21.15.32:0/1106668733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d3e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d8a00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21d40d0c80 0x7f21d40b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 -- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 msgr2=0x7f21d40cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 -- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21d40d0c80 msgr2=0x7f21d40b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21d40d0c80 0x7f21d40b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 -- 172.21.15.32:0/1106668733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d40b71a0 con 0x7f21d415e0b0 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d8a00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21d40d0c80 0x7f21d40b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.697+0000 7f21d9400640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d4151d80 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00c450 tx=0x7f21cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.698+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc018070 con 0x7f21d415e0b0 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.698+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc007500 con 0x7f21d415e0b0 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.698+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc004a20 con 0x7f21d415e0b0 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.698+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d40b74e0 con 0x7f21d415e0b0 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.698+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d40b7d20 con 0x7f21d415e0b0 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.699+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d4072bd0 con 0x7f21d415e0b0 2024-03-06T07:38:40.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.701+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21cc01b070 con 0x7f21d415e0b0 2024-03-06T07:38:40.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.702+0000 7f21d0c00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21b406b4d0 0x7f21b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.702+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f21cc089670 con 0x7f21d415e0b0 2024-03-06T07:38:40.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.702+0000 7f21d8a00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21b406b4d0 0x7f21b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.703+0000 7f21d8a00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21b406b4d0 0x7f21b406d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f21c4036000 tx=0x7f21c4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.707+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21d4072bd0 con 0x7f21d415e0b0 2024-03-06T07:38:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.946+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21d404f330 con 0x7f21d415e0b0 2024-03-06T07:38:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.947+0000 7f21d0c00640 1 -- 172.21.15.32:0/1106668733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f21d40b74e0 con 0x7f21d415e0b0 2024-03-06T07:38:40.949 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:40.949 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21b406b4d0 msgr2=0x7f21b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21b406b4d0 0x7f21b406d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f21c4036000 tx=0x7f21c4030000 comp rx=0 tx=0).stop 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 msgr2=0x7f21d4151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d4151d80 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00c450 tx=0x7f21cc00c910 comp rx=0 tx=0).stop 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 shutdown_connections 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f21b406b4d0 0x7f21b406d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21d415e0b0 0x7f21d4151d80 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f21d40d4a10 0x7f21d40cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 --2- 172.21.15.32:0/1106668733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21d40d0c80 0x7f21d40b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 >> 172.21.15.32:0/1106668733 conn(0x7f21d40d3d30 msgr2=0x7f21d406cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 shutdown_connections 2024-03-06T07:38:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:40.950+0000 7f21d9e00640 1 -- 172.21.15.32:0/1106668733 wait complete. 2024-03-06T07:38:41.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:41.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:42.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:42.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.405+0000 7f905a400640 1 Processor -- start 2024-03-06T07:38:42.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.405+0000 7f905a400640 1 -- start start 2024-03-06T07:38:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.406+0000 7f905a400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90540af1a0 0x7f905415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.406+0000 7f905a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af580 0x7f90540b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f905a400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90540cdc20 0x7f90540b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f905a400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905415cc40 con 0x7f90540cdc20 2024-03-06T07:38:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f905a400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905415cf20 con 0x7f90540af1a0 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f905a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905415d200 con 0x7f90540af580 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f9053e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90540cdc20 0x7f90540b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f9053e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90540cdc20 0x7f90540b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46234/0 (socket says 172.21.15.32:46234) 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f9053e00640 1 -- 172.21.15.32:0/2518009057 learned_addr learned my addr 172.21.15.32:0/2518009057 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f9052a00640 1 --2- 172.21.15.32:0/2518009057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af580 0x7f90540b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.407+0000 7f9053400640 1 --2- 172.21.15.32:0/2518009057 >> v2:172.21.15.39:3300/0 conn(0x7f90540af1a0 0x7f905415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.408+0000 7f9053e00640 1 -- 172.21.15.32:0/2518009057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af580 msgr2=0x7f90540b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.408+0000 7f9053e00640 1 --2- 172.21.15.32:0/2518009057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af580 0x7f90540b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.408+0000 7f9053e00640 1 -- 172.21.15.32:0/2518009057 >> v2:172.21.15.39:3300/0 conn(0x7f90540af1a0 msgr2=0x7f905415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.408+0000 7f9053e00640 1 --2- 172.21.15.32:0/2518009057 >> v2:172.21.15.39:3300/0 conn(0x7f90540af1a0 0x7f905415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.408+0000 7f9053e00640 1 -- 172.21.15.32:0/2518009057 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905415d4e0 con 0x7f90540cdc20 2024-03-06T07:38:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.409+0000 7f9053e00640 1 --2- 172.21.15.32:0/2518009057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540cdc20 0x7f90540b2dd0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f9054074960 tx=0x7f904c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78de2c1aa0f9b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.409+0000 7f9052000640 1 -- 172.21.15.32:0/2518009057 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c02c9f0 con 0x7f90540cdc20 2024-03-06T07:38:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.409+0000 7f9052000640 1 -- 172.21.15.32:0/2518009057 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c02cb90 con 0x7f90540cdc20 2024-03-06T07:38:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.409+0000 7f9052000640 1 -- 172.21.15.32:0/2518009057 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c0343f0 con 0x7f90540cdc20 2024-03-06T07:38:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 -- 172.21.15.32:0/2518009057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540cdc20 msgr2=0x7f90540b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 --2- 172.21.15.32:0/2518009057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540cdc20 0x7f90540b2dd0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f9054074960 tx=0x7f904c02be50 comp rx=0 tx=0).stop 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 -- 172.21.15.32:0/2518009057 shutdown_connections 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 --2- 172.21.15.32:0/2518009057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540cdc20 0x7f90540b2dd0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 --2- 172.21.15.32:0/2518009057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af580 0x7f90540b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 --2- 172.21.15.32:0/2518009057 >> v2:172.21.15.39:3300/0 conn(0x7f90540af1a0 0x7f905415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 -- 172.21.15.32:0/2518009057 >> 172.21.15.32:0/2518009057 conn(0x7f905406afb0 msgr2=0x7f905406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 -- 172.21.15.32:0/2518009057 shutdown_connections 2024-03-06T07:38:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.410+0000 7f905a400640 1 -- 172.21.15.32:0/2518009057 wait complete. 2024-03-06T07:38:42.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.412+0000 7f905a400640 1 Processor -- start 2024-03-06T07:38:42.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.412+0000 7f905a400640 1 -- start start 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f905a400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af1a0 0x7f905414c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f905a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 0x7f90540ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f905a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90540cdc20 0x7f90540ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f905a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540b3490 con 0x7f90540af580 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f905a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540b3310 con 0x7f90540cdc20 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f905a400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90540b3610 con 0x7f90540af1a0 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9052a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 0x7f90540ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9052a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 0x7f90540ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46244/0 (socket says 172.21.15.32:46244) 2024-03-06T07:38:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9052a00640 1 -- 172.21.15.32:0/1106511418 learned_addr learned my addr 172.21.15.32:0/1106511418 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9053400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af1a0 0x7f905414c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9052a00640 1 -- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af1a0 msgr2=0x7f905414c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9052a00640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af1a0 0x7f905414c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9052a00640 1 -- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90540cdc20 msgr2=0x7f90540ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.413+0000 7f9053e00640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90540cdc20 0x7f90540ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f9052a00640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90540cdc20 0x7f90540ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f9052a00640 1 -- 172.21.15.32:0/1106511418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90540cb590 con 0x7f90540af580 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f9053400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af1a0 0x7f905414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f9052a00640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 0x7f90540ce740 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f903c00d830 tx=0x7f903c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c00b6c0 con 0x7f90540af580 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c019070 con 0x7f90540af580 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c00ba80 con 0x7f90540af580 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90540cb870 con 0x7f90540af580 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.414+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90541547c0 con 0x7f90540af580 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.416+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9054069510 con 0x7f90540af580 2024-03-06T07:38:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.416+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f903c01e460 con 0x7f90540af580 2024-03-06T07:38:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.417+0000 7f904b400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f903006b4d0 0x7f903006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.417+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f903c08a900 con 0x7f90540af580 2024-03-06T07:38:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.418+0000 7f9053400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f903006b4d0 0x7f903006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.419+0000 7f9053400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f903006b4d0 0x7f903006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9044006fd0 tx=0x7f9044008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.423+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f903c059730 con 0x7f90540af580 2024-03-06T07:38:42.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.660+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f905404f3a0 con 0x7f90540af580 2024-03-06T07:38:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.661+0000 7f904b400640 1 -- 172.21.15.32:0/1106511418 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f903c059030 con 0x7f90540af580 2024-03-06T07:38:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:42.662 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:42.663 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f903006b4d0 msgr2=0x7f903006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f903006b4d0 0x7f903006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9044006fd0 tx=0x7f9044008040 comp rx=0 tx=0).stop 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 msgr2=0x7f90540ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 0x7f90540ce740 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f903c00d830 tx=0x7f903c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 shutdown_connections 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f903006b4d0 0x7f903006d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90540cdc20 0x7f90540ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90540af580 0x7f90540ce740 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 --2- 172.21.15.32:0/1106511418 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f90540af1a0 0x7f905414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 >> 172.21.15.32:0/1106511418 conn(0x7f905406afb0 msgr2=0x7f90540acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 shutdown_connections 2024-03-06T07:38:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:42.664+0000 7f905a400640 1 -- 172.21.15.32:0/1106511418 wait complete. 2024-03-06T07:38:42.887 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:42.887 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:43.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.109+0000 7f644ca00640 1 Processor -- start 2024-03-06T07:38:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.109+0000 7f644ca00640 1 -- start start 2024-03-06T07:38:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f644ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f644815e0b0 0x7f64480cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f644ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f64480d0e50 0x7f64480b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f644ca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480b3710 con 0x7f64480d4a10 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f644ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64480b09e0 con 0x7f64480d0e50 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f644ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64480b39f0 con 0x7f644815e0b0 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f6447e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f64480d0e50 0x7f64480b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f6447e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f64480d0e50 0x7f64480b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46254/0 (socket says 172.21.15.32:46254) 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f6447e00640 1 -- 172.21.15.32:0/708032014 learned_addr learned my addr 172.21.15.32:0/708032014 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.111+0000 7f6447400640 1 --2- 172.21.15.32:0/708032014 >> v2:172.21.15.39:3300/0 conn(0x7f644815e0b0 0x7f64480cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.112+0000 7f6447e00640 1 -- 172.21.15.32:0/708032014 >> v2:172.21.15.39:3300/0 conn(0x7f644815e0b0 msgr2=0x7f64480cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.112+0000 7f6447e00640 1 --2- 172.21.15.32:0/708032014 >> v2:172.21.15.39:3300/0 conn(0x7f644815e0b0 0x7f64480cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.112+0000 7f6447e00640 1 -- 172.21.15.32:0/708032014 >> v1:172.21.15.39:6789/0 conn(0x7f64480d4a10 legacy=0x7f64480d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.112+0000 7f6447e00640 1 -- 172.21.15.32:0/708032014 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480b3cd0 con 0x7f64480d0e50 2024-03-06T07:38:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.112+0000 7f6447400640 1 --2- 172.21.15.32:0/708032014 >> v2:172.21.15.39:3300/0 conn(0x7f644815e0b0 0x7f64480cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.113+0000 7f6447e00640 1 --2- 172.21.15.32:0/708032014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d0e50 0x7f64480b44d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f64480b68c0 tx=0x7f644002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b3e67cecf55a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.113+0000 7f6446000640 1 -- 172.21.15.32:0/708032014 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644002ca80 con 0x7f64480d0e50 2024-03-06T07:38:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.113+0000 7f6446000640 1 -- 172.21.15.32:0/708032014 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644002cc20 con 0x7f64480d0e50 2024-03-06T07:38:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.113+0000 7f6446000640 1 -- 172.21.15.32:0/708032014 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64400343f0 con 0x7f64480d0e50 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 -- 172.21.15.32:0/708032014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d0e50 msgr2=0x7f64480b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 --2- 172.21.15.32:0/708032014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d0e50 0x7f64480b44d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f64480b68c0 tx=0x7f644002bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 -- 172.21.15.32:0/708032014 shutdown_connections 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 --2- 172.21.15.32:0/708032014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d0e50 0x7f64480b44d0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 --2- 172.21.15.32:0/708032014 >> v2:172.21.15.39:3300/0 conn(0x7f644815e0b0 0x7f64480cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 -- 172.21.15.32:0/708032014 >> 172.21.15.32:0/708032014 conn(0x7f64480d3d30 msgr2=0x7f64480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 -- 172.21.15.32:0/708032014 shutdown_connections 2024-03-06T07:38:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.114+0000 7f644ca00640 1 -- 172.21.15.32:0/708032014 wait complete. 2024-03-06T07:38:44.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.116+0000 7f644ca00640 1 Processor -- start 2024-03-06T07:38:44.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.116+0000 7f644ca00640 1 -- start start 2024-03-06T07:38:44.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.117+0000 7f644ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f64480d0e50 0x7f644806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.117+0000 7f644ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 0x7f644806c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.117+0000 7f644ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644815e0b0 0x7f64480cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f644ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64480cfc60 con 0x7f64480d4a10 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f644ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644815e490 con 0x7f644815e0b0 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f644ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64480cf850 con 0x7f64480d0e50 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 0x7f644806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 0x7f644806c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46270/0 (socket says 172.21.15.32:46270) 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 -- 172.21.15.32:0/180242536 learned_addr learned my addr 172.21.15.32:0/180242536 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6447e00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644815e0b0 0x7f64480cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 -- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f64480d0e50 msgr2=0x7f644806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6447400640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f64480d0e50 0x7f644806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f64480d0e50 0x7f644806bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 -- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644815e0b0 msgr2=0x7f64480cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644815e0b0 0x7f64480cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.118+0000 7f6446a00640 1 -- 172.21.15.32:0/180242536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480cafa0 con 0x7f64480d4a10 2024-03-06T07:38:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.119+0000 7f6446a00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 0x7f644806c480 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f643400c400 tx=0x7f643400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.121+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434011790 con 0x7f64480d4a10 2024-03-06T07:38:44.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.121+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6434017070 con 0x7f64480d4a10 2024-03-06T07:38:44.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.121+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64480cb280 con 0x7f64480d4a10 2024-03-06T07:38:44.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.121+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434011bb0 con 0x7f64480d4a10 2024-03-06T07:38:44.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.122+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64480cb840 con 0x7f64480d4a10 2024-03-06T07:38:44.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.123+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644804f330 con 0x7f64480d4a10 2024-03-06T07:38:44.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.124+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f643401c460 con 0x7f64480d4a10 2024-03-06T07:38:44.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.125+0000 7f643f400640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.125+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f643408a1c0 con 0x7f64480d4a10 2024-03-06T07:38:44.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.125+0000 7f6447400640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.126+0000 7f6447400640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f642806b4d0 0x7f642806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f64480b67f0 tx=0x7f6438008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.129+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6434058ed0 con 0x7f64480d4a10 2024-03-06T07:38:44.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.370+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6448161640 con 0x7f64480d4a10 2024-03-06T07:38:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.371+0000 7f643f400640 1 -- 172.21.15.32:0/180242536 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f64340587d0 con 0x7f64480d4a10 2024-03-06T07:38:44.372 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:44.372 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:44.372 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f642806b4d0 msgr2=0x7f642806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f642806b4d0 0x7f642806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f64480b67f0 tx=0x7f6438008040 comp rx=0 tx=0).stop 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 msgr2=0x7f644806c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 0x7f644806c480 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f643400c400 tx=0x7f643400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 shutdown_connections 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644815e0b0 0x7f64480cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64480d4a10 0x7f644806c480 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 --2- 172.21.15.32:0/180242536 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f64480d0e50 0x7f644806bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 >> 172.21.15.32:0/180242536 conn(0x7f64480d3d30 msgr2=0x7f64480b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 shutdown_connections 2024-03-06T07:38:44.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:44.373+0000 7f644ca00640 1 -- 172.21.15.32:0/180242536 wait complete. 2024-03-06T07:38:44.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:44.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:45.404 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:45.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:45.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.826+0000 7f19d3200640 1 Processor -- start 2024-03-06T07:38:45.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.827+0000 7f19d3200640 1 -- start start 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.829+0000 7f19d3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19d4144c00 0x7f19d4145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.829+0000 7f19d3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d40d0910 con 0x7f19d406b4f0 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d40d0bf0 con 0x7f19d4144c00 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40d0ed0 con 0x7f19d41457c0 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19d4144c00 0x7f19d4145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19d4144c00 0x7f19d4145280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46274/0 (socket says 172.21.15.32:46274) 2024-03-06T07:38:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d2800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f19d406b4f0 0x7f19d415f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49156/0 (socket says 172.21.15.32:49156) 2024-03-06T07:38:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d1e00640 1 -- 172.21.15.32:0/3602469396 learned_addr learned my addr 172.21.15.32:0/3602469396 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19cbe00640 1 --2- 172.21.15.32:0/3602469396 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d1e00640 1 -- 172.21.15.32:0/3602469396 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 msgr2=0x7f19d406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.830+0000 7f19d1e00640 1 --2- 172.21.15.32:0/3602469396 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.831+0000 7f19d1e00640 1 -- 172.21.15.32:0/3602469396 >> v1:172.21.15.32:6789/0 conn(0x7f19d406b4f0 legacy=0x7f19d415f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.831+0000 7f19d1e00640 1 -- 172.21.15.32:0/3602469396 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40d11b0 con 0x7f19d4144c00 2024-03-06T07:38:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.832+0000 7f19d1e00640 1 --2- 172.21.15.32:0/3602469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d4144c00 0x7f19d4145280 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009870 tx=0x7f19c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30182bcf45688d1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.832+0000 7f19d1400640 1 -- 172.21.15.32:0/3602469396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4034070 con 0x7f19d4144c00 2024-03-06T07:38:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.832+0000 7f19d1400640 1 -- 172.21.15.32:0/3602469396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4030840 con 0x7f19d4144c00 2024-03-06T07:38:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.832+0000 7f19d1400640 1 -- 172.21.15.32:0/3602469396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4030c10 con 0x7f19d4144c00 2024-03-06T07:38:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 -- 172.21.15.32:0/3602469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d4144c00 msgr2=0x7f19d4145280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 --2- 172.21.15.32:0/3602469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d4144c00 0x7f19d4145280 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009870 tx=0x7f19c402b980 comp rx=0 tx=0).stop 2024-03-06T07:38:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 -- 172.21.15.32:0/3602469396 shutdown_connections 2024-03-06T07:38:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 --2- 172.21.15.32:0/3602469396 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 --2- 172.21.15.32:0/3602469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d4144c00 0x7f19d4145280 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 -- 172.21.15.32:0/3602469396 >> 172.21.15.32:0/3602469396 conn(0x7f19d406c0c0 msgr2=0x7f19d406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 -- 172.21.15.32:0/3602469396 shutdown_connections 2024-03-06T07:38:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.833+0000 7f19d3200640 1 -- 172.21.15.32:0/3602469396 wait complete. 2024-03-06T07:38:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.836+0000 7f19d3200640 1 Processor -- start 2024-03-06T07:38:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.836+0000 7f19d3200640 1 -- start start 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 0x7f19d415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19d4144c00 0x7f19d4158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d4134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40d19b0 con 0x7f19d406b4f0 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d4144fe0 con 0x7f19d4144c00 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40d0140 con 0x7f19d41457c0 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 0x7f19d415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 0x7f19d415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46286/0 (socket says 172.21.15.32:46286) 2024-03-06T07:38:45.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 -- 172.21.15.32:0/180071302 learned_addr learned my addr 172.21.15.32:0/180071302 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 -- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 msgr2=0x7f19d4134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d2800640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d4134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19cbe00640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19d4144c00 0x7f19d4158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d4134dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 -- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19d4144c00 msgr2=0x7f19d4158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19d4144c00 0x7f19d4158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.838+0000 7f19d1e00640 1 -- 172.21.15.32:0/180071302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40ae7c0 con 0x7f19d406b4f0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.839+0000 7f19d1e00640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 0x7f19d415b8c0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f19c402b9c0 tx=0x7f19c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.839+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4048070 con 0x7f19d406b4f0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.839+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4030640 con 0x7f19d406b4f0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.839+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d40afd80 con 0x7f19d406b4f0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.839+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4034040 con 0x7f19d406b4f0 2024-03-06T07:38:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.839+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d4135310 con 0x7f19d406b4f0 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.840+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d404f330 con 0x7f19d406b4f0 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.841+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19c404b070 con 0x7f19d406b4f0 2024-03-06T07:38:45.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.842+0000 7f19ca000640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19b006b500 0x7f19b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.842+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f19c40ac2d0 con 0x7f19d406b4f0 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.843+0000 7f19cbe00640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19b006b500 0x7f19b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.843+0000 7f19cbe00640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19b006b500 0x7f19b006d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f19d41440b0 tx=0x7f19bc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:45.847+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19c407b190 con 0x7f19d406b4f0 2024-03-06T07:38:46.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.106+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19d40722f0 con 0x7f19d406b4f0 2024-03-06T07:38:46.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.107+0000 7f19ca000640 1 -- 172.21.15.32:0/180071302 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f19c407aa90 con 0x7f19d406b4f0 2024-03-06T07:38:46.109 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:46.109 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:46.109 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:46.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19b006b500 msgr2=0x7f19b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19b006b500 0x7f19b006d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f19d41440b0 tx=0x7f19bc008040 comp rx=0 tx=0).stop 2024-03-06T07:38:46.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 msgr2=0x7f19d415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 0x7f19d415b8c0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f19c402b9c0 tx=0x7f19c4004730 comp rx=0 tx=0).stop 2024-03-06T07:38:46.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 shutdown_connections 2024-03-06T07:38:46.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f19b006b500 0x7f19b006d9c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f19d41457c0 0x7f19d4134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19d4144c00 0x7f19d4158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 --2- 172.21.15.32:0/180071302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d406b4f0 0x7f19d415b8c0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.110+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 >> 172.21.15.32:0/180071302 conn(0x7f19d406c0c0 msgr2=0x7f19d40cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.111+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 shutdown_connections 2024-03-06T07:38:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:46.111+0000 7f19d3200640 1 -- 172.21.15.32:0/180071302 wait complete. 2024-03-06T07:38:46.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:46.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:47.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:47.406 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:47.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.561+0000 7efee8c00640 1 Processor -- start 2024-03-06T07:38:47.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.562+0000 7efee8c00640 1 -- start start 2024-03-06T07:38:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.563+0000 7efee8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efee40b6000 0x7efee4162750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.563+0000 7efee8c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee4162c90 0x7efee40ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.563+0000 7efee8c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee41639f0 con 0x7efee40b3d70 2024-03-06T07:38:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.563+0000 7efee8c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee4163430 con 0x7efee40b6000 2024-03-06T07:38:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.563+0000 7efee8c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee4163710 con 0x7efee4162c90 2024-03-06T07:38:47.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.564+0000 7efee1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efee40b6000 0x7efee4162750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.564+0000 7efee2800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee4162c90 0x7efee40ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.564+0000 7efee1e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efee40b3d70 0x7efee40b4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.32:60854/0 (socket says 172.21.15.32:60854) 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.564+0000 7efee1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efee40b6000 0x7efee4162750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:37476/0 (socket says 172.21.15.32:37476) 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.564+0000 7efee1e00640 1 -- 172.21.15.32:0/3237494319 learned_addr learned my addr 172.21.15.32:0/3237494319 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.565+0000 7efee1400640 1 -- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee4162c90 msgr2=0x7efee40ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.565+0000 7efee1400640 1 --2- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee4162c90 0x7efee40ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.565+0000 7efee1400640 1 -- 172.21.15.32:0/3237494319 >> v1:172.21.15.39:6789/0 conn(0x7efee40b3d70 legacy=0x7efee40b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.565+0000 7efee1400640 1 -- 172.21.15.32:0/3237494319 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40addf0 con 0x7efee40b6000 2024-03-06T07:38:47.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.568+0000 7efee1400640 1 --2- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee4162750 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efecc0097a0 tx=0x7efecc02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e46a0ebe4c4af85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee0a00640 1 -- 172.21.15.32:0/3237494319 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc037070 con 0x7efee40b6000 2024-03-06T07:38:47.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee0a00640 1 -- 172.21.15.32:0/3237494319 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc030710 con 0x7efee40b6000 2024-03-06T07:38:47.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee0a00640 1 -- 172.21.15.32:0/3237494319 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc030ae0 con 0x7efee40b6000 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee8c00640 1 -- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 msgr2=0x7efee4162750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee8c00640 1 --2- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee4162750 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efecc0097a0 tx=0x7efecc02b710 comp rx=0 tx=0).stop 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee8c00640 1 -- 172.21.15.32:0/3237494319 shutdown_connections 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee8c00640 1 --2- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee4162c90 0x7efee40ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee8c00640 1 --2- 172.21.15.32:0/3237494319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee4162750 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.569+0000 7efee8c00640 1 -- 172.21.15.32:0/3237494319 >> 172.21.15.32:0/3237494319 conn(0x7efee406c2a0 msgr2=0x7efee406c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.570+0000 7efee8c00640 1 -- 172.21.15.32:0/3237494319 shutdown_connections 2024-03-06T07:38:47.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.570+0000 7efee8c00640 1 -- 172.21.15.32:0/3237494319 wait complete. 2024-03-06T07:38:47.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.571+0000 7efee8c00640 1 Processor -- start 2024-03-06T07:38:47.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.572+0000 7efee8c00640 1 -- start start 2024-03-06T07:38:47.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee8c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee40b3d70 0x7efee415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee41fb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 0x7efee41584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee8c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee40aeb30 con 0x7efee4162c90 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee8c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee4163140 con 0x7efee40b6000 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee8c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee40aecb0 con 0x7efee40b3d70 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 0x7efee41584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 0x7efee41584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46294/0 (socket says 172.21.15.32:46294) 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee2800640 1 -- 172.21.15.32:0/1601526129 learned_addr learned my addr 172.21.15.32:0/1601526129 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee1400640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee41fb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee1e00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee40b3d70 0x7efee415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee2800640 1 -- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee40b3d70 msgr2=0x7efee415a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee2800640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee40b3d70 0x7efee415a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.573+0000 7efee2800640 1 -- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 msgr2=0x7efee41fb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efee2800640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee41fb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efee2800640 1 -- 172.21.15.32:0/1601526129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee41593b0 con 0x7efee4162c90 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efee2800640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 0x7efee41584d0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7efedc00e7f0 tx=0x7efedc00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc017070 con 0x7efee4162c90 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc00ee40 con 0x7efee4162c90 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee4159690 con 0x7efee4162c90 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc00cb80 con 0x7efee4162c90 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.574+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee4159c50 con 0x7efee4162c90 2024-03-06T07:38:47.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.576+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee404f3a0 con 0x7efee4162c90 2024-03-06T07:38:47.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.577+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efedc01a070 con 0x7efee4162c90 2024-03-06T07:38:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.578+0000 7efeda000640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efec006b520 0x7efec006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.579+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efedc08a070 con 0x7efee4162c90 2024-03-06T07:38:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.579+0000 7efee1e00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efec006b520 0x7efec006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.579+0000 7efee1e00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efec006b520 0x7efec006d9e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efed40012e0 tx=0x7efed4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.582+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efedc058f20 con 0x7efee4162c90 2024-03-06T07:38:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.833+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efee40ae910 con 0x7efee4162c90 2024-03-06T07:38:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.834+0000 7efeda000640 1 -- 172.21.15.32:0/1601526129 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7efedc058820 con 0x7efee4162c90 2024-03-06T07:38:47.837 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:47.837 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efec006b520 msgr2=0x7efec006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efec006b520 0x7efec006d9e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efed40012e0 tx=0x7efed4009040 comp rx=0 tx=0).stop 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 msgr2=0x7efee41584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 0x7efee41584d0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7efedc00e7f0 tx=0x7efedc00ebf0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 shutdown_connections 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efec006b520 0x7efec006d9e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efee4162c90 0x7efee41584d0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efee40b6000 0x7efee41fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.837+0000 7efee8c00640 1 --2- 172.21.15.32:0/1601526129 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efee40b3d70 0x7efee415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.838+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 >> 172.21.15.32:0/1601526129 conn(0x7efee406c2a0 msgr2=0x7efee40b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.838+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 shutdown_connections 2024-03-06T07:38:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:47.838+0000 7efee8c00640 1 -- 172.21.15.32:0/1601526129 wait complete. 2024-03-06T07:38:48.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:48.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:49.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.296+0000 7f4ea4c00640 1 Processor -- start 2024-03-06T07:38:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.297+0000 7f4ea4c00640 1 -- start start 2024-03-06T07:38:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.298+0000 7f4ea4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 0x7f4ea00b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.298+0000 7f4ea4c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea006bec0 con 0x7f4ea006afb0 2024-03-06T07:38:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.298+0000 7f4ea4c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea00b0670 con 0x7f4ea015e200 2024-03-06T07:38:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.298+0000 7f4ea4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea00ae380 con 0x7f4ea00afdf0 2024-03-06T07:38:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.299+0000 7f4e9e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ea006afb0 0x7f4ea006dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58810/0 (socket says 172.21.15.32:58810) 2024-03-06T07:38:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.299+0000 7f4e9e800640 1 -- 172.21.15.32:0/3704106699 learned_addr learned my addr 172.21.15.32:0/3704106699 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.299+0000 7f4e9d400640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 0x7f4ea00b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.299+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710783711 0 0) 0x7f4ea006bec0 con 0x7f4ea006afb0 2024-03-06T07:38:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.299+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e84003520 con 0x7f4ea006afb0 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797795605 0 0) 0x7f4e84003520 con 0x7f4ea006afb0 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 msgr2=0x7f4ea00b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 0x7f4ea00b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 >> v1:172.21.15.39:6789/0 conn(0x7f4ea015e200 legacy=0x7f4ea00af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea00ae660 con 0x7f4ea006afb0 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f4e98001690 con 0x7f4ea006afb0 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 >> v1:172.21.15.32:6789/0 conn(0x7f4ea006afb0 legacy=0x7f4ea006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4e84003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e84004800 0x7f4e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 0x7f4e840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea00ae380 con 0x7f4e84003a50 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8400ba70 con 0x7f4e84004800 2024-03-06T07:38:49.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8400bc40 con 0x7f4ea00afdf0 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9de00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4e84003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9e800640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 0x7f4e840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.300+0000 7f4e9d400640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e84004800 0x7f4e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.301+0000 7f4e9de00640 1 -- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 msgr2=0x7f4e840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.301+0000 7f4e9de00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea00afdf0 0x7f4e840051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.301+0000 7f4e9de00640 1 -- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e84004800 msgr2=0x7f4e84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.301+0000 7f4e9de00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e84004800 0x7f4e84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.301+0000 7f4e9de00640 1 -- 172.21.15.32:0/3704106699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e84003520 con 0x7f4e84003a50 2024-03-06T07:38:49.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.301+0000 7f4e9d400640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e84004800 0x7f4e84004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:49.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.302+0000 7f4e9de00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4e84003e30 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f4e90001f60 tx=0x7f4e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2f14efbd554920 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.302+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90001b30 con 0x7f4e84003a50 2024-03-06T07:38:49.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.302+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9002d6e0 con 0x7f4e84003a50 2024-03-06T07:38:49.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.302+0000 7f4e9ca00640 1 -- 172.21.15.32:0/3704106699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9002dad0 con 0x7f4e84003a50 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4ea4c00640 1 -- 172.21.15.32:0/3704106699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 msgr2=0x7f4e84003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/3704106699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4e84003e30 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f4e90001f60 tx=0x7f4e84005830 comp rx=0 tx=0).stop 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4e9e800640 1 -- 172.21.15.32:0/3704106699 reap_dead start 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4ea4c00640 1 -- 172.21.15.32:0/3704106699 shutdown_connections 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4ea4c00640 1 -- 172.21.15.32:0/3704106699 >> 172.21.15.32:0/3704106699 conn(0x7f4ea00b2ae0 msgr2=0x7f4ea00b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4ea4c00640 1 -- 172.21.15.32:0/3704106699 shutdown_connections 2024-03-06T07:38:49.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.303+0000 7f4ea4c00640 1 -- 172.21.15.32:0/3704106699 wait complete. 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.305+0000 7f4ea4c00640 1 Processor -- start 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.305+0000 7f4ea4c00640 1 -- start start 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4ea4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4ea014c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4ea4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea015d100 0x7f4ea014bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4ea4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ea006ec00 0x7f4ea006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4ea4c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea00b3de0 con 0x7f4e84003a50 2024-03-06T07:38:49.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4ea4c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea00aec20 con 0x7f4ea006ec00 2024-03-06T07:38:49.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4ea4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea00b3f60 con 0x7f4ea015d100 2024-03-06T07:38:49.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4ea014c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4ea014c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52032/0 (socket says 172.21.15.32:52032) 2024-03-06T07:38:49.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9de00640 1 -- 172.21.15.32:0/352619937 learned_addr learned my addr 172.21.15.32:0/352619937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9d400640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea015d100 0x7f4ea014bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9de00640 1 -- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea015d100 msgr2=0x7f4ea014bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9de00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea015d100 0x7f4ea014bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9e800640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ea006ec00 0x7f4ea006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.306+0000 7f4e9de00640 1 -- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ea006ec00 msgr2=0x7f4ea006f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e9de00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ea006ec00 0x7f4ea006f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e9de00640 1 -- 172.21.15.32:0/352619937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea015d580 con 0x7f4e84003a50 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e9e800640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ea006ec00 0x7f4ea006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e9de00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4ea014c3c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f4e9002c120 tx=0x7f4e900015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90044020 con 0x7f4e84003a50 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e90003e10 con 0x7f4e84003a50 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90001b30 con 0x7f4e84003a50 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea015e830 con 0x7f4e84003a50 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.307+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea015edf0 con 0x7f4e84003a50 2024-03-06T07:38:49.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.309+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea006f5c0 con 0x7f4e84003a50 2024-03-06T07:38:49.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.310+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e90047020 con 0x7f4e84003a50 2024-03-06T07:38:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.311+0000 7f4e96000640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e7c06b500 0x7f4e7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.311+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4e900ac500 con 0x7f4e84003a50 2024-03-06T07:38:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.312+0000 7f4e9d400640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e7c06b500 0x7f4e7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.312+0000 7f4e9d400640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e7c06b500 0x7f4e7c06d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4e88006ce0 tx=0x7f4e88008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.316+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e9007b330 con 0x7f4e84003a50 2024-03-06T07:38:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:49.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.554+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ea004f3a0 con 0x7f4e84003a50 2024-03-06T07:38:49.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.555+0000 7f4e96000640 1 -- 172.21.15.32:0/352619937 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f4e9007ac30 con 0x7f4e84003a50 2024-03-06T07:38:49.557 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:49.557 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e7c06b500 msgr2=0x7f4e7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e7c06b500 0x7f4e7c06d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4e88006ce0 tx=0x7f4e88008040 comp rx=0 tx=0).stop 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 msgr2=0x7f4ea014c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4ea014c3c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f4e9002c120 tx=0x7f4e900015b0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 shutdown_connections 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e7c06b500 0x7f4e7c06d9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ea006ec00 0x7f4ea006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4ea015d100 0x7f4ea014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.557+0000 7f4ea4c00640 1 --2- 172.21.15.32:0/352619937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e84003a50 0x7f4ea014c3c0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.558+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 >> 172.21.15.32:0/352619937 conn(0x7f4ea00b2ae0 msgr2=0x7f4ea00b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.558+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 shutdown_connections 2024-03-06T07:38:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:49.558+0000 7f4ea4c00640 1 -- 172.21.15.32:0/352619937 wait complete. 2024-03-06T07:38:49.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:49.780 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:50.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:51.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.003+0000 7fad1f200640 1 Processor -- start 2024-03-06T07:38:51.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.004+0000 7fad1f200640 1 -- start start 2024-03-06T07:38:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.005+0000 7fad1f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fad2015e0b0 0x7fad200cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.005+0000 7fad1f200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad200b3710 con 0x7fad200d4a10 2024-03-06T07:38:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.005+0000 7fad1f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad200b39f0 con 0x7fad200d0e50 2024-03-06T07:38:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.005+0000 7fad1f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad200b09e0 con 0x7fad2015e0b0 2024-03-06T07:38:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.006+0000 7fad15400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fad200d4a10 0x7fad200d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58818/0 (socket says 172.21.15.32:58818) 2024-03-06T07:38:51.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.006+0000 7fad15400640 1 -- 172.21.15.32:0/61517113 learned_addr learned my addr 172.21.15.32:0/61517113 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.006+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265226227 0 0) 0x7fad200b3710 con 0x7fad200d4a10 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.006+0000 7fad1de00640 1 --2- 172.21.15.32:0/61517113 >> v2:172.21.15.39:3300/0 conn(0x7fad2015e0b0 0x7fad200cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.006+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad04003520 con 0x7fad200d4a10 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804740424 0 0) 0x7fad04003520 con 0x7fad200d4a10 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 >> v2:172.21.15.39:3300/0 conn(0x7fad2015e0b0 msgr2=0x7fad200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 --2- 172.21.15.32:0/61517113 >> v2:172.21.15.39:3300/0 conn(0x7fad2015e0b0 0x7fad200cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 >> v1:172.21.15.39:6789/0 conn(0x7fad200d0e50 legacy=0x7fad200b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad200b3cd0 con 0x7fad200d4a10 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fad08001670 con 0x7fad200d4a10 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 >> v1:172.21.15.32:6789/0 conn(0x7fad200d4a10 legacy=0x7fad200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04003a00 0x7fad04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad04004800 0x7fad04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad040051c0 0x7fad0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0400c090 con 0x7fad04004800 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0400c240 con 0x7fad040051c0 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0400c440 con 0x7fad04003a00 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad15400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad04004800 0x7fad04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.007+0000 7fad1e800640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad040051c0 0x7fad0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad1de00640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04003a00 0x7fad04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad15400640 1 -- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04003a00 msgr2=0x7fad04003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad15400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04003a00 0x7fad04003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad15400640 1 -- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad040051c0 msgr2=0x7fad0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad15400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad040051c0 0x7fad0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad15400640 1 -- 172.21.15.32:0/61517113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04003520 con 0x7fad04004800 2024-03-06T07:38:51.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.008+0000 7fad1e800640 1 -- 172.21.15.32:0/61517113 reap_dead start 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.009+0000 7fad15400640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad04004800 0x7fad04004c80 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c0e0 tx=0x7fad04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b746e53941cd784f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.009+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08007c40 con 0x7fad04004800 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.009+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad08030070 con 0x7fad04004800 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.009+0000 7fad1d400640 1 -- 172.21.15.32:0/61517113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08005d70 con 0x7fad04004800 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 -- 172.21.15.32:0/61517113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad04004800 msgr2=0x7fad04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad04004800 0x7fad04004c80 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c0e0 tx=0x7fad04005830 comp rx=0 tx=0).stop 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 -- 172.21.15.32:0/61517113 shutdown_connections 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 --2- 172.21.15.32:0/61517113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad04004800 0x7fad04004c80 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 -- 172.21.15.32:0/61517113 >> 172.21.15.32:0/61517113 conn(0x7fad200d3d30 msgr2=0x7fad200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:51.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 -- 172.21.15.32:0/61517113 shutdown_connections 2024-03-06T07:38:51.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.010+0000 7fad1f200640 1 -- 172.21.15.32:0/61517113 wait complete. 2024-03-06T07:38:51.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.012+0000 7fad1f200640 1 Processor -- start 2024-03-06T07:38:51.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.012+0000 7fad1f200640 1 -- start start 2024-03-06T07:38:51.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad1f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04004800 0x7fad2015e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad1f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 0x7fad200b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad1f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad2014fcf0 0x7fad20150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad1f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad200cfc60 con 0x7fad200cd8e0 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad1f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2015e490 con 0x7fad2014fcf0 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad1f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad200d0740 con 0x7fad04004800 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad15400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 0x7fad200b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad15400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 0x7fad200b8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52038/0 (socket says 172.21.15.32:52038) 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.013+0000 7fad15400640 1 -- 172.21.15.32:0/2751029137 learned_addr learned my addr 172.21.15.32:0/2751029137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad1de00640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04004800 0x7fad2015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad1e800640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad2014fcf0 0x7fad20150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad15400640 1 -- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04004800 msgr2=0x7fad2015e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad15400640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04004800 0x7fad2015e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad15400640 1 -- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad2014fcf0 msgr2=0x7fad20150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad15400640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad2014fcf0 0x7fad20150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad15400640 1 -- 172.21.15.32:0/2751029137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2015ef10 con 0x7fad200cd8e0 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad1de00640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04004800 0x7fad2015e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad1e800640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad2014fcf0 0x7fad20150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad15400640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 0x7fad200b8d90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fad0802c480 tx=0x7fad0802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08004bf0 con 0x7fad200cd8e0 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad08000ed0 con 0x7fad200cd8e0 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08007070 con 0x7fad200cd8e0 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2015f1f0 con 0x7fad200cd8e0 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.014+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2015f7b0 con 0x7fad200cd8e0 2024-03-06T07:38:51.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.016+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2004f330 con 0x7fad200cd8e0 2024-03-06T07:38:51.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.017+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad08048020 con 0x7fad200cd8e0 2024-03-06T07:38:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.017+0000 7fad16a00640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b4d0 0x7facfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.018+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad080ab5c0 con 0x7fad200cd8e0 2024-03-06T07:38:51.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.018+0000 7fad1de00640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b4d0 0x7facfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.019+0000 7fad1de00640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b4d0 0x7facfc06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fad20069ad0 tx=0x7fad10008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:51.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.022+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad0807a400 con 0x7fad200cd8e0 2024-03-06T07:38:51.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:51.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.266+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad2015d2f0 con 0x7fad200cd8e0 2024-03-06T07:38:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.267+0000 7fad16a00640 1 -- 172.21.15.32:0/2751029137 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fad08079d00 con 0x7fad200cd8e0 2024-03-06T07:38:51.270 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:51.270 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b4d0 msgr2=0x7facfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b4d0 0x7facfc06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fad20069ad0 tx=0x7fad10008040 comp rx=0 tx=0).stop 2024-03-06T07:38:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 msgr2=0x7fad200b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 0x7fad200b8d90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fad0802c480 tx=0x7fad0802e070 comp rx=0 tx=0).stop 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 shutdown_connections 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7facfc06b4d0 0x7facfc06d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad2014fcf0 0x7fad20150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad200cd8e0 0x7fad200b8d90 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 --2- 172.21.15.32:0/2751029137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fad04004800 0x7fad2015e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 >> 172.21.15.32:0/2751029137 conn(0x7fad200d3d30 msgr2=0x7fad200b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 shutdown_connections 2024-03-06T07:38:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:51.270+0000 7fad1f200640 1 -- 172.21.15.32:0/2751029137 wait complete. 2024-03-06T07:38:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:51.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:51.495 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:52.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.721+0000 7f08c9000640 1 Processor -- start 2024-03-06T07:38:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.721+0000 7f08c9000640 1 -- start start 2024-03-06T07:38:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c9000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f08c40cd060 0x7f08c415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08c415e810 0x7f08c40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c9000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c415ed10 con 0x7f08c41552b0 2024-03-06T07:38:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c9000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c415eff0 con 0x7f08c415e810 2024-03-06T07:38:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c9000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c415f2d0 con 0x7f08c40cd060 2024-03-06T07:38:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08c415e810 0x7f08c40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08c415e810 0x7f08c40b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52046/0 (socket says 172.21.15.32:52046) 2024-03-06T07:38:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08c3e00640 1 -- 172.21.15.32:0/2063009412 learned_addr learned my addr 172.21.15.32:0/2063009412 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.723+0000 7f08bbe00640 1 --2- 172.21.15.32:0/2063009412 >> v2:172.21.15.39:3300/0 conn(0x7f08c40cd060 0x7f08c415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.724+0000 7f08c3e00640 1 -- 172.21.15.32:0/2063009412 >> v2:172.21.15.39:3300/0 conn(0x7f08c40cd060 msgr2=0x7f08c415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.724+0000 7f08c3e00640 1 --2- 172.21.15.32:0/2063009412 >> v2:172.21.15.39:3300/0 conn(0x7f08c40cd060 0x7f08c415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.724+0000 7f08c3e00640 1 -- 172.21.15.32:0/2063009412 >> v1:172.21.15.32:6789/0 conn(0x7f08c41552b0 legacy=0x7f08c40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.724+0000 7f08c3e00640 1 -- 172.21.15.32:0/2063009412 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c415f5b0 con 0x7f08c415e810 2024-03-06T07:38:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.724+0000 7f08bbe00640 1 --2- 172.21.15.32:0/2063009412 >> v2:172.21.15.39:3300/0 conn(0x7f08c40cd060 0x7f08c415b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:52.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.725+0000 7f08c3e00640 1 --2- 172.21.15.32:0/2063009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c40b3cb0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f08c40cf1d0 tx=0x7f08bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b65fdda9c63d6a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.725+0000 7f08c2a00640 1 -- 172.21.15.32:0/2063009412 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc02ca80 con 0x7f08c415e810 2024-03-06T07:38:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.725+0000 7f08c2a00640 1 -- 172.21.15.32:0/2063009412 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08bc02cc20 con 0x7f08c415e810 2024-03-06T07:38:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.725+0000 7f08c2a00640 1 -- 172.21.15.32:0/2063009412 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc0343f0 con 0x7f08c415e810 2024-03-06T07:38:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 -- 172.21.15.32:0/2063009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 msgr2=0x7f08c40b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 --2- 172.21.15.32:0/2063009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c40b3cb0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f08c40cf1d0 tx=0x7f08bc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 -- 172.21.15.32:0/2063009412 shutdown_connections 2024-03-06T07:38:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 --2- 172.21.15.32:0/2063009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c40b3cb0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 --2- 172.21.15.32:0/2063009412 >> v2:172.21.15.39:3300/0 conn(0x7f08c40cd060 0x7f08c415b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 -- 172.21.15.32:0/2063009412 >> 172.21.15.32:0/2063009412 conn(0x7f08c406b3d0 msgr2=0x7f08c406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 -- 172.21.15.32:0/2063009412 shutdown_connections 2024-03-06T07:38:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.726+0000 7f08c9000640 1 -- 172.21.15.32:0/2063009412 wait complete. 2024-03-06T07:38:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.728+0000 7f08c9000640 1 Processor -- start 2024-03-06T07:38:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.728+0000 7f08c9000640 1 -- start start 2024-03-06T07:38:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c9000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08c40cd060 0x7f08c4157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08c41552b0 0x7f08c415d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c4141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c9000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c406dd30 con 0x7f08c415e810 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c40b3760 con 0x7f08c41552b0 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c9000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c406deb0 con 0x7f08c40cd060 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c4141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c4141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52060/0 (socket says 172.21.15.32:52060) 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 -- 172.21.15.32:0/4154168408 learned_addr learned my addr 172.21.15.32:0/4154168408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08bbe00640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08c41552b0 0x7f08c415d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3400640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08c40cd060 0x7f08c4157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 -- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08c40cd060 msgr2=0x7f08c4157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08c40cd060 0x7f08c4157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 -- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08c41552b0 msgr2=0x7f08c415d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08c41552b0 0x7f08c415d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.729+0000 7f08c3e00640 1 -- 172.21.15.32:0/4154168408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c415a0a0 con 0x7f08c415e810 2024-03-06T07:38:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.730+0000 7f08c3e00640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c4141250 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f08bc036040 tx=0x7f08bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.730+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc043020 con 0x7f08c415e810 2024-03-06T07:38:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.730+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08bc03e450 con 0x7f08c415e810 2024-03-06T07:38:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.730+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc009510 con 0x7f08c415e810 2024-03-06T07:38:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.730+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c415a380 con 0x7f08c415e810 2024-03-06T07:38:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.730+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c415a940 con 0x7f08c415e810 2024-03-06T07:38:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.731+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c404f330 con 0x7f08c415e810 2024-03-06T07:38:52.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.733+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08bc046020 con 0x7f08c415e810 2024-03-06T07:38:52.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.734+0000 7f08bb400640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.734+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f08bc0acc00 con 0x7f08c415e810 2024-03-06T07:38:52.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.735+0000 7f08c3400640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.735+0000 7f08c3400640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f08b40015a0 tx=0x7f08b4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.739+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08bc04d030 con 0x7f08c415e810 2024-03-06T07:38:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.979+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08c40cf4a0 con 0x7f08c415e810 2024-03-06T07:38:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.980+0000 7f08bb400640 1 -- 172.21.15.32:0/4154168408 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f08bc07b9a0 con 0x7f08c415e810 2024-03-06T07:38:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:52.983 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f08a006b4d0 msgr2=0x7f08a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f08b40015a0 tx=0x7f08b4009040 comp rx=0 tx=0).stop 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 msgr2=0x7f08c4141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c4141250 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f08bc036040 tx=0x7f08bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 shutdown_connections 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08c415e810 0x7f08c4141250 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08c41552b0 0x7f08c415d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 --2- 172.21.15.32:0/4154168408 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f08c40cd060 0x7f08c4157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.983+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 >> 172.21.15.32:0/4154168408 conn(0x7f08c406b3d0 msgr2=0x7f08c40cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.984+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 shutdown_connections 2024-03-06T07:38:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:52.984+0000 7f08c9000640 1 -- 172.21.15.32:0/4154168408 wait complete. 2024-03-06T07:38:53.208 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:53.208 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:54.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.437+0000 7f03cdc00640 1 Processor -- start 2024-03-06T07:38:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.437+0000 7f03cdc00640 1 -- start start 2024-03-06T07:38:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.440+0000 7f03cdc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f03c806c8b0 0x7f03c806ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03cdc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80cf1b0 con 0x7f03c8066d00 2024-03-06T07:38:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03cdc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c806d420 con 0x7f03c80ce370 2024-03-06T07:38:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03cdc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c815e470 con 0x7f03c806c8b0 2024-03-06T07:38:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03c7400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f03c8066d00 0x7f03c815ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58832/0 (socket says 172.21.15.32:58832) 2024-03-06T07:38:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03c7400640 1 -- 172.21.15.32:0/3421289212 learned_addr learned my addr 172.21.15.32:0/3421289212 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:54.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03c7e00640 1 --2- 172.21.15.32:0/3421289212 >> v2:172.21.15.39:3300/0 conn(0x7f03c806c8b0 0x7f03c806ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879623295 0 0) 0x7f03c80cf1b0 con 0x7f03c8066d00 2024-03-06T07:38:54.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.441+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ac003520 con 0x7f03c8066d00 2024-03-06T07:38:54.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 622854126 0 0) 0x7f03ac003520 con 0x7f03c8066d00 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 >> v2:172.21.15.39:3300/0 conn(0x7f03c806c8b0 msgr2=0x7f03c806ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 --2- 172.21.15.32:0/3421289212 >> v2:172.21.15.39:3300/0 conn(0x7f03c806c8b0 0x7f03c806ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 >> v1:172.21.15.39:6789/0 conn(0x7f03c80ce370 legacy=0x7f03c80ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c815e690 con 0x7f03c8066d00 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f03b4001670 con 0x7f03c8066d00 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 >> v1:172.21.15.32:6789/0 conn(0x7f03c8066d00 legacy=0x7f03c815ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03ac003a00 0x7f03ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03ac004800 0x7f03ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c815e470 con 0x7f03ac0051c0 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac00c0c0 con 0x7f03ac003a00 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.442+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac00c2c0 con 0x7f03ac004800 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03cd200640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03c7e00640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03ac003a00 0x7f03ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03c7400640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03ac004800 0x7f03ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03cd200640 1 -- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03ac004800 msgr2=0x7f03ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03cd200640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03ac004800 0x7f03ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03cd200640 1 -- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03ac003a00 msgr2=0x7f03ac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03cd200640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03ac003a00 0x7f03ac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.443+0000 7f03cd200640 1 -- 172.21.15.32:0/3421289212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac003520 con 0x7f03ac0051c0 2024-03-06T07:38:54.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.445+0000 7f03cd200640 1 -- 172.21.15.32:0/3421289212 reap_dead start 2024-03-06T07:38:54.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.445+0000 7f03cd200640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03ac00b9c0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f03c0001e50 tx=0x7f03ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ce67f834e2fff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.445+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0001a70 con 0x7f03ac0051c0 2024-03-06T07:38:54.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.445+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c0003b90 con 0x7f03ac0051c0 2024-03-06T07:38:54.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.445+0000 7f03c6a00640 1 -- 172.21.15.32:0/3421289212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c00050d0 con 0x7f03ac0051c0 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 -- 172.21.15.32:0/3421289212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 msgr2=0x7f03ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03ac00b9c0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f03c0001e50 tx=0x7f03ac005830 comp rx=0 tx=0).stop 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 -- 172.21.15.32:0/3421289212 shutdown_connections 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 --2- 172.21.15.32:0/3421289212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03ac00b9c0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 -- 172.21.15.32:0/3421289212 >> 172.21.15.32:0/3421289212 conn(0x7f03c80ccf70 msgr2=0x7f03c80cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 -- 172.21.15.32:0/3421289212 shutdown_connections 2024-03-06T07:38:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.446+0000 7f03cdc00640 1 -- 172.21.15.32:0/3421289212 wait complete. 2024-03-06T07:38:54.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.448+0000 7f03cdc00640 1 Processor -- start 2024-03-06T07:38:54.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.449+0000 7f03cdc00640 1 -- start start 2024-03-06T07:38:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03c8156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cdc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03c8161800 0x7f03c8161c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cdc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03c80adc00 0x7f03c80ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cdc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806f2a0 con 0x7f03ac0051c0 2024-03-06T07:38:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cdc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806f120 con 0x7f03c80adc00 2024-03-06T07:38:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cdc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c815ea00 con 0x7f03c8161800 2024-03-06T07:38:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03c8156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03c8156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52082/0 (socket says 172.21.15.32:52082) 2024-03-06T07:38:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 -- 172.21.15.32:0/619307390 learned_addr learned my addr 172.21.15.32:0/619307390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03cd200640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03c80adc00 0x7f03c80ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7400640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03c8161800 0x7f03c8161c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 -- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03c8161800 msgr2=0x7f03c8161c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03c8161800 0x7f03c8161c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 -- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03c80adc00 msgr2=0x7f03c80ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03c80adc00 0x7f03c80ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.450+0000 7f03c7e00640 1 -- 172.21.15.32:0/619307390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80ae930 con 0x7f03ac0051c0 2024-03-06T07:38:54.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.451+0000 7f03c7e00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03c8156b90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f03b8011910 tx=0x7f03b8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.451+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b800fa90 con 0x7f03ac0051c0 2024-03-06T07:38:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.451+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b800fc30 con 0x7f03ac0051c0 2024-03-06T07:38:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.451+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c80aec10 con 0x7f03ac0051c0 2024-03-06T07:38:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.451+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8014440 con 0x7f03ac0051c0 2024-03-06T07:38:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.451+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c80af1d0 con 0x7f03ac0051c0 2024-03-06T07:38:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.453+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c81621c0 con 0x7f03ac0051c0 2024-03-06T07:38:54.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.454+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03b8012040 con 0x7f03ac0051c0 2024-03-06T07:38:54.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.455+0000 7f03bfe00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f03a806b500 0x7f03a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.455+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f03b808a870 con 0x7f03ac0051c0 2024-03-06T07:38:54.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.456+0000 7f03c7400640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f03a806b500 0x7f03a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.456+0000 7f03c7400640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f03a806b500 0x7f03a806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f03b4001eb0 tx=0x7f03b4000f50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.460+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03b8019030 con 0x7f03ac0051c0 2024-03-06T07:38:54.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.720+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c804f330 con 0x7f03ac0051c0 2024-03-06T07:38:54.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.720+0000 7f03bfe00640 1 -- 172.21.15.32:0/619307390 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f03b8059610 con 0x7f03ac0051c0 2024-03-06T07:38:54.723 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:54.723 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:54.723 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:54.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f03a806b500 msgr2=0x7f03a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f03a806b500 0x7f03a806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f03b4001eb0 tx=0x7f03b4000f50 comp rx=0 tx=0).stop 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 msgr2=0x7f03c8156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03c8156b90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f03b8011910 tx=0x7f03b8011dd0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 shutdown_connections 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f03a806b500 0x7f03a806d9c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03c80adc00 0x7f03c80ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f03c8161800 0x7f03c8161c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 --2- 172.21.15.32:0/619307390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03ac0051c0 0x7f03c8156b90 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 >> 172.21.15.32:0/619307390 conn(0x7f03c80ccf70 msgr2=0x7f03c80713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 shutdown_connections 2024-03-06T07:38:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:54.724+0000 7f03cdc00640 1 -- 172.21.15.32:0/619307390 wait complete. 2024-03-06T07:38:54.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:54.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:55.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:56.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.216+0000 7f8de5c00640 1 Processor -- start 2024-03-06T07:38:56.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.217+0000 7f8de5c00640 1 -- start start 2024-03-06T07:38:56.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.218+0000 7f8de5c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8de00cf5c0 0x7f8de00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.218+0000 7f8de5c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de0071050 0x7f8de0071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.218+0000 7f8de5c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de00b4860 con 0x7f8de00b3c90 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.218+0000 7f8de5c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de00ce160 con 0x7f8de00cf5c0 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.218+0000 7f8de5c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00b4b40 con 0x7f8de0071050 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.219+0000 7f8ddf400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de0071050 0x7f8de0071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.219+0000 7f8dde000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8de00cf5c0 0x7f8de00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.219+0000 7f8dde000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8de00cf5c0 0x7f8de00cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:38012/0 (socket says 172.21.15.32:38012) 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.219+0000 7f8ddf400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de0071050 0x7f8de0071490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:50382/0 (socket says 172.21.15.32:50382) 2024-03-06T07:38:56.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.219+0000 7f8dde000640 1 -- 172.21.15.32:0/2985386580 learned_addr learned my addr 172.21.15.32:0/2985386580 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:56.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.219+0000 7f8dde000640 1 -- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de0071050 msgr2=0x7f8de0071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.220+0000 7f8dde000640 1 --2- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de0071050 0x7f8de0071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.220+0000 7f8dde000640 1 -- 172.21.15.32:0/2985386580 >> v1:172.21.15.39:6789/0 conn(0x7f8de00b3c90 legacy=0x7f8de00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.220+0000 7f8dde000640 1 -- 172.21.15.32:0/2985386580 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00b4e20 con 0x7f8de00cf5c0 2024-03-06T07:38:56.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.221+0000 7f8dde000640 1 --2- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de00cf5c0 0x7f8de00cdc20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009870 tx=0x7f8dc802dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=171aafa55598cfb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.221+0000 7f8ddd600640 1 -- 172.21.15.32:0/2985386580 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc803b070 con 0x7f8de00cf5c0 2024-03-06T07:38:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.221+0000 7f8ddd600640 1 -- 172.21.15.32:0/2985386580 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc802e9e0 con 0x7f8de00cf5c0 2024-03-06T07:38:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.221+0000 7f8ddd600640 1 -- 172.21.15.32:0/2985386580 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc802edb0 con 0x7f8de00cf5c0 2024-03-06T07:38:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 -- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de00cf5c0 msgr2=0x7f8de00cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de00cf5c0 0x7f8de00cdc20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009870 tx=0x7f8dc802dc30 comp rx=0 tx=0).stop 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 -- 172.21.15.32:0/2985386580 shutdown_connections 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de0071050 0x7f8de0071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2985386580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de00cf5c0 0x7f8de00cdc20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 -- 172.21.15.32:0/2985386580 >> 172.21.15.32:0/2985386580 conn(0x7f8de006afb0 msgr2=0x7f8de006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 -- 172.21.15.32:0/2985386580 shutdown_connections 2024-03-06T07:38:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.222+0000 7f8de5c00640 1 -- 172.21.15.32:0/2985386580 wait complete. 2024-03-06T07:38:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.224+0000 7f8de5c00640 1 Processor -- start 2024-03-06T07:38:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.224+0000 7f8de5c00640 1 -- start start 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8de5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de0071050 0x7f8de014ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8de5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 0x7f8de01f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8de5c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de00cf5c0 0x7f8de00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8de5c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00719d0 con 0x7f8de00b3c90 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8de5c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de015b340 con 0x7f8de0071050 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8de5c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0071b50 con 0x7f8de00cf5c0 2024-03-06T07:38:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8dde000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 0x7f8de01f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8dde000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 0x7f8de01f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52088/0 (socket says 172.21.15.32:52088) 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.225+0000 7f8dde000640 1 -- 172.21.15.32:0/2365614565 learned_addr learned my addr 172.21.15.32:0/2365614565 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8ddea00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de0071050 0x7f8de014ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8ddf400640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de00cf5c0 0x7f8de00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8dde000640 1 -- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de00cf5c0 msgr2=0x7f8de00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8dde000640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de00cf5c0 0x7f8de00c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8dde000640 1 -- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de0071050 msgr2=0x7f8de014ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8dde000640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de0071050 0x7f8de014ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8dde000640 1 -- 172.21.15.32:0/2365614565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00cade0 con 0x7f8de00b3c90 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.226+0000 7f8dde000640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 0x7f8de01f39d0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f8dc802dc70 tx=0x7f8dc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.227+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8046070 con 0x7f8de00b3c90 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.227+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8036d80 con 0x7f8de00b3c90 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.227+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc803b070 con 0x7f8de00b3c90 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.227+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de00cb0c0 con 0x7f8de00b3c90 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.227+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de00cb680 con 0x7f8de00b3c90 2024-03-06T07:38:56.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.228+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de004f3a0 con 0x7f8de00b3c90 2024-03-06T07:38:56.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.229+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dc803f070 con 0x7f8de00b3c90 2024-03-06T07:38:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.230+0000 7f8dd6a00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.230+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8dc80ab660 con 0x7f8de00b3c90 2024-03-06T07:38:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.230+0000 7f8ddea00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.231+0000 7f8ddea00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0001640 tx=0x7f8dd0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.235+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8dc807a530 con 0x7f8de00b3c90 2024-03-06T07:38:56.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.476+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8de006e1a0 con 0x7f8de00b3c90 2024-03-06T07:38:56.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.477+0000 7f8dd6a00640 1 -- 172.21.15.32:0/2365614565 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f8dc8079e30 con 0x7f8de00b3c90 2024-03-06T07:38:56.479 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:56.480 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:56.480 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8dbc06b4d0 msgr2=0x7f8dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0001640 tx=0x7f8dd0009040 comp rx=0 tx=0).stop 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 msgr2=0x7f8de01f39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 0x7f8de01f39d0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f8dc802dc70 tx=0x7f8dc80046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 shutdown_connections 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8de00cf5c0 0x7f8de00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de00b3c90 0x7f8de01f39d0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 --2- 172.21.15.32:0/2365614565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8de0071050 0x7f8de014ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 >> 172.21.15.32:0/2365614565 conn(0x7f8de006afb0 msgr2=0x7f8de015a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 shutdown_connections 2024-03-06T07:38:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:56.480+0000 7f8de5c00640 1 -- 172.21.15.32:0/2365614565 wait complete. 2024-03-06T07:38:56.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:56.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:57.416 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:57.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:57.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.926+0000 7fcdd4a00640 1 Processor -- start 2024-03-06T07:38:57.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.927+0000 7fcdd4a00640 1 -- start start 2024-03-06T07:38:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.928+0000 7fcdd4a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcdd006ef10 0x7fcdd00cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.928+0000 7fcdd4a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd00cf1b0 con 0x7fcdd0066d00 2024-03-06T07:38:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.928+0000 7fcdd4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd006d420 con 0x7fcdd006c8b0 2024-03-06T07:38:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.928+0000 7fcdd4a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd015e470 con 0x7fcdd006ef10 2024-03-06T07:38:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.929+0000 7fcdcea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcdd0066d00 0x7fcdd00ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45738/0 (socket says 172.21.15.32:45738) 2024-03-06T07:38:57.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.929+0000 7fcdcea00640 1 -- 172.21.15.32:0/480210574 learned_addr learned my addr 172.21.15.32:0/480210574 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.929+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/480210574 >> v2:172.21.15.39:3300/0 conn(0x7fcdd006ef10 0x7fcdd00cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.929+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582500227 0 0) 0x7fcdd00cf1b0 con 0x7fcdd0066d00 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.929+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb4003520 con 0x7fcdd0066d00 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494303130 0 0) 0x7fcdd006d420 con 0x7fcdd006c8b0 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd00cf1b0 con 0x7fcdd006c8b0 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3418080347 0 0) 0x7fcdb4003520 con 0x7fcdd0066d00 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 >> v2:172.21.15.39:3300/0 conn(0x7fcdd006ef10 msgr2=0x7fcdd00cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 --2- 172.21.15.32:0/480210574 >> v2:172.21.15.39:3300/0 conn(0x7fcdd006ef10 0x7fcdd00cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 >> v1:172.21.15.39:6789/0 conn(0x7fcdd006c8b0 legacy=0x7fcdd015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd015e690 con 0x7fcdd0066d00 2024-03-06T07:38:57.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fcdbc001670 con 0x7fcdd0066d00 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 >> v1:172.21.15.32:6789/0 conn(0x7fcdd0066d00 legacy=0x7fcdd00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb4003b90 0x7fcdb4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdb4004800 0x7fcdb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb40051c0 0x7fcdb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd015e470 con 0x7fcdb40051c0 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb400c090 con 0x7fcdb4004800 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb400c290 con 0x7fcdb4003b90 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.930+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb40051c0 0x7fcdb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcf400640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb4003b90 0x7fcdb4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcea00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdb4004800 0x7fcdb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcfe00640 1 -- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb4003b90 msgr2=0x7fcdb4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb4003b90 0x7fcdb4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcfe00640 1 -- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdb4004800 msgr2=0x7fcdb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:57.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdb4004800 0x7fcdb4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.931+0000 7fcdcfe00640 1 -- 172.21.15.32:0/480210574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4003520 con 0x7fcdb40051c0 2024-03-06T07:38:57.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.932+0000 7fcdcfe00640 1 -- 172.21.15.32:0/480210574 reap_dead start 2024-03-06T07:38:57.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.932+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb40051c0 0x7fcdb400b9c0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009fc0 tx=0x7fcdb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6c0ae0d6ccf7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.932+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8009c10 con 0x7fcdb40051c0 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.932+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc802cc60 con 0x7fcdb40051c0 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.932+0000 7fcdce000640 1 -- 172.21.15.32:0/480210574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8003bd0 con 0x7fcdb40051c0 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 -- 172.21.15.32:0/480210574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb40051c0 msgr2=0x7fcdb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb40051c0 0x7fcdb400b9c0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009fc0 tx=0x7fcdb4005830 comp rx=0 tx=0).stop 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 -- 172.21.15.32:0/480210574 shutdown_connections 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/480210574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdb40051c0 0x7fcdb400b9c0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 -- 172.21.15.32:0/480210574 >> 172.21.15.32:0/480210574 conn(0x7fcdd00ccf70 msgr2=0x7fcdd00cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 -- 172.21.15.32:0/480210574 shutdown_connections 2024-03-06T07:38:57.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.933+0000 7fcdd4a00640 1 -- 172.21.15.32:0/480210574 wait complete. 2024-03-06T07:38:57.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.935+0000 7fcdd4a00640 1 Processor -- start 2024-03-06T07:38:57.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.935+0000 7fcdd4a00640 1 -- start start 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.936+0000 7fcdd4a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb40051c0 0x7fcdd00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.936+0000 7fcdd4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 0x7fcdd0155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdd4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd013ff70 0x7fcdd00aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdd4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd015f220 con 0x7fcdd0154570 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdd4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd015f0a0 con 0x7fcdd013ff70 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdd4a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd015f3a0 con 0x7fcdb40051c0 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 0x7fcdd0155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 0x7fcdd0155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50862/0 (socket says 172.21.15.32:50862) 2024-03-06T07:38:57.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 -- 172.21.15.32:0/2906242931 learned_addr learned my addr 172.21.15.32:0/2906242931 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcf400640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb40051c0 0x7fcdd00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 -- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb40051c0 msgr2=0x7fcdd00b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb40051c0 0x7fcdd00b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd013ff70 0x7fcdd00aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 -- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd013ff70 msgr2=0x7fcdd00aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd013ff70 0x7fcdd00aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcea00640 1 -- 172.21.15.32:0/2906242931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0156080 con 0x7fcdd0154570 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcf400640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb40051c0 0x7fcdd00b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.937+0000 7fcdcfe00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd013ff70 0x7fcdd00aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:57.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.938+0000 7fcdcea00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 0x7fcdd0155b40 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc001400 tx=0x7fcdbc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.938+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc019070 con 0x7fcdd0154570 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.938+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc0072d0 con 0x7fcdd0154570 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.938+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0047e0 con 0x7fcdd0154570 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.938+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd0154aa0 con 0x7fcdd0154570 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.938+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd0155060 con 0x7fcdd0154570 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.940+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd0069060 con 0x7fcdd0154570 2024-03-06T07:38:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.940+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdbc01c070 con 0x7fcdd0154570 2024-03-06T07:38:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.941+0000 7fcdc7400640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdb006b500 0x7fcdb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.941+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcdbc089c30 con 0x7fcdd0154570 2024-03-06T07:38:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.942+0000 7fcdcf400640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdb006b500 0x7fcdb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.943+0000 7fcdcf400640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdb006b500 0x7fcdb006d9c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0007850 tx=0x7fcdc0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:57.946+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdd0069060 con 0x7fcdd0154570 2024-03-06T07:38:58.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.189+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd004f330 con 0x7fcdd0154570 2024-03-06T07:38:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.190+0000 7fcdc7400640 1 -- 172.21.15.32:0/2906242931 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fcdbc0589d0 con 0x7fcdd0154570 2024-03-06T07:38:58.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:58.192 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.192+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdb006b500 msgr2=0x7fcdb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.192+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdb006b500 0x7fcdb006d9c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0007850 tx=0x7fcdc0009040 comp rx=0 tx=0).stop 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.192+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 msgr2=0x7fcdd0155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.192+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 0x7fcdd0155b40 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc001400 tx=0x7fcdbc00dc00 comp rx=0 tx=0).stop 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.192+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 shutdown_connections 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcdb006b500 0x7fcdb006d9c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd013ff70 0x7fcdd00aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd0154570 0x7fcdd0155b40 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 --2- 172.21.15.32:0/2906242931 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcdb40051c0 0x7fcdd00b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 >> 172.21.15.32:0/2906242931 conn(0x7fcdd00ccf70 msgr2=0x7fcdd006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 shutdown_connections 2024-03-06T07:38:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:58.193+0000 7fcdd4a00640 1 -- 172.21.15.32:0/2906242931 wait complete. 2024-03-06T07:38:58.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:38:58.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:38:59.418 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:59.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.645+0000 7f523ce00640 1 Processor -- start 2024-03-06T07:38:59.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.646+0000 7f523ce00640 1 -- start start 2024-03-06T07:38:59.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f523ce00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f523815a930 0x7f52380ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f523ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 0x7f52380ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f523ce00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5238071640 con 0x7f523815a550 2024-03-06T07:38:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f523ce00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52380af6f0 con 0x7f523815a930 2024-03-06T07:38:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f523ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5238071920 con 0x7f52380ce9f0 2024-03-06T07:38:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f5235e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f523815a550 0x7f52380aefe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45752/0 (socket says 172.21.15.32:45752) 2024-03-06T07:38:59.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.648+0000 7f5235e00640 1 -- 172.21.15.32:0/3359902609 learned_addr learned my addr 172.21.15.32:0/3359902609 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962599155 0 0) 0x7f5238071640 con 0x7f523815a550 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f522fe00640 1 --2- 172.21.15.32:0/3359902609 >> v2:172.21.15.39:3300/0 conn(0x7f523815a930 0x7f52380ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5236800640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 0x7f52380ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c003520 con 0x7f523815a550 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2069393668 0 0) 0x7f521c003520 con 0x7f523815a550 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 msgr2=0x7f52380ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 0x7f52380ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 >> v2:172.21.15.39:3300/0 conn(0x7f523815a930 msgr2=0x7f52380ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 --2- 172.21.15.32:0/3359902609 >> v2:172.21.15.39:3300/0 conn(0x7f523815a930 0x7f52380ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238071c00 con 0x7f523815a550 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f5228001690 con 0x7f523815a550 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 >> v1:172.21.15.32:6789/0 conn(0x7f523815a550 legacy=0x7f52380aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 0x7f521c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c004080 0x7f521c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f521c004d40 0x7f521c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52380af6f0 con 0x7f521c004080 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238071920 con 0x7f521c004d40 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.649+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c00bc10 con 0x7f52380ce9f0 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f522fe00640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c004080 0x7f521c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f5235e00640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 0x7f521c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f5236800640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f521c004d40 0x7f521c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f522fe00640 1 -- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 msgr2=0x7f521c003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f522fe00640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f52380ce9f0 0x7f521c003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f522fe00640 1 -- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f521c004d40 msgr2=0x7f521c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f522fe00640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f521c004d40 0x7f521c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.650+0000 7f522fe00640 1 -- 172.21.15.32:0/3359902609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c003520 con 0x7f521c004080 2024-03-06T07:38:59.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.652+0000 7f522fe00640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c004080 0x7f521c004800 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f522000ba00 tx=0x7f521c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d035b05ba8cd18f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522000b600 con 0x7f521c004080 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5220038070 con 0x7f521c004080 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f5235400640 1 -- 172.21.15.32:0/3359902609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220033660 con 0x7f521c004080 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f523ce00640 1 -- 172.21.15.32:0/3359902609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c004080 msgr2=0x7f521c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f523ce00640 1 --2- 172.21.15.32:0/3359902609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c004080 0x7f521c004800 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f522000ba00 tx=0x7f521c005830 comp rx=0 tx=0).stop 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f5236800640 1 -- 172.21.15.32:0/3359902609 reap_dead start 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f523ce00640 1 -- 172.21.15.32:0/3359902609 shutdown_connections 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f523ce00640 1 -- 172.21.15.32:0/3359902609 >> 172.21.15.32:0/3359902609 conn(0x7f523806afb0 msgr2=0x7f523806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f523ce00640 1 -- 172.21.15.32:0/3359902609 shutdown_connections 2024-03-06T07:38:59.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.653+0000 7f523ce00640 1 -- 172.21.15.32:0/3359902609 wait complete. 2024-03-06T07:38:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.655+0000 7f523ce00640 1 Processor -- start 2024-03-06T07:38:59.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.655+0000 7f523ce00640 1 -- start start 2024-03-06T07:38:59.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f523ce00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f521c004080 0x7f523815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f523ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 0x7f523815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f523ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238151730 0x7f5238151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f523ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52380b52b0 con 0x7f52381f48f0 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f523ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52380b5130 con 0x7f5238151730 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f523ce00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52380cdc20 con 0x7f521c004080 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f522fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 0x7f523815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f522fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 0x7f523815b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50888/0 (socket says 172.21.15.32:50888) 2024-03-06T07:38:59.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.656+0000 7f522fe00640 1 -- 172.21.15.32:0/921578282 learned_addr learned my addr 172.21.15.32:0/921578282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:38:59.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f5236800640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238151730 0x7f5238151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f5235e00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f521c004080 0x7f523815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522fe00640 1 -- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f521c004080 msgr2=0x7f523815ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522fe00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f521c004080 0x7f523815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522fe00640 1 -- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238151730 msgr2=0x7f5238151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522fe00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238151730 0x7f5238151bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522fe00640 1 -- 172.21.15.32:0/921578282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238152100 con 0x7f52381f48f0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522fe00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 0x7f523815b890 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f5220035040 tx=0x7f5220004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.657+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220042070 con 0x7f52381f48f0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.658+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5220004020 con 0x7f52381f48f0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.658+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220004410 con 0x7f52381f48f0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.658+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52381598c0 con 0x7f52381f48f0 2024-03-06T07:38:59.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.658+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52380ca450 con 0x7f52381f48f0 2024-03-06T07:38:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.659+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523804f3a0 con 0x7f52381f48f0 2024-03-06T07:38:59.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.660+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5220003610 con 0x7f52381f48f0 2024-03-06T07:38:59.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.661+0000 7f522e000640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f521406b4d0 0x7f521406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.661+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f52200ac1b0 con 0x7f52381f48f0 2024-03-06T07:38:59.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.661+0000 7f5235e00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f521406b4d0 0x7f521406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.662+0000 7f5235e00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f521406b4d0 0x7f521406d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5228001050 tx=0x7f5228001890 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.666+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f522007aec0 con 0x7f52381f48f0 2024-03-06T07:38:59.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.981+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52380af360 con 0x7f52381f48f0 2024-03-06T07:38:59.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.982+0000 7f522e000640 1 -- 172.21.15.32:0/921578282 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f522007a7c0 con 0x7f52381f48f0 2024-03-06T07:38:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:38:59.986 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:38:59.986 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:38:59.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f521406b4d0 msgr2=0x7f521406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f521406b4d0 0x7f521406d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5228001050 tx=0x7f5228001890 comp rx=0 tx=0).stop 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 msgr2=0x7f523815b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 0x7f523815b890 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f5220035040 tx=0x7f5220004730 comp rx=0 tx=0).stop 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 shutdown_connections 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f521406b4d0 0x7f521406d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238151730 0x7f5238151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52381f48f0 0x7f523815b890 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 --2- 172.21.15.32:0/921578282 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f521c004080 0x7f523815ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 >> 172.21.15.32:0/921578282 conn(0x7f523806afb0 msgr2=0x7f523806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 shutdown_connections 2024-03-06T07:38:59.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:38:59.985+0000 7f523ce00640 1 -- 172.21.15.32:0/921578282 wait complete. 2024-03-06T07:39:00.207 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:00.208 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:01.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.429+0000 7f4e6ca00640 1 Processor -- start 2024-03-06T07:39:01.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.429+0000 7f4e6ca00640 1 -- start start 2024-03-06T07:39:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.431+0000 7f4e6ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e681552b0 0x7f4e6815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.431+0000 7f4e6ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e680b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.431+0000 7f4e6ca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6815ed10 con 0x7f4e6815e810 2024-03-06T07:39:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.431+0000 7f4e6ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6815eff0 con 0x7f4e681552b0 2024-03-06T07:39:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.431+0000 7f4e6ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6815f2d0 con 0x7f4e680cd060 2024-03-06T07:39:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.432+0000 7f4e5ea00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4e6815e810 0x7f4e680cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45754/0 (socket says 172.21.15.32:45754) 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.432+0000 7f4e5ea00640 1 -- 172.21.15.32:0/4198217109 learned_addr learned my addr 172.21.15.32:0/4198217109 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.432+0000 7f4e67400640 1 --2- 172.21.15.32:0/4198217109 >> v2:172.21.15.39:3300/0 conn(0x7f4e681552b0 0x7f4e6815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.432+0000 7f4e67e00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e680b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.432+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161081353 0 0) 0x7f4e6815ed10 con 0x7f4e6815e810 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.432+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e4c003520 con 0x7f4e6815e810 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402722915 0 0) 0x7f4e4c003520 con 0x7f4e6815e810 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 msgr2=0x7f4e680b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e680b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 >> v2:172.21.15.39:3300/0 conn(0x7f4e681552b0 msgr2=0x7f4e6815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 --2- 172.21.15.32:0/4198217109 >> v2:172.21.15.39:3300/0 conn(0x7f4e681552b0 0x7f4e6815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6815f5b0 con 0x7f4e6815e810 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f4e50001670 con 0x7f4e6815e810 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 >> v1:172.21.15.32:6789/0 conn(0x7f4e6815e810 legacy=0x7f4e680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c004800 0x7f4e4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6815eff0 con 0x7f4e4c0039b0 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6815f2d0 con 0x7f4e4c004800 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c00dc50 con 0x7f4e680cd060 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.433+0000 7f4e67400640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e5ea00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c004800 0x7f4e4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67400640 1 -- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 msgr2=0x7f4e4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67e00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67400640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67400640 1 -- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c004800 msgr2=0x7f4e4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67400640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c004800 0x7f4e4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67400640 1 -- 172.21.15.32:0/4198217109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c003520 con 0x7f4e4c0039b0 2024-03-06T07:39:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.434+0000 7f4e67e00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680cd060 0x7f4e4c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:01.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.435+0000 7f4e67400640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e4c003dd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009ac0 tx=0x7f4e4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdec4cf751bd74b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.435+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58039070 con 0x7f4e4c0039b0 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.435+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5802c710 con 0x7f4e4c0039b0 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.435+0000 7f4e66a00640 1 -- 172.21.15.32:0/4198217109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58009690 con 0x7f4e4c0039b0 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e6ca00640 1 -- 172.21.15.32:0/4198217109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 msgr2=0x7f4e4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/4198217109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e4c003dd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009ac0 tx=0x7f4e4c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e67e00640 1 -- 172.21.15.32:0/4198217109 reap_dead start 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e6ca00640 1 -- 172.21.15.32:0/4198217109 shutdown_connections 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e6ca00640 1 -- 172.21.15.32:0/4198217109 >> 172.21.15.32:0/4198217109 conn(0x7f4e6806b3d0 msgr2=0x7f4e6806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e6ca00640 1 -- 172.21.15.32:0/4198217109 shutdown_connections 2024-03-06T07:39:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.436+0000 7f4e6ca00640 1 -- 172.21.15.32:0/4198217109 wait complete. 2024-03-06T07:39:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.438+0000 7f4e6ca00640 1 Processor -- start 2024-03-06T07:39:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.438+0000 7f4e6ca00640 1 -- start start 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.439+0000 7f4e6ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e6814a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.439+0000 7f4e6ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e681e4630 0x7f4e68149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.439+0000 7f4e6ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680b4150 0x7f4e680b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.439+0000 7f4e6ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6806dd30 con 0x7f4e4c0039b0 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.439+0000 7f4e6ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680b3760 con 0x7f4e681e4630 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.439+0000 7f4e6ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6806deb0 con 0x7f4e680b4150 2024-03-06T07:39:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e6814a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e6814a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50904/0 (socket says 172.21.15.32:50904) 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 -- 172.21.15.32:0/527858680 learned_addr learned my addr 172.21.15.32:0/527858680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e5ea00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e681e4630 0x7f4e68149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 -- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680b4150 msgr2=0x7f4e680b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680b4150 0x7f4e680b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 -- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e681e4630 msgr2=0x7f4e68149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e681e4630 0x7f4e68149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 -- 172.21.15.32:0/527858680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6814a490 con 0x7f4e4c0039b0 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.440+0000 7f4e67400640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e6814a690 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f4e58000c00 tx=0x7f4e58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.441+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58039050 con 0x7f4e4c0039b0 2024-03-06T07:39:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.441+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5803a040 con 0x7f4e4c0039b0 2024-03-06T07:39:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.441+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e680b1530 con 0x7f4e4c0039b0 2024-03-06T07:39:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.441+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58005bb0 con 0x7f4e4c0039b0 2024-03-06T07:39:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.441+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6814f360 con 0x7f4e4c0039b0 2024-03-06T07:39:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.442+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6815b530 con 0x7f4e4c0039b0 2024-03-06T07:39:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.443+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e58047020 con 0x7f4e4c0039b0 2024-03-06T07:39:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.444+0000 7f4e5fe00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e4406b500 0x7f4e4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.444+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4e580abb50 con 0x7f4e4c0039b0 2024-03-06T07:39:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.444+0000 7f4e5ea00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e4406b500 0x7f4e4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.445+0000 7f4e5ea00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e4406b500 0x7f4e4406d9c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4e50001eb0 tx=0x7f4e50000ed0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.451+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e6815b530 con 0x7f4e4c0039b0 2024-03-06T07:39:01.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.687+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e6804f330 con 0x7f4e4c0039b0 2024-03-06T07:39:01.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.689+0000 7f4e5fe00640 1 -- 172.21.15.32:0/527858680 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f4e6814a490 con 0x7f4e4c0039b0 2024-03-06T07:39:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:01.691 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:01.691 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e4406b500 msgr2=0x7f4e4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e4406b500 0x7f4e4406d9c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4e50001eb0 tx=0x7f4e50000ed0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 msgr2=0x7f4e6814a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e6814a690 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f4e58000c00 tx=0x7f4e58004730 comp rx=0 tx=0).stop 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 shutdown_connections 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4e4406b500 0x7f4e4406d9c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4e680b4150 0x7f4e680b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e681e4630 0x7f4e68149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 --2- 172.21.15.32:0/527858680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0039b0 0x7f4e6814a690 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.691+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 >> 172.21.15.32:0/527858680 conn(0x7f4e6806b3d0 msgr2=0x7f4e68151e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.692+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 shutdown_connections 2024-03-06T07:39:01.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:01.692+0000 7f4e6ca00640 1 -- 172.21.15.32:0/527858680 wait complete. 2024-03-06T07:39:01.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:01.915 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:02.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.141+0000 7f6b83200640 1 Processor -- start 2024-03-06T07:39:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.142+0000 7f6b83200640 1 -- start start 2024-03-06T07:39:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b83200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6b840b0f90 0x7f6b840cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b83200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b840cc730 0x7f6b840d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b83200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840b1370 con 0x7f6b840d05e0 2024-03-06T07:39:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b83200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b840d0a10 con 0x7f6b840cc730 2024-03-06T07:39:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b83200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b840ccf80 con 0x7f6b840b0f90 2024-03-06T07:39:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b79400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b840cc730 0x7f6b840d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b79400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b840cc730 0x7f6b840d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50920/0 (socket says 172.21.15.32:50920) 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.143+0000 7f6b79400640 1 -- 172.21.15.32:0/2303666168 learned_addr learned my addr 172.21.15.32:0/2303666168 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.144+0000 7f6b81e00640 1 --2- 172.21.15.32:0/2303666168 >> v2:172.21.15.39:3300/0 conn(0x7f6b840b0f90 0x7f6b840cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.144+0000 7f6b79400640 1 -- 172.21.15.32:0/2303666168 >> v2:172.21.15.39:3300/0 conn(0x7f6b840b0f90 msgr2=0x7f6b840cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.144+0000 7f6b79400640 1 --2- 172.21.15.32:0/2303666168 >> v2:172.21.15.39:3300/0 conn(0x7f6b840b0f90 0x7f6b840cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.144+0000 7f6b79400640 1 -- 172.21.15.32:0/2303666168 >> v1:172.21.15.32:6789/0 conn(0x7f6b840d05e0 legacy=0x7f6b8415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.144+0000 7f6b79400640 1 -- 172.21.15.32:0/2303666168 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840cd260 con 0x7f6b840cc730 2024-03-06T07:39:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.144+0000 7f6b81e00640 1 --2- 172.21.15.32:0/2303666168 >> v2:172.21.15.39:3300/0 conn(0x7f6b840b0f90 0x7f6b840cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:03.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.145+0000 7f6b79400640 1 --2- 172.21.15.32:0/2303666168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b840d00a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f6b84071b80 tx=0x7f6b6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dbd27ef0422ab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:03.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.145+0000 7f6b81400640 1 -- 172.21.15.32:0/2303666168 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c02ca80 con 0x7f6b840cc730 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.145+0000 7f6b81400640 1 -- 172.21.15.32:0/2303666168 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c02cc20 con 0x7f6b840cc730 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.145+0000 7f6b81400640 1 -- 172.21.15.32:0/2303666168 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0343f0 con 0x7f6b840cc730 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 -- 172.21.15.32:0/2303666168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 msgr2=0x7f6b840d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 --2- 172.21.15.32:0/2303666168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b840d00a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f6b84071b80 tx=0x7f6b6c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 -- 172.21.15.32:0/2303666168 shutdown_connections 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 --2- 172.21.15.32:0/2303666168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b840d00a0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 --2- 172.21.15.32:0/2303666168 >> v2:172.21.15.39:3300/0 conn(0x7f6b840b0f90 0x7f6b840cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 -- 172.21.15.32:0/2303666168 >> 172.21.15.32:0/2303666168 conn(0x7f6b840d1ae0 msgr2=0x7f6b840d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 -- 172.21.15.32:0/2303666168 shutdown_connections 2024-03-06T07:39:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.146+0000 7f6b83200640 1 -- 172.21.15.32:0/2303666168 wait complete. 2024-03-06T07:39:03.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.149+0000 7f6b83200640 1 Processor -- start 2024-03-06T07:39:03.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.149+0000 7f6b83200640 1 -- start start 2024-03-06T07:39:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b83200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b840b0f90 0x7f6b84157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b83200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b841f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b83200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b840d05e0 0x7f6b841f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b83200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840ccba0 con 0x7f6b840cc730 2024-03-06T07:39:03.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b83200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840cd820 con 0x7f6b840d05e0 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b83200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840ccd20 con 0x7f6b840b0f90 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b841f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b841f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50932/0 (socket says 172.21.15.32:50932) 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 -- 172.21.15.32:0/1267502615 learned_addr learned my addr 172.21.15.32:0/1267502615 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 -- 172.21.15.32:0/1267502615 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b840b0f90 msgr2=0x7f6b84157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b840b0f90 0x7f6b84157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 -- 172.21.15.32:0/1267502615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b840d05e0 msgr2=0x7f6b841f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b840d05e0 0x7f6b841f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.151+0000 7f6b79400640 1 -- 172.21.15.32:0/1267502615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8415b830 con 0x7f6b840cc730 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.152+0000 7f6b79400640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b841f4fd0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c000f80 tx=0x7f6b6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.152+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c043020 con 0x7f6b840cc730 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.152+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c03e450 con 0x7f6b840cc730 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.152+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c03e840 con 0x7f6b840cc730 2024-03-06T07:39:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.152+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8415bb10 con 0x7f6b840cc730 2024-03-06T07:39:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.152+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8415c0d0 con 0x7f6b840cc730 2024-03-06T07:39:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.154+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8404f330 con 0x7f6b840cc730 2024-03-06T07:39:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.154+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b6c046020 con 0x7f6b840cc730 2024-03-06T07:39:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.155+0000 7f6b7aa00640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b6006b520 0x7f6b6006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.155+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6b6c0aca00 con 0x7f6b840cc730 2024-03-06T07:39:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.156+0000 7f6b81e00640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b6006b520 0x7f6b6006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.156+0000 7f6b81e00640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b6006b520 0x7f6b6006d9e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.164+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b6c04d030 con 0x7f6b840cc730 2024-03-06T07:39:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.405+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8406f770 con 0x7f6b840cc730 2024-03-06T07:39:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.406+0000 7f6b7aa00640 1 -- 172.21.15.32:0/1267502615 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f6b6c07bad0 con 0x7f6b840cc730 2024-03-06T07:39:03.409 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:03.409 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.408+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b6006b520 msgr2=0x7f6b6006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.408+0000 7f6b83200640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b6006b520 0x7f6b6006d9e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74008040 comp rx=0 tx=0).stop 2024-03-06T07:39:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 msgr2=0x7f6b841f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b841f4fd0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c000f80 tx=0x7f6b6c004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 shutdown_connections 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b6006b520 0x7f6b6006d9e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b840d05e0 0x7f6b841f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b840cc730 0x7f6b841f4fd0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 --2- 172.21.15.32:0/1267502615 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b840b0f90 0x7f6b84157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 >> 172.21.15.32:0/1267502615 conn(0x7f6b840d1ae0 msgr2=0x7f6b84160fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 shutdown_connections 2024-03-06T07:39:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:03.409+0000 7f6b83200640 1 -- 172.21.15.32:0/1267502615 wait complete. 2024-03-06T07:39:03.422 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:03.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:03.633 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:04.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.873+0000 7f5fc1e00640 1 Processor -- start 2024-03-06T07:39:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.874+0000 7f5fc1e00640 1 -- start start 2024-03-06T07:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5fbc0d0cd0 0x7f5fbc0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc1e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fbc0d2dc0 con 0x7f5fbc15e070 2024-03-06T07:39:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc1e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fbc0d18c0 con 0x7f5fbc0d2960 2024-03-06T07:39:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc1e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d1ba0 con 0x7f5fbc0d0cd0 2024-03-06T07:39:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc0a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5fbc15e070 0x7f5fbc0cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45772/0 (socket says 172.21.15.32:45772) 2024-03-06T07:39:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc0a00640 1 -- 172.21.15.32:0/1358792275 learned_addr learned my addr 172.21.15.32:0/1358792275 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.875+0000 7f5fc1400640 1 --2- 172.21.15.32:0/1358792275 >> v2:172.21.15.32:3300/0 conn(0x7f5fbc0d0cd0 0x7f5fbc0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.876+0000 7f5fbb400640 1 -- 172.21.15.32:0/1358792275 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870856803 0 0) 0x7f5fbc0d2dc0 con 0x7f5fbc15e070 2024-03-06T07:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.876+0000 7f5fbb400640 1 -- 172.21.15.32:0/1358792275 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa4003520 con 0x7f5fbc15e070 2024-03-06T07:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.876+0000 7f5fc1400640 1 -- 172.21.15.32:0/1358792275 >> v1:172.21.15.39:6789/0 conn(0x7f5fbc0d2960 legacy=0x7f5fbc0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.876+0000 7f5fc1400640 1 -- 172.21.15.32:0/1358792275 >> v1:172.21.15.32:6789/0 conn(0x7f5fbc15e070 legacy=0x7f5fbc0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.876+0000 7f5fc1400640 1 -- 172.21.15.32:0/1358792275 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc0d1e80 con 0x7f5fbc0d0cd0 2024-03-06T07:39:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.877+0000 7f5fc1400640 1 --2- 172.21.15.32:0/1358792275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0b3630 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009d90 tx=0x7f5fb402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee649dfc65f1a728 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.877+0000 7f5fbb400640 1 -- 172.21.15.32:0/1358792275 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4039070 con 0x7f5fbc0d0cd0 2024-03-06T07:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.877+0000 7f5fbb400640 1 -- 172.21.15.32:0/1358792275 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb402ccf0 con 0x7f5fbc0d0cd0 2024-03-06T07:39:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.877+0000 7f5fbb400640 1 -- 172.21.15.32:0/1358792275 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4034640 con 0x7f5fbc0d0cd0 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.878+0000 7f5fc1e00640 1 -- 172.21.15.32:0/1358792275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d0cd0 msgr2=0x7f5fbc0b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.878+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/1358792275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0b3630 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009d90 tx=0x7f5fb402bf40 comp rx=0 tx=0).stop 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.878+0000 7f5fc1e00640 1 -- 172.21.15.32:0/1358792275 shutdown_connections 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.878+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/1358792275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0b3630 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.878+0000 7f5fc1e00640 1 -- 172.21.15.32:0/1358792275 >> 172.21.15.32:0/1358792275 conn(0x7f5fbc08d2a0 msgr2=0x7f5fbc0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.878+0000 7f5fc1e00640 1 -- 172.21.15.32:0/1358792275 shutdown_connections 2024-03-06T07:39:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.879+0000 7f5fc1e00640 1 -- 172.21.15.32:0/1358792275 wait complete. 2024-03-06T07:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.880+0000 7f5fc1e00640 1 Processor -- start 2024-03-06T07:39:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.880+0000 7f5fc1e00640 1 -- start start 2024-03-06T07:39:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.881+0000 7f5fc1e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.881+0000 7f5fc1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 0x7f5fbc153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.881+0000 7f5fc1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fbc15e070 0x7f5fbc0cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.881+0000 7f5fc1e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d45b0 con 0x7f5fbc0d2960 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.881+0000 7f5fc1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d2440 con 0x7f5fbc15e070 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.881+0000 7f5fc1e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d4730 con 0x7f5fbc0d0cd0 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 0x7f5fbc153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 0x7f5fbc153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50948/0 (socket says 172.21.15.32:50948) 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 -- 172.21.15.32:0/4181256519 learned_addr learned my addr 172.21.15.32:0/4181256519 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fc0a00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 -- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5fbc0d0cd0 msgr2=0x7f5fbc0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 -- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fbc15e070 msgr2=0x7f5fbc0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fbc15e070 0x7f5fbc0cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.882+0000 7f5fbbe00640 1 -- 172.21.15.32:0/4181256519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc0cd580 con 0x7f5fbc0d2960 2024-03-06T07:39:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.883+0000 7f5fbbe00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 0x7f5fbc153b10 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800c400 tx=0x7f5fa800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.883+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80052d0 con 0x7f5fbc0d2960 2024-03-06T07:39:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.883+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa8005490 con 0x7f5fbc0d2960 2024-03-06T07:39:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.883+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8007260 con 0x7f5fbc0d2960 2024-03-06T07:39:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.883+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fbc0cd820 con 0x7f5fbc0d2960 2024-03-06T07:39:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.883+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fbc0cdde0 con 0x7f5fbc0d2960 2024-03-06T07:39:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.884+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fbc0ce1c0 con 0x7f5fbc0d2960 2024-03-06T07:39:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.885+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fa8007400 con 0x7f5fbc0d2960 2024-03-06T07:39:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.886+0000 7f5fb8c00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.886+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5fa8089820 con 0x7f5fbc0d2960 2024-03-06T07:39:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.887+0000 7f5fc0a00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.887+0000 7f5fc0a00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00011e0 tx=0x7f5fb0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:04.891+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fbc0ce1c0 con 0x7f5fbc0d2960 2024-03-06T07:39:05.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.133+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fbc04f330 con 0x7f5fbc0d2960 2024-03-06T07:39:05.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.134+0000 7f5fb8c00640 1 -- 172.21.15.32:0/4181256519 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f5fa80585d0 con 0x7f5fbc0d2960 2024-03-06T07:39:05.136 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:05.136 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:05.137 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f9c06b4d0 msgr2=0x7f5f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00011e0 tx=0x7f5fb0009040 comp rx=0 tx=0).stop 2024-03-06T07:39:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 msgr2=0x7f5fbc153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 0x7f5fbc153b10 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800c400 tx=0x7f5fa800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 shutdown_connections 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5fbc15e070 0x7f5fbc0cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fbc0d2960 0x7f5fbc153b10 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 --2- 172.21.15.32:0/4181256519 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5fbc0d0cd0 0x7f5fbc0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 >> 172.21.15.32:0/4181256519 conn(0x7f5fbc08d2a0 msgr2=0x7f5fbc06d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 shutdown_connections 2024-03-06T07:39:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:05.137+0000 7f5fc1e00640 1 -- 172.21.15.32:0/4181256519 wait complete. 2024-03-06T07:39:05.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:05.360 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:05.424 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:06.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:06.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.589+0000 7f7bede00640 1 Processor -- start 2024-03-06T07:39:06.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.589+0000 7f7bede00640 1 -- start start 2024-03-06T07:39:06.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.590+0000 7f7bede00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be8070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.590+0000 7f7bede00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 0x7f7be80cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.590+0000 7f7bede00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be80cd0c0 con 0x7f7be815f450 2024-03-06T07:39:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.590+0000 7f7bede00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be80d29e0 con 0x7f7be80700c0 2024-03-06T07:39:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.590+0000 7f7bede00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80cd3a0 con 0x7f7be80d04b0 2024-03-06T07:39:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.591+0000 7f7bed400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 0x7f7be80cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.591+0000 7f7bed400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 0x7f7be80cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:38018/0 (socket says 172.21.15.32:38018) 2024-03-06T07:39:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.591+0000 7f7bed400640 1 -- 172.21.15.32:0/334867357 learned_addr learned my addr 172.21.15.32:0/334867357 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.591+0000 7f7be7e00640 1 --2- 172.21.15.32:0/334867357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be8070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7be7400640 1 -- 172.21.15.32:0/334867357 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721016292 0 0) 0x7f7be80cd0c0 con 0x7f7be815f450 2024-03-06T07:39:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7be7400640 1 -- 172.21.15.32:0/334867357 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bcc003520 con 0x7f7be815f450 2024-03-06T07:39:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7be7e00640 1 -- 172.21.15.32:0/334867357 >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 msgr2=0x7f7be80cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7be7e00640 1 --2- 172.21.15.32:0/334867357 >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 0x7f7be80cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7be7e00640 1 -- 172.21.15.32:0/334867357 >> v1:172.21.15.39:6789/0 conn(0x7f7be815f450 legacy=0x7f7be80cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7be7e00640 1 -- 172.21.15.32:0/334867357 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80cd680 con 0x7f7be80d04b0 2024-03-06T07:39:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.592+0000 7f7bed400640 1 --2- 172.21.15.32:0/334867357 >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 0x7f7be80cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.594+0000 7f7be7e00640 1 --2- 172.21.15.32:0/334867357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be8070990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40097a0 tx=0x7f7bd402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e26274b252308bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.594+0000 7f7be7400640 1 -- 172.21.15.32:0/334867357 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4035020 con 0x7f7be80d04b0 2024-03-06T07:39:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.594+0000 7f7be7400640 1 -- 172.21.15.32:0/334867357 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4031810 con 0x7f7be80d04b0 2024-03-06T07:39:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.594+0000 7f7be7400640 1 -- 172.21.15.32:0/334867357 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4031bc0 con 0x7f7be80d04b0 2024-03-06T07:39:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 -- 172.21.15.32:0/334867357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 msgr2=0x7f7be8070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 --2- 172.21.15.32:0/334867357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be8070990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40097a0 tx=0x7f7bd402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 -- 172.21.15.32:0/334867357 shutdown_connections 2024-03-06T07:39:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 --2- 172.21.15.32:0/334867357 >> v2:172.21.15.39:3300/0 conn(0x7f7be80700c0 0x7f7be80cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 --2- 172.21.15.32:0/334867357 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be8070990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 -- 172.21.15.32:0/334867357 >> 172.21.15.32:0/334867357 conn(0x7f7be80d1ae0 msgr2=0x7f7be80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 -- 172.21.15.32:0/334867357 shutdown_connections 2024-03-06T07:39:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.595+0000 7f7bede00640 1 -- 172.21.15.32:0/334867357 wait complete. 2024-03-06T07:39:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.597+0000 7f7bede00640 1 Processor -- start 2024-03-06T07:39:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.597+0000 7f7bede00640 1 -- start start 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7bede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 0x7f7be81f1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7bede00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be80c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7bede00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7be815f450 0x7f7be80c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7bede00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815f830 con 0x7f7be80700c0 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7bede00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80ae8b0 con 0x7f7be815f450 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7bede00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815f9b0 con 0x7f7be80d04b0 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7beca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 0x7f7be81f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7beca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 0x7f7be81f1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50950/0 (socket says 172.21.15.32:50950) 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.598+0000 7f7beca00640 1 -- 172.21.15.32:0/3055823913 learned_addr learned my addr 172.21.15.32:0/3055823913 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7be7e00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be80c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7bed400640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7be815f450 0x7f7be80c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7beca00640 1 -- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 msgr2=0x7f7be80c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7beca00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be80c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7beca00640 1 -- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7be815f450 msgr2=0x7f7be80c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7beca00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7be815f450 0x7f7be80c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7beca00640 1 -- 172.21.15.32:0/3055823913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80c5c20 con 0x7f7be80700c0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.599+0000 7f7beca00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 0x7f7be81f1390 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800c530 tx=0x7f7bd800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.600+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80052d0 con 0x7f7be80700c0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.600+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8005490 con 0x7f7be80700c0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.600+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8012870 con 0x7f7be80700c0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.600+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be80c5f00 con 0x7f7be80700c0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.600+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be8156c30 con 0x7f7be80700c0 2024-03-06T07:39:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.601+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be80c6420 con 0x7f7be80700c0 2024-03-06T07:39:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.604+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bd8012a10 con 0x7f7be80700c0 2024-03-06T07:39:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.605+0000 7f7be4c00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bc806b4d0 0x7f7bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.605+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7bd8089e80 con 0x7f7be80700c0 2024-03-06T07:39:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.606+0000 7f7be7e00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bc806b4d0 0x7f7bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.606+0000 7f7be7e00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bc806b4d0 0x7f7bc806d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036000 tx=0x7f7bd4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.611+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bd8058c30 con 0x7f7be80700c0 2024-03-06T07:39:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.890+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7be8156a20 con 0x7f7be80700c0 2024-03-06T07:39:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.891+0000 7f7be4c00640 1 -- 172.21.15.32:0/3055823913 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f7bd8058530 con 0x7f7be80700c0 2024-03-06T07:39:06.894 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:06.894 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.893+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bc806b4d0 msgr2=0x7f7bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.893+0000 7f7bede00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bc806b4d0 0x7f7bc806d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036000 tx=0x7f7bd4008030 comp rx=0 tx=0).stop 2024-03-06T07:39:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.893+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 msgr2=0x7f7be81f1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.893+0000 7f7bede00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 0x7f7be81f1390 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800c530 tx=0x7f7bd800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 shutdown_connections 2024-03-06T07:39:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7bc806b4d0 0x7f7bc806d990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7be815f450 0x7f7be80c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7be80d04b0 0x7f7be80c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 --2- 172.21.15.32:0/3055823913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be80700c0 0x7f7be81f1390 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 >> 172.21.15.32:0/3055823913 conn(0x7f7be80d1ae0 msgr2=0x7f7be80c8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 shutdown_connections 2024-03-06T07:39:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:06.894+0000 7f7bede00640 1 -- 172.21.15.32:0/3055823913 wait complete. 2024-03-06T07:39:07.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:07.121 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:07.426 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:08.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:08.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.350+0000 7ff73a200640 1 Processor -- start 2024-03-06T07:39:08.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.351+0000 7ff73a200640 1 -- start start 2024-03-06T07:39:08.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.352+0000 7ff73a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d4a10 0x7ff7340d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.352+0000 7ff73a200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7340d0c80 0x7ff7340b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.352+0000 7ff73a200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7340b3710 con 0x7ff73415e0b0 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.352+0000 7ff73a200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7340b09e0 con 0x7ff7340d0c80 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.352+0000 7ff73a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340cfa20 con 0x7ff7340d4a10 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff738e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff73415e0b0 0x7ff7340cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57922/0 (socket says 172.21.15.32:57922) 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff738e00640 1 -- 172.21.15.32:0/2192582861 learned_addr learned my addr 172.21.15.32:0/2192582861 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff739800640 1 --2- 172.21.15.32:0/2192582861 >> v2:172.21.15.32:3300/0 conn(0x7ff7340d0c80 0x7ff7340b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff733e00640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d4a10 0x7ff7340d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff739800640 1 -- 172.21.15.32:0/2192582861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d4a10 msgr2=0x7ff7340d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff739800640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d4a10 0x7ff7340d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff739800640 1 -- 172.21.15.32:0/2192582861 >> v1:172.21.15.32:6789/0 conn(0x7ff73415e0b0 legacy=0x7ff7340cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff739800640 1 -- 172.21.15.32:0/2192582861 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7340b39f0 con 0x7ff7340d0c80 2024-03-06T07:39:08.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.353+0000 7ff733e00640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d4a10 0x7ff7340d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.354+0000 7ff739800640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7340d0c80 0x7ff7340b44d0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009d90 tx=0x7ff72c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63603149f308cb92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff733400640 1 -- 172.21.15.32:0/2192582861 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c039070 con 0x7ff7340d0c80 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff733400640 1 -- 172.21.15.32:0/2192582861 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c02cc60 con 0x7ff7340d0c80 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff733400640 1 -- 172.21.15.32:0/2192582861 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c034660 con 0x7ff7340d0c80 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 -- 172.21.15.32:0/2192582861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7340d0c80 msgr2=0x7ff7340b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7340d0c80 0x7ff7340b44d0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009d90 tx=0x7ff72c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 -- 172.21.15.32:0/2192582861 shutdown_connections 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7340d0c80 0x7ff7340b44d0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 --2- 172.21.15.32:0/2192582861 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d4a10 0x7ff7340d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 -- 172.21.15.32:0/2192582861 >> 172.21.15.32:0/2192582861 conn(0x7ff7340d3d30 msgr2=0x7ff7340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 -- 172.21.15.32:0/2192582861 shutdown_connections 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.355+0000 7ff73a200640 1 -- 172.21.15.32:0/2192582861 wait complete. 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.357+0000 7ff73a200640 1 Processor -- start 2024-03-06T07:39:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.357+0000 7ff73a200640 1 -- start start 2024-03-06T07:39:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff73a200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d0c80 0x7ff734150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff73a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7340d4a10 0x7ff7340cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff73a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 0x7ff734160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff73a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73415e490 con 0x7ff73415e0b0 2024-03-06T07:39:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff73a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340b3fb0 con 0x7ff7340d4a10 2024-03-06T07:39:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff73a200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7340cfc60 con 0x7ff7340d0c80 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff739800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 0x7ff734160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff739800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 0x7ff734160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58070/0 (socket says 172.21.15.32:58070) 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.358+0000 7ff739800640 1 -- 172.21.15.32:0/3942727362 learned_addr learned my addr 172.21.15.32:0/3942727362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff733e00640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7340d4a10 0x7ff7340cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff739800640 1 -- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d0c80 msgr2=0x7ff734150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff738e00640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d0c80 0x7ff734150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff739800640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d0c80 0x7ff734150170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff739800640 1 -- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7340d4a10 msgr2=0x7ff7340cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff739800640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7340d4a10 0x7ff7340cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff739800640 1 -- 172.21.15.32:0/3942727362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7341506b0 con 0x7ff73415e0b0 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff733e00640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7340d4a10 0x7ff7340cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff739800640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 0x7ff734160490 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009eb0 tx=0x7ff72c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.359+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c039050 con 0x7ff73415e0b0 2024-03-06T07:39:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.360+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c045070 con 0x7ff73415e0b0 2024-03-06T07:39:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.360+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c03e830 con 0x7ff73415e0b0 2024-03-06T07:39:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.360+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7340cafa0 con 0x7ff73415e0b0 2024-03-06T07:39:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.360+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7340cb560 con 0x7ff73415e0b0 2024-03-06T07:39:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.361+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73404f330 con 0x7ff73415e0b0 2024-03-06T07:39:08.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.362+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff72c03e9d0 con 0x7ff73415e0b0 2024-03-06T07:39:08.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.363+0000 7ff730c00640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff71406b500 0x7ff71406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.364+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff72c0ad1d0 con 0x7ff73415e0b0 2024-03-06T07:39:08.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.364+0000 7ff738e00640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff71406b500 0x7ff71406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.364+0000 7ff738e00640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff71406b500 0x7ff71406d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff728000f60 tx=0x7ff728009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.368+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff72c07c090 con 0x7ff73415e0b0 2024-03-06T07:39:08.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.612+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7340b2be0 con 0x7ff73415e0b0 2024-03-06T07:39:08.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.613+0000 7ff730c00640 1 -- 172.21.15.32:0/3942727362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7ff72c07b990 con 0x7ff73415e0b0 2024-03-06T07:39:08.616 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:08.616 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:08.616 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff71406b500 msgr2=0x7ff71406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff71406b500 0x7ff71406d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff728000f60 tx=0x7ff728009040 comp rx=0 tx=0).stop 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 msgr2=0x7ff734160490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 0x7ff734160490 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009eb0 tx=0x7ff72c004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 shutdown_connections 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff71406b500 0x7ff71406d9c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff73415e0b0 0x7ff734160490 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7340d4a10 0x7ff7340cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 --2- 172.21.15.32:0/3942727362 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff7340d0c80 0x7ff734150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 >> 172.21.15.32:0/3942727362 conn(0x7ff7340d3d30 msgr2=0x7ff7340b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.616+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 shutdown_connections 2024-03-06T07:39:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:08.617+0000 7ff73a200640 1 -- 172.21.15.32:0/3942727362 wait complete. 2024-03-06T07:39:08.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:08.838 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:09.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:10.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.062+0000 7fa7eca00640 1 Processor -- start 2024-03-06T07:39:10.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.063+0000 7fa7eca00640 1 -- start start 2024-03-06T07:39:10.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.064+0000 7fa7eca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7e815e070 0x7fa7e80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.064+0000 7fa7eca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e80cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.064+0000 7fa7eca00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e80d18c0 con 0x7fa7e80cf410 2024-03-06T07:39:10.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.064+0000 7fa7eca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e80d1ba0 con 0x7fa7e815e070 2024-03-06T07:39:10.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.064+0000 7fa7eca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80d1e80 con 0x7fa7e80d0cd0 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.065+0000 7fa7dde00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7e80cf410 0x7fa7e80d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57936/0 (socket says 172.21.15.32:57936) 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.065+0000 7fa7dde00640 1 -- 172.21.15.32:0/3967652391 learned_addr learned my addr 172.21.15.32:0/3967652391 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.065+0000 7fa7e6800640 1 --2- 172.21.15.32:0/3967652391 >> v2:172.21.15.39:3300/0 conn(0x7fa7e815e070 0x7fa7e80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.065+0000 7fa7e7200640 1 --2- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e80cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.065+0000 7fa7e5e00640 1 -- 172.21.15.32:0/3967652391 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113322311 0 0) 0x7fa7e80d18c0 con 0x7fa7e80cf410 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.065+0000 7fa7e5e00640 1 -- 172.21.15.32:0/3967652391 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc003520 con 0x7fa7e80cf410 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.066+0000 7fa7e6800640 1 -- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 msgr2=0x7fa7e80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:10.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.066+0000 7fa7e6800640 1 --2- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e80cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.066+0000 7fa7e6800640 1 -- 172.21.15.32:0/3967652391 >> v1:172.21.15.32:6789/0 conn(0x7fa7e80cf410 legacy=0x7fa7e80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.066+0000 7fa7e6800640 1 -- 172.21.15.32:0/3967652391 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80d2160 con 0x7fa7e815e070 2024-03-06T07:39:10.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.067+0000 7fa7e6800640 1 --2- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e815e070 0x7fa7e80d2960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009d90 tx=0x7fa7d802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ca3af27ea28c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:10.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7e5e00640 1 -- 172.21.15.32:0/3967652391 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8039070 con 0x7fa7e815e070 2024-03-06T07:39:10.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7e5e00640 1 -- 172.21.15.32:0/3967652391 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d802cc60 con 0x7fa7e815e070 2024-03-06T07:39:10.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7e5e00640 1 -- 172.21.15.32:0/3967652391 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8034640 con 0x7fa7e815e070 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 -- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e815e070 msgr2=0x7fa7e80d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 --2- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e815e070 0x7fa7e80d2960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009d90 tx=0x7fa7d802beb0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 -- 172.21.15.32:0/3967652391 shutdown_connections 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 --2- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 --2- 172.21.15.32:0/3967652391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e815e070 0x7fa7e80d2960 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 -- 172.21.15.32:0/3967652391 >> 172.21.15.32:0/3967652391 conn(0x7fa7e808d2a0 msgr2=0x7fa7e80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.068+0000 7fa7eca00640 1 -- 172.21.15.32:0/3967652391 shutdown_connections 2024-03-06T07:39:10.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.069+0000 7fa7eca00640 1 -- 172.21.15.32:0/3967652391 wait complete. 2024-03-06T07:39:10.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.071+0000 7fa7eca00640 1 Processor -- start 2024-03-06T07:39:10.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.071+0000 7fa7eca00640 1 -- start start 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7eca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e80cf410 0x7fa7e8158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7eca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e815e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7eca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 0x7fa7e80b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7eca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80d4730 con 0x7fa7e815e070 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7eca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80d45b0 con 0x7fa7e80cf410 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7eca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b3630 con 0x7fa7e80d0cd0 2024-03-06T07:39:10.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7e7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 0x7fa7e80b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.072+0000 7fa7e7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 0x7fa7e80b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58072/0 (socket says 172.21.15.32:58072) 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 -- 172.21.15.32:0/129419853 learned_addr learned my addr 172.21.15.32:0/129419853 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e6800640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e80cf410 0x7fa7e8158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7dde00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 -- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 msgr2=0x7fa7e815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 -- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e80cf410 msgr2=0x7fa7e8158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e80cf410 0x7fa7e8158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 -- 172.21.15.32:0/129419853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e815edc0 con 0x7fa7e815e070 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.073+0000 7fa7e7200640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 0x7fa7e80b1ca0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000e850 tx=0x7fa7e000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.074+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0017070 con 0x7fa7e815e070 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.074+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e000c940 con 0x7fa7e815e070 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.074+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e815f0a0 con 0x7fa7e815e070 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.074+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e000cd30 con 0x7fa7e815e070 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.074+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e80b72e0 con 0x7fa7e815e070 2024-03-06T07:39:10.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.075+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e804f330 con 0x7fa7e815e070 2024-03-06T07:39:10.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.076+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7e001a070 con 0x7fa7e815e070 2024-03-06T07:39:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.077+0000 7fa7df400640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7c406b500 0x7fa7c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.077+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7e008a5e0 con 0x7fa7e815e070 2024-03-06T07:39:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.077+0000 7fa7e6800640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7c406b500 0x7fa7c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.078+0000 7fa7e6800640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7c406b500 0x7fa7c406d9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8036000 tx=0x7fa7d8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:10.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.081+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7e0059410 con 0x7fa7e815e070 2024-03-06T07:39:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.323+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7e80b70b0 con 0x7fa7e815e070 2024-03-06T07:39:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.324+0000 7fa7df400640 1 -- 172.21.15.32:0/129419853 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7fa7e0058d10 con 0x7fa7e815e070 2024-03-06T07:39:10.327 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:10.327 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7c406b500 msgr2=0x7fa7c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7c406b500 0x7fa7c406d9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8036000 tx=0x7fa7d8030000 comp rx=0 tx=0).stop 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 msgr2=0x7fa7e80b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 0x7fa7e80b1ca0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000e850 tx=0x7fa7e000ed10 comp rx=0 tx=0).stop 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 shutdown_connections 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa7c406b500 0x7fa7c406d9c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7e815e070 0x7fa7e80b1ca0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa7e80d0cd0 0x7fa7e815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 --2- 172.21.15.32:0/129419853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e80cf410 0x7fa7e8158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 >> 172.21.15.32:0/129419853 conn(0x7fa7e808d2a0 msgr2=0x7fa7e80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 shutdown_connections 2024-03-06T07:39:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:10.327+0000 7fa7eca00640 1 -- 172.21.15.32:0/129419853 wait complete. 2024-03-06T07:39:10.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:10.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:11.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:11.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.769+0000 7f6770a00640 1 Processor -- start 2024-03-06T07:39:11.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.769+0000 7f6770a00640 1 -- start start 2024-03-06T07:39:11.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f6770a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f676c070ee0 0x7f676c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f6770a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f676c0712c0 0x7f676c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f6770a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676c0ce090 con 0x7f676c072da0 2024-03-06T07:39:11.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f6770a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0ceca0 con 0x7f676c0712c0 2024-03-06T07:39:11.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f6770a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0ce370 con 0x7f676c070ee0 2024-03-06T07:39:11.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f676a800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f676c0712c0 0x7f676c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f676a800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f676c0712c0 0x7f676c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58084/0 (socket says 172.21.15.32:58084) 2024-03-06T07:39:11.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.771+0000 7f676a800640 1 -- 172.21.15.32:0/1744030769 learned_addr learned my addr 172.21.15.32:0/1744030769 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:11.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.772+0000 7f6769e00640 1 --2- 172.21.15.32:0/1744030769 >> v2:172.21.15.39:3300/0 conn(0x7f676c070ee0 0x7f676c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.772+0000 7f676a800640 1 -- 172.21.15.32:0/1744030769 >> v2:172.21.15.39:3300/0 conn(0x7f676c070ee0 msgr2=0x7f676c162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:11.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.772+0000 7f676a800640 1 --2- 172.21.15.32:0/1744030769 >> v2:172.21.15.39:3300/0 conn(0x7f676c070ee0 0x7f676c162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.772+0000 7f676a800640 1 -- 172.21.15.32:0/1744030769 >> v1:172.21.15.39:6789/0 conn(0x7f676c072da0 legacy=0x7f676c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:11.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.772+0000 7f676a800640 1 -- 172.21.15.32:0/1744030769 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c0ce650 con 0x7f676c0712c0 2024-03-06T07:39:11.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.773+0000 7f676a800640 1 --2- 172.21.15.32:0/1744030769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c06e7d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f676c073610 tx=0x7f676402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972ef16fb92080ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.773+0000 7f6768a00640 1 -- 172.21.15.32:0/1744030769 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676402ca80 con 0x7f676c0712c0 2024-03-06T07:39:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.773+0000 7f6768a00640 1 -- 172.21.15.32:0/1744030769 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676402cc20 con 0x7f676c0712c0 2024-03-06T07:39:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.773+0000 7f6768a00640 1 -- 172.21.15.32:0/1744030769 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67640343f0 con 0x7f676c0712c0 2024-03-06T07:39:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 -- 172.21.15.32:0/1744030769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 msgr2=0x7f676c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 --2- 172.21.15.32:0/1744030769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c06e7d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f676c073610 tx=0x7f676402bee0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 -- 172.21.15.32:0/1744030769 shutdown_connections 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 --2- 172.21.15.32:0/1744030769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c06e7d0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 --2- 172.21.15.32:0/1744030769 >> v2:172.21.15.39:3300/0 conn(0x7f676c070ee0 0x7f676c162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 -- 172.21.15.32:0/1744030769 >> 172.21.15.32:0/1744030769 conn(0x7f676c06afb0 msgr2=0x7f676c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 -- 172.21.15.32:0/1744030769 shutdown_connections 2024-03-06T07:39:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.774+0000 7f6770a00640 1 -- 172.21.15.32:0/1744030769 wait complete. 2024-03-06T07:39:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.777+0000 7f6770a00640 1 Processor -- start 2024-03-06T07:39:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.777+0000 7f6770a00640 1 -- start start 2024-03-06T07:39:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.778+0000 7f6770a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f676c070ee0 0x7f676c09d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6770a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c14c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6770a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f676c072da0 0x7f676c14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6770a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0cdda0 con 0x7f676c0712c0 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6770a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0cdc20 con 0x7f676c070ee0 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6770a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0cf130 con 0x7f676c072da0 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c14c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c14c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58100/0 (socket says 172.21.15.32:58100) 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 -- 172.21.15.32:0/1705567455 learned_addr learned my addr 172.21.15.32:0/1705567455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769e00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f676c070ee0 0x7f676c09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f676a800640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f676c072da0 0x7f676c14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 -- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f676c072da0 msgr2=0x7f676c14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f676c072da0 0x7f676c14f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 -- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f676c070ee0 msgr2=0x7f676c09d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f676c070ee0 0x7f676c09d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.779+0000 7f6769400640 1 -- 172.21.15.32:0/1705567455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c09da50 con 0x7f676c0712c0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.780+0000 7f6769400640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c14c7d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f675800b420 tx=0x7f675800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.780+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758011830 con 0x7f676c0712c0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.780+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6758016070 con 0x7f676c0712c0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.780+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c14cd10 con 0x7f676c0712c0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.780+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758011c10 con 0x7f676c0712c0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.780+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c09ca30 con 0x7f676c0712c0 2024-03-06T07:39:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.782+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c04f3a0 con 0x7f676c0712c0 2024-03-06T07:39:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.783+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f675801b460 con 0x7f676c0712c0 2024-03-06T07:39:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.784+0000 7f6762000640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f674c06b4d0 0x7f674c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.784+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6758089e10 con 0x7f676c0712c0 2024-03-06T07:39:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.784+0000 7f6769e00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f674c06b4d0 0x7f674c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.785+0000 7f6769e00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f674c06b4d0 0x7f674c06d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f675c006fd0 tx=0x7f675c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:11.789+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f675808d030 con 0x7f676c0712c0 2024-03-06T07:39:12.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.097+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676c0cc600 con 0x7f676c0712c0 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.098+0000 7f6762000640 1 -- 172.21.15.32:0/1705567455 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f6758020020 con 0x7f676c0712c0 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:12.101 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f674c06b4d0 msgr2=0x7f674c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f674c06b4d0 0x7f674c06d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f675c006fd0 tx=0x7f675c008040 comp rx=0 tx=0).stop 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 msgr2=0x7f676c14c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c14c7d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f675800b420 tx=0x7f675800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 shutdown_connections 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f674c06b4d0 0x7f674c06d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f676c072da0 0x7f676c14f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f676c0712c0 0x7f676c14c7d0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 --2- 172.21.15.32:0/1705567455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f676c070ee0 0x7f676c09d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 >> 172.21.15.32:0/1705567455 conn(0x7f676c06afb0 msgr2=0x7f676c0af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 shutdown_connections 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:12.100+0000 7f6770a00640 1 -- 172.21.15.32:0/1705567455 wait complete. 2024-03-06T07:39:12.321 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:12.321 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:13.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:13.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.542+0000 7f2b1fc00640 1 Processor -- start 2024-03-06T07:39:13.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.543+0000 7f2b1fc00640 1 -- start start 2024-03-06T07:39:13.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.544+0000 7f2b1fc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b18072da0 0x7f2b18162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.544+0000 7f2b1fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b1806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.544+0000 7f2b1fc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b180ce090 con 0x7f2b18070ee0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.544+0000 7f2b1fc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b180ceca0 con 0x7f2b18072da0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.544+0000 7f2b1fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180ce370 con 0x7f2b180712c0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b18072da0 0x7f2b18162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b1d200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2b18070ee0 0x7f2b1806e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57938/0 (socket says 172.21.15.32:57938) 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b18072da0 0x7f2b18162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58114/0 (socket says 172.21.15.32:58114) 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b1d200640 1 -- 172.21.15.32:0/2566449089 learned_addr learned my addr 172.21.15.32:0/2566449089 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17e00640 1 -- 172.21.15.32:0/2566449089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 msgr2=0x7f2b1806e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b1dc00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b1806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17e00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b1806e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17e00640 1 -- 172.21.15.32:0/2566449089 >> v1:172.21.15.32:6789/0 conn(0x7f2b18070ee0 legacy=0x7f2b1806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17e00640 1 -- 172.21.15.32:0/2566449089 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180ce650 con 0x7f2b18072da0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b17400640 1 -- 172.21.15.32:0/2566449089 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324894661 0 0) 0x7f2b180ce090 con 0x7f2b18070ee0 2024-03-06T07:39:13.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.545+0000 7f2b1dc00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b1806e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.546+0000 7f2b17e00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b18162c40 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f2b180cee20 tx=0x7f2b0802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e71df6c523fde37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b17400640 1 -- 172.21.15.32:0/2566449089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08035020 con 0x7f2b18072da0 2024-03-06T07:39:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b17400640 1 -- 172.21.15.32:0/2566449089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b08031800 con 0x7f2b18072da0 2024-03-06T07:39:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b17400640 1 -- 172.21.15.32:0/2566449089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08031bd0 con 0x7f2b18072da0 2024-03-06T07:39:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 -- 172.21.15.32:0/2566449089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 msgr2=0x7f2b18162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b18162c40 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f2b180cee20 tx=0x7f2b0802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 -- 172.21.15.32:0/2566449089 shutdown_connections 2024-03-06T07:39:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b18162c40 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/2566449089 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b1806e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 -- 172.21.15.32:0/2566449089 >> 172.21.15.32:0/2566449089 conn(0x7f2b1806afb0 msgr2=0x7f2b1806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 -- 172.21.15.32:0/2566449089 shutdown_connections 2024-03-06T07:39:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.547+0000 7f2b1fc00640 1 -- 172.21.15.32:0/2566449089 wait complete. 2024-03-06T07:39:13.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.550+0000 7f2b1fc00640 1 Processor -- start 2024-03-06T07:39:13.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.550+0000 7f2b1fc00640 1 -- start start 2024-03-06T07:39:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b18070ee0 0x7f2b1814eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1fc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b18152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b1809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cdda0 con 0x7f2b18072da0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cdc20 con 0x7f2b18070ee0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1fc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cf130 con 0x7f2b180712c0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b1809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b1809aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58122/0 (socket says 172.21.15.32:58122) 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.552+0000 7f2b1dc00640 1 -- 172.21.15.32:0/1077515501 learned_addr learned my addr 172.21.15.32:0/1077515501 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b17e00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b18152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1d200640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b18070ee0 0x7f2b1814eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1dc00640 1 -- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 msgr2=0x7f2b18152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1dc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b18152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1dc00640 1 -- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b18070ee0 msgr2=0x7f2b1814eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1dc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b18070ee0 0x7f2b1814eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1dc00640 1 -- 172.21.15.32:0/1077515501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1809b4d0 con 0x7f2b18072da0 2024-03-06T07:39:13.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1d200640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b18070ee0 0x7f2b1814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:13.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.553+0000 7f2b1dc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b1809aa10 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000cef0 tx=0x7f2b1000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.554+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1001a070 con 0x7f2b18072da0 2024-03-06T07:39:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.554+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1000d860 con 0x7f2b18072da0 2024-03-06T07:39:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.554+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1000dc50 con 0x7f2b18072da0 2024-03-06T07:39:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.554+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1809b7b0 con 0x7f2b18072da0 2024-03-06T07:39:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.554+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1809bd70 con 0x7f2b18072da0 2024-03-06T07:39:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.555+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b18069510 con 0x7f2b18072da0 2024-03-06T07:39:13.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.556+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b1001d070 con 0x7f2b18072da0 2024-03-06T07:39:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.557+0000 7f2b14c00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2afc06b4d0 0x7f2afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.557+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b1008abb0 con 0x7f2b18072da0 2024-03-06T07:39:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.557+0000 7f2b1d200640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2afc06b4d0 0x7f2afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.558+0000 7f2b1d200640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2afc06b4d0 0x7f2afc06d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000f60 tx=0x7f2b0c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.566+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b10059a10 con 0x7f2b18072da0 2024-03-06T07:39:13.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.983+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b1804f3a0 con 0x7f2b18072da0 2024-03-06T07:39:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.984+0000 7f2b14c00640 1 -- 172.21.15.32:0/1077515501 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f2b10059310 con 0x7f2b18072da0 2024-03-06T07:39:13.986 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:13.986 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.986+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2afc06b4d0 msgr2=0x7f2afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.986+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2afc06b4d0 0x7f2afc06d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000f60 tx=0x7f2b0c009040 comp rx=0 tx=0).stop 2024-03-06T07:39:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.986+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 msgr2=0x7f2b1809aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.986+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b1809aa10 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000cef0 tx=0x7f2b1000a710 comp rx=0 tx=0).stop 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 shutdown_connections 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2afc06b4d0 0x7f2afc06d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b18072da0 0x7f2b1809aa10 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2b180712c0 0x7f2b18152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 --2- 172.21.15.32:0/1077515501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b18070ee0 0x7f2b1814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 >> 172.21.15.32:0/1077515501 conn(0x7f2b1806afb0 msgr2=0x7f2b180b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 shutdown_connections 2024-03-06T07:39:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:13.987+0000 7f2b1fc00640 1 -- 172.21.15.32:0/1077515501 wait complete. 2024-03-06T07:39:14.210 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:14.210 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:15.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:15.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.427+0000 7f6abde00640 1 Processor -- start 2024-03-06T07:39:15.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.428+0000 7f6abde00640 1 -- start start 2024-03-06T07:39:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.429+0000 7f6abde00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 0x7f6ab81652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.429+0000 7f6abde00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab8071640 con 0x7f6ab815a550 2024-03-06T07:39:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.429+0000 7f6abde00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab8071920 con 0x7f6ab815a930 2024-03-06T07:39:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.429+0000 7f6abde00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80af6f0 con 0x7f6ab80cebc0 2024-03-06T07:39:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.430+0000 7f6ab6a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6ab815a550 0x7f6ab80aefe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57948/0 (socket says 172.21.15.32:57948) 2024-03-06T07:39:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.430+0000 7f6ab6a00640 1 -- 172.21.15.32:0/3989487853 learned_addr learned my addr 172.21.15.32:0/3989487853 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.430+0000 7f6ab7400640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 0x7f6ab81652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.430+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547650153 0 0) 0x7f6ab8071640 con 0x7f6ab815a550 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.430+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a9c003520 con 0x7f6ab815a550 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017965188 0 0) 0x7f6ab8071920 con 0x7f6ab815a930 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab8071640 con 0x7f6ab815a930 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4278343131 0 0) 0x7f6a9c003520 con 0x7f6ab815a550 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 msgr2=0x7f6ab81652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 0x7f6ab81652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 >> v1:172.21.15.39:6789/0 conn(0x7f6ab815a930 legacy=0x7f6ab80ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8071c00 con 0x7f6ab815a550 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6aa80019f0 con 0x7f6ab815a550 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 >> v1:172.21.15.32:6789/0 conn(0x7f6ab815a550 legacy=0x7f6ab80aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a9c003be0 0x7f6a9c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a9c004800 0x7f6a9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 0x7f6a9c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80af6f0 con 0x7f6a9c004800 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c00bb20 con 0x7f6a9c003be0 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c00bd20 con 0x7f6ab80cebc0 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6aae000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a9c004800 0x7f6a9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.431+0000 7f6ab6a00640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a9c003be0 0x7f6a9c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.436 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.432+0000 7f6aae000640 1 -- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 msgr2=0x7f6a9c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.432+0000 7f6aae000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6ab80cebc0 0x7f6a9c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.432+0000 7f6aae000640 1 -- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a9c003be0 msgr2=0x7f6a9c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.432+0000 7f6aae000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a9c003be0 0x7f6a9c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.432+0000 7f6aae000640 1 -- 172.21.15.32:0/3989487853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c003520 con 0x7f6a9c004800 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.433+0000 7f6aae000640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a9c004800 0x7f6a9c004c80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00018d0 tx=0x7f6a9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a162525dfa02a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.433+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0037070 con 0x7f6a9c004800 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.433+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa00052d0 con 0x7f6a9c004800 2024-03-06T07:39:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.433+0000 7f6ab6000640 1 -- 172.21.15.32:0/3989487853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00014f0 con 0x7f6a9c004800 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.433+0000 7f6abde00640 1 -- 172.21.15.32:0/3989487853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a9c004800 msgr2=0x7f6a9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.434+0000 7f6abde00640 1 --2- 172.21.15.32:0/3989487853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a9c004800 0x7f6a9c004c80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00018d0 tx=0x7f6a9c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.434+0000 7f6ab7400640 1 -- 172.21.15.32:0/3989487853 reap_dead start 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.434+0000 7f6abde00640 1 -- 172.21.15.32:0/3989487853 shutdown_connections 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.434+0000 7f6abde00640 1 -- 172.21.15.32:0/3989487853 >> 172.21.15.32:0/3989487853 conn(0x7f6ab806afb0 msgr2=0x7f6ab806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.434+0000 7f6abde00640 1 -- 172.21.15.32:0/3989487853 shutdown_connections 2024-03-06T07:39:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.434+0000 7f6abde00640 1 -- 172.21.15.32:0/3989487853 wait complete. 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.435+0000 7f6abde00640 1 Processor -- start 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.436+0000 7f6abde00640 1 -- start start 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.436+0000 7f6abde00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a9c004800 0x7f6ab8155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6abde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 0x7f6ab81e81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6abde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ab81e86f0 0x7f6ab81f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6abde00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80b52b0 con 0x7f6ab81e7820 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6abde00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80b5130 con 0x7f6ab81e86f0 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6abde00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80ade50 con 0x7f6a9c004800 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 0x7f6ab81e81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 0x7f6ab81e81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58154/0 (socket says 172.21.15.32:58154) 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 -- 172.21.15.32:0/1203411854 learned_addr learned my addr 172.21.15.32:0/1203411854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6ab7400640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ab81e86f0 0x7f6ab81f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6ab6a00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a9c004800 0x7f6ab8155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 -- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a9c004800 msgr2=0x7f6ab8155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a9c004800 0x7f6ab8155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 -- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ab81e86f0 msgr2=0x7f6ab81f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ab81e86f0 0x7f6ab81f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6aae000640 1 -- 172.21.15.32:0/1203411854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8153320 con 0x7f6ab81e7820 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.437+0000 7f6ab6a00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a9c004800 0x7f6ab8155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:15.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.438+0000 7f6aae000640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 0x7f6ab81e81b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00018d0 tx=0x7f6aa00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.438+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0037070 con 0x7f6ab81e7820 2024-03-06T07:39:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.438+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0038040 con 0x7f6ab81e7820 2024-03-06T07:39:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.438+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00014f0 con 0x7f6ab81e7820 2024-03-06T07:39:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.438+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab8155710 con 0x7f6ab81e7820 2024-03-06T07:39:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.438+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab8155cd0 con 0x7f6ab81e7820 2024-03-06T07:39:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.440+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8069510 con 0x7f6ab81e7820 2024-03-06T07:39:15.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.441+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6aa00052d0 con 0x7f6ab81e7820 2024-03-06T07:39:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.442+0000 7f6aafe00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a9406b4d0 0x7f6a9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.442+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6aa00aba10 con 0x7f6ab81e7820 2024-03-06T07:39:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.442+0000 7f6ab6a00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a9406b4d0 0x7f6a9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.443+0000 7f6ab6a00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a9406b4d0 0x7f6a9406d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8002230 tx=0x7f6aa80014d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.447+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6aa007a280 con 0x7f6ab81e7820 2024-03-06T07:39:15.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.704+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ab804f3a0 con 0x7f6ab81e7820 2024-03-06T07:39:15.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.705+0000 7f6aafe00640 1 -- 172.21.15.32:0/1203411854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f6aa007e320 con 0x7f6ab81e7820 2024-03-06T07:39:15.708 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:15.708 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:15.708 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:15.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a9406b4d0 msgr2=0x7f6a9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a9406b4d0 0x7f6a9406d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8002230 tx=0x7f6aa80014d0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 msgr2=0x7f6ab81e81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 0x7f6ab81e81b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00018d0 tx=0x7f6aa00012d0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 shutdown_connections 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6a9406b4d0 0x7f6a9406d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ab81e86f0 0x7f6ab81f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ab81e7820 0x7f6ab81e81b0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 --2- 172.21.15.32:0/1203411854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6a9c004800 0x7f6ab8155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 >> 172.21.15.32:0/1203411854 conn(0x7f6ab806afb0 msgr2=0x7f6ab81590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 shutdown_connections 2024-03-06T07:39:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:15.708+0000 7f6abde00640 1 -- 172.21.15.32:0/1203411854 wait complete. 2024-03-06T07:39:15.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:15.938 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:16.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:17.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.157+0000 7f41b9c00640 1 Processor -- start 2024-03-06T07:39:17.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.157+0000 7f41b9c00640 1 -- start start 2024-03-06T07:39:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.158+0000 7f41b9c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41b415e2e0 0x7f41b40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b9c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41b40b50e0 0x7f41b40d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b9c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40b3940 con 0x7f41b40d0d70 2024-03-06T07:39:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b9c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b40d4e70 con 0x7f41b40b50e0 2024-03-06T07:39:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b9c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b40b3c40 con 0x7f41b415e2e0 2024-03-06T07:39:17.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41b40b50e0 0x7f41b40d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b3400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41b40b50e0 0x7f41b40d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58170/0 (socket says 172.21.15.32:58170) 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b3400640 1 -- 172.21.15.32:0/1340619564 learned_addr learned my addr 172.21.15.32:0/1340619564 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.159+0000 7f41b2a00640 1 --2- 172.21.15.32:0/1340619564 >> v2:172.21.15.39:3300/0 conn(0x7f41b415e2e0 0x7f41b40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.160+0000 7f41b3400640 1 -- 172.21.15.32:0/1340619564 >> v2:172.21.15.39:3300/0 conn(0x7f41b415e2e0 msgr2=0x7f41b40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.160+0000 7f41b3400640 1 --2- 172.21.15.32:0/1340619564 >> v2:172.21.15.39:3300/0 conn(0x7f41b415e2e0 0x7f41b40d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.160+0000 7f41b3400640 1 -- 172.21.15.32:0/1340619564 >> v1:172.21.15.39:6789/0 conn(0x7f41b40d0d70 legacy=0x7f41b40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.160+0000 7f41b3400640 1 -- 172.21.15.32:0/1340619564 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40cf580 con 0x7f41b40b50e0 2024-03-06T07:39:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.162+0000 7f41b3400640 1 --2- 172.21.15.32:0/1340619564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b40b50e0 0x7f41b40d11d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f41b408d8b0 tx=0x7f41ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2327f4689b9ec711 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.163+0000 7f41b1600640 1 -- 172.21.15.32:0/1340619564 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac034070 con 0x7f41b40b50e0 2024-03-06T07:39:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.163+0000 7f41b1600640 1 -- 172.21.15.32:0/1340619564 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac030860 con 0x7f41b40b50e0 2024-03-06T07:39:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.163+0000 7f41b1600640 1 -- 172.21.15.32:0/1340619564 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac030c30 con 0x7f41b40b50e0 2024-03-06T07:39:17.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.163+0000 7f41b9c00640 1 -- 172.21.15.32:0/1340619564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b40b50e0 msgr2=0x7f41b40d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.163+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1340619564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b40b50e0 0x7f41b40d11d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f41b408d8b0 tx=0x7f41ac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.164+0000 7f41b9c00640 1 -- 172.21.15.32:0/1340619564 shutdown_connections 2024-03-06T07:39:17.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.164+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1340619564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b40b50e0 0x7f41b40d11d0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.164+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1340619564 >> v2:172.21.15.39:3300/0 conn(0x7f41b415e2e0 0x7f41b40d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.164+0000 7f41b9c00640 1 -- 172.21.15.32:0/1340619564 >> 172.21.15.32:0/1340619564 conn(0x7f41b40d3f60 msgr2=0x7f41b40d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:17.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.164+0000 7f41b9c00640 1 -- 172.21.15.32:0/1340619564 shutdown_connections 2024-03-06T07:39:17.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.164+0000 7f41b9c00640 1 -- 172.21.15.32:0/1340619564 wait complete. 2024-03-06T07:39:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.166+0000 7f41b9c00640 1 Processor -- start 2024-03-06T07:39:17.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.167+0000 7f41b9c00640 1 -- start start 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.168+0000 7f41b9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41b40b50e0 0x7f41b4161090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.168+0000 7f41b9c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f41b40d0d70 0x7f41b40cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.168+0000 7f41b9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 0x7f41b40b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.168+0000 7f41b9c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40cfe20 con 0x7f41b415e2e0 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.168+0000 7f41b9c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40cfb40 con 0x7f41b40b50e0 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.168+0000 7f41b9c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40d2240 con 0x7f41b40d0d70 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 0x7f41b40b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 0x7f41b40b8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58186/0 (socket says 172.21.15.32:58186) 2024-03-06T07:39:17.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 -- 172.21.15.32:0/1003846214 learned_addr learned my addr 172.21.15.32:0/1003846214 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b2a00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41b40b50e0 0x7f41b4161090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 -- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f41b40d0d70 msgr2=0x7f41b40cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b2000640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f41b40d0d70 0x7f41b40cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f41b40d0d70 0x7f41b40cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 -- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41b40b50e0 msgr2=0x7f41b4161090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41b40b50e0 0x7f41b4161090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b3400640 1 -- 172.21.15.32:0/1003846214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b41615d0 con 0x7f41b415e2e0 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.169+0000 7f41b2a00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41b40b50e0 0x7f41b4161090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.170+0000 7f41b3400640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 0x7f41b40b8d50 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f41ac02b9e0 tx=0x7f41ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.170+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac048070 con 0x7f41b415e2e0 2024-03-06T07:39:17.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.170+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac030660 con 0x7f41b415e2e0 2024-03-06T07:39:17.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.170+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac034040 con 0x7f41b415e2e0 2024-03-06T07:39:17.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.170+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b414f970 con 0x7f41b415e2e0 2024-03-06T07:39:17.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.170+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b40cb0d0 con 0x7f41b415e2e0 2024-03-06T07:39:17.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.171+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b404f3a0 con 0x7f41b415e2e0 2024-03-06T07:39:17.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.173+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41ac04b070 con 0x7f41b415e2e0 2024-03-06T07:39:17.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.173+0000 7f41aaa00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f419006b4d0 0x7f419006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.173+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f41ac0ac360 con 0x7f41b415e2e0 2024-03-06T07:39:17.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.174+0000 7f41b2a00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f419006b4d0 0x7f419006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.174+0000 7f41b2a00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f419006b4d0 0x7f419006d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007440 tx=0x7f41a400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:17.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.181+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41ac07b220 con 0x7f41b415e2e0 2024-03-06T07:39:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.427+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41b406bfd0 con 0x7f41b415e2e0 2024-03-06T07:39:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.428+0000 7f41aaa00640 1 -- 172.21.15.32:0/1003846214 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f41ac030830 con 0x7f41b415e2e0 2024-03-06T07:39:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:17.431 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f419006b4d0 msgr2=0x7f419006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f419006b4d0 0x7f419006d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007440 tx=0x7f41a400b040 comp rx=0 tx=0).stop 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 msgr2=0x7f41b40b8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 0x7f41b40b8d50 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f41ac02b9e0 tx=0x7f41ac004730 comp rx=0 tx=0).stop 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 shutdown_connections 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f419006b4d0 0x7f419006d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b415e2e0 0x7f41b40b8d50 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f41b40d0d70 0x7f41b40cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 --2- 172.21.15.32:0/1003846214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41b40b50e0 0x7f41b4161090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 >> 172.21.15.32:0/1003846214 conn(0x7f41b40d3f60 msgr2=0x7f41b40d1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 shutdown_connections 2024-03-06T07:39:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:17.431+0000 7f41b9c00640 1 -- 172.21.15.32:0/1003846214 wait complete. 2024-03-06T07:39:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:17.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:17.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:18.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.928+0000 7f99a3c00640 1 Processor -- start 2024-03-06T07:39:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.928+0000 7f99a3c00640 1 -- start start 2024-03-06T07:39:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.930+0000 7f99a3c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 0x7f999c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.930+0000 7f99a3c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f999c0cf5c0 0x7f999c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.930+0000 7f99a3c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999c0b45b0 con 0x7f999c06cb10 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.930+0000 7f99a3c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999c0ce160 con 0x7f999c0cf5c0 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.930+0000 7f99a3c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b4860 con 0x7f999c0b3c90 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.931+0000 7f99a1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 0x7f999c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.931+0000 7f999be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f999c0cf5c0 0x7f999c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.931+0000 7f99a1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 0x7f999c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:55978/0 (socket says 172.21.15.32:55978) 2024-03-06T07:39:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.931+0000 7f99a1200640 1 -- 172.21.15.32:0/2326712336 learned_addr learned my addr 172.21.15.32:0/2326712336 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.932+0000 7f99a1200640 1 -- 172.21.15.32:0/2326712336 >> v2:172.21.15.39:3300/0 conn(0x7f999c0cf5c0 msgr2=0x7f999c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.932+0000 7f99a1200640 1 --2- 172.21.15.32:0/2326712336 >> v2:172.21.15.39:3300/0 conn(0x7f999c0cf5c0 0x7f999c0b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.932+0000 7f99a1200640 1 -- 172.21.15.32:0/2326712336 >> v1:172.21.15.39:6789/0 conn(0x7f999c06cb10 legacy=0x7f999c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.932+0000 7f99a1200640 1 -- 172.21.15.32:0/2326712336 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c0b4b40 con 0x7f999c0b3c90 2024-03-06T07:39:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.932+0000 7f999be00640 1 --2- 172.21.15.32:0/2326712336 >> v2:172.21.15.39:3300/0 conn(0x7f999c0cf5c0 0x7f999c0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.933+0000 7f99a1200640 1 --2- 172.21.15.32:0/2326712336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 0x7f999c0cdc20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9990009870 tx=0x7f999002dc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a32563b54b5f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.933+0000 7f999b400640 1 -- 172.21.15.32:0/2326712336 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999003b070 con 0x7f999c0b3c90 2024-03-06T07:39:18.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f999b400640 1 -- 172.21.15.32:0/2326712336 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999002e9d0 con 0x7f999c0b3c90 2024-03-06T07:39:18.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f999b400640 1 -- 172.21.15.32:0/2326712336 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999002eda0 con 0x7f999c0b3c90 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 -- 172.21.15.32:0/2326712336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 msgr2=0x7f999c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2326712336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 0x7f999c0cdc20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9990009870 tx=0x7f999002dc20 comp rx=0 tx=0).stop 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 -- 172.21.15.32:0/2326712336 shutdown_connections 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2326712336 >> v2:172.21.15.39:3300/0 conn(0x7f999c0cf5c0 0x7f999c0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2326712336 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0b3c90 0x7f999c0cdc20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 -- 172.21.15.32:0/2326712336 >> 172.21.15.32:0/2326712336 conn(0x7f999c06afb0 msgr2=0x7f999c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 -- 172.21.15.32:0/2326712336 shutdown_connections 2024-03-06T07:39:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.934+0000 7f99a3c00640 1 -- 172.21.15.32:0/2326712336 wait complete. 2024-03-06T07:39:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.936+0000 7f99a3c00640 1 Processor -- start 2024-03-06T07:39:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.936+0000 7f99a3c00640 1 -- start start 2024-03-06T07:39:18.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f99a3c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c06cb10 0x7f999c1e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f99a3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 0x7f999c153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f99a3c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0cf5c0 0x7f999c153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f99a3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c15b340 con 0x7f999c0b3c90 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f99a3c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b5100 con 0x7f999c06cb10 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f99a3c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c06d690 con 0x7f999c0cf5c0 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.937+0000 7f999be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 0x7f999c153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 0x7f999c153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45428/0 (socket says 172.21.15.32:45428) 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 -- 172.21.15.32:0/2070944279 learned_addr learned my addr 172.21.15.32:0/2070944279 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f99a1c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0cf5c0 0x7f999c153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f99a1200640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c06cb10 0x7f999c1e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 -- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0cf5c0 msgr2=0x7f999c153ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0cf5c0 0x7f999c153ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 -- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c06cb10 msgr2=0x7f999c1e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c06cb10 0x7f999c1e7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 -- 172.21.15.32:0/2070944279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c09c930 con 0x7f999c0b3c90 2024-03-06T07:39:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f99a1200640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c06cb10 0x7f999c1e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f99a1c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0cf5c0 0x7f999c153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.938+0000 7f999be00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 0x7f999c153590 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f998800e960 tx=0x7f998800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.939+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988017070 con 0x7f999c0b3c90 2024-03-06T07:39:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.939+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998800ca00 con 0x7f999c0b3c90 2024-03-06T07:39:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.939+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998800cdd0 con 0x7f999c0b3c90 2024-03-06T07:39:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.939+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c09cc10 con 0x7f999c0b3c90 2024-03-06T07:39:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.939+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c09d4b0 con 0x7f999c0b3c90 2024-03-06T07:39:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.940+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c069510 con 0x7f999c0b3c90 2024-03-06T07:39:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.941+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f998801a070 con 0x7f999c0b3c90 2024-03-06T07:39:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.942+0000 7f9998c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f997c06b4d0 0x7f997c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.942+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f998808a550 con 0x7f999c0b3c90 2024-03-06T07:39:18.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.942+0000 7f99a1200640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f997c06b4d0 0x7f997c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.943+0000 7f99a1200640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f997c06b4d0 0x7f997c06d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9990038000 tx=0x7f9990032000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:18.947+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99880592f0 con 0x7f999c0b3c90 2024-03-06T07:39:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.224+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999c04f3a0 con 0x7f999c0b3c90 2024-03-06T07:39:19.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.225+0000 7f9998c00640 1 -- 172.21.15.32:0/2070944279 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f9988058bf0 con 0x7f999c0b3c90 2024-03-06T07:39:19.228 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:19.228 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:19.228 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f997c06b4d0 msgr2=0x7f997c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f997c06b4d0 0x7f997c06d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9990038000 tx=0x7f9990032000 comp rx=0 tx=0).stop 2024-03-06T07:39:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 msgr2=0x7f999c153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 0x7f999c153590 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f998800e960 tx=0x7f998800ee20 comp rx=0 tx=0).stop 2024-03-06T07:39:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 shutdown_connections 2024-03-06T07:39:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f997c06b4d0 0x7f997c06d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f999c0cf5c0 0x7f999c153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f999c0b3c90 0x7f999c153590 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 --2- 172.21.15.32:0/2070944279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f999c06cb10 0x7f999c1e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 >> 172.21.15.32:0/2070944279 conn(0x7f999c06afb0 msgr2=0x7f999c06daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:19.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 shutdown_connections 2024-03-06T07:39:19.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:19.228+0000 7f99a3c00640 1 -- 172.21.15.32:0/2070944279 wait complete. 2024-03-06T07:39:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:19.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:19.452 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:20.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:20.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.678+0000 7f37e0200640 1 Processor -- start 2024-03-06T07:39:20.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.678+0000 7f37e0200640 1 -- start start 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37e0200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37d815faa0 0x7f37d80d1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37e0200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d80d2130 0x7f37d80af030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37e0200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d815ffd0 con 0x7f37d815be40 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37e0200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d81602b0 con 0x7f37d815faa0 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37e0200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d8160590 con 0x7f37d80d2130 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37dce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37d815faa0 0x7f37d80d1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37dce00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37d815faa0 0x7f37d80d1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45438/0 (socket says 172.21.15.32:45438) 2024-03-06T07:39:20.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.679+0000 7f37dce00640 1 -- 172.21.15.32:0/731531041 learned_addr learned my addr 172.21.15.32:0/731531041 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.680+0000 7f37de200640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d80d2130 0x7f37d80af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.680+0000 7f37dce00640 1 -- 172.21.15.32:0/731531041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d80d2130 msgr2=0x7f37d80af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.680+0000 7f37dce00640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d80d2130 0x7f37d80af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.680+0000 7f37dce00640 1 -- 172.21.15.32:0/731531041 >> v1:172.21.15.39:6789/0 conn(0x7f37d815be40 legacy=0x7f37d80cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.680+0000 7f37dce00640 1 -- 172.21.15.32:0/731531041 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d8160870 con 0x7f37d815faa0 2024-03-06T07:39:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.680+0000 7f37de200640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d80d2130 0x7f37d80af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.681+0000 7f37dce00640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d815faa0 0x7f37d80d1bf0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009870 tx=0x7f37c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad067a8454b487e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.681+0000 7f37d7e00640 1 -- 172.21.15.32:0/731531041 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4035020 con 0x7f37d815faa0 2024-03-06T07:39:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.681+0000 7f37d7e00640 1 -- 172.21.15.32:0/731531041 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c4031640 con 0x7f37d815faa0 2024-03-06T07:39:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.681+0000 7f37d7e00640 1 -- 172.21.15.32:0/731531041 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c402d040 con 0x7f37d815faa0 2024-03-06T07:39:20.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 -- 172.21.15.32:0/731531041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d815faa0 msgr2=0x7f37d80d1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d815faa0 0x7f37d80d1bf0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009870 tx=0x7f37c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 -- 172.21.15.32:0/731531041 shutdown_connections 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d80d2130 0x7f37d80af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 --2- 172.21.15.32:0/731531041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d815faa0 0x7f37d80d1bf0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 -- 172.21.15.32:0/731531041 >> 172.21.15.32:0/731531041 conn(0x7f37d809aa10 msgr2=0x7f37d809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 -- 172.21.15.32:0/731531041 shutdown_connections 2024-03-06T07:39:20.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.682+0000 7f37e0200640 1 -- 172.21.15.32:0/731531041 wait complete. 2024-03-06T07:39:20.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.684+0000 7f37e0200640 1 Processor -- start 2024-03-06T07:39:20.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.684+0000 7f37e0200640 1 -- start start 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37e0200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 0x7f37d815ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37e0200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d815be40 0x7f37d814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37e0200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37d815faa0 0x7f37d80afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37e0200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80af6f0 con 0x7f37d80d2130 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37e0200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80af570 con 0x7f37d815faa0 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37e0200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80af870 con 0x7f37d815be40 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37dd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 0x7f37d815ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37dd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 0x7f37d815ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45454/0 (socket says 172.21.15.32:45454) 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.685+0000 7f37dd800640 1 -- 172.21.15.32:0/804060494 learned_addr learned my addr 172.21.15.32:0/804060494 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:20.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dce00640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d815be40 0x7f37d814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37de200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37d815faa0 0x7f37d80afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dd800640 1 -- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d815be40 msgr2=0x7f37d814ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dd800640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d815be40 0x7f37d814ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dd800640 1 -- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37d815faa0 msgr2=0x7f37d80afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dd800640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37d815faa0 0x7f37d80afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dd800640 1 -- 172.21.15.32:0/804060494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d80b3410 con 0x7f37d80d2130 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37dd800640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 0x7f37d815ae80 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00c430 tx=0x7f37cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.686+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc011790 con 0x7f37d80d2130 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.687+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc017070 con 0x7f37d80d2130 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.687+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d80b5b20 con 0x7f37d80d2130 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.687+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc011bb0 con 0x7f37d80d2130 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.687+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d80b60e0 con 0x7f37d80d2130 2024-03-06T07:39:20.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.688+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d804f3a0 con 0x7f37d80d2130 2024-03-06T07:39:20.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.689+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37cc01c460 con 0x7f37d80d2130 2024-03-06T07:39:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.692+0000 7f37d5600640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f37b806b4d0 0x7f37b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.692+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37cc08a130 con 0x7f37d80d2130 2024-03-06T07:39:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.692+0000 7f37dce00640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f37b806b4d0 0x7f37b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.692+0000 7f37dce00640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f37b806b4d0 0x7f37b806d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f37c4036000 tx=0x7f37c403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:20.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.695+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37cc058f70 con 0x7f37d80d2130 2024-03-06T07:39:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.937+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37d815d040 con 0x7f37d80d2130 2024-03-06T07:39:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.938+0000 7f37d5600640 1 -- 172.21.15.32:0/804060494 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f37cc058870 con 0x7f37d80d2130 2024-03-06T07:39:20.941 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:20.941 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f37b806b4d0 msgr2=0x7f37b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f37b806b4d0 0x7f37b806d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f37c4036000 tx=0x7f37c403c040 comp rx=0 tx=0).stop 2024-03-06T07:39:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 msgr2=0x7f37d815ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 0x7f37d815ae80 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00c430 tx=0x7f37cc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 shutdown_connections 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f37b806b4d0 0x7f37b806d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37d815faa0 0x7f37d80afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f37d815be40 0x7f37d814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 --2- 172.21.15.32:0/804060494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d80d2130 0x7f37d815ae80 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 >> 172.21.15.32:0/804060494 conn(0x7f37d809aa10 msgr2=0x7f37d80b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 shutdown_connections 2024-03-06T07:39:20.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:20.941+0000 7f37e0200640 1 -- 172.21.15.32:0/804060494 wait complete. 2024-03-06T07:39:21.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:21.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:21.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:21.369 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:22.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.396+0000 7f0453200640 1 Processor -- start 2024-03-06T07:39:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.396+0000 7f0453200640 1 -- start start 2024-03-06T07:39:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0453200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 0x7f0454161c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0453200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540cff30 con 0x7f0454158460 2024-03-06T07:39:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0453200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04541626d0 con 0x7f0454162150 2024-03-06T07:39:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0453200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04541629b0 con 0x7f0454158840 2024-03-06T07:39:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0451e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0454158460 0x7f04540d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52972/0 (socket says 172.21.15.32:52972) 2024-03-06T07:39:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0451e00640 1 -- 172.21.15.32:0/1232379678 learned_addr learned my addr 172.21.15.32:0/1232379678 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.398+0000 7f0451400640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 0x7f0454161c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934046959 0 0) 0x7f04540cff30 con 0x7f0454158460 2024-03-06T07:39:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438003520 con 0x7f0454158460 2024-03-06T07:39:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450394807 0 0) 0x7f0438003520 con 0x7f0454158460 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 msgr2=0x7f0454161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 0x7f0454161c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 >> v1:172.21.15.39:6789/0 conn(0x7f0454162150 legacy=0x7f04540b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04540d01d0 con 0x7f0454158460 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f0444001670 con 0x7f0454158460 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 >> v1:172.21.15.32:6789/0 conn(0x7f0454158460 legacy=0x7f04540d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04380039e0 0x7f0438003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 0x7f0438004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0438004d40 0x7f04380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04541629b0 con 0x7f04380039e0 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043800bb50 con 0x7f0438004d40 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043800bd50 con 0x7f0454158840 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.399+0000 7f0451e00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04380039e0 0x7f0438003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.400+0000 7f0451400640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 0x7f0438004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.400+0000 7f0451e00640 1 -- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 msgr2=0x7f0438004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.400+0000 7f0451e00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454158840 0x7f0438004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.400+0000 7f0451e00640 1 -- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0438004d40 msgr2=0x7f04380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.400+0000 7f0451e00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0438004d40 0x7f04380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.400+0000 7f0451e00640 1 -- 172.21.15.32:0/1232379678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438003520 con 0x7f04380039e0 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.401+0000 7f0451e00640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04380039e0 0x7f0438003e00 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f044400c190 tx=0x7f0438005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efc08856d382f1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.401+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444007c60 con 0x7f04380039e0 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.401+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0444030070 con 0x7f04380039e0 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.401+0000 7f0450a00640 1 -- 172.21.15.32:0/1232379678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444005d50 con 0x7f04380039e0 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0453200640 1 -- 172.21.15.32:0/1232379678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04380039e0 msgr2=0x7f0438003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0453200640 1 --2- 172.21.15.32:0/1232379678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04380039e0 0x7f0438003e00 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f044400c190 tx=0x7f0438005830 comp rx=0 tx=0).stop 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0452800640 1 -- 172.21.15.32:0/1232379678 reap_dead start 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0453200640 1 -- 172.21.15.32:0/1232379678 shutdown_connections 2024-03-06T07:39:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0453200640 1 -- 172.21.15.32:0/1232379678 >> 172.21.15.32:0/1232379678 conn(0x7f04540b5100 msgr2=0x7f04540b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0453200640 1 -- 172.21.15.32:0/1232379678 shutdown_connections 2024-03-06T07:39:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.402+0000 7f0453200640 1 -- 172.21.15.32:0/1232379678 wait complete. 2024-03-06T07:39:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.404+0000 7f0453200640 1 Processor -- start 2024-03-06T07:39:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.404+0000 7f0453200640 1 -- start start 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0453200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04380039e0 0x7f04540ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0453200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 0x7f0454155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0453200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454155870 0x7f04541f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0453200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04540b1a20 con 0x7f0454099830 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0453200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045415e8c0 con 0x7f04380039e0 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0453200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04540b1ba0 con 0x7f0454155870 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0451400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 0x7f0454155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0451400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 0x7f0454155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45476/0 (socket says 172.21.15.32:45476) 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.405+0000 7f0451400640 1 -- 172.21.15.32:0/2851260154 learned_addr learned my addr 172.21.15.32:0/2851260154 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0452800640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454155870 0x7f04541f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451e00640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04380039e0 0x7f04540ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451400640 1 -- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454155870 msgr2=0x7f04541f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451400640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454155870 0x7f04541f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451400640 1 -- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04380039e0 msgr2=0x7f04540ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451400640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04380039e0 0x7f04540ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451400640 1 -- 172.21.15.32:0/2851260154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04540aeb60 con 0x7f0454099830 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f0451400640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 0x7f0454155330 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f04400108c0 tx=0x7f0440010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.406+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044000f040 con 0x7f0454099830 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.407+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044000eb20 con 0x7f0454099830 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.407+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04540ad580 con 0x7f0454099830 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.407+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044001c400 con 0x7f0454099830 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.407+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04540adb40 con 0x7f0454099830 2024-03-06T07:39:22.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.408+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045404f330 con 0x7f0454099830 2024-03-06T07:39:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.409+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f044001c5a0 con 0x7f0454099830 2024-03-06T07:39:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.410+0000 7f044a000640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f043406b500 0x7f043406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.410+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f044008a840 con 0x7f0454099830 2024-03-06T07:39:22.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.411+0000 7f0451e00640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f043406b500 0x7f043406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.411+0000 7f0451e00640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f043406b500 0x7f043406d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f044400c160 tx=0x7f0444039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.415+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0440018030 con 0x7f0454099830 2024-03-06T07:39:22.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.683+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f045406b850 con 0x7f0454099830 2024-03-06T07:39:22.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.684+0000 7f044a000640 1 -- 172.21.15.32:0/2851260154 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f0440059670 con 0x7f0454099830 2024-03-06T07:39:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:22.685 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:22.685 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:22.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f043406b500 msgr2=0x7f043406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f043406b500 0x7f043406d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f044400c160 tx=0x7f0444039040 comp rx=0 tx=0).stop 2024-03-06T07:39:22.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 msgr2=0x7f0454155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 0x7f0454155330 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f04400108c0 tx=0x7f0440010d80 comp rx=0 tx=0).stop 2024-03-06T07:39:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 shutdown_connections 2024-03-06T07:39:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f043406b500 0x7f043406d9c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0454155870 0x7f04541f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0454099830 0x7f0454155330 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 --2- 172.21.15.32:0/2851260154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04380039e0 0x7f04540ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.686+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 >> 172.21.15.32:0/2851260154 conn(0x7f04540b5100 msgr2=0x7f04540af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:22.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.687+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 shutdown_connections 2024-03-06T07:39:22.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:22.687+0000 7f0453200640 1 -- 172.21.15.32:0/2851260154 wait complete. 2024-03-06T07:39:22.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:22.907 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:23.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.122+0000 7f8609800640 1 Processor -- start 2024-03-06T07:39:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.122+0000 7f8609800640 1 -- start start 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8609800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8604070f10 0x7f86040cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8609800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86040cce90 0x7f860415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8609800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040714d0 con 0x7f8604145300 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8609800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040cd300 con 0x7f86040cce90 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8609800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040717b0 con 0x7f8604070f10 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8608e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86040cce90 0x7f860415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8603e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8604145300 0x7f86040b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52974/0 (socket says 172.21.15.32:52974) 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8608e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86040cce90 0x7f860415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45484/0 (socket says 172.21.15.32:45484) 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.124+0000 7f8603e00640 1 -- 172.21.15.32:0/3392781571 learned_addr learned my addr 172.21.15.32:0/3392781571 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.125+0000 7f85fb400640 1 --2- 172.21.15.32:0/3392781571 >> v2:172.21.15.39:3300/0 conn(0x7f8604070f10 0x7f86040cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.125+0000 7f8608e00640 1 -- 172.21.15.32:0/3392781571 >> v2:172.21.15.39:3300/0 conn(0x7f8604070f10 msgr2=0x7f86040cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.125+0000 7f8608e00640 1 --2- 172.21.15.32:0/3392781571 >> v2:172.21.15.39:3300/0 conn(0x7f8604070f10 0x7f86040cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.125+0000 7f8608e00640 1 -- 172.21.15.32:0/3392781571 >> v1:172.21.15.32:6789/0 conn(0x7f8604145300 legacy=0x7f86040b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.125+0000 7f8608e00640 1 -- 172.21.15.32:0/3392781571 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604071a90 con 0x7f86040cce90 2024-03-06T07:39:24.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.125+0000 7f8603400640 1 -- 172.21.15.32:0/3392781571 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502490739 0 0) 0x7f86040714d0 con 0x7f8604145300 2024-03-06T07:39:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.126+0000 7f8608e00640 1 --2- 172.21.15.32:0/3392781571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86040cce90 0x7f860415c570 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f86040b5150 tx=0x7f85fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657128efbaaa1407 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8603400640 1 -- 172.21.15.32:0/3392781571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc039070 con 0x7f86040cce90 2024-03-06T07:39:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8603400640 1 -- 172.21.15.32:0/3392781571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc02cbf0 con 0x7f86040cce90 2024-03-06T07:39:24.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8603400640 1 -- 172.21.15.32:0/3392781571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc034440 con 0x7f86040cce90 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8609800640 1 -- 172.21.15.32:0/3392781571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86040cce90 msgr2=0x7f860415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8609800640 1 --2- 172.21.15.32:0/3392781571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86040cce90 0x7f860415c570 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f86040b5150 tx=0x7f85fc02be40 comp rx=0 tx=0).stop 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8609800640 1 -- 172.21.15.32:0/3392781571 shutdown_connections 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8609800640 1 --2- 172.21.15.32:0/3392781571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86040cce90 0x7f860415c570 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8609800640 1 --2- 172.21.15.32:0/3392781571 >> v2:172.21.15.39:3300/0 conn(0x7f8604070f10 0x7f86040cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.127+0000 7f8609800640 1 -- 172.21.15.32:0/3392781571 >> 172.21.15.32:0/3392781571 conn(0x7f860406b3d0 msgr2=0x7f860406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.128+0000 7f8609800640 1 -- 172.21.15.32:0/3392781571 shutdown_connections 2024-03-06T07:39:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.128+0000 7f8609800640 1 -- 172.21.15.32:0/3392781571 wait complete. 2024-03-06T07:39:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.129+0000 7f8609800640 1 Processor -- start 2024-03-06T07:39:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.129+0000 7f8609800640 1 -- start start 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.130+0000 7f8609800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 0x7f86041f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.130+0000 7f8609800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86040cce90 0x7f86041e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.130+0000 7f8609800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8604145300 0x7f860406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.130+0000 7f8609800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860415cab0 con 0x7f8604070f10 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.130+0000 7f8609800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041456e0 con 0x7f86040cce90 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.130+0000 7f8609800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860415cc30 con 0x7f8604145300 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 0x7f86041f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 0x7f86041f61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45486/0 (socket says 172.21.15.32:45486) 2024-03-06T07:39:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 -- 172.21.15.32:0/1725693695 learned_addr learned my addr 172.21.15.32:0/1725693695 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f85fb400640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86040cce90 0x7f86041e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8608e00640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8604145300 0x7f860406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 -- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8604145300 msgr2=0x7f860406d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8604145300 0x7f860406d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 -- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86040cce90 msgr2=0x7f86041e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86040cce90 0x7f86041e8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.131+0000 7f8603e00640 1 -- 172.21.15.32:0/1725693695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86041f66f0 con 0x7f8604070f10 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.132+0000 7f8603e00640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 0x7f86041f61b0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f85f400c460 tx=0x7f85f400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.132+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40074e0 con 0x7f8604070f10 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.132+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f40047e0 con 0x7f8604070f10 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.132+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4004bd0 con 0x7f8604070f10 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.132+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86041e44c0 con 0x7f8604070f10 2024-03-06T07:39:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.132+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86041e4a80 con 0x7f8604070f10 2024-03-06T07:39:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.133+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860404f330 con 0x7f8604070f10 2024-03-06T07:39:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.134+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85f4011070 con 0x7f8604070f10 2024-03-06T07:39:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.135+0000 7f8600c00640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.135+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85f4089ea0 con 0x7f8604070f10 2024-03-06T07:39:24.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.136+0000 7f85fb400640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.136+0000 7f85fb400640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f860406e9a0 tx=0x7f85ec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.140+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85f4058c40 con 0x7f8604070f10 2024-03-06T07:39:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.383+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86041e50d0 con 0x7f8604070f10 2024-03-06T07:39:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.384+0000 7f8600c00640 1 -- 172.21.15.32:0/1725693695 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3462 (secure 0 0 0) 0x7f85f4058540 con 0x7f8604070f10 2024-03-06T07:39:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:24.385 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:37:55.811074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.387+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85e006b4d0 msgr2=0x7f85e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.387+0000 7f8609800640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f860406e9a0 tx=0x7f85ec008040 comp rx=0 tx=0).stop 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.387+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 msgr2=0x7f86041f61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.387+0000 7f8609800640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 0x7f86041f61b0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f85f400c460 tx=0x7f85f400c920 comp rx=0 tx=0).stop 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 shutdown_connections 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8604145300 0x7f860406d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86040cce90 0x7f86041e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 --2- 172.21.15.32:0/1725693695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8604070f10 0x7f86041f61b0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 >> 172.21.15.32:0/1725693695 conn(0x7f860406b3d0 msgr2=0x7f8604072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 shutdown_connections 2024-03-06T07:39:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:24.388+0000 7f8609800640 1 -- 172.21.15.32:0/1725693695 wait complete. 2024-03-06T07:39:24.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:37:55.811074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:24.611 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:25.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:25.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.860+0000 7f14ca400640 1 Processor -- start 2024-03-06T07:39:25.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.861+0000 7f14ca400640 1 -- start start 2024-03-06T07:39:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.862+0000 7f14ca400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c41477e0 0x7f14c4162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14ca400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f14c406b6c0 0x7f14c40d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14ca400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c4147bc0 con 0x7f14c406ef20 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14ca400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c4161f90 con 0x7f14c41477e0 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14ca400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c4162960 con 0x7f14c406b6c0 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14c9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c41477e0 0x7f14c4162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14c9000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c41477e0 0x7f14c4162420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45496/0 (socket says 172.21.15.32:45496) 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14c9000640 1 -- 172.21.15.32:0/1864057591 learned_addr learned my addr 172.21.15.32:0/1864057591 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14c9a00640 1 --2- 172.21.15.32:0/1864057591 >> v2:172.21.15.39:3300/0 conn(0x7f14c406b6c0 0x7f14c40d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14c9000640 1 -- 172.21.15.32:0/1864057591 >> v2:172.21.15.39:3300/0 conn(0x7f14c406b6c0 msgr2=0x7f14c40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.863+0000 7f14c9000640 1 --2- 172.21.15.32:0/1864057591 >> v2:172.21.15.39:3300/0 conn(0x7f14c406b6c0 0x7f14c40d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.864+0000 7f14c9000640 1 -- 172.21.15.32:0/1864057591 >> v1:172.21.15.39:6789/0 conn(0x7f14c406ef20 legacy=0x7f14c406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:25.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.864+0000 7f14c9000640 1 -- 172.21.15.32:0/1864057591 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40b3d70 con 0x7f14c41477e0 2024-03-06T07:39:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.865+0000 7f14c9000640 1 --2- 172.21.15.32:0/1864057591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c41477e0 0x7f14c4162420 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f14c40d1c10 tx=0x7f14b402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe4ff7d66f07bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.865+0000 7f14c3400640 1 -- 172.21.15.32:0/1864057591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b402c9f0 con 0x7f14c41477e0 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14c3400640 1 -- 172.21.15.32:0/1864057591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b402cb90 con 0x7f14c41477e0 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14c3400640 1 -- 172.21.15.32:0/1864057591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b40343f0 con 0x7f14c41477e0 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 -- 172.21.15.32:0/1864057591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c41477e0 msgr2=0x7f14c4162420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 --2- 172.21.15.32:0/1864057591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c41477e0 0x7f14c4162420 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f14c40d1c10 tx=0x7f14b402be50 comp rx=0 tx=0).stop 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 -- 172.21.15.32:0/1864057591 shutdown_connections 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 --2- 172.21.15.32:0/1864057591 >> v2:172.21.15.39:3300/0 conn(0x7f14c406b6c0 0x7f14c40d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 --2- 172.21.15.32:0/1864057591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c41477e0 0x7f14c4162420 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 -- 172.21.15.32:0/1864057591 >> 172.21.15.32:0/1864057591 conn(0x7f14c406c0c0 msgr2=0x7f14c406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 -- 172.21.15.32:0/1864057591 shutdown_connections 2024-03-06T07:39:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.866+0000 7f14ca400640 1 -- 172.21.15.32:0/1864057591 wait complete. 2024-03-06T07:39:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.868+0000 7f14ca400640 1 Processor -- start 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.868+0000 7f14ca400640 1 -- start start 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14ca400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 0x7f14c413c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14ca400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c406ef20 0x7f14c413cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14ca400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c41477e0 0x7f14c41418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14ca400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4161c20 con 0x7f14c406b6c0 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14ca400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b4330 con 0x7f14c406ef20 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14ca400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c406f8f0 con 0x7f14c41477e0 2024-03-06T07:39:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14c9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 0x7f14c413c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.869+0000 7f14c9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 0x7f14c413c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45504/0 (socket says 172.21.15.32:45504) 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 -- 172.21.15.32:0/306541846 learned_addr learned my addr 172.21.15.32:0/306541846 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 -- 172.21.15.32:0/306541846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c41477e0 msgr2=0x7f14c41418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c41477e0 0x7f14c41418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 -- 172.21.15.32:0/306541846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c406ef20 msgr2=0x7f14c413cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c406ef20 0x7f14c413cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 -- 172.21.15.32:0/306541846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c413fdc0 con 0x7f14c406b6c0 2024-03-06T07:39:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.870+0000 7f14c9000640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 0x7f14c413c8c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000f80 tx=0x7f14b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.871+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043020 con 0x7f14c406b6c0 2024-03-06T07:39:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.871+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b403e450 con 0x7f14c406b6c0 2024-03-06T07:39:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.871+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4009510 con 0x7f14c406b6c0 2024-03-06T07:39:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.871+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c41400a0 con 0x7f14c406b6c0 2024-03-06T07:39:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.871+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c41413c0 con 0x7f14c406b6c0 2024-03-06T07:39:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.872+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c404f330 con 0x7f14c406b6c0 2024-03-06T07:39:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.874+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14b4046020 con 0x7f14c406b6c0 2024-03-06T07:39:25.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.875+0000 7f14c0c00640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a006b4d0 0x7f14a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.875+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f14b40acb70 con 0x7f14c406b6c0 2024-03-06T07:39:25.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.875+0000 7f14c3e00640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a006b4d0 0x7f14a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.876+0000 7f14c3e00640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a006b4d0 0x7f14a006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f14c40b4a10 tx=0x7f14ac0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:25.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:25.885+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14b404d030 con 0x7f14c406b6c0 2024-03-06T07:39:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.121+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14c40724b0 con 0x7f14c406b6c0 2024-03-06T07:39:26.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.122+0000 7f14c0c00640 1 -- 172.21.15.32:0/306541846 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f14b407b9a0 con 0x7f14c406b6c0 2024-03-06T07:39:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:26.124 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:26.124 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:26.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a006b4d0 msgr2=0x7f14a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a006b4d0 0x7f14a006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f14c40b4a10 tx=0x7f14ac0079b0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 msgr2=0x7f14c413c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 0x7f14c413c8c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000f80 tx=0x7f14b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 shutdown_connections 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f14a006b4d0 0x7f14a006d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f14c41477e0 0x7f14c41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c406ef20 0x7f14c413cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 --2- 172.21.15.32:0/306541846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c406b6c0 0x7f14c413c8c0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 >> 172.21.15.32:0/306541846 conn(0x7f14c406c0c0 msgr2=0x7f14c406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.124+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 shutdown_connections 2024-03-06T07:39:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:26.125+0000 7f14ca400640 1 -- 172.21.15.32:0/306541846 wait complete. 2024-03-06T07:39:26.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:26.346 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:27.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:27.446 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.597+0000 7fd979400640 1 Processor -- start 2024-03-06T07:39:27.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.598+0000 7fd979400640 1 -- start start 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.599+0000 7fd979400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 0x7fd9740adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.599+0000 7fd979400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd9740b5990 0x7fd97415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.599+0000 7fd979400640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740b4990 con 0x7fd97415ee70 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.599+0000 7fd979400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd974144bb0 con 0x7fd9740b5990 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.599+0000 7fd979400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9740b4c70 con 0x7fd9741447d0 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.600+0000 7fd973200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd97415ee70 0x7fd9740af790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52978/0 (socket says 172.21.15.32:52978) 2024-03-06T07:39:27.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.600+0000 7fd973200640 1 -- 172.21.15.32:0/1077864550 learned_addr learned my addr 172.21.15.32:0/1077864550 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:27.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.600+0000 7fd972800640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 0x7fd9740adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.600+0000 7fd96be00640 1 --2- 172.21.15.32:0/1077864550 >> v2:172.21.15.39:3300/0 conn(0x7fd9740b5990 0x7fd97415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.600+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846566990 0 0) 0x7fd9740b4990 con 0x7fd97415ee70 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.600+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd958003520 con 0x7fd97415ee70 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1016030435 0 0) 0x7fd958003520 con 0x7fd97415ee70 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 msgr2=0x7fd9740adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 0x7fd9740adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 >> v2:172.21.15.39:3300/0 conn(0x7fd9740b5990 msgr2=0x7fd97415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 --2- 172.21.15.32:0/1077864550 >> v2:172.21.15.39:3300/0 conn(0x7fd9740b5990 0x7fd97415e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740b4f50 con 0x7fd97415ee70 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd96c0019f0 con 0x7fd97415ee70 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 >> v1:172.21.15.32:6789/0 conn(0x7fd97415ee70 legacy=0x7fd9740af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958003a20 0x7fd958003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd958004800 0x7fd958004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 0x7fd9580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974144bb0 con 0x7fd958004800 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9740b4c70 con 0x7fd958003a20 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95800dbd0 con 0x7fd9741447d0 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.601+0000 7fd96be00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd958004800 0x7fd958004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.602+0000 7fd972800640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958003a20 0x7fd958003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.602+0000 7fd96be00640 1 -- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 msgr2=0x7fd9580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.602+0000 7fd96be00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9741447d0 0x7fd9580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.602+0000 7fd96be00640 1 -- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958003a20 msgr2=0x7fd958003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.602+0000 7fd96be00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958003a20 0x7fd958003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.602+0000 7fd96be00640 1 -- 172.21.15.32:0/1077864550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958003520 con 0x7fd958004800 2024-03-06T07:39:27.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.603+0000 7fd96be00640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd958004800 0x7fd958004c80 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009af0 tx=0x7fd958005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4f2a9371af9e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:27.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.603+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c039070 con 0x7fd958004800 2024-03-06T07:39:27.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.603+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c02c710 con 0x7fd958004800 2024-03-06T07:39:27.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.603+0000 7fd971e00640 1 -- 172.21.15.32:0/1077864550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c009690 con 0x7fd958004800 2024-03-06T07:39:27.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd979400640 1 -- 172.21.15.32:0/1077864550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd958004800 msgr2=0x7fd958004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd979400640 1 --2- 172.21.15.32:0/1077864550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd958004800 0x7fd958004c80 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009af0 tx=0x7fd958005830 comp rx=0 tx=0).stop 2024-03-06T07:39:27.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd973200640 1 -- 172.21.15.32:0/1077864550 reap_dead start 2024-03-06T07:39:27.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd979400640 1 -- 172.21.15.32:0/1077864550 shutdown_connections 2024-03-06T07:39:27.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd979400640 1 -- 172.21.15.32:0/1077864550 >> 172.21.15.32:0/1077864550 conn(0x7fd9740b2ae0 msgr2=0x7fd9740b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:27.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd979400640 1 -- 172.21.15.32:0/1077864550 shutdown_connections 2024-03-06T07:39:27.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.604+0000 7fd979400640 1 -- 172.21.15.32:0/1077864550 wait complete. 2024-03-06T07:39:27.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.607+0000 7fd979400640 1 Processor -- start 2024-03-06T07:39:27.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.607+0000 7fd979400640 1 -- start start 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd979400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958004800 0x7fd97413a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd979400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9740ce1d0 0x7fd9740ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd979400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 0x7fd9740cc980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd979400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97406f100 con 0x7fd9740cc500 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd979400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97406ef80 con 0x7fd958004800 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd979400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97406f280 con 0x7fd9740ce1d0 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd973200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 0x7fd9740cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd973200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 0x7fd9740cc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45526/0 (socket says 172.21.15.32:45526) 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd973200640 1 -- 172.21.15.32:0/2735241255 learned_addr learned my addr 172.21.15.32:0/2735241255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd972800640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958004800 0x7fd97413a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.609+0000 7fd96be00640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9740ce1d0 0x7fd9740ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd973200640 1 -- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9740ce1d0 msgr2=0x7fd9740ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd973200640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9740ce1d0 0x7fd9740ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd973200640 1 -- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958004800 msgr2=0x7fd97413a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd973200640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958004800 0x7fd97413a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd973200640 1 -- 172.21.15.32:0/2735241255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740cec70 con 0x7fd9740cc500 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd972800640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958004800 0x7fd97413a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd973200640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 0x7fd9740cc980 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fd96c001370 tx=0x7fd96c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00ce50 con 0x7fd9740cc500 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c00a5b0 con 0x7fd9740cc500 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00a9a0 con 0x7fd9740cc500 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9740cef50 con 0x7fd9740cc500 2024-03-06T07:39:27.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.610+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9740cf620 con 0x7fd9740cc500 2024-03-06T07:39:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.612+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97404f3a0 con 0x7fd9740cc500 2024-03-06T07:39:27.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.613+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd96c011070 con 0x7fd9740cc500 2024-03-06T07:39:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.614+0000 7fd96aa00640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95006b4d0 0x7fd95006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.615+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd96c0897e0 con 0x7fd9740cc500 2024-03-06T07:39:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.615+0000 7fd972800640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95006b4d0 0x7fd95006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.615+0000 7fd972800640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95006b4d0 0x7fd95006d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd964006ce0 tx=0x7fd964008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:27.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.619+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd96c058580 con 0x7fd9740cc500 2024-03-06T07:39:27.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.869+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd974070170 con 0x7fd9740cc500 2024-03-06T07:39:27.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.870+0000 7fd96aa00640 1 -- 172.21.15.32:0/2735241255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fd96c057e80 con 0x7fd9740cc500 2024-03-06T07:39:27.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:27.872 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:27.872 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95006b4d0 msgr2=0x7fd95006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95006b4d0 0x7fd95006d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd964006ce0 tx=0x7fd964008040 comp rx=0 tx=0).stop 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 msgr2=0x7fd9740cc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 0x7fd9740cc980 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fd96c001370 tx=0x7fd96c00cc00 comp rx=0 tx=0).stop 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 shutdown_connections 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd95006b4d0 0x7fd95006d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9740cc500 0x7fd9740cc980 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd9740ce1d0 0x7fd9740ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 --2- 172.21.15.32:0/2735241255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd958004800 0x7fd97413a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 >> 172.21.15.32:0/2735241255 conn(0x7fd9740b2ae0 msgr2=0x7fd9740b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:27.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 shutdown_connections 2024-03-06T07:39:27.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:27.873+0000 7fd979400640 1 -- 172.21.15.32:0/2735241255 wait complete. 2024-03-06T07:39:28.094 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:28.094 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:29.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.321+0000 7fd1c1a00640 1 Processor -- start 2024-03-06T07:39:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.322+0000 7fd1c1a00640 1 -- start start 2024-03-06T07:39:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.323+0000 7fd1c1a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1bc0d07b0 0x7fd1bc15d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.323+0000 7fd1c1a00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0b1370 con 0x7fd1bc0b0f90 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1c1a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0ccf80 con 0x7fd1bc0cf080 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1c1a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc0cd260 con 0x7fd1bc0d07b0 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1bbe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd1bc0b0f90 0x7fd1bc0cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37122/0 (socket says 172.21.15.32:37122) 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1bbe00640 1 -- 172.21.15.32:0/582402669 learned_addr learned my addr 172.21.15.32:0/582402669 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1c1000640 1 --2- 172.21.15.32:0/582402669 >> v2:172.21.15.39:3300/0 conn(0x7fd1bc0d07b0 0x7fd1bc15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283464649 0 0) 0x7fd1bc0b1370 con 0x7fd1bc0b0f90 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.324+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a0003520 con 0x7fd1bc0b0f90 2024-03-06T07:39:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443332126 0 0) 0x7fd1a0003520 con 0x7fd1bc0b0f90 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 >> v2:172.21.15.39:3300/0 conn(0x7fd1bc0d07b0 msgr2=0x7fd1bc15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 --2- 172.21.15.32:0/582402669 >> v2:172.21.15.39:3300/0 conn(0x7fd1bc0d07b0 0x7fd1bc15d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 >> v1:172.21.15.39:6789/0 conn(0x7fd1bc0cf080 legacy=0x7fd1bc0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0cd540 con 0x7fd1bc0b0f90 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd1ac001670 con 0x7fd1bc0b0f90 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 >> v1:172.21.15.32:6789/0 conn(0x7fd1bc0b0f90 legacy=0x7fd1bc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1a0003a00 0x7fd1a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a0004800 0x7fd1a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0005210 0x7fd1a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0cd260 con 0x7fd1a0004800 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a000c160 con 0x7fd1a0005210 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a000c360 con 0x7fd1a0003a00 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a0004800 0x7fd1a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1bbe00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1a0003a00 0x7fd1a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.325+0000 7fd1c1000640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0005210 0x7fd1a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1b3e00640 1 -- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1a0003a00 msgr2=0x7fd1a0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1a0003a00 0x7fd1a0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1b3e00640 1 -- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0005210 msgr2=0x7fd1a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0005210 0x7fd1a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1b3e00640 1 -- 172.21.15.32:0/582402669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a0003520 con 0x7fd1a0004800 2024-03-06T07:39:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1c1000640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0005210 0x7fd1a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.326+0000 7fd1c1000640 1 -- 172.21.15.32:0/582402669 reap_dead start 2024-03-06T07:39:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.327+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a0004800 0x7fd1a0004c80 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4001720 tx=0x7fd1a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b926dd6f688edbd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.327+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4001370 con 0x7fd1a0004800 2024-03-06T07:39:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.327+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a40052d0 con 0x7fd1a0004800 2024-03-06T07:39:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.327+0000 7fd1bb400640 1 -- 172.21.15.32:0/582402669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4003830 con 0x7fd1a0004800 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 -- 172.21.15.32:0/582402669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a0004800 msgr2=0x7fd1a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a0004800 0x7fd1a0004c80 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4001720 tx=0x7fd1a0005830 comp rx=0 tx=0).stop 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 -- 172.21.15.32:0/582402669 shutdown_connections 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/582402669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a0004800 0x7fd1a0004c80 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 -- 172.21.15.32:0/582402669 >> 172.21.15.32:0/582402669 conn(0x7fd1bc0d1ae0 msgr2=0x7fd1bc0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 -- 172.21.15.32:0/582402669 shutdown_connections 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.328+0000 7fd1c1a00640 1 -- 172.21.15.32:0/582402669 wait complete. 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.330+0000 7fd1c1a00640 1 Processor -- start 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.330+0000 7fd1c1a00640 1 -- start start 2024-03-06T07:39:29.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1c1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0004800 0x7fd1bc0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1c1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 0x7fd1bc0cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1c1a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1bc0cbae0 0x7fd1bc0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1c1a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0d0220 con 0x7fd1bc14fe30 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1c1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0d00a0 con 0x7fd1a0004800 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1c1a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0d03a0 con 0x7fd1bc0cbae0 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 0x7fd1bc0cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 0x7fd1bc0cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41878/0 (socket says 172.21.15.32:41878) 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.331+0000 7fd1b3e00640 1 -- 172.21.15.32:0/1768152916 learned_addr learned my addr 172.21.15.32:0/1768152916 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1c1000640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1bc0cbae0 0x7fd1bc0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1bbe00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0004800 0x7fd1bc0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b3e00640 1 -- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1bc0cbae0 msgr2=0x7fd1bc0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1bc0cbae0 0x7fd1bc0c8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b3e00640 1 -- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0004800 msgr2=0x7fd1bc0b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0004800 0x7fd1bc0b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b3e00640 1 -- 172.21.15.32:0/1768152916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0b2140 con 0x7fd1bc14fe30 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1bbe00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0004800 0x7fd1bc0b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 0x7fd1bc0cb5a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4034040 tx=0x7fd1a4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.332+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4041020 con 0x7fd1bc14fe30 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.333+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4005480 con 0x7fd1bc14fe30 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.333+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0b2420 con 0x7fd1bc14fe30 2024-03-06T07:39:29.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.333+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4007a30 con 0x7fd1bc14fe30 2024-03-06T07:39:29.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.333+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc0b29e0 con 0x7fd1bc14fe30 2024-03-06T07:39:29.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.334+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc068120 con 0x7fd1bc14fe30 2024-03-06T07:39:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.335+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1a4044020 con 0x7fd1bc14fe30 2024-03-06T07:39:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.336+0000 7fd1b8c00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19806b4d0 0x7fd19806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.337+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1a40abd80 con 0x7fd1bc14fe30 2024-03-06T07:39:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.337+0000 7fd1bbe00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19806b4d0 0x7fd19806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:29.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.337+0000 7fd1bbe00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19806b4d0 0x7fd19806d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac001eb0 tx=0x7fd1ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:29.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.341+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1bc068120 con 0x7fd1bc14fe30 2024-03-06T07:39:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.581+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1bc0aed30 con 0x7fd1bc14fe30 2024-03-06T07:39:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.582+0000 7fd1b8c00640 1 -- 172.21.15.32:0/1768152916 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fd1a404b030 con 0x7fd1bc14fe30 2024-03-06T07:39:29.583 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:29.584 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:29.584 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.584+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19806b4d0 msgr2=0x7fd19806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.584+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19806b4d0 0x7fd19806d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac001eb0 tx=0x7fd1ac009040 comp rx=0 tx=0).stop 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 msgr2=0x7fd1bc0cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 0x7fd1bc0cb5a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4034040 tx=0x7fd1a4001120 comp rx=0 tx=0).stop 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 shutdown_connections 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd19806b4d0 0x7fd19806d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd1bc0cbae0 0x7fd1bc0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc14fe30 0x7fd1bc0cb5a0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 --2- 172.21.15.32:0/1768152916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a0004800 0x7fd1bc0b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 >> 172.21.15.32:0/1768152916 conn(0x7fd1bc0d1ae0 msgr2=0x7fd1bc0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 shutdown_connections 2024-03-06T07:39:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:29.585+0000 7fd1c1a00640 1 -- 172.21.15.32:0/1768152916 wait complete. 2024-03-06T07:39:29.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:29.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:30.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.030+0000 7fb0c0c00640 1 Processor -- start 2024-03-06T07:39:31.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.030+0000 7fb0c0c00640 1 -- start start 2024-03-06T07:39:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0c0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc070f10 0x7fb0bc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0c0c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0bc0cce90 0x7fb0bc15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0c0c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc0714d0 con 0x7fb0bc145300 2024-03-06T07:39:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0c0c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc0cd300 con 0x7fb0bc0cce90 2024-03-06T07:39:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0c0c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0717b0 con 0x7fb0bc070f10 2024-03-06T07:39:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0bbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0bc0cce90 0x7fb0bc15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0bbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0bc0cce90 0x7fb0bc15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41884/0 (socket says 172.21.15.32:41884) 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.032+0000 7fb0bbe00640 1 -- 172.21.15.32:0/281616177 learned_addr learned my addr 172.21.15.32:0/281616177 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.033+0000 7fb0baa00640 1 --2- 172.21.15.32:0/281616177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc070f10 0x7fb0bc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.033+0000 7fb0bbe00640 1 -- 172.21.15.32:0/281616177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc070f10 msgr2=0x7fb0bc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.033+0000 7fb0bbe00640 1 --2- 172.21.15.32:0/281616177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc070f10 0x7fb0bc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.033+0000 7fb0bbe00640 1 -- 172.21.15.32:0/281616177 >> v1:172.21.15.39:6789/0 conn(0x7fb0bc145300 legacy=0x7fb0bc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.033+0000 7fb0bbe00640 1 -- 172.21.15.32:0/281616177 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc071a90 con 0x7fb0bc0cce90 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.034+0000 7fb0bbe00640 1 --2- 172.21.15.32:0/281616177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc0cce90 0x7fb0bc15c570 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009e50 tx=0x7fb0b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6d4395ab2c0dab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0ba000640 1 -- 172.21.15.32:0/281616177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b402cb00 con 0x7fb0bc0cce90 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0ba000640 1 -- 172.21.15.32:0/281616177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b402cca0 con 0x7fb0bc0cce90 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0ba000640 1 -- 172.21.15.32:0/281616177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4034610 con 0x7fb0bc0cce90 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0c0c00640 1 -- 172.21.15.32:0/281616177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc0cce90 msgr2=0x7fb0bc15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/281616177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc0cce90 0x7fb0bc15c570 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009e50 tx=0x7fb0b402bf60 comp rx=0 tx=0).stop 2024-03-06T07:39:31.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0c0c00640 1 -- 172.21.15.32:0/281616177 shutdown_connections 2024-03-06T07:39:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/281616177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc0cce90 0x7fb0bc15c570 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/281616177 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc070f10 0x7fb0bc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.035+0000 7fb0c0c00640 1 -- 172.21.15.32:0/281616177 >> 172.21.15.32:0/281616177 conn(0x7fb0bc06b3d0 msgr2=0x7fb0bc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.036+0000 7fb0c0c00640 1 -- 172.21.15.32:0/281616177 shutdown_connections 2024-03-06T07:39:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.036+0000 7fb0c0c00640 1 -- 172.21.15.32:0/281616177 wait complete. 2024-03-06T07:39:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.037+0000 7fb0c0c00640 1 Processor -- start 2024-03-06T07:39:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.038+0000 7fb0c0c00640 1 -- start start 2024-03-06T07:39:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.038+0000 7fb0c0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc070f10 0x7fb0bc143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.038+0000 7fb0c0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc0cce90 0x7fb0bc1e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0c0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 0x7fb0bc1e8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0c0c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc15cab0 con 0x7fb0bc145300 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0c0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc1456e0 con 0x7fb0bc070f10 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0c0c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc15cc30 con 0x7fb0bc0cce90 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 0x7fb0bc1e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 0x7fb0bc1e8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41896/0 (socket says 172.21.15.32:41896) 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 -- 172.21.15.32:0/661706653 learned_addr learned my addr 172.21.15.32:0/661706653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0baa00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc0cce90 0x7fb0bc1e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bb400640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc070f10 0x7fb0bc143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 -- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc0cce90 msgr2=0x7fb0bc1e83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc0cce90 0x7fb0bc1e83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 -- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc070f10 msgr2=0x7fb0bc143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc070f10 0x7fb0bc143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.039+0000 7fb0bbe00640 1 -- 172.21.15.32:0/661706653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc1e7ef0 con 0x7fb0bc145300 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.040+0000 7fb0bbe00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 0x7fb0bc1e8930 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4036040 tx=0x7fb0b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.040+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4043020 con 0x7fb0bc145300 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.040+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b402ce50 con 0x7fb0bc145300 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.040+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4009510 con 0x7fb0bc145300 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.040+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc1e9200 con 0x7fb0bc145300 2024-03-06T07:39:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.041+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc1e8e70 con 0x7fb0bc145300 2024-03-06T07:39:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.042+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc0b28b0 con 0x7fb0bc145300 2024-03-06T07:39:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.044+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0b4046020 con 0x7fb0bc145300 2024-03-06T07:39:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.044+0000 7fb0b3400640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb09c06b4d0 0x7fb09c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.045+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb0b40ac8b0 con 0x7fb0bc145300 2024-03-06T07:39:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.045+0000 7fb0bb400640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb09c06b4d0 0x7fb09c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.046+0000 7fb0bb400640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb09c06b4d0 0x7fb09c06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac007fd0 tx=0x7fb0ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.050+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0bc0b28b0 con 0x7fb0bc145300 2024-03-06T07:39:31.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.288+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0bc1e81d0 con 0x7fb0bc145300 2024-03-06T07:39:31.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.289+0000 7fb0b3400640 1 -- 172.21.15.32:0/661706653 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fb0b404d030 con 0x7fb0bc145300 2024-03-06T07:39:31.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:31.291 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:31.291 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:31.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb09c06b4d0 msgr2=0x7fb09c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb09c06b4d0 0x7fb09c06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac007fd0 tx=0x7fb0ac009040 comp rx=0 tx=0).stop 2024-03-06T07:39:31.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 msgr2=0x7fb0bc1e8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 0x7fb0bc1e8930 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4036040 tx=0x7fb0b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 shutdown_connections 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb09c06b4d0 0x7fb09c06d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0bc145300 0x7fb0bc1e8930 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb0bc0cce90 0x7fb0bc1e83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 --2- 172.21.15.32:0/661706653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc070f10 0x7fb0bc143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.291+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 >> 172.21.15.32:0/661706653 conn(0x7fb0bc06b3d0 msgr2=0x7fb0bc0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.292+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 shutdown_connections 2024-03-06T07:39:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:31.292+0000 7fb0c0c00640 1 -- 172.21.15.32:0/661706653 wait complete. 2024-03-06T07:39:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:31.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:31.512 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:32.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.740+0000 7f2aa7200640 1 Processor -- start 2024-03-06T07:39:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.741+0000 7f2aa7200640 1 -- start start 2024-03-06T07:39:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.742+0000 7f2aa7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2aa8145300 0x7f2aa80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.742+0000 7f2aa7200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa80714d0 con 0x7f2aa80cd060 2024-03-06T07:39:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.743+0000 7f2aa7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa80717b0 con 0x7f2aa8070f10 2024-03-06T07:39:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.743+0000 7f2aa7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa8071a90 con 0x7f2aa8145300 2024-03-06T07:39:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.743+0000 7f2aa6800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2aa80cd060 0x7f2aa815c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37130/0 (socket says 172.21.15.32:37130) 2024-03-06T07:39:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.743+0000 7f2aa6800640 1 -- 172.21.15.32:0/3326992634 learned_addr learned my addr 172.21.15.32:0/3326992634 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.743+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/3326992634 >> v2:172.21.15.39:3300/0 conn(0x7f2aa8145300 0x7f2aa80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.743+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469498080 0 0) 0x7f2aa80714d0 con 0x7f2aa80cd060 2024-03-06T07:39:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c003520 con 0x7f2aa80cd060 2024-03-06T07:39:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760677998 0 0) 0x7f2a8c003520 con 0x7f2aa80cd060 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 >> v2:172.21.15.39:3300/0 conn(0x7f2aa8145300 msgr2=0x7f2aa80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 --2- 172.21.15.32:0/3326992634 >> v2:172.21.15.39:3300/0 conn(0x7f2aa8145300 0x7f2aa80b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 >> v1:172.21.15.39:6789/0 conn(0x7f2aa8070f10 legacy=0x7f2aa80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8071d70 con 0x7f2aa80cd060 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f2aa00019f0 con 0x7f2aa80cd060 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 >> v1:172.21.15.32:6789/0 conn(0x7f2aa80cd060 legacy=0x7f2aa815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a8c003a00 0x7f2a8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c005210 0x7f2a8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8071a90 con 0x7f2a8c003a00 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c00c160 con 0x7f2a8c004800 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.744+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c00c360 con 0x7f2a8c005210 2024-03-06T07:39:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a8c003a00 0x7f2a8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2a9d400640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2aa6800640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c005210 0x7f2a8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2aa5e00640 1 -- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c005210 msgr2=0x7f2a8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c005210 0x7f2a8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2aa5e00640 1 -- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8c004800 msgr2=0x7f2a8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.745+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a8c004800 0x7f2a8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.746+0000 7f2aa5e00640 1 -- 172.21.15.32:0/3326992634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c003520 con 0x7f2a8c003a00 2024-03-06T07:39:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.746+0000 7f2aa6800640 1 -- 172.21.15.32:0/3326992634 reap_dead start 2024-03-06T07:39:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.747+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a8c003a00 0x7f2a8c003e20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009a90 tx=0x7f2a8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8680ba6c409d7cba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.748+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98039070 con 0x7f2a8c003a00 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.748+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9802c6b0 con 0x7f2a8c003a00 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.748+0000 7f2aa5400640 1 -- 172.21.15.32:0/3326992634 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98009690 con 0x7f2a8c003a00 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.748+0000 7f2aa7200640 1 -- 172.21.15.32:0/3326992634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a8c003a00 msgr2=0x7f2a8c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.748+0000 7f2aa7200640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a8c003a00 0x7f2a8c003e20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009a90 tx=0x7f2a8c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.749+0000 7f2aa7200640 1 -- 172.21.15.32:0/3326992634 shutdown_connections 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.749+0000 7f2aa7200640 1 --2- 172.21.15.32:0/3326992634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a8c003a00 0x7f2a8c003e20 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.749+0000 7f2aa7200640 1 -- 172.21.15.32:0/3326992634 >> 172.21.15.32:0/3326992634 conn(0x7f2aa806b3d0 msgr2=0x7f2aa806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.749+0000 7f2aa7200640 1 -- 172.21.15.32:0/3326992634 shutdown_connections 2024-03-06T07:39:32.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.749+0000 7f2aa7200640 1 -- 172.21.15.32:0/3326992634 wait complete. 2024-03-06T07:39:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.751+0000 7f2aa7200640 1 Processor -- start 2024-03-06T07:39:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.751+0000 7f2aa7200640 1 -- start start 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa7200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c003a00 0x7f2aa80cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 0x7f2aa80b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa80b1d30 0x7f2aa8135990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80ccad0 con 0x7f2aa80cb660 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80cc950 con 0x7f2aa80b1d30 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa7200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80ccc50 con 0x7f2a8c003a00 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2a9d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 0x7f2aa80b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2a9d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 0x7f2aa80b17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41916/0 (socket says 172.21.15.32:41916) 2024-03-06T07:39:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2a9d400640 1 -- 172.21.15.32:0/2053698064 learned_addr learned my addr 172.21.15.32:0/2053698064 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c003a00 0x7f2aa80cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.752+0000 7f2aa6800640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa80b1d30 0x7f2aa8135990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9d400640 1 -- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c003a00 msgr2=0x7f2aa80cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9d400640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c003a00 0x7f2aa80cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9d400640 1 -- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa80b1d30 msgr2=0x7f2aa8135990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9d400640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa80b1d30 0x7f2aa8135990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9d400640 1 -- 172.21.15.32:0/2053698064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8137ba0 con 0x7f2aa80cb660 2024-03-06T07:39:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9d400640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 0x7f2aa80b17f0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000f540 tx=0x7f2a9000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a900052e0 con 0x7f2aa80cb660 2024-03-06T07:39:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90018070 con 0x7f2aa80cb660 2024-03-06T07:39:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90003830 con 0x7f2aa80cb660 2024-03-06T07:39:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa8138a30 con 0x7f2aa80cb660 2024-03-06T07:39:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.753+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa8138ff0 con 0x7f2aa80cb660 2024-03-06T07:39:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.755+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa815efc0 con 0x7f2aa80cb660 2024-03-06T07:39:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.756+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a900039d0 con 0x7f2aa80cb660 2024-03-06T07:39:32.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.757+0000 7f2a9ea00640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8406b520 0x7f2a8406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.757+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a90089ff0 con 0x7f2aa80cb660 2024-03-06T07:39:32.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.757+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8406b520 0x7f2a8406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.758+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8406b520 0x7f2a8406d9e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2a98036000 tx=0x7f2a98030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:32.762+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2aa815efc0 con 0x7f2aa80cb660 2024-03-06T07:39:33.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.100+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa80b2110 con 0x7f2aa80cb660 2024-03-06T07:39:33.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.100+0000 7f2a9ea00640 1 -- 172.21.15.32:0/2053698064 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f2a90058de0 con 0x7f2aa80cb660 2024-03-06T07:39:33.102 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:33.102 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:33.102 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:33.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8406b520 msgr2=0x7f2a8406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8406b520 0x7f2a8406d9e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2a98036000 tx=0x7f2a98030000 comp rx=0 tx=0).stop 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 msgr2=0x7f2aa80b17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 0x7f2aa80b17f0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000f540 tx=0x7f2a9000fa00 comp rx=0 tx=0).stop 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 shutdown_connections 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2a8406b520 0x7f2a8406d9e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aa80b1d30 0x7f2aa8135990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aa80cb660 0x7f2aa80b17f0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 --2- 172.21.15.32:0/2053698064 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2a8c003a00 0x7f2aa80cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 >> 172.21.15.32:0/2053698064 conn(0x7f2aa806b3d0 msgr2=0x7f2aa8072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 shutdown_connections 2024-03-06T07:39:33.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:33.103+0000 7f2aa7200640 1 -- 172.21.15.32:0/2053698064 wait complete. 2024-03-06T07:39:33.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:33.327 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:34.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.544+0000 7f6813e00640 1 Processor -- start 2024-03-06T07:39:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.545+0000 7f6813e00640 1 -- start start 2024-03-06T07:39:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.546+0000 7f6813e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 0x7f681406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.546+0000 7f6813e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f68140ce370 0x7f68140ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.546+0000 7f6813e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68140cf1b0 con 0x7f6814066d00 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.546+0000 7f6813e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681415e470 con 0x7f68140ce370 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.546+0000 7f6813e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681415ea00 con 0x7f681406c8b0 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.547+0000 7f680be00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6814066d00 0x7f681415ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37134/0 (socket says 172.21.15.32:37134) 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.547+0000 7f680be00640 1 -- 172.21.15.32:0/4200621296 learned_addr learned my addr 172.21.15.32:0/4200621296 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.547+0000 7f6812a00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 0x7f681406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.547+0000 7f6813400640 1 --2- 172.21.15.32:0/4200621296 >> v2:172.21.15.39:3300/0 conn(0x7f68140ce370 0x7f68140ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.547+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687731913 0 0) 0x7f68140cf1b0 con 0x7f6814066d00 2024-03-06T07:39:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.547+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f8003520 con 0x7f6814066d00 2024-03-06T07:39:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192387328 0 0) 0x7f67f8003520 con 0x7f6814066d00 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 msgr2=0x7f681406ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 0x7f681406ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 >> v2:172.21.15.39:3300/0 conn(0x7f68140ce370 msgr2=0x7f68140ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 --2- 172.21.15.32:0/4200621296 >> v2:172.21.15.39:3300/0 conn(0x7f68140ce370 0x7f68140ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681406d420 con 0x7f6814066d00 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f67fc001670 con 0x7f6814066d00 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 >> v1:172.21.15.32:6789/0 conn(0x7f6814066d00 legacy=0x7f681415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 0x7f67f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8003f80 0x7f67f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f8004d40 0x7f67f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681415e470 con 0x7f67f8003f80 2024-03-06T07:39:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681415ea00 con 0x7f67f8004d40 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f800b9c0 con 0x7f681406c8b0 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.548+0000 7f680be00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8003f80 0x7f67f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f6812a00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 0x7f67f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f6813400640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f8004d40 0x7f67f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f680be00640 1 -- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 msgr2=0x7f67f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f680be00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f681406c8b0 0x7f67f80039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f680be00640 1 -- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f8004d40 msgr2=0x7f67f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f680be00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f8004d40 0x7f67f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f680be00640 1 -- 172.21.15.32:0/4200621296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8003520 con 0x7f67f8003f80 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.549+0000 7f6813400640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67f8004d40 0x7f67f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.550+0000 7f680be00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8003f80 0x7f67f8004800 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c160 tx=0x7f67f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb3fe6965383b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.550+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc007c80 con 0x7f67f8003f80 2024-03-06T07:39:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.550+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc030070 con 0x7f67f8003f80 2024-03-06T07:39:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.550+0000 7f6812000640 1 -- 172.21.15.32:0/4200621296 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc005d70 con 0x7f67f8003f80 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813e00640 1 -- 172.21.15.32:0/4200621296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8003f80 msgr2=0x7f67f8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813e00640 1 --2- 172.21.15.32:0/4200621296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8003f80 0x7f67f8004800 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c160 tx=0x7f67f8005830 comp rx=0 tx=0).stop 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813400640 1 -- 172.21.15.32:0/4200621296 reap_dead start 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813e00640 1 -- 172.21.15.32:0/4200621296 shutdown_connections 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813e00640 1 -- 172.21.15.32:0/4200621296 >> 172.21.15.32:0/4200621296 conn(0x7f68140ccf70 msgr2=0x7f68140cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813e00640 1 -- 172.21.15.32:0/4200621296 shutdown_connections 2024-03-06T07:39:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.551+0000 7f6813e00640 1 -- 172.21.15.32:0/4200621296 wait complete. 2024-03-06T07:39:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.554+0000 7f6813e00640 1 Processor -- start 2024-03-06T07:39:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.554+0000 7f6813e00640 1 -- start start 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f8003f80 0x7f6814158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 0x7f68140b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f681415b310 0x7f681415b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140ceec0 con 0x7f6814145730 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140ced40 con 0x7f681415b310 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681406ef10 con 0x7f67f8003f80 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f680be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 0x7f68140b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f680be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 0x7f68140b3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41932/0 (socket says 172.21.15.32:41932) 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f680be00640 1 -- 172.21.15.32:0/964903515 learned_addr learned my addr 172.21.15.32:0/964903515 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.556+0000 7f6813400640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f681415b310 0x7f681415b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f6812a00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f8003f80 0x7f6814158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680be00640 1 -- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f8003f80 msgr2=0x7f6814158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680be00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f8003f80 0x7f6814158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680be00640 1 -- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f681415b310 msgr2=0x7f681415b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680be00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f681415b310 0x7f681415b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680be00640 1 -- 172.21.15.32:0/964903515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140b36a0 con 0x7f6814145730 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680be00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 0x7f68140b3160 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c280 tx=0x7f67fc001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc007c80 con 0x7f6814145730 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc004c10 con 0x7f6814145730 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68140b3940 con 0x7f6814145730 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.557+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc001bc0 con 0x7f6814145730 2024-03-06T07:39:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.558+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68140b3f00 con 0x7f6814145730 2024-03-06T07:39:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.559+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681404f330 con 0x7f6814145730 2024-03-06T07:39:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.560+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67fc048020 con 0x7f6814145730 2024-03-06T07:39:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.561+0000 7f680aa00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67f006b500 0x7f67f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.561+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f67fc0ab5e0 con 0x7f6814145730 2024-03-06T07:39:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.561+0000 7f6812a00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67f006b500 0x7f67f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.562+0000 7f6812a00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67f006b500 0x7f67f006d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6804009930 tx=0x7f6804008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:34.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.565+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67fc07a380 con 0x7f6814145730 2024-03-06T07:39:34.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.877+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68140cdde0 con 0x7f6814145730 2024-03-06T07:39:34.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.878+0000 7f680aa00640 1 -- 172.21.15.32:0/964903515 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f67fc079c80 con 0x7f6814145730 2024-03-06T07:39:34.879 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:34.880 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:34.880 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67f006b500 msgr2=0x7f67f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67f006b500 0x7f67f006d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6804009930 tx=0x7f6804008040 comp rx=0 tx=0).stop 2024-03-06T07:39:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 msgr2=0x7f68140b3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 0x7f68140b3160 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c280 tx=0x7f67fc001970 comp rx=0 tx=0).stop 2024-03-06T07:39:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 shutdown_connections 2024-03-06T07:39:34.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f67f006b500 0x7f67f006d9c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f681415b310 0x7f681415b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6814145730 0x7f68140b3160 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 --2- 172.21.15.32:0/964903515 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f67f8003f80 0x7f6814158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.880+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 >> 172.21.15.32:0/964903515 conn(0x7f68140ccf70 msgr2=0x7f681415f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.881+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 shutdown_connections 2024-03-06T07:39:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:34.881+0000 7f6813e00640 1 -- 172.21.15.32:0/964903515 wait complete. 2024-03-06T07:39:35.103 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:35.103 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:35.454 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:36.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:36.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.325+0000 7f1262e00640 1 Processor -- start 2024-03-06T07:39:36.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.325+0000 7f1262e00640 1 -- start start 2024-03-06T07:39:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.326+0000 7f1262e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f125c15e0b0 0x7f125c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.327+0000 7f1262e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125c0b3710 con 0x7f125c0d0e50 2024-03-06T07:39:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.327+0000 7f1262e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125c0b39f0 con 0x7f125c0d4a10 2024-03-06T07:39:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.327+0000 7f1262e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c0b09e0 con 0x7f125c15e0b0 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.327+0000 7f1262400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f125c0d0e50 0x7f125c0b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37142/0 (socket says 172.21.15.32:37142) 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.327+0000 7f1262400640 1 -- 172.21.15.32:0/1091419053 learned_addr learned my addr 172.21.15.32:0/1091419053 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.327+0000 7f1261a00640 1 --2- 172.21.15.32:0/1091419053 >> v2:172.21.15.39:3300/0 conn(0x7f125c15e0b0 0x7f125c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.328+0000 7f125be00640 1 -- 172.21.15.32:0/1091419053 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700147165 0 0) 0x7f125c0b3710 con 0x7f125c0d0e50 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.328+0000 7f125be00640 1 -- 172.21.15.32:0/1091419053 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1244003520 con 0x7f125c0d0e50 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.328+0000 7f1261a00640 1 -- 172.21.15.32:0/1091419053 >> v1:172.21.15.39:6789/0 conn(0x7f125c0d4a10 legacy=0x7f125c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.328+0000 7f1261a00640 1 -- 172.21.15.32:0/1091419053 >> v1:172.21.15.32:6789/0 conn(0x7f125c0d0e50 legacy=0x7f125c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.328+0000 7f1261a00640 1 -- 172.21.15.32:0/1091419053 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c0b3cd0 con 0x7f125c15e0b0 2024-03-06T07:39:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.329+0000 7f1261a00640 1 --2- 172.21.15.32:0/1091419053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c15e0b0 0x7f125c0cf310 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f12500097a0 tx=0x7f125002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b7a1718ab1d3889 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.330+0000 7f125be00640 1 -- 172.21.15.32:0/1091419053 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250034070 con 0x7f125c15e0b0 2024-03-06T07:39:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.330+0000 7f125be00640 1 -- 172.21.15.32:0/1091419053 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250030610 con 0x7f125c15e0b0 2024-03-06T07:39:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.330+0000 7f125be00640 1 -- 172.21.15.32:0/1091419053 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500309c0 con 0x7f125c15e0b0 2024-03-06T07:39:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 -- 172.21.15.32:0/1091419053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c15e0b0 msgr2=0x7f125c0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 --2- 172.21.15.32:0/1091419053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c15e0b0 0x7f125c0cf310 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f12500097a0 tx=0x7f125002b960 comp rx=0 tx=0).stop 2024-03-06T07:39:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 -- 172.21.15.32:0/1091419053 shutdown_connections 2024-03-06T07:39:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 --2- 172.21.15.32:0/1091419053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c15e0b0 0x7f125c0cf310 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 -- 172.21.15.32:0/1091419053 >> 172.21.15.32:0/1091419053 conn(0x7f125c0d3d30 msgr2=0x7f125c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 -- 172.21.15.32:0/1091419053 shutdown_connections 2024-03-06T07:39:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.331+0000 7f1262e00640 1 -- 172.21.15.32:0/1091419053 wait complete. 2024-03-06T07:39:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.333+0000 7f1262e00640 1 Processor -- start 2024-03-06T07:39:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.333+0000 7f1262e00640 1 -- start start 2024-03-06T07:39:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1262e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c0d0e50 0x7f125c0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1262e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 0x7f125c15f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1262e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f125c15e0b0 0x7f125c160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1262e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0cfc60 con 0x7f125c0d4a10 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1262e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c15e490 con 0x7f125c0d0e50 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1262e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0cf850 con 0x7f125c15e0b0 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 0x7f125c15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 0x7f125c15f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41934/0 (socket says 172.21.15.32:41934) 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 -- 172.21.15.32:0/792408277 learned_addr learned my addr 172.21.15.32:0/792408277 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 -- 172.21.15.32:0/792408277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f125c15e0b0 msgr2=0x7f125c160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f125c15e0b0 0x7f125c160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 -- 172.21.15.32:0/792408277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c0d0e50 msgr2=0x7f125c0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c0d0e50 0x7f125c0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.334+0000 7f1261000640 1 -- 172.21.15.32:0/792408277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c06c380 con 0x7f125c0d4a10 2024-03-06T07:39:36.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.335+0000 7f1261000640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 0x7f125c15f020 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f124800c430 tx=0x7f124800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.335+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248017070 con 0x7f125c0d4a10 2024-03-06T07:39:36.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.335+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12480052d0 con 0x7f125c0d4a10 2024-03-06T07:39:36.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.335+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248003830 con 0x7f125c0d4a10 2024-03-06T07:39:36.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.335+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c15eb10 con 0x7f125c0d4a10 2024-03-06T07:39:36.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.335+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125c06c8e0 con 0x7f125c0d4a10 2024-03-06T07:39:36.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.336+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125c04f330 con 0x7f125c0d4a10 2024-03-06T07:39:36.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.337+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f124801a070 con 0x7f125c0d4a10 2024-03-06T07:39:36.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.338+0000 7f1259600640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f123c06b4d0 0x7f123c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.338+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1248089620 con 0x7f125c0d4a10 2024-03-06T07:39:36.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.339+0000 7f1261a00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f123c06b4d0 0x7f123c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.339+0000 7f1261a00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f123c06b4d0 0x7f123c06d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f1250030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.343+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f125c06c380 con 0x7f125c0d4a10 2024-03-06T07:39:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.666+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f125c0b2c90 con 0x7f125c0d4a10 2024-03-06T07:39:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.667+0000 7f1259600640 1 -- 172.21.15.32:0/792408277 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f1248058470 con 0x7f125c0d4a10 2024-03-06T07:39:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:36.736 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.669+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f123c06b4d0 msgr2=0x7f123c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.669+0000 7f1262e00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f123c06b4d0 0x7f123c06d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f1250030000 comp rx=0 tx=0).stop 2024-03-06T07:39:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 msgr2=0x7f125c15f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 0x7f125c15f020 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f124800c430 tx=0x7f124800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 shutdown_connections 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f123c06b4d0 0x7f123c06d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f125c15e0b0 0x7f125c160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f125c0d4a10 0x7f125c15f020 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 --2- 172.21.15.32:0/792408277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f125c0d0e50 0x7f125c0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 >> 172.21.15.32:0/792408277 conn(0x7f125c0d3d30 msgr2=0x7f125c0b5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 shutdown_connections 2024-03-06T07:39:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:36.670+0000 7f1262e00640 1 -- 172.21.15.32:0/792408277 wait complete. 2024-03-06T07:39:36.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:36.941 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:37.456 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:37.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.169+0000 7f9bd1400640 1 Processor -- start 2024-03-06T07:39:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.170+0000 7f9bd1400640 1 -- start start 2024-03-06T07:39:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bd1400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9bcc0d2b40 0x7f9bcc0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d1480 0x7f9bcc0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bd1400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0b3940 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bd1400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0d4e70 con 0x7f9bcc0d2b40 2024-03-06T07:39:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bd1400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0b3c40 con 0x7f9bcc0d1480 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bc1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9bcc0d2b40 0x7f9bcc0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bc1e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9bcc0d2b40 0x7f9bcc0d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37610/0 (socket says 172.21.15.32:37610) 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.171+0000 7f9bc1e00640 1 -- 172.21.15.32:0/1152373195 learned_addr learned my addr 172.21.15.32:0/1152373195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.172+0000 7f9bcb200640 1 --2- 172.21.15.32:0/1152373195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d1480 0x7f9bcc0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.172+0000 7f9bc1e00640 1 -- 172.21.15.32:0/1152373195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d1480 msgr2=0x7f9bcc0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.172+0000 7f9bc1e00640 1 --2- 172.21.15.32:0/1152373195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d1480 0x7f9bcc0b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.172+0000 7f9bc1e00640 1 -- 172.21.15.32:0/1152373195 >> v1:172.21.15.39:6789/0 conn(0x7f9bcc15e2e0 legacy=0x7f9bcc0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.172+0000 7f9bc1e00640 1 -- 172.21.15.32:0/1152373195 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0cf580 con 0x7f9bcc0d2b40 2024-03-06T07:39:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.173+0000 7f9bc1e00640 1 --2- 172.21.15.32:0/1152373195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc0d2b40 0x7f9bcc0d0f40 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009870 tx=0x7f9bb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3905b000650417 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bc9e00640 1 -- 172.21.15.32:0/1152373195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4035020 con 0x7f9bcc0d2b40 2024-03-06T07:39:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bc9e00640 1 -- 172.21.15.32:0/1152373195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb4031640 con 0x7f9bcc0d2b40 2024-03-06T07:39:38.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bc9e00640 1 -- 172.21.15.32:0/1152373195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb402d040 con 0x7f9bcc0d2b40 2024-03-06T07:39:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bd1400640 1 -- 172.21.15.32:0/1152373195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc0d2b40 msgr2=0x7f9bcc0d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bd1400640 1 --2- 172.21.15.32:0/1152373195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc0d2b40 0x7f9bcc0d0f40 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009870 tx=0x7f9bb402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bd1400640 1 -- 172.21.15.32:0/1152373195 shutdown_connections 2024-03-06T07:39:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bd1400640 1 --2- 172.21.15.32:0/1152373195 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d1480 0x7f9bcc0b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bd1400640 1 --2- 172.21.15.32:0/1152373195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc0d2b40 0x7f9bcc0d0f40 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.174+0000 7f9bd1400640 1 -- 172.21.15.32:0/1152373195 >> 172.21.15.32:0/1152373195 conn(0x7f9bcc0d3f60 msgr2=0x7f9bcc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.175+0000 7f9bd1400640 1 -- 172.21.15.32:0/1152373195 shutdown_connections 2024-03-06T07:39:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.175+0000 7f9bd1400640 1 -- 172.21.15.32:0/1152373195 wait complete. 2024-03-06T07:39:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.176+0000 7f9bd1400640 1 Processor -- start 2024-03-06T07:39:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.177+0000 7f9bd1400640 1 -- start start 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d2b40 0x7f9bcc152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 0x7f9bcc0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bcc153610 0x7f9bcc0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bd1400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0cfe20 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bd1400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0cfb40 con 0x7f9bcc153610 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bd1400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0b4610 con 0x7f9bcc0d2b40 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 0x7f9bcc0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 0x7f9bcc0c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37618/0 (socket says 172.21.15.32:37618) 2024-03-06T07:39:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 -- 172.21.15.32:0/2212077575 learned_addr learned my addr 172.21.15.32:0/2212077575 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bca800640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d2b40 0x7f9bcc152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bcb200640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bcc153610 0x7f9bcc0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 -- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d2b40 msgr2=0x7f9bcc152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d2b40 0x7f9bcc152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 -- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bcc153610 msgr2=0x7f9bcc0c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bcc153610 0x7f9bcc0c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.178+0000 7f9bc1e00640 1 -- 172.21.15.32:0/2212077575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0b9ab0 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.179+0000 7f9bc1e00640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 0x7f9bcc0c9640 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4000c00 tx=0x7f9bb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.179+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4040070 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.179+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb40399f0 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.179+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc0b9210 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.179+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4035040 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.179+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc0b97d0 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.181+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc04f3a0 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.183+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb4038020 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.183+0000 7f9bc3400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba806b500 0x7f9ba806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.184+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9bb40abc80 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.184+0000 7f9bca800640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba806b500 0x7f9ba806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.184+0000 7f9bca800640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba806b500 0x7f9ba806d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009f20 tx=0x7f9bbc0094d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.188+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb407aa20 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.424+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bcc0d01e0 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.425+0000 7f9bc3400640 1 -- 172.21.15.32:0/2212077575 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f9bb407a320 con 0x7f9bcc15e2e0 2024-03-06T07:39:38.428 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:38.428 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:38.428 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba806b500 msgr2=0x7f9ba806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba806b500 0x7f9ba806d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009f20 tx=0x7f9bbc0094d0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 msgr2=0x7f9bcc0c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 0x7f9bcc0c9640 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4000c00 tx=0x7f9bb40046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 shutdown_connections 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ba806b500 0x7f9ba806d9c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bcc153610 0x7f9bcc0c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bcc15e2e0 0x7f9bcc0c9640 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 --2- 172.21.15.32:0/2212077575 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9bcc0d2b40 0x7f9bcc152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 >> 172.21.15.32:0/2212077575 conn(0x7f9bcc0d3f60 msgr2=0x7f9bcc0b60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 shutdown_connections 2024-03-06T07:39:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:38.428+0000 7f9bd1400640 1 -- 172.21.15.32:0/2212077575 wait complete. 2024-03-06T07:39:38.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:38.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:39.458 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:39.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:39.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.881+0000 7f3c21600640 1 Processor -- start 2024-03-06T07:39:39.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.882+0000 7f3c21600640 1 -- start start 2024-03-06T07:39:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.883+0000 7f3c21600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c1c0af1a0 0x7f3c1c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.883+0000 7f3c21600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0af580 0x7f3c1c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.883+0000 7f3c21600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cdc20 0x7f3c1c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c21600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c15cc40 con 0x7f3c1c0af1a0 2024-03-06T07:39:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c21600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c15cf20 con 0x7f3c1c0cdc20 2024-03-06T07:39:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c21600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c15d200 con 0x7f3c1c0af580 2024-03-06T07:39:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c1aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c1c0af1a0 0x7f3c1c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c1aa00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c1c0af1a0 0x7f3c1c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37630/0 (socket says 172.21.15.32:37630) 2024-03-06T07:39:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c1aa00640 1 -- 172.21.15.32:0/25370805 learned_addr learned my addr 172.21.15.32:0/25370805 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c13e00640 1 --2- 172.21.15.32:0/25370805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0af580 0x7f3c1c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.884+0000 7f3c1b400640 1 --2- 172.21.15.32:0/25370805 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cdc20 0x7f3c1c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.885+0000 7f3c1aa00640 1 -- 172.21.15.32:0/25370805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0af580 msgr2=0x7f3c1c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.885+0000 7f3c1aa00640 1 --2- 172.21.15.32:0/25370805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0af580 0x7f3c1c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.885+0000 7f3c1aa00640 1 -- 172.21.15.32:0/25370805 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cdc20 msgr2=0x7f3c1c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.885+0000 7f3c1aa00640 1 --2- 172.21.15.32:0/25370805 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cdc20 0x7f3c1c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.885+0000 7f3c1aa00640 1 -- 172.21.15.32:0/25370805 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c15d4e0 con 0x7f3c1c0af1a0 2024-03-06T07:39:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.886+0000 7f3c1aa00640 1 --2- 172.21.15.32:0/25370805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c15c600 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0cda30 tx=0x7f3c0c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7e80be8e7613ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.887+0000 7f3c1a000640 1 -- 172.21.15.32:0/25370805 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c02e070 con 0x7f3c1c0af1a0 2024-03-06T07:39:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.887+0000 7f3c1a000640 1 -- 172.21.15.32:0/25370805 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c037070 con 0x7f3c1c0af1a0 2024-03-06T07:39:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.887+0000 7f3c1a000640 1 -- 172.21.15.32:0/25370805 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c031a00 con 0x7f3c1c0af1a0 2024-03-06T07:39:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 -- 172.21.15.32:0/25370805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af1a0 msgr2=0x7f3c1c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 --2- 172.21.15.32:0/25370805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c15c600 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0cda30 tx=0x7f3c0c02b880 comp rx=0 tx=0).stop 2024-03-06T07:39:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 -- 172.21.15.32:0/25370805 shutdown_connections 2024-03-06T07:39:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 --2- 172.21.15.32:0/25370805 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cdc20 0x7f3c1c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 --2- 172.21.15.32:0/25370805 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0af580 0x7f3c1c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 --2- 172.21.15.32:0/25370805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c15c600 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 -- 172.21.15.32:0/25370805 >> 172.21.15.32:0/25370805 conn(0x7f3c1c06afb0 msgr2=0x7f3c1c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 -- 172.21.15.32:0/25370805 shutdown_connections 2024-03-06T07:39:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.888+0000 7f3c21600640 1 -- 172.21.15.32:0/25370805 wait complete. 2024-03-06T07:39:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.890+0000 7f3c21600640 1 Processor -- start 2024-03-06T07:39:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.891+0000 7f3c21600640 1 -- start start 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c21600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c1477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c21600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 0x7f3c1c1f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c21600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0cdc20 0x7f3c1c157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c21600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b3490 con 0x7f3c1c0af580 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c21600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b3310 con 0x7f3c1c0af1a0 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c21600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b3610 con 0x7f3c1c0cdc20 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 0x7f3c1c1f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.892+0000 7f3c13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 0x7f3c1c1f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37644/0 (socket says 172.21.15.32:37644) 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 -- 172.21.15.32:0/3190998487 learned_addr learned my addr 172.21.15.32:0/3190998487 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c1aa00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c1477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c1b400640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0cdc20 0x7f3c1c157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 -- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0cdc20 msgr2=0x7f3c1c157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0cdc20 0x7f3c1c157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 -- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0af1a0 msgr2=0x7f3c1c1477e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c1477e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 -- 172.21.15.32:0/3190998487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c156990 con 0x7f3c1c0af580 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.893+0000 7f3c13e00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 0x7f3c1c1f2640 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400d830 tx=0x7f3c0400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.894+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04018020 con 0x7f3c1c0af580 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.894+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c040096a0 con 0x7f3c1c0af580 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.894+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c156c70 con 0x7f3c1c0af580 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.894+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04009a70 con 0x7f3c1c0af580 2024-03-06T07:39:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.894+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c157230 con 0x7f3c1c0af580 2024-03-06T07:39:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.895+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c069510 con 0x7f3c1c0af580 2024-03-06T07:39:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.897+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c0401b020 con 0x7f3c1c0af580 2024-03-06T07:39:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.898+0000 7f3c12a00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bf806b4d0 0x7f3bf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.898+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3c0408a6a0 con 0x7f3c1c0af580 2024-03-06T07:39:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.898+0000 7f3c1aa00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bf806b4d0 0x7f3bf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.899+0000 7f3c1aa00640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bf806b4d0 0x7f3bf806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c036000 tx=0x7f3c0c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:39.903+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c04022030 con 0x7f3c1c0af580 2024-03-06T07:39:40.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.143+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c1c0719a0 con 0x7f3c1c0af580 2024-03-06T07:39:40.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.144+0000 7f3c12a00640 1 -- 172.21.15.32:0/3190998487 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f3c04009e20 con 0x7f3c1c0af580 2024-03-06T07:39:40.147 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:40.147 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:40.147 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bf806b4d0 msgr2=0x7f3bf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bf806b4d0 0x7f3bf806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c036000 tx=0x7f3c0c03d040 comp rx=0 tx=0).stop 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 msgr2=0x7f3c1c1f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 0x7f3c1c1f2640 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400d830 tx=0x7f3c0400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 shutdown_connections 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3bf806b4d0 0x7f3bf806d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3c1c0cdc20 0x7f3c1c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0af580 0x7f3c1c1f2640 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 --2- 172.21.15.32:0/3190998487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0af1a0 0x7f3c1c1477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 >> 172.21.15.32:0/3190998487 conn(0x7f3c1c06afb0 msgr2=0x7f3c1c0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 shutdown_connections 2024-03-06T07:39:40.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:40.147+0000 7f3c21600640 1 -- 172.21.15.32:0/3190998487 wait complete. 2024-03-06T07:39:40.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:40.375 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:41.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:41.460 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:41.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.630+0000 7f3ace400640 1 Processor -- start 2024-03-06T07:39:41.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.631+0000 7f3ace400640 1 -- start start 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.633+0000 7f3ace400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ac815e070 0x7f3ac80d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.633+0000 7f3ace400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80cf410 0x7f3ac80cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3ace400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80d18c0 con 0x7f3ac80d05c0 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3ace400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac80d1ba0 con 0x7f3ac815e070 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3ace400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80d1e80 con 0x7f3ac80cf410 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ac815e070 0x7f3ac80d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ac815e070 0x7f3ac80d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37654/0 (socket says 172.21.15.32:37654) 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 -- 172.21.15.32:0/3860150103 learned_addr learned my addr 172.21.15.32:0/3860150103 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3ac7e00640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80cf410 0x7f3ac80cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 -- 172.21.15.32:0/3860150103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80cf410 msgr2=0x7f3ac80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80cf410 0x7f3ac80cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 -- 172.21.15.32:0/3860150103 >> v1:172.21.15.39:6789/0 conn(0x7f3ac80d05c0 legacy=0x7f3ac80b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:41.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3acd000640 1 -- 172.21.15.32:0/3860150103 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac80d2160 con 0x7f3ac815e070 2024-03-06T07:39:41.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.634+0000 7f3ac7e00640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80cf410 0x7f3ac80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.635+0000 7f3acd000640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac80d2960 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009870 tx=0x7f3abc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe135def5ac1868 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.636+0000 7f3ac7400640 1 -- 172.21.15.32:0/3860150103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc035020 con 0x7f3ac815e070 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.636+0000 7f3ac7400640 1 -- 172.21.15.32:0/3860150103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc031640 con 0x7f3ac815e070 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.636+0000 7f3ac7400640 1 -- 172.21.15.32:0/3860150103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc02d040 con 0x7f3ac815e070 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.636+0000 7f3ace400640 1 -- 172.21.15.32:0/3860150103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 msgr2=0x7f3ac80d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.636+0000 7f3ace400640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac80d2960 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009870 tx=0x7f3abc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.637+0000 7f3ace400640 1 -- 172.21.15.32:0/3860150103 shutdown_connections 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.637+0000 7f3ace400640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80cf410 0x7f3ac80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.637+0000 7f3ace400640 1 --2- 172.21.15.32:0/3860150103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac80d2960 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.637+0000 7f3ace400640 1 -- 172.21.15.32:0/3860150103 >> 172.21.15.32:0/3860150103 conn(0x7f3ac808d2a0 msgr2=0x7f3ac80d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.637+0000 7f3ace400640 1 -- 172.21.15.32:0/3860150103 shutdown_connections 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.637+0000 7f3ace400640 1 -- 172.21.15.32:0/3860150103 wait complete. 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.638+0000 7f3ace400640 1 Processor -- start 2024-03-06T07:39:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.638+0000 7f3ace400640 1 -- start start 2024-03-06T07:39:41.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.639+0000 7f3ace400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac80cf410 0x7f3ac80b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.639+0000 7f3ace400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80d05c0 0x7f3ac8160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.639+0000 7f3ace400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac8154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.639+0000 7f3ace400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80d4730 con 0x7f3ac815e070 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.639+0000 7f3ace400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80d45b0 con 0x7f3ac80cf410 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.639+0000 7f3ace400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80d0a50 con 0x7f3ac80d05c0 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac8154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac8154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37668/0 (socket says 172.21.15.32:37668) 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 -- 172.21.15.32:0/2595922364 learned_addr learned my addr 172.21.15.32:0/2595922364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3ac7e00640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80d05c0 0x7f3ac8160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 -- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80d05c0 msgr2=0x7f3ac8160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80d05c0 0x7f3ac8160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 -- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac80cf410 msgr2=0x7f3ac80b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:41.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acd000640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac80cf410 0x7f3ac80b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac80cf410 0x7f3ac80b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acda00640 1 -- 172.21.15.32:0/2595922364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac81f84c0 con 0x7f3ac815e070 2024-03-06T07:39:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3ac7e00640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80d05c0 0x7f3ac8160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.640+0000 7f3acd000640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac80cf410 0x7f3ac80b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:41.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.641+0000 7f3acda00640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac8154d70 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c7b0 tx=0x7f3ac000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.641+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0004b60 con 0x7f3ac815e070 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.641+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac0004d00 con 0x7f3ac815e070 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.641+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000aa30 con 0x7f3ac815e070 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.641+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac80b7fc0 con 0x7f3ac815e070 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.641+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac80b8580 con 0x7f3ac815e070 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.642+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac804f330 con 0x7f3ac815e070 2024-03-06T07:39:41.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.643+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ac000abd0 con 0x7f3ac815e070 2024-03-06T07:39:41.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.644+0000 7f3ac4c00640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3aa806b4d0 0x7f3aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.644+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3ac008a4a0 con 0x7f3ac815e070 2024-03-06T07:39:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.645+0000 7f3acd000640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3aa806b4d0 0x7f3aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.645+0000 7f3acd000640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3aa806b4d0 0x7f3aa806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3abc036000 tx=0x7f3abc008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:41.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:41.654+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ac0020070 con 0x7f3ac815e070 2024-03-06T07:39:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.014+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ac806c2c0 con 0x7f3ac815e070 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.015+0000 7f3ac4c00640 1 -- 172.21.15.32:0/2595922364 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f3ac00592e0 con 0x7f3ac815e070 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.018+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3aa806b4d0 msgr2=0x7f3aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.018+0000 7f3ace400640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3aa806b4d0 0x7f3aa806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3abc036000 tx=0x7f3abc008030 comp rx=0 tx=0).stop 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.018+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 msgr2=0x7f3ac8154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.018+0000 7f3ace400640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac8154d70 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c7b0 tx=0x7f3ac000cc70 comp rx=0 tx=0).stop 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 shutdown_connections 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3aa806b4d0 0x7f3aa806d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac815e070 0x7f3ac8154d70 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3ac80d05c0 0x7f3ac8160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 --2- 172.21.15.32:0/2595922364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac80cf410 0x7f3ac80b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 >> 172.21.15.32:0/2595922364 conn(0x7f3ac808d2a0 msgr2=0x7f3ac806d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 shutdown_connections 2024-03-06T07:39:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:42.019+0000 7f3ace400640 1 -- 172.21.15.32:0/2595922364 wait complete. 2024-03-06T07:39:42.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:42.243 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:43.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.469+0000 7fa01ee00640 1 Processor -- start 2024-03-06T07:39:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.470+0000 7fa01ee00640 1 -- start start 2024-03-06T07:39:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa01ee00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa0180b3c90 0x7fa0180cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa01ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 0x7fa018071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa01ee00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0180b4860 con 0x7fa0180cf5c0 2024-03-06T07:39:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa01ee00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0180ce160 con 0x7fa0180b3c90 2024-03-06T07:39:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa01ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180b4b40 con 0x7fa018071050 2024-03-06T07:39:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa00fe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0180cf5c0 0x7fa0180b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38670/0 (socket says 172.21.15.32:38670) 2024-03-06T07:39:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.472+0000 7fa00fe00640 1 -- 172.21.15.32:0/777199704 learned_addr learned my addr 172.21.15.32:0/777199704 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017e00640 1 --2- 172.21.15.32:0/777199704 >> v2:172.21.15.39:3300/0 conn(0x7fa0180b3c90 0x7fa0180cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa01ce00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 0x7fa018071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249666917 0 0) 0x7fa0180b4860 con 0x7fa0180cf5c0 2024-03-06T07:39:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ffc003520 con 0x7fa0180cf5c0 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 537271576 0 0) 0x7f9ffc003520 con 0x7fa0180cf5c0 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 msgr2=0x7fa018071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 0x7fa018071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 >> v2:172.21.15.39:3300/0 conn(0x7fa0180b3c90 msgr2=0x7fa0180cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 --2- 172.21.15.32:0/777199704 >> v2:172.21.15.39:3300/0 conn(0x7fa0180b3c90 0x7fa0180cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0180b4e20 con 0x7fa0180cf5c0 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fa000001670 con 0x7fa0180cf5c0 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 >> v1:172.21.15.32:6789/0 conn(0x7fa0180cf5c0 legacy=0x7fa0180b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 0x7f9ffc003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ffc003fa0 0x7f9ffc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc004d40 0x7f9ffc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180ce160 con 0x7f9ffc004d40 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180b4b40 con 0x7f9ffc003fa0 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.473+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc00bb30 con 0x7fa018071050 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa01ce00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc004d40 0x7f9ffc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa017e00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 0x7f9ffc003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa00fe00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ffc003fa0 0x7f9ffc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa01ce00640 1 -- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 msgr2=0x7f9ffc003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa01ce00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa018071050 0x7f9ffc003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa01ce00640 1 -- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ffc003fa0 msgr2=0x7f9ffc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa01ce00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ffc003fa0 0x7f9ffc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa01ce00640 1 -- 172.21.15.32:0/777199704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc003520 con 0x7f9ffc004d40 2024-03-06T07:39:43.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.474+0000 7fa00fe00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ffc003fa0 0x7f9ffc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.476+0000 7fa01ce00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc004d40 0x7f9ffc0051c0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fa010007360 tx=0x7f9ffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de37fb2ae1bf377 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.476+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010039070 con 0x7f9ffc004d40 2024-03-06T07:39:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.476+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01002ccd0 con 0x7f9ffc004d40 2024-03-06T07:39:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.476+0000 7fa017400640 1 -- 172.21.15.32:0/777199704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010009be0 con 0x7f9ffc004d40 2024-03-06T07:39:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ee00640 1 -- 172.21.15.32:0/777199704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc004d40 msgr2=0x7f9ffc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ee00640 1 --2- 172.21.15.32:0/777199704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ffc004d40 0x7f9ffc0051c0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fa010007360 tx=0x7f9ffc005830 comp rx=0 tx=0).stop 2024-03-06T07:39:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ce00640 1 -- 172.21.15.32:0/777199704 reap_dead start 2024-03-06T07:39:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ee00640 1 -- 172.21.15.32:0/777199704 shutdown_connections 2024-03-06T07:39:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ee00640 1 -- 172.21.15.32:0/777199704 >> 172.21.15.32:0/777199704 conn(0x7fa01806afb0 msgr2=0x7fa01806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ee00640 1 -- 172.21.15.32:0/777199704 shutdown_connections 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.477+0000 7fa01ee00640 1 -- 172.21.15.32:0/777199704 wait complete. 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.479+0000 7fa01ee00640 1 Processor -- start 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.479+0000 7fa01ee00640 1 -- start start 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ee00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ffc004d40 0x7fa018143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa018155b10 0x7fa01809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 0x7fa0181466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018071490 con 0x7fa01809d170 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01815b340 con 0x7fa018155b10 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ee00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa018071610 con 0x7f9ffc004d40 2024-03-06T07:39:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 0x7fa0181466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 0x7fa0181466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37690/0 (socket says 172.21.15.32:37690) 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.480+0000 7fa01ce00640 1 -- 172.21.15.32:0/4236508169 learned_addr learned my addr 172.21.15.32:0/4236508169 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa00fe00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa018155b10 0x7fa01809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa017e00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ffc004d40 0x7fa018143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ce00640 1 -- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ffc004d40 msgr2=0x7fa018143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ce00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ffc004d40 0x7fa018143d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ce00640 1 -- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa018155b10 msgr2=0x7fa01809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ce00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa018155b10 0x7fa01809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ce00640 1 -- 172.21.15.32:0/4236508169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa018144a00 con 0x7fa01809d170 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa017e00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ffc004d40 0x7fa018143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ce00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 0x7fa0181466f0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa010002230 tx=0x7fa010004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010039030 con 0x7fa01809d170 2024-03-06T07:39:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa010043070 con 0x7fa01809d170 2024-03-06T07:39:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010009be0 con 0x7fa01809d170 2024-03-06T07:39:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.481+0000 7fa01ee00640 1 -- 172.21.15.32:0/4236508169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa018144ce0 con 0x7fa01809d170 2024-03-06T07:39:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.482+0000 7fa01ee00640 1 -- 172.21.15.32:0/4236508169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0181452a0 con 0x7fa01809d170 2024-03-06T07:39:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.483+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa018068720 con 0x7fa01809d170 2024-03-06T07:39:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.485+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa01003d7f0 con 0x7fa01809d170 2024-03-06T07:39:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.486+0000 7fa014c00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.486+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0100ac5e0 con 0x7fa01809d170 2024-03-06T07:39:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.486+0000 7fa017e00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.487+0000 7fa017e00640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ff406b4d0 0x7f9ff406d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa008009a80 tx=0x7fa008008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.492+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa018068720 con 0x7fa01809d170 2024-03-06T07:39:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.735+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0180b16a0 con 0x7fa01809d170 2024-03-06T07:39:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.736+0000 7fa014c00640 1 -- 172.21.15.32:0/4236508169 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fa01007b2f0 con 0x7fa01809d170 2024-03-06T07:39:43.739 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:43.739 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:43.739 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:43.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ff406b4d0 msgr2=0x7f9ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ff406b4d0 0x7f9ff406d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa008009a80 tx=0x7fa008008040 comp rx=0 tx=0).stop 2024-03-06T07:39:43.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 msgr2=0x7fa0181466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 0x7fa0181466f0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa010002230 tx=0x7fa010004ab0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 shutdown_connections 2024-03-06T07:39:43.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ff406b4d0 0x7f9ff406d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01809d170 0x7fa0181466f0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa018155b10 0x7fa01809cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 --2- 172.21.15.32:0/4236508169 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ffc004d40 0x7fa018143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 >> 172.21.15.32:0/4236508169 conn(0x7fa01806afb0 msgr2=0x7fa0181592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 shutdown_connections 2024-03-06T07:39:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:43.739+0000 7fa00d600640 1 -- 172.21.15.32:0/4236508169 wait complete. 2024-03-06T07:39:43.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:43.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:44.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:45.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.242+0000 7f79b6000640 1 Processor -- start 2024-03-06T07:39:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.242+0000 7f79b6000640 1 -- start start 2024-03-06T07:39:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b6000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f79b0070f10 0x7f79b00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b6000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79b00cce90 0x7f79b015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b6000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00714d0 con 0x7f79b0145300 2024-03-06T07:39:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b6000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00cd300 con 0x7f79b00cce90 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b6000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00717b0 con 0x7f79b0070f10 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b4c00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f79b0145300 0x7f79b00b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38686/0 (socket says 172.21.15.32:38686) 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b4c00640 1 -- 172.21.15.32:0/22634704 learned_addr learned my addr 172.21.15.32:0/22634704 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.244+0000 7f79b5600640 1 --2- 172.21.15.32:0/22634704 >> v2:172.21.15.32:3300/0 conn(0x7f79b00cce90 0x7f79b015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79afe00640 1 --2- 172.21.15.32:0/22634704 >> v2:172.21.15.39:3300/0 conn(0x7f79b0070f10 0x7f79b00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79af400640 1 -- 172.21.15.32:0/22634704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495449566 0 0) 0x7f79b00714d0 con 0x7f79b0145300 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79af400640 1 -- 172.21.15.32:0/22634704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7998003520 con 0x7f79b0145300 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79b5600640 1 -- 172.21.15.32:0/22634704 >> v2:172.21.15.39:3300/0 conn(0x7f79b0070f10 msgr2=0x7f79b00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79b5600640 1 --2- 172.21.15.32:0/22634704 >> v2:172.21.15.39:3300/0 conn(0x7f79b0070f10 0x7f79b00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79b5600640 1 -- 172.21.15.32:0/22634704 >> v1:172.21.15.32:6789/0 conn(0x7f79b0145300 legacy=0x7f79b00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79b5600640 1 -- 172.21.15.32:0/22634704 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0071a90 con 0x7f79b00cce90 2024-03-06T07:39:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.245+0000 7f79afe00640 1 --2- 172.21.15.32:0/22634704 >> v2:172.21.15.39:3300/0 conn(0x7f79b0070f10 0x7f79b00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.246+0000 7f79b5600640 1 --2- 172.21.15.32:0/22634704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b00cce90 0x7f79b015c570 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f79b00b5150 tx=0x7f79a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66de798e14847ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.246+0000 7f79af400640 1 -- 172.21.15.32:0/22634704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8039070 con 0x7f79b00cce90 2024-03-06T07:39:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.246+0000 7f79af400640 1 -- 172.21.15.32:0/22634704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a802cbf0 con 0x7f79b00cce90 2024-03-06T07:39:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.246+0000 7f79af400640 1 -- 172.21.15.32:0/22634704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8034440 con 0x7f79b00cce90 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 -- 172.21.15.32:0/22634704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b00cce90 msgr2=0x7f79b015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 --2- 172.21.15.32:0/22634704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b00cce90 0x7f79b015c570 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f79b00b5150 tx=0x7f79a802be40 comp rx=0 tx=0).stop 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 -- 172.21.15.32:0/22634704 shutdown_connections 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 --2- 172.21.15.32:0/22634704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b00cce90 0x7f79b015c570 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 --2- 172.21.15.32:0/22634704 >> v2:172.21.15.39:3300/0 conn(0x7f79b0070f10 0x7f79b00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 -- 172.21.15.32:0/22634704 >> 172.21.15.32:0/22634704 conn(0x7f79b006b3d0 msgr2=0x7f79b006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 -- 172.21.15.32:0/22634704 shutdown_connections 2024-03-06T07:39:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.247+0000 7f79b6000640 1 -- 172.21.15.32:0/22634704 wait complete. 2024-03-06T07:39:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.249+0000 7f79b6000640 1 Processor -- start 2024-03-06T07:39:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.249+0000 7f79b6000640 1 -- start start 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b6000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79b0070f10 0x7f79b01e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79b00cce90 0x7f79b0140d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 0x7f79b01e5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b015cab0 con 0x7f79b0145300 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b01456e0 con 0x7f79b00cce90 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b6000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b015cc30 con 0x7f79b0070f10 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 0x7f79b01e5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 0x7f79b01e5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37702/0 (socket says 172.21.15.32:37702) 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.250+0000 7f79b5600640 1 -- 172.21.15.32:0/2213374028 learned_addr learned my addr 172.21.15.32:0/2213374028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b4c00640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79b0070f10 0x7f79b01e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79afe00640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79b00cce90 0x7f79b0140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b5600640 1 -- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79b0070f10 msgr2=0x7f79b01e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b5600640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79b0070f10 0x7f79b01e6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b5600640 1 -- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79b00cce90 msgr2=0x7f79b0140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b5600640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79b00cce90 0x7f79b0140d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b5600640 1 -- 172.21.15.32:0/2213374028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b01e5ba0 con 0x7f79b0145300 2024-03-06T07:39:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.251+0000 7f79b5600640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 0x7f79b01e5470 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009e80 tx=0x7f79a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:45.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.252+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8039050 con 0x7f79b0145300 2024-03-06T07:39:45.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.252+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8045070 con 0x7f79b0145300 2024-03-06T07:39:45.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.252+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a803e830 con 0x7f79b0145300 2024-03-06T07:39:45.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.252+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b01e5e80 con 0x7f79b0145300 2024-03-06T07:39:45.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.252+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b01e6440 con 0x7f79b0145300 2024-03-06T07:39:45.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.253+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b004f330 con 0x7f79b0145300 2024-03-06T07:39:45.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.254+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79a803e9d0 con 0x7f79b0145300 2024-03-06T07:39:45.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.255+0000 7f79acc00640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f799006b4d0 0x7f799006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:45.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.255+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f79a80ad330 con 0x7f79b0145300 2024-03-06T07:39:45.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.255+0000 7f79b4c00640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f799006b4d0 0x7f799006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:45.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.256+0000 7f79b4c00640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f799006b4d0 0x7f799006d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f79a4005880 tx=0x7f79a4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:45.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.259+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79a807c280 con 0x7f79b0145300 2024-03-06T07:39:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:45.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.500+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b015c890 con 0x7f79b0145300 2024-03-06T07:39:45.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.502+0000 7f79acc00640 1 -- 172.21.15.32:0/2213374028 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f79a807bb80 con 0x7f79b0145300 2024-03-06T07:39:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:45.505 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:45.505 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f799006b4d0 msgr2=0x7f799006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f799006b4d0 0x7f799006d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f79a4005880 tx=0x7f79a4009040 comp rx=0 tx=0).stop 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 msgr2=0x7f79b01e5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 0x7f79b01e5470 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009e80 tx=0x7f79a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 shutdown_connections 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f799006b4d0 0x7f799006d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b0145300 0x7f79b01e5470 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79b00cce90 0x7f79b0140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 --2- 172.21.15.32:0/2213374028 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f79b0070f10 0x7f79b01e6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:45.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 >> 172.21.15.32:0/2213374028 conn(0x7f79b006b3d0 msgr2=0x7f79b00cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:45.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 shutdown_connections 2024-03-06T07:39:45.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:45.506+0000 7f79b6000640 1 -- 172.21.15.32:0/2213374028 wait complete. 2024-03-06T07:39:45.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:45.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:46.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.955+0000 7f5c65e00640 1 Processor -- start 2024-03-06T07:39:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.956+0000 7f5c65e00640 1 -- start start 2024-03-06T07:39:46.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.957+0000 7f5c65e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5c600d0e50 0x7f5c60168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.957+0000 7f5c65e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c600b3710 con 0x7f5c6015e0b0 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.957+0000 7f5c65e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c600b39f0 con 0x7f5c600d4a10 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.957+0000 7f5c65e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c600b09e0 con 0x7f5c600d0e50 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.958+0000 7f5c64a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5c6015e0b0 0x7f5c600cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38688/0 (socket says 172.21.15.32:38688) 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.958+0000 7f5c64a00640 1 -- 172.21.15.32:0/481295107 learned_addr learned my addr 172.21.15.32:0/481295107 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.958+0000 7f5c65400640 1 --2- 172.21.15.32:0/481295107 >> v2:172.21.15.39:3300/0 conn(0x7f5c600d0e50 0x7f5c60168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.958+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814827895 0 0) 0x7f5c600b3710 con 0x7f5c6015e0b0 2024-03-06T07:39:46.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.958+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c48003520 con 0x7f5c6015e0b0 2024-03-06T07:39:46.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1399356169 0 0) 0x7f5c48003520 con 0x7f5c6015e0b0 2024-03-06T07:39:46.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 >> v2:172.21.15.39:3300/0 conn(0x7f5c600d0e50 msgr2=0x7f5c60168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:46.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 --2- 172.21.15.32:0/481295107 >> v2:172.21.15.39:3300/0 conn(0x7f5c600d0e50 0x7f5c60168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:46.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 >> v1:172.21.15.39:6789/0 conn(0x7f5c600d4a10 legacy=0x7f5c600d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c600b3cd0 con 0x7f5c6015e0b0 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f5c54001670 con 0x7f5c6015e0b0 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 >> v1:172.21.15.32:6789/0 conn(0x7f5c6015e0b0 legacy=0x7f5c600cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480039b0 0x7f5c48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c48004800 0x7f5c48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c480051c0 0x7f5c4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4800c030 con 0x7f5c480051c0 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4800c1e0 con 0x7f5c48004800 2024-03-06T07:39:46.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4800c3e0 con 0x7f5c480039b0 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c65400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c480051c0 0x7f5c4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.959+0000 7f5c5fe00640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c48004800 0x7f5c48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.960+0000 7f5c65400640 1 -- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480039b0 msgr2=0x7f5c48003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.960+0000 7f5c64a00640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480039b0 0x7f5c48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.960+0000 7f5c65400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480039b0 0x7f5c48003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.960+0000 7f5c65400640 1 -- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c48004800 msgr2=0x7f5c48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.960+0000 7f5c65400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c48004800 0x7f5c48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.960+0000 7f5c65400640 1 -- 172.21.15.32:0/481295107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c48003520 con 0x7f5c480051c0 2024-03-06T07:39:46.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.961+0000 7f5c65400640 1 -- 172.21.15.32:0/481295107 reap_dead start 2024-03-06T07:39:46.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.961+0000 7f5c65400640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c480051c0 0x7f5c4800b9c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009f90 tx=0x7f5c48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5d3a81f3e62b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:46.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58009c10 con 0x7f5c480051c0 2024-03-06T07:39:46.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5802cc20 con 0x7f5c480051c0 2024-03-06T07:39:46.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c5f400640 1 -- 172.21.15.32:0/481295107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58003990 con 0x7f5c480051c0 2024-03-06T07:39:46.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c65e00640 1 -- 172.21.15.32:0/481295107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c480051c0 msgr2=0x7f5c4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c65e00640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c480051c0 0x7f5c4800b9c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009f90 tx=0x7f5c48005830 comp rx=0 tx=0).stop 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c65e00640 1 -- 172.21.15.32:0/481295107 shutdown_connections 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c65e00640 1 --2- 172.21.15.32:0/481295107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c480051c0 0x7f5c4800b9c0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.962+0000 7f5c65e00640 1 -- 172.21.15.32:0/481295107 >> 172.21.15.32:0/481295107 conn(0x7f5c600d3d30 msgr2=0x7f5c600d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.963+0000 7f5c65e00640 1 -- 172.21.15.32:0/481295107 shutdown_connections 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.963+0000 7f5c65e00640 1 -- 172.21.15.32:0/481295107 wait complete. 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.964+0000 7f5c65e00640 1 Processor -- start 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.964+0000 7f5c65e00640 1 -- start start 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.965+0000 7f5c65e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480051c0 0x7f5c601f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.965+0000 7f5c65e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c600cd6f0 0x7f5c600c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.965+0000 7f5c65e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 0x7f5c600cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.965+0000 7f5c65e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c600cfc60 con 0x7f5c601f77f0 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.965+0000 7f5c65e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6015e490 con 0x7f5c600cd6f0 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.965+0000 7f5c65e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c600d0740 con 0x7f5c480051c0 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 0x7f5c600cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 0x7f5c600cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37728/0 (socket says 172.21.15.32:37728) 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 -- 172.21.15.32:0/4046755442 learned_addr learned my addr 172.21.15.32:0/4046755442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:46.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c64a00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480051c0 0x7f5c601f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c5fe00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c600cd6f0 0x7f5c600c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 -- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480051c0 msgr2=0x7f5c601f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480051c0 0x7f5c601f3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 -- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c600cd6f0 msgr2=0x7f5c600c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c600cd6f0 0x7f5c600c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.966+0000 7f5c65400640 1 -- 172.21.15.32:0/4046755442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c601f4260 con 0x7f5c601f77f0 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.967+0000 7f5c65400640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 0x7f5c600cae00 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800e200 tx=0x7f5c58009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.967+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58003e10 con 0x7f5c601f77f0 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.967+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c58044070 con 0x7f5c601f77f0 2024-03-06T07:39:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.967+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c601f4540 con 0x7f5c601f77f0 2024-03-06T07:39:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.967+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c580047e0 con 0x7f5c601f77f0 2024-03-06T07:39:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.967+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c601f72d0 con 0x7f5c601f77f0 2024-03-06T07:39:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.969+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c601f4a60 con 0x7f5c601f77f0 2024-03-06T07:39:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.969+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c5803d040 con 0x7f5c601f77f0 2024-03-06T07:39:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.970+0000 7f5c5cc00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c4006b4d0 0x7f5c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.970+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5c580ac6f0 con 0x7f5c601f77f0 2024-03-06T07:39:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.970+0000 7f5c64a00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c4006b4d0 0x7f5c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.971+0000 7f5c64a00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c4006b4d0 0x7f5c4006d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5c54001eb0 tx=0x7f5c54009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:46.975+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c5807b5b0 con 0x7f5c601f77f0 2024-03-06T07:39:47.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.356+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c6004f330 con 0x7f5c601f77f0 2024-03-06T07:39:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.357+0000 7f5c5cc00640 1 -- 172.21.15.32:0/4046755442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f5c5807aeb0 con 0x7f5c601f77f0 2024-03-06T07:39:47.360 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:47.360 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c4006b4d0 msgr2=0x7f5c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c4006b4d0 0x7f5c4006d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5c54001eb0 tx=0x7f5c54009040 comp rx=0 tx=0).stop 2024-03-06T07:39:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 msgr2=0x7f5c600cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 0x7f5c600cae00 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800e200 tx=0x7f5c58009750 comp rx=0 tx=0).stop 2024-03-06T07:39:47.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 shutdown_connections 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c4006b4d0 0x7f5c4006d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c601f77f0 0x7f5c600cae00 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c600cd6f0 0x7f5c600c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 --2- 172.21.15.32:0/4046755442 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5c480051c0 0x7f5c601f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 >> 172.21.15.32:0/4046755442 conn(0x7f5c600d3d30 msgr2=0x7f5c600cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.360+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 shutdown_connections 2024-03-06T07:39:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:47.361+0000 7f5c65e00640 1 -- 172.21.15.32:0/4046755442 wait complete. 2024-03-06T07:39:47.466 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:47.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:47.584 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:48.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.798+0000 7ffac5000640 1 Processor -- start 2024-03-06T07:39:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.798+0000 7ffac5000640 1 -- start start 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.799+0000 7ffac5000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffac015e070 0x7ffac00d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffac5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffac00d0cd0 0x7ffac00cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffac5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00d18c0 con 0x7ffac00cf410 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffac5000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac00d1ba0 con 0x7ffac015e070 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffac5000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac00d1e80 con 0x7ffac00d0cd0 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffabf400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffac015e070 0x7ffac00d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffabf400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffac015e070 0x7ffac00d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35442/0 (socket says 172.21.15.32:35442) 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffabf400640 1 -- 172.21.15.32:0/2831656357 learned_addr learned my addr 172.21.15.32:0/2831656357 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffabfe00640 1 --2- 172.21.15.32:0/2831656357 >> v2:172.21.15.39:3300/0 conn(0x7ffac00d0cd0 0x7ffac00cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffabf400640 1 -- 172.21.15.32:0/2831656357 >> v2:172.21.15.39:3300/0 conn(0x7ffac00d0cd0 msgr2=0x7ffac00cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.800+0000 7ffabf400640 1 --2- 172.21.15.32:0/2831656357 >> v2:172.21.15.39:3300/0 conn(0x7ffac00d0cd0 0x7ffac00cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.801+0000 7ffabf400640 1 -- 172.21.15.32:0/2831656357 >> v1:172.21.15.39:6789/0 conn(0x7ffac00cf410 legacy=0x7ffac00d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.801+0000 7ffabf400640 1 -- 172.21.15.32:0/2831656357 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00d2160 con 0x7ffac015e070 2024-03-06T07:39:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.801+0000 7ffabfe00640 1 --2- 172.21.15.32:0/2831656357 >> v2:172.21.15.39:3300/0 conn(0x7ffac00d0cd0 0x7ffac00cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.802+0000 7ffabf400640 1 --2- 172.21.15.32:0/2831656357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac00d2960 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ffac008c1b0 tx=0x7ffab002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c2bb4c902500d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffabea00640 1 -- 172.21.15.32:0/2831656357 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab002ca80 con 0x7ffac015e070 2024-03-06T07:39:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffabea00640 1 -- 172.21.15.32:0/2831656357 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab002cc20 con 0x7ffac015e070 2024-03-06T07:39:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffabea00640 1 -- 172.21.15.32:0/2831656357 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab00343f0 con 0x7ffac015e070 2024-03-06T07:39:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 -- 172.21.15.32:0/2831656357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 msgr2=0x7ffac00d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 --2- 172.21.15.32:0/2831656357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac00d2960 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ffac008c1b0 tx=0x7ffab002bee0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 -- 172.21.15.32:0/2831656357 shutdown_connections 2024-03-06T07:39:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 --2- 172.21.15.32:0/2831656357 >> v2:172.21.15.39:3300/0 conn(0x7ffac00d0cd0 0x7ffac00cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 --2- 172.21.15.32:0/2831656357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac00d2960 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 -- 172.21.15.32:0/2831656357 >> 172.21.15.32:0/2831656357 conn(0x7ffac008d2a0 msgr2=0x7ffac00d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 -- 172.21.15.32:0/2831656357 shutdown_connections 2024-03-06T07:39:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.803+0000 7ffac5000640 1 -- 172.21.15.32:0/2831656357 wait complete. 2024-03-06T07:39:48.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.806+0000 7ffac5000640 1 Processor -- start 2024-03-06T07:39:48.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.806+0000 7ffac5000640 1 -- start start 2024-03-06T07:39:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffac5000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffac00cf410 0x7ffac0158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffac5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac00d0cd0 0x7ffac0160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffac5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac01606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffac5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d4730 con 0x7ffac015e070 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffac5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00d45b0 con 0x7ffac00d0cd0 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffac5000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00b3630 con 0x7ffac00cf410 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffabfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac01606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffabfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac01606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35452/0 (socket says 172.21.15.32:35452) 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.808+0000 7ffabfe00640 1 -- 172.21.15.32:0/3671731188 learned_addr learned my addr 172.21.15.32:0/3671731188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabf400640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffac00cf410 0x7ffac0158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffab6a00640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac00d0cd0 0x7ffac0160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabfe00640 1 -- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffac00cf410 msgr2=0x7ffac0158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabfe00640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffac00cf410 0x7ffac0158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabfe00640 1 -- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac00d0cd0 msgr2=0x7ffac0160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabfe00640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac00d0cd0 0x7ffac0160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabfe00640 1 -- 172.21.15.32:0/3671731188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac015e9f0 con 0x7ffac015e070 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.809+0000 7ffabfe00640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac01606a0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c820 tx=0x7ffab800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.810+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8015070 con 0x7ffac015e070 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.810+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8004240 con 0x7ffac015e070 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.810+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac015ecd0 con 0x7ffac015e070 2024-03-06T07:39:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.810+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab800ce10 con 0x7ffac015e070 2024-03-06T07:39:48.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.810+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac015f290 con 0x7ffac015e070 2024-03-06T07:39:48.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.811+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac004f330 con 0x7ffac015e070 2024-03-06T07:39:48.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.812+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffab8007830 con 0x7ffac015e070 2024-03-06T07:39:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.813+0000 7ffab7e00640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.814+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffab808a3b0 con 0x7ffac015e070 2024-03-06T07:39:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.814+0000 7ffabf400640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.814+0000 7ffabf400640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ffab0035000 tx=0x7ffab0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:48.817+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffab8059280 con 0x7ffac015e070 2024-03-06T07:39:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.102+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffac015d720 con 0x7ffac015e070 2024-03-06T07:39:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.103+0000 7ffab7e00640 1 -- 172.21.15.32:0/3671731188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7ffab8058b80 con 0x7ffac015e070 2024-03-06T07:39:49.105 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:49.105 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa9c06b4d0 msgr2=0x7ffa9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ffab0035000 tx=0x7ffab0030000 comp rx=0 tx=0).stop 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 msgr2=0x7ffac01606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac01606a0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c820 tx=0x7ffab800cce0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 shutdown_connections 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ffa9c06b4d0 0x7ffa9c06d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffac015e070 0x7ffac01606a0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac00d0cd0 0x7ffac0160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 --2- 172.21.15.32:0/3671731188 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ffac00cf410 0x7ffac0158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.105+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 >> 172.21.15.32:0/3671731188 conn(0x7ffac008d2a0 msgr2=0x7ffac006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.106+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 shutdown_connections 2024-03-06T07:39:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:49.106+0000 7ffac5000640 1 -- 172.21.15.32:0/3671731188 wait complete. 2024-03-06T07:39:49.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:49.330 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:50.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.551+0000 7f760f200640 1 Processor -- start 2024-03-06T07:39:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.552+0000 7f760f200640 1 -- start start 2024-03-06T07:39:50.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.553+0000 7f760f200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f761015e0b0 0x7f76100cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.553+0000 7f760f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d4a10 0x7f76100d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.553+0000 7f760f200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100b3710 con 0x7f76100d0c80 2024-03-06T07:39:50.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.553+0000 7f760f200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76100b09e0 con 0x7f761015e0b0 2024-03-06T07:39:50.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.553+0000 7f760f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76100b39f0 con 0x7f76100d4a10 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760de00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f761015e0b0 0x7f76100cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760e800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f76100d0c80 0x7f76100d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47766/0 (socket says 172.21.15.32:47766) 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760e800640 1 -- 172.21.15.32:0/163432830 learned_addr learned my addr 172.21.15.32:0/163432830 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f7607e00640 1 --2- 172.21.15.32:0/163432830 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d4a10 0x7f76100d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760de00640 1 -- 172.21.15.32:0/163432830 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d4a10 msgr2=0x7f76100d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760de00640 1 --2- 172.21.15.32:0/163432830 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d4a10 0x7f76100d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760de00640 1 -- 172.21.15.32:0/163432830 >> v1:172.21.15.32:6789/0 conn(0x7f76100d0c80 legacy=0x7f76100d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.554+0000 7f760de00640 1 -- 172.21.15.32:0/163432830 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100b3cd0 con 0x7f761015e0b0 2024-03-06T07:39:50.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.556+0000 7f760de00640 1 --2- 172.21.15.32:0/163432830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cf310 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f76000097a0 tx=0x7f760002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c9ecda3e940ace server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:50.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.556+0000 7f760d400640 1 -- 172.21.15.32:0/163432830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600035020 con 0x7f761015e0b0 2024-03-06T07:39:50.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.556+0000 7f760d400640 1 -- 172.21.15.32:0/163432830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600031810 con 0x7f761015e0b0 2024-03-06T07:39:50.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.556+0000 7f760d400640 1 -- 172.21.15.32:0/163432830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600031be0 con 0x7f761015e0b0 2024-03-06T07:39:50.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 -- 172.21.15.32:0/163432830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 msgr2=0x7f76100cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 --2- 172.21.15.32:0/163432830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cf310 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f76000097a0 tx=0x7f760002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 -- 172.21.15.32:0/163432830 shutdown_connections 2024-03-06T07:39:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 --2- 172.21.15.32:0/163432830 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d4a10 0x7f76100d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 --2- 172.21.15.32:0/163432830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cf310 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 -- 172.21.15.32:0/163432830 >> 172.21.15.32:0/163432830 conn(0x7f76100d3d30 msgr2=0x7f76100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 -- 172.21.15.32:0/163432830 shutdown_connections 2024-03-06T07:39:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.557+0000 7f760f200640 1 -- 172.21.15.32:0/163432830 wait complete. 2024-03-06T07:39:50.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.559+0000 7f760f200640 1 Processor -- start 2024-03-06T07:39:50.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.559+0000 7f760f200640 1 -- start start 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d0c80 0x7f7610157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76100d4a10 0x7f761015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76100cfc60 con 0x7f761015e0b0 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761015e490 con 0x7f76100d4a10 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76100b54e0 con 0x7f76100d0c80 2024-03-06T07:39:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35470/0 (socket says 172.21.15.32:35470) 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.560+0000 7f760e800640 1 -- 172.21.15.32:0/1365293137 learned_addr learned my addr 172.21.15.32:0/1365293137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760de00640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d0c80 0x7f7610157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f7607e00640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76100d4a10 0x7f761015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760e800640 1 -- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d0c80 msgr2=0x7f7610157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760e800640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d0c80 0x7f7610157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760e800640 1 -- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76100d4a10 msgr2=0x7f761015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760e800640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76100d4a10 0x7f761015efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760e800640 1 -- 172.21.15.32:0/1365293137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100b7a90 con 0x7f761015e0b0 2024-03-06T07:39:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f760e800640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cc660 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f760800cbf0 tx=0x7f760800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.561+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608017070 con 0x7f761015e0b0 2024-03-06T07:39:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.562+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760800ad40 con 0x7f761015e0b0 2024-03-06T07:39:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.562+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608012600 con 0x7f761015e0b0 2024-03-06T07:39:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.562+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76100b7d70 con 0x7f761015e0b0 2024-03-06T07:39:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.562+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76100b8610 con 0x7f761015e0b0 2024-03-06T07:39:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.563+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76100b8810 con 0x7f761015e0b0 2024-03-06T07:39:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.564+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f760801a070 con 0x7f761015e0b0 2024-03-06T07:39:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.565+0000 7f7606000640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f75ec06b500 0x7f75ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.566+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7608089f40 con 0x7f761015e0b0 2024-03-06T07:39:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.566+0000 7f760de00640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f75ec06b500 0x7f75ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.566+0000 7f760de00640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f75ec06b500 0x7f75ec06d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7600036000 tx=0x7f760003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:50.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.570+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76100b8810 con 0x7f761015e0b0 2024-03-06T07:39:50.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.876+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761004f330 con 0x7f761015e0b0 2024-03-06T07:39:50.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.877+0000 7f7606000640 1 -- 172.21.15.32:0/1365293137 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f7608058690 con 0x7f761015e0b0 2024-03-06T07:39:50.880 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:50.880 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:50.880 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:50.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f75ec06b500 msgr2=0x7f75ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f75ec06b500 0x7f75ec06d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7600036000 tx=0x7f760003c040 comp rx=0 tx=0).stop 2024-03-06T07:39:50.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 msgr2=0x7f76100cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:50.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cc660 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f760800cbf0 tx=0x7f760800a430 comp rx=0 tx=0).stop 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 shutdown_connections 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f75ec06b500 0x7f75ec06d9c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761015e0b0 0x7f76100cc660 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76100d4a10 0x7f761015efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 --2- 172.21.15.32:0/1365293137 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76100d0c80 0x7f7610157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.880+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 >> 172.21.15.32:0/1365293137 conn(0x7f76100d3d30 msgr2=0x7f76100d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.881+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 shutdown_connections 2024-03-06T07:39:50.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:50.881+0000 7f760f200640 1 -- 172.21.15.32:0/1365293137 wait complete. 2024-03-06T07:39:51.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:51.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:51.470 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:51.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:51.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:52.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:52.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.323+0000 7fedd1200640 1 Processor -- start 2024-03-06T07:39:52.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.323+0000 7fedd1200640 1 -- start start 2024-03-06T07:39:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.325+0000 7fedd1200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fedcc0cd060 0x7fedcc1667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.325+0000 7fedd1200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedcc0714d0 con 0x7fedcc070f10 2024-03-06T07:39:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.325+0000 7fedd1200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedcc0717b0 con 0x7fedcc145300 2024-03-06T07:39:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.325+0000 7fedd1200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedcc071a90 con 0x7fedcc0cd060 2024-03-06T07:39:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedc3e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fedcc070f10 0x7fedcc0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47770/0 (socket says 172.21.15.32:47770) 2024-03-06T07:39:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedc3e00640 1 -- 172.21.15.32:0/2921792345 learned_addr learned my addr 172.21.15.32:0/2921792345 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:52.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcbe00640 1 --2- 172.21.15.32:0/2921792345 >> v2:172.21.15.39:3300/0 conn(0x7fedcc0cd060 0x7fedcc1667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028524430 0 0) 0x7fedcc0714d0 con 0x7fedcc070f10 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0003520 con 0x7fedcc070f10 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 185385343 0 0) 0x7fedb0003520 con 0x7fedcc070f10 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 >> v2:172.21.15.39:3300/0 conn(0x7fedcc0cd060 msgr2=0x7fedcc1667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcaa00640 1 --2- 172.21.15.32:0/2921792345 >> v2:172.21.15.39:3300/0 conn(0x7fedcc0cd060 0x7fedcc1667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.326+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 >> v1:172.21.15.39:6789/0 conn(0x7fedcc145300 legacy=0x7fedcc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc071d70 con 0x7fedcc070f10 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fedb4001670 con 0x7fedcc070f10 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 >> v1:172.21.15.32:6789/0 conn(0x7fedcc070f10 legacy=0x7fedcc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb0003a00 0x7fedb0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0004800 0x7fedb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb0005210 0x7fedb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc071a90 con 0x7fedb0005210 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb000c160 con 0x7fedb0003a00 2024-03-06T07:39:52.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb000c360 con 0x7fedb0004800 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcbe00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb0005210 0x7fedb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedcb400640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb0003a00 0x7fedb0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.327+0000 7fedc3e00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0004800 0x7fedb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedcbe00640 1 -- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0004800 msgr2=0x7fedb0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedcbe00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0004800 0x7fedb0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedcbe00640 1 -- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb0003a00 msgr2=0x7fedb0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedcbe00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb0003a00 0x7fedb0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedcbe00640 1 -- 172.21.15.32:0/2921792345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb0003520 con 0x7fedb0005210 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedc3e00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0004800 0x7fedb0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:52.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.328+0000 7fedcb400640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb0003a00 0x7fedb0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.329+0000 7fedcbe00640 1 -- 172.21.15.32:0/2921792345 reap_dead start 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.329+0000 7fedcbe00640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb0005210 0x7fedb000b9c0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fedcc072090 tx=0x7fedb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3a4af9673910e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.329+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4009c10 con 0x7fedb0005210 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.329+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc402cbd0 con 0x7fedb0005210 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.329+0000 7fedcaa00640 1 -- 172.21.15.32:0/2921792345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4003990 con 0x7fedb0005210 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 -- 172.21.15.32:0/2921792345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb0005210 msgr2=0x7fedb000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb0005210 0x7fedb000b9c0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fedcc072090 tx=0x7fedb0005830 comp rx=0 tx=0).stop 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 -- 172.21.15.32:0/2921792345 shutdown_connections 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 --2- 172.21.15.32:0/2921792345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb0005210 0x7fedb000b9c0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 -- 172.21.15.32:0/2921792345 >> 172.21.15.32:0/2921792345 conn(0x7fedcc06b3d0 msgr2=0x7fedcc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 -- 172.21.15.32:0/2921792345 shutdown_connections 2024-03-06T07:39:52.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.330+0000 7fedd1200640 1 -- 172.21.15.32:0/2921792345 wait complete. 2024-03-06T07:39:52.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.332+0000 7fedd1200640 1 Processor -- start 2024-03-06T07:39:52.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.332+0000 7fedd1200640 1 -- start start 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedd1200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0005210 0x7fedcc140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedd1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 0x7fedcc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedd1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedcc06cd00 0x7fedcc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedd1200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc0ccad0 con 0x7fedcc140ad0 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedd1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc0cc950 con 0x7fedcc06cd00 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedd1200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc0ccc50 con 0x7fedb0005210 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 0x7fedcc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 0x7fedcc06c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35498/0 (socket says 172.21.15.32:35498) 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 -- 172.21.15.32:0/3644440278 learned_addr learned my addr 172.21.15.32:0/3644440278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedcbe00640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedcc06cd00 0x7fedcc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedcb400640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0005210 0x7fedcc140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 -- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0005210 msgr2=0x7fedcc140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0005210 0x7fedcc140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 -- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedcc06cd00 msgr2=0x7fedcc06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedcc06cd00 0x7fedcc06d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.333+0000 7fedc3e00640 1 -- 172.21.15.32:0/3644440278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc06da70 con 0x7fedcc140ad0 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.334+0000 7fedc3e00640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 0x7fedcc06c7c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fedb400d730 tx=0x7fedb400dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.334+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4019070 con 0x7fedcc140ad0 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.334+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb40072d0 con 0x7fedcc140ad0 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.334+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40047e0 con 0x7fedcc140ad0 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.334+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedcc1410e0 con 0x7fedcc140ad0 2024-03-06T07:39:52.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.334+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc1416a0 con 0x7fedcc140ad0 2024-03-06T07:39:52.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.335+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc072d40 con 0x7fedcc140ad0 2024-03-06T07:39:52.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.336+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedb4004b90 con 0x7fedcc140ad0 2024-03-06T07:39:52.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.337+0000 7fedc3400640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feda806b500 0x7feda806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:52.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.337+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fedb4089e70 con 0x7fedcc140ad0 2024-03-06T07:39:52.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.338+0000 7fedcb400640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feda806b500 0x7feda806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:52.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.338+0000 7fedcb400640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feda806b500 0x7feda806d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fedbc000c00 tx=0x7fedbc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:52.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.342+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedcc072d40 con 0x7fedcc140ad0 2024-03-06T07:39:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.586+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedcc04f330 con 0x7fedcc140ad0 2024-03-06T07:39:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.587+0000 7fedc3400640 1 -- 172.21.15.32:0/3644440278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fedb4058c10 con 0x7fedcc140ad0 2024-03-06T07:39:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:52.590 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feda806b500 msgr2=0x7feda806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feda806b500 0x7feda806d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fedbc000c00 tx=0x7fedbc009040 comp rx=0 tx=0).stop 2024-03-06T07:39:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 msgr2=0x7fedcc06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 0x7fedcc06c7c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fedb400d730 tx=0x7fedb400dbf0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 shutdown_connections 2024-03-06T07:39:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feda806b500 0x7feda806d9c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedcc06cd00 0x7fedcc06d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedcc140ad0 0x7fedcc06c7c0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 --2- 172.21.15.32:0/3644440278 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fedb0005210 0x7fedcc140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.590+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 >> 172.21.15.32:0/3644440278 conn(0x7fedcc06b3d0 msgr2=0x7fedcc0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.591+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 shutdown_connections 2024-03-06T07:39:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:52.591+0000 7fedd1200640 1 -- 172.21.15.32:0/3644440278 wait complete. 2024-03-06T07:39:52.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:52.813 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:53.472 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:53.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:54.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.039+0000 7f6e1be00640 1 Processor -- start 2024-03-06T07:39:54.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.039+0000 7f6e1be00640 1 -- start start 2024-03-06T07:39:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.041+0000 7f6e1be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 0x7f6e1c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.041+0000 7f6e1be00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0d1120 con 0x7f6e1c0cf460 2024-03-06T07:39:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.041+0000 7f6e1be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0d1400 con 0x7f6e1c15dec0 2024-03-06T07:39:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.041+0000 7f6e1be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0b0ea0 con 0x7f6e1c15e2a0 2024-03-06T07:39:54.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.041+0000 7f6e1b400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6e1c0cf460 0x7f6e1c0b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47778/0 (socket says 172.21.15.32:47778) 2024-03-06T07:39:54.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.041+0000 7f6e1b400640 1 -- 172.21.15.32:0/4005005976 learned_addr learned my addr 172.21.15.32:0/4005005976 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:54.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e13e00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 0x7f6e1c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655878966 0 0) 0x7f6e1c0d1120 con 0x7f6e1c0cf460 2024-03-06T07:39:54.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e00003520 con 0x7f6e1c0cf460 2024-03-06T07:39:54.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1139246606 0 0) 0x7f6e00003520 con 0x7f6e1c0cf460 2024-03-06T07:39:54.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 msgr2=0x7f6e1c0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:54.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 0x7f6e1c0cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 >> v1:172.21.15.39:6789/0 conn(0x7f6e1c15dec0 legacy=0x7f6e1c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0b0b30 con 0x7f6e1c0cf460 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f6e140019f0 con 0x7f6e1c0cf460 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 >> v1:172.21.15.32:6789/0 conn(0x7f6e1c0cf460 legacy=0x7f6e1c0b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e000039e0 0x7f6e00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e00004800 0x7f6e00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 0x7f6e000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0b0ea0 con 0x7f6e000039e0 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.042+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0000ba70 con 0x7f6e00004800 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0000bc70 con 0x7f6e1c15e2a0 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e000039e0 0x7f6e00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e13e00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e00004800 0x7f6e00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1b400640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 0x7f6e000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1aa00640 1 -- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 msgr2=0x7f6e000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:54.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c15e2a0 0x7f6e000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1aa00640 1 -- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e00004800 msgr2=0x7f6e00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:54.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.043+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e00004800 0x7f6e00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.044+0000 7f6e1aa00640 1 -- 172.21.15.32:0/4005005976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00003520 con 0x7f6e000039e0 2024-03-06T07:39:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.045+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e000039e0 0x7f6e00003e00 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c001720 tx=0x7f6e00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e0291c75d67f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.045+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c001370 con 0x7f6e000039e0 2024-03-06T07:39:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.045+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c0052d0 con 0x7f6e000039e0 2024-03-06T07:39:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.045+0000 7f6e1a000640 1 -- 172.21.15.32:0/4005005976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c003830 con 0x7f6e000039e0 2024-03-06T07:39:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.045+0000 7f6e1be00640 1 -- 172.21.15.32:0/4005005976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e000039e0 msgr2=0x7f6e00003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.045+0000 7f6e1be00640 1 --2- 172.21.15.32:0/4005005976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e000039e0 0x7f6e00003e00 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c001720 tx=0x7f6e00005830 comp rx=0 tx=0).stop 2024-03-06T07:39:54.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.046+0000 7f6e1b400640 1 -- 172.21.15.32:0/4005005976 reap_dead start 2024-03-06T07:39:54.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.046+0000 7f6e1be00640 1 -- 172.21.15.32:0/4005005976 shutdown_connections 2024-03-06T07:39:54.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.046+0000 7f6e1be00640 1 -- 172.21.15.32:0/4005005976 >> 172.21.15.32:0/4005005976 conn(0x7f6e1c0d3d40 msgr2=0x7f6e1c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:54.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.046+0000 7f6e1be00640 1 -- 172.21.15.32:0/4005005976 shutdown_connections 2024-03-06T07:39:54.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.046+0000 7f6e1be00640 1 -- 172.21.15.32:0/4005005976 wait complete. 2024-03-06T07:39:54.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.048+0000 7f6e1be00640 1 Processor -- start 2024-03-06T07:39:54.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.048+0000 7f6e1be00640 1 -- start start 2024-03-06T07:39:54.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e000039e0 0x7f6e1c0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 0x7f6e1c1f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1be00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c1f68a0 0x7f6e1c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0d0ec0 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0d0d40 con 0x7f6e000039e0 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1be00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0cef20 con 0x7f6e1c1f68a0 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 0x7f6e1c1f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 0x7f6e1c1f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35510/0 (socket says 172.21.15.32:35510) 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 -- 172.21.15.32:0/3911103847 learned_addr learned my addr 172.21.15.32:0/3911103847 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e000039e0 0x7f6e1c0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e1b400640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c1f68a0 0x7f6e1c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 -- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c1f68a0 msgr2=0x7f6e1c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c1f68a0 0x7f6e1c06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 -- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e000039e0 msgr2=0x7f6e1c0b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.049+0000 7f6e13e00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e000039e0 0x7f6e1c0b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e13e00640 1 -- 172.21.15.32:0/3911103847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c15e720 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e13e00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 0x7f6e1c1f4ce0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f6e04011600 tx=0x7f6e04011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400f920 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0400fac0 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400fe70 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c1f4170 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.050+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c1f4730 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.052+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c068120 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.053+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e04015460 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.053+0000 7f6e12a00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6df806b4d0 0x7f6df806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:54.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.054+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6e0408ab70 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.054+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6df806b4d0 0x7f6df806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:54.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.054+0000 7f6e1aa00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6df806b4d0 0x7f6df806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c033000 tx=0x7f6e0c009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:54.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.062+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e1c068120 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.309+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e1c04f330 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.309+0000 7f6e12a00640 1 -- 172.21.15.32:0/3911103847 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f6e04019030 con 0x7f6e1c1f3b20 2024-03-06T07:39:54.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:54.312 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:54.312 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6df806b4d0 msgr2=0x7f6df806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6df806b4d0 0x7f6df806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c033000 tx=0x7f6e0c009030 comp rx=0 tx=0).stop 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 msgr2=0x7f6e1c1f4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 0x7f6e1c1f4ce0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f6e04011600 tx=0x7f6e04011ac0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 shutdown_connections 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6df806b4d0 0x7f6df806d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6e1c1f68a0 0x7f6e1c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e1c1f3b20 0x7f6e1c1f4ce0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 --2- 172.21.15.32:0/3911103847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e000039e0 0x7f6e1c0b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 >> 172.21.15.32:0/3911103847 conn(0x7f6e1c0d3d40 msgr2=0x7f6e1c0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 shutdown_connections 2024-03-06T07:39:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:54.312+0000 7f6e1be00640 1 -- 172.21.15.32:0/3911103847 wait complete. 2024-03-06T07:39:54.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:54.534 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:55.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.787+0000 7efc0d200640 1 Processor -- start 2024-03-06T07:39:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.787+0000 7efc0d200640 1 -- start start 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.788+0000 7efc0d200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc080cd060 0x7efc0815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.788+0000 7efc0d200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efc0815e810 0x7efc080b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.788+0000 7efc0d200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0815ed10 con 0x7efc081552b0 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.788+0000 7efc0d200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0815eff0 con 0x7efc080cd060 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.788+0000 7efc0d200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0815f2d0 con 0x7efc0815e810 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efc07400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efc081552b0 0x7efc080cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47782/0 (socket says 172.21.15.32:47782) 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efc07400640 1 -- 172.21.15.32:0/930837713 learned_addr learned my addr 172.21.15.32:0/930837713 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efbffe00640 1 --2- 172.21.15.32:0/930837713 >> v2:172.21.15.32:3300/0 conn(0x7efc080cd060 0x7efc0815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efc07e00640 1 --2- 172.21.15.32:0/930837713 >> v2:172.21.15.39:3300/0 conn(0x7efc0815e810 0x7efc080b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efbffe00640 1 -- 172.21.15.32:0/930837713 >> v2:172.21.15.39:3300/0 conn(0x7efc0815e810 msgr2=0x7efc080b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efbffe00640 1 --2- 172.21.15.32:0/930837713 >> v2:172.21.15.39:3300/0 conn(0x7efc0815e810 0x7efc080b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efbffe00640 1 -- 172.21.15.32:0/930837713 >> v1:172.21.15.32:6789/0 conn(0x7efc081552b0 legacy=0x7efc080cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efbffe00640 1 -- 172.21.15.32:0/930837713 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0815f5b0 con 0x7efc080cd060 2024-03-06T07:39:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efc07e00640 1 --2- 172.21.15.32:0/930837713 >> v2:172.21.15.39:3300/0 conn(0x7efc0815e810 0x7efc080b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.789+0000 7efc06a00640 1 -- 172.21.15.32:0/930837713 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716367038 0 0) 0x7efc0815ed10 con 0x7efc081552b0 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.790+0000 7efbffe00640 1 --2- 172.21.15.32:0/930837713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc0815b1d0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7efc080af700 tx=0x7efbf002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80dd6bd28e303213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc06a00640 1 -- 172.21.15.32:0/930837713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf00315a0 con 0x7efc080cd060 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc06a00640 1 -- 172.21.15.32:0/930837713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf0037070 con 0x7efc080cd060 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc06a00640 1 -- 172.21.15.32:0/930837713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0031960 con 0x7efc080cd060 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 -- 172.21.15.32:0/930837713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 msgr2=0x7efc0815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 --2- 172.21.15.32:0/930837713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc0815b1d0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7efc080af700 tx=0x7efbf002b840 comp rx=0 tx=0).stop 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 -- 172.21.15.32:0/930837713 shutdown_connections 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 --2- 172.21.15.32:0/930837713 >> v2:172.21.15.39:3300/0 conn(0x7efc0815e810 0x7efc080b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 --2- 172.21.15.32:0/930837713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc0815b1d0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 -- 172.21.15.32:0/930837713 >> 172.21.15.32:0/930837713 conn(0x7efc0806b3d0 msgr2=0x7efc0806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.791+0000 7efc0d200640 1 -- 172.21.15.32:0/930837713 shutdown_connections 2024-03-06T07:39:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.792+0000 7efc0d200640 1 -- 172.21.15.32:0/930837713 wait complete. 2024-03-06T07:39:55.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.793+0000 7efc0d200640 1 Processor -- start 2024-03-06T07:39:55.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.793+0000 7efc0d200640 1 -- start start 2024-03-06T07:39:55.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc0d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc08149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc0d200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc081552b0 0x7efc0814a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc0d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc0815e810 0x7efc0814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc0d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0806dd30 con 0x7efc080cd060 2024-03-06T07:39:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc0d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080b3760 con 0x7efc0815e810 2024-03-06T07:39:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc0d200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0806deb0 con 0x7efc081552b0 2024-03-06T07:39:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc08149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc08149b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35520/0 (socket says 172.21.15.32:35520) 2024-03-06T07:39:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.794+0000 7efc07400640 1 -- 172.21.15.32:0/3288747141 learned_addr learned my addr 172.21.15.32:0/3288747141 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efc07400640 1 -- 172.21.15.32:0/3288747141 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc081552b0 msgr2=0x7efc0814a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efc07400640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc081552b0 0x7efc0814a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efc07400640 1 -- 172.21.15.32:0/3288747141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc0815e810 msgr2=0x7efc0814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efc07400640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc0815e810 0x7efc0814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efc07400640 1 -- 172.21.15.32:0/3288747141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0814a580 con 0x7efc080cd060 2024-03-06T07:39:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efc07400640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc08149b00 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c500 tx=0x7efbf800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.795+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8018070 con 0x7efc080cd060 2024-03-06T07:39:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.796+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf8007560 con 0x7efc080cd060 2024-03-06T07:39:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.796+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0814aca0 con 0x7efc080cd060 2024-03-06T07:39:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.796+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8004a20 con 0x7efc080cd060 2024-03-06T07:39:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.796+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0814b260 con 0x7efc080cd060 2024-03-06T07:39:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.797+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0804f330 con 0x7efc080cd060 2024-03-06T07:39:55.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.798+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efbf801b070 con 0x7efc080cd060 2024-03-06T07:39:55.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.799+0000 7efbff400640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe406b500 0x7efbe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:55.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.799+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efbf80894b0 con 0x7efc080cd060 2024-03-06T07:39:55.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.799+0000 7efbffe00640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe406b500 0x7efbe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:55.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.800+0000 7efbffe00640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe406b500 0x7efbe406d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efbf0036000 tx=0x7efbf0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:55.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:55.803+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efbf8058380 con 0x7efc080cd060 2024-03-06T07:39:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.046+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc0806e580 con 0x7efc080cd060 2024-03-06T07:39:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.047+0000 7efbff400640 1 -- 172.21.15.32:0/3288747141 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7efbf805e080 con 0x7efc080cd060 2024-03-06T07:39:56.049 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:56.049 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.049+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe406b500 msgr2=0x7efbe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.049+0000 7efc0d200640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe406b500 0x7efbe406d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efbf0036000 tx=0x7efbf0030000 comp rx=0 tx=0).stop 2024-03-06T07:39:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.049+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 msgr2=0x7efc08149b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.049+0000 7efc0d200640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc08149b00 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c500 tx=0x7efbf800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 shutdown_connections 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efbe406b500 0x7efbe406d9c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc0815e810 0x7efc0814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efc081552b0 0x7efc0814a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 --2- 172.21.15.32:0/3288747141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080cd060 0x7efc08149b00 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 >> 172.21.15.32:0/3288747141 conn(0x7efc0806b3d0 msgr2=0x7efc08070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 shutdown_connections 2024-03-06T07:39:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:56.050+0000 7efc0d200640 1 -- 172.21.15.32:0/3288747141 wait complete. 2024-03-06T07:39:56.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:56.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:57.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:57.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.496+0000 7f7a8ea00640 1 Processor -- start 2024-03-06T07:39:57.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.496+0000 7f7a8ea00640 1 -- start start 2024-03-06T07:39:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a8815e0b0 0x7f7a880cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7a880d0e50 0x7f7a880b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8ea00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a880b3710 con 0x7f7a880d4a10 2024-03-06T07:39:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8ea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a880b09e0 con 0x7f7a8815e0b0 2024-03-06T07:39:57.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8ea00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a880b39f0 con 0x7f7a880d0e50 2024-03-06T07:39:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8d600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a8815e0b0 0x7f7a880cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8cc00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7a880d4a10 0x7f7a880d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47786/0 (socket says 172.21.15.32:47786) 2024-03-06T07:39:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8d600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7a8815e0b0 0x7f7a880cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35522/0 (socket says 172.21.15.32:35522) 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.498+0000 7f7a8cc00640 1 -- 172.21.15.32:0/4226850480 learned_addr learned my addr 172.21.15.32:0/4226850480 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.499+0000 7f7a8e000640 1 --2- 172.21.15.32:0/4226850480 >> v2:172.21.15.39:3300/0 conn(0x7f7a880d0e50 0x7f7a880b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.499+0000 7f7a8d600640 1 -- 172.21.15.32:0/4226850480 >> v2:172.21.15.39:3300/0 conn(0x7f7a880d0e50 msgr2=0x7f7a880b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.499+0000 7f7a8d600640 1 --2- 172.21.15.32:0/4226850480 >> v2:172.21.15.39:3300/0 conn(0x7f7a880d0e50 0x7f7a880b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.499+0000 7f7a8d600640 1 -- 172.21.15.32:0/4226850480 >> v1:172.21.15.32:6789/0 conn(0x7f7a880d4a10 legacy=0x7f7a880d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.499+0000 7f7a8d600640 1 -- 172.21.15.32:0/4226850480 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880b3cd0 con 0x7f7a8815e0b0 2024-03-06T07:39:57.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.499+0000 7f7a87e00640 1 -- 172.21.15.32:0/4226850480 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912883746 0 0) 0x7f7a880b3710 con 0x7f7a880d4a10 2024-03-06T07:39:57.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.501+0000 7f7a8d600640 1 --2- 172.21.15.32:0/4226850480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8815e0b0 0x7f7a880cf310 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f7a8808d6d0 tx=0x7f7a7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a2ea01bcd767f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:57.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.501+0000 7f7a87e00640 1 -- 172.21.15.32:0/4226850480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c035020 con 0x7f7a8815e0b0 2024-03-06T07:39:57.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.501+0000 7f7a87e00640 1 -- 172.21.15.32:0/4226850480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c031830 con 0x7f7a8815e0b0 2024-03-06T07:39:57.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.501+0000 7f7a87e00640 1 -- 172.21.15.32:0/4226850480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c031c00 con 0x7f7a8815e0b0 2024-03-06T07:39:57.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 -- 172.21.15.32:0/4226850480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8815e0b0 msgr2=0x7f7a880cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/4226850480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8815e0b0 0x7f7a880cf310 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f7a8808d6d0 tx=0x7f7a7c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 -- 172.21.15.32:0/4226850480 shutdown_connections 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/4226850480 >> v2:172.21.15.39:3300/0 conn(0x7f7a880d0e50 0x7f7a880b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/4226850480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8815e0b0 0x7f7a880cf310 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 -- 172.21.15.32:0/4226850480 >> 172.21.15.32:0/4226850480 conn(0x7f7a880d3d30 msgr2=0x7f7a880d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 -- 172.21.15.32:0/4226850480 shutdown_connections 2024-03-06T07:39:57.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.502+0000 7f7a8ea00640 1 -- 172.21.15.32:0/4226850480 wait complete. 2024-03-06T07:39:57.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.505+0000 7f7a8ea00640 1 Processor -- start 2024-03-06T07:39:57.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.505+0000 7f7a8ea00640 1 -- start start 2024-03-06T07:39:57.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a880d0e50 0x7f7a881612d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 0x7f7a88160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a8815e0b0 0x7f7a8815fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880cfc60 con 0x7f7a880d4a10 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8815e490 con 0x7f7a880d0e50 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880cf850 con 0x7f7a8815e0b0 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.506+0000 7f7a8cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 0x7f7a88160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 0x7f7a88160940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35538/0 (socket says 172.21.15.32:35538) 2024-03-06T07:39:57.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 -- 172.21.15.32:0/1548219101 learned_addr learned my addr 172.21.15.32:0/1548219101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:57.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8e000640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a8815e0b0 0x7f7a8815fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8d600640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a880d0e50 0x7f7a881612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 -- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a8815e0b0 msgr2=0x7f7a8815fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a8815e0b0 0x7f7a8815fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 -- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a880d0e50 msgr2=0x7f7a881612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a880d0e50 0x7f7a881612d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 -- 172.21.15.32:0/1548219101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880ca480 con 0x7f7a880d4a10 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8e000640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a8815e0b0 0x7f7a8815fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.507+0000 7f7a8cc00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 0x7f7a88160940 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400c4b0 tx=0x7f7a7400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.508+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74018070 con 0x7f7a880d4a10 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.508+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a74007500 con 0x7f7a880d4a10 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.508+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8815a7b0 con 0x7f7a880d4a10 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.508+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74004a20 con 0x7f7a880d4a10 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.508+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8815aa20 con 0x7f7a880d4a10 2024-03-06T07:39:57.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.509+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8804f330 con 0x7f7a880d4a10 2024-03-06T07:39:57.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.511+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a7401b070 con 0x7f7a880d4a10 2024-03-06T07:39:57.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.512+0000 7f7a85600640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a6806b4d0 0x7f7a6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:57.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.512+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a740895c0 con 0x7f7a880d4a10 2024-03-06T07:39:57.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.513+0000 7f7a8d600640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a6806b4d0 0x7f7a6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:57.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.513+0000 7f7a8d600640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a6806b4d0 0x7f7a6806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:57.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.517+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a74058420 con 0x7f7a880d4a10 2024-03-06T07:39:57.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.757+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a88074430 con 0x7f7a880d4a10 2024-03-06T07:39:57.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.758+0000 7f7a85600640 1 -- 172.21.15.32:0/1548219101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f7a74006e20 con 0x7f7a880d4a10 2024-03-06T07:39:57.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:57.761 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:57.761 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a6806b4d0 msgr2=0x7f7a6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a6806b4d0 0x7f7a6806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 msgr2=0x7f7a88160940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 0x7f7a88160940 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400c4b0 tx=0x7f7a7400c970 comp rx=0 tx=0).stop 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 shutdown_connections 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f7a6806b4d0 0x7f7a6806d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f7a8815e0b0 0x7f7a8815fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a880d4a10 0x7f7a88160940 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 --2- 172.21.15.32:0/1548219101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a880d0e50 0x7f7a881612d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.760+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 >> 172.21.15.32:0/1548219101 conn(0x7f7a880d3d30 msgr2=0x7f7a880b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.761+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 shutdown_connections 2024-03-06T07:39:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:57.761+0000 7f7a8ea00640 1 -- 172.21.15.32:0/1548219101 wait complete. 2024-03-06T07:39:57.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:57.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:39:58.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.210+0000 7f2efbe00640 1 Processor -- start 2024-03-06T07:39:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.211+0000 7f2efbe00640 1 -- start start 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.212+0000 7f2efbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2efc151b30 0x7f2efc06f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.212+0000 7f2efbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc15cbb0 0x7f2efc0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.212+0000 7f2efbe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc06e4b0 con 0x7f2efc0b4650 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.212+0000 7f2efbe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc0b5000 con 0x7f2efc151b30 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.212+0000 7f2efbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc06e790 con 0x7f2efc15cbb0 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.213+0000 7f2efa000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2efc0b4650 0x7f2efc15c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54876/0 (socket says 172.21.15.32:54876) 2024-03-06T07:39:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.213+0000 7f2efa000640 1 -- 172.21.15.32:0/1203484925 learned_addr learned my addr 172.21.15.32:0/1203484925 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:39:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.213+0000 7f2efb400640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc15cbb0 0x7f2efc0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.213+0000 7f2efaa00640 1 --2- 172.21.15.32:0/1203484925 >> v2:172.21.15.39:3300/0 conn(0x7f2efc151b30 0x7f2efc06f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.213+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325583293 0 0) 0x7f2efc06e4b0 con 0x7f2efc0b4650 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.213+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0005730 con 0x7f2efc0b4650 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594153708 0 0) 0x7f2ee0005730 con 0x7f2efc0b4650 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc15cbb0 msgr2=0x7f2efc0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc15cbb0 0x7f2efc0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 >> v2:172.21.15.39:3300/0 conn(0x7f2efc151b30 msgr2=0x7f2efc06f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 --2- 172.21.15.32:0/1203484925 >> v2:172.21.15.39:3300/0 conn(0x7f2efc151b30 0x7f2efc06f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc06ea70 con 0x7f2efc0b4650 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f2ee8001670 con 0x7f2efc0b4650 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 >> v1:172.21.15.32:6789/0 conn(0x7f2efc0b4650 legacy=0x7f2efc15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ee0005bc0 0x7f2ee0005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2ee0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee00073d0 0x7f2ee000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0b5000 con 0x7f2ee0006a10 2024-03-06T07:39:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc06e790 con 0x7f2ee00073d0 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000e190 con 0x7f2ee0005bc0 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.214+0000 7f2efa000640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2ee0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efb400640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee00073d0 0x7f2ee000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efaa00640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ee0005bc0 0x7f2ee0005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efa000640 1 -- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ee0005bc0 msgr2=0x7f2ee0005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efa000640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ee0005bc0 0x7f2ee0005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efa000640 1 -- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee00073d0 msgr2=0x7f2ee000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efa000640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee00073d0 0x7f2ee000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efa000640 1 -- 172.21.15.32:0/1203484925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee0005730 con 0x7f2ee0006a10 2024-03-06T07:39:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.215+0000 7f2efaa00640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2ee0005bc0 0x7f2ee0005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.216+0000 7f2efa000640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2ee0006e90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c190 tx=0x7f2ee0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2044492c4530d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8007c60 con 0x7f2ee0006a10 2024-03-06T07:39:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee8030070 con 0x7f2ee0006a10 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2ef9600640 1 -- 172.21.15.32:0/1203484925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8005d30 con 0x7f2ee0006a10 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2efbe00640 1 -- 172.21.15.32:0/1203484925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 msgr2=0x7f2ee0006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2efbe00640 1 --2- 172.21.15.32:0/1203484925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2ee0006e90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c190 tx=0x7f2ee0007a40 comp rx=0 tx=0).stop 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2efb400640 1 -- 172.21.15.32:0/1203484925 reap_dead start 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2efbe00640 1 -- 172.21.15.32:0/1203484925 shutdown_connections 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.217+0000 7f2efbe00640 1 -- 172.21.15.32:0/1203484925 >> 172.21.15.32:0/1203484925 conn(0x7f2efc06c0c0 msgr2=0x7f2efc06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.218+0000 7f2efbe00640 1 -- 172.21.15.32:0/1203484925 shutdown_connections 2024-03-06T07:39:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.218+0000 7f2efbe00640 1 -- 172.21.15.32:0/1203484925 wait complete. 2024-03-06T07:39:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.219+0000 7f2efbe00640 1 Processor -- start 2024-03-06T07:39:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.220+0000 7f2efbe00640 1 -- start start 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2efc13c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efbe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc13ce00 0x7f2efc154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2efc14e130 0x7f2efc14e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc06f7b0 con 0x7f2ee0006a10 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc151f10 con 0x7f2efc14e130 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efbe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc13d300 con 0x7f2efc13ce00 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2efc13c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2efc13c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58074/0 (socket says 172.21.15.32:58074) 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efaa00640 1 -- 172.21.15.32:0/2051785727 learned_addr learned my addr 172.21.15.32:0/2051785727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:39:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efa000640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc13ce00 0x7f2efc154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efb400640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2efc14e130 0x7f2efc14e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efaa00640 1 -- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc13ce00 msgr2=0x7f2efc154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efaa00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc13ce00 0x7f2efc154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.221+0000 7f2efaa00640 1 -- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2efc14e130 msgr2=0x7f2efc14e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2efaa00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2efc14e130 0x7f2efc14e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2efaa00640 1 -- 172.21.15.32:0/2051785727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc13d5e0 con 0x7f2ee0006a10 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2efb400640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2efc14e130 0x7f2efc14e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2efaa00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2efc13c490 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00f3d0 tx=0x7f2eec00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec003990 con 0x7f2ee0006a10 2024-03-06T07:39:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec003b30 con 0x7f2ee0006a10 2024-03-06T07:39:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec004020 con 0x7f2ee0006a10 2024-03-06T07:39:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc13d8c0 con 0x7f2ee0006a10 2024-03-06T07:39:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.222+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc13c9d0 con 0x7f2ee0006a10 2024-03-06T07:39:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.224+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc04f330 con 0x7f2ee0006a10 2024-03-06T07:39:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.225+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2eec005d10 con 0x7f2ee0006a10 2024-03-06T07:39:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.225+0000 7f2ef3400640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2edc06b500 0x7f2edc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.226+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2eec08ac50 con 0x7f2ee0006a10 2024-03-06T07:39:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.226+0000 7f2efa000640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2edc06b500 0x7f2edc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.226+0000 7f2efa000640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2edc06b500 0x7f2edc06d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c190 tx=0x7f2ee8007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:59.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.230+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2eec024020 con 0x7f2ee0006a10 2024-03-06T07:39:59.478 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:59.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.496+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2efc06e110 con 0x7f2ee0006a10 2024-03-06T07:39:59.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.497+0000 7f2ef3400640 1 -- 172.21.15.32:0/2051785727 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f2eec059a80 con 0x7f2ee0006a10 2024-03-06T07:39:59.500 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:39:59.500 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:39:59.500 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2edc06b500 msgr2=0x7f2edc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2edc06b500 0x7f2edc06d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c190 tx=0x7f2ee8007000 comp rx=0 tx=0).stop 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 msgr2=0x7f2efc13c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2efc13c490 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00f3d0 tx=0x7f2eec00f890 comp rx=0 tx=0).stop 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 shutdown_connections 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2edc06b500 0x7f2edc06d9c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2efc14e130 0x7f2efc14e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2efc13ce00 0x7f2efc154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 --2- 172.21.15.32:0/2051785727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0006a10 0x7f2efc13c490 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:59.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.500+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 >> 172.21.15.32:0/2051785727 conn(0x7f2efc06c0c0 msgr2=0x7f2efc0cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:59.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.501+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 shutdown_connections 2024-03-06T07:39:59.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:39:59.501+0000 7f2efbe00640 1 -- 172.21.15.32:0/2051785727 wait complete. 2024-03-06T07:39:59.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:39:59.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:00.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:00.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.944+0000 7f614ca00640 1 Processor -- start 2024-03-06T07:40:00.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.944+0000 7f614ca00640 1 -- start start 2024-03-06T07:40:00.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.946+0000 7f614ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f61480b3c90 0x7f61480cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.946+0000 7f614ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480cf5c0 0x7f61480b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.946+0000 7f614ca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480b45b0 con 0x7f614806cb10 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.946+0000 7f614ca00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61480ce160 con 0x7f61480b3c90 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.946+0000 7f614ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61480b4860 con 0x7f61480cf5c0 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.946+0000 7f6145e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f61480b3c90 0x7f61480cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f61480b3c90 0x7f61480cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58082/0 (socket says 172.21.15.32:58082) 2024-03-06T07:40:00.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145e00640 1 -- 172.21.15.32:0/1853431005 learned_addr learned my addr 172.21.15.32:0/1853431005 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:00.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145400640 1 --2- 172.21.15.32:0/1853431005 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480cf5c0 0x7f61480b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145e00640 1 -- 172.21.15.32:0/1853431005 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480cf5c0 msgr2=0x7f61480b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:00.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145e00640 1 --2- 172.21.15.32:0/1853431005 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480cf5c0 0x7f61480b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145e00640 1 -- 172.21.15.32:0/1853431005 >> v1:172.21.15.39:6789/0 conn(0x7f614806cb10 legacy=0x7f614806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:00.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.947+0000 7f6145e00640 1 -- 172.21.15.32:0/1853431005 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61480b4b40 con 0x7f61480b3c90 2024-03-06T07:40:00.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.948+0000 7f6145e00640 1 --2- 172.21.15.32:0/1853431005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61480b3c90 0x7f61480cdc20 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f613800bb10 tx=0x7f613802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2f03240a330400 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:00.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.948+0000 7f6144a00640 1 -- 172.21.15.32:0/1853431005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613803b070 con 0x7f61480b3c90 2024-03-06T07:40:00.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f6144a00640 1 -- 172.21.15.32:0/1853431005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138036440 con 0x7f61480b3c90 2024-03-06T07:40:00.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f6144a00640 1 -- 172.21.15.32:0/1853431005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138036810 con 0x7f61480b3c90 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 -- 172.21.15.32:0/1853431005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61480b3c90 msgr2=0x7f61480cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 --2- 172.21.15.32:0/1853431005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61480b3c90 0x7f61480cdc20 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f613800bb10 tx=0x7f613802e150 comp rx=0 tx=0).stop 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 -- 172.21.15.32:0/1853431005 shutdown_connections 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 --2- 172.21.15.32:0/1853431005 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480cf5c0 0x7f61480b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 --2- 172.21.15.32:0/1853431005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61480b3c90 0x7f61480cdc20 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 -- 172.21.15.32:0/1853431005 >> 172.21.15.32:0/1853431005 conn(0x7f614806afb0 msgr2=0x7f614806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 -- 172.21.15.32:0/1853431005 shutdown_connections 2024-03-06T07:40:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.949+0000 7f614ca00640 1 -- 172.21.15.32:0/1853431005 wait complete. 2024-03-06T07:40:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.952+0000 7f614ca00640 1 Processor -- start 2024-03-06T07:40:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.952+0000 7f614ca00640 1 -- start start 2024-03-06T07:40:00.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.953+0000 7f614ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 0x7f6148157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.953+0000 7f614ca00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480b3c90 0x7f6148154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.953+0000 7f614ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61480cf5c0 0x7f61481f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.953+0000 7f614ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614815b340 con 0x7f614806cb10 2024-03-06T07:40:00.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f614ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61480b5100 con 0x7f61480cf5c0 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f614ca00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614806d6b0 con 0x7f61480b3c90 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 0x7f6148157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 0x7f6148157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58090/0 (socket says 172.21.15.32:58090) 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 -- 172.21.15.32:0/2931764881 learned_addr learned my addr 172.21.15.32:0/2931764881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145400640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480b3c90 0x7f6148154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 -- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480b3c90 msgr2=0x7f6148154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6146800640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61480cf5c0 0x7f61481f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480b3c90 0x7f6148154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 -- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61480cf5c0 msgr2=0x7f61481f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61480cf5c0 0x7f61481f4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145e00640 1 -- 172.21.15.32:0/2931764881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148151e60 con 0x7f614806cb10 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.954+0000 7f6145400640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480b3c90 0x7f6148154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.955+0000 7f6145e00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 0x7f6148157390 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f613800b2c0 tx=0x7f61380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.955+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138046070 con 0x7f614806cb10 2024-03-06T07:40:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.955+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613803d050 con 0x7f614806cb10 2024-03-06T07:40:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.955+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148157a00 con 0x7f614806cb10 2024-03-06T07:40:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.955+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613803b030 con 0x7f614806cb10 2024-03-06T07:40:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.955+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614809bb90 con 0x7f614806cb10 2024-03-06T07:40:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.957+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148069510 con 0x7f614806cb10 2024-03-06T07:40:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.958+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6138049070 con 0x7f614806cb10 2024-03-06T07:40:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.959+0000 7f613e000640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f612806b4d0 0x7f612806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.959+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f61380abc00 con 0x7f614806cb10 2024-03-06T07:40:00.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.960+0000 7f6145400640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f612806b4d0 0x7f612806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.960+0000 7f6145400640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f612806b4d0 0x7f612806d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6134006fd0 tx=0x7f6134008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:00.969+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61380b2080 con 0x7f614806cb10 2024-03-06T07:40:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.211+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f614804f3a0 con 0x7f614806cb10 2024-03-06T07:40:01.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.212+0000 7f613e000640 1 -- 172.21.15.32:0/2931764881 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f613807aad0 con 0x7f614806cb10 2024-03-06T07:40:01.214 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:01.214 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:01.215 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f612806b4d0 msgr2=0x7f612806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f612806b4d0 0x7f612806d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6134006fd0 tx=0x7f6134008040 comp rx=0 tx=0).stop 2024-03-06T07:40:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 msgr2=0x7f6148157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 0x7f6148157390 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f613800b2c0 tx=0x7f61380046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 shutdown_connections 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f612806b4d0 0x7f612806d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61480cf5c0 0x7f61481f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f61480b3c90 0x7f6148154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 --2- 172.21.15.32:0/2931764881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614806cb10 0x7f6148157390 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 >> 172.21.15.32:0/2931764881 conn(0x7f614806afb0 msgr2=0x7f61481586e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 shutdown_connections 2024-03-06T07:40:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:01.215+0000 7f614ca00640 1 -- 172.21.15.32:0/2931764881 wait complete. 2024-03-06T07:40:01.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:01.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:01.480 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:02.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.661+0000 7fcbf4200640 1 Processor -- start 2024-03-06T07:40:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.661+0000 7fcbf4200640 1 -- start start 2024-03-06T07:40:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.663+0000 7fcbf4200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcbec06e240 0x7fcbec0ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.663+0000 7fcbf4200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0ce9a0 con 0x7fcbec071380 2024-03-06T07:40:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.663+0000 7fcbf4200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0cec80 con 0x7fcbec161670 2024-03-06T07:40:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.663+0000 7fcbf4200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbec0730d0 con 0x7fcbec06e240 2024-03-06T07:40:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.664+0000 7fcbf0e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcbec071380 0x7fcbec0ad090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54882/0 (socket says 172.21.15.32:54882) 2024-03-06T07:40:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.664+0000 7fcbf0e00640 1 -- 172.21.15.32:0/4152825797 learned_addr learned my addr 172.21.15.32:0/4152825797 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.664+0000 7fcbf2200640 1 --2- 172.21.15.32:0/4152825797 >> v2:172.21.15.39:3300/0 conn(0x7fcbec06e240 0x7fcbec0ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.664+0000 7fcbebe00640 1 -- 172.21.15.32:0/4152825797 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056681708 0 0) 0x7fcbec0ce9a0 con 0x7fcbec071380 2024-03-06T07:40:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.664+0000 7fcbebe00640 1 -- 172.21.15.32:0/4152825797 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4003520 con 0x7fcbec071380 2024-03-06T07:40:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.665+0000 7fcbf2200640 1 -- 172.21.15.32:0/4152825797 >> v1:172.21.15.39:6789/0 conn(0x7fcbec161670 legacy=0x7fcbec070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.665+0000 7fcbf2200640 1 -- 172.21.15.32:0/4152825797 >> v1:172.21.15.32:6789/0 conn(0x7fcbec071380 legacy=0x7fcbec0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.665+0000 7fcbebe00640 1 -- 172.21.15.32:0/4152825797 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2131623321 0 0) 0x7fcbd4003520 con 0x7fcbec071380 2024-03-06T07:40:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.665+0000 7fcbf2200640 1 -- 172.21.15.32:0/4152825797 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0cef60 con 0x7fcbec06e240 2024-03-06T07:40:02.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.668+0000 7fcbf2200640 1 --2- 172.21.15.32:0/4152825797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec06e240 0x7fcbec0ad7a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009dc0 tx=0x7fcbe402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b48f55718a3c3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:02.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.668+0000 7fcbebe00640 1 -- 172.21.15.32:0/4152825797 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4039070 con 0x7fcbec06e240 2024-03-06T07:40:02.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.668+0000 7fcbebe00640 1 -- 172.21.15.32:0/4152825797 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe402c8a0 con 0x7fcbec06e240 2024-03-06T07:40:02.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.668+0000 7fcbebe00640 1 -- 172.21.15.32:0/4152825797 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8001670 con 0x7fcbec06e240 2024-03-06T07:40:02.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 -- 172.21.15.32:0/4152825797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec06e240 msgr2=0x7fcbec0ad7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 --2- 172.21.15.32:0/4152825797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec06e240 0x7fcbec0ad7a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009dc0 tx=0x7fcbe402bd70 comp rx=0 tx=0).stop 2024-03-06T07:40:02.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 -- 172.21.15.32:0/4152825797 shutdown_connections 2024-03-06T07:40:02.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 --2- 172.21.15.32:0/4152825797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec06e240 0x7fcbec0ad7a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 -- 172.21.15.32:0/4152825797 >> 172.21.15.32:0/4152825797 conn(0x7fcbec06afb0 msgr2=0x7fcbec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:02.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 -- 172.21.15.32:0/4152825797 shutdown_connections 2024-03-06T07:40:02.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.669+0000 7fcbf4200640 1 -- 172.21.15.32:0/4152825797 wait complete. 2024-03-06T07:40:02.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.671+0000 7fcbf4200640 1 Processor -- start 2024-03-06T07:40:02.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.671+0000 7fcbf4200640 1 -- start start 2024-03-06T07:40:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf4200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcbec06e240 0x7fcbec0cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf4200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec071380 0x7fcbec0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf4200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 0x7fcbec150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf4200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0ce600 con 0x7fcbec161670 2024-03-06T07:40:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf4200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0cf520 con 0x7fcbec071380 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf4200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0ce780 con 0x7fcbec06e240 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 0x7fcbec150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 0x7fcbec150880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58104/0 (socket says 172.21.15.32:58104) 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.672+0000 7fcbf2200640 1 -- 172.21.15.32:0/2828639854 learned_addr learned my addr 172.21.15.32:0/2828639854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf1800640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcbec06e240 0x7fcbec0cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf0e00640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec071380 0x7fcbec0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf2200640 1 -- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcbec06e240 msgr2=0x7fcbec0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf2200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcbec06e240 0x7fcbec0cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf2200640 1 -- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec071380 msgr2=0x7fcbec0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf2200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec071380 0x7fcbec0ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf2200640 1 -- 172.21.15.32:0/2828639854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0cab10 con 0x7fcbec161670 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.673+0000 7fcbf2200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 0x7fcbec150880 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4037040 tx=0x7fcbe4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.674+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4044020 con 0x7fcbec161670 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.674+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe4034d90 con 0x7fcbec161670 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.674+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec0c9530 con 0x7fcbec161670 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.674+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4039040 con 0x7fcbec161670 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.674+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec0c9af0 con 0x7fcbec161670 2024-03-06T07:40:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.675+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec04f3a0 con 0x7fcbec161670 2024-03-06T07:40:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.676+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbe4005d10 con 0x7fcbec161670 2024-03-06T07:40:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.677+0000 7fcbe9600640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.678+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcbe40ac960 con 0x7fcbec161670 2024-03-06T07:40:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.678+0000 7fcbf1800640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:02.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.679+0000 7fcbf1800640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0000f60 tx=0x7fcbe0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.682+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbe404e070 con 0x7fcbec161670 2024-03-06T07:40:02.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.921+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbec150dc0 con 0x7fcbec161670 2024-03-06T07:40:02.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.922+0000 7fcbe9600640 1 -- 172.21.15.32:0/2828639854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fcbe407b700 con 0x7fcbec161670 2024-03-06T07:40:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:02.925 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:02.925 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:02.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcbcc06b4d0 msgr2=0x7fcbcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0000f60 tx=0x7fcbe0009040 comp rx=0 tx=0).stop 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 msgr2=0x7fcbec150880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 0x7fcbec150880 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4037040 tx=0x7fcbe4004a40 comp rx=0 tx=0).stop 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 shutdown_connections 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbec161670 0x7fcbec150880 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbec071380 0x7fcbec0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 --2- 172.21.15.32:0/2828639854 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fcbec06e240 0x7fcbec0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.925+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 >> 172.21.15.32:0/2828639854 conn(0x7fcbec06afb0 msgr2=0x7fcbec06e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.926+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 shutdown_connections 2024-03-06T07:40:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:02.926+0000 7fcbf4200640 1 -- 172.21.15.32:0/2828639854 wait complete. 2024-03-06T07:40:03.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:03.157 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:04.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.381+0000 7f0362600640 1 Processor -- start 2024-03-06T07:40:04.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.381+0000 7f0362600640 1 -- start start 2024-03-06T07:40:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.383+0000 7f0362600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f035c168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.383+0000 7f0362600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0b3710 con 0x7f035c15e0b0 2024-03-06T07:40:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.383+0000 7f0362600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0b39f0 con 0x7f035c0d4a10 2024-03-06T07:40:04.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.383+0000 7f0362600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0b09e0 con 0x7f035c0d0e50 2024-03-06T07:40:04.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.383+0000 7f0361200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f035c15e0b0 0x7f035c0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54884/0 (socket says 172.21.15.32:54884) 2024-03-06T07:40:04.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.383+0000 7f0361200640 1 -- 172.21.15.32:0/3172588446 learned_addr learned my addr 172.21.15.32:0/3172588446 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:04.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f0361c00640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f035c168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320993635 0 0) 0x7f035c0b3710 con 0x7f035c15e0b0 2024-03-06T07:40:04.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0340003520 con 0x7f035c15e0b0 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728747485 0 0) 0x7f0340003520 con 0x7f035c15e0b0 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 msgr2=0x7f035c168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f035c168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 >> v1:172.21.15.39:6789/0 conn(0x7f035c0d4a10 legacy=0x7f035c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0b3cd0 con 0x7f035c15e0b0 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f034c001690 con 0x7f035c15e0b0 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 >> v1:172.21.15.32:6789/0 conn(0x7f035c15e0b0 legacy=0x7f035c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f03400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340003f20 0x7f0340004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340004d40 0x7f03400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034000b9f0 con 0x7f0340004d40 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034000bba0 con 0x7f0340003f20 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.384+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034000bda0 con 0x7f035c0d0e50 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361c00640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340004d40 0x7f03400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f035be00640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340003f20 0x7f0340004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361200640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f03400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361c00640 1 -- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 msgr2=0x7f03400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361c00640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f03400039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361c00640 1 -- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340003f20 msgr2=0x7f0340004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361c00640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340003f20 0x7f0340004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361c00640 1 -- 172.21.15.32:0/3172588446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0340003520 con 0x7f0340004d40 2024-03-06T07:40:04.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.385+0000 7f0361200640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0d0e50 0x7f03400039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:04.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.386+0000 7f0361c00640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340004d40 0x7f03400051c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f0354009a90 tx=0x7f0340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2901c83143f59e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:04.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354039070 con 0x7f0340004d40 2024-03-06T07:40:04.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035402c680 con 0x7f0340004d40 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f035b400640 1 -- 172.21.15.32:0/3172588446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354009690 con 0x7f0340004d40 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0362600640 1 -- 172.21.15.32:0/3172588446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340004d40 msgr2=0x7f03400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0362600640 1 --2- 172.21.15.32:0/3172588446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340004d40 0x7f03400051c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f0354009a90 tx=0x7f0340005830 comp rx=0 tx=0).stop 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0361c00640 1 -- 172.21.15.32:0/3172588446 reap_dead start 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0362600640 1 -- 172.21.15.32:0/3172588446 shutdown_connections 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0362600640 1 -- 172.21.15.32:0/3172588446 >> 172.21.15.32:0/3172588446 conn(0x7f035c0d3d30 msgr2=0x7f035c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0362600640 1 -- 172.21.15.32:0/3172588446 shutdown_connections 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.387+0000 7f0362600640 1 -- 172.21.15.32:0/3172588446 wait complete. 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.389+0000 7f0362600640 1 Processor -- start 2024-03-06T07:40:04.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.389+0000 7f0362600640 1 -- start start 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.390+0000 7f0362600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340004d40 0x7f035c1612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.390+0000 7f0362600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0cae00 0x7f035c1f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.390+0000 7f0362600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 0x7f035c0b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.390+0000 7f0362600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c15e490 con 0x7f035c0b7a70 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.390+0000 7f0362600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0cfa20 con 0x7f0340004d40 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.390+0000 7f0362600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0cfc60 con 0x7f035c0cae00 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 0x7f035c0b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 0x7f035c0b7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58134/0 (socket says 172.21.15.32:58134) 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 -- 172.21.15.32:0/1380472325 learned_addr learned my addr 172.21.15.32:0/1380472325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361200640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340004d40 0x7f035c1612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f035be00640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0cae00 0x7f035c1f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 -- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0cae00 msgr2=0x7f035c1f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0cae00 0x7f035c1f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 -- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340004d40 msgr2=0x7f035c1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340004d40 0x7f035c1612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 -- 172.21.15.32:0/1380472325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0cd3f0 con 0x7f035c0b7a70 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.391+0000 7f0361c00640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 0x7f035c0b7ef0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f0354009bb0 tx=0x7f0354004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.392+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354044070 con 0x7f035c0b7a70 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.392+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035403a040 con 0x7f035c0b7a70 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.392+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354039070 con 0x7f035c0b7a70 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.392+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c0cd6d0 con 0x7f035c0b7a70 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.392+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035c0cdc90 con 0x7f035c0b7a70 2024-03-06T07:40:04.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.393+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c04f330 con 0x7f035c0b7a70 2024-03-06T07:40:04.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.394+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0354047070 con 0x7f035c0b7a70 2024-03-06T07:40:04.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.395+0000 7f0358c00640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f033c06b4d0 0x7f033c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:04.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.395+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f03540aac70 con 0x7f035c0b7a70 2024-03-06T07:40:04.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.395+0000 7f0361200640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f033c06b4d0 0x7f033c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:04.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.396+0000 7f0361200640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f033c06b4d0 0x7f033c06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f034c001800 tx=0x7f034c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:04.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.399+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03540793c0 con 0x7f035c0b7a70 2024-03-06T07:40:04.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.638+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f035c0b51d0 con 0x7f035c0b7a70 2024-03-06T07:40:04.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.639+0000 7f0358c00640 1 -- 172.21.15.32:0/1380472325 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f035407d460 con 0x7f035c0b7a70 2024-03-06T07:40:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:04.642 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:04.642 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f033c06b4d0 msgr2=0x7f033c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f033c06b4d0 0x7f033c06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f034c001800 tx=0x7f034c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 msgr2=0x7f035c0b7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 0x7f035c0b7ef0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f0354009bb0 tx=0x7f0354004730 comp rx=0 tx=0).stop 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 shutdown_connections 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f033c06b4d0 0x7f033c06d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c0b7a70 0x7f035c0b7ef0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f035c0cae00 0x7f035c1f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 --2- 172.21.15.32:0/1380472325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0340004d40 0x7f035c1612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.641+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 >> 172.21.15.32:0/1380472325 conn(0x7f035c0d3d30 msgr2=0x7f035c0b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.642+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 shutdown_connections 2024-03-06T07:40:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:04.642+0000 7f0362600640 1 -- 172.21.15.32:0/1380472325 wait complete. 2024-03-06T07:40:04.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:04.863 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:05.484 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:05.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:06.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.085+0000 7fb22bc00640 1 Processor -- start 2024-03-06T07:40:06.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.085+0000 7fb22bc00640 1 -- start start 2024-03-06T07:40:06.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb22bc00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2240cdc20 0x7fb22415f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb22bc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb2240722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb22bc00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2240ce0b0 con 0x7fb22415af80 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb22bc00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb224072820 con 0x7fb2240cdc20 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb22bc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240b4640 con 0x7fb224071e90 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb229200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb22415af80 0x7fb2240b4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54900/0 (socket says 172.21.15.32:54900) 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb229200640 1 -- 172.21.15.32:0/2412259401 learned_addr learned my addr 172.21.15.32:0/2412259401 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:06.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.087+0000 7fb21be00640 1 --2- 172.21.15.32:0/2412259401 >> v2:172.21.15.32:3300/0 conn(0x7fb2240cdc20 0x7fb22415f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb229c00640 1 --2- 172.21.15.32:0/2412259401 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb2240722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb223e00640 1 -- 172.21.15.32:0/2412259401 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935886438 0 0) 0x7fb2240ce0b0 con 0x7fb22415af80 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb223e00640 1 -- 172.21.15.32:0/2412259401 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb208003520 con 0x7fb22415af80 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb21be00640 1 -- 172.21.15.32:0/2412259401 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 msgr2=0x7fb2240722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb21be00640 1 --2- 172.21.15.32:0/2412259401 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb2240722e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb21be00640 1 -- 172.21.15.32:0/2412259401 >> v1:172.21.15.32:6789/0 conn(0x7fb22415af80 legacy=0x7fb2240b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.088+0000 7fb21be00640 1 -- 172.21.15.32:0/2412259401 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2240b3ff0 con 0x7fb2240cdc20 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.089+0000 7fb21be00640 1 --2- 172.21.15.32:0/2412259401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2240cdc20 0x7fb22415f560 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0097a0 tx=0x7fb20c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fac384fa8b4f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb223e00640 1 -- 172.21.15.32:0/2412259401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0315a0 con 0x7fb2240cdc20 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb223e00640 1 -- 172.21.15.32:0/2412259401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c037070 con 0x7fb2240cdc20 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb223e00640 1 -- 172.21.15.32:0/2412259401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c031940 con 0x7fb2240cdc20 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 -- 172.21.15.32:0/2412259401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2240cdc20 msgr2=0x7fb22415f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2412259401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2240cdc20 0x7fb22415f560 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0097a0 tx=0x7fb20c02b840 comp rx=0 tx=0).stop 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 -- 172.21.15.32:0/2412259401 shutdown_connections 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2412259401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2240cdc20 0x7fb22415f560 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2412259401 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb2240722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 -- 172.21.15.32:0/2412259401 >> 172.21.15.32:0/2412259401 conn(0x7fb22406afb0 msgr2=0x7fb22406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 -- 172.21.15.32:0/2412259401 shutdown_connections 2024-03-06T07:40:06.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.090+0000 7fb22bc00640 1 -- 172.21.15.32:0/2412259401 wait complete. 2024-03-06T07:40:06.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.092+0000 7fb22bc00640 1 Processor -- start 2024-03-06T07:40:06.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.092+0000 7fb22bc00640 1 -- start start 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb22bc00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb224150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb22bc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 0x7fb2241e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb22bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb22406d3b0 0x7fb22406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb22bc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240cf780 con 0x7fb22415af80 2024-03-06T07:40:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb22bc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240b3cc0 con 0x7fb22406d3b0 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb22bc00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22415b360 con 0x7fb224071e90 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.093+0000 7fb21be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 0x7fb2241e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 0x7fb2241e83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58142/0 (socket says 172.21.15.32:58142) 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 -- 172.21.15.32:0/2957022545 learned_addr learned my addr 172.21.15.32:0/2957022545 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb229200640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb224150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb229c00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb22406d3b0 0x7fb22406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 -- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 msgr2=0x7fb224150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb224150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 -- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb22406d3b0 msgr2=0x7fb22406d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb22406d3b0 0x7fb22406d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 -- 172.21.15.32:0/2957022545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2241e7340 con 0x7fb22415af80 2024-03-06T07:40:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb229200640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb224150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:06.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.094+0000 7fb21be00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 0x7fb2241e83e0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fb20c034040 tx=0x7fb20c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:06.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.095+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c032430 con 0x7fb22415af80 2024-03-06T07:40:06.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.095+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c0325d0 con 0x7fb22415af80 2024-03-06T07:40:06.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.095+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c032980 con 0x7fb22415af80 2024-03-06T07:40:06.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.095+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2241e7620 con 0x7fb22415af80 2024-03-06T07:40:06.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.095+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2241e7be0 con 0x7fb22415af80 2024-03-06T07:40:06.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.096+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb224069510 con 0x7fb22415af80 2024-03-06T07:40:06.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.097+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb20c032b20 con 0x7fb22415af80 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.098+0000 7fb221600640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb20006b4d0 0x7fb20006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.098+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb20c02f030 con 0x7fb22415af80 2024-03-06T07:40:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.098+0000 7fb229200640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb20006b4d0 0x7fb20006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:06.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.099+0000 7fb229200640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb20006b4d0 0x7fb20006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb2140012e0 tx=0x7fb214009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:06.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.103+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb20c040070 con 0x7fb22415af80 2024-03-06T07:40:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.342+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2240cc590 con 0x7fb22415af80 2024-03-06T07:40:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.342+0000 7fb221600640 1 -- 172.21.15.32:0/2957022545 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fb20c07a810 con 0x7fb22415af80 2024-03-06T07:40:06.345 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:06.345 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.346+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb20006b4d0 msgr2=0x7fb20006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.346+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb20006b4d0 0x7fb20006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb2140012e0 tx=0x7fb214009040 comp rx=0 tx=0).stop 2024-03-06T07:40:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 msgr2=0x7fb2241e83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 0x7fb2241e83e0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fb20c034040 tx=0x7fb20c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 shutdown_connections 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fb20006b4d0 0x7fb20006d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb22406d3b0 0x7fb22406d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22415af80 0x7fb2241e83e0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 --2- 172.21.15.32:0/2957022545 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fb224071e90 0x7fb224150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 >> 172.21.15.32:0/2957022545 conn(0x7fb22406afb0 msgr2=0x7fb2240b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 shutdown_connections 2024-03-06T07:40:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:06.347+0000 7fb22bc00640 1 -- 172.21.15.32:0/2957022545 wait complete. 2024-03-06T07:40:06.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:06.570 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:07.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:07.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.784+0000 7fc8ed200640 1 Processor -- start 2024-03-06T07:40:07.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.784+0000 7fc8ed200640 1 -- start start 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.785+0000 7fc8ed200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc8e80d04b0 0x7fc8e8070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.785+0000 7fc8ed200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8e80700c0 0x7fc8e80cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.785+0000 7fc8ed200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e80cd0c0 con 0x7fc8e815f450 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.785+0000 7fc8ed200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e80d29e0 con 0x7fc8e80700c0 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.785+0000 7fc8ed200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e80cd3a0 con 0x7fc8e80d04b0 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8e80700c0 0x7fc8e80cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8e80700c0 0x7fc8e80cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58146/0 (socket says 172.21.15.32:58146) 2024-03-06T07:40:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 -- 172.21.15.32:0/1203418920 learned_addr learned my addr 172.21.15.32:0/1203418920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8dea00640 1 --2- 172.21.15.32:0/1203418920 >> v2:172.21.15.39:3300/0 conn(0x7fc8e80d04b0 0x7fc8e8070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 -- 172.21.15.32:0/1203418920 >> v2:172.21.15.39:3300/0 conn(0x7fc8e80d04b0 msgr2=0x7fc8e8070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 --2- 172.21.15.32:0/1203418920 >> v2:172.21.15.39:3300/0 conn(0x7fc8e80d04b0 0x7fc8e8070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 -- 172.21.15.32:0/1203418920 >> v1:172.21.15.39:6789/0 conn(0x7fc8e815f450 legacy=0x7fc8e80cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.786+0000 7fc8e7e00640 1 -- 172.21.15.32:0/1203418920 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e80cd680 con 0x7fc8e80700c0 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.787+0000 7fc8e7e00640 1 --2- 172.21.15.32:0/1203418920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80700c0 0x7fc8e80cde80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc8e806b9c0 tx=0x7fc8e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fa33da80dc8a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.788+0000 7fc8e6a00640 1 -- 172.21.15.32:0/1203418920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e002ca80 con 0x7fc8e80700c0 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.788+0000 7fc8e6a00640 1 -- 172.21.15.32:0/1203418920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e002cc20 con 0x7fc8e80700c0 2024-03-06T07:40:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.788+0000 7fc8e6a00640 1 -- 172.21.15.32:0/1203418920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e00343f0 con 0x7fc8e80700c0 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.788+0000 7fc8ed200640 1 -- 172.21.15.32:0/1203418920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80700c0 msgr2=0x7fc8e80cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.788+0000 7fc8ed200640 1 --2- 172.21.15.32:0/1203418920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80700c0 0x7fc8e80cde80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc8e806b9c0 tx=0x7fc8e002bee0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.789+0000 7fc8ed200640 1 -- 172.21.15.32:0/1203418920 shutdown_connections 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.789+0000 7fc8ed200640 1 --2- 172.21.15.32:0/1203418920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80700c0 0x7fc8e80cde80 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.789+0000 7fc8ed200640 1 --2- 172.21.15.32:0/1203418920 >> v2:172.21.15.39:3300/0 conn(0x7fc8e80d04b0 0x7fc8e8070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.789+0000 7fc8ed200640 1 -- 172.21.15.32:0/1203418920 >> 172.21.15.32:0/1203418920 conn(0x7fc8e80d1ae0 msgr2=0x7fc8e80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.789+0000 7fc8ed200640 1 -- 172.21.15.32:0/1203418920 shutdown_connections 2024-03-06T07:40:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.789+0000 7fc8ed200640 1 -- 172.21.15.32:0/1203418920 wait complete. 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.791+0000 7fc8ed200640 1 Processor -- start 2024-03-06T07:40:07.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.791+0000 7fc8ed200640 1 -- start start 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8ed200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc8e80700c0 0x7fc8e81580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8ed200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 0x7fc8e80b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8ed200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8e815f450 0x7fc8e815c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8ed200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e815f830 con 0x7fc8e80d04b0 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8ed200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e80ae8b0 con 0x7fc8e815f450 2024-03-06T07:40:07.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8ed200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e815f9b0 con 0x7fc8e80700c0 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 0x7fc8e80b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 0x7fc8e80b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58156/0 (socket says 172.21.15.32:58156) 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8dea00640 1 -- 172.21.15.32:0/57109320 learned_addr learned my addr 172.21.15.32:0/57109320 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8e7e00640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8e815f450 0x7fc8e815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.793+0000 7fc8e7400640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc8e80700c0 0x7fc8e81580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dea00640 1 -- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc8e80700c0 msgr2=0x7fc8e81580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dea00640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc8e80700c0 0x7fc8e81580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dea00640 1 -- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8e815f450 msgr2=0x7fc8e815c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dea00640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8e815f450 0x7fc8e815c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dea00640 1 -- 172.21.15.32:0/57109320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e80b0a60 con 0x7fc8e80d04b0 2024-03-06T07:40:07.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dea00640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 0x7fc8e80b0010 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000d800 tx=0x7fc8d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000b6c0 con 0x7fc8e80d04b0 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d0019070 con 0x7fc8e80d04b0 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.794+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e80c8d50 con 0x7fc8e80d04b0 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.795+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000ba80 con 0x7fc8e80d04b0 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.795+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e80c9310 con 0x7fc8e80d04b0 2024-03-06T07:40:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.796+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e804f330 con 0x7fc8e80d04b0 2024-03-06T07:40:07.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.797+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8d000bc20 con 0x7fc8e80d04b0 2024-03-06T07:40:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.798+0000 7fc8dfe00640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc8c406b500 0x7fc8c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.799+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc8d008a740 con 0x7fc8e80d04b0 2024-03-06T07:40:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.799+0000 7fc8e7400640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc8c406b500 0x7fc8c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.799+0000 7fc8e7400640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc8c406b500 0x7fc8c406d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8006fd0 tx=0x7fc8d8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:07.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:07.802+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8d001c360 con 0x7fc8e80d04b0 2024-03-06T07:40:08.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.057+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8e815dd30 con 0x7fc8e80d04b0 2024-03-06T07:40:08.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.058+0000 7fc8dfe00640 1 -- 172.21.15.32:0/57109320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fc8d00594e0 con 0x7fc8e80d04b0 2024-03-06T07:40:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:08.060 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc8c406b500 msgr2=0x7fc8c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc8c406b500 0x7fc8c406d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8006fd0 tx=0x7fc8d8008040 comp rx=0 tx=0).stop 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 msgr2=0x7fc8e80b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 0x7fc8e80b0010 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000d800 tx=0x7fc8d000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 shutdown_connections 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc8c406b500 0x7fc8c406d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8e815f450 0x7fc8e815c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8e80d04b0 0x7fc8e80b0010 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 --2- 172.21.15.32:0/57109320 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc8e80700c0 0x7fc8e81580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 >> 172.21.15.32:0/57109320 conn(0x7fc8e80d1ae0 msgr2=0x7fc8e815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 shutdown_connections 2024-03-06T07:40:08.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:08.061+0000 7fc8ed200640 1 -- 172.21.15.32:0/57109320 wait complete. 2024-03-06T07:40:08.283 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:08.283 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:09.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:09.488 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.510+0000 7fae06800640 1 Processor -- start 2024-03-06T07:40:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.510+0000 7fae06800640 1 -- start start 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.511+0000 7fae06800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fae000cd060 0x7fae0015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.511+0000 7fae06800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fae000b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.511+0000 7fae06800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0015ed10 con 0x7fae001552b0 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.511+0000 7fae06800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0015eff0 con 0x7fae000cd060 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.511+0000 7fae06800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0015f2d0 con 0x7fae0015e810 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fae05400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fae001552b0 0x7fae000cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53592/0 (socket says 172.21.15.32:53592) 2024-03-06T07:40:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fae05400640 1 -- 172.21.15.32:0/2935857109 learned_addr learned my addr 172.21.15.32:0/2935857109 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fae05e00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fae000b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fae04a00640 1 --2- 172.21.15.32:0/2935857109 >> v2:172.21.15.39:3300/0 conn(0x7fae000cd060 0x7fae0015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293101360 0 0) 0x7fae0015ed10 con 0x7fae001552b0 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade4003520 con 0x7fae001552b0 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.512+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911747877 0 0) 0x7fade4003520 con 0x7fae001552b0 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 msgr2=0x7fae000b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fae000b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 >> v2:172.21.15.39:3300/0 conn(0x7fae000cd060 msgr2=0x7fae0015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 --2- 172.21.15.32:0/2935857109 >> v2:172.21.15.39:3300/0 conn(0x7fae000cd060 0x7fae0015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0015f5b0 con 0x7fae001552b0 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fadf0001a10 con 0x7fae001552b0 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 >> v1:172.21.15.32:6789/0 conn(0x7fae001552b0 legacy=0x7fae000cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade40039b0 0x7fade4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fade4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade4004d40 0x7fade40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0015eff0 con 0x7fade4004d40 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0015f2d0 con 0x7fade40039b0 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade400dc00 con 0x7fae0015e810 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fae05e00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade4004d40 0x7fade40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fae05400640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade40039b0 0x7fade4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.513+0000 7fae04a00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fade4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae05e00640 1 -- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 msgr2=0x7fade4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae05e00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fade4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae05e00640 1 -- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade40039b0 msgr2=0x7fade4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae05e00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade40039b0 0x7fade4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae05e00640 1 -- 172.21.15.32:0/2935857109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade4003520 con 0x7fade4004d40 2024-03-06T07:40:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae05400640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade40039b0 0x7fade4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.514+0000 7fae04a00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0015e810 0x7fade4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.515+0000 7fae05e00640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade4004d40 0x7fade40051c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fae0015ec40 tx=0x7fade4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16adb7197329f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8039070 con 0x7fade4004d40 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf802c660 con 0x7fade4004d40 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fadffe00640 1 -- 172.21.15.32:0/2935857109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8009690 con 0x7fade4004d40 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fae06800640 1 -- 172.21.15.32:0/2935857109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade4004d40 msgr2=0x7fade40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fae06800640 1 --2- 172.21.15.32:0/2935857109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fade4004d40 0x7fade40051c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fae0015ec40 tx=0x7fade4005830 comp rx=0 tx=0).stop 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fae05e00640 1 -- 172.21.15.32:0/2935857109 reap_dead start 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fae06800640 1 -- 172.21.15.32:0/2935857109 shutdown_connections 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.516+0000 7fae06800640 1 -- 172.21.15.32:0/2935857109 >> 172.21.15.32:0/2935857109 conn(0x7fae0006b3d0 msgr2=0x7fae0006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.517+0000 7fae06800640 1 -- 172.21.15.32:0/2935857109 shutdown_connections 2024-03-06T07:40:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.517+0000 7fae06800640 1 -- 172.21.15.32:0/2935857109 wait complete. 2024-03-06T07:40:09.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.518+0000 7fae06800640 1 Processor -- start 2024-03-06T07:40:09.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.519+0000 7fae06800640 1 -- start start 2024-03-06T07:40:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae06800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0014d8f0 0x7fae0014b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae06800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 0x7fae001f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae06800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae0014dcd0 0x7fae001f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae06800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0006dd30 con 0x7fae0015a4d0 2024-03-06T07:40:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae06800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae000b3760 con 0x7fae0014dcd0 2024-03-06T07:40:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae06800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0006deb0 con 0x7fae0014d8f0 2024-03-06T07:40:09.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 0x7fae001f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 0x7fae001f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48850/0 (socket says 172.21.15.32:48850) 2024-03-06T07:40:09.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 -- 172.21.15.32:0/3120004352 learned_addr learned my addr 172.21.15.32:0/3120004352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae05e00640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae0014dcd0 0x7fae001f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae05400640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0014d8f0 0x7fae0014b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 -- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0014d8f0 msgr2=0x7fae0014b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0014d8f0 0x7fae0014b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 -- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae0014dcd0 msgr2=0x7fae001f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae0014dcd0 0x7fae001f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae04a00640 1 -- 172.21.15.32:0/3120004352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000b08b0 con 0x7fae0015a4d0 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.520+0000 7fae05e00640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae0014dcd0 0x7fae001f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.521+0000 7fae04a00640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 0x7fae001f3580 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fadec0109d0 tx=0x7fadec010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.521+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec00f040 con 0x7fae0015a4d0 2024-03-06T07:40:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.521+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec00ec20 con 0x7fae0015a4d0 2024-03-06T07:40:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.521+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec01c400 con 0x7fae0015a4d0 2024-03-06T07:40:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.521+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0014be60 con 0x7fae0015a4d0 2024-03-06T07:40:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.521+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae000b0e50 con 0x7fae0015a4d0 2024-03-06T07:40:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.522+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae00072af0 con 0x7fae0015a4d0 2024-03-06T07:40:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.523+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadec01c5a0 con 0x7fae0015a4d0 2024-03-06T07:40:09.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.526+0000 7fadfd600640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fade006b410 0x7fade006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:09.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.526+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fadec08aab0 con 0x7fae0015a4d0 2024-03-06T07:40:09.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.526+0000 7fae05400640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fade006b410 0x7fade006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:09.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.526+0000 7fae05400640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fade006b410 0x7fade006d8d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fadf0001350 tx=0x7fadf0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.532+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae00072af0 con 0x7fae0015a4d0 2024-03-06T07:40:09.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.777+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae0004f330 con 0x7fae0015a4d0 2024-03-06T07:40:09.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.777+0000 7fadfd600640 1 -- 172.21.15.32:0/3120004352 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fadec018030 con 0x7fae0015a4d0 2024-03-06T07:40:09.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:09.780 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:09.780 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:09.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fade006b410 msgr2=0x7fade006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fade006b410 0x7fade006d8d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fadf0001350 tx=0x7fadf0009040 comp rx=0 tx=0).stop 2024-03-06T07:40:09.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 msgr2=0x7fae001f3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:09.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 0x7fae001f3580 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fadec0109d0 tx=0x7fadec010e90 comp rx=0 tx=0).stop 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 shutdown_connections 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fade006b410 0x7fade006d8d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae0014dcd0 0x7fae001f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae0015a4d0 0x7fae001f3580 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 --2- 172.21.15.32:0/3120004352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fae0014d8f0 0x7fae0014b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 >> 172.21.15.32:0/3120004352 conn(0x7fae0006b3d0 msgr2=0x7fae000cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 shutdown_connections 2024-03-06T07:40:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:09.780+0000 7fae06800640 1 -- 172.21.15.32:0/3120004352 wait complete. 2024-03-06T07:40:10.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:10.010 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:11.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:11.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.234+0000 7f0f7e400640 1 Processor -- start 2024-03-06T07:40:11.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.234+0000 7f0f7e400640 1 -- start start 2024-03-06T07:40:11.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.236+0000 7f0f7e400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0f780cd760 0x7f0f7815d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.236+0000 7f0f7e400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f7815d970 0x7f0f7806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.236+0000 7f0f7e400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f78159cc0 con 0x7f0f781598e0 2024-03-06T07:40:11.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.236+0000 7f0f7e400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7806e3f0 con 0x7f0f780cd760 2024-03-06T07:40:11.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.236+0000 7f0f7e400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780729b0 con 0x7f0f7815d970 2024-03-06T07:40:11.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f7da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f7815d970 0x7f0f7806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f77e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0f780cd760 0x7f0f7815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f7da00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f7815d970 0x7f0f7806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:51050/0 (socket says 172.21.15.32:51050) 2024-03-06T07:40:11.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f7da00640 1 -- 172.21.15.32:0/1205047578 learned_addr learned my addr 172.21.15.32:0/1205047578 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:11.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f77e00640 1 -- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f7815d970 msgr2=0x7f0f7806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:11.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f77e00640 1 --2- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f7815d970 0x7f0f7806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f77e00640 1 -- 172.21.15.32:0/1205047578 >> v1:172.21.15.39:6789/0 conn(0x7f0f781598e0 legacy=0x7f0f7806f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.237+0000 7f0f77e00640 1 -- 172.21.15.32:0/1205047578 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f78072c50 con 0x7f0f780cd760 2024-03-06T07:40:11.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.239+0000 7f0f77e00640 1 --2- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f780cd760 0x7f0f7815d430 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0f640097a0 tx=0x7f0f6402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61761af21bef4c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:11.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.239+0000 7f0f77400640 1 -- 172.21.15.32:0/1205047578 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64035020 con 0x7f0f780cd760 2024-03-06T07:40:11.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.239+0000 7f0f77400640 1 -- 172.21.15.32:0/1205047578 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64031830 con 0x7f0f780cd760 2024-03-06T07:40:11.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f77400640 1 -- 172.21.15.32:0/1205047578 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64031c00 con 0x7f0f780cd760 2024-03-06T07:40:11.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 -- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f780cd760 msgr2=0x7f0f7815d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f780cd760 0x7f0f7815d430 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0f640097a0 tx=0x7f0f6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 -- 172.21.15.32:0/1205047578 shutdown_connections 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f7815d970 0x7f0f7806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1205047578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f780cd760 0x7f0f7815d430 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 -- 172.21.15.32:0/1205047578 >> 172.21.15.32:0/1205047578 conn(0x7f0f7806c0c0 msgr2=0x7f0f7806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 -- 172.21.15.32:0/1205047578 shutdown_connections 2024-03-06T07:40:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.240+0000 7f0f7e400640 1 -- 172.21.15.32:0/1205047578 wait complete. 2024-03-06T07:40:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.242+0000 7f0f7e400640 1 Processor -- start 2024-03-06T07:40:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.242+0000 7f0f7e400640 1 -- start start 2024-03-06T07:40:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f7e400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f780cd760 0x7f0f78149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f7e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 0x7f0f780b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f7e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f7815d970 0x7f0f780b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f7e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7806e080 con 0x7f0f781598e0 2024-03-06T07:40:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f7e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7806dda0 con 0x7f0f7815d970 2024-03-06T07:40:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f7e400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7806b4f0 con 0x7f0f780cd760 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.243+0000 7f0f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 0x7f0f780b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 0x7f0f780b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48854/0 (socket says 172.21.15.32:48854) 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 -- 172.21.15.32:0/1362764216 learned_addr learned my addr 172.21.15.32:0/1362764216 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f7d000640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f780cd760 0x7f0f78149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f7da00640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f7815d970 0x7f0f780b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 -- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f780cd760 msgr2=0x7f0f78149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f780cd760 0x7f0f78149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 -- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f7815d970 msgr2=0x7f0f780b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f7815d970 0x7f0f780b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f77e00640 1 -- 172.21.15.32:0/1362764216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780b0ea0 con 0x7f0f781598e0 2024-03-06T07:40:11.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.244+0000 7f0f7da00640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f7815d970 0x7f0f780b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.245+0000 7f0f77e00640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 0x7f0f780b0670 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009890 tx=0x7f0f640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.245+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64035040 con 0x7f0f781598e0 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.245+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6403b920 con 0x7f0f781598e0 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.245+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6403bd10 con 0x7f0f781598e0 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.245+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f780b1180 con 0x7f0f781598e0 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.245+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f780b0bb0 con 0x7f0f781598e0 2024-03-06T07:40:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.246+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7804f330 con 0x7f0f781598e0 2024-03-06T07:40:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.247+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f64038020 con 0x7f0f781598e0 2024-03-06T07:40:11.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.248+0000 7f0f74c00640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f5806b520 0x7f0f5806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:11.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.248+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f640ac300 con 0x7f0f781598e0 2024-03-06T07:40:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.249+0000 7f0f7d000640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f5806b520 0x7f0f5806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.250+0000 7f0f7d000640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f5806b520 0x7f0f5806d9e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c001640 tx=0x7f0f6c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:11.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.253+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f6407b1d0 con 0x7f0f781598e0 2024-03-06T07:40:11.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.484+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f780d10e0 con 0x7f0f781598e0 2024-03-06T07:40:11.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.484+0000 7f0f74c00640 1 -- 172.21.15.32:0/1362764216 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f0f64031800 con 0x7f0f781598e0 2024-03-06T07:40:11.486 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:11.486 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:11.486 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:11.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.487+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f5806b520 msgr2=0x7f0f5806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.487+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f5806b520 0x7f0f5806d9e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c001640 tx=0x7f0f6c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:11.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.487+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 msgr2=0x7f0f780b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:11.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.487+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 0x7f0f780b0670 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009890 tx=0x7f0f640046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 shutdown_connections 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0f5806b520 0x7f0f5806d9e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f7815d970 0x7f0f780b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f781598e0 0x7f0f780b0670 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 --2- 172.21.15.32:0/1362764216 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0f780cd760 0x7f0f78149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 >> 172.21.15.32:0/1362764216 conn(0x7f0f7806c0c0 msgr2=0x7f0f7806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 shutdown_connections 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:11.488+0000 7f0f7e400640 1 -- 172.21.15.32:0/1362764216 wait complete. 2024-03-06T07:40:11.490 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:11.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:11.715 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:12.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:12.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.931+0000 7f0ca4000640 1 Processor -- start 2024-03-06T07:40:12.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.932+0000 7f0ca4000640 1 -- start start 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.933+0000 7f0ca4000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af1a0 0x7f0c9c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.933+0000 7f0ca4000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c9c0af580 0x7f0c9c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.933+0000 7f0ca4000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9c15cc40 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.933+0000 7f0ca4000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9c15cf20 con 0x7f0c9c0af580 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.933+0000 7f0ca4000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c15d200 con 0x7f0c9c0af1a0 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c9c0af580 0x7f0c9c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c9c0af580 0x7f0c9c0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48860/0 (socket says 172.21.15.32:48860) 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 -- 172.21.15.32:0/2911363633 learned_addr learned my addr 172.21.15.32:0/2911363633 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca1600640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af1a0 0x7f0c9c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 -- 172.21.15.32:0/2911363633 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af1a0 msgr2=0x7f0c9c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af1a0 0x7f0c9c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 -- 172.21.15.32:0/2911363633 >> v1:172.21.15.39:6789/0 conn(0x7f0c9c0cdc20 legacy=0x7f0c9c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca0c00640 1 -- 172.21.15.32:0/2911363633 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c15d4e0 con 0x7f0c9c0af580 2024-03-06T07:40:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.934+0000 7f0ca1600640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af1a0 0x7f0c9c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.935+0000 7f0ca0c00640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0af580 0x7f0c9c0b4c20 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009870 tx=0x7f0c8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393d32ea5cbf754c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.936+0000 7f0c9be00640 1 -- 172.21.15.32:0/2911363633 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880317c0 con 0x7f0c9c0af580 2024-03-06T07:40:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.936+0000 7f0c9be00640 1 -- 172.21.15.32:0/2911363633 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88037070 con 0x7f0c9c0af580 2024-03-06T07:40:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.936+0000 7f0c9be00640 1 -- 172.21.15.32:0/2911363633 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88031b80 con 0x7f0c9c0af580 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.936+0000 7f0ca4000640 1 -- 172.21.15.32:0/2911363633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0af580 msgr2=0x7f0c9c0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.936+0000 7f0ca4000640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0af580 0x7f0c9c0b4c20 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009870 tx=0x7f0c8802b860 comp rx=0 tx=0).stop 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.937+0000 7f0ca4000640 1 -- 172.21.15.32:0/2911363633 shutdown_connections 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.937+0000 7f0ca4000640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0af580 0x7f0c9c0b4c20 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.937+0000 7f0ca4000640 1 --2- 172.21.15.32:0/2911363633 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af1a0 0x7f0c9c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.937+0000 7f0ca4000640 1 -- 172.21.15.32:0/2911363633 >> 172.21.15.32:0/2911363633 conn(0x7f0c9c06afb0 msgr2=0x7f0c9c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.937+0000 7f0ca4000640 1 -- 172.21.15.32:0/2911363633 shutdown_connections 2024-03-06T07:40:12.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.937+0000 7f0ca4000640 1 -- 172.21.15.32:0/2911363633 wait complete. 2024-03-06T07:40:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.938+0000 7f0ca4000640 1 Processor -- start 2024-03-06T07:40:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.939+0000 7f0ca4000640 1 -- start start 2024-03-06T07:40:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca4000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9c0af1a0 0x7f0c9c14afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca4000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af580 0x7f0c9c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 0x7f0c9c15eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca4000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c0b3660 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca4000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c0b34e0 con 0x7f0c9c0af1a0 2024-03-06T07:40:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca4000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c0b37e0 con 0x7f0c9c0af580 2024-03-06T07:40:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 0x7f0c9c15eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 0x7f0c9c15eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48872/0 (socket says 172.21.15.32:48872) 2024-03-06T07:40:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 -- 172.21.15.32:0/1873901980 learned_addr learned my addr 172.21.15.32:0/1873901980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca1600640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9c0af1a0 0x7f0c9c14afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca0c00640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af580 0x7f0c9c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 -- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af580 msgr2=0x7f0c9c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af580 0x7f0c9c14d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 -- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9c0af1a0 msgr2=0x7f0c9c14afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9c0af1a0 0x7f0c9c14afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.940+0000 7f0ca2000640 1 -- 172.21.15.32:0/1873901980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c14e310 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0ca1600640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9c0af1a0 0x7f0c9c14afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0ca0c00640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af580 0x7f0c9c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0ca2000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 0x7f0c9c15eb40 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400b9a0 tx=0x7f0c9400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94019070 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9400cba0 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94014440 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c14e5f0 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.941+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c09c980 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.942+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9c069510 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.943+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c9401c070 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.944+0000 7f0c99600640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:12.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.944+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0c94089f20 con 0x7f0c9c0cdc20 2024-03-06T07:40:12.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.945+0000 7f0ca1600640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:12.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.945+0000 7f0ca1600640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0c90006fd0 tx=0x7f0c90008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:12.949+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c94058cc0 con 0x7f0c9c0cdc20 2024-03-06T07:40:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.192+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c9c071d80 con 0x7f0c9c0cdc20 2024-03-06T07:40:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.192+0000 7f0c99600640 1 -- 172.21.15.32:0/1873901980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f0c940585c0 con 0x7f0c9c0cdc20 2024-03-06T07:40:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:13.194 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c7c06b4d0 msgr2=0x7f0c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0c90006fd0 tx=0x7f0c90008040 comp rx=0 tx=0).stop 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 msgr2=0x7f0c9c15eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 0x7f0c9c15eb40 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400b9a0 tx=0x7f0c9400be60 comp rx=0 tx=0).stop 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 shutdown_connections 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0cdc20 0x7f0c9c15eb40 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0c9c0af580 0x7f0c9c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 --2- 172.21.15.32:0/1873901980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9c0af1a0 0x7f0c9c14afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 >> 172.21.15.32:0/1873901980 conn(0x7f0c9c06afb0 msgr2=0x7f0c9c0b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 shutdown_connections 2024-03-06T07:40:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:13.195+0000 7f0ca4000640 1 -- 172.21.15.32:0/1873901980 wait complete. 2024-03-06T07:40:13.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:13.416 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:14.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:14.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.665+0000 7f0e02000640 1 Processor -- start 2024-03-06T07:40:14.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.666+0000 7f0e02000640 1 -- start start 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.667+0000 7f0e02000640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dfc070f10 0x7f0dfc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.667+0000 7f0e02000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0dfc0cce90 0x7f0dfc15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.667+0000 7f0e02000640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc0714d0 con 0x7f0dfc145300 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.667+0000 7f0e02000640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc0cd300 con 0x7f0dfc070f10 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.667+0000 7f0e02000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc0717b0 con 0x7f0dfc0cce90 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dfc070f10 0x7f0dfc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dfc070f10 0x7f0dfc0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48878/0 (socket says 172.21.15.32:48878) 2024-03-06T07:40:14.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 -- 172.21.15.32:0/3955434320 learned_addr learned my addr 172.21.15.32:0/3955434320 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0e01600640 1 --2- 172.21.15.32:0/3955434320 >> v2:172.21.15.39:3300/0 conn(0x7f0dfc0cce90 0x7f0dfc15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 -- 172.21.15.32:0/3955434320 >> v2:172.21.15.39:3300/0 conn(0x7f0dfc0cce90 msgr2=0x7f0dfc15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 --2- 172.21.15.32:0/3955434320 >> v2:172.21.15.39:3300/0 conn(0x7f0dfc0cce90 0x7f0dfc15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 -- 172.21.15.32:0/3955434320 >> v1:172.21.15.32:6789/0 conn(0x7f0dfc145300 legacy=0x7f0dfc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0dfbe00640 1 -- 172.21.15.32:0/3955434320 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc071a90 con 0x7f0dfc070f10 2024-03-06T07:40:14.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.668+0000 7f0e01600640 1 --2- 172.21.15.32:0/3955434320 >> v2:172.21.15.39:3300/0 conn(0x7f0dfc0cce90 0x7f0dfc15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.670+0000 7f0dfbe00640 1 --2- 172.21.15.32:0/3955434320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc0cc950 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0cc500 tx=0x7f0dec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88147d71d21fae6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.670+0000 7f0dfb400640 1 -- 172.21.15.32:0/3955434320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec02ca80 con 0x7f0dfc070f10 2024-03-06T07:40:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.670+0000 7f0dfb400640 1 -- 172.21.15.32:0/3955434320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec02cc20 con 0x7f0dfc070f10 2024-03-06T07:40:14.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.670+0000 7f0dfb400640 1 -- 172.21.15.32:0/3955434320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0343f0 con 0x7f0dfc070f10 2024-03-06T07:40:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.670+0000 7f0e02000640 1 -- 172.21.15.32:0/3955434320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 msgr2=0x7f0dfc0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.670+0000 7f0e02000640 1 --2- 172.21.15.32:0/3955434320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc0cc950 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0cc500 tx=0x7f0dec02bee0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.671+0000 7f0e02000640 1 -- 172.21.15.32:0/3955434320 shutdown_connections 2024-03-06T07:40:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.671+0000 7f0e02000640 1 --2- 172.21.15.32:0/3955434320 >> v2:172.21.15.39:3300/0 conn(0x7f0dfc0cce90 0x7f0dfc15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.671+0000 7f0e02000640 1 --2- 172.21.15.32:0/3955434320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc0cc950 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.671+0000 7f0e02000640 1 -- 172.21.15.32:0/3955434320 >> 172.21.15.32:0/3955434320 conn(0x7f0dfc06b3d0 msgr2=0x7f0dfc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.671+0000 7f0e02000640 1 -- 172.21.15.32:0/3955434320 shutdown_connections 2024-03-06T07:40:14.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.671+0000 7f0e02000640 1 -- 172.21.15.32:0/3955434320 wait complete. 2024-03-06T07:40:14.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.673+0000 7f0e02000640 1 Processor -- start 2024-03-06T07:40:14.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.674+0000 7f0e02000640 1 -- start start 2024-03-06T07:40:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e02000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc1e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e02000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dfc0cce90 0x7f0dfc1f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e02000640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dfc145300 0x7f0dfc136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e02000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc15cab0 con 0x7f0dfc070f10 2024-03-06T07:40:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e02000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc1456e0 con 0x7f0dfc0cce90 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e02000640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc15cc30 con 0x7f0dfc145300 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e00c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc1e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e00c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc1e4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48884/0 (socket says 172.21.15.32:48884) 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.675+0000 7f0e00c00640 1 -- 172.21.15.32:0/3784877378 learned_addr learned my addr 172.21.15.32:0/3784877378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0dfbe00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dfc0cce90 0x7f0dfc1f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e01600640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dfc145300 0x7f0dfc136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e00c00640 1 -- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dfc145300 msgr2=0x7f0dfc136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e00c00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dfc145300 0x7f0dfc136040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e00c00640 1 -- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dfc0cce90 msgr2=0x7f0dfc1f80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e00c00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dfc0cce90 0x7f0dfc1f80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e00c00640 1 -- 172.21.15.32:0/3784877378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc1e4ae0 con 0x7f0dfc070f10 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e01600640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dfc145300 0x7f0dfc136040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0e00c00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc1e4420 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0de400c3c0 tx=0x7f0de400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.676+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de40052d0 con 0x7f0dfc070f10 2024-03-06T07:40:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.677+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de4005490 con 0x7f0dfc070f10 2024-03-06T07:40:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.677+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de4007260 con 0x7f0dfc070f10 2024-03-06T07:40:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.677+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc1e4dc0 con 0x7f0dfc070f10 2024-03-06T07:40:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.677+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc1e5380 con 0x7f0dfc070f10 2024-03-06T07:40:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.678+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc0b3b70 con 0x7f0dfc070f10 2024-03-06T07:40:14.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.679+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0de4007400 con 0x7f0dfc070f10 2024-03-06T07:40:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.680+0000 7f0df8c00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0dd806b500 0x7f0dd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.680+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0de4089730 con 0x7f0dfc070f10 2024-03-06T07:40:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.680+0000 7f0dfbe00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0dd806b500 0x7f0dd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:14.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.681+0000 7f0dfbe00640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0dd806b500 0x7f0dd806d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0dec035000 tx=0x7f0dec030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.685+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dfc0b3b70 con 0x7f0dfc070f10 2024-03-06T07:40:14.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.925+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dfc067830 con 0x7f0dfc070f10 2024-03-06T07:40:14.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.925+0000 7f0df8c00640 1 -- 172.21.15.32:0/3784877378 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f0dfc067830 con 0x7f0dfc070f10 2024-03-06T07:40:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:14.927 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:14.927 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:14.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0dd806b500 msgr2=0x7f0dd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0dd806b500 0x7f0dd806d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0dec035000 tx=0x7f0dec030000 comp rx=0 tx=0).stop 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 msgr2=0x7f0dfc1e4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc1e4420 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0de400c3c0 tx=0x7f0de400c880 comp rx=0 tx=0).stop 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 shutdown_connections 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0dd806b500 0x7f0dd806d9c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0dfc145300 0x7f0dfc136040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0dfc0cce90 0x7f0dfc1f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 --2- 172.21.15.32:0/3784877378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dfc070f10 0x7f0dfc1e4420 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 >> 172.21.15.32:0/3784877378 conn(0x7f0dfc06b3d0 msgr2=0x7f0dfc15cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 shutdown_connections 2024-03-06T07:40:14.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:14.928+0000 7f0e02000640 1 -- 172.21.15.32:0/3784877378 wait complete. 2024-03-06T07:40:15.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:15.148 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:16.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:16.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.373+0000 7fd6ca600640 1 Processor -- start 2024-03-06T07:40:16.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.374+0000 7fd6ca600640 1 -- start start 2024-03-06T07:40:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.375+0000 7fd6ca600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 0x7fd6c415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.375+0000 7fd6ca600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c415cc40 con 0x7fd6c40af580 2024-03-06T07:40:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.375+0000 7fd6ca600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c415cf20 con 0x7fd6c40af1a0 2024-03-06T07:40:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.375+0000 7fd6ca600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c415d200 con 0x7fd6c40b34e0 2024-03-06T07:40:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.376+0000 7fd6baa00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd6c40af580 0x7fd6c40b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53604/0 (socket says 172.21.15.32:53604) 2024-03-06T07:40:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.376+0000 7fd6baa00640 1 -- 172.21.15.32:0/479003128 learned_addr learned my addr 172.21.15.32:0/479003128 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.376+0000 7fd6c3e00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 0x7fd6c415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.376+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906037430 0 0) 0x7fd6c415cc40 con 0x7fd6c40af580 2024-03-06T07:40:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.376+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a8003520 con 0x7fd6c40af580 2024-03-06T07:40:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494032053 0 0) 0x7fd6a8003520 con 0x7fd6c40af580 2024-03-06T07:40:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 msgr2=0x7fd6c415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 0x7fd6c415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 >> v1:172.21.15.39:6789/0 conn(0x7fd6c40af1a0 legacy=0x7fd6c40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c415d4e0 con 0x7fd6c40af580 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fd6ac001670 con 0x7fd6c40af580 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 >> v1:172.21.15.32:6789/0 conn(0x7fd6c40af580 legacy=0x7fd6c40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a8003a50 0x7fd6a8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 0x7fd6a8005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c415d200 con 0x7fd6a8003a50 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a800bbb0 con 0x7fd6a8004800 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a800bdb0 con 0x7fd6c40b34e0 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.377+0000 7fd6c3400640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a8003a50 0x7fd6a8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6c3e00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 0x7fd6a8005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6baa00640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6c3400640 1 -- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 msgr2=0x7fd6a8005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6c3400640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6c40b34e0 0x7fd6a8005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6c3400640 1 -- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6a8004800 msgr2=0x7fd6a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6c3400640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6a8004800 0x7fd6a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.378+0000 7fd6c3400640 1 -- 172.21.15.32:0/479003128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8003520 con 0x7fd6a8003a50 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.379+0000 7fd6c3400640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a8003a50 0x7fd6a8003e70 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4001720 tx=0x7fd6a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f56439c13a14ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.379+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4001370 con 0x7fd6a8003a50 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.379+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b40052d0 con 0x7fd6a8003a50 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.379+0000 7fd6c2a00640 1 -- 172.21.15.32:0/479003128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4003830 con 0x7fd6a8003a50 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6ca600640 1 -- 172.21.15.32:0/479003128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a8003a50 msgr2=0x7fd6a8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6ca600640 1 --2- 172.21.15.32:0/479003128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a8003a50 0x7fd6a8003e70 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4001720 tx=0x7fd6a8005830 comp rx=0 tx=0).stop 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6c3e00640 1 -- 172.21.15.32:0/479003128 reap_dead start 2024-03-06T07:40:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6ca600640 1 -- 172.21.15.32:0/479003128 shutdown_connections 2024-03-06T07:40:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6ca600640 1 -- 172.21.15.32:0/479003128 >> 172.21.15.32:0/479003128 conn(0x7fd6c406afb0 msgr2=0x7fd6c406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6ca600640 1 -- 172.21.15.32:0/479003128 shutdown_connections 2024-03-06T07:40:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.380+0000 7fd6ca600640 1 -- 172.21.15.32:0/479003128 wait complete. 2024-03-06T07:40:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.382+0000 7fd6ca600640 1 Processor -- start 2024-03-06T07:40:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.382+0000 7fd6ca600640 1 -- start start 2024-03-06T07:40:16.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6ca600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6a8003a50 0x7fd6c414c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6ca600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6c41592c0 0x7fd6c41f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6ca600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 0x7fd6c415fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6ca600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c415c600 con 0x7fd6c415fa70 2024-03-06T07:40:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6ca600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40cdeb0 con 0x7fd6c41592c0 2024-03-06T07:40:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6ca600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c415c780 con 0x7fd6a8003a50 2024-03-06T07:40:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 0x7fd6c415fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 0x7fd6c415fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48906/0 (socket says 172.21.15.32:48906) 2024-03-06T07:40:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6c3e00640 1 -- 172.21.15.32:0/711325637 learned_addr learned my addr 172.21.15.32:0/711325637 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.383+0000 7fd6baa00640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6c41592c0 0x7fd6c41f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3400640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6a8003a50 0x7fd6c414c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3e00640 1 -- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6a8003a50 msgr2=0x7fd6c414c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3e00640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6a8003a50 0x7fd6c414c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3e00640 1 -- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6c41592c0 msgr2=0x7fd6c41f59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3e00640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6c41592c0 0x7fd6c41f59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3e00640 1 -- 172.21.15.32:0/711325637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c409ac60 con 0x7fd6c415fa70 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6baa00640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6c41592c0 0x7fd6c41f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6c3e00640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 0x7fd6c415fef0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fd6c409e100 tx=0x7fd6bc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc003990 con 0x7fd6c415fa70 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc01b070 con 0x7fd6c415fa70 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc003b70 con 0x7fd6c415fa70 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c409af40 con 0x7fd6c415fa70 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.384+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c41f19a0 con 0x7fd6c415fa70 2024-03-06T07:40:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.386+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4069510 con 0x7fd6c415fa70 2024-03-06T07:40:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.387+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6bc005d10 con 0x7fd6c415fa70 2024-03-06T07:40:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.388+0000 7fd6bbe00640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6a006b500 0x7fd6a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.388+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6bc08a7d0 con 0x7fd6c415fa70 2024-03-06T07:40:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.389+0000 7fd6c3400640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6a006b500 0x7fd6a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.389+0000 7fd6c3400640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6a006b500 0x7fd6a006d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4033000 tx=0x7fd6b4009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.393+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6bc025020 con 0x7fd6c415fa70 2024-03-06T07:40:16.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.634+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6c404f3a0 con 0x7fd6c415fa70 2024-03-06T07:40:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.636+0000 7fd6bbe00640 1 -- 172.21.15.32:0/711325637 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fd6bc014350 con 0x7fd6c415fa70 2024-03-06T07:40:16.637 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:16.637 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6a006b500 msgr2=0x7fd6a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6a006b500 0x7fd6a006d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4033000 tx=0x7fd6b4009030 comp rx=0 tx=0).stop 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 msgr2=0x7fd6c415fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 0x7fd6c415fef0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fd6c409e100 tx=0x7fd6bc0115a0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 shutdown_connections 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fd6a006b500 0x7fd6a006d9c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c415fa70 0x7fd6c415fef0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6c41592c0 0x7fd6c41f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 --2- 172.21.15.32:0/711325637 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fd6a8003a50 0x7fd6c414c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.638+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 >> 172.21.15.32:0/711325637 conn(0x7fd6c406afb0 msgr2=0x7fd6c40b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:16.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.639+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 shutdown_connections 2024-03-06T07:40:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:16.639+0000 7fd6ca600640 1 -- 172.21.15.32:0/711325637 wait complete. 2024-03-06T07:40:16.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:16.868 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:17.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.086+0000 7ff38d600640 1 Processor -- start 2024-03-06T07:40:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.086+0000 7ff38d600640 1 -- start start 2024-03-06T07:40:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff38d600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 0x7ff3880cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff38d600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff388071050 0x7ff3880714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff38d600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3880b4860 con 0x7ff3880b3c90 2024-03-06T07:40:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff38d600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3880ce160 con 0x7ff388071050 2024-03-06T07:40:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff38d600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880b4b40 con 0x7ff3880cf5c0 2024-03-06T07:40:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff386a00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff3880b3c90 0x7ff3880b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51942/0 (socket says 172.21.15.32:51942) 2024-03-06T07:40:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.088+0000 7ff386a00640 1 -- 172.21.15.32:0/1543586190 learned_addr learned my addr 172.21.15.32:0/1543586190 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff387400640 1 --2- 172.21.15.32:0/1543586190 >> v2:172.21.15.39:3300/0 conn(0x7ff388071050 0x7ff3880714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff37e000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 0x7ff3880cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986604240 0 0) 0x7ff3880b4860 con 0x7ff3880b3c90 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36c003520 con 0x7ff3880b3c90 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409687186 0 0) 0x7ff36c003520 con 0x7ff3880b3c90 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 msgr2=0x7ff3880cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 0x7ff3880cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 >> v2:172.21.15.39:3300/0 conn(0x7ff388071050 msgr2=0x7ff3880714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 --2- 172.21.15.32:0/1543586190 >> v2:172.21.15.39:3300/0 conn(0x7ff388071050 0x7ff3880714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880b4e20 con 0x7ff3880b3c90 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7ff3780019f0 con 0x7ff3880b3c90 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 >> v1:172.21.15.32:6789/0 conn(0x7ff3880b3c90 legacy=0x7ff3880b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.089+0000 7ff386000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 0x7ff36c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff386000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff36c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff386000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff36c004d40 0x7ff36c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880ce160 con 0x7ff36c003fa0 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880b4b40 con 0x7ff36c004d40 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c00ba70 con 0x7ff3880cf5c0 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff37e000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff36c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff386a00640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 0x7ff36c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff387400640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff36c004d40 0x7ff36c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff37e000640 1 -- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 msgr2=0x7ff36c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff37e000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff3880cf5c0 0x7ff36c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff37e000640 1 -- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff36c004d40 msgr2=0x7ff36c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff37e000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff36c004d40 0x7ff36c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff37e000640 1 -- 172.21.15.32:0/1543586190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c003520 con 0x7ff36c003fa0 2024-03-06T07:40:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.090+0000 7ff387400640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff36c004d40 0x7ff36c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.091+0000 7ff37e000640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff36c004800 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff370009ac0 tx=0x7ff36c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5966293b1eda65c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370039070 con 0x7ff36c003fa0 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37002c6b0 con 0x7ff36c003fa0 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff386000640 1 -- 172.21.15.32:0/1543586190 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370009690 con 0x7ff36c003fa0 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff38d600640 1 -- 172.21.15.32:0/1543586190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 msgr2=0x7ff36c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff38d600640 1 --2- 172.21.15.32:0/1543586190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff36c004800 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff370009ac0 tx=0x7ff36c005830 comp rx=0 tx=0).stop 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff387400640 1 -- 172.21.15.32:0/1543586190 reap_dead start 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff38d600640 1 -- 172.21.15.32:0/1543586190 shutdown_connections 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.092+0000 7ff38d600640 1 -- 172.21.15.32:0/1543586190 >> 172.21.15.32:0/1543586190 conn(0x7ff38806afb0 msgr2=0x7ff38806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.093+0000 7ff38d600640 1 -- 172.21.15.32:0/1543586190 shutdown_connections 2024-03-06T07:40:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.093+0000 7ff38d600640 1 -- 172.21.15.32:0/1543586190 wait complete. 2024-03-06T07:40:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.094+0000 7ff38d600640 1 Processor -- start 2024-03-06T07:40:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.094+0000 7ff38d600640 1 -- start start 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.095+0000 7ff38d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff38809d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.095+0000 7ff38d600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff388146410 0x7ff388146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.095+0000 7ff38d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff388152f80 0x7ff388153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.095+0000 7ff38d600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880cdc20 con 0x7ff36c003fa0 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.095+0000 7ff38d600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815b340 con 0x7ff388152f80 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.095+0000 7ff38d600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880cdda0 con 0x7ff388146410 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff38809d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff38809d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38376/0 (socket says 172.21.15.32:38376) 2024-03-06T07:40:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 -- 172.21.15.32:0/4122187492 learned_addr learned my addr 172.21.15.32:0/4122187492 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:18.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff37e000640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff388146410 0x7ff388146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff387400640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff388152f80 0x7ff388153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 -- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff388146410 msgr2=0x7ff388146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff388146410 0x7ff388146850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 -- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff388152f80 msgr2=0x7ff388153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff388152f80 0x7ff388153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.096+0000 7ff386a00640 1 -- 172.21.15.32:0/4122187492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388146d90 con 0x7ff36c003fa0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.097+0000 7ff386a00640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff38809d4e0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7ff378007a40 tx=0x7ff37800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.097+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378005e50 con 0x7ff36c003fa0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.097+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff378018070 con 0x7ff36c003fa0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.097+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378013640 con 0x7ff36c003fa0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.097+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff388153c70 con 0x7ff36c003fa0 2024-03-06T07:40:18.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.097+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff388154230 con 0x7ff36c003fa0 2024-03-06T07:40:18.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.098+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff388069510 con 0x7ff36c003fa0 2024-03-06T07:40:18.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.100+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3780137e0 con 0x7ff36c003fa0 2024-03-06T07:40:18.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.101+0000 7ff37f400640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff36406b6a0 0x7ff36406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:18.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.101+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff37808a8d0 con 0x7ff36c003fa0 2024-03-06T07:40:18.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.101+0000 7ff37e000640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff36406b6a0 0x7ff36406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:18.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.102+0000 7ff37e000640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff36406b6a0 0x7ff36406db60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff370036000 tx=0x7ff370030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:18.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.110+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3780595e0 con 0x7ff36c003fa0 2024-03-06T07:40:18.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.369+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff38806d5c0 con 0x7ff36c003fa0 2024-03-06T07:40:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.370+0000 7ff37f400640 1 -- 172.21.15.32:0/4122187492 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7ff378058ee0 con 0x7ff36c003fa0 2024-03-06T07:40:18.372 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:18.372 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.373+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff36406b6a0 msgr2=0x7ff36406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.373+0000 7ff38d600640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff36406b6a0 0x7ff36406db60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff370036000 tx=0x7ff370030000 comp rx=0 tx=0).stop 2024-03-06T07:40:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.373+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 msgr2=0x7ff38809d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.373+0000 7ff38d600640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff38809d4e0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7ff378007a40 tx=0x7ff37800e530 comp rx=0 tx=0).stop 2024-03-06T07:40:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 shutdown_connections 2024-03-06T07:40:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff36406b6a0 0x7ff36406db60 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff388152f80 0x7ff388153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff388146410 0x7ff388146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 --2- 172.21.15.32:0/4122187492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36c003fa0 0x7ff38809d4e0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 >> 172.21.15.32:0/4122187492 conn(0x7ff38806afb0 msgr2=0x7ff38806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 shutdown_connections 2024-03-06T07:40:18.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:18.374+0000 7ff38d600640 1 -- 172.21.15.32:0/4122187492 wait complete. 2024-03-06T07:40:18.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:18.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:19.498 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:19.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.828+0000 7f8d85a00640 1 Processor -- start 2024-03-06T07:40:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.828+0000 7f8d85a00640 1 -- start start 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.829+0000 7f8d85a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d800af1a0 0x7f8d8015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.829+0000 7f8d85a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8d800af580 0x7f8d800b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d85a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800cdc20 0x7f8d800b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d85a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8015cc40 con 0x7f8d800af1a0 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d85a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8015cf20 con 0x7f8d800af580 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d85a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8015d200 con 0x7f8d800cdc20 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d7ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d800af1a0 0x7f8d8015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d7ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d800af1a0 0x7f8d8015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38378/0 (socket says 172.21.15.32:38378) 2024-03-06T07:40:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d7ea00640 1 -- 172.21.15.32:0/1548896361 learned_addr learned my addr 172.21.15.32:0/1548896361 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d76000640 1 --2- 172.21.15.32:0/1548896361 >> v2:172.21.15.39:3300/0 conn(0x7f8d800af580 0x7f8d800b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.830+0000 7f8d7f400640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800cdc20 0x7f8d800b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d7ea00640 1 -- 172.21.15.32:0/1548896361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800cdc20 msgr2=0x7f8d800b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d7ea00640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800cdc20 0x7f8d800b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d7ea00640 1 -- 172.21.15.32:0/1548896361 >> v2:172.21.15.39:3300/0 conn(0x7f8d800af580 msgr2=0x7f8d800b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d7ea00640 1 --2- 172.21.15.32:0/1548896361 >> v2:172.21.15.39:3300/0 conn(0x7f8d800af580 0x7f8d800b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d7ea00640 1 -- 172.21.15.32:0/1548896361 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8015d4e0 con 0x7f8d800af1a0 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d76000640 1 --2- 172.21.15.32:0/1548896361 >> v2:172.21.15.39:3300/0 conn(0x7f8d800af580 0x7f8d800b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.831+0000 7f8d7f400640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800cdc20 0x7f8d800b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.832+0000 7f8d7ea00640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800af1a0 0x7f8d8015c600 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f8d800b54c0 tx=0x7f8d6802b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e60404288ce5ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.832+0000 7f8d7e000640 1 -- 172.21.15.32:0/1548896361 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d680317f0 con 0x7f8d800af1a0 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.832+0000 7f8d7e000640 1 -- 172.21.15.32:0/1548896361 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68037070 con 0x7f8d800af1a0 2024-03-06T07:40:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.832+0000 7f8d7e000640 1 -- 172.21.15.32:0/1548896361 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68031bb0 con 0x7f8d800af1a0 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 -- 172.21.15.32:0/1548896361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800af1a0 msgr2=0x7f8d8015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800af1a0 0x7f8d8015c600 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f8d800b54c0 tx=0x7f8d6802b890 comp rx=0 tx=0).stop 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 -- 172.21.15.32:0/1548896361 shutdown_connections 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800cdc20 0x7f8d800b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1548896361 >> v2:172.21.15.39:3300/0 conn(0x7f8d800af580 0x7f8d800b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1548896361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800af1a0 0x7f8d8015c600 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 -- 172.21.15.32:0/1548896361 >> 172.21.15.32:0/1548896361 conn(0x7f8d8006afb0 msgr2=0x7f8d8006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 -- 172.21.15.32:0/1548896361 shutdown_connections 2024-03-06T07:40:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.833+0000 7f8d85a00640 1 -- 172.21.15.32:0/1548896361 wait complete. 2024-03-06T07:40:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.835+0000 7f8d85a00640 1 Processor -- start 2024-03-06T07:40:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.835+0000 7f8d85a00640 1 -- start start 2024-03-06T07:40:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d85a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d800af1a0 0x7f8d80151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d85a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800af580 0x7f8d8014b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d85a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 0x7f8d8009b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d85a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800b3490 con 0x7f8d800cdc20 2024-03-06T07:40:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d85a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800b3310 con 0x7f8d800af1a0 2024-03-06T07:40:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d85a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800b3610 con 0x7f8d800af580 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d7f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 0x7f8d8009b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d7f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 0x7f8d8009b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38390/0 (socket says 172.21.15.32:38390) 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d7f400640 1 -- 172.21.15.32:0/1780243844 learned_addr learned my addr 172.21.15.32:0/1780243844 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d7ea00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d800af1a0 0x7f8d80151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.836+0000 7f8d76000640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800af580 0x7f8d8014b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d7f400640 1 -- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800af580 msgr2=0x7f8d8014b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d7f400640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800af580 0x7f8d8014b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d7f400640 1 -- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d800af1a0 msgr2=0x7f8d80151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d7f400640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d800af1a0 0x7f8d80151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d7f400640 1 -- 172.21.15.32:0/1780243844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8009c5a0 con 0x7f8d800cdc20 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d7f400640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 0x7f8d8009b9e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800cd80 tx=0x7f8d7800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800f680 con 0x7f8d800cdc20 2024-03-06T07:40:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7800f840 con 0x7f8d800cdc20 2024-03-06T07:40:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800fc30 con 0x7f8d800cdc20 2024-03-06T07:40:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d801517d0 con 0x7f8d800cdc20 2024-03-06T07:40:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.837+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d801507b0 con 0x7f8d800cdc20 2024-03-06T07:40:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.839+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8004f3a0 con 0x7f8d800cdc20 2024-03-06T07:40:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.840+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d7800e3e0 con 0x7f8d800cdc20 2024-03-06T07:40:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.840+0000 7f8d77400640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d5c06b4d0 0x7f8d5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.841+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8d7808afd0 con 0x7f8d800cdc20 2024-03-06T07:40:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.841+0000 7f8d7ea00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d5c06b4d0 0x7f8d5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.841+0000 7f8d7ea00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d5c06b4d0 0x7f8d5c06d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8d68036000 tx=0x7f8d6803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:19.849+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d8009c5a0 con 0x7f8d800cdc20 2024-03-06T07:40:20.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.093+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d801520a0 con 0x7f8d800cdc20 2024-03-06T07:40:20.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.094+0000 7f8d77400640 1 -- 172.21.15.32:0/1780243844 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f8d78014030 con 0x7f8d800cdc20 2024-03-06T07:40:20.097 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:20.097 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:20.097 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:20.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d5c06b4d0 msgr2=0x7f8d5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:20.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d5c06b4d0 0x7f8d5c06d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8d68036000 tx=0x7f8d6803d040 comp rx=0 tx=0).stop 2024-03-06T07:40:20.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 msgr2=0x7f8d8009b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:20.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 0x7f8d8009b9e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800cd80 tx=0x7f8d7800a650 comp rx=0 tx=0).stop 2024-03-06T07:40:20.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 shutdown_connections 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f8d5c06b4d0 0x7f8d5c06d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d800cdc20 0x7f8d8009b9e0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f8d800af580 0x7f8d8014b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 --2- 172.21.15.32:0/1780243844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d800af1a0 0x7f8d80151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 >> 172.21.15.32:0/1780243844 conn(0x7f8d8006afb0 msgr2=0x7f8d800acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 shutdown_connections 2024-03-06T07:40:20.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:20.097+0000 7f8d85a00640 1 -- 172.21.15.32:0/1780243844 wait complete. 2024-03-06T07:40:20.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:20.323 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:21.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:21.500 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:21.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.549+0000 7f9ae9400640 1 Processor -- start 2024-03-06T07:40:21.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.549+0000 7f9ae9400640 1 -- start start 2024-03-06T07:40:21.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.550+0000 7f9ae9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9ae40d0040 0x7f9ae40d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.550+0000 7f9ae9400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae415fc60 0x7f9ae41600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.550+0000 7f9ae9400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40af220 con 0x7f9ae4145a90 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.550+0000 7f9ae9400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae406ba00 con 0x7f9ae40d0040 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.550+0000 7f9ae9400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40cd830 con 0x7f9ae415fc60 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.551+0000 7f9adbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9ae40d0040 0x7f9ae40d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.551+0000 7f9adbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9ae40d0040 0x7f9ae40d04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:55732/0 (socket says 172.21.15.32:55732) 2024-03-06T07:40:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.551+0000 7f9adbe00640 1 -- 172.21.15.32:0/1498097966 learned_addr learned my addr 172.21.15.32:0/1498097966 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.551+0000 7f9ae3200640 1 --2- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae415fc60 0x7f9ae41600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.552+0000 7f9adbe00640 1 -- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae415fc60 msgr2=0x7f9ae41600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.552+0000 7f9adbe00640 1 --2- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae415fc60 0x7f9ae41600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.552+0000 7f9adbe00640 1 -- 172.21.15.32:0/1498097966 >> v1:172.21.15.39:6789/0 conn(0x7f9ae4145a90 legacy=0x7f9ae406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.552+0000 7f9adbe00640 1 -- 172.21.15.32:0/1498097966 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40cdb10 con 0x7f9ae40d0040 2024-03-06T07:40:21.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.555+0000 7f9adbe00640 1 --2- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae40d0040 0x7f9ae40d04c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009870 tx=0x7f9acc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89ce4c9466d47953 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:21.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.555+0000 7f9ae1e00640 1 -- 172.21.15.32:0/1498097966 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc034070 con 0x7f9ae40d0040 2024-03-06T07:40:21.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.556+0000 7f9ae1e00640 1 -- 172.21.15.32:0/1498097966 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc030840 con 0x7f9ae40d0040 2024-03-06T07:40:21.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.556+0000 7f9ae1e00640 1 -- 172.21.15.32:0/1498097966 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc030c10 con 0x7f9ae40d0040 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.556+0000 7f9ae9400640 1 -- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae40d0040 msgr2=0x7f9ae40d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.556+0000 7f9ae9400640 1 --2- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae40d0040 0x7f9ae40d04c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009870 tx=0x7f9acc02b980 comp rx=0 tx=0).stop 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.556+0000 7f9ae9400640 1 -- 172.21.15.32:0/1498097966 shutdown_connections 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.557+0000 7f9ae9400640 1 --2- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae415fc60 0x7f9ae41600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.557+0000 7f9ae9400640 1 --2- 172.21.15.32:0/1498097966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae40d0040 0x7f9ae40d04c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.557+0000 7f9ae9400640 1 -- 172.21.15.32:0/1498097966 >> 172.21.15.32:0/1498097966 conn(0x7f9ae40d0d80 msgr2=0x7f9ae40d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.557+0000 7f9ae9400640 1 -- 172.21.15.32:0/1498097966 shutdown_connections 2024-03-06T07:40:21.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.557+0000 7f9ae9400640 1 -- 172.21.15.32:0/1498097966 wait complete. 2024-03-06T07:40:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.559+0000 7f9ae9400640 1 Processor -- start 2024-03-06T07:40:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.559+0000 7f9ae9400640 1 -- start start 2024-03-06T07:40:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae9400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae40d0040 0x7f9ae415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae4145a90 0x7f9ae40ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 0x7f9ae40cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae9400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae406b6c0 con 0x7f9ae415fc60 2024-03-06T07:40:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae9400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40ce0d0 con 0x7f9ae4145a90 2024-03-06T07:40:21.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae9400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40ce5f0 con 0x7f9ae40d0040 2024-03-06T07:40:21.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 0x7f9ae40cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 0x7f9ae40cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38394/0 (socket says 172.21.15.32:38394) 2024-03-06T07:40:21.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae3200640 1 -- 172.21.15.32:0/3796629305 learned_addr learned my addr 172.21.15.32:0/3796629305 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:21.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9ae2800640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae40d0040 0x7f9ae415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.560+0000 7f9adbe00640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae4145a90 0x7f9ae40ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9ae3200640 1 -- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae40d0040 msgr2=0x7f9ae415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9ae3200640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae40d0040 0x7f9ae415ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9ae3200640 1 -- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae4145a90 msgr2=0x7f9ae40ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9ae3200640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae4145a90 0x7f9ae40ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9ae3200640 1 -- 172.21.15.32:0/3796629305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40b43a0 con 0x7f9ae415fc60 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9adbe00640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae4145a90 0x7f9ae40ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.561+0000 7f9ae3200640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 0x7f9ae40cab70 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d30 tx=0x7f9adc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.562+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc018020 con 0x7f9ae415fc60 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.562+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc014480 con 0x7f9ae415fc60 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.562+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae415ecd0 con 0x7f9ae415fc60 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.562+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc014870 con 0x7f9ae415fc60 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.562+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae415efb0 con 0x7f9ae415fc60 2024-03-06T07:40:21.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.563+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae404f3a0 con 0x7f9ae415fc60 2024-03-06T07:40:21.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.564+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9adc005d10 con 0x7f9ae415fc60 2024-03-06T07:40:21.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.566+0000 7f9adaa00640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ac006b520 0x7f9ac006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:21.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.566+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9adc08ac00 con 0x7f9ae415fc60 2024-03-06T07:40:21.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.566+0000 7f9ae2800640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ac006b520 0x7f9ac006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.567+0000 7f9ae2800640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ac006b520 0x7f9ac006d9e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4001780 tx=0x7f9ad4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.571+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9adc022070 con 0x7f9ae415fc60 2024-03-06T07:40:21.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:21.651 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:21.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.816+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ae4071f50 con 0x7f9ae415fc60 2024-03-06T07:40:21.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.817+0000 7f9adaa00640 1 -- 172.21.15.32:0/3796629305 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f9adc010040 con 0x7f9ae415fc60 2024-03-06T07:40:21.818 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:21.818 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:21.819 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:21.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ac006b520 msgr2=0x7f9ac006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ac006b520 0x7f9ac006d9e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4001780 tx=0x7f9ad4009040 comp rx=0 tx=0).stop 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 msgr2=0x7f9ae40cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 0x7f9ae40cab70 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d30 tx=0x7f9adc00c490 comp rx=0 tx=0).stop 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 shutdown_connections 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f9ac006b520 0x7f9ac006d9e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae415fc60 0x7f9ae40cab70 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ae4145a90 0x7f9ae40ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 --2- 172.21.15.32:0/3796629305 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f9ae40d0040 0x7f9ae415ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 >> 172.21.15.32:0/3796629305 conn(0x7f9ae40d0d80 msgr2=0x7f9ae40cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 shutdown_connections 2024-03-06T07:40:21.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:21.820+0000 7f9ae9400640 1 -- 172.21.15.32:0/3796629305 wait complete. 2024-03-06T07:40:22.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:22.042 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:23.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.265+0000 7f5f53400640 1 Processor -- start 2024-03-06T07:40:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.265+0000 7f5f53400640 1 -- start start 2024-03-06T07:40:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.266+0000 7f5f53400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f4c0cfd90 0x7f5f4c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.266+0000 7f5f53400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0d2760 0x7f5f4c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.266+0000 7f5f53400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0cea50 con 0x7f5f4c161b40 2024-03-06T07:40:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f53400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c0b5770 con 0x7f5f4c0d2760 2024-03-06T07:40:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f53400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cf900 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f51400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0d2760 0x7f5f4c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f51400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f4c0d2760 0x7f5f4c0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38402/0 (socket says 172.21.15.32:38402) 2024-03-06T07:40:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f51400640 1 -- 172.21.15.32:0/272937262 learned_addr learned my addr 172.21.15.32:0/272937262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f43e00640 1 --2- 172.21.15.32:0/272937262 >> v2:172.21.15.39:3300/0 conn(0x7f5f4c0cfd90 0x7f5f4c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f51400640 1 -- 172.21.15.32:0/272937262 >> v2:172.21.15.39:3300/0 conn(0x7f5f4c0cfd90 msgr2=0x7f5f4c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f51400640 1 --2- 172.21.15.32:0/272937262 >> v2:172.21.15.39:3300/0 conn(0x7f5f4c0cfd90 0x7f5f4c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.267+0000 7f5f51400640 1 -- 172.21.15.32:0/272937262 >> v1:172.21.15.39:6789/0 conn(0x7f5f4c161b40 legacy=0x7f5f4c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.268+0000 7f5f51400640 1 -- 172.21.15.32:0/272937262 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0cecf0 con 0x7f5f4c0d2760 2024-03-06T07:40:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.269+0000 7f5f51400640 1 --2- 172.21.15.32:0/272937262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0d2760 0x7f5f4c0b3740 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0b9fa0 tx=0x7f5f4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b583d712ef3456c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.269+0000 7f5f4be00640 1 -- 172.21.15.32:0/272937262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44035020 con 0x7f5f4c0d2760 2024-03-06T07:40:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.269+0000 7f5f4be00640 1 -- 172.21.15.32:0/272937262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44031870 con 0x7f5f4c0d2760 2024-03-06T07:40:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.269+0000 7f5f4be00640 1 -- 172.21.15.32:0/272937262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44031c40 con 0x7f5f4c0d2760 2024-03-06T07:40:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 -- 172.21.15.32:0/272937262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0d2760 msgr2=0x7f5f4c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 --2- 172.21.15.32:0/272937262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0d2760 0x7f5f4c0b3740 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0b9fa0 tx=0x7f5f4402b910 comp rx=0 tx=0).stop 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 -- 172.21.15.32:0/272937262 shutdown_connections 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 --2- 172.21.15.32:0/272937262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0d2760 0x7f5f4c0b3740 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 --2- 172.21.15.32:0/272937262 >> v2:172.21.15.39:3300/0 conn(0x7f5f4c0cfd90 0x7f5f4c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 -- 172.21.15.32:0/272937262 >> 172.21.15.32:0/272937262 conn(0x7f5f4c09d220 msgr2=0x7f5f4c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 -- 172.21.15.32:0/272937262 shutdown_connections 2024-03-06T07:40:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.270+0000 7f5f53400640 1 -- 172.21.15.32:0/272937262 wait complete. 2024-03-06T07:40:23.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.272+0000 7f5f53400640 1 Processor -- start 2024-03-06T07:40:23.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.272+0000 7f5f53400640 1 -- start start 2024-03-06T07:40:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.273+0000 7f5f53400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 0x7f5f4c155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.273+0000 7f5f53400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f4c0d2760 0x7f5f4c0c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.273+0000 7f5f53400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f4c161b40 0x7f5f4c0b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.273+0000 7f5f53400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cf590 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.273+0000 7f5f53400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cf2b0 con 0x7f5f4c161b40 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.273+0000 7f5f53400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0b2ff0 con 0x7f5f4c0d2760 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 0x7f5f4c155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 0x7f5f4c155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38414/0 (socket says 172.21.15.32:38414) 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 -- 172.21.15.32:0/604150628 learned_addr learned my addr 172.21.15.32:0/604150628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f51400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f4c161b40 0x7f5f4c0b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f43e00640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f4c0d2760 0x7f5f4c0c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 -- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f4c0d2760 msgr2=0x7f5f4c0c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f4c0d2760 0x7f5f4c0c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 -- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f4c161b40 msgr2=0x7f5f4c0b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f4c161b40 0x7f5f4c0b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.274+0000 7f5f50a00640 1 -- 172.21.15.32:0/604150628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0b8060 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.275+0000 7f5f50a00640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 0x7f5f4c155a80 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c7a0 tx=0x7f5f3c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.275+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c017070 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.275+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c003da0 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.275+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c0b8340 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.275+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c0052b0 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.275+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c0b8900 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.276+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c04f3a0 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.277+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f3c01a070 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.279+0000 7f5f49600640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f2c06b500 0x7f5f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.279+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f3c089aa0 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.279+0000 7f5f43e00640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f2c06b500 0x7f5f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:23.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.280+0000 7f5f43e00640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f2c06b500 0x7f5f2c06d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5f38006fd0 tx=0x7f5f38008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.283+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f3c0587b0 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.502 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:23.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.539+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4c0d4e00 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.540+0000 7f5f49600640 1 -- 172.21.15.32:0/604150628 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f5f3c0580b0 con 0x7f5f4c0cfd90 2024-03-06T07:40:23.542 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:23.542 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:23.542 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:23.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f2c06b500 msgr2=0x7f5f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f2c06b500 0x7f5f2c06d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5f38006fd0 tx=0x7f5f38008040 comp rx=0 tx=0).stop 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 msgr2=0x7f5f4c155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 0x7f5f4c155a80 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c7a0 tx=0x7f5f3c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 shutdown_connections 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5f2c06b500 0x7f5f2c06d9c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f4c161b40 0x7f5f4c0b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5f4c0d2760 0x7f5f4c0c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 --2- 172.21.15.32:0/604150628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f4c0cfd90 0x7f5f4c155a80 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 >> 172.21.15.32:0/604150628 conn(0x7f5f4c09d220 msgr2=0x7f5f4c0b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 shutdown_connections 2024-03-06T07:40:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:23.542+0000 7f5f53400640 1 -- 172.21.15.32:0/604150628 wait complete. 2024-03-06T07:40:23.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:23.771 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:24.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:24.998+0000 7f1b88a00640 1 Processor -- start 2024-03-06T07:40:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:24.998+0000 7f1b88a00640 1 -- start start 2024-03-06T07:40:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b88a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b8415e070 0x7f1b840d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b88a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1b840cf410 0x7f1b840cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b88a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b840d18c0 con 0x7f1b840d05c0 2024-03-06T07:40:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b88a00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840d1ba0 con 0x7f1b8415e070 2024-03-06T07:40:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b88a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840d1e80 con 0x7f1b840cf410 2024-03-06T07:40:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b83400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b8415e070 0x7f1b840d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b83400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b8415e070 0x7f1b840d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38416/0 (socket says 172.21.15.32:38416) 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.000+0000 7f1b83400640 1 -- 172.21.15.32:0/3277363541 learned_addr learned my addr 172.21.15.32:0/3277363541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.001+0000 7f1b82a00640 1 --2- 172.21.15.32:0/3277363541 >> v2:172.21.15.39:3300/0 conn(0x7f1b840cf410 0x7f1b840cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.001+0000 7f1b83400640 1 -- 172.21.15.32:0/3277363541 >> v2:172.21.15.39:3300/0 conn(0x7f1b840cf410 msgr2=0x7f1b840cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.001+0000 7f1b83400640 1 --2- 172.21.15.32:0/3277363541 >> v2:172.21.15.39:3300/0 conn(0x7f1b840cf410 0x7f1b840cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.001+0000 7f1b83400640 1 -- 172.21.15.32:0/3277363541 >> v1:172.21.15.39:6789/0 conn(0x7f1b840d05c0 legacy=0x7f1b840b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:25.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.001+0000 7f1b83400640 1 -- 172.21.15.32:0/3277363541 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840d2160 con 0x7f1b8415e070 2024-03-06T07:40:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.002+0000 7f1b83400640 1 --2- 172.21.15.32:0/3277363541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840d2960 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f1b840b6880 tx=0x7f1b7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8f8a20092e58f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.002+0000 7f1b82000640 1 -- 172.21.15.32:0/3277363541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7402ca80 con 0x7f1b8415e070 2024-03-06T07:40:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.002+0000 7f1b82000640 1 -- 172.21.15.32:0/3277363541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7402cc20 con 0x7f1b8415e070 2024-03-06T07:40:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.002+0000 7f1b82000640 1 -- 172.21.15.32:0/3277363541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b740343f0 con 0x7f1b8415e070 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 -- 172.21.15.32:0/3277363541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 msgr2=0x7f1b840d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 --2- 172.21.15.32:0/3277363541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840d2960 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f1b840b6880 tx=0x7f1b7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 -- 172.21.15.32:0/3277363541 shutdown_connections 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 --2- 172.21.15.32:0/3277363541 >> v2:172.21.15.39:3300/0 conn(0x7f1b840cf410 0x7f1b840cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 --2- 172.21.15.32:0/3277363541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840d2960 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 -- 172.21.15.32:0/3277363541 >> 172.21.15.32:0/3277363541 conn(0x7f1b8408d2a0 msgr2=0x7f1b840d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 -- 172.21.15.32:0/3277363541 shutdown_connections 2024-03-06T07:40:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.003+0000 7f1b88a00640 1 -- 172.21.15.32:0/3277363541 wait complete. 2024-03-06T07:40:25.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.005+0000 7f1b88a00640 1 Processor -- start 2024-03-06T07:40:25.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.005+0000 7f1b88a00640 1 -- start start 2024-03-06T07:40:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.006+0000 7f1b88a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b840cf410 0x7f1b841584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.006+0000 7f1b88a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b840d05c0 0x7f1b840b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.006+0000 7f1b88a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.006+0000 7f1b88a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d4730 con 0x7f1b8415e070 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.006+0000 7f1b88a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d45b0 con 0x7f1b840cf410 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.006+0000 7f1b88a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d0a30 con 0x7f1b840d05c0 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840b6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38420/0 (socket says 172.21.15.32:38420) 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 -- 172.21.15.32:0/2224581421 learned_addr learned my addr 172.21.15.32:0/2224581421 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83400640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b840cf410 0x7f1b841584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b82a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b840d05c0 0x7f1b840b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 -- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b840d05c0 msgr2=0x7f1b840b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b840d05c0 0x7f1b840b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 -- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b840cf410 msgr2=0x7f1b841584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b840cf410 0x7f1b841584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b83e00640 1 -- 172.21.15.32:0/2224581421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84159930 con 0x7f1b8415e070 2024-03-06T07:40:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.007+0000 7f1b82a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b840d05c0 0x7f1b840b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:40:25.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.008+0000 7f1b83e00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840b6e50 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00c410 tx=0x7f1b7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:25.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.008+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c011790 con 0x7f1b8415e070 2024-03-06T07:40:25.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.008+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c017070 con 0x7f1b8415e070 2024-03-06T07:40:25.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.008+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c011bb0 con 0x7f1b8415e070 2024-03-06T07:40:25.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.008+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b840cbe40 con 0x7f1b8415e070 2024-03-06T07:40:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.008+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b840cc400 con 0x7f1b8415e070 2024-03-06T07:40:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.009+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8404f330 con 0x7f1b8415e070 2024-03-06T07:40:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.010+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b7c01c460 con 0x7f1b8415e070 2024-03-06T07:40:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.011+0000 7f1b7b400640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.011+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1b7c089e30 con 0x7f1b8415e070 2024-03-06T07:40:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.012+0000 7f1b83400640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.012+0000 7f1b83400640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b6006b4d0 0x7f1b6006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1b74035000 tx=0x7f1b74030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:25.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.016+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b7c021020 con 0x7f1b8415e070 2024-03-06T07:40:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.350+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b840d10c0 con 0x7f1b8415e070 2024-03-06T07:40:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.351+0000 7f1b7b400640 1 -- 172.21.15.32:0/2224581421 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f1b7c058bd0 con 0x7f1b8415e070 2024-03-06T07:40:25.353 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b6006b4d0 msgr2=0x7f1b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b6006b4d0 0x7f1b6006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1b74035000 tx=0x7f1b74030000 comp rx=0 tx=0).stop 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 msgr2=0x7f1b840b6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840b6e50 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00c410 tx=0x7f1b7c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 shutdown_connections 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b8415e070 0x7f1b840b6e50 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f1b840d05c0 0x7f1b840b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 --2- 172.21.15.32:0/2224581421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b840cf410 0x7f1b841584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.353+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 >> 172.21.15.32:0/2224581421 conn(0x7f1b8408d2a0 msgr2=0x7f1b840b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.354+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 shutdown_connections 2024-03-06T07:40:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:25.354+0000 7f1b88a00640 1 -- 172.21.15.32:0/2224581421 wait complete. 2024-03-06T07:40:25.504 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:25.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:25.577 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:26.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:26.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.819+0000 7f980ea00640 1 Processor -- start 2024-03-06T07:40:26.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.819+0000 7f980ea00640 1 -- start start 2024-03-06T07:40:26.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.821+0000 7f980ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f980815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.821+0000 7f980ea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f98080cdc20 0x7f98080b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.821+0000 7f980ea00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980815cc40 con 0x7f98080af580 2024-03-06T07:40:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.821+0000 7f980ea00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980815cf20 con 0x7f98080cdc20 2024-03-06T07:40:26.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.821+0000 7f980ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980815d200 con 0x7f98080af1a0 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.821+0000 7f980ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f98080cdc20 0x7f98080b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f97ff400640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f98080af580 0x7f98080b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51956/0 (socket says 172.21.15.32:51956) 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f980ca00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f98080cdc20 0x7f98080b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38424/0 (socket says 172.21.15.32:38424) 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f97ff400640 1 -- 172.21.15.32:0/2854053057 learned_addr learned my addr 172.21.15.32:0/2854053057 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f9807e00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f980815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f980ca00640 1 -- 172.21.15.32:0/2854053057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 msgr2=0x7f980815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f980ca00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f980815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f980ca00640 1 -- 172.21.15.32:0/2854053057 >> v1:172.21.15.32:6789/0 conn(0x7f98080af580 legacy=0x7f98080b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f9807400640 1 -- 172.21.15.32:0/2854053057 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079099580 0 0) 0x7f980815cc40 con 0x7f98080af580 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f980ca00640 1 -- 172.21.15.32:0/2854053057 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980815d4e0 con 0x7f98080cdc20 2024-03-06T07:40:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.822+0000 7f9807e00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f980815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:40:26.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.825+0000 7f980ca00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080cdc20 0x7f98080b2dd0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f9800009d90 tx=0x7f980002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1024404f3763560b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:26.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.825+0000 7f9807400640 1 -- 172.21.15.32:0/2854053057 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800039070 con 0x7f98080cdc20 2024-03-06T07:40:26.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.825+0000 7f9807400640 1 -- 172.21.15.32:0/2854053057 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980002cbc0 con 0x7f98080cdc20 2024-03-06T07:40:26.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.825+0000 7f9807400640 1 -- 172.21.15.32:0/2854053057 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800034440 con 0x7f98080cdc20 2024-03-06T07:40:26.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 -- 172.21.15.32:0/2854053057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080cdc20 msgr2=0x7f98080b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080cdc20 0x7f98080b2dd0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f9800009d90 tx=0x7f980002be10 comp rx=0 tx=0).stop 2024-03-06T07:40:26.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 -- 172.21.15.32:0/2854053057 shutdown_connections 2024-03-06T07:40:26.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080cdc20 0x7f98080b2dd0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 --2- 172.21.15.32:0/2854053057 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f980815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 -- 172.21.15.32:0/2854053057 >> 172.21.15.32:0/2854053057 conn(0x7f980806afb0 msgr2=0x7f980806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:26.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 -- 172.21.15.32:0/2854053057 shutdown_connections 2024-03-06T07:40:26.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.826+0000 7f980ea00640 1 -- 172.21.15.32:0/2854053057 wait complete. 2024-03-06T07:40:26.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.829+0000 7f980ea00640 1 Processor -- start 2024-03-06T07:40:26.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.829+0000 7f980ea00640 1 -- start start 2024-03-06T07:40:26.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f980ea00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f98081f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f980ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 0x7f9808156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f980ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98080cdc20 0x7f98080ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f980ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98080b36a0 con 0x7f98080af580 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f980ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98080b3520 con 0x7f98080cdc20 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f980ea00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98080b3820 con 0x7f98080af1a0 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f97ff400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 0x7f9808156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f97ff400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 0x7f9808156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38428/0 (socket says 172.21.15.32:38428) 2024-03-06T07:40:26.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.831+0000 7f97ff400640 1 -- 172.21.15.32:0/2496605659 learned_addr learned my addr 172.21.15.32:0/2496605659 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f980ca00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98080cdc20 0x7f98080ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f97ff400640 1 -- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 msgr2=0x7f98081f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f97ff400640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f98081f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f97ff400640 1 -- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98080cdc20 msgr2=0x7f98080ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:26.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f97ff400640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98080cdc20 0x7f98080ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f97ff400640 1 -- 172.21.15.32:0/2496605659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98081f2270 con 0x7f98080af580 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f97ff400640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 0x7f9808156a10 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c4d0 tx=0x7f97f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0018070 con 0x7f98080af580 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f0007500 con 0x7f98080af580 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0004a20 con 0x7f98080af580 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98081f2550 con 0x7f98080af580 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.832+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98081580b0 con 0x7f98080af580 2024-03-06T07:40:26.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.834+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9808069510 con 0x7f98080af580 2024-03-06T07:40:26.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.835+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97f0004bc0 con 0x7f98080af580 2024-03-06T07:40:26.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.835+0000 7f9804c00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97e406b4d0 0x7f97e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:26.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.836+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f97f00895f0 con 0x7f98080af580 2024-03-06T07:40:26.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.836+0000 7f9807e00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97e406b4d0 0x7f97e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:26.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.836+0000 7f9807e00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97e406b4d0 0x7f97e406d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f97f8006fd0 tx=0x7f97f8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:26.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:26.840+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97f0058390 con 0x7f98080af580 2024-03-06T07:40:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.107+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9808068300 con 0x7f98080af580 2024-03-06T07:40:27.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.108+0000 7f9804c00640 1 -- 172.21.15.32:0/2496605659 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f97f0006e20 con 0x7f98080af580 2024-03-06T07:40:27.110 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:27.110 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:27.111 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97e406b4d0 msgr2=0x7f97e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97e406b4d0 0x7f97e406d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f97f8006fd0 tx=0x7f97f8008040 comp rx=0 tx=0).stop 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 msgr2=0x7f9808156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 0x7f9808156a10 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c4d0 tx=0x7f97f000c990 comp rx=0 tx=0).stop 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 shutdown_connections 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f97e406b4d0 0x7f97e406d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98080cdc20 0x7f98080ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98080af580 0x7f9808156a10 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 --2- 172.21.15.32:0/2496605659 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f98080af1a0 0x7f98081f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.110+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 >> 172.21.15.32:0/2496605659 conn(0x7f980806afb0 msgr2=0x7f98080cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.111+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 shutdown_connections 2024-03-06T07:40:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:27.111+0000 7f980ea00640 1 -- 172.21.15.32:0/2496605659 wait complete. 2024-03-06T07:40:27.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:27.332 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:28.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.550+0000 7f0495600640 1 Processor -- start 2024-03-06T07:40:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.551+0000 7f0495600640 1 -- start start 2024-03-06T07:40:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.552+0000 7f0495600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04900af1a0 0x7f049015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.552+0000 7f0495600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f04900cdc20 0x7f04900b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.552+0000 7f0495600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049015cc40 con 0x7f04900af580 2024-03-06T07:40:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.552+0000 7f0495600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049015cf20 con 0x7f04900cdc20 2024-03-06T07:40:28.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.552+0000 7f0495600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049015d200 con 0x7f04900af1a0 2024-03-06T07:40:28.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.552+0000 7f048f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f04900cdc20 0x7f04900b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048f400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f04900cdc20 0x7f04900b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44878/0 (socket says 172.21.15.32:44878) 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048f400640 1 -- 172.21.15.32:0/1411468525 learned_addr learned my addr 172.21.15.32:0/1411468525 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048ea00640 1 --2- 172.21.15.32:0/1411468525 >> v2:172.21.15.39:3300/0 conn(0x7f04900af1a0 0x7f049015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048f400640 1 -- 172.21.15.32:0/1411468525 >> v2:172.21.15.39:3300/0 conn(0x7f04900af1a0 msgr2=0x7f049015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048f400640 1 --2- 172.21.15.32:0/1411468525 >> v2:172.21.15.39:3300/0 conn(0x7f04900af1a0 0x7f049015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048f400640 1 -- 172.21.15.32:0/1411468525 >> v1:172.21.15.39:6789/0 conn(0x7f04900af580 legacy=0x7f04900b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.553+0000 7f048f400640 1 -- 172.21.15.32:0/1411468525 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049015d4e0 con 0x7f04900cdc20 2024-03-06T07:40:28.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.555+0000 7f048f400640 1 --2- 172.21.15.32:0/1411468525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900cdc20 0x7f04900b2dd0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f0490074960 tx=0x7f048802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2b3c6642685b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:28.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.555+0000 7f048d600640 1 -- 172.21.15.32:0/1411468525 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048802c9f0 con 0x7f04900cdc20 2024-03-06T07:40:28.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.555+0000 7f048d600640 1 -- 172.21.15.32:0/1411468525 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048802cb90 con 0x7f04900cdc20 2024-03-06T07:40:28.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.555+0000 7f048d600640 1 -- 172.21.15.32:0/1411468525 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04880343f0 con 0x7f04900cdc20 2024-03-06T07:40:28.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 -- 172.21.15.32:0/1411468525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900cdc20 msgr2=0x7f04900b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 --2- 172.21.15.32:0/1411468525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900cdc20 0x7f04900b2dd0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f0490074960 tx=0x7f048802be50 comp rx=0 tx=0).stop 2024-03-06T07:40:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 -- 172.21.15.32:0/1411468525 shutdown_connections 2024-03-06T07:40:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 --2- 172.21.15.32:0/1411468525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900cdc20 0x7f04900b2dd0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 --2- 172.21.15.32:0/1411468525 >> v2:172.21.15.39:3300/0 conn(0x7f04900af1a0 0x7f049015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 -- 172.21.15.32:0/1411468525 >> 172.21.15.32:0/1411468525 conn(0x7f049006afb0 msgr2=0x7f049006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.556+0000 7f0495600640 1 -- 172.21.15.32:0/1411468525 shutdown_connections 2024-03-06T07:40:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.557+0000 7f0495600640 1 -- 172.21.15.32:0/1411468525 wait complete. 2024-03-06T07:40:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.559+0000 7f0495600640 1 Processor -- start 2024-03-06T07:40:28.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.559+0000 7f0495600640 1 -- start start 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f0495600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 0x7f049009ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f0495600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f04900af580 0x7f04901503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f0495600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04900cdc20 0x7f0490158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f0495600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04900b36a0 con 0x7f04900af1a0 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f0495600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04900b3520 con 0x7f04900cdc20 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f0495600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04900b3820 con 0x7f04900af580 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f048ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 0x7f049009ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f048ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 0x7f049009ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44892/0 (socket says 172.21.15.32:44892) 2024-03-06T07:40:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.561+0000 7f048ea00640 1 -- 172.21.15.32:0/620869241 learned_addr learned my addr 172.21.15.32:0/620869241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048f400640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04900cdc20 0x7f0490158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048ea00640 1 -- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f04900af580 msgr2=0x7f04901503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048ea00640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f04900af580 0x7f04901503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048ea00640 1 -- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04900cdc20 msgr2=0x7f0490158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048ea00640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04900cdc20 0x7f0490158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048ea00640 1 -- 172.21.15.32:0/620869241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049009ac60 con 0x7f04900af1a0 2024-03-06T07:40:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048f400640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04900cdc20 0x7f0490158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f048ea00640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 0x7f049009ba20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f048000d800 tx=0x7f048000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048000b6c0 con 0x7f04900af1a0 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0480019070 con 0x7f04900af1a0 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.562+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048000ba80 con 0x7f04900af1a0 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.563+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049009af40 con 0x7f04900af1a0 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.563+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049009b500 con 0x7f04900af1a0 2024-03-06T07:40:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.564+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0490069510 con 0x7f04900af1a0 2024-03-06T07:40:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.565+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f048000bc20 con 0x7f04900af1a0 2024-03-06T07:40:28.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.566+0000 7f0486a00640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f047006b4d0 0x7f047006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:28.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.566+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f048008a7c0 con 0x7f04900af1a0 2024-03-06T07:40:28.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.566+0000 7f048e000640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f047006b4d0 0x7f047006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:28.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.567+0000 7f048e000640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f047006b4d0 0x7f047006d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f049015f740 tx=0x7f047c0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:28.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.571+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f048001c360 con 0x7f04900af1a0 2024-03-06T07:40:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.807+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f049004f3a0 con 0x7f04900af1a0 2024-03-06T07:40:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.808+0000 7f0486a00640 1 -- 172.21.15.32:0/620869241 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f0480059560 con 0x7f04900af1a0 2024-03-06T07:40:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:28.810 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f047006b4d0 msgr2=0x7f047006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f047006b4d0 0x7f047006d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f049015f740 tx=0x7f047c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 msgr2=0x7f049009ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 0x7f049009ba20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f048000d800 tx=0x7f048000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 shutdown_connections 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f047006b4d0 0x7f047006d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04900cdc20 0x7f0490158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f04900af580 0x7f04901503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 --2- 172.21.15.32:0/620869241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04900af1a0 0x7f049009ba20 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 >> 172.21.15.32:0/620869241 conn(0x7f049006afb0 msgr2=0x7f049015bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 shutdown_connections 2024-03-06T07:40:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:28.810+0000 7f0495600640 1 -- 172.21.15.32:0/620869241 wait complete. 2024-03-06T07:40:29.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:29.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:29.509 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:30.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:30.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.262+0000 7f76b3200640 1 Processor -- start 2024-03-06T07:40:30.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.262+0000 7f76b3200640 1 -- start start 2024-03-06T07:40:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.263+0000 7f76b3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76b4070f10 0x7f76b40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.263+0000 7f76b3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f76b40cce90 0x7f76b415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.263+0000 7f76b3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40714d0 con 0x7f76b4145300 2024-03-06T07:40:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.263+0000 7f76b3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b40cd300 con 0x7f76b4070f10 2024-03-06T07:40:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.263+0000 7f76b3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b40717b0 con 0x7f76b40cce90 2024-03-06T07:40:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76b4070f10 0x7f76b40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76b4070f10 0x7f76b40cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44898/0 (socket says 172.21.15.32:44898) 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 -- 172.21.15.32:0/2972639191 learned_addr learned my addr 172.21.15.32:0/2972639191 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76b2800640 1 --2- 172.21.15.32:0/2972639191 >> v2:172.21.15.39:3300/0 conn(0x7f76b40cce90 0x7f76b415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 -- 172.21.15.32:0/2972639191 >> v2:172.21.15.39:3300/0 conn(0x7f76b40cce90 msgr2=0x7f76b415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 --2- 172.21.15.32:0/2972639191 >> v2:172.21.15.39:3300/0 conn(0x7f76b40cce90 0x7f76b415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 -- 172.21.15.32:0/2972639191 >> v1:172.21.15.39:6789/0 conn(0x7f76b4145300 legacy=0x7f76b40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.264+0000 7f76a9400640 1 -- 172.21.15.32:0/2972639191 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b4071a90 con 0x7f76b4070f10 2024-03-06T07:40:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.266+0000 7f76a9400640 1 --2- 172.21.15.32:0/2972639191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b40cc950 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f76b40cc500 tx=0x7f769c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc864a3736f1cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.266+0000 7f76b1400640 1 -- 172.21.15.32:0/2972639191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c02e070 con 0x7f76b4070f10 2024-03-06T07:40:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.266+0000 7f76b1400640 1 -- 172.21.15.32:0/2972639191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769c037070 con 0x7f76b4070f10 2024-03-06T07:40:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.266+0000 7f76b1400640 1 -- 172.21.15.32:0/2972639191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c031a00 con 0x7f76b4070f10 2024-03-06T07:40:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.266+0000 7f76b3200640 1 -- 172.21.15.32:0/2972639191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 msgr2=0x7f76b40cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.266+0000 7f76b3200640 1 --2- 172.21.15.32:0/2972639191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b40cc950 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f76b40cc500 tx=0x7f769c02b880 comp rx=0 tx=0).stop 2024-03-06T07:40:30.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.267+0000 7f76b3200640 1 -- 172.21.15.32:0/2972639191 shutdown_connections 2024-03-06T07:40:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.267+0000 7f76b3200640 1 --2- 172.21.15.32:0/2972639191 >> v2:172.21.15.39:3300/0 conn(0x7f76b40cce90 0x7f76b415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.267+0000 7f76b3200640 1 --2- 172.21.15.32:0/2972639191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b40cc950 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.267+0000 7f76b3200640 1 -- 172.21.15.32:0/2972639191 >> 172.21.15.32:0/2972639191 conn(0x7f76b406b3d0 msgr2=0x7f76b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.267+0000 7f76b3200640 1 -- 172.21.15.32:0/2972639191 shutdown_connections 2024-03-06T07:40:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.267+0000 7f76b3200640 1 -- 172.21.15.32:0/2972639191 wait complete. 2024-03-06T07:40:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.268+0000 7f76b3200640 1 Processor -- start 2024-03-06T07:40:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 -- start start 2024-03-06T07:40:30.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76b40cce90 0x7f76b41e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76b4145300 0x7f76b41e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b415cab0 con 0x7f76b4070f10 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b41456e0 con 0x7f76b40cce90 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.269+0000 7f76b3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b415cc30 con 0x7f76b4145300 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b415b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44906/0 (socket says 172.21.15.32:44906) 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 -- 172.21.15.32:0/2949524503 learned_addr learned my addr 172.21.15.32:0/2949524503 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76a9400640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76b40cce90 0x7f76b41e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b2800640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76b4145300 0x7f76b41e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 -- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76b4145300 msgr2=0x7f76b41e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76b4145300 0x7f76b41e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 -- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76b40cce90 msgr2=0x7f76b41e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76b40cce90 0x7f76b41e4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b1e00640 1 -- 172.21.15.32:0/2949524503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b415db70 con 0x7f76b4070f10 2024-03-06T07:40:30.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.270+0000 7f76b2800640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76b4145300 0x7f76b41e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.271+0000 7f76b1e00640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b415b330 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f76a400b3e0 tx=0x7f76a400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.271+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4015020 con 0x7f76b4070f10 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.271+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4011850 con 0x7f76b4070f10 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.271+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4011c40 con 0x7f76b4070f10 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.271+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b415de10 con 0x7f76b4070f10 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.271+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b415e3d0 con 0x7f76b4070f10 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.272+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b406df00 con 0x7f76b4070f10 2024-03-06T07:40:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.273+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76a4018020 con 0x7f76b4070f10 2024-03-06T07:40:30.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.274+0000 7f76aaa00640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f769006b4d0 0x7f769006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:30.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.274+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76a408a1c0 con 0x7f76b4070f10 2024-03-06T07:40:30.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.275+0000 7f76a9400640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f769006b4d0 0x7f769006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:30.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.275+0000 7f76a9400640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f769006b4d0 0x7f769006d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f769c036000 tx=0x7f769c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:30.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.279+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76b406df00 con 0x7f76b4070f10 2024-03-06T07:40:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.519+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76b404f330 con 0x7f76b4070f10 2024-03-06T07:40:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.520+0000 7f76aaa00640 1 -- 172.21.15.32:0/2949524503 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f76a401f020 con 0x7f76b4070f10 2024-03-06T07:40:30.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:30.523 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:30.523 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f769006b4d0 msgr2=0x7f769006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f769006b4d0 0x7f769006d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f769c036000 tx=0x7f769c03d040 comp rx=0 tx=0).stop 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 msgr2=0x7f76b415b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b415b330 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f76a400b3e0 tx=0x7f76a400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 shutdown_connections 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f769006b4d0 0x7f769006d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f76b4145300 0x7f76b41e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76b40cce90 0x7f76b41e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 --2- 172.21.15.32:0/2949524503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4070f10 0x7f76b415b330 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.523+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 >> 172.21.15.32:0/2949524503 conn(0x7f76b406b3d0 msgr2=0x7f76b40b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.524+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 shutdown_connections 2024-03-06T07:40:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:30.524+0000 7f76b3200640 1 -- 172.21.15.32:0/2949524503 wait complete. 2024-03-06T07:40:30.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:30.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:31.511 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:31.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.002+0000 7feac7c00640 1 Processor -- start 2024-03-06T07:40:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.002+0000 7feac7c00640 1 -- start start 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.003+0000 7feac7c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feac00af580 0x7feac015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.003+0000 7feac7c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7feac00cdc20 0x7feac00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.003+0000 7feac7c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac015cc40 con 0x7feac00af1a0 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.003+0000 7feac7c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac015cf20 con 0x7feac00af580 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.003+0000 7feac7c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac015d200 con 0x7feac00cdc20 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feac5200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feac00af1a0 0x7feac00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56932/0 (socket says 172.21.15.32:56932) 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feac5200640 1 -- 172.21.15.32:0/1774104470 learned_addr learned my addr 172.21.15.32:0/1774104470 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feabfe00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feac00af580 0x7feac015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feac5c00640 1 --2- 172.21.15.32:0/1774104470 >> v2:172.21.15.39:3300/0 conn(0x7feac00cdc20 0x7feac00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feabf400640 1 -- 172.21.15.32:0/1774104470 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177931263 0 0) 0x7feac015cc40 con 0x7feac00af1a0 2024-03-06T07:40:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feabfe00640 1 -- 172.21.15.32:0/1774104470 >> v2:172.21.15.39:3300/0 conn(0x7feac00cdc20 msgr2=0x7feac00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feabfe00640 1 --2- 172.21.15.32:0/1774104470 >> v2:172.21.15.39:3300/0 conn(0x7feac00cdc20 0x7feac00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feabfe00640 1 -- 172.21.15.32:0/1774104470 >> v1:172.21.15.32:6789/0 conn(0x7feac00af1a0 legacy=0x7feac00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.004+0000 7feabfe00640 1 -- 172.21.15.32:0/1774104470 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac015d4e0 con 0x7feac00af580 2024-03-06T07:40:32.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.006+0000 7feabfe00640 1 --2- 172.21.15.32:0/1774104470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00af580 0x7feac015c600 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7feac006c0c0 tx=0x7feaa802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d82812f7011dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.006+0000 7feabf400640 1 -- 172.21.15.32:0/1774104470 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa80315a0 con 0x7feac00af580 2024-03-06T07:40:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.006+0000 7feabf400640 1 -- 172.21.15.32:0/1774104470 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8037070 con 0x7feac00af580 2024-03-06T07:40:32.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feabf400640 1 -- 172.21.15.32:0/1774104470 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8031960 con 0x7feac00af580 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feac7c00640 1 -- 172.21.15.32:0/1774104470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00af580 msgr2=0x7feac015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feac7c00640 1 --2- 172.21.15.32:0/1774104470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00af580 0x7feac015c600 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7feac006c0c0 tx=0x7feaa802b840 comp rx=0 tx=0).stop 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feac7c00640 1 -- 172.21.15.32:0/1774104470 shutdown_connections 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feac7c00640 1 --2- 172.21.15.32:0/1774104470 >> v2:172.21.15.39:3300/0 conn(0x7feac00cdc20 0x7feac00b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feac7c00640 1 --2- 172.21.15.32:0/1774104470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00af580 0x7feac015c600 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.007+0000 7feac7c00640 1 -- 172.21.15.32:0/1774104470 >> 172.21.15.32:0/1774104470 conn(0x7feac006afb0 msgr2=0x7feac006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.008+0000 7feac7c00640 1 -- 172.21.15.32:0/1774104470 shutdown_connections 2024-03-06T07:40:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.008+0000 7feac7c00640 1 -- 172.21.15.32:0/1774104470 wait complete. 2024-03-06T07:40:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.009+0000 7feac7c00640 1 Processor -- start 2024-03-06T07:40:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.009+0000 7feac7c00640 1 -- start start 2024-03-06T07:40:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.010+0000 7feac7c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00af1a0 0x7feac0154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.010+0000 7feac7c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00af580 0x7feac01f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.010+0000 7feac7c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 0x7feac009b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.010+0000 7feac7c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac00b36a0 con 0x7feac00cdc20 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.010+0000 7feac7c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac00b3520 con 0x7feac00af580 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.010+0000 7feac7c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac00b3820 con 0x7feac00af1a0 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 0x7feac009b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 0x7feac009b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44926/0 (socket says 172.21.15.32:44926) 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 -- 172.21.15.32:0/1078124915 learned_addr learned my addr 172.21.15.32:0/1078124915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5200640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00af1a0 0x7feac0154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feabfe00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00af580 0x7feac01f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 -- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00af1a0 msgr2=0x7feac0154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00af1a0 0x7feac0154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 -- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00af580 msgr2=0x7feac01f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00af580 0x7feac01f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feac5c00640 1 -- 172.21.15.32:0/1078124915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac01f2360 con 0x7feac00cdc20 2024-03-06T07:40:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.011+0000 7feabfe00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00af580 0x7feac01f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.012+0000 7feac5c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 0x7feac009b640 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7feab800d5d0 tx=0x7feab800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.012+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab80125f0 con 0x7feac00cdc20 2024-03-06T07:40:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.012+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab8019070 con 0x7feac00cdc20 2024-03-06T07:40:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.012+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab80129f0 con 0x7feac00cdc20 2024-03-06T07:40:32.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.012+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac01f2640 con 0x7feac00cdc20 2024-03-06T07:40:32.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.012+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac0158530 con 0x7feac00cdc20 2024-03-06T07:40:32.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.013+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac0069510 con 0x7feac00cdc20 2024-03-06T07:40:32.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.015+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feab8012b90 con 0x7feac00cdc20 2024-03-06T07:40:32.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.016+0000 7feabcc00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea9c06b500 0x7fea9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:32.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.016+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7feab808a620 con 0x7feac00cdc20 2024-03-06T07:40:32.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.016+0000 7feac5200640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea9c06b500 0x7fea9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.017+0000 7feac5200640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea9c06b500 0x7fea9c06d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7feab00012e0 tx=0x7feab0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:32.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.020+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feab8059470 con 0x7feac00cdc20 2024-03-06T07:40:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.263+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feac004f3a0 con 0x7feac00cdc20 2024-03-06T07:40:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.263+0000 7feabcc00640 1 -- 172.21.15.32:0/1078124915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7feab8058d70 con 0x7feac00cdc20 2024-03-06T07:40:32.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:32.266 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea9c06b500 msgr2=0x7fea9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea9c06b500 0x7fea9c06d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7feab00012e0 tx=0x7feab0009040 comp rx=0 tx=0).stop 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 msgr2=0x7feac009b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 0x7feac009b640 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7feab800d5d0 tx=0x7feab800da90 comp rx=0 tx=0).stop 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 shutdown_connections 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea9c06b500 0x7fea9c06d9c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac00cdc20 0x7feac009b640 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00af580 0x7feac01f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 --2- 172.21.15.32:0/1078124915 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00af1a0 0x7feac0154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 >> 172.21.15.32:0/1078124915 conn(0x7feac006afb0 msgr2=0x7feac0072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 shutdown_connections 2024-03-06T07:40:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:32.266+0000 7feac7c00640 1 -- 172.21.15.32:0/1078124915 wait complete. 2024-03-06T07:40:32.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:32.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:33.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.707+0000 7f6c77e00640 1 Processor -- start 2024-03-06T07:40:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.707+0000 7f6c77e00640 1 -- start start 2024-03-06T07:40:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c77e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c78145300 0x7f6c780b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c77e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78070f10 0x7f6c780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c77e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c780714d0 con 0x7f6c780cce90 2024-03-06T07:40:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c77e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c780cd300 con 0x7f6c78145300 2024-03-06T07:40:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c77e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c780717b0 con 0x7f6c78070f10 2024-03-06T07:40:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c76a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c78145300 0x7f6c780b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c76a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c78145300 0x7f6c780b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44932/0 (socket says 172.21.15.32:44932) 2024-03-06T07:40:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.709+0000 7f6c76a00640 1 -- 172.21.15.32:0/3715193955 learned_addr learned my addr 172.21.15.32:0/3715193955 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.710+0000 7f6c6e000640 1 --2- 172.21.15.32:0/3715193955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78070f10 0x7f6c780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.710+0000 7f6c76a00640 1 -- 172.21.15.32:0/3715193955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78070f10 msgr2=0x7f6c780cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.710+0000 7f6c76a00640 1 --2- 172.21.15.32:0/3715193955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78070f10 0x7f6c780cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.710+0000 7f6c76a00640 1 -- 172.21.15.32:0/3715193955 >> v1:172.21.15.39:6789/0 conn(0x7f6c780cce90 legacy=0x7f6c780cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.710+0000 7f6c76a00640 1 -- 172.21.15.32:0/3715193955 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78071a90 con 0x7f6c78145300 2024-03-06T07:40:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.711+0000 7f6c76a00640 1 --2- 172.21.15.32:0/3715193955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78145300 0x7f6c780b49a0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009870 tx=0x7f6c6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15332c4650733747 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.711+0000 7f6c76000640 1 -- 172.21.15.32:0/3715193955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68034070 con 0x7f6c78145300 2024-03-06T07:40:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c76000640 1 -- 172.21.15.32:0/3715193955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c68030840 con 0x7f6c78145300 2024-03-06T07:40:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c76000640 1 -- 172.21.15.32:0/3715193955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68030c10 con 0x7f6c78145300 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 -- 172.21.15.32:0/3715193955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78145300 msgr2=0x7f6c780b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 --2- 172.21.15.32:0/3715193955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78145300 0x7f6c780b49a0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009870 tx=0x7f6c6802b980 comp rx=0 tx=0).stop 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 -- 172.21.15.32:0/3715193955 shutdown_connections 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 --2- 172.21.15.32:0/3715193955 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78070f10 0x7f6c780cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 --2- 172.21.15.32:0/3715193955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78145300 0x7f6c780b49a0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 -- 172.21.15.32:0/3715193955 >> 172.21.15.32:0/3715193955 conn(0x7f6c7806b3d0 msgr2=0x7f6c7806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 -- 172.21.15.32:0/3715193955 shutdown_connections 2024-03-06T07:40:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.712+0000 7f6c77e00640 1 -- 172.21.15.32:0/3715193955 wait complete. 2024-03-06T07:40:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.715+0000 7f6c77e00640 1 Processor -- start 2024-03-06T07:40:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.715+0000 7f6c77e00640 1 -- start start 2024-03-06T07:40:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 0x7f6c7813cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c77e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c780cce90 0x7f6c7815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c77e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78145300 0x7f6c780a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c77e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c781456e0 con 0x7f6c78070f10 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c77e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c780b4ee0 con 0x7f6c780cce90 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c77e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7815d580 con 0x7f6c78145300 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 0x7f6c7813cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 0x7f6c7813cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44948/0 (socket says 172.21.15.32:44948) 2024-03-06T07:40:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.717+0000 7f6c76a00640 1 -- 172.21.15.32:0/2877568391 learned_addr learned my addr 172.21.15.32:0/2877568391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c6e000640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c780cce90 0x7f6c7815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c76a00640 1 -- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78145300 msgr2=0x7f6c780a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c76a00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78145300 0x7f6c780a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c76a00640 1 -- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c780cce90 msgr2=0x7f6c7815b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c76a00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c780cce90 0x7f6c7815b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c76a00640 1 -- 172.21.15.32:0/2877568391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c780aa050 con 0x7f6c78070f10 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c76a00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 0x7f6c7813cb90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f6c68000c00 tx=0x7f6c680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68034040 con 0x7f6c78070f10 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c68039d90 con 0x7f6c78070f10 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.718+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c780aa330 con 0x7f6c78070f10 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.719+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68043610 con 0x7f6c78070f10 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.719+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c780ad5f0 con 0x7f6c78070f10 2024-03-06T07:40:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.720+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7804f330 con 0x7f6c78070f10 2024-03-06T07:40:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.721+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c6804a060 con 0x7f6c78070f10 2024-03-06T07:40:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.722+0000 7f6c6f400640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6c5406b520 0x7f6c5406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.722+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6c680ac310 con 0x7f6c78070f10 2024-03-06T07:40:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.722+0000 7f6c6e000640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6c5406b520 0x7f6c5406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.723+0000 7f6c6e000640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6c5406b520 0x7f6c5406d9e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6c78141500 tx=0x7f6c60008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.731+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c6803e070 con 0x7f6c78070f10 2024-03-06T07:40:33.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.972+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c780aa6c0 con 0x7f6c78070f10 2024-03-06T07:40:33.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.972+0000 7f6c6f400640 1 -- 172.21.15.32:0/2877568391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f6c6802d330 con 0x7f6c78070f10 2024-03-06T07:40:33.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:33.975 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:33.975 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:33.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6c5406b520 msgr2=0x7f6c5406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6c5406b520 0x7f6c5406d9e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6c78141500 tx=0x7f6c60008040 comp rx=0 tx=0).stop 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 msgr2=0x7f6c7813cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 0x7f6c7813cb90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f6c68000c00 tx=0x7f6c680046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 shutdown_connections 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6c5406b520 0x7f6c5406d9e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6c78145300 0x7f6c780a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c780cce90 0x7f6c7815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 --2- 172.21.15.32:0/2877568391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c78070f10 0x7f6c7813cb90 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 >> 172.21.15.32:0/2877568391 conn(0x7f6c7806b3d0 msgr2=0x7f6c78072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 shutdown_connections 2024-03-06T07:40:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:33.975+0000 7f6c77e00640 1 -- 172.21.15.32:0/2877568391 wait complete. 2024-03-06T07:40:34.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:34.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:35.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:35.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.415+0000 7efce0c00640 1 Processor -- start 2024-03-06T07:40:35.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.416+0000 7efce0c00640 1 -- start start 2024-03-06T07:40:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.419+0000 7efce0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efcdc15e200 0x7efcdc0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.419+0000 7efce0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc0afdf0 0x7efcdc0af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.419+0000 7efce0c00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc06bec0 con 0x7efcdc06afb0 2024-03-06T07:40:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.419+0000 7efce0c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0b0670 con 0x7efcdc15e200 2024-03-06T07:40:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.419+0000 7efce0c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0ae380 con 0x7efcdc0afdf0 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.420+0000 7efcda800640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efcdc06afb0 0x7efcdc06dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56948/0 (socket says 172.21.15.32:56948) 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.420+0000 7efcda800640 1 -- 172.21.15.32:0/565043121 learned_addr learned my addr 172.21.15.32:0/565043121 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.420+0000 7efcd1400640 1 --2- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc0afdf0 0x7efcdc0af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.420+0000 7efcd9e00640 1 --2- 172.21.15.32:0/565043121 >> v2:172.21.15.39:3300/0 conn(0x7efcdc15e200 0x7efcdc0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.420+0000 7efcd9400640 1 -- 172.21.15.32:0/565043121 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507376009 0 0) 0x7efcdc06bec0 con 0x7efcdc06afb0 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.420+0000 7efcd9400640 1 -- 172.21.15.32:0/565043121 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc0003520 con 0x7efcdc06afb0 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.421+0000 7efcd9e00640 1 -- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc0afdf0 msgr2=0x7efcdc0af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:35.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.421+0000 7efcd9e00640 1 --2- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc0afdf0 0x7efcdc0af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.421+0000 7efcd9400640 1 -- 172.21.15.32:0/565043121 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3654886100 0 0) 0x7efcc0003520 con 0x7efcdc06afb0 2024-03-06T07:40:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.421+0000 7efcd9e00640 1 -- 172.21.15.32:0/565043121 >> v1:172.21.15.32:6789/0 conn(0x7efcdc06afb0 legacy=0x7efcdc06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.421+0000 7efcd9e00640 1 -- 172.21.15.32:0/565043121 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0ae660 con 0x7efcdc15e200 2024-03-06T07:40:35.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.422+0000 7efcd9e00640 1 --2- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc15e200 0x7efcdc0b3de0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7efccc009d90 tx=0x7efccc02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=745cd84ea58871cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efcd9400640 1 -- 172.21.15.32:0/565043121 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc039070 con 0x7efcdc15e200 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efcd9400640 1 -- 172.21.15.32:0/565043121 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc02c8a0 con 0x7efcdc15e200 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efcd9400640 1 -- 172.21.15.32:0/565043121 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4001690 con 0x7efcdc15e200 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 -- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc15e200 msgr2=0x7efcdc0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 --2- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc15e200 0x7efcdc0b3de0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7efccc009d90 tx=0x7efccc02bd70 comp rx=0 tx=0).stop 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 -- 172.21.15.32:0/565043121 shutdown_connections 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 --2- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc0afdf0 0x7efcdc0af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 --2- 172.21.15.32:0/565043121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc15e200 0x7efcdc0b3de0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 -- 172.21.15.32:0/565043121 >> 172.21.15.32:0/565043121 conn(0x7efcdc0b2ae0 msgr2=0x7efcdc0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.423+0000 7efce0c00640 1 -- 172.21.15.32:0/565043121 shutdown_connections 2024-03-06T07:40:35.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.424+0000 7efce0c00640 1 -- 172.21.15.32:0/565043121 wait complete. 2024-03-06T07:40:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.425+0000 7efce0c00640 1 Processor -- start 2024-03-06T07:40:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.425+0000 7efce0c00640 1 -- start start 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.426+0000 7efce0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc06afb0 0x7efcdc146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.426+0000 7efce0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 0x7efcdc15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.426+0000 7efce0c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc15e200 0x7efcdc154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.426+0000 7efce0c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0af680 con 0x7efcdc0afdf0 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.426+0000 7efce0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0aec20 con 0x7efcdc06afb0 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.426+0000 7efce0c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc06e2c0 con 0x7efcdc15e200 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 0x7efcdc15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 0x7efcdc15c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44956/0 (socket says 172.21.15.32:44956) 2024-03-06T07:40:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 -- 172.21.15.32:0/1448312276 learned_addr learned my addr 172.21.15.32:0/1448312276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd9e00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc06afb0 0x7efcdc146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcda800640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc15e200 0x7efcdc154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 -- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc15e200 msgr2=0x7efcdc154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc15e200 0x7efcdc154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 -- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc06afb0 msgr2=0x7efcdc146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc06afb0 0x7efcdc146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd1400640 1 -- 172.21.15.32:0/1448312276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc159000 con 0x7efcdc0afdf0 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.427+0000 7efcd9e00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc06afb0 0x7efcdc146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:40:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.428+0000 7efcd1400640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 0x7efcdc15c2f0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7efcc400e980 tx=0x7efcc400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.428+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4017070 con 0x7efcdc0afdf0 2024-03-06T07:40:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.428+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc400ca00 con 0x7efcdc0afdf0 2024-03-06T07:40:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.428+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc400cdd0 con 0x7efcdc0afdf0 2024-03-06T07:40:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.428+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc1592e0 con 0x7efcdc0afdf0 2024-03-06T07:40:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.428+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc159b80 con 0x7efcdc0afdf0 2024-03-06T07:40:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.429+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc1594b0 con 0x7efcdc0afdf0 2024-03-06T07:40:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.430+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcc401a070 con 0x7efcdc0afdf0 2024-03-06T07:40:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.431+0000 7efcd2a00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.431+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcc408a360 con 0x7efcdc0afdf0 2024-03-06T07:40:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.432+0000 7efcd9e00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:35.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.432+0000 7efcd9e00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcb806b4d0 0x7efcb806d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7efccc036000 tx=0x7efccc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:35.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.440+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcc4059070 con 0x7efcdc0afdf0 2024-03-06T07:40:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:35.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.679+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcdc04f3a0 con 0x7efcdc0afdf0 2024-03-06T07:40:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.680+0000 7efcd2a00640 1 -- 172.21.15.32:0/1448312276 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7efcc4058970 con 0x7efcdc0afdf0 2024-03-06T07:40:35.683 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:35.683 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcb806b4d0 msgr2=0x7efcb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcb806b4d0 0x7efcb806d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7efccc036000 tx=0x7efccc030000 comp rx=0 tx=0).stop 2024-03-06T07:40:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 msgr2=0x7efcdc15c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 0x7efcdc15c2f0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7efcc400e980 tx=0x7efcc400ee40 comp rx=0 tx=0).stop 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 shutdown_connections 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7efcdc15e200 0x7efcdc154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcdc0afdf0 0x7efcdc15c2f0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 --2- 172.21.15.32:0/1448312276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc06afb0 0x7efcdc146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 >> 172.21.15.32:0/1448312276 conn(0x7efcdc0b2ae0 msgr2=0x7efcdc162150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 shutdown_connections 2024-03-06T07:40:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:35.683+0000 7efce0c00640 1 -- 172.21.15.32:0/1448312276 wait complete. 2024-03-06T07:40:35.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:35.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:36.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.126+0000 7f5cafe00640 1 Processor -- start 2024-03-06T07:40:37.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.126+0000 7f5cafe00640 1 -- start start 2024-03-06T07:40:37.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.128+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 0x7f5ca815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.128+0000 7f5cafe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca815cc40 con 0x7f5ca80cdc20 2024-03-06T07:40:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.128+0000 7f5cafe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca815cf20 con 0x7f5ca80af1a0 2024-03-06T07:40:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.128+0000 7f5cafe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca815d200 con 0x7f5ca80af580 2024-03-06T07:40:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.128+0000 7f5cade00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5ca80cdc20 0x7f5ca80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56958/0 (socket says 172.21.15.32:56958) 2024-03-06T07:40:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.128+0000 7f5cade00640 1 -- 172.21.15.32:0/2221545287 learned_addr learned my addr 172.21.15.32:0/2221545287 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:37.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5caca00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 0x7f5ca815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003389613 0 0) 0x7f5ca815cc40 con 0x7f5ca80cdc20 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c90003520 con 0x7f5ca80cdc20 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515897711 0 0) 0x7f5c90003520 con 0x7f5ca80cdc20 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 msgr2=0x7f5ca815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 0x7f5ca815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 >> v1:172.21.15.39:6789/0 conn(0x7f5ca80af1a0 legacy=0x7f5ca80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca815d4e0 con 0x7f5ca80cdc20 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f5ca00019f0 con 0x7f5ca80cdc20 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 >> v1:172.21.15.32:6789/0 conn(0x7f5ca80cdc20 legacy=0x7f5ca80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c90003a50 0x7f5c90003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 0x7f5c90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5c900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca815d200 con 0x7f5c90004d40 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9000bbd0 con 0x7f5c90003a50 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.129+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9000bdd0 con 0x7f5ca80af580 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cade00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5c900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5caca00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 0x7f5c90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cad400640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c90003a50 0x7f5c90003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cade00640 1 -- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 msgr2=0x7f5c90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cade00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca80af580 0x7f5c90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cade00640 1 -- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c90003a50 msgr2=0x7f5c90003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cade00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c90003a50 0x7f5c90003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.130+0000 7f5cade00640 1 -- 172.21.15.32:0/2221545287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90003520 con 0x7f5c90004d40 2024-03-06T07:40:37.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.131+0000 7f5cade00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5c900051c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000a7b0 tx=0x7f5c90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d775c1f80a4f672f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca000a3d0 con 0x7f5c90004d40 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca0034650 con 0x7f5c90004d40 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5ca7e00640 1 -- 172.21.15.32:0/2221545287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0034a80 con 0x7f5c90004d40 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5cafe00640 1 -- 172.21.15.32:0/2221545287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 msgr2=0x7f5c900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5cafe00640 1 --2- 172.21.15.32:0/2221545287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5c900051c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000a7b0 tx=0x7f5c90005830 comp rx=0 tx=0).stop 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5cade00640 1 -- 172.21.15.32:0/2221545287 reap_dead start 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5cafe00640 1 -- 172.21.15.32:0/2221545287 shutdown_connections 2024-03-06T07:40:37.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.132+0000 7f5cafe00640 1 -- 172.21.15.32:0/2221545287 >> 172.21.15.32:0/2221545287 conn(0x7f5ca806afb0 msgr2=0x7f5ca806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.133+0000 7f5cafe00640 1 -- 172.21.15.32:0/2221545287 shutdown_connections 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.133+0000 7f5cafe00640 1 -- 172.21.15.32:0/2221545287 wait complete. 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.134+0000 7f5cafe00640 1 Processor -- start 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.134+0000 7f5cafe00640 1 -- start start 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.135+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5ca81f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.135+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca809d1c0 0x7f5ca814f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.135+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca814f900 0x7f5ca80ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.135+0000 7f5cafe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca815c600 con 0x7f5c90004d40 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.135+0000 7f5cafe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80b36f0 con 0x7f5ca814f900 2024-03-06T07:40:37.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.135+0000 7f5cafe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca815c780 con 0x7f5ca809d1c0 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5ca81f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5ca81f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44976/0 (socket says 172.21.15.32:44976) 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 -- 172.21.15.32:0/3862119073 learned_addr learned my addr 172.21.15.32:0/3862119073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cade00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca814f900 0x7f5ca80ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5caca00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca809d1c0 0x7f5ca814f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 -- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca809d1c0 msgr2=0x7f5ca814f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca809d1c0 0x7f5ca814f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 -- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca814f900 msgr2=0x7f5ca80ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca814f900 0x7f5ca80ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cad400640 1 -- 172.21.15.32:0/3862119073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80c95c0 con 0x7f5c90004d40 2024-03-06T07:40:37.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.136+0000 7f5cade00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca814f900 0x7f5ca80ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.137+0000 7f5cad400640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5ca81f4140 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00e580 tx=0x7f5c9c00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.137+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c0052e0 con 0x7f5c90004d40 2024-03-06T07:40:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.137+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c018070 con 0x7f5c90004d40 2024-03-06T07:40:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.137+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c003810 con 0x7f5c90004d40 2024-03-06T07:40:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.137+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca80c98a0 con 0x7f5c90004d40 2024-03-06T07:40:37.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.137+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca80c9e60 con 0x7f5c90004d40 2024-03-06T07:40:37.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.138+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca8069510 con 0x7f5c90004d40 2024-03-06T07:40:37.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.139+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c9c0039b0 con 0x7f5c90004d40 2024-03-06T07:40:37.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.140+0000 7f5ca5600640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c8806b4d0 0x7f5c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.140+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5c9c089f90 con 0x7f5c90004d40 2024-03-06T07:40:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.141+0000 7f5caca00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c8806b4d0 0x7f5c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.142+0000 7f5caca00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c8806b4d0 0x7f5c8806d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80cb5e0 tx=0x7f5c94008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.145+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c9c058d40 con 0x7f5c90004d40 2024-03-06T07:40:37.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.391+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca80ace50 con 0x7f5c90004d40 2024-03-06T07:40:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.392+0000 7f5ca5600640 1 -- 172.21.15.32:0/3862119073 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f5c9c058640 con 0x7f5c90004d40 2024-03-06T07:40:37.394 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:37.395 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c8806b4d0 msgr2=0x7f5c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c8806b4d0 0x7f5c8806d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80cb5e0 tx=0x7f5c94008040 comp rx=0 tx=0).stop 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 msgr2=0x7f5ca81f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5ca81f4140 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00e580 tx=0x7f5c9c00ea40 comp rx=0 tx=0).stop 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 shutdown_connections 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f5c8806b4d0 0x7f5c8806d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca814f900 0x7f5ca80ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f5ca809d1c0 0x7f5ca814f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 --2- 172.21.15.32:0/3862119073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c90004d40 0x7f5ca81f4140 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.394+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 >> 172.21.15.32:0/3862119073 conn(0x7f5ca806afb0 msgr2=0x7f5ca8073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.395+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 shutdown_connections 2024-03-06T07:40:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:37.395+0000 7f5cafe00640 1 -- 172.21.15.32:0/3862119073 wait complete. 2024-03-06T07:40:37.517 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:37.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:37.617 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:38.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.835+0000 7ff8b3200640 1 Processor -- start 2024-03-06T07:40:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.835+0000 7ff8b3200640 1 -- start start 2024-03-06T07:40:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.836+0000 7ff8b3200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff8b40d0e50 0x7ff8b4168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.836+0000 7ff8b3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40b3710 con 0x7ff8b415e0b0 2024-03-06T07:40:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.836+0000 7ff8b3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40b39f0 con 0x7ff8b40d4a10 2024-03-06T07:40:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.836+0000 7ff8b3200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b40b09e0 con 0x7ff8b40d0e50 2024-03-06T07:40:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.836+0000 7ff8b1e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff8b415e0b0 0x7ff8b40cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53182/0 (socket says 172.21.15.32:53182) 2024-03-06T07:40:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.836+0000 7ff8b1e00640 1 -- 172.21.15.32:0/2254824410 learned_addr learned my addr 172.21.15.32:0/2254824410 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b2800640 1 --2- 172.21.15.32:0/2254824410 >> v2:172.21.15.32:3300/0 conn(0x7ff8b40d0e50 0x7ff8b4168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b1400640 1 -- 172.21.15.32:0/2254824410 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635343886 0 0) 0x7ff8b40b3710 con 0x7ff8b415e0b0 2024-03-06T07:40:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b1400640 1 -- 172.21.15.32:0/2254824410 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff898003520 con 0x7ff8b415e0b0 2024-03-06T07:40:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b1400640 1 -- 172.21.15.32:0/2254824410 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445857589 0 0) 0x7ff8b40b39f0 con 0x7ff8b40d4a10 2024-03-06T07:40:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b2800640 1 -- 172.21.15.32:0/2254824410 >> v1:172.21.15.39:6789/0 conn(0x7ff8b40d4a10 legacy=0x7ff8b40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b2800640 1 -- 172.21.15.32:0/2254824410 >> v1:172.21.15.32:6789/0 conn(0x7ff8b415e0b0 legacy=0x7ff8b40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.837+0000 7ff8b2800640 1 -- 172.21.15.32:0/2254824410 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40b3cd0 con 0x7ff8b40d0e50 2024-03-06T07:40:38.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.839+0000 7ff8b2800640 1 --2- 172.21.15.32:0/2254824410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b40d0e50 0x7ff8b4168c50 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009d90 tx=0x7ff8ac02bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716df7f88faf93ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:38.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.839+0000 7ff8b1400640 1 -- 172.21.15.32:0/2254824410 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac039070 con 0x7ff8b40d0e50 2024-03-06T07:40:38.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.839+0000 7ff8b1400640 1 -- 172.21.15.32:0/2254824410 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac02c860 con 0x7ff8b40d0e50 2024-03-06T07:40:38.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.839+0000 7ff8b1400640 1 -- 172.21.15.32:0/2254824410 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4001670 con 0x7ff8b40d0e50 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.839+0000 7ff8b3200640 1 -- 172.21.15.32:0/2254824410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b40d0e50 msgr2=0x7ff8b4168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.839+0000 7ff8b3200640 1 --2- 172.21.15.32:0/2254824410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b40d0e50 0x7ff8b4168c50 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009d90 tx=0x7ff8ac02bd30 comp rx=0 tx=0).stop 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.840+0000 7ff8b3200640 1 -- 172.21.15.32:0/2254824410 shutdown_connections 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.840+0000 7ff8b3200640 1 --2- 172.21.15.32:0/2254824410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b40d0e50 0x7ff8b4168c50 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.840+0000 7ff8b3200640 1 -- 172.21.15.32:0/2254824410 >> 172.21.15.32:0/2254824410 conn(0x7ff8b40d3d30 msgr2=0x7ff8b40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.840+0000 7ff8b3200640 1 -- 172.21.15.32:0/2254824410 shutdown_connections 2024-03-06T07:40:38.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.840+0000 7ff8b3200640 1 -- 172.21.15.32:0/2254824410 wait complete. 2024-03-06T07:40:38.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.841+0000 7ff8b3200640 1 Processor -- start 2024-03-06T07:40:38.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 -- start start 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b40d0e50 0x7ff8b415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b40d4a10 0x7ff8b40b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 0x7ff8b415be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b415e490 con 0x7ff8b415e0b0 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40cfa20 con 0x7ff8b40d0e50 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.842+0000 7ff8b3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40cfc60 con 0x7ff8b40d4a10 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 0x7ff8b415be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 0x7ff8b415be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44138/0 (socket says 172.21.15.32:44138) 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 -- 172.21.15.32:0/1121069029 learned_addr learned my addr 172.21.15.32:0/1121069029 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b1e00640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b40d0e50 0x7ff8b415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8abe00640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b40d4a10 0x7ff8b40b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 -- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b40d4a10 msgr2=0x7ff8b40b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b40d4a10 0x7ff8b40b8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 -- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b40d0e50 msgr2=0x7ff8b415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b40d0e50 0x7ff8b415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.843+0000 7ff8b2800640 1 -- 172.21.15.32:0/1121069029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b415ff40 con 0x7ff8b415e0b0 2024-03-06T07:40:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.844+0000 7ff8b2800640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 0x7ff8b415be20 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac037040 tx=0x7ff8ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.844+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac039040 con 0x7ff8b415e0b0 2024-03-06T07:40:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.844+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac034d10 con 0x7ff8b415e0b0 2024-03-06T07:40:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.844+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac009450 con 0x7ff8b415e0b0 2024-03-06T07:40:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.844+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b415eea0 con 0x7ff8b415e0b0 2024-03-06T07:40:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.844+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b415f460 con 0x7ff8b415e0b0 2024-03-06T07:40:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.846+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b404f330 con 0x7ff8b415e0b0 2024-03-06T07:40:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.846+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8ac003bd0 con 0x7ff8b415e0b0 2024-03-06T07:40:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.847+0000 7ff8aa000640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.847+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff8ac0acba0 con 0x7ff8b415e0b0 2024-03-06T07:40:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.848+0000 7ff8b1e00640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.848+0000 7ff8b1e00640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89006b4d0 0x7ff89006d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000c00 tx=0x7ff8a40018f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:38.852+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8ac04e070 con 0x7ff8b415e0b0 2024-03-06T07:40:39.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.091+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8b40b35a0 con 0x7ff8b415e0b0 2024-03-06T07:40:39.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.092+0000 7ff8aa000640 1 -- 172.21.15.32:0/1121069029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7ff8ac07b9d0 con 0x7ff8b415e0b0 2024-03-06T07:40:39.095 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89006b4d0 msgr2=0x7ff89006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89006b4d0 0x7ff89006d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000c00 tx=0x7ff8a40018f0 comp rx=0 tx=0).stop 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 msgr2=0x7ff8b415be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 0x7ff8b415be20 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac037040 tx=0x7ff8ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 shutdown_connections 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b415e0b0 0x7ff8b415be20 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7ff8b40d4a10 0x7ff8b40b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 --2- 172.21.15.32:0/1121069029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8b40d0e50 0x7ff8b415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.095+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 >> 172.21.15.32:0/1121069029 conn(0x7ff8b40d3d30 msgr2=0x7ff8b40b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.096+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 shutdown_connections 2024-03-06T07:40:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:39.096+0000 7ff8b3200640 1 -- 172.21.15.32:0/1121069029 wait complete. 2024-03-06T07:40:39.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:39.317 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:39.519 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:40.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:40.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.539+0000 7f3b2a600640 1 Processor -- start 2024-03-06T07:40:40.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.539+0000 7f3b2a600640 1 -- start start 2024-03-06T07:40:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b2a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 0x7f3b240b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b2a600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3b240cd060 0x7f3b2415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b2a600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240714d0 con 0x7f3b24070f10 2024-03-06T07:40:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b2a600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240717b0 con 0x7f3b240cd060 2024-03-06T07:40:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b2a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24071a90 con 0x7f3b24145300 2024-03-06T07:40:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b29200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 0x7f3b240b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.541+0000 7f3b29200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 0x7f3b240b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:37114/0 (socket says 172.21.15.32:37114) 2024-03-06T07:40:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.542+0000 7f3b29200640 1 -- 172.21.15.32:0/3710821133 learned_addr learned my addr 172.21.15.32:0/3710821133 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.542+0000 7f3b29c00640 1 --2- 172.21.15.32:0/3710821133 >> v2:172.21.15.39:3300/0 conn(0x7f3b240cd060 0x7f3b2415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.542+0000 7f3b29200640 1 -- 172.21.15.32:0/3710821133 >> v2:172.21.15.39:3300/0 conn(0x7f3b240cd060 msgr2=0x7f3b2415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.542+0000 7f3b29200640 1 --2- 172.21.15.32:0/3710821133 >> v2:172.21.15.39:3300/0 conn(0x7f3b240cd060 0x7f3b2415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.542+0000 7f3b29200640 1 -- 172.21.15.32:0/3710821133 >> v1:172.21.15.39:6789/0 conn(0x7f3b24070f10 legacy=0x7f3b240cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.542+0000 7f3b29200640 1 -- 172.21.15.32:0/3710821133 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24071d70 con 0x7f3b24145300 2024-03-06T07:40:40.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.544+0000 7f3b29200640 1 --2- 172.21.15.32:0/3710821133 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 0x7f3b240b49a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a3772aed3b8cb42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:40.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b23400640 1 -- 172.21.15.32:0/3710821133 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18034070 con 0x7f3b24145300 2024-03-06T07:40:40.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b23400640 1 -- 172.21.15.32:0/3710821133 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b18030840 con 0x7f3b24145300 2024-03-06T07:40:40.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b23400640 1 -- 172.21.15.32:0/3710821133 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18030c10 con 0x7f3b24145300 2024-03-06T07:40:40.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b2a600640 1 -- 172.21.15.32:0/3710821133 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 msgr2=0x7f3b240b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3710821133 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 0x7f3b240b49a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1802b980 comp rx=0 tx=0).stop 2024-03-06T07:40:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b2a600640 1 -- 172.21.15.32:0/3710821133 shutdown_connections 2024-03-06T07:40:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3710821133 >> v2:172.21.15.39:3300/0 conn(0x7f3b240cd060 0x7f3b2415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3710821133 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24145300 0x7f3b240b49a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.545+0000 7f3b2a600640 1 -- 172.21.15.32:0/3710821133 >> 172.21.15.32:0/3710821133 conn(0x7f3b2406b3d0 msgr2=0x7f3b2406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.546+0000 7f3b2a600640 1 -- 172.21.15.32:0/3710821133 shutdown_connections 2024-03-06T07:40:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.546+0000 7f3b2a600640 1 -- 172.21.15.32:0/3710821133 wait complete. 2024-03-06T07:40:40.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.547+0000 7f3b2a600640 1 Processor -- start 2024-03-06T07:40:40.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.547+0000 7f3b2a600640 1 -- start start 2024-03-06T07:40:40.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.548+0000 7f3b2a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24070f10 0x7f3b2413f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.548+0000 7f3b2a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b240cd060 0x7f3b2413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.548+0000 7f3b2a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 0x7f3b2413dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.548+0000 7f3b2a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2415cab0 con 0x7f3b24145300 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.548+0000 7f3b2a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b241456e0 con 0x7f3b240cd060 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.548+0000 7f3b2a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2415cc30 con 0x7f3b24070f10 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 0x7f3b2413dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 0x7f3b2413dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44152/0 (socket says 172.21.15.32:44152) 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 -- 172.21.15.32:0/3691348500 learned_addr learned my addr 172.21.15.32:0/3691348500 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29200640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24070f10 0x7f3b2413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b23e00640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b240cd060 0x7f3b2413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 -- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24070f10 msgr2=0x7f3b2413f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24070f10 0x7f3b2413f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 -- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b240cd060 msgr2=0x7f3b2413d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b240cd060 0x7f3b2413d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.549+0000 7f3b29c00640 1 -- 172.21.15.32:0/3691348500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2413e6f0 con 0x7f3b24145300 2024-03-06T07:40:40.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.550+0000 7f3b29c00640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 0x7f3b2413dca0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009dd0 tx=0x7f3b1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:40.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.550+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c018020 con 0x7f3b24145300 2024-03-06T07:40:40.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.550+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c014480 con 0x7f3b24145300 2024-03-06T07:40:40.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.550+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c014870 con 0x7f3b24145300 2024-03-06T07:40:40.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.550+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2413e9d0 con 0x7f3b24145300 2024-03-06T07:40:40.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.550+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2413ef90 con 0x7f3b24145300 2024-03-06T07:40:40.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.551+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2404f330 con 0x7f3b24145300 2024-03-06T07:40:40.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.553+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b1c005d10 con 0x7f3b24145300 2024-03-06T07:40:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.554+0000 7f3b20c00640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b0406b4d0 0x7f3b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.554+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3b1c08b120 con 0x7f3b24145300 2024-03-06T07:40:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.555+0000 7f3b29200640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b0406b4d0 0x7f3b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.555+0000 7f3b29200640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b0406b4d0 0x7f3b0406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3b18036000 tx=0x7f3b1803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:40.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.563+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b1c022030 con 0x7f3b24145300 2024-03-06T07:40:40.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.808+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b2415be10 con 0x7f3b24145300 2024-03-06T07:40:40.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.809+0000 7f3b20c00640 1 -- 172.21.15.32:0/3691348500 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f3b1c010040 con 0x7f3b24145300 2024-03-06T07:40:40.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:40.811 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:40.812 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.811+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b0406b4d0 msgr2=0x7f3b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.811+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b0406b4d0 0x7f3b0406d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3b18036000 tx=0x7f3b1803b040 comp rx=0 tx=0).stop 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.811+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 msgr2=0x7f3b2413dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.811+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 0x7f3b2413dca0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009dd0 tx=0x7f3b1c00c490 comp rx=0 tx=0).stop 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 shutdown_connections 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f3b0406b4d0 0x7f3b0406d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b24145300 0x7f3b2413dca0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b240cd060 0x7f3b2413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 --2- 172.21.15.32:0/3691348500 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f3b24070f10 0x7f3b2413f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:40.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 >> 172.21.15.32:0/3691348500 conn(0x7f3b2406b3d0 msgr2=0x7f3b240723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:40.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 shutdown_connections 2024-03-06T07:40:40.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:40.812+0000 7f3b2a600640 1 -- 172.21.15.32:0/3691348500 wait complete. 2024-03-06T07:40:41.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:41.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:42.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.259+0000 7fea8f200640 1 Processor -- start 2024-03-06T07:40:42.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.260+0000 7fea8f200640 1 -- start start 2024-03-06T07:40:42.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.263+0000 7fea8f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fea88147870 0x7fea880b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.263+0000 7fea8f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 0x7fea8806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.263+0000 7fea8f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8806fb90 con 0x7fea880ad880 2024-03-06T07:40:42.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.263+0000 7fea8f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea88147c50 con 0x7fea88147870 2024-03-06T07:40:42.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.263+0000 7fea8f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b3b60 con 0x7fea8806e710 2024-03-06T07:40:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.264+0000 7fea8d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 0x7fea8806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.264+0000 7fea8de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fea88147870 0x7fea880b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.264+0000 7fea8d400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 0x7fea8806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:37132/0 (socket says 172.21.15.32:37132) 2024-03-06T07:40:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.264+0000 7fea8d400640 1 -- 172.21.15.32:0/1169640307 learned_addr learned my addr 172.21.15.32:0/1169640307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.265+0000 7fea8d400640 1 -- 172.21.15.32:0/1169640307 >> v2:172.21.15.39:3300/0 conn(0x7fea88147870 msgr2=0x7fea880b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:42.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.265+0000 7fea8d400640 1 --2- 172.21.15.32:0/1169640307 >> v2:172.21.15.39:3300/0 conn(0x7fea88147870 0x7fea880b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.265+0000 7fea8d400640 1 -- 172.21.15.32:0/1169640307 >> v1:172.21.15.39:6789/0 conn(0x7fea880ad880 legacy=0x7fea880d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.265+0000 7fea8d400640 1 -- 172.21.15.32:0/1169640307 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8806ebf0 con 0x7fea8806e710 2024-03-06T07:40:42.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.266+0000 7fea8d400640 1 --2- 172.21.15.32:0/1169640307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 0x7fea8806afb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea78009870 tx=0x7fea7802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fdc2a605f5403ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.267+0000 7fea8ca00640 1 -- 172.21.15.32:0/1169640307 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78034070 con 0x7fea8806e710 2024-03-06T07:40:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.267+0000 7fea8ca00640 1 -- 172.21.15.32:0/1169640307 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea78030840 con 0x7fea8806e710 2024-03-06T07:40:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.267+0000 7fea8ca00640 1 -- 172.21.15.32:0/1169640307 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78030c10 con 0x7fea8806e710 2024-03-06T07:40:42.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 -- 172.21.15.32:0/1169640307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 msgr2=0x7fea8806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 --2- 172.21.15.32:0/1169640307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 0x7fea8806afb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea78009870 tx=0x7fea7802b980 comp rx=0 tx=0).stop 2024-03-06T07:40:42.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 -- 172.21.15.32:0/1169640307 shutdown_connections 2024-03-06T07:40:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 --2- 172.21.15.32:0/1169640307 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea8806e710 0x7fea8806afb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 --2- 172.21.15.32:0/1169640307 >> v2:172.21.15.39:3300/0 conn(0x7fea88147870 0x7fea880b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 -- 172.21.15.32:0/1169640307 >> 172.21.15.32:0/1169640307 conn(0x7fea8806c0c0 msgr2=0x7fea8806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 -- 172.21.15.32:0/1169640307 shutdown_connections 2024-03-06T07:40:42.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.268+0000 7fea8f200640 1 -- 172.21.15.32:0/1169640307 wait complete. 2024-03-06T07:40:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.272+0000 7fea8f200640 1 Processor -- start 2024-03-06T07:40:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.272+0000 7fea8f200640 1 -- start start 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.274+0000 7fea8f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 0x7fea8815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.274+0000 7fea8f200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea880ad880 0x7fea8815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.274+0000 7fea8f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea88147870 0x7fea8813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b4370 con 0x7fea8806e710 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b4090 con 0x7fea88147870 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8f200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b5da0 con 0x7fea880ad880 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 0x7fea8815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 0x7fea8815ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44156/0 (socket says 172.21.15.32:44156) 2024-03-06T07:40:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 -- 172.21.15.32:0/574295660 learned_addr learned my addr 172.21.15.32:0/574295660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 -- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea880ad880 msgr2=0x7fea8815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8d400640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea880ad880 0x7fea8815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8e800640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea88147870 0x7fea8813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea880ad880 0x7fea8815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 -- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea88147870 msgr2=0x7fea8813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea88147870 0x7fea8813fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8de00640 1 -- 172.21.15.32:0/574295660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8815e4c0 con 0x7fea8806e710 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8d400640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea880ad880 0x7fea8815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.275+0000 7fea8e800640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea88147870 0x7fea8813fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.276+0000 7fea8de00640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 0x7fea8815ce80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00e9b0 tx=0x7fea7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.276+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c017070 con 0x7fea8806e710 2024-03-06T07:40:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.276+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c00c9a0 con 0x7fea8806e710 2024-03-06T07:40:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.276+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c00cd90 con 0x7fea8806e710 2024-03-06T07:40:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.276+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8815e7a0 con 0x7fea8806e710 2024-03-06T07:40:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.276+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8815ed60 con 0x7fea8806e710 2024-03-06T07:40:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.277+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8804f330 con 0x7fea8806e710 2024-03-06T07:40:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.278+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea7c01a070 con 0x7fea8806e710 2024-03-06T07:40:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.280+0000 7fea82000640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.280+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea7c08a670 con 0x7fea8806e710 2024-03-06T07:40:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.280+0000 7fea8d400640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.281+0000 7fea8d400640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b4d0 0x7fea6c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea78036000 tx=0x7fea7803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:42.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.285+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea7c059410 con 0x7fea8806e710 2024-03-06T07:40:42.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.547+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea88144c20 con 0x7fea8806e710 2024-03-06T07:40:42.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.548+0000 7fea82000640 1 -- 172.21.15.32:0/574295660 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fea7c058d10 con 0x7fea8806e710 2024-03-06T07:40:42.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:42.551 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:42.551 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b4d0 msgr2=0x7fea6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b4d0 0x7fea6c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea78036000 tx=0x7fea7803b040 comp rx=0 tx=0).stop 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 msgr2=0x7fea8815ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 0x7fea8815ce80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00e9b0 tx=0x7fea7c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 shutdown_connections 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea88147870 0x7fea8813fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fea880ad880 0x7fea8815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 --2- 172.21.15.32:0/574295660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea8806e710 0x7fea8815ce80 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:42.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 >> 172.21.15.32:0/574295660 conn(0x7fea8806c0c0 msgr2=0x7fea8806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:42.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 shutdown_connections 2024-03-06T07:40:42.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:42.552+0000 7fea8f200640 1 -- 172.21.15.32:0/574295660 wait complete. 2024-03-06T07:40:42.774 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:42.774 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:43.523 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:43.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:44.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.001+0000 7f0ab5400640 1 Processor -- start 2024-03-06T07:40:44.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.001+0000 7f0ab5400640 1 -- start start 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.003+0000 7f0ab5400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ab015e200 0x7f0ab00b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.003+0000 7f0ab5400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ab006afb0 0x7f0ab006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.003+0000 7f0ab5400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab006bec0 con 0x7f0ab00afdf0 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.003+0000 7f0ab5400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab00b0670 con 0x7f0ab015e200 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.003+0000 7f0ab5400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab00ae380 con 0x7f0ab006afb0 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ab015e200 0x7f0ab00b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ab015e200 0x7f0ab00b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44158/0 (socket says 172.21.15.32:44158) 2024-03-06T07:40:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 -- 172.21.15.32:0/1405367570 learned_addr learned my addr 172.21.15.32:0/1405367570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aaf200640 1 --2- 172.21.15.32:0/1405367570 >> v2:172.21.15.39:3300/0 conn(0x7f0ab006afb0 0x7f0ab006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 -- 172.21.15.32:0/1405367570 >> v2:172.21.15.39:3300/0 conn(0x7f0ab006afb0 msgr2=0x7f0ab006b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 --2- 172.21.15.32:0/1405367570 >> v2:172.21.15.39:3300/0 conn(0x7f0ab006afb0 0x7f0ab006b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 -- 172.21.15.32:0/1405367570 >> v1:172.21.15.39:6789/0 conn(0x7f0ab00afdf0 legacy=0x7f0ab00af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.004+0000 7f0aae800640 1 -- 172.21.15.32:0/1405367570 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00ae660 con 0x7f0ab015e200 2024-03-06T07:40:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.005+0000 7f0aae800640 1 --2- 172.21.15.32:0/1405367570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab015e200 0x7f0ab00b3de0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0162c90 tx=0x7f0aa002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ebaa52e64a26a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.006+0000 7f0aade00640 1 -- 172.21.15.32:0/1405367570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa002ca80 con 0x7f0ab015e200 2024-03-06T07:40:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.006+0000 7f0aade00640 1 -- 172.21.15.32:0/1405367570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa002cc20 con 0x7f0ab015e200 2024-03-06T07:40:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.006+0000 7f0aade00640 1 -- 172.21.15.32:0/1405367570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa00343f0 con 0x7f0ab015e200 2024-03-06T07:40:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 -- 172.21.15.32:0/1405367570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab015e200 msgr2=0x7f0ab00b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 --2- 172.21.15.32:0/1405367570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab015e200 0x7f0ab00b3de0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0162c90 tx=0x7f0aa002bee0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 -- 172.21.15.32:0/1405367570 shutdown_connections 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 --2- 172.21.15.32:0/1405367570 >> v2:172.21.15.39:3300/0 conn(0x7f0ab006afb0 0x7f0ab006b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 --2- 172.21.15.32:0/1405367570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab015e200 0x7f0ab00b3de0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 -- 172.21.15.32:0/1405367570 >> 172.21.15.32:0/1405367570 conn(0x7f0ab00b2ae0 msgr2=0x7f0ab00b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 -- 172.21.15.32:0/1405367570 shutdown_connections 2024-03-06T07:40:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.007+0000 7f0ab5400640 1 -- 172.21.15.32:0/1405367570 wait complete. 2024-03-06T07:40:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.010+0000 7f0ab5400640 1 Processor -- start 2024-03-06T07:40:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.010+0000 7f0ab5400640 1 -- start start 2024-03-06T07:40:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.011+0000 7f0ab5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 0x7f0ab0148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0ab5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab00afdf0 0x7f0ab015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0ab5400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ab015e200 0x7f0ab0153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0ab5400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab006dbb0 con 0x7f0ab006afb0 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0ab5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00aec20 con 0x7f0ab00afdf0 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0ab5400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab006dd30 con 0x7f0ab015e200 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 0x7f0ab0148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 0x7f0ab0148580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44168/0 (socket says 172.21.15.32:44168) 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 -- 172.21.15.32:0/4243959606 learned_addr learned my addr 172.21.15.32:0/4243959606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aa5e00640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab00afdf0 0x7f0ab015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aaf200640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ab015e200 0x7f0ab0153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 -- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ab015e200 msgr2=0x7f0ab0153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ab015e200 0x7f0ab0153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 -- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab00afdf0 msgr2=0x7f0ab015a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab00afdf0 0x7f0ab015a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aae800640 1 -- 172.21.15.32:0/4243959606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab015c830 con 0x7f0ab006afb0 2024-03-06T07:40:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.012+0000 7f0aaf200640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ab015e200 0x7f0ab0153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.013+0000 7f0aae800640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 0x7f0ab0148580 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0036040 tx=0x7f0aa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.013+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0043020 con 0x7f0ab006afb0 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.013+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa003e450 con 0x7f0ab006afb0 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.013+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0009510 con 0x7f0ab006afb0 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.013+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab015cb10 con 0x7f0ab006afb0 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.013+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab015d0d0 con 0x7f0ab006afb0 2024-03-06T07:40:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.014+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab004f3a0 con 0x7f0ab006afb0 2024-03-06T07:40:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.016+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aa0046020 con 0x7f0ab006afb0 2024-03-06T07:40:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.016+0000 7f0aa7400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.017+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0aa00acaa0 con 0x7f0ab006afb0 2024-03-06T07:40:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.017+0000 7f0aa5e00640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:44.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.018+0000 7f0aa5e00640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0ab01f75c0 tx=0x7f0a98009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:44.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.022+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0aa004d030 con 0x7f0ab006afb0 2024-03-06T07:40:44.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.258+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab0073050 con 0x7f0ab006afb0 2024-03-06T07:40:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.259+0000 7f0aa7400640 1 -- 172.21.15.32:0/4243959606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f0aa007b8d0 con 0x7f0ab006afb0 2024-03-06T07:40:44.262 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:44.262 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a8c06b4d0 msgr2=0x7f0a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0ab01f75c0 tx=0x7f0a98009040 comp rx=0 tx=0).stop 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 msgr2=0x7f0ab0148580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 0x7f0ab0148580 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0036040 tx=0x7f0aa0004a40 comp rx=0 tx=0).stop 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 shutdown_connections 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f0ab015e200 0x7f0ab0153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ab00afdf0 0x7f0ab015a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 --2- 172.21.15.32:0/4243959606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ab006afb0 0x7f0ab0148580 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 >> 172.21.15.32:0/4243959606 conn(0x7f0ab00b2ae0 msgr2=0x7f0ab00d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 shutdown_connections 2024-03-06T07:40:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:44.262+0000 7f0ab5400640 1 -- 172.21.15.32:0/4243959606 wait complete. 2024-03-06T07:40:44.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:44.487 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:45.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:45.525 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:45.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.715+0000 7f4b06800640 1 Processor -- start 2024-03-06T07:40:45.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.715+0000 7f4b06800640 1 -- start start 2024-03-06T07:40:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.716+0000 7f4b06800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4b001607e0 0x7f4b000d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.716+0000 7f4b06800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000ceac0 con 0x7f4b000b3f60 2024-03-06T07:40:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4b06800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000b43f0 con 0x7f4b000d27d0 2024-03-06T07:40:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4b06800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b000cf690 con 0x7f4b001607e0 2024-03-06T07:40:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4aff400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4b001607e0 0x7f4b000d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4aff400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4b001607e0 0x7f4b000d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44170/0 (socket says 172.21.15.32:44170) 2024-03-06T07:40:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4aff400640 1 -- 172.21.15.32:0/2456745557 learned_addr learned my addr 172.21.15.32:0/2456745557 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4affe00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4b000b3f60 0x7f4b00160bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53186/0 (socket says 172.21.15.32:53186) 2024-03-06T07:40:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4aff400640 1 -- 172.21.15.32:0/2456745557 >> v1:172.21.15.39:6789/0 conn(0x7f4b000d27d0 legacy=0x7f4b000b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4aff400640 1 -- 172.21.15.32:0/2456745557 >> v1:172.21.15.32:6789/0 conn(0x7f4b000b3f60 legacy=0x7f4b00160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.718+0000 7f4aff400640 1 -- 172.21.15.32:0/2456745557 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b000ced60 con 0x7f4b001607e0 2024-03-06T07:40:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.717+0000 7f4afe000640 1 -- 172.21.15.32:0/2456745557 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990390349 0 0) 0x7f4b000ceac0 con 0x7f4b000b3f60 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.719+0000 7f4aff400640 1 --2- 172.21.15.32:0/2456745557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b001607e0 0x7f4b000d4a10 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009870 tx=0x7f4af002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1521ac2c5710d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.719+0000 7f4afe000640 1 -- 172.21.15.32:0/2456745557 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0034070 con 0x7f4b001607e0 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.719+0000 7f4afe000640 1 -- 172.21.15.32:0/2456745557 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af0030860 con 0x7f4b001607e0 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.719+0000 7f4afe000640 1 -- 172.21.15.32:0/2456745557 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0030c30 con 0x7f4b001607e0 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 -- 172.21.15.32:0/2456745557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b001607e0 msgr2=0x7f4b000d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 --2- 172.21.15.32:0/2456745557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b001607e0 0x7f4b000d4a10 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009870 tx=0x7f4af002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 -- 172.21.15.32:0/2456745557 shutdown_connections 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 --2- 172.21.15.32:0/2456745557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b001607e0 0x7f4b000d4a10 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 -- 172.21.15.32:0/2456745557 >> 172.21.15.32:0/2456745557 conn(0x7f4b000d3ef0 msgr2=0x7f4b000d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:45.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 -- 172.21.15.32:0/2456745557 shutdown_connections 2024-03-06T07:40:45.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.720+0000 7f4b06800640 1 -- 172.21.15.32:0/2456745557 wait complete. 2024-03-06T07:40:45.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.721+0000 7f4b06800640 1 Processor -- start 2024-03-06T07:40:45.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.722+0000 7f4b06800640 1 -- start start 2024-03-06T07:40:45.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4b06800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b000b3f60 0x7f4b000b9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4b06800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 0x7f4b00152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4b06800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4b001607e0 0x7f4b000d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4b06800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000d2c10 con 0x7f4b000d27d0 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4b06800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000cf320 con 0x7f4b000b3f60 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4b06800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000cfb20 con 0x7f4b001607e0 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 0x7f4b00152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 0x7f4b00152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44178/0 (socket says 172.21.15.32:44178) 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 -- 172.21.15.32:0/1217776352 learned_addr learned my addr 172.21.15.32:0/1217776352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4aff400640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b000b3f60 0x7f4b000b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4affe00640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4b001607e0 0x7f4b000d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 -- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4b001607e0 msgr2=0x7f4b000d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4b001607e0 0x7f4b000d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 -- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b000b3f60 msgr2=0x7f4b000b9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b000b3f60 0x7f4b000b9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4afea00640 1 -- 172.21.15.32:0/1217776352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b001528b0 con 0x7f4b000d27d0 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.723+0000 7f4affe00640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4b001607e0 0x7f4b000d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.724+0000 7f4afea00640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 0x7f4b00152370 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00c430 tx=0x7f4aec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.724+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec011790 con 0x7f4b000d27d0 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.724+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aec017070 con 0x7f4b000d27d0 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.724+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec011bb0 con 0x7f4b000d27d0 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.724+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b000cb1a0 con 0x7f4b000d27d0 2024-03-06T07:40:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.724+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b000cb760 con 0x7f4b000d27d0 2024-03-06T07:40:45.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.725+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0004f3a0 con 0x7f4b000d27d0 2024-03-06T07:40:45.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.726+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4aec01c460 con 0x7f4b000d27d0 2024-03-06T07:40:45.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.727+0000 7f4af7400640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ae006b500 0x7f4ae006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:45.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.727+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4aec08a0e0 con 0x7f4b000d27d0 2024-03-06T07:40:45.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.728+0000 7f4aff400640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ae006b500 0x7f4ae006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:45.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.728+0000 7f4aff400640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ae006b500 0x7f4ae006d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4af0036000 tx=0x7f4af003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:45.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.732+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b001528b0 con 0x7f4b000d27d0 2024-03-06T07:40:45.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.972+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b0006d210 con 0x7f4b000d27d0 2024-03-06T07:40:45.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.974+0000 7f4af7400640 1 -- 172.21.15.32:0/1217776352 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f4aec058f10 con 0x7f4b000d27d0 2024-03-06T07:40:45.976 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:45.977 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:45.977 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.976+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ae006b500 msgr2=0x7f4ae006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.976+0000 7f4b06800640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ae006b500 0x7f4ae006d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4af0036000 tx=0x7f4af003b040 comp rx=0 tx=0).stop 2024-03-06T07:40:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.976+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 msgr2=0x7f4b00152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.976+0000 7f4b06800640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 0x7f4b00152370 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00c430 tx=0x7f4aec00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 shutdown_connections 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f4ae006b500 0x7f4ae006d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f4b001607e0 0x7f4b000d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b000d27d0 0x7f4b00152370 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 --2- 172.21.15.32:0/1217776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b000b3f60 0x7f4b000b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 >> 172.21.15.32:0/1217776352 conn(0x7f4b000d3ef0 msgr2=0x7f4b0006c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 shutdown_connections 2024-03-06T07:40:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:45.977+0000 7f4b06800640 1 -- 172.21.15.32:0/1217776352 wait complete. 2024-03-06T07:40:46.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:46.202 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:47.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.433+0000 7fa6fb600640 1 Processor -- start 2024-03-06T07:40:47.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.433+0000 7fa6fb600640 1 -- start start 2024-03-06T07:40:47.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.434+0000 7fa6fb600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa6f40b3c90 0x7fa6f40cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.434+0000 7fa6fb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6fb600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f40b4860 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6fb600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f40ce160 con 0x7fa6f40b3c90 2024-03-06T07:40:47.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6fb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40b4b40 con 0x7fa6f4071050 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa6f40b3c90 0x7fa6f40cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa6f40b3c90 0x7fa6f40cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44192/0 (socket says 172.21.15.32:44192) 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 -- 172.21.15.32:0/1477341769 learned_addr learned my addr 172.21.15.32:0/1477341769 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f9600640 1 --2- 172.21.15.32:0/1477341769 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 -- 172.21.15.32:0/1477341769 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 msgr2=0x7fa6f4071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 --2- 172.21.15.32:0/1477341769 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 -- 172.21.15.32:0/1477341769 >> v1:172.21.15.39:6789/0 conn(0x7fa6f40cf5c0 legacy=0x7fa6f40b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:47.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.435+0000 7fa6f8c00640 1 -- 172.21.15.32:0/1477341769 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40b4e20 con 0x7fa6f40b3c90 2024-03-06T07:40:47.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.437+0000 7fa6f8c00640 1 --2- 172.21.15.32:0/1477341769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40b3c90 0x7fa6f40cdc20 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009870 tx=0x7fa6e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba9193586c90e95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:47.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.437+0000 7fa6f3400640 1 -- 172.21.15.32:0/1477341769 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4034070 con 0x7fa6f40b3c90 2024-03-06T07:40:47.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.437+0000 7fa6f3400640 1 -- 172.21.15.32:0/1477341769 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e4030840 con 0x7fa6f40b3c90 2024-03-06T07:40:47.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.437+0000 7fa6f3400640 1 -- 172.21.15.32:0/1477341769 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4030c10 con 0x7fa6f40b3c90 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 -- 172.21.15.32:0/1477341769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40b3c90 msgr2=0x7fa6f40cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 --2- 172.21.15.32:0/1477341769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40b3c90 0x7fa6f40cdc20 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009870 tx=0x7fa6e402b980 comp rx=0 tx=0).stop 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 -- 172.21.15.32:0/1477341769 shutdown_connections 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 --2- 172.21.15.32:0/1477341769 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 --2- 172.21.15.32:0/1477341769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40b3c90 0x7fa6f40cdc20 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 -- 172.21.15.32:0/1477341769 >> 172.21.15.32:0/1477341769 conn(0x7fa6f406afb0 msgr2=0x7fa6f406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 -- 172.21.15.32:0/1477341769 shutdown_connections 2024-03-06T07:40:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.438+0000 7fa6fb600640 1 -- 172.21.15.32:0/1477341769 wait complete. 2024-03-06T07:40:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.440+0000 7fa6fb600640 1 Processor -- start 2024-03-06T07:40:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.440+0000 7fa6fb600640 1 -- start start 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6fb600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6fb600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f40b3c90 0x7fa6f4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6fb600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 0x7fa6f4151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6fb600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40719d0 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6fb600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415b340 con 0x7fa6f40b3c90 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6fb600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4071b50 con 0x7fa6f4071050 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 0x7fa6f4151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 0x7fa6f4151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44204/0 (socket says 172.21.15.32:44204) 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.441+0000 7fa6f9600640 1 -- 172.21.15.32:0/50134123 learned_addr learned my addr 172.21.15.32:0/50134123 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:47.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f3e00640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f40b3c90 0x7fa6f4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f8c00640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f9600640 1 -- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 msgr2=0x7fa6f4157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f9600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f9600640 1 -- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f40b3c90 msgr2=0x7fa6f4159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f9600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f40b3c90 0x7fa6f4159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f9600640 1 -- 172.21.15.32:0/50134123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f41f4c60 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f9600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 0x7fa6f4151270 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec007e40 tx=0x7fa6ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec014620 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.442+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec019070 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.443+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec014a20 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.443+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f4156830 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.443+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f415a220 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.444+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f4069510 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.445+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6ec014bc0 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.446+0000 7fa6f0c00640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6d406b4d0 0x7fa6d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.446+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa6ec08ab90 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.446+0000 7fa6f8c00640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6d406b4d0 0x7fa6d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:47.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.447+0000 7fa6f8c00640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6d406b4d0 0x7fa6d406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4036000 tx=0x7fa6e403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:47.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.451+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6ec0599c0 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:47.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.688+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6f4072e10 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.689+0000 7fa6f0c00640 1 -- 172.21.15.32:0/50134123 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fa6ec0592c0 con 0x7fa6f40cf5c0 2024-03-06T07:40:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:47.692 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:47.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6d406b4d0 msgr2=0x7fa6d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6d406b4d0 0x7fa6d406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4036000 tx=0x7fa6e403b040 comp rx=0 tx=0).stop 2024-03-06T07:40:47.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 msgr2=0x7fa6f4151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 0x7fa6f4151270 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec007e40 tx=0x7fa6ec00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 shutdown_connections 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fa6d406b4d0 0x7fa6d406d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f40cf5c0 0x7fa6f4151270 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.691+0000 7fa6fb600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f40b3c90 0x7fa6f4159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.692+0000 7fa6fb600640 1 --2- 172.21.15.32:0/50134123 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fa6f4071050 0x7fa6f4157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.692+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 >> 172.21.15.32:0/50134123 conn(0x7fa6f406afb0 msgr2=0x7fa6f4155e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.692+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 shutdown_connections 2024-03-06T07:40:47.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:47.692+0000 7fa6fb600640 1 -- 172.21.15.32:0/50134123 wait complete. 2024-03-06T07:40:47.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:47.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:48.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.138+0000 7f309fe00640 1 Processor -- start 2024-03-06T07:40:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.139+0000 7f309fe00640 1 -- start start 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.140+0000 7f309fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 0x7f30a01667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.140+0000 7f309fe00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a00714d0 con 0x7f30a0070f10 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.140+0000 7f309fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a00717b0 con 0x7f30a0145300 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.141+0000 7f309fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0071a90 con 0x7f30a00cd060 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.141+0000 7f309e000640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30a0070f10 0x7f30a00cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47784/0 (socket says 172.21.15.32:47784) 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.141+0000 7f309e000640 1 -- 172.21.15.32:0/2571614322 learned_addr learned my addr 172.21.15.32:0/2571614322 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:49.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.141+0000 7f309f400640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 0x7f30a01667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.141+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380623884 0 0) 0x7f30a00714d0 con 0x7f30a0070f10 2024-03-06T07:40:49.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.141+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3084003520 con 0x7f30a0070f10 2024-03-06T07:40:49.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774883469 0 0) 0x7f3084003520 con 0x7f30a0070f10 2024-03-06T07:40:49.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 msgr2=0x7f30a01667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 0x7f30a01667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 >> v1:172.21.15.39:6789/0 conn(0x7f30a0145300 legacy=0x7f30a00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a0071d70 con 0x7f30a0070f10 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7f3088001670 con 0x7f30a0070f10 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 >> v1:172.21.15.32:6789/0 conn(0x7f30a0070f10 legacy=0x7f30a00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30840039e0 0x7f3084003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 0x7f3084004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0071a90 con 0x7f3084004d40 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308400bb20 con 0x7f30840039e0 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308400bd20 con 0x7f30a00cd060 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.142+0000 7f309f400640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309ea00640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30840039e0 0x7f3084003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309e000640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 0x7f3084004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309f400640 1 -- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 msgr2=0x7f3084004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:49.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309f400640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a00cd060 0x7f3084004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309f400640 1 -- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30840039e0 msgr2=0x7f3084003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:49.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309f400640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30840039e0 0x7f3084003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.143+0000 7f309f400640 1 -- 172.21.15.32:0/2571614322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084003520 con 0x7f3084004d40 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.144+0000 7f309f400640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30840051c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f3098009ac0 tx=0x7f3084005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed1cb3d43a6a023 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.144+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098039070 con 0x7f3084004d40 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.144+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309802c6b0 con 0x7f3084004d40 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.144+0000 7f309d600640 1 -- 172.21.15.32:0/2571614322 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098009690 con 0x7f3084004d40 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309fe00640 1 -- 172.21.15.32:0/2571614322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 msgr2=0x7f30840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309fe00640 1 --2- 172.21.15.32:0/2571614322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30840051c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f3098009ac0 tx=0x7f3084005830 comp rx=0 tx=0).stop 2024-03-06T07:40:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309f400640 1 -- 172.21.15.32:0/2571614322 reap_dead start 2024-03-06T07:40:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309fe00640 1 -- 172.21.15.32:0/2571614322 shutdown_connections 2024-03-06T07:40:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309fe00640 1 -- 172.21.15.32:0/2571614322 >> 172.21.15.32:0/2571614322 conn(0x7f30a006b3d0 msgr2=0x7f30a006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309fe00640 1 -- 172.21.15.32:0/2571614322 shutdown_connections 2024-03-06T07:40:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.145+0000 7f309fe00640 1 -- 172.21.15.32:0/2571614322 wait complete. 2024-03-06T07:40:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.147+0000 7f309fe00640 1 Processor -- start 2024-03-06T07:40:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.147+0000 7f309fe00640 1 -- start start 2024-03-06T07:40:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30a013be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309fe00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a013c380 0x7f30a013da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30a013df70 0x7f30a013a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a015d700 con 0x7f3084004d40 2024-03-06T07:40:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a015d580 con 0x7f30a013df70 2024-03-06T07:40:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309fe00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a015d880 con 0x7f30a013c380 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30a013be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30a013be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58522/0 (socket says 172.21.15.32:58522) 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.148+0000 7f309ea00640 1 -- 172.21.15.32:0/2773108669 learned_addr learned my addr 172.21.15.32:0/2773108669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309e000640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a013c380 0x7f30a013da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309f400640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30a013df70 0x7f30a013a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309ea00640 1 -- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a013c380 msgr2=0x7f30a013da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309ea00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a013c380 0x7f30a013da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309ea00640 1 -- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30a013df70 msgr2=0x7f30a013a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309ea00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30a013df70 0x7f30a013a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309ea00640 1 -- 172.21.15.32:0/2773108669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00b10d0 con 0x7f3084004d40 2024-03-06T07:40:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309f400640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30a013df70 0x7f30a013a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.149+0000 7f309ea00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30a013be40 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f309000e8c0 tx=0x7f309000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.150+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090003b80 con 0x7f3084004d40 2024-03-06T07:40:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.150+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3090018070 con 0x7f3084004d40 2024-03-06T07:40:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.150+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30900050d0 con 0x7f3084004d40 2024-03-06T07:40:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.150+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a0139660 con 0x7f3084004d40 2024-03-06T07:40:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.150+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a0139c20 con 0x7f3084004d40 2024-03-06T07:40:49.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.151+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a004f330 con 0x7f3084004d40 2024-03-06T07:40:49.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.152+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3090005270 con 0x7f3084004d40 2024-03-06T07:40:49.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.153+0000 7f3096a00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307c06b4d0 0x7f307c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:49.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.153+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f309008a2e0 con 0x7f3084004d40 2024-03-06T07:40:49.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.153+0000 7f309e000640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307c06b4d0 0x7f307c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:49.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.154+0000 7f309e000640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307c06b4d0 0x7f307c06d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f30a00cbf50 tx=0x7f3088001d70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:49.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.157+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f309001a310 con 0x7f3084004d40 2024-03-06T07:40:49.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.397+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30a0068320 con 0x7f3084004d40 2024-03-06T07:40:49.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.398+0000 7f3096a00640 1 -- 172.21.15.32:0/2773108669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f3090059110 con 0x7f3084004d40 2024-03-06T07:40:49.400 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:49.401 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:49.401 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.400+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307c06b4d0 msgr2=0x7f307c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.400+0000 7f309fe00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307c06b4d0 0x7f307c06d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f30a00cbf50 tx=0x7f3088001d70 comp rx=0 tx=0).stop 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.400+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 msgr2=0x7f30a013be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.400+0000 7f309fe00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30a013be40 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f309000e8c0 tx=0x7f309000ed80 comp rx=0 tx=0).stop 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 shutdown_connections 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f307c06b4d0 0x7f307c06d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30a013df70 0x7f30a013a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f30a013c380 0x7f30a013da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 --2- 172.21.15.32:0/2773108669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3084004d40 0x7f30a013be40 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:49.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 >> 172.21.15.32:0/2773108669 conn(0x7f30a006b3d0 msgr2=0x7f30a0142180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:49.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 shutdown_connections 2024-03-06T07:40:49.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:49.401+0000 7f309fe00640 1 -- 172.21.15.32:0/2773108669 wait complete. 2024-03-06T07:40:49.529 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:49.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:49.626 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:50.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:50.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.852+0000 7f1266800640 1 Processor -- start 2024-03-06T07:40:50.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.852+0000 7f1266800640 1 -- start start 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.853+0000 7f1266800640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12600d4a10 0x7f12600d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.853+0000 7f1266800640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600b3710 con 0x7f12600d0c80 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.853+0000 7f1266800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600b39f0 con 0x7f126015e0b0 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.853+0000 7f1266800640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600d10f0 con 0x7f12600d4a10 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.854+0000 7f1264a00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12600d4a10 0x7f12600d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.854+0000 7f1265e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12600d0c80 0x7f12600b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47796/0 (socket says 172.21.15.32:47796) 2024-03-06T07:40:50.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.854+0000 7f1265e00640 1 -- 172.21.15.32:0/4272892162 learned_addr learned my addr 172.21.15.32:0/4272892162 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:50.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.855+0000 7f1264a00640 1 -- 172.21.15.32:0/4272892162 >> v1:172.21.15.39:6789/0 conn(0x7f126015e0b0 legacy=0x7f12600cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.855+0000 7f1264a00640 1 -- 172.21.15.32:0/4272892162 >> v1:172.21.15.32:6789/0 conn(0x7f12600d0c80 legacy=0x7f12600b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.854+0000 7f125fe00640 1 -- 172.21.15.32:0/4272892162 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745820549 0 0) 0x7f12600b3710 con 0x7f12600d0c80 2024-03-06T07:40:50.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.855+0000 7f1264a00640 1 -- 172.21.15.32:0/4272892162 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600b3cd0 con 0x7f12600d4a10 2024-03-06T07:40:50.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.856+0000 7f1264a00640 1 --2- 172.21.15.32:0/4272892162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12600d4a10 0x7f12600d0740 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f12540097a0 tx=0x7f125402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1d0db2e86c5419 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:50.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.856+0000 7f125fe00640 1 -- 172.21.15.32:0/4272892162 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254034070 con 0x7f12600d4a10 2024-03-06T07:40:50.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.856+0000 7f125fe00640 1 -- 172.21.15.32:0/4272892162 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1254030610 con 0x7f12600d4a10 2024-03-06T07:40:50.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.856+0000 7f125fe00640 1 -- 172.21.15.32:0/4272892162 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540309e0 con 0x7f12600d4a10 2024-03-06T07:40:50.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 -- 172.21.15.32:0/4272892162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12600d4a10 msgr2=0x7f12600d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 --2- 172.21.15.32:0/4272892162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12600d4a10 0x7f12600d0740 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f12540097a0 tx=0x7f125402b960 comp rx=0 tx=0).stop 2024-03-06T07:40:50.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 -- 172.21.15.32:0/4272892162 shutdown_connections 2024-03-06T07:40:50.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 --2- 172.21.15.32:0/4272892162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12600d4a10 0x7f12600d0740 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 -- 172.21.15.32:0/4272892162 >> 172.21.15.32:0/4272892162 conn(0x7f12600d3d30 msgr2=0x7f12600d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:50.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 -- 172.21.15.32:0/4272892162 shutdown_connections 2024-03-06T07:40:50.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.857+0000 7f1266800640 1 -- 172.21.15.32:0/4272892162 wait complete. 2024-03-06T07:40:50.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.859+0000 7f1266800640 1 Processor -- start 2024-03-06T07:40:50.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.859+0000 7f1266800640 1 -- start start 2024-03-06T07:40:50.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1266800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12600d0c80 0x7f1260158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1266800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12600d4a10 0x7f126014f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1266800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 0x7f12601611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1266800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600cfa20 con 0x7f126015e0b0 2024-03-06T07:40:50.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1266800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126015e490 con 0x7f12600d0c80 2024-03-06T07:40:50.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1266800640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600cfc60 con 0x7f12600d4a10 2024-03-06T07:40:50.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1265e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 0x7f12601611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1265e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 0x7f12601611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58542/0 (socket says 172.21.15.32:58542) 2024-03-06T07:40:50.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.860+0000 7f1265e00640 1 -- 172.21.15.32:0/3089288976 learned_addr learned my addr 172.21.15.32:0/3089288976 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:50.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1264a00640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12600d4a10 0x7f126014f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265400640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12600d0c80 0x7f1260158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265e00640 1 -- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12600d4a10 msgr2=0x7f126014f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265e00640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12600d4a10 0x7f126014f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265e00640 1 -- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12600d0c80 msgr2=0x7f1260158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265e00640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12600d0c80 0x7f1260158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265e00640 1 -- 172.21.15.32:0/3089288976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12601592e0 con 0x7f126015e0b0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265400640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12600d0c80 0x7f1260158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f1265e00640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 0x7f12601611b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f125800caf0 tx=0x7f12580021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800ac00 con 0x7f126015e0b0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.861+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800ada0 con 0x7f126015e0b0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.862+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12601595c0 con 0x7f126015e0b0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.862+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258012640 con 0x7f126015e0b0 2024-03-06T07:40:50.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.862+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1260159b80 con 0x7f126015e0b0 2024-03-06T07:40:50.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.863+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126004f330 con 0x7f126015e0b0 2024-03-06T07:40:50.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.864+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12580127e0 con 0x7f126015e0b0 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.865+0000 7f125d600640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f124006b4d0 0x7f124006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.865+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f125808a880 con 0x7f126015e0b0 2024-03-06T07:40:50.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.865+0000 7f1265400640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f124006b4d0 0x7f124006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:50.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.866+0000 7f1265400640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f124006b4d0 0x7f124006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f124c000f60 tx=0x7f124c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:50.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:50.870+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12580596c0 con 0x7f126015e0b0 2024-03-06T07:40:51.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.112+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12600b1880 con 0x7f126015e0b0 2024-03-06T07:40:51.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.113+0000 7f125d600640 1 -- 172.21.15.32:0/3089288976 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7f1258058fc0 con 0x7f126015e0b0 2024-03-06T07:40:51.116 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:51.116 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:51.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f124006b4d0 msgr2=0x7f124006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:51.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f124006b4d0 0x7f124006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f124c000f60 tx=0x7f124c009040 comp rx=0 tx=0).stop 2024-03-06T07:40:51.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 msgr2=0x7f12601611b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:51.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 0x7f12601611b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f125800caf0 tx=0x7f12580021c0 comp rx=0 tx=0).stop 2024-03-06T07:40:51.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 shutdown_connections 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f124006b4d0 0x7f124006d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f126015e0b0 0x7f12601611b0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f12600d4a10 0x7f126014f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 --2- 172.21.15.32:0/3089288976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12600d0c80 0x7f1260158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.116+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 >> 172.21.15.32:0/3089288976 conn(0x7f12600d3d30 msgr2=0x7f12600b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.117+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 shutdown_connections 2024-03-06T07:40:51.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:51.117+0000 7f1266800640 1 -- 172.21.15.32:0/3089288976 wait complete. 2024-03-06T07:40:51.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:51.341 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:51.532 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:51.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:51.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:52.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:52.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.608+0000 7fafc9400640 1 Processor -- start 2024-03-06T07:40:52.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.608+0000 7fafc9400640 1 -- start start 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc9400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafc415efc0 0x7fafc40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fafc41621d0 0x7fafc4070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc9400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc41627f0 con 0x7fafc40cce90 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc9400640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc415f3a0 con 0x7fafc415efc0 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc9400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc406d190 con 0x7fafc41621d0 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafc415efc0 0x7fafc40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc3e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafc415efc0 0x7fafc40cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58544/0 (socket says 172.21.15.32:58544) 2024-03-06T07:40:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.610+0000 7fafc3e00640 1 -- 172.21.15.32:0/3210085363 learned_addr learned my addr 172.21.15.32:0/3210085363 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.611+0000 7fafc8a00640 1 --2- 172.21.15.32:0/3210085363 >> v2:172.21.15.39:3300/0 conn(0x7fafc41621d0 0x7fafc4070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.611+0000 7fafc3e00640 1 -- 172.21.15.32:0/3210085363 >> v2:172.21.15.39:3300/0 conn(0x7fafc41621d0 msgr2=0x7fafc4070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.611+0000 7fafc3e00640 1 --2- 172.21.15.32:0/3210085363 >> v2:172.21.15.39:3300/0 conn(0x7fafc41621d0 0x7fafc4070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.611+0000 7fafc3e00640 1 -- 172.21.15.32:0/3210085363 >> v1:172.21.15.39:6789/0 conn(0x7fafc40cce90 legacy=0x7fafc4161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.611+0000 7fafc3e00640 1 -- 172.21.15.32:0/3210085363 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc406c860 con 0x7fafc415efc0 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.611+0000 7fafc8a00640 1 --2- 172.21.15.32:0/3210085363 >> v2:172.21.15.39:3300/0 conn(0x7fafc41621d0 0x7fafc4070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.612+0000 7fafc3e00640 1 --2- 172.21.15.32:0/3210085363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc40cc950 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fafc40709d0 tx=0x7fafb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4eb1b321ae33938 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:52.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.612+0000 7fafc2a00640 1 -- 172.21.15.32:0/3210085363 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4035020 con 0x7fafc415efc0 2024-03-06T07:40:52.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.612+0000 7fafc2a00640 1 -- 172.21.15.32:0/3210085363 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb4031870 con 0x7fafc415efc0 2024-03-06T07:40:52.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.612+0000 7fafc2a00640 1 -- 172.21.15.32:0/3210085363 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4031c40 con 0x7fafc415efc0 2024-03-06T07:40:52.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 -- 172.21.15.32:0/3210085363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 msgr2=0x7fafc40cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 --2- 172.21.15.32:0/3210085363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc40cc950 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fafc40709d0 tx=0x7fafb402b910 comp rx=0 tx=0).stop 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 -- 172.21.15.32:0/3210085363 shutdown_connections 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 --2- 172.21.15.32:0/3210085363 >> v2:172.21.15.39:3300/0 conn(0x7fafc41621d0 0x7fafc4070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 --2- 172.21.15.32:0/3210085363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc40cc950 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 -- 172.21.15.32:0/3210085363 >> 172.21.15.32:0/3210085363 conn(0x7fafc406b3d0 msgr2=0x7fafc406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 -- 172.21.15.32:0/3210085363 shutdown_connections 2024-03-06T07:40:52.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.613+0000 7fafc9400640 1 -- 172.21.15.32:0/3210085363 wait complete. 2024-03-06T07:40:52.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.615+0000 7fafc9400640 1 Processor -- start 2024-03-06T07:40:52.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.615+0000 7fafc9400640 1 -- start start 2024-03-06T07:40:52.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc40cce90 0x7fafc415a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc41f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc9400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafc41621d0 0x7fafc41f7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc9400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc406d620 con 0x7fafc415efc0 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc9400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc406ce20 con 0x7fafc40cce90 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc9400640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc41e80e0 con 0x7fafc41621d0 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc41f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc41f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58556/0 (socket says 172.21.15.32:58556) 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 -- 172.21.15.32:0/1918755253 learned_addr learned my addr 172.21.15.32:0/1918755253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 -- 172.21.15.32:0/1918755253 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafc41621d0 msgr2=0x7fafc41f7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafc41621d0 0x7fafc41f7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 -- 172.21.15.32:0/1918755253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc40cce90 msgr2=0x7fafc415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc40cce90 0x7fafc415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.616+0000 7fafc3400640 1 -- 172.21.15.32:0/1918755253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc415ad50 con 0x7fafc415efc0 2024-03-06T07:40:52.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.617+0000 7fafc3400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc41f73e0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c410 tx=0x7fafac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:52.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.617+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac017070 con 0x7fafc415efc0 2024-03-06T07:40:52.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.617+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac0035f0 con 0x7fafc415efc0 2024-03-06T07:40:52.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.617+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac003a20 con 0x7fafc415efc0 2024-03-06T07:40:52.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.617+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc41e8330 con 0x7fafc415efc0 2024-03-06T07:40:52.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.617+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc41e88f0 con 0x7fafc415efc0 2024-03-06T07:40:52.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.618+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc404f330 con 0x7fafc415efc0 2024-03-06T07:40:52.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.620+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafac003bc0 con 0x7fafc415efc0 2024-03-06T07:40:52.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.621+0000 7fafbbe00640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fafa006b4d0 0x7fafa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:52.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.621+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fafac089620 con 0x7fafc415efc0 2024-03-06T07:40:52.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.621+0000 7fafc3e00640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fafa006b4d0 0x7fafa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:52.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.622+0000 7fafc3e00640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fafa006b4d0 0x7fafa006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fafb4036000 tx=0x7fafb4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:52.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.626+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafac0583c0 con 0x7fafc415efc0 2024-03-06T07:40:52.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.873+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafc41e8fb0 con 0x7fafc415efc0 2024-03-06T07:40:52.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.874+0000 7fafbbe00640 1 -- 172.21.15.32:0/1918755253 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fafac0010c0 con 0x7fafc415efc0 2024-03-06T07:40:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:52.877 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:52.878 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:52.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fafa006b4d0 msgr2=0x7fafa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fafa006b4d0 0x7fafa006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fafb4036000 tx=0x7fafb4030000 comp rx=0 tx=0).stop 2024-03-06T07:40:52.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 msgr2=0x7fafc41f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:52.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc41f73e0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c410 tx=0x7fafac00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 shutdown_connections 2024-03-06T07:40:52.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fafa006b4d0 0x7fafa006d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fafc41621d0 0x7fafc41f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafc415efc0 0x7fafc41f73e0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.877+0000 7fafc9400640 1 --2- 172.21.15.32:0/1918755253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc40cce90 0x7fafc415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:52.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.878+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 >> 172.21.15.32:0/1918755253 conn(0x7fafc406b3d0 msgr2=0x7fafc40b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:52.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.878+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 shutdown_connections 2024-03-06T07:40:52.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:52.878+0000 7fafc9400640 1 -- 172.21.15.32:0/1918755253 wait complete. 2024-03-06T07:40:53.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:53.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:54.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.342+0000 7fdad3200640 1 Processor -- start 2024-03-06T07:40:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.342+0000 7fdad3200640 1 -- start start 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.345+0000 7fdad3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 0x7fdad4168b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.345+0000 7fdad3200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad415e2b0 con 0x7fdad415ded0 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.345+0000 7fdad3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad40b6970 con 0x7fdad406c490 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.345+0000 7fdad3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad406c8f0 con 0x7fdad406d1c0 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.345+0000 7fdad1e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdad415ded0 0x7fdad40b61b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47804/0 (socket says 172.21.15.32:47804) 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.345+0000 7fdad1e00640 1 -- 172.21.15.32:0/1453078000 learned_addr learned my addr 172.21.15.32:0/1453078000 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad2800640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 0x7fdad4168b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454345975 0 0) 0x7fdad415e2b0 con 0x7fdad415ded0 2024-03-06T07:40:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8003520 con 0x7fdad415ded0 2024-03-06T07:40:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2379499969 0 0) 0x7fdab8003520 con 0x7fdad415ded0 2024-03-06T07:40:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 msgr2=0x7fdad4168b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 0x7fdad4168b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 >> v1:172.21.15.39:6789/0 conn(0x7fdad406c490 legacy=0x7fdad406cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad40b6c50 con 0x7fdad415ded0 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2411787964 0 0) 0x7fdac40019f0 con 0x7fdad415ded0 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.346+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 >> v1:172.21.15.32:6789/0 conn(0x7fdad415ded0 legacy=0x7fdad40b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad0a00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdab8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad0a00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 0x7fdab8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad0a00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdab8004d40 0x7fdab80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab800ba70 con 0x7fdab80039e0 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab800bc20 con 0x7fdab8004d40 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab800be20 con 0x7fdad406d1c0 2024-03-06T07:40:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdab8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad1400640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 0x7fdab8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.347+0000 7fdad2800640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdab8004d40 0x7fdab80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.348+0000 7fdad1e00640 1 -- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 msgr2=0x7fdab8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.348+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad406d1c0 0x7fdab8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.348+0000 7fdad1e00640 1 -- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdab8004d40 msgr2=0x7fdab80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.348+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdab8004d40 0x7fdab80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.348+0000 7fdad1e00640 1 -- 172.21.15.32:0/1453078000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab8003520 con 0x7fdab80039e0 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.349+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdab8003e00 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fdac400a830 tx=0x7fdab8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a929383efdb539eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.349+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400a480 con 0x7fdab80039e0 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.349+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac4034630 con 0x7fdab80039e0 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.349+0000 7fdad0a00640 1 -- 172.21.15.32:0/1453078000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4034a40 con 0x7fdab80039e0 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad3200640 1 -- 172.21.15.32:0/1453078000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 msgr2=0x7fdab8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad3200640 1 --2- 172.21.15.32:0/1453078000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdab8003e00 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fdac400a830 tx=0x7fdab8005830 comp rx=0 tx=0).stop 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad2800640 1 -- 172.21.15.32:0/1453078000 reap_dead start 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad3200640 1 -- 172.21.15.32:0/1453078000 shutdown_connections 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad3200640 1 -- 172.21.15.32:0/1453078000 >> 172.21.15.32:0/1453078000 conn(0x7fdad40731f0 msgr2=0x7fdad409aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad3200640 1 -- 172.21.15.32:0/1453078000 shutdown_connections 2024-03-06T07:40:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.350+0000 7fdad3200640 1 -- 172.21.15.32:0/1453078000 wait complete. 2024-03-06T07:40:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.351+0000 7fdad3200640 1 Processor -- start 2024-03-06T07:40:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.352+0000 7fdad3200640 1 -- start start 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdad4153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad3200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad4157a30 0x7fdad4157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad4156a90 0x7fdad4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40b0340 con 0x7fdab80039e0 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40b6f30 con 0x7fdad4156a90 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad3200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40b04c0 con 0x7fdad4157a30 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdad4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdad4153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58566/0 (socket says 172.21.15.32:58566) 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad1e00640 1 -- 172.21.15.32:0/1832456061 learned_addr learned my addr 172.21.15.32:0/1832456061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad1400640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad4157a30 0x7fdad4157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.353+0000 7fdad2800640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad4156a90 0x7fdad4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad1e00640 1 -- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad4157a30 msgr2=0x7fdad4157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad4157a30 0x7fdad4157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad1e00640 1 -- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad4156a90 msgr2=0x7fdad4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad4156a90 0x7fdad4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad1e00640 1 -- 172.21.15.32:0/1832456061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad4155cd0 con 0x7fdab80039e0 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad1e00640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdad4153950 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fdac400a950 tx=0x7fdad40aee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400a480 con 0x7fdab80039e0 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac4034430 con 0x7fdab80039e0 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad4155fb0 con 0x7fdab80039e0 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.354+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4004d70 con 0x7fdab80039e0 2024-03-06T07:40:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.355+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad4156570 con 0x7fdab80039e0 2024-03-06T07:40:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.356+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad404f330 con 0x7fdab80039e0 2024-03-06T07:40:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.357+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdac4047020 con 0x7fdab80039e0 2024-03-06T07:40:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.358+0000 7fdaca000640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdab006b4d0 0x7fdab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.359+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdac40ac250 con 0x7fdab80039e0 2024-03-06T07:40:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.359+0000 7fdad1400640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdab006b4d0 0x7fdab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.359+0000 7fdad1400640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdab006b4d0 0x7fdab006d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fdad406e740 tx=0x7fdabc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:54.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.362+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdac407aff0 con 0x7fdab80039e0 2024-03-06T07:40:54.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.601+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdad4162840 con 0x7fdab80039e0 2024-03-06T07:40:54.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.602+0000 7fdaca000640 1 -- 172.21.15.32:0/1832456061 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fdac4041e40 con 0x7fdab80039e0 2024-03-06T07:40:54.605 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:54.605 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:54.605 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdab006b4d0 msgr2=0x7fdab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdab006b4d0 0x7fdab006d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fdad406e740 tx=0x7fdabc009040 comp rx=0 tx=0).stop 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 msgr2=0x7fdad4153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdad4153950 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fdac400a950 tx=0x7fdad40aee10 comp rx=0 tx=0).stop 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 shutdown_connections 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fdab006b4d0 0x7fdab006d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad4156a90 0x7fdad4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fdad4157a30 0x7fdad4157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 --2- 172.21.15.32:0/1832456061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdab80039e0 0x7fdad4153950 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 >> 172.21.15.32:0/1832456061 conn(0x7fdad40731f0 msgr2=0x7fdad40b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 shutdown_connections 2024-03-06T07:40:54.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:54.605+0000 7fdad3200640 1 -- 172.21.15.32:0/1832456061 wait complete. 2024-03-06T07:40:54.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:54.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:55.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:40:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.046+0000 7fab46200640 1 Processor -- start 2024-03-06T07:40:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.046+0000 7fab46200640 1 -- start start 2024-03-06T07:40:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.048+0000 7fab46200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 0x7fab4015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.048+0000 7fab46200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fab400af580 0x7fab400b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.048+0000 7fab46200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4015cc40 con 0x7fab400cdc20 2024-03-06T07:40:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.048+0000 7fab46200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4015cf20 con 0x7fab400af580 2024-03-06T07:40:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.048+0000 7fab46200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4015d200 con 0x7fab400af1a0 2024-03-06T07:40:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.049+0000 7fab3f400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 0x7fab4015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.049+0000 7fab3ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fab400af580 0x7fab400b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.049+0000 7fab3f400640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 0x7fab4015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3301/0 says I am v2:172.21.15.32:39744/0 (socket says 172.21.15.32:39744) 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.049+0000 7fab3f400640 1 -- 172.21.15.32:0/1583995518 learned_addr learned my addr 172.21.15.32:0/1583995518 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.050+0000 7fab3f400640 1 -- 172.21.15.32:0/1583995518 >> v2:172.21.15.39:3300/0 conn(0x7fab400af580 msgr2=0x7fab400b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.050+0000 7fab3f400640 1 --2- 172.21.15.32:0/1583995518 >> v2:172.21.15.39:3300/0 conn(0x7fab400af580 0x7fab400b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.050+0000 7fab3f400640 1 -- 172.21.15.32:0/1583995518 >> v1:172.21.15.39:6789/0 conn(0x7fab400cdc20 legacy=0x7fab400b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.050+0000 7fab3f400640 1 -- 172.21.15.32:0/1583995518 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4015d4e0 con 0x7fab400af1a0 2024-03-06T07:40:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.051+0000 7fab3f400640 1 --2- 172.21.15.32:0/1583995518 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 0x7fab4015c600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fab30009870 tx=0x7fab3002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eefc562fb50eff81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.052+0000 7fab3e000640 1 -- 172.21.15.32:0/1583995518 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30035020 con 0x7fab400af1a0 2024-03-06T07:40:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.052+0000 7fab3e000640 1 -- 172.21.15.32:0/1583995518 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30031640 con 0x7fab400af1a0 2024-03-06T07:40:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.052+0000 7fab3e000640 1 -- 172.21.15.32:0/1583995518 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3002d040 con 0x7fab400af1a0 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 -- 172.21.15.32:0/1583995518 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 msgr2=0x7fab4015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 --2- 172.21.15.32:0/1583995518 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 0x7fab4015c600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fab30009870 tx=0x7fab3002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 -- 172.21.15.32:0/1583995518 shutdown_connections 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 --2- 172.21.15.32:0/1583995518 >> v2:172.21.15.39:3300/0 conn(0x7fab400af580 0x7fab400b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 --2- 172.21.15.32:0/1583995518 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400af1a0 0x7fab4015c600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 -- 172.21.15.32:0/1583995518 >> 172.21.15.32:0/1583995518 conn(0x7fab4006afb0 msgr2=0x7fab4006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 -- 172.21.15.32:0/1583995518 shutdown_connections 2024-03-06T07:40:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.053+0000 7fab46200640 1 -- 172.21.15.32:0/1583995518 wait complete. 2024-03-06T07:40:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.055+0000 7fab46200640 1 Processor -- start 2024-03-06T07:40:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.055+0000 7fab46200640 1 -- start start 2024-03-06T07:40:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.056+0000 7fab46200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 0x7fab401f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.056+0000 7fab46200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab400af580 0x7fab4009af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.056+0000 7fab46200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400cdc20 0x7fab4014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.056+0000 7fab46200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab400b3660 con 0x7fab400af1a0 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.056+0000 7fab46200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab400b34e0 con 0x7fab400af580 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.056+0000 7fab46200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab400b37e0 con 0x7fab400cdc20 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 0x7fab401f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 0x7fab401f4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58580/0 (socket says 172.21.15.32:58580) 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 -- 172.21.15.32:0/3054194221 learned_addr learned my addr 172.21.15.32:0/3054194221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3ea00640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab400af580 0x7fab4009af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3fe00640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400cdc20 0x7fab4014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 -- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400cdc20 msgr2=0x7fab4014e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400cdc20 0x7fab4014e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 -- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab400af580 msgr2=0x7fab4009af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab400af580 0x7fab4009af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.057+0000 7fab3f400640 1 -- 172.21.15.32:0/3054194221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4009dcd0 con 0x7fab400af1a0 2024-03-06T07:40:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.058+0000 7fab3f400640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 0x7fab401f4c20 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fab3002b930 tx=0x7fab300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.058+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30040070 con 0x7fab400af1a0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.058+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3003e070 con 0x7fab400af1a0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.058+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30035040 con 0x7fab400af1a0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.058+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4006cd60 con 0x7fab400af1a0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.058+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4006d320 con 0x7fab400af1a0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.059+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4004f3a0 con 0x7fab400af1a0 2024-03-06T07:40:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.060+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab30038020 con 0x7fab400af1a0 2024-03-06T07:40:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.062+0000 7fab37400640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab2006b4d0 0x7fab2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.062+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fab300abc20 con 0x7fab400af1a0 2024-03-06T07:40:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.062+0000 7fab3ea00640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab2006b4d0 0x7fab2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:56.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.063+0000 7fab3ea00640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab2006b4d0 0x7fab2006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fab401528f0 tx=0x7fab2c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.066+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab3007aa50 con 0x7fab400af1a0 2024-03-06T07:40:56.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.311+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab4006cb10 con 0x7fab400af1a0 2024-03-06T07:40:56.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.312+0000 7fab37400640 1 -- 172.21.15.32:0/3054194221 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3461 (secure 0 0 0) 0x7fab3007a350 con 0x7fab400af1a0 2024-03-06T07:40:56.314 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:40:56.314 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:36:55.275098+0000","modified":"2024-03-06T07:39:25.398550+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8313,"mds_1":6574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6574":{"gid":6574,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.39:6834/1843750889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1843750889},{"type":"v1","addr":"172.21.15.39:6834","nonce":1843750889}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:37:55.811074+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8313":{"gid":8313,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6836/294605993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6835","nonce":294605993},{"type":"v1","addr":"172.21.15.32:6836","nonce":294605993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8313,"qdb_cluster":[8313]},"id":9}]} 2024-03-06T07:40:56.315 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2024-03-06T07:40:56.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.314+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab2006b4d0 msgr2=0x7fab2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:56.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.314+0000 7fab46200640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab2006b4d0 0x7fab2006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fab401528f0 tx=0x7fab2c008040 comp rx=0 tx=0).stop 2024-03-06T07:40:56.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.314+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 msgr2=0x7fab401f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:56.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.314+0000 7fab46200640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 0x7fab401f4c20 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fab3002b930 tx=0x7fab300046c0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 shutdown_connections 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fab2006b4d0 0x7fab2006d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fab400cdc20 0x7fab4014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab400af580 0x7fab4009af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 --2- 172.21.15.32:0/3054194221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab400af1a0 0x7fab401f4c20 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 >> 172.21.15.32:0/3054194221 conn(0x7fab4006afb0 msgr2=0x7fab400ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 shutdown_connections 2024-03-06T07:40:56.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:56.315+0000 7fab46200640 1 -- 172.21.15.32:0/3054194221 wait complete. 2024-03-06T07:40:56.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:36:55.275098+0000', 'modified': '2024-03-06T07:39:25.398550+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8313, 'mds_1': 6574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6574': {'gid': 6574, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.39:6834/1843750889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 1843750889}, {'type': 'v1', 'addr': '172.21.15.39:6834', 'nonce': 1843750889}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:37:55.811074+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8313': {'gid': 8313, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6836/294605993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6835', 'nonce': 294605993}, {'type': 'v1', 'addr': '172.21.15.32:6836', 'nonce': 294605993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8313, 'qdb_cluster': [8313]} 2024-03-06T07:40:56.537 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6574:up:stopping 2024-03-06T07:40:57.537 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:57.538 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:36:55.275098+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6574": { "addr": "172.21.15.39:6834/1843750889", "addrs": { "addrvec": [ { "addr": "172.21.15.39:6832", "nonce": 1843750889, "type": "v2" }, { "addr": "172.21.15.39:6834", "nonce": 1843750889, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6574, "incarnation": 93, "join_fscid": -1, "laggy_since": "2024-03-06T07:37:55.811074+0000", "name": "b", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_8313": { "addr": "172.21.15.32:6836/294605993", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6835", "nonce": 294605993, "type": "v2" }, { "addr": "172.21.15.32:6836", "nonce": 294605993, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8313, "incarnation": 87, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-03-06T07:39:25.398550+0000", "qdb_cluster": [ 8313 ], "qdb_leader": 8313, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8313, "mds_1": 6574 } } } ], "standbys": [] } 2024-03-06T07:40:57.538 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:40:57.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:40:57.657 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:40:57.794 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:40:57.794 INFO:tasks.cephfs.mount:Terminating background process 2024-03-06T07:40:57.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:57 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:40:59.539 INFO:teuthology.orchestra.run.smithi032.stderr:open_dir_background: keeping dir open 2024-03-06T07:40:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:40:59.571 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:40:59.572 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:40:59.670 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-06T07:40:59.670 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:40:59.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:40:59.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.938+0000 7fc2b4e00640 1 Processor -- start 2024-03-06T07:40:59.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.939+0000 7fc2b4e00640 1 -- start start 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2b4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2b00d2b40 0x7fc2b00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2b4e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc2b00d1480 0x7fc2b00b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2b4e00640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b00b3940 con 0x7fc2b015e2e0 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2b4e00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b00d4e70 con 0x7fc2b00d1480 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2b4e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b00b3c40 con 0x7fc2b00d2b40 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2ade00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc2b015e2e0 0x7fc2b00d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55282/0 (socket says 172.21.15.32:55282) 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2ade00640 1 -- 172.21.15.32:0/3626102319 learned_addr learned my addr 172.21.15.32:0/3626102319 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.940+0000 7fc2ae800640 1 --2- 172.21.15.32:0/3626102319 >> v2:172.21.15.32:3300/0 conn(0x7fc2b00d1480 0x7fc2b00b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.941+0000 7fc2a5400640 1 --2- 172.21.15.32:0/3626102319 >> v2:172.21.15.39:3300/0 conn(0x7fc2b00d2b40 0x7fc2b00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.941+0000 7fc2ae800640 1 -- 172.21.15.32:0/3626102319 >> v2:172.21.15.39:3300/0 conn(0x7fc2b00d2b40 msgr2=0x7fc2b00d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:40:59.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.941+0000 7fc2ae800640 1 --2- 172.21.15.32:0/3626102319 >> v2:172.21.15.39:3300/0 conn(0x7fc2b00d2b40 0x7fc2b00d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.941+0000 7fc2ae800640 1 -- 172.21.15.32:0/3626102319 >> v1:172.21.15.32:6789/0 conn(0x7fc2b015e2e0 legacy=0x7fc2b00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.941+0000 7fc2ae800640 1 -- 172.21.15.32:0/3626102319 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b00cf580 con 0x7fc2b00d1480 2024-03-06T07:40:59.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.941+0000 7fc2ad400640 1 -- 172.21.15.32:0/3626102319 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196550575 0 0) 0x7fc2b00b3940 con 0x7fc2b015e2e0 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.942+0000 7fc2ae800640 1 --2- 172.21.15.32:0/3626102319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00d1480 0x7fc2b00b40d0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00d3510 tx=0x7fc2a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a61aa9f7aa1091 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2ad400640 1 -- 172.21.15.32:0/3626102319 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8035020 con 0x7fc2b00d1480 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2ad400640 1 -- 172.21.15.32:0/3626102319 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8031830 con 0x7fc2b00d1480 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2ad400640 1 -- 172.21.15.32:0/3626102319 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8031c00 con 0x7fc2b00d1480 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 -- 172.21.15.32:0/3626102319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00d1480 msgr2=0x7fc2b00b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/3626102319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00d1480 0x7fc2b00b40d0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00d3510 tx=0x7fc2a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 -- 172.21.15.32:0/3626102319 shutdown_connections 2024-03-06T07:40:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/3626102319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00d1480 0x7fc2b00b40d0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/3626102319 >> v2:172.21.15.39:3300/0 conn(0x7fc2b00d2b40 0x7fc2b00d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 -- 172.21.15.32:0/3626102319 >> 172.21.15.32:0/3626102319 conn(0x7fc2b00d3f60 msgr2=0x7fc2b00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:40:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.943+0000 7fc2b4e00640 1 -- 172.21.15.32:0/3626102319 shutdown_connections 2024-03-06T07:40:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.944+0000 7fc2b4e00640 1 -- 172.21.15.32:0/3626102319 wait complete. 2024-03-06T07:40:59.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.946+0000 7fc2b4e00640 1 Processor -- start 2024-03-06T07:40:59.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.946+0000 7fc2b4e00640 1 -- start start 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.947+0000 7fc2b4e00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2b00d2b40 0x7fc2b01534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.947+0000 7fc2b4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 0x7fc2b00b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2b4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2b00c97d0 0x7fc2b015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2b4e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00cfe20 con 0x7fc2b015e2e0 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2b4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00cfb40 con 0x7fc2b00c97d0 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2b4e00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00b4610 con 0x7fc2b00d2b40 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 0x7fc2b00b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 0x7fc2b00b81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41042/0 (socket says 172.21.15.32:41042) 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 -- 172.21.15.32:0/2228510076 learned_addr learned my addr 172.21.15.32:0/2228510076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2ae800640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2b00c97d0 0x7fc2b015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2ade00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2b00d2b40 0x7fc2b01534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 -- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2b00d2b40 msgr2=0x7fc2b01534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2b00d2b40 0x7fc2b01534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 -- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2b00c97d0 msgr2=0x7fc2b015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:40:59.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2b00c97d0 0x7fc2b015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:40:59.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2a5400640 1 -- 172.21.15.32:0/2228510076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b0160a40 con 0x7fc2b015e2e0 2024-03-06T07:40:59.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.948+0000 7fc2ade00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2b00d2b40 0x7fc2b01534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:40:59.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.949+0000 7fc2a5400640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 0x7fc2b00b81d0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fc29800e980 tx=0x7fc29800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.949+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298017070 con 0x7fc2b015e2e0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.949+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29800ca00 con 0x7fc2b015e2e0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.949+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29800cdd0 con 0x7fc2b015e2e0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.949+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b0153a30 con 0x7fc2b015e2e0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.949+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b00b89b0 con 0x7fc2b015e2e0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.950+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b004f3a0 con 0x7fc2b015e2e0 2024-03-06T07:40:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.951+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc29801a070 con 0x7fc2b015e2e0 2024-03-06T07:40:59.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.953+0000 7fc2a6a00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:40:59.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.953+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc29808a2c0 con 0x7fc2b015e2e0 2024-03-06T07:40:59.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.953+0000 7fc2ade00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:40:59.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.954+0000 7fc2ade00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0007fd0 tx=0x7fc2a0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:40:59.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:40:59.957+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2b0153a30 con 0x7fc2b015e2e0 2024-03-06T07:41:00.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.158+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2b00b5310 con 0x7fc2b015e2e0 2024-03-06T07:41:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.161+0000 7fc2a6a00640 1 -- 172.21.15.32:0/2228510076 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16613 (secure 0 0 0) 0x7fc2b0160a40 con 0x7fc2b015e2e0 2024-03-06T07:41:00.163 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-06T07:41:00.163 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":131,"fsid":"e91bbd07-5556-4949-b381-99a73b242ae4","created":"2024-03-06T07:28:40.936855+0000","modified":"2024-03-06T07:37:49.435081+0000","last_up_change":"2024-03-06T07:28:52.988900+0000","last_in_change":"2024-03-06T07:28:44.501404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:28:54.261946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:36:53.245687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:36:54.277198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"381a2d5a-6911-40e0-93b2-252fbd01807f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6818","nonce":3728617102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6822","nonce":3728617102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6827","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6829","nonce":3728617102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6823","nonce":3728617102},{"type":"v1","addr":"172.21.15.32:6825","nonce":3728617102}]},"public_addr":"172.21.15.32:6818/3728617102","cluster_addr":"172.21.15.32:6822/3728617102","heartbeat_back_addr":"172.21.15.32:6829/3728617102","heartbeat_front_addr":"172.21.15.32:6825/3728617102","state":["exists","up"]},{"osd":1,"uuid":"ec67ac0c-0584-4a74-8116-1480561db4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6817","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6819","nonce":50998742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6821","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6824","nonce":50998742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6831","nonce":50998742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":50998742},{"type":"v1","addr":"172.21.15.32:6828","nonce":50998742}]},"public_addr":"172.21.15.32:6819/50998742","cluster_addr":"172.21.15.32:6824/50998742","heartbeat_back_addr":"172.21.15.32:6831/50998742","heartbeat_front_addr":"172.21.15.32:6828/50998742","state":["exists","up"]},{"osd":2,"uuid":"c45c108c-3f06-4132-9202-b9553ebeb4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6801","nonce":1600044506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6803","nonce":1600044506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6807","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6809","nonce":1600044506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1600044506},{"type":"v1","addr":"172.21.15.32:6805","nonce":1600044506}]},"public_addr":"172.21.15.32:6801/1600044506","cluster_addr":"172.21.15.32:6803/1600044506","heartbeat_back_addr":"172.21.15.32:6809/1600044506","heartbeat_front_addr":"172.21.15.32:6805/1600044506","state":["exists","up"]},{"osd":3,"uuid":"27319d10-b65b-4934-ab15-58281f33ecc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6808","nonce":238384581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6811","nonce":238384581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6815","nonce":238384581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":238384581},{"type":"v1","addr":"172.21.15.32:6813","nonce":238384581}]},"public_addr":"172.21.15.32:6808/238384581","cluster_addr":"172.21.15.32:6811/238384581","heartbeat_back_addr":"172.21.15.32:6815/238384581","heartbeat_front_addr":"172.21.15.32:6813/238384581","state":["exists","up"]},{"osd":4,"uuid":"c234adf2-055d-4dd0-adac-7e68bc54ebd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6801","nonce":89256438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6803","nonce":89256438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6807","nonce":89256438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":89256438},{"type":"v1","addr":"172.21.15.39:6805","nonce":89256438}]},"public_addr":"172.21.15.39:6801/89256438","cluster_addr":"172.21.15.39:6803/89256438","heartbeat_back_addr":"172.21.15.39:6807/89256438","heartbeat_front_addr":"172.21.15.39:6805/89256438","state":["exists","up"]},{"osd":5,"uuid":"3be1b0dc-2152-420f-b869-5cb82c5136a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6817","nonce":969666024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6819","nonce":969666024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6824","nonce":969666024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":969666024},{"type":"v1","addr":"172.21.15.39:6821","nonce":969666024}]},"public_addr":"172.21.15.39:6817/969666024","cluster_addr":"172.21.15.39:6819/969666024","heartbeat_back_addr":"172.21.15.39:6824/969666024","heartbeat_front_addr":"172.21.15.39:6821/969666024","state":["exists","up"]},{"osd":6,"uuid":"39783030-63d8-4a9d-8909-7bac4d53da31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6809","nonce":2944464285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6811","nonce":2944464285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6815","nonce":2944464285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2944464285},{"type":"v1","addr":"172.21.15.39:6813","nonce":2944464285}]},"public_addr":"172.21.15.39:6809/2944464285","cluster_addr":"172.21.15.39:6811/2944464285","heartbeat_back_addr":"172.21.15.39:6815/2944464285","heartbeat_front_addr":"172.21.15.39:6813/2944464285","state":["exists","up"]},{"osd":7,"uuid":"f10c34e2-acc4-460e-97f6-7da3d7ae631c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6823","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6825","nonce":3019464823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6827","nonce":3019464823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6831","nonce":3019464823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3019464823},{"type":"v1","addr":"172.21.15.39:6829","nonce":3019464823}]},"public_addr":"172.21.15.39:6825/3019464823","cluster_addr":"172.21.15.39:6827/3019464823","heartbeat_back_addr":"172.21.15.39:6831/3019464823","heartbeat_front_addr":"172.21.15.39:6829/3019464823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.852229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.888095+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.865131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.772680+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.814912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:50.044552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:28:49.849536+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.3f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:41:00.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 msgr2=0x7fc28c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0007fd0 tx=0x7fc2a0009040 comp rx=0 tx=0).stop 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 msgr2=0x7fc2b00b81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 0x7fc2b00b81d0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fc29800e980 tx=0x7fc29800ee40 comp rx=0 tx=0).stop 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 shutdown_connections 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2b00c97d0 0x7fc2b015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b015e2e0 0x7fc2b00b81d0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/2228510076 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7fc2b00d2b40 0x7fc2b01534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 >> 172.21.15.32:0/2228510076 conn(0x7fc2b00d3f60 msgr2=0x7fc2b006c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 shutdown_connections 2024-03-06T07:41:00.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.164+0000 7fc2b4e00640 1 -- 172.21.15.32:0/2228510076 wait complete. 2024-03-06T07:41:00.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:28:54.261946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:36:53.245687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:36:54.277198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:41:00.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:41:00.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.390+0000 7f226a600640 1 Processor -- start 2024-03-06T07:41:00.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.391+0000 7f226a600640 1 -- start start 2024-03-06T07:41:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.392+0000 7f226a600640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22640d07d0 0x7f22640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.392+0000 7f226a600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f22640d4930 0x7f226415dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.392+0000 7f226a600640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22640cf780 con 0x7f22640d0bb0 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.392+0000 7f226a600640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22640b3770 con 0x7f22640d07d0 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.392+0000 7f226a600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22640cfa60 con 0x7f22640d4930 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22640d07d0 0x7f22640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22640d07d0 0x7f22640b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41046/0 (socket says 172.21.15.32:41046) 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 -- 172.21.15.32:0/2336477918 learned_addr learned my addr 172.21.15.32:0/2336477918 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263e00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f22640d0bb0 0x7f22640d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55290/0 (socket says 172.21.15.32:55290) 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2262a00640 1 --2- 172.21.15.32:0/2336477918 >> v2:172.21.15.39:3300/0 conn(0x7f22640d4930 0x7f226415dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 -- 172.21.15.32:0/2336477918 >> v2:172.21.15.39:3300/0 conn(0x7f22640d4930 msgr2=0x7f226415dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 --2- 172.21.15.32:0/2336477918 >> v2:172.21.15.39:3300/0 conn(0x7f22640d4930 0x7f226415dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 -- 172.21.15.32:0/2336477918 >> v1:172.21.15.32:6789/0 conn(0x7f22640d0bb0 legacy=0x7f22640d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.393+0000 7f2263400640 1 -- 172.21.15.32:0/2336477918 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640cfd40 con 0x7f22640d07d0 2024-03-06T07:41:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.394+0000 7f2263400640 1 --2- 172.21.15.32:0/2336477918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d07d0 0x7f22640b4f80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f226408c800 tx=0x7f225402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b902a449d86c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f2262000640 1 -- 172.21.15.32:0/2336477918 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225402f850 con 0x7f22640d07d0 2024-03-06T07:41:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f2262000640 1 -- 172.21.15.32:0/2336477918 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225402f9f0 con 0x7f22640d07d0 2024-03-06T07:41:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f2262000640 1 -- 172.21.15.32:0/2336477918 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225402fdc0 con 0x7f22640d07d0 2024-03-06T07:41:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f226a600640 1 -- 172.21.15.32:0/2336477918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d07d0 msgr2=0x7f22640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f226a600640 1 --2- 172.21.15.32:0/2336477918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d07d0 0x7f22640b4f80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f226408c800 tx=0x7f225402ba30 comp rx=0 tx=0).stop 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f226a600640 1 -- 172.21.15.32:0/2336477918 shutdown_connections 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f226a600640 1 --2- 172.21.15.32:0/2336477918 >> v2:172.21.15.39:3300/0 conn(0x7f22640d4930 0x7f226415dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.395+0000 7f226a600640 1 --2- 172.21.15.32:0/2336477918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d07d0 0x7f22640b4f80 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.396+0000 7f226a600640 1 -- 172.21.15.32:0/2336477918 >> 172.21.15.32:0/2336477918 conn(0x7f22640d3e10 msgr2=0x7f22640d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.396+0000 7f226a600640 1 -- 172.21.15.32:0/2336477918 shutdown_connections 2024-03-06T07:41:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.396+0000 7f226a600640 1 -- 172.21.15.32:0/2336477918 wait complete. 2024-03-06T07:41:00.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.398+0000 7f226a600640 1 Processor -- start 2024-03-06T07:41:00.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.398+0000 7f226a600640 1 -- start start 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f226a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22640d07d0 0x7f22640ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f226a600640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22640d0bb0 0x7f22640cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f226a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 0x7f22640b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f226a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640b4d90 con 0x7f22640d4930 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f226a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640b4c10 con 0x7f22640d07d0 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f226a600640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226415e4f0 con 0x7f22640d0bb0 2024-03-06T07:41:00.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f2263e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 0x7f22640b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f2263e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 0x7f22640b8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41050/0 (socket says 172.21.15.32:41050) 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f2263e00640 1 -- 172.21.15.32:0/2552618308 learned_addr learned my addr 172.21.15.32:0/2552618308 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f2263400640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22640d07d0 0x7f22640ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.399+0000 7f2262a00640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22640d0bb0 0x7f22640cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f2263e00640 1 -- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22640d0bb0 msgr2=0x7f22640cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f2263e00640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22640d0bb0 0x7f22640cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f2263e00640 1 -- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22640d07d0 msgr2=0x7f22640ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f2263e00640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22640d07d0 0x7f22640ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f2263e00640 1 -- 172.21.15.32:0/2552618308 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640cd560 con 0x7f22640d4930 2024-03-06T07:41:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f2263e00640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 0x7f22640b8b40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f225c00c780 tx=0x7f225c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c017070 con 0x7f22640d4930 2024-03-06T07:41:00.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00a640 con 0x7f22640d4930 2024-03-06T07:41:00.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00a9f0 con 0x7f22640d4930 2024-03-06T07:41:00.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22640cd840 con 0x7f22640d4930 2024-03-06T07:41:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.400+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22640cde00 con 0x7f22640d4930 2024-03-06T07:41:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.402+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226404f330 con 0x7f22640d4930 2024-03-06T07:41:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.403+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f225c01a070 con 0x7f22640d4930 2024-03-06T07:41:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.404+0000 7f225b400640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f224406b4d0 0x7f224406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.404+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f225c0896f0 con 0x7f22640d4930 2024-03-06T07:41:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.405+0000 7f2263400640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f224406b4d0 0x7f224406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.405+0000 7f2263400640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f224406b4d0 0x7f224406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2254036000 tx=0x7f2254008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.409+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f225c057ee0 con 0x7f22640d4930 2024-03-06T07:41:00.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.610+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22640b1420 con 0x7f22640d4930 2024-03-06T07:41:00.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.611+0000 7f225b400640 1 -- 172.21.15.32:0/2552618308 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f225c05bf80 con 0x7f22640d4930 2024-03-06T07:41:00.612 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-03-06T07:41:00.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f224406b4d0 msgr2=0x7f224406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f224406b4d0 0x7f224406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2254036000 tx=0x7f2254008030 comp rx=0 tx=0).stop 2024-03-06T07:41:00.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 msgr2=0x7f22640b8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 0x7f22640b8b40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f225c00c780 tx=0x7f225c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:41:00.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 shutdown_connections 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f224406b4d0 0x7f224406d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22640d4930 0x7f22640b8b40 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f22640d0bb0 0x7f22640cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.613+0000 7f226a600640 1 --2- 172.21.15.32:0/2552618308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22640d07d0 0x7f22640ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.614+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 >> 172.21.15.32:0/2552618308 conn(0x7f22640d3e10 msgr2=0x7f226406cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.614+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 shutdown_connections 2024-03-06T07:41:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.614+0000 7f226a600640 1 -- 172.21.15.32:0/2552618308 wait complete. 2024-03-06T07:41:00.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:41:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.848+0000 7feac4c00640 1 Processor -- start 2024-03-06T07:41:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.848+0000 7feac4c00640 1 -- start start 2024-03-06T07:41:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feac4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac0145430 0x7feac015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feac4c00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feac015cc90 0x7feac00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feac4c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac00b3fe0 con 0x7feac00b2080 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feac4c00640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac015fe40 con 0x7feac015cc90 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feac4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac0160120 con 0x7feac0145430 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feabea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feac015cc90 0x7feac00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feabea00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feac015cc90 0x7feac00b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41062/0 (socket says 172.21.15.32:41062) 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.850+0000 7feabea00640 1 -- 172.21.15.32:0/3332541930 learned_addr learned my addr 172.21.15.32:0/3332541930 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.851+0000 7feabf400640 1 --2- 172.21.15.32:0/3332541930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac0145430 0x7feac015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.851+0000 7feabea00640 1 -- 172.21.15.32:0/3332541930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac0145430 msgr2=0x7feac015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.851+0000 7feabea00640 1 --2- 172.21.15.32:0/3332541930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac0145430 0x7feac015f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.851+0000 7feabea00640 1 -- 172.21.15.32:0/3332541930 >> v1:172.21.15.39:6789/0 conn(0x7feac00b2080 legacy=0x7feac006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.851+0000 7feabea00640 1 -- 172.21.15.32:0/3332541930 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac0160400 con 0x7feac015cc90 2024-03-06T07:41:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.852+0000 7feabea00640 1 --2- 172.21.15.32:0/3332541930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac015cc90 0x7feac00b1b40 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7feaac009870 tx=0x7feaac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cedbe38051210b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.852+0000 7feabe000640 1 -- 172.21.15.32:0/3332541930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac035020 con 0x7feac015cc90 2024-03-06T07:41:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.852+0000 7feabe000640 1 -- 172.21.15.32:0/3332541930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac031640 con 0x7feac015cc90 2024-03-06T07:41:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feabe000640 1 -- 172.21.15.32:0/3332541930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac02d040 con 0x7feac015cc90 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feac4c00640 1 -- 172.21.15.32:0/3332541930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac015cc90 msgr2=0x7feac00b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feac4c00640 1 --2- 172.21.15.32:0/3332541930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac015cc90 0x7feac00b1b40 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7feaac009870 tx=0x7feaac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feac4c00640 1 -- 172.21.15.32:0/3332541930 shutdown_connections 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feac4c00640 1 --2- 172.21.15.32:0/3332541930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac015cc90 0x7feac00b1b40 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feac4c00640 1 --2- 172.21.15.32:0/3332541930 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac0145430 0x7feac015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.853+0000 7feac4c00640 1 -- 172.21.15.32:0/3332541930 >> 172.21.15.32:0/3332541930 conn(0x7feac006b3a0 msgr2=0x7feac006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.854+0000 7feac4c00640 1 -- 172.21.15.32:0/3332541930 shutdown_connections 2024-03-06T07:41:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.854+0000 7feac4c00640 1 -- 172.21.15.32:0/3332541930 wait complete. 2024-03-06T07:41:00.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.856+0000 7feac4c00640 1 Processor -- start 2024-03-06T07:41:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.857+0000 7feac4c00640 1 -- start start 2024-03-06T07:41:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.858+0000 7feac4c00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00b2080 0x7feac013ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.858+0000 7feac4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 0x7feac013f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.858+0000 7feac4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac015cc90 0x7feac0136c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.858+0000 7feac4c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0145810 con 0x7feac0145430 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.858+0000 7feac4c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac015d0d0 con 0x7feac015cc90 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.858+0000 7feac4c00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac0145990 con 0x7feac00b2080 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 0x7feac013f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 0x7feac013f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41064/0 (socket says 172.21.15.32:41064) 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 -- 172.21.15.32:0/981122170 learned_addr learned my addr 172.21.15.32:0/981122170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabf400640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00b2080 0x7feac013ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 -- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00b2080 msgr2=0x7feac013ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00b2080 0x7feac013ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabfe00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac015cc90 0x7feac0136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 -- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac015cc90 msgr2=0x7feac0136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac015cc90 0x7feac0136c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.859+0000 7feabea00640 1 -- 172.21.15.32:0/981122170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac013a1b0 con 0x7feac0145430 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.860+0000 7feabea00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 0x7feac013f100 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7feaac02b930 tx=0x7feaac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.860+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac03aa60 con 0x7feac0145430 2024-03-06T07:41:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.860+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac03ac20 con 0x7feac0145430 2024-03-06T07:41:00.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.860+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac0443f0 con 0x7feac0145430 2024-03-06T07:41:00.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.860+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac01f1d50 con 0x7feac0145430 2024-03-06T07:41:00.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.860+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac01f1ff0 con 0x7feac0145430 2024-03-06T07:41:00.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.861+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac004f330 con 0x7feac0145430 2024-03-06T07:41:00.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.862+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feaac04b060 con 0x7feac0145430 2024-03-06T07:41:00.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.863+0000 7feab7400640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feaa006b520 0x7feaa006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:00.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.863+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7feaac0abe00 con 0x7feac0145430 2024-03-06T07:41:00.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.864+0000 7feabf400640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feaa006b520 0x7feaa006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:00.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.864+0000 7feabf400640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feaa006b520 0x7feaa006d9e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feab0006fd0 tx=0x7feab0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:00.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:00.872+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feac013a1b0 con 0x7feac0145430 2024-03-06T07:41:01.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.070+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feac00ce0e0 con 0x7feac0145430 2024-03-06T07:41:01.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.071+0000 7feab7400640 1 -- 172.21.15.32:0/981122170 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7feaac03f070 con 0x7feac0145430 2024-03-06T07:41:01.073 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feaa006b520 msgr2=0x7feaa006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feaa006b520 0x7feaa006d9e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feab0006fd0 tx=0x7feab0008040 comp rx=0 tx=0).stop 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 msgr2=0x7feac013f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 0x7feac013f100 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7feaac02b930 tx=0x7feaac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 shutdown_connections 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7feaa006b520 0x7feaa006d9e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac015cc90 0x7feac0136c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feac0145430 0x7feac013f100 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 --2- 172.21.15.32:0/981122170 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7feac00b2080 0x7feac013ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 >> 172.21.15.32:0/981122170 conn(0x7feac006b3a0 msgr2=0x7feac013fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:01.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 shutdown_connections 2024-03-06T07:41:01.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.074+0000 7feac4c00640 1 -- 172.21.15.32:0/981122170 wait complete. 2024-03-06T07:41:01.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:41:01.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.306+0000 7f2feb200640 1 Processor -- start 2024-03-06T07:41:01.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.306+0000 7f2feb200640 1 -- start start 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2feb200640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fec15dee0 0x7f2fec0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2feb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2fec0d0d40 0x7f2fec0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2feb200640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec0ceca0 con 0x7f2fec0d4860 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2feb200640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec0cf5d0 con 0x7f2fec15dee0 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2feb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec0cef80 con 0x7f2fec0d0d40 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2fe9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fec15dee0 0x7f2fec0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2fe9e00640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fec15dee0 0x7f2fec0d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41076/0 (socket says 172.21.15.32:41076) 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2fe9e00640 1 -- 172.21.15.32:0/4055426063 learned_addr learned my addr 172.21.15.32:0/4055426063 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.308+0000 7f2fe3e00640 1 --1- 172.21.15.32:0/4055426063 >> v1:172.21.15.32:6789/0 conn(0x7f2fec0d4860 0x7f2fec0d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55298/0 (socket says 172.21.15.32:55298) 2024-03-06T07:41:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.309+0000 7f2fea800640 1 --2- 172.21.15.32:0/4055426063 >> v2:172.21.15.39:3300/0 conn(0x7f2fec0d0d40 0x7f2fec0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.309+0000 7f2fe9e00640 1 -- 172.21.15.32:0/4055426063 >> v2:172.21.15.39:3300/0 conn(0x7f2fec0d0d40 msgr2=0x7f2fec0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.309+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/4055426063 >> v2:172.21.15.39:3300/0 conn(0x7f2fec0d0d40 0x7f2fec0b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.309+0000 7f2fe9e00640 1 -- 172.21.15.32:0/4055426063 >> v1:172.21.15.32:6789/0 conn(0x7f2fec0d4860 legacy=0x7f2fec0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.309+0000 7f2fe9e00640 1 -- 172.21.15.32:0/4055426063 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0cf260 con 0x7f2fec15dee0 2024-03-06T07:41:01.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.309+0000 7f2fe9400640 1 -- 172.21.15.32:0/4055426063 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293757816 0 0) 0x7f2fec0ceca0 con 0x7f2fec0d4860 2024-03-06T07:41:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.311+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/4055426063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec15dee0 0x7f2fec0d18a0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2fec06b2d0 tx=0x7f2fdc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1b7185b50724b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.311+0000 7f2fe9400640 1 -- 172.21.15.32:0/4055426063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc034070 con 0x7f2fec15dee0 2024-03-06T07:41:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.311+0000 7f2fe9400640 1 -- 172.21.15.32:0/4055426063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc030860 con 0x7f2fec15dee0 2024-03-06T07:41:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.311+0000 7f2fe9400640 1 -- 172.21.15.32:0/4055426063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc030c30 con 0x7f2fec15dee0 2024-03-06T07:41:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 -- 172.21.15.32:0/4055426063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec15dee0 msgr2=0x7f2fec0d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 --2- 172.21.15.32:0/4055426063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec15dee0 0x7f2fec0d18a0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2fec06b2d0 tx=0x7f2fdc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 -- 172.21.15.32:0/4055426063 shutdown_connections 2024-03-06T07:41:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 --2- 172.21.15.32:0/4055426063 >> v2:172.21.15.39:3300/0 conn(0x7f2fec0d0d40 0x7f2fec0b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 --2- 172.21.15.32:0/4055426063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec15dee0 0x7f2fec0d18a0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 -- 172.21.15.32:0/4055426063 >> 172.21.15.32:0/4055426063 conn(0x7f2fec0d3cd0 msgr2=0x7f2fec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 -- 172.21.15.32:0/4055426063 shutdown_connections 2024-03-06T07:41:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.312+0000 7f2feb200640 1 -- 172.21.15.32:0/4055426063 wait complete. 2024-03-06T07:41:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.314+0000 7f2feb200640 1 Processor -- start 2024-03-06T07:41:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.314+0000 7f2feb200640 1 -- start start 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2feb200640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fec0d0d40 0x7f2fec14b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2feb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 0x7f2fec156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2feb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15dee0 0x7f2fec06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2feb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0d1de0 con 0x7f2fec0d4860 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2feb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec15e2c0 con 0x7f2fec15dee0 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2feb200640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0d1f60 con 0x7f2fec0d0d40 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 0x7f2fec156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 0x7f2fec156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41092/0 (socket says 172.21.15.32:41092) 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 -- 172.21.15.32:0/3612095121 learned_addr learned my addr 172.21.15.32:0/3612095121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fea800640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15dee0 0x7f2fec06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 -- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fec0d0d40 msgr2=0x7f2fec14b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fec0d0d40 0x7f2fec14b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 -- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15dee0 msgr2=0x7f2fec06ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15dee0 0x7f2fec06ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.315+0000 7f2fe3e00640 1 -- 172.21.15.32:0/3612095121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec1571b0 con 0x7f2fec0d4860 2024-03-06T07:41:01.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.316+0000 7f2fe3e00640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 0x7f2fec156c70 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c500 tx=0x7f2fd400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.316+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4018070 con 0x7f2fec0d4860 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.316+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd4007580 con 0x7f2fec0d4860 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.316+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4004a20 con 0x7f2fec0d4860 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.316+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec157470 con 0x7f2fec0d4860 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.316+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec157a30 con 0x7f2fec0d4860 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.317+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec04f330 con 0x7f2fec0d4860 2024-03-06T07:41:01.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.318+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd401b070 con 0x7f2fec0d4860 2024-03-06T07:41:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.320+0000 7f2fe2000640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.320+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2fd4089530 con 0x7f2fec0d4860 2024-03-06T07:41:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.320+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.321+0000 7f2fe9e00640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc806b4d0 0x7f2fc806d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc036000 tx=0x7f2fdc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:01.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.324+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd4058360 con 0x7f2fec0d4860 2024-03-06T07:41:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.522+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2fec08d7d0 con 0x7f2fec0d4860 2024-03-06T07:41:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.523+0000 7f2fe2000640 1 -- 172.21.15.32:0/3612095121 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2fd4057c60 con 0x7f2fec0d4860 2024-03-06T07:41:01.525 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-03-06T07:41:01.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc806b4d0 msgr2=0x7f2fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc806b4d0 0x7f2fc806d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc036000 tx=0x7f2fdc030000 comp rx=0 tx=0).stop 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 msgr2=0x7f2fec156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 0x7f2fec156c70 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c500 tx=0x7f2fd400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 shutdown_connections 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15dee0 0x7f2fec06ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d4860 0x7f2fec156c70 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 --2- 172.21.15.32:0/3612095121 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f2fec0d0d40 0x7f2fec14b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 >> 172.21.15.32:0/3612095121 conn(0x7f2fec0d3cd0 msgr2=0x7f2fec0cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 shutdown_connections 2024-03-06T07:41:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.526+0000 7f2feb200640 1 -- 172.21.15.32:0/3612095121 wait complete. 2024-03-06T07:41:01.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:41:01.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.761+0000 7f6b80a00640 1 Processor -- start 2024-03-06T07:41:01.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.761+0000 7f6b80a00640 1 -- start start 2024-03-06T07:41:01.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.763+0000 7f6b80a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 0x7f6b7c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.763+0000 7f6b80a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.763+0000 7f6b80a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c0cea50 con 0x7f6b7c161b40 2024-03-06T07:41:01.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.763+0000 7f6b80a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c0b5770 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.763+0000 7f6b80a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0cf900 con 0x7f6b7c0d2760 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.763+0000 7f6b73e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 0x7f6b7c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b7a800640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b73e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 0x7f6b7c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.32:60362/0 (socket says 172.21.15.32:60362) 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b73e00640 1 -- 172.21.15.32:0/285345945 learned_addr learned my addr 172.21.15.32:0/285345945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b7a800640 1 -- 172.21.15.32:0/285345945 >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 msgr2=0x7f6b7c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b7a800640 1 --2- 172.21.15.32:0/285345945 >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 0x7f6b7c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b7a800640 1 -- 172.21.15.32:0/285345945 >> v1:172.21.15.39:6789/0 conn(0x7f6b7c161b40 legacy=0x7f6b7c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b7a800640 1 -- 172.21.15.32:0/285345945 --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c0cecf0 con 0x7f6b7c0d2760 2024-03-06T07:41:01.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.764+0000 7f6b73e00640 1 --2- 172.21.15.32:0/285345945 >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 0x7f6b7c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:41:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.766+0000 7f6b7a800640 1 --2- 172.21.15.32:0/285345945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0b3740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009d90 tx=0x7f6b7402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10d6536d0aa5376e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.767+0000 7f6b79400640 1 -- 172.21.15.32:0/285345945 <== mon.2 v2:172.21.15.39:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74039070 con 0x7f6b7c0d2760 2024-03-06T07:41:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.767+0000 7f6b79400640 1 -- 172.21.15.32:0/285345945 <== mon.2 v2:172.21.15.39:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7402cc50 con 0x7f6b7c0d2760 2024-03-06T07:41:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.767+0000 7f6b79400640 1 -- 172.21.15.32:0/285345945 <== mon.2 v2:172.21.15.39:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74034440 con 0x7f6b7c0d2760 2024-03-06T07:41:01.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 -- 172.21.15.32:0/285345945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 msgr2=0x7f6b7c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 --2- 172.21.15.32:0/285345945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0b3740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009d90 tx=0x7f6b7402bea0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 -- 172.21.15.32:0/285345945 shutdown_connections 2024-03-06T07:41:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 --2- 172.21.15.32:0/285345945 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0b3740 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 --2- 172.21.15.32:0/285345945 >> v2:172.21.15.39:3300/0 conn(0x7f6b7c0cfd90 0x7f6b7c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 -- 172.21.15.32:0/285345945 >> 172.21.15.32:0/285345945 conn(0x7f6b7c09d220 msgr2=0x7f6b7c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 -- 172.21.15.32:0/285345945 shutdown_connections 2024-03-06T07:41:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.768+0000 7f6b80a00640 1 -- 172.21.15.32:0/285345945 wait complete. 2024-03-06T07:41:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.770+0000 7f6b80a00640 1 Processor -- start 2024-03-06T07:41:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.770+0000 7f6b80a00640 1 -- start start 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b80a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 0x7f6b7c0c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b80a00640 1 --2- >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b80a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b7c161b40 0x7f6b7c0c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b80a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0cf590 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b80a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0cf2b0 con 0x7f6b7c161b40 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b80a00640 1 -- --> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0b2ff0 con 0x7f6b7c0d2760 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b79e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 0x7f6b7c0c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b79e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 0x7f6b7c0c89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41108/0 (socket says 172.21.15.32:41108) 2024-03-06T07:41:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b79e00640 1 -- 172.21.15.32:0/2952127304 learned_addr learned my addr 172.21.15.32:0/2952127304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b73e00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.772+0000 7f6b7a800640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b7c161b40 0x7f6b7c0c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b79e00640 1 -- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 msgr2=0x7f6b7c0c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b79e00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b79e00640 1 -- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b7c161b40 msgr2=0x7f6b7c0c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b79e00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b7c161b40 0x7f6b7c0c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b79e00640 1 -- 172.21.15.32:0/2952127304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c0c7c10 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b79e00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 0x7f6b7c0c89d0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00c430 tx=0x7f6b6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0052d0 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c005490 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.773+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c0c7ef0 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.774+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c007260 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.774+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c0c9990 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.775+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c04f3a0 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.776+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b6c007400 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.777+0000 7f6b72000640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b5806b4d0 0x7f6b5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:41:01.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.777+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6b6c089800 con 0x7f6b7c0cfd90 2024-03-06T07:41:01.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.777+0000 7f6b73e00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b5806b4d0 0x7f6b5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:41:01.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.778+0000 7f6b73e00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b5806b4d0 0x7f6b5806d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0b9fa0 tx=0x7f6b64008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:41:01.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:01.783+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b6c0585a0 con 0x7f6b7c0cfd90 2024-03-06T07:41:02.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.000+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6b7c0d1110 con 0x7f6b7c0cfd90 2024-03-06T07:41:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.001+0000 7f6b72000640 1 -- 172.21.15.32:0/2952127304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v131) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6b6c057ea0 con 0x7f6b7c0cfd90 2024-03-06T07:41:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b5806b4d0 msgr2=0x7f6b5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b5806b4d0 0x7f6b5806d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0b9fa0 tx=0x7f6b64008040 comp rx=0 tx=0).stop 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 msgr2=0x7f6b7c0c89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 0x7f6b7c0c89d0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00c430 tx=0x7f6b6c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 shutdown_connections 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:6832/1434773126,v1:172.21.15.32:6833/1434773126] conn(0x7f6b5806b4d0 0x7f6b5806d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b7c161b40 0x7f6b7c0c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] conn(0x7f6b7c0d2760 0x7f6b7c0c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 --2- 172.21.15.32:0/2952127304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7c0cfd90 0x7f6b7c0c89d0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:41:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 >> 172.21.15.32:0/2952127304 conn(0x7f6b7c09d220 msgr2=0x7f6b7c06bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:41:02.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.003+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 shutdown_connections 2024-03-06T07:41:02.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-06T07:41:02.004+0000 7f6b80a00640 1 -- 172.21.15.32:0/2952127304 wait complete. 2024-03-06T07:41:02.023 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:41:02.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:41:02.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:41:21.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:21.961 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:52.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:52.131 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:22.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:22.274 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:52.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:52.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:22.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:22.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:52.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:52.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:23.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:23.059 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:53.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:53.206 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:23.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:23.378 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:53.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:53.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:02.029 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-03-06T07:46:02.032 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-06T07:46:23.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:23.664 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:53.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:53.841 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:23.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:23.990 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:54.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:54.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:24.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:24.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:54.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:54.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:24.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:24.627 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:54.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:54.777 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:24.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:24.931 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:55.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:55.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:25.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:25.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:55.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:55.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:25.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:25.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:55.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:55.694 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:25.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:25.873 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:56.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:56.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:26.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:26.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:56.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:56.366 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:26.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:26.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:56.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:56.665 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:26.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:26.828 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:56.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:56.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:27.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:27.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:57.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:57.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:27.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:27.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:57.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:57.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:27.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:27.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:57.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:57.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:28.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:28.096 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:58.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:58.287 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:28.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:28.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:58.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:58.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:28.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:28.767 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:58.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:58.906 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:29.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:29.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:59.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:59.192 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:29.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:29.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:59.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:59.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:29.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:29.641 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:59.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:59.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:29.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:29.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:00.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:00.090 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:30.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:30.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:00.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:00.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:30.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:30.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:00.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:00.677 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:30.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:30.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:00.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:00.962 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:31.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:31.127 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:01.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:01.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:31.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:31.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:01.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:01.601 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:31.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:31.748 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:01.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:01.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:32.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:32.044 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:02.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:02.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:32.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:32.348 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:02.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:02.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:32.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:32.664 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:02.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:02.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:32.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:32.969 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:03.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:03.114 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:33.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:33.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:03.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:03.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:33.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:33.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:03.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:03.745 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:33.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:33.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:04.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:04.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:34.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:34.208 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:04.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:04.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:34.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:34.552 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:04.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:04.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:34.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:34.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:04.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:04.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:35.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:35.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:05.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:05.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:35.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:35.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:05.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:05.575 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:35.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:35.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:05.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:05.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:36.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:36.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:06.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:06.167 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:36.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:36.312 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:06.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:06.463 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:36.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:36.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:06.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:06.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:36.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:36.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:07.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:07.062 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:37.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:37.226 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:07.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:07.371 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:37.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:37.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:07.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:07.655 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:37.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:37.804 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:07.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:07.951 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:38.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:38.115 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:08.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:08.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:38.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:38.435 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:08.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:08.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:38.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:38.726 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:08.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:08.875 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:39.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:39.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:09.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:09.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:39.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:39.327 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:09.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:09.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:39.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:39.618 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:09.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:09.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:39.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:39.917 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:10.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:10.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:40.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:40.239 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:41.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:41.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:41.387446+0000) 2024-03-06T08:40:42.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:42.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:42.387729+0000) 2024-03-06T08:40:43.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:43.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:43.387961+0000) 2024-03-06T08:40:44.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:44.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:44.388185+0000) 2024-03-06T08:40:45.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:45.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:45.388431+0000) 2024-03-06T08:40:46.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:46.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:46.388732+0000) 2024-03-06T08:40:47.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:47.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:47.388971+0000) 2024-03-06T08:40:48.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:48.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:48.389228+0000) 2024-03-06T08:40:49.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:49.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:49.389487+0000) 2024-03-06T08:40:50.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:50.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:50.389744+0000) 2024-03-06T08:40:51.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:51.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:51.389983+0000) 2024-03-06T08:40:52.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:52.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:52.390216+0000) 2024-03-06T08:40:53.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:53.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:53.390444+0000) 2024-03-06T08:40:54.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:54.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:54.390734+0000) 2024-03-06T08:40:55.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:55.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:55.390984+0000) 2024-03-06T08:40:56.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:56.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:56.391213+0000) 2024-03-06T08:40:57.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:57.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:57.391483+0000) 2024-03-06T08:40:58.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:58.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:58.391796+0000) 2024-03-06T08:40:59.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:40:59.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:59.392081+0000) 2024-03-06T08:41:00.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:00.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:00.392369+0000) 2024-03-06T08:41:01.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:01.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:01.392726+0000) 2024-03-06T08:41:02.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:02.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:02.393016+0000) 2024-03-06T08:41:03.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:03.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:03.393316+0000) 2024-03-06T08:41:04.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:04.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:04.393661+0000) 2024-03-06T08:41:05.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:05.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:05.394006+0000) 2024-03-06T08:41:06.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:06.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:06.394322+0000) 2024-03-06T08:41:07.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:07.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:07.394603+0000) 2024-03-06T08:41:08.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:08.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:08.394893+0000) 2024-03-06T08:41:09.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:09.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:09.395164+0000) 2024-03-06T08:41:10.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:10.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:10.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:10.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:10.395442+0000) 2024-03-06T08:41:11.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:11.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:11.395783+0000) 2024-03-06T08:41:12.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:12.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:12.396092+0000) 2024-03-06T08:41:13.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:13.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:13.396387+0000) 2024-03-06T08:41:14.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:14.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:14.396714+0000) 2024-03-06T08:41:15.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:15.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:15.397047+0000) 2024-03-06T08:41:16.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:16.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:16.397326+0000) 2024-03-06T08:41:17.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:17.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:17.397674+0000) 2024-03-06T08:41:18.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:18.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:18.397990+0000) 2024-03-06T08:41:19.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:19.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:19.398274+0000) 2024-03-06T08:41:20.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:20.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:20.398569+0000) 2024-03-06T08:41:21.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:21.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:21.398836+0000) 2024-03-06T08:41:22.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:22.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:22.399155+0000) 2024-03-06T08:41:23.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:23.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:23.399436+0000) 2024-03-06T08:41:24.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:24.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:24.399749+0000) 2024-03-06T08:41:25.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:25.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:25.400087+0000) 2024-03-06T08:41:26.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:26.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:26.400382+0000) 2024-03-06T08:41:27.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:27.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:27.400706+0000) 2024-03-06T08:41:28.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:28.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:28.401022+0000) 2024-03-06T08:41:29.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:29.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:29.401309+0000) 2024-03-06T08:41:30.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:30.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:30.401601+0000) 2024-03-06T08:41:31.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:31.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:31.401898+0000) 2024-03-06T08:41:32.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:32.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:32.402182+0000) 2024-03-06T08:41:33.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:33.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:33.402453+0000) 2024-03-06T08:41:34.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:34.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:34.402750+0000) 2024-03-06T08:41:35.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:35.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:35.403063+0000) 2024-03-06T08:41:36.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:36.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:36.403357+0000) 2024-03-06T08:41:37.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:37.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:37.403704+0000) 2024-03-06T08:41:38.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:38.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:38.403958+0000) 2024-03-06T08:41:39.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:39.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:39.404240+0000) 2024-03-06T08:41:40.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:40.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:40.404523+0000) 2024-03-06T08:41:40.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:40.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:41.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:41.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:41.404838+0000) 2024-03-06T08:41:42.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:42.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:42.405110+0000) 2024-03-06T08:41:43.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:43.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:43.405357+0000) 2024-03-06T08:41:44.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:44.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:44.405672+0000) 2024-03-06T08:41:45.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:45.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:45.405993+0000) 2024-03-06T08:41:46.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:46.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:46.406304+0000) 2024-03-06T08:41:47.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:47.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:47.406592+0000) 2024-03-06T08:41:48.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:48.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:48.406950+0000) 2024-03-06T08:41:49.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:49.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:49.407236+0000) 2024-03-06T08:41:50.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:50.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:50.407548+0000) 2024-03-06T08:41:51.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:51.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:51.407851+0000) 2024-03-06T08:41:52.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:52.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:52.408158+0000) 2024-03-06T08:41:53.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:53.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:53.408461+0000) 2024-03-06T08:41:54.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:54.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:54.408772+0000) 2024-03-06T08:41:55.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:55.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:55.409086+0000) 2024-03-06T08:41:56.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:56.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:56.409398+0000) 2024-03-06T08:41:57.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:57.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:57.409729+0000) 2024-03-06T08:41:58.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:58.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:58.410009+0000) 2024-03-06T08:41:59.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:41:59.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:59.410316+0000) 2024-03-06T08:42:00.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:00.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:00.410589+0000) 2024-03-06T08:42:01.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:01.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:01.410937+0000) 2024-03-06T08:42:02.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:02.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:02.411255+0000) 2024-03-06T08:42:03.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:03.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:03.411535+0000) 2024-03-06T08:42:04.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:04.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:04.411833+0000) 2024-03-06T08:42:05.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:05.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:05.412140+0000) 2024-03-06T08:42:06.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:06.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:06.412375+0000) 2024-03-06T08:42:07.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:07.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:07.412742+0000) 2024-03-06T08:42:08.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:08.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:08.413047+0000) 2024-03-06T08:42:09.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:09.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:09.413327+0000) 2024-03-06T08:42:10.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:10.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:10.413674+0000) 2024-03-06T08:42:10.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:10.671 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:11.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:11.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:11.413987+0000) 2024-03-06T08:42:12.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:12.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:12.414279+0000) 2024-03-06T08:42:13.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:13.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:13.414575+0000) 2024-03-06T08:42:14.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:14.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:14.414874+0000) 2024-03-06T08:42:15.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:15.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:15.415198+0000) 2024-03-06T08:42:16.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:16.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:16.415502+0000) 2024-03-06T08:42:17.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:17.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:17.415780+0000) 2024-03-06T08:42:18.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:18.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:18.416071+0000) 2024-03-06T08:42:19.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:19.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:19.416358+0000) 2024-03-06T08:42:20.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:20.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:20.416740+0000) 2024-03-06T08:42:21.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:21.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:21.417017+0000) 2024-03-06T08:42:22.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:22.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:22.417247+0000) 2024-03-06T08:42:23.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:23.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:23.417482+0000) 2024-03-06T08:42:24.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:24.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:24.417743+0000) 2024-03-06T08:42:25.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:25.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:25.418007+0000) 2024-03-06T08:42:26.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:26.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:26.418240+0000) 2024-03-06T08:42:27.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:27.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:27.418491+0000) 2024-03-06T08:42:28.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:28.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:28.418750+0000) 2024-03-06T08:42:29.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:29.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:29.418933+0000) 2024-03-06T08:42:30.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:30.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:30.419168+0000) 2024-03-06T08:42:31.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:31.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:31.419421+0000) 2024-03-06T08:42:32.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:32.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:32.419738+0000) 2024-03-06T08:42:33.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:33.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:33.419987+0000) 2024-03-06T08:42:34.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:34.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:34.420227+0000) 2024-03-06T08:42:35.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:35.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:35.420497+0000) 2024-03-06T08:42:36.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:36.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:36.420736+0000) 2024-03-06T08:42:37.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:37.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:37.420964+0000) 2024-03-06T08:42:38.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:38.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:38.421145+0000) 2024-03-06T08:42:39.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:39.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:39.421389+0000) 2024-03-06T08:42:40.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:40.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:40.421681+0000) 2024-03-06T08:42:40.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:40.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:41.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:41.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:41.421930+0000) 2024-03-06T08:42:42.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:42.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:42.422153+0000) 2024-03-06T08:42:43.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:43.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:43.422401+0000) 2024-03-06T08:42:44.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:44.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:44.422696+0000) 2024-03-06T08:42:45.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:45.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:45.422910+0000) 2024-03-06T08:42:46.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:46.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:46.423162+0000) 2024-03-06T08:42:47.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:47.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:47.423414+0000) 2024-03-06T08:42:48.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:48.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:48.423693+0000) 2024-03-06T08:42:49.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:49.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:49.423926+0000) 2024-03-06T08:42:50.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:50.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:50.424189+0000) 2024-03-06T08:42:51.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:51.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:51.424449+0000) 2024-03-06T08:42:52.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:52.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:52.424740+0000) 2024-03-06T08:42:53.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:53.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:53.424983+0000) 2024-03-06T08:42:54.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:54.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:54.425233+0000) 2024-03-06T08:42:55.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:55.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:55.425483+0000) 2024-03-06T08:42:56.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:56.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:56.425760+0000) 2024-03-06T08:42:57.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:57.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:57.425990+0000) 2024-03-06T08:42:58.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:58.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:58.426251+0000) 2024-03-06T08:42:59.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:42:59.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:59.426500+0000) 2024-03-06T08:43:00.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:00.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:00.426762+0000) 2024-03-06T08:43:01.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:01.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:01.427023+0000) 2024-03-06T08:43:02.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:02.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:02.427262+0000) 2024-03-06T08:43:03.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:03.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:03.427514+0000) 2024-03-06T08:43:04.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:04.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:04.427765+0000) 2024-03-06T08:43:05.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:05.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:05.428004+0000) 2024-03-06T08:43:06.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:06.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:06.428239+0000) 2024-03-06T08:43:07.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:07.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:07.428476+0000) 2024-03-06T08:43:08.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:08.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:08.428803+0000) 2024-03-06T08:43:09.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:09.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:09.429033+0000) 2024-03-06T08:43:10.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:10.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:10.429215+0000) 2024-03-06T08:43:10.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:10.966 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:11.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:11.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:11.429447+0000) 2024-03-06T08:43:12.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:12.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:12.429796+0000) 2024-03-06T08:43:13.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:13.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:13.430050+0000) 2024-03-06T08:43:14.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:14.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:14.430299+0000) 2024-03-06T08:43:15.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:15.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:15.430562+0000) 2024-03-06T08:43:16.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:16.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:16.430799+0000) 2024-03-06T08:43:17.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:17.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:17.431054+0000) 2024-03-06T08:43:18.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:18.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:18.431238+0000) 2024-03-06T08:43:19.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:19.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:19.431418+0000) 2024-03-06T08:43:20.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:20.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:20.431744+0000) 2024-03-06T08:43:21.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:21.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:21.431986+0000) 2024-03-06T08:43:22.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:22.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:22.432237+0000) 2024-03-06T08:43:23.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:23.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:23.432425+0000) 2024-03-06T08:43:24.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:24.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:24.432813+0000) 2024-03-06T08:43:25.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:25.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:25.433118+0000) 2024-03-06T08:43:26.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:26.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:26.433349+0000) 2024-03-06T08:43:27.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:27.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:27.433588+0000) 2024-03-06T08:43:28.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:28.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:28.433844+0000) 2024-03-06T08:43:29.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:29.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:29.434110+0000) 2024-03-06T08:43:30.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:30.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:30.434352+0000) 2024-03-06T08:43:31.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:31.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:31.434592+0000) 2024-03-06T08:43:32.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:32.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:32.434832+0000) 2024-03-06T08:43:33.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:33.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:33.435093+0000) 2024-03-06T08:43:34.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:34.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:34.435348+0000) 2024-03-06T08:43:35.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:35.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:35.435600+0000) 2024-03-06T08:43:36.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:36.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:36.435897+0000) 2024-03-06T08:43:37.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:37.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:37.436125+0000) 2024-03-06T08:43:38.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:38.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:38.436379+0000) 2024-03-06T08:43:39.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:39.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:39.436618+0000) 2024-03-06T08:43:40.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:40.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:40.436938+0000) 2024-03-06T08:43:41.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:41.096 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:41.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:41.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:41.437198+0000) 2024-03-06T08:43:42.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:42.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:42.437455+0000) 2024-03-06T08:43:43.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:43.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:43.437749+0000) 2024-03-06T08:43:44.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:44.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:44.438003+0000) 2024-03-06T08:43:45.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:45.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:45.438263+0000) 2024-03-06T08:43:46.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:46.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:46.438493+0000) 2024-03-06T08:43:47.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:47.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:47.438753+0000) 2024-03-06T08:43:48.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:48.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:48.438985+0000) 2024-03-06T08:43:49.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:49.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:49.439237+0000) 2024-03-06T08:43:50.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:50.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:50.439487+0000) 2024-03-06T08:43:51.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:51.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:51.439748+0000) 2024-03-06T08:43:52.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:52.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:52.439979+0000) 2024-03-06T08:43:53.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:53.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:53.440245+0000) 2024-03-06T08:43:54.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:54.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:54.440479+0000) 2024-03-06T08:43:55.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:55.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:55.440747+0000) 2024-03-06T08:43:56.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:56.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:56.440992+0000) 2024-03-06T08:43:57.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:57.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:57.441247+0000) 2024-03-06T08:43:58.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:58.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:58.441488+0000) 2024-03-06T08:43:59.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:43:59.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:59.441746+0000) 2024-03-06T08:44:00.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:00.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:00.441976+0000) 2024-03-06T08:44:01.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:01.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:01.442216+0000) 2024-03-06T08:44:02.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:02.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:02.442454+0000) 2024-03-06T08:44:03.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:03.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:03.442767+0000) 2024-03-06T08:44:04.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:04.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:04.443030+0000) 2024-03-06T08:44:05.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:05.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:05.443278+0000) 2024-03-06T08:44:06.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:06.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:06.443564+0000) 2024-03-06T08:44:07.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:07.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:07.443801+0000) 2024-03-06T08:44:08.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:08.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:08.443988+0000) 2024-03-06T08:44:09.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:09.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:09.444181+0000) 2024-03-06T08:44:10.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:10.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:10.444362+0000) 2024-03-06T08:44:11.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:11.243 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:11.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:11.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:11.444560+0000) 2024-03-06T08:44:12.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:12.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:12.444797+0000) 2024-03-06T08:44:13.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:13.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:13.445054+0000) 2024-03-06T08:44:14.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:14.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:14.445304+0000) 2024-03-06T08:44:15.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:15.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:15.445579+0000) 2024-03-06T08:44:16.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:16.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:16.445832+0000) 2024-03-06T08:44:17.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:17.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:17.446096+0000) 2024-03-06T08:44:18.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:18.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:18.446333+0000) 2024-03-06T08:44:19.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:19.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:19.446587+0000) 2024-03-06T08:44:20.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:20.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:20.446823+0000) 2024-03-06T08:44:21.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:21.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:21.447090+0000) 2024-03-06T08:44:22.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:22.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:22.447351+0000) 2024-03-06T08:44:23.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:23.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:23.447608+0000) 2024-03-06T08:44:24.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:24.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:24.447913+0000) 2024-03-06T08:44:25.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:25.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:25.448184+0000) 2024-03-06T08:44:26.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:26.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:26.448439+0000) 2024-03-06T08:44:27.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:27.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:27.448708+0000) 2024-03-06T08:44:28.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:28.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:28.448962+0000) 2024-03-06T08:44:29.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:29.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:29.449218+0000) 2024-03-06T08:44:30.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:30.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:30.449467+0000) 2024-03-06T08:44:31.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:31.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:31.449759+0000) 2024-03-06T08:44:32.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:32.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:32.449935+0000) 2024-03-06T08:44:33.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:33.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:33.450205+0000) 2024-03-06T08:44:34.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:34.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:34.450434+0000) 2024-03-06T08:44:35.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:35.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:35.450746+0000) 2024-03-06T08:44:36.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:36.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:36.450991+0000) 2024-03-06T08:44:37.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:37.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:37.451185+0000) 2024-03-06T08:44:38.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:38.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:38.451437+0000) 2024-03-06T08:44:39.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:39.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:39.451743+0000) 2024-03-06T08:44:40.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:40.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:40.451929+0000) 2024-03-06T08:44:41.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:41.389 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:41.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:41.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:41.452124+0000) 2024-03-06T08:44:42.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:42.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:42.452376+0000) 2024-03-06T08:44:43.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:43.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:43.452605+0000) 2024-03-06T08:44:44.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:44.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:44.452884+0000) 2024-03-06T08:44:45.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:45.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:45.453148+0000) 2024-03-06T08:44:46.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:46.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:46.453380+0000) 2024-03-06T08:44:47.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:47.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:47.453680+0000) 2024-03-06T08:44:48.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:48.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:48.453944+0000) 2024-03-06T08:44:49.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:49.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:49.454184+0000) 2024-03-06T08:44:50.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:50.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:50.454431+0000) 2024-03-06T08:44:51.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:51.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:51.454749+0000) 2024-03-06T08:44:52.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:52.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:52.454982+0000) 2024-03-06T08:44:53.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:53.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:53.455237+0000) 2024-03-06T08:44:54.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:54.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:54.455521+0000) 2024-03-06T08:44:55.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:55.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:55.455793+0000) 2024-03-06T08:44:56.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:56.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:56.456053+0000) 2024-03-06T08:44:57.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:57.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:57.456310+0000) 2024-03-06T08:44:58.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:58.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:58.456541+0000) 2024-03-06T08:44:59.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:44:59.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:59.456797+0000) 2024-03-06T08:45:00.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:00.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:00.457028+0000) 2024-03-06T08:45:01.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:01.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:01.457290+0000) 2024-03-06T08:45:02.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:02.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:02.457540+0000) 2024-03-06T08:45:03.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:03.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:03.457778+0000) 2024-03-06T08:45:04.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:04.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:04.458009+0000) 2024-03-06T08:45:05.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:05.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:05.458269+0000) 2024-03-06T08:45:06.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:06.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:06.458503+0000) 2024-03-06T08:45:07.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:07.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:07.458757+0000) 2024-03-06T08:45:08.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:08.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:08.459011+0000) 2024-03-06T08:45:09.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:09.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:09.459250+0000) 2024-03-06T08:45:10.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:10.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:10.459506+0000) 2024-03-06T08:45:11.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:11.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:11.459771+0000) 2024-03-06T08:45:11.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:11.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:12.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:12.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:12.459967+0000) 2024-03-06T08:45:13.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:13.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:13.460229+0000) 2024-03-06T08:45:14.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:14.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:14.460482+0000) 2024-03-06T08:45:15.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:15.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:15.460753+0000) 2024-03-06T08:45:16.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:16.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:16.461005+0000) 2024-03-06T08:45:17.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:17.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:17.461253+0000) 2024-03-06T08:45:18.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:18.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:18.461502+0000) 2024-03-06T08:45:19.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:19.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:19.461768+0000) 2024-03-06T08:45:20.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:20.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:20.462025+0000) 2024-03-06T08:45:21.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:21.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:21.462289+0000) 2024-03-06T08:45:22.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:22.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:22.462541+0000) 2024-03-06T08:45:23.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:23.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:23.462782+0000) 2024-03-06T08:45:24.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:24.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:24.463026+0000) 2024-03-06T08:45:25.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:25.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:25.463283+0000) 2024-03-06T08:45:26.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:26.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:26.463543+0000) 2024-03-06T08:45:27.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:27.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:27.463804+0000) 2024-03-06T08:45:28.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:28.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:28.464045+0000) 2024-03-06T08:45:29.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:29.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:29.464281+0000) 2024-03-06T08:45:30.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:30.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:30.464535+0000) 2024-03-06T08:45:31.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:31.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:31.464804+0000) 2024-03-06T08:45:32.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:32.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:32.465069+0000) 2024-03-06T08:45:33.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:33.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:33.465311+0000) 2024-03-06T08:45:34.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:34.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:34.465550+0000) 2024-03-06T08:45:35.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:35.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:35.465815+0000) 2024-03-06T08:45:36.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:36.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:36.466065+0000) 2024-03-06T08:45:37.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:37.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:37.466333+0000) 2024-03-06T08:45:38.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:38.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:38.466575+0000) 2024-03-06T08:45:39.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:39.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:39.466796+0000) 2024-03-06T08:45:40.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:40.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:40.467077+0000) 2024-03-06T08:45:41.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:41.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:41.467361+0000) 2024-03-06T08:45:41.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:41.700 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:42.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:42.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:42.467616+0000) 2024-03-06T08:45:43.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:43.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:43.467881+0000) 2024-03-06T08:45:44.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:44.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:44.468141+0000) 2024-03-06T08:45:45.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:45.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:45.468426+0000) 2024-03-06T08:45:46.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:46.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:46.468804+0000) 2024-03-06T08:45:47.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:47.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:47.469058+0000) 2024-03-06T08:45:48.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:48.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:48.469273+0000) 2024-03-06T08:45:49.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:49.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:49.469524+0000) 2024-03-06T08:45:50.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:50.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:50.469803+0000) 2024-03-06T08:45:51.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:51.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:51.470078+0000) 2024-03-06T08:45:52.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:52.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:52.470346+0000) 2024-03-06T08:45:53.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:53.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:53.470709+0000) 2024-03-06T08:45:54.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:54.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:54.470981+0000) 2024-03-06T08:45:55.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:55.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:55.471224+0000) 2024-03-06T08:45:56.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:56.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:56.471461+0000) 2024-03-06T08:45:57.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:57.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:57.471675+0000) 2024-03-06T08:45:58.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:58.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:58.471936+0000) 2024-03-06T08:45:59.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:45:59.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:59.472173+0000) 2024-03-06T08:46:00.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:00.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:00.472426+0000) 2024-03-06T08:46:01.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:01.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:01.472725+0000) 2024-03-06T08:46:02.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:02.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:02.472987+0000) 2024-03-06T08:46:03.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:03.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:03.473264+0000) 2024-03-06T08:46:04.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:04.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:04.473522+0000) 2024-03-06T08:46:05.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:05.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:05.473810+0000) 2024-03-06T08:46:06.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:06.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:06.474079+0000) 2024-03-06T08:46:07.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:07.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:07.474298+0000) 2024-03-06T08:46:08.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:08.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:08.474568+0000) 2024-03-06T08:46:09.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:09.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:09.474843+0000) 2024-03-06T08:46:10.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:10.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:10.475109+0000) 2024-03-06T08:46:11.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:11.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:11.475387+0000) 2024-03-06T08:46:11.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:11.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:12.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:12.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:12.475701+0000) 2024-03-06T08:46:13.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:13.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:13.475968+0000) 2024-03-06T08:46:14.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:14.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:14.476250+0000) 2024-03-06T08:46:15.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:15.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:15.476466+0000) 2024-03-06T08:46:16.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:16.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:16.476742+0000) 2024-03-06T08:46:17.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:17.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:17.477042+0000) 2024-03-06T08:46:18.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:18.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:18.477323+0000) 2024-03-06T08:46:19.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:19.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:19.477673+0000) 2024-03-06T08:46:20.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:20.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:20.477987+0000) 2024-03-06T08:46:21.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:21.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:21.478302+0000) 2024-03-06T08:46:22.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:22.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:22.478617+0000) 2024-03-06T08:46:23.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:23.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:23.478965+0000) 2024-03-06T08:46:24.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:24.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:24.479244+0000) 2024-03-06T08:46:25.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:25.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:25.479542+0000) 2024-03-06T08:46:26.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:26.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:26.479865+0000) 2024-03-06T08:46:27.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:27.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:27.480122+0000) 2024-03-06T08:46:28.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:28.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:28.480358+0000) 2024-03-06T08:46:29.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:29.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:29.480608+0000) 2024-03-06T08:46:30.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:30.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:30.480892+0000) 2024-03-06T08:46:31.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:31.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:31.481159+0000) 2024-03-06T08:46:32.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:32.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:32.481392+0000) 2024-03-06T08:46:33.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:33.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:33.481662+0000) 2024-03-06T08:46:34.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:34.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:34.481907+0000) 2024-03-06T08:46:35.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:35.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:35.482174+0000) 2024-03-06T08:46:36.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:36.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:36.482435+0000) 2024-03-06T08:46:37.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:37.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:37.482851+0000) 2024-03-06T08:46:38.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:38.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:38.483127+0000) 2024-03-06T08:46:39.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:39.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:39.483382+0000) 2024-03-06T08:46:40.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:40.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:40.483581+0000) 2024-03-06T08:46:41.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:41.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:41.483824+0000) 2024-03-06T08:46:41.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:41.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:42.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:42.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:42.484068+0000) 2024-03-06T08:46:43.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:43.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:43.484254+0000) 2024-03-06T08:46:44.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:44.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:44.484490+0000) 2024-03-06T08:46:45.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:45.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:45.484752+0000) 2024-03-06T08:46:46.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:46.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:46.485006+0000) 2024-03-06T08:46:47.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:47.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:47.485262+0000) 2024-03-06T08:46:48.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:48.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:48.485494+0000) 2024-03-06T08:46:49.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:49.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:49.485750+0000) 2024-03-06T08:46:50.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:50.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:50.486002+0000) 2024-03-06T08:46:51.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:51.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:51.486267+0000) 2024-03-06T08:46:52.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:52.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:52.486525+0000) 2024-03-06T08:46:53.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:53.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:53.486812+0000) 2024-03-06T08:46:54.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:54.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:54.487089+0000) 2024-03-06T08:46:55.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:55.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:55.487364+0000) 2024-03-06T08:46:56.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:56.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:56.487619+0000) 2024-03-06T08:46:57.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:57.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:57.487916+0000) 2024-03-06T08:46:58.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:58.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:58.488174+0000) 2024-03-06T08:46:59.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:46:59.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:59.488438+0000) 2024-03-06T08:47:00.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:00.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:00.488734+0000) 2024-03-06T08:47:01.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:01.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:01.489014+0000) 2024-03-06T08:47:02.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:02.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:02.489286+0000) 2024-03-06T08:47:03.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:03.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:03.489553+0000) 2024-03-06T08:47:04.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:04.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:04.489832+0000) 2024-03-06T08:47:05.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:05.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:05.490101+0000) 2024-03-06T08:47:06.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:06.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:06.490374+0000) 2024-03-06T08:47:07.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:07.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:07.490676+0000) 2024-03-06T08:47:08.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:08.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:08.490929+0000) 2024-03-06T08:47:09.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:09.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:09.491206+0000) 2024-03-06T08:47:10.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:10.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:10.491477+0000) 2024-03-06T08:47:11.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:11.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:11.491760+0000) 2024-03-06T08:47:12.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:12.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:12.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:12.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:12.491968+0000) 2024-03-06T08:47:13.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:13.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:13.492172+0000) 2024-03-06T08:47:14.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:14.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:14.492367+0000) 2024-03-06T08:47:15.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:15.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:15.492657+0000) 2024-03-06T08:47:16.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:16.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:16.492910+0000) 2024-03-06T08:47:17.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:17.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:17.493185+0000) 2024-03-06T08:47:18.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:18.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:18.493456+0000) 2024-03-06T08:47:19.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:19.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:19.493741+0000) 2024-03-06T08:47:20.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:20.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:20.493990+0000) 2024-03-06T08:47:21.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:21.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:21.494254+0000) 2024-03-06T08:47:22.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:22.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:22.494509+0000) 2024-03-06T08:47:23.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:23.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:23.494763+0000) 2024-03-06T08:47:24.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:24.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:24.495015+0000) 2024-03-06T08:47:25.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:25.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:25.495287+0000) 2024-03-06T08:47:26.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:26.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:26.495542+0000) 2024-03-06T08:47:27.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:27.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:27.495811+0000) 2024-03-06T08:47:28.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:28.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:28.496063+0000) 2024-03-06T08:47:29.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:29.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:29.496321+0000) 2024-03-06T08:47:30.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:30.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:30.496586+0000) 2024-03-06T08:47:31.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:31.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:31.496893+0000) 2024-03-06T08:47:32.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:32.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:32.497147+0000) 2024-03-06T08:47:33.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:33.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:33.497421+0000) 2024-03-06T08:47:34.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:34.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:34.497618+0000) 2024-03-06T08:47:35.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:35.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:35.497962+0000) 2024-03-06T08:47:36.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:36.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:36.498238+0000) 2024-03-06T08:47:37.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:37.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:37.498514+0000) 2024-03-06T08:47:38.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:38.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:38.498771+0000) 2024-03-06T08:47:39.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:39.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:39.499034+0000) 2024-03-06T08:47:40.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:40.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:40.499268+0000) 2024-03-06T08:47:41.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:41.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:41.499527+0000) 2024-03-06T08:47:42.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:42.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:42.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:42.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:42.499791+0000) 2024-03-06T08:47:43.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:43.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:43.500045+0000) 2024-03-06T08:47:44.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:44.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:44.500322+0000) 2024-03-06T08:47:45.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:45.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:45.500707+0000) 2024-03-06T08:47:46.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:46.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:46.500954+0000) 2024-03-06T08:47:47.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:47.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:47.501232+0000) 2024-03-06T08:47:48.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:48.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:48.501499+0000) 2024-03-06T08:47:49.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:49.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:49.501781+0000) 2024-03-06T08:47:50.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:50.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:50.502047+0000) 2024-03-06T08:47:51.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:51.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:51.502307+0000) 2024-03-06T08:47:52.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:52.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:52.502561+0000) 2024-03-06T08:47:53.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:53.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:53.502802+0000) 2024-03-06T08:47:54.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:54.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:54.503047+0000) 2024-03-06T08:47:55.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:55.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:55.503245+0000) 2024-03-06T08:47:56.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:56.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:56.503498+0000) 2024-03-06T08:47:57.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:57.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:57.503815+0000) 2024-03-06T08:47:58.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:58.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:58.504064+0000) 2024-03-06T08:47:59.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:47:59.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:59.504307+0000) 2024-03-06T08:48:00.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:00.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:00.504558+0000) 2024-03-06T08:48:01.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:01.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:01.504772+0000) 2024-03-06T08:48:02.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:02.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:02.505006+0000) 2024-03-06T08:48:03.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:03.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:03.505256+0000) 2024-03-06T08:48:04.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:04.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:04.505525+0000) 2024-03-06T08:48:05.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:05.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:05.505807+0000) 2024-03-06T08:48:06.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:06.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:06.506082+0000) 2024-03-06T08:48:07.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:07.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:07.506280+0000) 2024-03-06T08:48:08.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:08.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:08.506512+0000) 2024-03-06T08:48:09.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:09.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:09.506792+0000) 2024-03-06T08:48:10.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:10.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:10.507058+0000) 2024-03-06T08:48:11.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:11.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:11.507292+0000) 2024-03-06T08:48:12.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:12.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:12.507571+0000) 2024-03-06T08:48:12.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:12.543 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:13.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:13.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:13.507829+0000) 2024-03-06T08:48:14.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:14.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:14.508070+0000) 2024-03-06T08:48:15.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:15.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:15.508333+0000) 2024-03-06T08:48:16.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:16.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:16.508606+0000) 2024-03-06T08:48:17.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:17.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:17.508945+0000) 2024-03-06T08:48:18.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:18.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:18.509227+0000) 2024-03-06T08:48:19.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:19.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:19.509504+0000) 2024-03-06T08:48:20.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:20.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:20.509777+0000) 2024-03-06T08:48:21.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:21.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:21.510032+0000) 2024-03-06T08:48:22.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:22.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:22.510289+0000) 2024-03-06T08:48:23.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:23.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:23.510565+0000) 2024-03-06T08:48:24.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:24.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:24.510822+0000) 2024-03-06T08:48:25.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:25.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:25.511114+0000) 2024-03-06T08:48:26.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:26.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:26.511382+0000) 2024-03-06T08:48:27.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:27.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:27.511693+0000) 2024-03-06T08:48:28.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:28.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:28.511969+0000) 2024-03-06T08:48:29.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:29.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:29.512243+0000) 2024-03-06T08:48:30.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:30.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:30.512453+0000) 2024-03-06T08:48:31.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:31.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:31.512748+0000) 2024-03-06T08:48:32.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:32.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:32.513019+0000) 2024-03-06T08:48:33.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:33.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:33.513292+0000) 2024-03-06T08:48:34.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:34.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:34.513559+0000) 2024-03-06T08:48:35.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:35.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:35.513823+0000) 2024-03-06T08:48:36.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:36.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:36.514096+0000) 2024-03-06T08:48:37.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:37.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:37.514377+0000) 2024-03-06T08:48:38.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:38.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:38.514675+0000) 2024-03-06T08:48:39.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:39.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:39.514932+0000) 2024-03-06T08:48:40.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:40.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:40.515209+0000) 2024-03-06T08:48:41.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:41.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:41.515490+0000) 2024-03-06T08:48:42.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:42.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:42.515746+0000) 2024-03-06T08:48:42.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:42.691 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:43.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:43.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:43.516014+0000) 2024-03-06T08:48:44.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:44.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:44.516327+0000) 2024-03-06T08:48:45.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:45.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:45.516611+0000) 2024-03-06T08:48:46.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:46.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:46.516905+0000) 2024-03-06T08:48:47.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:47.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:47.517167+0000) 2024-03-06T08:48:48.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:48.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:48.517416+0000) 2024-03-06T08:48:49.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:49.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:49.517714+0000) 2024-03-06T08:48:50.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:50.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:50.517967+0000) 2024-03-06T08:48:51.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:51.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:51.518240+0000) 2024-03-06T08:48:52.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:52.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:52.518529+0000) 2024-03-06T08:48:53.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:53.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:53.518807+0000) 2024-03-06T08:48:54.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:54.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:54.519079+0000) 2024-03-06T08:48:55.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:55.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:55.519371+0000) 2024-03-06T08:48:56.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:56.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:56.519674+0000) 2024-03-06T08:48:57.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:57.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:57.519965+0000) 2024-03-06T08:48:58.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:58.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:58.520216+0000) 2024-03-06T08:48:59.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:48:59.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:59.520497+0000) 2024-03-06T08:49:00.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:00.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:00.520757+0000) 2024-03-06T08:49:01.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:01.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:01.521022+0000) 2024-03-06T08:49:02.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:02.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:02.521263+0000) 2024-03-06T08:49:03.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:03.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:03.521522+0000) 2024-03-06T08:49:04.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:04.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:04.521812+0000) 2024-03-06T08:49:05.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:05.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:05.522086+0000) 2024-03-06T08:49:06.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:06.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:06.522330+0000) 2024-03-06T08:49:07.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:07.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:07.522595+0000) 2024-03-06T08:49:08.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:08.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:08.522861+0000) 2024-03-06T08:49:09.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:09.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:09.523125+0000) 2024-03-06T08:49:10.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:10.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:10.523364+0000) 2024-03-06T08:49:11.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:11.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:11.523620+0000) 2024-03-06T08:49:12.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:12.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:12.523851+0000) 2024-03-06T08:49:12.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:12.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:13.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:13.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:13.524111+0000) 2024-03-06T08:49:14.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:14.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:14.524372+0000) 2024-03-06T08:49:15.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:15.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:15.524666+0000) 2024-03-06T08:49:16.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:16.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:16.524910+0000) 2024-03-06T08:49:17.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:17.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:17.525159+0000) 2024-03-06T08:49:18.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:18.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:18.525404+0000) 2024-03-06T08:49:19.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:19.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:19.525688+0000) 2024-03-06T08:49:20.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:20.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:20.525940+0000) 2024-03-06T08:49:21.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:21.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:21.526185+0000) 2024-03-06T08:49:22.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:22.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:22.526424+0000) 2024-03-06T08:49:23.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:23.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:23.526701+0000) 2024-03-06T08:49:24.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:24.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:24.526955+0000) 2024-03-06T08:49:25.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:25.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:25.527223+0000) 2024-03-06T08:49:26.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:26.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:26.527463+0000) 2024-03-06T08:49:27.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:27.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:27.527767+0000) 2024-03-06T08:49:28.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:28.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:28.528004+0000) 2024-03-06T08:49:29.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:29.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:29.528264+0000) 2024-03-06T08:49:30.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:30.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:30.528506+0000) 2024-03-06T08:49:31.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:31.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:31.528752+0000) 2024-03-06T08:49:32.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:32.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:32.528989+0000) 2024-03-06T08:49:33.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:33.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:33.529251+0000) 2024-03-06T08:49:34.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:34.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:34.529484+0000) 2024-03-06T08:49:35.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:35.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:35.529742+0000) 2024-03-06T08:49:36.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:36.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:36.529933+0000) 2024-03-06T08:49:37.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:37.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:37.530205+0000) 2024-03-06T08:49:38.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:38.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:38.530462+0000) 2024-03-06T08:49:39.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:39.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:39.530747+0000) 2024-03-06T08:49:40.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:40.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:40.531000+0000) 2024-03-06T08:49:41.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:41.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:41.531262+0000) 2024-03-06T08:49:42.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:42.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:42.531580+0000) 2024-03-06T08:49:42.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:42.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:43.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:43.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:43.531834+0000) 2024-03-06T08:49:44.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:44.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:44.532069+0000) 2024-03-06T08:49:45.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:45.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:45.532321+0000) 2024-03-06T08:49:46.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:46.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:46.532562+0000) 2024-03-06T08:49:47.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:47.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:47.532827+0000) 2024-03-06T08:49:48.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:48.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:48.533078+0000) 2024-03-06T08:49:49.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:49.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:49.533325+0000) 2024-03-06T08:49:50.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:50.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:50.533585+0000) 2024-03-06T08:49:51.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:51.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:51.533846+0000) 2024-03-06T08:49:52.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:52.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:52.534116+0000) 2024-03-06T08:49:53.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:53.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:53.534381+0000) 2024-03-06T08:49:54.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:54.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:54.534664+0000) 2024-03-06T08:49:55.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:55.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:55.534873+0000) 2024-03-06T08:49:56.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:56.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:56.535130+0000) 2024-03-06T08:49:57.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:57.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:57.535392+0000) 2024-03-06T08:49:58.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:58.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:58.535702+0000) 2024-03-06T08:49:59.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:49:59.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:59.535965+0000) 2024-03-06T08:50:00.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:00.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:00.536217+0000) 2024-03-06T08:50:01.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:01.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:01.536473+0000) 2024-03-06T08:50:02.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:02.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:02.536750+0000) 2024-03-06T08:50:03.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:03.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:03.537014+0000) 2024-03-06T08:50:04.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:04.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:04.537283+0000) 2024-03-06T08:50:05.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:05.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:05.537552+0000) 2024-03-06T08:50:06.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:06.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:06.537813+0000) 2024-03-06T08:50:07.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:07.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:07.538052+0000) 2024-03-06T08:50:08.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:08.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:08.538307+0000) 2024-03-06T08:50:09.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:09.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:09.538575+0000) 2024-03-06T08:50:10.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:10.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:10.538843+0000) 2024-03-06T08:50:11.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:11.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:11.539097+0000) 2024-03-06T08:50:12.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:12.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:12.539328+0000) 2024-03-06T08:50:13.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:13.201 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:13.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:13.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:13.539576+0000) 2024-03-06T08:50:14.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:14.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:14.539815+0000) 2024-03-06T08:50:15.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:15.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:15.540071+0000) 2024-03-06T08:50:16.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:16.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:16.540259+0000) 2024-03-06T08:50:17.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:17.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:17.540507+0000) 2024-03-06T08:50:18.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:18.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:18.540754+0000) 2024-03-06T08:50:19.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:19.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:19.541017+0000) 2024-03-06T08:50:20.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:20.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:20.541252+0000) 2024-03-06T08:50:21.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:21.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:21.541531+0000) 2024-03-06T08:50:22.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:22.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:22.541805+0000) 2024-03-06T08:50:23.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:23.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:23.542067+0000) 2024-03-06T08:50:24.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:24.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:24.542331+0000) 2024-03-06T08:50:25.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:25.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:25.542605+0000) 2024-03-06T08:50:26.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:26.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:26.542943+0000) 2024-03-06T08:50:27.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:27.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:27.543221+0000) 2024-03-06T08:50:28.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:28.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:28.543485+0000) 2024-03-06T08:50:29.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:29.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:29.543747+0000) 2024-03-06T08:50:30.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:30.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:30.544003+0000) 2024-03-06T08:50:31.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:31.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:31.544281+0000) 2024-03-06T08:50:32.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:32.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:32.544537+0000) 2024-03-06T08:50:33.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:33.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:33.544818+0000) 2024-03-06T08:50:34.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:34.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:34.545098+0000) 2024-03-06T08:50:35.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:35.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:35.545326+0000) 2024-03-06T08:50:36.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:36.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:36.545686+0000) 2024-03-06T08:50:37.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:37.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:37.545956+0000) 2024-03-06T08:50:38.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:38.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:38.546246+0000) 2024-03-06T08:50:39.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:39.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:39.546531+0000) 2024-03-06T08:50:40.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:40.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:40.546750+0000) 2024-03-06T08:50:41.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:41.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:41.547016+0000) 2024-03-06T08:50:42.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:42.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:42.547233+0000) 2024-03-06T08:50:43.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:43.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:43.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:43.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:43.547453+0000) 2024-03-06T08:50:44.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:44.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:44.547726+0000) 2024-03-06T08:50:45.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:45.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:45.547977+0000) 2024-03-06T08:50:46.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:46.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:46.548230+0000) 2024-03-06T08:50:47.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:47.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:47.548469+0000) 2024-03-06T08:50:48.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:48.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:48.548758+0000) 2024-03-06T08:50:49.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:49.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:49.549041+0000) 2024-03-06T08:50:50.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:50.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:50.549294+0000) 2024-03-06T08:50:51.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:51.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:51.549546+0000) 2024-03-06T08:50:52.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:52.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:52.549819+0000) 2024-03-06T08:50:53.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:53.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:53.550095+0000) 2024-03-06T08:50:54.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:54.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:54.550348+0000) 2024-03-06T08:50:55.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:55.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:55.550599+0000) 2024-03-06T08:50:56.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:56.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:56.550833+0000) 2024-03-06T08:50:57.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:57.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:57.551075+0000) 2024-03-06T08:50:58.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:58.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:58.551257+0000) 2024-03-06T08:50:59.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:50:59.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:59.551509+0000) 2024-03-06T08:51:00.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:00.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:00.551742+0000) 2024-03-06T08:51:01.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:01.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:01.551999+0000) 2024-03-06T08:51:02.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:02.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:02.552248+0000) 2024-03-06T08:51:03.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:03.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:03.552448+0000) 2024-03-06T08:51:04.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:04.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:04.552734+0000) 2024-03-06T08:51:05.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:05.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:05.552998+0000) 2024-03-06T08:51:06.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:06.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:06.553226+0000) 2024-03-06T08:51:07.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:07.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:07.553490+0000) 2024-03-06T08:51:08.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:08.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:08.553747+0000) 2024-03-06T08:51:09.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:09.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:09.553998+0000) 2024-03-06T08:51:10.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:10.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:10.554262+0000) 2024-03-06T08:51:11.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:11.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:11.554469+0000) 2024-03-06T08:51:12.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:12.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:12.554766+0000) 2024-03-06T08:51:13.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:13.499 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:13.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:13.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:13.554936+0000) 2024-03-06T08:51:14.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:14.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:14.555180+0000) 2024-03-06T08:51:15.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:15.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:15.555449+0000) 2024-03-06T08:51:16.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:16.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:16.555758+0000) 2024-03-06T08:51:17.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:17.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:17.556034+0000) 2024-03-06T08:51:18.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:18.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:18.556304+0000) 2024-03-06T08:51:19.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:19.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:19.556562+0000) 2024-03-06T08:51:20.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:20.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:20.556816+0000) 2024-03-06T08:51:21.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:21.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:21.557059+0000) 2024-03-06T08:51:22.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:22.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:22.557298+0000) 2024-03-06T08:51:23.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:23.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:23.557539+0000) 2024-03-06T08:51:24.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:24.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:24.557789+0000) 2024-03-06T08:51:25.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:25.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:25.558055+0000) 2024-03-06T08:51:26.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:26.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:26.558310+0000) 2024-03-06T08:51:27.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:27.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:27.558574+0000) 2024-03-06T08:51:28.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:28.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:28.558808+0000) 2024-03-06T08:51:29.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:29.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:29.559043+0000) 2024-03-06T08:51:30.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:30.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:30.559222+0000) 2024-03-06T08:51:31.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:31.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:31.559481+0000) 2024-03-06T08:51:32.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:32.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:32.559754+0000) 2024-03-06T08:51:33.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:33.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:33.560013+0000) 2024-03-06T08:51:34.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:34.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:34.560298+0000) 2024-03-06T08:51:35.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:35.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:35.560563+0000) 2024-03-06T08:51:36.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:36.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:36.560820+0000) 2024-03-06T08:51:37.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:37.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:37.561088+0000) 2024-03-06T08:51:38.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:38.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:38.561345+0000) 2024-03-06T08:51:39.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:39.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:39.561583+0000) 2024-03-06T08:51:40.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:40.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:40.561819+0000) 2024-03-06T08:51:41.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:41.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:41.562086+0000) 2024-03-06T08:51:42.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:42.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:42.562362+0000) 2024-03-06T08:51:43.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:43.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:43.562626+0000) 2024-03-06T08:51:43.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:43.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:44.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:44.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:44.562911+0000) 2024-03-06T08:51:45.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:45.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:45.563160+0000) 2024-03-06T08:51:46.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:46.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:46.563424+0000) 2024-03-06T08:51:47.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:47.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:47.563711+0000) 2024-03-06T08:51:48.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:48.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:48.563961+0000) 2024-03-06T08:51:49.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:49.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:49.564208+0000) 2024-03-06T08:51:50.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:50.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:50.564460+0000) 2024-03-06T08:51:51.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:51.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:51.564741+0000) 2024-03-06T08:51:52.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:52.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:52.564994+0000) 2024-03-06T08:51:53.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:53.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:53.565251+0000) 2024-03-06T08:51:54.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:54.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:54.565503+0000) 2024-03-06T08:51:55.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:55.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:55.565757+0000) 2024-03-06T08:51:56.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:56.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:56.566010+0000) 2024-03-06T08:51:57.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:57.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:57.566268+0000) 2024-03-06T08:51:58.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:58.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:58.566524+0000) 2024-03-06T08:51:59.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:51:59.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:59.566747+0000) 2024-03-06T08:52:00.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:00.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:00.566994+0000) 2024-03-06T08:52:01.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:01.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:01.567241+0000) 2024-03-06T08:52:02.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:02.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:02.567496+0000) 2024-03-06T08:52:03.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:03.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:03.567755+0000) 2024-03-06T08:52:04.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:04.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:04.568001+0000) 2024-03-06T08:52:05.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:05.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:05.568262+0000) 2024-03-06T08:52:06.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:06.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:06.568517+0000) 2024-03-06T08:52:07.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:07.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:07.568758+0000) 2024-03-06T08:52:08.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:08.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:08.568993+0000) 2024-03-06T08:52:09.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:09.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:09.569247+0000) 2024-03-06T08:52:10.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:10.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:10.569496+0000) 2024-03-06T08:52:11.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:11.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:11.569735+0000) 2024-03-06T08:52:12.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:12.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:12.569973+0000) 2024-03-06T08:52:13.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:13.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:13.570210+0000) 2024-03-06T08:52:13.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:13.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:14.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:14.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:14.570445+0000) 2024-03-06T08:52:15.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:15.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:15.570752+0000) 2024-03-06T08:52:16.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:16.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:16.570984+0000) 2024-03-06T08:52:17.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:17.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:17.571278+0000) 2024-03-06T08:52:18.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:18.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:18.571527+0000) 2024-03-06T08:52:19.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:19.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:19.571777+0000) 2024-03-06T08:52:20.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:20.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:20.572009+0000) 2024-03-06T08:52:21.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:21.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:21.572270+0000) 2024-03-06T08:52:22.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:22.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:22.572509+0000) 2024-03-06T08:52:23.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:23.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:23.572763+0000) 2024-03-06T08:52:24.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:24.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:24.573021+0000) 2024-03-06T08:52:25.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:25.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:25.573269+0000) 2024-03-06T08:52:26.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:26.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:26.573500+0000) 2024-03-06T08:52:27.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:27.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:27.573747+0000) 2024-03-06T08:52:28.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:28.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:28.573981+0000) 2024-03-06T08:52:29.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:29.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:29.574220+0000) 2024-03-06T08:52:30.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:30.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:30.574472+0000) 2024-03-06T08:52:31.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:31.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:31.574741+0000) 2024-03-06T08:52:32.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:32.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:32.574996+0000) 2024-03-06T08:52:33.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:33.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:33.575263+0000) 2024-03-06T08:52:34.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:34.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:34.575510+0000) 2024-03-06T08:52:35.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:35.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:35.575760+0000) 2024-03-06T08:52:36.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:36.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:36.576016+0000) 2024-03-06T08:52:37.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:37.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:37.576276+0000) 2024-03-06T08:52:38.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:38.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:38.576534+0000) 2024-03-06T08:52:39.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:39.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:39.576791+0000) 2024-03-06T08:52:40.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:40.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:40.577022+0000) 2024-03-06T08:52:41.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:41.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:41.577258+0000) 2024-03-06T08:52:42.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:42.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:42.577436+0000) 2024-03-06T08:52:43.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:43.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:43.577749+0000) 2024-03-06T08:52:43.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:43.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:44.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:44.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:44.578007+0000) 2024-03-06T08:52:45.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:45.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:45.578334+0000) 2024-03-06T08:52:46.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:46.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:46.578576+0000) 2024-03-06T08:52:47.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:47.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:47.578841+0000) 2024-03-06T08:52:48.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:48.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:48.579131+0000) 2024-03-06T08:52:49.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:49.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:49.579370+0000) 2024-03-06T08:52:50.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:50.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:50.579733+0000) 2024-03-06T08:52:51.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:51.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:51.580035+0000) 2024-03-06T08:52:52.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:52.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:52.580328+0000) 2024-03-06T08:52:53.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:53.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:53.580616+0000) 2024-03-06T08:52:54.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:54.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:54.580903+0000) 2024-03-06T08:52:55.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:55.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:55.581200+0000) 2024-03-06T08:52:56.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:56.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:56.581510+0000) 2024-03-06T08:52:57.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:57.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:57.581813+0000) 2024-03-06T08:52:58.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:58.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:58.582129+0000) 2024-03-06T08:52:59.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:52:59.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:59.582446+0000) 2024-03-06T08:53:00.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:00.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:00.582776+0000) 2024-03-06T08:53:01.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:01.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:01.583085+0000) 2024-03-06T08:53:02.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:02.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:02.583324+0000) 2024-03-06T08:53:03.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:03.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:03.583601+0000) 2024-03-06T08:53:04.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:04.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:04.583894+0000) 2024-03-06T08:53:05.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:05.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:05.584160+0000) 2024-03-06T08:53:06.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:06.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:06.584399+0000) 2024-03-06T08:53:07.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:07.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:07.584665+0000) 2024-03-06T08:53:08.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:08.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:08.584920+0000) 2024-03-06T08:53:09.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:09.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:09.585166+0000) 2024-03-06T08:53:10.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:10.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:10.585402+0000) 2024-03-06T08:53:11.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:11.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:11.585667+0000) 2024-03-06T08:53:12.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:12.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:12.585917+0000) 2024-03-06T08:53:13.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:13.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:13.586174+0000) 2024-03-06T08:53:14.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:14.116 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:14.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:14.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:14.586398+0000) 2024-03-06T08:53:15.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:15.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:15.586689+0000) 2024-03-06T08:53:16.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:16.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:16.586918+0000) 2024-03-06T08:53:17.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:17.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:17.587222+0000) 2024-03-06T08:53:18.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:18.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:18.587494+0000) 2024-03-06T08:53:19.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:19.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:19.587754+0000) 2024-03-06T08:53:20.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:20.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:20.588009+0000) 2024-03-06T08:53:21.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:21.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:21.588257+0000) 2024-03-06T08:53:22.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:22.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:22.588503+0000) 2024-03-06T08:53:23.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:23.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:23.588766+0000) 2024-03-06T08:53:24.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:24.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:24.589026+0000) 2024-03-06T08:53:25.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:25.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:25.589292+0000) 2024-03-06T08:53:26.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:26.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:26.589515+0000) 2024-03-06T08:53:27.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:27.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:27.589777+0000) 2024-03-06T08:53:28.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:28.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:28.590033+0000) 2024-03-06T08:53:29.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:29.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:29.590270+0000) 2024-03-06T08:53:30.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:30.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:30.590521+0000) 2024-03-06T08:53:31.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:31.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:31.590758+0000) 2024-03-06T08:53:32.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:32.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:32.591012+0000) 2024-03-06T08:53:33.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:33.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:33.591267+0000) 2024-03-06T08:53:34.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:34.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:34.591495+0000) 2024-03-06T08:53:35.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:35.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:35.591762+0000) 2024-03-06T08:53:36.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:36.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:36.591993+0000) 2024-03-06T08:53:37.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:37.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:37.592232+0000) 2024-03-06T08:53:38.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:38.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:38.592470+0000) 2024-03-06T08:53:39.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:39.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:39.592741+0000) 2024-03-06T08:53:40.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:40.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:40.592991+0000) 2024-03-06T08:53:41.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:41.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:41.593252+0000) 2024-03-06T08:53:42.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:42.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:42.593492+0000) 2024-03-06T08:53:43.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:43.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:43.593760+0000) 2024-03-06T08:53:44.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:44.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:44.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:44.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:44.594007+0000) 2024-03-06T08:53:45.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:45.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:45.594270+0000) 2024-03-06T08:53:46.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:46.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:46.594500+0000) 2024-03-06T08:53:47.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:47.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:47.594752+0000) 2024-03-06T08:53:48.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:48.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:48.595005+0000) 2024-03-06T08:53:49.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:49.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:49.595260+0000) 2024-03-06T08:53:50.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:50.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:50.595494+0000) 2024-03-06T08:53:51.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:51.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:51.595772+0000) 2024-03-06T08:53:52.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:52.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:52.596025+0000) 2024-03-06T08:53:53.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:53.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:53.596264+0000) 2024-03-06T08:53:54.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:54.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:54.596521+0000) 2024-03-06T08:53:55.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:55.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:55.596765+0000) 2024-03-06T08:53:56.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:56.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:56.596997+0000) 2024-03-06T08:53:57.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:57.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:57.597252+0000) 2024-03-06T08:53:58.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:58.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:58.597488+0000) 2024-03-06T08:53:59.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:53:59.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:59.597751+0000) 2024-03-06T08:54:00.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:00.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:00.597976+0000) 2024-03-06T08:54:01.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:01.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:01.598215+0000) 2024-03-06T08:54:02.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:02.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:02.598450+0000) 2024-03-06T08:54:03.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:03.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:03.598749+0000) 2024-03-06T08:54:04.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:04.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:04.598976+0000) 2024-03-06T08:54:05.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:05.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:05.599244+0000) 2024-03-06T08:54:06.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:06.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:06.599479+0000) 2024-03-06T08:54:07.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:07.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:07.599755+0000) 2024-03-06T08:54:08.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:08.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:08.600017+0000) 2024-03-06T08:54:09.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:09.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:09.600277+0000) 2024-03-06T08:54:10.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:10.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:10.600533+0000) 2024-03-06T08:54:11.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:11.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:11.600795+0000) 2024-03-06T08:54:12.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:12.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:12.601029+0000) 2024-03-06T08:54:13.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:13.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:13.601281+0000) 2024-03-06T08:54:14.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:14.414 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:14.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:14.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:14.601539+0000) 2024-03-06T08:54:15.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:15.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:15.601901+0000) 2024-03-06T08:54:16.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:16.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:16.602133+0000) 2024-03-06T08:54:17.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:17.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:17.602368+0000) 2024-03-06T08:54:18.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:18.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:18.602606+0000) 2024-03-06T08:54:19.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:19.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:19.602877+0000) 2024-03-06T08:54:20.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:20.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:20.603182+0000) 2024-03-06T08:54:21.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:21.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:21.603416+0000) 2024-03-06T08:54:22.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:22.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:22.603699+0000) 2024-03-06T08:54:23.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:23.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:23.603961+0000) 2024-03-06T08:54:24.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:24.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:24.604195+0000) 2024-03-06T08:54:25.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:25.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:25.604466+0000) 2024-03-06T08:54:26.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:26.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:26.604734+0000) 2024-03-06T08:54:27.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:27.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:27.604972+0000) 2024-03-06T08:54:28.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:28.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:28.605222+0000) 2024-03-06T08:54:29.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:29.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:29.605414+0000) 2024-03-06T08:54:30.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:30.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:30.605697+0000) 2024-03-06T08:54:31.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:31.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:31.605960+0000) 2024-03-06T08:54:32.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:32.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:32.606213+0000) 2024-03-06T08:54:33.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:33.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:33.606471+0000) 2024-03-06T08:54:34.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:34.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:34.606748+0000) 2024-03-06T08:54:35.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:35.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:35.606973+0000) 2024-03-06T08:54:36.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:36.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:36.607254+0000) 2024-03-06T08:54:37.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:37.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:37.607513+0000) 2024-03-06T08:54:38.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:38.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:38.607750+0000) 2024-03-06T08:54:39.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:39.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:39.607998+0000) 2024-03-06T08:54:40.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:40.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:40.608252+0000) 2024-03-06T08:54:41.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:41.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:41.608491+0000) 2024-03-06T08:54:42.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:42.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:42.608728+0000) 2024-03-06T08:54:43.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:43.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:43.608989+0000) 2024-03-06T08:54:44.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:44.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:44.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:44.609247+0000) 2024-03-06T08:54:44.651 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:45.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:45.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:45.609512+0000) 2024-03-06T08:54:46.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:46.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:46.609747+0000) 2024-03-06T08:54:47.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:47.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:47.610010+0000) 2024-03-06T08:54:48.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:48.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:48.610266+0000) 2024-03-06T08:54:49.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:49.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:49.610523+0000) 2024-03-06T08:54:50.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:50.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:50.610782+0000) 2024-03-06T08:54:51.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:51.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:51.611038+0000) 2024-03-06T08:54:52.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:52.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:52.611295+0000) 2024-03-06T08:54:53.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:53.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:53.611558+0000) 2024-03-06T08:54:54.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:54.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:54.611855+0000) 2024-03-06T08:54:55.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:55.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:55.612133+0000) 2024-03-06T08:54:56.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:56.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:56.612398+0000) 2024-03-06T08:54:57.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:57.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:57.612699+0000) 2024-03-06T08:54:58.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:58.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:58.612937+0000) 2024-03-06T08:54:59.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:54:59.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:59.613134+0000) 2024-03-06T08:55:00.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:00.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:00.613371+0000) 2024-03-06T08:55:01.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:01.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:01.613605+0000) 2024-03-06T08:55:02.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:02.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:02.613899+0000) 2024-03-06T08:55:03.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:03.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:03.614165+0000) 2024-03-06T08:55:04.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:04.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:04.614423+0000) 2024-03-06T08:55:05.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:05.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:05.614684+0000) 2024-03-06T08:55:06.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:06.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:06.614887+0000) 2024-03-06T08:55:07.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:07.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:07.615145+0000) 2024-03-06T08:55:08.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:08.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:08.615404+0000) 2024-03-06T08:55:09.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:09.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:09.615729+0000) 2024-03-06T08:55:10.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:10.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:10.616026+0000) 2024-03-06T08:55:11.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:11.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:11.616348+0000) 2024-03-06T08:55:12.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:12.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:12.616711+0000) 2024-03-06T08:55:13.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:13.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:13.617073+0000) 2024-03-06T08:55:14.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:14.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:14.617390+0000) 2024-03-06T08:55:14.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:14.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:15.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:15.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:15.617759+0000) 2024-03-06T08:55:16.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:16.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:16.618083+0000) 2024-03-06T08:55:17.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:17.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:17.618337+0000) 2024-03-06T08:55:18.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:18.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:18.618671+0000) 2024-03-06T08:55:19.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:19.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:19.618938+0000) 2024-03-06T08:55:20.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:20.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:20.619197+0000) 2024-03-06T08:55:21.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:21.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:21.619423+0000) 2024-03-06T08:55:22.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:22.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:22.619695+0000) 2024-03-06T08:55:23.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:23.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:23.619954+0000) 2024-03-06T08:55:24.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:24.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:24.620215+0000) 2024-03-06T08:55:25.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:25.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:25.620487+0000) 2024-03-06T08:55:26.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:26.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:26.620745+0000) 2024-03-06T08:55:27.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:27.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:27.620979+0000) 2024-03-06T08:55:28.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:28.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:28.621174+0000) 2024-03-06T08:55:29.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:29.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:29.621431+0000) 2024-03-06T08:55:30.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:30.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:30.621728+0000) 2024-03-06T08:55:31.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:31.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:31.621967+0000) 2024-03-06T08:55:32.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:32.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:32.622211+0000) 2024-03-06T08:55:33.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:33.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:33.622460+0000) 2024-03-06T08:55:34.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:34.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:34.622734+0000) 2024-03-06T08:55:35.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:35.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:35.622996+0000) 2024-03-06T08:55:36.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:36.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:36.623247+0000) 2024-03-06T08:55:37.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:37.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:37.623488+0000) 2024-03-06T08:55:38.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:38.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:38.623748+0000) 2024-03-06T08:55:39.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:39.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:39.624007+0000) 2024-03-06T08:55:40.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:40.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:40.624258+0000) 2024-03-06T08:55:41.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:41.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:41.624503+0000) 2024-03-06T08:55:42.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:42.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:42.624736+0000) 2024-03-06T08:55:43.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:43.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:43.625004+0000) 2024-03-06T08:55:44.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:44.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:44.625259+0000) 2024-03-06T08:55:44.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:44.978 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:45.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:45.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:45.625512+0000) 2024-03-06T08:55:46.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:46.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:46.625743+0000) 2024-03-06T08:55:47.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:47.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:47.625995+0000) 2024-03-06T08:55:48.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:48.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:48.626254+0000) 2024-03-06T08:55:49.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:49.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:49.626503+0000) 2024-03-06T08:55:50.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:50.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:50.626730+0000) 2024-03-06T08:55:51.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:51.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:51.626914+0000) 2024-03-06T08:55:52.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:52.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:52.627177+0000) 2024-03-06T08:55:53.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:53.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:53.627436+0000) 2024-03-06T08:55:54.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:54.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:54.627737+0000) 2024-03-06T08:55:55.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:55.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:55.628005+0000) 2024-03-06T08:55:56.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:56.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:56.628188+0000) 2024-03-06T08:55:57.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:57.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:57.628432+0000) 2024-03-06T08:55:58.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:58.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:58.628731+0000) 2024-03-06T08:55:59.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:55:59.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:59.628992+0000) 2024-03-06T08:56:00.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:00.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:00.629225+0000) 2024-03-06T08:56:01.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:01.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:01.629471+0000) 2024-03-06T08:56:02.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:02.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:02.629749+0000) 2024-03-06T08:56:03.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:03.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:03.630004+0000) 2024-03-06T08:56:04.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:04.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:04.630185+0000) 2024-03-06T08:56:05.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:05.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:05.630453+0000) 2024-03-06T08:56:06.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:06.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:06.630738+0000) 2024-03-06T08:56:07.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:07.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:07.630987+0000) 2024-03-06T08:56:08.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:08.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:08.631223+0000) 2024-03-06T08:56:09.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:09.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:09.631460+0000) 2024-03-06T08:56:10.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:10.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:10.631745+0000) 2024-03-06T08:56:11.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:11.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:11.632001+0000) 2024-03-06T08:56:12.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:12.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:12.632303+0000) 2024-03-06T08:56:13.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:13.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:13.632550+0000) 2024-03-06T08:56:14.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:14.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:14.632792+0000) 2024-03-06T08:56:15.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:15.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:15.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:15.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:15.633056+0000) 2024-03-06T08:56:16.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:16.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:16.633314+0000) 2024-03-06T08:56:17.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:17.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:17.633561+0000) 2024-03-06T08:56:18.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:18.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:18.633761+0000) 2024-03-06T08:56:19.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:19.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:19.634015+0000) 2024-03-06T08:56:20.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:20.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:20.634266+0000) 2024-03-06T08:56:21.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:21.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:21.634495+0000) 2024-03-06T08:56:22.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:22.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:22.634776+0000) 2024-03-06T08:56:23.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:23.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:23.635023+0000) 2024-03-06T08:56:24.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:24.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:24.635273+0000) 2024-03-06T08:56:25.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:25.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:25.635520+0000) 2024-03-06T08:56:26.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:26.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:26.635772+0000) 2024-03-06T08:56:27.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:27.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:27.636024+0000) 2024-03-06T08:56:28.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:28.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:28.636288+0000) 2024-03-06T08:56:29.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:29.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:29.636545+0000) 2024-03-06T08:56:30.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:30.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:30.636778+0000) 2024-03-06T08:56:31.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:31.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:31.637025+0000) 2024-03-06T08:56:32.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:32.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:32.637264+0000) 2024-03-06T08:56:33.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:33.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:33.637539+0000) 2024-03-06T08:56:34.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:34.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:34.637792+0000) 2024-03-06T08:56:35.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:35.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:35.637992+0000) 2024-03-06T08:56:36.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:36.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:36.638227+0000) 2024-03-06T08:56:37.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:37.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:37.638465+0000) 2024-03-06T08:56:38.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:38.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:38.638753+0000) 2024-03-06T08:56:39.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:39.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:39.639013+0000) 2024-03-06T08:56:40.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:40.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:40.639250+0000) 2024-03-06T08:56:41.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:41.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:41.639488+0000) 2024-03-06T08:56:42.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:42.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:42.639760+0000) 2024-03-06T08:56:43.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:43.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:43.640007+0000) 2024-03-06T08:56:44.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:44.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:44.640264+0000) 2024-03-06T08:56:45.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:45.291 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:45.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:45.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:45.640537+0000) 2024-03-06T08:56:46.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:46.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:46.640788+0000) 2024-03-06T08:56:47.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:47.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:47.640982+0000) 2024-03-06T08:56:48.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:48.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:48.641238+0000) 2024-03-06T08:56:49.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:49.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:49.641481+0000) 2024-03-06T08:56:50.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:50.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:50.641730+0000) 2024-03-06T08:56:51.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:51.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:51.641987+0000) 2024-03-06T08:56:52.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:52.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:52.642244+0000) 2024-03-06T08:56:53.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:53.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:53.642504+0000) 2024-03-06T08:56:54.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:54.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:54.642763+0000) 2024-03-06T08:56:55.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:55.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:55.643009+0000) 2024-03-06T08:56:56.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:56.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:56.643250+0000) 2024-03-06T08:56:57.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:57.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:57.643503+0000) 2024-03-06T08:56:58.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:58.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:58.643758+0000) 2024-03-06T08:56:59.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:56:59.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:59.644001+0000) 2024-03-06T08:57:00.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:00.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:00.644293+0000) 2024-03-06T08:57:01.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:01.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:01.644543+0000) 2024-03-06T08:57:02.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:02.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:02.644789+0000) 2024-03-06T08:57:03.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:03.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:03.645032+0000) 2024-03-06T08:57:04.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:04.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:04.645266+0000) 2024-03-06T08:57:05.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:05.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:05.645535+0000) 2024-03-06T08:57:06.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:06.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:06.645776+0000) 2024-03-06T08:57:07.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:07.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:07.646011+0000) 2024-03-06T08:57:08.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:08.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:08.646199+0000) 2024-03-06T08:57:09.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:09.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:09.646460+0000) 2024-03-06T08:57:10.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:10.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:10.646755+0000) 2024-03-06T08:57:11.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:11.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:11.647013+0000) 2024-03-06T08:57:12.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:12.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:12.647269+0000) 2024-03-06T08:57:13.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:13.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:13.647534+0000) 2024-03-06T08:57:14.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:14.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:14.647789+0000) 2024-03-06T08:57:15.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:15.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:15.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:15.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:15.648082+0000) 2024-03-06T08:57:16.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:16.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:16.648355+0000) 2024-03-06T08:57:17.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:17.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:17.648613+0000) 2024-03-06T08:57:18.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:18.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:18.648947+0000) 2024-03-06T08:57:19.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:19.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:19.649209+0000) 2024-03-06T08:57:20.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:20.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:20.649487+0000) 2024-03-06T08:57:21.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:21.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:21.649767+0000) 2024-03-06T08:57:22.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:22.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:22.650034+0000) 2024-03-06T08:57:23.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:23.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:23.650294+0000) 2024-03-06T08:57:24.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:24.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:24.650548+0000) 2024-03-06T08:57:25.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:25.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:25.650818+0000) 2024-03-06T08:57:26.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:26.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:26.651019+0000) 2024-03-06T08:57:27.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:27.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:27.651269+0000) 2024-03-06T08:57:28.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:28.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:28.651504+0000) 2024-03-06T08:57:29.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:29.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:29.651746+0000) 2024-03-06T08:57:30.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:30.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:30.651972+0000) 2024-03-06T08:57:31.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:31.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:31.652238+0000) 2024-03-06T08:57:32.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:32.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:32.652487+0000) 2024-03-06T08:57:33.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:33.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:33.652746+0000) 2024-03-06T08:57:34.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:34.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:34.653001+0000) 2024-03-06T08:57:35.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:35.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:35.653275+0000) 2024-03-06T08:57:36.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:36.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:36.653532+0000) 2024-03-06T08:57:37.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:37.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:37.653798+0000) 2024-03-06T08:57:38.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:38.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:38.654036+0000) 2024-03-06T08:57:39.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:39.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:39.654300+0000) 2024-03-06T08:57:40.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:40.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:40.654566+0000) 2024-03-06T08:57:41.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:41.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:41.654814+0000) 2024-03-06T08:57:42.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:42.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:42.655081+0000) 2024-03-06T08:57:43.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:43.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:43.655316+0000) 2024-03-06T08:57:44.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:44.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:44.655532+0000) 2024-03-06T08:57:45.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:45.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:45.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:45.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:45.655834+0000) 2024-03-06T08:57:46.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:46.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:46.656085+0000) 2024-03-06T08:57:47.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:47.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:47.656363+0000) 2024-03-06T08:57:48.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:48.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:48.656676+0000) 2024-03-06T08:57:49.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:49.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:49.656916+0000) 2024-03-06T08:57:50.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:50.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:50.657115+0000) 2024-03-06T08:57:51.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:51.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:51.657368+0000) 2024-03-06T08:57:52.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:52.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:52.657614+0000) 2024-03-06T08:57:53.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:53.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:53.657942+0000) 2024-03-06T08:57:54.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:54.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:54.658194+0000) 2024-03-06T08:57:55.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:55.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:55.658455+0000) 2024-03-06T08:57:56.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:56.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:56.658752+0000) 2024-03-06T08:57:57.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:57.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:57.658989+0000) 2024-03-06T08:57:58.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:58.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:58.659271+0000) 2024-03-06T08:57:59.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:57:59.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:59.659492+0000) 2024-03-06T08:58:00.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:00.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:00.659792+0000) 2024-03-06T08:58:01.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:01.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:01.660019+0000) 2024-03-06T08:58:02.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:02.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:02.660274+0000) 2024-03-06T08:58:03.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:03.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:03.660488+0000) 2024-03-06T08:58:04.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:04.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:04.660715+0000) 2024-03-06T08:58:05.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:05.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:05.660947+0000) 2024-03-06T08:58:06.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:06.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:06.661182+0000) 2024-03-06T08:58:07.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:07.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:07.661417+0000) 2024-03-06T08:58:08.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:08.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:08.661715+0000) 2024-03-06T08:58:09.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:09.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:09.661937+0000) 2024-03-06T08:58:10.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:10.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:10.662130+0000) 2024-03-06T08:58:11.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:11.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:11.662374+0000) 2024-03-06T08:58:12.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:12.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:12.662624+0000) 2024-03-06T08:58:13.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:13.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:13.662929+0000) 2024-03-06T08:58:14.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:14.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:14.663177+0000) 2024-03-06T08:58:15.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:15.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:15.663422+0000) 2024-03-06T08:58:15.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:15.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:16.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:16.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:16.663617+0000) 2024-03-06T08:58:17.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:17.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:17.663891+0000) 2024-03-06T08:58:18.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:18.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:18.664098+0000) 2024-03-06T08:58:19.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:19.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:19.664294+0000) 2024-03-06T08:58:20.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:20.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:20.664501+0000) 2024-03-06T08:58:21.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:21.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:21.664759+0000) 2024-03-06T08:58:22.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:22.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:22.664974+0000) 2024-03-06T08:58:23.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:23.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:23.665185+0000) 2024-03-06T08:58:24.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:24.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:24.665390+0000) 2024-03-06T08:58:25.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:25.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:25.665612+0000) 2024-03-06T08:58:26.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:26.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:26.665869+0000) 2024-03-06T08:58:27.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:27.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:27.666078+0000) 2024-03-06T08:58:28.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:28.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:28.666287+0000) 2024-03-06T08:58:29.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:29.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:29.666506+0000) 2024-03-06T08:58:30.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:30.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:30.666754+0000) 2024-03-06T08:58:31.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:31.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:31.666971+0000) 2024-03-06T08:58:32.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:32.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:32.667178+0000) 2024-03-06T08:58:33.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:33.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:33.667419+0000) 2024-03-06T08:58:34.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:34.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:34.667751+0000) 2024-03-06T08:58:35.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:35.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:35.668008+0000) 2024-03-06T08:58:36.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:36.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:36.668217+0000) 2024-03-06T08:58:37.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:37.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:37.668481+0000) 2024-03-06T08:58:38.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:38.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:38.668739+0000) 2024-03-06T08:58:39.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:39.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:39.669012+0000) 2024-03-06T08:58:40.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:40.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:40.669228+0000) 2024-03-06T08:58:41.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:41.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:41.669489+0000) 2024-03-06T08:58:42.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:42.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:42.669734+0000) 2024-03-06T08:58:43.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:43.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:43.670006+0000) 2024-03-06T08:58:44.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:44.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:44.670272+0000) 2024-03-06T08:58:45.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:45.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:45.670555+0000) 2024-03-06T08:58:45.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:45.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:46.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:46.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:46.670842+0000) 2024-03-06T08:58:47.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:47.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:47.671101+0000) 2024-03-06T08:58:48.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:48.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:48.671362+0000) 2024-03-06T08:58:49.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:49.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:49.671601+0000) 2024-03-06T08:58:50.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:50.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:50.671861+0000) 2024-03-06T08:58:51.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:51.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:51.672098+0000) 2024-03-06T08:58:52.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:52.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:52.672333+0000) 2024-03-06T08:58:53.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:53.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:53.672579+0000) 2024-03-06T08:58:54.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:54.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:54.672856+0000) 2024-03-06T08:58:55.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:55.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:55.673173+0000) 2024-03-06T08:58:56.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:56.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:56.673450+0000) 2024-03-06T08:58:57.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:57.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:57.673678+0000) 2024-03-06T08:58:58.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:58.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:58.673942+0000) 2024-03-06T08:58:59.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:58:59.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:59.674210+0000) 2024-03-06T08:59:00.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:00.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:00.674475+0000) 2024-03-06T08:59:01.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:01.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:01.674755+0000) 2024-03-06T08:59:02.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:02.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:02.675007+0000) 2024-03-06T08:59:03.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:03.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:03.675279+0000) 2024-03-06T08:59:04.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:04.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:04.675553+0000) 2024-03-06T08:59:05.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:05.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:05.675823+0000) 2024-03-06T08:59:06.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:06.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:06.676070+0000) 2024-03-06T08:59:07.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:07.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:07.676323+0000) 2024-03-06T08:59:08.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:08.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:08.676597+0000) 2024-03-06T08:59:09.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:09.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:09.676925+0000) 2024-03-06T08:59:10.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:10.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:10.677183+0000) 2024-03-06T08:59:11.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:11.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:11.677437+0000) 2024-03-06T08:59:12.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:12.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:12.677716+0000) 2024-03-06T08:59:13.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:13.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:13.677975+0000) 2024-03-06T08:59:14.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:14.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:14.678229+0000) 2024-03-06T08:59:15.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:15.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:15.678495+0000) 2024-03-06T08:59:16.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:16.066 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:16.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:16.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:16.678733+0000) 2024-03-06T08:59:17.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:17.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:17.678970+0000) 2024-03-06T08:59:18.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:18.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:18.679231+0000) 2024-03-06T08:59:19.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:19.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:19.679471+0000) 2024-03-06T08:59:20.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:20.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:20.679729+0000) 2024-03-06T08:59:21.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:21.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:21.679985+0000) 2024-03-06T08:59:22.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:22.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:22.680217+0000) 2024-03-06T08:59:23.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:23.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:23.680475+0000) 2024-03-06T08:59:24.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:24.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:24.680738+0000) 2024-03-06T08:59:25.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:25.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:25.681006+0000) 2024-03-06T08:59:26.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:26.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:26.681237+0000) 2024-03-06T08:59:27.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:27.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:27.681469+0000) 2024-03-06T08:59:28.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:28.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:28.681748+0000) 2024-03-06T08:59:29.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:29.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:29.681986+0000) 2024-03-06T08:59:30.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:30.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:30.682236+0000) 2024-03-06T08:59:31.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:31.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:31.682472+0000) 2024-03-06T08:59:32.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:32.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:32.682744+0000) 2024-03-06T08:59:33.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:33.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:33.682982+0000) 2024-03-06T08:59:34.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:34.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:34.683237+0000) 2024-03-06T08:59:35.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:35.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:35.683508+0000) 2024-03-06T08:59:36.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:36.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:36.683746+0000) 2024-03-06T08:59:37.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:37.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:37.683928+0000) 2024-03-06T08:59:38.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:38.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:38.684181+0000) 2024-03-06T08:59:39.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:39.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:39.684443+0000) 2024-03-06T08:59:40.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:40.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:40.684725+0000) 2024-03-06T08:59:41.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:41.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:41.684962+0000) 2024-03-06T08:59:42.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:42.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:42.685216+0000) 2024-03-06T08:59:43.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:43.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:43.685463+0000) 2024-03-06T08:59:44.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:44.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:44.685741+0000) 2024-03-06T08:59:45.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:45.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:45.686050+0000) 2024-03-06T08:59:46.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:46.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:46.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:46.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:46.686297+0000) 2024-03-06T08:59:47.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:47.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:47.686533+0000) 2024-03-06T08:59:48.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:48.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:48.686759+0000) 2024-03-06T08:59:49.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:49.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:49.687000+0000) 2024-03-06T08:59:50.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:50.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:50.687197+0000) 2024-03-06T08:59:51.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:51.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:51.687434+0000) 2024-03-06T08:59:52.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:52.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:52.687732+0000) 2024-03-06T08:59:53.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:53.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:53.687991+0000) 2024-03-06T08:59:54.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:54.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:54.688253+0000) 2024-03-06T08:59:55.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:55.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:55.688514+0000) 2024-03-06T08:59:56.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:56.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:56.688749+0000) 2024-03-06T08:59:57.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:57.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:57.688940+0000) 2024-03-06T08:59:58.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:58.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:58.689182+0000) 2024-03-06T08:59:59.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T08:59:59.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:59.689451+0000) 2024-03-06T09:00:00.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:00.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:00.689740+0000) 2024-03-06T09:00:01.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:01.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:01.689999+0000) 2024-03-06T09:00:02.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:02.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:02.690250+0000) 2024-03-06T09:00:03.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:03.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:03.690493+0000) 2024-03-06T09:00:04.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:04.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:04.690746+0000) 2024-03-06T09:00:05.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:05.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:05.691018+0000) 2024-03-06T09:00:06.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:06.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:06.691266+0000) 2024-03-06T09:00:07.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:07.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:07.691507+0000) 2024-03-06T09:00:08.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:08.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:08.691762+0000) 2024-03-06T09:00:09.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:09.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:09.692009+0000) 2024-03-06T09:00:10.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:10.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:10.692262+0000) 2024-03-06T09:00:11.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:11.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:11.692501+0000) 2024-03-06T09:00:12.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:12.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:12.692756+0000) 2024-03-06T09:00:13.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:13.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:13.693001+0000) 2024-03-06T09:00:14.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:14.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:14.693257+0000) 2024-03-06T09:00:15.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:15.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:15.693531+0000) 2024-03-06T09:00:16.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:16.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:16.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:16.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:16.693808+0000) 2024-03-06T09:00:17.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:17.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:17.694056+0000) 2024-03-06T09:00:18.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:18.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:18.694309+0000) 2024-03-06T09:00:19.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:19.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:19.694577+0000) 2024-03-06T09:00:20.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:20.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:20.694829+0000) 2024-03-06T09:00:21.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:21.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:21.695068+0000) 2024-03-06T09:00:22.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:22.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:22.695323+0000) 2024-03-06T09:00:23.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:23.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:23.695514+0000) 2024-03-06T09:00:24.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:24.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:24.695768+0000) 2024-03-06T09:00:25.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:25.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:25.696021+0000) 2024-03-06T09:00:26.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:26.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:26.696259+0000) 2024-03-06T09:00:27.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:27.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:27.696510+0000) 2024-03-06T09:00:28.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:28.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:28.696763+0000) 2024-03-06T09:00:29.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:29.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:29.697038+0000) 2024-03-06T09:00:30.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:30.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:30.697294+0000) 2024-03-06T09:00:31.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:31.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:31.697573+0000) 2024-03-06T09:00:32.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:32.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:32.697847+0000) 2024-03-06T09:00:33.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:33.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:33.698132+0000) 2024-03-06T09:00:34.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:34.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:34.698404+0000) 2024-03-06T09:00:35.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:35.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:35.698730+0000) 2024-03-06T09:00:36.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:36.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:36.698905+0000) 2024-03-06T09:00:37.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:37.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:37.699090+0000) 2024-03-06T09:00:38.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:38.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:38.699327+0000) 2024-03-06T09:00:39.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:39.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:39.699582+0000) 2024-03-06T09:00:40.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:40.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:40.699838+0000) 2024-03-06T09:00:41.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:41.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:41.700085+0000) 2024-03-06T09:00:42.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:42.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:42.700343+0000) 2024-03-06T09:00:43.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:43.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:43.700614+0000) 2024-03-06T09:00:44.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:44.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:44.700985+0000) 2024-03-06T09:00:45.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:45.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:45.701255+0000) 2024-03-06T09:00:46.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:46.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:46.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:46.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:46.701493+0000) 2024-03-06T09:00:47.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:47.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:47.701769+0000) 2024-03-06T09:00:48.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:48.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:48.702026+0000) 2024-03-06T09:00:49.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:49.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:49.702304+0000) 2024-03-06T09:00:50.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:50.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:50.702566+0000) 2024-03-06T09:00:51.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:51.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:51.702812+0000) 2024-03-06T09:00:52.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:52.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:52.703053+0000) 2024-03-06T09:00:53.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:53.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:53.703314+0000) 2024-03-06T09:00:54.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:54.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:54.703566+0000) 2024-03-06T09:00:55.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:55.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:55.703825+0000) 2024-03-06T09:00:56.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:56.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:56.704072+0000) 2024-03-06T09:00:57.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:57.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:57.704334+0000) 2024-03-06T09:00:58.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:58.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:58.704576+0000) 2024-03-06T09:00:59.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:00:59.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:59.704842+0000) 2024-03-06T09:01:00.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:00.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:00.705092+0000) 2024-03-06T09:01:01.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:01.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:01.705288+0000) 2024-03-06T09:01:02.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:02.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:02.705576+0000) 2024-03-06T09:01:03.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:03.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:03.705853+0000) 2024-03-06T09:01:04.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:04.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:04.706123+0000) 2024-03-06T09:01:05.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:05.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:05.706331+0000) 2024-03-06T09:01:06.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:06.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:06.706487+0000) 2024-03-06T09:01:07.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:07.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:07.706752+0000) 2024-03-06T09:01:08.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:08.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:08.707006+0000) 2024-03-06T09:01:09.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:09.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:09.707277+0000) 2024-03-06T09:01:10.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:10.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:10.707527+0000) 2024-03-06T09:01:11.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:11.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:11.707778+0000) 2024-03-06T09:01:12.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:12.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:12.708019+0000) 2024-03-06T09:01:13.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:13.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:13.708296+0000) 2024-03-06T09:01:14.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:14.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:14.708546+0000) 2024-03-06T09:01:15.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:15.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:15.708809+0000) 2024-03-06T09:01:16.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:16.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:16.709044+0000) 2024-03-06T09:01:16.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:16.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:17.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:17.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:17.709326+0000) 2024-03-06T09:01:18.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:18.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:18.709600+0000) 2024-03-06T09:01:19.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:19.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:19.709906+0000) 2024-03-06T09:01:20.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:20.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:20.710163+0000) 2024-03-06T09:01:21.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:21.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:21.710382+0000) 2024-03-06T09:01:22.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:22.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:22.710721+0000) 2024-03-06T09:01:23.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:23.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:23.710985+0000) 2024-03-06T09:01:24.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:24.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:24.711256+0000) 2024-03-06T09:01:25.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:25.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:25.711552+0000) 2024-03-06T09:01:26.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:26.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:26.711837+0000) 2024-03-06T09:01:27.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:27.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:27.712126+0000) 2024-03-06T09:01:28.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:28.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:28.712401+0000) 2024-03-06T09:01:29.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:29.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:29.712732+0000) 2024-03-06T09:01:30.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:30.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:30.713017+0000) 2024-03-06T09:01:31.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:31.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:31.713290+0000) 2024-03-06T09:01:32.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:32.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:32.713498+0000) 2024-03-06T09:01:33.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:33.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:33.713761+0000) 2024-03-06T09:01:34.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:34.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:34.714041+0000) 2024-03-06T09:01:35.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:35.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:35.714343+0000) 2024-03-06T09:01:36.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:36.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:36.714592+0000) 2024-03-06T09:01:37.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:37.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:37.714972+0000) 2024-03-06T09:01:38.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:38.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:38.715246+0000) 2024-03-06T09:01:39.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:39.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:39.715535+0000) 2024-03-06T09:01:40.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:40.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:40.715811+0000) 2024-03-06T09:01:41.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:41.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:41.716072+0000) 2024-03-06T09:01:42.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:42.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:42.716331+0000) 2024-03-06T09:01:43.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:43.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:43.716607+0000) 2024-03-06T09:01:44.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:44.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:44.716933+0000) 2024-03-06T09:01:45.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:45.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:45.717228+0000) 2024-03-06T09:01:46.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:46.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:46.717502+0000) 2024-03-06T09:01:46.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:46.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:47.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:47.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:47.717746+0000) 2024-03-06T09:01:48.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:48.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:48.717931+0000) 2024-03-06T09:01:49.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:49.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:49.718165+0000) 2024-03-06T09:01:50.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:50.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:50.718722+0000) 2024-03-06T09:01:51.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:51.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:51.719003+0000) 2024-03-06T09:01:52.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:52.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:52.719192+0000) 2024-03-06T09:01:53.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:53.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:53.719425+0000) 2024-03-06T09:01:54.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:54.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:54.719696+0000) 2024-03-06T09:01:55.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:55.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:55.719968+0000) 2024-03-06T09:01:56.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:56.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:56.720205+0000) 2024-03-06T09:01:57.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:57.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:57.720483+0000) 2024-03-06T09:01:58.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:58.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:58.720759+0000) 2024-03-06T09:01:59.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:01:59.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:59.721022+0000) 2024-03-06T09:02:00.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:00.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:00.721274+0000) 2024-03-06T09:02:01.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:01.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:01.721536+0000) 2024-03-06T09:02:02.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:02.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:02.721783+0000) 2024-03-06T09:02:03.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:03.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:03.722044+0000) 2024-03-06T09:02:04.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:04.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:04.722311+0000) 2024-03-06T09:02:05.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:05.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:05.722742+0000) 2024-03-06T09:02:06.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:06.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:06.723027+0000) 2024-03-06T09:02:07.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:07.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:07.723295+0000) 2024-03-06T09:02:08.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:08.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:08.723526+0000) 2024-03-06T09:02:09.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:09.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:09.723763+0000) 2024-03-06T09:02:10.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:10.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:10.724020+0000) 2024-03-06T09:02:11.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:11.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:11.724277+0000) 2024-03-06T09:02:12.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:12.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:12.724526+0000) 2024-03-06T09:02:13.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:13.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:13.724785+0000) 2024-03-06T09:02:14.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:14.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:14.725037+0000) 2024-03-06T09:02:15.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:15.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:15.725307+0000) 2024-03-06T09:02:16.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:16.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:16.725544+0000) 2024-03-06T09:02:17.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:17.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:17.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:17.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:17.725780+0000) 2024-03-06T09:02:18.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:18.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:18.726011+0000) 2024-03-06T09:02:19.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:19.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:19.726250+0000) 2024-03-06T09:02:20.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:20.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:20.726502+0000) 2024-03-06T09:02:21.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:21.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:21.726752+0000) 2024-03-06T09:02:22.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:22.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:22.727004+0000) 2024-03-06T09:02:23.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:23.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:23.727264+0000) 2024-03-06T09:02:24.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:24.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:24.727516+0000) 2024-03-06T09:02:25.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:25.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:25.727789+0000) 2024-03-06T09:02:26.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:26.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:26.728046+0000) 2024-03-06T09:02:27.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:27.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:27.728296+0000) 2024-03-06T09:02:28.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:28.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:28.728547+0000) 2024-03-06T09:02:29.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:29.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:29.728796+0000) 2024-03-06T09:02:30.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:30.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:30.729047+0000) 2024-03-06T09:02:31.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:31.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:31.729307+0000) 2024-03-06T09:02:32.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:32.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:32.729559+0000) 2024-03-06T09:02:33.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:33.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:33.729817+0000) 2024-03-06T09:02:34.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:34.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:34.730066+0000) 2024-03-06T09:02:35.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:35.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:35.730325+0000) 2024-03-06T09:02:36.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:36.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:36.730670+0000) 2024-03-06T09:02:37.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:37.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:37.730918+0000) 2024-03-06T09:02:38.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:38.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:38.731168+0000) 2024-03-06T09:02:39.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:39.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:39.731429+0000) 2024-03-06T09:02:40.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:40.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:40.731742+0000) 2024-03-06T09:02:41.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:41.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:41.732004+0000) 2024-03-06T09:02:42.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:42.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:42.732239+0000) 2024-03-06T09:02:43.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:43.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:43.732506+0000) 2024-03-06T09:02:44.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:44.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:44.732775+0000) 2024-03-06T09:02:45.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:45.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:45.733048+0000) 2024-03-06T09:02:46.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:46.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:46.733298+0000) 2024-03-06T09:02:47.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:47.199 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:47.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:47.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:47.733549+0000) 2024-03-06T09:02:48.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:48.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:48.733791+0000) 2024-03-06T09:02:49.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:49.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:49.734037+0000) 2024-03-06T09:02:50.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:50.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:50.734303+0000) 2024-03-06T09:02:51.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:51.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:51.734565+0000) 2024-03-06T09:02:52.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:52.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:52.734805+0000) 2024-03-06T09:02:53.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:53.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:53.735069+0000) 2024-03-06T09:02:54.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:54.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:54.735320+0000) 2024-03-06T09:02:55.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:55.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:55.735576+0000) 2024-03-06T09:02:56.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:56.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:56.735833+0000) 2024-03-06T09:02:57.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:57.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:57.736028+0000) 2024-03-06T09:02:58.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:58.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:58.736278+0000) 2024-03-06T09:02:59.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:02:59.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:59.736536+0000) 2024-03-06T09:03:00.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:00.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:00.736810+0000) 2024-03-06T09:03:01.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:01.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:01.737036+0000) 2024-03-06T09:03:02.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:02.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:02.737307+0000) 2024-03-06T09:03:03.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:03.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:03.737563+0000) 2024-03-06T09:03:04.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:04.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:04.737797+0000) 2024-03-06T09:03:05.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:05.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:05.738062+0000) 2024-03-06T09:03:06.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:06.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:06.738311+0000) 2024-03-06T09:03:07.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:07.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:07.738545+0000) 2024-03-06T09:03:08.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:08.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:08.738799+0000) 2024-03-06T09:03:09.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:09.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:09.738980+0000) 2024-03-06T09:03:10.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:10.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:10.739229+0000) 2024-03-06T09:03:11.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:11.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:11.739467+0000) 2024-03-06T09:03:12.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:12.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:12.739744+0000) 2024-03-06T09:03:13.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:13.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:13.739987+0000) 2024-03-06T09:03:14.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:14.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:14.740224+0000) 2024-03-06T09:03:15.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:15.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:15.740516+0000) 2024-03-06T09:03:16.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:16.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:16.740750+0000) 2024-03-06T09:03:17.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:17.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:17.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:17.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:17.740958+0000) 2024-03-06T09:03:18.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:18.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:18.741206+0000) 2024-03-06T09:03:19.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:19.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:19.741452+0000) 2024-03-06T09:03:20.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:20.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:20.741750+0000) 2024-03-06T09:03:21.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:21.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:21.742019+0000) 2024-03-06T09:03:22.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:22.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:22.742270+0000) 2024-03-06T09:03:23.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:23.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:23.742526+0000) 2024-03-06T09:03:24.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:24.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:24.742784+0000) 2024-03-06T09:03:25.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:25.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:25.743064+0000) 2024-03-06T09:03:26.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:26.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:26.743315+0000) 2024-03-06T09:03:27.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:27.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:27.743505+0000) 2024-03-06T09:03:28.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:28.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:28.743744+0000) 2024-03-06T09:03:29.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:29.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:29.743993+0000) 2024-03-06T09:03:30.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:30.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:30.744241+0000) 2024-03-06T09:03:31.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:31.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:31.744499+0000) 2024-03-06T09:03:32.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:32.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:32.744753+0000) 2024-03-06T09:03:33.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:33.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:33.745012+0000) 2024-03-06T09:03:34.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:34.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:34.745320+0000) 2024-03-06T09:03:35.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:35.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:35.745592+0000) 2024-03-06T09:03:36.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:36.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:36.745834+0000) 2024-03-06T09:03:37.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:37.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:37.746088+0000) 2024-03-06T09:03:38.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:38.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:38.746336+0000) 2024-03-06T09:03:39.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:39.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:39.746579+0000) 2024-03-06T09:03:40.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:40.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:40.746831+0000) 2024-03-06T09:03:41.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:41.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:41.747103+0000) 2024-03-06T09:03:42.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:42.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:42.747338+0000) 2024-03-06T09:03:43.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:43.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:43.747602+0000) 2024-03-06T09:03:44.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:44.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:44.747900+0000) 2024-03-06T09:03:45.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:45.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:45.748160+0000) 2024-03-06T09:03:46.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:46.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:46.748395+0000) 2024-03-06T09:03:47.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:47.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:47.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:47.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:47.748703+0000) 2024-03-06T09:03:48.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:48.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:48.748956+0000) 2024-03-06T09:03:49.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:49.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:49.749212+0000) 2024-03-06T09:03:50.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:50.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:50.749467+0000) 2024-03-06T09:03:51.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:51.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:51.749741+0000) 2024-03-06T09:03:52.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:52.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:52.750003+0000) 2024-03-06T09:03:53.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:53.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:53.750264+0000) 2024-03-06T09:03:54.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:54.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:54.750512+0000) 2024-03-06T09:03:55.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:55.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:55.750771+0000) 2024-03-06T09:03:56.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:56.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:56.751024+0000) 2024-03-06T09:03:57.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:57.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:57.751262+0000) 2024-03-06T09:03:58.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:58.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:58.751516+0000) 2024-03-06T09:03:59.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:03:59.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:59.751776+0000) 2024-03-06T09:04:00.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:00.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:00.752029+0000) 2024-03-06T09:04:01.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:01.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:01.752305+0000) 2024-03-06T09:04:02.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:02.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:02.752558+0000) 2024-03-06T09:04:03.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:03.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:03.752800+0000) 2024-03-06T09:04:04.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:04.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:04.753045+0000) 2024-03-06T09:04:05.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:05.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:05.753320+0000) 2024-03-06T09:04:06.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:06.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:06.753575+0000) 2024-03-06T09:04:07.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:07.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:07.753831+0000) 2024-03-06T09:04:08.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:08.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:08.754079+0000) 2024-03-06T09:04:09.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:09.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:09.754337+0000) 2024-03-06T09:04:10.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:10.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:10.754574+0000) 2024-03-06T09:04:11.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:11.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:11.754839+0000) 2024-03-06T09:04:12.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:12.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:12.755074+0000) 2024-03-06T09:04:13.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:13.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:13.755331+0000) 2024-03-06T09:04:14.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:14.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:14.755584+0000) 2024-03-06T09:04:15.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:15.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:15.755850+0000) 2024-03-06T09:04:16.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:16.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:16.756035+0000) 2024-03-06T09:04:17.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:17.671 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:17.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:17.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:17.756198+0000) 2024-03-06T09:04:18.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:18.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:18.756413+0000) 2024-03-06T09:04:19.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:19.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:19.756703+0000) 2024-03-06T09:04:20.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:20.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:20.756947+0000) 2024-03-06T09:04:21.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:21.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:21.757192+0000) 2024-03-06T09:04:22.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:22.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:22.757431+0000) 2024-03-06T09:04:23.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:23.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:23.757737+0000) 2024-03-06T09:04:24.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:24.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:24.757981+0000) 2024-03-06T09:04:25.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:25.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:25.758259+0000) 2024-03-06T09:04:26.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:26.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:26.758513+0000) 2024-03-06T09:04:27.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:27.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:27.758774+0000) 2024-03-06T09:04:28.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:28.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:28.759095+0000) 2024-03-06T09:04:29.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:29.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:29.759352+0000) 2024-03-06T09:04:30.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:30.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:30.759608+0000) 2024-03-06T09:04:31.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:31.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:31.759916+0000) 2024-03-06T09:04:32.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:32.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:32.760158+0000) 2024-03-06T09:04:33.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:33.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:33.760421+0000) 2024-03-06T09:04:34.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:34.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:34.760716+0000) 2024-03-06T09:04:35.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:35.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:35.760978+0000) 2024-03-06T09:04:36.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:36.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:36.761241+0000) 2024-03-06T09:04:37.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:37.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:37.761512+0000) 2024-03-06T09:04:38.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:38.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:38.761767+0000) 2024-03-06T09:04:39.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:39.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:39.762012+0000) 2024-03-06T09:04:40.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:40.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:40.762274+0000) 2024-03-06T09:04:41.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:41.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:41.762528+0000) 2024-03-06T09:04:42.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:42.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:42.762812+0000) 2024-03-06T09:04:43.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:43.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:43.763011+0000) 2024-03-06T09:04:44.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:44.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:44.763249+0000) 2024-03-06T09:04:45.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:45.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:45.763525+0000) 2024-03-06T09:04:46.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:46.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:46.763772+0000) 2024-03-06T09:04:47.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:47.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:47.764038+0000) 2024-03-06T09:04:47.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:47.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:48.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:48.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:48.764288+0000) 2024-03-06T09:04:49.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:49.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:49.764553+0000) 2024-03-06T09:04:50.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:50.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:50.764819+0000) 2024-03-06T09:04:51.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:51.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:51.765078+0000) 2024-03-06T09:04:52.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:52.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:52.765317+0000) 2024-03-06T09:04:53.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:53.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:53.765570+0000) 2024-03-06T09:04:54.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:54.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:54.765829+0000) 2024-03-06T09:04:55.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:55.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:55.766107+0000) 2024-03-06T09:04:56.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:56.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:56.766349+0000) 2024-03-06T09:04:57.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:57.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:57.766613+0000) 2024-03-06T09:04:58.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:58.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:58.766907+0000) 2024-03-06T09:04:59.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:04:59.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:59.767178+0000) 2024-03-06T09:05:00.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:00.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:00.767448+0000) 2024-03-06T09:05:01.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:01.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:01.767751+0000) 2024-03-06T09:05:02.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:02.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:02.768008+0000) 2024-03-06T09:05:03.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:03.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:03.768259+0000) 2024-03-06T09:05:04.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:04.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:04.768517+0000) 2024-03-06T09:05:05.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:05.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:05.768796+0000) 2024-03-06T09:05:06.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:06.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:06.768995+0000) 2024-03-06T09:05:07.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:07.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:07.769257+0000) 2024-03-06T09:05:08.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:08.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:08.769508+0000) 2024-03-06T09:05:09.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:09.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:09.769764+0000) 2024-03-06T09:05:10.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:10.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:10.770058+0000) 2024-03-06T09:05:11.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:11.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:11.770324+0000) 2024-03-06T09:05:12.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:12.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:12.770563+0000) 2024-03-06T09:05:13.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:13.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:13.770808+0000) 2024-03-06T09:05:14.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:14.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:14.771058+0000) 2024-03-06T09:05:15.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:15.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:15.771331+0000) 2024-03-06T09:05:16.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:16.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:16.771597+0000) 2024-03-06T09:05:17.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:17.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:17.771870+0000) 2024-03-06T09:05:17.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:17.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:18.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:18.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:18.772136+0000) 2024-03-06T09:05:19.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:19.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:19.772411+0000) 2024-03-06T09:05:20.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:20.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:20.772724+0000) 2024-03-06T09:05:21.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:21.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:21.772990+0000) 2024-03-06T09:05:22.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:22.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:22.773240+0000) 2024-03-06T09:05:23.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:23.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:23.773504+0000) 2024-03-06T09:05:24.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:24.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:24.773760+0000) 2024-03-06T09:05:25.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:25.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:25.774037+0000) 2024-03-06T09:05:26.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:26.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:26.774294+0000) 2024-03-06T09:05:27.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:27.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:27.774550+0000) 2024-03-06T09:05:28.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:28.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:28.774814+0000) 2024-03-06T09:05:29.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:29.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:29.775080+0000) 2024-03-06T09:05:30.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:30.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:30.775329+0000) 2024-03-06T09:05:31.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:31.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:31.775606+0000) 2024-03-06T09:05:32.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:32.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:32.775904+0000) 2024-03-06T09:05:33.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:33.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:33.776151+0000) 2024-03-06T09:05:34.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:34.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:34.776392+0000) 2024-03-06T09:05:35.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:35.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:35.776687+0000) 2024-03-06T09:05:36.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:36.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:36.776953+0000) 2024-03-06T09:05:37.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:37.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:37.777217+0000) 2024-03-06T09:05:38.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:38.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:38.777471+0000) 2024-03-06T09:05:39.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:39.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:39.777756+0000) 2024-03-06T09:05:40.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:40.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:40.778006+0000) 2024-03-06T09:05:41.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:41.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:41.778252+0000) 2024-03-06T09:05:42.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:42.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:42.778501+0000) 2024-03-06T09:05:43.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:43.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:43.778762+0000) 2024-03-06T09:05:44.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:44.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:44.778993+0000) 2024-03-06T09:05:45.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:45.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:45.779257+0000) 2024-03-06T09:05:46.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:46.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:46.779515+0000) 2024-03-06T09:05:47.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:47.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:47.779793+0000) 2024-03-06T09:05:48.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:48.122 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:48.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:48.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:48.779992+0000) 2024-03-06T09:05:49.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:49.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:49.780182+0000) 2024-03-06T09:05:50.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:50.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:50.780422+0000) 2024-03-06T09:05:51.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:51.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:51.780739+0000) 2024-03-06T09:05:52.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:52.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:52.780940+0000) 2024-03-06T09:05:53.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:53.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:53.781156+0000) 2024-03-06T09:05:54.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:54.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:54.781405+0000) 2024-03-06T09:05:55.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:55.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:55.781701+0000) 2024-03-06T09:05:56.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:56.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:56.781960+0000) 2024-03-06T09:05:57.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:57.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:57.782214+0000) 2024-03-06T09:05:58.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:58.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:58.782436+0000) 2024-03-06T09:05:59.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:05:59.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:59.782734+0000) 2024-03-06T09:06:00.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:00.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:00.782994+0000) 2024-03-06T09:06:01.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:01.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:01.783252+0000) 2024-03-06T09:06:02.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:02.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:02.783483+0000) 2024-03-06T09:06:03.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:03.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:03.783760+0000) 2024-03-06T09:06:04.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:04.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:04.784015+0000) 2024-03-06T09:06:05.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:05.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:05.784269+0000) 2024-03-06T09:06:06.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:06.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:06.784521+0000) 2024-03-06T09:06:07.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:07.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:07.784775+0000) 2024-03-06T09:06:08.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:08.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:08.784960+0000) 2024-03-06T09:06:09.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:09.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:09.785262+0000) 2024-03-06T09:06:10.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:10.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:10.785499+0000) 2024-03-06T09:06:11.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:11.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:11.785752+0000) 2024-03-06T09:06:12.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:12.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:12.785997+0000) 2024-03-06T09:06:13.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:13.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:13.786245+0000) 2024-03-06T09:06:14.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:14.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:14.786501+0000) 2024-03-06T09:06:15.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:15.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:15.786773+0000) 2024-03-06T09:06:16.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:16.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:16.787025+0000) 2024-03-06T09:06:17.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:17.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:17.787258+0000) 2024-03-06T09:06:18.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:18.269 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:18.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:18.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:18.787511+0000) 2024-03-06T09:06:19.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:19.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:19.787768+0000) 2024-03-06T09:06:20.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:20.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:20.787957+0000) 2024-03-06T09:06:21.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:21.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:21.788221+0000) 2024-03-06T09:06:22.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:22.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:22.788473+0000) 2024-03-06T09:06:23.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:23.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:23.788755+0000) 2024-03-06T09:06:24.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:24.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:24.789005+0000) 2024-03-06T09:06:25.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:25.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:25.789279+0000) 2024-03-06T09:06:26.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:26.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:26.789535+0000) 2024-03-06T09:06:27.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:27.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:27.789789+0000) 2024-03-06T09:06:28.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:28.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:28.790043+0000) 2024-03-06T09:06:29.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:29.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:29.790288+0000) 2024-03-06T09:06:30.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:30.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:30.790545+0000) 2024-03-06T09:06:31.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:31.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:31.790810+0000) 2024-03-06T09:06:32.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:32.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:32.790988+0000) 2024-03-06T09:06:33.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:33.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:33.791245+0000) 2024-03-06T09:06:34.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:34.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:34.791493+0000) 2024-03-06T09:06:35.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:35.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:35.791750+0000) 2024-03-06T09:06:36.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:36.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:36.791990+0000) 2024-03-06T09:06:37.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:37.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:37.792224+0000) 2024-03-06T09:06:38.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:38.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:38.792456+0000) 2024-03-06T09:06:39.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:39.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:39.792751+0000) 2024-03-06T09:06:40.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:40.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:40.792990+0000) 2024-03-06T09:06:41.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:41.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:41.793263+0000) 2024-03-06T09:06:42.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:42.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:42.793501+0000) 2024-03-06T09:06:43.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:43.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:43.793755+0000) 2024-03-06T09:06:44.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:44.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:44.794007+0000) 2024-03-06T09:06:45.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:45.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:45.794282+0000) 2024-03-06T09:06:46.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:46.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:46.794520+0000) 2024-03-06T09:06:47.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:47.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:47.794779+0000) 2024-03-06T09:06:48.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:48.416 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:48.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:48.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:48.795015+0000) 2024-03-06T09:06:49.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:49.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:49.795258+0000) 2024-03-06T09:06:50.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:50.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:50.795507+0000) 2024-03-06T09:06:51.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:51.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:51.795783+0000) 2024-03-06T09:06:52.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:52.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:52.796039+0000) 2024-03-06T09:06:53.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:53.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:53.796296+0000) 2024-03-06T09:06:54.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:54.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:54.796552+0000) 2024-03-06T09:06:55.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:55.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:55.796825+0000) 2024-03-06T09:06:56.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:56.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:56.797090+0000) 2024-03-06T09:06:57.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:57.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:57.797354+0000) 2024-03-06T09:06:58.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:58.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:58.797601+0000) 2024-03-06T09:06:59.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:06:59.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:59.797893+0000) 2024-03-06T09:07:00.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:00.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:00.798129+0000) 2024-03-06T09:07:01.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:01.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:01.798425+0000) 2024-03-06T09:07:02.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:02.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:02.798747+0000) 2024-03-06T09:07:03.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:03.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:03.799016+0000) 2024-03-06T09:07:04.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:04.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:04.799301+0000) 2024-03-06T09:07:05.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:05.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:05.799583+0000) 2024-03-06T09:07:06.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:06.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:06.799815+0000) 2024-03-06T09:07:07.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:07.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:07.800062+0000) 2024-03-06T09:07:08.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:08.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:08.800294+0000) 2024-03-06T09:07:09.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:09.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:09.800521+0000) 2024-03-06T09:07:10.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:10.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:10.800785+0000) 2024-03-06T09:07:11.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:11.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:11.801029+0000) 2024-03-06T09:07:12.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:12.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:12.801271+0000) 2024-03-06T09:07:13.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:13.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:13.801518+0000) 2024-03-06T09:07:14.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:14.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:14.801743+0000) 2024-03-06T09:07:15.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:15.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:15.801952+0000) 2024-03-06T09:07:16.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:16.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:16.802183+0000) 2024-03-06T09:07:17.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:17.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:17.802411+0000) 2024-03-06T09:07:18.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:18.572 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:18.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:18.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:18.802687+0000) 2024-03-06T09:07:19.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:19.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:19.802925+0000) 2024-03-06T09:07:20.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:20.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:20.803105+0000) 2024-03-06T09:07:21.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:21.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:21.803308+0000) 2024-03-06T09:07:22.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:22.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:22.803507+0000) 2024-03-06T09:07:23.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:23.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:23.803751+0000) 2024-03-06T09:07:24.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:24.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:24.803952+0000) 2024-03-06T09:07:25.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:25.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:25.804174+0000) 2024-03-06T09:07:26.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:26.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:26.804427+0000) 2024-03-06T09:07:27.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:27.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:27.804665+0000) 2024-03-06T09:07:28.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:28.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:28.804866+0000) 2024-03-06T09:07:29.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:29.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:29.805067+0000) 2024-03-06T09:07:30.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:30.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:30.805330+0000) 2024-03-06T09:07:31.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:31.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:31.805585+0000) 2024-03-06T09:07:32.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:32.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:32.805868+0000) 2024-03-06T09:07:33.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:33.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:33.806078+0000) 2024-03-06T09:07:34.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:34.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:34.806343+0000) 2024-03-06T09:07:35.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:35.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:35.806619+0000) 2024-03-06T09:07:36.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:36.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:36.806894+0000) 2024-03-06T09:07:37.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:37.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:37.807101+0000) 2024-03-06T09:07:38.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:38.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:38.807316+0000) 2024-03-06T09:07:39.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:39.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:39.807575+0000) 2024-03-06T09:07:40.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:40.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:40.807839+0000) 2024-03-06T09:07:41.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:41.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:41.808061+0000) 2024-03-06T09:07:42.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:42.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:42.808325+0000) 2024-03-06T09:07:43.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:43.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:43.808539+0000) 2024-03-06T09:07:44.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:44.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:44.808797+0000) 2024-03-06T09:07:45.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:45.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:45.809092+0000) 2024-03-06T09:07:46.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:46.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:46.809352+0000) 2024-03-06T09:07:47.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:47.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:47.809669+0000) 2024-03-06T09:07:48.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:48.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:48.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:48.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:48.809859+0000) 2024-03-06T09:07:49.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:49.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:49.810105+0000) 2024-03-06T09:07:50.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:50.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:50.810397+0000) 2024-03-06T09:07:51.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:51.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:51.810735+0000) 2024-03-06T09:07:52.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:52.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:52.811014+0000) 2024-03-06T09:07:53.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:53.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:53.811297+0000) 2024-03-06T09:07:54.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:54.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:54.811548+0000) 2024-03-06T09:07:55.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:55.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:55.811829+0000) 2024-03-06T09:07:56.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:56.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:56.812058+0000) 2024-03-06T09:07:57.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:57.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:57.812337+0000) 2024-03-06T09:07:58.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:58.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:58.812606+0000) 2024-03-06T09:07:59.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:07:59.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:59.812935+0000) 2024-03-06T09:08:00.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:00.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:00.813229+0000) 2024-03-06T09:08:01.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:01.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:01.813513+0000) 2024-03-06T09:08:02.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:02.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:02.813784+0000) 2024-03-06T09:08:03.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:03.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:03.814049+0000) 2024-03-06T09:08:04.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:04.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:04.814335+0000) 2024-03-06T09:08:05.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:05.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:05.814625+0000) 2024-03-06T09:08:06.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:06.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:06.814928+0000) 2024-03-06T09:08:07.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:07.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:07.815194+0000) 2024-03-06T09:08:08.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:08.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:08.815444+0000) 2024-03-06T09:08:09.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:09.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:09.815763+0000) 2024-03-06T09:08:10.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:10.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:10.816040+0000) 2024-03-06T09:08:11.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:11.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:11.816305+0000) 2024-03-06T09:08:12.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:12.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:12.816613+0000) 2024-03-06T09:08:13.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:13.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:13.816925+0000) 2024-03-06T09:08:14.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:14.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:14.817198+0000) 2024-03-06T09:08:15.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:15.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:15.817473+0000) 2024-03-06T09:08:16.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:16.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:16.817736+0000) 2024-03-06T09:08:17.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:17.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:17.818003+0000) 2024-03-06T09:08:18.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:18.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:18.818259+0000) 2024-03-06T09:08:18.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:18.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:19.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:19.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:19.818525+0000) 2024-03-06T09:08:20.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:20.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:20.818787+0000) 2024-03-06T09:08:21.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:21.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:21.819031+0000) 2024-03-06T09:08:22.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:22.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:22.819290+0000) 2024-03-06T09:08:23.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:23.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:23.819485+0000) 2024-03-06T09:08:24.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:24.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:24.819786+0000) 2024-03-06T09:08:25.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:25.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:25.819993+0000) 2024-03-06T09:08:26.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:26.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:26.820189+0000) 2024-03-06T09:08:27.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:27.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:27.820379+0000) 2024-03-06T09:08:28.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:28.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:28.820689+0000) 2024-03-06T09:08:29.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:29.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:29.820925+0000) 2024-03-06T09:08:30.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:30.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:30.821168+0000) 2024-03-06T09:08:31.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:31.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:31.821417+0000) 2024-03-06T09:08:32.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:32.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:32.821730+0000) 2024-03-06T09:08:33.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:33.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:33.821934+0000) 2024-03-06T09:08:34.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:34.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:34.822198+0000) 2024-03-06T09:08:35.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:35.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:35.822462+0000) 2024-03-06T09:08:36.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:36.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:36.822740+0000) 2024-03-06T09:08:37.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:37.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:37.822987+0000) 2024-03-06T09:08:38.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:38.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:38.823243+0000) 2024-03-06T09:08:39.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:39.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:39.823501+0000) 2024-03-06T09:08:40.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:40.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:40.823751+0000) 2024-03-06T09:08:41.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:41.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:41.824022+0000) 2024-03-06T09:08:42.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:42.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:42.824304+0000) 2024-03-06T09:08:43.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:43.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:43.824583+0000) 2024-03-06T09:08:44.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:44.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:44.824893+0000) 2024-03-06T09:08:45.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:45.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:45.825217+0000) 2024-03-06T09:08:46.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:46.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:46.825502+0000) 2024-03-06T09:08:47.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:47.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:47.825755+0000) 2024-03-06T09:08:48.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:48.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:48.826036+0000) 2024-03-06T09:08:49.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:49.051 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:49.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:49.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:49.826292+0000) 2024-03-06T09:08:50.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:50.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:50.826570+0000) 2024-03-06T09:08:51.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:51.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:51.826830+0000) 2024-03-06T09:08:52.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:52.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:52.827105+0000) 2024-03-06T09:08:53.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:53.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:53.827393+0000) 2024-03-06T09:08:54.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:54.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:54.827672+0000) 2024-03-06T09:08:55.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:55.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:55.827989+0000) 2024-03-06T09:08:56.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:56.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:56.828271+0000) 2024-03-06T09:08:57.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:57.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:57.828536+0000) 2024-03-06T09:08:58.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:58.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:58.828798+0000) 2024-03-06T09:08:59.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:08:59.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:59.829065+0000) 2024-03-06T09:09:00.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:00.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:00.829333+0000) 2024-03-06T09:09:01.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:01.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:01.829558+0000) 2024-03-06T09:09:02.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:02.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:02.829843+0000) 2024-03-06T09:09:03.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:03.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:03.830105+0000) 2024-03-06T09:09:04.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:04.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:04.830366+0000) 2024-03-06T09:09:05.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:05.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:05.830757+0000) 2024-03-06T09:09:06.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:06.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:06.831034+0000) 2024-03-06T09:09:07.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:07.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:07.831245+0000) 2024-03-06T09:09:08.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:08.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:08.831523+0000) 2024-03-06T09:09:09.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:09.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:09.831791+0000) 2024-03-06T09:09:10.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:10.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:10.832070+0000) 2024-03-06T09:09:11.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:11.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:11.832363+0000) 2024-03-06T09:09:12.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:12.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:12.832674+0000) 2024-03-06T09:09:13.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:13.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:13.832959+0000) 2024-03-06T09:09:14.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:14.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:14.833163+0000) 2024-03-06T09:09:15.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:15.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:15.833421+0000) 2024-03-06T09:09:16.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:16.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:16.833724+0000) 2024-03-06T09:09:17.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:17.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:17.833983+0000) 2024-03-06T09:09:18.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:18.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:18.834225+0000) 2024-03-06T09:09:19.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:19.203 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:19.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:19.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:19.834523+0000) 2024-03-06T09:09:20.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:20.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:20.834770+0000) 2024-03-06T09:09:21.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:21.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:21.835030+0000) 2024-03-06T09:09:22.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:22.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:22.835285+0000) 2024-03-06T09:09:23.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:23.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:23.835547+0000) 2024-03-06T09:09:24.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:24.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:24.835803+0000) 2024-03-06T09:09:25.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:25.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:25.836071+0000) 2024-03-06T09:09:26.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:26.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:26.836330+0000) 2024-03-06T09:09:27.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:27.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:27.836574+0000) 2024-03-06T09:09:28.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:28.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:28.836829+0000) 2024-03-06T09:09:29.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:29.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:29.837082+0000) 2024-03-06T09:09:30.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:30.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:30.837341+0000) 2024-03-06T09:09:31.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:31.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:31.837594+0000) 2024-03-06T09:09:32.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:32.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:32.837833+0000) 2024-03-06T09:09:33.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:33.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:33.838085+0000) 2024-03-06T09:09:34.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:34.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:34.838336+0000) 2024-03-06T09:09:35.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:35.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:35.838607+0000) 2024-03-06T09:09:36.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:36.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:36.838910+0000) 2024-03-06T09:09:37.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:37.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:37.839177+0000) 2024-03-06T09:09:38.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:38.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:38.839418+0000) 2024-03-06T09:09:39.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:39.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:39.839704+0000) 2024-03-06T09:09:40.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:40.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:40.839962+0000) 2024-03-06T09:09:41.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:41.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:41.840247+0000) 2024-03-06T09:09:42.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:42.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:42.840499+0000) 2024-03-06T09:09:43.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:43.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:43.840757+0000) 2024-03-06T09:09:44.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:44.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:44.840948+0000) 2024-03-06T09:09:45.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:45.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:45.841150+0000) 2024-03-06T09:09:46.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:46.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:46.841418+0000) 2024-03-06T09:09:47.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:47.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:47.841686+0000) 2024-03-06T09:09:48.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:48.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:48.841943+0000) 2024-03-06T09:09:49.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:49.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:49.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:49.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:49.842211+0000) 2024-03-06T09:09:50.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:50.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:50.842463+0000) 2024-03-06T09:09:51.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:51.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:51.842759+0000) 2024-03-06T09:09:52.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:52.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:52.843011+0000) 2024-03-06T09:09:53.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:53.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:53.843279+0000) 2024-03-06T09:09:54.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:54.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:54.843531+0000) 2024-03-06T09:09:55.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:55.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:55.843794+0000) 2024-03-06T09:09:56.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:56.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:56.844047+0000) 2024-03-06T09:09:57.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:57.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:57.844285+0000) 2024-03-06T09:09:58.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:58.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:58.844590+0000) 2024-03-06T09:09:59.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:09:59.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:59.844833+0000) 2024-03-06T09:10:00.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:00.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:00.845068+0000) 2024-03-06T09:10:01.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:01.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:01.845328+0000) 2024-03-06T09:10:02.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:02.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:02.845566+0000) 2024-03-06T09:10:03.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:03.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:03.845814+0000) 2024-03-06T09:10:04.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:04.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:04.846067+0000) 2024-03-06T09:10:05.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:05.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:05.846326+0000) 2024-03-06T09:10:06.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:06.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:06.846577+0000) 2024-03-06T09:10:07.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:07.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:07.846834+0000) 2024-03-06T09:10:08.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:08.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:08.847091+0000) 2024-03-06T09:10:09.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:09.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:09.847352+0000) 2024-03-06T09:10:10.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:10.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:10.847583+0000) 2024-03-06T09:10:11.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:11.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:11.847851+0000) 2024-03-06T09:10:12.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:12.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:12.848111+0000) 2024-03-06T09:10:13.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:13.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:13.848306+0000) 2024-03-06T09:10:14.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:14.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:14.848543+0000) 2024-03-06T09:10:15.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:15.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:15.848808+0000) 2024-03-06T09:10:16.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:16.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:16.849057+0000) 2024-03-06T09:10:17.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:17.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:17.849254+0000) 2024-03-06T09:10:18.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:18.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:18.849494+0000) 2024-03-06T09:10:19.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:19.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:19.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:19.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:19.849770+0000) 2024-03-06T09:10:20.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:20.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:20.850006+0000) 2024-03-06T09:10:21.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:21.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:21.850215+0000) 2024-03-06T09:10:22.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:22.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:22.850454+0000) 2024-03-06T09:10:23.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:23.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:23.850750+0000) 2024-03-06T09:10:24.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:24.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:24.851001+0000) 2024-03-06T09:10:25.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:25.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:25.851253+0000) 2024-03-06T09:10:26.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:26.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:26.851515+0000) 2024-03-06T09:10:27.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:27.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:27.851764+0000) 2024-03-06T09:10:28.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:28.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:28.852017+0000) 2024-03-06T09:10:29.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:29.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:29.852270+0000) 2024-03-06T09:10:30.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:30.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:30.852504+0000) 2024-03-06T09:10:31.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:31.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:31.852763+0000) 2024-03-06T09:10:32.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:32.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:32.853009+0000) 2024-03-06T09:10:33.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:33.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:33.853256+0000) 2024-03-06T09:10:34.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:34.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:34.853512+0000) 2024-03-06T09:10:35.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:35.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:35.853778+0000) 2024-03-06T09:10:36.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:36.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:36.854033+0000) 2024-03-06T09:10:37.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:37.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:37.854294+0000) 2024-03-06T09:10:38.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:38.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:38.854525+0000) 2024-03-06T09:10:39.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:39.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:39.854785+0000) 2024-03-06T09:10:40.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:40.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:40.855017+0000) 2024-03-06T09:10:41.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:41.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:41.855271+0000) 2024-03-06T09:10:42.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:42.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:42.855529+0000) 2024-03-06T09:10:43.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:43.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:43.855786+0000) 2024-03-06T09:10:44.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:44.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:44.856042+0000) 2024-03-06T09:10:45.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:45.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:45.856306+0000) 2024-03-06T09:10:46.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:46.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:46.856564+0000) 2024-03-06T09:10:47.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:47.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:47.856819+0000) 2024-03-06T09:10:48.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:48.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:48.857072+0000) 2024-03-06T09:10:49.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:49.679 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:49.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:49.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:49.857344+0000) 2024-03-06T09:10:50.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:50.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:50.857596+0000) 2024-03-06T09:10:51.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:51.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:51.857849+0000) 2024-03-06T09:10:52.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:52.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:52.858148+0000) 2024-03-06T09:10:53.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:53.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:53.858409+0000) 2024-03-06T09:10:54.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:54.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:54.858671+0000) 2024-03-06T09:10:55.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:55.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:55.858882+0000) 2024-03-06T09:10:56.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:56.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:56.859126+0000) 2024-03-06T09:10:57.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:57.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:57.859392+0000) 2024-03-06T09:10:58.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:58.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:58.859657+0000) 2024-03-06T09:10:59.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:10:59.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:59.859900+0000) 2024-03-06T09:11:00.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:00.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:00.860127+0000) 2024-03-06T09:11:01.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:01.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:01.860388+0000) 2024-03-06T09:11:02.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:02.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:02.860666+0000) 2024-03-06T09:11:03.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:03.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:03.860930+0000) 2024-03-06T09:11:04.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:04.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:04.861181+0000) 2024-03-06T09:11:05.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:05.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:05.861427+0000) 2024-03-06T09:11:06.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:06.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:06.861714+0000) 2024-03-06T09:11:07.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:07.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:07.861962+0000) 2024-03-06T09:11:08.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:08.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:08.862217+0000) 2024-03-06T09:11:09.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:09.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:09.862473+0000) 2024-03-06T09:11:10.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:10.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:10.862747+0000) 2024-03-06T09:11:11.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:11.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:11.862998+0000) 2024-03-06T09:11:12.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:12.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:12.863252+0000) 2024-03-06T09:11:13.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:13.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:13.863499+0000) 2024-03-06T09:11:14.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:14.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:14.863752+0000) 2024-03-06T09:11:15.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:15.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:15.863998+0000) 2024-03-06T09:11:16.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:16.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:16.864252+0000) 2024-03-06T09:11:17.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:17.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:17.864503+0000) 2024-03-06T09:11:18.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:18.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:18.864758+0000) 2024-03-06T09:11:19.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:19.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:19.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:19.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:19.864945+0000) 2024-03-06T09:11:20.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:20.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:20.865215+0000) 2024-03-06T09:11:21.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:21.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:21.865543+0000) 2024-03-06T09:11:22.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:22.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:22.865817+0000) 2024-03-06T09:11:23.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:23.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:23.866066+0000) 2024-03-06T09:11:24.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:24.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:24.866305+0000) 2024-03-06T09:11:25.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:25.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:25.866569+0000) 2024-03-06T09:11:26.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:26.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:26.866827+0000) 2024-03-06T09:11:27.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:27.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:27.867084+0000) 2024-03-06T09:11:28.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:28.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:28.867314+0000) 2024-03-06T09:11:29.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:29.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:29.867554+0000) 2024-03-06T09:11:30.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:30.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:30.867796+0000) 2024-03-06T09:11:31.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:31.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:31.868046+0000) 2024-03-06T09:11:32.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:32.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:32.868308+0000) 2024-03-06T09:11:33.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:33.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:33.868565+0000) 2024-03-06T09:11:34.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:34.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:34.868815+0000) 2024-03-06T09:11:35.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:35.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:35.869088+0000) 2024-03-06T09:11:36.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:36.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:36.869337+0000) 2024-03-06T09:11:37.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:37.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:37.869623+0000) 2024-03-06T09:11:38.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:38.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:38.869939+0000) 2024-03-06T09:11:39.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:39.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:39.870215+0000) 2024-03-06T09:11:40.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:40.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:40.870481+0000) 2024-03-06T09:11:41.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:41.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:41.870756+0000) 2024-03-06T09:11:42.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:42.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:42.870973+0000) 2024-03-06T09:11:43.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:43.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:43.871248+0000) 2024-03-06T09:11:44.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:44.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:44.871445+0000) 2024-03-06T09:11:45.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:45.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:45.871702+0000) 2024-03-06T09:11:46.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:46.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:46.871908+0000) 2024-03-06T09:11:47.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:47.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:47.872148+0000) 2024-03-06T09:11:48.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:48.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:48.872389+0000) 2024-03-06T09:11:49.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:49.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:49.872663+0000) 2024-03-06T09:11:49.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:49.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:50.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:50.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:50.872957+0000) 2024-03-06T09:11:51.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:51.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:51.873215+0000) 2024-03-06T09:11:52.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:52.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:52.873486+0000) 2024-03-06T09:11:53.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:53.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:53.873748+0000) 2024-03-06T09:11:54.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:54.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:54.873999+0000) 2024-03-06T09:11:55.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:55.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:55.874263+0000) 2024-03-06T09:11:56.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:56.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:56.874510+0000) 2024-03-06T09:11:57.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:57.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:57.874766+0000) 2024-03-06T09:11:58.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:58.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:58.875000+0000) 2024-03-06T09:11:59.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:11:59.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:59.875248+0000) 2024-03-06T09:12:00.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:00.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:00.875488+0000) 2024-03-06T09:12:01.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:01.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:01.875744+0000) 2024-03-06T09:12:02.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:02.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:02.876011+0000) 2024-03-06T09:12:03.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:03.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:03.876254+0000) 2024-03-06T09:12:04.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:04.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:04.876508+0000) 2024-03-06T09:12:05.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:05.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:05.876775+0000) 2024-03-06T09:12:06.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:06.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:06.877009+0000) 2024-03-06T09:12:07.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:07.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:07.877284+0000) 2024-03-06T09:12:08.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:08.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:08.877539+0000) 2024-03-06T09:12:09.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:09.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:09.877795+0000) 2024-03-06T09:12:10.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:10.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:10.878053+0000) 2024-03-06T09:12:11.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:11.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:11.878316+0000) 2024-03-06T09:12:12.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:12.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:12.878590+0000) 2024-03-06T09:12:13.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:13.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:13.878857+0000) 2024-03-06T09:12:14.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:14.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:14.879117+0000) 2024-03-06T09:12:15.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:15.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:15.879370+0000) 2024-03-06T09:12:16.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:16.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:16.879660+0000) 2024-03-06T09:12:17.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:17.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:17.879928+0000) 2024-03-06T09:12:18.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:18.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:18.880168+0000) 2024-03-06T09:12:19.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:19.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:19.880431+0000) 2024-03-06T09:12:20.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:20.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:20.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:20.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:20.880727+0000) 2024-03-06T09:12:21.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:21.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:21.880974+0000) 2024-03-06T09:12:22.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:22.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:22.881243+0000) 2024-03-06T09:12:23.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:23.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:23.881506+0000) 2024-03-06T09:12:24.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:24.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:24.881767+0000) 2024-03-06T09:12:25.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:25.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:25.882022+0000) 2024-03-06T09:12:26.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:26.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:26.882259+0000) 2024-03-06T09:12:27.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:27.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:27.882512+0000) 2024-03-06T09:12:28.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:28.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:28.882762+0000) 2024-03-06T09:12:29.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:29.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:29.882957+0000) 2024-03-06T09:12:30.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:30.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:30.883218+0000) 2024-03-06T09:12:31.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:31.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:31.883411+0000) 2024-03-06T09:12:32.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:32.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:32.883706+0000) 2024-03-06T09:12:33.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:33.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:33.883982+0000) 2024-03-06T09:12:34.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:34.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:34.884238+0000) 2024-03-06T09:12:35.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:35.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:35.884505+0000) 2024-03-06T09:12:36.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:36.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:36.884788+0000) 2024-03-06T09:12:37.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:37.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:37.885056+0000) 2024-03-06T09:12:38.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:38.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:38.885364+0000) 2024-03-06T09:12:39.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:39.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:39.885615+0000) 2024-03-06T09:12:40.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:40.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:40.885888+0000) 2024-03-06T09:12:41.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:41.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:41.886128+0000) 2024-03-06T09:12:42.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:42.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:42.886342+0000) 2024-03-06T09:12:43.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:43.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:43.886610+0000) 2024-03-06T09:12:44.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:44.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:44.886903+0000) 2024-03-06T09:12:45.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:45.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:45.887173+0000) 2024-03-06T09:12:46.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:46.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:46.887428+0000) 2024-03-06T09:12:47.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:47.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:47.887739+0000) 2024-03-06T09:12:48.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:48.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:48.888004+0000) 2024-03-06T09:12:49.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:49.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:49.888246+0000) 2024-03-06T09:12:50.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:50.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:50.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:50.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:50.888509+0000) 2024-03-06T09:12:51.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:51.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:51.888756+0000) 2024-03-06T09:12:52.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:52.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:52.889013+0000) 2024-03-06T09:12:53.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:53.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:53.889248+0000) 2024-03-06T09:12:54.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:54.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:54.889529+0000) 2024-03-06T09:12:55.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:55.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:55.889768+0000) 2024-03-06T09:12:56.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:56.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:56.889959+0000) 2024-03-06T09:12:57.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:57.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:57.890212+0000) 2024-03-06T09:12:58.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:58.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:58.890468+0000) 2024-03-06T09:12:59.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:12:59.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:59.890746+0000) 2024-03-06T09:13:00.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:00.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:00.891002+0000) 2024-03-06T09:13:01.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:01.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:01.891255+0000) 2024-03-06T09:13:02.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:02.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:02.891496+0000) 2024-03-06T09:13:03.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:03.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:03.891763+0000) 2024-03-06T09:13:04.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:04.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:04.892002+0000) 2024-03-06T09:13:05.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:05.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:05.892245+0000) 2024-03-06T09:13:06.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:06.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:06.892479+0000) 2024-03-06T09:13:07.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:07.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:07.892750+0000) 2024-03-06T09:13:08.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:08.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:08.892945+0000) 2024-03-06T09:13:09.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:09.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:09.893176+0000) 2024-03-06T09:13:10.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:10.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:10.893428+0000) 2024-03-06T09:13:11.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:11.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:11.893713+0000) 2024-03-06T09:13:12.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:12.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:12.893945+0000) 2024-03-06T09:13:13.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:13.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:13.894183+0000) 2024-03-06T09:13:14.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:14.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:14.894429+0000) 2024-03-06T09:13:15.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:15.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:15.894691+0000) 2024-03-06T09:13:16.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:16.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:16.894953+0000) 2024-03-06T09:13:17.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:17.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:17.895208+0000) 2024-03-06T09:13:18.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:18.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:18.895461+0000) 2024-03-06T09:13:19.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:19.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:19.895741+0000) 2024-03-06T09:13:20.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:20.415 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:20.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:20.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:20.895975+0000) 2024-03-06T09:13:21.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:21.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:21.896240+0000) 2024-03-06T09:13:22.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:22.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:22.896475+0000) 2024-03-06T09:13:23.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:23.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:23.896735+0000) 2024-03-06T09:13:24.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:24.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:24.896977+0000) 2024-03-06T09:13:25.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:25.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:25.897222+0000) 2024-03-06T09:13:26.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:26.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:26.897452+0000) 2024-03-06T09:13:27.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:27.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:27.897744+0000) 2024-03-06T09:13:28.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:28.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:28.897985+0000) 2024-03-06T09:13:29.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:29.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:29.898209+0000) 2024-03-06T09:13:30.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:30.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:30.898457+0000) 2024-03-06T09:13:31.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:31.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:31.898732+0000) 2024-03-06T09:13:32.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:32.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:32.898970+0000) 2024-03-06T09:13:33.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:33.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:33.899226+0000) 2024-03-06T09:13:34.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:34.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:34.899454+0000) 2024-03-06T09:13:35.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:35.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:35.899753+0000) 2024-03-06T09:13:36.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:36.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:36.900006+0000) 2024-03-06T09:13:37.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:37.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:37.900303+0000) 2024-03-06T09:13:38.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:38.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:38.900553+0000) 2024-03-06T09:13:39.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:39.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:39.900805+0000) 2024-03-06T09:13:40.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:40.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:40.901038+0000) 2024-03-06T09:13:41.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:41.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:41.901295+0000) 2024-03-06T09:13:42.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:42.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:42.901555+0000) 2024-03-06T09:13:43.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:43.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:43.901795+0000) 2024-03-06T09:13:44.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:44.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:44.901981+0000) 2024-03-06T09:13:45.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:45.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:45.902247+0000) 2024-03-06T09:13:46.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:46.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:46.902500+0000) 2024-03-06T09:13:47.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:47.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:47.902765+0000) 2024-03-06T09:13:48.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:48.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:48.903024+0000) 2024-03-06T09:13:49.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:49.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:49.903261+0000) 2024-03-06T09:13:50.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:50.571 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:50.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:50.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:50.903516+0000) 2024-03-06T09:13:51.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:51.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:51.903775+0000) 2024-03-06T09:13:52.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:52.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:52.904023+0000) 2024-03-06T09:13:53.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:53.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:53.904291+0000) 2024-03-06T09:13:54.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:54.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:54.904516+0000) 2024-03-06T09:13:55.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:55.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:55.904767+0000) 2024-03-06T09:13:56.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:56.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:56.905009+0000) 2024-03-06T09:13:57.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:57.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:57.905269+0000) 2024-03-06T09:13:58.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:58.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:58.905524+0000) 2024-03-06T09:13:59.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:13:59.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:59.905780+0000) 2024-03-06T09:14:00.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:00.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:00.906035+0000) 2024-03-06T09:14:01.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:01.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:01.906293+0000) 2024-03-06T09:14:02.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:02.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:02.906548+0000) 2024-03-06T09:14:03.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:03.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:03.906789+0000) 2024-03-06T09:14:04.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:04.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:04.907044+0000) 2024-03-06T09:14:05.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:05.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:05.907321+0000) 2024-03-06T09:14:06.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:06.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:06.907586+0000) 2024-03-06T09:14:07.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:07.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:07.907899+0000) 2024-03-06T09:14:08.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:08.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:08.908155+0000) 2024-03-06T09:14:09.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:09.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:09.908430+0000) 2024-03-06T09:14:10.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:10.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:10.908737+0000) 2024-03-06T09:14:11.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:11.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:11.908948+0000) 2024-03-06T09:14:12.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:12.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:12.909219+0000) 2024-03-06T09:14:13.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:13.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:13.909475+0000) 2024-03-06T09:14:14.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:14.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:14.909751+0000) 2024-03-06T09:14:15.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:15.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:15.910051+0000) 2024-03-06T09:14:16.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:16.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:16.910300+0000) 2024-03-06T09:14:17.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:17.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:17.910593+0000) 2024-03-06T09:14:18.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:18.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:18.910900+0000) 2024-03-06T09:14:19.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:19.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:19.911112+0000) 2024-03-06T09:14:20.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:20.718 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:20.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:20.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:20.911407+0000) 2024-03-06T09:14:21.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:21.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:21.911716+0000) 2024-03-06T09:14:22.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:22.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:22.911931+0000) 2024-03-06T09:14:23.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:23.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:23.912194+0000) 2024-03-06T09:14:24.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:24.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:24.912451+0000) 2024-03-06T09:14:25.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:25.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:25.912756+0000) 2024-03-06T09:14:26.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:26.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:26.913021+0000) 2024-03-06T09:14:27.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:27.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:27.913305+0000) 2024-03-06T09:14:28.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:28.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:28.913577+0000) 2024-03-06T09:14:29.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:29.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:29.913862+0000) 2024-03-06T09:14:30.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:30.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:30.914128+0000) 2024-03-06T09:14:31.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:31.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:31.914408+0000) 2024-03-06T09:14:32.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:32.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:32.914714+0000) 2024-03-06T09:14:33.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:33.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:33.915003+0000) 2024-03-06T09:14:34.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:34.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:34.915198+0000) 2024-03-06T09:14:35.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:35.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:35.915463+0000) 2024-03-06T09:14:36.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:36.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:36.915791+0000) 2024-03-06T09:14:37.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:37.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:37.916008+0000) 2024-03-06T09:14:38.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:38.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:38.916206+0000) 2024-03-06T09:14:39.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:39.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:39.916412+0000) 2024-03-06T09:14:40.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:40.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:40.916702+0000) 2024-03-06T09:14:41.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:41.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:41.916976+0000) 2024-03-06T09:14:42.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:42.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:42.917236+0000) 2024-03-06T09:14:43.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:43.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:43.917514+0000) 2024-03-06T09:14:44.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:44.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:44.917780+0000) 2024-03-06T09:14:45.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:45.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:45.918068+0000) 2024-03-06T09:14:46.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:46.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:46.918342+0000) 2024-03-06T09:14:47.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:47.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:47.918623+0000) 2024-03-06T09:14:48.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:48.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:48.918959+0000) 2024-03-06T09:14:49.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:49.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:49.919159+0000) 2024-03-06T09:14:50.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:50.865 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:50.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:50.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:50.919390+0000) 2024-03-06T09:14:51.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:51.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:51.919668+0000) 2024-03-06T09:14:52.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:52.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:52.919931+0000) 2024-03-06T09:14:53.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:53.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:53.920176+0000) 2024-03-06T09:14:54.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:54.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:54.920418+0000) 2024-03-06T09:14:55.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:55.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:55.920705+0000) 2024-03-06T09:14:56.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:56.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:56.920948+0000) 2024-03-06T09:14:57.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:57.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:57.921227+0000) 2024-03-06T09:14:58.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:58.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:58.921457+0000) 2024-03-06T09:14:59.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:14:59.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:59.921755+0000) 2024-03-06T09:15:00.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:00.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:00.922010+0000) 2024-03-06T09:15:01.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:01.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:01.922264+0000) 2024-03-06T09:15:02.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:02.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:02.922529+0000) 2024-03-06T09:15:03.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:03.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:03.922823+0000) 2024-03-06T09:15:04.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:04.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:04.923048+0000) 2024-03-06T09:15:05.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:05.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:05.923294+0000) 2024-03-06T09:15:06.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:06.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:06.923540+0000) 2024-03-06T09:15:07.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:07.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:07.923807+0000) 2024-03-06T09:15:08.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:08.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:08.924049+0000) 2024-03-06T09:15:09.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:09.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:09.924283+0000) 2024-03-06T09:15:10.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:10.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:10.924539+0000) 2024-03-06T09:15:11.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:11.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:11.924796+0000) 2024-03-06T09:15:12.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:12.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:12.925064+0000) 2024-03-06T09:15:13.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:13.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:13.925313+0000) 2024-03-06T09:15:14.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:14.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:14.925545+0000) 2024-03-06T09:15:15.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:15.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:15.925808+0000) 2024-03-06T09:15:16.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:16.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:16.926069+0000) 2024-03-06T09:15:17.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:17.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:17.926339+0000) 2024-03-06T09:15:18.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:18.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:18.926618+0000) 2024-03-06T09:15:19.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:19.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:19.926931+0000) 2024-03-06T09:15:20.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:20.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:20.927186+0000) 2024-03-06T09:15:21.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:21.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:21.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:21.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:21.927452+0000) 2024-03-06T09:15:22.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:22.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:22.927767+0000) 2024-03-06T09:15:23.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:23.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:23.928047+0000) 2024-03-06T09:15:24.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:24.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:24.928303+0000) 2024-03-06T09:15:25.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:25.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:25.928587+0000) 2024-03-06T09:15:26.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:26.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:26.928825+0000) 2024-03-06T09:15:27.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:27.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:27.929121+0000) 2024-03-06T09:15:28.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:28.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:28.929434+0000) 2024-03-06T09:15:29.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:29.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:29.929830+0000) 2024-03-06T09:15:30.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:30.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:30.930126+0000) 2024-03-06T09:15:31.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:31.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:31.930377+0000) 2024-03-06T09:15:32.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:32.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:32.930737+0000) 2024-03-06T09:15:33.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:33.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:33.931058+0000) 2024-03-06T09:15:34.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:34.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:34.931376+0000) 2024-03-06T09:15:35.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:35.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:35.931741+0000) 2024-03-06T09:15:36.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:36.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:36.932035+0000) 2024-03-06T09:15:37.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:37.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:37.932357+0000) 2024-03-06T09:15:38.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:38.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:38.932733+0000) 2024-03-06T09:15:39.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:39.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:39.933064+0000) 2024-03-06T09:15:40.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:40.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:40.933372+0000) 2024-03-06T09:15:41.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:41.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:41.933727+0000) 2024-03-06T09:15:42.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:42.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:42.934047+0000) 2024-03-06T09:15:43.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:43.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:43.934376+0000) 2024-03-06T09:15:44.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:44.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:44.934707+0000) 2024-03-06T09:15:45.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:45.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:45.935018+0000) 2024-03-06T09:15:46.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:46.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:46.935315+0000) 2024-03-06T09:15:47.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:47.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:47.935676+0000) 2024-03-06T09:15:48.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:48.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:48.935919+0000) 2024-03-06T09:15:49.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:49.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:49.936245+0000) 2024-03-06T09:15:50.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:50.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:50.936546+0000) 2024-03-06T09:15:51.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:51.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:51.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:51.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:51.936870+0000) 2024-03-06T09:15:52.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:52.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:52.937206+0000) 2024-03-06T09:15:53.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:53.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:53.937519+0000) 2024-03-06T09:15:54.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:54.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:54.937821+0000) 2024-03-06T09:15:55.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:55.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:55.938160+0000) 2024-03-06T09:15:56.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:56.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:56.938480+0000) 2024-03-06T09:15:57.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:57.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:57.938807+0000) 2024-03-06T09:15:58.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:58.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:58.939030+0000) 2024-03-06T09:15:59.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:15:59.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:59.939307+0000) 2024-03-06T09:16:00.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:00.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:00.939611+0000) 2024-03-06T09:16:01.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:01.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:01.939976+0000) 2024-03-06T09:16:02.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:02.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:02.940238+0000) 2024-03-06T09:16:03.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:03.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:03.940547+0000) 2024-03-06T09:16:04.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:04.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:04.940870+0000) 2024-03-06T09:16:05.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:05.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:05.941179+0000) 2024-03-06T09:16:06.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:06.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:06.941497+0000) 2024-03-06T09:16:07.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:07.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:07.941806+0000) 2024-03-06T09:16:08.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:08.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:08.942114+0000) 2024-03-06T09:16:09.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:09.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:09.942427+0000) 2024-03-06T09:16:10.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:10.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:10.942760+0000) 2024-03-06T09:16:11.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:11.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:11.943077+0000) 2024-03-06T09:16:12.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:12.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:12.943343+0000) 2024-03-06T09:16:13.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:13.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:13.943685+0000) 2024-03-06T09:16:14.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:14.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:14.943996+0000) 2024-03-06T09:16:15.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:15.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:15.944310+0000) 2024-03-06T09:16:16.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:16.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:16.944679+0000) 2024-03-06T09:16:17.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:17.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:17.944943+0000) 2024-03-06T09:16:18.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:18.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:18.945256+0000) 2024-03-06T09:16:19.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:19.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:19.945550+0000) 2024-03-06T09:16:20.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:20.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:20.945870+0000) 2024-03-06T09:16:21.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:21.300 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:21.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:21.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:21.946111+0000) 2024-03-06T09:16:22.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:22.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:22.946364+0000) 2024-03-06T09:16:23.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:23.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:23.946677+0000) 2024-03-06T09:16:24.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:24.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:24.946897+0000) 2024-03-06T09:16:25.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:25.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:25.947144+0000) 2024-03-06T09:16:26.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:26.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:26.947374+0000) 2024-03-06T09:16:27.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:27.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:27.947600+0000) 2024-03-06T09:16:28.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:28.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:28.947925+0000) 2024-03-06T09:16:29.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:29.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:29.948166+0000) 2024-03-06T09:16:30.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:30.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:30.948428+0000) 2024-03-06T09:16:31.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:31.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:31.948733+0000) 2024-03-06T09:16:32.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:32.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:32.948987+0000) 2024-03-06T09:16:33.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:33.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:33.949252+0000) 2024-03-06T09:16:34.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:34.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:34.949503+0000) 2024-03-06T09:16:35.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:35.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:35.949763+0000) 2024-03-06T09:16:36.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:36.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:36.950024+0000) 2024-03-06T09:16:37.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:37.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:37.950286+0000) 2024-03-06T09:16:38.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:38.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:38.950572+0000) 2024-03-06T09:16:39.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:39.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:39.950847+0000) 2024-03-06T09:16:40.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:40.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:40.951043+0000) 2024-03-06T09:16:41.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:41.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:41.951299+0000) 2024-03-06T09:16:42.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:42.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:42.951538+0000) 2024-03-06T09:16:43.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:43.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:43.951817+0000) 2024-03-06T09:16:44.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:44.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:44.952054+0000) 2024-03-06T09:16:45.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:45.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:45.952292+0000) 2024-03-06T09:16:46.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:46.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:46.952542+0000) 2024-03-06T09:16:47.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:47.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:47.952786+0000) 2024-03-06T09:16:48.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:48.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:48.953055+0000) 2024-03-06T09:16:49.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:49.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:49.953312+0000) 2024-03-06T09:16:50.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:50.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:50.953546+0000) 2024-03-06T09:16:51.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:51.486 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:51.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:51.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:51.953814+0000) 2024-03-06T09:16:52.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:52.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:52.954075+0000) 2024-03-06T09:16:53.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:53.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:53.954347+0000) 2024-03-06T09:16:54.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:54.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:54.954614+0000) 2024-03-06T09:16:55.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:55.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:55.954915+0000) 2024-03-06T09:16:56.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:56.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:56.955179+0000) 2024-03-06T09:16:57.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:57.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:57.955438+0000) 2024-03-06T09:16:58.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:58.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:58.955747+0000) 2024-03-06T09:16:59.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:16:59.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:59.956014+0000) 2024-03-06T09:17:00.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:00.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:00.956258+0000) 2024-03-06T09:17:01.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:01.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:01.956512+0000) 2024-03-06T09:17:02.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:02.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:02.956776+0000) 2024-03-06T09:17:03.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:03.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:03.957028+0000) 2024-03-06T09:17:04.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:04.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:04.957270+0000) 2024-03-06T09:17:05.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:05.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:05.957538+0000) 2024-03-06T09:17:06.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:06.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:06.957795+0000) 2024-03-06T09:17:07.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:07.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:07.958055+0000) 2024-03-06T09:17:08.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:08.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:08.958331+0000) 2024-03-06T09:17:09.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:09.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:09.958593+0000) 2024-03-06T09:17:10.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:10.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:10.958837+0000) 2024-03-06T09:17:11.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:11.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:11.959097+0000) 2024-03-06T09:17:12.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:12.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:12.959338+0000) 2024-03-06T09:17:13.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:13.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:13.959612+0000) 2024-03-06T09:17:14.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:14.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:14.959835+0000) 2024-03-06T09:17:15.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:15.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:15.960105+0000) 2024-03-06T09:17:16.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:16.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:16.960349+0000) 2024-03-06T09:17:17.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:17.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:17.960604+0000) 2024-03-06T09:17:18.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:18.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:18.960925+0000) 2024-03-06T09:17:19.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:19.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:19.961208+0000) 2024-03-06T09:17:20.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:20.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:20.961475+0000) 2024-03-06T09:17:21.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:21.645 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:21.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:21.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:21.961747+0000) 2024-03-06T09:17:22.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:22.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:22.961985+0000) 2024-03-06T09:17:23.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:23.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:23.962233+0000) 2024-03-06T09:17:24.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:24.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:24.962489+0000) 2024-03-06T09:17:25.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:25.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:25.962758+0000) 2024-03-06T09:17:26.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:26.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:26.963053+0000) 2024-03-06T09:17:27.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:27.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:27.963327+0000) 2024-03-06T09:17:28.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:28.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:28.963555+0000) 2024-03-06T09:17:29.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:29.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:29.963833+0000) 2024-03-06T09:17:30.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:30.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:30.964062+0000) 2024-03-06T09:17:31.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:31.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:31.964305+0000) 2024-03-06T09:17:32.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:32.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:32.964545+0000) 2024-03-06T09:17:33.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:33.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:33.964815+0000) 2024-03-06T09:17:34.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:34.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:34.965057+0000) 2024-03-06T09:17:35.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:35.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:35.965334+0000) 2024-03-06T09:17:36.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:36.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:36.965596+0000) 2024-03-06T09:17:37.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:37.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:37.965870+0000) 2024-03-06T09:17:38.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:38.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:38.966126+0000) 2024-03-06T09:17:39.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:39.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:39.966385+0000) 2024-03-06T09:17:40.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:40.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:40.966590+0000) 2024-03-06T09:17:41.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:41.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:41.966825+0000) 2024-03-06T09:17:42.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:42.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:42.967079+0000) 2024-03-06T09:17:43.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:43.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:43.967280+0000) 2024-03-06T09:17:44.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:44.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:44.967510+0000) 2024-03-06T09:17:45.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:45.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:45.967793+0000) 2024-03-06T09:17:46.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:46.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:46.968028+0000) 2024-03-06T09:17:47.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:47.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:47.968310+0000) 2024-03-06T09:17:48.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:48.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:48.968620+0000) 2024-03-06T09:17:49.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:49.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:49.968988+0000) 2024-03-06T09:17:50.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:50.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:50.969304+0000) 2024-03-06T09:17:51.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:51.793 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:51.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:51.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:51.969683+0000) 2024-03-06T09:17:52.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:52.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:52.969969+0000) 2024-03-06T09:17:53.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:53.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:53.970281+0000) 2024-03-06T09:17:54.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:54.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:54.970596+0000) 2024-03-06T09:17:55.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:55.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:55.970953+0000) 2024-03-06T09:17:56.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:56.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:56.971263+0000) 2024-03-06T09:17:57.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:57.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:57.971538+0000) 2024-03-06T09:17:58.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:58.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:58.971857+0000) 2024-03-06T09:17:59.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:17:59.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:59.972143+0000) 2024-03-06T09:18:00.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:00.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:00.972443+0000) 2024-03-06T09:18:01.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:01.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:01.972783+0000) 2024-03-06T09:18:02.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:02.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:02.973079+0000) 2024-03-06T09:18:03.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:03.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:03.973358+0000) 2024-03-06T09:18:04.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:04.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:04.973688+0000) 2024-03-06T09:18:05.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:05.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:05.974012+0000) 2024-03-06T09:18:06.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:06.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:06.974331+0000) 2024-03-06T09:18:07.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:07.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:07.974677+0000) 2024-03-06T09:18:08.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:08.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:08.974984+0000) 2024-03-06T09:18:09.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:09.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:09.975290+0000) 2024-03-06T09:18:10.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:10.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:10.975605+0000) 2024-03-06T09:18:11.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:11.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:11.975974+0000) 2024-03-06T09:18:12.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:12.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:12.976278+0000) 2024-03-06T09:18:13.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:13.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:13.976583+0000) 2024-03-06T09:18:14.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:14.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:14.976925+0000) 2024-03-06T09:18:15.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:15.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:15.977235+0000) 2024-03-06T09:18:16.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:16.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:16.977561+0000) 2024-03-06T09:18:17.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:17.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:17.977858+0000) 2024-03-06T09:18:18.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:18.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:18.978168+0000) 2024-03-06T09:18:19.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:19.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:19.978460+0000) 2024-03-06T09:18:20.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:20.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:20.978817+0000) 2024-03-06T09:18:21.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:21.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:21.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:21.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:21.979059+0000) 2024-03-06T09:18:22.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:22.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:22.979358+0000) 2024-03-06T09:18:23.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:23.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:23.979685+0000) 2024-03-06T09:18:24.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:24.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:24.979992+0000) 2024-03-06T09:18:25.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:25.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:25.980297+0000) 2024-03-06T09:18:26.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:26.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:26.980606+0000) 2024-03-06T09:18:27.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:27.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:27.980949+0000) 2024-03-06T09:18:28.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:28.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:28.981263+0000) 2024-03-06T09:18:29.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:29.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:29.981549+0000) 2024-03-06T09:18:30.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:30.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:30.981854+0000) 2024-03-06T09:18:31.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:31.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:31.982132+0000) 2024-03-06T09:18:32.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:32.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:32.982448+0000) 2024-03-06T09:18:33.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:33.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:33.982776+0000) 2024-03-06T09:18:34.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:34.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:34.983084+0000) 2024-03-06T09:18:35.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:35.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:35.983390+0000) 2024-03-06T09:18:36.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:36.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:36.983744+0000) 2024-03-06T09:18:37.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:37.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:37.983993+0000) 2024-03-06T09:18:38.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:38.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:38.984213+0000) 2024-03-06T09:18:39.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:39.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:39.984520+0000) 2024-03-06T09:18:40.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:40.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:40.984815+0000) 2024-03-06T09:18:41.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:41.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:41.985071+0000) 2024-03-06T09:18:42.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:42.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:42.985339+0000) 2024-03-06T09:18:43.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:43.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:43.985716+0000) 2024-03-06T09:18:44.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:44.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:44.986032+0000) 2024-03-06T09:18:45.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:45.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:45.986363+0000) 2024-03-06T09:18:46.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:46.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:46.986751+0000) 2024-03-06T09:18:47.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:47.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:47.987085+0000) 2024-03-06T09:18:48.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:48.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:48.987339+0000) 2024-03-06T09:18:49.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:49.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:49.987578+0000) 2024-03-06T09:18:50.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:50.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:50.987823+0000) 2024-03-06T09:18:51.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:51.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:51.988086+0000) 2024-03-06T09:18:52.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:52.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:52.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:52.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:52.988340+0000) 2024-03-06T09:18:53.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:53.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:53.988609+0000) 2024-03-06T09:18:54.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:54.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:54.988855+0000) 2024-03-06T09:18:55.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:55.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:55.989063+0000) 2024-03-06T09:18:56.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:56.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:56.989321+0000) 2024-03-06T09:18:57.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:57.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:57.989594+0000) 2024-03-06T09:18:58.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:58.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:58.989859+0000) 2024-03-06T09:18:59.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:18:59.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:59.990104+0000) 2024-03-06T09:19:00.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:00.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:00.990367+0000) 2024-03-06T09:19:01.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:01.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:01.990609+0000) 2024-03-06T09:19:02.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:02.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:02.991174+0000) 2024-03-06T09:19:03.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:03.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:03.991427+0000) 2024-03-06T09:19:04.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:04.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:04.991717+0000) 2024-03-06T09:19:05.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:05.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:05.991988+0000) 2024-03-06T09:19:06.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:06.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:06.992249+0000) 2024-03-06T09:19:07.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:07.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:07.992510+0000) 2024-03-06T09:19:08.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:08.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:08.992780+0000) 2024-03-06T09:19:09.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:09.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:09.992985+0000) 2024-03-06T09:19:10.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:10.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:10.993245+0000) 2024-03-06T09:19:11.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:11.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:11.993499+0000) 2024-03-06T09:19:12.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:12.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:12.993745+0000) 2024-03-06T09:19:13.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:13.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:13.994017+0000) 2024-03-06T09:19:14.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:14.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:14.994255+0000) 2024-03-06T09:19:15.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:15.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:15.994505+0000) 2024-03-06T09:19:16.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:16.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:16.994746+0000) 2024-03-06T09:19:17.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:17.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:17.995001+0000) 2024-03-06T09:19:18.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:18.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:18.995276+0000) 2024-03-06T09:19:19.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:19.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:19.995551+0000) 2024-03-06T09:19:20.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:20.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:20.995812+0000) 2024-03-06T09:19:21.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:21.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:21.996001+0000) 2024-03-06T09:19:22.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:22.243 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:22.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:22.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:22.996200+0000) 2024-03-06T09:19:23.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:23.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:23.996476+0000) 2024-03-06T09:19:24.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:24.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:24.996751+0000) 2024-03-06T09:19:25.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:25.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:25.997009+0000) 2024-03-06T09:19:26.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:26.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:26.997209+0000) 2024-03-06T09:19:27.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:27.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:27.997474+0000) 2024-03-06T09:19:28.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:28.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:28.997766+0000) 2024-03-06T09:19:29.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:29.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:29.998029+0000) 2024-03-06T09:19:30.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:30.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:30.998284+0000) 2024-03-06T09:19:31.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:31.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:31.998519+0000) 2024-03-06T09:19:32.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:32.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:32.998770+0000) 2024-03-06T09:19:33.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:33.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:33.999015+0000) 2024-03-06T09:19:34.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:34.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:34.999270+0000) 2024-03-06T09:19:35.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:35.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:35.999536+0000) 2024-03-06T09:19:36.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:36.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:36.999792+0000) 2024-03-06T09:19:37.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:37.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:38.000032+0000) 2024-03-06T09:19:38.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:38.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:39.000229+0000) 2024-03-06T09:19:39.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:39.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:40.000496+0000) 2024-03-06T09:19:40.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:40.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:41.000737+0000) 2024-03-06T09:19:42.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:42.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:42.000997+0000) 2024-03-06T09:19:43.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:43.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:43.001247+0000) 2024-03-06T09:19:44.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:44.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:44.001515+0000) 2024-03-06T09:19:45.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:45.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:45.001751+0000) 2024-03-06T09:19:46.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:46.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:46.002018+0000) 2024-03-06T09:19:47.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:47.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:47.002263+0000) 2024-03-06T09:19:48.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:48.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:48.002504+0000) 2024-03-06T09:19:49.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:49.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:49.002771+0000) 2024-03-06T09:19:50.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:50.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:50.003030+0000) 2024-03-06T09:19:51.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:51.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:51.003297+0000) 2024-03-06T09:19:52.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:52.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:52.003518+0000) 2024-03-06T09:19:52.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:52.395 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:53.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:53.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:53.003808+0000) 2024-03-06T09:19:54.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:54.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:54.004126+0000) 2024-03-06T09:19:55.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:55.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:55.004447+0000) 2024-03-06T09:19:56.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:56.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:56.004772+0000) 2024-03-06T09:19:57.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:57.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:57.005085+0000) 2024-03-06T09:19:58.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:58.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:58.005398+0000) 2024-03-06T09:19:59.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:19:59.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:59.005741+0000) 2024-03-06T09:20:00.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:00.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:00.006019+0000) 2024-03-06T09:20:01.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:01.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:01.006250+0000) 2024-03-06T09:20:02.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:02.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:02.006567+0000) 2024-03-06T09:20:03.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:03.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:03.006815+0000) 2024-03-06T09:20:04.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:04.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:04.007050+0000) 2024-03-06T09:20:05.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:05.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:05.007250+0000) 2024-03-06T09:20:06.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:06.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:06.007503+0000) 2024-03-06T09:20:07.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:07.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:07.007733+0000) 2024-03-06T09:20:08.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:08.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:08.007986+0000) 2024-03-06T09:20:09.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:09.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:09.008229+0000) 2024-03-06T09:20:10.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:10.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:10.008478+0000) 2024-03-06T09:20:11.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:11.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:11.008756+0000) 2024-03-06T09:20:12.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:12.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:12.008993+0000) 2024-03-06T09:20:13.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:13.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:13.009226+0000) 2024-03-06T09:20:14.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:14.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:14.009433+0000) 2024-03-06T09:20:15.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:15.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:15.009622+0000) 2024-03-06T09:20:16.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:16.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:16.009853+0000) 2024-03-06T09:20:17.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:17.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:17.010029+0000) 2024-03-06T09:20:18.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:18.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:18.010211+0000) 2024-03-06T09:20:19.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:19.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:19.010431+0000) 2024-03-06T09:20:20.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:20.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:20.010679+0000) 2024-03-06T09:20:21.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:21.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:21.010893+0000) 2024-03-06T09:20:22.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:22.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:22.011095+0000) 2024-03-06T09:20:22.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:22.536 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:23.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:23.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:23.011319+0000) 2024-03-06T09:20:24.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:24.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:24.011509+0000) 2024-03-06T09:20:25.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:25.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:25.011865+0000) 2024-03-06T09:20:26.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:26.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:26.012149+0000) 2024-03-06T09:20:27.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:27.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:27.012440+0000) 2024-03-06T09:20:28.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:28.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:28.012760+0000) 2024-03-06T09:20:29.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:29.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:29.013002+0000) 2024-03-06T09:20:30.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:30.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:30.013264+0000) 2024-03-06T09:20:31.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:31.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:31.013559+0000) 2024-03-06T09:20:32.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:32.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:32.013878+0000) 2024-03-06T09:20:33.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:33.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:33.014107+0000) 2024-03-06T09:20:34.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:34.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:34.014345+0000) 2024-03-06T09:20:35.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:35.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:35.014677+0000) 2024-03-06T09:20:36.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:36.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:36.014986+0000) 2024-03-06T09:20:37.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:37.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:37.015282+0000) 2024-03-06T09:20:38.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:38.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:38.015568+0000) 2024-03-06T09:20:39.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:39.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:39.015851+0000) 2024-03-06T09:20:40.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:40.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:40.016129+0000) 2024-03-06T09:20:41.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:41.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:41.016381+0000) 2024-03-06T09:20:42.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:42.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:42.016620+0000) 2024-03-06T09:20:43.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:43.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:43.016894+0000) 2024-03-06T09:20:44.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:44.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:44.017085+0000) 2024-03-06T09:20:45.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:45.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:45.017336+0000) 2024-03-06T09:20:46.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:46.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:46.017593+0000) 2024-03-06T09:20:47.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:47.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:47.017841+0000) 2024-03-06T09:20:48.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:48.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:48.018142+0000) 2024-03-06T09:20:49.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:49.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:49.018426+0000) 2024-03-06T09:20:50.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:50.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:50.018769+0000) 2024-03-06T09:20:51.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:51.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:51.019058+0000) 2024-03-06T09:20:52.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:52.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:52.019311+0000) 2024-03-06T09:20:52.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:52.690 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:53.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:53.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:53.019692+0000) 2024-03-06T09:20:54.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:54.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:54.019955+0000) 2024-03-06T09:20:55.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:55.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:55.020253+0000) 2024-03-06T09:20:56.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:56.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:56.020559+0000) 2024-03-06T09:20:57.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:57.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:57.020882+0000) 2024-03-06T09:20:58.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:58.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:58.021198+0000) 2024-03-06T09:20:59.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:20:59.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:59.021513+0000) 2024-03-06T09:21:00.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:00.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:00.021841+0000) 2024-03-06T09:21:01.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:01.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:01.022118+0000) 2024-03-06T09:21:02.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:02.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:02.022434+0000) 2024-03-06T09:21:03.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:03.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:03.022769+0000) 2024-03-06T09:21:04.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:04.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:04.023084+0000) 2024-03-06T09:21:05.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:05.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:05.023413+0000) 2024-03-06T09:21:06.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:06.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:06.023771+0000) 2024-03-06T09:21:07.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:07.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:07.024090+0000) 2024-03-06T09:21:08.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:08.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:08.024342+0000) 2024-03-06T09:21:09.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:09.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:09.024579+0000) 2024-03-06T09:21:10.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:10.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:10.024831+0000) 2024-03-06T09:21:11.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:11.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:11.025032+0000) 2024-03-06T09:21:12.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:12.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:12.025274+0000) 2024-03-06T09:21:13.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:13.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:13.025523+0000) 2024-03-06T09:21:14.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:14.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:14.025765+0000) 2024-03-06T09:21:15.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:15.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:15.026023+0000) 2024-03-06T09:21:16.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:16.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:16.026293+0000) 2024-03-06T09:21:17.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:17.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:17.026552+0000) 2024-03-06T09:21:18.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:18.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:18.026813+0000) 2024-03-06T09:21:19.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:19.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:19.027067+0000) 2024-03-06T09:21:20.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:20.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:20.027318+0000) 2024-03-06T09:21:21.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:21.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:21.027557+0000) 2024-03-06T09:21:22.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:22.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:22.027796+0000) 2024-03-06T09:21:22.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:22.836 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:23.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:23.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:23.028047+0000) 2024-03-06T09:21:24.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:24.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:24.028307+0000) 2024-03-06T09:21:25.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:25.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:25.028575+0000) 2024-03-06T09:21:26.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:26.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:26.028815+0000) 2024-03-06T09:21:27.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:27.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:27.029050+0000) 2024-03-06T09:21:28.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:28.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:28.029308+0000) 2024-03-06T09:21:29.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:29.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:29.029563+0000) 2024-03-06T09:21:30.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:30.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:30.029820+0000) 2024-03-06T09:21:31.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:31.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:31.030052+0000) 2024-03-06T09:21:32.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:32.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:32.030316+0000) 2024-03-06T09:21:33.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:33.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:33.030569+0000) 2024-03-06T09:21:34.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:34.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:34.030831+0000) 2024-03-06T09:21:35.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:35.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:35.031101+0000) 2024-03-06T09:21:36.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:36.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:36.031340+0000) 2024-03-06T09:21:37.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:37.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:37.031586+0000) 2024-03-06T09:21:38.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:38.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:38.031832+0000) 2024-03-06T09:21:39.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:39.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:39.032071+0000) 2024-03-06T09:21:40.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:40.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:40.032339+0000) 2024-03-06T09:21:41.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:41.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:41.032594+0000) 2024-03-06T09:21:42.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:42.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:42.032853+0000) 2024-03-06T09:21:43.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:43.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:43.033055+0000) 2024-03-06T09:21:44.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:44.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:44.033319+0000) 2024-03-06T09:21:45.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:45.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:45.033586+0000) 2024-03-06T09:21:46.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:46.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:46.033799+0000) 2024-03-06T09:21:47.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:47.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:47.034059+0000) 2024-03-06T09:21:48.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:48.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:48.034328+0000) 2024-03-06T09:21:49.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:49.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:49.034577+0000) 2024-03-06T09:21:50.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:50.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:50.034831+0000) 2024-03-06T09:21:51.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:51.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:51.035088+0000) 2024-03-06T09:21:52.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:52.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:52.035332+0000) 2024-03-06T09:21:52.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:52.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:53.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:53.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:53.035526+0000) 2024-03-06T09:21:54.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:54.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:54.035769+0000) 2024-03-06T09:21:55.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:55.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:55.036041+0000) 2024-03-06T09:21:56.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:56.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:56.036265+0000) 2024-03-06T09:21:57.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:57.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:57.036526+0000) 2024-03-06T09:21:58.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:58.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:58.036789+0000) 2024-03-06T09:21:59.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:21:59.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:59.036988+0000) 2024-03-06T09:22:00.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:00.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:00.037185+0000) 2024-03-06T09:22:01.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:01.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:01.037439+0000) 2024-03-06T09:22:02.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:02.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:02.037734+0000) 2024-03-06T09:22:03.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:03.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:03.037993+0000) 2024-03-06T09:22:04.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:04.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:04.038234+0000) 2024-03-06T09:22:05.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:05.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:05.038543+0000) 2024-03-06T09:22:06.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:06.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:06.038812+0000) 2024-03-06T09:22:07.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:07.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:07.039012+0000) 2024-03-06T09:22:08.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:08.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:08.039256+0000) 2024-03-06T09:22:09.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:09.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:09.039494+0000) 2024-03-06T09:22:10.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:10.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:10.039766+0000) 2024-03-06T09:22:11.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:11.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:11.040020+0000) 2024-03-06T09:22:12.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:12.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:12.040221+0000) 2024-03-06T09:22:13.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:13.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:13.040448+0000) 2024-03-06T09:22:14.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:14.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:14.040730+0000) 2024-03-06T09:22:15.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:15.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:15.040971+0000) 2024-03-06T09:22:16.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:16.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:16.041177+0000) 2024-03-06T09:22:17.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:17.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:17.041416+0000) 2024-03-06T09:22:18.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:18.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:18.041719+0000) 2024-03-06T09:22:19.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:19.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:19.041956+0000) 2024-03-06T09:22:20.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:20.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:20.042218+0000) 2024-03-06T09:22:21.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:21.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:21.042475+0000) 2024-03-06T09:22:22.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:22.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:22.042742+0000) 2024-03-06T09:22:23.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:23.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:23.042996+0000) 2024-03-06T09:22:23.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:23.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:24.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:24.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:24.043247+0000) 2024-03-06T09:22:25.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:25.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:25.043481+0000) 2024-03-06T09:22:26.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:26.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:26.043770+0000) 2024-03-06T09:22:27.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:27.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:27.043964+0000) 2024-03-06T09:22:28.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:28.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:28.044235+0000) 2024-03-06T09:22:29.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:29.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:29.044487+0000) 2024-03-06T09:22:30.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:30.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:30.044752+0000) 2024-03-06T09:22:31.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:31.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:31.044994+0000) 2024-03-06T09:22:32.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:32.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:32.045254+0000) 2024-03-06T09:22:33.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:33.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:33.045495+0000) 2024-03-06T09:22:34.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:34.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:34.045756+0000) 2024-03-06T09:22:35.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:35.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:35.046009+0000) 2024-03-06T09:22:36.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:36.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:36.046279+0000) 2024-03-06T09:22:37.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:37.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:37.046519+0000) 2024-03-06T09:22:38.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:38.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:38.046747+0000) 2024-03-06T09:22:39.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:39.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:39.046987+0000) 2024-03-06T09:22:40.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:40.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:40.047235+0000) 2024-03-06T09:22:41.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:41.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:41.047487+0000) 2024-03-06T09:22:42.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:42.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:42.047746+0000) 2024-03-06T09:22:43.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:43.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:43.048003+0000) 2024-03-06T09:22:44.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:44.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:44.048188+0000) 2024-03-06T09:22:45.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:45.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:45.048419+0000) 2024-03-06T09:22:46.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:46.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:46.048705+0000) 2024-03-06T09:22:47.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:47.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:47.048969+0000) 2024-03-06T09:22:48.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:48.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:48.049225+0000) 2024-03-06T09:22:49.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:49.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:49.049479+0000) 2024-03-06T09:22:50.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:50.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:50.049740+0000) 2024-03-06T09:22:51.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:51.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:51.049997+0000) 2024-03-06T09:22:52.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:52.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:52.050252+0000) 2024-03-06T09:22:53.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:53.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:53.050541+0000) 2024-03-06T09:22:53.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:53.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:54.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:54.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:54.050801+0000) 2024-03-06T09:22:55.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:55.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:55.051036+0000) 2024-03-06T09:22:56.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:56.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:56.051245+0000) 2024-03-06T09:22:57.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:57.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:57.051497+0000) 2024-03-06T09:22:58.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:58.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:58.051741+0000) 2024-03-06T09:22:59.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:22:59.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:59.052020+0000) 2024-03-06T09:23:00.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:00.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:00.052277+0000) 2024-03-06T09:23:01.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:01.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:01.052540+0000) 2024-03-06T09:23:02.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:02.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:02.052826+0000) 2024-03-06T09:23:03.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:03.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:03.053098+0000) 2024-03-06T09:23:04.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:04.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:04.053370+0000) 2024-03-06T09:23:05.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:05.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:05.053623+0000) 2024-03-06T09:23:06.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:06.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:06.053964+0000) 2024-03-06T09:23:07.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:07.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:07.054215+0000) 2024-03-06T09:23:08.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:08.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:08.054476+0000) 2024-03-06T09:23:09.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:09.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:09.054753+0000) 2024-03-06T09:23:10.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:10.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:10.055031+0000) 2024-03-06T09:23:11.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:11.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:11.055285+0000) 2024-03-06T09:23:12.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:12.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:12.055544+0000) 2024-03-06T09:23:13.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:13.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:13.055811+0000) 2024-03-06T09:23:14.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:14.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:14.056086+0000) 2024-03-06T09:23:15.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:15.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:15.056357+0000) 2024-03-06T09:23:16.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:16.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:16.056668+0000) 2024-03-06T09:23:17.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:17.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:17.056931+0000) 2024-03-06T09:23:18.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:18.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:18.057187+0000) 2024-03-06T09:23:19.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:19.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:19.057446+0000) 2024-03-06T09:23:20.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:20.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:20.057731+0000) 2024-03-06T09:23:21.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:21.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:21.057988+0000) 2024-03-06T09:23:22.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:22.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:22.058239+0000) 2024-03-06T09:23:23.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:23.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:23.058476+0000) 2024-03-06T09:23:23.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:23.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:24.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:24.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:24.058763+0000) 2024-03-06T09:23:25.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:25.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:25.058998+0000) 2024-03-06T09:23:26.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:26.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:26.059269+0000) 2024-03-06T09:23:27.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:27.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:27.059507+0000) 2024-03-06T09:23:28.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:28.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:28.059761+0000) 2024-03-06T09:23:29.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:29.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:29.059995+0000) 2024-03-06T09:23:30.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:30.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:30.060240+0000) 2024-03-06T09:23:31.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:31.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:31.060491+0000) 2024-03-06T09:23:32.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:32.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:32.060744+0000) 2024-03-06T09:23:33.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:33.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:33.060994+0000) 2024-03-06T09:23:34.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:34.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:34.061254+0000) 2024-03-06T09:23:35.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:35.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:35.061501+0000) 2024-03-06T09:23:36.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:36.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:36.061754+0000) 2024-03-06T09:23:37.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:37.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:37.062002+0000) 2024-03-06T09:23:38.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:38.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:38.062259+0000) 2024-03-06T09:23:39.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:39.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:39.062481+0000) 2024-03-06T09:23:40.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:40.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:40.062751+0000) 2024-03-06T09:23:41.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:41.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:41.062945+0000) 2024-03-06T09:23:42.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:42.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:42.063203+0000) 2024-03-06T09:23:43.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:43.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:43.063460+0000) 2024-03-06T09:23:44.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:44.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:44.063764+0000) 2024-03-06T09:23:45.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:45.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:45.064021+0000) 2024-03-06T09:23:46.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:46.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:46.064292+0000) 2024-03-06T09:23:47.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:47.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:47.064533+0000) 2024-03-06T09:23:48.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:48.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:48.064783+0000) 2024-03-06T09:23:49.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:49.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:49.065037+0000) 2024-03-06T09:23:50.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:50.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:50.065300+0000) 2024-03-06T09:23:51.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:51.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:51.065622+0000) 2024-03-06T09:23:52.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:52.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:52.065911+0000) 2024-03-06T09:23:53.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:53.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:53.066165+0000) 2024-03-06T09:23:53.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:53.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:54.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:54.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:54.066397+0000) 2024-03-06T09:23:55.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:55.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:55.066739+0000) 2024-03-06T09:23:56.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:56.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:56.067033+0000) 2024-03-06T09:23:57.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:57.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:57.067302+0000) 2024-03-06T09:23:58.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:58.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:58.067559+0000) 2024-03-06T09:23:59.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:23:59.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:59.067858+0000) 2024-03-06T09:24:00.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:00.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:00.068163+0000) 2024-03-06T09:24:01.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:01.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:01.068477+0000) 2024-03-06T09:24:02.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:02.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:02.068785+0000) 2024-03-06T09:24:03.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:03.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:03.069070+0000) 2024-03-06T09:24:04.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:04.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:04.069376+0000) 2024-03-06T09:24:05.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:05.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:05.069730+0000) 2024-03-06T09:24:06.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:06.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:06.070058+0000) 2024-03-06T09:24:07.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:07.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:07.070345+0000) 2024-03-06T09:24:08.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:08.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:08.070707+0000) 2024-03-06T09:24:09.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:09.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:09.071010+0000) 2024-03-06T09:24:10.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:10.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:10.071323+0000) 2024-03-06T09:24:11.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:11.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:11.071680+0000) 2024-03-06T09:24:12.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:12.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:12.071999+0000) 2024-03-06T09:24:13.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:13.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:13.072287+0000) 2024-03-06T09:24:14.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:14.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:14.072585+0000) 2024-03-06T09:24:15.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:15.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:15.072950+0000) 2024-03-06T09:24:16.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:16.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:16.073280+0000) 2024-03-06T09:24:17.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:17.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:17.073587+0000) 2024-03-06T09:24:18.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:18.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:18.073947+0000) 2024-03-06T09:24:19.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:19.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:19.074249+0000) 2024-03-06T09:24:20.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:20.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:20.074581+0000) 2024-03-06T09:24:21.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:21.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:21.074890+0000) 2024-03-06T09:24:22.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:22.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:22.075206+0000) 2024-03-06T09:24:23.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:23.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:23.075519+0000) 2024-03-06T09:24:23.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:23.748 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:24.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:24.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:24.075829+0000) 2024-03-06T09:24:25.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:25.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:25.076083+0000) 2024-03-06T09:24:26.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:26.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:26.076296+0000) 2024-03-06T09:24:27.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:27.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:27.076537+0000) 2024-03-06T09:24:28.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:28.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:28.076783+0000) 2024-03-06T09:24:29.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:29.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:29.077041+0000) 2024-03-06T09:24:30.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:30.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:30.077316+0000) 2024-03-06T09:24:31.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:31.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:31.077580+0000) 2024-03-06T09:24:32.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:32.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:32.077826+0000) 2024-03-06T09:24:33.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:33.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:33.078081+0000) 2024-03-06T09:24:34.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:34.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:34.078362+0000) 2024-03-06T09:24:35.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:35.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:35.078743+0000) 2024-03-06T09:24:36.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:36.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:36.079042+0000) 2024-03-06T09:24:37.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:37.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:37.079361+0000) 2024-03-06T09:24:38.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:38.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:38.079705+0000) 2024-03-06T09:24:39.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:39.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:39.080019+0000) 2024-03-06T09:24:40.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:40.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:40.080326+0000) 2024-03-06T09:24:41.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:41.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:41.080670+0000) 2024-03-06T09:24:42.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:42.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:42.080986+0000) 2024-03-06T09:24:43.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:43.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:43.081305+0000) 2024-03-06T09:24:44.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:44.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:44.081677+0000) 2024-03-06T09:24:45.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:45.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:45.082034+0000) 2024-03-06T09:24:46.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:46.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:46.082343+0000) 2024-03-06T09:24:47.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:47.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:47.082684+0000) 2024-03-06T09:24:48.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:48.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:48.083004+0000) 2024-03-06T09:24:49.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:49.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:49.083315+0000) 2024-03-06T09:24:50.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:50.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:50.083687+0000) 2024-03-06T09:24:51.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:51.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:51.083972+0000) 2024-03-06T09:24:52.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:52.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:52.084218+0000) 2024-03-06T09:24:53.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:53.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:53.084401+0000) 2024-03-06T09:24:53.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:53.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:54.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:54.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:54.084666+0000) 2024-03-06T09:24:55.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:55.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:55.084874+0000) 2024-03-06T09:24:56.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:56.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:56.085131+0000) 2024-03-06T09:24:57.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:57.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:57.085340+0000) 2024-03-06T09:24:58.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:58.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:58.085607+0000) 2024-03-06T09:24:59.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:24:59.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:59.085910+0000) 2024-03-06T09:25:00.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:00.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:00.086172+0000) 2024-03-06T09:25:01.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:01.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:01.086422+0000) 2024-03-06T09:25:02.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:02.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:02.086735+0000) 2024-03-06T09:25:03.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:03.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:03.086986+0000) 2024-03-06T09:25:04.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:04.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:04.087225+0000) 2024-03-06T09:25:05.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:05.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:05.087490+0000) 2024-03-06T09:25:06.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:06.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:06.087770+0000) 2024-03-06T09:25:07.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:07.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:07.088004+0000) 2024-03-06T09:25:08.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:08.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:08.088261+0000) 2024-03-06T09:25:09.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:09.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:09.088517+0000) 2024-03-06T09:25:10.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:10.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:10.088781+0000) 2024-03-06T09:25:11.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:11.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:11.089031+0000) 2024-03-06T09:25:12.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:12.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:12.089316+0000) 2024-03-06T09:25:13.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:13.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:13.089602+0000) 2024-03-06T09:25:14.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:14.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:14.089932+0000) 2024-03-06T09:25:15.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:15.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:15.090253+0000) 2024-03-06T09:25:16.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:16.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:16.090562+0000) 2024-03-06T09:25:17.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:17.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:17.090844+0000) 2024-03-06T09:25:18.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:18.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:18.091141+0000) 2024-03-06T09:25:19.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:19.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:19.091391+0000) 2024-03-06T09:25:20.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:20.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:20.091745+0000) 2024-03-06T09:25:21.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:21.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:21.092049+0000) 2024-03-06T09:25:22.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:22.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:22.092348+0000) 2024-03-06T09:25:23.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:23.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:23.092715+0000) 2024-03-06T09:25:24.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:24.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:24.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:24.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:24.092935+0000) 2024-03-06T09:25:25.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:25.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:25.093213+0000) 2024-03-06T09:25:26.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:26.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:26.093532+0000) 2024-03-06T09:25:27.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:27.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:27.093870+0000) 2024-03-06T09:25:28.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:28.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:28.094188+0000) 2024-03-06T09:25:29.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:29.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:29.094509+0000) 2024-03-06T09:25:30.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:30.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:30.094835+0000) 2024-03-06T09:25:31.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:31.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:31.095157+0000) 2024-03-06T09:25:32.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:32.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:32.095476+0000) 2024-03-06T09:25:33.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:33.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:33.095794+0000) 2024-03-06T09:25:34.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:34.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:34.096118+0000) 2024-03-06T09:25:35.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:35.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:35.096372+0000) 2024-03-06T09:25:36.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:36.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:36.096743+0000) 2024-03-06T09:25:37.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:37.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:37.097053+0000) 2024-03-06T09:25:38.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:38.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:38.097361+0000) 2024-03-06T09:25:39.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:39.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:39.097681+0000) 2024-03-06T09:25:40.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:40.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:40.097945+0000) 2024-03-06T09:25:41.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:41.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:41.098176+0000) 2024-03-06T09:25:42.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:42.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:42.098427+0000) 2024-03-06T09:25:43.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:43.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:43.098808+0000) 2024-03-06T09:25:44.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:44.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:44.099098+0000) 2024-03-06T09:25:45.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:45.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:45.099408+0000) 2024-03-06T09:25:46.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:46.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:46.099749+0000) 2024-03-06T09:25:47.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:47.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:47.100061+0000) 2024-03-06T09:25:48.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:48.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:48.100352+0000) 2024-03-06T09:25:49.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:49.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:49.100591+0000) 2024-03-06T09:25:50.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:50.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:50.100862+0000) 2024-03-06T09:25:51.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:51.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:51.101109+0000) 2024-03-06T09:25:52.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:52.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:52.101373+0000) 2024-03-06T09:25:53.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:53.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:53.101692+0000) 2024-03-06T09:25:54.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:54.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:54.102002+0000) 2024-03-06T09:25:54.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:54.215 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:55.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:55.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:55.102301+0000) 2024-03-06T09:25:56.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:56.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:56.102616+0000) 2024-03-06T09:25:57.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:57.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:57.102959+0000) 2024-03-06T09:25:58.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:58.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:58.103210+0000) 2024-03-06T09:25:59.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:25:59.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:59.103465+0000) 2024-03-06T09:26:00.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:00.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:00.103746+0000) 2024-03-06T09:26:01.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:01.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:01.104009+0000) 2024-03-06T09:26:02.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:02.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:02.104280+0000) 2024-03-06T09:26:03.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:03.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:03.104574+0000) 2024-03-06T09:26:04.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:04.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:04.104871+0000) 2024-03-06T09:26:05.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:05.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:05.105174+0000) 2024-03-06T09:26:06.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:06.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:06.105428+0000) 2024-03-06T09:26:07.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:07.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:07.105764+0000) 2024-03-06T09:26:08.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:08.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:08.106010+0000) 2024-03-06T09:26:09.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:09.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:09.106246+0000) 2024-03-06T09:26:10.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:10.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:10.106486+0000) 2024-03-06T09:26:11.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:11.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:11.106806+0000) 2024-03-06T09:26:12.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:12.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:12.107172+0000) 2024-03-06T09:26:13.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:13.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:13.107481+0000) 2024-03-06T09:26:14.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:14.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:14.107774+0000) 2024-03-06T09:26:15.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:15.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:15.108086+0000) 2024-03-06T09:26:16.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:16.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:16.108417+0000) 2024-03-06T09:26:17.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:17.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:17.108773+0000) 2024-03-06T09:26:18.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:18.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:18.109081+0000) 2024-03-06T09:26:19.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:19.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:19.109376+0000) 2024-03-06T09:26:20.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:20.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:20.109718+0000) 2024-03-06T09:26:21.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:21.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:21.109967+0000) 2024-03-06T09:26:22.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:22.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:22.110222+0000) 2024-03-06T09:26:23.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:23.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:23.110474+0000) 2024-03-06T09:26:24.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:24.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:24.110756+0000) 2024-03-06T09:26:24.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:24.366 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:25.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:25.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:25.111032+0000) 2024-03-06T09:26:26.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:26.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:26.111304+0000) 2024-03-06T09:26:27.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:27.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:27.111571+0000) 2024-03-06T09:26:28.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:28.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:28.111825+0000) 2024-03-06T09:26:29.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:29.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:29.112095+0000) 2024-03-06T09:26:30.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:30.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:30.112302+0000) 2024-03-06T09:26:31.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:31.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:31.112610+0000) 2024-03-06T09:26:32.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:32.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:32.112906+0000) 2024-03-06T09:26:33.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:33.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:33.113138+0000) 2024-03-06T09:26:34.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:34.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:34.113395+0000) 2024-03-06T09:26:35.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:35.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:35.113692+0000) 2024-03-06T09:26:36.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:36.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:36.113946+0000) 2024-03-06T09:26:37.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:37.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:37.114182+0000) 2024-03-06T09:26:38.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:38.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:38.114368+0000) 2024-03-06T09:26:39.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:39.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:39.114611+0000) 2024-03-06T09:26:40.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:40.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:40.114851+0000) 2024-03-06T09:26:41.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:41.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:41.115085+0000) 2024-03-06T09:26:42.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:42.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:42.115330+0000) 2024-03-06T09:26:43.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:43.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:43.115583+0000) 2024-03-06T09:26:44.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:44.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:44.115854+0000) 2024-03-06T09:26:45.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:45.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:45.116130+0000) 2024-03-06T09:26:46.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:46.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:46.116412+0000) 2024-03-06T09:26:47.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:47.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:47.116726+0000) 2024-03-06T09:26:48.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:48.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:48.116985+0000) 2024-03-06T09:26:49.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:49.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:49.117259+0000) 2024-03-06T09:26:50.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:50.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:50.117512+0000) 2024-03-06T09:26:51.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:51.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:51.117858+0000) 2024-03-06T09:26:52.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:52.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:52.118165+0000) 2024-03-06T09:26:53.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:53.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:53.118432+0000) 2024-03-06T09:26:54.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:54.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:54.118746+0000) 2024-03-06T09:26:54.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:54.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:55.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:55.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:55.118984+0000) 2024-03-06T09:26:56.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:56.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:56.119254+0000) 2024-03-06T09:26:57.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:57.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:57.119439+0000) 2024-03-06T09:26:58.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:58.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:58.119733+0000) 2024-03-06T09:26:59.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:26:59.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:59.119966+0000) 2024-03-06T09:27:00.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:00.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:00.120202+0000) 2024-03-06T09:27:01.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:01.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:01.120461+0000) 2024-03-06T09:27:02.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:02.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:02.120743+0000) 2024-03-06T09:27:03.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:03.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:03.120998+0000) 2024-03-06T09:27:04.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:04.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:04.121257+0000) 2024-03-06T09:27:05.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:05.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:05.121491+0000) 2024-03-06T09:27:06.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:06.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:06.121746+0000) 2024-03-06T09:27:07.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:07.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:07.121994+0000) 2024-03-06T09:27:08.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:08.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:08.122254+0000) 2024-03-06T09:27:09.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:09.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:09.122490+0000) 2024-03-06T09:27:10.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:10.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:10.122746+0000) 2024-03-06T09:27:11.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:11.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:11.122980+0000) 2024-03-06T09:27:12.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:12.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:12.123205+0000) 2024-03-06T09:27:13.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:13.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:13.123480+0000) 2024-03-06T09:27:14.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:14.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:14.123766+0000) 2024-03-06T09:27:15.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:15.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:15.124046+0000) 2024-03-06T09:27:16.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:16.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:16.124328+0000) 2024-03-06T09:27:17.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:17.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:17.124585+0000) 2024-03-06T09:27:18.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:18.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:18.124832+0000) 2024-03-06T09:27:19.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:19.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:19.125016+0000) 2024-03-06T09:27:20.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:20.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:20.125352+0000) 2024-03-06T09:27:21.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:21.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:21.125619+0000) 2024-03-06T09:27:22.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:22.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:22.125912+0000) 2024-03-06T09:27:23.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:23.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:23.126104+0000) 2024-03-06T09:27:24.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:24.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:24.126456+0000) 2024-03-06T09:27:24.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:24.645 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:25.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:25.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:25.126768+0000) 2024-03-06T09:27:26.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:26.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:26.126976+0000) 2024-03-06T09:27:27.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:27.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:27.127223+0000) 2024-03-06T09:27:28.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:28.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:28.127616+0000) 2024-03-06T09:27:29.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:29.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:29.127943+0000) 2024-03-06T09:27:30.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:30.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:30.128310+0000) 2024-03-06T09:27:31.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:31.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:31.128564+0000) 2024-03-06T09:27:32.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:32.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:32.128829+0000) 2024-03-06T09:27:33.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:33.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:33.129065+0000) 2024-03-06T09:27:34.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:34.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:34.129327+0000) 2024-03-06T09:27:35.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:35.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:35.129583+0000) 2024-03-06T09:27:36.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:36.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:36.129825+0000) 2024-03-06T09:27:37.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:37.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:37.130116+0000) 2024-03-06T09:27:38.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:38.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:38.130376+0000) 2024-03-06T09:27:39.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:39.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:39.130684+0000) 2024-03-06T09:27:40.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:40.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:40.130935+0000) 2024-03-06T09:27:41.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:41.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:41.131191+0000) 2024-03-06T09:27:42.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:42.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:42.131448+0000) 2024-03-06T09:27:43.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:43.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:43.131735+0000) 2024-03-06T09:27:44.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:44.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:44.132001+0000) 2024-03-06T09:27:45.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:45.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:45.132299+0000) 2024-03-06T09:27:46.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:46.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:46.132621+0000) 2024-03-06T09:27:47.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:47.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:47.132953+0000) 2024-03-06T09:27:48.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:48.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:48.133252+0000) 2024-03-06T09:27:49.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:49.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:49.133536+0000) 2024-03-06T09:27:50.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:50.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:50.133841+0000) 2024-03-06T09:27:51.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:51.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:51.134151+0000) 2024-03-06T09:27:52.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:52.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:52.134392+0000) 2024-03-06T09:27:53.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:53.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:53.134742+0000) 2024-03-06T09:27:54.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:54.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:54.135046+0000) 2024-03-06T09:27:54.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:54.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:55.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:55.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:55.135310+0000) 2024-03-06T09:27:56.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:56.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:56.135667+0000) 2024-03-06T09:27:57.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:57.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:57.135981+0000) 2024-03-06T09:27:58.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:58.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:58.136224+0000) 2024-03-06T09:27:59.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:27:59.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:59.136524+0000) 2024-03-06T09:28:00.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:00.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:00.136889+0000) 2024-03-06T09:28:01.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:01.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:01.137201+0000) 2024-03-06T09:28:02.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:02.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:02.137460+0000) 2024-03-06T09:28:03.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:03.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:03.137780+0000) 2024-03-06T09:28:04.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:04.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:04.138002+0000) 2024-03-06T09:28:05.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:05.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:05.138240+0000) 2024-03-06T09:28:06.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:06.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:06.138559+0000) 2024-03-06T09:28:07.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:07.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:07.138766+0000) 2024-03-06T09:28:08.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:08.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:08.139012+0000) 2024-03-06T09:28:09.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:09.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:09.139278+0000) 2024-03-06T09:28:10.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:10.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:10.139533+0000) 2024-03-06T09:28:11.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:11.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:11.139787+0000) 2024-03-06T09:28:12.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:12.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:12.140059+0000) 2024-03-06T09:28:13.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:13.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:13.140313+0000) 2024-03-06T09:28:14.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:14.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:14.140569+0000) 2024-03-06T09:28:15.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:15.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:15.140801+0000) 2024-03-06T09:28:16.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:16.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:16.141013+0000) 2024-03-06T09:28:17.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:17.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:17.141244+0000) 2024-03-06T09:28:18.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:18.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:18.141479+0000) 2024-03-06T09:28:19.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:19.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:19.141776+0000) 2024-03-06T09:28:20.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:20.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:20.142088+0000) 2024-03-06T09:28:21.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:21.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:21.142391+0000) 2024-03-06T09:28:22.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:22.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:22.142757+0000) 2024-03-06T09:28:23.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:23.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:23.142984+0000) 2024-03-06T09:28:24.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:24.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:24.143199+0000) 2024-03-06T09:28:24.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:24.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:25.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:25.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:25.143446+0000) 2024-03-06T09:28:26.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:26.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:26.143781+0000) 2024-03-06T09:28:27.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:27.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:27.144078+0000) 2024-03-06T09:28:28.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:28.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:28.144393+0000) 2024-03-06T09:28:29.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:29.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:29.144791+0000) 2024-03-06T09:28:30.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:30.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:30.145088+0000) 2024-03-06T09:28:31.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:31.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:31.145361+0000) 2024-03-06T09:28:32.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:32.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:32.145583+0000) 2024-03-06T09:28:33.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:33.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:33.145815+0000) 2024-03-06T09:28:34.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:34.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:34.146069+0000) 2024-03-06T09:28:35.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:35.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:35.146256+0000) 2024-03-06T09:28:36.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:36.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:36.146538+0000) 2024-03-06T09:28:37.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:37.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:37.146777+0000) 2024-03-06T09:28:38.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:38.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:38.147030+0000) 2024-03-06T09:28:39.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:39.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:39.147347+0000) 2024-03-06T09:28:40.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:40.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:40.147616+0000) 2024-03-06T09:28:41.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:41.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:41.147934+0000) 2024-03-06T09:28:42.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:42.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:42.148198+0000) 2024-03-06T09:28:43.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:43.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:43.148479+0000) 2024-03-06T09:28:44.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:44.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:44.148757+0000) 2024-03-06T09:28:45.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:45.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:45.148955+0000) 2024-03-06T09:28:46.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:46.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:46.149251+0000) 2024-03-06T09:28:47.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:47.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:47.149761+0000) 2024-03-06T09:28:48.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:48.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:48.150038+0000) 2024-03-06T09:28:49.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:49.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:49.150291+0000) 2024-03-06T09:28:50.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:50.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:50.150567+0000) 2024-03-06T09:28:51.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:51.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:51.150850+0000) 2024-03-06T09:28:52.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:52.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:52.151146+0000) 2024-03-06T09:28:53.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:53.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:53.151420+0000) 2024-03-06T09:28:54.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:54.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:54.151723+0000) 2024-03-06T09:28:55.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:55.102 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:55.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:55.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:55.151908+0000) 2024-03-06T09:28:56.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:56.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:56.152171+0000) 2024-03-06T09:28:57.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:57.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:57.152405+0000) 2024-03-06T09:28:58.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:58.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:58.152670+0000) 2024-03-06T09:28:59.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:28:59.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:59.152908+0000) 2024-03-06T09:29:00.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:00.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:00.153170+0000) 2024-03-06T09:29:01.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:01.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:01.153434+0000) 2024-03-06T09:29:02.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:02.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:02.153735+0000) 2024-03-06T09:29:03.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:03.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:03.153910+0000) 2024-03-06T09:29:04.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:04.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:04.154098+0000) 2024-03-06T09:29:05.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:05.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:05.154276+0000) 2024-03-06T09:29:06.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:06.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:06.154589+0000) 2024-03-06T09:29:07.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:07.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:07.154849+0000) 2024-03-06T09:29:08.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:08.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:08.155106+0000) 2024-03-06T09:29:09.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:09.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:09.155356+0000) 2024-03-06T09:29:10.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:10.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:10.155610+0000) 2024-03-06T09:29:11.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:11.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:11.155833+0000) 2024-03-06T09:29:12.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:12.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:12.156103+0000) 2024-03-06T09:29:13.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:13.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:13.156343+0000) 2024-03-06T09:29:14.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:14.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:14.156573+0000) 2024-03-06T09:29:15.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:15.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:15.156813+0000) 2024-03-06T09:29:16.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:16.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:16.157029+0000) 2024-03-06T09:29:17.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:17.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:17.157249+0000) 2024-03-06T09:29:18.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:18.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:18.157504+0000) 2024-03-06T09:29:19.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:19.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:19.157747+0000) 2024-03-06T09:29:20.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:20.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:20.157988+0000) 2024-03-06T09:29:21.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:21.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:21.158251+0000) 2024-03-06T09:29:22.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:22.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:22.158592+0000) 2024-03-06T09:29:23.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:23.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:23.158832+0000) 2024-03-06T09:29:24.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:24.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:24.159071+0000) 2024-03-06T09:29:25.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:25.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:25.159313+0000) 2024-03-06T09:29:25.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:25.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:26.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:26.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:26.159584+0000) 2024-03-06T09:29:27.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:27.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:27.159857+0000) 2024-03-06T09:29:28.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:28.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:28.160064+0000) 2024-03-06T09:29:29.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:29.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:29.160315+0000) 2024-03-06T09:29:30.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:30.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:30.160563+0000) 2024-03-06T09:29:31.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:31.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:31.160814+0000) 2024-03-06T09:29:32.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:32.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:32.161085+0000) 2024-03-06T09:29:33.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:33.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:33.161341+0000) 2024-03-06T09:29:34.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:34.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:34.161580+0000) 2024-03-06T09:29:35.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:35.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:35.161826+0000) 2024-03-06T09:29:36.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:36.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:36.162105+0000) 2024-03-06T09:29:37.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:37.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:37.162356+0000) 2024-03-06T09:29:38.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:38.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:38.162595+0000) 2024-03-06T09:29:39.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:39.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:39.162843+0000) 2024-03-06T09:29:40.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:40.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:40.163081+0000) 2024-03-06T09:29:41.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:41.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:41.163345+0000) 2024-03-06T09:29:42.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:42.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:42.163605+0000) 2024-03-06T09:29:43.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:43.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:43.163843+0000) 2024-03-06T09:29:44.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:44.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:44.164104+0000) 2024-03-06T09:29:45.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:45.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:45.164360+0000) 2024-03-06T09:29:46.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:46.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:46.164675+0000) 2024-03-06T09:29:47.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:47.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:47.164947+0000) 2024-03-06T09:29:48.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:48.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:48.165207+0000) 2024-03-06T09:29:49.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:49.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:49.165455+0000) 2024-03-06T09:29:50.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:50.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:50.165733+0000) 2024-03-06T09:29:51.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:51.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:51.165972+0000) 2024-03-06T09:29:52.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:52.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:52.166225+0000) 2024-03-06T09:29:53.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:53.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:53.166470+0000) 2024-03-06T09:29:54.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:54.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:54.166756+0000) 2024-03-06T09:29:55.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:55.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:55.167006+0000) 2024-03-06T09:29:55.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:55.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:56.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:56.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:56.167263+0000) 2024-03-06T09:29:57.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:57.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:57.167504+0000) 2024-03-06T09:29:58.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:58.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:58.167724+0000) 2024-03-06T09:29:59.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:29:59.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:59.167947+0000) 2024-03-06T09:30:00.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:00.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:00.168222+0000) 2024-03-06T09:30:01.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:01.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:01.168503+0000) 2024-03-06T09:30:02.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:02.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:02.168776+0000) 2024-03-06T09:30:03.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:03.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:03.169046+0000) 2024-03-06T09:30:04.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:04.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:04.169326+0000) 2024-03-06T09:30:05.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:05.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:05.169605+0000) 2024-03-06T09:30:06.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:06.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:06.169939+0000) 2024-03-06T09:30:07.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:07.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:07.170154+0000) 2024-03-06T09:30:08.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:08.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:08.170366+0000) 2024-03-06T09:30:09.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:09.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:09.170674+0000) 2024-03-06T09:30:10.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:10.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:10.170953+0000) 2024-03-06T09:30:11.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:11.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:11.171211+0000) 2024-03-06T09:30:12.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:12.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:12.171501+0000) 2024-03-06T09:30:13.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:13.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:13.171759+0000) 2024-03-06T09:30:14.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:14.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:14.172024+0000) 2024-03-06T09:30:15.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:15.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:15.172307+0000) 2024-03-06T09:30:16.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:16.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:16.172605+0000) 2024-03-06T09:30:17.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:17.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:17.172946+0000) 2024-03-06T09:30:18.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:18.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:18.173213+0000) 2024-03-06T09:30:19.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:19.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:19.173482+0000) 2024-03-06T09:30:20.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:20.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:20.173741+0000) 2024-03-06T09:30:21.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:21.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:21.174061+0000) 2024-03-06T09:30:22.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:22.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:22.174338+0000) 2024-03-06T09:30:23.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:23.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:23.174599+0000) 2024-03-06T09:30:24.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:24.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:24.174905+0000) 2024-03-06T09:30:25.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:25.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:25.175177+0000) 2024-03-06T09:30:25.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:25.559 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:26.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:26.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:26.175407+0000) 2024-03-06T09:30:27.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:27.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:27.175752+0000) 2024-03-06T09:30:28.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:28.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:28.175996+0000) 2024-03-06T09:30:29.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:29.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:29.176268+0000) 2024-03-06T09:30:30.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:30.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:30.176544+0000) 2024-03-06T09:30:31.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:31.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:31.176807+0000) 2024-03-06T09:30:32.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:32.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:32.177037+0000) 2024-03-06T09:30:33.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:33.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:33.177307+0000) 2024-03-06T09:30:34.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:34.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:34.177515+0000) 2024-03-06T09:30:35.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:35.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:35.177769+0000) 2024-03-06T09:30:36.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:36.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:36.178066+0000) 2024-03-06T09:30:37.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:37.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:37.178335+0000) 2024-03-06T09:30:38.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:38.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:38.178572+0000) 2024-03-06T09:30:39.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:39.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:39.178820+0000) 2024-03-06T09:30:40.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:40.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:40.179015+0000) 2024-03-06T09:30:41.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:41.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:41.179246+0000) 2024-03-06T09:30:42.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:42.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:42.179539+0000) 2024-03-06T09:30:43.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:43.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:43.179808+0000) 2024-03-06T09:30:44.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:44.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:44.180085+0000) 2024-03-06T09:30:45.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:45.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:45.180340+0000) 2024-03-06T09:30:46.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:46.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:46.180677+0000) 2024-03-06T09:30:47.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:47.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:47.180969+0000) 2024-03-06T09:30:48.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:48.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:48.181226+0000) 2024-03-06T09:30:49.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:49.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:49.181464+0000) 2024-03-06T09:30:50.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:50.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:50.181770+0000) 2024-03-06T09:30:51.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:51.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:51.182031+0000) 2024-03-06T09:30:52.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:52.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:52.182323+0000) 2024-03-06T09:30:53.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:53.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:53.182576+0000) 2024-03-06T09:30:54.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:54.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:54.182763+0000) 2024-03-06T09:30:55.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:55.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:55.183038+0000) 2024-03-06T09:30:55.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:55.715 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:56.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:56.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:56.183339+0000) 2024-03-06T09:30:57.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:57.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:57.183592+0000) 2024-03-06T09:30:58.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:58.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:58.183888+0000) 2024-03-06T09:30:59.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:30:59.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:59.184209+0000) 2024-03-06T09:31:00.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:00.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:00.184519+0000) 2024-03-06T09:31:01.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:01.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:01.184834+0000) 2024-03-06T09:31:02.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:02.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:02.185135+0000) 2024-03-06T09:31:03.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:03.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:03.185454+0000) 2024-03-06T09:31:04.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:04.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:04.185803+0000) 2024-03-06T09:31:05.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:05.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:05.186101+0000) 2024-03-06T09:31:06.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:06.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:06.186446+0000) 2024-03-06T09:31:07.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:07.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:07.186776+0000) 2024-03-06T09:31:08.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:08.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:08.187097+0000) 2024-03-06T09:31:09.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:09.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:09.187327+0000) 2024-03-06T09:31:10.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:10.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:10.187623+0000) 2024-03-06T09:31:11.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:11.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:11.187991+0000) 2024-03-06T09:31:12.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:12.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:12.188293+0000) 2024-03-06T09:31:13.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:13.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:13.188613+0000) 2024-03-06T09:31:14.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:14.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:14.188959+0000) 2024-03-06T09:31:15.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:15.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:15.189247+0000) 2024-03-06T09:31:16.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:16.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:16.189556+0000) 2024-03-06T09:31:17.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:17.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:17.189872+0000) 2024-03-06T09:31:18.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:18.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:18.190185+0000) 2024-03-06T09:31:19.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:19.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:19.190547+0000) 2024-03-06T09:31:20.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:20.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:20.190816+0000) 2024-03-06T09:31:21.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:21.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:21.191114+0000) 2024-03-06T09:31:22.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:22.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:22.191433+0000) 2024-03-06T09:31:23.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:23.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:23.191710+0000) 2024-03-06T09:31:24.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:24.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:24.192038+0000) 2024-03-06T09:31:25.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:25.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:25.192345+0000) 2024-03-06T09:31:25.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:25.870 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:26.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:26.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:26.192699+0000) 2024-03-06T09:31:27.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:27.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:27.192962+0000) 2024-03-06T09:31:28.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:28.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:28.193247+0000) 2024-03-06T09:31:29.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:29.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:29.193519+0000) 2024-03-06T09:31:30.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:30.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:30.193798+0000) 2024-03-06T09:31:31.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:31.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:31.194097+0000) 2024-03-06T09:31:32.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:32.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:32.194395+0000) 2024-03-06T09:31:33.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:33.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:33.194736+0000) 2024-03-06T09:31:34.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:34.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:34.195039+0000) 2024-03-06T09:31:35.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:35.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:35.195344+0000) 2024-03-06T09:31:36.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:36.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:36.195676+0000) 2024-03-06T09:31:37.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:37.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:37.195990+0000) 2024-03-06T09:31:38.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:38.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:38.196267+0000) 2024-03-06T09:31:39.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:39.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:39.196556+0000) 2024-03-06T09:31:40.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:40.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:40.196856+0000) 2024-03-06T09:31:41.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:41.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:41.197107+0000) 2024-03-06T09:31:42.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:42.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:42.197407+0000) 2024-03-06T09:31:43.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:43.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:43.197693+0000) 2024-03-06T09:31:44.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:44.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:44.197954+0000) 2024-03-06T09:31:45.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:45.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:45.198179+0000) 2024-03-06T09:31:46.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:46.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:46.198437+0000) 2024-03-06T09:31:47.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:47.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:47.198733+0000) 2024-03-06T09:31:48.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:48.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:48.198913+0000) 2024-03-06T09:31:49.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:49.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:49.199181+0000) 2024-03-06T09:31:50.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:50.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:50.199483+0000) 2024-03-06T09:31:51.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:51.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:51.199817+0000) 2024-03-06T09:31:52.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:52.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:52.200072+0000) 2024-03-06T09:31:53.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:53.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:53.200334+0000) 2024-03-06T09:31:54.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:54.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:54.200591+0000) 2024-03-06T09:31:55.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:55.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:55.200821+0000) 2024-03-06T09:31:56.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:56.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:56.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:56.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:56.201070+0000) 2024-03-06T09:31:57.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:57.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:57.201324+0000) 2024-03-06T09:31:58.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:58.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:58.201585+0000) 2024-03-06T09:31:59.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:31:59.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:59.201864+0000) 2024-03-06T09:32:00.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:00.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:00.202126+0000) 2024-03-06T09:32:01.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:01.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:01.202313+0000) 2024-03-06T09:32:02.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:02.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:02.202574+0000) 2024-03-06T09:32:03.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:03.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:03.202756+0000) 2024-03-06T09:32:04.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:04.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:04.202974+0000) 2024-03-06T09:32:05.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:05.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:05.203215+0000) 2024-03-06T09:32:06.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:06.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:06.203475+0000) 2024-03-06T09:32:07.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:07.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:07.203926+0000) 2024-03-06T09:32:08.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:08.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:08.204238+0000) 2024-03-06T09:32:09.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:09.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:09.204550+0000) 2024-03-06T09:32:10.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:10.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:10.204865+0000) 2024-03-06T09:32:11.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:11.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:11.205175+0000) 2024-03-06T09:32:12.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:12.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:12.205496+0000) 2024-03-06T09:32:13.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:13.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:13.205810+0000) 2024-03-06T09:32:14.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:14.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:14.206126+0000) 2024-03-06T09:32:15.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:15.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:15.206432+0000) 2024-03-06T09:32:16.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:16.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:16.206785+0000) 2024-03-06T09:32:17.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:17.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:17.207100+0000) 2024-03-06T09:32:18.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:18.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:18.207401+0000) 2024-03-06T09:32:19.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:19.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:19.207724+0000) 2024-03-06T09:32:20.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:20.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:20.207977+0000) 2024-03-06T09:32:21.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:21.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:21.208257+0000) 2024-03-06T09:32:22.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:22.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:22.208569+0000) 2024-03-06T09:32:23.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:23.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:23.208852+0000) 2024-03-06T09:32:24.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:24.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:24.209171+0000) 2024-03-06T09:32:25.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:25.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:25.209474+0000) 2024-03-06T09:32:26.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:26.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:26.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:26.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:26.209729+0000) 2024-03-06T09:32:27.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:27.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:27.210004+0000) 2024-03-06T09:32:28.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:28.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:28.210296+0000) 2024-03-06T09:32:29.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:29.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:29.210589+0000) 2024-03-06T09:32:30.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:30.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:30.210930+0000) 2024-03-06T09:32:31.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:31.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:31.211225+0000) 2024-03-06T09:32:32.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:32.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:32.211524+0000) 2024-03-06T09:32:33.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:33.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:33.211815+0000) 2024-03-06T09:32:34.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:34.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:34.212105+0000) 2024-03-06T09:32:35.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:35.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:35.212408+0000) 2024-03-06T09:32:36.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:36.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:36.212762+0000) 2024-03-06T09:32:37.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:37.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:37.213079+0000) 2024-03-06T09:32:38.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:38.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:38.213399+0000) 2024-03-06T09:32:39.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:39.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:39.213755+0000) 2024-03-06T09:32:40.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:40.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:40.214066+0000) 2024-03-06T09:32:41.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:41.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:41.214372+0000) 2024-03-06T09:32:42.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:42.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:42.214739+0000) 2024-03-06T09:32:43.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:43.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:43.215036+0000) 2024-03-06T09:32:44.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:44.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:44.215325+0000) 2024-03-06T09:32:45.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:45.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:45.215618+0000) 2024-03-06T09:32:46.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:46.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:46.216028+0000) 2024-03-06T09:32:47.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:47.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:47.216346+0000) 2024-03-06T09:32:48.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:48.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:48.216714+0000) 2024-03-06T09:32:49.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:49.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:49.217013+0000) 2024-03-06T09:32:50.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:50.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:50.217327+0000) 2024-03-06T09:32:51.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:51.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:51.217677+0000) 2024-03-06T09:32:52.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:52.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:52.217984+0000) 2024-03-06T09:32:53.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:53.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:53.218279+0000) 2024-03-06T09:32:54.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:54.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:54.218578+0000) 2024-03-06T09:32:55.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:55.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:55.218936+0000) 2024-03-06T09:32:56.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:56.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:56.219257+0000) 2024-03-06T09:32:56.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:56.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:57.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:57.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:57.219565+0000) 2024-03-06T09:32:58.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:58.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:58.219801+0000) 2024-03-06T09:32:59.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:32:59.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:59.220089+0000) 2024-03-06T09:33:00.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:00.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:00.220410+0000) 2024-03-06T09:33:01.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:01.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:01.220759+0000) 2024-03-06T09:33:02.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:02.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:02.221058+0000) 2024-03-06T09:33:03.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:03.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:03.221372+0000) 2024-03-06T09:33:04.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:04.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:04.221746+0000) 2024-03-06T09:33:05.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:05.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:05.222042+0000) 2024-03-06T09:33:06.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:06.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:06.222357+0000) 2024-03-06T09:33:07.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:07.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:07.222707+0000) 2024-03-06T09:33:08.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:08.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:08.223018+0000) 2024-03-06T09:33:09.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:09.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:09.223307+0000) 2024-03-06T09:33:10.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:10.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:10.223576+0000) 2024-03-06T09:33:11.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:11.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:11.223828+0000) 2024-03-06T09:33:12.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:12.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:12.224068+0000) 2024-03-06T09:33:13.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:13.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:13.224322+0000) 2024-03-06T09:33:14.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:14.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:14.224558+0000) 2024-03-06T09:33:15.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:15.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:15.224807+0000) 2024-03-06T09:33:16.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:16.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:16.225072+0000) 2024-03-06T09:33:17.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:17.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:17.225325+0000) 2024-03-06T09:33:18.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:18.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:18.225621+0000) 2024-03-06T09:33:19.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:19.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:19.225965+0000) 2024-03-06T09:33:20.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:20.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:20.226280+0000) 2024-03-06T09:33:21.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:21.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:21.226575+0000) 2024-03-06T09:33:22.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:22.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:22.226883+0000) 2024-03-06T09:33:23.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:23.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:23.227188+0000) 2024-03-06T09:33:24.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:24.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:24.227503+0000) 2024-03-06T09:33:25.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:25.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:25.227794+0000) 2024-03-06T09:33:26.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:26.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:26.228147+0000) 2024-03-06T09:33:26.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:26.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:27.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:27.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:27.228457+0000) 2024-03-06T09:33:28.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:28.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:28.228793+0000) 2024-03-06T09:33:29.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:29.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:29.229105+0000) 2024-03-06T09:33:30.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:30.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:30.229415+0000) 2024-03-06T09:33:31.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:31.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:31.229760+0000) 2024-03-06T09:33:32.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:32.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:32.230073+0000) 2024-03-06T09:33:33.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:33.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:33.230360+0000) 2024-03-06T09:33:34.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:34.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:34.230737+0000) 2024-03-06T09:33:35.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:35.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:35.231068+0000) 2024-03-06T09:33:36.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:36.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:36.231320+0000) 2024-03-06T09:33:37.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:37.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:37.231575+0000) 2024-03-06T09:33:38.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:38.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:38.231837+0000) 2024-03-06T09:33:39.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:39.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:39.232085+0000) 2024-03-06T09:33:40.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:40.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:40.232313+0000) 2024-03-06T09:33:41.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:41.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:41.232615+0000) 2024-03-06T09:33:42.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:42.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:42.232982+0000) 2024-03-06T09:33:43.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:43.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:43.233277+0000) 2024-03-06T09:33:44.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:44.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:44.233595+0000) 2024-03-06T09:33:45.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:45.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:45.233926+0000) 2024-03-06T09:33:46.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:46.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:46.234233+0000) 2024-03-06T09:33:47.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:47.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:47.234539+0000) 2024-03-06T09:33:48.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:48.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:48.234845+0000) 2024-03-06T09:33:49.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:49.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:49.235146+0000) 2024-03-06T09:33:50.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:50.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:50.235446+0000) 2024-03-06T09:33:51.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:51.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:51.235782+0000) 2024-03-06T09:33:52.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:52.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:52.236110+0000) 2024-03-06T09:33:53.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:53.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:53.236432+0000) 2024-03-06T09:33:54.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:54.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:54.236831+0000) 2024-03-06T09:33:55.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:55.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:55.237127+0000) 2024-03-06T09:33:56.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:56.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:56.237433+0000) 2024-03-06T09:33:56.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:56.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:57.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:57.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:57.237764+0000) 2024-03-06T09:33:58.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:58.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:58.238005+0000) 2024-03-06T09:33:59.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:33:59.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:59.238265+0000) 2024-03-06T09:34:00.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:00.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:00.238524+0000) 2024-03-06T09:34:01.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:01.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:01.238776+0000) 2024-03-06T09:34:02.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:02.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:02.239042+0000) 2024-03-06T09:34:03.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:03.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:03.239309+0000) 2024-03-06T09:34:04.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:04.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:04.239508+0000) 2024-03-06T09:34:05.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:05.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:05.239759+0000) 2024-03-06T09:34:06.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:06.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:06.240020+0000) 2024-03-06T09:34:07.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:07.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:07.240289+0000) 2024-03-06T09:34:08.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:08.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:08.240552+0000) 2024-03-06T09:34:09.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:09.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:09.240804+0000) 2024-03-06T09:34:10.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:10.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:10.241058+0000) 2024-03-06T09:34:11.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:11.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:11.241290+0000) 2024-03-06T09:34:12.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:12.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:12.241539+0000) 2024-03-06T09:34:13.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:13.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:13.241807+0000) 2024-03-06T09:34:14.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:14.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:14.242036+0000) 2024-03-06T09:34:15.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:15.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:15.242292+0000) 2024-03-06T09:34:16.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:16.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:16.242557+0000) 2024-03-06T09:34:17.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:17.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:17.242767+0000) 2024-03-06T09:34:18.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:18.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:18.243040+0000) 2024-03-06T09:34:19.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:19.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:19.243294+0000) 2024-03-06T09:34:20.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:20.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:20.243550+0000) 2024-03-06T09:34:21.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:21.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:21.243795+0000) 2024-03-06T09:34:22.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:22.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:22.244043+0000) 2024-03-06T09:34:23.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:23.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:23.244327+0000) 2024-03-06T09:34:24.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:24.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:24.244587+0000) 2024-03-06T09:34:25.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:25.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:25.244837+0000) 2024-03-06T09:34:26.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:26.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:26.245084+0000) 2024-03-06T09:34:26.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:26.769 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:27.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:27.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:27.245349+0000) 2024-03-06T09:34:28.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:28.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:28.245598+0000) 2024-03-06T09:34:29.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:29.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:29.245856+0000) 2024-03-06T09:34:30.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:30.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:30.246122+0000) 2024-03-06T09:34:31.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:31.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:31.246397+0000) 2024-03-06T09:34:32.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:32.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:32.246754+0000) 2024-03-06T09:34:33.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:33.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:33.247061+0000) 2024-03-06T09:34:34.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:34.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:34.247361+0000) 2024-03-06T09:34:35.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:35.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:35.247716+0000) 2024-03-06T09:34:36.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:36.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:36.248038+0000) 2024-03-06T09:34:37.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:37.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:37.248349+0000) 2024-03-06T09:34:38.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:38.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:38.248706+0000) 2024-03-06T09:34:39.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:39.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:39.248975+0000) 2024-03-06T09:34:40.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:40.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:40.249222+0000) 2024-03-06T09:34:41.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:41.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:41.249460+0000) 2024-03-06T09:34:42.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:42.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:42.249775+0000) 2024-03-06T09:34:43.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:43.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:43.250047+0000) 2024-03-06T09:34:44.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:44.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:44.250343+0000) 2024-03-06T09:34:45.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:45.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:45.250700+0000) 2024-03-06T09:34:46.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:46.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:46.251030+0000) 2024-03-06T09:34:47.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:47.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:47.251346+0000) 2024-03-06T09:34:48.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:48.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:48.251705+0000) 2024-03-06T09:34:49.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:49.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:49.251992+0000) 2024-03-06T09:34:50.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:50.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:50.252288+0000) 2024-03-06T09:34:51.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:51.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:51.252586+0000) 2024-03-06T09:34:52.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:52.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:52.252930+0000) 2024-03-06T09:34:53.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:53.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:53.253247+0000) 2024-03-06T09:34:54.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:54.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:54.253561+0000) 2024-03-06T09:34:55.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:55.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:55.253868+0000) 2024-03-06T09:34:56.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:56.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:56.254201+0000) 2024-03-06T09:34:56.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:56.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:57.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:57.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:57.254505+0000) 2024-03-06T09:34:58.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:58.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:58.254807+0000) 2024-03-06T09:34:59.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:34:59.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:59.255133+0000) 2024-03-06T09:35:00.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:00.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:00.255434+0000) 2024-03-06T09:35:01.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:01.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:01.255781+0000) 2024-03-06T09:35:02.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:02.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:02.256084+0000) 2024-03-06T09:35:03.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:03.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:03.256376+0000) 2024-03-06T09:35:04.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:04.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:04.256723+0000) 2024-03-06T09:35:05.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:05.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:05.257019+0000) 2024-03-06T09:35:06.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:06.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:06.257345+0000) 2024-03-06T09:35:07.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:07.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:07.257599+0000) 2024-03-06T09:35:08.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:08.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:08.257906+0000) 2024-03-06T09:35:09.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:09.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:09.258141+0000) 2024-03-06T09:35:10.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:10.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:10.258395+0000) 2024-03-06T09:35:11.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:11.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:11.258687+0000) 2024-03-06T09:35:12.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:12.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:12.258952+0000) 2024-03-06T09:35:13.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:13.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:13.259195+0000) 2024-03-06T09:35:14.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:14.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:14.259451+0000) 2024-03-06T09:35:15.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:15.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:15.259744+0000) 2024-03-06T09:35:16.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:16.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:16.260013+0000) 2024-03-06T09:35:17.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:17.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:17.260257+0000) 2024-03-06T09:35:18.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:18.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:18.260526+0000) 2024-03-06T09:35:19.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:19.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:19.260779+0000) 2024-03-06T09:35:20.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:20.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:20.261037+0000) 2024-03-06T09:35:21.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:21.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:21.261304+0000) 2024-03-06T09:35:22.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:22.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:22.261624+0000) 2024-03-06T09:35:23.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:23.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:23.262005+0000) 2024-03-06T09:35:24.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:24.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:24.262319+0000) 2024-03-06T09:35:25.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:25.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:25.262676+0000) 2024-03-06T09:35:26.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:26.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:26.263035+0000) 2024-03-06T09:35:27.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:27.080 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:27.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:27.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:27.263343+0000) 2024-03-06T09:35:28.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:28.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:28.263723+0000) 2024-03-06T09:35:29.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:29.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:29.264007+0000) 2024-03-06T09:35:30.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:30.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:30.264319+0000) 2024-03-06T09:35:31.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:31.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:31.264606+0000) 2024-03-06T09:35:32.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:32.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:32.264960+0000) 2024-03-06T09:35:33.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:33.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:33.265272+0000) 2024-03-06T09:35:34.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:34.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:34.265560+0000) 2024-03-06T09:35:35.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:35.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:35.265859+0000) 2024-03-06T09:35:36.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:36.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:36.266187+0000) 2024-03-06T09:35:37.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:37.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:37.266475+0000) 2024-03-06T09:35:38.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:38.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:38.266790+0000) 2024-03-06T09:35:39.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:39.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:39.267109+0000) 2024-03-06T09:35:40.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:40.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:40.267416+0000) 2024-03-06T09:35:41.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:41.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:41.267746+0000) 2024-03-06T09:35:42.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:42.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:42.268032+0000) 2024-03-06T09:35:43.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:43.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:43.268333+0000) 2024-03-06T09:35:44.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:44.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:44.268683+0000) 2024-03-06T09:35:45.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:45.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:45.268996+0000) 2024-03-06T09:35:46.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:46.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:46.269328+0000) 2024-03-06T09:35:47.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:47.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:47.269673+0000) 2024-03-06T09:35:48.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:48.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:48.269970+0000) 2024-03-06T09:35:49.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:49.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:49.270268+0000) 2024-03-06T09:35:50.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:50.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:50.270579+0000) 2024-03-06T09:35:51.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:51.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:51.270898+0000) 2024-03-06T09:35:52.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:52.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:52.271214+0000) 2024-03-06T09:35:53.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:53.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:53.271510+0000) 2024-03-06T09:35:54.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:54.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:54.271803+0000) 2024-03-06T09:35:55.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:55.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:55.272083+0000) 2024-03-06T09:35:56.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:56.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:56.272319+0000) 2024-03-06T09:35:57.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:57.236 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:57.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:57.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:57.272577+0000) 2024-03-06T09:35:58.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:58.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:58.272897+0000) 2024-03-06T09:35:59.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:35:59.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:59.273148+0000) 2024-03-06T09:36:00.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:00.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:00.273457+0000) 2024-03-06T09:36:01.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:01.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:01.273784+0000) 2024-03-06T09:36:02.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:02.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:02.274089+0000) 2024-03-06T09:36:03.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:03.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:03.274296+0000) 2024-03-06T09:36:04.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:04.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:04.274552+0000) 2024-03-06T09:36:05.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:05.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:05.274810+0000) 2024-03-06T09:36:06.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:06.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:06.275076+0000) 2024-03-06T09:36:07.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:07.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:07.275315+0000) 2024-03-06T09:36:08.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:08.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:08.275509+0000) 2024-03-06T09:36:09.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:09.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:09.275743+0000) 2024-03-06T09:36:10.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:10.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:10.275931+0000) 2024-03-06T09:36:11.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:11.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:11.276176+0000) 2024-03-06T09:36:12.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:12.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:12.276432+0000) 2024-03-06T09:36:13.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:13.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:13.276715+0000) 2024-03-06T09:36:14.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:14.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:14.276962+0000) 2024-03-06T09:36:15.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:15.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:15.277215+0000) 2024-03-06T09:36:16.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:16.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:16.277482+0000) 2024-03-06T09:36:17.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:17.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:17.277746+0000) 2024-03-06T09:36:18.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:18.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:18.278009+0000) 2024-03-06T09:36:19.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:19.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:19.278237+0000) 2024-03-06T09:36:20.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:20.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:20.278494+0000) 2024-03-06T09:36:21.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:21.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:21.278746+0000) 2024-03-06T09:36:22.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:22.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:22.278972+0000) 2024-03-06T09:36:23.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:23.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:23.279240+0000) 2024-03-06T09:36:24.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:24.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:24.279435+0000) 2024-03-06T09:36:25.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:25.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:25.279718+0000) 2024-03-06T09:36:26.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:26.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:26.279975+0000) 2024-03-06T09:36:27.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:27.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:27.280228+0000) 2024-03-06T09:36:27.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:27.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:28.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:28.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:28.280487+0000) 2024-03-06T09:36:29.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:29.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:29.280740+0000) 2024-03-06T09:36:30.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:30.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:30.280971+0000) 2024-03-06T09:36:31.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:31.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:31.281223+0000) 2024-03-06T09:36:32.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:32.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:32.281481+0000) 2024-03-06T09:36:33.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:33.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:33.281752+0000) 2024-03-06T09:36:34.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:34.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:34.281992+0000) 2024-03-06T09:36:35.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:35.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:35.282240+0000) 2024-03-06T09:36:36.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:36.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:36.282505+0000) 2024-03-06T09:36:37.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:37.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:37.282740+0000) 2024-03-06T09:36:38.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:38.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:38.283002+0000) 2024-03-06T09:36:39.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:39.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:39.283190+0000) 2024-03-06T09:36:40.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:40.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:40.283464+0000) 2024-03-06T09:36:41.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:41.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:41.283775+0000) 2024-03-06T09:36:42.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:42.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:42.284094+0000) 2024-03-06T09:36:43.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:43.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:43.284357+0000) 2024-03-06T09:36:44.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:44.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:44.284604+0000) 2024-03-06T09:36:45.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:45.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:45.284933+0000) 2024-03-06T09:36:46.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:46.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:46.285207+0000) 2024-03-06T09:36:47.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:47.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:47.285454+0000) 2024-03-06T09:36:48.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:48.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:48.285742+0000) 2024-03-06T09:36:49.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:49.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:49.285971+0000) 2024-03-06T09:36:50.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:50.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:50.286214+0000) 2024-03-06T09:36:51.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:51.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:51.286443+0000) 2024-03-06T09:36:52.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:52.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:52.286735+0000) 2024-03-06T09:36:53.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:53.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:53.286926+0000) 2024-03-06T09:36:54.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:54.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:54.287166+0000) 2024-03-06T09:36:55.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:55.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:55.287420+0000) 2024-03-06T09:36:56.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:56.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:56.287730+0000) 2024-03-06T09:36:57.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:57.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:57.287927+0000) 2024-03-06T09:36:57.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:57.522 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:58.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:58.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:58.288165+0000) 2024-03-06T09:36:59.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:36:59.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:59.288421+0000) 2024-03-06T09:37:00.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:00.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:00.288690+0000) 2024-03-06T09:37:01.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:01.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:01.288950+0000) 2024-03-06T09:37:02.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:02.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:02.289214+0000) 2024-03-06T09:37:03.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:03.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:03.289471+0000) 2024-03-06T09:37:04.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:04.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:04.289758+0000) 2024-03-06T09:37:05.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:05.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:05.290008+0000) 2024-03-06T09:37:06.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:06.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:06.290274+0000) 2024-03-06T09:37:07.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:07.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:07.290505+0000) 2024-03-06T09:37:08.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:08.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:08.290767+0000) 2024-03-06T09:37:09.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:09.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:09.291046+0000) 2024-03-06T09:37:10.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:10.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:10.291305+0000) 2024-03-06T09:37:11.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:11.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:11.291447+0000) 2024-03-06T09:37:12.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:12.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:12.291740+0000) 2024-03-06T09:37:13.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:13.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:13.291995+0000) 2024-03-06T09:37:14.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:14.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:14.292253+0000) 2024-03-06T09:37:15.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:15.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:15.292504+0000) 2024-03-06T09:37:16.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:16.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:16.292779+0000) 2024-03-06T09:37:17.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:17.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:17.293030+0000) 2024-03-06T09:37:18.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:18.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:18.293302+0000) 2024-03-06T09:37:19.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:19.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:19.293559+0000) 2024-03-06T09:37:20.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:20.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:20.293815+0000) 2024-03-06T09:37:21.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:21.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:21.294022+0000) 2024-03-06T09:37:22.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:22.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:22.294286+0000) 2024-03-06T09:37:23.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:23.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:23.294562+0000) 2024-03-06T09:37:24.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:24.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:24.294840+0000) 2024-03-06T09:37:25.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:25.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:25.295041+0000) 2024-03-06T09:37:26.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:26.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:26.295306+0000) 2024-03-06T09:37:27.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:27.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:27.295546+0000) 2024-03-06T09:37:27.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:27.680 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:28.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:28.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:28.295757+0000) 2024-03-06T09:37:29.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:29.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:29.296007+0000) 2024-03-06T09:37:30.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:30.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:30.296201+0000) 2024-03-06T09:37:31.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:31.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:31.296434+0000) 2024-03-06T09:37:32.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:32.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:32.296720+0000) 2024-03-06T09:37:33.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:33.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:33.296983+0000) 2024-03-06T09:37:34.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:34.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:34.297228+0000) 2024-03-06T09:37:35.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:35.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:35.297478+0000) 2024-03-06T09:37:36.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:36.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:36.297750+0000) 2024-03-06T09:37:37.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:37.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:37.297983+0000) 2024-03-06T09:37:38.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:38.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:38.298247+0000) 2024-03-06T09:37:39.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:39.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:39.298505+0000) 2024-03-06T09:37:40.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:40.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:40.298761+0000) 2024-03-06T09:37:41.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:41.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:41.298961+0000) 2024-03-06T09:37:42.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:42.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:42.299222+0000) 2024-03-06T09:37:43.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:43.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:43.299490+0000) 2024-03-06T09:37:44.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:44.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:44.299755+0000) 2024-03-06T09:37:45.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:45.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:45.300003+0000) 2024-03-06T09:37:46.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:46.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:46.300248+0000) 2024-03-06T09:37:47.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:47.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:47.300498+0000) 2024-03-06T09:37:48.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:48.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:48.300761+0000) 2024-03-06T09:37:49.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:49.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:49.300995+0000) 2024-03-06T09:37:50.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:50.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:50.301232+0000) 2024-03-06T09:37:51.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:51.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:51.301474+0000) 2024-03-06T09:37:52.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:52.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:52.301755+0000) 2024-03-06T09:37:53.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:53.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:53.302007+0000) 2024-03-06T09:37:54.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:54.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:54.302245+0000) 2024-03-06T09:37:55.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:55.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:55.302494+0000) 2024-03-06T09:37:56.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:56.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:56.302756+0000) 2024-03-06T09:37:57.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:57.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:57.302939+0000) 2024-03-06T09:37:57.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:57.836 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:58.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:58.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:58.303203+0000) 2024-03-06T09:37:59.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:37:59.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:59.303467+0000) 2024-03-06T09:38:00.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:00.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:00.303755+0000) 2024-03-06T09:38:01.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:01.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:01.304004+0000) 2024-03-06T09:38:02.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:02.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:02.304264+0000) 2024-03-06T09:38:03.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:03.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:03.304520+0000) 2024-03-06T09:38:04.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:04.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:04.304768+0000) 2024-03-06T09:38:05.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:05.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:05.305024+0000) 2024-03-06T09:38:06.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:06.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:06.305257+0000) 2024-03-06T09:38:07.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:07.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:07.305495+0000) 2024-03-06T09:38:08.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:08.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:08.305747+0000) 2024-03-06T09:38:09.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:09.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:09.305985+0000) 2024-03-06T09:38:10.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:10.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:10.306212+0000) 2024-03-06T09:38:11.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:11.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:11.306452+0000) 2024-03-06T09:38:12.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:12.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:12.306732+0000) 2024-03-06T09:38:13.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:13.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:13.306994+0000) 2024-03-06T09:38:14.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:14.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:14.307255+0000) 2024-03-06T09:38:15.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:15.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:15.307488+0000) 2024-03-06T09:38:16.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:16.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:16.307742+0000) 2024-03-06T09:38:17.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:17.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:17.307975+0000) 2024-03-06T09:38:18.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:18.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:18.308211+0000) 2024-03-06T09:38:19.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:19.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:19.308459+0000) 2024-03-06T09:38:20.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:20.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:20.308739+0000) 2024-03-06T09:38:21.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:21.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:21.308990+0000) 2024-03-06T09:38:22.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:22.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:22.309241+0000) 2024-03-06T09:38:23.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:23.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:23.309485+0000) 2024-03-06T09:38:24.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:24.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:24.309751+0000) 2024-03-06T09:38:25.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:25.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:25.309927+0000) 2024-03-06T09:38:26.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:26.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:26.310188+0000) 2024-03-06T09:38:27.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:27.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:27.310441+0000) 2024-03-06T09:38:27.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:27.992 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:28.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:28.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:28.310733+0000) 2024-03-06T09:38:29.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:29.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:29.310979+0000) 2024-03-06T09:38:30.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:30.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:30.311216+0000) 2024-03-06T09:38:31.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:31.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:31.311457+0000) 2024-03-06T09:38:32.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:32.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:32.311748+0000) 2024-03-06T09:38:33.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:33.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:33.311983+0000) 2024-03-06T09:38:34.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:34.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:34.312243+0000) 2024-03-06T09:38:35.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:35.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:35.312478+0000) 2024-03-06T09:38:36.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:36.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:36.312747+0000) 2024-03-06T09:38:37.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:37.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:37.313054+0000) 2024-03-06T09:38:38.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:38.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:38.313300+0000) 2024-03-06T09:38:39.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:39.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:39.313489+0000) 2024-03-06T09:38:40.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:40.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:40.313752+0000) 2024-03-06T09:38:41.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:41.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:41.314007+0000) 2024-03-06T09:38:42.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:42.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:42.314178+0000) 2024-03-06T09:38:43.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:43.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:43.314423+0000) 2024-03-06T09:38:44.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:44.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:44.314707+0000) 2024-03-06T09:38:45.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:45.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:45.314942+0000) 2024-03-06T09:38:46.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:46.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:46.315188+0000) 2024-03-06T09:38:47.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:47.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:47.315449+0000) 2024-03-06T09:38:48.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:48.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:48.315747+0000) 2024-03-06T09:38:49.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:49.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:49.315997+0000) 2024-03-06T09:38:50.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:50.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:50.316230+0000) 2024-03-06T09:38:51.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:51.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:51.316469+0000) 2024-03-06T09:38:52.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:52.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:52.316745+0000) 2024-03-06T09:38:53.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:53.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:53.317008+0000) 2024-03-06T09:38:54.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:54.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:54.317259+0000) 2024-03-06T09:38:55.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:55.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:55.317518+0000) 2024-03-06T09:38:56.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:56.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:56.317765+0000) 2024-03-06T09:38:57.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:57.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:57.318000+0000) 2024-03-06T09:38:58.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:58.136 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:58.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:58.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:58.318204+0000) 2024-03-06T09:38:59.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:38:59.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:59.318462+0000) 2024-03-06T09:39:00.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:00.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:00.318604+0000) 2024-03-06T09:39:01.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:01.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:01.318824+0000) 2024-03-06T09:39:02.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:02.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:02.319060+0000) 2024-03-06T09:39:03.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:03.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:03.319327+0000) 2024-03-06T09:39:04.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:04.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:04.319596+0000) 2024-03-06T09:39:05.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:05.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:05.319829+0000) 2024-03-06T09:39:06.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:06.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:06.320092+0000) 2024-03-06T09:39:07.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:07.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:07.320348+0000) 2024-03-06T09:39:08.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:08.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:08.320616+0000) 2024-03-06T09:39:09.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:09.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:09.320929+0000) 2024-03-06T09:39:10.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:10.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:10.321167+0000) 2024-03-06T09:39:11.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:11.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:11.321424+0000) 2024-03-06T09:39:12.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:12.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:12.321725+0000) 2024-03-06T09:39:13.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:13.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:13.321992+0000) 2024-03-06T09:39:14.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:14.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:14.322267+0000) 2024-03-06T09:39:15.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:15.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:15.322518+0000) 2024-03-06T09:39:16.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:16.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:16.322763+0000) 2024-03-06T09:39:17.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:17.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:17.323014+0000) 2024-03-06T09:39:18.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:18.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:18.323273+0000) 2024-03-06T09:39:19.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:19.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:19.323522+0000) 2024-03-06T09:39:20.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:20.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:20.323741+0000) 2024-03-06T09:39:21.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:21.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:21.323998+0000) 2024-03-06T09:39:22.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:22.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:22.324245+0000) 2024-03-06T09:39:23.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:23.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:23.324494+0000) 2024-03-06T09:39:24.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:24.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:24.324748+0000) 2024-03-06T09:39:25.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:25.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:25.325006+0000) 2024-03-06T09:39:26.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:26.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:26.325268+0000) 2024-03-06T09:39:27.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:27.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:27.325521+0000) 2024-03-06T09:39:28.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:28.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:28.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:28.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:28.325730+0000) 2024-03-06T09:39:29.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:29.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:29.325963+0000) 2024-03-06T09:39:30.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:30.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:30.326217+0000) 2024-03-06T09:39:31.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:31.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:31.326442+0000) 2024-03-06T09:39:32.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:32.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:32.326749+0000) 2024-03-06T09:39:33.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:33.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:33.326951+0000) 2024-03-06T09:39:34.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:34.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:34.327216+0000) 2024-03-06T09:39:35.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:35.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:35.327517+0000) 2024-03-06T09:39:36.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:36.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:36.327807+0000) 2024-03-06T09:39:37.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:37.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:37.328051+0000) 2024-03-06T09:39:38.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:38.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:38.328332+0000) 2024-03-06T09:39:39.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:39.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:39.328617+0000) 2024-03-06T09:39:40.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:40.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:40.328953+0000) 2024-03-06T09:39:41.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:41.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:41.329208+0000) 2024-03-06T09:39:42.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:42.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:42.329479+0000) 2024-03-06T09:39:43.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:43.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:43.329750+0000) 2024-03-06T09:39:44.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:44.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:44.330033+0000) 2024-03-06T09:39:45.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:45.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:45.330304+0000) 2024-03-06T09:39:46.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:46.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:46.330589+0000) 2024-03-06T09:39:47.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:47.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:47.330881+0000) 2024-03-06T09:39:48.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:48.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:48.331143+0000) 2024-03-06T09:39:49.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:49.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:49.331412+0000) 2024-03-06T09:39:50.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:50.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:50.331728+0000) 2024-03-06T09:39:51.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:51.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:51.332008+0000) 2024-03-06T09:39:52.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:52.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:52.332278+0000) 2024-03-06T09:39:53.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:53.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:53.332515+0000) 2024-03-06T09:39:54.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:54.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:54.332784+0000) 2024-03-06T09:39:55.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:55.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:55.333036+0000) 2024-03-06T09:39:56.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:56.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:56.333290+0000) 2024-03-06T09:39:57.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:57.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:57.333556+0000) 2024-03-06T09:39:58.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:58.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:58.333816+0000) 2024-03-06T09:39:58.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:58.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:59.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:39:59.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:59.334056+0000) 2024-03-06T09:40:00.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:00.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:00.334305+0000) 2024-03-06T09:40:01.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:01.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:01.334546+0000) 2024-03-06T09:40:02.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:02.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:02.334787+0000) 2024-03-06T09:40:03.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:03.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:03.335042+0000) 2024-03-06T09:40:04.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:04.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:04.335301+0000) 2024-03-06T09:40:05.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:05.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:05.335553+0000) 2024-03-06T09:40:06.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:06.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:06.335799+0000) 2024-03-06T09:40:07.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:07.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:07.336056+0000) 2024-03-06T09:40:08.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:08.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:08.336319+0000) 2024-03-06T09:40:09.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:09.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:09.336553+0000) 2024-03-06T09:40:10.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:10.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:10.336796+0000) 2024-03-06T09:40:11.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:11.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:11.337083+0000) 2024-03-06T09:40:12.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:12.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:12.337396+0000) 2024-03-06T09:40:13.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:13.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:13.337754+0000) 2024-03-06T09:40:14.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:14.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:14.338078+0000) 2024-03-06T09:40:15.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:15.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:15.338378+0000) 2024-03-06T09:40:16.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:16.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:16.338743+0000) 2024-03-06T09:40:17.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:17.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:17.338973+0000) 2024-03-06T09:40:18.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:18.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:18.339274+0000) 2024-03-06T09:40:19.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:19.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:19.339575+0000) 2024-03-06T09:40:20.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:20.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:20.339889+0000) 2024-03-06T09:40:21.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:21.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:21.340126+0000) 2024-03-06T09:40:22.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:22.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:22.340421+0000) 2024-03-06T09:40:23.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:23.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:23.340828+0000) 2024-03-06T09:40:24.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:24.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:24.341139+0000) 2024-03-06T09:40:25.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:25.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:25.341424+0000) 2024-03-06T09:40:26.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:26.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:26.341781+0000) 2024-03-06T09:40:27.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:27.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:27.342089+0000) 2024-03-06T09:40:28.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:28.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:28.342383+0000) 2024-03-06T09:40:28.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:28.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:29.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:29.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:29.342728+0000) 2024-03-06T09:40:30.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:30.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:30.343054+0000) 2024-03-06T09:40:31.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:31.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:31.343369+0000) 2024-03-06T09:40:32.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:32.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:32.343735+0000) 2024-03-06T09:40:33.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:33.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:33.344046+0000) 2024-03-06T09:40:34.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:34.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:34.344341+0000) 2024-03-06T09:40:35.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:35.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:35.344671+0000) 2024-03-06T09:40:36.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:36.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:36.345017+0000) 2024-03-06T09:40:37.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:37.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:37.345317+0000) 2024-03-06T09:40:38.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:38.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:38.345622+0000) 2024-03-06T09:40:39.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:39.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:39.345931+0000) 2024-03-06T09:40:40.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:40.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:40.346187+0000) 2024-03-06T09:40:41.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:41.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:41.346423+0000) 2024-03-06T09:40:42.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:42.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:42.346708+0000) 2024-03-06T09:40:43.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:43.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:43.346953+0000) 2024-03-06T09:40:44.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:44.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:44.347189+0000) 2024-03-06T09:40:45.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:45.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:45.347515+0000) 2024-03-06T09:40:46.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:46.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:46.347824+0000) 2024-03-06T09:40:47.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:47.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:47.348138+0000) 2024-03-06T09:40:48.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:48.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:48.348428+0000) 2024-03-06T09:40:49.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:49.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:49.348764+0000) 2024-03-06T09:40:50.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:50.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:50.349080+0000) 2024-03-06T09:40:51.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:51.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:51.349389+0000) 2024-03-06T09:40:52.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:52.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:52.349730+0000) 2024-03-06T09:40:53.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:53.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:53.350023+0000) 2024-03-06T09:40:54.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:54.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:54.350324+0000) 2024-03-06T09:40:55.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:55.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:55.350624+0000) 2024-03-06T09:40:56.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:56.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:56.351024+0000) 2024-03-06T09:40:57.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:57.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:57.351248+0000) 2024-03-06T09:40:58.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:58.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:58.351555+0000) 2024-03-06T09:40:58.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:58.723 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:59.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:40:59.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:59.351830+0000) 2024-03-06T09:41:00.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:00.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:00.352068+0000) 2024-03-06T09:41:01.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:01.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:01.352316+0000) 2024-03-06T09:41:02.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:02.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:02.352564+0000) 2024-03-06T09:41:03.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:03.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:03.352850+0000) 2024-03-06T09:41:04.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:04.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:04.353161+0000) 2024-03-06T09:41:05.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:05.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:05.353454+0000) 2024-03-06T09:41:06.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:06.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:06.353796+0000) 2024-03-06T09:41:07.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:07.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:07.354101+0000) 2024-03-06T09:41:08.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:08.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:08.354386+0000) 2024-03-06T09:41:09.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:09.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:09.354734+0000) 2024-03-06T09:41:10.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:10.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:10.355058+0000) 2024-03-06T09:41:11.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:11.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:11.355379+0000) 2024-03-06T09:41:12.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:12.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:12.355610+0000) 2024-03-06T09:41:13.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:13.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:13.355973+0000) 2024-03-06T09:41:14.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:14.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:14.356436+0000) 2024-03-06T09:41:15.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:15.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:15.356712+0000) 2024-03-06T09:41:16.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:16.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:16.356972+0000) 2024-03-06T09:41:17.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:17.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:17.357224+0000) 2024-03-06T09:41:18.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:18.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:18.357475+0000) 2024-03-06T09:41:19.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:19.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:19.357749+0000) 2024-03-06T09:41:20.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:20.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:20.357989+0000) 2024-03-06T09:41:21.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:21.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:21.358240+0000) 2024-03-06T09:41:22.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:22.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:22.358492+0000) 2024-03-06T09:41:23.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:23.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:23.358744+0000) 2024-03-06T09:41:24.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:24.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:24.358982+0000) 2024-03-06T09:41:25.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:25.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:25.359169+0000) 2024-03-06T09:41:26.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:26.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:26.359359+0000) 2024-03-06T09:41:27.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:27.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:27.359620+0000) 2024-03-06T09:41:28.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:28.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:28.359894+0000) 2024-03-06T09:41:28.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:28.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:29.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:29.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:29.360084+0000) 2024-03-06T09:41:30.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:30.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:30.360327+0000) 2024-03-06T09:41:31.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:31.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:31.360569+0000) 2024-03-06T09:41:32.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:32.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:32.360808+0000) 2024-03-06T09:41:33.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:33.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:33.361043+0000) 2024-03-06T09:41:34.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:34.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:34.361278+0000) 2024-03-06T09:41:35.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:35.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:35.361506+0000) 2024-03-06T09:41:36.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:36.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:36.361775+0000) 2024-03-06T09:41:37.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:37.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:37.362006+0000) 2024-03-06T09:41:38.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:38.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:38.362318+0000) 2024-03-06T09:41:39.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:39.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:39.362547+0000) 2024-03-06T09:41:40.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:40.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:40.362756+0000) 2024-03-06T09:41:41.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:41.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:41.362948+0000) 2024-03-06T09:41:42.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:42.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:42.363211+0000) 2024-03-06T09:41:43.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:43.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:43.363417+0000) 2024-03-06T09:41:44.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:44.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:44.363721+0000) 2024-03-06T09:41:45.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:45.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:45.363976+0000) 2024-03-06T09:41:46.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:46.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:46.364234+0000) 2024-03-06T09:41:47.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:47.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:47.364465+0000) 2024-03-06T09:41:48.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:48.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:48.364687+0000) 2024-03-06T09:41:49.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:49.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:49.364928+0000) 2024-03-06T09:41:50.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:50.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:50.365164+0000) 2024-03-06T09:41:51.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:51.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:51.365419+0000) 2024-03-06T09:41:52.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:52.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:52.365725+0000) 2024-03-06T09:41:53.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:53.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:53.365982+0000) 2024-03-06T09:41:54.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:54.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:54.366181+0000) 2024-03-06T09:41:55.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:55.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:55.366430+0000) 2024-03-06T09:41:56.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:56.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:56.366743+0000) 2024-03-06T09:41:57.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:57.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:57.366975+0000) 2024-03-06T09:41:58.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:58.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:58.367226+0000) 2024-03-06T09:41:59.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:59.019 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:59.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:41:59.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:59.367493+0000) 2024-03-06T09:42:00.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:00.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:00.367752+0000) 2024-03-06T09:42:01.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:01.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:01.367984+0000) 2024-03-06T09:42:02.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:02.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:02.368225+0000) 2024-03-06T09:42:03.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:03.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:03.368488+0000) 2024-03-06T09:42:04.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:04.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:04.368793+0000) 2024-03-06T09:42:05.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:05.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:05.369093+0000) 2024-03-06T09:42:06.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:06.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:06.369401+0000) 2024-03-06T09:42:07.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:07.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:07.369748+0000) 2024-03-06T09:42:08.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:08.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:08.370064+0000) 2024-03-06T09:42:09.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:09.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:09.370304+0000) 2024-03-06T09:42:10.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:10.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:10.370613+0000) 2024-03-06T09:42:11.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:11.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:11.370970+0000) 2024-03-06T09:42:12.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:12.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:12.371293+0000) 2024-03-06T09:42:13.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:13.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:13.371538+0000) 2024-03-06T09:42:14.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:14.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:14.371839+0000) 2024-03-06T09:42:15.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:15.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:15.372150+0000) 2024-03-06T09:42:16.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:16.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:16.372478+0000) 2024-03-06T09:42:17.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:17.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:17.372777+0000) 2024-03-06T09:42:18.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:18.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:18.373056+0000) 2024-03-06T09:42:19.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:19.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:19.373366+0000) 2024-03-06T09:42:20.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:20.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:20.373721+0000) 2024-03-06T09:42:21.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:21.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:21.374059+0000) 2024-03-06T09:42:22.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:22.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:22.374369+0000) 2024-03-06T09:42:23.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:23.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:23.374715+0000) 2024-03-06T09:42:24.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:24.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:24.374967+0000) 2024-03-06T09:42:25.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:25.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:25.375283+0000) 2024-03-06T09:42:26.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:26.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:26.375599+0000) 2024-03-06T09:42:27.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:27.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:27.375935+0000) 2024-03-06T09:42:28.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:28.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:28.376230+0000) 2024-03-06T09:42:29.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:29.174 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:29.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:29.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:29.376506+0000) 2024-03-06T09:42:30.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:30.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:30.376770+0000) 2024-03-06T09:42:31.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:31.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:31.376996+0000) 2024-03-06T09:42:32.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:32.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:32.377234+0000) 2024-03-06T09:42:33.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:33.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:33.377483+0000) 2024-03-06T09:42:34.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:34.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:34.377740+0000) 2024-03-06T09:42:35.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:35.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:35.377995+0000) 2024-03-06T09:42:36.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:36.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:36.378265+0000) 2024-03-06T09:42:37.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:37.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:37.378503+0000) 2024-03-06T09:42:38.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:38.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:38.378778+0000) 2024-03-06T09:42:39.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:39.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:39.378986+0000) 2024-03-06T09:42:40.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:40.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:40.379244+0000) 2024-03-06T09:42:41.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:41.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:41.379491+0000) 2024-03-06T09:42:42.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:42.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:42.379752+0000) 2024-03-06T09:42:43.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:43.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:43.380036+0000) 2024-03-06T09:42:44.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:44.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:44.380342+0000) 2024-03-06T09:42:45.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:45.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:45.380694+0000) 2024-03-06T09:42:46.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:46.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:46.381050+0000) 2024-03-06T09:42:47.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:47.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:47.381335+0000) 2024-03-06T09:42:48.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:48.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:48.381669+0000) 2024-03-06T09:42:49.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:49.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:49.381963+0000) 2024-03-06T09:42:50.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:50.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:50.382284+0000) 2024-03-06T09:42:51.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:51.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:51.382569+0000) 2024-03-06T09:42:52.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:52.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:52.382893+0000) 2024-03-06T09:42:53.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:53.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:53.383166+0000) 2024-03-06T09:42:54.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:54.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:54.383447+0000) 2024-03-06T09:42:55.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:55.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:55.383730+0000) 2024-03-06T09:42:56.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:56.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:56.383984+0000) 2024-03-06T09:42:57.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:57.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:57.384226+0000) 2024-03-06T09:42:58.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:58.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:58.384479+0000) 2024-03-06T09:42:59.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:59.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:59.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:42:59.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:59.384668+0000) 2024-03-06T09:43:00.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:00.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:00.384927+0000) 2024-03-06T09:43:01.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:01.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:01.385176+0000) 2024-03-06T09:43:02.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:02.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:02.385431+0000) 2024-03-06T09:43:03.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:03.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:03.385721+0000) 2024-03-06T09:43:04.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:04.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:04.385969+0000) 2024-03-06T09:43:05.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:05.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:05.386218+0000) 2024-03-06T09:43:06.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:06.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:06.386483+0000) 2024-03-06T09:43:07.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:07.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:07.386743+0000) 2024-03-06T09:43:08.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:08.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:08.386995+0000) 2024-03-06T09:43:09.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:09.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:09.387259+0000) 2024-03-06T09:43:10.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:10.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:10.387521+0000) 2024-03-06T09:43:11.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:11.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:11.387775+0000) 2024-03-06T09:43:12.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:12.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:12.387958+0000) 2024-03-06T09:43:13.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:13.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:13.388210+0000) 2024-03-06T09:43:14.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:14.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:14.388515+0000) 2024-03-06T09:43:15.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:15.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:15.388774+0000) 2024-03-06T09:43:16.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:16.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:16.389037+0000) 2024-03-06T09:43:17.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:17.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:17.389289+0000) 2024-03-06T09:43:18.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:18.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:18.389546+0000) 2024-03-06T09:43:19.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:19.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:19.389806+0000) 2024-03-06T09:43:20.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:20.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:20.390051+0000) 2024-03-06T09:43:21.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:21.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:21.390222+0000) 2024-03-06T09:43:22.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:22.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:22.390455+0000) 2024-03-06T09:43:23.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:23.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:23.390745+0000) 2024-03-06T09:43:24.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:24.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:24.390989+0000) 2024-03-06T09:43:25.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:25.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:25.391246+0000) 2024-03-06T09:43:26.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:26.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:26.391508+0000) 2024-03-06T09:43:27.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:27.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:27.391739+0000) 2024-03-06T09:43:28.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:28.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:28.391996+0000) 2024-03-06T09:43:29.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:29.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:29.392254+0000) 2024-03-06T09:43:29.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:29.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:30.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:30.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:30.392512+0000) 2024-03-06T09:43:31.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:31.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:31.392736+0000) 2024-03-06T09:43:32.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:32.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:32.392983+0000) 2024-03-06T09:43:33.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:33.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:33.393214+0000) 2024-03-06T09:43:34.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:34.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:34.393484+0000) 2024-03-06T09:43:35.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:35.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:35.393755+0000) 2024-03-06T09:43:36.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:36.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:36.394023+0000) 2024-03-06T09:43:37.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:37.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:37.394274+0000) 2024-03-06T09:43:38.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:38.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:38.394534+0000) 2024-03-06T09:43:39.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:39.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:39.394800+0000) 2024-03-06T09:43:40.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:40.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:40.395058+0000) 2024-03-06T09:43:41.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:41.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:41.395315+0000) 2024-03-06T09:43:42.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:42.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:42.395548+0000) 2024-03-06T09:43:43.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:43.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:43.395781+0000) 2024-03-06T09:43:44.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:44.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:44.396043+0000) 2024-03-06T09:43:45.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:45.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:45.396293+0000) 2024-03-06T09:43:46.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:46.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:46.396538+0000) 2024-03-06T09:43:47.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:47.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:47.396788+0000) 2024-03-06T09:43:48.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:48.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:48.397045+0000) 2024-03-06T09:43:49.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:49.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:49.397308+0000) 2024-03-06T09:43:50.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:50.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:50.397578+0000) 2024-03-06T09:43:51.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:51.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:51.397831+0000) 2024-03-06T09:43:52.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:52.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:52.398065+0000) 2024-03-06T09:43:53.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:53.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:53.398294+0000) 2024-03-06T09:43:54.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:54.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:54.398540+0000) 2024-03-06T09:43:55.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:55.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:55.398800+0000) 2024-03-06T09:43:56.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:56.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:56.399050+0000) 2024-03-06T09:43:57.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:57.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:57.399302+0000) 2024-03-06T09:43:58.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:58.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:58.399531+0000) 2024-03-06T09:43:59.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:43:59.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:59.399777+0000) 2024-03-06T09:43:59.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:59.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:00.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:00.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:00.400044+0000) 2024-03-06T09:44:01.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:01.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:01.400293+0000) 2024-03-06T09:44:02.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:02.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:02.400528+0000) 2024-03-06T09:44:03.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:03.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:03.400782+0000) 2024-03-06T09:44:04.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:04.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:04.401046+0000) 2024-03-06T09:44:05.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:05.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:05.401309+0000) 2024-03-06T09:44:06.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:06.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:06.401559+0000) 2024-03-06T09:44:07.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:07.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:07.401815+0000) 2024-03-06T09:44:08.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:08.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:08.402058+0000) 2024-03-06T09:44:09.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:09.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:09.402323+0000) 2024-03-06T09:44:10.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:10.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:10.402590+0000) 2024-03-06T09:44:11.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:11.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:11.402823+0000) 2024-03-06T09:44:12.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:12.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:12.403018+0000) 2024-03-06T09:44:13.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:13.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:13.403296+0000) 2024-03-06T09:44:14.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:14.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:14.403543+0000) 2024-03-06T09:44:15.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:15.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:15.403794+0000) 2024-03-06T09:44:16.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:16.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:16.404065+0000) 2024-03-06T09:44:17.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:17.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:17.404296+0000) 2024-03-06T09:44:18.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:18.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:18.404530+0000) 2024-03-06T09:44:19.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:19.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:19.404784+0000) 2024-03-06T09:44:20.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:20.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:20.405045+0000) 2024-03-06T09:44:21.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:21.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:21.405298+0000) 2024-03-06T09:44:22.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:22.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:22.405529+0000) 2024-03-06T09:44:23.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:23.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:23.405763+0000) 2024-03-06T09:44:24.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:24.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:24.406011+0000) 2024-03-06T09:44:25.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:25.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:25.406266+0000) 2024-03-06T09:44:26.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:26.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:26.406523+0000) 2024-03-06T09:44:27.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:27.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:27.406772+0000) 2024-03-06T09:44:28.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:28.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:28.407020+0000) 2024-03-06T09:44:29.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:29.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:29.407302+0000) 2024-03-06T09:44:29.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:29.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:30.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:30.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:30.407614+0000) 2024-03-06T09:44:31.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:31.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:31.407956+0000) 2024-03-06T09:44:32.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:32.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:32.408269+0000) 2024-03-06T09:44:33.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:33.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:33.408497+0000) 2024-03-06T09:44:34.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:34.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:34.408801+0000) 2024-03-06T09:44:35.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:35.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:35.409115+0000) 2024-03-06T09:44:36.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:36.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:36.409438+0000) 2024-03-06T09:44:37.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:37.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:37.409788+0000) 2024-03-06T09:44:38.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:38.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:38.410062+0000) 2024-03-06T09:44:39.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:39.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:39.410320+0000) 2024-03-06T09:44:40.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:40.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:40.410579+0000) 2024-03-06T09:44:41.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:41.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:41.410813+0000) 2024-03-06T09:44:42.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:42.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:42.411088+0000) 2024-03-06T09:44:43.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:43.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:43.411401+0000) 2024-03-06T09:44:44.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:44.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:44.411759+0000) 2024-03-06T09:44:45.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:45.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:45.412062+0000) 2024-03-06T09:44:46.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:46.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:46.412385+0000) 2024-03-06T09:44:47.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:47.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:47.412731+0000) 2024-03-06T09:44:48.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:48.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:48.413051+0000) 2024-03-06T09:44:49.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:49.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:49.413349+0000) 2024-03-06T09:44:50.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:50.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:50.413700+0000) 2024-03-06T09:44:51.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:51.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:51.414009+0000) 2024-03-06T09:44:52.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:52.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:52.414304+0000) 2024-03-06T09:44:53.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:53.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:53.414589+0000) 2024-03-06T09:44:54.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:54.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:54.414963+0000) 2024-03-06T09:44:55.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:55.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:55.415276+0000) 2024-03-06T09:44:56.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:56.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:56.415576+0000) 2024-03-06T09:44:57.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:57.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:57.415859+0000) 2024-03-06T09:44:58.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:58.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:58.416112+0000) 2024-03-06T09:44:59.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:44:59.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:59.416426+0000) 2024-03-06T09:44:59.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:59.936 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:00.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:00.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:00.416794+0000) 2024-03-06T09:45:01.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:01.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:01.417187+0000) 2024-03-06T09:45:02.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:02.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:02.417495+0000) 2024-03-06T09:45:03.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:03.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:03.417809+0000) 2024-03-06T09:45:04.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:04.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:04.418124+0000) 2024-03-06T09:45:05.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:05.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:05.418406+0000) 2024-03-06T09:45:06.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:06.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:06.418766+0000) 2024-03-06T09:45:07.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:07.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:07.419035+0000) 2024-03-06T09:45:08.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:08.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:08.419350+0000) 2024-03-06T09:45:09.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:09.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:09.419655+0000) 2024-03-06T09:45:10.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:10.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:10.419949+0000) 2024-03-06T09:45:11.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:11.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:11.420239+0000) 2024-03-06T09:45:12.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:12.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:12.420531+0000) 2024-03-06T09:45:13.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:13.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:13.420824+0000) 2024-03-06T09:45:14.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:14.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:14.421130+0000) 2024-03-06T09:45:15.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:15.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:15.421359+0000) 2024-03-06T09:45:16.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:16.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:16.421705+0000) 2024-03-06T09:45:17.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:17.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:17.421993+0000) 2024-03-06T09:45:18.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:18.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:18.422298+0000) 2024-03-06T09:45:19.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:19.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:19.422599+0000) 2024-03-06T09:45:20.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:20.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:20.422969+0000) 2024-03-06T09:45:21.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:21.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:21.423258+0000) 2024-03-06T09:45:22.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:22.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:22.423556+0000) 2024-03-06T09:45:23.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:23.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:23.423851+0000) 2024-03-06T09:45:24.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:24.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:24.424144+0000) 2024-03-06T09:45:25.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:25.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:25.424440+0000) 2024-03-06T09:45:26.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:26.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:26.424785+0000) 2024-03-06T09:45:27.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:27.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:27.425076+0000) 2024-03-06T09:45:28.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:28.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:28.425370+0000) 2024-03-06T09:45:29.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:29.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:29.425718+0000) 2024-03-06T09:45:30.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:30.082 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:30.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:30.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:30.425967+0000) 2024-03-06T09:45:31.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:31.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:31.426271+0000) 2024-03-06T09:45:32.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:32.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:32.426583+0000) 2024-03-06T09:45:33.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:33.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:33.426902+0000) 2024-03-06T09:45:34.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:34.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:34.427159+0000) 2024-03-06T09:45:35.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:35.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:35.427466+0000) 2024-03-06T09:45:36.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:36.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:36.427783+0000) 2024-03-06T09:45:37.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:37.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:37.428030+0000) 2024-03-06T09:45:38.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:38.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:38.428316+0000) 2024-03-06T09:45:39.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:39.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:39.428597+0000) 2024-03-06T09:45:40.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:40.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:40.428950+0000) 2024-03-06T09:45:41.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:41.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:41.429264+0000) 2024-03-06T09:45:42.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:42.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:42.429566+0000) 2024-03-06T09:45:43.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:43.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:43.429844+0000) 2024-03-06T09:45:44.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:44.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:44.430166+0000) 2024-03-06T09:45:45.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:45.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:45.430458+0000) 2024-03-06T09:45:46.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:46.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:46.430801+0000) 2024-03-06T09:45:47.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:47.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:47.431103+0000) 2024-03-06T09:45:48.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:48.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:48.431404+0000) 2024-03-06T09:45:49.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:49.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:49.431802+0000) 2024-03-06T09:45:50.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:50.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:50.432087+0000) 2024-03-06T09:45:51.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:51.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:51.432315+0000) 2024-03-06T09:45:52.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:52.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:52.432542+0000) 2024-03-06T09:45:53.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:53.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:53.432779+0000) 2024-03-06T09:45:54.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:54.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:54.433013+0000) 2024-03-06T09:45:55.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:55.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:55.433265+0000) 2024-03-06T09:45:56.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:56.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:56.433514+0000) 2024-03-06T09:45:57.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:57.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:57.433773+0000) 2024-03-06T09:45:58.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:58.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:58.433914+0000) 2024-03-06T09:45:59.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:45:59.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:59.434137+0000) 2024-03-06T09:46:00.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:00.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:00.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:00.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:00.434405+0000) 2024-03-06T09:46:01.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:01.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:01.434696+0000) 2024-03-06T09:46:02.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:02.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:02.434956+0000) 2024-03-06T09:46:03.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:03.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:03.435209+0000) 2024-03-06T09:46:04.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:04.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:04.435466+0000) 2024-03-06T09:46:05.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:05.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:05.435745+0000) 2024-03-06T09:46:06.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:06.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:06.435994+0000) 2024-03-06T09:46:07.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:07.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:07.436235+0000) 2024-03-06T09:46:08.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:08.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:08.436479+0000) 2024-03-06T09:46:09.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:09.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:09.436736+0000) 2024-03-06T09:46:10.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:10.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:10.436989+0000) 2024-03-06T09:46:11.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:11.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:11.437266+0000) 2024-03-06T09:46:12.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:12.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:12.437476+0000) 2024-03-06T09:46:13.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:13.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:13.437756+0000) 2024-03-06T09:46:14.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:14.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:14.437970+0000) 2024-03-06T09:46:15.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:15.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:15.438227+0000) 2024-03-06T09:46:16.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:16.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:16.438509+0000) 2024-03-06T09:46:17.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:17.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:17.438780+0000) 2024-03-06T09:46:18.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:18.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:18.438988+0000) 2024-03-06T09:46:19.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:19.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:19.439234+0000) 2024-03-06T09:46:20.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:20.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:20.439490+0000) 2024-03-06T09:46:21.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:21.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:21.439770+0000) 2024-03-06T09:46:22.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:22.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:22.439965+0000) 2024-03-06T09:46:23.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:23.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:23.440227+0000) 2024-03-06T09:46:24.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:24.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:24.440480+0000) 2024-03-06T09:46:25.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:25.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:25.440739+0000) 2024-03-06T09:46:26.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:26.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:26.441008+0000) 2024-03-06T09:46:27.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:27.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:27.441262+0000) 2024-03-06T09:46:28.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:28.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:28.441521+0000) 2024-03-06T09:46:29.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:29.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:29.441803+0000) 2024-03-06T09:46:30.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:30.385 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:30.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:30.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:30.441993+0000) 2024-03-06T09:46:31.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:31.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:31.442224+0000) 2024-03-06T09:46:32.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:32.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:32.442480+0000) 2024-03-06T09:46:33.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:33.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:33.442750+0000) 2024-03-06T09:46:34.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:34.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:34.442972+0000) 2024-03-06T09:46:35.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:35.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:35.443228+0000) 2024-03-06T09:46:36.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:36.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:36.443502+0000) 2024-03-06T09:46:37.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:37.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:37.443762+0000) 2024-03-06T09:46:38.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:38.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:38.444006+0000) 2024-03-06T09:46:39.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:39.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:39.444259+0000) 2024-03-06T09:46:40.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:40.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:40.444508+0000) 2024-03-06T09:46:41.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:41.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:41.444763+0000) 2024-03-06T09:46:42.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:42.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:42.445041+0000) 2024-03-06T09:46:43.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:43.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:43.445293+0000) 2024-03-06T09:46:44.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:44.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:44.445527+0000) 2024-03-06T09:46:45.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:45.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:45.445780+0000) 2024-03-06T09:46:46.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:46.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:46.445987+0000) 2024-03-06T09:46:47.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:47.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:47.446276+0000) 2024-03-06T09:46:48.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:48.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:48.446461+0000) 2024-03-06T09:46:49.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:49.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:49.446743+0000) 2024-03-06T09:46:50.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:50.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:50.446981+0000) 2024-03-06T09:46:51.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:51.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:51.447165+0000) 2024-03-06T09:46:52.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:52.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:52.447418+0000) 2024-03-06T09:46:53.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:53.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:53.447706+0000) 2024-03-06T09:46:54.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:54.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:54.447941+0000) 2024-03-06T09:46:55.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:55.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:55.448199+0000) 2024-03-06T09:46:56.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:56.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:56.448454+0000) 2024-03-06T09:46:57.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:57.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:57.448749+0000) 2024-03-06T09:46:58.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:58.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:58.449008+0000) 2024-03-06T09:46:59.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:46:59.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:59.449262+0000) 2024-03-06T09:47:00.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:00.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:00.449544+0000) 2024-03-06T09:47:00.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:00.526 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:01.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:01.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:01.449805+0000) 2024-03-06T09:47:02.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:02.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:02.450037+0000) 2024-03-06T09:47:03.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:03.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:03.450285+0000) 2024-03-06T09:47:04.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:04.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:04.450529+0000) 2024-03-06T09:47:05.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:05.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:05.450770+0000) 2024-03-06T09:47:06.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:06.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:06.451028+0000) 2024-03-06T09:47:07.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:07.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:07.451270+0000) 2024-03-06T09:47:08.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:08.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:08.451526+0000) 2024-03-06T09:47:09.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:09.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:09.451763+0000) 2024-03-06T09:47:10.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:10.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:10.451958+0000) 2024-03-06T09:47:11.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:11.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:11.452217+0000) 2024-03-06T09:47:12.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:12.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:12.452470+0000) 2024-03-06T09:47:13.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:13.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:13.452753+0000) 2024-03-06T09:47:14.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:14.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:14.453010+0000) 2024-03-06T09:47:15.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:15.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:15.453274+0000) 2024-03-06T09:47:16.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:16.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:16.453545+0000) 2024-03-06T09:47:17.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:17.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:17.453780+0000) 2024-03-06T09:47:18.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:18.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:18.454019+0000) 2024-03-06T09:47:19.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:19.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:19.454285+0000) 2024-03-06T09:47:20.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:20.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:20.454527+0000) 2024-03-06T09:47:21.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:21.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:21.454789+0000) 2024-03-06T09:47:22.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:22.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:22.455047+0000) 2024-03-06T09:47:23.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:23.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:23.455303+0000) 2024-03-06T09:47:24.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:24.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:24.455561+0000) 2024-03-06T09:47:25.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:25.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:25.455817+0000) 2024-03-06T09:47:26.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:26.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:26.456071+0000) 2024-03-06T09:47:27.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:27.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:27.456322+0000) 2024-03-06T09:47:28.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:28.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:28.456567+0000) 2024-03-06T09:47:29.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:29.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:29.456821+0000) 2024-03-06T09:47:30.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:30.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:30.457076+0000) 2024-03-06T09:47:30.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:30.679 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:31.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:31.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:31.457336+0000) 2024-03-06T09:47:32.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:32.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:32.457590+0000) 2024-03-06T09:47:33.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:33.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:33.457827+0000) 2024-03-06T09:47:34.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:34.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:34.458070+0000) 2024-03-06T09:47:35.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:35.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:35.458313+0000) 2024-03-06T09:47:36.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:36.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:36.458572+0000) 2024-03-06T09:47:37.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:37.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:37.458806+0000) 2024-03-06T09:47:38.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:38.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:38.459061+0000) 2024-03-06T09:47:39.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:39.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:39.459313+0000) 2024-03-06T09:47:40.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:40.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:40.459573+0000) 2024-03-06T09:47:41.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:41.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:41.459827+0000) 2024-03-06T09:47:42.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:42.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:42.460079+0000) 2024-03-06T09:47:43.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:43.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:43.460336+0000) 2024-03-06T09:47:44.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:44.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:44.460577+0000) 2024-03-06T09:47:45.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:45.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:45.460839+0000) 2024-03-06T09:47:46.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:46.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:46.461169+0000) 2024-03-06T09:47:47.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:47.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:47.461420+0000) 2024-03-06T09:47:48.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:48.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:48.461704+0000) 2024-03-06T09:47:49.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:49.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:49.461942+0000) 2024-03-06T09:47:50.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:50.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:50.462204+0000) 2024-03-06T09:47:51.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:51.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:51.462463+0000) 2024-03-06T09:47:52.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:52.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:52.462766+0000) 2024-03-06T09:47:53.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:53.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:53.463073+0000) 2024-03-06T09:47:54.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:54.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:54.463385+0000) 2024-03-06T09:47:55.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:55.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:55.463738+0000) 2024-03-06T09:47:56.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:56.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:56.464039+0000) 2024-03-06T09:47:57.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:57.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:57.464352+0000) 2024-03-06T09:47:58.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:58.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:58.464611+0000) 2024-03-06T09:47:59.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:47:59.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:59.464831+0000) 2024-03-06T09:48:00.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:00.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:00.465082+0000) 2024-03-06T09:48:00.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:00.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:01.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:01.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:01.465287+0000) 2024-03-06T09:48:02.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:02.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:02.465523+0000) 2024-03-06T09:48:03.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:03.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:03.465769+0000) 2024-03-06T09:48:04.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:04.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:04.466015+0000) 2024-03-06T09:48:05.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:05.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:05.466207+0000) 2024-03-06T09:48:06.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:06.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:06.466479+0000) 2024-03-06T09:48:07.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:07.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:07.466734+0000) 2024-03-06T09:48:08.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:08.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:08.466991+0000) 2024-03-06T09:48:09.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:09.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:09.467178+0000) 2024-03-06T09:48:10.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:10.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:10.467445+0000) 2024-03-06T09:48:11.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:11.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:11.467754+0000) 2024-03-06T09:48:12.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:12.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:12.468006+0000) 2024-03-06T09:48:13.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:13.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:13.468261+0000) 2024-03-06T09:48:14.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:14.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:14.468516+0000) 2024-03-06T09:48:15.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:15.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:15.468774+0000) 2024-03-06T09:48:16.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:16.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:16.469031+0000) 2024-03-06T09:48:17.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:17.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:17.469279+0000) 2024-03-06T09:48:18.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:18.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:18.469537+0000) 2024-03-06T09:48:19.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:19.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:19.469786+0000) 2024-03-06T09:48:20.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:20.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:20.470052+0000) 2024-03-06T09:48:21.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:21.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:21.470307+0000) 2024-03-06T09:48:22.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:22.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:22.470552+0000) 2024-03-06T09:48:23.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:23.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:23.470810+0000) 2024-03-06T09:48:24.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:24.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:24.471051+0000) 2024-03-06T09:48:25.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:25.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:25.471292+0000) 2024-03-06T09:48:26.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:26.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:26.471567+0000) 2024-03-06T09:48:27.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:27.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:27.471861+0000) 2024-03-06T09:48:28.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:28.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:28.472166+0000) 2024-03-06T09:48:29.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:29.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:29.472476+0000) 2024-03-06T09:48:30.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:30.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:30.472803+0000) 2024-03-06T09:48:30.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:30.991 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:31.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:31.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:31.473103+0000) 2024-03-06T09:48:32.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:32.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:32.473407+0000) 2024-03-06T09:48:33.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:33.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:33.473743+0000) 2024-03-06T09:48:34.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:34.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:34.474032+0000) 2024-03-06T09:48:35.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:35.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:35.474353+0000) 2024-03-06T09:48:36.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:36.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:36.474686+0000) 2024-03-06T09:48:37.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:37.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:37.475019+0000) 2024-03-06T09:48:38.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:38.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:38.475306+0000) 2024-03-06T09:48:39.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:39.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:39.475622+0000) 2024-03-06T09:48:40.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:40.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:40.475976+0000) 2024-03-06T09:48:41.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:41.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:41.476292+0000) 2024-03-06T09:48:42.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:42.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:42.476594+0000) 2024-03-06T09:48:43.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:43.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:43.476958+0000) 2024-03-06T09:48:44.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:44.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:44.477277+0000) 2024-03-06T09:48:45.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:45.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:45.477592+0000) 2024-03-06T09:48:46.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:46.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:46.477970+0000) 2024-03-06T09:48:47.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:47.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:47.478269+0000) 2024-03-06T09:48:48.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:48.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:48.478579+0000) 2024-03-06T09:48:49.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:49.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:49.478889+0000) 2024-03-06T09:48:50.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:50.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:50.479204+0000) 2024-03-06T09:48:51.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:51.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:51.479520+0000) 2024-03-06T09:48:52.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:52.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:52.479819+0000) 2024-03-06T09:48:53.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:53.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:53.480130+0000) 2024-03-06T09:48:54.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:54.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:54.480414+0000) 2024-03-06T09:48:55.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:55.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:55.480761+0000) 2024-03-06T09:48:56.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:56.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:56.481082+0000) 2024-03-06T09:48:57.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:57.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:57.481368+0000) 2024-03-06T09:48:58.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:58.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:58.481717+0000) 2024-03-06T09:48:59.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:48:59.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:59.482035+0000) 2024-03-06T09:49:00.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:00.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:00.482280+0000) 2024-03-06T09:49:01.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:01.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:01.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:01.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:01.482576+0000) 2024-03-06T09:49:02.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:02.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:02.482875+0000) 2024-03-06T09:49:03.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:03.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:03.483166+0000) 2024-03-06T09:49:04.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:04.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:04.483452+0000) 2024-03-06T09:49:05.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:05.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:05.483788+0000) 2024-03-06T09:49:06.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:06.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:06.484052+0000) 2024-03-06T09:49:07.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:07.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:07.484333+0000) 2024-03-06T09:49:08.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:08.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:08.484685+0000) 2024-03-06T09:49:09.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:09.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:09.485010+0000) 2024-03-06T09:49:10.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:10.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:10.485303+0000) 2024-03-06T09:49:11.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:11.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:11.485618+0000) 2024-03-06T09:49:12.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:12.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:12.486001+0000) 2024-03-06T09:49:13.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:13.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:13.486308+0000) 2024-03-06T09:49:14.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:14.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:14.486616+0000) 2024-03-06T09:49:15.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:15.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:15.486976+0000) 2024-03-06T09:49:16.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:16.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:16.487298+0000) 2024-03-06T09:49:17.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:17.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:17.487615+0000) 2024-03-06T09:49:18.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:18.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:18.487976+0000) 2024-03-06T09:49:19.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:19.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:19.488268+0000) 2024-03-06T09:49:20.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:20.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:20.488576+0000) 2024-03-06T09:49:21.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:21.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:21.488868+0000) 2024-03-06T09:49:22.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:22.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:22.489165+0000) 2024-03-06T09:49:23.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:23.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:23.489471+0000) 2024-03-06T09:49:24.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:24.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:24.489789+0000) 2024-03-06T09:49:25.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:25.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:25.490083+0000) 2024-03-06T09:49:26.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:26.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:26.490393+0000) 2024-03-06T09:49:27.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:27.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:27.490725+0000) 2024-03-06T09:49:28.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:28.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:28.491047+0000) 2024-03-06T09:49:29.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:29.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:29.491358+0000) 2024-03-06T09:49:30.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:30.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:30.491698+0000) 2024-03-06T09:49:31.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:31.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:31.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:31.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:31.492013+0000) 2024-03-06T09:49:32.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:32.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:32.492311+0000) 2024-03-06T09:49:33.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:33.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:33.492606+0000) 2024-03-06T09:49:34.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:34.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:34.492995+0000) 2024-03-06T09:49:35.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:35.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:35.493317+0000) 2024-03-06T09:49:36.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:36.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:36.493667+0000) 2024-03-06T09:49:37.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:37.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:37.493980+0000) 2024-03-06T09:49:38.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:38.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:38.494376+0000) 2024-03-06T09:49:39.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:39.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:39.494735+0000) 2024-03-06T09:49:40.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:40.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:40.495060+0000) 2024-03-06T09:49:41.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:41.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:41.495340+0000) 2024-03-06T09:49:42.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:42.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:42.495669+0000) 2024-03-06T09:49:43.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:43.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:43.495970+0000) 2024-03-06T09:49:44.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:44.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:44.496261+0000) 2024-03-06T09:49:45.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:45.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:45.496562+0000) 2024-03-06T09:49:46.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:46.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:46.496887+0000) 2024-03-06T09:49:47.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:47.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:47.497179+0000) 2024-03-06T09:49:48.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:48.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:48.497465+0000) 2024-03-06T09:49:49.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:49.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:49.497786+0000) 2024-03-06T09:49:50.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:50.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:50.498074+0000) 2024-03-06T09:49:51.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:51.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:51.498345+0000) 2024-03-06T09:49:52.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:52.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:52.498595+0000) 2024-03-06T09:49:53.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:53.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:53.498856+0000) 2024-03-06T09:49:54.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:54.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:54.499137+0000) 2024-03-06T09:49:55.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:55.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:55.499408+0000) 2024-03-06T09:49:56.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:56.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:56.499701+0000) 2024-03-06T09:49:57.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:57.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:57.499951+0000) 2024-03-06T09:49:58.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:58.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:58.500187+0000) 2024-03-06T09:49:59.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:49:59.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:59.500424+0000) 2024-03-06T09:50:00.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:00.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:00.500722+0000) 2024-03-06T09:50:01.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:01.456 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:01.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:01.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:01.500914+0000) 2024-03-06T09:50:02.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:02.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:02.501165+0000) 2024-03-06T09:50:03.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:03.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:03.501417+0000) 2024-03-06T09:50:04.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:04.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:04.501706+0000) 2024-03-06T09:50:05.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:05.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:05.501895+0000) 2024-03-06T09:50:06.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:06.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:06.502161+0000) 2024-03-06T09:50:07.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:07.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:07.502412+0000) 2024-03-06T09:50:08.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:08.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:08.502717+0000) 2024-03-06T09:50:09.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:09.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:09.502982+0000) 2024-03-06T09:50:10.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:10.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:10.503219+0000) 2024-03-06T09:50:11.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:11.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:11.503453+0000) 2024-03-06T09:50:12.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:12.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:12.503732+0000) 2024-03-06T09:50:13.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:13.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:13.503997+0000) 2024-03-06T09:50:14.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:14.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:14.504251+0000) 2024-03-06T09:50:15.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:15.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:15.504484+0000) 2024-03-06T09:50:16.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:16.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:16.504750+0000) 2024-03-06T09:50:17.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:17.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:17.504998+0000) 2024-03-06T09:50:18.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:18.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:18.505252+0000) 2024-03-06T09:50:19.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:19.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:19.505439+0000) 2024-03-06T09:50:20.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:20.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:20.505722+0000) 2024-03-06T09:50:21.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:21.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:21.505973+0000) 2024-03-06T09:50:22.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:22.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:22.506229+0000) 2024-03-06T09:50:23.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:23.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:23.506478+0000) 2024-03-06T09:50:24.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:24.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:24.506740+0000) 2024-03-06T09:50:25.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:25.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:25.506976+0000) 2024-03-06T09:50:26.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:26.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:26.507236+0000) 2024-03-06T09:50:27.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:27.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:27.507490+0000) 2024-03-06T09:50:28.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:28.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:28.507749+0000) 2024-03-06T09:50:29.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:29.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:29.507985+0000) 2024-03-06T09:50:30.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:30.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:30.508223+0000) 2024-03-06T09:50:31.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:31.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:31.508485+0000) 2024-03-06T09:50:31.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:31.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:32.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:32.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:32.508716+0000) 2024-03-06T09:50:33.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:33.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:33.508960+0000) 2024-03-06T09:50:34.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:34.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:34.509191+0000) 2024-03-06T09:50:35.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:35.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:35.509449+0000) 2024-03-06T09:50:36.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:36.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:36.509755+0000) 2024-03-06T09:50:37.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:37.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:37.509994+0000) 2024-03-06T09:50:38.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:38.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:38.510249+0000) 2024-03-06T09:50:39.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:39.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:39.510482+0000) 2024-03-06T09:50:40.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:40.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:40.510708+0000) 2024-03-06T09:50:41.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:41.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:41.510957+0000) 2024-03-06T09:50:42.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:42.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:42.511205+0000) 2024-03-06T09:50:43.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:43.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:43.511459+0000) 2024-03-06T09:50:44.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:44.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:44.511746+0000) 2024-03-06T09:50:45.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:45.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:45.511997+0000) 2024-03-06T09:50:46.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:46.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:46.512193+0000) 2024-03-06T09:50:47.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:47.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:47.512447+0000) 2024-03-06T09:50:48.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:48.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:48.512677+0000) 2024-03-06T09:50:49.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:49.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:49.512872+0000) 2024-03-06T09:50:50.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:50.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:50.513126+0000) 2024-03-06T09:50:51.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:51.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:51.513366+0000) 2024-03-06T09:50:52.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:52.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:52.513604+0000) 2024-03-06T09:50:53.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:53.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:53.513897+0000) 2024-03-06T09:50:54.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:54.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:54.514079+0000) 2024-03-06T09:50:55.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:55.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:55.514269+0000) 2024-03-06T09:50:56.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:56.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:56.514516+0000) 2024-03-06T09:50:57.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:57.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:57.514767+0000) 2024-03-06T09:50:58.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:58.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:58.515009+0000) 2024-03-06T09:50:59.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:50:59.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:59.515198+0000) 2024-03-06T09:51:00.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:00.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:00.515452+0000) 2024-03-06T09:51:01.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:01.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:01.515661+0000) 2024-03-06T09:51:01.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:01.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:02.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:02.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:02.515896+0000) 2024-03-06T09:51:03.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:03.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:03.516115+0000) 2024-03-06T09:51:04.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:04.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:04.516371+0000) 2024-03-06T09:51:05.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:05.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:05.516579+0000) 2024-03-06T09:51:06.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:06.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:06.516828+0000) 2024-03-06T09:51:07.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:07.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:07.517049+0000) 2024-03-06T09:51:08.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:08.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:08.517329+0000) 2024-03-06T09:51:09.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:09.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:09.517552+0000) 2024-03-06T09:51:10.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:10.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:10.517752+0000) 2024-03-06T09:51:11.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:11.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:11.517948+0000) 2024-03-06T09:51:12.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:12.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:12.518182+0000) 2024-03-06T09:51:13.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:13.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:13.518412+0000) 2024-03-06T09:51:14.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:14.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:14.518712+0000) 2024-03-06T09:51:15.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:15.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:15.518962+0000) 2024-03-06T09:51:16.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:16.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:16.519231+0000) 2024-03-06T09:51:17.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:17.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:17.519489+0000) 2024-03-06T09:51:18.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:18.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:18.519750+0000) 2024-03-06T09:51:19.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:19.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:19.520023+0000) 2024-03-06T09:51:20.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:20.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:20.520299+0000) 2024-03-06T09:51:21.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:21.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:21.520577+0000) 2024-03-06T09:51:22.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:22.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:22.520851+0000) 2024-03-06T09:51:23.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:23.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:23.521125+0000) 2024-03-06T09:51:24.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:24.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:24.521398+0000) 2024-03-06T09:51:25.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:25.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:25.521714+0000) 2024-03-06T09:51:26.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:26.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:26.521981+0000) 2024-03-06T09:51:27.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:27.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:27.522177+0000) 2024-03-06T09:51:28.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:28.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:28.522457+0000) 2024-03-06T09:51:29.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:29.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:29.522744+0000) 2024-03-06T09:51:30.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:30.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:30.523016+0000) 2024-03-06T09:51:31.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:31.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:31.523300+0000) 2024-03-06T09:51:31.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:31.934 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:32.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:32.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:32.523563+0000) 2024-03-06T09:51:33.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:33.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:33.523816+0000) 2024-03-06T09:51:34.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:34.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:34.524071+0000) 2024-03-06T09:51:35.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:35.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:35.524397+0000) 2024-03-06T09:51:36.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:36.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:36.524702+0000) 2024-03-06T09:51:37.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:37.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:37.524971+0000) 2024-03-06T09:51:38.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:38.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:38.525253+0000) 2024-03-06T09:51:39.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:39.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:39.525503+0000) 2024-03-06T09:51:40.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:40.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:40.525754+0000) 2024-03-06T09:51:41.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:41.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:41.526037+0000) 2024-03-06T09:51:42.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:42.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:42.526309+0000) 2024-03-06T09:51:43.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:43.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:43.526567+0000) 2024-03-06T09:51:44.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:44.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:44.526844+0000) 2024-03-06T09:51:45.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:45.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:45.527085+0000) 2024-03-06T09:51:46.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:46.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:46.527377+0000) 2024-03-06T09:51:47.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:47.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:47.527678+0000) 2024-03-06T09:51:48.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:48.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:48.527961+0000) 2024-03-06T09:51:49.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:49.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:49.528229+0000) 2024-03-06T09:51:50.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:50.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:50.528483+0000) 2024-03-06T09:51:51.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:51.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:51.528755+0000) 2024-03-06T09:51:52.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:52.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:52.528953+0000) 2024-03-06T09:51:53.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:53.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:53.529227+0000) 2024-03-06T09:51:54.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:54.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:54.529485+0000) 2024-03-06T09:51:55.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:55.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:55.529735+0000) 2024-03-06T09:51:56.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:56.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:56.529942+0000) 2024-03-06T09:51:57.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:57.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:57.530199+0000) 2024-03-06T09:51:58.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:58.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:58.530454+0000) 2024-03-06T09:51:59.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:51:59.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:59.530752+0000) 2024-03-06T09:52:00.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:00.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:00.531014+0000) 2024-03-06T09:52:01.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:01.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:01.531275+0000) 2024-03-06T09:52:02.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:02.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:02.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:02.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:02.531538+0000) 2024-03-06T09:52:03.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:03.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:03.531777+0000) 2024-03-06T09:52:04.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:04.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:04.532034+0000) 2024-03-06T09:52:05.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:05.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:05.532281+0000) 2024-03-06T09:52:06.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:06.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:06.532554+0000) 2024-03-06T09:52:07.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:07.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:07.532790+0000) 2024-03-06T09:52:08.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:08.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:08.533028+0000) 2024-03-06T09:52:09.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:09.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:09.533271+0000) 2024-03-06T09:52:10.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:10.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:10.533515+0000) 2024-03-06T09:52:11.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:11.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:11.533754+0000) 2024-03-06T09:52:12.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:12.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:12.534022+0000) 2024-03-06T09:52:13.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:13.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:13.534278+0000) 2024-03-06T09:52:14.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:14.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:14.534523+0000) 2024-03-06T09:52:15.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:15.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:15.534758+0000) 2024-03-06T09:52:16.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:16.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:16.535033+0000) 2024-03-06T09:52:17.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:17.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:17.535291+0000) 2024-03-06T09:52:18.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:18.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:18.535546+0000) 2024-03-06T09:52:19.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:19.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:19.535797+0000) 2024-03-06T09:52:20.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:20.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:20.536056+0000) 2024-03-06T09:52:21.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:21.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:21.536320+0000) 2024-03-06T09:52:22.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:22.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:22.536556+0000) 2024-03-06T09:52:23.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:23.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:23.536809+0000) 2024-03-06T09:52:24.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:24.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:24.537047+0000) 2024-03-06T09:52:25.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:25.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:25.537278+0000) 2024-03-06T09:52:26.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:26.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:26.537497+0000) 2024-03-06T09:52:27.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:27.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:27.537759+0000) 2024-03-06T09:52:28.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:28.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:28.537956+0000) 2024-03-06T09:52:29.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:29.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:29.538263+0000) 2024-03-06T09:52:30.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:30.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:30.538521+0000) 2024-03-06T09:52:31.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:31.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:31.538785+0000) 2024-03-06T09:52:32.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:32.226 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:32.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:32.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:32.539020+0000) 2024-03-06T09:52:33.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:33.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:33.539264+0000) 2024-03-06T09:52:34.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:34.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:34.539523+0000) 2024-03-06T09:52:35.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:35.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:35.539777+0000) 2024-03-06T09:52:36.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:36.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:36.540031+0000) 2024-03-06T09:52:37.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:37.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:37.540267+0000) 2024-03-06T09:52:38.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:38.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:38.540512+0000) 2024-03-06T09:52:39.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:39.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:39.540761+0000) 2024-03-06T09:52:40.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:40.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:40.541000+0000) 2024-03-06T09:52:41.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:41.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:41.541261+0000) 2024-03-06T09:52:42.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:42.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:42.541507+0000) 2024-03-06T09:52:43.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:43.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:43.541759+0000) 2024-03-06T09:52:44.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:44.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:44.542025+0000) 2024-03-06T09:52:45.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:45.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:45.542281+0000) 2024-03-06T09:52:46.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:46.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:46.542529+0000) 2024-03-06T09:52:47.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:47.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:47.542779+0000) 2024-03-06T09:52:48.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:48.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:48.543040+0000) 2024-03-06T09:52:49.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:49.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:49.543298+0000) 2024-03-06T09:52:50.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:50.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:50.543556+0000) 2024-03-06T09:52:51.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:51.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:51.543820+0000) 2024-03-06T09:52:52.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:52.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:52.544073+0000) 2024-03-06T09:52:53.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:53.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:53.544309+0000) 2024-03-06T09:52:54.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:54.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:54.544568+0000) 2024-03-06T09:52:55.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:55.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:55.544822+0000) 2024-03-06T09:52:56.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:56.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:56.545133+0000) 2024-03-06T09:52:57.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:57.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:57.545380+0000) 2024-03-06T09:52:58.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:58.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:58.545664+0000) 2024-03-06T09:52:59.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:52:59.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:59.545895+0000) 2024-03-06T09:53:00.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:00.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:00.546134+0000) 2024-03-06T09:53:01.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:01.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:01.546384+0000) 2024-03-06T09:53:02.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:02.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:02.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:02.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:02.546699+0000) 2024-03-06T09:53:03.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:03.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:03.546959+0000) 2024-03-06T09:53:04.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:04.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:04.547217+0000) 2024-03-06T09:53:05.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:05.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:05.547449+0000) 2024-03-06T09:53:06.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:06.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:06.547746+0000) 2024-03-06T09:53:07.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:07.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:07.547983+0000) 2024-03-06T09:53:08.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:08.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:08.548232+0000) 2024-03-06T09:53:09.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:09.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:09.548477+0000) 2024-03-06T09:53:10.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:10.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:10.548734+0000) 2024-03-06T09:53:11.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:11.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:11.548999+0000) 2024-03-06T09:53:12.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:12.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:12.549224+0000) 2024-03-06T09:53:13.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:13.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:13.549485+0000) 2024-03-06T09:53:14.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:14.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:14.549746+0000) 2024-03-06T09:53:15.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:15.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:15.549977+0000) 2024-03-06T09:53:16.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:16.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:16.550269+0000) 2024-03-06T09:53:17.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:17.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:17.550516+0000) 2024-03-06T09:53:18.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:18.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:18.550772+0000) 2024-03-06T09:53:19.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:19.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:19.551019+0000) 2024-03-06T09:53:20.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:20.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:20.551271+0000) 2024-03-06T09:53:21.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:21.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:21.551541+0000) 2024-03-06T09:53:22.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:22.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:22.551806+0000) 2024-03-06T09:53:23.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:23.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:23.552043+0000) 2024-03-06T09:53:24.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:24.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:24.552300+0000) 2024-03-06T09:53:25.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:25.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:25.552532+0000) 2024-03-06T09:53:26.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:26.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:26.552800+0000) 2024-03-06T09:53:27.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:27.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:27.553110+0000) 2024-03-06T09:53:28.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:28.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:28.553372+0000) 2024-03-06T09:53:29.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:29.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:29.553661+0000) 2024-03-06T09:53:30.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:30.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:30.553926+0000) 2024-03-06T09:53:31.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:31.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:31.554195+0000) 2024-03-06T09:53:32.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:32.547 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:32.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:32.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:32.554489+0000) 2024-03-06T09:53:33.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:33.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:33.554754+0000) 2024-03-06T09:53:34.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:34.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:34.554992+0000) 2024-03-06T09:53:35.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:35.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:35.555226+0000) 2024-03-06T09:53:36.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:36.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:36.555481+0000) 2024-03-06T09:53:37.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:37.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:37.555793+0000) 2024-03-06T09:53:38.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:38.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:38.556021+0000) 2024-03-06T09:53:39.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:39.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:39.556251+0000) 2024-03-06T09:53:40.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:40.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:40.556508+0000) 2024-03-06T09:53:41.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:41.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:41.556758+0000) 2024-03-06T09:53:42.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:42.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:42.557023+0000) 2024-03-06T09:53:43.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:43.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:43.557282+0000) 2024-03-06T09:53:44.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:44.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:44.557537+0000) 2024-03-06T09:53:45.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:45.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:45.557791+0000) 2024-03-06T09:53:46.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:46.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:46.558044+0000) 2024-03-06T09:53:47.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:47.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:47.558312+0000) 2024-03-06T09:53:48.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:48.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:48.558574+0000) 2024-03-06T09:53:49.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:49.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:49.558807+0000) 2024-03-06T09:53:50.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:50.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:50.559060+0000) 2024-03-06T09:53:51.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:51.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:51.559249+0000) 2024-03-06T09:53:52.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:52.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:52.559500+0000) 2024-03-06T09:53:53.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:53.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:53.559734+0000) 2024-03-06T09:53:54.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:54.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:54.559966+0000) 2024-03-06T09:53:55.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:55.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:55.560210+0000) 2024-03-06T09:53:56.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:56.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:56.560481+0000) 2024-03-06T09:53:57.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:57.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:57.560752+0000) 2024-03-06T09:53:58.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:58.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:58.560992+0000) 2024-03-06T09:53:59.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:53:59.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:59.561247+0000) 2024-03-06T09:54:00.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:00.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:00.561506+0000) 2024-03-06T09:54:01.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:01.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:01.561741+0000) 2024-03-06T09:54:02.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:02.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:02.561976+0000) 2024-03-06T09:54:02.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:02.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:03.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:03.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:03.562174+0000) 2024-03-06T09:54:04.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:04.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:04.562426+0000) 2024-03-06T09:54:05.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:05.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:05.562728+0000) 2024-03-06T09:54:06.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:06.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:06.562977+0000) 2024-03-06T09:54:07.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:07.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:07.563219+0000) 2024-03-06T09:54:08.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:08.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:08.563465+0000) 2024-03-06T09:54:09.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:09.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:09.563747+0000) 2024-03-06T09:54:10.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:10.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:10.564054+0000) 2024-03-06T09:54:11.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:11.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:11.564299+0000) 2024-03-06T09:54:12.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:12.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:12.564558+0000) 2024-03-06T09:54:13.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:13.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:13.564813+0000) 2024-03-06T09:54:14.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:14.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:14.565004+0000) 2024-03-06T09:54:15.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:15.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:15.565233+0000) 2024-03-06T09:54:16.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:16.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:16.565508+0000) 2024-03-06T09:54:17.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:17.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:17.565736+0000) 2024-03-06T09:54:18.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:18.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:18.565999+0000) 2024-03-06T09:54:19.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:19.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:19.566245+0000) 2024-03-06T09:54:20.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:20.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:20.566485+0000) 2024-03-06T09:54:21.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:21.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:21.566740+0000) 2024-03-06T09:54:22.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:22.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:22.566994+0000) 2024-03-06T09:54:23.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:23.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:23.567229+0000) 2024-03-06T09:54:24.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:24.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:24.567472+0000) 2024-03-06T09:54:25.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:25.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:25.567751+0000) 2024-03-06T09:54:26.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:26.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:26.568000+0000) 2024-03-06T09:54:27.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:27.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:27.568255+0000) 2024-03-06T09:54:28.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:28.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:28.568497+0000) 2024-03-06T09:54:29.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:29.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:29.568748+0000) 2024-03-06T09:54:30.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:30.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:30.569007+0000) 2024-03-06T09:54:31.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:31.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:31.569251+0000) 2024-03-06T09:54:32.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:32.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:32.569503+0000) 2024-03-06T09:54:32.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:32.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:33.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:33.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:33.569750+0000) 2024-03-06T09:54:34.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:34.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:34.569987+0000) 2024-03-06T09:54:35.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:35.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:35.570218+0000) 2024-03-06T09:54:36.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:36.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:36.570475+0000) 2024-03-06T09:54:37.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:37.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:37.570766+0000) 2024-03-06T09:54:38.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:38.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:38.571024+0000) 2024-03-06T09:54:39.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:39.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:39.571207+0000) 2024-03-06T09:54:40.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:40.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:40.571447+0000) 2024-03-06T09:54:41.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:41.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:41.571763+0000) 2024-03-06T09:54:42.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:42.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:42.572018+0000) 2024-03-06T09:54:43.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:43.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:43.572271+0000) 2024-03-06T09:54:44.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:44.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:44.572531+0000) 2024-03-06T09:54:45.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:45.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:45.572765+0000) 2024-03-06T09:54:46.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:46.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:46.573029+0000) 2024-03-06T09:54:47.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:47.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:47.573223+0000) 2024-03-06T09:54:48.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:48.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:48.573478+0000) 2024-03-06T09:54:49.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:49.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:49.573731+0000) 2024-03-06T09:54:50.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:50.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:50.573969+0000) 2024-03-06T09:54:51.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:51.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:51.574199+0000) 2024-03-06T09:54:52.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:52.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:52.574457+0000) 2024-03-06T09:54:53.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:53.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:53.574759+0000) 2024-03-06T09:54:54.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:54.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:54.575005+0000) 2024-03-06T09:54:55.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:55.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:55.575195+0000) 2024-03-06T09:54:56.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:56.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:56.575467+0000) 2024-03-06T09:54:57.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:57.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:57.575739+0000) 2024-03-06T09:54:58.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:58.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:58.575983+0000) 2024-03-06T09:54:59.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:54:59.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:59.576230+0000) 2024-03-06T09:55:00.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:00.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:00.576483+0000) 2024-03-06T09:55:01.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:01.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:01.576744+0000) 2024-03-06T09:55:02.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:02.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:02.577006+0000) 2024-03-06T09:55:02.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:02.977 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:03.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:03.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:03.577263+0000) 2024-03-06T09:55:04.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:04.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:04.577502+0000) 2024-03-06T09:55:05.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:05.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:05.577764+0000) 2024-03-06T09:55:06.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:06.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:06.578022+0000) 2024-03-06T09:55:07.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:07.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:07.578274+0000) 2024-03-06T09:55:08.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:08.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:08.578515+0000) 2024-03-06T09:55:09.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:09.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:09.578770+0000) 2024-03-06T09:55:10.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:10.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:10.579027+0000) 2024-03-06T09:55:11.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:11.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:11.579209+0000) 2024-03-06T09:55:12.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:12.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:12.579465+0000) 2024-03-06T09:55:13.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:13.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:13.579739+0000) 2024-03-06T09:55:14.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:14.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:14.579996+0000) 2024-03-06T09:55:15.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:15.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:15.580242+0000) 2024-03-06T09:55:16.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:16.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:16.580488+0000) 2024-03-06T09:55:17.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:17.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:17.580745+0000) 2024-03-06T09:55:18.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:18.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:18.581013+0000) 2024-03-06T09:55:19.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:19.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:19.581268+0000) 2024-03-06T09:55:20.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:20.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:20.581508+0000) 2024-03-06T09:55:21.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:21.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:21.581762+0000) 2024-03-06T09:55:22.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:22.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:22.581991+0000) 2024-03-06T09:55:23.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:23.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:23.582247+0000) 2024-03-06T09:55:24.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:24.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:24.582477+0000) 2024-03-06T09:55:25.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:25.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:25.582732+0000) 2024-03-06T09:55:26.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:26.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:26.582996+0000) 2024-03-06T09:55:27.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:27.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:27.583244+0000) 2024-03-06T09:55:28.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:28.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:28.583487+0000) 2024-03-06T09:55:29.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:29.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:29.583753+0000) 2024-03-06T09:55:30.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:30.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:30.584008+0000) 2024-03-06T09:55:31.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:31.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:31.584236+0000) 2024-03-06T09:55:32.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:32.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:32.584480+0000) 2024-03-06T09:55:33.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:33.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:33.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:33.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:33.584746+0000) 2024-03-06T09:55:34.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:34.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:34.584982+0000) 2024-03-06T09:55:35.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:35.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:35.585194+0000) 2024-03-06T09:55:36.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:36.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:36.585511+0000) 2024-03-06T09:55:37.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:37.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:37.585811+0000) 2024-03-06T09:55:38.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:38.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:38.586114+0000) 2024-03-06T09:55:39.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:39.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:39.586416+0000) 2024-03-06T09:55:40.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:40.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:40.586779+0000) 2024-03-06T09:55:41.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:41.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:41.587085+0000) 2024-03-06T09:55:42.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:42.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:42.587396+0000) 2024-03-06T09:55:43.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:43.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:43.587739+0000) 2024-03-06T09:55:44.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:44.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:44.588024+0000) 2024-03-06T09:55:45.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:45.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:45.588294+0000) 2024-03-06T09:55:46.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:46.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:46.588536+0000) 2024-03-06T09:55:47.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:47.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:47.588849+0000) 2024-03-06T09:55:48.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:48.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:48.589090+0000) 2024-03-06T09:55:49.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:49.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:49.589377+0000) 2024-03-06T09:55:50.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:50.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:50.589719+0000) 2024-03-06T09:55:51.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:51.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:51.590018+0000) 2024-03-06T09:55:52.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:52.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:52.590311+0000) 2024-03-06T09:55:53.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:53.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:53.590625+0000) 2024-03-06T09:55:54.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:54.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:54.590943+0000) 2024-03-06T09:55:55.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:55.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:55.591190+0000) 2024-03-06T09:55:56.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:56.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:56.591438+0000) 2024-03-06T09:55:57.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:57.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:57.591742+0000) 2024-03-06T09:55:58.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:58.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:58.592002+0000) 2024-03-06T09:55:59.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:55:59.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:59.592262+0000) 2024-03-06T09:56:00.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:00.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:00.592549+0000) 2024-03-06T09:56:01.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:01.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:01.592770+0000) 2024-03-06T09:56:02.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:02.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:02.592982+0000) 2024-03-06T09:56:03.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:03.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:03.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:03.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:03.593204+0000) 2024-03-06T09:56:04.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:04.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:04.593409+0000) 2024-03-06T09:56:05.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:05.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:05.593597+0000) 2024-03-06T09:56:06.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:06.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:06.593795+0000) 2024-03-06T09:56:07.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:07.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:07.593980+0000) 2024-03-06T09:56:08.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:08.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:08.594177+0000) 2024-03-06T09:56:09.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:09.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:09.594366+0000) 2024-03-06T09:56:10.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:10.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:10.594560+0000) 2024-03-06T09:56:11.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:11.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:11.594746+0000) 2024-03-06T09:56:12.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:12.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:12.594938+0000) 2024-03-06T09:56:13.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:13.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:13.595188+0000) 2024-03-06T09:56:14.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:14.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:14.595381+0000) 2024-03-06T09:56:15.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:15.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:15.595518+0000) 2024-03-06T09:56:16.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:16.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:16.595736+0000) 2024-03-06T09:56:17.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:17.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:17.595969+0000) 2024-03-06T09:56:18.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:18.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:18.596213+0000) 2024-03-06T09:56:19.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:19.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:19.596441+0000) 2024-03-06T09:56:20.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:20.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:20.596659+0000) 2024-03-06T09:56:21.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:21.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:21.596907+0000) 2024-03-06T09:56:22.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:22.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:22.597158+0000) 2024-03-06T09:56:23.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:23.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:23.597394+0000) 2024-03-06T09:56:24.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:24.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:24.597670+0000) 2024-03-06T09:56:25.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:25.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:25.597866+0000) 2024-03-06T09:56:26.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:26.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:26.598121+0000) 2024-03-06T09:56:27.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:27.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:27.598376+0000) 2024-03-06T09:56:28.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:28.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:28.598679+0000) 2024-03-06T09:56:29.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:29.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:29.598941+0000) 2024-03-06T09:56:30.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:30.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:30.599136+0000) 2024-03-06T09:56:31.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:31.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:31.599395+0000) 2024-03-06T09:56:32.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:32.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:32.599670+0000) 2024-03-06T09:56:33.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:33.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:33.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:33.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:33.599924+0000) 2024-03-06T09:56:34.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:34.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:34.600163+0000) 2024-03-06T09:56:35.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:35.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:35.600400+0000) 2024-03-06T09:56:36.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:36.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:36.600698+0000) 2024-03-06T09:56:37.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:37.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:37.600938+0000) 2024-03-06T09:56:38.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:38.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:38.601217+0000) 2024-03-06T09:56:39.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:39.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:39.601529+0000) 2024-03-06T09:56:40.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:40.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:40.601832+0000) 2024-03-06T09:56:41.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:41.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:41.602153+0000) 2024-03-06T09:56:42.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:42.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:42.602475+0000) 2024-03-06T09:56:43.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:43.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:43.602798+0000) 2024-03-06T09:56:44.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:44.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:44.603052+0000) 2024-03-06T09:56:45.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:45.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:45.603334+0000) 2024-03-06T09:56:46.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:46.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:46.603682+0000) 2024-03-06T09:56:47.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:47.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:47.603995+0000) 2024-03-06T09:56:48.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:48.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:48.604282+0000) 2024-03-06T09:56:49.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:49.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:49.604549+0000) 2024-03-06T09:56:50.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:50.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:50.604807+0000) 2024-03-06T09:56:51.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:51.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:51.605054+0000) 2024-03-06T09:56:52.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:52.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:52.605310+0000) 2024-03-06T09:56:53.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:53.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:53.605554+0000) 2024-03-06T09:56:54.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:54.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:54.605809+0000) 2024-03-06T09:56:55.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:55.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:55.606064+0000) 2024-03-06T09:56:56.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:56.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:56.606314+0000) 2024-03-06T09:56:57.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:57.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:57.606564+0000) 2024-03-06T09:56:58.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:58.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:58.606826+0000) 2024-03-06T09:56:59.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:56:59.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:59.607070+0000) 2024-03-06T09:57:00.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:00.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:00.607308+0000) 2024-03-06T09:57:01.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:01.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:01.607543+0000) 2024-03-06T09:57:02.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:02.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:02.607785+0000) 2024-03-06T09:57:03.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:03.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:03.608033+0000) 2024-03-06T09:57:03.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:03.624 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:04.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:04.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:04.608295+0000) 2024-03-06T09:57:05.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:05.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:05.608525+0000) 2024-03-06T09:57:06.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:06.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:06.608823+0000) 2024-03-06T09:57:07.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:07.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:07.609061+0000) 2024-03-06T09:57:08.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:08.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:08.609322+0000) 2024-03-06T09:57:09.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:09.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:09.609559+0000) 2024-03-06T09:57:10.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:10.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:10.609814+0000) 2024-03-06T09:57:11.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:11.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:11.610064+0000) 2024-03-06T09:57:12.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:12.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:12.610314+0000) 2024-03-06T09:57:13.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:13.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:13.610498+0000) 2024-03-06T09:57:14.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:14.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:14.610768+0000) 2024-03-06T09:57:15.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:15.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:15.611018+0000) 2024-03-06T09:57:16.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:16.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:16.611229+0000) 2024-03-06T09:57:17.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:17.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:17.611484+0000) 2024-03-06T09:57:18.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:18.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:18.611743+0000) 2024-03-06T09:57:19.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:19.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:19.611996+0000) 2024-03-06T09:57:20.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:20.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:20.612225+0000) 2024-03-06T09:57:21.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:21.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:21.612475+0000) 2024-03-06T09:57:22.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:22.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:22.612733+0000) 2024-03-06T09:57:23.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:23.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:23.612994+0000) 2024-03-06T09:57:24.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:24.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:24.613233+0000) 2024-03-06T09:57:25.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:25.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:25.613484+0000) 2024-03-06T09:57:26.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:26.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:26.613751+0000) 2024-03-06T09:57:27.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:27.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:27.614010+0000) 2024-03-06T09:57:28.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:28.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:28.614271+0000) 2024-03-06T09:57:29.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:29.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:29.614502+0000) 2024-03-06T09:57:30.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:30.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:30.614740+0000) 2024-03-06T09:57:31.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:31.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:31.614971+0000) 2024-03-06T09:57:32.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:32.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:32.615212+0000) 2024-03-06T09:57:33.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:33.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:33.615411+0000) 2024-03-06T09:57:33.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:33.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:34.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:34.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:34.615713+0000) 2024-03-06T09:57:35.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:35.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:35.615964+0000) 2024-03-06T09:57:36.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:36.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:36.616208+0000) 2024-03-06T09:57:37.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:37.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:37.616445+0000) 2024-03-06T09:57:38.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:38.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:38.616746+0000) 2024-03-06T09:57:39.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:39.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:39.616971+0000) 2024-03-06T09:57:40.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:40.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:40.617169+0000) 2024-03-06T09:57:41.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:41.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:41.617402+0000) 2024-03-06T09:57:42.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:42.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:42.617689+0000) 2024-03-06T09:57:43.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:43.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:43.617963+0000) 2024-03-06T09:57:44.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:44.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:44.618233+0000) 2024-03-06T09:57:45.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:45.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:45.618503+0000) 2024-03-06T09:57:46.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:46.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:46.618786+0000) 2024-03-06T09:57:47.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:47.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:47.619060+0000) 2024-03-06T09:57:48.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:48.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:48.619328+0000) 2024-03-06T09:57:49.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:49.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:49.619598+0000) 2024-03-06T09:57:50.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:50.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:50.619894+0000) 2024-03-06T09:57:51.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:51.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:51.620163+0000) 2024-03-06T09:57:52.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:52.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:52.620410+0000) 2024-03-06T09:57:53.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:53.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:53.620757+0000) 2024-03-06T09:57:54.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:54.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:54.621041+0000) 2024-03-06T09:57:55.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:55.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:55.621291+0000) 2024-03-06T09:57:56.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:56.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:56.621572+0000) 2024-03-06T09:57:57.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:57.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:57.621817+0000) 2024-03-06T09:57:58.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:58.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:58.622085+0000) 2024-03-06T09:57:59.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:57:59.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:59.622345+0000) 2024-03-06T09:58:00.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:00.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:00.622605+0000) 2024-03-06T09:58:01.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:01.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:01.622919+0000) 2024-03-06T09:58:02.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:02.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:02.623195+0000) 2024-03-06T09:58:03.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:03.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:03.623476+0000) 2024-03-06T09:58:03.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:03.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:04.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:04.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:04.623756+0000) 2024-03-06T09:58:05.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:05.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:05.624059+0000) 2024-03-06T09:58:06.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:06.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:06.624283+0000) 2024-03-06T09:58:07.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:07.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:07.624555+0000) 2024-03-06T09:58:08.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:08.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:08.624848+0000) 2024-03-06T09:58:09.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:09.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:09.625098+0000) 2024-03-06T09:58:10.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:10.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:10.625373+0000) 2024-03-06T09:58:11.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:11.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:11.625673+0000) 2024-03-06T09:58:12.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:12.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:12.625953+0000) 2024-03-06T09:58:13.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:13.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:13.626196+0000) 2024-03-06T09:58:14.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:14.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:14.626449+0000) 2024-03-06T09:58:15.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:15.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:15.626745+0000) 2024-03-06T09:58:16.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:16.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:16.627018+0000) 2024-03-06T09:58:17.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:17.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:17.627276+0000) 2024-03-06T09:58:18.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:18.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:18.627551+0000) 2024-03-06T09:58:19.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:19.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:19.627784+0000) 2024-03-06T09:58:20.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:20.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:20.628039+0000) 2024-03-06T09:58:21.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:21.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:21.628294+0000) 2024-03-06T09:58:22.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:22.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:22.628533+0000) 2024-03-06T09:58:23.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:23.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:23.628795+0000) 2024-03-06T09:58:24.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:24.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:24.629051+0000) 2024-03-06T09:58:25.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:25.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:25.629305+0000) 2024-03-06T09:58:26.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:26.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:26.629553+0000) 2024-03-06T09:58:27.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:27.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:27.629795+0000) 2024-03-06T09:58:28.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:28.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:28.630057+0000) 2024-03-06T09:58:29.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:29.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:29.630297+0000) 2024-03-06T09:58:30.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:30.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:30.630531+0000) 2024-03-06T09:58:31.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:31.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:31.630779+0000) 2024-03-06T09:58:32.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:32.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:32.631021+0000) 2024-03-06T09:58:33.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:33.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:33.631281+0000) 2024-03-06T09:58:34.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:34.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:34.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:34.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:34.631515+0000) 2024-03-06T09:58:35.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:35.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:35.631746+0000) 2024-03-06T09:58:36.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:36.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:36.632067+0000) 2024-03-06T09:58:37.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:37.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:37.632371+0000) 2024-03-06T09:58:38.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:38.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:38.632674+0000) 2024-03-06T09:58:39.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:39.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:39.632935+0000) 2024-03-06T09:58:40.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:40.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:40.633188+0000) 2024-03-06T09:58:41.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:41.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:41.633440+0000) 2024-03-06T09:58:42.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:42.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:42.633670+0000) 2024-03-06T09:58:43.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:43.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:43.633925+0000) 2024-03-06T09:58:44.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:44.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:44.634196+0000) 2024-03-06T09:58:45.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:45.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:45.634448+0000) 2024-03-06T09:58:46.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:46.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:46.634744+0000) 2024-03-06T09:58:47.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:47.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:47.635004+0000) 2024-03-06T09:58:48.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:48.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:48.635264+0000) 2024-03-06T09:58:49.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:49.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:49.635495+0000) 2024-03-06T09:58:50.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:50.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:50.635731+0000) 2024-03-06T09:58:51.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:51.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:51.635962+0000) 2024-03-06T09:58:52.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:52.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:52.636210+0000) 2024-03-06T09:58:53.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:53.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:53.636520+0000) 2024-03-06T09:58:54.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:54.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:54.636819+0000) 2024-03-06T09:58:55.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:55.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:55.637107+0000) 2024-03-06T09:58:56.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:56.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:56.637410+0000) 2024-03-06T09:58:57.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:57.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:57.637745+0000) 2024-03-06T09:58:58.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:58.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:58.638031+0000) 2024-03-06T09:58:59.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:58:59.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:59.638319+0000) 2024-03-06T09:59:00.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:00.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:00.638613+0000) 2024-03-06T09:59:01.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:01.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:01.638984+0000) 2024-03-06T09:59:02.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:02.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:02.639270+0000) 2024-03-06T09:59:03.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:03.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:03.639567+0000) 2024-03-06T09:59:04.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:04.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:04.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:04.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:04.639880+0000) 2024-03-06T09:59:05.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:05.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:05.640172+0000) 2024-03-06T09:59:06.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:06.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:06.640497+0000) 2024-03-06T09:59:07.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:07.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:07.640817+0000) 2024-03-06T09:59:08.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:08.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:08.641106+0000) 2024-03-06T09:59:09.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:09.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:09.641384+0000) 2024-03-06T09:59:10.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:10.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:10.641729+0000) 2024-03-06T09:59:11.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:11.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:11.641967+0000) 2024-03-06T09:59:12.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:12.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:12.642281+0000) 2024-03-06T09:59:13.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:13.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:13.642577+0000) 2024-03-06T09:59:14.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:14.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:14.642875+0000) 2024-03-06T09:59:15.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:15.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:15.643156+0000) 2024-03-06T09:59:16.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:16.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:16.643470+0000) 2024-03-06T09:59:17.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:17.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:17.643766+0000) 2024-03-06T09:59:18.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:18.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:18.644016+0000) 2024-03-06T09:59:19.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:19.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:19.644314+0000) 2024-03-06T09:59:20.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:20.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:20.644599+0000) 2024-03-06T09:59:21.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:21.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:21.644923+0000) 2024-03-06T09:59:22.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:22.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:22.645180+0000) 2024-03-06T09:59:23.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:23.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:23.645436+0000) 2024-03-06T09:59:24.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:24.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:24.645730+0000) 2024-03-06T09:59:25.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:25.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:25.645982+0000) 2024-03-06T09:59:26.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:26.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:26.646242+0000) 2024-03-06T09:59:27.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:27.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:27.646493+0000) 2024-03-06T09:59:28.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:28.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:28.646751+0000) 2024-03-06T09:59:29.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:29.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:29.646987+0000) 2024-03-06T09:59:30.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:30.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:30.647227+0000) 2024-03-06T09:59:31.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:31.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:31.647474+0000) 2024-03-06T09:59:32.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:32.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:32.647751+0000) 2024-03-06T09:59:33.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:33.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:33.648008+0000) 2024-03-06T09:59:34.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:34.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:34.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:34.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:34.648265+0000) 2024-03-06T09:59:35.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:35.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:35.648452+0000) 2024-03-06T09:59:36.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:36.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:36.648751+0000) 2024-03-06T09:59:37.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:37.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:37.648999+0000) 2024-03-06T09:59:38.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:38.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:38.649241+0000) 2024-03-06T09:59:39.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:39.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:39.649525+0000) 2024-03-06T09:59:40.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:40.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:40.649780+0000) 2024-03-06T09:59:41.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:41.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:41.650014+0000) 2024-03-06T09:59:42.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:42.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:42.650262+0000) 2024-03-06T09:59:43.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:43.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:43.650515+0000) 2024-03-06T09:59:44.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:44.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:44.650772+0000) 2024-03-06T09:59:45.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:45.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:45.651008+0000) 2024-03-06T09:59:46.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:46.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:46.651252+0000) 2024-03-06T09:59:47.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:47.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:47.651483+0000) 2024-03-06T09:59:48.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:48.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:48.651753+0000) 2024-03-06T09:59:49.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:49.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:49.651992+0000) 2024-03-06T09:59:50.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:50.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:50.652233+0000) 2024-03-06T09:59:51.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:51.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:51.652534+0000) 2024-03-06T09:59:52.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:52.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:52.652776+0000) 2024-03-06T09:59:53.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:53.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:53.653008+0000) 2024-03-06T09:59:54.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:54.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:54.653247+0000) 2024-03-06T09:59:55.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:55.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:55.653501+0000) 2024-03-06T09:59:56.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:56.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:56.653769+0000) 2024-03-06T09:59:57.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:57.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:57.654005+0000) 2024-03-06T09:59:58.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:58.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:58.654247+0000) 2024-03-06T09:59:59.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T09:59:59.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:59.654502+0000) 2024-03-06T10:00:00.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:00.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:00.654760+0000) 2024-03-06T10:00:01.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:01.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:01.655011+0000) 2024-03-06T10:00:02.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:02.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:02.655262+0000) 2024-03-06T10:00:03.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:03.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:03.655502+0000) 2024-03-06T10:00:04.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:04.543 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:04.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:04.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:04.655739+0000) 2024-03-06T10:00:05.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:05.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:05.656014+0000) 2024-03-06T10:00:06.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:06.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:06.656281+0000) 2024-03-06T10:00:07.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:07.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:07.656515+0000) 2024-03-06T10:00:08.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:08.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:08.656772+0000) 2024-03-06T10:00:09.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:09.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:09.657015+0000) 2024-03-06T10:00:10.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:10.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:10.657246+0000) 2024-03-06T10:00:11.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:11.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:11.657496+0000) 2024-03-06T10:00:12.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:12.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:12.657749+0000) 2024-03-06T10:00:13.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:13.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:13.658009+0000) 2024-03-06T10:00:14.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:14.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:14.658269+0000) 2024-03-06T10:00:15.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:15.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:15.658519+0000) 2024-03-06T10:00:16.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:16.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:16.658792+0000) 2024-03-06T10:00:17.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:17.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:17.659047+0000) 2024-03-06T10:00:18.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:18.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:18.659287+0000) 2024-03-06T10:00:19.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:19.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:19.659526+0000) 2024-03-06T10:00:20.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:20.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:20.659764+0000) 2024-03-06T10:00:21.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:21.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:21.660002+0000) 2024-03-06T10:00:22.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:22.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:22.660246+0000) 2024-03-06T10:00:23.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:23.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:23.660501+0000) 2024-03-06T10:00:24.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:24.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:24.660765+0000) 2024-03-06T10:00:25.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:25.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:25.661019+0000) 2024-03-06T10:00:26.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:26.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:26.661285+0000) 2024-03-06T10:00:27.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:27.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:27.661533+0000) 2024-03-06T10:00:28.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:28.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:28.661795+0000) 2024-03-06T10:00:29.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:29.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:29.662061+0000) 2024-03-06T10:00:30.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:30.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:30.662314+0000) 2024-03-06T10:00:31.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:31.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:31.662569+0000) 2024-03-06T10:00:32.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:32.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:32.662821+0000) 2024-03-06T10:00:33.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:33.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:33.663081+0000) 2024-03-06T10:00:34.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:34.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:34.663333+0000) 2024-03-06T10:00:34.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:34.705 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:35.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:35.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:35.663586+0000) 2024-03-06T10:00:36.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:36.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:36.663855+0000) 2024-03-06T10:00:37.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:37.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:37.664084+0000) 2024-03-06T10:00:38.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:38.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:38.664347+0000) 2024-03-06T10:00:39.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:39.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:39.664689+0000) 2024-03-06T10:00:40.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:40.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:40.664949+0000) 2024-03-06T10:00:41.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:41.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:41.665202+0000) 2024-03-06T10:00:42.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:42.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:42.665454+0000) 2024-03-06T10:00:43.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:43.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:43.665785+0000) 2024-03-06T10:00:44.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:44.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:44.666078+0000) 2024-03-06T10:00:45.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:45.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:45.666391+0000) 2024-03-06T10:00:46.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:46.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:46.666733+0000) 2024-03-06T10:00:47.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:47.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:47.667027+0000) 2024-03-06T10:00:48.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:48.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:48.667307+0000) 2024-03-06T10:00:49.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:49.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:49.667622+0000) 2024-03-06T10:00:50.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:50.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:50.667995+0000) 2024-03-06T10:00:51.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:51.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:51.668304+0000) 2024-03-06T10:00:52.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:52.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:52.668614+0000) 2024-03-06T10:00:53.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:53.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:53.668972+0000) 2024-03-06T10:00:54.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:54.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:54.669277+0000) 2024-03-06T10:00:55.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:55.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:55.669575+0000) 2024-03-06T10:00:56.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:56.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:56.669929+0000) 2024-03-06T10:00:57.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:57.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:57.670221+0000) 2024-03-06T10:00:58.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:58.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:58.670539+0000) 2024-03-06T10:00:59.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:00:59.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:59.670860+0000) 2024-03-06T10:01:00.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:00.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:00.671173+0000) 2024-03-06T10:01:01.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:01.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:01.671412+0000) 2024-03-06T10:01:02.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:02.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:02.671684+0000) 2024-03-06T10:01:03.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:03.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:03.671980+0000) 2024-03-06T10:01:04.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:04.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:04.672293+0000) 2024-03-06T10:01:04.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:04.852 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:05.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:05.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:05.672582+0000) 2024-03-06T10:01:06.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:06.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:06.672942+0000) 2024-03-06T10:01:07.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:07.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:07.673228+0000) 2024-03-06T10:01:08.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:08.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:08.673545+0000) 2024-03-06T10:01:09.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:09.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:09.673835+0000) 2024-03-06T10:01:10.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:10.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:10.674145+0000) 2024-03-06T10:01:11.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:11.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:11.674425+0000) 2024-03-06T10:01:12.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:12.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:12.674753+0000) 2024-03-06T10:01:13.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:13.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:13.675054+0000) 2024-03-06T10:01:14.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:14.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:14.675347+0000) 2024-03-06T10:01:15.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:15.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:15.675687+0000) 2024-03-06T10:01:16.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:16.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:16.675967+0000) 2024-03-06T10:01:17.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:17.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:17.676285+0000) 2024-03-06T10:01:18.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:18.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:18.676599+0000) 2024-03-06T10:01:19.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:19.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:19.676940+0000) 2024-03-06T10:01:20.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:20.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:20.677232+0000) 2024-03-06T10:01:21.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:21.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:21.677537+0000) 2024-03-06T10:01:22.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:22.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:22.677850+0000) 2024-03-06T10:01:23.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:23.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:23.678155+0000) 2024-03-06T10:01:24.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:24.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:24.678461+0000) 2024-03-06T10:01:25.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:25.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:25.678770+0000) 2024-03-06T10:01:26.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:26.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:26.679102+0000) 2024-03-06T10:01:27.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:27.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:27.679357+0000) 2024-03-06T10:01:28.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:28.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:28.679613+0000) 2024-03-06T10:01:29.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:29.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:29.679837+0000) 2024-03-06T10:01:30.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:30.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:30.680110+0000) 2024-03-06T10:01:31.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:31.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:31.680290+0000) 2024-03-06T10:01:32.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:32.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:32.680544+0000) 2024-03-06T10:01:33.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:33.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:33.680816+0000) 2024-03-06T10:01:34.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:34.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:34.681079+0000) 2024-03-06T10:01:34.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:35.000 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:35.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:35.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:35.681336+0000) 2024-03-06T10:01:36.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:36.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:36.681603+0000) 2024-03-06T10:01:37.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:37.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:37.681876+0000) 2024-03-06T10:01:38.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:38.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:38.682170+0000) 2024-03-06T10:01:39.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:39.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:39.682421+0000) 2024-03-06T10:01:40.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:40.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:40.682699+0000) 2024-03-06T10:01:41.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:41.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:41.682962+0000) 2024-03-06T10:01:42.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:42.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:42.683216+0000) 2024-03-06T10:01:43.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:43.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:43.683473+0000) 2024-03-06T10:01:44.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:44.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:44.683769+0000) 2024-03-06T10:01:45.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:45.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:45.684002+0000) 2024-03-06T10:01:46.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:46.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:46.684270+0000) 2024-03-06T10:01:47.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:47.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:47.684506+0000) 2024-03-06T10:01:48.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:48.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:48.684767+0000) 2024-03-06T10:01:49.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:49.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:49.685033+0000) 2024-03-06T10:01:50.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:50.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:50.685291+0000) 2024-03-06T10:01:51.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:51.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:51.685550+0000) 2024-03-06T10:01:52.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:52.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:52.685806+0000) 2024-03-06T10:01:53.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:53.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:53.686060+0000) 2024-03-06T10:01:54.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:54.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:54.686324+0000) 2024-03-06T10:01:55.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:55.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:55.686584+0000) 2024-03-06T10:01:56.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:56.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:56.686840+0000) 2024-03-06T10:01:57.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:57.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:57.687093+0000) 2024-03-06T10:01:58.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:58.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:58.687351+0000) 2024-03-06T10:01:59.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:01:59.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:59.687618+0000) 2024-03-06T10:02:00.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:00.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:00.687917+0000) 2024-03-06T10:02:01.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:01.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:01.688162+0000) 2024-03-06T10:02:02.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:02.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:02.688396+0000) 2024-03-06T10:02:03.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:03.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:03.688696+0000) 2024-03-06T10:02:04.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:04.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:04.688968+0000) 2024-03-06T10:02:05.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:05.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:05.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:05.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:05.689229+0000) 2024-03-06T10:02:06.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:06.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:06.689478+0000) 2024-03-06T10:02:07.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:07.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:07.689754+0000) 2024-03-06T10:02:08.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:08.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:08.690002+0000) 2024-03-06T10:02:09.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:09.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:09.690270+0000) 2024-03-06T10:02:10.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:10.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:10.690501+0000) 2024-03-06T10:02:11.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:11.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:11.690759+0000) 2024-03-06T10:02:12.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:12.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:12.691062+0000) 2024-03-06T10:02:13.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:13.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:13.691358+0000) 2024-03-06T10:02:14.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:14.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:14.691720+0000) 2024-03-06T10:02:15.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:15.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:15.692025+0000) 2024-03-06T10:02:16.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:16.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:16.692353+0000) 2024-03-06T10:02:17.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:17.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:17.692675+0000) 2024-03-06T10:02:18.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:18.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:18.692985+0000) 2024-03-06T10:02:19.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:19.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:19.693265+0000) 2024-03-06T10:02:20.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:20.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:20.693584+0000) 2024-03-06T10:02:21.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:21.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:21.693905+0000) 2024-03-06T10:02:22.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:22.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:22.694205+0000) 2024-03-06T10:02:23.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:23.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:23.694483+0000) 2024-03-06T10:02:24.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:24.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:24.694759+0000) 2024-03-06T10:02:25.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:25.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:25.694948+0000) 2024-03-06T10:02:26.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:26.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:26.695198+0000) 2024-03-06T10:02:27.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:27.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:27.695478+0000) 2024-03-06T10:02:28.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:28.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:28.695798+0000) 2024-03-06T10:02:29.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:29.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:29.696095+0000) 2024-03-06T10:02:30.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:30.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:30.696408+0000) 2024-03-06T10:02:31.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:31.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:31.696721+0000) 2024-03-06T10:02:32.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:32.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:32.697009+0000) 2024-03-06T10:02:33.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:33.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:33.697263+0000) 2024-03-06T10:02:34.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:34.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:34.697518+0000) 2024-03-06T10:02:35.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:35.302 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:35.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:35.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:35.697769+0000) 2024-03-06T10:02:36.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:36.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:36.698047+0000) 2024-03-06T10:02:37.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:37.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:37.698333+0000) 2024-03-06T10:02:38.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:38.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:38.698574+0000) 2024-03-06T10:02:39.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:39.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:39.698881+0000) 2024-03-06T10:02:40.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:40.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:40.699195+0000) 2024-03-06T10:02:41.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:41.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:41.699462+0000) 2024-03-06T10:02:42.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:42.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:42.699754+0000) 2024-03-06T10:02:43.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:43.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:43.700016+0000) 2024-03-06T10:02:44.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:44.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:44.700288+0000) 2024-03-06T10:02:45.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:45.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:45.700527+0000) 2024-03-06T10:02:46.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:46.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:46.700782+0000) 2024-03-06T10:02:47.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:47.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:47.701076+0000) 2024-03-06T10:02:48.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:48.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:48.701392+0000) 2024-03-06T10:02:49.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:49.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:49.701731+0000) 2024-03-06T10:02:50.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:50.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:50.702020+0000) 2024-03-06T10:02:51.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:51.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:51.702312+0000) 2024-03-06T10:02:52.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:52.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:52.702609+0000) 2024-03-06T10:02:53.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:53.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:53.702973+0000) 2024-03-06T10:02:54.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:54.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:54.703286+0000) 2024-03-06T10:02:55.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:55.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:55.703599+0000) 2024-03-06T10:02:56.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:56.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:56.703896+0000) 2024-03-06T10:02:57.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:57.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:57.704186+0000) 2024-03-06T10:02:58.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:58.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:58.704476+0000) 2024-03-06T10:02:59.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:02:59.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:59.704799+0000) 2024-03-06T10:03:00.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:00.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:00.705099+0000) 2024-03-06T10:03:01.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:01.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:01.705386+0000) 2024-03-06T10:03:02.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:02.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:02.705725+0000) 2024-03-06T10:03:03.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:03.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:03.706035+0000) 2024-03-06T10:03:04.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:04.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:04.706347+0000) 2024-03-06T10:03:05.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:05.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:05.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:05.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:05.706688+0000) 2024-03-06T10:03:06.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:06.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:06.706977+0000) 2024-03-06T10:03:07.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:07.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:07.707276+0000) 2024-03-06T10:03:08.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:08.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:08.707585+0000) 2024-03-06T10:03:09.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:09.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:09.707911+0000) 2024-03-06T10:03:10.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:10.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:10.708206+0000) 2024-03-06T10:03:11.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:11.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:11.708520+0000) 2024-03-06T10:03:12.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:12.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:12.708835+0000) 2024-03-06T10:03:13.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:13.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:13.709128+0000) 2024-03-06T10:03:14.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:14.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:14.709444+0000) 2024-03-06T10:03:15.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:15.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:15.709786+0000) 2024-03-06T10:03:16.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:16.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:16.710090+0000) 2024-03-06T10:03:17.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:17.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:17.710374+0000) 2024-03-06T10:03:18.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:18.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:18.710740+0000) 2024-03-06T10:03:19.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:19.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:19.711037+0000) 2024-03-06T10:03:20.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:20.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:20.711328+0000) 2024-03-06T10:03:21.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:21.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:21.711596+0000) 2024-03-06T10:03:22.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:22.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:22.711949+0000) 2024-03-06T10:03:23.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:23.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:23.712209+0000) 2024-03-06T10:03:24.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:24.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:24.712501+0000) 2024-03-06T10:03:25.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:25.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:25.712793+0000) 2024-03-06T10:03:26.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:26.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:26.713099+0000) 2024-03-06T10:03:27.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:27.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:27.713413+0000) 2024-03-06T10:03:28.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:28.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:28.713770+0000) 2024-03-06T10:03:29.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:29.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:29.714073+0000) 2024-03-06T10:03:30.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:30.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:30.714441+0000) 2024-03-06T10:03:31.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:31.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:31.714790+0000) 2024-03-06T10:03:32.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:32.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:32.715100+0000) 2024-03-06T10:03:33.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:33.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:33.715402+0000) 2024-03-06T10:03:34.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:34.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:34.715755+0000) 2024-03-06T10:03:35.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:35.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:35.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:35.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:35.715944+0000) 2024-03-06T10:03:36.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:36.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:36.716204+0000) 2024-03-06T10:03:37.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:37.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:37.716484+0000) 2024-03-06T10:03:38.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:38.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:38.716754+0000) 2024-03-06T10:03:39.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:39.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:39.717054+0000) 2024-03-06T10:03:40.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:40.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:40.717355+0000) 2024-03-06T10:03:41.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:41.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:41.717706+0000) 2024-03-06T10:03:42.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:42.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:42.718010+0000) 2024-03-06T10:03:43.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:43.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:43.718328+0000) 2024-03-06T10:03:44.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:44.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:44.718680+0000) 2024-03-06T10:03:45.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:45.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:45.718982+0000) 2024-03-06T10:03:46.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:46.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:46.719311+0000) 2024-03-06T10:03:47.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:47.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:47.719572+0000) 2024-03-06T10:03:48.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:48.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:48.719857+0000) 2024-03-06T10:03:49.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:49.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:49.720156+0000) 2024-03-06T10:03:50.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:50.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:50.720468+0000) 2024-03-06T10:03:51.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:51.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:51.720774+0000) 2024-03-06T10:03:52.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:52.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:52.721087+0000) 2024-03-06T10:03:53.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:53.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:53.721398+0000) 2024-03-06T10:03:54.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:54.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:54.721685+0000) 2024-03-06T10:03:55.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:55.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:55.722022+0000) 2024-03-06T10:03:56.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:56.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:56.722330+0000) 2024-03-06T10:03:57.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:57.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:57.722675+0000) 2024-03-06T10:03:58.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:58.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:58.722977+0000) 2024-03-06T10:03:59.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:03:59.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:59.723289+0000) 2024-03-06T10:04:00.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:00.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:00.723576+0000) 2024-03-06T10:04:01.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:01.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:01.723857+0000) 2024-03-06T10:04:02.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:02.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:02.724151+0000) 2024-03-06T10:04:03.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:03.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:03.724451+0000) 2024-03-06T10:04:04.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:04.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:04.724772+0000) 2024-03-06T10:04:05.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:05.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:05.725073+0000) 2024-03-06T10:04:05.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:05.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:06.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:06.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:06.725401+0000) 2024-03-06T10:04:07.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:07.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:07.725748+0000) 2024-03-06T10:04:08.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:08.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:08.725993+0000) 2024-03-06T10:04:09.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:09.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:09.726306+0000) 2024-03-06T10:04:10.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:10.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:10.726606+0000) 2024-03-06T10:04:11.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:11.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:11.726957+0000) 2024-03-06T10:04:12.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:12.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:12.727257+0000) 2024-03-06T10:04:13.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:13.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:13.727536+0000) 2024-03-06T10:04:14.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:14.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:14.727845+0000) 2024-03-06T10:04:15.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:15.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:15.728159+0000) 2024-03-06T10:04:16.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:16.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:16.728483+0000) 2024-03-06T10:04:17.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:17.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:17.728779+0000) 2024-03-06T10:04:18.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:18.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:18.729084+0000) 2024-03-06T10:04:19.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:19.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:19.729397+0000) 2024-03-06T10:04:20.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:20.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:20.729745+0000) 2024-03-06T10:04:21.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:21.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:21.730041+0000) 2024-03-06T10:04:22.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:22.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:22.730332+0000) 2024-03-06T10:04:23.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:23.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:23.730608+0000) 2024-03-06T10:04:24.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:24.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:24.730933+0000) 2024-03-06T10:04:25.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:25.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:25.731252+0000) 2024-03-06T10:04:26.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:26.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:26.731576+0000) 2024-03-06T10:04:27.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:27.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:27.731823+0000) 2024-03-06T10:04:28.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:28.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:28.732079+0000) 2024-03-06T10:04:29.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:29.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:29.732389+0000) 2024-03-06T10:04:30.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:30.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:30.732684+0000) 2024-03-06T10:04:31.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:31.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:31.732963+0000) 2024-03-06T10:04:32.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:32.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:32.733272+0000) 2024-03-06T10:04:33.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:33.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:33.733573+0000) 2024-03-06T10:04:34.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:34.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:34.733878+0000) 2024-03-06T10:04:35.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:35.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:35.734186+0000) 2024-03-06T10:04:35.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:35.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:36.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:36.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:36.734483+0000) 2024-03-06T10:04:37.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:37.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:37.734758+0000) 2024-03-06T10:04:38.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:38.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:38.735055+0000) 2024-03-06T10:04:39.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:39.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:39.735355+0000) 2024-03-06T10:04:40.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:40.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:40.735703+0000) 2024-03-06T10:04:41.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:41.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:41.735976+0000) 2024-03-06T10:04:42.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:42.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:42.736288+0000) 2024-03-06T10:04:43.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:43.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:43.736585+0000) 2024-03-06T10:04:44.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:44.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:44.736918+0000) 2024-03-06T10:04:45.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:45.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:45.737217+0000) 2024-03-06T10:04:46.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:46.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:46.737525+0000) 2024-03-06T10:04:47.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:47.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:47.737786+0000) 2024-03-06T10:04:48.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:48.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:48.738105+0000) 2024-03-06T10:04:49.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:49.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:49.738413+0000) 2024-03-06T10:04:50.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:50.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:50.738689+0000) 2024-03-06T10:04:51.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:51.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:51.738937+0000) 2024-03-06T10:04:52.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:52.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:52.739249+0000) 2024-03-06T10:04:53.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:53.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:53.739543+0000) 2024-03-06T10:04:54.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:54.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:54.739851+0000) 2024-03-06T10:04:55.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:55.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:55.740160+0000) 2024-03-06T10:04:56.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:56.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:56.740486+0000) 2024-03-06T10:04:57.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:57.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:57.740778+0000) 2024-03-06T10:04:58.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:58.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:58.741069+0000) 2024-03-06T10:04:59.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:04:59.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:59.741317+0000) 2024-03-06T10:05:00.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:00.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:00.741677+0000) 2024-03-06T10:05:01.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:01.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:01.741976+0000) 2024-03-06T10:05:02.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:02.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:02.742272+0000) 2024-03-06T10:05:03.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:03.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:03.742572+0000) 2024-03-06T10:05:04.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:04.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:04.742864+0000) 2024-03-06T10:05:05.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:05.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:05.743119+0000) 2024-03-06T10:05:06.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:06.081 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:06.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:06.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:06.743392+0000) 2024-03-06T10:05:07.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:07.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:07.743670+0000) 2024-03-06T10:05:08.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:08.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:08.743920+0000) 2024-03-06T10:05:09.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:09.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:09.744158+0000) 2024-03-06T10:05:10.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:10.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:10.744422+0000) 2024-03-06T10:05:11.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:11.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:11.744718+0000) 2024-03-06T10:05:12.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:12.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:12.744976+0000) 2024-03-06T10:05:13.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:13.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:13.745209+0000) 2024-03-06T10:05:14.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:14.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:14.745452+0000) 2024-03-06T10:05:15.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:15.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:15.745757+0000) 2024-03-06T10:05:16.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:16.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:16.746006+0000) 2024-03-06T10:05:17.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:17.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:17.746240+0000) 2024-03-06T10:05:18.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:18.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:18.746426+0000) 2024-03-06T10:05:19.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:19.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:19.746725+0000) 2024-03-06T10:05:20.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:20.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:20.746984+0000) 2024-03-06T10:05:21.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:21.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:21.747238+0000) 2024-03-06T10:05:22.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:22.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:22.747494+0000) 2024-03-06T10:05:23.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:23.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:23.747760+0000) 2024-03-06T10:05:24.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:24.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:24.748024+0000) 2024-03-06T10:05:25.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:25.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:25.748301+0000) 2024-03-06T10:05:26.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:26.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:26.748577+0000) 2024-03-06T10:05:27.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:27.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:27.748882+0000) 2024-03-06T10:05:28.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:28.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:28.749127+0000) 2024-03-06T10:05:29.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:29.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:29.749382+0000) 2024-03-06T10:05:30.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:30.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:30.749623+0000) 2024-03-06T10:05:31.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:31.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:31.749901+0000) 2024-03-06T10:05:32.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:32.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:32.750146+0000) 2024-03-06T10:05:33.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:33.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:33.750383+0000) 2024-03-06T10:05:34.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:34.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:34.750673+0000) 2024-03-06T10:05:35.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:35.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:35.750940+0000) 2024-03-06T10:05:36.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:36.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:36.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:36.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:36.751212+0000) 2024-03-06T10:05:37.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:37.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:37.751458+0000) 2024-03-06T10:05:38.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:38.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:38.751745+0000) 2024-03-06T10:05:39.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:39.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:39.752024+0000) 2024-03-06T10:05:40.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:40.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:40.752303+0000) 2024-03-06T10:05:41.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:41.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:41.752549+0000) 2024-03-06T10:05:42.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:42.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:42.752820+0000) 2024-03-06T10:05:43.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:43.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:43.753095+0000) 2024-03-06T10:05:44.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:44.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:44.753355+0000) 2024-03-06T10:05:45.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:45.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:45.753610+0000) 2024-03-06T10:05:46.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:46.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:46.753945+0000) 2024-03-06T10:05:47.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:47.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:47.754240+0000) 2024-03-06T10:05:48.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:48.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:48.754516+0000) 2024-03-06T10:05:49.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:49.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:49.754776+0000) 2024-03-06T10:05:50.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:50.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:50.755024+0000) 2024-03-06T10:05:51.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:51.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:51.755277+0000) 2024-03-06T10:05:52.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:52.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:52.755533+0000) 2024-03-06T10:05:53.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:53.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:53.755778+0000) 2024-03-06T10:05:54.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:54.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:54.756040+0000) 2024-03-06T10:05:55.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:55.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:55.756311+0000) 2024-03-06T10:05:56.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:56.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:56.756603+0000) 2024-03-06T10:05:57.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:57.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:57.756923+0000) 2024-03-06T10:05:58.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:58.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:58.757180+0000) 2024-03-06T10:05:59.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:05:59.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:59.757455+0000) 2024-03-06T10:06:00.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:00.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:00.757759+0000) 2024-03-06T10:06:01.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:01.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:01.758005+0000) 2024-03-06T10:06:02.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:02.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:02.758257+0000) 2024-03-06T10:06:03.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:03.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:03.758523+0000) 2024-03-06T10:06:04.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:04.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:04.758791+0000) 2024-03-06T10:06:05.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:05.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:05.759048+0000) 2024-03-06T10:06:06.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:06.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:06.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:06.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:06.759299+0000) 2024-03-06T10:06:07.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:07.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:07.759539+0000) 2024-03-06T10:06:08.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:08.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:08.759801+0000) 2024-03-06T10:06:09.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:09.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:09.760066+0000) 2024-03-06T10:06:10.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:10.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:10.760346+0000) 2024-03-06T10:06:11.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:11.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:11.760618+0000) 2024-03-06T10:06:12.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:12.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:12.760939+0000) 2024-03-06T10:06:13.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:13.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:13.761195+0000) 2024-03-06T10:06:14.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:14.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:14.761441+0000) 2024-03-06T10:06:15.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:15.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:15.761780+0000) 2024-03-06T10:06:16.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:16.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:16.762049+0000) 2024-03-06T10:06:17.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:17.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:17.762281+0000) 2024-03-06T10:06:18.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:18.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:18.762546+0000) 2024-03-06T10:06:19.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:19.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:19.762851+0000) 2024-03-06T10:06:20.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:20.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:20.763159+0000) 2024-03-06T10:06:21.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:21.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:21.763471+0000) 2024-03-06T10:06:22.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:22.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:22.763799+0000) 2024-03-06T10:06:23.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:23.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:23.764078+0000) 2024-03-06T10:06:24.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:24.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:24.764388+0000) 2024-03-06T10:06:25.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:25.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:25.764719+0000) 2024-03-06T10:06:26.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:26.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:26.765053+0000) 2024-03-06T10:06:27.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:27.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:27.765346+0000) 2024-03-06T10:06:28.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:28.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:28.765689+0000) 2024-03-06T10:06:29.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:29.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:29.766001+0000) 2024-03-06T10:06:30.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:30.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:30.766317+0000) 2024-03-06T10:06:31.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:31.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:31.766612+0000) 2024-03-06T10:06:32.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:32.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:32.766959+0000) 2024-03-06T10:06:33.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:33.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:33.767247+0000) 2024-03-06T10:06:34.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:34.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:34.767546+0000) 2024-03-06T10:06:35.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:35.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:35.767858+0000) 2024-03-06T10:06:36.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:36.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:36.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:36.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:36.768183+0000) 2024-03-06T10:06:37.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:37.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:37.768494+0000) 2024-03-06T10:06:38.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:38.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:38.768800+0000) 2024-03-06T10:06:39.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:39.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:39.769120+0000) 2024-03-06T10:06:40.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:40.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:40.769426+0000) 2024-03-06T10:06:41.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:41.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:41.769742+0000) 2024-03-06T10:06:42.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:42.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:42.770049+0000) 2024-03-06T10:06:43.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:43.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:43.770347+0000) 2024-03-06T10:06:44.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:44.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:44.770713+0000) 2024-03-06T10:06:45.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:45.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:45.771037+0000) 2024-03-06T10:06:46.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:46.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:46.771367+0000) 2024-03-06T10:06:47.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:47.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:47.771721+0000) 2024-03-06T10:06:48.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:48.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:48.771958+0000) 2024-03-06T10:06:49.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:49.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:49.772253+0000) 2024-03-06T10:06:50.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:50.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:50.772554+0000) 2024-03-06T10:06:51.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:51.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:51.772836+0000) 2024-03-06T10:06:52.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:52.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:52.773146+0000) 2024-03-06T10:06:53.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:53.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:53.773435+0000) 2024-03-06T10:06:54.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:54.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:54.773783+0000) 2024-03-06T10:06:55.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:55.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:55.774083+0000) 2024-03-06T10:06:56.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:56.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:56.774402+0000) 2024-03-06T10:06:57.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:57.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:57.774717+0000) 2024-03-06T10:06:58.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:58.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:58.775028+0000) 2024-03-06T10:06:59.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:06:59.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:59.775308+0000) 2024-03-06T10:07:00.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:00.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:00.775590+0000) 2024-03-06T10:07:01.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:01.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:01.775896+0000) 2024-03-06T10:07:02.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:02.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:02.776173+0000) 2024-03-06T10:07:03.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:03.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:03.776445+0000) 2024-03-06T10:07:04.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:04.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:04.776756+0000) 2024-03-06T10:07:05.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:05.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:05.777048+0000) 2024-03-06T10:07:06.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:06.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:06.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:06.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:06.777248+0000) 2024-03-06T10:07:07.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:07.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:07.777501+0000) 2024-03-06T10:07:08.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:08.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:08.777778+0000) 2024-03-06T10:07:09.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:09.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:09.778052+0000) 2024-03-06T10:07:10.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:10.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:10.778330+0000) 2024-03-06T10:07:11.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:11.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:11.778583+0000) 2024-03-06T10:07:12.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:12.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:12.778894+0000) 2024-03-06T10:07:13.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:13.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:13.779129+0000) 2024-03-06T10:07:14.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:14.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:14.779418+0000) 2024-03-06T10:07:15.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:15.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:15.779733+0000) 2024-03-06T10:07:16.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:16.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:16.780029+0000) 2024-03-06T10:07:17.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:17.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:17.780291+0000) 2024-03-06T10:07:18.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:18.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:18.780559+0000) 2024-03-06T10:07:19.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:19.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:19.780831+0000) 2024-03-06T10:07:20.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:20.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:20.781076+0000) 2024-03-06T10:07:21.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:21.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:21.781315+0000) 2024-03-06T10:07:22.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:22.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:22.781567+0000) 2024-03-06T10:07:23.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:23.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:23.781810+0000) 2024-03-06T10:07:24.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:24.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:24.782081+0000) 2024-03-06T10:07:25.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:25.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:25.782345+0000) 2024-03-06T10:07:26.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:26.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:26.782611+0000) 2024-03-06T10:07:27.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:27.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:27.782879+0000) 2024-03-06T10:07:28.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:28.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:28.783141+0000) 2024-03-06T10:07:29.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:29.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:29.783344+0000) 2024-03-06T10:07:30.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:30.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:30.783616+0000) 2024-03-06T10:07:31.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:31.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:31.783921+0000) 2024-03-06T10:07:32.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:32.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:32.784176+0000) 2024-03-06T10:07:33.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:33.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:33.784421+0000) 2024-03-06T10:07:34.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:34.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:34.784720+0000) 2024-03-06T10:07:35.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:35.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:35.784977+0000) 2024-03-06T10:07:36.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:36.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:36.785237+0000) 2024-03-06T10:07:36.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:36.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:37.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:37.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:37.785494+0000) 2024-03-06T10:07:38.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:38.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:38.785772+0000) 2024-03-06T10:07:39.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:39.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:39.786032+0000) 2024-03-06T10:07:40.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:40.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:40.786316+0000) 2024-03-06T10:07:41.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:41.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:41.786582+0000) 2024-03-06T10:07:42.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:42.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:42.786797+0000) 2024-03-06T10:07:43.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:43.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:43.787051+0000) 2024-03-06T10:07:44.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:44.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:44.787315+0000) 2024-03-06T10:07:45.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:45.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:45.787569+0000) 2024-03-06T10:07:46.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:46.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:46.787861+0000) 2024-03-06T10:07:47.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:47.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:47.788111+0000) 2024-03-06T10:07:48.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:48.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:48.788384+0000) 2024-03-06T10:07:49.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:49.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:49.788699+0000) 2024-03-06T10:07:50.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:50.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:50.788951+0000) 2024-03-06T10:07:51.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:51.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:51.789215+0000) 2024-03-06T10:07:52.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:52.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:52.789489+0000) 2024-03-06T10:07:53.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:53.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:53.789786+0000) 2024-03-06T10:07:54.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:54.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:54.790118+0000) 2024-03-06T10:07:55.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:55.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:55.790390+0000) 2024-03-06T10:07:56.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:56.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:56.790739+0000) 2024-03-06T10:07:57.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:57.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:57.790994+0000) 2024-03-06T10:07:58.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:58.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:58.791252+0000) 2024-03-06T10:07:59.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:07:59.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:59.791510+0000) 2024-03-06T10:08:00.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:00.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:00.791766+0000) 2024-03-06T10:08:01.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:01.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:01.791998+0000) 2024-03-06T10:08:02.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:02.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:02.792256+0000) 2024-03-06T10:08:03.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:03.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:03.792481+0000) 2024-03-06T10:08:04.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:04.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:04.792750+0000) 2024-03-06T10:08:05.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:05.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:05.792999+0000) 2024-03-06T10:08:06.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:06.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:06.793249+0000) 2024-03-06T10:08:07.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:07.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:07.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:07.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:07.793515+0000) 2024-03-06T10:08:08.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:08.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:08.793769+0000) 2024-03-06T10:08:09.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:09.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:09.793985+0000) 2024-03-06T10:08:10.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:10.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:10.794251+0000) 2024-03-06T10:08:11.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:11.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:11.794483+0000) 2024-03-06T10:08:12.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:12.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:12.794755+0000) 2024-03-06T10:08:13.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:13.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:13.795025+0000) 2024-03-06T10:08:14.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:14.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:14.795296+0000) 2024-03-06T10:08:15.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:15.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:15.795570+0000) 2024-03-06T10:08:16.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:16.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:16.795901+0000) 2024-03-06T10:08:17.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:17.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:17.796210+0000) 2024-03-06T10:08:18.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:18.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:18.796504+0000) 2024-03-06T10:08:19.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:19.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:19.796809+0000) 2024-03-06T10:08:20.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:20.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:20.797086+0000) 2024-03-06T10:08:21.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:21.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:21.797395+0000) 2024-03-06T10:08:22.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:22.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:22.797745+0000) 2024-03-06T10:08:23.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:23.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:23.798033+0000) 2024-03-06T10:08:24.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:24.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:24.798334+0000) 2024-03-06T10:08:25.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:25.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:25.798587+0000) 2024-03-06T10:08:26.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:26.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:26.798876+0000) 2024-03-06T10:08:27.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:27.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:27.799186+0000) 2024-03-06T10:08:28.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:28.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:28.799507+0000) 2024-03-06T10:08:29.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:29.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:29.799804+0000) 2024-03-06T10:08:30.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:30.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:30.800102+0000) 2024-03-06T10:08:31.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:31.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:31.800409+0000) 2024-03-06T10:08:32.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:32.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:32.800752+0000) 2024-03-06T10:08:33.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:33.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:33.801050+0000) 2024-03-06T10:08:34.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:34.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:34.801360+0000) 2024-03-06T10:08:35.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:35.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:35.801698+0000) 2024-03-06T10:08:36.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:36.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:36.802000+0000) 2024-03-06T10:08:37.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:37.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:37.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:37.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:37.802234+0000) 2024-03-06T10:08:38.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:38.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:38.802536+0000) 2024-03-06T10:08:39.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:39.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:39.802842+0000) 2024-03-06T10:08:40.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:40.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:40.803144+0000) 2024-03-06T10:08:41.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:41.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:41.803430+0000) 2024-03-06T10:08:42.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:42.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:42.803761+0000) 2024-03-06T10:08:43.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:43.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:43.804068+0000) 2024-03-06T10:08:44.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:44.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:44.804356+0000) 2024-03-06T10:08:45.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:45.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:45.804721+0000) 2024-03-06T10:08:46.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:46.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:46.804973+0000) 2024-03-06T10:08:47.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:47.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:47.805294+0000) 2024-03-06T10:08:48.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:48.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:48.805610+0000) 2024-03-06T10:08:49.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:49.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:49.805958+0000) 2024-03-06T10:08:50.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:50.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:50.806264+0000) 2024-03-06T10:08:51.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:51.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:51.806579+0000) 2024-03-06T10:08:52.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:52.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:52.806884+0000) 2024-03-06T10:08:53.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:53.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:53.807173+0000) 2024-03-06T10:08:54.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:54.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:54.807468+0000) 2024-03-06T10:08:55.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:55.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:55.807784+0000) 2024-03-06T10:08:56.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:56.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:56.808085+0000) 2024-03-06T10:08:57.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:57.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:57.808396+0000) 2024-03-06T10:08:58.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:58.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:58.808741+0000) 2024-03-06T10:08:59.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:08:59.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:59.809058+0000) 2024-03-06T10:09:00.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:00.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:00.809358+0000) 2024-03-06T10:09:01.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:01.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:01.809685+0000) 2024-03-06T10:09:02.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:02.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:02.810003+0000) 2024-03-06T10:09:03.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:03.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:03.810304+0000) 2024-03-06T10:09:04.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:04.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:04.810619+0000) 2024-03-06T10:09:05.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:05.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:05.810964+0000) 2024-03-06T10:09:06.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:06.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:06.811232+0000) 2024-03-06T10:09:07.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:07.308 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:07.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:07.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:07.811555+0000) 2024-03-06T10:09:08.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:08.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:08.811876+0000) 2024-03-06T10:09:09.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:09.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:09.812195+0000) 2024-03-06T10:09:10.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:10.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:10.812494+0000) 2024-03-06T10:09:11.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:11.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:11.812803+0000) 2024-03-06T10:09:12.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:12.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:12.813118+0000) 2024-03-06T10:09:13.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:13.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:13.813436+0000) 2024-03-06T10:09:14.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:14.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:14.813781+0000) 2024-03-06T10:09:15.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:15.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:15.814084+0000) 2024-03-06T10:09:16.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:16.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:16.814400+0000) 2024-03-06T10:09:17.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:17.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:17.814756+0000) 2024-03-06T10:09:18.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:18.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:18.815063+0000) 2024-03-06T10:09:19.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:19.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:19.815381+0000) 2024-03-06T10:09:20.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:20.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:20.815625+0000) 2024-03-06T10:09:21.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:21.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:21.815972+0000) 2024-03-06T10:09:22.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:22.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:22.816256+0000) 2024-03-06T10:09:23.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:23.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:23.816525+0000) 2024-03-06T10:09:24.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:24.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:24.816822+0000) 2024-03-06T10:09:25.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:25.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:25.817089+0000) 2024-03-06T10:09:26.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:26.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:26.817370+0000) 2024-03-06T10:09:27.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:27.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:27.817659+0000) 2024-03-06T10:09:28.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:28.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:28.817915+0000) 2024-03-06T10:09:29.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:29.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:29.818154+0000) 2024-03-06T10:09:30.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:30.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:30.818409+0000) 2024-03-06T10:09:31.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:31.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:31.818706+0000) 2024-03-06T10:09:32.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:32.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:32.818891+0000) 2024-03-06T10:09:33.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:33.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:33.819199+0000) 2024-03-06T10:09:34.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:34.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:34.819459+0000) 2024-03-06T10:09:35.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:35.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:35.819754+0000) 2024-03-06T10:09:36.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:36.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:36.819955+0000) 2024-03-06T10:09:37.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:37.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:37.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:37.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:37.820152+0000) 2024-03-06T10:09:38.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:38.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:38.820342+0000) 2024-03-06T10:09:39.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:39.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:39.820595+0000) 2024-03-06T10:09:40.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:40.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:40.820840+0000) 2024-03-06T10:09:41.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:41.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:41.821109+0000) 2024-03-06T10:09:42.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:42.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:42.821364+0000) 2024-03-06T10:09:43.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:43.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:43.821669+0000) 2024-03-06T10:09:44.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:44.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:44.821866+0000) 2024-03-06T10:09:45.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:45.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:45.822104+0000) 2024-03-06T10:09:46.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:46.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:46.822369+0000) 2024-03-06T10:09:47.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:47.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:47.822623+0000) 2024-03-06T10:09:48.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:48.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:48.822845+0000) 2024-03-06T10:09:49.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:49.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:49.823036+0000) 2024-03-06T10:09:50.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:50.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:50.823237+0000) 2024-03-06T10:09:51.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:51.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:51.823484+0000) 2024-03-06T10:09:52.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:52.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:52.823737+0000) 2024-03-06T10:09:53.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:53.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:53.823964+0000) 2024-03-06T10:09:54.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:54.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:54.824221+0000) 2024-03-06T10:09:55.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:55.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:55.824457+0000) 2024-03-06T10:09:56.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:56.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:56.824745+0000) 2024-03-06T10:09:57.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:57.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:57.824991+0000) 2024-03-06T10:09:58.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:58.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:58.825243+0000) 2024-03-06T10:09:59.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:09:59.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:59.825477+0000) 2024-03-06T10:10:00.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:00.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:00.825759+0000) 2024-03-06T10:10:01.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:01.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:01.826022+0000) 2024-03-06T10:10:02.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:02.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:02.826285+0000) 2024-03-06T10:10:03.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:03.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:03.826550+0000) 2024-03-06T10:10:04.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:04.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:04.826814+0000) 2024-03-06T10:10:05.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:05.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:05.827080+0000) 2024-03-06T10:10:06.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:06.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:06.827349+0000) 2024-03-06T10:10:07.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:07.607 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:07.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:07.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:07.827618+0000) 2024-03-06T10:10:08.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:08.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:08.827911+0000) 2024-03-06T10:10:09.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:09.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:09.828159+0000) 2024-03-06T10:10:10.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:10.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:10.828408+0000) 2024-03-06T10:10:11.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:11.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:11.828693+0000) 2024-03-06T10:10:12.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:12.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:12.828956+0000) 2024-03-06T10:10:13.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:13.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:13.829202+0000) 2024-03-06T10:10:14.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:14.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:14.829445+0000) 2024-03-06T10:10:15.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:15.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:15.829740+0000) 2024-03-06T10:10:16.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:16.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:16.829954+0000) 2024-03-06T10:10:17.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:17.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:17.830197+0000) 2024-03-06T10:10:18.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:18.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:18.830442+0000) 2024-03-06T10:10:19.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:19.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:19.830737+0000) 2024-03-06T10:10:20.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:20.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:20.830984+0000) 2024-03-06T10:10:21.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:21.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:21.831176+0000) 2024-03-06T10:10:22.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:22.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:22.831383+0000) 2024-03-06T10:10:23.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:23.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:23.831684+0000) 2024-03-06T10:10:24.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:24.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:24.831929+0000) 2024-03-06T10:10:25.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:25.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:25.832116+0000) 2024-03-06T10:10:26.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:26.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:26.832432+0000) 2024-03-06T10:10:27.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:27.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:27.832735+0000) 2024-03-06T10:10:28.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:28.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:28.832977+0000) 2024-03-06T10:10:29.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:29.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:29.833237+0000) 2024-03-06T10:10:30.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:30.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:30.833481+0000) 2024-03-06T10:10:31.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:31.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:31.833761+0000) 2024-03-06T10:10:32.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:32.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:32.834016+0000) 2024-03-06T10:10:33.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:33.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:33.834283+0000) 2024-03-06T10:10:34.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:34.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:34.834528+0000) 2024-03-06T10:10:35.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:35.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:35.834790+0000) 2024-03-06T10:10:36.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:36.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:36.835061+0000) 2024-03-06T10:10:37.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:37.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:37.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:37.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:37.835269+0000) 2024-03-06T10:10:38.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:38.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:38.835550+0000) 2024-03-06T10:10:39.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:39.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:39.835798+0000) 2024-03-06T10:10:40.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:40.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:40.836042+0000) 2024-03-06T10:10:41.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:41.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:41.836280+0000) 2024-03-06T10:10:42.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:42.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:42.836510+0000) 2024-03-06T10:10:43.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:43.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:43.836807+0000) 2024-03-06T10:10:44.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:44.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:44.837086+0000) 2024-03-06T10:10:45.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:45.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:45.837401+0000) 2024-03-06T10:10:46.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:46.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:46.837751+0000) 2024-03-06T10:10:47.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:47.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:47.838064+0000) 2024-03-06T10:10:48.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:48.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:48.838371+0000) 2024-03-06T10:10:49.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:49.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:49.838728+0000) 2024-03-06T10:10:50.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:50.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:50.839042+0000) 2024-03-06T10:10:51.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:51.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:51.839354+0000) 2024-03-06T10:10:52.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:52.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:52.839673+0000) 2024-03-06T10:10:53.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:53.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:53.839969+0000) 2024-03-06T10:10:54.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:54.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:54.840270+0000) 2024-03-06T10:10:55.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:55.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:55.840578+0000) 2024-03-06T10:10:56.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:56.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:56.840933+0000) 2024-03-06T10:10:57.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:57.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:57.841167+0000) 2024-03-06T10:10:58.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:58.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:58.841471+0000) 2024-03-06T10:10:59.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:10:59.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:59.841784+0000) 2024-03-06T10:11:00.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:00.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:00.842087+0000) 2024-03-06T10:11:01.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:01.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:01.842413+0000) 2024-03-06T10:11:02.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:02.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:02.842768+0000) 2024-03-06T10:11:03.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:03.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:03.843064+0000) 2024-03-06T10:11:04.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:04.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:04.843376+0000) 2024-03-06T10:11:05.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:05.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:05.843733+0000) 2024-03-06T10:11:06.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:06.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:06.844047+0000) 2024-03-06T10:11:07.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:07.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:07.844324+0000) 2024-03-06T10:11:07.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:07.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:08.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:08.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:08.844681+0000) 2024-03-06T10:11:09.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:09.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:09.844984+0000) 2024-03-06T10:11:10.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:10.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:10.845302+0000) 2024-03-06T10:11:11.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:11.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:11.845609+0000) 2024-03-06T10:11:12.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:12.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:12.845953+0000) 2024-03-06T10:11:13.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:13.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:13.846249+0000) 2024-03-06T10:11:14.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:14.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:14.846605+0000) 2024-03-06T10:11:15.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:15.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:15.846960+0000) 2024-03-06T10:11:16.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:16.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:16.847265+0000) 2024-03-06T10:11:17.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:17.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:17.847571+0000) 2024-03-06T10:11:18.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:18.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:18.847862+0000) 2024-03-06T10:11:19.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:19.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:19.848158+0000) 2024-03-06T10:11:20.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:20.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:20.848443+0000) 2024-03-06T10:11:21.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:21.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:21.848773+0000) 2024-03-06T10:11:22.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:22.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:22.849085+0000) 2024-03-06T10:11:23.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:23.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:23.849392+0000) 2024-03-06T10:11:24.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:24.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:24.849743+0000) 2024-03-06T10:11:25.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:25.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:25.849981+0000) 2024-03-06T10:11:26.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:26.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:26.850307+0000) 2024-03-06T10:11:27.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:27.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:27.850601+0000) 2024-03-06T10:11:28.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:28.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:28.850940+0000) 2024-03-06T10:11:29.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:29.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:29.851183+0000) 2024-03-06T10:11:30.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:30.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:30.851496+0000) 2024-03-06T10:11:31.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:31.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:31.851800+0000) 2024-03-06T10:11:32.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:32.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:32.852116+0000) 2024-03-06T10:11:33.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:33.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:33.852433+0000) 2024-03-06T10:11:34.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:34.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:34.852752+0000) 2024-03-06T10:11:35.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:35.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:35.853060+0000) 2024-03-06T10:11:36.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:36.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:36.853348+0000) 2024-03-06T10:11:37.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:37.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:37.853683+0000) 2024-03-06T10:11:38.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:38.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:38.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:38.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:38.853976+0000) 2024-03-06T10:11:39.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:39.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:39.854277+0000) 2024-03-06T10:11:40.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:40.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:40.854564+0000) 2024-03-06T10:11:41.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:41.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:41.854861+0000) 2024-03-06T10:11:42.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:42.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:42.855165+0000) 2024-03-06T10:11:43.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:43.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:43.855478+0000) 2024-03-06T10:11:44.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:44.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:44.855782+0000) 2024-03-06T10:11:45.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:45.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:45.856095+0000) 2024-03-06T10:11:46.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:46.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:46.856358+0000) 2024-03-06T10:11:47.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:47.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:47.856704+0000) 2024-03-06T10:11:48.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:48.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:48.857020+0000) 2024-03-06T10:11:49.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:49.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:49.857312+0000) 2024-03-06T10:11:50.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:50.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:50.857600+0000) 2024-03-06T10:11:51.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:51.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:51.857976+0000) 2024-03-06T10:11:52.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:52.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:52.858272+0000) 2024-03-06T10:11:53.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:53.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:53.858573+0000) 2024-03-06T10:11:54.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:54.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:54.858863+0000) 2024-03-06T10:11:55.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:55.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:55.859170+0000) 2024-03-06T10:11:56.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:56.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:56.859498+0000) 2024-03-06T10:11:57.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:57.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:57.859795+0000) 2024-03-06T10:11:58.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:58.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:58.860090+0000) 2024-03-06T10:11:59.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:11:59.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:59.860385+0000) 2024-03-06T10:12:00.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:00.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:00.860739+0000) 2024-03-06T10:12:01.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:01.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:01.861050+0000) 2024-03-06T10:12:02.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:02.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:02.861400+0000) 2024-03-06T10:12:03.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:03.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:03.861734+0000) 2024-03-06T10:12:04.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:04.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:04.862014+0000) 2024-03-06T10:12:05.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:05.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:05.862301+0000) 2024-03-06T10:12:06.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:06.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:06.862621+0000) 2024-03-06T10:12:07.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:07.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:07.862993+0000) 2024-03-06T10:12:08.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:08.227 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:08.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:08.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:08.863305+0000) 2024-03-06T10:12:09.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:09.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:09.863615+0000) 2024-03-06T10:12:10.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:10.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:10.863957+0000) 2024-03-06T10:12:11.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:11.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:11.864274+0000) 2024-03-06T10:12:12.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:12.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:12.864559+0000) 2024-03-06T10:12:13.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:13.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:13.864864+0000) 2024-03-06T10:12:14.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:14.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:14.865182+0000) 2024-03-06T10:12:15.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:15.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:15.865497+0000) 2024-03-06T10:12:16.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:16.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:16.865812+0000) 2024-03-06T10:12:17.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:17.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:17.866105+0000) 2024-03-06T10:12:18.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:18.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:18.866418+0000) 2024-03-06T10:12:19.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:19.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:19.866749+0000) 2024-03-06T10:12:20.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:20.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:20.866993+0000) 2024-03-06T10:12:21.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:21.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:21.867226+0000) 2024-03-06T10:12:22.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:22.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:22.867520+0000) 2024-03-06T10:12:23.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:23.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:23.867814+0000) 2024-03-06T10:12:24.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:24.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:24.868110+0000) 2024-03-06T10:12:25.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:25.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:25.868418+0000) 2024-03-06T10:12:26.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:26.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:26.868727+0000) 2024-03-06T10:12:27.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:27.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:27.868981+0000) 2024-03-06T10:12:28.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:28.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:28.869242+0000) 2024-03-06T10:12:29.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:29.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:29.869483+0000) 2024-03-06T10:12:30.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:30.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:30.869757+0000) 2024-03-06T10:12:31.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:31.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:31.870008+0000) 2024-03-06T10:12:32.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:32.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:32.870252+0000) 2024-03-06T10:12:33.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:33.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:33.870495+0000) 2024-03-06T10:12:34.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:34.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:34.870757+0000) 2024-03-06T10:12:35.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:35.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:35.871015+0000) 2024-03-06T10:12:36.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:36.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:36.871260+0000) 2024-03-06T10:12:37.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:37.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:37.871507+0000) 2024-03-06T10:12:38.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:38.365 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:38.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:38.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:38.871757+0000) 2024-03-06T10:12:39.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:39.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:39.871950+0000) 2024-03-06T10:12:40.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:40.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:40.872155+0000) 2024-03-06T10:12:41.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:41.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:41.872408+0000) 2024-03-06T10:12:42.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:42.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:42.872699+0000) 2024-03-06T10:12:43.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:43.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:43.872963+0000) 2024-03-06T10:12:44.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:44.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:44.873210+0000) 2024-03-06T10:12:45.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:45.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:45.873476+0000) 2024-03-06T10:12:46.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:46.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:46.873749+0000) 2024-03-06T10:12:47.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:47.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:47.874000+0000) 2024-03-06T10:12:48.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:48.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:48.874255+0000) 2024-03-06T10:12:49.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:49.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:49.874508+0000) 2024-03-06T10:12:50.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:50.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:50.874770+0000) 2024-03-06T10:12:51.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:51.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:51.875039+0000) 2024-03-06T10:12:52.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:52.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:52.875283+0000) 2024-03-06T10:12:53.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:53.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:53.875526+0000) 2024-03-06T10:12:54.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:54.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:54.875785+0000) 2024-03-06T10:12:55.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:55.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:55.876121+0000) 2024-03-06T10:12:56.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:56.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:56.876370+0000) 2024-03-06T10:12:57.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:57.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:57.876668+0000) 2024-03-06T10:12:58.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:58.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:58.876938+0000) 2024-03-06T10:12:59.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:12:59.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:59.877190+0000) 2024-03-06T10:13:00.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:00.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:00.877421+0000) 2024-03-06T10:13:01.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:01.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:01.877763+0000) 2024-03-06T10:13:02.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:02.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:02.878059+0000) 2024-03-06T10:13:03.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:03.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:03.878346+0000) 2024-03-06T10:13:04.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:04.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:04.878679+0000) 2024-03-06T10:13:05.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:05.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:05.878967+0000) 2024-03-06T10:13:06.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:06.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:06.879283+0000) 2024-03-06T10:13:07.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:07.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:07.879594+0000) 2024-03-06T10:13:08.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:08.522 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:08.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:08.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:08.879948+0000) 2024-03-06T10:13:09.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:09.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:09.880245+0000) 2024-03-06T10:13:10.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:10.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:10.880549+0000) 2024-03-06T10:13:11.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:11.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:11.880844+0000) 2024-03-06T10:13:12.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:12.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:12.881146+0000) 2024-03-06T10:13:13.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:13.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:13.881441+0000) 2024-03-06T10:13:14.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:14.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:14.881770+0000) 2024-03-06T10:13:15.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:15.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:15.882088+0000) 2024-03-06T10:13:16.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:16.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:16.882410+0000) 2024-03-06T10:13:17.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:17.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:17.882721+0000) 2024-03-06T10:13:18.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:18.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:18.883002+0000) 2024-03-06T10:13:19.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:19.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:19.883302+0000) 2024-03-06T10:13:20.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:20.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:20.883614+0000) 2024-03-06T10:13:21.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:21.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:21.883979+0000) 2024-03-06T10:13:22.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:22.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:22.884262+0000) 2024-03-06T10:13:23.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:23.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:23.884549+0000) 2024-03-06T10:13:24.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:24.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:24.884848+0000) 2024-03-06T10:13:25.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:25.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:25.885155+0000) 2024-03-06T10:13:26.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:26.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:26.885470+0000) 2024-03-06T10:13:27.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:27.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:27.885773+0000) 2024-03-06T10:13:28.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:28.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:28.886102+0000) 2024-03-06T10:13:29.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:29.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:29.886410+0000) 2024-03-06T10:13:30.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:30.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:30.886761+0000) 2024-03-06T10:13:31.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:31.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:31.887062+0000) 2024-03-06T10:13:32.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:32.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:32.887371+0000) 2024-03-06T10:13:33.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:33.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:33.887723+0000) 2024-03-06T10:13:34.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:34.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:34.888041+0000) 2024-03-06T10:13:35.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:35.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:35.888330+0000) 2024-03-06T10:13:36.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:36.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:36.888623+0000) 2024-03-06T10:13:37.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:37.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:37.888961+0000) 2024-03-06T10:13:38.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:38.680 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:38.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:38.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:38.889242+0000) 2024-03-06T10:13:39.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:39.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:39.889522+0000) 2024-03-06T10:13:40.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:40.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:40.889799+0000) 2024-03-06T10:13:41.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:41.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:41.890089+0000) 2024-03-06T10:13:42.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:42.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:42.890320+0000) 2024-03-06T10:13:43.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:43.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:43.890662+0000) 2024-03-06T10:13:44.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:44.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:44.890905+0000) 2024-03-06T10:13:45.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:45.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:45.891139+0000) 2024-03-06T10:13:46.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:46.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:46.891385+0000) 2024-03-06T10:13:47.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:47.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:47.891662+0000) 2024-03-06T10:13:48.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:48.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:48.891925+0000) 2024-03-06T10:13:49.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:49.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:49.892171+0000) 2024-03-06T10:13:50.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:50.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:50.892408+0000) 2024-03-06T10:13:51.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:51.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:51.892701+0000) 2024-03-06T10:13:52.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:52.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:52.892958+0000) 2024-03-06T10:13:53.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:53.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:53.893197+0000) 2024-03-06T10:13:54.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:54.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:54.893439+0000) 2024-03-06T10:13:55.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:55.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:55.893739+0000) 2024-03-06T10:13:56.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:56.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:56.894011+0000) 2024-03-06T10:13:57.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:57.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:57.894270+0000) 2024-03-06T10:13:58.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:58.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:58.894510+0000) 2024-03-06T10:13:59.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:13:59.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:59.894759+0000) 2024-03-06T10:14:00.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:00.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:00.895013+0000) 2024-03-06T10:14:01.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:01.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:01.895198+0000) 2024-03-06T10:14:02.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:02.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:02.895439+0000) 2024-03-06T10:14:03.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:03.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:03.895718+0000) 2024-03-06T10:14:04.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:04.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:04.895978+0000) 2024-03-06T10:14:05.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:05.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:05.896239+0000) 2024-03-06T10:14:06.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:06.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:06.896508+0000) 2024-03-06T10:14:07.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:07.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:07.896793+0000) 2024-03-06T10:14:08.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:08.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:08.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:08.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:08.897034+0000) 2024-03-06T10:14:09.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:09.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:09.897323+0000) 2024-03-06T10:14:10.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:10.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:10.897585+0000) 2024-03-06T10:14:11.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:11.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:11.897894+0000) 2024-03-06T10:14:12.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:12.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:12.898172+0000) 2024-03-06T10:14:13.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:13.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:13.898452+0000) 2024-03-06T10:14:14.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:14.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:14.898765+0000) 2024-03-06T10:14:15.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:15.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:15.899047+0000) 2024-03-06T10:14:16.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:16.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:16.899337+0000) 2024-03-06T10:14:17.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:17.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:17.899582+0000) 2024-03-06T10:14:18.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:18.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:18.899827+0000) 2024-03-06T10:14:19.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:19.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:19.900073+0000) 2024-03-06T10:14:20.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:20.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:20.900325+0000) 2024-03-06T10:14:21.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:21.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:21.900592+0000) 2024-03-06T10:14:22.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:22.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:22.900862+0000) 2024-03-06T10:14:23.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:23.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:23.901126+0000) 2024-03-06T10:14:24.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:24.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:24.901387+0000) 2024-03-06T10:14:25.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:25.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:25.901583+0000) 2024-03-06T10:14:26.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:26.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:26.901872+0000) 2024-03-06T10:14:27.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:27.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:27.902077+0000) 2024-03-06T10:14:28.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:28.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:28.902333+0000) 2024-03-06T10:14:29.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:29.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:29.902594+0000) 2024-03-06T10:14:30.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:30.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:30.902865+0000) 2024-03-06T10:14:31.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:31.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:31.903197+0000) 2024-03-06T10:14:32.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:32.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:32.903479+0000) 2024-03-06T10:14:33.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:33.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:33.903756+0000) 2024-03-06T10:14:34.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:34.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:34.904020+0000) 2024-03-06T10:14:35.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:35.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:35.904299+0000) 2024-03-06T10:14:36.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:36.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:36.904593+0000) 2024-03-06T10:14:37.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:37.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:37.904865+0000) 2024-03-06T10:14:38.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:38.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:38.905083+0000) 2024-03-06T10:14:38.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:38.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:39.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:39.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:39.905300+0000) 2024-03-06T10:14:40.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:40.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:40.905574+0000) 2024-03-06T10:14:41.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:41.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:41.905852+0000) 2024-03-06T10:14:42.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:42.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:42.906127+0000) 2024-03-06T10:14:43.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:43.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:43.906395+0000) 2024-03-06T10:14:44.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:44.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:44.906697+0000) 2024-03-06T10:14:45.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:45.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:45.906956+0000) 2024-03-06T10:14:46.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:46.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:46.907228+0000) 2024-03-06T10:14:47.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:47.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:47.907490+0000) 2024-03-06T10:14:48.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:48.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:48.907756+0000) 2024-03-06T10:14:49.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:49.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:49.908013+0000) 2024-03-06T10:14:50.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:50.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:50.908286+0000) 2024-03-06T10:14:51.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:51.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:51.908544+0000) 2024-03-06T10:14:52.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:52.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:52.908801+0000) 2024-03-06T10:14:53.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:53.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:53.909054+0000) 2024-03-06T10:14:54.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:54.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:54.909311+0000) 2024-03-06T10:14:55.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:55.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:55.909562+0000) 2024-03-06T10:14:56.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:56.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:56.909847+0000) 2024-03-06T10:14:57.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:57.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:57.910117+0000) 2024-03-06T10:14:58.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:58.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:58.910397+0000) 2024-03-06T10:14:59.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:14:59.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:59.910687+0000) 2024-03-06T10:15:00.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:00.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:00.910949+0000) 2024-03-06T10:15:01.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:01.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:01.911195+0000) 2024-03-06T10:15:02.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:02.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:02.911445+0000) 2024-03-06T10:15:03.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:03.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:03.911734+0000) 2024-03-06T10:15:04.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:04.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:04.911987+0000) 2024-03-06T10:15:05.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:05.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:05.912237+0000) 2024-03-06T10:15:06.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:06.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:06.912491+0000) 2024-03-06T10:15:07.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:07.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:07.912746+0000) 2024-03-06T10:15:08.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:08.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:08.912985+0000) 2024-03-06T10:15:09.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:09.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:09.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:09.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:09.913241+0000) 2024-03-06T10:15:10.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:10.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:10.913502+0000) 2024-03-06T10:15:11.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:11.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:11.913741+0000) 2024-03-06T10:15:12.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:12.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:12.913997+0000) 2024-03-06T10:15:13.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:13.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:13.914259+0000) 2024-03-06T10:15:14.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:14.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:14.914502+0000) 2024-03-06T10:15:15.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:15.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:15.914743+0000) 2024-03-06T10:15:16.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:16.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:16.915024+0000) 2024-03-06T10:15:17.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:17.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:17.915288+0000) 2024-03-06T10:15:18.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:18.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:18.915542+0000) 2024-03-06T10:15:19.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:19.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:19.915814+0000) 2024-03-06T10:15:20.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:20.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:20.916095+0000) 2024-03-06T10:15:21.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:21.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:21.916363+0000) 2024-03-06T10:15:22.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:22.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:22.916676+0000) 2024-03-06T10:15:23.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:23.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:23.916956+0000) 2024-03-06T10:15:24.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:24.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:24.917230+0000) 2024-03-06T10:15:25.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:25.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:25.917486+0000) 2024-03-06T10:15:26.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:26.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:26.917770+0000) 2024-03-06T10:15:27.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:27.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:27.918056+0000) 2024-03-06T10:15:28.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:28.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:28.918337+0000) 2024-03-06T10:15:29.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:29.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:29.918594+0000) 2024-03-06T10:15:30.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:30.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:30.918917+0000) 2024-03-06T10:15:31.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:31.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:31.919194+0000) 2024-03-06T10:15:32.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:32.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:32.919476+0000) 2024-03-06T10:15:33.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:33.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:33.919755+0000) 2024-03-06T10:15:34.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:34.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:34.920007+0000) 2024-03-06T10:15:35.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:35.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:35.920245+0000) 2024-03-06T10:15:36.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:36.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:36.920499+0000) 2024-03-06T10:15:37.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:37.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:37.920749+0000) 2024-03-06T10:15:38.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:38.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:38.920999+0000) 2024-03-06T10:15:39.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:39.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:39.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:39.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:39.921252+0000) 2024-03-06T10:15:40.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:40.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:40.921486+0000) 2024-03-06T10:15:41.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:41.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:41.921744+0000) 2024-03-06T10:15:42.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:42.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:42.921984+0000) 2024-03-06T10:15:43.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:43.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:43.922224+0000) 2024-03-06T10:15:44.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:44.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:44.922468+0000) 2024-03-06T10:15:45.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:45.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:45.922760+0000) 2024-03-06T10:15:46.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:46.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:46.923034+0000) 2024-03-06T10:15:47.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:47.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:47.923273+0000) 2024-03-06T10:15:48.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:48.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:48.923535+0000) 2024-03-06T10:15:49.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:49.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:49.923785+0000) 2024-03-06T10:15:50.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:50.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:50.924041+0000) 2024-03-06T10:15:51.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:51.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:51.924297+0000) 2024-03-06T10:15:52.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:52.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:52.924543+0000) 2024-03-06T10:15:53.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:53.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:53.924782+0000) 2024-03-06T10:15:54.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:54.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:54.925051+0000) 2024-03-06T10:15:55.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:55.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:55.925308+0000) 2024-03-06T10:15:56.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:56.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:56.925580+0000) 2024-03-06T10:15:57.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:57.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:57.925843+0000) 2024-03-06T10:15:58.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:58.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:58.926098+0000) 2024-03-06T10:15:59.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:15:59.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:59.926353+0000) 2024-03-06T10:16:00.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:00.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:00.926591+0000) 2024-03-06T10:16:01.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:01.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:01.926850+0000) 2024-03-06T10:16:02.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:02.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:02.927115+0000) 2024-03-06T10:16:03.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:03.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:03.927382+0000) 2024-03-06T10:16:04.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:04.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:04.927664+0000) 2024-03-06T10:16:05.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:05.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:05.927924+0000) 2024-03-06T10:16:06.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:06.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:06.928178+0000) 2024-03-06T10:16:07.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:07.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:07.928461+0000) 2024-03-06T10:16:08.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:08.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:08.928738+0000) 2024-03-06T10:16:09.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:09.448 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:09.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:09.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:09.929014+0000) 2024-03-06T10:16:10.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:10.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:10.929277+0000) 2024-03-06T10:16:11.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:11.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:11.929534+0000) 2024-03-06T10:16:12.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:12.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:12.929807+0000) 2024-03-06T10:16:13.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:13.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:13.930057+0000) 2024-03-06T10:16:14.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:14.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:14.930310+0000) 2024-03-06T10:16:15.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:15.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:15.930552+0000) 2024-03-06T10:16:16.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:16.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:16.930820+0000) 2024-03-06T10:16:17.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:17.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:17.931064+0000) 2024-03-06T10:16:18.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:18.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:18.931337+0000) 2024-03-06T10:16:19.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:19.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:19.931596+0000) 2024-03-06T10:16:20.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:20.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:20.931863+0000) 2024-03-06T10:16:21.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:21.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:21.932053+0000) 2024-03-06T10:16:22.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:22.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:22.932313+0000) 2024-03-06T10:16:23.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:23.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:23.932576+0000) 2024-03-06T10:16:24.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:24.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:24.932822+0000) 2024-03-06T10:16:25.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:25.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:25.933073+0000) 2024-03-06T10:16:26.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:26.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:26.933344+0000) 2024-03-06T10:16:27.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:27.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:27.933605+0000) 2024-03-06T10:16:28.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:28.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:28.933902+0000) 2024-03-06T10:16:29.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:29.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:29.934154+0000) 2024-03-06T10:16:30.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:30.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:30.934412+0000) 2024-03-06T10:16:31.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:31.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:31.934686+0000) 2024-03-06T10:16:32.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:32.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:32.934962+0000) 2024-03-06T10:16:33.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:33.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:33.935217+0000) 2024-03-06T10:16:34.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:34.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:34.935473+0000) 2024-03-06T10:16:35.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:35.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:35.935753+0000) 2024-03-06T10:16:36.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:36.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:36.936040+0000) 2024-03-06T10:16:37.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:37.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:37.936325+0000) 2024-03-06T10:16:38.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:38.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:38.936587+0000) 2024-03-06T10:16:39.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:39.591 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:39.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:39.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:39.936853+0000) 2024-03-06T10:16:40.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:40.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:40.937110+0000) 2024-03-06T10:16:41.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:41.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:41.937381+0000) 2024-03-06T10:16:42.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:42.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:42.937685+0000) 2024-03-06T10:16:43.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:43.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:43.937945+0000) 2024-03-06T10:16:44.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:44.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:44.938218+0000) 2024-03-06T10:16:45.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:45.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:45.938490+0000) 2024-03-06T10:16:46.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:46.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:46.938781+0000) 2024-03-06T10:16:47.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:47.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:47.939044+0000) 2024-03-06T10:16:48.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:48.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:48.939323+0000) 2024-03-06T10:16:49.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:49.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:49.939564+0000) 2024-03-06T10:16:50.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:50.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:50.939837+0000) 2024-03-06T10:16:51.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:51.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:51.940113+0000) 2024-03-06T10:16:52.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:52.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:52.940397+0000) 2024-03-06T10:16:53.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:53.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:53.940706+0000) 2024-03-06T10:16:54.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:54.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:54.940980+0000) 2024-03-06T10:16:55.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:55.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:55.941253+0000) 2024-03-06T10:16:56.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:56.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:56.941526+0000) 2024-03-06T10:16:57.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:57.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:57.941779+0000) 2024-03-06T10:16:58.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:58.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:58.942038+0000) 2024-03-06T10:16:59.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:16:59.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:59.942299+0000) 2024-03-06T10:17:00.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:00.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:00.942536+0000) 2024-03-06T10:17:01.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:01.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:01.942890+0000) 2024-03-06T10:17:02.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:02.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:02.943146+0000) 2024-03-06T10:17:03.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:03.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:03.943339+0000) 2024-03-06T10:17:04.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:04.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:04.943582+0000) 2024-03-06T10:17:05.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:05.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:05.943841+0000) 2024-03-06T10:17:06.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:06.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:06.944141+0000) 2024-03-06T10:17:07.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:07.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:07.944398+0000) 2024-03-06T10:17:08.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:08.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:08.944733+0000) 2024-03-06T10:17:09.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:09.747 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:09.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:09.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:09.944996+0000) 2024-03-06T10:17:10.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:10.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:10.945255+0000) 2024-03-06T10:17:11.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:11.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:11.945512+0000) 2024-03-06T10:17:12.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:12.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:12.945776+0000) 2024-03-06T10:17:13.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:13.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:13.946039+0000) 2024-03-06T10:17:14.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:14.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:14.946277+0000) 2024-03-06T10:17:15.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:15.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:15.946534+0000) 2024-03-06T10:17:16.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:16.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:16.946794+0000) 2024-03-06T10:17:17.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:17.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:17.947061+0000) 2024-03-06T10:17:18.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:18.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:18.947302+0000) 2024-03-06T10:17:19.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:19.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:19.947547+0000) 2024-03-06T10:17:20.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:20.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:20.947806+0000) 2024-03-06T10:17:21.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:21.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:21.948073+0000) 2024-03-06T10:17:22.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:22.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:22.948349+0000) 2024-03-06T10:17:23.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:23.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:23.948609+0000) 2024-03-06T10:17:24.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:24.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:24.948867+0000) 2024-03-06T10:17:25.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:25.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:25.949123+0000) 2024-03-06T10:17:26.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:26.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:26.949419+0000) 2024-03-06T10:17:27.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:27.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:27.949725+0000) 2024-03-06T10:17:28.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:28.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:28.949989+0000) 2024-03-06T10:17:29.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:29.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:29.950234+0000) 2024-03-06T10:17:30.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:30.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:30.950509+0000) 2024-03-06T10:17:31.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:31.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:31.950793+0000) 2024-03-06T10:17:32.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:32.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:32.950999+0000) 2024-03-06T10:17:33.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:33.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:33.951271+0000) 2024-03-06T10:17:34.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:34.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:34.951546+0000) 2024-03-06T10:17:35.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:35.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:35.951791+0000) 2024-03-06T10:17:36.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:36.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:36.952042+0000) 2024-03-06T10:17:37.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:37.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:37.952256+0000) 2024-03-06T10:17:38.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:38.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:38.952514+0000) 2024-03-06T10:17:39.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:39.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:39.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:39.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:39.952711+0000) 2024-03-06T10:17:40.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:40.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:40.952956+0000) 2024-03-06T10:17:41.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:41.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:41.953218+0000) 2024-03-06T10:17:42.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:42.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:42.953482+0000) 2024-03-06T10:17:43.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:43.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:43.953756+0000) 2024-03-06T10:17:44.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:44.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:44.953994+0000) 2024-03-06T10:17:45.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:45.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:45.954247+0000) 2024-03-06T10:17:46.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:46.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:46.954513+0000) 2024-03-06T10:17:47.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:47.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:47.954742+0000) 2024-03-06T10:17:48.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:48.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:48.954984+0000) 2024-03-06T10:17:49.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:49.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:49.955272+0000) 2024-03-06T10:17:50.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:50.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:50.955454+0000) 2024-03-06T10:17:51.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:51.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:51.955744+0000) 2024-03-06T10:17:52.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:52.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:52.955996+0000) 2024-03-06T10:17:53.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:53.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:53.956236+0000) 2024-03-06T10:17:54.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:54.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:54.956561+0000) 2024-03-06T10:17:55.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:55.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:55.956791+0000) 2024-03-06T10:17:56.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:56.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:56.957061+0000) 2024-03-06T10:17:57.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:57.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:57.957291+0000) 2024-03-06T10:17:58.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:58.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:58.957548+0000) 2024-03-06T10:17:59.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:17:59.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:59.957782+0000) 2024-03-06T10:18:00.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:00.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:00.958040+0000) 2024-03-06T10:18:01.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:01.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:01.958241+0000) 2024-03-06T10:18:02.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:02.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:02.958491+0000) 2024-03-06T10:18:03.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:03.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:03.958749+0000) 2024-03-06T10:18:04.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:04.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:04.958993+0000) 2024-03-06T10:18:05.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:05.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:05.959234+0000) 2024-03-06T10:18:06.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:06.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:06.959486+0000) 2024-03-06T10:18:07.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:07.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:07.959752+0000) 2024-03-06T10:18:08.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:08.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:08.959998+0000) 2024-03-06T10:18:09.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:09.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:09.960248+0000) 2024-03-06T10:18:10.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:10.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:10.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:10.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:10.960504+0000) 2024-03-06T10:18:11.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:11.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:11.960746+0000) 2024-03-06T10:18:12.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:12.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:12.960998+0000) 2024-03-06T10:18:13.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:13.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:13.961245+0000) 2024-03-06T10:18:14.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:14.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:14.961497+0000) 2024-03-06T10:18:15.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:15.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:15.961761+0000) 2024-03-06T10:18:16.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:16.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:16.961957+0000) 2024-03-06T10:18:17.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:17.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:17.962188+0000) 2024-03-06T10:18:18.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:18.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:18.962424+0000) 2024-03-06T10:18:19.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:19.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:19.962706+0000) 2024-03-06T10:18:20.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:20.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:20.962958+0000) 2024-03-06T10:18:21.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:21.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:21.963201+0000) 2024-03-06T10:18:22.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:22.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:22.963441+0000) 2024-03-06T10:18:23.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:23.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:23.963753+0000) 2024-03-06T10:18:24.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:24.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:24.963947+0000) 2024-03-06T10:18:25.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:25.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:25.964203+0000) 2024-03-06T10:18:26.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:26.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:26.964474+0000) 2024-03-06T10:18:27.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:27.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:27.964758+0000) 2024-03-06T10:18:28.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:28.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:28.965023+0000) 2024-03-06T10:18:29.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:29.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:29.965282+0000) 2024-03-06T10:18:30.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:30.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:30.965539+0000) 2024-03-06T10:18:31.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:31.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:31.965816+0000) 2024-03-06T10:18:32.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:32.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:32.966086+0000) 2024-03-06T10:18:33.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:33.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:33.966360+0000) 2024-03-06T10:18:34.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:34.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:34.966614+0000) 2024-03-06T10:18:35.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:35.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:35.966896+0000) 2024-03-06T10:18:36.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:36.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:36.967188+0000) 2024-03-06T10:18:37.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:37.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:37.967463+0000) 2024-03-06T10:18:38.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:38.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:38.967760+0000) 2024-03-06T10:18:39.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:39.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:39.968014+0000) 2024-03-06T10:18:40.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:40.213 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:40.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:40.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:40.968277+0000) 2024-03-06T10:18:41.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:41.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:41.968506+0000) 2024-03-06T10:18:42.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:42.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:42.968809+0000) 2024-03-06T10:18:43.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:43.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:43.969064+0000) 2024-03-06T10:18:44.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:44.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:44.969332+0000) 2024-03-06T10:18:45.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:45.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:45.969592+0000) 2024-03-06T10:18:46.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:46.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:46.969902+0000) 2024-03-06T10:18:47.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:47.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:47.970143+0000) 2024-03-06T10:18:48.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:48.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:48.970405+0000) 2024-03-06T10:18:49.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:49.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:49.970722+0000) 2024-03-06T10:18:50.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:50.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:50.970959+0000) 2024-03-06T10:18:51.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:51.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:51.971145+0000) 2024-03-06T10:18:52.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:52.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:52.971397+0000) 2024-03-06T10:18:53.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:53.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:53.971687+0000) 2024-03-06T10:18:54.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:54.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:54.971939+0000) 2024-03-06T10:18:55.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:55.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:55.972193+0000) 2024-03-06T10:18:56.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:56.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:56.972463+0000) 2024-03-06T10:18:57.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:57.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:57.972737+0000) 2024-03-06T10:18:58.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:58.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:58.972929+0000) 2024-03-06T10:18:59.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:18:59.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:59.973182+0000) 2024-03-06T10:19:00.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:00.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:00.973438+0000) 2024-03-06T10:19:01.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:01.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:01.973724+0000) 2024-03-06T10:19:02.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:02.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:02.973980+0000) 2024-03-06T10:19:03.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:03.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:03.974225+0000) 2024-03-06T10:19:04.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:04.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:04.974467+0000) 2024-03-06T10:19:05.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:05.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:05.974756+0000) 2024-03-06T10:19:06.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:06.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:06.975004+0000) 2024-03-06T10:19:07.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:07.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:07.975198+0000) 2024-03-06T10:19:08.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:08.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:08.975440+0000) 2024-03-06T10:19:09.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:09.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:09.975742+0000) 2024-03-06T10:19:10.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:10.386 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:10.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:10.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:10.976000+0000) 2024-03-06T10:19:11.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:11.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:11.976236+0000) 2024-03-06T10:19:12.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:12.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:12.976479+0000) 2024-03-06T10:19:13.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:13.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:13.976740+0000) 2024-03-06T10:19:14.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:14.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:14.976934+0000) 2024-03-06T10:19:15.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:15.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:15.977183+0000) 2024-03-06T10:19:16.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:16.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:16.977446+0000) 2024-03-06T10:19:17.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:17.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:17.977753+0000) 2024-03-06T10:19:18.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:18.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:18.978015+0000) 2024-03-06T10:19:19.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:19.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:19.978276+0000) 2024-03-06T10:19:20.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:20.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:20.978539+0000) 2024-03-06T10:19:21.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:21.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:21.978778+0000) 2024-03-06T10:19:22.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:22.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:22.979034+0000) 2024-03-06T10:19:23.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:23.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:23.979291+0000) 2024-03-06T10:19:24.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:24.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:24.979549+0000) 2024-03-06T10:19:25.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:25.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:25.979788+0000) 2024-03-06T10:19:26.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:26.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:26.980055+0000) 2024-03-06T10:19:27.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:27.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:27.980296+0000) 2024-03-06T10:19:28.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:28.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:28.980558+0000) 2024-03-06T10:19:29.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:29.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:29.980811+0000) 2024-03-06T10:19:30.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:30.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:30.981052+0000) 2024-03-06T10:19:31.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:31.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:31.981303+0000) 2024-03-06T10:19:32.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:32.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:32.981536+0000) 2024-03-06T10:19:33.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:33.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:33.981781+0000) 2024-03-06T10:19:34.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:34.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:34.982027+0000) 2024-03-06T10:19:35.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:35.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:35.982275+0000) 2024-03-06T10:19:36.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:36.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:36.982523+0000) 2024-03-06T10:19:37.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:37.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:37.982777+0000) 2024-03-06T10:19:38.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:38.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:38.983040+0000) 2024-03-06T10:19:39.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:39.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:39.983303+0000) 2024-03-06T10:19:40.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:40.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:40.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:40.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:40.983562+0000) 2024-03-06T10:19:41.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:41.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:41.983813+0000) 2024-03-06T10:19:42.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:42.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:42.984068+0000) 2024-03-06T10:19:43.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:43.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:43.984323+0000) 2024-03-06T10:19:44.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:44.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:44.984564+0000) 2024-03-06T10:19:45.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:45.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:45.984826+0000) 2024-03-06T10:19:46.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:46.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:46.985081+0000) 2024-03-06T10:19:47.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:47.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:47.985332+0000) 2024-03-06T10:19:48.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:48.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:48.985599+0000) 2024-03-06T10:19:49.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:49.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:49.985891+0000) 2024-03-06T10:19:50.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:50.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:50.986149+0000) 2024-03-06T10:19:51.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:51.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:51.986419+0000) 2024-03-06T10:19:52.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:52.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:52.986722+0000) 2024-03-06T10:19:53.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:53.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:53.986976+0000) 2024-03-06T10:19:54.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:54.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:54.987219+0000) 2024-03-06T10:19:55.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:55.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:55.987440+0000) 2024-03-06T10:19:56.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:56.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:56.987721+0000) 2024-03-06T10:19:57.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:57.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:57.987914+0000) 2024-03-06T10:19:58.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:58.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:58.988173+0000) 2024-03-06T10:19:59.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:19:59.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:59.988425+0000) 2024-03-06T10:20:00.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:00.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:00.988730+0000) 2024-03-06T10:20:01.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:01.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:01.988965+0000) 2024-03-06T10:20:02.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:02.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:02.989208+0000) 2024-03-06T10:20:03.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:03.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:03.989456+0000) 2024-03-06T10:20:04.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:04.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:04.989747+0000) 2024-03-06T10:20:05.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:05.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:05.989946+0000) 2024-03-06T10:20:06.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:06.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:06.990190+0000) 2024-03-06T10:20:07.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:07.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:07.990435+0000) 2024-03-06T10:20:08.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:08.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:08.990733+0000) 2024-03-06T10:20:09.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:09.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:09.991011+0000) 2024-03-06T10:20:10.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:10.689 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:10.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:10.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:10.991297+0000) 2024-03-06T10:20:11.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:11.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:11.991557+0000) 2024-03-06T10:20:12.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:12.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:12.991814+0000) 2024-03-06T10:20:13.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:13.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:13.992091+0000) 2024-03-06T10:20:14.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:14.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:14.992361+0000) 2024-03-06T10:20:15.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:15.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:15.992660+0000) 2024-03-06T10:20:16.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:16.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:16.992925+0000) 2024-03-06T10:20:17.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:17.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:17.993162+0000) 2024-03-06T10:20:18.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:18.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:18.993420+0000) 2024-03-06T10:20:19.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:19.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:19.993692+0000) 2024-03-06T10:20:20.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:20.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:20.993947+0000) 2024-03-06T10:20:21.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:21.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:21.994199+0000) 2024-03-06T10:20:22.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:22.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:22.994448+0000) 2024-03-06T10:20:23.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:23.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:23.994799+0000) 2024-03-06T10:20:24.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:24.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:24.995053+0000) 2024-03-06T10:20:25.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:25.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:25.995308+0000) 2024-03-06T10:20:26.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:26.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:26.995546+0000) 2024-03-06T10:20:27.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:27.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:27.995820+0000) 2024-03-06T10:20:28.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:28.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:28.996032+0000) 2024-03-06T10:20:29.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:29.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:29.996328+0000) 2024-03-06T10:20:30.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:30.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:30.996590+0000) 2024-03-06T10:20:31.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:31.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:31.996905+0000) 2024-03-06T10:20:32.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:32.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:32.997154+0000) 2024-03-06T10:20:33.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:33.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:33.997432+0000) 2024-03-06T10:20:34.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:34.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:34.997755+0000) 2024-03-06T10:20:35.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:35.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:35.998032+0000) 2024-03-06T10:20:36.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:36.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:36.998297+0000) 2024-03-06T10:20:37.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:37.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:37.998563+0000) 2024-03-06T10:20:38.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:38.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:38.998842+0000) 2024-03-06T10:20:39.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:39.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:39.999127+0000) 2024-03-06T10:20:40.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:40.852 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:40.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:40.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:40.999340+0000) 2024-03-06T10:20:41.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:41.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:41.999589+0000) 2024-03-06T10:20:42.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:42.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:42.999908+0000) 2024-03-06T10:20:43.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:43.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:44.000165+0000) 2024-03-06T10:20:44.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:44.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:45.000443+0000) 2024-03-06T10:20:45.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:45.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:46.000861+0000) 2024-03-06T10:20:46.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:47.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:47.001197+0000) 2024-03-06T10:20:48.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:48.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:48.001510+0000) 2024-03-06T10:20:49.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:49.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:49.001797+0000) 2024-03-06T10:20:50.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:50.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:50.002175+0000) 2024-03-06T10:20:51.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:51.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:51.002463+0000) 2024-03-06T10:20:52.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:52.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:52.002817+0000) 2024-03-06T10:20:53.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:53.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:53.003027+0000) 2024-03-06T10:20:54.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:54.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:54.003237+0000) 2024-03-06T10:20:55.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:55.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:55.003493+0000) 2024-03-06T10:20:56.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:56.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:56.003749+0000) 2024-03-06T10:20:57.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:57.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:57.003998+0000) 2024-03-06T10:20:58.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:58.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:58.004237+0000) 2024-03-06T10:20:59.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:20:59.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:59.004489+0000) 2024-03-06T10:21:00.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:00.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:00.004743+0000) 2024-03-06T10:21:01.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:01.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:01.005001+0000) 2024-03-06T10:21:02.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:02.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:02.005239+0000) 2024-03-06T10:21:03.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:03.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:03.005498+0000) 2024-03-06T10:21:04.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:04.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:04.005755+0000) 2024-03-06T10:21:05.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:05.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:05.006023+0000) 2024-03-06T10:21:06.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:06.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:06.006276+0000) 2024-03-06T10:21:07.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:07.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:07.006545+0000) 2024-03-06T10:21:08.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:08.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:08.006760+0000) 2024-03-06T10:21:09.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:09.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:09.007024+0000) 2024-03-06T10:21:10.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:10.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:10.007262+0000) 2024-03-06T10:21:10.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:10.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:11.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:11.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:11.007490+0000) 2024-03-06T10:21:12.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:12.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:12.007811+0000) 2024-03-06T10:21:13.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:13.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:13.008053+0000) 2024-03-06T10:21:14.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:14.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:14.008321+0000) 2024-03-06T10:21:15.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:15.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:15.008531+0000) 2024-03-06T10:21:16.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:16.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:16.008768+0000) 2024-03-06T10:21:17.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:17.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:17.009021+0000) 2024-03-06T10:21:18.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:18.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:18.009288+0000) 2024-03-06T10:21:19.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:19.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:19.009554+0000) 2024-03-06T10:21:20.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:20.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:20.009808+0000) 2024-03-06T10:21:21.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:21.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:21.010066+0000) 2024-03-06T10:21:22.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:22.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:22.010329+0000) 2024-03-06T10:21:23.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:23.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:23.010586+0000) 2024-03-06T10:21:24.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:24.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:24.010861+0000) 2024-03-06T10:21:25.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:25.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:25.011121+0000) 2024-03-06T10:21:26.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:26.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:26.011357+0000) 2024-03-06T10:21:27.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:27.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:27.011604+0000) 2024-03-06T10:21:28.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:28.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:28.011884+0000) 2024-03-06T10:21:29.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:29.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:29.012147+0000) 2024-03-06T10:21:30.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:30.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:30.012409+0000) 2024-03-06T10:21:31.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:31.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:31.012695+0000) 2024-03-06T10:21:32.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:32.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:32.012963+0000) 2024-03-06T10:21:33.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:33.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:33.013228+0000) 2024-03-06T10:21:34.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:34.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:34.013470+0000) 2024-03-06T10:21:35.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:35.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:35.013758+0000) 2024-03-06T10:21:36.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:36.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:36.014002+0000) 2024-03-06T10:21:37.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:37.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:37.014197+0000) 2024-03-06T10:21:38.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:38.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:38.014439+0000) 2024-03-06T10:21:39.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:39.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:39.014752+0000) 2024-03-06T10:21:40.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:40.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:40.015024+0000) 2024-03-06T10:21:41.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:41.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:41.015282+0000) 2024-03-06T10:21:41.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:41.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:42.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:42.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:42.015554+0000) 2024-03-06T10:21:43.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:43.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:43.015817+0000) 2024-03-06T10:21:44.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:44.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:44.016020+0000) 2024-03-06T10:21:45.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:45.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:45.016272+0000) 2024-03-06T10:21:46.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:46.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:46.016530+0000) 2024-03-06T10:21:47.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:47.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:47.016797+0000) 2024-03-06T10:21:48.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:48.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:48.017035+0000) 2024-03-06T10:21:49.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:49.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:49.017292+0000) 2024-03-06T10:21:50.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:50.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:50.017538+0000) 2024-03-06T10:21:51.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:51.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:51.017778+0000) 2024-03-06T10:21:52.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:52.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:52.018044+0000) 2024-03-06T10:21:53.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:53.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:53.018293+0000) 2024-03-06T10:21:54.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:54.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:54.018550+0000) 2024-03-06T10:21:55.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:55.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:55.018797+0000) 2024-03-06T10:21:56.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:56.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:56.019049+0000) 2024-03-06T10:21:57.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:57.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:57.019310+0000) 2024-03-06T10:21:58.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:58.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:58.019567+0000) 2024-03-06T10:21:59.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:21:59.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:59.019814+0000) 2024-03-06T10:22:00.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:00.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:00.020142+0000) 2024-03-06T10:22:01.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:01.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:01.020400+0000) 2024-03-06T10:22:02.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:02.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:02.020685+0000) 2024-03-06T10:22:03.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:03.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:03.020912+0000) 2024-03-06T10:22:04.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:04.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:04.021113+0000) 2024-03-06T10:22:05.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:05.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:05.021370+0000) 2024-03-06T10:22:06.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:06.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:06.021604+0000) 2024-03-06T10:22:07.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:07.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:07.021902+0000) 2024-03-06T10:22:08.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:08.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:08.022156+0000) 2024-03-06T10:22:09.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:09.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:09.022427+0000) 2024-03-06T10:22:10.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:10.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:10.022715+0000) 2024-03-06T10:22:11.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:11.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:11.022977+0000) 2024-03-06T10:22:11.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:11.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:12.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:12.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:12.023163+0000) 2024-03-06T10:22:13.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:13.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:13.023349+0000) 2024-03-06T10:22:14.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:14.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:14.023602+0000) 2024-03-06T10:22:15.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:15.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:15.023886+0000) 2024-03-06T10:22:16.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:16.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:16.024140+0000) 2024-03-06T10:22:17.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:17.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:17.024409+0000) 2024-03-06T10:22:18.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:18.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:18.024688+0000) 2024-03-06T10:22:19.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:19.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:19.024949+0000) 2024-03-06T10:22:20.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:20.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:20.025212+0000) 2024-03-06T10:22:21.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:21.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:21.025452+0000) 2024-03-06T10:22:22.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:22.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:22.025739+0000) 2024-03-06T10:22:23.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:23.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:23.026001+0000) 2024-03-06T10:22:24.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:24.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:24.026235+0000) 2024-03-06T10:22:25.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:25.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:25.026473+0000) 2024-03-06T10:22:26.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:26.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:26.026757+0000) 2024-03-06T10:22:27.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:27.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:27.027012+0000) 2024-03-06T10:22:28.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:28.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:28.027270+0000) 2024-03-06T10:22:29.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:29.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:29.027525+0000) 2024-03-06T10:22:30.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:30.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:30.027783+0000) 2024-03-06T10:22:31.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:31.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:31.028047+0000) 2024-03-06T10:22:32.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:32.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:32.028303+0000) 2024-03-06T10:22:33.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:33.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:33.028546+0000) 2024-03-06T10:22:34.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:34.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:34.028788+0000) 2024-03-06T10:22:35.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:35.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:35.029048+0000) 2024-03-06T10:22:36.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:36.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:36.029300+0000) 2024-03-06T10:22:37.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:37.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:37.029570+0000) 2024-03-06T10:22:38.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:38.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:38.029809+0000) 2024-03-06T10:22:39.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:39.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:39.030048+0000) 2024-03-06T10:22:40.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:40.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:40.030303+0000) 2024-03-06T10:22:41.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:41.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:41.030496+0000) 2024-03-06T10:22:41.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:41.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:42.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:42.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:42.030754+0000) 2024-03-06T10:22:43.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:43.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:43.031011+0000) 2024-03-06T10:22:44.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:44.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:44.031271+0000) 2024-03-06T10:22:45.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:45.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:45.031512+0000) 2024-03-06T10:22:46.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:46.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:46.031749+0000) 2024-03-06T10:22:47.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:47.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:47.031998+0000) 2024-03-06T10:22:48.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:48.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:48.032254+0000) 2024-03-06T10:22:49.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:49.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:49.032497+0000) 2024-03-06T10:22:50.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:50.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:50.032754+0000) 2024-03-06T10:22:51.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:51.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:51.033010+0000) 2024-03-06T10:22:52.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:52.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:52.033203+0000) 2024-03-06T10:22:53.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:53.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:53.033445+0000) 2024-03-06T10:22:54.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:54.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:54.033745+0000) 2024-03-06T10:22:55.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:55.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:55.033986+0000) 2024-03-06T10:22:56.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:56.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:56.034216+0000) 2024-03-06T10:22:57.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:57.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:57.034411+0000) 2024-03-06T10:22:58.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:58.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:58.034733+0000) 2024-03-06T10:22:59.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:22:59.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:59.035039+0000) 2024-03-06T10:23:00.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:00.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:00.035302+0000) 2024-03-06T10:23:01.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:01.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:01.035614+0000) 2024-03-06T10:23:02.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:02.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:02.035928+0000) 2024-03-06T10:23:03.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:03.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:03.036233+0000) 2024-03-06T10:23:04.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:04.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:04.036515+0000) 2024-03-06T10:23:05.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:05.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:05.036811+0000) 2024-03-06T10:23:06.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:06.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:06.037124+0000) 2024-03-06T10:23:07.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:07.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:07.037442+0000) 2024-03-06T10:23:08.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:08.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:08.037774+0000) 2024-03-06T10:23:09.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:09.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:09.038089+0000) 2024-03-06T10:23:10.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:10.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:10.038392+0000) 2024-03-06T10:23:11.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:11.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:11.038754+0000) 2024-03-06T10:23:11.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:11.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:12.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:12.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:12.039068+0000) 2024-03-06T10:23:13.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:13.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:13.039346+0000) 2024-03-06T10:23:14.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:14.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:14.039602+0000) 2024-03-06T10:23:15.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:15.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:15.039901+0000) 2024-03-06T10:23:16.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:16.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:16.040199+0000) 2024-03-06T10:23:17.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:17.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:17.040522+0000) 2024-03-06T10:23:18.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:18.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:18.040828+0000) 2024-03-06T10:23:19.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:19.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:19.041117+0000) 2024-03-06T10:23:20.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:20.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:20.041390+0000) 2024-03-06T10:23:21.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:21.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:21.041683+0000) 2024-03-06T10:23:22.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:22.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:22.041987+0000) 2024-03-06T10:23:23.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:23.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:23.042302+0000) 2024-03-06T10:23:24.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:24.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:24.042607+0000) 2024-03-06T10:23:25.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:25.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:25.042938+0000) 2024-03-06T10:23:26.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:26.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:26.043227+0000) 2024-03-06T10:23:27.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:27.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:27.043551+0000) 2024-03-06T10:23:28.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:28.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:28.043855+0000) 2024-03-06T10:23:29.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:29.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:29.044156+0000) 2024-03-06T10:23:30.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:30.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:30.044464+0000) 2024-03-06T10:23:31.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:31.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:31.044763+0000) 2024-03-06T10:23:32.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:32.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:32.045051+0000) 2024-03-06T10:23:33.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:33.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:33.045329+0000) 2024-03-06T10:23:34.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:34.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:34.045614+0000) 2024-03-06T10:23:35.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:35.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:35.045965+0000) 2024-03-06T10:23:36.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:36.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:36.046258+0000) 2024-03-06T10:23:37.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:37.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:37.046553+0000) 2024-03-06T10:23:38.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:38.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:38.046864+0000) 2024-03-06T10:23:39.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:39.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:39.047176+0000) 2024-03-06T10:23:40.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:40.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:40.047467+0000) 2024-03-06T10:23:41.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:41.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:41.047789+0000) 2024-03-06T10:23:41.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:41.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:42.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:42.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:42.048118+0000) 2024-03-06T10:23:43.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:43.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:43.048428+0000) 2024-03-06T10:23:44.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:44.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:44.048761+0000) 2024-03-06T10:23:45.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:45.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:45.048995+0000) 2024-03-06T10:23:46.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:46.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:46.049240+0000) 2024-03-06T10:23:47.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:47.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:47.049551+0000) 2024-03-06T10:23:48.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:48.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:48.049790+0000) 2024-03-06T10:23:49.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:49.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:49.050055+0000) 2024-03-06T10:23:50.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:50.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:50.050317+0000) 2024-03-06T10:23:51.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:51.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:51.050575+0000) 2024-03-06T10:23:52.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:52.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:52.050810+0000) 2024-03-06T10:23:53.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:53.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:53.051085+0000) 2024-03-06T10:23:54.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:54.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:54.051320+0000) 2024-03-06T10:23:55.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:55.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:55.051580+0000) 2024-03-06T10:23:56.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:56.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:56.051836+0000) 2024-03-06T10:23:57.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:57.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:57.052100+0000) 2024-03-06T10:23:58.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:58.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:58.052343+0000) 2024-03-06T10:23:59.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:23:59.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:59.052598+0000) 2024-03-06T10:24:00.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:00.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:00.052853+0000) 2024-03-06T10:24:01.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:01.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:01.053116+0000) 2024-03-06T10:24:02.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:02.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:02.053371+0000) 2024-03-06T10:24:03.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:03.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:03.053673+0000) 2024-03-06T10:24:04.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:04.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:04.053935+0000) 2024-03-06T10:24:05.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:05.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:05.054132+0000) 2024-03-06T10:24:06.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:06.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:06.054357+0000) 2024-03-06T10:24:07.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:07.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:07.054608+0000) 2024-03-06T10:24:08.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:08.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:08.054820+0000) 2024-03-06T10:24:09.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:09.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:09.055076+0000) 2024-03-06T10:24:10.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:10.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:10.055330+0000) 2024-03-06T10:24:11.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:11.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:11.055556+0000) 2024-03-06T10:24:11.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:11.919 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:12.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:12.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:12.055778+0000) 2024-03-06T10:24:13.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:13.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:13.056042+0000) 2024-03-06T10:24:14.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:14.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:14.056298+0000) 2024-03-06T10:24:15.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:15.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:15.056550+0000) 2024-03-06T10:24:16.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:16.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:16.056801+0000) 2024-03-06T10:24:17.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:17.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:17.057066+0000) 2024-03-06T10:24:18.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:18.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:18.057320+0000) 2024-03-06T10:24:19.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:19.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:19.057587+0000) 2024-03-06T10:24:20.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:20.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:20.057845+0000) 2024-03-06T10:24:21.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:21.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:21.058099+0000) 2024-03-06T10:24:22.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:22.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:22.058353+0000) 2024-03-06T10:24:23.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:23.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:23.058588+0000) 2024-03-06T10:24:24.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:24.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:24.058830+0000) 2024-03-06T10:24:25.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:25.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:25.059078+0000) 2024-03-06T10:24:26.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:26.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:26.059313+0000) 2024-03-06T10:24:27.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:27.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:27.059581+0000) 2024-03-06T10:24:28.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:28.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:28.059790+0000) 2024-03-06T10:24:29.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:29.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:29.060046+0000) 2024-03-06T10:24:30.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:30.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:30.060310+0000) 2024-03-06T10:24:31.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:31.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:31.060550+0000) 2024-03-06T10:24:32.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:32.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:32.060789+0000) 2024-03-06T10:24:33.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:33.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:33.061025+0000) 2024-03-06T10:24:34.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:34.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:34.061268+0000) 2024-03-06T10:24:35.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:35.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:35.061515+0000) 2024-03-06T10:24:36.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:36.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:36.061752+0000) 2024-03-06T10:24:37.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:37.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:37.062018+0000) 2024-03-06T10:24:38.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:38.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:38.062275+0000) 2024-03-06T10:24:39.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:39.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:39.062520+0000) 2024-03-06T10:24:40.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:40.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:40.062781+0000) 2024-03-06T10:24:41.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:41.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:41.063020+0000) 2024-03-06T10:24:42.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:42.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:42.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:42.063258+0000) 2024-03-06T10:24:42.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:43.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:43.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:43.063501+0000) 2024-03-06T10:24:44.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:44.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:44.063754+0000) 2024-03-06T10:24:45.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:45.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:45.064013+0000) 2024-03-06T10:24:46.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:46.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:46.064263+0000) 2024-03-06T10:24:47.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:47.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:47.064530+0000) 2024-03-06T10:24:48.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:48.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:48.064790+0000) 2024-03-06T10:24:49.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:49.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:49.065051+0000) 2024-03-06T10:24:50.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:50.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:50.065318+0000) 2024-03-06T10:24:51.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:51.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:51.065581+0000) 2024-03-06T10:24:52.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:52.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:52.065839+0000) 2024-03-06T10:24:53.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:53.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:53.066026+0000) 2024-03-06T10:24:54.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:54.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:54.066297+0000) 2024-03-06T10:24:55.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:55.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:55.066538+0000) 2024-03-06T10:24:56.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:56.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:56.066801+0000) 2024-03-06T10:24:57.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:57.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:57.066993+0000) 2024-03-06T10:24:58.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:58.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:58.067246+0000) 2024-03-06T10:24:59.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:24:59.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:59.067489+0000) 2024-03-06T10:25:00.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:00.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:00.067734+0000) 2024-03-06T10:25:01.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:01.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:01.067998+0000) 2024-03-06T10:25:02.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:02.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:02.068263+0000) 2024-03-06T10:25:03.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:03.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:03.068497+0000) 2024-03-06T10:25:04.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:04.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:04.068762+0000) 2024-03-06T10:25:05.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:05.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:05.069026+0000) 2024-03-06T10:25:06.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:06.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:06.069285+0000) 2024-03-06T10:25:07.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:07.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:07.069555+0000) 2024-03-06T10:25:08.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:08.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:08.069811+0000) 2024-03-06T10:25:09.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:09.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:09.070052+0000) 2024-03-06T10:25:10.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:10.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:10.070288+0000) 2024-03-06T10:25:11.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:11.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:11.070529+0000) 2024-03-06T10:25:12.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:12.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:12.070772+0000) 2024-03-06T10:25:12.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:12.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:13.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:13.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:13.070986+0000) 2024-03-06T10:25:14.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:14.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:14.071185+0000) 2024-03-06T10:25:15.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:15.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:15.071389+0000) 2024-03-06T10:25:16.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:16.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:16.071667+0000) 2024-03-06T10:25:17.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:17.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:17.071930+0000) 2024-03-06T10:25:18.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:18.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:18.072165+0000) 2024-03-06T10:25:19.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:19.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:19.072413+0000) 2024-03-06T10:25:20.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:20.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:20.072739+0000) 2024-03-06T10:25:21.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:21.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:21.073011+0000) 2024-03-06T10:25:22.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:22.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:22.073261+0000) 2024-03-06T10:25:23.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:23.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:23.073568+0000) 2024-03-06T10:25:24.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:24.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:24.073845+0000) 2024-03-06T10:25:25.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:25.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:25.074125+0000) 2024-03-06T10:25:26.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:26.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:26.074396+0000) 2024-03-06T10:25:27.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:27.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:27.074729+0000) 2024-03-06T10:25:28.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:28.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:28.074981+0000) 2024-03-06T10:25:29.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:29.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:29.075189+0000) 2024-03-06T10:25:30.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:30.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:30.075473+0000) 2024-03-06T10:25:31.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:31.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:31.075753+0000) 2024-03-06T10:25:32.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:32.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:32.076003+0000) 2024-03-06T10:25:33.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:33.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:33.076269+0000) 2024-03-06T10:25:34.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:34.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:34.076551+0000) 2024-03-06T10:25:35.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:35.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:35.076820+0000) 2024-03-06T10:25:36.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:36.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:36.077092+0000) 2024-03-06T10:25:37.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:37.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:37.077379+0000) 2024-03-06T10:25:38.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:38.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:38.077659+0000) 2024-03-06T10:25:39.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:39.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:39.077920+0000) 2024-03-06T10:25:40.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:40.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:40.078173+0000) 2024-03-06T10:25:41.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:41.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:41.078419+0000) 2024-03-06T10:25:42.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:42.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:42.078690+0000) 2024-03-06T10:25:42.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:42.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:43.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:43.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:43.078936+0000) 2024-03-06T10:25:44.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:44.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:44.079186+0000) 2024-03-06T10:25:45.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:45.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:45.079444+0000) 2024-03-06T10:25:46.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:46.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:46.079722+0000) 2024-03-06T10:25:47.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:47.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:47.079972+0000) 2024-03-06T10:25:48.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:48.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:48.080225+0000) 2024-03-06T10:25:49.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:49.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:49.080488+0000) 2024-03-06T10:25:50.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:50.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:50.080746+0000) 2024-03-06T10:25:51.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:51.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:51.081001+0000) 2024-03-06T10:25:52.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:52.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:52.081255+0000) 2024-03-06T10:25:53.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:53.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:53.081434+0000) 2024-03-06T10:25:54.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:54.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:54.081734+0000) 2024-03-06T10:25:55.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:55.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:55.082007+0000) 2024-03-06T10:25:56.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:56.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:56.082266+0000) 2024-03-06T10:25:57.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:57.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:57.082512+0000) 2024-03-06T10:25:58.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:58.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:58.082762+0000) 2024-03-06T10:25:59.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:25:59.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:59.083006+0000) 2024-03-06T10:26:00.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:00.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:00.083269+0000) 2024-03-06T10:26:01.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:01.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:01.083523+0000) 2024-03-06T10:26:02.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:02.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:02.083786+0000) 2024-03-06T10:26:03.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:03.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:03.084035+0000) 2024-03-06T10:26:04.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:04.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:04.084288+0000) 2024-03-06T10:26:05.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:05.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:05.084540+0000) 2024-03-06T10:26:06.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:06.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:06.084801+0000) 2024-03-06T10:26:07.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:07.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:07.085001+0000) 2024-03-06T10:26:08.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:08.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:08.085254+0000) 2024-03-06T10:26:09.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:09.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:09.085506+0000) 2024-03-06T10:26:10.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:10.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:10.085779+0000) 2024-03-06T10:26:11.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:11.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:11.086073+0000) 2024-03-06T10:26:12.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:12.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:12.086309+0000) 2024-03-06T10:26:12.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:12.520 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:13.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:13.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:13.086560+0000) 2024-03-06T10:26:14.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:14.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:14.086824+0000) 2024-03-06T10:26:15.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:15.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:15.087082+0000) 2024-03-06T10:26:16.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:16.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:16.087341+0000) 2024-03-06T10:26:17.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:17.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:17.087615+0000) 2024-03-06T10:26:18.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:18.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:18.087906+0000) 2024-03-06T10:26:19.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:19.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:19.088150+0000) 2024-03-06T10:26:20.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:20.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:20.088406+0000) 2024-03-06T10:26:21.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:21.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:21.088710+0000) 2024-03-06T10:26:22.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:22.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:22.088898+0000) 2024-03-06T10:26:23.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:23.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:23.089150+0000) 2024-03-06T10:26:24.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:24.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:24.089341+0000) 2024-03-06T10:26:25.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:25.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:25.089591+0000) 2024-03-06T10:26:26.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:26.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:26.089849+0000) 2024-03-06T10:26:27.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:27.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:27.090118+0000) 2024-03-06T10:26:28.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:28.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:28.090370+0000) 2024-03-06T10:26:29.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:29.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:29.090666+0000) 2024-03-06T10:26:30.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:30.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:30.090928+0000) 2024-03-06T10:26:31.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:31.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:31.091171+0000) 2024-03-06T10:26:32.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:32.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:32.091429+0000) 2024-03-06T10:26:33.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:33.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:33.091713+0000) 2024-03-06T10:26:34.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:34.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:34.091903+0000) 2024-03-06T10:26:35.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:35.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:35.092159+0000) 2024-03-06T10:26:36.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:36.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:36.092419+0000) 2024-03-06T10:26:37.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:37.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:37.092768+0000) 2024-03-06T10:26:38.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:38.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:38.093038+0000) 2024-03-06T10:26:39.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:39.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:39.093292+0000) 2024-03-06T10:26:40.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:40.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:40.093529+0000) 2024-03-06T10:26:41.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:41.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:41.093765+0000) 2024-03-06T10:26:42.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:42.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:42.094025+0000) 2024-03-06T10:26:42.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:42.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:43.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:43.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:43.094270+0000) 2024-03-06T10:26:44.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:44.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:44.094512+0000) 2024-03-06T10:26:45.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:45.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:45.094743+0000) 2024-03-06T10:26:46.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:46.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:46.094997+0000) 2024-03-06T10:26:47.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:47.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:47.095262+0000) 2024-03-06T10:26:48.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:48.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:48.095498+0000) 2024-03-06T10:26:49.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:49.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:49.095764+0000) 2024-03-06T10:26:50.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:50.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:50.096018+0000) 2024-03-06T10:26:51.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:51.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:51.096251+0000) 2024-03-06T10:26:52.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:52.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:52.096509+0000) 2024-03-06T10:26:53.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:53.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:53.096768+0000) 2024-03-06T10:26:54.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:54.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:54.097036+0000) 2024-03-06T10:26:55.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:55.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:55.097276+0000) 2024-03-06T10:26:56.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:56.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:56.097513+0000) 2024-03-06T10:26:57.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:57.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:57.097778+0000) 2024-03-06T10:26:58.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:58.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:58.098029+0000) 2024-03-06T10:26:59.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:26:59.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:59.098303+0000) 2024-03-06T10:27:00.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:00.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:00.098564+0000) 2024-03-06T10:27:01.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:01.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:01.098823+0000) 2024-03-06T10:27:02.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:02.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:02.099058+0000) 2024-03-06T10:27:03.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:03.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:03.099286+0000) 2024-03-06T10:27:04.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:04.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:04.099527+0000) 2024-03-06T10:27:05.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:05.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:05.099786+0000) 2024-03-06T10:27:06.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:06.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:06.100040+0000) 2024-03-06T10:27:07.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:07.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:07.100288+0000) 2024-03-06T10:27:08.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:08.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:08.100540+0000) 2024-03-06T10:27:09.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:09.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:09.100782+0000) 2024-03-06T10:27:10.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:10.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:10.101070+0000) 2024-03-06T10:27:11.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:11.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:11.101319+0000) 2024-03-06T10:27:12.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:12.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:12.101564+0000) 2024-03-06T10:27:12.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:12.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:13.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:13.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:13.101825+0000) 2024-03-06T10:27:14.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:14.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:14.102070+0000) 2024-03-06T10:27:15.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:15.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:15.102315+0000) 2024-03-06T10:27:16.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:16.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:16.102553+0000) 2024-03-06T10:27:17.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:17.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:17.102823+0000) 2024-03-06T10:27:18.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:18.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:18.103083+0000) 2024-03-06T10:27:19.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:19.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:19.103326+0000) 2024-03-06T10:27:20.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:20.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:20.103595+0000) 2024-03-06T10:27:21.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:21.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:21.103852+0000) 2024-03-06T10:27:22.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:22.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:22.104097+0000) 2024-03-06T10:27:23.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:23.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:23.104349+0000) 2024-03-06T10:27:24.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:24.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:24.104607+0000) 2024-03-06T10:27:25.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:25.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:25.104838+0000) 2024-03-06T10:27:26.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:26.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:26.105096+0000) 2024-03-06T10:27:27.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:27.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:27.105333+0000) 2024-03-06T10:27:28.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:28.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:28.105582+0000) 2024-03-06T10:27:29.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:29.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:29.105810+0000) 2024-03-06T10:27:30.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:30.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:30.106071+0000) 2024-03-06T10:27:31.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:31.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:31.106307+0000) 2024-03-06T10:27:32.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:32.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:32.106497+0000) 2024-03-06T10:27:33.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:33.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:33.106737+0000) 2024-03-06T10:27:34.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:34.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:34.106971+0000) 2024-03-06T10:27:35.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:35.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:35.107209+0000) 2024-03-06T10:27:36.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:36.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:36.107454+0000) 2024-03-06T10:27:37.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:37.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:37.107754+0000) 2024-03-06T10:27:38.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:38.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:38.107990+0000) 2024-03-06T10:27:39.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:39.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:39.108210+0000) 2024-03-06T10:27:40.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:40.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:40.108466+0000) 2024-03-06T10:27:41.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:41.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:41.108735+0000) 2024-03-06T10:27:42.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:42.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:42.108999+0000) 2024-03-06T10:27:42.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:42.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:43.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:43.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:43.109270+0000) 2024-03-06T10:27:44.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:44.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:44.109550+0000) 2024-03-06T10:27:45.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:45.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:45.109801+0000) 2024-03-06T10:27:46.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:46.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:46.110078+0000) 2024-03-06T10:27:47.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:47.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:47.110345+0000) 2024-03-06T10:27:48.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:48.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:48.110616+0000) 2024-03-06T10:27:49.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:49.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:49.110923+0000) 2024-03-06T10:27:50.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:50.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:50.111203+0000) 2024-03-06T10:27:51.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:51.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:51.111470+0000) 2024-03-06T10:27:52.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:52.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:52.111752+0000) 2024-03-06T10:27:53.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:53.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:53.112022+0000) 2024-03-06T10:27:54.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:54.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:54.112289+0000) 2024-03-06T10:27:55.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:55.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:55.112555+0000) 2024-03-06T10:27:56.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:56.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:56.112833+0000) 2024-03-06T10:27:57.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:57.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:57.113142+0000) 2024-03-06T10:27:58.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:58.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:58.113463+0000) 2024-03-06T10:27:59.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:27:59.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:59.113739+0000) 2024-03-06T10:28:00.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:00.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:00.113950+0000) 2024-03-06T10:28:01.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:01.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:01.114200+0000) 2024-03-06T10:28:02.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:02.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:02.114478+0000) 2024-03-06T10:28:03.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:03.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:03.114751+0000) 2024-03-06T10:28:04.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:04.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:04.115025+0000) 2024-03-06T10:28:05.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:05.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:05.115280+0000) 2024-03-06T10:28:06.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:06.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:06.115549+0000) 2024-03-06T10:28:07.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:07.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:07.115811+0000) 2024-03-06T10:28:08.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:08.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:08.116055+0000) 2024-03-06T10:28:09.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:09.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:09.116299+0000) 2024-03-06T10:28:10.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:10.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:10.116555+0000) 2024-03-06T10:28:11.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:11.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:11.116794+0000) 2024-03-06T10:28:12.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:12.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:12.116983+0000) 2024-03-06T10:28:13.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:13.094 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:13.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:13.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:13.117181+0000) 2024-03-06T10:28:14.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:14.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:14.117430+0000) 2024-03-06T10:28:15.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:15.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:15.117739+0000) 2024-03-06T10:28:16.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:16.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:16.118013+0000) 2024-03-06T10:28:17.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:17.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:17.118282+0000) 2024-03-06T10:28:18.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:18.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:18.118555+0000) 2024-03-06T10:28:19.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:19.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:19.118833+0000) 2024-03-06T10:28:20.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:20.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:20.119093+0000) 2024-03-06T10:28:21.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:21.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:21.119368+0000) 2024-03-06T10:28:22.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:22.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:22.119682+0000) 2024-03-06T10:28:23.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:23.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:23.119935+0000) 2024-03-06T10:28:24.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:24.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:24.120222+0000) 2024-03-06T10:28:25.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:25.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:25.120480+0000) 2024-03-06T10:28:26.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:26.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:26.120740+0000) 2024-03-06T10:28:27.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:27.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:27.121000+0000) 2024-03-06T10:28:28.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:28.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:28.121212+0000) 2024-03-06T10:28:29.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:29.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:29.121478+0000) 2024-03-06T10:28:30.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:30.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:30.121723+0000) 2024-03-06T10:28:31.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:31.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:31.121996+0000) 2024-03-06T10:28:32.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:32.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:32.122274+0000) 2024-03-06T10:28:33.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:33.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:33.122527+0000) 2024-03-06T10:28:34.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:34.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:34.122800+0000) 2024-03-06T10:28:35.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:35.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:35.123068+0000) 2024-03-06T10:28:36.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:36.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:36.123325+0000) 2024-03-06T10:28:37.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:37.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:37.123596+0000) 2024-03-06T10:28:38.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:38.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:38.123912+0000) 2024-03-06T10:28:39.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:39.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:39.124181+0000) 2024-03-06T10:28:40.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:40.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:40.124441+0000) 2024-03-06T10:28:41.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:41.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:41.124722+0000) 2024-03-06T10:28:42.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:42.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:42.124994+0000) 2024-03-06T10:28:43.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:43.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:43.125243+0000) 2024-03-06T10:28:43.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:43.240 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:44.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:44.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:44.125519+0000) 2024-03-06T10:28:45.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:45.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:45.125799+0000) 2024-03-06T10:28:46.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:46.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:46.126131+0000) 2024-03-06T10:28:47.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:47.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:47.126417+0000) 2024-03-06T10:28:48.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:48.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:48.126730+0000) 2024-03-06T10:28:49.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:49.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:49.127006+0000) 2024-03-06T10:28:50.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:50.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:50.127282+0000) 2024-03-06T10:28:51.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:51.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:51.127555+0000) 2024-03-06T10:28:52.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:52.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:52.127820+0000) 2024-03-06T10:28:53.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:53.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:53.128087+0000) 2024-03-06T10:28:54.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:54.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:54.128371+0000) 2024-03-06T10:28:55.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:55.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:55.128675+0000) 2024-03-06T10:28:56.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:56.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:56.128963+0000) 2024-03-06T10:28:57.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:57.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:57.129238+0000) 2024-03-06T10:28:58.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:58.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:58.129522+0000) 2024-03-06T10:28:59.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:28:59.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:59.129789+0000) 2024-03-06T10:29:00.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:00.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:00.130051+0000) 2024-03-06T10:29:01.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:01.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:01.130326+0000) 2024-03-06T10:29:02.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:02.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:02.130587+0000) 2024-03-06T10:29:03.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:03.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:03.130888+0000) 2024-03-06T10:29:04.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:04.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:04.131159+0000) 2024-03-06T10:29:05.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:05.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:05.131418+0000) 2024-03-06T10:29:06.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:06.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:06.131744+0000) 2024-03-06T10:29:07.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:07.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:07.132017+0000) 2024-03-06T10:29:08.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:08.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:08.132271+0000) 2024-03-06T10:29:09.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:09.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:09.132525+0000) 2024-03-06T10:29:10.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:10.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:10.132780+0000) 2024-03-06T10:29:11.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:11.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:11.132975+0000) 2024-03-06T10:29:12.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:12.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:12.133187+0000) 2024-03-06T10:29:13.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:13.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:13.133383+0000) 2024-03-06T10:29:13.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:13.386 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:14.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:14.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:14.133685+0000) 2024-03-06T10:29:15.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:15.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:15.133918+0000) 2024-03-06T10:29:16.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:16.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:16.134171+0000) 2024-03-06T10:29:17.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:17.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:17.134434+0000) 2024-03-06T10:29:18.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:18.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:18.134710+0000) 2024-03-06T10:29:19.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:19.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:19.134950+0000) 2024-03-06T10:29:20.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:20.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:20.135212+0000) 2024-03-06T10:29:21.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:21.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:21.135464+0000) 2024-03-06T10:29:22.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:22.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:22.135750+0000) 2024-03-06T10:29:23.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:23.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:23.136004+0000) 2024-03-06T10:29:24.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:24.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:24.136262+0000) 2024-03-06T10:29:25.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:25.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:25.136493+0000) 2024-03-06T10:29:26.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:26.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:26.136753+0000) 2024-03-06T10:29:27.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:27.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:27.137018+0000) 2024-03-06T10:29:28.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:28.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:28.137320+0000) 2024-03-06T10:29:29.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:29.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:29.137506+0000) 2024-03-06T10:29:30.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:30.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:30.137756+0000) 2024-03-06T10:29:31.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:31.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:31.137991+0000) 2024-03-06T10:29:32.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:32.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:32.138226+0000) 2024-03-06T10:29:33.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:33.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:33.138473+0000) 2024-03-06T10:29:34.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:34.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:34.138753+0000) 2024-03-06T10:29:35.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:35.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:35.139014+0000) 2024-03-06T10:29:36.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:36.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:36.139274+0000) 2024-03-06T10:29:37.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:37.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:37.139518+0000) 2024-03-06T10:29:38.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:38.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:38.139768+0000) 2024-03-06T10:29:39.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:39.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:39.140003+0000) 2024-03-06T10:29:40.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:40.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:40.140197+0000) 2024-03-06T10:29:41.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:41.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:41.140430+0000) 2024-03-06T10:29:42.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:42.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:42.140727+0000) 2024-03-06T10:29:43.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:43.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:43.140977+0000) 2024-03-06T10:29:43.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:43.532 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:44.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:44.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:44.141234+0000) 2024-03-06T10:29:45.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:45.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:45.141490+0000) 2024-03-06T10:29:46.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:46.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:46.141753+0000) 2024-03-06T10:29:47.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:47.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:47.142018+0000) 2024-03-06T10:29:48.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:48.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:48.142266+0000) 2024-03-06T10:29:49.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:49.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:49.142497+0000) 2024-03-06T10:29:50.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:50.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:50.142765+0000) 2024-03-06T10:29:51.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:51.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:51.143001+0000) 2024-03-06T10:29:52.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:52.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:52.143245+0000) 2024-03-06T10:29:53.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:53.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:53.143477+0000) 2024-03-06T10:29:54.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:54.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:54.143749+0000) 2024-03-06T10:29:55.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:55.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:55.144002+0000) 2024-03-06T10:29:56.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:56.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:56.144245+0000) 2024-03-06T10:29:57.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:57.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:57.144504+0000) 2024-03-06T10:29:58.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:58.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:58.144758+0000) 2024-03-06T10:29:59.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:29:59.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:59.145014+0000) 2024-03-06T10:30:00.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:00.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:00.145276+0000) 2024-03-06T10:30:01.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:01.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:01.145512+0000) 2024-03-06T10:30:02.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:02.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:02.145751+0000) 2024-03-06T10:30:03.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:03.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:03.145990+0000) 2024-03-06T10:30:04.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:04.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:04.146238+0000) 2024-03-06T10:30:05.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:05.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:05.146571+0000) 2024-03-06T10:30:06.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:06.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:06.146834+0000) 2024-03-06T10:30:07.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:07.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:07.147067+0000) 2024-03-06T10:30:08.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:08.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:08.147322+0000) 2024-03-06T10:30:09.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:09.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:09.147574+0000) 2024-03-06T10:30:10.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:10.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:10.147837+0000) 2024-03-06T10:30:11.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:11.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:11.148095+0000) 2024-03-06T10:30:12.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:12.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:12.148273+0000) 2024-03-06T10:30:13.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:13.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:13.148528+0000) 2024-03-06T10:30:13.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:13.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:14.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:14.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:14.148795+0000) 2024-03-06T10:30:15.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:15.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:15.149033+0000) 2024-03-06T10:30:16.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:16.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:16.149224+0000) 2024-03-06T10:30:17.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:17.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:17.149476+0000) 2024-03-06T10:30:18.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:18.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:18.149745+0000) 2024-03-06T10:30:19.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:19.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:19.149986+0000) 2024-03-06T10:30:20.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:20.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:20.150245+0000) 2024-03-06T10:30:21.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:21.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:21.150500+0000) 2024-03-06T10:30:22.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:22.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:22.150753+0000) 2024-03-06T10:30:23.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:23.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:23.151002+0000) 2024-03-06T10:30:24.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:24.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:24.151239+0000) 2024-03-06T10:30:25.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:25.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:25.151488+0000) 2024-03-06T10:30:26.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:26.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:26.151769+0000) 2024-03-06T10:30:27.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:27.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:27.152037+0000) 2024-03-06T10:30:28.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:28.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:28.152285+0000) 2024-03-06T10:30:29.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:29.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:29.152514+0000) 2024-03-06T10:30:30.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:30.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:30.152781+0000) 2024-03-06T10:30:31.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:31.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:31.153011+0000) 2024-03-06T10:30:32.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:32.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:32.153249+0000) 2024-03-06T10:30:33.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:33.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:33.153489+0000) 2024-03-06T10:30:34.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:34.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:34.153711+0000) 2024-03-06T10:30:35.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:35.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:35.153906+0000) 2024-03-06T10:30:36.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:36.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:36.154166+0000) 2024-03-06T10:30:37.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:37.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:37.154432+0000) 2024-03-06T10:30:38.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:38.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:38.154665+0000) 2024-03-06T10:30:39.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:39.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:39.154926+0000) 2024-03-06T10:30:40.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:40.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:40.155164+0000) 2024-03-06T10:30:41.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:41.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:41.155414+0000) 2024-03-06T10:30:42.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:42.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:42.155708+0000) 2024-03-06T10:30:43.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:43.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:43.155904+0000) 2024-03-06T10:30:43.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:43.844 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:44.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:44.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:44.156160+0000) 2024-03-06T10:30:45.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:45.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:45.156396+0000) 2024-03-06T10:30:46.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:46.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:46.156669+0000) 2024-03-06T10:30:47.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:47.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:47.156940+0000) 2024-03-06T10:30:48.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:48.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:48.157179+0000) 2024-03-06T10:30:49.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:49.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:49.157364+0000) 2024-03-06T10:30:50.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:50.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:50.157656+0000) 2024-03-06T10:30:51.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:51.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:51.157836+0000) 2024-03-06T10:30:52.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:52.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:52.158099+0000) 2024-03-06T10:30:53.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:53.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:53.158360+0000) 2024-03-06T10:30:54.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:54.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:54.158576+0000) 2024-03-06T10:30:55.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:55.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:55.158832+0000) 2024-03-06T10:30:56.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:56.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:56.159087+0000) 2024-03-06T10:30:57.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:57.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:57.159348+0000) 2024-03-06T10:30:58.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:58.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:58.159603+0000) 2024-03-06T10:30:59.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:30:59.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:59.159907+0000) 2024-03-06T10:31:00.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:00.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:00.160177+0000) 2024-03-06T10:31:01.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:01.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:01.160411+0000) 2024-03-06T10:31:02.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:02.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:02.160704+0000) 2024-03-06T10:31:03.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:03.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:03.160946+0000) 2024-03-06T10:31:04.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:04.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:04.161190+0000) 2024-03-06T10:31:05.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:05.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:05.161369+0000) 2024-03-06T10:31:06.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:06.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:06.161661+0000) 2024-03-06T10:31:07.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:07.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:07.161905+0000) 2024-03-06T10:31:08.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:08.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:08.162141+0000) 2024-03-06T10:31:09.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:09.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:09.162381+0000) 2024-03-06T10:31:10.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:10.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:10.162657+0000) 2024-03-06T10:31:11.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:11.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:11.162909+0000) 2024-03-06T10:31:12.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:12.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:12.163173+0000) 2024-03-06T10:31:13.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:13.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:13.163419+0000) 2024-03-06T10:31:13.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:13.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:14.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:14.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:14.163705+0000) 2024-03-06T10:31:15.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:15.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:15.164019+0000) 2024-03-06T10:31:16.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:16.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:16.164300+0000) 2024-03-06T10:31:17.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:17.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:17.164586+0000) 2024-03-06T10:31:18.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:18.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:18.164895+0000) 2024-03-06T10:31:19.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:19.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:19.165147+0000) 2024-03-06T10:31:20.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:20.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:20.165429+0000) 2024-03-06T10:31:21.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:21.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:21.165721+0000) 2024-03-06T10:31:22.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:22.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:22.165973+0000) 2024-03-06T10:31:23.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:23.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:23.166226+0000) 2024-03-06T10:31:24.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:24.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:24.166487+0000) 2024-03-06T10:31:25.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:25.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:25.166761+0000) 2024-03-06T10:31:26.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:26.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:26.167040+0000) 2024-03-06T10:31:27.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:27.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:27.167329+0000) 2024-03-06T10:31:28.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:28.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:28.167612+0000) 2024-03-06T10:31:29.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:29.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:29.167924+0000) 2024-03-06T10:31:30.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:30.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:30.168194+0000) 2024-03-06T10:31:31.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:31.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:31.168439+0000) 2024-03-06T10:31:32.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:32.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:32.168746+0000) 2024-03-06T10:31:33.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:33.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:33.168991+0000) 2024-03-06T10:31:34.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:34.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:34.169235+0000) 2024-03-06T10:31:35.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:35.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:35.169516+0000) 2024-03-06T10:31:36.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:36.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:36.169796+0000) 2024-03-06T10:31:37.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:37.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:37.170057+0000) 2024-03-06T10:31:38.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:38.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:38.170322+0000) 2024-03-06T10:31:39.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:39.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:39.170588+0000) 2024-03-06T10:31:40.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:40.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:40.170916+0000) 2024-03-06T10:31:41.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:41.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:41.171119+0000) 2024-03-06T10:31:42.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:42.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:42.171361+0000) 2024-03-06T10:31:43.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:43.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:43.171610+0000) 2024-03-06T10:31:44.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:44.125 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:44.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:44.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:44.171829+0000) 2024-03-06T10:31:45.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:45.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:45.172056+0000) 2024-03-06T10:31:46.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:46.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:46.172314+0000) 2024-03-06T10:31:47.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:47.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:47.172567+0000) 2024-03-06T10:31:48.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:48.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:48.172819+0000) 2024-03-06T10:31:49.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:49.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:49.173066+0000) 2024-03-06T10:31:50.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:50.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:50.173326+0000) 2024-03-06T10:31:51.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:51.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:51.173573+0000) 2024-03-06T10:31:52.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:52.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:52.173832+0000) 2024-03-06T10:31:53.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:53.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:53.174087+0000) 2024-03-06T10:31:54.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:54.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:54.174343+0000) 2024-03-06T10:31:55.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:55.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:55.174594+0000) 2024-03-06T10:31:56.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:56.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:56.174865+0000) 2024-03-06T10:31:57.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:57.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:57.175068+0000) 2024-03-06T10:31:58.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:58.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:58.175312+0000) 2024-03-06T10:31:59.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:31:59.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:59.175543+0000) 2024-03-06T10:32:00.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:00.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:00.175796+0000) 2024-03-06T10:32:01.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:01.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:01.175990+0000) 2024-03-06T10:32:02.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:02.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:02.176248+0000) 2024-03-06T10:32:03.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:03.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:03.176446+0000) 2024-03-06T10:32:04.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:04.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:04.176759+0000) 2024-03-06T10:32:05.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:05.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:05.176993+0000) 2024-03-06T10:32:06.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:06.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:06.177251+0000) 2024-03-06T10:32:07.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:07.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:07.177514+0000) 2024-03-06T10:32:08.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:08.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:08.177826+0000) 2024-03-06T10:32:09.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:09.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:09.178062+0000) 2024-03-06T10:32:10.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:10.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:10.178320+0000) 2024-03-06T10:32:11.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:11.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:11.178555+0000) 2024-03-06T10:32:12.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:12.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:12.178820+0000) 2024-03-06T10:32:13.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:13.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:13.179074+0000) 2024-03-06T10:32:14.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:14.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:14.179322+0000) 2024-03-06T10:32:14.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:14.271 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:15.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:15.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:15.179572+0000) 2024-03-06T10:32:16.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:16.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:16.179814+0000) 2024-03-06T10:32:17.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:17.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:17.180056+0000) 2024-03-06T10:32:18.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:18.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:18.180310+0000) 2024-03-06T10:32:19.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:19.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:19.180562+0000) 2024-03-06T10:32:20.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:20.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:20.180830+0000) 2024-03-06T10:32:21.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:21.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:21.181018+0000) 2024-03-06T10:32:22.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:22.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:22.181275+0000) 2024-03-06T10:32:23.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:23.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:23.181535+0000) 2024-03-06T10:32:24.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:24.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:24.181777+0000) 2024-03-06T10:32:25.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:25.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:25.182013+0000) 2024-03-06T10:32:26.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:26.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:26.182271+0000) 2024-03-06T10:32:27.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:27.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:27.182539+0000) 2024-03-06T10:32:28.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:28.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:28.182779+0000) 2024-03-06T10:32:29.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:29.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:29.183036+0000) 2024-03-06T10:32:30.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:30.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:30.183288+0000) 2024-03-06T10:32:31.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:31.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:31.183529+0000) 2024-03-06T10:32:32.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:32.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:32.183791+0000) 2024-03-06T10:32:33.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:33.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:33.184041+0000) 2024-03-06T10:32:34.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:34.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:34.184287+0000) 2024-03-06T10:32:35.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:35.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:35.184535+0000) 2024-03-06T10:32:36.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:36.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:36.184797+0000) 2024-03-06T10:32:37.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:37.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:37.184989+0000) 2024-03-06T10:32:38.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:38.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:38.185183+0000) 2024-03-06T10:32:39.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:39.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:39.185434+0000) 2024-03-06T10:32:40.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:40.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:40.185749+0000) 2024-03-06T10:32:41.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:41.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:41.186006+0000) 2024-03-06T10:32:42.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:42.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:42.186263+0000) 2024-03-06T10:32:43.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:43.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:43.186470+0000) 2024-03-06T10:32:44.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:44.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:44.186760+0000) 2024-03-06T10:32:44.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:44.414 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:45.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:45.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:45.187008+0000) 2024-03-06T10:32:46.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:46.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:46.187220+0000) 2024-03-06T10:32:47.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:47.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:47.187465+0000) 2024-03-06T10:32:48.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:48.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:48.187720+0000) 2024-03-06T10:32:49.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:49.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:49.187980+0000) 2024-03-06T10:32:50.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:50.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:50.188268+0000) 2024-03-06T10:32:51.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:51.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:51.188476+0000) 2024-03-06T10:32:52.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:52.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:52.188743+0000) 2024-03-06T10:32:53.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:53.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:53.188992+0000) 2024-03-06T10:32:54.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:54.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:54.189255+0000) 2024-03-06T10:32:55.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:55.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:55.189509+0000) 2024-03-06T10:32:56.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:56.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:56.189836+0000) 2024-03-06T10:32:57.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:57.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:57.190108+0000) 2024-03-06T10:32:58.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:58.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:58.190365+0000) 2024-03-06T10:32:59.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:32:59.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:59.190667+0000) 2024-03-06T10:33:00.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:00.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:00.190929+0000) 2024-03-06T10:33:01.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:01.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:01.191205+0000) 2024-03-06T10:33:02.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:02.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:02.191434+0000) 2024-03-06T10:33:03.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:03.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:03.191720+0000) 2024-03-06T10:33:04.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:04.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:04.191938+0000) 2024-03-06T10:33:05.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:05.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:05.192133+0000) 2024-03-06T10:33:06.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:06.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:06.192339+0000) 2024-03-06T10:33:07.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:07.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:07.192624+0000) 2024-03-06T10:33:08.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:08.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:08.192913+0000) 2024-03-06T10:33:09.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:09.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:09.193163+0000) 2024-03-06T10:33:10.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:10.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:10.193418+0000) 2024-03-06T10:33:11.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:11.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:11.193703+0000) 2024-03-06T10:33:12.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:12.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:12.193956+0000) 2024-03-06T10:33:13.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:13.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:13.194206+0000) 2024-03-06T10:33:14.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:14.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:14.194464+0000) 2024-03-06T10:33:14.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:14.575 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:15.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:15.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:15.194692+0000) 2024-03-06T10:33:16.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:16.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:16.194900+0000) 2024-03-06T10:33:17.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:17.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:17.195157+0000) 2024-03-06T10:33:18.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:18.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:18.195411+0000) 2024-03-06T10:33:19.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:19.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:19.195709+0000) 2024-03-06T10:33:20.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:20.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:20.195984+0000) 2024-03-06T10:33:21.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:21.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:21.196235+0000) 2024-03-06T10:33:22.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:22.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:22.196496+0000) 2024-03-06T10:33:23.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:23.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:23.196755+0000) 2024-03-06T10:33:24.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:24.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:24.197028+0000) 2024-03-06T10:33:25.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:25.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:25.197278+0000) 2024-03-06T10:33:26.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:26.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:26.197532+0000) 2024-03-06T10:33:27.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:27.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:27.197798+0000) 2024-03-06T10:33:28.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:28.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:28.198060+0000) 2024-03-06T10:33:29.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:29.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:29.198323+0000) 2024-03-06T10:33:30.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:30.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:30.198573+0000) 2024-03-06T10:33:31.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:31.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:31.198845+0000) 2024-03-06T10:33:32.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:32.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:32.199124+0000) 2024-03-06T10:33:33.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:33.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:33.199393+0000) 2024-03-06T10:33:34.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:34.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:34.199696+0000) 2024-03-06T10:33:35.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:35.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:35.199968+0000) 2024-03-06T10:33:36.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:36.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:36.200245+0000) 2024-03-06T10:33:37.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:37.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:37.200541+0000) 2024-03-06T10:33:38.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:38.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:38.200774+0000) 2024-03-06T10:33:39.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:39.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:39.201065+0000) 2024-03-06T10:33:40.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:40.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:40.201305+0000) 2024-03-06T10:33:41.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:41.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:41.201547+0000) 2024-03-06T10:33:42.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:42.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:42.201797+0000) 2024-03-06T10:33:43.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:43.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:43.202018+0000) 2024-03-06T10:33:44.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:44.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:44.202289+0000) 2024-03-06T10:33:44.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:44.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:45.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:45.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:45.202558+0000) 2024-03-06T10:33:46.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:46.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:46.202820+0000) 2024-03-06T10:33:47.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:47.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:47.203061+0000) 2024-03-06T10:33:48.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:48.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:48.203317+0000) 2024-03-06T10:33:49.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:49.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:49.203568+0000) 2024-03-06T10:33:50.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:50.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:50.203796+0000) 2024-03-06T10:33:51.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:51.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:51.204023+0000) 2024-03-06T10:33:52.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:52.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:52.204291+0000) 2024-03-06T10:33:53.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:53.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:53.204543+0000) 2024-03-06T10:33:54.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:54.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:54.204794+0000) 2024-03-06T10:33:55.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:55.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:55.205031+0000) 2024-03-06T10:33:56.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:56.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:56.205271+0000) 2024-03-06T10:33:57.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:57.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:57.205529+0000) 2024-03-06T10:33:58.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:58.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:58.205784+0000) 2024-03-06T10:33:59.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:33:59.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:59.205971+0000) 2024-03-06T10:34:00.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:00.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:00.206241+0000) 2024-03-06T10:34:01.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:01.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:01.206499+0000) 2024-03-06T10:34:02.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:02.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:02.206765+0000) 2024-03-06T10:34:03.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:03.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:03.207008+0000) 2024-03-06T10:34:04.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:04.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:04.207205+0000) 2024-03-06T10:34:05.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:05.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:05.207439+0000) 2024-03-06T10:34:06.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:06.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:06.207758+0000) 2024-03-06T10:34:07.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:07.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:07.207996+0000) 2024-03-06T10:34:08.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:08.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:08.208258+0000) 2024-03-06T10:34:09.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:09.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:09.208499+0000) 2024-03-06T10:34:10.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:10.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:10.208758+0000) 2024-03-06T10:34:11.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:11.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:11.209013+0000) 2024-03-06T10:34:12.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:12.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:12.209281+0000) 2024-03-06T10:34:13.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:13.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:13.209535+0000) 2024-03-06T10:34:14.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:14.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:14.209783+0000) 2024-03-06T10:34:14.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:14.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:15.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:15.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:15.210038+0000) 2024-03-06T10:34:16.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:16.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:16.210309+0000) 2024-03-06T10:34:17.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:17.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:17.210562+0000) 2024-03-06T10:34:18.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:18.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:18.210798+0000) 2024-03-06T10:34:19.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:19.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:19.211049+0000) 2024-03-06T10:34:20.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:20.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:20.211317+0000) 2024-03-06T10:34:21.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:21.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:21.211567+0000) 2024-03-06T10:34:22.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:22.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:22.211810+0000) 2024-03-06T10:34:23.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:23.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:23.212083+0000) 2024-03-06T10:34:24.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:24.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:24.212348+0000) 2024-03-06T10:34:25.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:25.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:25.212596+0000) 2024-03-06T10:34:26.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:26.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:26.212909+0000) 2024-03-06T10:34:27.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:27.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:27.213188+0000) 2024-03-06T10:34:28.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:28.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:28.213455+0000) 2024-03-06T10:34:29.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:29.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:29.213753+0000) 2024-03-06T10:34:30.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:30.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:30.214040+0000) 2024-03-06T10:34:31.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:31.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:31.214308+0000) 2024-03-06T10:34:32.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:32.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:32.214578+0000) 2024-03-06T10:34:33.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:33.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:33.214849+0000) 2024-03-06T10:34:34.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:34.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:34.215124+0000) 2024-03-06T10:34:35.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:35.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:35.215363+0000) 2024-03-06T10:34:36.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:36.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:36.215675+0000) 2024-03-06T10:34:37.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:37.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:37.215964+0000) 2024-03-06T10:34:38.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:38.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:38.216280+0000) 2024-03-06T10:34:39.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:39.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:39.216552+0000) 2024-03-06T10:34:40.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:40.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:40.216810+0000) 2024-03-06T10:34:41.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:41.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:41.217080+0000) 2024-03-06T10:34:42.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:42.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:42.217367+0000) 2024-03-06T10:34:43.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:43.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:43.217670+0000) 2024-03-06T10:34:44.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:44.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:44.217924+0000) 2024-03-06T10:34:45.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:45.027 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:45.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:45.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:45.218183+0000) 2024-03-06T10:34:46.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:46.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:46.218447+0000) 2024-03-06T10:34:47.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:47.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:47.218744+0000) 2024-03-06T10:34:48.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:48.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:48.218966+0000) 2024-03-06T10:34:49.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:49.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:49.219270+0000) 2024-03-06T10:34:50.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:50.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:50.219561+0000) 2024-03-06T10:34:51.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:51.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:51.219811+0000) 2024-03-06T10:34:52.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:52.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:52.220090+0000) 2024-03-06T10:34:53.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:53.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:53.220357+0000) 2024-03-06T10:34:54.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:54.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:54.220671+0000) 2024-03-06T10:34:55.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:55.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:55.220952+0000) 2024-03-06T10:34:56.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:56.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:56.221230+0000) 2024-03-06T10:34:57.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:57.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:57.221521+0000) 2024-03-06T10:34:58.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:58.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:58.221752+0000) 2024-03-06T10:34:59.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:34:59.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:59.222023+0000) 2024-03-06T10:35:00.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:00.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:00.222297+0000) 2024-03-06T10:35:01.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:01.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:01.222562+0000) 2024-03-06T10:35:02.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:02.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:02.222793+0000) 2024-03-06T10:35:03.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:03.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:03.223043+0000) 2024-03-06T10:35:04.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:04.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:04.223301+0000) 2024-03-06T10:35:05.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:05.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:05.223560+0000) 2024-03-06T10:35:06.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:06.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:06.223827+0000) 2024-03-06T10:35:07.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:07.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:07.224042+0000) 2024-03-06T10:35:08.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:08.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:08.224310+0000) 2024-03-06T10:35:09.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:09.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:09.224564+0000) 2024-03-06T10:35:10.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:10.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:10.224826+0000) 2024-03-06T10:35:11.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:11.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:11.225080+0000) 2024-03-06T10:35:12.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:12.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:12.225326+0000) 2024-03-06T10:35:13.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:13.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:13.225515+0000) 2024-03-06T10:35:14.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:14.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:14.225775+0000) 2024-03-06T10:35:15.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:15.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:15.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:15.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:15.225949+0000) 2024-03-06T10:35:16.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:16.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:16.226195+0000) 2024-03-06T10:35:17.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:17.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:17.226467+0000) 2024-03-06T10:35:18.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:18.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:18.226762+0000) 2024-03-06T10:35:19.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:19.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:19.227014+0000) 2024-03-06T10:35:20.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:20.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:20.227332+0000) 2024-03-06T10:35:21.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:21.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:21.227602+0000) 2024-03-06T10:35:22.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:22.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:22.227885+0000) 2024-03-06T10:35:23.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:23.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:23.228124+0000) 2024-03-06T10:35:24.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:24.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:24.228383+0000) 2024-03-06T10:35:25.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:25.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:25.228661+0000) 2024-03-06T10:35:26.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:26.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:26.228968+0000) 2024-03-06T10:35:27.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:27.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:27.229197+0000) 2024-03-06T10:35:28.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:28.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:28.229456+0000) 2024-03-06T10:35:29.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:29.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:29.229750+0000) 2024-03-06T10:35:30.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:30.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:30.230014+0000) 2024-03-06T10:35:31.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:31.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:31.230274+0000) 2024-03-06T10:35:32.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:32.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:32.230524+0000) 2024-03-06T10:35:33.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:33.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:33.230766+0000) 2024-03-06T10:35:34.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:34.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:34.231027+0000) 2024-03-06T10:35:35.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:35.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:35.231281+0000) 2024-03-06T10:35:36.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:36.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:36.231546+0000) 2024-03-06T10:35:37.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:37.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:37.231797+0000) 2024-03-06T10:35:38.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:38.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:38.232056+0000) 2024-03-06T10:35:39.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:39.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:39.232310+0000) 2024-03-06T10:35:40.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:40.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:40.232544+0000) 2024-03-06T10:35:41.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:41.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:41.232774+0000) 2024-03-06T10:35:42.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:42.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:42.233026+0000) 2024-03-06T10:35:43.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:43.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:43.233282+0000) 2024-03-06T10:35:44.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:44.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:44.233528+0000) 2024-03-06T10:35:45.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:45.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:45.233782+0000) 2024-03-06T10:35:45.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:45.317 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:46.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:46.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:46.234048+0000) 2024-03-06T10:35:47.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:47.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:47.234250+0000) 2024-03-06T10:35:48.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:48.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:48.234511+0000) 2024-03-06T10:35:49.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:49.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:49.234731+0000) 2024-03-06T10:35:50.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:50.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:50.234974+0000) 2024-03-06T10:35:51.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:51.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:51.235261+0000) 2024-03-06T10:35:52.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:52.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:52.235505+0000) 2024-03-06T10:35:53.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:53.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:53.235739+0000) 2024-03-06T10:35:54.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:54.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:54.235999+0000) 2024-03-06T10:35:55.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:55.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:55.236251+0000) 2024-03-06T10:35:56.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:56.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:56.236518+0000) 2024-03-06T10:35:57.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:57.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:57.236763+0000) 2024-03-06T10:35:58.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:58.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:58.237025+0000) 2024-03-06T10:35:59.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:35:59.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:59.237282+0000) 2024-03-06T10:36:00.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:00.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:00.237471+0000) 2024-03-06T10:36:01.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:01.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:01.237745+0000) 2024-03-06T10:36:02.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:02.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:02.238001+0000) 2024-03-06T10:36:03.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:03.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:03.238259+0000) 2024-03-06T10:36:04.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:04.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:04.238522+0000) 2024-03-06T10:36:05.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:05.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:05.238968+0000) 2024-03-06T10:36:06.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:06.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:06.239234+0000) 2024-03-06T10:36:07.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:07.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:07.239483+0000) 2024-03-06T10:36:08.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:08.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:08.239766+0000) 2024-03-06T10:36:09.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:09.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:09.240020+0000) 2024-03-06T10:36:10.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:10.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:10.240284+0000) 2024-03-06T10:36:11.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:11.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:11.240537+0000) 2024-03-06T10:36:12.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:12.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:12.240780+0000) 2024-03-06T10:36:13.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:13.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:13.241038+0000) 2024-03-06T10:36:14.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:14.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:14.241353+0000) 2024-03-06T10:36:15.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:15.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:15.241587+0000) 2024-03-06T10:36:15.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:15.472 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:16.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:16.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:16.241854+0000) 2024-03-06T10:36:17.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:17.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:17.242102+0000) 2024-03-06T10:36:18.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:18.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:18.242299+0000) 2024-03-06T10:36:19.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:19.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:19.242533+0000) 2024-03-06T10:36:20.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:20.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:20.242780+0000) 2024-03-06T10:36:21.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:21.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:21.243008+0000) 2024-03-06T10:36:22.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:22.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:22.243224+0000) 2024-03-06T10:36:23.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:23.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:23.243483+0000) 2024-03-06T10:36:24.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:24.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:24.243784+0000) 2024-03-06T10:36:25.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:25.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:25.244044+0000) 2024-03-06T10:36:26.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:26.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:26.244308+0000) 2024-03-06T10:36:27.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:27.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:27.244522+0000) 2024-03-06T10:36:28.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:28.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:28.244777+0000) 2024-03-06T10:36:29.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:29.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:29.245030+0000) 2024-03-06T10:36:30.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:30.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:30.245297+0000) 2024-03-06T10:36:31.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:31.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:31.245545+0000) 2024-03-06T10:36:32.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:32.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:32.245774+0000) 2024-03-06T10:36:33.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:33.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:33.246034+0000) 2024-03-06T10:36:34.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:34.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:34.246295+0000) 2024-03-06T10:36:35.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:35.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:35.246554+0000) 2024-03-06T10:36:36.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:36.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:36.246807+0000) 2024-03-06T10:36:37.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:37.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:37.247010+0000) 2024-03-06T10:36:38.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:38.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:38.247272+0000) 2024-03-06T10:36:39.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:39.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:39.247540+0000) 2024-03-06T10:36:40.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:40.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:40.247790+0000) 2024-03-06T10:36:41.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:41.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:41.248044+0000) 2024-03-06T10:36:42.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:42.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:42.248235+0000) 2024-03-06T10:36:43.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:43.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:43.248472+0000) 2024-03-06T10:36:44.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:44.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:44.248712+0000) 2024-03-06T10:36:45.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:45.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:45.248964+0000) 2024-03-06T10:36:45.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:45.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:46.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:46.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:46.249203+0000) 2024-03-06T10:36:47.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:47.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:47.249439+0000) 2024-03-06T10:36:48.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:48.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:48.249757+0000) 2024-03-06T10:36:49.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:49.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:49.250015+0000) 2024-03-06T10:36:50.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:50.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:50.250280+0000) 2024-03-06T10:36:51.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:51.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:51.250550+0000) 2024-03-06T10:36:52.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:52.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:52.250814+0000) 2024-03-06T10:36:53.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:53.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:53.251021+0000) 2024-03-06T10:36:54.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:54.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:54.251288+0000) 2024-03-06T10:36:55.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:55.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:55.251536+0000) 2024-03-06T10:36:56.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:56.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:56.251833+0000) 2024-03-06T10:36:57.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:57.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:57.252094+0000) 2024-03-06T10:36:58.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:58.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:58.252364+0000) 2024-03-06T10:36:59.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:36:59.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:59.252617+0000) 2024-03-06T10:37:00.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:00.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:00.252925+0000) 2024-03-06T10:37:01.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:01.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:01.253175+0000) 2024-03-06T10:37:02.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:02.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:02.253436+0000) 2024-03-06T10:37:03.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:03.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:03.253725+0000) 2024-03-06T10:37:04.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:04.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:04.253979+0000) 2024-03-06T10:37:05.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:05.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:05.254241+0000) 2024-03-06T10:37:06.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:06.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:06.254516+0000) 2024-03-06T10:37:07.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:07.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:07.254840+0000) 2024-03-06T10:37:08.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:08.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:08.255051+0000) 2024-03-06T10:37:09.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:09.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:09.255314+0000) 2024-03-06T10:37:10.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:10.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:10.255587+0000) 2024-03-06T10:37:11.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:11.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:11.255890+0000) 2024-03-06T10:37:12.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:12.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:12.256172+0000) 2024-03-06T10:37:13.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:13.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:13.256387+0000) 2024-03-06T10:37:14.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:14.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:14.256701+0000) 2024-03-06T10:37:15.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:15.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:15.256948+0000) 2024-03-06T10:37:15.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:15.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:16.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:16.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:16.257206+0000) 2024-03-06T10:37:17.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:17.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:17.257479+0000) 2024-03-06T10:37:18.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:18.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:18.257741+0000) 2024-03-06T10:37:19.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:19.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:19.258005+0000) 2024-03-06T10:37:20.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:20.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:20.258284+0000) 2024-03-06T10:37:21.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:21.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:21.258537+0000) 2024-03-06T10:37:22.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:22.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:22.258791+0000) 2024-03-06T10:37:23.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:23.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:23.258999+0000) 2024-03-06T10:37:24.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:24.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:24.259274+0000) 2024-03-06T10:37:25.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:25.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:25.259524+0000) 2024-03-06T10:37:26.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:26.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:26.259772+0000) 2024-03-06T10:37:27.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:27.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:27.260081+0000) 2024-03-06T10:37:28.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:28.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:28.260363+0000) 2024-03-06T10:37:29.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:29.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:29.260610+0000) 2024-03-06T10:37:30.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:30.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:30.260878+0000) 2024-03-06T10:37:31.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:31.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:31.261136+0000) 2024-03-06T10:37:32.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:32.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:32.261345+0000) 2024-03-06T10:37:33.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:33.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:33.261614+0000) 2024-03-06T10:37:34.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:34.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:34.261934+0000) 2024-03-06T10:37:35.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:35.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:35.262199+0000) 2024-03-06T10:37:36.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:36.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:36.262471+0000) 2024-03-06T10:37:37.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:37.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:37.262732+0000) 2024-03-06T10:37:38.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:38.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:38.263020+0000) 2024-03-06T10:37:39.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:39.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:39.263309+0000) 2024-03-06T10:37:40.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:40.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:40.263566+0000) 2024-03-06T10:37:41.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:41.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:41.263779+0000) 2024-03-06T10:37:42.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:42.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:42.263986+0000) 2024-03-06T10:37:43.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:43.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:43.264256+0000) 2024-03-06T10:37:44.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:44.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:44.264519+0000) 2024-03-06T10:37:45.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:45.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:45.264783+0000) 2024-03-06T10:37:45.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:45.933 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:46.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:46.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:46.265063+0000) 2024-03-06T10:37:47.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:47.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:47.265347+0000) 2024-03-06T10:37:48.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:48.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:48.265608+0000) 2024-03-06T10:37:49.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:49.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:49.265877+0000) 2024-03-06T10:37:50.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:50.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:50.266110+0000) 2024-03-06T10:37:51.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:51.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:51.266299+0000) 2024-03-06T10:37:52.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:52.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:52.266568+0000) 2024-03-06T10:37:53.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:53.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:53.266802+0000) 2024-03-06T10:37:54.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:54.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:54.267074+0000) 2024-03-06T10:37:55.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:55.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:55.267332+0000) 2024-03-06T10:37:56.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:56.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:56.267609+0000) 2024-03-06T10:37:57.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:57.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:57.267939+0000) 2024-03-06T10:37:58.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:58.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:58.268219+0000) 2024-03-06T10:37:59.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:37:59.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:59.268487+0000) 2024-03-06T10:38:00.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:00.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:00.268751+0000) 2024-03-06T10:38:01.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:01.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:01.269019+0000) 2024-03-06T10:38:02.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:02.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:02.269300+0000) 2024-03-06T10:38:03.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:03.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:03.269496+0000) 2024-03-06T10:38:04.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:04.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:04.269777+0000) 2024-03-06T10:38:05.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:05.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:05.270027+0000) 2024-03-06T10:38:06.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:06.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:06.270287+0000) 2024-03-06T10:38:07.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:07.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:07.270564+0000) 2024-03-06T10:38:08.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:08.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:08.270849+0000) 2024-03-06T10:38:09.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:09.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:09.271105+0000) 2024-03-06T10:38:10.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:10.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:10.271379+0000) 2024-03-06T10:38:11.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:11.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:11.271671+0000) 2024-03-06T10:38:12.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:12.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:12.271929+0000) 2024-03-06T10:38:13.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:13.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:13.272117+0000) 2024-03-06T10:38:14.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:14.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:14.272366+0000) 2024-03-06T10:38:15.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:15.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:15.272605+0000) 2024-03-06T10:38:16.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:16.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:16.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:16.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:16.272839+0000) 2024-03-06T10:38:17.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:17.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:17.273033+0000) 2024-03-06T10:38:18.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:18.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:18.273227+0000) 2024-03-06T10:38:19.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:19.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:19.273473+0000) 2024-03-06T10:38:20.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:20.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:20.273753+0000) 2024-03-06T10:38:21.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:21.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:21.274010+0000) 2024-03-06T10:38:22.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:22.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:22.274277+0000) 2024-03-06T10:38:23.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:23.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:23.274531+0000) 2024-03-06T10:38:24.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:24.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:24.274819+0000) 2024-03-06T10:38:25.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:25.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:25.275114+0000) 2024-03-06T10:38:26.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:26.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:26.275399+0000) 2024-03-06T10:38:27.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:27.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:27.275766+0000) 2024-03-06T10:38:28.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:28.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:28.276048+0000) 2024-03-06T10:38:29.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:29.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:29.276281+0000) 2024-03-06T10:38:30.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:30.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:30.276539+0000) 2024-03-06T10:38:31.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:31.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:31.276788+0000) 2024-03-06T10:38:32.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:32.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:32.277052+0000) 2024-03-06T10:38:33.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:33.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:33.277299+0000) 2024-03-06T10:38:34.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:34.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:34.277583+0000) 2024-03-06T10:38:35.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:35.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:35.277885+0000) 2024-03-06T10:38:36.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:36.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:36.278205+0000) 2024-03-06T10:38:37.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:37.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:37.278522+0000) 2024-03-06T10:38:38.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:38.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:38.278805+0000) 2024-03-06T10:38:39.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:39.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:39.279079+0000) 2024-03-06T10:38:40.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:40.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:40.279324+0000) 2024-03-06T10:38:41.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:41.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:41.279557+0000) 2024-03-06T10:38:42.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:42.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:42.279816+0000) 2024-03-06T10:38:43.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:43.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:43.280130+0000) 2024-03-06T10:38:44.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:44.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:44.280392+0000) 2024-03-06T10:38:45.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:45.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:45.280675+0000) 2024-03-06T10:38:46.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:46.226 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:46.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:46.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:46.280896+0000) 2024-03-06T10:38:47.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:47.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:47.281159+0000) 2024-03-06T10:38:48.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:48.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:48.281441+0000) 2024-03-06T10:38:49.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:49.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:49.281745+0000) 2024-03-06T10:38:50.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:50.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:50.282046+0000) 2024-03-06T10:38:51.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:51.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:51.282280+0000) 2024-03-06T10:38:52.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:52.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:52.282524+0000) 2024-03-06T10:38:53.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:53.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:53.282777+0000) 2024-03-06T10:38:54.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:54.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:54.282969+0000) 2024-03-06T10:38:55.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:55.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:55.283221+0000) 2024-03-06T10:38:56.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:56.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:56.283462+0000) 2024-03-06T10:38:57.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:57.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:57.283741+0000) 2024-03-06T10:38:58.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:58.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:58.284007+0000) 2024-03-06T10:38:59.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:38:59.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:59.284242+0000) 2024-03-06T10:39:00.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:00.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:00.284484+0000) 2024-03-06T10:39:01.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:01.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:01.284752+0000) 2024-03-06T10:39:02.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:02.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:02.285003+0000) 2024-03-06T10:39:03.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:03.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:03.285245+0000) 2024-03-06T10:39:04.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:04.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:04.285512+0000) 2024-03-06T10:39:05.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:05.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:05.285749+0000) 2024-03-06T10:39:06.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:06.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:06.285989+0000) 2024-03-06T10:39:07.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:07.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:07.286250+0000) 2024-03-06T10:39:08.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:08.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:08.286536+0000) 2024-03-06T10:39:09.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:09.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:09.286786+0000) 2024-03-06T10:39:10.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:10.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:10.287005+0000) 2024-03-06T10:39:11.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:11.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:11.287266+0000) 2024-03-06T10:39:12.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:12.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:12.287544+0000) 2024-03-06T10:39:13.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:13.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:13.287821+0000) 2024-03-06T10:39:14.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:14.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:14.288100+0000) 2024-03-06T10:39:15.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:15.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:15.288389+0000) 2024-03-06T10:39:16.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:16.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:16.288697+0000) 2024-03-06T10:39:16.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:16.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:17.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:17.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:17.288961+0000) 2024-03-06T10:39:18.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:18.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:18.289245+0000) 2024-03-06T10:39:19.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:19.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:19.289508+0000) 2024-03-06T10:39:20.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:20.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:20.289789+0000) 2024-03-06T10:39:21.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:21.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:21.290057+0000) 2024-03-06T10:39:22.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:22.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:22.290355+0000) 2024-03-06T10:39:23.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:23.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:23.290666+0000) 2024-03-06T10:39:24.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:24.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:24.290947+0000) 2024-03-06T10:39:25.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:25.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:25.291203+0000) 2024-03-06T10:39:26.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:26.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:26.291470+0000) 2024-03-06T10:39:27.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:27.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:27.291748+0000) 2024-03-06T10:39:28.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:28.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:28.292035+0000) 2024-03-06T10:39:29.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:29.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:29.292245+0000) 2024-03-06T10:39:30.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:30.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:30.292521+0000) 2024-03-06T10:39:31.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:31.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:31.292787+0000) 2024-03-06T10:39:32.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:32.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:32.293069+0000) 2024-03-06T10:39:33.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:33.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:33.293344+0000) 2024-03-06T10:39:34.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:34.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:34.293690+0000) 2024-03-06T10:39:35.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:35.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:35.294011+0000) 2024-03-06T10:39:36.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:36.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:36.294359+0000) 2024-03-06T10:39:37.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:37.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:37.294705+0000) 2024-03-06T10:39:38.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:38.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:38.295034+0000) 2024-03-06T10:39:39.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:39.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:39.295332+0000) 2024-03-06T10:39:40.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:40.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:40.295678+0000) 2024-03-06T10:39:41.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:41.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:41.295984+0000) 2024-03-06T10:39:42.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:42.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:42.296282+0000) 2024-03-06T10:39:43.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:43.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:43.296569+0000) 2024-03-06T10:39:44.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:44.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:44.296892+0000) 2024-03-06T10:39:45.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:45.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:45.297190+0000) 2024-03-06T10:39:46.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:46.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:46.297485+0000) 2024-03-06T10:39:46.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:46.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:47.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:47.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:47.297787+0000) 2024-03-06T10:39:48.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:48.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:48.298050+0000) 2024-03-06T10:39:49.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:49.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:49.298338+0000) 2024-03-06T10:39:50.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:50.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:50.298671+0000) 2024-03-06T10:39:51.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:51.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:51.298979+0000) 2024-03-06T10:39:52.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:52.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:52.299327+0000) 2024-03-06T10:39:53.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:53.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:53.299683+0000) 2024-03-06T10:39:54.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:54.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:54.299951+0000) 2024-03-06T10:39:55.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:55.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:55.300269+0000) 2024-03-06T10:39:56.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:56.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:56.300564+0000) 2024-03-06T10:39:57.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:57.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:57.300862+0000) 2024-03-06T10:39:58.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:58.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:58.301119+0000) 2024-03-06T10:39:59.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:39:59.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:59.301433+0000) 2024-03-06T10:40:00.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:00.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:00.301773+0000) 2024-03-06T10:40:01.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:01.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:01.302075+0000) 2024-03-06T10:40:02.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:02.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:02.302391+0000) 2024-03-06T10:40:03.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:03.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:03.302735+0000) 2024-03-06T10:40:04.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:04.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:04.303038+0000) 2024-03-06T10:40:05.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:05.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:05.303339+0000) 2024-03-06T10:40:06.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:06.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:06.303571+0000) 2024-03-06T10:40:07.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:07.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:07.303873+0000) 2024-03-06T10:40:08.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:08.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:08.304109+0000) 2024-03-06T10:40:09.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:09.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:09.304436+0000) 2024-03-06T10:40:10.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:10.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:10.304795+0000) 2024-03-06T10:40:11.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:11.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:11.305101+0000) 2024-03-06T10:40:12.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:12.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:12.305346+0000) 2024-03-06T10:40:13.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:13.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:13.305667+0000) 2024-03-06T10:40:14.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:14.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:14.305941+0000) 2024-03-06T10:40:15.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:15.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:15.306169+0000) 2024-03-06T10:40:16.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:16.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:16.306484+0000) 2024-03-06T10:40:16.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:16.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:17.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:17.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:17.306827+0000) 2024-03-06T10:40:18.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:18.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:18.307063+0000) 2024-03-06T10:40:19.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:19.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:19.307339+0000) 2024-03-06T10:40:20.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:20.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:20.307598+0000) 2024-03-06T10:40:21.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:21.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:21.307832+0000) 2024-03-06T10:40:22.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:22.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:22.308096+0000) 2024-03-06T10:40:23.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:23.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:23.308343+0000) 2024-03-06T10:40:24.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:24.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:24.308691+0000) 2024-03-06T10:40:25.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:25.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:25.308942+0000) 2024-03-06T10:40:26.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:26.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:26.309122+0000) 2024-03-06T10:40:27.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:27.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:27.309405+0000) 2024-03-06T10:40:28.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:28.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:28.309736+0000) 2024-03-06T10:40:29.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:29.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:29.309999+0000) 2024-03-06T10:40:30.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:30.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:30.310254+0000) 2024-03-06T10:40:31.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:31.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:31.310486+0000) 2024-03-06T10:40:32.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:32.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:32.310759+0000) 2024-03-06T10:40:33.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:33.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:33.310988+0000) 2024-03-06T10:40:34.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:34.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:34.311244+0000) 2024-03-06T10:40:35.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:35.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:35.311490+0000) 2024-03-06T10:40:36.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:36.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:36.311746+0000) 2024-03-06T10:40:37.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:37.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:37.312008+0000) 2024-03-06T10:40:38.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:38.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:38.312266+0000) 2024-03-06T10:40:39.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:39.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:39.312515+0000) 2024-03-06T10:40:40.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:40.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:40.312752+0000) 2024-03-06T10:40:41.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:41.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:41.313023+0000) 2024-03-06T10:40:42.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:42.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:42.313349+0000) 2024-03-06T10:40:43.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:43.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:43.313696+0000) 2024-03-06T10:40:44.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:44.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:44.314017+0000) 2024-03-06T10:40:45.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:45.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:45.314328+0000) 2024-03-06T10:40:46.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:46.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:46.314671+0000) 2024-03-06T10:40:46.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:46.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:47.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:47.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:47.314925+0000) 2024-03-06T10:40:48.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:48.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:48.315229+0000) 2024-03-06T10:40:49.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:49.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:49.315478+0000) 2024-03-06T10:40:50.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:50.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:50.315766+0000) 2024-03-06T10:40:51.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:51.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:51.316017+0000) 2024-03-06T10:40:52.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:52.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:52.316248+0000) 2024-03-06T10:40:53.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:53.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:53.316446+0000) 2024-03-06T10:40:54.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:54.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:54.316738+0000) 2024-03-06T10:40:55.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:55.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:55.316989+0000) 2024-03-06T10:40:56.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:56.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:56.317237+0000) 2024-03-06T10:40:57.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:57.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:57.317481+0000) 2024-03-06T10:40:58.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:58.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:58.317709+0000) 2024-03-06T10:40:59.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:40:59.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:59.317960+0000) 2024-03-06T10:41:00.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:00.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:00.318221+0000) 2024-03-06T10:41:01.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:01.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:01.318462+0000) 2024-03-06T10:41:02.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:02.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:02.318749+0000) 2024-03-06T10:41:03.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:03.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:03.318984+0000) 2024-03-06T10:41:04.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:04.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:04.319236+0000) 2024-03-06T10:41:05.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:05.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:05.319468+0000) 2024-03-06T10:41:06.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:06.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:06.319759+0000) 2024-03-06T10:41:07.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:07.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:07.320011+0000) 2024-03-06T10:41:08.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:08.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:08.320270+0000) 2024-03-06T10:41:09.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:09.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:09.320520+0000) 2024-03-06T10:41:10.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:10.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:10.320786+0000) 2024-03-06T10:41:11.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:11.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:11.321020+0000) 2024-03-06T10:41:12.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:12.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:12.321343+0000) 2024-03-06T10:41:13.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:13.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:13.321595+0000) 2024-03-06T10:41:14.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:14.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:14.321848+0000) 2024-03-06T10:41:15.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:15.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:15.322116+0000) 2024-03-06T10:41:16.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:16.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:16.322388+0000) 2024-03-06T10:41:17.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:17.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:17.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:17.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:17.322677+0000) 2024-03-06T10:41:18.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:18.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:18.322923+0000) 2024-03-06T10:41:19.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:19.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:19.323159+0000) 2024-03-06T10:41:20.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:20.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:20.323404+0000) 2024-03-06T10:41:21.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:21.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:21.323690+0000) 2024-03-06T10:41:22.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:22.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:22.323942+0000) 2024-03-06T10:41:23.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:23.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:23.324180+0000) 2024-03-06T10:41:24.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:24.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:24.324426+0000) 2024-03-06T10:41:25.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:25.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:25.324700+0000) 2024-03-06T10:41:26.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:26.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:26.324957+0000) 2024-03-06T10:41:27.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:27.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:27.325149+0000) 2024-03-06T10:41:28.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:28.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:28.325413+0000) 2024-03-06T10:41:29.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:29.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:29.325717+0000) 2024-03-06T10:41:30.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:30.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:30.325955+0000) 2024-03-06T10:41:31.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:31.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:31.326195+0000) 2024-03-06T10:41:32.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:32.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:32.326431+0000) 2024-03-06T10:41:33.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:33.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:33.326727+0000) 2024-03-06T10:41:34.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:34.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:34.326988+0000) 2024-03-06T10:41:35.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:35.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:35.327244+0000) 2024-03-06T10:41:36.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:36.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:36.327499+0000) 2024-03-06T10:41:37.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:37.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:37.327769+0000) 2024-03-06T10:41:38.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:38.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:38.328030+0000) 2024-03-06T10:41:39.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:39.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:39.328263+0000) 2024-03-06T10:41:40.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:40.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:40.328508+0000) 2024-03-06T10:41:41.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:41.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:41.328759+0000) 2024-03-06T10:41:42.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:42.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:42.329031+0000) 2024-03-06T10:41:43.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:43.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:43.329281+0000) 2024-03-06T10:41:44.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:44.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:44.329559+0000) 2024-03-06T10:41:45.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:45.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:45.329805+0000) 2024-03-06T10:41:46.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:46.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:46.330062+0000) 2024-03-06T10:41:47.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:47.168 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:47.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:47.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:47.330328+0000) 2024-03-06T10:41:48.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:48.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:48.330579+0000) 2024-03-06T10:41:49.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:49.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:49.330835+0000) 2024-03-06T10:41:50.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:50.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:50.331106+0000) 2024-03-06T10:41:51.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:51.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:51.331364+0000) 2024-03-06T10:41:52.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:52.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:52.331613+0000) 2024-03-06T10:41:53.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:53.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:53.331910+0000) 2024-03-06T10:41:54.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:54.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:54.332112+0000) 2024-03-06T10:41:55.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:55.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:55.332365+0000) 2024-03-06T10:41:56.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:56.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:56.332568+0000) 2024-03-06T10:41:57.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:57.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:57.332842+0000) 2024-03-06T10:41:58.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:58.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:58.333093+0000) 2024-03-06T10:41:59.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:41:59.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:59.333335+0000) 2024-03-06T10:42:00.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:00.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:00.333597+0000) 2024-03-06T10:42:01.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:01.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:01.333861+0000) 2024-03-06T10:42:02.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:02.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:02.334098+0000) 2024-03-06T10:42:03.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:03.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:03.334357+0000) 2024-03-06T10:42:04.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:04.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:04.334610+0000) 2024-03-06T10:42:05.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:05.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:05.334899+0000) 2024-03-06T10:42:06.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:06.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:06.335164+0000) 2024-03-06T10:42:07.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:07.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:07.335444+0000) 2024-03-06T10:42:08.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:08.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:08.335731+0000) 2024-03-06T10:42:09.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:09.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:09.335927+0000) 2024-03-06T10:42:10.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:10.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:10.336122+0000) 2024-03-06T10:42:11.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:11.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:11.336365+0000) 2024-03-06T10:42:12.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:12.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:12.336617+0000) 2024-03-06T10:42:13.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:13.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:13.336927+0000) 2024-03-06T10:42:14.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:14.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:14.337187+0000) 2024-03-06T10:42:15.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:15.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:15.337437+0000) 2024-03-06T10:42:16.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:16.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:16.337742+0000) 2024-03-06T10:42:17.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:17.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:17.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:17.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:17.337942+0000) 2024-03-06T10:42:18.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:18.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:18.338204+0000) 2024-03-06T10:42:19.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:19.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:19.338451+0000) 2024-03-06T10:42:20.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:20.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:20.338765+0000) 2024-03-06T10:42:21.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:21.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:21.339020+0000) 2024-03-06T10:42:22.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:22.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:22.339229+0000) 2024-03-06T10:42:23.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:23.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:23.339495+0000) 2024-03-06T10:42:24.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:24.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:24.339829+0000) 2024-03-06T10:42:25.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:25.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:25.340072+0000) 2024-03-06T10:42:26.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:26.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:26.340264+0000) 2024-03-06T10:42:27.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:27.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:27.340517+0000) 2024-03-06T10:42:28.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:28.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:28.340811+0000) 2024-03-06T10:42:29.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:29.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:29.341044+0000) 2024-03-06T10:42:30.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:30.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:30.341278+0000) 2024-03-06T10:42:31.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:31.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:31.341508+0000) 2024-03-06T10:42:32.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:32.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:32.341762+0000) 2024-03-06T10:42:33.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:33.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:33.341995+0000) 2024-03-06T10:42:34.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:34.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:34.342256+0000) 2024-03-06T10:42:35.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:35.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:35.342495+0000) 2024-03-06T10:42:36.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:36.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:36.342748+0000) 2024-03-06T10:42:37.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:37.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:37.343005+0000) 2024-03-06T10:42:38.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:38.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:38.343266+0000) 2024-03-06T10:42:39.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:39.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:39.343510+0000) 2024-03-06T10:42:40.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:40.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:40.343769+0000) 2024-03-06T10:42:41.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:41.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:41.344000+0000) 2024-03-06T10:42:42.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:42.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:42.344190+0000) 2024-03-06T10:42:43.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:43.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:43.344416+0000) 2024-03-06T10:42:44.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:44.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:44.344615+0000) 2024-03-06T10:42:45.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:45.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:45.344824+0000) 2024-03-06T10:42:46.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:46.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:46.345051+0000) 2024-03-06T10:42:47.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:47.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:47.345297+0000) 2024-03-06T10:42:47.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:47.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:48.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:48.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:48.345536+0000) 2024-03-06T10:42:49.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:49.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:49.345785+0000) 2024-03-06T10:42:50.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:50.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:50.346039+0000) 2024-03-06T10:42:51.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:51.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:51.346268+0000) 2024-03-06T10:42:52.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:52.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:52.346501+0000) 2024-03-06T10:42:53.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:53.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:53.346733+0000) 2024-03-06T10:42:54.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:54.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:54.347047+0000) 2024-03-06T10:42:55.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:55.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:55.347280+0000) 2024-03-06T10:42:56.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:56.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:56.347532+0000) 2024-03-06T10:42:57.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:57.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:57.347788+0000) 2024-03-06T10:42:58.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:58.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:58.347982+0000) 2024-03-06T10:42:59.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:42:59.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:59.348229+0000) 2024-03-06T10:43:00.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:00.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:00.348471+0000) 2024-03-06T10:43:01.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:01.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:01.348734+0000) 2024-03-06T10:43:02.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:02.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:02.348992+0000) 2024-03-06T10:43:03.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:03.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:03.349227+0000) 2024-03-06T10:43:04.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:04.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:04.349493+0000) 2024-03-06T10:43:05.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:05.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:05.349746+0000) 2024-03-06T10:43:06.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:06.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:06.350005+0000) 2024-03-06T10:43:07.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:07.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:07.350266+0000) 2024-03-06T10:43:08.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:08.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:08.350499+0000) 2024-03-06T10:43:09.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:09.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:09.350748+0000) 2024-03-06T10:43:10.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:10.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:10.351013+0000) 2024-03-06T10:43:11.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:11.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:11.351249+0000) 2024-03-06T10:43:12.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:12.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:12.351510+0000) 2024-03-06T10:43:13.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:13.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:13.351758+0000) 2024-03-06T10:43:14.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:14.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:14.351958+0000) 2024-03-06T10:43:15.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:15.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:15.352209+0000) 2024-03-06T10:43:16.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:16.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:16.352465+0000) 2024-03-06T10:43:17.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:17.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:17.352733+0000) 2024-03-06T10:43:17.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:17.608 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:18.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:18.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:18.352995+0000) 2024-03-06T10:43:19.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:19.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:19.353249+0000) 2024-03-06T10:43:20.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:20.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:20.353511+0000) 2024-03-06T10:43:21.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:21.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:21.353741+0000) 2024-03-06T10:43:22.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:22.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:22.353993+0000) 2024-03-06T10:43:23.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:23.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:23.354231+0000) 2024-03-06T10:43:24.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:24.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:24.354485+0000) 2024-03-06T10:43:25.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:25.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:25.354731+0000) 2024-03-06T10:43:26.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:26.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:26.354930+0000) 2024-03-06T10:43:27.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:27.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:27.355190+0000) 2024-03-06T10:43:28.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:28.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:28.355448+0000) 2024-03-06T10:43:29.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:29.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:29.355730+0000) 2024-03-06T10:43:30.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:30.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:30.355978+0000) 2024-03-06T10:43:31.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:31.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:31.356234+0000) 2024-03-06T10:43:32.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:32.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:32.356475+0000) 2024-03-06T10:43:33.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:33.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:33.356728+0000) 2024-03-06T10:43:34.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:34.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:34.357009+0000) 2024-03-06T10:43:35.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:35.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:35.357243+0000) 2024-03-06T10:43:36.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:36.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:36.357499+0000) 2024-03-06T10:43:37.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:37.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:37.357759+0000) 2024-03-06T10:43:38.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:38.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:38.358011+0000) 2024-03-06T10:43:39.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:39.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:39.358264+0000) 2024-03-06T10:43:40.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:40.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:40.358502+0000) 2024-03-06T10:43:41.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:41.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:41.358749+0000) 2024-03-06T10:43:42.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:42.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:42.359045+0000) 2024-03-06T10:43:43.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:43.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:43.359278+0000) 2024-03-06T10:43:44.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:44.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:44.359543+0000) 2024-03-06T10:43:45.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:45.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:45.359789+0000) 2024-03-06T10:43:46.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:46.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:46.360056+0000) 2024-03-06T10:43:47.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:47.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:47.360306+0000) 2024-03-06T10:43:47.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:47.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:48.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:48.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:48.360551+0000) 2024-03-06T10:43:49.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:49.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:49.360804+0000) 2024-03-06T10:43:50.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:50.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:50.361065+0000) 2024-03-06T10:43:51.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:51.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:51.361297+0000) 2024-03-06T10:43:52.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:52.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:52.361549+0000) 2024-03-06T10:43:53.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:53.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:53.361801+0000) 2024-03-06T10:43:54.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:54.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:54.362079+0000) 2024-03-06T10:43:55.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:55.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:55.362351+0000) 2024-03-06T10:43:56.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:56.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:56.362683+0000) 2024-03-06T10:43:57.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:57.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:57.362933+0000) 2024-03-06T10:43:58.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:58.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:58.363223+0000) 2024-03-06T10:43:59.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:43:59.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:59.363509+0000) 2024-03-06T10:44:00.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:00.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:00.363760+0000) 2024-03-06T10:44:01.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:01.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:01.364024+0000) 2024-03-06T10:44:02.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:02.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:02.364305+0000) 2024-03-06T10:44:03.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:03.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:03.364577+0000) 2024-03-06T10:44:04.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:04.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:04.364878+0000) 2024-03-06T10:44:05.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:05.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:05.365152+0000) 2024-03-06T10:44:06.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:06.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:06.365428+0000) 2024-03-06T10:44:07.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:07.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:07.365747+0000) 2024-03-06T10:44:08.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:08.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:08.366006+0000) 2024-03-06T10:44:09.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:09.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:09.366293+0000) 2024-03-06T10:44:10.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:10.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:10.366569+0000) 2024-03-06T10:44:11.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:11.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:11.366820+0000) 2024-03-06T10:44:12.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:12.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:12.367102+0000) 2024-03-06T10:44:13.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:13.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:13.367355+0000) 2024-03-06T10:44:14.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:14.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:14.367713+0000) 2024-03-06T10:44:15.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:15.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:15.367989+0000) 2024-03-06T10:44:16.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:16.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:16.368252+0000) 2024-03-06T10:44:17.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:17.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:17.368547+0000) 2024-03-06T10:44:17.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:17.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:18.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:18.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:18.368805+0000) 2024-03-06T10:44:19.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:19.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:19.369076+0000) 2024-03-06T10:44:20.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:20.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:20.369338+0000) 2024-03-06T10:44:21.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:21.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:21.369611+0000) 2024-03-06T10:44:22.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:22.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:22.369940+0000) 2024-03-06T10:44:23.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:23.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:23.370133+0000) 2024-03-06T10:44:24.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:24.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:24.370390+0000) 2024-03-06T10:44:25.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:25.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:25.370692+0000) 2024-03-06T10:44:26.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:26.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:26.370894+0000) 2024-03-06T10:44:27.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:27.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:27.371157+0000) 2024-03-06T10:44:28.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:28.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:28.371406+0000) 2024-03-06T10:44:29.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:29.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:29.371705+0000) 2024-03-06T10:44:30.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:30.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:30.371951+0000) 2024-03-06T10:44:31.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:31.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:31.372191+0000) 2024-03-06T10:44:32.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:32.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:32.372440+0000) 2024-03-06T10:44:33.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:33.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:33.372743+0000) 2024-03-06T10:44:34.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:34.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:34.373015+0000) 2024-03-06T10:44:35.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:35.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:35.373320+0000) 2024-03-06T10:44:36.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:36.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:36.373580+0000) 2024-03-06T10:44:37.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:37.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:37.373802+0000) 2024-03-06T10:44:38.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:38.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:38.374088+0000) 2024-03-06T10:44:39.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:39.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:39.374359+0000) 2024-03-06T10:44:40.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:40.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:40.374687+0000) 2024-03-06T10:44:41.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:41.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:41.374971+0000) 2024-03-06T10:44:42.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:42.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:42.375231+0000) 2024-03-06T10:44:43.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:43.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:43.375492+0000) 2024-03-06T10:44:44.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:44.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:44.375740+0000) 2024-03-06T10:44:45.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:45.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:45.375995+0000) 2024-03-06T10:44:46.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:46.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:46.376234+0000) 2024-03-06T10:44:47.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:47.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:47.376454+0000) 2024-03-06T10:44:48.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:48.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:48.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:48.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:48.376765+0000) 2024-03-06T10:44:49.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:49.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:49.376966+0000) 2024-03-06T10:44:50.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:50.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:50.377226+0000) 2024-03-06T10:44:51.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:51.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:51.377484+0000) 2024-03-06T10:44:52.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:52.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:52.377768+0000) 2024-03-06T10:44:53.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:53.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:53.378007+0000) 2024-03-06T10:44:54.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:54.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:54.378277+0000) 2024-03-06T10:44:55.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:55.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:55.378520+0000) 2024-03-06T10:44:56.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:56.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:56.378822+0000) 2024-03-06T10:44:57.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:57.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:57.379129+0000) 2024-03-06T10:44:58.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:58.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:58.379394+0000) 2024-03-06T10:44:59.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:44:59.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:59.379676+0000) 2024-03-06T10:45:00.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:00.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:00.379945+0000) 2024-03-06T10:45:01.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:01.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:01.380204+0000) 2024-03-06T10:45:02.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:02.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:02.380468+0000) 2024-03-06T10:45:03.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:03.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:03.380747+0000) 2024-03-06T10:45:04.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:04.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:04.381021+0000) 2024-03-06T10:45:05.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:05.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:05.381312+0000) 2024-03-06T10:45:06.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:06.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:06.381555+0000) 2024-03-06T10:45:07.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:07.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:07.381799+0000) 2024-03-06T10:45:08.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:08.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:08.382061+0000) 2024-03-06T10:45:09.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:09.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:09.382283+0000) 2024-03-06T10:45:10.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:10.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:10.382518+0000) 2024-03-06T10:45:11.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:11.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:11.382801+0000) 2024-03-06T10:45:12.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:12.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:12.383251+0000) 2024-03-06T10:45:13.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:13.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:13.383525+0000) 2024-03-06T10:45:14.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:14.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:14.383787+0000) 2024-03-06T10:45:15.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:15.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:15.384064+0000) 2024-03-06T10:45:16.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:16.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:16.384329+0000) 2024-03-06T10:45:17.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:17.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:17.384594+0000) 2024-03-06T10:45:18.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:18.217 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:18.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:18.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:18.384918+0000) 2024-03-06T10:45:19.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:19.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:19.385214+0000) 2024-03-06T10:45:20.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:20.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:20.385441+0000) 2024-03-06T10:45:21.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:21.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:21.385709+0000) 2024-03-06T10:45:22.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:22.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:22.385898+0000) 2024-03-06T10:45:23.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:23.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:23.386185+0000) 2024-03-06T10:45:24.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:24.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:24.386444+0000) 2024-03-06T10:45:25.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:25.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:25.386742+0000) 2024-03-06T10:45:26.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:26.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:26.387005+0000) 2024-03-06T10:45:27.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:27.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:27.387267+0000) 2024-03-06T10:45:28.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:28.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:28.387491+0000) 2024-03-06T10:45:29.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:29.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:29.387750+0000) 2024-03-06T10:45:30.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:30.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:30.388014+0000) 2024-03-06T10:45:31.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:31.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:31.388247+0000) 2024-03-06T10:45:32.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:32.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:32.388499+0000) 2024-03-06T10:45:33.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:33.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:33.388738+0000) 2024-03-06T10:45:34.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:34.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:34.388998+0000) 2024-03-06T10:45:35.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:35.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:35.389176+0000) 2024-03-06T10:45:36.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:36.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:36.389431+0000) 2024-03-06T10:45:37.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:37.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:37.389727+0000) 2024-03-06T10:45:38.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:38.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:38.389994+0000) 2024-03-06T10:45:39.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:39.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:39.390247+0000) 2024-03-06T10:45:40.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:40.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:40.390495+0000) 2024-03-06T10:45:41.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:41.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:41.390747+0000) 2024-03-06T10:45:42.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:42.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:42.390982+0000) 2024-03-06T10:45:43.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:43.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:43.391182+0000) 2024-03-06T10:45:44.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:44.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:44.391373+0000) 2024-03-06T10:45:45.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:45.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:45.391566+0000) 2024-03-06T10:45:46.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:46.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:46.391833+0000) 2024-03-06T10:45:47.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:47.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:47.392092+0000) 2024-03-06T10:45:48.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:48.362 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:48.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:48.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:48.392283+0000) 2024-03-06T10:45:49.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:49.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:49.392525+0000) 2024-03-06T10:45:50.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:50.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:50.392741+0000) 2024-03-06T10:45:51.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:51.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:51.392978+0000) 2024-03-06T10:45:52.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:52.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:52.393239+0000) 2024-03-06T10:45:53.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:53.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:53.393500+0000) 2024-03-06T10:45:54.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:54.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:54.393760+0000) 2024-03-06T10:45:55.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:55.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:55.393992+0000) 2024-03-06T10:45:56.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:56.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:56.394248+0000) 2024-03-06T10:45:57.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:57.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:57.394505+0000) 2024-03-06T10:45:58.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:58.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:58.394776+0000) 2024-03-06T10:45:59.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:45:59.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:59.395028+0000) 2024-03-06T10:46:00.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:00.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:00.395289+0000) 2024-03-06T10:46:01.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:01.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:01.395531+0000) 2024-03-06T10:46:02.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:02.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:02.395764+0000) 2024-03-06T10:46:03.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:03.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:03.396007+0000) 2024-03-06T10:46:04.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:04.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:04.396264+0000) 2024-03-06T10:46:05.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:05.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:05.396518+0000) 2024-03-06T10:46:06.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:06.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:06.396770+0000) 2024-03-06T10:46:07.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:07.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:07.397029+0000) 2024-03-06T10:46:08.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:08.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:08.397300+0000) 2024-03-06T10:46:09.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:09.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:09.397550+0000) 2024-03-06T10:46:10.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:10.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:10.397812+0000) 2024-03-06T10:46:11.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:11.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:11.398036+0000) 2024-03-06T10:46:12.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:12.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:12.398271+0000) 2024-03-06T10:46:13.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:13.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:13.398510+0000) 2024-03-06T10:46:14.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:14.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:14.398755+0000) 2024-03-06T10:46:15.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:15.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:15.398957+0000) 2024-03-06T10:46:16.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:16.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:16.399230+0000) 2024-03-06T10:46:17.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:17.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:17.399488+0000) 2024-03-06T10:46:18.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:18.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:18.399753+0000) 2024-03-06T10:46:18.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:18.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:19.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:19.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:19.399992+0000) 2024-03-06T10:46:20.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:20.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:20.400235+0000) 2024-03-06T10:46:21.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:21.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:21.400469+0000) 2024-03-06T10:46:22.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:22.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:22.400816+0000) 2024-03-06T10:46:23.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:23.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:23.401054+0000) 2024-03-06T10:46:24.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:24.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:24.401246+0000) 2024-03-06T10:46:25.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:25.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:25.401497+0000) 2024-03-06T10:46:26.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:26.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:26.401761+0000) 2024-03-06T10:46:27.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:27.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:27.402028+0000) 2024-03-06T10:46:28.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:28.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:28.402294+0000) 2024-03-06T10:46:29.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:29.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:29.402519+0000) 2024-03-06T10:46:30.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:30.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:30.402780+0000) 2024-03-06T10:46:31.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:31.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:31.402984+0000) 2024-03-06T10:46:32.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:32.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:32.403257+0000) 2024-03-06T10:46:33.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:33.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:33.403516+0000) 2024-03-06T10:46:34.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:34.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:34.403771+0000) 2024-03-06T10:46:35.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:35.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:35.404019+0000) 2024-03-06T10:46:36.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:36.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:36.404276+0000) 2024-03-06T10:46:37.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:37.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:37.404523+0000) 2024-03-06T10:46:38.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:38.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:38.404800+0000) 2024-03-06T10:46:39.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:39.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:39.405026+0000) 2024-03-06T10:46:40.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:40.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:40.405273+0000) 2024-03-06T10:46:41.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:41.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:41.405528+0000) 2024-03-06T10:46:42.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:42.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:42.405764+0000) 2024-03-06T10:46:43.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:43.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:43.406011+0000) 2024-03-06T10:46:44.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:44.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:44.406263+0000) 2024-03-06T10:46:45.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:45.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:45.406503+0000) 2024-03-06T10:46:46.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:46.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:46.406751+0000) 2024-03-06T10:46:47.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:47.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:47.406992+0000) 2024-03-06T10:46:48.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:48.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:48.407242+0000) 2024-03-06T10:46:48.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:48.667 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:49.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:49.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:49.407477+0000) 2024-03-06T10:46:50.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:50.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:50.407743+0000) 2024-03-06T10:46:51.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:51.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:51.407996+0000) 2024-03-06T10:46:52.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:52.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:52.408256+0000) 2024-03-06T10:46:53.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:53.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:53.408518+0000) 2024-03-06T10:46:54.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:54.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:54.408787+0000) 2024-03-06T10:46:55.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:55.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:55.409017+0000) 2024-03-06T10:46:56.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:56.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:56.409267+0000) 2024-03-06T10:46:57.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:57.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:57.409536+0000) 2024-03-06T10:46:58.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:58.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:58.409812+0000) 2024-03-06T10:46:59.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:46:59.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:59.410055+0000) 2024-03-06T10:47:00.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:00.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:00.410317+0000) 2024-03-06T10:47:01.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:01.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:01.410573+0000) 2024-03-06T10:47:02.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:02.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:02.410820+0000) 2024-03-06T10:47:03.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:03.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:03.411053+0000) 2024-03-06T10:47:04.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:04.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:04.411348+0000) 2024-03-06T10:47:05.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:05.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:05.411586+0000) 2024-03-06T10:47:06.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:06.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:06.411822+0000) 2024-03-06T10:47:07.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:07.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:07.412022+0000) 2024-03-06T10:47:08.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:08.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:08.412265+0000) 2024-03-06T10:47:09.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:09.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:09.412495+0000) 2024-03-06T10:47:10.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:10.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:10.412767+0000) 2024-03-06T10:47:11.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:11.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:11.413018+0000) 2024-03-06T10:47:12.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:12.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:12.413288+0000) 2024-03-06T10:47:13.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:13.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:13.413547+0000) 2024-03-06T10:47:14.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:14.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:14.413819+0000) 2024-03-06T10:47:15.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:15.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:15.414058+0000) 2024-03-06T10:47:16.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:16.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:16.414319+0000) 2024-03-06T10:47:17.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:17.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:17.414584+0000) 2024-03-06T10:47:18.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:18.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:18.414809+0000) 2024-03-06T10:47:18.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:18.922 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:19.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:19.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:19.415075+0000) 2024-03-06T10:47:20.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:20.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:20.415344+0000) 2024-03-06T10:47:21.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:21.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:21.415604+0000) 2024-03-06T10:47:22.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:22.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:22.415899+0000) 2024-03-06T10:47:23.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:23.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:23.416155+0000) 2024-03-06T10:47:24.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:24.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:24.416418+0000) 2024-03-06T10:47:25.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:25.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:25.416725+0000) 2024-03-06T10:47:26.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:26.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:26.416986+0000) 2024-03-06T10:47:27.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:27.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:27.417237+0000) 2024-03-06T10:47:28.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:28.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:28.417482+0000) 2024-03-06T10:47:29.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:29.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:29.417742+0000) 2024-03-06T10:47:30.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:30.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:30.418016+0000) 2024-03-06T10:47:31.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:31.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:31.418263+0000) 2024-03-06T10:47:32.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:32.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:32.418524+0000) 2024-03-06T10:47:33.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:33.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:33.418783+0000) 2024-03-06T10:47:34.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:34.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:34.419047+0000) 2024-03-06T10:47:35.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:35.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:35.419312+0000) 2024-03-06T10:47:36.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:36.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:36.419565+0000) 2024-03-06T10:47:37.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:37.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:37.419831+0000) 2024-03-06T10:47:38.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:38.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:38.420089+0000) 2024-03-06T10:47:39.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:39.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:39.420348+0000) 2024-03-06T10:47:40.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:40.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:40.420604+0000) 2024-03-06T10:47:41.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:41.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:41.420876+0000) 2024-03-06T10:47:42.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:42.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:42.421072+0000) 2024-03-06T10:47:43.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:43.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:43.421326+0000) 2024-03-06T10:47:44.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:44.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:44.421587+0000) 2024-03-06T10:47:45.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:45.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:45.421864+0000) 2024-03-06T10:47:46.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:46.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:46.422122+0000) 2024-03-06T10:47:47.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:47.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:47.422399+0000) 2024-03-06T10:47:48.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:48.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:48.422681+0000) 2024-03-06T10:47:49.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:49.074 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:49.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:49.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:49.422955+0000) 2024-03-06T10:47:50.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:50.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:50.423206+0000) 2024-03-06T10:47:51.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:51.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:51.423490+0000) 2024-03-06T10:47:52.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:52.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:52.423837+0000) 2024-03-06T10:47:53.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:53.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:53.424097+0000) 2024-03-06T10:47:54.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:54.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:54.424505+0000) 2024-03-06T10:47:55.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:55.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:55.424791+0000) 2024-03-06T10:47:56.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:56.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:56.424988+0000) 2024-03-06T10:47:57.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:57.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:57.425259+0000) 2024-03-06T10:47:58.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:58.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:58.425490+0000) 2024-03-06T10:47:59.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:47:59.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:59.425755+0000) 2024-03-06T10:48:00.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:00.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:00.425981+0000) 2024-03-06T10:48:01.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:01.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:01.426191+0000) 2024-03-06T10:48:02.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:02.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:02.426447+0000) 2024-03-06T10:48:03.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:03.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:03.426745+0000) 2024-03-06T10:48:04.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:04.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:04.426998+0000) 2024-03-06T10:48:05.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:05.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:05.427236+0000) 2024-03-06T10:48:06.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:06.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:06.427476+0000) 2024-03-06T10:48:07.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:07.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:07.427806+0000) 2024-03-06T10:48:08.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:08.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:08.428038+0000) 2024-03-06T10:48:09.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:09.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:09.428276+0000) 2024-03-06T10:48:10.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:10.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:10.428544+0000) 2024-03-06T10:48:11.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:11.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:11.428807+0000) 2024-03-06T10:48:12.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:12.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:12.428998+0000) 2024-03-06T10:48:13.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:13.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:13.429255+0000) 2024-03-06T10:48:14.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:14.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:14.429501+0000) 2024-03-06T10:48:15.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:15.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:15.429759+0000) 2024-03-06T10:48:16.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:16.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:16.429954+0000) 2024-03-06T10:48:17.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:17.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:17.430153+0000) 2024-03-06T10:48:18.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:18.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:18.430408+0000) 2024-03-06T10:48:19.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:19.223 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:19.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:19.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:19.430716+0000) 2024-03-06T10:48:20.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:20.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:20.430988+0000) 2024-03-06T10:48:21.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:21.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:21.431244+0000) 2024-03-06T10:48:22.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:22.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:22.431507+0000) 2024-03-06T10:48:23.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:23.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:23.431781+0000) 2024-03-06T10:48:24.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:24.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:24.432029+0000) 2024-03-06T10:48:25.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:25.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:25.432276+0000) 2024-03-06T10:48:26.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:26.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:26.432534+0000) 2024-03-06T10:48:27.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:27.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:27.432778+0000) 2024-03-06T10:48:28.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:28.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:28.433029+0000) 2024-03-06T10:48:29.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:29.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:29.433295+0000) 2024-03-06T10:48:30.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:30.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:30.433587+0000) 2024-03-06T10:48:31.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:31.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:31.433908+0000) 2024-03-06T10:48:32.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:32.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:32.434206+0000) 2024-03-06T10:48:33.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:33.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:33.434452+0000) 2024-03-06T10:48:34.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:34.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:34.434759+0000) 2024-03-06T10:48:35.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:35.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:35.435028+0000) 2024-03-06T10:48:36.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:36.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:36.435291+0000) 2024-03-06T10:48:37.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:37.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:37.435535+0000) 2024-03-06T10:48:38.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:38.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:38.435751+0000) 2024-03-06T10:48:39.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:39.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:39.435952+0000) 2024-03-06T10:48:40.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:40.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:40.436311+0000) 2024-03-06T10:48:41.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:41.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:41.436537+0000) 2024-03-06T10:48:42.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:42.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:42.436784+0000) 2024-03-06T10:48:43.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:43.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:43.437018+0000) 2024-03-06T10:48:44.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:44.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:44.437268+0000) 2024-03-06T10:48:45.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:45.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:45.437533+0000) 2024-03-06T10:48:46.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:46.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:46.437808+0000) 2024-03-06T10:48:47.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:47.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:47.438013+0000) 2024-03-06T10:48:48.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:48.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:48.438259+0000) 2024-03-06T10:48:49.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:49.369 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:49.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:49.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:49.438499+0000) 2024-03-06T10:48:50.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:50.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:50.438765+0000) 2024-03-06T10:48:51.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:51.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:51.439020+0000) 2024-03-06T10:48:52.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:52.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:52.439215+0000) 2024-03-06T10:48:53.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:53.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:53.439465+0000) 2024-03-06T10:48:54.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:54.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:54.439778+0000) 2024-03-06T10:48:55.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:55.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:55.440050+0000) 2024-03-06T10:48:56.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:56.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:56.440302+0000) 2024-03-06T10:48:57.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:57.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:57.440547+0000) 2024-03-06T10:48:58.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:58.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:58.440796+0000) 2024-03-06T10:48:59.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:48:59.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:59.441063+0000) 2024-03-06T10:49:00.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:00.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:00.441330+0000) 2024-03-06T10:49:01.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:01.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:01.441582+0000) 2024-03-06T10:49:02.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:02.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:02.441844+0000) 2024-03-06T10:49:03.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:03.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:03.442097+0000) 2024-03-06T10:49:04.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:04.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:04.442349+0000) 2024-03-06T10:49:05.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:05.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:05.442611+0000) 2024-03-06T10:49:06.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:06.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:06.442890+0000) 2024-03-06T10:49:07.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:07.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:07.443081+0000) 2024-03-06T10:49:08.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:08.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:08.443320+0000) 2024-03-06T10:49:09.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:09.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:09.443568+0000) 2024-03-06T10:49:10.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:10.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:10.443837+0000) 2024-03-06T10:49:11.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:11.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:11.444025+0000) 2024-03-06T10:49:12.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:12.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:12.444294+0000) 2024-03-06T10:49:13.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:13.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:13.444544+0000) 2024-03-06T10:49:14.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:14.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:14.444810+0000) 2024-03-06T10:49:15.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:15.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:15.445073+0000) 2024-03-06T10:49:16.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:16.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:16.445317+0000) 2024-03-06T10:49:17.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:17.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:17.445571+0000) 2024-03-06T10:49:18.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:18.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:18.445814+0000) 2024-03-06T10:49:19.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:19.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:19.446079+0000) 2024-03-06T10:49:19.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:19.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:20.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:20.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:20.446286+0000) 2024-03-06T10:49:21.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:21.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:21.446550+0000) 2024-03-06T10:49:22.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:22.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:22.446809+0000) 2024-03-06T10:49:23.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:23.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:23.447072+0000) 2024-03-06T10:49:24.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:24.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:24.447338+0000) 2024-03-06T10:49:25.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:25.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:25.447594+0000) 2024-03-06T10:49:26.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:26.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:26.447867+0000) 2024-03-06T10:49:27.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:27.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:27.448122+0000) 2024-03-06T10:49:28.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:28.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:28.448392+0000) 2024-03-06T10:49:29.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:29.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:29.448667+0000) 2024-03-06T10:49:30.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:30.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:30.448929+0000) 2024-03-06T10:49:31.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:31.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:31.449189+0000) 2024-03-06T10:49:32.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:32.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:32.449431+0000) 2024-03-06T10:49:33.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:33.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:33.449723+0000) 2024-03-06T10:49:34.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:34.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:34.449937+0000) 2024-03-06T10:49:35.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:35.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:35.450202+0000) 2024-03-06T10:49:36.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:36.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:36.450448+0000) 2024-03-06T10:49:37.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:37.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:37.450743+0000) 2024-03-06T10:49:38.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:38.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:38.450998+0000) 2024-03-06T10:49:39.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:39.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:39.451192+0000) 2024-03-06T10:49:40.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:40.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:40.451384+0000) 2024-03-06T10:49:41.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:41.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:41.451660+0000) 2024-03-06T10:49:42.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:42.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:42.451906+0000) 2024-03-06T10:49:43.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:43.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:43.452173+0000) 2024-03-06T10:49:44.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:44.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:44.452442+0000) 2024-03-06T10:49:45.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:45.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:45.452726+0000) 2024-03-06T10:49:46.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:46.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:46.452988+0000) 2024-03-06T10:49:47.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:47.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:47.453240+0000) 2024-03-06T10:49:48.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:48.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:48.453497+0000) 2024-03-06T10:49:49.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:49.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:49.453756+0000) 2024-03-06T10:49:49.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:49.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:50.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:50.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:50.454018+0000) 2024-03-06T10:49:51.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:51.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:51.454284+0000) 2024-03-06T10:49:52.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:52.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:52.454549+0000) 2024-03-06T10:49:53.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:53.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:53.454809+0000) 2024-03-06T10:49:54.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:54.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:54.455074+0000) 2024-03-06T10:49:55.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:55.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:55.455271+0000) 2024-03-06T10:49:56.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:56.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:56.455530+0000) 2024-03-06T10:49:57.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:57.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:57.455801+0000) 2024-03-06T10:49:58.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:58.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:58.456062+0000) 2024-03-06T10:49:59.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:49:59.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:59.456335+0000) 2024-03-06T10:50:00.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:00.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:00.456601+0000) 2024-03-06T10:50:01.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:01.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:01.456895+0000) 2024-03-06T10:50:02.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:02.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:02.457138+0000) 2024-03-06T10:50:03.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:03.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:03.457384+0000) 2024-03-06T10:50:04.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:04.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:04.457578+0000) 2024-03-06T10:50:05.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:05.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:05.457835+0000) 2024-03-06T10:50:06.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:06.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:06.458087+0000) 2024-03-06T10:50:07.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:07.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:07.458301+0000) 2024-03-06T10:50:08.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:08.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:08.458534+0000) 2024-03-06T10:50:09.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:09.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:09.458805+0000) 2024-03-06T10:50:10.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:10.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:10.459021+0000) 2024-03-06T10:50:11.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:11.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:11.459214+0000) 2024-03-06T10:50:12.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:12.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:12.459489+0000) 2024-03-06T10:50:13.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:13.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:13.459737+0000) 2024-03-06T10:50:14.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:14.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:14.460022+0000) 2024-03-06T10:50:15.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:15.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:15.460292+0000) 2024-03-06T10:50:16.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:16.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:16.460577+0000) 2024-03-06T10:50:17.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:17.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:17.460905+0000) 2024-03-06T10:50:18.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:18.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:18.461180+0000) 2024-03-06T10:50:19.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:19.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:19.461463+0000) 2024-03-06T10:50:19.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:19.852 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:20.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:20.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:20.461767+0000) 2024-03-06T10:50:21.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:21.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:21.462038+0000) 2024-03-06T10:50:22.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:22.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:22.462317+0000) 2024-03-06T10:50:23.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:23.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:23.462570+0000) 2024-03-06T10:50:24.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:24.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:24.462857+0000) 2024-03-06T10:50:25.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:25.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:25.463106+0000) 2024-03-06T10:50:26.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:26.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:26.463373+0000) 2024-03-06T10:50:27.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:27.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:27.463740+0000) 2024-03-06T10:50:28.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:28.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:28.463992+0000) 2024-03-06T10:50:29.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:29.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:29.464281+0000) 2024-03-06T10:50:30.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:30.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:30.464537+0000) 2024-03-06T10:50:31.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:31.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:31.464771+0000) 2024-03-06T10:50:32.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:32.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:32.465031+0000) 2024-03-06T10:50:33.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:33.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:33.465287+0000) 2024-03-06T10:50:34.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:34.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:34.465538+0000) 2024-03-06T10:50:35.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:35.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:35.465749+0000) 2024-03-06T10:50:36.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:36.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:36.465996+0000) 2024-03-06T10:50:37.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:37.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:37.466255+0000) 2024-03-06T10:50:38.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:38.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:38.466516+0000) 2024-03-06T10:50:39.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:39.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:39.466777+0000) 2024-03-06T10:50:40.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:40.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:40.467023+0000) 2024-03-06T10:50:41.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:41.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:41.467302+0000) 2024-03-06T10:50:42.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:42.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:42.467571+0000) 2024-03-06T10:50:43.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:43.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:43.467822+0000) 2024-03-06T10:50:44.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:44.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:44.468121+0000) 2024-03-06T10:50:45.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:45.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:45.468436+0000) 2024-03-06T10:50:46.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:46.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:46.468797+0000) 2024-03-06T10:50:47.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:47.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:47.469112+0000) 2024-03-06T10:50:48.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:48.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:48.469318+0000) 2024-03-06T10:50:49.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:49.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:49.469511+0000) 2024-03-06T10:50:50.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:50.008 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:50.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:50.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:50.469760+0000) 2024-03-06T10:50:51.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:51.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:51.470009+0000) 2024-03-06T10:50:52.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:52.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:52.470267+0000) 2024-03-06T10:50:53.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:53.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:53.470492+0000) 2024-03-06T10:50:54.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:54.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:54.470751+0000) 2024-03-06T10:50:55.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:55.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:55.471003+0000) 2024-03-06T10:50:56.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:56.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:56.471268+0000) 2024-03-06T10:50:57.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:57.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:57.471527+0000) 2024-03-06T10:50:58.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:58.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:58.471778+0000) 2024-03-06T10:50:59.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:50:59.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:59.472036+0000) 2024-03-06T10:51:00.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:00.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:00.472274+0000) 2024-03-06T10:51:01.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:01.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:01.472525+0000) 2024-03-06T10:51:02.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:02.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:02.472756+0000) 2024-03-06T10:51:03.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:03.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:03.472998+0000) 2024-03-06T10:51:04.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:04.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:04.473257+0000) 2024-03-06T10:51:05.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:05.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:05.473504+0000) 2024-03-06T10:51:06.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:06.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:06.473766+0000) 2024-03-06T10:51:07.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:07.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:07.474044+0000) 2024-03-06T10:51:08.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:08.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:08.474319+0000) 2024-03-06T10:51:09.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:09.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:09.474584+0000) 2024-03-06T10:51:10.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:10.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:10.474910+0000) 2024-03-06T10:51:11.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:11.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:11.475155+0000) 2024-03-06T10:51:12.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:12.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:12.475439+0000) 2024-03-06T10:51:13.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:13.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:13.475764+0000) 2024-03-06T10:51:14.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:14.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:14.476048+0000) 2024-03-06T10:51:15.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:15.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:15.476319+0000) 2024-03-06T10:51:16.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:16.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:16.476588+0000) 2024-03-06T10:51:17.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:17.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:17.476905+0000) 2024-03-06T10:51:18.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:18.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:18.477113+0000) 2024-03-06T10:51:19.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:19.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:19.477379+0000) 2024-03-06T10:51:20.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:20.139 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:20.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:20.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:20.477686+0000) 2024-03-06T10:51:21.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:21.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:21.477947+0000) 2024-03-06T10:51:22.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:22.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:22.478213+0000) 2024-03-06T10:51:23.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:23.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:23.478450+0000) 2024-03-06T10:51:24.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:24.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:24.478696+0000) 2024-03-06T10:51:25.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:25.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:25.478957+0000) 2024-03-06T10:51:26.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:26.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:26.479249+0000) 2024-03-06T10:51:27.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:27.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:27.479513+0000) 2024-03-06T10:51:28.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:28.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:28.479779+0000) 2024-03-06T10:51:29.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:29.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:29.480040+0000) 2024-03-06T10:51:30.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:30.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:30.480283+0000) 2024-03-06T10:51:31.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:31.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:31.480522+0000) 2024-03-06T10:51:32.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:32.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:32.480781+0000) 2024-03-06T10:51:33.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:33.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:33.481024+0000) 2024-03-06T10:51:34.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:34.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:34.481286+0000) 2024-03-06T10:51:35.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:35.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:35.481536+0000) 2024-03-06T10:51:36.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:36.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:36.481803+0000) 2024-03-06T10:51:37.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:37.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:37.482086+0000) 2024-03-06T10:51:38.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:38.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:38.482335+0000) 2024-03-06T10:51:39.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:39.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:39.482585+0000) 2024-03-06T10:51:40.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:40.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:40.482822+0000) 2024-03-06T10:51:41.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:41.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:41.483055+0000) 2024-03-06T10:51:42.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:42.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:42.483260+0000) 2024-03-06T10:51:43.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:43.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:43.483511+0000) 2024-03-06T10:51:44.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:44.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:44.483779+0000) 2024-03-06T10:51:45.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:45.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:45.483975+0000) 2024-03-06T10:51:46.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:46.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:46.484224+0000) 2024-03-06T10:51:47.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:47.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:47.484497+0000) 2024-03-06T10:51:48.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:48.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:48.484757+0000) 2024-03-06T10:51:49.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:49.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:49.485023+0000) 2024-03-06T10:51:50.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:50.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:50.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:50.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:50.485301+0000) 2024-03-06T10:51:51.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:51.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:51.485622+0000) 2024-03-06T10:51:52.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:52.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:52.485999+0000) 2024-03-06T10:51:53.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:53.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:53.486279+0000) 2024-03-06T10:51:54.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:54.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:54.486524+0000) 2024-03-06T10:51:55.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:55.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:55.486789+0000) 2024-03-06T10:51:56.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:56.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:56.487038+0000) 2024-03-06T10:51:57.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:57.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:57.487310+0000) 2024-03-06T10:51:58.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:58.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:58.487569+0000) 2024-03-06T10:51:59.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:51:59.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:59.487815+0000) 2024-03-06T10:52:00.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:00.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:00.488060+0000) 2024-03-06T10:52:01.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:01.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:01.488315+0000) 2024-03-06T10:52:02.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:02.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:02.488576+0000) 2024-03-06T10:52:03.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:03.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:03.488889+0000) 2024-03-06T10:52:04.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:04.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:04.489087+0000) 2024-03-06T10:52:05.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:05.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:05.489331+0000) 2024-03-06T10:52:06.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:06.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:06.489600+0000) 2024-03-06T10:52:07.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:07.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:07.489913+0000) 2024-03-06T10:52:08.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:08.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:08.490119+0000) 2024-03-06T10:52:09.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:09.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:09.490312+0000) 2024-03-06T10:52:10.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:10.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:10.490574+0000) 2024-03-06T10:52:11.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:11.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:11.490822+0000) 2024-03-06T10:52:12.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:12.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:12.491059+0000) 2024-03-06T10:52:13.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:13.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:13.491303+0000) 2024-03-06T10:52:14.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:14.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:14.491564+0000) 2024-03-06T10:52:15.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:15.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:15.491828+0000) 2024-03-06T10:52:16.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:16.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:16.492095+0000) 2024-03-06T10:52:17.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:17.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:17.492363+0000) 2024-03-06T10:52:18.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:18.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:18.492658+0000) 2024-03-06T10:52:19.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:19.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:19.492941+0000) 2024-03-06T10:52:20.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:20.425 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:20.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:20.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:20.493293+0000) 2024-03-06T10:52:21.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:21.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:21.493566+0000) 2024-03-06T10:52:22.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:22.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:22.493829+0000) 2024-03-06T10:52:23.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:23.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:23.494105+0000) 2024-03-06T10:52:24.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:24.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:24.494386+0000) 2024-03-06T10:52:25.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:25.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:25.494685+0000) 2024-03-06T10:52:26.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:26.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:26.494970+0000) 2024-03-06T10:52:27.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:27.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:27.495289+0000) 2024-03-06T10:52:28.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:28.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:28.495587+0000) 2024-03-06T10:52:29.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:29.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:29.495926+0000) 2024-03-06T10:52:30.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:30.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:30.496230+0000) 2024-03-06T10:52:31.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:31.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:31.496537+0000) 2024-03-06T10:52:32.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:32.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:32.496855+0000) 2024-03-06T10:52:33.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:33.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:33.497096+0000) 2024-03-06T10:52:34.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:34.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:34.497419+0000) 2024-03-06T10:52:35.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:35.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:35.497755+0000) 2024-03-06T10:52:36.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:36.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:36.498063+0000) 2024-03-06T10:52:37.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:37.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:37.498317+0000) 2024-03-06T10:52:38.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:38.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:38.498615+0000) 2024-03-06T10:52:39.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:39.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:39.498932+0000) 2024-03-06T10:52:40.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:40.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:40.499199+0000) 2024-03-06T10:52:41.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:41.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:41.499389+0000) 2024-03-06T10:52:42.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:42.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:42.499587+0000) 2024-03-06T10:52:43.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:43.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:43.499750+0000) 2024-03-06T10:52:44.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:44.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:44.499970+0000) 2024-03-06T10:52:45.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:45.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:45.500167+0000) 2024-03-06T10:52:46.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:46.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:46.500386+0000) 2024-03-06T10:52:47.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:47.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:47.500696+0000) 2024-03-06T10:52:48.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:48.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:48.500954+0000) 2024-03-06T10:52:49.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:49.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:49.501214+0000) 2024-03-06T10:52:50.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:50.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:50.501495+0000) 2024-03-06T10:52:50.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:50.572 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:51.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:51.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:51.501818+0000) 2024-03-06T10:52:52.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:52.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:52.502074+0000) 2024-03-06T10:52:53.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:53.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:53.502334+0000) 2024-03-06T10:52:54.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:54.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:54.502584+0000) 2024-03-06T10:52:55.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:55.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:55.502883+0000) 2024-03-06T10:52:56.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:56.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:56.503169+0000) 2024-03-06T10:52:57.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:57.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:57.503441+0000) 2024-03-06T10:52:58.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:58.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:58.503738+0000) 2024-03-06T10:52:59.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:52:59.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:59.504005+0000) 2024-03-06T10:53:00.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:00.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:00.504287+0000) 2024-03-06T10:53:01.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:01.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:01.504586+0000) 2024-03-06T10:53:02.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:02.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:02.504880+0000) 2024-03-06T10:53:03.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:03.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:03.505117+0000) 2024-03-06T10:53:04.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:04.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:04.505385+0000) 2024-03-06T10:53:05.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:05.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:05.505670+0000) 2024-03-06T10:53:06.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:06.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:06.505957+0000) 2024-03-06T10:53:07.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:07.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:07.506247+0000) 2024-03-06T10:53:08.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:08.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:08.506504+0000) 2024-03-06T10:53:09.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:09.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:09.506753+0000) 2024-03-06T10:53:10.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:10.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:10.507039+0000) 2024-03-06T10:53:11.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:11.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:11.507322+0000) 2024-03-06T10:53:12.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:12.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:12.507592+0000) 2024-03-06T10:53:13.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:13.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:13.507893+0000) 2024-03-06T10:53:14.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:14.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:14.508181+0000) 2024-03-06T10:53:15.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:15.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:15.508454+0000) 2024-03-06T10:53:16.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:16.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:16.508763+0000) 2024-03-06T10:53:17.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:17.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:17.509051+0000) 2024-03-06T10:53:18.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:18.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:18.509335+0000) 2024-03-06T10:53:19.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:19.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:19.509606+0000) 2024-03-06T10:53:20.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:20.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:20.509919+0000) 2024-03-06T10:53:20.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:20.712 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:21.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:21.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:21.510166+0000) 2024-03-06T10:53:22.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:22.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:22.510356+0000) 2024-03-06T10:53:23.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:23.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:23.510613+0000) 2024-03-06T10:53:24.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:24.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:24.510903+0000) 2024-03-06T10:53:25.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:25.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:25.511151+0000) 2024-03-06T10:53:26.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:26.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:26.511417+0000) 2024-03-06T10:53:27.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:27.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:27.511720+0000) 2024-03-06T10:53:28.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:28.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:28.511994+0000) 2024-03-06T10:53:29.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:29.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:29.512224+0000) 2024-03-06T10:53:30.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:30.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:30.512490+0000) 2024-03-06T10:53:31.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:31.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:31.512753+0000) 2024-03-06T10:53:32.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:32.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:32.513014+0000) 2024-03-06T10:53:33.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:33.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:33.513268+0000) 2024-03-06T10:53:34.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:34.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:34.513533+0000) 2024-03-06T10:53:35.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:35.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:35.513796+0000) 2024-03-06T10:53:36.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:36.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:36.514091+0000) 2024-03-06T10:53:37.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:37.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:37.514372+0000) 2024-03-06T10:53:38.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:38.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:38.514662+0000) 2024-03-06T10:53:39.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:39.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:39.514855+0000) 2024-03-06T10:53:40.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:40.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:40.515119+0000) 2024-03-06T10:53:41.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:41.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:41.515312+0000) 2024-03-06T10:53:42.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:42.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:42.515577+0000) 2024-03-06T10:53:43.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:43.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:43.515805+0000) 2024-03-06T10:53:44.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:44.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:44.516053+0000) 2024-03-06T10:53:45.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:45.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:45.516334+0000) 2024-03-06T10:53:46.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:46.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:46.516625+0000) 2024-03-06T10:53:47.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:47.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:47.516917+0000) 2024-03-06T10:53:48.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:48.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:48.517178+0000) 2024-03-06T10:53:49.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:49.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:49.517369+0000) 2024-03-06T10:53:50.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:50.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:50.517723+0000) 2024-03-06T10:53:50.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:50.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:51.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:51.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:51.517937+0000) 2024-03-06T10:53:52.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:52.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:52.518206+0000) 2024-03-06T10:53:53.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:53.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:53.518403+0000) 2024-03-06T10:53:54.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:54.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:54.518708+0000) 2024-03-06T10:53:55.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:55.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:55.518955+0000) 2024-03-06T10:53:56.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:56.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:56.519237+0000) 2024-03-06T10:53:57.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:57.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:57.519503+0000) 2024-03-06T10:53:58.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:58.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:58.519775+0000) 2024-03-06T10:53:59.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:53:59.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:59.520010+0000) 2024-03-06T10:54:00.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:00.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:00.520278+0000) 2024-03-06T10:54:01.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:01.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:01.520554+0000) 2024-03-06T10:54:02.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:02.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:02.520832+0000) 2024-03-06T10:54:03.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:03.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:03.521041+0000) 2024-03-06T10:54:04.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:04.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:04.521302+0000) 2024-03-06T10:54:05.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:05.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:05.521549+0000) 2024-03-06T10:54:06.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:06.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:06.521812+0000) 2024-03-06T10:54:07.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:07.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:07.522083+0000) 2024-03-06T10:54:08.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:08.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:08.522335+0000) 2024-03-06T10:54:09.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:09.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:09.522621+0000) 2024-03-06T10:54:10.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:10.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:10.522898+0000) 2024-03-06T10:54:11.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:11.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:11.523166+0000) 2024-03-06T10:54:12.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:12.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:12.523409+0000) 2024-03-06T10:54:13.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:13.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:13.523613+0000) 2024-03-06T10:54:14.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:14.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:14.523909+0000) 2024-03-06T10:54:15.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:15.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:15.524175+0000) 2024-03-06T10:54:16.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:16.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:16.524440+0000) 2024-03-06T10:54:17.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:17.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:17.524745+0000) 2024-03-06T10:54:18.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:18.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:18.525005+0000) 2024-03-06T10:54:19.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:19.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:19.525250+0000) 2024-03-06T10:54:20.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:20.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:20.525506+0000) 2024-03-06T10:54:21.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:21.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:21.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:21.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:21.525758+0000) 2024-03-06T10:54:22.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:22.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:22.526026+0000) 2024-03-06T10:54:23.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:23.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:23.526229+0000) 2024-03-06T10:54:24.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:24.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:24.526484+0000) 2024-03-06T10:54:25.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:25.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:25.526744+0000) 2024-03-06T10:54:26.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:26.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:26.527015+0000) 2024-03-06T10:54:27.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:27.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:27.527283+0000) 2024-03-06T10:54:28.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:28.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:28.527527+0000) 2024-03-06T10:54:29.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:29.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:29.527781+0000) 2024-03-06T10:54:30.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:30.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:30.528050+0000) 2024-03-06T10:54:31.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:31.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:31.528298+0000) 2024-03-06T10:54:32.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:32.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:32.528560+0000) 2024-03-06T10:54:33.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:33.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:33.528808+0000) 2024-03-06T10:54:34.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:34.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:34.529073+0000) 2024-03-06T10:54:35.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:35.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:35.529334+0000) 2024-03-06T10:54:36.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:36.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:36.529595+0000) 2024-03-06T10:54:37.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:37.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:37.529899+0000) 2024-03-06T10:54:38.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:38.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:38.530191+0000) 2024-03-06T10:54:39.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:39.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:39.530435+0000) 2024-03-06T10:54:40.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:40.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:40.530744+0000) 2024-03-06T10:54:41.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:41.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:41.530975+0000) 2024-03-06T10:54:42.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:42.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:42.531243+0000) 2024-03-06T10:54:43.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:43.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:43.531495+0000) 2024-03-06T10:54:44.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:44.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:44.531741+0000) 2024-03-06T10:54:45.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:45.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:45.531950+0000) 2024-03-06T10:54:46.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:46.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:46.532214+0000) 2024-03-06T10:54:47.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:47.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:47.532468+0000) 2024-03-06T10:54:48.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:48.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:48.532762+0000) 2024-03-06T10:54:49.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:49.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:49.533015+0000) 2024-03-06T10:54:50.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:50.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:50.533226+0000) 2024-03-06T10:54:51.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:51.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:51.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:51.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:51.533472+0000) 2024-03-06T10:54:52.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:52.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:52.533754+0000) 2024-03-06T10:54:53.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:53.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:53.533969+0000) 2024-03-06T10:54:54.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:54.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:54.534242+0000) 2024-03-06T10:54:55.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:55.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:55.534493+0000) 2024-03-06T10:54:56.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:56.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:56.534761+0000) 2024-03-06T10:54:57.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:57.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:57.535035+0000) 2024-03-06T10:54:58.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:58.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:58.535226+0000) 2024-03-06T10:54:59.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:54:59.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:59.535458+0000) 2024-03-06T10:55:00.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:00.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:00.535749+0000) 2024-03-06T10:55:01.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:01.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:01.535980+0000) 2024-03-06T10:55:02.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:02.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:02.536246+0000) 2024-03-06T10:55:03.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:03.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:03.536496+0000) 2024-03-06T10:55:04.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:04.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:04.536755+0000) 2024-03-06T10:55:05.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:05.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:05.536994+0000) 2024-03-06T10:55:06.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:06.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:06.537189+0000) 2024-03-06T10:55:07.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:07.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:07.537385+0000) 2024-03-06T10:55:08.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:08.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:08.537664+0000) 2024-03-06T10:55:09.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:09.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:09.537874+0000) 2024-03-06T10:55:10.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:10.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:10.538145+0000) 2024-03-06T10:55:11.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:11.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:11.538385+0000) 2024-03-06T10:55:12.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:12.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:12.538678+0000) 2024-03-06T10:55:13.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:13.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:13.538933+0000) 2024-03-06T10:55:14.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:14.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:14.539191+0000) 2024-03-06T10:55:15.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:15.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:15.539435+0000) 2024-03-06T10:55:16.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:16.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:16.539664+0000) 2024-03-06T10:55:17.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:17.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:17.539909+0000) 2024-03-06T10:55:18.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:18.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:18.540148+0000) 2024-03-06T10:55:19.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:19.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:19.540380+0000) 2024-03-06T10:55:20.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:20.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:20.540679+0000) 2024-03-06T10:55:21.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:21.307 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:21.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:21.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:21.540960+0000) 2024-03-06T10:55:22.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:22.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:22.541149+0000) 2024-03-06T10:55:23.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:23.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:23.541381+0000) 2024-03-06T10:55:24.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:24.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:24.541664+0000) 2024-03-06T10:55:25.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:25.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:25.541936+0000) 2024-03-06T10:55:26.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:26.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:26.542192+0000) 2024-03-06T10:55:27.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:27.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:27.542456+0000) 2024-03-06T10:55:28.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:28.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:28.542733+0000) 2024-03-06T10:55:29.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:29.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:29.542996+0000) 2024-03-06T10:55:30.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:30.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:30.543263+0000) 2024-03-06T10:55:31.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:31.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:31.543520+0000) 2024-03-06T10:55:32.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:32.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:32.543798+0000) 2024-03-06T10:55:33.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:33.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:33.543987+0000) 2024-03-06T10:55:34.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:34.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:34.544252+0000) 2024-03-06T10:55:35.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:35.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:35.544529+0000) 2024-03-06T10:55:36.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:36.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:36.544802+0000) 2024-03-06T10:55:37.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:37.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:37.545065+0000) 2024-03-06T10:55:38.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:38.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:38.545325+0000) 2024-03-06T10:55:39.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:39.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:39.545509+0000) 2024-03-06T10:55:40.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:40.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:40.545785+0000) 2024-03-06T10:55:41.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:41.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:41.546039+0000) 2024-03-06T10:55:42.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:42.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:42.546280+0000) 2024-03-06T10:55:43.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:43.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:43.546536+0000) 2024-03-06T10:55:44.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:44.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:44.546804+0000) 2024-03-06T10:55:45.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:45.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:45.547069+0000) 2024-03-06T10:55:46.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:46.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:46.547333+0000) 2024-03-06T10:55:47.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:47.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:47.547584+0000) 2024-03-06T10:55:48.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:48.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:48.547842+0000) 2024-03-06T10:55:49.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:49.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:49.548090+0000) 2024-03-06T10:55:50.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:50.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:50.548353+0000) 2024-03-06T10:55:51.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:51.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:51.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:51.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:51.548608+0000) 2024-03-06T10:55:52.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:52.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:52.548911+0000) 2024-03-06T10:55:53.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:53.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:53.549196+0000) 2024-03-06T10:55:54.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:54.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:54.549508+0000) 2024-03-06T10:55:55.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:55.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:55.549806+0000) 2024-03-06T10:55:56.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:56.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:56.550127+0000) 2024-03-06T10:55:57.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:57.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:57.550428+0000) 2024-03-06T10:55:58.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:58.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:58.550785+0000) 2024-03-06T10:55:59.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:55:59.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:59.551097+0000) 2024-03-06T10:56:00.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:00.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:00.551415+0000) 2024-03-06T10:56:01.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:01.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:01.551762+0000) 2024-03-06T10:56:02.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:02.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:02.552068+0000) 2024-03-06T10:56:03.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:03.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:03.552377+0000) 2024-03-06T10:56:04.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:04.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:04.552735+0000) 2024-03-06T10:56:05.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:05.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:05.553038+0000) 2024-03-06T10:56:06.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:06.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:06.553361+0000) 2024-03-06T10:56:07.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:07.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:07.553737+0000) 2024-03-06T10:56:08.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:08.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:08.554110+0000) 2024-03-06T10:56:09.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:09.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:09.554420+0000) 2024-03-06T10:56:10.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:10.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:10.554774+0000) 2024-03-06T10:56:11.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:11.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:11.555070+0000) 2024-03-06T10:56:12.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:12.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:12.555378+0000) 2024-03-06T10:56:13.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:13.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:13.555701+0000) 2024-03-06T10:56:14.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:14.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:14.556021+0000) 2024-03-06T10:56:15.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:15.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:15.556281+0000) 2024-03-06T10:56:16.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:16.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:16.556599+0000) 2024-03-06T10:56:17.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:17.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:17.556970+0000) 2024-03-06T10:56:18.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:18.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:18.557274+0000) 2024-03-06T10:56:19.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:19.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:19.557560+0000) 2024-03-06T10:56:20.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:20.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:20.557889+0000) 2024-03-06T10:56:21.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:21.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:21.558203+0000) 2024-03-06T10:56:21.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:21.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:22.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:22.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:22.558524+0000) 2024-03-06T10:56:23.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:23.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:23.558830+0000) 2024-03-06T10:56:24.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:24.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:24.559143+0000) 2024-03-06T10:56:25.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:25.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:25.559383+0000) 2024-03-06T10:56:26.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:26.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:26.559740+0000) 2024-03-06T10:56:27.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:27.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:27.560058+0000) 2024-03-06T10:56:28.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:28.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:28.560387+0000) 2024-03-06T10:56:29.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:29.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:29.560618+0000) 2024-03-06T10:56:30.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:30.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:30.560925+0000) 2024-03-06T10:56:31.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:31.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:31.561230+0000) 2024-03-06T10:56:32.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:32.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:32.561476+0000) 2024-03-06T10:56:33.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:33.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:33.561796+0000) 2024-03-06T10:56:34.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:34.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:34.562112+0000) 2024-03-06T10:56:35.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:35.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:35.562351+0000) 2024-03-06T10:56:36.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:36.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:36.562691+0000) 2024-03-06T10:56:37.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:37.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:37.562959+0000) 2024-03-06T10:56:38.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:38.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:38.563272+0000) 2024-03-06T10:56:39.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:39.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:39.563594+0000) 2024-03-06T10:56:40.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:40.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:40.563972+0000) 2024-03-06T10:56:41.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:41.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:41.564286+0000) 2024-03-06T10:56:42.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:42.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:42.564595+0000) 2024-03-06T10:56:43.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:43.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:43.564960+0000) 2024-03-06T10:56:44.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:44.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:44.565272+0000) 2024-03-06T10:56:45.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:45.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:45.565573+0000) 2024-03-06T10:56:46.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:46.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:46.565868+0000) 2024-03-06T10:56:47.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:47.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:47.566195+0000) 2024-03-06T10:56:48.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:48.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:48.566518+0000) 2024-03-06T10:56:49.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:49.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:49.566816+0000) 2024-03-06T10:56:50.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:50.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:50.567128+0000) 2024-03-06T10:56:51.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:51.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:51.567418+0000) 2024-03-06T10:56:51.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:51.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:52.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:52.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:52.567780+0000) 2024-03-06T10:56:53.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:53.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:53.568086+0000) 2024-03-06T10:56:54.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:54.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:54.568396+0000) 2024-03-06T10:56:55.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:55.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:55.568751+0000) 2024-03-06T10:56:56.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:56.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:56.569058+0000) 2024-03-06T10:56:57.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:57.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:57.569390+0000) 2024-03-06T10:56:58.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:58.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:58.569744+0000) 2024-03-06T10:56:59.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:56:59.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:59.569972+0000) 2024-03-06T10:57:00.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:00.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:00.570214+0000) 2024-03-06T10:57:01.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:01.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:01.570464+0000) 2024-03-06T10:57:02.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:02.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:02.570765+0000) 2024-03-06T10:57:03.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:03.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:03.570964+0000) 2024-03-06T10:57:04.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:04.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:04.571227+0000) 2024-03-06T10:57:05.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:05.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:05.571435+0000) 2024-03-06T10:57:06.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:06.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:06.571767+0000) 2024-03-06T10:57:07.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:07.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:07.572049+0000) 2024-03-06T10:57:08.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:08.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:08.572305+0000) 2024-03-06T10:57:09.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:09.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:09.572554+0000) 2024-03-06T10:57:10.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:10.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:10.572811+0000) 2024-03-06T10:57:11.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:11.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:11.573055+0000) 2024-03-06T10:57:12.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:12.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:12.573328+0000) 2024-03-06T10:57:13.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:13.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:13.573589+0000) 2024-03-06T10:57:14.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:14.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:14.573854+0000) 2024-03-06T10:57:15.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:15.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:15.574108+0000) 2024-03-06T10:57:16.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:16.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:16.574373+0000) 2024-03-06T10:57:17.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:17.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:17.574686+0000) 2024-03-06T10:57:18.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:18.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:18.574937+0000) 2024-03-06T10:57:19.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:19.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:19.575135+0000) 2024-03-06T10:57:20.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:20.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:20.575363+0000) 2024-03-06T10:57:21.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:21.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:21.575671+0000) 2024-03-06T10:57:21.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:21.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:22.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:22.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:22.575988+0000) 2024-03-06T10:57:23.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:23.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:23.576298+0000) 2024-03-06T10:57:24.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:24.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:24.576566+0000) 2024-03-06T10:57:25.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:25.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:25.576871+0000) 2024-03-06T10:57:26.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:26.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:26.577169+0000) 2024-03-06T10:57:27.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:27.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:27.577486+0000) 2024-03-06T10:57:28.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:28.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:28.577777+0000) 2024-03-06T10:57:29.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:29.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:29.578072+0000) 2024-03-06T10:57:30.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:30.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:30.578382+0000) 2024-03-06T10:57:31.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:31.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:31.578732+0000) 2024-03-06T10:57:32.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:32.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:32.579057+0000) 2024-03-06T10:57:33.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:33.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:33.579353+0000) 2024-03-06T10:57:34.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:34.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:34.579717+0000) 2024-03-06T10:57:35.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:35.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:35.580015+0000) 2024-03-06T10:57:36.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:36.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:36.580326+0000) 2024-03-06T10:57:37.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:37.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:37.580709+0000) 2024-03-06T10:57:38.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:38.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:38.580957+0000) 2024-03-06T10:57:39.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:39.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:39.581258+0000) 2024-03-06T10:57:40.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:40.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:40.581578+0000) 2024-03-06T10:57:41.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:41.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:41.581840+0000) 2024-03-06T10:57:42.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:42.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:42.582149+0000) 2024-03-06T10:57:43.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:43.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:43.582400+0000) 2024-03-06T10:57:44.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:44.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:44.582731+0000) 2024-03-06T10:57:45.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:45.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:45.583031+0000) 2024-03-06T10:57:46.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:46.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:46.583354+0000) 2024-03-06T10:57:47.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:47.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:47.583737+0000) 2024-03-06T10:57:48.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:48.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:48.584056+0000) 2024-03-06T10:57:49.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:49.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:49.584360+0000) 2024-03-06T10:57:50.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:50.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:50.584707+0000) 2024-03-06T10:57:51.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:51.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:51.584965+0000) 2024-03-06T10:57:52.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:52.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:52.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:52.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:52.585275+0000) 2024-03-06T10:57:53.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:53.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:53.585580+0000) 2024-03-06T10:57:54.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:54.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:54.585903+0000) 2024-03-06T10:57:55.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:55.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:55.586263+0000) 2024-03-06T10:57:56.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:56.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:56.586552+0000) 2024-03-06T10:57:57.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:57.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:57.586853+0000) 2024-03-06T10:57:58.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:58.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:58.587153+0000) 2024-03-06T10:57:59.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:57:59.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:59.587459+0000) 2024-03-06T10:58:00.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:00.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:00.587793+0000) 2024-03-06T10:58:01.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:01.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:01.588108+0000) 2024-03-06T10:58:02.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:02.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:02.588406+0000) 2024-03-06T10:58:03.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:03.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:03.588766+0000) 2024-03-06T10:58:04.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:04.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:04.589084+0000) 2024-03-06T10:58:05.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:05.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:05.589404+0000) 2024-03-06T10:58:06.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:06.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:06.589765+0000) 2024-03-06T10:58:07.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:07.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:07.590068+0000) 2024-03-06T10:58:08.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:08.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:08.590315+0000) 2024-03-06T10:58:09.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:09.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:09.590571+0000) 2024-03-06T10:58:10.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:10.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:10.590834+0000) 2024-03-06T10:58:11.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:11.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:11.591092+0000) 2024-03-06T10:58:12.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:12.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:12.591362+0000) 2024-03-06T10:58:13.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:13.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:13.591623+0000) 2024-03-06T10:58:14.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:14.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:14.591933+0000) 2024-03-06T10:58:15.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:15.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:15.592193+0000) 2024-03-06T10:58:16.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:16.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:16.592461+0000) 2024-03-06T10:58:17.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:17.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:17.592759+0000) 2024-03-06T10:58:18.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:18.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:18.593013+0000) 2024-03-06T10:58:19.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:19.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:19.593281+0000) 2024-03-06T10:58:20.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:20.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:20.593513+0000) 2024-03-06T10:58:21.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:21.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:21.593774+0000) 2024-03-06T10:58:22.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:22.209 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:22.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:22.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:22.594047+0000) 2024-03-06T10:58:23.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:23.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:23.594303+0000) 2024-03-06T10:58:24.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:24.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:24.594590+0000) 2024-03-06T10:58:25.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:25.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:25.594902+0000) 2024-03-06T10:58:26.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:26.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:26.595121+0000) 2024-03-06T10:58:27.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:27.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:27.595404+0000) 2024-03-06T10:58:28.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:28.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:28.595694+0000) 2024-03-06T10:58:29.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:29.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:29.595933+0000) 2024-03-06T10:58:30.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:30.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:30.596189+0000) 2024-03-06T10:58:31.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:31.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:31.596435+0000) 2024-03-06T10:58:32.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:32.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:32.596749+0000) 2024-03-06T10:58:33.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:33.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:33.596995+0000) 2024-03-06T10:58:34.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:34.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:34.597182+0000) 2024-03-06T10:58:35.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:35.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:35.597435+0000) 2024-03-06T10:58:36.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:36.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:36.597750+0000) 2024-03-06T10:58:37.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:37.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:37.598021+0000) 2024-03-06T10:58:38.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:38.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:38.598274+0000) 2024-03-06T10:58:39.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:39.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:39.598537+0000) 2024-03-06T10:58:40.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:40.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:40.598798+0000) 2024-03-06T10:58:41.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:41.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:41.599030+0000) 2024-03-06T10:58:42.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:42.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:42.599303+0000) 2024-03-06T10:58:43.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:43.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:43.599583+0000) 2024-03-06T10:58:44.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:44.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:44.599899+0000) 2024-03-06T10:58:45.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:45.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:45.600160+0000) 2024-03-06T10:58:46.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:46.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:46.600442+0000) 2024-03-06T10:58:47.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:47.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:47.600763+0000) 2024-03-06T10:58:48.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:48.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:48.601044+0000) 2024-03-06T10:58:49.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:49.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:49.601320+0000) 2024-03-06T10:58:50.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:50.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:50.601604+0000) 2024-03-06T10:58:51.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:51.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:51.601942+0000) 2024-03-06T10:58:52.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:52.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:52.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:52.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:52.602228+0000) 2024-03-06T10:58:53.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:53.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:53.602516+0000) 2024-03-06T10:58:54.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:54.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:54.602790+0000) 2024-03-06T10:58:55.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:55.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:55.603080+0000) 2024-03-06T10:58:56.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:56.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:56.603367+0000) 2024-03-06T10:58:57.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:57.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:57.603698+0000) 2024-03-06T10:58:58.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:58.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:58.603963+0000) 2024-03-06T10:58:59.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:58:59.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:59.604219+0000) 2024-03-06T10:59:00.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:00.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:00.604483+0000) 2024-03-06T10:59:01.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:01.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:01.604761+0000) 2024-03-06T10:59:02.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:02.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:02.605025+0000) 2024-03-06T10:59:03.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:03.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:03.605287+0000) 2024-03-06T10:59:04.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:04.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:04.605553+0000) 2024-03-06T10:59:05.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:05.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:05.605790+0000) 2024-03-06T10:59:06.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:06.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:06.606057+0000) 2024-03-06T10:59:07.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:07.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:07.606321+0000) 2024-03-06T10:59:08.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:08.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:08.606588+0000) 2024-03-06T10:59:09.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:09.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:09.606908+0000) 2024-03-06T10:59:10.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:10.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:10.607125+0000) 2024-03-06T10:59:11.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:11.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:11.607380+0000) 2024-03-06T10:59:12.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:12.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:12.607704+0000) 2024-03-06T10:59:13.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:13.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:13.607978+0000) 2024-03-06T10:59:14.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:14.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:14.608182+0000) 2024-03-06T10:59:15.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:15.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:15.608418+0000) 2024-03-06T10:59:16.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:16.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:16.608728+0000) 2024-03-06T10:59:17.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:17.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:17.608925+0000) 2024-03-06T10:59:18.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:18.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:18.609123+0000) 2024-03-06T10:59:19.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:19.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:19.609376+0000) 2024-03-06T10:59:20.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:20.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:20.609572+0000) 2024-03-06T10:59:21.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:21.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:21.609825+0000) 2024-03-06T10:59:22.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:22.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:22.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:22.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:22.610008+0000) 2024-03-06T10:59:23.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:23.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:23.610248+0000) 2024-03-06T10:59:24.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:24.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:24.610514+0000) 2024-03-06T10:59:25.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:25.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:25.610769+0000) 2024-03-06T10:59:26.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:26.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:26.611035+0000) 2024-03-06T10:59:27.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:27.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:27.611304+0000) 2024-03-06T10:59:28.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:28.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:28.611579+0000) 2024-03-06T10:59:29.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:29.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:29.611832+0000) 2024-03-06T10:59:30.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:30.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:30.612030+0000) 2024-03-06T10:59:31.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:31.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:31.612213+0000) 2024-03-06T10:59:32.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:32.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:32.612473+0000) 2024-03-06T10:59:33.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:33.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:33.612731+0000) 2024-03-06T10:59:34.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:34.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:34.612991+0000) 2024-03-06T10:59:35.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:35.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:35.613244+0000) 2024-03-06T10:59:36.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:36.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:36.613572+0000) 2024-03-06T10:59:37.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:37.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:37.613826+0000) 2024-03-06T10:59:38.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:38.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:38.614092+0000) 2024-03-06T10:59:39.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:39.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:39.614353+0000) 2024-03-06T10:59:40.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:40.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:40.614610+0000) 2024-03-06T10:59:41.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:41.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:41.614898+0000) 2024-03-06T10:59:42.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:42.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:42.615146+0000) 2024-03-06T10:59:43.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:43.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:43.615400+0000) 2024-03-06T10:59:44.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:44.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:44.615682+0000) 2024-03-06T10:59:45.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:45.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:45.615879+0000) 2024-03-06T10:59:46.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:46.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:46.616125+0000) 2024-03-06T10:59:47.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:47.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:47.616403+0000) 2024-03-06T10:59:48.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:48.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:48.616706+0000) 2024-03-06T10:59:49.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:49.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:49.616966+0000) 2024-03-06T10:59:50.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:50.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:50.617231+0000) 2024-03-06T10:59:51.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:51.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:51.617486+0000) 2024-03-06T10:59:52.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:52.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:52.617769+0000) 2024-03-06T10:59:52.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:52.695 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:53.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:53.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:53.618033+0000) 2024-03-06T10:59:54.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:54.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:54.618296+0000) 2024-03-06T10:59:55.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:55.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:55.618532+0000) 2024-03-06T10:59:56.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:56.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:56.618803+0000) 2024-03-06T10:59:57.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:57.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:57.619076+0000) 2024-03-06T10:59:58.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:58.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:58.619287+0000) 2024-03-06T10:59:59.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T10:59:59.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:59.619555+0000) 2024-03-06T11:00:00.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:00.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:00.619821+0000) 2024-03-06T11:00:01.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:01.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:01.620093+0000) 2024-03-06T11:00:02.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:02.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:02.620348+0000) 2024-03-06T11:00:03.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:03.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:03.620614+0000) 2024-03-06T11:00:04.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:04.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:04.620910+0000) 2024-03-06T11:00:05.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:05.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:05.621164+0000) 2024-03-06T11:00:06.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:06.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:06.621419+0000) 2024-03-06T11:00:07.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:07.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:07.621737+0000) 2024-03-06T11:00:08.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:08.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:08.622005+0000) 2024-03-06T11:00:09.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:09.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:09.622235+0000) 2024-03-06T11:00:10.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:10.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:10.622480+0000) 2024-03-06T11:00:11.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:11.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:11.622770+0000) 2024-03-06T11:00:12.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:12.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:12.623030+0000) 2024-03-06T11:00:13.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:13.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:13.623291+0000) 2024-03-06T11:00:14.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:14.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:14.623560+0000) 2024-03-06T11:00:15.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:15.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:15.623824+0000) 2024-03-06T11:00:16.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:16.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:16.624087+0000) 2024-03-06T11:00:17.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:17.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:17.624426+0000) 2024-03-06T11:00:18.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:18.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:18.624752+0000) 2024-03-06T11:00:19.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:19.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:19.625062+0000) 2024-03-06T11:00:20.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:20.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:20.625323+0000) 2024-03-06T11:00:21.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:21.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:21.625598+0000) 2024-03-06T11:00:22.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:22.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:22.625861+0000) 2024-03-06T11:00:22.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:22.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:23.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:23.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:23.626132+0000) 2024-03-06T11:00:24.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:24.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:24.626401+0000) 2024-03-06T11:00:25.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:25.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:25.626671+0000) 2024-03-06T11:00:26.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:26.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:26.626923+0000) 2024-03-06T11:00:27.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:27.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:27.627189+0000) 2024-03-06T11:00:28.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:28.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:28.627436+0000) 2024-03-06T11:00:29.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:29.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:29.627723+0000) 2024-03-06T11:00:30.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:30.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:30.627921+0000) 2024-03-06T11:00:31.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:31.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:31.628191+0000) 2024-03-06T11:00:32.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:32.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:32.628458+0000) 2024-03-06T11:00:33.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:33.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:33.628750+0000) 2024-03-06T11:00:34.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:34.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:34.629027+0000) 2024-03-06T11:00:35.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:35.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:35.629304+0000) 2024-03-06T11:00:36.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:36.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:36.629593+0000) 2024-03-06T11:00:37.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:37.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:37.629911+0000) 2024-03-06T11:00:38.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:38.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:38.630194+0000) 2024-03-06T11:00:39.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:39.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:39.630448+0000) 2024-03-06T11:00:40.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:40.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:40.630761+0000) 2024-03-06T11:00:41.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:41.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:41.631034+0000) 2024-03-06T11:00:42.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:42.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:42.631300+0000) 2024-03-06T11:00:43.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:43.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:43.631547+0000) 2024-03-06T11:00:44.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:44.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:44.631822+0000) 2024-03-06T11:00:45.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:45.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:45.632060+0000) 2024-03-06T11:00:46.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:46.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:46.632268+0000) 2024-03-06T11:00:47.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:47.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:47.632542+0000) 2024-03-06T11:00:48.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:48.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:48.632814+0000) 2024-03-06T11:00:49.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:49.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:49.633056+0000) 2024-03-06T11:00:50.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:50.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:50.633328+0000) 2024-03-06T11:00:51.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:51.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:51.633601+0000) 2024-03-06T11:00:52.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:52.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:52.633889+0000) 2024-03-06T11:00:52.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:52.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:53.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:53.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:53.634122+0000) 2024-03-06T11:00:54.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:54.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:54.634375+0000) 2024-03-06T11:00:55.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:55.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:55.634657+0000) 2024-03-06T11:00:56.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:56.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:56.634886+0000) 2024-03-06T11:00:57.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:57.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:57.635141+0000) 2024-03-06T11:00:58.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:58.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:58.635409+0000) 2024-03-06T11:00:59.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:00:59.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:59.635692+0000) 2024-03-06T11:01:00.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:00.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:00.635954+0000) 2024-03-06T11:01:01.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:01.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:01.636222+0000) 2024-03-06T11:01:02.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:02.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:02.636475+0000) 2024-03-06T11:01:03.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:03.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:03.636741+0000) 2024-03-06T11:01:04.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:04.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:04.636932+0000) 2024-03-06T11:01:05.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:05.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:05.637120+0000) 2024-03-06T11:01:06.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:06.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:06.637321+0000) 2024-03-06T11:01:07.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:07.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:07.637570+0000) 2024-03-06T11:01:08.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:08.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:08.637796+0000) 2024-03-06T11:01:09.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:09.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:09.637975+0000) 2024-03-06T11:01:10.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:10.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:10.638167+0000) 2024-03-06T11:01:11.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:11.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:11.638419+0000) 2024-03-06T11:01:12.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:12.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:12.638705+0000) 2024-03-06T11:01:13.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:13.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:13.638961+0000) 2024-03-06T11:01:14.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:14.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:14.639217+0000) 2024-03-06T11:01:15.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:15.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:15.639473+0000) 2024-03-06T11:01:16.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:16.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:16.639734+0000) 2024-03-06T11:01:17.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:17.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:17.639993+0000) 2024-03-06T11:01:18.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:18.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:18.640311+0000) 2024-03-06T11:01:19.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:19.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:19.640565+0000) 2024-03-06T11:01:20.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:20.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:20.640838+0000) 2024-03-06T11:01:21.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:21.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:21.641087+0000) 2024-03-06T11:01:22.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:22.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:22.641354+0000) 2024-03-06T11:01:23.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:23.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:23.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:23.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:23.641617+0000) 2024-03-06T11:01:24.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:24.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:24.641910+0000) 2024-03-06T11:01:25.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:25.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:25.642160+0000) 2024-03-06T11:01:26.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:26.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:26.642419+0000) 2024-03-06T11:01:27.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:27.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:27.642758+0000) 2024-03-06T11:01:28.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:28.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:28.642979+0000) 2024-03-06T11:01:29.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:29.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:29.643238+0000) 2024-03-06T11:01:30.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:30.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:30.643496+0000) 2024-03-06T11:01:31.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:31.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:31.643754+0000) 2024-03-06T11:01:32.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:32.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:32.644002+0000) 2024-03-06T11:01:33.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:33.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:33.644270+0000) 2024-03-06T11:01:34.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:34.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:34.644534+0000) 2024-03-06T11:01:35.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:35.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:35.644777+0000) 2024-03-06T11:01:36.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:36.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:36.645040+0000) 2024-03-06T11:01:37.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:37.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:37.645321+0000) 2024-03-06T11:01:38.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:38.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:38.645570+0000) 2024-03-06T11:01:39.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:39.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:39.645809+0000) 2024-03-06T11:01:40.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:40.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:40.646071+0000) 2024-03-06T11:01:41.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:41.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:41.646337+0000) 2024-03-06T11:01:42.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:42.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:42.646601+0000) 2024-03-06T11:01:43.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:43.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:43.646891+0000) 2024-03-06T11:01:44.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:44.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:44.647161+0000) 2024-03-06T11:01:45.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:45.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:45.647427+0000) 2024-03-06T11:01:46.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:46.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:46.647732+0000) 2024-03-06T11:01:47.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:47.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:47.648011+0000) 2024-03-06T11:01:48.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:48.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:48.648283+0000) 2024-03-06T11:01:49.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:49.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:49.648547+0000) 2024-03-06T11:01:50.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:50.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:50.648801+0000) 2024-03-06T11:01:51.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:51.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:51.649062+0000) 2024-03-06T11:01:52.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:52.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:52.649330+0000) 2024-03-06T11:01:53.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:53.306 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:53.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:53.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:53.649600+0000) 2024-03-06T11:01:54.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:54.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:54.649905+0000) 2024-03-06T11:01:55.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:55.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:55.650152+0000) 2024-03-06T11:01:56.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:56.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:56.650418+0000) 2024-03-06T11:01:57.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:57.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:57.650741+0000) 2024-03-06T11:01:58.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:58.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:58.651010+0000) 2024-03-06T11:01:59.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:01:59.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:59.651215+0000) 2024-03-06T11:02:00.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:00.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:00.651475+0000) 2024-03-06T11:02:01.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:01.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:01.651755+0000) 2024-03-06T11:02:02.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:02.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:02.652010+0000) 2024-03-06T11:02:03.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:03.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:03.652261+0000) 2024-03-06T11:02:04.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:04.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:04.652502+0000) 2024-03-06T11:02:05.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:05.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:05.652750+0000) 2024-03-06T11:02:06.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:06.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:06.653068+0000) 2024-03-06T11:02:07.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:07.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:07.653340+0000) 2024-03-06T11:02:08.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:08.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:08.653537+0000) 2024-03-06T11:02:09.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:09.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:09.653784+0000) 2024-03-06T11:02:10.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:10.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:10.654054+0000) 2024-03-06T11:02:11.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:11.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:11.654312+0000) 2024-03-06T11:02:12.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:12.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:12.654571+0000) 2024-03-06T11:02:13.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:13.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:13.654842+0000) 2024-03-06T11:02:14.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:14.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:14.655087+0000) 2024-03-06T11:02:15.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:15.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:15.655337+0000) 2024-03-06T11:02:16.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:16.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:16.655611+0000) 2024-03-06T11:02:17.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:17.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:17.655908+0000) 2024-03-06T11:02:18.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:18.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:18.656177+0000) 2024-03-06T11:02:19.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:19.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:19.656416+0000) 2024-03-06T11:02:20.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:20.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:20.656720+0000) 2024-03-06T11:02:21.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:21.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:21.656983+0000) 2024-03-06T11:02:22.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:22.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:22.657260+0000) 2024-03-06T11:02:23.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:23.465 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:23.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:23.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:23.657532+0000) 2024-03-06T11:02:24.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:24.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:24.657795+0000) 2024-03-06T11:02:25.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:25.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:25.658058+0000) 2024-03-06T11:02:26.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:26.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:26.658302+0000) 2024-03-06T11:02:27.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:27.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:27.658559+0000) 2024-03-06T11:02:28.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:28.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:28.658837+0000) 2024-03-06T11:02:29.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:29.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:29.659091+0000) 2024-03-06T11:02:30.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:30.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:30.659353+0000) 2024-03-06T11:02:31.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:31.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:31.659594+0000) 2024-03-06T11:02:32.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:32.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:32.659857+0000) 2024-03-06T11:02:33.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:33.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:33.660117+0000) 2024-03-06T11:02:34.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:34.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:34.660372+0000) 2024-03-06T11:02:35.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:35.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:35.660625+0000) 2024-03-06T11:02:36.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:36.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:36.660925+0000) 2024-03-06T11:02:37.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:37.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:37.661121+0000) 2024-03-06T11:02:38.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:38.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:38.661372+0000) 2024-03-06T11:02:39.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:39.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:39.661612+0000) 2024-03-06T11:02:40.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:40.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:40.661898+0000) 2024-03-06T11:02:41.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:41.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:41.662155+0000) 2024-03-06T11:02:42.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:42.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:42.662420+0000) 2024-03-06T11:02:43.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:43.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:43.662736+0000) 2024-03-06T11:02:44.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:44.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:44.662994+0000) 2024-03-06T11:02:45.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:45.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:45.663247+0000) 2024-03-06T11:02:46.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:46.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:46.663494+0000) 2024-03-06T11:02:47.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:47.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:47.663767+0000) 2024-03-06T11:02:48.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:48.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:48.664042+0000) 2024-03-06T11:02:49.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:49.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:49.664303+0000) 2024-03-06T11:02:50.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:50.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:50.664545+0000) 2024-03-06T11:02:51.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:51.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:51.664801+0000) 2024-03-06T11:02:52.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:52.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:52.665053+0000) 2024-03-06T11:02:53.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:53.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:53.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:53.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:53.665292+0000) 2024-03-06T11:02:54.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:54.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:54.665594+0000) 2024-03-06T11:02:55.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:55.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:55.665845+0000) 2024-03-06T11:02:56.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:56.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:56.666082+0000) 2024-03-06T11:02:57.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:57.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:57.666360+0000) 2024-03-06T11:02:58.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:58.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:58.666617+0000) 2024-03-06T11:02:59.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:02:59.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:59.666926+0000) 2024-03-06T11:03:00.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:00.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:00.667168+0000) 2024-03-06T11:03:01.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:01.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:01.667422+0000) 2024-03-06T11:03:02.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:02.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:02.667663+0000) 2024-03-06T11:03:03.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:03.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:03.667885+0000) 2024-03-06T11:03:04.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:04.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:04.668071+0000) 2024-03-06T11:03:05.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:05.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:05.668300+0000) 2024-03-06T11:03:06.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:06.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:06.668548+0000) 2024-03-06T11:03:07.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:07.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:07.668804+0000) 2024-03-06T11:03:08.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:08.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:08.669056+0000) 2024-03-06T11:03:09.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:09.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:09.669311+0000) 2024-03-06T11:03:10.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:10.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:10.669569+0000) 2024-03-06T11:03:11.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:11.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:11.669792+0000) 2024-03-06T11:03:12.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:12.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:12.670056+0000) 2024-03-06T11:03:13.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:13.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:13.670288+0000) 2024-03-06T11:03:14.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:14.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:14.670543+0000) 2024-03-06T11:03:15.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:15.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:15.670785+0000) 2024-03-06T11:03:16.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:16.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:16.671027+0000) 2024-03-06T11:03:17.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:17.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:17.671237+0000) 2024-03-06T11:03:18.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:18.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:18.671482+0000) 2024-03-06T11:03:19.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:19.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:19.671744+0000) 2024-03-06T11:03:20.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:20.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:20.672015+0000) 2024-03-06T11:03:21.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:21.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:21.672273+0000) 2024-03-06T11:03:22.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:22.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:22.672519+0000) 2024-03-06T11:03:23.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:23.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:23.672777+0000) 2024-03-06T11:03:23.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:23.748 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:24.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:24.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:24.673041+0000) 2024-03-06T11:03:25.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:25.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:25.673298+0000) 2024-03-06T11:03:26.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:26.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:26.673563+0000) 2024-03-06T11:03:27.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:27.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:27.673821+0000) 2024-03-06T11:03:28.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:28.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:28.674091+0000) 2024-03-06T11:03:29.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:29.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:29.674335+0000) 2024-03-06T11:03:30.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:30.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:30.674579+0000) 2024-03-06T11:03:31.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:31.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:31.674829+0000) 2024-03-06T11:03:32.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:32.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:32.675088+0000) 2024-03-06T11:03:33.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:33.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:33.675346+0000) 2024-03-06T11:03:34.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:34.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:34.675587+0000) 2024-03-06T11:03:35.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:35.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:35.675808+0000) 2024-03-06T11:03:36.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:36.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:36.676077+0000) 2024-03-06T11:03:37.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:37.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:37.676334+0000) 2024-03-06T11:03:38.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:38.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:38.676600+0000) 2024-03-06T11:03:39.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:39.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:39.676909+0000) 2024-03-06T11:03:40.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:40.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:40.677155+0000) 2024-03-06T11:03:41.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:41.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:41.677397+0000) 2024-03-06T11:03:42.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:42.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:42.677713+0000) 2024-03-06T11:03:43.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:43.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:43.677977+0000) 2024-03-06T11:03:44.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:44.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:44.678227+0000) 2024-03-06T11:03:45.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:45.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:45.678488+0000) 2024-03-06T11:03:46.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:46.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:46.678756+0000) 2024-03-06T11:03:47.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:47.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:47.679065+0000) 2024-03-06T11:03:48.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:48.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:48.679334+0000) 2024-03-06T11:03:49.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:49.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:49.679601+0000) 2024-03-06T11:03:50.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:50.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:50.679903+0000) 2024-03-06T11:03:51.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:51.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:51.680180+0000) 2024-03-06T11:03:52.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:52.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:52.680474+0000) 2024-03-06T11:03:53.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:53.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:53.680753+0000) 2024-03-06T11:03:53.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:53.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:54.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:54.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:54.681022+0000) 2024-03-06T11:03:55.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:55.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:55.681298+0000) 2024-03-06T11:03:56.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:56.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:56.681568+0000) 2024-03-06T11:03:57.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:57.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:57.681852+0000) 2024-03-06T11:03:58.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:58.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:58.682120+0000) 2024-03-06T11:03:59.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:03:59.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:59.682393+0000) 2024-03-06T11:04:00.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:00.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:00.682724+0000) 2024-03-06T11:04:01.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:01.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:01.683027+0000) 2024-03-06T11:04:02.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:02.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:02.683313+0000) 2024-03-06T11:04:03.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:03.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:03.683601+0000) 2024-03-06T11:04:04.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:04.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:04.683858+0000) 2024-03-06T11:04:05.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:05.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:05.684114+0000) 2024-03-06T11:04:06.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:06.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:06.684377+0000) 2024-03-06T11:04:07.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:07.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:07.684694+0000) 2024-03-06T11:04:08.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:08.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:08.684968+0000) 2024-03-06T11:04:09.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:09.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:09.685210+0000) 2024-03-06T11:04:10.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:10.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:10.685471+0000) 2024-03-06T11:04:11.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:11.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:11.685752+0000) 2024-03-06T11:04:12.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:12.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:12.686022+0000) 2024-03-06T11:04:13.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:13.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:13.686263+0000) 2024-03-06T11:04:14.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:14.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:14.686511+0000) 2024-03-06T11:04:15.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:15.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:15.686780+0000) 2024-03-06T11:04:16.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:16.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:16.687041+0000) 2024-03-06T11:04:17.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:17.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:17.687323+0000) 2024-03-06T11:04:18.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:18.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:18.687572+0000) 2024-03-06T11:04:19.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:19.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:19.687804+0000) 2024-03-06T11:04:20.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:20.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:20.688043+0000) 2024-03-06T11:04:21.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:21.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:21.688286+0000) 2024-03-06T11:04:22.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:22.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:22.688524+0000) 2024-03-06T11:04:23.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:23.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:23.688748+0000) 2024-03-06T11:04:24.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:24.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:24.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:24.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:24.689018+0000) 2024-03-06T11:04:25.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:25.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:25.689280+0000) 2024-03-06T11:04:26.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:26.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:26.689527+0000) 2024-03-06T11:04:27.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:27.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:27.689795+0000) 2024-03-06T11:04:28.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:28.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:28.690045+0000) 2024-03-06T11:04:29.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:29.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:29.690304+0000) 2024-03-06T11:04:30.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:30.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:30.690544+0000) 2024-03-06T11:04:31.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:31.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:31.690807+0000) 2024-03-06T11:04:32.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:32.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:32.691058+0000) 2024-03-06T11:04:33.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:33.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:33.691304+0000) 2024-03-06T11:04:34.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:34.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:34.691565+0000) 2024-03-06T11:04:35.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:35.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:35.691812+0000) 2024-03-06T11:04:36.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:36.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:36.692071+0000) 2024-03-06T11:04:37.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:37.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:37.692328+0000) 2024-03-06T11:04:38.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:38.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:38.692593+0000) 2024-03-06T11:04:39.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:39.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:39.692871+0000) 2024-03-06T11:04:40.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:40.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:40.693081+0000) 2024-03-06T11:04:41.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:41.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:41.693338+0000) 2024-03-06T11:04:42.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:42.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:42.693606+0000) 2024-03-06T11:04:43.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:43.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:43.693891+0000) 2024-03-06T11:04:44.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:44.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:44.694156+0000) 2024-03-06T11:04:45.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:45.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:45.694409+0000) 2024-03-06T11:04:46.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:46.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:46.694667+0000) 2024-03-06T11:04:47.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:47.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:47.694884+0000) 2024-03-06T11:04:48.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:48.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:48.695081+0000) 2024-03-06T11:04:49.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:49.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:49.695333+0000) 2024-03-06T11:04:50.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:50.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:50.695594+0000) 2024-03-06T11:04:51.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:51.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:51.695840+0000) 2024-03-06T11:04:52.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:52.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:52.696065+0000) 2024-03-06T11:04:53.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:53.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:53.696385+0000) 2024-03-06T11:04:54.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:54.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:54.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:54.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:54.696759+0000) 2024-03-06T11:04:55.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:55.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:55.697055+0000) 2024-03-06T11:04:56.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:56.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:56.697349+0000) 2024-03-06T11:04:57.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:57.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:57.697733+0000) 2024-03-06T11:04:58.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:58.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:58.698058+0000) 2024-03-06T11:04:59.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:04:59.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:59.698352+0000) 2024-03-06T11:05:00.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:00.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:00.698707+0000) 2024-03-06T11:05:01.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:01.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:01.698949+0000) 2024-03-06T11:05:02.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:02.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:02.699283+0000) 2024-03-06T11:05:03.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:03.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:03.699589+0000) 2024-03-06T11:05:04.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:04.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:04.699960+0000) 2024-03-06T11:05:05.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:05.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:05.700255+0000) 2024-03-06T11:05:06.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:06.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:06.700576+0000) 2024-03-06T11:05:07.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:07.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:07.700885+0000) 2024-03-06T11:05:08.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:08.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:08.701204+0000) 2024-03-06T11:05:09.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:09.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:09.701500+0000) 2024-03-06T11:05:10.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:10.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:10.701805+0000) 2024-03-06T11:05:11.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:11.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:11.702126+0000) 2024-03-06T11:05:12.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:12.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:12.702431+0000) 2024-03-06T11:05:13.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:13.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:13.702748+0000) 2024-03-06T11:05:14.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:14.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:14.703021+0000) 2024-03-06T11:05:15.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:15.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:15.703280+0000) 2024-03-06T11:05:16.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:16.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:16.703539+0000) 2024-03-06T11:05:17.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:17.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:17.703799+0000) 2024-03-06T11:05:18.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:18.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:18.704073+0000) 2024-03-06T11:05:19.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:19.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:19.704345+0000) 2024-03-06T11:05:20.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:20.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:20.704618+0000) 2024-03-06T11:05:21.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:21.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:21.704901+0000) 2024-03-06T11:05:22.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:22.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:22.705171+0000) 2024-03-06T11:05:23.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:23.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:23.705492+0000) 2024-03-06T11:05:24.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:24.335 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:24.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:24.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:24.705762+0000) 2024-03-06T11:05:25.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:25.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:25.706028+0000) 2024-03-06T11:05:26.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:26.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:26.706293+0000) 2024-03-06T11:05:27.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:27.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:27.706544+0000) 2024-03-06T11:05:28.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:28.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:28.706793+0000) 2024-03-06T11:05:29.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:29.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:29.707058+0000) 2024-03-06T11:05:30.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:30.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:30.707321+0000) 2024-03-06T11:05:31.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:31.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:31.707578+0000) 2024-03-06T11:05:32.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:32.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:32.707857+0000) 2024-03-06T11:05:33.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:33.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:33.708105+0000) 2024-03-06T11:05:34.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:34.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:34.708365+0000) 2024-03-06T11:05:35.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:35.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:35.708606+0000) 2024-03-06T11:05:36.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:36.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:36.708891+0000) 2024-03-06T11:05:37.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:37.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:37.709166+0000) 2024-03-06T11:05:38.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:38.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:38.709435+0000) 2024-03-06T11:05:39.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:39.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:39.709726+0000) 2024-03-06T11:05:40.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:40.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:40.709984+0000) 2024-03-06T11:05:41.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:41.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:41.710234+0000) 2024-03-06T11:05:42.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:42.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:42.710506+0000) 2024-03-06T11:05:43.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:43.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:43.710745+0000) 2024-03-06T11:05:44.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:44.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:44.711014+0000) 2024-03-06T11:05:45.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:45.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:45.711260+0000) 2024-03-06T11:05:46.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:46.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:46.711520+0000) 2024-03-06T11:05:47.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:47.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:47.711781+0000) 2024-03-06T11:05:48.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:48.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:48.712027+0000) 2024-03-06T11:05:49.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:49.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:49.712283+0000) 2024-03-06T11:05:50.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:50.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:50.712538+0000) 2024-03-06T11:05:51.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:51.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:51.712801+0000) 2024-03-06T11:05:52.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:52.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:52.713069+0000) 2024-03-06T11:05:53.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:53.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:53.713329+0000) 2024-03-06T11:05:54.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:54.490 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:54.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:54.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:54.713588+0000) 2024-03-06T11:05:55.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:55.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:55.713856+0000) 2024-03-06T11:05:56.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:56.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:56.714116+0000) 2024-03-06T11:05:57.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:57.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:57.714371+0000) 2024-03-06T11:05:58.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:58.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:58.714688+0000) 2024-03-06T11:05:59.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:05:59.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:59.714955+0000) 2024-03-06T11:06:00.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:00.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:00.715225+0000) 2024-03-06T11:06:01.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:01.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:01.715469+0000) 2024-03-06T11:06:02.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:02.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:02.715737+0000) 2024-03-06T11:06:03.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:03.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:03.715995+0000) 2024-03-06T11:06:04.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:04.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:04.716247+0000) 2024-03-06T11:06:05.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:05.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:05.716479+0000) 2024-03-06T11:06:06.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:06.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:06.716751+0000) 2024-03-06T11:06:07.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:07.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:07.717023+0000) 2024-03-06T11:06:08.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:08.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:08.717294+0000) 2024-03-06T11:06:09.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:09.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:09.717560+0000) 2024-03-06T11:06:10.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:10.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:10.717804+0000) 2024-03-06T11:06:11.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:11.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:11.718035+0000) 2024-03-06T11:06:12.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:12.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:12.718283+0000) 2024-03-06T11:06:13.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:13.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:13.718544+0000) 2024-03-06T11:06:14.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:14.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:14.718822+0000) 2024-03-06T11:06:15.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:15.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:15.719022+0000) 2024-03-06T11:06:16.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:16.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:16.719326+0000) 2024-03-06T11:06:17.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:17.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:17.719582+0000) 2024-03-06T11:06:18.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:18.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:18.719835+0000) 2024-03-06T11:06:19.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:19.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:19.720106+0000) 2024-03-06T11:06:20.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:20.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:20.720372+0000) 2024-03-06T11:06:21.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:21.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:21.720666+0000) 2024-03-06T11:06:22.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:22.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:22.720937+0000) 2024-03-06T11:06:23.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:23.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:23.721141+0000) 2024-03-06T11:06:24.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:24.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:24.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:24.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:24.721395+0000) 2024-03-06T11:06:25.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:25.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:25.721605+0000) 2024-03-06T11:06:26.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:26.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:26.721856+0000) 2024-03-06T11:06:27.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:27.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:27.722113+0000) 2024-03-06T11:06:28.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:28.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:28.722411+0000) 2024-03-06T11:06:29.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:29.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:29.722771+0000) 2024-03-06T11:06:30.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:30.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:30.723078+0000) 2024-03-06T11:06:31.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:31.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:31.723380+0000) 2024-03-06T11:06:32.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:32.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:32.723738+0000) 2024-03-06T11:06:33.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:33.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:33.723958+0000) 2024-03-06T11:06:34.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:34.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:34.724247+0000) 2024-03-06T11:06:35.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:35.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:35.724508+0000) 2024-03-06T11:06:36.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:36.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:36.724784+0000) 2024-03-06T11:06:37.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:37.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:37.725065+0000) 2024-03-06T11:06:38.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:38.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:38.725352+0000) 2024-03-06T11:06:39.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:39.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:39.725681+0000) 2024-03-06T11:06:40.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:40.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:40.725953+0000) 2024-03-06T11:06:41.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:41.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:41.726223+0000) 2024-03-06T11:06:42.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:42.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:42.726481+0000) 2024-03-06T11:06:43.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:43.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:43.726747+0000) 2024-03-06T11:06:44.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:44.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:44.727017+0000) 2024-03-06T11:06:45.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:45.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:45.727283+0000) 2024-03-06T11:06:46.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:46.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:46.727545+0000) 2024-03-06T11:06:47.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:47.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:47.727833+0000) 2024-03-06T11:06:48.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:48.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:48.728119+0000) 2024-03-06T11:06:49.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:49.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:49.728411+0000) 2024-03-06T11:06:50.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:50.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:50.728727+0000) 2024-03-06T11:06:51.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:51.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:51.728995+0000) 2024-03-06T11:06:52.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:52.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:52.729275+0000) 2024-03-06T11:06:53.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:53.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:53.729562+0000) 2024-03-06T11:06:54.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:54.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:54.729838+0000) 2024-03-06T11:06:54.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:54.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:55.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:55.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:55.730124+0000) 2024-03-06T11:06:56.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:56.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:56.730393+0000) 2024-03-06T11:06:57.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:57.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:57.730608+0000) 2024-03-06T11:06:58.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:58.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:58.730935+0000) 2024-03-06T11:06:59.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:06:59.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:59.731224+0000) 2024-03-06T11:07:00.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:00.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:00.731486+0000) 2024-03-06T11:07:01.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:01.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:01.731755+0000) 2024-03-06T11:07:02.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:02.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:02.732030+0000) 2024-03-06T11:07:03.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:03.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:03.732306+0000) 2024-03-06T11:07:04.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:04.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:04.732687+0000) 2024-03-06T11:07:05.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:05.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:05.732931+0000) 2024-03-06T11:07:06.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:06.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:06.733194+0000) 2024-03-06T11:07:07.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:07.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:07.733463+0000) 2024-03-06T11:07:08.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:08.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:08.733767+0000) 2024-03-06T11:07:09.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:09.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:09.733979+0000) 2024-03-06T11:07:10.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:10.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:10.734172+0000) 2024-03-06T11:07:11.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:11.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:11.734370+0000) 2024-03-06T11:07:12.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:12.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:12.734573+0000) 2024-03-06T11:07:13.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:13.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:13.734864+0000) 2024-03-06T11:07:14.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:14.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:14.735170+0000) 2024-03-06T11:07:15.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:15.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:15.735409+0000) 2024-03-06T11:07:16.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:16.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:16.735712+0000) 2024-03-06T11:07:17.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:17.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:17.735981+0000) 2024-03-06T11:07:18.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:18.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:18.736248+0000) 2024-03-06T11:07:19.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:19.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:19.736490+0000) 2024-03-06T11:07:20.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:20.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:20.736762+0000) 2024-03-06T11:07:21.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:21.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:21.737016+0000) 2024-03-06T11:07:22.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:22.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:22.737264+0000) 2024-03-06T11:07:23.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:23.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:23.737513+0000) 2024-03-06T11:07:24.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:24.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:24.737781+0000) 2024-03-06T11:07:24.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:24.914 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:25.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:25.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:25.737981+0000) 2024-03-06T11:07:26.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:26.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:26.738224+0000) 2024-03-06T11:07:27.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:27.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:27.738478+0000) 2024-03-06T11:07:28.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:28.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:28.738763+0000) 2024-03-06T11:07:29.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:29.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:29.739045+0000) 2024-03-06T11:07:30.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:30.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:30.739303+0000) 2024-03-06T11:07:31.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:31.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:31.739581+0000) 2024-03-06T11:07:32.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:32.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:32.739869+0000) 2024-03-06T11:07:33.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:33.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:33.740087+0000) 2024-03-06T11:07:34.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:34.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:34.740303+0000) 2024-03-06T11:07:35.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:35.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:35.740564+0000) 2024-03-06T11:07:36.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:36.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:36.740822+0000) 2024-03-06T11:07:37.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:37.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:37.741084+0000) 2024-03-06T11:07:38.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:38.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:38.741376+0000) 2024-03-06T11:07:39.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:39.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:39.741718+0000) 2024-03-06T11:07:40.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:40.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:40.741980+0000) 2024-03-06T11:07:41.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:41.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:41.742289+0000) 2024-03-06T11:07:42.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:42.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:42.742589+0000) 2024-03-06T11:07:43.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:43.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:43.742934+0000) 2024-03-06T11:07:44.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:44.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:44.743258+0000) 2024-03-06T11:07:45.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:45.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:45.743577+0000) 2024-03-06T11:07:46.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:46.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:46.743911+0000) 2024-03-06T11:07:47.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:47.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:47.744235+0000) 2024-03-06T11:07:48.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:48.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:48.744534+0000) 2024-03-06T11:07:49.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:49.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:49.744783+0000) 2024-03-06T11:07:50.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:50.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:50.745060+0000) 2024-03-06T11:07:51.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:51.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:51.745331+0000) 2024-03-06T11:07:52.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:52.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:52.745709+0000) 2024-03-06T11:07:53.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:53.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:53.745917+0000) 2024-03-06T11:07:54.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:54.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:54.746186+0000) 2024-03-06T11:07:55.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:55.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:55.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:55.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:55.746453+0000) 2024-03-06T11:07:56.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:56.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:56.746724+0000) 2024-03-06T11:07:57.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:57.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:57.746969+0000) 2024-03-06T11:07:58.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:58.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:58.747235+0000) 2024-03-06T11:07:59.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:07:59.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:59.747503+0000) 2024-03-06T11:08:00.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:00.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:00.747761+0000) 2024-03-06T11:08:01.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:01.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:01.748026+0000) 2024-03-06T11:08:02.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:02.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:02.748284+0000) 2024-03-06T11:08:03.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:03.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:03.748529+0000) 2024-03-06T11:08:04.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:04.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:04.748793+0000) 2024-03-06T11:08:05.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:05.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:05.749032+0000) 2024-03-06T11:08:06.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:06.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:06.749279+0000) 2024-03-06T11:08:07.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:07.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:07.749476+0000) 2024-03-06T11:08:08.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:08.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:08.749794+0000) 2024-03-06T11:08:09.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:09.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:09.750068+0000) 2024-03-06T11:08:10.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:10.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:10.750330+0000) 2024-03-06T11:08:11.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:11.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:11.750567+0000) 2024-03-06T11:08:12.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:12.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:12.750822+0000) 2024-03-06T11:08:13.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:13.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:13.751083+0000) 2024-03-06T11:08:14.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:14.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:14.751353+0000) 2024-03-06T11:08:15.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:15.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:15.751599+0000) 2024-03-06T11:08:16.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:16.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:16.751869+0000) 2024-03-06T11:08:17.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:17.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:17.752135+0000) 2024-03-06T11:08:18.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:18.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:18.752404+0000) 2024-03-06T11:08:19.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:19.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:19.752662+0000) 2024-03-06T11:08:20.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:20.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:20.752936+0000) 2024-03-06T11:08:21.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:21.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:21.753183+0000) 2024-03-06T11:08:22.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:22.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:22.753432+0000) 2024-03-06T11:08:23.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:23.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:23.753741+0000) 2024-03-06T11:08:24.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:24.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:24.754019+0000) 2024-03-06T11:08:25.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:25.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:25.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:25.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:25.754282+0000) 2024-03-06T11:08:26.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:26.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:26.754535+0000) 2024-03-06T11:08:27.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:27.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:27.754780+0000) 2024-03-06T11:08:28.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:28.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:28.755043+0000) 2024-03-06T11:08:29.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:29.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:29.755308+0000) 2024-03-06T11:08:30.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:30.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:30.755577+0000) 2024-03-06T11:08:31.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:31.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:31.755835+0000) 2024-03-06T11:08:32.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:32.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:32.756093+0000) 2024-03-06T11:08:33.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:33.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:33.756285+0000) 2024-03-06T11:08:34.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:34.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:34.756529+0000) 2024-03-06T11:08:35.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:35.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:35.756777+0000) 2024-03-06T11:08:36.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:36.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:36.757023+0000) 2024-03-06T11:08:37.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:37.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:37.757281+0000) 2024-03-06T11:08:38.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:38.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:38.757531+0000) 2024-03-06T11:08:39.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:39.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:39.757792+0000) 2024-03-06T11:08:40.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:40.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:40.758067+0000) 2024-03-06T11:08:41.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:41.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:41.758327+0000) 2024-03-06T11:08:42.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:42.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:42.758568+0000) 2024-03-06T11:08:43.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:43.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:43.758836+0000) 2024-03-06T11:08:44.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:44.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:44.759082+0000) 2024-03-06T11:08:45.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:45.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:45.759346+0000) 2024-03-06T11:08:46.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:46.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:46.759680+0000) 2024-03-06T11:08:47.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:47.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:47.759890+0000) 2024-03-06T11:08:48.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:48.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:48.760141+0000) 2024-03-06T11:08:49.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:49.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:49.760328+0000) 2024-03-06T11:08:50.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:50.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:50.760594+0000) 2024-03-06T11:08:51.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:51.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:51.760840+0000) 2024-03-06T11:08:52.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:52.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:52.761099+0000) 2024-03-06T11:08:53.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:53.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:53.761355+0000) 2024-03-06T11:08:54.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:54.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:54.761615+0000) 2024-03-06T11:08:55.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:55.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:55.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:55.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:55.761927+0000) 2024-03-06T11:08:56.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:56.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:56.762197+0000) 2024-03-06T11:08:57.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:57.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:57.762469+0000) 2024-03-06T11:08:58.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:58.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:58.762770+0000) 2024-03-06T11:08:59.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:08:59.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:59.763031+0000) 2024-03-06T11:09:00.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:00.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:00.763300+0000) 2024-03-06T11:09:01.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:01.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:01.763585+0000) 2024-03-06T11:09:02.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:02.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:02.763831+0000) 2024-03-06T11:09:03.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:03.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:03.764085+0000) 2024-03-06T11:09:04.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:04.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:04.764308+0000) 2024-03-06T11:09:05.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:05.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:05.764562+0000) 2024-03-06T11:09:06.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:06.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:06.764792+0000) 2024-03-06T11:09:07.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:07.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:07.765039+0000) 2024-03-06T11:09:08.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:08.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:08.765322+0000) 2024-03-06T11:09:09.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:09.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:09.765546+0000) 2024-03-06T11:09:10.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:10.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:10.765836+0000) 2024-03-06T11:09:11.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:11.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:11.766051+0000) 2024-03-06T11:09:12.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:12.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:12.766336+0000) 2024-03-06T11:09:13.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:13.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:13.766596+0000) 2024-03-06T11:09:14.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:14.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:14.766855+0000) 2024-03-06T11:09:15.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:15.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:15.767063+0000) 2024-03-06T11:09:16.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:16.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:16.767325+0000) 2024-03-06T11:09:17.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:17.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:17.767576+0000) 2024-03-06T11:09:18.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:18.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:18.767899+0000) 2024-03-06T11:09:19.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:19.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:19.768140+0000) 2024-03-06T11:09:20.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:20.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:20.768473+0000) 2024-03-06T11:09:21.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:21.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:21.768780+0000) 2024-03-06T11:09:22.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:22.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:22.769097+0000) 2024-03-06T11:09:23.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:23.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:23.769340+0000) 2024-03-06T11:09:24.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:24.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:24.769683+0000) 2024-03-06T11:09:25.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:25.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:25.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:25.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:25.769953+0000) 2024-03-06T11:09:26.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:26.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:26.770248+0000) 2024-03-06T11:09:27.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:27.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:27.770542+0000) 2024-03-06T11:09:28.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:28.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:28.770801+0000) 2024-03-06T11:09:29.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:29.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:29.771038+0000) 2024-03-06T11:09:30.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:30.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:30.771328+0000) 2024-03-06T11:09:31.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:31.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:31.771607+0000) 2024-03-06T11:09:32.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:32.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:32.771952+0000) 2024-03-06T11:09:33.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:33.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:33.772200+0000) 2024-03-06T11:09:34.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:34.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:34.772551+0000) 2024-03-06T11:09:35.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:35.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:35.772796+0000) 2024-03-06T11:09:36.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:36.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:36.773092+0000) 2024-03-06T11:09:37.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:37.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:37.773400+0000) 2024-03-06T11:09:38.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:38.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:38.773771+0000) 2024-03-06T11:09:39.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:39.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:39.774073+0000) 2024-03-06T11:09:40.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:40.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:40.774371+0000) 2024-03-06T11:09:41.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:41.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:41.774720+0000) 2024-03-06T11:09:42.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:42.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:42.775045+0000) 2024-03-06T11:09:43.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:43.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:43.775309+0000) 2024-03-06T11:09:44.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:44.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:44.775683+0000) 2024-03-06T11:09:45.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:45.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:45.776009+0000) 2024-03-06T11:09:46.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:46.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:46.776330+0000) 2024-03-06T11:09:47.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:47.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:47.776701+0000) 2024-03-06T11:09:48.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:48.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:48.777033+0000) 2024-03-06T11:09:49.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:49.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:49.777351+0000) 2024-03-06T11:09:50.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:50.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:50.777707+0000) 2024-03-06T11:09:51.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:51.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:51.778022+0000) 2024-03-06T11:09:52.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:52.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:52.778327+0000) 2024-03-06T11:09:53.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:53.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:53.778717+0000) 2024-03-06T11:09:54.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:54.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:54.779013+0000) 2024-03-06T11:09:55.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:55.642 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:55.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:55.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:55.779310+0000) 2024-03-06T11:09:56.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:56.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:56.779679+0000) 2024-03-06T11:09:57.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:57.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:57.780040+0000) 2024-03-06T11:09:58.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:58.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:58.780326+0000) 2024-03-06T11:09:59.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:09:59.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:59.780569+0000) 2024-03-06T11:10:00.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:00.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:00.780881+0000) 2024-03-06T11:10:01.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:01.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:01.781202+0000) 2024-03-06T11:10:02.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:02.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:02.781528+0000) 2024-03-06T11:10:03.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:03.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:03.781847+0000) 2024-03-06T11:10:04.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:04.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:04.782163+0000) 2024-03-06T11:10:05.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:05.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:05.782492+0000) 2024-03-06T11:10:06.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:06.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:06.782794+0000) 2024-03-06T11:10:07.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:07.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:07.783089+0000) 2024-03-06T11:10:08.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:08.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:08.783340+0000) 2024-03-06T11:10:09.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:09.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:09.783588+0000) 2024-03-06T11:10:10.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:10.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:10.783856+0000) 2024-03-06T11:10:11.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:11.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:11.784096+0000) 2024-03-06T11:10:12.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:12.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:12.784361+0000) 2024-03-06T11:10:13.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:13.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:13.784673+0000) 2024-03-06T11:10:14.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:14.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:14.784921+0000) 2024-03-06T11:10:15.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:15.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:15.785186+0000) 2024-03-06T11:10:16.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:16.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:16.785450+0000) 2024-03-06T11:10:17.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:17.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:17.785741+0000) 2024-03-06T11:10:18.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:18.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:18.786011+0000) 2024-03-06T11:10:19.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:19.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:19.786255+0000) 2024-03-06T11:10:20.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:20.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:20.786502+0000) 2024-03-06T11:10:21.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:21.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:21.786749+0000) 2024-03-06T11:10:22.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:22.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:22.787058+0000) 2024-03-06T11:10:23.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:23.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:23.787320+0000) 2024-03-06T11:10:24.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:24.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:24.787572+0000) 2024-03-06T11:10:25.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:25.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:25.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:25.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:25.787824+0000) 2024-03-06T11:10:26.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:26.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:26.788027+0000) 2024-03-06T11:10:27.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:27.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:27.788233+0000) 2024-03-06T11:10:28.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:28.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:28.788471+0000) 2024-03-06T11:10:29.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:29.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:29.788757+0000) 2024-03-06T11:10:30.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:30.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:30.789018+0000) 2024-03-06T11:10:31.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:31.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:31.789276+0000) 2024-03-06T11:10:32.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:32.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:32.789542+0000) 2024-03-06T11:10:33.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:33.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:33.789805+0000) 2024-03-06T11:10:34.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:34.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:34.790068+0000) 2024-03-06T11:10:35.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:35.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:35.790334+0000) 2024-03-06T11:10:36.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:36.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:36.790591+0000) 2024-03-06T11:10:37.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:37.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:37.790889+0000) 2024-03-06T11:10:38.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:38.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:38.791161+0000) 2024-03-06T11:10:39.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:39.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:39.791421+0000) 2024-03-06T11:10:40.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:40.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:40.791702+0000) 2024-03-06T11:10:41.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:41.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:41.791943+0000) 2024-03-06T11:10:42.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:42.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:42.792205+0000) 2024-03-06T11:10:43.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:43.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:43.792467+0000) 2024-03-06T11:10:44.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:44.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:44.792769+0000) 2024-03-06T11:10:45.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:45.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:45.793031+0000) 2024-03-06T11:10:46.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:46.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:46.793295+0000) 2024-03-06T11:10:47.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:47.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:47.793545+0000) 2024-03-06T11:10:48.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:48.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:48.793794+0000) 2024-03-06T11:10:49.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:49.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:49.794039+0000) 2024-03-06T11:10:50.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:50.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:50.794296+0000) 2024-03-06T11:10:51.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:51.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:51.794551+0000) 2024-03-06T11:10:52.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:52.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:52.794814+0000) 2024-03-06T11:10:53.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:53.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:53.795075+0000) 2024-03-06T11:10:54.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:54.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:54.795340+0000) 2024-03-06T11:10:55.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:55.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:55.795591+0000) 2024-03-06T11:10:55.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:55.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:56.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:56.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:56.795846+0000) 2024-03-06T11:10:57.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:57.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:57.796041+0000) 2024-03-06T11:10:58.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:58.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:58.796299+0000) 2024-03-06T11:10:59.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:10:59.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:59.796560+0000) 2024-03-06T11:11:00.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:00.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:00.796821+0000) 2024-03-06T11:11:01.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:01.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:01.797082+0000) 2024-03-06T11:11:02.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:02.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:02.797275+0000) 2024-03-06T11:11:03.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:03.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:03.797537+0000) 2024-03-06T11:11:04.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:04.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:04.797792+0000) 2024-03-06T11:11:05.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:05.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:05.798059+0000) 2024-03-06T11:11:06.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:06.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:06.798303+0000) 2024-03-06T11:11:07.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:07.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:07.798571+0000) 2024-03-06T11:11:08.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:08.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:08.798835+0000) 2024-03-06T11:11:09.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:09.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:09.799097+0000) 2024-03-06T11:11:10.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:10.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:10.799366+0000) 2024-03-06T11:11:11.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:11.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:11.799625+0000) 2024-03-06T11:11:12.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:12.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:12.799867+0000) 2024-03-06T11:11:13.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:13.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:13.800142+0000) 2024-03-06T11:11:14.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:14.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:14.800407+0000) 2024-03-06T11:11:15.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:15.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:15.800774+0000) 2024-03-06T11:11:16.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:16.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:16.801038+0000) 2024-03-06T11:11:17.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:17.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:17.801290+0000) 2024-03-06T11:11:18.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:18.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:18.801529+0000) 2024-03-06T11:11:19.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:19.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:19.801797+0000) 2024-03-06T11:11:20.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:20.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:20.802071+0000) 2024-03-06T11:11:21.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:21.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:21.802326+0000) 2024-03-06T11:11:22.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:22.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:22.802593+0000) 2024-03-06T11:11:23.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:23.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:23.802843+0000) 2024-03-06T11:11:24.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:24.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:24.803091+0000) 2024-03-06T11:11:25.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:25.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:25.803347+0000) 2024-03-06T11:11:26.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:26.094 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:26.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:26.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:26.803572+0000) 2024-03-06T11:11:27.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:27.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:27.803828+0000) 2024-03-06T11:11:28.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:28.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:28.804092+0000) 2024-03-06T11:11:29.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:29.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:29.804340+0000) 2024-03-06T11:11:30.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:30.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:30.804591+0000) 2024-03-06T11:11:31.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:31.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:31.804865+0000) 2024-03-06T11:11:32.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:32.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:32.805126+0000) 2024-03-06T11:11:33.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:33.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:33.805384+0000) 2024-03-06T11:11:34.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:34.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:34.805688+0000) 2024-03-06T11:11:35.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:35.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:35.805954+0000) 2024-03-06T11:11:36.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:36.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:36.806224+0000) 2024-03-06T11:11:37.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:37.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:37.806459+0000) 2024-03-06T11:11:38.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:38.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:38.806757+0000) 2024-03-06T11:11:39.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:39.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:39.807018+0000) 2024-03-06T11:11:40.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:40.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:40.807223+0000) 2024-03-06T11:11:41.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:41.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:41.807481+0000) 2024-03-06T11:11:42.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:42.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:42.807758+0000) 2024-03-06T11:11:43.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:43.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:43.807994+0000) 2024-03-06T11:11:44.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:44.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:44.808261+0000) 2024-03-06T11:11:45.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:45.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:45.808527+0000) 2024-03-06T11:11:46.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:46.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:46.808773+0000) 2024-03-06T11:11:47.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:47.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:47.809026+0000) 2024-03-06T11:11:48.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:48.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:48.809286+0000) 2024-03-06T11:11:49.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:49.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:49.809533+0000) 2024-03-06T11:11:50.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:50.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:50.809803+0000) 2024-03-06T11:11:51.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:51.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:51.809995+0000) 2024-03-06T11:11:52.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:52.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:52.810241+0000) 2024-03-06T11:11:53.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:53.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:53.810504+0000) 2024-03-06T11:11:54.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:54.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:54.810761+0000) 2024-03-06T11:11:55.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:55.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:55.811009+0000) 2024-03-06T11:11:56.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:56.291 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:56.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:56.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:56.811258+0000) 2024-03-06T11:11:57.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:57.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:57.811508+0000) 2024-03-06T11:11:58.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:58.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:58.811775+0000) 2024-03-06T11:11:59.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:11:59.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:59.812033+0000) 2024-03-06T11:12:00.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:00.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:00.812287+0000) 2024-03-06T11:12:01.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:01.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:01.812541+0000) 2024-03-06T11:12:02.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:02.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:02.812784+0000) 2024-03-06T11:12:03.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:03.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:03.813099+0000) 2024-03-06T11:12:04.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:04.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:04.813362+0000) 2024-03-06T11:12:05.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:05.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:05.813616+0000) 2024-03-06T11:12:06.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:06.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:06.813922+0000) 2024-03-06T11:12:07.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:07.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:07.814192+0000) 2024-03-06T11:12:08.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:08.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:08.814447+0000) 2024-03-06T11:12:09.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:09.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:09.814760+0000) 2024-03-06T11:12:10.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:10.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:10.815024+0000) 2024-03-06T11:12:11.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:11.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:11.815284+0000) 2024-03-06T11:12:12.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:12.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:12.815518+0000) 2024-03-06T11:12:13.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:13.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:13.815819+0000) 2024-03-06T11:12:14.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:14.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:14.816113+0000) 2024-03-06T11:12:15.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:15.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:15.816414+0000) 2024-03-06T11:12:16.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:16.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:16.816778+0000) 2024-03-06T11:12:17.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:17.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:17.817096+0000) 2024-03-06T11:12:18.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:18.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:18.817414+0000) 2024-03-06T11:12:19.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:19.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:19.817751+0000) 2024-03-06T11:12:20.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:20.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:20.818003+0000) 2024-03-06T11:12:21.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:21.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:21.818265+0000) 2024-03-06T11:12:22.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:22.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:22.818544+0000) 2024-03-06T11:12:23.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:23.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:23.818772+0000) 2024-03-06T11:12:24.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:24.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:24.819062+0000) 2024-03-06T11:12:25.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:25.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:25.819337+0000) 2024-03-06T11:12:26.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:26.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:26.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:26.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:26.819621+0000) 2024-03-06T11:12:27.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:27.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:27.819982+0000) 2024-03-06T11:12:28.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:28.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:28.820246+0000) 2024-03-06T11:12:29.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:29.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:29.820519+0000) 2024-03-06T11:12:30.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:30.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:30.820798+0000) 2024-03-06T11:12:31.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:31.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:31.821047+0000) 2024-03-06T11:12:32.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:32.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:32.821246+0000) 2024-03-06T11:12:33.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:33.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:33.821524+0000) 2024-03-06T11:12:34.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:34.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:34.821835+0000) 2024-03-06T11:12:35.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:35.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:35.822093+0000) 2024-03-06T11:12:36.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:36.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:36.822343+0000) 2024-03-06T11:12:37.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:37.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:37.822608+0000) 2024-03-06T11:12:38.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:38.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:38.822940+0000) 2024-03-06T11:12:39.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:39.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:39.823215+0000) 2024-03-06T11:12:40.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:40.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:40.823480+0000) 2024-03-06T11:12:41.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:41.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:41.823767+0000) 2024-03-06T11:12:42.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:42.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:42.823985+0000) 2024-03-06T11:12:43.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:43.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:43.824254+0000) 2024-03-06T11:12:44.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:44.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:44.824525+0000) 2024-03-06T11:12:45.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:45.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:45.824791+0000) 2024-03-06T11:12:46.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:46.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:46.825043+0000) 2024-03-06T11:12:47.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:47.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:47.825327+0000) 2024-03-06T11:12:48.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:48.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:48.825591+0000) 2024-03-06T11:12:49.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:49.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:49.825875+0000) 2024-03-06T11:12:50.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:50.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:50.826123+0000) 2024-03-06T11:12:51.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:51.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:51.826378+0000) 2024-03-06T11:12:52.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:52.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:52.826665+0000) 2024-03-06T11:12:53.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:53.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:53.826923+0000) 2024-03-06T11:12:54.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:54.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:54.827187+0000) 2024-03-06T11:12:55.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:55.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:55.827427+0000) 2024-03-06T11:12:56.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:56.591 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:56.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:56.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:56.827745+0000) 2024-03-06T11:12:57.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:57.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:57.828019+0000) 2024-03-06T11:12:58.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:58.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:58.828301+0000) 2024-03-06T11:12:59.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:12:59.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:59.828563+0000) 2024-03-06T11:13:00.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:00.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:00.828821+0000) 2024-03-06T11:13:01.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:01.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:01.829098+0000) 2024-03-06T11:13:02.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:02.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:02.829420+0000) 2024-03-06T11:13:03.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:03.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:03.829709+0000) 2024-03-06T11:13:04.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:04.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:04.830005+0000) 2024-03-06T11:13:05.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:05.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:05.830272+0000) 2024-03-06T11:13:06.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:06.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:06.830548+0000) 2024-03-06T11:13:07.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:07.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:07.830842+0000) 2024-03-06T11:13:08.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:08.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:08.831119+0000) 2024-03-06T11:13:09.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:09.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:09.831380+0000) 2024-03-06T11:13:10.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:10.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:10.831693+0000) 2024-03-06T11:13:11.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:11.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:11.831972+0000) 2024-03-06T11:13:12.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:12.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:12.832237+0000) 2024-03-06T11:13:13.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:13.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:13.832510+0000) 2024-03-06T11:13:14.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:14.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:14.832772+0000) 2024-03-06T11:13:15.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:15.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:15.833041+0000) 2024-03-06T11:13:16.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:16.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:16.833309+0000) 2024-03-06T11:13:17.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:17.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:17.833530+0000) 2024-03-06T11:13:18.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:18.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:18.833778+0000) 2024-03-06T11:13:19.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:19.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:19.833968+0000) 2024-03-06T11:13:20.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:20.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:20.834209+0000) 2024-03-06T11:13:21.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:21.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:21.834471+0000) 2024-03-06T11:13:22.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:22.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:22.834755+0000) 2024-03-06T11:13:23.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:23.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:23.834997+0000) 2024-03-06T11:13:24.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:24.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:24.835237+0000) 2024-03-06T11:13:25.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:25.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:25.835482+0000) 2024-03-06T11:13:26.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:26.729 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:26.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:26.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:26.835758+0000) 2024-03-06T11:13:27.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:27.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:27.836019+0000) 2024-03-06T11:13:28.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:28.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:28.836283+0000) 2024-03-06T11:13:29.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:29.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:29.836550+0000) 2024-03-06T11:13:30.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:30.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:30.836842+0000) 2024-03-06T11:13:31.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:31.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:31.837119+0000) 2024-03-06T11:13:32.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:32.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:32.837387+0000) 2024-03-06T11:13:33.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:33.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:33.837677+0000) 2024-03-06T11:13:34.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:34.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:34.837930+0000) 2024-03-06T11:13:35.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:35.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:35.838185+0000) 2024-03-06T11:13:36.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:36.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:36.838450+0000) 2024-03-06T11:13:37.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:37.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:37.838736+0000) 2024-03-06T11:13:38.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:38.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:38.838997+0000) 2024-03-06T11:13:39.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:39.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:39.839259+0000) 2024-03-06T11:13:40.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:40.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:40.839502+0000) 2024-03-06T11:13:41.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:41.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:41.839765+0000) 2024-03-06T11:13:42.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:42.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:42.840013+0000) 2024-03-06T11:13:43.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:43.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:43.840251+0000) 2024-03-06T11:13:44.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:44.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:44.840571+0000) 2024-03-06T11:13:45.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:45.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:45.840834+0000) 2024-03-06T11:13:46.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:46.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:46.841077+0000) 2024-03-06T11:13:47.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:47.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:47.841328+0000) 2024-03-06T11:13:48.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:48.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:48.841588+0000) 2024-03-06T11:13:49.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:49.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:49.841856+0000) 2024-03-06T11:13:50.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:50.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:50.842117+0000) 2024-03-06T11:13:51.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:51.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:51.842377+0000) 2024-03-06T11:13:52.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:52.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:52.842672+0000) 2024-03-06T11:13:53.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:53.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:53.842911+0000) 2024-03-06T11:13:54.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:54.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:54.843159+0000) 2024-03-06T11:13:55.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:55.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:55.843415+0000) 2024-03-06T11:13:56.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:56.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:56.843729+0000) 2024-03-06T11:13:56.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:56.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:57.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:57.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:57.843951+0000) 2024-03-06T11:13:58.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:58.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:58.844206+0000) 2024-03-06T11:13:59.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:13:59.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:59.844472+0000) 2024-03-06T11:14:00.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:00.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:00.844747+0000) 2024-03-06T11:14:01.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:01.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:01.844942+0000) 2024-03-06T11:14:02.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:02.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:02.845196+0000) 2024-03-06T11:14:03.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:03.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:03.845448+0000) 2024-03-06T11:14:04.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:04.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:04.845732+0000) 2024-03-06T11:14:05.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:05.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:05.845971+0000) 2024-03-06T11:14:06.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:06.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:06.846219+0000) 2024-03-06T11:14:07.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:07.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:07.846417+0000) 2024-03-06T11:14:08.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:08.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:08.846675+0000) 2024-03-06T11:14:09.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:09.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:09.846926+0000) 2024-03-06T11:14:10.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:10.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:10.847123+0000) 2024-03-06T11:14:11.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:11.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:11.847384+0000) 2024-03-06T11:14:12.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:12.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:12.847617+0000) 2024-03-06T11:14:13.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:13.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:13.847910+0000) 2024-03-06T11:14:14.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:14.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:14.848181+0000) 2024-03-06T11:14:15.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:15.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:15.848439+0000) 2024-03-06T11:14:16.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:16.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:16.848739+0000) 2024-03-06T11:14:17.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:17.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:17.848983+0000) 2024-03-06T11:14:18.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:18.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:18.849181+0000) 2024-03-06T11:14:19.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:19.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:19.849432+0000) 2024-03-06T11:14:20.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:20.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:20.849732+0000) 2024-03-06T11:14:21.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:21.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:21.849991+0000) 2024-03-06T11:14:22.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:22.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:22.850233+0000) 2024-03-06T11:14:23.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:23.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:23.850484+0000) 2024-03-06T11:14:24.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:24.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:24.850764+0000) 2024-03-06T11:14:25.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:25.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:25.851020+0000) 2024-03-06T11:14:26.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:26.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:26.851286+0000) 2024-03-06T11:14:27.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:27.026 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:27.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:27.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:27.851537+0000) 2024-03-06T11:14:28.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:28.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:28.851789+0000) 2024-03-06T11:14:29.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:29.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:29.852059+0000) 2024-03-06T11:14:30.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:30.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:30.852296+0000) 2024-03-06T11:14:31.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:31.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:31.852569+0000) 2024-03-06T11:14:32.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:32.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:32.852882+0000) 2024-03-06T11:14:33.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:33.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:33.853086+0000) 2024-03-06T11:14:34.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:34.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:34.853356+0000) 2024-03-06T11:14:35.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:35.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:35.853596+0000) 2024-03-06T11:14:36.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:36.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:36.853854+0000) 2024-03-06T11:14:37.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:37.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:37.854107+0000) 2024-03-06T11:14:38.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:38.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:38.854363+0000) 2024-03-06T11:14:39.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:39.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:39.854618+0000) 2024-03-06T11:14:40.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:40.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:40.854854+0000) 2024-03-06T11:14:41.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:41.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:41.855120+0000) 2024-03-06T11:14:42.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:42.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:42.855358+0000) 2024-03-06T11:14:43.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:43.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:43.855605+0000) 2024-03-06T11:14:44.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:44.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:44.855899+0000) 2024-03-06T11:14:45.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:45.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:45.856160+0000) 2024-03-06T11:14:46.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:46.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:46.856425+0000) 2024-03-06T11:14:47.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:47.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:47.856723+0000) 2024-03-06T11:14:48.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:48.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:48.856973+0000) 2024-03-06T11:14:49.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:49.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:49.857243+0000) 2024-03-06T11:14:50.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:50.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:50.857507+0000) 2024-03-06T11:14:51.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:51.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:51.857775+0000) 2024-03-06T11:14:52.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:52.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:52.857996+0000) 2024-03-06T11:14:53.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:53.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:53.858253+0000) 2024-03-06T11:14:54.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:54.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:54.858520+0000) 2024-03-06T11:14:55.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:55.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:55.858773+0000) 2024-03-06T11:14:56.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:56.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:56.859042+0000) 2024-03-06T11:14:57.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:57.186 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:57.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:57.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:57.859239+0000) 2024-03-06T11:14:58.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:58.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:58.859502+0000) 2024-03-06T11:14:59.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:14:59.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:59.859763+0000) 2024-03-06T11:15:00.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:00.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:00.860009+0000) 2024-03-06T11:15:01.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:01.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:01.860252+0000) 2024-03-06T11:15:02.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:02.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:02.860494+0000) 2024-03-06T11:15:03.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:03.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:03.860735+0000) 2024-03-06T11:15:04.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:04.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:04.860986+0000) 2024-03-06T11:15:05.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:05.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:05.861247+0000) 2024-03-06T11:15:06.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:06.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:06.861500+0000) 2024-03-06T11:15:07.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:07.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:07.861754+0000) 2024-03-06T11:15:08.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:08.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:08.862006+0000) 2024-03-06T11:15:09.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:09.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:09.862272+0000) 2024-03-06T11:15:10.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:10.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:10.862525+0000) 2024-03-06T11:15:11.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:11.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:11.862818+0000) 2024-03-06T11:15:12.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:12.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:12.863054+0000) 2024-03-06T11:15:13.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:13.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:13.863307+0000) 2024-03-06T11:15:14.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:14.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:14.863578+0000) 2024-03-06T11:15:15.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:15.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:15.863838+0000) 2024-03-06T11:15:16.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:16.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:16.864109+0000) 2024-03-06T11:15:17.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:17.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:17.864377+0000) 2024-03-06T11:15:18.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:18.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:18.864576+0000) 2024-03-06T11:15:19.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:19.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:19.864839+0000) 2024-03-06T11:15:20.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:20.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:20.865160+0000) 2024-03-06T11:15:21.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:21.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:21.865409+0000) 2024-03-06T11:15:22.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:22.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:22.865664+0000) 2024-03-06T11:15:23.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:23.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:23.865957+0000) 2024-03-06T11:15:24.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:24.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:24.866166+0000) 2024-03-06T11:15:25.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:25.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:25.866424+0000) 2024-03-06T11:15:26.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:26.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:26.866715+0000) 2024-03-06T11:15:27.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:27.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:27.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:27.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:27.866992+0000) 2024-03-06T11:15:28.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:28.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:28.867245+0000) 2024-03-06T11:15:29.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:29.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:29.867497+0000) 2024-03-06T11:15:30.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:30.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:30.867758+0000) 2024-03-06T11:15:31.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:31.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:31.868027+0000) 2024-03-06T11:15:32.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:32.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:32.868268+0000) 2024-03-06T11:15:33.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:33.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:33.868505+0000) 2024-03-06T11:15:34.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:34.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:34.868755+0000) 2024-03-06T11:15:35.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:35.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:35.869009+0000) 2024-03-06T11:15:36.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:36.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:36.869276+0000) 2024-03-06T11:15:37.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:37.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:37.869526+0000) 2024-03-06T11:15:38.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:38.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:38.869805+0000) 2024-03-06T11:15:39.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:39.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:39.870078+0000) 2024-03-06T11:15:40.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:40.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:40.870353+0000) 2024-03-06T11:15:41.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:41.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:41.870662+0000) 2024-03-06T11:15:42.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:42.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:42.870930+0000) 2024-03-06T11:15:43.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:43.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:43.871188+0000) 2024-03-06T11:15:44.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:44.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:44.871447+0000) 2024-03-06T11:15:45.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:45.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:45.871746+0000) 2024-03-06T11:15:46.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:46.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:46.872010+0000) 2024-03-06T11:15:47.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:47.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:47.872273+0000) 2024-03-06T11:15:48.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:48.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:48.872518+0000) 2024-03-06T11:15:49.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:49.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:49.872767+0000) 2024-03-06T11:15:50.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:50.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:50.873053+0000) 2024-03-06T11:15:51.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:51.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:51.873337+0000) 2024-03-06T11:15:52.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:52.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:52.873599+0000) 2024-03-06T11:15:53.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:53.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:53.873920+0000) 2024-03-06T11:15:54.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:54.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:54.874176+0000) 2024-03-06T11:15:55.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:55.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:55.874417+0000) 2024-03-06T11:15:56.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:56.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:56.874714+0000) 2024-03-06T11:15:57.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:57.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:57.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:57.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:57.874958+0000) 2024-03-06T11:15:58.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:58.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:58.875209+0000) 2024-03-06T11:15:59.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:15:59.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:59.875451+0000) 2024-03-06T11:16:00.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:00.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:00.875744+0000) 2024-03-06T11:16:01.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:01.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:01.875997+0000) 2024-03-06T11:16:02.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:02.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:02.876253+0000) 2024-03-06T11:16:03.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:03.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:03.876512+0000) 2024-03-06T11:16:04.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:04.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:04.876768+0000) 2024-03-06T11:16:05.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:05.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:05.877025+0000) 2024-03-06T11:16:06.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:06.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:06.877318+0000) 2024-03-06T11:16:07.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:07.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:07.877598+0000) 2024-03-06T11:16:08.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:08.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:08.877894+0000) 2024-03-06T11:16:09.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:09.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:09.878145+0000) 2024-03-06T11:16:10.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:10.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:10.878420+0000) 2024-03-06T11:16:11.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:11.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:11.878744+0000) 2024-03-06T11:16:12.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:12.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:12.878997+0000) 2024-03-06T11:16:13.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:13.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:13.879255+0000) 2024-03-06T11:16:14.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:14.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:14.879521+0000) 2024-03-06T11:16:15.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:15.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:15.879783+0000) 2024-03-06T11:16:16.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:16.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:16.880036+0000) 2024-03-06T11:16:17.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:17.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:17.880315+0000) 2024-03-06T11:16:18.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:18.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:18.880595+0000) 2024-03-06T11:16:19.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:19.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:19.880863+0000) 2024-03-06T11:16:20.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:20.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:20.881125+0000) 2024-03-06T11:16:21.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:21.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:21.881375+0000) 2024-03-06T11:16:22.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:22.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:22.881663+0000) 2024-03-06T11:16:23.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:23.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:23.881936+0000) 2024-03-06T11:16:24.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:24.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:24.882197+0000) 2024-03-06T11:16:25.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:25.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:25.882494+0000) 2024-03-06T11:16:26.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:26.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:26.882782+0000) 2024-03-06T11:16:27.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:27.623 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:27.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:27.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:27.883045+0000) 2024-03-06T11:16:28.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:28.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:28.883310+0000) 2024-03-06T11:16:29.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:29.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:29.883576+0000) 2024-03-06T11:16:30.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:30.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:30.883854+0000) 2024-03-06T11:16:31.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:31.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:31.884085+0000) 2024-03-06T11:16:32.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:32.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:32.884357+0000) 2024-03-06T11:16:33.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:33.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:33.884615+0000) 2024-03-06T11:16:34.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:34.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:34.884921+0000) 2024-03-06T11:16:35.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:35.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:35.885208+0000) 2024-03-06T11:16:36.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:36.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:36.885478+0000) 2024-03-06T11:16:37.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:37.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:37.885740+0000) 2024-03-06T11:16:38.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:38.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:38.886020+0000) 2024-03-06T11:16:39.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:39.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:39.886294+0000) 2024-03-06T11:16:40.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:40.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:40.886563+0000) 2024-03-06T11:16:41.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:41.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:41.886836+0000) 2024-03-06T11:16:42.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:42.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:42.887104+0000) 2024-03-06T11:16:43.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:43.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:43.887349+0000) 2024-03-06T11:16:44.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:44.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:44.887668+0000) 2024-03-06T11:16:45.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:45.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:45.887938+0000) 2024-03-06T11:16:46.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:46.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:46.888228+0000) 2024-03-06T11:16:47.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:47.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:47.888495+0000) 2024-03-06T11:16:48.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:48.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:48.888768+0000) 2024-03-06T11:16:49.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:49.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:49.889010+0000) 2024-03-06T11:16:50.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:50.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:50.889274+0000) 2024-03-06T11:16:51.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:51.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:51.889539+0000) 2024-03-06T11:16:52.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:52.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:52.889794+0000) 2024-03-06T11:16:53.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:53.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:53.890056+0000) 2024-03-06T11:16:54.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:54.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:54.890256+0000) 2024-03-06T11:16:55.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:55.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:55.890522+0000) 2024-03-06T11:16:56.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:56.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:56.890783+0000) 2024-03-06T11:16:57.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:57.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:57.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:57.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:57.890945+0000) 2024-03-06T11:16:58.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:58.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:58.891197+0000) 2024-03-06T11:16:59.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:16:59.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:59.891486+0000) 2024-03-06T11:17:00.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:00.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:00.891764+0000) 2024-03-06T11:17:01.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:01.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:01.892041+0000) 2024-03-06T11:17:02.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:02.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:02.892300+0000) 2024-03-06T11:17:03.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:03.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:03.892562+0000) 2024-03-06T11:17:04.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:04.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:04.892825+0000) 2024-03-06T11:17:05.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:05.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:05.893085+0000) 2024-03-06T11:17:06.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:06.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:06.893348+0000) 2024-03-06T11:17:07.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:07.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:07.893700+0000) 2024-03-06T11:17:08.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:08.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:08.893949+0000) 2024-03-06T11:17:09.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:09.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:09.894205+0000) 2024-03-06T11:17:10.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:10.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:10.894451+0000) 2024-03-06T11:17:11.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:11.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:11.894739+0000) 2024-03-06T11:17:12.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:12.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:12.894992+0000) 2024-03-06T11:17:13.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:13.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:13.895227+0000) 2024-03-06T11:17:14.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:14.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:14.895480+0000) 2024-03-06T11:17:15.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:15.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:15.895747+0000) 2024-03-06T11:17:16.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:16.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:16.895994+0000) 2024-03-06T11:17:17.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:17.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:17.896243+0000) 2024-03-06T11:17:18.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:18.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:18.896494+0000) 2024-03-06T11:17:19.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:19.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:19.896743+0000) 2024-03-06T11:17:20.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:20.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:20.897005+0000) 2024-03-06T11:17:21.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:21.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:21.897261+0000) 2024-03-06T11:17:22.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:22.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:22.897520+0000) 2024-03-06T11:17:23.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:23.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:23.897778+0000) 2024-03-06T11:17:24.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:24.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:24.898039+0000) 2024-03-06T11:17:25.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:25.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:25.898284+0000) 2024-03-06T11:17:26.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:26.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:26.898535+0000) 2024-03-06T11:17:27.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:27.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:27.898792+0000) 2024-03-06T11:17:27.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:27.916 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:28.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:28.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:28.899060+0000) 2024-03-06T11:17:29.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:29.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:29.899249+0000) 2024-03-06T11:17:30.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:30.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:30.899442+0000) 2024-03-06T11:17:31.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:31.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:31.899716+0000) 2024-03-06T11:17:32.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:32.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:32.899967+0000) 2024-03-06T11:17:33.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:33.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:33.900152+0000) 2024-03-06T11:17:34.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:34.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:34.900397+0000) 2024-03-06T11:17:35.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:35.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:35.900680+0000) 2024-03-06T11:17:36.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:36.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:36.900930+0000) 2024-03-06T11:17:37.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:37.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:37.901187+0000) 2024-03-06T11:17:38.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:38.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:38.901436+0000) 2024-03-06T11:17:39.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:39.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:39.901682+0000) 2024-03-06T11:17:40.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:40.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:40.901948+0000) 2024-03-06T11:17:41.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:41.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:41.902177+0000) 2024-03-06T11:17:42.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:42.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:42.902412+0000) 2024-03-06T11:17:43.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:43.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:43.902676+0000) 2024-03-06T11:17:44.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:44.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:44.902946+0000) 2024-03-06T11:17:45.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:45.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:45.903189+0000) 2024-03-06T11:17:46.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:46.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:46.903456+0000) 2024-03-06T11:17:47.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:47.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:47.903747+0000) 2024-03-06T11:17:48.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:48.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:48.903992+0000) 2024-03-06T11:17:49.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:49.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:49.904257+0000) 2024-03-06T11:17:50.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:50.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:50.904576+0000) 2024-03-06T11:17:51.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:51.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:51.904807+0000) 2024-03-06T11:17:52.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:52.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:52.905065+0000) 2024-03-06T11:17:53.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:53.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:53.905254+0000) 2024-03-06T11:17:54.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:54.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:54.905492+0000) 2024-03-06T11:17:55.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:55.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:55.905765+0000) 2024-03-06T11:17:56.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:56.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:56.906023+0000) 2024-03-06T11:17:57.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:57.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:57.906313+0000) 2024-03-06T11:17:58.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:58.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:58.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:58.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:58.906580+0000) 2024-03-06T11:17:59.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:17:59.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:59.906845+0000) 2024-03-06T11:18:00.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:00.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:00.907131+0000) 2024-03-06T11:18:01.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:01.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:01.907405+0000) 2024-03-06T11:18:02.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:02.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:02.907724+0000) 2024-03-06T11:18:03.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:03.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:03.907999+0000) 2024-03-06T11:18:04.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:04.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:04.908265+0000) 2024-03-06T11:18:05.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:05.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:05.908529+0000) 2024-03-06T11:18:06.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:06.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:06.908796+0000) 2024-03-06T11:18:07.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:07.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:07.909059+0000) 2024-03-06T11:18:08.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:08.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:08.909348+0000) 2024-03-06T11:18:09.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:09.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:09.909620+0000) 2024-03-06T11:18:10.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:10.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:10.909950+0000) 2024-03-06T11:18:11.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:11.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:11.910189+0000) 2024-03-06T11:18:12.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:12.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:12.910443+0000) 2024-03-06T11:18:13.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:13.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:13.910753+0000) 2024-03-06T11:18:14.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:14.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:14.911008+0000) 2024-03-06T11:18:15.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:15.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:15.911256+0000) 2024-03-06T11:18:16.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:16.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:16.911506+0000) 2024-03-06T11:18:17.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:17.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:17.911767+0000) 2024-03-06T11:18:18.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:18.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:18.912009+0000) 2024-03-06T11:18:19.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:19.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:19.912267+0000) 2024-03-06T11:18:20.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:20.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:20.912518+0000) 2024-03-06T11:18:21.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:21.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:21.912780+0000) 2024-03-06T11:18:22.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:22.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:22.913023+0000) 2024-03-06T11:18:23.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:23.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:23.913208+0000) 2024-03-06T11:18:24.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:24.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:24.913461+0000) 2024-03-06T11:18:25.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:25.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:25.913769+0000) 2024-03-06T11:18:26.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:26.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:26.914026+0000) 2024-03-06T11:18:27.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:27.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:27.914308+0000) 2024-03-06T11:18:28.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:28.209 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:28.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:28.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:28.914594+0000) 2024-03-06T11:18:29.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:29.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:29.914914+0000) 2024-03-06T11:18:30.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:30.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:30.915194+0000) 2024-03-06T11:18:31.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:31.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:31.915478+0000) 2024-03-06T11:18:32.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:32.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:32.915805+0000) 2024-03-06T11:18:33.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:33.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:33.916120+0000) 2024-03-06T11:18:34.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:34.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:34.916437+0000) 2024-03-06T11:18:35.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:35.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:35.916786+0000) 2024-03-06T11:18:36.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:36.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:36.917108+0000) 2024-03-06T11:18:37.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:37.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:37.917392+0000) 2024-03-06T11:18:38.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:38.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:38.917701+0000) 2024-03-06T11:18:39.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:39.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:39.917937+0000) 2024-03-06T11:18:40.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:40.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:40.918200+0000) 2024-03-06T11:18:41.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:41.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:41.918492+0000) 2024-03-06T11:18:42.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:42.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:42.918758+0000) 2024-03-06T11:18:43.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:43.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:43.919099+0000) 2024-03-06T11:18:44.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:44.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:44.919419+0000) 2024-03-06T11:18:45.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:45.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:45.919753+0000) 2024-03-06T11:18:46.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:46.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:46.920001+0000) 2024-03-06T11:18:47.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:47.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:47.920298+0000) 2024-03-06T11:18:48.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:48.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:48.920597+0000) 2024-03-06T11:18:49.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:49.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:49.920898+0000) 2024-03-06T11:18:50.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:50.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:50.921201+0000) 2024-03-06T11:18:51.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:51.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:51.921515+0000) 2024-03-06T11:18:52.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:52.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:52.921833+0000) 2024-03-06T11:18:53.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:53.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:53.922149+0000) 2024-03-06T11:18:54.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:54.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:54.922463+0000) 2024-03-06T11:18:55.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:55.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:55.922783+0000) 2024-03-06T11:18:56.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:56.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:56.923106+0000) 2024-03-06T11:18:57.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:57.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:57.923355+0000) 2024-03-06T11:18:58.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:58.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:58.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:58.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:58.923612+0000) 2024-03-06T11:18:59.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:18:59.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:59.924010+0000) 2024-03-06T11:19:00.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:00.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:00.924309+0000) 2024-03-06T11:19:01.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:01.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:01.924611+0000) 2024-03-06T11:19:02.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:02.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:02.924944+0000) 2024-03-06T11:19:03.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:03.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:03.925162+0000) 2024-03-06T11:19:04.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:04.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:04.925456+0000) 2024-03-06T11:19:05.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:05.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:05.925784+0000) 2024-03-06T11:19:06.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:06.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:06.926029+0000) 2024-03-06T11:19:07.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:07.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:07.926309+0000) 2024-03-06T11:19:08.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:08.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:08.926553+0000) 2024-03-06T11:19:09.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:09.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:09.926825+0000) 2024-03-06T11:19:10.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:10.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:10.927118+0000) 2024-03-06T11:19:11.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:11.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:11.927429+0000) 2024-03-06T11:19:12.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:12.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:12.927712+0000) 2024-03-06T11:19:13.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:13.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:13.927997+0000) 2024-03-06T11:19:14.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:14.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:14.928300+0000) 2024-03-06T11:19:15.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:15.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:15.928607+0000) 2024-03-06T11:19:16.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:16.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:16.928963+0000) 2024-03-06T11:19:17.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:17.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:17.929238+0000) 2024-03-06T11:19:18.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:18.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:18.929446+0000) 2024-03-06T11:19:19.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:19.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:19.929777+0000) 2024-03-06T11:19:20.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:20.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:20.930074+0000) 2024-03-06T11:19:21.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:21.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:21.930380+0000) 2024-03-06T11:19:22.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:22.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:22.930677+0000) 2024-03-06T11:19:23.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:23.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:23.930984+0000) 2024-03-06T11:19:24.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:24.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:24.931264+0000) 2024-03-06T11:19:25.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:25.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:25.931567+0000) 2024-03-06T11:19:26.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:26.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:26.931880+0000) 2024-03-06T11:19:27.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:27.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:27.932165+0000) 2024-03-06T11:19:28.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:28.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:28.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:28.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:28.932429+0000) 2024-03-06T11:19:29.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:29.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:29.932771+0000) 2024-03-06T11:19:30.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:30.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:30.933076+0000) 2024-03-06T11:19:31.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:31.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:31.933365+0000) 2024-03-06T11:19:32.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:32.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:32.933660+0000) 2024-03-06T11:19:33.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:33.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:33.933950+0000) 2024-03-06T11:19:34.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:34.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:34.934184+0000) 2024-03-06T11:19:35.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:35.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:35.934520+0000) 2024-03-06T11:19:36.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:36.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:36.934832+0000) 2024-03-06T11:19:37.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:37.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:37.935103+0000) 2024-03-06T11:19:38.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:38.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:38.935303+0000) 2024-03-06T11:19:39.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:39.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:39.935572+0000) 2024-03-06T11:19:40.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:40.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:40.935879+0000) 2024-03-06T11:19:41.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:41.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:41.936184+0000) 2024-03-06T11:19:42.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:42.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:42.936456+0000) 2024-03-06T11:19:43.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:43.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:43.936792+0000) 2024-03-06T11:19:44.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:44.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:44.937092+0000) 2024-03-06T11:19:45.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:45.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:45.937414+0000) 2024-03-06T11:19:46.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:46.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:46.937783+0000) 2024-03-06T11:19:47.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:47.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:47.938075+0000) 2024-03-06T11:19:48.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:48.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:48.938264+0000) 2024-03-06T11:19:49.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:49.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:49.938565+0000) 2024-03-06T11:19:50.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:50.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:50.938855+0000) 2024-03-06T11:19:51.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:51.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:51.939152+0000) 2024-03-06T11:19:52.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:52.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:52.939423+0000) 2024-03-06T11:19:53.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:53.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:53.939740+0000) 2024-03-06T11:19:54.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:54.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:54.940042+0000) 2024-03-06T11:19:55.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:55.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:55.940331+0000) 2024-03-06T11:19:56.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:56.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:56.940687+0000) 2024-03-06T11:19:57.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:57.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:57.940964+0000) 2024-03-06T11:19:58.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:58.676 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:58.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:58.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:58.941217+0000) 2024-03-06T11:19:59.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:19:59.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:59.941509+0000) 2024-03-06T11:20:00.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:00.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:00.941805+0000) 2024-03-06T11:20:01.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:01.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:01.942122+0000) 2024-03-06T11:20:02.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:02.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:02.942323+0000) 2024-03-06T11:20:03.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:03.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:03.942585+0000) 2024-03-06T11:20:04.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:04.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:04.942904+0000) 2024-03-06T11:20:05.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:05.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:05.943154+0000) 2024-03-06T11:20:06.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:06.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:06.943415+0000) 2024-03-06T11:20:07.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:07.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:07.943727+0000) 2024-03-06T11:20:08.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:08.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:08.943922+0000) 2024-03-06T11:20:09.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:09.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:09.944156+0000) 2024-03-06T11:20:10.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:10.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:10.944412+0000) 2024-03-06T11:20:11.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:11.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:11.944685+0000) 2024-03-06T11:20:12.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:12.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:12.944943+0000) 2024-03-06T11:20:13.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:13.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:13.945184+0000) 2024-03-06T11:20:14.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:14.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:14.945394+0000) 2024-03-06T11:20:15.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:15.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:15.945721+0000) 2024-03-06T11:20:16.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:16.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:16.945941+0000) 2024-03-06T11:20:17.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:17.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:17.946151+0000) 2024-03-06T11:20:18.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:18.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:18.946416+0000) 2024-03-06T11:20:19.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:19.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:19.946737+0000) 2024-03-06T11:20:20.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:20.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:20.946983+0000) 2024-03-06T11:20:21.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:21.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:21.947222+0000) 2024-03-06T11:20:22.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:22.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:22.947476+0000) 2024-03-06T11:20:23.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:23.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:23.947747+0000) 2024-03-06T11:20:24.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:24.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:24.948008+0000) 2024-03-06T11:20:25.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:25.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:25.948245+0000) 2024-03-06T11:20:26.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:26.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:26.948514+0000) 2024-03-06T11:20:27.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:27.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:27.948780+0000) 2024-03-06T11:20:28.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:28.818 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:28.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:28.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:28.949050+0000) 2024-03-06T11:20:29.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:29.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:29.949301+0000) 2024-03-06T11:20:30.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:30.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:30.949537+0000) 2024-03-06T11:20:31.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:31.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:31.949800+0000) 2024-03-06T11:20:32.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:32.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:32.950061+0000) 2024-03-06T11:20:33.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:33.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:33.950302+0000) 2024-03-06T11:20:34.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:34.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:34.950544+0000) 2024-03-06T11:20:35.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:35.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:35.950782+0000) 2024-03-06T11:20:36.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:36.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:36.951040+0000) 2024-03-06T11:20:37.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:37.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:37.951319+0000) 2024-03-06T11:20:38.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:38.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:38.951586+0000) 2024-03-06T11:20:39.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:39.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:39.951844+0000) 2024-03-06T11:20:40.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:40.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:40.952108+0000) 2024-03-06T11:20:41.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:41.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:41.952299+0000) 2024-03-06T11:20:42.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:42.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:42.952537+0000) 2024-03-06T11:20:43.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:43.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:43.952775+0000) 2024-03-06T11:20:44.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:44.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:44.952966+0000) 2024-03-06T11:20:45.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:45.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:45.953222+0000) 2024-03-06T11:20:46.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:46.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:46.953465+0000) 2024-03-06T11:20:47.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:47.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:47.953751+0000) 2024-03-06T11:20:48.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:48.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:48.953997+0000) 2024-03-06T11:20:49.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:49.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:49.954181+0000) 2024-03-06T11:20:50.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:50.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:50.954439+0000) 2024-03-06T11:20:51.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:51.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:51.954736+0000) 2024-03-06T11:20:52.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:52.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:52.954947+0000) 2024-03-06T11:20:53.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:53.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:53.955189+0000) 2024-03-06T11:20:54.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:54.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:54.955448+0000) 2024-03-06T11:20:55.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:55.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:55.955739+0000) 2024-03-06T11:20:56.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:56.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:56.956004+0000) 2024-03-06T11:20:57.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:57.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:57.956262+0000) 2024-03-06T11:20:58.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:58.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:58.956504+0000) 2024-03-06T11:20:58.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:58.980 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:59.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:20:59.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:59.956766+0000) 2024-03-06T11:21:00.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:00.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:00.957017+0000) 2024-03-06T11:21:01.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:01.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:01.957274+0000) 2024-03-06T11:21:02.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:02.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:02.957532+0000) 2024-03-06T11:21:03.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:03.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:03.957783+0000) 2024-03-06T11:21:04.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:04.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:04.958030+0000) 2024-03-06T11:21:05.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:05.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:05.958276+0000) 2024-03-06T11:21:06.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:06.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:06.958529+0000) 2024-03-06T11:21:07.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:07.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:07.958789+0000) 2024-03-06T11:21:08.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:08.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:08.959049+0000) 2024-03-06T11:21:09.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:09.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:09.959311+0000) 2024-03-06T11:21:10.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:10.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:10.959570+0000) 2024-03-06T11:21:11.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:11.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:11.959811+0000) 2024-03-06T11:21:12.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:12.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:12.960088+0000) 2024-03-06T11:21:13.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:13.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:13.960346+0000) 2024-03-06T11:21:14.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:14.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:14.960615+0000) 2024-03-06T11:21:15.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:15.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:15.960905+0000) 2024-03-06T11:21:16.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:16.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:16.961149+0000) 2024-03-06T11:21:17.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:17.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:17.961429+0000) 2024-03-06T11:21:18.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:18.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:18.961779+0000) 2024-03-06T11:21:19.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:19.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:19.962022+0000) 2024-03-06T11:21:20.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:20.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:20.962268+0000) 2024-03-06T11:21:21.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:21.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:21.962529+0000) 2024-03-06T11:21:22.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:22.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:22.962781+0000) 2024-03-06T11:21:23.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:23.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:23.963012+0000) 2024-03-06T11:21:24.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:24.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:24.963203+0000) 2024-03-06T11:21:25.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:25.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:25.963391+0000) 2024-03-06T11:21:26.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:26.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:26.963675+0000) 2024-03-06T11:21:27.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:27.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:27.963894+0000) 2024-03-06T11:21:28.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:28.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:28.964100+0000) 2024-03-06T11:21:29.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:29.120 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:29.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:29.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:29.964370+0000) 2024-03-06T11:21:30.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:30.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:30.964678+0000) 2024-03-06T11:21:31.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:31.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:31.964878+0000) 2024-03-06T11:21:32.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:32.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:32.965147+0000) 2024-03-06T11:21:33.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:33.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:33.965389+0000) 2024-03-06T11:21:34.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:34.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:34.965670+0000) 2024-03-06T11:21:35.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:35.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:35.965921+0000) 2024-03-06T11:21:36.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:36.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:36.966176+0000) 2024-03-06T11:21:37.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:37.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:37.966380+0000) 2024-03-06T11:21:38.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:38.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:38.966574+0000) 2024-03-06T11:21:39.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:39.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:39.966815+0000) 2024-03-06T11:21:40.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:40.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:40.967072+0000) 2024-03-06T11:21:41.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:41.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:41.967331+0000) 2024-03-06T11:21:42.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:42.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:42.967585+0000) 2024-03-06T11:21:43.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:43.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:43.967831+0000) 2024-03-06T11:21:44.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:44.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:44.968124+0000) 2024-03-06T11:21:45.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:45.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:45.968396+0000) 2024-03-06T11:21:46.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:46.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:46.968618+0000) 2024-03-06T11:21:47.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:47.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:47.968942+0000) 2024-03-06T11:21:48.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:48.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:48.969223+0000) 2024-03-06T11:21:49.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:49.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:49.969501+0000) 2024-03-06T11:21:50.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:50.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:50.969780+0000) 2024-03-06T11:21:51.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:51.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:51.970057+0000) 2024-03-06T11:21:52.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:52.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:52.970304+0000) 2024-03-06T11:21:53.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:53.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:53.970545+0000) 2024-03-06T11:21:54.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:54.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:54.970809+0000) 2024-03-06T11:21:55.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:55.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:55.971057+0000) 2024-03-06T11:21:56.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:56.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:56.971327+0000) 2024-03-06T11:21:57.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:57.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:57.971569+0000) 2024-03-06T11:21:58.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:58.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:58.971766+0000) 2024-03-06T11:21:59.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:59.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:59.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:21:59.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:59.972033+0000) 2024-03-06T11:22:00.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:00.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:00.972347+0000) 2024-03-06T11:22:01.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:01.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:01.972617+0000) 2024-03-06T11:22:02.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:02.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:02.972903+0000) 2024-03-06T11:22:03.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:03.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:03.973139+0000) 2024-03-06T11:22:04.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:04.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:04.973389+0000) 2024-03-06T11:22:05.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:05.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:05.973618+0000) 2024-03-06T11:22:06.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:06.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:06.973920+0000) 2024-03-06T11:22:07.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:07.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:07.974185+0000) 2024-03-06T11:22:08.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:08.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:08.974449+0000) 2024-03-06T11:22:09.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:09.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:09.974735+0000) 2024-03-06T11:22:10.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:10.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:10.974998+0000) 2024-03-06T11:22:11.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:11.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:11.975243+0000) 2024-03-06T11:22:12.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:12.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:12.975480+0000) 2024-03-06T11:22:13.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:13.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:13.975742+0000) 2024-03-06T11:22:14.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:14.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:14.975987+0000) 2024-03-06T11:22:15.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:15.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:15.976229+0000) 2024-03-06T11:22:16.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:16.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:16.976500+0000) 2024-03-06T11:22:17.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:17.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:17.976745+0000) 2024-03-06T11:22:18.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:18.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:18.976946+0000) 2024-03-06T11:22:19.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:19.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:19.977195+0000) 2024-03-06T11:22:20.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:20.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:20.977426+0000) 2024-03-06T11:22:21.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:21.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:21.977718+0000) 2024-03-06T11:22:22.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:22.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:22.977964+0000) 2024-03-06T11:22:23.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:23.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:23.978208+0000) 2024-03-06T11:22:24.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:24.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:24.978467+0000) 2024-03-06T11:22:25.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:25.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:25.978750+0000) 2024-03-06T11:22:26.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:26.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:26.979015+0000) 2024-03-06T11:22:27.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:27.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:27.979288+0000) 2024-03-06T11:22:28.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:28.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:28.979530+0000) 2024-03-06T11:22:29.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:29.440 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:29.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:29.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:29.979768+0000) 2024-03-06T11:22:30.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:30.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:30.980031+0000) 2024-03-06T11:22:31.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:31.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:31.980271+0000) 2024-03-06T11:22:32.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:32.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:32.980523+0000) 2024-03-06T11:22:33.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:33.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:33.980792+0000) 2024-03-06T11:22:34.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:34.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:34.981057+0000) 2024-03-06T11:22:35.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:35.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:35.981305+0000) 2024-03-06T11:22:36.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:36.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:36.981570+0000) 2024-03-06T11:22:37.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:37.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:37.981736+0000) 2024-03-06T11:22:38.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:38.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:38.981979+0000) 2024-03-06T11:22:39.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:39.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:39.982236+0000) 2024-03-06T11:22:40.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:40.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:40.982482+0000) 2024-03-06T11:22:41.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:41.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:41.982751+0000) 2024-03-06T11:22:42.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:42.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:42.983015+0000) 2024-03-06T11:22:43.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:43.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:43.983204+0000) 2024-03-06T11:22:44.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:44.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:44.983454+0000) 2024-03-06T11:22:45.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:45.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:45.983743+0000) 2024-03-06T11:22:46.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:46.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:46.984009+0000) 2024-03-06T11:22:47.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:47.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:47.984267+0000) 2024-03-06T11:22:48.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:48.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:48.984608+0000) 2024-03-06T11:22:49.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:49.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:49.984891+0000) 2024-03-06T11:22:50.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:50.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:50.985150+0000) 2024-03-06T11:22:51.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:51.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:51.985396+0000) 2024-03-06T11:22:52.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:52.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:52.985711+0000) 2024-03-06T11:22:53.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:53.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:53.985968+0000) 2024-03-06T11:22:54.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:54.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:54.986231+0000) 2024-03-06T11:22:55.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:55.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:55.986474+0000) 2024-03-06T11:22:56.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:56.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:56.986811+0000) 2024-03-06T11:22:57.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:57.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:57.987068+0000) 2024-03-06T11:22:58.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:58.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:58.987313+0000) 2024-03-06T11:22:59.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:59.592 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:59.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:22:59.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:59.987571+0000) 2024-03-06T11:23:00.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:00.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:00.987806+0000) 2024-03-06T11:23:01.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:01.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:01.988001+0000) 2024-03-06T11:23:02.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:02.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:02.988248+0000) 2024-03-06T11:23:03.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:03.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:03.988514+0000) 2024-03-06T11:23:04.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:04.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:04.988749+0000) 2024-03-06T11:23:05.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:05.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:05.989003+0000) 2024-03-06T11:23:06.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:06.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:06.989255+0000) 2024-03-06T11:23:07.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:07.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:07.989495+0000) 2024-03-06T11:23:08.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:08.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:08.989743+0000) 2024-03-06T11:23:09.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:09.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:09.990000+0000) 2024-03-06T11:23:10.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:10.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:10.990275+0000) 2024-03-06T11:23:11.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:11.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:11.990543+0000) 2024-03-06T11:23:12.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:12.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:12.990805+0000) 2024-03-06T11:23:13.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:13.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:13.991039+0000) 2024-03-06T11:23:14.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:14.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:14.991296+0000) 2024-03-06T11:23:15.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:15.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:15.991527+0000) 2024-03-06T11:23:16.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:16.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:16.991795+0000) 2024-03-06T11:23:17.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:17.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:17.991996+0000) 2024-03-06T11:23:18.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:18.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:18.992245+0000) 2024-03-06T11:23:19.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:19.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:19.992481+0000) 2024-03-06T11:23:20.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:20.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:20.992748+0000) 2024-03-06T11:23:21.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:21.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:21.992952+0000) 2024-03-06T11:23:22.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:22.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:22.993214+0000) 2024-03-06T11:23:23.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:23.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:23.993464+0000) 2024-03-06T11:23:24.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:24.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:24.993757+0000) 2024-03-06T11:23:25.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:25.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:25.994017+0000) 2024-03-06T11:23:26.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:26.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:26.994299+0000) 2024-03-06T11:23:27.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:27.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:27.994547+0000) 2024-03-06T11:23:28.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:28.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:28.994813+0000) 2024-03-06T11:23:29.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:29.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:29.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:29.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:29.995053+0000) 2024-03-06T11:23:30.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:30.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:30.995300+0000) 2024-03-06T11:23:31.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:31.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:31.995560+0000) 2024-03-06T11:23:32.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:32.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:32.995809+0000) 2024-03-06T11:23:33.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:33.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:33.996050+0000) 2024-03-06T11:23:34.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:34.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:34.996303+0000) 2024-03-06T11:23:35.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:35.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:35.996547+0000) 2024-03-06T11:23:36.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:36.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:36.996816+0000) 2024-03-06T11:23:37.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:37.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:37.997080+0000) 2024-03-06T11:23:38.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:38.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:38.997332+0000) 2024-03-06T11:23:39.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:39.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:39.997605+0000) 2024-03-06T11:23:40.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:40.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:40.997905+0000) 2024-03-06T11:23:41.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:41.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:41.998164+0000) 2024-03-06T11:23:42.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:42.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:42.998403+0000) 2024-03-06T11:23:43.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:43.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:43.998720+0000) 2024-03-06T11:23:44.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:44.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:44.998982+0000) 2024-03-06T11:23:45.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:45.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:45.999255+0000) 2024-03-06T11:23:46.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:46.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:46.999503+0000) 2024-03-06T11:23:47.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:47.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:47.999753+0000) 2024-03-06T11:23:48.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:48.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:48.999960+0000) 2024-03-06T11:23:50.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:49.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:50.000221+0000) 2024-03-06T11:23:51.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:50.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:51.000410+0000) 2024-03-06T11:23:52.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:51.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:52.000721+0000) 2024-03-06T11:23:53.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:52.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:53.000983+0000) 2024-03-06T11:23:54.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:53.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:54.001230+0000) 2024-03-06T11:23:55.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:55.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:55.001490+0000) 2024-03-06T11:23:56.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:56.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:56.001744+0000) 2024-03-06T11:23:57.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:57.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:57.002017+0000) 2024-03-06T11:23:58.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:58.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:58.002291+0000) 2024-03-06T11:23:59.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:23:59.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:59.002540+0000) 2024-03-06T11:23:59.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:59.961 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:00.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:00.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:00.002764+0000) 2024-03-06T11:24:01.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:01.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:01.002946+0000) 2024-03-06T11:24:02.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:02.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:02.003212+0000) 2024-03-06T11:24:03.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:03.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:03.003491+0000) 2024-03-06T11:24:04.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:04.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:04.003775+0000) 2024-03-06T11:24:05.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:05.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:05.004051+0000) 2024-03-06T11:24:06.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:06.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:06.004318+0000) 2024-03-06T11:24:07.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:07.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:07.004569+0000) 2024-03-06T11:24:08.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:08.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:08.004836+0000) 2024-03-06T11:24:09.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:09.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:09.005087+0000) 2024-03-06T11:24:10.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:10.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:10.005341+0000) 2024-03-06T11:24:11.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:11.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:11.005582+0000) 2024-03-06T11:24:12.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:12.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:12.005849+0000) 2024-03-06T11:24:13.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:13.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:13.006054+0000) 2024-03-06T11:24:14.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:14.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:14.006304+0000) 2024-03-06T11:24:15.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:15.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:15.006545+0000) 2024-03-06T11:24:16.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:16.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:16.006800+0000) 2024-03-06T11:24:17.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:17.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:17.007102+0000) 2024-03-06T11:24:18.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:18.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:18.007368+0000) 2024-03-06T11:24:19.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:19.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:19.007674+0000) 2024-03-06T11:24:20.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:20.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:20.007941+0000) 2024-03-06T11:24:21.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:21.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:21.008219+0000) 2024-03-06T11:24:22.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:22.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:22.008493+0000) 2024-03-06T11:24:23.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:23.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:23.008778+0000) 2024-03-06T11:24:24.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:24.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:24.009043+0000) 2024-03-06T11:24:25.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:25.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:25.009312+0000) 2024-03-06T11:24:26.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:26.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:26.009561+0000) 2024-03-06T11:24:27.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:27.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:27.009826+0000) 2024-03-06T11:24:28.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:28.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:28.010023+0000) 2024-03-06T11:24:29.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:29.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:29.010244+0000) 2024-03-06T11:24:30.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:30.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:30.010526+0000) 2024-03-06T11:24:30.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:30.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:31.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:31.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:31.010800+0000) 2024-03-06T11:24:32.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:32.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:32.011066+0000) 2024-03-06T11:24:33.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:33.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:33.011263+0000) 2024-03-06T11:24:34.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:34.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:34.011531+0000) 2024-03-06T11:24:35.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:35.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:35.011790+0000) 2024-03-06T11:24:36.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:36.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:36.012114+0000) 2024-03-06T11:24:37.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:37.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:37.012351+0000) 2024-03-06T11:24:38.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:38.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:38.012664+0000) 2024-03-06T11:24:39.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:39.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:39.012932+0000) 2024-03-06T11:24:40.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:40.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:40.013188+0000) 2024-03-06T11:24:41.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:41.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:41.013454+0000) 2024-03-06T11:24:42.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:42.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:42.013758+0000) 2024-03-06T11:24:43.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:43.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:43.014003+0000) 2024-03-06T11:24:44.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:44.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:44.014274+0000) 2024-03-06T11:24:45.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:45.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:45.014542+0000) 2024-03-06T11:24:46.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:46.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:46.014781+0000) 2024-03-06T11:24:47.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:47.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:47.015057+0000) 2024-03-06T11:24:48.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:48.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:48.015332+0000) 2024-03-06T11:24:49.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:49.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:49.015598+0000) 2024-03-06T11:24:50.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:50.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:50.015863+0000) 2024-03-06T11:24:51.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:51.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:51.016116+0000) 2024-03-06T11:24:52.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:52.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:52.016363+0000) 2024-03-06T11:24:53.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:53.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:53.016675+0000) 2024-03-06T11:24:54.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:54.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:54.016958+0000) 2024-03-06T11:24:55.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:55.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:55.017252+0000) 2024-03-06T11:24:56.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:56.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:56.017507+0000) 2024-03-06T11:24:57.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:57.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:57.017777+0000) 2024-03-06T11:24:58.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:58.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:58.018053+0000) 2024-03-06T11:24:59.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:24:59.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:59.018332+0000) 2024-03-06T11:25:00.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:00.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:00.018613+0000) 2024-03-06T11:25:00.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:00.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:01.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:01.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:01.018945+0000) 2024-03-06T11:25:02.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:02.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:02.019131+0000) 2024-03-06T11:25:03.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:03.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:03.019373+0000) 2024-03-06T11:25:04.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:04.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:04.019581+0000) 2024-03-06T11:25:05.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:05.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:05.019858+0000) 2024-03-06T11:25:06.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:06.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:06.020117+0000) 2024-03-06T11:25:07.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:07.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:07.020394+0000) 2024-03-06T11:25:08.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:08.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:08.020709+0000) 2024-03-06T11:25:09.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:09.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:09.020980+0000) 2024-03-06T11:25:10.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:10.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:10.021239+0000) 2024-03-06T11:25:11.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:11.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:11.021495+0000) 2024-03-06T11:25:12.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:12.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:12.021777+0000) 2024-03-06T11:25:13.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:13.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:13.022042+0000) 2024-03-06T11:25:14.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:14.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:14.022293+0000) 2024-03-06T11:25:15.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:15.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:15.022550+0000) 2024-03-06T11:25:16.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:16.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:16.022791+0000) 2024-03-06T11:25:17.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:17.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:17.022985+0000) 2024-03-06T11:25:18.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:18.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:18.023234+0000) 2024-03-06T11:25:19.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:19.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:19.023493+0000) 2024-03-06T11:25:20.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:20.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:20.023757+0000) 2024-03-06T11:25:21.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:21.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:21.024021+0000) 2024-03-06T11:25:22.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:22.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:22.024256+0000) 2024-03-06T11:25:23.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:23.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:23.024516+0000) 2024-03-06T11:25:24.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:24.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:24.024795+0000) 2024-03-06T11:25:25.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:25.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:25.025043+0000) 2024-03-06T11:25:26.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:26.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:26.025276+0000) 2024-03-06T11:25:27.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:27.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:27.025530+0000) 2024-03-06T11:25:28.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:28.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:28.025802+0000) 2024-03-06T11:25:29.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:29.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:29.026109+0000) 2024-03-06T11:25:30.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:30.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:30.026348+0000) 2024-03-06T11:25:30.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:30.404 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:31.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:31.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:31.026621+0000) 2024-03-06T11:25:32.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:32.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:32.026861+0000) 2024-03-06T11:25:33.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:33.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:33.027128+0000) 2024-03-06T11:25:34.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:34.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:34.027380+0000) 2024-03-06T11:25:35.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:35.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:35.027687+0000) 2024-03-06T11:25:36.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:36.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:36.027954+0000) 2024-03-06T11:25:37.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:37.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:37.028221+0000) 2024-03-06T11:25:38.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:38.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:38.028482+0000) 2024-03-06T11:25:39.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:39.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:39.028781+0000) 2024-03-06T11:25:40.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:40.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:40.029037+0000) 2024-03-06T11:25:41.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:41.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:41.029224+0000) 2024-03-06T11:25:42.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:42.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:42.029478+0000) 2024-03-06T11:25:43.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:43.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:43.029748+0000) 2024-03-06T11:25:44.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:44.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:44.030014+0000) 2024-03-06T11:25:45.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:45.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:45.030286+0000) 2024-03-06T11:25:46.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:46.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:46.030551+0000) 2024-03-06T11:25:47.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:47.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:47.030802+0000) 2024-03-06T11:25:48.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:48.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:48.031049+0000) 2024-03-06T11:25:49.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:49.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:49.031256+0000) 2024-03-06T11:25:50.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:50.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:50.031510+0000) 2024-03-06T11:25:51.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:51.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:51.031772+0000) 2024-03-06T11:25:52.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:52.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:52.032030+0000) 2024-03-06T11:25:53.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:53.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:53.032230+0000) 2024-03-06T11:25:54.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:54.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:54.032471+0000) 2024-03-06T11:25:55.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:55.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:55.032767+0000) 2024-03-06T11:25:56.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:56.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:56.033037+0000) 2024-03-06T11:25:57.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:57.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:57.033295+0000) 2024-03-06T11:25:58.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:58.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:58.033562+0000) 2024-03-06T11:25:59.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:25:59.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:59.033849+0000) 2024-03-06T11:26:00.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:00.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:00.034114+0000) 2024-03-06T11:26:00.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:00.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:01.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:01.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:01.034333+0000) 2024-03-06T11:26:02.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:02.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:02.034540+0000) 2024-03-06T11:26:03.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:03.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:03.034793+0000) 2024-03-06T11:26:04.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:04.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:04.035043+0000) 2024-03-06T11:26:05.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:05.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:05.035288+0000) 2024-03-06T11:26:06.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:06.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:06.035523+0000) 2024-03-06T11:26:07.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:07.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:07.035824+0000) 2024-03-06T11:26:08.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:08.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:08.036073+0000) 2024-03-06T11:26:09.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:09.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:09.036351+0000) 2024-03-06T11:26:10.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:10.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:10.036577+0000) 2024-03-06T11:26:11.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:11.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:11.036794+0000) 2024-03-06T11:26:12.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:12.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:12.037047+0000) 2024-03-06T11:26:13.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:13.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:13.037295+0000) 2024-03-06T11:26:14.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:14.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:14.037524+0000) 2024-03-06T11:26:15.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:15.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:15.037775+0000) 2024-03-06T11:26:16.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:16.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:16.038036+0000) 2024-03-06T11:26:17.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:17.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:17.038336+0000) 2024-03-06T11:26:18.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:18.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:18.038606+0000) 2024-03-06T11:26:19.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:19.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:19.038897+0000) 2024-03-06T11:26:20.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:20.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:20.039156+0000) 2024-03-06T11:26:21.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:21.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:21.039409+0000) 2024-03-06T11:26:22.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:22.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:22.039720+0000) 2024-03-06T11:26:23.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:23.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:23.039970+0000) 2024-03-06T11:26:24.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:24.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:24.040220+0000) 2024-03-06T11:26:25.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:25.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:25.040487+0000) 2024-03-06T11:26:26.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:26.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:26.040747+0000) 2024-03-06T11:26:27.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:27.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:27.041003+0000) 2024-03-06T11:26:28.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:28.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:28.041252+0000) 2024-03-06T11:26:29.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:29.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:29.041498+0000) 2024-03-06T11:26:30.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:30.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:30.041739+0000) 2024-03-06T11:26:30.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:30.689 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:31.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:31.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:31.041946+0000) 2024-03-06T11:26:32.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:32.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:32.042181+0000) 2024-03-06T11:26:33.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:33.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:33.042447+0000) 2024-03-06T11:26:34.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:34.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:34.042733+0000) 2024-03-06T11:26:35.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:35.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:35.042999+0000) 2024-03-06T11:26:36.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:36.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:36.043256+0000) 2024-03-06T11:26:37.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:37.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:37.043500+0000) 2024-03-06T11:26:38.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:38.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:38.043770+0000) 2024-03-06T11:26:39.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:39.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:39.044033+0000) 2024-03-06T11:26:40.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:40.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:40.044286+0000) 2024-03-06T11:26:41.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:41.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:41.044546+0000) 2024-03-06T11:26:42.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:42.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:42.044809+0000) 2024-03-06T11:26:43.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:43.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:43.045067+0000) 2024-03-06T11:26:44.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:44.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:44.045331+0000) 2024-03-06T11:26:45.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:45.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:45.045595+0000) 2024-03-06T11:26:46.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:46.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:46.045877+0000) 2024-03-06T11:26:47.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:47.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:47.046133+0000) 2024-03-06T11:26:48.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:48.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:48.046399+0000) 2024-03-06T11:26:49.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:49.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:49.046711+0000) 2024-03-06T11:26:50.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:50.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:50.046951+0000) 2024-03-06T11:26:51.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:51.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:51.047192+0000) 2024-03-06T11:26:52.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:52.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:52.047429+0000) 2024-03-06T11:26:53.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:53.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:53.047727+0000) 2024-03-06T11:26:54.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:54.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:54.047976+0000) 2024-03-06T11:26:55.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:55.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:55.048245+0000) 2024-03-06T11:26:56.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:56.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:56.048521+0000) 2024-03-06T11:26:57.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:57.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:57.048804+0000) 2024-03-06T11:26:58.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:58.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:58.049091+0000) 2024-03-06T11:26:59.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:26:59.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:59.049340+0000) 2024-03-06T11:27:00.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:00.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:00.049671+0000) 2024-03-06T11:27:00.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:00.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:01.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:01.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:01.049885+0000) 2024-03-06T11:27:02.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:02.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:02.050117+0000) 2024-03-06T11:27:03.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:03.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:03.050382+0000) 2024-03-06T11:27:04.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:04.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:04.050708+0000) 2024-03-06T11:27:05.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:05.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:05.050930+0000) 2024-03-06T11:27:06.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:06.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:06.051165+0000) 2024-03-06T11:27:07.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:07.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:07.051410+0000) 2024-03-06T11:27:08.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:08.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:08.051720+0000) 2024-03-06T11:27:09.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:09.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:09.051994+0000) 2024-03-06T11:27:10.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:10.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:10.052230+0000) 2024-03-06T11:27:11.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:11.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:11.052469+0000) 2024-03-06T11:27:12.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:12.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:12.052739+0000) 2024-03-06T11:27:13.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:13.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:13.052977+0000) 2024-03-06T11:27:14.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:14.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:14.053215+0000) 2024-03-06T11:27:15.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:15.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:15.053480+0000) 2024-03-06T11:27:16.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:16.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:16.053753+0000) 2024-03-06T11:27:17.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:17.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:17.054014+0000) 2024-03-06T11:27:18.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:18.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:18.054257+0000) 2024-03-06T11:27:19.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:19.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:19.054518+0000) 2024-03-06T11:27:20.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:20.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:20.054755+0000) 2024-03-06T11:27:21.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:21.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:21.054951+0000) 2024-03-06T11:27:22.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:22.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:22.055198+0000) 2024-03-06T11:27:23.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:23.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:23.055454+0000) 2024-03-06T11:27:24.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:24.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:24.055730+0000) 2024-03-06T11:27:25.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:25.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:25.055997+0000) 2024-03-06T11:27:26.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:26.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:26.056256+0000) 2024-03-06T11:27:27.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:27.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:27.056514+0000) 2024-03-06T11:27:28.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:28.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:28.056797+0000) 2024-03-06T11:27:29.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:29.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:29.057065+0000) 2024-03-06T11:27:30.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:30.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:30.057321+0000) 2024-03-06T11:27:30.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:30.966 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:31.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:31.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:31.057552+0000) 2024-03-06T11:27:32.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:32.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:32.057848+0000) 2024-03-06T11:27:33.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:33.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:33.058182+0000) 2024-03-06T11:27:34.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:34.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:34.058466+0000) 2024-03-06T11:27:35.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:35.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:35.058785+0000) 2024-03-06T11:27:36.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:36.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:36.059100+0000) 2024-03-06T11:27:37.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:37.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:37.059399+0000) 2024-03-06T11:27:38.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:38.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:38.059766+0000) 2024-03-06T11:27:39.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:39.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:39.060013+0000) 2024-03-06T11:27:40.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:40.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:40.060264+0000) 2024-03-06T11:27:41.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:41.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:41.060494+0000) 2024-03-06T11:27:42.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:42.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:42.060754+0000) 2024-03-06T11:27:43.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:43.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:43.061001+0000) 2024-03-06T11:27:44.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:44.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:44.061259+0000) 2024-03-06T11:27:45.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:45.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:45.061524+0000) 2024-03-06T11:27:46.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:46.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:46.061783+0000) 2024-03-06T11:27:47.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:47.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:47.062048+0000) 2024-03-06T11:27:48.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:48.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:48.062289+0000) 2024-03-06T11:27:49.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:49.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:49.062593+0000) 2024-03-06T11:27:50.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:50.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:50.062954+0000) 2024-03-06T11:27:51.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:51.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:51.063262+0000) 2024-03-06T11:27:52.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:52.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:52.063551+0000) 2024-03-06T11:27:53.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:53.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:53.063865+0000) 2024-03-06T11:27:54.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:54.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:54.064122+0000) 2024-03-06T11:27:55.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:55.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:55.064431+0000) 2024-03-06T11:27:56.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:56.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:56.064732+0000) 2024-03-06T11:27:57.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:57.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:57.065028+0000) 2024-03-06T11:27:58.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:58.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:58.065344+0000) 2024-03-06T11:27:59.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:27:59.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:59.065746+0000) 2024-03-06T11:28:00.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:00.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:00.066057+0000) 2024-03-06T11:28:01.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:01.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:01.066379+0000) 2024-03-06T11:28:01.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:01.122 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:02.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:02.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:02.066712+0000) 2024-03-06T11:28:03.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:03.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:03.066976+0000) 2024-03-06T11:28:04.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:04.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:04.067240+0000) 2024-03-06T11:28:05.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:05.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:05.067535+0000) 2024-03-06T11:28:06.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:06.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:06.067850+0000) 2024-03-06T11:28:07.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:07.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:07.068137+0000) 2024-03-06T11:28:08.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:08.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:08.068458+0000) 2024-03-06T11:28:09.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:09.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:09.068755+0000) 2024-03-06T11:28:10.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:10.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:10.069003+0000) 2024-03-06T11:28:11.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:11.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:11.069315+0000) 2024-03-06T11:28:12.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:12.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:12.069691+0000) 2024-03-06T11:28:13.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:13.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:13.069991+0000) 2024-03-06T11:28:14.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:14.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:14.070292+0000) 2024-03-06T11:28:15.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:15.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:15.070540+0000) 2024-03-06T11:28:16.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:16.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:16.070844+0000) 2024-03-06T11:28:17.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:17.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:17.071158+0000) 2024-03-06T11:28:18.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:18.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:18.071479+0000) 2024-03-06T11:28:19.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:19.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:19.071789+0000) 2024-03-06T11:28:20.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:20.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:20.072039+0000) 2024-03-06T11:28:21.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:21.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:21.072358+0000) 2024-03-06T11:28:22.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:22.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:22.072692+0000) 2024-03-06T11:28:23.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:23.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:23.072997+0000) 2024-03-06T11:28:24.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:24.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:24.073256+0000) 2024-03-06T11:28:25.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:25.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:25.073520+0000) 2024-03-06T11:28:26.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:26.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:26.073777+0000) 2024-03-06T11:28:27.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:27.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:27.074038+0000) 2024-03-06T11:28:28.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:28.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:28.074330+0000) 2024-03-06T11:28:29.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:29.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:29.074669+0000) 2024-03-06T11:28:30.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:30.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:30.074967+0000) 2024-03-06T11:28:31.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:31.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:31.075274+0000) 2024-03-06T11:28:31.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:31.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:32.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:32.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:32.075531+0000) 2024-03-06T11:28:33.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:33.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:33.075863+0000) 2024-03-06T11:28:34.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:34.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:34.076190+0000) 2024-03-06T11:28:35.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:35.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:35.076499+0000) 2024-03-06T11:28:36.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:36.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:36.076811+0000) 2024-03-06T11:28:37.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:37.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:37.077130+0000) 2024-03-06T11:28:38.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:38.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:38.077436+0000) 2024-03-06T11:28:39.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:39.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:39.077780+0000) 2024-03-06T11:28:40.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:40.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:40.078095+0000) 2024-03-06T11:28:41.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:41.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:41.078401+0000) 2024-03-06T11:28:42.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:42.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:42.078750+0000) 2024-03-06T11:28:43.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:43.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:43.079057+0000) 2024-03-06T11:28:44.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:44.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:44.079352+0000) 2024-03-06T11:28:45.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:45.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:45.079701+0000) 2024-03-06T11:28:46.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:46.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:46.079957+0000) 2024-03-06T11:28:47.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:47.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:47.080306+0000) 2024-03-06T11:28:48.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:48.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:48.080692+0000) 2024-03-06T11:28:49.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:49.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:49.080993+0000) 2024-03-06T11:28:50.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:50.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:50.081280+0000) 2024-03-06T11:28:51.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:51.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:51.081596+0000) 2024-03-06T11:28:52.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:52.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:52.081952+0000) 2024-03-06T11:28:53.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:53.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:53.082278+0000) 2024-03-06T11:28:54.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:54.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:54.082603+0000) 2024-03-06T11:28:55.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:55.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:55.082982+0000) 2024-03-06T11:28:56.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:56.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:56.083281+0000) 2024-03-06T11:28:57.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:57.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:57.083589+0000) 2024-03-06T11:28:58.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:58.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:58.083919+0000) 2024-03-06T11:28:59.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:28:59.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:59.084214+0000) 2024-03-06T11:29:00.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:00.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:00.084523+0000) 2024-03-06T11:29:01.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:01.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:01.084823+0000) 2024-03-06T11:29:01.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:01.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:02.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:02.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:02.085117+0000) 2024-03-06T11:29:03.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:03.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:03.085419+0000) 2024-03-06T11:29:04.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:04.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:04.085777+0000) 2024-03-06T11:29:05.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:05.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:05.086086+0000) 2024-03-06T11:29:06.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:06.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:06.086394+0000) 2024-03-06T11:29:07.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:07.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:07.086730+0000) 2024-03-06T11:29:08.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:08.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:08.087040+0000) 2024-03-06T11:29:09.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:09.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:09.087331+0000) 2024-03-06T11:29:10.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:10.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:10.087584+0000) 2024-03-06T11:29:11.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:11.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:11.087929+0000) 2024-03-06T11:29:12.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:12.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:12.088250+0000) 2024-03-06T11:29:13.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:13.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:13.088568+0000) 2024-03-06T11:29:14.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:14.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:14.088854+0000) 2024-03-06T11:29:15.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:15.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:15.089150+0000) 2024-03-06T11:29:16.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:16.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:16.089402+0000) 2024-03-06T11:29:17.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:17.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:17.089756+0000) 2024-03-06T11:29:18.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:18.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:18.090076+0000) 2024-03-06T11:29:19.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:19.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:19.090361+0000) 2024-03-06T11:29:20.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:20.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:20.090676+0000) 2024-03-06T11:29:21.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:21.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:21.090993+0000) 2024-03-06T11:29:22.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:22.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:22.091296+0000) 2024-03-06T11:29:23.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:23.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:23.091600+0000) 2024-03-06T11:29:24.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:24.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:24.091911+0000) 2024-03-06T11:29:25.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:25.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:25.092166+0000) 2024-03-06T11:29:26.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:26.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:26.092402+0000) 2024-03-06T11:29:27.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:27.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:27.092673+0000) 2024-03-06T11:29:28.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:28.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:28.092954+0000) 2024-03-06T11:29:29.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:29.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:29.093197+0000) 2024-03-06T11:29:30.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:30.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:30.093391+0000) 2024-03-06T11:29:31.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:31.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:31.093673+0000) 2024-03-06T11:29:31.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:31.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:32.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:32.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:32.093915+0000) 2024-03-06T11:29:33.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:33.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:33.094164+0000) 2024-03-06T11:29:34.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:34.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:34.094429+0000) 2024-03-06T11:29:35.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:35.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:35.094785+0000) 2024-03-06T11:29:36.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:36.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:36.095051+0000) 2024-03-06T11:29:37.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:37.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:37.095305+0000) 2024-03-06T11:29:38.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:38.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:38.095569+0000) 2024-03-06T11:29:39.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:39.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:39.095760+0000) 2024-03-06T11:29:40.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:40.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:40.096005+0000) 2024-03-06T11:29:41.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:41.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:41.096234+0000) 2024-03-06T11:29:42.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:42.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:42.096479+0000) 2024-03-06T11:29:43.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:43.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:43.096738+0000) 2024-03-06T11:29:44.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:44.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:44.096939+0000) 2024-03-06T11:29:45.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:45.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:45.097207+0000) 2024-03-06T11:29:46.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:46.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:46.097474+0000) 2024-03-06T11:29:47.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:47.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:47.097756+0000) 2024-03-06T11:29:48.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:48.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:48.098027+0000) 2024-03-06T11:29:49.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:49.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:49.098299+0000) 2024-03-06T11:29:50.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:50.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:50.098588+0000) 2024-03-06T11:29:51.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:51.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:51.098908+0000) 2024-03-06T11:29:52.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:52.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:52.099188+0000) 2024-03-06T11:29:53.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:53.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:53.099381+0000) 2024-03-06T11:29:54.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:54.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:54.099685+0000) 2024-03-06T11:29:55.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:55.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:55.099954+0000) 2024-03-06T11:29:56.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:56.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:56.100165+0000) 2024-03-06T11:29:57.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:57.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:57.100426+0000) 2024-03-06T11:29:58.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:58.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:58.100741+0000) 2024-03-06T11:29:59.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:29:59.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:59.101001+0000) 2024-03-06T11:30:00.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:00.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:00.101270+0000) 2024-03-06T11:30:01.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:01.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:01.101543+0000) 2024-03-06T11:30:01.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:01.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:02.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:02.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:02.101834+0000) 2024-03-06T11:30:03.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:03.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:03.102092+0000) 2024-03-06T11:30:04.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:04.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:04.102376+0000) 2024-03-06T11:30:05.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:05.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:05.102618+0000) 2024-03-06T11:30:06.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:06.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:06.102868+0000) 2024-03-06T11:30:07.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:07.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:07.103078+0000) 2024-03-06T11:30:08.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:08.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:08.103339+0000) 2024-03-06T11:30:09.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:09.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:09.103617+0000) 2024-03-06T11:30:10.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:10.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:10.103976+0000) 2024-03-06T11:30:11.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:11.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:11.104253+0000) 2024-03-06T11:30:12.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:12.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:12.104531+0000) 2024-03-06T11:30:13.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:13.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:13.104793+0000) 2024-03-06T11:30:14.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:14.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:14.105043+0000) 2024-03-06T11:30:15.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:15.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:15.105282+0000) 2024-03-06T11:30:16.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:16.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:16.105559+0000) 2024-03-06T11:30:17.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:17.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:17.105819+0000) 2024-03-06T11:30:18.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:18.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:18.106119+0000) 2024-03-06T11:30:19.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:19.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:19.106376+0000) 2024-03-06T11:30:20.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:20.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:20.106707+0000) 2024-03-06T11:30:21.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:21.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:21.106919+0000) 2024-03-06T11:30:22.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:22.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:22.107177+0000) 2024-03-06T11:30:23.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:23.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:23.107429+0000) 2024-03-06T11:30:24.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:24.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:24.107736+0000) 2024-03-06T11:30:25.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:25.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:25.107994+0000) 2024-03-06T11:30:26.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:26.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:26.108271+0000) 2024-03-06T11:30:27.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:27.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:27.108518+0000) 2024-03-06T11:30:28.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:28.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:28.108838+0000) 2024-03-06T11:30:29.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:29.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:29.109119+0000) 2024-03-06T11:30:30.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:30.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:30.109405+0000) 2024-03-06T11:30:31.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:31.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:31.109696+0000) 2024-03-06T11:30:31.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:31.833 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:32.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:32.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:32.109965+0000) 2024-03-06T11:30:33.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:33.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:33.110245+0000) 2024-03-06T11:30:34.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:34.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:34.110529+0000) 2024-03-06T11:30:35.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:35.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:35.110782+0000) 2024-03-06T11:30:36.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:36.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:36.111046+0000) 2024-03-06T11:30:37.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:37.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:37.111297+0000) 2024-03-06T11:30:38.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:38.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:38.111572+0000) 2024-03-06T11:30:39.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:39.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:39.111818+0000) 2024-03-06T11:30:40.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:40.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:40.112059+0000) 2024-03-06T11:30:41.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:41.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:41.112316+0000) 2024-03-06T11:30:42.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:42.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:42.112595+0000) 2024-03-06T11:30:43.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:43.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:43.112909+0000) 2024-03-06T11:30:44.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:44.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:44.113171+0000) 2024-03-06T11:30:45.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:45.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:45.113442+0000) 2024-03-06T11:30:46.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:46.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:46.113746+0000) 2024-03-06T11:30:47.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:47.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:47.113995+0000) 2024-03-06T11:30:48.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:48.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:48.114260+0000) 2024-03-06T11:30:49.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:49.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:49.114544+0000) 2024-03-06T11:30:50.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:50.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:50.114797+0000) 2024-03-06T11:30:51.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:51.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:51.115048+0000) 2024-03-06T11:30:52.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:52.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:52.115299+0000) 2024-03-06T11:30:53.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:53.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:53.115551+0000) 2024-03-06T11:30:54.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:54.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:54.115813+0000) 2024-03-06T11:30:55.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:55.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:55.116070+0000) 2024-03-06T11:30:56.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:56.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:56.116273+0000) 2024-03-06T11:30:57.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:57.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:57.116532+0000) 2024-03-06T11:30:58.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:58.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:58.116799+0000) 2024-03-06T11:30:59.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:30:59.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:59.117041+0000) 2024-03-06T11:31:00.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:00.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:00.117289+0000) 2024-03-06T11:31:01.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:01.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:01.117537+0000) 2024-03-06T11:31:01.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:01.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:02.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:02.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:02.117814+0000) 2024-03-06T11:31:03.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:03.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:03.118128+0000) 2024-03-06T11:31:04.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:04.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:04.118449+0000) 2024-03-06T11:31:05.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:05.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:05.118788+0000) 2024-03-06T11:31:06.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:06.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:06.119077+0000) 2024-03-06T11:31:07.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:07.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:07.119360+0000) 2024-03-06T11:31:08.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:08.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:08.119686+0000) 2024-03-06T11:31:09.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:09.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:09.119955+0000) 2024-03-06T11:31:10.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:10.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:10.120151+0000) 2024-03-06T11:31:11.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:11.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:11.120385+0000) 2024-03-06T11:31:12.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:12.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:12.120675+0000) 2024-03-06T11:31:13.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:13.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:13.120951+0000) 2024-03-06T11:31:14.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:14.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:14.121185+0000) 2024-03-06T11:31:15.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:15.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:15.121438+0000) 2024-03-06T11:31:16.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:16.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:16.121742+0000) 2024-03-06T11:31:17.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:17.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:17.121982+0000) 2024-03-06T11:31:18.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:18.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:18.122246+0000) 2024-03-06T11:31:19.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:19.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:19.122503+0000) 2024-03-06T11:31:20.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:20.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:20.122809+0000) 2024-03-06T11:31:21.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:21.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:21.123041+0000) 2024-03-06T11:31:22.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:22.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:22.123290+0000) 2024-03-06T11:31:23.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:23.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:23.123517+0000) 2024-03-06T11:31:24.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:24.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:24.123771+0000) 2024-03-06T11:31:25.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:25.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:25.124002+0000) 2024-03-06T11:31:26.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:26.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:26.124211+0000) 2024-03-06T11:31:27.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:27.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:27.124408+0000) 2024-03-06T11:31:28.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:28.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:28.124734+0000) 2024-03-06T11:31:29.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:29.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:29.124991+0000) 2024-03-06T11:31:30.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:30.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:30.125276+0000) 2024-03-06T11:31:31.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:31.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:31.125537+0000) 2024-03-06T11:31:32.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:32.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:32.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:32.125789+0000) 2024-03-06T11:31:32.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:33.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:33.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:33.126048+0000) 2024-03-06T11:31:34.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:34.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:34.126344+0000) 2024-03-06T11:31:35.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:35.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:35.126606+0000) 2024-03-06T11:31:36.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:36.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:36.126891+0000) 2024-03-06T11:31:37.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:37.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:37.127110+0000) 2024-03-06T11:31:38.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:38.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:38.127326+0000) 2024-03-06T11:31:39.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:39.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:39.127570+0000) 2024-03-06T11:31:40.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:40.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:40.127816+0000) 2024-03-06T11:31:41.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:41.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:41.128090+0000) 2024-03-06T11:31:42.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:42.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:42.128359+0000) 2024-03-06T11:31:43.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:43.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:43.128605+0000) 2024-03-06T11:31:44.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:44.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:44.128932+0000) 2024-03-06T11:31:45.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:45.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:45.129184+0000) 2024-03-06T11:31:46.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:46.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:46.129456+0000) 2024-03-06T11:31:47.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:47.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:47.129782+0000) 2024-03-06T11:31:48.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:48.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:48.130076+0000) 2024-03-06T11:31:49.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:49.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:49.130288+0000) 2024-03-06T11:31:50.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:50.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:50.130503+0000) 2024-03-06T11:31:51.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:51.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:51.130769+0000) 2024-03-06T11:31:52.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:52.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:52.131029+0000) 2024-03-06T11:31:53.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:53.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:53.131326+0000) 2024-03-06T11:31:54.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:54.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:54.131616+0000) 2024-03-06T11:31:55.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:55.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:55.131942+0000) 2024-03-06T11:31:56.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:56.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:56.132210+0000) 2024-03-06T11:31:57.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:57.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:57.132423+0000) 2024-03-06T11:31:58.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:58.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:58.132709+0000) 2024-03-06T11:31:59.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:31:59.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:59.132974+0000) 2024-03-06T11:32:00.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:00.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:00.133225+0000) 2024-03-06T11:32:01.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:01.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:01.133453+0000) 2024-03-06T11:32:02.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:02.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:02.133748+0000) 2024-03-06T11:32:02.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:02.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:03.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:03.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:03.133960+0000) 2024-03-06T11:32:04.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:04.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:04.134243+0000) 2024-03-06T11:32:05.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:05.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:05.134492+0000) 2024-03-06T11:32:06.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:06.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:06.134804+0000) 2024-03-06T11:32:07.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:07.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:07.135041+0000) 2024-03-06T11:32:08.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:08.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:08.135270+0000) 2024-03-06T11:32:09.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:09.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:09.135490+0000) 2024-03-06T11:32:10.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:10.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:10.135740+0000) 2024-03-06T11:32:11.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:11.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:11.135938+0000) 2024-03-06T11:32:12.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:12.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:12.136181+0000) 2024-03-06T11:32:13.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:13.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:13.136389+0000) 2024-03-06T11:32:14.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:14.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:14.136609+0000) 2024-03-06T11:32:15.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:15.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:15.136870+0000) 2024-03-06T11:32:16.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:16.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:16.137060+0000) 2024-03-06T11:32:17.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:17.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:17.137250+0000) 2024-03-06T11:32:18.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:18.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:18.137476+0000) 2024-03-06T11:32:19.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:19.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:19.137708+0000) 2024-03-06T11:32:20.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:20.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:20.137905+0000) 2024-03-06T11:32:21.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:21.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:21.138135+0000) 2024-03-06T11:32:22.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:22.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:22.138343+0000) 2024-03-06T11:32:23.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:23.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:23.138569+0000) 2024-03-06T11:32:24.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:24.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:24.138795+0000) 2024-03-06T11:32:25.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:25.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:25.139020+0000) 2024-03-06T11:32:26.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:26.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:26.139220+0000) 2024-03-06T11:32:27.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:27.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:27.139402+0000) 2024-03-06T11:32:28.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:28.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:28.139655+0000) 2024-03-06T11:32:29.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:29.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:29.139866+0000) 2024-03-06T11:32:30.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:30.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:30.140065+0000) 2024-03-06T11:32:31.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:31.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:31.140300+0000) 2024-03-06T11:32:32.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:32.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:32.140507+0000) 2024-03-06T11:32:32.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:32.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:33.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:33.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:33.140741+0000) 2024-03-06T11:32:34.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:34.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:34.140978+0000) 2024-03-06T11:32:35.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:35.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:35.141185+0000) 2024-03-06T11:32:36.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:36.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:36.141412+0000) 2024-03-06T11:32:37.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:37.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:37.141613+0000) 2024-03-06T11:32:38.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:38.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:38.141933+0000) 2024-03-06T11:32:39.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:39.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:39.142179+0000) 2024-03-06T11:32:40.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:40.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:40.142397+0000) 2024-03-06T11:32:41.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:41.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:41.142705+0000) 2024-03-06T11:32:42.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:42.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:42.142954+0000) 2024-03-06T11:32:43.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:43.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:43.143164+0000) 2024-03-06T11:32:44.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:44.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:44.143431+0000) 2024-03-06T11:32:45.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:45.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:45.143804+0000) 2024-03-06T11:32:46.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:46.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:46.144052+0000) 2024-03-06T11:32:47.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:47.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:47.144291+0000) 2024-03-06T11:32:48.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:48.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:48.144553+0000) 2024-03-06T11:32:49.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:49.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:49.144774+0000) 2024-03-06T11:32:50.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:50.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:50.144978+0000) 2024-03-06T11:32:51.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:51.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:51.145243+0000) 2024-03-06T11:32:52.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:52.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:52.145517+0000) 2024-03-06T11:32:53.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:53.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:53.145758+0000) 2024-03-06T11:32:54.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:54.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:54.145962+0000) 2024-03-06T11:32:55.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:55.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:55.146204+0000) 2024-03-06T11:32:56.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:56.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:56.146453+0000) 2024-03-06T11:32:57.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:57.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:57.146746+0000) 2024-03-06T11:32:58.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:58.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:58.146949+0000) 2024-03-06T11:32:59.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:32:59.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:59.147142+0000) 2024-03-06T11:33:00.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:00.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:00.147336+0000) 2024-03-06T11:33:01.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:01.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:01.147558+0000) 2024-03-06T11:33:02.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:02.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:02.147804+0000) 2024-03-06T11:33:02.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:02.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:03.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:03.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:03.148061+0000) 2024-03-06T11:33:04.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:04.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:04.148284+0000) 2024-03-06T11:33:05.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:05.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:05.148482+0000) 2024-03-06T11:33:06.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:06.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:06.148772+0000) 2024-03-06T11:33:07.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:07.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:07.148971+0000) 2024-03-06T11:33:08.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:08.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:08.149178+0000) 2024-03-06T11:33:09.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:09.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:09.149379+0000) 2024-03-06T11:33:10.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:10.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:10.149574+0000) 2024-03-06T11:33:11.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:11.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:11.149812+0000) 2024-03-06T11:33:12.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:12.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:12.150045+0000) 2024-03-06T11:33:13.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:13.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:13.150273+0000) 2024-03-06T11:33:14.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:14.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:14.150512+0000) 2024-03-06T11:33:15.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:15.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:15.150767+0000) 2024-03-06T11:33:16.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:16.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:16.151023+0000) 2024-03-06T11:33:17.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:17.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:17.151230+0000) 2024-03-06T11:33:18.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:18.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:18.151441+0000) 2024-03-06T11:33:19.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:19.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:19.151687+0000) 2024-03-06T11:33:20.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:20.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:20.151886+0000) 2024-03-06T11:33:21.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:21.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:21.152133+0000) 2024-03-06T11:33:22.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:22.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:22.152344+0000) 2024-03-06T11:33:23.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:23.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:23.152543+0000) 2024-03-06T11:33:24.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:24.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:24.152811+0000) 2024-03-06T11:33:25.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:25.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:25.153041+0000) 2024-03-06T11:33:26.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:26.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:26.153313+0000) 2024-03-06T11:33:27.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:27.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:27.153526+0000) 2024-03-06T11:33:28.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:28.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:28.153743+0000) 2024-03-06T11:33:29.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:29.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:29.153939+0000) 2024-03-06T11:33:30.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:30.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:30.154147+0000) 2024-03-06T11:33:31.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:31.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:31.154352+0000) 2024-03-06T11:33:32.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:32.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:32.154546+0000) 2024-03-06T11:33:32.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:32.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:33.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:33.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:33.154795+0000) 2024-03-06T11:33:34.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:34.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:34.155047+0000) 2024-03-06T11:33:35.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:35.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:35.155264+0000) 2024-03-06T11:33:36.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:36.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:36.155480+0000) 2024-03-06T11:33:37.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:37.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:37.155701+0000) 2024-03-06T11:33:38.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:38.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:38.155908+0000) 2024-03-06T11:33:39.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:39.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:39.156098+0000) 2024-03-06T11:33:40.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:40.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:40.156296+0000) 2024-03-06T11:33:41.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:41.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:41.156495+0000) 2024-03-06T11:33:42.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:42.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:42.156750+0000) 2024-03-06T11:33:43.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:43.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:43.156949+0000) 2024-03-06T11:33:44.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:44.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:44.157174+0000) 2024-03-06T11:33:45.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:45.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:45.157451+0000) 2024-03-06T11:33:46.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:46.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:46.157741+0000) 2024-03-06T11:33:47.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:47.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:47.157974+0000) 2024-03-06T11:33:48.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:48.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:48.158228+0000) 2024-03-06T11:33:49.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:49.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:49.158466+0000) 2024-03-06T11:33:50.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:50.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:50.158735+0000) 2024-03-06T11:33:51.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:51.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:51.158969+0000) 2024-03-06T11:33:52.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:52.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:52.159160+0000) 2024-03-06T11:33:53.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:53.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:53.159358+0000) 2024-03-06T11:33:54.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:54.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:54.159557+0000) 2024-03-06T11:33:55.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:55.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:55.159769+0000) 2024-03-06T11:33:56.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:56.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:56.159962+0000) 2024-03-06T11:33:57.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:57.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:57.160165+0000) 2024-03-06T11:33:58.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:58.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:58.160425+0000) 2024-03-06T11:33:59.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:33:59.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:59.160714+0000) 2024-03-06T11:34:00.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:00.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:00.160956+0000) 2024-03-06T11:34:01.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:01.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:01.161242+0000) 2024-03-06T11:34:02.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:02.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:02.161507+0000) 2024-03-06T11:34:02.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:02.897 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:03.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:03.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:03.161803+0000) 2024-03-06T11:34:04.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:04.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:04.162010+0000) 2024-03-06T11:34:05.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:05.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:05.162266+0000) 2024-03-06T11:34:06.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:06.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:06.162527+0000) 2024-03-06T11:34:07.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:07.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:07.162791+0000) 2024-03-06T11:34:08.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:08.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:08.163034+0000) 2024-03-06T11:34:09.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:09.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:09.163224+0000) 2024-03-06T11:34:10.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:10.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:10.163465+0000) 2024-03-06T11:34:11.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:11.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:11.163733+0000) 2024-03-06T11:34:12.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:12.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:12.163916+0000) 2024-03-06T11:34:13.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:13.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:13.164167+0000) 2024-03-06T11:34:14.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:14.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:14.164403+0000) 2024-03-06T11:34:15.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:15.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:15.164721+0000) 2024-03-06T11:34:16.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:16.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:16.164965+0000) 2024-03-06T11:34:17.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:17.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:17.165221+0000) 2024-03-06T11:34:18.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:18.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:18.165483+0000) 2024-03-06T11:34:19.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:19.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:19.165739+0000) 2024-03-06T11:34:20.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:20.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:20.165929+0000) 2024-03-06T11:34:21.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:21.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:21.166185+0000) 2024-03-06T11:34:22.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:22.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:22.166373+0000) 2024-03-06T11:34:23.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:23.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:23.166607+0000) 2024-03-06T11:34:24.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:24.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:24.166880+0000) 2024-03-06T11:34:25.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:25.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:25.167128+0000) 2024-03-06T11:34:26.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:26.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:26.167387+0000) 2024-03-06T11:34:27.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:27.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:27.167657+0000) 2024-03-06T11:34:28.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:28.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:28.167931+0000) 2024-03-06T11:34:29.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:29.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:29.168157+0000) 2024-03-06T11:34:30.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:30.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:30.168405+0000) 2024-03-06T11:34:31.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:31.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:31.168717+0000) 2024-03-06T11:34:32.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:32.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:32.168953+0000) 2024-03-06T11:34:33.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:33.034 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:33.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:33.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:33.169284+0000) 2024-03-06T11:34:34.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:34.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:34.169520+0000) 2024-03-06T11:34:35.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:35.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:35.169776+0000) 2024-03-06T11:34:36.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:36.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:36.169974+0000) 2024-03-06T11:34:37.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:37.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:37.170227+0000) 2024-03-06T11:34:38.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:38.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:38.170484+0000) 2024-03-06T11:34:39.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:39.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:39.170755+0000) 2024-03-06T11:34:40.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:40.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:40.171009+0000) 2024-03-06T11:34:41.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:41.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:41.171274+0000) 2024-03-06T11:34:42.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:42.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:42.171526+0000) 2024-03-06T11:34:43.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:43.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:43.171763+0000) 2024-03-06T11:34:44.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:44.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:44.172027+0000) 2024-03-06T11:34:45.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:45.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:45.172281+0000) 2024-03-06T11:34:46.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:46.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:46.172541+0000) 2024-03-06T11:34:47.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:47.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:47.172774+0000) 2024-03-06T11:34:48.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:48.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:48.173031+0000) 2024-03-06T11:34:49.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:49.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:49.173270+0000) 2024-03-06T11:34:50.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:50.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:50.173545+0000) 2024-03-06T11:34:51.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:51.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:51.173806+0000) 2024-03-06T11:34:52.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:52.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:52.174038+0000) 2024-03-06T11:34:53.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:53.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:53.174270+0000) 2024-03-06T11:34:54.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:54.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:54.174506+0000) 2024-03-06T11:34:55.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:55.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:55.174769+0000) 2024-03-06T11:34:56.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:56.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:56.175016+0000) 2024-03-06T11:34:57.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:57.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:57.175272+0000) 2024-03-06T11:34:58.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:58.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:58.175512+0000) 2024-03-06T11:34:59.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:34:59.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:59.175773+0000) 2024-03-06T11:35:00.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:00.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:00.176005+0000) 2024-03-06T11:35:01.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:01.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:01.176271+0000) 2024-03-06T11:35:02.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:02.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:02.176504+0000) 2024-03-06T11:35:03.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:03.173 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:03.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:03.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:03.176745+0000) 2024-03-06T11:35:04.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:04.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:04.176997+0000) 2024-03-06T11:35:05.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:05.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:05.177262+0000) 2024-03-06T11:35:06.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:06.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:06.177523+0000) 2024-03-06T11:35:07.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:07.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:07.177769+0000) 2024-03-06T11:35:08.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:08.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:08.178029+0000) 2024-03-06T11:35:09.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:09.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:09.178284+0000) 2024-03-06T11:35:10.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:10.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:10.178525+0000) 2024-03-06T11:35:11.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:11.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:11.178789+0000) 2024-03-06T11:35:12.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:12.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:12.179038+0000) 2024-03-06T11:35:13.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:13.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:13.179272+0000) 2024-03-06T11:35:14.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:14.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:14.179507+0000) 2024-03-06T11:35:15.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:15.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:15.179767+0000) 2024-03-06T11:35:16.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:16.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:16.179969+0000) 2024-03-06T11:35:17.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:17.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:17.180208+0000) 2024-03-06T11:35:18.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:18.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:18.180479+0000) 2024-03-06T11:35:19.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:19.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:19.180768+0000) 2024-03-06T11:35:20.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:20.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:20.180956+0000) 2024-03-06T11:35:21.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:21.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:21.181140+0000) 2024-03-06T11:35:22.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:22.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:22.181318+0000) 2024-03-06T11:35:23.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:23.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:23.181584+0000) 2024-03-06T11:35:24.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:24.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:24.181830+0000) 2024-03-06T11:35:25.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:25.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:25.182080+0000) 2024-03-06T11:35:26.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:26.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:26.182406+0000) 2024-03-06T11:35:27.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:27.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:27.182759+0000) 2024-03-06T11:35:28.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:28.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:28.183018+0000) 2024-03-06T11:35:29.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:29.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:29.183271+0000) 2024-03-06T11:35:30.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:30.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:30.183527+0000) 2024-03-06T11:35:31.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:31.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:31.183779+0000) 2024-03-06T11:35:32.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:32.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:32.184052+0000) 2024-03-06T11:35:33.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:33.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:33.184285+0000) 2024-03-06T11:35:33.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:33.312 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:34.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:34.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:34.184540+0000) 2024-03-06T11:35:35.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:35.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:35.184798+0000) 2024-03-06T11:35:36.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:36.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:36.185046+0000) 2024-03-06T11:35:37.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:37.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:37.185309+0000) 2024-03-06T11:35:38.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:38.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:38.185551+0000) 2024-03-06T11:35:39.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:39.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:39.185788+0000) 2024-03-06T11:35:40.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:40.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:40.186025+0000) 2024-03-06T11:35:41.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:41.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:41.186278+0000) 2024-03-06T11:35:42.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:42.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:42.186527+0000) 2024-03-06T11:35:43.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:43.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:43.186760+0000) 2024-03-06T11:35:44.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:44.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:44.187011+0000) 2024-03-06T11:35:45.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:45.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:45.187253+0000) 2024-03-06T11:35:46.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:46.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:46.187488+0000) 2024-03-06T11:35:47.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:47.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:47.187781+0000) 2024-03-06T11:35:48.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:48.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:48.188047+0000) 2024-03-06T11:35:49.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:49.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:49.188240+0000) 2024-03-06T11:35:50.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:50.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:50.188487+0000) 2024-03-06T11:35:51.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:51.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:51.188753+0000) 2024-03-06T11:35:52.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:52.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:52.188986+0000) 2024-03-06T11:35:53.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:53.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:53.189223+0000) 2024-03-06T11:35:54.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:54.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:54.189462+0000) 2024-03-06T11:35:55.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:55.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:55.189748+0000) 2024-03-06T11:35:56.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:56.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:56.189996+0000) 2024-03-06T11:35:57.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:57.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:57.190237+0000) 2024-03-06T11:35:58.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:58.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:58.190478+0000) 2024-03-06T11:35:59.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:35:59.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:59.190739+0000) 2024-03-06T11:36:00.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:00.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:00.190980+0000) 2024-03-06T11:36:01.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:01.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:01.191237+0000) 2024-03-06T11:36:02.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:02.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:02.191434+0000) 2024-03-06T11:36:03.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:03.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:03.191732+0000) 2024-03-06T11:36:03.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:03.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:04.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:04.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:04.191975+0000) 2024-03-06T11:36:05.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:05.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:05.192232+0000) 2024-03-06T11:36:06.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:06.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:06.192490+0000) 2024-03-06T11:36:07.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:07.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:07.192751+0000) 2024-03-06T11:36:08.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:08.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:08.193015+0000) 2024-03-06T11:36:09.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:09.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:09.193277+0000) 2024-03-06T11:36:10.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:10.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:10.193536+0000) 2024-03-06T11:36:11.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:11.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:11.193791+0000) 2024-03-06T11:36:12.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:12.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:12.194043+0000) 2024-03-06T11:36:13.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:13.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:13.194290+0000) 2024-03-06T11:36:14.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:14.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:14.194543+0000) 2024-03-06T11:36:15.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:15.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:15.194849+0000) 2024-03-06T11:36:16.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:16.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:16.195086+0000) 2024-03-06T11:36:17.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:17.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:17.195330+0000) 2024-03-06T11:36:18.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:18.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:18.195585+0000) 2024-03-06T11:36:19.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:19.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:19.195837+0000) 2024-03-06T11:36:20.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:20.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:20.196099+0000) 2024-03-06T11:36:21.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:21.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:21.196349+0000) 2024-03-06T11:36:22.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:22.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:22.196589+0000) 2024-03-06T11:36:23.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:23.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:23.196829+0000) 2024-03-06T11:36:24.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:24.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:24.197072+0000) 2024-03-06T11:36:25.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:25.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:25.197332+0000) 2024-03-06T11:36:26.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:26.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:26.197514+0000) 2024-03-06T11:36:27.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:27.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:27.197755+0000) 2024-03-06T11:36:28.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:28.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:28.198023+0000) 2024-03-06T11:36:29.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:29.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:29.198229+0000) 2024-03-06T11:36:30.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:30.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:30.198482+0000) 2024-03-06T11:36:31.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:31.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:31.198764+0000) 2024-03-06T11:36:32.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:32.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:32.199024+0000) 2024-03-06T11:36:33.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:33.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:33.199277+0000) 2024-03-06T11:36:33.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:33.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:34.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:34.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:34.199535+0000) 2024-03-06T11:36:35.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:35.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:35.199794+0000) 2024-03-06T11:36:36.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:36.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:36.200051+0000) 2024-03-06T11:36:37.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:37.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:37.200309+0000) 2024-03-06T11:36:38.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:38.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:38.200573+0000) 2024-03-06T11:36:39.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:39.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:39.200820+0000) 2024-03-06T11:36:40.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:40.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:40.201007+0000) 2024-03-06T11:36:41.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:41.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:41.201267+0000) 2024-03-06T11:36:42.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:42.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:42.201522+0000) 2024-03-06T11:36:43.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:43.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:43.201761+0000) 2024-03-06T11:36:44.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:44.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:44.202010+0000) 2024-03-06T11:36:45.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:45.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:45.202241+0000) 2024-03-06T11:36:46.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:46.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:46.202505+0000) 2024-03-06T11:36:47.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:47.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:47.202759+0000) 2024-03-06T11:36:48.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:48.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:48.203020+0000) 2024-03-06T11:36:49.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:49.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:49.203283+0000) 2024-03-06T11:36:50.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:50.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:50.203529+0000) 2024-03-06T11:36:51.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:51.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:51.203773+0000) 2024-03-06T11:36:52.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:52.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:52.204011+0000) 2024-03-06T11:36:53.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:53.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:53.204257+0000) 2024-03-06T11:36:54.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:54.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:54.204442+0000) 2024-03-06T11:36:55.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:55.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:55.204746+0000) 2024-03-06T11:36:56.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:56.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:56.204983+0000) 2024-03-06T11:36:57.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:57.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:57.205248+0000) 2024-03-06T11:36:58.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:58.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:58.205495+0000) 2024-03-06T11:36:59.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:36:59.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:59.205735+0000) 2024-03-06T11:37:00.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:00.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:00.205994+0000) 2024-03-06T11:37:01.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:01.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:01.206254+0000) 2024-03-06T11:37:02.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:02.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:02.206510+0000) 2024-03-06T11:37:03.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:03.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:03.206828+0000) 2024-03-06T11:37:03.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:03.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:04.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:04.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:04.207089+0000) 2024-03-06T11:37:05.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:05.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:05.207331+0000) 2024-03-06T11:37:06.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:06.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:06.207561+0000) 2024-03-06T11:37:07.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:07.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:07.207827+0000) 2024-03-06T11:37:08.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:08.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:08.208093+0000) 2024-03-06T11:37:09.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:09.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:09.208349+0000) 2024-03-06T11:37:10.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:10.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:10.208600+0000) 2024-03-06T11:37:11.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:11.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:11.208880+0000) 2024-03-06T11:37:12.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:12.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:12.209152+0000) 2024-03-06T11:37:13.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:13.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:13.209413+0000) 2024-03-06T11:37:14.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:14.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:14.209743+0000) 2024-03-06T11:37:15.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:15.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:15.210010+0000) 2024-03-06T11:37:16.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:16.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:16.210271+0000) 2024-03-06T11:37:17.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:17.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:17.210537+0000) 2024-03-06T11:37:18.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:18.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:18.210831+0000) 2024-03-06T11:37:19.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:19.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:19.211087+0000) 2024-03-06T11:37:20.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:20.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:20.211326+0000) 2024-03-06T11:37:21.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:21.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:21.211575+0000) 2024-03-06T11:37:22.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:22.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:22.211826+0000) 2024-03-06T11:37:23.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:23.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:23.212085+0000) 2024-03-06T11:37:24.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:24.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:24.212347+0000) 2024-03-06T11:37:25.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:25.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:25.212615+0000) 2024-03-06T11:37:26.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:26.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:26.212879+0000) 2024-03-06T11:37:27.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:27.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:27.213143+0000) 2024-03-06T11:37:28.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:28.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:28.213347+0000) 2024-03-06T11:37:29.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:29.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:29.213609+0000) 2024-03-06T11:37:30.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:30.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:30.213937+0000) 2024-03-06T11:37:31.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:31.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:31.214169+0000) 2024-03-06T11:37:32.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:32.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:32.214435+0000) 2024-03-06T11:37:33.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:33.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:33.214742+0000) 2024-03-06T11:37:33.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:33.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:34.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:34.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:34.215006+0000) 2024-03-06T11:37:35.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:35.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:35.215205+0000) 2024-03-06T11:37:36.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:36.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:36.215432+0000) 2024-03-06T11:37:37.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:37.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:37.215655+0000) 2024-03-06T11:37:38.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:38.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:38.215905+0000) 2024-03-06T11:37:39.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:39.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:39.216080+0000) 2024-03-06T11:37:40.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:40.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:40.216302+0000) 2024-03-06T11:37:41.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:41.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:41.216573+0000) 2024-03-06T11:37:42.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:42.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:42.216770+0000) 2024-03-06T11:37:43.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:43.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:43.217004+0000) 2024-03-06T11:37:44.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:44.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:44.217248+0000) 2024-03-06T11:37:45.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:45.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:45.217527+0000) 2024-03-06T11:37:46.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:46.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:46.217761+0000) 2024-03-06T11:37:47.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:47.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:47.218015+0000) 2024-03-06T11:37:48.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:48.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:48.218202+0000) 2024-03-06T11:37:49.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:49.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:49.218467+0000) 2024-03-06T11:37:50.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:50.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:50.218756+0000) 2024-03-06T11:37:51.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:51.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:51.218965+0000) 2024-03-06T11:37:52.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:52.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:52.219213+0000) 2024-03-06T11:37:53.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:53.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:53.219455+0000) 2024-03-06T11:37:54.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:54.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:54.219718+0000) 2024-03-06T11:37:55.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:55.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:55.219909+0000) 2024-03-06T11:37:56.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:56.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:56.220217+0000) 2024-03-06T11:37:57.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:57.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:57.220462+0000) 2024-03-06T11:37:58.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:58.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:58.220741+0000) 2024-03-06T11:37:59.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:37:59.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:59.220999+0000) 2024-03-06T11:38:00.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:00.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:00.221265+0000) 2024-03-06T11:38:01.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:01.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:01.221505+0000) 2024-03-06T11:38:02.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:02.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:02.221791+0000) 2024-03-06T11:38:03.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:03.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:03.222043+0000) 2024-03-06T11:38:04.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:04.085 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:04.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:04.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:04.222246+0000) 2024-03-06T11:38:05.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:05.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:05.222512+0000) 2024-03-06T11:38:06.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:06.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:06.222765+0000) 2024-03-06T11:38:07.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:07.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:07.223047+0000) 2024-03-06T11:38:08.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:08.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:08.223311+0000) 2024-03-06T11:38:09.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:09.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:09.223585+0000) 2024-03-06T11:38:10.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:10.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:10.223881+0000) 2024-03-06T11:38:11.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:11.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:11.224151+0000) 2024-03-06T11:38:12.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:12.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:12.224408+0000) 2024-03-06T11:38:13.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:13.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:13.224705+0000) 2024-03-06T11:38:14.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:14.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:14.224953+0000) 2024-03-06T11:38:15.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:15.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:15.225207+0000) 2024-03-06T11:38:16.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:16.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:16.225472+0000) 2024-03-06T11:38:17.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:17.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:17.225769+0000) 2024-03-06T11:38:18.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:18.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:18.226030+0000) 2024-03-06T11:38:19.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:19.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:19.226286+0000) 2024-03-06T11:38:20.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:20.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:20.226551+0000) 2024-03-06T11:38:21.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:21.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:21.226816+0000) 2024-03-06T11:38:22.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:22.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:22.227027+0000) 2024-03-06T11:38:23.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:23.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:23.227247+0000) 2024-03-06T11:38:24.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:24.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:24.227504+0000) 2024-03-06T11:38:25.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:25.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:25.227758+0000) 2024-03-06T11:38:26.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:26.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:26.228008+0000) 2024-03-06T11:38:27.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:27.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:27.228258+0000) 2024-03-06T11:38:28.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:28.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:28.228524+0000) 2024-03-06T11:38:29.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:29.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:29.228777+0000) 2024-03-06T11:38:30.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:30.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:30.229019+0000) 2024-03-06T11:38:31.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:31.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:31.229223+0000) 2024-03-06T11:38:32.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:32.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:32.229479+0000) 2024-03-06T11:38:33.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:33.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:33.229733+0000) 2024-03-06T11:38:34.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:34.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:34.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:34.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:34.229966+0000) 2024-03-06T11:38:35.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:35.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:35.230227+0000) 2024-03-06T11:38:36.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:36.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:36.230483+0000) 2024-03-06T11:38:37.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:37.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:37.230735+0000) 2024-03-06T11:38:38.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:38.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:38.230994+0000) 2024-03-06T11:38:39.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:39.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:39.231233+0000) 2024-03-06T11:38:40.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:40.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:40.231488+0000) 2024-03-06T11:38:41.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:41.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:41.231749+0000) 2024-03-06T11:38:42.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:42.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:42.231987+0000) 2024-03-06T11:38:43.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:43.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:43.232223+0000) 2024-03-06T11:38:44.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:44.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:44.232531+0000) 2024-03-06T11:38:45.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:45.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:45.232797+0000) 2024-03-06T11:38:46.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:46.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:46.233060+0000) 2024-03-06T11:38:47.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:47.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:47.233326+0000) 2024-03-06T11:38:48.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:48.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:48.233574+0000) 2024-03-06T11:38:49.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:49.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:49.233815+0000) 2024-03-06T11:38:50.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:50.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:50.234066+0000) 2024-03-06T11:38:51.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:51.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:51.234255+0000) 2024-03-06T11:38:52.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:52.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:52.234510+0000) 2024-03-06T11:38:53.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:53.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:53.234772+0000) 2024-03-06T11:38:54.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:54.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:54.235029+0000) 2024-03-06T11:38:55.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:55.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:55.235323+0000) 2024-03-06T11:38:56.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:56.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:56.235625+0000) 2024-03-06T11:38:57.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:57.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:57.235964+0000) 2024-03-06T11:38:58.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:58.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:58.236228+0000) 2024-03-06T11:38:59.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:38:59.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:59.236458+0000) 2024-03-06T11:39:00.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:00.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:00.236808+0000) 2024-03-06T11:39:01.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:01.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:01.237050+0000) 2024-03-06T11:39:02.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:02.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:02.237303+0000) 2024-03-06T11:39:03.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:03.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:03.237563+0000) 2024-03-06T11:39:04.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:04.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:04.237766+0000) 2024-03-06T11:39:04.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:04.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:05.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:05.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:05.238030+0000) 2024-03-06T11:39:06.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:06.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:06.238240+0000) 2024-03-06T11:39:07.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:07.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:07.238505+0000) 2024-03-06T11:39:08.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:08.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:08.238757+0000) 2024-03-06T11:39:09.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:09.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:09.239020+0000) 2024-03-06T11:39:10.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:10.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:10.239281+0000) 2024-03-06T11:39:11.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:11.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:11.239557+0000) 2024-03-06T11:39:12.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:12.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:12.239824+0000) 2024-03-06T11:39:13.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:13.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:13.240085+0000) 2024-03-06T11:39:14.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:14.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:14.240354+0000) 2024-03-06T11:39:15.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:15.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:15.240617+0000) 2024-03-06T11:39:16.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:16.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:16.240980+0000) 2024-03-06T11:39:17.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:17.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:17.241291+0000) 2024-03-06T11:39:18.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:18.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:18.241581+0000) 2024-03-06T11:39:19.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:19.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:19.241883+0000) 2024-03-06T11:39:20.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:20.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:20.242162+0000) 2024-03-06T11:39:21.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:21.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:21.242474+0000) 2024-03-06T11:39:22.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:22.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:22.242800+0000) 2024-03-06T11:39:23.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:23.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:23.243092+0000) 2024-03-06T11:39:24.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:24.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:24.243365+0000) 2024-03-06T11:39:25.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:25.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:25.243734+0000) 2024-03-06T11:39:26.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:26.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:26.244036+0000) 2024-03-06T11:39:27.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:27.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:27.244337+0000) 2024-03-06T11:39:28.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:28.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:28.244710+0000) 2024-03-06T11:39:29.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:29.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:29.245018+0000) 2024-03-06T11:39:30.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:30.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:30.245323+0000) 2024-03-06T11:39:31.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:31.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:31.245679+0000) 2024-03-06T11:39:32.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:32.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:32.245998+0000) 2024-03-06T11:39:33.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:33.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:33.246289+0000) 2024-03-06T11:39:34.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:34.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:34.246527+0000) 2024-03-06T11:39:34.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:34.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:35.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:35.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:35.246835+0000) 2024-03-06T11:39:36.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:36.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:36.247100+0000) 2024-03-06T11:39:37.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:37.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:37.247370+0000) 2024-03-06T11:39:38.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:38.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:38.247680+0000) 2024-03-06T11:39:39.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:39.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:39.247942+0000) 2024-03-06T11:39:40.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:40.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:40.248168+0000) 2024-03-06T11:39:41.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:41.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:41.248473+0000) 2024-03-06T11:39:42.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:42.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:42.248810+0000) 2024-03-06T11:39:43.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:43.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:43.249119+0000) 2024-03-06T11:39:44.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:44.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:44.249416+0000) 2024-03-06T11:39:45.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:45.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:45.249778+0000) 2024-03-06T11:39:46.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:46.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:46.250083+0000) 2024-03-06T11:39:47.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:47.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:47.250368+0000) 2024-03-06T11:39:48.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:48.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:48.250714+0000) 2024-03-06T11:39:49.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:49.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:49.251034+0000) 2024-03-06T11:39:50.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:50.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:50.251332+0000) 2024-03-06T11:39:51.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:51.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:51.251709+0000) 2024-03-06T11:39:52.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:52.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:52.251997+0000) 2024-03-06T11:39:53.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:53.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:53.252243+0000) 2024-03-06T11:39:54.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:54.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:54.252471+0000) 2024-03-06T11:39:55.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:55.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:55.252754+0000) 2024-03-06T11:39:56.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:56.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:56.252991+0000) 2024-03-06T11:39:57.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:57.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:57.253225+0000) 2024-03-06T11:39:58.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:58.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:58.253469+0000) 2024-03-06T11:39:59.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:39:59.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:59.253764+0000) 2024-03-06T11:40:00.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:00.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:00.254013+0000) 2024-03-06T11:40:01.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:01.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:01.254199+0000) 2024-03-06T11:40:02.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:02.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:02.254434+0000) 2024-03-06T11:40:03.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:03.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:03.254729+0000) 2024-03-06T11:40:04.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:04.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:04.254964+0000) 2024-03-06T11:40:04.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:04.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:05.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:05.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:05.255218+0000) 2024-03-06T11:40:06.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:06.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:06.255416+0000) 2024-03-06T11:40:07.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:07.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:07.255693+0000) 2024-03-06T11:40:08.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:08.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:08.255934+0000) 2024-03-06T11:40:09.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:09.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:09.256130+0000) 2024-03-06T11:40:10.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:10.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:10.256370+0000) 2024-03-06T11:40:11.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:11.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:11.256555+0000) 2024-03-06T11:40:12.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:12.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:12.256800+0000) 2024-03-06T11:40:13.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:13.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:13.257061+0000) 2024-03-06T11:40:14.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:14.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:14.257329+0000) 2024-03-06T11:40:15.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:15.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:15.257571+0000) 2024-03-06T11:40:16.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:16.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:16.257808+0000) 2024-03-06T11:40:17.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:17.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:17.258069+0000) 2024-03-06T11:40:18.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:18.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:18.258264+0000) 2024-03-06T11:40:19.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:19.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:19.258518+0000) 2024-03-06T11:40:20.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:20.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:20.258766+0000) 2024-03-06T11:40:21.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:21.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:21.258995+0000) 2024-03-06T11:40:22.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:22.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:22.259230+0000) 2024-03-06T11:40:23.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:23.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:23.259490+0000) 2024-03-06T11:40:24.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:24.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:24.259745+0000) 2024-03-06T11:40:25.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:25.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:25.260053+0000) 2024-03-06T11:40:26.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:26.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:26.260309+0000) 2024-03-06T11:40:27.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:27.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:27.260563+0000) 2024-03-06T11:40:28.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:28.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:28.260822+0000) 2024-03-06T11:40:29.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:29.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:29.261078+0000) 2024-03-06T11:40:30.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:30.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:30.261326+0000) 2024-03-06T11:40:31.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:31.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:31.261593+0000) 2024-03-06T11:40:32.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:32.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:32.261812+0000) 2024-03-06T11:40:33.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:33.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:33.262066+0000) 2024-03-06T11:40:34.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:34.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:34.262297+0000) 2024-03-06T11:40:34.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:34.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:35.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:35.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:35.262540+0000) 2024-03-06T11:40:36.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:36.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:36.262798+0000) 2024-03-06T11:40:37.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:37.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:37.263062+0000) 2024-03-06T11:40:38.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:38.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:38.263359+0000) 2024-03-06T11:40:39.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:39.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:39.263600+0000) 2024-03-06T11:40:40.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:40.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:40.263840+0000) 2024-03-06T11:40:41.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:41.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:41.264097+0000) 2024-03-06T11:40:42.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:42.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:42.264328+0000) 2024-03-06T11:40:43.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:43.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:43.264560+0000) 2024-03-06T11:40:44.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:44.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:44.264806+0000) 2024-03-06T11:40:45.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:45.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:45.265052+0000) 2024-03-06T11:40:46.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:46.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:46.265282+0000) 2024-03-06T11:40:47.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:47.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:47.265521+0000) 2024-03-06T11:40:48.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:48.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:48.265776+0000) 2024-03-06T11:40:49.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:49.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:49.266011+0000) 2024-03-06T11:40:50.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:50.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:50.266249+0000) 2024-03-06T11:40:51.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:51.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:51.266514+0000) 2024-03-06T11:40:52.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:52.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:52.266753+0000) 2024-03-06T11:40:53.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:53.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:53.267009+0000) 2024-03-06T11:40:54.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:54.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:54.267235+0000) 2024-03-06T11:40:55.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:55.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:55.267476+0000) 2024-03-06T11:40:56.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:56.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:56.267735+0000) 2024-03-06T11:40:57.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:57.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:57.267995+0000) 2024-03-06T11:40:58.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:58.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:58.268262+0000) 2024-03-06T11:40:59.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:40:59.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:59.268521+0000) 2024-03-06T11:41:00.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:00.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:00.268771+0000) 2024-03-06T11:41:01.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:01.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:01.269063+0000) 2024-03-06T11:41:02.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:02.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:02.269335+0000) 2024-03-06T11:41:03.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:03.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:03.269616+0000) 2024-03-06T11:41:04.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:04.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:04.269924+0000) 2024-03-06T11:41:04.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:04.980 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:05.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:05.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:05.270203+0000) 2024-03-06T11:41:06.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:06.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:06.270454+0000) 2024-03-06T11:41:07.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:07.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:07.270743+0000) 2024-03-06T11:41:08.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:08.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:08.270999+0000) 2024-03-06T11:41:09.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:09.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:09.271266+0000) 2024-03-06T11:41:10.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:10.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:10.271529+0000) 2024-03-06T11:41:11.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:11.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:11.271788+0000) 2024-03-06T11:41:12.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:12.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:12.272049+0000) 2024-03-06T11:41:13.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:13.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:13.272338+0000) 2024-03-06T11:41:14.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:14.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:14.272575+0000) 2024-03-06T11:41:15.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:15.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:15.272815+0000) 2024-03-06T11:41:16.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:16.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:16.273079+0000) 2024-03-06T11:41:17.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:17.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:17.273322+0000) 2024-03-06T11:41:18.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:18.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:18.273586+0000) 2024-03-06T11:41:19.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:19.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:19.273808+0000) 2024-03-06T11:41:20.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:20.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:20.274068+0000) 2024-03-06T11:41:21.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:21.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:21.274327+0000) 2024-03-06T11:41:22.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:22.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:22.274578+0000) 2024-03-06T11:41:23.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:23.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:23.274814+0000) 2024-03-06T11:41:24.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:24.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:24.275064+0000) 2024-03-06T11:41:25.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:25.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:25.275314+0000) 2024-03-06T11:41:26.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:26.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:26.275671+0000) 2024-03-06T11:41:27.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:27.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:27.275991+0000) 2024-03-06T11:41:28.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:28.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:28.276304+0000) 2024-03-06T11:41:29.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:29.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:29.276589+0000) 2024-03-06T11:41:30.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:30.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:30.276896+0000) 2024-03-06T11:41:31.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:31.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:31.277171+0000) 2024-03-06T11:41:32.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:32.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:32.277423+0000) 2024-03-06T11:41:33.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:33.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:33.277714+0000) 2024-03-06T11:41:34.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:34.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:34.277948+0000) 2024-03-06T11:41:35.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:35.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:35.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:35.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:35.278209+0000) 2024-03-06T11:41:36.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:36.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:36.278452+0000) 2024-03-06T11:41:37.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:37.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:37.278737+0000) 2024-03-06T11:41:38.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:38.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:38.278996+0000) 2024-03-06T11:41:39.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:39.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:39.279243+0000) 2024-03-06T11:41:40.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:40.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:40.279482+0000) 2024-03-06T11:41:41.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:41.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:41.279783+0000) 2024-03-06T11:41:42.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:42.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:42.280097+0000) 2024-03-06T11:41:43.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:43.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:43.280412+0000) 2024-03-06T11:41:44.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:44.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:44.280743+0000) 2024-03-06T11:41:45.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:45.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:45.281060+0000) 2024-03-06T11:41:46.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:46.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:46.281346+0000) 2024-03-06T11:41:47.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:47.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:47.281711+0000) 2024-03-06T11:41:48.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:48.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:48.282031+0000) 2024-03-06T11:41:49.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:49.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:49.282334+0000) 2024-03-06T11:41:50.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:50.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:50.282685+0000) 2024-03-06T11:41:51.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:51.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:51.283010+0000) 2024-03-06T11:41:52.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:52.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:52.283333+0000) 2024-03-06T11:41:53.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:53.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:53.283690+0000) 2024-03-06T11:41:54.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:54.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:54.283982+0000) 2024-03-06T11:41:55.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:55.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:55.284303+0000) 2024-03-06T11:41:56.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:56.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:56.284620+0000) 2024-03-06T11:41:57.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:57.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:57.284985+0000) 2024-03-06T11:41:58.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:58.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:58.285318+0000) 2024-03-06T11:41:59.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:41:59.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:59.285625+0000) 2024-03-06T11:42:00.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:00.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:00.285976+0000) 2024-03-06T11:42:01.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:01.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:01.286320+0000) 2024-03-06T11:42:02.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:02.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:02.286623+0000) 2024-03-06T11:42:03.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:03.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:03.287004+0000) 2024-03-06T11:42:04.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:04.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:04.287320+0000) 2024-03-06T11:42:05.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:05.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:05.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:05.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:05.287513+0000) 2024-03-06T11:42:06.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:06.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:06.287780+0000) 2024-03-06T11:42:07.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:07.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:07.288078+0000) 2024-03-06T11:42:08.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:08.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:08.288336+0000) 2024-03-06T11:42:09.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:09.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:09.288598+0000) 2024-03-06T11:42:10.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:10.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:10.288856+0000) 2024-03-06T11:42:11.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:11.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:11.289120+0000) 2024-03-06T11:42:12.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:12.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:12.289371+0000) 2024-03-06T11:42:13.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:13.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:13.289687+0000) 2024-03-06T11:42:14.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:14.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:14.289983+0000) 2024-03-06T11:42:15.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:15.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:15.290286+0000) 2024-03-06T11:42:16.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:16.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:16.290593+0000) 2024-03-06T11:42:17.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:17.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:17.290885+0000) 2024-03-06T11:42:18.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:18.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:18.291206+0000) 2024-03-06T11:42:19.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:19.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:19.291518+0000) 2024-03-06T11:42:20.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:20.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:20.291801+0000) 2024-03-06T11:42:21.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:21.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:21.292040+0000) 2024-03-06T11:42:22.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:22.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:22.292354+0000) 2024-03-06T11:42:23.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:23.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:23.292683+0000) 2024-03-06T11:42:24.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:24.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:24.292992+0000) 2024-03-06T11:42:25.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:25.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:25.293287+0000) 2024-03-06T11:42:26.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:26.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:26.293557+0000) 2024-03-06T11:42:27.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:27.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:27.293847+0000) 2024-03-06T11:42:28.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:28.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:28.294162+0000) 2024-03-06T11:42:29.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:29.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:29.294476+0000) 2024-03-06T11:42:30.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:30.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:30.294780+0000) 2024-03-06T11:42:31.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:31.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:31.295066+0000) 2024-03-06T11:42:32.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:32.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:32.295356+0000) 2024-03-06T11:42:33.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:33.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:33.295714+0000) 2024-03-06T11:42:34.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:34.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:34.296024+0000) 2024-03-06T11:42:35.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:35.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:35.296242+0000) 2024-03-06T11:42:35.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:35.414 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:36.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:36.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:36.296483+0000) 2024-03-06T11:42:37.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:37.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:37.296762+0000) 2024-03-06T11:42:38.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:38.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:38.297032+0000) 2024-03-06T11:42:39.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:39.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:39.297280+0000) 2024-03-06T11:42:40.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:40.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:40.297524+0000) 2024-03-06T11:42:41.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:41.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:41.297790+0000) 2024-03-06T11:42:42.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:42.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:42.298043+0000) 2024-03-06T11:42:43.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:43.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:43.298279+0000) 2024-03-06T11:42:44.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:44.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:44.298518+0000) 2024-03-06T11:42:45.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:45.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:45.298752+0000) 2024-03-06T11:42:46.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:46.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:46.299009+0000) 2024-03-06T11:42:47.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:47.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:47.299253+0000) 2024-03-06T11:42:48.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:48.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:48.299515+0000) 2024-03-06T11:42:49.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:49.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:49.299773+0000) 2024-03-06T11:42:50.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:50.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:50.300024+0000) 2024-03-06T11:42:51.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:51.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:51.300287+0000) 2024-03-06T11:42:52.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:52.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:52.300529+0000) 2024-03-06T11:42:53.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:53.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:53.300764+0000) 2024-03-06T11:42:54.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:54.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:54.300998+0000) 2024-03-06T11:42:55.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:55.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:55.301244+0000) 2024-03-06T11:42:56.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:56.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:56.301506+0000) 2024-03-06T11:42:57.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:57.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:57.301768+0000) 2024-03-06T11:42:58.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:58.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:58.302031+0000) 2024-03-06T11:42:59.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:42:59.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:59.302270+0000) 2024-03-06T11:43:00.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:00.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:00.302522+0000) 2024-03-06T11:43:01.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:01.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:01.302776+0000) 2024-03-06T11:43:02.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:02.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:02.303025+0000) 2024-03-06T11:43:03.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:03.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:03.303283+0000) 2024-03-06T11:43:04.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:04.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:04.303534+0000) 2024-03-06T11:43:05.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:05.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:05.303792+0000) 2024-03-06T11:43:05.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:05.568 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:06.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:06.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:06.304053+0000) 2024-03-06T11:43:07.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:07.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:07.304303+0000) 2024-03-06T11:43:08.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:08.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:08.304556+0000) 2024-03-06T11:43:09.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:09.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:09.304795+0000) 2024-03-06T11:43:10.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:10.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:10.304987+0000) 2024-03-06T11:43:11.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:11.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:11.305246+0000) 2024-03-06T11:43:12.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:12.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:12.305425+0000) 2024-03-06T11:43:13.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:13.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:13.305697+0000) 2024-03-06T11:43:14.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:14.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:14.305954+0000) 2024-03-06T11:43:15.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:15.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:15.306214+0000) 2024-03-06T11:43:16.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:16.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:16.306453+0000) 2024-03-06T11:43:17.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:17.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:17.306744+0000) 2024-03-06T11:43:18.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:18.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:18.307018+0000) 2024-03-06T11:43:19.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:19.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:19.307274+0000) 2024-03-06T11:43:20.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:20.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:20.307524+0000) 2024-03-06T11:43:21.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:21.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:21.307778+0000) 2024-03-06T11:43:22.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:22.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:22.308012+0000) 2024-03-06T11:43:23.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:23.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:23.308251+0000) 2024-03-06T11:43:24.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:24.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:24.308490+0000) 2024-03-06T11:43:25.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:25.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:25.308731+0000) 2024-03-06T11:43:26.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:26.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:26.308913+0000) 2024-03-06T11:43:27.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:27.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:27.309161+0000) 2024-03-06T11:43:28.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:28.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:28.309402+0000) 2024-03-06T11:43:29.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:29.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:29.309707+0000) 2024-03-06T11:43:30.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:30.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:30.309960+0000) 2024-03-06T11:43:31.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:31.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:31.310220+0000) 2024-03-06T11:43:32.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:32.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:32.310466+0000) 2024-03-06T11:43:33.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:33.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:33.310754+0000) 2024-03-06T11:43:34.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:34.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:34.310989+0000) 2024-03-06T11:43:35.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:35.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:35.311245+0000) 2024-03-06T11:43:35.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:35.699 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:36.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:36.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:36.311508+0000) 2024-03-06T11:43:37.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:37.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:37.311755+0000) 2024-03-06T11:43:38.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:38.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:38.312003+0000) 2024-03-06T11:43:39.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:39.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:39.312258+0000) 2024-03-06T11:43:40.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:40.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:40.312500+0000) 2024-03-06T11:43:41.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:41.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:41.312756+0000) 2024-03-06T11:43:42.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:42.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:42.312932+0000) 2024-03-06T11:43:43.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:43.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:43.313164+0000) 2024-03-06T11:43:44.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:44.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:44.313348+0000) 2024-03-06T11:43:45.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:45.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:45.313601+0000) 2024-03-06T11:43:46.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:46.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:46.313882+0000) 2024-03-06T11:43:47.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:47.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:47.314125+0000) 2024-03-06T11:43:48.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:48.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:48.314442+0000) 2024-03-06T11:43:49.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:49.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:49.314734+0000) 2024-03-06T11:43:50.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:50.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:50.314933+0000) 2024-03-06T11:43:51.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:51.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:51.315125+0000) 2024-03-06T11:43:52.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:52.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:52.315372+0000) 2024-03-06T11:43:53.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:53.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:53.315614+0000) 2024-03-06T11:43:54.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:54.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:54.315915+0000) 2024-03-06T11:43:55.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:55.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:55.316164+0000) 2024-03-06T11:43:56.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:56.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:56.316406+0000) 2024-03-06T11:43:57.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:57.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:57.316723+0000) 2024-03-06T11:43:58.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:58.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:58.316993+0000) 2024-03-06T11:43:59.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:43:59.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:59.317239+0000) 2024-03-06T11:44:00.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:00.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:00.317473+0000) 2024-03-06T11:44:01.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:01.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:01.317732+0000) 2024-03-06T11:44:02.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:02.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:02.317970+0000) 2024-03-06T11:44:03.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:03.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:03.318243+0000) 2024-03-06T11:44:04.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:04.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:04.318502+0000) 2024-03-06T11:44:05.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:05.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:05.318775+0000) 2024-03-06T11:44:05.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:05.845 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:06.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:06.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:06.318965+0000) 2024-03-06T11:44:07.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:07.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:07.319222+0000) 2024-03-06T11:44:08.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:08.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:08.319475+0000) 2024-03-06T11:44:09.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:09.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:09.319763+0000) 2024-03-06T11:44:10.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:10.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:10.320003+0000) 2024-03-06T11:44:11.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:11.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:11.320249+0000) 2024-03-06T11:44:12.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:12.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:12.320489+0000) 2024-03-06T11:44:13.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:13.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:13.320750+0000) 2024-03-06T11:44:14.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:14.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:14.321014+0000) 2024-03-06T11:44:15.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:15.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:15.321208+0000) 2024-03-06T11:44:16.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:16.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:16.321471+0000) 2024-03-06T11:44:17.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:17.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:17.321749+0000) 2024-03-06T11:44:18.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:18.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:18.322011+0000) 2024-03-06T11:44:19.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:19.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:19.322246+0000) 2024-03-06T11:44:20.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:20.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:20.322569+0000) 2024-03-06T11:44:21.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:21.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:21.322805+0000) 2024-03-06T11:44:22.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:22.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:22.323045+0000) 2024-03-06T11:44:23.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:23.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:23.323289+0000) 2024-03-06T11:44:24.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:24.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:24.323536+0000) 2024-03-06T11:44:25.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:25.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:25.323786+0000) 2024-03-06T11:44:26.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:26.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:26.324040+0000) 2024-03-06T11:44:27.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:27.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:27.324279+0000) 2024-03-06T11:44:28.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:28.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:28.324551+0000) 2024-03-06T11:44:29.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:29.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:29.324787+0000) 2024-03-06T11:44:30.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:30.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:30.325037+0000) 2024-03-06T11:44:31.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:31.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:31.325339+0000) 2024-03-06T11:44:32.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:32.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:32.325599+0000) 2024-03-06T11:44:33.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:33.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:33.325860+0000) 2024-03-06T11:44:34.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:34.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:34.326095+0000) 2024-03-06T11:44:35.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:35.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:35.326353+0000) 2024-03-06T11:44:35.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:35.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:36.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:36.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:36.326554+0000) 2024-03-06T11:44:37.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:37.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:37.326815+0000) 2024-03-06T11:44:38.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:38.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:38.327078+0000) 2024-03-06T11:44:39.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:39.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:39.327318+0000) 2024-03-06T11:44:40.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:40.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:40.327578+0000) 2024-03-06T11:44:41.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:41.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:41.327830+0000) 2024-03-06T11:44:42.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:42.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:42.328085+0000) 2024-03-06T11:44:43.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:43.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:43.328345+0000) 2024-03-06T11:44:44.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:44.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:44.328607+0000) 2024-03-06T11:44:45.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:45.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:45.328915+0000) 2024-03-06T11:44:46.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:46.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:46.329168+0000) 2024-03-06T11:44:47.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:47.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:47.329427+0000) 2024-03-06T11:44:48.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:48.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:48.329719+0000) 2024-03-06T11:44:49.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:49.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:49.329964+0000) 2024-03-06T11:44:50.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:50.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:50.330219+0000) 2024-03-06T11:44:51.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:51.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:51.330456+0000) 2024-03-06T11:44:52.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:52.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:52.330745+0000) 2024-03-06T11:44:53.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:53.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:53.330999+0000) 2024-03-06T11:44:54.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:54.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:54.331237+0000) 2024-03-06T11:44:55.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:55.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:55.331482+0000) 2024-03-06T11:44:56.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:56.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:56.331740+0000) 2024-03-06T11:44:57.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:57.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:57.331997+0000) 2024-03-06T11:44:58.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:58.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:58.332261+0000) 2024-03-06T11:44:59.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:44:59.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:59.332524+0000) 2024-03-06T11:45:00.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:00.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:00.332778+0000) 2024-03-06T11:45:01.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:01.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:01.333040+0000) 2024-03-06T11:45:02.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:02.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:02.333269+0000) 2024-03-06T11:45:03.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:03.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:03.333574+0000) 2024-03-06T11:45:04.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:04.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:04.333873+0000) 2024-03-06T11:45:05.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:05.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:05.334186+0000) 2024-03-06T11:45:06.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:06.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:06.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:06.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:06.334502+0000) 2024-03-06T11:45:07.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:07.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:07.334786+0000) 2024-03-06T11:45:08.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:08.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:08.335109+0000) 2024-03-06T11:45:09.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:09.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:09.335416+0000) 2024-03-06T11:45:10.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:10.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:10.335769+0000) 2024-03-06T11:45:11.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:11.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:11.336051+0000) 2024-03-06T11:45:12.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:12.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:12.336334+0000) 2024-03-06T11:45:13.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:13.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:13.336695+0000) 2024-03-06T11:45:14.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:14.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:14.337021+0000) 2024-03-06T11:45:15.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:15.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:15.337298+0000) 2024-03-06T11:45:16.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:16.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:16.337594+0000) 2024-03-06T11:45:17.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:17.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:17.337931+0000) 2024-03-06T11:45:18.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:18.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:18.338232+0000) 2024-03-06T11:45:19.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:19.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:19.338528+0000) 2024-03-06T11:45:20.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:20.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:20.338834+0000) 2024-03-06T11:45:21.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:21.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:21.339149+0000) 2024-03-06T11:45:22.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:22.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:22.339450+0000) 2024-03-06T11:45:23.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:23.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:23.339770+0000) 2024-03-06T11:45:24.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:24.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:24.340059+0000) 2024-03-06T11:45:25.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:25.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:25.340365+0000) 2024-03-06T11:45:26.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:26.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:26.340715+0000) 2024-03-06T11:45:27.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:27.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:27.341024+0000) 2024-03-06T11:45:28.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:28.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:28.341284+0000) 2024-03-06T11:45:29.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:29.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:29.341531+0000) 2024-03-06T11:45:30.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:30.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:30.341822+0000) 2024-03-06T11:45:31.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:31.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:31.342047+0000) 2024-03-06T11:45:32.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:32.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:32.342347+0000) 2024-03-06T11:45:33.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:33.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:33.342696+0000) 2024-03-06T11:45:34.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:34.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:34.343003+0000) 2024-03-06T11:45:35.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:35.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:35.343310+0000) 2024-03-06T11:45:36.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:36.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:36.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:36.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:36.343510+0000) 2024-03-06T11:45:37.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:37.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:37.343755+0000) 2024-03-06T11:45:38.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:38.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:38.344019+0000) 2024-03-06T11:45:39.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:39.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:39.344285+0000) 2024-03-06T11:45:40.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:40.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:40.344534+0000) 2024-03-06T11:45:41.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:41.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:41.344864+0000) 2024-03-06T11:45:42.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:42.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:42.345174+0000) 2024-03-06T11:45:43.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:43.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:43.345464+0000) 2024-03-06T11:45:44.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:44.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:44.345772+0000) 2024-03-06T11:45:45.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:45.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:45.346066+0000) 2024-03-06T11:45:46.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:46.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:46.346370+0000) 2024-03-06T11:45:47.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:47.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:47.346730+0000) 2024-03-06T11:45:48.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:48.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:48.347050+0000) 2024-03-06T11:45:49.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:49.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:49.347344+0000) 2024-03-06T11:45:50.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:50.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:50.347608+0000) 2024-03-06T11:45:51.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:51.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:51.347901+0000) 2024-03-06T11:45:52.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:52.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:52.348157+0000) 2024-03-06T11:45:53.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:53.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:53.348407+0000) 2024-03-06T11:45:54.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:54.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:54.348720+0000) 2024-03-06T11:45:55.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:55.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:55.348978+0000) 2024-03-06T11:45:56.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:56.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:56.349213+0000) 2024-03-06T11:45:57.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:57.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:57.349449+0000) 2024-03-06T11:45:58.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:58.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:58.349747+0000) 2024-03-06T11:45:59.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:45:59.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:59.350008+0000) 2024-03-06T11:46:00.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:00.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:00.350239+0000) 2024-03-06T11:46:01.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:01.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:01.350484+0000) 2024-03-06T11:46:02.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:02.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:02.350746+0000) 2024-03-06T11:46:03.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:03.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:03.350998+0000) 2024-03-06T11:46:04.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:04.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:04.351255+0000) 2024-03-06T11:46:05.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:05.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:05.351514+0000) 2024-03-06T11:46:06.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:06.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:06.351765+0000) 2024-03-06T11:46:06.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:06.432 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:07.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:07.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:07.352025+0000) 2024-03-06T11:46:08.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:08.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:08.352289+0000) 2024-03-06T11:46:09.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:09.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:09.352536+0000) 2024-03-06T11:46:10.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:10.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:10.352771+0000) 2024-03-06T11:46:11.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:11.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:11.352949+0000) 2024-03-06T11:46:12.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:12.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:12.353243+0000) 2024-03-06T11:46:13.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:13.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:13.353483+0000) 2024-03-06T11:46:14.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:14.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:14.353735+0000) 2024-03-06T11:46:15.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:15.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:15.353990+0000) 2024-03-06T11:46:16.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:16.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:16.354239+0000) 2024-03-06T11:46:17.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:17.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:17.354503+0000) 2024-03-06T11:46:18.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:18.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:18.354789+0000) 2024-03-06T11:46:19.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:19.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:19.355025+0000) 2024-03-06T11:46:20.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:20.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:20.355275+0000) 2024-03-06T11:46:21.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:21.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:21.355536+0000) 2024-03-06T11:46:22.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:22.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:22.355797+0000) 2024-03-06T11:46:23.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:23.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:23.356051+0000) 2024-03-06T11:46:24.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:24.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:24.356293+0000) 2024-03-06T11:46:25.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:25.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:25.356550+0000) 2024-03-06T11:46:26.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:26.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:26.356813+0000) 2024-03-06T11:46:27.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:27.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:27.357026+0000) 2024-03-06T11:46:28.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:28.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:28.357305+0000) 2024-03-06T11:46:29.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:29.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:29.357583+0000) 2024-03-06T11:46:30.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:30.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:30.357849+0000) 2024-03-06T11:46:31.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:31.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:31.358118+0000) 2024-03-06T11:46:32.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:32.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:32.358377+0000) 2024-03-06T11:46:33.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:33.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:33.358686+0000) 2024-03-06T11:46:34.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:34.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:34.358968+0000) 2024-03-06T11:46:35.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:35.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:35.359222+0000) 2024-03-06T11:46:36.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:36.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:36.359474+0000) 2024-03-06T11:46:36.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:36.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:37.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:37.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:37.359737+0000) 2024-03-06T11:46:38.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:38.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:38.360017+0000) 2024-03-06T11:46:39.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:39.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:39.360247+0000) 2024-03-06T11:46:40.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:40.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:40.360503+0000) 2024-03-06T11:46:41.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:41.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:41.360774+0000) 2024-03-06T11:46:42.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:42.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:42.360991+0000) 2024-03-06T11:46:43.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:43.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:43.361261+0000) 2024-03-06T11:46:44.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:44.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:44.361536+0000) 2024-03-06T11:46:45.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:45.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:45.361812+0000) 2024-03-06T11:46:46.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:46.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:46.362068+0000) 2024-03-06T11:46:47.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:47.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:47.362346+0000) 2024-03-06T11:46:48.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:48.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:48.362625+0000) 2024-03-06T11:46:49.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:49.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:49.362955+0000) 2024-03-06T11:46:50.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:50.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:50.363206+0000) 2024-03-06T11:46:51.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:51.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:51.363479+0000) 2024-03-06T11:46:52.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:52.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:52.363756+0000) 2024-03-06T11:46:53.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:53.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:53.364001+0000) 2024-03-06T11:46:54.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:54.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:54.364254+0000) 2024-03-06T11:46:55.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:55.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:55.364514+0000) 2024-03-06T11:46:56.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:56.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:56.364786+0000) 2024-03-06T11:46:57.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:57.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:57.365051+0000) 2024-03-06T11:46:58.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:58.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:58.365297+0000) 2024-03-06T11:46:59.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:46:59.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:59.365530+0000) 2024-03-06T11:47:00.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:00.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:00.365790+0000) 2024-03-06T11:47:01.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:01.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:01.366031+0000) 2024-03-06T11:47:02.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:02.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:02.366296+0000) 2024-03-06T11:47:03.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:03.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:03.366557+0000) 2024-03-06T11:47:04.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:04.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:04.366822+0000) 2024-03-06T11:47:05.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:05.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:05.367079+0000) 2024-03-06T11:47:06.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:06.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:06.367314+0000) 2024-03-06T11:47:06.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:06.735 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:07.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:07.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:07.367561+0000) 2024-03-06T11:47:08.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:08.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:08.367816+0000) 2024-03-06T11:47:09.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:09.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:09.368058+0000) 2024-03-06T11:47:10.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:10.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:10.368301+0000) 2024-03-06T11:47:11.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:11.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:11.368615+0000) 2024-03-06T11:47:12.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:12.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:12.368934+0000) 2024-03-06T11:47:13.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:13.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:13.369193+0000) 2024-03-06T11:47:14.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:14.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:14.369457+0000) 2024-03-06T11:47:15.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:15.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:15.369738+0000) 2024-03-06T11:47:16.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:16.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:16.369994+0000) 2024-03-06T11:47:17.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:17.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:17.370254+0000) 2024-03-06T11:47:18.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:18.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:18.370518+0000) 2024-03-06T11:47:19.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:19.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:19.370783+0000) 2024-03-06T11:47:20.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:20.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:20.371013+0000) 2024-03-06T11:47:21.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:21.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:21.371219+0000) 2024-03-06T11:47:22.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:22.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:22.371420+0000) 2024-03-06T11:47:23.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:23.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:23.371694+0000) 2024-03-06T11:47:24.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:24.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:24.371956+0000) 2024-03-06T11:47:25.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:25.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:25.372237+0000) 2024-03-06T11:47:26.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:26.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:26.372493+0000) 2024-03-06T11:47:27.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:27.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:27.372758+0000) 2024-03-06T11:47:28.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:28.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:28.373022+0000) 2024-03-06T11:47:29.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:29.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:29.373271+0000) 2024-03-06T11:47:30.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:30.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:30.373517+0000) 2024-03-06T11:47:31.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:31.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:31.373746+0000) 2024-03-06T11:47:32.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:32.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:32.374020+0000) 2024-03-06T11:47:33.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:33.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:33.374271+0000) 2024-03-06T11:47:34.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:34.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:34.374532+0000) 2024-03-06T11:47:35.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:35.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:35.374795+0000) 2024-03-06T11:47:36.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:36.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:36.375023+0000) 2024-03-06T11:47:36.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:36.874 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:37.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:37.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:37.375262+0000) 2024-03-06T11:47:38.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:38.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:38.375510+0000) 2024-03-06T11:47:39.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:39.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:39.375760+0000) 2024-03-06T11:47:40.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:40.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:40.376017+0000) 2024-03-06T11:47:41.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:41.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:41.376283+0000) 2024-03-06T11:47:42.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:42.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:42.376494+0000) 2024-03-06T11:47:43.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:43.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:43.376757+0000) 2024-03-06T11:47:44.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:44.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:44.377021+0000) 2024-03-06T11:47:45.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:45.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:45.377267+0000) 2024-03-06T11:47:46.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:46.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:46.377530+0000) 2024-03-06T11:47:47.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:47.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:47.377774+0000) 2024-03-06T11:47:48.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:48.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:48.378044+0000) 2024-03-06T11:47:49.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:49.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:49.378302+0000) 2024-03-06T11:47:50.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:50.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:50.378555+0000) 2024-03-06T11:47:51.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:51.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:51.378793+0000) 2024-03-06T11:47:52.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:52.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:52.379026+0000) 2024-03-06T11:47:53.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:53.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:53.379225+0000) 2024-03-06T11:47:54.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:54.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:54.379419+0000) 2024-03-06T11:47:55.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:55.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:55.379694+0000) 2024-03-06T11:47:56.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:56.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:56.379954+0000) 2024-03-06T11:47:57.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:57.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:57.380194+0000) 2024-03-06T11:47:58.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:58.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:58.380461+0000) 2024-03-06T11:47:59.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:47:59.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:59.380810+0000) 2024-03-06T11:48:00.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:00.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:00.381041+0000) 2024-03-06T11:48:01.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:01.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:01.381303+0000) 2024-03-06T11:48:02.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:02.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:02.381542+0000) 2024-03-06T11:48:03.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:03.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:03.381780+0000) 2024-03-06T11:48:04.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:04.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:04.382044+0000) 2024-03-06T11:48:05.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:05.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:05.382295+0000) 2024-03-06T11:48:06.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:06.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:06.382552+0000) 2024-03-06T11:48:07.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:07.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:07.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:07.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:07.382806+0000) 2024-03-06T11:48:08.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:08.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:08.383059+0000) 2024-03-06T11:48:09.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:09.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:09.383310+0000) 2024-03-06T11:48:10.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:10.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:10.383566+0000) 2024-03-06T11:48:11.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:11.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:11.383820+0000) 2024-03-06T11:48:12.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:12.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:12.384074+0000) 2024-03-06T11:48:13.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:13.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:13.384313+0000) 2024-03-06T11:48:14.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:14.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:14.384559+0000) 2024-03-06T11:48:15.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:15.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:15.384827+0000) 2024-03-06T11:48:16.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:16.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:16.385079+0000) 2024-03-06T11:48:17.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:17.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:17.385330+0000) 2024-03-06T11:48:18.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:18.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:18.385573+0000) 2024-03-06T11:48:19.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:19.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:19.385811+0000) 2024-03-06T11:48:20.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:20.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:20.386068+0000) 2024-03-06T11:48:21.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:21.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:21.386321+0000) 2024-03-06T11:48:22.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:22.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:22.386601+0000) 2024-03-06T11:48:23.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:23.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:23.386915+0000) 2024-03-06T11:48:24.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:24.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:24.387118+0000) 2024-03-06T11:48:25.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:25.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:25.387377+0000) 2024-03-06T11:48:26.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:26.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:26.387663+0000) 2024-03-06T11:48:27.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:27.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:27.387915+0000) 2024-03-06T11:48:28.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:28.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:28.388177+0000) 2024-03-06T11:48:29.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:29.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:29.388417+0000) 2024-03-06T11:48:30.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:30.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:30.388687+0000) 2024-03-06T11:48:31.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:31.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:31.388945+0000) 2024-03-06T11:48:32.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:32.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:32.389199+0000) 2024-03-06T11:48:33.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:33.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:33.389479+0000) 2024-03-06T11:48:34.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:34.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:34.389744+0000) 2024-03-06T11:48:35.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:35.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:35.390002+0000) 2024-03-06T11:48:36.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:36.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:36.390244+0000) 2024-03-06T11:48:37.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:37.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:37.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:37.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:37.390488+0000) 2024-03-06T11:48:38.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:38.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:38.390741+0000) 2024-03-06T11:48:39.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:39.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:39.390986+0000) 2024-03-06T11:48:40.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:40.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:40.391241+0000) 2024-03-06T11:48:41.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:41.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:41.391557+0000) 2024-03-06T11:48:42.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:42.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:42.391809+0000) 2024-03-06T11:48:43.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:43.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:43.392064+0000) 2024-03-06T11:48:44.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:44.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:44.392307+0000) 2024-03-06T11:48:45.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:45.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:45.392545+0000) 2024-03-06T11:48:46.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:46.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:46.392798+0000) 2024-03-06T11:48:47.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:47.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:47.393056+0000) 2024-03-06T11:48:48.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:48.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:48.393300+0000) 2024-03-06T11:48:49.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:49.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:49.393567+0000) 2024-03-06T11:48:50.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:50.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:50.393785+0000) 2024-03-06T11:48:51.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:51.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:51.394025+0000) 2024-03-06T11:48:52.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:52.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:52.394287+0000) 2024-03-06T11:48:53.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:53.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:53.394558+0000) 2024-03-06T11:48:54.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:54.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:54.394825+0000) 2024-03-06T11:48:55.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:55.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:55.395093+0000) 2024-03-06T11:48:56.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:56.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:56.395351+0000) 2024-03-06T11:48:57.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:57.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:57.395618+0000) 2024-03-06T11:48:58.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:58.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:58.395927+0000) 2024-03-06T11:48:59.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:48:59.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:59.396133+0000) 2024-03-06T11:49:00.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:00.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:00.396371+0000) 2024-03-06T11:49:01.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:01.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:01.396558+0000) 2024-03-06T11:49:02.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:02.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:02.396754+0000) 2024-03-06T11:49:03.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:03.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:03.397013+0000) 2024-03-06T11:49:04.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:04.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:04.397266+0000) 2024-03-06T11:49:05.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:05.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:05.397509+0000) 2024-03-06T11:49:06.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:06.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:06.397756+0000) 2024-03-06T11:49:07.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:07.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:07.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:07.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:07.397958+0000) 2024-03-06T11:49:08.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:08.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:08.398228+0000) 2024-03-06T11:49:09.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:09.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:09.398492+0000) 2024-03-06T11:49:10.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:10.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:10.398747+0000) 2024-03-06T11:49:11.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:11.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:11.399010+0000) 2024-03-06T11:49:12.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:12.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:12.399264+0000) 2024-03-06T11:49:13.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:13.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:13.399527+0000) 2024-03-06T11:49:14.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:14.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:14.399796+0000) 2024-03-06T11:49:15.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:15.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:15.400058+0000) 2024-03-06T11:49:16.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:16.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:16.400313+0000) 2024-03-06T11:49:17.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:17.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:17.400552+0000) 2024-03-06T11:49:18.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:18.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:18.400758+0000) 2024-03-06T11:49:19.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:19.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:19.401014+0000) 2024-03-06T11:49:20.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:20.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:20.401204+0000) 2024-03-06T11:49:21.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:21.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:21.401396+0000) 2024-03-06T11:49:22.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:22.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:22.401669+0000) 2024-03-06T11:49:23.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:23.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:23.401861+0000) 2024-03-06T11:49:24.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:24.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:24.402048+0000) 2024-03-06T11:49:25.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:25.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:25.402246+0000) 2024-03-06T11:49:26.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:26.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:26.402478+0000) 2024-03-06T11:49:27.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:27.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:27.402737+0000) 2024-03-06T11:49:28.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:28.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:28.402940+0000) 2024-03-06T11:49:29.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:29.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:29.403252+0000) 2024-03-06T11:49:30.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:30.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:30.403512+0000) 2024-03-06T11:49:31.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:31.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:31.403774+0000) 2024-03-06T11:49:32.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:32.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:32.404010+0000) 2024-03-06T11:49:33.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:33.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:33.404263+0000) 2024-03-06T11:49:34.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:34.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:34.404517+0000) 2024-03-06T11:49:35.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:35.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:35.404779+0000) 2024-03-06T11:49:36.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:36.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:36.405019+0000) 2024-03-06T11:49:37.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:37.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:37.405266+0000) 2024-03-06T11:49:37.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:37.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:38.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:38.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:38.405520+0000) 2024-03-06T11:49:39.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:39.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:39.405772+0000) 2024-03-06T11:49:40.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:40.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:40.406028+0000) 2024-03-06T11:49:41.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:41.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:41.406264+0000) 2024-03-06T11:49:42.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:42.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:42.406512+0000) 2024-03-06T11:49:43.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:43.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:43.406749+0000) 2024-03-06T11:49:44.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:44.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:44.406989+0000) 2024-03-06T11:49:45.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:45.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:45.407245+0000) 2024-03-06T11:49:46.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:46.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:46.407498+0000) 2024-03-06T11:49:47.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:47.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:47.407751+0000) 2024-03-06T11:49:48.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:48.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:48.407996+0000) 2024-03-06T11:49:49.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:49.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:49.408235+0000) 2024-03-06T11:49:50.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:50.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:50.408424+0000) 2024-03-06T11:49:51.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:51.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:51.408773+0000) 2024-03-06T11:49:52.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:52.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:52.408961+0000) 2024-03-06T11:49:53.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:53.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:53.409215+0000) 2024-03-06T11:49:54.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:54.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:54.409464+0000) 2024-03-06T11:49:55.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:55.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:55.409751+0000) 2024-03-06T11:49:56.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:56.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:56.409999+0000) 2024-03-06T11:49:57.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:57.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:57.410260+0000) 2024-03-06T11:49:58.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:58.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:58.410513+0000) 2024-03-06T11:49:59.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:49:59.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:59.410752+0000) 2024-03-06T11:50:00.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:00.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:00.411005+0000) 2024-03-06T11:50:01.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:01.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:01.411242+0000) 2024-03-06T11:50:02.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:02.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:02.411493+0000) 2024-03-06T11:50:03.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:03.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:03.411752+0000) 2024-03-06T11:50:04.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:04.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:04.411947+0000) 2024-03-06T11:50:05.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:05.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:05.412201+0000) 2024-03-06T11:50:06.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:06.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:06.412496+0000) 2024-03-06T11:50:07.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:07.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:07.412800+0000) 2024-03-06T11:50:07.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:07.626 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:08.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:08.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:08.413122+0000) 2024-03-06T11:50:09.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:09.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:09.413374+0000) 2024-03-06T11:50:10.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:10.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:10.413663+0000) 2024-03-06T11:50:11.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:11.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:11.413921+0000) 2024-03-06T11:50:12.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:12.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:12.414176+0000) 2024-03-06T11:50:13.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:13.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:13.414413+0000) 2024-03-06T11:50:14.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:14.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:14.414682+0000) 2024-03-06T11:50:15.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:15.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:15.414896+0000) 2024-03-06T11:50:16.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:16.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:16.415148+0000) 2024-03-06T11:50:17.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:17.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:17.415413+0000) 2024-03-06T11:50:18.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:18.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:18.415714+0000) 2024-03-06T11:50:19.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:19.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:19.415980+0000) 2024-03-06T11:50:20.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:20.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:20.416220+0000) 2024-03-06T11:50:21.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:21.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:21.416460+0000) 2024-03-06T11:50:22.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:22.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:22.416758+0000) 2024-03-06T11:50:23.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:23.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:23.417021+0000) 2024-03-06T11:50:24.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:24.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:24.417276+0000) 2024-03-06T11:50:25.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:25.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:25.417522+0000) 2024-03-06T11:50:26.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:26.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:26.417761+0000) 2024-03-06T11:50:27.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:27.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:27.418013+0000) 2024-03-06T11:50:28.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:28.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:28.418283+0000) 2024-03-06T11:50:29.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:29.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:29.418467+0000) 2024-03-06T11:50:30.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:30.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:30.418748+0000) 2024-03-06T11:50:31.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:31.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:31.418986+0000) 2024-03-06T11:50:32.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:32.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:32.419244+0000) 2024-03-06T11:50:33.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:33.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:33.419514+0000) 2024-03-06T11:50:34.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:34.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:34.419750+0000) 2024-03-06T11:50:35.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:35.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:35.419993+0000) 2024-03-06T11:50:36.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:36.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:36.420241+0000) 2024-03-06T11:50:37.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:37.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:37.420486+0000) 2024-03-06T11:50:37.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:37.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:38.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:38.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:38.420752+0000) 2024-03-06T11:50:39.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:39.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:39.420978+0000) 2024-03-06T11:50:40.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:40.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:40.421213+0000) 2024-03-06T11:50:41.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:41.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:41.421476+0000) 2024-03-06T11:50:42.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:42.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:42.421731+0000) 2024-03-06T11:50:43.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:43.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:43.421973+0000) 2024-03-06T11:50:44.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:44.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:44.422232+0000) 2024-03-06T11:50:45.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:45.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:45.422470+0000) 2024-03-06T11:50:46.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:46.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:46.422748+0000) 2024-03-06T11:50:47.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:47.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:47.423004+0000) 2024-03-06T11:50:48.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:48.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:48.423274+0000) 2024-03-06T11:50:49.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:49.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:49.423473+0000) 2024-03-06T11:50:50.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:50.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:50.423742+0000) 2024-03-06T11:50:51.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:51.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:51.423998+0000) 2024-03-06T11:50:52.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:52.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:52.424235+0000) 2024-03-06T11:50:53.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:53.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:53.424478+0000) 2024-03-06T11:50:54.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:54.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:54.424747+0000) 2024-03-06T11:50:55.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:55.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:55.425011+0000) 2024-03-06T11:50:56.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:56.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:56.425293+0000) 2024-03-06T11:50:57.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:57.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:57.425559+0000) 2024-03-06T11:50:58.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:58.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:58.425753+0000) 2024-03-06T11:50:59.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:50:59.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:59.426008+0000) 2024-03-06T11:51:00.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:00.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:00.426267+0000) 2024-03-06T11:51:01.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:01.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:01.426528+0000) 2024-03-06T11:51:02.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:02.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:02.426789+0000) 2024-03-06T11:51:03.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:03.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:03.427064+0000) 2024-03-06T11:51:04.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:04.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:04.427303+0000) 2024-03-06T11:51:05.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:05.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:05.427551+0000) 2024-03-06T11:51:06.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:06.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:06.427797+0000) 2024-03-06T11:51:07.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:07.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:07.428041+0000) 2024-03-06T11:51:07.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:07.916 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:08.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:08.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:08.428318+0000) 2024-03-06T11:51:09.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:09.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:09.428574+0000) 2024-03-06T11:51:10.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:10.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:10.428816+0000) 2024-03-06T11:51:11.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:11.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:11.429060+0000) 2024-03-06T11:51:12.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:12.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:12.429319+0000) 2024-03-06T11:51:13.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:13.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:13.429573+0000) 2024-03-06T11:51:14.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:14.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:14.429834+0000) 2024-03-06T11:51:15.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:15.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:15.430093+0000) 2024-03-06T11:51:16.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:16.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:16.430409+0000) 2024-03-06T11:51:17.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:17.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:17.430686+0000) 2024-03-06T11:51:18.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:18.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:18.430953+0000) 2024-03-06T11:51:19.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:19.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:19.431211+0000) 2024-03-06T11:51:20.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:20.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:20.431453+0000) 2024-03-06T11:51:21.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:21.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:21.431741+0000) 2024-03-06T11:51:22.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:22.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:22.431995+0000) 2024-03-06T11:51:23.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:23.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:23.432252+0000) 2024-03-06T11:51:24.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:24.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:24.432505+0000) 2024-03-06T11:51:25.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:25.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:25.432754+0000) 2024-03-06T11:51:26.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:26.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:26.432993+0000) 2024-03-06T11:51:27.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:27.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:27.433234+0000) 2024-03-06T11:51:28.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:28.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:28.433509+0000) 2024-03-06T11:51:29.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:29.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:29.433762+0000) 2024-03-06T11:51:30.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:30.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:30.434005+0000) 2024-03-06T11:51:31.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:31.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:31.434188+0000) 2024-03-06T11:51:32.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:32.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:32.434442+0000) 2024-03-06T11:51:33.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:33.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:33.434678+0000) 2024-03-06T11:51:34.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:34.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:34.434917+0000) 2024-03-06T11:51:35.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:35.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:35.435171+0000) 2024-03-06T11:51:36.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:36.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:36.435427+0000) 2024-03-06T11:51:37.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:37.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:37.435731+0000) 2024-03-06T11:51:38.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:38.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:38.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:38.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:38.435938+0000) 2024-03-06T11:51:39.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:39.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:39.436196+0000) 2024-03-06T11:51:40.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:40.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:40.436438+0000) 2024-03-06T11:51:41.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:41.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:41.436719+0000) 2024-03-06T11:51:42.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:42.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:42.436957+0000) 2024-03-06T11:51:43.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:43.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:43.437217+0000) 2024-03-06T11:51:44.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:44.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:44.437477+0000) 2024-03-06T11:51:45.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:45.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:45.437738+0000) 2024-03-06T11:51:46.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:46.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:46.437929+0000) 2024-03-06T11:51:47.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:47.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:47.438192+0000) 2024-03-06T11:51:48.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:48.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:48.438445+0000) 2024-03-06T11:51:49.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:49.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:49.438722+0000) 2024-03-06T11:51:50.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:50.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:50.438915+0000) 2024-03-06T11:51:51.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:51.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:51.439169+0000) 2024-03-06T11:51:52.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:52.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:52.439425+0000) 2024-03-06T11:51:53.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:53.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:53.439718+0000) 2024-03-06T11:51:54.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:54.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:54.439945+0000) 2024-03-06T11:51:55.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:55.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:55.440146+0000) 2024-03-06T11:51:56.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:56.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:56.440380+0000) 2024-03-06T11:51:57.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:57.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:57.440674+0000) 2024-03-06T11:51:58.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:58.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:58.440956+0000) 2024-03-06T11:51:59.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:51:59.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:59.441219+0000) 2024-03-06T11:52:00.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:00.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:00.441479+0000) 2024-03-06T11:52:01.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:01.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:01.441736+0000) 2024-03-06T11:52:02.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:02.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:02.441974+0000) 2024-03-06T11:52:03.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:03.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:03.442221+0000) 2024-03-06T11:52:04.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:04.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:04.442473+0000) 2024-03-06T11:52:05.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:05.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:05.442749+0000) 2024-03-06T11:52:06.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:06.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:06.442951+0000) 2024-03-06T11:52:07.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:07.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:07.443193+0000) 2024-03-06T11:52:08.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:08.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:08.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:08.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:08.443446+0000) 2024-03-06T11:52:09.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:09.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:09.443804+0000) 2024-03-06T11:52:10.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:10.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:10.444066+0000) 2024-03-06T11:52:11.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:11.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:11.444304+0000) 2024-03-06T11:52:12.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:12.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:12.444539+0000) 2024-03-06T11:52:13.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:13.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:13.444815+0000) 2024-03-06T11:52:14.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:14.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:14.445056+0000) 2024-03-06T11:52:15.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:15.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:15.445297+0000) 2024-03-06T11:52:16.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:16.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:16.445548+0000) 2024-03-06T11:52:17.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:17.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:17.445811+0000) 2024-03-06T11:52:18.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:18.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:18.446080+0000) 2024-03-06T11:52:19.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:19.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:19.446307+0000) 2024-03-06T11:52:20.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:20.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:20.446583+0000) 2024-03-06T11:52:21.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:21.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:21.446840+0000) 2024-03-06T11:52:22.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:22.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:22.447098+0000) 2024-03-06T11:52:23.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:23.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:23.447299+0000) 2024-03-06T11:52:24.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:24.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:24.447535+0000) 2024-03-06T11:52:25.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:25.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:25.447793+0000) 2024-03-06T11:52:26.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:26.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:26.448048+0000) 2024-03-06T11:52:27.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:27.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:27.448295+0000) 2024-03-06T11:52:28.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:28.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:28.448562+0000) 2024-03-06T11:52:29.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:29.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:29.448818+0000) 2024-03-06T11:52:30.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:30.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:30.449065+0000) 2024-03-06T11:52:31.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:31.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:31.449331+0000) 2024-03-06T11:52:32.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:32.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:32.449570+0000) 2024-03-06T11:52:33.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:33.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:33.449795+0000) 2024-03-06T11:52:34.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:34.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:34.450042+0000) 2024-03-06T11:52:35.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:35.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:35.450292+0000) 2024-03-06T11:52:36.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:36.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:36.450535+0000) 2024-03-06T11:52:37.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:37.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:37.450791+0000) 2024-03-06T11:52:38.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:38.364 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:38.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:38.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:38.451031+0000) 2024-03-06T11:52:39.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:39.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:39.451312+0000) 2024-03-06T11:52:40.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:40.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:40.451683+0000) 2024-03-06T11:52:41.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:41.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:41.452007+0000) 2024-03-06T11:52:42.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:42.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:42.452311+0000) 2024-03-06T11:52:43.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:43.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:43.452614+0000) 2024-03-06T11:52:44.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:44.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:44.452967+0000) 2024-03-06T11:52:45.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:45.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:45.453258+0000) 2024-03-06T11:52:46.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:46.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:46.453573+0000) 2024-03-06T11:52:47.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:47.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:47.453860+0000) 2024-03-06T11:52:48.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:48.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:48.454189+0000) 2024-03-06T11:52:49.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:49.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:49.454495+0000) 2024-03-06T11:52:50.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:50.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:50.454797+0000) 2024-03-06T11:52:51.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:51.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:51.455104+0000) 2024-03-06T11:52:52.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:52.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:52.455464+0000) 2024-03-06T11:52:53.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:53.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:53.455787+0000) 2024-03-06T11:52:54.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:54.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:54.456084+0000) 2024-03-06T11:52:55.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:55.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:55.456391+0000) 2024-03-06T11:52:56.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:56.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:56.456749+0000) 2024-03-06T11:52:57.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:57.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:57.457044+0000) 2024-03-06T11:52:58.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:58.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:58.457348+0000) 2024-03-06T11:52:59.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:52:59.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:59.457702+0000) 2024-03-06T11:53:00.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:00.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:00.458021+0000) 2024-03-06T11:53:01.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:01.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:01.458320+0000) 2024-03-06T11:53:02.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:02.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:02.458561+0000) 2024-03-06T11:53:03.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:03.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:03.458829+0000) 2024-03-06T11:53:04.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:04.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:04.459141+0000) 2024-03-06T11:53:05.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:05.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:05.459453+0000) 2024-03-06T11:53:06.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:06.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:06.459785+0000) 2024-03-06T11:53:07.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:07.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:07.460097+0000) 2024-03-06T11:53:08.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:08.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:08.460353+0000) 2024-03-06T11:53:08.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:08.507 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:09.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:09.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:09.460723+0000) 2024-03-06T11:53:10.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:10.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:10.461019+0000) 2024-03-06T11:53:11.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:11.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:11.461336+0000) 2024-03-06T11:53:12.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:12.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:12.461686+0000) 2024-03-06T11:53:13.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:13.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:13.461940+0000) 2024-03-06T11:53:14.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:14.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:14.462244+0000) 2024-03-06T11:53:15.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:15.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:15.462534+0000) 2024-03-06T11:53:16.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:16.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:16.462787+0000) 2024-03-06T11:53:17.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:17.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:17.463116+0000) 2024-03-06T11:53:18.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:18.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:18.463443+0000) 2024-03-06T11:53:19.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:19.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:19.463781+0000) 2024-03-06T11:53:20.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:20.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:20.464095+0000) 2024-03-06T11:53:21.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:21.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:21.464393+0000) 2024-03-06T11:53:22.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:22.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:22.464660+0000) 2024-03-06T11:53:23.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:23.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:23.464989+0000) 2024-03-06T11:53:24.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:24.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:24.465255+0000) 2024-03-06T11:53:25.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:25.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:25.465503+0000) 2024-03-06T11:53:26.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:26.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:26.465745+0000) 2024-03-06T11:53:27.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:27.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:27.466002+0000) 2024-03-06T11:53:28.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:28.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:28.466269+0000) 2024-03-06T11:53:29.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:29.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:29.466543+0000) 2024-03-06T11:53:30.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:30.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:30.466826+0000) 2024-03-06T11:53:31.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:31.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:31.467101+0000) 2024-03-06T11:53:32.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:32.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:32.467374+0000) 2024-03-06T11:53:33.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:33.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:33.467674+0000) 2024-03-06T11:53:34.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:34.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:34.467928+0000) 2024-03-06T11:53:35.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:35.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:35.468186+0000) 2024-03-06T11:53:36.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:36.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:36.468458+0000) 2024-03-06T11:53:37.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:37.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:37.468761+0000) 2024-03-06T11:53:38.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:38.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:38.469029+0000) 2024-03-06T11:53:38.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:38.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:39.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:39.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:39.469307+0000) 2024-03-06T11:53:40.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:40.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:40.469681+0000) 2024-03-06T11:53:41.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:41.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:41.469972+0000) 2024-03-06T11:53:42.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:42.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:42.470243+0000) 2024-03-06T11:53:43.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:43.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:43.470521+0000) 2024-03-06T11:53:44.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:44.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:44.470784+0000) 2024-03-06T11:53:45.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:45.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:45.471046+0000) 2024-03-06T11:53:46.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:46.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:46.471320+0000) 2024-03-06T11:53:47.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:47.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:47.471593+0000) 2024-03-06T11:53:48.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:48.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:48.471888+0000) 2024-03-06T11:53:49.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:49.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:49.472147+0000) 2024-03-06T11:53:50.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:50.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:50.472416+0000) 2024-03-06T11:53:51.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:51.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:51.472733+0000) 2024-03-06T11:53:52.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:52.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:52.472991+0000) 2024-03-06T11:53:53.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:53.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:53.473267+0000) 2024-03-06T11:53:54.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:54.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:54.473544+0000) 2024-03-06T11:53:55.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:55.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:55.473822+0000) 2024-03-06T11:53:56.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:56.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:56.474098+0000) 2024-03-06T11:53:57.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:57.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:57.474372+0000) 2024-03-06T11:53:58.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:58.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:58.474694+0000) 2024-03-06T11:53:59.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:53:59.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:59.474952+0000) 2024-03-06T11:54:00.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:00.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:00.475207+0000) 2024-03-06T11:54:01.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:01.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:01.475414+0000) 2024-03-06T11:54:02.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:02.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:02.475738+0000) 2024-03-06T11:54:03.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:03.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:03.475936+0000) 2024-03-06T11:54:04.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:04.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:04.476195+0000) 2024-03-06T11:54:05.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:05.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:05.476445+0000) 2024-03-06T11:54:06.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:06.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:06.476737+0000) 2024-03-06T11:54:07.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:07.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:07.477024+0000) 2024-03-06T11:54:08.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:08.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:08.477291+0000) 2024-03-06T11:54:08.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:08.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:09.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:09.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:09.477524+0000) 2024-03-06T11:54:10.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:10.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:10.477792+0000) 2024-03-06T11:54:11.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:11.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:11.478026+0000) 2024-03-06T11:54:12.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:12.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:12.478284+0000) 2024-03-06T11:54:13.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:13.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:13.478544+0000) 2024-03-06T11:54:14.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:14.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:14.478825+0000) 2024-03-06T11:54:15.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:15.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:15.479105+0000) 2024-03-06T11:54:16.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:16.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:16.479365+0000) 2024-03-06T11:54:17.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:17.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:17.479622+0000) 2024-03-06T11:54:18.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:18.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:18.479916+0000) 2024-03-06T11:54:19.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:19.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:19.480147+0000) 2024-03-06T11:54:20.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:20.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:20.480401+0000) 2024-03-06T11:54:21.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:21.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:21.480596+0000) 2024-03-06T11:54:22.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:22.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:22.480861+0000) 2024-03-06T11:54:23.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:23.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:23.481121+0000) 2024-03-06T11:54:24.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:24.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:24.481380+0000) 2024-03-06T11:54:25.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:25.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:25.481692+0000) 2024-03-06T11:54:26.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:26.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:26.481969+0000) 2024-03-06T11:54:27.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:27.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:27.482211+0000) 2024-03-06T11:54:28.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:28.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:28.482499+0000) 2024-03-06T11:54:29.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:29.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:29.482750+0000) 2024-03-06T11:54:30.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:30.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:30.483014+0000) 2024-03-06T11:54:31.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:31.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:31.483247+0000) 2024-03-06T11:54:32.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:32.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:32.483500+0000) 2024-03-06T11:54:33.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:33.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:33.483754+0000) 2024-03-06T11:54:34.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:34.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:34.483947+0000) 2024-03-06T11:54:35.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:35.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:35.484138+0000) 2024-03-06T11:54:36.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:36.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:36.484383+0000) 2024-03-06T11:54:37.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:37.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:37.484669+0000) 2024-03-06T11:54:38.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:38.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:38.484951+0000) 2024-03-06T11:54:38.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:38.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:39.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:39.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:39.485265+0000) 2024-03-06T11:54:40.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:40.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:40.485526+0000) 2024-03-06T11:54:41.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:41.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:41.485765+0000) 2024-03-06T11:54:42.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:42.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:42.486000+0000) 2024-03-06T11:54:43.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:43.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:43.486256+0000) 2024-03-06T11:54:44.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:44.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:44.486515+0000) 2024-03-06T11:54:45.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:45.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:45.486769+0000) 2024-03-06T11:54:46.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:46.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:46.487029+0000) 2024-03-06T11:54:47.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:47.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:47.487289+0000) 2024-03-06T11:54:48.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:48.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:48.487568+0000) 2024-03-06T11:54:49.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:49.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:49.487843+0000) 2024-03-06T11:54:50.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:50.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:50.488116+0000) 2024-03-06T11:54:51.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:51.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:51.488390+0000) 2024-03-06T11:54:52.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:52.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:52.488680+0000) 2024-03-06T11:54:53.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:53.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:53.488951+0000) 2024-03-06T11:54:54.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:54.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:54.489227+0000) 2024-03-06T11:54:55.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:55.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:55.489508+0000) 2024-03-06T11:54:56.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:56.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:56.489823+0000) 2024-03-06T11:54:57.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:57.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:57.490080+0000) 2024-03-06T11:54:58.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:58.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:58.490323+0000) 2024-03-06T11:54:59.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:54:59.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:59.490584+0000) 2024-03-06T11:55:00.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:00.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:00.490839+0000) 2024-03-06T11:55:01.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:01.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:01.491076+0000) 2024-03-06T11:55:02.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:02.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:02.491299+0000) 2024-03-06T11:55:03.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:03.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:03.491551+0000) 2024-03-06T11:55:04.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:04.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:04.491806+0000) 2024-03-06T11:55:05.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:05.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:05.492064+0000) 2024-03-06T11:55:06.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:06.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:06.492324+0000) 2024-03-06T11:55:07.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:07.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:07.492578+0000) 2024-03-06T11:55:08.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:08.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:08.492842+0000) 2024-03-06T11:55:09.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:09.115 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:09.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:09.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:09.493100+0000) 2024-03-06T11:55:10.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:10.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:10.493356+0000) 2024-03-06T11:55:11.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:11.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:11.493621+0000) 2024-03-06T11:55:12.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:12.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:12.493892+0000) 2024-03-06T11:55:13.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:13.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:13.494131+0000) 2024-03-06T11:55:14.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:14.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:14.494369+0000) 2024-03-06T11:55:15.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:15.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:15.494611+0000) 2024-03-06T11:55:16.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:16.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:16.494900+0000) 2024-03-06T11:55:17.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:17.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:17.495151+0000) 2024-03-06T11:55:18.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:18.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:18.495412+0000) 2024-03-06T11:55:19.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:19.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:19.495715+0000) 2024-03-06T11:55:20.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:20.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:20.495979+0000) 2024-03-06T11:55:21.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:21.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:21.496244+0000) 2024-03-06T11:55:22.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:22.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:22.496502+0000) 2024-03-06T11:55:23.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:23.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:23.496757+0000) 2024-03-06T11:55:24.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:24.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:24.497014+0000) 2024-03-06T11:55:25.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:25.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:25.497268+0000) 2024-03-06T11:55:26.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:26.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:26.497526+0000) 2024-03-06T11:55:27.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:27.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:27.497834+0000) 2024-03-06T11:55:28.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:28.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:28.498092+0000) 2024-03-06T11:55:29.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:29.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:29.498335+0000) 2024-03-06T11:55:30.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:30.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:30.498583+0000) 2024-03-06T11:55:31.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:31.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:31.498816+0000) 2024-03-06T11:55:32.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:32.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:32.499043+0000) 2024-03-06T11:55:33.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:33.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:33.499337+0000) 2024-03-06T11:55:34.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:34.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:34.499593+0000) 2024-03-06T11:55:35.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:35.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:35.499831+0000) 2024-03-06T11:55:36.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:36.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:36.500092+0000) 2024-03-06T11:55:37.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:37.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:37.500320+0000) 2024-03-06T11:55:38.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:38.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:38.500585+0000) 2024-03-06T11:55:39.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:39.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:39.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:39.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:39.500851+0000) 2024-03-06T11:55:40.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:40.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:40.501087+0000) 2024-03-06T11:55:41.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:41.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:41.501328+0000) 2024-03-06T11:55:42.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:42.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:42.501581+0000) 2024-03-06T11:55:43.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:43.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:43.501821+0000) 2024-03-06T11:55:44.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:44.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:44.502063+0000) 2024-03-06T11:55:45.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:45.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:45.502309+0000) 2024-03-06T11:55:46.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:46.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:46.502555+0000) 2024-03-06T11:55:47.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:47.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:47.502792+0000) 2024-03-06T11:55:48.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:48.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:48.503052+0000) 2024-03-06T11:55:49.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:49.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:49.503307+0000) 2024-03-06T11:55:50.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:50.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:50.503539+0000) 2024-03-06T11:55:51.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:51.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:51.503774+0000) 2024-03-06T11:55:52.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:52.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:52.504026+0000) 2024-03-06T11:55:53.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:53.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:53.504264+0000) 2024-03-06T11:55:54.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:54.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:54.504516+0000) 2024-03-06T11:55:55.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:55.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:55.504767+0000) 2024-03-06T11:55:56.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:56.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:56.505026+0000) 2024-03-06T11:55:57.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:57.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:57.505282+0000) 2024-03-06T11:55:58.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:58.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:58.505552+0000) 2024-03-06T11:55:59.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:55:59.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:59.505812+0000) 2024-03-06T11:56:00.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:00.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:00.506057+0000) 2024-03-06T11:56:01.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:01.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:01.506316+0000) 2024-03-06T11:56:02.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:02.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:02.506554+0000) 2024-03-06T11:56:03.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:03.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:03.506796+0000) 2024-03-06T11:56:04.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:04.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:04.507055+0000) 2024-03-06T11:56:05.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:05.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:05.507312+0000) 2024-03-06T11:56:06.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:06.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:06.507549+0000) 2024-03-06T11:56:07.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:07.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:07.507785+0000) 2024-03-06T11:56:08.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:08.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:08.508043+0000) 2024-03-06T11:56:09.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:09.395 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:09.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:09.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:09.508323+0000) 2024-03-06T11:56:10.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:10.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:10.508619+0000) 2024-03-06T11:56:11.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:11.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:11.508965+0000) 2024-03-06T11:56:12.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:12.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:12.509251+0000) 2024-03-06T11:56:13.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:13.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:13.509507+0000) 2024-03-06T11:56:14.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:14.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:14.509765+0000) 2024-03-06T11:56:15.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:15.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:15.510071+0000) 2024-03-06T11:56:16.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:16.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:16.510324+0000) 2024-03-06T11:56:17.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:17.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:17.510564+0000) 2024-03-06T11:56:18.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:18.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:18.510834+0000) 2024-03-06T11:56:19.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:19.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:19.511073+0000) 2024-03-06T11:56:20.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:20.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:20.511379+0000) 2024-03-06T11:56:21.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:21.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:21.511720+0000) 2024-03-06T11:56:22.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:22.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:22.512018+0000) 2024-03-06T11:56:23.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:23.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:23.512328+0000) 2024-03-06T11:56:24.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:24.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:24.512679+0000) 2024-03-06T11:56:25.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:25.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:25.512972+0000) 2024-03-06T11:56:26.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:26.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:26.513283+0000) 2024-03-06T11:56:27.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:27.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:27.513581+0000) 2024-03-06T11:56:28.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:28.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:28.513936+0000) 2024-03-06T11:56:29.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:29.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:29.514221+0000) 2024-03-06T11:56:30.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:30.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:30.514497+0000) 2024-03-06T11:56:31.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:31.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:31.514791+0000) 2024-03-06T11:56:32.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:32.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:32.515067+0000) 2024-03-06T11:56:33.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:33.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:33.515315+0000) 2024-03-06T11:56:34.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:34.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:34.515584+0000) 2024-03-06T11:56:35.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:35.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:35.515825+0000) 2024-03-06T11:56:36.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:36.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:36.516071+0000) 2024-03-06T11:56:37.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:37.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:37.516312+0000) 2024-03-06T11:56:38.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:38.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:38.516559+0000) 2024-03-06T11:56:39.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:39.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:39.516805+0000) 2024-03-06T11:56:39.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:39.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:40.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:40.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:40.517067+0000) 2024-03-06T11:56:41.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:41.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:41.517325+0000) 2024-03-06T11:56:42.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:42.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:42.517582+0000) 2024-03-06T11:56:43.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:43.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:43.517837+0000) 2024-03-06T11:56:44.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:44.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:44.518091+0000) 2024-03-06T11:56:45.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:45.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:45.518389+0000) 2024-03-06T11:56:46.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:46.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:46.518601+0000) 2024-03-06T11:56:47.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:47.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:47.518908+0000) 2024-03-06T11:56:48.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:48.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:48.519197+0000) 2024-03-06T11:56:49.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:49.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:49.519438+0000) 2024-03-06T11:56:50.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:50.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:50.519743+0000) 2024-03-06T11:56:51.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:51.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:51.520000+0000) 2024-03-06T11:56:52.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:52.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:52.520251+0000) 2024-03-06T11:56:53.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:53.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:53.520529+0000) 2024-03-06T11:56:54.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:54.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:54.520805+0000) 2024-03-06T11:56:55.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:55.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:55.521075+0000) 2024-03-06T11:56:56.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:56.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:56.521313+0000) 2024-03-06T11:56:57.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:57.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:57.521582+0000) 2024-03-06T11:56:58.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:58.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:58.521887+0000) 2024-03-06T11:56:59.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:56:59.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:59.522149+0000) 2024-03-06T11:57:00.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:00.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:00.522346+0000) 2024-03-06T11:57:01.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:01.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:01.522582+0000) 2024-03-06T11:57:02.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:02.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:02.522812+0000) 2024-03-06T11:57:03.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:03.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:03.523071+0000) 2024-03-06T11:57:04.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:04.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:04.523330+0000) 2024-03-06T11:57:05.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:05.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:05.523573+0000) 2024-03-06T11:57:06.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:06.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:06.523813+0000) 2024-03-06T11:57:07.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:07.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:07.524050+0000) 2024-03-06T11:57:08.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:08.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:08.524355+0000) 2024-03-06T11:57:09.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:09.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:09.524606+0000) 2024-03-06T11:57:09.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:09.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:10.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:10.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:10.524905+0000) 2024-03-06T11:57:11.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:11.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:11.525170+0000) 2024-03-06T11:57:12.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:12.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:12.525425+0000) 2024-03-06T11:57:13.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:13.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:13.525716+0000) 2024-03-06T11:57:14.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:14.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:14.525975+0000) 2024-03-06T11:57:15.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:15.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:15.526231+0000) 2024-03-06T11:57:16.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:16.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:16.526489+0000) 2024-03-06T11:57:17.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:17.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:17.526745+0000) 2024-03-06T11:57:18.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:18.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:18.527038+0000) 2024-03-06T11:57:19.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:19.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:19.527312+0000) 2024-03-06T11:57:20.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:20.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:20.527583+0000) 2024-03-06T11:57:21.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:21.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:21.527862+0000) 2024-03-06T11:57:22.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:22.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:22.528137+0000) 2024-03-06T11:57:23.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:23.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:23.528408+0000) 2024-03-06T11:57:24.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:24.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:24.528710+0000) 2024-03-06T11:57:25.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:25.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:25.528970+0000) 2024-03-06T11:57:26.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:26.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:26.529246+0000) 2024-03-06T11:57:27.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:27.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:27.529527+0000) 2024-03-06T11:57:28.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:28.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:28.529797+0000) 2024-03-06T11:57:29.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:29.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:29.530071+0000) 2024-03-06T11:57:30.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:30.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:30.530324+0000) 2024-03-06T11:57:31.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:31.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:31.530598+0000) 2024-03-06T11:57:32.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:32.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:32.530917+0000) 2024-03-06T11:57:33.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:33.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:33.531170+0000) 2024-03-06T11:57:34.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:34.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:34.531444+0000) 2024-03-06T11:57:35.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:35.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:35.531744+0000) 2024-03-06T11:57:36.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:36.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:36.532012+0000) 2024-03-06T11:57:37.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:37.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:37.532286+0000) 2024-03-06T11:57:38.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:38.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:38.532568+0000) 2024-03-06T11:57:39.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:39.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:39.532846+0000) 2024-03-06T11:57:39.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:39.861 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:40.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:40.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:40.533053+0000) 2024-03-06T11:57:41.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:41.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:41.533329+0000) 2024-03-06T11:57:42.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:42.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:42.533587+0000) 2024-03-06T11:57:43.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:43.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:43.533886+0000) 2024-03-06T11:57:44.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:44.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:44.534166+0000) 2024-03-06T11:57:45.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:45.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:45.534436+0000) 2024-03-06T11:57:46.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:46.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:46.534751+0000) 2024-03-06T11:57:47.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:47.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:47.535018+0000) 2024-03-06T11:57:48.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:48.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:48.535295+0000) 2024-03-06T11:57:49.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:49.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:49.535559+0000) 2024-03-06T11:57:50.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:50.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:50.535816+0000) 2024-03-06T11:57:51.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:51.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:51.536104+0000) 2024-03-06T11:57:52.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:52.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:52.536364+0000) 2024-03-06T11:57:53.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:53.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:53.536618+0000) 2024-03-06T11:57:54.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:54.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:54.536931+0000) 2024-03-06T11:57:55.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:55.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:55.537185+0000) 2024-03-06T11:57:56.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:56.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:56.537453+0000) 2024-03-06T11:57:57.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:57.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:57.537753+0000) 2024-03-06T11:57:58.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:58.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:58.538008+0000) 2024-03-06T11:57:59.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:57:59.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:59.538268+0000) 2024-03-06T11:58:00.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:00.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:00.538525+0000) 2024-03-06T11:58:01.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:01.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:01.538816+0000) 2024-03-06T11:58:02.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:02.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:02.539080+0000) 2024-03-06T11:58:03.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:03.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:03.539339+0000) 2024-03-06T11:58:04.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:04.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:04.539608+0000) 2024-03-06T11:58:05.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:05.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:05.539908+0000) 2024-03-06T11:58:06.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:06.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:06.540187+0000) 2024-03-06T11:58:07.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:07.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:07.540465+0000) 2024-03-06T11:58:08.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:08.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:08.540760+0000) 2024-03-06T11:58:09.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:09.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:09.541073+0000) 2024-03-06T11:58:10.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:10.020 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:10.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:10.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:10.541354+0000) 2024-03-06T11:58:11.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:11.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:11.541669+0000) 2024-03-06T11:58:12.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:12.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:12.541951+0000) 2024-03-06T11:58:13.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:13.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:13.542223+0000) 2024-03-06T11:58:14.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:14.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:14.542503+0000) 2024-03-06T11:58:15.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:15.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:15.542800+0000) 2024-03-06T11:58:16.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:16.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:16.543084+0000) 2024-03-06T11:58:17.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:17.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:17.543346+0000) 2024-03-06T11:58:18.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:18.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:18.543680+0000) 2024-03-06T11:58:19.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:19.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:19.543956+0000) 2024-03-06T11:58:20.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:20.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:20.544218+0000) 2024-03-06T11:58:21.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:21.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:21.544502+0000) 2024-03-06T11:58:22.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:22.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:22.544754+0000) 2024-03-06T11:58:23.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:23.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:23.545032+0000) 2024-03-06T11:58:24.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:24.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:24.545304+0000) 2024-03-06T11:58:25.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:25.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:25.545576+0000) 2024-03-06T11:58:26.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:26.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:26.545862+0000) 2024-03-06T11:58:27.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:27.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:27.546134+0000) 2024-03-06T11:58:28.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:28.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:28.546414+0000) 2024-03-06T11:58:29.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:29.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:29.546720+0000) 2024-03-06T11:58:30.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:30.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:30.546987+0000) 2024-03-06T11:58:31.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:31.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:31.547259+0000) 2024-03-06T11:58:32.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:32.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:32.547519+0000) 2024-03-06T11:58:33.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:33.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:33.547801+0000) 2024-03-06T11:58:34.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:34.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:34.548074+0000) 2024-03-06T11:58:35.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:35.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:35.548330+0000) 2024-03-06T11:58:36.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:36.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:36.548604+0000) 2024-03-06T11:58:37.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:37.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:37.548916+0000) 2024-03-06T11:58:38.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:38.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:38.549183+0000) 2024-03-06T11:58:39.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:39.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:39.549459+0000) 2024-03-06T11:58:40.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:40.167 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:40.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:40.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:40.549729+0000) 2024-03-06T11:58:41.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:41.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:41.550011+0000) 2024-03-06T11:58:42.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:42.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:42.550284+0000) 2024-03-06T11:58:43.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:43.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:43.550558+0000) 2024-03-06T11:58:44.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:44.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:44.550901+0000) 2024-03-06T11:58:45.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:45.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:45.551110+0000) 2024-03-06T11:58:46.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:46.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:46.551380+0000) 2024-03-06T11:58:47.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:47.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:47.551705+0000) 2024-03-06T11:58:48.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:48.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:48.552001+0000) 2024-03-06T11:58:49.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:49.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:49.552260+0000) 2024-03-06T11:58:50.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:50.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:50.552507+0000) 2024-03-06T11:58:51.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:51.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:51.552775+0000) 2024-03-06T11:58:52.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:52.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:52.553053+0000) 2024-03-06T11:58:53.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:53.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:53.553312+0000) 2024-03-06T11:58:54.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:54.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:54.553586+0000) 2024-03-06T11:58:55.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:55.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:55.553915+0000) 2024-03-06T11:58:56.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:56.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:56.554176+0000) 2024-03-06T11:58:57.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:57.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:57.554445+0000) 2024-03-06T11:58:58.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:58.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:58.554767+0000) 2024-03-06T11:58:59.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:58:59.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:59.555045+0000) 2024-03-06T11:59:00.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:00.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:00.555332+0000) 2024-03-06T11:59:01.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:01.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:01.555606+0000) 2024-03-06T11:59:02.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:02.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:02.555930+0000) 2024-03-06T11:59:03.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:03.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:03.556196+0000) 2024-03-06T11:59:04.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:04.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:04.556472+0000) 2024-03-06T11:59:05.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:05.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:05.556769+0000) 2024-03-06T11:59:06.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:06.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:06.556979+0000) 2024-03-06T11:59:07.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:07.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:07.557213+0000) 2024-03-06T11:59:08.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:08.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:08.557480+0000) 2024-03-06T11:59:09.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:09.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:09.557735+0000) 2024-03-06T11:59:10.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:10.323 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:10.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:10.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:10.558004+0000) 2024-03-06T11:59:11.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:11.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:11.558274+0000) 2024-03-06T11:59:12.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:12.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:12.558536+0000) 2024-03-06T11:59:13.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:13.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:13.558796+0000) 2024-03-06T11:59:14.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:14.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:14.559034+0000) 2024-03-06T11:59:15.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:15.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:15.559278+0000) 2024-03-06T11:59:16.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:16.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:16.559540+0000) 2024-03-06T11:59:17.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:17.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:17.559829+0000) 2024-03-06T11:59:18.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:18.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:18.560117+0000) 2024-03-06T11:59:19.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:19.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:19.560411+0000) 2024-03-06T11:59:20.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:20.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:20.560697+0000) 2024-03-06T11:59:21.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:21.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:21.560962+0000) 2024-03-06T11:59:22.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:22.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:22.561228+0000) 2024-03-06T11:59:23.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:23.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:23.561416+0000) 2024-03-06T11:59:24.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:24.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:24.561700+0000) 2024-03-06T11:59:25.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:25.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:25.561948+0000) 2024-03-06T11:59:26.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:26.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:26.562201+0000) 2024-03-06T11:59:27.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:27.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:27.562452+0000) 2024-03-06T11:59:28.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:28.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:28.562742+0000) 2024-03-06T11:59:29.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:29.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:29.563000+0000) 2024-03-06T11:59:30.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:30.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:30.563260+0000) 2024-03-06T11:59:31.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:31.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:31.563516+0000) 2024-03-06T11:59:32.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:32.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:32.563785+0000) 2024-03-06T11:59:33.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:33.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:33.563998+0000) 2024-03-06T11:59:34.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:34.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:34.564253+0000) 2024-03-06T11:59:35.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:35.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:35.564499+0000) 2024-03-06T11:59:36.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:36.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:36.564754+0000) 2024-03-06T11:59:37.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:37.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:37.565072+0000) 2024-03-06T11:59:38.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:38.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:38.565344+0000) 2024-03-06T11:59:39.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:39.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:39.565600+0000) 2024-03-06T11:59:40.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:40.469 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:40.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:40.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:40.565795+0000) 2024-03-06T11:59:41.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:41.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:41.566022+0000) 2024-03-06T11:59:42.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:42.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:42.566287+0000) 2024-03-06T11:59:43.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:43.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:43.566538+0000) 2024-03-06T11:59:44.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:44.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:44.566795+0000) 2024-03-06T11:59:45.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:45.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:45.567035+0000) 2024-03-06T11:59:46.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:46.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:46.567292+0000) 2024-03-06T11:59:47.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:47.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:47.567549+0000) 2024-03-06T11:59:48.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:48.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:48.567804+0000) 2024-03-06T11:59:49.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:49.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:49.567998+0000) 2024-03-06T11:59:50.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:50.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:50.568245+0000) 2024-03-06T11:59:51.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:51.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:51.568477+0000) 2024-03-06T11:59:52.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:52.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:52.568744+0000) 2024-03-06T11:59:53.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:53.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:53.568980+0000) 2024-03-06T11:59:54.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:54.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:54.569220+0000) 2024-03-06T11:59:55.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:55.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:55.569478+0000) 2024-03-06T11:59:56.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:56.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:56.569755+0000) 2024-03-06T11:59:57.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:57.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:57.569994+0000) 2024-03-06T11:59:58.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:58.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:58.570262+0000) 2024-03-06T11:59:59.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T11:59:59.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:59.570495+0000) 2024-03-06T12:00:00.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:00.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:00.570765+0000) 2024-03-06T12:00:01.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:01.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:01.571022+0000) 2024-03-06T12:00:02.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:02.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:02.571263+0000) 2024-03-06T12:00:03.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:03.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:03.571455+0000) 2024-03-06T12:00:04.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:04.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:04.571746+0000) 2024-03-06T12:00:05.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:05.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:05.572008+0000) 2024-03-06T12:00:06.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:06.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:06.572265+0000) 2024-03-06T12:00:07.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:07.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:07.572523+0000) 2024-03-06T12:00:08.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:08.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:08.572790+0000) 2024-03-06T12:00:09.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:09.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:09.573035+0000) 2024-03-06T12:00:10.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:10.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:10.573356+0000) 2024-03-06T12:00:10.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:10.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:11.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:11.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:11.573671+0000) 2024-03-06T12:00:12.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:12.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:12.573938+0000) 2024-03-06T12:00:13.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:13.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:13.574177+0000) 2024-03-06T12:00:14.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:14.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:14.574440+0000) 2024-03-06T12:00:15.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:15.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:15.574738+0000) 2024-03-06T12:00:16.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:16.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:16.575015+0000) 2024-03-06T12:00:17.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:17.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:17.575293+0000) 2024-03-06T12:00:18.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:18.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:18.575558+0000) 2024-03-06T12:00:19.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:19.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:19.575813+0000) 2024-03-06T12:00:20.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:20.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:20.576076+0000) 2024-03-06T12:00:21.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:21.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:21.576331+0000) 2024-03-06T12:00:22.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:22.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:22.576590+0000) 2024-03-06T12:00:23.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:23.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:23.576901+0000) 2024-03-06T12:00:24.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:24.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:24.577166+0000) 2024-03-06T12:00:25.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:25.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:25.577477+0000) 2024-03-06T12:00:26.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:26.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:26.577755+0000) 2024-03-06T12:00:27.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:27.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:27.578012+0000) 2024-03-06T12:00:28.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:28.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:28.578293+0000) 2024-03-06T12:00:29.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:29.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:29.578560+0000) 2024-03-06T12:00:30.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:30.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:30.578829+0000) 2024-03-06T12:00:31.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:31.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:31.579085+0000) 2024-03-06T12:00:32.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:32.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:32.579359+0000) 2024-03-06T12:00:33.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:33.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:33.579674+0000) 2024-03-06T12:00:34.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:34.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:34.579928+0000) 2024-03-06T12:00:35.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:35.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:35.580186+0000) 2024-03-06T12:00:36.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:36.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:36.580458+0000) 2024-03-06T12:00:37.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:37.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:37.580766+0000) 2024-03-06T12:00:38.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:38.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:38.581040+0000) 2024-03-06T12:00:39.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:39.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:39.581293+0000) 2024-03-06T12:00:40.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:40.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:40.581549+0000) 2024-03-06T12:00:40.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:40.783 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:41.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:41.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:41.581809+0000) 2024-03-06T12:00:42.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:42.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:42.582012+0000) 2024-03-06T12:00:43.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:43.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:43.582264+0000) 2024-03-06T12:00:44.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:44.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:44.582513+0000) 2024-03-06T12:00:45.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:45.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:45.582756+0000) 2024-03-06T12:00:46.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:46.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:46.583014+0000) 2024-03-06T12:00:47.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:47.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:47.583260+0000) 2024-03-06T12:00:48.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:48.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:48.583529+0000) 2024-03-06T12:00:49.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:49.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:49.583784+0000) 2024-03-06T12:00:50.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:50.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:50.584042+0000) 2024-03-06T12:00:51.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:51.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:51.584248+0000) 2024-03-06T12:00:52.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:52.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:52.584523+0000) 2024-03-06T12:00:53.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:53.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:53.584777+0000) 2024-03-06T12:00:54.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:54.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:54.585054+0000) 2024-03-06T12:00:55.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:55.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:55.585311+0000) 2024-03-06T12:00:56.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:56.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:56.585587+0000) 2024-03-06T12:00:57.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:57.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:57.585909+0000) 2024-03-06T12:00:58.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:58.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:58.586192+0000) 2024-03-06T12:00:59.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:00:59.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:59.586468+0000) 2024-03-06T12:01:00.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:00.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:00.586763+0000) 2024-03-06T12:01:01.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:01.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:01.587026+0000) 2024-03-06T12:01:02.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:02.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:02.587285+0000) 2024-03-06T12:01:03.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:03.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:03.587524+0000) 2024-03-06T12:01:04.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:04.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:04.587761+0000) 2024-03-06T12:01:05.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:05.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:05.588032+0000) 2024-03-06T12:01:06.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:06.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:06.588285+0000) 2024-03-06T12:01:07.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:07.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:07.588540+0000) 2024-03-06T12:01:08.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:08.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:08.588809+0000) 2024-03-06T12:01:09.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:09.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:09.589078+0000) 2024-03-06T12:01:10.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:10.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:10.589334+0000) 2024-03-06T12:01:10.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:10.934 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:11.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:11.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:11.589593+0000) 2024-03-06T12:01:12.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:12.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:12.589895+0000) 2024-03-06T12:01:13.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:13.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:13.590227+0000) 2024-03-06T12:01:14.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:14.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:14.590478+0000) 2024-03-06T12:01:15.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:15.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:15.590749+0000) 2024-03-06T12:01:16.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:16.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:16.591009+0000) 2024-03-06T12:01:17.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:17.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:17.591270+0000) 2024-03-06T12:01:18.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:18.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:18.591539+0000) 2024-03-06T12:01:19.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:19.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:19.591810+0000) 2024-03-06T12:01:20.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:20.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:20.592084+0000) 2024-03-06T12:01:21.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:21.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:21.592358+0000) 2024-03-06T12:01:22.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:22.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:22.592614+0000) 2024-03-06T12:01:23.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:23.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:23.592944+0000) 2024-03-06T12:01:24.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:24.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:24.593217+0000) 2024-03-06T12:01:25.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:25.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:25.593492+0000) 2024-03-06T12:01:26.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:26.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:26.593764+0000) 2024-03-06T12:01:27.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:27.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:27.594035+0000) 2024-03-06T12:01:28.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:28.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:28.594303+0000) 2024-03-06T12:01:29.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:29.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:29.594561+0000) 2024-03-06T12:01:30.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:30.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:30.594858+0000) 2024-03-06T12:01:31.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:31.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:31.595137+0000) 2024-03-06T12:01:32.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:32.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:32.595412+0000) 2024-03-06T12:01:33.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:33.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:33.595737+0000) 2024-03-06T12:01:34.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:34.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:34.596013+0000) 2024-03-06T12:01:35.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:35.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:35.596294+0000) 2024-03-06T12:01:36.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:36.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:36.596564+0000) 2024-03-06T12:01:37.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:37.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:37.596838+0000) 2024-03-06T12:01:38.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:38.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:38.597127+0000) 2024-03-06T12:01:39.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:39.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:39.597390+0000) 2024-03-06T12:01:40.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:40.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:40.597719+0000) 2024-03-06T12:01:41.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:41.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:41.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:41.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:41.597981+0000) 2024-03-06T12:01:42.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:42.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:42.598258+0000) 2024-03-06T12:01:43.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:43.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:43.598510+0000) 2024-03-06T12:01:44.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:44.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:44.598782+0000) 2024-03-06T12:01:45.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:45.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:45.599042+0000) 2024-03-06T12:01:46.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:46.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:46.599245+0000) 2024-03-06T12:01:47.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:47.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:47.599512+0000) 2024-03-06T12:01:48.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:48.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:48.599780+0000) 2024-03-06T12:01:49.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:49.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:49.600015+0000) 2024-03-06T12:01:50.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:50.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:50.600275+0000) 2024-03-06T12:01:51.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:51.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:51.600510+0000) 2024-03-06T12:01:52.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:52.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:52.600774+0000) 2024-03-06T12:01:53.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:53.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:53.601046+0000) 2024-03-06T12:01:54.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:54.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:54.601323+0000) 2024-03-06T12:01:55.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:55.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:55.601603+0000) 2024-03-06T12:01:56.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:56.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:56.601921+0000) 2024-03-06T12:01:57.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:57.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:57.602201+0000) 2024-03-06T12:01:58.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:58.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:58.602474+0000) 2024-03-06T12:01:59.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:01:59.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:59.602754+0000) 2024-03-06T12:02:00.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:00.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:00.602975+0000) 2024-03-06T12:02:01.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:01.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:01.603257+0000) 2024-03-06T12:02:02.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:02.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:02.603518+0000) 2024-03-06T12:02:03.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:03.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:03.603791+0000) 2024-03-06T12:02:04.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:04.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:04.604073+0000) 2024-03-06T12:02:05.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:05.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:05.604334+0000) 2024-03-06T12:02:06.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:06.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:06.604615+0000) 2024-03-06T12:02:07.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:07.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:07.604982+0000) 2024-03-06T12:02:08.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:08.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:08.605276+0000) 2024-03-06T12:02:09.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:09.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:09.605548+0000) 2024-03-06T12:02:10.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:10.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:10.605826+0000) 2024-03-06T12:02:11.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:11.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:11.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:11.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:11.606106+0000) 2024-03-06T12:02:12.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:12.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:12.606373+0000) 2024-03-06T12:02:13.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:13.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:13.606669+0000) 2024-03-06T12:02:14.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:14.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:14.606954+0000) 2024-03-06T12:02:15.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:15.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:15.607193+0000) 2024-03-06T12:02:16.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:16.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:16.607438+0000) 2024-03-06T12:02:17.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:17.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:17.607741+0000) 2024-03-06T12:02:18.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:18.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:18.608002+0000) 2024-03-06T12:02:19.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:19.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:19.608232+0000) 2024-03-06T12:02:20.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:20.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:20.608468+0000) 2024-03-06T12:02:21.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:21.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:21.608767+0000) 2024-03-06T12:02:22.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:22.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:22.609030+0000) 2024-03-06T12:02:23.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:23.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:23.609277+0000) 2024-03-06T12:02:24.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:24.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:24.609558+0000) 2024-03-06T12:02:25.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:25.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:25.609825+0000) 2024-03-06T12:02:26.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:26.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:26.610036+0000) 2024-03-06T12:02:27.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:27.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:27.610308+0000) 2024-03-06T12:02:28.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:28.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:28.610604+0000) 2024-03-06T12:02:29.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:29.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:29.610882+0000) 2024-03-06T12:02:30.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:30.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:30.611166+0000) 2024-03-06T12:02:31.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:31.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:31.611432+0000) 2024-03-06T12:02:32.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:32.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:32.611760+0000) 2024-03-06T12:02:33.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:33.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:33.612046+0000) 2024-03-06T12:02:34.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:34.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:34.612245+0000) 2024-03-06T12:02:35.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:35.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:35.612514+0000) 2024-03-06T12:02:36.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:36.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:36.612756+0000) 2024-03-06T12:02:37.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:37.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:37.613024+0000) 2024-03-06T12:02:38.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:38.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:38.613278+0000) 2024-03-06T12:02:39.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:39.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:39.613539+0000) 2024-03-06T12:02:40.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:40.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:40.613785+0000) 2024-03-06T12:02:41.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:41.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:41.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:41.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:41.614056+0000) 2024-03-06T12:02:42.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:42.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:42.614315+0000) 2024-03-06T12:02:43.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:43.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:43.614560+0000) 2024-03-06T12:02:44.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:44.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:44.614818+0000) 2024-03-06T12:02:45.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:45.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:45.615064+0000) 2024-03-06T12:02:46.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:46.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:46.615306+0000) 2024-03-06T12:02:47.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:47.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:47.615573+0000) 2024-03-06T12:02:48.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:48.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:48.615847+0000) 2024-03-06T12:02:49.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:49.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:49.616137+0000) 2024-03-06T12:02:50.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:50.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:50.616385+0000) 2024-03-06T12:02:51.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:51.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:51.616661+0000) 2024-03-06T12:02:52.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:52.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:52.616929+0000) 2024-03-06T12:02:53.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:53.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:53.617191+0000) 2024-03-06T12:02:54.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:54.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:54.617436+0000) 2024-03-06T12:02:55.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:55.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:55.617812+0000) 2024-03-06T12:02:56.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:56.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:56.618101+0000) 2024-03-06T12:02:57.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:57.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:57.618374+0000) 2024-03-06T12:02:58.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:58.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:58.618658+0000) 2024-03-06T12:02:59.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:02:59.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:59.618895+0000) 2024-03-06T12:03:00.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:00.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:00.619142+0000) 2024-03-06T12:03:01.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:01.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:01.619374+0000) 2024-03-06T12:03:02.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:02.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:02.619662+0000) 2024-03-06T12:03:03.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:03.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:03.619866+0000) 2024-03-06T12:03:04.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:04.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:04.620129+0000) 2024-03-06T12:03:05.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:05.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:05.620433+0000) 2024-03-06T12:03:06.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:06.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:06.620780+0000) 2024-03-06T12:03:07.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:07.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:07.621085+0000) 2024-03-06T12:03:08.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:08.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:08.621403+0000) 2024-03-06T12:03:09.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:09.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:09.621742+0000) 2024-03-06T12:03:10.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:10.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:10.622053+0000) 2024-03-06T12:03:11.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:11.527 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:11.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:11.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:11.622250+0000) 2024-03-06T12:03:12.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:12.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:12.622537+0000) 2024-03-06T12:03:13.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:13.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:13.622823+0000) 2024-03-06T12:03:14.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:14.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:14.623082+0000) 2024-03-06T12:03:15.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:15.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:15.623338+0000) 2024-03-06T12:03:16.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:16.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:16.623602+0000) 2024-03-06T12:03:17.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:17.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:17.623901+0000) 2024-03-06T12:03:18.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:18.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:18.624176+0000) 2024-03-06T12:03:19.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:19.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:19.624437+0000) 2024-03-06T12:03:20.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:20.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:20.624818+0000) 2024-03-06T12:03:21.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:21.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:21.625038+0000) 2024-03-06T12:03:22.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:22.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:22.625273+0000) 2024-03-06T12:03:23.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:23.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:23.625523+0000) 2024-03-06T12:03:24.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:24.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:24.625774+0000) 2024-03-06T12:03:25.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:25.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:25.625979+0000) 2024-03-06T12:03:26.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:26.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:26.626179+0000) 2024-03-06T12:03:27.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:27.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:27.626428+0000) 2024-03-06T12:03:28.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:28.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:28.626937+0000) 2024-03-06T12:03:29.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:29.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:29.627215+0000) 2024-03-06T12:03:30.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:30.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:30.627470+0000) 2024-03-06T12:03:31.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:31.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:31.627764+0000) 2024-03-06T12:03:32.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:32.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:32.628035+0000) 2024-03-06T12:03:33.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:33.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:33.628278+0000) 2024-03-06T12:03:34.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:34.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:34.628518+0000) 2024-03-06T12:03:35.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:35.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:35.628747+0000) 2024-03-06T12:03:36.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:36.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:36.629007+0000) 2024-03-06T12:03:37.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:37.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:37.629309+0000) 2024-03-06T12:03:38.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:38.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:38.629623+0000) 2024-03-06T12:03:39.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:39.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:39.629996+0000) 2024-03-06T12:03:40.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:40.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:40.630265+0000) 2024-03-06T12:03:41.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:41.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:41.630596+0000) 2024-03-06T12:03:41.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:41.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:42.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:42.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:42.630940+0000) 2024-03-06T12:03:43.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:43.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:43.631239+0000) 2024-03-06T12:03:44.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:44.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:44.631503+0000) 2024-03-06T12:03:45.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:45.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:45.631787+0000) 2024-03-06T12:03:46.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:46.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:46.632057+0000) 2024-03-06T12:03:47.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:47.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:47.632293+0000) 2024-03-06T12:03:48.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:48.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:48.632564+0000) 2024-03-06T12:03:49.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:49.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:49.632814+0000) 2024-03-06T12:03:50.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:50.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:50.633074+0000) 2024-03-06T12:03:51.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:51.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:51.633272+0000) 2024-03-06T12:03:52.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:52.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:52.633510+0000) 2024-03-06T12:03:53.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:53.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:53.633776+0000) 2024-03-06T12:03:54.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:54.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:54.633967+0000) 2024-03-06T12:03:55.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:55.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:55.634230+0000) 2024-03-06T12:03:56.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:56.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:56.634492+0000) 2024-03-06T12:03:57.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:57.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:57.634747+0000) 2024-03-06T12:03:58.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:58.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:58.634988+0000) 2024-03-06T12:03:59.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:03:59.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:59.635182+0000) 2024-03-06T12:04:00.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:00.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:00.635436+0000) 2024-03-06T12:04:01.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:01.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:01.635739+0000) 2024-03-06T12:04:02.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:02.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:02.635978+0000) 2024-03-06T12:04:03.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:03.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:03.636220+0000) 2024-03-06T12:04:04.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:04.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:04.636482+0000) 2024-03-06T12:04:05.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:05.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:05.636730+0000) 2024-03-06T12:04:06.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:06.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:06.636987+0000) 2024-03-06T12:04:07.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:07.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:07.637229+0000) 2024-03-06T12:04:08.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:08.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:08.637494+0000) 2024-03-06T12:04:09.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:09.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:09.637752+0000) 2024-03-06T12:04:10.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:10.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:10.638005+0000) 2024-03-06T12:04:11.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:11.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:11.638263+0000) 2024-03-06T12:04:11.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:11.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:12.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:12.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:12.638525+0000) 2024-03-06T12:04:13.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:13.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:13.638785+0000) 2024-03-06T12:04:14.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:14.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:14.639046+0000) 2024-03-06T12:04:15.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:15.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:15.639279+0000) 2024-03-06T12:04:16.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:16.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:16.639539+0000) 2024-03-06T12:04:17.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:17.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:17.639797+0000) 2024-03-06T12:04:18.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:18.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:18.640045+0000) 2024-03-06T12:04:19.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:19.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:19.640300+0000) 2024-03-06T12:04:20.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:20.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:20.640583+0000) 2024-03-06T12:04:21.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:21.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:21.640850+0000) 2024-03-06T12:04:22.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:22.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:22.641136+0000) 2024-03-06T12:04:23.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:23.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:23.641451+0000) 2024-03-06T12:04:24.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:24.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:24.641796+0000) 2024-03-06T12:04:25.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:25.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:25.642101+0000) 2024-03-06T12:04:26.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:26.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:26.642403+0000) 2024-03-06T12:04:27.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:27.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:27.642751+0000) 2024-03-06T12:04:28.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:28.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:28.643005+0000) 2024-03-06T12:04:29.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:29.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:29.643315+0000) 2024-03-06T12:04:30.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:30.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:30.643603+0000) 2024-03-06T12:04:31.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:31.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:31.643964+0000) 2024-03-06T12:04:32.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:32.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:32.644274+0000) 2024-03-06T12:04:33.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:33.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:33.644578+0000) 2024-03-06T12:04:34.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:34.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:34.644902+0000) 2024-03-06T12:04:35.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:35.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:35.645196+0000) 2024-03-06T12:04:36.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:36.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:36.645542+0000) 2024-03-06T12:04:37.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:37.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:37.645854+0000) 2024-03-06T12:04:38.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:38.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:38.646185+0000) 2024-03-06T12:04:39.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:39.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:39.646488+0000) 2024-03-06T12:04:40.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:40.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:40.646794+0000) 2024-03-06T12:04:41.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:41.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:41.647086+0000) 2024-03-06T12:04:41.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:41.981 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:42.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:42.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:42.647396+0000) 2024-03-06T12:04:43.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:43.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:43.647745+0000) 2024-03-06T12:04:44.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:44.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:44.648022+0000) 2024-03-06T12:04:45.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:45.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:45.648309+0000) 2024-03-06T12:04:46.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:46.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:46.648613+0000) 2024-03-06T12:04:47.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:47.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:47.648985+0000) 2024-03-06T12:04:48.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:48.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:48.649327+0000) 2024-03-06T12:04:49.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:49.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:49.649625+0000) 2024-03-06T12:04:50.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:50.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:50.649991+0000) 2024-03-06T12:04:51.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:51.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:51.650288+0000) 2024-03-06T12:04:52.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:52.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:52.650570+0000) 2024-03-06T12:04:53.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:53.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:53.650884+0000) 2024-03-06T12:04:54.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:54.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:54.651180+0000) 2024-03-06T12:04:55.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:55.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:55.651490+0000) 2024-03-06T12:04:56.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:56.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:56.651805+0000) 2024-03-06T12:04:57.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:57.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:57.652097+0000) 2024-03-06T12:04:58.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:58.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:58.652426+0000) 2024-03-06T12:04:59.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:04:59.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:59.652771+0000) 2024-03-06T12:05:00.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:00.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:00.653070+0000) 2024-03-06T12:05:01.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:01.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:01.653375+0000) 2024-03-06T12:05:02.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:02.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:02.653720+0000) 2024-03-06T12:05:03.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:03.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:03.654035+0000) 2024-03-06T12:05:04.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:04.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:04.654357+0000) 2024-03-06T12:05:05.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:05.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:05.654723+0000) 2024-03-06T12:05:06.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:06.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:06.655018+0000) 2024-03-06T12:05:07.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:07.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:07.655334+0000) 2024-03-06T12:05:08.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:08.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:08.655686+0000) 2024-03-06T12:05:09.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:09.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:09.656001+0000) 2024-03-06T12:05:10.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:10.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:10.656320+0000) 2024-03-06T12:05:11.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:11.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:11.656683+0000) 2024-03-06T12:05:12.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:12.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:12.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:12.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:12.657016+0000) 2024-03-06T12:05:13.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:13.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:13.657315+0000) 2024-03-06T12:05:14.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:14.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:14.657681+0000) 2024-03-06T12:05:15.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:15.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:15.657928+0000) 2024-03-06T12:05:16.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:16.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:16.658261+0000) 2024-03-06T12:05:17.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:17.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:17.658544+0000) 2024-03-06T12:05:18.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:18.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:18.658854+0000) 2024-03-06T12:05:19.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:19.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:19.659162+0000) 2024-03-06T12:05:20.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:20.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:20.659420+0000) 2024-03-06T12:05:21.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:21.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:21.659784+0000) 2024-03-06T12:05:22.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:22.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:22.660100+0000) 2024-03-06T12:05:23.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:23.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:23.660407+0000) 2024-03-06T12:05:24.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:24.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:24.660808+0000) 2024-03-06T12:05:25.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:25.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:25.661079+0000) 2024-03-06T12:05:26.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:26.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:26.661331+0000) 2024-03-06T12:05:27.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:27.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:27.661589+0000) 2024-03-06T12:05:28.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:28.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:28.661901+0000) 2024-03-06T12:05:29.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:29.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:29.662164+0000) 2024-03-06T12:05:30.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:30.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:30.662421+0000) 2024-03-06T12:05:31.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:31.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:31.662715+0000) 2024-03-06T12:05:32.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:32.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:32.662975+0000) 2024-03-06T12:05:33.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:33.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:33.663232+0000) 2024-03-06T12:05:34.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:34.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:34.663425+0000) 2024-03-06T12:05:35.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:35.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:35.663732+0000) 2024-03-06T12:05:36.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:36.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:36.663977+0000) 2024-03-06T12:05:37.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:37.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:37.664235+0000) 2024-03-06T12:05:38.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:38.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:38.664511+0000) 2024-03-06T12:05:39.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:39.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:39.664766+0000) 2024-03-06T12:05:40.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:40.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:40.665031+0000) 2024-03-06T12:05:41.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:41.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:41.665302+0000) 2024-03-06T12:05:42.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:42.274 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:42.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:42.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:42.665567+0000) 2024-03-06T12:05:43.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:43.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:43.665835+0000) 2024-03-06T12:05:44.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:44.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:44.666057+0000) 2024-03-06T12:05:45.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:45.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:45.666301+0000) 2024-03-06T12:05:46.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:46.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:46.666549+0000) 2024-03-06T12:05:47.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:47.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:47.666796+0000) 2024-03-06T12:05:48.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:48.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:48.667075+0000) 2024-03-06T12:05:49.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:49.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:49.667260+0000) 2024-03-06T12:05:50.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:50.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:50.667517+0000) 2024-03-06T12:05:51.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:51.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:51.667768+0000) 2024-03-06T12:05:52.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:52.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:52.668000+0000) 2024-03-06T12:05:53.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:53.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:53.668271+0000) 2024-03-06T12:05:54.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:54.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:54.668525+0000) 2024-03-06T12:05:55.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:55.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:55.668781+0000) 2024-03-06T12:05:56.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:56.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:56.669050+0000) 2024-03-06T12:05:57.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:57.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:57.669290+0000) 2024-03-06T12:05:58.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:58.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:58.669506+0000) 2024-03-06T12:05:59.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:05:59.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:59.669753+0000) 2024-03-06T12:06:00.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:00.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:00.669957+0000) 2024-03-06T12:06:01.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:01.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:01.670214+0000) 2024-03-06T12:06:02.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:02.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:02.670453+0000) 2024-03-06T12:06:03.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:03.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:03.670760+0000) 2024-03-06T12:06:04.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:04.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:04.670996+0000) 2024-03-06T12:06:05.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:05.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:05.671255+0000) 2024-03-06T12:06:06.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:06.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:06.671538+0000) 2024-03-06T12:06:07.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:07.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:07.671784+0000) 2024-03-06T12:06:08.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:08.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:08.672038+0000) 2024-03-06T12:06:09.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:09.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:09.672279+0000) 2024-03-06T12:06:10.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:10.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:10.672536+0000) 2024-03-06T12:06:11.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:11.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:11.672783+0000) 2024-03-06T12:06:12.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:12.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:12.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:12.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:12.673102+0000) 2024-03-06T12:06:13.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:13.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:13.673384+0000) 2024-03-06T12:06:14.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:14.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:14.673625+0000) 2024-03-06T12:06:15.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:15.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:15.673903+0000) 2024-03-06T12:06:16.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:16.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:16.674114+0000) 2024-03-06T12:06:17.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:17.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:17.674375+0000) 2024-03-06T12:06:18.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:18.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:18.674688+0000) 2024-03-06T12:06:19.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:19.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:19.674947+0000) 2024-03-06T12:06:20.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:20.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:20.675188+0000) 2024-03-06T12:06:21.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:21.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:21.675438+0000) 2024-03-06T12:06:22.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:22.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:22.675735+0000) 2024-03-06T12:06:23.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:23.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:23.675944+0000) 2024-03-06T12:06:24.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:24.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:24.676205+0000) 2024-03-06T12:06:25.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:25.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:25.676442+0000) 2024-03-06T12:06:26.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:26.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:26.676678+0000) 2024-03-06T12:06:27.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:27.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:27.676939+0000) 2024-03-06T12:06:28.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:28.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:28.677210+0000) 2024-03-06T12:06:29.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:29.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:29.677450+0000) 2024-03-06T12:06:30.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:30.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:30.677754+0000) 2024-03-06T12:06:31.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:31.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:31.677970+0000) 2024-03-06T12:06:32.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:32.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:32.678237+0000) 2024-03-06T12:06:33.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:33.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:33.678521+0000) 2024-03-06T12:06:34.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:34.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:34.678792+0000) 2024-03-06T12:06:35.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:35.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:35.679012+0000) 2024-03-06T12:06:36.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:36.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:36.679297+0000) 2024-03-06T12:06:37.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:37.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:37.679561+0000) 2024-03-06T12:06:38.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:38.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:38.679868+0000) 2024-03-06T12:06:39.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:39.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:39.680149+0000) 2024-03-06T12:06:40.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:40.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:40.680421+0000) 2024-03-06T12:06:41.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:41.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:41.680751+0000) 2024-03-06T12:06:42.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:42.592 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:42.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:42.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:42.680978+0000) 2024-03-06T12:06:43.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:43.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:43.681246+0000) 2024-03-06T12:06:44.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:44.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:44.681522+0000) 2024-03-06T12:06:45.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:45.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:45.681798+0000) 2024-03-06T12:06:46.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:46.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:46.682068+0000) 2024-03-06T12:06:47.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:47.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:47.682287+0000) 2024-03-06T12:06:48.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:48.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:48.682564+0000) 2024-03-06T12:06:49.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:49.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:49.682851+0000) 2024-03-06T12:06:50.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:50.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:50.683122+0000) 2024-03-06T12:06:51.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:51.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:51.683409+0000) 2024-03-06T12:06:52.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:52.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:52.683720+0000) 2024-03-06T12:06:53.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:53.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:53.684004+0000) 2024-03-06T12:06:54.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:54.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:54.684266+0000) 2024-03-06T12:06:55.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:55.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:55.684554+0000) 2024-03-06T12:06:56.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:56.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:56.684824+0000) 2024-03-06T12:06:57.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:57.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:57.685088+0000) 2024-03-06T12:06:58.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:58.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:58.685326+0000) 2024-03-06T12:06:59.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:06:59.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:59.685591+0000) 2024-03-06T12:07:00.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:00.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:00.685883+0000) 2024-03-06T12:07:01.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:01.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:01.686151+0000) 2024-03-06T12:07:02.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:02.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:02.686422+0000) 2024-03-06T12:07:03.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:03.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:03.686747+0000) 2024-03-06T12:07:04.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:04.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:04.687031+0000) 2024-03-06T12:07:05.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:05.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:05.687349+0000) 2024-03-06T12:07:06.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:06.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:06.687617+0000) 2024-03-06T12:07:07.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:07.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:07.687944+0000) 2024-03-06T12:07:08.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:08.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:08.688243+0000) 2024-03-06T12:07:09.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:09.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:09.688502+0000) 2024-03-06T12:07:10.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:10.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:10.688789+0000) 2024-03-06T12:07:11.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:11.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:11.689072+0000) 2024-03-06T12:07:12.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:12.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:12.689357+0000) 2024-03-06T12:07:12.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:12.738 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:13.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:13.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:13.689667+0000) 2024-03-06T12:07:14.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:14.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:14.689943+0000) 2024-03-06T12:07:15.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:15.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:15.690220+0000) 2024-03-06T12:07:16.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:16.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:16.690484+0000) 2024-03-06T12:07:17.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:17.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:17.690757+0000) 2024-03-06T12:07:18.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:18.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:18.691054+0000) 2024-03-06T12:07:19.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:19.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:19.691335+0000) 2024-03-06T12:07:20.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:20.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:20.691603+0000) 2024-03-06T12:07:21.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:21.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:21.691957+0000) 2024-03-06T12:07:22.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:22.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:22.692219+0000) 2024-03-06T12:07:23.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:23.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:23.692507+0000) 2024-03-06T12:07:24.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:24.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:24.692785+0000) 2024-03-06T12:07:25.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:25.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:25.693062+0000) 2024-03-06T12:07:26.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:26.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:26.693336+0000) 2024-03-06T12:07:27.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:27.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:27.693623+0000) 2024-03-06T12:07:28.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:28.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:28.693919+0000) 2024-03-06T12:07:29.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:29.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:29.694170+0000) 2024-03-06T12:07:30.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:30.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:30.694445+0000) 2024-03-06T12:07:31.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:31.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:31.694761+0000) 2024-03-06T12:07:32.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:32.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:32.695047+0000) 2024-03-06T12:07:33.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:33.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:33.695279+0000) 2024-03-06T12:07:34.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:34.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:34.695563+0000) 2024-03-06T12:07:35.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:35.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:35.695834+0000) 2024-03-06T12:07:36.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:36.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:36.696088+0000) 2024-03-06T12:07:37.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:37.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:37.696365+0000) 2024-03-06T12:07:38.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:38.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:38.696671+0000) 2024-03-06T12:07:39.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:39.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:39.696933+0000) 2024-03-06T12:07:40.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:40.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:40.697153+0000) 2024-03-06T12:07:41.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:41.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:41.697417+0000) 2024-03-06T12:07:42.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:42.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:42.697721+0000) 2024-03-06T12:07:42.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:42.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:43.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:43.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:43.698002+0000) 2024-03-06T12:07:44.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:44.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:44.698269+0000) 2024-03-06T12:07:45.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:45.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:45.698550+0000) 2024-03-06T12:07:46.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:46.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:46.698834+0000) 2024-03-06T12:07:47.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:47.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:47.699096+0000) 2024-03-06T12:07:48.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:48.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:48.699394+0000) 2024-03-06T12:07:49.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:49.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:49.699620+0000) 2024-03-06T12:07:50.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:50.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:50.699953+0000) 2024-03-06T12:07:51.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:51.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:51.700229+0000) 2024-03-06T12:07:52.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:52.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:52.700515+0000) 2024-03-06T12:07:53.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:53.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:53.700828+0000) 2024-03-06T12:07:54.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:54.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:54.701106+0000) 2024-03-06T12:07:55.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:55.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:55.701391+0000) 2024-03-06T12:07:56.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:56.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:56.701606+0000) 2024-03-06T12:07:57.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:57.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:57.701938+0000) 2024-03-06T12:07:58.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:58.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:58.702249+0000) 2024-03-06T12:07:59.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:07:59.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:59.702519+0000) 2024-03-06T12:08:00.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:00.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:00.702803+0000) 2024-03-06T12:08:01.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:01.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:01.703066+0000) 2024-03-06T12:08:02.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:02.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:02.703328+0000) 2024-03-06T12:08:03.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:03.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:03.703591+0000) 2024-03-06T12:08:04.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:04.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:04.703897+0000) 2024-03-06T12:08:05.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:05.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:05.704085+0000) 2024-03-06T12:08:06.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:06.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:06.704346+0000) 2024-03-06T12:08:07.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:07.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:07.704676+0000) 2024-03-06T12:08:08.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:08.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:08.704986+0000) 2024-03-06T12:08:09.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:09.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:09.705276+0000) 2024-03-06T12:08:10.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:10.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:10.705536+0000) 2024-03-06T12:08:11.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:11.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:11.705785+0000) 2024-03-06T12:08:12.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:12.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:12.706075+0000) 2024-03-06T12:08:13.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:13.047 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:13.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:13.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:13.706350+0000) 2024-03-06T12:08:14.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:14.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:14.706677+0000) 2024-03-06T12:08:15.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:15.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:15.706933+0000) 2024-03-06T12:08:16.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:16.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:16.707213+0000) 2024-03-06T12:08:17.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:17.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:17.707490+0000) 2024-03-06T12:08:18.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:18.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:18.707766+0000) 2024-03-06T12:08:19.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:19.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:19.708039+0000) 2024-03-06T12:08:20.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:20.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:20.708316+0000) 2024-03-06T12:08:21.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:21.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:21.708587+0000) 2024-03-06T12:08:22.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:22.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:22.708901+0000) 2024-03-06T12:08:23.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:23.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:23.709165+0000) 2024-03-06T12:08:24.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:24.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:24.709443+0000) 2024-03-06T12:08:25.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:25.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:25.709677+0000) 2024-03-06T12:08:26.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:26.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:26.709881+0000) 2024-03-06T12:08:27.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:27.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:27.710163+0000) 2024-03-06T12:08:28.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:28.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:28.710432+0000) 2024-03-06T12:08:29.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:29.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:29.710746+0000) 2024-03-06T12:08:30.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:30.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:30.711002+0000) 2024-03-06T12:08:31.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:31.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:31.711278+0000) 2024-03-06T12:08:32.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:32.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:32.711550+0000) 2024-03-06T12:08:33.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:33.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:33.711804+0000) 2024-03-06T12:08:34.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:34.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:34.712036+0000) 2024-03-06T12:08:35.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:35.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:35.712286+0000) 2024-03-06T12:08:36.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:36.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:36.712553+0000) 2024-03-06T12:08:37.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:37.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:37.712814+0000) 2024-03-06T12:08:38.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:38.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:38.713077+0000) 2024-03-06T12:08:39.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:39.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:39.713344+0000) 2024-03-06T12:08:40.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:40.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:40.713607+0000) 2024-03-06T12:08:41.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:41.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:41.713955+0000) 2024-03-06T12:08:42.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:42.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:42.714208+0000) 2024-03-06T12:08:43.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:43.179 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:43.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:43.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:43.714404+0000) 2024-03-06T12:08:44.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:44.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:44.714707+0000) 2024-03-06T12:08:45.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:45.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:45.714938+0000) 2024-03-06T12:08:46.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:46.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:46.715176+0000) 2024-03-06T12:08:47.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:47.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:47.715437+0000) 2024-03-06T12:08:48.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:48.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:48.715752+0000) 2024-03-06T12:08:49.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:49.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:49.715962+0000) 2024-03-06T12:08:50.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:50.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:50.716207+0000) 2024-03-06T12:08:51.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:51.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:51.716470+0000) 2024-03-06T12:08:52.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:52.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:52.716779+0000) 2024-03-06T12:08:53.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:53.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:53.717036+0000) 2024-03-06T12:08:54.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:54.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:54.717297+0000) 2024-03-06T12:08:55.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:55.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:55.717537+0000) 2024-03-06T12:08:56.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:56.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:56.717789+0000) 2024-03-06T12:08:57.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:57.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:57.718033+0000) 2024-03-06T12:08:58.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:58.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:58.718281+0000) 2024-03-06T12:08:59.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:08:59.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:59.718521+0000) 2024-03-06T12:09:00.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:00.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:00.718780+0000) 2024-03-06T12:09:01.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:01.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:01.719021+0000) 2024-03-06T12:09:02.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:02.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:02.719279+0000) 2024-03-06T12:09:03.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:03.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:03.719544+0000) 2024-03-06T12:09:04.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:04.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:04.719806+0000) 2024-03-06T12:09:05.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:05.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:05.720064+0000) 2024-03-06T12:09:06.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:06.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:06.720322+0000) 2024-03-06T12:09:07.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:07.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:07.720516+0000) 2024-03-06T12:09:08.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:08.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:08.720780+0000) 2024-03-06T12:09:09.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:09.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:09.720994+0000) 2024-03-06T12:09:10.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:10.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:10.721283+0000) 2024-03-06T12:09:11.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:11.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:11.721561+0000) 2024-03-06T12:09:12.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:12.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:12.721862+0000) 2024-03-06T12:09:13.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:13.335 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:13.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:13.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:13.722151+0000) 2024-03-06T12:09:14.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:14.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:14.722436+0000) 2024-03-06T12:09:15.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:15.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:15.722746+0000) 2024-03-06T12:09:16.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:16.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:16.723000+0000) 2024-03-06T12:09:17.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:17.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:17.723267+0000) 2024-03-06T12:09:18.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:18.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:18.723541+0000) 2024-03-06T12:09:19.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:19.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:19.723818+0000) 2024-03-06T12:09:20.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:20.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:20.724044+0000) 2024-03-06T12:09:21.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:21.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:21.724315+0000) 2024-03-06T12:09:22.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:22.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:22.724577+0000) 2024-03-06T12:09:23.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:23.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:23.724858+0000) 2024-03-06T12:09:24.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:24.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:24.725143+0000) 2024-03-06T12:09:25.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:25.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:25.725409+0000) 2024-03-06T12:09:26.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:26.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:26.725727+0000) 2024-03-06T12:09:27.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:27.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:27.726002+0000) 2024-03-06T12:09:28.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:28.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:28.726283+0000) 2024-03-06T12:09:29.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:29.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:29.726566+0000) 2024-03-06T12:09:30.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:30.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:30.726819+0000) 2024-03-06T12:09:31.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:31.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:31.727096+0000) 2024-03-06T12:09:32.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:32.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:32.727310+0000) 2024-03-06T12:09:33.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:33.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:33.727570+0000) 2024-03-06T12:09:34.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:34.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:34.727870+0000) 2024-03-06T12:09:35.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:35.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:35.728133+0000) 2024-03-06T12:09:36.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:36.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:36.728397+0000) 2024-03-06T12:09:37.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:37.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:37.728732+0000) 2024-03-06T12:09:38.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:38.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:38.729009+0000) 2024-03-06T12:09:39.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:39.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:39.729292+0000) 2024-03-06T12:09:40.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:40.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:40.729567+0000) 2024-03-06T12:09:41.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:41.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:41.729835+0000) 2024-03-06T12:09:42.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:42.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:42.730119+0000) 2024-03-06T12:09:43.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:43.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:43.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:43.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:43.730343+0000) 2024-03-06T12:09:44.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:44.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:44.730622+0000) 2024-03-06T12:09:45.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:45.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:45.730948+0000) 2024-03-06T12:09:46.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:46.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:46.731220+0000) 2024-03-06T12:09:47.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:47.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:47.731484+0000) 2024-03-06T12:09:48.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:48.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:48.731779+0000) 2024-03-06T12:09:49.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:49.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:49.732030+0000) 2024-03-06T12:09:50.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:50.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:50.732304+0000) 2024-03-06T12:09:51.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:51.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:51.732572+0000) 2024-03-06T12:09:52.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:52.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:52.732843+0000) 2024-03-06T12:09:53.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:53.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:53.733124+0000) 2024-03-06T12:09:54.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:54.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:54.733390+0000) 2024-03-06T12:09:55.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:55.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:55.733674+0000) 2024-03-06T12:09:56.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:56.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:56.733944+0000) 2024-03-06T12:09:57.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:57.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:57.734226+0000) 2024-03-06T12:09:58.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:58.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:58.734516+0000) 2024-03-06T12:09:59.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:09:59.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:59.734780+0000) 2024-03-06T12:10:00.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:00.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:00.735068+0000) 2024-03-06T12:10:01.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:01.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:01.735354+0000) 2024-03-06T12:10:02.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:02.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:02.735672+0000) 2024-03-06T12:10:03.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:03.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:03.735963+0000) 2024-03-06T12:10:04.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:04.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:04.736245+0000) 2024-03-06T12:10:05.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:05.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:05.736522+0000) 2024-03-06T12:10:06.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:06.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:06.736804+0000) 2024-03-06T12:10:07.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:07.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:07.737078+0000) 2024-03-06T12:10:08.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:08.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:08.737333+0000) 2024-03-06T12:10:09.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:09.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:09.737571+0000) 2024-03-06T12:10:10.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:10.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:10.737843+0000) 2024-03-06T12:10:11.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:11.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:11.738121+0000) 2024-03-06T12:10:12.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:12.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:12.738406+0000) 2024-03-06T12:10:13.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:13.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:13.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:13.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:13.738615+0000) 2024-03-06T12:10:14.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:14.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:14.738909+0000) 2024-03-06T12:10:15.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:15.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:15.739194+0000) 2024-03-06T12:10:16.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:16.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:16.739476+0000) 2024-03-06T12:10:17.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:17.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:17.739754+0000) 2024-03-06T12:10:18.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:18.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:18.740045+0000) 2024-03-06T12:10:19.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:19.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:19.740335+0000) 2024-03-06T12:10:20.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:20.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:20.740615+0000) 2024-03-06T12:10:21.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:21.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:21.740930+0000) 2024-03-06T12:10:22.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:22.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:22.741187+0000) 2024-03-06T12:10:23.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:23.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:23.741445+0000) 2024-03-06T12:10:24.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:24.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:24.741725+0000) 2024-03-06T12:10:25.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:25.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:25.741967+0000) 2024-03-06T12:10:26.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:26.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:26.742221+0000) 2024-03-06T12:10:27.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:27.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:27.742488+0000) 2024-03-06T12:10:28.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:28.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:28.742800+0000) 2024-03-06T12:10:29.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:29.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:29.743066+0000) 2024-03-06T12:10:30.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:30.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:30.743335+0000) 2024-03-06T12:10:31.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:31.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:31.743580+0000) 2024-03-06T12:10:32.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:32.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:32.743850+0000) 2024-03-06T12:10:33.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:33.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:33.744113+0000) 2024-03-06T12:10:34.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:34.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:34.744371+0000) 2024-03-06T12:10:35.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:35.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:35.744665+0000) 2024-03-06T12:10:36.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:36.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:36.744919+0000) 2024-03-06T12:10:37.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:37.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:37.745186+0000) 2024-03-06T12:10:38.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:38.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:38.745445+0000) 2024-03-06T12:10:39.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:39.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:39.745771+0000) 2024-03-06T12:10:40.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:40.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:40.746040+0000) 2024-03-06T12:10:41.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:41.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:41.746281+0000) 2024-03-06T12:10:42.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:42.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:42.746529+0000) 2024-03-06T12:10:43.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:43.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:43.746776+0000) 2024-03-06T12:10:43.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:43.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:44.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:44.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:44.746992+0000) 2024-03-06T12:10:45.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:45.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:45.747181+0000) 2024-03-06T12:10:46.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:46.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:46.747444+0000) 2024-03-06T12:10:47.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:47.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:47.747749+0000) 2024-03-06T12:10:48.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:48.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:48.748028+0000) 2024-03-06T12:10:49.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:49.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:49.748282+0000) 2024-03-06T12:10:50.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:50.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:50.748525+0000) 2024-03-06T12:10:51.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:51.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:51.748784+0000) 2024-03-06T12:10:52.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:52.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:52.748989+0000) 2024-03-06T12:10:53.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:53.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:53.749235+0000) 2024-03-06T12:10:54.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:54.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:54.749485+0000) 2024-03-06T12:10:55.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:55.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:55.749752+0000) 2024-03-06T12:10:56.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:56.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:56.749999+0000) 2024-03-06T12:10:57.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:57.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:57.750205+0000) 2024-03-06T12:10:58.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:58.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:58.750480+0000) 2024-03-06T12:10:59.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:10:59.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:59.750752+0000) 2024-03-06T12:11:00.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:00.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:00.750999+0000) 2024-03-06T12:11:01.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:01.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:01.751260+0000) 2024-03-06T12:11:02.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:02.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:02.751527+0000) 2024-03-06T12:11:03.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:03.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:03.751769+0000) 2024-03-06T12:11:04.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:04.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:04.752050+0000) 2024-03-06T12:11:05.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:05.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:05.752295+0000) 2024-03-06T12:11:06.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:06.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:06.752555+0000) 2024-03-06T12:11:07.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:07.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:07.752826+0000) 2024-03-06T12:11:08.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:08.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:08.753107+0000) 2024-03-06T12:11:09.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:09.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:09.753376+0000) 2024-03-06T12:11:10.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:10.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:10.753690+0000) 2024-03-06T12:11:11.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:11.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:11.753997+0000) 2024-03-06T12:11:12.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:12.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:12.754269+0000) 2024-03-06T12:11:13.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:13.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:13.754513+0000) 2024-03-06T12:11:13.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:13.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:14.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:14.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:14.754783+0000) 2024-03-06T12:11:15.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:15.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:15.755025+0000) 2024-03-06T12:11:16.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:16.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:16.755263+0000) 2024-03-06T12:11:17.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:17.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:17.755518+0000) 2024-03-06T12:11:18.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:18.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:18.755780+0000) 2024-03-06T12:11:19.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:19.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:19.756037+0000) 2024-03-06T12:11:20.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:20.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:20.756294+0000) 2024-03-06T12:11:21.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:21.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:21.756556+0000) 2024-03-06T12:11:22.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:22.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:22.756822+0000) 2024-03-06T12:11:23.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:23.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:23.757085+0000) 2024-03-06T12:11:24.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:24.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:24.757369+0000) 2024-03-06T12:11:25.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:25.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:25.757663+0000) 2024-03-06T12:11:26.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:26.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:26.757929+0000) 2024-03-06T12:11:27.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:27.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:27.758186+0000) 2024-03-06T12:11:28.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:28.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:28.758464+0000) 2024-03-06T12:11:29.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:29.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:29.758764+0000) 2024-03-06T12:11:30.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:30.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:30.759032+0000) 2024-03-06T12:11:31.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:31.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:31.759273+0000) 2024-03-06T12:11:32.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:32.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:32.759541+0000) 2024-03-06T12:11:33.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:33.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:33.759810+0000) 2024-03-06T12:11:34.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:34.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:34.760085+0000) 2024-03-06T12:11:35.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:35.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:35.760335+0000) 2024-03-06T12:11:36.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:36.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:36.760593+0000) 2024-03-06T12:11:37.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:37.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:37.760872+0000) 2024-03-06T12:11:38.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:38.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:38.761151+0000) 2024-03-06T12:11:39.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:39.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:39.761425+0000) 2024-03-06T12:11:40.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:40.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:40.761743+0000) 2024-03-06T12:11:41.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:41.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:41.761992+0000) 2024-03-06T12:11:42.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:42.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:42.762260+0000) 2024-03-06T12:11:43.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:43.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:43.762531+0000) 2024-03-06T12:11:44.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:44.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:44.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:44.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:44.762805+0000) 2024-03-06T12:11:45.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:45.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:45.763053+0000) 2024-03-06T12:11:46.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:46.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:46.763240+0000) 2024-03-06T12:11:47.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:47.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:47.763496+0000) 2024-03-06T12:11:48.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:48.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:48.763770+0000) 2024-03-06T12:11:49.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:49.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:49.764040+0000) 2024-03-06T12:11:50.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:50.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:50.764270+0000) 2024-03-06T12:11:51.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:51.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:51.764529+0000) 2024-03-06T12:11:52.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:52.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:52.764800+0000) 2024-03-06T12:11:53.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:53.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:53.765064+0000) 2024-03-06T12:11:54.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:54.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:54.765332+0000) 2024-03-06T12:11:55.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:55.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:55.765608+0000) 2024-03-06T12:11:56.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:56.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:56.765936+0000) 2024-03-06T12:11:57.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:57.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:57.766217+0000) 2024-03-06T12:11:58.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:58.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:58.766510+0000) 2024-03-06T12:11:59.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:11:59.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:59.766779+0000) 2024-03-06T12:12:00.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:00.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:00.767072+0000) 2024-03-06T12:12:01.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:01.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:01.767358+0000) 2024-03-06T12:12:02.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:02.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:02.767621+0000) 2024-03-06T12:12:03.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:03.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:03.767946+0000) 2024-03-06T12:12:04.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:04.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:04.768267+0000) 2024-03-06T12:12:05.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:05.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:05.768516+0000) 2024-03-06T12:12:06.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:06.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:06.768775+0000) 2024-03-06T12:12:07.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:07.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:07.769058+0000) 2024-03-06T12:12:08.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:08.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:08.769333+0000) 2024-03-06T12:12:09.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:09.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:09.769611+0000) 2024-03-06T12:12:10.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:10.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:10.769934+0000) 2024-03-06T12:12:11.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:11.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:11.770216+0000) 2024-03-06T12:12:12.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:12.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:12.770479+0000) 2024-03-06T12:12:13.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:13.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:13.770743+0000) 2024-03-06T12:12:14.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:14.260 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:14.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:14.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:14.771037+0000) 2024-03-06T12:12:15.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:15.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:15.771312+0000) 2024-03-06T12:12:16.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:16.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:16.771587+0000) 2024-03-06T12:12:17.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:17.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:17.771881+0000) 2024-03-06T12:12:18.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:18.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:18.772169+0000) 2024-03-06T12:12:19.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:19.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:19.772438+0000) 2024-03-06T12:12:20.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:20.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:20.772739+0000) 2024-03-06T12:12:21.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:21.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:21.773018+0000) 2024-03-06T12:12:22.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:22.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:22.773310+0000) 2024-03-06T12:12:23.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:23.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:23.773590+0000) 2024-03-06T12:12:24.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:24.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:24.773848+0000) 2024-03-06T12:12:25.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:25.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:25.774110+0000) 2024-03-06T12:12:26.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:26.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:26.774383+0000) 2024-03-06T12:12:27.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:27.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:27.774695+0000) 2024-03-06T12:12:28.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:28.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:28.774985+0000) 2024-03-06T12:12:29.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:29.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:29.775264+0000) 2024-03-06T12:12:30.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:30.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:30.775549+0000) 2024-03-06T12:12:31.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:31.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:31.775830+0000) 2024-03-06T12:12:32.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:32.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:32.776088+0000) 2024-03-06T12:12:33.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:33.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:33.776371+0000) 2024-03-06T12:12:34.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:34.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:34.776677+0000) 2024-03-06T12:12:35.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:35.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:35.776948+0000) 2024-03-06T12:12:36.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:36.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:36.777232+0000) 2024-03-06T12:12:37.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:37.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:37.777492+0000) 2024-03-06T12:12:38.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:38.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:38.777762+0000) 2024-03-06T12:12:39.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:39.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:39.778022+0000) 2024-03-06T12:12:40.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:40.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:40.778305+0000) 2024-03-06T12:12:41.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:41.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:41.778580+0000) 2024-03-06T12:12:42.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:42.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:42.778799+0000) 2024-03-06T12:12:43.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:43.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:43.779009+0000) 2024-03-06T12:12:44.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:44.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:44.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:44.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:44.779223+0000) 2024-03-06T12:12:45.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:45.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:45.779428+0000) 2024-03-06T12:12:46.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:46.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:46.779737+0000) 2024-03-06T12:12:47.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:47.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:47.780016+0000) 2024-03-06T12:12:48.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:48.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:48.780249+0000) 2024-03-06T12:12:49.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:49.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:49.780524+0000) 2024-03-06T12:12:50.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:50.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:50.780814+0000) 2024-03-06T12:12:51.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:51.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:51.781095+0000) 2024-03-06T12:12:52.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:52.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:52.781417+0000) 2024-03-06T12:12:53.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:53.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:53.781737+0000) 2024-03-06T12:12:54.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:54.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:54.782028+0000) 2024-03-06T12:12:55.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:55.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:55.782296+0000) 2024-03-06T12:12:56.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:56.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:56.782565+0000) 2024-03-06T12:12:57.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:57.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:57.782816+0000) 2024-03-06T12:12:58.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:58.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:58.783099+0000) 2024-03-06T12:12:59.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:12:59.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:59.783307+0000) 2024-03-06T12:13:00.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:00.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:00.783578+0000) 2024-03-06T12:13:01.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:01.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:01.783855+0000) 2024-03-06T12:13:02.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:02.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:02.784112+0000) 2024-03-06T12:13:03.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:03.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:03.784400+0000) 2024-03-06T12:13:04.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:04.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:04.784731+0000) 2024-03-06T12:13:05.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:05.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:05.784987+0000) 2024-03-06T12:13:06.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:06.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:06.785267+0000) 2024-03-06T12:13:07.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:07.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:07.785521+0000) 2024-03-06T12:13:08.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:08.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:08.785799+0000) 2024-03-06T12:13:09.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:09.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:09.786050+0000) 2024-03-06T12:13:10.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:10.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:10.786320+0000) 2024-03-06T12:13:11.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:11.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:11.786548+0000) 2024-03-06T12:13:12.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:12.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:12.786821+0000) 2024-03-06T12:13:13.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:13.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:13.787041+0000) 2024-03-06T12:13:14.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:14.553 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:14.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:14.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:14.787326+0000) 2024-03-06T12:13:15.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:15.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:15.787584+0000) 2024-03-06T12:13:16.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:16.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:16.787900+0000) 2024-03-06T12:13:17.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:17.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:17.788202+0000) 2024-03-06T12:13:18.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:18.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:18.788525+0000) 2024-03-06T12:13:19.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:19.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:19.788773+0000) 2024-03-06T12:13:20.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:20.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:20.789094+0000) 2024-03-06T12:13:21.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:21.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:21.789406+0000) 2024-03-06T12:13:22.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:22.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:22.789746+0000) 2024-03-06T12:13:23.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:23.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:23.790077+0000) 2024-03-06T12:13:24.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:24.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:24.790373+0000) 2024-03-06T12:13:25.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:25.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:25.790741+0000) 2024-03-06T12:13:26.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:26.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:26.790991+0000) 2024-03-06T12:13:27.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:27.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:27.791292+0000) 2024-03-06T12:13:28.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:28.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:28.791675+0000) 2024-03-06T12:13:29.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:29.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:29.791972+0000) 2024-03-06T12:13:30.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:30.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:30.792285+0000) 2024-03-06T12:13:31.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:31.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:31.792587+0000) 2024-03-06T12:13:32.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:32.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:32.792952+0000) 2024-03-06T12:13:33.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:33.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:33.793249+0000) 2024-03-06T12:13:34.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:34.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:34.793555+0000) 2024-03-06T12:13:35.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:35.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:35.793861+0000) 2024-03-06T12:13:36.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:36.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:36.794160+0000) 2024-03-06T12:13:37.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:37.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:37.794470+0000) 2024-03-06T12:13:38.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:38.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:38.794796+0000) 2024-03-06T12:13:39.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:39.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:39.795072+0000) 2024-03-06T12:13:40.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:40.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:40.795377+0000) 2024-03-06T12:13:41.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:41.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:41.795703+0000) 2024-03-06T12:13:42.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:42.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:42.796032+0000) 2024-03-06T12:13:43.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:43.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:43.796349+0000) 2024-03-06T12:13:44.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:44.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:44.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:44.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:44.796581+0000) 2024-03-06T12:13:45.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:45.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:45.796846+0000) 2024-03-06T12:13:46.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:46.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:46.797138+0000) 2024-03-06T12:13:47.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:47.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:47.797447+0000) 2024-03-06T12:13:48.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:48.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:48.797797+0000) 2024-03-06T12:13:49.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:49.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:49.798115+0000) 2024-03-06T12:13:50.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:50.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:50.798440+0000) 2024-03-06T12:13:51.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:51.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:51.798841+0000) 2024-03-06T12:13:52.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:52.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:52.799160+0000) 2024-03-06T12:13:53.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:53.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:53.799481+0000) 2024-03-06T12:13:54.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:54.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:54.799792+0000) 2024-03-06T12:13:55.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:55.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:55.800104+0000) 2024-03-06T12:13:56.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:56.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:56.800421+0000) 2024-03-06T12:13:57.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:57.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:57.800760+0000) 2024-03-06T12:13:58.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:58.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:58.801039+0000) 2024-03-06T12:13:59.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:13:59.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:59.801287+0000) 2024-03-06T12:14:00.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:00.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:00.801570+0000) 2024-03-06T12:14:01.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:01.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:01.801883+0000) 2024-03-06T12:14:02.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:02.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:02.802133+0000) 2024-03-06T12:14:03.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:03.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:03.802423+0000) 2024-03-06T12:14:04.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:04.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:04.802769+0000) 2024-03-06T12:14:05.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:05.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:05.803077+0000) 2024-03-06T12:14:06.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:06.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:06.803390+0000) 2024-03-06T12:14:07.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:07.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:07.803744+0000) 2024-03-06T12:14:08.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:08.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:08.804016+0000) 2024-03-06T12:14:09.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:09.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:09.804324+0000) 2024-03-06T12:14:10.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:10.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:10.804625+0000) 2024-03-06T12:14:11.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:11.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:11.804970+0000) 2024-03-06T12:14:12.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:12.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:12.805258+0000) 2024-03-06T12:14:13.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:13.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:13.805556+0000) 2024-03-06T12:14:14.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:14.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:14.805869+0000) 2024-03-06T12:14:14.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:14.865 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:15.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:15.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:15.806194+0000) 2024-03-06T12:14:16.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:16.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:16.806503+0000) 2024-03-06T12:14:17.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:17.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:17.806788+0000) 2024-03-06T12:14:18.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:18.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:18.807127+0000) 2024-03-06T12:14:19.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:19.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:19.807435+0000) 2024-03-06T12:14:20.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:20.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:20.807770+0000) 2024-03-06T12:14:21.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:21.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:21.808049+0000) 2024-03-06T12:14:22.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:22.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:22.808284+0000) 2024-03-06T12:14:23.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:23.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:23.808591+0000) 2024-03-06T12:14:24.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:24.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:24.808953+0000) 2024-03-06T12:14:25.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:25.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:25.809264+0000) 2024-03-06T12:14:26.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:26.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:26.809565+0000) 2024-03-06T12:14:27.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:27.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:27.809868+0000) 2024-03-06T12:14:28.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:28.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:28.810192+0000) 2024-03-06T12:14:29.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:29.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:29.810504+0000) 2024-03-06T12:14:30.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:30.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:30.810824+0000) 2024-03-06T12:14:31.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:31.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:31.811136+0000) 2024-03-06T12:14:32.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:32.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:32.811461+0000) 2024-03-06T12:14:33.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:33.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:33.811812+0000) 2024-03-06T12:14:34.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:34.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:34.812128+0000) 2024-03-06T12:14:35.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:35.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:35.812432+0000) 2024-03-06T12:14:36.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:36.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:36.812767+0000) 2024-03-06T12:14:37.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:37.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:37.813085+0000) 2024-03-06T12:14:38.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:38.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:38.813409+0000) 2024-03-06T12:14:39.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:39.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:39.813822+0000) 2024-03-06T12:14:40.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:40.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:40.814084+0000) 2024-03-06T12:14:41.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:41.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:41.814332+0000) 2024-03-06T12:14:42.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:42.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:42.814689+0000) 2024-03-06T12:14:43.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:43.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:43.815023+0000) 2024-03-06T12:14:44.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:44.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:44.815332+0000) 2024-03-06T12:14:45.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:45.010 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:45.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:45.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:45.815672+0000) 2024-03-06T12:14:46.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:46.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:46.815985+0000) 2024-03-06T12:14:47.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:47.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:47.816295+0000) 2024-03-06T12:14:48.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:48.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:48.816590+0000) 2024-03-06T12:14:49.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:49.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:49.816951+0000) 2024-03-06T12:14:50.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:50.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:50.817274+0000) 2024-03-06T12:14:51.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:51.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:51.817589+0000) 2024-03-06T12:14:52.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:52.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:52.817947+0000) 2024-03-06T12:14:53.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:53.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:53.818248+0000) 2024-03-06T12:14:54.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:54.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:54.818564+0000) 2024-03-06T12:14:55.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:55.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:55.818831+0000) 2024-03-06T12:14:56.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:56.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:56.819090+0000) 2024-03-06T12:14:57.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:57.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:57.819346+0000) 2024-03-06T12:14:58.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:58.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:58.819680+0000) 2024-03-06T12:14:59.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:14:59.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:59.819929+0000) 2024-03-06T12:15:00.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:00.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:00.820244+0000) 2024-03-06T12:15:01.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:01.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:01.820544+0000) 2024-03-06T12:15:02.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:02.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:02.820865+0000) 2024-03-06T12:15:03.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:03.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:03.821180+0000) 2024-03-06T12:15:04.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:04.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:04.821503+0000) 2024-03-06T12:15:05.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:05.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:05.821817+0000) 2024-03-06T12:15:06.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:06.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:06.822120+0000) 2024-03-06T12:15:07.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:07.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:07.822425+0000) 2024-03-06T12:15:08.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:08.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:08.822797+0000) 2024-03-06T12:15:09.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:09.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:09.823085+0000) 2024-03-06T12:15:10.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:10.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:10.823384+0000) 2024-03-06T12:15:11.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:11.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:11.823730+0000) 2024-03-06T12:15:12.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:12.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:12.824036+0000) 2024-03-06T12:15:13.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:13.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:13.824359+0000) 2024-03-06T12:15:14.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:14.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:14.824721+0000) 2024-03-06T12:15:15.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:15.165 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:15.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:15.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:15.825047+0000) 2024-03-06T12:15:16.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:16.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:16.825352+0000) 2024-03-06T12:15:17.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:17.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:17.825715+0000) 2024-03-06T12:15:18.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:18.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:18.826056+0000) 2024-03-06T12:15:19.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:19.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:19.826355+0000) 2024-03-06T12:15:20.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:20.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:20.826717+0000) 2024-03-06T12:15:21.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:21.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:21.827080+0000) 2024-03-06T12:15:22.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:22.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:22.827388+0000) 2024-03-06T12:15:23.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:23.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:23.827745+0000) 2024-03-06T12:15:24.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:24.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:24.828067+0000) 2024-03-06T12:15:25.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:25.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:25.828377+0000) 2024-03-06T12:15:26.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:26.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:26.828737+0000) 2024-03-06T12:15:27.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:27.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:27.829019+0000) 2024-03-06T12:15:28.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:28.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:28.829357+0000) 2024-03-06T12:15:29.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:29.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:29.829706+0000) 2024-03-06T12:15:30.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:30.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:30.830024+0000) 2024-03-06T12:15:31.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:31.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:31.830336+0000) 2024-03-06T12:15:32.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:32.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:32.830685+0000) 2024-03-06T12:15:33.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:33.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:33.830954+0000) 2024-03-06T12:15:34.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:34.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:34.831261+0000) 2024-03-06T12:15:35.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:35.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:35.831572+0000) 2024-03-06T12:15:36.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:36.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:36.831881+0000) 2024-03-06T12:15:37.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:37.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:37.832201+0000) 2024-03-06T12:15:38.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:38.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:38.832524+0000) 2024-03-06T12:15:39.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:39.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:39.832777+0000) 2024-03-06T12:15:40.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:40.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:40.832981+0000) 2024-03-06T12:15:41.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:41.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:41.833233+0000) 2024-03-06T12:15:42.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:42.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:42.833489+0000) 2024-03-06T12:15:43.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:43.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:43.833760+0000) 2024-03-06T12:15:44.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:44.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:44.834020+0000) 2024-03-06T12:15:45.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:45.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:45.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:45.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:45.834287+0000) 2024-03-06T12:15:46.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:46.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:46.834527+0000) 2024-03-06T12:15:47.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:47.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:47.834787+0000) 2024-03-06T12:15:48.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:48.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:48.835068+0000) 2024-03-06T12:15:49.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:49.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:49.835340+0000) 2024-03-06T12:15:50.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:50.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:50.835681+0000) 2024-03-06T12:15:51.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:51.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:51.836004+0000) 2024-03-06T12:15:52.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:52.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:52.836329+0000) 2024-03-06T12:15:53.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:53.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:53.836688+0000) 2024-03-06T12:15:54.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:54.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:54.837015+0000) 2024-03-06T12:15:55.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:55.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:55.837321+0000) 2024-03-06T12:15:56.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:56.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:56.837617+0000) 2024-03-06T12:15:57.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:57.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:57.837958+0000) 2024-03-06T12:15:58.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:58.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:58.838254+0000) 2024-03-06T12:15:59.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:15:59.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:59.838555+0000) 2024-03-06T12:16:00.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:00.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:00.838868+0000) 2024-03-06T12:16:01.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:01.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:01.839107+0000) 2024-03-06T12:16:02.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:02.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:02.839417+0000) 2024-03-06T12:16:03.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:03.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:03.839785+0000) 2024-03-06T12:16:04.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:04.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:04.840086+0000) 2024-03-06T12:16:05.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:05.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:05.840399+0000) 2024-03-06T12:16:06.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:06.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:06.840739+0000) 2024-03-06T12:16:07.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:07.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:07.841052+0000) 2024-03-06T12:16:08.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:08.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:08.841377+0000) 2024-03-06T12:16:09.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:09.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:09.841778+0000) 2024-03-06T12:16:10.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:10.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:10.842088+0000) 2024-03-06T12:16:11.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:11.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:11.842408+0000) 2024-03-06T12:16:12.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:12.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:12.842735+0000) 2024-03-06T12:16:13.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:13.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:13.843041+0000) 2024-03-06T12:16:14.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:14.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:14.843344+0000) 2024-03-06T12:16:15.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:15.486 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:15.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:15.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:15.843705+0000) 2024-03-06T12:16:16.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:16.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:16.844027+0000) 2024-03-06T12:16:17.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:17.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:17.844324+0000) 2024-03-06T12:16:18.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:18.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:18.844707+0000) 2024-03-06T12:16:19.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:19.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:19.845031+0000) 2024-03-06T12:16:20.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:20.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:20.845351+0000) 2024-03-06T12:16:21.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:21.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:21.845713+0000) 2024-03-06T12:16:22.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:22.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:22.846007+0000) 2024-03-06T12:16:23.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:23.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:23.846314+0000) 2024-03-06T12:16:24.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:24.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:24.846619+0000) 2024-03-06T12:16:25.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:25.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:25.846983+0000) 2024-03-06T12:16:26.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:26.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:26.847289+0000) 2024-03-06T12:16:27.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:27.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:27.847595+0000) 2024-03-06T12:16:28.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:28.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:28.847910+0000) 2024-03-06T12:16:29.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:29.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:29.848225+0000) 2024-03-06T12:16:30.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:30.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:30.848523+0000) 2024-03-06T12:16:31.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:31.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:31.848831+0000) 2024-03-06T12:16:32.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:32.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:32.849157+0000) 2024-03-06T12:16:33.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:33.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:33.849468+0000) 2024-03-06T12:16:34.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:34.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:34.849806+0000) 2024-03-06T12:16:35.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:35.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:35.850068+0000) 2024-03-06T12:16:36.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:36.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:36.850339+0000) 2024-03-06T12:16:37.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:37.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:37.850684+0000) 2024-03-06T12:16:38.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:38.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:38.851012+0000) 2024-03-06T12:16:39.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:39.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:39.851309+0000) 2024-03-06T12:16:40.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:40.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:40.851606+0000) 2024-03-06T12:16:41.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:41.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:41.851967+0000) 2024-03-06T12:16:42.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:42.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:42.852280+0000) 2024-03-06T12:16:43.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:43.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:43.852601+0000) 2024-03-06T12:16:44.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:44.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:44.852984+0000) 2024-03-06T12:16:45.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:45.635 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:45.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:45.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:45.853290+0000) 2024-03-06T12:16:46.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:46.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:46.853535+0000) 2024-03-06T12:16:47.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:47.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:47.853833+0000) 2024-03-06T12:16:48.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:48.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:48.854172+0000) 2024-03-06T12:16:49.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:49.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:49.854494+0000) 2024-03-06T12:16:50.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:50.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:50.854793+0000) 2024-03-06T12:16:51.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:51.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:51.855041+0000) 2024-03-06T12:16:52.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:52.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:52.855295+0000) 2024-03-06T12:16:53.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:53.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:53.855529+0000) 2024-03-06T12:16:54.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:54.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:54.855759+0000) 2024-03-06T12:16:55.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:55.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:55.856017+0000) 2024-03-06T12:16:56.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:56.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:56.856282+0000) 2024-03-06T12:16:57.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:57.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:57.856538+0000) 2024-03-06T12:16:58.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:58.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:58.856794+0000) 2024-03-06T12:16:59.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:16:59.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:59.857056+0000) 2024-03-06T12:17:00.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:00.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:00.857295+0000) 2024-03-06T12:17:01.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:01.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:01.857491+0000) 2024-03-06T12:17:02.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:02.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:02.857765+0000) 2024-03-06T12:17:03.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:03.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:03.858048+0000) 2024-03-06T12:17:04.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:04.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:04.858295+0000) 2024-03-06T12:17:05.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:05.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:05.858533+0000) 2024-03-06T12:17:06.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:06.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:06.858787+0000) 2024-03-06T12:17:07.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:07.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:07.859043+0000) 2024-03-06T12:17:08.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:08.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:08.859244+0000) 2024-03-06T12:17:09.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:09.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:09.859503+0000) 2024-03-06T12:17:10.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:10.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:10.859765+0000) 2024-03-06T12:17:11.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:11.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:11.860011+0000) 2024-03-06T12:17:12.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:12.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:12.860252+0000) 2024-03-06T12:17:13.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:13.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:13.860522+0000) 2024-03-06T12:17:14.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:14.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:14.860757+0000) 2024-03-06T12:17:15.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:15.777 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:15.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:15.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:15.861007+0000) 2024-03-06T12:17:16.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:16.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:16.861280+0000) 2024-03-06T12:17:17.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:17.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:17.861526+0000) 2024-03-06T12:17:18.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:18.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:18.861822+0000) 2024-03-06T12:17:19.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:19.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:19.862097+0000) 2024-03-06T12:17:20.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:20.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:20.862373+0000) 2024-03-06T12:17:21.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:21.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:21.862670+0000) 2024-03-06T12:17:22.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:22.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:22.862932+0000) 2024-03-06T12:17:23.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:23.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:23.863203+0000) 2024-03-06T12:17:24.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:24.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:24.863478+0000) 2024-03-06T12:17:25.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:25.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:25.863752+0000) 2024-03-06T12:17:26.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:26.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:26.864034+0000) 2024-03-06T12:17:27.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:27.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:27.864286+0000) 2024-03-06T12:17:28.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:28.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:28.864575+0000) 2024-03-06T12:17:29.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:29.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:29.864856+0000) 2024-03-06T12:17:30.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:30.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:30.865126+0000) 2024-03-06T12:17:31.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:31.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:31.865385+0000) 2024-03-06T12:17:32.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:32.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:32.865677+0000) 2024-03-06T12:17:33.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:33.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:33.865949+0000) 2024-03-06T12:17:34.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:34.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:34.866229+0000) 2024-03-06T12:17:35.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:35.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:35.866507+0000) 2024-03-06T12:17:36.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:36.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:36.866768+0000) 2024-03-06T12:17:37.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:37.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:37.867027+0000) 2024-03-06T12:17:38.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:38.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:38.867318+0000) 2024-03-06T12:17:39.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:39.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:39.867596+0000) 2024-03-06T12:17:40.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:40.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:40.867912+0000) 2024-03-06T12:17:41.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:41.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:41.868170+0000) 2024-03-06T12:17:42.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:42.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:42.868433+0000) 2024-03-06T12:17:43.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:43.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:43.868741+0000) 2024-03-06T12:17:44.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:44.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:44.869011+0000) 2024-03-06T12:17:45.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:45.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:45.869262+0000) 2024-03-06T12:17:45.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:45.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:46.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:46.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:46.869525+0000) 2024-03-06T12:17:47.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:47.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:47.869795+0000) 2024-03-06T12:17:48.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:48.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:48.870065+0000) 2024-03-06T12:17:49.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:49.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:49.870342+0000) 2024-03-06T12:17:50.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:50.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:50.870601+0000) 2024-03-06T12:17:51.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:51.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:51.870906+0000) 2024-03-06T12:17:52.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:52.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:52.871177+0000) 2024-03-06T12:17:53.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:53.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:53.871460+0000) 2024-03-06T12:17:54.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:54.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:54.871742+0000) 2024-03-06T12:17:55.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:55.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:55.872011+0000) 2024-03-06T12:17:56.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:56.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:56.872339+0000) 2024-03-06T12:17:57.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:57.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:57.872611+0000) 2024-03-06T12:17:58.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:58.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:58.872943+0000) 2024-03-06T12:17:59.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:17:59.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:59.873228+0000) 2024-03-06T12:18:00.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:00.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:00.873506+0000) 2024-03-06T12:18:01.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:01.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:01.873787+0000) 2024-03-06T12:18:02.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:02.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:02.874059+0000) 2024-03-06T12:18:03.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:03.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:03.874312+0000) 2024-03-06T12:18:04.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:04.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:04.874570+0000) 2024-03-06T12:18:05.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:05.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:05.874824+0000) 2024-03-06T12:18:06.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:06.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:06.875106+0000) 2024-03-06T12:18:07.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:07.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:07.875377+0000) 2024-03-06T12:18:08.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:08.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:08.875674+0000) 2024-03-06T12:18:09.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:09.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:09.875933+0000) 2024-03-06T12:18:10.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:10.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:10.876196+0000) 2024-03-06T12:18:11.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:11.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:11.876479+0000) 2024-03-06T12:18:12.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:12.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:12.876753+0000) 2024-03-06T12:18:13.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:13.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:13.877005+0000) 2024-03-06T12:18:14.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:14.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:14.877196+0000) 2024-03-06T12:18:15.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:15.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:15.877457+0000) 2024-03-06T12:18:16.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:16.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:16.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:16.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:16.877760+0000) 2024-03-06T12:18:17.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:17.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:17.878011+0000) 2024-03-06T12:18:18.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:18.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:18.878289+0000) 2024-03-06T12:18:19.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:19.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:19.878532+0000) 2024-03-06T12:18:20.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:20.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:20.878811+0000) 2024-03-06T12:18:21.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:21.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:21.879005+0000) 2024-03-06T12:18:22.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:22.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:22.879194+0000) 2024-03-06T12:18:23.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:23.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:23.879384+0000) 2024-03-06T12:18:24.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:24.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:24.879710+0000) 2024-03-06T12:18:25.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:25.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:25.879945+0000) 2024-03-06T12:18:26.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:26.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:26.880151+0000) 2024-03-06T12:18:27.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:27.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:27.880406+0000) 2024-03-06T12:18:28.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:28.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:28.880735+0000) 2024-03-06T12:18:29.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:29.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:29.880998+0000) 2024-03-06T12:18:30.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:30.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:30.881261+0000) 2024-03-06T12:18:31.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:31.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:31.881532+0000) 2024-03-06T12:18:32.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:32.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:32.881791+0000) 2024-03-06T12:18:33.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:33.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:33.882032+0000) 2024-03-06T12:18:34.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:34.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:34.882296+0000) 2024-03-06T12:18:35.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:35.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:35.882540+0000) 2024-03-06T12:18:36.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:36.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:36.882803+0000) 2024-03-06T12:18:37.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:37.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:37.883043+0000) 2024-03-06T12:18:38.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:38.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:38.883321+0000) 2024-03-06T12:18:39.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:39.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:39.883576+0000) 2024-03-06T12:18:40.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:40.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:40.883799+0000) 2024-03-06T12:18:41.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:41.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:41.884061+0000) 2024-03-06T12:18:42.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:42.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:42.884319+0000) 2024-03-06T12:18:43.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:43.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:43.884579+0000) 2024-03-06T12:18:44.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:44.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:44.884825+0000) 2024-03-06T12:18:45.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:45.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:45.885082+0000) 2024-03-06T12:18:46.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:46.216 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:46.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:46.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:46.885346+0000) 2024-03-06T12:18:47.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:47.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:47.885595+0000) 2024-03-06T12:18:48.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:48.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:48.885910+0000) 2024-03-06T12:18:49.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:49.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:49.886202+0000) 2024-03-06T12:18:50.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:50.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:50.886471+0000) 2024-03-06T12:18:51.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:51.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:51.886768+0000) 2024-03-06T12:18:52.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:52.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:52.887027+0000) 2024-03-06T12:18:53.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:53.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:53.887283+0000) 2024-03-06T12:18:54.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:54.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:54.887550+0000) 2024-03-06T12:18:55.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:55.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:55.887814+0000) 2024-03-06T12:18:56.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:56.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:56.888080+0000) 2024-03-06T12:18:57.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:57.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:57.888277+0000) 2024-03-06T12:18:58.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:58.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:58.888487+0000) 2024-03-06T12:18:59.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:18:59.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:59.888755+0000) 2024-03-06T12:19:00.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:00.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:00.889016+0000) 2024-03-06T12:19:01.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:01.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:01.889284+0000) 2024-03-06T12:19:02.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:02.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:02.889546+0000) 2024-03-06T12:19:03.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:03.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:03.889806+0000) 2024-03-06T12:19:04.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:04.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:04.890078+0000) 2024-03-06T12:19:05.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:05.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:05.890333+0000) 2024-03-06T12:19:06.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:06.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:06.890573+0000) 2024-03-06T12:19:07.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:07.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:07.890806+0000) 2024-03-06T12:19:08.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:08.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:08.891067+0000) 2024-03-06T12:19:09.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:09.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:09.891347+0000) 2024-03-06T12:19:10.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:10.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:10.891604+0000) 2024-03-06T12:19:11.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:11.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:11.891901+0000) 2024-03-06T12:19:12.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:12.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:12.892179+0000) 2024-03-06T12:19:13.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:13.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:13.892390+0000) 2024-03-06T12:19:14.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:14.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:14.892730+0000) 2024-03-06T12:19:15.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:15.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:15.892958+0000) 2024-03-06T12:19:16.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:16.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:16.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:16.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:16.893211+0000) 2024-03-06T12:19:17.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:17.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:17.893433+0000) 2024-03-06T12:19:18.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:18.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:18.893722+0000) 2024-03-06T12:19:19.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:19.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:19.893968+0000) 2024-03-06T12:19:20.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:20.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:20.894224+0000) 2024-03-06T12:19:21.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:21.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:21.894483+0000) 2024-03-06T12:19:22.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:22.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:22.894747+0000) 2024-03-06T12:19:23.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:23.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:23.895003+0000) 2024-03-06T12:19:24.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:24.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:24.895192+0000) 2024-03-06T12:19:25.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:25.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:25.895451+0000) 2024-03-06T12:19:26.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:26.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:26.895753+0000) 2024-03-06T12:19:27.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:27.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:27.895961+0000) 2024-03-06T12:19:28.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:28.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:28.896249+0000) 2024-03-06T12:19:29.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:29.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:29.896500+0000) 2024-03-06T12:19:30.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:30.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:30.896740+0000) 2024-03-06T12:19:31.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:31.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:31.896991+0000) 2024-03-06T12:19:32.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:32.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:32.897298+0000) 2024-03-06T12:19:33.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:33.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:33.897557+0000) 2024-03-06T12:19:34.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:34.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:34.897822+0000) 2024-03-06T12:19:35.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:35.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:35.898056+0000) 2024-03-06T12:19:36.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:36.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:36.898330+0000) 2024-03-06T12:19:37.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:37.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:37.898572+0000) 2024-03-06T12:19:38.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:38.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:38.898758+0000) 2024-03-06T12:19:39.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:39.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:39.899010+0000) 2024-03-06T12:19:40.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:40.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:40.899269+0000) 2024-03-06T12:19:41.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:41.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:41.899515+0000) 2024-03-06T12:19:42.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:42.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:42.899774+0000) 2024-03-06T12:19:43.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:43.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:43.900009+0000) 2024-03-06T12:19:44.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:44.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:44.900253+0000) 2024-03-06T12:19:45.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:45.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:45.900509+0000) 2024-03-06T12:19:46.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:46.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:46.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:46.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:46.900770+0000) 2024-03-06T12:19:47.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:47.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:47.901006+0000) 2024-03-06T12:19:48.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:48.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:48.901261+0000) 2024-03-06T12:19:49.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:49.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:49.901505+0000) 2024-03-06T12:19:50.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:50.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:50.901795+0000) 2024-03-06T12:19:51.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:51.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:51.902047+0000) 2024-03-06T12:19:52.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:52.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:52.902255+0000) 2024-03-06T12:19:53.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:53.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:53.902488+0000) 2024-03-06T12:19:54.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:54.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:54.902749+0000) 2024-03-06T12:19:55.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:55.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:55.902981+0000) 2024-03-06T12:19:56.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:56.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:56.903263+0000) 2024-03-06T12:19:57.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:57.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:57.903529+0000) 2024-03-06T12:19:58.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:58.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:58.903813+0000) 2024-03-06T12:19:59.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:19:59.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:59.904081+0000) 2024-03-06T12:20:00.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:00.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:00.904294+0000) 2024-03-06T12:20:01.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:01.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:01.904518+0000) 2024-03-06T12:20:02.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:02.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:02.904760+0000) 2024-03-06T12:20:03.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:03.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:03.905010+0000) 2024-03-06T12:20:04.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:04.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:04.905246+0000) 2024-03-06T12:20:05.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:05.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:05.905480+0000) 2024-03-06T12:20:06.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:06.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:06.905747+0000) 2024-03-06T12:20:07.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:07.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:07.905935+0000) 2024-03-06T12:20:08.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:08.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:08.906204+0000) 2024-03-06T12:20:09.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:09.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:09.906440+0000) 2024-03-06T12:20:10.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:10.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:10.906751+0000) 2024-03-06T12:20:11.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:11.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:11.906986+0000) 2024-03-06T12:20:12.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:12.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:12.907250+0000) 2024-03-06T12:20:13.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:13.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:13.907482+0000) 2024-03-06T12:20:14.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:14.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:14.907762+0000) 2024-03-06T12:20:15.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:15.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:15.907958+0000) 2024-03-06T12:20:16.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:16.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:16.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:16.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:16.908234+0000) 2024-03-06T12:20:17.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:17.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:17.908439+0000) 2024-03-06T12:20:18.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:18.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:18.908766+0000) 2024-03-06T12:20:19.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:19.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:19.909016+0000) 2024-03-06T12:20:20.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:20.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:20.909342+0000) 2024-03-06T12:20:21.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:21.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:21.909599+0000) 2024-03-06T12:20:22.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:22.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:22.909867+0000) 2024-03-06T12:20:23.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:23.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:23.910115+0000) 2024-03-06T12:20:24.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:24.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:24.910381+0000) 2024-03-06T12:20:25.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:25.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:25.910665+0000) 2024-03-06T12:20:26.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:26.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:26.910915+0000) 2024-03-06T12:20:27.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:27.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:27.911147+0000) 2024-03-06T12:20:28.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:28.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:28.911409+0000) 2024-03-06T12:20:29.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:29.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:29.911683+0000) 2024-03-06T12:20:30.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:30.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:30.911915+0000) 2024-03-06T12:20:31.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:31.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:31.912168+0000) 2024-03-06T12:20:32.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:32.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:32.912365+0000) 2024-03-06T12:20:33.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:33.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:33.912597+0000) 2024-03-06T12:20:34.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:34.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:34.912799+0000) 2024-03-06T12:20:35.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:35.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:35.913033+0000) 2024-03-06T12:20:36.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:36.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:36.913299+0000) 2024-03-06T12:20:37.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:37.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:37.913546+0000) 2024-03-06T12:20:38.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:38.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:38.913836+0000) 2024-03-06T12:20:39.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:39.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:39.914051+0000) 2024-03-06T12:20:40.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:40.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:40.914334+0000) 2024-03-06T12:20:41.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:41.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:41.914586+0000) 2024-03-06T12:20:42.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:42.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:42.914905+0000) 2024-03-06T12:20:43.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:43.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:43.915137+0000) 2024-03-06T12:20:44.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:44.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:44.915382+0000) 2024-03-06T12:20:45.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:45.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:45.915663+0000) 2024-03-06T12:20:46.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:46.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:46.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:46.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:46.915940+0000) 2024-03-06T12:20:47.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:47.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:47.916178+0000) 2024-03-06T12:20:48.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:48.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:48.916388+0000) 2024-03-06T12:20:49.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:49.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:49.916622+0000) 2024-03-06T12:20:50.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:50.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:50.916925+0000) 2024-03-06T12:20:51.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:51.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:51.917155+0000) 2024-03-06T12:20:52.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:52.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:52.917405+0000) 2024-03-06T12:20:53.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:53.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:53.917678+0000) 2024-03-06T12:20:54.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:54.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:54.917886+0000) 2024-03-06T12:20:55.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:55.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:55.918133+0000) 2024-03-06T12:20:56.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:56.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:56.918383+0000) 2024-03-06T12:20:57.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:57.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:57.918675+0000) 2024-03-06T12:20:58.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:58.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:58.918879+0000) 2024-03-06T12:20:59.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:20:59.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:59.919115+0000) 2024-03-06T12:21:00.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:00.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:00.919392+0000) 2024-03-06T12:21:01.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:01.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:01.919692+0000) 2024-03-06T12:21:02.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:02.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:02.919926+0000) 2024-03-06T12:21:03.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:03.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:03.920162+0000) 2024-03-06T12:21:04.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:04.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:04.920435+0000) 2024-03-06T12:21:05.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:05.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:05.920725+0000) 2024-03-06T12:21:06.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:06.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:06.920989+0000) 2024-03-06T12:21:07.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:07.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:07.921222+0000) 2024-03-06T12:21:08.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:08.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:08.921499+0000) 2024-03-06T12:21:09.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:09.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:09.921732+0000) 2024-03-06T12:21:10.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:10.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:10.921931+0000) 2024-03-06T12:21:11.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:11.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:11.922212+0000) 2024-03-06T12:21:12.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:12.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:12.922491+0000) 2024-03-06T12:21:13.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:13.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:13.922755+0000) 2024-03-06T12:21:14.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:14.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:14.922964+0000) 2024-03-06T12:21:15.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:15.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:15.923210+0000) 2024-03-06T12:21:16.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:16.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:16.923495+0000) 2024-03-06T12:21:16.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:16.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:17.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:17.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:17.923746+0000) 2024-03-06T12:21:18.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:18.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:18.923964+0000) 2024-03-06T12:21:19.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:19.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:19.924303+0000) 2024-03-06T12:21:20.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:20.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:20.924586+0000) 2024-03-06T12:21:21.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:21.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:21.924877+0000) 2024-03-06T12:21:22.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:22.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:22.925133+0000) 2024-03-06T12:21:23.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:23.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:23.925397+0000) 2024-03-06T12:21:24.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:24.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:24.925743+0000) 2024-03-06T12:21:25.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:25.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:25.926013+0000) 2024-03-06T12:21:26.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:26.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:26.926302+0000) 2024-03-06T12:21:27.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:27.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:27.926498+0000) 2024-03-06T12:21:28.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:28.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:28.926768+0000) 2024-03-06T12:21:29.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:29.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:29.927036+0000) 2024-03-06T12:21:30.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:30.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:30.927295+0000) 2024-03-06T12:21:31.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:31.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:31.927558+0000) 2024-03-06T12:21:32.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:32.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:32.927815+0000) 2024-03-06T12:21:33.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:33.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:33.928066+0000) 2024-03-06T12:21:34.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:34.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:34.928334+0000) 2024-03-06T12:21:35.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:35.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:35.928588+0000) 2024-03-06T12:21:36.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:36.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:36.928909+0000) 2024-03-06T12:21:37.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:37.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:37.929117+0000) 2024-03-06T12:21:38.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:38.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:38.929363+0000) 2024-03-06T12:21:39.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:39.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:39.929611+0000) 2024-03-06T12:21:40.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:40.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:40.929892+0000) 2024-03-06T12:21:41.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:41.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:41.930069+0000) 2024-03-06T12:21:42.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:42.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:42.930313+0000) 2024-03-06T12:21:43.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:43.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:43.930568+0000) 2024-03-06T12:21:44.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:44.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:44.930831+0000) 2024-03-06T12:21:45.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:45.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:45.931071+0000) 2024-03-06T12:21:46.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:46.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:46.931309+0000) 2024-03-06T12:21:47.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:47.091 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:47.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:47.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:47.931539+0000) 2024-03-06T12:21:48.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:48.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:48.931802+0000) 2024-03-06T12:21:49.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:49.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:49.932054+0000) 2024-03-06T12:21:50.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:50.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:50.932311+0000) 2024-03-06T12:21:51.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:51.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:51.932562+0000) 2024-03-06T12:21:52.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:52.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:52.932807+0000) 2024-03-06T12:21:53.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:53.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:53.933040+0000) 2024-03-06T12:21:54.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:54.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:54.933281+0000) 2024-03-06T12:21:55.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:55.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:55.933512+0000) 2024-03-06T12:21:56.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:56.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:56.933767+0000) 2024-03-06T12:21:57.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:57.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:57.934004+0000) 2024-03-06T12:21:58.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:58.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:58.934255+0000) 2024-03-06T12:21:59.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:21:59.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:59.934509+0000) 2024-03-06T12:22:00.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:00.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:00.934750+0000) 2024-03-06T12:22:01.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:01.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:01.935000+0000) 2024-03-06T12:22:02.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:02.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:02.935243+0000) 2024-03-06T12:22:03.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:03.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:03.935473+0000) 2024-03-06T12:22:04.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:04.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:04.935753+0000) 2024-03-06T12:22:05.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:05.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:05.935998+0000) 2024-03-06T12:22:06.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:06.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:06.936256+0000) 2024-03-06T12:22:07.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:07.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:07.936566+0000) 2024-03-06T12:22:08.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:08.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:08.936838+0000) 2024-03-06T12:22:09.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:09.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:09.937027+0000) 2024-03-06T12:22:10.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:10.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:10.937289+0000) 2024-03-06T12:22:11.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:11.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:11.937539+0000) 2024-03-06T12:22:12.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:12.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:12.937899+0000) 2024-03-06T12:22:13.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:13.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:13.938172+0000) 2024-03-06T12:22:14.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:14.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:14.938410+0000) 2024-03-06T12:22:15.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:15.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:15.938686+0000) 2024-03-06T12:22:16.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:16.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:16.938951+0000) 2024-03-06T12:22:17.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:17.250 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:17.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:17.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:17.939202+0000) 2024-03-06T12:22:18.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:18.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:18.939470+0000) 2024-03-06T12:22:19.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:19.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:19.939756+0000) 2024-03-06T12:22:20.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:20.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:20.939995+0000) 2024-03-06T12:22:21.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:21.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:21.940233+0000) 2024-03-06T12:22:22.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:22.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:22.940425+0000) 2024-03-06T12:22:23.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:23.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:23.940714+0000) 2024-03-06T12:22:24.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:24.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:24.940979+0000) 2024-03-06T12:22:25.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:25.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:25.941229+0000) 2024-03-06T12:22:26.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:26.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:26.941471+0000) 2024-03-06T12:22:27.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:27.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:27.941755+0000) 2024-03-06T12:22:28.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:28.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:28.942016+0000) 2024-03-06T12:22:29.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:29.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:29.942268+0000) 2024-03-06T12:22:30.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:30.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:30.942511+0000) 2024-03-06T12:22:31.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:31.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:31.942746+0000) 2024-03-06T12:22:32.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:32.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:32.943003+0000) 2024-03-06T12:22:33.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:33.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:33.943248+0000) 2024-03-06T12:22:34.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:34.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:34.943507+0000) 2024-03-06T12:22:35.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:35.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:35.943758+0000) 2024-03-06T12:22:36.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:36.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:36.944107+0000) 2024-03-06T12:22:37.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:37.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:37.944374+0000) 2024-03-06T12:22:38.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:38.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:38.944729+0000) 2024-03-06T12:22:39.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:39.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:39.945000+0000) 2024-03-06T12:22:40.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:40.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:40.945261+0000) 2024-03-06T12:22:41.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:41.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:41.945519+0000) 2024-03-06T12:22:42.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:42.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:42.945766+0000) 2024-03-06T12:22:43.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:43.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:43.946003+0000) 2024-03-06T12:22:44.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:44.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:44.946264+0000) 2024-03-06T12:22:45.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:45.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:45.946497+0000) 2024-03-06T12:22:46.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:46.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:46.946760+0000) 2024-03-06T12:22:47.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:47.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:47.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:47.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:47.947010+0000) 2024-03-06T12:22:48.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:48.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:48.947223+0000) 2024-03-06T12:22:49.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:49.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:49.947482+0000) 2024-03-06T12:22:50.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:50.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:50.947744+0000) 2024-03-06T12:22:51.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:51.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:51.947984+0000) 2024-03-06T12:22:52.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:52.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:52.948237+0000) 2024-03-06T12:22:53.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:53.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:53.948472+0000) 2024-03-06T12:22:54.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:54.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:54.948746+0000) 2024-03-06T12:22:55.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:55.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:55.949055+0000) 2024-03-06T12:22:56.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:56.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:56.949323+0000) 2024-03-06T12:22:57.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:57.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:57.949573+0000) 2024-03-06T12:22:58.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:58.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:58.949853+0000) 2024-03-06T12:22:59.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:22:59.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:59.950088+0000) 2024-03-06T12:23:00.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:00.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:00.950338+0000) 2024-03-06T12:23:01.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:01.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:01.950581+0000) 2024-03-06T12:23:02.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:02.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:02.950849+0000) 2024-03-06T12:23:03.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:03.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:03.951101+0000) 2024-03-06T12:23:04.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:04.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:04.951353+0000) 2024-03-06T12:23:05.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:05.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:05.951600+0000) 2024-03-06T12:23:06.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:06.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:06.951839+0000) 2024-03-06T12:23:07.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:07.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:07.952101+0000) 2024-03-06T12:23:08.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:08.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:08.952372+0000) 2024-03-06T12:23:09.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:09.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:09.952607+0000) 2024-03-06T12:23:10.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:10.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:10.952892+0000) 2024-03-06T12:23:11.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:11.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:11.953148+0000) 2024-03-06T12:23:12.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:12.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:12.953393+0000) 2024-03-06T12:23:13.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:13.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:13.953727+0000) 2024-03-06T12:23:14.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:14.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:14.953990+0000) 2024-03-06T12:23:15.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:15.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:15.954246+0000) 2024-03-06T12:23:16.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:16.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:16.954507+0000) 2024-03-06T12:23:17.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:17.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:17.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:17.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:17.954783+0000) 2024-03-06T12:23:18.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:18.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:18.955047+0000) 2024-03-06T12:23:19.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:19.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:19.955280+0000) 2024-03-06T12:23:20.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:20.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:20.955514+0000) 2024-03-06T12:23:21.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:21.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:21.955771+0000) 2024-03-06T12:23:22.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:22.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:22.955983+0000) 2024-03-06T12:23:23.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:23.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:23.956208+0000) 2024-03-06T12:23:24.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:24.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:24.956474+0000) 2024-03-06T12:23:25.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:25.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:25.956741+0000) 2024-03-06T12:23:26.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:26.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:26.956998+0000) 2024-03-06T12:23:27.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:27.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:27.957238+0000) 2024-03-06T12:23:28.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:28.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:28.957494+0000) 2024-03-06T12:23:29.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:29.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:29.957725+0000) 2024-03-06T12:23:30.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:30.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:30.957977+0000) 2024-03-06T12:23:31.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:31.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:31.958219+0000) 2024-03-06T12:23:32.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:32.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:32.958508+0000) 2024-03-06T12:23:33.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:33.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:33.958762+0000) 2024-03-06T12:23:34.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:34.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:34.959037+0000) 2024-03-06T12:23:35.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:35.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:35.959316+0000) 2024-03-06T12:23:36.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:36.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:36.959577+0000) 2024-03-06T12:23:37.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:37.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:37.959858+0000) 2024-03-06T12:23:38.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:38.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:38.960147+0000) 2024-03-06T12:23:39.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:39.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:39.960516+0000) 2024-03-06T12:23:40.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:40.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:40.960807+0000) 2024-03-06T12:23:41.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:41.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:41.961010+0000) 2024-03-06T12:23:42.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:42.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:42.961268+0000) 2024-03-06T12:23:43.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:43.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:43.961545+0000) 2024-03-06T12:23:44.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:44.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:44.961801+0000) 2024-03-06T12:23:45.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:45.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:45.962021+0000) 2024-03-06T12:23:46.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:46.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:46.962210+0000) 2024-03-06T12:23:47.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:47.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:47.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:47.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:47.962409+0000) 2024-03-06T12:23:48.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:48.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:48.962614+0000) 2024-03-06T12:23:49.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:49.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:49.962869+0000) 2024-03-06T12:23:50.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:50.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:50.963076+0000) 2024-03-06T12:23:51.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:51.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:51.963257+0000) 2024-03-06T12:23:52.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:52.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:52.963449+0000) 2024-03-06T12:23:53.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:53.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:53.963704+0000) 2024-03-06T12:23:54.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:54.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:54.963997+0000) 2024-03-06T12:23:55.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:55.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:55.964177+0000) 2024-03-06T12:23:56.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:56.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:56.964424+0000) 2024-03-06T12:23:57.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:57.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:57.964716+0000) 2024-03-06T12:23:58.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:58.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:58.964969+0000) 2024-03-06T12:23:59.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:23:59.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:59.965226+0000) 2024-03-06T12:24:00.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:00.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:00.965479+0000) 2024-03-06T12:24:01.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:01.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:01.965745+0000) 2024-03-06T12:24:02.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:02.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:02.965943+0000) 2024-03-06T12:24:03.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:03.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:03.966124+0000) 2024-03-06T12:24:04.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:04.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:04.966370+0000) 2024-03-06T12:24:05.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:05.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:05.966618+0000) 2024-03-06T12:24:06.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:06.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:06.966890+0000) 2024-03-06T12:24:07.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:07.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:07.967146+0000) 2024-03-06T12:24:08.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:08.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:08.967357+0000) 2024-03-06T12:24:09.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:09.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:09.967607+0000) 2024-03-06T12:24:10.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:10.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:10.967805+0000) 2024-03-06T12:24:11.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:11.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:11.968060+0000) 2024-03-06T12:24:12.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:12.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:12.968308+0000) 2024-03-06T12:24:13.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:13.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:13.968567+0000) 2024-03-06T12:24:14.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:14.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:14.968799+0000) 2024-03-06T12:24:15.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:15.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:15.969039+0000) 2024-03-06T12:24:16.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:16.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:16.969297+0000) 2024-03-06T12:24:17.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:17.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:17.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:17.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:17.969485+0000) 2024-03-06T12:24:18.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:18.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:18.969738+0000) 2024-03-06T12:24:19.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:19.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:19.969977+0000) 2024-03-06T12:24:20.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:20.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:20.970239+0000) 2024-03-06T12:24:21.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:21.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:21.970491+0000) 2024-03-06T12:24:22.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:22.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:22.970759+0000) 2024-03-06T12:24:23.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:23.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:23.971013+0000) 2024-03-06T12:24:24.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:24.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:24.971270+0000) 2024-03-06T12:24:25.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:25.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:25.971510+0000) 2024-03-06T12:24:26.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:26.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:26.971754+0000) 2024-03-06T12:24:27.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:27.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:27.972009+0000) 2024-03-06T12:24:28.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:28.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:28.972260+0000) 2024-03-06T12:24:29.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:29.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:29.972490+0000) 2024-03-06T12:24:30.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:30.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:30.972743+0000) 2024-03-06T12:24:31.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:31.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:31.972989+0000) 2024-03-06T12:24:32.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:32.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:32.973239+0000) 2024-03-06T12:24:33.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:33.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:33.973480+0000) 2024-03-06T12:24:34.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:34.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:34.973741+0000) 2024-03-06T12:24:35.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:35.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:35.973975+0000) 2024-03-06T12:24:36.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:36.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:36.974225+0000) 2024-03-06T12:24:37.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:37.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:37.974477+0000) 2024-03-06T12:24:38.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:38.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:38.974752+0000) 2024-03-06T12:24:39.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:39.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:39.975003+0000) 2024-03-06T12:24:40.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:40.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:40.975269+0000) 2024-03-06T12:24:41.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:41.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:41.975520+0000) 2024-03-06T12:24:42.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:42.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:42.975768+0000) 2024-03-06T12:24:43.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:43.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:43.976019+0000) 2024-03-06T12:24:44.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:44.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:44.976286+0000) 2024-03-06T12:24:45.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:45.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:45.976512+0000) 2024-03-06T12:24:46.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:46.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:46.976746+0000) 2024-03-06T12:24:47.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:47.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:47.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:47.976984+0000) 2024-03-06T12:24:47.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:48.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:48.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:48.977256+0000) 2024-03-06T12:24:49.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:49.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:49.977495+0000) 2024-03-06T12:24:50.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:50.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:50.977760+0000) 2024-03-06T12:24:51.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:51.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:51.977948+0000) 2024-03-06T12:24:52.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:52.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:52.978212+0000) 2024-03-06T12:24:53.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:53.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:53.978442+0000) 2024-03-06T12:24:54.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:54.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:54.978725+0000) 2024-03-06T12:24:55.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:55.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:55.978987+0000) 2024-03-06T12:24:56.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:56.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:56.979233+0000) 2024-03-06T12:24:57.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:57.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:57.979486+0000) 2024-03-06T12:24:58.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:58.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:58.979734+0000) 2024-03-06T12:24:59.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:24:59.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:59.979964+0000) 2024-03-06T12:25:00.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:00.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:00.980215+0000) 2024-03-06T12:25:01.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:01.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:01.980456+0000) 2024-03-06T12:25:02.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:02.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:02.980774+0000) 2024-03-06T12:25:03.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:03.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:03.981027+0000) 2024-03-06T12:25:04.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:04.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:04.981219+0000) 2024-03-06T12:25:05.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:05.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:05.981451+0000) 2024-03-06T12:25:06.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:06.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:06.981759+0000) 2024-03-06T12:25:07.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:07.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:07.981996+0000) 2024-03-06T12:25:08.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:08.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:08.982260+0000) 2024-03-06T12:25:09.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:09.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:09.982492+0000) 2024-03-06T12:25:10.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:10.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:10.982754+0000) 2024-03-06T12:25:11.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:11.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:11.983005+0000) 2024-03-06T12:25:12.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:12.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:12.983286+0000) 2024-03-06T12:25:13.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:13.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:13.983525+0000) 2024-03-06T12:25:14.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:14.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:14.983787+0000) 2024-03-06T12:25:15.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:15.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:15.984042+0000) 2024-03-06T12:25:16.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:16.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:16.984305+0000) 2024-03-06T12:25:17.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:17.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:17.984560+0000) 2024-03-06T12:25:18.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:18.124 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:18.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:18.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:18.984826+0000) 2024-03-06T12:25:19.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:19.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:19.985066+0000) 2024-03-06T12:25:20.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:20.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:20.985332+0000) 2024-03-06T12:25:21.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:21.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:21.985586+0000) 2024-03-06T12:25:22.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:22.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:22.985858+0000) 2024-03-06T12:25:23.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:23.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:23.986114+0000) 2024-03-06T12:25:24.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:24.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:24.986424+0000) 2024-03-06T12:25:25.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:25.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:25.986713+0000) 2024-03-06T12:25:26.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:26.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:26.986954+0000) 2024-03-06T12:25:27.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:27.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:27.987187+0000) 2024-03-06T12:25:28.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:28.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:28.987444+0000) 2024-03-06T12:25:29.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:29.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:29.987665+0000) 2024-03-06T12:25:30.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:30.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:30.987919+0000) 2024-03-06T12:25:31.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:31.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:31.988204+0000) 2024-03-06T12:25:32.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:32.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:32.988505+0000) 2024-03-06T12:25:33.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:33.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:33.988819+0000) 2024-03-06T12:25:34.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:34.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:34.989059+0000) 2024-03-06T12:25:35.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:35.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:35.989354+0000) 2024-03-06T12:25:36.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:36.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:36.989620+0000) 2024-03-06T12:25:37.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:37.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:37.989968+0000) 2024-03-06T12:25:38.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:38.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:38.990263+0000) 2024-03-06T12:25:39.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:39.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:39.990557+0000) 2024-03-06T12:25:40.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:40.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:40.990858+0000) 2024-03-06T12:25:41.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:41.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:41.991166+0000) 2024-03-06T12:25:42.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:42.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:42.991477+0000) 2024-03-06T12:25:43.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:43.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:43.991770+0000) 2024-03-06T12:25:44.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:44.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:44.992088+0000) 2024-03-06T12:25:45.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:45.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:45.992383+0000) 2024-03-06T12:25:46.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:46.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:46.992730+0000) 2024-03-06T12:25:47.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:47.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:47.993049+0000) 2024-03-06T12:25:48.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:48.277 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:48.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:48.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:48.993372+0000) 2024-03-06T12:25:49.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:49.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:49.993697+0000) 2024-03-06T12:25:50.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:50.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:50.994023+0000) 2024-03-06T12:25:51.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:51.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:51.994252+0000) 2024-03-06T12:25:52.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:52.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:52.994561+0000) 2024-03-06T12:25:53.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:53.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:53.994851+0000) 2024-03-06T12:25:54.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:54.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:54.995156+0000) 2024-03-06T12:25:55.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:55.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:55.995420+0000) 2024-03-06T12:25:56.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:56.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:56.995732+0000) 2024-03-06T12:25:57.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:57.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:57.996005+0000) 2024-03-06T12:25:58.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:58.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:58.996275+0000) 2024-03-06T12:25:59.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:25:59.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:59.996508+0000) 2024-03-06T12:26:00.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:00.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:00.996772+0000) 2024-03-06T12:26:01.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:01.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:01.997030+0000) 2024-03-06T12:26:02.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:02.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:02.997288+0000) 2024-03-06T12:26:03.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:03.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:03.997557+0000) 2024-03-06T12:26:04.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:04.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:04.997858+0000) 2024-03-06T12:26:05.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:05.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:05.998173+0000) 2024-03-06T12:26:06.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:06.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:06.998449+0000) 2024-03-06T12:26:07.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:07.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:07.998722+0000) 2024-03-06T12:26:08.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:08.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:08.998967+0000) 2024-03-06T12:26:09.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:09.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:09.999219+0000) 2024-03-06T12:26:10.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:10.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:10.999475+0000) 2024-03-06T12:26:11.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:11.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:11.999734+0000) 2024-03-06T12:26:12.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:12.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:12.999993+0000) 2024-03-06T12:26:13.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:13.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:14.000241+0000) 2024-03-06T12:26:14.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:15.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:15.000481+0000) 2024-03-06T12:26:15.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:16.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:16.000764+0000) 2024-03-06T12:26:16.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:17.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:17.001032+0000) 2024-03-06T12:26:17.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:18.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:18.001335+0000) 2024-03-06T12:26:18.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:18.445 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:18.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:19.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:19.001610+0000) 2024-03-06T12:26:19.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:20.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:20.001882+0000) 2024-03-06T12:26:20.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:21.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:21.002124+0000) 2024-03-06T12:26:21.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:22.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:22.002358+0000) 2024-03-06T12:26:22.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:23.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:23.002612+0000) 2024-03-06T12:26:23.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:24.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:24.002913+0000) 2024-03-06T12:26:24.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:25.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:25.003109+0000) 2024-03-06T12:26:25.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:26.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:26.003345+0000) 2024-03-06T12:26:26.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:27.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:27.003581+0000) 2024-03-06T12:26:27.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:28.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:28.003757+0000) 2024-03-06T12:26:28.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:29.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:29.004018+0000) 2024-03-06T12:26:29.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:30.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:30.004256+0000) 2024-03-06T12:26:30.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:31.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:31.004501+0000) 2024-03-06T12:26:31.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:32.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:32.004744+0000) 2024-03-06T12:26:32.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:33.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:33.004995+0000) 2024-03-06T12:26:34.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:34.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:34.005247+0000) 2024-03-06T12:26:35.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:35.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:35.005495+0000) 2024-03-06T12:26:36.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:36.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:36.005751+0000) 2024-03-06T12:26:37.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:37.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:37.005997+0000) 2024-03-06T12:26:38.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:38.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:38.006250+0000) 2024-03-06T12:26:39.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:39.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:39.006524+0000) 2024-03-06T12:26:40.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:40.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:40.006777+0000) 2024-03-06T12:26:41.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:41.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:41.007042+0000) 2024-03-06T12:26:42.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:42.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:42.007297+0000) 2024-03-06T12:26:43.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:43.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:43.007495+0000) 2024-03-06T12:26:44.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:44.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:44.007748+0000) 2024-03-06T12:26:45.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:45.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:45.008015+0000) 2024-03-06T12:26:46.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:46.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:46.008269+0000) 2024-03-06T12:26:47.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:47.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:47.008531+0000) 2024-03-06T12:26:48.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:48.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:48.008785+0000) 2024-03-06T12:26:48.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:48.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:49.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:49.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:49.009060+0000) 2024-03-06T12:26:50.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:50.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:50.009302+0000) 2024-03-06T12:26:51.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:51.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:51.009537+0000) 2024-03-06T12:26:52.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:52.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:52.009771+0000) 2024-03-06T12:26:53.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:53.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:53.010027+0000) 2024-03-06T12:26:54.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:54.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:54.010279+0000) 2024-03-06T12:26:55.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:55.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:55.010538+0000) 2024-03-06T12:26:56.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:56.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:56.010790+0000) 2024-03-06T12:26:57.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:57.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:57.011047+0000) 2024-03-06T12:26:58.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:58.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:58.011299+0000) 2024-03-06T12:26:59.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:26:59.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:59.011546+0000) 2024-03-06T12:27:00.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:00.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:00.011794+0000) 2024-03-06T12:27:01.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:01.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:01.012053+0000) 2024-03-06T12:27:02.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:02.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:02.012286+0000) 2024-03-06T12:27:03.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:03.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:03.012540+0000) 2024-03-06T12:27:04.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:04.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:04.012793+0000) 2024-03-06T12:27:05.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:05.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:05.013036+0000) 2024-03-06T12:27:06.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:06.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:06.013341+0000) 2024-03-06T12:27:07.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:07.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:07.013596+0000) 2024-03-06T12:27:08.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:08.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:08.013850+0000) 2024-03-06T12:27:09.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:09.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:09.014121+0000) 2024-03-06T12:27:10.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:10.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:10.014351+0000) 2024-03-06T12:27:11.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:11.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:11.014590+0000) 2024-03-06T12:27:12.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:12.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:12.014843+0000) 2024-03-06T12:27:13.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:13.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:13.015088+0000) 2024-03-06T12:27:14.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:14.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:14.015345+0000) 2024-03-06T12:27:15.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:15.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:15.015600+0000) 2024-03-06T12:27:16.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:16.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:16.015872+0000) 2024-03-06T12:27:17.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:17.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:17.016134+0000) 2024-03-06T12:27:18.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:18.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:18.016388+0000) 2024-03-06T12:27:18.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:18.747 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:19.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:19.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:19.016705+0000) 2024-03-06T12:27:20.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:20.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:20.016939+0000) 2024-03-06T12:27:21.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:21.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:21.017186+0000) 2024-03-06T12:27:22.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:22.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:22.017430+0000) 2024-03-06T12:27:23.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:23.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:23.017756+0000) 2024-03-06T12:27:24.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:24.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:24.018007+0000) 2024-03-06T12:27:25.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:25.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:25.018268+0000) 2024-03-06T12:27:26.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:26.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:26.018518+0000) 2024-03-06T12:27:27.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:27.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:27.018787+0000) 2024-03-06T12:27:28.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:28.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:28.019045+0000) 2024-03-06T12:27:29.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:29.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:29.019295+0000) 2024-03-06T12:27:30.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:30.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:30.019529+0000) 2024-03-06T12:27:31.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:31.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:31.019787+0000) 2024-03-06T12:27:32.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:32.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:32.020038+0000) 2024-03-06T12:27:33.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:33.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:33.020295+0000) 2024-03-06T12:27:34.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:34.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:34.020547+0000) 2024-03-06T12:27:35.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:35.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:35.020801+0000) 2024-03-06T12:27:36.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:36.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:36.021058+0000) 2024-03-06T12:27:37.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:37.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:37.021317+0000) 2024-03-06T12:27:38.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:38.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:38.021566+0000) 2024-03-06T12:27:39.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:39.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:39.021833+0000) 2024-03-06T12:27:40.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:40.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:40.022073+0000) 2024-03-06T12:27:41.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:41.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:41.022314+0000) 2024-03-06T12:27:42.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:42.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:42.022548+0000) 2024-03-06T12:27:43.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:43.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:43.022806+0000) 2024-03-06T12:27:44.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:44.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:44.023061+0000) 2024-03-06T12:27:45.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:45.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:45.023323+0000) 2024-03-06T12:27:46.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:46.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:46.023556+0000) 2024-03-06T12:27:47.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:47.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:47.023798+0000) 2024-03-06T12:27:48.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:48.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:48.024053+0000) 2024-03-06T12:27:48.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:48.894 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:49.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:49.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:49.024317+0000) 2024-03-06T12:27:50.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:50.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:50.024505+0000) 2024-03-06T12:27:51.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:51.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:51.024758+0000) 2024-03-06T12:27:52.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:52.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:52.025009+0000) 2024-03-06T12:27:53.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:53.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:53.025252+0000) 2024-03-06T12:27:54.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:54.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:54.025574+0000) 2024-03-06T12:27:55.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:55.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:55.025816+0000) 2024-03-06T12:27:56.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:56.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:56.026073+0000) 2024-03-06T12:27:57.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:57.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:57.026335+0000) 2024-03-06T12:27:58.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:58.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:58.026585+0000) 2024-03-06T12:27:59.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:27:59.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:59.026855+0000) 2024-03-06T12:28:00.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:00.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:00.027107+0000) 2024-03-06T12:28:01.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:01.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:01.027343+0000) 2024-03-06T12:28:02.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:02.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:02.027595+0000) 2024-03-06T12:28:03.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:03.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:03.027846+0000) 2024-03-06T12:28:04.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:04.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:04.028028+0000) 2024-03-06T12:28:05.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:05.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:05.028283+0000) 2024-03-06T12:28:06.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:06.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:06.028517+0000) 2024-03-06T12:28:07.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:07.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:07.028785+0000) 2024-03-06T12:28:08.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:08.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:08.029021+0000) 2024-03-06T12:28:09.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:09.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:09.029296+0000) 2024-03-06T12:28:10.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:10.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:10.029546+0000) 2024-03-06T12:28:11.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:11.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:11.029797+0000) 2024-03-06T12:28:12.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:12.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:12.030046+0000) 2024-03-06T12:28:13.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:13.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:13.030247+0000) 2024-03-06T12:28:14.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:14.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:14.030496+0000) 2024-03-06T12:28:15.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:15.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:15.030752+0000) 2024-03-06T12:28:16.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:16.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:16.030963+0000) 2024-03-06T12:28:17.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:17.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:17.031235+0000) 2024-03-06T12:28:18.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:18.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:18.031468+0000) 2024-03-06T12:28:19.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:19.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:19.031773+0000) 2024-03-06T12:28:19.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:19.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:20.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:20.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:20.032008+0000) 2024-03-06T12:28:21.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:21.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:21.032261+0000) 2024-03-06T12:28:22.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:22.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:22.032511+0000) 2024-03-06T12:28:23.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:23.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:23.032762+0000) 2024-03-06T12:28:24.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:24.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:24.033017+0000) 2024-03-06T12:28:25.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:25.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:25.033280+0000) 2024-03-06T12:28:26.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:26.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:26.033513+0000) 2024-03-06T12:28:27.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:27.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:27.033777+0000) 2024-03-06T12:28:28.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:28.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:28.034008+0000) 2024-03-06T12:28:29.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:29.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:29.034268+0000) 2024-03-06T12:28:30.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:30.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:30.034515+0000) 2024-03-06T12:28:31.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:31.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:31.034778+0000) 2024-03-06T12:28:32.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:32.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:32.035033+0000) 2024-03-06T12:28:33.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:33.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:33.035279+0000) 2024-03-06T12:28:34.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:34.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:34.035530+0000) 2024-03-06T12:28:35.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:35.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:35.035789+0000) 2024-03-06T12:28:36.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:36.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:36.036013+0000) 2024-03-06T12:28:37.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:37.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:37.036213+0000) 2024-03-06T12:28:38.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:38.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:38.036464+0000) 2024-03-06T12:28:39.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:39.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:39.036767+0000) 2024-03-06T12:28:40.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:40.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:40.037016+0000) 2024-03-06T12:28:41.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:41.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:41.037288+0000) 2024-03-06T12:28:42.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:42.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:42.037538+0000) 2024-03-06T12:28:43.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:43.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:43.037783+0000) 2024-03-06T12:28:44.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:44.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:44.037991+0000) 2024-03-06T12:28:45.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:45.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:45.038230+0000) 2024-03-06T12:28:46.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:46.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:46.038466+0000) 2024-03-06T12:28:47.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:47.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:47.038766+0000) 2024-03-06T12:28:48.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:48.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:48.039052+0000) 2024-03-06T12:28:49.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:49.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:49.039303+0000) 2024-03-06T12:28:49.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:49.179 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:50.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:50.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:50.039559+0000) 2024-03-06T12:28:51.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:51.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:51.039814+0000) 2024-03-06T12:28:52.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:52.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:52.040081+0000) 2024-03-06T12:28:53.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:53.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:53.040334+0000) 2024-03-06T12:28:54.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:54.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:54.040566+0000) 2024-03-06T12:28:55.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:55.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:55.040827+0000) 2024-03-06T12:28:56.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:56.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:56.041082+0000) 2024-03-06T12:28:57.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:57.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:57.041333+0000) 2024-03-06T12:28:58.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:58.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:58.041585+0000) 2024-03-06T12:28:59.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:28:59.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:59.041848+0000) 2024-03-06T12:29:00.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:00.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:00.042097+0000) 2024-03-06T12:29:01.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:01.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:01.042352+0000) 2024-03-06T12:29:02.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:02.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:02.042584+0000) 2024-03-06T12:29:03.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:03.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:03.042841+0000) 2024-03-06T12:29:04.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:04.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:04.043095+0000) 2024-03-06T12:29:05.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:05.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:05.043347+0000) 2024-03-06T12:29:06.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:06.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:06.043579+0000) 2024-03-06T12:29:07.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:07.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:07.043769+0000) 2024-03-06T12:29:08.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:08.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:08.043955+0000) 2024-03-06T12:29:09.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:09.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:09.044238+0000) 2024-03-06T12:29:10.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:10.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:10.044471+0000) 2024-03-06T12:29:11.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:11.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:11.044755+0000) 2024-03-06T12:29:12.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:12.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:12.044989+0000) 2024-03-06T12:29:13.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:13.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:13.045250+0000) 2024-03-06T12:29:14.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:14.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:14.045497+0000) 2024-03-06T12:29:15.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:15.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:15.045760+0000) 2024-03-06T12:29:16.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:16.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:16.046016+0000) 2024-03-06T12:29:17.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:17.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:17.046268+0000) 2024-03-06T12:29:18.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:18.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:18.046505+0000) 2024-03-06T12:29:19.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:19.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:19.046761+0000) 2024-03-06T12:29:19.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:19.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:20.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:20.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:20.047019+0000) 2024-03-06T12:29:21.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:21.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:21.047298+0000) 2024-03-06T12:29:22.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:22.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:22.047567+0000) 2024-03-06T12:29:23.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:23.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:23.047830+0000) 2024-03-06T12:29:24.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:24.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:24.048100+0000) 2024-03-06T12:29:25.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:25.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:25.048359+0000) 2024-03-06T12:29:26.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:26.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:26.048607+0000) 2024-03-06T12:29:27.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:27.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:27.048911+0000) 2024-03-06T12:29:28.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:28.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:28.049143+0000) 2024-03-06T12:29:29.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:29.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:29.049422+0000) 2024-03-06T12:29:30.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:30.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:30.049692+0000) 2024-03-06T12:29:31.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:31.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:31.049934+0000) 2024-03-06T12:29:32.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:32.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:32.050195+0000) 2024-03-06T12:29:33.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:33.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:33.050450+0000) 2024-03-06T12:29:34.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:34.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:34.050742+0000) 2024-03-06T12:29:35.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:35.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:35.051020+0000) 2024-03-06T12:29:36.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:36.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:36.051343+0000) 2024-03-06T12:29:37.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:37.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:37.051616+0000) 2024-03-06T12:29:38.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:38.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:38.051897+0000) 2024-03-06T12:29:39.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:39.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:39.052193+0000) 2024-03-06T12:29:40.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:40.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:40.052442+0000) 2024-03-06T12:29:41.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:41.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:41.052747+0000) 2024-03-06T12:29:42.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:42.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:42.053016+0000) 2024-03-06T12:29:43.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:43.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:43.053291+0000) 2024-03-06T12:29:44.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:44.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:44.053563+0000) 2024-03-06T12:29:45.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:45.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:45.053858+0000) 2024-03-06T12:29:46.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:46.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:46.054148+0000) 2024-03-06T12:29:47.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:47.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:47.054437+0000) 2024-03-06T12:29:48.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:48.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:48.054790+0000) 2024-03-06T12:29:49.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:49.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:49.055117+0000) 2024-03-06T12:29:49.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:49.496 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:50.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:50.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:50.055409+0000) 2024-03-06T12:29:51.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:51.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:51.055770+0000) 2024-03-06T12:29:52.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:52.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:52.056046+0000) 2024-03-06T12:29:53.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:53.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:53.056276+0000) 2024-03-06T12:29:54.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:54.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:54.056531+0000) 2024-03-06T12:29:55.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:55.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:55.056790+0000) 2024-03-06T12:29:56.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:56.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:56.057057+0000) 2024-03-06T12:29:57.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:57.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:57.057328+0000) 2024-03-06T12:29:58.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:58.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:58.057567+0000) 2024-03-06T12:29:59.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:29:59.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:59.057870+0000) 2024-03-06T12:30:00.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:00.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:00.058185+0000) 2024-03-06T12:30:01.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:01.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:01.058484+0000) 2024-03-06T12:30:02.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:02.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:02.058789+0000) 2024-03-06T12:30:03.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:03.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:03.059041+0000) 2024-03-06T12:30:04.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:04.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:04.059335+0000) 2024-03-06T12:30:05.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:05.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:05.059563+0000) 2024-03-06T12:30:06.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:06.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:06.059861+0000) 2024-03-06T12:30:07.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:07.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:07.060184+0000) 2024-03-06T12:30:08.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:08.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:08.060496+0000) 2024-03-06T12:30:09.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:09.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:09.060794+0000) 2024-03-06T12:30:10.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:10.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:10.061093+0000) 2024-03-06T12:30:11.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:11.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:11.061389+0000) 2024-03-06T12:30:12.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:12.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:12.061719+0000) 2024-03-06T12:30:13.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:13.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:13.061993+0000) 2024-03-06T12:30:14.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:14.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:14.062276+0000) 2024-03-06T12:30:15.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:15.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:15.062583+0000) 2024-03-06T12:30:16.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:16.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:16.062926+0000) 2024-03-06T12:30:17.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:17.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:17.063249+0000) 2024-03-06T12:30:18.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:18.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:18.063566+0000) 2024-03-06T12:30:19.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:19.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:19.063910+0000) 2024-03-06T12:30:19.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:19.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:20.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:20.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:20.064197+0000) 2024-03-06T12:30:21.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:21.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:21.064471+0000) 2024-03-06T12:30:22.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:22.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:22.064761+0000) 2024-03-06T12:30:23.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:23.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:23.065080+0000) 2024-03-06T12:30:24.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:24.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:24.065454+0000) 2024-03-06T12:30:25.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:25.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:25.065735+0000) 2024-03-06T12:30:26.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:26.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:26.066008+0000) 2024-03-06T12:30:27.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:27.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:27.066273+0000) 2024-03-06T12:30:28.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:28.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:28.066528+0000) 2024-03-06T12:30:29.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:29.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:29.066828+0000) 2024-03-06T12:30:30.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:30.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:30.067100+0000) 2024-03-06T12:30:31.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:31.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:31.067381+0000) 2024-03-06T12:30:32.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:32.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:32.067688+0000) 2024-03-06T12:30:33.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:33.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:33.067958+0000) 2024-03-06T12:30:34.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:34.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:34.068239+0000) 2024-03-06T12:30:35.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:35.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:35.068515+0000) 2024-03-06T12:30:36.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:36.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:36.068783+0000) 2024-03-06T12:30:37.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:37.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:37.069035+0000) 2024-03-06T12:30:38.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:38.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:38.069271+0000) 2024-03-06T12:30:39.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:39.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:39.069542+0000) 2024-03-06T12:30:40.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:40.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:40.069825+0000) 2024-03-06T12:30:41.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:41.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:41.070086+0000) 2024-03-06T12:30:42.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:42.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:42.070343+0000) 2024-03-06T12:30:43.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:43.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:43.070602+0000) 2024-03-06T12:30:44.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:44.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:44.070894+0000) 2024-03-06T12:30:45.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:45.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:45.071098+0000) 2024-03-06T12:30:46.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:46.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:46.071324+0000) 2024-03-06T12:30:47.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:47.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:47.071602+0000) 2024-03-06T12:30:48.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:48.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:48.071894+0000) 2024-03-06T12:30:49.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:49.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:49.072163+0000) 2024-03-06T12:30:49.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:49.769 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:50.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:50.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:50.072416+0000) 2024-03-06T12:30:51.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:51.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:51.072728+0000) 2024-03-06T12:30:52.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:52.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:52.072993+0000) 2024-03-06T12:30:53.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:53.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:53.073247+0000) 2024-03-06T12:30:54.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:54.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:54.073481+0000) 2024-03-06T12:30:55.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:55.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:55.073749+0000) 2024-03-06T12:30:56.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:56.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:56.074005+0000) 2024-03-06T12:30:57.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:57.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:57.074281+0000) 2024-03-06T12:30:58.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:58.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:58.074491+0000) 2024-03-06T12:30:59.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:30:59.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:59.074790+0000) 2024-03-06T12:31:00.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:00.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:00.075064+0000) 2024-03-06T12:31:01.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:01.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:01.075339+0000) 2024-03-06T12:31:02.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:02.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:02.075604+0000) 2024-03-06T12:31:03.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:03.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:03.075904+0000) 2024-03-06T12:31:04.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:04.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:04.076154+0000) 2024-03-06T12:31:05.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:05.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:05.076402+0000) 2024-03-06T12:31:06.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:06.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:06.076671+0000) 2024-03-06T12:31:07.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:07.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:07.076917+0000) 2024-03-06T12:31:08.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:08.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:08.077164+0000) 2024-03-06T12:31:09.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:09.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:09.077368+0000) 2024-03-06T12:31:10.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:10.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:10.077618+0000) 2024-03-06T12:31:11.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:11.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:11.077928+0000) 2024-03-06T12:31:12.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:12.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:12.078115+0000) 2024-03-06T12:31:13.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:13.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:13.078376+0000) 2024-03-06T12:31:14.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:14.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:14.078662+0000) 2024-03-06T12:31:15.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:15.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:15.078851+0000) 2024-03-06T12:31:16.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:16.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:16.079083+0000) 2024-03-06T12:31:17.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:17.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:17.079386+0000) 2024-03-06T12:31:18.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:18.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:18.079621+0000) 2024-03-06T12:31:19.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:19.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:19.079982+0000) 2024-03-06T12:31:19.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:19.922 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:20.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:20.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:20.080251+0000) 2024-03-06T12:31:21.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:21.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:21.080508+0000) 2024-03-06T12:31:22.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:22.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:22.080768+0000) 2024-03-06T12:31:23.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:23.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:23.081033+0000) 2024-03-06T12:31:24.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:24.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:24.081220+0000) 2024-03-06T12:31:25.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:25.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:25.081478+0000) 2024-03-06T12:31:26.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:26.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:26.081728+0000) 2024-03-06T12:31:27.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:27.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:27.081981+0000) 2024-03-06T12:31:28.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:28.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:28.082242+0000) 2024-03-06T12:31:29.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:29.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:29.082517+0000) 2024-03-06T12:31:30.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:30.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:30.082748+0000) 2024-03-06T12:31:31.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:31.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:31.083012+0000) 2024-03-06T12:31:32.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:32.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:32.083206+0000) 2024-03-06T12:31:33.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:33.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:33.083461+0000) 2024-03-06T12:31:34.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:34.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:34.083749+0000) 2024-03-06T12:31:35.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:35.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:35.083987+0000) 2024-03-06T12:31:36.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:36.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:36.084227+0000) 2024-03-06T12:31:37.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:37.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:37.084481+0000) 2024-03-06T12:31:38.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:38.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:38.084747+0000) 2024-03-06T12:31:39.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:39.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:39.085016+0000) 2024-03-06T12:31:40.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:40.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:40.085271+0000) 2024-03-06T12:31:41.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:41.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:41.085525+0000) 2024-03-06T12:31:42.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:42.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:42.085776+0000) 2024-03-06T12:31:43.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:43.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:43.086020+0000) 2024-03-06T12:31:44.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:44.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:44.086211+0000) 2024-03-06T12:31:45.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:45.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:45.086441+0000) 2024-03-06T12:31:46.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:46.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:46.086730+0000) 2024-03-06T12:31:47.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:47.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:47.086994+0000) 2024-03-06T12:31:48.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:48.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:48.087267+0000) 2024-03-06T12:31:49.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:49.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:49.087560+0000) 2024-03-06T12:31:50.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:50.060 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:50.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:50.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:50.087746+0000) 2024-03-06T12:31:51.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:51.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:51.088003+0000) 2024-03-06T12:31:52.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:52.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:52.088253+0000) 2024-03-06T12:31:53.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:53.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:53.088532+0000) 2024-03-06T12:31:54.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:54.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:54.088783+0000) 2024-03-06T12:31:55.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:55.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:55.089060+0000) 2024-03-06T12:31:56.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:56.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:56.089335+0000) 2024-03-06T12:31:57.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:57.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:57.089587+0000) 2024-03-06T12:31:58.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:58.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:58.089877+0000) 2024-03-06T12:31:59.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:31:59.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:59.090143+0000) 2024-03-06T12:32:00.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:00.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:00.090391+0000) 2024-03-06T12:32:01.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:01.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:01.090715+0000) 2024-03-06T12:32:02.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:02.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:02.090969+0000) 2024-03-06T12:32:03.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:03.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:03.091210+0000) 2024-03-06T12:32:04.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:04.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:04.091447+0000) 2024-03-06T12:32:05.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:05.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:05.091801+0000) 2024-03-06T12:32:06.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:06.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:06.092041+0000) 2024-03-06T12:32:07.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:07.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:07.092240+0000) 2024-03-06T12:32:08.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:08.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:08.092488+0000) 2024-03-06T12:32:09.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:09.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:09.092768+0000) 2024-03-06T12:32:10.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:10.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:10.093015+0000) 2024-03-06T12:32:11.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:11.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:11.093226+0000) 2024-03-06T12:32:12.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:12.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:12.093480+0000) 2024-03-06T12:32:13.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:13.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:13.093768+0000) 2024-03-06T12:32:14.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:14.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:14.094054+0000) 2024-03-06T12:32:15.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:15.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:15.094332+0000) 2024-03-06T12:32:16.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:16.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:16.094588+0000) 2024-03-06T12:32:17.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:17.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:17.094851+0000) 2024-03-06T12:32:18.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:18.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:18.095106+0000) 2024-03-06T12:32:19.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:19.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:19.095359+0000) 2024-03-06T12:32:20.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:20.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:20.095614+0000) 2024-03-06T12:32:20.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:20.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:21.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:21.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:21.095922+0000) 2024-03-06T12:32:22.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:22.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:22.096173+0000) 2024-03-06T12:32:23.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:23.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:23.096438+0000) 2024-03-06T12:32:24.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:24.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:24.096721+0000) 2024-03-06T12:32:25.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:25.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:25.096990+0000) 2024-03-06T12:32:26.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:26.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:26.097184+0000) 2024-03-06T12:32:27.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:27.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:27.097411+0000) 2024-03-06T12:32:28.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:28.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:28.097716+0000) 2024-03-06T12:32:29.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:29.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:29.097986+0000) 2024-03-06T12:32:30.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:30.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:30.098235+0000) 2024-03-06T12:32:31.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:31.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:31.098495+0000) 2024-03-06T12:32:32.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:32.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:32.098752+0000) 2024-03-06T12:32:33.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:33.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:33.098968+0000) 2024-03-06T12:32:34.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:34.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:34.099200+0000) 2024-03-06T12:32:35.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:35.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:35.099452+0000) 2024-03-06T12:32:36.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:36.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:36.099733+0000) 2024-03-06T12:32:37.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:37.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:37.100002+0000) 2024-03-06T12:32:38.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:38.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:38.100235+0000) 2024-03-06T12:32:39.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:39.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:39.100500+0000) 2024-03-06T12:32:40.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:40.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:40.100755+0000) 2024-03-06T12:32:41.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:41.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:41.101014+0000) 2024-03-06T12:32:42.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:42.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:42.101252+0000) 2024-03-06T12:32:43.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:43.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:43.101508+0000) 2024-03-06T12:32:44.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:44.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:44.101740+0000) 2024-03-06T12:32:45.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:45.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:45.102004+0000) 2024-03-06T12:32:46.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:46.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:46.102251+0000) 2024-03-06T12:32:47.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:47.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:47.102505+0000) 2024-03-06T12:32:48.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:48.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:48.102766+0000) 2024-03-06T12:32:49.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:49.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:49.103035+0000) 2024-03-06T12:32:50.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:50.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:50.103291+0000) 2024-03-06T12:32:50.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:50.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:51.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:51.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:51.103540+0000) 2024-03-06T12:32:52.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:52.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:52.103769+0000) 2024-03-06T12:32:53.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:53.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:53.104080+0000) 2024-03-06T12:32:54.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:54.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:54.104315+0000) 2024-03-06T12:32:55.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:55.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:55.104576+0000) 2024-03-06T12:32:56.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:56.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:56.104811+0000) 2024-03-06T12:32:57.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:57.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:57.105048+0000) 2024-03-06T12:32:58.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:58.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:58.105280+0000) 2024-03-06T12:32:59.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:32:59.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:59.105550+0000) 2024-03-06T12:33:00.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:00.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:00.105783+0000) 2024-03-06T12:33:01.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:01.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:01.105985+0000) 2024-03-06T12:33:02.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:02.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:02.106219+0000) 2024-03-06T12:33:03.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:03.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:03.106453+0000) 2024-03-06T12:33:04.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:04.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:04.106759+0000) 2024-03-06T12:33:05.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:05.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:05.107009+0000) 2024-03-06T12:33:06.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:06.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:06.107245+0000) 2024-03-06T12:33:07.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:07.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:07.107489+0000) 2024-03-06T12:33:08.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:08.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:08.107728+0000) 2024-03-06T12:33:09.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:09.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:09.108017+0000) 2024-03-06T12:33:10.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:10.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:10.108273+0000) 2024-03-06T12:33:11.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:11.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:11.108519+0000) 2024-03-06T12:33:12.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:12.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:12.108769+0000) 2024-03-06T12:33:13.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:13.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:13.109011+0000) 2024-03-06T12:33:14.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:14.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:14.109262+0000) 2024-03-06T12:33:15.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:15.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:15.109520+0000) 2024-03-06T12:33:16.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:16.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:16.109778+0000) 2024-03-06T12:33:17.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:17.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:17.110013+0000) 2024-03-06T12:33:18.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:18.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:18.110265+0000) 2024-03-06T12:33:19.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:19.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:19.110512+0000) 2024-03-06T12:33:20.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:20.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:20.110748+0000) 2024-03-06T12:33:20.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:20.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:21.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:21.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:21.111008+0000) 2024-03-06T12:33:22.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:22.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:22.111258+0000) 2024-03-06T12:33:23.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:23.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:23.111518+0000) 2024-03-06T12:33:24.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:24.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:24.111766+0000) 2024-03-06T12:33:25.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:25.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:25.112030+0000) 2024-03-06T12:33:26.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:26.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:26.112279+0000) 2024-03-06T12:33:27.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:27.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:27.112525+0000) 2024-03-06T12:33:28.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:28.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:28.112779+0000) 2024-03-06T12:33:29.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:29.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:29.113048+0000) 2024-03-06T12:33:30.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:30.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:30.113287+0000) 2024-03-06T12:33:31.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:31.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:31.113537+0000) 2024-03-06T12:33:32.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:32.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:32.113772+0000) 2024-03-06T12:33:33.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:33.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:33.114015+0000) 2024-03-06T12:33:34.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:34.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:34.114262+0000) 2024-03-06T12:33:35.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:35.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:35.114536+0000) 2024-03-06T12:33:36.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:36.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:36.114813+0000) 2024-03-06T12:33:37.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:37.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:37.115073+0000) 2024-03-06T12:33:38.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:38.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:38.115321+0000) 2024-03-06T12:33:39.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:39.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:39.115515+0000) 2024-03-06T12:33:40.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:40.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:40.115762+0000) 2024-03-06T12:33:41.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:41.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:41.116019+0000) 2024-03-06T12:33:42.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:42.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:42.116269+0000) 2024-03-06T12:33:43.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:43.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:43.116530+0000) 2024-03-06T12:33:44.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:44.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:44.116779+0000) 2024-03-06T12:33:45.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:45.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:45.117031+0000) 2024-03-06T12:33:46.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:46.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:46.117336+0000) 2024-03-06T12:33:47.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:47.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:47.117589+0000) 2024-03-06T12:33:48.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:48.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:48.117822+0000) 2024-03-06T12:33:49.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:49.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:49.118097+0000) 2024-03-06T12:33:50.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:50.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:50.118333+0000) 2024-03-06T12:33:50.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:50.660 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:51.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:51.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:51.118592+0000) 2024-03-06T12:33:52.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:52.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:52.118854+0000) 2024-03-06T12:33:53.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:53.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:53.119055+0000) 2024-03-06T12:33:54.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:54.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:54.119309+0000) 2024-03-06T12:33:55.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:55.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:55.119605+0000) 2024-03-06T12:33:56.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:56.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:56.119948+0000) 2024-03-06T12:33:57.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:57.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:57.120265+0000) 2024-03-06T12:33:58.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:58.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:58.120552+0000) 2024-03-06T12:33:59.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:33:59.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:59.120879+0000) 2024-03-06T12:34:00.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:00.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:00.121174+0000) 2024-03-06T12:34:01.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:01.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:01.121417+0000) 2024-03-06T12:34:02.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:02.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:02.121743+0000) 2024-03-06T12:34:03.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:03.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:03.122040+0000) 2024-03-06T12:34:04.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:04.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:04.122345+0000) 2024-03-06T12:34:05.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:05.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:05.122705+0000) 2024-03-06T12:34:06.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:06.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:06.122946+0000) 2024-03-06T12:34:07.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:07.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:07.123241+0000) 2024-03-06T12:34:08.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:08.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:08.123530+0000) 2024-03-06T12:34:09.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:09.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:09.123827+0000) 2024-03-06T12:34:10.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:10.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:10.124134+0000) 2024-03-06T12:34:11.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:11.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:11.124455+0000) 2024-03-06T12:34:12.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:12.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:12.124778+0000) 2024-03-06T12:34:13.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:13.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:13.125079+0000) 2024-03-06T12:34:14.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:14.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:14.125377+0000) 2024-03-06T12:34:15.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:15.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:15.125712+0000) 2024-03-06T12:34:16.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:16.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:16.125964+0000) 2024-03-06T12:34:17.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:17.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:17.126271+0000) 2024-03-06T12:34:18.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:18.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:18.126572+0000) 2024-03-06T12:34:19.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:19.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:19.126939+0000) 2024-03-06T12:34:20.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:20.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:20.127248+0000) 2024-03-06T12:34:20.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:20.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:21.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:21.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:21.127554+0000) 2024-03-06T12:34:22.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:22.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:22.127868+0000) 2024-03-06T12:34:23.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:23.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:23.128156+0000) 2024-03-06T12:34:24.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:24.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:24.128461+0000) 2024-03-06T12:34:25.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:25.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:25.128802+0000) 2024-03-06T12:34:26.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:26.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:26.129120+0000) 2024-03-06T12:34:27.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:27.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:27.129424+0000) 2024-03-06T12:34:28.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:28.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:28.129770+0000) 2024-03-06T12:34:29.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:29.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:29.130089+0000) 2024-03-06T12:34:30.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:30.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:30.130368+0000) 2024-03-06T12:34:31.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:31.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:31.130712+0000) 2024-03-06T12:34:32.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:32.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:32.130960+0000) 2024-03-06T12:34:33.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:33.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:33.131207+0000) 2024-03-06T12:34:34.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:34.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:34.131532+0000) 2024-03-06T12:34:35.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:35.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:35.131855+0000) 2024-03-06T12:34:36.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:36.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:36.132143+0000) 2024-03-06T12:34:37.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:37.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:37.132457+0000) 2024-03-06T12:34:38.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:38.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:38.132790+0000) 2024-03-06T12:34:39.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:39.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:39.133116+0000) 2024-03-06T12:34:40.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:40.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:40.133375+0000) 2024-03-06T12:34:41.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:41.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:41.133715+0000) 2024-03-06T12:34:42.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:42.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:42.134017+0000) 2024-03-06T12:34:43.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:43.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:43.134327+0000) 2024-03-06T12:34:44.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:44.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:44.134668+0000) 2024-03-06T12:34:45.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:45.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:45.134998+0000) 2024-03-06T12:34:46.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:46.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:46.135284+0000) 2024-03-06T12:34:47.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:47.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:47.135580+0000) 2024-03-06T12:34:48.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:48.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:48.135877+0000) 2024-03-06T12:34:49.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:49.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:49.136177+0000) 2024-03-06T12:34:50.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:50.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:50.136478+0000) 2024-03-06T12:34:50.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:50.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:51.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:51.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:51.136813+0000) 2024-03-06T12:34:52.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:52.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:52.137116+0000) 2024-03-06T12:34:53.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:53.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:53.137436+0000) 2024-03-06T12:34:54.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:54.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:54.137768+0000) 2024-03-06T12:34:55.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:55.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:55.138062+0000) 2024-03-06T12:34:56.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:56.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:56.138319+0000) 2024-03-06T12:34:57.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:57.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:57.138576+0000) 2024-03-06T12:34:58.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:58.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:58.138824+0000) 2024-03-06T12:34:59.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:34:59.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:59.139032+0000) 2024-03-06T12:35:00.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:00.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:00.139285+0000) 2024-03-06T12:35:01.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:01.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:01.139482+0000) 2024-03-06T12:35:02.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:02.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:02.139731+0000) 2024-03-06T12:35:03.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:03.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:03.139996+0000) 2024-03-06T12:35:04.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:04.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:04.140248+0000) 2024-03-06T12:35:05.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:05.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:05.140514+0000) 2024-03-06T12:35:06.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:06.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:06.140760+0000) 2024-03-06T12:35:07.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:07.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:07.141018+0000) 2024-03-06T12:35:08.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:08.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:08.141257+0000) 2024-03-06T12:35:09.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:09.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:09.141527+0000) 2024-03-06T12:35:10.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:10.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:10.141788+0000) 2024-03-06T12:35:11.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:11.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:11.142049+0000) 2024-03-06T12:35:12.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:12.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:12.142234+0000) 2024-03-06T12:35:13.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:13.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:13.142500+0000) 2024-03-06T12:35:14.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:14.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:14.142755+0000) 2024-03-06T12:35:15.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:15.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:15.143010+0000) 2024-03-06T12:35:16.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:16.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:16.143248+0000) 2024-03-06T12:35:17.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:17.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:17.143519+0000) 2024-03-06T12:35:18.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:18.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:18.143762+0000) 2024-03-06T12:35:19.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:19.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:19.144014+0000) 2024-03-06T12:35:20.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:20.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:20.144261+0000) 2024-03-06T12:35:21.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:21.129 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:21.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:21.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:21.144497+0000) 2024-03-06T12:35:22.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:22.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:22.144799+0000) 2024-03-06T12:35:23.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:23.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:23.145064+0000) 2024-03-06T12:35:24.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:24.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:24.145294+0000) 2024-03-06T12:35:25.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:25.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:25.145539+0000) 2024-03-06T12:35:26.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:26.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:26.145812+0000) 2024-03-06T12:35:27.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:27.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:27.146072+0000) 2024-03-06T12:35:28.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:28.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:28.146322+0000) 2024-03-06T12:35:29.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:29.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:29.146571+0000) 2024-03-06T12:35:30.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:30.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:30.146805+0000) 2024-03-06T12:35:31.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:31.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:31.147006+0000) 2024-03-06T12:35:32.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:32.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:32.147242+0000) 2024-03-06T12:35:33.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:33.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:33.147501+0000) 2024-03-06T12:35:34.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:34.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:34.147741+0000) 2024-03-06T12:35:35.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:35.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:35.147981+0000) 2024-03-06T12:35:36.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:36.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:36.148231+0000) 2024-03-06T12:35:37.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:37.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:37.148494+0000) 2024-03-06T12:35:38.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:38.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:38.148755+0000) 2024-03-06T12:35:39.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:39.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:39.149016+0000) 2024-03-06T12:35:40.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:40.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:40.149264+0000) 2024-03-06T12:35:41.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:41.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:41.149524+0000) 2024-03-06T12:35:42.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:42.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:42.149779+0000) 2024-03-06T12:35:43.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:43.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:43.150057+0000) 2024-03-06T12:35:44.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:44.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:44.150311+0000) 2024-03-06T12:35:45.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:45.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:45.150550+0000) 2024-03-06T12:35:46.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:46.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:46.150793+0000) 2024-03-06T12:35:47.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:47.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:47.151031+0000) 2024-03-06T12:35:48.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:48.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:48.151272+0000) 2024-03-06T12:35:49.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:49.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:49.151534+0000) 2024-03-06T12:35:50.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:50.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:50.151784+0000) 2024-03-06T12:35:51.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:51.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:51.151981+0000) 2024-03-06T12:35:51.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:51.275 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:52.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:52.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:52.152234+0000) 2024-03-06T12:35:53.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:53.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:53.152498+0000) 2024-03-06T12:35:54.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:54.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:54.152745+0000) 2024-03-06T12:35:55.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:55.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:55.153005+0000) 2024-03-06T12:35:56.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:56.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:56.153258+0000) 2024-03-06T12:35:57.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:57.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:57.153515+0000) 2024-03-06T12:35:58.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:58.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:58.153773+0000) 2024-03-06T12:35:59.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:35:59.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:59.154044+0000) 2024-03-06T12:36:00.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:00.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:00.154281+0000) 2024-03-06T12:36:01.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:01.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:01.154543+0000) 2024-03-06T12:36:02.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:02.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:02.154828+0000) 2024-03-06T12:36:03.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:03.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:03.155030+0000) 2024-03-06T12:36:04.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:04.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:04.155225+0000) 2024-03-06T12:36:05.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:05.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:05.155457+0000) 2024-03-06T12:36:06.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:06.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:06.155725+0000) 2024-03-06T12:36:07.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:07.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:07.155972+0000) 2024-03-06T12:36:08.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:08.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:08.156217+0000) 2024-03-06T12:36:09.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:09.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:09.156479+0000) 2024-03-06T12:36:10.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:10.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:10.156753+0000) 2024-03-06T12:36:11.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:11.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:11.156995+0000) 2024-03-06T12:36:12.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:12.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:12.157236+0000) 2024-03-06T12:36:13.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:13.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:13.157496+0000) 2024-03-06T12:36:14.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:14.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:14.157753+0000) 2024-03-06T12:36:15.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:15.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:15.158008+0000) 2024-03-06T12:36:16.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:16.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:16.158274+0000) 2024-03-06T12:36:17.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:17.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:17.158515+0000) 2024-03-06T12:36:18.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:18.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:18.158749+0000) 2024-03-06T12:36:19.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:19.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:19.159006+0000) 2024-03-06T12:36:20.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:20.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:20.159195+0000) 2024-03-06T12:36:21.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:21.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:21.159456+0000) 2024-03-06T12:36:21.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:21.428 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:22.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:22.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:22.159748+0000) 2024-03-06T12:36:23.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:23.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:23.160012+0000) 2024-03-06T12:36:24.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:24.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:24.160266+0000) 2024-03-06T12:36:25.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:25.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:25.160509+0000) 2024-03-06T12:36:26.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:26.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:26.160762+0000) 2024-03-06T12:36:27.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:27.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:27.161017+0000) 2024-03-06T12:36:28.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:28.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:28.161267+0000) 2024-03-06T12:36:29.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:29.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:29.161520+0000) 2024-03-06T12:36:30.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:30.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:30.161750+0000) 2024-03-06T12:36:31.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:31.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:31.162011+0000) 2024-03-06T12:36:32.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:32.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:32.162262+0000) 2024-03-06T12:36:33.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:33.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:33.162499+0000) 2024-03-06T12:36:34.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:34.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:34.162760+0000) 2024-03-06T12:36:35.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:35.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:35.162994+0000) 2024-03-06T12:36:36.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:36.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:36.163230+0000) 2024-03-06T12:36:37.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:37.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:37.163478+0000) 2024-03-06T12:36:38.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:38.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:38.163727+0000) 2024-03-06T12:36:39.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:39.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:39.164002+0000) 2024-03-06T12:36:40.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:40.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:40.164263+0000) 2024-03-06T12:36:41.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:41.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:41.164465+0000) 2024-03-06T12:36:42.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:42.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:42.164676+0000) 2024-03-06T12:36:43.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:43.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:43.164937+0000) 2024-03-06T12:36:44.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:44.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:44.165190+0000) 2024-03-06T12:36:45.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:45.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:45.165435+0000) 2024-03-06T12:36:46.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:46.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:46.165738+0000) 2024-03-06T12:36:47.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:47.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:47.165983+0000) 2024-03-06T12:36:48.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:48.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:48.166175+0000) 2024-03-06T12:36:49.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:49.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:49.166439+0000) 2024-03-06T12:36:50.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:50.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:50.166736+0000) 2024-03-06T12:36:51.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:51.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:51.167001+0000) 2024-03-06T12:36:51.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:51.567 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:52.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:52.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:52.167271+0000) 2024-03-06T12:36:53.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:53.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:53.167506+0000) 2024-03-06T12:36:54.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:54.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:54.167754+0000) 2024-03-06T12:36:55.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:55.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:55.168017+0000) 2024-03-06T12:36:56.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:56.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:56.168254+0000) 2024-03-06T12:36:57.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:57.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:57.168519+0000) 2024-03-06T12:36:58.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:58.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:58.168756+0000) 2024-03-06T12:36:59.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:36:59.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:59.169022+0000) 2024-03-06T12:37:00.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:00.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:00.169198+0000) 2024-03-06T12:37:01.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:01.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:01.169476+0000) 2024-03-06T12:37:02.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:02.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:02.169748+0000) 2024-03-06T12:37:03.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:03.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:03.169981+0000) 2024-03-06T12:37:04.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:04.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:04.170288+0000) 2024-03-06T12:37:05.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:05.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:05.170518+0000) 2024-03-06T12:37:06.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:06.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:06.170782+0000) 2024-03-06T12:37:07.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:07.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:07.170983+0000) 2024-03-06T12:37:08.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:08.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:08.171218+0000) 2024-03-06T12:37:09.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:09.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:09.171479+0000) 2024-03-06T12:37:10.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:10.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:10.171753+0000) 2024-03-06T12:37:11.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:11.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:11.172015+0000) 2024-03-06T12:37:12.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:12.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:12.172251+0000) 2024-03-06T12:37:13.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:13.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:13.172510+0000) 2024-03-06T12:37:14.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:14.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:14.172773+0000) 2024-03-06T12:37:15.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:15.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:15.173034+0000) 2024-03-06T12:37:16.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:16.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:16.173270+0000) 2024-03-06T12:37:17.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:17.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:17.173511+0000) 2024-03-06T12:37:18.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:18.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:18.173753+0000) 2024-03-06T12:37:19.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:19.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:19.174013+0000) 2024-03-06T12:37:20.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:20.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:20.174274+0000) 2024-03-06T12:37:21.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:21.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:21.174533+0000) 2024-03-06T12:37:21.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:21.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:22.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:22.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:22.174767+0000) 2024-03-06T12:37:23.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:23.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:23.175028+0000) 2024-03-06T12:37:24.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:24.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:24.175315+0000) 2024-03-06T12:37:25.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:25.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:25.175620+0000) 2024-03-06T12:37:26.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:26.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:26.176013+0000) 2024-03-06T12:37:27.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:27.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:27.176337+0000) 2024-03-06T12:37:28.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:28.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:28.176667+0000) 2024-03-06T12:37:29.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:29.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:29.177022+0000) 2024-03-06T12:37:30.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:30.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:30.177316+0000) 2024-03-06T12:37:31.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:31.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:31.177554+0000) 2024-03-06T12:37:32.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:32.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:32.177834+0000) 2024-03-06T12:37:33.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:33.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:33.178144+0000) 2024-03-06T12:37:34.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:34.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:34.178433+0000) 2024-03-06T12:37:35.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:35.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:35.178781+0000) 2024-03-06T12:37:36.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:36.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:36.179060+0000) 2024-03-06T12:37:37.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:37.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:37.179350+0000) 2024-03-06T12:37:38.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:38.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:38.179619+0000) 2024-03-06T12:37:39.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:39.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:39.180006+0000) 2024-03-06T12:37:40.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:40.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:40.180287+0000) 2024-03-06T12:37:41.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:41.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:41.180619+0000) 2024-03-06T12:37:42.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:42.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:42.180981+0000) 2024-03-06T12:37:43.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:43.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:43.181278+0000) 2024-03-06T12:37:44.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:44.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:44.181583+0000) 2024-03-06T12:37:45.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:45.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:45.181852+0000) 2024-03-06T12:37:46.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:46.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:46.182135+0000) 2024-03-06T12:37:47.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:47.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:47.182419+0000) 2024-03-06T12:37:48.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:48.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:48.182765+0000) 2024-03-06T12:37:49.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:49.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:49.183080+0000) 2024-03-06T12:37:50.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:50.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:50.183370+0000) 2024-03-06T12:37:51.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:51.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:51.183699+0000) 2024-03-06T12:37:51.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:51.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:52.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:52.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:52.184040+0000) 2024-03-06T12:37:53.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:53.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:53.184344+0000) 2024-03-06T12:37:54.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:54.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:54.184693+0000) 2024-03-06T12:37:55.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:55.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:55.185008+0000) 2024-03-06T12:37:56.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:56.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:56.185319+0000) 2024-03-06T12:37:57.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:57.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:57.185672+0000) 2024-03-06T12:37:58.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:58.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:58.185989+0000) 2024-03-06T12:37:59.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:37:59.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:59.186301+0000) 2024-03-06T12:38:00.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:00.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:00.186606+0000) 2024-03-06T12:38:01.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:01.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:01.186954+0000) 2024-03-06T12:38:02.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:02.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:02.187259+0000) 2024-03-06T12:38:03.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:03.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:03.187569+0000) 2024-03-06T12:38:04.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:04.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:04.187887+0000) 2024-03-06T12:38:05.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:05.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:05.188210+0000) 2024-03-06T12:38:06.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:06.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:06.188503+0000) 2024-03-06T12:38:07.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:07.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:07.188820+0000) 2024-03-06T12:38:08.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:08.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:08.189118+0000) 2024-03-06T12:38:09.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:09.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:09.189449+0000) 2024-03-06T12:38:10.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:10.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:10.189786+0000) 2024-03-06T12:38:11.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:11.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:11.190093+0000) 2024-03-06T12:38:12.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:12.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:12.190411+0000) 2024-03-06T12:38:13.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:13.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:13.190729+0000) 2024-03-06T12:38:14.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:14.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:14.191043+0000) 2024-03-06T12:38:15.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:15.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:15.191359+0000) 2024-03-06T12:38:16.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:16.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:16.191701+0000) 2024-03-06T12:38:17.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:17.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:17.191987+0000) 2024-03-06T12:38:18.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:18.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:18.192252+0000) 2024-03-06T12:38:19.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:19.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:19.192518+0000) 2024-03-06T12:38:20.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:20.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:20.192765+0000) 2024-03-06T12:38:21.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:21.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:21.193006+0000) 2024-03-06T12:38:22.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:22.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:22.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:22.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:22.193266+0000) 2024-03-06T12:38:23.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:23.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:23.193513+0000) 2024-03-06T12:38:24.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:24.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:24.193772+0000) 2024-03-06T12:38:25.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:25.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:25.194011+0000) 2024-03-06T12:38:26.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:26.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:26.194306+0000) 2024-03-06T12:38:27.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:27.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:27.194539+0000) 2024-03-06T12:38:28.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:28.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:28.194822+0000) 2024-03-06T12:38:29.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:29.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:29.195122+0000) 2024-03-06T12:38:30.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:30.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:30.195424+0000) 2024-03-06T12:38:31.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:31.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:31.195785+0000) 2024-03-06T12:38:32.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:32.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:32.196066+0000) 2024-03-06T12:38:33.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:33.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:33.196332+0000) 2024-03-06T12:38:34.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:34.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:34.196584+0000) 2024-03-06T12:38:35.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:35.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:35.196841+0000) 2024-03-06T12:38:36.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:36.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:36.197081+0000) 2024-03-06T12:38:37.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:37.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:37.197341+0000) 2024-03-06T12:38:38.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:38.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:38.197598+0000) 2024-03-06T12:38:39.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:39.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:39.197924+0000) 2024-03-06T12:38:40.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:40.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:40.198180+0000) 2024-03-06T12:38:41.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:41.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:41.198446+0000) 2024-03-06T12:38:42.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:42.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:42.198749+0000) 2024-03-06T12:38:43.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:43.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:43.199014+0000) 2024-03-06T12:38:44.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:44.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:44.199267+0000) 2024-03-06T12:38:45.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:45.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:45.199565+0000) 2024-03-06T12:38:46.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:46.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:46.199823+0000) 2024-03-06T12:38:47.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:47.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:47.200080+0000) 2024-03-06T12:38:48.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:48.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:48.200259+0000) 2024-03-06T12:38:49.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:49.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:49.200539+0000) 2024-03-06T12:38:50.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:50.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:50.200796+0000) 2024-03-06T12:38:51.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:51.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:51.201098+0000) 2024-03-06T12:38:52.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:52.172 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:52.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:52.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:52.201384+0000) 2024-03-06T12:38:53.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:53.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:53.201727+0000) 2024-03-06T12:38:54.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:54.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:54.202016+0000) 2024-03-06T12:38:55.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:55.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:55.202329+0000) 2024-03-06T12:38:56.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:56.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:56.202577+0000) 2024-03-06T12:38:57.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:57.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:57.202839+0000) 2024-03-06T12:38:58.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:58.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:58.203091+0000) 2024-03-06T12:38:59.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:38:59.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:59.203343+0000) 2024-03-06T12:39:00.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:00.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:00.203571+0000) 2024-03-06T12:39:01.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:01.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:01.203832+0000) 2024-03-06T12:39:02.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:02.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:02.204075+0000) 2024-03-06T12:39:03.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:03.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:03.204314+0000) 2024-03-06T12:39:04.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:04.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:04.204567+0000) 2024-03-06T12:39:05.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:05.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:05.204810+0000) 2024-03-06T12:39:06.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:06.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:06.205067+0000) 2024-03-06T12:39:07.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:07.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:07.205319+0000) 2024-03-06T12:39:08.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:08.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:08.205563+0000) 2024-03-06T12:39:09.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:09.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:09.205801+0000) 2024-03-06T12:39:10.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:10.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:10.206044+0000) 2024-03-06T12:39:11.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:11.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:11.206242+0000) 2024-03-06T12:39:12.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:12.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:12.206486+0000) 2024-03-06T12:39:13.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:13.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:13.206772+0000) 2024-03-06T12:39:14.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:14.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:14.207009+0000) 2024-03-06T12:39:15.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:15.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:15.207268+0000) 2024-03-06T12:39:16.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:16.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:16.207512+0000) 2024-03-06T12:39:17.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:17.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:17.207755+0000) 2024-03-06T12:39:18.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:18.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:18.208008+0000) 2024-03-06T12:39:19.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:19.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:19.208254+0000) 2024-03-06T12:39:20.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:20.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:20.208499+0000) 2024-03-06T12:39:21.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:21.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:21.208775+0000) 2024-03-06T12:39:22.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:22.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:22.209028+0000) 2024-03-06T12:39:22.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:22.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:23.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:23.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:23.209286+0000) 2024-03-06T12:39:24.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:24.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:24.209543+0000) 2024-03-06T12:39:25.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:25.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:25.209808+0000) 2024-03-06T12:39:26.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:26.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:26.210073+0000) 2024-03-06T12:39:27.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:27.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:27.210334+0000) 2024-03-06T12:39:28.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:28.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:28.210578+0000) 2024-03-06T12:39:29.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:29.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:29.210830+0000) 2024-03-06T12:39:30.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:30.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:30.211069+0000) 2024-03-06T12:39:31.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:31.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:31.211329+0000) 2024-03-06T12:39:32.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:32.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:32.211564+0000) 2024-03-06T12:39:33.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:33.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:33.211874+0000) 2024-03-06T12:39:34.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:34.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:34.212109+0000) 2024-03-06T12:39:35.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:35.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:35.212348+0000) 2024-03-06T12:39:36.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:36.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:36.212601+0000) 2024-03-06T12:39:37.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:37.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:37.212905+0000) 2024-03-06T12:39:38.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:38.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:38.213139+0000) 2024-03-06T12:39:39.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:39.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:39.213416+0000) 2024-03-06T12:39:40.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:40.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:40.213593+0000) 2024-03-06T12:39:41.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:41.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:41.213831+0000) 2024-03-06T12:39:42.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:42.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:42.214082+0000) 2024-03-06T12:39:43.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:43.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:43.214371+0000) 2024-03-06T12:39:44.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:44.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:44.214677+0000) 2024-03-06T12:39:45.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:45.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:45.214960+0000) 2024-03-06T12:39:46.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:46.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:46.215252+0000) 2024-03-06T12:39:47.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:47.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:47.215529+0000) 2024-03-06T12:39:48.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:48.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:48.215796+0000) 2024-03-06T12:39:49.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:49.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:49.216079+0000) 2024-03-06T12:39:50.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:50.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:50.216339+0000) 2024-03-06T12:39:51.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:51.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:51.216620+0000) 2024-03-06T12:39:52.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:52.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:52.216876+0000) 2024-03-06T12:39:52.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:52.475 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:53.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:53.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:53.217171+0000) 2024-03-06T12:39:54.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:54.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:54.217429+0000) 2024-03-06T12:39:55.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:55.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:55.217727+0000) 2024-03-06T12:39:56.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:56.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:56.217993+0000) 2024-03-06T12:39:57.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:57.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:57.218253+0000) 2024-03-06T12:39:58.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:58.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:58.218503+0000) 2024-03-06T12:39:59.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:39:59.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:59.218815+0000) 2024-03-06T12:40:00.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:00.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:00.219085+0000) 2024-03-06T12:40:01.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:01.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:01.219363+0000) 2024-03-06T12:40:02.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:02.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:02.219575+0000) 2024-03-06T12:40:03.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:03.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:03.219858+0000) 2024-03-06T12:40:04.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:04.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:04.220070+0000) 2024-03-06T12:40:05.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:05.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:05.220332+0000) 2024-03-06T12:40:06.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:06.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:06.220599+0000) 2024-03-06T12:40:07.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:07.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:07.220926+0000) 2024-03-06T12:40:08.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:08.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:08.221178+0000) 2024-03-06T12:40:09.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:09.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:09.221481+0000) 2024-03-06T12:40:10.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:10.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:10.221738+0000) 2024-03-06T12:40:11.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:11.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:11.222012+0000) 2024-03-06T12:40:12.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:12.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:12.222277+0000) 2024-03-06T12:40:13.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:13.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:13.222549+0000) 2024-03-06T12:40:14.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:14.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:14.222821+0000) 2024-03-06T12:40:15.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:15.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:15.223086+0000) 2024-03-06T12:40:16.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:16.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:16.223348+0000) 2024-03-06T12:40:17.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:17.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:17.223685+0000) 2024-03-06T12:40:18.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:18.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:18.223978+0000) 2024-03-06T12:40:19.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:19.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:19.224263+0000) 2024-03-06T12:40:20.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:20.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:20.224532+0000) 2024-03-06T12:40:21.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:21.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:21.224809+0000) 2024-03-06T12:40:22.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:22.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:22.225061+0000) 2024-03-06T12:40:22.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:22.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:23.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:23.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:23.225324+0000) 2024-03-06T12:40:24.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:24.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:24.225571+0000) 2024-03-06T12:40:25.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:25.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:25.225816+0000) 2024-03-06T12:40:26.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:26.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:26.226065+0000) 2024-03-06T12:40:27.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:27.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:27.226315+0000) 2024-03-06T12:40:28.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:28.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:28.226562+0000) 2024-03-06T12:40:29.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:29.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:29.226823+0000) 2024-03-06T12:40:30.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:30.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:30.227072+0000) 2024-03-06T12:40:31.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:31.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:31.227321+0000) 2024-03-06T12:40:32.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:32.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:32.227673+0000) 2024-03-06T12:40:33.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:33.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:33.227958+0000) 2024-03-06T12:40:34.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:34.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:34.228238+0000) 2024-03-06T12:40:35.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:35.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:35.228517+0000) 2024-03-06T12:40:36.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:36.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:36.228774+0000) 2024-03-06T12:40:37.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:37.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:37.229059+0000) 2024-03-06T12:40:38.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:38.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:38.229299+0000) 2024-03-06T12:40:39.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:39.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:39.229594+0000) 2024-03-06T12:40:40.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:40.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:40.229919+0000) 2024-03-06T12:40:41.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:41.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:41.230197+0000) 2024-03-06T12:40:42.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:42.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:42.230470+0000) 2024-03-06T12:40:43.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:43.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:43.230745+0000) 2024-03-06T12:40:44.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:44.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:44.231015+0000) 2024-03-06T12:40:45.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:45.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:45.231274+0000) 2024-03-06T12:40:46.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:46.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:46.231520+0000) 2024-03-06T12:40:47.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:47.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:47.231825+0000) 2024-03-06T12:40:48.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:48.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:48.232106+0000) 2024-03-06T12:40:49.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:49.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:49.232387+0000) 2024-03-06T12:40:50.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:50.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:50.232678+0000) 2024-03-06T12:40:51.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:51.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:51.232943+0000) 2024-03-06T12:40:52.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:52.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:52.233187+0000) 2024-03-06T12:40:52.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:52.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:53.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:53.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:53.233428+0000) 2024-03-06T12:40:54.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:54.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:54.233721+0000) 2024-03-06T12:40:55.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:55.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:55.233973+0000) 2024-03-06T12:40:56.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:56.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:56.234211+0000) 2024-03-06T12:40:57.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:57.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:57.234469+0000) 2024-03-06T12:40:58.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:58.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:58.234771+0000) 2024-03-06T12:40:59.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:40:59.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:59.235052+0000) 2024-03-06T12:41:00.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:00.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:00.235312+0000) 2024-03-06T12:41:01.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:01.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:01.235567+0000) 2024-03-06T12:41:02.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:02.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:02.235816+0000) 2024-03-06T12:41:03.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:03.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:03.236056+0000) 2024-03-06T12:41:04.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:04.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:04.236318+0000) 2024-03-06T12:41:05.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:05.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:05.236563+0000) 2024-03-06T12:41:06.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:06.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:06.236797+0000) 2024-03-06T12:41:07.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:07.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:07.237056+0000) 2024-03-06T12:41:08.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:08.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:08.237237+0000) 2024-03-06T12:41:09.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:09.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:09.237508+0000) 2024-03-06T12:41:10.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:10.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:10.237760+0000) 2024-03-06T12:41:11.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:11.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:11.238014+0000) 2024-03-06T12:41:12.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:12.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:12.238246+0000) 2024-03-06T12:41:13.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:13.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:13.238507+0000) 2024-03-06T12:41:14.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:14.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:14.238767+0000) 2024-03-06T12:41:15.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:15.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:15.239029+0000) 2024-03-06T12:41:16.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:16.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:16.239277+0000) 2024-03-06T12:41:17.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:17.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:17.239543+0000) 2024-03-06T12:41:18.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:18.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:18.239790+0000) 2024-03-06T12:41:19.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:19.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:19.240038+0000) 2024-03-06T12:41:20.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:20.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:20.240326+0000) 2024-03-06T12:41:21.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:21.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:21.240567+0000) 2024-03-06T12:41:22.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:22.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:22.240804+0000) 2024-03-06T12:41:22.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:22.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:23.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:23.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:23.241064+0000) 2024-03-06T12:41:24.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:24.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:24.241313+0000) 2024-03-06T12:41:25.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:25.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:25.241577+0000) 2024-03-06T12:41:26.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:26.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:26.241808+0000) 2024-03-06T12:41:27.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:27.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:27.242049+0000) 2024-03-06T12:41:28.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:28.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:28.242290+0000) 2024-03-06T12:41:29.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:29.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:29.242538+0000) 2024-03-06T12:41:30.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:30.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:30.242772+0000) 2024-03-06T12:41:31.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:31.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:31.243031+0000) 2024-03-06T12:41:32.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:32.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:32.243282+0000) 2024-03-06T12:41:33.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:33.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:33.243542+0000) 2024-03-06T12:41:34.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:34.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:34.243795+0000) 2024-03-06T12:41:35.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:35.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:35.244038+0000) 2024-03-06T12:41:36.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:36.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:36.244229+0000) 2024-03-06T12:41:37.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:37.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:37.244490+0000) 2024-03-06T12:41:38.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:38.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:38.244736+0000) 2024-03-06T12:41:39.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:39.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:39.245002+0000) 2024-03-06T12:41:40.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:40.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:40.245254+0000) 2024-03-06T12:41:41.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:41.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:41.245504+0000) 2024-03-06T12:41:42.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:42.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:42.245746+0000) 2024-03-06T12:41:43.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:43.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:43.246010+0000) 2024-03-06T12:41:44.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:44.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:44.246264+0000) 2024-03-06T12:41:45.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:45.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:45.246527+0000) 2024-03-06T12:41:46.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:46.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:46.246786+0000) 2024-03-06T12:41:47.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:47.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:47.247048+0000) 2024-03-06T12:41:48.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:48.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:48.247282+0000) 2024-03-06T12:41:49.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:49.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:49.247520+0000) 2024-03-06T12:41:50.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:50.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:50.247753+0000) 2024-03-06T12:41:51.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:51.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:51.247979+0000) 2024-03-06T12:41:52.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:52.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:52.248220+0000) 2024-03-06T12:41:53.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:53.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:53.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:53.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:53.248471+0000) 2024-03-06T12:41:54.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:54.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:54.248769+0000) 2024-03-06T12:41:55.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:55.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:55.249051+0000) 2024-03-06T12:41:56.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:56.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:56.249298+0000) 2024-03-06T12:41:57.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:57.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:57.249584+0000) 2024-03-06T12:41:58.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:58.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:58.249899+0000) 2024-03-06T12:41:59.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:41:59.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:59.250172+0000) 2024-03-06T12:42:00.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:00.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:00.250439+0000) 2024-03-06T12:42:01.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:01.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:01.250747+0000) 2024-03-06T12:42:02.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:02.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:02.251070+0000) 2024-03-06T12:42:03.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:03.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:03.251342+0000) 2024-03-06T12:42:04.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:04.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:04.251602+0000) 2024-03-06T12:42:05.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:05.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:05.251933+0000) 2024-03-06T12:42:06.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:06.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:06.252180+0000) 2024-03-06T12:42:07.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:07.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:07.252456+0000) 2024-03-06T12:42:08.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:08.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:08.252755+0000) 2024-03-06T12:42:09.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:09.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:09.253051+0000) 2024-03-06T12:42:10.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:10.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:10.253322+0000) 2024-03-06T12:42:11.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:11.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:11.253597+0000) 2024-03-06T12:42:12.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:12.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:12.253885+0000) 2024-03-06T12:42:13.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:13.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:13.254166+0000) 2024-03-06T12:42:14.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:14.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:14.254437+0000) 2024-03-06T12:42:15.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:15.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:15.254753+0000) 2024-03-06T12:42:16.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:16.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:16.255066+0000) 2024-03-06T12:42:17.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:17.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:17.255370+0000) 2024-03-06T12:42:18.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:18.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:18.255712+0000) 2024-03-06T12:42:19.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:19.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:19.256030+0000) 2024-03-06T12:42:20.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:20.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:20.256348+0000) 2024-03-06T12:42:21.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:21.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:21.256683+0000) 2024-03-06T12:42:22.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:22.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:22.256988+0000) 2024-03-06T12:42:23.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:23.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:23.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:23.257279+0000) 2024-03-06T12:42:23.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:24.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:24.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:24.257547+0000) 2024-03-06T12:42:25.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:25.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:25.257801+0000) 2024-03-06T12:42:26.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:26.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:26.258051+0000) 2024-03-06T12:42:27.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:27.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:27.258295+0000) 2024-03-06T12:42:28.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:28.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:28.258527+0000) 2024-03-06T12:42:29.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:29.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:29.258805+0000) 2024-03-06T12:42:30.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:30.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:30.259006+0000) 2024-03-06T12:42:31.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:31.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:31.259263+0000) 2024-03-06T12:42:32.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:32.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:32.259515+0000) 2024-03-06T12:42:33.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:33.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:33.259787+0000) 2024-03-06T12:42:34.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:34.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:34.260022+0000) 2024-03-06T12:42:35.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:35.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:35.260282+0000) 2024-03-06T12:42:36.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:36.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:36.260529+0000) 2024-03-06T12:42:37.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:37.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:37.260782+0000) 2024-03-06T12:42:38.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:38.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:38.261037+0000) 2024-03-06T12:42:39.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:39.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:39.261312+0000) 2024-03-06T12:42:40.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:40.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:40.261570+0000) 2024-03-06T12:42:41.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:41.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:41.261812+0000) 2024-03-06T12:42:42.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:42.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:42.262063+0000) 2024-03-06T12:42:43.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:43.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:43.262323+0000) 2024-03-06T12:42:44.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:44.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:44.262557+0000) 2024-03-06T12:42:45.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:45.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:45.262834+0000) 2024-03-06T12:42:46.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:46.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:46.263022+0000) 2024-03-06T12:42:47.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:47.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:47.263277+0000) 2024-03-06T12:42:48.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:48.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:48.263520+0000) 2024-03-06T12:42:49.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:49.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:49.263778+0000) 2024-03-06T12:42:50.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:50.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:50.264079+0000) 2024-03-06T12:42:51.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:51.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:51.264341+0000) 2024-03-06T12:42:52.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:52.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:52.264599+0000) 2024-03-06T12:42:53.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:53.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:53.264837+0000) 2024-03-06T12:42:53.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:53.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:54.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:54.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:54.265085+0000) 2024-03-06T12:42:55.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:55.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:55.265350+0000) 2024-03-06T12:42:56.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:56.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:56.265602+0000) 2024-03-06T12:42:57.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:57.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:57.265928+0000) 2024-03-06T12:42:58.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:58.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:58.266181+0000) 2024-03-06T12:42:59.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:42:59.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:59.266429+0000) 2024-03-06T12:43:00.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:00.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:00.266617+0000) 2024-03-06T12:43:01.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:01.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:01.266944+0000) 2024-03-06T12:43:02.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:02.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:02.267191+0000) 2024-03-06T12:43:03.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:03.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:03.267485+0000) 2024-03-06T12:43:04.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:04.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:04.267766+0000) 2024-03-06T12:43:05.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:05.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:05.268035+0000) 2024-03-06T12:43:06.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:06.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:06.268282+0000) 2024-03-06T12:43:07.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:07.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:07.268560+0000) 2024-03-06T12:43:08.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:08.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:08.268813+0000) 2024-03-06T12:43:09.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:09.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:09.269081+0000) 2024-03-06T12:43:10.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:10.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:10.269340+0000) 2024-03-06T12:43:11.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:11.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:11.269606+0000) 2024-03-06T12:43:12.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:12.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:12.269897+0000) 2024-03-06T12:43:13.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:13.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:13.270157+0000) 2024-03-06T12:43:14.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:14.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:14.270394+0000) 2024-03-06T12:43:15.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:15.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:15.270680+0000) 2024-03-06T12:43:16.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:16.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:16.270929+0000) 2024-03-06T12:43:17.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:17.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:17.271137+0000) 2024-03-06T12:43:18.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:18.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:18.271389+0000) 2024-03-06T12:43:19.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:19.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:19.271716+0000) 2024-03-06T12:43:20.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:20.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:20.271970+0000) 2024-03-06T12:43:21.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:21.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:21.272235+0000) 2024-03-06T12:43:22.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:22.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:22.272499+0000) 2024-03-06T12:43:23.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:23.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:23.272761+0000) 2024-03-06T12:43:23.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:23.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:24.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:24.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:24.273004+0000) 2024-03-06T12:43:25.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:25.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:25.273247+0000) 2024-03-06T12:43:26.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:26.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:26.273483+0000) 2024-03-06T12:43:27.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:27.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:27.273772+0000) 2024-03-06T12:43:28.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:28.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:28.274014+0000) 2024-03-06T12:43:29.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:29.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:29.274258+0000) 2024-03-06T12:43:30.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:30.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:30.274515+0000) 2024-03-06T12:43:31.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:31.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:31.274756+0000) 2024-03-06T12:43:32.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:32.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:32.274996+0000) 2024-03-06T12:43:33.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:33.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:33.275240+0000) 2024-03-06T12:43:34.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:34.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:34.275479+0000) 2024-03-06T12:43:35.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:35.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:35.275741+0000) 2024-03-06T12:43:36.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:36.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:36.276014+0000) 2024-03-06T12:43:37.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:37.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:37.276337+0000) 2024-03-06T12:43:38.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:38.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:38.276685+0000) 2024-03-06T12:43:39.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:39.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:39.277001+0000) 2024-03-06T12:43:40.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:40.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:40.277310+0000) 2024-03-06T12:43:41.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:41.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:41.277613+0000) 2024-03-06T12:43:42.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:42.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:42.277952+0000) 2024-03-06T12:43:43.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:43.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:43.278246+0000) 2024-03-06T12:43:44.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:44.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:44.278538+0000) 2024-03-06T12:43:45.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:45.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:45.278814+0000) 2024-03-06T12:43:46.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:46.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:46.279134+0000) 2024-03-06T12:43:47.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:47.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:47.279435+0000) 2024-03-06T12:43:48.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:48.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:48.279779+0000) 2024-03-06T12:43:49.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:49.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:49.280087+0000) 2024-03-06T12:43:50.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:50.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:50.280361+0000) 2024-03-06T12:43:51.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:51.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:51.280664+0000) 2024-03-06T12:43:52.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:52.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:52.280912+0000) 2024-03-06T12:43:53.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:53.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:53.281150+0000) 2024-03-06T12:43:53.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:53.695 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:54.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:54.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:54.281426+0000) 2024-03-06T12:43:55.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:55.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:55.281781+0000) 2024-03-06T12:43:56.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:56.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:56.282048+0000) 2024-03-06T12:43:57.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:57.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:57.282318+0000) 2024-03-06T12:43:58.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:58.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:58.282535+0000) 2024-03-06T12:43:59.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:43:59.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:59.282786+0000) 2024-03-06T12:44:00.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:00.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:00.283016+0000) 2024-03-06T12:44:01.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:01.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:01.283325+0000) 2024-03-06T12:44:02.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:02.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:02.283690+0000) 2024-03-06T12:44:03.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:03.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:03.283993+0000) 2024-03-06T12:44:04.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:04.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:04.284242+0000) 2024-03-06T12:44:05.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:05.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:05.284507+0000) 2024-03-06T12:44:06.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:06.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:06.284755+0000) 2024-03-06T12:44:07.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:07.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:07.285009+0000) 2024-03-06T12:44:08.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:08.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:08.285239+0000) 2024-03-06T12:44:09.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:09.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:09.285489+0000) 2024-03-06T12:44:10.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:10.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:10.285868+0000) 2024-03-06T12:44:11.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:11.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:11.286120+0000) 2024-03-06T12:44:12.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:12.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:12.286362+0000) 2024-03-06T12:44:13.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:13.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:13.286592+0000) 2024-03-06T12:44:14.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:14.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:14.286832+0000) 2024-03-06T12:44:15.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:15.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:15.287098+0000) 2024-03-06T12:44:16.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:16.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:16.287359+0000) 2024-03-06T12:44:17.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:17.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:17.287604+0000) 2024-03-06T12:44:18.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:18.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:18.287872+0000) 2024-03-06T12:44:19.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:19.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:19.288122+0000) 2024-03-06T12:44:20.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:20.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:20.288360+0000) 2024-03-06T12:44:21.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:21.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:21.288676+0000) 2024-03-06T12:44:22.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:22.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:22.288926+0000) 2024-03-06T12:44:23.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:23.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:23.289192+0000) 2024-03-06T12:44:23.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:23.858 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:24.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:24.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:24.289429+0000) 2024-03-06T12:44:25.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:25.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:25.289738+0000) 2024-03-06T12:44:26.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:26.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:26.289990+0000) 2024-03-06T12:44:27.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:27.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:27.290232+0000) 2024-03-06T12:44:28.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:28.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:28.290418+0000) 2024-03-06T12:44:29.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:29.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:29.290731+0000) 2024-03-06T12:44:30.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:30.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:30.290934+0000) 2024-03-06T12:44:31.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:31.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:31.291193+0000) 2024-03-06T12:44:32.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:32.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:32.291442+0000) 2024-03-06T12:44:33.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:33.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:33.291735+0000) 2024-03-06T12:44:34.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:34.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:34.291968+0000) 2024-03-06T12:44:35.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:35.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:35.292235+0000) 2024-03-06T12:44:36.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:36.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:36.292491+0000) 2024-03-06T12:44:37.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:37.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:37.292801+0000) 2024-03-06T12:44:38.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:38.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:38.292997+0000) 2024-03-06T12:44:39.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:39.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:39.293255+0000) 2024-03-06T12:44:40.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:40.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:40.293507+0000) 2024-03-06T12:44:41.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:41.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:41.293767+0000) 2024-03-06T12:44:42.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:42.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:42.294023+0000) 2024-03-06T12:44:43.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:43.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:43.294269+0000) 2024-03-06T12:44:44.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:44.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:44.294521+0000) 2024-03-06T12:44:45.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:45.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:45.294785+0000) 2024-03-06T12:44:46.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:46.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:46.295033+0000) 2024-03-06T12:44:47.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:47.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:47.295295+0000) 2024-03-06T12:44:48.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:48.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:48.295484+0000) 2024-03-06T12:44:49.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:49.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:49.295753+0000) 2024-03-06T12:44:50.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:50.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:50.296014+0000) 2024-03-06T12:44:51.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:51.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:51.296293+0000) 2024-03-06T12:44:52.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:52.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:52.296546+0000) 2024-03-06T12:44:53.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:53.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:53.296776+0000) 2024-03-06T12:44:54.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:54.031 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:54.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:54.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:54.297042+0000) 2024-03-06T12:44:55.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:55.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:55.297313+0000) 2024-03-06T12:44:56.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:56.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:56.297562+0000) 2024-03-06T12:44:57.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:57.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:57.297822+0000) 2024-03-06T12:44:58.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:58.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:58.298076+0000) 2024-03-06T12:44:59.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:44:59.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:59.298345+0000) 2024-03-06T12:45:00.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:00.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:00.298603+0000) 2024-03-06T12:45:01.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:01.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:01.298901+0000) 2024-03-06T12:45:02.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:02.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:02.299142+0000) 2024-03-06T12:45:03.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:03.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:03.299420+0000) 2024-03-06T12:45:04.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:04.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:04.299717+0000) 2024-03-06T12:45:05.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:05.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:05.299985+0000) 2024-03-06T12:45:06.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:06.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:06.300218+0000) 2024-03-06T12:45:07.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:07.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:07.300461+0000) 2024-03-06T12:45:08.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:08.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:08.300738+0000) 2024-03-06T12:45:09.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:09.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:09.301002+0000) 2024-03-06T12:45:10.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:10.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:10.301199+0000) 2024-03-06T12:45:11.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:11.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:11.301462+0000) 2024-03-06T12:45:12.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:12.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:12.301731+0000) 2024-03-06T12:45:13.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:13.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:13.301991+0000) 2024-03-06T12:45:14.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:14.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:14.302225+0000) 2024-03-06T12:45:15.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:15.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:15.302493+0000) 2024-03-06T12:45:16.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:16.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:16.302729+0000) 2024-03-06T12:45:17.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:17.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:17.302970+0000) 2024-03-06T12:45:18.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:18.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:18.303222+0000) 2024-03-06T12:45:19.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:19.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:19.303496+0000) 2024-03-06T12:45:20.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:20.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:20.303767+0000) 2024-03-06T12:45:21.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:21.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:21.304056+0000) 2024-03-06T12:45:22.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:22.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:22.304329+0000) 2024-03-06T12:45:23.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:23.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:23.304563+0000) 2024-03-06T12:45:24.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:24.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:24.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:24.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:24.304821+0000) 2024-03-06T12:45:25.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:25.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:25.305083+0000) 2024-03-06T12:45:26.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:26.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:26.305336+0000) 2024-03-06T12:45:27.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:27.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:27.305578+0000) 2024-03-06T12:45:28.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:28.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:28.305827+0000) 2024-03-06T12:45:29.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:29.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:29.306033+0000) 2024-03-06T12:45:30.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:30.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:30.306392+0000) 2024-03-06T12:45:31.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:31.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:31.306692+0000) 2024-03-06T12:45:32.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:32.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:32.306945+0000) 2024-03-06T12:45:33.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:33.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:33.307182+0000) 2024-03-06T12:45:34.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:34.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:34.307437+0000) 2024-03-06T12:45:35.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:35.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:35.307728+0000) 2024-03-06T12:45:36.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:36.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:36.307974+0000) 2024-03-06T12:45:37.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:37.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:37.308241+0000) 2024-03-06T12:45:38.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:38.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:38.308493+0000) 2024-03-06T12:45:39.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:39.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:39.308738+0000) 2024-03-06T12:45:40.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:40.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:40.308994+0000) 2024-03-06T12:45:41.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:41.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:41.309257+0000) 2024-03-06T12:45:42.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:42.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:42.309495+0000) 2024-03-06T12:45:43.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:43.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:43.309741+0000) 2024-03-06T12:45:44.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:44.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:44.309934+0000) 2024-03-06T12:45:45.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:45.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:45.310180+0000) 2024-03-06T12:45:46.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:46.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:46.310413+0000) 2024-03-06T12:45:47.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:47.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:47.310684+0000) 2024-03-06T12:45:48.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:48.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:48.310920+0000) 2024-03-06T12:45:49.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:49.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:49.311205+0000) 2024-03-06T12:45:50.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:50.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:50.311445+0000) 2024-03-06T12:45:51.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:51.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:51.311742+0000) 2024-03-06T12:45:52.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:52.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:52.312002+0000) 2024-03-06T12:45:53.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:53.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:53.312261+0000) 2024-03-06T12:45:54.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:54.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:54.312514+0000) 2024-03-06T12:45:54.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:54.333 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:55.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:55.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:55.312787+0000) 2024-03-06T12:45:56.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:56.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:56.313040+0000) 2024-03-06T12:45:57.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:57.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:57.313308+0000) 2024-03-06T12:45:58.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:58.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:58.313562+0000) 2024-03-06T12:45:59.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:45:59.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:59.313835+0000) 2024-03-06T12:46:00.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:00.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:00.314078+0000) 2024-03-06T12:46:01.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:01.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:01.314270+0000) 2024-03-06T12:46:02.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:02.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:02.314510+0000) 2024-03-06T12:46:03.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:03.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:03.314747+0000) 2024-03-06T12:46:04.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:04.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:04.314977+0000) 2024-03-06T12:46:05.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:05.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:05.315226+0000) 2024-03-06T12:46:06.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:06.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:06.315467+0000) 2024-03-06T12:46:07.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:07.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:07.315719+0000) 2024-03-06T12:46:08.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:08.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:08.315964+0000) 2024-03-06T12:46:09.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:09.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:09.316172+0000) 2024-03-06T12:46:10.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:10.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:10.316420+0000) 2024-03-06T12:46:11.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:11.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:11.316694+0000) 2024-03-06T12:46:12.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:12.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:12.316884+0000) 2024-03-06T12:46:13.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:13.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:13.317129+0000) 2024-03-06T12:46:14.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:14.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:14.317362+0000) 2024-03-06T12:46:15.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:15.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:15.317550+0000) 2024-03-06T12:46:16.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:16.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:16.317756+0000) 2024-03-06T12:46:17.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:17.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:17.318009+0000) 2024-03-06T12:46:18.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:18.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:18.318248+0000) 2024-03-06T12:46:19.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:19.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:19.318521+0000) 2024-03-06T12:46:20.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:20.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:20.318771+0000) 2024-03-06T12:46:21.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:21.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:21.319035+0000) 2024-03-06T12:46:22.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:22.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:22.319267+0000) 2024-03-06T12:46:23.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:23.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:23.319521+0000) 2024-03-06T12:46:24.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:24.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:24.319778+0000) 2024-03-06T12:46:24.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:24.472 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:25.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:25.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:25.320045+0000) 2024-03-06T12:46:26.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:26.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:26.320314+0000) 2024-03-06T12:46:27.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:27.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:27.320573+0000) 2024-03-06T12:46:28.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:28.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:28.320840+0000) 2024-03-06T12:46:29.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:29.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:29.321107+0000) 2024-03-06T12:46:30.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:30.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:30.321337+0000) 2024-03-06T12:46:31.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:31.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:31.321596+0000) 2024-03-06T12:46:32.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:32.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:32.321858+0000) 2024-03-06T12:46:33.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:33.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:33.322117+0000) 2024-03-06T12:46:34.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:34.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:34.322371+0000) 2024-03-06T12:46:35.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:35.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:35.322605+0000) 2024-03-06T12:46:36.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:36.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:36.322794+0000) 2024-03-06T12:46:37.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:37.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:37.323046+0000) 2024-03-06T12:46:38.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:38.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:38.323300+0000) 2024-03-06T12:46:39.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:39.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:39.323591+0000) 2024-03-06T12:46:40.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:40.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:40.323902+0000) 2024-03-06T12:46:41.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:41.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:41.324183+0000) 2024-03-06T12:46:42.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:42.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:42.324422+0000) 2024-03-06T12:46:43.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:43.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:43.324713+0000) 2024-03-06T12:46:44.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:44.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:44.324988+0000) 2024-03-06T12:46:45.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:45.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:45.325253+0000) 2024-03-06T12:46:46.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:46.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:46.325519+0000) 2024-03-06T12:46:47.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:47.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:47.325780+0000) 2024-03-06T12:46:48.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:48.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:48.326031+0000) 2024-03-06T12:46:49.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:49.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:49.326320+0000) 2024-03-06T12:46:50.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:50.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:50.326595+0000) 2024-03-06T12:46:51.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:51.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:51.326927+0000) 2024-03-06T12:46:52.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:52.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:52.327132+0000) 2024-03-06T12:46:53.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:53.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:53.327384+0000) 2024-03-06T12:46:54.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:54.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:54.327703+0000) 2024-03-06T12:46:54.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:54.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:55.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:55.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:55.327986+0000) 2024-03-06T12:46:56.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:56.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:56.328263+0000) 2024-03-06T12:46:57.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:57.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:57.328549+0000) 2024-03-06T12:46:58.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:58.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:58.328821+0000) 2024-03-06T12:46:59.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:46:59.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:59.329109+0000) 2024-03-06T12:47:00.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:00.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:00.329386+0000) 2024-03-06T12:47:01.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:01.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:01.329753+0000) 2024-03-06T12:47:02.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:02.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:02.330006+0000) 2024-03-06T12:47:03.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:03.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:03.330266+0000) 2024-03-06T12:47:04.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:04.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:04.330531+0000) 2024-03-06T12:47:05.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:05.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:05.330808+0000) 2024-03-06T12:47:06.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:06.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:06.331087+0000) 2024-03-06T12:47:07.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:07.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:07.331356+0000) 2024-03-06T12:47:08.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:08.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:08.331622+0000) 2024-03-06T12:47:09.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:09.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:09.331954+0000) 2024-03-06T12:47:10.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:10.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:10.332224+0000) 2024-03-06T12:47:11.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:11.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:11.332516+0000) 2024-03-06T12:47:12.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:12.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:12.332778+0000) 2024-03-06T12:47:13.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:13.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:13.333034+0000) 2024-03-06T12:47:14.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:14.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:14.333286+0000) 2024-03-06T12:47:15.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:15.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:15.333546+0000) 2024-03-06T12:47:16.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:16.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:16.333799+0000) 2024-03-06T12:47:17.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:17.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:17.334056+0000) 2024-03-06T12:47:18.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:18.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:18.334245+0000) 2024-03-06T12:47:19.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:19.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:19.334491+0000) 2024-03-06T12:47:20.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:20.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:20.334754+0000) 2024-03-06T12:47:21.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:21.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:21.335014+0000) 2024-03-06T12:47:22.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:22.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:22.335282+0000) 2024-03-06T12:47:23.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:23.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:23.335536+0000) 2024-03-06T12:47:24.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:24.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:24.335771+0000) 2024-03-06T12:47:24.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:24.766 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:25.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:25.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:25.336035+0000) 2024-03-06T12:47:26.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:26.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:26.336289+0000) 2024-03-06T12:47:27.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:27.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:27.336551+0000) 2024-03-06T12:47:28.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:28.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:28.336812+0000) 2024-03-06T12:47:29.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:29.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:29.337064+0000) 2024-03-06T12:47:30.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:30.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:30.337298+0000) 2024-03-06T12:47:31.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:31.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:31.337543+0000) 2024-03-06T12:47:32.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:32.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:32.337784+0000) 2024-03-06T12:47:33.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:33.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:33.338039+0000) 2024-03-06T12:47:34.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:34.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:34.338290+0000) 2024-03-06T12:47:35.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:35.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:35.338474+0000) 2024-03-06T12:47:36.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:36.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:36.338750+0000) 2024-03-06T12:47:37.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:37.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:37.339012+0000) 2024-03-06T12:47:38.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:38.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:38.339270+0000) 2024-03-06T12:47:39.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:39.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:39.339524+0000) 2024-03-06T12:47:40.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:40.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:40.339774+0000) 2024-03-06T12:47:41.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:41.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:41.340038+0000) 2024-03-06T12:47:42.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:42.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:42.340285+0000) 2024-03-06T12:47:43.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:43.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:43.340524+0000) 2024-03-06T12:47:44.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:44.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:44.340757+0000) 2024-03-06T12:47:45.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:45.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:45.340995+0000) 2024-03-06T12:47:46.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:46.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:46.341252+0000) 2024-03-06T12:47:47.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:47.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:47.341516+0000) 2024-03-06T12:47:48.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:48.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:48.341782+0000) 2024-03-06T12:47:49.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:49.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:49.342035+0000) 2024-03-06T12:47:50.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:50.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:50.342286+0000) 2024-03-06T12:47:51.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:51.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:51.342525+0000) 2024-03-06T12:47:52.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:52.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:52.342763+0000) 2024-03-06T12:47:53.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:53.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:53.343002+0000) 2024-03-06T12:47:54.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:54.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:54.343235+0000) 2024-03-06T12:47:54.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:54.929 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:55.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:55.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:55.343469+0000) 2024-03-06T12:47:56.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:56.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:56.343744+0000) 2024-03-06T12:47:57.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:57.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:57.343995+0000) 2024-03-06T12:47:58.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:58.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:58.344237+0000) 2024-03-06T12:47:59.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:47:59.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:59.344442+0000) 2024-03-06T12:48:00.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:00.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:00.344784+0000) 2024-03-06T12:48:01.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:01.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:01.345050+0000) 2024-03-06T12:48:02.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:02.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:02.345320+0000) 2024-03-06T12:48:03.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:03.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:03.345571+0000) 2024-03-06T12:48:04.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:04.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:04.345826+0000) 2024-03-06T12:48:05.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:05.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:05.346087+0000) 2024-03-06T12:48:06.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:06.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:06.346337+0000) 2024-03-06T12:48:07.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:07.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:07.346592+0000) 2024-03-06T12:48:08.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:08.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:08.346848+0000) 2024-03-06T12:48:09.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:09.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:09.347060+0000) 2024-03-06T12:48:10.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:10.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:10.347297+0000) 2024-03-06T12:48:11.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:11.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:11.347541+0000) 2024-03-06T12:48:12.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:12.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:12.347784+0000) 2024-03-06T12:48:13.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:13.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:13.347972+0000) 2024-03-06T12:48:14.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:14.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:14.348162+0000) 2024-03-06T12:48:15.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:15.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:15.348422+0000) 2024-03-06T12:48:16.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:16.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:16.348694+0000) 2024-03-06T12:48:17.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:17.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:17.348892+0000) 2024-03-06T12:48:18.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:18.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:18.349100+0000) 2024-03-06T12:48:19.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:19.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:19.349363+0000) 2024-03-06T12:48:20.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:20.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:20.349600+0000) 2024-03-06T12:48:21.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:21.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:21.349903+0000) 2024-03-06T12:48:22.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:22.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:22.350143+0000) 2024-03-06T12:48:23.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:23.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:23.350404+0000) 2024-03-06T12:48:24.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:24.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:24.350589+0000) 2024-03-06T12:48:25.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:25.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:25.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:25.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:25.350839+0000) 2024-03-06T12:48:26.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:26.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:26.351094+0000) 2024-03-06T12:48:27.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:27.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:27.351339+0000) 2024-03-06T12:48:28.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:28.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:28.351725+0000) 2024-03-06T12:48:29.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:29.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:29.352012+0000) 2024-03-06T12:48:30.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:30.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:30.352247+0000) 2024-03-06T12:48:31.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:31.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:31.352511+0000) 2024-03-06T12:48:32.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:32.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:32.352762+0000) 2024-03-06T12:48:33.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:33.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:33.353032+0000) 2024-03-06T12:48:34.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:34.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:34.353274+0000) 2024-03-06T12:48:35.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:35.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:35.353533+0000) 2024-03-06T12:48:36.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:36.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:36.353786+0000) 2024-03-06T12:48:37.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:37.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:37.354031+0000) 2024-03-06T12:48:38.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:38.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:38.354222+0000) 2024-03-06T12:48:39.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:39.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:39.354493+0000) 2024-03-06T12:48:40.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:40.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:40.354751+0000) 2024-03-06T12:48:41.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:41.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:41.355009+0000) 2024-03-06T12:48:42.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:42.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:42.355257+0000) 2024-03-06T12:48:43.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:43.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:43.355516+0000) 2024-03-06T12:48:44.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:44.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:44.355767+0000) 2024-03-06T12:48:45.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:45.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:45.356026+0000) 2024-03-06T12:48:46.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:46.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:46.356204+0000) 2024-03-06T12:48:47.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:47.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:47.356470+0000) 2024-03-06T12:48:48.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:48.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:48.356813+0000) 2024-03-06T12:48:49.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:49.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:49.357014+0000) 2024-03-06T12:48:50.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:50.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:50.357250+0000) 2024-03-06T12:48:51.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:51.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:51.357520+0000) 2024-03-06T12:48:52.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:52.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:52.357762+0000) 2024-03-06T12:48:53.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:53.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:53.358011+0000) 2024-03-06T12:48:54.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:54.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:54.358262+0000) 2024-03-06T12:48:55.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:55.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:55.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:55.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:55.358535+0000) 2024-03-06T12:48:56.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:56.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:56.358790+0000) 2024-03-06T12:48:57.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:57.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:57.359026+0000) 2024-03-06T12:48:58.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:58.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:58.359277+0000) 2024-03-06T12:48:59.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:48:59.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:59.359529+0000) 2024-03-06T12:49:00.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:00.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:00.359761+0000) 2024-03-06T12:49:01.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:01.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:01.359950+0000) 2024-03-06T12:49:02.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:02.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:02.360190+0000) 2024-03-06T12:49:03.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:03.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:03.360451+0000) 2024-03-06T12:49:04.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:04.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:04.360758+0000) 2024-03-06T12:49:05.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:05.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:05.361016+0000) 2024-03-06T12:49:06.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:06.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:06.361284+0000) 2024-03-06T12:49:07.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:07.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:07.361488+0000) 2024-03-06T12:49:08.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:08.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:08.361759+0000) 2024-03-06T12:49:09.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:09.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:09.362026+0000) 2024-03-06T12:49:10.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:10.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:10.362259+0000) 2024-03-06T12:49:11.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:11.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:11.362524+0000) 2024-03-06T12:49:12.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:12.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:12.362760+0000) 2024-03-06T12:49:13.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:13.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:13.362942+0000) 2024-03-06T12:49:14.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:14.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:14.363185+0000) 2024-03-06T12:49:15.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:15.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:15.363440+0000) 2024-03-06T12:49:16.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:16.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:16.363747+0000) 2024-03-06T12:49:17.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:17.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:17.364011+0000) 2024-03-06T12:49:18.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:18.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:18.364272+0000) 2024-03-06T12:49:19.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:19.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:19.364544+0000) 2024-03-06T12:49:20.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:20.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:20.364798+0000) 2024-03-06T12:49:21.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:21.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:21.365059+0000) 2024-03-06T12:49:22.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:22.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:22.365307+0000) 2024-03-06T12:49:23.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:23.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:23.365559+0000) 2024-03-06T12:49:24.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:24.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:24.365793+0000) 2024-03-06T12:49:25.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:25.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:25.365982+0000) 2024-03-06T12:49:25.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:25.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:26.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:26.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:26.366236+0000) 2024-03-06T12:49:27.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:27.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:27.366531+0000) 2024-03-06T12:49:28.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:28.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:28.366817+0000) 2024-03-06T12:49:29.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:29.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:29.367068+0000) 2024-03-06T12:49:30.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:30.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:30.367305+0000) 2024-03-06T12:49:31.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:31.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:31.367559+0000) 2024-03-06T12:49:32.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:32.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:32.367814+0000) 2024-03-06T12:49:33.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:33.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:33.368065+0000) 2024-03-06T12:49:34.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:34.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:34.368315+0000) 2024-03-06T12:49:35.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:35.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:35.368571+0000) 2024-03-06T12:49:36.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:36.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:36.368863+0000) 2024-03-06T12:49:37.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:37.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:37.369123+0000) 2024-03-06T12:49:38.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:38.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:38.369372+0000) 2024-03-06T12:49:39.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:39.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:39.369669+0000) 2024-03-06T12:49:40.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:40.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:40.369921+0000) 2024-03-06T12:49:41.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:41.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:41.370184+0000) 2024-03-06T12:49:42.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:42.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:42.370438+0000) 2024-03-06T12:49:43.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:43.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:43.370724+0000) 2024-03-06T12:49:44.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:44.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:44.370982+0000) 2024-03-06T12:49:45.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:45.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:45.371249+0000) 2024-03-06T12:49:46.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:46.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:46.371485+0000) 2024-03-06T12:49:47.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:47.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:47.371749+0000) 2024-03-06T12:49:48.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:48.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:48.371992+0000) 2024-03-06T12:49:49.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:49.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:49.372246+0000) 2024-03-06T12:49:50.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:50.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:50.372495+0000) 2024-03-06T12:49:51.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:51.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:51.372747+0000) 2024-03-06T12:49:52.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:52.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:52.373027+0000) 2024-03-06T12:49:53.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:53.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:53.373349+0000) 2024-03-06T12:49:54.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:54.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:54.373683+0000) 2024-03-06T12:49:55.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:55.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:55.373981+0000) 2024-03-06T12:49:55.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:55.520 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:56.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:56.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:56.374307+0000) 2024-03-06T12:49:57.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:57.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:57.374597+0000) 2024-03-06T12:49:58.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:58.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:58.374940+0000) 2024-03-06T12:49:59.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:49:59.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:59.375259+0000) 2024-03-06T12:50:00.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:00.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:00.375572+0000) 2024-03-06T12:50:01.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:01.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:01.375877+0000) 2024-03-06T12:50:02.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:02.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:02.376194+0000) 2024-03-06T12:50:03.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:03.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:03.376492+0000) 2024-03-06T12:50:04.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:04.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:04.376802+0000) 2024-03-06T12:50:05.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:05.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:05.377040+0000) 2024-03-06T12:50:06.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:06.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:06.377296+0000) 2024-03-06T12:50:07.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:07.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:07.377597+0000) 2024-03-06T12:50:08.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:08.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:08.377947+0000) 2024-03-06T12:50:09.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:09.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:09.378266+0000) 2024-03-06T12:50:10.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:10.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:10.378491+0000) 2024-03-06T12:50:11.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:11.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:11.378812+0000) 2024-03-06T12:50:12.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:12.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:12.379123+0000) 2024-03-06T12:50:13.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:13.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:13.379423+0000) 2024-03-06T12:50:14.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:14.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:14.379749+0000) 2024-03-06T12:50:15.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:15.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:15.380041+0000) 2024-03-06T12:50:16.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:16.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:16.380358+0000) 2024-03-06T12:50:17.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:17.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:17.380712+0000) 2024-03-06T12:50:18.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:18.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:18.381081+0000) 2024-03-06T12:50:19.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:19.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:19.381398+0000) 2024-03-06T12:50:20.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:20.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:20.381730+0000) 2024-03-06T12:50:21.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:21.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:21.382030+0000) 2024-03-06T12:50:22.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:22.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:22.382341+0000) 2024-03-06T12:50:23.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:23.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:23.382704+0000) 2024-03-06T12:50:24.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:24.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:24.382961+0000) 2024-03-06T12:50:25.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:25.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:25.383242+0000) 2024-03-06T12:50:25.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:25.679 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:26.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:26.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:26.383556+0000) 2024-03-06T12:50:27.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:27.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:27.383867+0000) 2024-03-06T12:50:28.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:28.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:28.384164+0000) 2024-03-06T12:50:29.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:29.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:29.384492+0000) 2024-03-06T12:50:30.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:30.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:30.384806+0000) 2024-03-06T12:50:31.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:31.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:31.385128+0000) 2024-03-06T12:50:32.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:32.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:32.385440+0000) 2024-03-06T12:50:33.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:33.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:33.385784+0000) 2024-03-06T12:50:34.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:34.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:34.386084+0000) 2024-03-06T12:50:35.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:35.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:35.386379+0000) 2024-03-06T12:50:36.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:36.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:36.386758+0000) 2024-03-06T12:50:37.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:37.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:37.387074+0000) 2024-03-06T12:50:38.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:38.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:38.387362+0000) 2024-03-06T12:50:39.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:39.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:39.387738+0000) 2024-03-06T12:50:40.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:40.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:40.388030+0000) 2024-03-06T12:50:41.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:41.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:41.388258+0000) 2024-03-06T12:50:42.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:42.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:42.388489+0000) 2024-03-06T12:50:43.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:43.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:43.388750+0000) 2024-03-06T12:50:44.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:44.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:44.388982+0000) 2024-03-06T12:50:45.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:45.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:45.389224+0000) 2024-03-06T12:50:46.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:46.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:46.389459+0000) 2024-03-06T12:50:47.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:47.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:47.389735+0000) 2024-03-06T12:50:48.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:48.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:48.389988+0000) 2024-03-06T12:50:49.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:49.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:49.390245+0000) 2024-03-06T12:50:50.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:50.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:50.390506+0000) 2024-03-06T12:50:51.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:51.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:51.390765+0000) 2024-03-06T12:50:52.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:52.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:52.391017+0000) 2024-03-06T12:50:53.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:53.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:53.391254+0000) 2024-03-06T12:50:54.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:54.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:54.391498+0000) 2024-03-06T12:50:55.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:55.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:55.391753+0000) 2024-03-06T12:50:55.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:55.837 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:56.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:56.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:56.391989+0000) 2024-03-06T12:50:57.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:57.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:57.392229+0000) 2024-03-06T12:50:58.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:58.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:58.392459+0000) 2024-03-06T12:50:59.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:50:59.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:59.392749+0000) 2024-03-06T12:51:00.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:00.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:00.392984+0000) 2024-03-06T12:51:01.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:01.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:01.393248+0000) 2024-03-06T12:51:02.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:02.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:02.393498+0000) 2024-03-06T12:51:03.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:03.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:03.393750+0000) 2024-03-06T12:51:04.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:04.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:04.394001+0000) 2024-03-06T12:51:05.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:05.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:05.394255+0000) 2024-03-06T12:51:06.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:06.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:06.394490+0000) 2024-03-06T12:51:07.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:07.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:07.394748+0000) 2024-03-06T12:51:08.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:08.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:08.394994+0000) 2024-03-06T12:51:09.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:09.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:09.395261+0000) 2024-03-06T12:51:10.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:10.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:10.395459+0000) 2024-03-06T12:51:11.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:11.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:11.395742+0000) 2024-03-06T12:51:12.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:12.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:12.395998+0000) 2024-03-06T12:51:13.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:13.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:13.396231+0000) 2024-03-06T12:51:14.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:14.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:14.396481+0000) 2024-03-06T12:51:15.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:15.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:15.396737+0000) 2024-03-06T12:51:16.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:16.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:16.396974+0000) 2024-03-06T12:51:17.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:17.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:17.397223+0000) 2024-03-06T12:51:18.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:18.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:18.397462+0000) 2024-03-06T12:51:19.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:19.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:19.397758+0000) 2024-03-06T12:51:20.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:20.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:20.398010+0000) 2024-03-06T12:51:21.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:21.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:21.398247+0000) 2024-03-06T12:51:22.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:22.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:22.398497+0000) 2024-03-06T12:51:23.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:23.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:23.398748+0000) 2024-03-06T12:51:24.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:24.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:24.398932+0000) 2024-03-06T12:51:25.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:25.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:25.399159+0000) 2024-03-06T12:51:25.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:25.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:26.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:26.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:26.399399+0000) 2024-03-06T12:51:27.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:27.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:27.399674+0000) 2024-03-06T12:51:28.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:28.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:28.399928+0000) 2024-03-06T12:51:29.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:29.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:29.400195+0000) 2024-03-06T12:51:30.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:30.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:30.400450+0000) 2024-03-06T12:51:31.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:31.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:31.400733+0000) 2024-03-06T12:51:32.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:32.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:32.400968+0000) 2024-03-06T12:51:33.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:33.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:33.401150+0000) 2024-03-06T12:51:34.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:34.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:34.401402+0000) 2024-03-06T12:51:35.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:35.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:35.401672+0000) 2024-03-06T12:51:36.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:36.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:36.401910+0000) 2024-03-06T12:51:37.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:37.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:37.402148+0000) 2024-03-06T12:51:38.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:38.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:38.402399+0000) 2024-03-06T12:51:39.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:39.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:39.402693+0000) 2024-03-06T12:51:40.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:40.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:40.402874+0000) 2024-03-06T12:51:41.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:41.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:41.403116+0000) 2024-03-06T12:51:42.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:42.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:42.403348+0000) 2024-03-06T12:51:43.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:43.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:43.403585+0000) 2024-03-06T12:51:44.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:44.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:44.403823+0000) 2024-03-06T12:51:45.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:45.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:45.404080+0000) 2024-03-06T12:51:46.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:46.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:46.404311+0000) 2024-03-06T12:51:47.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:47.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:47.404551+0000) 2024-03-06T12:51:48.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:48.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:48.404791+0000) 2024-03-06T12:51:49.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:49.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:49.405052+0000) 2024-03-06T12:51:50.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:50.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:50.405301+0000) 2024-03-06T12:51:51.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:51.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:51.405561+0000) 2024-03-06T12:51:52.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:52.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:52.405753+0000) 2024-03-06T12:51:53.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:53.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:53.406006+0000) 2024-03-06T12:51:54.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:54.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:54.406260+0000) 2024-03-06T12:51:55.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:55.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:55.406506+0000) 2024-03-06T12:51:56.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:56.139 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:56.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:56.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:56.406789+0000) 2024-03-06T12:51:57.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:57.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:57.407039+0000) 2024-03-06T12:51:58.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:58.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:58.407287+0000) 2024-03-06T12:51:59.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:51:59.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:59.407557+0000) 2024-03-06T12:52:00.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:00.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:00.407751+0000) 2024-03-06T12:52:01.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:01.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:01.408013+0000) 2024-03-06T12:52:02.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:02.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:02.408282+0000) 2024-03-06T12:52:03.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:03.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:03.408509+0000) 2024-03-06T12:52:04.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:04.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:04.408765+0000) 2024-03-06T12:52:05.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:05.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:05.409079+0000) 2024-03-06T12:52:06.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:06.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:06.409259+0000) 2024-03-06T12:52:07.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:07.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:07.409497+0000) 2024-03-06T12:52:08.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:08.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:08.409749+0000) 2024-03-06T12:52:09.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:09.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:09.410015+0000) 2024-03-06T12:52:10.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:10.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:10.410272+0000) 2024-03-06T12:52:11.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:11.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:11.410513+0000) 2024-03-06T12:52:12.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:12.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:12.410766+0000) 2024-03-06T12:52:13.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:13.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:13.411018+0000) 2024-03-06T12:52:14.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:14.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:14.411209+0000) 2024-03-06T12:52:15.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:15.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:15.411459+0000) 2024-03-06T12:52:16.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:16.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:16.411748+0000) 2024-03-06T12:52:17.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:17.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:17.411999+0000) 2024-03-06T12:52:18.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:18.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:18.412241+0000) 2024-03-06T12:52:19.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:19.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:19.412498+0000) 2024-03-06T12:52:20.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:20.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:20.412751+0000) 2024-03-06T12:52:21.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:21.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:21.412992+0000) 2024-03-06T12:52:22.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:22.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:22.413241+0000) 2024-03-06T12:52:23.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:23.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:23.413480+0000) 2024-03-06T12:52:24.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:24.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:24.413764+0000) 2024-03-06T12:52:25.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:25.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:25.414007+0000) 2024-03-06T12:52:26.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:26.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:26.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:26.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:26.414296+0000) 2024-03-06T12:52:27.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:27.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:27.414613+0000) 2024-03-06T12:52:28.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:28.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:28.414918+0000) 2024-03-06T12:52:29.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:29.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:29.415215+0000) 2024-03-06T12:52:30.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:30.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:30.415463+0000) 2024-03-06T12:52:31.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:31.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:31.415750+0000) 2024-03-06T12:52:32.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:32.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:32.415949+0000) 2024-03-06T12:52:33.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:33.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:33.416185+0000) 2024-03-06T12:52:34.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:34.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:34.416402+0000) 2024-03-06T12:52:35.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:35.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:35.416694+0000) 2024-03-06T12:52:36.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:36.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:36.416929+0000) 2024-03-06T12:52:37.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:37.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:37.417195+0000) 2024-03-06T12:52:38.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:38.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:38.417448+0000) 2024-03-06T12:52:39.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:39.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:39.417773+0000) 2024-03-06T12:52:40.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:40.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:40.418030+0000) 2024-03-06T12:52:41.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:41.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:41.418270+0000) 2024-03-06T12:52:42.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:42.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:42.418507+0000) 2024-03-06T12:52:43.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:43.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:43.418743+0000) 2024-03-06T12:52:44.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:44.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:44.418966+0000) 2024-03-06T12:52:45.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:45.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:45.419228+0000) 2024-03-06T12:52:46.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:46.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:46.419482+0000) 2024-03-06T12:52:47.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:47.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:47.419759+0000) 2024-03-06T12:52:48.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:48.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:48.419973+0000) 2024-03-06T12:52:49.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:49.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:49.420246+0000) 2024-03-06T12:52:50.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:50.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:50.420506+0000) 2024-03-06T12:52:51.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:51.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:51.420799+0000) 2024-03-06T12:52:52.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:52.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:52.421076+0000) 2024-03-06T12:52:53.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:53.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:53.421329+0000) 2024-03-06T12:52:54.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:54.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:54.421616+0000) 2024-03-06T12:52:55.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:55.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:55.421945+0000) 2024-03-06T12:52:56.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:56.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:56.422218+0000) 2024-03-06T12:52:56.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:56.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:57.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:57.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:57.422494+0000) 2024-03-06T12:52:58.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:58.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:58.422830+0000) 2024-03-06T12:52:59.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:52:59.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:59.423106+0000) 2024-03-06T12:53:00.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:00.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:00.423357+0000) 2024-03-06T12:53:01.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:01.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:01.423611+0000) 2024-03-06T12:53:02.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:02.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:02.423925+0000) 2024-03-06T12:53:03.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:03.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:03.424181+0000) 2024-03-06T12:53:04.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:04.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:04.424436+0000) 2024-03-06T12:53:05.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:05.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:05.424710+0000) 2024-03-06T12:53:06.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:06.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:06.424947+0000) 2024-03-06T12:53:07.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:07.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:07.425193+0000) 2024-03-06T12:53:08.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:08.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:08.425435+0000) 2024-03-06T12:53:09.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:09.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:09.425680+0000) 2024-03-06T12:53:10.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:10.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:10.425939+0000) 2024-03-06T12:53:11.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:11.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:11.426175+0000) 2024-03-06T12:53:12.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:12.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:12.426369+0000) 2024-03-06T12:53:13.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:13.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:13.426606+0000) 2024-03-06T12:53:14.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:14.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:14.426930+0000) 2024-03-06T12:53:15.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:15.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:15.427187+0000) 2024-03-06T12:53:16.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:16.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:16.427436+0000) 2024-03-06T12:53:17.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:17.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:17.427746+0000) 2024-03-06T12:53:18.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:18.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:18.427980+0000) 2024-03-06T12:53:19.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:19.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:19.428239+0000) 2024-03-06T12:53:20.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:20.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:20.428494+0000) 2024-03-06T12:53:21.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:21.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:21.428756+0000) 2024-03-06T12:53:22.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:22.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:22.428942+0000) 2024-03-06T12:53:23.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:23.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:23.429178+0000) 2024-03-06T12:53:24.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:24.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:24.429432+0000) 2024-03-06T12:53:25.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:25.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:25.429743+0000) 2024-03-06T12:53:26.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:26.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:26.429994+0000) 2024-03-06T12:53:26.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:26.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:27.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:27.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:27.430258+0000) 2024-03-06T12:53:28.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:28.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:28.430495+0000) 2024-03-06T12:53:29.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:29.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:29.430768+0000) 2024-03-06T12:53:30.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:30.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:30.431022+0000) 2024-03-06T12:53:31.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:31.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:31.431279+0000) 2024-03-06T12:53:32.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:32.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:32.431536+0000) 2024-03-06T12:53:33.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:33.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:33.431798+0000) 2024-03-06T12:53:34.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:34.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:34.432048+0000) 2024-03-06T12:53:35.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:35.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:35.432308+0000) 2024-03-06T12:53:36.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:36.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:36.432518+0000) 2024-03-06T12:53:37.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:37.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:37.432775+0000) 2024-03-06T12:53:38.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:38.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:38.433031+0000) 2024-03-06T12:53:39.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:39.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:39.433295+0000) 2024-03-06T12:53:40.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:40.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:40.433530+0000) 2024-03-06T12:53:41.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:41.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:41.433792+0000) 2024-03-06T12:53:42.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:42.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:42.434041+0000) 2024-03-06T12:53:43.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:43.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:43.434295+0000) 2024-03-06T12:53:44.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:44.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:44.434534+0000) 2024-03-06T12:53:45.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:45.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:45.434796+0000) 2024-03-06T12:53:46.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:46.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:46.435099+0000) 2024-03-06T12:53:47.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:47.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:47.435359+0000) 2024-03-06T12:53:48.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:48.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:48.435599+0000) 2024-03-06T12:53:49.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:49.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:49.435907+0000) 2024-03-06T12:53:50.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:50.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:50.436161+0000) 2024-03-06T12:53:51.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:51.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:51.436353+0000) 2024-03-06T12:53:52.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:52.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:52.436611+0000) 2024-03-06T12:53:53.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:53.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:53.436886+0000) 2024-03-06T12:53:54.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:54.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:54.437067+0000) 2024-03-06T12:53:55.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:55.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:55.437325+0000) 2024-03-06T12:53:56.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:56.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:56.437556+0000) 2024-03-06T12:53:56.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:56.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:57.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:57.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:57.437813+0000) 2024-03-06T12:53:58.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:58.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:58.438000+0000) 2024-03-06T12:53:59.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:53:59.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:59.438267+0000) 2024-03-06T12:54:00.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:00.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:00.438527+0000) 2024-03-06T12:54:01.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:01.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:01.438779+0000) 2024-03-06T12:54:02.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:02.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:02.438974+0000) 2024-03-06T12:54:03.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:03.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:03.439229+0000) 2024-03-06T12:54:04.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:04.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:04.439462+0000) 2024-03-06T12:54:05.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:05.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:05.439756+0000) 2024-03-06T12:54:06.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:06.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:06.439985+0000) 2024-03-06T12:54:07.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:07.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:07.440231+0000) 2024-03-06T12:54:08.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:08.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:08.440489+0000) 2024-03-06T12:54:09.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:09.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:09.440752+0000) 2024-03-06T12:54:10.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:10.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:10.440989+0000) 2024-03-06T12:54:11.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:11.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:11.441235+0000) 2024-03-06T12:54:12.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:12.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:12.441472+0000) 2024-03-06T12:54:13.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:13.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:13.441766+0000) 2024-03-06T12:54:14.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:14.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:14.442012+0000) 2024-03-06T12:54:15.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:15.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:15.442272+0000) 2024-03-06T12:54:16.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:16.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:16.442558+0000) 2024-03-06T12:54:17.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:17.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:17.442827+0000) 2024-03-06T12:54:18.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:18.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:18.443026+0000) 2024-03-06T12:54:19.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:19.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:19.443288+0000) 2024-03-06T12:54:20.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:20.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:20.443542+0000) 2024-03-06T12:54:21.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:21.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:21.443810+0000) 2024-03-06T12:54:22.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:22.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:22.444060+0000) 2024-03-06T12:54:23.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:23.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:23.444294+0000) 2024-03-06T12:54:24.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:24.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:24.444554+0000) 2024-03-06T12:54:25.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:25.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:25.444813+0000) 2024-03-06T12:54:26.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:26.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:26.445065+0000) 2024-03-06T12:54:26.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:26.895 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:27.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:27.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:27.445325+0000) 2024-03-06T12:54:28.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:28.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:28.445556+0000) 2024-03-06T12:54:29.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:29.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:29.445818+0000) 2024-03-06T12:54:30.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:30.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:30.446006+0000) 2024-03-06T12:54:31.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:31.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:31.446256+0000) 2024-03-06T12:54:32.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:32.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:32.446519+0000) 2024-03-06T12:54:33.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:33.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:33.446778+0000) 2024-03-06T12:54:34.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:34.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:34.447064+0000) 2024-03-06T12:54:35.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:35.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:35.447325+0000) 2024-03-06T12:54:36.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:36.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:36.447572+0000) 2024-03-06T12:54:37.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:37.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:37.447814+0000) 2024-03-06T12:54:38.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:38.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:38.448066+0000) 2024-03-06T12:54:39.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:39.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:39.448332+0000) 2024-03-06T12:54:40.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:40.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:40.448573+0000) 2024-03-06T12:54:41.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:41.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:41.448812+0000) 2024-03-06T12:54:42.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:42.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:42.449060+0000) 2024-03-06T12:54:43.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:43.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:43.449305+0000) 2024-03-06T12:54:44.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:44.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:44.449556+0000) 2024-03-06T12:54:45.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:45.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:45.449815+0000) 2024-03-06T12:54:46.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:46.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:46.450065+0000) 2024-03-06T12:54:47.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:47.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:47.450337+0000) 2024-03-06T12:54:48.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:48.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:48.450591+0000) 2024-03-06T12:54:49.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:49.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:49.450896+0000) 2024-03-06T12:54:50.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:50.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:50.451158+0000) 2024-03-06T12:54:51.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:51.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:51.451410+0000) 2024-03-06T12:54:52.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:52.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:52.451714+0000) 2024-03-06T12:54:53.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:53.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:53.451978+0000) 2024-03-06T12:54:54.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:54.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:54.452232+0000) 2024-03-06T12:54:55.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:55.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:55.452495+0000) 2024-03-06T12:54:56.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:56.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:56.452764+0000) 2024-03-06T12:54:57.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:57.039 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:57.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:57.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:57.453033+0000) 2024-03-06T12:54:58.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:58.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:58.453297+0000) 2024-03-06T12:54:59.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:54:59.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:59.453543+0000) 2024-03-06T12:55:00.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:00.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:00.453794+0000) 2024-03-06T12:55:01.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:01.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:01.454040+0000) 2024-03-06T12:55:02.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:02.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:02.454307+0000) 2024-03-06T12:55:03.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:03.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:03.454534+0000) 2024-03-06T12:55:04.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:04.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:04.454786+0000) 2024-03-06T12:55:05.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:05.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:05.455047+0000) 2024-03-06T12:55:06.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:06.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:06.455274+0000) 2024-03-06T12:55:07.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:07.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:07.455528+0000) 2024-03-06T12:55:08.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:08.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:08.455762+0000) 2024-03-06T12:55:09.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:09.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:09.456006+0000) 2024-03-06T12:55:10.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:10.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:10.456254+0000) 2024-03-06T12:55:11.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:11.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:11.456501+0000) 2024-03-06T12:55:12.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:12.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:12.456768+0000) 2024-03-06T12:55:13.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:13.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:13.456959+0000) 2024-03-06T12:55:14.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:14.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:14.457208+0000) 2024-03-06T12:55:15.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:15.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:15.457470+0000) 2024-03-06T12:55:16.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:16.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:16.457752+0000) 2024-03-06T12:55:17.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:17.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:17.457989+0000) 2024-03-06T12:55:18.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:18.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:18.458249+0000) 2024-03-06T12:55:19.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:19.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:19.458514+0000) 2024-03-06T12:55:20.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:20.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:20.458772+0000) 2024-03-06T12:55:21.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:21.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:21.458996+0000) 2024-03-06T12:55:22.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:22.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:22.459232+0000) 2024-03-06T12:55:23.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:23.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:23.459474+0000) 2024-03-06T12:55:24.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:24.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:24.459738+0000) 2024-03-06T12:55:25.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:25.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:25.459988+0000) 2024-03-06T12:55:26.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:26.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:26.460255+0000) 2024-03-06T12:55:27.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:27.203 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:27.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:27.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:27.460469+0000) 2024-03-06T12:55:28.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:28.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:28.460808+0000) 2024-03-06T12:55:29.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:29.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:29.461069+0000) 2024-03-06T12:55:30.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:30.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:30.461339+0000) 2024-03-06T12:55:31.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:31.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:31.461602+0000) 2024-03-06T12:55:32.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:32.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:32.461874+0000) 2024-03-06T12:55:33.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:33.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:33.462130+0000) 2024-03-06T12:55:34.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:34.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:34.462394+0000) 2024-03-06T12:55:35.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:35.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:35.462696+0000) 2024-03-06T12:55:36.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:36.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:36.462930+0000) 2024-03-06T12:55:37.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:37.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:37.463126+0000) 2024-03-06T12:55:38.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:38.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:38.463380+0000) 2024-03-06T12:55:39.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:39.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:39.463662+0000) 2024-03-06T12:55:40.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:40.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:40.463901+0000) 2024-03-06T12:55:41.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:41.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:41.464142+0000) 2024-03-06T12:55:42.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:42.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:42.464373+0000) 2024-03-06T12:55:43.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:43.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:43.464679+0000) 2024-03-06T12:55:44.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:44.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:44.464932+0000) 2024-03-06T12:55:45.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:45.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:45.465180+0000) 2024-03-06T12:55:46.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:46.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:46.465441+0000) 2024-03-06T12:55:47.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:47.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:47.465724+0000) 2024-03-06T12:55:48.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:48.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:48.465958+0000) 2024-03-06T12:55:49.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:49.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:49.466198+0000) 2024-03-06T12:55:50.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:50.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:50.466451+0000) 2024-03-06T12:55:51.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:51.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:51.466721+0000) 2024-03-06T12:55:52.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:52.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:52.466974+0000) 2024-03-06T12:55:53.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:53.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:53.467231+0000) 2024-03-06T12:55:54.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:54.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:54.467487+0000) 2024-03-06T12:55:55.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:55.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:55.467745+0000) 2024-03-06T12:55:56.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:56.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:56.468004+0000) 2024-03-06T12:55:57.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:57.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:57.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:57.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:57.468217+0000) 2024-03-06T12:55:58.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:58.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:58.468467+0000) 2024-03-06T12:55:59.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:55:59.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:59.468737+0000) 2024-03-06T12:56:00.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:00.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:00.468995+0000) 2024-03-06T12:56:01.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:01.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:01.469255+0000) 2024-03-06T12:56:02.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:02.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:02.469512+0000) 2024-03-06T12:56:03.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:03.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:03.469769+0000) 2024-03-06T12:56:04.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:04.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:04.470024+0000) 2024-03-06T12:56:05.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:05.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:05.470288+0000) 2024-03-06T12:56:06.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:06.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:06.470541+0000) 2024-03-06T12:56:07.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:07.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:07.470784+0000) 2024-03-06T12:56:08.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:08.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:08.471023+0000) 2024-03-06T12:56:09.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:09.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:09.471270+0000) 2024-03-06T12:56:10.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:10.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:10.471520+0000) 2024-03-06T12:56:11.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:11.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:11.471755+0000) 2024-03-06T12:56:12.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:12.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:12.471988+0000) 2024-03-06T12:56:13.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:13.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:13.472224+0000) 2024-03-06T12:56:14.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:14.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:14.472482+0000) 2024-03-06T12:56:15.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:15.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:15.472736+0000) 2024-03-06T12:56:16.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:16.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:16.473050+0000) 2024-03-06T12:56:17.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:17.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:17.473305+0000) 2024-03-06T12:56:18.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:18.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:18.473574+0000) 2024-03-06T12:56:19.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:19.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:19.473832+0000) 2024-03-06T12:56:20.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:20.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:20.474088+0000) 2024-03-06T12:56:21.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:21.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:21.474340+0000) 2024-03-06T12:56:22.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:22.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:22.474578+0000) 2024-03-06T12:56:23.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:23.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:23.474832+0000) 2024-03-06T12:56:24.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:24.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:24.475092+0000) 2024-03-06T12:56:25.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:25.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:25.475284+0000) 2024-03-06T12:56:26.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:26.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:26.475530+0000) 2024-03-06T12:56:27.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:27.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:27.475776+0000) 2024-03-06T12:56:27.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:27.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:28.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:28.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:28.476016+0000) 2024-03-06T12:56:29.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:29.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:29.476271+0000) 2024-03-06T12:56:30.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:30.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:30.476536+0000) 2024-03-06T12:56:31.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:31.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:31.476781+0000) 2024-03-06T12:56:32.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:32.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:32.477042+0000) 2024-03-06T12:56:33.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:33.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:33.477236+0000) 2024-03-06T12:56:34.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:34.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:34.477490+0000) 2024-03-06T12:56:35.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:35.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:35.477745+0000) 2024-03-06T12:56:36.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:36.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:36.477999+0000) 2024-03-06T12:56:37.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:37.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:37.478244+0000) 2024-03-06T12:56:38.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:38.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:38.478476+0000) 2024-03-06T12:56:39.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:39.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:39.478746+0000) 2024-03-06T12:56:40.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:40.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:40.478990+0000) 2024-03-06T12:56:41.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:41.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:41.479177+0000) 2024-03-06T12:56:42.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:42.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:42.479432+0000) 2024-03-06T12:56:43.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:43.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:43.479676+0000) 2024-03-06T12:56:44.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:44.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:44.479932+0000) 2024-03-06T12:56:45.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:45.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:45.480183+0000) 2024-03-06T12:56:46.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:46.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:46.480439+0000) 2024-03-06T12:56:47.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:47.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:47.480617+0000) 2024-03-06T12:56:48.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:48.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:48.480909+0000) 2024-03-06T12:56:49.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:49.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:49.481146+0000) 2024-03-06T12:56:50.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:50.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:50.481407+0000) 2024-03-06T12:56:51.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:51.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:51.481671+0000) 2024-03-06T12:56:52.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:52.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:52.481920+0000) 2024-03-06T12:56:53.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:53.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:53.482111+0000) 2024-03-06T12:56:54.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:54.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:54.482350+0000) 2024-03-06T12:56:55.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:55.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:55.482587+0000) 2024-03-06T12:56:56.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:56.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:56.482901+0000) 2024-03-06T12:56:57.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:57.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:57.483151+0000) 2024-03-06T12:56:57.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:57.645 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:58.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:58.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:58.483338+0000) 2024-03-06T12:56:59.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:56:59.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:59.483594+0000) 2024-03-06T12:57:00.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:00.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:00.483862+0000) 2024-03-06T12:57:01.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:01.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:01.484108+0000) 2024-03-06T12:57:02.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:02.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:02.484351+0000) 2024-03-06T12:57:03.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:03.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:03.484602+0000) 2024-03-06T12:57:04.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:04.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:04.484938+0000) 2024-03-06T12:57:05.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:05.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:05.485204+0000) 2024-03-06T12:57:06.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:06.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:06.485461+0000) 2024-03-06T12:57:07.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:07.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:07.485741+0000) 2024-03-06T12:57:08.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:08.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:08.485951+0000) 2024-03-06T12:57:09.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:09.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:09.486199+0000) 2024-03-06T12:57:10.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:10.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:10.486467+0000) 2024-03-06T12:57:11.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:11.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:11.486744+0000) 2024-03-06T12:57:12.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:12.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:12.487002+0000) 2024-03-06T12:57:13.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:13.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:13.487258+0000) 2024-03-06T12:57:14.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:14.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:14.487517+0000) 2024-03-06T12:57:15.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:15.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:15.487760+0000) 2024-03-06T12:57:16.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:16.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:16.488003+0000) 2024-03-06T12:57:17.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:17.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:17.488245+0000) 2024-03-06T12:57:18.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:18.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:18.488498+0000) 2024-03-06T12:57:19.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:19.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:19.488737+0000) 2024-03-06T12:57:20.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:20.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:20.489007+0000) 2024-03-06T12:57:21.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:21.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:21.489263+0000) 2024-03-06T12:57:22.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:22.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:22.489469+0000) 2024-03-06T12:57:23.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:23.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:23.489751+0000) 2024-03-06T12:57:24.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:24.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:24.489994+0000) 2024-03-06T12:57:25.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:25.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:25.490212+0000) 2024-03-06T12:57:26.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:26.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:26.490490+0000) 2024-03-06T12:57:27.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:27.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:27.490733+0000) 2024-03-06T12:57:27.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:27.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:28.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:28.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:28.490969+0000) 2024-03-06T12:57:29.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:29.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:29.491214+0000) 2024-03-06T12:57:30.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:30.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:30.491488+0000) 2024-03-06T12:57:31.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:31.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:31.491739+0000) 2024-03-06T12:57:32.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:32.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:32.491993+0000) 2024-03-06T12:57:33.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:33.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:33.492248+0000) 2024-03-06T12:57:34.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:34.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:34.492490+0000) 2024-03-06T12:57:35.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:35.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:35.492760+0000) 2024-03-06T12:57:36.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:36.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:36.493019+0000) 2024-03-06T12:57:37.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:37.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:37.493252+0000) 2024-03-06T12:57:38.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:38.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:38.493489+0000) 2024-03-06T12:57:39.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:39.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:39.493732+0000) 2024-03-06T12:57:40.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:40.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:40.493943+0000) 2024-03-06T12:57:41.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:41.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:41.494169+0000) 2024-03-06T12:57:42.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:42.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:42.494426+0000) 2024-03-06T12:57:43.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:43.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:43.494729+0000) 2024-03-06T12:57:44.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:44.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:44.494994+0000) 2024-03-06T12:57:45.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:45.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:45.495258+0000) 2024-03-06T12:57:46.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:46.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:46.495499+0000) 2024-03-06T12:57:47.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:47.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:47.495729+0000) 2024-03-06T12:57:48.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:48.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:48.495978+0000) 2024-03-06T12:57:49.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:49.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:49.496239+0000) 2024-03-06T12:57:50.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:50.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:50.496508+0000) 2024-03-06T12:57:51.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:51.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:51.496759+0000) 2024-03-06T12:57:52.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:52.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:52.496998+0000) 2024-03-06T12:57:53.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:53.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:53.497249+0000) 2024-03-06T12:57:54.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:54.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:54.497507+0000) 2024-03-06T12:57:55.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:55.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:55.497777+0000) 2024-03-06T12:57:56.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:56.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:56.498052+0000) 2024-03-06T12:57:57.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:57.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:57.498360+0000) 2024-03-06T12:57:57.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:57.931 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:58.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:58.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:58.498593+0000) 2024-03-06T12:57:59.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:57:59.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:59.498892+0000) 2024-03-06T12:58:00.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:00.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:00.499162+0000) 2024-03-06T12:58:01.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:01.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:01.499344+0000) 2024-03-06T12:58:02.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:02.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:02.499600+0000) 2024-03-06T12:58:03.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:03.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:03.499875+0000) 2024-03-06T12:58:04.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:04.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:04.500140+0000) 2024-03-06T12:58:05.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:05.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:05.500403+0000) 2024-03-06T12:58:06.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:06.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:06.500699+0000) 2024-03-06T12:58:07.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:07.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:07.500927+0000) 2024-03-06T12:58:08.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:08.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:08.501173+0000) 2024-03-06T12:58:09.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:09.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:09.501433+0000) 2024-03-06T12:58:10.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:10.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:10.501742+0000) 2024-03-06T12:58:11.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:11.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:11.501993+0000) 2024-03-06T12:58:12.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:12.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:12.502235+0000) 2024-03-06T12:58:13.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:13.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:13.502476+0000) 2024-03-06T12:58:14.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:14.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:14.502752+0000) 2024-03-06T12:58:15.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:15.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:15.502985+0000) 2024-03-06T12:58:16.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:16.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:16.503244+0000) 2024-03-06T12:58:17.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:17.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:17.503497+0000) 2024-03-06T12:58:18.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:18.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:18.503756+0000) 2024-03-06T12:58:19.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:19.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:19.504017+0000) 2024-03-06T12:58:20.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:20.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:20.504281+0000) 2024-03-06T12:58:21.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:21.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:21.504551+0000) 2024-03-06T12:58:22.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:22.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:22.504759+0000) 2024-03-06T12:58:23.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:23.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:23.505035+0000) 2024-03-06T12:58:24.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:24.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:24.505299+0000) 2024-03-06T12:58:25.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:25.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:25.505542+0000) 2024-03-06T12:58:26.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:26.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:26.505801+0000) 2024-03-06T12:58:27.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:27.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:27.506066+0000) 2024-03-06T12:58:28.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:28.087 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:28.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:28.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:28.506349+0000) 2024-03-06T12:58:29.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:29.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:29.506617+0000) 2024-03-06T12:58:30.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:30.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:30.506874+0000) 2024-03-06T12:58:31.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:31.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:31.507083+0000) 2024-03-06T12:58:32.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:32.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:32.507361+0000) 2024-03-06T12:58:33.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:33.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:33.507566+0000) 2024-03-06T12:58:34.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:34.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:34.507853+0000) 2024-03-06T12:58:35.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:35.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:35.508098+0000) 2024-03-06T12:58:36.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:36.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:36.508352+0000) 2024-03-06T12:58:37.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:37.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:37.508606+0000) 2024-03-06T12:58:38.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:38.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:38.508925+0000) 2024-03-06T12:58:39.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:39.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:39.509214+0000) 2024-03-06T12:58:40.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:40.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:40.509496+0000) 2024-03-06T12:58:41.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:41.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:41.509757+0000) 2024-03-06T12:58:42.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:42.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:42.510040+0000) 2024-03-06T12:58:43.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:43.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:43.510315+0000) 2024-03-06T12:58:44.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:44.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:44.510558+0000) 2024-03-06T12:58:45.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:45.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:45.510809+0000) 2024-03-06T12:58:46.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:46.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:46.511064+0000) 2024-03-06T12:58:47.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:47.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:47.511298+0000) 2024-03-06T12:58:48.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:48.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:48.511551+0000) 2024-03-06T12:58:49.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:49.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:49.511826+0000) 2024-03-06T12:58:50.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:50.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:50.512109+0000) 2024-03-06T12:58:51.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:51.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:51.512383+0000) 2024-03-06T12:58:52.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:52.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:52.512682+0000) 2024-03-06T12:58:53.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:53.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:53.512946+0000) 2024-03-06T12:58:54.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:54.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:54.513221+0000) 2024-03-06T12:58:55.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:55.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:55.513497+0000) 2024-03-06T12:58:56.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:56.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:56.513777+0000) 2024-03-06T12:58:57.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:57.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:57.514051+0000) 2024-03-06T12:58:58.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:58.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:58.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:58.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:58.514337+0000) 2024-03-06T12:58:59.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:58:59.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:59.514614+0000) 2024-03-06T12:59:00.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:00.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:00.514923+0000) 2024-03-06T12:59:01.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:01.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:01.515197+0000) 2024-03-06T12:59:02.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:02.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:02.515440+0000) 2024-03-06T12:59:03.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:03.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:03.515731+0000) 2024-03-06T12:59:04.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:04.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:04.515975+0000) 2024-03-06T12:59:05.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:05.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:05.516219+0000) 2024-03-06T12:59:06.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:06.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:06.516472+0000) 2024-03-06T12:59:07.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:07.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:07.516722+0000) 2024-03-06T12:59:08.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:08.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:08.516912+0000) 2024-03-06T12:59:09.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:09.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:09.517169+0000) 2024-03-06T12:59:10.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:10.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:10.517430+0000) 2024-03-06T12:59:11.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:11.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:11.517620+0000) 2024-03-06T12:59:12.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:12.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:12.517867+0000) 2024-03-06T12:59:13.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:13.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:13.518107+0000) 2024-03-06T12:59:14.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:14.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:14.518343+0000) 2024-03-06T12:59:15.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:15.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:15.518571+0000) 2024-03-06T12:59:16.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:16.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:16.518806+0000) 2024-03-06T12:59:17.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:17.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:17.519008+0000) 2024-03-06T12:59:18.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:18.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:18.519239+0000) 2024-03-06T12:59:19.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:19.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:19.519469+0000) 2024-03-06T12:59:20.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:20.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:20.519753+0000) 2024-03-06T12:59:21.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:21.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:21.519985+0000) 2024-03-06T12:59:22.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:22.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:22.520220+0000) 2024-03-06T12:59:23.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:23.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:23.520462+0000) 2024-03-06T12:59:24.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:24.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:24.520754+0000) 2024-03-06T12:59:25.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:25.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:25.521000+0000) 2024-03-06T12:59:26.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:26.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:26.521249+0000) 2024-03-06T12:59:27.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:27.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:27.521503+0000) 2024-03-06T12:59:28.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:28.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:28.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:28.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:28.521764+0000) 2024-03-06T12:59:29.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:29.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:29.522006+0000) 2024-03-06T12:59:30.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:30.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:30.522262+0000) 2024-03-06T12:59:31.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:31.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:31.522521+0000) 2024-03-06T12:59:32.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:32.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:32.522778+0000) 2024-03-06T12:59:33.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:33.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:33.523073+0000) 2024-03-06T12:59:34.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:34.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:34.523323+0000) 2024-03-06T12:59:35.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:35.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:35.523545+0000) 2024-03-06T12:59:36.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:36.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:36.523807+0000) 2024-03-06T12:59:37.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:37.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:37.524050+0000) 2024-03-06T12:59:38.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:38.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:38.524249+0000) 2024-03-06T12:59:39.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:39.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:39.524500+0000) 2024-03-06T12:59:40.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:40.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:40.524750+0000) 2024-03-06T12:59:41.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:41.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:41.525009+0000) 2024-03-06T12:59:42.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:42.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:42.525251+0000) 2024-03-06T12:59:43.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:43.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:43.525491+0000) 2024-03-06T12:59:44.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:44.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:44.525745+0000) 2024-03-06T12:59:45.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:45.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:45.525982+0000) 2024-03-06T12:59:46.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:46.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:46.526239+0000) 2024-03-06T12:59:47.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:47.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:47.526459+0000) 2024-03-06T12:59:48.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:48.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:48.526763+0000) 2024-03-06T12:59:49.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:49.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:49.526970+0000) 2024-03-06T12:59:50.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:50.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:50.527211+0000) 2024-03-06T12:59:51.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:51.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:51.527452+0000) 2024-03-06T12:59:52.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:52.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:52.527755+0000) 2024-03-06T12:59:53.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:53.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:53.528017+0000) 2024-03-06T12:59:54.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:54.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:54.528258+0000) 2024-03-06T12:59:55.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:55.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:55.528517+0000) 2024-03-06T12:59:56.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:56.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:56.528758+0000) 2024-03-06T12:59:57.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:57.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:57.528942+0000) 2024-03-06T12:59:58.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:58.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:58.529183+0000) 2024-03-06T12:59:58.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:58.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:59.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T12:59:59.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:59.529436+0000) 2024-03-06T13:00:00.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:00.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:00.529732+0000) 2024-03-06T13:00:01.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:01.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:01.529974+0000) 2024-03-06T13:00:02.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:02.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:02.530218+0000) 2024-03-06T13:00:03.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:03.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:03.530453+0000) 2024-03-06T13:00:04.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:04.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:04.530735+0000) 2024-03-06T13:00:05.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:05.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:05.530988+0000) 2024-03-06T13:00:06.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:06.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:06.531237+0000) 2024-03-06T13:00:07.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:07.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:07.531481+0000) 2024-03-06T13:00:08.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:08.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:08.531745+0000) 2024-03-06T13:00:09.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:09.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:09.532006+0000) 2024-03-06T13:00:10.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:10.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:10.532246+0000) 2024-03-06T13:00:11.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:11.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:11.532431+0000) 2024-03-06T13:00:12.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:12.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:12.532765+0000) 2024-03-06T13:00:13.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:13.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:13.533058+0000) 2024-03-06T13:00:14.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:14.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:14.533344+0000) 2024-03-06T13:00:15.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:15.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:15.533709+0000) 2024-03-06T13:00:16.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:16.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:16.534008+0000) 2024-03-06T13:00:17.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:17.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:17.534302+0000) 2024-03-06T13:00:18.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:18.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:18.534603+0000) 2024-03-06T13:00:19.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:19.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:19.535011+0000) 2024-03-06T13:00:20.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:20.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:20.535323+0000) 2024-03-06T13:00:21.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:21.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:21.535685+0000) 2024-03-06T13:00:22.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:22.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:22.535983+0000) 2024-03-06T13:00:23.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:23.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:23.536280+0000) 2024-03-06T13:00:24.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:24.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:24.536591+0000) 2024-03-06T13:00:25.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:25.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:25.536954+0000) 2024-03-06T13:00:26.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:26.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:26.537296+0000) 2024-03-06T13:00:27.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:27.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:27.537611+0000) 2024-03-06T13:00:28.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:28.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:28.537920+0000) 2024-03-06T13:00:28.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:28.700 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:29.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:29.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:29.538236+0000) 2024-03-06T13:00:30.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:30.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:30.538542+0000) 2024-03-06T13:00:31.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:31.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:31.538853+0000) 2024-03-06T13:00:32.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:32.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:32.539164+0000) 2024-03-06T13:00:33.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:33.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:33.539477+0000) 2024-03-06T13:00:34.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:34.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:34.539778+0000) 2024-03-06T13:00:35.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:35.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:35.540085+0000) 2024-03-06T13:00:36.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:36.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:36.540386+0000) 2024-03-06T13:00:37.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:37.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:37.540758+0000) 2024-03-06T13:00:38.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:38.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:38.541046+0000) 2024-03-06T13:00:39.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:39.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:39.541351+0000) 2024-03-06T13:00:40.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:40.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:40.541597+0000) 2024-03-06T13:00:41.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:41.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:41.541944+0000) 2024-03-06T13:00:42.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:42.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:42.542208+0000) 2024-03-06T13:00:43.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:43.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:43.542487+0000) 2024-03-06T13:00:44.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:44.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:44.542800+0000) 2024-03-06T13:00:45.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:45.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:45.543116+0000) 2024-03-06T13:00:46.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:46.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:46.543439+0000) 2024-03-06T13:00:47.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:47.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:47.543768+0000) 2024-03-06T13:00:48.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:48.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:48.544072+0000) 2024-03-06T13:00:49.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:49.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:49.544396+0000) 2024-03-06T13:00:50.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:50.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:50.544747+0000) 2024-03-06T13:00:51.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:51.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:51.545051+0000) 2024-03-06T13:00:52.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:52.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:52.545296+0000) 2024-03-06T13:00:53.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:53.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:53.545588+0000) 2024-03-06T13:00:54.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:54.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:54.545910+0000) 2024-03-06T13:00:55.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:55.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:55.546163+0000) 2024-03-06T13:00:56.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:56.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:56.546428+0000) 2024-03-06T13:00:57.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:57.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:57.546704+0000) 2024-03-06T13:00:58.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:58.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:58.546969+0000) 2024-03-06T13:00:58.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:58.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:59.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:00:59.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:59.547174+0000) 2024-03-06T13:01:00.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:00.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:00.547424+0000) 2024-03-06T13:01:01.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:01.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:01.547750+0000) 2024-03-06T13:01:02.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:02.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:02.548036+0000) 2024-03-06T13:01:03.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:03.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:03.548296+0000) 2024-03-06T13:01:04.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:04.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:04.548573+0000) 2024-03-06T13:01:05.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:05.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:05.548865+0000) 2024-03-06T13:01:06.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:06.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:06.549121+0000) 2024-03-06T13:01:07.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:07.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:07.549382+0000) 2024-03-06T13:01:08.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:08.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:08.549667+0000) 2024-03-06T13:01:09.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:09.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:09.549964+0000) 2024-03-06T13:01:10.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:10.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:10.550242+0000) 2024-03-06T13:01:11.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:11.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:11.550509+0000) 2024-03-06T13:01:12.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:12.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:12.550790+0000) 2024-03-06T13:01:13.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:13.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:13.551050+0000) 2024-03-06T13:01:14.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:14.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:14.551298+0000) 2024-03-06T13:01:15.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:15.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:15.551535+0000) 2024-03-06T13:01:16.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:16.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:16.551797+0000) 2024-03-06T13:01:17.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:17.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:17.552039+0000) 2024-03-06T13:01:18.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:18.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:18.552297+0000) 2024-03-06T13:01:19.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:19.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:19.552554+0000) 2024-03-06T13:01:20.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:20.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:20.552817+0000) 2024-03-06T13:01:21.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:21.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:21.553068+0000) 2024-03-06T13:01:22.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:22.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:22.553317+0000) 2024-03-06T13:01:23.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:23.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:23.553521+0000) 2024-03-06T13:01:24.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:24.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:24.553790+0000) 2024-03-06T13:01:25.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:25.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:25.554024+0000) 2024-03-06T13:01:26.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:26.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:26.554274+0000) 2024-03-06T13:01:27.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:27.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:27.554496+0000) 2024-03-06T13:01:28.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:28.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:28.554780+0000) 2024-03-06T13:01:29.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:29.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:29.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:29.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:29.555058+0000) 2024-03-06T13:01:30.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:30.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:30.555320+0000) 2024-03-06T13:01:31.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:31.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:31.555554+0000) 2024-03-06T13:01:32.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:32.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:32.555750+0000) 2024-03-06T13:01:33.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:33.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:33.555936+0000) 2024-03-06T13:01:34.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:34.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:34.556213+0000) 2024-03-06T13:01:35.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:35.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:35.556450+0000) 2024-03-06T13:01:36.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:36.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:36.556780+0000) 2024-03-06T13:01:37.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:37.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:37.557030+0000) 2024-03-06T13:01:38.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:38.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:38.557345+0000) 2024-03-06T13:01:39.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:39.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:39.557713+0000) 2024-03-06T13:01:40.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:40.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:40.558033+0000) 2024-03-06T13:01:41.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:41.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:41.558329+0000) 2024-03-06T13:01:42.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:42.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:42.558674+0000) 2024-03-06T13:01:43.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:43.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:43.558973+0000) 2024-03-06T13:01:44.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:44.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:44.559282+0000) 2024-03-06T13:01:45.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:45.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:45.559572+0000) 2024-03-06T13:01:46.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:46.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:46.559862+0000) 2024-03-06T13:01:47.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:47.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:47.560180+0000) 2024-03-06T13:01:48.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:48.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:48.560411+0000) 2024-03-06T13:01:49.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:49.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:49.560753+0000) 2024-03-06T13:01:50.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:50.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:50.561055+0000) 2024-03-06T13:01:51.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:51.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:51.561375+0000) 2024-03-06T13:01:52.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:52.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:52.561735+0000) 2024-03-06T13:01:53.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:53.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:53.562043+0000) 2024-03-06T13:01:54.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:54.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:54.562331+0000) 2024-03-06T13:01:55.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:55.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:55.562614+0000) 2024-03-06T13:01:56.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:56.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:56.562980+0000) 2024-03-06T13:01:57.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:57.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:57.563264+0000) 2024-03-06T13:01:58.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:58.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:58.563547+0000) 2024-03-06T13:01:59.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:59.158 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:59.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:01:59.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:59.563868+0000) 2024-03-06T13:02:00.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:00.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:00.564111+0000) 2024-03-06T13:02:01.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:01.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:01.564446+0000) 2024-03-06T13:02:02.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:02.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:02.564843+0000) 2024-03-06T13:02:03.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:03.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:03.565143+0000) 2024-03-06T13:02:04.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:04.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:04.565443+0000) 2024-03-06T13:02:05.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:05.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:05.565795+0000) 2024-03-06T13:02:06.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:06.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:06.566117+0000) 2024-03-06T13:02:07.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:07.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:07.566359+0000) 2024-03-06T13:02:08.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:08.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:08.566681+0000) 2024-03-06T13:02:09.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:09.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:09.566982+0000) 2024-03-06T13:02:10.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:10.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:10.567284+0000) 2024-03-06T13:02:11.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:11.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:11.567592+0000) 2024-03-06T13:02:12.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:12.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:12.567921+0000) 2024-03-06T13:02:13.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:13.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:13.568222+0000) 2024-03-06T13:02:14.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:14.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:14.568519+0000) 2024-03-06T13:02:15.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:15.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:15.568840+0000) 2024-03-06T13:02:16.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:16.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:16.569165+0000) 2024-03-06T13:02:17.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:17.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:17.569471+0000) 2024-03-06T13:02:18.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:18.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:18.569791+0000) 2024-03-06T13:02:19.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:19.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:19.570095+0000) 2024-03-06T13:02:20.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:20.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:20.570407+0000) 2024-03-06T13:02:21.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:21.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:21.570756+0000) 2024-03-06T13:02:22.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:22.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:22.570994+0000) 2024-03-06T13:02:23.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:23.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:23.571294+0000) 2024-03-06T13:02:24.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:24.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:24.571591+0000) 2024-03-06T13:02:25.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:25.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:25.571956+0000) 2024-03-06T13:02:26.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:26.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:26.572251+0000) 2024-03-06T13:02:27.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:27.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:27.572552+0000) 2024-03-06T13:02:28.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:28.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:28.572810+0000) 2024-03-06T13:02:29.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:29.313 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:29.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:29.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:29.573091+0000) 2024-03-06T13:02:30.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:30.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:30.573348+0000) 2024-03-06T13:02:31.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:31.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:31.573620+0000) 2024-03-06T13:02:32.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:32.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:32.573893+0000) 2024-03-06T13:02:33.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:33.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:33.574145+0000) 2024-03-06T13:02:34.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:34.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:34.574387+0000) 2024-03-06T13:02:35.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:35.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:35.574671+0000) 2024-03-06T13:02:36.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:36.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:36.574861+0000) 2024-03-06T13:02:37.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:37.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:37.575097+0000) 2024-03-06T13:02:38.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:38.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:38.575275+0000) 2024-03-06T13:02:39.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:39.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:39.575527+0000) 2024-03-06T13:02:40.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:40.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:40.575764+0000) 2024-03-06T13:02:41.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:41.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:41.575971+0000) 2024-03-06T13:02:42.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:42.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:42.576225+0000) 2024-03-06T13:02:43.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:43.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:43.576478+0000) 2024-03-06T13:02:44.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:44.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:44.576748+0000) 2024-03-06T13:02:45.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:45.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:45.577006+0000) 2024-03-06T13:02:46.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:46.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:46.577201+0000) 2024-03-06T13:02:47.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:47.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:47.577446+0000) 2024-03-06T13:02:48.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:48.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:48.577732+0000) 2024-03-06T13:02:49.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:49.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:49.577928+0000) 2024-03-06T13:02:50.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:50.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:50.578181+0000) 2024-03-06T13:02:51.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:51.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:51.578422+0000) 2024-03-06T13:02:52.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:52.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:52.578728+0000) 2024-03-06T13:02:53.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:53.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:53.578959+0000) 2024-03-06T13:02:54.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:54.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:54.579214+0000) 2024-03-06T13:02:55.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:55.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:55.579447+0000) 2024-03-06T13:02:56.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:56.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:56.579739+0000) 2024-03-06T13:02:57.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:57.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:57.579980+0000) 2024-03-06T13:02:58.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:58.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:58.580231+0000) 2024-03-06T13:02:59.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:59.465 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:59.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:02:59.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:59.580450+0000) 2024-03-06T13:03:00.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:00.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:00.580748+0000) 2024-03-06T13:03:01.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:01.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:01.580995+0000) 2024-03-06T13:03:02.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:02.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:02.581250+0000) 2024-03-06T13:03:03.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:03.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:03.581507+0000) 2024-03-06T13:03:04.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:04.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:04.581774+0000) 2024-03-06T13:03:05.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:05.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:05.582033+0000) 2024-03-06T13:03:06.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:06.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:06.582285+0000) 2024-03-06T13:03:07.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:07.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:07.582540+0000) 2024-03-06T13:03:08.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:08.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:08.582774+0000) 2024-03-06T13:03:09.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:09.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:09.583029+0000) 2024-03-06T13:03:10.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:10.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:10.583275+0000) 2024-03-06T13:03:11.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:11.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:11.583516+0000) 2024-03-06T13:03:12.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:12.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:12.583753+0000) 2024-03-06T13:03:13.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:13.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:13.584013+0000) 2024-03-06T13:03:14.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:14.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:14.584267+0000) 2024-03-06T13:03:15.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:15.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:15.584527+0000) 2024-03-06T13:03:16.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:16.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:16.584759+0000) 2024-03-06T13:03:17.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:17.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:17.585020+0000) 2024-03-06T13:03:18.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:18.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:18.585272+0000) 2024-03-06T13:03:19.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:19.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:19.585556+0000) 2024-03-06T13:03:20.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:20.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:20.585815+0000) 2024-03-06T13:03:21.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:21.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:21.586053+0000) 2024-03-06T13:03:22.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:22.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:22.586289+0000) 2024-03-06T13:03:23.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:23.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:23.586549+0000) 2024-03-06T13:03:24.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:24.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:24.586812+0000) 2024-03-06T13:03:25.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:25.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:25.587069+0000) 2024-03-06T13:03:26.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:26.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:26.587327+0000) 2024-03-06T13:03:27.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:27.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:27.587592+0000) 2024-03-06T13:03:28.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:28.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:28.587814+0000) 2024-03-06T13:03:29.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:29.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:29.588083+0000) 2024-03-06T13:03:29.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:29.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:30.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:30.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:30.588324+0000) 2024-03-06T13:03:31.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:31.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:31.588577+0000) 2024-03-06T13:03:32.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:32.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:32.588832+0000) 2024-03-06T13:03:33.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:33.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:33.589026+0000) 2024-03-06T13:03:34.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:34.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:34.589288+0000) 2024-03-06T13:03:35.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:35.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:35.589544+0000) 2024-03-06T13:03:36.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:36.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:36.589804+0000) 2024-03-06T13:03:37.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:37.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:37.590064+0000) 2024-03-06T13:03:38.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:38.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:38.590300+0000) 2024-03-06T13:03:39.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:39.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:39.590553+0000) 2024-03-06T13:03:40.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:40.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:40.590792+0000) 2024-03-06T13:03:41.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:41.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:41.591087+0000) 2024-03-06T13:03:42.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:42.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:42.591356+0000) 2024-03-06T13:03:43.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:43.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:43.591612+0000) 2024-03-06T13:03:44.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:44.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:44.591956+0000) 2024-03-06T13:03:45.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:45.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:45.592163+0000) 2024-03-06T13:03:46.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:46.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:46.592419+0000) 2024-03-06T13:03:47.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:47.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:47.592722+0000) 2024-03-06T13:03:48.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:48.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:48.592988+0000) 2024-03-06T13:03:49.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:49.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:49.593198+0000) 2024-03-06T13:03:50.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:50.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:50.593441+0000) 2024-03-06T13:03:51.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:51.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:51.593724+0000) 2024-03-06T13:03:52.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:52.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:52.593906+0000) 2024-03-06T13:03:53.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:53.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:53.594144+0000) 2024-03-06T13:03:54.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:54.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:54.594386+0000) 2024-03-06T13:03:55.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:55.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:55.594689+0000) 2024-03-06T13:03:56.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:56.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:56.594949+0000) 2024-03-06T13:03:57.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:57.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:57.595190+0000) 2024-03-06T13:03:58.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:58.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:58.595372+0000) 2024-03-06T13:03:59.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:03:59.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:59.595659+0000) 2024-03-06T13:03:59.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:59.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:00.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:00.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:00.595904+0000) 2024-03-06T13:04:01.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:01.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:01.596162+0000) 2024-03-06T13:04:02.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:02.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:02.596411+0000) 2024-03-06T13:04:03.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:03.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:03.596731+0000) 2024-03-06T13:04:04.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:04.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:04.596977+0000) 2024-03-06T13:04:05.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:05.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:05.597220+0000) 2024-03-06T13:04:06.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:06.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:06.597483+0000) 2024-03-06T13:04:07.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:07.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:07.597752+0000) 2024-03-06T13:04:08.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:08.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:08.597989+0000) 2024-03-06T13:04:09.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:09.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:09.598252+0000) 2024-03-06T13:04:10.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:10.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:10.598503+0000) 2024-03-06T13:04:11.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:11.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:11.598777+0000) 2024-03-06T13:04:12.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:12.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:12.599033+0000) 2024-03-06T13:04:13.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:13.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:13.599270+0000) 2024-03-06T13:04:14.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:14.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:14.599530+0000) 2024-03-06T13:04:15.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:15.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:15.599792+0000) 2024-03-06T13:04:16.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:16.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:16.600046+0000) 2024-03-06T13:04:17.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:17.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:17.600307+0000) 2024-03-06T13:04:18.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:18.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:18.600561+0000) 2024-03-06T13:04:19.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:19.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:19.600820+0000) 2024-03-06T13:04:20.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:20.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:20.601053+0000) 2024-03-06T13:04:21.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:21.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:21.601327+0000) 2024-03-06T13:04:22.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:22.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:22.601581+0000) 2024-03-06T13:04:23.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:23.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:23.601828+0000) 2024-03-06T13:04:24.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:24.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:24.602089+0000) 2024-03-06T13:04:25.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:25.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:25.602350+0000) 2024-03-06T13:04:26.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:26.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:26.602602+0000) 2024-03-06T13:04:27.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:27.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:27.602882+0000) 2024-03-06T13:04:28.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:28.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:28.603132+0000) 2024-03-06T13:04:29.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:29.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:29.603402+0000) 2024-03-06T13:04:29.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:29.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:30.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:30.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:30.603675+0000) 2024-03-06T13:04:31.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:31.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:31.603920+0000) 2024-03-06T13:04:32.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:32.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:32.604208+0000) 2024-03-06T13:04:33.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:33.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:33.604447+0000) 2024-03-06T13:04:34.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:34.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:34.604759+0000) 2024-03-06T13:04:35.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:35.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:35.605029+0000) 2024-03-06T13:04:36.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:36.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:36.605261+0000) 2024-03-06T13:04:37.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:37.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:37.605508+0000) 2024-03-06T13:04:38.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:38.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:38.605741+0000) 2024-03-06T13:04:39.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:39.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:39.606011+0000) 2024-03-06T13:04:40.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:40.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:40.606272+0000) 2024-03-06T13:04:41.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:41.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:41.606536+0000) 2024-03-06T13:04:42.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:42.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:42.606775+0000) 2024-03-06T13:04:43.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:43.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:43.607009+0000) 2024-03-06T13:04:44.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:44.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:44.607273+0000) 2024-03-06T13:04:45.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:45.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:45.607521+0000) 2024-03-06T13:04:46.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:46.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:46.607763+0000) 2024-03-06T13:04:47.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:47.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:47.608019+0000) 2024-03-06T13:04:48.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:48.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:48.608214+0000) 2024-03-06T13:04:49.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:49.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:49.608465+0000) 2024-03-06T13:04:50.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:50.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:50.608753+0000) 2024-03-06T13:04:51.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:51.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:51.609015+0000) 2024-03-06T13:04:52.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:52.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:52.609203+0000) 2024-03-06T13:04:53.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:53.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:53.609406+0000) 2024-03-06T13:04:54.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:54.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:54.609689+0000) 2024-03-06T13:04:55.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:55.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:55.609958+0000) 2024-03-06T13:04:56.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:56.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:56.610201+0000) 2024-03-06T13:04:57.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:57.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:57.610427+0000) 2024-03-06T13:04:58.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:58.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:58.610700+0000) 2024-03-06T13:04:59.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:04:59.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:59.610958+0000) 2024-03-06T13:05:00.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:00.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:00.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:00.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:00.611171+0000) 2024-03-06T13:05:01.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:01.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:01.611413+0000) 2024-03-06T13:05:02.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:02.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:02.611724+0000) 2024-03-06T13:05:03.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:03.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:03.611941+0000) 2024-03-06T13:05:04.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:04.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:04.612214+0000) 2024-03-06T13:05:05.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:05.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:05.612495+0000) 2024-03-06T13:05:06.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:06.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:06.612756+0000) 2024-03-06T13:05:07.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:07.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:07.613010+0000) 2024-03-06T13:05:08.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:08.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:08.613261+0000) 2024-03-06T13:05:09.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:09.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:09.613524+0000) 2024-03-06T13:05:10.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:10.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:10.613786+0000) 2024-03-06T13:05:11.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:11.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:11.613983+0000) 2024-03-06T13:05:12.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:12.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:12.614245+0000) 2024-03-06T13:05:13.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:13.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:13.614490+0000) 2024-03-06T13:05:14.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:14.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:14.614734+0000) 2024-03-06T13:05:15.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:15.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:15.614997+0000) 2024-03-06T13:05:16.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:16.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:16.615255+0000) 2024-03-06T13:05:17.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:17.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:17.615501+0000) 2024-03-06T13:05:18.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:18.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:18.615762+0000) 2024-03-06T13:05:19.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:19.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:19.616010+0000) 2024-03-06T13:05:20.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:20.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:20.616209+0000) 2024-03-06T13:05:21.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:21.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:21.616472+0000) 2024-03-06T13:05:22.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:22.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:22.616754+0000) 2024-03-06T13:05:23.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:23.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:23.617018+0000) 2024-03-06T13:05:24.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:24.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:24.617277+0000) 2024-03-06T13:05:25.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:25.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:25.617587+0000) 2024-03-06T13:05:26.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:26.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:26.617741+0000) 2024-03-06T13:05:27.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:27.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:27.618000+0000) 2024-03-06T13:05:28.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:28.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:28.618233+0000) 2024-03-06T13:05:29.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:29.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:29.618499+0000) 2024-03-06T13:05:30.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:30.210 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:30.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:30.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:30.618781+0000) 2024-03-06T13:05:31.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:31.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:31.619027+0000) 2024-03-06T13:05:32.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:32.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:32.619279+0000) 2024-03-06T13:05:33.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:33.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:33.619514+0000) 2024-03-06T13:05:34.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:34.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:34.619761+0000) 2024-03-06T13:05:35.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:35.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:35.620005+0000) 2024-03-06T13:05:36.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:36.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:36.620241+0000) 2024-03-06T13:05:37.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:37.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:37.620503+0000) 2024-03-06T13:05:38.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:38.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:38.620746+0000) 2024-03-06T13:05:39.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:39.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:39.621010+0000) 2024-03-06T13:05:40.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:40.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:40.621246+0000) 2024-03-06T13:05:41.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:41.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:41.621502+0000) 2024-03-06T13:05:42.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:42.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:42.621763+0000) 2024-03-06T13:05:43.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:43.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:43.622017+0000) 2024-03-06T13:05:44.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:44.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:44.622252+0000) 2024-03-06T13:05:45.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:45.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:45.622500+0000) 2024-03-06T13:05:46.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:46.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:46.622757+0000) 2024-03-06T13:05:47.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:47.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:47.622997+0000) 2024-03-06T13:05:48.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:48.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:48.623251+0000) 2024-03-06T13:05:49.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:49.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:49.623504+0000) 2024-03-06T13:05:50.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:50.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:50.623756+0000) 2024-03-06T13:05:51.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:51.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:51.623993+0000) 2024-03-06T13:05:52.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:52.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:52.624257+0000) 2024-03-06T13:05:53.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:53.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:53.624488+0000) 2024-03-06T13:05:54.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:54.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:54.624708+0000) 2024-03-06T13:05:55.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:55.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:55.624955+0000) 2024-03-06T13:05:56.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:56.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:56.625201+0000) 2024-03-06T13:05:57.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:57.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:57.625479+0000) 2024-03-06T13:05:58.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:58.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:58.625730+0000) 2024-03-06T13:05:59.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:05:59.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:59.626002+0000) 2024-03-06T13:06:00.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:00.356 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:00.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:00.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:00.626278+0000) 2024-03-06T13:06:01.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:01.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:01.626474+0000) 2024-03-06T13:06:02.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:02.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:02.626739+0000) 2024-03-06T13:06:03.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:03.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:03.626975+0000) 2024-03-06T13:06:04.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:04.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:04.627226+0000) 2024-03-06T13:06:05.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:05.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:05.627462+0000) 2024-03-06T13:06:06.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:06.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:06.627734+0000) 2024-03-06T13:06:07.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:07.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:07.627930+0000) 2024-03-06T13:06:08.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:08.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:08.628171+0000) 2024-03-06T13:06:09.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:09.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:09.628431+0000) 2024-03-06T13:06:10.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:10.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:10.628730+0000) 2024-03-06T13:06:11.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:11.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:11.628970+0000) 2024-03-06T13:06:12.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:12.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:12.629215+0000) 2024-03-06T13:06:13.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:13.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:13.629507+0000) 2024-03-06T13:06:14.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:14.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:14.629761+0000) 2024-03-06T13:06:15.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:15.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:15.630019+0000) 2024-03-06T13:06:16.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:16.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:16.630275+0000) 2024-03-06T13:06:17.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:17.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:17.630524+0000) 2024-03-06T13:06:18.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:18.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:18.630777+0000) 2024-03-06T13:06:19.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:19.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:19.631042+0000) 2024-03-06T13:06:20.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:20.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:20.631294+0000) 2024-03-06T13:06:21.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:21.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:21.631552+0000) 2024-03-06T13:06:22.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:22.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:22.631816+0000) 2024-03-06T13:06:23.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:23.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:23.632052+0000) 2024-03-06T13:06:24.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:24.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:24.632304+0000) 2024-03-06T13:06:25.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:25.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:25.632560+0000) 2024-03-06T13:06:26.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:26.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:26.632795+0000) 2024-03-06T13:06:27.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:27.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:27.633045+0000) 2024-03-06T13:06:28.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:28.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:28.633284+0000) 2024-03-06T13:06:29.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:29.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:29.633542+0000) 2024-03-06T13:06:30.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:30.516 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:30.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:30.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:30.633812+0000) 2024-03-06T13:06:31.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:31.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:31.634079+0000) 2024-03-06T13:06:32.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:32.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:32.634350+0000) 2024-03-06T13:06:33.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:33.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:33.634593+0000) 2024-03-06T13:06:34.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:34.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:34.634835+0000) 2024-03-06T13:06:35.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:35.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:35.635080+0000) 2024-03-06T13:06:36.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:36.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:36.635327+0000) 2024-03-06T13:06:37.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:37.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:37.635576+0000) 2024-03-06T13:06:38.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:38.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:38.635834+0000) 2024-03-06T13:06:39.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:39.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:39.636105+0000) 2024-03-06T13:06:40.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:40.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:40.636345+0000) 2024-03-06T13:06:41.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:41.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:41.636608+0000) 2024-03-06T13:06:42.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:42.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:42.636845+0000) 2024-03-06T13:06:43.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:43.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:43.637097+0000) 2024-03-06T13:06:44.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:44.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:44.637346+0000) 2024-03-06T13:06:45.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:45.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:45.637607+0000) 2024-03-06T13:06:46.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:46.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:46.637893+0000) 2024-03-06T13:06:47.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:47.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:47.638161+0000) 2024-03-06T13:06:48.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:48.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:48.638400+0000) 2024-03-06T13:06:49.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:49.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:49.638594+0000) 2024-03-06T13:06:50.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:50.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:50.638839+0000) 2024-03-06T13:06:51.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:51.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:51.639096+0000) 2024-03-06T13:06:52.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:52.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:52.639362+0000) 2024-03-06T13:06:53.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:53.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:53.639625+0000) 2024-03-06T13:06:54.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:54.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:54.639854+0000) 2024-03-06T13:06:55.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:55.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:55.640116+0000) 2024-03-06T13:06:56.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:56.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:56.640308+0000) 2024-03-06T13:06:57.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:57.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:57.640556+0000) 2024-03-06T13:06:58.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:58.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:58.640788+0000) 2024-03-06T13:06:59.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:06:59.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:59.641039+0000) 2024-03-06T13:07:00.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:00.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:00.641274+0000) 2024-03-06T13:07:00.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:00.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:01.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:01.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:01.641541+0000) 2024-03-06T13:07:02.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:02.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:02.641787+0000) 2024-03-06T13:07:03.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:03.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:03.642023+0000) 2024-03-06T13:07:04.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:04.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:04.642259+0000) 2024-03-06T13:07:05.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:05.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:05.642460+0000) 2024-03-06T13:07:06.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:06.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:06.642744+0000) 2024-03-06T13:07:07.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:07.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:07.643009+0000) 2024-03-06T13:07:08.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:08.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:08.643261+0000) 2024-03-06T13:07:09.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:09.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:09.643523+0000) 2024-03-06T13:07:10.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:10.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:10.643759+0000) 2024-03-06T13:07:11.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:11.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:11.644023+0000) 2024-03-06T13:07:12.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:12.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:12.644358+0000) 2024-03-06T13:07:13.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:13.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:13.644596+0000) 2024-03-06T13:07:14.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:14.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:14.644855+0000) 2024-03-06T13:07:15.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:15.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:15.645117+0000) 2024-03-06T13:07:16.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:16.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:16.645367+0000) 2024-03-06T13:07:17.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:17.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:17.645612+0000) 2024-03-06T13:07:18.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:18.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:18.645892+0000) 2024-03-06T13:07:19.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:19.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:19.646138+0000) 2024-03-06T13:07:20.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:20.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:20.646378+0000) 2024-03-06T13:07:21.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:21.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:21.646661+0000) 2024-03-06T13:07:22.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:22.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:22.646918+0000) 2024-03-06T13:07:23.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:23.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:23.647151+0000) 2024-03-06T13:07:24.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:24.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:24.647399+0000) 2024-03-06T13:07:25.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:25.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:25.647692+0000) 2024-03-06T13:07:26.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:26.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:26.647949+0000) 2024-03-06T13:07:27.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:27.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:27.648182+0000) 2024-03-06T13:07:28.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:28.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:28.648416+0000) 2024-03-06T13:07:29.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:29.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:29.648739+0000) 2024-03-06T13:07:30.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:30.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:30.648977+0000) 2024-03-06T13:07:30.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:30.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:31.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:31.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:31.649219+0000) 2024-03-06T13:07:32.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:32.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:32.649453+0000) 2024-03-06T13:07:33.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:33.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:33.649734+0000) 2024-03-06T13:07:34.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:34.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:34.649968+0000) 2024-03-06T13:07:35.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:35.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:35.650207+0000) 2024-03-06T13:07:36.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:36.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:36.650480+0000) 2024-03-06T13:07:37.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:37.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:37.650762+0000) 2024-03-06T13:07:38.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:38.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:38.650993+0000) 2024-03-06T13:07:39.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:39.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:39.651259+0000) 2024-03-06T13:07:40.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:40.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:40.651518+0000) 2024-03-06T13:07:41.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:41.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:41.651761+0000) 2024-03-06T13:07:42.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:42.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:42.652017+0000) 2024-03-06T13:07:43.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:43.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:43.652197+0000) 2024-03-06T13:07:44.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:44.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:44.652456+0000) 2024-03-06T13:07:45.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:45.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:45.652752+0000) 2024-03-06T13:07:46.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:46.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:46.653002+0000) 2024-03-06T13:07:47.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:47.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:47.653267+0000) 2024-03-06T13:07:48.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:48.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:48.653519+0000) 2024-03-06T13:07:49.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:49.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:49.653769+0000) 2024-03-06T13:07:50.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:50.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:50.654005+0000) 2024-03-06T13:07:51.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:51.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:51.654253+0000) 2024-03-06T13:07:52.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:52.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:52.654505+0000) 2024-03-06T13:07:53.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:53.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:53.654779+0000) 2024-03-06T13:07:54.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:54.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:54.655109+0000) 2024-03-06T13:07:55.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:55.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:55.655372+0000) 2024-03-06T13:07:56.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:56.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:56.655620+0000) 2024-03-06T13:07:57.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:57.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:57.655888+0000) 2024-03-06T13:07:58.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:58.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:58.656195+0000) 2024-03-06T13:07:59.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:07:59.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:59.656515+0000) 2024-03-06T13:08:00.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:00.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:00.656790+0000) 2024-03-06T13:08:00.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:00.981 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:01.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:01.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:01.657027+0000) 2024-03-06T13:08:02.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:02.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:02.657259+0000) 2024-03-06T13:08:03.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:03.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:03.657497+0000) 2024-03-06T13:08:04.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:04.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:04.657737+0000) 2024-03-06T13:08:05.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:05.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:05.657935+0000) 2024-03-06T13:08:06.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:06.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:06.658188+0000) 2024-03-06T13:08:07.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:07.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:07.658446+0000) 2024-03-06T13:08:08.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:08.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:08.658745+0000) 2024-03-06T13:08:09.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:09.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:09.658940+0000) 2024-03-06T13:08:10.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:10.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:10.659139+0000) 2024-03-06T13:08:11.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:11.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:11.659324+0000) 2024-03-06T13:08:12.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:12.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:12.659562+0000) 2024-03-06T13:08:13.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:13.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:13.659828+0000) 2024-03-06T13:08:14.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:14.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:14.660021+0000) 2024-03-06T13:08:15.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:15.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:15.660256+0000) 2024-03-06T13:08:16.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:16.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:16.660509+0000) 2024-03-06T13:08:17.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:17.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:17.660749+0000) 2024-03-06T13:08:18.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:18.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:18.661005+0000) 2024-03-06T13:08:19.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:19.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:19.661209+0000) 2024-03-06T13:08:20.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:20.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:20.661450+0000) 2024-03-06T13:08:21.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:21.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:21.661747+0000) 2024-03-06T13:08:22.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:22.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:22.661986+0000) 2024-03-06T13:08:23.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:23.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:23.662395+0000) 2024-03-06T13:08:24.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:24.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:24.662715+0000) 2024-03-06T13:08:25.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:25.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:25.662959+0000) 2024-03-06T13:08:26.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:26.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:26.663198+0000) 2024-03-06T13:08:27.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:27.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:27.663434+0000) 2024-03-06T13:08:28.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:28.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:28.663721+0000) 2024-03-06T13:08:29.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:29.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:29.663989+0000) 2024-03-06T13:08:30.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:30.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:30.664231+0000) 2024-03-06T13:08:31.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:31.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:31.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:31.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:31.664473+0000) 2024-03-06T13:08:32.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:32.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:32.664753+0000) 2024-03-06T13:08:33.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:33.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:33.665016+0000) 2024-03-06T13:08:34.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:34.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:34.665261+0000) 2024-03-06T13:08:35.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:35.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:35.665509+0000) 2024-03-06T13:08:36.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:36.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:36.665748+0000) 2024-03-06T13:08:37.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:37.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:37.666007+0000) 2024-03-06T13:08:38.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:38.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:38.666265+0000) 2024-03-06T13:08:39.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:39.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:39.666534+0000) 2024-03-06T13:08:40.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:40.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:40.666793+0000) 2024-03-06T13:08:41.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:41.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:41.667037+0000) 2024-03-06T13:08:42.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:42.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:42.667344+0000) 2024-03-06T13:08:43.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:43.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:43.667530+0000) 2024-03-06T13:08:44.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:44.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:44.667765+0000) 2024-03-06T13:08:45.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:45.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:45.668033+0000) 2024-03-06T13:08:46.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:46.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:46.668288+0000) 2024-03-06T13:08:47.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:47.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:47.668539+0000) 2024-03-06T13:08:48.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:48.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:48.668798+0000) 2024-03-06T13:08:49.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:49.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:49.669064+0000) 2024-03-06T13:08:50.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:50.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:50.669308+0000) 2024-03-06T13:08:51.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:51.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:51.669569+0000) 2024-03-06T13:08:52.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:52.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:52.669825+0000) 2024-03-06T13:08:53.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:53.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:53.670089+0000) 2024-03-06T13:08:54.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:54.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:54.670346+0000) 2024-03-06T13:08:55.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:55.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:55.670615+0000) 2024-03-06T13:08:56.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:56.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:56.670918+0000) 2024-03-06T13:08:57.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:57.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:57.671117+0000) 2024-03-06T13:08:58.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:58.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:58.671387+0000) 2024-03-06T13:08:59.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:08:59.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:59.671711+0000) 2024-03-06T13:09:00.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:00.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:00.671959+0000) 2024-03-06T13:09:01.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:01.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:01.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:01.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:01.672151+0000) 2024-03-06T13:09:02.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:02.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:02.672393+0000) 2024-03-06T13:09:03.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:03.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:03.672674+0000) 2024-03-06T13:09:04.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:04.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:04.672926+0000) 2024-03-06T13:09:05.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:05.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:05.673194+0000) 2024-03-06T13:09:06.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:06.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:06.673438+0000) 2024-03-06T13:09:07.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:07.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:07.673747+0000) 2024-03-06T13:09:08.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:08.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:08.673986+0000) 2024-03-06T13:09:09.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:09.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:09.674261+0000) 2024-03-06T13:09:10.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:10.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:10.674524+0000) 2024-03-06T13:09:11.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:11.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:11.674767+0000) 2024-03-06T13:09:12.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:12.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:12.674977+0000) 2024-03-06T13:09:13.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:13.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:13.675238+0000) 2024-03-06T13:09:14.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:14.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:14.675476+0000) 2024-03-06T13:09:15.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:15.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:15.675761+0000) 2024-03-06T13:09:16.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:16.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:16.675957+0000) 2024-03-06T13:09:17.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:17.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:17.676197+0000) 2024-03-06T13:09:18.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:18.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:18.676452+0000) 2024-03-06T13:09:19.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:19.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:19.676754+0000) 2024-03-06T13:09:20.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:20.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:20.677005+0000) 2024-03-06T13:09:21.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:21.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:21.677199+0000) 2024-03-06T13:09:22.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:22.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:22.677427+0000) 2024-03-06T13:09:23.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:23.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:23.677711+0000) 2024-03-06T13:09:24.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:24.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:24.677959+0000) 2024-03-06T13:09:25.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:25.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:25.678222+0000) 2024-03-06T13:09:26.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:26.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:26.678468+0000) 2024-03-06T13:09:27.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:27.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:27.678743+0000) 2024-03-06T13:09:28.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:28.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:28.678979+0000) 2024-03-06T13:09:29.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:29.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:29.679229+0000) 2024-03-06T13:09:30.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:30.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:30.679525+0000) 2024-03-06T13:09:31.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:31.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:31.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:31.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:31.679774+0000) 2024-03-06T13:09:32.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:32.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:32.680013+0000) 2024-03-06T13:09:33.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:33.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:33.680260+0000) 2024-03-06T13:09:34.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:34.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:34.680503+0000) 2024-03-06T13:09:35.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:35.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:35.680749+0000) 2024-03-06T13:09:36.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:36.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:36.680986+0000) 2024-03-06T13:09:37.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:37.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:37.681250+0000) 2024-03-06T13:09:38.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:38.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:38.681513+0000) 2024-03-06T13:09:39.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:39.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:39.681765+0000) 2024-03-06T13:09:40.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:40.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:40.681958+0000) 2024-03-06T13:09:41.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:41.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:41.682155+0000) 2024-03-06T13:09:42.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:42.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:42.682403+0000) 2024-03-06T13:09:43.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:43.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:43.682668+0000) 2024-03-06T13:09:44.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:44.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:44.682932+0000) 2024-03-06T13:09:45.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:45.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:45.683176+0000) 2024-03-06T13:09:46.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:46.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:46.683434+0000) 2024-03-06T13:09:47.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:47.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:47.683730+0000) 2024-03-06T13:09:48.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:48.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:48.683965+0000) 2024-03-06T13:09:49.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:49.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:49.684231+0000) 2024-03-06T13:09:50.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:50.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:50.684489+0000) 2024-03-06T13:09:51.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:51.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:51.684751+0000) 2024-03-06T13:09:52.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:52.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:52.685007+0000) 2024-03-06T13:09:53.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:53.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:53.685266+0000) 2024-03-06T13:09:54.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:54.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:54.685515+0000) 2024-03-06T13:09:55.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:55.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:55.685782+0000) 2024-03-06T13:09:56.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:56.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:56.686040+0000) 2024-03-06T13:09:57.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:57.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:57.686299+0000) 2024-03-06T13:09:58.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:58.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:58.686534+0000) 2024-03-06T13:09:59.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:09:59.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:59.686781+0000) 2024-03-06T13:10:00.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:00.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:00.687012+0000) 2024-03-06T13:10:01.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:01.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:01.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:01.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:01.687263+0000) 2024-03-06T13:10:02.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:02.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:02.687517+0000) 2024-03-06T13:10:03.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:03.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:03.687762+0000) 2024-03-06T13:10:04.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:04.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:04.688014+0000) 2024-03-06T13:10:05.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:05.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:05.688265+0000) 2024-03-06T13:10:06.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:06.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:06.688515+0000) 2024-03-06T13:10:07.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:07.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:07.688772+0000) 2024-03-06T13:10:08.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:08.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:08.689030+0000) 2024-03-06T13:10:09.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:09.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:09.689287+0000) 2024-03-06T13:10:10.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:10.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:10.689466+0000) 2024-03-06T13:10:11.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:11.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:11.689743+0000) 2024-03-06T13:10:12.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:12.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:12.690019+0000) 2024-03-06T13:10:13.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:13.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:13.690275+0000) 2024-03-06T13:10:14.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:14.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:14.690523+0000) 2024-03-06T13:10:15.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:15.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:15.690775+0000) 2024-03-06T13:10:16.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:16.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:16.690967+0000) 2024-03-06T13:10:17.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:17.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:17.691221+0000) 2024-03-06T13:10:18.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:18.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:18.691467+0000) 2024-03-06T13:10:19.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:19.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:19.691746+0000) 2024-03-06T13:10:20.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:20.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:20.691996+0000) 2024-03-06T13:10:21.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:21.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:21.692259+0000) 2024-03-06T13:10:22.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:22.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:22.692510+0000) 2024-03-06T13:10:23.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:23.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:23.692768+0000) 2024-03-06T13:10:24.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:24.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:24.693029+0000) 2024-03-06T13:10:25.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:25.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:25.693276+0000) 2024-03-06T13:10:26.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:26.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:26.693527+0000) 2024-03-06T13:10:27.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:27.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:27.693780+0000) 2024-03-06T13:10:28.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:28.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:28.694016+0000) 2024-03-06T13:10:29.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:29.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:29.694281+0000) 2024-03-06T13:10:30.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:30.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:30.694517+0000) 2024-03-06T13:10:31.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:31.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:31.694744+0000) 2024-03-06T13:10:31.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:31.742 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:32.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:32.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:32.694986+0000) 2024-03-06T13:10:33.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:33.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:33.695222+0000) 2024-03-06T13:10:34.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:34.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:34.695482+0000) 2024-03-06T13:10:35.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:35.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:35.695747+0000) 2024-03-06T13:10:36.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:36.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:36.695994+0000) 2024-03-06T13:10:37.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:37.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:37.696249+0000) 2024-03-06T13:10:38.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:38.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:38.696502+0000) 2024-03-06T13:10:39.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:39.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:39.696768+0000) 2024-03-06T13:10:40.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:40.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:40.697024+0000) 2024-03-06T13:10:41.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:41.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:41.697289+0000) 2024-03-06T13:10:42.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:42.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:42.697525+0000) 2024-03-06T13:10:43.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:43.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:43.697781+0000) 2024-03-06T13:10:44.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:44.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:44.698014+0000) 2024-03-06T13:10:45.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:45.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:45.698274+0000) 2024-03-06T13:10:46.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:46.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:46.698528+0000) 2024-03-06T13:10:47.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:47.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:47.698775+0000) 2024-03-06T13:10:48.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:48.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:48.699018+0000) 2024-03-06T13:10:49.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:49.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:49.699269+0000) 2024-03-06T13:10:50.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:50.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:50.699531+0000) 2024-03-06T13:10:51.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:51.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:51.699794+0000) 2024-03-06T13:10:52.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:52.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:52.700049+0000) 2024-03-06T13:10:53.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:53.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:53.700318+0000) 2024-03-06T13:10:54.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:54.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:54.700569+0000) 2024-03-06T13:10:55.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:55.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:55.700830+0000) 2024-03-06T13:10:56.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:56.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:56.701087+0000) 2024-03-06T13:10:57.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:57.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:57.701344+0000) 2024-03-06T13:10:58.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:58.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:58.701620+0000) 2024-03-06T13:10:59.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:10:59.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:59.701946+0000) 2024-03-06T13:11:00.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:00.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:00.702240+0000) 2024-03-06T13:11:01.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:01.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:01.702559+0000) 2024-03-06T13:11:01.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:01.890 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:02.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:02.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:02.702861+0000) 2024-03-06T13:11:03.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:03.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:03.703167+0000) 2024-03-06T13:11:04.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:04.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:04.703464+0000) 2024-03-06T13:11:05.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:05.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:05.703762+0000) 2024-03-06T13:11:06.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:06.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:06.704016+0000) 2024-03-06T13:11:07.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:07.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:07.704253+0000) 2024-03-06T13:11:08.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:08.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:08.704497+0000) 2024-03-06T13:11:09.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:09.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:09.704767+0000) 2024-03-06T13:11:10.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:10.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:10.705054+0000) 2024-03-06T13:11:11.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:11.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:11.705358+0000) 2024-03-06T13:11:12.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:12.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:12.705699+0000) 2024-03-06T13:11:13.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:13.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:13.706006+0000) 2024-03-06T13:11:14.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:14.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:14.706312+0000) 2024-03-06T13:11:15.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:15.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:15.706581+0000) 2024-03-06T13:11:16.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:16.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:16.706867+0000) 2024-03-06T13:11:17.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:17.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:17.707207+0000) 2024-03-06T13:11:18.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:18.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:18.707514+0000) 2024-03-06T13:11:19.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:19.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:19.707843+0000) 2024-03-06T13:11:20.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:20.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:20.708153+0000) 2024-03-06T13:11:21.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:21.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:21.708456+0000) 2024-03-06T13:11:22.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:22.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:22.708788+0000) 2024-03-06T13:11:23.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:23.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:23.709071+0000) 2024-03-06T13:11:24.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:24.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:24.709307+0000) 2024-03-06T13:11:25.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:25.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:25.709543+0000) 2024-03-06T13:11:26.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:26.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:26.709784+0000) 2024-03-06T13:11:27.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:27.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:27.709978+0000) 2024-03-06T13:11:28.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:28.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:28.710225+0000) 2024-03-06T13:11:29.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:29.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:29.710477+0000) 2024-03-06T13:11:30.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:30.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:30.710734+0000) 2024-03-06T13:11:31.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:31.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:31.710999+0000) 2024-03-06T13:11:32.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:32.044 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:32.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:32.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:32.711251+0000) 2024-03-06T13:11:33.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:33.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:33.711518+0000) 2024-03-06T13:11:34.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:34.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:34.711765+0000) 2024-03-06T13:11:35.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:35.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:35.712022+0000) 2024-03-06T13:11:36.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:36.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:36.712261+0000) 2024-03-06T13:11:37.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:37.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:37.712524+0000) 2024-03-06T13:11:38.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:38.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:38.712797+0000) 2024-03-06T13:11:39.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:39.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:39.713064+0000) 2024-03-06T13:11:40.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:40.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:40.713316+0000) 2024-03-06T13:11:41.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:41.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:41.713567+0000) 2024-03-06T13:11:42.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:42.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:42.713807+0000) 2024-03-06T13:11:43.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:43.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:43.714071+0000) 2024-03-06T13:11:44.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:44.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:44.714304+0000) 2024-03-06T13:11:45.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:45.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:45.714543+0000) 2024-03-06T13:11:46.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:46.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:46.714799+0000) 2024-03-06T13:11:47.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:47.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:47.715036+0000) 2024-03-06T13:11:48.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:48.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:48.715300+0000) 2024-03-06T13:11:49.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:49.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:49.715553+0000) 2024-03-06T13:11:50.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:50.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:50.715785+0000) 2024-03-06T13:11:51.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:51.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:51.716008+0000) 2024-03-06T13:11:52.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:52.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:52.716206+0000) 2024-03-06T13:11:53.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:53.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:53.716395+0000) 2024-03-06T13:11:54.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:54.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:54.716726+0000) 2024-03-06T13:11:55.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:55.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:55.716985+0000) 2024-03-06T13:11:56.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:56.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:56.717232+0000) 2024-03-06T13:11:57.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:57.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:57.717478+0000) 2024-03-06T13:11:58.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:58.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:58.717743+0000) 2024-03-06T13:11:59.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:11:59.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:59.718012+0000) 2024-03-06T13:12:00.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:00.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:00.718258+0000) 2024-03-06T13:12:01.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:01.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:01.718507+0000) 2024-03-06T13:12:02.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:02.228 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:02.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:02.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:02.718752+0000) 2024-03-06T13:12:03.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:03.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:03.718998+0000) 2024-03-06T13:12:04.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:04.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:04.719249+0000) 2024-03-06T13:12:05.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:05.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:05.719474+0000) 2024-03-06T13:12:06.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:06.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:06.719774+0000) 2024-03-06T13:12:07.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:07.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:07.720024+0000) 2024-03-06T13:12:08.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:08.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:08.720279+0000) 2024-03-06T13:12:09.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:09.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:09.720553+0000) 2024-03-06T13:12:10.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:10.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:10.720860+0000) 2024-03-06T13:12:11.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:11.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:11.721100+0000) 2024-03-06T13:12:12.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:12.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:12.721354+0000) 2024-03-06T13:12:13.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:13.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:13.721603+0000) 2024-03-06T13:12:14.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:14.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:14.721894+0000) 2024-03-06T13:12:15.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:15.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:15.722149+0000) 2024-03-06T13:12:16.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:16.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:16.722411+0000) 2024-03-06T13:12:17.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:17.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:17.722714+0000) 2024-03-06T13:12:18.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:18.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:18.722948+0000) 2024-03-06T13:12:19.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:19.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:19.723196+0000) 2024-03-06T13:12:20.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:20.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:20.723433+0000) 2024-03-06T13:12:21.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:21.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:21.723744+0000) 2024-03-06T13:12:22.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:22.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:22.723960+0000) 2024-03-06T13:12:23.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:23.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:23.724216+0000) 2024-03-06T13:12:24.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:24.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:24.724476+0000) 2024-03-06T13:12:25.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:25.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:25.724743+0000) 2024-03-06T13:12:26.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:26.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:26.725009+0000) 2024-03-06T13:12:27.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:27.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:27.725281+0000) 2024-03-06T13:12:28.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:28.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:28.725537+0000) 2024-03-06T13:12:29.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:29.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:29.725825+0000) 2024-03-06T13:12:30.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:30.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:30.726092+0000) 2024-03-06T13:12:31.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:31.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:31.726368+0000) 2024-03-06T13:12:32.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:32.372 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:32.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:32.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:32.726686+0000) 2024-03-06T13:12:33.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:33.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:33.726964+0000) 2024-03-06T13:12:34.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:34.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:34.727239+0000) 2024-03-06T13:12:35.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:35.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:35.727515+0000) 2024-03-06T13:12:36.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:36.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:36.727774+0000) 2024-03-06T13:12:37.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:37.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:37.728056+0000) 2024-03-06T13:12:38.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:38.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:38.728332+0000) 2024-03-06T13:12:39.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:39.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:39.728620+0000) 2024-03-06T13:12:40.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:40.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:40.728929+0000) 2024-03-06T13:12:41.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:41.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:41.729188+0000) 2024-03-06T13:12:42.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:42.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:42.729449+0000) 2024-03-06T13:12:43.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:43.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:43.729758+0000) 2024-03-06T13:12:44.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:44.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:44.730020+0000) 2024-03-06T13:12:45.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:45.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:45.730304+0000) 2024-03-06T13:12:46.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:46.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:46.730579+0000) 2024-03-06T13:12:47.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:47.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:47.730860+0000) 2024-03-06T13:12:48.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:48.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:48.731117+0000) 2024-03-06T13:12:49.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:49.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:49.731312+0000) 2024-03-06T13:12:50.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:50.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:50.731551+0000) 2024-03-06T13:12:51.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:51.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:51.731803+0000) 2024-03-06T13:12:52.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:52.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:52.732057+0000) 2024-03-06T13:12:53.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:53.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:53.732371+0000) 2024-03-06T13:12:54.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:54.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:54.732670+0000) 2024-03-06T13:12:55.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:55.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:55.732872+0000) 2024-03-06T13:12:56.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:56.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:56.733134+0000) 2024-03-06T13:12:57.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:57.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:57.733382+0000) 2024-03-06T13:12:58.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:58.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:58.733670+0000) 2024-03-06T13:12:59.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:12:59.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:59.733948+0000) 2024-03-06T13:13:00.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:00.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:00.734212+0000) 2024-03-06T13:13:01.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:01.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:01.734479+0000) 2024-03-06T13:13:02.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:02.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:02.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:02.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:02.734744+0000) 2024-03-06T13:13:03.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:03.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:03.734991+0000) 2024-03-06T13:13:04.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:04.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:04.735260+0000) 2024-03-06T13:13:05.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:05.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:05.735523+0000) 2024-03-06T13:13:06.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:06.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:06.735784+0000) 2024-03-06T13:13:07.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:07.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:07.736043+0000) 2024-03-06T13:13:08.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:08.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:08.736283+0000) 2024-03-06T13:13:09.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:09.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:09.736565+0000) 2024-03-06T13:13:10.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:10.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:10.736825+0000) 2024-03-06T13:13:11.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:11.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:11.737077+0000) 2024-03-06T13:13:12.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:12.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:12.737318+0000) 2024-03-06T13:13:13.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:13.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:13.737551+0000) 2024-03-06T13:13:14.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:14.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:14.737791+0000) 2024-03-06T13:13:15.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:15.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:15.737968+0000) 2024-03-06T13:13:16.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:16.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:16.738156+0000) 2024-03-06T13:13:17.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:17.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:17.738399+0000) 2024-03-06T13:13:18.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:18.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:18.738665+0000) 2024-03-06T13:13:19.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:19.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:19.738925+0000) 2024-03-06T13:13:20.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:20.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:20.739161+0000) 2024-03-06T13:13:21.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:21.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:21.739425+0000) 2024-03-06T13:13:22.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:22.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:22.739726+0000) 2024-03-06T13:13:23.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:23.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:23.739977+0000) 2024-03-06T13:13:24.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:24.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:24.740208+0000) 2024-03-06T13:13:25.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:25.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:25.740465+0000) 2024-03-06T13:13:26.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:26.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:26.740769+0000) 2024-03-06T13:13:27.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:27.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:27.741025+0000) 2024-03-06T13:13:28.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:28.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:28.741277+0000) 2024-03-06T13:13:29.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:29.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:29.741545+0000) 2024-03-06T13:13:30.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:30.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:30.741799+0000) 2024-03-06T13:13:31.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:31.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:31.741999+0000) 2024-03-06T13:13:32.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:32.658 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:32.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:32.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:32.742202+0000) 2024-03-06T13:13:33.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:33.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:33.742448+0000) 2024-03-06T13:13:34.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:34.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:34.742806+0000) 2024-03-06T13:13:35.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:35.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:35.743022+0000) 2024-03-06T13:13:36.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:36.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:36.743287+0000) 2024-03-06T13:13:37.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:37.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:37.743550+0000) 2024-03-06T13:13:38.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:38.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:38.743802+0000) 2024-03-06T13:13:39.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:39.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:39.744073+0000) 2024-03-06T13:13:40.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:40.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:40.744320+0000) 2024-03-06T13:13:41.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:41.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:41.744623+0000) 2024-03-06T13:13:42.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:42.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:42.744918+0000) 2024-03-06T13:13:43.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:43.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:43.745177+0000) 2024-03-06T13:13:44.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:44.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:44.745445+0000) 2024-03-06T13:13:45.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:45.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:45.745730+0000) 2024-03-06T13:13:46.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:46.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:46.745994+0000) 2024-03-06T13:13:47.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:47.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:47.746196+0000) 2024-03-06T13:13:48.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:48.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:48.746429+0000) 2024-03-06T13:13:49.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:49.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:49.746713+0000) 2024-03-06T13:13:50.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:50.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:50.746904+0000) 2024-03-06T13:13:51.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:51.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:51.747105+0000) 2024-03-06T13:13:52.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:52.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:52.747362+0000) 2024-03-06T13:13:53.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:53.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:53.747617+0000) 2024-03-06T13:13:54.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:54.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:54.747897+0000) 2024-03-06T13:13:55.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:55.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:55.748133+0000) 2024-03-06T13:13:56.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:56.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:56.748395+0000) 2024-03-06T13:13:57.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:57.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:57.748701+0000) 2024-03-06T13:13:58.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:58.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:58.748889+0000) 2024-03-06T13:13:59.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:13:59.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:59.749164+0000) 2024-03-06T13:14:00.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:00.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:00.749417+0000) 2024-03-06T13:14:01.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:01.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:01.749748+0000) 2024-03-06T13:14:02.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:02.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:02.750021+0000) 2024-03-06T13:14:02.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:02.797 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:03.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:03.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:03.750255+0000) 2024-03-06T13:14:04.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:04.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:04.750490+0000) 2024-03-06T13:14:05.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:05.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:05.750750+0000) 2024-03-06T13:14:06.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:06.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:06.750949+0000) 2024-03-06T13:14:07.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:07.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:07.751211+0000) 2024-03-06T13:14:08.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:08.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:08.751457+0000) 2024-03-06T13:14:09.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:09.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:09.751748+0000) 2024-03-06T13:14:10.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:10.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:10.751942+0000) 2024-03-06T13:14:11.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:11.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:11.752152+0000) 2024-03-06T13:14:12.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:12.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:12.752420+0000) 2024-03-06T13:14:13.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:13.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:13.752691+0000) 2024-03-06T13:14:14.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:14.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:14.752948+0000) 2024-03-06T13:14:15.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:15.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:15.753182+0000) 2024-03-06T13:14:16.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:16.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:16.753436+0000) 2024-03-06T13:14:17.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:17.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:17.753741+0000) 2024-03-06T13:14:18.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:18.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:18.753981+0000) 2024-03-06T13:14:19.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:19.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:19.754233+0000) 2024-03-06T13:14:20.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:20.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:20.754483+0000) 2024-03-06T13:14:21.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:21.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:21.754740+0000) 2024-03-06T13:14:22.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:22.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:22.754973+0000) 2024-03-06T13:14:23.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:23.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:23.755208+0000) 2024-03-06T13:14:24.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:24.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:24.755463+0000) 2024-03-06T13:14:25.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:25.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:25.755741+0000) 2024-03-06T13:14:26.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:26.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:26.755967+0000) 2024-03-06T13:14:27.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:27.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:27.756224+0000) 2024-03-06T13:14:28.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:28.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:28.756475+0000) 2024-03-06T13:14:29.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:29.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:29.756745+0000) 2024-03-06T13:14:30.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:30.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:30.756979+0000) 2024-03-06T13:14:31.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:31.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:31.757223+0000) 2024-03-06T13:14:32.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:32.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:32.757476+0000) 2024-03-06T13:14:32.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:32.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:33.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:33.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:33.757674+0000) 2024-03-06T13:14:34.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:34.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:34.757922+0000) 2024-03-06T13:14:35.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:35.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:35.758119+0000) 2024-03-06T13:14:36.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:36.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:36.758375+0000) 2024-03-06T13:14:37.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:37.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:37.758622+0000) 2024-03-06T13:14:38.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:38.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:38.758918+0000) 2024-03-06T13:14:39.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:39.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:39.759226+0000) 2024-03-06T13:14:40.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:40.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:40.759540+0000) 2024-03-06T13:14:41.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:41.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:41.759821+0000) 2024-03-06T13:14:42.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:42.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:42.760094+0000) 2024-03-06T13:14:43.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:43.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:43.760375+0000) 2024-03-06T13:14:44.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:44.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:44.760671+0000) 2024-03-06T13:14:45.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:45.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:45.760875+0000) 2024-03-06T13:14:46.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:46.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:46.761146+0000) 2024-03-06T13:14:47.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:47.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:47.761410+0000) 2024-03-06T13:14:48.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:48.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:48.761697+0000) 2024-03-06T13:14:49.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:49.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:49.761958+0000) 2024-03-06T13:14:50.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:50.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:50.762191+0000) 2024-03-06T13:14:51.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:51.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:51.762442+0000) 2024-03-06T13:14:52.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:52.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:52.762736+0000) 2024-03-06T13:14:53.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:53.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:53.763011+0000) 2024-03-06T13:14:54.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:54.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:54.763291+0000) 2024-03-06T13:14:55.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:55.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:55.763544+0000) 2024-03-06T13:14:56.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:56.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:56.763814+0000) 2024-03-06T13:14:57.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:57.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:57.764075+0000) 2024-03-06T13:14:58.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:58.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:58.764340+0000) 2024-03-06T13:14:59.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:14:59.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:59.764559+0000) 2024-03-06T13:15:00.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:00.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:00.764830+0000) 2024-03-06T13:15:01.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:01.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:01.765105+0000) 2024-03-06T13:15:02.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:02.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:02.765355+0000) 2024-03-06T13:15:03.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:03.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:03.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:03.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:03.765609+0000) 2024-03-06T13:15:04.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:04.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:04.765898+0000) 2024-03-06T13:15:05.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:05.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:05.766108+0000) 2024-03-06T13:15:06.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:06.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:06.766376+0000) 2024-03-06T13:15:07.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:07.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:07.766672+0000) 2024-03-06T13:15:08.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:08.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:08.766942+0000) 2024-03-06T13:15:09.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:09.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:09.767170+0000) 2024-03-06T13:15:10.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:10.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:10.767442+0000) 2024-03-06T13:15:11.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:11.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:11.767748+0000) 2024-03-06T13:15:12.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:12.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:12.767962+0000) 2024-03-06T13:15:13.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:13.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:13.768211+0000) 2024-03-06T13:15:14.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:14.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:14.768493+0000) 2024-03-06T13:15:15.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:15.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:15.768792+0000) 2024-03-06T13:15:16.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:16.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:16.769044+0000) 2024-03-06T13:15:17.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:17.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:17.769292+0000) 2024-03-06T13:15:18.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:18.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:18.769535+0000) 2024-03-06T13:15:19.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:19.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:19.769803+0000) 2024-03-06T13:15:20.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:20.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:20.770031+0000) 2024-03-06T13:15:21.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:21.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:21.770215+0000) 2024-03-06T13:15:22.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:22.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:22.770456+0000) 2024-03-06T13:15:23.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:23.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:23.770750+0000) 2024-03-06T13:15:24.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:24.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:24.771006+0000) 2024-03-06T13:15:25.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:25.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:25.771260+0000) 2024-03-06T13:15:26.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:26.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:26.771527+0000) 2024-03-06T13:15:27.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:27.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:27.771789+0000) 2024-03-06T13:15:28.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:28.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:28.772090+0000) 2024-03-06T13:15:29.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:29.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:29.772358+0000) 2024-03-06T13:15:30.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:30.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:30.772611+0000) 2024-03-06T13:15:31.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:31.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:31.772909+0000) 2024-03-06T13:15:32.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:32.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:32.773158+0000) 2024-03-06T13:15:33.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:33.254 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:33.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:33.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:33.773409+0000) 2024-03-06T13:15:34.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:34.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:34.773695+0000) 2024-03-06T13:15:35.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:35.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:35.773907+0000) 2024-03-06T13:15:36.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:36.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:36.774181+0000) 2024-03-06T13:15:37.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:37.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:37.774457+0000) 2024-03-06T13:15:38.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:38.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:38.774753+0000) 2024-03-06T13:15:39.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:39.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:39.775044+0000) 2024-03-06T13:15:40.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:40.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:40.775298+0000) 2024-03-06T13:15:41.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:41.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:41.775556+0000) 2024-03-06T13:15:42.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:42.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:42.775807+0000) 2024-03-06T13:15:43.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:43.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:43.776079+0000) 2024-03-06T13:15:44.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:44.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:44.776344+0000) 2024-03-06T13:15:45.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:45.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:45.776624+0000) 2024-03-06T13:15:46.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:46.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:46.776964+0000) 2024-03-06T13:15:47.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:47.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:47.777234+0000) 2024-03-06T13:15:48.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:48.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:48.777504+0000) 2024-03-06T13:15:49.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:49.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:49.777795+0000) 2024-03-06T13:15:50.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:50.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:50.778052+0000) 2024-03-06T13:15:51.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:51.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:51.778305+0000) 2024-03-06T13:15:52.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:52.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:52.778579+0000) 2024-03-06T13:15:53.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:53.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:53.778856+0000) 2024-03-06T13:15:54.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:54.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:54.779075+0000) 2024-03-06T13:15:55.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:55.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:55.779350+0000) 2024-03-06T13:15:56.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:56.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:56.779623+0000) 2024-03-06T13:15:57.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:57.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:57.779872+0000) 2024-03-06T13:15:58.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:58.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:58.780144+0000) 2024-03-06T13:15:59.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:15:59.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:59.780436+0000) 2024-03-06T13:16:00.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:00.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:00.780737+0000) 2024-03-06T13:16:01.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:01.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:01.780976+0000) 2024-03-06T13:16:02.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:02.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:02.781239+0000) 2024-03-06T13:16:03.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:03.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:03.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:03.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:03.781500+0000) 2024-03-06T13:16:04.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:04.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:04.781747+0000) 2024-03-06T13:16:05.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:05.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:05.782018+0000) 2024-03-06T13:16:06.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:06.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:06.782291+0000) 2024-03-06T13:16:07.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:07.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:07.782552+0000) 2024-03-06T13:16:08.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:08.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:08.782756+0000) 2024-03-06T13:16:09.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:09.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:09.783026+0000) 2024-03-06T13:16:10.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:10.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:10.783289+0000) 2024-03-06T13:16:11.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:11.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:11.783551+0000) 2024-03-06T13:16:12.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:12.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:12.783809+0000) 2024-03-06T13:16:13.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:13.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:13.783990+0000) 2024-03-06T13:16:14.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:14.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:14.784245+0000) 2024-03-06T13:16:15.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:15.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:15.784518+0000) 2024-03-06T13:16:16.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:16.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:16.784831+0000) 2024-03-06T13:16:17.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:17.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:17.785039+0000) 2024-03-06T13:16:18.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:18.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:18.785294+0000) 2024-03-06T13:16:19.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:19.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:19.785586+0000) 2024-03-06T13:16:20.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:20.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:20.785913+0000) 2024-03-06T13:16:21.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:21.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:21.786186+0000) 2024-03-06T13:16:22.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:22.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:22.786465+0000) 2024-03-06T13:16:23.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:23.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:23.786762+0000) 2024-03-06T13:16:24.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:24.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:24.787052+0000) 2024-03-06T13:16:25.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:25.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:25.787315+0000) 2024-03-06T13:16:26.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:26.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:26.787589+0000) 2024-03-06T13:16:27.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:27.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:27.787911+0000) 2024-03-06T13:16:28.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:28.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:28.788163+0000) 2024-03-06T13:16:29.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:29.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:29.788460+0000) 2024-03-06T13:16:30.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:30.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:30.788766+0000) 2024-03-06T13:16:31.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:31.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:31.789041+0000) 2024-03-06T13:16:32.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:32.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:32.789317+0000) 2024-03-06T13:16:33.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:33.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:33.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:33.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:33.789589+0000) 2024-03-06T13:16:34.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:34.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:34.789910+0000) 2024-03-06T13:16:35.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:35.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:35.790128+0000) 2024-03-06T13:16:36.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:36.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:36.790401+0000) 2024-03-06T13:16:37.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:37.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:37.790720+0000) 2024-03-06T13:16:38.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:38.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:38.791003+0000) 2024-03-06T13:16:39.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:39.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:39.791284+0000) 2024-03-06T13:16:40.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:40.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:40.791545+0000) 2024-03-06T13:16:41.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:41.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:41.791827+0000) 2024-03-06T13:16:42.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:42.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:42.792110+0000) 2024-03-06T13:16:43.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:43.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:43.792390+0000) 2024-03-06T13:16:44.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:44.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:44.792682+0000) 2024-03-06T13:16:45.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:45.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:45.792979+0000) 2024-03-06T13:16:46.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:46.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:46.793239+0000) 2024-03-06T13:16:47.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:47.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:47.793506+0000) 2024-03-06T13:16:48.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:48.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:48.793753+0000) 2024-03-06T13:16:49.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:49.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:49.794009+0000) 2024-03-06T13:16:50.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:50.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:50.794251+0000) 2024-03-06T13:16:51.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:51.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:51.794509+0000) 2024-03-06T13:16:52.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:52.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:52.794778+0000) 2024-03-06T13:16:53.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:53.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:53.795047+0000) 2024-03-06T13:16:54.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:54.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:54.795247+0000) 2024-03-06T13:16:55.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:55.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:55.795492+0000) 2024-03-06T13:16:56.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:56.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:56.795748+0000) 2024-03-06T13:16:57.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:57.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:57.795950+0000) 2024-03-06T13:16:58.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:58.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:58.796227+0000) 2024-03-06T13:16:59.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:16:59.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:59.796476+0000) 2024-03-06T13:17:00.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:00.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:00.796732+0000) 2024-03-06T13:17:01.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:01.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:01.796971+0000) 2024-03-06T13:17:02.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:02.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:02.797208+0000) 2024-03-06T13:17:03.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:03.689 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:03.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:03.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:03.797413+0000) 2024-03-06T13:17:04.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:04.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:04.797688+0000) 2024-03-06T13:17:05.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:05.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:05.798014+0000) 2024-03-06T13:17:06.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:06.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:06.798334+0000) 2024-03-06T13:17:07.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:07.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:07.798623+0000) 2024-03-06T13:17:08.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:08.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:08.798947+0000) 2024-03-06T13:17:09.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:09.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:09.799263+0000) 2024-03-06T13:17:10.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:10.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:10.799553+0000) 2024-03-06T13:17:11.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:11.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:11.799861+0000) 2024-03-06T13:17:12.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:12.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:12.800164+0000) 2024-03-06T13:17:13.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:13.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:13.800479+0000) 2024-03-06T13:17:14.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:14.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:14.800785+0000) 2024-03-06T13:17:15.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:15.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:15.801086+0000) 2024-03-06T13:17:16.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:16.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:16.801394+0000) 2024-03-06T13:17:17.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:17.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:17.801739+0000) 2024-03-06T13:17:18.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:18.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:18.802029+0000) 2024-03-06T13:17:19.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:19.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:19.802343+0000) 2024-03-06T13:17:20.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:20.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:20.802612+0000) 2024-03-06T13:17:21.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:21.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:21.802905+0000) 2024-03-06T13:17:22.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:22.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:22.803143+0000) 2024-03-06T13:17:23.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:23.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:23.803391+0000) 2024-03-06T13:17:24.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:24.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:24.803652+0000) 2024-03-06T13:17:25.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:25.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:25.803912+0000) 2024-03-06T13:17:26.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:26.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:26.804157+0000) 2024-03-06T13:17:27.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:27.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:27.804454+0000) 2024-03-06T13:17:28.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:28.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:28.804747+0000) 2024-03-06T13:17:29.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:29.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:29.805013+0000) 2024-03-06T13:17:30.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:30.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:30.805265+0000) 2024-03-06T13:17:31.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:31.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:31.805522+0000) 2024-03-06T13:17:32.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:32.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:32.805760+0000) 2024-03-06T13:17:33.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:33.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:33.806016+0000) 2024-03-06T13:17:33.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:33.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:34.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:34.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:34.806253+0000) 2024-03-06T13:17:35.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:35.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:35.806517+0000) 2024-03-06T13:17:36.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:36.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:36.806772+0000) 2024-03-06T13:17:37.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:37.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:37.807027+0000) 2024-03-06T13:17:38.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:38.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:38.807218+0000) 2024-03-06T13:17:39.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:39.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:39.807435+0000) 2024-03-06T13:17:40.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:40.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:40.807612+0000) 2024-03-06T13:17:41.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:41.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:41.807841+0000) 2024-03-06T13:17:42.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:42.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:42.808040+0000) 2024-03-06T13:17:43.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:43.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:43.808297+0000) 2024-03-06T13:17:44.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:44.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:44.808555+0000) 2024-03-06T13:17:45.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:45.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:45.808819+0000) 2024-03-06T13:17:46.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:46.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:46.809001+0000) 2024-03-06T13:17:47.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:47.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:47.809263+0000) 2024-03-06T13:17:48.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:48.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:48.809525+0000) 2024-03-06T13:17:49.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:49.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:49.809790+0000) 2024-03-06T13:17:50.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:50.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:50.809977+0000) 2024-03-06T13:17:51.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:51.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:51.810232+0000) 2024-03-06T13:17:52.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:52.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:52.810492+0000) 2024-03-06T13:17:53.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:53.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:53.810746+0000) 2024-03-06T13:17:54.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:54.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:54.810968+0000) 2024-03-06T13:17:55.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:55.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:55.811213+0000) 2024-03-06T13:17:56.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:56.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:56.811453+0000) 2024-03-06T13:17:57.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:57.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:57.811751+0000) 2024-03-06T13:17:58.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:58.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:58.812007+0000) 2024-03-06T13:17:59.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:17:59.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:59.812264+0000) 2024-03-06T13:18:00.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:00.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:00.812513+0000) 2024-03-06T13:18:01.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:01.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:01.812749+0000) 2024-03-06T13:18:02.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:02.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:02.813008+0000) 2024-03-06T13:18:03.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:03.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:03.813243+0000) 2024-03-06T13:18:03.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:03.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:04.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:04.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:04.813480+0000) 2024-03-06T13:18:05.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:05.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:05.813755+0000) 2024-03-06T13:18:06.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:06.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:06.814005+0000) 2024-03-06T13:18:07.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:07.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:07.814270+0000) 2024-03-06T13:18:08.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:08.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:08.814530+0000) 2024-03-06T13:18:09.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:09.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:09.814776+0000) 2024-03-06T13:18:10.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:10.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:10.815028+0000) 2024-03-06T13:18:11.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:11.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:11.815296+0000) 2024-03-06T13:18:12.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:12.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:12.815558+0000) 2024-03-06T13:18:13.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:13.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:13.815821+0000) 2024-03-06T13:18:14.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:14.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:14.816067+0000) 2024-03-06T13:18:15.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:15.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:15.816306+0000) 2024-03-06T13:18:16.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:16.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:16.816556+0000) 2024-03-06T13:18:17.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:17.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:17.816824+0000) 2024-03-06T13:18:18.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:18.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:18.817083+0000) 2024-03-06T13:18:19.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:19.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:19.817346+0000) 2024-03-06T13:18:20.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:20.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:20.817597+0000) 2024-03-06T13:18:21.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:21.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:21.817847+0000) 2024-03-06T13:18:22.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:22.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:22.818093+0000) 2024-03-06T13:18:23.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:23.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:23.818331+0000) 2024-03-06T13:18:24.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:24.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:24.818568+0000) 2024-03-06T13:18:25.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:25.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:25.818824+0000) 2024-03-06T13:18:26.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:26.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:26.819048+0000) 2024-03-06T13:18:27.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:27.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:27.819308+0000) 2024-03-06T13:18:28.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:28.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:28.819544+0000) 2024-03-06T13:18:29.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:29.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:29.819790+0000) 2024-03-06T13:18:30.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:30.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:30.820049+0000) 2024-03-06T13:18:31.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:31.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:31.820292+0000) 2024-03-06T13:18:32.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:32.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:32.820523+0000) 2024-03-06T13:18:33.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:33.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:33.820761+0000) 2024-03-06T13:18:34.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:34.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:34.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:34.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:34.820951+0000) 2024-03-06T13:18:35.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:35.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:35.821158+0000) 2024-03-06T13:18:36.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:36.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:36.821410+0000) 2024-03-06T13:18:37.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:37.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:37.821702+0000) 2024-03-06T13:18:38.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:38.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:38.821956+0000) 2024-03-06T13:18:39.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:39.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:39.822226+0000) 2024-03-06T13:18:40.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:40.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:40.822464+0000) 2024-03-06T13:18:41.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:41.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:41.822776+0000) 2024-03-06T13:18:42.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:42.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:42.823043+0000) 2024-03-06T13:18:43.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:43.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:43.823275+0000) 2024-03-06T13:18:44.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:44.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:44.823531+0000) 2024-03-06T13:18:45.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:45.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:45.823827+0000) 2024-03-06T13:18:46.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:46.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:46.824072+0000) 2024-03-06T13:18:47.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:47.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:47.824304+0000) 2024-03-06T13:18:48.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:48.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:48.824539+0000) 2024-03-06T13:18:49.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:49.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:49.824802+0000) 2024-03-06T13:18:50.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:50.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:50.824995+0000) 2024-03-06T13:18:51.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:51.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:51.825231+0000) 2024-03-06T13:18:52.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:52.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:52.825484+0000) 2024-03-06T13:18:53.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:53.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:53.825738+0000) 2024-03-06T13:18:54.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:54.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:54.825993+0000) 2024-03-06T13:18:55.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:55.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:55.826235+0000) 2024-03-06T13:18:56.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:56.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:56.826482+0000) 2024-03-06T13:18:57.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:57.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:57.826743+0000) 2024-03-06T13:18:58.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:58.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:58.826994+0000) 2024-03-06T13:18:59.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:18:59.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:59.827248+0000) 2024-03-06T13:19:00.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:00.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:00.827506+0000) 2024-03-06T13:19:01.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:01.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:01.827747+0000) 2024-03-06T13:19:02.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:02.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:02.828001+0000) 2024-03-06T13:19:03.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:03.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:03.828258+0000) 2024-03-06T13:19:04.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:04.296 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:04.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:04.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:04.828495+0000) 2024-03-06T13:19:05.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:05.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:05.828743+0000) 2024-03-06T13:19:06.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:06.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:06.828991+0000) 2024-03-06T13:19:07.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:07.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:07.829235+0000) 2024-03-06T13:19:08.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:08.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:08.829486+0000) 2024-03-06T13:19:09.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:09.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:09.829746+0000) 2024-03-06T13:19:10.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:10.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:10.830007+0000) 2024-03-06T13:19:11.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:11.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:11.830265+0000) 2024-03-06T13:19:12.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:12.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:12.830506+0000) 2024-03-06T13:19:13.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:13.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:13.830746+0000) 2024-03-06T13:19:14.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:14.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:14.830998+0000) 2024-03-06T13:19:15.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:15.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:15.831261+0000) 2024-03-06T13:19:16.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:16.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:16.831514+0000) 2024-03-06T13:19:17.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:17.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:17.831769+0000) 2024-03-06T13:19:18.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:18.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:18.832022+0000) 2024-03-06T13:19:19.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:19.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:19.832287+0000) 2024-03-06T13:19:20.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:20.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:20.832526+0000) 2024-03-06T13:19:21.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:21.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:21.832786+0000) 2024-03-06T13:19:22.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:22.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:22.833031+0000) 2024-03-06T13:19:23.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:23.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:23.833287+0000) 2024-03-06T13:19:24.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:24.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:24.833541+0000) 2024-03-06T13:19:25.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:25.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:25.833811+0000) 2024-03-06T13:19:26.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:26.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:26.834071+0000) 2024-03-06T13:19:27.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:27.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:27.834338+0000) 2024-03-06T13:19:28.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:28.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:28.834601+0000) 2024-03-06T13:19:29.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:29.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:29.834925+0000) 2024-03-06T13:19:30.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:30.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:30.835132+0000) 2024-03-06T13:19:31.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:31.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:31.835390+0000) 2024-03-06T13:19:32.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:32.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:32.835583+0000) 2024-03-06T13:19:33.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:33.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:33.835878+0000) 2024-03-06T13:19:34.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:34.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:34.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:34.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:34.836094+0000) 2024-03-06T13:19:35.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:35.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:35.836368+0000) 2024-03-06T13:19:36.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:36.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:36.836670+0000) 2024-03-06T13:19:37.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:37.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:37.836950+0000) 2024-03-06T13:19:38.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:38.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:38.837284+0000) 2024-03-06T13:19:39.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:39.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:39.837561+0000) 2024-03-06T13:19:40.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:40.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:40.837844+0000) 2024-03-06T13:19:41.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:41.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:41.838120+0000) 2024-03-06T13:19:42.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:42.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:42.838376+0000) 2024-03-06T13:19:43.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:43.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:43.838696+0000) 2024-03-06T13:19:44.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:44.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:44.838986+0000) 2024-03-06T13:19:45.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:45.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:45.839237+0000) 2024-03-06T13:19:46.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:46.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:46.839503+0000) 2024-03-06T13:19:47.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:47.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:47.839779+0000) 2024-03-06T13:19:48.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:48.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:48.840009+0000) 2024-03-06T13:19:49.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:49.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:49.840287+0000) 2024-03-06T13:19:50.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:50.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:50.840545+0000) 2024-03-06T13:19:51.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:51.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:51.840799+0000) 2024-03-06T13:19:52.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:52.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:52.841038+0000) 2024-03-06T13:19:53.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:53.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:53.841304+0000) 2024-03-06T13:19:54.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:54.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:54.841565+0000) 2024-03-06T13:19:55.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:55.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:55.841815+0000) 2024-03-06T13:19:56.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:56.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:56.842090+0000) 2024-03-06T13:19:57.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:57.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:57.842362+0000) 2024-03-06T13:19:58.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:58.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:58.842663+0000) 2024-03-06T13:19:59.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:19:59.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:59.842952+0000) 2024-03-06T13:20:00.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:00.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:00.843154+0000) 2024-03-06T13:20:01.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:01.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:01.843342+0000) 2024-03-06T13:20:02.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:02.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:02.843579+0000) 2024-03-06T13:20:03.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:03.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:03.843866+0000) 2024-03-06T13:20:04.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:04.600 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:04.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:04.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:04.844145+0000) 2024-03-06T13:20:05.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:05.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:05.844427+0000) 2024-03-06T13:20:06.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:06.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:06.844726+0000) 2024-03-06T13:20:07.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:07.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:07.845001+0000) 2024-03-06T13:20:08.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:08.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:08.845258+0000) 2024-03-06T13:20:09.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:09.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:09.845539+0000) 2024-03-06T13:20:10.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:10.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:10.845824+0000) 2024-03-06T13:20:11.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:11.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:11.846095+0000) 2024-03-06T13:20:12.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:12.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:12.846369+0000) 2024-03-06T13:20:13.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:13.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:13.846671+0000) 2024-03-06T13:20:14.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:14.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:14.846934+0000) 2024-03-06T13:20:15.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:15.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:15.847152+0000) 2024-03-06T13:20:16.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:16.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:16.847358+0000) 2024-03-06T13:20:17.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:17.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:17.847620+0000) 2024-03-06T13:20:18.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:18.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:18.847971+0000) 2024-03-06T13:20:19.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:19.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:19.848239+0000) 2024-03-06T13:20:20.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:20.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:20.848499+0000) 2024-03-06T13:20:21.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:21.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:21.848762+0000) 2024-03-06T13:20:22.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:22.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:22.848973+0000) 2024-03-06T13:20:23.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:23.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:23.849232+0000) 2024-03-06T13:20:24.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:24.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:24.849489+0000) 2024-03-06T13:20:25.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:25.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:25.849759+0000) 2024-03-06T13:20:26.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:26.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:26.850223+0000) 2024-03-06T13:20:27.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:27.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:27.850480+0000) 2024-03-06T13:20:28.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:28.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:28.850756+0000) 2024-03-06T13:20:29.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:29.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:29.851013+0000) 2024-03-06T13:20:30.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:30.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:30.851276+0000) 2024-03-06T13:20:31.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:31.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:31.851510+0000) 2024-03-06T13:20:32.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:32.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:32.851780+0000) 2024-03-06T13:20:33.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:33.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:33.852040+0000) 2024-03-06T13:20:34.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:34.757 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:34.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:34.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:34.852217+0000) 2024-03-06T13:20:35.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:35.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:35.852490+0000) 2024-03-06T13:20:36.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:36.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:36.852750+0000) 2024-03-06T13:20:37.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:37.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:37.852983+0000) 2024-03-06T13:20:38.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:38.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:38.853223+0000) 2024-03-06T13:20:39.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:39.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:39.853488+0000) 2024-03-06T13:20:40.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:40.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:40.853756+0000) 2024-03-06T13:20:41.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:41.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:41.853958+0000) 2024-03-06T13:20:42.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:42.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:42.854227+0000) 2024-03-06T13:20:43.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:43.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:43.854460+0000) 2024-03-06T13:20:44.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:44.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:44.854735+0000) 2024-03-06T13:20:45.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:45.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:45.855000+0000) 2024-03-06T13:20:46.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:46.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:46.855193+0000) 2024-03-06T13:20:47.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:47.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:47.855487+0000) 2024-03-06T13:20:48.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:48.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:48.855752+0000) 2024-03-06T13:20:49.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:49.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:49.856018+0000) 2024-03-06T13:20:50.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:50.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:50.856261+0000) 2024-03-06T13:20:51.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:51.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:51.856518+0000) 2024-03-06T13:20:52.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:52.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:52.856759+0000) 2024-03-06T13:20:53.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:53.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:53.857020+0000) 2024-03-06T13:20:54.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:54.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:54.857250+0000) 2024-03-06T13:20:55.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:55.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:55.857454+0000) 2024-03-06T13:20:56.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:56.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:56.857755+0000) 2024-03-06T13:20:57.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:57.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:57.858018+0000) 2024-03-06T13:20:58.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:58.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:58.858199+0000) 2024-03-06T13:20:59.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:20:59.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:59.858394+0000) 2024-03-06T13:21:00.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:00.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:00.858677+0000) 2024-03-06T13:21:01.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:01.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:01.858924+0000) 2024-03-06T13:21:02.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:02.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:02.859180+0000) 2024-03-06T13:21:03.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:03.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:03.859420+0000) 2024-03-06T13:21:04.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:04.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:04.859725+0000) 2024-03-06T13:21:04.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:04.903 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:05.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:05.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:05.859991+0000) 2024-03-06T13:21:06.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:06.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:06.860242+0000) 2024-03-06T13:21:07.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:07.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:07.860495+0000) 2024-03-06T13:21:08.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:08.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:08.860754+0000) 2024-03-06T13:21:09.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:09.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:09.861060+0000) 2024-03-06T13:21:10.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:10.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:10.861320+0000) 2024-03-06T13:21:11.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:11.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:11.861689+0000) 2024-03-06T13:21:12.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:12.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:12.861966+0000) 2024-03-06T13:21:13.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:13.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:13.862237+0000) 2024-03-06T13:21:14.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:14.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:14.862507+0000) 2024-03-06T13:21:15.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:15.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:15.862763+0000) 2024-03-06T13:21:16.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:16.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:16.863032+0000) 2024-03-06T13:21:17.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:17.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:17.863303+0000) 2024-03-06T13:21:18.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:18.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:18.863560+0000) 2024-03-06T13:21:19.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:19.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:19.863794+0000) 2024-03-06T13:21:20.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:20.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:20.864011+0000) 2024-03-06T13:21:21.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:21.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:21.864292+0000) 2024-03-06T13:21:22.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:22.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:22.864568+0000) 2024-03-06T13:21:23.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:23.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:23.864823+0000) 2024-03-06T13:21:24.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:24.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:24.865084+0000) 2024-03-06T13:21:25.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:25.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:25.865360+0000) 2024-03-06T13:21:26.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:26.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:26.865609+0000) 2024-03-06T13:21:27.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:27.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:27.865872+0000) 2024-03-06T13:21:28.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:28.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:28.866159+0000) 2024-03-06T13:21:29.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:29.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:29.866460+0000) 2024-03-06T13:21:30.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:30.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:30.866749+0000) 2024-03-06T13:21:31.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:31.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:31.867005+0000) 2024-03-06T13:21:32.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:32.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:32.867257+0000) 2024-03-06T13:21:33.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:33.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:33.867508+0000) 2024-03-06T13:21:34.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:34.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:34.867779+0000) 2024-03-06T13:21:35.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:35.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:35.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:35.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:35.867985+0000) 2024-03-06T13:21:36.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:36.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:36.868234+0000) 2024-03-06T13:21:37.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:37.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:37.868512+0000) 2024-03-06T13:21:38.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:38.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:38.868771+0000) 2024-03-06T13:21:39.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:39.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:39.869038+0000) 2024-03-06T13:21:40.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:40.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:40.869287+0000) 2024-03-06T13:21:41.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:41.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:41.869545+0000) 2024-03-06T13:21:42.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:42.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:42.869789+0000) 2024-03-06T13:21:43.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:43.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:43.870021+0000) 2024-03-06T13:21:44.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:44.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:44.870275+0000) 2024-03-06T13:21:45.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:45.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:45.870542+0000) 2024-03-06T13:21:46.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:46.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:46.870797+0000) 2024-03-06T13:21:47.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:47.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:47.871055+0000) 2024-03-06T13:21:48.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:48.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:48.871317+0000) 2024-03-06T13:21:49.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:49.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:49.871557+0000) 2024-03-06T13:21:50.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:50.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:50.871784+0000) 2024-03-06T13:21:51.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:51.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:51.872054+0000) 2024-03-06T13:21:52.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:52.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:52.872314+0000) 2024-03-06T13:21:53.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:53.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:53.872597+0000) 2024-03-06T13:21:54.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:54.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:54.872812+0000) 2024-03-06T13:21:55.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:55.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:55.873051+0000) 2024-03-06T13:21:56.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:56.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:56.873290+0000) 2024-03-06T13:21:57.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:57.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:57.873528+0000) 2024-03-06T13:21:58.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:58.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:58.873777+0000) 2024-03-06T13:21:59.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:21:59.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:59.874048+0000) 2024-03-06T13:22:00.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:00.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:00.874291+0000) 2024-03-06T13:22:01.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:01.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:01.874576+0000) 2024-03-06T13:22:02.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:02.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:02.874822+0000) 2024-03-06T13:22:03.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:03.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:03.875052+0000) 2024-03-06T13:22:04.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:04.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:04.875244+0000) 2024-03-06T13:22:05.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:05.188 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:05.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:05.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:05.875507+0000) 2024-03-06T13:22:06.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:06.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:06.875750+0000) 2024-03-06T13:22:07.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:07.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:07.876013+0000) 2024-03-06T13:22:08.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:08.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:08.876497+0000) 2024-03-06T13:22:09.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:09.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:09.876850+0000) 2024-03-06T13:22:10.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:10.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:10.877135+0000) 2024-03-06T13:22:11.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:11.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:11.877371+0000) 2024-03-06T13:22:12.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:12.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:12.877732+0000) 2024-03-06T13:22:13.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:13.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:13.878047+0000) 2024-03-06T13:22:14.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:14.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:14.878352+0000) 2024-03-06T13:22:15.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:15.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:15.878704+0000) 2024-03-06T13:22:16.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:16.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:16.878960+0000) 2024-03-06T13:22:17.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:17.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:17.879272+0000) 2024-03-06T13:22:18.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:18.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:18.879569+0000) 2024-03-06T13:22:19.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:19.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:19.879946+0000) 2024-03-06T13:22:20.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:20.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:20.880251+0000) 2024-03-06T13:22:21.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:21.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:21.880546+0000) 2024-03-06T13:22:22.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:22.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:22.880837+0000) 2024-03-06T13:22:23.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:23.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:23.881135+0000) 2024-03-06T13:22:24.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:24.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:24.881467+0000) 2024-03-06T13:22:25.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:25.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:25.881744+0000) 2024-03-06T13:22:26.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:26.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:26.881978+0000) 2024-03-06T13:22:27.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:27.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:27.882214+0000) 2024-03-06T13:22:28.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:28.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:28.882459+0000) 2024-03-06T13:22:29.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:29.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:29.882768+0000) 2024-03-06T13:22:30.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:30.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:30.883006+0000) 2024-03-06T13:22:31.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:31.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:31.883256+0000) 2024-03-06T13:22:32.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:32.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:32.883441+0000) 2024-03-06T13:22:33.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:33.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:33.883756+0000) 2024-03-06T13:22:34.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:34.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:34.883946+0000) 2024-03-06T13:22:35.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:35.345 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:35.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:35.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:35.884201+0000) 2024-03-06T13:22:36.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:36.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:36.884433+0000) 2024-03-06T13:22:37.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:37.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:37.884721+0000) 2024-03-06T13:22:38.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:38.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:38.884953+0000) 2024-03-06T13:22:39.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:39.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:39.885229+0000) 2024-03-06T13:22:40.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:40.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:40.885425+0000) 2024-03-06T13:22:41.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:41.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:41.885734+0000) 2024-03-06T13:22:42.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:42.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:42.885975+0000) 2024-03-06T13:22:43.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:43.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:43.886232+0000) 2024-03-06T13:22:44.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:44.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:44.886467+0000) 2024-03-06T13:22:45.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:45.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:45.886758+0000) 2024-03-06T13:22:46.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:46.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:46.886996+0000) 2024-03-06T13:22:47.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:47.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:47.887240+0000) 2024-03-06T13:22:48.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:48.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:48.887441+0000) 2024-03-06T13:22:49.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:49.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:49.887750+0000) 2024-03-06T13:22:50.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:50.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:50.887989+0000) 2024-03-06T13:22:51.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:51.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:51.888233+0000) 2024-03-06T13:22:52.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:52.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:52.888487+0000) 2024-03-06T13:22:53.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:53.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:53.888747+0000) 2024-03-06T13:22:54.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:54.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:54.889006+0000) 2024-03-06T13:22:55.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:55.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:55.889231+0000) 2024-03-06T13:22:56.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:56.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:56.889545+0000) 2024-03-06T13:22:57.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:57.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:57.889799+0000) 2024-03-06T13:22:58.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:58.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:58.890055+0000) 2024-03-06T13:22:59.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:22:59.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:59.890324+0000) 2024-03-06T13:23:00.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:00.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:00.890595+0000) 2024-03-06T13:23:01.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:01.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:01.890887+0000) 2024-03-06T13:23:02.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:02.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:02.891159+0000) 2024-03-06T13:23:03.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:03.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:03.891441+0000) 2024-03-06T13:23:04.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:04.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:04.891753+0000) 2024-03-06T13:23:05.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:05.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:05.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:05.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:05.892009+0000) 2024-03-06T13:23:06.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:06.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:06.892264+0000) 2024-03-06T13:23:07.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:07.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:07.892523+0000) 2024-03-06T13:23:08.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:08.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:08.892793+0000) 2024-03-06T13:23:09.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:09.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:09.893073+0000) 2024-03-06T13:23:10.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:10.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:10.893345+0000) 2024-03-06T13:23:11.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:11.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:11.893606+0000) 2024-03-06T13:23:12.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:12.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:12.893939+0000) 2024-03-06T13:23:13.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:13.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:13.894200+0000) 2024-03-06T13:23:14.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:14.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:14.894462+0000) 2024-03-06T13:23:15.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:15.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:15.894737+0000) 2024-03-06T13:23:16.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:16.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:16.895016+0000) 2024-03-06T13:23:17.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:17.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:17.895272+0000) 2024-03-06T13:23:18.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:18.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:18.895527+0000) 2024-03-06T13:23:19.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:19.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:19.895831+0000) 2024-03-06T13:23:20.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:20.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:20.896048+0000) 2024-03-06T13:23:21.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:21.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:21.896324+0000) 2024-03-06T13:23:22.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:22.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:22.896604+0000) 2024-03-06T13:23:23.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:23.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:23.896874+0000) 2024-03-06T13:23:24.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:24.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:24.897149+0000) 2024-03-06T13:23:25.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:25.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:25.897405+0000) 2024-03-06T13:23:26.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:26.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:26.897724+0000) 2024-03-06T13:23:27.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:27.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:27.897978+0000) 2024-03-06T13:23:28.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:28.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:28.898264+0000) 2024-03-06T13:23:29.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:29.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:29.898555+0000) 2024-03-06T13:23:30.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:30.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:30.898814+0000) 2024-03-06T13:23:31.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:31.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:31.899076+0000) 2024-03-06T13:23:32.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:32.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:32.899331+0000) 2024-03-06T13:23:33.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:33.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:33.899615+0000) 2024-03-06T13:23:34.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:34.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:34.899939+0000) 2024-03-06T13:23:35.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:35.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:35.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:35.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:35.900217+0000) 2024-03-06T13:23:36.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:36.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:36.900483+0000) 2024-03-06T13:23:37.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:37.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:37.900757+0000) 2024-03-06T13:23:38.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:38.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:38.901012+0000) 2024-03-06T13:23:39.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:39.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:39.901297+0000) 2024-03-06T13:23:40.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:40.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:40.901573+0000) 2024-03-06T13:23:41.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:41.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:41.901850+0000) 2024-03-06T13:23:42.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:42.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:42.902105+0000) 2024-03-06T13:23:43.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:43.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:43.902385+0000) 2024-03-06T13:23:44.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:44.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:44.902758+0000) 2024-03-06T13:23:45.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:45.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:45.903021+0000) 2024-03-06T13:23:46.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:46.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:46.903282+0000) 2024-03-06T13:23:47.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:47.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:47.903498+0000) 2024-03-06T13:23:48.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:48.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:48.903776+0000) 2024-03-06T13:23:49.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:49.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:49.904054+0000) 2024-03-06T13:23:50.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:50.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:50.904311+0000) 2024-03-06T13:23:51.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:51.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:51.904552+0000) 2024-03-06T13:23:52.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:52.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:52.904752+0000) 2024-03-06T13:23:53.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:53.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:53.904994+0000) 2024-03-06T13:23:54.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:54.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:54.905252+0000) 2024-03-06T13:23:55.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:55.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:55.905484+0000) 2024-03-06T13:23:56.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:56.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:56.905753+0000) 2024-03-06T13:23:57.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:57.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:57.906000+0000) 2024-03-06T13:23:58.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:58.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:58.906245+0000) 2024-03-06T13:23:59.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:23:59.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:59.906514+0000) 2024-03-06T13:24:00.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:00.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:00.906782+0000) 2024-03-06T13:24:01.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:01.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:01.907029+0000) 2024-03-06T13:24:02.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:02.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:02.907283+0000) 2024-03-06T13:24:03.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:03.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:03.907529+0000) 2024-03-06T13:24:04.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:04.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:04.907770+0000) 2024-03-06T13:24:05.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:05.790 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:05.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:05.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:05.908029+0000) 2024-03-06T13:24:06.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:06.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:06.908327+0000) 2024-03-06T13:24:07.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:07.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:07.908519+0000) 2024-03-06T13:24:08.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:08.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:08.908783+0000) 2024-03-06T13:24:09.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:09.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:09.909062+0000) 2024-03-06T13:24:10.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:10.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:10.909293+0000) 2024-03-06T13:24:11.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:11.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:11.909552+0000) 2024-03-06T13:24:12.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:12.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:12.909804+0000) 2024-03-06T13:24:13.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:13.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:13.910058+0000) 2024-03-06T13:24:14.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:14.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:14.910301+0000) 2024-03-06T13:24:15.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:15.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:15.910542+0000) 2024-03-06T13:24:16.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:16.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:16.910823+0000) 2024-03-06T13:24:17.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:17.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:17.911102+0000) 2024-03-06T13:24:18.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:18.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:18.911396+0000) 2024-03-06T13:24:19.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:19.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:19.911725+0000) 2024-03-06T13:24:20.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:20.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:20.912041+0000) 2024-03-06T13:24:21.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:21.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:21.912334+0000) 2024-03-06T13:24:22.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:22.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:22.912681+0000) 2024-03-06T13:24:23.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:23.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:23.912941+0000) 2024-03-06T13:24:24.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:24.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:24.913219+0000) 2024-03-06T13:24:25.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:25.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:25.913497+0000) 2024-03-06T13:24:26.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:26.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:26.913809+0000) 2024-03-06T13:24:27.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:27.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:27.914118+0000) 2024-03-06T13:24:28.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:28.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:28.914399+0000) 2024-03-06T13:24:29.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:29.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:29.914736+0000) 2024-03-06T13:24:30.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:30.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:30.915040+0000) 2024-03-06T13:24:31.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:31.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:31.915333+0000) 2024-03-06T13:24:32.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:32.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:32.915607+0000) 2024-03-06T13:24:33.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:33.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:33.915942+0000) 2024-03-06T13:24:34.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:34.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:34.916239+0000) 2024-03-06T13:24:35.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:35.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:35.916557+0000) 2024-03-06T13:24:35.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:35.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:36.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:36.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:36.916842+0000) 2024-03-06T13:24:37.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:37.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:37.917217+0000) 2024-03-06T13:24:38.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:38.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:38.917520+0000) 2024-03-06T13:24:39.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:39.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:39.917826+0000) 2024-03-06T13:24:40.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:40.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:40.918042+0000) 2024-03-06T13:24:41.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:41.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:41.918304+0000) 2024-03-06T13:24:42.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:42.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:42.918563+0000) 2024-03-06T13:24:43.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:43.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:43.918811+0000) 2024-03-06T13:24:44.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:44.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:44.919046+0000) 2024-03-06T13:24:45.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:45.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:45.919297+0000) 2024-03-06T13:24:46.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:46.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:46.919555+0000) 2024-03-06T13:24:47.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:47.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:47.919796+0000) 2024-03-06T13:24:48.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:48.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:48.920058+0000) 2024-03-06T13:24:49.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:49.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:49.920321+0000) 2024-03-06T13:24:50.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:50.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:50.920623+0000) 2024-03-06T13:24:51.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:51.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:51.920932+0000) 2024-03-06T13:24:52.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:52.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:52.921227+0000) 2024-03-06T13:24:53.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:53.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:53.921540+0000) 2024-03-06T13:24:54.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:54.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:54.921844+0000) 2024-03-06T13:24:55.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:55.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:55.922145+0000) 2024-03-06T13:24:56.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:56.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:56.922442+0000) 2024-03-06T13:24:57.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:57.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:57.922766+0000) 2024-03-06T13:24:58.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:58.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:58.923092+0000) 2024-03-06T13:24:59.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:24:59.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:59.923418+0000) 2024-03-06T13:25:00.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:00.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:00.923769+0000) 2024-03-06T13:25:01.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:01.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:01.924074+0000) 2024-03-06T13:25:02.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:02.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:02.924378+0000) 2024-03-06T13:25:03.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:03.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:03.924720+0000) 2024-03-06T13:25:04.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:04.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:04.925004+0000) 2024-03-06T13:25:05.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:05.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:05.925283+0000) 2024-03-06T13:25:06.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:06.102 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:06.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:06.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:06.925533+0000) 2024-03-06T13:25:07.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:07.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:07.925796+0000) 2024-03-06T13:25:08.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:08.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:08.926041+0000) 2024-03-06T13:25:09.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:09.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:09.926308+0000) 2024-03-06T13:25:10.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:10.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:10.926552+0000) 2024-03-06T13:25:11.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:11.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:11.926755+0000) 2024-03-06T13:25:12.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:12.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:12.926993+0000) 2024-03-06T13:25:13.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:13.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:13.927183+0000) 2024-03-06T13:25:14.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:14.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:14.927364+0000) 2024-03-06T13:25:15.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:15.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:15.927622+0000) 2024-03-06T13:25:16.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:16.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:16.927969+0000) 2024-03-06T13:25:17.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:17.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:17.928169+0000) 2024-03-06T13:25:18.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:18.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:18.928420+0000) 2024-03-06T13:25:19.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:19.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:19.928739+0000) 2024-03-06T13:25:20.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:20.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:20.928953+0000) 2024-03-06T13:25:21.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:21.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:21.929213+0000) 2024-03-06T13:25:22.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:22.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:22.929465+0000) 2024-03-06T13:25:23.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:23.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:23.929757+0000) 2024-03-06T13:25:24.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:24.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:24.929945+0000) 2024-03-06T13:25:25.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:25.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:25.930254+0000) 2024-03-06T13:25:26.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:26.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:26.930517+0000) 2024-03-06T13:25:27.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:27.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:27.930759+0000) 2024-03-06T13:25:28.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:28.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:28.930959+0000) 2024-03-06T13:25:29.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:29.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:29.931230+0000) 2024-03-06T13:25:30.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:30.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:30.931474+0000) 2024-03-06T13:25:31.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:31.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:31.931764+0000) 2024-03-06T13:25:32.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:32.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:32.931963+0000) 2024-03-06T13:25:33.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:33.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:33.932198+0000) 2024-03-06T13:25:34.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:34.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:34.932436+0000) 2024-03-06T13:25:35.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:35.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:35.932728+0000) 2024-03-06T13:25:36.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:36.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:36.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:36.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:36.932959+0000) 2024-03-06T13:25:37.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:37.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:37.933215+0000) 2024-03-06T13:25:38.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:38.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:38.933461+0000) 2024-03-06T13:25:39.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:39.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:39.933736+0000) 2024-03-06T13:25:40.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:40.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:40.933988+0000) 2024-03-06T13:25:41.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:41.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:41.934250+0000) 2024-03-06T13:25:42.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:42.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:42.934493+0000) 2024-03-06T13:25:43.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:43.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:43.934728+0000) 2024-03-06T13:25:44.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:44.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:44.934987+0000) 2024-03-06T13:25:45.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:45.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:45.935213+0000) 2024-03-06T13:25:46.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:46.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:46.935476+0000) 2024-03-06T13:25:47.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:47.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:47.935741+0000) 2024-03-06T13:25:48.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:48.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:48.935974+0000) 2024-03-06T13:25:49.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:49.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:49.936244+0000) 2024-03-06T13:25:50.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:50.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:50.936481+0000) 2024-03-06T13:25:51.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:51.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:51.936768+0000) 2024-03-06T13:25:52.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:52.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:52.937025+0000) 2024-03-06T13:25:53.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:53.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:53.937280+0000) 2024-03-06T13:25:54.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:54.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:54.937510+0000) 2024-03-06T13:25:55.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:55.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:55.937766+0000) 2024-03-06T13:25:56.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:56.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:56.938017+0000) 2024-03-06T13:25:57.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:57.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:57.938265+0000) 2024-03-06T13:25:58.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:58.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:58.938520+0000) 2024-03-06T13:25:59.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:25:59.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:59.938790+0000) 2024-03-06T13:26:00.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:00.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:00.939040+0000) 2024-03-06T13:26:01.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:01.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:01.939271+0000) 2024-03-06T13:26:02.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:02.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:02.939529+0000) 2024-03-06T13:26:03.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:03.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:03.939763+0000) 2024-03-06T13:26:04.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:04.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:04.939955+0000) 2024-03-06T13:26:05.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:05.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:05.940194+0000) 2024-03-06T13:26:06.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:06.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:06.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:06.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:06.940440+0000) 2024-03-06T13:26:07.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:07.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:07.940741+0000) 2024-03-06T13:26:08.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:08.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:08.940992+0000) 2024-03-06T13:26:09.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:09.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:09.941259+0000) 2024-03-06T13:26:10.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:10.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:10.941511+0000) 2024-03-06T13:26:11.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:11.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:11.941778+0000) 2024-03-06T13:26:12.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:12.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:12.942019+0000) 2024-03-06T13:26:13.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:13.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:13.942324+0000) 2024-03-06T13:26:14.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:14.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:14.942581+0000) 2024-03-06T13:26:15.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:15.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:15.942840+0000) 2024-03-06T13:26:16.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:16.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:16.943082+0000) 2024-03-06T13:26:17.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:17.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:17.943333+0000) 2024-03-06T13:26:18.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:18.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:18.943593+0000) 2024-03-06T13:26:19.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:19.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:19.943881+0000) 2024-03-06T13:26:20.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:20.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:20.944144+0000) 2024-03-06T13:26:21.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:21.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:21.944396+0000) 2024-03-06T13:26:22.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:22.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:22.944691+0000) 2024-03-06T13:26:23.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:23.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:23.944953+0000) 2024-03-06T13:26:24.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:24.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:24.945198+0000) 2024-03-06T13:26:25.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:25.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:25.945453+0000) 2024-03-06T13:26:26.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:26.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:26.945739+0000) 2024-03-06T13:26:27.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:27.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:27.945972+0000) 2024-03-06T13:26:28.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:28.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:28.946211+0000) 2024-03-06T13:26:29.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:29.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:29.946483+0000) 2024-03-06T13:26:30.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:30.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:30.946748+0000) 2024-03-06T13:26:31.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:31.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:31.947011+0000) 2024-03-06T13:26:32.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:32.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:32.947215+0000) 2024-03-06T13:26:33.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:33.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:33.947454+0000) 2024-03-06T13:26:34.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:34.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:34.947745+0000) 2024-03-06T13:26:35.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:35.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:35.947987+0000) 2024-03-06T13:26:36.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:36.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:36.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:36.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:36.948191+0000) 2024-03-06T13:26:37.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:37.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:37.948454+0000) 2024-03-06T13:26:38.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:38.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:38.948751+0000) 2024-03-06T13:26:39.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:39.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:39.948999+0000) 2024-03-06T13:26:40.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:40.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:40.949245+0000) 2024-03-06T13:26:41.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:41.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:41.949485+0000) 2024-03-06T13:26:42.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:42.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:42.949736+0000) 2024-03-06T13:26:43.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:43.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:43.949989+0000) 2024-03-06T13:26:44.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:44.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:44.950245+0000) 2024-03-06T13:26:45.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:45.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:45.950500+0000) 2024-03-06T13:26:46.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:46.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:46.950766+0000) 2024-03-06T13:26:47.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:47.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:47.950961+0000) 2024-03-06T13:26:48.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:48.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:48.951220+0000) 2024-03-06T13:26:49.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:49.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:49.951482+0000) 2024-03-06T13:26:50.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:50.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:50.951750+0000) 2024-03-06T13:26:51.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:51.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:51.952006+0000) 2024-03-06T13:26:52.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:52.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:52.952255+0000) 2024-03-06T13:26:53.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:53.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:53.952515+0000) 2024-03-06T13:26:54.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:54.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:54.952772+0000) 2024-03-06T13:26:55.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:55.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:55.952986+0000) 2024-03-06T13:26:56.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:56.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:56.953229+0000) 2024-03-06T13:26:57.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:57.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:57.953494+0000) 2024-03-06T13:26:58.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:58.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:58.953754+0000) 2024-03-06T13:26:59.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:26:59.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:59.954025+0000) 2024-03-06T13:27:00.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:00.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:00.954286+0000) 2024-03-06T13:27:01.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:01.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:01.954580+0000) 2024-03-06T13:27:02.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:02.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:02.954887+0000) 2024-03-06T13:27:03.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:03.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:03.955177+0000) 2024-03-06T13:27:04.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:04.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:04.955469+0000) 2024-03-06T13:27:05.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:05.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:05.955793+0000) 2024-03-06T13:27:06.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:06.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:06.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:06.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:06.956123+0000) 2024-03-06T13:27:07.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:07.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:07.956410+0000) 2024-03-06T13:27:08.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:08.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:08.956746+0000) 2024-03-06T13:27:09.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:09.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:09.957081+0000) 2024-03-06T13:27:10.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:10.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:10.957394+0000) 2024-03-06T13:27:11.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:11.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:11.957751+0000) 2024-03-06T13:27:12.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:12.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:12.957999+0000) 2024-03-06T13:27:13.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:13.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:13.958297+0000) 2024-03-06T13:27:14.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:14.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:14.958603+0000) 2024-03-06T13:27:15.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:15.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:15.958965+0000) 2024-03-06T13:27:16.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:16.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:16.959266+0000) 2024-03-06T13:27:17.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:17.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:17.959559+0000) 2024-03-06T13:27:18.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:18.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:18.959860+0000) 2024-03-06T13:27:19.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:19.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:19.960182+0000) 2024-03-06T13:27:20.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:20.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:20.960485+0000) 2024-03-06T13:27:21.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:21.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:21.960798+0000) 2024-03-06T13:27:22.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:22.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:22.961109+0000) 2024-03-06T13:27:23.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:23.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:23.961431+0000) 2024-03-06T13:27:24.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:24.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:24.961768+0000) 2024-03-06T13:27:25.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:25.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:25.962088+0000) 2024-03-06T13:27:26.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:26.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:26.962387+0000) 2024-03-06T13:27:27.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:27.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:27.962666+0000) 2024-03-06T13:27:28.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:28.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:28.962978+0000) 2024-03-06T13:27:29.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:29.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:29.963250+0000) 2024-03-06T13:27:30.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:30.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:30.963558+0000) 2024-03-06T13:27:31.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:31.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:31.963854+0000) 2024-03-06T13:27:32.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:32.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:32.964126+0000) 2024-03-06T13:27:33.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:33.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:33.964426+0000) 2024-03-06T13:27:34.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:34.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:34.964772+0000) 2024-03-06T13:27:35.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:35.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:35.965096+0000) 2024-03-06T13:27:36.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:36.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:36.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:36.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:36.965382+0000) 2024-03-06T13:27:37.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:37.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:37.965742+0000) 2024-03-06T13:27:38.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:38.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:38.965990+0000) 2024-03-06T13:27:39.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:39.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:39.966295+0000) 2024-03-06T13:27:40.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:40.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:40.966623+0000) 2024-03-06T13:27:41.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:41.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:41.966930+0000) 2024-03-06T13:27:42.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:42.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:42.967203+0000) 2024-03-06T13:27:43.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:43.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:43.967406+0000) 2024-03-06T13:27:44.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:44.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:44.967658+0000) 2024-03-06T13:27:45.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:45.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:45.967911+0000) 2024-03-06T13:27:46.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:46.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:46.968161+0000) 2024-03-06T13:27:47.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:47.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:47.968400+0000) 2024-03-06T13:27:48.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:48.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:48.968667+0000) 2024-03-06T13:27:49.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:49.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:49.968936+0000) 2024-03-06T13:27:50.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:50.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:50.969190+0000) 2024-03-06T13:27:51.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:51.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:51.969444+0000) 2024-03-06T13:27:52.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:52.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:52.969747+0000) 2024-03-06T13:27:53.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:53.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:53.969998+0000) 2024-03-06T13:27:54.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:54.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:54.970296+0000) 2024-03-06T13:27:55.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:55.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:55.970533+0000) 2024-03-06T13:27:56.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:56.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:56.970773+0000) 2024-03-06T13:27:57.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:57.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:57.971029+0000) 2024-03-06T13:27:58.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:58.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:58.971281+0000) 2024-03-06T13:27:59.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:27:59.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:59.971559+0000) 2024-03-06T13:28:00.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:00.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:00.971826+0000) 2024-03-06T13:28:01.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:01.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:01.972025+0000) 2024-03-06T13:28:02.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:02.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:02.972282+0000) 2024-03-06T13:28:03.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:03.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:03.972528+0000) 2024-03-06T13:28:04.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:04.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:04.972771+0000) 2024-03-06T13:28:05.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:05.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:05.973030+0000) 2024-03-06T13:28:06.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:06.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:06.973282+0000) 2024-03-06T13:28:06.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:06.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:07.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:07.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:07.973546+0000) 2024-03-06T13:28:08.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:08.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:08.973814+0000) 2024-03-06T13:28:09.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:09.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:09.974089+0000) 2024-03-06T13:28:10.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:10.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:10.974330+0000) 2024-03-06T13:28:11.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:11.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:11.974580+0000) 2024-03-06T13:28:12.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:12.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:12.974819+0000) 2024-03-06T13:28:13.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:13.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:13.975028+0000) 2024-03-06T13:28:14.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:14.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:14.975318+0000) 2024-03-06T13:28:15.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:15.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:15.975620+0000) 2024-03-06T13:28:16.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:16.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:16.975982+0000) 2024-03-06T13:28:17.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:17.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:17.976289+0000) 2024-03-06T13:28:18.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:18.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:18.976584+0000) 2024-03-06T13:28:19.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:19.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:19.976966+0000) 2024-03-06T13:28:20.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:20.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:20.977248+0000) 2024-03-06T13:28:21.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:21.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:21.977530+0000) 2024-03-06T13:28:22.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:22.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:22.977845+0000) 2024-03-06T13:28:23.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:23.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:23.978082+0000) 2024-03-06T13:28:24.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:24.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:24.978323+0000) 2024-03-06T13:28:25.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:25.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:25.978576+0000) 2024-03-06T13:28:26.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:26.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:26.978831+0000) 2024-03-06T13:28:27.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:27.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:27.979064+0000) 2024-03-06T13:28:28.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:28.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:28.979247+0000) 2024-03-06T13:28:29.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:29.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:29.979514+0000) 2024-03-06T13:28:30.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:30.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:30.979755+0000) 2024-03-06T13:28:31.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:31.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:31.979996+0000) 2024-03-06T13:28:32.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:32.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:32.980258+0000) 2024-03-06T13:28:33.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:33.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:33.980549+0000) 2024-03-06T13:28:34.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:34.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:34.980843+0000) 2024-03-06T13:28:35.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:35.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:35.981146+0000) 2024-03-06T13:28:36.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:36.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:36.981493+0000) 2024-03-06T13:28:37.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:37.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:37.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:37.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:37.981805+0000) 2024-03-06T13:28:38.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:38.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:38.982111+0000) 2024-03-06T13:28:39.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:39.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:39.982428+0000) 2024-03-06T13:28:40.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:40.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:40.982738+0000) 2024-03-06T13:28:41.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:41.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:41.983000+0000) 2024-03-06T13:28:42.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:42.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:42.983247+0000) 2024-03-06T13:28:43.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:43.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:43.983539+0000) 2024-03-06T13:28:44.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:44.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:44.983839+0000) 2024-03-06T13:28:45.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:45.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:45.984136+0000) 2024-03-06T13:28:46.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:46.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:46.984441+0000) 2024-03-06T13:28:47.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:47.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:47.984781+0000) 2024-03-06T13:28:48.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:48.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:48.985067+0000) 2024-03-06T13:28:49.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:49.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:49.985377+0000) 2024-03-06T13:28:50.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:50.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:50.985726+0000) 2024-03-06T13:28:51.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:51.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:51.986031+0000) 2024-03-06T13:28:52.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:52.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:52.986339+0000) 2024-03-06T13:28:53.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:53.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:53.986608+0000) 2024-03-06T13:28:54.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:54.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:54.986889+0000) 2024-03-06T13:28:55.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:55.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:55.987170+0000) 2024-03-06T13:28:56.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:56.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:56.987468+0000) 2024-03-06T13:28:57.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:57.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:57.987764+0000) 2024-03-06T13:28:58.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:58.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:58.988013+0000) 2024-03-06T13:28:59.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:28:59.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:59.988280+0000) 2024-03-06T13:29:00.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:00.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:00.988536+0000) 2024-03-06T13:29:01.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:01.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:01.988774+0000) 2024-03-06T13:29:02.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:02.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:02.989034+0000) 2024-03-06T13:29:03.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:03.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:03.989295+0000) 2024-03-06T13:29:04.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:04.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:04.989557+0000) 2024-03-06T13:29:05.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:05.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:05.989798+0000) 2024-03-06T13:29:06.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:06.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:06.990032+0000) 2024-03-06T13:29:07.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:07.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:07.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:07.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:07.990298+0000) 2024-03-06T13:29:08.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:08.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:08.990557+0000) 2024-03-06T13:29:09.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:09.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:09.990828+0000) 2024-03-06T13:29:10.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:10.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:10.991085+0000) 2024-03-06T13:29:11.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:11.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:11.991275+0000) 2024-03-06T13:29:12.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:12.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:12.991540+0000) 2024-03-06T13:29:13.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:13.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:13.991794+0000) 2024-03-06T13:29:14.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:14.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:14.992051+0000) 2024-03-06T13:29:15.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:15.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:15.992299+0000) 2024-03-06T13:29:16.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:16.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:16.992540+0000) 2024-03-06T13:29:17.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:17.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:17.992790+0000) 2024-03-06T13:29:18.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:18.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:18.993039+0000) 2024-03-06T13:29:19.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:19.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:19.993296+0000) 2024-03-06T13:29:20.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:20.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:20.993553+0000) 2024-03-06T13:29:21.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:21.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:21.993805+0000) 2024-03-06T13:29:22.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:22.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:22.994070+0000) 2024-03-06T13:29:23.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:23.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:23.994338+0000) 2024-03-06T13:29:24.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:24.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:24.994567+0000) 2024-03-06T13:29:25.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:25.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:25.994768+0000) 2024-03-06T13:29:26.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:26.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:26.995024+0000) 2024-03-06T13:29:27.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:27.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:27.995298+0000) 2024-03-06T13:29:28.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:28.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:28.995555+0000) 2024-03-06T13:29:29.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:29.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:29.995802+0000) 2024-03-06T13:29:30.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:30.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:30.996057+0000) 2024-03-06T13:29:31.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:31.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:31.996296+0000) 2024-03-06T13:29:32.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:32.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:32.996562+0000) 2024-03-06T13:29:33.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:33.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:33.996795+0000) 2024-03-06T13:29:34.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:34.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:34.997051+0000) 2024-03-06T13:29:35.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:35.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:35.997310+0000) 2024-03-06T13:29:36.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:36.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:36.997569+0000) 2024-03-06T13:29:37.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:37.462 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:37.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:37.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:37.997838+0000) 2024-03-06T13:29:38.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:38.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:38.998097+0000) 2024-03-06T13:29:39.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:39.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:39.998360+0000) 2024-03-06T13:29:40.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:40.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:40.998592+0000) 2024-03-06T13:29:41.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:41.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:41.998846+0000) 2024-03-06T13:29:42.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:42.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:42.999108+0000) 2024-03-06T13:29:43.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:43.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:43.999367+0000) 2024-03-06T13:29:44.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:44.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:44.999663+0000) 2024-03-06T13:29:45.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:45.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:45.999959+0000) 2024-03-06T13:29:46.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:46.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:47.000196+0000) 2024-03-06T13:29:47.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:47.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:48.000456+0000) 2024-03-06T13:29:48.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:49.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:49.000747+0000) 2024-03-06T13:29:49.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:50.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:50.000985+0000) 2024-03-06T13:29:50.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:51.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:51.001243+0000) 2024-03-06T13:29:51.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:52.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:52.001501+0000) 2024-03-06T13:29:52.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:53.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:53.001754+0000) 2024-03-06T13:29:53.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:54.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:54.001944+0000) 2024-03-06T13:29:54.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:55.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:55.002179+0000) 2024-03-06T13:29:55.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:56.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:56.002423+0000) 2024-03-06T13:29:56.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:57.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:57.002692+0000) 2024-03-06T13:29:57.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:58.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:58.002946+0000) 2024-03-06T13:29:58.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:29:59.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:59.003202+0000) 2024-03-06T13:29:59.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:00.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:00.003455+0000) 2024-03-06T13:30:00.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:01.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:01.003752+0000) 2024-03-06T13:30:01.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:02.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:02.003993+0000) 2024-03-06T13:30:02.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:03.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:03.004263+0000) 2024-03-06T13:30:03.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:04.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:04.004519+0000) 2024-03-06T13:30:04.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:05.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:05.004792+0000) 2024-03-06T13:30:05.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:06.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:06.005055+0000) 2024-03-06T13:30:06.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:07.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:07.005331+0000) 2024-03-06T13:30:07.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:07.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:07.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:08.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:08.005616+0000) 2024-03-06T13:30:08.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:09.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:09.005996+0000) 2024-03-06T13:30:09.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:10.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:10.006251+0000) 2024-03-06T13:30:10.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:11.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:11.006510+0000) 2024-03-06T13:30:11.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:12.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:12.006761+0000) 2024-03-06T13:30:12.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:13.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:13.007029+0000) 2024-03-06T13:30:13.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:14.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:14.007283+0000) 2024-03-06T13:30:14.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:15.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:15.007527+0000) 2024-03-06T13:30:15.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:16.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:16.007785+0000) 2024-03-06T13:30:16.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:17.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:17.008052+0000) 2024-03-06T13:30:17.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:18.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:18.008322+0000) 2024-03-06T13:30:18.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:19.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:19.008570+0000) 2024-03-06T13:30:19.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:20.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:20.008885+0000) 2024-03-06T13:30:20.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:21.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:21.009126+0000) 2024-03-06T13:30:21.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:22.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:22.009361+0000) 2024-03-06T13:30:22.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:23.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:23.009564+0000) 2024-03-06T13:30:23.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:24.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:24.009845+0000) 2024-03-06T13:30:24.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:25.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:25.010140+0000) 2024-03-06T13:30:25.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:26.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:26.010332+0000) 2024-03-06T13:30:27.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:27.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:27.010580+0000) 2024-03-06T13:30:28.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:28.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:28.010760+0000) 2024-03-06T13:30:29.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:29.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:29.011003+0000) 2024-03-06T13:30:30.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:30.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:30.011267+0000) 2024-03-06T13:30:31.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:31.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:31.011527+0000) 2024-03-06T13:30:32.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:32.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:32.011760+0000) 2024-03-06T13:30:33.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:33.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:33.011959+0000) 2024-03-06T13:30:34.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:34.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:34.012160+0000) 2024-03-06T13:30:35.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:35.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:35.012443+0000) 2024-03-06T13:30:36.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:36.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:36.012769+0000) 2024-03-06T13:30:37.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:37.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:37.013021+0000) 2024-03-06T13:30:37.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:37.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:38.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:38.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:38.013299+0000) 2024-03-06T13:30:39.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:39.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:39.013582+0000) 2024-03-06T13:30:40.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:40.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:40.013847+0000) 2024-03-06T13:30:41.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:41.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:41.014113+0000) 2024-03-06T13:30:42.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:42.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:42.014378+0000) 2024-03-06T13:30:43.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:43.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:43.014689+0000) 2024-03-06T13:30:44.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:44.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:44.014939+0000) 2024-03-06T13:30:45.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:45.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:45.015182+0000) 2024-03-06T13:30:46.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:46.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:46.015442+0000) 2024-03-06T13:30:47.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:47.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:47.015755+0000) 2024-03-06T13:30:48.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:48.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:48.016020+0000) 2024-03-06T13:30:49.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:49.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:49.016291+0000) 2024-03-06T13:30:50.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:50.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:50.016546+0000) 2024-03-06T13:30:51.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:51.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:51.016829+0000) 2024-03-06T13:30:52.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:52.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:52.017061+0000) 2024-03-06T13:30:53.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:53.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:53.017343+0000) 2024-03-06T13:30:54.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:54.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:54.017554+0000) 2024-03-06T13:30:55.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:55.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:55.017796+0000) 2024-03-06T13:30:56.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:56.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:56.018065+0000) 2024-03-06T13:30:57.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:57.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:57.018336+0000) 2024-03-06T13:30:58.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:58.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:58.018581+0000) 2024-03-06T13:30:59.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:30:59.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:59.018819+0000) 2024-03-06T13:31:00.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:00.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:00.019132+0000) 2024-03-06T13:31:01.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:01.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:01.019402+0000) 2024-03-06T13:31:02.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:02.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:02.019673+0000) 2024-03-06T13:31:03.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:03.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:03.019936+0000) 2024-03-06T13:31:04.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:04.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:04.020122+0000) 2024-03-06T13:31:05.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:05.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:05.020320+0000) 2024-03-06T13:31:06.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:06.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:06.020565+0000) 2024-03-06T13:31:07.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:07.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:07.020791+0000) 2024-03-06T13:31:07.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:07.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:08.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:08.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:08.021005+0000) 2024-03-06T13:31:09.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:09.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:09.021245+0000) 2024-03-06T13:31:10.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:10.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:10.021490+0000) 2024-03-06T13:31:11.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:11.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:11.021755+0000) 2024-03-06T13:31:12.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:12.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:12.022011+0000) 2024-03-06T13:31:13.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:13.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:13.022340+0000) 2024-03-06T13:31:14.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:14.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:14.022576+0000) 2024-03-06T13:31:15.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:15.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:15.022818+0000) 2024-03-06T13:31:16.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:16.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:16.023061+0000) 2024-03-06T13:31:17.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:17.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:17.023313+0000) 2024-03-06T13:31:18.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:18.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:18.023549+0000) 2024-03-06T13:31:19.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:19.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:19.023801+0000) 2024-03-06T13:31:20.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:20.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:20.023995+0000) 2024-03-06T13:31:21.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:21.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:21.024238+0000) 2024-03-06T13:31:22.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:22.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:22.024483+0000) 2024-03-06T13:31:23.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:23.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:23.024760+0000) 2024-03-06T13:31:24.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:24.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:24.025021+0000) 2024-03-06T13:31:25.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:25.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:25.025276+0000) 2024-03-06T13:31:26.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:26.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:26.025543+0000) 2024-03-06T13:31:27.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:27.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:27.025791+0000) 2024-03-06T13:31:28.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:28.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:28.026050+0000) 2024-03-06T13:31:29.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:29.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:29.026316+0000) 2024-03-06T13:31:30.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:30.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:30.026584+0000) 2024-03-06T13:31:31.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:31.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:31.026849+0000) 2024-03-06T13:31:32.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:32.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:32.027280+0000) 2024-03-06T13:31:33.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:33.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:33.027531+0000) 2024-03-06T13:31:34.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:34.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:34.027794+0000) 2024-03-06T13:31:35.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:35.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:35.028048+0000) 2024-03-06T13:31:36.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:36.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:36.028249+0000) 2024-03-06T13:31:37.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:37.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:37.028488+0000) 2024-03-06T13:31:38.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:38.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:38.028744+0000) 2024-03-06T13:31:38.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:38.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:39.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:39.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:39.028943+0000) 2024-03-06T13:31:40.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:40.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:40.029205+0000) 2024-03-06T13:31:41.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:41.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:41.029449+0000) 2024-03-06T13:31:42.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:42.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:42.029743+0000) 2024-03-06T13:31:43.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:43.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:43.029976+0000) 2024-03-06T13:31:44.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:44.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:44.030226+0000) 2024-03-06T13:31:45.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:45.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:45.030474+0000) 2024-03-06T13:31:46.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:46.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:46.030766+0000) 2024-03-06T13:31:47.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:47.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:47.031026+0000) 2024-03-06T13:31:48.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:48.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:48.031264+0000) 2024-03-06T13:31:49.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:49.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:49.031491+0000) 2024-03-06T13:31:50.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:50.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:50.031772+0000) 2024-03-06T13:31:51.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:51.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:51.032034+0000) 2024-03-06T13:31:52.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:52.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:52.032298+0000) 2024-03-06T13:31:53.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:53.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:53.032558+0000) 2024-03-06T13:31:54.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:54.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:54.032759+0000) 2024-03-06T13:31:55.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:55.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:55.033012+0000) 2024-03-06T13:31:56.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:56.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:56.033280+0000) 2024-03-06T13:31:57.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:57.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:57.033531+0000) 2024-03-06T13:31:58.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:58.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:58.033768+0000) 2024-03-06T13:31:59.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:31:59.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:59.034018+0000) 2024-03-06T13:32:00.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:00.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:00.034276+0000) 2024-03-06T13:32:01.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:01.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:01.034525+0000) 2024-03-06T13:32:02.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:02.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:02.034806+0000) 2024-03-06T13:32:03.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:03.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:03.035063+0000) 2024-03-06T13:32:04.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:04.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:04.035255+0000) 2024-03-06T13:32:05.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:05.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:05.035504+0000) 2024-03-06T13:32:06.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:06.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:06.035831+0000) 2024-03-06T13:32:07.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:07.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:07.036087+0000) 2024-03-06T13:32:08.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:08.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:08.036281+0000) 2024-03-06T13:32:08.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:08.299 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:09.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:09.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:09.036551+0000) 2024-03-06T13:32:10.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:10.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:10.036800+0000) 2024-03-06T13:32:11.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:11.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:11.037052+0000) 2024-03-06T13:32:12.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:12.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:12.037292+0000) 2024-03-06T13:32:13.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:13.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:13.037528+0000) 2024-03-06T13:32:14.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:14.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:14.037783+0000) 2024-03-06T13:32:15.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:15.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:15.038034+0000) 2024-03-06T13:32:16.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:16.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:16.038242+0000) 2024-03-06T13:32:17.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:17.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:17.038506+0000) 2024-03-06T13:32:18.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:18.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:18.038771+0000) 2024-03-06T13:32:19.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:19.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:19.039004+0000) 2024-03-06T13:32:20.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:20.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:20.039266+0000) 2024-03-06T13:32:21.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:21.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:21.039528+0000) 2024-03-06T13:32:22.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:22.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:22.039787+0000) 2024-03-06T13:32:23.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:23.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:23.040021+0000) 2024-03-06T13:32:24.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:24.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:24.040274+0000) 2024-03-06T13:32:25.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:25.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:25.040536+0000) 2024-03-06T13:32:26.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:26.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:26.040787+0000) 2024-03-06T13:32:27.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:27.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:27.041048+0000) 2024-03-06T13:32:28.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:28.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:28.041281+0000) 2024-03-06T13:32:29.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:29.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:29.041536+0000) 2024-03-06T13:32:30.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:30.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:30.041792+0000) 2024-03-06T13:32:31.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:31.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:31.042036+0000) 2024-03-06T13:32:32.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:32.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:32.042232+0000) 2024-03-06T13:32:33.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:33.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:33.042491+0000) 2024-03-06T13:32:34.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:34.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:34.042742+0000) 2024-03-06T13:32:35.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:35.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:35.042968+0000) 2024-03-06T13:32:36.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:36.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:36.043233+0000) 2024-03-06T13:32:37.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:37.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:37.043439+0000) 2024-03-06T13:32:38.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:38.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:38.043753+0000) 2024-03-06T13:32:38.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:38.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:39.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:39.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:39.044021+0000) 2024-03-06T13:32:40.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:40.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:40.044291+0000) 2024-03-06T13:32:41.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:41.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:41.044572+0000) 2024-03-06T13:32:42.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:42.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:42.044813+0000) 2024-03-06T13:32:43.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:43.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:43.045070+0000) 2024-03-06T13:32:44.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:44.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:44.045316+0000) 2024-03-06T13:32:45.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:45.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:45.045563+0000) 2024-03-06T13:32:46.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:46.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:46.045817+0000) 2024-03-06T13:32:47.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:47.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:47.046069+0000) 2024-03-06T13:32:48.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:48.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:48.046265+0000) 2024-03-06T13:32:49.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:49.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:49.046537+0000) 2024-03-06T13:32:50.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:50.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:50.046817+0000) 2024-03-06T13:32:51.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:51.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:51.047084+0000) 2024-03-06T13:32:52.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:52.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:52.047354+0000) 2024-03-06T13:32:53.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:53.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:53.047586+0000) 2024-03-06T13:32:54.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:54.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:54.047902+0000) 2024-03-06T13:32:55.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:55.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:55.048139+0000) 2024-03-06T13:32:56.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:56.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:56.048354+0000) 2024-03-06T13:32:57.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:57.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:57.048617+0000) 2024-03-06T13:32:58.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:58.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:58.048915+0000) 2024-03-06T13:32:59.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:32:59.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:59.049169+0000) 2024-03-06T13:33:00.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:00.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:00.049437+0000) 2024-03-06T13:33:01.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:01.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:01.049734+0000) 2024-03-06T13:33:02.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:02.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:02.050000+0000) 2024-03-06T13:33:03.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:03.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:03.050271+0000) 2024-03-06T13:33:04.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:04.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:04.050536+0000) 2024-03-06T13:33:05.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:05.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:05.050800+0000) 2024-03-06T13:33:06.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:06.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:06.051050+0000) 2024-03-06T13:33:07.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:07.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:07.051319+0000) 2024-03-06T13:33:08.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:08.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:08.051573+0000) 2024-03-06T13:33:08.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:08.717 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:09.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:09.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:09.051822+0000) 2024-03-06T13:33:10.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:10.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:10.052079+0000) 2024-03-06T13:33:11.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:11.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:11.052332+0000) 2024-03-06T13:33:12.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:12.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:12.052582+0000) 2024-03-06T13:33:13.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:13.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:13.052842+0000) 2024-03-06T13:33:14.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:14.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:14.053112+0000) 2024-03-06T13:33:15.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:15.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:15.053371+0000) 2024-03-06T13:33:16.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:16.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:16.053663+0000) 2024-03-06T13:33:17.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:17.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:17.053869+0000) 2024-03-06T13:33:18.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:18.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:18.054124+0000) 2024-03-06T13:33:19.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:19.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:19.054380+0000) 2024-03-06T13:33:20.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:20.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:20.054676+0000) 2024-03-06T13:33:21.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:21.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:21.054937+0000) 2024-03-06T13:33:22.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:22.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:22.055185+0000) 2024-03-06T13:33:23.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:23.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:23.055380+0000) 2024-03-06T13:33:24.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:24.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:24.055683+0000) 2024-03-06T13:33:25.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:25.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:25.055944+0000) 2024-03-06T13:33:26.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:26.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:26.056208+0000) 2024-03-06T13:33:27.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:27.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:27.056451+0000) 2024-03-06T13:33:28.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:28.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:28.056736+0000) 2024-03-06T13:33:29.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:29.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:29.056964+0000) 2024-03-06T13:33:30.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:30.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:30.057221+0000) 2024-03-06T13:33:31.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:31.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:31.057468+0000) 2024-03-06T13:33:32.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:32.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:32.057757+0000) 2024-03-06T13:33:33.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:33.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:33.058013+0000) 2024-03-06T13:33:34.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:34.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:34.058272+0000) 2024-03-06T13:33:35.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:35.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:35.058531+0000) 2024-03-06T13:33:36.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:36.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:36.058792+0000) 2024-03-06T13:33:37.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:37.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:37.059043+0000) 2024-03-06T13:33:38.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:38.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:38.059302+0000) 2024-03-06T13:33:38.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:38.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:39.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:39.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:39.059583+0000) 2024-03-06T13:33:40.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:40.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:40.059843+0000) 2024-03-06T13:33:41.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:41.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:41.060099+0000) 2024-03-06T13:33:42.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:42.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:42.060367+0000) 2024-03-06T13:33:43.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:43.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:43.060671+0000) 2024-03-06T13:33:44.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:44.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:44.060908+0000) 2024-03-06T13:33:45.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:45.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:45.061161+0000) 2024-03-06T13:33:46.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:46.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:46.061428+0000) 2024-03-06T13:33:47.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:47.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:47.061743+0000) 2024-03-06T13:33:48.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:48.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:48.061990+0000) 2024-03-06T13:33:49.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:49.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:49.062264+0000) 2024-03-06T13:33:50.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:50.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:50.062531+0000) 2024-03-06T13:33:51.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:51.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:51.062764+0000) 2024-03-06T13:33:52.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:52.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:52.063013+0000) 2024-03-06T13:33:53.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:53.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:53.063309+0000) 2024-03-06T13:33:54.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:54.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:54.063558+0000) 2024-03-06T13:33:55.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:55.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:55.063762+0000) 2024-03-06T13:33:56.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:56.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:56.063965+0000) 2024-03-06T13:33:57.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:57.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:57.064165+0000) 2024-03-06T13:33:58.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:58.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:58.064385+0000) 2024-03-06T13:33:59.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:33:59.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:59.064688+0000) 2024-03-06T13:34:00.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:00.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:00.064972+0000) 2024-03-06T13:34:01.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:01.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:01.065233+0000) 2024-03-06T13:34:02.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:02.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:02.065494+0000) 2024-03-06T13:34:03.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:03.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:03.065751+0000) 2024-03-06T13:34:04.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:04.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:04.066019+0000) 2024-03-06T13:34:05.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:05.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:05.066276+0000) 2024-03-06T13:34:06.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:06.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:06.066522+0000) 2024-03-06T13:34:07.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:07.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:07.066758+0000) 2024-03-06T13:34:08.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:08.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:08.067001+0000) 2024-03-06T13:34:09.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:09.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:09.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:09.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:09.067164+0000) 2024-03-06T13:34:10.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:10.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:10.067419+0000) 2024-03-06T13:34:11.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:11.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:11.067704+0000) 2024-03-06T13:34:12.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:12.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:12.067954+0000) 2024-03-06T13:34:13.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:13.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:13.068133+0000) 2024-03-06T13:34:14.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:14.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:14.068402+0000) 2024-03-06T13:34:15.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:15.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:15.068713+0000) 2024-03-06T13:34:16.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:16.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:16.068967+0000) 2024-03-06T13:34:17.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:17.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:17.069226+0000) 2024-03-06T13:34:18.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:18.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:18.069483+0000) 2024-03-06T13:34:19.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:19.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:19.069755+0000) 2024-03-06T13:34:20.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:20.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:20.070030+0000) 2024-03-06T13:34:21.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:21.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:21.070281+0000) 2024-03-06T13:34:22.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:22.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:22.070522+0000) 2024-03-06T13:34:23.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:23.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:23.070770+0000) 2024-03-06T13:34:24.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:24.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:24.071045+0000) 2024-03-06T13:34:25.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:25.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:25.071306+0000) 2024-03-06T13:34:26.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:26.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:26.071543+0000) 2024-03-06T13:34:27.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:27.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:27.071793+0000) 2024-03-06T13:34:28.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:28.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:28.072056+0000) 2024-03-06T13:34:29.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:29.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:29.072258+0000) 2024-03-06T13:34:30.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:30.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:30.072506+0000) 2024-03-06T13:34:31.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:31.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:31.072758+0000) 2024-03-06T13:34:32.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:32.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:32.073024+0000) 2024-03-06T13:34:33.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:33.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:33.073264+0000) 2024-03-06T13:34:34.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:34.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:34.073527+0000) 2024-03-06T13:34:35.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:35.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:35.073807+0000) 2024-03-06T13:34:36.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:36.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:36.074041+0000) 2024-03-06T13:34:37.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:37.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:37.074283+0000) 2024-03-06T13:34:38.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:38.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:38.074522+0000) 2024-03-06T13:34:39.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:39.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:39.074772+0000) 2024-03-06T13:34:39.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:39.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:40.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:40.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:40.075041+0000) 2024-03-06T13:34:41.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:41.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:41.075291+0000) 2024-03-06T13:34:42.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:42.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:42.075550+0000) 2024-03-06T13:34:43.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:43.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:43.075795+0000) 2024-03-06T13:34:44.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:44.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:44.076064+0000) 2024-03-06T13:34:45.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:45.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:45.076312+0000) 2024-03-06T13:34:46.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:46.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:46.076567+0000) 2024-03-06T13:34:47.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:47.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:47.076827+0000) 2024-03-06T13:34:48.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:48.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:48.077066+0000) 2024-03-06T13:34:49.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:49.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:49.077339+0000) 2024-03-06T13:34:50.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:50.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:50.077602+0000) 2024-03-06T13:34:51.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:51.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:51.077894+0000) 2024-03-06T13:34:52.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:52.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:52.078152+0000) 2024-03-06T13:34:53.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:53.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:53.078405+0000) 2024-03-06T13:34:54.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:54.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:54.078670+0000) 2024-03-06T13:34:55.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:55.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:55.078907+0000) 2024-03-06T13:34:56.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:56.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:56.079161+0000) 2024-03-06T13:34:57.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:57.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:57.079424+0000) 2024-03-06T13:34:58.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:58.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:58.079738+0000) 2024-03-06T13:34:59.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:34:59.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:59.079949+0000) 2024-03-06T13:35:00.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:00.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:00.080218+0000) 2024-03-06T13:35:01.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:01.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:01.080471+0000) 2024-03-06T13:35:02.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:02.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:02.080758+0000) 2024-03-06T13:35:03.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:03.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:03.081010+0000) 2024-03-06T13:35:04.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:04.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:04.081247+0000) 2024-03-06T13:35:05.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:05.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:05.081513+0000) 2024-03-06T13:35:06.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:06.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:06.081779+0000) 2024-03-06T13:35:07.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:07.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:07.082019+0000) 2024-03-06T13:35:08.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:08.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:08.082266+0000) 2024-03-06T13:35:09.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:09.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:09.082505+0000) 2024-03-06T13:35:09.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:09.310 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:10.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:10.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:10.082759+0000) 2024-03-06T13:35:11.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:11.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:11.083002+0000) 2024-03-06T13:35:12.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:12.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:12.083199+0000) 2024-03-06T13:35:13.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:13.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:13.083429+0000) 2024-03-06T13:35:14.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:14.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:14.083731+0000) 2024-03-06T13:35:15.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:15.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:15.083985+0000) 2024-03-06T13:35:16.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:16.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:16.084244+0000) 2024-03-06T13:35:17.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:17.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:17.084505+0000) 2024-03-06T13:35:18.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:18.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:18.084766+0000) 2024-03-06T13:35:19.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:19.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:19.085020+0000) 2024-03-06T13:35:20.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:20.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:20.085292+0000) 2024-03-06T13:35:21.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:21.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:21.085553+0000) 2024-03-06T13:35:22.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:22.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:22.085841+0000) 2024-03-06T13:35:23.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:23.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:23.086131+0000) 2024-03-06T13:35:24.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:24.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:24.086384+0000) 2024-03-06T13:35:25.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:25.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:25.086621+0000) 2024-03-06T13:35:26.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:26.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:26.086865+0000) 2024-03-06T13:35:27.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:27.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:27.087062+0000) 2024-03-06T13:35:28.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:28.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:28.087326+0000) 2024-03-06T13:35:29.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:29.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:29.087607+0000) 2024-03-06T13:35:30.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:30.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:30.087919+0000) 2024-03-06T13:35:31.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:31.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:31.088190+0000) 2024-03-06T13:35:32.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:32.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:32.088442+0000) 2024-03-06T13:35:33.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:33.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:33.088721+0000) 2024-03-06T13:35:34.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:34.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:34.088976+0000) 2024-03-06T13:35:35.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:35.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:35.089230+0000) 2024-03-06T13:35:36.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:36.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:36.089473+0000) 2024-03-06T13:35:37.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:37.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:37.089706+0000) 2024-03-06T13:35:38.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:38.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:38.089951+0000) 2024-03-06T13:35:39.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:39.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:39.090187+0000) 2024-03-06T13:35:39.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:39.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:40.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:40.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:40.090459+0000) 2024-03-06T13:35:41.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:41.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:41.090724+0000) 2024-03-06T13:35:42.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:42.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:42.090984+0000) 2024-03-06T13:35:43.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:43.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:43.091219+0000) 2024-03-06T13:35:44.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:44.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:44.091478+0000) 2024-03-06T13:35:45.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:45.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:45.091749+0000) 2024-03-06T13:35:46.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:46.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:46.092008+0000) 2024-03-06T13:35:47.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:47.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:47.092272+0000) 2024-03-06T13:35:48.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:48.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:48.092529+0000) 2024-03-06T13:35:49.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:49.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:49.092779+0000) 2024-03-06T13:35:50.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:50.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:50.093061+0000) 2024-03-06T13:35:51.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:51.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:51.093318+0000) 2024-03-06T13:35:52.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:52.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:52.093581+0000) 2024-03-06T13:35:53.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:53.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:53.093830+0000) 2024-03-06T13:35:54.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:54.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:54.094081+0000) 2024-03-06T13:35:55.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:55.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:55.094271+0000) 2024-03-06T13:35:56.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:56.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:56.094504+0000) 2024-03-06T13:35:57.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:57.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:57.094740+0000) 2024-03-06T13:35:58.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:58.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:58.094983+0000) 2024-03-06T13:35:59.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:35:59.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:59.095236+0000) 2024-03-06T13:36:00.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:00.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:00.095492+0000) 2024-03-06T13:36:01.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:01.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:01.095758+0000) 2024-03-06T13:36:02.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:02.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:02.095964+0000) 2024-03-06T13:36:03.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:03.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:03.096220+0000) 2024-03-06T13:36:04.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:04.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:04.096451+0000) 2024-03-06T13:36:05.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:05.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:05.096741+0000) 2024-03-06T13:36:06.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:06.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:06.096984+0000) 2024-03-06T13:36:07.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:07.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:07.097259+0000) 2024-03-06T13:36:08.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:08.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:08.097528+0000) 2024-03-06T13:36:09.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:09.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:09.097806+0000) 2024-03-06T13:36:09.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:09.619 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:10.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:10.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:10.098102+0000) 2024-03-06T13:36:11.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:11.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:11.098444+0000) 2024-03-06T13:36:12.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:12.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:12.098772+0000) 2024-03-06T13:36:13.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:13.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:13.099044+0000) 2024-03-06T13:36:14.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:14.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:14.099302+0000) 2024-03-06T13:36:15.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:15.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:15.099578+0000) 2024-03-06T13:36:16.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:16.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:16.099864+0000) 2024-03-06T13:36:17.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:17.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:17.100133+0000) 2024-03-06T13:36:18.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:18.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:18.100414+0000) 2024-03-06T13:36:19.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:19.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:19.100702+0000) 2024-03-06T13:36:20.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:20.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:20.100986+0000) 2024-03-06T13:36:21.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:21.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:21.101274+0000) 2024-03-06T13:36:22.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:22.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:22.101499+0000) 2024-03-06T13:36:23.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:23.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:23.101787+0000) 2024-03-06T13:36:24.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:24.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:24.102064+0000) 2024-03-06T13:36:25.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:25.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:25.102264+0000) 2024-03-06T13:36:26.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:26.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:26.102534+0000) 2024-03-06T13:36:27.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:27.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:27.102813+0000) 2024-03-06T13:36:28.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:28.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:28.103092+0000) 2024-03-06T13:36:29.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:29.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:29.103374+0000) 2024-03-06T13:36:30.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:30.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:30.103746+0000) 2024-03-06T13:36:31.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:31.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:31.103982+0000) 2024-03-06T13:36:32.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:32.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:32.104232+0000) 2024-03-06T13:36:33.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:33.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:33.104482+0000) 2024-03-06T13:36:34.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:34.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:34.104765+0000) 2024-03-06T13:36:35.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:35.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:35.105022+0000) 2024-03-06T13:36:36.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:36.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:36.105276+0000) 2024-03-06T13:36:37.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:37.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:37.105523+0000) 2024-03-06T13:36:38.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:38.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:38.105781+0000) 2024-03-06T13:36:39.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:39.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:39.106037+0000) 2024-03-06T13:36:39.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:39.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:40.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:40.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:40.106304+0000) 2024-03-06T13:36:41.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:41.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:41.106549+0000) 2024-03-06T13:36:42.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:42.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:42.106796+0000) 2024-03-06T13:36:43.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:43.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:43.107042+0000) 2024-03-06T13:36:44.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:44.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:44.107300+0000) 2024-03-06T13:36:45.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:45.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:45.107540+0000) 2024-03-06T13:36:46.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:46.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:46.107793+0000) 2024-03-06T13:36:47.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:47.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:47.108048+0000) 2024-03-06T13:36:48.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:48.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:48.108263+0000) 2024-03-06T13:36:49.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:49.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:49.108525+0000) 2024-03-06T13:36:50.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:50.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:50.108815+0000) 2024-03-06T13:36:51.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:51.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:51.109065+0000) 2024-03-06T13:36:52.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:52.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:52.109322+0000) 2024-03-06T13:36:53.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:53.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:53.109600+0000) 2024-03-06T13:36:54.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:54.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:54.109934+0000) 2024-03-06T13:36:55.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:55.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:55.110216+0000) 2024-03-06T13:36:56.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:56.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:56.110475+0000) 2024-03-06T13:36:57.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:57.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:57.110751+0000) 2024-03-06T13:36:58.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:58.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:58.111008+0000) 2024-03-06T13:36:59.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:36:59.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:59.111277+0000) 2024-03-06T13:37:00.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:00.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:00.111551+0000) 2024-03-06T13:37:01.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:01.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:01.111800+0000) 2024-03-06T13:37:02.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:02.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:02.112075+0000) 2024-03-06T13:37:03.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:03.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:03.112334+0000) 2024-03-06T13:37:04.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:04.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:04.112592+0000) 2024-03-06T13:37:05.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:05.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:05.112904+0000) 2024-03-06T13:37:06.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:06.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:06.113165+0000) 2024-03-06T13:37:07.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:07.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:07.113430+0000) 2024-03-06T13:37:08.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:08.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:08.113748+0000) 2024-03-06T13:37:09.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:09.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:09.113997+0000) 2024-03-06T13:37:09.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:09.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:10.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:10.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:10.114313+0000) 2024-03-06T13:37:11.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:11.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:11.114585+0000) 2024-03-06T13:37:12.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:12.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:12.114882+0000) 2024-03-06T13:37:13.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:13.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:13.115157+0000) 2024-03-06T13:37:14.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:14.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:14.115432+0000) 2024-03-06T13:37:15.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:15.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:15.115748+0000) 2024-03-06T13:37:16.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:16.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:16.116011+0000) 2024-03-06T13:37:17.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:17.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:17.116264+0000) 2024-03-06T13:37:18.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:18.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:18.116550+0000) 2024-03-06T13:37:19.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:19.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:19.116804+0000) 2024-03-06T13:37:20.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:20.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:20.117073+0000) 2024-03-06T13:37:21.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:21.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:21.117330+0000) 2024-03-06T13:37:22.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:22.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:22.117592+0000) 2024-03-06T13:37:23.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:23.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:23.117843+0000) 2024-03-06T13:37:24.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:24.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:24.118086+0000) 2024-03-06T13:37:25.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:25.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:25.118349+0000) 2024-03-06T13:37:26.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:26.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:26.118616+0000) 2024-03-06T13:37:27.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:27.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:27.118919+0000) 2024-03-06T13:37:28.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:28.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:28.119160+0000) 2024-03-06T13:37:29.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:29.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:29.119405+0000) 2024-03-06T13:37:30.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:30.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:30.119709+0000) 2024-03-06T13:37:31.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:31.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:31.119977+0000) 2024-03-06T13:37:32.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:32.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:32.120246+0000) 2024-03-06T13:37:33.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:33.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:33.120472+0000) 2024-03-06T13:37:34.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:34.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:34.120730+0000) 2024-03-06T13:37:35.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:35.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:35.120967+0000) 2024-03-06T13:37:36.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:36.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:36.121211+0000) 2024-03-06T13:37:37.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:37.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:37.121484+0000) 2024-03-06T13:37:38.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:38.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:38.121742+0000) 2024-03-06T13:37:39.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:39.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:39.121993+0000) 2024-03-06T13:37:40.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:40.068 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:40.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:40.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:40.122236+0000) 2024-03-06T13:37:41.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:41.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:41.122490+0000) 2024-03-06T13:37:42.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:42.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:42.122800+0000) 2024-03-06T13:37:43.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:43.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:43.123052+0000) 2024-03-06T13:37:44.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:44.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:44.123292+0000) 2024-03-06T13:37:45.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:45.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:45.123556+0000) 2024-03-06T13:37:46.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:46.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:46.123812+0000) 2024-03-06T13:37:47.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:47.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:47.124011+0000) 2024-03-06T13:37:48.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:48.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:48.124247+0000) 2024-03-06T13:37:49.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:49.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:49.124498+0000) 2024-03-06T13:37:50.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:50.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:50.124780+0000) 2024-03-06T13:37:51.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:51.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:51.125038+0000) 2024-03-06T13:37:52.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:52.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:52.125305+0000) 2024-03-06T13:37:53.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:53.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:53.125550+0000) 2024-03-06T13:37:54.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:54.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:54.125800+0000) 2024-03-06T13:37:55.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:55.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:55.126069+0000) 2024-03-06T13:37:56.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:56.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:56.126310+0000) 2024-03-06T13:37:57.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:57.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:57.126555+0000) 2024-03-06T13:37:58.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:58.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:58.126849+0000) 2024-03-06T13:37:59.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:37:59.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:59.127090+0000) 2024-03-06T13:38:00.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:00.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:00.127369+0000) 2024-03-06T13:38:01.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:01.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:01.127664+0000) 2024-03-06T13:38:02.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:02.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:02.127867+0000) 2024-03-06T13:38:03.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:03.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:03.128095+0000) 2024-03-06T13:38:04.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:04.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:04.128346+0000) 2024-03-06T13:38:05.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:05.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:05.128615+0000) 2024-03-06T13:38:06.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:06.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:06.128912+0000) 2024-03-06T13:38:07.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:07.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:07.129113+0000) 2024-03-06T13:38:08.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:08.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:08.129371+0000) 2024-03-06T13:38:09.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:09.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:09.129614+0000) 2024-03-06T13:38:10.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:10.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:10.129919+0000) 2024-03-06T13:38:10.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:10.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:11.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:11.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:11.130190+0000) 2024-03-06T13:38:12.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:12.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:12.130453+0000) 2024-03-06T13:38:13.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:13.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:13.130738+0000) 2024-03-06T13:38:14.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:14.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:14.130996+0000) 2024-03-06T13:38:15.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:15.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:15.131260+0000) 2024-03-06T13:38:16.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:16.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:16.131489+0000) 2024-03-06T13:38:17.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:17.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:17.131748+0000) 2024-03-06T13:38:18.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:18.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:18.131987+0000) 2024-03-06T13:38:19.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:19.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:19.132249+0000) 2024-03-06T13:38:20.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:20.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:20.132525+0000) 2024-03-06T13:38:21.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:21.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:21.132759+0000) 2024-03-06T13:38:22.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:22.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:22.133031+0000) 2024-03-06T13:38:23.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:23.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:23.133283+0000) 2024-03-06T13:38:24.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:24.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:24.133543+0000) 2024-03-06T13:38:25.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:25.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:25.133817+0000) 2024-03-06T13:38:26.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:26.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:26.134058+0000) 2024-03-06T13:38:27.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:27.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:27.134301+0000) 2024-03-06T13:38:28.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:28.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:28.134547+0000) 2024-03-06T13:38:29.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:29.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:29.134789+0000) 2024-03-06T13:38:30.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:30.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:30.135070+0000) 2024-03-06T13:38:31.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:31.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:31.135328+0000) 2024-03-06T13:38:32.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:32.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:32.135580+0000) 2024-03-06T13:38:33.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:33.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:33.135825+0000) 2024-03-06T13:38:34.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:34.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:34.136083+0000) 2024-03-06T13:38:35.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:35.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:35.136341+0000) 2024-03-06T13:38:36.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:36.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:36.136598+0000) 2024-03-06T13:38:37.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:37.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:37.136879+0000) 2024-03-06T13:38:38.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:38.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:38.137147+0000) 2024-03-06T13:38:39.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:39.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:39.137388+0000) 2024-03-06T13:38:40.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:40.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:40.137703+0000) 2024-03-06T13:38:40.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:40.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:41.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:41.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:41.137951+0000) 2024-03-06T13:38:42.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:42.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:42.138213+0000) 2024-03-06T13:38:43.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:43.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:43.138457+0000) 2024-03-06T13:38:44.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:44.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:44.138737+0000) 2024-03-06T13:38:45.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:45.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:45.138976+0000) 2024-03-06T13:38:46.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:46.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:46.139232+0000) 2024-03-06T13:38:47.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:47.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:47.139491+0000) 2024-03-06T13:38:48.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:48.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:48.139750+0000) 2024-03-06T13:38:49.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:49.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:49.139955+0000) 2024-03-06T13:38:50.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:50.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:50.140272+0000) 2024-03-06T13:38:51.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:51.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:51.140516+0000) 2024-03-06T13:38:52.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:52.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:52.140770+0000) 2024-03-06T13:38:53.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:53.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:53.141028+0000) 2024-03-06T13:38:54.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:54.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:54.141266+0000) 2024-03-06T13:38:55.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:55.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:55.141516+0000) 2024-03-06T13:38:56.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:56.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:56.141758+0000) 2024-03-06T13:38:57.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:57.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:57.142020+0000) 2024-03-06T13:38:58.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:58.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:58.142210+0000) 2024-03-06T13:38:59.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:38:59.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:59.142464+0000) 2024-03-06T13:39:00.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:00.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:00.142771+0000) 2024-03-06T13:39:01.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:01.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:01.143012+0000) 2024-03-06T13:39:02.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:02.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:02.143203+0000) 2024-03-06T13:39:03.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:03.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:03.143444+0000) 2024-03-06T13:39:04.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:04.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:04.143736+0000) 2024-03-06T13:39:05.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:05.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:05.143991+0000) 2024-03-06T13:39:06.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:06.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:06.144253+0000) 2024-03-06T13:39:07.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:07.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:07.144497+0000) 2024-03-06T13:39:08.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:08.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:08.144756+0000) 2024-03-06T13:39:09.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:09.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:09.145027+0000) 2024-03-06T13:39:10.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:10.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:10.145320+0000) 2024-03-06T13:39:10.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:10.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:11.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:11.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:11.145587+0000) 2024-03-06T13:39:12.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:12.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:12.145854+0000) 2024-03-06T13:39:13.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:13.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:13.146119+0000) 2024-03-06T13:39:14.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:14.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:14.146374+0000) 2024-03-06T13:39:15.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:15.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:15.146665+0000) 2024-03-06T13:39:16.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:16.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:16.146909+0000) 2024-03-06T13:39:17.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:17.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:17.147105+0000) 2024-03-06T13:39:18.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:18.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:18.147408+0000) 2024-03-06T13:39:19.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:19.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:19.147705+0000) 2024-03-06T13:39:20.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:20.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:20.147972+0000) 2024-03-06T13:39:21.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:21.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:21.148292+0000) 2024-03-06T13:39:22.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:22.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:22.148601+0000) 2024-03-06T13:39:23.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:23.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:23.148945+0000) 2024-03-06T13:39:24.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:24.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:24.149162+0000) 2024-03-06T13:39:25.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:25.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:25.149420+0000) 2024-03-06T13:39:26.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:26.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:26.149720+0000) 2024-03-06T13:39:27.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:27.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:27.149979+0000) 2024-03-06T13:39:28.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:28.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:28.150240+0000) 2024-03-06T13:39:29.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:29.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:29.150503+0000) 2024-03-06T13:39:30.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:30.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:30.150816+0000) 2024-03-06T13:39:31.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:31.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:31.151017+0000) 2024-03-06T13:39:32.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:32.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:32.151253+0000) 2024-03-06T13:39:33.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:33.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:33.151504+0000) 2024-03-06T13:39:34.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:34.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:34.151817+0000) 2024-03-06T13:39:35.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:35.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:35.152006+0000) 2024-03-06T13:39:36.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:36.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:36.152192+0000) 2024-03-06T13:39:37.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:37.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:37.152424+0000) 2024-03-06T13:39:38.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:38.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:38.152699+0000) 2024-03-06T13:39:39.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:39.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:39.152936+0000) 2024-03-06T13:39:40.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:40.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:40.153198+0000) 2024-03-06T13:39:40.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:40.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:41.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:41.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:41.153445+0000) 2024-03-06T13:39:42.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:42.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:42.153747+0000) 2024-03-06T13:39:43.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:43.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:43.154010+0000) 2024-03-06T13:39:44.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:44.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:44.154265+0000) 2024-03-06T13:39:45.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:45.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:45.154566+0000) 2024-03-06T13:39:46.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:46.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:46.154826+0000) 2024-03-06T13:39:47.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:47.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:47.155016+0000) 2024-03-06T13:39:48.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:48.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:48.155268+0000) 2024-03-06T13:39:49.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:49.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:49.155519+0000) 2024-03-06T13:39:50.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:50.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:50.155768+0000) 2024-03-06T13:39:51.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:51.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:51.156020+0000) 2024-03-06T13:39:52.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:52.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:52.156261+0000) 2024-03-06T13:39:53.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:53.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:53.156507+0000) 2024-03-06T13:39:54.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:54.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:54.156756+0000) 2024-03-06T13:39:55.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:55.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:55.156941+0000) 2024-03-06T13:39:56.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:56.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:56.157199+0000) 2024-03-06T13:39:57.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:57.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:57.157455+0000) 2024-03-06T13:39:58.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:58.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:58.157780+0000) 2024-03-06T13:39:59.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:39:59.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:59.157971+0000) 2024-03-06T13:40:00.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:00.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:00.158239+0000) 2024-03-06T13:40:01.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:01.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:01.158500+0000) 2024-03-06T13:40:02.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:02.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:02.158754+0000) 2024-03-06T13:40:03.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:03.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:03.158988+0000) 2024-03-06T13:40:04.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:04.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:04.159237+0000) 2024-03-06T13:40:05.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:05.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:05.159470+0000) 2024-03-06T13:40:06.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:06.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:06.159766+0000) 2024-03-06T13:40:07.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:07.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:07.160020+0000) 2024-03-06T13:40:08.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:08.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:08.160220+0000) 2024-03-06T13:40:09.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:09.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:09.160480+0000) 2024-03-06T13:40:10.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:10.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:10.160787+0000) 2024-03-06T13:40:10.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:10.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:11.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:11.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:11.161091+0000) 2024-03-06T13:40:12.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:12.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:12.161387+0000) 2024-03-06T13:40:13.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:13.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:13.161745+0000) 2024-03-06T13:40:14.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:14.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:14.161995+0000) 2024-03-06T13:40:15.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:15.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:15.162252+0000) 2024-03-06T13:40:16.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:16.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:16.162486+0000) 2024-03-06T13:40:17.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:17.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:17.162768+0000) 2024-03-06T13:40:18.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:18.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:18.163026+0000) 2024-03-06T13:40:19.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:19.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:19.163279+0000) 2024-03-06T13:40:20.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:20.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:20.163541+0000) 2024-03-06T13:40:21.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:21.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:21.163773+0000) 2024-03-06T13:40:22.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:22.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:22.164068+0000) 2024-03-06T13:40:23.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:23.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:23.164317+0000) 2024-03-06T13:40:24.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:24.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:24.164569+0000) 2024-03-06T13:40:25.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:25.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:25.164829+0000) 2024-03-06T13:40:26.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:26.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:26.165087+0000) 2024-03-06T13:40:27.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:27.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:27.165341+0000) 2024-03-06T13:40:28.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:28.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:28.165580+0000) 2024-03-06T13:40:29.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:29.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:29.165832+0000) 2024-03-06T13:40:30.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:30.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:30.166100+0000) 2024-03-06T13:40:31.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:31.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:31.166354+0000) 2024-03-06T13:40:32.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:32.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:32.166598+0000) 2024-03-06T13:40:33.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:33.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:33.166862+0000) 2024-03-06T13:40:34.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:34.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:34.167095+0000) 2024-03-06T13:40:35.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:35.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:35.167330+0000) 2024-03-06T13:40:36.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:36.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:36.167586+0000) 2024-03-06T13:40:37.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:37.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:37.167844+0000) 2024-03-06T13:40:38.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:38.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:38.168082+0000) 2024-03-06T13:40:39.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:39.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:39.168318+0000) 2024-03-06T13:40:40.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:40.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:40.168591+0000) 2024-03-06T13:40:40.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:40.953 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:41.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:41.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:41.168863+0000) 2024-03-06T13:40:42.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:42.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:42.169118+0000) 2024-03-06T13:40:43.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:43.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:43.169388+0000) 2024-03-06T13:40:44.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:44.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:44.169662+0000) 2024-03-06T13:40:45.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:45.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:45.169917+0000) 2024-03-06T13:40:46.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:46.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:46.170165+0000) 2024-03-06T13:40:47.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:47.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:47.170395+0000) 2024-03-06T13:40:48.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:48.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:48.170702+0000) 2024-03-06T13:40:49.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:49.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:49.170972+0000) 2024-03-06T13:40:50.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:50.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:50.171223+0000) 2024-03-06T13:40:51.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:51.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:51.171462+0000) 2024-03-06T13:40:52.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:52.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:52.171685+0000) 2024-03-06T13:40:53.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:53.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:53.171889+0000) 2024-03-06T13:40:54.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:54.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:54.172140+0000) 2024-03-06T13:40:55.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:55.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:55.172417+0000) 2024-03-06T13:40:56.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:56.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:56.172777+0000) 2024-03-06T13:40:57.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:57.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:57.173071+0000) 2024-03-06T13:40:58.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:58.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:58.173383+0000) 2024-03-06T13:40:59.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:40:59.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:59.173692+0000) 2024-03-06T13:41:00.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:00.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:00.173955+0000) 2024-03-06T13:41:01.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:01.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:01.174216+0000) 2024-03-06T13:41:02.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:02.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:02.174454+0000) 2024-03-06T13:41:03.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:03.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:03.174761+0000) 2024-03-06T13:41:04.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:04.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:04.175022+0000) 2024-03-06T13:41:05.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:05.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:05.175217+0000) 2024-03-06T13:41:06.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:06.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:06.175476+0000) 2024-03-06T13:41:07.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:07.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:07.175740+0000) 2024-03-06T13:41:08.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:08.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:08.176001+0000) 2024-03-06T13:41:09.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:09.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:09.176246+0000) 2024-03-06T13:41:10.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:10.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:10.176495+0000) 2024-03-06T13:41:11.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:11.095 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:11.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:11.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:11.176714+0000) 2024-03-06T13:41:12.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:12.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:12.176969+0000) 2024-03-06T13:41:13.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:13.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:13.177255+0000) 2024-03-06T13:41:14.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:14.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:14.177528+0000) 2024-03-06T13:41:15.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:15.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:15.177787+0000) 2024-03-06T13:41:16.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:16.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:16.178007+0000) 2024-03-06T13:41:17.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:17.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:17.178277+0000) 2024-03-06T13:41:18.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:18.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:18.178557+0000) 2024-03-06T13:41:19.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:19.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:19.178755+0000) 2024-03-06T13:41:20.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:20.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:20.179031+0000) 2024-03-06T13:41:21.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:21.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:21.179369+0000) 2024-03-06T13:41:22.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:22.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:22.179678+0000) 2024-03-06T13:41:23.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:23.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:23.179968+0000) 2024-03-06T13:41:24.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:24.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:24.180221+0000) 2024-03-06T13:41:25.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:25.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:25.180486+0000) 2024-03-06T13:41:26.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:26.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:26.180765+0000) 2024-03-06T13:41:27.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:27.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:27.181036+0000) 2024-03-06T13:41:28.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:28.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:28.181305+0000) 2024-03-06T13:41:29.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:29.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:29.181585+0000) 2024-03-06T13:41:30.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:30.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:30.181899+0000) 2024-03-06T13:41:31.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:31.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:31.182181+0000) 2024-03-06T13:41:32.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:32.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:32.182452+0000) 2024-03-06T13:41:33.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:33.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:33.182736+0000) 2024-03-06T13:41:34.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:34.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:34.183014+0000) 2024-03-06T13:41:35.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:35.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:35.183291+0000) 2024-03-06T13:41:36.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:36.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:36.183573+0000) 2024-03-06T13:41:37.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:37.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:37.183843+0000) 2024-03-06T13:41:38.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:38.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:38.184141+0000) 2024-03-06T13:41:39.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:39.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:39.184416+0000) 2024-03-06T13:41:40.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:40.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:40.184733+0000) 2024-03-06T13:41:41.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:41.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:41.184950+0000) 2024-03-06T13:41:41.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:41.244 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:42.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:42.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:42.185220+0000) 2024-03-06T13:41:43.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:43.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:43.185502+0000) 2024-03-06T13:41:44.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:44.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:44.185758+0000) 2024-03-06T13:41:45.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:45.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:45.186016+0000) 2024-03-06T13:41:46.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:46.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:46.186274+0000) 2024-03-06T13:41:47.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:47.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:47.186517+0000) 2024-03-06T13:41:48.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:48.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:48.186779+0000) 2024-03-06T13:41:49.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:49.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:49.187042+0000) 2024-03-06T13:41:50.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:50.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:50.187249+0000) 2024-03-06T13:41:51.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:51.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:51.187525+0000) 2024-03-06T13:41:52.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:52.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:52.187787+0000) 2024-03-06T13:41:53.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:53.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:53.188073+0000) 2024-03-06T13:41:54.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:54.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:54.188351+0000) 2024-03-06T13:41:55.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:55.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:55.188623+0000) 2024-03-06T13:41:56.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:56.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:56.188947+0000) 2024-03-06T13:41:57.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:57.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:57.189147+0000) 2024-03-06T13:41:58.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:58.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:58.189423+0000) 2024-03-06T13:41:59.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:41:59.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:59.189730+0000) 2024-03-06T13:42:00.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:00.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:00.189997+0000) 2024-03-06T13:42:01.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:01.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:01.190183+0000) 2024-03-06T13:42:02.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:02.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:02.190450+0000) 2024-03-06T13:42:03.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:03.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:03.190735+0000) 2024-03-06T13:42:04.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:04.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:04.190936+0000) 2024-03-06T13:42:05.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:05.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:05.191208+0000) 2024-03-06T13:42:06.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:06.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:06.191464+0000) 2024-03-06T13:42:07.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:07.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:07.191734+0000) 2024-03-06T13:42:08.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:08.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:08.192019+0000) 2024-03-06T13:42:09.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:09.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:09.192354+0000) 2024-03-06T13:42:10.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:10.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:10.192662+0000) 2024-03-06T13:42:11.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:11.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:11.192917+0000) 2024-03-06T13:42:11.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:11.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:12.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:12.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:12.193173+0000) 2024-03-06T13:42:13.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:13.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:13.193423+0000) 2024-03-06T13:42:14.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:14.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:14.193712+0000) 2024-03-06T13:42:15.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:15.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:15.193902+0000) 2024-03-06T13:42:16.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:16.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:16.194153+0000) 2024-03-06T13:42:17.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:17.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:17.194404+0000) 2024-03-06T13:42:18.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:18.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:18.194706+0000) 2024-03-06T13:42:19.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:19.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:19.194961+0000) 2024-03-06T13:42:20.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:20.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:20.195242+0000) 2024-03-06T13:42:21.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:21.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:21.195502+0000) 2024-03-06T13:42:22.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:22.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:22.195745+0000) 2024-03-06T13:42:23.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:23.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:23.196024+0000) 2024-03-06T13:42:24.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:24.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:24.196298+0000) 2024-03-06T13:42:25.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:25.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:25.196552+0000) 2024-03-06T13:42:26.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:26.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:26.196817+0000) 2024-03-06T13:42:27.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:27.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:27.197085+0000) 2024-03-06T13:42:28.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:28.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:28.197343+0000) 2024-03-06T13:42:29.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:29.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:29.197597+0000) 2024-03-06T13:42:30.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:30.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:30.197921+0000) 2024-03-06T13:42:31.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:31.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:31.198206+0000) 2024-03-06T13:42:32.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:32.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:32.198413+0000) 2024-03-06T13:42:33.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:33.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:33.198669+0000) 2024-03-06T13:42:34.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:34.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:34.198924+0000) 2024-03-06T13:42:35.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:35.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:35.199178+0000) 2024-03-06T13:42:36.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:36.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:36.199462+0000) 2024-03-06T13:42:37.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:37.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:37.199737+0000) 2024-03-06T13:42:38.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:38.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:38.200005+0000) 2024-03-06T13:42:39.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:39.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:39.200273+0000) 2024-03-06T13:42:40.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:40.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:40.200556+0000) 2024-03-06T13:42:41.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:41.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:41.200813+0000) 2024-03-06T13:42:41.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:41.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:42.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:42.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:42.201073+0000) 2024-03-06T13:42:43.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:43.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:43.201353+0000) 2024-03-06T13:42:44.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:44.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:44.201618+0000) 2024-03-06T13:42:45.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:45.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:45.201922+0000) 2024-03-06T13:42:46.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:46.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:46.202173+0000) 2024-03-06T13:42:47.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:47.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:47.202372+0000) 2024-03-06T13:42:48.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:48.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:48.202672+0000) 2024-03-06T13:42:49.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:49.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:49.202911+0000) 2024-03-06T13:42:50.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:50.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:50.203172+0000) 2024-03-06T13:42:51.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:51.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:51.203426+0000) 2024-03-06T13:42:52.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:52.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:52.203702+0000) 2024-03-06T13:42:53.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:53.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:53.203957+0000) 2024-03-06T13:42:54.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:54.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:54.204211+0000) 2024-03-06T13:42:55.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:55.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:55.204465+0000) 2024-03-06T13:42:56.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:56.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:56.204754+0000) 2024-03-06T13:42:57.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:57.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:57.205085+0000) 2024-03-06T13:42:58.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:58.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:58.205374+0000) 2024-03-06T13:42:59.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:42:59.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:59.205699+0000) 2024-03-06T13:43:00.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:00.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:00.205980+0000) 2024-03-06T13:43:01.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:01.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:01.206258+0000) 2024-03-06T13:43:02.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:02.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:02.206520+0000) 2024-03-06T13:43:03.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:03.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:03.206799+0000) 2024-03-06T13:43:04.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:04.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:04.207067+0000) 2024-03-06T13:43:05.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:05.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:05.207336+0000) 2024-03-06T13:43:06.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:06.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:06.207623+0000) 2024-03-06T13:43:07.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:07.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:07.207927+0000) 2024-03-06T13:43:08.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:08.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:08.208186+0000) 2024-03-06T13:43:09.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:09.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:09.208455+0000) 2024-03-06T13:43:10.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:10.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:10.208767+0000) 2024-03-06T13:43:11.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:11.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:11.209038+0000) 2024-03-06T13:43:11.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:11.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:12.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:12.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:12.209310+0000) 2024-03-06T13:43:13.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:13.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:13.209598+0000) 2024-03-06T13:43:14.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:14.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:14.209920+0000) 2024-03-06T13:43:15.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:15.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:15.210141+0000) 2024-03-06T13:43:16.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:16.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:16.210371+0000) 2024-03-06T13:43:17.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:17.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:17.210570+0000) 2024-03-06T13:43:18.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:18.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:18.210837+0000) 2024-03-06T13:43:19.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:19.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:19.211112+0000) 2024-03-06T13:43:20.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:20.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:20.211400+0000) 2024-03-06T13:43:21.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:21.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:21.211705+0000) 2024-03-06T13:43:22.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:22.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:22.211984+0000) 2024-03-06T13:43:23.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:23.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:23.212244+0000) 2024-03-06T13:43:24.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:24.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:24.212500+0000) 2024-03-06T13:43:25.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:25.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:25.212752+0000) 2024-03-06T13:43:26.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:26.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:26.213040+0000) 2024-03-06T13:43:27.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:27.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:27.213312+0000) 2024-03-06T13:43:28.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:28.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:28.213562+0000) 2024-03-06T13:43:29.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:29.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:29.213760+0000) 2024-03-06T13:43:30.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:30.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:30.214022+0000) 2024-03-06T13:43:31.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:31.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:31.214250+0000) 2024-03-06T13:43:32.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:32.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:32.214512+0000) 2024-03-06T13:43:33.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:33.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:33.214805+0000) 2024-03-06T13:43:34.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:34.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:34.215090+0000) 2024-03-06T13:43:35.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:35.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:35.215279+0000) 2024-03-06T13:43:36.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:36.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:36.215555+0000) 2024-03-06T13:43:37.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:37.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:37.215754+0000) 2024-03-06T13:43:38.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:38.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:38.215996+0000) 2024-03-06T13:43:39.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:39.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:39.216288+0000) 2024-03-06T13:43:40.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:40.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:40.216541+0000) 2024-03-06T13:43:41.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:41.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:41.216800+0000) 2024-03-06T13:43:41.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:41.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:42.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:42.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:42.217030+0000) 2024-03-06T13:43:43.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:43.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:43.217287+0000) 2024-03-06T13:43:44.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:44.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:44.217531+0000) 2024-03-06T13:43:45.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:45.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:45.217788+0000) 2024-03-06T13:43:46.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:46.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:46.218037+0000) 2024-03-06T13:43:47.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:47.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:47.218278+0000) 2024-03-06T13:43:48.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:48.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:48.218515+0000) 2024-03-06T13:43:49.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:49.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:49.218767+0000) 2024-03-06T13:43:50.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:50.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:50.219034+0000) 2024-03-06T13:43:51.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:51.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:51.219328+0000) 2024-03-06T13:43:52.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:52.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:52.219675+0000) 2024-03-06T13:43:53.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:53.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:53.219959+0000) 2024-03-06T13:43:54.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:54.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:54.220203+0000) 2024-03-06T13:43:55.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:55.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:55.220440+0000) 2024-03-06T13:43:56.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:56.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:56.220757+0000) 2024-03-06T13:43:57.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:57.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:57.221014+0000) 2024-03-06T13:43:58.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:58.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:58.221271+0000) 2024-03-06T13:43:59.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:43:59.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:59.221526+0000) 2024-03-06T13:44:00.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:00.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:00.221779+0000) 2024-03-06T13:44:01.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:01.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:01.222059+0000) 2024-03-06T13:44:02.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:02.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:02.222318+0000) 2024-03-06T13:44:03.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:03.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:03.222557+0000) 2024-03-06T13:44:04.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:04.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:04.222811+0000) 2024-03-06T13:44:05.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:05.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:05.222998+0000) 2024-03-06T13:44:06.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:06.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:06.223258+0000) 2024-03-06T13:44:07.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:07.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:07.223501+0000) 2024-03-06T13:44:08.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:08.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:08.223760+0000) 2024-03-06T13:44:09.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:09.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:09.224014+0000) 2024-03-06T13:44:10.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:10.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:10.224285+0000) 2024-03-06T13:44:11.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:11.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:11.224525+0000) 2024-03-06T13:44:11.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:12.000 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:12.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:12.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:12.224816+0000) 2024-03-06T13:44:13.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:13.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:13.225016+0000) 2024-03-06T13:44:14.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:14.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:14.225272+0000) 2024-03-06T13:44:15.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:15.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:15.225527+0000) 2024-03-06T13:44:16.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:16.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:16.225804+0000) 2024-03-06T13:44:17.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:17.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:17.226090+0000) 2024-03-06T13:44:18.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:18.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:18.226349+0000) 2024-03-06T13:44:19.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:19.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:19.226608+0000) 2024-03-06T13:44:20.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:20.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:20.226941+0000) 2024-03-06T13:44:21.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:21.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:21.227199+0000) 2024-03-06T13:44:22.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:22.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:22.227462+0000) 2024-03-06T13:44:23.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:23.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:23.227758+0000) 2024-03-06T13:44:24.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:24.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:24.227999+0000) 2024-03-06T13:44:25.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:25.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:25.228264+0000) 2024-03-06T13:44:26.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:26.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:26.228497+0000) 2024-03-06T13:44:27.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:27.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:27.228741+0000) 2024-03-06T13:44:28.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:28.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:28.228973+0000) 2024-03-06T13:44:29.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:29.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:29.229214+0000) 2024-03-06T13:44:30.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:30.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:30.229459+0000) 2024-03-06T13:44:31.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:31.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:31.229762+0000) 2024-03-06T13:44:32.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:32.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:32.230022+0000) 2024-03-06T13:44:33.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:33.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:33.230241+0000) 2024-03-06T13:44:34.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:34.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:34.230518+0000) 2024-03-06T13:44:35.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:35.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:35.230784+0000) 2024-03-06T13:44:36.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:36.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:36.231039+0000) 2024-03-06T13:44:37.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:37.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:37.231310+0000) 2024-03-06T13:44:38.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:38.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:38.231586+0000) 2024-03-06T13:44:39.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:39.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:39.231912+0000) 2024-03-06T13:44:40.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:40.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:40.232197+0000) 2024-03-06T13:44:41.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:41.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:41.232440+0000) 2024-03-06T13:44:42.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:42.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:42.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:42.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:42.232649+0000) 2024-03-06T13:44:43.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:43.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:43.232905+0000) 2024-03-06T13:44:44.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:44.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:44.233150+0000) 2024-03-06T13:44:45.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:45.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:45.233416+0000) 2024-03-06T13:44:46.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:46.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:46.233719+0000) 2024-03-06T13:44:47.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:47.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:47.233978+0000) 2024-03-06T13:44:48.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:48.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:48.234244+0000) 2024-03-06T13:44:49.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:49.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:49.234525+0000) 2024-03-06T13:44:50.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:50.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:50.234811+0000) 2024-03-06T13:44:51.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:51.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:51.235056+0000) 2024-03-06T13:44:52.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:52.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:52.235329+0000) 2024-03-06T13:44:53.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:53.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:53.235601+0000) 2024-03-06T13:44:54.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:54.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:54.235976+0000) 2024-03-06T13:44:55.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:55.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:55.236185+0000) 2024-03-06T13:44:56.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:56.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:56.236449+0000) 2024-03-06T13:44:57.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:57.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:57.236748+0000) 2024-03-06T13:44:58.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:58.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:58.236989+0000) 2024-03-06T13:44:59.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:44:59.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:59.237257+0000) 2024-03-06T13:45:00.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:00.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:00.237520+0000) 2024-03-06T13:45:01.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:01.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:01.237775+0000) 2024-03-06T13:45:02.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:02.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:02.237964+0000) 2024-03-06T13:45:03.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:03.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:03.238203+0000) 2024-03-06T13:45:04.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:04.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:04.238455+0000) 2024-03-06T13:45:05.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:05.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:05.238743+0000) 2024-03-06T13:45:06.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:06.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:06.238990+0000) 2024-03-06T13:45:07.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:07.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:07.239248+0000) 2024-03-06T13:45:08.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:08.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:08.239487+0000) 2024-03-06T13:45:09.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:09.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:09.239754+0000) 2024-03-06T13:45:10.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:10.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:10.240049+0000) 2024-03-06T13:45:11.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:11.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:11.240320+0000) 2024-03-06T13:45:12.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:12.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:12.240587+0000) 2024-03-06T13:45:12.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:12.295 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:13.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:13.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:13.240917+0000) 2024-03-06T13:45:14.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:14.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:14.241176+0000) 2024-03-06T13:45:15.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:15.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:15.241451+0000) 2024-03-06T13:45:16.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:16.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:16.241748+0000) 2024-03-06T13:45:17.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:17.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:17.242004+0000) 2024-03-06T13:45:18.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:18.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:18.242260+0000) 2024-03-06T13:45:19.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:19.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:19.242533+0000) 2024-03-06T13:45:20.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:20.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:20.242824+0000) 2024-03-06T13:45:21.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:21.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:21.243082+0000) 2024-03-06T13:45:22.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:22.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:22.243360+0000) 2024-03-06T13:45:23.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:23.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:23.243675+0000) 2024-03-06T13:45:24.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:24.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:24.243899+0000) 2024-03-06T13:45:25.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:25.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:25.244172+0000) 2024-03-06T13:45:26.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:26.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:26.244533+0000) 2024-03-06T13:45:27.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:27.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:27.244812+0000) 2024-03-06T13:45:28.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:28.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:28.245076+0000) 2024-03-06T13:45:29.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:29.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:29.245339+0000) 2024-03-06T13:45:30.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:30.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:30.245672+0000) 2024-03-06T13:45:31.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:31.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:31.245906+0000) 2024-03-06T13:45:32.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:32.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:32.246150+0000) 2024-03-06T13:45:33.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:33.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:33.246396+0000) 2024-03-06T13:45:34.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:34.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:34.246691+0000) 2024-03-06T13:45:35.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:35.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:35.246946+0000) 2024-03-06T13:45:36.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:36.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:36.247200+0000) 2024-03-06T13:45:37.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:37.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:37.247466+0000) 2024-03-06T13:45:38.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:38.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:38.247762+0000) 2024-03-06T13:45:39.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:39.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:39.247962+0000) 2024-03-06T13:45:40.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:40.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:40.248212+0000) 2024-03-06T13:45:41.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:41.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:41.248464+0000) 2024-03-06T13:45:42.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:42.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:42.248744+0000) 2024-03-06T13:45:42.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:42.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:43.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:43.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:43.248995+0000) 2024-03-06T13:45:44.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:44.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:44.249267+0000) 2024-03-06T13:45:45.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:45.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:45.249544+0000) 2024-03-06T13:45:46.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:46.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:46.249816+0000) 2024-03-06T13:45:47.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:47.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:47.250097+0000) 2024-03-06T13:45:48.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:48.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:48.250371+0000) 2024-03-06T13:45:49.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:49.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:49.250671+0000) 2024-03-06T13:45:50.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:50.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:50.250945+0000) 2024-03-06T13:45:51.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:51.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:51.251181+0000) 2024-03-06T13:45:52.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:52.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:52.251446+0000) 2024-03-06T13:45:53.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:53.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:53.251724+0000) 2024-03-06T13:45:54.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:54.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:54.251962+0000) 2024-03-06T13:45:55.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:55.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:55.252196+0000) 2024-03-06T13:45:56.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:56.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:56.252455+0000) 2024-03-06T13:45:57.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:57.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:57.252754+0000) 2024-03-06T13:45:58.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:58.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:58.252953+0000) 2024-03-06T13:45:59.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:45:59.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:59.253194+0000) 2024-03-06T13:46:00.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:00.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:00.253449+0000) 2024-03-06T13:46:01.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:01.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:01.253772+0000) 2024-03-06T13:46:02.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:02.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:02.254085+0000) 2024-03-06T13:46:03.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:03.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:03.254357+0000) 2024-03-06T13:46:04.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:04.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:04.254623+0000) 2024-03-06T13:46:05.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:05.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:05.254931+0000) 2024-03-06T13:46:06.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:06.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:06.255192+0000) 2024-03-06T13:46:07.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:07.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:07.255385+0000) 2024-03-06T13:46:08.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:08.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:08.255664+0000) 2024-03-06T13:46:09.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:09.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:09.255935+0000) 2024-03-06T13:46:10.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:10.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:10.256209+0000) 2024-03-06T13:46:11.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:11.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:11.256449+0000) 2024-03-06T13:46:12.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:12.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:12.256753+0000) 2024-03-06T13:46:12.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:12.598 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:13.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:13.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:13.257016+0000) 2024-03-06T13:46:14.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:14.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:14.257276+0000) 2024-03-06T13:46:15.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:15.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:15.257539+0000) 2024-03-06T13:46:16.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:16.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:16.257776+0000) 2024-03-06T13:46:17.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:17.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:17.258023+0000) 2024-03-06T13:46:18.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:18.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:18.258279+0000) 2024-03-06T13:46:19.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:19.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:19.258590+0000) 2024-03-06T13:46:20.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:20.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:20.258894+0000) 2024-03-06T13:46:21.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:21.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:21.259157+0000) 2024-03-06T13:46:22.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:22.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:22.259398+0000) 2024-03-06T13:46:23.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:23.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:23.259687+0000) 2024-03-06T13:46:24.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:24.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:24.259942+0000) 2024-03-06T13:46:25.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:25.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:25.260184+0000) 2024-03-06T13:46:26.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:26.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:26.260426+0000) 2024-03-06T13:46:27.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:27.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:27.260713+0000) 2024-03-06T13:46:28.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:28.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:28.260954+0000) 2024-03-06T13:46:29.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:29.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:29.261219+0000) 2024-03-06T13:46:30.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:30.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:30.261485+0000) 2024-03-06T13:46:31.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:31.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:31.261743+0000) 2024-03-06T13:46:32.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:32.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:32.262011+0000) 2024-03-06T13:46:33.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:33.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:33.262273+0000) 2024-03-06T13:46:34.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:34.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:34.262544+0000) 2024-03-06T13:46:35.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:35.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:35.262806+0000) 2024-03-06T13:46:36.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:36.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:36.263031+0000) 2024-03-06T13:46:37.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:37.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:37.263272+0000) 2024-03-06T13:46:38.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:38.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:38.263509+0000) 2024-03-06T13:46:39.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:39.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:39.263773+0000) 2024-03-06T13:46:40.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:40.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:40.264026+0000) 2024-03-06T13:46:41.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:41.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:41.264295+0000) 2024-03-06T13:46:42.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:42.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:42.264549+0000) 2024-03-06T13:46:42.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:42.752 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:43.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:43.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:43.264817+0000) 2024-03-06T13:46:44.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:44.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:44.265059+0000) 2024-03-06T13:46:45.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:45.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:45.265318+0000) 2024-03-06T13:46:46.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:46.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:46.265579+0000) 2024-03-06T13:46:47.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:47.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:47.265821+0000) 2024-03-06T13:46:48.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:48.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:48.266062+0000) 2024-03-06T13:46:49.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:49.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:49.266334+0000) 2024-03-06T13:46:50.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:50.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:50.266581+0000) 2024-03-06T13:46:51.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:51.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:51.266826+0000) 2024-03-06T13:46:52.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:52.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:52.267092+0000) 2024-03-06T13:46:53.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:53.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:53.267348+0000) 2024-03-06T13:46:54.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:54.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:54.267613+0000) 2024-03-06T13:46:55.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:55.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:55.267896+0000) 2024-03-06T13:46:56.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:56.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:56.268140+0000) 2024-03-06T13:46:57.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:57.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:57.268393+0000) 2024-03-06T13:46:58.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:58.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:58.268667+0000) 2024-03-06T13:46:59.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:46:59.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:59.268935+0000) 2024-03-06T13:47:00.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:00.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:00.269199+0000) 2024-03-06T13:47:01.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:01.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:01.269440+0000) 2024-03-06T13:47:02.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:02.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:02.269749+0000) 2024-03-06T13:47:03.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:03.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:03.270007+0000) 2024-03-06T13:47:04.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:04.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:04.270272+0000) 2024-03-06T13:47:05.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:05.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:05.270538+0000) 2024-03-06T13:47:06.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:06.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:06.270798+0000) 2024-03-06T13:47:07.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:07.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:07.271269+0000) 2024-03-06T13:47:08.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:08.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:08.271527+0000) 2024-03-06T13:47:09.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:09.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:09.271789+0000) 2024-03-06T13:47:10.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:10.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:10.272012+0000) 2024-03-06T13:47:11.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:11.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:11.272261+0000) 2024-03-06T13:47:12.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:12.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:12.272533+0000) 2024-03-06T13:47:12.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:12.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:13.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:13.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:13.272809+0000) 2024-03-06T13:47:14.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:14.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:14.273082+0000) 2024-03-06T13:47:15.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:15.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:15.273343+0000) 2024-03-06T13:47:16.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:16.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:16.273583+0000) 2024-03-06T13:47:17.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:17.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:17.273852+0000) 2024-03-06T13:47:18.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:18.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:18.274046+0000) 2024-03-06T13:47:19.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:19.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:19.274318+0000) 2024-03-06T13:47:20.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:20.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:20.274577+0000) 2024-03-06T13:47:21.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:21.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:21.274825+0000) 2024-03-06T13:47:22.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:22.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:22.275084+0000) 2024-03-06T13:47:23.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:23.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:23.275325+0000) 2024-03-06T13:47:24.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:24.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:24.275574+0000) 2024-03-06T13:47:25.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:25.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:25.275817+0000) 2024-03-06T13:47:26.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:26.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:26.276074+0000) 2024-03-06T13:47:27.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:27.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:27.276329+0000) 2024-03-06T13:47:28.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:28.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:28.276586+0000) 2024-03-06T13:47:29.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:29.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:29.276858+0000) 2024-03-06T13:47:30.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:30.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:30.277104+0000) 2024-03-06T13:47:31.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:31.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:31.277340+0000) 2024-03-06T13:47:32.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:32.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:32.277589+0000) 2024-03-06T13:47:33.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:33.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:33.277794+0000) 2024-03-06T13:47:34.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:34.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:34.278037+0000) 2024-03-06T13:47:35.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:35.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:35.278297+0000) 2024-03-06T13:47:36.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:36.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:36.278559+0000) 2024-03-06T13:47:37.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:37.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:37.278825+0000) 2024-03-06T13:47:38.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:38.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:38.279079+0000) 2024-03-06T13:47:39.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:39.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:39.279347+0000) 2024-03-06T13:47:40.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:40.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:40.279622+0000) 2024-03-06T13:47:41.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:41.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:41.279922+0000) 2024-03-06T13:47:42.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:42.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:42.280199+0000) 2024-03-06T13:47:43.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:43.056 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:43.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:43.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:43.280476+0000) 2024-03-06T13:47:44.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:44.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:44.280771+0000) 2024-03-06T13:47:45.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:45.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:45.281034+0000) 2024-03-06T13:47:46.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:46.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:46.281288+0000) 2024-03-06T13:47:47.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:47.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:47.281533+0000) 2024-03-06T13:47:48.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:48.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:48.281765+0000) 2024-03-06T13:47:49.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:49.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:49.282023+0000) 2024-03-06T13:47:50.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:50.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:50.282347+0000) 2024-03-06T13:47:51.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:51.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:51.282583+0000) 2024-03-06T13:47:52.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:52.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:52.282853+0000) 2024-03-06T13:47:53.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:53.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:53.283098+0000) 2024-03-06T13:47:54.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:54.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:54.283342+0000) 2024-03-06T13:47:55.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:55.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:55.283591+0000) 2024-03-06T13:47:56.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:56.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:56.283843+0000) 2024-03-06T13:47:57.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:57.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:57.284111+0000) 2024-03-06T13:47:58.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:58.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:58.284363+0000) 2024-03-06T13:47:59.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:47:59.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:59.284620+0000) 2024-03-06T13:48:00.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:00.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:00.284873+0000) 2024-03-06T13:48:01.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:01.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:01.285062+0000) 2024-03-06T13:48:02.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:02.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:02.285330+0000) 2024-03-06T13:48:03.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:03.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:03.285581+0000) 2024-03-06T13:48:04.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:04.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:04.285822+0000) 2024-03-06T13:48:05.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:05.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:05.286067+0000) 2024-03-06T13:48:06.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:06.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:06.286309+0000) 2024-03-06T13:48:07.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:07.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:07.286586+0000) 2024-03-06T13:48:08.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:08.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:08.286842+0000) 2024-03-06T13:48:09.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:09.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:09.287039+0000) 2024-03-06T13:48:10.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:10.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:10.287288+0000) 2024-03-06T13:48:11.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:11.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:11.287540+0000) 2024-03-06T13:48:12.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:12.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:12.287810+0000) 2024-03-06T13:48:13.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:13.198 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:13.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:13.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:13.288042+0000) 2024-03-06T13:48:14.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:14.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:14.288330+0000) 2024-03-06T13:48:15.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:15.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:15.288577+0000) 2024-03-06T13:48:16.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:16.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:16.288878+0000) 2024-03-06T13:48:17.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:17.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:17.289165+0000) 2024-03-06T13:48:18.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:18.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:18.289407+0000) 2024-03-06T13:48:19.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:19.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:19.289740+0000) 2024-03-06T13:48:20.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:20.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:20.290085+0000) 2024-03-06T13:48:21.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:21.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:21.290299+0000) 2024-03-06T13:48:22.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:22.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:22.290545+0000) 2024-03-06T13:48:23.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:23.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:23.290799+0000) 2024-03-06T13:48:24.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:24.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:24.291092+0000) 2024-03-06T13:48:25.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:25.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:25.291354+0000) 2024-03-06T13:48:26.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:26.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:26.291623+0000) 2024-03-06T13:48:27.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:27.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:27.291929+0000) 2024-03-06T13:48:28.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:28.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:28.292199+0000) 2024-03-06T13:48:29.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:29.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:29.292483+0000) 2024-03-06T13:48:30.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:30.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:30.292800+0000) 2024-03-06T13:48:31.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:31.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:31.293054+0000) 2024-03-06T13:48:32.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:32.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:32.293318+0000) 2024-03-06T13:48:33.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:33.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:33.293581+0000) 2024-03-06T13:48:34.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:34.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:34.293861+0000) 2024-03-06T13:48:35.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:35.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:35.294134+0000) 2024-03-06T13:48:36.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:36.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:36.294375+0000) 2024-03-06T13:48:37.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:37.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:37.294684+0000) 2024-03-06T13:48:38.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:38.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:38.294954+0000) 2024-03-06T13:48:39.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:39.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:39.295231+0000) 2024-03-06T13:48:40.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:40.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:40.295555+0000) 2024-03-06T13:48:41.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:41.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:41.295818+0000) 2024-03-06T13:48:42.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:42.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:42.296079+0000) 2024-03-06T13:48:43.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:43.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:43.296322+0000) 2024-03-06T13:48:43.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:43.370 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:44.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:44.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:44.296606+0000) 2024-03-06T13:48:45.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:45.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:45.296856+0000) 2024-03-06T13:48:46.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:46.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:46.297118+0000) 2024-03-06T13:48:47.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:47.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:47.297362+0000) 2024-03-06T13:48:48.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:48.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:48.297622+0000) 2024-03-06T13:48:49.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:49.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:49.298009+0000) 2024-03-06T13:48:50.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:50.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:50.298320+0000) 2024-03-06T13:48:51.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:51.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:51.298583+0000) 2024-03-06T13:48:52.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:52.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:52.298830+0000) 2024-03-06T13:48:53.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:53.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:53.299087+0000) 2024-03-06T13:48:54.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:54.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:54.299344+0000) 2024-03-06T13:48:55.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:55.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:55.299586+0000) 2024-03-06T13:48:56.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:56.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:56.299845+0000) 2024-03-06T13:48:57.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:57.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:57.300104+0000) 2024-03-06T13:48:58.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:58.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:58.300347+0000) 2024-03-06T13:48:59.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:48:59.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:59.300556+0000) 2024-03-06T13:49:00.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:00.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:00.300809+0000) 2024-03-06T13:49:01.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:01.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:01.301048+0000) 2024-03-06T13:49:02.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:02.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:02.301302+0000) 2024-03-06T13:49:03.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:03.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:03.301561+0000) 2024-03-06T13:49:04.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:04.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:04.301822+0000) 2024-03-06T13:49:05.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:05.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:05.302095+0000) 2024-03-06T13:49:06.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:06.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:06.302348+0000) 2024-03-06T13:49:07.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:07.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:07.302605+0000) 2024-03-06T13:49:08.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:08.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:08.302902+0000) 2024-03-06T13:49:09.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:09.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:09.303145+0000) 2024-03-06T13:49:10.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:10.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:10.303397+0000) 2024-03-06T13:49:11.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:11.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:11.303684+0000) 2024-03-06T13:49:12.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:12.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:12.303930+0000) 2024-03-06T13:49:13.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:13.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:13.304173+0000) 2024-03-06T13:49:13.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:13.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:14.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:14.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:14.304412+0000) 2024-03-06T13:49:15.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:15.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:15.304725+0000) 2024-03-06T13:49:16.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:16.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:16.304977+0000) 2024-03-06T13:49:17.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:17.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:17.305163+0000) 2024-03-06T13:49:18.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:18.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:18.305403+0000) 2024-03-06T13:49:19.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:19.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:19.305681+0000) 2024-03-06T13:49:20.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:20.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:20.305951+0000) 2024-03-06T13:49:21.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:21.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:21.306204+0000) 2024-03-06T13:49:22.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:22.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:22.306442+0000) 2024-03-06T13:49:23.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:23.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:23.306746+0000) 2024-03-06T13:49:24.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:24.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:24.306992+0000) 2024-03-06T13:49:25.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:25.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:25.307255+0000) 2024-03-06T13:49:26.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:26.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:26.307457+0000) 2024-03-06T13:49:27.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:27.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:27.307776+0000) 2024-03-06T13:49:28.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:28.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:28.308024+0000) 2024-03-06T13:49:29.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:29.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:29.308312+0000) 2024-03-06T13:49:30.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:30.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:30.308684+0000) 2024-03-06T13:49:31.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:31.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:31.308959+0000) 2024-03-06T13:49:32.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:32.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:32.309216+0000) 2024-03-06T13:49:33.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:33.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:33.309471+0000) 2024-03-06T13:49:34.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:34.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:34.309807+0000) 2024-03-06T13:49:35.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:35.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:35.310072+0000) 2024-03-06T13:49:36.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:36.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:36.310332+0000) 2024-03-06T13:49:37.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:37.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:37.310684+0000) 2024-03-06T13:49:38.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:38.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:38.310942+0000) 2024-03-06T13:49:39.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:39.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:39.311233+0000) 2024-03-06T13:49:40.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:40.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:40.311483+0000) 2024-03-06T13:49:41.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:41.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:41.311792+0000) 2024-03-06T13:49:42.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:42.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:42.312051+0000) 2024-03-06T13:49:43.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:43.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:43.312305+0000) 2024-03-06T13:49:43.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:43.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:44.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:44.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:44.312590+0000) 2024-03-06T13:49:45.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:45.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:45.312902+0000) 2024-03-06T13:49:46.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:46.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:46.313144+0000) 2024-03-06T13:49:47.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:47.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:47.313386+0000) 2024-03-06T13:49:48.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:48.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:48.313661+0000) 2024-03-06T13:49:49.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:49.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:49.313978+0000) 2024-03-06T13:49:50.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:50.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:50.314303+0000) 2024-03-06T13:49:51.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:51.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:51.314552+0000) 2024-03-06T13:49:52.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:52.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:52.314814+0000) 2024-03-06T13:49:53.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:53.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:53.315060+0000) 2024-03-06T13:49:54.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:54.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:54.315358+0000) 2024-03-06T13:49:55.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:55.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:55.315620+0000) 2024-03-06T13:49:56.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:56.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:56.315921+0000) 2024-03-06T13:49:57.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:57.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:57.316157+0000) 2024-03-06T13:49:58.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:58.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:58.316426+0000) 2024-03-06T13:49:59.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:49:59.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:59.316734+0000) 2024-03-06T13:50:00.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:00.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:00.317048+0000) 2024-03-06T13:50:01.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:01.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:01.317323+0000) 2024-03-06T13:50:02.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:02.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:02.317584+0000) 2024-03-06T13:50:03.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:03.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:03.317837+0000) 2024-03-06T13:50:04.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:04.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:04.318135+0000) 2024-03-06T13:50:05.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:05.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:05.318383+0000) 2024-03-06T13:50:06.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:06.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:06.318677+0000) 2024-03-06T13:50:07.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:07.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:07.318940+0000) 2024-03-06T13:50:08.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:08.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:08.319188+0000) 2024-03-06T13:50:09.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:09.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:09.319488+0000) 2024-03-06T13:50:10.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:10.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:10.319832+0000) 2024-03-06T13:50:11.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:11.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:11.320084+0000) 2024-03-06T13:50:12.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:12.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:12.320323+0000) 2024-03-06T13:50:13.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:13.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:13.320569+0000) 2024-03-06T13:50:13.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:13.828 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:14.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:14.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:14.320870+0000) 2024-03-06T13:50:15.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:15.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:15.321148+0000) 2024-03-06T13:50:16.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:16.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:16.321411+0000) 2024-03-06T13:50:17.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:17.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:17.321712+0000) 2024-03-06T13:50:18.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:18.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:18.321975+0000) 2024-03-06T13:50:19.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:19.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:19.322225+0000) 2024-03-06T13:50:20.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:20.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:20.322496+0000) 2024-03-06T13:50:21.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:21.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:21.322753+0000) 2024-03-06T13:50:22.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:22.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:22.323008+0000) 2024-03-06T13:50:23.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:23.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:23.323190+0000) 2024-03-06T13:50:24.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:24.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:24.323432+0000) 2024-03-06T13:50:25.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:25.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:25.323804+0000) 2024-03-06T13:50:26.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:26.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:26.324044+0000) 2024-03-06T13:50:27.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:27.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:27.324297+0000) 2024-03-06T13:50:28.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:28.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:28.324537+0000) 2024-03-06T13:50:29.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:29.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:29.324801+0000) 2024-03-06T13:50:30.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:30.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:30.325076+0000) 2024-03-06T13:50:31.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:31.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:31.325336+0000) 2024-03-06T13:50:32.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:32.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:32.325527+0000) 2024-03-06T13:50:33.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:33.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:33.325795+0000) 2024-03-06T13:50:34.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:34.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:34.326041+0000) 2024-03-06T13:50:35.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:35.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:35.326310+0000) 2024-03-06T13:50:36.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:36.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:36.326547+0000) 2024-03-06T13:50:37.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:37.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:37.326810+0000) 2024-03-06T13:50:38.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:38.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:38.327068+0000) 2024-03-06T13:50:39.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:39.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:39.327327+0000) 2024-03-06T13:50:40.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:40.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:40.327606+0000) 2024-03-06T13:50:41.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:41.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:41.327908+0000) 2024-03-06T13:50:42.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:42.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:42.328148+0000) 2024-03-06T13:50:43.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:43.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:43.328408+0000) 2024-03-06T13:50:43.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:43.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:44.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:44.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:44.328715+0000) 2024-03-06T13:50:45.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:45.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:45.328964+0000) 2024-03-06T13:50:46.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:46.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:46.329222+0000) 2024-03-06T13:50:47.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:47.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:47.329470+0000) 2024-03-06T13:50:48.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:48.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:48.329756+0000) 2024-03-06T13:50:49.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:49.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:49.330003+0000) 2024-03-06T13:50:50.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:50.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:50.330277+0000) 2024-03-06T13:50:51.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:51.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:51.330515+0000) 2024-03-06T13:50:52.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:52.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:52.330768+0000) 2024-03-06T13:50:53.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:53.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:53.331025+0000) 2024-03-06T13:50:54.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:54.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:54.331269+0000) 2024-03-06T13:50:55.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:55.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:55.331517+0000) 2024-03-06T13:50:56.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:56.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:56.331781+0000) 2024-03-06T13:50:57.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:57.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:57.332050+0000) 2024-03-06T13:50:58.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:58.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:58.332311+0000) 2024-03-06T13:50:59.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:50:59.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:59.332570+0000) 2024-03-06T13:51:00.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:00.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:00.332845+0000) 2024-03-06T13:51:01.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:01.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:01.333094+0000) 2024-03-06T13:51:02.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:02.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:02.333350+0000) 2024-03-06T13:51:03.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:03.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:03.333596+0000) 2024-03-06T13:51:04.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:04.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:04.333864+0000) 2024-03-06T13:51:05.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:05.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:05.334122+0000) 2024-03-06T13:51:06.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:06.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:06.334380+0000) 2024-03-06T13:51:07.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:07.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:07.334664+0000) 2024-03-06T13:51:08.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:08.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:08.334927+0000) 2024-03-06T13:51:09.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:09.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:09.335165+0000) 2024-03-06T13:51:10.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:10.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:10.335427+0000) 2024-03-06T13:51:11.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:11.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:11.335716+0000) 2024-03-06T13:51:12.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:12.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:12.335987+0000) 2024-03-06T13:51:13.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:13.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:13.336248+0000) 2024-03-06T13:51:14.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:14.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:14.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:14.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:14.336526+0000) 2024-03-06T13:51:15.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:15.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:15.336783+0000) 2024-03-06T13:51:16.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:16.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:16.337016+0000) 2024-03-06T13:51:17.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:17.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:17.337255+0000) 2024-03-06T13:51:18.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:18.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:18.337561+0000) 2024-03-06T13:51:19.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:19.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:19.337816+0000) 2024-03-06T13:51:20.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:20.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:20.338084+0000) 2024-03-06T13:51:21.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:21.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:21.338350+0000) 2024-03-06T13:51:22.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:22.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:22.338585+0000) 2024-03-06T13:51:23.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:23.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:23.338847+0000) 2024-03-06T13:51:24.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:24.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:24.339088+0000) 2024-03-06T13:51:25.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:25.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:25.339350+0000) 2024-03-06T13:51:26.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:26.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:26.339607+0000) 2024-03-06T13:51:27.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:27.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:27.339888+0000) 2024-03-06T13:51:28.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:28.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:28.340134+0000) 2024-03-06T13:51:29.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:29.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:29.340381+0000) 2024-03-06T13:51:30.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:30.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:30.340684+0000) 2024-03-06T13:51:31.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:31.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:31.340952+0000) 2024-03-06T13:51:32.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:32.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:32.341211+0000) 2024-03-06T13:51:33.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:33.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:33.341476+0000) 2024-03-06T13:51:34.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:34.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:34.341734+0000) 2024-03-06T13:51:35.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:35.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:35.341981+0000) 2024-03-06T13:51:36.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:36.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:36.342241+0000) 2024-03-06T13:51:37.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:37.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:37.342532+0000) 2024-03-06T13:51:38.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:38.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:38.342842+0000) 2024-03-06T13:51:39.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:39.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:39.343092+0000) 2024-03-06T13:51:40.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:40.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:40.343445+0000) 2024-03-06T13:51:41.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:41.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:41.343777+0000) 2024-03-06T13:51:42.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:42.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:42.344076+0000) 2024-03-06T13:51:43.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:43.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:43.344381+0000) 2024-03-06T13:51:44.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:44.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:44.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:44.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:44.344582+0000) 2024-03-06T13:51:45.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:45.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:45.344834+0000) 2024-03-06T13:51:46.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:46.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:46.345083+0000) 2024-03-06T13:51:47.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:47.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:47.345342+0000) 2024-03-06T13:51:48.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:48.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:48.345585+0000) 2024-03-06T13:51:49.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:49.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:49.345881+0000) 2024-03-06T13:51:50.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:50.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:50.346197+0000) 2024-03-06T13:51:51.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:51.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:51.346506+0000) 2024-03-06T13:51:52.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:52.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:52.346812+0000) 2024-03-06T13:51:53.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:53.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:53.347063+0000) 2024-03-06T13:51:54.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:54.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:54.347371+0000) 2024-03-06T13:51:55.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:55.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:55.347742+0000) 2024-03-06T13:51:56.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:56.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:56.348059+0000) 2024-03-06T13:51:57.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:57.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:57.348361+0000) 2024-03-06T13:51:58.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:58.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:58.348725+0000) 2024-03-06T13:51:59.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:51:59.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:59.349040+0000) 2024-03-06T13:52:00.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:00.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:00.349373+0000) 2024-03-06T13:52:01.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:01.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:01.349672+0000) 2024-03-06T13:52:02.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:02.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:02.349967+0000) 2024-03-06T13:52:03.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:03.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:03.350253+0000) 2024-03-06T13:52:04.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:04.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:04.350562+0000) 2024-03-06T13:52:05.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:05.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:05.350877+0000) 2024-03-06T13:52:06.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:06.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:06.351163+0000) 2024-03-06T13:52:07.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:07.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:07.351598+0000) 2024-03-06T13:52:08.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:08.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:08.351954+0000) 2024-03-06T13:52:09.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:09.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:09.352213+0000) 2024-03-06T13:52:10.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:10.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:10.352494+0000) 2024-03-06T13:52:11.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:11.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:11.352746+0000) 2024-03-06T13:52:12.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:12.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:12.352980+0000) 2024-03-06T13:52:13.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:13.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:13.353226+0000) 2024-03-06T13:52:14.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:14.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:14.353811+0000) 2024-03-06T13:52:14.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:14.415 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:15.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:15.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:15.354133+0000) 2024-03-06T13:52:16.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:16.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:16.354451+0000) 2024-03-06T13:52:17.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:17.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:17.354761+0000) 2024-03-06T13:52:18.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:18.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:18.355035+0000) 2024-03-06T13:52:19.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:19.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:19.355224+0000) 2024-03-06T13:52:20.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:20.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:20.355498+0000) 2024-03-06T13:52:21.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:21.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:21.355757+0000) 2024-03-06T13:52:22.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:22.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:22.355995+0000) 2024-03-06T13:52:23.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:23.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:23.356262+0000) 2024-03-06T13:52:24.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:24.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:24.356444+0000) 2024-03-06T13:52:25.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:25.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:25.356758+0000) 2024-03-06T13:52:26.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:26.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:26.356998+0000) 2024-03-06T13:52:27.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:27.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:27.357263+0000) 2024-03-06T13:52:28.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:28.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:28.357524+0000) 2024-03-06T13:52:29.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:29.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:29.357783+0000) 2024-03-06T13:52:30.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:30.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:30.358042+0000) 2024-03-06T13:52:31.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:31.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:31.358291+0000) 2024-03-06T13:52:32.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:32.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:32.358553+0000) 2024-03-06T13:52:33.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:33.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:33.358802+0000) 2024-03-06T13:52:34.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:34.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:34.358990+0000) 2024-03-06T13:52:35.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:35.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:35.359240+0000) 2024-03-06T13:52:36.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:36.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:36.359494+0000) 2024-03-06T13:52:37.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:37.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:37.359756+0000) 2024-03-06T13:52:38.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:38.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:38.360030+0000) 2024-03-06T13:52:39.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:39.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:39.360242+0000) 2024-03-06T13:52:40.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:40.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:40.360496+0000) 2024-03-06T13:52:41.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:41.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:41.360739+0000) 2024-03-06T13:52:42.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:42.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:42.361011+0000) 2024-03-06T13:52:43.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:43.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:43.361272+0000) 2024-03-06T13:52:44.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:44.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:44.361512+0000) 2024-03-06T13:52:44.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:44.571 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:45.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:45.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:45.361777+0000) 2024-03-06T13:52:46.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:46.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:46.362009+0000) 2024-03-06T13:52:47.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:47.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:47.362266+0000) 2024-03-06T13:52:48.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:48.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:48.362538+0000) 2024-03-06T13:52:49.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:49.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:49.362751+0000) 2024-03-06T13:52:50.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:50.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:50.363009+0000) 2024-03-06T13:52:51.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:51.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:51.363247+0000) 2024-03-06T13:52:52.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:52.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:52.363505+0000) 2024-03-06T13:52:53.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:53.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:53.363772+0000) 2024-03-06T13:52:54.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:54.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:54.364068+0000) 2024-03-06T13:52:55.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:55.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:55.364331+0000) 2024-03-06T13:52:56.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:56.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:56.364589+0000) 2024-03-06T13:52:57.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:57.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:57.364831+0000) 2024-03-06T13:52:58.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:58.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:58.365083+0000) 2024-03-06T13:52:59.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:52:59.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:59.365322+0000) 2024-03-06T13:53:00.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:00.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:00.365581+0000) 2024-03-06T13:53:01.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:01.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:01.365842+0000) 2024-03-06T13:53:02.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:02.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:02.366096+0000) 2024-03-06T13:53:03.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:03.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:03.366359+0000) 2024-03-06T13:53:04.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:04.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:04.366620+0000) 2024-03-06T13:53:05.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:05.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:05.366849+0000) 2024-03-06T13:53:06.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:06.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:06.367109+0000) 2024-03-06T13:53:07.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:07.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:07.367364+0000) 2024-03-06T13:53:08.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:08.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:08.367609+0000) 2024-03-06T13:53:09.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:09.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:09.367939+0000) 2024-03-06T13:53:10.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:10.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:10.368231+0000) 2024-03-06T13:53:11.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:11.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:11.368493+0000) 2024-03-06T13:53:12.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:12.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:12.368759+0000) 2024-03-06T13:53:13.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:13.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:13.369046+0000) 2024-03-06T13:53:14.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:14.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:14.369241+0000) 2024-03-06T13:53:14.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:14.718 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:15.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:15.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:15.369525+0000) 2024-03-06T13:53:16.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:16.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:16.369804+0000) 2024-03-06T13:53:17.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:17.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:17.370075+0000) 2024-03-06T13:53:18.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:18.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:18.370373+0000) 2024-03-06T13:53:19.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:19.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:19.370672+0000) 2024-03-06T13:53:20.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:20.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:20.370923+0000) 2024-03-06T13:53:21.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:21.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:21.371185+0000) 2024-03-06T13:53:22.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:22.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:22.371419+0000) 2024-03-06T13:53:23.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:23.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:23.371731+0000) 2024-03-06T13:53:24.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:24.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:24.372002+0000) 2024-03-06T13:53:25.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:25.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:25.372285+0000) 2024-03-06T13:53:26.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:26.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:26.372568+0000) 2024-03-06T13:53:27.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:27.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:27.372878+0000) 2024-03-06T13:53:28.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:28.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:28.373171+0000) 2024-03-06T13:53:29.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:29.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:29.373467+0000) 2024-03-06T13:53:30.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:30.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:30.373780+0000) 2024-03-06T13:53:31.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:31.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:31.374089+0000) 2024-03-06T13:53:32.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:32.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:32.374322+0000) 2024-03-06T13:53:33.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:33.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:33.374677+0000) 2024-03-06T13:53:34.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:34.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:34.374991+0000) 2024-03-06T13:53:35.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:35.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:35.375293+0000) 2024-03-06T13:53:36.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:36.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:36.375615+0000) 2024-03-06T13:53:37.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:37.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:37.375909+0000) 2024-03-06T13:53:38.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:38.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:38.376223+0000) 2024-03-06T13:53:39.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:39.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:39.376515+0000) 2024-03-06T13:53:40.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:40.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:40.376860+0000) 2024-03-06T13:53:41.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:41.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:41.377161+0000) 2024-03-06T13:53:42.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:42.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:42.377455+0000) 2024-03-06T13:53:43.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:43.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:43.377774+0000) 2024-03-06T13:53:44.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:44.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:44.378071+0000) 2024-03-06T13:53:44.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:44.856 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:45.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:45.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:45.378377+0000) 2024-03-06T13:53:46.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:46.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:46.378743+0000) 2024-03-06T13:53:47.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:47.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:47.379050+0000) 2024-03-06T13:53:48.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:48.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:48.379348+0000) 2024-03-06T13:53:49.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:49.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:49.379684+0000) 2024-03-06T13:53:50.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:50.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:50.379984+0000) 2024-03-06T13:53:51.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:51.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:51.380285+0000) 2024-03-06T13:53:52.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:52.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:52.380574+0000) 2024-03-06T13:53:53.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:53.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:53.380869+0000) 2024-03-06T13:53:54.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:54.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:54.381162+0000) 2024-03-06T13:53:55.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:55.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:55.381459+0000) 2024-03-06T13:53:56.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:56.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:56.381766+0000) 2024-03-06T13:53:57.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:57.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:57.382078+0000) 2024-03-06T13:53:58.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:58.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:58.382390+0000) 2024-03-06T13:53:59.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:53:59.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:59.382735+0000) 2024-03-06T13:54:00.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:00.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:00.383009+0000) 2024-03-06T13:54:01.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:01.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:01.383318+0000) 2024-03-06T13:54:02.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:02.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:02.383681+0000) 2024-03-06T13:54:03.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:03.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:03.383999+0000) 2024-03-06T13:54:04.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:04.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:04.384315+0000) 2024-03-06T13:54:05.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:05.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:05.384682+0000) 2024-03-06T13:54:06.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:06.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:06.385001+0000) 2024-03-06T13:54:07.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:07.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:07.385327+0000) 2024-03-06T13:54:08.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:08.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:08.385684+0000) 2024-03-06T13:54:09.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:09.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:09.386002+0000) 2024-03-06T13:54:10.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:10.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:10.386294+0000) 2024-03-06T13:54:11.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:11.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:11.386597+0000) 2024-03-06T13:54:12.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:12.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:12.386945+0000) 2024-03-06T13:54:13.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:13.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:13.387250+0000) 2024-03-06T13:54:14.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:14.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:14.387538+0000) 2024-03-06T13:54:14.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:14.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:15.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:15.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:15.387800+0000) 2024-03-06T13:54:16.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:16.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:16.388094+0000) 2024-03-06T13:54:17.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:17.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:17.388393+0000) 2024-03-06T13:54:18.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:18.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:18.388740+0000) 2024-03-06T13:54:19.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:19.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:19.389043+0000) 2024-03-06T13:54:20.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:20.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:20.389342+0000) 2024-03-06T13:54:21.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:21.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:21.389720+0000) 2024-03-06T13:54:22.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:22.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:22.390038+0000) 2024-03-06T13:54:23.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:23.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:23.390329+0000) 2024-03-06T13:54:24.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:24.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:24.390689+0000) 2024-03-06T13:54:25.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:25.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:25.390978+0000) 2024-03-06T13:54:26.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:26.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:26.391290+0000) 2024-03-06T13:54:27.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:27.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:27.391528+0000) 2024-03-06T13:54:28.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:28.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:28.391809+0000) 2024-03-06T13:54:29.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:29.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:29.392087+0000) 2024-03-06T13:54:30.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:30.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:30.392364+0000) 2024-03-06T13:54:31.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:31.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:31.392620+0000) 2024-03-06T13:54:32.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:32.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:32.392861+0000) 2024-03-06T13:54:33.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:33.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:33.393111+0000) 2024-03-06T13:54:34.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:34.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:34.393389+0000) 2024-03-06T13:54:35.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:35.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:35.393757+0000) 2024-03-06T13:54:36.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:36.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:36.394035+0000) 2024-03-06T13:54:37.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:37.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:37.394308+0000) 2024-03-06T13:54:38.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:38.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:38.394553+0000) 2024-03-06T13:54:39.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:39.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:39.394842+0000) 2024-03-06T13:54:40.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:40.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:40.395117+0000) 2024-03-06T13:54:41.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:41.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:41.395387+0000) 2024-03-06T13:54:42.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:42.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:42.395677+0000) 2024-03-06T13:54:43.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:43.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:43.395894+0000) 2024-03-06T13:54:44.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:44.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:44.396171+0000) 2024-03-06T13:54:45.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:45.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:45.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:45.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:45.396460+0000) 2024-03-06T13:54:46.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:46.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:46.396742+0000) 2024-03-06T13:54:47.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:47.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:47.397025+0000) 2024-03-06T13:54:48.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:48.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:48.397236+0000) 2024-03-06T13:54:49.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:49.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:49.397518+0000) 2024-03-06T13:54:50.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:50.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:50.397789+0000) 2024-03-06T13:54:51.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:51.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:51.398075+0000) 2024-03-06T13:54:52.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:52.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:52.398348+0000) 2024-03-06T13:54:53.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:53.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:53.398603+0000) 2024-03-06T13:54:54.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:54.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:54.398928+0000) 2024-03-06T13:54:55.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:55.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:55.399191+0000) 2024-03-06T13:54:56.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:56.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:56.399455+0000) 2024-03-06T13:54:57.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:57.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:57.399742+0000) 2024-03-06T13:54:58.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:58.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:58.399957+0000) 2024-03-06T13:54:59.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:54:59.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:59.400228+0000) 2024-03-06T13:55:00.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:00.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:00.400516+0000) 2024-03-06T13:55:01.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:01.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:01.400791+0000) 2024-03-06T13:55:02.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:02.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:02.401050+0000) 2024-03-06T13:55:03.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:03.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:03.401315+0000) 2024-03-06T13:55:04.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:04.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:04.401596+0000) 2024-03-06T13:55:05.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:05.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:05.401859+0000) 2024-03-06T13:55:06.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:06.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:06.402117+0000) 2024-03-06T13:55:07.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:07.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:07.402330+0000) 2024-03-06T13:55:08.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:08.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:08.402550+0000) 2024-03-06T13:55:09.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:09.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:09.402771+0000) 2024-03-06T13:55:10.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:10.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:10.403044+0000) 2024-03-06T13:55:11.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:11.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:11.403321+0000) 2024-03-06T13:55:12.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:12.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:12.403602+0000) 2024-03-06T13:55:13.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:13.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:13.403932+0000) 2024-03-06T13:55:14.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:14.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:14.404180+0000) 2024-03-06T13:55:15.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:15.281 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:15.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:15.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:15.404462+0000) 2024-03-06T13:55:16.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:16.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:16.404758+0000) 2024-03-06T13:55:17.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:17.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:17.405038+0000) 2024-03-06T13:55:18.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:18.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:18.405251+0000) 2024-03-06T13:55:19.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:19.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:19.405533+0000) 2024-03-06T13:55:20.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:20.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:20.405799+0000) 2024-03-06T13:55:21.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:21.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:21.406058+0000) 2024-03-06T13:55:22.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:22.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:22.406268+0000) 2024-03-06T13:55:23.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:23.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:23.406596+0000) 2024-03-06T13:55:24.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:24.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:24.406919+0000) 2024-03-06T13:55:25.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:25.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:25.407193+0000) 2024-03-06T13:55:26.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:26.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:26.407479+0000) 2024-03-06T13:55:27.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:27.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:27.407739+0000) 2024-03-06T13:55:28.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:28.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:28.407998+0000) 2024-03-06T13:55:29.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:29.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:29.408259+0000) 2024-03-06T13:55:30.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:30.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:30.408525+0000) 2024-03-06T13:55:31.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:31.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:31.408770+0000) 2024-03-06T13:55:32.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:32.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:32.409044+0000) 2024-03-06T13:55:33.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:33.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:33.409307+0000) 2024-03-06T13:55:34.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:34.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:34.409589+0000) 2024-03-06T13:55:35.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:35.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:35.409853+0000) 2024-03-06T13:55:36.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:36.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:36.410117+0000) 2024-03-06T13:55:37.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:37.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:37.410377+0000) 2024-03-06T13:55:38.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:38.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:38.410671+0000) 2024-03-06T13:55:39.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:39.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:39.410929+0000) 2024-03-06T13:55:40.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:40.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:40.411193+0000) 2024-03-06T13:55:41.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:41.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:41.411458+0000) 2024-03-06T13:55:42.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:42.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:42.411746+0000) 2024-03-06T13:55:43.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:43.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:43.411973+0000) 2024-03-06T13:55:44.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:44.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:44.412239+0000) 2024-03-06T13:55:45.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:45.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:45.412507+0000) 2024-03-06T13:55:45.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:45.435 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:46.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:46.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:46.412809+0000) 2024-03-06T13:55:47.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:47.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:47.413069+0000) 2024-03-06T13:55:48.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:48.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:48.413330+0000) 2024-03-06T13:55:49.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:49.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:49.413608+0000) 2024-03-06T13:55:50.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:50.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:50.413966+0000) 2024-03-06T13:55:51.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:51.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:51.414240+0000) 2024-03-06T13:55:52.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:52.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:52.414493+0000) 2024-03-06T13:55:53.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:53.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:53.414765+0000) 2024-03-06T13:55:54.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:54.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:54.415030+0000) 2024-03-06T13:55:55.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:55.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:55.415297+0000) 2024-03-06T13:55:56.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:56.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:56.415562+0000) 2024-03-06T13:55:57.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:57.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:57.415825+0000) 2024-03-06T13:55:58.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:58.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:58.416064+0000) 2024-03-06T13:55:59.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:55:59.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:59.416325+0000) 2024-03-06T13:56:00.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:00.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:00.416597+0000) 2024-03-06T13:56:01.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:01.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:01.416866+0000) 2024-03-06T13:56:02.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:02.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:02.417108+0000) 2024-03-06T13:56:03.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:03.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:03.417349+0000) 2024-03-06T13:56:04.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:04.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:04.417595+0000) 2024-03-06T13:56:05.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:05.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:05.417848+0000) 2024-03-06T13:56:06.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:06.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:06.418117+0000) 2024-03-06T13:56:07.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:07.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:07.418377+0000) 2024-03-06T13:56:08.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:08.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:08.418624+0000) 2024-03-06T13:56:09.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:09.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:09.418907+0000) 2024-03-06T13:56:10.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:10.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:10.419174+0000) 2024-03-06T13:56:11.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:11.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:11.419441+0000) 2024-03-06T13:56:12.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:12.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:12.419739+0000) 2024-03-06T13:56:13.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:13.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:13.419995+0000) 2024-03-06T13:56:14.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:14.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:14.420261+0000) 2024-03-06T13:56:15.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:15.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:15.420518+0000) 2024-03-06T13:56:15.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:15.574 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:16.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:16.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:16.420777+0000) 2024-03-06T13:56:17.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:17.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:17.421020+0000) 2024-03-06T13:56:18.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:18.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:18.421274+0000) 2024-03-06T13:56:19.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:19.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:19.421512+0000) 2024-03-06T13:56:20.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:20.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:20.421794+0000) 2024-03-06T13:56:21.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:21.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:21.422052+0000) 2024-03-06T13:56:22.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:22.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:22.422292+0000) 2024-03-06T13:56:23.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:23.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:23.422551+0000) 2024-03-06T13:56:24.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:24.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:24.422807+0000) 2024-03-06T13:56:25.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:25.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:25.423073+0000) 2024-03-06T13:56:26.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:26.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:26.423342+0000) 2024-03-06T13:56:27.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:27.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:27.423602+0000) 2024-03-06T13:56:28.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:28.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:28.423880+0000) 2024-03-06T13:56:29.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:29.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:29.424082+0000) 2024-03-06T13:56:30.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:30.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:30.424348+0000) 2024-03-06T13:56:31.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:31.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:31.424605+0000) 2024-03-06T13:56:32.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:32.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:32.424893+0000) 2024-03-06T13:56:33.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:33.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:33.425080+0000) 2024-03-06T13:56:34.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:34.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:34.425336+0000) 2024-03-06T13:56:35.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:35.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:35.425590+0000) 2024-03-06T13:56:36.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:36.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:36.425847+0000) 2024-03-06T13:56:37.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:37.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:37.426105+0000) 2024-03-06T13:56:38.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:38.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:38.426368+0000) 2024-03-06T13:56:39.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:39.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:39.426665+0000) 2024-03-06T13:56:40.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:40.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:40.426954+0000) 2024-03-06T13:56:41.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:41.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:41.427209+0000) 2024-03-06T13:56:42.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:42.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:42.427464+0000) 2024-03-06T13:56:43.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:43.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:43.427755+0000) 2024-03-06T13:56:44.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:44.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:44.428010+0000) 2024-03-06T13:56:45.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:45.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:45.428266+0000) 2024-03-06T13:56:45.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:45.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:46.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:46.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:46.428541+0000) 2024-03-06T13:56:47.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:47.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:47.428784+0000) 2024-03-06T13:56:48.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:48.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:48.429041+0000) 2024-03-06T13:56:49.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:49.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:49.429308+0000) 2024-03-06T13:56:50.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:50.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:50.429572+0000) 2024-03-06T13:56:51.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:51.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:51.429836+0000) 2024-03-06T13:56:52.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:52.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:52.430080+0000) 2024-03-06T13:56:53.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:53.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:53.430324+0000) 2024-03-06T13:56:54.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:54.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:54.430597+0000) 2024-03-06T13:56:55.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:55.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:55.430839+0000) 2024-03-06T13:56:56.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:56.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:56.431122+0000) 2024-03-06T13:56:57.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:57.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:57.431385+0000) 2024-03-06T13:56:58.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:58.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:58.431674+0000) 2024-03-06T13:56:59.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:56:59.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:59.431941+0000) 2024-03-06T13:57:00.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:00.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:00.432185+0000) 2024-03-06T13:57:01.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:01.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:01.432428+0000) 2024-03-06T13:57:02.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:02.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:02.432723+0000) 2024-03-06T13:57:03.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:03.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:03.432988+0000) 2024-03-06T13:57:04.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:04.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:04.433256+0000) 2024-03-06T13:57:05.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:05.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:05.433583+0000) 2024-03-06T13:57:06.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:06.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:06.433860+0000) 2024-03-06T13:57:07.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:07.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:07.434117+0000) 2024-03-06T13:57:08.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:08.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:08.434355+0000) 2024-03-06T13:57:09.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:09.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:09.434623+0000) 2024-03-06T13:57:10.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:10.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:10.434869+0000) 2024-03-06T13:57:11.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:11.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:11.435101+0000) 2024-03-06T13:57:12.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:12.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:12.435287+0000) 2024-03-06T13:57:13.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:13.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:13.435528+0000) 2024-03-06T13:57:14.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:14.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:14.435795+0000) 2024-03-06T13:57:15.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:15.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:15.436036+0000) 2024-03-06T13:57:15.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:15.879 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:16.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:16.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:16.436300+0000) 2024-03-06T13:57:17.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:17.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:17.436541+0000) 2024-03-06T13:57:18.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:18.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:18.436784+0000) 2024-03-06T13:57:19.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:19.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:19.436992+0000) 2024-03-06T13:57:20.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:20.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:20.437200+0000) 2024-03-06T13:57:21.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:21.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:21.437406+0000) 2024-03-06T13:57:22.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:22.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:22.437680+0000) 2024-03-06T13:57:23.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:23.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:23.437873+0000) 2024-03-06T13:57:24.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:24.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:24.438075+0000) 2024-03-06T13:57:25.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:25.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:25.438327+0000) 2024-03-06T13:57:26.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:26.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:26.438578+0000) 2024-03-06T13:57:27.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:27.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:27.438817+0000) 2024-03-06T13:57:28.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:28.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:28.439072+0000) 2024-03-06T13:57:29.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:29.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:29.439278+0000) 2024-03-06T13:57:30.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:30.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:30.439524+0000) 2024-03-06T13:57:31.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:31.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:31.439786+0000) 2024-03-06T13:57:32.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:32.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:32.440036+0000) 2024-03-06T13:57:33.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:33.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:33.440231+0000) 2024-03-06T13:57:34.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:34.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:34.440496+0000) 2024-03-06T13:57:35.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:35.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:35.440767+0000) 2024-03-06T13:57:36.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:36.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:36.441032+0000) 2024-03-06T13:57:37.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:37.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:37.441227+0000) 2024-03-06T13:57:38.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:38.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:38.441485+0000) 2024-03-06T13:57:39.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:39.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:39.441754+0000) 2024-03-06T13:57:40.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:40.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:40.442021+0000) 2024-03-06T13:57:41.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:41.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:41.442207+0000) 2024-03-06T13:57:42.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:42.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:42.442472+0000) 2024-03-06T13:57:43.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:43.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:43.442757+0000) 2024-03-06T13:57:44.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:44.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:44.443025+0000) 2024-03-06T13:57:45.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:45.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:45.443242+0000) 2024-03-06T13:57:46.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:46.033 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:46.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:46.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:46.443527+0000) 2024-03-06T13:57:47.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:47.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:47.443769+0000) 2024-03-06T13:57:48.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:48.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:48.444027+0000) 2024-03-06T13:57:49.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:49.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:49.444275+0000) 2024-03-06T13:57:50.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:50.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:50.444546+0000) 2024-03-06T13:57:51.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:51.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:51.444774+0000) 2024-03-06T13:57:52.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:52.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:52.445016+0000) 2024-03-06T13:57:53.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:53.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:53.445315+0000) 2024-03-06T13:57:54.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:54.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:54.445577+0000) 2024-03-06T13:57:55.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:55.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:55.445836+0000) 2024-03-06T13:57:56.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:56.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:56.446075+0000) 2024-03-06T13:57:57.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:57.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:57.446333+0000) 2024-03-06T13:57:58.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:58.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:58.446597+0000) 2024-03-06T13:57:59.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:57:59.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:59.446841+0000) 2024-03-06T13:58:00.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:00.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:00.447051+0000) 2024-03-06T13:58:01.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:01.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:01.447287+0000) 2024-03-06T13:58:02.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:02.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:02.447543+0000) 2024-03-06T13:58:03.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:03.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:03.447793+0000) 2024-03-06T13:58:04.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:04.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:04.448048+0000) 2024-03-06T13:58:05.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:05.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:05.448315+0000) 2024-03-06T13:58:06.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:06.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:06.448568+0000) 2024-03-06T13:58:07.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:07.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:07.448815+0000) 2024-03-06T13:58:08.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:08.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:08.449067+0000) 2024-03-06T13:58:09.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:09.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:09.449326+0000) 2024-03-06T13:58:10.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:10.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:10.449577+0000) 2024-03-06T13:58:11.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:11.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:11.449815+0000) 2024-03-06T13:58:12.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:12.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:12.450071+0000) 2024-03-06T13:58:13.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:13.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:13.450330+0000) 2024-03-06T13:58:14.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:14.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:14.450586+0000) 2024-03-06T13:58:15.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:15.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:15.450843+0000) 2024-03-06T13:58:16.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:16.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:16.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:16.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:16.451057+0000) 2024-03-06T13:58:17.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:17.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:17.451307+0000) 2024-03-06T13:58:18.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:18.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:18.451544+0000) 2024-03-06T13:58:19.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:19.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:19.451781+0000) 2024-03-06T13:58:20.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:20.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:20.452053+0000) 2024-03-06T13:58:21.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:21.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:21.452304+0000) 2024-03-06T13:58:22.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:22.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:22.452566+0000) 2024-03-06T13:58:23.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:23.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:23.452803+0000) 2024-03-06T13:58:24.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:24.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:24.453060+0000) 2024-03-06T13:58:25.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:25.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:25.453302+0000) 2024-03-06T13:58:26.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:26.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:26.453541+0000) 2024-03-06T13:58:27.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:27.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:27.453764+0000) 2024-03-06T13:58:28.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:28.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:28.453995+0000) 2024-03-06T13:58:29.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:29.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:29.454233+0000) 2024-03-06T13:58:30.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:30.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:30.454484+0000) 2024-03-06T13:58:31.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:31.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:31.454746+0000) 2024-03-06T13:58:32.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:32.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:32.454941+0000) 2024-03-06T13:58:33.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:33.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:33.455181+0000) 2024-03-06T13:58:34.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:34.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:34.455433+0000) 2024-03-06T13:58:35.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:35.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:35.455736+0000) 2024-03-06T13:58:36.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:36.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:36.455995+0000) 2024-03-06T13:58:37.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:37.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:37.456255+0000) 2024-03-06T13:58:38.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:38.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:38.456456+0000) 2024-03-06T13:58:39.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:39.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:39.456755+0000) 2024-03-06T13:58:40.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:40.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:40.457025+0000) 2024-03-06T13:58:41.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:41.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:41.457213+0000) 2024-03-06T13:58:42.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:42.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:42.457456+0000) 2024-03-06T13:58:43.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:43.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:43.457751+0000) 2024-03-06T13:58:44.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:44.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:44.458010+0000) 2024-03-06T13:58:45.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:45.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:45.458251+0000) 2024-03-06T13:58:46.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:46.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:46.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:46.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:46.458564+0000) 2024-03-06T13:58:47.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:47.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:47.458802+0000) 2024-03-06T13:58:48.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:48.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:48.459061+0000) 2024-03-06T13:58:49.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:49.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:49.459320+0000) 2024-03-06T13:58:50.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:50.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:50.459588+0000) 2024-03-06T13:58:51.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:51.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:51.459834+0000) 2024-03-06T13:58:52.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:52.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:52.460072+0000) 2024-03-06T13:58:53.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:53.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:53.460329+0000) 2024-03-06T13:58:54.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:54.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:54.460583+0000) 2024-03-06T13:58:55.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:55.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:55.460808+0000) 2024-03-06T13:58:56.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:56.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:56.461071+0000) 2024-03-06T13:58:57.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:57.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:57.461314+0000) 2024-03-06T13:58:58.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:58.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:58.461572+0000) 2024-03-06T13:58:59.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:58:59.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:59.461807+0000) 2024-03-06T13:59:00.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:00.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:00.462016+0000) 2024-03-06T13:59:01.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:01.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:01.462266+0000) 2024-03-06T13:59:02.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:02.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:02.462518+0000) 2024-03-06T13:59:03.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:03.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:03.462773+0000) 2024-03-06T13:59:04.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:04.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:04.463030+0000) 2024-03-06T13:59:05.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:05.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:05.463217+0000) 2024-03-06T13:59:06.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:06.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:06.463475+0000) 2024-03-06T13:59:07.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:07.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:07.463756+0000) 2024-03-06T13:59:08.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:08.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:08.464012+0000) 2024-03-06T13:59:09.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:09.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:09.464266+0000) 2024-03-06T13:59:10.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:10.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:10.464521+0000) 2024-03-06T13:59:11.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:11.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:11.464792+0000) 2024-03-06T13:59:12.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:12.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:12.465046+0000) 2024-03-06T13:59:13.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:13.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:13.465314+0000) 2024-03-06T13:59:14.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:14.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:14.465567+0000) 2024-03-06T13:59:15.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:15.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:15.465809+0000) 2024-03-06T13:59:16.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:16.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:16.466049+0000) 2024-03-06T13:59:16.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:16.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:17.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:17.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:17.466299+0000) 2024-03-06T13:59:18.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:18.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:18.466555+0000) 2024-03-06T13:59:19.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:19.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:19.466790+0000) 2024-03-06T13:59:20.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:20.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:20.467065+0000) 2024-03-06T13:59:21.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:21.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:21.467316+0000) 2024-03-06T13:59:22.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:22.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:22.467560+0000) 2024-03-06T13:59:23.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:23.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:23.467808+0000) 2024-03-06T13:59:24.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:24.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:24.468063+0000) 2024-03-06T13:59:25.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:25.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:25.468346+0000) 2024-03-06T13:59:26.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:26.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:26.468606+0000) 2024-03-06T13:59:27.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:27.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:27.468918+0000) 2024-03-06T13:59:28.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:28.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:28.469139+0000) 2024-03-06T13:59:29.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:29.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:29.469397+0000) 2024-03-06T13:59:30.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:30.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:30.469714+0000) 2024-03-06T13:59:31.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:31.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:31.469980+0000) 2024-03-06T13:59:32.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:32.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:32.470257+0000) 2024-03-06T13:59:33.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:33.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:33.470516+0000) 2024-03-06T13:59:34.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:34.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:34.470847+0000) 2024-03-06T13:59:35.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:35.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:35.471102+0000) 2024-03-06T13:59:36.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:36.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:36.471389+0000) 2024-03-06T13:59:37.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:37.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:37.471699+0000) 2024-03-06T13:59:38.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:38.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:38.471940+0000) 2024-03-06T13:59:39.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:39.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:39.472187+0000) 2024-03-06T13:59:40.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:40.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:40.472452+0000) 2024-03-06T13:59:41.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:41.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:41.472749+0000) 2024-03-06T13:59:42.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:42.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:42.473005+0000) 2024-03-06T13:59:43.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:43.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:43.473262+0000) 2024-03-06T13:59:44.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:44.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:44.473521+0000) 2024-03-06T13:59:45.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:45.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:45.473781+0000) 2024-03-06T13:59:46.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:46.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:46.474026+0000) 2024-03-06T13:59:46.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:46.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:47.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:47.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:47.474278+0000) 2024-03-06T13:59:48.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:48.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:48.474479+0000) 2024-03-06T13:59:49.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:49.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:49.474761+0000) 2024-03-06T13:59:50.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:50.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:50.475047+0000) 2024-03-06T13:59:51.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:51.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:51.475309+0000) 2024-03-06T13:59:52.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:52.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:52.475525+0000) 2024-03-06T13:59:53.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:53.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:53.475806+0000) 2024-03-06T13:59:54.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:54.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:54.476079+0000) 2024-03-06T13:59:55.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:55.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:55.476287+0000) 2024-03-06T13:59:56.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:56.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:56.476553+0000) 2024-03-06T13:59:57.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:57.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:57.476812+0000) 2024-03-06T13:59:58.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:58.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:58.477086+0000) 2024-03-06T13:59:59.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T13:59:59.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:59.477349+0000) 2024-03-06T14:00:00.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:00.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:00.477663+0000) 2024-03-06T14:00:01.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:01.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:01.477908+0000) 2024-03-06T14:00:02.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:02.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:02.478164+0000) 2024-03-06T14:00:03.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:03.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:03.478405+0000) 2024-03-06T14:00:04.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:04.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:04.478684+0000) 2024-03-06T14:00:05.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:05.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:05.478881+0000) 2024-03-06T14:00:06.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:06.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:06.479139+0000) 2024-03-06T14:00:07.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:07.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:07.479403+0000) 2024-03-06T14:00:08.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:08.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:08.479705+0000) 2024-03-06T14:00:09.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:09.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:09.479965+0000) 2024-03-06T14:00:10.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:10.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:10.480254+0000) 2024-03-06T14:00:11.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:11.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:11.480536+0000) 2024-03-06T14:00:12.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:12.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:12.480814+0000) 2024-03-06T14:00:13.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:13.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:13.481088+0000) 2024-03-06T14:00:14.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:14.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:14.481345+0000) 2024-03-06T14:00:15.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:15.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:15.481666+0000) 2024-03-06T14:00:16.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:16.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:16.481892+0000) 2024-03-06T14:00:16.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:16.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:17.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:17.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:17.482194+0000) 2024-03-06T14:00:18.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:18.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:18.482457+0000) 2024-03-06T14:00:19.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:19.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:19.482736+0000) 2024-03-06T14:00:20.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:20.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:20.483034+0000) 2024-03-06T14:00:21.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:21.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:21.483293+0000) 2024-03-06T14:00:22.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:22.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:22.483604+0000) 2024-03-06T14:00:23.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:23.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:23.483939+0000) 2024-03-06T14:00:24.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:24.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:24.484182+0000) 2024-03-06T14:00:25.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:25.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:25.484424+0000) 2024-03-06T14:00:26.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:26.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:26.484718+0000) 2024-03-06T14:00:27.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:27.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:27.484980+0000) 2024-03-06T14:00:28.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:28.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:28.485241+0000) 2024-03-06T14:00:29.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:29.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:29.485492+0000) 2024-03-06T14:00:30.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:30.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:30.485772+0000) 2024-03-06T14:00:31.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:31.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:31.486030+0000) 2024-03-06T14:00:32.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:32.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:32.486277+0000) 2024-03-06T14:00:33.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:33.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:33.486592+0000) 2024-03-06T14:00:34.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:34.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:34.486855+0000) 2024-03-06T14:00:35.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:35.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:35.487117+0000) 2024-03-06T14:00:36.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:36.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:36.487297+0000) 2024-03-06T14:00:37.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:37.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:37.487550+0000) 2024-03-06T14:00:38.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:38.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:38.487792+0000) 2024-03-06T14:00:39.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:39.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:39.488045+0000) 2024-03-06T14:00:40.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:40.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:40.488263+0000) 2024-03-06T14:00:41.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:41.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:41.488520+0000) 2024-03-06T14:00:42.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:42.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:42.488781+0000) 2024-03-06T14:00:43.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:43.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:43.489045+0000) 2024-03-06T14:00:44.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:44.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:44.489298+0000) 2024-03-06T14:00:45.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:45.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:45.489564+0000) 2024-03-06T14:00:46.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:46.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:46.489819+0000) 2024-03-06T14:00:46.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:46.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:47.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:47.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:47.490086+0000) 2024-03-06T14:00:48.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:48.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:48.490339+0000) 2024-03-06T14:00:49.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:49.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:49.490602+0000) 2024-03-06T14:00:50.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:50.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:50.490855+0000) 2024-03-06T14:00:51.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:51.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:51.491093+0000) 2024-03-06T14:00:52.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:52.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:52.491368+0000) 2024-03-06T14:00:53.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:53.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:53.491625+0000) 2024-03-06T14:00:54.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:54.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:54.491957+0000) 2024-03-06T14:00:55.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:55.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:55.492237+0000) 2024-03-06T14:00:56.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:56.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:56.492509+0000) 2024-03-06T14:00:57.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:57.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:57.492789+0000) 2024-03-06T14:00:58.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:58.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:58.493048+0000) 2024-03-06T14:00:59.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:00:59.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:59.493309+0000) 2024-03-06T14:01:00.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:00.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:00.493550+0000) 2024-03-06T14:01:01.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:01.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:01.493834+0000) 2024-03-06T14:01:02.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:02.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:02.494083+0000) 2024-03-06T14:01:03.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:03.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:03.494363+0000) 2024-03-06T14:01:04.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:04.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:04.494622+0000) 2024-03-06T14:01:05.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:05.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:05.494964+0000) 2024-03-06T14:01:06.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:06.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:06.495244+0000) 2024-03-06T14:01:07.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:07.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:07.495528+0000) 2024-03-06T14:01:08.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:08.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:08.495785+0000) 2024-03-06T14:01:09.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:09.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:09.496040+0000) 2024-03-06T14:01:10.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:10.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:10.496340+0000) 2024-03-06T14:01:11.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:11.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:11.496624+0000) 2024-03-06T14:01:12.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:12.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:12.496940+0000) 2024-03-06T14:01:13.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:13.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:13.497219+0000) 2024-03-06T14:01:14.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:14.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:14.497495+0000) 2024-03-06T14:01:15.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:15.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:15.497969+0000) 2024-03-06T14:01:16.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:16.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:16.498238+0000) 2024-03-06T14:01:17.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:17.095 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:17.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:17.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:17.498521+0000) 2024-03-06T14:01:18.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:18.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:18.498786+0000) 2024-03-06T14:01:19.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:19.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:19.498969+0000) 2024-03-06T14:01:20.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:20.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:20.499187+0000) 2024-03-06T14:01:21.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:21.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:21.499438+0000) 2024-03-06T14:01:22.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:22.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:22.499743+0000) 2024-03-06T14:01:23.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:23.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:23.500000+0000) 2024-03-06T14:01:24.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:24.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:24.500237+0000) 2024-03-06T14:01:25.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:25.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:25.500502+0000) 2024-03-06T14:01:26.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:26.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:26.500759+0000) 2024-03-06T14:01:27.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:27.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:27.500998+0000) 2024-03-06T14:01:28.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:28.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:28.501241+0000) 2024-03-06T14:01:29.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:29.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:29.501501+0000) 2024-03-06T14:01:30.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:30.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:30.501786+0000) 2024-03-06T14:01:31.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:31.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:31.502022+0000) 2024-03-06T14:01:32.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:32.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:32.502272+0000) 2024-03-06T14:01:33.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:33.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:33.502527+0000) 2024-03-06T14:01:34.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:34.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:34.502782+0000) 2024-03-06T14:01:35.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:35.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:35.502988+0000) 2024-03-06T14:01:36.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:36.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:36.503249+0000) 2024-03-06T14:01:37.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:37.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:37.503511+0000) 2024-03-06T14:01:38.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:38.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:38.503767+0000) 2024-03-06T14:01:39.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:39.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:39.503950+0000) 2024-03-06T14:01:40.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:40.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:40.504217+0000) 2024-03-06T14:01:41.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:41.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:41.504490+0000) 2024-03-06T14:01:42.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:42.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:42.504782+0000) 2024-03-06T14:01:43.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:43.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:43.505063+0000) 2024-03-06T14:01:44.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:44.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:44.505275+0000) 2024-03-06T14:01:45.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:45.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:45.505542+0000) 2024-03-06T14:01:46.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:46.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:46.505795+0000) 2024-03-06T14:01:47.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:47.238 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:47.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:47.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:47.506066+0000) 2024-03-06T14:01:48.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:48.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:48.506273+0000) 2024-03-06T14:01:49.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:49.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:49.506527+0000) 2024-03-06T14:01:50.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:50.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:50.506816+0000) 2024-03-06T14:01:51.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:51.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:51.507079+0000) 2024-03-06T14:01:52.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:52.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:52.507361+0000) 2024-03-06T14:01:53.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:53.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:53.507672+0000) 2024-03-06T14:01:54.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:54.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:54.507931+0000) 2024-03-06T14:01:55.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:55.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:55.508213+0000) 2024-03-06T14:01:56.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:56.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:56.508487+0000) 2024-03-06T14:01:57.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:57.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:57.508773+0000) 2024-03-06T14:01:58.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:58.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:58.509052+0000) 2024-03-06T14:01:59.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:01:59.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:59.509331+0000) 2024-03-06T14:02:00.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:00.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:00.509670+0000) 2024-03-06T14:02:01.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:01.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:01.509921+0000) 2024-03-06T14:02:02.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:02.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:02.510200+0000) 2024-03-06T14:02:03.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:03.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:03.510523+0000) 2024-03-06T14:02:04.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:04.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:04.510785+0000) 2024-03-06T14:02:05.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:05.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:05.511051+0000) 2024-03-06T14:02:06.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:06.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:06.511313+0000) 2024-03-06T14:02:07.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:07.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:07.511557+0000) 2024-03-06T14:02:08.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:08.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:08.511832+0000) 2024-03-06T14:02:09.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:09.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:09.512095+0000) 2024-03-06T14:02:10.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:10.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:10.512394+0000) 2024-03-06T14:02:11.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:11.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:11.512673+0000) 2024-03-06T14:02:12.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:12.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:12.512960+0000) 2024-03-06T14:02:13.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:13.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:13.513237+0000) 2024-03-06T14:02:14.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:14.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:14.513509+0000) 2024-03-06T14:02:15.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:15.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:15.513798+0000) 2024-03-06T14:02:16.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:16.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:16.514056+0000) 2024-03-06T14:02:17.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:17.377 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:17.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:17.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:17.514271+0000) 2024-03-06T14:02:18.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:18.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:18.514540+0000) 2024-03-06T14:02:19.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:19.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:19.514797+0000) 2024-03-06T14:02:20.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:20.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:20.515097+0000) 2024-03-06T14:02:21.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:21.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:21.515376+0000) 2024-03-06T14:02:22.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:22.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:22.515669+0000) 2024-03-06T14:02:23.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:23.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:23.515950+0000) 2024-03-06T14:02:24.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:24.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:24.516224+0000) 2024-03-06T14:02:25.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:25.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:25.516486+0000) 2024-03-06T14:02:26.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:26.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:26.516769+0000) 2024-03-06T14:02:27.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:27.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:27.517032+0000) 2024-03-06T14:02:28.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:28.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:28.517308+0000) 2024-03-06T14:02:29.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:29.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:29.517564+0000) 2024-03-06T14:02:30.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:30.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:30.517843+0000) 2024-03-06T14:02:31.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:31.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:31.518119+0000) 2024-03-06T14:02:32.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:32.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:32.518398+0000) 2024-03-06T14:02:33.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:33.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:33.518705+0000) 2024-03-06T14:02:34.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:34.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:34.518980+0000) 2024-03-06T14:02:35.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:35.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:35.519252+0000) 2024-03-06T14:02:36.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:36.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:36.519499+0000) 2024-03-06T14:02:37.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:37.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:37.519756+0000) 2024-03-06T14:02:38.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:38.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:38.520019+0000) 2024-03-06T14:02:39.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:39.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:39.520278+0000) 2024-03-06T14:02:40.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:40.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:40.520565+0000) 2024-03-06T14:02:41.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:41.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:41.520828+0000) 2024-03-06T14:02:42.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:42.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:42.521099+0000) 2024-03-06T14:02:43.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:43.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:43.521363+0000) 2024-03-06T14:02:44.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:44.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:44.521617+0000) 2024-03-06T14:02:45.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:45.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:45.521850+0000) 2024-03-06T14:02:46.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:46.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:46.522127+0000) 2024-03-06T14:02:47.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:47.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:47.522400+0000) 2024-03-06T14:02:47.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:47.532 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:48.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:48.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:48.522720+0000) 2024-03-06T14:02:49.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:49.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:49.522980+0000) 2024-03-06T14:02:50.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:50.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:50.523273+0000) 2024-03-06T14:02:51.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:51.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:51.523534+0000) 2024-03-06T14:02:52.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:52.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:52.523811+0000) 2024-03-06T14:02:53.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:53.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:53.524065+0000) 2024-03-06T14:02:54.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:54.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:54.524341+0000) 2024-03-06T14:02:55.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:55.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:55.524599+0000) 2024-03-06T14:02:56.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:56.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:56.524940+0000) 2024-03-06T14:02:57.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:57.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:57.525210+0000) 2024-03-06T14:02:58.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:58.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:58.525497+0000) 2024-03-06T14:02:59.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:02:59.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:59.525750+0000) 2024-03-06T14:03:00.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:00.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:00.526047+0000) 2024-03-06T14:03:01.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:01.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:01.526351+0000) 2024-03-06T14:03:02.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:02.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:02.526563+0000) 2024-03-06T14:03:03.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:03.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:03.526814+0000) 2024-03-06T14:03:04.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:04.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:04.527086+0000) 2024-03-06T14:03:05.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:05.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:05.527308+0000) 2024-03-06T14:03:06.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:06.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:06.527556+0000) 2024-03-06T14:03:07.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:07.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:07.528068+0000) 2024-03-06T14:03:08.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:08.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:08.528319+0000) 2024-03-06T14:03:09.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:09.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:09.528552+0000) 2024-03-06T14:03:10.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:10.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:10.528800+0000) 2024-03-06T14:03:11.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:11.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:11.529040+0000) 2024-03-06T14:03:12.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:12.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:12.529284+0000) 2024-03-06T14:03:13.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:13.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:13.529539+0000) 2024-03-06T14:03:14.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:14.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:14.529775+0000) 2024-03-06T14:03:15.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:15.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:15.530026+0000) 2024-03-06T14:03:16.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:16.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:16.530280+0000) 2024-03-06T14:03:17.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:17.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:17.530521+0000) 2024-03-06T14:03:17.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:17.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:18.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:18.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:18.530776+0000) 2024-03-06T14:03:19.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:19.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:19.531032+0000) 2024-03-06T14:03:20.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:20.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:20.531299+0000) 2024-03-06T14:03:21.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:21.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:21.531541+0000) 2024-03-06T14:03:22.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:22.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:22.531780+0000) 2024-03-06T14:03:23.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:23.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:23.532046+0000) 2024-03-06T14:03:24.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:24.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:24.532240+0000) 2024-03-06T14:03:25.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:25.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:25.532495+0000) 2024-03-06T14:03:26.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:26.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:26.532751+0000) 2024-03-06T14:03:27.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:27.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:27.533008+0000) 2024-03-06T14:03:28.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:28.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:28.533266+0000) 2024-03-06T14:03:29.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:29.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:29.533528+0000) 2024-03-06T14:03:30.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:30.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:30.533780+0000) 2024-03-06T14:03:31.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:31.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:31.534019+0000) 2024-03-06T14:03:32.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:32.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:32.534258+0000) 2024-03-06T14:03:33.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:33.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:33.534528+0000) 2024-03-06T14:03:34.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:34.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:34.534788+0000) 2024-03-06T14:03:35.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:35.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:35.535047+0000) 2024-03-06T14:03:36.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:36.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:36.535285+0000) 2024-03-06T14:03:37.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:37.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:37.535546+0000) 2024-03-06T14:03:38.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:38.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:38.535805+0000) 2024-03-06T14:03:39.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:39.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:39.536052+0000) 2024-03-06T14:03:40.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:40.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:40.536290+0000) 2024-03-06T14:03:41.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:41.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:41.536580+0000) 2024-03-06T14:03:42.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:42.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:42.536894+0000) 2024-03-06T14:03:43.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:43.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:43.537203+0000) 2024-03-06T14:03:44.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:44.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:44.537457+0000) 2024-03-06T14:03:45.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:45.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:45.537753+0000) 2024-03-06T14:03:46.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:46.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:46.537994+0000) 2024-03-06T14:03:47.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:47.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:47.538249+0000) 2024-03-06T14:03:47.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:47.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:48.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:48.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:48.538504+0000) 2024-03-06T14:03:49.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:49.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:49.538759+0000) 2024-03-06T14:03:50.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:50.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:50.539028+0000) 2024-03-06T14:03:51.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:51.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:51.539278+0000) 2024-03-06T14:03:52.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:52.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:52.539538+0000) 2024-03-06T14:03:53.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:53.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:53.539782+0000) 2024-03-06T14:03:54.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:54.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:54.540034+0000) 2024-03-06T14:03:55.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:55.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:55.540286+0000) 2024-03-06T14:03:56.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:56.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:56.540542+0000) 2024-03-06T14:03:57.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:57.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:57.540793+0000) 2024-03-06T14:03:58.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:58.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:58.541052+0000) 2024-03-06T14:03:59.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:03:59.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:59.541302+0000) 2024-03-06T14:04:00.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:00.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:00.541564+0000) 2024-03-06T14:04:01.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:01.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:01.541804+0000) 2024-03-06T14:04:02.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:02.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:02.542060+0000) 2024-03-06T14:04:03.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:03.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:03.542326+0000) 2024-03-06T14:04:04.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:04.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:04.542574+0000) 2024-03-06T14:04:05.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:05.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:05.542807+0000) 2024-03-06T14:04:06.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:06.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:06.543069+0000) 2024-03-06T14:04:07.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:07.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:07.543252+0000) 2024-03-06T14:04:08.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:08.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:08.543496+0000) 2024-03-06T14:04:09.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:09.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:09.543736+0000) 2024-03-06T14:04:10.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:10.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:10.543997+0000) 2024-03-06T14:04:11.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:11.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:11.544248+0000) 2024-03-06T14:04:12.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:12.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:12.544493+0000) 2024-03-06T14:04:13.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:13.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:13.544759+0000) 2024-03-06T14:04:14.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:14.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:14.544957+0000) 2024-03-06T14:04:15.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:15.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:15.545208+0000) 2024-03-06T14:04:16.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:16.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:16.545446+0000) 2024-03-06T14:04:17.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:17.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:17.545739+0000) 2024-03-06T14:04:17.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:17.965 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:18.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:18.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:18.545982+0000) 2024-03-06T14:04:19.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:19.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:19.546242+0000) 2024-03-06T14:04:20.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:20.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:20.546510+0000) 2024-03-06T14:04:21.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:21.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:21.546763+0000) 2024-03-06T14:04:22.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:22.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:22.547014+0000) 2024-03-06T14:04:23.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:23.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:23.547279+0000) 2024-03-06T14:04:24.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:24.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:24.547535+0000) 2024-03-06T14:04:25.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:25.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:25.547793+0000) 2024-03-06T14:04:26.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:26.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:26.548031+0000) 2024-03-06T14:04:27.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:27.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:27.548223+0000) 2024-03-06T14:04:28.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:28.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:28.548466+0000) 2024-03-06T14:04:29.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:29.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:29.548758+0000) 2024-03-06T14:04:30.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:30.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:30.549029+0000) 2024-03-06T14:04:31.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:31.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:31.549275+0000) 2024-03-06T14:04:32.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:32.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:32.549512+0000) 2024-03-06T14:04:33.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:33.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:33.549784+0000) 2024-03-06T14:04:34.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:34.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:34.550020+0000) 2024-03-06T14:04:35.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:35.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:35.550282+0000) 2024-03-06T14:04:36.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:36.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:36.550538+0000) 2024-03-06T14:04:37.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:37.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:37.550795+0000) 2024-03-06T14:04:38.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:38.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:38.551103+0000) 2024-03-06T14:04:39.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:39.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:39.551344+0000) 2024-03-06T14:04:40.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:40.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:40.551619+0000) 2024-03-06T14:04:41.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:41.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:41.551907+0000) 2024-03-06T14:04:42.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:42.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:42.552162+0000) 2024-03-06T14:04:43.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:43.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:43.552426+0000) 2024-03-06T14:04:44.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:44.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:44.552703+0000) 2024-03-06T14:04:45.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:45.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:45.552957+0000) 2024-03-06T14:04:46.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:46.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:46.553213+0000) 2024-03-06T14:04:47.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:47.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:47.553475+0000) 2024-03-06T14:04:48.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:48.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:48.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:48.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:48.553752+0000) 2024-03-06T14:04:49.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:49.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:49.553944+0000) 2024-03-06T14:04:50.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:50.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:50.554210+0000) 2024-03-06T14:04:51.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:51.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:51.554473+0000) 2024-03-06T14:04:52.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:52.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:52.554754+0000) 2024-03-06T14:04:53.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:53.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:53.554997+0000) 2024-03-06T14:04:54.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:54.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:54.555264+0000) 2024-03-06T14:04:55.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:55.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:55.555526+0000) 2024-03-06T14:04:56.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:56.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:56.555770+0000) 2024-03-06T14:04:57.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:57.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:57.556002+0000) 2024-03-06T14:04:58.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:58.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:58.556205+0000) 2024-03-06T14:04:59.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:04:59.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:59.556466+0000) 2024-03-06T14:05:00.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:00.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:00.556734+0000) 2024-03-06T14:05:01.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:01.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:01.556995+0000) 2024-03-06T14:05:02.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:02.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:02.557190+0000) 2024-03-06T14:05:03.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:03.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:03.557448+0000) 2024-03-06T14:05:04.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:04.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:04.557744+0000) 2024-03-06T14:05:05.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:05.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:05.558003+0000) 2024-03-06T14:05:06.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:06.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:06.558268+0000) 2024-03-06T14:05:07.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:07.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:07.558502+0000) 2024-03-06T14:05:08.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:08.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:08.558738+0000) 2024-03-06T14:05:09.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:09.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:09.558991+0000) 2024-03-06T14:05:10.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:10.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:10.559255+0000) 2024-03-06T14:05:11.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:11.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:11.559534+0000) 2024-03-06T14:05:12.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:12.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:12.559795+0000) 2024-03-06T14:05:13.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:13.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:13.560032+0000) 2024-03-06T14:05:14.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:14.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:14.560286+0000) 2024-03-06T14:05:15.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:15.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:15.560544+0000) 2024-03-06T14:05:16.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:16.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:16.560809+0000) 2024-03-06T14:05:17.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:17.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:17.561066+0000) 2024-03-06T14:05:18.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:18.258 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:18.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:18.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:18.561339+0000) 2024-03-06T14:05:19.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:19.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:19.561580+0000) 2024-03-06T14:05:20.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:20.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:20.561856+0000) 2024-03-06T14:05:21.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:21.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:21.562104+0000) 2024-03-06T14:05:22.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:22.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:22.562344+0000) 2024-03-06T14:05:23.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:23.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:23.562604+0000) 2024-03-06T14:05:24.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:24.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:24.562892+0000) 2024-03-06T14:05:25.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:25.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:25.563076+0000) 2024-03-06T14:05:26.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:26.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:26.563322+0000) 2024-03-06T14:05:27.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:27.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:27.563583+0000) 2024-03-06T14:05:28.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:28.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:28.563850+0000) 2024-03-06T14:05:29.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:29.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:29.564106+0000) 2024-03-06T14:05:30.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:30.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:30.564373+0000) 2024-03-06T14:05:31.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:31.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:31.564678+0000) 2024-03-06T14:05:32.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:32.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:32.564941+0000) 2024-03-06T14:05:33.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:33.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:33.565184+0000) 2024-03-06T14:05:34.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:34.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:34.565436+0000) 2024-03-06T14:05:35.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:35.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:35.565736+0000) 2024-03-06T14:05:36.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:36.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:36.565990+0000) 2024-03-06T14:05:37.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:37.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:37.566249+0000) 2024-03-06T14:05:38.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:38.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:38.566518+0000) 2024-03-06T14:05:39.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:39.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:39.566772+0000) 2024-03-06T14:05:40.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:40.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:40.567036+0000) 2024-03-06T14:05:41.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:41.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:41.567285+0000) 2024-03-06T14:05:42.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:42.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:42.567529+0000) 2024-03-06T14:05:43.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:43.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:43.567792+0000) 2024-03-06T14:05:44.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:44.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:44.568031+0000) 2024-03-06T14:05:45.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:45.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:45.568235+0000) 2024-03-06T14:05:46.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:46.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:46.568497+0000) 2024-03-06T14:05:47.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:47.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:47.568754+0000) 2024-03-06T14:05:48.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:48.417 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:48.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:48.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:48.568974+0000) 2024-03-06T14:05:49.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:49.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:49.569229+0000) 2024-03-06T14:05:50.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:50.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:50.569424+0000) 2024-03-06T14:05:51.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:51.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:51.569719+0000) 2024-03-06T14:05:52.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:52.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:52.569986+0000) 2024-03-06T14:05:53.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:53.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:53.570235+0000) 2024-03-06T14:05:54.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:54.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:54.570501+0000) 2024-03-06T14:05:55.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:55.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:55.570751+0000) 2024-03-06T14:05:56.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:56.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:56.571024+0000) 2024-03-06T14:05:57.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:57.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:57.571259+0000) 2024-03-06T14:05:58.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:58.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:58.571532+0000) 2024-03-06T14:05:59.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:05:59.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:59.571769+0000) 2024-03-06T14:06:00.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:00.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:00.571973+0000) 2024-03-06T14:06:01.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:01.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:01.572224+0000) 2024-03-06T14:06:02.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:02.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:02.572483+0000) 2024-03-06T14:06:03.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:03.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:03.572766+0000) 2024-03-06T14:06:04.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:04.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:04.573021+0000) 2024-03-06T14:06:05.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:05.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:05.573254+0000) 2024-03-06T14:06:06.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:06.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:06.573449+0000) 2024-03-06T14:06:07.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:07.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:07.573737+0000) 2024-03-06T14:06:08.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:08.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:08.574000+0000) 2024-03-06T14:06:09.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:09.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:09.574241+0000) 2024-03-06T14:06:10.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:10.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:10.574497+0000) 2024-03-06T14:06:11.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:11.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:11.574768+0000) 2024-03-06T14:06:12.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:12.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:12.575023+0000) 2024-03-06T14:06:13.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:13.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:13.575277+0000) 2024-03-06T14:06:14.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:14.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:14.575591+0000) 2024-03-06T14:06:15.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:15.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:15.575858+0000) 2024-03-06T14:06:16.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:16.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:16.576066+0000) 2024-03-06T14:06:17.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:17.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:17.576304+0000) 2024-03-06T14:06:18.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:18.571 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:18.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:18.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:18.576540+0000) 2024-03-06T14:06:19.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:19.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:19.576808+0000) 2024-03-06T14:06:20.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:20.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:20.577066+0000) 2024-03-06T14:06:21.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:21.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:21.577329+0000) 2024-03-06T14:06:22.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:22.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:22.577585+0000) 2024-03-06T14:06:23.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:23.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:23.577852+0000) 2024-03-06T14:06:24.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:24.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:24.578093+0000) 2024-03-06T14:06:25.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:25.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:25.578356+0000) 2024-03-06T14:06:26.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:26.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:26.578624+0000) 2024-03-06T14:06:27.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:27.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:27.578931+0000) 2024-03-06T14:06:28.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:28.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:28.579137+0000) 2024-03-06T14:06:29.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:29.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:29.579373+0000) 2024-03-06T14:06:30.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:30.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:30.579664+0000) 2024-03-06T14:06:31.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:31.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:31.579938+0000) 2024-03-06T14:06:32.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:32.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:32.580203+0000) 2024-03-06T14:06:33.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:33.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:33.580460+0000) 2024-03-06T14:06:34.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:34.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:34.580683+0000) 2024-03-06T14:06:35.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:35.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:35.580873+0000) 2024-03-06T14:06:36.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:36.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:36.581129+0000) 2024-03-06T14:06:37.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:37.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:37.581377+0000) 2024-03-06T14:06:38.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:38.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:38.581621+0000) 2024-03-06T14:06:39.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:39.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:39.581906+0000) 2024-03-06T14:06:40.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:40.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:40.582177+0000) 2024-03-06T14:06:41.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:41.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:41.582429+0000) 2024-03-06T14:06:42.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:42.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:42.582706+0000) 2024-03-06T14:06:43.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:43.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:43.582950+0000) 2024-03-06T14:06:44.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:44.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:44.583207+0000) 2024-03-06T14:06:45.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:45.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:45.583453+0000) 2024-03-06T14:06:46.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:46.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:46.583758+0000) 2024-03-06T14:06:47.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:47.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:47.584021+0000) 2024-03-06T14:06:48.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:48.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:48.584284+0000) 2024-03-06T14:06:48.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:48.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:49.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:49.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:49.584526+0000) 2024-03-06T14:06:50.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:50.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:50.584794+0000) 2024-03-06T14:06:51.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:51.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:51.585037+0000) 2024-03-06T14:06:52.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:52.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:52.585290+0000) 2024-03-06T14:06:53.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:53.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:53.585555+0000) 2024-03-06T14:06:54.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:54.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:54.585812+0000) 2024-03-06T14:06:55.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:55.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:55.586051+0000) 2024-03-06T14:06:56.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:56.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:56.586298+0000) 2024-03-06T14:06:57.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:57.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:57.586550+0000) 2024-03-06T14:06:58.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:58.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:58.586820+0000) 2024-03-06T14:06:59.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:06:59.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:59.587081+0000) 2024-03-06T14:07:00.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:00.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:00.587346+0000) 2024-03-06T14:07:01.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:01.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:01.587581+0000) 2024-03-06T14:07:02.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:02.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:02.587894+0000) 2024-03-06T14:07:03.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:03.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:03.588141+0000) 2024-03-06T14:07:04.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:04.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:04.588380+0000) 2024-03-06T14:07:05.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:05.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:05.588655+0000) 2024-03-06T14:07:06.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:06.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:06.588913+0000) 2024-03-06T14:07:07.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:07.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:07.589146+0000) 2024-03-06T14:07:08.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:08.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:08.589392+0000) 2024-03-06T14:07:09.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:09.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:09.589696+0000) 2024-03-06T14:07:10.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:10.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:10.589966+0000) 2024-03-06T14:07:11.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:11.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:11.590156+0000) 2024-03-06T14:07:12.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:12.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:12.590419+0000) 2024-03-06T14:07:13.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:13.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:13.590722+0000) 2024-03-06T14:07:14.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:14.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:14.590962+0000) 2024-03-06T14:07:15.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:15.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:15.591223+0000) 2024-03-06T14:07:16.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:16.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:16.591455+0000) 2024-03-06T14:07:17.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:17.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:17.591788+0000) 2024-03-06T14:07:18.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:18.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:18.592090+0000) 2024-03-06T14:07:18.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:18.865 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:19.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:19.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:19.592397+0000) 2024-03-06T14:07:20.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:20.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:20.592773+0000) 2024-03-06T14:07:21.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:21.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:21.593091+0000) 2024-03-06T14:07:22.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:22.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:22.593396+0000) 2024-03-06T14:07:23.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:23.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:23.593750+0000) 2024-03-06T14:07:24.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:24.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:24.594034+0000) 2024-03-06T14:07:25.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:25.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:25.594340+0000) 2024-03-06T14:07:26.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:26.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:26.594675+0000) 2024-03-06T14:07:27.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:27.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:27.595002+0000) 2024-03-06T14:07:28.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:28.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:28.595315+0000) 2024-03-06T14:07:29.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:29.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:29.595598+0000) 2024-03-06T14:07:30.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:30.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:30.595968+0000) 2024-03-06T14:07:31.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:31.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:31.596277+0000) 2024-03-06T14:07:32.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:32.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:32.596573+0000) 2024-03-06T14:07:33.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:33.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:33.596878+0000) 2024-03-06T14:07:34.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:34.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:34.597169+0000) 2024-03-06T14:07:35.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:35.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:35.597461+0000) 2024-03-06T14:07:36.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:36.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:36.597799+0000) 2024-03-06T14:07:37.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:37.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:37.598104+0000) 2024-03-06T14:07:38.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:38.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:38.598397+0000) 2024-03-06T14:07:39.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:39.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:39.598758+0000) 2024-03-06T14:07:40.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:40.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:40.599083+0000) 2024-03-06T14:07:41.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:41.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:41.599398+0000) 2024-03-06T14:07:42.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:42.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:42.599758+0000) 2024-03-06T14:07:43.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:43.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:43.600060+0000) 2024-03-06T14:07:44.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:44.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:44.600366+0000) 2024-03-06T14:07:45.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:45.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:45.600703+0000) 2024-03-06T14:07:46.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:46.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:46.601033+0000) 2024-03-06T14:07:47.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:47.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:47.601345+0000) 2024-03-06T14:07:48.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:48.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:48.601657+0000) 2024-03-06T14:07:49.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:49.036 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:49.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:49.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:49.601981+0000) 2024-03-06T14:07:50.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:50.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:50.602303+0000) 2024-03-06T14:07:51.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:51.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:51.602597+0000) 2024-03-06T14:07:52.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:52.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:52.602947+0000) 2024-03-06T14:07:53.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:53.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:53.603243+0000) 2024-03-06T14:07:54.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:54.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:54.603536+0000) 2024-03-06T14:07:55.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:55.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:55.603844+0000) 2024-03-06T14:07:56.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:56.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:56.604138+0000) 2024-03-06T14:07:57.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:57.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:57.604431+0000) 2024-03-06T14:07:58.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:58.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:58.604796+0000) 2024-03-06T14:07:59.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:07:59.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:59.605055+0000) 2024-03-06T14:08:00.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:00.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:00.605381+0000) 2024-03-06T14:08:01.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:01.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:01.605761+0000) 2024-03-06T14:08:02.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:02.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:02.606084+0000) 2024-03-06T14:08:03.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:03.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:03.606404+0000) 2024-03-06T14:08:04.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:04.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:04.606760+0000) 2024-03-06T14:08:05.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:05.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:05.607034+0000) 2024-03-06T14:08:06.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:06.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:06.607319+0000) 2024-03-06T14:08:07.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:07.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:07.607562+0000) 2024-03-06T14:08:08.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:08.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:08.607879+0000) 2024-03-06T14:08:09.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:09.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:09.608165+0000) 2024-03-06T14:08:10.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:10.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:10.608493+0000) 2024-03-06T14:08:11.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:11.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:11.608795+0000) 2024-03-06T14:08:12.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:12.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:12.609103+0000) 2024-03-06T14:08:13.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:13.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:13.609345+0000) 2024-03-06T14:08:14.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:14.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:14.609717+0000) 2024-03-06T14:08:15.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:15.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:15.610012+0000) 2024-03-06T14:08:16.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:16.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:16.610300+0000) 2024-03-06T14:08:17.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:17.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:17.610592+0000) 2024-03-06T14:08:18.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:18.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:18.610904+0000) 2024-03-06T14:08:19.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:19.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:19.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:19.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:19.611226+0000) 2024-03-06T14:08:20.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:20.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:20.611558+0000) 2024-03-06T14:08:21.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:21.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:21.611874+0000) 2024-03-06T14:08:22.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:22.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:22.612161+0000) 2024-03-06T14:08:23.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:23.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:23.612475+0000) 2024-03-06T14:08:24.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:24.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:24.612778+0000) 2024-03-06T14:08:25.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:25.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:25.613085+0000) 2024-03-06T14:08:26.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:26.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:26.613388+0000) 2024-03-06T14:08:27.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:27.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:27.613745+0000) 2024-03-06T14:08:28.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:28.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:28.613978+0000) 2024-03-06T14:08:29.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:29.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:29.614257+0000) 2024-03-06T14:08:30.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:30.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:30.614576+0000) 2024-03-06T14:08:31.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:31.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:31.614872+0000) 2024-03-06T14:08:32.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:32.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:32.615186+0000) 2024-03-06T14:08:33.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:33.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:33.615500+0000) 2024-03-06T14:08:34.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:34.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:34.615810+0000) 2024-03-06T14:08:35.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:35.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:35.616134+0000) 2024-03-06T14:08:36.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:36.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:36.616450+0000) 2024-03-06T14:08:37.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:37.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:37.616787+0000) 2024-03-06T14:08:38.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:38.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:38.617094+0000) 2024-03-06T14:08:39.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:39.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:39.617395+0000) 2024-03-06T14:08:40.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:40.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:40.617759+0000) 2024-03-06T14:08:41.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:41.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:41.618047+0000) 2024-03-06T14:08:42.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:42.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:42.618345+0000) 2024-03-06T14:08:43.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:43.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:43.618616+0000) 2024-03-06T14:08:44.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:44.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:44.618916+0000) 2024-03-06T14:08:45.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:45.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:45.619114+0000) 2024-03-06T14:08:46.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:46.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:46.619305+0000) 2024-03-06T14:08:47.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:47.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:47.619547+0000) 2024-03-06T14:08:48.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:48.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:48.619762+0000) 2024-03-06T14:08:49.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:49.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:49.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:49.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:49.620072+0000) 2024-03-06T14:08:50.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:50.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:50.620346+0000) 2024-03-06T14:08:51.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:51.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:51.620593+0000) 2024-03-06T14:08:52.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:52.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:52.620836+0000) 2024-03-06T14:08:53.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:53.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:53.621095+0000) 2024-03-06T14:08:54.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:54.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:54.621354+0000) 2024-03-06T14:08:55.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:55.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:55.621539+0000) 2024-03-06T14:08:56.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:56.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:56.621812+0000) 2024-03-06T14:08:57.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:57.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:57.622096+0000) 2024-03-06T14:08:58.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:58.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:58.622351+0000) 2024-03-06T14:08:59.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:08:59.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:59.622666+0000) 2024-03-06T14:09:00.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:00.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:00.622948+0000) 2024-03-06T14:09:01.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:01.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:01.623195+0000) 2024-03-06T14:09:02.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:02.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:02.623385+0000) 2024-03-06T14:09:03.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:03.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:03.623621+0000) 2024-03-06T14:09:04.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:04.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:04.623926+0000) 2024-03-06T14:09:05.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:05.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:05.624189+0000) 2024-03-06T14:09:06.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:06.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:06.624450+0000) 2024-03-06T14:09:07.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:07.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:07.624750+0000) 2024-03-06T14:09:08.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:08.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:08.624983+0000) 2024-03-06T14:09:09.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:09.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:09.625248+0000) 2024-03-06T14:09:10.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:10.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:10.625500+0000) 2024-03-06T14:09:11.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:11.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:11.625740+0000) 2024-03-06T14:09:12.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:12.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:12.625991+0000) 2024-03-06T14:09:13.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:13.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:13.626225+0000) 2024-03-06T14:09:14.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:14.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:14.626483+0000) 2024-03-06T14:09:15.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:15.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:15.626757+0000) 2024-03-06T14:09:16.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:16.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:16.627013+0000) 2024-03-06T14:09:17.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:17.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:17.627262+0000) 2024-03-06T14:09:18.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:18.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:18.627501+0000) 2024-03-06T14:09:19.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:19.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:19.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:19.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:19.627785+0000) 2024-03-06T14:09:20.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:20.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:20.628060+0000) 2024-03-06T14:09:21.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:21.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:21.628317+0000) 2024-03-06T14:09:22.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:22.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:22.628594+0000) 2024-03-06T14:09:23.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:23.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:23.628895+0000) 2024-03-06T14:09:24.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:24.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:24.629142+0000) 2024-03-06T14:09:25.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:25.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:25.629379+0000) 2024-03-06T14:09:26.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:26.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:26.629617+0000) 2024-03-06T14:09:27.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:27.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:27.629918+0000) 2024-03-06T14:09:28.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:28.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:28.630177+0000) 2024-03-06T14:09:29.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:29.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:29.630442+0000) 2024-03-06T14:09:30.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:30.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:30.630748+0000) 2024-03-06T14:09:31.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:31.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:31.631020+0000) 2024-03-06T14:09:32.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:32.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:32.631263+0000) 2024-03-06T14:09:33.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:33.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:33.631505+0000) 2024-03-06T14:09:34.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:34.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:34.631747+0000) 2024-03-06T14:09:35.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:35.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:35.632006+0000) 2024-03-06T14:09:36.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:36.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:36.632260+0000) 2024-03-06T14:09:37.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:37.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:37.632575+0000) 2024-03-06T14:09:38.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:38.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:38.632849+0000) 2024-03-06T14:09:39.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:39.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:39.633083+0000) 2024-03-06T14:09:40.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:40.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:40.633336+0000) 2024-03-06T14:09:41.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:41.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:41.633587+0000) 2024-03-06T14:09:42.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:42.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:42.633851+0000) 2024-03-06T14:09:43.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:43.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:43.634095+0000) 2024-03-06T14:09:44.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:44.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:44.634341+0000) 2024-03-06T14:09:45.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:45.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:45.634583+0000) 2024-03-06T14:09:46.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:46.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:46.634830+0000) 2024-03-06T14:09:47.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:47.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:47.635076+0000) 2024-03-06T14:09:48.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:48.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:48.635320+0000) 2024-03-06T14:09:49.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:49.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:49.635566+0000) 2024-03-06T14:09:49.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:49.660 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:50.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:50.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:50.635835+0000) 2024-03-06T14:09:51.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:51.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:51.636086+0000) 2024-03-06T14:09:52.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:52.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:52.636320+0000) 2024-03-06T14:09:53.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:53.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:53.636564+0000) 2024-03-06T14:09:54.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:54.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:54.636811+0000) 2024-03-06T14:09:55.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:55.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:55.637071+0000) 2024-03-06T14:09:56.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:56.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:56.637331+0000) 2024-03-06T14:09:57.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:57.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:57.637590+0000) 2024-03-06T14:09:58.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:58.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:58.637851+0000) 2024-03-06T14:09:59.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:09:59.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:59.638116+0000) 2024-03-06T14:10:00.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:00.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:00.638389+0000) 2024-03-06T14:10:01.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:01.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:01.638687+0000) 2024-03-06T14:10:02.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:02.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:02.638930+0000) 2024-03-06T14:10:03.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:03.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:03.639193+0000) 2024-03-06T14:10:04.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:04.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:04.639463+0000) 2024-03-06T14:10:05.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:05.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:05.639755+0000) 2024-03-06T14:10:06.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:06.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:06.640020+0000) 2024-03-06T14:10:07.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:07.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:07.640213+0000) 2024-03-06T14:10:08.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:08.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:08.640453+0000) 2024-03-06T14:10:09.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:09.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:09.640746+0000) 2024-03-06T14:10:10.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:10.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:10.641002+0000) 2024-03-06T14:10:11.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:11.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:11.641255+0000) 2024-03-06T14:10:12.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:12.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:12.641513+0000) 2024-03-06T14:10:13.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:13.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:13.641766+0000) 2024-03-06T14:10:14.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:14.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:14.642034+0000) 2024-03-06T14:10:15.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:15.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:15.642290+0000) 2024-03-06T14:10:16.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:16.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:16.642554+0000) 2024-03-06T14:10:17.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:17.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:17.642797+0000) 2024-03-06T14:10:18.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:18.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:18.643047+0000) 2024-03-06T14:10:19.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:19.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:19.643355+0000) 2024-03-06T14:10:19.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:19.806 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:20.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:20.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:20.643625+0000) 2024-03-06T14:10:21.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:21.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:21.643917+0000) 2024-03-06T14:10:22.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:22.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:22.644126+0000) 2024-03-06T14:10:23.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:23.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:23.644366+0000) 2024-03-06T14:10:24.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:24.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:24.644617+0000) 2024-03-06T14:10:25.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:25.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:25.644903+0000) 2024-03-06T14:10:26.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:26.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:26.645160+0000) 2024-03-06T14:10:27.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:27.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:27.645432+0000) 2024-03-06T14:10:28.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:28.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:28.645735+0000) 2024-03-06T14:10:29.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:29.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:29.645995+0000) 2024-03-06T14:10:30.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:30.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:30.646253+0000) 2024-03-06T14:10:31.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:31.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:31.646512+0000) 2024-03-06T14:10:32.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:32.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:32.646800+0000) 2024-03-06T14:10:33.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:33.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:33.647042+0000) 2024-03-06T14:10:34.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:34.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:34.647307+0000) 2024-03-06T14:10:35.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:35.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:35.647567+0000) 2024-03-06T14:10:36.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:36.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:36.647805+0000) 2024-03-06T14:10:37.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:37.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:37.648086+0000) 2024-03-06T14:10:38.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:38.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:38.648337+0000) 2024-03-06T14:10:39.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:39.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:39.648598+0000) 2024-03-06T14:10:40.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:40.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:40.648954+0000) 2024-03-06T14:10:41.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:41.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:41.649262+0000) 2024-03-06T14:10:42.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:42.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:42.649565+0000) 2024-03-06T14:10:43.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:43.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:43.649908+0000) 2024-03-06T14:10:44.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:44.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:44.650208+0000) 2024-03-06T14:10:45.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:45.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:45.650449+0000) 2024-03-06T14:10:46.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:46.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:46.650792+0000) 2024-03-06T14:10:47.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:47.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:47.651112+0000) 2024-03-06T14:10:48.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:48.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:48.651374+0000) 2024-03-06T14:10:49.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:49.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:49.651687+0000) 2024-03-06T14:10:49.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:49.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:50.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:50.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:50.651908+0000) 2024-03-06T14:10:51.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:51.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:51.652170+0000) 2024-03-06T14:10:52.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:52.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:52.652422+0000) 2024-03-06T14:10:53.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:53.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:53.652609+0000) 2024-03-06T14:10:54.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:54.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:54.652892+0000) 2024-03-06T14:10:55.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:55.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:55.653145+0000) 2024-03-06T14:10:56.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:56.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:56.653406+0000) 2024-03-06T14:10:57.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:57.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:57.653716+0000) 2024-03-06T14:10:58.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:58.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:58.653958+0000) 2024-03-06T14:10:59.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:10:59.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:59.654233+0000) 2024-03-06T14:11:00.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:00.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:00.654500+0000) 2024-03-06T14:11:01.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:01.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:01.654761+0000) 2024-03-06T14:11:02.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:02.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:02.655020+0000) 2024-03-06T14:11:03.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:03.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:03.655265+0000) 2024-03-06T14:11:04.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:04.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:04.655468+0000) 2024-03-06T14:11:05.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:05.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:05.655754+0000) 2024-03-06T14:11:06.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:06.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:06.656000+0000) 2024-03-06T14:11:07.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:07.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:07.656242+0000) 2024-03-06T14:11:08.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:08.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:08.656499+0000) 2024-03-06T14:11:09.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:09.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:09.656768+0000) 2024-03-06T14:11:10.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:10.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:10.657036+0000) 2024-03-06T14:11:11.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:11.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:11.657290+0000) 2024-03-06T14:11:12.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:12.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:12.657543+0000) 2024-03-06T14:11:13.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:13.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:13.657799+0000) 2024-03-06T14:11:14.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:14.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:14.658051+0000) 2024-03-06T14:11:15.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:15.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:15.658295+0000) 2024-03-06T14:11:16.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:16.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:16.658554+0000) 2024-03-06T14:11:17.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:17.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:17.658808+0000) 2024-03-06T14:11:18.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:18.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:18.659012+0000) 2024-03-06T14:11:19.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:19.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:19.659280+0000) 2024-03-06T14:11:20.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:20.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:20.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:20.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:20.659544+0000) 2024-03-06T14:11:21.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:21.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:21.659781+0000) 2024-03-06T14:11:22.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:22.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:22.660048+0000) 2024-03-06T14:11:23.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:23.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:23.660308+0000) 2024-03-06T14:11:24.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:24.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:24.660569+0000) 2024-03-06T14:11:25.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:25.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:25.660827+0000) 2024-03-06T14:11:26.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:26.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:26.661090+0000) 2024-03-06T14:11:27.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:27.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:27.661367+0000) 2024-03-06T14:11:28.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:28.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:28.661624+0000) 2024-03-06T14:11:29.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:29.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:29.661930+0000) 2024-03-06T14:11:30.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:30.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:30.662203+0000) 2024-03-06T14:11:31.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:31.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:31.662458+0000) 2024-03-06T14:11:32.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:32.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:32.662751+0000) 2024-03-06T14:11:33.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:33.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:33.662986+0000) 2024-03-06T14:11:34.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:34.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:34.663260+0000) 2024-03-06T14:11:35.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:35.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:35.663505+0000) 2024-03-06T14:11:36.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:36.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:36.663749+0000) 2024-03-06T14:11:37.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:37.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:37.664013+0000) 2024-03-06T14:11:38.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:38.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:38.664279+0000) 2024-03-06T14:11:39.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:39.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:39.664545+0000) 2024-03-06T14:11:40.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:40.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:40.664815+0000) 2024-03-06T14:11:41.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:41.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:41.665070+0000) 2024-03-06T14:11:42.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:42.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:42.665327+0000) 2024-03-06T14:11:43.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:43.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:43.665583+0000) 2024-03-06T14:11:44.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:44.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:44.665846+0000) 2024-03-06T14:11:45.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:45.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:45.666113+0000) 2024-03-06T14:11:46.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:46.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:46.666372+0000) 2024-03-06T14:11:47.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:47.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:47.666662+0000) 2024-03-06T14:11:48.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:48.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:48.666906+0000) 2024-03-06T14:11:49.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:49.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:49.667165+0000) 2024-03-06T14:11:50.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:50.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:50.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:50.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:50.667444+0000) 2024-03-06T14:11:51.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:51.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:51.667727+0000) 2024-03-06T14:11:52.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:52.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:52.667986+0000) 2024-03-06T14:11:53.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:53.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:53.668248+0000) 2024-03-06T14:11:54.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:54.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:54.668497+0000) 2024-03-06T14:11:55.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:55.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:55.668772+0000) 2024-03-06T14:11:56.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:56.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:56.669028+0000) 2024-03-06T14:11:57.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:57.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:57.669277+0000) 2024-03-06T14:11:58.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:58.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:58.669517+0000) 2024-03-06T14:11:59.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:11:59.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:59.669762+0000) 2024-03-06T14:12:00.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:00.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:00.669970+0000) 2024-03-06T14:12:01.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:01.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:01.670218+0000) 2024-03-06T14:12:02.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:02.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:02.670463+0000) 2024-03-06T14:12:03.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:03.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:03.670760+0000) 2024-03-06T14:12:04.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:04.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:04.671046+0000) 2024-03-06T14:12:05.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:05.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:05.671361+0000) 2024-03-06T14:12:06.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:06.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:06.671760+0000) 2024-03-06T14:12:07.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:07.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:07.672054+0000) 2024-03-06T14:12:08.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:08.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:08.672354+0000) 2024-03-06T14:12:09.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:09.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:09.672688+0000) 2024-03-06T14:12:10.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:10.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:10.673044+0000) 2024-03-06T14:12:11.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:11.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:11.673355+0000) 2024-03-06T14:12:12.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:12.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:12.673689+0000) 2024-03-06T14:12:13.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:13.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:13.674000+0000) 2024-03-06T14:12:14.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:14.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:14.674318+0000) 2024-03-06T14:12:15.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:15.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:15.674675+0000) 2024-03-06T14:12:16.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:16.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:16.675001+0000) 2024-03-06T14:12:17.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:17.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:17.675294+0000) 2024-03-06T14:12:18.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:18.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:18.675606+0000) 2024-03-06T14:12:19.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:19.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:19.675953+0000) 2024-03-06T14:12:20.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:20.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:20.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:20.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:20.676272+0000) 2024-03-06T14:12:21.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:21.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:21.676575+0000) 2024-03-06T14:12:22.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:22.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:22.676890+0000) 2024-03-06T14:12:23.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:23.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:23.677168+0000) 2024-03-06T14:12:24.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:24.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:24.677477+0000) 2024-03-06T14:12:25.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:25.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:25.677770+0000) 2024-03-06T14:12:26.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:26.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:26.678060+0000) 2024-03-06T14:12:27.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:27.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:27.678368+0000) 2024-03-06T14:12:28.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:28.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:28.678609+0000) 2024-03-06T14:12:29.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:29.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:29.678971+0000) 2024-03-06T14:12:30.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:30.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:30.679288+0000) 2024-03-06T14:12:31.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:31.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:31.679607+0000) 2024-03-06T14:12:32.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:32.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:32.679956+0000) 2024-03-06T14:12:33.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:33.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:33.680206+0000) 2024-03-06T14:12:34.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:34.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:34.680453+0000) 2024-03-06T14:12:35.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:35.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:35.680789+0000) 2024-03-06T14:12:36.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:36.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:36.681102+0000) 2024-03-06T14:12:37.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:37.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:37.681406+0000) 2024-03-06T14:12:38.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:38.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:38.681756+0000) 2024-03-06T14:12:39.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:39.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:39.682073+0000) 2024-03-06T14:12:40.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:40.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:40.682383+0000) 2024-03-06T14:12:41.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:41.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:41.682738+0000) 2024-03-06T14:12:42.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:42.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:42.683057+0000) 2024-03-06T14:12:43.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:43.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:43.683360+0000) 2024-03-06T14:12:44.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:44.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:44.683702+0000) 2024-03-06T14:12:45.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:45.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:45.684018+0000) 2024-03-06T14:12:46.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:46.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:46.684333+0000) 2024-03-06T14:12:47.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:47.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:47.684671+0000) 2024-03-06T14:12:48.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:48.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:48.684966+0000) 2024-03-06T14:12:49.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:49.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:49.685255+0000) 2024-03-06T14:12:50.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:50.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:50.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:50.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:50.685546+0000) 2024-03-06T14:12:51.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:51.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:51.685850+0000) 2024-03-06T14:12:52.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:52.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:52.686149+0000) 2024-03-06T14:12:53.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:53.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:53.686395+0000) 2024-03-06T14:12:54.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:54.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:54.686733+0000) 2024-03-06T14:12:55.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:55.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:55.687022+0000) 2024-03-06T14:12:56.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:56.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:56.687318+0000) 2024-03-06T14:12:57.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:57.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:57.687619+0000) 2024-03-06T14:12:58.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:58.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:58.687985+0000) 2024-03-06T14:12:59.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:12:59.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:59.688344+0000) 2024-03-06T14:13:00.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:00.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:00.688702+0000) 2024-03-06T14:13:01.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:01.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:01.688957+0000) 2024-03-06T14:13:02.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:02.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:02.689195+0000) 2024-03-06T14:13:03.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:03.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:03.689513+0000) 2024-03-06T14:13:04.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:04.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:04.689814+0000) 2024-03-06T14:13:05.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:05.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:05.690124+0000) 2024-03-06T14:13:06.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:06.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:06.690406+0000) 2024-03-06T14:13:07.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:07.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:07.690760+0000) 2024-03-06T14:13:08.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:08.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:08.691047+0000) 2024-03-06T14:13:09.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:09.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:09.691350+0000) 2024-03-06T14:13:10.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:10.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:10.691721+0000) 2024-03-06T14:13:11.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:11.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:11.692036+0000) 2024-03-06T14:13:12.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:12.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:12.692333+0000) 2024-03-06T14:13:13.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:13.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:13.692689+0000) 2024-03-06T14:13:14.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:14.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:14.692988+0000) 2024-03-06T14:13:15.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:15.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:15.693281+0000) 2024-03-06T14:13:16.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:16.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:16.693589+0000) 2024-03-06T14:13:17.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:17.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:17.693919+0000) 2024-03-06T14:13:18.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:18.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:18.694225+0000) 2024-03-06T14:13:19.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:19.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:19.694520+0000) 2024-03-06T14:13:20.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:20.689 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:20.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:20.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:20.694797+0000) 2024-03-06T14:13:21.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:21.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:21.695087+0000) 2024-03-06T14:13:22.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:22.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:22.695405+0000) 2024-03-06T14:13:23.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:23.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:23.695745+0000) 2024-03-06T14:13:24.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:24.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:24.696064+0000) 2024-03-06T14:13:25.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:25.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:25.696352+0000) 2024-03-06T14:13:26.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:26.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:26.696674+0000) 2024-03-06T14:13:27.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:27.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:27.696989+0000) 2024-03-06T14:13:28.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:28.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:28.697285+0000) 2024-03-06T14:13:29.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:29.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:29.697577+0000) 2024-03-06T14:13:30.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:30.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:30.697940+0000) 2024-03-06T14:13:31.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:31.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:31.698186+0000) 2024-03-06T14:13:32.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:32.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:32.698504+0000) 2024-03-06T14:13:33.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:33.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:33.698807+0000) 2024-03-06T14:13:34.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:34.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:34.699104+0000) 2024-03-06T14:13:35.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:35.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:35.699425+0000) 2024-03-06T14:13:36.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:36.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:36.699763+0000) 2024-03-06T14:13:37.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:37.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:37.699996+0000) 2024-03-06T14:13:38.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:38.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:38.700287+0000) 2024-03-06T14:13:39.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:39.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:39.700590+0000) 2024-03-06T14:13:40.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:40.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:40.700957+0000) 2024-03-06T14:13:41.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:41.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:41.701248+0000) 2024-03-06T14:13:42.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:42.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:42.701552+0000) 2024-03-06T14:13:43.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:43.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:43.701851+0000) 2024-03-06T14:13:44.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:44.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:44.702088+0000) 2024-03-06T14:13:45.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:45.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:45.702395+0000) 2024-03-06T14:13:46.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:46.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:46.702681+0000) 2024-03-06T14:13:47.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:47.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:47.702988+0000) 2024-03-06T14:13:48.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:48.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:48.703268+0000) 2024-03-06T14:13:49.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:49.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:49.703580+0000) 2024-03-06T14:13:50.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:50.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:50.703943+0000) 2024-03-06T14:13:50.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:50.837 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:51.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:51.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:51.704242+0000) 2024-03-06T14:13:52.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:52.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:52.704548+0000) 2024-03-06T14:13:53.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:53.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:53.704855+0000) 2024-03-06T14:13:54.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:54.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:54.705157+0000) 2024-03-06T14:13:55.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:55.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:55.705468+0000) 2024-03-06T14:13:56.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:56.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:56.705782+0000) 2024-03-06T14:13:57.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:57.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:57.706084+0000) 2024-03-06T14:13:58.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:58.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:58.706399+0000) 2024-03-06T14:13:59.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:13:59.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:59.706727+0000) 2024-03-06T14:14:00.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:00.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:00.707045+0000) 2024-03-06T14:14:01.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:01.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:01.707352+0000) 2024-03-06T14:14:02.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:02.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:02.707715+0000) 2024-03-06T14:14:03.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:03.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:03.708010+0000) 2024-03-06T14:14:04.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:04.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:04.708313+0000) 2024-03-06T14:14:05.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:05.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:05.708611+0000) 2024-03-06T14:14:06.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:06.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:06.708987+0000) 2024-03-06T14:14:07.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:07.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:07.709305+0000) 2024-03-06T14:14:08.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:08.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:08.709597+0000) 2024-03-06T14:14:09.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:09.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:09.709953+0000) 2024-03-06T14:14:10.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:10.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:10.710270+0000) 2024-03-06T14:14:11.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:11.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:11.710561+0000) 2024-03-06T14:14:12.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:12.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:12.710866+0000) 2024-03-06T14:14:13.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:13.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:13.711154+0000) 2024-03-06T14:14:14.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:14.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:14.711459+0000) 2024-03-06T14:14:15.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:15.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:15.711792+0000) 2024-03-06T14:14:16.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:16.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:16.712083+0000) 2024-03-06T14:14:17.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:17.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:17.712397+0000) 2024-03-06T14:14:18.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:18.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:18.712738+0000) 2024-03-06T14:14:19.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:19.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:19.713039+0000) 2024-03-06T14:14:20.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:20.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:20.713278+0000) 2024-03-06T14:14:20.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:20.978 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:21.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:21.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:21.713573+0000) 2024-03-06T14:14:22.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:22.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:22.713866+0000) 2024-03-06T14:14:23.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:23.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:23.714155+0000) 2024-03-06T14:14:24.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:24.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:24.714441+0000) 2024-03-06T14:14:25.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:25.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:25.714771+0000) 2024-03-06T14:14:26.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:26.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:26.715082+0000) 2024-03-06T14:14:27.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:27.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:27.715348+0000) 2024-03-06T14:14:28.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:28.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:28.715673+0000) 2024-03-06T14:14:29.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:29.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:29.715985+0000) 2024-03-06T14:14:30.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:30.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:30.716366+0000) 2024-03-06T14:14:31.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:31.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:31.716728+0000) 2024-03-06T14:14:32.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:32.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:32.717050+0000) 2024-03-06T14:14:33.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:33.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:33.717363+0000) 2024-03-06T14:14:34.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:34.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:34.717720+0000) 2024-03-06T14:14:35.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:35.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:35.718051+0000) 2024-03-06T14:14:36.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:36.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:36.718361+0000) 2024-03-06T14:14:37.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:37.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:37.718721+0000) 2024-03-06T14:14:38.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:38.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:38.719025+0000) 2024-03-06T14:14:39.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:39.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:39.719338+0000) 2024-03-06T14:14:40.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:40.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:40.719585+0000) 2024-03-06T14:14:41.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:41.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:41.719932+0000) 2024-03-06T14:14:42.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:42.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:42.720250+0000) 2024-03-06T14:14:43.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:43.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:43.720550+0000) 2024-03-06T14:14:44.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:44.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:44.720866+0000) 2024-03-06T14:14:45.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:45.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:45.721175+0000) 2024-03-06T14:14:46.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:46.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:46.721492+0000) 2024-03-06T14:14:47.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:47.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:47.721838+0000) 2024-03-06T14:14:48.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:48.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:48.722152+0000) 2024-03-06T14:14:49.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:49.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:49.722454+0000) 2024-03-06T14:14:50.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:50.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:50.722757+0000) 2024-03-06T14:14:51.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:51.133 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:51.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:51.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:51.723065+0000) 2024-03-06T14:14:52.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:52.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:52.723375+0000) 2024-03-06T14:14:53.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:53.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:53.723714+0000) 2024-03-06T14:14:54.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:54.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:54.723953+0000) 2024-03-06T14:14:55.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:55.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:55.724264+0000) 2024-03-06T14:14:56.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:56.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:56.724557+0000) 2024-03-06T14:14:57.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:57.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:57.724870+0000) 2024-03-06T14:14:58.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:58.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:58.725146+0000) 2024-03-06T14:14:59.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:14:59.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:59.725449+0000) 2024-03-06T14:15:00.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:00.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:00.725772+0000) 2024-03-06T14:15:01.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:01.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:01.726088+0000) 2024-03-06T14:15:02.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:02.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:02.726370+0000) 2024-03-06T14:15:03.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:03.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:03.726729+0000) 2024-03-06T14:15:04.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:04.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:04.727021+0000) 2024-03-06T14:15:05.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:05.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:05.727311+0000) 2024-03-06T14:15:06.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:06.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:06.727623+0000) 2024-03-06T14:15:07.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:07.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:07.727945+0000) 2024-03-06T14:15:08.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:08.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:08.728203+0000) 2024-03-06T14:15:09.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:09.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:09.728463+0000) 2024-03-06T14:15:10.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:10.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:10.728748+0000) 2024-03-06T14:15:11.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:11.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:11.729005+0000) 2024-03-06T14:15:12.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:12.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:12.729246+0000) 2024-03-06T14:15:13.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:13.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:13.729513+0000) 2024-03-06T14:15:14.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:14.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:14.729766+0000) 2024-03-06T14:15:15.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:15.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:15.730029+0000) 2024-03-06T14:15:16.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:16.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:16.730220+0000) 2024-03-06T14:15:17.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:17.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:17.730456+0000) 2024-03-06T14:15:18.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:18.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:18.730741+0000) 2024-03-06T14:15:19.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:19.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:19.730988+0000) 2024-03-06T14:15:20.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:20.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:20.731246+0000) 2024-03-06T14:15:21.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:21.276 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:21.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:21.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:21.731514+0000) 2024-03-06T14:15:22.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:22.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:22.731772+0000) 2024-03-06T14:15:23.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:23.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:23.731982+0000) 2024-03-06T14:15:24.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:24.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:24.732237+0000) 2024-03-06T14:15:25.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:25.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:25.732482+0000) 2024-03-06T14:15:26.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:26.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:26.732743+0000) 2024-03-06T14:15:27.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:27.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:27.733000+0000) 2024-03-06T14:15:28.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:28.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:28.733250+0000) 2024-03-06T14:15:29.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:29.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:29.733564+0000) 2024-03-06T14:15:30.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:30.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:30.733828+0000) 2024-03-06T14:15:31.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:31.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:31.734086+0000) 2024-03-06T14:15:32.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:32.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:32.734340+0000) 2024-03-06T14:15:33.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:33.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:33.734602+0000) 2024-03-06T14:15:34.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:34.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:34.734909+0000) 2024-03-06T14:15:35.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:35.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:35.735190+0000) 2024-03-06T14:15:36.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:36.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:36.735451+0000) 2024-03-06T14:15:37.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:37.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:37.735729+0000) 2024-03-06T14:15:38.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:38.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:38.735988+0000) 2024-03-06T14:15:39.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:39.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:39.736237+0000) 2024-03-06T14:15:40.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:40.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:40.736513+0000) 2024-03-06T14:15:41.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:41.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:41.736778+0000) 2024-03-06T14:15:42.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:42.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:42.737021+0000) 2024-03-06T14:15:43.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:43.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:43.737276+0000) 2024-03-06T14:15:44.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:44.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:44.737508+0000) 2024-03-06T14:15:45.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:45.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:45.737763+0000) 2024-03-06T14:15:46.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:46.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:46.738002+0000) 2024-03-06T14:15:47.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:47.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:47.738256+0000) 2024-03-06T14:15:48.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:48.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:48.738506+0000) 2024-03-06T14:15:49.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:49.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:49.738757+0000) 2024-03-06T14:15:50.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:50.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:50.739026+0000) 2024-03-06T14:15:51.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:51.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:51.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:51.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:51.739288+0000) 2024-03-06T14:15:52.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:52.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:52.739516+0000) 2024-03-06T14:15:53.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:53.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:53.739778+0000) 2024-03-06T14:15:54.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:54.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:54.739971+0000) 2024-03-06T14:15:55.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:55.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:55.740252+0000) 2024-03-06T14:15:56.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:56.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:56.740492+0000) 2024-03-06T14:15:57.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:57.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:57.740748+0000) 2024-03-06T14:15:58.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:58.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:58.740999+0000) 2024-03-06T14:15:59.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:15:59.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:59.741238+0000) 2024-03-06T14:16:00.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:00.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:00.741511+0000) 2024-03-06T14:16:01.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:01.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:01.741769+0000) 2024-03-06T14:16:02.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:02.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:02.742020+0000) 2024-03-06T14:16:03.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:03.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:03.742277+0000) 2024-03-06T14:16:04.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:04.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:04.742537+0000) 2024-03-06T14:16:05.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:05.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:05.742807+0000) 2024-03-06T14:16:06.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:06.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:06.743044+0000) 2024-03-06T14:16:07.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:07.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:07.743296+0000) 2024-03-06T14:16:08.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:08.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:08.743488+0000) 2024-03-06T14:16:09.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:09.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:09.743772+0000) 2024-03-06T14:16:10.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:10.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:10.744030+0000) 2024-03-06T14:16:11.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:11.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:11.744288+0000) 2024-03-06T14:16:12.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:12.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:12.744517+0000) 2024-03-06T14:16:13.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:13.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:13.744775+0000) 2024-03-06T14:16:14.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:14.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:14.745047+0000) 2024-03-06T14:16:15.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:15.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:15.745282+0000) 2024-03-06T14:16:16.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:16.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:16.745518+0000) 2024-03-06T14:16:17.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:17.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:17.745821+0000) 2024-03-06T14:16:18.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:18.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:18.746077+0000) 2024-03-06T14:16:19.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:19.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:19.746341+0000) 2024-03-06T14:16:20.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:20.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:20.746601+0000) 2024-03-06T14:16:21.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:21.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:21.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:21.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:21.746872+0000) 2024-03-06T14:16:22.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:22.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:22.747124+0000) 2024-03-06T14:16:23.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:23.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:23.747361+0000) 2024-03-06T14:16:24.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:24.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:24.747612+0000) 2024-03-06T14:16:25.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:25.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:25.747890+0000) 2024-03-06T14:16:26.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:26.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:26.748118+0000) 2024-03-06T14:16:27.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:27.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:27.748372+0000) 2024-03-06T14:16:28.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:28.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:28.748609+0000) 2024-03-06T14:16:29.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:29.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:29.748879+0000) 2024-03-06T14:16:30.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:30.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:30.749129+0000) 2024-03-06T14:16:31.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:31.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:31.749367+0000) 2024-03-06T14:16:32.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:32.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:32.749607+0000) 2024-03-06T14:16:33.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:33.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:33.749900+0000) 2024-03-06T14:16:34.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:34.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:34.750155+0000) 2024-03-06T14:16:35.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:35.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:35.750412+0000) 2024-03-06T14:16:36.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:36.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:36.750678+0000) 2024-03-06T14:16:37.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:37.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:37.750935+0000) 2024-03-06T14:16:38.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:38.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:38.751129+0000) 2024-03-06T14:16:39.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:39.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:39.751370+0000) 2024-03-06T14:16:40.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:40.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:40.751612+0000) 2024-03-06T14:16:41.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:41.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:41.751886+0000) 2024-03-06T14:16:42.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:42.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:42.752137+0000) 2024-03-06T14:16:43.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:43.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:43.752380+0000) 2024-03-06T14:16:44.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:44.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:44.752613+0000) 2024-03-06T14:16:45.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:45.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:45.752893+0000) 2024-03-06T14:16:46.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:46.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:46.753134+0000) 2024-03-06T14:16:47.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:47.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:47.753379+0000) 2024-03-06T14:16:48.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:48.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:48.753580+0000) 2024-03-06T14:16:49.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:49.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:49.753827+0000) 2024-03-06T14:16:50.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:50.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:50.754090+0000) 2024-03-06T14:16:51.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:51.725 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:51.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:51.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:51.754295+0000) 2024-03-06T14:16:52.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:52.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:52.754550+0000) 2024-03-06T14:16:53.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:53.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:53.754822+0000) 2024-03-06T14:16:54.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:54.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:54.755074+0000) 2024-03-06T14:16:55.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:55.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:55.755318+0000) 2024-03-06T14:16:56.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:56.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:56.755567+0000) 2024-03-06T14:16:57.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:57.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:57.755812+0000) 2024-03-06T14:16:58.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:58.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:58.756045+0000) 2024-03-06T14:16:59.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:16:59.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:59.756309+0000) 2024-03-06T14:17:00.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:00.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:00.756572+0000) 2024-03-06T14:17:01.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:01.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:01.756827+0000) 2024-03-06T14:17:02.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:02.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:02.757127+0000) 2024-03-06T14:17:03.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:03.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:03.757441+0000) 2024-03-06T14:17:04.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:04.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:04.757779+0000) 2024-03-06T14:17:05.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:05.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:05.758055+0000) 2024-03-06T14:17:06.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:06.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:06.758315+0000) 2024-03-06T14:17:07.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:07.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:07.758574+0000) 2024-03-06T14:17:08.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:08.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:08.758803+0000) 2024-03-06T14:17:09.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:09.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:09.759058+0000) 2024-03-06T14:17:10.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:10.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:10.759319+0000) 2024-03-06T14:17:11.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:11.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:11.759578+0000) 2024-03-06T14:17:12.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:12.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:12.759831+0000) 2024-03-06T14:17:13.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:13.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:13.760064+0000) 2024-03-06T14:17:14.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:14.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:14.760297+0000) 2024-03-06T14:17:15.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:15.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:15.760544+0000) 2024-03-06T14:17:16.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:16.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:16.760758+0000) 2024-03-06T14:17:17.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:17.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:17.761017+0000) 2024-03-06T14:17:18.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:18.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:18.761198+0000) 2024-03-06T14:17:19.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:19.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:19.761505+0000) 2024-03-06T14:17:20.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:20.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:20.761773+0000) 2024-03-06T14:17:21.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:21.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:21.762038+0000) 2024-03-06T14:17:21.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:21.874 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:22.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:22.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:22.762283+0000) 2024-03-06T14:17:23.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:23.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:23.762538+0000) 2024-03-06T14:17:24.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:24.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:24.762789+0000) 2024-03-06T14:17:25.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:25.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:25.763044+0000) 2024-03-06T14:17:26.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:26.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:26.763232+0000) 2024-03-06T14:17:27.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:27.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:27.763487+0000) 2024-03-06T14:17:28.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:28.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:28.763741+0000) 2024-03-06T14:17:29.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:29.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:29.763938+0000) 2024-03-06T14:17:30.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:30.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:30.764200+0000) 2024-03-06T14:17:31.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:31.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:31.764458+0000) 2024-03-06T14:17:32.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:32.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:32.764757+0000) 2024-03-06T14:17:33.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:33.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:33.765015+0000) 2024-03-06T14:17:34.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:34.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:34.765204+0000) 2024-03-06T14:17:35.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:35.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:35.765460+0000) 2024-03-06T14:17:36.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:36.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:36.765748+0000) 2024-03-06T14:17:37.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:37.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:37.766009+0000) 2024-03-06T14:17:38.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:38.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:38.766264+0000) 2024-03-06T14:17:39.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:39.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:39.766520+0000) 2024-03-06T14:17:40.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:40.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:40.766783+0000) 2024-03-06T14:17:41.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:41.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:41.767001+0000) 2024-03-06T14:17:42.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:42.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:42.767297+0000) 2024-03-06T14:17:43.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:43.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:43.767587+0000) 2024-03-06T14:17:44.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:44.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:44.767904+0000) 2024-03-06T14:17:45.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:45.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:45.768152+0000) 2024-03-06T14:17:46.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:46.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:46.768389+0000) 2024-03-06T14:17:47.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:47.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:47.768674+0000) 2024-03-06T14:17:48.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:48.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:48.768910+0000) 2024-03-06T14:17:49.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:49.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:49.769152+0000) 2024-03-06T14:17:50.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:50.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:50.769411+0000) 2024-03-06T14:17:51.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:51.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:51.769684+0000) 2024-03-06T14:17:52.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:52.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:52.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:52.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:52.769938+0000) 2024-03-06T14:17:53.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:53.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:53.770182+0000) 2024-03-06T14:17:54.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:54.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:54.770438+0000) 2024-03-06T14:17:55.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:55.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:55.770717+0000) 2024-03-06T14:17:56.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:56.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:56.770970+0000) 2024-03-06T14:17:57.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:57.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:57.771232+0000) 2024-03-06T14:17:58.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:58.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:58.771469+0000) 2024-03-06T14:17:59.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:17:59.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:59.771769+0000) 2024-03-06T14:18:00.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:00.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:00.772011+0000) 2024-03-06T14:18:01.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:01.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:01.772276+0000) 2024-03-06T14:18:02.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:02.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:02.772510+0000) 2024-03-06T14:18:03.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:03.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:03.772769+0000) 2024-03-06T14:18:04.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:04.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:04.773024+0000) 2024-03-06T14:18:05.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:05.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:05.773285+0000) 2024-03-06T14:18:06.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:06.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:06.773539+0000) 2024-03-06T14:18:07.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:07.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:07.773794+0000) 2024-03-06T14:18:08.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:08.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:08.774042+0000) 2024-03-06T14:18:09.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:09.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:09.774283+0000) 2024-03-06T14:18:10.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:10.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:10.774486+0000) 2024-03-06T14:18:11.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:11.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:11.774765+0000) 2024-03-06T14:18:12.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:12.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:12.774949+0000) 2024-03-06T14:18:13.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:13.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:13.775183+0000) 2024-03-06T14:18:14.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:14.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:14.775435+0000) 2024-03-06T14:18:15.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:15.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:15.775720+0000) 2024-03-06T14:18:16.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:16.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:16.775928+0000) 2024-03-06T14:18:17.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:17.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:17.776160+0000) 2024-03-06T14:18:18.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:18.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:18.776393+0000) 2024-03-06T14:18:19.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:19.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:19.776702+0000) 2024-03-06T14:18:20.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:20.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:20.776950+0000) 2024-03-06T14:18:21.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:21.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:21.777208+0000) 2024-03-06T14:18:22.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:22.174 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:22.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:22.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:22.777449+0000) 2024-03-06T14:18:23.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:23.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:23.777727+0000) 2024-03-06T14:18:24.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:24.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:24.777986+0000) 2024-03-06T14:18:25.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:25.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:25.778225+0000) 2024-03-06T14:18:26.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:26.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:26.778408+0000) 2024-03-06T14:18:27.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:27.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:27.778684+0000) 2024-03-06T14:18:28.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:28.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:28.778942+0000) 2024-03-06T14:18:29.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:29.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:29.779186+0000) 2024-03-06T14:18:30.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:30.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:30.779389+0000) 2024-03-06T14:18:31.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:31.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:31.779669+0000) 2024-03-06T14:18:32.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:32.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:32.779913+0000) 2024-03-06T14:18:33.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:33.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:33.780166+0000) 2024-03-06T14:18:34.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:34.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:34.780402+0000) 2024-03-06T14:18:35.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:35.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:35.780704+0000) 2024-03-06T14:18:36.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:36.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:36.780961+0000) 2024-03-06T14:18:37.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:37.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:37.781200+0000) 2024-03-06T14:18:38.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:38.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:38.781450+0000) 2024-03-06T14:18:39.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:39.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:39.781763+0000) 2024-03-06T14:18:40.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:40.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:40.782054+0000) 2024-03-06T14:18:41.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:41.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:41.782301+0000) 2024-03-06T14:18:42.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:42.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:42.782537+0000) 2024-03-06T14:18:43.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:43.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:43.782788+0000) 2024-03-06T14:18:44.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:44.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:44.783039+0000) 2024-03-06T14:18:45.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:45.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:45.783295+0000) 2024-03-06T14:18:46.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:46.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:46.783545+0000) 2024-03-06T14:18:47.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:47.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:47.783818+0000) 2024-03-06T14:18:48.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:48.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:48.784052+0000) 2024-03-06T14:18:49.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:49.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:49.784303+0000) 2024-03-06T14:18:50.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:50.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:50.784548+0000) 2024-03-06T14:18:51.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:51.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:51.784800+0000) 2024-03-06T14:18:52.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:52.316 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:52.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:52.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:52.785032+0000) 2024-03-06T14:18:53.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:53.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:53.785289+0000) 2024-03-06T14:18:54.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:54.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:54.785515+0000) 2024-03-06T14:18:55.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:55.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:55.785769+0000) 2024-03-06T14:18:56.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:56.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:56.786019+0000) 2024-03-06T14:18:57.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:57.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:57.786241+0000) 2024-03-06T14:18:58.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:58.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:58.786487+0000) 2024-03-06T14:18:59.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:18:59.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:59.786746+0000) 2024-03-06T14:19:00.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:00.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:00.787013+0000) 2024-03-06T14:19:01.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:01.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:01.787275+0000) 2024-03-06T14:19:02.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:02.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:02.787525+0000) 2024-03-06T14:19:03.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:03.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:03.787769+0000) 2024-03-06T14:19:04.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:04.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:04.788020+0000) 2024-03-06T14:19:05.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:05.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:05.788280+0000) 2024-03-06T14:19:06.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:06.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:06.788518+0000) 2024-03-06T14:19:07.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:07.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:07.788785+0000) 2024-03-06T14:19:08.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:08.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:08.789030+0000) 2024-03-06T14:19:09.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:09.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:09.789297+0000) 2024-03-06T14:19:10.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:10.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:10.789558+0000) 2024-03-06T14:19:11.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:11.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:11.789742+0000) 2024-03-06T14:19:12.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:12.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:12.789979+0000) 2024-03-06T14:19:13.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:13.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:13.790231+0000) 2024-03-06T14:19:14.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:14.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:14.790473+0000) 2024-03-06T14:19:15.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:15.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:15.790738+0000) 2024-03-06T14:19:16.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:16.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:16.790988+0000) 2024-03-06T14:19:17.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:17.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:17.791233+0000) 2024-03-06T14:19:18.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:18.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:18.791465+0000) 2024-03-06T14:19:19.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:19.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:19.791765+0000) 2024-03-06T14:19:20.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:20.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:20.792027+0000) 2024-03-06T14:19:21.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:21.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:21.792283+0000) 2024-03-06T14:19:22.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:22.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:22.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:22.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:22.792545+0000) 2024-03-06T14:19:23.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:23.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:23.792801+0000) 2024-03-06T14:19:24.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:24.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:24.793034+0000) 2024-03-06T14:19:25.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:25.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:25.793274+0000) 2024-03-06T14:19:26.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:26.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:26.793586+0000) 2024-03-06T14:19:27.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:27.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:27.793851+0000) 2024-03-06T14:19:28.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:28.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:28.794103+0000) 2024-03-06T14:19:29.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:29.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:29.794370+0000) 2024-03-06T14:19:30.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:30.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:30.794667+0000) 2024-03-06T14:19:31.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:31.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:31.794917+0000) 2024-03-06T14:19:32.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:32.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:32.795153+0000) 2024-03-06T14:19:33.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:33.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:33.795386+0000) 2024-03-06T14:19:34.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:34.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:34.795722+0000) 2024-03-06T14:19:35.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:35.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:35.795973+0000) 2024-03-06T14:19:36.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:36.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:36.796220+0000) 2024-03-06T14:19:37.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:37.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:37.796459+0000) 2024-03-06T14:19:38.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:38.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:38.796723+0000) 2024-03-06T14:19:39.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:39.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:39.796968+0000) 2024-03-06T14:19:40.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:40.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:40.797214+0000) 2024-03-06T14:19:41.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:41.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:41.797458+0000) 2024-03-06T14:19:42.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:42.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:42.797742+0000) 2024-03-06T14:19:43.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:43.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:43.797930+0000) 2024-03-06T14:19:44.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:44.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:44.798187+0000) 2024-03-06T14:19:45.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:45.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:45.798445+0000) 2024-03-06T14:19:46.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:46.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:46.798729+0000) 2024-03-06T14:19:47.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:47.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:47.798989+0000) 2024-03-06T14:19:48.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:48.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:48.799243+0000) 2024-03-06T14:19:49.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:49.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:49.799494+0000) 2024-03-06T14:19:50.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:50.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:50.799806+0000) 2024-03-06T14:19:51.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:51.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:51.800076+0000) 2024-03-06T14:19:52.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:52.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:52.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:52.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:52.800331+0000) 2024-03-06T14:19:53.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:53.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:53.800588+0000) 2024-03-06T14:19:54.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:54.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:54.800846+0000) 2024-03-06T14:19:55.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:55.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:55.801090+0000) 2024-03-06T14:19:56.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:56.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:56.801343+0000) 2024-03-06T14:19:57.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:57.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:57.801582+0000) 2024-03-06T14:19:58.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:58.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:58.801833+0000) 2024-03-06T14:19:59.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:19:59.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:59.802105+0000) 2024-03-06T14:20:00.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:00.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:00.802369+0000) 2024-03-06T14:20:01.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:01.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:01.802667+0000) 2024-03-06T14:20:02.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:02.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:02.802893+0000) 2024-03-06T14:20:03.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:03.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:03.803077+0000) 2024-03-06T14:20:04.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:04.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:04.803342+0000) 2024-03-06T14:20:05.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:05.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:05.803583+0000) 2024-03-06T14:20:06.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:06.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:06.803841+0000) 2024-03-06T14:20:07.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:07.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:07.804100+0000) 2024-03-06T14:20:08.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:08.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:08.804283+0000) 2024-03-06T14:20:09.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:09.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:09.804547+0000) 2024-03-06T14:20:10.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:10.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:10.804795+0000) 2024-03-06T14:20:11.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:11.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:11.805007+0000) 2024-03-06T14:20:12.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:12.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:12.805251+0000) 2024-03-06T14:20:13.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:13.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:13.805485+0000) 2024-03-06T14:20:14.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:14.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:14.805758+0000) 2024-03-06T14:20:15.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:15.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:15.806011+0000) 2024-03-06T14:20:16.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:16.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:16.806271+0000) 2024-03-06T14:20:17.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:17.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:17.806460+0000) 2024-03-06T14:20:18.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:18.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:18.806729+0000) 2024-03-06T14:20:19.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:19.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:19.806985+0000) 2024-03-06T14:20:20.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:20.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:20.807250+0000) 2024-03-06T14:20:21.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:21.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:21.807516+0000) 2024-03-06T14:20:22.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:22.756 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:22.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:22.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:22.807841+0000) 2024-03-06T14:20:23.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:23.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:23.808112+0000) 2024-03-06T14:20:24.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:24.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:24.808351+0000) 2024-03-06T14:20:25.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:25.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:25.808591+0000) 2024-03-06T14:20:26.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:26.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:26.808824+0000) 2024-03-06T14:20:27.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:27.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:27.809108+0000) 2024-03-06T14:20:28.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:28.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:28.809358+0000) 2024-03-06T14:20:29.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:29.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:29.809663+0000) 2024-03-06T14:20:30.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:30.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:30.809938+0000) 2024-03-06T14:20:31.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:31.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:31.810126+0000) 2024-03-06T14:20:32.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:32.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:32.810380+0000) 2024-03-06T14:20:33.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:33.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:33.810667+0000) 2024-03-06T14:20:34.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:34.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:34.810940+0000) 2024-03-06T14:20:35.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:35.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:35.811196+0000) 2024-03-06T14:20:36.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:36.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:36.811454+0000) 2024-03-06T14:20:37.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:37.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:37.811749+0000) 2024-03-06T14:20:38.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:38.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:38.811944+0000) 2024-03-06T14:20:39.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:39.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:39.812190+0000) 2024-03-06T14:20:40.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:40.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:40.812451+0000) 2024-03-06T14:20:41.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:41.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:41.812756+0000) 2024-03-06T14:20:42.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:42.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:42.813015+0000) 2024-03-06T14:20:43.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:43.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:43.813272+0000) 2024-03-06T14:20:44.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:44.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:44.813536+0000) 2024-03-06T14:20:45.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:45.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:45.813769+0000) 2024-03-06T14:20:46.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:46.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:46.814003+0000) 2024-03-06T14:20:47.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:47.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:47.814249+0000) 2024-03-06T14:20:48.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:48.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:48.814509+0000) 2024-03-06T14:20:49.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:49.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:49.814782+0000) 2024-03-06T14:20:50.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:50.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:50.815047+0000) 2024-03-06T14:20:51.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:51.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:51.815299+0000) 2024-03-06T14:20:52.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:52.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:52.815526+0000) 2024-03-06T14:20:52.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:52.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:53.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:53.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:53.815781+0000) 2024-03-06T14:20:54.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:54.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:54.815974+0000) 2024-03-06T14:20:55.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:55.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:55.816214+0000) 2024-03-06T14:20:56.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:56.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:56.816476+0000) 2024-03-06T14:20:57.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:57.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:57.816760+0000) 2024-03-06T14:20:58.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:58.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:58.817012+0000) 2024-03-06T14:20:59.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:20:59.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:59.817269+0000) 2024-03-06T14:21:00.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:00.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:00.817510+0000) 2024-03-06T14:21:01.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:01.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:01.817763+0000) 2024-03-06T14:21:02.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:02.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:02.818020+0000) 2024-03-06T14:21:03.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:03.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:03.818275+0000) 2024-03-06T14:21:04.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:04.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:04.818510+0000) 2024-03-06T14:21:05.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:05.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:05.818781+0000) 2024-03-06T14:21:06.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:06.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:06.819018+0000) 2024-03-06T14:21:07.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:07.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:07.819274+0000) 2024-03-06T14:21:08.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:08.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:08.819508+0000) 2024-03-06T14:21:09.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:09.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:09.819751+0000) 2024-03-06T14:21:10.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:10.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:10.819956+0000) 2024-03-06T14:21:11.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:11.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:11.820153+0000) 2024-03-06T14:21:12.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:12.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:12.820338+0000) 2024-03-06T14:21:13.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:13.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:13.820518+0000) 2024-03-06T14:21:14.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:14.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:14.820756+0000) 2024-03-06T14:21:15.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:15.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:15.821078+0000) 2024-03-06T14:21:16.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:16.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:16.821334+0000) 2024-03-06T14:21:17.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:17.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:17.821604+0000) 2024-03-06T14:21:18.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:18.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:18.821905+0000) 2024-03-06T14:21:19.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:19.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:19.822137+0000) 2024-03-06T14:21:20.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:20.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:20.822381+0000) 2024-03-06T14:21:21.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:21.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:21.822572+0000) 2024-03-06T14:21:22.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:22.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:22.822806+0000) 2024-03-06T14:21:23.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:23.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:23.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:23.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:23.823076+0000) 2024-03-06T14:21:24.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:24.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:24.823326+0000) 2024-03-06T14:21:25.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:25.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:25.823567+0000) 2024-03-06T14:21:26.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:26.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:26.823807+0000) 2024-03-06T14:21:27.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:27.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:27.823992+0000) 2024-03-06T14:21:28.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:28.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:28.824253+0000) 2024-03-06T14:21:29.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:29.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:29.824497+0000) 2024-03-06T14:21:30.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:30.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:30.824761+0000) 2024-03-06T14:21:31.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:31.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:31.825019+0000) 2024-03-06T14:21:32.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:32.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:32.825270+0000) 2024-03-06T14:21:33.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:33.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:33.825510+0000) 2024-03-06T14:21:34.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:34.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:34.825748+0000) 2024-03-06T14:21:35.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:35.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:35.826011+0000) 2024-03-06T14:21:36.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:36.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:36.826263+0000) 2024-03-06T14:21:37.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:37.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:37.826530+0000) 2024-03-06T14:21:38.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:38.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:38.826788+0000) 2024-03-06T14:21:39.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:39.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:39.827006+0000) 2024-03-06T14:21:40.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:40.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:40.827249+0000) 2024-03-06T14:21:41.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:41.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:41.827430+0000) 2024-03-06T14:21:42.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:42.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:42.827705+0000) 2024-03-06T14:21:43.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:43.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:43.827949+0000) 2024-03-06T14:21:44.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:44.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:44.828208+0000) 2024-03-06T14:21:45.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:45.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:45.828465+0000) 2024-03-06T14:21:46.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:46.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:46.828753+0000) 2024-03-06T14:21:47.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:47.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:47.829014+0000) 2024-03-06T14:21:48.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:48.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:48.829248+0000) 2024-03-06T14:21:49.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:49.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:49.829476+0000) 2024-03-06T14:21:50.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:50.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:50.829723+0000) 2024-03-06T14:21:51.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:51.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:51.829966+0000) 2024-03-06T14:21:52.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:52.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:52.830198+0000) 2024-03-06T14:21:53.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:53.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:53.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:53.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:53.830491+0000) 2024-03-06T14:21:54.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:54.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:54.830741+0000) 2024-03-06T14:21:55.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:55.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:55.830946+0000) 2024-03-06T14:21:56.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:56.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:56.831216+0000) 2024-03-06T14:21:57.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:57.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:57.831488+0000) 2024-03-06T14:21:58.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:58.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:58.831757+0000) 2024-03-06T14:21:59.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:21:59.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:59.832023+0000) 2024-03-06T14:22:00.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:00.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:00.832290+0000) 2024-03-06T14:22:01.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:01.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:01.832539+0000) 2024-03-06T14:22:02.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:02.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:02.832805+0000) 2024-03-06T14:22:03.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:03.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:03.833140+0000) 2024-03-06T14:22:04.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:04.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:04.833388+0000) 2024-03-06T14:22:05.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:05.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:05.833621+0000) 2024-03-06T14:22:06.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:06.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:06.833904+0000) 2024-03-06T14:22:07.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:07.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:07.834170+0000) 2024-03-06T14:22:08.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:08.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:08.834389+0000) 2024-03-06T14:22:09.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:09.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:09.834623+0000) 2024-03-06T14:22:10.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:10.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:10.834896+0000) 2024-03-06T14:22:11.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:11.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:11.835130+0000) 2024-03-06T14:22:12.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:12.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:12.835384+0000) 2024-03-06T14:22:13.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:13.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:13.835663+0000) 2024-03-06T14:22:14.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:14.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:14.835905+0000) 2024-03-06T14:22:15.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:15.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:15.836145+0000) 2024-03-06T14:22:16.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:16.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:16.836395+0000) 2024-03-06T14:22:17.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:17.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:17.836663+0000) 2024-03-06T14:22:18.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:18.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:18.836901+0000) 2024-03-06T14:22:19.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:19.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:19.837148+0000) 2024-03-06T14:22:20.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:20.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:20.837388+0000) 2024-03-06T14:22:21.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:21.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:21.837660+0000) 2024-03-06T14:22:22.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:22.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:22.837900+0000) 2024-03-06T14:22:23.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:23.322 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:23.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:23.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:23.838158+0000) 2024-03-06T14:22:24.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:24.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:24.838415+0000) 2024-03-06T14:22:25.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:25.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:25.838745+0000) 2024-03-06T14:22:26.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:26.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:26.838993+0000) 2024-03-06T14:22:27.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:27.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:27.839250+0000) 2024-03-06T14:22:28.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:28.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:28.839482+0000) 2024-03-06T14:22:29.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:29.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:29.839731+0000) 2024-03-06T14:22:30.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:30.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:30.839993+0000) 2024-03-06T14:22:31.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:31.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:31.840237+0000) 2024-03-06T14:22:32.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:32.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:32.840487+0000) 2024-03-06T14:22:33.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:33.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:33.840737+0000) 2024-03-06T14:22:34.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:34.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:34.840969+0000) 2024-03-06T14:22:35.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:35.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:35.841230+0000) 2024-03-06T14:22:36.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:36.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:36.841489+0000) 2024-03-06T14:22:37.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:37.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:37.841752+0000) 2024-03-06T14:22:38.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:38.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:38.842002+0000) 2024-03-06T14:22:39.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:39.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:39.842255+0000) 2024-03-06T14:22:40.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:40.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:40.842529+0000) 2024-03-06T14:22:41.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:41.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:41.842806+0000) 2024-03-06T14:22:42.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:42.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:42.843074+0000) 2024-03-06T14:22:43.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:43.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:43.843336+0000) 2024-03-06T14:22:44.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:44.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:44.843603+0000) 2024-03-06T14:22:45.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:45.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:45.843940+0000) 2024-03-06T14:22:46.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:46.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:46.844212+0000) 2024-03-06T14:22:47.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:47.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:47.844470+0000) 2024-03-06T14:22:48.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:48.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:48.844734+0000) 2024-03-06T14:22:49.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:49.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:49.844998+0000) 2024-03-06T14:22:50.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:50.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:50.845258+0000) 2024-03-06T14:22:51.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:51.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:51.845495+0000) 2024-03-06T14:22:52.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:52.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:52.845767+0000) 2024-03-06T14:22:53.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:53.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:53.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:53.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:53.846020+0000) 2024-03-06T14:22:54.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:54.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:54.846273+0000) 2024-03-06T14:22:55.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:55.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:55.846551+0000) 2024-03-06T14:22:56.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:56.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:56.846826+0000) 2024-03-06T14:22:57.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:57.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:57.847282+0000) 2024-03-06T14:22:58.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:58.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:58.847547+0000) 2024-03-06T14:22:59.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:22:59.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:59.847810+0000) 2024-03-06T14:23:00.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:00.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:00.848093+0000) 2024-03-06T14:23:01.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:01.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:01.848346+0000) 2024-03-06T14:23:02.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:02.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:02.848689+0000) 2024-03-06T14:23:03.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:03.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:03.848960+0000) 2024-03-06T14:23:04.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:04.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:04.849194+0000) 2024-03-06T14:23:05.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:05.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:05.849444+0000) 2024-03-06T14:23:06.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:06.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:06.849749+0000) 2024-03-06T14:23:07.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:07.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:07.850015+0000) 2024-03-06T14:23:08.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:08.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:08.850289+0000) 2024-03-06T14:23:09.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:09.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:09.850548+0000) 2024-03-06T14:23:10.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:10.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:10.850791+0000) 2024-03-06T14:23:11.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:11.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:11.851034+0000) 2024-03-06T14:23:12.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:12.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:12.851268+0000) 2024-03-06T14:23:13.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:13.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:13.851507+0000) 2024-03-06T14:23:14.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:14.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:14.851766+0000) 2024-03-06T14:23:15.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:15.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:15.852039+0000) 2024-03-06T14:23:16.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:16.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:16.852238+0000) 2024-03-06T14:23:17.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:17.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:17.852496+0000) 2024-03-06T14:23:18.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:18.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:18.852755+0000) 2024-03-06T14:23:19.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:19.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:19.853002+0000) 2024-03-06T14:23:20.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:20.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:20.853281+0000) 2024-03-06T14:23:21.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:21.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:21.853547+0000) 2024-03-06T14:23:22.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:22.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:22.853816+0000) 2024-03-06T14:23:23.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:23.624 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:23.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:23.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:23.854074+0000) 2024-03-06T14:23:24.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:24.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:24.854340+0000) 2024-03-06T14:23:25.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:25.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:25.854599+0000) 2024-03-06T14:23:26.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:26.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:26.854913+0000) 2024-03-06T14:23:27.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:27.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:27.855167+0000) 2024-03-06T14:23:28.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:28.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:28.855417+0000) 2024-03-06T14:23:29.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:29.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:29.855720+0000) 2024-03-06T14:23:30.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:30.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:30.855926+0000) 2024-03-06T14:23:31.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:31.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:31.856189+0000) 2024-03-06T14:23:32.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:32.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:32.856449+0000) 2024-03-06T14:23:33.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:33.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:33.856745+0000) 2024-03-06T14:23:34.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:34.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:34.856990+0000) 2024-03-06T14:23:35.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:35.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:35.857239+0000) 2024-03-06T14:23:36.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:36.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:36.857477+0000) 2024-03-06T14:23:37.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:37.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:37.857752+0000) 2024-03-06T14:23:38.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:38.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:38.857946+0000) 2024-03-06T14:23:39.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:39.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:39.858174+0000) 2024-03-06T14:23:40.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:40.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:40.858434+0000) 2024-03-06T14:23:41.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:41.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:41.858729+0000) 2024-03-06T14:23:42.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:42.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:42.859014+0000) 2024-03-06T14:23:43.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:43.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:43.859292+0000) 2024-03-06T14:23:44.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:44.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:44.859547+0000) 2024-03-06T14:23:45.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:45.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:45.859872+0000) 2024-03-06T14:23:46.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:46.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:46.860124+0000) 2024-03-06T14:23:47.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:47.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:47.860365+0000) 2024-03-06T14:23:48.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:48.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:48.860615+0000) 2024-03-06T14:23:49.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:49.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:49.860881+0000) 2024-03-06T14:23:50.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:50.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:50.861131+0000) 2024-03-06T14:23:51.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:51.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:51.861358+0000) 2024-03-06T14:23:52.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:52.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:52.861609+0000) 2024-03-06T14:23:53.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:53.784 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:53.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:53.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:53.861902+0000) 2024-03-06T14:23:54.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:54.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:54.862175+0000) 2024-03-06T14:23:55.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:55.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:55.862447+0000) 2024-03-06T14:23:56.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:56.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:56.862738+0000) 2024-03-06T14:23:57.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:57.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:57.863012+0000) 2024-03-06T14:23:58.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:58.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:58.863257+0000) 2024-03-06T14:23:59.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:23:59.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:59.863524+0000) 2024-03-06T14:24:00.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:00.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:00.863788+0000) 2024-03-06T14:24:01.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:01.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:01.864044+0000) 2024-03-06T14:24:02.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:02.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:02.864296+0000) 2024-03-06T14:24:03.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:03.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:03.864495+0000) 2024-03-06T14:24:04.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:04.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:04.864751+0000) 2024-03-06T14:24:05.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:05.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:05.865035+0000) 2024-03-06T14:24:06.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:06.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:06.865248+0000) 2024-03-06T14:24:07.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:07.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:07.865521+0000) 2024-03-06T14:24:08.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:08.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:08.865776+0000) 2024-03-06T14:24:09.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:09.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:09.866049+0000) 2024-03-06T14:24:10.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:10.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:10.866331+0000) 2024-03-06T14:24:11.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:11.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:11.866600+0000) 2024-03-06T14:24:12.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:12.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:12.866922+0000) 2024-03-06T14:24:13.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:13.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:13.867190+0000) 2024-03-06T14:24:14.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:14.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:14.867456+0000) 2024-03-06T14:24:15.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:15.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:15.867778+0000) 2024-03-06T14:24:16.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:16.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:16.868058+0000) 2024-03-06T14:24:17.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:17.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:17.868340+0000) 2024-03-06T14:24:18.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:18.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:18.868590+0000) 2024-03-06T14:24:19.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:19.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:19.868891+0000) 2024-03-06T14:24:20.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:20.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:20.869175+0000) 2024-03-06T14:24:21.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:21.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:21.869437+0000) 2024-03-06T14:24:22.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:22.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:22.869741+0000) 2024-03-06T14:24:23.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:23.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:23.870010+0000) 2024-03-06T14:24:23.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:23.923 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:24.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:24.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:24.870278+0000) 2024-03-06T14:24:25.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:25.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:25.870540+0000) 2024-03-06T14:24:26.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:26.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:26.870804+0000) 2024-03-06T14:24:27.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:27.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:27.871067+0000) 2024-03-06T14:24:28.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:28.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:28.871266+0000) 2024-03-06T14:24:29.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:29.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:29.871512+0000) 2024-03-06T14:24:30.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:30.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:30.871777+0000) 2024-03-06T14:24:31.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:31.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:31.872032+0000) 2024-03-06T14:24:32.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:32.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:32.872294+0000) 2024-03-06T14:24:33.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:33.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:33.872669+0000) 2024-03-06T14:24:34.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:34.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:34.872953+0000) 2024-03-06T14:24:35.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:35.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:35.873257+0000) 2024-03-06T14:24:36.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:36.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:36.873526+0000) 2024-03-06T14:24:37.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:37.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:37.873800+0000) 2024-03-06T14:24:38.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:38.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:38.874073+0000) 2024-03-06T14:24:39.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:39.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:39.874319+0000) 2024-03-06T14:24:40.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:40.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:40.874597+0000) 2024-03-06T14:24:41.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:41.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:41.874853+0000) 2024-03-06T14:24:42.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:42.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:42.875031+0000) 2024-03-06T14:24:43.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:43.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:43.875279+0000) 2024-03-06T14:24:44.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:44.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:44.875512+0000) 2024-03-06T14:24:45.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:45.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:45.875796+0000) 2024-03-06T14:24:46.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:46.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:46.876045+0000) 2024-03-06T14:24:47.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:47.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:47.876323+0000) 2024-03-06T14:24:48.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:48.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:48.876594+0000) 2024-03-06T14:24:49.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:49.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:49.876915+0000) 2024-03-06T14:24:50.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:50.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:50.877197+0000) 2024-03-06T14:24:51.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:51.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:51.877438+0000) 2024-03-06T14:24:52.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:52.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:52.877742+0000) 2024-03-06T14:24:53.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:53.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:53.878008+0000) 2024-03-06T14:24:54.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:54.082 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:54.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:54.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:54.878277+0000) 2024-03-06T14:24:55.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:55.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:55.878557+0000) 2024-03-06T14:24:56.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:56.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:56.878827+0000) 2024-03-06T14:24:57.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:57.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:57.879092+0000) 2024-03-06T14:24:58.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:58.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:58.879360+0000) 2024-03-06T14:24:59.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:24:59.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:59.879610+0000) 2024-03-06T14:25:00.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:00.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:00.879963+0000) 2024-03-06T14:25:01.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:01.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:01.880223+0000) 2024-03-06T14:25:02.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:02.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:02.880496+0000) 2024-03-06T14:25:03.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:03.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:03.880767+0000) 2024-03-06T14:25:04.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:04.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:04.881033+0000) 2024-03-06T14:25:05.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:05.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:05.881321+0000) 2024-03-06T14:25:06.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:06.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:06.881574+0000) 2024-03-06T14:25:07.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:07.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:07.881860+0000) 2024-03-06T14:25:08.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:08.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:08.882123+0000) 2024-03-06T14:25:09.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:09.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:09.882386+0000) 2024-03-06T14:25:10.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:10.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:10.882698+0000) 2024-03-06T14:25:11.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:11.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:11.882909+0000) 2024-03-06T14:25:12.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:12.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:12.883154+0000) 2024-03-06T14:25:13.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:13.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:13.883410+0000) 2024-03-06T14:25:14.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:14.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:14.883670+0000) 2024-03-06T14:25:15.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:15.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:15.883920+0000) 2024-03-06T14:25:16.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:16.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:16.884162+0000) 2024-03-06T14:25:17.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:17.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:17.884407+0000) 2024-03-06T14:25:18.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:18.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:18.884669+0000) 2024-03-06T14:25:19.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:19.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:19.884917+0000) 2024-03-06T14:25:20.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:20.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:20.885188+0000) 2024-03-06T14:25:21.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:21.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:21.885463+0000) 2024-03-06T14:25:22.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:22.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:22.885802+0000) 2024-03-06T14:25:23.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:23.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:23.886112+0000) 2024-03-06T14:25:24.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:24.216 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:24.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:24.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:24.886403+0000) 2024-03-06T14:25:25.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:25.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:25.886693+0000) 2024-03-06T14:25:26.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:26.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:26.886951+0000) 2024-03-06T14:25:27.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:27.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:27.887232+0000) 2024-03-06T14:25:28.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:28.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:28.887499+0000) 2024-03-06T14:25:29.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:29.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:29.887765+0000) 2024-03-06T14:25:30.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:30.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:30.888042+0000) 2024-03-06T14:25:31.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:31.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:31.888322+0000) 2024-03-06T14:25:32.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:32.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:32.888594+0000) 2024-03-06T14:25:33.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:33.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:33.888911+0000) 2024-03-06T14:25:34.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:34.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:34.889161+0000) 2024-03-06T14:25:35.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:35.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:35.889424+0000) 2024-03-06T14:25:36.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:36.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:36.889724+0000) 2024-03-06T14:25:37.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:37.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:37.889929+0000) 2024-03-06T14:25:38.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:38.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:38.890167+0000) 2024-03-06T14:25:39.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:39.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:39.890397+0000) 2024-03-06T14:25:40.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:40.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:40.890595+0000) 2024-03-06T14:25:41.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:41.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:41.890875+0000) 2024-03-06T14:25:42.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:42.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:42.891123+0000) 2024-03-06T14:25:43.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:43.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:43.891370+0000) 2024-03-06T14:25:44.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:44.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:44.891617+0000) 2024-03-06T14:25:45.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:45.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:45.891927+0000) 2024-03-06T14:25:46.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:46.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:46.892158+0000) 2024-03-06T14:25:47.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:47.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:47.892409+0000) 2024-03-06T14:25:48.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:48.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:48.892691+0000) 2024-03-06T14:25:49.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:49.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:49.892940+0000) 2024-03-06T14:25:50.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:50.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:50.893223+0000) 2024-03-06T14:25:51.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:51.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:51.893480+0000) 2024-03-06T14:25:52.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:52.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:52.893737+0000) 2024-03-06T14:25:53.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:53.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:53.893992+0000) 2024-03-06T14:25:54.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:54.371 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:54.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:54.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:54.894188+0000) 2024-03-06T14:25:55.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:55.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:55.894438+0000) 2024-03-06T14:25:56.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:56.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:56.894733+0000) 2024-03-06T14:25:57.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:57.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:57.894998+0000) 2024-03-06T14:25:58.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:58.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:58.895268+0000) 2024-03-06T14:25:59.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:25:59.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:59.895533+0000) 2024-03-06T14:26:00.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:00.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:00.895809+0000) 2024-03-06T14:26:01.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:01.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:01.896084+0000) 2024-03-06T14:26:02.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:02.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:02.896364+0000) 2024-03-06T14:26:03.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:03.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:03.896662+0000) 2024-03-06T14:26:04.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:04.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:04.896861+0000) 2024-03-06T14:26:05.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:05.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:05.897113+0000) 2024-03-06T14:26:06.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:06.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:06.897364+0000) 2024-03-06T14:26:07.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:07.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:07.897619+0000) 2024-03-06T14:26:08.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:08.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:08.897940+0000) 2024-03-06T14:26:09.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:09.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:09.898187+0000) 2024-03-06T14:26:10.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:10.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:10.898450+0000) 2024-03-06T14:26:11.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:11.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:11.898729+0000) 2024-03-06T14:26:12.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:12.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:12.898970+0000) 2024-03-06T14:26:13.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:13.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:13.899224+0000) 2024-03-06T14:26:14.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:14.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:14.899516+0000) 2024-03-06T14:26:15.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:15.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:15.899765+0000) 2024-03-06T14:26:16.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:16.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:16.900017+0000) 2024-03-06T14:26:17.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:17.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:17.900279+0000) 2024-03-06T14:26:18.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:18.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:18.900532+0000) 2024-03-06T14:26:19.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:19.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:19.900766+0000) 2024-03-06T14:26:20.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:20.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:20.901027+0000) 2024-03-06T14:26:21.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:21.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:21.901268+0000) 2024-03-06T14:26:22.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:22.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:22.901494+0000) 2024-03-06T14:26:23.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:23.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:23.901741+0000) 2024-03-06T14:26:24.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:24.536 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:24.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:24.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:24.901987+0000) 2024-03-06T14:26:25.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:25.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:25.902246+0000) 2024-03-06T14:26:26.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:26.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:26.902500+0000) 2024-03-06T14:26:27.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:27.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:27.902757+0000) 2024-03-06T14:26:28.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:28.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:28.902987+0000) 2024-03-06T14:26:29.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:29.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:29.903222+0000) 2024-03-06T14:26:30.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:30.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:30.903467+0000) 2024-03-06T14:26:31.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:31.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:31.903763+0000) 2024-03-06T14:26:32.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:32.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:32.904028+0000) 2024-03-06T14:26:33.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:33.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:33.904287+0000) 2024-03-06T14:26:34.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:34.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:34.904536+0000) 2024-03-06T14:26:35.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:35.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:35.904791+0000) 2024-03-06T14:26:36.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:36.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:36.905041+0000) 2024-03-06T14:26:37.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:37.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:37.905269+0000) 2024-03-06T14:26:38.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:38.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:38.905518+0000) 2024-03-06T14:26:39.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:39.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:39.905767+0000) 2024-03-06T14:26:40.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:40.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:40.906029+0000) 2024-03-06T14:26:41.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:41.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:41.906277+0000) 2024-03-06T14:26:42.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:42.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:42.906529+0000) 2024-03-06T14:26:43.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:43.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:43.906793+0000) 2024-03-06T14:26:44.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:44.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:44.907022+0000) 2024-03-06T14:26:45.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:45.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:45.907271+0000) 2024-03-06T14:26:46.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:46.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:46.907500+0000) 2024-03-06T14:26:47.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:47.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:47.907752+0000) 2024-03-06T14:26:48.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:48.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:48.908001+0000) 2024-03-06T14:26:49.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:49.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:49.908255+0000) 2024-03-06T14:26:50.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:50.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:50.908513+0000) 2024-03-06T14:26:51.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:51.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:51.908767+0000) 2024-03-06T14:26:52.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:52.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:52.908949+0000) 2024-03-06T14:26:53.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:53.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:53.909188+0000) 2024-03-06T14:26:54.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:54.691 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:54.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:54.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:54.909433+0000) 2024-03-06T14:26:55.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:55.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:55.909668+0000) 2024-03-06T14:26:56.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:56.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:56.909927+0000) 2024-03-06T14:26:57.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:57.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:57.910188+0000) 2024-03-06T14:26:58.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:58.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:58.910419+0000) 2024-03-06T14:26:59.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:26:59.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:59.910747+0000) 2024-03-06T14:27:00.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:00.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:00.911012+0000) 2024-03-06T14:27:01.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:01.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:01.911271+0000) 2024-03-06T14:27:02.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:02.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:02.911508+0000) 2024-03-06T14:27:03.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:03.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:03.911759+0000) 2024-03-06T14:27:04.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:04.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:04.912015+0000) 2024-03-06T14:27:05.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:05.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:05.912274+0000) 2024-03-06T14:27:06.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:06.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:06.912519+0000) 2024-03-06T14:27:07.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:07.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:07.912779+0000) 2024-03-06T14:27:08.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:08.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:08.913015+0000) 2024-03-06T14:27:09.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:09.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:09.913279+0000) 2024-03-06T14:27:10.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:10.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:10.913548+0000) 2024-03-06T14:27:11.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:11.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:11.913799+0000) 2024-03-06T14:27:12.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:12.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:12.914055+0000) 2024-03-06T14:27:13.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:13.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:13.914365+0000) 2024-03-06T14:27:14.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:14.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:14.914622+0000) 2024-03-06T14:27:15.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:15.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:15.914909+0000) 2024-03-06T14:27:16.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:16.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:16.915168+0000) 2024-03-06T14:27:17.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:17.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:17.915418+0000) 2024-03-06T14:27:18.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:18.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:18.915598+0000) 2024-03-06T14:27:19.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:19.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:19.915857+0000) 2024-03-06T14:27:20.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:20.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:20.916106+0000) 2024-03-06T14:27:21.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:21.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:21.916351+0000) 2024-03-06T14:27:22.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:22.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:22.916604+0000) 2024-03-06T14:27:23.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:23.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:23.916888+0000) 2024-03-06T14:27:24.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:24.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:24.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:24.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:24.917129+0000) 2024-03-06T14:27:25.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:25.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:25.917409+0000) 2024-03-06T14:27:26.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:26.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:26.917695+0000) 2024-03-06T14:27:27.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:27.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:27.917931+0000) 2024-03-06T14:27:28.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:28.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:28.918164+0000) 2024-03-06T14:27:29.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:29.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:29.918429+0000) 2024-03-06T14:27:30.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:30.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:30.918729+0000) 2024-03-06T14:27:31.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:31.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:31.918998+0000) 2024-03-06T14:27:32.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:32.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:32.919249+0000) 2024-03-06T14:27:33.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:33.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:33.919508+0000) 2024-03-06T14:27:34.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:34.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:34.919747+0000) 2024-03-06T14:27:35.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:35.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:35.920003+0000) 2024-03-06T14:27:36.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:36.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:36.920263+0000) 2024-03-06T14:27:37.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:37.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:37.920519+0000) 2024-03-06T14:27:38.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:38.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:38.920771+0000) 2024-03-06T14:27:39.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:39.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:39.921034+0000) 2024-03-06T14:27:40.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:40.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:40.921284+0000) 2024-03-06T14:27:41.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:41.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:41.921546+0000) 2024-03-06T14:27:42.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:42.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:42.921804+0000) 2024-03-06T14:27:43.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:43.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:43.922057+0000) 2024-03-06T14:27:44.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:44.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:44.922337+0000) 2024-03-06T14:27:45.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:45.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:45.922682+0000) 2024-03-06T14:27:46.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:46.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:46.922999+0000) 2024-03-06T14:27:47.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:47.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:47.923306+0000) 2024-03-06T14:27:48.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:48.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:48.923541+0000) 2024-03-06T14:27:49.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:49.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:49.923829+0000) 2024-03-06T14:27:50.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:50.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:50.924152+0000) 2024-03-06T14:27:51.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:51.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:51.924465+0000) 2024-03-06T14:27:52.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:52.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:52.924768+0000) 2024-03-06T14:27:53.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:53.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:53.925004+0000) 2024-03-06T14:27:54.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:54.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:54.925313+0000) 2024-03-06T14:27:54.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:54.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:55.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:55.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:55.925725+0000) 2024-03-06T14:27:56.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:56.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:56.926041+0000) 2024-03-06T14:27:57.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:57.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:57.926337+0000) 2024-03-06T14:27:58.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:58.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:58.926683+0000) 2024-03-06T14:27:59.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:27:59.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:59.926989+0000) 2024-03-06T14:28:00.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:00.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:00.927294+0000) 2024-03-06T14:28:01.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:01.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:01.927587+0000) 2024-03-06T14:28:02.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:02.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:02.927926+0000) 2024-03-06T14:28:03.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:03.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:03.928240+0000) 2024-03-06T14:28:04.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:04.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:04.928545+0000) 2024-03-06T14:28:05.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:05.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:05.928862+0000) 2024-03-06T14:28:06.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:06.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:06.929166+0000) 2024-03-06T14:28:07.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:07.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:07.929480+0000) 2024-03-06T14:28:08.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:08.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:08.929772+0000) 2024-03-06T14:28:09.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:09.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:09.930058+0000) 2024-03-06T14:28:10.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:10.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:10.930373+0000) 2024-03-06T14:28:11.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:11.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:11.930704+0000) 2024-03-06T14:28:12.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:12.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:12.931019+0000) 2024-03-06T14:28:13.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:13.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:13.931289+0000) 2024-03-06T14:28:14.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:14.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:14.931524+0000) 2024-03-06T14:28:15.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:15.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:15.931788+0000) 2024-03-06T14:28:16.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:16.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:16.932048+0000) 2024-03-06T14:28:17.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:17.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:17.932322+0000) 2024-03-06T14:28:18.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:18.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:18.932522+0000) 2024-03-06T14:28:19.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:19.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:19.932802+0000) 2024-03-06T14:28:20.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:20.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:20.933069+0000) 2024-03-06T14:28:21.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:21.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:21.933331+0000) 2024-03-06T14:28:22.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:22.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:22.933531+0000) 2024-03-06T14:28:23.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:23.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:23.933781+0000) 2024-03-06T14:28:24.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:24.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:24.934014+0000) 2024-03-06T14:28:25.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:25.123 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:25.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:25.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:25.934275+0000) 2024-03-06T14:28:26.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:26.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:26.934527+0000) 2024-03-06T14:28:27.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:27.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:27.934770+0000) 2024-03-06T14:28:28.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:28.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:28.935023+0000) 2024-03-06T14:28:29.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:29.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:29.935278+0000) 2024-03-06T14:28:30.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:30.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:30.935521+0000) 2024-03-06T14:28:31.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:31.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:31.935787+0000) 2024-03-06T14:28:32.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:32.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:32.936040+0000) 2024-03-06T14:28:33.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:33.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:33.936296+0000) 2024-03-06T14:28:34.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:34.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:34.936530+0000) 2024-03-06T14:28:35.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:35.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:35.936788+0000) 2024-03-06T14:28:36.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:36.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:36.937045+0000) 2024-03-06T14:28:37.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:37.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:37.937311+0000) 2024-03-06T14:28:38.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:38.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:38.937542+0000) 2024-03-06T14:28:39.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:39.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:39.937797+0000) 2024-03-06T14:28:40.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:40.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:40.937998+0000) 2024-03-06T14:28:41.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:41.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:41.938252+0000) 2024-03-06T14:28:42.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:42.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:42.938507+0000) 2024-03-06T14:28:43.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:43.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:43.938798+0000) 2024-03-06T14:28:44.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:44.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:44.939046+0000) 2024-03-06T14:28:45.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:45.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:45.939292+0000) 2024-03-06T14:28:46.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:46.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:46.939530+0000) 2024-03-06T14:28:47.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:47.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:47.939773+0000) 2024-03-06T14:28:48.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:48.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:48.940030+0000) 2024-03-06T14:28:49.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:49.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:49.940292+0000) 2024-03-06T14:28:50.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:50.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:50.940545+0000) 2024-03-06T14:28:51.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:51.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:51.940796+0000) 2024-03-06T14:28:52.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:52.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:52.941035+0000) 2024-03-06T14:28:53.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:53.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:53.941230+0000) 2024-03-06T14:28:54.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:54.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:54.941458+0000) 2024-03-06T14:28:55.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:55.281 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:55.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:55.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:55.941736+0000) 2024-03-06T14:28:56.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:56.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:56.941956+0000) 2024-03-06T14:28:57.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:57.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:57.942170+0000) 2024-03-06T14:28:58.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:58.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:58.942425+0000) 2024-03-06T14:28:59.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:28:59.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:59.942723+0000) 2024-03-06T14:29:00.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:00.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:00.942965+0000) 2024-03-06T14:29:01.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:01.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:01.943155+0000) 2024-03-06T14:29:02.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:02.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:02.943391+0000) 2024-03-06T14:29:03.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:03.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:03.943671+0000) 2024-03-06T14:29:04.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:04.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:04.943923+0000) 2024-03-06T14:29:05.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:05.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:05.944163+0000) 2024-03-06T14:29:06.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:06.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:06.944415+0000) 2024-03-06T14:29:07.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:07.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:07.944692+0000) 2024-03-06T14:29:08.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:08.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:08.944928+0000) 2024-03-06T14:29:09.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:09.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:09.945184+0000) 2024-03-06T14:29:10.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:10.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:10.945431+0000) 2024-03-06T14:29:11.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:11.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:11.945724+0000) 2024-03-06T14:29:12.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:12.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:12.945911+0000) 2024-03-06T14:29:13.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:13.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:13.946149+0000) 2024-03-06T14:29:14.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:14.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:14.946380+0000) 2024-03-06T14:29:15.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:15.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:15.946612+0000) 2024-03-06T14:29:16.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:16.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:16.946910+0000) 2024-03-06T14:29:17.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:17.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:17.947170+0000) 2024-03-06T14:29:18.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:18.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:18.947352+0000) 2024-03-06T14:29:19.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:19.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:19.947606+0000) 2024-03-06T14:29:20.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:20.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:20.947907+0000) 2024-03-06T14:29:21.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:21.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:21.948176+0000) 2024-03-06T14:29:22.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:22.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:22.948428+0000) 2024-03-06T14:29:23.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:23.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:23.948739+0000) 2024-03-06T14:29:24.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:24.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:24.948989+0000) 2024-03-06T14:29:25.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:25.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:25.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:25.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:25.949248+0000) 2024-03-06T14:29:26.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:26.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:26.949500+0000) 2024-03-06T14:29:27.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:27.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:27.949754+0000) 2024-03-06T14:29:28.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:28.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:28.950005+0000) 2024-03-06T14:29:29.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:29.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:29.950270+0000) 2024-03-06T14:29:30.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:30.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:30.950508+0000) 2024-03-06T14:29:31.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:31.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:31.950831+0000) 2024-03-06T14:29:32.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:32.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:32.951083+0000) 2024-03-06T14:29:33.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:33.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:33.951333+0000) 2024-03-06T14:29:34.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:34.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:34.951595+0000) 2024-03-06T14:29:35.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:35.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:35.951924+0000) 2024-03-06T14:29:36.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:36.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:36.952187+0000) 2024-03-06T14:29:37.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:37.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:37.952436+0000) 2024-03-06T14:29:38.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:38.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:38.952734+0000) 2024-03-06T14:29:39.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:39.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:39.952974+0000) 2024-03-06T14:29:40.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:40.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:40.953230+0000) 2024-03-06T14:29:41.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:41.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:41.953477+0000) 2024-03-06T14:29:42.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:42.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:42.953757+0000) 2024-03-06T14:29:43.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:43.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:43.954000+0000) 2024-03-06T14:29:44.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:44.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:44.954242+0000) 2024-03-06T14:29:45.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:45.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:45.954482+0000) 2024-03-06T14:29:46.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:46.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:46.954783+0000) 2024-03-06T14:29:47.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:47.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:47.955043+0000) 2024-03-06T14:29:48.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:48.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:48.955293+0000) 2024-03-06T14:29:49.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:49.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:49.955557+0000) 2024-03-06T14:29:50.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:50.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:50.955813+0000) 2024-03-06T14:29:51.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:51.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:51.956074+0000) 2024-03-06T14:29:52.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:52.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:52.956326+0000) 2024-03-06T14:29:53.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:53.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:53.956591+0000) 2024-03-06T14:29:54.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:54.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:54.956855+0000) 2024-03-06T14:29:55.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:55.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:55.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:55.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:55.957124+0000) 2024-03-06T14:29:56.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:56.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:56.957371+0000) 2024-03-06T14:29:57.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:57.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:57.957611+0000) 2024-03-06T14:29:58.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:58.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:58.957901+0000) 2024-03-06T14:29:59.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:29:59.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:59.958100+0000) 2024-03-06T14:30:00.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:00.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:00.958363+0000) 2024-03-06T14:30:01.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:01.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:01.958620+0000) 2024-03-06T14:30:02.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:02.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:02.958919+0000) 2024-03-06T14:30:03.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:03.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:03.959187+0000) 2024-03-06T14:30:04.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:04.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:04.959436+0000) 2024-03-06T14:30:05.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:05.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:05.959734+0000) 2024-03-06T14:30:06.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:06.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:06.959969+0000) 2024-03-06T14:30:07.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:07.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:07.960214+0000) 2024-03-06T14:30:08.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:08.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:08.960459+0000) 2024-03-06T14:30:09.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:09.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:09.960750+0000) 2024-03-06T14:30:10.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:10.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:10.960944+0000) 2024-03-06T14:30:11.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:11.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:11.961182+0000) 2024-03-06T14:30:12.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:12.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:12.961423+0000) 2024-03-06T14:30:13.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:13.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:13.961717+0000) 2024-03-06T14:30:14.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:14.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:14.961977+0000) 2024-03-06T14:30:15.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:15.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:15.962169+0000) 2024-03-06T14:30:16.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:16.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:16.962423+0000) 2024-03-06T14:30:17.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:17.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:17.962721+0000) 2024-03-06T14:30:18.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:18.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:18.962975+0000) 2024-03-06T14:30:19.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:19.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:19.963265+0000) 2024-03-06T14:30:20.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:20.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:20.963517+0000) 2024-03-06T14:30:21.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:21.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:21.963762+0000) 2024-03-06T14:30:22.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:22.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:22.964015+0000) 2024-03-06T14:30:23.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:23.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:23.964277+0000) 2024-03-06T14:30:24.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:24.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:24.964532+0000) 2024-03-06T14:30:25.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:25.740 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:25.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:25.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:25.964806+0000) 2024-03-06T14:30:26.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:26.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:26.965061+0000) 2024-03-06T14:30:27.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:27.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:27.965301+0000) 2024-03-06T14:30:28.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:28.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:28.965538+0000) 2024-03-06T14:30:29.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:29.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:29.965790+0000) 2024-03-06T14:30:30.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:30.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:30.966057+0000) 2024-03-06T14:30:31.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:31.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:31.966314+0000) 2024-03-06T14:30:32.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:32.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:32.966568+0000) 2024-03-06T14:30:33.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:33.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:33.966813+0000) 2024-03-06T14:30:34.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:34.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:34.967048+0000) 2024-03-06T14:30:35.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:35.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:35.967293+0000) 2024-03-06T14:30:36.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:36.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:36.967543+0000) 2024-03-06T14:30:37.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:37.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:37.967803+0000) 2024-03-06T14:30:38.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:38.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:38.968056+0000) 2024-03-06T14:30:39.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:39.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:39.968292+0000) 2024-03-06T14:30:40.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:40.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:40.968539+0000) 2024-03-06T14:30:41.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:41.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:41.968790+0000) 2024-03-06T14:30:42.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:42.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:42.969037+0000) 2024-03-06T14:30:43.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:43.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:43.969293+0000) 2024-03-06T14:30:44.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:44.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:44.969533+0000) 2024-03-06T14:30:45.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:45.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:45.969790+0000) 2024-03-06T14:30:46.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:46.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:46.970049+0000) 2024-03-06T14:30:47.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:47.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:47.970305+0000) 2024-03-06T14:30:48.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:48.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:48.970559+0000) 2024-03-06T14:30:49.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:49.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:49.970808+0000) 2024-03-06T14:30:50.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:50.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:50.971071+0000) 2024-03-06T14:30:51.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:51.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:51.971315+0000) 2024-03-06T14:30:52.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:52.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:52.971546+0000) 2024-03-06T14:30:53.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:53.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:53.971781+0000) 2024-03-06T14:30:54.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:54.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:54.972042+0000) 2024-03-06T14:30:55.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:55.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:55.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:55.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:55.972277+0000) 2024-03-06T14:30:56.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:56.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:56.972515+0000) 2024-03-06T14:30:57.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:57.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:57.972779+0000) 2024-03-06T14:30:58.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:58.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:58.973032+0000) 2024-03-06T14:30:59.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:30:59.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:59.973274+0000) 2024-03-06T14:31:00.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:00.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:00.973534+0000) 2024-03-06T14:31:01.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:01.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:01.973780+0000) 2024-03-06T14:31:02.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:02.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:02.974037+0000) 2024-03-06T14:31:03.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:03.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:03.974293+0000) 2024-03-06T14:31:04.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:04.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:04.974541+0000) 2024-03-06T14:31:05.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:05.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:05.974820+0000) 2024-03-06T14:31:06.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:06.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:06.975055+0000) 2024-03-06T14:31:07.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:07.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:07.975317+0000) 2024-03-06T14:31:08.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:08.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:08.975576+0000) 2024-03-06T14:31:09.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:09.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:09.975815+0000) 2024-03-06T14:31:10.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:10.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:10.976059+0000) 2024-03-06T14:31:11.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:11.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:11.976242+0000) 2024-03-06T14:31:12.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:12.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:12.976485+0000) 2024-03-06T14:31:13.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:13.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:13.976749+0000) 2024-03-06T14:31:14.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:14.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:14.977003+0000) 2024-03-06T14:31:15.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:15.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:15.977258+0000) 2024-03-06T14:31:16.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:16.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:16.977507+0000) 2024-03-06T14:31:17.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:17.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:17.977766+0000) 2024-03-06T14:31:18.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:18.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:18.978076+0000) 2024-03-06T14:31:19.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:19.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:19.978330+0000) 2024-03-06T14:31:20.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:20.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:20.978571+0000) 2024-03-06T14:31:21.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:21.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:21.978825+0000) 2024-03-06T14:31:22.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:22.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:22.979078+0000) 2024-03-06T14:31:23.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:23.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:23.979335+0000) 2024-03-06T14:31:24.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:24.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:24.979580+0000) 2024-03-06T14:31:25.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:25.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:25.979813+0000) 2024-03-06T14:31:26.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:26.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:26.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:26.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:26.980067+0000) 2024-03-06T14:31:27.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:27.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:27.980310+0000) 2024-03-06T14:31:28.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:28.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:28.980545+0000) 2024-03-06T14:31:29.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:29.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:29.980812+0000) 2024-03-06T14:31:30.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:30.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:30.981016+0000) 2024-03-06T14:31:31.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:31.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:31.981266+0000) 2024-03-06T14:31:32.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:32.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:32.981515+0000) 2024-03-06T14:31:33.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:33.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:33.981748+0000) 2024-03-06T14:31:34.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:34.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:34.982008+0000) 2024-03-06T14:31:35.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:35.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:35.982267+0000) 2024-03-06T14:31:36.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:36.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:36.982515+0000) 2024-03-06T14:31:37.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:37.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:37.982772+0000) 2024-03-06T14:31:38.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:38.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:38.983015+0000) 2024-03-06T14:31:39.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:39.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:39.983258+0000) 2024-03-06T14:31:40.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:40.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:40.983501+0000) 2024-03-06T14:31:41.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:41.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:41.983756+0000) 2024-03-06T14:31:42.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:42.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:42.984009+0000) 2024-03-06T14:31:43.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:43.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:43.984259+0000) 2024-03-06T14:31:44.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:44.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:44.984512+0000) 2024-03-06T14:31:45.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:45.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:45.984790+0000) 2024-03-06T14:31:46.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:46.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:46.985049+0000) 2024-03-06T14:31:47.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:47.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:47.985290+0000) 2024-03-06T14:31:48.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:48.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:48.985538+0000) 2024-03-06T14:31:49.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:49.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:49.985787+0000) 2024-03-06T14:31:50.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:50.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:50.985988+0000) 2024-03-06T14:31:51.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:51.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:51.986176+0000) 2024-03-06T14:31:52.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:52.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:52.986424+0000) 2024-03-06T14:31:53.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:53.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:53.986723+0000) 2024-03-06T14:31:54.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:54.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:54.986972+0000) 2024-03-06T14:31:55.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:55.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:55.987232+0000) 2024-03-06T14:31:56.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:56.208 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:56.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:56.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:56.987468+0000) 2024-03-06T14:31:57.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:57.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:57.987736+0000) 2024-03-06T14:31:58.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:58.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:58.987985+0000) 2024-03-06T14:31:59.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:31:59.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:59.988249+0000) 2024-03-06T14:32:00.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:00.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:00.988493+0000) 2024-03-06T14:32:01.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:01.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:01.988807+0000) 2024-03-06T14:32:02.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:02.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:02.989058+0000) 2024-03-06T14:32:03.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:03.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:03.989308+0000) 2024-03-06T14:32:04.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:04.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:04.989557+0000) 2024-03-06T14:32:05.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:05.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:05.989814+0000) 2024-03-06T14:32:06.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:06.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:06.990046+0000) 2024-03-06T14:32:07.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:07.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:07.990308+0000) 2024-03-06T14:32:08.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:08.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:08.990540+0000) 2024-03-06T14:32:09.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:09.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:09.990782+0000) 2024-03-06T14:32:10.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:10.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:10.991033+0000) 2024-03-06T14:32:11.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:11.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:11.991276+0000) 2024-03-06T14:32:12.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:12.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:12.991525+0000) 2024-03-06T14:32:13.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:13.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:13.991757+0000) 2024-03-06T14:32:14.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:14.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:14.991952+0000) 2024-03-06T14:32:15.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:15.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:15.992143+0000) 2024-03-06T14:32:16.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:16.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:16.992394+0000) 2024-03-06T14:32:17.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:17.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:17.992685+0000) 2024-03-06T14:32:18.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:18.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:18.992874+0000) 2024-03-06T14:32:19.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:19.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:19.993135+0000) 2024-03-06T14:32:20.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:20.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:20.993393+0000) 2024-03-06T14:32:21.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:21.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:21.993686+0000) 2024-03-06T14:32:22.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:22.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:22.993923+0000) 2024-03-06T14:32:23.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:23.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:23.994176+0000) 2024-03-06T14:32:24.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:24.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:24.994356+0000) 2024-03-06T14:32:25.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:25.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:25.994612+0000) 2024-03-06T14:32:26.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:26.351 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:26.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:26.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:26.994889+0000) 2024-03-06T14:32:27.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:27.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:27.995142+0000) 2024-03-06T14:32:28.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:28.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:28.995379+0000) 2024-03-06T14:32:29.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:29.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:29.995678+0000) 2024-03-06T14:32:30.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:30.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:30.995948+0000) 2024-03-06T14:32:31.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:31.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:31.996140+0000) 2024-03-06T14:32:32.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:32.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:32.996399+0000) 2024-03-06T14:32:33.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:33.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:33.996679+0000) 2024-03-06T14:32:34.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:34.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:34.996928+0000) 2024-03-06T14:32:35.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:35.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:35.997173+0000) 2024-03-06T14:32:36.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:36.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:36.997429+0000) 2024-03-06T14:32:37.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:37.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:37.997735+0000) 2024-03-06T14:32:38.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:38.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:38.997977+0000) 2024-03-06T14:32:39.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:39.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:39.998211+0000) 2024-03-06T14:32:40.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:40.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:40.998460+0000) 2024-03-06T14:32:41.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:41.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:41.998763+0000) 2024-03-06T14:32:42.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:42.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:42.999001+0000) 2024-03-06T14:32:43.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:43.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:43.999266+0000) 2024-03-06T14:32:44.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:44.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:44.999451+0000) 2024-03-06T14:32:45.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:45.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:45.999732+0000) 2024-03-06T14:32:46.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:46.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:46.999963+0000) 2024-03-06T14:32:47.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:48.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:48.000229+0000) 2024-03-06T14:32:48.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:49.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:49.000487+0000) 2024-03-06T14:32:49.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:50.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:50.000753+0000) 2024-03-06T14:32:50.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:50.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:51.001017+0000) 2024-03-06T14:32:51.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:52.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:52.001282+0000) 2024-03-06T14:32:52.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:53.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:53.001520+0000) 2024-03-06T14:32:54.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:54.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:54.001776+0000) 2024-03-06T14:32:55.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:55.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:55.002071+0000) 2024-03-06T14:32:56.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:56.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:56.002272+0000) 2024-03-06T14:32:56.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:56.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:57.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:57.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:57.002513+0000) 2024-03-06T14:32:58.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:58.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:58.002799+0000) 2024-03-06T14:32:59.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:32:59.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:59.003054+0000) 2024-03-06T14:33:00.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:00.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:00.003341+0000) 2024-03-06T14:33:01.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:01.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:01.003582+0000) 2024-03-06T14:33:02.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:02.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:02.003821+0000) 2024-03-06T14:33:03.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:03.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:03.004071+0000) 2024-03-06T14:33:04.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:04.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:04.004319+0000) 2024-03-06T14:33:05.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:05.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:05.004572+0000) 2024-03-06T14:33:06.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:06.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:06.004828+0000) 2024-03-06T14:33:07.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:07.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:07.005078+0000) 2024-03-06T14:33:08.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:08.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:08.005322+0000) 2024-03-06T14:33:09.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:09.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:09.005553+0000) 2024-03-06T14:33:10.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:10.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:10.005802+0000) 2024-03-06T14:33:11.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:11.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:11.006061+0000) 2024-03-06T14:33:12.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:12.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:12.006310+0000) 2024-03-06T14:33:13.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:13.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:13.006558+0000) 2024-03-06T14:33:14.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:14.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:14.006816+0000) 2024-03-06T14:33:15.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:15.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:15.007058+0000) 2024-03-06T14:33:16.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:16.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:16.007312+0000) 2024-03-06T14:33:17.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:17.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:17.007541+0000) 2024-03-06T14:33:18.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:18.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:18.007801+0000) 2024-03-06T14:33:19.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:19.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:19.008049+0000) 2024-03-06T14:33:20.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:20.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:20.008306+0000) 2024-03-06T14:33:21.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:21.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:21.008571+0000) 2024-03-06T14:33:22.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:22.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:22.008829+0000) 2024-03-06T14:33:23.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:23.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:23.009086+0000) 2024-03-06T14:33:24.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:24.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:24.009345+0000) 2024-03-06T14:33:25.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:25.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:25.009594+0000) 2024-03-06T14:33:26.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:26.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:26.009866+0000) 2024-03-06T14:33:26.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:26.645 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:27.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:27.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:27.010102+0000) 2024-03-06T14:33:28.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:28.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:28.010359+0000) 2024-03-06T14:33:29.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:29.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:29.010612+0000) 2024-03-06T14:33:30.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:30.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:30.010907+0000) 2024-03-06T14:33:31.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:31.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:31.011168+0000) 2024-03-06T14:33:32.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:32.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:32.011427+0000) 2024-03-06T14:33:33.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:33.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:33.011710+0000) 2024-03-06T14:33:34.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:34.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:34.011972+0000) 2024-03-06T14:33:35.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:35.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:35.012233+0000) 2024-03-06T14:33:36.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:36.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:36.012480+0000) 2024-03-06T14:33:37.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:37.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:37.012756+0000) 2024-03-06T14:33:38.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:38.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:38.013004+0000) 2024-03-06T14:33:39.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:39.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:39.013244+0000) 2024-03-06T14:33:40.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:40.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:40.013484+0000) 2024-03-06T14:33:41.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:41.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:41.013735+0000) 2024-03-06T14:33:42.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:42.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:42.013985+0000) 2024-03-06T14:33:43.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:43.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:43.014298+0000) 2024-03-06T14:33:44.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:44.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:44.014550+0000) 2024-03-06T14:33:45.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:45.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:45.014806+0000) 2024-03-06T14:33:46.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:46.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:46.015065+0000) 2024-03-06T14:33:47.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:47.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:47.015315+0000) 2024-03-06T14:33:48.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:48.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:48.015565+0000) 2024-03-06T14:33:49.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:49.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:49.015833+0000) 2024-03-06T14:33:50.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:50.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:50.016112+0000) 2024-03-06T14:33:51.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:51.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:51.016383+0000) 2024-03-06T14:33:52.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:52.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:52.016706+0000) 2024-03-06T14:33:53.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:53.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:53.016986+0000) 2024-03-06T14:33:54.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:54.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:54.017231+0000) 2024-03-06T14:33:55.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:55.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:55.017498+0000) 2024-03-06T14:33:56.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:56.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:56.017754+0000) 2024-03-06T14:33:56.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:56.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:57.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:57.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:57.018027+0000) 2024-03-06T14:33:58.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:58.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:58.018220+0000) 2024-03-06T14:33:59.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:33:59.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:59.018456+0000) 2024-03-06T14:34:00.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:00.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:00.018736+0000) 2024-03-06T14:34:01.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:01.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:01.018999+0000) 2024-03-06T14:34:02.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:02.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:02.019259+0000) 2024-03-06T14:34:03.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:03.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:03.019540+0000) 2024-03-06T14:34:04.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:04.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:04.019812+0000) 2024-03-06T14:34:05.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:05.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:05.020092+0000) 2024-03-06T14:34:06.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:06.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:06.020341+0000) 2024-03-06T14:34:07.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:07.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:07.020577+0000) 2024-03-06T14:34:08.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:08.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:08.020848+0000) 2024-03-06T14:34:09.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:09.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:09.021131+0000) 2024-03-06T14:34:10.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:10.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:10.021395+0000) 2024-03-06T14:34:11.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:11.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:11.021716+0000) 2024-03-06T14:34:12.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:12.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:12.021990+0000) 2024-03-06T14:34:13.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:13.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:13.022274+0000) 2024-03-06T14:34:14.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:14.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:14.022509+0000) 2024-03-06T14:34:15.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:15.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:15.022766+0000) 2024-03-06T14:34:16.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:16.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:16.023021+0000) 2024-03-06T14:34:17.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:17.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:17.023274+0000) 2024-03-06T14:34:18.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:18.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:18.023552+0000) 2024-03-06T14:34:19.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:19.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:19.023807+0000) 2024-03-06T14:34:20.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:20.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:20.024077+0000) 2024-03-06T14:34:21.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:21.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:21.024324+0000) 2024-03-06T14:34:22.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:22.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:22.024578+0000) 2024-03-06T14:34:23.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:23.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:23.024840+0000) 2024-03-06T14:34:24.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:24.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:24.025093+0000) 2024-03-06T14:34:25.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:25.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:25.025346+0000) 2024-03-06T14:34:26.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:26.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:26.025693+0000) 2024-03-06T14:34:26.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:26.935 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:27.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:27.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:27.025936+0000) 2024-03-06T14:34:28.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:28.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:28.026151+0000) 2024-03-06T14:34:29.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:29.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:29.026404+0000) 2024-03-06T14:34:30.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:30.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:30.026704+0000) 2024-03-06T14:34:31.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:31.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:31.026972+0000) 2024-03-06T14:34:32.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:32.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:32.027215+0000) 2024-03-06T14:34:33.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:33.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:33.027463+0000) 2024-03-06T14:34:34.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:34.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:34.027772+0000) 2024-03-06T14:34:35.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:35.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:35.028044+0000) 2024-03-06T14:34:36.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:36.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:36.028256+0000) 2024-03-06T14:34:37.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:37.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:37.028495+0000) 2024-03-06T14:34:38.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:38.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:38.028747+0000) 2024-03-06T14:34:39.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:39.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:39.029021+0000) 2024-03-06T14:34:40.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:40.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:40.029305+0000) 2024-03-06T14:34:41.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:41.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:41.029572+0000) 2024-03-06T14:34:42.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:42.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:42.029911+0000) 2024-03-06T14:34:43.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:43.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:43.030185+0000) 2024-03-06T14:34:44.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:44.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:44.030438+0000) 2024-03-06T14:34:45.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:45.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:45.030763+0000) 2024-03-06T14:34:46.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:46.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:46.030985+0000) 2024-03-06T14:34:47.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:47.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:47.031189+0000) 2024-03-06T14:34:48.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:48.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:48.031394+0000) 2024-03-06T14:34:49.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:49.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:49.031712+0000) 2024-03-06T14:34:50.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:50.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:50.031986+0000) 2024-03-06T14:34:51.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:51.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:51.032193+0000) 2024-03-06T14:34:52.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:52.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:52.032461+0000) 2024-03-06T14:34:53.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:53.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:53.032729+0000) 2024-03-06T14:34:54.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:54.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:54.032983+0000) 2024-03-06T14:34:55.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:55.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:55.033261+0000) 2024-03-06T14:34:56.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:56.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:56.033500+0000) 2024-03-06T14:34:57.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:57.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:57.033750+0000) 2024-03-06T14:34:57.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:57.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:58.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:58.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:58.034002+0000) 2024-03-06T14:34:59.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:34:59.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:59.034255+0000) 2024-03-06T14:35:00.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:00.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:00.034520+0000) 2024-03-06T14:35:01.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:01.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:01.034820+0000) 2024-03-06T14:35:02.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:02.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:02.035058+0000) 2024-03-06T14:35:03.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:03.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:03.035295+0000) 2024-03-06T14:35:04.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:04.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:04.035559+0000) 2024-03-06T14:35:05.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:05.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:05.035774+0000) 2024-03-06T14:35:06.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:06.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:06.036056+0000) 2024-03-06T14:35:07.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:07.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:07.036324+0000) 2024-03-06T14:35:08.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:08.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:08.036580+0000) 2024-03-06T14:35:09.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:09.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:09.036853+0000) 2024-03-06T14:35:10.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:10.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:10.037077+0000) 2024-03-06T14:35:11.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:11.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:11.037359+0000) 2024-03-06T14:35:12.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:12.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:12.037708+0000) 2024-03-06T14:35:13.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:13.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:13.037929+0000) 2024-03-06T14:35:14.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:14.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:14.038213+0000) 2024-03-06T14:35:15.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:15.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:15.038480+0000) 2024-03-06T14:35:16.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:16.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:16.038737+0000) 2024-03-06T14:35:17.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:17.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:17.039003+0000) 2024-03-06T14:35:18.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:18.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:18.039276+0000) 2024-03-06T14:35:19.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:19.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:19.039511+0000) 2024-03-06T14:35:20.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:20.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:20.039754+0000) 2024-03-06T14:35:21.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:21.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:21.040020+0000) 2024-03-06T14:35:22.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:22.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:22.040278+0000) 2024-03-06T14:35:23.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:23.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:23.040516+0000) 2024-03-06T14:35:24.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:24.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:24.040792+0000) 2024-03-06T14:35:25.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:25.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:25.041066+0000) 2024-03-06T14:35:26.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:26.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:26.041322+0000) 2024-03-06T14:35:27.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:27.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:27.041589+0000) 2024-03-06T14:35:27.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:27.242 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:28.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:28.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:28.041914+0000) 2024-03-06T14:35:29.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:29.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:29.042158+0000) 2024-03-06T14:35:30.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:30.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:30.042473+0000) 2024-03-06T14:35:31.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:31.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:31.042732+0000) 2024-03-06T14:35:32.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:32.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:32.042980+0000) 2024-03-06T14:35:33.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:33.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:33.043207+0000) 2024-03-06T14:35:34.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:34.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:34.043464+0000) 2024-03-06T14:35:35.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:35.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:35.043789+0000) 2024-03-06T14:35:36.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:36.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:36.044048+0000) 2024-03-06T14:35:37.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:37.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:37.044301+0000) 2024-03-06T14:35:38.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:38.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:38.044538+0000) 2024-03-06T14:35:39.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:39.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:39.044798+0000) 2024-03-06T14:35:40.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:40.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:40.045000+0000) 2024-03-06T14:35:41.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:41.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:41.045265+0000) 2024-03-06T14:35:42.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:42.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:42.045523+0000) 2024-03-06T14:35:43.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:43.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:43.045782+0000) 2024-03-06T14:35:44.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:44.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:44.046040+0000) 2024-03-06T14:35:45.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:45.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:45.046302+0000) 2024-03-06T14:35:46.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:46.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:46.046572+0000) 2024-03-06T14:35:47.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:47.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:47.046829+0000) 2024-03-06T14:35:48.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:48.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:48.047090+0000) 2024-03-06T14:35:49.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:49.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:49.047323+0000) 2024-03-06T14:35:50.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:50.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:50.047565+0000) 2024-03-06T14:35:51.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:51.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:51.047837+0000) 2024-03-06T14:35:52.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:52.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:52.048091+0000) 2024-03-06T14:35:53.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:53.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:53.048346+0000) 2024-03-06T14:35:54.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:54.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:54.048609+0000) 2024-03-06T14:35:55.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:55.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:55.048924+0000) 2024-03-06T14:35:56.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:56.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:56.049168+0000) 2024-03-06T14:35:57.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:57.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:57.049421+0000) 2024-03-06T14:35:57.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:57.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:58.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:58.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:58.049734+0000) 2024-03-06T14:35:59.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:35:59.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:59.049982+0000) 2024-03-06T14:36:00.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:00.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:00.050219+0000) 2024-03-06T14:36:01.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:01.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:01.050470+0000) 2024-03-06T14:36:02.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:02.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:02.050704+0000) 2024-03-06T14:36:03.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:03.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:03.050960+0000) 2024-03-06T14:36:04.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:04.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:04.051215+0000) 2024-03-06T14:36:05.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:05.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:05.051467+0000) 2024-03-06T14:36:06.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:06.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:06.051766+0000) 2024-03-06T14:36:07.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:07.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:07.052024+0000) 2024-03-06T14:36:08.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:08.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:08.052281+0000) 2024-03-06T14:36:09.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:09.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:09.052535+0000) 2024-03-06T14:36:10.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:10.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:10.052795+0000) 2024-03-06T14:36:11.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:11.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:11.053055+0000) 2024-03-06T14:36:12.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:12.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:12.053348+0000) 2024-03-06T14:36:13.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:13.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:13.053599+0000) 2024-03-06T14:36:14.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:14.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:14.053889+0000) 2024-03-06T14:36:15.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:15.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:15.054140+0000) 2024-03-06T14:36:16.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:16.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:16.054398+0000) 2024-03-06T14:36:17.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:17.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:17.054670+0000) 2024-03-06T14:36:18.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:18.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:18.054938+0000) 2024-03-06T14:36:19.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:19.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:19.055189+0000) 2024-03-06T14:36:20.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:20.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:20.055437+0000) 2024-03-06T14:36:21.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:21.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:21.055742+0000) 2024-03-06T14:36:22.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:22.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:22.055991+0000) 2024-03-06T14:36:23.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:23.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:23.056223+0000) 2024-03-06T14:36:24.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:24.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:24.056476+0000) 2024-03-06T14:36:25.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:25.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:25.056731+0000) 2024-03-06T14:36:26.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:26.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:26.056931+0000) 2024-03-06T14:36:27.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:27.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:27.057178+0000) 2024-03-06T14:36:27.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:27.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:28.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:28.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:28.057424+0000) 2024-03-06T14:36:29.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:29.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:29.057711+0000) 2024-03-06T14:36:30.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:30.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:30.057959+0000) 2024-03-06T14:36:31.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:31.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:31.058212+0000) 2024-03-06T14:36:32.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:32.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:32.058462+0000) 2024-03-06T14:36:33.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:33.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:33.058777+0000) 2024-03-06T14:36:34.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:34.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:34.059009+0000) 2024-03-06T14:36:35.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:35.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:35.059253+0000) 2024-03-06T14:36:36.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:36.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:36.059493+0000) 2024-03-06T14:36:37.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:37.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:37.059749+0000) 2024-03-06T14:36:38.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:38.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:38.059983+0000) 2024-03-06T14:36:39.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:39.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:39.060177+0000) 2024-03-06T14:36:40.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:40.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:40.060423+0000) 2024-03-06T14:36:41.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:41.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:41.060730+0000) 2024-03-06T14:36:42.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:42.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:42.060940+0000) 2024-03-06T14:36:43.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:43.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:43.061196+0000) 2024-03-06T14:36:44.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:44.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:44.061440+0000) 2024-03-06T14:36:45.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:45.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:45.061715+0000) 2024-03-06T14:36:46.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:46.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:46.061979+0000) 2024-03-06T14:36:47.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:47.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:47.062214+0000) 2024-03-06T14:36:48.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:48.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:48.062475+0000) 2024-03-06T14:36:49.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:49.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:49.062740+0000) 2024-03-06T14:36:50.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:50.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:50.063003+0000) 2024-03-06T14:36:51.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:51.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:51.063268+0000) 2024-03-06T14:36:52.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:52.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:52.063524+0000) 2024-03-06T14:36:53.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:53.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:53.063740+0000) 2024-03-06T14:36:54.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:54.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:54.063931+0000) 2024-03-06T14:36:55.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:55.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:55.064168+0000) 2024-03-06T14:36:56.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:56.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:56.064431+0000) 2024-03-06T14:36:57.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:57.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:57.064721+0000) 2024-03-06T14:36:57.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:57.679 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:58.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:58.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:58.064963+0000) 2024-03-06T14:36:59.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:36:59.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:59.065208+0000) 2024-03-06T14:37:00.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:00.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:00.065498+0000) 2024-03-06T14:37:01.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:01.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:01.065744+0000) 2024-03-06T14:37:02.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:02.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:02.066001+0000) 2024-03-06T14:37:03.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:03.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:03.066235+0000) 2024-03-06T14:37:04.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:04.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:04.066414+0000) 2024-03-06T14:37:05.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:05.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:05.066693+0000) 2024-03-06T14:37:06.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:06.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:06.066960+0000) 2024-03-06T14:37:07.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:07.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:07.067212+0000) 2024-03-06T14:37:08.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:08.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:08.067472+0000) 2024-03-06T14:37:09.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:09.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:09.067749+0000) 2024-03-06T14:37:10.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:10.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:10.067986+0000) 2024-03-06T14:37:11.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:11.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:11.068230+0000) 2024-03-06T14:37:12.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:12.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:12.068478+0000) 2024-03-06T14:37:13.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:13.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:13.068759+0000) 2024-03-06T14:37:14.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:14.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:14.069014+0000) 2024-03-06T14:37:15.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:15.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:15.069275+0000) 2024-03-06T14:37:16.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:16.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:16.069536+0000) 2024-03-06T14:37:17.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:17.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:17.069791+0000) 2024-03-06T14:37:18.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:18.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:18.069991+0000) 2024-03-06T14:37:19.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:19.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:19.070231+0000) 2024-03-06T14:37:20.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:20.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:20.070488+0000) 2024-03-06T14:37:21.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:21.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:21.070737+0000) 2024-03-06T14:37:22.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:22.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:22.070987+0000) 2024-03-06T14:37:23.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:23.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:23.071242+0000) 2024-03-06T14:37:24.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:24.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:24.071497+0000) 2024-03-06T14:37:25.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:25.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:25.071741+0000) 2024-03-06T14:37:26.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:26.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:26.071984+0000) 2024-03-06T14:37:27.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:27.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:27.072224+0000) 2024-03-06T14:37:27.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:27.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:28.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:28.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:28.072471+0000) 2024-03-06T14:37:29.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:29.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:29.072795+0000) 2024-03-06T14:37:30.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:30.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:30.073043+0000) 2024-03-06T14:37:31.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:31.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:31.073305+0000) 2024-03-06T14:37:32.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:32.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:32.073543+0000) 2024-03-06T14:37:33.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:33.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:33.073797+0000) 2024-03-06T14:37:34.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:34.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:34.073982+0000) 2024-03-06T14:37:35.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:35.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:35.074235+0000) 2024-03-06T14:37:36.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:36.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:36.074485+0000) 2024-03-06T14:37:37.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:37.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:37.074744+0000) 2024-03-06T14:37:38.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:38.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:38.074987+0000) 2024-03-06T14:37:39.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:39.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:39.075179+0000) 2024-03-06T14:37:40.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:40.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:40.075434+0000) 2024-03-06T14:37:41.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:41.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:41.075755+0000) 2024-03-06T14:37:42.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:42.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:42.075995+0000) 2024-03-06T14:37:43.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:43.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:43.076253+0000) 2024-03-06T14:37:44.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:44.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:44.076511+0000) 2024-03-06T14:37:45.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:45.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:45.076761+0000) 2024-03-06T14:37:46.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:46.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:46.077019+0000) 2024-03-06T14:37:47.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:47.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:47.077213+0000) 2024-03-06T14:37:48.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:48.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:48.077447+0000) 2024-03-06T14:37:49.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:49.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:49.077747+0000) 2024-03-06T14:37:50.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:50.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:50.077982+0000) 2024-03-06T14:37:51.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:51.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:51.078248+0000) 2024-03-06T14:37:52.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:52.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:52.078506+0000) 2024-03-06T14:37:53.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:53.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:53.078768+0000) 2024-03-06T14:37:54.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:54.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:54.079031+0000) 2024-03-06T14:37:55.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:55.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:55.079264+0000) 2024-03-06T14:37:56.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:56.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:56.079523+0000) 2024-03-06T14:37:57.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:57.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:57.079759+0000) 2024-03-06T14:37:58.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:58.054 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:58.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:58.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:58.079917+0000) 2024-03-06T14:37:59.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:37:59.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:59.080153+0000) 2024-03-06T14:38:00.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:00.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:00.080351+0000) 2024-03-06T14:38:01.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:01.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:01.080615+0000) 2024-03-06T14:38:02.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:02.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:02.080925+0000) 2024-03-06T14:38:03.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:03.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:03.081179+0000) 2024-03-06T14:38:04.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:04.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:04.081435+0000) 2024-03-06T14:38:05.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:05.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:05.081752+0000) 2024-03-06T14:38:06.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:06.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:06.081950+0000) 2024-03-06T14:38:07.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:07.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:07.082209+0000) 2024-03-06T14:38:08.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:08.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:08.082471+0000) 2024-03-06T14:38:09.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:09.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:09.082755+0000) 2024-03-06T14:38:10.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:10.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:10.083017+0000) 2024-03-06T14:38:11.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:11.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:11.083268+0000) 2024-03-06T14:38:12.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:12.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:12.083506+0000) 2024-03-06T14:38:13.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:13.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:13.083754+0000) 2024-03-06T14:38:14.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:14.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:14.084018+0000) 2024-03-06T14:38:15.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:15.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:15.084278+0000) 2024-03-06T14:38:16.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:16.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:16.084519+0000) 2024-03-06T14:38:17.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:17.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:17.084773+0000) 2024-03-06T14:38:18.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:18.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:18.085034+0000) 2024-03-06T14:38:19.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:19.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:19.085290+0000) 2024-03-06T14:38:20.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:20.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:20.085546+0000) 2024-03-06T14:38:21.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:21.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:21.085791+0000) 2024-03-06T14:38:22.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:22.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:22.085976+0000) 2024-03-06T14:38:23.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:23.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:23.086217+0000) 2024-03-06T14:38:24.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:24.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:24.086479+0000) 2024-03-06T14:38:25.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:25.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:25.086822+0000) 2024-03-06T14:38:26.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:26.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:26.087070+0000) 2024-03-06T14:38:27.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:27.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:27.087304+0000) 2024-03-06T14:38:28.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:28.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:28.087545+0000) 2024-03-06T14:38:28.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:28.195 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:29.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:29.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:29.087793+0000) 2024-03-06T14:38:30.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:30.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:30.087976+0000) 2024-03-06T14:38:31.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:31.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:31.088223+0000) 2024-03-06T14:38:32.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:32.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:32.088464+0000) 2024-03-06T14:38:33.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:33.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:33.088765+0000) 2024-03-06T14:38:34.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:34.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:34.088984+0000) 2024-03-06T14:38:35.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:35.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:35.089178+0000) 2024-03-06T14:38:36.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:36.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:36.089445+0000) 2024-03-06T14:38:37.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:37.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:37.089734+0000) 2024-03-06T14:38:38.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:38.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:38.089972+0000) 2024-03-06T14:38:39.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:39.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:39.090244+0000) 2024-03-06T14:38:40.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:40.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:40.090495+0000) 2024-03-06T14:38:41.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:41.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:41.090807+0000) 2024-03-06T14:38:42.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:42.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:42.091122+0000) 2024-03-06T14:38:43.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:43.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:43.091363+0000) 2024-03-06T14:38:44.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:44.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:44.091609+0000) 2024-03-06T14:38:45.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:45.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:45.091904+0000) 2024-03-06T14:38:46.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:46.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:46.092144+0000) 2024-03-06T14:38:47.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:47.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:47.092448+0000) 2024-03-06T14:38:48.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:48.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:48.092713+0000) 2024-03-06T14:38:49.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:49.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:49.092998+0000) 2024-03-06T14:38:50.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:50.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:50.093276+0000) 2024-03-06T14:38:51.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:51.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:51.093545+0000) 2024-03-06T14:38:52.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:52.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:52.093815+0000) 2024-03-06T14:38:53.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:53.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:53.094079+0000) 2024-03-06T14:38:54.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:54.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:54.094369+0000) 2024-03-06T14:38:55.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:55.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:55.094622+0000) 2024-03-06T14:38:56.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:56.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:56.094948+0000) 2024-03-06T14:38:57.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:57.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:57.095210+0000) 2024-03-06T14:38:58.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:58.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:58.095458+0000) 2024-03-06T14:38:58.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:58.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:59.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:38:59.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:59.095766+0000) 2024-03-06T14:39:00.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:00.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:00.095952+0000) 2024-03-06T14:39:01.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:01.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:01.096213+0000) 2024-03-06T14:39:02.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:02.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:02.096414+0000) 2024-03-06T14:39:03.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:03.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:03.096808+0000) 2024-03-06T14:39:04.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:04.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:04.097088+0000) 2024-03-06T14:39:05.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:05.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:05.097398+0000) 2024-03-06T14:39:06.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:06.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:06.097684+0000) 2024-03-06T14:39:07.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:07.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:07.097957+0000) 2024-03-06T14:39:08.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:08.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:08.098221+0000) 2024-03-06T14:39:09.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:09.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:09.098480+0000) 2024-03-06T14:39:10.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:10.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:10.098743+0000) 2024-03-06T14:39:11.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:11.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:11.099028+0000) 2024-03-06T14:39:12.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:12.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:12.099274+0000) 2024-03-06T14:39:13.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:13.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:13.099558+0000) 2024-03-06T14:39:14.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:14.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:14.099839+0000) 2024-03-06T14:39:15.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:15.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:15.100132+0000) 2024-03-06T14:39:16.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:16.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:16.100439+0000) 2024-03-06T14:39:17.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:17.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:17.100775+0000) 2024-03-06T14:39:18.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:18.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:18.101091+0000) 2024-03-06T14:39:19.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:19.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:19.101394+0000) 2024-03-06T14:39:20.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:20.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:20.101666+0000) 2024-03-06T14:39:21.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:21.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:21.101992+0000) 2024-03-06T14:39:22.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:22.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:22.102278+0000) 2024-03-06T14:39:23.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:23.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:23.102576+0000) 2024-03-06T14:39:24.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:24.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:24.102879+0000) 2024-03-06T14:39:25.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:25.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:25.103185+0000) 2024-03-06T14:39:26.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:26.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:26.103495+0000) 2024-03-06T14:39:27.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:27.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:27.103808+0000) 2024-03-06T14:39:28.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:28.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:28.104049+0000) 2024-03-06T14:39:28.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:28.498 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:29.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:29.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:29.104355+0000) 2024-03-06T14:39:30.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:30.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:30.104711+0000) 2024-03-06T14:39:31.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:31.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:31.105065+0000) 2024-03-06T14:39:32.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:32.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:32.105376+0000) 2024-03-06T14:39:33.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:33.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:33.105712+0000) 2024-03-06T14:39:34.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:34.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:34.106014+0000) 2024-03-06T14:39:35.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:35.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:35.106302+0000) 2024-03-06T14:39:36.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:36.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:36.106616+0000) 2024-03-06T14:39:37.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:37.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:37.106892+0000) 2024-03-06T14:39:38.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:38.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:38.107197+0000) 2024-03-06T14:39:39.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:39.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:39.107486+0000) 2024-03-06T14:39:40.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:40.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:40.107839+0000) 2024-03-06T14:39:41.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:41.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:41.108151+0000) 2024-03-06T14:39:42.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:42.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:42.108450+0000) 2024-03-06T14:39:43.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:43.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:43.108777+0000) 2024-03-06T14:39:44.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:44.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:44.109095+0000) 2024-03-06T14:39:45.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:45.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:45.109385+0000) 2024-03-06T14:39:46.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:46.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:46.109735+0000) 2024-03-06T14:39:47.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:47.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:47.110040+0000) 2024-03-06T14:39:48.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:48.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:48.110361+0000) 2024-03-06T14:39:49.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:49.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:49.110719+0000) 2024-03-06T14:39:50.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:50.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:50.111024+0000) 2024-03-06T14:39:51.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:51.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:51.111293+0000) 2024-03-06T14:39:52.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:52.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:52.111614+0000) 2024-03-06T14:39:53.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:53.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:53.111909+0000) 2024-03-06T14:39:54.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:54.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:54.112144+0000) 2024-03-06T14:39:55.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:55.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:55.112381+0000) 2024-03-06T14:39:56.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:56.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:56.112664+0000) 2024-03-06T14:39:57.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:57.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:57.112918+0000) 2024-03-06T14:39:58.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:58.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:58.113170+0000) 2024-03-06T14:39:58.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:58.654 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:59.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:39:59.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:59.113406+0000) 2024-03-06T14:40:00.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:00.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:00.113706+0000) 2024-03-06T14:40:01.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:01.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:01.113963+0000) 2024-03-06T14:40:02.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:02.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:02.114223+0000) 2024-03-06T14:40:03.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:03.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:03.114474+0000) 2024-03-06T14:40:04.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:04.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:04.114751+0000) 2024-03-06T14:40:05.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:05.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:05.114951+0000) 2024-03-06T14:40:06.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:06.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:06.115214+0000) 2024-03-06T14:40:07.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:07.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:07.115459+0000) 2024-03-06T14:40:08.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:08.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:08.115742+0000) 2024-03-06T14:40:09.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:09.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:09.116178+0000) 2024-03-06T14:40:10.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:10.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:10.116462+0000) 2024-03-06T14:40:11.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:11.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:11.116725+0000) 2024-03-06T14:40:12.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:12.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:12.116976+0000) 2024-03-06T14:40:13.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:13.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:13.117238+0000) 2024-03-06T14:40:14.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:14.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:14.117485+0000) 2024-03-06T14:40:15.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:15.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:15.117740+0000) 2024-03-06T14:40:16.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:16.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:16.118000+0000) 2024-03-06T14:40:17.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:17.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:17.118361+0000) 2024-03-06T14:40:18.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:18.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:18.118825+0000) 2024-03-06T14:40:19.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:19.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:19.119078+0000) 2024-03-06T14:40:20.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:20.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:20.119264+0000) 2024-03-06T14:40:21.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:21.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:21.119519+0000) 2024-03-06T14:40:22.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:22.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:22.119796+0000) 2024-03-06T14:40:23.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:23.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:23.120074+0000) 2024-03-06T14:40:24.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:24.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:24.120351+0000) 2024-03-06T14:40:25.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:25.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:25.120780+0000) 2024-03-06T14:40:26.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:26.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:26.121096+0000) 2024-03-06T14:40:27.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:27.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:27.121411+0000) 2024-03-06T14:40:28.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:28.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:28.121792+0000) 2024-03-06T14:40:28.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:28.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:29.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:29.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:29.122051+0000) 2024-03-06T14:40:30.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:30.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:30.122255+0000) 2024-03-06T14:40:31.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:31.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:31.122530+0000) 2024-03-06T14:40:32.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:32.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:32.122772+0000) 2024-03-06T14:40:33.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:33.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:33.123016+0000) 2024-03-06T14:40:34.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:34.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:34.123284+0000) 2024-03-06T14:40:35.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:35.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:35.123545+0000) 2024-03-06T14:40:36.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:36.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:36.123788+0000) 2024-03-06T14:40:37.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:37.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:37.124026+0000) 2024-03-06T14:40:38.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:38.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:38.124220+0000) 2024-03-06T14:40:39.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:39.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:39.124490+0000) 2024-03-06T14:40:40.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:40.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:40.124785+0000) 2024-03-06T14:40:41.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:41.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:41.125223+0000) 2024-03-06T14:40:42.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:42.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:42.125546+0000) 2024-03-06T14:40:43.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:43.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:43.125816+0000) 2024-03-06T14:40:44.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:44.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:44.126114+0000) 2024-03-06T14:40:45.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:45.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:45.126360+0000) 2024-03-06T14:40:46.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:46.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:46.126604+0000) 2024-03-06T14:40:47.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:47.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:47.126937+0000) 2024-03-06T14:40:48.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:48.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:48.127242+0000) 2024-03-06T14:40:49.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:49.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:49.127557+0000) 2024-03-06T14:40:50.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:50.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:50.127871+0000) 2024-03-06T14:40:51.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:51.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:51.128157+0000) 2024-03-06T14:40:52.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:52.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:52.128430+0000) 2024-03-06T14:40:53.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:53.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:53.128686+0000) 2024-03-06T14:40:54.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:54.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:54.128929+0000) 2024-03-06T14:40:55.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:55.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:55.129187+0000) 2024-03-06T14:40:56.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:56.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:56.129490+0000) 2024-03-06T14:40:57.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:57.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:57.129757+0000) 2024-03-06T14:40:58.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:58.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:58.130005+0000) 2024-03-06T14:40:58.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:58.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:59.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:40:59.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:59.130330+0000) 2024-03-06T14:41:00.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:00.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:00.130622+0000) 2024-03-06T14:41:01.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:01.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:01.131017+0000) 2024-03-06T14:41:02.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:02.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:02.131310+0000) 2024-03-06T14:41:03.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:03.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:03.131593+0000) 2024-03-06T14:41:04.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:04.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:04.131919+0000) 2024-03-06T14:41:05.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:05.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:05.132161+0000) 2024-03-06T14:41:06.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:06.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:06.132467+0000) 2024-03-06T14:41:07.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:07.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:07.132774+0000) 2024-03-06T14:41:08.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:08.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:08.133053+0000) 2024-03-06T14:41:09.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:09.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:09.133357+0000) 2024-03-06T14:41:10.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:10.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:10.133699+0000) 2024-03-06T14:41:11.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:11.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:11.134031+0000) 2024-03-06T14:41:12.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:12.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:12.134322+0000) 2024-03-06T14:41:13.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:13.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:13.134684+0000) 2024-03-06T14:41:14.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:14.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:14.135010+0000) 2024-03-06T14:41:15.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:15.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:15.135260+0000) 2024-03-06T14:41:16.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:16.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:16.135565+0000) 2024-03-06T14:41:17.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:17.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:17.135873+0000) 2024-03-06T14:41:18.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:18.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:18.136154+0000) 2024-03-06T14:41:19.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:19.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:19.136469+0000) 2024-03-06T14:41:20.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:20.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:20.136783+0000) 2024-03-06T14:41:21.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:21.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:21.137098+0000) 2024-03-06T14:41:22.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:22.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:22.137400+0000) 2024-03-06T14:41:23.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:23.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:23.137733+0000) 2024-03-06T14:41:24.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:24.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:24.138019+0000) 2024-03-06T14:41:25.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:25.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:25.138331+0000) 2024-03-06T14:41:26.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:26.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:26.138683+0000) 2024-03-06T14:41:27.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:27.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:27.138983+0000) 2024-03-06T14:41:28.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:28.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:28.139297+0000) 2024-03-06T14:41:29.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:29.086 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:29.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:29.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:29.139520+0000) 2024-03-06T14:41:30.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:30.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:30.139816+0000) 2024-03-06T14:41:31.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:31.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:31.140162+0000) 2024-03-06T14:41:32.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:32.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:32.140470+0000) 2024-03-06T14:41:33.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:33.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:33.140786+0000) 2024-03-06T14:41:34.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:34.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:34.141076+0000) 2024-03-06T14:41:35.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:35.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:35.141387+0000) 2024-03-06T14:41:36.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:36.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:36.141713+0000) 2024-03-06T14:41:37.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:37.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:37.142013+0000) 2024-03-06T14:41:38.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:38.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:38.142300+0000) 2024-03-06T14:41:39.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:39.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:39.142578+0000) 2024-03-06T14:41:40.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:40.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:40.142857+0000) 2024-03-06T14:41:41.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:41.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:41.143172+0000) 2024-03-06T14:41:42.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:42.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:42.143457+0000) 2024-03-06T14:41:43.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:43.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:43.143780+0000) 2024-03-06T14:41:44.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:44.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:44.144031+0000) 2024-03-06T14:41:45.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:45.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:45.144351+0000) 2024-03-06T14:41:46.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:46.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:46.144712+0000) 2024-03-06T14:41:47.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:47.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:47.144963+0000) 2024-03-06T14:41:48.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:48.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:48.145248+0000) 2024-03-06T14:41:49.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:49.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:49.145551+0000) 2024-03-06T14:41:50.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:50.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:50.145832+0000) 2024-03-06T14:41:51.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:51.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:51.146132+0000) 2024-03-06T14:41:52.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:52.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:52.146430+0000) 2024-03-06T14:41:53.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:53.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:53.146778+0000) 2024-03-06T14:41:54.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:54.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:54.147077+0000) 2024-03-06T14:41:55.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:55.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:55.147379+0000) 2024-03-06T14:41:56.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:56.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:56.147721+0000) 2024-03-06T14:41:57.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:57.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:57.148039+0000) 2024-03-06T14:41:58.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:58.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:58.148351+0000) 2024-03-06T14:41:59.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:41:59.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:59.148696+0000) 2024-03-06T14:41:59.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:59.242 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:00.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:00.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:00.148947+0000) 2024-03-06T14:42:01.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:01.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:01.149229+0000) 2024-03-06T14:42:02.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:02.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:02.149519+0000) 2024-03-06T14:42:03.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:03.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:03.149803+0000) 2024-03-06T14:42:04.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:04.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:04.150097+0000) 2024-03-06T14:42:05.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:05.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:05.150411+0000) 2024-03-06T14:42:06.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:06.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:06.150752+0000) 2024-03-06T14:42:07.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:07.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:07.151045+0000) 2024-03-06T14:42:08.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:08.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:08.151282+0000) 2024-03-06T14:42:09.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:09.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:09.151596+0000) 2024-03-06T14:42:10.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:10.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:10.152010+0000) 2024-03-06T14:42:11.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:11.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:11.152312+0000) 2024-03-06T14:42:12.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:12.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:12.152613+0000) 2024-03-06T14:42:13.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:13.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:13.152952+0000) 2024-03-06T14:42:14.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:14.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:14.153270+0000) 2024-03-06T14:42:15.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:15.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:15.153573+0000) 2024-03-06T14:42:16.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:16.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:16.153896+0000) 2024-03-06T14:42:17.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:17.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:17.154197+0000) 2024-03-06T14:42:18.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:18.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:18.154481+0000) 2024-03-06T14:42:19.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:19.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:19.154783+0000) 2024-03-06T14:42:20.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:20.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:20.155085+0000) 2024-03-06T14:42:21.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:21.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:21.155387+0000) 2024-03-06T14:42:22.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:22.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:22.155731+0000) 2024-03-06T14:42:23.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:23.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:23.156037+0000) 2024-03-06T14:42:24.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:24.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:24.156293+0000) 2024-03-06T14:42:25.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:25.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:25.156610+0000) 2024-03-06T14:42:26.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:26.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:26.156978+0000) 2024-03-06T14:42:27.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:27.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:27.157285+0000) 2024-03-06T14:42:28.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:28.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:28.157560+0000) 2024-03-06T14:42:29.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:29.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:29.157825+0000) 2024-03-06T14:42:29.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:29.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:30.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:30.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:30.158083+0000) 2024-03-06T14:42:31.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:31.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:31.158277+0000) 2024-03-06T14:42:32.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:32.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:32.158537+0000) 2024-03-06T14:42:33.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:33.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:33.158761+0000) 2024-03-06T14:42:34.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:34.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:34.159040+0000) 2024-03-06T14:42:35.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:35.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:35.159248+0000) 2024-03-06T14:42:36.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:36.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:36.159504+0000) 2024-03-06T14:42:37.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:37.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:37.159774+0000) 2024-03-06T14:42:38.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:38.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:38.160044+0000) 2024-03-06T14:42:39.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:39.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:39.160319+0000) 2024-03-06T14:42:40.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:40.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:40.160584+0000) 2024-03-06T14:42:41.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:41.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:41.160867+0000) 2024-03-06T14:42:42.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:42.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:42.161147+0000) 2024-03-06T14:42:43.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:43.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:43.161438+0000) 2024-03-06T14:42:44.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:44.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:44.161686+0000) 2024-03-06T14:42:45.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:45.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:45.161962+0000) 2024-03-06T14:42:46.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:46.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:46.162233+0000) 2024-03-06T14:42:47.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:47.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:47.162496+0000) 2024-03-06T14:42:48.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:48.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:48.162771+0000) 2024-03-06T14:42:49.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:49.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:49.162985+0000) 2024-03-06T14:42:50.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:50.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:50.163267+0000) 2024-03-06T14:42:51.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:51.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:51.163552+0000) 2024-03-06T14:42:52.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:52.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:52.163816+0000) 2024-03-06T14:42:53.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:53.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:53.164066+0000) 2024-03-06T14:42:54.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:54.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:54.164318+0000) 2024-03-06T14:42:55.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:55.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:55.164574+0000) 2024-03-06T14:42:56.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:56.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:56.164831+0000) 2024-03-06T14:42:57.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:57.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:57.165046+0000) 2024-03-06T14:42:58.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:58.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:58.165358+0000) 2024-03-06T14:42:59.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:42:59.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:59.165593+0000) 2024-03-06T14:42:59.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:59.548 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:00.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:00.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:00.165861+0000) 2024-03-06T14:43:01.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:01.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:01.166123+0000) 2024-03-06T14:43:02.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:02.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:02.166363+0000) 2024-03-06T14:43:03.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:03.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:03.166592+0000) 2024-03-06T14:43:04.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:04.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:04.166841+0000) 2024-03-06T14:43:05.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:05.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:05.167079+0000) 2024-03-06T14:43:06.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:06.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:06.167274+0000) 2024-03-06T14:43:07.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:07.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:07.167524+0000) 2024-03-06T14:43:08.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:08.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:08.167761+0000) 2024-03-06T14:43:09.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:09.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:09.168013+0000) 2024-03-06T14:43:10.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:10.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:10.168272+0000) 2024-03-06T14:43:11.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:11.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:11.168565+0000) 2024-03-06T14:43:12.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:12.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:12.168873+0000) 2024-03-06T14:43:13.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:13.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:13.169155+0000) 2024-03-06T14:43:14.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:14.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:14.169438+0000) 2024-03-06T14:43:15.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:15.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:15.169780+0000) 2024-03-06T14:43:16.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:16.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:16.170084+0000) 2024-03-06T14:43:17.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:17.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:17.170396+0000) 2024-03-06T14:43:18.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:18.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:18.170731+0000) 2024-03-06T14:43:19.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:19.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:19.171021+0000) 2024-03-06T14:43:20.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:20.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:20.171262+0000) 2024-03-06T14:43:21.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:21.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:21.171536+0000) 2024-03-06T14:43:22.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:22.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:22.171851+0000) 2024-03-06T14:43:23.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:23.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:23.172141+0000) 2024-03-06T14:43:24.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:24.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:24.172420+0000) 2024-03-06T14:43:25.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:25.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:25.172746+0000) 2024-03-06T14:43:26.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:26.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:26.172983+0000) 2024-03-06T14:43:27.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:27.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:27.173285+0000) 2024-03-06T14:43:28.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:28.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:28.173587+0000) 2024-03-06T14:43:29.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:29.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:29.173926+0000) 2024-03-06T14:43:29.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:29.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:30.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:30.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:30.174198+0000) 2024-03-06T14:43:31.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:31.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:31.174503+0000) 2024-03-06T14:43:32.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:32.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:32.174809+0000) 2024-03-06T14:43:33.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:33.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:33.175120+0000) 2024-03-06T14:43:34.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:34.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:34.175435+0000) 2024-03-06T14:43:35.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:35.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:35.175766+0000) 2024-03-06T14:43:36.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:36.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:36.176013+0000) 2024-03-06T14:43:37.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:37.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:37.176203+0000) 2024-03-06T14:43:38.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:38.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:38.176457+0000) 2024-03-06T14:43:39.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:39.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:39.176737+0000) 2024-03-06T14:43:40.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:40.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:40.176989+0000) 2024-03-06T14:43:41.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:41.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:41.177241+0000) 2024-03-06T14:43:42.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:42.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:42.177466+0000) 2024-03-06T14:43:43.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:43.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:43.177746+0000) 2024-03-06T14:43:44.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:44.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:44.177994+0000) 2024-03-06T14:43:45.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:45.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:45.178235+0000) 2024-03-06T14:43:46.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:46.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:46.178540+0000) 2024-03-06T14:43:47.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:47.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:47.178795+0000) 2024-03-06T14:43:48.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:48.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:48.179057+0000) 2024-03-06T14:43:49.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:49.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:49.179290+0000) 2024-03-06T14:43:50.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:50.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:50.179554+0000) 2024-03-06T14:43:51.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:51.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:51.179809+0000) 2024-03-06T14:43:52.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:52.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:52.180050+0000) 2024-03-06T14:43:53.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:53.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:53.180280+0000) 2024-03-06T14:43:54.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:54.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:54.180521+0000) 2024-03-06T14:43:55.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:55.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:55.180754+0000) 2024-03-06T14:43:56.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:56.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:56.181032+0000) 2024-03-06T14:43:57.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:57.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:57.181286+0000) 2024-03-06T14:43:58.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:58.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:58.181542+0000) 2024-03-06T14:43:59.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:43:59.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:59.181793+0000) 2024-03-06T14:43:59.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:59.856 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:00.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:00.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:00.182033+0000) 2024-03-06T14:44:01.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:01.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:01.182298+0000) 2024-03-06T14:44:02.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:02.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:02.182544+0000) 2024-03-06T14:44:03.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:03.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:03.182775+0000) 2024-03-06T14:44:04.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:04.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:04.183007+0000) 2024-03-06T14:44:05.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:05.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:05.183241+0000) 2024-03-06T14:44:06.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:06.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:06.183477+0000) 2024-03-06T14:44:07.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:07.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:07.183762+0000) 2024-03-06T14:44:08.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:08.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:08.184000+0000) 2024-03-06T14:44:09.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:09.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:09.184187+0000) 2024-03-06T14:44:10.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:10.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:10.184429+0000) 2024-03-06T14:44:11.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:11.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:11.184733+0000) 2024-03-06T14:44:12.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:12.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:12.184997+0000) 2024-03-06T14:44:13.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:13.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:13.185234+0000) 2024-03-06T14:44:14.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:14.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:14.185492+0000) 2024-03-06T14:44:15.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:15.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:15.185740+0000) 2024-03-06T14:44:16.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:16.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:16.185928+0000) 2024-03-06T14:44:17.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:17.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:17.186190+0000) 2024-03-06T14:44:18.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:18.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:18.186452+0000) 2024-03-06T14:44:19.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:19.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:19.186749+0000) 2024-03-06T14:44:20.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:20.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:20.186978+0000) 2024-03-06T14:44:21.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:21.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:21.187238+0000) 2024-03-06T14:44:22.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:22.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:22.187479+0000) 2024-03-06T14:44:23.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:23.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:23.187764+0000) 2024-03-06T14:44:24.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:24.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:24.188084+0000) 2024-03-06T14:44:25.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:25.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:25.188409+0000) 2024-03-06T14:44:26.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:26.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:26.188747+0000) 2024-03-06T14:44:27.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:27.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:27.189052+0000) 2024-03-06T14:44:28.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:28.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:28.189360+0000) 2024-03-06T14:44:29.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:29.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:29.189706+0000) 2024-03-06T14:44:29.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:29.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:30.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:30.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:30.189957+0000) 2024-03-06T14:44:31.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:31.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:31.190279+0000) 2024-03-06T14:44:32.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:32.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:32.190563+0000) 2024-03-06T14:44:33.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:33.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:33.190864+0000) 2024-03-06T14:44:34.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:34.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:34.191158+0000) 2024-03-06T14:44:35.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:35.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:35.191458+0000) 2024-03-06T14:44:36.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:36.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:36.191784+0000) 2024-03-06T14:44:37.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:37.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:37.192064+0000) 2024-03-06T14:44:38.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:38.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:38.192313+0000) 2024-03-06T14:44:39.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:39.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:39.192684+0000) 2024-03-06T14:44:40.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:40.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:40.192946+0000) 2024-03-06T14:44:41.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:41.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:41.193186+0000) 2024-03-06T14:44:42.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:42.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:42.193446+0000) 2024-03-06T14:44:43.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:43.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:43.193722+0000) 2024-03-06T14:44:44.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:44.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:44.193981+0000) 2024-03-06T14:44:45.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:45.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:45.194243+0000) 2024-03-06T14:44:46.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:46.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:46.194511+0000) 2024-03-06T14:44:47.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:47.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:47.194786+0000) 2024-03-06T14:44:48.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:48.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:48.195094+0000) 2024-03-06T14:44:49.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:49.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:49.195389+0000) 2024-03-06T14:44:50.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:50.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:50.195731+0000) 2024-03-06T14:44:51.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:51.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:51.196036+0000) 2024-03-06T14:44:52.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:52.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:52.196282+0000) 2024-03-06T14:44:53.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:53.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:53.196530+0000) 2024-03-06T14:44:54.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:54.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:54.196789+0000) 2024-03-06T14:44:55.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:55.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:55.197083+0000) 2024-03-06T14:44:56.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:56.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:56.197338+0000) 2024-03-06T14:44:57.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:57.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:57.197604+0000) 2024-03-06T14:44:58.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:58.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:58.197885+0000) 2024-03-06T14:44:59.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:44:59.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:59.198107+0000) 2024-03-06T14:45:00.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:00.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:00.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:00.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:00.198361+0000) 2024-03-06T14:45:01.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:01.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:01.198663+0000) 2024-03-06T14:45:02.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:02.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:02.198932+0000) 2024-03-06T14:45:03.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:03.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:03.199169+0000) 2024-03-06T14:45:04.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:04.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:04.199410+0000) 2024-03-06T14:45:05.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:05.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:05.199713+0000) 2024-03-06T14:45:06.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:06.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:06.199977+0000) 2024-03-06T14:45:07.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:07.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:07.200233+0000) 2024-03-06T14:45:08.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:08.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:08.200476+0000) 2024-03-06T14:45:09.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:09.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:09.200767+0000) 2024-03-06T14:45:10.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:10.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:10.201038+0000) 2024-03-06T14:45:11.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:11.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:11.201322+0000) 2024-03-06T14:45:12.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:12.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:12.201581+0000) 2024-03-06T14:45:13.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:13.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:13.201823+0000) 2024-03-06T14:45:14.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:14.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:14.202075+0000) 2024-03-06T14:45:15.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:15.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:15.202327+0000) 2024-03-06T14:45:16.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:16.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:16.202590+0000) 2024-03-06T14:45:17.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:17.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:17.202941+0000) 2024-03-06T14:45:18.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:18.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:18.203154+0000) 2024-03-06T14:45:19.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:19.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:19.203404+0000) 2024-03-06T14:45:20.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:20.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:20.203717+0000) 2024-03-06T14:45:21.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:21.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:21.203942+0000) 2024-03-06T14:45:22.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:22.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:22.204204+0000) 2024-03-06T14:45:23.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:23.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:23.204449+0000) 2024-03-06T14:45:24.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:24.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:24.204741+0000) 2024-03-06T14:45:25.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:25.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:25.204977+0000) 2024-03-06T14:45:26.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:26.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:26.205238+0000) 2024-03-06T14:45:27.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:27.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:27.205496+0000) 2024-03-06T14:45:28.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:28.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:28.205754+0000) 2024-03-06T14:45:29.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:29.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:29.206006+0000) 2024-03-06T14:45:30.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:30.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:30.206272+0000) 2024-03-06T14:45:30.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:30.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:31.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:31.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:31.206516+0000) 2024-03-06T14:45:32.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:32.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:32.206770+0000) 2024-03-06T14:45:33.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:33.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:33.207017+0000) 2024-03-06T14:45:34.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:34.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:34.207266+0000) 2024-03-06T14:45:35.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:35.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:35.207517+0000) 2024-03-06T14:45:36.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:36.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:36.207775+0000) 2024-03-06T14:45:37.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:37.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:37.208040+0000) 2024-03-06T14:45:38.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:38.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:38.208293+0000) 2024-03-06T14:45:39.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:39.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:39.208543+0000) 2024-03-06T14:45:40.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:40.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:40.208797+0000) 2024-03-06T14:45:41.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:41.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:41.209060+0000) 2024-03-06T14:45:42.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:42.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:42.209322+0000) 2024-03-06T14:45:43.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:43.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:43.209577+0000) 2024-03-06T14:45:44.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:44.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:44.209834+0000) 2024-03-06T14:45:45.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:45.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:45.210087+0000) 2024-03-06T14:45:46.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:46.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:46.210346+0000) 2024-03-06T14:45:47.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:47.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:47.210581+0000) 2024-03-06T14:45:48.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:48.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:48.210813+0000) 2024-03-06T14:45:49.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:49.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:49.211091+0000) 2024-03-06T14:45:50.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:50.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:50.211288+0000) 2024-03-06T14:45:51.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:51.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:51.211534+0000) 2024-03-06T14:45:52.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:52.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:52.211792+0000) 2024-03-06T14:45:53.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:53.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:53.212082+0000) 2024-03-06T14:45:54.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:54.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:54.212387+0000) 2024-03-06T14:45:55.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:55.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:55.212733+0000) 2024-03-06T14:45:56.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:56.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:56.213045+0000) 2024-03-06T14:45:57.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:57.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:57.213349+0000) 2024-03-06T14:45:58.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:58.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:58.213713+0000) 2024-03-06T14:45:59.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:45:59.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:59.214001+0000) 2024-03-06T14:46:00.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:00.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:00.214299+0000) 2024-03-06T14:46:00.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:00.486 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:01.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:01.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:01.214579+0000) 2024-03-06T14:46:02.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:02.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:02.214830+0000) 2024-03-06T14:46:03.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:03.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:03.215060+0000) 2024-03-06T14:46:04.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:04.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:04.215359+0000) 2024-03-06T14:46:05.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:05.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:05.215715+0000) 2024-03-06T14:46:06.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:06.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:06.216033+0000) 2024-03-06T14:46:07.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:07.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:07.216337+0000) 2024-03-06T14:46:08.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:08.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:08.216687+0000) 2024-03-06T14:46:09.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:09.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:09.216980+0000) 2024-03-06T14:46:10.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:10.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:10.217254+0000) 2024-03-06T14:46:11.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:11.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:11.217552+0000) 2024-03-06T14:46:12.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:12.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:12.217834+0000) 2024-03-06T14:46:13.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:13.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:13.218074+0000) 2024-03-06T14:46:14.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:14.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:14.218358+0000) 2024-03-06T14:46:15.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:15.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:15.218742+0000) 2024-03-06T14:46:16.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:16.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:16.219033+0000) 2024-03-06T14:46:17.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:17.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:17.219303+0000) 2024-03-06T14:46:18.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:18.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:18.219587+0000) 2024-03-06T14:46:19.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:19.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:19.219931+0000) 2024-03-06T14:46:20.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:20.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:20.220213+0000) 2024-03-06T14:46:21.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:21.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:21.220550+0000) 2024-03-06T14:46:22.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:22.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:22.220851+0000) 2024-03-06T14:46:23.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:23.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:23.221161+0000) 2024-03-06T14:46:24.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:24.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:24.221458+0000) 2024-03-06T14:46:25.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:25.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:25.221790+0000) 2024-03-06T14:46:26.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:26.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:26.222091+0000) 2024-03-06T14:46:27.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:27.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:27.222398+0000) 2024-03-06T14:46:28.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:28.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:28.222728+0000) 2024-03-06T14:46:29.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:29.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:29.223040+0000) 2024-03-06T14:46:30.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:30.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:30.223351+0000) 2024-03-06T14:46:30.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:30.628 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:31.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:31.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:31.223671+0000) 2024-03-06T14:46:32.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:32.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:32.223859+0000) 2024-03-06T14:46:33.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:33.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:33.224087+0000) 2024-03-06T14:46:34.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:34.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:34.224320+0000) 2024-03-06T14:46:35.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:35.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:35.224574+0000) 2024-03-06T14:46:36.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:36.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:36.224838+0000) 2024-03-06T14:46:37.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:37.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:37.225091+0000) 2024-03-06T14:46:38.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:38.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:38.225354+0000) 2024-03-06T14:46:39.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:39.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:39.225604+0000) 2024-03-06T14:46:40.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:40.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:40.225909+0000) 2024-03-06T14:46:41.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:41.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:41.226177+0000) 2024-03-06T14:46:42.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:42.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:42.226433+0000) 2024-03-06T14:46:43.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:43.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:43.226728+0000) 2024-03-06T14:46:44.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:44.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:44.226983+0000) 2024-03-06T14:46:45.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:45.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:45.227234+0000) 2024-03-06T14:46:46.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:46.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:46.227427+0000) 2024-03-06T14:46:47.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:47.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:47.227731+0000) 2024-03-06T14:46:48.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:48.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:48.227972+0000) 2024-03-06T14:46:49.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:49.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:49.228230+0000) 2024-03-06T14:46:50.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:50.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:50.228490+0000) 2024-03-06T14:46:51.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:51.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:51.228755+0000) 2024-03-06T14:46:52.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:52.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:52.228997+0000) 2024-03-06T14:46:53.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:53.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:53.229182+0000) 2024-03-06T14:46:54.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:54.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:54.229424+0000) 2024-03-06T14:46:55.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:55.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:55.229718+0000) 2024-03-06T14:46:56.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:56.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:56.229979+0000) 2024-03-06T14:46:57.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:57.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:57.230233+0000) 2024-03-06T14:46:58.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:58.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:58.230493+0000) 2024-03-06T14:46:59.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:46:59.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:59.230750+0000) 2024-03-06T14:47:00.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:00.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:00.231004+0000) 2024-03-06T14:47:00.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:00.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:01.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:01.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:01.231264+0000) 2024-03-06T14:47:02.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:02.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:02.231516+0000) 2024-03-06T14:47:03.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:03.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:03.231775+0000) 2024-03-06T14:47:04.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:04.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:04.232030+0000) 2024-03-06T14:47:05.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:05.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:05.232286+0000) 2024-03-06T14:47:06.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:06.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:06.232524+0000) 2024-03-06T14:47:07.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:07.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:07.232777+0000) 2024-03-06T14:47:08.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:08.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:08.233089+0000) 2024-03-06T14:47:09.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:09.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:09.233266+0000) 2024-03-06T14:47:10.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:10.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:10.233519+0000) 2024-03-06T14:47:11.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:11.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:11.233769+0000) 2024-03-06T14:47:12.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:12.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:12.234005+0000) 2024-03-06T14:47:13.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:13.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:13.234261+0000) 2024-03-06T14:47:14.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:14.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:14.234447+0000) 2024-03-06T14:47:15.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:15.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:15.234734+0000) 2024-03-06T14:47:16.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:16.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:16.234995+0000) 2024-03-06T14:47:17.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:17.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:17.235235+0000) 2024-03-06T14:47:18.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:18.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:18.235489+0000) 2024-03-06T14:47:19.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:19.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:19.235743+0000) 2024-03-06T14:47:20.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:20.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:20.235996+0000) 2024-03-06T14:47:21.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:21.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:21.236280+0000) 2024-03-06T14:47:22.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:22.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:22.236516+0000) 2024-03-06T14:47:23.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:23.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:23.236753+0000) 2024-03-06T14:47:24.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:24.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:24.236995+0000) 2024-03-06T14:47:25.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:25.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:25.237250+0000) 2024-03-06T14:47:26.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:26.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:26.237511+0000) 2024-03-06T14:47:27.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:27.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:27.237764+0000) 2024-03-06T14:47:28.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:28.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:28.237957+0000) 2024-03-06T14:47:29.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:29.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:29.238209+0000) 2024-03-06T14:47:30.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:30.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:30.238473+0000) 2024-03-06T14:47:30.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:30.929 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:31.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:31.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:31.238757+0000) 2024-03-06T14:47:32.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:32.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:32.239013+0000) 2024-03-06T14:47:33.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:33.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:33.239250+0000) 2024-03-06T14:47:34.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:34.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:34.239485+0000) 2024-03-06T14:47:35.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:35.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:35.239750+0000) 2024-03-06T14:47:36.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:36.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:36.239992+0000) 2024-03-06T14:47:37.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:37.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:37.240248+0000) 2024-03-06T14:47:38.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:38.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:38.240508+0000) 2024-03-06T14:47:39.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:39.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:39.240782+0000) 2024-03-06T14:47:40.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:40.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:40.241083+0000) 2024-03-06T14:47:41.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:41.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:41.241396+0000) 2024-03-06T14:47:42.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:42.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:42.241731+0000) 2024-03-06T14:47:43.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:43.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:43.241992+0000) 2024-03-06T14:47:44.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:44.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:44.242272+0000) 2024-03-06T14:47:45.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:45.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:45.242551+0000) 2024-03-06T14:47:46.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:46.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:46.242855+0000) 2024-03-06T14:47:47.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:47.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:47.243110+0000) 2024-03-06T14:47:48.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:48.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:48.243369+0000) 2024-03-06T14:47:49.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:49.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:49.243623+0000) 2024-03-06T14:47:50.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:50.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:50.243891+0000) 2024-03-06T14:47:51.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:51.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:51.244142+0000) 2024-03-06T14:47:52.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:52.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:52.244395+0000) 2024-03-06T14:47:53.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:53.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:53.244696+0000) 2024-03-06T14:47:54.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:54.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:54.244971+0000) 2024-03-06T14:47:55.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:55.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:55.245281+0000) 2024-03-06T14:47:56.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:56.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:56.245616+0000) 2024-03-06T14:47:57.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:57.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:57.245980+0000) 2024-03-06T14:47:58.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:58.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:58.246265+0000) 2024-03-06T14:47:59.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:47:59.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:59.246576+0000) 2024-03-06T14:48:00.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:00.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:00.246895+0000) 2024-03-06T14:48:01.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:01.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:01.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:01.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:01.247200+0000) 2024-03-06T14:48:02.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:02.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:02.247508+0000) 2024-03-06T14:48:03.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:03.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:03.247812+0000) 2024-03-06T14:48:04.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:04.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:04.248124+0000) 2024-03-06T14:48:05.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:05.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:05.248438+0000) 2024-03-06T14:48:06.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:06.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:06.248779+0000) 2024-03-06T14:48:07.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:07.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:07.249064+0000) 2024-03-06T14:48:08.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:08.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:08.249360+0000) 2024-03-06T14:48:09.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:09.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:09.249707+0000) 2024-03-06T14:48:10.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:10.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:10.250020+0000) 2024-03-06T14:48:11.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:11.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:11.250348+0000) 2024-03-06T14:48:12.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:12.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:12.250676+0000) 2024-03-06T14:48:13.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:13.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:13.250998+0000) 2024-03-06T14:48:14.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:14.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:14.251311+0000) 2024-03-06T14:48:15.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:15.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:15.251602+0000) 2024-03-06T14:48:16.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:16.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:16.251963+0000) 2024-03-06T14:48:17.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:17.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:17.252262+0000) 2024-03-06T14:48:18.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:18.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:18.252573+0000) 2024-03-06T14:48:19.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:19.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:19.252890+0000) 2024-03-06T14:48:20.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:20.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:20.253189+0000) 2024-03-06T14:48:21.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:21.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:21.253492+0000) 2024-03-06T14:48:22.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:22.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:22.253810+0000) 2024-03-06T14:48:23.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:23.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:23.254123+0000) 2024-03-06T14:48:24.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:24.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:24.254408+0000) 2024-03-06T14:48:25.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:25.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:25.254746+0000) 2024-03-06T14:48:26.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:26.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:26.254999+0000) 2024-03-06T14:48:27.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:27.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:27.255304+0000) 2024-03-06T14:48:28.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:28.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:28.255605+0000) 2024-03-06T14:48:29.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:29.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:29.255946+0000) 2024-03-06T14:48:30.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:30.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:30.256244+0000) 2024-03-06T14:48:31.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:31.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:31.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:31.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:31.256460+0000) 2024-03-06T14:48:32.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:32.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:32.256797+0000) 2024-03-06T14:48:33.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:33.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:33.257116+0000) 2024-03-06T14:48:34.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:34.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:34.257424+0000) 2024-03-06T14:48:35.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:35.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:35.257777+0000) 2024-03-06T14:48:36.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:36.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:36.258069+0000) 2024-03-06T14:48:37.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:37.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:37.258393+0000) 2024-03-06T14:48:38.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:38.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:38.258755+0000) 2024-03-06T14:48:39.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:39.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:39.259047+0000) 2024-03-06T14:48:40.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:40.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:40.259356+0000) 2024-03-06T14:48:41.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:41.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:41.259725+0000) 2024-03-06T14:48:42.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:42.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:42.260048+0000) 2024-03-06T14:48:43.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:43.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:43.260362+0000) 2024-03-06T14:48:44.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:44.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:44.260773+0000) 2024-03-06T14:48:45.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:45.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:45.261090+0000) 2024-03-06T14:48:46.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:46.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:46.261376+0000) 2024-03-06T14:48:47.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:47.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:47.261730+0000) 2024-03-06T14:48:48.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:48.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:48.262004+0000) 2024-03-06T14:48:49.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:49.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:49.262281+0000) 2024-03-06T14:48:50.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:50.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:50.262549+0000) 2024-03-06T14:48:51.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:51.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:51.262861+0000) 2024-03-06T14:48:52.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:52.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:52.263135+0000) 2024-03-06T14:48:53.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:53.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:53.263387+0000) 2024-03-06T14:48:54.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:54.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:54.263585+0000) 2024-03-06T14:48:55.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:55.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:55.263837+0000) 2024-03-06T14:48:56.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:56.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:56.264092+0000) 2024-03-06T14:48:57.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:57.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:57.264331+0000) 2024-03-06T14:48:58.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:58.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:58.264574+0000) 2024-03-06T14:48:59.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:48:59.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:59.264805+0000) 2024-03-06T14:49:00.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:00.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:00.265062+0000) 2024-03-06T14:49:01.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:01.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:01.265322+0000) 2024-03-06T14:49:01.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:01.477 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:02.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:02.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:02.265583+0000) 2024-03-06T14:49:03.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:03.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:03.265821+0000) 2024-03-06T14:49:04.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:04.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:04.266008+0000) 2024-03-06T14:49:05.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:05.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:05.266195+0000) 2024-03-06T14:49:06.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:06.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:06.266379+0000) 2024-03-06T14:49:07.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:07.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:07.266703+0000) 2024-03-06T14:49:08.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:08.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:08.266913+0000) 2024-03-06T14:49:09.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:09.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:09.267160+0000) 2024-03-06T14:49:10.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:10.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:10.267419+0000) 2024-03-06T14:49:11.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:11.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:11.267725+0000) 2024-03-06T14:49:12.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:12.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:12.267981+0000) 2024-03-06T14:49:13.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:13.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:13.268222+0000) 2024-03-06T14:49:14.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:14.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:14.268457+0000) 2024-03-06T14:49:15.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:15.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:15.268748+0000) 2024-03-06T14:49:16.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:16.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:16.269012+0000) 2024-03-06T14:49:17.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:17.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:17.269265+0000) 2024-03-06T14:49:18.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:18.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:18.269519+0000) 2024-03-06T14:49:19.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:19.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:19.269748+0000) 2024-03-06T14:49:20.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:20.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:20.269984+0000) 2024-03-06T14:49:21.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:21.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:21.270249+0000) 2024-03-06T14:49:22.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:22.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:22.270509+0000) 2024-03-06T14:49:23.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:23.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:23.270760+0000) 2024-03-06T14:49:24.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:24.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:24.271018+0000) 2024-03-06T14:49:25.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:25.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:25.271249+0000) 2024-03-06T14:49:26.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:26.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:26.271501+0000) 2024-03-06T14:49:27.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:27.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:27.271754+0000) 2024-03-06T14:49:28.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:28.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:28.271997+0000) 2024-03-06T14:49:29.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:29.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:29.272231+0000) 2024-03-06T14:49:30.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:30.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:30.272489+0000) 2024-03-06T14:49:31.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:31.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:31.272732+0000) 2024-03-06T14:49:31.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:31.626 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:32.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:32.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:32.272973+0000) 2024-03-06T14:49:33.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:33.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:33.273227+0000) 2024-03-06T14:49:34.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:34.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:34.273504+0000) 2024-03-06T14:49:35.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:35.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:35.273796+0000) 2024-03-06T14:49:36.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:36.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:36.274023+0000) 2024-03-06T14:49:37.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:37.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:37.274257+0000) 2024-03-06T14:49:38.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:38.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:38.274792+0000) 2024-03-06T14:49:39.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:39.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:39.275089+0000) 2024-03-06T14:49:40.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:40.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:40.275407+0000) 2024-03-06T14:49:41.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:41.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:41.275752+0000) 2024-03-06T14:49:42.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:42.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:42.276029+0000) 2024-03-06T14:49:43.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:43.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:43.276332+0000) 2024-03-06T14:49:44.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:44.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:44.276684+0000) 2024-03-06T14:49:45.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:45.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:45.276980+0000) 2024-03-06T14:49:46.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:46.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:46.277286+0000) 2024-03-06T14:49:47.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:47.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:47.277567+0000) 2024-03-06T14:49:48.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:48.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:48.277876+0000) 2024-03-06T14:49:49.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:49.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:49.278177+0000) 2024-03-06T14:49:50.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:50.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:50.278474+0000) 2024-03-06T14:49:51.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:51.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:51.278795+0000) 2024-03-06T14:49:52.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:52.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:52.279099+0000) 2024-03-06T14:49:53.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:53.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:53.279412+0000) 2024-03-06T14:49:54.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:54.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:54.279755+0000) 2024-03-06T14:49:55.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:55.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:55.280069+0000) 2024-03-06T14:49:56.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:56.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:56.280362+0000) 2024-03-06T14:49:57.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:57.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:57.280700+0000) 2024-03-06T14:49:58.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:58.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:58.281005+0000) 2024-03-06T14:49:59.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:49:59.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:59.281298+0000) 2024-03-06T14:50:00.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:00.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:00.281594+0000) 2024-03-06T14:50:01.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:01.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:01.281955+0000) 2024-03-06T14:50:01.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:01.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:02.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:02.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:02.282249+0000) 2024-03-06T14:50:03.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:03.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:03.282558+0000) 2024-03-06T14:50:04.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:04.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:04.282860+0000) 2024-03-06T14:50:05.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:05.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:05.283063+0000) 2024-03-06T14:50:06.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:06.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:06.283297+0000) 2024-03-06T14:50:07.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:07.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:07.283552+0000) 2024-03-06T14:50:08.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:08.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:08.283783+0000) 2024-03-06T14:50:09.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:09.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:09.283974+0000) 2024-03-06T14:50:10.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:10.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:10.284170+0000) 2024-03-06T14:50:11.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:11.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:11.284414+0000) 2024-03-06T14:50:12.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:12.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:12.284676+0000) 2024-03-06T14:50:13.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:13.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:13.284944+0000) 2024-03-06T14:50:14.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:14.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:14.285202+0000) 2024-03-06T14:50:15.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:15.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:15.285459+0000) 2024-03-06T14:50:16.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:16.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:16.285753+0000) 2024-03-06T14:50:17.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:17.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:17.286005+0000) 2024-03-06T14:50:18.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:18.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:18.286266+0000) 2024-03-06T14:50:19.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:19.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:19.286515+0000) 2024-03-06T14:50:20.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:20.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:20.286770+0000) 2024-03-06T14:50:21.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:21.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:21.287022+0000) 2024-03-06T14:50:22.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:22.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:22.287285+0000) 2024-03-06T14:50:23.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:23.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:23.287535+0000) 2024-03-06T14:50:24.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:24.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:24.287776+0000) 2024-03-06T14:50:25.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:25.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:25.288028+0000) 2024-03-06T14:50:26.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:26.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:26.288344+0000) 2024-03-06T14:50:27.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:27.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:27.288585+0000) 2024-03-06T14:50:28.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:28.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:28.288827+0000) 2024-03-06T14:50:29.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:29.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:29.289074+0000) 2024-03-06T14:50:30.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:30.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:30.289321+0000) 2024-03-06T14:50:31.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:31.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:31.289576+0000) 2024-03-06T14:50:31.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:31.920 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:32.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:32.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:32.289755+0000) 2024-03-06T14:50:33.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:33.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:33.289992+0000) 2024-03-06T14:50:34.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:34.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:34.290175+0000) 2024-03-06T14:50:35.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:35.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:35.290430+0000) 2024-03-06T14:50:36.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:36.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:36.290719+0000) 2024-03-06T14:50:37.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:37.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:37.290947+0000) 2024-03-06T14:50:38.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:38.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:38.291205+0000) 2024-03-06T14:50:39.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:39.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:39.291437+0000) 2024-03-06T14:50:40.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:40.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:40.291739+0000) 2024-03-06T14:50:41.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:41.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:41.292007+0000) 2024-03-06T14:50:42.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:42.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:42.292255+0000) 2024-03-06T14:50:43.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:43.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:43.292510+0000) 2024-03-06T14:50:44.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:44.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:44.292747+0000) 2024-03-06T14:50:45.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:45.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:45.292922+0000) 2024-03-06T14:50:46.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:46.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:46.293178+0000) 2024-03-06T14:50:47.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:47.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:47.293439+0000) 2024-03-06T14:50:48.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:48.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:48.293729+0000) 2024-03-06T14:50:49.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:49.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:49.293963+0000) 2024-03-06T14:50:50.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:50.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:50.294203+0000) 2024-03-06T14:50:51.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:51.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:51.294467+0000) 2024-03-06T14:50:52.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:52.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:52.294743+0000) 2024-03-06T14:50:53.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:53.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:53.294994+0000) 2024-03-06T14:50:54.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:54.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:54.295243+0000) 2024-03-06T14:50:55.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:55.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:55.295476+0000) 2024-03-06T14:50:56.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:56.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:56.295731+0000) 2024-03-06T14:50:57.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:57.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:57.295967+0000) 2024-03-06T14:50:58.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:58.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:58.296164+0000) 2024-03-06T14:50:59.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:50:59.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:59.296393+0000) 2024-03-06T14:51:00.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:00.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:00.296737+0000) 2024-03-06T14:51:01.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:01.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:01.297059+0000) 2024-03-06T14:51:02.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:02.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:02.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:02.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:02.297365+0000) 2024-03-06T14:51:03.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:03.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:03.297717+0000) 2024-03-06T14:51:04.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:04.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:04.298036+0000) 2024-03-06T14:51:05.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:05.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:05.298337+0000) 2024-03-06T14:51:06.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:06.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:06.298710+0000) 2024-03-06T14:51:07.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:07.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:07.299040+0000) 2024-03-06T14:51:08.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:08.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:08.299330+0000) 2024-03-06T14:51:09.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:09.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:09.299686+0000) 2024-03-06T14:51:10.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:10.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:10.300008+0000) 2024-03-06T14:51:11.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:11.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:11.300333+0000) 2024-03-06T14:51:12.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:12.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:12.300683+0000) 2024-03-06T14:51:13.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:13.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:13.300982+0000) 2024-03-06T14:51:14.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:14.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:14.301357+0000) 2024-03-06T14:51:15.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:15.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:15.301693+0000) 2024-03-06T14:51:16.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:16.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:16.301995+0000) 2024-03-06T14:51:17.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:17.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:17.302278+0000) 2024-03-06T14:51:18.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:18.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:18.302595+0000) 2024-03-06T14:51:19.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:19.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:19.302941+0000) 2024-03-06T14:51:20.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:20.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:20.303245+0000) 2024-03-06T14:51:21.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:21.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:21.303574+0000) 2024-03-06T14:51:22.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:22.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:22.303878+0000) 2024-03-06T14:51:23.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:23.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:23.304118+0000) 2024-03-06T14:51:24.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:24.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:24.304419+0000) 2024-03-06T14:51:25.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:25.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:25.304757+0000) 2024-03-06T14:51:26.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:26.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:26.305075+0000) 2024-03-06T14:51:27.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:27.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:27.305390+0000) 2024-03-06T14:51:28.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:28.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:28.305724+0000) 2024-03-06T14:51:29.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:29.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:29.306045+0000) 2024-03-06T14:51:30.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:30.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:30.306334+0000) 2024-03-06T14:51:31.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:31.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:31.306569+0000) 2024-03-06T14:51:32.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:32.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:32.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:32.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:32.306782+0000) 2024-03-06T14:51:33.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:33.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:33.307058+0000) 2024-03-06T14:51:34.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:34.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:34.307359+0000) 2024-03-06T14:51:35.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:35.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:35.307693+0000) 2024-03-06T14:51:36.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:36.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:36.308005+0000) 2024-03-06T14:51:37.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:37.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:37.308296+0000) 2024-03-06T14:51:38.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:38.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:38.308613+0000) 2024-03-06T14:51:39.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:39.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:39.308973+0000) 2024-03-06T14:51:40.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:40.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:40.309254+0000) 2024-03-06T14:51:41.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:41.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:41.309579+0000) 2024-03-06T14:51:42.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:42.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:42.309892+0000) 2024-03-06T14:51:43.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:43.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:43.310189+0000) 2024-03-06T14:51:44.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:44.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:44.310503+0000) 2024-03-06T14:51:45.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:45.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:45.310825+0000) 2024-03-06T14:51:46.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:46.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:46.311137+0000) 2024-03-06T14:51:47.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:47.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:47.311424+0000) 2024-03-06T14:51:48.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:48.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:48.311730+0000) 2024-03-06T14:51:49.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:49.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:49.311973+0000) 2024-03-06T14:51:50.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:50.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:50.312242+0000) 2024-03-06T14:51:51.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:51.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:51.312528+0000) 2024-03-06T14:51:52.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:52.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:52.312800+0000) 2024-03-06T14:51:53.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:53.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:53.313055+0000) 2024-03-06T14:51:54.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:54.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:54.313337+0000) 2024-03-06T14:51:55.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:55.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:55.313674+0000) 2024-03-06T14:51:56.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:56.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:56.313973+0000) 2024-03-06T14:51:57.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:57.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:57.314255+0000) 2024-03-06T14:51:58.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:58.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:58.314528+0000) 2024-03-06T14:51:59.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:51:59.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:59.314779+0000) 2024-03-06T14:52:00.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:00.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:00.314989+0000) 2024-03-06T14:52:01.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:01.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:01.315271+0000) 2024-03-06T14:52:02.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:02.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:02.315582+0000) 2024-03-06T14:52:02.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:02.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:03.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:03.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:03.315867+0000) 2024-03-06T14:52:04.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:04.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:04.316170+0000) 2024-03-06T14:52:05.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:05.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:05.316477+0000) 2024-03-06T14:52:06.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:06.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:06.316787+0000) 2024-03-06T14:52:07.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:07.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:07.317149+0000) 2024-03-06T14:52:08.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:08.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:08.317444+0000) 2024-03-06T14:52:09.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:09.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:09.317790+0000) 2024-03-06T14:52:10.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:10.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:10.318087+0000) 2024-03-06T14:52:11.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:11.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:11.318393+0000) 2024-03-06T14:52:12.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:12.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:12.318739+0000) 2024-03-06T14:52:13.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:13.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:13.319062+0000) 2024-03-06T14:52:14.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:14.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:14.319374+0000) 2024-03-06T14:52:15.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:15.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:15.319606+0000) 2024-03-06T14:52:16.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:16.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:16.319937+0000) 2024-03-06T14:52:17.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:17.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:17.320238+0000) 2024-03-06T14:52:18.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:18.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:18.320540+0000) 2024-03-06T14:52:19.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:19.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:19.320855+0000) 2024-03-06T14:52:20.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:20.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:20.321178+0000) 2024-03-06T14:52:21.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:21.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:21.321507+0000) 2024-03-06T14:52:22.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:22.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:22.321819+0000) 2024-03-06T14:52:23.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:23.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:23.322106+0000) 2024-03-06T14:52:24.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:24.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:24.322429+0000) 2024-03-06T14:52:25.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:25.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:25.322777+0000) 2024-03-06T14:52:26.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:26.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:26.323081+0000) 2024-03-06T14:52:27.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:27.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:27.323389+0000) 2024-03-06T14:52:28.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:28.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:28.323740+0000) 2024-03-06T14:52:29.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:29.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:29.323983+0000) 2024-03-06T14:52:30.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:30.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:30.324275+0000) 2024-03-06T14:52:31.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:31.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:31.324569+0000) 2024-03-06T14:52:32.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:32.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:32.324882+0000) 2024-03-06T14:52:32.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:32.509 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:33.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:33.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:33.325203+0000) 2024-03-06T14:52:34.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:34.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:34.325496+0000) 2024-03-06T14:52:35.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:35.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:35.325813+0000) 2024-03-06T14:52:36.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:36.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:36.326106+0000) 2024-03-06T14:52:37.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:37.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:37.326404+0000) 2024-03-06T14:52:38.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:38.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:38.326748+0000) 2024-03-06T14:52:39.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:39.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:39.327043+0000) 2024-03-06T14:52:40.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:40.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:40.327357+0000) 2024-03-06T14:52:41.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:41.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:41.327737+0000) 2024-03-06T14:52:42.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:42.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:42.328039+0000) 2024-03-06T14:52:43.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:43.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:43.328271+0000) 2024-03-06T14:52:44.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:44.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:44.328581+0000) 2024-03-06T14:52:45.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:45.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:45.328884+0000) 2024-03-06T14:52:46.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:46.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:46.329204+0000) 2024-03-06T14:52:47.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:47.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:47.329502+0000) 2024-03-06T14:52:48.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:48.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:48.329794+0000) 2024-03-06T14:52:49.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:49.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:49.330093+0000) 2024-03-06T14:52:50.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:50.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:50.330381+0000) 2024-03-06T14:52:51.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:51.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:51.330709+0000) 2024-03-06T14:52:52.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:52.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:52.330904+0000) 2024-03-06T14:52:53.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:53.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:53.331140+0000) 2024-03-06T14:52:54.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:54.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:54.331395+0000) 2024-03-06T14:52:55.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:55.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:55.331724+0000) 2024-03-06T14:52:56.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:56.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:56.331976+0000) 2024-03-06T14:52:57.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:57.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:57.332240+0000) 2024-03-06T14:52:58.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:58.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:58.332489+0000) 2024-03-06T14:52:59.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:52:59.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:59.332744+0000) 2024-03-06T14:53:00.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:00.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:00.332973+0000) 2024-03-06T14:53:01.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:01.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:01.333225+0000) 2024-03-06T14:53:02.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:02.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:02.333493+0000) 2024-03-06T14:53:02.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:02.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:03.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:03.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:03.333731+0000) 2024-03-06T14:53:04.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:04.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:04.333979+0000) 2024-03-06T14:53:05.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:05.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:05.334237+0000) 2024-03-06T14:53:06.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:06.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:06.334487+0000) 2024-03-06T14:53:07.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:07.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:07.334737+0000) 2024-03-06T14:53:08.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:08.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:08.334921+0000) 2024-03-06T14:53:09.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:09.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:09.335179+0000) 2024-03-06T14:53:10.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:10.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:10.335429+0000) 2024-03-06T14:53:11.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:11.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:11.335735+0000) 2024-03-06T14:53:12.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:12.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:12.335989+0000) 2024-03-06T14:53:13.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:13.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:13.336194+0000) 2024-03-06T14:53:14.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:14.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:14.336452+0000) 2024-03-06T14:53:15.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:15.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:15.336738+0000) 2024-03-06T14:53:16.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:16.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:16.336999+0000) 2024-03-06T14:53:17.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:17.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:17.337268+0000) 2024-03-06T14:53:18.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:18.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:18.337523+0000) 2024-03-06T14:53:19.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:19.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:19.337795+0000) 2024-03-06T14:53:20.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:20.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:20.338070+0000) 2024-03-06T14:53:21.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:21.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:21.338353+0000) 2024-03-06T14:53:22.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:22.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:22.338668+0000) 2024-03-06T14:53:23.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:23.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:23.338946+0000) 2024-03-06T14:53:24.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:24.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:24.339223+0000) 2024-03-06T14:53:25.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:25.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:25.339489+0000) 2024-03-06T14:53:26.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:26.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:26.339763+0000) 2024-03-06T14:53:27.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:27.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:27.340034+0000) 2024-03-06T14:53:28.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:28.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:28.340306+0000) 2024-03-06T14:53:29.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:29.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:29.340583+0000) 2024-03-06T14:53:30.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:30.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:30.340857+0000) 2024-03-06T14:53:31.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:31.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:31.341114+0000) 2024-03-06T14:53:32.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:32.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:32.341369+0000) 2024-03-06T14:53:32.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:32.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:33.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:33.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:33.341671+0000) 2024-03-06T14:53:34.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:34.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:34.341940+0000) 2024-03-06T14:53:35.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:35.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:35.342206+0000) 2024-03-06T14:53:36.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:36.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:36.342484+0000) 2024-03-06T14:53:37.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:37.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:37.342768+0000) 2024-03-06T14:53:38.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:38.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:38.343053+0000) 2024-03-06T14:53:39.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:39.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:39.343300+0000) 2024-03-06T14:53:40.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:40.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:40.343545+0000) 2024-03-06T14:53:41.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:41.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:41.343815+0000) 2024-03-06T14:53:42.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:42.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:42.344092+0000) 2024-03-06T14:53:43.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:43.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:43.344365+0000) 2024-03-06T14:53:44.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:44.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:44.344670+0000) 2024-03-06T14:53:45.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:45.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:45.344927+0000) 2024-03-06T14:53:46.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:46.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:46.345176+0000) 2024-03-06T14:53:47.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:47.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:47.345447+0000) 2024-03-06T14:53:48.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:48.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:48.345728+0000) 2024-03-06T14:53:49.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:49.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:49.345985+0000) 2024-03-06T14:53:50.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:50.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:50.346218+0000) 2024-03-06T14:53:51.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:51.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:51.346481+0000) 2024-03-06T14:53:52.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:52.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:52.346768+0000) 2024-03-06T14:53:53.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:53.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:53.347025+0000) 2024-03-06T14:53:54.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:54.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:54.347287+0000) 2024-03-06T14:53:55.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:55.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:55.347528+0000) 2024-03-06T14:53:56.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:56.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:56.347783+0000) 2024-03-06T14:53:57.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:57.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:57.348035+0000) 2024-03-06T14:53:58.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:58.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:58.348298+0000) 2024-03-06T14:53:59.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:53:59.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:59.348539+0000) 2024-03-06T14:54:00.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:00.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:00.348778+0000) 2024-03-06T14:54:01.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:01.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:01.349029+0000) 2024-03-06T14:54:02.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:02.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:02.349275+0000) 2024-03-06T14:54:02.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:02.951 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:03.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:03.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:03.349537+0000) 2024-03-06T14:54:04.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:04.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:04.349795+0000) 2024-03-06T14:54:05.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:05.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:05.350052+0000) 2024-03-06T14:54:06.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:06.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:06.350307+0000) 2024-03-06T14:54:07.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:07.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:07.350557+0000) 2024-03-06T14:54:08.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:08.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:08.350815+0000) 2024-03-06T14:54:09.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:09.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:09.351049+0000) 2024-03-06T14:54:10.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:10.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:10.351308+0000) 2024-03-06T14:54:11.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:11.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:11.351557+0000) 2024-03-06T14:54:12.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:12.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:12.351808+0000) 2024-03-06T14:54:13.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:13.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:13.352044+0000) 2024-03-06T14:54:14.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:14.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:14.352290+0000) 2024-03-06T14:54:15.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:15.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:15.352563+0000) 2024-03-06T14:54:16.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:16.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:16.352851+0000) 2024-03-06T14:54:17.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:17.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:17.353084+0000) 2024-03-06T14:54:18.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:18.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:18.353286+0000) 2024-03-06T14:54:19.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:19.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:19.353477+0000) 2024-03-06T14:54:20.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:20.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:20.353735+0000) 2024-03-06T14:54:21.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:21.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:21.353997+0000) 2024-03-06T14:54:22.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:22.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:22.354259+0000) 2024-03-06T14:54:23.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:23.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:23.354509+0000) 2024-03-06T14:54:24.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:24.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:24.354792+0000) 2024-03-06T14:54:25.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:25.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:25.355039+0000) 2024-03-06T14:54:26.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:26.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:26.355299+0000) 2024-03-06T14:54:27.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:27.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:27.355533+0000) 2024-03-06T14:54:28.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:28.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:28.355773+0000) 2024-03-06T14:54:29.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:29.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:29.356012+0000) 2024-03-06T14:54:30.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:30.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:30.356255+0000) 2024-03-06T14:54:31.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:31.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:31.356523+0000) 2024-03-06T14:54:32.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:32.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:32.356780+0000) 2024-03-06T14:54:33.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:33.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:33.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:33.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:33.357039+0000) 2024-03-06T14:54:34.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:34.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:34.357281+0000) 2024-03-06T14:54:35.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:35.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:35.357539+0000) 2024-03-06T14:54:36.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:36.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:36.357856+0000) 2024-03-06T14:54:37.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:37.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:37.358110+0000) 2024-03-06T14:54:38.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:38.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:38.358366+0000) 2024-03-06T14:54:39.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:39.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:39.358612+0000) 2024-03-06T14:54:40.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:40.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:40.358902+0000) 2024-03-06T14:54:41.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:41.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:41.359169+0000) 2024-03-06T14:54:42.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:42.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:42.359402+0000) 2024-03-06T14:54:43.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:43.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:43.359692+0000) 2024-03-06T14:54:44.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:44.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:44.359951+0000) 2024-03-06T14:54:45.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:45.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:45.360205+0000) 2024-03-06T14:54:46.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:46.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:46.360441+0000) 2024-03-06T14:54:47.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:47.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:47.360732+0000) 2024-03-06T14:54:48.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:48.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:48.360946+0000) 2024-03-06T14:54:49.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:49.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:49.361157+0000) 2024-03-06T14:54:50.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:50.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:50.361431+0000) 2024-03-06T14:54:51.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:51.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:51.361787+0000) 2024-03-06T14:54:52.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:52.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:52.362045+0000) 2024-03-06T14:54:53.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:53.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:53.362230+0000) 2024-03-06T14:54:54.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:54.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:54.362506+0000) 2024-03-06T14:54:55.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:55.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:55.362786+0000) 2024-03-06T14:54:56.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:56.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:56.363007+0000) 2024-03-06T14:54:57.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:57.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:57.363271+0000) 2024-03-06T14:54:58.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:58.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:58.363530+0000) 2024-03-06T14:54:59.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:54:59.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:59.363781+0000) 2024-03-06T14:55:00.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:00.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:00.364047+0000) 2024-03-06T14:55:01.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:01.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:01.364320+0000) 2024-03-06T14:55:02.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:02.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:02.364590+0000) 2024-03-06T14:55:03.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:03.260 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:03.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:03.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:03.364813+0000) 2024-03-06T14:55:04.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:04.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:04.365010+0000) 2024-03-06T14:55:05.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:05.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:05.365278+0000) 2024-03-06T14:55:06.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:06.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:06.365539+0000) 2024-03-06T14:55:07.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:07.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:07.365810+0000) 2024-03-06T14:55:08.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:08.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:08.366090+0000) 2024-03-06T14:55:09.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:09.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:09.366343+0000) 2024-03-06T14:55:10.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:10.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:10.366605+0000) 2024-03-06T14:55:11.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:11.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:11.366925+0000) 2024-03-06T14:55:12.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:12.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:12.367196+0000) 2024-03-06T14:55:13.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:13.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:13.367455+0000) 2024-03-06T14:55:14.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:14.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:14.367754+0000) 2024-03-06T14:55:15.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:15.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:15.368014+0000) 2024-03-06T14:55:16.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:16.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:16.368267+0000) 2024-03-06T14:55:17.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:17.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:17.368516+0000) 2024-03-06T14:55:18.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:18.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:18.368788+0000) 2024-03-06T14:55:19.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:19.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:19.369038+0000) 2024-03-06T14:55:20.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:20.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:20.369303+0000) 2024-03-06T14:55:21.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:21.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:21.369597+0000) 2024-03-06T14:55:22.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:22.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:22.369896+0000) 2024-03-06T14:55:23.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:23.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:23.370102+0000) 2024-03-06T14:55:24.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:24.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:24.370417+0000) 2024-03-06T14:55:25.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:25.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:25.370715+0000) 2024-03-06T14:55:26.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:26.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:26.370968+0000) 2024-03-06T14:55:27.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:27.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:27.371242+0000) 2024-03-06T14:55:28.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:28.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:28.371490+0000) 2024-03-06T14:55:29.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:29.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:29.371755+0000) 2024-03-06T14:55:30.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:30.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:30.372028+0000) 2024-03-06T14:55:31.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:31.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:31.372301+0000) 2024-03-06T14:55:32.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:32.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:32.372573+0000) 2024-03-06T14:55:33.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:33.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:33.372847+0000) 2024-03-06T14:55:33.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:33.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:34.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:34.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:34.373113+0000) 2024-03-06T14:55:35.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:35.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:35.373388+0000) 2024-03-06T14:55:36.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:36.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:36.373705+0000) 2024-03-06T14:55:37.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:37.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:37.373980+0000) 2024-03-06T14:55:38.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:38.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:38.374269+0000) 2024-03-06T14:55:39.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:39.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:39.374536+0000) 2024-03-06T14:55:40.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:40.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:40.374794+0000) 2024-03-06T14:55:41.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:41.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:41.375057+0000) 2024-03-06T14:55:42.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:42.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:42.375296+0000) 2024-03-06T14:55:43.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:43.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:43.375523+0000) 2024-03-06T14:55:44.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:44.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:44.375787+0000) 2024-03-06T14:55:45.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:45.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:45.376043+0000) 2024-03-06T14:55:46.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:46.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:46.376335+0000) 2024-03-06T14:55:47.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:47.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:47.376672+0000) 2024-03-06T14:55:48.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:48.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:48.376932+0000) 2024-03-06T14:55:49.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:49.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:49.377245+0000) 2024-03-06T14:55:50.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:50.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:50.377535+0000) 2024-03-06T14:55:51.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:51.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:51.377856+0000) 2024-03-06T14:55:52.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:52.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:52.378154+0000) 2024-03-06T14:55:53.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:53.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:53.378448+0000) 2024-03-06T14:55:54.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:54.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:54.378733+0000) 2024-03-06T14:55:55.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:55.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:55.379026+0000) 2024-03-06T14:55:56.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:56.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:56.379298+0000) 2024-03-06T14:55:57.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:57.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:57.379611+0000) 2024-03-06T14:55:58.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:58.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:58.379930+0000) 2024-03-06T14:55:59.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:55:59.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:59.380194+0000) 2024-03-06T14:56:00.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:00.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:00.380383+0000) 2024-03-06T14:56:01.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:01.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:01.380669+0000) 2024-03-06T14:56:02.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:02.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:02.380914+0000) 2024-03-06T14:56:03.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:03.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:03.381103+0000) 2024-03-06T14:56:03.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:03.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:04.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:04.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:04.381363+0000) 2024-03-06T14:56:05.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:05.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:05.381606+0000) 2024-03-06T14:56:06.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:06.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:06.381858+0000) 2024-03-06T14:56:07.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:07.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:07.382117+0000) 2024-03-06T14:56:08.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:08.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:08.382378+0000) 2024-03-06T14:56:09.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:09.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:09.382586+0000) 2024-03-06T14:56:10.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:10.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:10.382896+0000) 2024-03-06T14:56:11.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:11.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:11.383148+0000) 2024-03-06T14:56:12.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:12.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:12.383461+0000) 2024-03-06T14:56:13.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:13.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:13.383741+0000) 2024-03-06T14:56:14.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:14.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:14.383990+0000) 2024-03-06T14:56:15.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:15.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:15.384178+0000) 2024-03-06T14:56:16.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:16.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:16.384439+0000) 2024-03-06T14:56:17.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:17.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:17.384743+0000) 2024-03-06T14:56:18.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:18.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:18.384981+0000) 2024-03-06T14:56:19.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:19.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:19.385212+0000) 2024-03-06T14:56:20.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:20.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:20.385452+0000) 2024-03-06T14:56:21.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:21.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:21.385733+0000) 2024-03-06T14:56:22.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:22.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:22.385992+0000) 2024-03-06T14:56:23.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:23.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:23.386242+0000) 2024-03-06T14:56:24.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:24.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:24.386498+0000) 2024-03-06T14:56:25.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:25.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:25.386773+0000) 2024-03-06T14:56:26.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:26.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:26.387026+0000) 2024-03-06T14:56:27.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:27.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:27.387292+0000) 2024-03-06T14:56:28.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:28.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:28.387554+0000) 2024-03-06T14:56:29.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:29.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:29.387802+0000) 2024-03-06T14:56:30.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:30.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:30.388052+0000) 2024-03-06T14:56:31.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:31.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:31.388314+0000) 2024-03-06T14:56:32.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:32.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:32.388502+0000) 2024-03-06T14:56:33.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:33.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:33.388745+0000) 2024-03-06T14:56:33.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:33.690 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:34.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:34.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:34.389012+0000) 2024-03-06T14:56:35.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:35.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:35.389261+0000) 2024-03-06T14:56:36.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:36.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:36.389512+0000) 2024-03-06T14:56:37.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:37.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:37.389766+0000) 2024-03-06T14:56:38.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:38.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:38.390013+0000) 2024-03-06T14:56:39.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:39.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:39.390264+0000) 2024-03-06T14:56:40.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:40.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:40.390520+0000) 2024-03-06T14:56:41.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:41.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:41.390787+0000) 2024-03-06T14:56:42.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:42.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:42.391025+0000) 2024-03-06T14:56:43.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:43.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:43.391263+0000) 2024-03-06T14:56:44.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:44.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:44.391498+0000) 2024-03-06T14:56:45.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:45.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:45.391748+0000) 2024-03-06T14:56:46.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:46.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:46.391995+0000) 2024-03-06T14:56:47.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:47.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:47.392174+0000) 2024-03-06T14:56:48.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:48.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:48.392433+0000) 2024-03-06T14:56:49.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:49.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:49.392721+0000) 2024-03-06T14:56:50.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:50.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:50.392967+0000) 2024-03-06T14:56:51.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:51.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:51.393211+0000) 2024-03-06T14:56:52.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:52.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:52.393463+0000) 2024-03-06T14:56:53.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:53.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:53.393750+0000) 2024-03-06T14:56:54.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:54.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:54.394006+0000) 2024-03-06T14:56:55.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:55.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:55.394245+0000) 2024-03-06T14:56:56.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:56.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:56.394507+0000) 2024-03-06T14:56:57.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:57.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:57.394797+0000) 2024-03-06T14:56:58.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:58.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:58.395033+0000) 2024-03-06T14:56:59.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:56:59.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:59.395285+0000) 2024-03-06T14:57:00.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:00.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:00.395526+0000) 2024-03-06T14:57:01.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:01.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:01.395789+0000) 2024-03-06T14:57:02.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:02.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:02.395969+0000) 2024-03-06T14:57:03.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:03.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:03.396227+0000) 2024-03-06T14:57:03.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:03.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:04.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:04.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:04.396468+0000) 2024-03-06T14:57:05.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:05.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:05.396723+0000) 2024-03-06T14:57:06.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:06.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:06.396982+0000) 2024-03-06T14:57:07.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:07.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:07.397169+0000) 2024-03-06T14:57:08.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:08.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:08.397417+0000) 2024-03-06T14:57:09.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:09.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:09.397710+0000) 2024-03-06T14:57:10.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:10.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:10.397944+0000) 2024-03-06T14:57:11.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:11.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:11.398204+0000) 2024-03-06T14:57:12.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:12.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:12.398460+0000) 2024-03-06T14:57:13.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:13.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:13.398726+0000) 2024-03-06T14:57:14.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:14.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:14.398980+0000) 2024-03-06T14:57:15.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:15.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:15.399241+0000) 2024-03-06T14:57:16.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:16.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:16.399501+0000) 2024-03-06T14:57:17.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:17.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:17.399759+0000) 2024-03-06T14:57:18.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:18.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:18.400025+0000) 2024-03-06T14:57:19.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:19.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:19.400259+0000) 2024-03-06T14:57:20.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:20.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:20.400498+0000) 2024-03-06T14:57:21.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:21.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:21.400763+0000) 2024-03-06T14:57:22.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:22.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:22.400992+0000) 2024-03-06T14:57:23.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:23.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:23.401234+0000) 2024-03-06T14:57:24.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:24.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:24.401470+0000) 2024-03-06T14:57:25.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:25.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:25.401742+0000) 2024-03-06T14:57:26.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:26.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:26.401980+0000) 2024-03-06T14:57:27.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:27.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:27.402229+0000) 2024-03-06T14:57:28.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:28.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:28.402473+0000) 2024-03-06T14:57:29.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:29.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:29.402753+0000) 2024-03-06T14:57:30.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:30.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:30.403012+0000) 2024-03-06T14:57:31.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:31.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:31.403274+0000) 2024-03-06T14:57:32.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:32.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:32.403535+0000) 2024-03-06T14:57:33.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:33.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:33.403806+0000) 2024-03-06T14:57:33.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:33.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:34.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:34.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:34.404085+0000) 2024-03-06T14:57:35.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:35.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:35.404362+0000) 2024-03-06T14:57:36.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:36.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:36.404616+0000) 2024-03-06T14:57:37.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:37.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:37.404913+0000) 2024-03-06T14:57:38.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:38.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:38.405201+0000) 2024-03-06T14:57:39.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:39.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:39.405463+0000) 2024-03-06T14:57:40.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:40.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:40.405765+0000) 2024-03-06T14:57:41.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:41.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:41.406019+0000) 2024-03-06T14:57:42.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:42.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:42.406276+0000) 2024-03-06T14:57:43.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:43.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:43.406548+0000) 2024-03-06T14:57:44.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:44.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:44.406801+0000) 2024-03-06T14:57:45.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:45.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:45.407077+0000) 2024-03-06T14:57:46.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:46.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:46.407350+0000) 2024-03-06T14:57:47.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:47.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:47.407669+0000) 2024-03-06T14:57:48.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:48.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:48.407923+0000) 2024-03-06T14:57:49.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:49.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:49.408187+0000) 2024-03-06T14:57:50.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:50.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:50.408453+0000) 2024-03-06T14:57:51.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:51.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:51.408738+0000) 2024-03-06T14:57:52.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:52.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:52.408990+0000) 2024-03-06T14:57:53.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:53.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:53.409266+0000) 2024-03-06T14:57:54.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:54.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:54.409526+0000) 2024-03-06T14:57:55.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:55.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:55.409789+0000) 2024-03-06T14:57:56.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:56.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:56.410069+0000) 2024-03-06T14:57:57.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:57.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:57.410338+0000) 2024-03-06T14:57:58.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:58.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:58.410583+0000) 2024-03-06T14:57:59.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:57:59.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:59.410893+0000) 2024-03-06T14:58:00.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:00.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:00.411148+0000) 2024-03-06T14:58:01.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:01.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:01.411402+0000) 2024-03-06T14:58:02.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:02.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:02.411701+0000) 2024-03-06T14:58:03.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:03.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:03.411988+0000) 2024-03-06T14:58:04.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:04.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:04.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:04.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:04.412270+0000) 2024-03-06T14:58:05.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:05.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:05.412547+0000) 2024-03-06T14:58:06.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:06.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:06.412804+0000) 2024-03-06T14:58:07.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:07.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:07.413040+0000) 2024-03-06T14:58:08.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:08.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:08.413278+0000) 2024-03-06T14:58:09.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:09.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:09.413520+0000) 2024-03-06T14:58:10.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:10.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:10.413765+0000) 2024-03-06T14:58:11.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:11.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:11.414036+0000) 2024-03-06T14:58:12.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:12.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:12.414281+0000) 2024-03-06T14:58:13.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:13.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:13.414521+0000) 2024-03-06T14:58:14.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:14.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:14.414783+0000) 2024-03-06T14:58:15.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:15.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:15.415043+0000) 2024-03-06T14:58:16.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:16.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:16.415297+0000) 2024-03-06T14:58:17.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:17.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:17.415552+0000) 2024-03-06T14:58:18.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:18.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:18.415829+0000) 2024-03-06T14:58:19.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:19.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:19.416086+0000) 2024-03-06T14:58:20.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:20.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:20.416360+0000) 2024-03-06T14:58:21.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:21.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:21.416677+0000) 2024-03-06T14:58:22.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:22.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:22.416952+0000) 2024-03-06T14:58:23.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:23.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:23.417198+0000) 2024-03-06T14:58:24.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:24.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:24.417452+0000) 2024-03-06T14:58:25.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:25.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:25.417742+0000) 2024-03-06T14:58:26.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:26.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:26.418018+0000) 2024-03-06T14:58:27.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:27.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:27.418292+0000) 2024-03-06T14:58:28.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:28.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:28.418562+0000) 2024-03-06T14:58:29.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:29.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:29.418817+0000) 2024-03-06T14:58:30.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:30.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:30.419073+0000) 2024-03-06T14:58:31.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:31.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:31.419364+0000) 2024-03-06T14:58:32.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:32.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:32.419679+0000) 2024-03-06T14:58:33.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:33.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:33.419957+0000) 2024-03-06T14:58:34.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:34.275 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:34.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:34.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:34.420194+0000) 2024-03-06T14:58:35.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:35.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:35.420506+0000) 2024-03-06T14:58:36.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:36.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:36.420779+0000) 2024-03-06T14:58:37.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:37.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:37.421025+0000) 2024-03-06T14:58:38.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:38.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:38.421302+0000) 2024-03-06T14:58:39.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:39.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:39.421589+0000) 2024-03-06T14:58:40.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:40.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:40.421906+0000) 2024-03-06T14:58:41.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:41.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:41.422157+0000) 2024-03-06T14:58:42.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:42.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:42.422464+0000) 2024-03-06T14:58:43.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:43.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:43.422733+0000) 2024-03-06T14:58:44.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:44.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:44.422918+0000) 2024-03-06T14:58:45.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:45.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:45.423170+0000) 2024-03-06T14:58:46.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:46.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:46.423415+0000) 2024-03-06T14:58:47.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:47.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:47.423710+0000) 2024-03-06T14:58:48.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:48.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:48.423968+0000) 2024-03-06T14:58:49.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:49.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:49.424219+0000) 2024-03-06T14:58:50.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:50.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:50.424482+0000) 2024-03-06T14:58:51.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:51.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:51.424728+0000) 2024-03-06T14:58:52.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:52.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:52.424964+0000) 2024-03-06T14:58:53.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:53.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:53.425224+0000) 2024-03-06T14:58:54.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:54.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:54.425473+0000) 2024-03-06T14:58:55.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:55.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:55.425710+0000) 2024-03-06T14:58:56.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:56.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:56.425897+0000) 2024-03-06T14:58:57.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:57.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:57.426155+0000) 2024-03-06T14:58:58.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:58.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:58.426397+0000) 2024-03-06T14:58:59.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:58:59.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:59.426695+0000) 2024-03-06T14:59:00.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:00.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:00.426959+0000) 2024-03-06T14:59:01.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:01.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:01.427220+0000) 2024-03-06T14:59:02.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:02.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:02.427472+0000) 2024-03-06T14:59:03.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:03.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:03.427751+0000) 2024-03-06T14:59:04.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:04.428 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:04.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:04.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:04.427948+0000) 2024-03-06T14:59:05.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:05.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:05.428210+0000) 2024-03-06T14:59:06.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:06.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:06.428473+0000) 2024-03-06T14:59:07.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:07.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:07.428739+0000) 2024-03-06T14:59:08.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:08.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:08.429012+0000) 2024-03-06T14:59:09.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:09.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:09.429246+0000) 2024-03-06T14:59:10.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:10.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:10.429508+0000) 2024-03-06T14:59:11.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:11.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:11.429771+0000) 2024-03-06T14:59:12.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:12.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:12.430010+0000) 2024-03-06T14:59:13.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:13.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:13.430210+0000) 2024-03-06T14:59:14.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:14.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:14.430458+0000) 2024-03-06T14:59:15.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:15.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:15.430731+0000) 2024-03-06T14:59:16.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:16.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:16.430992+0000) 2024-03-06T14:59:17.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:17.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:17.431171+0000) 2024-03-06T14:59:18.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:18.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:18.431421+0000) 2024-03-06T14:59:19.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:19.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:19.431673+0000) 2024-03-06T14:59:20.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:20.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:20.431938+0000) 2024-03-06T14:59:21.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:21.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:21.432165+0000) 2024-03-06T14:59:22.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:22.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:22.432448+0000) 2024-03-06T14:59:23.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:23.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:23.432717+0000) 2024-03-06T14:59:24.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:24.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:24.432981+0000) 2024-03-06T14:59:25.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:25.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:25.433232+0000) 2024-03-06T14:59:26.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:26.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:26.433495+0000) 2024-03-06T14:59:27.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:27.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:27.433724+0000) 2024-03-06T14:59:28.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:28.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:28.433985+0000) 2024-03-06T14:59:29.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:29.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:29.434179+0000) 2024-03-06T14:59:30.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:30.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:30.434443+0000) 2024-03-06T14:59:31.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:31.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:31.434721+0000) 2024-03-06T14:59:32.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:32.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:32.434991+0000) 2024-03-06T14:59:33.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:33.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:33.435258+0000) 2024-03-06T14:59:34.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:34.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:34.435515+0000) 2024-03-06T14:59:34.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:34.570 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:35.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:35.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:35.435822+0000) 2024-03-06T14:59:36.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:36.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:36.436081+0000) 2024-03-06T14:59:37.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:37.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:37.436344+0000) 2024-03-06T14:59:38.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:38.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:38.436598+0000) 2024-03-06T14:59:39.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:39.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:39.436846+0000) 2024-03-06T14:59:40.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:40.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:40.437094+0000) 2024-03-06T14:59:41.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:41.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:41.437353+0000) 2024-03-06T14:59:42.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:42.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:42.437564+0000) 2024-03-06T14:59:43.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:43.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:43.437820+0000) 2024-03-06T14:59:44.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:44.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:44.438081+0000) 2024-03-06T14:59:45.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:45.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:45.438361+0000) 2024-03-06T14:59:46.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:46.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:46.438624+0000) 2024-03-06T14:59:47.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:47.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:47.438958+0000) 2024-03-06T14:59:48.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:48.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:48.439233+0000) 2024-03-06T14:59:49.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:49.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:49.439484+0000) 2024-03-06T14:59:50.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:50.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:50.439744+0000) 2024-03-06T14:59:51.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:51.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:51.440004+0000) 2024-03-06T14:59:52.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:52.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:52.440324+0000) 2024-03-06T14:59:53.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:53.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:53.440586+0000) 2024-03-06T14:59:54.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:54.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:54.440827+0000) 2024-03-06T14:59:55.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:55.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:55.441081+0000) 2024-03-06T14:59:56.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:56.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:56.441344+0000) 2024-03-06T14:59:57.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:57.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:57.441606+0000) 2024-03-06T14:59:58.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:58.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:58.441929+0000) 2024-03-06T14:59:59.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T14:59:59.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:59.442199+0000) 2024-03-06T15:00:00.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:00.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:00.442401+0000) 2024-03-06T15:00:01.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:01.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:01.442717+0000) 2024-03-06T15:00:02.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:02.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:02.442935+0000) 2024-03-06T15:00:03.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:03.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:03.443148+0000) 2024-03-06T15:00:04.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:04.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:04.443411+0000) 2024-03-06T15:00:04.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:04.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:05.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:05.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:05.443704+0000) 2024-03-06T15:00:06.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:06.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:06.443937+0000) 2024-03-06T15:00:07.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:07.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:07.444148+0000) 2024-03-06T15:00:08.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:08.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:08.444369+0000) 2024-03-06T15:00:09.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:09.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:09.444579+0000) 2024-03-06T15:00:10.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:10.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:10.444747+0000) 2024-03-06T15:00:11.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:11.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:11.445041+0000) 2024-03-06T15:00:12.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:12.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:12.445292+0000) 2024-03-06T15:00:13.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:13.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:13.445537+0000) 2024-03-06T15:00:14.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:14.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:14.445813+0000) 2024-03-06T15:00:15.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:15.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:15.446050+0000) 2024-03-06T15:00:16.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:16.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:16.446310+0000) 2024-03-06T15:00:17.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:17.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:17.446564+0000) 2024-03-06T15:00:18.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:18.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:18.446833+0000) 2024-03-06T15:00:19.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:19.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:19.447087+0000) 2024-03-06T15:00:20.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:20.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:20.447352+0000) 2024-03-06T15:00:21.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:21.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:21.447599+0000) 2024-03-06T15:00:22.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:22.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:22.447892+0000) 2024-03-06T15:00:23.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:23.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:23.448203+0000) 2024-03-06T15:00:24.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:24.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:24.448464+0000) 2024-03-06T15:00:25.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:25.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:25.448762+0000) 2024-03-06T15:00:26.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:26.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:26.449021+0000) 2024-03-06T15:00:27.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:27.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:27.449264+0000) 2024-03-06T15:00:28.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:28.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:28.449528+0000) 2024-03-06T15:00:29.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:29.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:29.449803+0000) 2024-03-06T15:00:30.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:30.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:30.450062+0000) 2024-03-06T15:00:31.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:31.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:31.450326+0000) 2024-03-06T15:00:32.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:32.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:32.450587+0000) 2024-03-06T15:00:33.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:33.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:33.450852+0000) 2024-03-06T15:00:34.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:34.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:34.451088+0000) 2024-03-06T15:00:34.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:34.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:35.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:35.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:35.451331+0000) 2024-03-06T15:00:36.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:36.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:36.451571+0000) 2024-03-06T15:00:37.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:37.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:37.451809+0000) 2024-03-06T15:00:38.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:38.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:38.452052+0000) 2024-03-06T15:00:39.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:39.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:39.452305+0000) 2024-03-06T15:00:40.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:40.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:40.452553+0000) 2024-03-06T15:00:41.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:41.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:41.452815+0000) 2024-03-06T15:00:42.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:42.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:42.453070+0000) 2024-03-06T15:00:43.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:43.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:43.453320+0000) 2024-03-06T15:00:44.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:44.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:44.453591+0000) 2024-03-06T15:00:45.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:45.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:45.453908+0000) 2024-03-06T15:00:46.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:46.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:46.454190+0000) 2024-03-06T15:00:47.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:47.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:47.454463+0000) 2024-03-06T15:00:48.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:48.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:48.454743+0000) 2024-03-06T15:00:49.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:49.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:49.455019+0000) 2024-03-06T15:00:50.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:50.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:50.455233+0000) 2024-03-06T15:00:51.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:51.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:51.455526+0000) 2024-03-06T15:00:52.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:52.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:52.455788+0000) 2024-03-06T15:00:53.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:53.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:53.456049+0000) 2024-03-06T15:00:54.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:54.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:54.456330+0000) 2024-03-06T15:00:55.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:55.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:55.456582+0000) 2024-03-06T15:00:56.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:56.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:56.456865+0000) 2024-03-06T15:00:57.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:57.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:57.457115+0000) 2024-03-06T15:00:58.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:58.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:58.457366+0000) 2024-03-06T15:00:59.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:00:59.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:59.457665+0000) 2024-03-06T15:01:00.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:00.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:00.457950+0000) 2024-03-06T15:01:01.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:01.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:01.458225+0000) 2024-03-06T15:01:02.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:02.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:02.458482+0000) 2024-03-06T15:01:03.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:03.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:03.458734+0000) 2024-03-06T15:01:04.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:04.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:04.459026+0000) 2024-03-06T15:01:05.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:05.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:05.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:05.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:05.459307+0000) 2024-03-06T15:01:06.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:06.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:06.459597+0000) 2024-03-06T15:01:07.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:07.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:07.459881+0000) 2024-03-06T15:01:08.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:08.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:08.460152+0000) 2024-03-06T15:01:09.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:09.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:09.460366+0000) 2024-03-06T15:01:10.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:10.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:10.460664+0000) 2024-03-06T15:01:11.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:11.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:11.460977+0000) 2024-03-06T15:01:12.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:12.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:12.461257+0000) 2024-03-06T15:01:13.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:13.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:13.461524+0000) 2024-03-06T15:01:14.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:14.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:14.461813+0000) 2024-03-06T15:01:15.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:15.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:15.462064+0000) 2024-03-06T15:01:16.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:16.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:16.462326+0000) 2024-03-06T15:01:17.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:17.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:17.462550+0000) 2024-03-06T15:01:18.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:18.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:18.462806+0000) 2024-03-06T15:01:19.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:19.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:19.463071+0000) 2024-03-06T15:01:20.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:20.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:20.463320+0000) 2024-03-06T15:01:21.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:21.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:21.463514+0000) 2024-03-06T15:01:22.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:22.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:22.463825+0000) 2024-03-06T15:01:23.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:23.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:23.464073+0000) 2024-03-06T15:01:24.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:24.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:24.464324+0000) 2024-03-06T15:01:25.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:25.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:25.464555+0000) 2024-03-06T15:01:26.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:26.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:26.464815+0000) 2024-03-06T15:01:27.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:27.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:27.465076+0000) 2024-03-06T15:01:28.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:28.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:28.465326+0000) 2024-03-06T15:01:29.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:29.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:29.465597+0000) 2024-03-06T15:01:30.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:30.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:30.465862+0000) 2024-03-06T15:01:31.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:31.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:31.466102+0000) 2024-03-06T15:01:32.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:32.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:32.466360+0000) 2024-03-06T15:01:33.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:33.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:33.466594+0000) 2024-03-06T15:01:34.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:34.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:34.466866+0000) 2024-03-06T15:01:35.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:35.175 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:35.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:35.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:35.467060+0000) 2024-03-06T15:01:36.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:36.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:36.467306+0000) 2024-03-06T15:01:37.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:37.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:37.467557+0000) 2024-03-06T15:01:38.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:38.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:38.467807+0000) 2024-03-06T15:01:39.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:39.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:39.468179+0000) 2024-03-06T15:01:40.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:40.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:40.468452+0000) 2024-03-06T15:01:41.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:41.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:41.468749+0000) 2024-03-06T15:01:42.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:42.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:42.469002+0000) 2024-03-06T15:01:43.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:43.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:43.469298+0000) 2024-03-06T15:01:44.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:44.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:44.469609+0000) 2024-03-06T15:01:45.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:45.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:45.469982+0000) 2024-03-06T15:01:46.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:46.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:46.470273+0000) 2024-03-06T15:01:47.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:47.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:47.470502+0000) 2024-03-06T15:01:48.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:48.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:48.470783+0000) 2024-03-06T15:01:49.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:49.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:49.471083+0000) 2024-03-06T15:01:50.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:50.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:50.471397+0000) 2024-03-06T15:01:51.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:51.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:51.471752+0000) 2024-03-06T15:01:52.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:52.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:52.472057+0000) 2024-03-06T15:01:53.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:53.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:53.472329+0000) 2024-03-06T15:01:54.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:54.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:54.472679+0000) 2024-03-06T15:01:55.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:55.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:55.473014+0000) 2024-03-06T15:01:56.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:56.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:56.473337+0000) 2024-03-06T15:01:57.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:57.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:57.473672+0000) 2024-03-06T15:01:58.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:58.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:58.473934+0000) 2024-03-06T15:01:59.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:01:59.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:59.474246+0000) 2024-03-06T15:02:00.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:00.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:00.474540+0000) 2024-03-06T15:02:01.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:01.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:01.474795+0000) 2024-03-06T15:02:02.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:02.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:02.475093+0000) 2024-03-06T15:02:03.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:03.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:03.475328+0000) 2024-03-06T15:02:04.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:04.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:04.475687+0000) 2024-03-06T15:02:05.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:05.322 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:05.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:05.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:05.475973+0000) 2024-03-06T15:02:06.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:06.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:06.476289+0000) 2024-03-06T15:02:07.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:07.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:07.476586+0000) 2024-03-06T15:02:08.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:08.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:08.476890+0000) 2024-03-06T15:02:09.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:09.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:09.477215+0000) 2024-03-06T15:02:10.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:10.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:10.477574+0000) 2024-03-06T15:02:11.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:11.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:11.477922+0000) 2024-03-06T15:02:12.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:12.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:12.478225+0000) 2024-03-06T15:02:13.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:13.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:13.478538+0000) 2024-03-06T15:02:14.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:14.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:14.478831+0000) 2024-03-06T15:02:15.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:15.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:15.479145+0000) 2024-03-06T15:02:16.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:16.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:16.479447+0000) 2024-03-06T15:02:17.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:17.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:17.479789+0000) 2024-03-06T15:02:18.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:18.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:18.480055+0000) 2024-03-06T15:02:19.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:19.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:19.480361+0000) 2024-03-06T15:02:20.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:20.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:20.480724+0000) 2024-03-06T15:02:21.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:21.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:21.481056+0000) 2024-03-06T15:02:22.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:22.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:22.481370+0000) 2024-03-06T15:02:23.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:23.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:23.481719+0000) 2024-03-06T15:02:24.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:24.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:24.482040+0000) 2024-03-06T15:02:25.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:25.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:25.482348+0000) 2024-03-06T15:02:26.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:26.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:26.482697+0000) 2024-03-06T15:02:27.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:27.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:27.482995+0000) 2024-03-06T15:02:28.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:28.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:28.483310+0000) 2024-03-06T15:02:29.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:29.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:29.483615+0000) 2024-03-06T15:02:30.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:30.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:30.483994+0000) 2024-03-06T15:02:31.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:31.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:31.484302+0000) 2024-03-06T15:02:32.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:32.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:32.484623+0000) 2024-03-06T15:02:33.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:33.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:33.484974+0000) 2024-03-06T15:02:34.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:34.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:34.485263+0000) 2024-03-06T15:02:35.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:35.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:35.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:35.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:35.485484+0000) 2024-03-06T15:02:36.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:36.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:36.485784+0000) 2024-03-06T15:02:37.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:37.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:37.486096+0000) 2024-03-06T15:02:38.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:38.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:38.486426+0000) 2024-03-06T15:02:39.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:39.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:39.486788+0000) 2024-03-06T15:02:40.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:40.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:40.487110+0000) 2024-03-06T15:02:41.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:41.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:41.487437+0000) 2024-03-06T15:02:42.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:42.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:42.487774+0000) 2024-03-06T15:02:43.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:43.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:43.488093+0000) 2024-03-06T15:02:44.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:44.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:44.488415+0000) 2024-03-06T15:02:45.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:45.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:45.488703+0000) 2024-03-06T15:02:46.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:46.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:46.488972+0000) 2024-03-06T15:02:47.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:47.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:47.489235+0000) 2024-03-06T15:02:48.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:48.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:48.489513+0000) 2024-03-06T15:02:49.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:49.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:49.489799+0000) 2024-03-06T15:02:50.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:50.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:50.490116+0000) 2024-03-06T15:02:51.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:51.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:51.490460+0000) 2024-03-06T15:02:52.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:52.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:52.490827+0000) 2024-03-06T15:02:53.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:53.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:53.491140+0000) 2024-03-06T15:02:54.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:54.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:54.491461+0000) 2024-03-06T15:02:55.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:55.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:55.491794+0000) 2024-03-06T15:02:56.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:56.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:56.492047+0000) 2024-03-06T15:02:57.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:57.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:57.492341+0000) 2024-03-06T15:02:58.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:58.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:58.492681+0000) 2024-03-06T15:02:59.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:02:59.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:59.492923+0000) 2024-03-06T15:03:00.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:00.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:00.493204+0000) 2024-03-06T15:03:01.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:01.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:01.493519+0000) 2024-03-06T15:03:02.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:02.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:02.493835+0000) 2024-03-06T15:03:03.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:03.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:03.494146+0000) 2024-03-06T15:03:04.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:04.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:04.494447+0000) 2024-03-06T15:03:05.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:05.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:05.494800+0000) 2024-03-06T15:03:05.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:05.616 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:06.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:06.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:06.495108+0000) 2024-03-06T15:03:07.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:07.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:07.495398+0000) 2024-03-06T15:03:08.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:08.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:08.495740+0000) 2024-03-06T15:03:09.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:09.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:09.496039+0000) 2024-03-06T15:03:10.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:10.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:10.496328+0000) 2024-03-06T15:03:11.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:11.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:11.496703+0000) 2024-03-06T15:03:12.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:12.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:12.497026+0000) 2024-03-06T15:03:13.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:13.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:13.497334+0000) 2024-03-06T15:03:14.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:14.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:14.497672+0000) 2024-03-06T15:03:15.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:15.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:15.497989+0000) 2024-03-06T15:03:16.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:16.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:16.498324+0000) 2024-03-06T15:03:17.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:17.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:17.498620+0000) 2024-03-06T15:03:18.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:18.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:18.499000+0000) 2024-03-06T15:03:19.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:19.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:19.499289+0000) 2024-03-06T15:03:20.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:20.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:20.499600+0000) 2024-03-06T15:03:21.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:21.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:21.499931+0000) 2024-03-06T15:03:22.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:22.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:22.500163+0000) 2024-03-06T15:03:23.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:23.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:23.500415+0000) 2024-03-06T15:03:24.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:24.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:24.500719+0000) 2024-03-06T15:03:25.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:25.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:25.500979+0000) 2024-03-06T15:03:26.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:26.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:26.501234+0000) 2024-03-06T15:03:27.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:27.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:27.501482+0000) 2024-03-06T15:03:28.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:28.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:28.501708+0000) 2024-03-06T15:03:29.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:29.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:29.501896+0000) 2024-03-06T15:03:30.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:30.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:30.502160+0000) 2024-03-06T15:03:31.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:31.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:31.502434+0000) 2024-03-06T15:03:32.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:32.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:32.502742+0000) 2024-03-06T15:03:33.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:33.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:33.502991+0000) 2024-03-06T15:03:34.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:34.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:34.503182+0000) 2024-03-06T15:03:35.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:35.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:35.503445+0000) 2024-03-06T15:03:35.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:35.789 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:36.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:36.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:36.503752+0000) 2024-03-06T15:03:37.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:37.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:37.504010+0000) 2024-03-06T15:03:38.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:38.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:38.504252+0000) 2024-03-06T15:03:39.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:39.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:39.504503+0000) 2024-03-06T15:03:40.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:40.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:40.504792+0000) 2024-03-06T15:03:41.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:41.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:41.505072+0000) 2024-03-06T15:03:42.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:42.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:42.505321+0000) 2024-03-06T15:03:43.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:43.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:43.505553+0000) 2024-03-06T15:03:44.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:44.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:44.505806+0000) 2024-03-06T15:03:45.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:45.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:45.506061+0000) 2024-03-06T15:03:46.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:46.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:46.506310+0000) 2024-03-06T15:03:47.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:47.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:47.506560+0000) 2024-03-06T15:03:48.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:48.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:48.506817+0000) 2024-03-06T15:03:49.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:49.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:49.507062+0000) 2024-03-06T15:03:50.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:50.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:50.507324+0000) 2024-03-06T15:03:51.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:51.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:51.507603+0000) 2024-03-06T15:03:52.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:52.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:52.507909+0000) 2024-03-06T15:03:53.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:53.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:53.508161+0000) 2024-03-06T15:03:54.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:54.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:54.508415+0000) 2024-03-06T15:03:55.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:55.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:55.508702+0000) 2024-03-06T15:03:56.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:56.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:56.508974+0000) 2024-03-06T15:03:57.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:57.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:57.509249+0000) 2024-03-06T15:03:58.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:58.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:58.509528+0000) 2024-03-06T15:03:59.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:03:59.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:59.509793+0000) 2024-03-06T15:04:00.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:00.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:00.510051+0000) 2024-03-06T15:04:01.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:01.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:01.510343+0000) 2024-03-06T15:04:02.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:02.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:02.510619+0000) 2024-03-06T15:04:03.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:03.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:03.510932+0000) 2024-03-06T15:04:04.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:04.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:04.511201+0000) 2024-03-06T15:04:05.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:05.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:05.511473+0000) 2024-03-06T15:04:05.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:05.937 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:06.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:06.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:06.511752+0000) 2024-03-06T15:04:07.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:07.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:07.512005+0000) 2024-03-06T15:04:08.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:08.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:08.512259+0000) 2024-03-06T15:04:09.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:09.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:09.512508+0000) 2024-03-06T15:04:10.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:10.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:10.512782+0000) 2024-03-06T15:04:11.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:11.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:11.513065+0000) 2024-03-06T15:04:12.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:12.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:12.513283+0000) 2024-03-06T15:04:13.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:13.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:13.513523+0000) 2024-03-06T15:04:14.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:14.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:14.513776+0000) 2024-03-06T15:04:15.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:15.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:15.514033+0000) 2024-03-06T15:04:16.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:16.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:16.514279+0000) 2024-03-06T15:04:17.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:17.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:17.514530+0000) 2024-03-06T15:04:18.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:18.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:18.514769+0000) 2024-03-06T15:04:19.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:19.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:19.515018+0000) 2024-03-06T15:04:20.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:20.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:20.515279+0000) 2024-03-06T15:04:21.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:21.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:21.515540+0000) 2024-03-06T15:04:22.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:22.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:22.515780+0000) 2024-03-06T15:04:23.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:23.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:23.516037+0000) 2024-03-06T15:04:24.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:24.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:24.516279+0000) 2024-03-06T15:04:25.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:25.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:25.516517+0000) 2024-03-06T15:04:26.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:26.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:26.516778+0000) 2024-03-06T15:04:27.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:27.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:27.517029+0000) 2024-03-06T15:04:28.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:28.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:28.517350+0000) 2024-03-06T15:04:29.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:29.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:29.517692+0000) 2024-03-06T15:04:30.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:30.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:30.518009+0000) 2024-03-06T15:04:31.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:31.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:31.518330+0000) 2024-03-06T15:04:32.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:32.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:32.518626+0000) 2024-03-06T15:04:33.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:33.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:33.518992+0000) 2024-03-06T15:04:34.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:34.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:34.519301+0000) 2024-03-06T15:04:35.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:35.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:35.519603+0000) 2024-03-06T15:04:36.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:36.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:36.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:36.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:36.519966+0000) 2024-03-06T15:04:37.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:37.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:37.520267+0000) 2024-03-06T15:04:38.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:38.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:38.520590+0000) 2024-03-06T15:04:39.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:39.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:39.520938+0000) 2024-03-06T15:04:40.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:40.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:40.521242+0000) 2024-03-06T15:04:41.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:41.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:41.521554+0000) 2024-03-06T15:04:42.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:42.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:42.521872+0000) 2024-03-06T15:04:43.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:43.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:43.522186+0000) 2024-03-06T15:04:44.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:44.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:44.522492+0000) 2024-03-06T15:04:45.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:45.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:45.522800+0000) 2024-03-06T15:04:46.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:46.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:46.523112+0000) 2024-03-06T15:04:47.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:47.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:47.523414+0000) 2024-03-06T15:04:48.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:48.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:48.523770+0000) 2024-03-06T15:04:49.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:49.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:49.524082+0000) 2024-03-06T15:04:50.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:50.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:50.524373+0000) 2024-03-06T15:04:51.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:51.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:51.524719+0000) 2024-03-06T15:04:52.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:52.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:52.525017+0000) 2024-03-06T15:04:53.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:53.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:53.525334+0000) 2024-03-06T15:04:54.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:54.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:54.525685+0000) 2024-03-06T15:04:55.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:55.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:55.525979+0000) 2024-03-06T15:04:56.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:56.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:56.526240+0000) 2024-03-06T15:04:57.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:57.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:57.526476+0000) 2024-03-06T15:04:58.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:58.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:58.526756+0000) 2024-03-06T15:04:59.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:04:59.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:59.526989+0000) 2024-03-06T15:05:00.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:00.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:00.527242+0000) 2024-03-06T15:05:01.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:01.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:01.527496+0000) 2024-03-06T15:05:02.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:02.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:02.527743+0000) 2024-03-06T15:05:03.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:03.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:03.527999+0000) 2024-03-06T15:05:04.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:04.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:04.528257+0000) 2024-03-06T15:05:05.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:05.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:05.528523+0000) 2024-03-06T15:05:06.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:06.231 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:06.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:06.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:06.528808+0000) 2024-03-06T15:05:07.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:07.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:07.529042+0000) 2024-03-06T15:05:08.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:08.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:08.529298+0000) 2024-03-06T15:05:09.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:09.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:09.529531+0000) 2024-03-06T15:05:10.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:10.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:10.529784+0000) 2024-03-06T15:05:11.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:11.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:11.530018+0000) 2024-03-06T15:05:12.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:12.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:12.530262+0000) 2024-03-06T15:05:13.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:13.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:13.530515+0000) 2024-03-06T15:05:14.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:14.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:14.530751+0000) 2024-03-06T15:05:15.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:15.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:15.531016+0000) 2024-03-06T15:05:16.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:16.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:16.531261+0000) 2024-03-06T15:05:17.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:17.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:17.531499+0000) 2024-03-06T15:05:18.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:18.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:18.531758+0000) 2024-03-06T15:05:19.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:19.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:19.532005+0000) 2024-03-06T15:05:20.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:20.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:20.532258+0000) 2024-03-06T15:05:21.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:21.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:21.532508+0000) 2024-03-06T15:05:22.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:22.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:22.532766+0000) 2024-03-06T15:05:23.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:23.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:23.533025+0000) 2024-03-06T15:05:24.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:24.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:24.533280+0000) 2024-03-06T15:05:25.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:25.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:25.533540+0000) 2024-03-06T15:05:26.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:26.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:26.533795+0000) 2024-03-06T15:05:27.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:27.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:27.534033+0000) 2024-03-06T15:05:28.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:28.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:28.534298+0000) 2024-03-06T15:05:29.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:29.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:29.534479+0000) 2024-03-06T15:05:30.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:30.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:30.534761+0000) 2024-03-06T15:05:31.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:31.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:31.535009+0000) 2024-03-06T15:05:32.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:32.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:32.535204+0000) 2024-03-06T15:05:33.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:33.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:33.535453+0000) 2024-03-06T15:05:34.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:34.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:34.535743+0000) 2024-03-06T15:05:35.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:35.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:35.536007+0000) 2024-03-06T15:05:36.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:36.375 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:36.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:36.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:36.536278+0000) 2024-03-06T15:05:37.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:37.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:37.536540+0000) 2024-03-06T15:05:38.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:38.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:38.536806+0000) 2024-03-06T15:05:39.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:39.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:39.537054+0000) 2024-03-06T15:05:40.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:40.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:40.537313+0000) 2024-03-06T15:05:41.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:41.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:41.537572+0000) 2024-03-06T15:05:42.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:42.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:42.537839+0000) 2024-03-06T15:05:43.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:43.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:43.538097+0000) 2024-03-06T15:05:44.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:44.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:44.538350+0000) 2024-03-06T15:05:45.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:45.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:45.538613+0000) 2024-03-06T15:05:46.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:46.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:46.538908+0000) 2024-03-06T15:05:47.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:47.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:47.539159+0000) 2024-03-06T15:05:48.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:48.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:48.539405+0000) 2024-03-06T15:05:49.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:49.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:49.539685+0000) 2024-03-06T15:05:50.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:50.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:50.539950+0000) 2024-03-06T15:05:51.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:51.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:51.540215+0000) 2024-03-06T15:05:52.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:52.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:52.540407+0000) 2024-03-06T15:05:53.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:53.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:53.540704+0000) 2024-03-06T15:05:54.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:54.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:54.540909+0000) 2024-03-06T15:05:55.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:55.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:55.541158+0000) 2024-03-06T15:05:56.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:56.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:56.541404+0000) 2024-03-06T15:05:57.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:57.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:57.541699+0000) 2024-03-06T15:05:58.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:58.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:58.541945+0000) 2024-03-06T15:05:59.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:05:59.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:59.542183+0000) 2024-03-06T15:06:00.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:00.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:00.542440+0000) 2024-03-06T15:06:01.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:01.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:01.542755+0000) 2024-03-06T15:06:02.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:02.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:02.543025+0000) 2024-03-06T15:06:03.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:03.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:03.543264+0000) 2024-03-06T15:06:04.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:04.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:04.543528+0000) 2024-03-06T15:06:05.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:05.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:05.543761+0000) 2024-03-06T15:06:06.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:06.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:06.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:06.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:06.543962+0000) 2024-03-06T15:06:07.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:07.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:07.544220+0000) 2024-03-06T15:06:08.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:08.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:08.544484+0000) 2024-03-06T15:06:09.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:09.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:09.544753+0000) 2024-03-06T15:06:10.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:10.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:10.545028+0000) 2024-03-06T15:06:11.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:11.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:11.545295+0000) 2024-03-06T15:06:12.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:12.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:12.545541+0000) 2024-03-06T15:06:13.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:13.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:13.545805+0000) 2024-03-06T15:06:14.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:14.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:14.546065+0000) 2024-03-06T15:06:15.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:15.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:15.546316+0000) 2024-03-06T15:06:16.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:16.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:16.546566+0000) 2024-03-06T15:06:17.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:17.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:17.546829+0000) 2024-03-06T15:06:18.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:18.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:18.547092+0000) 2024-03-06T15:06:19.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:19.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:19.547282+0000) 2024-03-06T15:06:20.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:20.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:20.547605+0000) 2024-03-06T15:06:21.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:21.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:21.547886+0000) 2024-03-06T15:06:22.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:22.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:22.548134+0000) 2024-03-06T15:06:23.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:23.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:23.548393+0000) 2024-03-06T15:06:24.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:24.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:24.548665+0000) 2024-03-06T15:06:25.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:25.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:25.548941+0000) 2024-03-06T15:06:26.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:26.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:26.549212+0000) 2024-03-06T15:06:27.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:27.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:27.549403+0000) 2024-03-06T15:06:28.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:28.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:28.549675+0000) 2024-03-06T15:06:29.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:29.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:29.549926+0000) 2024-03-06T15:06:30.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:30.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:30.550182+0000) 2024-03-06T15:06:31.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:31.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:31.550449+0000) 2024-03-06T15:06:32.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:32.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:32.550768+0000) 2024-03-06T15:06:33.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:33.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:33.550969+0000) 2024-03-06T15:06:34.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:34.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:34.551225+0000) 2024-03-06T15:06:35.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:35.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:35.551473+0000) 2024-03-06T15:06:36.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:36.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:36.551765+0000) 2024-03-06T15:06:36.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:36.691 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:37.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:37.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:37.552031+0000) 2024-03-06T15:06:38.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:38.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:38.552292+0000) 2024-03-06T15:06:39.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:39.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:39.552556+0000) 2024-03-06T15:06:40.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:40.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:40.552819+0000) 2024-03-06T15:06:41.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:41.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:41.553090+0000) 2024-03-06T15:06:42.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:42.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:42.553358+0000) 2024-03-06T15:06:43.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:43.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:43.553613+0000) 2024-03-06T15:06:44.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:44.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:44.553917+0000) 2024-03-06T15:06:45.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:45.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:45.554183+0000) 2024-03-06T15:06:46.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:46.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:46.554448+0000) 2024-03-06T15:06:47.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:47.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:47.554745+0000) 2024-03-06T15:06:48.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:48.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:48.555007+0000) 2024-03-06T15:06:49.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:49.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:49.555271+0000) 2024-03-06T15:06:50.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:50.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:50.555541+0000) 2024-03-06T15:06:51.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:51.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:51.555801+0000) 2024-03-06T15:06:52.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:52.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:52.556071+0000) 2024-03-06T15:06:53.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:53.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:53.556336+0000) 2024-03-06T15:06:54.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:54.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:54.556594+0000) 2024-03-06T15:06:55.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:55.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:55.556842+0000) 2024-03-06T15:06:56.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:56.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:56.557049+0000) 2024-03-06T15:06:57.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:57.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:57.557320+0000) 2024-03-06T15:06:58.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:58.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:58.557580+0000) 2024-03-06T15:06:59.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:06:59.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:59.557835+0000) 2024-03-06T15:07:00.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:00.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:00.558106+0000) 2024-03-06T15:07:01.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:01.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:01.558360+0000) 2024-03-06T15:07:02.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:02.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:02.558608+0000) 2024-03-06T15:07:03.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:03.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:03.558827+0000) 2024-03-06T15:07:04.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:04.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:04.559079+0000) 2024-03-06T15:07:05.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:05.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:05.559322+0000) 2024-03-06T15:07:06.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:06.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:06.559566+0000) 2024-03-06T15:07:06.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:06.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:07.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:07.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:07.559837+0000) 2024-03-06T15:07:08.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:08.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:08.560033+0000) 2024-03-06T15:07:09.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:09.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:09.560281+0000) 2024-03-06T15:07:10.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:10.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:10.560516+0000) 2024-03-06T15:07:11.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:11.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:11.560801+0000) 2024-03-06T15:07:12.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:12.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:12.561086+0000) 2024-03-06T15:07:13.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:13.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:13.561344+0000) 2024-03-06T15:07:14.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:14.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:14.561599+0000) 2024-03-06T15:07:15.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:15.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:15.561869+0000) 2024-03-06T15:07:16.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:16.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:16.562124+0000) 2024-03-06T15:07:17.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:17.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:17.562368+0000) 2024-03-06T15:07:18.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:18.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:18.562624+0000) 2024-03-06T15:07:19.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:19.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:19.562895+0000) 2024-03-06T15:07:20.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:20.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:20.563156+0000) 2024-03-06T15:07:21.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:21.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:21.563417+0000) 2024-03-06T15:07:22.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:22.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:22.563708+0000) 2024-03-06T15:07:23.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:23.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:23.563965+0000) 2024-03-06T15:07:24.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:24.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:24.564200+0000) 2024-03-06T15:07:25.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:25.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:25.564434+0000) 2024-03-06T15:07:26.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:26.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:26.564726+0000) 2024-03-06T15:07:27.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:27.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:27.564986+0000) 2024-03-06T15:07:28.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:28.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:28.565244+0000) 2024-03-06T15:07:29.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:29.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:29.565495+0000) 2024-03-06T15:07:30.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:30.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:30.565780+0000) 2024-03-06T15:07:31.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:31.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:31.566059+0000) 2024-03-06T15:07:32.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:32.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:32.566376+0000) 2024-03-06T15:07:33.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:33.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:33.566733+0000) 2024-03-06T15:07:34.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:34.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:34.567041+0000) 2024-03-06T15:07:35.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:35.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:35.567327+0000) 2024-03-06T15:07:36.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:36.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:36.567682+0000) 2024-03-06T15:07:36.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:37.004 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:37.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:37.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:37.567989+0000) 2024-03-06T15:07:38.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:38.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:38.568305+0000) 2024-03-06T15:07:39.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:39.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:39.568603+0000) 2024-03-06T15:07:40.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:40.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:40.568970+0000) 2024-03-06T15:07:41.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:41.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:41.569285+0000) 2024-03-06T15:07:42.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:42.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:42.569601+0000) 2024-03-06T15:07:43.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:43.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:43.569940+0000) 2024-03-06T15:07:44.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:44.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:44.570229+0000) 2024-03-06T15:07:45.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:45.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:45.570534+0000) 2024-03-06T15:07:46.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:46.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:46.570849+0000) 2024-03-06T15:07:47.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:47.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:47.571164+0000) 2024-03-06T15:07:48.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:48.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:48.571446+0000) 2024-03-06T15:07:49.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:49.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:49.571766+0000) 2024-03-06T15:07:50.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:50.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:50.572063+0000) 2024-03-06T15:07:51.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:51.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:51.572418+0000) 2024-03-06T15:07:52.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:52.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:52.572747+0000) 2024-03-06T15:07:53.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:53.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:53.573035+0000) 2024-03-06T15:07:54.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:54.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:54.573342+0000) 2024-03-06T15:07:55.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:55.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:55.573674+0000) 2024-03-06T15:07:56.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:56.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:56.573979+0000) 2024-03-06T15:07:57.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:57.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:57.574286+0000) 2024-03-06T15:07:58.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:58.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:58.574517+0000) 2024-03-06T15:07:59.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:07:59.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:59.574819+0000) 2024-03-06T15:08:00.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:00.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:00.575127+0000) 2024-03-06T15:08:01.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:01.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:01.575442+0000) 2024-03-06T15:08:02.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:02.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:02.575785+0000) 2024-03-06T15:08:03.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:03.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:03.576085+0000) 2024-03-06T15:08:04.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:04.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:04.576403+0000) 2024-03-06T15:08:05.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:05.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:05.576750+0000) 2024-03-06T15:08:06.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:06.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:06.577066+0000) 2024-03-06T15:08:07.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:07.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:07.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:07.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:07.577362+0000) 2024-03-06T15:08:08.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:08.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:08.577714+0000) 2024-03-06T15:08:09.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:09.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:09.578018+0000) 2024-03-06T15:08:10.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:10.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:10.578336+0000) 2024-03-06T15:08:11.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:11.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:11.578716+0000) 2024-03-06T15:08:12.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:12.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:12.578977+0000) 2024-03-06T15:08:13.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:13.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:13.579230+0000) 2024-03-06T15:08:14.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:14.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:14.579472+0000) 2024-03-06T15:08:15.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:15.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:15.579750+0000) 2024-03-06T15:08:16.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:16.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:16.580007+0000) 2024-03-06T15:08:17.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:17.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:17.580282+0000) 2024-03-06T15:08:18.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:18.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:18.580549+0000) 2024-03-06T15:08:19.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:19.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:19.580808+0000) 2024-03-06T15:08:20.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:20.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:20.581065+0000) 2024-03-06T15:08:21.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:21.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:21.581314+0000) 2024-03-06T15:08:22.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:22.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:22.581566+0000) 2024-03-06T15:08:23.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:23.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:23.581807+0000) 2024-03-06T15:08:24.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:24.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:24.582073+0000) 2024-03-06T15:08:25.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:25.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:25.582270+0000) 2024-03-06T15:08:26.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:26.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:26.582532+0000) 2024-03-06T15:08:27.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:27.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:27.582783+0000) 2024-03-06T15:08:28.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:28.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:28.583048+0000) 2024-03-06T15:08:29.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:29.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:29.583287+0000) 2024-03-06T15:08:30.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:30.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:30.583530+0000) 2024-03-06T15:08:31.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:31.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:31.583799+0000) 2024-03-06T15:08:32.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:32.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:32.584067+0000) 2024-03-06T15:08:33.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:33.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:33.584333+0000) 2024-03-06T15:08:34.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:34.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:34.584596+0000) 2024-03-06T15:08:35.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:35.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:35.584813+0000) 2024-03-06T15:08:36.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:36.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:36.585154+0000) 2024-03-06T15:08:37.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:37.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:37.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:37.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:37.585455+0000) 2024-03-06T15:08:38.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:38.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:38.585800+0000) 2024-03-06T15:08:39.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:39.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:39.586107+0000) 2024-03-06T15:08:40.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:40.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:40.586347+0000) 2024-03-06T15:08:41.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:41.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:41.586618+0000) 2024-03-06T15:08:42.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:42.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:42.586911+0000) 2024-03-06T15:08:43.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:43.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:43.587169+0000) 2024-03-06T15:08:44.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:44.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:44.587408+0000) 2024-03-06T15:08:45.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:45.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:45.587687+0000) 2024-03-06T15:08:46.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:46.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:46.588016+0000) 2024-03-06T15:08:47.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:47.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:47.588338+0000) 2024-03-06T15:08:48.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:48.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:48.588672+0000) 2024-03-06T15:08:49.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:49.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:49.588988+0000) 2024-03-06T15:08:50.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:50.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:50.589337+0000) 2024-03-06T15:08:51.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:51.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:51.589716+0000) 2024-03-06T15:08:52.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:52.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:52.590021+0000) 2024-03-06T15:08:53.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:53.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:53.590348+0000) 2024-03-06T15:08:54.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:54.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:54.590587+0000) 2024-03-06T15:08:55.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:55.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:55.590936+0000) 2024-03-06T15:08:56.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:56.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:56.591192+0000) 2024-03-06T15:08:57.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:57.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:57.591478+0000) 2024-03-06T15:08:58.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:58.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:58.591767+0000) 2024-03-06T15:08:59.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:08:59.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:59.592064+0000) 2024-03-06T15:09:00.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:00.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:00.592365+0000) 2024-03-06T15:09:01.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:01.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:01.592687+0000) 2024-03-06T15:09:02.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:02.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:02.592965+0000) 2024-03-06T15:09:03.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:03.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:03.593263+0000) 2024-03-06T15:09:04.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:04.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:04.593517+0000) 2024-03-06T15:09:05.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:05.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:05.593826+0000) 2024-03-06T15:09:06.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:06.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:06.594151+0000) 2024-03-06T15:09:07.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:07.444 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:07.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:07.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:07.594459+0000) 2024-03-06T15:09:08.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:08.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:08.594807+0000) 2024-03-06T15:09:09.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:09.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:09.595055+0000) 2024-03-06T15:09:10.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:10.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:10.595351+0000) 2024-03-06T15:09:11.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:11.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:11.595730+0000) 2024-03-06T15:09:12.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:12.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:12.596033+0000) 2024-03-06T15:09:13.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:13.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:13.596313+0000) 2024-03-06T15:09:14.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:14.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:14.596610+0000) 2024-03-06T15:09:15.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:15.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:15.596957+0000) 2024-03-06T15:09:16.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:16.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:16.597242+0000) 2024-03-06T15:09:17.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:17.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:17.597539+0000) 2024-03-06T15:09:18.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:18.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:18.597793+0000) 2024-03-06T15:09:19.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:19.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:19.598075+0000) 2024-03-06T15:09:20.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:20.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:20.598390+0000) 2024-03-06T15:09:21.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:21.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:21.598742+0000) 2024-03-06T15:09:22.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:22.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:22.599046+0000) 2024-03-06T15:09:23.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:23.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:23.599346+0000) 2024-03-06T15:09:24.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:24.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:24.599681+0000) 2024-03-06T15:09:25.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:25.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:25.599978+0000) 2024-03-06T15:09:26.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:26.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:26.600215+0000) 2024-03-06T15:09:27.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:27.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:27.600524+0000) 2024-03-06T15:09:28.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:28.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:28.600812+0000) 2024-03-06T15:09:29.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:29.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:29.601122+0000) 2024-03-06T15:09:30.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:30.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:30.601435+0000) 2024-03-06T15:09:31.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:31.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:31.601783+0000) 2024-03-06T15:09:32.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:32.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:32.602034+0000) 2024-03-06T15:09:33.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:33.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:33.602345+0000) 2024-03-06T15:09:34.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:34.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:34.602677+0000) 2024-03-06T15:09:35.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:35.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:35.602921+0000) 2024-03-06T15:09:36.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:36.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:36.603226+0000) 2024-03-06T15:09:37.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:37.591 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:37.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:37.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:37.603493+0000) 2024-03-06T15:09:38.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:38.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:38.603845+0000) 2024-03-06T15:09:39.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:39.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:39.604145+0000) 2024-03-06T15:09:40.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:40.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:40.604385+0000) 2024-03-06T15:09:41.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:41.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:41.604757+0000) 2024-03-06T15:09:42.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:42.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:42.605072+0000) 2024-03-06T15:09:43.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:43.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:43.605384+0000) 2024-03-06T15:09:44.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:44.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:44.605727+0000) 2024-03-06T15:09:45.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:45.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:45.606026+0000) 2024-03-06T15:09:46.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:46.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:46.606326+0000) 2024-03-06T15:09:47.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:47.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:47.606622+0000) 2024-03-06T15:09:48.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:48.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:48.606969+0000) 2024-03-06T15:09:49.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:49.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:49.607266+0000) 2024-03-06T15:09:50.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:50.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:50.607572+0000) 2024-03-06T15:09:51.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:51.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:51.607953+0000) 2024-03-06T15:09:52.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:52.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:52.608265+0000) 2024-03-06T15:09:53.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:53.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:53.608514+0000) 2024-03-06T15:09:54.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:54.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:54.608825+0000) 2024-03-06T15:09:55.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:55.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:55.609064+0000) 2024-03-06T15:09:56.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:56.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:56.609363+0000) 2024-03-06T15:09:57.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:57.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:57.609688+0000) 2024-03-06T15:09:58.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:58.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:58.610007+0000) 2024-03-06T15:09:59.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:09:59.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:59.610330+0000) 2024-03-06T15:10:00.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:00.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:00.610670+0000) 2024-03-06T15:10:01.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:01.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:01.610927+0000) 2024-03-06T15:10:02.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:02.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:02.611248+0000) 2024-03-06T15:10:03.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:03.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:03.611565+0000) 2024-03-06T15:10:04.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:04.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:04.611861+0000) 2024-03-06T15:10:05.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:05.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:05.612163+0000) 2024-03-06T15:10:06.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:06.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:06.612476+0000) 2024-03-06T15:10:07.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:07.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:07.612794+0000) 2024-03-06T15:10:07.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:07.737 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:08.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:08.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:08.613117+0000) 2024-03-06T15:10:09.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:09.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:09.613446+0000) 2024-03-06T15:10:10.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:10.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:10.613776+0000) 2024-03-06T15:10:11.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:11.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:11.614084+0000) 2024-03-06T15:10:12.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:12.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:12.614389+0000) 2024-03-06T15:10:13.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:13.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:13.614743+0000) 2024-03-06T15:10:14.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:14.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:14.615059+0000) 2024-03-06T15:10:15.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:15.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:15.615375+0000) 2024-03-06T15:10:16.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:16.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:16.615717+0000) 2024-03-06T15:10:17.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:17.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:17.616035+0000) 2024-03-06T15:10:18.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:18.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:18.616337+0000) 2024-03-06T15:10:19.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:19.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:19.616689+0000) 2024-03-06T15:10:20.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:20.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:20.616983+0000) 2024-03-06T15:10:21.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:21.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:21.617272+0000) 2024-03-06T15:10:22.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:22.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:22.617521+0000) 2024-03-06T15:10:23.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:23.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:23.617783+0000) 2024-03-06T15:10:24.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:24.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:24.618043+0000) 2024-03-06T15:10:25.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:25.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:25.618341+0000) 2024-03-06T15:10:26.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:26.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:26.618761+0000) 2024-03-06T15:10:27.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:27.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:27.618983+0000) 2024-03-06T15:10:28.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:28.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:28.619281+0000) 2024-03-06T15:10:29.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:29.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:29.619596+0000) 2024-03-06T15:10:30.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:30.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:30.619971+0000) 2024-03-06T15:10:31.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:31.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:31.620311+0000) 2024-03-06T15:10:32.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:32.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:32.620679+0000) 2024-03-06T15:10:33.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:33.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:33.620991+0000) 2024-03-06T15:10:34.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:34.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:34.621310+0000) 2024-03-06T15:10:35.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:35.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:35.621594+0000) 2024-03-06T15:10:36.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:36.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:36.621924+0000) 2024-03-06T15:10:37.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:37.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:37.622246+0000) 2024-03-06T15:10:37.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:37.874 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:38.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:38.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:38.622567+0000) 2024-03-06T15:10:39.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:39.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:39.622882+0000) 2024-03-06T15:10:40.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:40.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:40.623189+0000) 2024-03-06T15:10:41.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:41.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:41.623529+0000) 2024-03-06T15:10:42.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:42.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:42.623823+0000) 2024-03-06T15:10:43.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:43.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:43.624145+0000) 2024-03-06T15:10:44.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:44.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:44.624445+0000) 2024-03-06T15:10:45.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:45.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:45.624777+0000) 2024-03-06T15:10:46.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:46.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:46.625091+0000) 2024-03-06T15:10:47.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:47.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:47.625379+0000) 2024-03-06T15:10:48.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:48.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:48.625742+0000) 2024-03-06T15:10:49.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:49.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:49.626021+0000) 2024-03-06T15:10:50.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:50.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:50.626335+0000) 2024-03-06T15:10:51.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:51.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:51.626723+0000) 2024-03-06T15:10:52.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:52.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:52.627049+0000) 2024-03-06T15:10:53.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:53.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:53.627349+0000) 2024-03-06T15:10:54.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:54.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:54.627586+0000) 2024-03-06T15:10:55.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:55.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:55.627869+0000) 2024-03-06T15:10:56.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:56.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:56.628175+0000) 2024-03-06T15:10:57.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:57.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:57.628488+0000) 2024-03-06T15:10:58.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:58.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:58.628802+0000) 2024-03-06T15:10:59.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:10:59.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:59.629090+0000) 2024-03-06T15:11:00.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:00.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:00.629398+0000) 2024-03-06T15:11:01.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:01.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:01.629757+0000) 2024-03-06T15:11:02.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:02.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:02.630058+0000) 2024-03-06T15:11:03.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:03.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:03.630379+0000) 2024-03-06T15:11:04.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:04.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:04.630716+0000) 2024-03-06T15:11:05.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:05.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:05.631034+0000) 2024-03-06T15:11:06.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:06.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:06.631348+0000) 2024-03-06T15:11:07.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:07.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:07.631697+0000) 2024-03-06T15:11:08.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:08.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:08.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:08.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:08.631992+0000) 2024-03-06T15:11:09.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:09.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:09.632277+0000) 2024-03-06T15:11:10.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:10.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:10.632572+0000) 2024-03-06T15:11:11.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:11.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:11.632915+0000) 2024-03-06T15:11:12.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:12.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:12.633214+0000) 2024-03-06T15:11:13.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:13.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:13.633529+0000) 2024-03-06T15:11:14.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:14.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:14.633831+0000) 2024-03-06T15:11:15.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:15.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:15.634082+0000) 2024-03-06T15:11:16.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:16.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:16.634409+0000) 2024-03-06T15:11:17.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:17.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:17.634739+0000) 2024-03-06T15:11:18.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:18.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:18.635045+0000) 2024-03-06T15:11:19.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:19.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:19.635394+0000) 2024-03-06T15:11:20.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:20.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:20.635731+0000) 2024-03-06T15:11:21.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:21.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:21.636025+0000) 2024-03-06T15:11:22.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:22.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:22.636268+0000) 2024-03-06T15:11:23.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:23.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:23.636504+0000) 2024-03-06T15:11:24.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:24.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:24.636758+0000) 2024-03-06T15:11:25.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:25.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:25.637025+0000) 2024-03-06T15:11:26.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:26.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:26.637313+0000) 2024-03-06T15:11:27.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:27.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:27.637577+0000) 2024-03-06T15:11:28.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:28.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:28.637840+0000) 2024-03-06T15:11:29.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:29.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:29.638079+0000) 2024-03-06T15:11:30.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:30.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:30.638357+0000) 2024-03-06T15:11:31.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:31.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:31.638677+0000) 2024-03-06T15:11:32.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:32.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:32.638956+0000) 2024-03-06T15:11:33.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:33.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:33.639207+0000) 2024-03-06T15:11:34.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:34.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:34.639446+0000) 2024-03-06T15:11:35.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:35.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:35.639729+0000) 2024-03-06T15:11:36.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:36.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:36.639979+0000) 2024-03-06T15:11:37.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:37.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:37.640241+0000) 2024-03-06T15:11:38.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:38.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:38.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:38.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:38.640493+0000) 2024-03-06T15:11:39.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:39.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:39.640735+0000) 2024-03-06T15:11:40.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:40.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:40.640986+0000) 2024-03-06T15:11:41.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:41.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:41.641248+0000) 2024-03-06T15:11:42.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:42.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:42.641512+0000) 2024-03-06T15:11:43.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:43.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:43.641775+0000) 2024-03-06T15:11:44.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:44.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:44.642024+0000) 2024-03-06T15:11:45.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:45.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:45.642290+0000) 2024-03-06T15:11:46.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:46.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:46.642562+0000) 2024-03-06T15:11:47.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:47.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:47.642843+0000) 2024-03-06T15:11:48.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:48.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:48.643050+0000) 2024-03-06T15:11:49.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:49.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:49.643297+0000) 2024-03-06T15:11:50.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:50.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:50.643548+0000) 2024-03-06T15:11:51.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:51.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:51.643830+0000) 2024-03-06T15:11:52.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:52.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:52.644022+0000) 2024-03-06T15:11:53.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:53.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:53.644285+0000) 2024-03-06T15:11:54.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:54.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:54.644551+0000) 2024-03-06T15:11:55.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:55.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:55.644793+0000) 2024-03-06T15:11:56.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:56.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:56.644997+0000) 2024-03-06T15:11:57.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:57.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:57.645254+0000) 2024-03-06T15:11:58.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:58.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:58.645520+0000) 2024-03-06T15:11:59.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:11:59.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:59.645777+0000) 2024-03-06T15:12:00.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:00.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:00.645983+0000) 2024-03-06T15:12:01.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:01.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:01.646255+0000) 2024-03-06T15:12:02.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:02.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:02.646526+0000) 2024-03-06T15:12:03.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:03.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:03.646775+0000) 2024-03-06T15:12:04.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:04.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:04.647018+0000) 2024-03-06T15:12:05.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:05.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:05.647267+0000) 2024-03-06T15:12:06.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:06.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:06.647534+0000) 2024-03-06T15:12:07.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:07.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:07.647802+0000) 2024-03-06T15:12:08.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:08.317 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:08.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:08.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:08.648075+0000) 2024-03-06T15:12:09.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:09.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:09.648339+0000) 2024-03-06T15:12:10.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:10.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:10.648606+0000) 2024-03-06T15:12:11.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:11.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:11.648919+0000) 2024-03-06T15:12:12.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:12.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:12.649160+0000) 2024-03-06T15:12:13.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:13.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:13.649414+0000) 2024-03-06T15:12:14.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:14.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:14.649724+0000) 2024-03-06T15:12:15.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:15.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:15.649966+0000) 2024-03-06T15:12:16.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:16.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:16.650214+0000) 2024-03-06T15:12:17.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:17.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:17.650467+0000) 2024-03-06T15:12:18.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:18.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:18.650778+0000) 2024-03-06T15:12:19.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:19.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:19.651003+0000) 2024-03-06T15:12:20.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:20.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:20.651270+0000) 2024-03-06T15:12:21.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:21.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:21.651500+0000) 2024-03-06T15:12:22.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:22.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:22.651773+0000) 2024-03-06T15:12:23.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:23.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:23.652065+0000) 2024-03-06T15:12:24.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:24.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:24.652325+0000) 2024-03-06T15:12:25.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:25.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:25.652587+0000) 2024-03-06T15:12:26.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:26.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:26.652805+0000) 2024-03-06T15:12:27.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:27.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:27.653033+0000) 2024-03-06T15:12:28.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:28.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:28.653310+0000) 2024-03-06T15:12:29.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:29.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:29.653585+0000) 2024-03-06T15:12:30.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:30.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:30.653901+0000) 2024-03-06T15:12:31.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:31.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:31.654140+0000) 2024-03-06T15:12:32.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:32.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:32.654404+0000) 2024-03-06T15:12:33.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:33.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:33.654719+0000) 2024-03-06T15:12:34.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:34.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:34.655006+0000) 2024-03-06T15:12:35.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:35.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:35.655189+0000) 2024-03-06T15:12:36.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:36.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:36.655447+0000) 2024-03-06T15:12:37.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:37.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:37.655748+0000) 2024-03-06T15:12:38.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:38.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:38.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:38.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:38.655998+0000) 2024-03-06T15:12:39.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:39.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:39.656255+0000) 2024-03-06T15:12:40.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:40.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:40.656456+0000) 2024-03-06T15:12:41.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:41.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:41.656764+0000) 2024-03-06T15:12:42.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:42.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:42.657028+0000) 2024-03-06T15:12:43.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:43.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:43.657278+0000) 2024-03-06T15:12:44.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:44.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:44.657526+0000) 2024-03-06T15:12:45.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:45.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:45.657791+0000) 2024-03-06T15:12:46.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:46.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:46.658022+0000) 2024-03-06T15:12:47.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:47.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:47.658269+0000) 2024-03-06T15:12:48.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:48.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:48.658464+0000) 2024-03-06T15:12:49.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:49.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:49.658758+0000) 2024-03-06T15:12:50.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:50.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:50.658993+0000) 2024-03-06T15:12:51.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:51.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:51.659239+0000) 2024-03-06T15:12:52.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:52.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:52.659479+0000) 2024-03-06T15:12:53.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:53.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:53.659764+0000) 2024-03-06T15:12:54.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:54.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:54.660015+0000) 2024-03-06T15:12:55.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:55.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:55.660314+0000) 2024-03-06T15:12:56.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:56.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:56.660583+0000) 2024-03-06T15:12:57.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:57.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:57.660843+0000) 2024-03-06T15:12:58.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:58.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:58.661106+0000) 2024-03-06T15:12:59.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:12:59.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:59.661355+0000) 2024-03-06T15:13:00.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:00.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:00.661591+0000) 2024-03-06T15:13:01.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:01.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:01.661874+0000) 2024-03-06T15:13:02.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:02.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:02.662125+0000) 2024-03-06T15:13:03.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:03.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:03.662382+0000) 2024-03-06T15:13:04.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:04.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:04.662567+0000) 2024-03-06T15:13:05.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:05.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:05.662827+0000) 2024-03-06T15:13:06.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:06.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:06.663088+0000) 2024-03-06T15:13:07.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:07.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:07.663344+0000) 2024-03-06T15:13:08.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:08.601 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:08.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:08.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:08.663593+0000) 2024-03-06T15:13:09.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:09.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:09.663850+0000) 2024-03-06T15:13:10.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:10.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:10.664088+0000) 2024-03-06T15:13:11.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:11.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:11.664332+0000) 2024-03-06T15:13:12.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:12.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:12.664587+0000) 2024-03-06T15:13:13.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:13.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:13.664854+0000) 2024-03-06T15:13:14.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:14.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:14.665115+0000) 2024-03-06T15:13:15.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:15.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:15.665347+0000) 2024-03-06T15:13:16.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:16.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:16.665706+0000) 2024-03-06T15:13:17.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:17.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:17.666057+0000) 2024-03-06T15:13:18.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:18.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:18.666319+0000) 2024-03-06T15:13:19.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:19.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:19.666551+0000) 2024-03-06T15:13:20.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:20.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:20.666807+0000) 2024-03-06T15:13:21.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:21.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:21.667055+0000) 2024-03-06T15:13:22.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:22.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:22.667318+0000) 2024-03-06T15:13:23.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:23.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:23.667561+0000) 2024-03-06T15:13:24.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:24.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:24.667813+0000) 2024-03-06T15:13:25.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:25.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:25.668062+0000) 2024-03-06T15:13:26.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:26.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:26.668295+0000) 2024-03-06T15:13:27.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:27.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:27.668554+0000) 2024-03-06T15:13:28.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:28.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:28.668821+0000) 2024-03-06T15:13:29.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:29.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:29.669055+0000) 2024-03-06T15:13:30.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:30.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:30.669323+0000) 2024-03-06T15:13:31.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:31.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:31.669597+0000) 2024-03-06T15:13:32.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:32.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:32.669861+0000) 2024-03-06T15:13:33.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:33.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:33.670099+0000) 2024-03-06T15:13:34.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:34.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:34.670283+0000) 2024-03-06T15:13:35.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:35.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:35.670530+0000) 2024-03-06T15:13:36.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:36.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:36.670777+0000) 2024-03-06T15:13:37.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:37.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:37.671046+0000) 2024-03-06T15:13:38.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:38.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:38.671309+0000) 2024-03-06T15:13:38.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:38.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:39.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:39.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:39.671561+0000) 2024-03-06T15:13:40.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:40.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:40.671812+0000) 2024-03-06T15:13:41.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:41.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:41.672074+0000) 2024-03-06T15:13:42.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:42.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:42.672342+0000) 2024-03-06T15:13:43.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:43.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:43.672615+0000) 2024-03-06T15:13:44.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:44.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:44.672903+0000) 2024-03-06T15:13:45.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:45.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:45.673164+0000) 2024-03-06T15:13:46.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:46.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:46.673352+0000) 2024-03-06T15:13:47.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:47.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:47.673551+0000) 2024-03-06T15:13:48.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:48.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:48.673870+0000) 2024-03-06T15:13:49.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:49.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:49.674127+0000) 2024-03-06T15:13:50.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:50.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:50.674390+0000) 2024-03-06T15:13:51.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:51.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:51.674698+0000) 2024-03-06T15:13:52.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:52.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:52.674956+0000) 2024-03-06T15:13:53.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:53.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:53.675224+0000) 2024-03-06T15:13:54.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:54.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:54.675492+0000) 2024-03-06T15:13:55.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:55.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:55.675732+0000) 2024-03-06T15:13:56.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:56.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:56.675996+0000) 2024-03-06T15:13:57.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:57.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:57.676242+0000) 2024-03-06T15:13:58.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:58.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:58.676489+0000) 2024-03-06T15:13:59.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:13:59.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:59.676734+0000) 2024-03-06T15:14:00.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:00.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:00.676985+0000) 2024-03-06T15:14:01.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:01.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:01.677265+0000) 2024-03-06T15:14:02.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:02.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:02.677497+0000) 2024-03-06T15:14:03.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:03.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:03.677761+0000) 2024-03-06T15:14:04.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:04.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:04.678030+0000) 2024-03-06T15:14:05.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:05.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:05.678278+0000) 2024-03-06T15:14:06.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:06.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:06.678535+0000) 2024-03-06T15:14:07.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:07.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:07.678800+0000) 2024-03-06T15:14:08.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:08.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:08.679066+0000) 2024-03-06T15:14:08.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:08.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:09.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:09.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:09.679327+0000) 2024-03-06T15:14:10.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:10.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:10.679591+0000) 2024-03-06T15:14:11.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:11.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:11.679887+0000) 2024-03-06T15:14:12.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:12.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:12.680138+0000) 2024-03-06T15:14:13.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:13.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:13.680388+0000) 2024-03-06T15:14:14.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:14.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:14.680674+0000) 2024-03-06T15:14:15.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:15.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:15.680936+0000) 2024-03-06T15:14:16.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:16.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:16.681201+0000) 2024-03-06T15:14:17.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:17.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:17.681451+0000) 2024-03-06T15:14:18.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:18.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:18.681743+0000) 2024-03-06T15:14:19.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:19.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:19.681998+0000) 2024-03-06T15:14:20.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:20.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:20.682266+0000) 2024-03-06T15:14:21.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:21.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:21.682533+0000) 2024-03-06T15:14:22.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:22.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:22.682799+0000) 2024-03-06T15:14:23.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:23.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:23.683054+0000) 2024-03-06T15:14:24.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:24.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:24.683327+0000) 2024-03-06T15:14:25.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:25.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:25.683596+0000) 2024-03-06T15:14:26.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:26.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:26.683865+0000) 2024-03-06T15:14:27.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:27.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:27.684116+0000) 2024-03-06T15:14:28.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:28.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:28.684370+0000) 2024-03-06T15:14:29.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:29.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:29.684618+0000) 2024-03-06T15:14:30.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:30.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:30.684906+0000) 2024-03-06T15:14:31.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:31.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:31.685117+0000) 2024-03-06T15:14:32.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:32.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:32.685364+0000) 2024-03-06T15:14:33.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:33.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:33.685662+0000) 2024-03-06T15:14:34.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:34.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:34.685931+0000) 2024-03-06T15:14:35.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:35.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:35.686196+0000) 2024-03-06T15:14:36.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:36.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:36.686494+0000) 2024-03-06T15:14:37.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:37.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:37.686762+0000) 2024-03-06T15:14:38.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:38.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:38.687025+0000) 2024-03-06T15:14:39.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:39.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:39.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:39.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:39.687295+0000) 2024-03-06T15:14:40.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:40.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:40.687580+0000) 2024-03-06T15:14:41.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:41.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:41.687893+0000) 2024-03-06T15:14:42.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:42.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:42.688166+0000) 2024-03-06T15:14:43.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:43.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:43.688427+0000) 2024-03-06T15:14:44.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:44.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:44.688747+0000) 2024-03-06T15:14:45.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:45.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:45.688940+0000) 2024-03-06T15:14:46.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:46.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:46.689130+0000) 2024-03-06T15:14:47.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:47.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:47.689387+0000) 2024-03-06T15:14:48.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:48.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:48.689703+0000) 2024-03-06T15:14:49.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:49.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:49.689974+0000) 2024-03-06T15:14:50.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:50.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:50.690229+0000) 2024-03-06T15:14:51.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:51.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:51.690471+0000) 2024-03-06T15:14:52.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:52.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:52.690761+0000) 2024-03-06T15:14:53.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:53.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:53.690993+0000) 2024-03-06T15:14:54.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:54.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:54.691251+0000) 2024-03-06T15:14:55.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:55.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:55.691482+0000) 2024-03-06T15:14:56.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:56.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:56.691759+0000) 2024-03-06T15:14:57.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:57.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:57.692013+0000) 2024-03-06T15:14:58.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:58.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:58.692260+0000) 2024-03-06T15:14:59.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:14:59.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:59.692514+0000) 2024-03-06T15:15:00.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:00.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:00.692762+0000) 2024-03-06T15:15:01.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:01.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:01.693003+0000) 2024-03-06T15:15:02.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:02.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:02.693200+0000) 2024-03-06T15:15:03.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:03.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:03.693439+0000) 2024-03-06T15:15:04.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:04.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:04.693740+0000) 2024-03-06T15:15:05.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:05.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:05.693995+0000) 2024-03-06T15:15:06.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:06.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:06.694252+0000) 2024-03-06T15:15:07.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:07.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:07.694497+0000) 2024-03-06T15:15:08.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:08.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:08.694739+0000) 2024-03-06T15:15:09.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:09.211 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:09.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:09.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:09.694940+0000) 2024-03-06T15:15:10.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:10.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:10.695199+0000) 2024-03-06T15:15:11.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:11.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:11.695444+0000) 2024-03-06T15:15:12.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:12.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:12.695747+0000) 2024-03-06T15:15:13.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:13.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:13.696006+0000) 2024-03-06T15:15:14.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:14.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:14.696279+0000) 2024-03-06T15:15:15.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:15.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:15.696545+0000) 2024-03-06T15:15:16.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:16.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:16.696759+0000) 2024-03-06T15:15:17.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:17.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:17.697041+0000) 2024-03-06T15:15:18.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:18.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:18.697321+0000) 2024-03-06T15:15:19.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:19.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:19.697576+0000) 2024-03-06T15:15:20.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:20.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:20.697860+0000) 2024-03-06T15:15:21.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:21.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:21.698111+0000) 2024-03-06T15:15:22.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:22.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:22.698361+0000) 2024-03-06T15:15:23.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:23.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:23.698587+0000) 2024-03-06T15:15:24.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:24.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:24.698885+0000) 2024-03-06T15:15:25.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:25.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:25.699117+0000) 2024-03-06T15:15:26.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:26.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:26.699357+0000) 2024-03-06T15:15:27.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:27.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:27.699612+0000) 2024-03-06T15:15:28.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:28.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:28.699904+0000) 2024-03-06T15:15:29.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:29.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:29.700104+0000) 2024-03-06T15:15:30.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:30.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:30.700360+0000) 2024-03-06T15:15:31.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:31.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:31.700621+0000) 2024-03-06T15:15:32.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:32.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:32.700930+0000) 2024-03-06T15:15:33.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:33.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:33.701167+0000) 2024-03-06T15:15:34.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:34.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:34.701425+0000) 2024-03-06T15:15:35.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:35.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:35.701724+0000) 2024-03-06T15:15:36.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:36.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:36.701982+0000) 2024-03-06T15:15:37.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:37.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:37.702256+0000) 2024-03-06T15:15:38.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:38.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:38.702509+0000) 2024-03-06T15:15:39.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:39.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:39.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:39.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:39.702775+0000) 2024-03-06T15:15:40.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:40.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:40.703039+0000) 2024-03-06T15:15:41.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:41.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:41.703230+0000) 2024-03-06T15:15:42.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:42.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:42.703499+0000) 2024-03-06T15:15:43.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:43.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:43.703734+0000) 2024-03-06T15:15:44.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:44.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:44.703989+0000) 2024-03-06T15:15:45.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:45.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:45.704256+0000) 2024-03-06T15:15:46.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:46.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:46.704521+0000) 2024-03-06T15:15:47.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:47.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:47.704783+0000) 2024-03-06T15:15:48.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:48.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:48.705028+0000) 2024-03-06T15:15:49.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:49.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:49.705281+0000) 2024-03-06T15:15:50.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:50.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:50.705546+0000) 2024-03-06T15:15:51.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:51.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:51.705786+0000) 2024-03-06T15:15:52.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:52.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:52.706041+0000) 2024-03-06T15:15:53.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:53.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:53.706283+0000) 2024-03-06T15:15:54.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:54.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:54.706527+0000) 2024-03-06T15:15:55.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:55.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:55.706778+0000) 2024-03-06T15:15:56.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:56.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:56.707036+0000) 2024-03-06T15:15:57.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:57.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:57.707308+0000) 2024-03-06T15:15:58.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:58.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:58.707516+0000) 2024-03-06T15:15:59.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:15:59.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:59.707775+0000) 2024-03-06T15:16:00.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:00.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:00.708019+0000) 2024-03-06T15:16:01.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:01.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:01.708292+0000) 2024-03-06T15:16:02.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:02.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:02.708547+0000) 2024-03-06T15:16:03.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:03.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:03.708765+0000) 2024-03-06T15:16:04.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:04.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:04.709009+0000) 2024-03-06T15:16:05.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:05.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:05.709243+0000) 2024-03-06T15:16:06.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:06.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:06.709485+0000) 2024-03-06T15:16:07.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:07.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:07.709749+0000) 2024-03-06T15:16:08.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:08.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:08.709987+0000) 2024-03-06T15:16:09.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:09.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:09.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:09.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:09.710258+0000) 2024-03-06T15:16:10.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:10.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:10.710501+0000) 2024-03-06T15:16:11.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:11.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:11.710775+0000) 2024-03-06T15:16:12.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:12.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:12.711030+0000) 2024-03-06T15:16:13.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:13.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:13.711224+0000) 2024-03-06T15:16:14.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:14.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:14.711490+0000) 2024-03-06T15:16:15.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:15.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:15.711742+0000) 2024-03-06T15:16:16.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:16.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:16.711996+0000) 2024-03-06T15:16:17.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:17.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:17.712252+0000) 2024-03-06T15:16:18.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:18.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:18.712549+0000) 2024-03-06T15:16:19.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:19.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:19.712805+0000) 2024-03-06T15:16:20.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:20.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:20.713061+0000) 2024-03-06T15:16:21.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:21.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:21.713328+0000) 2024-03-06T15:16:22.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:22.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:22.713589+0000) 2024-03-06T15:16:23.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:23.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:23.713826+0000) 2024-03-06T15:16:24.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:24.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:24.714090+0000) 2024-03-06T15:16:25.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:25.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:25.714333+0000) 2024-03-06T15:16:26.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:26.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:26.714570+0000) 2024-03-06T15:16:27.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:27.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:27.714822+0000) 2024-03-06T15:16:28.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:28.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:28.715080+0000) 2024-03-06T15:16:29.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:29.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:29.715337+0000) 2024-03-06T15:16:30.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:30.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:30.715596+0000) 2024-03-06T15:16:31.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:31.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:31.715897+0000) 2024-03-06T15:16:32.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:32.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:32.716157+0000) 2024-03-06T15:16:33.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:33.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:33.716418+0000) 2024-03-06T15:16:34.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:34.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:34.716726+0000) 2024-03-06T15:16:35.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:35.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:35.716991+0000) 2024-03-06T15:16:36.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:36.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:36.717248+0000) 2024-03-06T15:16:37.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:37.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:37.717478+0000) 2024-03-06T15:16:38.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:38.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:38.717749+0000) 2024-03-06T15:16:39.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:39.658 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:39.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:39.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:39.717926+0000) 2024-03-06T15:16:40.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:40.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:40.718159+0000) 2024-03-06T15:16:41.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:41.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:41.718403+0000) 2024-03-06T15:16:42.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:42.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:42.718684+0000) 2024-03-06T15:16:43.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:43.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:43.718943+0000) 2024-03-06T15:16:44.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:44.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:44.719203+0000) 2024-03-06T15:16:45.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:45.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:45.719455+0000) 2024-03-06T15:16:46.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:46.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:46.719753+0000) 2024-03-06T15:16:47.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:47.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:47.719987+0000) 2024-03-06T15:16:48.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:48.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:48.720228+0000) 2024-03-06T15:16:49.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:49.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:49.720471+0000) 2024-03-06T15:16:50.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:50.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:50.720736+0000) 2024-03-06T15:16:51.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:51.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:51.720998+0000) 2024-03-06T15:16:52.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:52.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:52.721242+0000) 2024-03-06T15:16:53.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:53.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:53.721478+0000) 2024-03-06T15:16:54.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:54.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:54.721753+0000) 2024-03-06T15:16:55.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:55.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:55.722004+0000) 2024-03-06T15:16:56.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:56.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:56.722251+0000) 2024-03-06T15:16:57.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:57.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:57.722453+0000) 2024-03-06T15:16:58.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:58.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:58.722735+0000) 2024-03-06T15:16:59.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:16:59.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:59.722976+0000) 2024-03-06T15:17:00.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:00.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:00.723233+0000) 2024-03-06T15:17:01.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:01.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:01.723497+0000) 2024-03-06T15:17:02.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:02.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:02.723755+0000) 2024-03-06T15:17:03.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:03.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:03.724005+0000) 2024-03-06T15:17:04.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:04.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:04.724271+0000) 2024-03-06T15:17:05.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:05.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:05.724525+0000) 2024-03-06T15:17:06.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:06.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:06.724820+0000) 2024-03-06T15:17:07.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:07.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:07.725073+0000) 2024-03-06T15:17:08.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:08.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:08.725313+0000) 2024-03-06T15:17:09.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:09.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:09.725573+0000) 2024-03-06T15:17:09.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:09.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:10.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:10.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:10.725834+0000) 2024-03-06T15:17:11.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:11.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:11.726081+0000) 2024-03-06T15:17:12.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:12.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:12.726348+0000) 2024-03-06T15:17:13.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:13.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:13.726615+0000) 2024-03-06T15:17:14.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:14.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:14.726922+0000) 2024-03-06T15:17:15.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:15.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:15.727154+0000) 2024-03-06T15:17:16.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:16.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:16.727396+0000) 2024-03-06T15:17:17.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:17.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:17.727687+0000) 2024-03-06T15:17:18.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:18.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:18.727922+0000) 2024-03-06T15:17:19.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:19.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:19.728187+0000) 2024-03-06T15:17:20.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:20.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:20.728446+0000) 2024-03-06T15:17:21.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:21.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:21.728733+0000) 2024-03-06T15:17:22.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:22.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:22.728980+0000) 2024-03-06T15:17:23.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:23.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:23.729222+0000) 2024-03-06T15:17:24.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:24.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:24.729468+0000) 2024-03-06T15:17:25.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:25.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:25.729745+0000) 2024-03-06T15:17:26.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:26.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:26.729993+0000) 2024-03-06T15:17:27.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:27.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:27.730249+0000) 2024-03-06T15:17:28.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:28.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:28.730484+0000) 2024-03-06T15:17:29.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:29.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:29.730749+0000) 2024-03-06T15:17:30.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:30.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:30.730941+0000) 2024-03-06T15:17:31.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:31.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:31.731193+0000) 2024-03-06T15:17:32.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:32.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:32.731381+0000) 2024-03-06T15:17:33.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:33.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:33.731618+0000) 2024-03-06T15:17:34.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:34.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:34.731919+0000) 2024-03-06T15:17:35.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:35.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:35.732168+0000) 2024-03-06T15:17:36.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:36.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:36.732421+0000) 2024-03-06T15:17:37.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:37.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:37.732702+0000) 2024-03-06T15:17:38.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:38.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:38.732963+0000) 2024-03-06T15:17:39.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:39.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:39.733223+0000) 2024-03-06T15:17:39.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:39.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:40.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:40.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:40.733486+0000) 2024-03-06T15:17:41.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:41.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:41.733750+0000) 2024-03-06T15:17:42.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:42.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:42.734012+0000) 2024-03-06T15:17:43.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:43.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:43.734275+0000) 2024-03-06T15:17:44.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:44.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:44.734524+0000) 2024-03-06T15:17:45.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:45.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:45.734787+0000) 2024-03-06T15:17:46.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:46.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:46.735025+0000) 2024-03-06T15:17:47.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:47.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:47.735287+0000) 2024-03-06T15:17:48.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:48.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:48.735538+0000) 2024-03-06T15:17:49.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:49.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:49.735814+0000) 2024-03-06T15:17:50.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:50.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:50.736055+0000) 2024-03-06T15:17:51.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:51.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:51.736305+0000) 2024-03-06T15:17:52.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:52.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:52.736564+0000) 2024-03-06T15:17:53.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:53.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:53.736829+0000) 2024-03-06T15:17:54.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:54.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:54.737148+0000) 2024-03-06T15:17:55.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:55.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:55.737336+0000) 2024-03-06T15:17:56.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:56.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:56.737594+0000) 2024-03-06T15:17:57.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:57.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:57.737860+0000) 2024-03-06T15:17:58.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:58.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:58.738122+0000) 2024-03-06T15:17:59.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:17:59.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:59.738390+0000) 2024-03-06T15:18:00.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:00.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:00.738697+0000) 2024-03-06T15:18:01.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:01.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:01.738906+0000) 2024-03-06T15:18:02.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:02.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:02.739174+0000) 2024-03-06T15:18:03.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:03.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:03.739441+0000) 2024-03-06T15:18:04.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:04.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:04.739748+0000) 2024-03-06T15:18:05.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:05.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:05.739987+0000) 2024-03-06T15:18:06.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:06.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:06.740253+0000) 2024-03-06T15:18:07.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:07.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:07.740518+0000) 2024-03-06T15:18:08.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:08.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:08.740770+0000) 2024-03-06T15:18:09.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:09.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:09.741022+0000) 2024-03-06T15:18:10.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:10.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:10.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:10.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:10.741292+0000) 2024-03-06T15:18:11.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:11.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:11.741545+0000) 2024-03-06T15:18:12.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:12.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:12.741807+0000) 2024-03-06T15:18:13.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:13.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:13.742071+0000) 2024-03-06T15:18:14.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:14.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:14.742349+0000) 2024-03-06T15:18:15.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:15.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:15.742612+0000) 2024-03-06T15:18:16.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:16.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:16.742902+0000) 2024-03-06T15:18:17.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:17.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:17.743136+0000) 2024-03-06T15:18:18.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:18.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:18.743387+0000) 2024-03-06T15:18:19.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:19.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:19.743684+0000) 2024-03-06T15:18:20.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:20.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:20.743944+0000) 2024-03-06T15:18:21.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:21.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:21.744220+0000) 2024-03-06T15:18:22.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:22.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:22.744479+0000) 2024-03-06T15:18:23.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:23.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:23.744754+0000) 2024-03-06T15:18:24.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:24.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:24.744959+0000) 2024-03-06T15:18:25.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:25.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:25.745205+0000) 2024-03-06T15:18:26.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:26.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:26.745454+0000) 2024-03-06T15:18:27.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:27.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:27.745750+0000) 2024-03-06T15:18:28.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:28.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:28.746000+0000) 2024-03-06T15:18:29.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:29.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:29.746248+0000) 2024-03-06T15:18:30.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:30.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:30.746507+0000) 2024-03-06T15:18:31.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:31.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:31.746764+0000) 2024-03-06T15:18:32.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:32.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:32.747042+0000) 2024-03-06T15:18:33.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:33.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:33.747280+0000) 2024-03-06T15:18:34.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:34.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:34.747544+0000) 2024-03-06T15:18:35.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:35.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:35.747796+0000) 2024-03-06T15:18:36.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:36.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:36.748001+0000) 2024-03-06T15:18:37.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:37.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:37.748257+0000) 2024-03-06T15:18:38.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:38.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:38.748550+0000) 2024-03-06T15:18:39.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:39.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:39.748812+0000) 2024-03-06T15:18:40.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:40.272 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:40.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:40.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:40.749064+0000) 2024-03-06T15:18:41.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:41.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:41.749349+0000) 2024-03-06T15:18:42.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:42.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:42.749573+0000) 2024-03-06T15:18:43.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:43.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:43.749800+0000) 2024-03-06T15:18:44.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:44.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:44.750091+0000) 2024-03-06T15:18:45.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:45.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:45.750351+0000) 2024-03-06T15:18:46.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:46.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:46.750714+0000) 2024-03-06T15:18:47.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:47.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:47.751029+0000) 2024-03-06T15:18:48.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:48.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:48.751325+0000) 2024-03-06T15:18:49.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:49.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:49.751675+0000) 2024-03-06T15:18:50.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:50.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:50.751970+0000) 2024-03-06T15:18:51.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:51.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:51.752288+0000) 2024-03-06T15:18:52.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:52.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:52.752571+0000) 2024-03-06T15:18:53.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:53.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:53.752871+0000) 2024-03-06T15:18:54.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:54.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:54.753165+0000) 2024-03-06T15:18:55.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:55.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:55.753451+0000) 2024-03-06T15:18:56.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:56.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:56.753797+0000) 2024-03-06T15:18:57.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:57.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:57.754089+0000) 2024-03-06T15:18:58.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:58.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:58.754411+0000) 2024-03-06T15:18:59.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:18:59.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:59.754747+0000) 2024-03-06T15:19:00.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:00.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:00.755055+0000) 2024-03-06T15:19:01.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:01.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:01.755313+0000) 2024-03-06T15:19:02.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:02.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:02.755565+0000) 2024-03-06T15:19:03.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:03.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:03.755817+0000) 2024-03-06T15:19:04.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:04.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:04.756044+0000) 2024-03-06T15:19:05.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:05.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:05.756300+0000) 2024-03-06T15:19:06.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:06.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:06.756547+0000) 2024-03-06T15:19:07.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:07.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:07.756794+0000) 2024-03-06T15:19:08.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:08.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:08.757042+0000) 2024-03-06T15:19:09.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:09.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:09.757280+0000) 2024-03-06T15:19:10.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:10.444 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:10.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:10.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:10.757550+0000) 2024-03-06T15:19:11.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:11.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:11.757812+0000) 2024-03-06T15:19:12.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:12.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:12.758065+0000) 2024-03-06T15:19:13.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:13.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:13.758309+0000) 2024-03-06T15:19:14.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:14.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:14.758547+0000) 2024-03-06T15:19:15.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:15.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:15.758800+0000) 2024-03-06T15:19:16.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:16.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:16.759049+0000) 2024-03-06T15:19:17.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:17.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:17.759301+0000) 2024-03-06T15:19:18.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:18.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:18.759566+0000) 2024-03-06T15:19:19.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:19.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:19.759821+0000) 2024-03-06T15:19:20.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:20.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:20.760050+0000) 2024-03-06T15:19:21.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:21.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:21.760322+0000) 2024-03-06T15:19:22.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:22.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:22.760574+0000) 2024-03-06T15:19:23.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:23.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:23.760825+0000) 2024-03-06T15:19:24.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:24.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:24.761132+0000) 2024-03-06T15:19:25.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:25.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:25.761444+0000) 2024-03-06T15:19:26.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:26.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:26.761777+0000) 2024-03-06T15:19:27.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:27.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:27.762094+0000) 2024-03-06T15:19:28.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:28.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:28.762404+0000) 2024-03-06T15:19:29.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:29.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:29.762745+0000) 2024-03-06T15:19:30.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:30.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:30.763057+0000) 2024-03-06T15:19:31.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:31.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:31.763373+0000) 2024-03-06T15:19:32.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:32.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:32.763730+0000) 2024-03-06T15:19:33.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:33.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:33.764041+0000) 2024-03-06T15:19:34.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:34.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:34.764334+0000) 2024-03-06T15:19:35.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:35.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:35.764721+0000) 2024-03-06T15:19:36.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:36.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:36.765024+0000) 2024-03-06T15:19:37.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:37.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:37.765262+0000) 2024-03-06T15:19:38.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:38.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:38.765528+0000) 2024-03-06T15:19:39.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:39.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:39.765777+0000) 2024-03-06T15:19:40.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:40.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:40.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:40.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:40.766027+0000) 2024-03-06T15:19:41.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:41.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:41.766278+0000) 2024-03-06T15:19:42.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:42.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:42.766523+0000) 2024-03-06T15:19:43.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:43.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:43.766791+0000) 2024-03-06T15:19:44.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:44.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:44.767051+0000) 2024-03-06T15:19:45.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:45.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:45.767293+0000) 2024-03-06T15:19:46.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:46.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:46.767536+0000) 2024-03-06T15:19:47.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:47.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:47.767785+0000) 2024-03-06T15:19:48.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:48.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:48.768032+0000) 2024-03-06T15:19:49.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:49.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:49.768282+0000) 2024-03-06T15:19:50.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:50.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:50.768553+0000) 2024-03-06T15:19:51.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:51.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:51.768756+0000) 2024-03-06T15:19:52.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:52.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:52.769012+0000) 2024-03-06T15:19:53.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:53.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:53.769280+0000) 2024-03-06T15:19:54.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:54.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:54.769521+0000) 2024-03-06T15:19:55.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:55.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:55.769781+0000) 2024-03-06T15:19:56.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:56.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:56.770028+0000) 2024-03-06T15:19:57.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:57.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:57.770283+0000) 2024-03-06T15:19:58.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:58.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:58.770547+0000) 2024-03-06T15:19:59.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:19:59.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:59.770799+0000) 2024-03-06T15:20:00.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:00.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:00.771066+0000) 2024-03-06T15:20:01.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:01.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:01.771284+0000) 2024-03-06T15:20:02.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:02.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:02.771552+0000) 2024-03-06T15:20:03.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:03.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:03.771819+0000) 2024-03-06T15:20:04.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:04.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:04.772079+0000) 2024-03-06T15:20:05.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:05.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:05.772267+0000) 2024-03-06T15:20:06.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:06.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:06.772508+0000) 2024-03-06T15:20:07.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:07.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:07.772750+0000) 2024-03-06T15:20:08.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:08.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:08.773014+0000) 2024-03-06T15:20:09.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:09.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:09.773239+0000) 2024-03-06T15:20:10.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:10.756 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:10.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:10.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:10.773492+0000) 2024-03-06T15:20:11.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:11.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:11.773761+0000) 2024-03-06T15:20:12.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:12.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:12.774022+0000) 2024-03-06T15:20:13.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:13.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:13.774290+0000) 2024-03-06T15:20:14.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:14.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:14.774531+0000) 2024-03-06T15:20:15.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:15.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:15.774800+0000) 2024-03-06T15:20:16.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:16.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:16.775046+0000) 2024-03-06T15:20:17.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:17.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:17.775303+0000) 2024-03-06T15:20:18.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:18.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:18.775585+0000) 2024-03-06T15:20:19.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:19.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:19.775828+0000) 2024-03-06T15:20:20.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:20.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:20.776066+0000) 2024-03-06T15:20:21.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:21.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:21.776331+0000) 2024-03-06T15:20:22.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:22.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:22.776572+0000) 2024-03-06T15:20:23.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:23.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:23.776872+0000) 2024-03-06T15:20:24.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:24.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:24.777127+0000) 2024-03-06T15:20:25.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:25.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:25.777375+0000) 2024-03-06T15:20:26.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:26.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:26.777667+0000) 2024-03-06T15:20:27.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:27.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:27.777926+0000) 2024-03-06T15:20:28.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:28.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:28.778194+0000) 2024-03-06T15:20:29.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:29.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:29.778451+0000) 2024-03-06T15:20:30.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:30.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:30.778771+0000) 2024-03-06T15:20:31.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:31.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:31.779017+0000) 2024-03-06T15:20:32.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:32.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:32.779255+0000) 2024-03-06T15:20:33.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:33.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:33.779523+0000) 2024-03-06T15:20:34.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:34.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:34.779780+0000) 2024-03-06T15:20:35.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:35.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:35.780032+0000) 2024-03-06T15:20:36.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:36.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:36.780267+0000) 2024-03-06T15:20:37.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:37.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:37.780514+0000) 2024-03-06T15:20:38.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:38.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:38.780761+0000) 2024-03-06T15:20:39.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:39.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:39.780996+0000) 2024-03-06T15:20:40.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:40.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:40.781242+0000) 2024-03-06T15:20:40.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:40.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:41.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:41.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:41.781496+0000) 2024-03-06T15:20:42.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:42.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:42.781737+0000) 2024-03-06T15:20:43.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:43.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:43.782007+0000) 2024-03-06T15:20:44.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:44.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:44.782243+0000) 2024-03-06T15:20:45.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:45.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:45.782496+0000) 2024-03-06T15:20:46.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:46.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:46.782750+0000) 2024-03-06T15:20:47.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:47.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:47.782997+0000) 2024-03-06T15:20:48.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:48.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:48.783232+0000) 2024-03-06T15:20:49.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:49.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:49.783486+0000) 2024-03-06T15:20:50.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:50.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:50.783741+0000) 2024-03-06T15:20:51.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:51.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:51.784005+0000) 2024-03-06T15:20:52.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:52.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:52.784250+0000) 2024-03-06T15:20:53.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:53.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:53.784482+0000) 2024-03-06T15:20:54.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:54.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:54.784738+0000) 2024-03-06T15:20:55.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:55.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:55.784977+0000) 2024-03-06T15:20:56.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:56.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:56.785238+0000) 2024-03-06T15:20:57.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:57.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:57.785498+0000) 2024-03-06T15:20:58.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:58.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:58.785737+0000) 2024-03-06T15:20:59.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:20:59.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:59.785989+0000) 2024-03-06T15:21:00.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:00.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:00.786235+0000) 2024-03-06T15:21:01.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:01.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:01.786499+0000) 2024-03-06T15:21:02.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:02.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:02.786743+0000) 2024-03-06T15:21:03.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:03.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:03.787031+0000) 2024-03-06T15:21:04.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:04.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:04.787290+0000) 2024-03-06T15:21:05.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:05.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:05.787540+0000) 2024-03-06T15:21:06.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:06.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:06.787796+0000) 2024-03-06T15:21:07.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:07.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:07.788048+0000) 2024-03-06T15:21:08.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:08.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:08.788291+0000) 2024-03-06T15:21:09.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:09.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:09.788546+0000) 2024-03-06T15:21:10.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:10.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:10.788787+0000) 2024-03-06T15:21:11.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:11.047 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:11.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:11.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:11.789053+0000) 2024-03-06T15:21:12.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:12.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:12.789289+0000) 2024-03-06T15:21:13.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:13.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:13.789526+0000) 2024-03-06T15:21:14.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:14.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:14.789780+0000) 2024-03-06T15:21:15.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:15.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:15.790039+0000) 2024-03-06T15:21:16.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:16.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:16.790355+0000) 2024-03-06T15:21:17.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:17.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:17.790588+0000) 2024-03-06T15:21:18.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:18.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:18.790895+0000) 2024-03-06T15:21:19.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:19.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:19.791125+0000) 2024-03-06T15:21:20.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:20.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:20.791374+0000) 2024-03-06T15:21:21.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:21.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:21.791675+0000) 2024-03-06T15:21:22.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:22.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:22.791936+0000) 2024-03-06T15:21:23.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:23.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:23.792185+0000) 2024-03-06T15:21:24.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:24.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:24.792435+0000) 2024-03-06T15:21:25.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:25.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:25.792742+0000) 2024-03-06T15:21:26.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:26.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:26.792995+0000) 2024-03-06T15:21:27.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:27.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:27.793235+0000) 2024-03-06T15:21:28.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:28.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:28.793475+0000) 2024-03-06T15:21:29.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:29.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:29.793766+0000) 2024-03-06T15:21:30.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:30.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:30.794032+0000) 2024-03-06T15:21:31.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:31.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:31.794237+0000) 2024-03-06T15:21:32.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:32.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:32.794499+0000) 2024-03-06T15:21:33.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:33.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:33.794753+0000) 2024-03-06T15:21:34.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:34.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:34.795012+0000) 2024-03-06T15:21:35.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:35.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:35.795279+0000) 2024-03-06T15:21:36.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:36.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:36.795526+0000) 2024-03-06T15:21:37.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:37.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:37.795767+0000) 2024-03-06T15:21:38.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:38.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:38.795969+0000) 2024-03-06T15:21:39.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:39.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:39.796212+0000) 2024-03-06T15:21:40.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:40.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:40.796476+0000) 2024-03-06T15:21:41.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:41.187 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:41.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:41.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:41.796750+0000) 2024-03-06T15:21:42.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:42.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:42.797012+0000) 2024-03-06T15:21:43.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:43.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:43.797254+0000) 2024-03-06T15:21:44.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:44.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:44.797449+0000) 2024-03-06T15:21:45.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:45.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:45.797743+0000) 2024-03-06T15:21:46.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:46.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:46.798010+0000) 2024-03-06T15:21:47.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:47.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:47.798279+0000) 2024-03-06T15:21:48.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:48.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:48.798562+0000) 2024-03-06T15:21:49.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:49.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:49.798828+0000) 2024-03-06T15:21:50.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:50.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:50.799106+0000) 2024-03-06T15:21:51.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:51.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:51.799384+0000) 2024-03-06T15:21:52.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:52.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:52.799674+0000) 2024-03-06T15:21:53.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:53.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:53.799954+0000) 2024-03-06T15:21:54.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:54.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:54.800232+0000) 2024-03-06T15:21:55.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:55.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:55.800486+0000) 2024-03-06T15:21:56.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:56.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:56.800761+0000) 2024-03-06T15:21:57.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:57.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:57.801018+0000) 2024-03-06T15:21:58.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:58.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:58.801273+0000) 2024-03-06T15:21:59.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:21:59.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:59.801550+0000) 2024-03-06T15:22:00.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:00.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:00.801827+0000) 2024-03-06T15:22:01.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:01.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:01.802103+0000) 2024-03-06T15:22:02.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:02.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:02.802365+0000) 2024-03-06T15:22:03.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:03.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:03.802681+0000) 2024-03-06T15:22:04.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:04.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:04.803018+0000) 2024-03-06T15:22:05.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:05.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:05.803261+0000) 2024-03-06T15:22:06.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:06.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:06.803503+0000) 2024-03-06T15:22:07.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:07.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:07.803746+0000) 2024-03-06T15:22:08.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:08.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:08.804000+0000) 2024-03-06T15:22:09.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:09.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:09.804259+0000) 2024-03-06T15:22:10.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:10.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:10.804529+0000) 2024-03-06T15:22:11.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:11.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:11.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:11.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:11.804808+0000) 2024-03-06T15:22:12.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:12.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:12.805049+0000) 2024-03-06T15:22:13.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:13.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:13.805288+0000) 2024-03-06T15:22:14.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:14.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:14.805536+0000) 2024-03-06T15:22:15.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:15.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:15.805802+0000) 2024-03-06T15:22:16.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:16.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:16.806061+0000) 2024-03-06T15:22:17.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:17.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:17.806305+0000) 2024-03-06T15:22:18.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:18.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:18.806561+0000) 2024-03-06T15:22:19.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:19.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:19.806811+0000) 2024-03-06T15:22:20.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:20.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:20.807083+0000) 2024-03-06T15:22:21.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:21.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:21.807348+0000) 2024-03-06T15:22:22.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:22.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:22.807591+0000) 2024-03-06T15:22:23.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:23.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:23.807829+0000) 2024-03-06T15:22:24.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:24.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:24.808086+0000) 2024-03-06T15:22:25.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:25.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:25.808352+0000) 2024-03-06T15:22:26.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:26.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:26.808572+0000) 2024-03-06T15:22:27.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:27.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:27.808785+0000) 2024-03-06T15:22:28.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:28.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:28.809046+0000) 2024-03-06T15:22:29.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:29.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:29.809286+0000) 2024-03-06T15:22:30.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:30.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:30.809541+0000) 2024-03-06T15:22:31.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:31.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:31.809809+0000) 2024-03-06T15:22:32.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:32.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:32.810085+0000) 2024-03-06T15:22:33.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:33.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:33.810270+0000) 2024-03-06T15:22:34.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:34.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:34.810512+0000) 2024-03-06T15:22:35.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:35.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:35.810760+0000) 2024-03-06T15:22:36.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:36.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:36.811006+0000) 2024-03-06T15:22:37.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:37.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:37.811200+0000) 2024-03-06T15:22:38.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:38.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:38.811459+0000) 2024-03-06T15:22:39.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:39.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:39.811760+0000) 2024-03-06T15:22:40.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:40.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:40.811967+0000) 2024-03-06T15:22:41.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:41.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:41.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:41.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:41.812250+0000) 2024-03-06T15:22:42.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:42.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:42.812500+0000) 2024-03-06T15:22:43.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:43.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:43.812755+0000) 2024-03-06T15:22:44.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:44.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:44.813044+0000) 2024-03-06T15:22:45.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:45.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:45.813338+0000) 2024-03-06T15:22:46.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:46.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:46.813686+0000) 2024-03-06T15:22:47.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:47.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:47.814004+0000) 2024-03-06T15:22:48.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:48.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:48.814287+0000) 2024-03-06T15:22:49.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:49.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:49.814596+0000) 2024-03-06T15:22:50.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:50.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:50.814967+0000) 2024-03-06T15:22:51.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:51.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:51.815239+0000) 2024-03-06T15:22:52.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:52.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:52.815551+0000) 2024-03-06T15:22:53.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:53.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:53.815811+0000) 2024-03-06T15:22:54.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:54.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:54.816012+0000) 2024-03-06T15:22:55.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:55.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:55.816209+0000) 2024-03-06T15:22:56.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:56.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:56.816404+0000) 2024-03-06T15:22:57.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:57.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:57.816706+0000) 2024-03-06T15:22:58.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:58.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:58.816954+0000) 2024-03-06T15:22:59.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:22:59.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:59.817216+0000) 2024-03-06T15:23:00.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:00.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:00.817485+0000) 2024-03-06T15:23:01.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:01.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:01.817759+0000) 2024-03-06T15:23:02.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:02.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:02.818000+0000) 2024-03-06T15:23:03.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:03.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:03.818241+0000) 2024-03-06T15:23:04.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:04.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:04.818493+0000) 2024-03-06T15:23:05.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:05.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:05.818762+0000) 2024-03-06T15:23:06.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:06.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:06.818982+0000) 2024-03-06T15:23:07.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:07.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:07.819179+0000) 2024-03-06T15:23:08.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:08.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:08.819428+0000) 2024-03-06T15:23:09.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:09.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:09.819736+0000) 2024-03-06T15:23:10.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:10.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:10.819971+0000) 2024-03-06T15:23:11.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:11.790 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:11.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:11.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:11.820188+0000) 2024-03-06T15:23:12.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:12.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:12.820481+0000) 2024-03-06T15:23:13.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:13.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:13.820757+0000) 2024-03-06T15:23:14.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:14.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:14.821002+0000) 2024-03-06T15:23:15.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:15.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:15.821191+0000) 2024-03-06T15:23:16.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:16.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:16.821383+0000) 2024-03-06T15:23:17.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:17.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:17.821569+0000) 2024-03-06T15:23:18.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:18.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:18.821815+0000) 2024-03-06T15:23:19.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:19.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:19.822052+0000) 2024-03-06T15:23:20.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:20.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:20.822309+0000) 2024-03-06T15:23:21.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:21.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:21.822501+0000) 2024-03-06T15:23:22.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:22.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:22.822761+0000) 2024-03-06T15:23:23.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:23.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:23.823013+0000) 2024-03-06T15:23:24.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:24.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:24.823261+0000) 2024-03-06T15:23:25.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:25.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:25.823504+0000) 2024-03-06T15:23:26.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:26.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:26.823757+0000) 2024-03-06T15:23:27.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:27.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:27.824010+0000) 2024-03-06T15:23:28.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:28.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:28.824266+0000) 2024-03-06T15:23:29.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:29.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:29.824520+0000) 2024-03-06T15:23:30.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:30.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:30.824773+0000) 2024-03-06T15:23:31.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:31.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:31.825014+0000) 2024-03-06T15:23:32.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:32.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:32.825247+0000) 2024-03-06T15:23:33.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:33.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:33.825485+0000) 2024-03-06T15:23:34.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:34.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:34.825738+0000) 2024-03-06T15:23:35.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:35.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:35.825987+0000) 2024-03-06T15:23:36.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:36.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:36.826232+0000) 2024-03-06T15:23:37.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:37.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:37.826495+0000) 2024-03-06T15:23:38.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:38.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:38.826773+0000) 2024-03-06T15:23:39.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:39.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:39.827033+0000) 2024-03-06T15:23:40.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:40.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:40.827283+0000) 2024-03-06T15:23:41.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:41.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:41.827530+0000) 2024-03-06T15:23:41.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:41.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:42.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:42.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:42.827789+0000) 2024-03-06T15:23:43.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:43.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:43.828062+0000) 2024-03-06T15:23:44.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:44.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:44.828306+0000) 2024-03-06T15:23:45.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:45.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:45.828563+0000) 2024-03-06T15:23:46.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:46.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:46.828742+0000) 2024-03-06T15:23:47.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:47.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:47.828989+0000) 2024-03-06T15:23:48.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:48.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:48.829247+0000) 2024-03-06T15:23:49.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:49.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:49.829505+0000) 2024-03-06T15:23:50.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:50.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:50.829747+0000) 2024-03-06T15:23:51.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:51.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:51.830013+0000) 2024-03-06T15:23:52.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:52.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:52.830254+0000) 2024-03-06T15:23:53.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:53.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:53.830522+0000) 2024-03-06T15:23:54.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:54.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:54.830774+0000) 2024-03-06T15:23:55.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:55.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:55.831023+0000) 2024-03-06T15:23:56.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:56.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:56.831254+0000) 2024-03-06T15:23:57.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:57.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:57.831492+0000) 2024-03-06T15:23:58.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:58.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:58.831753+0000) 2024-03-06T15:23:59.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:23:59.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:59.832011+0000) 2024-03-06T15:24:00.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:00.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:00.832274+0000) 2024-03-06T15:24:01.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:01.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:01.832546+0000) 2024-03-06T15:24:02.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:02.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:02.832784+0000) 2024-03-06T15:24:03.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:03.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:03.833020+0000) 2024-03-06T15:24:04.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:04.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:04.833274+0000) 2024-03-06T15:24:05.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:05.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:05.833520+0000) 2024-03-06T15:24:06.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:06.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:06.833792+0000) 2024-03-06T15:24:07.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:07.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:07.834052+0000) 2024-03-06T15:24:08.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:08.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:08.834317+0000) 2024-03-06T15:24:09.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:09.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:09.834598+0000) 2024-03-06T15:24:10.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:10.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:10.834872+0000) 2024-03-06T15:24:11.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:11.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:11.835115+0000) 2024-03-06T15:24:12.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:12.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:12.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:12.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:12.835364+0000) 2024-03-06T15:24:13.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:13.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:13.835603+0000) 2024-03-06T15:24:14.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:14.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:14.835890+0000) 2024-03-06T15:24:15.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:15.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:15.836132+0000) 2024-03-06T15:24:16.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:16.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:16.836395+0000) 2024-03-06T15:24:17.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:17.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:17.836693+0000) 2024-03-06T15:24:18.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:18.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:18.836942+0000) 2024-03-06T15:24:19.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:19.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:19.837195+0000) 2024-03-06T15:24:20.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:20.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:20.837429+0000) 2024-03-06T15:24:21.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:21.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:21.837723+0000) 2024-03-06T15:24:22.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:22.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:22.837980+0000) 2024-03-06T15:24:23.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:23.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:23.838219+0000) 2024-03-06T15:24:24.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:24.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:24.838467+0000) 2024-03-06T15:24:25.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:25.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:25.838756+0000) 2024-03-06T15:24:26.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:26.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:26.839004+0000) 2024-03-06T15:24:27.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:27.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:27.839240+0000) 2024-03-06T15:24:28.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:28.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:28.839502+0000) 2024-03-06T15:24:29.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:29.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:29.839765+0000) 2024-03-06T15:24:30.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:30.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:30.840024+0000) 2024-03-06T15:24:31.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:31.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:31.840265+0000) 2024-03-06T15:24:32.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:32.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:32.840506+0000) 2024-03-06T15:24:33.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:33.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:33.840752+0000) 2024-03-06T15:24:34.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:34.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:34.840959+0000) 2024-03-06T15:24:35.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:35.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:35.841188+0000) 2024-03-06T15:24:36.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:36.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:36.841449+0000) 2024-03-06T15:24:37.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:37.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:37.841732+0000) 2024-03-06T15:24:38.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:38.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:38.841986+0000) 2024-03-06T15:24:39.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:39.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:39.842310+0000) 2024-03-06T15:24:40.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:40.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:40.842565+0000) 2024-03-06T15:24:41.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:41.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:41.842832+0000) 2024-03-06T15:24:42.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:42.231 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:42.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:42.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:42.843094+0000) 2024-03-06T15:24:43.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:43.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:43.843347+0000) 2024-03-06T15:24:44.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:44.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:44.843616+0000) 2024-03-06T15:24:45.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:45.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:45.843908+0000) 2024-03-06T15:24:46.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:46.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:46.844162+0000) 2024-03-06T15:24:47.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:47.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:47.844414+0000) 2024-03-06T15:24:48.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:48.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:48.844696+0000) 2024-03-06T15:24:49.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:49.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:49.844966+0000) 2024-03-06T15:24:50.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:50.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:50.845222+0000) 2024-03-06T15:24:51.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:51.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:51.845490+0000) 2024-03-06T15:24:52.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:52.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:52.845743+0000) 2024-03-06T15:24:53.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:53.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:53.845995+0000) 2024-03-06T15:24:54.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:54.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:54.846285+0000) 2024-03-06T15:24:55.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:55.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:55.846554+0000) 2024-03-06T15:24:56.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:56.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:56.846752+0000) 2024-03-06T15:24:57.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:57.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:57.847003+0000) 2024-03-06T15:24:58.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:58.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:58.847198+0000) 2024-03-06T15:24:59.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:24:59.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:59.847422+0000) 2024-03-06T15:25:00.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:00.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:00.847700+0000) 2024-03-06T15:25:01.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:01.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:01.847958+0000) 2024-03-06T15:25:02.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:02.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:02.848217+0000) 2024-03-06T15:25:03.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:03.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:03.848487+0000) 2024-03-06T15:25:04.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:04.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:04.848756+0000) 2024-03-06T15:25:05.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:05.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:05.849016+0000) 2024-03-06T15:25:06.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:06.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:06.849215+0000) 2024-03-06T15:25:07.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:07.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:07.849477+0000) 2024-03-06T15:25:08.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:08.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:08.849753+0000) 2024-03-06T15:25:09.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:09.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:09.850027+0000) 2024-03-06T15:25:10.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:10.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:10.850292+0000) 2024-03-06T15:25:11.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:11.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:11.850573+0000) 2024-03-06T15:25:12.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:12.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:12.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:12.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:12.850837+0000) 2024-03-06T15:25:13.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:13.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:13.851102+0000) 2024-03-06T15:25:14.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:14.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:14.851371+0000) 2024-03-06T15:25:15.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:15.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:15.851608+0000) 2024-03-06T15:25:16.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:16.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:16.851908+0000) 2024-03-06T15:25:17.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:17.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:17.852171+0000) 2024-03-06T15:25:18.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:18.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:18.852376+0000) 2024-03-06T15:25:19.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:19.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:19.852663+0000) 2024-03-06T15:25:20.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:20.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:20.852877+0000) 2024-03-06T15:25:21.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:21.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:21.853153+0000) 2024-03-06T15:25:22.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:22.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:22.853460+0000) 2024-03-06T15:25:23.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:23.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:23.853739+0000) 2024-03-06T15:25:24.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:24.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:24.853992+0000) 2024-03-06T15:25:25.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:25.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:25.854211+0000) 2024-03-06T15:25:26.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:26.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:26.854451+0000) 2024-03-06T15:25:27.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:27.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:27.854752+0000) 2024-03-06T15:25:28.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:28.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:28.854997+0000) 2024-03-06T15:25:29.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:29.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:29.855239+0000) 2024-03-06T15:25:30.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:30.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:30.855492+0000) 2024-03-06T15:25:31.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:31.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:31.855754+0000) 2024-03-06T15:25:32.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:32.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:32.856045+0000) 2024-03-06T15:25:33.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:33.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:33.856365+0000) 2024-03-06T15:25:34.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:34.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:34.856728+0000) 2024-03-06T15:25:35.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:35.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:35.857043+0000) 2024-03-06T15:25:36.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:36.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:36.857339+0000) 2024-03-06T15:25:37.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:37.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:37.857590+0000) 2024-03-06T15:25:38.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:38.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:38.857865+0000) 2024-03-06T15:25:39.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:39.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:39.858153+0000) 2024-03-06T15:25:40.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:40.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:40.858389+0000) 2024-03-06T15:25:41.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:41.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:41.858679+0000) 2024-03-06T15:25:42.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:42.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:42.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:42.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:42.859035+0000) 2024-03-06T15:25:43.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:43.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:43.859303+0000) 2024-03-06T15:25:44.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:44.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:44.859557+0000) 2024-03-06T15:25:45.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:45.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:45.859810+0000) 2024-03-06T15:25:46.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:46.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:46.860071+0000) 2024-03-06T15:25:47.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:47.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:47.860325+0000) 2024-03-06T15:25:48.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:48.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:48.860567+0000) 2024-03-06T15:25:49.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:49.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:49.860817+0000) 2024-03-06T15:25:50.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:50.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:50.861075+0000) 2024-03-06T15:25:51.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:51.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:51.861337+0000) 2024-03-06T15:25:52.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:52.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:52.861574+0000) 2024-03-06T15:25:53.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:53.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:53.861826+0000) 2024-03-06T15:25:54.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:54.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:54.862085+0000) 2024-03-06T15:25:55.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:55.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:55.862320+0000) 2024-03-06T15:25:56.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:56.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:56.862559+0000) 2024-03-06T15:25:57.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:57.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:57.862789+0000) 2024-03-06T15:25:58.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:58.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:58.863041+0000) 2024-03-06T15:25:59.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:25:59.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:59.863301+0000) 2024-03-06T15:26:00.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:00.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:00.863556+0000) 2024-03-06T15:26:01.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:01.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:01.863820+0000) 2024-03-06T15:26:02.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:02.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:02.864064+0000) 2024-03-06T15:26:03.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:03.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:03.864315+0000) 2024-03-06T15:26:04.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:04.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:04.864570+0000) 2024-03-06T15:26:05.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:05.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:05.864820+0000) 2024-03-06T15:26:06.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:06.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:06.865082+0000) 2024-03-06T15:26:07.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:07.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:07.865312+0000) 2024-03-06T15:26:08.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:08.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:08.865551+0000) 2024-03-06T15:26:09.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:09.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:09.865811+0000) 2024-03-06T15:26:10.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:10.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:10.866047+0000) 2024-03-06T15:26:11.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:11.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:11.866316+0000) 2024-03-06T15:26:12.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:12.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:12.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:12.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:12.866590+0000) 2024-03-06T15:26:13.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:13.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:13.866837+0000) 2024-03-06T15:26:14.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:14.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:14.867064+0000) 2024-03-06T15:26:15.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:15.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:15.867301+0000) 2024-03-06T15:26:16.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:16.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:16.867497+0000) 2024-03-06T15:26:17.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:17.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:17.867740+0000) 2024-03-06T15:26:18.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:18.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:18.867995+0000) 2024-03-06T15:26:19.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:19.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:19.868248+0000) 2024-03-06T15:26:20.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:20.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:20.868486+0000) 2024-03-06T15:26:21.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:21.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:21.868758+0000) 2024-03-06T15:26:22.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:22.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:22.869019+0000) 2024-03-06T15:26:23.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:23.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:23.869268+0000) 2024-03-06T15:26:24.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:24.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:24.869524+0000) 2024-03-06T15:26:25.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:25.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:25.869783+0000) 2024-03-06T15:26:26.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:26.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:26.870026+0000) 2024-03-06T15:26:27.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:27.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:27.870264+0000) 2024-03-06T15:26:28.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:28.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:28.870513+0000) 2024-03-06T15:26:29.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:29.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:29.870765+0000) 2024-03-06T15:26:30.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:30.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:30.871027+0000) 2024-03-06T15:26:31.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:31.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:31.871305+0000) 2024-03-06T15:26:32.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:32.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:32.871565+0000) 2024-03-06T15:26:33.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:33.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:33.871828+0000) 2024-03-06T15:26:34.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:34.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:34.872087+0000) 2024-03-06T15:26:35.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:35.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:35.872346+0000) 2024-03-06T15:26:36.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:36.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:36.872599+0000) 2024-03-06T15:26:37.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:37.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:37.872875+0000) 2024-03-06T15:26:38.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:38.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:38.873133+0000) 2024-03-06T15:26:39.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:39.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:39.873377+0000) 2024-03-06T15:26:40.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:40.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:40.873680+0000) 2024-03-06T15:26:41.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:41.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:41.873971+0000) 2024-03-06T15:26:42.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:42.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:42.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:42.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:42.874225+0000) 2024-03-06T15:26:43.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:43.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:43.874496+0000) 2024-03-06T15:26:44.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:44.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:44.874753+0000) 2024-03-06T15:26:45.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:45.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:45.875008+0000) 2024-03-06T15:26:46.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:46.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:46.875282+0000) 2024-03-06T15:26:47.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:47.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:47.875541+0000) 2024-03-06T15:26:48.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:48.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:48.875801+0000) 2024-03-06T15:26:49.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:49.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:49.876066+0000) 2024-03-06T15:26:50.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:50.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:50.876338+0000) 2024-03-06T15:26:51.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:51.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:51.876617+0000) 2024-03-06T15:26:52.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:52.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:52.876923+0000) 2024-03-06T15:26:53.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:53.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:53.877180+0000) 2024-03-06T15:26:54.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:54.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:54.877421+0000) 2024-03-06T15:26:55.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:55.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:55.877707+0000) 2024-03-06T15:26:56.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:56.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:56.877952+0000) 2024-03-06T15:26:57.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:57.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:57.878207+0000) 2024-03-06T15:26:58.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:58.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:58.878473+0000) 2024-03-06T15:26:59.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:26:59.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:59.878751+0000) 2024-03-06T15:27:00.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:00.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:00.879013+0000) 2024-03-06T15:27:01.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:01.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:01.879280+0000) 2024-03-06T15:27:02.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:02.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:02.879525+0000) 2024-03-06T15:27:03.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:03.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:03.879838+0000) 2024-03-06T15:27:04.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:04.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:04.880074+0000) 2024-03-06T15:27:05.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:05.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:05.880323+0000) 2024-03-06T15:27:06.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:06.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:06.880561+0000) 2024-03-06T15:27:07.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:07.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:07.880773+0000) 2024-03-06T15:27:08.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:08.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:08.881045+0000) 2024-03-06T15:27:09.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:09.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:09.881308+0000) 2024-03-06T15:27:10.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:10.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:10.881566+0000) 2024-03-06T15:27:11.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:11.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:11.881836+0000) 2024-03-06T15:27:12.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:12.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:12.882093+0000) 2024-03-06T15:27:12.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:12.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:13.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:13.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:13.882336+0000) 2024-03-06T15:27:14.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:14.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:14.882572+0000) 2024-03-06T15:27:15.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:15.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:15.882811+0000) 2024-03-06T15:27:16.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:16.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:16.883076+0000) 2024-03-06T15:27:17.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:17.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:17.883317+0000) 2024-03-06T15:27:18.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:18.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:18.883570+0000) 2024-03-06T15:27:19.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:19.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:19.883802+0000) 2024-03-06T15:27:20.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:20.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:20.884042+0000) 2024-03-06T15:27:21.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:21.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:21.884310+0000) 2024-03-06T15:27:22.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:22.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:22.884557+0000) 2024-03-06T15:27:23.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:23.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:23.884787+0000) 2024-03-06T15:27:24.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:24.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:24.884980+0000) 2024-03-06T15:27:25.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:25.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:25.885214+0000) 2024-03-06T15:27:26.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:26.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:26.885469+0000) 2024-03-06T15:27:27.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:27.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:27.885730+0000) 2024-03-06T15:27:28.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:28.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:28.885987+0000) 2024-03-06T15:27:29.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:29.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:29.886222+0000) 2024-03-06T15:27:30.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:30.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:30.886481+0000) 2024-03-06T15:27:31.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:31.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:31.886757+0000) 2024-03-06T15:27:32.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:32.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:32.887011+0000) 2024-03-06T15:27:33.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:33.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:33.887261+0000) 2024-03-06T15:27:34.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:34.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:34.887495+0000) 2024-03-06T15:27:35.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:35.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:35.887758+0000) 2024-03-06T15:27:36.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:36.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:36.887962+0000) 2024-03-06T15:27:37.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:37.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:37.888215+0000) 2024-03-06T15:27:38.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:38.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:38.888454+0000) 2024-03-06T15:27:39.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:39.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:39.888753+0000) 2024-03-06T15:27:40.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:40.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:40.888999+0000) 2024-03-06T15:27:41.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:41.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:41.889268+0000) 2024-03-06T15:27:42.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:42.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:42.889522+0000) 2024-03-06T15:27:43.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:43.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:43.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:43.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:43.889774+0000) 2024-03-06T15:27:44.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:44.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:44.890033+0000) 2024-03-06T15:27:45.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:45.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:45.890229+0000) 2024-03-06T15:27:46.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:46.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:46.890530+0000) 2024-03-06T15:27:47.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:47.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:47.890779+0000) 2024-03-06T15:27:48.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:48.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:48.891031+0000) 2024-03-06T15:27:49.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:49.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:49.891263+0000) 2024-03-06T15:27:50.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:50.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:50.891529+0000) 2024-03-06T15:27:51.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:51.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:51.891813+0000) 2024-03-06T15:27:52.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:52.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:52.892073+0000) 2024-03-06T15:27:53.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:53.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:53.892309+0000) 2024-03-06T15:27:54.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:54.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:54.892547+0000) 2024-03-06T15:27:55.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:55.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:55.892793+0000) 2024-03-06T15:27:56.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:56.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:56.893029+0000) 2024-03-06T15:27:57.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:57.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:57.893297+0000) 2024-03-06T15:27:58.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:58.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:58.893560+0000) 2024-03-06T15:27:59.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:27:59.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:59.893816+0000) 2024-03-06T15:28:00.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:00.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:00.894081+0000) 2024-03-06T15:28:01.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:01.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:01.894340+0000) 2024-03-06T15:28:02.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:02.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:02.894702+0000) 2024-03-06T15:28:03.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:03.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:03.894950+0000) 2024-03-06T15:28:04.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:04.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:04.895197+0000) 2024-03-06T15:28:05.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:05.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:05.895459+0000) 2024-03-06T15:28:06.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:06.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:06.895761+0000) 2024-03-06T15:28:07.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:07.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:07.896016+0000) 2024-03-06T15:28:08.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:08.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:08.896284+0000) 2024-03-06T15:28:09.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:09.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:09.896547+0000) 2024-03-06T15:28:10.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:10.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:10.896800+0000) 2024-03-06T15:28:11.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:11.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:11.897058+0000) 2024-03-06T15:28:12.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:12.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:12.897325+0000) 2024-03-06T15:28:13.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:13.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:13.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:13.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:13.897579+0000) 2024-03-06T15:28:14.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:14.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:14.897839+0000) 2024-03-06T15:28:15.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:15.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:15.898103+0000) 2024-03-06T15:28:16.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:16.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:16.898379+0000) 2024-03-06T15:28:17.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:17.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:17.898674+0000) 2024-03-06T15:28:18.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:18.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:18.898866+0000) 2024-03-06T15:28:19.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:19.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:19.899119+0000) 2024-03-06T15:28:20.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:20.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:20.899388+0000) 2024-03-06T15:28:21.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:21.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:21.899590+0000) 2024-03-06T15:28:22.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:22.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:22.899842+0000) 2024-03-06T15:28:23.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:23.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:23.900092+0000) 2024-03-06T15:28:24.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:24.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:24.900278+0000) 2024-03-06T15:28:25.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:25.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:25.900538+0000) 2024-03-06T15:28:26.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:26.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:26.900774+0000) 2024-03-06T15:28:27.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:27.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:27.900978+0000) 2024-03-06T15:28:28.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:28.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:28.901231+0000) 2024-03-06T15:28:29.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:29.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:29.901461+0000) 2024-03-06T15:28:30.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:30.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:30.901746+0000) 2024-03-06T15:28:31.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:31.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:31.902025+0000) 2024-03-06T15:28:32.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:32.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:32.902276+0000) 2024-03-06T15:28:33.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:33.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:33.902554+0000) 2024-03-06T15:28:34.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:34.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:34.902837+0000) 2024-03-06T15:28:35.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:35.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:35.903102+0000) 2024-03-06T15:28:36.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:36.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:36.903351+0000) 2024-03-06T15:28:37.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:37.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:37.903598+0000) 2024-03-06T15:28:38.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:38.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:38.903934+0000) 2024-03-06T15:28:39.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:39.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:39.904199+0000) 2024-03-06T15:28:40.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:40.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:40.904464+0000) 2024-03-06T15:28:41.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:41.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:41.904755+0000) 2024-03-06T15:28:42.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:42.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:42.905028+0000) 2024-03-06T15:28:43.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:43.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:43.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:43.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:43.905313+0000) 2024-03-06T15:28:44.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:44.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:44.905592+0000) 2024-03-06T15:28:45.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:45.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:45.905912+0000) 2024-03-06T15:28:46.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:46.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:46.906189+0000) 2024-03-06T15:28:47.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:47.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:47.906436+0000) 2024-03-06T15:28:48.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:48.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:48.906743+0000) 2024-03-06T15:28:49.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:49.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:49.907021+0000) 2024-03-06T15:28:50.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:50.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:50.907356+0000) 2024-03-06T15:28:51.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:51.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:51.907621+0000) 2024-03-06T15:28:52.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:52.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:52.907925+0000) 2024-03-06T15:28:53.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:53.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:53.908197+0000) 2024-03-06T15:28:54.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:54.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:54.908470+0000) 2024-03-06T15:28:55.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:55.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:55.908756+0000) 2024-03-06T15:28:56.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:56.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:56.909010+0000) 2024-03-06T15:28:57.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:57.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:57.909265+0000) 2024-03-06T15:28:58.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:58.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:58.909542+0000) 2024-03-06T15:28:59.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:28:59.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:59.909810+0000) 2024-03-06T15:29:00.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:00.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:00.910092+0000) 2024-03-06T15:29:01.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:01.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:01.910404+0000) 2024-03-06T15:29:02.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:02.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:02.910727+0000) 2024-03-06T15:29:03.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:03.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:03.911004+0000) 2024-03-06T15:29:04.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:04.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:04.911277+0000) 2024-03-06T15:29:05.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:05.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:05.911541+0000) 2024-03-06T15:29:06.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:06.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:06.911804+0000) 2024-03-06T15:29:07.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:07.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:07.912054+0000) 2024-03-06T15:29:08.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:08.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:08.912325+0000) 2024-03-06T15:29:09.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:09.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:09.912593+0000) 2024-03-06T15:29:10.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:10.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:10.912909+0000) 2024-03-06T15:29:11.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:11.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:11.913181+0000) 2024-03-06T15:29:12.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:12.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:12.913419+0000) 2024-03-06T15:29:13.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:13.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:13.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:13.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:13.913738+0000) 2024-03-06T15:29:14.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:14.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:14.913982+0000) 2024-03-06T15:29:15.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:15.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:15.914234+0000) 2024-03-06T15:29:16.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:16.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:16.914498+0000) 2024-03-06T15:29:17.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:17.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:17.914740+0000) 2024-03-06T15:29:18.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:18.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:18.914981+0000) 2024-03-06T15:29:19.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:19.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:19.915223+0000) 2024-03-06T15:29:20.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:20.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:20.915423+0000) 2024-03-06T15:29:21.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:21.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:21.915730+0000) 2024-03-06T15:29:22.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:22.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:22.915989+0000) 2024-03-06T15:29:23.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:23.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:23.916251+0000) 2024-03-06T15:29:24.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:24.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:24.916495+0000) 2024-03-06T15:29:25.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:25.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:25.916748+0000) 2024-03-06T15:29:26.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:26.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:26.917010+0000) 2024-03-06T15:29:27.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:27.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:27.917266+0000) 2024-03-06T15:29:28.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:28.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:28.917528+0000) 2024-03-06T15:29:29.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:29.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:29.917787+0000) 2024-03-06T15:29:30.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:30.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:30.918037+0000) 2024-03-06T15:29:31.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:31.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:31.918304+0000) 2024-03-06T15:29:32.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:32.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:32.918621+0000) 2024-03-06T15:29:33.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:33.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:33.918943+0000) 2024-03-06T15:29:34.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:34.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:34.919177+0000) 2024-03-06T15:29:35.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:35.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:35.919446+0000) 2024-03-06T15:29:36.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:36.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:36.919667+0000) 2024-03-06T15:29:37.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:37.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:37.919867+0000) 2024-03-06T15:29:38.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:38.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:38.920104+0000) 2024-03-06T15:29:39.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:39.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:39.920325+0000) 2024-03-06T15:29:40.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:40.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:40.920534+0000) 2024-03-06T15:29:41.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:41.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:41.920774+0000) 2024-03-06T15:29:42.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:42.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:42.921046+0000) 2024-03-06T15:29:43.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:43.714 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:43.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:43.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:43.921305+0000) 2024-03-06T15:29:44.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:44.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:44.921562+0000) 2024-03-06T15:29:45.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:45.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:45.921807+0000) 2024-03-06T15:29:46.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:46.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:46.922044+0000) 2024-03-06T15:29:47.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:47.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:47.922284+0000) 2024-03-06T15:29:48.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:48.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:48.922524+0000) 2024-03-06T15:29:49.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:49.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:49.922759+0000) 2024-03-06T15:29:50.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:50.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:50.923024+0000) 2024-03-06T15:29:51.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:51.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:51.923263+0000) 2024-03-06T15:29:52.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:52.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:52.923528+0000) 2024-03-06T15:29:53.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:53.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:53.923765+0000) 2024-03-06T15:29:54.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:54.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:54.924022+0000) 2024-03-06T15:29:55.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:55.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:55.924274+0000) 2024-03-06T15:29:56.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:56.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:56.924533+0000) 2024-03-06T15:29:57.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:57.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:57.924790+0000) 2024-03-06T15:29:58.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:58.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:58.925042+0000) 2024-03-06T15:29:59.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:29:59.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:59.925293+0000) 2024-03-06T15:30:00.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:00.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:00.925547+0000) 2024-03-06T15:30:01.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:01.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:01.925810+0000) 2024-03-06T15:30:02.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:02.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:02.925995+0000) 2024-03-06T15:30:03.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:03.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:03.926250+0000) 2024-03-06T15:30:04.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:04.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:04.926508+0000) 2024-03-06T15:30:05.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:05.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:05.926761+0000) 2024-03-06T15:30:06.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:06.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:06.927019+0000) 2024-03-06T15:30:07.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:07.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:07.927275+0000) 2024-03-06T15:30:08.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:08.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:08.927536+0000) 2024-03-06T15:30:09.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:09.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:09.927789+0000) 2024-03-06T15:30:10.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:10.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:10.928054+0000) 2024-03-06T15:30:11.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:11.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:11.928318+0000) 2024-03-06T15:30:12.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:12.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:12.928556+0000) 2024-03-06T15:30:13.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:13.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:13.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:13.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:13.928763+0000) 2024-03-06T15:30:14.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:14.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:14.928950+0000) 2024-03-06T15:30:15.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:15.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:15.929203+0000) 2024-03-06T15:30:16.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:16.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:16.929442+0000) 2024-03-06T15:30:17.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:17.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:17.929729+0000) 2024-03-06T15:30:18.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:18.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:18.929975+0000) 2024-03-06T15:30:19.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:19.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:19.930218+0000) 2024-03-06T15:30:20.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:20.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:20.930526+0000) 2024-03-06T15:30:21.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:21.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:21.930793+0000) 2024-03-06T15:30:22.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:22.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:22.931031+0000) 2024-03-06T15:30:23.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:23.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:23.931292+0000) 2024-03-06T15:30:24.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:24.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:24.931549+0000) 2024-03-06T15:30:25.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:25.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:25.931809+0000) 2024-03-06T15:30:26.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:26.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:26.932014+0000) 2024-03-06T15:30:27.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:27.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:27.932212+0000) 2024-03-06T15:30:28.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:28.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:28.932459+0000) 2024-03-06T15:30:29.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:29.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:29.932674+0000) 2024-03-06T15:30:30.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:30.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:30.932923+0000) 2024-03-06T15:30:31.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:31.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:31.933176+0000) 2024-03-06T15:30:32.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:32.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:32.933427+0000) 2024-03-06T15:30:33.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:33.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:33.933721+0000) 2024-03-06T15:30:34.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:34.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:34.933985+0000) 2024-03-06T15:30:35.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:35.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:35.934220+0000) 2024-03-06T15:30:36.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:36.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:36.934494+0000) 2024-03-06T15:30:37.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:37.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:37.934762+0000) 2024-03-06T15:30:38.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:38.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:38.935022+0000) 2024-03-06T15:30:39.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:39.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:39.935265+0000) 2024-03-06T15:30:40.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:40.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:40.935529+0000) 2024-03-06T15:30:41.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:41.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:41.935805+0000) 2024-03-06T15:30:42.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:42.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:42.936056+0000) 2024-03-06T15:30:43.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:43.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:43.936314+0000) 2024-03-06T15:30:44.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:44.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:44.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:44.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:44.936574+0000) 2024-03-06T15:30:45.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:45.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:45.936791+0000) 2024-03-06T15:30:46.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:46.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:46.937035+0000) 2024-03-06T15:30:47.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:47.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:47.937323+0000) 2024-03-06T15:30:48.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:48.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:48.937594+0000) 2024-03-06T15:30:49.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:49.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:49.937911+0000) 2024-03-06T15:30:50.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:50.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:50.938141+0000) 2024-03-06T15:30:51.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:51.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:51.938403+0000) 2024-03-06T15:30:52.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:52.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:52.938696+0000) 2024-03-06T15:30:53.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:53.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:53.938891+0000) 2024-03-06T15:30:54.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:54.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:54.939141+0000) 2024-03-06T15:30:55.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:55.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:55.939389+0000) 2024-03-06T15:30:56.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:56.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:56.939667+0000) 2024-03-06T15:30:57.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:57.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:57.939915+0000) 2024-03-06T15:30:58.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:58.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:58.940167+0000) 2024-03-06T15:30:59.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:30:59.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:59.940404+0000) 2024-03-06T15:31:00.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:00.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:00.940747+0000) 2024-03-06T15:31:01.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:01.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:01.940997+0000) 2024-03-06T15:31:02.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:02.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:02.941230+0000) 2024-03-06T15:31:03.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:03.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:03.941469+0000) 2024-03-06T15:31:04.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:04.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:04.941749+0000) 2024-03-06T15:31:05.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:05.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:05.941982+0000) 2024-03-06T15:31:06.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:06.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:06.942165+0000) 2024-03-06T15:31:07.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:07.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:07.942431+0000) 2024-03-06T15:31:08.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:08.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:08.942740+0000) 2024-03-06T15:31:09.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:09.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:09.942969+0000) 2024-03-06T15:31:10.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:10.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:10.943202+0000) 2024-03-06T15:31:11.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:11.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:11.943444+0000) 2024-03-06T15:31:12.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:12.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:12.943735+0000) 2024-03-06T15:31:13.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:13.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:13.943991+0000) 2024-03-06T15:31:14.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:14.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:14.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:14.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:14.944255+0000) 2024-03-06T15:31:15.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:15.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:15.944486+0000) 2024-03-06T15:31:16.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:16.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:16.944774+0000) 2024-03-06T15:31:17.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:17.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:17.945006+0000) 2024-03-06T15:31:18.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:18.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:18.945251+0000) 2024-03-06T15:31:19.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:19.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:19.945510+0000) 2024-03-06T15:31:20.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:20.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:20.945765+0000) 2024-03-06T15:31:21.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:21.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:21.946026+0000) 2024-03-06T15:31:22.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:22.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:22.946286+0000) 2024-03-06T15:31:23.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:23.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:23.946537+0000) 2024-03-06T15:31:24.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:24.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:24.946775+0000) 2024-03-06T15:31:25.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:25.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:25.947027+0000) 2024-03-06T15:31:26.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:26.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:26.947286+0000) 2024-03-06T15:31:27.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:27.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:27.947543+0000) 2024-03-06T15:31:28.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:28.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:28.947784+0000) 2024-03-06T15:31:29.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:29.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:29.947971+0000) 2024-03-06T15:31:30.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:30.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:30.948226+0000) 2024-03-06T15:31:31.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:31.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:31.948481+0000) 2024-03-06T15:31:32.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:32.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:32.948753+0000) 2024-03-06T15:31:33.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:33.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:33.949003+0000) 2024-03-06T15:31:34.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:34.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:34.949263+0000) 2024-03-06T15:31:35.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:35.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:35.949519+0000) 2024-03-06T15:31:36.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:36.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:36.949779+0000) 2024-03-06T15:31:37.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:37.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:37.950027+0000) 2024-03-06T15:31:38.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:38.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:38.950287+0000) 2024-03-06T15:31:39.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:39.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:39.950548+0000) 2024-03-06T15:31:40.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:40.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:40.950799+0000) 2024-03-06T15:31:41.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:41.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:41.950998+0000) 2024-03-06T15:31:42.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:42.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:42.951248+0000) 2024-03-06T15:31:43.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:43.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:43.951488+0000) 2024-03-06T15:31:44.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:44.336 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:44.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:44.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:44.951747+0000) 2024-03-06T15:31:45.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:45.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:45.951990+0000) 2024-03-06T15:31:46.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:46.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:46.952249+0000) 2024-03-06T15:31:47.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:47.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:47.952503+0000) 2024-03-06T15:31:48.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:48.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:48.952763+0000) 2024-03-06T15:31:49.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:49.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:49.953006+0000) 2024-03-06T15:31:50.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:50.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:50.953254+0000) 2024-03-06T15:31:51.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:51.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:51.953508+0000) 2024-03-06T15:31:52.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:52.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:52.953767+0000) 2024-03-06T15:31:53.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:53.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:53.954019+0000) 2024-03-06T15:31:54.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:54.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:54.954229+0000) 2024-03-06T15:31:55.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:55.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:55.954487+0000) 2024-03-06T15:31:56.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:56.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:56.954746+0000) 2024-03-06T15:31:57.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:57.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:57.954944+0000) 2024-03-06T15:31:58.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:58.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:58.955200+0000) 2024-03-06T15:31:59.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:31:59.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:59.955450+0000) 2024-03-06T15:32:00.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:00.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:00.955744+0000) 2024-03-06T15:32:01.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:01.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:01.956016+0000) 2024-03-06T15:32:02.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:02.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:02.956284+0000) 2024-03-06T15:32:03.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:03.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:03.956522+0000) 2024-03-06T15:32:04.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:04.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:04.956773+0000) 2024-03-06T15:32:05.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:05.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:05.957021+0000) 2024-03-06T15:32:06.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:06.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:06.957270+0000) 2024-03-06T15:32:07.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:07.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:07.957593+0000) 2024-03-06T15:32:08.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:08.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:08.957857+0000) 2024-03-06T15:32:09.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:09.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:09.958094+0000) 2024-03-06T15:32:10.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:10.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:10.958330+0000) 2024-03-06T15:32:11.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:11.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:11.958597+0000) 2024-03-06T15:32:12.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:12.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:12.958852+0000) 2024-03-06T15:32:13.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:13.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:13.959085+0000) 2024-03-06T15:32:14.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:14.495 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:14.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:14.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:14.959330+0000) 2024-03-06T15:32:15.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:15.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:15.959564+0000) 2024-03-06T15:32:16.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:16.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:16.959810+0000) 2024-03-06T15:32:17.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:17.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:17.960066+0000) 2024-03-06T15:32:18.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:18.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:18.960265+0000) 2024-03-06T15:32:19.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:19.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:19.960446+0000) 2024-03-06T15:32:20.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:20.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:20.960739+0000) 2024-03-06T15:32:21.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:21.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:21.961010+0000) 2024-03-06T15:32:22.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:22.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:22.961277+0000) 2024-03-06T15:32:23.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:23.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:23.961460+0000) 2024-03-06T15:32:24.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:24.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:24.961751+0000) 2024-03-06T15:32:25.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:25.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:25.961985+0000) 2024-03-06T15:32:26.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:26.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:26.962227+0000) 2024-03-06T15:32:27.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:27.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:27.962478+0000) 2024-03-06T15:32:28.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:28.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:28.962758+0000) 2024-03-06T15:32:29.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:29.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:29.962987+0000) 2024-03-06T15:32:30.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:30.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:30.963226+0000) 2024-03-06T15:32:31.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:31.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:31.963472+0000) 2024-03-06T15:32:32.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:32.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:32.963755+0000) 2024-03-06T15:32:33.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:33.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:33.964000+0000) 2024-03-06T15:32:34.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:34.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:34.964263+0000) 2024-03-06T15:32:35.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:35.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:35.964519+0000) 2024-03-06T15:32:36.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:36.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:36.964788+0000) 2024-03-06T15:32:37.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:37.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:37.965044+0000) 2024-03-06T15:32:38.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:38.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:38.965312+0000) 2024-03-06T15:32:39.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:39.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:39.965551+0000) 2024-03-06T15:32:40.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:40.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:40.965813+0000) 2024-03-06T15:32:41.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:41.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:41.966086+0000) 2024-03-06T15:32:42.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:42.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:42.966289+0000) 2024-03-06T15:32:43.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:43.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:43.966520+0000) 2024-03-06T15:32:44.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:44.638 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:44.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:44.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:44.966775+0000) 2024-03-06T15:32:45.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:45.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:45.967004+0000) 2024-03-06T15:32:46.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:46.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:46.967191+0000) 2024-03-06T15:32:47.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:47.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:47.967432+0000) 2024-03-06T15:32:48.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:48.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:48.967753+0000) 2024-03-06T15:32:49.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:49.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:49.967986+0000) 2024-03-06T15:32:50.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:50.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:50.968252+0000) 2024-03-06T15:32:51.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:51.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:51.968518+0000) 2024-03-06T15:32:52.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:52.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:52.968779+0000) 2024-03-06T15:32:53.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:53.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:53.969035+0000) 2024-03-06T15:32:54.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:54.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:54.969271+0000) 2024-03-06T15:32:55.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:55.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:55.969499+0000) 2024-03-06T15:32:56.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:56.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:56.969743+0000) 2024-03-06T15:32:57.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:57.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:57.969996+0000) 2024-03-06T15:32:58.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:58.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:58.970265+0000) 2024-03-06T15:32:59.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:32:59.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:59.970513+0000) 2024-03-06T15:33:00.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:00.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:00.970834+0000) 2024-03-06T15:33:01.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:01.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:01.971102+0000) 2024-03-06T15:33:02.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:02.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:02.971360+0000) 2024-03-06T15:33:03.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:03.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:03.971609+0000) 2024-03-06T15:33:04.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:04.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:04.971905+0000) 2024-03-06T15:33:05.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:05.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:05.972142+0000) 2024-03-06T15:33:06.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:06.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:06.972414+0000) 2024-03-06T15:33:07.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:07.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:07.972720+0000) 2024-03-06T15:33:08.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:08.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:08.972970+0000) 2024-03-06T15:33:09.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:09.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:09.973150+0000) 2024-03-06T15:33:10.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:10.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:10.973411+0000) 2024-03-06T15:33:11.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:11.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:11.973715+0000) 2024-03-06T15:33:12.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:12.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:12.973991+0000) 2024-03-06T15:33:13.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:13.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:13.974239+0000) 2024-03-06T15:33:14.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:14.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:14.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:14.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:14.974496+0000) 2024-03-06T15:33:15.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:15.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:15.974760+0000) 2024-03-06T15:33:16.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:16.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:16.974999+0000) 2024-03-06T15:33:17.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:17.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:17.975257+0000) 2024-03-06T15:33:18.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:18.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:18.975504+0000) 2024-03-06T15:33:19.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:19.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:19.975766+0000) 2024-03-06T15:33:20.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:20.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:20.976032+0000) 2024-03-06T15:33:21.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:21.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:21.976309+0000) 2024-03-06T15:33:22.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:22.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:22.976573+0000) 2024-03-06T15:33:23.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:23.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:23.976759+0000) 2024-03-06T15:33:24.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:24.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:24.977016+0000) 2024-03-06T15:33:25.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:25.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:25.977264+0000) 2024-03-06T15:33:26.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:26.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:26.977512+0000) 2024-03-06T15:33:27.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:27.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:27.977747+0000) 2024-03-06T15:33:28.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:28.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:28.977938+0000) 2024-03-06T15:33:29.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:29.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:29.978176+0000) 2024-03-06T15:33:30.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:30.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:30.978433+0000) 2024-03-06T15:33:31.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:31.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:31.978756+0000) 2024-03-06T15:33:32.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:32.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:32.979011+0000) 2024-03-06T15:33:33.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:33.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:33.979253+0000) 2024-03-06T15:33:34.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:34.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:34.979509+0000) 2024-03-06T15:33:35.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:35.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:35.979738+0000) 2024-03-06T15:33:36.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:36.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:36.980002+0000) 2024-03-06T15:33:37.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:37.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:37.980255+0000) 2024-03-06T15:33:38.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:38.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:38.980494+0000) 2024-03-06T15:33:39.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:39.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:39.980771+0000) 2024-03-06T15:33:40.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:40.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:40.981035+0000) 2024-03-06T15:33:41.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:41.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:41.981304+0000) 2024-03-06T15:33:42.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:42.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:42.981489+0000) 2024-03-06T15:33:43.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:43.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:43.981816+0000) 2024-03-06T15:33:44.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:44.937 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:44.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:44.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:44.981983+0000) 2024-03-06T15:33:45.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:45.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:45.982218+0000) 2024-03-06T15:33:46.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:46.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:46.982495+0000) 2024-03-06T15:33:47.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:47.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:47.982751+0000) 2024-03-06T15:33:48.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:48.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:48.983019+0000) 2024-03-06T15:33:49.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:49.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:49.983278+0000) 2024-03-06T15:33:50.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:50.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:50.983539+0000) 2024-03-06T15:33:51.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:51.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:51.983821+0000) 2024-03-06T15:33:52.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:52.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:52.984075+0000) 2024-03-06T15:33:53.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:53.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:53.984332+0000) 2024-03-06T15:33:54.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:54.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:54.984594+0000) 2024-03-06T15:33:55.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:55.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:55.984852+0000) 2024-03-06T15:33:56.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:56.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:56.985101+0000) 2024-03-06T15:33:57.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:57.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:57.985331+0000) 2024-03-06T15:33:58.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:58.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:58.985593+0000) 2024-03-06T15:33:59.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:33:59.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:59.985851+0000) 2024-03-06T15:34:00.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:00.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:00.986097+0000) 2024-03-06T15:34:01.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:01.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:01.986356+0000) 2024-03-06T15:34:02.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:02.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:02.986619+0000) 2024-03-06T15:34:03.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:03.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:03.986923+0000) 2024-03-06T15:34:04.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:04.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:04.987191+0000) 2024-03-06T15:34:05.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:05.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:05.987449+0000) 2024-03-06T15:34:06.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:06.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:06.987774+0000) 2024-03-06T15:34:07.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:07.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:07.988061+0000) 2024-03-06T15:34:08.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:08.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:08.988296+0000) 2024-03-06T15:34:09.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:09.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:09.988562+0000) 2024-03-06T15:34:10.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:10.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:10.988816+0000) 2024-03-06T15:34:11.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:11.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:11.989044+0000) 2024-03-06T15:34:12.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:12.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:12.989250+0000) 2024-03-06T15:34:13.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:13.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:13.989453+0000) 2024-03-06T15:34:14.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:14.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:14.989679+0000) 2024-03-06T15:34:15.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:15.080 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:15.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:15.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:15.989901+0000) 2024-03-06T15:34:16.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:16.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:16.990114+0000) 2024-03-06T15:34:17.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:17.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:17.990310+0000) 2024-03-06T15:34:18.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:18.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:18.990501+0000) 2024-03-06T15:34:19.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:19.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:19.990717+0000) 2024-03-06T15:34:20.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:20.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:20.990897+0000) 2024-03-06T15:34:21.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:21.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:21.991096+0000) 2024-03-06T15:34:22.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:22.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:22.991345+0000) 2024-03-06T15:34:23.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:23.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:23.991530+0000) 2024-03-06T15:34:24.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:24.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:24.991785+0000) 2024-03-06T15:34:25.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:25.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:25.992044+0000) 2024-03-06T15:34:26.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:26.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:26.992237+0000) 2024-03-06T15:34:27.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:27.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:27.992478+0000) 2024-03-06T15:34:28.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:28.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:28.992753+0000) 2024-03-06T15:34:29.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:29.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:29.992936+0000) 2024-03-06T15:34:30.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:30.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:30.993133+0000) 2024-03-06T15:34:31.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:31.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:31.993435+0000) 2024-03-06T15:34:32.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:32.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:32.993729+0000) 2024-03-06T15:34:33.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:33.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:33.993986+0000) 2024-03-06T15:34:35.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:34.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:34.994239+0000) 2024-03-06T15:34:35.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:35.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:35.994497+0000) 2024-03-06T15:34:36.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:36.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:36.994754+0000) 2024-03-06T15:34:37.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:37.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:37.994976+0000) 2024-03-06T15:34:38.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:38.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:38.995233+0000) 2024-03-06T15:34:39.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:39.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:39.995463+0000) 2024-03-06T15:34:40.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:40.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:40.995749+0000) 2024-03-06T15:34:42.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:41.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:41.996015+0000) 2024-03-06T15:34:43.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:42.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:42.996280+0000) 2024-03-06T15:34:44.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:43.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:43.996521+0000) 2024-03-06T15:34:45.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:44.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:44.996778+0000) 2024-03-06T15:34:45.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:45.350 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:46.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:45.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:45.996972+0000) 2024-03-06T15:34:47.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:46.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:46.997222+0000) 2024-03-06T15:34:48.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:47.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:47.997487+0000) 2024-03-06T15:34:49.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:48.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:48.997763+0000) 2024-03-06T15:34:50.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:49.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:49.997998+0000) 2024-03-06T15:34:51.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:50.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:50.998235+0000) 2024-03-06T15:34:52.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:51.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:51.998479+0000) 2024-03-06T15:34:53.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:52.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:52.998740+0000) 2024-03-06T15:34:54.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:53.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:53.998935+0000) 2024-03-06T15:34:55.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:54.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:54.999188+0000) 2024-03-06T15:34:56.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:55.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:55.999440+0000) 2024-03-06T15:34:57.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:56.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:56.999743+0000) 2024-03-06T15:34:58.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:57.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:58.000004+0000) 2024-03-06T15:34:59.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:34:59.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:59.000277+0000) 2024-03-06T15:35:00.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:00.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:00.000532+0000) 2024-03-06T15:35:01.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:01.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:01.000775+0000) 2024-03-06T15:35:02.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:02.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:02.000976+0000) 2024-03-06T15:35:03.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:03.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:03.001178+0000) 2024-03-06T15:35:04.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:04.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:04.001431+0000) 2024-03-06T15:35:05.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:05.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:05.001695+0000) 2024-03-06T15:35:06.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:06.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:06.001888+0000) 2024-03-06T15:35:07.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:07.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:07.002081+0000) 2024-03-06T15:35:08.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:08.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:08.002274+0000) 2024-03-06T15:35:09.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:09.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:09.002534+0000) 2024-03-06T15:35:10.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:10.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:10.002790+0000) 2024-03-06T15:35:11.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:11.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:11.003027+0000) 2024-03-06T15:35:12.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:12.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:12.003270+0000) 2024-03-06T15:35:13.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:13.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:13.003511+0000) 2024-03-06T15:35:14.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:14.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:14.003753+0000) 2024-03-06T15:35:15.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:15.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:15.003991+0000) 2024-03-06T15:35:15.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:15.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:16.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:16.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:16.004240+0000) 2024-03-06T15:35:17.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:17.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:17.004487+0000) 2024-03-06T15:35:18.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:18.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:18.004768+0000) 2024-03-06T15:35:19.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:19.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:19.004964+0000) 2024-03-06T15:35:20.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:20.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:20.005221+0000) 2024-03-06T15:35:21.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:21.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:21.005475+0000) 2024-03-06T15:35:22.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:22.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:22.005741+0000) 2024-03-06T15:35:23.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:23.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:23.005999+0000) 2024-03-06T15:35:24.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:24.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:24.006251+0000) 2024-03-06T15:35:25.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:25.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:25.006508+0000) 2024-03-06T15:35:26.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:26.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:26.006759+0000) 2024-03-06T15:35:27.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:27.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:27.006998+0000) 2024-03-06T15:35:28.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:28.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:28.007235+0000) 2024-03-06T15:35:29.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:29.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:29.007498+0000) 2024-03-06T15:35:30.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:30.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:30.007752+0000) 2024-03-06T15:35:31.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:31.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:31.008075+0000) 2024-03-06T15:35:32.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:32.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:32.008321+0000) 2024-03-06T15:35:33.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:33.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:33.008580+0000) 2024-03-06T15:35:34.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:34.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:34.008840+0000) 2024-03-06T15:35:35.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:35.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:35.009094+0000) 2024-03-06T15:35:36.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:36.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:36.009346+0000) 2024-03-06T15:35:37.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:37.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:37.009604+0000) 2024-03-06T15:35:38.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:38.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:38.009885+0000) 2024-03-06T15:35:39.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:39.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:39.010131+0000) 2024-03-06T15:35:40.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:40.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:40.010381+0000) 2024-03-06T15:35:41.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:41.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:41.010623+0000) 2024-03-06T15:35:42.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:42.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:42.010931+0000) 2024-03-06T15:35:43.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:43.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:43.011176+0000) 2024-03-06T15:35:44.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:44.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:44.011416+0000) 2024-03-06T15:35:45.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:45.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:45.011717+0000) 2024-03-06T15:35:45.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:45.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:46.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:46.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:46.011974+0000) 2024-03-06T15:35:47.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:47.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:47.012231+0000) 2024-03-06T15:35:48.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:48.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:48.012420+0000) 2024-03-06T15:35:49.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:49.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:49.012735+0000) 2024-03-06T15:35:50.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:50.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:50.012999+0000) 2024-03-06T15:35:51.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:51.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:51.013268+0000) 2024-03-06T15:35:52.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:52.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:52.013536+0000) 2024-03-06T15:35:53.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:53.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:53.013782+0000) 2024-03-06T15:35:54.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:54.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:54.013998+0000) 2024-03-06T15:35:55.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:55.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:55.014204+0000) 2024-03-06T15:35:56.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:56.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:56.014407+0000) 2024-03-06T15:35:57.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:57.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:57.014705+0000) 2024-03-06T15:35:58.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:58.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:58.014973+0000) 2024-03-06T15:35:59.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:35:59.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:59.015251+0000) 2024-03-06T15:36:00.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:00.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:00.015476+0000) 2024-03-06T15:36:01.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:01.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:01.015752+0000) 2024-03-06T15:36:02.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:02.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:02.016000+0000) 2024-03-06T15:36:03.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:03.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:03.016288+0000) 2024-03-06T15:36:04.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:04.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:04.016559+0000) 2024-03-06T15:36:05.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:05.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:05.016835+0000) 2024-03-06T15:36:06.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:06.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:06.017106+0000) 2024-03-06T15:36:07.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:07.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:07.017362+0000) 2024-03-06T15:36:08.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:08.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:08.017623+0000) 2024-03-06T15:36:09.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:09.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:09.017943+0000) 2024-03-06T15:36:10.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:10.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:10.018203+0000) 2024-03-06T15:36:11.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:11.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:11.018442+0000) 2024-03-06T15:36:12.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:12.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:12.018739+0000) 2024-03-06T15:36:13.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:13.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:13.018988+0000) 2024-03-06T15:36:14.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:14.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:14.019222+0000) 2024-03-06T15:36:15.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:15.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:15.019472+0000) 2024-03-06T15:36:15.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:15.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:16.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:16.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:16.019763+0000) 2024-03-06T15:36:17.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:17.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:17.019997+0000) 2024-03-06T15:36:18.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:18.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:18.020281+0000) 2024-03-06T15:36:19.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:19.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:19.020597+0000) 2024-03-06T15:36:20.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:20.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:20.020859+0000) 2024-03-06T15:36:21.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:21.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:21.021117+0000) 2024-03-06T15:36:22.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:22.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:22.021391+0000) 2024-03-06T15:36:23.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:23.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:23.021583+0000) 2024-03-06T15:36:24.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:24.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:24.021851+0000) 2024-03-06T15:36:25.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:25.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:25.022039+0000) 2024-03-06T15:36:26.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:26.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:26.022303+0000) 2024-03-06T15:36:27.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:27.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:27.022557+0000) 2024-03-06T15:36:28.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:28.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:28.022817+0000) 2024-03-06T15:36:29.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:29.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:29.023006+0000) 2024-03-06T15:36:30.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:30.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:30.023252+0000) 2024-03-06T15:36:31.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:31.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:31.023444+0000) 2024-03-06T15:36:32.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:32.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:32.023752+0000) 2024-03-06T15:36:33.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:33.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:33.024007+0000) 2024-03-06T15:36:34.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:34.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:34.024217+0000) 2024-03-06T15:36:35.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:35.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:35.024481+0000) 2024-03-06T15:36:36.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:36.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:36.024752+0000) 2024-03-06T15:36:37.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:37.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:37.024968+0000) 2024-03-06T15:36:38.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:38.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:38.025226+0000) 2024-03-06T15:36:39.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:39.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:39.025486+0000) 2024-03-06T15:36:40.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:40.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:40.025773+0000) 2024-03-06T15:36:41.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:41.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:41.026011+0000) 2024-03-06T15:36:42.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:42.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:42.026213+0000) 2024-03-06T15:36:43.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:43.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:43.026456+0000) 2024-03-06T15:36:44.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:44.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:44.026746+0000) 2024-03-06T15:36:45.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:45.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:45.026998+0000) 2024-03-06T15:36:45.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:45.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:46.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:46.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:46.027213+0000) 2024-03-06T15:36:47.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:47.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:47.027465+0000) 2024-03-06T15:36:48.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:48.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:48.027756+0000) 2024-03-06T15:36:49.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:49.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:49.028027+0000) 2024-03-06T15:36:50.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:50.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:50.028219+0000) 2024-03-06T15:36:51.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:51.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:51.028457+0000) 2024-03-06T15:36:52.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:52.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:52.028753+0000) 2024-03-06T15:36:53.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:53.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:53.029008+0000) 2024-03-06T15:36:54.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:54.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:54.029271+0000) 2024-03-06T15:36:55.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:55.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:55.029532+0000) 2024-03-06T15:36:56.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:56.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:56.029775+0000) 2024-03-06T15:36:57.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:57.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:57.030036+0000) 2024-03-06T15:36:58.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:58.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:58.030293+0000) 2024-03-06T15:36:59.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:36:59.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:59.030498+0000) 2024-03-06T15:37:00.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:00.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:00.030748+0000) 2024-03-06T15:37:01.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:01.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:01.031005+0000) 2024-03-06T15:37:02.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:02.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:02.031261+0000) 2024-03-06T15:37:03.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:03.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:03.031534+0000) 2024-03-06T15:37:04.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:04.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:04.031816+0000) 2024-03-06T15:37:05.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:05.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:05.032091+0000) 2024-03-06T15:37:06.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:06.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:06.032385+0000) 2024-03-06T15:37:07.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:07.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:07.032811+0000) 2024-03-06T15:37:08.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:08.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:08.033123+0000) 2024-03-06T15:37:09.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:09.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:09.033430+0000) 2024-03-06T15:37:10.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:10.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:10.033712+0000) 2024-03-06T15:37:11.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:11.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:11.033941+0000) 2024-03-06T15:37:12.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:12.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:12.034184+0000) 2024-03-06T15:37:13.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:13.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:13.034493+0000) 2024-03-06T15:37:14.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:14.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:14.034763+0000) 2024-03-06T15:37:15.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:15.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:15.035018+0000) 2024-03-06T15:37:16.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:16.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:16.035201+0000) 2024-03-06T15:37:16.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:16.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:17.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:17.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:17.035463+0000) 2024-03-06T15:37:18.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:18.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:18.035739+0000) 2024-03-06T15:37:19.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:19.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:19.036003+0000) 2024-03-06T15:37:20.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:20.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:20.036198+0000) 2024-03-06T15:37:21.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:21.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:21.036469+0000) 2024-03-06T15:37:22.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:22.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:22.036756+0000) 2024-03-06T15:37:23.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:23.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:23.036947+0000) 2024-03-06T15:37:24.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:24.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:24.037201+0000) 2024-03-06T15:37:25.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:25.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:25.037454+0000) 2024-03-06T15:37:26.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:26.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:26.037745+0000) 2024-03-06T15:37:27.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:27.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:27.038016+0000) 2024-03-06T15:37:28.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:28.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:28.038278+0000) 2024-03-06T15:37:29.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:29.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:29.038535+0000) 2024-03-06T15:37:30.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:30.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:30.038801+0000) 2024-03-06T15:37:31.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:31.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:31.039074+0000) 2024-03-06T15:37:32.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:32.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:32.039317+0000) 2024-03-06T15:37:33.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:33.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:33.039590+0000) 2024-03-06T15:37:34.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:34.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:34.039907+0000) 2024-03-06T15:37:35.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:35.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:35.040127+0000) 2024-03-06T15:37:36.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:36.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:36.040396+0000) 2024-03-06T15:37:37.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:37.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:37.040687+0000) 2024-03-06T15:37:38.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:38.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:38.040950+0000) 2024-03-06T15:37:39.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:39.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:39.041194+0000) 2024-03-06T15:37:40.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:40.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:40.041440+0000) 2024-03-06T15:37:41.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:41.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:41.041735+0000) 2024-03-06T15:37:42.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:42.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:42.041998+0000) 2024-03-06T15:37:43.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:43.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:43.042266+0000) 2024-03-06T15:37:44.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:44.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:44.042524+0000) 2024-03-06T15:37:45.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:45.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:45.042783+0000) 2024-03-06T15:37:46.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:46.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:46.043038+0000) 2024-03-06T15:37:46.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:46.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:47.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:47.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:47.043301+0000) 2024-03-06T15:37:48.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:48.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:48.043558+0000) 2024-03-06T15:37:49.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:49.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:49.043797+0000) 2024-03-06T15:37:50.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:50.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:50.043976+0000) 2024-03-06T15:37:51.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:51.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:51.044226+0000) 2024-03-06T15:37:52.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:52.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:52.044463+0000) 2024-03-06T15:37:53.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:53.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:53.044758+0000) 2024-03-06T15:37:54.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:54.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:54.045003+0000) 2024-03-06T15:37:55.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:55.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:55.045261+0000) 2024-03-06T15:37:56.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:56.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:56.045515+0000) 2024-03-06T15:37:57.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:57.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:57.045752+0000) 2024-03-06T15:37:58.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:58.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:58.046013+0000) 2024-03-06T15:37:59.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:37:59.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:59.046273+0000) 2024-03-06T15:38:00.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:00.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:00.046514+0000) 2024-03-06T15:38:01.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:01.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:01.046769+0000) 2024-03-06T15:38:02.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:02.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:02.047034+0000) 2024-03-06T15:38:03.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:03.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:03.047303+0000) 2024-03-06T15:38:04.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:04.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:04.047539+0000) 2024-03-06T15:38:05.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:05.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:05.047782+0000) 2024-03-06T15:38:06.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:06.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:06.047967+0000) 2024-03-06T15:38:07.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:07.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:07.048226+0000) 2024-03-06T15:38:08.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:08.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:08.048469+0000) 2024-03-06T15:38:09.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:09.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:09.048756+0000) 2024-03-06T15:38:10.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:10.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:10.049001+0000) 2024-03-06T15:38:11.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:11.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:11.049260+0000) 2024-03-06T15:38:12.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:12.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:12.049517+0000) 2024-03-06T15:38:13.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:13.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:13.049765+0000) 2024-03-06T15:38:14.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:14.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:14.050015+0000) 2024-03-06T15:38:15.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:15.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:15.050264+0000) 2024-03-06T15:38:16.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:16.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:16.050537+0000) 2024-03-06T15:38:16.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:16.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:17.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:17.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:17.050800+0000) 2024-03-06T15:38:18.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:18.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:18.051047+0000) 2024-03-06T15:38:19.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:19.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:19.051304+0000) 2024-03-06T15:38:20.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:20.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:20.051559+0000) 2024-03-06T15:38:21.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:21.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:21.051816+0000) 2024-03-06T15:38:22.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:22.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:22.052080+0000) 2024-03-06T15:38:23.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:23.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:23.052355+0000) 2024-03-06T15:38:24.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:24.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:24.052609+0000) 2024-03-06T15:38:25.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:25.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:25.052891+0000) 2024-03-06T15:38:26.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:26.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:26.053168+0000) 2024-03-06T15:38:27.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:27.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:27.053479+0000) 2024-03-06T15:38:28.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:28.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:28.053796+0000) 2024-03-06T15:38:29.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:29.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:29.054091+0000) 2024-03-06T15:38:30.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:30.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:30.054398+0000) 2024-03-06T15:38:31.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:31.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:31.054747+0000) 2024-03-06T15:38:32.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:32.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:32.055072+0000) 2024-03-06T15:38:33.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:33.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:33.055385+0000) 2024-03-06T15:38:34.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:34.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:34.055735+0000) 2024-03-06T15:38:35.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:35.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:35.055978+0000) 2024-03-06T15:38:36.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:36.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:36.056219+0000) 2024-03-06T15:38:37.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:37.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:37.056496+0000) 2024-03-06T15:38:38.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:38.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:38.056798+0000) 2024-03-06T15:38:39.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:39.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:39.057085+0000) 2024-03-06T15:38:40.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:40.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:40.057366+0000) 2024-03-06T15:38:41.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:41.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:41.057687+0000) 2024-03-06T15:38:42.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:42.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:42.057950+0000) 2024-03-06T15:38:43.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:43.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:43.058223+0000) 2024-03-06T15:38:44.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:44.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:44.058476+0000) 2024-03-06T15:38:45.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:45.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:45.058733+0000) 2024-03-06T15:38:46.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:46.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:46.058965+0000) 2024-03-06T15:38:46.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:46.659 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:47.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:47.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:47.059225+0000) 2024-03-06T15:38:48.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:48.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:48.059491+0000) 2024-03-06T15:38:49.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:49.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:49.059789+0000) 2024-03-06T15:38:50.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:50.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:50.060106+0000) 2024-03-06T15:38:51.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:51.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:51.060367+0000) 2024-03-06T15:38:52.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:52.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:52.060664+0000) 2024-03-06T15:38:53.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:53.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:53.060981+0000) 2024-03-06T15:38:54.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:54.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:54.061232+0000) 2024-03-06T15:38:55.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:55.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:55.061494+0000) 2024-03-06T15:38:56.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:56.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:56.061747+0000) 2024-03-06T15:38:57.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:57.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:57.062002+0000) 2024-03-06T15:38:58.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:58.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:58.062251+0000) 2024-03-06T15:38:59.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:38:59.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:59.062508+0000) 2024-03-06T15:39:00.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:00.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:00.062747+0000) 2024-03-06T15:39:01.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:01.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:01.062941+0000) 2024-03-06T15:39:02.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:02.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:02.063203+0000) 2024-03-06T15:39:03.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:03.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:03.063439+0000) 2024-03-06T15:39:04.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:04.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:04.063718+0000) 2024-03-06T15:39:05.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:05.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:05.063990+0000) 2024-03-06T15:39:06.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:06.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:06.064236+0000) 2024-03-06T15:39:07.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:07.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:07.064475+0000) 2024-03-06T15:39:08.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:08.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:08.064746+0000) 2024-03-06T15:39:09.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:09.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:09.065004+0000) 2024-03-06T15:39:10.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:10.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:10.065199+0000) 2024-03-06T15:39:11.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:11.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:11.065436+0000) 2024-03-06T15:39:12.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:12.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:12.065723+0000) 2024-03-06T15:39:13.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:13.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:13.065982+0000) 2024-03-06T15:39:14.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:14.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:14.066225+0000) 2024-03-06T15:39:15.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:15.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:15.066413+0000) 2024-03-06T15:39:16.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:16.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:16.066703+0000) 2024-03-06T15:39:16.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:16.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:17.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:17.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:17.066976+0000) 2024-03-06T15:39:18.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:18.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:18.067218+0000) 2024-03-06T15:39:19.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:19.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:19.067468+0000) 2024-03-06T15:39:20.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:20.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:20.067756+0000) 2024-03-06T15:39:21.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:21.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:21.067997+0000) 2024-03-06T15:39:22.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:22.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:22.068253+0000) 2024-03-06T15:39:23.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:23.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:23.068512+0000) 2024-03-06T15:39:24.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:24.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:24.068744+0000) 2024-03-06T15:39:25.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:25.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:25.069010+0000) 2024-03-06T15:39:26.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:26.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:26.069249+0000) 2024-03-06T15:39:27.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:27.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:27.069502+0000) 2024-03-06T15:39:28.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:28.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:28.069749+0000) 2024-03-06T15:39:29.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:29.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:29.069987+0000) 2024-03-06T15:39:30.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:30.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:30.070249+0000) 2024-03-06T15:39:31.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:31.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:31.070490+0000) 2024-03-06T15:39:32.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:32.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:32.070762+0000) 2024-03-06T15:39:33.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:33.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:33.070987+0000) 2024-03-06T15:39:34.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:34.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:34.071209+0000) 2024-03-06T15:39:35.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:35.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:35.071464+0000) 2024-03-06T15:39:36.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:36.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:36.071804+0000) 2024-03-06T15:39:37.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:37.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:37.072058+0000) 2024-03-06T15:39:38.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:38.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:38.072310+0000) 2024-03-06T15:39:39.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:39.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:39.072552+0000) 2024-03-06T15:39:40.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:40.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:40.072808+0000) 2024-03-06T15:39:41.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:41.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:41.073049+0000) 2024-03-06T15:39:42.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:42.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:42.073292+0000) 2024-03-06T15:39:43.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:43.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:43.073515+0000) 2024-03-06T15:39:44.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:44.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:44.073756+0000) 2024-03-06T15:39:45.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:45.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:45.073994+0000) 2024-03-06T15:39:46.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:46.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:46.074228+0000) 2024-03-06T15:39:46.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:46.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:47.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:47.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:47.074511+0000) 2024-03-06T15:39:48.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:48.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:48.074743+0000) 2024-03-06T15:39:49.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:49.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:49.074977+0000) 2024-03-06T15:39:50.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:50.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:50.075220+0000) 2024-03-06T15:39:51.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:51.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:51.075460+0000) 2024-03-06T15:39:52.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:52.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:52.075760+0000) 2024-03-06T15:39:53.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:53.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:53.076014+0000) 2024-03-06T15:39:54.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:54.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:54.076245+0000) 2024-03-06T15:39:55.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:55.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:55.076496+0000) 2024-03-06T15:39:56.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:56.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:56.076738+0000) 2024-03-06T15:39:57.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:57.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:57.076976+0000) 2024-03-06T15:39:58.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:58.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:58.077234+0000) 2024-03-06T15:39:59.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:39:59.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:59.077489+0000) 2024-03-06T15:40:00.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:00.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:00.077710+0000) 2024-03-06T15:40:01.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:01.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:01.077975+0000) 2024-03-06T15:40:02.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:02.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:02.078237+0000) 2024-03-06T15:40:03.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:03.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:03.078468+0000) 2024-03-06T15:40:04.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:04.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:04.078751+0000) 2024-03-06T15:40:05.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:05.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:05.078992+0000) 2024-03-06T15:40:06.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:06.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:06.079222+0000) 2024-03-06T15:40:07.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:07.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:07.079476+0000) 2024-03-06T15:40:08.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:08.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:08.079739+0000) 2024-03-06T15:40:09.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:09.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:09.079996+0000) 2024-03-06T15:40:10.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:10.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:10.080249+0000) 2024-03-06T15:40:11.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:11.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:11.080507+0000) 2024-03-06T15:40:12.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:12.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:12.080768+0000) 2024-03-06T15:40:13.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:13.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:13.081006+0000) 2024-03-06T15:40:14.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:14.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:14.081264+0000) 2024-03-06T15:40:15.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:15.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:15.081529+0000) 2024-03-06T15:40:16.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:16.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:16.081758+0000) 2024-03-06T15:40:17.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:17.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:17.082010+0000) 2024-03-06T15:40:17.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:17.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:18.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:18.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:18.082262+0000) 2024-03-06T15:40:19.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:19.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:19.082528+0000) 2024-03-06T15:40:20.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:20.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:20.082783+0000) 2024-03-06T15:40:21.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:21.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:21.083022+0000) 2024-03-06T15:40:22.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:22.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:22.083282+0000) 2024-03-06T15:40:23.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:23.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:23.083526+0000) 2024-03-06T15:40:24.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:24.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:24.083793+0000) 2024-03-06T15:40:25.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:25.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:25.084026+0000) 2024-03-06T15:40:26.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:26.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:26.084260+0000) 2024-03-06T15:40:27.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:27.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:27.084496+0000) 2024-03-06T15:40:28.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:28.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:28.084753+0000) 2024-03-06T15:40:29.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:29.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:29.085070+0000) 2024-03-06T15:40:30.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:30.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:30.085326+0000) 2024-03-06T15:40:31.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:31.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:31.085585+0000) 2024-03-06T15:40:32.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:32.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:32.085836+0000) 2024-03-06T15:40:33.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:33.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:33.086077+0000) 2024-03-06T15:40:34.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:34.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:34.086333+0000) 2024-03-06T15:40:35.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:35.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:35.086588+0000) 2024-03-06T15:40:36.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:36.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:36.086839+0000) 2024-03-06T15:40:37.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:37.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:37.087092+0000) 2024-03-06T15:40:38.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:38.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:38.087325+0000) 2024-03-06T15:40:39.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:39.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:39.087589+0000) 2024-03-06T15:40:40.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:40.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:40.087846+0000) 2024-03-06T15:40:41.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:41.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:41.088100+0000) 2024-03-06T15:40:42.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:42.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:42.088367+0000) 2024-03-06T15:40:43.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:43.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:43.088624+0000) 2024-03-06T15:40:44.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:44.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:44.088916+0000) 2024-03-06T15:40:45.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:45.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:45.089113+0000) 2024-03-06T15:40:46.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:46.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:46.089346+0000) 2024-03-06T15:40:47.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:47.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:47.089588+0000) 2024-03-06T15:40:47.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:47.272 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:48.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:48.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:48.089842+0000) 2024-03-06T15:40:49.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:49.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:49.090083+0000) 2024-03-06T15:40:50.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:50.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:50.090340+0000) 2024-03-06T15:40:51.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:51.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:51.090583+0000) 2024-03-06T15:40:52.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:52.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:52.090822+0000) 2024-03-06T15:40:53.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:53.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:53.091075+0000) 2024-03-06T15:40:54.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:54.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:54.091335+0000) 2024-03-06T15:40:55.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:55.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:55.091575+0000) 2024-03-06T15:40:56.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:56.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:56.091834+0000) 2024-03-06T15:40:57.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:57.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:57.092067+0000) 2024-03-06T15:40:58.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:58.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:58.092315+0000) 2024-03-06T15:40:59.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:40:59.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:59.092553+0000) 2024-03-06T15:41:00.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:00.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:00.092808+0000) 2024-03-06T15:41:01.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:01.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:01.093040+0000) 2024-03-06T15:41:02.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:02.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:02.093272+0000) 2024-03-06T15:41:03.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:03.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:03.093509+0000) 2024-03-06T15:41:04.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:04.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:04.093755+0000) 2024-03-06T15:41:05.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:05.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:05.093991+0000) 2024-03-06T15:41:06.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:06.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:06.094223+0000) 2024-03-06T15:41:07.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:07.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:07.094483+0000) 2024-03-06T15:41:08.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:08.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:08.094754+0000) 2024-03-06T15:41:09.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:09.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:09.095002+0000) 2024-03-06T15:41:10.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:10.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:10.095271+0000) 2024-03-06T15:41:11.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:11.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:11.095518+0000) 2024-03-06T15:41:12.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:12.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:12.095775+0000) 2024-03-06T15:41:13.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:13.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:13.096030+0000) 2024-03-06T15:41:14.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:14.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:14.096213+0000) 2024-03-06T15:41:15.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:15.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:15.096474+0000) 2024-03-06T15:41:16.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:16.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:16.096744+0000) 2024-03-06T15:41:17.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:17.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:17.097050+0000) 2024-03-06T15:41:17.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:17.420 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:18.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:18.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:18.097306+0000) 2024-03-06T15:41:19.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:19.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:19.097546+0000) 2024-03-06T15:41:20.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:20.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:20.097798+0000) 2024-03-06T15:41:21.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:21.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:21.098035+0000) 2024-03-06T15:41:22.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:22.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:22.098299+0000) 2024-03-06T15:41:23.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:23.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:23.098562+0000) 2024-03-06T15:41:24.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:24.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:24.098803+0000) 2024-03-06T15:41:25.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:25.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:25.099066+0000) 2024-03-06T15:41:26.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:26.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:26.099301+0000) 2024-03-06T15:41:27.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:27.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:27.099486+0000) 2024-03-06T15:41:28.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:28.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:28.099737+0000) 2024-03-06T15:41:29.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:29.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:29.099974+0000) 2024-03-06T15:41:30.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:30.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:30.100234+0000) 2024-03-06T15:41:31.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:31.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:31.100487+0000) 2024-03-06T15:41:32.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:32.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:32.100730+0000) 2024-03-06T15:41:33.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:33.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:33.100964+0000) 2024-03-06T15:41:34.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:34.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:34.101196+0000) 2024-03-06T15:41:35.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:35.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:35.101447+0000) 2024-03-06T15:41:36.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:36.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:36.101736+0000) 2024-03-06T15:41:37.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:37.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:37.101992+0000) 2024-03-06T15:41:38.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:38.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:38.102228+0000) 2024-03-06T15:41:39.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:39.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:39.102474+0000) 2024-03-06T15:41:40.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:40.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:40.102742+0000) 2024-03-06T15:41:41.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:41.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:41.103004+0000) 2024-03-06T15:41:42.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:42.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:42.103198+0000) 2024-03-06T15:41:43.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:43.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:43.103436+0000) 2024-03-06T15:41:44.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:44.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:44.103738+0000) 2024-03-06T15:41:45.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:45.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:45.104001+0000) 2024-03-06T15:41:46.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:46.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:46.104248+0000) 2024-03-06T15:41:47.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:47.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:47.104509+0000) 2024-03-06T15:41:47.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:47.558 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:48.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:48.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:48.104760+0000) 2024-03-06T15:41:49.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:49.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:49.105018+0000) 2024-03-06T15:41:50.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:50.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:50.105257+0000) 2024-03-06T15:41:51.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:51.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:51.105501+0000) 2024-03-06T15:41:52.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:52.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:52.105761+0000) 2024-03-06T15:41:53.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:53.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:53.106018+0000) 2024-03-06T15:41:54.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:54.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:54.106276+0000) 2024-03-06T15:41:55.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:55.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:55.106549+0000) 2024-03-06T15:41:56.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:56.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:56.106785+0000) 2024-03-06T15:41:57.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:57.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:57.106977+0000) 2024-03-06T15:41:58.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:58.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:58.107213+0000) 2024-03-06T15:41:59.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:41:59.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:59.107476+0000) 2024-03-06T15:42:00.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:00.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:00.107760+0000) 2024-03-06T15:42:01.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:01.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:01.107991+0000) 2024-03-06T15:42:02.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:02.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:02.108237+0000) 2024-03-06T15:42:03.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:03.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:03.108476+0000) 2024-03-06T15:42:04.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:04.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:04.108760+0000) 2024-03-06T15:42:05.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:05.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:05.109075+0000) 2024-03-06T15:42:06.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:06.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:06.109310+0000) 2024-03-06T15:42:07.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:07.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:07.109535+0000) 2024-03-06T15:42:08.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:08.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:08.109783+0000) 2024-03-06T15:42:09.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:09.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:09.110041+0000) 2024-03-06T15:42:10.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:10.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:10.110311+0000) 2024-03-06T15:42:11.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:11.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:11.110561+0000) 2024-03-06T15:42:12.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:12.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:12.110803+0000) 2024-03-06T15:42:13.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:13.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:13.110989+0000) 2024-03-06T15:42:14.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:14.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:14.111252+0000) 2024-03-06T15:42:15.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:15.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:15.111512+0000) 2024-03-06T15:42:16.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:16.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:16.111766+0000) 2024-03-06T15:42:17.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:17.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:17.112028+0000) 2024-03-06T15:42:17.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:17.714 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:18.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:18.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:18.112284+0000) 2024-03-06T15:42:19.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:19.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:19.112546+0000) 2024-03-06T15:42:20.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:20.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:20.112796+0000) 2024-03-06T15:42:21.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:21.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:21.113037+0000) 2024-03-06T15:42:22.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:22.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:22.113304+0000) 2024-03-06T15:42:23.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:23.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:23.113556+0000) 2024-03-06T15:42:24.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:24.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:24.113803+0000) 2024-03-06T15:42:25.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:25.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:25.114060+0000) 2024-03-06T15:42:26.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:26.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:26.114297+0000) 2024-03-06T15:42:27.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:27.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:27.114564+0000) 2024-03-06T15:42:28.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:28.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:28.114805+0000) 2024-03-06T15:42:29.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:29.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:29.115035+0000) 2024-03-06T15:42:30.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:30.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:30.115273+0000) 2024-03-06T15:42:31.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:31.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:31.115466+0000) 2024-03-06T15:42:32.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:32.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:32.115750+0000) 2024-03-06T15:42:33.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:33.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:33.115993+0000) 2024-03-06T15:42:34.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:34.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:34.116236+0000) 2024-03-06T15:42:35.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:35.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:35.116486+0000) 2024-03-06T15:42:36.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:36.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:36.116738+0000) 2024-03-06T15:42:37.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:37.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:37.116993+0000) 2024-03-06T15:42:38.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:38.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:38.117245+0000) 2024-03-06T15:42:39.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:39.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:39.117510+0000) 2024-03-06T15:42:40.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:40.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:40.117773+0000) 2024-03-06T15:42:41.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:41.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:41.118033+0000) 2024-03-06T15:42:42.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:42.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:42.118306+0000) 2024-03-06T15:42:43.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:43.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:43.118547+0000) 2024-03-06T15:42:44.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:44.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:44.118806+0000) 2024-03-06T15:42:45.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:45.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:45.119076+0000) 2024-03-06T15:42:46.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:46.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:46.119326+0000) 2024-03-06T15:42:47.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:47.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:47.119582+0000) 2024-03-06T15:42:47.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:47.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:48.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:48.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:48.119825+0000) 2024-03-06T15:42:49.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:49.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:49.120094+0000) 2024-03-06T15:42:50.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:50.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:50.120349+0000) 2024-03-06T15:42:51.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:51.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:51.120585+0000) 2024-03-06T15:42:52.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:52.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:52.120854+0000) 2024-03-06T15:42:53.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:53.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:53.121101+0000) 2024-03-06T15:42:54.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:54.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:54.121293+0000) 2024-03-06T15:42:55.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:55.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:55.121550+0000) 2024-03-06T15:42:56.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:56.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:56.121809+0000) 2024-03-06T15:42:57.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:57.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:57.122036+0000) 2024-03-06T15:42:58.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:58.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:58.122264+0000) 2024-03-06T15:42:59.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:42:59.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:59.122588+0000) 2024-03-06T15:43:00.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:00.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:00.122835+0000) 2024-03-06T15:43:01.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:01.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:01.123070+0000) 2024-03-06T15:43:02.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:02.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:02.123268+0000) 2024-03-06T15:43:03.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:03.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:03.123508+0000) 2024-03-06T15:43:04.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:04.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:04.123761+0000) 2024-03-06T15:43:05.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:05.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:05.124008+0000) 2024-03-06T15:43:06.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:06.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:06.124270+0000) 2024-03-06T15:43:07.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:07.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:07.124526+0000) 2024-03-06T15:43:08.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:08.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:08.124770+0000) 2024-03-06T15:43:09.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:09.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:09.125008+0000) 2024-03-06T15:43:10.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:10.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:10.125264+0000) 2024-03-06T15:43:11.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:11.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:11.125505+0000) 2024-03-06T15:43:12.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:12.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:12.125759+0000) 2024-03-06T15:43:13.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:13.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:13.126011+0000) 2024-03-06T15:43:14.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:14.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:14.126279+0000) 2024-03-06T15:43:15.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:15.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:15.126521+0000) 2024-03-06T15:43:16.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:16.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:16.126779+0000) 2024-03-06T15:43:17.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:17.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:17.126976+0000) 2024-03-06T15:43:18.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:18.025 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:18.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:18.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:18.127177+0000) 2024-03-06T15:43:19.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:19.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:19.127444+0000) 2024-03-06T15:43:20.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:20.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:20.127666+0000) 2024-03-06T15:43:21.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:21.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:21.127927+0000) 2024-03-06T15:43:22.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:22.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:22.128160+0000) 2024-03-06T15:43:23.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:23.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:23.128418+0000) 2024-03-06T15:43:24.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:24.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:24.128670+0000) 2024-03-06T15:43:25.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:25.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:25.128919+0000) 2024-03-06T15:43:26.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:26.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:26.129154+0000) 2024-03-06T15:43:27.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:27.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:27.129413+0000) 2024-03-06T15:43:28.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:28.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:28.129675+0000) 2024-03-06T15:43:29.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:29.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:29.129928+0000) 2024-03-06T15:43:30.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:30.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:30.130161+0000) 2024-03-06T15:43:31.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:31.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:31.130409+0000) 2024-03-06T15:43:32.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:32.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:32.130690+0000) 2024-03-06T15:43:33.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:33.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:33.130876+0000) 2024-03-06T15:43:34.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:34.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:34.131116+0000) 2024-03-06T15:43:35.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:35.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:35.131378+0000) 2024-03-06T15:43:36.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:36.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:36.131667+0000) 2024-03-06T15:43:37.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:37.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:37.131926+0000) 2024-03-06T15:43:38.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:38.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:38.132182+0000) 2024-03-06T15:43:39.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:39.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:39.132437+0000) 2024-03-06T15:43:40.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:40.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:40.132779+0000) 2024-03-06T15:43:41.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:41.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:41.133048+0000) 2024-03-06T15:43:42.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:42.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:42.133316+0000) 2024-03-06T15:43:43.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:43.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:43.133508+0000) 2024-03-06T15:43:44.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:44.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:44.133764+0000) 2024-03-06T15:43:45.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:45.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:45.134022+0000) 2024-03-06T15:43:46.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:46.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:46.134258+0000) 2024-03-06T15:43:47.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:47.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:47.134508+0000) 2024-03-06T15:43:48.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:48.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:48.134756+0000) 2024-03-06T15:43:48.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:48.218 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:49.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:49.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:49.135014+0000) 2024-03-06T15:43:50.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:50.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:50.135275+0000) 2024-03-06T15:43:51.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:51.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:51.135459+0000) 2024-03-06T15:43:52.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:52.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:52.135733+0000) 2024-03-06T15:43:53.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:53.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:53.135964+0000) 2024-03-06T15:43:54.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:54.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:54.136218+0000) 2024-03-06T15:43:55.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:55.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:55.136476+0000) 2024-03-06T15:43:56.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:56.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:56.136744+0000) 2024-03-06T15:43:57.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:57.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:57.136979+0000) 2024-03-06T15:43:58.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:58.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:58.137210+0000) 2024-03-06T15:43:59.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:43:59.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:59.137466+0000) 2024-03-06T15:44:00.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:00.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:00.137696+0000) 2024-03-06T15:44:01.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:01.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:01.137953+0000) 2024-03-06T15:44:02.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:02.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:02.138198+0000) 2024-03-06T15:44:03.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:03.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:03.138454+0000) 2024-03-06T15:44:04.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:04.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:04.138732+0000) 2024-03-06T15:44:05.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:05.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:05.138934+0000) 2024-03-06T15:44:06.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:06.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:06.139196+0000) 2024-03-06T15:44:07.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:07.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:07.139453+0000) 2024-03-06T15:44:08.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:08.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:08.139746+0000) 2024-03-06T15:44:09.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:09.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:09.140005+0000) 2024-03-06T15:44:10.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:10.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:10.140257+0000) 2024-03-06T15:44:11.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:11.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:11.140524+0000) 2024-03-06T15:44:12.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:12.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:12.140785+0000) 2024-03-06T15:44:13.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:13.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:13.140983+0000) 2024-03-06T15:44:14.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:14.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:14.141229+0000) 2024-03-06T15:44:15.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:15.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:15.141472+0000) 2024-03-06T15:44:16.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:16.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:16.141738+0000) 2024-03-06T15:44:17.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:17.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:17.141999+0000) 2024-03-06T15:44:18.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:18.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:18.142247+0000) 2024-03-06T15:44:18.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:18.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:19.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:19.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:19.142488+0000) 2024-03-06T15:44:20.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:20.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:20.142754+0000) 2024-03-06T15:44:21.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:21.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:21.142991+0000) 2024-03-06T15:44:22.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:22.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:22.143248+0000) 2024-03-06T15:44:23.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:23.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:23.143484+0000) 2024-03-06T15:44:24.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:24.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:24.143745+0000) 2024-03-06T15:44:25.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:25.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:25.143985+0000) 2024-03-06T15:44:26.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:26.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:26.144250+0000) 2024-03-06T15:44:27.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:27.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:27.144508+0000) 2024-03-06T15:44:28.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:28.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:28.144758+0000) 2024-03-06T15:44:29.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:29.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:29.144998+0000) 2024-03-06T15:44:30.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:30.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:30.145246+0000) 2024-03-06T15:44:31.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:31.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:31.145484+0000) 2024-03-06T15:44:32.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:32.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:32.145728+0000) 2024-03-06T15:44:33.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:33.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:33.145992+0000) 2024-03-06T15:44:34.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:34.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:34.146236+0000) 2024-03-06T15:44:35.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:35.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:35.146542+0000) 2024-03-06T15:44:36.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:36.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:36.146781+0000) 2024-03-06T15:44:37.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:37.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:37.147024+0000) 2024-03-06T15:44:38.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:38.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:38.147265+0000) 2024-03-06T15:44:39.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:39.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:39.147518+0000) 2024-03-06T15:44:40.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:40.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:40.147786+0000) 2024-03-06T15:44:41.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:41.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:41.147981+0000) 2024-03-06T15:44:42.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:42.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:42.148224+0000) 2024-03-06T15:44:43.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:43.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:43.148479+0000) 2024-03-06T15:44:44.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:44.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:44.148740+0000) 2024-03-06T15:44:45.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:45.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:45.148984+0000) 2024-03-06T15:44:46.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:46.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:46.149239+0000) 2024-03-06T15:44:47.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:47.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:47.149493+0000) 2024-03-06T15:44:48.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:48.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:48.149744+0000) 2024-03-06T15:44:48.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:48.508 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:49.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:49.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:49.150007+0000) 2024-03-06T15:44:50.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:50.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:50.150273+0000) 2024-03-06T15:44:51.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:51.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:51.150528+0000) 2024-03-06T15:44:52.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:52.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:52.150793+0000) 2024-03-06T15:44:53.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:53.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:53.151029+0000) 2024-03-06T15:44:54.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:54.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:54.151204+0000) 2024-03-06T15:44:55.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:55.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:55.151459+0000) 2024-03-06T15:44:56.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:56.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:56.151769+0000) 2024-03-06T15:44:57.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:57.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:57.152031+0000) 2024-03-06T15:44:58.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:58.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:58.152280+0000) 2024-03-06T15:44:59.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:44:59.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:59.152521+0000) 2024-03-06T15:45:00.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:00.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:00.152780+0000) 2024-03-06T15:45:01.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:01.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:01.153023+0000) 2024-03-06T15:45:02.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:02.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:02.153262+0000) 2024-03-06T15:45:03.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:03.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:03.153439+0000) 2024-03-06T15:45:04.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:04.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:04.153731+0000) 2024-03-06T15:45:05.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:05.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:05.154177+0000) 2024-03-06T15:45:06.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:06.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:06.154438+0000) 2024-03-06T15:45:07.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:07.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:07.154742+0000) 2024-03-06T15:45:08.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:08.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:08.154992+0000) 2024-03-06T15:45:09.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:09.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:09.155245+0000) 2024-03-06T15:45:10.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:10.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:10.155483+0000) 2024-03-06T15:45:11.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:11.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:11.155748+0000) 2024-03-06T15:45:12.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:12.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:12.156010+0000) 2024-03-06T15:45:13.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:13.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:13.156227+0000) 2024-03-06T15:45:14.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:14.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:14.156477+0000) 2024-03-06T15:45:15.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:15.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:15.156741+0000) 2024-03-06T15:45:16.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:16.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:16.156987+0000) 2024-03-06T15:45:17.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:17.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:17.157186+0000) 2024-03-06T15:45:18.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:18.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:18.157426+0000) 2024-03-06T15:45:18.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:18.664 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:19.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:19.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:19.157740+0000) 2024-03-06T15:45:20.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:20.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:20.157976+0000) 2024-03-06T15:45:21.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:21.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:21.158237+0000) 2024-03-06T15:45:22.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:22.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:22.158523+0000) 2024-03-06T15:45:23.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:23.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:23.158922+0000) 2024-03-06T15:45:24.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:24.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:24.159173+0000) 2024-03-06T15:45:25.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:25.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:25.159429+0000) 2024-03-06T15:45:26.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:26.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:26.159887+0000) 2024-03-06T15:45:27.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:27.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:27.160153+0000) 2024-03-06T15:45:28.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:28.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:28.160439+0000) 2024-03-06T15:45:29.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:29.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:29.160773+0000) 2024-03-06T15:45:30.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:30.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:30.161165+0000) 2024-03-06T15:45:31.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:31.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:31.161389+0000) 2024-03-06T15:45:32.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:32.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:32.161662+0000) 2024-03-06T15:45:33.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:33.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:33.161905+0000) 2024-03-06T15:45:34.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:34.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:34.162162+0000) 2024-03-06T15:45:35.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:35.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:35.162336+0000) 2024-03-06T15:45:36.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:36.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:36.162572+0000) 2024-03-06T15:45:37.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:37.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:37.162830+0000) 2024-03-06T15:45:38.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:38.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:38.163018+0000) 2024-03-06T15:45:39.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:39.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:39.163355+0000) 2024-03-06T15:45:40.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:40.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:40.163620+0000) 2024-03-06T15:45:41.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:41.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:41.163929+0000) 2024-03-06T15:45:42.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:42.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:42.164120+0000) 2024-03-06T15:45:43.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:43.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:43.164372+0000) 2024-03-06T15:45:44.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:44.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:44.164621+0000) 2024-03-06T15:45:45.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:45.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:45.164921+0000) 2024-03-06T15:45:46.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:46.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:46.165188+0000) 2024-03-06T15:45:47.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:47.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:47.165430+0000) 2024-03-06T15:45:48.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:48.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:48.165695+0000) 2024-03-06T15:45:48.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:48.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:49.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:49.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:49.165953+0000) 2024-03-06T15:45:50.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:50.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:50.166151+0000) 2024-03-06T15:45:51.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:51.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:51.166410+0000) 2024-03-06T15:45:52.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:52.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:52.166708+0000) 2024-03-06T15:45:53.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:53.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:53.166956+0000) 2024-03-06T15:45:54.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:54.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:54.167193+0000) 2024-03-06T15:45:55.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:55.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:55.167434+0000) 2024-03-06T15:45:56.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:56.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:56.167730+0000) 2024-03-06T15:45:57.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:57.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:57.167968+0000) 2024-03-06T15:45:58.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:58.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:58.168201+0000) 2024-03-06T15:45:59.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:45:59.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:59.168461+0000) 2024-03-06T15:46:00.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:00.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:00.168757+0000) 2024-03-06T15:46:01.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:01.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:01.169018+0000) 2024-03-06T15:46:02.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:02.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:02.169279+0000) 2024-03-06T15:46:03.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:03.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:03.169525+0000) 2024-03-06T15:46:04.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:04.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:04.169760+0000) 2024-03-06T15:46:05.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:05.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:05.169999+0000) 2024-03-06T15:46:06.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:06.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:06.170263+0000) 2024-03-06T15:46:07.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:07.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:07.170505+0000) 2024-03-06T15:46:08.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:08.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:08.170783+0000) 2024-03-06T15:46:09.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:09.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:09.171018+0000) 2024-03-06T15:46:10.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:10.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:10.171279+0000) 2024-03-06T15:46:11.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:11.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:11.171556+0000) 2024-03-06T15:46:12.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:12.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:12.171833+0000) 2024-03-06T15:46:13.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:13.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:13.172069+0000) 2024-03-06T15:46:14.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:14.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:14.172304+0000) 2024-03-06T15:46:15.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:15.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:15.172561+0000) 2024-03-06T15:46:16.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:16.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:16.172879+0000) 2024-03-06T15:46:17.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:17.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:17.173073+0000) 2024-03-06T15:46:18.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:18.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:18.173310+0000) 2024-03-06T15:46:19.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:19.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:19.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:19.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:19.173524+0000) 2024-03-06T15:46:20.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:20.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:20.173773+0000) 2024-03-06T15:46:21.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:21.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:21.174037+0000) 2024-03-06T15:46:22.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:22.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:22.174294+0000) 2024-03-06T15:46:23.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:23.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:23.174537+0000) 2024-03-06T15:46:24.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:24.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:24.174794+0000) 2024-03-06T15:46:25.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:25.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:25.175049+0000) 2024-03-06T15:46:26.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:26.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:26.175309+0000) 2024-03-06T15:46:27.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:27.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:27.175553+0000) 2024-03-06T15:46:28.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:28.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:28.175799+0000) 2024-03-06T15:46:29.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:29.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:29.176050+0000) 2024-03-06T15:46:30.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:30.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:30.176310+0000) 2024-03-06T15:46:31.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:31.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:31.176496+0000) 2024-03-06T15:46:32.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:32.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:32.176740+0000) 2024-03-06T15:46:33.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:33.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:33.176978+0000) 2024-03-06T15:46:34.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:34.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:34.177215+0000) 2024-03-06T15:46:35.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:35.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:35.177469+0000) 2024-03-06T15:46:36.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:36.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:36.177751+0000) 2024-03-06T15:46:37.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:37.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:37.178009+0000) 2024-03-06T15:46:38.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:38.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:38.178258+0000) 2024-03-06T15:46:39.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:39.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:39.178496+0000) 2024-03-06T15:46:40.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:40.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:40.178753+0000) 2024-03-06T15:46:41.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:41.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:41.179000+0000) 2024-03-06T15:46:42.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:42.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:42.179189+0000) 2024-03-06T15:46:43.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:43.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:43.179444+0000) 2024-03-06T15:46:44.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:44.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:44.179722+0000) 2024-03-06T15:46:45.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:45.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:45.179961+0000) 2024-03-06T15:46:46.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:46.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:46.180229+0000) 2024-03-06T15:46:47.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:47.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:47.180484+0000) 2024-03-06T15:46:48.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:48.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:48.180745+0000) 2024-03-06T15:46:49.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:49.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:49.180940+0000) 2024-03-06T15:46:49.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:49.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:50.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:50.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:50.181206+0000) 2024-03-06T15:46:51.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:51.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:51.181453+0000) 2024-03-06T15:46:52.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:52.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:52.181743+0000) 2024-03-06T15:46:53.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:53.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:53.181990+0000) 2024-03-06T15:46:54.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:54.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:54.182239+0000) 2024-03-06T15:46:55.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:55.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:55.182498+0000) 2024-03-06T15:46:56.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:56.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:56.182762+0000) 2024-03-06T15:46:57.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:57.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:57.183002+0000) 2024-03-06T15:46:58.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:58.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:58.183241+0000) 2024-03-06T15:46:59.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:46:59.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:59.183492+0000) 2024-03-06T15:47:00.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:00.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:00.183752+0000) 2024-03-06T15:47:01.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:01.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:01.184019+0000) 2024-03-06T15:47:02.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:02.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:02.184227+0000) 2024-03-06T15:47:03.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:03.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:03.184484+0000) 2024-03-06T15:47:04.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:04.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:04.184782+0000) 2024-03-06T15:47:05.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:05.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:05.185039+0000) 2024-03-06T15:47:06.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:06.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:06.185298+0000) 2024-03-06T15:47:07.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:07.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:07.185552+0000) 2024-03-06T15:47:08.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:08.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:08.185784+0000) 2024-03-06T15:47:09.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:09.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:09.186041+0000) 2024-03-06T15:47:10.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:10.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:10.186286+0000) 2024-03-06T15:47:11.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:11.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:11.186528+0000) 2024-03-06T15:47:12.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:12.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:12.186787+0000) 2024-03-06T15:47:13.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:13.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:13.187043+0000) 2024-03-06T15:47:14.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:14.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:14.187292+0000) 2024-03-06T15:47:15.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:15.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:15.187501+0000) 2024-03-06T15:47:16.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:16.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:16.187757+0000) 2024-03-06T15:47:17.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:17.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:17.188004+0000) 2024-03-06T15:47:18.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:18.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:18.188227+0000) 2024-03-06T15:47:19.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:19.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:19.188497+0000) 2024-03-06T15:47:19.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:19.389 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:20.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:20.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:20.188772+0000) 2024-03-06T15:47:21.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:21.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:21.189015+0000) 2024-03-06T15:47:22.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:22.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:22.189255+0000) 2024-03-06T15:47:23.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:23.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:23.189496+0000) 2024-03-06T15:47:24.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:24.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:24.189724+0000) 2024-03-06T15:47:25.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:25.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:25.189995+0000) 2024-03-06T15:47:26.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:26.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:26.190246+0000) 2024-03-06T15:47:27.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:27.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:27.190483+0000) 2024-03-06T15:47:28.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:28.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:28.190736+0000) 2024-03-06T15:47:29.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:29.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:29.190994+0000) 2024-03-06T15:47:30.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:30.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:30.191262+0000) 2024-03-06T15:47:31.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:31.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:31.191527+0000) 2024-03-06T15:47:32.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:32.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:32.191777+0000) 2024-03-06T15:47:33.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:33.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:33.192026+0000) 2024-03-06T15:47:34.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:34.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:34.192271+0000) 2024-03-06T15:47:35.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:35.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:35.192485+0000) 2024-03-06T15:47:36.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:36.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:36.192740+0000) 2024-03-06T15:47:37.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:37.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:37.192972+0000) 2024-03-06T15:47:38.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:38.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:38.193150+0000) 2024-03-06T15:47:39.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:39.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:39.193391+0000) 2024-03-06T15:47:40.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:40.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:40.193691+0000) 2024-03-06T15:47:41.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:41.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:41.193958+0000) 2024-03-06T15:47:42.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:42.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:42.194219+0000) 2024-03-06T15:47:43.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:43.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:43.194474+0000) 2024-03-06T15:47:44.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:44.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:44.194721+0000) 2024-03-06T15:47:45.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:45.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:45.194973+0000) 2024-03-06T15:47:46.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:46.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:46.195224+0000) 2024-03-06T15:47:47.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:47.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:47.195492+0000) 2024-03-06T15:47:48.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:48.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:48.195744+0000) 2024-03-06T15:47:49.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:49.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:49.196002+0000) 2024-03-06T15:47:49.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:49.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:50.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:50.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:50.196285+0000) 2024-03-06T15:47:51.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:51.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:51.196519+0000) 2024-03-06T15:47:52.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:52.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:52.196770+0000) 2024-03-06T15:47:53.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:53.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:53.197012+0000) 2024-03-06T15:47:54.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:54.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:54.197260+0000) 2024-03-06T15:47:55.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:55.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:55.197521+0000) 2024-03-06T15:47:56.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:56.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:56.197747+0000) 2024-03-06T15:47:57.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:57.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:57.198040+0000) 2024-03-06T15:47:58.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:58.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:58.198278+0000) 2024-03-06T15:47:59.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:47:59.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:59.198524+0000) 2024-03-06T15:48:00.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:00.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:00.198791+0000) 2024-03-06T15:48:01.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:01.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:01.199032+0000) 2024-03-06T15:48:02.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:02.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:02.199277+0000) 2024-03-06T15:48:03.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:03.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:03.199547+0000) 2024-03-06T15:48:04.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:04.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:04.199791+0000) 2024-03-06T15:48:05.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:05.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:05.199984+0000) 2024-03-06T15:48:06.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:06.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:06.200214+0000) 2024-03-06T15:48:07.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:07.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:07.200455+0000) 2024-03-06T15:48:08.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:08.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:08.200712+0000) 2024-03-06T15:48:09.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:09.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:09.200972+0000) 2024-03-06T15:48:10.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:10.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:10.201221+0000) 2024-03-06T15:48:11.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:11.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:11.201417+0000) 2024-03-06T15:48:12.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:12.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:12.201731+0000) 2024-03-06T15:48:13.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:13.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:13.201982+0000) 2024-03-06T15:48:14.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:14.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:14.202251+0000) 2024-03-06T15:48:15.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:15.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:15.202522+0000) 2024-03-06T15:48:16.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:16.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:16.202792+0000) 2024-03-06T15:48:17.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:17.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:17.203050+0000) 2024-03-06T15:48:18.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:18.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:18.203300+0000) 2024-03-06T15:48:19.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:19.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:19.203565+0000) 2024-03-06T15:48:19.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:19.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:20.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:20.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:20.203829+0000) 2024-03-06T15:48:21.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:21.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:21.204017+0000) 2024-03-06T15:48:22.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:22.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:22.204263+0000) 2024-03-06T15:48:23.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:23.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:23.204517+0000) 2024-03-06T15:48:24.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:24.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:24.204761+0000) 2024-03-06T15:48:25.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:25.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:25.204998+0000) 2024-03-06T15:48:26.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:26.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:26.205229+0000) 2024-03-06T15:48:27.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:27.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:27.205463+0000) 2024-03-06T15:48:28.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:28.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:28.205737+0000) 2024-03-06T15:48:29.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:29.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:29.205989+0000) 2024-03-06T15:48:30.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:30.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:30.206208+0000) 2024-03-06T15:48:31.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:31.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:31.206449+0000) 2024-03-06T15:48:32.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:32.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:32.206751+0000) 2024-03-06T15:48:33.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:33.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:33.206996+0000) 2024-03-06T15:48:34.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:34.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:34.207183+0000) 2024-03-06T15:48:35.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:35.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:35.207431+0000) 2024-03-06T15:48:36.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:36.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:36.207735+0000) 2024-03-06T15:48:37.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:37.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:37.207976+0000) 2024-03-06T15:48:38.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:38.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:38.208217+0000) 2024-03-06T15:48:39.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:39.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:39.208472+0000) 2024-03-06T15:48:40.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:40.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:40.208741+0000) 2024-03-06T15:48:41.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:41.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:41.209000+0000) 2024-03-06T15:48:42.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:42.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:42.209265+0000) 2024-03-06T15:48:43.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:43.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:43.209522+0000) 2024-03-06T15:48:44.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:44.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:44.209751+0000) 2024-03-06T15:48:45.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:45.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:45.210064+0000) 2024-03-06T15:48:46.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:46.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:46.210319+0000) 2024-03-06T15:48:47.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:47.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:47.210587+0000) 2024-03-06T15:48:48.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:48.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:48.210835+0000) 2024-03-06T15:48:49.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:49.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:49.211088+0000) 2024-03-06T15:48:49.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:49.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:50.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:50.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:50.211333+0000) 2024-03-06T15:48:51.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:51.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:51.211575+0000) 2024-03-06T15:48:52.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:52.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:52.211823+0000) 2024-03-06T15:48:53.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:53.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:53.212082+0000) 2024-03-06T15:48:54.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:54.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:54.212333+0000) 2024-03-06T15:48:55.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:55.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:55.212587+0000) 2024-03-06T15:48:56.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:56.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:56.212832+0000) 2024-03-06T15:48:57.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:57.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:57.213078+0000) 2024-03-06T15:48:58.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:58.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:58.213327+0000) 2024-03-06T15:48:59.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:48:59.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:59.213580+0000) 2024-03-06T15:49:00.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:00.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:00.213824+0000) 2024-03-06T15:49:01.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:01.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:01.214065+0000) 2024-03-06T15:49:02.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:02.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:02.214333+0000) 2024-03-06T15:49:03.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:03.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:03.214572+0000) 2024-03-06T15:49:04.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:04.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:04.214817+0000) 2024-03-06T15:49:05.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:05.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:05.215072+0000) 2024-03-06T15:49:06.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:06.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:06.215334+0000) 2024-03-06T15:49:07.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:07.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:07.215584+0000) 2024-03-06T15:49:08.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:08.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:08.215831+0000) 2024-03-06T15:49:09.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:09.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:09.216062+0000) 2024-03-06T15:49:10.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:10.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:10.216292+0000) 2024-03-06T15:49:11.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:11.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:11.216524+0000) 2024-03-06T15:49:12.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:12.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:12.216802+0000) 2024-03-06T15:49:13.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:13.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:13.217033+0000) 2024-03-06T15:49:14.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:14.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:14.217281+0000) 2024-03-06T15:49:15.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:15.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:15.217538+0000) 2024-03-06T15:49:16.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:16.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:16.217790+0000) 2024-03-06T15:49:17.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:17.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:17.218029+0000) 2024-03-06T15:49:18.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:18.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:18.218255+0000) 2024-03-06T15:49:19.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:19.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:19.218505+0000) 2024-03-06T15:49:19.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:19.960 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:20.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:20.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:20.218795+0000) 2024-03-06T15:49:21.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:21.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:21.219073+0000) 2024-03-06T15:49:22.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:22.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:22.219346+0000) 2024-03-06T15:49:23.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:23.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:23.219608+0000) 2024-03-06T15:49:24.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:24.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:24.219823+0000) 2024-03-06T15:49:25.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:25.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:25.220067+0000) 2024-03-06T15:49:26.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:26.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:26.220308+0000) 2024-03-06T15:49:27.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:27.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:27.220546+0000) 2024-03-06T15:49:28.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:28.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:28.220781+0000) 2024-03-06T15:49:29.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:29.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:29.220971+0000) 2024-03-06T15:49:30.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:30.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:30.221265+0000) 2024-03-06T15:49:31.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:31.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:31.221586+0000) 2024-03-06T15:49:32.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:32.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:32.221960+0000) 2024-03-06T15:49:33.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:33.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:33.222325+0000) 2024-03-06T15:49:34.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:34.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:34.222577+0000) 2024-03-06T15:49:35.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:35.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:35.222841+0000) 2024-03-06T15:49:36.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:36.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:36.223078+0000) 2024-03-06T15:49:37.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:37.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:37.223391+0000) 2024-03-06T15:49:38.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:38.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:38.223722+0000) 2024-03-06T15:49:39.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:39.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:39.223979+0000) 2024-03-06T15:49:40.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:40.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:40.224257+0000) 2024-03-06T15:49:41.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:41.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:41.224545+0000) 2024-03-06T15:49:42.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:42.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:42.224868+0000) 2024-03-06T15:49:43.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:43.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:43.225184+0000) 2024-03-06T15:49:44.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:44.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:44.225446+0000) 2024-03-06T15:49:45.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:45.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:45.225735+0000) 2024-03-06T15:49:46.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:46.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:46.225950+0000) 2024-03-06T15:49:47.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:47.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:47.226253+0000) 2024-03-06T15:49:48.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:48.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:48.226520+0000) 2024-03-06T15:49:49.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:49.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:49.226817+0000) 2024-03-06T15:49:50.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:50.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:50.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:50.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:50.227025+0000) 2024-03-06T15:49:51.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:51.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:51.227246+0000) 2024-03-06T15:49:52.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:52.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:52.227494+0000) 2024-03-06T15:49:53.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:53.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:53.227734+0000) 2024-03-06T15:49:54.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:54.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:54.227968+0000) 2024-03-06T15:49:55.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:55.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:55.228223+0000) 2024-03-06T15:49:56.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:56.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:56.228486+0000) 2024-03-06T15:49:57.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:57.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:57.228727+0000) 2024-03-06T15:49:58.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:58.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:58.228967+0000) 2024-03-06T15:49:59.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:49:59.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:59.229204+0000) 2024-03-06T15:50:00.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:00.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:00.229447+0000) 2024-03-06T15:50:01.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:01.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:01.229759+0000) 2024-03-06T15:50:02.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:02.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:02.230025+0000) 2024-03-06T15:50:03.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:03.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:03.230284+0000) 2024-03-06T15:50:04.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:04.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:04.230534+0000) 2024-03-06T15:50:05.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:05.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:05.230732+0000) 2024-03-06T15:50:06.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:06.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:06.230986+0000) 2024-03-06T15:50:07.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:07.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:07.231244+0000) 2024-03-06T15:50:08.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:08.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:08.231488+0000) 2024-03-06T15:50:09.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:09.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:09.231742+0000) 2024-03-06T15:50:10.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:10.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:10.231993+0000) 2024-03-06T15:50:11.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:11.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:11.232252+0000) 2024-03-06T15:50:12.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:12.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:12.232506+0000) 2024-03-06T15:50:13.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:13.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:13.232760+0000) 2024-03-06T15:50:14.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:14.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:14.232941+0000) 2024-03-06T15:50:15.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:15.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:15.233179+0000) 2024-03-06T15:50:16.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:16.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:16.233441+0000) 2024-03-06T15:50:17.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:17.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:17.233739+0000) 2024-03-06T15:50:18.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:18.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:18.233999+0000) 2024-03-06T15:50:19.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:19.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:19.234182+0000) 2024-03-06T15:50:20.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:20.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:20.234425+0000) 2024-03-06T15:50:20.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:20.244 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:21.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:21.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:21.234724+0000) 2024-03-06T15:50:22.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:22.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:22.234979+0000) 2024-03-06T15:50:23.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:23.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:23.235219+0000) 2024-03-06T15:50:24.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:24.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:24.235453+0000) 2024-03-06T15:50:25.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:25.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:25.235731+0000) 2024-03-06T15:50:26.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:26.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:26.235992+0000) 2024-03-06T15:50:27.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:27.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:27.236254+0000) 2024-03-06T15:50:28.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:28.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:28.236491+0000) 2024-03-06T15:50:29.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:29.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:29.236735+0000) 2024-03-06T15:50:30.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:30.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:30.236982+0000) 2024-03-06T15:50:31.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:31.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:31.237238+0000) 2024-03-06T15:50:32.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:32.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:32.237508+0000) 2024-03-06T15:50:33.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:33.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:33.237741+0000) 2024-03-06T15:50:34.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:34.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:34.237992+0000) 2024-03-06T15:50:35.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:35.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:35.238254+0000) 2024-03-06T15:50:36.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:36.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:36.238497+0000) 2024-03-06T15:50:37.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:37.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:37.238759+0000) 2024-03-06T15:50:38.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:38.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:38.239004+0000) 2024-03-06T15:50:39.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:39.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:39.239262+0000) 2024-03-06T15:50:40.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:40.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:40.239498+0000) 2024-03-06T15:50:41.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:41.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:41.239741+0000) 2024-03-06T15:50:42.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:42.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:42.240014+0000) 2024-03-06T15:50:43.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:43.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:43.240259+0000) 2024-03-06T15:50:44.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:44.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:44.240436+0000) 2024-03-06T15:50:45.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:45.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:45.240714+0000) 2024-03-06T15:50:46.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:46.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:46.240956+0000) 2024-03-06T15:50:47.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:47.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:47.241220+0000) 2024-03-06T15:50:48.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:48.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:48.241449+0000) 2024-03-06T15:50:49.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:49.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:49.241739+0000) 2024-03-06T15:50:50.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:50.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:50.241965+0000) 2024-03-06T15:50:50.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:50.424 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:51.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:51.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:51.242210+0000) 2024-03-06T15:50:52.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:52.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:52.242464+0000) 2024-03-06T15:50:53.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:53.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:53.242765+0000) 2024-03-06T15:50:54.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:54.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:54.242999+0000) 2024-03-06T15:50:55.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:55.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:55.243245+0000) 2024-03-06T15:50:56.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:56.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:56.243485+0000) 2024-03-06T15:50:57.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:57.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:57.243747+0000) 2024-03-06T15:50:58.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:58.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:58.243975+0000) 2024-03-06T15:50:59.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:50:59.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:59.244207+0000) 2024-03-06T15:51:00.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:00.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:00.244449+0000) 2024-03-06T15:51:01.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:01.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:01.244753+0000) 2024-03-06T15:51:02.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:02.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:02.245024+0000) 2024-03-06T15:51:03.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:03.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:03.245278+0000) 2024-03-06T15:51:04.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:04.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:04.245528+0000) 2024-03-06T15:51:05.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:05.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:05.245781+0000) 2024-03-06T15:51:06.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:06.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:06.246024+0000) 2024-03-06T15:51:07.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:07.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:07.246266+0000) 2024-03-06T15:51:08.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:08.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:08.246494+0000) 2024-03-06T15:51:09.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:09.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:09.246752+0000) 2024-03-06T15:51:10.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:10.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:10.246934+0000) 2024-03-06T15:51:11.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:11.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:11.247195+0000) 2024-03-06T15:51:12.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:12.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:12.247459+0000) 2024-03-06T15:51:13.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:13.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:13.247759+0000) 2024-03-06T15:51:14.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:14.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:14.247989+0000) 2024-03-06T15:51:15.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:15.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:15.248281+0000) 2024-03-06T15:51:16.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:16.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:16.248550+0000) 2024-03-06T15:51:17.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:17.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:17.248794+0000) 2024-03-06T15:51:18.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:18.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:18.249040+0000) 2024-03-06T15:51:19.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:19.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:19.249289+0000) 2024-03-06T15:51:20.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:20.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:20.249535+0000) 2024-03-06T15:51:20.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:20.571 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:21.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:21.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:21.249799+0000) 2024-03-06T15:51:22.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:22.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:22.250052+0000) 2024-03-06T15:51:23.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:23.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:23.250302+0000) 2024-03-06T15:51:24.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:24.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:24.250536+0000) 2024-03-06T15:51:25.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:25.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:25.250788+0000) 2024-03-06T15:51:26.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:26.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:26.251046+0000) 2024-03-06T15:51:27.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:27.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:27.251312+0000) 2024-03-06T15:51:28.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:28.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:28.251556+0000) 2024-03-06T15:51:29.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:29.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:29.251818+0000) 2024-03-06T15:51:30.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:30.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:30.252070+0000) 2024-03-06T15:51:31.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:31.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:31.252337+0000) 2024-03-06T15:51:32.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:32.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:32.252600+0000) 2024-03-06T15:51:33.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:33.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:33.252902+0000) 2024-03-06T15:51:34.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:34.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:34.253154+0000) 2024-03-06T15:51:35.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:35.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:35.253418+0000) 2024-03-06T15:51:36.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:36.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:36.253724+0000) 2024-03-06T15:51:37.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:37.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:37.253994+0000) 2024-03-06T15:51:38.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:38.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:38.254250+0000) 2024-03-06T15:51:39.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:39.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:39.254428+0000) 2024-03-06T15:51:40.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:40.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:40.254693+0000) 2024-03-06T15:51:41.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:41.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:41.254913+0000) 2024-03-06T15:51:42.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:42.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:42.255171+0000) 2024-03-06T15:51:43.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:43.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:43.255422+0000) 2024-03-06T15:51:44.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:44.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:44.255711+0000) 2024-03-06T15:51:45.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:45.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:45.255965+0000) 2024-03-06T15:51:46.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:46.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:46.256234+0000) 2024-03-06T15:51:47.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:47.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:47.256501+0000) 2024-03-06T15:51:48.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:48.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:48.256757+0000) 2024-03-06T15:51:49.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:49.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:49.257012+0000) 2024-03-06T15:51:50.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:50.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:50.257270+0000) 2024-03-06T15:51:50.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:50.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:51.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:51.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:51.257514+0000) 2024-03-06T15:51:52.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:52.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:52.257778+0000) 2024-03-06T15:51:53.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:53.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:53.258024+0000) 2024-03-06T15:51:54.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:54.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:54.258213+0000) 2024-03-06T15:51:55.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:55.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:55.258446+0000) 2024-03-06T15:51:56.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:56.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:56.258751+0000) 2024-03-06T15:51:57.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:57.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:57.259001+0000) 2024-03-06T15:51:58.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:58.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:58.259253+0000) 2024-03-06T15:51:59.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:51:59.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:59.259495+0000) 2024-03-06T15:52:00.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:00.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:00.259749+0000) 2024-03-06T15:52:01.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:01.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:01.260018+0000) 2024-03-06T15:52:02.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:02.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:02.260280+0000) 2024-03-06T15:52:03.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:03.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:03.260573+0000) 2024-03-06T15:52:04.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:04.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:04.260805+0000) 2024-03-06T15:52:05.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:05.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:05.261044+0000) 2024-03-06T15:52:06.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:06.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:06.261311+0000) 2024-03-06T15:52:07.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:07.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:07.261572+0000) 2024-03-06T15:52:08.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:08.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:08.261811+0000) 2024-03-06T15:52:09.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:09.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:09.262043+0000) 2024-03-06T15:52:10.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:10.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:10.262295+0000) 2024-03-06T15:52:11.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:11.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:11.262564+0000) 2024-03-06T15:52:12.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:12.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:12.262815+0000) 2024-03-06T15:52:13.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:13.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:13.263072+0000) 2024-03-06T15:52:14.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:14.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:14.263324+0000) 2024-03-06T15:52:15.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:15.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:15.263562+0000) 2024-03-06T15:52:16.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:16.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:16.263832+0000) 2024-03-06T15:52:17.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:17.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:17.264032+0000) 2024-03-06T15:52:18.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:18.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:18.264225+0000) 2024-03-06T15:52:19.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:19.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:19.264451+0000) 2024-03-06T15:52:20.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:20.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:20.264690+0000) 2024-03-06T15:52:20.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:20.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:21.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:21.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:21.264939+0000) 2024-03-06T15:52:22.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:22.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:22.265185+0000) 2024-03-06T15:52:23.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:23.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:23.265439+0000) 2024-03-06T15:52:24.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:24.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:24.265738+0000) 2024-03-06T15:52:25.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:25.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:25.265991+0000) 2024-03-06T15:52:26.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:26.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:26.266256+0000) 2024-03-06T15:52:27.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:27.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:27.266495+0000) 2024-03-06T15:52:28.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:28.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:28.266748+0000) 2024-03-06T15:52:29.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:29.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:29.266984+0000) 2024-03-06T15:52:30.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:30.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:30.267232+0000) 2024-03-06T15:52:31.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:31.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:31.267474+0000) 2024-03-06T15:52:32.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:32.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:32.267739+0000) 2024-03-06T15:52:33.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:33.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:33.267982+0000) 2024-03-06T15:52:34.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:34.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:34.268220+0000) 2024-03-06T15:52:35.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:35.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:35.268465+0000) 2024-03-06T15:52:36.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:36.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:36.268737+0000) 2024-03-06T15:52:37.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:37.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:37.268993+0000) 2024-03-06T15:52:38.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:38.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:38.269256+0000) 2024-03-06T15:52:39.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:39.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:39.269489+0000) 2024-03-06T15:52:40.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:40.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:40.269752+0000) 2024-03-06T15:52:41.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:41.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:41.270010+0000) 2024-03-06T15:52:42.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:42.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:42.270280+0000) 2024-03-06T15:52:43.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:43.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:43.270549+0000) 2024-03-06T15:52:44.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:44.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:44.270784+0000) 2024-03-06T15:52:45.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:45.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:45.271024+0000) 2024-03-06T15:52:46.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:46.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:46.271268+0000) 2024-03-06T15:52:47.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:47.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:47.271523+0000) 2024-03-06T15:52:48.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:48.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:48.271781+0000) 2024-03-06T15:52:49.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:49.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:49.272017+0000) 2024-03-06T15:52:50.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:50.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:50.272253+0000) 2024-03-06T15:52:51.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:51.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:51.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:51.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:51.272503+0000) 2024-03-06T15:52:52.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:52.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:52.272761+0000) 2024-03-06T15:52:53.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:53.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:53.273010+0000) 2024-03-06T15:52:54.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:54.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:54.273263+0000) 2024-03-06T15:52:55.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:55.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:55.273515+0000) 2024-03-06T15:52:56.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:56.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:56.273851+0000) 2024-03-06T15:52:57.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:57.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:57.274091+0000) 2024-03-06T15:52:58.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:58.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:58.274352+0000) 2024-03-06T15:52:59.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:52:59.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:59.274596+0000) 2024-03-06T15:53:00.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:00.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:00.274835+0000) 2024-03-06T15:53:01.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:01.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:01.275066+0000) 2024-03-06T15:53:02.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:02.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:02.275266+0000) 2024-03-06T15:53:03.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:03.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:03.275530+0000) 2024-03-06T15:53:04.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:04.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:04.275781+0000) 2024-03-06T15:53:05.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:05.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:05.276036+0000) 2024-03-06T15:53:06.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:06.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:06.276289+0000) 2024-03-06T15:53:07.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:07.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:07.276546+0000) 2024-03-06T15:53:08.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:08.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:08.276809+0000) 2024-03-06T15:53:09.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:09.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:09.277058+0000) 2024-03-06T15:53:10.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:10.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:10.277241+0000) 2024-03-06T15:53:11.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:11.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:11.277498+0000) 2024-03-06T15:53:12.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:12.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:12.277761+0000) 2024-03-06T15:53:13.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:13.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:13.278024+0000) 2024-03-06T15:53:14.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:14.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:14.278279+0000) 2024-03-06T15:53:15.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:15.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:15.278518+0000) 2024-03-06T15:53:16.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:16.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:16.278756+0000) 2024-03-06T15:53:17.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:17.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:17.279018+0000) 2024-03-06T15:53:18.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:18.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:18.279214+0000) 2024-03-06T15:53:19.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:19.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:19.279445+0000) 2024-03-06T15:53:20.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:20.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:20.279710+0000) 2024-03-06T15:53:21.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:21.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:21.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:21.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:21.279959+0000) 2024-03-06T15:53:22.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:22.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:22.280264+0000) 2024-03-06T15:53:23.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:23.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:23.280528+0000) 2024-03-06T15:53:24.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:24.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:24.280787+0000) 2024-03-06T15:53:25.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:25.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:25.281040+0000) 2024-03-06T15:53:26.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:26.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:26.281231+0000) 2024-03-06T15:53:27.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:27.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:27.281480+0000) 2024-03-06T15:53:28.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:28.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:28.281764+0000) 2024-03-06T15:53:29.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:29.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:29.282007+0000) 2024-03-06T15:53:30.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:30.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:30.282242+0000) 2024-03-06T15:53:31.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:31.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:31.282472+0000) 2024-03-06T15:53:32.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:32.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:32.282740+0000) 2024-03-06T15:53:33.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:33.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:33.282998+0000) 2024-03-06T15:53:34.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:34.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:34.283250+0000) 2024-03-06T15:53:35.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:35.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:35.283500+0000) 2024-03-06T15:53:36.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:36.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:36.283751+0000) 2024-03-06T15:53:37.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:37.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:37.284032+0000) 2024-03-06T15:53:38.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:38.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:38.284243+0000) 2024-03-06T15:53:39.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:39.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:39.284509+0000) 2024-03-06T15:53:40.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:40.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:40.284748+0000) 2024-03-06T15:53:41.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:41.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:41.284955+0000) 2024-03-06T15:53:42.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:42.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:42.285223+0000) 2024-03-06T15:53:43.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:43.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:43.285470+0000) 2024-03-06T15:53:44.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:44.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:44.285815+0000) 2024-03-06T15:53:45.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:45.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:45.286001+0000) 2024-03-06T15:53:46.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:46.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:46.286252+0000) 2024-03-06T15:53:47.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:47.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:47.286525+0000) 2024-03-06T15:53:48.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:48.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:48.286800+0000) 2024-03-06T15:53:49.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:49.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:49.286989+0000) 2024-03-06T15:53:50.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:50.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:50.287226+0000) 2024-03-06T15:53:51.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:51.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:51.287461+0000) 2024-03-06T15:53:51.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:51.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:52.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:52.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:52.287733+0000) 2024-03-06T15:53:53.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:53.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:53.287957+0000) 2024-03-06T15:53:54.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:54.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:54.288230+0000) 2024-03-06T15:53:55.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:55.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:55.288474+0000) 2024-03-06T15:53:56.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:56.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:56.288767+0000) 2024-03-06T15:53:57.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:57.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:57.289027+0000) 2024-03-06T15:53:58.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:58.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:58.289296+0000) 2024-03-06T15:53:59.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:53:59.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:59.289531+0000) 2024-03-06T15:54:00.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:00.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:00.289792+0000) 2024-03-06T15:54:01.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:01.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:01.290045+0000) 2024-03-06T15:54:02.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:02.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:02.290296+0000) 2024-03-06T15:54:03.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:03.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:03.290538+0000) 2024-03-06T15:54:04.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:04.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:04.290794+0000) 2024-03-06T15:54:05.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:05.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:05.290993+0000) 2024-03-06T15:54:06.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:06.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:06.291250+0000) 2024-03-06T15:54:07.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:07.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:07.291492+0000) 2024-03-06T15:54:08.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:08.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:08.291736+0000) 2024-03-06T15:54:09.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:09.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:09.291989+0000) 2024-03-06T15:54:10.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:10.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:10.292243+0000) 2024-03-06T15:54:11.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:11.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:11.292499+0000) 2024-03-06T15:54:12.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:12.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:12.292763+0000) 2024-03-06T15:54:13.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:13.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:13.293002+0000) 2024-03-06T15:54:14.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:14.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:14.293252+0000) 2024-03-06T15:54:15.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:15.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:15.293505+0000) 2024-03-06T15:54:16.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:16.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:16.293759+0000) 2024-03-06T15:54:17.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:17.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:17.294005+0000) 2024-03-06T15:54:18.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:18.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:18.294261+0000) 2024-03-06T15:54:19.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:19.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:19.294500+0000) 2024-03-06T15:54:20.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:20.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:20.294733+0000) 2024-03-06T15:54:21.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:21.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:21.294996+0000) 2024-03-06T15:54:21.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:21.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:22.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:22.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:22.295204+0000) 2024-03-06T15:54:23.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:23.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:23.295469+0000) 2024-03-06T15:54:24.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:24.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:24.295736+0000) 2024-03-06T15:54:25.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:25.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:25.296005+0000) 2024-03-06T15:54:26.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:26.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:26.296276+0000) 2024-03-06T15:54:27.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:27.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:27.296562+0000) 2024-03-06T15:54:28.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:28.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:28.296835+0000) 2024-03-06T15:54:29.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:29.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:29.297103+0000) 2024-03-06T15:54:30.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:30.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:30.297361+0000) 2024-03-06T15:54:31.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:31.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:31.297680+0000) 2024-03-06T15:54:32.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:32.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:32.297985+0000) 2024-03-06T15:54:33.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:33.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:33.298263+0000) 2024-03-06T15:54:34.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:34.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:34.298526+0000) 2024-03-06T15:54:35.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:35.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:35.298802+0000) 2024-03-06T15:54:36.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:36.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:36.299078+0000) 2024-03-06T15:54:37.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:37.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:37.299283+0000) 2024-03-06T15:54:38.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:38.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:38.299541+0000) 2024-03-06T15:54:39.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:39.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:39.299764+0000) 2024-03-06T15:54:40.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:40.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:40.299952+0000) 2024-03-06T15:54:41.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:41.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:41.300204+0000) 2024-03-06T15:54:42.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:42.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:42.300455+0000) 2024-03-06T15:54:43.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:43.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:43.300804+0000) 2024-03-06T15:54:44.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:44.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:44.301059+0000) 2024-03-06T15:54:45.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:45.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:45.301311+0000) 2024-03-06T15:54:46.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:46.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:46.301547+0000) 2024-03-06T15:54:47.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:47.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:47.301804+0000) 2024-03-06T15:54:48.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:48.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:48.302045+0000) 2024-03-06T15:54:49.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:49.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:49.302277+0000) 2024-03-06T15:54:50.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:50.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:50.302531+0000) 2024-03-06T15:54:51.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:51.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:51.302789+0000) 2024-03-06T15:54:51.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:51.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:52.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:52.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:52.303062+0000) 2024-03-06T15:54:53.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:53.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:53.303325+0000) 2024-03-06T15:54:54.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:54.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:54.303570+0000) 2024-03-06T15:54:55.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:55.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:55.303824+0000) 2024-03-06T15:54:56.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:56.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:56.304082+0000) 2024-03-06T15:54:57.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:57.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:57.304320+0000) 2024-03-06T15:54:58.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:58.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:58.304562+0000) 2024-03-06T15:54:59.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:54:59.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:59.304819+0000) 2024-03-06T15:55:00.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:00.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:00.305074+0000) 2024-03-06T15:55:01.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:01.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:01.305326+0000) 2024-03-06T15:55:02.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:02.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:02.305596+0000) 2024-03-06T15:55:03.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:03.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:03.305843+0000) 2024-03-06T15:55:04.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:04.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:04.306077+0000) 2024-03-06T15:55:05.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:05.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:05.306321+0000) 2024-03-06T15:55:06.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:06.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:06.306578+0000) 2024-03-06T15:55:07.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:07.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:07.306826+0000) 2024-03-06T15:55:08.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:08.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:08.307045+0000) 2024-03-06T15:55:09.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:09.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:09.307300+0000) 2024-03-06T15:55:10.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:10.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:10.307569+0000) 2024-03-06T15:55:11.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:11.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:11.307904+0000) 2024-03-06T15:55:12.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:12.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:12.308129+0000) 2024-03-06T15:55:13.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:13.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:13.308404+0000) 2024-03-06T15:55:14.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:14.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:14.308713+0000) 2024-03-06T15:55:15.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:15.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:15.308927+0000) 2024-03-06T15:55:16.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:16.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:16.309208+0000) 2024-03-06T15:55:17.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:17.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:17.309483+0000) 2024-03-06T15:55:18.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:18.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:18.309740+0000) 2024-03-06T15:55:19.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:19.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:19.309982+0000) 2024-03-06T15:55:20.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:20.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:20.310238+0000) 2024-03-06T15:55:21.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:21.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:21.310502+0000) 2024-03-06T15:55:21.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:21.793 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:22.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:22.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:22.310776+0000) 2024-03-06T15:55:23.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:23.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:23.311026+0000) 2024-03-06T15:55:24.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:24.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:24.311265+0000) 2024-03-06T15:55:25.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:25.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:25.311511+0000) 2024-03-06T15:55:26.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:26.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:26.311776+0000) 2024-03-06T15:55:27.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:27.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:27.311983+0000) 2024-03-06T15:55:28.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:28.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:28.312184+0000) 2024-03-06T15:55:29.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:29.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:29.312420+0000) 2024-03-06T15:55:30.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:30.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:30.312719+0000) 2024-03-06T15:55:31.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:31.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:31.313034+0000) 2024-03-06T15:55:32.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:32.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:32.313304+0000) 2024-03-06T15:55:33.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:33.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:33.313514+0000) 2024-03-06T15:55:34.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:34.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:34.313786+0000) 2024-03-06T15:55:35.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:35.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:35.314025+0000) 2024-03-06T15:55:36.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:36.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:36.314284+0000) 2024-03-06T15:55:37.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:37.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:37.314531+0000) 2024-03-06T15:55:38.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:38.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:38.314796+0000) 2024-03-06T15:55:39.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:39.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:39.315034+0000) 2024-03-06T15:55:40.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:40.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:40.315272+0000) 2024-03-06T15:55:41.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:41.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:41.315536+0000) 2024-03-06T15:55:42.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:42.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:42.315812+0000) 2024-03-06T15:55:43.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:43.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:43.316027+0000) 2024-03-06T15:55:44.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:44.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:44.316215+0000) 2024-03-06T15:55:45.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:45.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:45.316456+0000) 2024-03-06T15:55:46.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:46.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:46.316742+0000) 2024-03-06T15:55:47.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:47.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:47.316994+0000) 2024-03-06T15:55:48.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:48.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:48.317248+0000) 2024-03-06T15:55:49.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:49.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:49.317492+0000) 2024-03-06T15:55:50.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:50.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:50.317744+0000) 2024-03-06T15:55:51.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:51.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:51.318005+0000) 2024-03-06T15:55:51.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:51.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:52.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:52.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:52.318292+0000) 2024-03-06T15:55:53.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:53.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:53.318539+0000) 2024-03-06T15:55:54.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:54.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:54.318793+0000) 2024-03-06T15:55:55.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:55.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:55.319035+0000) 2024-03-06T15:55:56.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:56.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:56.319232+0000) 2024-03-06T15:55:57.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:57.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:57.319440+0000) 2024-03-06T15:55:58.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:58.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:58.319738+0000) 2024-03-06T15:55:59.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:55:59.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:59.320003+0000) 2024-03-06T15:56:00.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:00.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:00.320199+0000) 2024-03-06T15:56:01.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:01.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:01.320475+0000) 2024-03-06T15:56:02.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:02.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:02.320801+0000) 2024-03-06T15:56:03.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:03.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:03.321094+0000) 2024-03-06T15:56:04.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:04.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:04.321397+0000) 2024-03-06T15:56:05.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:05.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:05.321749+0000) 2024-03-06T15:56:06.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:06.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:06.322066+0000) 2024-03-06T15:56:07.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:07.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:07.322394+0000) 2024-03-06T15:56:08.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:08.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:08.322764+0000) 2024-03-06T15:56:09.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:09.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:09.323070+0000) 2024-03-06T15:56:10.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:10.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:10.323328+0000) 2024-03-06T15:56:11.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:11.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:11.323675+0000) 2024-03-06T15:56:12.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:12.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:12.323999+0000) 2024-03-06T15:56:13.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:13.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:13.324368+0000) 2024-03-06T15:56:14.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:14.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:14.324699+0000) 2024-03-06T15:56:15.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:15.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:15.325026+0000) 2024-03-06T15:56:16.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:16.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:16.325267+0000) 2024-03-06T15:56:17.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:17.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:17.325591+0000) 2024-03-06T15:56:18.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:18.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:18.325898+0000) 2024-03-06T15:56:19.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:19.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:19.326099+0000) 2024-03-06T15:56:20.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:20.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:20.326298+0000) 2024-03-06T15:56:21.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:21.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:21.326546+0000) 2024-03-06T15:56:22.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:22.120 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:22.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:22.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:22.326831+0000) 2024-03-06T15:56:23.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:23.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:23.327116+0000) 2024-03-06T15:56:24.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:24.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:24.327370+0000) 2024-03-06T15:56:25.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:25.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:25.327578+0000) 2024-03-06T15:56:26.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:26.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:26.327855+0000) 2024-03-06T15:56:27.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:27.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:27.328151+0000) 2024-03-06T15:56:28.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:28.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:28.328424+0000) 2024-03-06T15:56:29.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:29.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:29.328734+0000) 2024-03-06T15:56:30.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:30.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:30.328999+0000) 2024-03-06T15:56:31.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:31.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:31.329270+0000) 2024-03-06T15:56:32.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:32.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:32.329526+0000) 2024-03-06T15:56:33.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:33.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:33.329810+0000) 2024-03-06T15:56:34.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:34.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:34.330059+0000) 2024-03-06T15:56:35.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:35.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:35.330324+0000) 2024-03-06T15:56:36.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:36.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:36.330575+0000) 2024-03-06T15:56:37.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:37.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:37.330816+0000) 2024-03-06T15:56:38.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:38.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:38.331061+0000) 2024-03-06T15:56:39.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:39.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:39.331244+0000) 2024-03-06T15:56:40.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:40.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:40.331522+0000) 2024-03-06T15:56:41.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:41.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:41.331834+0000) 2024-03-06T15:56:42.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:42.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:42.332141+0000) 2024-03-06T15:56:43.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:43.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:43.332443+0000) 2024-03-06T15:56:44.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:44.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:44.332768+0000) 2024-03-06T15:56:45.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:45.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:45.333064+0000) 2024-03-06T15:56:46.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:46.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:46.333389+0000) 2024-03-06T15:56:47.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:47.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:47.333754+0000) 2024-03-06T15:56:48.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:48.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:48.333979+0000) 2024-03-06T15:56:49.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:49.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:49.334230+0000) 2024-03-06T15:56:50.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:50.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:50.334486+0000) 2024-03-06T15:56:51.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:51.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:51.334743+0000) 2024-03-06T15:56:52.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:52.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:52.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:52.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:52.334927+0000) 2024-03-06T15:56:53.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:53.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:53.335188+0000) 2024-03-06T15:56:54.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:54.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:54.335443+0000) 2024-03-06T15:56:55.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:55.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:55.335735+0000) 2024-03-06T15:56:56.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:56.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:56.335967+0000) 2024-03-06T15:56:57.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:57.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:57.336234+0000) 2024-03-06T15:56:58.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:58.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:58.336517+0000) 2024-03-06T15:56:59.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:56:59.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:59.336763+0000) 2024-03-06T15:57:00.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:00.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:00.337029+0000) 2024-03-06T15:57:01.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:01.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:01.337342+0000) 2024-03-06T15:57:02.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:02.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:02.337675+0000) 2024-03-06T15:57:03.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:03.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:03.337934+0000) 2024-03-06T15:57:04.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:04.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:04.338192+0000) 2024-03-06T15:57:05.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:05.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:05.338513+0000) 2024-03-06T15:57:06.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:06.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:06.338808+0000) 2024-03-06T15:57:07.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:07.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:07.339062+0000) 2024-03-06T15:57:08.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:08.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:08.339326+0000) 2024-03-06T15:57:09.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:09.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:09.339581+0000) 2024-03-06T15:57:10.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:10.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:10.339822+0000) 2024-03-06T15:57:11.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:11.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:11.340032+0000) 2024-03-06T15:57:12.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:12.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:12.340251+0000) 2024-03-06T15:57:13.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:13.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:13.340587+0000) 2024-03-06T15:57:14.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:14.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:14.340932+0000) 2024-03-06T15:57:15.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:15.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:15.341240+0000) 2024-03-06T15:57:16.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:16.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:16.341556+0000) 2024-03-06T15:57:17.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:17.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:17.341858+0000) 2024-03-06T15:57:18.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:18.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:18.342164+0000) 2024-03-06T15:57:19.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:19.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:19.342464+0000) 2024-03-06T15:57:20.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:20.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:20.342753+0000) 2024-03-06T15:57:21.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:21.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:21.343038+0000) 2024-03-06T15:57:22.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:22.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:22.343373+0000) 2024-03-06T15:57:22.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:22.415 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:23.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:23.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:23.343703+0000) 2024-03-06T15:57:24.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:24.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:24.344019+0000) 2024-03-06T15:57:25.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:25.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:25.344259+0000) 2024-03-06T15:57:26.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:26.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:26.344504+0000) 2024-03-06T15:57:27.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:27.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:27.344814+0000) 2024-03-06T15:57:28.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:28.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:28.345103+0000) 2024-03-06T15:57:29.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:29.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:29.345394+0000) 2024-03-06T15:57:30.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:30.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:30.345741+0000) 2024-03-06T15:57:31.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:31.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:31.346032+0000) 2024-03-06T15:57:32.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:32.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:32.346354+0000) 2024-03-06T15:57:33.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:33.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:33.346720+0000) 2024-03-06T15:57:34.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:34.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:34.347005+0000) 2024-03-06T15:57:35.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:35.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:35.347218+0000) 2024-03-06T15:57:36.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:36.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:36.347469+0000) 2024-03-06T15:57:37.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:37.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:37.347708+0000) 2024-03-06T15:57:38.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:38.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:38.347961+0000) 2024-03-06T15:57:39.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:39.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:39.348216+0000) 2024-03-06T15:57:40.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:40.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:40.348451+0000) 2024-03-06T15:57:41.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:41.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:41.348742+0000) 2024-03-06T15:57:42.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:42.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:42.348997+0000) 2024-03-06T15:57:43.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:43.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:43.349256+0000) 2024-03-06T15:57:44.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:44.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:44.349501+0000) 2024-03-06T15:57:45.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:45.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:45.349750+0000) 2024-03-06T15:57:46.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:46.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:46.350012+0000) 2024-03-06T15:57:47.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:47.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:47.350198+0000) 2024-03-06T15:57:48.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:48.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:48.350453+0000) 2024-03-06T15:57:49.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:49.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:49.350734+0000) 2024-03-06T15:57:50.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:50.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:50.350993+0000) 2024-03-06T15:57:51.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:51.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:51.351251+0000) 2024-03-06T15:57:52.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:52.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:52.351507+0000) 2024-03-06T15:57:52.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:52.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:53.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:53.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:53.351781+0000) 2024-03-06T15:57:54.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:54.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:54.352092+0000) 2024-03-06T15:57:55.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:55.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:55.352382+0000) 2024-03-06T15:57:56.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:56.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:56.352736+0000) 2024-03-06T15:57:57.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:57.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:57.353052+0000) 2024-03-06T15:57:58.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:58.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:58.353356+0000) 2024-03-06T15:57:59.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:57:59.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:59.353686+0000) 2024-03-06T15:58:00.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:00.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:00.353978+0000) 2024-03-06T15:58:01.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:01.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:01.354235+0000) 2024-03-06T15:58:02.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:02.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:02.354520+0000) 2024-03-06T15:58:03.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:03.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:03.354765+0000) 2024-03-06T15:58:04.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:04.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:04.355012+0000) 2024-03-06T15:58:05.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:05.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:05.355307+0000) 2024-03-06T15:58:06.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:06.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:06.355589+0000) 2024-03-06T15:58:07.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:07.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:07.355948+0000) 2024-03-06T15:58:08.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:08.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:08.356242+0000) 2024-03-06T15:58:09.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:09.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:09.356501+0000) 2024-03-06T15:58:10.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:10.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:10.356757+0000) 2024-03-06T15:58:11.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:11.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:11.357017+0000) 2024-03-06T15:58:12.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:12.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:12.357279+0000) 2024-03-06T15:58:13.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:13.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:13.357552+0000) 2024-03-06T15:58:14.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:14.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:14.357828+0000) 2024-03-06T15:58:15.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:15.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:15.358099+0000) 2024-03-06T15:58:16.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:16.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:16.358318+0000) 2024-03-06T15:58:17.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:17.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:17.358579+0000) 2024-03-06T15:58:18.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:18.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:18.358846+0000) 2024-03-06T15:58:19.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:19.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:19.359159+0000) 2024-03-06T15:58:20.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:20.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:20.359443+0000) 2024-03-06T15:58:21.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:21.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:21.359778+0000) 2024-03-06T15:58:22.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:22.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:22.360102+0000) 2024-03-06T15:58:22.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:22.716 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:23.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:23.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:23.360384+0000) 2024-03-06T15:58:24.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:24.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:24.360729+0000) 2024-03-06T15:58:25.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:25.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:25.360983+0000) 2024-03-06T15:58:26.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:26.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:26.361231+0000) 2024-03-06T15:58:27.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:27.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:27.361535+0000) 2024-03-06T15:58:28.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:28.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:28.361838+0000) 2024-03-06T15:58:29.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:29.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:29.362124+0000) 2024-03-06T15:58:30.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:30.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:30.362358+0000) 2024-03-06T15:58:31.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:31.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:31.362694+0000) 2024-03-06T15:58:32.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:32.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:32.363028+0000) 2024-03-06T15:58:33.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:33.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:33.363269+0000) 2024-03-06T15:58:34.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:34.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:34.363550+0000) 2024-03-06T15:58:35.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:35.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:35.363857+0000) 2024-03-06T15:58:36.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:36.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:36.364148+0000) 2024-03-06T15:58:37.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:37.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:37.364432+0000) 2024-03-06T15:58:38.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:38.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:38.364794+0000) 2024-03-06T15:58:39.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:39.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:39.365104+0000) 2024-03-06T15:58:40.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:40.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:40.365418+0000) 2024-03-06T15:58:41.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:41.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:41.365706+0000) 2024-03-06T15:58:42.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:42.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:42.366035+0000) 2024-03-06T15:58:43.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:43.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:43.366346+0000) 2024-03-06T15:58:44.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:44.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:44.366687+0000) 2024-03-06T15:58:45.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:45.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:45.366924+0000) 2024-03-06T15:58:46.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:46.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:46.367232+0000) 2024-03-06T15:58:47.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:47.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:47.367562+0000) 2024-03-06T15:58:48.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:48.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:48.367921+0000) 2024-03-06T15:58:49.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:49.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:49.368169+0000) 2024-03-06T15:58:50.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:50.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:50.368436+0000) 2024-03-06T15:58:51.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:51.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:51.368711+0000) 2024-03-06T15:58:52.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:52.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:52.368981+0000) 2024-03-06T15:58:52.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:52.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:53.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:53.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:53.369241+0000) 2024-03-06T15:58:54.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:54.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:54.369475+0000) 2024-03-06T15:58:55.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:55.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:55.369749+0000) 2024-03-06T15:58:56.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:56.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:56.369996+0000) 2024-03-06T15:58:57.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:57.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:57.370190+0000) 2024-03-06T15:58:58.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:58.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:58.370450+0000) 2024-03-06T15:58:59.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:58:59.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:59.370735+0000) 2024-03-06T15:59:00.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:00.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:00.370971+0000) 2024-03-06T15:59:01.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:01.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:01.371226+0000) 2024-03-06T15:59:02.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:02.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:02.371462+0000) 2024-03-06T15:59:03.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:03.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:03.371744+0000) 2024-03-06T15:59:04.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:04.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:04.371969+0000) 2024-03-06T15:59:05.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:05.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:05.372169+0000) 2024-03-06T15:59:06.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:06.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:06.372406+0000) 2024-03-06T15:59:07.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:07.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:07.372700+0000) 2024-03-06T15:59:08.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:08.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:08.372960+0000) 2024-03-06T15:59:09.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:09.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:09.373220+0000) 2024-03-06T15:59:10.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:10.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:10.373460+0000) 2024-03-06T15:59:11.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:11.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:11.373737+0000) 2024-03-06T15:59:12.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:12.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:12.374009+0000) 2024-03-06T15:59:13.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:13.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:13.374260+0000) 2024-03-06T15:59:14.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:14.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:14.374519+0000) 2024-03-06T15:59:15.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:15.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:15.374785+0000) 2024-03-06T15:59:16.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:16.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:16.375026+0000) 2024-03-06T15:59:17.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:17.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:17.375290+0000) 2024-03-06T15:59:18.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:18.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:18.375541+0000) 2024-03-06T15:59:19.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:19.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:19.375794+0000) 2024-03-06T15:59:20.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:20.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:20.375987+0000) 2024-03-06T15:59:21.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:21.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:21.376183+0000) 2024-03-06T15:59:22.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:22.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:22.376447+0000) 2024-03-06T15:59:22.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:23.005 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:23.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:23.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:23.376739+0000) 2024-03-06T15:59:24.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:24.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:24.376978+0000) 2024-03-06T15:59:25.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:25.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:25.377240+0000) 2024-03-06T15:59:26.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:26.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:26.377506+0000) 2024-03-06T15:59:27.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:27.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:27.377782+0000) 2024-03-06T15:59:28.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:28.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:28.377988+0000) 2024-03-06T15:59:29.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:29.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:29.378198+0000) 2024-03-06T15:59:30.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:30.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:30.378473+0000) 2024-03-06T15:59:31.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:31.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:31.378750+0000) 2024-03-06T15:59:32.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:32.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:32.379021+0000) 2024-03-06T15:59:33.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:33.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:33.379270+0000) 2024-03-06T15:59:34.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:34.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:34.379520+0000) 2024-03-06T15:59:35.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:35.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:35.379766+0000) 2024-03-06T15:59:36.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:36.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:36.379965+0000) 2024-03-06T15:59:37.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:37.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:37.380246+0000) 2024-03-06T15:59:38.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:38.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:38.380538+0000) 2024-03-06T15:59:39.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:39.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:39.380840+0000) 2024-03-06T15:59:40.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:40.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:40.381142+0000) 2024-03-06T15:59:41.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:41.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:41.381483+0000) 2024-03-06T15:59:42.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:42.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:42.381805+0000) 2024-03-06T15:59:43.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:43.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:43.382061+0000) 2024-03-06T15:59:44.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:44.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:44.382366+0000) 2024-03-06T15:59:45.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:45.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:45.382611+0000) 2024-03-06T15:59:46.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:46.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:46.382965+0000) 2024-03-06T15:59:47.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:47.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:47.383229+0000) 2024-03-06T15:59:48.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:48.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:48.383501+0000) 2024-03-06T15:59:49.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:49.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:49.383793+0000) 2024-03-06T15:59:50.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:50.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:50.384111+0000) 2024-03-06T15:59:51.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:51.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:51.384397+0000) 2024-03-06T15:59:52.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:52.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:52.384757+0000) 2024-03-06T15:59:53.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:53.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:53.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:53.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:53.385052+0000) 2024-03-06T15:59:54.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:54.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:54.385361+0000) 2024-03-06T15:59:55.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:55.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:55.385608+0000) 2024-03-06T15:59:56.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:56.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:56.385876+0000) 2024-03-06T15:59:57.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:57.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:57.386143+0000) 2024-03-06T15:59:58.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:58.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:58.386414+0000) 2024-03-06T15:59:59.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T15:59:59.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:59.386738+0000) 2024-03-06T16:00:00.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:00.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:00.387000+0000) 2024-03-06T16:00:01.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:01.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:01.387318+0000) 2024-03-06T16:00:02.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:02.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:02.387686+0000) 2024-03-06T16:00:03.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:03.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:03.387983+0000) 2024-03-06T16:00:04.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:04.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:04.388287+0000) 2024-03-06T16:00:05.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:05.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:05.388579+0000) 2024-03-06T16:00:06.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:06.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:06.388882+0000) 2024-03-06T16:00:07.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:07.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:07.389180+0000) 2024-03-06T16:00:08.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:08.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:08.389482+0000) 2024-03-06T16:00:09.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:09.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:09.389800+0000) 2024-03-06T16:00:10.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:10.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:10.390121+0000) 2024-03-06T16:00:11.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:11.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:11.390430+0000) 2024-03-06T16:00:12.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:12.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:12.390784+0000) 2024-03-06T16:00:13.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:13.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:13.391083+0000) 2024-03-06T16:00:14.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:14.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:14.391398+0000) 2024-03-06T16:00:15.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:15.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:15.391734+0000) 2024-03-06T16:00:16.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:16.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:16.392030+0000) 2024-03-06T16:00:17.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:17.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:17.392330+0000) 2024-03-06T16:00:18.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:18.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:18.392623+0000) 2024-03-06T16:00:19.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:19.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:19.392981+0000) 2024-03-06T16:00:20.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:20.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:20.393286+0000) 2024-03-06T16:00:21.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:21.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:21.393590+0000) 2024-03-06T16:00:22.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:22.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:22.393942+0000) 2024-03-06T16:00:23.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:23.305 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:23.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:23.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:23.394150+0000) 2024-03-06T16:00:24.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:24.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:24.394514+0000) 2024-03-06T16:00:25.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:25.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:25.394835+0000) 2024-03-06T16:00:26.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:26.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:26.395136+0000) 2024-03-06T16:00:27.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:27.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:27.395437+0000) 2024-03-06T16:00:28.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:28.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:28.395785+0000) 2024-03-06T16:00:29.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:29.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:29.396071+0000) 2024-03-06T16:00:30.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:30.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:30.396320+0000) 2024-03-06T16:00:31.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:31.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:31.396558+0000) 2024-03-06T16:00:32.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:32.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:32.396830+0000) 2024-03-06T16:00:33.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:33.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:33.397049+0000) 2024-03-06T16:00:34.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:34.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:34.397321+0000) 2024-03-06T16:00:35.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:35.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:35.397600+0000) 2024-03-06T16:00:36.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:36.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:36.397910+0000) 2024-03-06T16:00:37.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:37.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:37.398186+0000) 2024-03-06T16:00:38.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:38.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:38.398455+0000) 2024-03-06T16:00:39.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:39.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:39.398755+0000) 2024-03-06T16:00:40.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:40.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:40.399018+0000) 2024-03-06T16:00:41.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:41.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:41.399304+0000) 2024-03-06T16:00:42.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:42.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:42.399573+0000) 2024-03-06T16:00:43.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:43.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:43.399846+0000) 2024-03-06T16:00:44.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:44.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:44.400101+0000) 2024-03-06T16:00:45.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:45.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:45.400357+0000) 2024-03-06T16:00:46.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:46.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:46.400593+0000) 2024-03-06T16:00:47.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:47.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:47.400855+0000) 2024-03-06T16:00:48.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:48.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:48.401120+0000) 2024-03-06T16:00:49.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:49.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:49.401393+0000) 2024-03-06T16:00:50.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:50.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:50.401694+0000) 2024-03-06T16:00:51.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:51.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:51.401956+0000) 2024-03-06T16:00:52.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:52.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:52.402229+0000) 2024-03-06T16:00:53.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:53.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:53.402481+0000) 2024-03-06T16:00:53.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:53.464 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:54.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:54.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:54.402754+0000) 2024-03-06T16:00:55.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:55.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:55.403010+0000) 2024-03-06T16:00:56.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:56.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:56.403253+0000) 2024-03-06T16:00:57.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:57.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:57.403515+0000) 2024-03-06T16:00:58.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:58.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:58.403777+0000) 2024-03-06T16:00:59.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:00:59.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:59.404034+0000) 2024-03-06T16:01:00.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:00.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:00.404280+0000) 2024-03-06T16:01:01.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:01.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:01.404539+0000) 2024-03-06T16:01:02.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:02.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:02.404807+0000) 2024-03-06T16:01:03.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:03.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:03.405058+0000) 2024-03-06T16:01:04.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:04.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:04.405303+0000) 2024-03-06T16:01:05.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:05.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:05.405565+0000) 2024-03-06T16:01:06.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:06.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:06.405833+0000) 2024-03-06T16:01:07.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:07.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:07.406099+0000) 2024-03-06T16:01:08.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:08.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:08.406369+0000) 2024-03-06T16:01:09.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:09.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:09.406624+0000) 2024-03-06T16:01:10.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:10.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:10.406901+0000) 2024-03-06T16:01:11.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:11.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:11.407139+0000) 2024-03-06T16:01:12.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:12.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:12.407391+0000) 2024-03-06T16:01:13.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:13.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:13.407728+0000) 2024-03-06T16:01:14.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:14.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:14.408010+0000) 2024-03-06T16:01:15.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:15.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:15.408267+0000) 2024-03-06T16:01:16.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:16.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:16.408536+0000) 2024-03-06T16:01:17.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:17.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:17.408794+0000) 2024-03-06T16:01:18.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:18.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:18.409082+0000) 2024-03-06T16:01:19.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:19.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:19.409353+0000) 2024-03-06T16:01:20.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:20.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:20.409597+0000) 2024-03-06T16:01:21.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:21.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:21.409836+0000) 2024-03-06T16:01:22.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:22.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:22.410087+0000) 2024-03-06T16:01:23.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:23.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:23.410351+0000) 2024-03-06T16:01:23.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:23.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:24.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:24.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:24.410609+0000) 2024-03-06T16:01:25.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:25.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:25.410881+0000) 2024-03-06T16:01:26.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:26.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:26.411075+0000) 2024-03-06T16:01:27.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:27.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:27.411317+0000) 2024-03-06T16:01:28.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:28.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:28.411590+0000) 2024-03-06T16:01:29.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:29.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:29.411840+0000) 2024-03-06T16:01:30.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:30.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:30.412101+0000) 2024-03-06T16:01:31.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:31.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:31.412355+0000) 2024-03-06T16:01:32.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:32.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:32.412618+0000) 2024-03-06T16:01:33.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:33.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:33.412917+0000) 2024-03-06T16:01:34.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:34.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:34.413180+0000) 2024-03-06T16:01:35.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:35.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:35.413446+0000) 2024-03-06T16:01:36.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:36.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:36.413739+0000) 2024-03-06T16:01:37.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:37.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:37.413976+0000) 2024-03-06T16:01:38.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:38.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:38.414246+0000) 2024-03-06T16:01:39.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:39.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:39.414486+0000) 2024-03-06T16:01:40.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:40.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:40.414755+0000) 2024-03-06T16:01:41.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:41.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:41.414989+0000) 2024-03-06T16:01:42.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:42.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:42.415322+0000) 2024-03-06T16:01:43.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:43.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:43.415616+0000) 2024-03-06T16:01:44.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:44.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:44.415981+0000) 2024-03-06T16:01:45.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:45.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:45.416273+0000) 2024-03-06T16:01:46.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:46.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:46.416586+0000) 2024-03-06T16:01:47.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:47.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:47.416848+0000) 2024-03-06T16:01:48.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:48.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:48.417126+0000) 2024-03-06T16:01:49.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:49.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:49.417390+0000) 2024-03-06T16:01:50.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:50.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:50.417668+0000) 2024-03-06T16:01:51.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:51.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:51.417908+0000) 2024-03-06T16:01:52.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:52.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:52.418155+0000) 2024-03-06T16:01:53.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:53.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:53.418401+0000) 2024-03-06T16:01:53.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:53.773 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:54.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:54.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:54.418675+0000) 2024-03-06T16:01:55.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:55.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:55.418925+0000) 2024-03-06T16:01:56.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:56.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:56.419178+0000) 2024-03-06T16:01:57.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:57.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:57.419417+0000) 2024-03-06T16:01:58.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:58.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:58.419702+0000) 2024-03-06T16:01:59.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:01:59.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:59.419943+0000) 2024-03-06T16:02:00.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:00.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:00.420204+0000) 2024-03-06T16:02:01.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:01.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:01.420460+0000) 2024-03-06T16:02:02.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:02.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:02.420749+0000) 2024-03-06T16:02:03.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:03.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:03.421003+0000) 2024-03-06T16:02:04.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:04.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:04.421257+0000) 2024-03-06T16:02:05.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:05.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:05.421513+0000) 2024-03-06T16:02:06.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:06.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:06.421783+0000) 2024-03-06T16:02:07.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:07.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:07.422021+0000) 2024-03-06T16:02:08.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:08.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:08.422269+0000) 2024-03-06T16:02:09.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:09.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:09.422512+0000) 2024-03-06T16:02:10.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:10.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:10.422746+0000) 2024-03-06T16:02:11.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:11.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:11.423031+0000) 2024-03-06T16:02:12.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:12.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:12.423284+0000) 2024-03-06T16:02:13.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:13.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:13.423517+0000) 2024-03-06T16:02:14.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:14.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:14.423755+0000) 2024-03-06T16:02:15.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:15.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:15.424012+0000) 2024-03-06T16:02:16.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:16.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:16.424274+0000) 2024-03-06T16:02:17.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:17.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:17.424545+0000) 2024-03-06T16:02:18.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:18.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:18.424786+0000) 2024-03-06T16:02:19.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:19.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:19.425023+0000) 2024-03-06T16:02:20.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:20.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:20.425280+0000) 2024-03-06T16:02:21.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:21.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:21.425539+0000) 2024-03-06T16:02:22.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:22.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:22.425808+0000) 2024-03-06T16:02:23.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:23.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:23.426074+0000) 2024-03-06T16:02:23.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:23.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:24.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:24.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:24.426349+0000) 2024-03-06T16:02:25.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:25.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:25.426672+0000) 2024-03-06T16:02:26.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:26.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:26.426921+0000) 2024-03-06T16:02:27.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:27.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:27.427196+0000) 2024-03-06T16:02:28.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:28.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:28.427451+0000) 2024-03-06T16:02:29.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:29.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:29.427741+0000) 2024-03-06T16:02:30.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:30.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:30.428002+0000) 2024-03-06T16:02:31.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:31.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:31.428257+0000) 2024-03-06T16:02:32.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:32.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:32.428534+0000) 2024-03-06T16:02:33.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:33.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:33.428811+0000) 2024-03-06T16:02:34.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:34.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:34.429072+0000) 2024-03-06T16:02:35.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:35.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:35.429329+0000) 2024-03-06T16:02:36.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:36.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:36.429582+0000) 2024-03-06T16:02:37.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:37.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:37.429848+0000) 2024-03-06T16:02:38.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:38.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:38.430120+0000) 2024-03-06T16:02:39.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:39.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:39.430371+0000) 2024-03-06T16:02:40.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:40.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:40.430680+0000) 2024-03-06T16:02:41.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:41.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:41.430941+0000) 2024-03-06T16:02:42.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:42.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:42.431226+0000) 2024-03-06T16:02:43.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:43.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:43.431488+0000) 2024-03-06T16:02:44.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:44.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:44.431768+0000) 2024-03-06T16:02:45.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:45.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:45.432027+0000) 2024-03-06T16:02:46.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:46.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:46.432303+0000) 2024-03-06T16:02:47.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:47.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:47.432586+0000) 2024-03-06T16:02:48.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:48.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:48.432903+0000) 2024-03-06T16:02:49.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:49.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:49.433183+0000) 2024-03-06T16:02:50.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:50.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:50.433459+0000) 2024-03-06T16:02:51.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:51.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:51.433756+0000) 2024-03-06T16:02:52.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:52.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:52.434044+0000) 2024-03-06T16:02:53.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:53.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:53.434307+0000) 2024-03-06T16:02:54.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:54.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:54.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:54.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:54.434565+0000) 2024-03-06T16:02:55.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:55.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:55.434839+0000) 2024-03-06T16:02:56.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:56.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:56.435096+0000) 2024-03-06T16:02:57.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:57.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:57.435380+0000) 2024-03-06T16:02:58.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:58.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:58.435667+0000) 2024-03-06T16:02:59.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:02:59.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:59.435992+0000) 2024-03-06T16:03:00.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:00.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:00.436272+0000) 2024-03-06T16:03:01.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:01.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:01.436525+0000) 2024-03-06T16:03:02.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:02.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:02.436776+0000) 2024-03-06T16:03:03.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:03.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:03.437009+0000) 2024-03-06T16:03:04.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:04.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:04.437268+0000) 2024-03-06T16:03:05.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:05.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:05.437521+0000) 2024-03-06T16:03:06.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:06.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:06.437803+0000) 2024-03-06T16:03:07.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:07.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:07.438058+0000) 2024-03-06T16:03:08.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:08.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:08.438243+0000) 2024-03-06T16:03:09.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:09.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:09.438486+0000) 2024-03-06T16:03:10.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:10.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:10.438759+0000) 2024-03-06T16:03:11.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:11.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:11.439019+0000) 2024-03-06T16:03:12.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:12.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:12.439293+0000) 2024-03-06T16:03:13.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:13.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:13.439553+0000) 2024-03-06T16:03:14.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:14.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:14.439829+0000) 2024-03-06T16:03:15.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:15.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:15.440085+0000) 2024-03-06T16:03:16.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:16.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:16.440359+0000) 2024-03-06T16:03:17.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:17.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:17.440671+0000) 2024-03-06T16:03:18.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:18.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:18.440886+0000) 2024-03-06T16:03:19.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:19.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:19.441152+0000) 2024-03-06T16:03:20.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:20.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:20.441413+0000) 2024-03-06T16:03:21.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:21.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:21.441735+0000) 2024-03-06T16:03:22.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:22.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:22.442001+0000) 2024-03-06T16:03:23.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:23.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:23.442267+0000) 2024-03-06T16:03:24.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:24.227 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:24.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:24.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:24.442547+0000) 2024-03-06T16:03:25.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:25.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:25.442826+0000) 2024-03-06T16:03:26.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:26.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:26.443048+0000) 2024-03-06T16:03:27.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:27.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:27.443321+0000) 2024-03-06T16:03:28.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:28.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:28.443599+0000) 2024-03-06T16:03:29.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:29.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:29.443925+0000) 2024-03-06T16:03:30.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:30.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:30.444142+0000) 2024-03-06T16:03:31.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:31.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:31.444399+0000) 2024-03-06T16:03:32.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:32.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:32.444726+0000) 2024-03-06T16:03:33.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:33.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:33.444998+0000) 2024-03-06T16:03:34.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:34.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:34.445254+0000) 2024-03-06T16:03:35.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:35.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:35.445529+0000) 2024-03-06T16:03:36.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:36.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:36.445807+0000) 2024-03-06T16:03:37.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:37.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:37.446078+0000) 2024-03-06T16:03:38.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:38.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:38.446320+0000) 2024-03-06T16:03:39.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:39.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:39.446577+0000) 2024-03-06T16:03:40.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:40.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:40.446791+0000) 2024-03-06T16:03:41.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:41.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:41.447048+0000) 2024-03-06T16:03:42.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:42.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:42.447318+0000) 2024-03-06T16:03:43.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:43.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:43.447573+0000) 2024-03-06T16:03:44.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:44.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:44.447826+0000) 2024-03-06T16:03:45.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:45.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:45.448086+0000) 2024-03-06T16:03:46.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:46.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:46.448339+0000) 2024-03-06T16:03:47.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:47.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:47.448668+0000) 2024-03-06T16:03:48.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:48.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:48.448930+0000) 2024-03-06T16:03:49.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:49.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:49.449182+0000) 2024-03-06T16:03:50.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:50.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:50.449428+0000) 2024-03-06T16:03:51.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:51.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:51.449613+0000) 2024-03-06T16:03:52.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:52.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:52.449902+0000) 2024-03-06T16:03:53.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:53.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:53.450161+0000) 2024-03-06T16:03:54.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:54.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:54.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:54.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:54.450393+0000) 2024-03-06T16:03:55.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:55.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:55.450734+0000) 2024-03-06T16:03:56.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:56.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:56.451019+0000) 2024-03-06T16:03:57.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:57.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:57.451311+0000) 2024-03-06T16:03:58.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:58.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:58.451614+0000) 2024-03-06T16:03:59.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:03:59.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:59.451966+0000) 2024-03-06T16:04:00.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:00.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:00.452276+0000) 2024-03-06T16:04:01.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:01.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:01.452546+0000) 2024-03-06T16:04:02.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:02.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:02.452793+0000) 2024-03-06T16:04:03.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:03.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:03.453032+0000) 2024-03-06T16:04:04.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:04.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:04.453265+0000) 2024-03-06T16:04:05.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:05.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:05.453503+0000) 2024-03-06T16:04:06.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:06.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:06.453746+0000) 2024-03-06T16:04:07.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:07.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:07.453999+0000) 2024-03-06T16:04:08.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:08.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:08.454259+0000) 2024-03-06T16:04:09.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:09.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:09.454510+0000) 2024-03-06T16:04:10.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:10.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:10.454768+0000) 2024-03-06T16:04:11.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:11.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:11.455028+0000) 2024-03-06T16:04:12.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:12.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:12.455350+0000) 2024-03-06T16:04:13.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:13.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:13.455711+0000) 2024-03-06T16:04:14.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:14.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:14.456019+0000) 2024-03-06T16:04:15.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:15.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:15.456318+0000) 2024-03-06T16:04:16.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:16.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:16.456683+0000) 2024-03-06T16:04:17.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:17.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:17.456927+0000) 2024-03-06T16:04:18.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:18.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:18.457229+0000) 2024-03-06T16:04:19.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:19.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:19.457529+0000) 2024-03-06T16:04:20.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:20.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:20.457798+0000) 2024-03-06T16:04:21.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:21.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:21.458099+0000) 2024-03-06T16:04:22.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:22.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:22.458431+0000) 2024-03-06T16:04:23.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:23.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:23.458784+0000) 2024-03-06T16:04:24.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:24.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:24.459092+0000) 2024-03-06T16:04:24.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:24.514 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:25.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:25.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:25.459385+0000) 2024-03-06T16:04:26.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:26.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:26.459746+0000) 2024-03-06T16:04:27.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:27.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:27.460054+0000) 2024-03-06T16:04:28.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:28.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:28.460373+0000) 2024-03-06T16:04:29.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:29.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:29.460791+0000) 2024-03-06T16:04:30.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:30.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:30.461046+0000) 2024-03-06T16:04:31.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:31.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:31.461281+0000) 2024-03-06T16:04:32.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:32.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:32.461596+0000) 2024-03-06T16:04:33.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:33.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:33.461969+0000) 2024-03-06T16:04:34.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:34.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:34.462275+0000) 2024-03-06T16:04:35.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:35.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:35.462540+0000) 2024-03-06T16:04:36.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:36.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:36.462785+0000) 2024-03-06T16:04:37.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:37.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:37.463043+0000) 2024-03-06T16:04:38.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:38.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:38.463307+0000) 2024-03-06T16:04:39.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:39.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:39.463553+0000) 2024-03-06T16:04:40.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:40.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:40.463827+0000) 2024-03-06T16:04:41.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:41.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:41.464089+0000) 2024-03-06T16:04:42.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:42.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:42.464338+0000) 2024-03-06T16:04:43.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:43.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:43.464576+0000) 2024-03-06T16:04:44.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:44.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:44.464838+0000) 2024-03-06T16:04:45.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:45.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:45.465090+0000) 2024-03-06T16:04:46.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:46.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:46.465344+0000) 2024-03-06T16:04:47.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:47.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:47.465598+0000) 2024-03-06T16:04:48.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:48.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:48.465844+0000) 2024-03-06T16:04:49.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:49.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:49.466101+0000) 2024-03-06T16:04:50.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:50.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:50.466353+0000) 2024-03-06T16:04:51.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:51.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:51.466612+0000) 2024-03-06T16:04:52.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:52.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:52.466897+0000) 2024-03-06T16:04:53.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:53.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:53.467130+0000) 2024-03-06T16:04:54.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:54.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:54.467381+0000) 2024-03-06T16:04:54.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:54.661 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:55.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:55.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:55.467677+0000) 2024-03-06T16:04:56.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:56.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:56.467912+0000) 2024-03-06T16:04:57.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:57.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:57.468161+0000) 2024-03-06T16:04:58.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:58.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:58.468392+0000) 2024-03-06T16:04:59.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:04:59.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:59.468674+0000) 2024-03-06T16:05:00.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:00.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:00.468932+0000) 2024-03-06T16:05:01.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:01.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:01.469193+0000) 2024-03-06T16:05:02.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:02.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:02.469449+0000) 2024-03-06T16:05:03.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:03.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:03.469736+0000) 2024-03-06T16:05:04.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:04.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:04.469991+0000) 2024-03-06T16:05:05.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:05.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:05.470250+0000) 2024-03-06T16:05:06.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:06.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:06.470507+0000) 2024-03-06T16:05:07.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:07.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:07.470766+0000) 2024-03-06T16:05:08.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:08.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:08.471024+0000) 2024-03-06T16:05:09.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:09.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:09.471277+0000) 2024-03-06T16:05:10.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:10.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:10.471540+0000) 2024-03-06T16:05:11.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:11.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:11.471765+0000) 2024-03-06T16:05:12.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:12.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:12.472031+0000) 2024-03-06T16:05:13.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:13.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:13.472232+0000) 2024-03-06T16:05:14.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:14.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:14.472428+0000) 2024-03-06T16:05:15.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:15.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:15.472702+0000) 2024-03-06T16:05:16.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:16.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:16.472936+0000) 2024-03-06T16:05:17.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:17.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:17.473171+0000) 2024-03-06T16:05:18.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:18.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:18.473457+0000) 2024-03-06T16:05:19.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:19.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:19.473731+0000) 2024-03-06T16:05:20.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:20.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:20.473962+0000) 2024-03-06T16:05:21.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:21.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:21.474256+0000) 2024-03-06T16:05:22.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:22.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:22.474560+0000) 2024-03-06T16:05:23.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:23.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:23.474839+0000) 2024-03-06T16:05:24.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:24.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:24.475063+0000) 2024-03-06T16:05:24.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:24.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:25.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:25.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:25.475377+0000) 2024-03-06T16:05:26.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:26.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:26.475733+0000) 2024-03-06T16:05:27.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:27.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:27.476040+0000) 2024-03-06T16:05:28.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:28.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:28.476340+0000) 2024-03-06T16:05:29.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:29.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:29.476660+0000) 2024-03-06T16:05:30.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:30.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:30.476985+0000) 2024-03-06T16:05:31.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:31.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:31.477285+0000) 2024-03-06T16:05:32.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:32.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:32.477592+0000) 2024-03-06T16:05:33.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:33.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:33.477948+0000) 2024-03-06T16:05:34.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:34.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:34.478262+0000) 2024-03-06T16:05:35.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:35.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:35.478582+0000) 2024-03-06T16:05:36.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:36.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:36.478877+0000) 2024-03-06T16:05:37.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:37.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:37.479190+0000) 2024-03-06T16:05:38.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:38.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:38.479487+0000) 2024-03-06T16:05:39.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:39.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:39.479777+0000) 2024-03-06T16:05:40.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:40.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:40.480056+0000) 2024-03-06T16:05:41.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:41.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:41.480353+0000) 2024-03-06T16:05:42.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:42.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:42.480705+0000) 2024-03-06T16:05:43.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:43.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:43.481010+0000) 2024-03-06T16:05:44.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:44.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:44.481298+0000) 2024-03-06T16:05:45.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:45.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:45.481624+0000) 2024-03-06T16:05:46.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:46.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:46.481979+0000) 2024-03-06T16:05:47.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:47.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:47.482285+0000) 2024-03-06T16:05:48.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:48.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:48.482515+0000) 2024-03-06T16:05:49.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:49.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:49.482840+0000) 2024-03-06T16:05:50.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:50.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:50.483141+0000) 2024-03-06T16:05:51.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:51.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:51.483450+0000) 2024-03-06T16:05:52.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:52.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:52.483758+0000) 2024-03-06T16:05:53.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:53.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:53.484020+0000) 2024-03-06T16:05:54.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:54.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:54.484277+0000) 2024-03-06T16:05:54.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:54.953 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:55.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:55.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:55.484548+0000) 2024-03-06T16:05:56.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:56.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:56.484805+0000) 2024-03-06T16:05:57.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:57.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:57.485100+0000) 2024-03-06T16:05:58.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:58.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:58.485401+0000) 2024-03-06T16:05:59.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:05:59.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:59.485757+0000) 2024-03-06T16:06:00.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:00.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:00.486034+0000) 2024-03-06T16:06:01.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:01.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:01.486275+0000) 2024-03-06T16:06:02.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:02.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:02.486521+0000) 2024-03-06T16:06:03.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:03.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:03.486778+0000) 2024-03-06T16:06:04.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:04.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:04.487042+0000) 2024-03-06T16:06:05.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:05.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:05.487286+0000) 2024-03-06T16:06:06.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:06.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:06.487467+0000) 2024-03-06T16:06:07.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:07.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:07.487747+0000) 2024-03-06T16:06:08.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:08.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:08.488002+0000) 2024-03-06T16:06:09.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:09.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:09.488253+0000) 2024-03-06T16:06:10.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:10.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:10.488515+0000) 2024-03-06T16:06:11.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:11.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:11.488775+0000) 2024-03-06T16:06:12.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:12.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:12.489021+0000) 2024-03-06T16:06:13.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:13.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:13.489254+0000) 2024-03-06T16:06:14.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:14.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:14.489496+0000) 2024-03-06T16:06:15.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:15.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:15.489738+0000) 2024-03-06T16:06:16.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:16.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:16.490001+0000) 2024-03-06T16:06:17.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:17.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:17.490185+0000) 2024-03-06T16:06:18.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:18.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:18.490366+0000) 2024-03-06T16:06:19.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:19.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:19.490623+0000) 2024-03-06T16:06:20.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:20.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:20.490925+0000) 2024-03-06T16:06:21.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:21.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:21.491163+0000) 2024-03-06T16:06:22.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:22.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:22.491427+0000) 2024-03-06T16:06:23.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:23.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:23.491687+0000) 2024-03-06T16:06:24.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:24.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:24.491922+0000) 2024-03-06T16:06:25.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:25.110 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:25.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:25.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:25.492197+0000) 2024-03-06T16:06:26.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:26.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:26.492391+0000) 2024-03-06T16:06:27.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:27.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:27.492673+0000) 2024-03-06T16:06:28.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:28.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:28.492925+0000) 2024-03-06T16:06:29.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:29.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:29.493156+0000) 2024-03-06T16:06:30.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:30.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:30.493405+0000) 2024-03-06T16:06:31.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:31.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:31.493700+0000) 2024-03-06T16:06:32.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:32.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:32.493938+0000) 2024-03-06T16:06:33.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:33.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:33.494179+0000) 2024-03-06T16:06:34.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:34.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:34.494417+0000) 2024-03-06T16:06:35.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:35.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:35.494701+0000) 2024-03-06T16:06:36.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:36.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:36.494962+0000) 2024-03-06T16:06:37.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:37.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:37.495274+0000) 2024-03-06T16:06:38.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:38.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:38.495535+0000) 2024-03-06T16:06:39.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:39.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:39.495762+0000) 2024-03-06T16:06:40.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:40.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:40.496023+0000) 2024-03-06T16:06:41.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:41.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:41.496270+0000) 2024-03-06T16:06:42.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:42.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:42.496545+0000) 2024-03-06T16:06:43.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:43.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:43.496790+0000) 2024-03-06T16:06:44.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:44.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:44.497020+0000) 2024-03-06T16:06:45.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:45.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:45.497216+0000) 2024-03-06T16:06:46.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:46.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:46.497469+0000) 2024-03-06T16:06:47.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:47.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:47.497681+0000) 2024-03-06T16:06:48.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:48.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:48.497927+0000) 2024-03-06T16:06:49.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:49.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:49.498123+0000) 2024-03-06T16:06:50.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:50.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:50.498365+0000) 2024-03-06T16:06:51.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:51.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:51.498623+0000) 2024-03-06T16:06:52.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:52.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:52.498904+0000) 2024-03-06T16:06:53.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:53.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:53.499159+0000) 2024-03-06T16:06:54.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:54.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:54.499393+0000) 2024-03-06T16:06:55.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:55.258 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:55.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:55.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:55.499663+0000) 2024-03-06T16:06:56.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:56.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:56.499895+0000) 2024-03-06T16:06:57.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:57.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:57.500154+0000) 2024-03-06T16:06:58.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:58.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:58.500411+0000) 2024-03-06T16:06:59.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:06:59.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:59.500598+0000) 2024-03-06T16:07:00.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:00.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:00.500862+0000) 2024-03-06T16:07:01.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:01.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:01.501055+0000) 2024-03-06T16:07:02.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:02.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:02.501323+0000) 2024-03-06T16:07:03.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:03.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:03.501546+0000) 2024-03-06T16:07:04.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:04.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:04.501785+0000) 2024-03-06T16:07:05.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:05.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:05.502021+0000) 2024-03-06T16:07:06.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:06.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:06.502284+0000) 2024-03-06T16:07:07.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:07.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:07.502509+0000) 2024-03-06T16:07:08.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:08.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:08.502763+0000) 2024-03-06T16:07:09.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:09.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:09.503089+0000) 2024-03-06T16:07:10.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:10.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:10.503332+0000) 2024-03-06T16:07:11.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:11.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:11.503562+0000) 2024-03-06T16:07:12.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:12.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:12.503830+0000) 2024-03-06T16:07:13.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:13.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:13.504066+0000) 2024-03-06T16:07:14.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:14.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:14.504299+0000) 2024-03-06T16:07:15.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:15.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:15.504547+0000) 2024-03-06T16:07:16.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:16.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:16.504795+0000) 2024-03-06T16:07:17.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:17.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:17.505048+0000) 2024-03-06T16:07:18.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:18.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:18.505285+0000) 2024-03-06T16:07:19.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:19.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:19.505535+0000) 2024-03-06T16:07:20.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:20.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:20.505799+0000) 2024-03-06T16:07:21.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:21.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:21.506031+0000) 2024-03-06T16:07:22.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:22.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:22.506296+0000) 2024-03-06T16:07:23.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:23.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:23.506549+0000) 2024-03-06T16:07:24.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:24.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:24.506795+0000) 2024-03-06T16:07:25.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:25.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:25.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:25.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:25.507031+0000) 2024-03-06T16:07:26.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:26.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:26.507279+0000) 2024-03-06T16:07:27.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:27.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:27.507527+0000) 2024-03-06T16:07:28.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:28.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:28.507782+0000) 2024-03-06T16:07:29.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:29.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:29.508036+0000) 2024-03-06T16:07:30.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:30.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:30.508224+0000) 2024-03-06T16:07:31.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:31.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:31.508475+0000) 2024-03-06T16:07:32.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:32.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:32.508743+0000) 2024-03-06T16:07:33.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:33.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:33.508972+0000) 2024-03-06T16:07:34.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:34.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:34.509220+0000) 2024-03-06T16:07:35.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:35.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:35.509451+0000) 2024-03-06T16:07:36.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:36.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:36.509748+0000) 2024-03-06T16:07:37.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:37.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:37.509989+0000) 2024-03-06T16:07:38.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:38.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:38.510244+0000) 2024-03-06T16:07:39.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:39.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:39.510479+0000) 2024-03-06T16:07:40.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:40.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:40.510732+0000) 2024-03-06T16:07:41.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:41.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:41.510970+0000) 2024-03-06T16:07:42.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:42.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:42.511232+0000) 2024-03-06T16:07:43.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:43.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:43.511462+0000) 2024-03-06T16:07:44.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:44.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:44.511735+0000) 2024-03-06T16:07:45.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:45.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:45.511911+0000) 2024-03-06T16:07:46.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:46.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:46.512160+0000) 2024-03-06T16:07:47.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:47.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:47.512418+0000) 2024-03-06T16:07:48.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:48.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:48.512714+0000) 2024-03-06T16:07:49.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:49.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:49.512963+0000) 2024-03-06T16:07:50.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:50.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:50.513200+0000) 2024-03-06T16:07:51.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:51.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:51.513431+0000) 2024-03-06T16:07:52.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:52.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:52.513657+0000) 2024-03-06T16:07:53.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:53.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:53.513848+0000) 2024-03-06T16:07:54.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:54.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:54.514036+0000) 2024-03-06T16:07:55.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:55.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:55.514212+0000) 2024-03-06T16:07:55.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:55.548 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:56.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:56.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:56.514393+0000) 2024-03-06T16:07:57.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:57.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:57.514745+0000) 2024-03-06T16:07:58.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:58.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:58.514990+0000) 2024-03-06T16:07:59.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:07:59.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:59.515167+0000) 2024-03-06T16:08:00.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:00.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:00.515424+0000) 2024-03-06T16:08:01.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:01.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:01.515728+0000) 2024-03-06T16:08:02.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:02.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:02.515979+0000) 2024-03-06T16:08:03.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:03.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:03.516228+0000) 2024-03-06T16:08:04.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:04.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:04.516491+0000) 2024-03-06T16:08:05.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:05.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:05.516745+0000) 2024-03-06T16:08:06.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:06.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:06.516989+0000) 2024-03-06T16:08:07.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:07.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:07.517242+0000) 2024-03-06T16:08:08.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:08.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:08.517498+0000) 2024-03-06T16:08:09.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:09.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:09.517731+0000) 2024-03-06T16:08:10.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:10.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:10.517926+0000) 2024-03-06T16:08:11.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:11.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:11.518157+0000) 2024-03-06T16:08:12.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:12.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:12.518409+0000) 2024-03-06T16:08:13.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:13.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:13.518682+0000) 2024-03-06T16:08:14.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:14.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:14.518915+0000) 2024-03-06T16:08:15.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:15.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:15.519147+0000) 2024-03-06T16:08:16.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:16.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:16.519399+0000) 2024-03-06T16:08:17.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:17.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:17.519675+0000) 2024-03-06T16:08:18.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:18.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:18.519939+0000) 2024-03-06T16:08:19.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:19.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:19.520189+0000) 2024-03-06T16:08:20.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:20.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:20.520449+0000) 2024-03-06T16:08:21.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:21.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:21.520738+0000) 2024-03-06T16:08:22.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:22.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:22.521004+0000) 2024-03-06T16:08:23.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:23.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:23.521255+0000) 2024-03-06T16:08:24.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:24.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:24.521496+0000) 2024-03-06T16:08:25.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:25.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:25.521753+0000) 2024-03-06T16:08:25.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:25.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:26.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:26.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:26.522009+0000) 2024-03-06T16:08:27.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:27.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:27.522198+0000) 2024-03-06T16:08:28.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:28.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:28.522439+0000) 2024-03-06T16:08:29.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:29.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:29.522729+0000) 2024-03-06T16:08:30.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:30.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:30.522967+0000) 2024-03-06T16:08:31.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:31.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:31.523217+0000) 2024-03-06T16:08:32.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:32.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:32.523497+0000) 2024-03-06T16:08:33.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:33.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:33.523732+0000) 2024-03-06T16:08:34.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:34.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:34.523962+0000) 2024-03-06T16:08:35.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:35.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:35.524211+0000) 2024-03-06T16:08:36.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:36.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:36.524465+0000) 2024-03-06T16:08:37.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:37.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:37.524751+0000) 2024-03-06T16:08:38.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:38.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:38.524979+0000) 2024-03-06T16:08:39.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:39.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:39.525232+0000) 2024-03-06T16:08:40.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:40.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:40.525492+0000) 2024-03-06T16:08:41.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:41.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:41.525706+0000) 2024-03-06T16:08:42.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:42.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:42.525969+0000) 2024-03-06T16:08:43.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:43.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:43.526232+0000) 2024-03-06T16:08:44.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:44.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:44.526467+0000) 2024-03-06T16:08:45.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:45.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:45.526798+0000) 2024-03-06T16:08:46.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:46.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:46.527056+0000) 2024-03-06T16:08:47.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:47.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:47.527304+0000) 2024-03-06T16:08:48.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:48.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:48.527535+0000) 2024-03-06T16:08:49.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:49.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:49.527788+0000) 2024-03-06T16:08:50.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:50.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:50.528032+0000) 2024-03-06T16:08:51.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:51.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:51.528285+0000) 2024-03-06T16:08:52.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:52.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:52.528544+0000) 2024-03-06T16:08:53.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:53.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:53.528796+0000) 2024-03-06T16:08:54.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:54.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:54.529077+0000) 2024-03-06T16:08:55.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:55.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:55.529349+0000) 2024-03-06T16:08:55.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:55.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:56.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:56.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:56.529609+0000) 2024-03-06T16:08:57.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:57.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:57.529911+0000) 2024-03-06T16:08:58.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:58.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:58.530165+0000) 2024-03-06T16:08:59.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:08:59.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:59.530389+0000) 2024-03-06T16:09:00.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:00.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:00.530691+0000) 2024-03-06T16:09:01.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:01.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:01.530965+0000) 2024-03-06T16:09:02.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:02.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:02.531230+0000) 2024-03-06T16:09:03.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:03.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:03.531462+0000) 2024-03-06T16:09:04.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:04.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:04.531745+0000) 2024-03-06T16:09:05.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:05.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:05.531993+0000) 2024-03-06T16:09:06.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:06.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:06.532251+0000) 2024-03-06T16:09:07.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:07.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:07.532501+0000) 2024-03-06T16:09:08.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:08.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:08.532747+0000) 2024-03-06T16:09:09.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:09.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:09.532999+0000) 2024-03-06T16:09:10.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:10.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:10.533250+0000) 2024-03-06T16:09:11.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:11.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:11.533512+0000) 2024-03-06T16:09:12.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:12.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:12.533777+0000) 2024-03-06T16:09:13.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:13.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:13.533975+0000) 2024-03-06T16:09:14.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:14.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:14.534234+0000) 2024-03-06T16:09:15.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:15.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:15.534490+0000) 2024-03-06T16:09:16.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:16.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:16.534770+0000) 2024-03-06T16:09:17.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:17.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:17.535024+0000) 2024-03-06T16:09:18.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:18.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:18.535285+0000) 2024-03-06T16:09:19.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:19.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:19.535543+0000) 2024-03-06T16:09:20.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:20.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:20.535800+0000) 2024-03-06T16:09:21.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:21.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:21.536073+0000) 2024-03-06T16:09:22.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:22.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:22.536336+0000) 2024-03-06T16:09:23.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:23.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:23.536536+0000) 2024-03-06T16:09:24.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:24.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:24.536811+0000) 2024-03-06T16:09:25.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:25.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:25.537075+0000) 2024-03-06T16:09:26.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:26.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:26.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:26.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:26.537325+0000) 2024-03-06T16:09:27.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:27.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:27.537571+0000) 2024-03-06T16:09:28.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:28.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:28.537833+0000) 2024-03-06T16:09:29.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:29.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:29.538089+0000) 2024-03-06T16:09:30.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:30.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:30.538361+0000) 2024-03-06T16:09:31.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:31.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:31.538612+0000) 2024-03-06T16:09:32.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:32.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:32.538904+0000) 2024-03-06T16:09:33.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:33.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:33.539139+0000) 2024-03-06T16:09:34.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:34.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:34.539484+0000) 2024-03-06T16:09:35.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:35.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:35.539782+0000) 2024-03-06T16:09:36.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:36.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:36.540040+0000) 2024-03-06T16:09:37.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:37.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:37.540258+0000) 2024-03-06T16:09:38.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:38.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:38.540572+0000) 2024-03-06T16:09:39.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:39.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:39.540871+0000) 2024-03-06T16:09:40.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:40.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:40.541165+0000) 2024-03-06T16:09:41.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:41.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:41.541453+0000) 2024-03-06T16:09:42.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:42.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:42.541795+0000) 2024-03-06T16:09:43.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:43.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:43.542087+0000) 2024-03-06T16:09:44.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:44.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:44.542385+0000) 2024-03-06T16:09:45.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:45.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:45.542762+0000) 2024-03-06T16:09:46.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:46.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:46.543031+0000) 2024-03-06T16:09:47.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:47.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:47.543283+0000) 2024-03-06T16:09:48.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:48.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:48.543539+0000) 2024-03-06T16:09:49.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:49.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:49.543806+0000) 2024-03-06T16:09:50.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:50.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:50.544048+0000) 2024-03-06T16:09:51.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:51.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:51.544300+0000) 2024-03-06T16:09:52.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:52.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:52.544544+0000) 2024-03-06T16:09:53.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:53.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:53.544793+0000) 2024-03-06T16:09:54.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:54.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:54.544991+0000) 2024-03-06T16:09:55.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:55.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:55.545246+0000) 2024-03-06T16:09:56.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:56.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:56.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:56.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:56.545488+0000) 2024-03-06T16:09:57.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:57.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:57.545739+0000) 2024-03-06T16:09:58.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:58.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:58.545985+0000) 2024-03-06T16:09:59.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:09:59.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:59.546214+0000) 2024-03-06T16:10:00.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:00.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:00.546457+0000) 2024-03-06T16:10:01.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:01.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:01.546751+0000) 2024-03-06T16:10:02.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:02.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:02.547014+0000) 2024-03-06T16:10:03.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:03.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:03.547262+0000) 2024-03-06T16:10:04.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:04.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:04.547513+0000) 2024-03-06T16:10:05.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:05.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:05.547768+0000) 2024-03-06T16:10:06.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:06.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:06.548038+0000) 2024-03-06T16:10:07.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:07.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:07.548328+0000) 2024-03-06T16:10:08.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:08.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:08.548623+0000) 2024-03-06T16:10:09.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:09.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:09.548957+0000) 2024-03-06T16:10:10.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:10.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:10.549204+0000) 2024-03-06T16:10:11.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:11.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:11.549491+0000) 2024-03-06T16:10:12.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:12.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:12.549815+0000) 2024-03-06T16:10:13.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:13.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:13.550112+0000) 2024-03-06T16:10:14.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:14.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:14.550426+0000) 2024-03-06T16:10:15.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:15.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:15.550779+0000) 2024-03-06T16:10:16.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:16.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:16.551088+0000) 2024-03-06T16:10:17.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:17.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:17.551367+0000) 2024-03-06T16:10:18.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:18.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:18.551694+0000) 2024-03-06T16:10:19.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:19.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:19.552004+0000) 2024-03-06T16:10:20.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:20.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:20.552301+0000) 2024-03-06T16:10:21.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:21.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:21.552605+0000) 2024-03-06T16:10:22.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:22.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:22.552950+0000) 2024-03-06T16:10:23.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:23.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:23.553234+0000) 2024-03-06T16:10:24.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:24.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:24.553522+0000) 2024-03-06T16:10:25.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:25.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:25.553831+0000) 2024-03-06T16:10:26.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:26.292 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:26.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:26.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:26.554157+0000) 2024-03-06T16:10:27.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:27.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:27.554524+0000) 2024-03-06T16:10:28.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:28.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:28.554838+0000) 2024-03-06T16:10:29.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:29.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:29.555133+0000) 2024-03-06T16:10:30.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:30.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:30.555423+0000) 2024-03-06T16:10:31.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:31.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:31.555754+0000) 2024-03-06T16:10:32.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:32.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:32.556082+0000) 2024-03-06T16:10:33.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:33.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:33.556364+0000) 2024-03-06T16:10:34.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:34.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:34.556708+0000) 2024-03-06T16:10:35.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:35.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:35.557010+0000) 2024-03-06T16:10:36.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:36.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:36.557326+0000) 2024-03-06T16:10:37.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:37.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:37.557670+0000) 2024-03-06T16:10:38.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:38.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:38.557989+0000) 2024-03-06T16:10:39.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:39.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:39.558286+0000) 2024-03-06T16:10:40.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:40.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:40.558615+0000) 2024-03-06T16:10:41.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:41.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:41.558968+0000) 2024-03-06T16:10:42.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:42.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:42.559227+0000) 2024-03-06T16:10:43.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:43.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:43.559483+0000) 2024-03-06T16:10:44.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:44.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:44.559793+0000) 2024-03-06T16:10:45.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:45.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:45.560077+0000) 2024-03-06T16:10:46.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:46.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:46.560358+0000) 2024-03-06T16:10:47.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:47.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:47.560594+0000) 2024-03-06T16:10:48.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:48.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:48.560842+0000) 2024-03-06T16:10:49.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:49.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:49.561130+0000) 2024-03-06T16:10:50.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:50.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:50.561449+0000) 2024-03-06T16:10:51.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:51.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:51.561781+0000) 2024-03-06T16:10:52.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:52.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:52.562096+0000) 2024-03-06T16:10:53.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:53.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:53.562374+0000) 2024-03-06T16:10:54.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:54.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:54.562722+0000) 2024-03-06T16:10:55.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:55.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:55.563030+0000) 2024-03-06T16:10:56.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:56.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:56.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:56.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:56.563354+0000) 2024-03-06T16:10:57.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:57.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:57.563679+0000) 2024-03-06T16:10:58.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:58.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:58.563970+0000) 2024-03-06T16:10:59.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:10:59.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:59.564278+0000) 2024-03-06T16:11:00.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:00.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:00.564578+0000) 2024-03-06T16:11:01.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:01.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:01.564888+0000) 2024-03-06T16:11:02.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:02.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:02.565178+0000) 2024-03-06T16:11:03.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:03.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:03.565457+0000) 2024-03-06T16:11:04.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:04.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:04.565778+0000) 2024-03-06T16:11:05.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:05.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:05.566003+0000) 2024-03-06T16:11:06.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:06.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:06.566312+0000) 2024-03-06T16:11:07.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:07.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:07.566606+0000) 2024-03-06T16:11:08.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:08.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:08.566977+0000) 2024-03-06T16:11:09.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:09.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:09.567284+0000) 2024-03-06T16:11:10.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:10.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:10.567593+0000) 2024-03-06T16:11:11.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:11.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:11.567951+0000) 2024-03-06T16:11:12.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:12.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:12.568246+0000) 2024-03-06T16:11:13.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:13.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:13.568554+0000) 2024-03-06T16:11:14.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:14.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:14.568855+0000) 2024-03-06T16:11:15.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:15.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:15.569195+0000) 2024-03-06T16:11:16.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:16.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:16.569515+0000) 2024-03-06T16:11:17.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:17.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:17.569827+0000) 2024-03-06T16:11:18.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:18.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:18.570147+0000) 2024-03-06T16:11:19.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:19.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:19.570437+0000) 2024-03-06T16:11:20.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:20.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:20.570738+0000) 2024-03-06T16:11:21.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:21.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:21.571004+0000) 2024-03-06T16:11:22.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:22.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:22.571306+0000) 2024-03-06T16:11:23.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:23.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:23.571595+0000) 2024-03-06T16:11:24.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:24.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:24.571934+0000) 2024-03-06T16:11:25.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:25.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:25.572238+0000) 2024-03-06T16:11:26.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:26.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:26.572555+0000) 2024-03-06T16:11:26.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:26.578 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:27.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:27.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:27.572868+0000) 2024-03-06T16:11:28.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:28.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:28.573144+0000) 2024-03-06T16:11:29.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:29.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:29.573431+0000) 2024-03-06T16:11:30.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:30.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:30.573780+0000) 2024-03-06T16:11:31.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:31.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:31.574085+0000) 2024-03-06T16:11:32.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:32.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:32.574412+0000) 2024-03-06T16:11:33.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:33.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:33.574737+0000) 2024-03-06T16:11:34.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:34.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:34.575050+0000) 2024-03-06T16:11:35.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:35.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:35.575327+0000) 2024-03-06T16:11:36.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:36.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:36.575685+0000) 2024-03-06T16:11:37.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:37.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:37.576010+0000) 2024-03-06T16:11:38.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:38.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:38.576263+0000) 2024-03-06T16:11:39.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:39.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:39.576563+0000) 2024-03-06T16:11:40.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:40.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:40.576880+0000) 2024-03-06T16:11:41.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:41.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:41.577172+0000) 2024-03-06T16:11:42.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:42.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:42.577490+0000) 2024-03-06T16:11:43.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:43.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:43.577786+0000) 2024-03-06T16:11:44.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:44.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:44.578088+0000) 2024-03-06T16:11:45.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:45.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:45.578408+0000) 2024-03-06T16:11:46.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:46.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:46.578750+0000) 2024-03-06T16:11:47.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:47.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:47.578989+0000) 2024-03-06T16:11:48.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:48.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:48.579244+0000) 2024-03-06T16:11:49.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:49.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:49.579535+0000) 2024-03-06T16:11:50.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:50.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:50.579851+0000) 2024-03-06T16:11:51.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:51.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:51.580163+0000) 2024-03-06T16:11:52.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:52.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:52.580468+0000) 2024-03-06T16:11:53.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:53.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:53.580784+0000) 2024-03-06T16:11:54.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:54.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:54.581074+0000) 2024-03-06T16:11:55.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:55.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:55.581357+0000) 2024-03-06T16:11:56.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:56.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:56.581696+0000) 2024-03-06T16:11:56.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:56.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:57.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:57.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:57.582012+0000) 2024-03-06T16:11:58.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:58.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:58.582263+0000) 2024-03-06T16:11:59.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:11:59.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:59.582567+0000) 2024-03-06T16:12:00.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:00.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:00.582858+0000) 2024-03-06T16:12:01.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:01.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:01.583125+0000) 2024-03-06T16:12:02.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:02.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:02.583374+0000) 2024-03-06T16:12:03.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:03.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:03.583717+0000) 2024-03-06T16:12:04.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:04.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:04.584035+0000) 2024-03-06T16:12:05.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:05.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:05.584331+0000) 2024-03-06T16:12:06.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:06.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:06.584683+0000) 2024-03-06T16:12:07.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:07.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:07.584991+0000) 2024-03-06T16:12:08.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:08.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:08.585337+0000) 2024-03-06T16:12:09.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:09.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:09.585625+0000) 2024-03-06T16:12:10.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:10.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:10.585984+0000) 2024-03-06T16:12:11.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:11.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:11.586277+0000) 2024-03-06T16:12:12.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:12.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:12.586572+0000) 2024-03-06T16:12:13.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:13.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:13.586875+0000) 2024-03-06T16:12:14.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:14.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:14.587165+0000) 2024-03-06T16:12:15.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:15.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:15.587452+0000) 2024-03-06T16:12:16.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:16.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:16.587792+0000) 2024-03-06T16:12:17.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:17.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:17.588076+0000) 2024-03-06T16:12:18.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:18.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:18.588394+0000) 2024-03-06T16:12:19.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:19.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:19.588743+0000) 2024-03-06T16:12:20.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:20.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:20.589031+0000) 2024-03-06T16:12:21.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:21.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:21.589334+0000) 2024-03-06T16:12:22.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:22.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:22.589689+0000) 2024-03-06T16:12:23.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:23.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:23.589975+0000) 2024-03-06T16:12:24.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:24.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:24.590296+0000) 2024-03-06T16:12:25.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:25.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:25.590602+0000) 2024-03-06T16:12:26.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:26.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:26.590940+0000) 2024-03-06T16:12:26.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:26.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:27.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:27.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:27.591251+0000) 2024-03-06T16:12:28.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:28.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:28.591501+0000) 2024-03-06T16:12:29.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:29.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:29.591815+0000) 2024-03-06T16:12:30.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:30.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:30.592132+0000) 2024-03-06T16:12:31.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:31.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:31.592445+0000) 2024-03-06T16:12:32.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:32.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:32.592793+0000) 2024-03-06T16:12:33.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:33.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:33.593090+0000) 2024-03-06T16:12:34.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:34.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:34.593387+0000) 2024-03-06T16:12:35.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:35.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:35.593716+0000) 2024-03-06T16:12:36.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:36.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:36.594024+0000) 2024-03-06T16:12:37.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:37.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:37.594306+0000) 2024-03-06T16:12:38.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:38.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:38.594576+0000) 2024-03-06T16:12:39.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:39.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:39.594820+0000) 2024-03-06T16:12:40.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:40.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:40.595068+0000) 2024-03-06T16:12:41.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:41.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:41.595307+0000) 2024-03-06T16:12:42.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:42.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:42.595598+0000) 2024-03-06T16:12:43.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:43.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:43.595801+0000) 2024-03-06T16:12:44.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:44.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:44.596126+0000) 2024-03-06T16:12:45.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:45.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:45.596373+0000) 2024-03-06T16:12:46.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:46.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:46.596703+0000) 2024-03-06T16:12:47.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:47.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:47.596962+0000) 2024-03-06T16:12:48.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:48.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:48.597223+0000) 2024-03-06T16:12:49.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:49.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:49.597483+0000) 2024-03-06T16:12:50.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:50.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:50.597865+0000) 2024-03-06T16:12:51.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:51.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:51.598099+0000) 2024-03-06T16:12:52.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:52.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:52.598431+0000) 2024-03-06T16:12:53.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:53.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:53.598750+0000) 2024-03-06T16:12:54.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:54.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:54.598986+0000) 2024-03-06T16:12:55.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:55.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:55.599214+0000) 2024-03-06T16:12:56.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:56.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:56.599566+0000) 2024-03-06T16:12:57.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:57.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:57.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:57.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:57.599836+0000) 2024-03-06T16:12:58.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:58.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:58.600072+0000) 2024-03-06T16:12:59.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:12:59.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:59.600307+0000) 2024-03-06T16:13:00.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:00.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:00.600612+0000) 2024-03-06T16:13:01.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:01.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:01.600884+0000) 2024-03-06T16:13:02.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:02.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:02.601191+0000) 2024-03-06T16:13:03.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:03.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:03.601469+0000) 2024-03-06T16:13:04.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:04.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:04.601784+0000) 2024-03-06T16:13:05.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:05.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:05.602089+0000) 2024-03-06T16:13:06.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:06.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:06.602384+0000) 2024-03-06T16:13:07.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:07.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:07.602714+0000) 2024-03-06T16:13:08.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:08.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:08.603003+0000) 2024-03-06T16:13:09.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:09.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:09.603299+0000) 2024-03-06T16:13:10.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:10.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:10.603620+0000) 2024-03-06T16:13:11.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:11.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:11.603924+0000) 2024-03-06T16:13:12.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:12.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:12.604242+0000) 2024-03-06T16:13:13.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:13.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:13.604540+0000) 2024-03-06T16:13:14.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:14.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:14.604850+0000) 2024-03-06T16:13:15.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:15.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:15.605079+0000) 2024-03-06T16:13:16.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:16.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:16.605389+0000) 2024-03-06T16:13:17.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:17.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:17.605737+0000) 2024-03-06T16:13:18.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:18.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:18.606053+0000) 2024-03-06T16:13:19.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:19.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:19.606347+0000) 2024-03-06T16:13:20.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:20.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:20.606695+0000) 2024-03-06T16:13:21.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:21.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:21.606996+0000) 2024-03-06T16:13:22.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:22.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:22.607327+0000) 2024-03-06T16:13:23.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:23.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:23.607677+0000) 2024-03-06T16:13:24.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:24.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:24.607974+0000) 2024-03-06T16:13:25.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:25.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:25.608271+0000) 2024-03-06T16:13:26.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:26.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:26.608499+0000) 2024-03-06T16:13:27.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:27.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:27.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:27.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:27.608819+0000) 2024-03-06T16:13:28.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:28.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:28.609075+0000) 2024-03-06T16:13:29.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:29.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:29.609377+0000) 2024-03-06T16:13:30.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:30.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:30.609726+0000) 2024-03-06T16:13:31.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:31.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:31.610022+0000) 2024-03-06T16:13:32.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:32.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:32.610325+0000) 2024-03-06T16:13:33.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:33.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:33.610558+0000) 2024-03-06T16:13:34.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:34.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:34.610805+0000) 2024-03-06T16:13:35.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:35.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:35.611056+0000) 2024-03-06T16:13:36.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:36.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:36.611308+0000) 2024-03-06T16:13:37.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:37.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:37.611545+0000) 2024-03-06T16:13:38.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:38.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:38.611791+0000) 2024-03-06T16:13:39.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:39.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:39.612050+0000) 2024-03-06T16:13:40.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:40.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:40.612286+0000) 2024-03-06T16:13:41.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:41.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:41.612527+0000) 2024-03-06T16:13:42.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:42.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:42.612795+0000) 2024-03-06T16:13:43.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:43.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:43.613024+0000) 2024-03-06T16:13:44.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:44.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:44.613267+0000) 2024-03-06T16:13:45.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:45.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:45.613444+0000) 2024-03-06T16:13:46.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:46.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:46.613745+0000) 2024-03-06T16:13:47.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:47.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:47.614003+0000) 2024-03-06T16:13:48.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:48.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:48.614239+0000) 2024-03-06T16:13:49.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:49.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:49.614472+0000) 2024-03-06T16:13:50.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:50.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:50.614741+0000) 2024-03-06T16:13:51.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:51.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:51.614987+0000) 2024-03-06T16:13:52.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:52.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:52.615237+0000) 2024-03-06T16:13:53.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:53.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:53.615529+0000) 2024-03-06T16:13:54.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:54.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:54.615835+0000) 2024-03-06T16:13:55.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:55.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:55.616073+0000) 2024-03-06T16:13:56.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:56.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:56.616368+0000) 2024-03-06T16:13:57.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:57.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:57.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:57.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:57.616698+0000) 2024-03-06T16:13:58.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:58.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:58.617009+0000) 2024-03-06T16:13:59.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:13:59.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:59.617233+0000) 2024-03-06T16:14:00.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:00.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:00.617540+0000) 2024-03-06T16:14:01.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:01.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:01.617833+0000) 2024-03-06T16:14:02.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:02.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:02.618099+0000) 2024-03-06T16:14:03.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:03.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:03.618412+0000) 2024-03-06T16:14:04.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:04.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:04.618762+0000) 2024-03-06T16:14:05.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:05.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:05.619073+0000) 2024-03-06T16:14:06.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:06.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:06.619355+0000) 2024-03-06T16:14:07.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:07.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:07.619708+0000) 2024-03-06T16:14:08.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:08.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:08.620014+0000) 2024-03-06T16:14:09.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:09.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:09.620294+0000) 2024-03-06T16:14:10.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:10.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:10.620581+0000) 2024-03-06T16:14:11.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:11.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:11.620909+0000) 2024-03-06T16:14:12.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:12.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:12.621157+0000) 2024-03-06T16:14:13.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:13.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:13.621451+0000) 2024-03-06T16:14:14.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:14.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:14.621766+0000) 2024-03-06T16:14:15.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:15.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:15.622059+0000) 2024-03-06T16:14:16.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:16.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:16.622360+0000) 2024-03-06T16:14:17.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:17.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:17.622716+0000) 2024-03-06T16:14:18.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:18.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:18.623003+0000) 2024-03-06T16:14:19.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:19.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:19.623236+0000) 2024-03-06T16:14:20.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:20.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:20.623548+0000) 2024-03-06T16:14:21.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:21.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:21.623845+0000) 2024-03-06T16:14:22.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:22.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:22.624139+0000) 2024-03-06T16:14:23.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:23.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:23.624439+0000) 2024-03-06T16:14:24.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:24.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:24.624775+0000) 2024-03-06T16:14:25.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:25.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:25.625055+0000) 2024-03-06T16:14:26.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:26.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:26.625340+0000) 2024-03-06T16:14:27.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:27.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:27.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:27.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:27.625624+0000) 2024-03-06T16:14:28.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:28.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:28.625879+0000) 2024-03-06T16:14:29.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:29.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:29.626118+0000) 2024-03-06T16:14:30.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:30.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:30.626378+0000) 2024-03-06T16:14:31.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:31.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:31.626610+0000) 2024-03-06T16:14:32.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:32.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:32.626875+0000) 2024-03-06T16:14:33.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:33.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:33.627067+0000) 2024-03-06T16:14:34.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:34.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:34.627314+0000) 2024-03-06T16:14:35.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:35.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:35.627558+0000) 2024-03-06T16:14:36.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:36.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:36.627820+0000) 2024-03-06T16:14:37.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:37.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:37.628161+0000) 2024-03-06T16:14:38.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:38.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:38.628476+0000) 2024-03-06T16:14:39.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:39.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:39.628782+0000) 2024-03-06T16:14:40.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:40.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:40.629078+0000) 2024-03-06T16:14:41.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:41.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:41.629389+0000) 2024-03-06T16:14:42.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:42.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:42.629749+0000) 2024-03-06T16:14:43.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:43.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:43.629977+0000) 2024-03-06T16:14:44.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:44.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:44.630223+0000) 2024-03-06T16:14:45.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:45.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:45.630536+0000) 2024-03-06T16:14:46.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:46.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:46.630842+0000) 2024-03-06T16:14:47.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:47.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:47.631109+0000) 2024-03-06T16:14:48.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:48.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:48.631353+0000) 2024-03-06T16:14:49.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:49.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:49.631583+0000) 2024-03-06T16:14:50.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:50.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:50.631828+0000) 2024-03-06T16:14:51.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:51.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:51.632068+0000) 2024-03-06T16:14:52.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:52.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:52.632341+0000) 2024-03-06T16:14:53.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:53.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:53.632586+0000) 2024-03-06T16:14:54.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:54.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:54.632838+0000) 2024-03-06T16:14:55.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:55.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:55.633107+0000) 2024-03-06T16:14:56.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:56.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:56.633403+0000) 2024-03-06T16:14:57.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:57.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:57.633625+0000) 2024-03-06T16:14:57.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:57.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:58.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:58.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:58.633975+0000) 2024-03-06T16:14:59.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:14:59.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:59.634293+0000) 2024-03-06T16:15:00.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:00.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:00.634589+0000) 2024-03-06T16:15:01.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:01.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:01.634938+0000) 2024-03-06T16:15:02.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:02.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:02.635244+0000) 2024-03-06T16:15:03.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:03.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:03.635553+0000) 2024-03-06T16:15:04.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:04.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:04.635861+0000) 2024-03-06T16:15:05.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:05.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:05.636179+0000) 2024-03-06T16:15:06.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:06.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:06.636455+0000) 2024-03-06T16:15:07.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:07.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:07.636779+0000) 2024-03-06T16:15:08.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:08.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:08.637030+0000) 2024-03-06T16:15:09.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:09.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:09.637342+0000) 2024-03-06T16:15:10.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:10.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:10.637701+0000) 2024-03-06T16:15:11.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:11.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:11.638015+0000) 2024-03-06T16:15:12.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:12.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:12.638334+0000) 2024-03-06T16:15:13.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:13.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:13.638617+0000) 2024-03-06T16:15:14.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:14.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:14.638962+0000) 2024-03-06T16:15:15.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:15.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:15.639211+0000) 2024-03-06T16:15:16.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:16.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:16.639518+0000) 2024-03-06T16:15:17.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:17.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:17.639837+0000) 2024-03-06T16:15:18.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:18.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:18.640132+0000) 2024-03-06T16:15:19.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:19.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:19.640409+0000) 2024-03-06T16:15:20.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:20.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:20.640752+0000) 2024-03-06T16:15:21.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:21.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:21.641073+0000) 2024-03-06T16:15:22.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:22.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:22.641396+0000) 2024-03-06T16:15:23.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:23.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:23.641721+0000) 2024-03-06T16:15:24.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:24.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:24.641971+0000) 2024-03-06T16:15:25.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:25.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:25.642272+0000) 2024-03-06T16:15:26.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:26.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:26.642479+0000) 2024-03-06T16:15:27.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:27.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:27.642753+0000) 2024-03-06T16:15:27.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:27.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:28.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:28.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:28.643034+0000) 2024-03-06T16:15:29.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:29.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:29.643290+0000) 2024-03-06T16:15:30.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:30.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:30.643567+0000) 2024-03-06T16:15:31.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:31.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:31.643849+0000) 2024-03-06T16:15:32.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:32.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:32.644116+0000) 2024-03-06T16:15:33.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:33.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:33.644322+0000) 2024-03-06T16:15:34.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:34.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:34.644582+0000) 2024-03-06T16:15:35.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:35.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:35.644837+0000) 2024-03-06T16:15:36.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:36.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:36.645110+0000) 2024-03-06T16:15:37.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:37.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:37.645387+0000) 2024-03-06T16:15:38.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:38.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:38.645701+0000) 2024-03-06T16:15:39.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:39.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:39.645973+0000) 2024-03-06T16:15:40.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:40.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:40.646235+0000) 2024-03-06T16:15:41.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:41.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:41.646488+0000) 2024-03-06T16:15:42.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:42.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:42.646765+0000) 2024-03-06T16:15:43.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:43.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:43.647037+0000) 2024-03-06T16:15:44.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:44.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:44.647289+0000) 2024-03-06T16:15:45.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:45.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:45.647552+0000) 2024-03-06T16:15:46.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:46.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:46.647810+0000) 2024-03-06T16:15:47.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:47.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:47.648047+0000) 2024-03-06T16:15:48.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:48.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:48.648277+0000) 2024-03-06T16:15:49.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:49.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:49.648527+0000) 2024-03-06T16:15:50.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:50.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:50.648781+0000) 2024-03-06T16:15:51.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:51.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:51.649011+0000) 2024-03-06T16:15:52.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:52.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:52.649279+0000) 2024-03-06T16:15:53.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:53.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:53.649528+0000) 2024-03-06T16:15:54.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:54.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:54.649761+0000) 2024-03-06T16:15:55.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:55.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:55.649938+0000) 2024-03-06T16:15:56.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:56.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:56.650178+0000) 2024-03-06T16:15:57.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:57.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:57.650432+0000) 2024-03-06T16:15:57.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:57.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:58.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:58.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:58.650754+0000) 2024-03-06T16:15:59.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:15:59.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:59.651001+0000) 2024-03-06T16:16:00.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:00.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:00.651261+0000) 2024-03-06T16:16:01.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:01.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:01.651499+0000) 2024-03-06T16:16:02.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:02.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:02.651778+0000) 2024-03-06T16:16:03.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:03.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:03.652020+0000) 2024-03-06T16:16:04.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:04.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:04.652274+0000) 2024-03-06T16:16:05.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:05.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:05.652546+0000) 2024-03-06T16:16:06.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:06.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:06.652809+0000) 2024-03-06T16:16:07.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:07.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:07.653067+0000) 2024-03-06T16:16:08.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:08.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:08.653331+0000) 2024-03-06T16:16:09.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:09.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:09.653599+0000) 2024-03-06T16:16:10.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:10.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:10.653882+0000) 2024-03-06T16:16:11.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:11.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:11.654132+0000) 2024-03-06T16:16:12.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:12.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:12.654515+0000) 2024-03-06T16:16:13.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:13.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:13.654843+0000) 2024-03-06T16:16:14.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:14.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:14.655079+0000) 2024-03-06T16:16:15.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:15.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:15.655311+0000) 2024-03-06T16:16:16.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:16.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:16.655563+0000) 2024-03-06T16:16:17.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:17.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:17.655808+0000) 2024-03-06T16:16:18.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:18.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:18.656072+0000) 2024-03-06T16:16:19.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:19.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:19.656321+0000) 2024-03-06T16:16:20.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:20.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:20.656576+0000) 2024-03-06T16:16:21.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:21.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:21.656827+0000) 2024-03-06T16:16:22.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:22.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:22.657089+0000) 2024-03-06T16:16:23.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:23.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:23.657349+0000) 2024-03-06T16:16:24.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:24.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:24.657591+0000) 2024-03-06T16:16:25.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:25.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:25.657846+0000) 2024-03-06T16:16:26.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:26.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:26.658104+0000) 2024-03-06T16:16:27.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:27.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:27.658344+0000) 2024-03-06T16:16:28.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:28.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:28.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:28.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:28.658594+0000) 2024-03-06T16:16:29.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:29.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:29.658853+0000) 2024-03-06T16:16:30.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:30.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:30.659112+0000) 2024-03-06T16:16:31.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:31.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:31.659361+0000) 2024-03-06T16:16:32.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:32.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:32.659611+0000) 2024-03-06T16:16:33.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:33.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:33.659897+0000) 2024-03-06T16:16:34.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:34.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:34.660118+0000) 2024-03-06T16:16:35.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:35.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:35.660377+0000) 2024-03-06T16:16:36.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:36.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:36.660674+0000) 2024-03-06T16:16:37.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:37.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:37.660950+0000) 2024-03-06T16:16:38.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:38.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:38.661167+0000) 2024-03-06T16:16:39.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:39.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:39.661416+0000) 2024-03-06T16:16:40.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:40.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:40.661732+0000) 2024-03-06T16:16:41.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:41.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:41.662006+0000) 2024-03-06T16:16:42.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:42.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:42.662293+0000) 2024-03-06T16:16:43.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:43.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:43.662529+0000) 2024-03-06T16:16:44.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:44.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:44.662790+0000) 2024-03-06T16:16:45.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:45.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:45.663046+0000) 2024-03-06T16:16:46.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:46.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:46.663312+0000) 2024-03-06T16:16:47.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:47.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:47.663564+0000) 2024-03-06T16:16:48.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:48.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:48.663805+0000) 2024-03-06T16:16:49.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:49.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:49.664074+0000) 2024-03-06T16:16:50.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:50.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:50.664346+0000) 2024-03-06T16:16:51.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:51.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:51.664617+0000) 2024-03-06T16:16:52.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:52.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:52.664876+0000) 2024-03-06T16:16:53.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:53.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:53.665145+0000) 2024-03-06T16:16:54.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:54.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:54.665403+0000) 2024-03-06T16:16:55.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:55.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:55.665712+0000) 2024-03-06T16:16:56.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:56.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:56.665972+0000) 2024-03-06T16:16:57.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:57.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:57.666238+0000) 2024-03-06T16:16:58.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:58.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:58.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:58.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:58.666485+0000) 2024-03-06T16:16:59.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:16:59.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:59.666725+0000) 2024-03-06T16:17:00.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:00.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:00.666993+0000) 2024-03-06T16:17:01.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:01.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:01.667240+0000) 2024-03-06T16:17:02.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:02.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:02.667484+0000) 2024-03-06T16:17:03.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:03.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:03.667758+0000) 2024-03-06T16:17:04.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:04.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:04.668011+0000) 2024-03-06T16:17:05.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:05.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:05.668242+0000) 2024-03-06T16:17:06.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:06.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:06.668497+0000) 2024-03-06T16:17:07.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:07.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:07.668807+0000) 2024-03-06T16:17:08.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:08.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:08.669044+0000) 2024-03-06T16:17:09.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:09.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:09.669282+0000) 2024-03-06T16:17:10.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:10.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:10.669541+0000) 2024-03-06T16:17:11.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:11.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:11.669795+0000) 2024-03-06T16:17:12.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:12.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:12.670001+0000) 2024-03-06T16:17:13.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:13.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:13.670238+0000) 2024-03-06T16:17:14.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:14.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:14.670478+0000) 2024-03-06T16:17:15.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:15.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:15.670746+0000) 2024-03-06T16:17:16.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:16.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:16.670999+0000) 2024-03-06T16:17:17.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:17.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:17.671282+0000) 2024-03-06T16:17:18.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:18.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:18.671522+0000) 2024-03-06T16:17:19.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:19.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:19.671768+0000) 2024-03-06T16:17:20.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:20.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:20.672024+0000) 2024-03-06T16:17:21.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:21.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:21.672277+0000) 2024-03-06T16:17:22.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:22.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:22.672542+0000) 2024-03-06T16:17:23.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:23.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:23.672773+0000) 2024-03-06T16:17:24.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:24.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:24.672999+0000) 2024-03-06T16:17:25.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:25.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:25.673231+0000) 2024-03-06T16:17:26.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:26.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:26.673484+0000) 2024-03-06T16:17:27.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:27.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:27.673746+0000) 2024-03-06T16:17:28.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:28.434 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:28.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:28.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:28.673990+0000) 2024-03-06T16:17:29.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:29.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:29.674177+0000) 2024-03-06T16:17:30.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:30.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:30.674419+0000) 2024-03-06T16:17:31.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:31.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:31.674723+0000) 2024-03-06T16:17:32.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:32.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:32.674973+0000) 2024-03-06T16:17:33.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:33.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:33.675194+0000) 2024-03-06T16:17:34.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:34.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:34.675425+0000) 2024-03-06T16:17:35.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:35.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:35.675685+0000) 2024-03-06T16:17:36.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:36.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:36.675938+0000) 2024-03-06T16:17:37.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:37.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:37.676171+0000) 2024-03-06T16:17:38.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:38.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:38.676413+0000) 2024-03-06T16:17:39.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:39.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:39.676691+0000) 2024-03-06T16:17:40.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:40.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:40.676935+0000) 2024-03-06T16:17:41.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:41.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:41.677186+0000) 2024-03-06T16:17:42.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:42.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:42.677448+0000) 2024-03-06T16:17:43.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:43.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:43.677731+0000) 2024-03-06T16:17:44.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:44.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:44.677997+0000) 2024-03-06T16:17:45.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:45.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:45.678249+0000) 2024-03-06T16:17:46.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:46.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:46.678509+0000) 2024-03-06T16:17:47.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:47.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:47.678747+0000) 2024-03-06T16:17:48.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:48.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:48.678991+0000) 2024-03-06T16:17:49.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:49.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:49.679181+0000) 2024-03-06T16:17:50.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:50.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:50.679421+0000) 2024-03-06T16:17:51.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:51.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:51.679681+0000) 2024-03-06T16:17:52.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:52.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:52.679952+0000) 2024-03-06T16:17:53.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:53.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:53.680200+0000) 2024-03-06T16:17:54.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:54.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:54.680453+0000) 2024-03-06T16:17:55.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:55.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:55.680778+0000) 2024-03-06T16:17:56.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:56.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:56.680975+0000) 2024-03-06T16:17:57.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:57.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:57.681221+0000) 2024-03-06T16:17:58.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:58.578 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:58.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:58.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:58.681408+0000) 2024-03-06T16:17:59.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:17:59.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:59.681686+0000) 2024-03-06T16:18:00.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:00.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:00.681929+0000) 2024-03-06T16:18:01.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:01.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:01.682200+0000) 2024-03-06T16:18:02.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:02.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:02.682472+0000) 2024-03-06T16:18:03.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:03.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:03.682751+0000) 2024-03-06T16:18:04.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:04.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:04.683005+0000) 2024-03-06T16:18:05.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:05.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:05.683204+0000) 2024-03-06T16:18:06.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:06.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:06.683469+0000) 2024-03-06T16:18:07.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:07.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:07.683744+0000) 2024-03-06T16:18:08.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:08.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:08.684006+0000) 2024-03-06T16:18:09.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:09.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:09.684253+0000) 2024-03-06T16:18:10.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:10.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:10.684501+0000) 2024-03-06T16:18:11.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:11.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:11.684742+0000) 2024-03-06T16:18:12.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:12.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:12.685020+0000) 2024-03-06T16:18:13.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:13.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:13.685259+0000) 2024-03-06T16:18:14.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:14.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:14.685512+0000) 2024-03-06T16:18:15.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:15.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:15.685747+0000) 2024-03-06T16:18:16.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:16.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:16.685966+0000) 2024-03-06T16:18:17.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:17.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:17.686225+0000) 2024-03-06T16:18:18.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:18.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:18.686470+0000) 2024-03-06T16:18:19.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:19.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:19.686747+0000) 2024-03-06T16:18:20.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:20.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:20.686981+0000) 2024-03-06T16:18:21.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:21.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:21.687235+0000) 2024-03-06T16:18:22.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:22.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:22.687483+0000) 2024-03-06T16:18:23.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:23.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:23.687750+0000) 2024-03-06T16:18:24.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:24.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:24.688005+0000) 2024-03-06T16:18:25.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:25.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:25.688258+0000) 2024-03-06T16:18:26.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:26.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:26.688498+0000) 2024-03-06T16:18:27.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:27.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:27.688752+0000) 2024-03-06T16:18:28.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:28.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:28.688990+0000) 2024-03-06T16:18:28.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:28.721 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:29.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:29.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:29.689244+0000) 2024-03-06T16:18:30.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:30.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:30.689482+0000) 2024-03-06T16:18:31.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:31.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:31.689730+0000) 2024-03-06T16:18:32.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:32.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:32.689925+0000) 2024-03-06T16:18:33.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:33.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:33.690177+0000) 2024-03-06T16:18:34.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:34.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:34.690414+0000) 2024-03-06T16:18:35.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:35.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:35.690708+0000) 2024-03-06T16:18:36.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:36.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:36.690968+0000) 2024-03-06T16:18:37.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:37.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:37.691205+0000) 2024-03-06T16:18:38.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:38.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:38.691456+0000) 2024-03-06T16:18:39.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:39.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:39.691729+0000) 2024-03-06T16:18:40.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:40.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:40.691983+0000) 2024-03-06T16:18:41.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:41.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:41.692232+0000) 2024-03-06T16:18:42.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:42.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:42.692499+0000) 2024-03-06T16:18:43.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:43.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:43.692802+0000) 2024-03-06T16:18:44.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:44.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:44.693056+0000) 2024-03-06T16:18:45.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:45.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:45.693278+0000) 2024-03-06T16:18:46.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:46.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:46.693517+0000) 2024-03-06T16:18:47.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:47.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:47.693771+0000) 2024-03-06T16:18:48.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:48.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:48.694021+0000) 2024-03-06T16:18:49.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:49.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:49.694308+0000) 2024-03-06T16:18:50.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:50.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:50.694564+0000) 2024-03-06T16:18:51.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:51.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:51.694797+0000) 2024-03-06T16:18:52.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:52.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:52.695043+0000) 2024-03-06T16:18:53.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:53.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:53.695300+0000) 2024-03-06T16:18:54.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:54.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:54.695547+0000) 2024-03-06T16:18:55.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:55.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:55.695803+0000) 2024-03-06T16:18:56.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:56.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:56.696067+0000) 2024-03-06T16:18:57.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:57.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:57.696333+0000) 2024-03-06T16:18:58.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:58.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:58.696593+0000) 2024-03-06T16:18:58.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:58.861 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:59.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:18:59.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:59.696853+0000) 2024-03-06T16:19:00.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:00.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:00.697106+0000) 2024-03-06T16:19:01.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:01.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:01.697356+0000) 2024-03-06T16:19:02.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:02.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:02.697664+0000) 2024-03-06T16:19:03.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:03.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:03.697910+0000) 2024-03-06T16:19:04.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:04.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:04.698171+0000) 2024-03-06T16:19:05.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:05.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:05.698414+0000) 2024-03-06T16:19:06.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:06.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:06.698683+0000) 2024-03-06T16:19:07.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:07.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:07.698936+0000) 2024-03-06T16:19:08.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:08.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:08.699176+0000) 2024-03-06T16:19:09.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:09.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:09.699408+0000) 2024-03-06T16:19:10.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:10.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:10.699701+0000) 2024-03-06T16:19:11.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:11.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:11.699940+0000) 2024-03-06T16:19:12.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:12.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:12.700211+0000) 2024-03-06T16:19:13.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:13.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:13.700447+0000) 2024-03-06T16:19:14.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:14.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:14.700754+0000) 2024-03-06T16:19:15.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:15.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:15.701027+0000) 2024-03-06T16:19:16.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:16.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:16.701290+0000) 2024-03-06T16:19:17.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:17.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:17.701561+0000) 2024-03-06T16:19:18.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:18.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:18.701826+0000) 2024-03-06T16:19:19.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:19.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:19.702075+0000) 2024-03-06T16:19:20.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:20.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:20.702279+0000) 2024-03-06T16:19:21.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:21.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:21.702541+0000) 2024-03-06T16:19:22.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:22.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:22.702791+0000) 2024-03-06T16:19:23.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:23.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:23.703037+0000) 2024-03-06T16:19:24.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:24.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:24.703300+0000) 2024-03-06T16:19:25.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:25.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:25.703551+0000) 2024-03-06T16:19:26.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:26.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:26.703807+0000) 2024-03-06T16:19:27.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:27.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:27.703985+0000) 2024-03-06T16:19:28.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:28.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:28.704246+0000) 2024-03-06T16:19:29.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:29.016 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:29.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:29.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:29.704479+0000) 2024-03-06T16:19:30.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:30.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:30.704746+0000) 2024-03-06T16:19:31.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:31.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:31.704999+0000) 2024-03-06T16:19:32.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:32.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:32.705267+0000) 2024-03-06T16:19:33.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:33.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:33.705517+0000) 2024-03-06T16:19:34.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:34.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:34.705776+0000) 2024-03-06T16:19:35.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:35.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:35.706012+0000) 2024-03-06T16:19:36.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:36.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:36.706306+0000) 2024-03-06T16:19:37.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:37.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:37.706551+0000) 2024-03-06T16:19:38.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:38.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:38.706798+0000) 2024-03-06T16:19:39.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:39.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:39.706978+0000) 2024-03-06T16:19:40.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:40.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:40.707234+0000) 2024-03-06T16:19:41.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:41.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:41.707418+0000) 2024-03-06T16:19:42.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:42.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:42.707718+0000) 2024-03-06T16:19:43.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:43.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:43.707955+0000) 2024-03-06T16:19:44.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:44.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:44.708197+0000) 2024-03-06T16:19:45.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:45.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:45.708446+0000) 2024-03-06T16:19:46.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:46.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:46.708742+0000) 2024-03-06T16:19:47.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:47.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:47.709003+0000) 2024-03-06T16:19:48.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:48.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:48.709267+0000) 2024-03-06T16:19:49.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:49.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:49.709525+0000) 2024-03-06T16:19:50.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:50.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:50.709782+0000) 2024-03-06T16:19:51.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:51.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:51.710008+0000) 2024-03-06T16:19:52.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:52.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:52.710270+0000) 2024-03-06T16:19:53.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:53.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:53.710527+0000) 2024-03-06T16:19:54.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:54.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:54.710769+0000) 2024-03-06T16:19:55.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:55.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:55.710957+0000) 2024-03-06T16:19:56.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:56.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:56.711211+0000) 2024-03-06T16:19:57.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:57.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:57.711464+0000) 2024-03-06T16:19:58.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:58.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:58.711754+0000) 2024-03-06T16:19:59.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:59.164 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:59.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:19:59.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:59.711996+0000) 2024-03-06T16:20:00.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:00.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:00.712180+0000) 2024-03-06T16:20:01.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:01.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:01.712420+0000) 2024-03-06T16:20:02.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:02.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:02.712734+0000) 2024-03-06T16:20:03.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:03.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:03.713000+0000) 2024-03-06T16:20:04.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:04.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:04.713254+0000) 2024-03-06T16:20:05.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:05.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:05.713482+0000) 2024-03-06T16:20:06.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:06.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:06.713743+0000) 2024-03-06T16:20:07.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:07.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:07.713998+0000) 2024-03-06T16:20:08.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:08.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:08.714251+0000) 2024-03-06T16:20:09.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:09.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:09.714493+0000) 2024-03-06T16:20:10.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:10.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:10.714750+0000) 2024-03-06T16:20:11.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:11.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:11.714984+0000) 2024-03-06T16:20:12.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:12.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:12.715191+0000) 2024-03-06T16:20:13.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:13.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:13.715434+0000) 2024-03-06T16:20:14.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:14.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:14.715734+0000) 2024-03-06T16:20:15.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:15.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:15.715924+0000) 2024-03-06T16:20:16.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:16.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:16.716181+0000) 2024-03-06T16:20:17.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:17.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:17.716440+0000) 2024-03-06T16:20:18.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:18.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:18.716722+0000) 2024-03-06T16:20:19.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:19.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:19.716988+0000) 2024-03-06T16:20:20.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:20.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:20.717238+0000) 2024-03-06T16:20:21.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:21.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:21.717499+0000) 2024-03-06T16:20:22.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:22.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:22.717773+0000) 2024-03-06T16:20:23.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:23.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:23.718040+0000) 2024-03-06T16:20:24.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:24.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:24.718359+0000) 2024-03-06T16:20:25.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:25.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:25.718621+0000) 2024-03-06T16:20:26.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:26.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:26.718920+0000) 2024-03-06T16:20:27.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:27.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:27.719177+0000) 2024-03-06T16:20:28.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:28.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:28.719378+0000) 2024-03-06T16:20:29.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:29.322 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:29.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:29.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:29.719672+0000) 2024-03-06T16:20:30.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:30.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:30.719913+0000) 2024-03-06T16:20:31.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:31.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:31.720146+0000) 2024-03-06T16:20:32.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:32.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:32.720412+0000) 2024-03-06T16:20:33.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:33.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:33.720688+0000) 2024-03-06T16:20:34.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:34.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:34.720924+0000) 2024-03-06T16:20:35.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:35.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:35.721159+0000) 2024-03-06T16:20:36.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:36.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:36.721419+0000) 2024-03-06T16:20:37.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:37.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:37.721599+0000) 2024-03-06T16:20:38.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:38.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:38.721830+0000) 2024-03-06T16:20:39.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:39.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:39.722077+0000) 2024-03-06T16:20:40.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:40.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:40.722343+0000) 2024-03-06T16:20:41.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:41.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:41.722594+0000) 2024-03-06T16:20:42.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:42.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:42.722883+0000) 2024-03-06T16:20:43.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:43.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:43.723144+0000) 2024-03-06T16:20:44.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:44.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:44.723397+0000) 2024-03-06T16:20:45.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:45.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:45.723688+0000) 2024-03-06T16:20:46.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:46.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:46.723919+0000) 2024-03-06T16:20:47.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:47.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:47.724166+0000) 2024-03-06T16:20:48.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:48.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:48.724414+0000) 2024-03-06T16:20:49.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:49.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:49.724689+0000) 2024-03-06T16:20:50.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:50.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:50.724949+0000) 2024-03-06T16:20:51.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:51.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:51.725188+0000) 2024-03-06T16:20:52.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:52.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:52.725449+0000) 2024-03-06T16:20:53.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:53.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:53.725749+0000) 2024-03-06T16:20:54.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:54.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:54.725973+0000) 2024-03-06T16:20:55.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:55.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:55.726206+0000) 2024-03-06T16:20:56.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:56.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:56.726452+0000) 2024-03-06T16:20:57.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:57.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:57.726735+0000) 2024-03-06T16:20:58.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:58.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:58.726994+0000) 2024-03-06T16:20:59.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:59.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:59.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:20:59.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:59.727239+0000) 2024-03-06T16:21:00.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:00.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:00.727497+0000) 2024-03-06T16:21:01.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:01.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:01.727752+0000) 2024-03-06T16:21:02.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:02.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:02.727999+0000) 2024-03-06T16:21:03.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:03.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:03.728236+0000) 2024-03-06T16:21:04.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:04.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:04.728475+0000) 2024-03-06T16:21:05.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:05.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:05.728752+0000) 2024-03-06T16:21:06.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:06.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:06.729003+0000) 2024-03-06T16:21:07.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:07.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:07.729236+0000) 2024-03-06T16:21:08.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:08.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:08.729474+0000) 2024-03-06T16:21:09.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:09.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:09.729725+0000) 2024-03-06T16:21:10.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:10.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:10.729984+0000) 2024-03-06T16:21:11.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:11.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:11.730230+0000) 2024-03-06T16:21:12.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:12.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:12.730482+0000) 2024-03-06T16:21:13.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:13.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:13.730738+0000) 2024-03-06T16:21:14.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:14.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:14.730940+0000) 2024-03-06T16:21:15.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:15.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:15.731189+0000) 2024-03-06T16:21:16.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:16.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:16.731428+0000) 2024-03-06T16:21:17.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:17.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:17.731719+0000) 2024-03-06T16:21:18.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:18.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:18.731974+0000) 2024-03-06T16:21:19.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:19.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:19.732217+0000) 2024-03-06T16:21:20.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:20.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:20.732480+0000) 2024-03-06T16:21:21.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:21.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:21.732713+0000) 2024-03-06T16:21:22.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:22.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:22.732965+0000) 2024-03-06T16:21:23.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:23.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:23.733220+0000) 2024-03-06T16:21:24.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:24.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:24.733476+0000) 2024-03-06T16:21:25.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:25.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:25.733739+0000) 2024-03-06T16:21:26.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:26.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:26.733995+0000) 2024-03-06T16:21:27.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:27.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:27.734231+0000) 2024-03-06T16:21:28.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:28.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:28.734490+0000) 2024-03-06T16:21:29.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:29.623 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:29.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:29.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:29.734778+0000) 2024-03-06T16:21:30.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:30.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:30.735018+0000) 2024-03-06T16:21:31.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:31.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:31.735248+0000) 2024-03-06T16:21:32.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:32.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:32.735514+0000) 2024-03-06T16:21:33.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:33.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:33.735789+0000) 2024-03-06T16:21:34.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:34.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:34.735968+0000) 2024-03-06T16:21:35.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:35.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:35.736208+0000) 2024-03-06T16:21:36.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:36.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:36.736393+0000) 2024-03-06T16:21:37.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:37.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:37.736672+0000) 2024-03-06T16:21:38.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:38.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:38.736936+0000) 2024-03-06T16:21:39.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:39.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:39.737185+0000) 2024-03-06T16:21:40.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:40.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:40.737442+0000) 2024-03-06T16:21:41.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:41.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:41.737733+0000) 2024-03-06T16:21:42.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:42.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:42.737998+0000) 2024-03-06T16:21:43.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:43.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:43.738251+0000) 2024-03-06T16:21:44.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:44.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:44.738437+0000) 2024-03-06T16:21:45.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:45.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:45.738741+0000) 2024-03-06T16:21:46.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:46.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:46.738972+0000) 2024-03-06T16:21:47.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:47.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:47.739202+0000) 2024-03-06T16:21:48.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:48.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:48.739459+0000) 2024-03-06T16:21:49.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:49.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:49.739754+0000) 2024-03-06T16:21:50.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:50.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:50.740005+0000) 2024-03-06T16:21:51.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:51.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:51.740253+0000) 2024-03-06T16:21:52.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:52.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:52.740498+0000) 2024-03-06T16:21:53.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:53.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:53.740750+0000) 2024-03-06T16:21:54.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:54.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:54.741006+0000) 2024-03-06T16:21:55.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:55.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:55.741244+0000) 2024-03-06T16:21:56.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:56.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:56.741429+0000) 2024-03-06T16:21:57.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:57.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:57.741698+0000) 2024-03-06T16:21:58.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:58.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:58.741953+0000) 2024-03-06T16:21:59.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:21:59.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:59.742202+0000) 2024-03-06T16:21:59.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:59.768 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:00.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:00.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:00.742440+0000) 2024-03-06T16:22:01.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:01.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:01.742720+0000) 2024-03-06T16:22:02.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:02.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:02.742974+0000) 2024-03-06T16:22:03.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:03.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:03.743208+0000) 2024-03-06T16:22:04.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:04.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:04.743390+0000) 2024-03-06T16:22:05.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:05.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:05.743662+0000) 2024-03-06T16:22:06.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:06.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:06.743915+0000) 2024-03-06T16:22:07.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:07.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:07.744167+0000) 2024-03-06T16:22:08.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:08.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:08.744411+0000) 2024-03-06T16:22:09.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:09.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:09.744655+0000) 2024-03-06T16:22:10.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:10.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:10.744885+0000) 2024-03-06T16:22:11.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:11.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:11.745119+0000) 2024-03-06T16:22:12.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:12.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:12.745367+0000) 2024-03-06T16:22:13.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:13.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:13.745606+0000) 2024-03-06T16:22:14.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:14.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:14.745899+0000) 2024-03-06T16:22:15.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:15.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:15.746123+0000) 2024-03-06T16:22:16.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:16.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:16.746323+0000) 2024-03-06T16:22:17.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:17.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:17.746509+0000) 2024-03-06T16:22:18.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:18.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:18.746741+0000) 2024-03-06T16:22:19.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:19.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:19.746965+0000) 2024-03-06T16:22:20.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:20.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:20.747222+0000) 2024-03-06T16:22:21.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:21.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:21.747442+0000) 2024-03-06T16:22:22.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:22.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:22.747717+0000) 2024-03-06T16:22:23.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:23.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:23.747957+0000) 2024-03-06T16:22:24.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:24.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:24.748210+0000) 2024-03-06T16:22:25.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:25.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:25.748444+0000) 2024-03-06T16:22:26.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:26.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:26.748733+0000) 2024-03-06T16:22:27.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:27.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:27.748982+0000) 2024-03-06T16:22:28.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:28.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:28.749169+0000) 2024-03-06T16:22:29.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:29.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:29.749418+0000) 2024-03-06T16:22:29.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:29.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:30.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:30.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:30.749685+0000) 2024-03-06T16:22:31.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:31.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:31.749914+0000) 2024-03-06T16:22:32.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:32.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:32.750188+0000) 2024-03-06T16:22:33.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:33.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:33.750380+0000) 2024-03-06T16:22:34.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:34.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:34.750611+0000) 2024-03-06T16:22:35.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:35.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:35.750893+0000) 2024-03-06T16:22:36.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:36.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:36.751141+0000) 2024-03-06T16:22:37.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:37.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:37.751384+0000) 2024-03-06T16:22:38.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:38.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:38.751669+0000) 2024-03-06T16:22:39.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:39.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:39.751902+0000) 2024-03-06T16:22:40.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:40.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:40.752153+0000) 2024-03-06T16:22:41.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:41.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:41.752384+0000) 2024-03-06T16:22:42.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:42.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:42.752669+0000) 2024-03-06T16:22:43.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:43.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:43.752945+0000) 2024-03-06T16:22:44.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:44.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:44.753195+0000) 2024-03-06T16:22:45.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:45.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:45.753434+0000) 2024-03-06T16:22:46.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:46.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:46.753724+0000) 2024-03-06T16:22:47.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:47.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:47.753981+0000) 2024-03-06T16:22:48.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:48.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:48.754249+0000) 2024-03-06T16:22:49.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:49.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:49.754498+0000) 2024-03-06T16:22:50.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:50.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:50.754752+0000) 2024-03-06T16:22:51.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:51.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:51.755006+0000) 2024-03-06T16:22:52.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:52.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:52.755267+0000) 2024-03-06T16:22:53.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:53.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:53.755569+0000) 2024-03-06T16:22:54.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:54.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:54.755822+0000) 2024-03-06T16:22:55.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:55.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:55.756074+0000) 2024-03-06T16:22:56.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:56.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:56.756327+0000) 2024-03-06T16:22:57.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:57.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:57.756673+0000) 2024-03-06T16:22:58.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:58.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:58.756942+0000) 2024-03-06T16:22:59.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:22:59.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:59.757127+0000) 2024-03-06T16:23:00.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:00.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:00.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:00.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:00.757389+0000) 2024-03-06T16:23:01.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:01.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:01.757618+0000) 2024-03-06T16:23:02.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:02.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:02.757914+0000) 2024-03-06T16:23:03.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:03.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:03.758142+0000) 2024-03-06T16:23:04.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:04.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:04.758386+0000) 2024-03-06T16:23:05.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:05.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:05.758658+0000) 2024-03-06T16:23:06.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:06.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:06.758920+0000) 2024-03-06T16:23:07.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:07.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:07.759169+0000) 2024-03-06T16:23:08.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:08.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:08.759400+0000) 2024-03-06T16:23:09.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:09.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:09.759709+0000) 2024-03-06T16:23:10.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:10.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:10.759981+0000) 2024-03-06T16:23:11.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:11.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:11.760232+0000) 2024-03-06T16:23:12.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:12.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:12.760483+0000) 2024-03-06T16:23:13.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:13.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:13.760745+0000) 2024-03-06T16:23:14.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:14.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:14.760999+0000) 2024-03-06T16:23:15.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:15.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:15.761235+0000) 2024-03-06T16:23:16.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:16.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:16.761488+0000) 2024-03-06T16:23:17.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:17.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:17.761734+0000) 2024-03-06T16:23:18.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:18.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:18.761917+0000) 2024-03-06T16:23:19.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:19.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:19.762172+0000) 2024-03-06T16:23:20.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:20.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:20.762417+0000) 2024-03-06T16:23:21.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:21.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:21.762678+0000) 2024-03-06T16:23:22.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:22.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:22.762945+0000) 2024-03-06T16:23:23.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:23.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:23.763128+0000) 2024-03-06T16:23:24.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:24.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:24.763362+0000) 2024-03-06T16:23:25.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:25.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:25.763614+0000) 2024-03-06T16:23:26.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:26.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:26.763907+0000) 2024-03-06T16:23:27.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:27.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:27.764155+0000) 2024-03-06T16:23:28.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:28.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:28.764413+0000) 2024-03-06T16:23:29.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:29.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:29.764716+0000) 2024-03-06T16:23:30.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:30.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:30.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:30.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:30.764974+0000) 2024-03-06T16:23:31.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:31.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:31.765232+0000) 2024-03-06T16:23:32.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:32.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:32.765476+0000) 2024-03-06T16:23:33.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:33.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:33.765745+0000) 2024-03-06T16:23:34.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:34.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:34.765996+0000) 2024-03-06T16:23:35.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:35.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:35.766234+0000) 2024-03-06T16:23:36.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:36.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:36.766497+0000) 2024-03-06T16:23:37.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:37.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:37.766748+0000) 2024-03-06T16:23:38.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:38.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:38.767004+0000) 2024-03-06T16:23:39.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:39.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:39.767257+0000) 2024-03-06T16:23:40.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:40.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:40.767503+0000) 2024-03-06T16:23:41.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:41.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:41.767791+0000) 2024-03-06T16:23:42.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:42.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:42.768043+0000) 2024-03-06T16:23:43.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:43.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:43.768294+0000) 2024-03-06T16:23:44.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:44.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:44.768535+0000) 2024-03-06T16:23:45.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:45.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:45.768782+0000) 2024-03-06T16:23:46.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:46.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:46.769043+0000) 2024-03-06T16:23:47.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:47.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:47.769290+0000) 2024-03-06T16:23:48.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:48.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:48.769555+0000) 2024-03-06T16:23:49.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:49.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:49.769805+0000) 2024-03-06T16:23:50.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:50.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:50.769991+0000) 2024-03-06T16:23:51.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:51.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:51.770238+0000) 2024-03-06T16:23:52.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:52.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:52.770511+0000) 2024-03-06T16:23:53.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:53.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:53.770761+0000) 2024-03-06T16:23:54.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:54.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:54.771066+0000) 2024-03-06T16:23:55.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:55.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:55.771358+0000) 2024-03-06T16:23:56.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:56.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:56.771722+0000) 2024-03-06T16:23:57.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:57.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:57.772021+0000) 2024-03-06T16:23:58.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:58.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:58.772322+0000) 2024-03-06T16:23:59.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:23:59.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:59.772622+0000) 2024-03-06T16:24:00.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:00.378 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:00.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:00.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:00.772950+0000) 2024-03-06T16:24:01.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:01.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:01.773259+0000) 2024-03-06T16:24:02.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:02.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:02.773573+0000) 2024-03-06T16:24:03.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:03.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:03.773862+0000) 2024-03-06T16:24:04.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:04.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:04.774167+0000) 2024-03-06T16:24:05.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:05.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:05.774466+0000) 2024-03-06T16:24:06.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:06.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:06.774775+0000) 2024-03-06T16:24:07.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:07.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:07.775074+0000) 2024-03-06T16:24:08.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:08.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:08.775366+0000) 2024-03-06T16:24:09.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:09.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:09.775710+0000) 2024-03-06T16:24:10.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:10.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:10.776032+0000) 2024-03-06T16:24:11.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:11.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:11.776333+0000) 2024-03-06T16:24:12.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:12.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:12.776698+0000) 2024-03-06T16:24:13.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:13.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:13.777005+0000) 2024-03-06T16:24:14.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:14.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:14.777283+0000) 2024-03-06T16:24:15.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:15.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:15.777577+0000) 2024-03-06T16:24:16.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:16.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:16.777898+0000) 2024-03-06T16:24:17.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:17.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:17.778196+0000) 2024-03-06T16:24:18.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:18.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:18.778438+0000) 2024-03-06T16:24:19.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:19.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:19.778780+0000) 2024-03-06T16:24:20.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:20.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:20.779091+0000) 2024-03-06T16:24:21.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:21.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:21.779404+0000) 2024-03-06T16:24:22.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:22.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:22.779770+0000) 2024-03-06T16:24:23.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:23.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:23.780053+0000) 2024-03-06T16:24:24.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:24.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:24.780342+0000) 2024-03-06T16:24:25.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:25.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:25.780625+0000) 2024-03-06T16:24:26.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:26.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:26.780994+0000) 2024-03-06T16:24:27.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:27.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:27.781250+0000) 2024-03-06T16:24:28.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:28.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:28.781509+0000) 2024-03-06T16:24:29.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:29.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:29.781788+0000) 2024-03-06T16:24:30.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:30.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:30.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:30.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:30.782094+0000) 2024-03-06T16:24:31.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:31.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:31.782383+0000) 2024-03-06T16:24:32.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:32.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:32.782745+0000) 2024-03-06T16:24:33.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:33.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:33.783040+0000) 2024-03-06T16:24:34.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:34.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:34.783341+0000) 2024-03-06T16:24:35.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:35.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:35.783719+0000) 2024-03-06T16:24:36.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:36.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:36.784023+0000) 2024-03-06T16:24:37.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:37.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:37.784330+0000) 2024-03-06T16:24:38.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:38.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:38.784574+0000) 2024-03-06T16:24:39.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:39.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:39.784855+0000) 2024-03-06T16:24:40.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:40.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:40.785138+0000) 2024-03-06T16:24:41.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:41.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:41.785429+0000) 2024-03-06T16:24:42.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:42.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:42.785752+0000) 2024-03-06T16:24:43.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:43.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:43.785982+0000) 2024-03-06T16:24:44.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:44.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:44.786243+0000) 2024-03-06T16:24:45.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:45.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:45.786494+0000) 2024-03-06T16:24:46.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:46.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:46.786756+0000) 2024-03-06T16:24:47.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:47.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:47.786988+0000) 2024-03-06T16:24:48.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:48.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:48.787260+0000) 2024-03-06T16:24:49.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:49.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:49.787519+0000) 2024-03-06T16:24:50.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:50.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:50.787775+0000) 2024-03-06T16:24:51.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:51.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:51.788003+0000) 2024-03-06T16:24:52.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:52.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:52.788266+0000) 2024-03-06T16:24:53.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:53.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:53.788474+0000) 2024-03-06T16:24:54.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:54.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:54.788767+0000) 2024-03-06T16:24:55.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:55.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:55.789018+0000) 2024-03-06T16:24:56.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:56.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:56.789309+0000) 2024-03-06T16:24:57.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:57.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:57.789578+0000) 2024-03-06T16:24:58.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:58.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:58.789839+0000) 2024-03-06T16:24:59.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:24:59.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:59.790111+0000) 2024-03-06T16:25:00.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:00.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:00.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:00.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:00.790323+0000) 2024-03-06T16:25:01.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:01.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:01.790564+0000) 2024-03-06T16:25:02.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:02.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:02.790847+0000) 2024-03-06T16:25:03.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:03.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:03.791102+0000) 2024-03-06T16:25:04.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:04.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:04.791362+0000) 2024-03-06T16:25:05.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:05.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:05.791593+0000) 2024-03-06T16:25:06.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:06.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:06.791842+0000) 2024-03-06T16:25:07.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:07.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:07.792095+0000) 2024-03-06T16:25:08.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:08.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:08.792361+0000) 2024-03-06T16:25:09.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:09.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:09.792612+0000) 2024-03-06T16:25:10.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:10.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:10.792862+0000) 2024-03-06T16:25:11.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:11.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:11.793053+0000) 2024-03-06T16:25:12.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:12.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:12.793309+0000) 2024-03-06T16:25:13.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:13.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:13.793540+0000) 2024-03-06T16:25:14.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:14.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:14.793778+0000) 2024-03-06T16:25:15.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:15.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:15.794010+0000) 2024-03-06T16:25:16.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:16.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:16.794291+0000) 2024-03-06T16:25:17.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:17.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:17.794560+0000) 2024-03-06T16:25:18.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:18.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:18.794832+0000) 2024-03-06T16:25:19.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:19.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:19.795093+0000) 2024-03-06T16:25:20.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:20.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:20.795317+0000) 2024-03-06T16:25:21.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:21.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:21.795563+0000) 2024-03-06T16:25:22.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:22.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:22.795840+0000) 2024-03-06T16:25:23.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:23.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:23.796107+0000) 2024-03-06T16:25:24.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:24.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:24.796342+0000) 2024-03-06T16:25:25.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:25.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:25.796574+0000) 2024-03-06T16:25:26.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:26.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:26.796826+0000) 2024-03-06T16:25:27.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:27.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:27.797096+0000) 2024-03-06T16:25:28.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:28.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:28.797406+0000) 2024-03-06T16:25:29.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:29.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:29.797600+0000) 2024-03-06T16:25:30.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:30.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:30.797893+0000) 2024-03-06T16:25:30.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:30.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:31.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:31.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:31.798084+0000) 2024-03-06T16:25:32.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:32.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:32.798356+0000) 2024-03-06T16:25:33.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:33.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:33.798604+0000) 2024-03-06T16:25:34.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:34.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:34.798932+0000) 2024-03-06T16:25:35.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:35.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:35.799172+0000) 2024-03-06T16:25:36.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:36.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:36.799410+0000) 2024-03-06T16:25:37.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:37.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:37.799595+0000) 2024-03-06T16:25:38.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:38.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:38.799859+0000) 2024-03-06T16:25:39.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:39.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:39.800114+0000) 2024-03-06T16:25:40.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:40.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:40.800375+0000) 2024-03-06T16:25:41.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:41.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:41.800623+0000) 2024-03-06T16:25:42.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:42.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:42.800928+0000) 2024-03-06T16:25:43.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:43.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:43.801178+0000) 2024-03-06T16:25:44.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:44.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:44.801375+0000) 2024-03-06T16:25:45.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:45.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:45.801681+0000) 2024-03-06T16:25:46.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:46.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:46.801940+0000) 2024-03-06T16:25:47.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:47.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:47.802172+0000) 2024-03-06T16:25:48.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:48.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:48.802427+0000) 2024-03-06T16:25:49.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:49.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:49.802714+0000) 2024-03-06T16:25:50.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:50.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:50.802908+0000) 2024-03-06T16:25:51.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:51.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:51.803095+0000) 2024-03-06T16:25:52.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:52.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:52.803350+0000) 2024-03-06T16:25:53.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:53.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:53.803596+0000) 2024-03-06T16:25:54.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:54.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:54.803846+0000) 2024-03-06T16:25:55.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:55.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:55.804069+0000) 2024-03-06T16:25:56.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:56.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:56.804334+0000) 2024-03-06T16:25:57.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:57.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:57.804568+0000) 2024-03-06T16:25:58.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:58.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:58.804823+0000) 2024-03-06T16:25:59.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:25:59.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:59.805073+0000) 2024-03-06T16:26:00.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:00.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:00.805319+0000) 2024-03-06T16:26:00.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:00.969 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:01.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:01.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:01.805567+0000) 2024-03-06T16:26:02.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:02.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:02.805832+0000) 2024-03-06T16:26:03.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:03.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:03.806082+0000) 2024-03-06T16:26:04.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:04.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:04.806324+0000) 2024-03-06T16:26:05.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:05.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:05.806562+0000) 2024-03-06T16:26:06.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:06.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:06.806816+0000) 2024-03-06T16:26:07.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:07.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:07.807067+0000) 2024-03-06T16:26:08.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:08.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:08.807311+0000) 2024-03-06T16:26:09.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:09.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:09.807559+0000) 2024-03-06T16:26:10.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:10.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:10.807824+0000) 2024-03-06T16:26:11.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:11.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:11.808018+0000) 2024-03-06T16:26:12.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:12.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:12.808284+0000) 2024-03-06T16:26:13.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:13.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:13.808515+0000) 2024-03-06T16:26:14.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:14.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:14.808769+0000) 2024-03-06T16:26:15.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:15.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:15.809023+0000) 2024-03-06T16:26:16.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:16.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:16.809263+0000) 2024-03-06T16:26:17.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:17.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:17.809511+0000) 2024-03-06T16:26:18.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:18.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:18.809749+0000) 2024-03-06T16:26:19.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:19.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:19.810002+0000) 2024-03-06T16:26:20.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:20.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:20.810186+0000) 2024-03-06T16:26:21.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:21.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:21.810470+0000) 2024-03-06T16:26:22.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:22.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:22.810763+0000) 2024-03-06T16:26:23.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:23.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:23.810987+0000) 2024-03-06T16:26:24.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:24.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:24.811237+0000) 2024-03-06T16:26:25.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:25.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:25.811471+0000) 2024-03-06T16:26:26.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:26.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:26.811737+0000) 2024-03-06T16:26:27.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:27.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:27.811986+0000) 2024-03-06T16:26:28.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:28.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:28.812245+0000) 2024-03-06T16:26:29.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:29.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:29.812497+0000) 2024-03-06T16:26:30.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:30.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:30.812753+0000) 2024-03-06T16:26:31.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:31.121 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:31.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:31.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:31.812942+0000) 2024-03-06T16:26:32.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:32.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:32.813214+0000) 2024-03-06T16:26:33.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:33.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:33.813466+0000) 2024-03-06T16:26:34.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:34.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:34.813762+0000) 2024-03-06T16:26:35.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:35.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:35.814014+0000) 2024-03-06T16:26:36.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:36.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:36.814270+0000) 2024-03-06T16:26:37.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:37.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:37.814525+0000) 2024-03-06T16:26:38.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:38.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:38.814790+0000) 2024-03-06T16:26:39.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:39.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:39.815023+0000) 2024-03-06T16:26:40.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:40.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:40.815289+0000) 2024-03-06T16:26:41.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:41.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:41.815476+0000) 2024-03-06T16:26:42.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:42.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:42.815732+0000) 2024-03-06T16:26:43.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:43.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:43.815967+0000) 2024-03-06T16:26:44.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:44.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:44.816207+0000) 2024-03-06T16:26:45.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:45.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:45.816461+0000) 2024-03-06T16:26:46.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:46.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:46.816776+0000) 2024-03-06T16:26:47.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:47.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:47.817028+0000) 2024-03-06T16:26:48.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:48.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:48.817297+0000) 2024-03-06T16:26:49.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:49.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:49.817538+0000) 2024-03-06T16:26:50.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:50.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:50.817762+0000) 2024-03-06T16:26:51.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:51.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:51.818014+0000) 2024-03-06T16:26:52.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:52.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:52.818302+0000) 2024-03-06T16:26:53.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:53.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:53.818559+0000) 2024-03-06T16:26:54.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:54.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:54.818840+0000) 2024-03-06T16:26:55.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:55.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:55.819098+0000) 2024-03-06T16:26:56.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:56.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:56.819361+0000) 2024-03-06T16:26:57.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:57.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:57.819610+0000) 2024-03-06T16:26:58.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:58.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:58.819907+0000) 2024-03-06T16:26:59.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:26:59.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:59.820176+0000) 2024-03-06T16:27:00.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:00.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:00.820426+0000) 2024-03-06T16:27:01.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:01.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:01.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:01.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:01.820731+0000) 2024-03-06T16:27:02.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:02.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:02.821000+0000) 2024-03-06T16:27:03.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:03.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:03.821191+0000) 2024-03-06T16:27:04.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:04.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:04.821501+0000) 2024-03-06T16:27:05.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:05.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:05.821782+0000) 2024-03-06T16:27:06.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:06.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:06.821997+0000) 2024-03-06T16:27:07.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:07.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:07.822277+0000) 2024-03-06T16:27:08.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:08.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:08.822552+0000) 2024-03-06T16:27:09.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:09.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:09.822806+0000) 2024-03-06T16:27:10.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:10.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:10.823088+0000) 2024-03-06T16:27:11.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:11.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:11.823358+0000) 2024-03-06T16:27:12.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:12.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:12.823669+0000) 2024-03-06T16:27:13.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:13.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:13.823946+0000) 2024-03-06T16:27:14.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:14.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:14.824213+0000) 2024-03-06T16:27:15.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:15.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:15.824490+0000) 2024-03-06T16:27:16.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:16.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:16.824751+0000) 2024-03-06T16:27:17.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:17.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:17.825021+0000) 2024-03-06T16:27:18.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:18.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:18.825280+0000) 2024-03-06T16:27:19.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:19.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:19.825534+0000) 2024-03-06T16:27:20.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:20.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:20.825810+0000) 2024-03-06T16:27:21.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:21.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:21.826071+0000) 2024-03-06T16:27:22.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:22.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:22.826352+0000) 2024-03-06T16:27:23.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:23.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:23.826589+0000) 2024-03-06T16:27:24.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:24.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:24.826850+0000) 2024-03-06T16:27:25.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:25.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:25.827114+0000) 2024-03-06T16:27:26.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:26.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:26.827373+0000) 2024-03-06T16:27:27.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:27.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:27.827625+0000) 2024-03-06T16:27:28.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:28.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:28.827947+0000) 2024-03-06T16:27:29.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:29.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:29.828223+0000) 2024-03-06T16:27:30.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:30.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:30.828509+0000) 2024-03-06T16:27:31.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:31.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:31.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:31.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:31.828786+0000) 2024-03-06T16:27:32.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:32.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:32.829075+0000) 2024-03-06T16:27:33.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:33.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:33.829345+0000) 2024-03-06T16:27:34.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:34.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:34.829610+0000) 2024-03-06T16:27:35.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:35.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:35.829938+0000) 2024-03-06T16:27:36.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:36.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:36.830223+0000) 2024-03-06T16:27:37.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:37.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:37.830487+0000) 2024-03-06T16:27:38.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:38.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:38.830771+0000) 2024-03-06T16:27:39.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:39.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:39.831031+0000) 2024-03-06T16:27:40.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:40.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:40.831309+0000) 2024-03-06T16:27:41.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:41.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:41.831582+0000) 2024-03-06T16:27:42.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:42.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:42.831914+0000) 2024-03-06T16:27:43.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:43.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:43.832187+0000) 2024-03-06T16:27:44.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:44.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:44.832466+0000) 2024-03-06T16:27:45.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:45.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:45.832735+0000) 2024-03-06T16:27:46.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:46.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:46.832999+0000) 2024-03-06T16:27:47.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:47.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:47.833273+0000) 2024-03-06T16:27:48.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:48.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:48.833547+0000) 2024-03-06T16:27:49.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:49.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:49.833822+0000) 2024-03-06T16:27:50.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:50.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:50.834098+0000) 2024-03-06T16:27:51.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:51.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:51.834370+0000) 2024-03-06T16:27:52.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:52.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:52.834904+0000) 2024-03-06T16:27:53.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:53.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:53.835175+0000) 2024-03-06T16:27:54.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:54.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:54.835435+0000) 2024-03-06T16:27:55.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:55.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:55.835793+0000) 2024-03-06T16:27:56.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:56.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:56.836051+0000) 2024-03-06T16:27:57.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:57.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:57.836273+0000) 2024-03-06T16:27:58.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:58.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:58.836534+0000) 2024-03-06T16:27:59.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:27:59.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:59.836790+0000) 2024-03-06T16:28:00.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:00.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:00.837000+0000) 2024-03-06T16:28:01.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:01.548 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:01.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:01.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:01.837206+0000) 2024-03-06T16:28:02.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:02.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:02.837466+0000) 2024-03-06T16:28:03.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:03.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:03.837756+0000) 2024-03-06T16:28:04.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:04.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:04.837956+0000) 2024-03-06T16:28:05.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:05.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:05.838151+0000) 2024-03-06T16:28:06.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:06.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:06.838351+0000) 2024-03-06T16:28:07.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:07.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:07.838535+0000) 2024-03-06T16:28:08.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:08.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:08.838818+0000) 2024-03-06T16:28:09.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:09.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:09.839090+0000) 2024-03-06T16:28:10.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:10.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:10.839338+0000) 2024-03-06T16:28:11.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:11.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:11.839577+0000) 2024-03-06T16:28:12.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:12.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:12.839840+0000) 2024-03-06T16:28:13.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:13.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:13.840100+0000) 2024-03-06T16:28:14.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:14.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:14.840344+0000) 2024-03-06T16:28:15.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:15.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:15.840581+0000) 2024-03-06T16:28:16.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:16.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:16.840844+0000) 2024-03-06T16:28:17.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:17.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:17.841085+0000) 2024-03-06T16:28:18.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:18.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:18.841328+0000) 2024-03-06T16:28:19.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:19.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:19.841588+0000) 2024-03-06T16:28:20.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:20.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:20.841856+0000) 2024-03-06T16:28:21.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:21.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:21.842112+0000) 2024-03-06T16:28:22.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:22.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:22.842364+0000) 2024-03-06T16:28:23.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:23.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:23.842598+0000) 2024-03-06T16:28:24.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:24.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:24.842856+0000) 2024-03-06T16:28:25.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:25.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:25.843046+0000) 2024-03-06T16:28:26.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:26.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:26.843307+0000) 2024-03-06T16:28:27.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:27.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:27.843598+0000) 2024-03-06T16:28:28.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:28.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:28.843890+0000) 2024-03-06T16:28:29.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:29.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:29.844181+0000) 2024-03-06T16:28:30.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:30.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:30.844417+0000) 2024-03-06T16:28:31.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:31.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:31.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:31.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:31.844761+0000) 2024-03-06T16:28:32.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:32.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:32.845095+0000) 2024-03-06T16:28:33.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:33.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:33.845399+0000) 2024-03-06T16:28:34.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:34.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:34.845760+0000) 2024-03-06T16:28:35.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:35.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:35.846065+0000) 2024-03-06T16:28:36.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:36.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:36.846365+0000) 2024-03-06T16:28:37.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:37.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:37.846701+0000) 2024-03-06T16:28:38.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:38.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:38.847022+0000) 2024-03-06T16:28:39.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:39.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:39.847343+0000) 2024-03-06T16:28:40.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:40.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:40.847593+0000) 2024-03-06T16:28:41.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:41.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:41.847932+0000) 2024-03-06T16:28:42.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:42.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:42.848255+0000) 2024-03-06T16:28:43.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:43.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:43.848570+0000) 2024-03-06T16:28:44.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:44.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:44.848841+0000) 2024-03-06T16:28:45.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:45.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:45.849158+0000) 2024-03-06T16:28:46.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:46.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:46.849480+0000) 2024-03-06T16:28:47.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:47.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:47.849792+0000) 2024-03-06T16:28:48.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:48.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:48.850112+0000) 2024-03-06T16:28:49.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:49.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:49.850435+0000) 2024-03-06T16:28:50.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:50.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:50.850798+0000) 2024-03-06T16:28:51.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:51.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:51.851082+0000) 2024-03-06T16:28:52.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:52.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:52.851415+0000) 2024-03-06T16:28:53.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:53.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:53.851742+0000) 2024-03-06T16:28:54.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:54.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:54.852063+0000) 2024-03-06T16:28:55.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:55.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:55.852376+0000) 2024-03-06T16:28:56.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:56.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:56.852744+0000) 2024-03-06T16:28:57.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:57.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:57.853031+0000) 2024-03-06T16:28:58.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:58.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:58.853335+0000) 2024-03-06T16:28:59.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:28:59.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:59.853697+0000) 2024-03-06T16:29:00.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:00.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:00.854014+0000) 2024-03-06T16:29:01.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:01.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:01.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:01.854321+0000) 2024-03-06T16:29:01.860 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:02.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:02.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:02.854675+0000) 2024-03-06T16:29:03.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:03.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:03.854973+0000) 2024-03-06T16:29:04.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:04.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:04.855254+0000) 2024-03-06T16:29:05.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:05.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:05.855549+0000) 2024-03-06T16:29:06.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:06.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:06.855873+0000) 2024-03-06T16:29:07.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:07.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:07.856179+0000) 2024-03-06T16:29:08.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:08.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:08.856495+0000) 2024-03-06T16:29:09.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:09.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:09.856798+0000) 2024-03-06T16:29:10.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:10.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:10.857111+0000) 2024-03-06T16:29:11.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:11.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:11.857400+0000) 2024-03-06T16:29:12.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:12.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:12.857772+0000) 2024-03-06T16:29:13.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:13.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:13.858071+0000) 2024-03-06T16:29:14.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:14.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:14.858399+0000) 2024-03-06T16:29:15.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:15.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:15.858729+0000) 2024-03-06T16:29:16.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:16.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:16.858966+0000) 2024-03-06T16:29:17.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:17.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:17.859200+0000) 2024-03-06T16:29:18.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:18.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:18.859482+0000) 2024-03-06T16:29:19.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:19.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:19.859757+0000) 2024-03-06T16:29:20.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:20.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:20.859998+0000) 2024-03-06T16:29:21.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:21.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:21.860291+0000) 2024-03-06T16:29:22.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:22.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:22.860597+0000) 2024-03-06T16:29:23.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:23.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:23.860943+0000) 2024-03-06T16:29:24.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:24.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:24.861261+0000) 2024-03-06T16:29:25.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:25.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:25.861583+0000) 2024-03-06T16:29:26.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:26.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:26.861904+0000) 2024-03-06T16:29:27.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:27.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:27.862203+0000) 2024-03-06T16:29:28.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:28.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:28.862524+0000) 2024-03-06T16:29:29.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:29.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:29.862811+0000) 2024-03-06T16:29:30.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:30.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:30.863115+0000) 2024-03-06T16:29:31.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:31.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:31.863415+0000) 2024-03-06T16:29:32.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:32.010 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:32.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:32.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:32.863773+0000) 2024-03-06T16:29:33.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:33.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:33.864069+0000) 2024-03-06T16:29:34.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:34.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:34.864384+0000) 2024-03-06T16:29:35.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:35.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:35.864727+0000) 2024-03-06T16:29:36.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:36.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:36.865051+0000) 2024-03-06T16:29:37.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:37.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:37.865353+0000) 2024-03-06T16:29:38.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:38.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:38.865702+0000) 2024-03-06T16:29:39.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:39.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:39.866039+0000) 2024-03-06T16:29:40.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:40.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:40.866357+0000) 2024-03-06T16:29:41.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:41.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:41.866704+0000) 2024-03-06T16:29:42.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:42.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:42.867057+0000) 2024-03-06T16:29:43.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:43.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:43.867283+0000) 2024-03-06T16:29:44.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:44.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:44.867595+0000) 2024-03-06T16:29:45.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:45.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:45.867943+0000) 2024-03-06T16:29:46.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:46.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:46.868256+0000) 2024-03-06T16:29:47.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:47.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:47.868561+0000) 2024-03-06T16:29:48.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:48.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:48.868879+0000) 2024-03-06T16:29:49.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:49.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:49.869156+0000) 2024-03-06T16:29:50.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:50.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:50.869489+0000) 2024-03-06T16:29:51.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:51.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:51.869797+0000) 2024-03-06T16:29:52.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:52.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:52.870128+0000) 2024-03-06T16:29:53.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:53.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:53.870412+0000) 2024-03-06T16:29:54.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:54.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:54.870752+0000) 2024-03-06T16:29:55.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:55.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:55.870995+0000) 2024-03-06T16:29:56.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:56.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:56.871231+0000) 2024-03-06T16:29:57.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:57.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:57.871527+0000) 2024-03-06T16:29:58.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:58.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:58.871847+0000) 2024-03-06T16:29:59.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:29:59.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:59.872131+0000) 2024-03-06T16:30:00.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:00.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:00.872427+0000) 2024-03-06T16:30:01.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:01.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:01.872763+0000) 2024-03-06T16:30:02.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:02.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:02.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:02.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:02.873091+0000) 2024-03-06T16:30:03.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:03.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:03.873396+0000) 2024-03-06T16:30:04.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:04.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:04.873732+0000) 2024-03-06T16:30:05.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:05.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:05.874022+0000) 2024-03-06T16:30:06.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:06.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:06.874323+0000) 2024-03-06T16:30:07.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:07.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:07.874622+0000) 2024-03-06T16:30:08.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:08.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:08.874970+0000) 2024-03-06T16:30:09.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:09.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:09.875259+0000) 2024-03-06T16:30:10.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:10.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:10.875563+0000) 2024-03-06T16:30:11.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:11.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:11.875872+0000) 2024-03-06T16:30:12.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:12.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:12.876138+0000) 2024-03-06T16:30:13.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:13.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:13.876435+0000) 2024-03-06T16:30:14.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:14.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:14.876787+0000) 2024-03-06T16:30:15.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:15.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:15.877115+0000) 2024-03-06T16:30:16.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:16.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:16.877411+0000) 2024-03-06T16:30:17.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:17.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:17.877736+0000) 2024-03-06T16:30:18.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:18.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:18.878034+0000) 2024-03-06T16:30:19.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:19.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:19.878319+0000) 2024-03-06T16:30:20.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:20.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:20.878623+0000) 2024-03-06T16:30:21.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:21.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:21.878970+0000) 2024-03-06T16:30:22.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:22.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:22.879301+0000) 2024-03-06T16:30:23.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:23.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:23.879606+0000) 2024-03-06T16:30:24.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:24.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:24.879919+0000) 2024-03-06T16:30:25.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:25.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:25.880196+0000) 2024-03-06T16:30:26.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:26.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:26.880437+0000) 2024-03-06T16:30:27.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:27.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:27.880811+0000) 2024-03-06T16:30:28.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:28.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:28.881138+0000) 2024-03-06T16:30:29.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:29.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:29.881432+0000) 2024-03-06T16:30:30.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:30.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:30.881801+0000) 2024-03-06T16:30:31.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:31.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:31.882098+0000) 2024-03-06T16:30:32.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:32.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:32.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:32.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:32.882428+0000) 2024-03-06T16:30:33.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:33.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:33.882771+0000) 2024-03-06T16:30:34.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:34.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:34.883155+0000) 2024-03-06T16:30:35.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:35.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:35.883479+0000) 2024-03-06T16:30:36.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:36.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:36.883863+0000) 2024-03-06T16:30:37.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:37.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:37.884126+0000) 2024-03-06T16:30:38.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:38.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:38.884379+0000) 2024-03-06T16:30:39.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:39.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:39.884716+0000) 2024-03-06T16:30:40.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:40.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:40.884975+0000) 2024-03-06T16:30:41.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:41.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:41.885212+0000) 2024-03-06T16:30:42.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:42.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:42.885518+0000) 2024-03-06T16:30:43.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:43.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:43.885922+0000) 2024-03-06T16:30:44.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:44.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:44.886161+0000) 2024-03-06T16:30:45.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:45.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:45.886452+0000) 2024-03-06T16:30:46.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:46.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:46.886779+0000) 2024-03-06T16:30:47.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:47.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:47.887034+0000) 2024-03-06T16:30:48.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:48.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:48.887284+0000) 2024-03-06T16:30:49.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:49.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:49.887574+0000) 2024-03-06T16:30:50.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:50.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:50.887869+0000) 2024-03-06T16:30:51.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:51.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:51.888112+0000) 2024-03-06T16:30:52.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:52.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:52.888423+0000) 2024-03-06T16:30:53.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:53.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:53.888750+0000) 2024-03-06T16:30:54.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:54.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:54.889048+0000) 2024-03-06T16:30:55.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:55.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:55.889299+0000) 2024-03-06T16:30:56.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:56.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:56.889555+0000) 2024-03-06T16:30:57.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:57.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:57.889824+0000) 2024-03-06T16:30:58.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:58.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:58.890148+0000) 2024-03-06T16:30:59.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:30:59.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:59.890386+0000) 2024-03-06T16:31:00.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:00.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:00.890741+0000) 2024-03-06T16:31:01.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:01.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:01.891025+0000) 2024-03-06T16:31:02.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:02.465 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:02.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:02.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:02.891358+0000) 2024-03-06T16:31:03.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:03.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:03.891688+0000) 2024-03-06T16:31:04.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:04.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:04.891989+0000) 2024-03-06T16:31:05.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:05.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:05.892267+0000) 2024-03-06T16:31:06.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:06.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:06.892559+0000) 2024-03-06T16:31:07.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:07.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:07.892870+0000) 2024-03-06T16:31:08.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:08.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:08.893174+0000) 2024-03-06T16:31:09.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:09.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:09.893521+0000) 2024-03-06T16:31:10.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:10.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:10.893831+0000) 2024-03-06T16:31:11.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:11.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:11.894137+0000) 2024-03-06T16:31:12.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:12.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:12.894454+0000) 2024-03-06T16:31:13.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:13.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:13.894776+0000) 2024-03-06T16:31:14.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:14.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:14.895056+0000) 2024-03-06T16:31:15.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:15.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:15.895361+0000) 2024-03-06T16:31:16.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:16.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:16.895714+0000) 2024-03-06T16:31:17.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:17.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:17.896017+0000) 2024-03-06T16:31:18.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:18.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:18.896321+0000) 2024-03-06T16:31:19.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:19.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:19.896662+0000) 2024-03-06T16:31:20.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:20.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:20.896916+0000) 2024-03-06T16:31:21.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:21.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:21.897223+0000) 2024-03-06T16:31:22.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:22.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:22.897538+0000) 2024-03-06T16:31:23.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:23.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:23.897854+0000) 2024-03-06T16:31:24.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:24.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:24.898176+0000) 2024-03-06T16:31:25.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:25.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:25.898491+0000) 2024-03-06T16:31:26.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:26.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:26.898812+0000) 2024-03-06T16:31:27.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:27.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:27.899095+0000) 2024-03-06T16:31:28.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:28.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:28.899421+0000) 2024-03-06T16:31:29.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:29.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:29.899735+0000) 2024-03-06T16:31:30.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:30.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:30.900063+0000) 2024-03-06T16:31:31.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:31.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:31.900316+0000) 2024-03-06T16:31:32.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:32.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:32.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:32.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:32.900563+0000) 2024-03-06T16:31:33.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:33.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:33.900801+0000) 2024-03-06T16:31:34.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:34.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:34.901068+0000) 2024-03-06T16:31:35.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:35.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:35.901327+0000) 2024-03-06T16:31:36.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:36.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:36.901595+0000) 2024-03-06T16:31:37.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:37.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:37.901837+0000) 2024-03-06T16:31:38.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:38.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:38.902041+0000) 2024-03-06T16:31:39.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:39.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:39.902280+0000) 2024-03-06T16:31:40.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:40.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:40.902542+0000) 2024-03-06T16:31:41.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:41.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:41.902801+0000) 2024-03-06T16:31:42.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:42.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:42.903077+0000) 2024-03-06T16:31:43.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:43.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:43.903315+0000) 2024-03-06T16:31:44.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:44.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:44.903575+0000) 2024-03-06T16:31:45.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:45.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:45.903743+0000) 2024-03-06T16:31:46.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:46.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:46.904006+0000) 2024-03-06T16:31:47.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:47.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:47.904262+0000) 2024-03-06T16:31:48.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:48.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:48.904526+0000) 2024-03-06T16:31:49.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:49.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:49.904755+0000) 2024-03-06T16:31:50.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:50.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:50.905032+0000) 2024-03-06T16:31:51.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:51.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:51.905285+0000) 2024-03-06T16:31:52.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:52.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:52.905511+0000) 2024-03-06T16:31:53.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:53.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:53.905766+0000) 2024-03-06T16:31:54.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:54.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:54.905968+0000) 2024-03-06T16:31:55.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:55.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:55.906199+0000) 2024-03-06T16:31:56.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:56.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:56.906461+0000) 2024-03-06T16:31:57.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:57.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:57.906760+0000) 2024-03-06T16:31:58.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:58.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:58.907003+0000) 2024-03-06T16:31:59.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:31:59.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:59.907242+0000) 2024-03-06T16:32:00.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:00.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:00.907505+0000) 2024-03-06T16:32:01.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:01.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:01.907732+0000) 2024-03-06T16:32:02.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:02.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:02.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:02.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:02.907997+0000) 2024-03-06T16:32:03.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:03.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:03.908285+0000) 2024-03-06T16:32:04.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:04.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:04.908530+0000) 2024-03-06T16:32:05.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:05.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:05.908784+0000) 2024-03-06T16:32:06.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:06.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:06.908973+0000) 2024-03-06T16:32:07.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:07.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:07.909224+0000) 2024-03-06T16:32:08.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:08.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:08.909476+0000) 2024-03-06T16:32:09.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:09.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:09.909747+0000) 2024-03-06T16:32:10.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:10.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:10.910013+0000) 2024-03-06T16:32:11.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:11.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:11.910271+0000) 2024-03-06T16:32:12.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:12.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:12.910541+0000) 2024-03-06T16:32:13.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:13.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:13.910795+0000) 2024-03-06T16:32:14.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:14.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:14.911028+0000) 2024-03-06T16:32:15.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:15.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:15.911250+0000) 2024-03-06T16:32:16.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:16.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:16.911514+0000) 2024-03-06T16:32:17.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:17.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:17.911752+0000) 2024-03-06T16:32:18.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:18.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:18.911951+0000) 2024-03-06T16:32:19.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:19.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:19.912209+0000) 2024-03-06T16:32:20.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:20.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:20.912460+0000) 2024-03-06T16:32:21.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:21.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:21.912734+0000) 2024-03-06T16:32:22.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:22.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:22.912988+0000) 2024-03-06T16:32:23.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:23.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:23.913181+0000) 2024-03-06T16:32:24.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:24.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:24.913374+0000) 2024-03-06T16:32:25.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:25.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:25.913564+0000) 2024-03-06T16:32:26.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:26.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:26.913841+0000) 2024-03-06T16:32:27.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:27.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:27.914085+0000) 2024-03-06T16:32:28.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:28.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:28.914348+0000) 2024-03-06T16:32:29.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:29.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:29.914606+0000) 2024-03-06T16:32:30.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:30.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:30.914914+0000) 2024-03-06T16:32:31.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:31.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:31.915154+0000) 2024-03-06T16:32:32.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:32.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:32.915404+0000) 2024-03-06T16:32:32.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:32.931 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:33.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:33.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:33.915589+0000) 2024-03-06T16:32:34.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:34.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:34.915854+0000) 2024-03-06T16:32:35.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:35.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:35.916122+0000) 2024-03-06T16:32:36.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:36.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:36.916387+0000) 2024-03-06T16:32:37.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:37.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:37.916687+0000) 2024-03-06T16:32:38.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:38.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:38.916964+0000) 2024-03-06T16:32:39.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:39.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:39.917202+0000) 2024-03-06T16:32:40.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:40.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:40.917469+0000) 2024-03-06T16:32:41.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:41.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:41.917768+0000) 2024-03-06T16:32:42.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:42.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:42.918055+0000) 2024-03-06T16:32:43.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:43.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:43.918312+0000) 2024-03-06T16:32:44.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:44.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:44.918570+0000) 2024-03-06T16:32:45.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:45.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:45.918818+0000) 2024-03-06T16:32:46.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:46.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:46.919082+0000) 2024-03-06T16:32:47.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:47.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:47.919314+0000) 2024-03-06T16:32:48.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:48.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:48.919585+0000) 2024-03-06T16:32:49.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:49.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:49.919846+0000) 2024-03-06T16:32:50.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:50.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:50.920097+0000) 2024-03-06T16:32:51.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:51.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:51.920366+0000) 2024-03-06T16:32:52.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:52.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:52.920684+0000) 2024-03-06T16:32:53.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:53.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:53.920951+0000) 2024-03-06T16:32:54.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:54.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:54.921193+0000) 2024-03-06T16:32:55.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:55.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:55.921447+0000) 2024-03-06T16:32:56.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:56.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:56.921813+0000) 2024-03-06T16:32:57.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:57.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:57.922076+0000) 2024-03-06T16:32:58.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:58.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:58.922316+0000) 2024-03-06T16:32:59.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:32:59.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:59.922546+0000) 2024-03-06T16:33:00.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:00.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:00.922793+0000) 2024-03-06T16:33:01.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:01.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:01.922982+0000) 2024-03-06T16:33:02.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:02.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:02.923237+0000) 2024-03-06T16:33:03.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:03.070 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:03.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:03.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:03.923483+0000) 2024-03-06T16:33:04.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:04.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:04.923763+0000) 2024-03-06T16:33:05.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:05.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:05.924031+0000) 2024-03-06T16:33:06.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:06.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:06.924299+0000) 2024-03-06T16:33:07.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:07.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:07.924554+0000) 2024-03-06T16:33:08.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:08.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:08.924793+0000) 2024-03-06T16:33:09.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:09.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:09.925046+0000) 2024-03-06T16:33:10.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:10.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:10.925316+0000) 2024-03-06T16:33:11.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:11.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:11.925567+0000) 2024-03-06T16:33:12.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:12.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:12.925821+0000) 2024-03-06T16:33:13.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:13.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:13.926075+0000) 2024-03-06T16:33:14.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:14.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:14.926304+0000) 2024-03-06T16:33:15.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:15.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:15.926553+0000) 2024-03-06T16:33:16.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:16.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:16.926817+0000) 2024-03-06T16:33:17.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:17.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:17.927067+0000) 2024-03-06T16:33:18.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:18.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:18.927333+0000) 2024-03-06T16:33:19.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:19.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:19.927566+0000) 2024-03-06T16:33:20.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:20.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:20.927832+0000) 2024-03-06T16:33:21.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:21.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:21.928081+0000) 2024-03-06T16:33:22.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:22.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:22.928335+0000) 2024-03-06T16:33:23.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:23.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:23.928578+0000) 2024-03-06T16:33:24.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:24.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:24.928756+0000) 2024-03-06T16:33:25.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:25.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:25.929017+0000) 2024-03-06T16:33:26.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:26.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:26.929287+0000) 2024-03-06T16:33:27.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:27.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:27.929536+0000) 2024-03-06T16:33:28.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:28.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:28.929807+0000) 2024-03-06T16:33:29.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:29.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:29.930054+0000) 2024-03-06T16:33:30.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:30.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:30.930301+0000) 2024-03-06T16:33:31.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:31.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:31.930535+0000) 2024-03-06T16:33:32.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:32.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:32.930793+0000) 2024-03-06T16:33:33.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:33.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:33.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:33.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:33.931053+0000) 2024-03-06T16:33:34.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:34.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:34.931321+0000) 2024-03-06T16:33:35.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:35.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:35.931515+0000) 2024-03-06T16:33:36.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:36.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:36.931765+0000) 2024-03-06T16:33:37.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:37.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:37.932005+0000) 2024-03-06T16:33:38.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:38.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:38.932252+0000) 2024-03-06T16:33:39.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:39.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:39.932502+0000) 2024-03-06T16:33:40.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:40.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:40.932769+0000) 2024-03-06T16:33:41.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:41.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:41.933019+0000) 2024-03-06T16:33:42.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:42.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:42.933288+0000) 2024-03-06T16:33:43.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:43.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:43.933547+0000) 2024-03-06T16:33:44.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:44.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:44.933828+0000) 2024-03-06T16:33:45.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:45.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:45.934089+0000) 2024-03-06T16:33:46.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:46.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:46.934381+0000) 2024-03-06T16:33:47.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:47.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:47.934689+0000) 2024-03-06T16:33:48.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:48.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:48.934979+0000) 2024-03-06T16:33:49.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:49.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:49.935277+0000) 2024-03-06T16:33:50.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:50.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:50.935560+0000) 2024-03-06T16:33:51.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:51.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:51.935820+0000) 2024-03-06T16:33:52.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:52.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:52.936118+0000) 2024-03-06T16:33:53.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:53.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:53.936371+0000) 2024-03-06T16:33:54.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:54.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:54.936695+0000) 2024-03-06T16:33:55.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:55.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:55.936981+0000) 2024-03-06T16:33:56.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:56.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:56.937237+0000) 2024-03-06T16:33:57.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:57.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:57.937493+0000) 2024-03-06T16:33:58.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:58.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:58.937730+0000) 2024-03-06T16:33:59.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:33:59.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:59.937955+0000) 2024-03-06T16:34:00.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:00.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:00.938220+0000) 2024-03-06T16:34:01.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:01.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:01.938470+0000) 2024-03-06T16:34:02.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:02.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:02.938748+0000) 2024-03-06T16:34:03.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:03.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:03.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:03.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:03.938985+0000) 2024-03-06T16:34:04.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:04.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:04.939239+0000) 2024-03-06T16:34:05.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:05.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:05.939478+0000) 2024-03-06T16:34:06.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:06.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:06.939746+0000) 2024-03-06T16:34:07.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:07.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:07.940003+0000) 2024-03-06T16:34:08.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:08.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:08.940256+0000) 2024-03-06T16:34:09.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:09.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:09.940550+0000) 2024-03-06T16:34:10.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:10.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:10.940831+0000) 2024-03-06T16:34:11.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:11.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:11.941129+0000) 2024-03-06T16:34:12.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:12.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:12.941420+0000) 2024-03-06T16:34:13.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:13.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:13.941671+0000) 2024-03-06T16:34:14.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:14.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:14.941892+0000) 2024-03-06T16:34:15.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:15.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:15.942177+0000) 2024-03-06T16:34:16.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:16.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:16.942465+0000) 2024-03-06T16:34:17.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:17.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:17.942801+0000) 2024-03-06T16:34:18.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:18.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:18.943082+0000) 2024-03-06T16:34:19.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:19.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:19.943288+0000) 2024-03-06T16:34:20.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:20.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:20.943594+0000) 2024-03-06T16:34:21.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:21.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:21.943962+0000) 2024-03-06T16:34:22.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:22.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:22.944291+0000) 2024-03-06T16:34:23.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:23.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:23.944585+0000) 2024-03-06T16:34:24.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:24.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:24.944948+0000) 2024-03-06T16:34:25.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:25.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:25.945232+0000) 2024-03-06T16:34:26.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:26.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:26.945483+0000) 2024-03-06T16:34:27.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:27.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:27.945778+0000) 2024-03-06T16:34:28.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:28.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:28.946107+0000) 2024-03-06T16:34:29.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:29.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:29.946411+0000) 2024-03-06T16:34:30.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:30.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:30.946779+0000) 2024-03-06T16:34:31.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:31.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:31.947052+0000) 2024-03-06T16:34:32.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:32.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:32.947361+0000) 2024-03-06T16:34:33.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:33.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:33.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:33.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:33.947622+0000) 2024-03-06T16:34:34.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:34.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:34.947931+0000) 2024-03-06T16:34:35.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:35.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:35.948189+0000) 2024-03-06T16:34:36.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:36.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:36.948456+0000) 2024-03-06T16:34:37.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:37.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:37.948803+0000) 2024-03-06T16:34:38.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:38.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:38.949064+0000) 2024-03-06T16:34:39.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:39.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:39.949263+0000) 2024-03-06T16:34:40.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:40.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:40.949525+0000) 2024-03-06T16:34:41.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:41.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:41.949767+0000) 2024-03-06T16:34:42.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:42.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:42.950050+0000) 2024-03-06T16:34:43.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:43.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:43.950282+0000) 2024-03-06T16:34:44.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:44.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:44.950531+0000) 2024-03-06T16:34:45.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:45.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:45.950782+0000) 2024-03-06T16:34:46.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:46.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:46.951056+0000) 2024-03-06T16:34:47.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:47.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:47.951315+0000) 2024-03-06T16:34:48.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:48.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:48.951587+0000) 2024-03-06T16:34:49.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:49.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:49.951826+0000) 2024-03-06T16:34:50.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:50.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:50.952089+0000) 2024-03-06T16:34:51.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:51.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:51.952338+0000) 2024-03-06T16:34:52.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:52.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:52.952598+0000) 2024-03-06T16:34:53.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:53.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:53.952841+0000) 2024-03-06T16:34:54.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:54.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:54.953037+0000) 2024-03-06T16:34:55.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:55.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:55.953292+0000) 2024-03-06T16:34:56.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:56.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:56.953543+0000) 2024-03-06T16:34:57.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:57.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:57.953801+0000) 2024-03-06T16:34:58.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:58.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:58.954070+0000) 2024-03-06T16:34:59.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:34:59.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:59.954340+0000) 2024-03-06T16:35:00.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:00.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:00.954618+0000) 2024-03-06T16:35:01.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:01.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:01.954936+0000) 2024-03-06T16:35:02.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:02.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:02.955230+0000) 2024-03-06T16:35:03.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:03.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:03.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:03.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:03.955513+0000) 2024-03-06T16:35:04.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:04.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:04.955776+0000) 2024-03-06T16:35:05.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:05.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:05.956055+0000) 2024-03-06T16:35:06.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:06.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:06.956335+0000) 2024-03-06T16:35:07.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:07.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:07.956582+0000) 2024-03-06T16:35:08.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:08.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:08.956858+0000) 2024-03-06T16:35:09.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:09.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:09.957117+0000) 2024-03-06T16:35:10.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:10.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:10.957324+0000) 2024-03-06T16:35:11.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:11.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:11.957607+0000) 2024-03-06T16:35:12.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:12.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:12.957977+0000) 2024-03-06T16:35:13.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:13.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:13.958229+0000) 2024-03-06T16:35:14.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:14.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:14.958509+0000) 2024-03-06T16:35:15.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:15.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:15.958763+0000) 2024-03-06T16:35:16.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:16.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:16.959037+0000) 2024-03-06T16:35:17.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:17.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:17.959297+0000) 2024-03-06T16:35:18.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:18.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:18.959549+0000) 2024-03-06T16:35:19.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:19.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:19.959825+0000) 2024-03-06T16:35:20.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:20.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:20.960091+0000) 2024-03-06T16:35:21.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:21.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:21.960354+0000) 2024-03-06T16:35:22.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:22.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:22.960676+0000) 2024-03-06T16:35:23.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:23.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:23.960938+0000) 2024-03-06T16:35:24.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:24.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:24.961199+0000) 2024-03-06T16:35:25.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:25.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:25.961471+0000) 2024-03-06T16:35:26.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:26.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:26.961753+0000) 2024-03-06T16:35:27.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:27.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:27.962042+0000) 2024-03-06T16:35:28.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:28.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:28.962297+0000) 2024-03-06T16:35:29.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:29.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:29.962542+0000) 2024-03-06T16:35:30.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:30.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:30.962822+0000) 2024-03-06T16:35:31.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:31.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:31.963076+0000) 2024-03-06T16:35:32.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:32.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:32.963369+0000) 2024-03-06T16:35:33.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:33.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:33.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:33.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:33.963665+0000) 2024-03-06T16:35:34.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:34.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:34.963946+0000) 2024-03-06T16:35:35.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:35.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:35.964197+0000) 2024-03-06T16:35:36.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:36.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:36.964481+0000) 2024-03-06T16:35:37.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:37.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:37.964732+0000) 2024-03-06T16:35:38.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:38.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:38.965014+0000) 2024-03-06T16:35:39.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:39.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:39.965264+0000) 2024-03-06T16:35:40.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:40.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:40.965525+0000) 2024-03-06T16:35:41.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:41.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:41.965794+0000) 2024-03-06T16:35:42.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:42.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:42.966086+0000) 2024-03-06T16:35:43.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:43.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:43.966334+0000) 2024-03-06T16:35:44.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:44.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:44.966597+0000) 2024-03-06T16:35:45.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:45.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:45.966887+0000) 2024-03-06T16:35:46.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:46.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:46.967150+0000) 2024-03-06T16:35:47.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:47.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:47.967438+0000) 2024-03-06T16:35:48.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:48.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:48.967730+0000) 2024-03-06T16:35:49.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:49.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:49.967985+0000) 2024-03-06T16:35:50.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:50.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:50.968253+0000) 2024-03-06T16:35:51.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:51.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:51.968506+0000) 2024-03-06T16:35:52.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:52.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:52.968798+0000) 2024-03-06T16:35:53.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:53.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:53.969067+0000) 2024-03-06T16:35:54.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:54.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:54.969354+0000) 2024-03-06T16:35:55.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:55.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:55.969588+0000) 2024-03-06T16:35:56.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:56.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:56.969935+0000) 2024-03-06T16:35:57.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:57.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:57.970213+0000) 2024-03-06T16:35:58.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:58.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:58.970493+0000) 2024-03-06T16:35:59.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:35:59.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:59.970766+0000) 2024-03-06T16:36:00.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:00.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:00.971027+0000) 2024-03-06T16:36:01.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:01.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:01.971229+0000) 2024-03-06T16:36:02.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:02.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:02.971523+0000) 2024-03-06T16:36:03.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:03.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:03.971777+0000) 2024-03-06T16:36:04.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:04.019 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:04.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:04.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:04.972062+0000) 2024-03-06T16:36:05.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:05.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:05.972330+0000) 2024-03-06T16:36:06.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:06.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:06.972612+0000) 2024-03-06T16:36:07.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:07.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:07.972906+0000) 2024-03-06T16:36:08.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:08.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:08.973173+0000) 2024-03-06T16:36:09.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:09.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:09.973441+0000) 2024-03-06T16:36:10.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:10.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:10.973760+0000) 2024-03-06T16:36:11.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:11.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:11.974031+0000) 2024-03-06T16:36:12.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:12.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:12.974321+0000) 2024-03-06T16:36:13.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:13.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:13.974578+0000) 2024-03-06T16:36:14.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:14.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:14.974864+0000) 2024-03-06T16:36:15.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:15.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:15.975115+0000) 2024-03-06T16:36:16.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:16.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:16.975379+0000) 2024-03-06T16:36:17.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:17.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:17.975688+0000) 2024-03-06T16:36:18.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:18.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:18.975981+0000) 2024-03-06T16:36:19.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:19.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:19.976256+0000) 2024-03-06T16:36:20.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:20.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:20.976536+0000) 2024-03-06T16:36:21.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:21.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:21.976793+0000) 2024-03-06T16:36:22.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:22.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:22.977080+0000) 2024-03-06T16:36:23.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:23.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:23.977331+0000) 2024-03-06T16:36:24.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:24.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:24.977619+0000) 2024-03-06T16:36:25.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:25.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:25.977919+0000) 2024-03-06T16:36:26.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:26.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:26.978200+0000) 2024-03-06T16:36:27.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:27.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:27.978468+0000) 2024-03-06T16:36:28.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:28.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:28.978775+0000) 2024-03-06T16:36:29.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:29.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:29.979046+0000) 2024-03-06T16:36:30.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:30.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:30.979326+0000) 2024-03-06T16:36:31.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:31.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:31.979559+0000) 2024-03-06T16:36:32.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:32.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:32.979817+0000) 2024-03-06T16:36:33.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:33.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:33.980071+0000) 2024-03-06T16:36:34.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:34.188 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:34.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:34.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:34.980288+0000) 2024-03-06T16:36:35.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:35.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:35.980521+0000) 2024-03-06T16:36:36.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:36.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:36.980771+0000) 2024-03-06T16:36:37.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:37.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:37.981004+0000) 2024-03-06T16:36:38.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:38.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:38.981269+0000) 2024-03-06T16:36:39.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:39.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:39.981520+0000) 2024-03-06T16:36:40.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:40.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:40.981786+0000) 2024-03-06T16:36:41.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:41.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:41.982021+0000) 2024-03-06T16:36:42.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:42.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:42.982291+0000) 2024-03-06T16:36:43.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:43.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:43.982520+0000) 2024-03-06T16:36:44.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:44.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:44.982785+0000) 2024-03-06T16:36:45.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:45.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:45.982992+0000) 2024-03-06T16:36:46.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:46.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:46.983203+0000) 2024-03-06T16:36:47.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:47.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:47.983406+0000) 2024-03-06T16:36:48.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:48.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:48.983735+0000) 2024-03-06T16:36:49.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:49.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:49.984007+0000) 2024-03-06T16:36:50.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:50.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:50.984278+0000) 2024-03-06T16:36:51.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:51.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:51.984545+0000) 2024-03-06T16:36:52.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:52.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:52.984818+0000) 2024-03-06T16:36:53.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:53.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:53.985055+0000) 2024-03-06T16:36:54.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:54.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:54.985309+0000) 2024-03-06T16:36:55.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:55.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:55.985559+0000) 2024-03-06T16:36:56.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:56.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:56.985818+0000) 2024-03-06T16:36:57.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:57.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:57.986072+0000) 2024-03-06T16:36:58.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:58.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:58.986338+0000) 2024-03-06T16:36:59.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:36:59.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:59.986572+0000) 2024-03-06T16:37:00.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:00.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:00.986809+0000) 2024-03-06T16:37:01.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:01.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:01.987053+0000) 2024-03-06T16:37:02.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:02.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:02.987264+0000) 2024-03-06T16:37:03.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:03.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:03.987502+0000) 2024-03-06T16:37:04.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:04.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:04.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:04.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:04.987758+0000) 2024-03-06T16:37:05.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:05.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:05.988005+0000) 2024-03-06T16:37:06.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:06.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:06.988270+0000) 2024-03-06T16:37:07.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:07.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:07.988558+0000) 2024-03-06T16:37:08.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:08.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:08.988815+0000) 2024-03-06T16:37:09.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:09.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:09.989065+0000) 2024-03-06T16:37:10.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:10.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:10.989304+0000) 2024-03-06T16:37:11.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:11.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:11.989556+0000) 2024-03-06T16:37:12.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:12.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:12.989834+0000) 2024-03-06T16:37:13.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:13.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:13.990083+0000) 2024-03-06T16:37:14.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:14.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:14.990335+0000) 2024-03-06T16:37:15.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:15.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:15.990577+0000) 2024-03-06T16:37:16.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:16.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:16.990785+0000) 2024-03-06T16:37:17.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:17.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:17.991047+0000) 2024-03-06T16:37:18.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:18.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:18.991299+0000) 2024-03-06T16:37:19.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:19.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:19.991546+0000) 2024-03-06T16:37:20.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:20.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:20.991806+0000) 2024-03-06T16:37:21.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:21.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:21.992042+0000) 2024-03-06T16:37:22.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:22.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:22.992289+0000) 2024-03-06T16:37:23.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:23.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:23.992546+0000) 2024-03-06T16:37:24.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:24.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:24.992795+0000) 2024-03-06T16:37:25.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:25.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:25.993048+0000) 2024-03-06T16:37:26.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:26.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:26.993313+0000) 2024-03-06T16:37:27.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:27.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:27.993565+0000) 2024-03-06T16:37:28.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:28.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:28.993815+0000) 2024-03-06T16:37:29.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:29.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:29.994047+0000) 2024-03-06T16:37:30.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:30.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:30.994293+0000) 2024-03-06T16:37:31.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:31.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:31.994567+0000) 2024-03-06T16:37:32.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:32.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:32.994894+0000) 2024-03-06T16:37:33.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:33.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:33.995147+0000) 2024-03-06T16:37:34.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:34.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:34.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:34.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:34.995418+0000) 2024-03-06T16:37:35.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:35.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:35.995709+0000) 2024-03-06T16:37:36.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:36.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:36.995907+0000) 2024-03-06T16:37:37.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:37.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:37.996166+0000) 2024-03-06T16:37:38.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:38.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:38.996432+0000) 2024-03-06T16:37:39.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:39.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:39.996720+0000) 2024-03-06T16:37:40.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:40.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:40.996964+0000) 2024-03-06T16:37:41.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:41.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:41.997232+0000) 2024-03-06T16:37:42.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:42.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:42.997492+0000) 2024-03-06T16:37:43.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:43.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:43.997749+0000) 2024-03-06T16:37:44.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:44.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:44.997955+0000) 2024-03-06T16:37:45.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:45.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:45.998188+0000) 2024-03-06T16:37:46.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:46.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:46.998435+0000) 2024-03-06T16:37:47.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:47.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:47.998741+0000) 2024-03-06T16:37:48.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:48.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:48.999006+0000) 2024-03-06T16:37:49.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:49.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:49.999262+0000) 2024-03-06T16:37:51.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:50.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:50.999520+0000) 2024-03-06T16:37:52.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:51.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:51.999783+0000) 2024-03-06T16:37:53.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:52.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:53.000055+0000) 2024-03-06T16:37:54.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:53.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:54.000306+0000) 2024-03-06T16:37:55.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:54.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:55.000576+0000) 2024-03-06T16:37:56.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:55.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:56.000863+0000) 2024-03-06T16:37:57.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:57.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:57.001128+0000) 2024-03-06T16:37:58.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:58.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:58.001386+0000) 2024-03-06T16:37:59.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:37:58.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:59.001603+0000) 2024-03-06T16:38:00.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:00.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:00.001878+0000) 2024-03-06T16:38:01.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:01.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:01.002145+0000) 2024-03-06T16:38:02.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:02.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:02.002409+0000) 2024-03-06T16:38:03.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:03.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:03.002733+0000) 2024-03-06T16:38:04.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:04.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:04.002986+0000) 2024-03-06T16:38:04.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:04.638 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:05.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:05.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:05.003242+0000) 2024-03-06T16:38:06.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:06.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:06.003488+0000) 2024-03-06T16:38:07.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:07.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:07.003747+0000) 2024-03-06T16:38:08.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:08.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:08.003944+0000) 2024-03-06T16:38:09.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:09.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:09.004210+0000) 2024-03-06T16:38:10.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:10.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:10.004450+0000) 2024-03-06T16:38:11.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:11.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:11.004731+0000) 2024-03-06T16:38:12.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:12.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:12.004939+0000) 2024-03-06T16:38:13.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:13.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:13.005144+0000) 2024-03-06T16:38:14.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:14.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:14.005349+0000) 2024-03-06T16:38:15.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:15.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:15.005609+0000) 2024-03-06T16:38:16.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:16.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:16.005919+0000) 2024-03-06T16:38:17.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:17.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:17.006184+0000) 2024-03-06T16:38:18.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:18.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:18.006439+0000) 2024-03-06T16:38:19.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:19.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:19.006760+0000) 2024-03-06T16:38:20.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:20.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:20.007017+0000) 2024-03-06T16:38:21.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:21.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:21.007280+0000) 2024-03-06T16:38:22.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:22.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:22.007529+0000) 2024-03-06T16:38:23.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:23.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:23.007807+0000) 2024-03-06T16:38:24.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:24.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:24.008073+0000) 2024-03-06T16:38:25.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:25.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:25.008332+0000) 2024-03-06T16:38:26.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:26.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:26.008588+0000) 2024-03-06T16:38:27.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:27.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:27.008837+0000) 2024-03-06T16:38:28.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:28.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:28.009033+0000) 2024-03-06T16:38:29.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:29.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:29.009281+0000) 2024-03-06T16:38:30.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:30.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:30.009520+0000) 2024-03-06T16:38:31.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:31.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:31.009765+0000) 2024-03-06T16:38:32.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:32.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:32.010015+0000) 2024-03-06T16:38:33.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:33.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:33.010236+0000) 2024-03-06T16:38:34.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:34.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:34.010476+0000) 2024-03-06T16:38:34.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:34.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:35.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:35.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:35.010779+0000) 2024-03-06T16:38:36.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:36.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:36.011028+0000) 2024-03-06T16:38:37.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:37.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:37.011299+0000) 2024-03-06T16:38:38.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:38.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:38.011570+0000) 2024-03-06T16:38:39.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:39.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:39.011817+0000) 2024-03-06T16:38:40.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:40.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:40.012075+0000) 2024-03-06T16:38:41.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:41.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:41.012321+0000) 2024-03-06T16:38:42.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:42.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:42.012584+0000) 2024-03-06T16:38:43.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:43.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:43.012844+0000) 2024-03-06T16:38:44.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:44.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:44.013100+0000) 2024-03-06T16:38:45.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:45.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:45.013364+0000) 2024-03-06T16:38:46.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:46.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:46.013566+0000) 2024-03-06T16:38:47.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:47.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:47.013812+0000) 2024-03-06T16:38:48.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:48.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:48.014062+0000) 2024-03-06T16:38:49.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:49.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:49.014381+0000) 2024-03-06T16:38:50.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:50.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:50.014624+0000) 2024-03-06T16:38:51.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:51.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:51.014919+0000) 2024-03-06T16:38:52.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:52.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:52.015179+0000) 2024-03-06T16:38:53.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:53.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:53.015460+0000) 2024-03-06T16:38:54.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:54.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:54.015746+0000) 2024-03-06T16:38:55.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:55.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:55.015988+0000) 2024-03-06T16:38:56.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:56.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:56.016242+0000) 2024-03-06T16:38:57.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:57.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:57.016506+0000) 2024-03-06T16:38:58.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:58.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:58.016783+0000) 2024-03-06T16:38:59.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:38:59.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:59.017042+0000) 2024-03-06T16:39:00.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:00.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:00.017297+0000) 2024-03-06T16:39:01.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:01.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:01.017579+0000) 2024-03-06T16:39:02.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:02.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:02.017851+0000) 2024-03-06T16:39:03.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:03.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:03.018148+0000) 2024-03-06T16:39:04.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:04.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:04.018419+0000) 2024-03-06T16:39:04.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:04.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:05.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:05.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:05.018652+0000) 2024-03-06T16:39:06.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:06.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:06.018905+0000) 2024-03-06T16:39:07.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:07.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:07.019189+0000) 2024-03-06T16:39:08.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:08.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:08.019461+0000) 2024-03-06T16:39:09.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:09.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:09.019775+0000) 2024-03-06T16:39:10.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:10.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:10.020022+0000) 2024-03-06T16:39:11.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:11.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:11.020288+0000) 2024-03-06T16:39:12.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:12.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:12.020528+0000) 2024-03-06T16:39:13.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:13.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:13.020799+0000) 2024-03-06T16:39:14.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:14.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:14.021067+0000) 2024-03-06T16:39:15.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:15.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:15.021347+0000) 2024-03-06T16:39:16.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:16.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:16.021609+0000) 2024-03-06T16:39:17.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:17.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:17.021937+0000) 2024-03-06T16:39:18.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:18.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:18.022200+0000) 2024-03-06T16:39:19.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:19.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:19.022486+0000) 2024-03-06T16:39:20.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:20.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:20.022737+0000) 2024-03-06T16:39:21.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:21.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:21.023017+0000) 2024-03-06T16:39:22.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:22.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:22.023299+0000) 2024-03-06T16:39:23.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:23.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:23.023595+0000) 2024-03-06T16:39:24.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:24.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:24.023903+0000) 2024-03-06T16:39:25.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:25.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:25.024176+0000) 2024-03-06T16:39:26.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:26.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:26.024436+0000) 2024-03-06T16:39:27.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:27.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:27.024769+0000) 2024-03-06T16:39:28.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:28.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:28.025045+0000) 2024-03-06T16:39:29.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:29.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:29.025327+0000) 2024-03-06T16:39:30.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:30.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:30.025572+0000) 2024-03-06T16:39:31.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:31.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:31.025860+0000) 2024-03-06T16:39:32.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:32.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:32.026131+0000) 2024-03-06T16:39:33.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:33.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:33.026423+0000) 2024-03-06T16:39:34.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:34.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:34.026752+0000) 2024-03-06T16:39:35.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:35.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:35.027030+0000) 2024-03-06T16:39:35.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:35.089 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:36.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:36.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:36.027280+0000) 2024-03-06T16:39:37.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:37.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:37.027614+0000) 2024-03-06T16:39:38.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:38.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:38.027949+0000) 2024-03-06T16:39:39.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:39.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:39.028169+0000) 2024-03-06T16:39:40.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:40.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:40.028436+0000) 2024-03-06T16:39:41.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:41.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:41.028748+0000) 2024-03-06T16:39:42.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:42.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:42.029019+0000) 2024-03-06T16:39:43.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:43.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:43.029299+0000) 2024-03-06T16:39:44.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:44.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:44.029568+0000) 2024-03-06T16:39:45.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:45.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:45.029853+0000) 2024-03-06T16:39:46.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:46.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:46.030120+0000) 2024-03-06T16:39:47.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:47.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:47.030385+0000) 2024-03-06T16:39:48.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:48.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:48.030722+0000) 2024-03-06T16:39:49.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:49.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:49.030948+0000) 2024-03-06T16:39:50.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:50.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:50.031227+0000) 2024-03-06T16:39:51.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:51.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:51.031488+0000) 2024-03-06T16:39:52.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:52.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:52.031764+0000) 2024-03-06T16:39:53.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:53.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:53.032047+0000) 2024-03-06T16:39:54.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:54.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:54.032311+0000) 2024-03-06T16:39:55.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:55.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:55.032586+0000) 2024-03-06T16:39:56.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:56.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:56.032911+0000) 2024-03-06T16:39:57.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:57.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:57.033192+0000) 2024-03-06T16:39:58.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:58.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:58.033491+0000) 2024-03-06T16:39:59.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:39:59.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:59.033776+0000) 2024-03-06T16:40:00.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:00.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:00.034072+0000) 2024-03-06T16:40:01.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:01.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:01.034369+0000) 2024-03-06T16:40:02.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:02.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:02.034717+0000) 2024-03-06T16:40:03.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:03.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:03.035042+0000) 2024-03-06T16:40:04.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:04.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:04.035336+0000) 2024-03-06T16:40:05.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:05.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:05.035589+0000) 2024-03-06T16:40:05.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:05.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:06.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:06.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:06.035825+0000) 2024-03-06T16:40:07.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:07.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:07.036088+0000) 2024-03-06T16:40:08.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:08.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:08.036289+0000) 2024-03-06T16:40:09.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:09.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:09.036562+0000) 2024-03-06T16:40:10.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:10.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:10.036799+0000) 2024-03-06T16:40:11.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:11.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:11.037061+0000) 2024-03-06T16:40:12.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:12.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:12.037302+0000) 2024-03-06T16:40:13.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:13.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:13.037578+0000) 2024-03-06T16:40:14.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:14.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:14.037839+0000) 2024-03-06T16:40:15.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:15.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:15.038082+0000) 2024-03-06T16:40:16.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:16.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:16.038339+0000) 2024-03-06T16:40:17.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:17.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:17.038609+0000) 2024-03-06T16:40:18.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:18.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:18.038915+0000) 2024-03-06T16:40:19.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:19.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:19.039173+0000) 2024-03-06T16:40:20.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:20.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:20.039413+0000) 2024-03-06T16:40:21.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:21.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:21.039731+0000) 2024-03-06T16:40:22.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:22.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:22.039987+0000) 2024-03-06T16:40:23.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:23.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:23.040263+0000) 2024-03-06T16:40:24.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:24.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:24.040496+0000) 2024-03-06T16:40:25.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:25.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:25.040809+0000) 2024-03-06T16:40:26.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:26.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:26.041051+0000) 2024-03-06T16:40:27.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:27.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:27.041295+0000) 2024-03-06T16:40:28.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:28.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:28.041532+0000) 2024-03-06T16:40:29.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:29.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:29.041816+0000) 2024-03-06T16:40:30.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:30.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:30.042085+0000) 2024-03-06T16:40:31.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:31.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:31.042343+0000) 2024-03-06T16:40:32.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:32.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:32.042607+0000) 2024-03-06T16:40:33.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:33.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:33.042933+0000) 2024-03-06T16:40:34.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:34.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:34.043127+0000) 2024-03-06T16:40:35.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:35.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:35.043324+0000) 2024-03-06T16:40:35.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:35.407 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:36.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:36.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:36.043578+0000) 2024-03-06T16:40:37.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:37.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:37.043848+0000) 2024-03-06T16:40:38.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:38.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:38.044149+0000) 2024-03-06T16:40:39.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:39.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:39.044408+0000) 2024-03-06T16:40:40.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:40.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:40.044737+0000) 2024-03-06T16:40:41.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:41.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:41.044984+0000) 2024-03-06T16:40:42.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:42.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:42.045241+0000) 2024-03-06T16:40:43.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:43.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:43.045519+0000) 2024-03-06T16:40:44.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:44.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:44.045774+0000) 2024-03-06T16:40:45.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:45.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:45.045985+0000) 2024-03-06T16:40:46.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:46.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:46.046245+0000) 2024-03-06T16:40:47.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:47.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:47.046501+0000) 2024-03-06T16:40:48.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:48.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:48.046759+0000) 2024-03-06T16:40:49.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:49.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:49.047026+0000) 2024-03-06T16:40:50.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:50.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:50.047288+0000) 2024-03-06T16:40:51.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:51.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:51.047538+0000) 2024-03-06T16:40:52.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:52.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:52.047795+0000) 2024-03-06T16:40:53.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:53.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:53.047995+0000) 2024-03-06T16:40:54.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:54.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:54.048238+0000) 2024-03-06T16:40:55.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:55.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:55.048498+0000) 2024-03-06T16:40:56.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:56.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:56.048784+0000) 2024-03-06T16:40:57.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:57.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:57.049101+0000) 2024-03-06T16:40:58.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:58.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:58.049406+0000) 2024-03-06T16:40:59.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:40:59.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:59.049743+0000) 2024-03-06T16:41:00.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:00.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:00.049999+0000) 2024-03-06T16:41:01.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:01.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:01.050250+0000) 2024-03-06T16:41:02.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:02.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:02.050501+0000) 2024-03-06T16:41:03.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:03.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:03.050775+0000) 2024-03-06T16:41:04.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:04.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:04.051026+0000) 2024-03-06T16:41:05.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:05.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:05.051262+0000) 2024-03-06T16:41:05.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:05.546 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:06.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:06.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:06.051518+0000) 2024-03-06T16:41:07.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:07.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:07.051811+0000) 2024-03-06T16:41:08.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:08.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:08.052082+0000) 2024-03-06T16:41:09.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:09.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:09.052360+0000) 2024-03-06T16:41:10.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:10.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:10.052609+0000) 2024-03-06T16:41:11.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:11.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:11.052935+0000) 2024-03-06T16:41:12.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:12.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:12.053185+0000) 2024-03-06T16:41:13.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:13.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:13.053462+0000) 2024-03-06T16:41:14.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:14.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:14.053727+0000) 2024-03-06T16:41:15.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:15.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:15.053992+0000) 2024-03-06T16:41:16.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:16.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:16.054241+0000) 2024-03-06T16:41:17.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:17.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:17.054503+0000) 2024-03-06T16:41:18.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:18.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:18.054791+0000) 2024-03-06T16:41:19.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:19.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:19.055040+0000) 2024-03-06T16:41:20.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:20.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:20.055271+0000) 2024-03-06T16:41:21.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:21.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:21.055465+0000) 2024-03-06T16:41:22.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:22.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:22.055749+0000) 2024-03-06T16:41:23.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:23.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:23.056029+0000) 2024-03-06T16:41:24.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:24.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:24.056289+0000) 2024-03-06T16:41:25.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:25.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:25.056555+0000) 2024-03-06T16:41:26.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:26.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:26.056793+0000) 2024-03-06T16:41:27.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:27.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:27.057030+0000) 2024-03-06T16:41:28.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:28.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:28.057270+0000) 2024-03-06T16:41:29.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:29.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:29.057526+0000) 2024-03-06T16:41:30.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:30.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:30.057785+0000) 2024-03-06T16:41:31.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:31.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:31.058043+0000) 2024-03-06T16:41:32.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:32.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:32.058300+0000) 2024-03-06T16:41:33.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:33.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:33.058557+0000) 2024-03-06T16:41:34.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:34.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:34.058813+0000) 2024-03-06T16:41:35.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:35.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:35.059080+0000) 2024-03-06T16:41:35.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:35.770 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:36.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:36.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:36.059363+0000) 2024-03-06T16:41:37.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:37.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:37.059740+0000) 2024-03-06T16:41:38.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:38.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:38.060045+0000) 2024-03-06T16:41:39.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:39.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:39.060366+0000) 2024-03-06T16:41:40.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:40.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:40.060669+0000) 2024-03-06T16:41:41.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:41.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:41.060917+0000) 2024-03-06T16:41:42.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:42.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:42.061107+0000) 2024-03-06T16:41:43.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:43.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:43.061314+0000) 2024-03-06T16:41:44.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:44.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:44.061572+0000) 2024-03-06T16:41:45.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:45.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:45.061834+0000) 2024-03-06T16:41:46.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:46.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:46.062015+0000) 2024-03-06T16:41:47.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:47.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:47.062277+0000) 2024-03-06T16:41:48.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:48.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:48.062538+0000) 2024-03-06T16:41:49.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:49.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:49.062784+0000) 2024-03-06T16:41:50.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:50.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:50.063039+0000) 2024-03-06T16:41:51.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:51.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:51.063280+0000) 2024-03-06T16:41:52.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:52.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:52.063520+0000) 2024-03-06T16:41:53.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:53.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:53.063788+0000) 2024-03-06T16:41:54.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:54.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:54.064055+0000) 2024-03-06T16:41:55.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:55.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:55.064329+0000) 2024-03-06T16:41:56.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:56.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:56.064568+0000) 2024-03-06T16:41:57.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:57.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:57.064820+0000) 2024-03-06T16:41:58.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:58.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:58.065076+0000) 2024-03-06T16:41:59.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:41:59.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:59.065338+0000) 2024-03-06T16:42:00.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:00.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:00.065591+0000) 2024-03-06T16:42:01.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:01.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:01.065838+0000) 2024-03-06T16:42:02.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:02.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:02.066095+0000) 2024-03-06T16:42:03.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:03.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:03.066369+0000) 2024-03-06T16:42:04.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:04.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:04.066606+0000) 2024-03-06T16:42:05.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:05.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:05.066891+0000) 2024-03-06T16:42:05.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:05.920 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:06.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:06.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:06.067182+0000) 2024-03-06T16:42:07.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:07.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:07.067501+0000) 2024-03-06T16:42:08.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:08.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:08.067792+0000) 2024-03-06T16:42:09.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:09.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:09.068087+0000) 2024-03-06T16:42:10.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:10.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:10.068366+0000) 2024-03-06T16:42:11.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:11.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:11.068613+0000) 2024-03-06T16:42:12.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:12.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:12.068930+0000) 2024-03-06T16:42:13.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:13.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:13.069200+0000) 2024-03-06T16:42:14.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:14.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:14.069461+0000) 2024-03-06T16:42:15.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:15.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:15.069755+0000) 2024-03-06T16:42:16.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:16.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:16.070015+0000) 2024-03-06T16:42:17.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:17.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:17.070263+0000) 2024-03-06T16:42:18.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:18.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:18.070529+0000) 2024-03-06T16:42:19.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:19.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:19.070766+0000) 2024-03-06T16:42:20.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:20.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:20.070962+0000) 2024-03-06T16:42:21.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:21.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:21.071207+0000) 2024-03-06T16:42:22.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:22.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:22.071463+0000) 2024-03-06T16:42:23.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:23.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:23.071770+0000) 2024-03-06T16:42:24.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:24.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:24.071977+0000) 2024-03-06T16:42:25.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:25.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:25.072168+0000) 2024-03-06T16:42:26.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:26.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:26.072395+0000) 2024-03-06T16:42:27.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:27.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:27.072715+0000) 2024-03-06T16:42:28.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:28.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:28.072935+0000) 2024-03-06T16:42:29.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:29.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:29.073194+0000) 2024-03-06T16:42:30.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:30.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:30.073453+0000) 2024-03-06T16:42:31.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:31.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:31.073764+0000) 2024-03-06T16:42:32.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:32.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:32.074023+0000) 2024-03-06T16:42:33.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:33.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:33.074303+0000) 2024-03-06T16:42:34.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:34.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:34.074572+0000) 2024-03-06T16:42:35.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:35.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:35.074842+0000) 2024-03-06T16:42:36.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:36.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:36.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:36.075084+0000) 2024-03-06T16:42:36.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:37.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:37.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:37.075342+0000) 2024-03-06T16:42:38.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:38.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:38.075598+0000) 2024-03-06T16:42:39.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:39.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:39.075890+0000) 2024-03-06T16:42:40.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:40.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:40.076140+0000) 2024-03-06T16:42:41.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:41.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:41.076390+0000) 2024-03-06T16:42:42.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:42.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:42.076575+0000) 2024-03-06T16:42:43.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:43.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:43.076847+0000) 2024-03-06T16:42:44.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:44.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:44.077102+0000) 2024-03-06T16:42:45.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:45.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:45.077363+0000) 2024-03-06T16:42:46.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:46.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:46.077603+0000) 2024-03-06T16:42:47.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:47.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:47.077914+0000) 2024-03-06T16:42:48.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:48.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:48.078110+0000) 2024-03-06T16:42:49.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:49.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:49.078386+0000) 2024-03-06T16:42:50.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:50.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:50.078684+0000) 2024-03-06T16:42:51.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:51.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:51.078932+0000) 2024-03-06T16:42:52.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:52.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:52.079173+0000) 2024-03-06T16:42:53.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:53.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:53.079426+0000) 2024-03-06T16:42:54.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:54.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:54.079774+0000) 2024-03-06T16:42:55.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:55.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:55.080026+0000) 2024-03-06T16:42:56.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:56.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:56.080270+0000) 2024-03-06T16:42:57.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:57.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:57.080522+0000) 2024-03-06T16:42:58.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:58.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:58.080789+0000) 2024-03-06T16:42:59.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:42:59.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:59.080986+0000) 2024-03-06T16:43:00.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:00.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:00.081179+0000) 2024-03-06T16:43:01.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:01.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:01.081426+0000) 2024-03-06T16:43:02.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:02.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:02.081740+0000) 2024-03-06T16:43:03.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:03.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:03.082005+0000) 2024-03-06T16:43:04.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:04.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:04.082263+0000) 2024-03-06T16:43:05.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:05.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:05.082531+0000) 2024-03-06T16:43:06.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:06.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:06.082781+0000) 2024-03-06T16:43:06.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:06.227 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:07.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:07.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:07.083048+0000) 2024-03-06T16:43:08.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:08.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:08.083298+0000) 2024-03-06T16:43:09.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:09.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:09.083546+0000) 2024-03-06T16:43:10.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:10.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:10.083801+0000) 2024-03-06T16:43:11.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:11.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:11.084043+0000) 2024-03-06T16:43:12.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:12.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:12.084295+0000) 2024-03-06T16:43:13.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:13.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:13.084552+0000) 2024-03-06T16:43:14.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:14.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:14.084798+0000) 2024-03-06T16:43:15.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:15.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:15.085047+0000) 2024-03-06T16:43:16.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:16.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:16.085284+0000) 2024-03-06T16:43:17.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:17.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:17.085542+0000) 2024-03-06T16:43:18.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:18.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:18.085789+0000) 2024-03-06T16:43:19.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:19.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:19.086026+0000) 2024-03-06T16:43:20.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:20.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:20.086274+0000) 2024-03-06T16:43:21.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:21.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:21.086513+0000) 2024-03-06T16:43:22.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:22.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:22.086749+0000) 2024-03-06T16:43:23.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:23.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:23.086999+0000) 2024-03-06T16:43:24.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:24.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:24.087259+0000) 2024-03-06T16:43:25.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:25.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:25.087510+0000) 2024-03-06T16:43:26.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:26.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:26.087749+0000) 2024-03-06T16:43:27.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:27.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:27.088011+0000) 2024-03-06T16:43:28.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:28.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:28.088201+0000) 2024-03-06T16:43:29.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:29.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:29.088448+0000) 2024-03-06T16:43:30.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:30.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:30.088744+0000) 2024-03-06T16:43:31.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:31.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:31.089003+0000) 2024-03-06T16:43:32.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:32.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:32.089241+0000) 2024-03-06T16:43:33.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:33.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:33.089512+0000) 2024-03-06T16:43:34.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:34.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:34.089793+0000) 2024-03-06T16:43:35.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:35.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:35.090059+0000) 2024-03-06T16:43:36.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:36.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:36.090328+0000) 2024-03-06T16:43:36.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:36.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:37.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:37.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:37.090580+0000) 2024-03-06T16:43:38.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:38.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:38.090836+0000) 2024-03-06T16:43:39.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:39.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:39.091104+0000) 2024-03-06T16:43:40.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:40.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:40.091344+0000) 2024-03-06T16:43:41.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:41.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:41.091584+0000) 2024-03-06T16:43:42.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:42.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:42.091793+0000) 2024-03-06T16:43:43.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:43.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:43.092069+0000) 2024-03-06T16:43:44.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:44.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:44.092274+0000) 2024-03-06T16:43:45.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:45.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:45.092540+0000) 2024-03-06T16:43:46.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:46.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:46.092799+0000) 2024-03-06T16:43:47.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:47.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:47.093066+0000) 2024-03-06T16:43:48.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:48.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:48.093355+0000) 2024-03-06T16:43:49.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:49.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:49.093625+0000) 2024-03-06T16:43:50.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:50.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:50.093948+0000) 2024-03-06T16:43:51.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:51.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:51.094193+0000) 2024-03-06T16:43:52.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:52.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:52.094436+0000) 2024-03-06T16:43:53.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:53.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:53.094753+0000) 2024-03-06T16:43:54.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:54.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:54.095022+0000) 2024-03-06T16:43:55.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:55.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:55.095283+0000) 2024-03-06T16:43:56.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:56.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:56.095470+0000) 2024-03-06T16:43:57.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:57.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:57.095761+0000) 2024-03-06T16:43:58.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:58.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:58.096019+0000) 2024-03-06T16:43:59.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:43:59.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:59.096287+0000) 2024-03-06T16:44:00.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:00.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:00.096542+0000) 2024-03-06T16:44:01.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:01.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:01.096801+0000) 2024-03-06T16:44:02.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:02.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:02.097058+0000) 2024-03-06T16:44:03.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:03.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:03.097317+0000) 2024-03-06T16:44:04.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:04.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:04.097590+0000) 2024-03-06T16:44:05.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:05.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:05.097827+0000) 2024-03-06T16:44:06.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:06.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:06.098082+0000) 2024-03-06T16:44:06.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:06.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:07.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:07.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:07.098343+0000) 2024-03-06T16:44:08.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:08.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:08.098587+0000) 2024-03-06T16:44:09.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:09.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:09.098865+0000) 2024-03-06T16:44:10.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:10.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:10.099055+0000) 2024-03-06T16:44:11.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:11.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:11.099306+0000) 2024-03-06T16:44:12.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:12.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:12.099573+0000) 2024-03-06T16:44:13.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:13.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:13.099819+0000) 2024-03-06T16:44:14.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:14.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:14.100030+0000) 2024-03-06T16:44:15.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:15.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:15.100265+0000) 2024-03-06T16:44:16.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:16.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:16.100535+0000) 2024-03-06T16:44:17.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:17.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:17.100777+0000) 2024-03-06T16:44:18.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:18.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:18.101037+0000) 2024-03-06T16:44:19.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:19.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:19.101286+0000) 2024-03-06T16:44:20.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:20.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:20.101529+0000) 2024-03-06T16:44:21.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:21.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:21.101773+0000) 2024-03-06T16:44:22.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:22.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:22.102018+0000) 2024-03-06T16:44:23.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:23.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:23.102286+0000) 2024-03-06T16:44:24.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:24.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:24.102544+0000) 2024-03-06T16:44:25.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:25.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:25.102805+0000) 2024-03-06T16:44:26.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:26.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:26.103047+0000) 2024-03-06T16:44:27.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:27.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:27.103300+0000) 2024-03-06T16:44:28.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:28.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:28.103558+0000) 2024-03-06T16:44:29.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:29.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:29.103821+0000) 2024-03-06T16:44:30.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:30.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:30.104078+0000) 2024-03-06T16:44:31.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:31.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:31.104318+0000) 2024-03-06T16:44:32.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:32.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:32.104572+0000) 2024-03-06T16:44:33.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:33.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:33.104845+0000) 2024-03-06T16:44:34.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:34.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:34.105111+0000) 2024-03-06T16:44:35.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:35.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:35.105359+0000) 2024-03-06T16:44:36.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:36.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:36.105722+0000) 2024-03-06T16:44:36.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:36.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:37.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:37.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:37.105992+0000) 2024-03-06T16:44:38.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:38.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:38.106242+0000) 2024-03-06T16:44:39.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:39.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:39.106491+0000) 2024-03-06T16:44:40.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:40.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:40.106750+0000) 2024-03-06T16:44:41.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:41.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:41.106965+0000) 2024-03-06T16:44:42.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:42.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:42.107227+0000) 2024-03-06T16:44:43.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:43.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:43.107482+0000) 2024-03-06T16:44:44.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:44.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:44.107765+0000) 2024-03-06T16:44:45.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:45.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:45.108026+0000) 2024-03-06T16:44:46.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:46.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:46.108288+0000) 2024-03-06T16:44:47.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:47.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:47.108547+0000) 2024-03-06T16:44:48.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:48.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:48.108801+0000) 2024-03-06T16:44:49.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:49.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:49.109051+0000) 2024-03-06T16:44:50.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:50.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:50.109294+0000) 2024-03-06T16:44:51.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:51.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:51.109558+0000) 2024-03-06T16:44:52.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:52.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:52.109818+0000) 2024-03-06T16:44:53.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:53.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:53.110087+0000) 2024-03-06T16:44:54.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:54.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:54.110344+0000) 2024-03-06T16:44:55.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:55.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:55.110607+0000) 2024-03-06T16:44:56.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:56.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:56.110906+0000) 2024-03-06T16:44:57.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:57.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:57.111171+0000) 2024-03-06T16:44:58.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:58.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:58.111421+0000) 2024-03-06T16:44:59.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:44:59.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:59.111614+0000) 2024-03-06T16:45:00.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:00.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:00.111927+0000) 2024-03-06T16:45:01.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:01.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:01.112184+0000) 2024-03-06T16:45:02.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:02.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:02.112432+0000) 2024-03-06T16:45:03.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:03.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:03.112736+0000) 2024-03-06T16:45:04.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:04.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:04.112990+0000) 2024-03-06T16:45:05.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:05.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:05.113233+0000) 2024-03-06T16:45:06.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:06.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:06.113457+0000) 2024-03-06T16:45:06.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:06.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:07.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:07.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:07.113764+0000) 2024-03-06T16:45:08.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:08.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:08.113951+0000) 2024-03-06T16:45:09.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:09.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:09.114217+0000) 2024-03-06T16:45:10.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:10.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:10.114463+0000) 2024-03-06T16:45:11.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:11.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:11.114743+0000) 2024-03-06T16:45:12.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:12.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:12.114973+0000) 2024-03-06T16:45:13.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:13.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:13.115228+0000) 2024-03-06T16:45:14.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:14.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:14.115471+0000) 2024-03-06T16:45:15.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:15.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:15.115771+0000) 2024-03-06T16:45:16.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:16.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:16.116020+0000) 2024-03-06T16:45:17.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:17.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:17.116286+0000) 2024-03-06T16:45:18.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:18.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:18.116529+0000) 2024-03-06T16:45:19.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:19.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:19.116783+0000) 2024-03-06T16:45:20.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:20.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:20.116962+0000) 2024-03-06T16:45:21.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:21.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:21.117215+0000) 2024-03-06T16:45:22.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:22.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:22.117469+0000) 2024-03-06T16:45:23.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:23.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:23.117745+0000) 2024-03-06T16:45:24.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:24.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:24.118065+0000) 2024-03-06T16:45:25.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:25.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:25.118261+0000) 2024-03-06T16:45:26.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:26.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:26.118488+0000) 2024-03-06T16:45:27.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:27.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:27.118751+0000) 2024-03-06T16:45:28.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:28.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:28.118985+0000) 2024-03-06T16:45:29.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:29.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:29.119254+0000) 2024-03-06T16:45:30.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:30.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:30.119510+0000) 2024-03-06T16:45:31.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:31.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:31.119765+0000) 2024-03-06T16:45:32.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:32.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:32.120007+0000) 2024-03-06T16:45:33.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:33.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:33.120319+0000) 2024-03-06T16:45:34.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:34.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:34.120583+0000) 2024-03-06T16:45:35.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:35.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:35.120832+0000) 2024-03-06T16:45:36.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:36.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:36.121076+0000) 2024-03-06T16:45:36.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:36.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:37.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:37.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:37.121729+0000) 2024-03-06T16:45:38.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:38.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:38.122019+0000) 2024-03-06T16:45:39.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:39.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:39.122286+0000) 2024-03-06T16:45:40.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:40.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:40.122551+0000) 2024-03-06T16:45:41.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:41.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:41.122817+0000) 2024-03-06T16:45:42.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:42.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:42.123060+0000) 2024-03-06T16:45:43.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:43.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:43.123324+0000) 2024-03-06T16:45:44.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:44.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:44.123565+0000) 2024-03-06T16:45:45.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:45.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:45.123808+0000) 2024-03-06T16:45:46.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:46.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:46.124055+0000) 2024-03-06T16:45:47.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:47.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:47.124319+0000) 2024-03-06T16:45:48.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:48.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:48.124565+0000) 2024-03-06T16:45:49.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:49.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:49.124836+0000) 2024-03-06T16:45:50.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:50.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:50.125029+0000) 2024-03-06T16:45:51.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:51.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:51.125293+0000) 2024-03-06T16:45:52.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:52.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:52.125556+0000) 2024-03-06T16:45:53.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:53.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:53.125876+0000) 2024-03-06T16:45:54.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:54.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:54.126138+0000) 2024-03-06T16:45:55.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:55.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:55.126407+0000) 2024-03-06T16:45:56.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:56.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:56.126720+0000) 2024-03-06T16:45:57.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:57.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:57.126986+0000) 2024-03-06T16:45:58.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:58.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:58.127244+0000) 2024-03-06T16:45:59.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:45:59.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:59.127493+0000) 2024-03-06T16:46:00.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:00.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:00.127753+0000) 2024-03-06T16:46:01.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:01.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:01.128027+0000) 2024-03-06T16:46:02.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:02.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:02.128256+0000) 2024-03-06T16:46:03.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:03.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:03.128459+0000) 2024-03-06T16:46:04.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:04.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:04.128750+0000) 2024-03-06T16:46:05.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:05.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:05.129018+0000) 2024-03-06T16:46:06.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:06.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:06.129283+0000) 2024-03-06T16:46:07.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:07.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:07.129522+0000) 2024-03-06T16:46:07.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:07.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:08.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:08.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:08.129783+0000) 2024-03-06T16:46:09.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:09.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:09.130031+0000) 2024-03-06T16:46:10.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:10.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:10.130292+0000) 2024-03-06T16:46:11.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:11.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:11.130559+0000) 2024-03-06T16:46:12.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:12.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:12.130817+0000) 2024-03-06T16:46:13.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:13.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:13.131086+0000) 2024-03-06T16:46:14.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:14.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:14.131283+0000) 2024-03-06T16:46:15.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:15.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:15.131531+0000) 2024-03-06T16:46:16.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:16.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:16.131799+0000) 2024-03-06T16:46:17.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:17.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:17.132048+0000) 2024-03-06T16:46:18.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:18.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:18.132294+0000) 2024-03-06T16:46:19.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:19.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:19.132559+0000) 2024-03-06T16:46:20.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:20.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:20.132806+0000) 2024-03-06T16:46:21.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:21.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:21.133050+0000) 2024-03-06T16:46:22.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:22.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:22.133320+0000) 2024-03-06T16:46:23.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:23.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:23.133596+0000) 2024-03-06T16:46:24.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:24.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:24.133874+0000) 2024-03-06T16:46:25.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:25.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:25.134141+0000) 2024-03-06T16:46:26.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:26.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:26.134405+0000) 2024-03-06T16:46:27.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:27.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:27.134695+0000) 2024-03-06T16:46:28.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:28.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:28.134945+0000) 2024-03-06T16:46:29.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:29.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:29.135209+0000) 2024-03-06T16:46:30.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:30.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:30.135453+0000) 2024-03-06T16:46:31.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:31.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:31.135737+0000) 2024-03-06T16:46:32.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:32.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:32.135994+0000) 2024-03-06T16:46:33.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:33.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:33.136279+0000) 2024-03-06T16:46:34.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:34.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:34.136528+0000) 2024-03-06T16:46:35.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:35.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:35.136783+0000) 2024-03-06T16:46:36.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:36.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:36.137023+0000) 2024-03-06T16:46:37.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:37.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:37.137218+0000) 2024-03-06T16:46:37.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:37.297 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:38.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:38.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:38.137468+0000) 2024-03-06T16:46:39.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:39.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:39.137762+0000) 2024-03-06T16:46:40.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:40.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:40.138027+0000) 2024-03-06T16:46:41.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:41.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:41.138264+0000) 2024-03-06T16:46:42.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:42.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:42.138523+0000) 2024-03-06T16:46:43.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:43.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:43.138791+0000) 2024-03-06T16:46:44.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:44.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:44.138988+0000) 2024-03-06T16:46:45.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:45.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:45.139175+0000) 2024-03-06T16:46:46.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:46.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:46.139355+0000) 2024-03-06T16:46:47.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:47.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:47.139539+0000) 2024-03-06T16:46:48.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:48.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:48.139789+0000) 2024-03-06T16:46:49.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:49.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:49.140038+0000) 2024-03-06T16:46:50.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:50.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:50.140289+0000) 2024-03-06T16:46:51.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:51.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:51.140542+0000) 2024-03-06T16:46:52.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:52.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:52.140774+0000) 2024-03-06T16:46:53.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:53.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:53.141022+0000) 2024-03-06T16:46:54.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:54.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:54.141283+0000) 2024-03-06T16:46:55.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:55.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:55.141528+0000) 2024-03-06T16:46:56.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:56.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:56.141767+0000) 2024-03-06T16:46:57.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:57.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:57.142031+0000) 2024-03-06T16:46:58.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:58.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:58.142285+0000) 2024-03-06T16:46:59.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:46:59.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:59.142535+0000) 2024-03-06T16:47:00.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:00.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:00.142792+0000) 2024-03-06T16:47:01.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:01.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:01.143027+0000) 2024-03-06T16:47:02.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:02.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:02.143282+0000) 2024-03-06T16:47:03.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:03.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:03.143550+0000) 2024-03-06T16:47:04.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:04.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:04.143798+0000) 2024-03-06T16:47:05.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:05.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:05.144110+0000) 2024-03-06T16:47:06.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:06.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:06.144354+0000) 2024-03-06T16:47:07.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:07.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:07.144616+0000) 2024-03-06T16:47:07.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:07.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:08.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:08.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:08.144904+0000) 2024-03-06T16:47:09.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:09.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:09.145175+0000) 2024-03-06T16:47:10.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:10.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:10.145444+0000) 2024-03-06T16:47:11.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:11.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:11.145747+0000) 2024-03-06T16:47:12.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:12.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:12.146004+0000) 2024-03-06T16:47:13.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:13.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:13.146259+0000) 2024-03-06T16:47:14.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:14.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:14.146509+0000) 2024-03-06T16:47:15.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:15.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:15.146745+0000) 2024-03-06T16:47:16.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:16.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:16.146988+0000) 2024-03-06T16:47:17.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:17.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:17.147231+0000) 2024-03-06T16:47:18.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:18.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:18.147462+0000) 2024-03-06T16:47:19.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:19.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:19.147766+0000) 2024-03-06T16:47:20.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:20.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:20.147996+0000) 2024-03-06T16:47:21.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:21.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:21.148235+0000) 2024-03-06T16:47:22.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:22.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:22.148485+0000) 2024-03-06T16:47:23.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:23.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:23.148756+0000) 2024-03-06T16:47:24.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:24.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:24.149019+0000) 2024-03-06T16:47:25.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:25.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:25.149272+0000) 2024-03-06T16:47:26.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:26.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:26.149534+0000) 2024-03-06T16:47:27.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:27.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:27.149779+0000) 2024-03-06T16:47:28.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:28.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:28.150032+0000) 2024-03-06T16:47:29.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:29.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:29.150262+0000) 2024-03-06T16:47:30.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:30.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:30.150508+0000) 2024-03-06T16:47:31.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:31.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:31.150768+0000) 2024-03-06T16:47:32.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:32.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:32.151022+0000) 2024-03-06T16:47:33.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:33.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:33.151285+0000) 2024-03-06T16:47:34.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:34.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:34.151535+0000) 2024-03-06T16:47:35.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:35.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:35.151796+0000) 2024-03-06T16:47:36.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:36.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:36.151990+0000) 2024-03-06T16:47:37.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:37.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:37.152259+0000) 2024-03-06T16:47:37.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:37.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:38.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:38.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:38.152514+0000) 2024-03-06T16:47:39.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:39.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:39.152790+0000) 2024-03-06T16:47:40.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:40.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:40.153047+0000) 2024-03-06T16:47:41.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:41.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:41.153293+0000) 2024-03-06T16:47:42.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:42.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:42.153479+0000) 2024-03-06T16:47:43.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:43.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:43.153739+0000) 2024-03-06T16:47:44.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:44.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:44.153987+0000) 2024-03-06T16:47:45.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:45.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:45.154280+0000) 2024-03-06T16:47:46.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:46.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:46.154527+0000) 2024-03-06T16:47:47.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:47.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:47.154773+0000) 2024-03-06T16:47:48.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:48.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:48.155038+0000) 2024-03-06T16:47:49.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:49.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:49.155302+0000) 2024-03-06T16:47:50.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:50.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:50.155568+0000) 2024-03-06T16:47:51.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:51.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:51.155819+0000) 2024-03-06T16:47:52.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:52.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:52.156076+0000) 2024-03-06T16:47:53.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:53.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:53.156290+0000) 2024-03-06T16:47:54.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:54.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:54.156550+0000) 2024-03-06T16:47:55.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:55.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:55.156818+0000) 2024-03-06T16:47:56.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:56.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:56.157074+0000) 2024-03-06T16:47:57.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:57.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:57.157324+0000) 2024-03-06T16:47:58.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:58.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:58.157581+0000) 2024-03-06T16:47:59.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:47:59.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:59.157820+0000) 2024-03-06T16:48:00.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:00.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:00.158068+0000) 2024-03-06T16:48:01.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:01.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:01.158336+0000) 2024-03-06T16:48:02.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:02.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:02.158573+0000) 2024-03-06T16:48:03.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:03.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:03.158849+0000) 2024-03-06T16:48:04.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:04.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:04.159092+0000) 2024-03-06T16:48:05.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:05.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:05.159341+0000) 2024-03-06T16:48:06.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:06.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:06.159605+0000) 2024-03-06T16:48:07.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:07.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:07.159911+0000) 2024-03-06T16:48:07.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:07.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:08.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:08.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:08.160183+0000) 2024-03-06T16:48:09.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:09.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:09.160433+0000) 2024-03-06T16:48:10.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:10.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:10.160738+0000) 2024-03-06T16:48:11.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:11.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:11.160993+0000) 2024-03-06T16:48:12.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:12.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:12.161231+0000) 2024-03-06T16:48:13.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:13.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:13.161431+0000) 2024-03-06T16:48:14.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:14.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:14.161731+0000) 2024-03-06T16:48:15.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:15.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:15.161994+0000) 2024-03-06T16:48:16.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:16.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:16.162256+0000) 2024-03-06T16:48:17.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:17.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:17.162523+0000) 2024-03-06T16:48:18.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:18.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:18.162787+0000) 2024-03-06T16:48:19.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:19.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:19.163027+0000) 2024-03-06T16:48:20.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:20.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:20.163298+0000) 2024-03-06T16:48:21.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:21.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:21.163571+0000) 2024-03-06T16:48:22.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:22.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:22.163827+0000) 2024-03-06T16:48:23.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:23.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:23.164039+0000) 2024-03-06T16:48:24.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:24.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:24.164279+0000) 2024-03-06T16:48:25.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:25.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:25.164532+0000) 2024-03-06T16:48:26.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:26.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:26.164814+0000) 2024-03-06T16:48:27.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:27.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:27.165010+0000) 2024-03-06T16:48:28.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:28.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:28.165253+0000) 2024-03-06T16:48:29.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:29.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:29.165505+0000) 2024-03-06T16:48:30.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:30.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:30.165786+0000) 2024-03-06T16:48:31.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:31.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:31.166048+0000) 2024-03-06T16:48:32.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:32.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:32.166334+0000) 2024-03-06T16:48:33.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:33.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:33.166589+0000) 2024-03-06T16:48:34.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:34.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:34.166834+0000) 2024-03-06T16:48:35.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:35.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:35.167097+0000) 2024-03-06T16:48:36.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:36.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:36.167364+0000) 2024-03-06T16:48:37.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:37.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:37.167622+0000) 2024-03-06T16:48:37.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:37.890 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:38.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:38.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:38.167911+0000) 2024-03-06T16:48:39.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:39.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:39.168174+0000) 2024-03-06T16:48:40.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:40.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:40.168423+0000) 2024-03-06T16:48:41.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:41.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:41.168717+0000) 2024-03-06T16:48:42.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:42.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:42.168914+0000) 2024-03-06T16:48:43.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:43.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:43.169181+0000) 2024-03-06T16:48:44.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:44.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:44.169444+0000) 2024-03-06T16:48:45.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:45.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:45.169754+0000) 2024-03-06T16:48:46.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:46.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:46.170071+0000) 2024-03-06T16:48:47.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:47.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:47.170322+0000) 2024-03-06T16:48:48.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:48.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:48.170584+0000) 2024-03-06T16:48:49.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:49.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:49.170824+0000) 2024-03-06T16:48:50.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:50.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:50.171022+0000) 2024-03-06T16:48:51.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:51.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:51.171273+0000) 2024-03-06T16:48:52.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:52.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:52.171535+0000) 2024-03-06T16:48:53.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:53.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:53.171759+0000) 2024-03-06T16:48:54.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:54.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:54.172016+0000) 2024-03-06T16:48:55.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:55.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:55.172277+0000) 2024-03-06T16:48:56.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:56.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:56.172501+0000) 2024-03-06T16:48:57.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:57.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:57.172770+0000) 2024-03-06T16:48:58.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:58.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:58.173024+0000) 2024-03-06T16:48:59.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:48:59.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:59.173223+0000) 2024-03-06T16:49:00.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:00.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:00.173579+0000) 2024-03-06T16:49:01.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:01.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:01.173821+0000) 2024-03-06T16:49:02.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:02.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:02.174011+0000) 2024-03-06T16:49:03.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:03.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:03.174244+0000) 2024-03-06T16:49:04.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:04.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:04.174491+0000) 2024-03-06T16:49:05.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:05.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:05.174745+0000) 2024-03-06T16:49:06.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:06.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:06.174992+0000) 2024-03-06T16:49:07.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:07.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:07.175257+0000) 2024-03-06T16:49:08.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:08.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:08.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:08.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:08.175509+0000) 2024-03-06T16:49:09.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:09.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:09.175788+0000) 2024-03-06T16:49:10.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:10.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:10.175986+0000) 2024-03-06T16:49:11.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:11.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:11.176246+0000) 2024-03-06T16:49:12.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:12.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:12.176499+0000) 2024-03-06T16:49:13.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:13.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:13.176772+0000) 2024-03-06T16:49:14.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:14.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:14.177011+0000) 2024-03-06T16:49:15.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:15.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:15.177282+0000) 2024-03-06T16:49:16.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:16.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:16.177540+0000) 2024-03-06T16:49:17.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:17.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:17.177801+0000) 2024-03-06T16:49:18.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:18.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:18.178046+0000) 2024-03-06T16:49:19.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:19.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:19.178308+0000) 2024-03-06T16:49:20.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:20.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:20.178567+0000) 2024-03-06T16:49:21.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:21.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:21.178836+0000) 2024-03-06T16:49:22.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:22.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:22.179091+0000) 2024-03-06T16:49:23.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:23.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:23.179362+0000) 2024-03-06T16:49:24.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:24.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:24.179623+0000) 2024-03-06T16:49:25.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:25.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:25.179932+0000) 2024-03-06T16:49:26.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:26.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:26.180183+0000) 2024-03-06T16:49:27.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:27.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:27.180449+0000) 2024-03-06T16:49:28.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:28.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:28.180746+0000) 2024-03-06T16:49:29.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:29.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:29.181010+0000) 2024-03-06T16:49:30.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:30.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:30.181278+0000) 2024-03-06T16:49:31.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:31.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:31.181542+0000) 2024-03-06T16:49:32.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:32.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:32.181799+0000) 2024-03-06T16:49:33.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:33.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:33.182071+0000) 2024-03-06T16:49:34.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:34.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:34.182380+0000) 2024-03-06T16:49:35.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:35.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:35.182690+0000) 2024-03-06T16:49:36.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:36.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:36.182910+0000) 2024-03-06T16:49:37.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:37.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:37.183177+0000) 2024-03-06T16:49:38.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:38.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:38.183424+0000) 2024-03-06T16:49:38.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:38.216 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:39.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:39.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:39.183738+0000) 2024-03-06T16:49:40.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:40.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:40.184006+0000) 2024-03-06T16:49:41.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:41.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:41.184271+0000) 2024-03-06T16:49:42.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:42.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:42.184509+0000) 2024-03-06T16:49:43.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:43.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:43.184785+0000) 2024-03-06T16:49:44.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:44.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:44.185037+0000) 2024-03-06T16:49:45.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:45.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:45.185282+0000) 2024-03-06T16:49:46.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:46.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:46.185548+0000) 2024-03-06T16:49:47.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:47.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:47.185801+0000) 2024-03-06T16:49:48.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:48.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:48.186059+0000) 2024-03-06T16:49:49.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:49.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:49.186317+0000) 2024-03-06T16:49:50.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:50.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:50.186556+0000) 2024-03-06T16:49:51.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:51.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:51.186822+0000) 2024-03-06T16:49:52.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:52.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:52.187079+0000) 2024-03-06T16:49:53.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:53.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:53.187350+0000) 2024-03-06T16:49:54.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:54.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:54.187603+0000) 2024-03-06T16:49:55.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:55.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:55.187907+0000) 2024-03-06T16:49:56.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:56.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:56.188167+0000) 2024-03-06T16:49:57.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:57.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:57.188426+0000) 2024-03-06T16:49:58.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:58.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:58.188743+0000) 2024-03-06T16:49:59.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:49:59.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:59.188944+0000) 2024-03-06T16:50:00.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:00.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:00.189203+0000) 2024-03-06T16:50:01.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:01.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:01.189475+0000) 2024-03-06T16:50:02.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:02.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:02.189733+0000) 2024-03-06T16:50:03.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:03.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:03.190006+0000) 2024-03-06T16:50:04.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:04.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:04.190262+0000) 2024-03-06T16:50:05.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:05.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:05.190537+0000) 2024-03-06T16:50:06.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:06.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:06.190760+0000) 2024-03-06T16:50:07.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:07.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:07.191028+0000) 2024-03-06T16:50:08.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:08.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:08.191213+0000) 2024-03-06T16:50:08.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:08.370 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:09.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:09.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:09.191474+0000) 2024-03-06T16:50:10.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:10.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:10.191760+0000) 2024-03-06T16:50:11.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:11.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:11.192024+0000) 2024-03-06T16:50:12.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:12.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:12.192282+0000) 2024-03-06T16:50:13.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:13.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:13.192549+0000) 2024-03-06T16:50:14.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:14.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:14.192806+0000) 2024-03-06T16:50:15.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:15.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:15.193051+0000) 2024-03-06T16:50:16.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:16.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:16.193299+0000) 2024-03-06T16:50:17.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:17.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:17.193559+0000) 2024-03-06T16:50:18.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:18.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:18.193817+0000) 2024-03-06T16:50:19.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:19.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:19.194078+0000) 2024-03-06T16:50:20.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:20.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:20.194340+0000) 2024-03-06T16:50:21.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:21.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:21.194587+0000) 2024-03-06T16:50:22.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:22.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:22.194815+0000) 2024-03-06T16:50:23.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:23.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:23.195085+0000) 2024-03-06T16:50:24.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:24.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:24.195320+0000) 2024-03-06T16:50:25.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:25.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:25.195588+0000) 2024-03-06T16:50:26.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:26.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:26.195846+0000) 2024-03-06T16:50:27.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:27.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:27.196053+0000) 2024-03-06T16:50:28.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:28.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:28.196290+0000) 2024-03-06T16:50:29.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:29.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:29.196541+0000) 2024-03-06T16:50:30.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:30.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:30.196780+0000) 2024-03-06T16:50:31.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:31.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:31.197040+0000) 2024-03-06T16:50:32.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:32.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:32.197301+0000) 2024-03-06T16:50:33.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:33.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:33.197566+0000) 2024-03-06T16:50:34.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:34.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:34.197804+0000) 2024-03-06T16:50:35.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:35.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:35.198054+0000) 2024-03-06T16:50:36.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:36.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:36.198245+0000) 2024-03-06T16:50:37.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:37.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:37.198478+0000) 2024-03-06T16:50:38.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:38.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:38.198738+0000) 2024-03-06T16:50:38.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:38.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:39.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:39.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:39.199002+0000) 2024-03-06T16:50:40.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:40.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:40.199260+0000) 2024-03-06T16:50:41.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:41.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:41.199476+0000) 2024-03-06T16:50:42.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:42.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:42.199738+0000) 2024-03-06T16:50:43.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:43.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:43.200025+0000) 2024-03-06T16:50:44.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:44.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:44.200259+0000) 2024-03-06T16:50:45.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:45.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:45.200506+0000) 2024-03-06T16:50:46.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:46.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:46.200752+0000) 2024-03-06T16:50:47.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:47.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:47.201010+0000) 2024-03-06T16:50:48.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:48.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:48.201264+0000) 2024-03-06T16:50:49.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:49.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:49.201528+0000) 2024-03-06T16:50:50.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:50.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:50.201797+0000) 2024-03-06T16:50:51.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:51.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:51.202060+0000) 2024-03-06T16:50:52.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:52.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:52.202321+0000) 2024-03-06T16:50:53.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:53.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:53.202592+0000) 2024-03-06T16:50:54.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:54.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:54.202861+0000) 2024-03-06T16:50:55.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:55.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:55.203126+0000) 2024-03-06T16:50:56.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:56.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:56.203383+0000) 2024-03-06T16:50:57.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:57.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:57.203698+0000) 2024-03-06T16:50:58.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:58.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:58.203946+0000) 2024-03-06T16:50:59.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:50:59.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:59.204210+0000) 2024-03-06T16:51:00.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:00.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:00.204479+0000) 2024-03-06T16:51:01.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:01.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:01.204766+0000) 2024-03-06T16:51:02.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:02.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:02.205025+0000) 2024-03-06T16:51:03.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:03.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:03.205277+0000) 2024-03-06T16:51:04.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:04.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:04.205563+0000) 2024-03-06T16:51:05.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:05.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:05.205865+0000) 2024-03-06T16:51:06.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:06.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:06.206174+0000) 2024-03-06T16:51:07.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:07.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:07.206470+0000) 2024-03-06T16:51:08.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:08.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:08.206796+0000) 2024-03-06T16:51:08.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:08.652 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:09.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:09.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:09.207108+0000) 2024-03-06T16:51:10.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:10.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:10.207425+0000) 2024-03-06T16:51:11.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:11.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:11.207726+0000) 2024-03-06T16:51:12.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:12.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:12.208019+0000) 2024-03-06T16:51:13.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:13.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:13.208330+0000) 2024-03-06T16:51:14.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:14.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:14.208682+0000) 2024-03-06T16:51:15.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:15.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:15.208986+0000) 2024-03-06T16:51:16.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:16.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:16.209335+0000) 2024-03-06T16:51:17.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:17.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:17.209687+0000) 2024-03-06T16:51:18.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:18.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:18.210001+0000) 2024-03-06T16:51:19.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:19.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:19.210319+0000) 2024-03-06T16:51:20.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:20.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:20.210667+0000) 2024-03-06T16:51:21.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:21.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:21.210975+0000) 2024-03-06T16:51:22.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:22.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:22.211285+0000) 2024-03-06T16:51:23.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:23.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:23.211619+0000) 2024-03-06T16:51:24.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:24.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:24.212004+0000) 2024-03-06T16:51:25.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:25.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:25.212249+0000) 2024-03-06T16:51:26.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:26.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:26.212536+0000) 2024-03-06T16:51:27.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:27.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:27.212837+0000) 2024-03-06T16:51:28.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:28.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:28.213135+0000) 2024-03-06T16:51:29.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:29.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:29.213422+0000) 2024-03-06T16:51:30.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:30.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:30.213779+0000) 2024-03-06T16:51:31.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:31.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:31.214090+0000) 2024-03-06T16:51:32.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:32.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:32.214391+0000) 2024-03-06T16:51:33.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:33.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:33.214760+0000) 2024-03-06T16:51:34.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:34.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:34.215048+0000) 2024-03-06T16:51:35.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:35.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:35.215360+0000) 2024-03-06T16:51:36.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:36.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:36.215732+0000) 2024-03-06T16:51:37.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:37.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:37.216047+0000) 2024-03-06T16:51:38.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:38.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:38.216369+0000) 2024-03-06T16:51:38.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:38.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:39.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:39.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:39.216742+0000) 2024-03-06T16:51:40.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:40.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:40.217004+0000) 2024-03-06T16:51:41.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:41.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:41.217272+0000) 2024-03-06T16:51:42.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:42.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:42.217536+0000) 2024-03-06T16:51:43.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:43.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:43.217815+0000) 2024-03-06T16:51:44.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:44.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:44.218077+0000) 2024-03-06T16:51:45.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:45.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:45.218344+0000) 2024-03-06T16:51:46.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:46.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:46.218589+0000) 2024-03-06T16:51:47.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:47.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:47.218851+0000) 2024-03-06T16:51:48.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:48.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:48.219085+0000) 2024-03-06T16:51:49.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:49.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:49.219329+0000) 2024-03-06T16:51:50.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:50.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:50.219594+0000) 2024-03-06T16:51:51.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:51.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:51.219862+0000) 2024-03-06T16:51:52.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:52.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:52.220119+0000) 2024-03-06T16:51:53.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:53.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:53.220380+0000) 2024-03-06T16:51:54.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:54.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:54.220617+0000) 2024-03-06T16:51:55.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:55.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:55.220892+0000) 2024-03-06T16:51:56.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:56.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:56.221152+0000) 2024-03-06T16:51:57.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:57.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:57.221403+0000) 2024-03-06T16:51:58.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:58.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:58.221720+0000) 2024-03-06T16:51:59.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:51:59.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:59.221975+0000) 2024-03-06T16:52:00.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:00.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:00.222230+0000) 2024-03-06T16:52:01.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:01.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:01.222491+0000) 2024-03-06T16:52:02.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:02.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:02.222741+0000) 2024-03-06T16:52:03.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:03.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:03.222957+0000) 2024-03-06T16:52:04.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:04.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:04.223268+0000) 2024-03-06T16:52:05.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:05.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:05.223582+0000) 2024-03-06T16:52:06.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:06.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:06.223926+0000) 2024-03-06T16:52:07.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:07.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:07.224181+0000) 2024-03-06T16:52:08.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:08.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:08.224496+0000) 2024-03-06T16:52:08.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:08.961 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:09.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:09.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:09.224863+0000) 2024-03-06T16:52:10.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:10.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:10.225171+0000) 2024-03-06T16:52:11.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:11.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:11.225488+0000) 2024-03-06T16:52:12.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:12.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:12.225800+0000) 2024-03-06T16:52:13.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:13.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:13.226136+0000) 2024-03-06T16:52:14.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:14.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:14.226436+0000) 2024-03-06T16:52:15.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:15.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:15.226788+0000) 2024-03-06T16:52:16.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:16.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:16.227103+0000) 2024-03-06T16:52:17.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:17.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:17.227411+0000) 2024-03-06T16:52:18.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:18.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:18.227764+0000) 2024-03-06T16:52:19.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:19.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:19.228067+0000) 2024-03-06T16:52:20.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:20.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:20.228366+0000) 2024-03-06T16:52:21.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:21.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:21.228717+0000) 2024-03-06T16:52:22.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:22.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:22.228988+0000) 2024-03-06T16:52:23.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:23.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:23.229185+0000) 2024-03-06T16:52:24.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:24.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:24.229440+0000) 2024-03-06T16:52:25.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:25.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:25.229733+0000) 2024-03-06T16:52:26.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:26.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:26.230012+0000) 2024-03-06T16:52:27.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:27.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:27.230277+0000) 2024-03-06T16:52:28.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:28.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:28.230535+0000) 2024-03-06T16:52:29.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:29.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:29.230796+0000) 2024-03-06T16:52:30.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:30.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:30.231086+0000) 2024-03-06T16:52:31.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:31.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:31.231381+0000) 2024-03-06T16:52:32.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:32.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:32.231732+0000) 2024-03-06T16:52:33.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:33.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:33.232055+0000) 2024-03-06T16:52:34.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:34.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:34.232363+0000) 2024-03-06T16:52:35.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:35.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:35.232731+0000) 2024-03-06T16:52:36.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:36.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:36.233030+0000) 2024-03-06T16:52:37.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:37.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:37.233293+0000) 2024-03-06T16:52:38.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:38.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:38.233601+0000) 2024-03-06T16:52:39.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:39.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:39.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:39.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:39.233876+0000) 2024-03-06T16:52:40.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:40.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:40.234180+0000) 2024-03-06T16:52:41.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:41.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:41.234481+0000) 2024-03-06T16:52:42.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:42.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:42.234791+0000) 2024-03-06T16:52:43.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:43.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:43.235126+0000) 2024-03-06T16:52:44.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:44.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:44.235428+0000) 2024-03-06T16:52:45.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:45.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:45.235718+0000) 2024-03-06T16:52:46.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:46.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:46.235972+0000) 2024-03-06T16:52:47.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:47.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:47.236291+0000) 2024-03-06T16:52:48.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:48.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:48.236604+0000) 2024-03-06T16:52:49.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:49.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:49.236963+0000) 2024-03-06T16:52:50.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:50.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:50.237291+0000) 2024-03-06T16:52:51.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:51.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:51.237610+0000) 2024-03-06T16:52:52.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:52.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:52.237951+0000) 2024-03-06T16:52:53.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:53.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:53.238285+0000) 2024-03-06T16:52:54.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:54.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:54.238603+0000) 2024-03-06T16:52:55.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:55.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:55.238947+0000) 2024-03-06T16:52:56.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:56.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:56.239201+0000) 2024-03-06T16:52:57.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:57.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:57.239491+0000) 2024-03-06T16:52:58.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:58.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:58.239804+0000) 2024-03-06T16:52:59.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:52:59.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:59.240103+0000) 2024-03-06T16:53:00.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:00.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:00.240415+0000) 2024-03-06T16:53:01.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:01.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:01.240755+0000) 2024-03-06T16:53:02.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:02.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:02.241055+0000) 2024-03-06T16:53:03.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:03.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:03.241365+0000) 2024-03-06T16:53:04.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:04.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:04.241732+0000) 2024-03-06T16:53:05.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:05.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:05.242035+0000) 2024-03-06T16:53:06.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:06.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:06.242354+0000) 2024-03-06T16:53:07.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:07.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:07.242733+0000) 2024-03-06T16:53:08.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:08.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:08.243026+0000) 2024-03-06T16:53:09.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:09.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:09.243330+0000) 2024-03-06T16:53:09.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:09.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:10.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:10.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:10.243681+0000) 2024-03-06T16:53:11.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:11.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:11.243947+0000) 2024-03-06T16:53:12.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:12.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:12.244181+0000) 2024-03-06T16:53:13.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:13.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:13.244492+0000) 2024-03-06T16:53:14.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:14.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:14.244803+0000) 2024-03-06T16:53:15.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:15.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:15.245103+0000) 2024-03-06T16:53:16.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:16.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:16.245418+0000) 2024-03-06T16:53:17.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:17.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:17.245753+0000) 2024-03-06T16:53:18.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:18.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:18.246061+0000) 2024-03-06T16:53:19.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:19.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:19.246381+0000) 2024-03-06T16:53:20.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:20.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:20.246723+0000) 2024-03-06T16:53:21.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:21.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:21.247026+0000) 2024-03-06T16:53:22.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:22.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:22.247345+0000) 2024-03-06T16:53:23.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:23.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:23.247619+0000) 2024-03-06T16:53:24.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:24.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:24.247928+0000) 2024-03-06T16:53:25.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:25.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:25.248214+0000) 2024-03-06T16:53:26.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:26.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:26.248544+0000) 2024-03-06T16:53:27.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:27.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:27.248836+0000) 2024-03-06T16:53:28.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:28.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:28.249133+0000) 2024-03-06T16:53:29.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:29.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:29.249432+0000) 2024-03-06T16:53:30.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:30.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:30.249784+0000) 2024-03-06T16:53:31.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:31.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:31.250107+0000) 2024-03-06T16:53:32.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:32.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:32.250404+0000) 2024-03-06T16:53:33.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:33.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:33.250751+0000) 2024-03-06T16:53:34.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:34.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:34.251049+0000) 2024-03-06T16:53:35.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:35.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:35.251364+0000) 2024-03-06T16:53:36.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:36.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:36.251732+0000) 2024-03-06T16:53:37.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:37.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:37.252033+0000) 2024-03-06T16:53:38.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:38.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:38.252338+0000) 2024-03-06T16:53:39.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:39.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:39.252705+0000) 2024-03-06T16:53:39.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:39.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:40.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:40.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:40.252984+0000) 2024-03-06T16:53:41.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:41.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:41.253254+0000) 2024-03-06T16:53:42.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:42.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:42.253521+0000) 2024-03-06T16:53:43.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:43.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:43.253800+0000) 2024-03-06T16:53:44.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:44.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:44.254039+0000) 2024-03-06T16:53:45.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:45.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:45.254333+0000) 2024-03-06T16:53:46.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:46.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:46.254595+0000) 2024-03-06T16:53:47.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:47.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:47.254816+0000) 2024-03-06T16:53:48.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:48.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:48.255085+0000) 2024-03-06T16:53:49.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:49.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:49.255276+0000) 2024-03-06T16:53:50.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:50.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:50.255537+0000) 2024-03-06T16:53:51.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:51.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:51.255809+0000) 2024-03-06T16:53:52.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:52.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:52.256053+0000) 2024-03-06T16:53:53.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:53.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:53.256314+0000) 2024-03-06T16:53:54.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:54.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:54.256574+0000) 2024-03-06T16:53:55.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:55.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:55.256846+0000) 2024-03-06T16:53:56.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:56.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:56.257141+0000) 2024-03-06T16:53:57.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:57.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:57.257426+0000) 2024-03-06T16:53:58.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:58.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:58.257703+0000) 2024-03-06T16:53:59.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:53:59.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:59.257983+0000) 2024-03-06T16:54:00.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:00.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:00.258239+0000) 2024-03-06T16:54:01.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:01.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:01.258522+0000) 2024-03-06T16:54:02.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:02.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:02.258794+0000) 2024-03-06T16:54:03.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:03.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:03.259112+0000) 2024-03-06T16:54:04.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:04.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:04.259427+0000) 2024-03-06T16:54:05.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:05.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:05.259788+0000) 2024-03-06T16:54:06.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:06.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:06.260076+0000) 2024-03-06T16:54:07.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:07.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:07.260337+0000) 2024-03-06T16:54:08.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:08.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:08.260686+0000) 2024-03-06T16:54:09.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:09.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:09.261012+0000) 2024-03-06T16:54:09.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:09.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:10.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:10.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:10.261337+0000) 2024-03-06T16:54:11.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:11.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:11.261717+0000) 2024-03-06T16:54:12.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:12.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:12.262026+0000) 2024-03-06T16:54:13.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:13.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:13.262354+0000) 2024-03-06T16:54:14.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:14.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:14.262724+0000) 2024-03-06T16:54:15.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:15.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:15.262983+0000) 2024-03-06T16:54:16.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:16.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:16.263307+0000) 2024-03-06T16:54:17.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:17.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:17.263610+0000) 2024-03-06T16:54:18.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:18.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:18.263920+0000) 2024-03-06T16:54:19.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:19.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:19.264155+0000) 2024-03-06T16:54:20.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:20.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:20.264480+0000) 2024-03-06T16:54:21.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:21.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:21.264805+0000) 2024-03-06T16:54:22.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:22.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:22.265100+0000) 2024-03-06T16:54:23.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:23.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:23.265438+0000) 2024-03-06T16:54:24.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:24.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:24.265783+0000) 2024-03-06T16:54:25.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:25.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:25.266042+0000) 2024-03-06T16:54:26.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:26.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:26.266298+0000) 2024-03-06T16:54:27.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:27.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:27.266605+0000) 2024-03-06T16:54:28.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:28.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:28.266971+0000) 2024-03-06T16:54:29.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:29.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:29.267273+0000) 2024-03-06T16:54:30.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:30.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:30.267562+0000) 2024-03-06T16:54:31.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:31.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:31.267874+0000) 2024-03-06T16:54:32.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:32.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:32.268141+0000) 2024-03-06T16:54:33.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:33.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:33.268464+0000) 2024-03-06T16:54:34.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:34.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:34.268763+0000) 2024-03-06T16:54:35.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:35.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:35.269009+0000) 2024-03-06T16:54:36.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:36.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:36.269272+0000) 2024-03-06T16:54:37.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:37.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:37.269538+0000) 2024-03-06T16:54:38.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:38.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:38.269788+0000) 2024-03-06T16:54:39.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:39.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:39.270052+0000) 2024-03-06T16:54:39.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:39.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:40.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:40.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:40.270306+0000) 2024-03-06T16:54:41.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:41.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:41.270570+0000) 2024-03-06T16:54:42.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:42.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:42.270830+0000) 2024-03-06T16:54:43.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:43.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:43.271092+0000) 2024-03-06T16:54:44.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:44.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:44.271332+0000) 2024-03-06T16:54:45.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:45.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:45.271589+0000) 2024-03-06T16:54:46.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:46.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:46.271850+0000) 2024-03-06T16:54:47.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:47.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:47.272116+0000) 2024-03-06T16:54:48.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:48.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:48.272383+0000) 2024-03-06T16:54:49.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:49.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:49.272691+0000) 2024-03-06T16:54:50.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:50.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:50.272934+0000) 2024-03-06T16:54:51.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:51.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:51.273186+0000) 2024-03-06T16:54:52.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:52.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:52.273443+0000) 2024-03-06T16:54:53.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:53.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:53.273761+0000) 2024-03-06T16:54:54.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:54.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:54.274006+0000) 2024-03-06T16:54:55.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:55.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:55.274274+0000) 2024-03-06T16:54:56.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:56.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:56.274517+0000) 2024-03-06T16:54:57.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:57.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:57.274781+0000) 2024-03-06T16:54:58.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:58.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:58.275040+0000) 2024-03-06T16:54:59.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:54:59.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:59.275308+0000) 2024-03-06T16:55:00.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:00.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:00.275573+0000) 2024-03-06T16:55:01.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:01.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:01.275839+0000) 2024-03-06T16:55:02.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:02.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:02.276109+0000) 2024-03-06T16:55:03.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:03.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:03.276384+0000) 2024-03-06T16:55:04.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:04.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:04.276692+0000) 2024-03-06T16:55:05.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:05.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:05.276969+0000) 2024-03-06T16:55:06.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:06.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:06.277225+0000) 2024-03-06T16:55:07.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:07.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:07.277479+0000) 2024-03-06T16:55:08.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:08.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:08.277769+0000) 2024-03-06T16:55:09.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:09.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:09.277961+0000) 2024-03-06T16:55:09.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:09.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:10.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:10.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:10.278229+0000) 2024-03-06T16:55:11.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:11.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:11.278479+0000) 2024-03-06T16:55:12.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:12.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:12.278741+0000) 2024-03-06T16:55:13.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:13.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:13.279032+0000) 2024-03-06T16:55:14.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:14.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:14.279306+0000) 2024-03-06T16:55:15.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:15.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:15.279552+0000) 2024-03-06T16:55:16.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:16.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:16.279815+0000) 2024-03-06T16:55:17.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:17.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:17.280079+0000) 2024-03-06T16:55:18.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:18.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:18.280352+0000) 2024-03-06T16:55:19.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:19.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:19.280598+0000) 2024-03-06T16:55:20.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:20.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:20.280855+0000) 2024-03-06T16:55:21.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:21.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:21.281127+0000) 2024-03-06T16:55:22.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:22.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:22.281367+0000) 2024-03-06T16:55:23.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:23.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:23.281571+0000) 2024-03-06T16:55:24.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:24.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:24.281831+0000) 2024-03-06T16:55:25.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:25.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:25.282078+0000) 2024-03-06T16:55:26.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:26.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:26.282345+0000) 2024-03-06T16:55:27.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:27.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:27.282611+0000) 2024-03-06T16:55:28.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:28.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:28.282930+0000) 2024-03-06T16:55:29.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:29.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:29.283189+0000) 2024-03-06T16:55:30.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:30.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:30.283427+0000) 2024-03-06T16:55:31.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:31.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:31.283748+0000) 2024-03-06T16:55:32.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:32.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:32.284021+0000) 2024-03-06T16:55:33.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:33.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:33.284295+0000) 2024-03-06T16:55:34.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:34.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:34.284533+0000) 2024-03-06T16:55:35.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:35.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:35.284806+0000) 2024-03-06T16:55:36.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:36.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:36.285041+0000) 2024-03-06T16:55:37.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:37.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:37.285288+0000) 2024-03-06T16:55:38.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:38.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:38.285547+0000) 2024-03-06T16:55:39.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:39.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:39.285793+0000) 2024-03-06T16:55:40.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:40.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:40.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:40.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:40.286063+0000) 2024-03-06T16:55:41.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:41.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:41.286261+0000) 2024-03-06T16:55:42.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:42.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:42.286507+0000) 2024-03-06T16:55:43.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:43.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:43.286785+0000) 2024-03-06T16:55:44.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:44.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:44.287019+0000) 2024-03-06T16:55:45.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:45.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:45.287267+0000) 2024-03-06T16:55:46.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:46.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:46.287532+0000) 2024-03-06T16:55:47.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:47.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:47.287798+0000) 2024-03-06T16:55:48.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:48.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:48.288042+0000) 2024-03-06T16:55:49.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:49.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:49.288292+0000) 2024-03-06T16:55:50.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:50.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:50.288544+0000) 2024-03-06T16:55:51.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:51.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:51.288806+0000) 2024-03-06T16:55:52.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:52.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:52.289067+0000) 2024-03-06T16:55:53.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:53.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:53.289340+0000) 2024-03-06T16:55:54.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:54.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:54.289591+0000) 2024-03-06T16:55:55.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:55.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:55.289858+0000) 2024-03-06T16:55:56.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:56.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:56.290116+0000) 2024-03-06T16:55:57.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:57.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:57.290367+0000) 2024-03-06T16:55:58.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:58.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:58.290668+0000) 2024-03-06T16:55:59.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:55:59.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:59.290952+0000) 2024-03-06T16:56:00.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:00.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:00.291207+0000) 2024-03-06T16:56:01.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:01.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:01.291481+0000) 2024-03-06T16:56:02.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:02.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:02.291740+0000) 2024-03-06T16:56:03.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:03.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:03.292019+0000) 2024-03-06T16:56:04.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:04.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:04.292281+0000) 2024-03-06T16:56:05.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:05.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:05.292486+0000) 2024-03-06T16:56:06.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:06.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:06.292730+0000) 2024-03-06T16:56:07.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:07.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:07.292998+0000) 2024-03-06T16:56:08.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:08.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:08.293257+0000) 2024-03-06T16:56:09.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:09.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:09.293523+0000) 2024-03-06T16:56:10.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:10.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:10.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:10.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:10.293750+0000) 2024-03-06T16:56:11.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:11.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:11.294010+0000) 2024-03-06T16:56:12.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:12.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:12.294267+0000) 2024-03-06T16:56:13.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:13.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:13.294515+0000) 2024-03-06T16:56:14.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:14.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:14.294755+0000) 2024-03-06T16:56:15.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:15.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:15.295025+0000) 2024-03-06T16:56:16.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:16.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:16.295275+0000) 2024-03-06T16:56:17.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:17.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:17.295524+0000) 2024-03-06T16:56:18.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:18.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:18.295779+0000) 2024-03-06T16:56:19.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:19.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:19.296025+0000) 2024-03-06T16:56:20.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:20.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:20.296268+0000) 2024-03-06T16:56:21.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:21.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:21.296541+0000) 2024-03-06T16:56:22.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:22.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:22.296803+0000) 2024-03-06T16:56:23.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:23.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:23.297063+0000) 2024-03-06T16:56:24.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:24.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:24.297317+0000) 2024-03-06T16:56:25.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:25.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:25.297580+0000) 2024-03-06T16:56:26.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:26.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:26.297846+0000) 2024-03-06T16:56:27.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:27.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:27.298094+0000) 2024-03-06T16:56:28.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:28.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:28.298352+0000) 2024-03-06T16:56:29.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:29.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:29.298611+0000) 2024-03-06T16:56:30.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:30.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:30.298914+0000) 2024-03-06T16:56:31.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:31.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:31.299122+0000) 2024-03-06T16:56:32.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:32.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:32.299314+0000) 2024-03-06T16:56:33.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:33.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:33.299567+0000) 2024-03-06T16:56:34.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:34.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:34.299827+0000) 2024-03-06T16:56:35.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:35.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:35.300022+0000) 2024-03-06T16:56:36.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:36.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:36.300294+0000) 2024-03-06T16:56:37.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:37.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:37.300550+0000) 2024-03-06T16:56:38.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:38.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:38.300804+0000) 2024-03-06T16:56:39.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:39.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:39.301066+0000) 2024-03-06T16:56:40.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:40.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:40.301332+0000) 2024-03-06T16:56:40.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:40.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:41.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:41.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:41.301578+0000) 2024-03-06T16:56:42.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:42.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:42.301761+0000) 2024-03-06T16:56:43.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:43.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:43.302035+0000) 2024-03-06T16:56:44.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:44.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:44.302295+0000) 2024-03-06T16:56:45.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:45.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:45.302559+0000) 2024-03-06T16:56:46.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:46.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:46.302804+0000) 2024-03-06T16:56:47.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:47.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:47.303057+0000) 2024-03-06T16:56:48.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:48.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:48.303240+0000) 2024-03-06T16:56:49.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:49.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:49.303491+0000) 2024-03-06T16:56:50.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:50.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:50.303748+0000) 2024-03-06T16:56:51.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:51.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:51.304018+0000) 2024-03-06T16:56:52.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:52.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:52.304276+0000) 2024-03-06T16:56:53.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:53.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:53.304553+0000) 2024-03-06T16:56:54.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:54.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:54.304789+0000) 2024-03-06T16:56:55.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:55.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:55.305065+0000) 2024-03-06T16:56:56.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:56.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:56.305329+0000) 2024-03-06T16:56:57.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:57.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:57.305589+0000) 2024-03-06T16:56:58.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:58.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:58.305853+0000) 2024-03-06T16:56:59.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:56:59.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:59.306117+0000) 2024-03-06T16:57:00.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:00.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:00.306358+0000) 2024-03-06T16:57:01.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:01.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:01.306622+0000) 2024-03-06T16:57:02.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:02.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:02.306885+0000) 2024-03-06T16:57:03.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:03.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:03.307094+0000) 2024-03-06T16:57:04.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:04.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:04.307349+0000) 2024-03-06T16:57:05.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:05.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:05.307603+0000) 2024-03-06T16:57:06.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:06.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:06.307918+0000) 2024-03-06T16:57:07.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:07.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:07.308169+0000) 2024-03-06T16:57:08.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:08.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:08.308494+0000) 2024-03-06T16:57:09.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:09.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:09.308762+0000) 2024-03-06T16:57:10.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:10.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:10.309022+0000) 2024-03-06T16:57:10.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:10.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:11.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:11.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:11.309288+0000) 2024-03-06T16:57:12.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:12.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:12.309553+0000) 2024-03-06T16:57:13.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:13.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:13.309810+0000) 2024-03-06T16:57:14.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:14.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:14.310045+0000) 2024-03-06T16:57:15.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:15.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:15.310282+0000) 2024-03-06T16:57:16.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:16.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:16.310527+0000) 2024-03-06T16:57:17.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:17.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:17.310783+0000) 2024-03-06T16:57:18.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:18.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:18.311044+0000) 2024-03-06T16:57:19.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:19.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:19.311274+0000) 2024-03-06T16:57:20.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:20.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:20.311532+0000) 2024-03-06T16:57:21.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:21.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:21.311779+0000) 2024-03-06T16:57:22.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:22.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:22.311977+0000) 2024-03-06T16:57:23.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:23.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:23.312265+0000) 2024-03-06T16:57:24.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:24.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:24.312527+0000) 2024-03-06T16:57:25.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:25.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:25.312774+0000) 2024-03-06T16:57:26.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:26.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:26.313026+0000) 2024-03-06T16:57:27.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:27.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:27.313294+0000) 2024-03-06T16:57:28.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:28.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:28.313553+0000) 2024-03-06T16:57:29.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:29.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:29.313820+0000) 2024-03-06T16:57:30.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:30.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:30.314083+0000) 2024-03-06T16:57:31.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:31.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:31.314281+0000) 2024-03-06T16:57:32.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:32.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:32.314556+0000) 2024-03-06T16:57:33.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:33.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:33.314769+0000) 2024-03-06T16:57:34.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:34.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:34.315027+0000) 2024-03-06T16:57:35.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:35.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:35.315272+0000) 2024-03-06T16:57:36.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:36.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:36.315508+0000) 2024-03-06T16:57:37.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:37.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:37.315767+0000) 2024-03-06T16:57:38.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:38.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:38.316015+0000) 2024-03-06T16:57:39.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:39.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:39.316280+0000) 2024-03-06T16:57:40.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:40.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:40.316542+0000) 2024-03-06T16:57:40.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:40.663 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:41.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:41.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:41.316774+0000) 2024-03-06T16:57:42.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:42.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:42.317041+0000) 2024-03-06T16:57:43.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:43.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:43.317357+0000) 2024-03-06T16:57:44.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:44.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:44.317592+0000) 2024-03-06T16:57:45.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:45.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:45.317850+0000) 2024-03-06T16:57:46.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:46.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:46.318096+0000) 2024-03-06T16:57:47.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:47.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:47.318349+0000) 2024-03-06T16:57:48.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:48.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:48.318606+0000) 2024-03-06T16:57:49.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:49.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:49.318907+0000) 2024-03-06T16:57:50.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:50.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:50.319198+0000) 2024-03-06T16:57:51.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:51.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:51.319460+0000) 2024-03-06T16:57:52.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:52.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:52.319782+0000) 2024-03-06T16:57:53.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:53.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:53.320057+0000) 2024-03-06T16:57:54.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:54.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:54.320297+0000) 2024-03-06T16:57:55.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:55.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:55.320559+0000) 2024-03-06T16:57:56.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:56.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:56.320818+0000) 2024-03-06T16:57:57.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:57.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:57.321073+0000) 2024-03-06T16:57:58.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:58.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:58.321313+0000) 2024-03-06T16:57:59.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:57:59.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:59.321507+0000) 2024-03-06T16:58:00.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:00.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:00.321755+0000) 2024-03-06T16:58:01.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:01.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:01.322025+0000) 2024-03-06T16:58:02.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:02.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:02.322294+0000) 2024-03-06T16:58:03.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:03.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:03.322539+0000) 2024-03-06T16:58:04.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:04.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:04.322795+0000) 2024-03-06T16:58:05.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:05.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:05.323057+0000) 2024-03-06T16:58:06.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:06.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:06.323302+0000) 2024-03-06T16:58:07.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:07.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:07.323572+0000) 2024-03-06T16:58:08.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:08.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:08.323824+0000) 2024-03-06T16:58:09.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:09.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:09.324091+0000) 2024-03-06T16:58:10.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:10.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:10.324332+0000) 2024-03-06T16:58:10.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:10.809 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:11.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:11.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:11.324585+0000) 2024-03-06T16:58:12.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:12.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:12.324835+0000) 2024-03-06T16:58:13.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:13.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:13.325097+0000) 2024-03-06T16:58:14.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:14.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:14.325357+0000) 2024-03-06T16:58:15.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:15.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:15.325609+0000) 2024-03-06T16:58:16.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:16.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:16.325904+0000) 2024-03-06T16:58:17.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:17.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:17.326153+0000) 2024-03-06T16:58:18.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:18.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:18.326404+0000) 2024-03-06T16:58:19.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:19.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:19.326685+0000) 2024-03-06T16:58:20.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:20.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:20.326941+0000) 2024-03-06T16:58:21.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:21.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:21.327207+0000) 2024-03-06T16:58:22.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:22.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:22.327477+0000) 2024-03-06T16:58:23.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:23.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:23.327750+0000) 2024-03-06T16:58:24.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:24.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:24.327989+0000) 2024-03-06T16:58:25.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:25.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:25.328240+0000) 2024-03-06T16:58:26.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:26.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:26.328484+0000) 2024-03-06T16:58:27.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:27.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:27.328741+0000) 2024-03-06T16:58:28.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:28.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:28.329008+0000) 2024-03-06T16:58:29.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:29.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:29.329269+0000) 2024-03-06T16:58:30.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:30.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:30.329506+0000) 2024-03-06T16:58:31.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:31.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:31.329749+0000) 2024-03-06T16:58:32.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:32.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:32.330010+0000) 2024-03-06T16:58:33.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:33.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:33.330212+0000) 2024-03-06T16:58:34.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:34.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:34.330456+0000) 2024-03-06T16:58:35.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:35.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:35.330770+0000) 2024-03-06T16:58:36.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:36.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:36.331025+0000) 2024-03-06T16:58:37.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:37.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:37.331273+0000) 2024-03-06T16:58:38.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:38.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:38.331509+0000) 2024-03-06T16:58:39.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:39.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:39.331773+0000) 2024-03-06T16:58:40.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:40.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:40.332008+0000) 2024-03-06T16:58:40.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:40.958 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:41.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:41.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:41.332269+0000) 2024-03-06T16:58:42.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:42.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:42.332457+0000) 2024-03-06T16:58:43.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:43.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:43.332755+0000) 2024-03-06T16:58:44.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:44.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:44.333055+0000) 2024-03-06T16:58:45.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:45.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:45.333316+0000) 2024-03-06T16:58:46.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:46.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:46.333576+0000) 2024-03-06T16:58:47.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:47.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:47.333846+0000) 2024-03-06T16:58:48.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:48.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:48.334105+0000) 2024-03-06T16:58:49.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:49.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:49.334353+0000) 2024-03-06T16:58:50.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:50.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:50.334617+0000) 2024-03-06T16:58:51.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:51.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:51.334865+0000) 2024-03-06T16:58:52.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:52.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:52.335129+0000) 2024-03-06T16:58:53.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:53.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:53.335406+0000) 2024-03-06T16:58:54.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:54.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:54.335698+0000) 2024-03-06T16:58:55.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:55.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:55.335955+0000) 2024-03-06T16:58:56.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:56.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:56.336203+0000) 2024-03-06T16:58:57.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:57.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:57.336451+0000) 2024-03-06T16:58:58.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:58.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:58.336718+0000) 2024-03-06T16:58:59.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:58:59.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:59.336973+0000) 2024-03-06T16:59:00.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:00.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:00.337230+0000) 2024-03-06T16:59:01.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:01.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:01.337490+0000) 2024-03-06T16:59:02.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:02.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:02.337770+0000) 2024-03-06T16:59:03.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:03.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:03.338028+0000) 2024-03-06T16:59:04.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:04.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:04.338272+0000) 2024-03-06T16:59:05.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:05.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:05.338538+0000) 2024-03-06T16:59:06.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:06.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:06.338782+0000) 2024-03-06T16:59:07.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:07.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:07.339047+0000) 2024-03-06T16:59:08.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:08.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:08.339311+0000) 2024-03-06T16:59:09.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:09.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:09.339571+0000) 2024-03-06T16:59:10.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:10.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:10.339805+0000) 2024-03-06T16:59:11.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:11.109 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:11.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:11.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:11.340053+0000) 2024-03-06T16:59:12.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:12.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:12.340307+0000) 2024-03-06T16:59:13.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:13.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:13.340585+0000) 2024-03-06T16:59:14.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:14.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:14.340835+0000) 2024-03-06T16:59:15.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:15.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:15.341113+0000) 2024-03-06T16:59:16.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:16.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:16.341377+0000) 2024-03-06T16:59:17.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:17.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:17.341684+0000) 2024-03-06T16:59:18.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:18.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:18.341951+0000) 2024-03-06T16:59:19.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:19.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:19.342214+0000) 2024-03-06T16:59:20.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:20.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:20.342465+0000) 2024-03-06T16:59:21.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:21.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:21.342757+0000) 2024-03-06T16:59:22.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:22.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:22.343020+0000) 2024-03-06T16:59:23.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:23.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:23.343303+0000) 2024-03-06T16:59:24.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:24.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:24.343555+0000) 2024-03-06T16:59:25.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:25.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:25.343854+0000) 2024-03-06T16:59:26.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:26.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:26.344116+0000) 2024-03-06T16:59:27.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:27.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:27.344382+0000) 2024-03-06T16:59:28.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:28.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:28.344681+0000) 2024-03-06T16:59:29.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:29.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:29.344952+0000) 2024-03-06T16:59:30.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:30.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:30.345194+0000) 2024-03-06T16:59:31.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:31.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:31.345460+0000) 2024-03-06T16:59:32.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:32.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:32.345759+0000) 2024-03-06T16:59:33.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:33.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:33.346040+0000) 2024-03-06T16:59:34.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:34.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:34.346282+0000) 2024-03-06T16:59:35.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:35.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:35.346542+0000) 2024-03-06T16:59:36.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:36.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:36.346786+0000) 2024-03-06T16:59:37.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:37.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:37.347111+0000) 2024-03-06T16:59:38.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:38.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:38.347353+0000) 2024-03-06T16:59:39.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:39.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:39.347592+0000) 2024-03-06T16:59:40.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:40.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:40.347843+0000) 2024-03-06T16:59:41.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:41.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:41.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:41.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:41.348035+0000) 2024-03-06T16:59:42.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:42.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:42.348275+0000) 2024-03-06T16:59:43.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:43.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:43.348534+0000) 2024-03-06T16:59:44.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:44.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:44.348823+0000) 2024-03-06T16:59:45.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:45.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:45.349077+0000) 2024-03-06T16:59:46.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:46.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:46.349333+0000) 2024-03-06T16:59:47.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:47.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:47.349582+0000) 2024-03-06T16:59:48.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:48.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:48.349841+0000) 2024-03-06T16:59:49.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:49.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:49.350082+0000) 2024-03-06T16:59:50.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:50.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:50.350338+0000) 2024-03-06T16:59:51.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:51.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:51.350603+0000) 2024-03-06T16:59:52.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:52.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:52.350854+0000) 2024-03-06T16:59:53.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:53.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:53.351137+0000) 2024-03-06T16:59:54.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:54.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:54.351366+0000) 2024-03-06T16:59:55.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:55.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:55.351617+0000) 2024-03-06T16:59:56.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:56.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:56.351910+0000) 2024-03-06T16:59:57.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:57.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:57.352168+0000) 2024-03-06T16:59:58.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:58.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:58.352411+0000) 2024-03-06T16:59:59.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T16:59:59.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:59.352672+0000) 2024-03-06T17:00:00.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:00.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:00.352932+0000) 2024-03-06T17:00:01.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:01.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:01.353191+0000) 2024-03-06T17:00:02.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:02.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:02.353452+0000) 2024-03-06T17:00:03.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:03.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:03.353741+0000) 2024-03-06T17:00:04.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:04.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:04.353984+0000) 2024-03-06T17:00:05.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:05.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:05.354240+0000) 2024-03-06T17:00:06.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:06.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:06.354498+0000) 2024-03-06T17:00:07.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:07.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:07.354745+0000) 2024-03-06T17:00:08.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:08.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:08.354995+0000) 2024-03-06T17:00:09.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:09.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:09.355259+0000) 2024-03-06T17:00:10.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:10.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:10.355505+0000) 2024-03-06T17:00:11.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:11.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:11.355765+0000) 2024-03-06T17:00:11.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:11.407 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:12.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:12.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:12.356009+0000) 2024-03-06T17:00:13.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:13.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:13.356281+0000) 2024-03-06T17:00:14.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:14.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:14.356545+0000) 2024-03-06T17:00:15.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:15.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:15.356806+0000) 2024-03-06T17:00:16.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:16.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:16.357054+0000) 2024-03-06T17:00:17.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:17.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:17.357300+0000) 2024-03-06T17:00:18.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:18.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:18.357566+0000) 2024-03-06T17:00:19.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:19.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:19.357835+0000) 2024-03-06T17:00:20.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:20.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:20.358088+0000) 2024-03-06T17:00:21.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:21.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:21.358325+0000) 2024-03-06T17:00:22.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:22.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:22.358579+0000) 2024-03-06T17:00:23.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:23.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:23.358853+0000) 2024-03-06T17:00:24.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:24.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:24.359109+0000) 2024-03-06T17:00:25.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:25.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:25.359374+0000) 2024-03-06T17:00:26.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:26.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:26.359666+0000) 2024-03-06T17:00:27.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:27.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:27.359920+0000) 2024-03-06T17:00:28.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:28.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:28.360182+0000) 2024-03-06T17:00:29.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:29.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:29.360442+0000) 2024-03-06T17:00:30.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:30.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:30.360769+0000) 2024-03-06T17:00:31.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:31.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:31.361016+0000) 2024-03-06T17:00:32.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:32.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:32.361262+0000) 2024-03-06T17:00:33.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:33.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:33.361543+0000) 2024-03-06T17:00:34.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:34.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:34.361802+0000) 2024-03-06T17:00:35.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:35.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:35.362065+0000) 2024-03-06T17:00:36.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:36.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:36.362313+0000) 2024-03-06T17:00:37.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:37.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:37.362557+0000) 2024-03-06T17:00:38.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:38.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:38.362830+0000) 2024-03-06T17:00:39.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:39.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:39.363099+0000) 2024-03-06T17:00:40.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:40.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:40.363344+0000) 2024-03-06T17:00:41.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:41.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:41.363665+0000) 2024-03-06T17:00:41.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:41.566 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:42.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:42.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:42.363933+0000) 2024-03-06T17:00:43.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:43.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:43.364208+0000) 2024-03-06T17:00:44.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:44.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:44.364496+0000) 2024-03-06T17:00:45.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:45.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:45.364762+0000) 2024-03-06T17:00:46.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:46.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:46.365048+0000) 2024-03-06T17:00:47.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:47.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:47.365338+0000) 2024-03-06T17:00:48.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:48.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:48.365621+0000) 2024-03-06T17:00:49.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:49.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:49.365945+0000) 2024-03-06T17:00:50.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:50.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:50.366219+0000) 2024-03-06T17:00:51.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:51.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:51.366488+0000) 2024-03-06T17:00:52.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:52.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:52.366766+0000) 2024-03-06T17:00:53.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:53.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:53.367087+0000) 2024-03-06T17:00:54.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:54.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:54.367355+0000) 2024-03-06T17:00:55.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:55.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:55.367617+0000) 2024-03-06T17:00:56.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:56.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:56.367889+0000) 2024-03-06T17:00:57.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:57.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:57.368132+0000) 2024-03-06T17:00:58.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:58.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:58.368401+0000) 2024-03-06T17:00:59.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:00:59.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:59.368690+0000) 2024-03-06T17:01:00.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:00.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:00.368962+0000) 2024-03-06T17:01:01.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:01.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:01.369153+0000) 2024-03-06T17:01:02.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:02.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:02.369413+0000) 2024-03-06T17:01:03.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:03.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:03.369749+0000) 2024-03-06T17:01:04.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:04.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:04.370008+0000) 2024-03-06T17:01:05.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:05.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:05.370260+0000) 2024-03-06T17:01:06.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:06.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:06.370519+0000) 2024-03-06T17:01:07.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:07.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:07.370770+0000) 2024-03-06T17:01:08.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:08.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:08.371016+0000) 2024-03-06T17:01:09.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:09.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:09.371296+0000) 2024-03-06T17:01:10.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:10.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:10.371538+0000) 2024-03-06T17:01:11.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:11.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:11.371789+0000) 2024-03-06T17:01:11.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:11.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:12.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:12.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:12.372047+0000) 2024-03-06T17:01:13.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:13.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:13.372330+0000) 2024-03-06T17:01:14.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:14.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:14.372587+0000) 2024-03-06T17:01:15.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:15.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:15.372846+0000) 2024-03-06T17:01:16.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:16.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:16.373106+0000) 2024-03-06T17:01:17.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:17.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:17.373375+0000) 2024-03-06T17:01:18.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:18.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:18.373721+0000) 2024-03-06T17:01:19.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:19.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:19.373984+0000) 2024-03-06T17:01:20.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:20.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:20.374183+0000) 2024-03-06T17:01:21.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:21.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:21.374461+0000) 2024-03-06T17:01:22.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:22.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:22.374751+0000) 2024-03-06T17:01:23.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:23.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:23.375029+0000) 2024-03-06T17:01:24.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:24.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:24.375292+0000) 2024-03-06T17:01:25.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:25.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:25.375533+0000) 2024-03-06T17:01:26.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:26.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:26.375797+0000) 2024-03-06T17:01:27.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:27.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:27.376064+0000) 2024-03-06T17:01:28.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:28.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:28.376270+0000) 2024-03-06T17:01:29.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:29.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:29.376533+0000) 2024-03-06T17:01:30.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:30.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:30.376789+0000) 2024-03-06T17:01:31.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:31.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:31.376985+0000) 2024-03-06T17:01:32.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:32.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:32.377235+0000) 2024-03-06T17:01:33.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:33.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:33.377447+0000) 2024-03-06T17:01:34.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:34.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:34.377773+0000) 2024-03-06T17:01:35.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:35.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:35.378086+0000) 2024-03-06T17:01:36.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:36.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:36.378354+0000) 2024-03-06T17:01:37.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:37.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:37.378586+0000) 2024-03-06T17:01:38.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:38.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:38.378846+0000) 2024-03-06T17:01:39.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:39.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:39.379125+0000) 2024-03-06T17:01:40.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:40.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:40.379402+0000) 2024-03-06T17:01:41.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:41.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:41.379756+0000) 2024-03-06T17:01:41.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:41.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:42.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:42.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:42.380040+0000) 2024-03-06T17:01:43.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:43.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:43.380334+0000) 2024-03-06T17:01:44.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:44.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:44.380597+0000) 2024-03-06T17:01:45.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:45.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:45.380863+0000) 2024-03-06T17:01:46.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:46.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:46.381129+0000) 2024-03-06T17:01:47.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:47.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:47.381380+0000) 2024-03-06T17:01:48.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:48.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:48.381681+0000) 2024-03-06T17:01:49.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:49.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:49.381943+0000) 2024-03-06T17:01:50.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:50.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:50.382206+0000) 2024-03-06T17:01:51.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:51.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:51.382403+0000) 2024-03-06T17:01:52.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:52.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:52.382728+0000) 2024-03-06T17:01:53.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:53.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:53.382987+0000) 2024-03-06T17:01:54.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:54.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:54.383246+0000) 2024-03-06T17:01:55.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:55.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:55.383505+0000) 2024-03-06T17:01:56.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:56.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:56.383750+0000) 2024-03-06T17:01:57.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:57.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:57.384012+0000) 2024-03-06T17:01:58.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:58.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:58.384272+0000) 2024-03-06T17:01:59.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:01:59.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:59.384515+0000) 2024-03-06T17:02:00.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:00.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:00.384756+0000) 2024-03-06T17:02:01.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:01.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:01.385008+0000) 2024-03-06T17:02:02.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:02.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:02.385261+0000) 2024-03-06T17:02:03.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:03.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:03.385530+0000) 2024-03-06T17:02:04.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:04.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:04.385766+0000) 2024-03-06T17:02:05.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:05.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:05.386031+0000) 2024-03-06T17:02:06.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:06.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:06.386291+0000) 2024-03-06T17:02:07.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:07.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:07.386540+0000) 2024-03-06T17:02:08.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:08.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:08.386804+0000) 2024-03-06T17:02:09.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:09.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:09.387054+0000) 2024-03-06T17:02:10.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:10.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:10.387337+0000) 2024-03-06T17:02:11.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:11.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:11.387619+0000) 2024-03-06T17:02:11.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:11.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:12.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:12.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:12.387932+0000) 2024-03-06T17:02:13.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:13.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:13.388226+0000) 2024-03-06T17:02:14.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:14.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:14.388483+0000) 2024-03-06T17:02:15.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:15.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:15.388746+0000) 2024-03-06T17:02:16.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:16.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:16.389009+0000) 2024-03-06T17:02:17.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:17.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:17.389277+0000) 2024-03-06T17:02:18.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:18.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:18.389532+0000) 2024-03-06T17:02:19.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:19.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:19.389749+0000) 2024-03-06T17:02:20.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:20.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:20.390010+0000) 2024-03-06T17:02:21.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:21.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:21.390288+0000) 2024-03-06T17:02:22.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:22.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:22.390549+0000) 2024-03-06T17:02:23.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:23.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:23.390842+0000) 2024-03-06T17:02:24.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:24.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:24.391100+0000) 2024-03-06T17:02:25.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:25.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:25.391310+0000) 2024-03-06T17:02:26.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:26.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:26.391587+0000) 2024-03-06T17:02:27.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:27.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:27.391894+0000) 2024-03-06T17:02:28.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:28.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:28.392168+0000) 2024-03-06T17:02:29.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:29.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:29.392448+0000) 2024-03-06T17:02:30.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:30.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:30.392778+0000) 2024-03-06T17:02:31.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:31.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:31.392997+0000) 2024-03-06T17:02:32.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:32.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:32.393270+0000) 2024-03-06T17:02:33.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:33.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:33.393541+0000) 2024-03-06T17:02:34.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:34.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:34.393829+0000) 2024-03-06T17:02:35.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:35.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:35.394080+0000) 2024-03-06T17:02:36.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:36.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:36.394340+0000) 2024-03-06T17:02:37.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:37.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:37.394612+0000) 2024-03-06T17:02:38.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:38.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:38.394967+0000) 2024-03-06T17:02:39.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:39.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:39.395245+0000) 2024-03-06T17:02:40.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:40.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:40.395518+0000) 2024-03-06T17:02:41.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:41.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:41.395799+0000) 2024-03-06T17:02:42.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:42.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:42.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:42.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:42.396083+0000) 2024-03-06T17:02:43.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:43.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:43.396338+0000) 2024-03-06T17:02:44.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:44.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:44.396622+0000) 2024-03-06T17:02:45.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:45.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:45.396906+0000) 2024-03-06T17:02:46.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:46.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:46.397114+0000) 2024-03-06T17:02:47.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:47.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:47.397392+0000) 2024-03-06T17:02:48.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:48.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:48.397706+0000) 2024-03-06T17:02:49.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:49.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:49.397975+0000) 2024-03-06T17:02:50.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:50.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:50.398249+0000) 2024-03-06T17:02:51.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:51.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:51.398533+0000) 2024-03-06T17:02:52.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:52.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:52.398808+0000) 2024-03-06T17:02:53.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:53.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:53.399086+0000) 2024-03-06T17:02:54.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:54.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:54.399343+0000) 2024-03-06T17:02:55.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:55.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:55.399600+0000) 2024-03-06T17:02:56.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:56.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:56.399921+0000) 2024-03-06T17:02:57.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:57.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:57.400207+0000) 2024-03-06T17:02:58.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:58.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:58.400488+0000) 2024-03-06T17:02:59.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:02:59.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:59.400748+0000) 2024-03-06T17:03:00.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:00.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:00.401013+0000) 2024-03-06T17:03:01.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:01.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:01.401264+0000) 2024-03-06T17:03:02.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:02.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:02.401550+0000) 2024-03-06T17:03:03.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:03.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:03.401849+0000) 2024-03-06T17:03:04.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:04.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:04.402114+0000) 2024-03-06T17:03:05.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:05.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:05.402392+0000) 2024-03-06T17:03:06.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:06.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:06.402720+0000) 2024-03-06T17:03:07.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:07.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:07.402998+0000) 2024-03-06T17:03:08.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:08.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:08.403282+0000) 2024-03-06T17:03:09.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:09.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:09.403495+0000) 2024-03-06T17:03:10.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:10.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:10.403743+0000) 2024-03-06T17:03:11.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:11.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:11.404007+0000) 2024-03-06T17:03:12.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:12.281 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:12.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:12.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:12.404251+0000) 2024-03-06T17:03:13.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:13.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:13.404547+0000) 2024-03-06T17:03:14.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:14.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:14.404792+0000) 2024-03-06T17:03:15.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:15.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:15.404985+0000) 2024-03-06T17:03:16.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:16.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:16.405191+0000) 2024-03-06T17:03:17.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:17.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:17.405467+0000) 2024-03-06T17:03:18.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:18.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:18.405752+0000) 2024-03-06T17:03:19.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:19.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:19.406011+0000) 2024-03-06T17:03:20.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:20.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:20.406283+0000) 2024-03-06T17:03:21.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:21.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:21.406570+0000) 2024-03-06T17:03:22.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:22.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:22.406824+0000) 2024-03-06T17:03:23.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:23.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:23.407082+0000) 2024-03-06T17:03:24.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:24.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:24.407336+0000) 2024-03-06T17:03:25.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:25.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:25.407566+0000) 2024-03-06T17:03:26.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:26.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:26.407822+0000) 2024-03-06T17:03:27.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:27.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:27.408067+0000) 2024-03-06T17:03:28.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:28.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:28.408329+0000) 2024-03-06T17:03:29.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:29.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:29.408577+0000) 2024-03-06T17:03:30.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:30.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:30.408839+0000) 2024-03-06T17:03:31.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:31.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:31.409083+0000) 2024-03-06T17:03:32.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:32.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:32.409326+0000) 2024-03-06T17:03:33.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:33.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:33.409606+0000) 2024-03-06T17:03:34.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:34.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:34.409920+0000) 2024-03-06T17:03:35.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:35.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:35.410116+0000) 2024-03-06T17:03:36.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:36.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:36.410380+0000) 2024-03-06T17:03:37.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:37.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:37.410672+0000) 2024-03-06T17:03:38.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:38.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:38.410932+0000) 2024-03-06T17:03:39.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:39.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:39.411176+0000) 2024-03-06T17:03:40.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:40.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:40.411420+0000) 2024-03-06T17:03:41.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:41.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:41.411724+0000) 2024-03-06T17:03:42.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:42.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:42.411987+0000) 2024-03-06T17:03:42.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:42.422 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:43.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:43.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:43.412282+0000) 2024-03-06T17:03:44.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:44.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:44.412535+0000) 2024-03-06T17:03:45.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:45.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:45.412783+0000) 2024-03-06T17:03:46.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:46.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:46.413064+0000) 2024-03-06T17:03:47.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:47.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:47.413321+0000) 2024-03-06T17:03:48.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:48.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:48.413672+0000) 2024-03-06T17:03:49.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:49.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:49.413938+0000) 2024-03-06T17:03:50.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:50.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:50.414177+0000) 2024-03-06T17:03:51.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:51.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:51.414411+0000) 2024-03-06T17:03:52.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:52.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:52.414708+0000) 2024-03-06T17:03:53.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:53.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:53.414986+0000) 2024-03-06T17:03:54.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:54.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:54.415282+0000) 2024-03-06T17:03:55.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:55.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:55.415599+0000) 2024-03-06T17:03:56.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:56.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:56.415875+0000) 2024-03-06T17:03:57.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:57.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:57.416170+0000) 2024-03-06T17:03:58.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:58.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:58.416498+0000) 2024-03-06T17:03:59.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:03:59.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:59.416808+0000) 2024-03-06T17:04:00.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:00.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:00.417125+0000) 2024-03-06T17:04:01.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:01.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:01.417446+0000) 2024-03-06T17:04:02.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:02.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:02.417786+0000) 2024-03-06T17:04:03.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:03.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:03.418094+0000) 2024-03-06T17:04:04.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:04.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:04.418405+0000) 2024-03-06T17:04:05.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:05.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:05.418753+0000) 2024-03-06T17:04:06.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:06.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:06.419067+0000) 2024-03-06T17:04:07.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:07.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:07.419342+0000) 2024-03-06T17:04:08.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:08.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:08.419672+0000) 2024-03-06T17:04:09.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:09.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:09.419973+0000) 2024-03-06T17:04:10.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:10.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:10.420295+0000) 2024-03-06T17:04:11.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:11.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:11.420615+0000) 2024-03-06T17:04:12.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:12.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:12.420960+0000) 2024-03-06T17:04:12.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:12.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:13.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:13.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:13.421269+0000) 2024-03-06T17:04:14.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:14.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:14.421580+0000) 2024-03-06T17:04:15.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:15.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:15.421883+0000) 2024-03-06T17:04:16.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:16.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:16.422112+0000) 2024-03-06T17:04:17.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:17.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:17.422418+0000) 2024-03-06T17:04:18.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:18.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:18.422758+0000) 2024-03-06T17:04:19.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:19.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:19.423077+0000) 2024-03-06T17:04:20.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:20.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:20.423394+0000) 2024-03-06T17:04:21.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:21.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:21.423742+0000) 2024-03-06T17:04:22.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:22.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:22.424055+0000) 2024-03-06T17:04:23.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:23.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:23.424379+0000) 2024-03-06T17:04:24.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:24.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:24.424749+0000) 2024-03-06T17:04:25.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:25.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:25.425052+0000) 2024-03-06T17:04:26.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:26.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:26.425282+0000) 2024-03-06T17:04:27.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:27.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:27.425579+0000) 2024-03-06T17:04:28.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:28.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:28.425868+0000) 2024-03-06T17:04:29.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:29.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:29.426191+0000) 2024-03-06T17:04:30.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:30.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:30.426501+0000) 2024-03-06T17:04:31.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:31.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:31.426826+0000) 2024-03-06T17:04:32.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:32.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:32.427086+0000) 2024-03-06T17:04:33.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:33.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:33.427416+0000) 2024-03-06T17:04:34.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:34.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:34.427783+0000) 2024-03-06T17:04:35.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:35.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:35.428054+0000) 2024-03-06T17:04:36.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:36.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:36.428416+0000) 2024-03-06T17:04:37.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:37.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:37.428744+0000) 2024-03-06T17:04:38.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:38.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:38.429049+0000) 2024-03-06T17:04:39.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:39.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:39.429337+0000) 2024-03-06T17:04:40.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:40.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:40.429624+0000) 2024-03-06T17:04:41.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:41.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:41.429976+0000) 2024-03-06T17:04:42.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:42.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:42.430306+0000) 2024-03-06T17:04:42.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:42.741 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:43.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:43.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:43.430692+0000) 2024-03-06T17:04:44.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:44.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:44.431016+0000) 2024-03-06T17:04:45.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:45.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:45.431313+0000) 2024-03-06T17:04:46.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:46.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:46.431608+0000) 2024-03-06T17:04:47.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:47.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:47.431968+0000) 2024-03-06T17:04:48.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:48.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:48.432290+0000) 2024-03-06T17:04:49.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:49.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:49.432579+0000) 2024-03-06T17:04:50.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:50.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:50.432884+0000) 2024-03-06T17:04:51.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:51.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:51.433202+0000) 2024-03-06T17:04:52.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:52.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:52.433457+0000) 2024-03-06T17:04:53.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:53.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:53.433747+0000) 2024-03-06T17:04:54.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:54.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:54.433948+0000) 2024-03-06T17:04:55.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:55.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:55.434189+0000) 2024-03-06T17:04:56.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:56.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:56.434442+0000) 2024-03-06T17:04:57.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:57.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:57.434738+0000) 2024-03-06T17:04:58.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:58.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:58.435004+0000) 2024-03-06T17:04:59.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:04:59.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:59.435273+0000) 2024-03-06T17:05:00.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:00.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:00.435515+0000) 2024-03-06T17:05:01.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:01.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:01.435768+0000) 2024-03-06T17:05:02.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:02.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:02.436035+0000) 2024-03-06T17:05:03.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:03.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:03.436293+0000) 2024-03-06T17:05:04.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:04.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:04.436550+0000) 2024-03-06T17:05:05.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:05.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:05.436792+0000) 2024-03-06T17:05:06.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:06.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:06.437034+0000) 2024-03-06T17:05:07.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:07.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:07.437283+0000) 2024-03-06T17:05:08.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:08.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:08.437543+0000) 2024-03-06T17:05:09.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:09.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:09.437809+0000) 2024-03-06T17:05:10.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:10.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:10.438078+0000) 2024-03-06T17:05:11.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:11.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:11.438311+0000) 2024-03-06T17:05:12.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:12.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:12.438558+0000) 2024-03-06T17:05:12.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:12.882 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:13.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:13.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:13.438827+0000) 2024-03-06T17:05:14.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:14.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:14.439084+0000) 2024-03-06T17:05:15.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:15.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:15.439280+0000) 2024-03-06T17:05:16.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:16.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:16.439519+0000) 2024-03-06T17:05:17.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:17.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:17.439746+0000) 2024-03-06T17:05:18.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:18.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:18.439954+0000) 2024-03-06T17:05:19.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:19.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:19.440208+0000) 2024-03-06T17:05:20.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:20.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:20.440448+0000) 2024-03-06T17:05:21.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:21.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:21.440766+0000) 2024-03-06T17:05:22.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:22.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:22.440958+0000) 2024-03-06T17:05:23.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:23.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:23.441233+0000) 2024-03-06T17:05:24.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:24.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:24.441494+0000) 2024-03-06T17:05:25.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:25.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:25.441744+0000) 2024-03-06T17:05:26.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:26.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:26.441950+0000) 2024-03-06T17:05:27.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:27.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:27.442210+0000) 2024-03-06T17:05:28.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:28.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:28.442479+0000) 2024-03-06T17:05:29.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:29.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:29.442796+0000) 2024-03-06T17:05:30.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:30.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:30.443036+0000) 2024-03-06T17:05:31.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:31.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:31.443282+0000) 2024-03-06T17:05:32.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:32.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:32.443540+0000) 2024-03-06T17:05:33.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:33.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:33.443820+0000) 2024-03-06T17:05:34.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:34.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:34.444072+0000) 2024-03-06T17:05:35.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:35.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:35.444312+0000) 2024-03-06T17:05:36.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:36.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:36.444564+0000) 2024-03-06T17:05:37.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:37.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:37.444819+0000) 2024-03-06T17:05:38.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:38.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:38.445094+0000) 2024-03-06T17:05:39.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:39.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:39.445370+0000) 2024-03-06T17:05:40.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:40.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:40.445615+0000) 2024-03-06T17:05:41.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:41.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:41.445899+0000) 2024-03-06T17:05:42.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:42.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:42.446164+0000) 2024-03-06T17:05:43.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:43.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:43.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:43.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:43.446425+0000) 2024-03-06T17:05:44.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:44.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:44.446754+0000) 2024-03-06T17:05:45.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:45.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:45.447022+0000) 2024-03-06T17:05:46.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:46.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:46.447273+0000) 2024-03-06T17:05:47.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:47.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:47.447532+0000) 2024-03-06T17:05:48.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:48.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:48.447813+0000) 2024-03-06T17:05:49.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:49.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:49.448062+0000) 2024-03-06T17:05:50.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:50.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:50.448325+0000) 2024-03-06T17:05:51.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:51.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:51.448580+0000) 2024-03-06T17:05:52.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:52.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:52.448829+0000) 2024-03-06T17:05:53.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:53.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:53.449109+0000) 2024-03-06T17:05:54.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:54.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:54.449313+0000) 2024-03-06T17:05:55.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:55.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:55.449558+0000) 2024-03-06T17:05:56.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:56.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:56.449821+0000) 2024-03-06T17:05:57.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:57.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:57.450069+0000) 2024-03-06T17:05:58.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:58.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:58.450329+0000) 2024-03-06T17:05:59.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:05:59.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:59.450608+0000) 2024-03-06T17:06:00.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:00.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:00.450943+0000) 2024-03-06T17:06:01.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:01.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:01.451209+0000) 2024-03-06T17:06:02.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:02.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:02.451470+0000) 2024-03-06T17:06:03.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:03.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:03.451759+0000) 2024-03-06T17:06:04.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:04.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:04.452005+0000) 2024-03-06T17:06:05.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:05.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:05.452262+0000) 2024-03-06T17:06:06.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:06.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:06.452513+0000) 2024-03-06T17:06:07.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:07.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:07.452755+0000) 2024-03-06T17:06:08.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:08.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:08.453028+0000) 2024-03-06T17:06:09.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:09.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:09.453296+0000) 2024-03-06T17:06:10.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:10.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:10.453542+0000) 2024-03-06T17:06:11.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:11.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:11.453785+0000) 2024-03-06T17:06:12.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:12.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:12.454034+0000) 2024-03-06T17:06:13.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:13.182 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:13.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:13.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:13.454304+0000) 2024-03-06T17:06:14.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:14.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:14.454567+0000) 2024-03-06T17:06:15.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:15.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:15.454790+0000) 2024-03-06T17:06:16.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:16.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:16.454984+0000) 2024-03-06T17:06:17.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:17.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:17.455174+0000) 2024-03-06T17:06:18.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:18.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:18.455367+0000) 2024-03-06T17:06:19.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:19.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:19.455676+0000) 2024-03-06T17:06:20.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:20.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:20.455938+0000) 2024-03-06T17:06:21.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:21.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:21.456171+0000) 2024-03-06T17:06:22.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:22.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:22.456427+0000) 2024-03-06T17:06:23.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:23.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:23.456736+0000) 2024-03-06T17:06:24.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:24.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:24.456971+0000) 2024-03-06T17:06:25.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:25.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:25.457214+0000) 2024-03-06T17:06:26.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:26.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:26.457456+0000) 2024-03-06T17:06:27.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:27.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:27.457770+0000) 2024-03-06T17:06:28.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:28.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:28.457960+0000) 2024-03-06T17:06:29.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:29.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:29.458209+0000) 2024-03-06T17:06:30.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:30.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:30.458463+0000) 2024-03-06T17:06:31.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:31.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:31.458744+0000) 2024-03-06T17:06:32.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:32.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:32.459007+0000) 2024-03-06T17:06:33.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:33.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:33.459254+0000) 2024-03-06T17:06:34.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:34.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:34.459525+0000) 2024-03-06T17:06:35.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:35.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:35.459788+0000) 2024-03-06T17:06:36.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:36.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:36.460034+0000) 2024-03-06T17:06:37.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:37.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:37.460291+0000) 2024-03-06T17:06:38.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:38.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:38.460551+0000) 2024-03-06T17:06:39.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:39.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:39.460828+0000) 2024-03-06T17:06:40.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:40.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:40.461069+0000) 2024-03-06T17:06:41.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:41.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:41.461261+0000) 2024-03-06T17:06:42.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:42.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:42.461494+0000) 2024-03-06T17:06:43.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:43.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:43.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:43.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:43.461791+0000) 2024-03-06T17:06:44.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:44.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:44.462041+0000) 2024-03-06T17:06:45.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:45.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:45.462279+0000) 2024-03-06T17:06:46.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:46.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:46.462541+0000) 2024-03-06T17:06:47.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:47.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:47.462785+0000) 2024-03-06T17:06:48.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:48.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:48.463034+0000) 2024-03-06T17:06:49.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:49.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:49.463310+0000) 2024-03-06T17:06:50.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:50.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:50.463551+0000) 2024-03-06T17:06:51.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:51.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:51.463797+0000) 2024-03-06T17:06:52.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:52.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:52.464053+0000) 2024-03-06T17:06:53.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:53.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:53.464330+0000) 2024-03-06T17:06:54.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:54.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:54.464584+0000) 2024-03-06T17:06:55.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:55.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:55.464823+0000) 2024-03-06T17:06:56.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:56.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:56.465061+0000) 2024-03-06T17:06:57.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:57.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:57.465323+0000) 2024-03-06T17:06:58.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:58.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:58.465583+0000) 2024-03-06T17:06:59.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:06:59.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:59.465842+0000) 2024-03-06T17:07:00.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:00.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:00.466083+0000) 2024-03-06T17:07:01.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:01.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:01.466326+0000) 2024-03-06T17:07:02.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:02.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:02.466566+0000) 2024-03-06T17:07:03.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:03.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:03.466824+0000) 2024-03-06T17:07:04.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:04.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:04.467020+0000) 2024-03-06T17:07:05.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:05.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:05.467321+0000) 2024-03-06T17:07:06.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:06.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:06.467576+0000) 2024-03-06T17:07:07.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:07.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:07.467825+0000) 2024-03-06T17:07:08.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:08.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:08.468037+0000) 2024-03-06T17:07:09.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:09.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:09.468302+0000) 2024-03-06T17:07:10.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:10.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:10.468564+0000) 2024-03-06T17:07:11.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:11.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:11.468809+0000) 2024-03-06T17:07:12.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:12.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:12.469053+0000) 2024-03-06T17:07:13.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:13.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:13.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:13.469304+0000) 2024-03-06T17:07:13.472 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:14.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:14.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:14.469567+0000) 2024-03-06T17:07:15.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:15.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:15.469830+0000) 2024-03-06T17:07:16.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:16.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:16.470092+0000) 2024-03-06T17:07:17.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:17.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:17.470354+0000) 2024-03-06T17:07:18.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:18.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:18.470596+0000) 2024-03-06T17:07:19.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:19.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:19.470871+0000) 2024-03-06T17:07:20.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:20.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:20.471062+0000) 2024-03-06T17:07:21.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:21.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:21.471313+0000) 2024-03-06T17:07:22.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:22.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:22.471555+0000) 2024-03-06T17:07:23.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:23.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:23.471806+0000) 2024-03-06T17:07:24.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:24.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:24.472073+0000) 2024-03-06T17:07:25.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:25.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:25.472340+0000) 2024-03-06T17:07:26.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:26.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:26.472601+0000) 2024-03-06T17:07:27.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:27.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:27.472898+0000) 2024-03-06T17:07:28.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:28.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:28.473152+0000) 2024-03-06T17:07:29.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:29.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:29.473421+0000) 2024-03-06T17:07:30.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:30.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:30.473731+0000) 2024-03-06T17:07:31.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:31.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:31.473978+0000) 2024-03-06T17:07:32.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:32.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:32.474215+0000) 2024-03-06T17:07:33.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:33.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:33.474472+0000) 2024-03-06T17:07:34.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:34.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:34.474757+0000) 2024-03-06T17:07:35.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:35.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:35.474978+0000) 2024-03-06T17:07:36.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:36.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:36.475199+0000) 2024-03-06T17:07:37.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:37.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:37.475449+0000) 2024-03-06T17:07:38.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:38.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:38.475746+0000) 2024-03-06T17:07:39.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:39.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:39.475957+0000) 2024-03-06T17:07:40.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:40.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:40.476187+0000) 2024-03-06T17:07:41.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:41.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:41.476444+0000) 2024-03-06T17:07:42.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:42.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:42.476760+0000) 2024-03-06T17:07:43.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:43.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:43.477007+0000) 2024-03-06T17:07:43.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:43.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:44.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:44.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:44.477276+0000) 2024-03-06T17:07:45.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:45.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:45.477545+0000) 2024-03-06T17:07:46.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:46.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:46.477808+0000) 2024-03-06T17:07:47.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:47.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:47.478072+0000) 2024-03-06T17:07:48.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:48.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:48.478269+0000) 2024-03-06T17:07:49.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:49.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:49.478522+0000) 2024-03-06T17:07:50.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:50.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:50.478782+0000) 2024-03-06T17:07:51.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:51.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:51.479035+0000) 2024-03-06T17:07:52.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:52.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:52.479294+0000) 2024-03-06T17:07:53.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:53.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:53.479552+0000) 2024-03-06T17:07:54.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:54.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:54.479821+0000) 2024-03-06T17:07:55.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:55.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:55.480064+0000) 2024-03-06T17:07:56.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:56.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:56.480317+0000) 2024-03-06T17:07:57.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:57.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:57.480544+0000) 2024-03-06T17:07:58.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:58.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:58.480857+0000) 2024-03-06T17:07:59.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:07:59.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:59.481110+0000) 2024-03-06T17:08:00.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:00.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:00.481349+0000) 2024-03-06T17:08:01.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:01.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:01.481589+0000) 2024-03-06T17:08:02.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:02.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:02.481842+0000) 2024-03-06T17:08:03.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:03.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:03.482116+0000) 2024-03-06T17:08:04.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:04.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:04.482381+0000) 2024-03-06T17:08:05.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:05.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:05.482669+0000) 2024-03-06T17:08:06.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:06.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:06.482947+0000) 2024-03-06T17:08:07.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:07.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:07.483202+0000) 2024-03-06T17:08:08.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:08.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:08.483467+0000) 2024-03-06T17:08:09.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:09.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:09.483771+0000) 2024-03-06T17:08:10.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:10.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:10.484014+0000) 2024-03-06T17:08:11.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:11.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:11.484274+0000) 2024-03-06T17:08:12.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:12.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:12.484530+0000) 2024-03-06T17:08:13.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:13.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:13.484803+0000) 2024-03-06T17:08:13.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:13.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:14.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:14.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:14.485070+0000) 2024-03-06T17:08:15.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:15.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:15.485332+0000) 2024-03-06T17:08:16.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:16.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:16.485588+0000) 2024-03-06T17:08:17.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:17.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:17.485848+0000) 2024-03-06T17:08:18.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:18.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:18.486110+0000) 2024-03-06T17:08:19.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:19.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:19.486377+0000) 2024-03-06T17:08:20.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:20.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:20.486571+0000) 2024-03-06T17:08:21.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:21.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:21.486827+0000) 2024-03-06T17:08:22.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:22.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:22.487089+0000) 2024-03-06T17:08:23.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:23.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:23.487342+0000) 2024-03-06T17:08:24.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:24.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:24.487594+0000) 2024-03-06T17:08:25.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:25.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:25.487867+0000) 2024-03-06T17:08:26.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:26.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:26.488130+0000) 2024-03-06T17:08:27.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:27.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:27.488383+0000) 2024-03-06T17:08:28.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:28.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:28.488707+0000) 2024-03-06T17:08:29.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:29.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:29.488981+0000) 2024-03-06T17:08:30.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:30.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:30.489243+0000) 2024-03-06T17:08:31.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:31.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:31.489433+0000) 2024-03-06T17:08:32.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:32.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:32.489730+0000) 2024-03-06T17:08:33.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:33.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:33.489982+0000) 2024-03-06T17:08:34.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:34.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:34.490248+0000) 2024-03-06T17:08:35.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:35.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:35.490496+0000) 2024-03-06T17:08:36.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:36.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:36.490758+0000) 2024-03-06T17:08:37.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:37.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:37.491006+0000) 2024-03-06T17:08:38.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:38.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:38.491270+0000) 2024-03-06T17:08:39.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:39.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:39.491542+0000) 2024-03-06T17:08:40.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:40.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:40.491801+0000) 2024-03-06T17:08:41.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:41.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:41.492061+0000) 2024-03-06T17:08:42.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:42.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:42.492303+0000) 2024-03-06T17:08:43.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:43.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:43.492570+0000) 2024-03-06T17:08:43.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:43.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:44.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:44.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:44.492821+0000) 2024-03-06T17:08:45.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:45.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:45.493082+0000) 2024-03-06T17:08:46.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:46.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:46.493320+0000) 2024-03-06T17:08:47.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:47.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:47.493579+0000) 2024-03-06T17:08:48.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:48.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:48.493838+0000) 2024-03-06T17:08:49.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:49.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:49.494096+0000) 2024-03-06T17:08:50.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:50.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:50.494338+0000) 2024-03-06T17:08:51.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:51.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:51.494594+0000) 2024-03-06T17:08:52.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:52.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:52.494860+0000) 2024-03-06T17:08:53.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:53.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:53.495122+0000) 2024-03-06T17:08:54.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:54.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:54.495388+0000) 2024-03-06T17:08:55.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:55.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:55.495688+0000) 2024-03-06T17:08:56.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:56.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:56.495936+0000) 2024-03-06T17:08:57.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:57.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:57.496178+0000) 2024-03-06T17:08:58.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:58.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:58.496425+0000) 2024-03-06T17:08:59.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:08:59.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:59.496727+0000) 2024-03-06T17:09:00.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:00.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:00.496980+0000) 2024-03-06T17:09:01.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:01.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:01.497217+0000) 2024-03-06T17:09:02.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:02.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:02.497466+0000) 2024-03-06T17:09:03.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:03.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:03.497754+0000) 2024-03-06T17:09:04.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:04.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:04.498020+0000) 2024-03-06T17:09:05.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:05.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:05.498269+0000) 2024-03-06T17:09:06.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:06.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:06.498528+0000) 2024-03-06T17:09:07.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:07.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:07.498783+0000) 2024-03-06T17:09:08.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:08.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:08.499017+0000) 2024-03-06T17:09:09.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:09.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:09.499256+0000) 2024-03-06T17:09:10.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:10.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:10.499502+0000) 2024-03-06T17:09:11.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:11.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:11.499764+0000) 2024-03-06T17:09:12.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:12.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:12.500028+0000) 2024-03-06T17:09:13.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:13.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:13.500290+0000) 2024-03-06T17:09:14.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:14.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:14.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:14.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:14.500555+0000) 2024-03-06T17:09:15.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:15.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:15.500808+0000) 2024-03-06T17:09:16.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:16.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:16.501072+0000) 2024-03-06T17:09:17.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:17.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:17.501330+0000) 2024-03-06T17:09:18.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:18.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:18.501577+0000) 2024-03-06T17:09:19.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:19.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:19.501843+0000) 2024-03-06T17:09:20.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:20.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:20.502100+0000) 2024-03-06T17:09:21.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:21.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:21.502355+0000) 2024-03-06T17:09:22.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:22.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:22.502608+0000) 2024-03-06T17:09:23.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:23.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:23.502945+0000) 2024-03-06T17:09:24.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:24.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:24.503209+0000) 2024-03-06T17:09:25.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:25.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:25.503471+0000) 2024-03-06T17:09:26.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:26.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:26.503753+0000) 2024-03-06T17:09:27.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:27.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:27.504013+0000) 2024-03-06T17:09:28.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:28.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:28.504290+0000) 2024-03-06T17:09:29.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:29.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:29.504572+0000) 2024-03-06T17:09:30.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:30.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:30.504847+0000) 2024-03-06T17:09:31.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:31.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:31.505125+0000) 2024-03-06T17:09:32.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:32.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:32.505394+0000) 2024-03-06T17:09:33.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:33.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:33.505737+0000) 2024-03-06T17:09:34.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:34.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:34.506071+0000) 2024-03-06T17:09:35.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:35.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:35.506338+0000) 2024-03-06T17:09:36.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:36.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:36.506611+0000) 2024-03-06T17:09:37.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:37.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:37.506950+0000) 2024-03-06T17:09:38.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:38.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:38.507229+0000) 2024-03-06T17:09:39.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:39.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:39.507505+0000) 2024-03-06T17:09:40.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:40.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:40.507767+0000) 2024-03-06T17:09:41.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:41.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:41.508043+0000) 2024-03-06T17:09:42.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:42.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:42.508322+0000) 2024-03-06T17:09:43.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:43.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:43.508615+0000) 2024-03-06T17:09:44.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:44.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:44.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:44.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:44.508942+0000) 2024-03-06T17:09:45.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:45.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:45.509189+0000) 2024-03-06T17:09:46.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:46.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:46.509442+0000) 2024-03-06T17:09:47.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:47.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:47.509733+0000) 2024-03-06T17:09:48.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:48.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:48.509963+0000) 2024-03-06T17:09:49.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:49.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:49.510220+0000) 2024-03-06T17:09:50.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:50.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:50.510464+0000) 2024-03-06T17:09:51.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:51.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:51.510740+0000) 2024-03-06T17:09:52.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:52.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:52.510974+0000) 2024-03-06T17:09:53.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:53.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:53.511227+0000) 2024-03-06T17:09:54.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:54.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:54.511476+0000) 2024-03-06T17:09:55.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:55.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:55.511763+0000) 2024-03-06T17:09:56.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:56.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:56.512018+0000) 2024-03-06T17:09:57.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:57.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:57.512279+0000) 2024-03-06T17:09:58.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:58.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:58.512488+0000) 2024-03-06T17:09:59.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:09:59.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:59.512743+0000) 2024-03-06T17:10:00.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:00.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:00.513006+0000) 2024-03-06T17:10:01.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:01.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:01.513267+0000) 2024-03-06T17:10:02.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:02.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:02.513507+0000) 2024-03-06T17:10:03.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:03.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:03.513756+0000) 2024-03-06T17:10:04.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:04.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:04.514023+0000) 2024-03-06T17:10:05.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:05.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:05.514286+0000) 2024-03-06T17:10:06.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:06.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:06.514523+0000) 2024-03-06T17:10:07.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:07.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:07.514785+0000) 2024-03-06T17:10:08.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:08.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:08.515046+0000) 2024-03-06T17:10:09.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:09.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:09.515307+0000) 2024-03-06T17:10:10.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:10.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:10.515605+0000) 2024-03-06T17:10:11.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:11.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:11.515983+0000) 2024-03-06T17:10:12.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:12.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:12.516195+0000) 2024-03-06T17:10:13.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:13.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:13.516465+0000) 2024-03-06T17:10:14.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:14.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:14.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:14.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:14.516786+0000) 2024-03-06T17:10:15.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:15.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:15.517088+0000) 2024-03-06T17:10:16.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:16.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:16.517407+0000) 2024-03-06T17:10:17.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:17.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:17.517774+0000) 2024-03-06T17:10:18.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:18.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:18.518030+0000) 2024-03-06T17:10:19.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:19.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:19.518341+0000) 2024-03-06T17:10:20.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:20.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:20.518693+0000) 2024-03-06T17:10:21.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:21.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:21.519005+0000) 2024-03-06T17:10:22.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:22.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:22.519315+0000) 2024-03-06T17:10:23.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:23.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:23.519688+0000) 2024-03-06T17:10:24.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:24.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:24.520010+0000) 2024-03-06T17:10:25.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:25.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:25.520338+0000) 2024-03-06T17:10:26.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:26.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:26.520675+0000) 2024-03-06T17:10:27.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:27.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:27.520905+0000) 2024-03-06T17:10:28.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:28.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:28.521269+0000) 2024-03-06T17:10:29.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:29.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:29.521586+0000) 2024-03-06T17:10:30.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:30.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:30.521938+0000) 2024-03-06T17:10:31.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:31.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:31.522253+0000) 2024-03-06T17:10:32.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:32.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:32.522566+0000) 2024-03-06T17:10:33.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:33.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:33.522883+0000) 2024-03-06T17:10:34.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:34.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:34.523133+0000) 2024-03-06T17:10:35.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:35.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:35.523381+0000) 2024-03-06T17:10:36.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:36.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:36.523731+0000) 2024-03-06T17:10:37.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:37.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:37.524051+0000) 2024-03-06T17:10:38.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:38.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:38.524353+0000) 2024-03-06T17:10:39.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:39.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:39.524707+0000) 2024-03-06T17:10:40.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:40.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:40.525020+0000) 2024-03-06T17:10:41.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:41.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:41.525322+0000) 2024-03-06T17:10:42.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:42.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:42.525681+0000) 2024-03-06T17:10:43.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:43.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:43.526019+0000) 2024-03-06T17:10:44.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:44.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:44.526321+0000) 2024-03-06T17:10:44.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:44.559 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:45.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:45.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:45.526615+0000) 2024-03-06T17:10:46.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:46.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:46.526994+0000) 2024-03-06T17:10:47.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:47.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:47.527312+0000) 2024-03-06T17:10:48.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:48.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:48.527679+0000) 2024-03-06T17:10:49.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:49.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:49.527993+0000) 2024-03-06T17:10:50.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:50.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:50.528307+0000) 2024-03-06T17:10:51.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:51.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:51.528600+0000) 2024-03-06T17:10:52.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:52.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:52.528954+0000) 2024-03-06T17:10:53.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:53.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:53.529253+0000) 2024-03-06T17:10:54.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:54.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:54.529536+0000) 2024-03-06T17:10:55.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:55.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:55.529752+0000) 2024-03-06T17:10:56.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:56.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:56.530048+0000) 2024-03-06T17:10:57.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:57.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:57.530366+0000) 2024-03-06T17:10:58.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:58.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:58.530743+0000) 2024-03-06T17:10:59.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:10:59.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:59.531037+0000) 2024-03-06T17:11:00.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:00.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:00.531334+0000) 2024-03-06T17:11:01.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:01.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:01.531618+0000) 2024-03-06T17:11:02.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:02.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:02.531931+0000) 2024-03-06T17:11:03.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:03.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:03.532206+0000) 2024-03-06T17:11:04.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:04.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:04.532473+0000) 2024-03-06T17:11:05.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:05.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:05.532761+0000) 2024-03-06T17:11:06.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:06.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:06.533022+0000) 2024-03-06T17:11:07.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:07.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:07.533218+0000) 2024-03-06T17:11:08.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:08.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:08.533479+0000) 2024-03-06T17:11:09.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:09.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:09.533764+0000) 2024-03-06T17:11:10.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:10.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:10.534015+0000) 2024-03-06T17:11:11.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:11.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:11.534198+0000) 2024-03-06T17:11:12.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:12.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:12.534445+0000) 2024-03-06T17:11:13.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:13.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:13.534753+0000) 2024-03-06T17:11:14.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:14.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:14.535010+0000) 2024-03-06T17:11:14.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:14.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:15.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:15.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:15.535223+0000) 2024-03-06T17:11:16.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:16.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:16.535532+0000) 2024-03-06T17:11:17.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:17.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:17.535808+0000) 2024-03-06T17:11:18.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:18.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:18.536069+0000) 2024-03-06T17:11:19.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:19.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:19.536315+0000) 2024-03-06T17:11:20.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:20.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:20.536609+0000) 2024-03-06T17:11:21.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:21.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:21.536967+0000) 2024-03-06T17:11:22.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:22.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:22.537274+0000) 2024-03-06T17:11:23.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:23.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:23.537601+0000) 2024-03-06T17:11:24.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:24.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:24.537914+0000) 2024-03-06T17:11:25.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:25.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:25.538235+0000) 2024-03-06T17:11:26.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:26.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:26.538527+0000) 2024-03-06T17:11:27.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:27.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:27.538807+0000) 2024-03-06T17:11:28.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:28.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:28.539122+0000) 2024-03-06T17:11:29.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:29.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:29.539423+0000) 2024-03-06T17:11:30.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:30.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:30.539772+0000) 2024-03-06T17:11:31.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:31.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:31.540073+0000) 2024-03-06T17:11:32.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:32.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:32.540367+0000) 2024-03-06T17:11:33.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:33.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:33.540719+0000) 2024-03-06T17:11:34.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:34.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:34.541031+0000) 2024-03-06T17:11:35.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:35.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:35.541332+0000) 2024-03-06T17:11:36.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:36.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:36.541677+0000) 2024-03-06T17:11:37.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:37.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:37.541989+0000) 2024-03-06T17:11:38.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:38.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:38.542313+0000) 2024-03-06T17:11:39.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:39.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:39.542599+0000) 2024-03-06T17:11:40.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:40.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:40.542959+0000) 2024-03-06T17:11:41.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:41.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:41.543273+0000) 2024-03-06T17:11:42.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:42.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:42.543586+0000) 2024-03-06T17:11:43.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:43.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:43.543938+0000) 2024-03-06T17:11:44.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:44.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:44.544274+0000) 2024-03-06T17:11:44.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:44.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:45.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:45.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:45.544593+0000) 2024-03-06T17:11:46.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:46.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:46.544942+0000) 2024-03-06T17:11:47.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:47.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:47.545239+0000) 2024-03-06T17:11:48.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:48.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:48.545502+0000) 2024-03-06T17:11:49.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:49.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:49.545746+0000) 2024-03-06T17:11:50.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:50.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:50.546002+0000) 2024-03-06T17:11:51.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:51.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:51.546246+0000) 2024-03-06T17:11:52.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:52.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:52.546491+0000) 2024-03-06T17:11:53.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:53.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:53.546758+0000) 2024-03-06T17:11:54.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:54.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:54.546955+0000) 2024-03-06T17:11:55.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:55.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:55.547198+0000) 2024-03-06T17:11:56.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:56.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:56.547462+0000) 2024-03-06T17:11:57.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:57.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:57.547740+0000) 2024-03-06T17:11:58.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:58.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:58.548011+0000) 2024-03-06T17:11:59.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:11:59.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:59.548271+0000) 2024-03-06T17:12:00.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:00.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:00.548537+0000) 2024-03-06T17:12:01.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:01.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:01.548771+0000) 2024-03-06T17:12:02.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:02.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:02.549009+0000) 2024-03-06T17:12:03.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:03.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:03.549262+0000) 2024-03-06T17:12:04.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:04.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:04.549570+0000) 2024-03-06T17:12:05.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:05.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:05.549836+0000) 2024-03-06T17:12:06.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:06.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:06.550100+0000) 2024-03-06T17:12:07.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:07.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:07.550343+0000) 2024-03-06T17:12:08.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:08.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:08.550609+0000) 2024-03-06T17:12:09.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:09.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:09.550845+0000) 2024-03-06T17:12:10.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:10.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:10.551114+0000) 2024-03-06T17:12:11.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:11.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:11.551372+0000) 2024-03-06T17:12:12.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:12.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:12.551665+0000) 2024-03-06T17:12:13.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:13.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:13.551911+0000) 2024-03-06T17:12:14.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:14.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:14.552177+0000) 2024-03-06T17:12:14.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:14.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:15.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:15.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:15.552415+0000) 2024-03-06T17:12:16.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:16.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:16.552713+0000) 2024-03-06T17:12:17.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:17.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:17.552968+0000) 2024-03-06T17:12:18.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:18.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:18.553225+0000) 2024-03-06T17:12:19.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:19.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:19.553490+0000) 2024-03-06T17:12:20.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:20.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:20.553768+0000) 2024-03-06T17:12:21.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:21.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:21.554015+0000) 2024-03-06T17:12:22.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:22.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:22.554277+0000) 2024-03-06T17:12:23.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:23.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:23.554541+0000) 2024-03-06T17:12:24.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:24.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:24.554783+0000) 2024-03-06T17:12:25.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:25.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:25.555046+0000) 2024-03-06T17:12:26.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:26.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:26.555286+0000) 2024-03-06T17:12:27.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:27.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:27.555531+0000) 2024-03-06T17:12:28.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:28.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:28.555793+0000) 2024-03-06T17:12:29.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:29.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:29.556041+0000) 2024-03-06T17:12:30.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:30.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:30.556297+0000) 2024-03-06T17:12:31.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:31.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:31.556540+0000) 2024-03-06T17:12:32.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:32.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:32.556796+0000) 2024-03-06T17:12:33.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:33.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:33.557053+0000) 2024-03-06T17:12:34.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:34.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:34.557317+0000) 2024-03-06T17:12:35.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:35.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:35.557587+0000) 2024-03-06T17:12:36.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:36.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:36.557861+0000) 2024-03-06T17:12:37.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:37.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:37.558104+0000) 2024-03-06T17:12:38.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:38.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:38.558330+0000) 2024-03-06T17:12:39.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:39.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:39.558604+0000) 2024-03-06T17:12:40.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:40.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:40.558914+0000) 2024-03-06T17:12:41.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:41.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:41.559175+0000) 2024-03-06T17:12:42.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:42.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:42.559418+0000) 2024-03-06T17:12:43.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:43.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:43.559696+0000) 2024-03-06T17:12:44.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:44.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:44.559964+0000) 2024-03-06T17:12:45.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:45.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:45.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:45.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:45.560161+0000) 2024-03-06T17:12:46.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:46.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:46.560409+0000) 2024-03-06T17:12:47.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:47.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:47.560695+0000) 2024-03-06T17:12:48.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:48.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:48.560928+0000) 2024-03-06T17:12:49.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:49.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:49.561186+0000) 2024-03-06T17:12:50.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:50.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:50.561453+0000) 2024-03-06T17:12:51.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:51.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:51.561740+0000) 2024-03-06T17:12:52.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:52.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:52.562011+0000) 2024-03-06T17:12:53.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:53.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:53.562249+0000) 2024-03-06T17:12:54.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:54.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:54.562512+0000) 2024-03-06T17:12:55.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:55.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:55.562779+0000) 2024-03-06T17:12:56.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:56.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:56.563040+0000) 2024-03-06T17:12:57.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:57.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:57.563431+0000) 2024-03-06T17:12:58.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:58.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:58.563730+0000) 2024-03-06T17:12:59.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:12:59.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:59.563958+0000) 2024-03-06T17:13:00.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:00.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:00.564213+0000) 2024-03-06T17:13:01.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:01.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:01.564458+0000) 2024-03-06T17:13:02.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:02.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:02.564768+0000) 2024-03-06T17:13:03.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:03.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:03.565030+0000) 2024-03-06T17:13:04.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:04.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:04.565261+0000) 2024-03-06T17:13:05.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:05.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:05.565731+0000) 2024-03-06T17:13:06.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:06.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:06.565996+0000) 2024-03-06T17:13:07.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:07.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:07.566226+0000) 2024-03-06T17:13:08.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:08.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:08.566482+0000) 2024-03-06T17:13:09.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:09.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:09.566742+0000) 2024-03-06T17:13:10.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:10.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:10.567009+0000) 2024-03-06T17:13:11.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:11.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:11.567268+0000) 2024-03-06T17:13:12.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:12.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:12.567605+0000) 2024-03-06T17:13:13.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:13.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:13.567930+0000) 2024-03-06T17:13:14.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:14.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:14.568195+0000) 2024-03-06T17:13:15.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:15.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:15.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:15.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:15.568437+0000) 2024-03-06T17:13:16.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:16.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:16.568747+0000) 2024-03-06T17:13:17.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:17.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:17.568982+0000) 2024-03-06T17:13:18.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:18.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:18.569249+0000) 2024-03-06T17:13:19.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:19.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:19.569497+0000) 2024-03-06T17:13:20.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:20.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:20.569790+0000) 2024-03-06T17:13:21.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:21.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:21.570062+0000) 2024-03-06T17:13:22.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:22.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:22.570346+0000) 2024-03-06T17:13:23.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:23.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:23.570623+0000) 2024-03-06T17:13:24.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:24.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:24.570932+0000) 2024-03-06T17:13:25.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:25.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:25.571179+0000) 2024-03-06T17:13:26.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:26.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:26.571424+0000) 2024-03-06T17:13:27.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:27.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:27.571700+0000) 2024-03-06T17:13:28.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:28.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:28.571921+0000) 2024-03-06T17:13:29.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:29.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:29.572160+0000) 2024-03-06T17:13:30.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:30.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:30.572406+0000) 2024-03-06T17:13:31.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:31.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:31.572705+0000) 2024-03-06T17:13:32.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:32.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:32.572978+0000) 2024-03-06T17:13:33.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:33.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:33.573239+0000) 2024-03-06T17:13:34.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:34.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:34.573488+0000) 2024-03-06T17:13:35.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:35.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:35.573752+0000) 2024-03-06T17:13:36.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:36.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:36.573952+0000) 2024-03-06T17:13:37.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:37.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:37.574202+0000) 2024-03-06T17:13:38.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:38.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:38.574468+0000) 2024-03-06T17:13:39.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:39.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:39.574750+0000) 2024-03-06T17:13:40.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:40.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:40.575006+0000) 2024-03-06T17:13:41.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:41.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:41.575257+0000) 2024-03-06T17:13:42.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:42.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:42.575528+0000) 2024-03-06T17:13:43.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:43.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:43.575785+0000) 2024-03-06T17:13:44.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:44.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:44.575986+0000) 2024-03-06T17:13:45.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:45.456 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:45.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:45.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:45.576224+0000) 2024-03-06T17:13:46.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:46.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:46.576480+0000) 2024-03-06T17:13:47.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:47.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:47.576733+0000) 2024-03-06T17:13:48.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:48.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:48.576923+0000) 2024-03-06T17:13:49.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:49.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:49.577151+0000) 2024-03-06T17:13:50.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:50.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:50.577420+0000) 2024-03-06T17:13:51.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:51.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:51.577728+0000) 2024-03-06T17:13:52.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:52.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:52.577941+0000) 2024-03-06T17:13:53.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:53.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:53.578203+0000) 2024-03-06T17:13:54.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:54.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:54.578472+0000) 2024-03-06T17:13:55.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:55.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:55.578759+0000) 2024-03-06T17:13:56.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:56.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:56.579011+0000) 2024-03-06T17:13:57.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:57.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:57.579260+0000) 2024-03-06T17:13:58.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:58.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:58.579521+0000) 2024-03-06T17:13:59.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:13:59.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:59.579769+0000) 2024-03-06T17:14:00.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:00.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:00.580021+0000) 2024-03-06T17:14:01.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:01.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:01.580266+0000) 2024-03-06T17:14:02.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:02.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:02.580521+0000) 2024-03-06T17:14:03.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:03.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:03.580777+0000) 2024-03-06T17:14:04.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:04.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:04.581039+0000) 2024-03-06T17:14:05.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:05.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:05.581296+0000) 2024-03-06T17:14:06.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:06.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:06.581568+0000) 2024-03-06T17:14:07.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:07.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:07.581812+0000) 2024-03-06T17:14:08.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:08.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:08.582086+0000) 2024-03-06T17:14:09.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:09.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:09.582334+0000) 2024-03-06T17:14:10.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:10.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:10.582588+0000) 2024-03-06T17:14:11.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:11.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:11.582846+0000) 2024-03-06T17:14:12.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:12.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:12.583094+0000) 2024-03-06T17:14:13.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:13.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:13.583334+0000) 2024-03-06T17:14:14.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:14.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:14.583583+0000) 2024-03-06T17:14:15.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:15.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:15.583848+0000) 2024-03-06T17:14:15.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:15.604 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:16.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:16.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:16.584050+0000) 2024-03-06T17:14:17.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:17.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:17.584476+0000) 2024-03-06T17:14:18.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:18.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:18.584757+0000) 2024-03-06T17:14:19.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:19.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:19.584998+0000) 2024-03-06T17:14:20.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:20.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:20.585246+0000) 2024-03-06T17:14:21.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:21.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:21.585501+0000) 2024-03-06T17:14:22.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:22.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:22.585766+0000) 2024-03-06T17:14:23.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:23.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:23.585960+0000) 2024-03-06T17:14:24.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:24.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:24.586206+0000) 2024-03-06T17:14:25.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:25.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:25.586459+0000) 2024-03-06T17:14:26.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:26.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:26.586760+0000) 2024-03-06T17:14:27.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:27.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:27.586998+0000) 2024-03-06T17:14:28.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:28.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:28.587262+0000) 2024-03-06T17:14:29.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:29.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:29.587537+0000) 2024-03-06T17:14:30.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:30.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:30.587803+0000) 2024-03-06T17:14:31.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:31.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:31.588065+0000) 2024-03-06T17:14:32.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:32.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:32.588315+0000) 2024-03-06T17:14:33.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:33.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:33.588588+0000) 2024-03-06T17:14:34.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:34.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:34.588939+0000) 2024-03-06T17:14:35.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:35.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:35.589198+0000) 2024-03-06T17:14:36.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:36.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:36.589454+0000) 2024-03-06T17:14:37.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:37.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:37.589728+0000) 2024-03-06T17:14:38.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:38.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:38.589989+0000) 2024-03-06T17:14:39.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:39.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:39.590234+0000) 2024-03-06T17:14:40.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:40.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:40.590509+0000) 2024-03-06T17:14:41.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:41.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:41.590797+0000) 2024-03-06T17:14:42.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:42.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:42.591057+0000) 2024-03-06T17:14:43.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:43.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:43.591308+0000) 2024-03-06T17:14:44.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:44.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:44.591557+0000) 2024-03-06T17:14:45.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:45.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:45.591822+0000) 2024-03-06T17:14:45.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:45.752 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:46.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:46.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:46.592085+0000) 2024-03-06T17:14:47.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:47.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:47.592261+0000) 2024-03-06T17:14:48.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:48.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:48.592522+0000) 2024-03-06T17:14:49.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:49.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:49.592803+0000) 2024-03-06T17:14:50.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:50.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:50.593090+0000) 2024-03-06T17:14:51.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:51.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:51.593346+0000) 2024-03-06T17:14:52.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:52.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:52.593602+0000) 2024-03-06T17:14:53.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:53.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:53.593882+0000) 2024-03-06T17:14:54.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:54.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:54.594135+0000) 2024-03-06T17:14:55.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:55.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:55.594392+0000) 2024-03-06T17:14:56.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:56.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:56.594691+0000) 2024-03-06T17:14:57.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:57.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:57.595109+0000) 2024-03-06T17:14:58.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:58.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:58.595374+0000) 2024-03-06T17:14:59.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:14:59.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:59.595567+0000) 2024-03-06T17:15:00.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:00.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:00.595837+0000) 2024-03-06T17:15:01.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:01.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:01.596100+0000) 2024-03-06T17:15:02.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:02.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:02.596379+0000) 2024-03-06T17:15:03.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:03.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:03.596703+0000) 2024-03-06T17:15:04.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:04.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:04.596993+0000) 2024-03-06T17:15:05.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:05.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:05.597278+0000) 2024-03-06T17:15:06.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:06.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:06.597543+0000) 2024-03-06T17:15:07.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:07.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:07.597812+0000) 2024-03-06T17:15:08.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:08.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:08.598093+0000) 2024-03-06T17:15:09.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:09.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:09.598339+0000) 2024-03-06T17:15:10.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:10.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:10.598674+0000) 2024-03-06T17:15:11.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:11.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:11.598953+0000) 2024-03-06T17:15:12.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:12.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:12.599246+0000) 2024-03-06T17:15:13.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:13.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:13.599513+0000) 2024-03-06T17:15:14.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:14.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:14.599823+0000) 2024-03-06T17:15:15.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:15.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:15.600070+0000) 2024-03-06T17:15:15.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:15.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:16.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:16.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:16.600338+0000) 2024-03-06T17:15:17.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:17.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:17.600586+0000) 2024-03-06T17:15:18.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:18.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:18.600839+0000) 2024-03-06T17:15:19.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:19.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:19.601075+0000) 2024-03-06T17:15:20.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:20.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:20.601330+0000) 2024-03-06T17:15:21.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:21.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:21.601538+0000) 2024-03-06T17:15:22.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:22.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:22.601825+0000) 2024-03-06T17:15:23.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:23.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:23.602092+0000) 2024-03-06T17:15:24.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:24.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:24.602344+0000) 2024-03-06T17:15:25.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:25.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:25.602575+0000) 2024-03-06T17:15:26.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:26.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:26.602869+0000) 2024-03-06T17:15:27.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:27.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:27.603116+0000) 2024-03-06T17:15:28.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:28.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:28.603389+0000) 2024-03-06T17:15:29.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:29.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:29.603587+0000) 2024-03-06T17:15:30.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:30.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:30.603854+0000) 2024-03-06T17:15:31.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:31.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:31.604117+0000) 2024-03-06T17:15:32.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:32.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:32.604401+0000) 2024-03-06T17:15:33.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:33.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:33.604696+0000) 2024-03-06T17:15:34.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:34.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:34.604974+0000) 2024-03-06T17:15:35.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:35.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:35.605236+0000) 2024-03-06T17:15:36.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:36.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:36.605505+0000) 2024-03-06T17:15:37.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:37.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:37.605774+0000) 2024-03-06T17:15:38.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:38.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:38.606044+0000) 2024-03-06T17:15:39.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:39.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:39.606310+0000) 2024-03-06T17:15:40.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:40.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:40.606585+0000) 2024-03-06T17:15:41.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:41.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:41.606878+0000) 2024-03-06T17:15:42.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:42.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:42.607143+0000) 2024-03-06T17:15:43.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:43.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:43.607421+0000) 2024-03-06T17:15:44.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:44.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:44.607721+0000) 2024-03-06T17:15:45.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:45.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:45.607923+0000) 2024-03-06T17:15:46.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:46.057 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:46.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:46.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:46.608140+0000) 2024-03-06T17:15:47.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:47.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:47.608385+0000) 2024-03-06T17:15:48.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:48.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:48.608582+0000) 2024-03-06T17:15:49.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:49.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:49.608831+0000) 2024-03-06T17:15:50.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:50.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:50.609095+0000) 2024-03-06T17:15:51.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:51.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:51.609343+0000) 2024-03-06T17:15:52.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:52.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:52.609594+0000) 2024-03-06T17:15:53.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:53.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:53.609846+0000) 2024-03-06T17:15:54.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:54.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:54.610110+0000) 2024-03-06T17:15:55.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:55.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:55.610278+0000) 2024-03-06T17:15:56.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:56.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:56.610552+0000) 2024-03-06T17:15:57.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:57.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:57.610777+0000) 2024-03-06T17:15:58.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:58.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:58.610973+0000) 2024-03-06T17:15:59.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:15:59.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:59.611215+0000) 2024-03-06T17:16:00.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:00.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:00.611410+0000) 2024-03-06T17:16:01.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:01.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:01.611607+0000) 2024-03-06T17:16:02.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:02.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:02.611894+0000) 2024-03-06T17:16:03.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:03.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:03.612157+0000) 2024-03-06T17:16:04.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:04.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:04.612418+0000) 2024-03-06T17:16:05.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:05.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:05.612696+0000) 2024-03-06T17:16:06.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:06.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:06.612941+0000) 2024-03-06T17:16:07.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:07.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:07.613176+0000) 2024-03-06T17:16:08.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:08.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:08.613431+0000) 2024-03-06T17:16:09.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:09.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:09.613730+0000) 2024-03-06T17:16:10.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:10.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:10.613998+0000) 2024-03-06T17:16:11.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:11.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:11.614257+0000) 2024-03-06T17:16:12.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:12.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:12.614519+0000) 2024-03-06T17:16:13.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:13.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:13.614747+0000) 2024-03-06T17:16:14.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:14.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:14.615057+0000) 2024-03-06T17:16:15.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:15.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:15.615308+0000) 2024-03-06T17:16:16.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:16.204 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:16.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:16.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:16.615569+0000) 2024-03-06T17:16:17.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:17.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:17.615821+0000) 2024-03-06T17:16:18.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:18.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:18.616073+0000) 2024-03-06T17:16:19.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:19.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:19.616302+0000) 2024-03-06T17:16:20.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:20.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:20.616564+0000) 2024-03-06T17:16:21.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:21.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:21.616814+0000) 2024-03-06T17:16:22.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:22.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:22.617077+0000) 2024-03-06T17:16:23.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:23.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:23.617311+0000) 2024-03-06T17:16:24.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:24.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:24.617595+0000) 2024-03-06T17:16:25.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:25.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:25.617802+0000) 2024-03-06T17:16:26.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:26.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:26.618080+0000) 2024-03-06T17:16:27.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:27.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:27.618347+0000) 2024-03-06T17:16:28.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:28.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:28.618609+0000) 2024-03-06T17:16:29.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:29.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:29.618885+0000) 2024-03-06T17:16:30.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:30.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:30.619146+0000) 2024-03-06T17:16:31.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:31.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:31.619410+0000) 2024-03-06T17:16:32.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:32.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:32.619687+0000) 2024-03-06T17:16:33.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:33.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:33.619985+0000) 2024-03-06T17:16:34.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:34.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:34.620252+0000) 2024-03-06T17:16:35.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:35.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:35.620496+0000) 2024-03-06T17:16:36.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:36.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:36.620768+0000) 2024-03-06T17:16:37.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:37.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:37.620992+0000) 2024-03-06T17:16:38.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:38.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:38.621248+0000) 2024-03-06T17:16:39.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:39.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:39.621479+0000) 2024-03-06T17:16:40.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:40.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:40.621766+0000) 2024-03-06T17:16:41.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:41.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:41.621998+0000) 2024-03-06T17:16:42.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:42.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:42.622262+0000) 2024-03-06T17:16:43.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:43.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:43.622548+0000) 2024-03-06T17:16:44.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:44.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:44.622820+0000) 2024-03-06T17:16:45.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:45.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:45.623096+0000) 2024-03-06T17:16:46.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:46.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:46.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:46.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:46.623359+0000) 2024-03-06T17:16:47.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:47.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:47.623624+0000) 2024-03-06T17:16:48.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:48.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:48.623903+0000) 2024-03-06T17:16:49.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:49.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:49.624180+0000) 2024-03-06T17:16:50.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:50.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:50.624446+0000) 2024-03-06T17:16:51.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:51.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:51.624768+0000) 2024-03-06T17:16:52.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:52.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:52.625035+0000) 2024-03-06T17:16:53.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:53.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:53.625329+0000) 2024-03-06T17:16:54.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:54.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:54.625543+0000) 2024-03-06T17:16:55.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:55.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:55.625815+0000) 2024-03-06T17:16:56.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:56.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:56.626098+0000) 2024-03-06T17:16:57.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:57.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:57.626373+0000) 2024-03-06T17:16:58.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:58.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:58.626622+0000) 2024-03-06T17:16:59.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:16:59.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:59.626936+0000) 2024-03-06T17:17:00.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:00.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:00.627204+0000) 2024-03-06T17:17:01.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:01.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:01.627482+0000) 2024-03-06T17:17:02.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:02.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:02.627805+0000) 2024-03-06T17:17:03.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:03.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:03.628091+0000) 2024-03-06T17:17:04.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:04.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:04.628344+0000) 2024-03-06T17:17:05.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:05.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:05.628555+0000) 2024-03-06T17:17:06.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:06.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:06.628771+0000) 2024-03-06T17:17:07.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:07.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:07.629034+0000) 2024-03-06T17:17:08.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:08.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:08.629215+0000) 2024-03-06T17:17:09.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:09.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:09.629447+0000) 2024-03-06T17:17:10.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:10.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:10.629735+0000) 2024-03-06T17:17:11.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:11.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:11.629939+0000) 2024-03-06T17:17:12.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:12.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:12.630191+0000) 2024-03-06T17:17:13.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:13.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:13.630463+0000) 2024-03-06T17:17:14.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:14.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:14.630752+0000) 2024-03-06T17:17:15.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:15.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:15.630990+0000) 2024-03-06T17:17:16.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:16.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:16.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:16.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:16.631171+0000) 2024-03-06T17:17:17.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:17.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:17.631412+0000) 2024-03-06T17:17:18.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:18.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:18.631703+0000) 2024-03-06T17:17:19.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:19.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:19.631958+0000) 2024-03-06T17:17:20.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:20.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:20.632203+0000) 2024-03-06T17:17:21.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:21.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:21.632453+0000) 2024-03-06T17:17:22.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:22.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:22.632748+0000) 2024-03-06T17:17:23.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:23.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:23.633024+0000) 2024-03-06T17:17:24.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:24.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:24.633283+0000) 2024-03-06T17:17:25.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:25.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:25.633466+0000) 2024-03-06T17:17:26.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:26.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:26.633742+0000) 2024-03-06T17:17:27.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:27.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:27.634001+0000) 2024-03-06T17:17:28.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:28.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:28.634257+0000) 2024-03-06T17:17:29.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:29.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:29.634504+0000) 2024-03-06T17:17:30.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:30.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:30.634761+0000) 2024-03-06T17:17:31.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:31.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:31.635009+0000) 2024-03-06T17:17:32.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:32.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:32.635262+0000) 2024-03-06T17:17:33.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:33.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:33.635520+0000) 2024-03-06T17:17:34.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:34.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:34.635762+0000) 2024-03-06T17:17:35.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:35.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:35.636017+0000) 2024-03-06T17:17:36.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:36.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:36.636257+0000) 2024-03-06T17:17:37.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:37.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:37.636458+0000) 2024-03-06T17:17:38.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:38.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:38.636737+0000) 2024-03-06T17:17:39.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:39.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:39.636975+0000) 2024-03-06T17:17:40.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:40.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:40.637217+0000) 2024-03-06T17:17:41.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:41.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:41.637477+0000) 2024-03-06T17:17:42.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:42.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:42.637737+0000) 2024-03-06T17:17:43.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:43.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:43.637992+0000) 2024-03-06T17:17:44.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:44.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:44.638250+0000) 2024-03-06T17:17:45.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:45.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:45.638491+0000) 2024-03-06T17:17:46.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:46.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:46.638741+0000) 2024-03-06T17:17:46.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:46.670 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:47.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:47.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:47.638990+0000) 2024-03-06T17:17:48.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:48.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:48.639252+0000) 2024-03-06T17:17:49.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:49.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:49.639495+0000) 2024-03-06T17:17:50.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:50.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:50.639732+0000) 2024-03-06T17:17:51.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:51.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:51.639980+0000) 2024-03-06T17:17:52.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:52.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:52.640245+0000) 2024-03-06T17:17:53.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:53.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:53.640474+0000) 2024-03-06T17:17:54.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:54.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:54.640750+0000) 2024-03-06T17:17:55.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:55.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:55.640978+0000) 2024-03-06T17:17:56.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:56.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:56.641299+0000) 2024-03-06T17:17:57.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:57.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:57.641539+0000) 2024-03-06T17:17:58.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:58.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:58.641797+0000) 2024-03-06T17:17:59.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:17:59.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:59.642032+0000) 2024-03-06T17:18:00.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:00.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:00.642295+0000) 2024-03-06T17:18:01.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:01.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:01.642557+0000) 2024-03-06T17:18:02.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:02.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:02.642806+0000) 2024-03-06T17:18:03.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:03.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:03.643082+0000) 2024-03-06T17:18:04.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:04.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:04.643341+0000) 2024-03-06T17:18:05.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:05.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:05.643572+0000) 2024-03-06T17:18:06.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:06.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:06.643835+0000) 2024-03-06T17:18:07.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:07.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:07.644093+0000) 2024-03-06T17:18:08.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:08.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:08.644292+0000) 2024-03-06T17:18:09.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:09.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:09.644480+0000) 2024-03-06T17:18:10.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:10.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:10.644738+0000) 2024-03-06T17:18:11.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:11.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:11.644939+0000) 2024-03-06T17:18:12.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:12.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:12.645193+0000) 2024-03-06T17:18:13.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:13.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:13.645458+0000) 2024-03-06T17:18:14.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:14.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:14.645750+0000) 2024-03-06T17:18:15.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:15.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:15.645990+0000) 2024-03-06T17:18:16.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:16.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:16.646238+0000) 2024-03-06T17:18:16.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:16.806 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:17.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:17.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:17.646486+0000) 2024-03-06T17:18:18.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:18.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:18.646744+0000) 2024-03-06T17:18:19.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:19.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:19.646998+0000) 2024-03-06T17:18:20.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:20.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:20.647231+0000) 2024-03-06T17:18:21.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:21.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:21.647486+0000) 2024-03-06T17:18:22.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:22.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:22.647762+0000) 2024-03-06T17:18:23.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:23.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:23.648025+0000) 2024-03-06T17:18:24.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:24.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:24.648266+0000) 2024-03-06T17:18:25.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:25.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:25.648511+0000) 2024-03-06T17:18:26.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:26.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:26.648772+0000) 2024-03-06T17:18:27.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:27.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:27.649031+0000) 2024-03-06T17:18:28.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:28.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:28.649299+0000) 2024-03-06T17:18:29.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:29.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:29.649492+0000) 2024-03-06T17:18:30.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:30.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:30.649744+0000) 2024-03-06T17:18:31.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:31.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:31.649985+0000) 2024-03-06T17:18:32.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:32.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:32.650250+0000) 2024-03-06T17:18:33.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:33.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:33.650497+0000) 2024-03-06T17:18:34.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:34.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:34.650750+0000) 2024-03-06T17:18:35.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:35.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:35.651010+0000) 2024-03-06T17:18:36.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:36.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:36.651267+0000) 2024-03-06T17:18:37.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:37.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:37.651530+0000) 2024-03-06T17:18:38.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:38.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:38.651795+0000) 2024-03-06T17:18:39.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:39.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:39.651979+0000) 2024-03-06T17:18:40.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:40.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:40.652238+0000) 2024-03-06T17:18:41.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:41.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:41.652492+0000) 2024-03-06T17:18:42.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:42.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:42.652751+0000) 2024-03-06T17:18:43.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:43.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:43.653002+0000) 2024-03-06T17:18:44.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:44.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:44.653271+0000) 2024-03-06T17:18:45.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:45.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:45.653508+0000) 2024-03-06T17:18:46.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:46.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:46.653753+0000) 2024-03-06T17:18:46.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:46.961 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:47.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:47.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:47.653964+0000) 2024-03-06T17:18:48.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:48.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:48.654214+0000) 2024-03-06T17:18:49.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:49.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:49.654518+0000) 2024-03-06T17:18:50.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:50.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:50.654785+0000) 2024-03-06T17:18:51.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:51.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:51.655037+0000) 2024-03-06T17:18:52.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:52.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:52.655298+0000) 2024-03-06T17:18:53.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:53.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:53.655549+0000) 2024-03-06T17:18:54.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:54.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:54.655789+0000) 2024-03-06T17:18:55.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:55.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:55.656046+0000) 2024-03-06T17:18:56.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:56.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:56.656238+0000) 2024-03-06T17:18:57.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:57.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:57.656501+0000) 2024-03-06T17:18:58.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:58.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:58.656770+0000) 2024-03-06T17:18:59.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:18:59.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:59.657030+0000) 2024-03-06T17:19:00.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:00.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:00.657273+0000) 2024-03-06T17:19:01.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:01.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:01.657489+0000) 2024-03-06T17:19:02.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:02.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:02.657763+0000) 2024-03-06T17:19:03.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:03.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:03.658037+0000) 2024-03-06T17:19:04.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:04.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:04.658293+0000) 2024-03-06T17:19:05.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:05.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:05.658550+0000) 2024-03-06T17:19:06.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:06.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:06.658820+0000) 2024-03-06T17:19:07.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:07.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:07.659057+0000) 2024-03-06T17:19:08.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:08.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:08.659300+0000) 2024-03-06T17:19:09.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:09.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:09.659561+0000) 2024-03-06T17:19:10.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:10.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:10.659798+0000) 2024-03-06T17:19:11.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:11.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:11.660062+0000) 2024-03-06T17:19:12.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:12.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:12.660327+0000) 2024-03-06T17:19:13.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:13.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:13.660591+0000) 2024-03-06T17:19:14.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:14.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:14.660821+0000) 2024-03-06T17:19:15.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:15.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:15.661018+0000) 2024-03-06T17:19:16.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:16.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:16.661284+0000) 2024-03-06T17:19:17.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:17.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:17.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:17.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:17.661519+0000) 2024-03-06T17:19:18.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:18.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:18.661772+0000) 2024-03-06T17:19:19.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:19.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:19.662031+0000) 2024-03-06T17:19:20.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:20.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:20.662285+0000) 2024-03-06T17:19:21.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:21.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:21.662566+0000) 2024-03-06T17:19:22.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:22.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:22.662827+0000) 2024-03-06T17:19:23.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:23.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:23.663072+0000) 2024-03-06T17:19:24.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:24.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:24.663316+0000) 2024-03-06T17:19:25.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:25.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:25.663497+0000) 2024-03-06T17:19:26.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:26.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:26.663770+0000) 2024-03-06T17:19:27.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:27.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:27.664031+0000) 2024-03-06T17:19:28.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:28.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:28.664288+0000) 2024-03-06T17:19:29.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:29.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:29.664565+0000) 2024-03-06T17:19:30.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:30.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:30.664823+0000) 2024-03-06T17:19:31.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:31.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:31.665100+0000) 2024-03-06T17:19:32.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:32.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:32.665408+0000) 2024-03-06T17:19:33.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:33.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:33.665730+0000) 2024-03-06T17:19:34.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:34.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:34.665994+0000) 2024-03-06T17:19:35.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:35.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:35.666245+0000) 2024-03-06T17:19:36.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:36.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:36.666515+0000) 2024-03-06T17:19:37.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:37.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:37.666762+0000) 2024-03-06T17:19:38.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:38.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:38.667018+0000) 2024-03-06T17:19:39.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:39.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:39.667271+0000) 2024-03-06T17:19:40.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:40.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:40.667523+0000) 2024-03-06T17:19:41.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:41.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:41.667787+0000) 2024-03-06T17:19:42.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:42.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:42.668048+0000) 2024-03-06T17:19:43.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:43.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:43.668239+0000) 2024-03-06T17:19:44.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:44.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:44.668482+0000) 2024-03-06T17:19:45.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:45.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:45.668739+0000) 2024-03-06T17:19:46.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:46.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:46.668997+0000) 2024-03-06T17:19:47.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:47.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:47.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:47.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:47.669266+0000) 2024-03-06T17:19:48.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:48.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:48.669526+0000) 2024-03-06T17:19:49.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:49.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:49.669757+0000) 2024-03-06T17:19:50.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:50.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:50.670014+0000) 2024-03-06T17:19:51.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:51.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:51.670293+0000) 2024-03-06T17:19:52.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:52.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:52.670576+0000) 2024-03-06T17:19:53.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:53.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:53.670902+0000) 2024-03-06T17:19:54.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:54.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:54.671175+0000) 2024-03-06T17:19:55.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:55.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:55.671446+0000) 2024-03-06T17:19:56.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:56.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:56.671761+0000) 2024-03-06T17:19:57.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:57.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:57.672023+0000) 2024-03-06T17:19:58.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:58.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:58.672307+0000) 2024-03-06T17:19:59.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:19:59.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:59.672560+0000) 2024-03-06T17:20:00.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:00.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:00.672847+0000) 2024-03-06T17:20:01.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:01.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:01.673131+0000) 2024-03-06T17:20:02.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:02.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:02.673398+0000) 2024-03-06T17:20:03.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:03.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:03.673740+0000) 2024-03-06T17:20:04.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:04.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:04.674003+0000) 2024-03-06T17:20:05.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:05.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:05.674259+0000) 2024-03-06T17:20:06.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:06.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:06.674526+0000) 2024-03-06T17:20:07.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:07.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:07.674794+0000) 2024-03-06T17:20:08.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:08.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:08.674995+0000) 2024-03-06T17:20:09.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:09.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:09.675189+0000) 2024-03-06T17:20:10.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:10.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:10.675433+0000) 2024-03-06T17:20:11.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:11.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:11.675618+0000) 2024-03-06T17:20:12.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:12.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:12.675857+0000) 2024-03-06T17:20:13.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:13.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:13.676119+0000) 2024-03-06T17:20:14.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:14.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:14.676367+0000) 2024-03-06T17:20:15.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:15.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:15.676617+0000) 2024-03-06T17:20:16.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:16.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:16.676922+0000) 2024-03-06T17:20:17.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:17.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:17.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:17.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:17.677176+0000) 2024-03-06T17:20:18.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:18.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:18.677418+0000) 2024-03-06T17:20:19.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:19.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:19.677700+0000) 2024-03-06T17:20:20.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:20.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:20.677955+0000) 2024-03-06T17:20:21.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:21.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:21.678214+0000) 2024-03-06T17:20:22.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:22.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:22.678483+0000) 2024-03-06T17:20:23.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:23.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:23.678741+0000) 2024-03-06T17:20:24.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:24.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:24.678991+0000) 2024-03-06T17:20:25.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:25.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:25.679319+0000) 2024-03-06T17:20:26.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:26.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:26.679580+0000) 2024-03-06T17:20:27.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:27.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:27.679837+0000) 2024-03-06T17:20:28.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:28.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:28.680098+0000) 2024-03-06T17:20:29.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:29.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:29.680352+0000) 2024-03-06T17:20:30.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:30.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:30.680609+0000) 2024-03-06T17:20:31.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:31.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:31.680903+0000) 2024-03-06T17:20:32.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:32.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:32.681150+0000) 2024-03-06T17:20:33.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:33.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:33.681429+0000) 2024-03-06T17:20:34.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:34.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:34.681716+0000) 2024-03-06T17:20:35.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:35.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:35.681971+0000) 2024-03-06T17:20:36.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:36.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:36.682231+0000) 2024-03-06T17:20:37.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:37.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:37.682477+0000) 2024-03-06T17:20:38.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:38.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:38.682745+0000) 2024-03-06T17:20:39.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:39.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:39.683010+0000) 2024-03-06T17:20:40.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:40.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:40.683246+0000) 2024-03-06T17:20:41.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:41.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:41.683502+0000) 2024-03-06T17:20:42.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:42.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:42.683752+0000) 2024-03-06T17:20:43.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:43.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:43.684031+0000) 2024-03-06T17:20:44.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:44.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:44.684284+0000) 2024-03-06T17:20:45.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:45.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:45.684521+0000) 2024-03-06T17:20:46.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:46.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:46.684775+0000) 2024-03-06T17:20:47.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:47.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:47.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:47.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:47.685044+0000) 2024-03-06T17:20:48.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:48.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:48.685306+0000) 2024-03-06T17:20:49.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:49.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:49.685557+0000) 2024-03-06T17:20:50.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:50.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:50.685813+0000) 2024-03-06T17:20:51.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:51.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:51.686059+0000) 2024-03-06T17:20:52.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:52.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:52.686320+0000) 2024-03-06T17:20:53.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:53.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:53.686576+0000) 2024-03-06T17:20:54.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:54.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:54.686817+0000) 2024-03-06T17:20:55.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:55.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:55.687068+0000) 2024-03-06T17:20:56.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:56.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:56.687321+0000) 2024-03-06T17:20:57.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:57.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:57.687586+0000) 2024-03-06T17:20:58.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:58.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:58.687747+0000) 2024-03-06T17:20:59.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:20:59.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:59.688003+0000) 2024-03-06T17:21:00.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:00.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:00.688253+0000) 2024-03-06T17:21:01.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:01.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:01.688507+0000) 2024-03-06T17:21:02.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:02.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:02.688766+0000) 2024-03-06T17:21:03.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:03.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:03.689034+0000) 2024-03-06T17:21:04.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:04.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:04.689268+0000) 2024-03-06T17:21:05.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:05.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:05.689512+0000) 2024-03-06T17:21:06.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:06.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:06.689779+0000) 2024-03-06T17:21:07.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:07.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:07.690041+0000) 2024-03-06T17:21:08.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:08.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:08.690233+0000) 2024-03-06T17:21:09.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:09.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:09.690484+0000) 2024-03-06T17:21:10.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:10.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:10.690758+0000) 2024-03-06T17:21:11.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:11.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:11.690995+0000) 2024-03-06T17:21:12.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:12.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:12.691230+0000) 2024-03-06T17:21:13.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:13.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:13.691562+0000) 2024-03-06T17:21:14.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:14.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:14.691796+0000) 2024-03-06T17:21:15.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:15.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:15.692049+0000) 2024-03-06T17:21:16.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:16.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:16.692289+0000) 2024-03-06T17:21:17.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:17.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:17.692530+0000) 2024-03-06T17:21:17.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:17.707 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:18.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:18.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:18.692777+0000) 2024-03-06T17:21:19.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:19.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:19.692957+0000) 2024-03-06T17:21:20.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:20.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:20.693190+0000) 2024-03-06T17:21:21.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:21.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:21.693450+0000) 2024-03-06T17:21:22.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:22.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:22.693750+0000) 2024-03-06T17:21:23.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:23.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:23.694002+0000) 2024-03-06T17:21:24.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:24.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:24.694239+0000) 2024-03-06T17:21:25.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:25.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:25.694491+0000) 2024-03-06T17:21:26.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:26.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:26.694750+0000) 2024-03-06T17:21:27.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:27.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:27.695015+0000) 2024-03-06T17:21:28.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:28.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:28.695276+0000) 2024-03-06T17:21:29.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:29.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:29.695548+0000) 2024-03-06T17:21:30.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:30.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:30.695805+0000) 2024-03-06T17:21:31.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:31.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:31.696064+0000) 2024-03-06T17:21:32.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:32.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:32.696327+0000) 2024-03-06T17:21:33.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:33.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:33.696596+0000) 2024-03-06T17:21:34.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:34.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:34.696856+0000) 2024-03-06T17:21:35.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:35.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:35.697105+0000) 2024-03-06T17:21:36.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:36.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:36.697359+0000) 2024-03-06T17:21:37.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:37.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:37.697625+0000) 2024-03-06T17:21:38.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:38.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:38.697924+0000) 2024-03-06T17:21:39.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:39.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:39.698160+0000) 2024-03-06T17:21:40.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:40.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:40.698403+0000) 2024-03-06T17:21:41.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:41.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:41.698684+0000) 2024-03-06T17:21:42.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:42.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:42.698962+0000) 2024-03-06T17:21:43.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:43.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:43.699231+0000) 2024-03-06T17:21:44.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:44.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:44.699485+0000) 2024-03-06T17:21:45.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:45.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:45.699747+0000) 2024-03-06T17:21:46.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:46.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:46.699985+0000) 2024-03-06T17:21:47.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:47.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:47.700226+0000) 2024-03-06T17:21:47.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:47.852 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:48.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:48.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:48.700468+0000) 2024-03-06T17:21:49.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:49.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:49.700728+0000) 2024-03-06T17:21:50.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:50.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:50.700967+0000) 2024-03-06T17:21:51.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:51.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:51.701183+0000) 2024-03-06T17:21:52.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:52.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:52.701446+0000) 2024-03-06T17:21:53.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:53.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:53.701742+0000) 2024-03-06T17:21:54.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:54.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:54.701994+0000) 2024-03-06T17:21:55.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:55.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:55.702247+0000) 2024-03-06T17:21:56.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:56.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:56.702487+0000) 2024-03-06T17:21:57.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:57.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:57.702747+0000) 2024-03-06T17:21:58.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:58.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:58.703005+0000) 2024-03-06T17:21:59.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:21:59.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:59.703255+0000) 2024-03-06T17:22:00.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:00.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:00.703512+0000) 2024-03-06T17:22:01.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:01.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:01.703779+0000) 2024-03-06T17:22:02.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:02.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:02.704038+0000) 2024-03-06T17:22:03.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:03.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:03.704289+0000) 2024-03-06T17:22:04.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:04.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:04.704471+0000) 2024-03-06T17:22:05.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:05.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:05.704701+0000) 2024-03-06T17:22:06.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:06.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:06.704958+0000) 2024-03-06T17:22:07.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:07.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:07.705221+0000) 2024-03-06T17:22:08.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:08.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:08.705461+0000) 2024-03-06T17:22:09.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:09.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:09.705724+0000) 2024-03-06T17:22:10.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:10.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:10.705969+0000) 2024-03-06T17:22:11.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:11.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:11.706215+0000) 2024-03-06T17:22:12.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:12.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:12.706475+0000) 2024-03-06T17:22:13.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:13.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:13.706749+0000) 2024-03-06T17:22:14.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:14.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:14.707002+0000) 2024-03-06T17:22:15.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:15.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:15.707251+0000) 2024-03-06T17:22:16.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:16.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:16.707507+0000) 2024-03-06T17:22:17.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:17.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:17.707775+0000) 2024-03-06T17:22:18.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:18.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:18.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:18.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:18.708038+0000) 2024-03-06T17:22:19.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:19.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:19.708279+0000) 2024-03-06T17:22:20.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:20.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:20.708534+0000) 2024-03-06T17:22:21.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:21.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:21.708787+0000) 2024-03-06T17:22:22.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:22.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:22.709064+0000) 2024-03-06T17:22:23.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:23.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:23.709265+0000) 2024-03-06T17:22:24.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:24.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:24.709505+0000) 2024-03-06T17:22:25.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:25.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:25.709747+0000) 2024-03-06T17:22:26.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:26.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:26.710005+0000) 2024-03-06T17:22:27.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:27.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:27.710301+0000) 2024-03-06T17:22:28.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:28.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:28.710578+0000) 2024-03-06T17:22:29.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:29.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:29.710883+0000) 2024-03-06T17:22:30.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:30.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:30.711195+0000) 2024-03-06T17:22:31.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:31.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:31.711488+0000) 2024-03-06T17:22:32.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:32.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:32.711795+0000) 2024-03-06T17:22:33.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:33.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:33.712099+0000) 2024-03-06T17:22:34.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:34.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:34.712415+0000) 2024-03-06T17:22:35.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:35.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:35.712749+0000) 2024-03-06T17:22:36.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:36.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:36.713039+0000) 2024-03-06T17:22:37.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:37.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:37.713329+0000) 2024-03-06T17:22:38.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:38.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:38.713693+0000) 2024-03-06T17:22:39.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:39.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:39.713986+0000) 2024-03-06T17:22:40.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:40.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:40.714296+0000) 2024-03-06T17:22:41.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:41.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:41.714581+0000) 2024-03-06T17:22:42.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:42.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:42.714836+0000) 2024-03-06T17:22:43.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:43.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:43.715050+0000) 2024-03-06T17:22:44.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:44.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:44.715304+0000) 2024-03-06T17:22:45.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:45.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:45.715559+0000) 2024-03-06T17:22:46.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:46.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:46.715818+0000) 2024-03-06T17:22:47.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:47.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:47.716077+0000) 2024-03-06T17:22:48.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:48.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:48.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:48.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:48.716322+0000) 2024-03-06T17:22:49.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:49.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:49.716565+0000) 2024-03-06T17:22:50.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:50.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:50.716803+0000) 2024-03-06T17:22:51.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:51.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:51.717042+0000) 2024-03-06T17:22:52.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:52.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:52.717306+0000) 2024-03-06T17:22:53.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:53.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:53.717557+0000) 2024-03-06T17:22:54.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:54.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:54.717816+0000) 2024-03-06T17:22:55.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:55.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:55.718071+0000) 2024-03-06T17:22:56.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:56.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:56.718331+0000) 2024-03-06T17:22:57.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:57.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:57.718599+0000) 2024-03-06T17:22:58.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:58.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:58.718898+0000) 2024-03-06T17:22:59.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:22:59.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:59.719157+0000) 2024-03-06T17:23:00.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:00.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:00.719412+0000) 2024-03-06T17:23:01.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:01.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:01.719708+0000) 2024-03-06T17:23:02.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:02.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:02.719974+0000) 2024-03-06T17:23:03.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:03.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:03.720253+0000) 2024-03-06T17:23:04.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:04.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:04.720489+0000) 2024-03-06T17:23:05.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:05.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:05.720764+0000) 2024-03-06T17:23:06.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:06.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:06.721030+0000) 2024-03-06T17:23:07.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:07.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:07.721285+0000) 2024-03-06T17:23:08.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:08.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:08.721544+0000) 2024-03-06T17:23:09.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:09.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:09.721795+0000) 2024-03-06T17:23:10.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:10.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:10.722052+0000) 2024-03-06T17:23:11.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:11.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:11.722314+0000) 2024-03-06T17:23:12.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:12.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:12.722571+0000) 2024-03-06T17:23:13.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:13.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:13.722852+0000) 2024-03-06T17:23:14.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:14.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:14.723041+0000) 2024-03-06T17:23:15.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:15.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:15.723295+0000) 2024-03-06T17:23:16.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:16.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:16.723551+0000) 2024-03-06T17:23:17.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:17.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:17.723749+0000) 2024-03-06T17:23:18.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:18.420 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:18.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:18.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:18.723995+0000) 2024-03-06T17:23:19.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:19.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:19.724244+0000) 2024-03-06T17:23:20.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:20.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:20.724505+0000) 2024-03-06T17:23:21.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:21.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:21.724748+0000) 2024-03-06T17:23:22.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:22.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:22.725004+0000) 2024-03-06T17:23:23.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:23.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:23.725212+0000) 2024-03-06T17:23:24.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:24.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:24.725446+0000) 2024-03-06T17:23:25.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:25.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:25.725736+0000) 2024-03-06T17:23:26.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:26.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:26.726002+0000) 2024-03-06T17:23:27.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:27.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:27.726250+0000) 2024-03-06T17:23:28.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:28.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:28.726446+0000) 2024-03-06T17:23:29.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:29.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:29.726777+0000) 2024-03-06T17:23:30.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:30.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:30.727033+0000) 2024-03-06T17:23:31.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:31.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:31.727272+0000) 2024-03-06T17:23:32.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:32.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:32.727536+0000) 2024-03-06T17:23:33.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:33.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:33.727788+0000) 2024-03-06T17:23:34.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:34.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:34.728025+0000) 2024-03-06T17:23:35.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:35.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:35.728272+0000) 2024-03-06T17:23:36.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:36.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:36.728532+0000) 2024-03-06T17:23:37.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:37.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:37.728817+0000) 2024-03-06T17:23:38.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:38.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:38.729090+0000) 2024-03-06T17:23:39.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:39.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:39.729343+0000) 2024-03-06T17:23:40.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:40.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:40.729589+0000) 2024-03-06T17:23:41.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:41.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:41.729846+0000) 2024-03-06T17:23:42.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:42.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:42.730116+0000) 2024-03-06T17:23:43.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:43.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:43.730316+0000) 2024-03-06T17:23:44.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:44.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:44.730552+0000) 2024-03-06T17:23:45.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:45.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:45.730814+0000) 2024-03-06T17:23:46.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:46.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:46.731012+0000) 2024-03-06T17:23:47.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:47.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:47.731277+0000) 2024-03-06T17:23:48.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:48.566 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:48.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:48.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:48.731616+0000) 2024-03-06T17:23:49.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:49.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:49.731915+0000) 2024-03-06T17:23:50.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:50.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:50.732173+0000) 2024-03-06T17:23:51.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:51.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:51.732436+0000) 2024-03-06T17:23:52.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:52.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:52.732741+0000) 2024-03-06T17:23:53.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:53.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:53.732957+0000) 2024-03-06T17:23:54.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:54.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:54.733143+0000) 2024-03-06T17:23:55.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:55.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:55.733384+0000) 2024-03-06T17:23:56.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:56.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:56.733671+0000) 2024-03-06T17:23:57.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:57.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:57.733944+0000) 2024-03-06T17:23:58.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:58.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:58.734220+0000) 2024-03-06T17:23:59.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:23:59.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:59.734478+0000) 2024-03-06T17:24:00.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:00.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:00.734741+0000) 2024-03-06T17:24:01.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:01.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:01.734956+0000) 2024-03-06T17:24:02.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:02.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:02.735205+0000) 2024-03-06T17:24:03.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:03.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:03.735419+0000) 2024-03-06T17:24:04.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:04.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:04.735713+0000) 2024-03-06T17:24:05.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:05.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:05.735927+0000) 2024-03-06T17:24:06.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:06.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:06.736173+0000) 2024-03-06T17:24:07.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:07.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:07.736329+0000) 2024-03-06T17:24:08.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:08.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:08.736576+0000) 2024-03-06T17:24:09.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:09.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:09.736831+0000) 2024-03-06T17:24:10.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:10.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:10.737078+0000) 2024-03-06T17:24:11.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:11.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:11.737311+0000) 2024-03-06T17:24:12.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:12.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:12.737575+0000) 2024-03-06T17:24:13.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:13.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:13.737842+0000) 2024-03-06T17:24:14.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:14.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:14.738094+0000) 2024-03-06T17:24:15.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:15.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:15.738355+0000) 2024-03-06T17:24:16.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:16.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:16.738617+0000) 2024-03-06T17:24:17.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:17.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:17.738924+0000) 2024-03-06T17:24:18.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:18.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:18.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:18.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:18.739118+0000) 2024-03-06T17:24:19.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:19.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:19.739369+0000) 2024-03-06T17:24:20.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:20.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:20.739620+0000) 2024-03-06T17:24:21.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:21.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:21.739884+0000) 2024-03-06T17:24:22.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:22.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:22.740108+0000) 2024-03-06T17:24:23.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:23.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:23.740428+0000) 2024-03-06T17:24:24.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:24.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:24.740769+0000) 2024-03-06T17:24:25.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:25.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:25.741081+0000) 2024-03-06T17:24:26.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:26.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:26.741374+0000) 2024-03-06T17:24:27.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:27.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:27.741706+0000) 2024-03-06T17:24:28.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:28.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:28.742021+0000) 2024-03-06T17:24:29.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:29.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:29.742316+0000) 2024-03-06T17:24:30.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:30.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:30.742732+0000) 2024-03-06T17:24:31.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:31.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:31.743029+0000) 2024-03-06T17:24:32.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:32.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:32.743284+0000) 2024-03-06T17:24:33.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:33.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:33.743589+0000) 2024-03-06T17:24:34.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:34.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:34.743923+0000) 2024-03-06T17:24:35.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:35.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:35.744238+0000) 2024-03-06T17:24:36.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:36.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:36.744538+0000) 2024-03-06T17:24:37.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:37.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:37.744847+0000) 2024-03-06T17:24:38.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:38.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:38.745138+0000) 2024-03-06T17:24:39.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:39.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:39.745445+0000) 2024-03-06T17:24:40.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:40.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:40.745783+0000) 2024-03-06T17:24:41.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:41.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:41.746077+0000) 2024-03-06T17:24:42.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:42.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:42.746395+0000) 2024-03-06T17:24:43.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:43.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:43.746748+0000) 2024-03-06T17:24:44.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:44.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:44.747057+0000) 2024-03-06T17:24:45.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:45.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:45.747345+0000) 2024-03-06T17:24:46.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:46.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:46.747683+0000) 2024-03-06T17:24:47.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:47.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:47.747990+0000) 2024-03-06T17:24:48.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:48.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:48.748283+0000) 2024-03-06T17:24:48.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:48.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:49.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:49.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:49.748590+0000) 2024-03-06T17:24:50.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:50.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:50.748949+0000) 2024-03-06T17:24:51.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:51.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:51.749246+0000) 2024-03-06T17:24:52.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:52.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:52.749543+0000) 2024-03-06T17:24:53.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:53.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:53.749859+0000) 2024-03-06T17:24:54.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:54.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:54.750159+0000) 2024-03-06T17:24:55.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:55.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:55.750455+0000) 2024-03-06T17:24:56.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:56.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:56.750777+0000) 2024-03-06T17:24:57.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:57.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:57.751076+0000) 2024-03-06T17:24:58.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:58.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:58.751362+0000) 2024-03-06T17:24:59.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:24:59.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:59.751668+0000) 2024-03-06T17:25:00.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:00.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:00.751920+0000) 2024-03-06T17:25:01.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:01.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:01.752200+0000) 2024-03-06T17:25:02.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:02.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:02.752440+0000) 2024-03-06T17:25:03.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:03.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:03.752731+0000) 2024-03-06T17:25:04.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:04.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:04.752988+0000) 2024-03-06T17:25:05.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:05.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:05.753243+0000) 2024-03-06T17:25:06.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:06.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:06.753496+0000) 2024-03-06T17:25:07.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:07.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:07.753756+0000) 2024-03-06T17:25:08.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:08.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:08.754039+0000) 2024-03-06T17:25:09.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:09.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:09.754289+0000) 2024-03-06T17:25:10.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:10.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:10.754537+0000) 2024-03-06T17:25:11.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:11.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:11.754795+0000) 2024-03-06T17:25:12.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:12.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:12.755053+0000) 2024-03-06T17:25:13.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:13.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:13.755307+0000) 2024-03-06T17:25:14.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:14.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:14.755512+0000) 2024-03-06T17:25:15.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:15.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:15.755766+0000) 2024-03-06T17:25:16.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:16.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:16.756003+0000) 2024-03-06T17:25:17.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:17.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:17.756258+0000) 2024-03-06T17:25:18.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:18.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:18.756502+0000) 2024-03-06T17:25:18.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:19.007 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:19.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:19.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:19.756739+0000) 2024-03-06T17:25:20.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:20.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:20.757000+0000) 2024-03-06T17:25:21.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:21.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:21.757257+0000) 2024-03-06T17:25:22.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:22.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:22.757505+0000) 2024-03-06T17:25:23.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:23.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:23.757752+0000) 2024-03-06T17:25:24.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:24.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:24.758043+0000) 2024-03-06T17:25:25.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:25.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:25.758290+0000) 2024-03-06T17:25:26.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:26.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:26.758532+0000) 2024-03-06T17:25:27.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:27.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:27.758787+0000) 2024-03-06T17:25:28.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:28.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:28.759033+0000) 2024-03-06T17:25:29.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:29.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:29.759292+0000) 2024-03-06T17:25:30.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:30.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:30.759547+0000) 2024-03-06T17:25:31.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:31.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:31.759812+0000) 2024-03-06T17:25:32.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:32.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:32.760049+0000) 2024-03-06T17:25:33.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:33.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:33.760318+0000) 2024-03-06T17:25:34.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:34.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:34.760559+0000) 2024-03-06T17:25:35.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:35.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:35.760808+0000) 2024-03-06T17:25:36.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:36.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:36.761045+0000) 2024-03-06T17:25:37.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:37.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:37.761307+0000) 2024-03-06T17:25:38.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:38.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:38.761567+0000) 2024-03-06T17:25:39.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:39.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:39.761818+0000) 2024-03-06T17:25:40.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:40.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:40.762071+0000) 2024-03-06T17:25:41.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:41.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:41.762308+0000) 2024-03-06T17:25:42.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:42.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:42.762562+0000) 2024-03-06T17:25:43.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:43.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:43.762822+0000) 2024-03-06T17:25:44.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:44.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:44.763078+0000) 2024-03-06T17:25:45.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:45.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:45.763320+0000) 2024-03-06T17:25:46.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:46.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:46.763507+0000) 2024-03-06T17:25:47.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:47.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:47.763758+0000) 2024-03-06T17:25:48.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:48.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:48.763956+0000) 2024-03-06T17:25:49.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:49.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:49.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:49.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:49.764152+0000) 2024-03-06T17:25:50.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:50.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:50.764389+0000) 2024-03-06T17:25:51.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:51.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:51.764666+0000) 2024-03-06T17:25:52.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:52.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:52.764925+0000) 2024-03-06T17:25:53.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:53.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:53.765180+0000) 2024-03-06T17:25:54.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:54.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:54.765418+0000) 2024-03-06T17:25:55.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:55.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:55.765697+0000) 2024-03-06T17:25:56.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:56.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:56.765841+0000) 2024-03-06T17:25:57.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:57.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:57.766083+0000) 2024-03-06T17:25:58.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:58.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:58.766343+0000) 2024-03-06T17:25:59.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:25:59.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:59.766595+0000) 2024-03-06T17:26:00.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:00.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:00.766843+0000) 2024-03-06T17:26:01.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:01.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:01.767078+0000) 2024-03-06T17:26:02.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:02.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:02.767326+0000) 2024-03-06T17:26:03.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:03.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:03.767575+0000) 2024-03-06T17:26:04.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:04.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:04.767811+0000) 2024-03-06T17:26:05.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:05.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:05.768066+0000) 2024-03-06T17:26:06.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:06.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:06.768330+0000) 2024-03-06T17:26:07.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:07.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:07.768571+0000) 2024-03-06T17:26:08.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:08.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:08.768819+0000) 2024-03-06T17:26:09.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:09.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:09.769049+0000) 2024-03-06T17:26:10.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:10.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:10.769308+0000) 2024-03-06T17:26:11.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:11.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:11.769573+0000) 2024-03-06T17:26:12.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:12.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:12.769837+0000) 2024-03-06T17:26:13.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:13.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:13.770111+0000) 2024-03-06T17:26:14.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:14.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:14.770361+0000) 2024-03-06T17:26:15.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:15.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:15.770599+0000) 2024-03-06T17:26:16.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:16.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:16.770844+0000) 2024-03-06T17:26:17.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:17.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:17.771157+0000) 2024-03-06T17:26:18.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:18.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:18.771372+0000) 2024-03-06T17:26:19.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:19.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:19.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:19.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:19.771662+0000) 2024-03-06T17:26:20.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:20.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:20.771926+0000) 2024-03-06T17:26:21.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:21.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:21.772158+0000) 2024-03-06T17:26:22.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:22.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:22.772347+0000) 2024-03-06T17:26:23.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:23.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:23.772598+0000) 2024-03-06T17:26:24.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:24.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:24.772767+0000) 2024-03-06T17:26:25.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:25.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:25.773017+0000) 2024-03-06T17:26:26.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:26.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:26.773259+0000) 2024-03-06T17:26:27.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:27.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:27.773526+0000) 2024-03-06T17:26:28.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:28.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:28.773789+0000) 2024-03-06T17:26:29.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:29.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:29.774042+0000) 2024-03-06T17:26:30.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:30.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:30.774309+0000) 2024-03-06T17:26:31.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:31.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:31.774491+0000) 2024-03-06T17:26:32.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:32.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:32.774803+0000) 2024-03-06T17:26:33.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:33.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:33.775062+0000) 2024-03-06T17:26:34.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:34.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:34.775320+0000) 2024-03-06T17:26:35.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:35.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:35.775562+0000) 2024-03-06T17:26:36.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:36.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:36.775805+0000) 2024-03-06T17:26:37.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:37.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:37.776038+0000) 2024-03-06T17:26:38.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:38.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:38.776290+0000) 2024-03-06T17:26:39.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:39.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:39.776544+0000) 2024-03-06T17:26:40.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:40.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:40.776807+0000) 2024-03-06T17:26:41.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:41.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:41.777043+0000) 2024-03-06T17:26:42.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:42.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:42.777288+0000) 2024-03-06T17:26:43.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:43.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:43.777568+0000) 2024-03-06T17:26:44.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:44.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:44.777803+0000) 2024-03-06T17:26:45.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:45.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:45.778056+0000) 2024-03-06T17:26:46.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:46.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:46.778312+0000) 2024-03-06T17:26:47.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:47.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:47.778548+0000) 2024-03-06T17:26:48.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:48.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:48.778834+0000) 2024-03-06T17:26:49.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:49.482 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:49.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:49.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:49.779096+0000) 2024-03-06T17:26:50.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:50.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:50.779364+0000) 2024-03-06T17:26:51.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:51.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:51.779612+0000) 2024-03-06T17:26:52.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:52.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:52.779895+0000) 2024-03-06T17:26:53.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:53.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:53.780155+0000) 2024-03-06T17:26:54.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:54.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:54.780401+0000) 2024-03-06T17:26:55.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:55.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:55.780688+0000) 2024-03-06T17:26:56.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:56.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:56.780949+0000) 2024-03-06T17:26:57.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:57.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:57.781188+0000) 2024-03-06T17:26:58.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:58.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:58.781465+0000) 2024-03-06T17:26:59.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:26:59.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:59.781732+0000) 2024-03-06T17:27:00.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:00.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:00.781995+0000) 2024-03-06T17:27:01.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:01.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:01.782254+0000) 2024-03-06T17:27:02.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:02.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:02.782508+0000) 2024-03-06T17:27:03.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:03.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:03.782768+0000) 2024-03-06T17:27:04.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:04.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:04.783025+0000) 2024-03-06T17:27:05.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:05.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:05.783283+0000) 2024-03-06T17:27:06.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:06.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:06.783550+0000) 2024-03-06T17:27:07.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:07.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:07.783810+0000) 2024-03-06T17:27:08.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:08.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:08.784051+0000) 2024-03-06T17:27:09.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:09.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:09.784309+0000) 2024-03-06T17:27:10.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:10.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:10.784622+0000) 2024-03-06T17:27:11.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:11.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:11.784906+0000) 2024-03-06T17:27:12.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:12.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:12.785167+0000) 2024-03-06T17:27:13.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:13.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:13.785436+0000) 2024-03-06T17:27:14.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:14.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:14.785738+0000) 2024-03-06T17:27:15.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:15.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:15.785989+0000) 2024-03-06T17:27:16.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:16.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:16.786248+0000) 2024-03-06T17:27:17.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:17.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:17.786502+0000) 2024-03-06T17:27:18.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:18.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:18.786750+0000) 2024-03-06T17:27:19.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:19.629 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:19.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:19.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:19.786971+0000) 2024-03-06T17:27:20.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:20.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:20.787209+0000) 2024-03-06T17:27:21.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:21.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:21.787450+0000) 2024-03-06T17:27:22.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:22.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:22.787736+0000) 2024-03-06T17:27:23.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:23.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:23.788000+0000) 2024-03-06T17:27:24.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:24.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:24.788189+0000) 2024-03-06T17:27:25.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:25.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:25.788446+0000) 2024-03-06T17:27:26.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:26.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:26.788717+0000) 2024-03-06T17:27:27.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:27.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:27.788975+0000) 2024-03-06T17:27:28.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:28.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:28.789181+0000) 2024-03-06T17:27:29.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:29.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:29.789435+0000) 2024-03-06T17:27:30.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:30.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:30.789730+0000) 2024-03-06T17:27:31.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:31.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:31.789964+0000) 2024-03-06T17:27:32.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:32.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:32.790224+0000) 2024-03-06T17:27:33.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:33.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:33.790423+0000) 2024-03-06T17:27:34.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:34.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:34.790709+0000) 2024-03-06T17:27:35.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:35.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:35.790964+0000) 2024-03-06T17:27:36.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:36.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:36.791223+0000) 2024-03-06T17:27:37.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:37.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:37.791470+0000) 2024-03-06T17:27:38.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:38.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:38.791752+0000) 2024-03-06T17:27:39.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:39.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:39.792008+0000) 2024-03-06T17:27:40.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:40.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:40.792265+0000) 2024-03-06T17:27:41.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:41.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:41.792516+0000) 2024-03-06T17:27:42.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:42.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:42.792788+0000) 2024-03-06T17:27:43.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:43.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:43.793059+0000) 2024-03-06T17:27:44.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:44.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:44.793315+0000) 2024-03-06T17:27:45.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:45.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:45.793518+0000) 2024-03-06T17:27:46.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:46.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:46.793794+0000) 2024-03-06T17:27:47.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:47.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:47.794059+0000) 2024-03-06T17:27:48.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:48.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:48.794315+0000) 2024-03-06T17:27:49.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:49.777 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:49.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:49.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:49.794595+0000) 2024-03-06T17:27:50.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:50.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:50.794917+0000) 2024-03-06T17:27:51.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:51.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:51.795162+0000) 2024-03-06T17:27:52.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:52.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:52.795415+0000) 2024-03-06T17:27:53.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:53.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:53.795735+0000) 2024-03-06T17:27:54.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:54.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:54.796049+0000) 2024-03-06T17:27:55.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:55.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:55.796368+0000) 2024-03-06T17:27:56.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:56.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:56.796737+0000) 2024-03-06T17:27:57.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:57.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:57.797049+0000) 2024-03-06T17:27:58.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:58.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:58.797417+0000) 2024-03-06T17:27:59.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:27:59.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:59.797679+0000) 2024-03-06T17:28:00.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:00.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:00.798003+0000) 2024-03-06T17:28:01.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:01.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:01.798331+0000) 2024-03-06T17:28:02.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:02.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:02.798670+0000) 2024-03-06T17:28:03.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:03.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:03.799022+0000) 2024-03-06T17:28:04.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:04.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:04.799265+0000) 2024-03-06T17:28:05.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:05.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:05.799498+0000) 2024-03-06T17:28:06.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:06.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:06.799821+0000) 2024-03-06T17:28:07.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:07.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:07.800055+0000) 2024-03-06T17:28:08.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:08.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:08.800306+0000) 2024-03-06T17:28:09.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:09.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:09.800539+0000) 2024-03-06T17:28:10.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:10.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:10.800802+0000) 2024-03-06T17:28:11.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:11.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:11.801040+0000) 2024-03-06T17:28:12.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:12.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:12.801324+0000) 2024-03-06T17:28:13.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:13.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:13.801697+0000) 2024-03-06T17:28:14.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:14.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:14.801993+0000) 2024-03-06T17:28:15.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:15.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:15.802280+0000) 2024-03-06T17:28:16.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:16.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:16.802569+0000) 2024-03-06T17:28:17.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:17.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:17.802872+0000) 2024-03-06T17:28:18.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:18.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:18.803188+0000) 2024-03-06T17:28:19.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:19.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:19.803489+0000) 2024-03-06T17:28:19.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:19.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:20.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:20.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:20.803791+0000) 2024-03-06T17:28:21.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:21.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:21.804092+0000) 2024-03-06T17:28:22.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:22.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:22.804397+0000) 2024-03-06T17:28:23.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:23.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:23.804775+0000) 2024-03-06T17:28:24.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:24.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:24.805065+0000) 2024-03-06T17:28:25.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:25.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:25.805372+0000) 2024-03-06T17:28:26.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:26.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:26.805725+0000) 2024-03-06T17:28:27.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:27.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:27.806037+0000) 2024-03-06T17:28:28.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:28.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:28.806329+0000) 2024-03-06T17:28:29.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:29.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:29.806673+0000) 2024-03-06T17:28:30.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:30.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:30.806971+0000) 2024-03-06T17:28:31.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:31.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:31.807312+0000) 2024-03-06T17:28:32.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:32.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:32.807616+0000) 2024-03-06T17:28:33.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:33.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:33.807973+0000) 2024-03-06T17:28:34.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:34.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:34.808230+0000) 2024-03-06T17:28:35.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:35.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:35.808493+0000) 2024-03-06T17:28:36.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:36.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:36.808746+0000) 2024-03-06T17:28:37.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:37.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:37.808999+0000) 2024-03-06T17:28:38.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:38.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:38.809245+0000) 2024-03-06T17:28:39.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:39.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:39.809485+0000) 2024-03-06T17:28:40.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:40.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:40.809747+0000) 2024-03-06T17:28:41.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:41.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:41.810036+0000) 2024-03-06T17:28:42.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:42.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:42.810276+0000) 2024-03-06T17:28:43.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:43.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:43.810555+0000) 2024-03-06T17:28:44.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:44.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:44.810806+0000) 2024-03-06T17:28:45.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:45.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:45.811040+0000) 2024-03-06T17:28:46.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:46.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:46.811288+0000) 2024-03-06T17:28:47.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:47.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:47.811548+0000) 2024-03-06T17:28:48.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:48.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:48.811810+0000) 2024-03-06T17:28:49.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:49.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:49.811999+0000) 2024-03-06T17:28:50.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:50.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:50.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:50.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:50.812200+0000) 2024-03-06T17:28:51.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:51.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:51.812467+0000) 2024-03-06T17:28:52.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:52.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:52.812754+0000) 2024-03-06T17:28:53.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:53.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:53.813030+0000) 2024-03-06T17:28:54.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:54.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:54.813275+0000) 2024-03-06T17:28:55.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:55.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:55.813528+0000) 2024-03-06T17:28:56.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:56.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:56.813785+0000) 2024-03-06T17:28:57.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:57.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:57.814062+0000) 2024-03-06T17:28:58.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:58.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:58.814313+0000) 2024-03-06T17:28:59.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:28:59.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:59.814545+0000) 2024-03-06T17:29:00.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:00.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:00.814798+0000) 2024-03-06T17:29:01.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:01.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:01.815066+0000) 2024-03-06T17:29:02.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:02.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:02.815366+0000) 2024-03-06T17:29:03.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:03.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:03.815721+0000) 2024-03-06T17:29:04.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:04.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:04.816005+0000) 2024-03-06T17:29:05.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:05.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:05.816279+0000) 2024-03-06T17:29:06.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:06.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:06.816593+0000) 2024-03-06T17:29:07.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:07.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:07.816962+0000) 2024-03-06T17:29:08.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:08.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:08.817271+0000) 2024-03-06T17:29:09.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:09.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:09.817563+0000) 2024-03-06T17:29:10.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:10.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:10.817863+0000) 2024-03-06T17:29:11.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:11.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:11.818161+0000) 2024-03-06T17:29:12.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:12.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:12.818449+0000) 2024-03-06T17:29:13.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:13.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:13.818782+0000) 2024-03-06T17:29:14.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:14.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:14.819070+0000) 2024-03-06T17:29:15.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:15.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:15.819384+0000) 2024-03-06T17:29:16.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:16.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:16.819757+0000) 2024-03-06T17:29:17.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:17.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:17.820052+0000) 2024-03-06T17:29:18.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:18.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:18.820367+0000) 2024-03-06T17:29:19.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:19.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:19.820664+0000) 2024-03-06T17:29:20.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:20.226 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:20.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:20.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:20.820936+0000) 2024-03-06T17:29:21.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:21.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:21.821269+0000) 2024-03-06T17:29:22.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:22.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:22.821581+0000) 2024-03-06T17:29:23.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:23.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:23.821951+0000) 2024-03-06T17:29:24.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:24.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:24.822270+0000) 2024-03-06T17:29:25.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:25.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:25.822592+0000) 2024-03-06T17:29:26.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:26.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:26.822952+0000) 2024-03-06T17:29:27.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:27.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:27.823252+0000) 2024-03-06T17:29:28.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:28.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:28.823546+0000) 2024-03-06T17:29:29.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:29.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:29.823848+0000) 2024-03-06T17:29:30.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:30.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:30.824160+0000) 2024-03-06T17:29:31.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:31.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:31.824476+0000) 2024-03-06T17:29:32.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:32.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:32.824790+0000) 2024-03-06T17:29:33.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:33.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:33.825104+0000) 2024-03-06T17:29:34.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:34.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:34.825421+0000) 2024-03-06T17:29:35.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:35.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:35.825772+0000) 2024-03-06T17:29:36.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:36.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:36.826087+0000) 2024-03-06T17:29:37.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:37.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:37.826389+0000) 2024-03-06T17:29:38.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:38.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:38.826725+0000) 2024-03-06T17:29:39.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:39.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:39.827024+0000) 2024-03-06T17:29:40.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:40.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:40.827329+0000) 2024-03-06T17:29:41.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:41.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:41.827621+0000) 2024-03-06T17:29:42.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:42.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:42.827997+0000) 2024-03-06T17:29:43.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:43.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:43.828327+0000) 2024-03-06T17:29:44.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:44.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:44.828665+0000) 2024-03-06T17:29:45.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:45.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:45.828985+0000) 2024-03-06T17:29:46.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:46.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:46.829295+0000) 2024-03-06T17:29:47.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:47.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:47.829613+0000) 2024-03-06T17:29:48.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:48.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:48.829948+0000) 2024-03-06T17:29:49.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:49.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:49.830205+0000) 2024-03-06T17:29:50.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:50.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:50.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:50.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:50.830470+0000) 2024-03-06T17:29:51.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:51.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:51.830756+0000) 2024-03-06T17:29:52.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:52.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:52.830993+0000) 2024-03-06T17:29:53.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:53.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:53.831203+0000) 2024-03-06T17:29:54.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:54.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:54.831461+0000) 2024-03-06T17:29:55.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:55.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:55.831751+0000) 2024-03-06T17:29:56.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:56.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:56.832010+0000) 2024-03-06T17:29:57.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:57.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:57.832267+0000) 2024-03-06T17:29:58.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:58.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:58.832507+0000) 2024-03-06T17:29:59.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:29:59.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:59.832752+0000) 2024-03-06T17:30:00.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:00.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:00.832997+0000) 2024-03-06T17:30:01.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:01.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:01.833254+0000) 2024-03-06T17:30:02.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:02.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:02.833514+0000) 2024-03-06T17:30:03.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:03.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:03.833765+0000) 2024-03-06T17:30:04.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:04.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:04.834010+0000) 2024-03-06T17:30:05.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:05.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:05.834263+0000) 2024-03-06T17:30:06.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:06.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:06.834522+0000) 2024-03-06T17:30:07.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:07.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:07.834812+0000) 2024-03-06T17:30:08.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:08.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:08.835052+0000) 2024-03-06T17:30:09.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:09.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:09.835286+0000) 2024-03-06T17:30:10.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:10.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:10.835528+0000) 2024-03-06T17:30:11.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:11.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:11.835794+0000) 2024-03-06T17:30:12.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:12.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:12.836077+0000) 2024-03-06T17:30:13.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:13.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:13.836319+0000) 2024-03-06T17:30:14.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:14.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:14.836685+0000) 2024-03-06T17:30:15.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:15.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:15.836918+0000) 2024-03-06T17:30:16.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:16.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:16.837216+0000) 2024-03-06T17:30:17.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:17.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:17.837420+0000) 2024-03-06T17:30:18.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:18.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:18.837707+0000) 2024-03-06T17:30:19.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:19.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:19.837899+0000) 2024-03-06T17:30:20.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:20.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:20.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:20.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:20.838143+0000) 2024-03-06T17:30:21.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:21.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:21.838403+0000) 2024-03-06T17:30:22.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:22.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:22.838699+0000) 2024-03-06T17:30:23.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:23.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:23.838947+0000) 2024-03-06T17:30:24.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:24.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:24.839200+0000) 2024-03-06T17:30:25.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:25.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:25.839458+0000) 2024-03-06T17:30:26.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:26.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:26.839755+0000) 2024-03-06T17:30:27.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:27.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:27.839996+0000) 2024-03-06T17:30:28.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:28.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:28.840297+0000) 2024-03-06T17:30:29.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:29.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:29.840546+0000) 2024-03-06T17:30:30.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:30.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:30.840789+0000) 2024-03-06T17:30:31.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:31.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:31.841043+0000) 2024-03-06T17:30:32.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:32.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:32.841258+0000) 2024-03-06T17:30:33.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:33.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:33.841506+0000) 2024-03-06T17:30:34.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:34.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:34.841757+0000) 2024-03-06T17:30:35.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:35.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:35.841997+0000) 2024-03-06T17:30:36.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:36.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:36.842189+0000) 2024-03-06T17:30:37.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:37.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:37.842452+0000) 2024-03-06T17:30:38.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:38.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:38.842753+0000) 2024-03-06T17:30:39.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:39.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:39.843005+0000) 2024-03-06T17:30:40.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:40.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:40.843219+0000) 2024-03-06T17:30:41.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:41.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:41.843480+0000) 2024-03-06T17:30:42.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:42.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:42.843728+0000) 2024-03-06T17:30:43.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:43.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:43.843994+0000) 2024-03-06T17:30:44.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:44.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:44.844260+0000) 2024-03-06T17:30:45.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:45.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:45.844521+0000) 2024-03-06T17:30:46.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:46.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:46.844779+0000) 2024-03-06T17:30:47.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:47.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:47.845035+0000) 2024-03-06T17:30:48.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:48.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:48.845282+0000) 2024-03-06T17:30:49.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:49.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:49.845541+0000) 2024-03-06T17:30:50.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:50.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:50.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:50.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:50.846001+0000) 2024-03-06T17:30:51.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:51.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:51.846234+0000) 2024-03-06T17:30:52.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:52.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:52.846474+0000) 2024-03-06T17:30:53.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:53.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:53.846741+0000) 2024-03-06T17:30:54.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:54.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:54.846985+0000) 2024-03-06T17:30:55.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:55.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:55.847227+0000) 2024-03-06T17:30:56.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:56.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:56.847468+0000) 2024-03-06T17:30:57.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:57.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:57.847759+0000) 2024-03-06T17:30:58.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:58.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:58.848011+0000) 2024-03-06T17:30:59.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:30:59.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:59.848250+0000) 2024-03-06T17:31:00.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:00.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:00.848492+0000) 2024-03-06T17:31:01.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:01.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:01.848749+0000) 2024-03-06T17:31:02.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:02.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:02.848989+0000) 2024-03-06T17:31:03.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:03.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:03.849257+0000) 2024-03-06T17:31:04.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:04.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:04.849514+0000) 2024-03-06T17:31:05.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:05.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:05.849760+0000) 2024-03-06T17:31:06.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:06.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:06.850016+0000) 2024-03-06T17:31:07.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:07.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:07.850255+0000) 2024-03-06T17:31:08.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:08.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:08.850493+0000) 2024-03-06T17:31:09.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:09.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:09.850749+0000) 2024-03-06T17:31:10.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:10.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:10.851007+0000) 2024-03-06T17:31:11.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:11.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:11.851265+0000) 2024-03-06T17:31:12.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:12.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:12.851508+0000) 2024-03-06T17:31:13.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:13.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:13.851765+0000) 2024-03-06T17:31:14.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:14.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:14.851958+0000) 2024-03-06T17:31:15.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:15.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:15.852214+0000) 2024-03-06T17:31:16.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:16.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:16.852528+0000) 2024-03-06T17:31:17.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:17.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:17.852787+0000) 2024-03-06T17:31:18.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:18.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:18.853050+0000) 2024-03-06T17:31:19.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:19.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:19.853238+0000) 2024-03-06T17:31:20.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:20.818 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:20.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:20.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:20.853434+0000) 2024-03-06T17:31:21.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:21.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:21.853732+0000) 2024-03-06T17:31:22.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:22.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:22.853971+0000) 2024-03-06T17:31:23.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:23.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:23.854238+0000) 2024-03-06T17:31:24.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:24.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:24.854499+0000) 2024-03-06T17:31:25.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:25.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:25.854737+0000) 2024-03-06T17:31:26.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:26.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:26.854984+0000) 2024-03-06T17:31:27.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:27.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:27.855236+0000) 2024-03-06T17:31:28.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:28.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:28.855495+0000) 2024-03-06T17:31:29.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:29.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:29.855752+0000) 2024-03-06T17:31:30.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:30.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:30.855994+0000) 2024-03-06T17:31:31.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:31.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:31.856252+0000) 2024-03-06T17:31:32.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:32.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:32.856512+0000) 2024-03-06T17:31:33.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:33.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:33.856770+0000) 2024-03-06T17:31:34.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:34.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:34.857020+0000) 2024-03-06T17:31:35.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:35.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:35.857289+0000) 2024-03-06T17:31:36.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:36.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:36.857535+0000) 2024-03-06T17:31:37.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:37.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:37.857803+0000) 2024-03-06T17:31:38.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:38.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:38.858062+0000) 2024-03-06T17:31:39.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:39.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:39.858309+0000) 2024-03-06T17:31:40.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:40.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:40.858497+0000) 2024-03-06T17:31:41.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:41.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:41.858738+0000) 2024-03-06T17:31:42.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:42.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:42.858977+0000) 2024-03-06T17:31:43.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:43.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:43.859254+0000) 2024-03-06T17:31:44.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:44.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:44.859503+0000) 2024-03-06T17:31:45.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:45.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:45.859779+0000) 2024-03-06T17:31:46.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:46.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:46.860041+0000) 2024-03-06T17:31:47.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:47.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:47.860237+0000) 2024-03-06T17:31:48.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:48.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:48.860475+0000) 2024-03-06T17:31:49.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:49.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:49.860705+0000) 2024-03-06T17:31:50.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:50.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:50.860906+0000) 2024-03-06T17:31:50.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:50.978 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:51.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:51.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:51.861164+0000) 2024-03-06T17:31:52.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:52.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:52.861419+0000) 2024-03-06T17:31:53.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:53.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:53.861739+0000) 2024-03-06T17:31:54.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:54.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:54.862010+0000) 2024-03-06T17:31:55.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:55.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:55.862263+0000) 2024-03-06T17:31:56.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:56.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:56.862504+0000) 2024-03-06T17:31:57.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:57.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:57.862749+0000) 2024-03-06T17:31:58.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:58.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:58.862986+0000) 2024-03-06T17:31:59.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:31:59.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:59.863241+0000) 2024-03-06T17:32:00.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:00.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:00.863496+0000) 2024-03-06T17:32:01.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:01.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:01.863755+0000) 2024-03-06T17:32:02.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:02.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:02.863990+0000) 2024-03-06T17:32:03.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:03.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:03.864237+0000) 2024-03-06T17:32:04.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:04.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:04.864493+0000) 2024-03-06T17:32:05.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:05.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:05.864729+0000) 2024-03-06T17:32:06.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:06.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:06.864981+0000) 2024-03-06T17:32:07.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:07.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:07.865221+0000) 2024-03-06T17:32:08.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:08.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:08.865476+0000) 2024-03-06T17:32:09.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:09.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:09.865794+0000) 2024-03-06T17:32:10.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:10.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:10.866047+0000) 2024-03-06T17:32:11.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:11.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:11.866231+0000) 2024-03-06T17:32:12.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:12.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:12.866488+0000) 2024-03-06T17:32:13.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:13.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:13.866755+0000) 2024-03-06T17:32:14.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:14.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:14.866990+0000) 2024-03-06T17:32:15.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:15.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:15.867186+0000) 2024-03-06T17:32:16.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:16.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:16.867439+0000) 2024-03-06T17:32:17.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:17.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:17.867724+0000) 2024-03-06T17:32:18.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:18.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:18.867962+0000) 2024-03-06T17:32:19.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:19.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:19.868206+0000) 2024-03-06T17:32:20.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:20.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:20.868463+0000) 2024-03-06T17:32:21.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:21.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:21.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:21.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:21.868684+0000) 2024-03-06T17:32:22.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:22.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:22.868924+0000) 2024-03-06T17:32:23.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:23.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:23.869172+0000) 2024-03-06T17:32:24.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:24.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:24.869407+0000) 2024-03-06T17:32:25.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:25.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:25.869677+0000) 2024-03-06T17:32:26.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:26.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:26.869922+0000) 2024-03-06T17:32:27.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:27.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:27.870170+0000) 2024-03-06T17:32:28.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:28.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:28.870410+0000) 2024-03-06T17:32:29.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:29.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:29.870716+0000) 2024-03-06T17:32:30.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:30.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:30.870954+0000) 2024-03-06T17:32:31.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:31.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:31.871217+0000) 2024-03-06T17:32:32.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:32.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:32.871445+0000) 2024-03-06T17:32:33.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:33.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:33.871753+0000) 2024-03-06T17:32:34.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:34.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:34.872007+0000) 2024-03-06T17:32:35.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:35.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:35.872211+0000) 2024-03-06T17:32:36.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:36.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:36.872465+0000) 2024-03-06T17:32:37.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:37.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:37.872759+0000) 2024-03-06T17:32:38.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:38.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:38.873011+0000) 2024-03-06T17:32:39.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:39.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:39.873266+0000) 2024-03-06T17:32:40.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:40.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:40.873527+0000) 2024-03-06T17:32:41.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:41.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:41.873792+0000) 2024-03-06T17:32:42.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:42.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:42.874036+0000) 2024-03-06T17:32:43.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:43.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:43.874298+0000) 2024-03-06T17:32:44.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:44.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:44.874554+0000) 2024-03-06T17:32:45.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:45.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:45.874790+0000) 2024-03-06T17:32:46.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:46.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:46.875026+0000) 2024-03-06T17:32:47.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:47.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:47.875280+0000) 2024-03-06T17:32:48.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:48.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:48.875541+0000) 2024-03-06T17:32:49.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:49.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:49.875796+0000) 2024-03-06T17:32:50.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:50.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:50.876051+0000) 2024-03-06T17:32:51.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:51.281 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:51.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:51.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:51.876249+0000) 2024-03-06T17:32:52.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:52.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:52.876558+0000) 2024-03-06T17:32:53.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:53.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:53.876834+0000) 2024-03-06T17:32:54.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:54.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:54.877082+0000) 2024-03-06T17:32:55.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:55.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:55.877332+0000) 2024-03-06T17:32:56.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:56.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:56.877608+0000) 2024-03-06T17:32:57.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:57.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:57.877891+0000) 2024-03-06T17:32:58.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:58.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:58.878145+0000) 2024-03-06T17:32:59.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:32:59.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:59.878342+0000) 2024-03-06T17:33:00.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:00.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:00.878604+0000) 2024-03-06T17:33:01.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:01.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:01.878875+0000) 2024-03-06T17:33:02.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:02.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:02.879129+0000) 2024-03-06T17:33:03.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:03.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:03.879392+0000) 2024-03-06T17:33:04.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:04.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:04.879730+0000) 2024-03-06T17:33:05.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:05.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:05.879962+0000) 2024-03-06T17:33:06.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:06.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:06.880201+0000) 2024-03-06T17:33:07.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:07.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:07.880456+0000) 2024-03-06T17:33:08.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:08.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:08.880759+0000) 2024-03-06T17:33:09.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:09.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:09.880999+0000) 2024-03-06T17:33:10.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:10.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:10.881256+0000) 2024-03-06T17:33:11.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:11.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:11.881512+0000) 2024-03-06T17:33:12.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:12.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:12.881781+0000) 2024-03-06T17:33:13.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:13.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:13.882009+0000) 2024-03-06T17:33:14.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:14.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:14.882283+0000) 2024-03-06T17:33:15.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:15.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:15.882501+0000) 2024-03-06T17:33:16.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:16.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:16.882756+0000) 2024-03-06T17:33:17.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:17.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:17.883038+0000) 2024-03-06T17:33:18.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:18.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:18.883321+0000) 2024-03-06T17:33:19.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:19.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:19.883599+0000) 2024-03-06T17:33:20.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:20.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:20.883902+0000) 2024-03-06T17:33:21.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:21.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:21.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:21.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:21.884171+0000) 2024-03-06T17:33:22.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:22.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:22.884382+0000) 2024-03-06T17:33:23.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:23.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:23.884672+0000) 2024-03-06T17:33:24.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:24.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:24.884926+0000) 2024-03-06T17:33:25.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:25.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:25.885174+0000) 2024-03-06T17:33:26.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:26.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:26.885409+0000) 2024-03-06T17:33:27.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:27.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:27.885614+0000) 2024-03-06T17:33:28.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:28.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:28.885919+0000) 2024-03-06T17:33:29.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:29.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:29.886140+0000) 2024-03-06T17:33:30.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:30.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:30.886412+0000) 2024-03-06T17:33:31.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:31.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:31.886712+0000) 2024-03-06T17:33:32.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:32.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:32.886970+0000) 2024-03-06T17:33:33.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:33.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:33.887222+0000) 2024-03-06T17:33:34.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:34.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:34.887476+0000) 2024-03-06T17:33:35.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:35.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:35.887795+0000) 2024-03-06T17:33:36.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:36.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:36.888091+0000) 2024-03-06T17:33:37.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:37.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:37.888381+0000) 2024-03-06T17:33:38.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:38.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:38.888721+0000) 2024-03-06T17:33:39.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:39.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:39.889003+0000) 2024-03-06T17:33:40.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:40.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:40.889306+0000) 2024-03-06T17:33:41.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:41.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:41.889598+0000) 2024-03-06T17:33:42.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:42.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:42.889937+0000) 2024-03-06T17:33:43.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:43.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:43.890261+0000) 2024-03-06T17:33:44.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:44.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:44.890551+0000) 2024-03-06T17:33:45.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:45.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:45.890921+0000) 2024-03-06T17:33:46.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:46.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:46.891194+0000) 2024-03-06T17:33:47.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:47.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:47.891445+0000) 2024-03-06T17:33:48.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:48.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:48.891689+0000) 2024-03-06T17:33:49.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:49.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:49.891924+0000) 2024-03-06T17:33:50.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:50.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:50.892119+0000) 2024-03-06T17:33:51.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:51.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:51.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:51.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:51.892374+0000) 2024-03-06T17:33:52.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:52.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:52.892623+0000) 2024-03-06T17:33:53.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:53.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:53.892925+0000) 2024-03-06T17:33:54.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:54.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:54.893162+0000) 2024-03-06T17:33:55.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:55.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:55.893420+0000) 2024-03-06T17:33:56.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:56.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:56.893747+0000) 2024-03-06T17:33:57.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:57.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:57.893998+0000) 2024-03-06T17:33:58.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:58.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:58.894315+0000) 2024-03-06T17:33:59.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:33:59.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:59.894615+0000) 2024-03-06T17:34:00.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:00.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:00.894985+0000) 2024-03-06T17:34:01.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:01.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:01.895226+0000) 2024-03-06T17:34:02.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:02.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:02.895529+0000) 2024-03-06T17:34:03.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:03.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:03.895795+0000) 2024-03-06T17:34:04.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:04.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:04.896032+0000) 2024-03-06T17:34:05.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:05.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:05.896283+0000) 2024-03-06T17:34:06.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:06.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:06.896594+0000) 2024-03-06T17:34:07.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:07.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:07.896936+0000) 2024-03-06T17:34:08.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:08.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:08.897238+0000) 2024-03-06T17:34:09.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:09.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:09.897551+0000) 2024-03-06T17:34:10.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:10.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:10.897836+0000) 2024-03-06T17:34:11.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:11.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:11.898125+0000) 2024-03-06T17:34:12.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:12.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:12.898433+0000) 2024-03-06T17:34:13.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:13.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:13.898792+0000) 2024-03-06T17:34:14.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:14.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:14.899077+0000) 2024-03-06T17:34:15.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:15.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:15.899401+0000) 2024-03-06T17:34:16.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:16.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:16.899736+0000) 2024-03-06T17:34:17.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:17.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:17.900045+0000) 2024-03-06T17:34:18.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:18.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:18.900356+0000) 2024-03-06T17:34:19.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:19.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:19.900706+0000) 2024-03-06T17:34:20.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:20.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:20.901014+0000) 2024-03-06T17:34:21.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:21.723 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:21.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:21.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:21.901322+0000) 2024-03-06T17:34:22.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:22.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:22.901681+0000) 2024-03-06T17:34:23.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:23.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:23.902034+0000) 2024-03-06T17:34:24.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:24.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:24.902286+0000) 2024-03-06T17:34:25.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:25.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:25.902549+0000) 2024-03-06T17:34:26.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:26.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:26.902790+0000) 2024-03-06T17:34:27.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:27.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:27.903060+0000) 2024-03-06T17:34:28.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:28.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:28.903320+0000) 2024-03-06T17:34:29.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:29.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:29.903571+0000) 2024-03-06T17:34:30.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:30.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:30.903808+0000) 2024-03-06T17:34:31.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:31.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:31.904079+0000) 2024-03-06T17:34:32.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:32.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:32.904357+0000) 2024-03-06T17:34:33.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:33.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:33.904685+0000) 2024-03-06T17:34:34.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:34.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:34.904930+0000) 2024-03-06T17:34:35.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:35.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:35.905172+0000) 2024-03-06T17:34:36.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:36.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:36.905412+0000) 2024-03-06T17:34:37.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:37.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:37.905702+0000) 2024-03-06T17:34:38.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:38.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:38.906013+0000) 2024-03-06T17:34:39.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:39.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:39.906275+0000) 2024-03-06T17:34:40.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:40.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:40.906533+0000) 2024-03-06T17:34:41.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:41.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:41.906823+0000) 2024-03-06T17:34:42.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:42.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:42.907087+0000) 2024-03-06T17:34:43.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:43.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:43.907331+0000) 2024-03-06T17:34:44.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:44.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:44.907571+0000) 2024-03-06T17:34:45.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:45.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:45.907811+0000) 2024-03-06T17:34:46.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:46.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:46.908050+0000) 2024-03-06T17:34:47.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:47.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:47.908309+0000) 2024-03-06T17:34:48.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:48.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:48.908549+0000) 2024-03-06T17:34:49.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:49.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:49.908787+0000) 2024-03-06T17:34:50.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:50.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:50.909043+0000) 2024-03-06T17:34:51.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:51.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:51.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:51.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:51.909261+0000) 2024-03-06T17:34:52.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:52.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:52.909535+0000) 2024-03-06T17:34:53.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:53.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:53.909822+0000) 2024-03-06T17:34:54.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:54.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:54.910075+0000) 2024-03-06T17:34:55.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:55.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:55.910335+0000) 2024-03-06T17:34:56.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:56.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:56.910579+0000) 2024-03-06T17:34:57.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:57.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:57.910836+0000) 2024-03-06T17:34:58.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:58.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:58.911094+0000) 2024-03-06T17:34:59.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:34:59.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:59.911345+0000) 2024-03-06T17:35:00.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:00.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:00.911607+0000) 2024-03-06T17:35:01.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:01.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:01.911906+0000) 2024-03-06T17:35:02.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:02.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:02.912161+0000) 2024-03-06T17:35:03.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:03.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:03.912432+0000) 2024-03-06T17:35:04.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:04.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:04.912723+0000) 2024-03-06T17:35:05.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:05.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:05.912984+0000) 2024-03-06T17:35:06.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:06.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:06.913178+0000) 2024-03-06T17:35:07.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:07.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:07.913416+0000) 2024-03-06T17:35:08.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:08.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:08.913707+0000) 2024-03-06T17:35:09.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:09.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:09.913959+0000) 2024-03-06T17:35:10.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:10.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:10.914225+0000) 2024-03-06T17:35:11.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:11.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:11.914461+0000) 2024-03-06T17:35:12.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:12.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:12.914738+0000) 2024-03-06T17:35:13.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:13.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:13.914995+0000) 2024-03-06T17:35:14.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:14.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:14.915250+0000) 2024-03-06T17:35:15.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:15.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:15.915509+0000) 2024-03-06T17:35:16.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:16.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:16.915766+0000) 2024-03-06T17:35:17.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:17.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:17.916006+0000) 2024-03-06T17:35:18.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:18.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:18.916243+0000) 2024-03-06T17:35:19.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:19.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:19.916491+0000) 2024-03-06T17:35:20.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:20.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:20.916753+0000) 2024-03-06T17:35:21.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:21.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:21.917028+0000) 2024-03-06T17:35:22.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:22.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:22.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:22.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:22.917228+0000) 2024-03-06T17:35:23.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:23.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:23.917492+0000) 2024-03-06T17:35:24.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:24.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:24.917734+0000) 2024-03-06T17:35:25.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:25.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:25.918000+0000) 2024-03-06T17:35:26.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:26.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:26.918298+0000) 2024-03-06T17:35:27.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:27.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:27.918552+0000) 2024-03-06T17:35:28.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:28.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:28.918803+0000) 2024-03-06T17:35:29.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:29.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:29.919060+0000) 2024-03-06T17:35:30.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:30.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:30.919308+0000) 2024-03-06T17:35:31.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:31.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:31.919568+0000) 2024-03-06T17:35:32.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:32.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:32.919828+0000) 2024-03-06T17:35:33.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:33.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:33.920098+0000) 2024-03-06T17:35:34.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:34.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:34.920289+0000) 2024-03-06T17:35:35.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:35.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:35.920479+0000) 2024-03-06T17:35:36.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:36.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:36.920767+0000) 2024-03-06T17:35:37.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:37.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:37.921030+0000) 2024-03-06T17:35:38.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:38.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:38.921281+0000) 2024-03-06T17:35:39.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:39.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:39.921510+0000) 2024-03-06T17:35:40.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:40.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:40.921759+0000) 2024-03-06T17:35:41.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:41.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:41.921969+0000) 2024-03-06T17:35:42.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:42.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:42.922186+0000) 2024-03-06T17:35:43.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:43.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:43.922419+0000) 2024-03-06T17:35:44.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:44.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:44.922729+0000) 2024-03-06T17:35:45.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:45.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:45.923020+0000) 2024-03-06T17:35:46.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:46.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:46.923223+0000) 2024-03-06T17:35:47.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:47.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:47.923481+0000) 2024-03-06T17:35:48.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:48.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:48.923736+0000) 2024-03-06T17:35:49.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:49.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:49.923972+0000) 2024-03-06T17:35:50.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:50.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:50.924229+0000) 2024-03-06T17:35:51.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:51.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:51.924482+0000) 2024-03-06T17:35:52.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:52.158 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:52.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:52.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:52.924711+0000) 2024-03-06T17:35:53.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:53.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:53.924979+0000) 2024-03-06T17:35:54.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:54.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:54.925215+0000) 2024-03-06T17:35:55.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:55.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:55.925478+0000) 2024-03-06T17:35:56.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:56.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:56.925729+0000) 2024-03-06T17:35:57.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:57.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:57.925962+0000) 2024-03-06T17:35:58.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:58.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:58.926218+0000) 2024-03-06T17:35:59.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:35:59.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:59.926470+0000) 2024-03-06T17:36:00.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:00.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:00.926758+0000) 2024-03-06T17:36:01.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:01.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:01.927018+0000) 2024-03-06T17:36:02.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:02.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:02.927250+0000) 2024-03-06T17:36:03.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:03.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:03.927517+0000) 2024-03-06T17:36:04.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:04.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:04.927774+0000) 2024-03-06T17:36:05.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:05.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:05.928040+0000) 2024-03-06T17:36:06.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:06.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:06.928221+0000) 2024-03-06T17:36:07.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:07.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:07.928482+0000) 2024-03-06T17:36:08.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:08.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:08.928738+0000) 2024-03-06T17:36:09.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:09.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:09.928980+0000) 2024-03-06T17:36:10.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:10.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:10.929223+0000) 2024-03-06T17:36:11.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:11.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:11.929464+0000) 2024-03-06T17:36:12.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:12.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:12.929747+0000) 2024-03-06T17:36:13.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:13.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:13.930014+0000) 2024-03-06T17:36:14.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:14.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:14.930334+0000) 2024-03-06T17:36:15.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:15.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:15.930590+0000) 2024-03-06T17:36:16.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:16.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:16.930830+0000) 2024-03-06T17:36:17.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:17.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:17.931093+0000) 2024-03-06T17:36:18.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:18.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:18.931352+0000) 2024-03-06T17:36:19.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:19.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:19.931604+0000) 2024-03-06T17:36:20.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:20.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:20.931838+0000) 2024-03-06T17:36:21.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:21.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:21.932072+0000) 2024-03-06T17:36:22.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:22.295 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:22.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:22.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:22.932314+0000) 2024-03-06T17:36:23.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:23.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:23.932562+0000) 2024-03-06T17:36:24.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:24.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:24.932817+0000) 2024-03-06T17:36:25.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:25.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:25.933073+0000) 2024-03-06T17:36:26.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:26.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:26.933323+0000) 2024-03-06T17:36:27.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:27.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:27.933550+0000) 2024-03-06T17:36:28.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:28.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:28.933788+0000) 2024-03-06T17:36:29.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:29.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:29.934021+0000) 2024-03-06T17:36:30.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:30.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:30.934269+0000) 2024-03-06T17:36:31.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:31.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:31.934510+0000) 2024-03-06T17:36:32.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:32.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:32.934762+0000) 2024-03-06T17:36:33.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:33.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:33.935020+0000) 2024-03-06T17:36:34.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:34.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:34.935272+0000) 2024-03-06T17:36:35.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:35.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:35.935474+0000) 2024-03-06T17:36:36.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:36.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:36.935731+0000) 2024-03-06T17:36:37.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:37.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:37.935983+0000) 2024-03-06T17:36:38.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:38.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:38.936240+0000) 2024-03-06T17:36:39.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:39.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:39.936478+0000) 2024-03-06T17:36:40.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:40.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:40.936763+0000) 2024-03-06T17:36:41.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:41.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:41.937005+0000) 2024-03-06T17:36:42.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:42.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:42.937256+0000) 2024-03-06T17:36:43.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:43.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:43.937517+0000) 2024-03-06T17:36:44.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:44.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:44.937770+0000) 2024-03-06T17:36:45.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:45.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:45.938043+0000) 2024-03-06T17:36:46.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:46.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:46.938299+0000) 2024-03-06T17:36:47.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:47.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:47.938556+0000) 2024-03-06T17:36:48.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:48.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:48.938813+0000) 2024-03-06T17:36:49.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:49.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:49.939050+0000) 2024-03-06T17:36:50.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:50.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:50.939315+0000) 2024-03-06T17:36:51.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:51.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:51.939575+0000) 2024-03-06T17:36:52.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:52.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:52.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:52.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:52.939823+0000) 2024-03-06T17:36:53.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:53.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:53.940073+0000) 2024-03-06T17:36:54.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:54.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:54.940324+0000) 2024-03-06T17:36:55.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:55.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:55.940584+0000) 2024-03-06T17:36:56.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:56.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:56.940854+0000) 2024-03-06T17:36:57.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:57.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:57.941098+0000) 2024-03-06T17:36:58.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:58.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:58.941338+0000) 2024-03-06T17:36:59.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:36:59.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:59.941594+0000) 2024-03-06T17:37:00.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:00.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:00.941855+0000) 2024-03-06T17:37:01.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:01.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:01.942104+0000) 2024-03-06T17:37:02.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:02.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:02.942361+0000) 2024-03-06T17:37:03.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:03.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:03.942610+0000) 2024-03-06T17:37:04.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:04.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:04.942903+0000) 2024-03-06T17:37:05.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:05.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:05.943107+0000) 2024-03-06T17:37:06.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:06.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:06.943363+0000) 2024-03-06T17:37:07.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:07.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:07.943620+0000) 2024-03-06T17:37:08.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:08.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:08.943927+0000) 2024-03-06T17:37:09.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:09.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:09.944164+0000) 2024-03-06T17:37:10.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:10.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:10.944412+0000) 2024-03-06T17:37:11.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:11.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:11.944704+0000) 2024-03-06T17:37:12.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:12.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:12.944965+0000) 2024-03-06T17:37:13.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:13.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:13.945217+0000) 2024-03-06T17:37:14.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:14.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:14.945467+0000) 2024-03-06T17:37:15.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:15.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:15.945764+0000) 2024-03-06T17:37:16.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:16.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:16.946022+0000) 2024-03-06T17:37:17.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:17.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:17.946280+0000) 2024-03-06T17:37:18.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:18.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:18.946536+0000) 2024-03-06T17:37:19.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:19.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:19.946788+0000) 2024-03-06T17:37:20.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:20.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:20.947052+0000) 2024-03-06T17:37:21.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:21.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:21.947316+0000) 2024-03-06T17:37:22.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:22.598 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:22.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:22.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:22.947569+0000) 2024-03-06T17:37:23.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:23.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:23.947845+0000) 2024-03-06T17:37:24.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:24.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:24.948084+0000) 2024-03-06T17:37:25.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:25.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:25.948338+0000) 2024-03-06T17:37:26.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:26.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:26.948606+0000) 2024-03-06T17:37:27.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:27.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:27.948881+0000) 2024-03-06T17:37:28.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:28.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:28.949137+0000) 2024-03-06T17:37:29.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:29.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:29.949404+0000) 2024-03-06T17:37:30.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:30.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:30.949697+0000) 2024-03-06T17:37:31.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:31.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:31.949959+0000) 2024-03-06T17:37:32.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:32.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:32.950199+0000) 2024-03-06T17:37:33.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:33.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:33.950475+0000) 2024-03-06T17:37:34.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:34.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:34.950751+0000) 2024-03-06T17:37:35.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:35.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:35.951026+0000) 2024-03-06T17:37:36.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:36.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:36.951289+0000) 2024-03-06T17:37:37.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:37.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:37.951534+0000) 2024-03-06T17:37:38.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:38.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:38.951781+0000) 2024-03-06T17:37:39.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:39.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:39.952027+0000) 2024-03-06T17:37:40.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:40.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:40.952286+0000) 2024-03-06T17:37:41.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:41.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:41.952537+0000) 2024-03-06T17:37:42.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:42.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:42.952776+0000) 2024-03-06T17:37:43.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:43.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:43.953035+0000) 2024-03-06T17:37:44.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:44.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:44.953270+0000) 2024-03-06T17:37:45.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:45.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:45.953542+0000) 2024-03-06T17:37:46.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:46.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:46.953752+0000) 2024-03-06T17:37:47.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:47.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:47.954013+0000) 2024-03-06T17:37:48.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:48.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:48.954254+0000) 2024-03-06T17:37:49.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:49.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:49.954504+0000) 2024-03-06T17:37:50.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:50.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:50.954770+0000) 2024-03-06T17:37:51.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:51.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:51.955033+0000) 2024-03-06T17:37:52.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:52.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:52.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:52.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:52.955286+0000) 2024-03-06T17:37:53.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:53.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:53.955559+0000) 2024-03-06T17:37:54.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:54.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:54.955796+0000) 2024-03-06T17:37:55.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:55.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:55.956057+0000) 2024-03-06T17:37:56.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:56.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:56.956369+0000) 2024-03-06T17:37:57.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:57.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:57.956666+0000) 2024-03-06T17:37:58.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:58.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:58.956899+0000) 2024-03-06T17:37:59.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:37:59.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:59.957132+0000) 2024-03-06T17:38:00.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:00.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:00.957380+0000) 2024-03-06T17:38:01.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:01.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:01.957672+0000) 2024-03-06T17:38:02.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:02.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:02.957944+0000) 2024-03-06T17:38:03.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:03.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:03.958203+0000) 2024-03-06T17:38:04.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:04.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:04.958451+0000) 2024-03-06T17:38:05.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:05.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:05.958731+0000) 2024-03-06T17:38:06.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:06.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:06.958972+0000) 2024-03-06T17:38:07.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:07.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:07.959215+0000) 2024-03-06T17:38:08.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:08.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:08.959464+0000) 2024-03-06T17:38:09.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:09.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:09.959741+0000) 2024-03-06T17:38:10.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:10.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:10.959983+0000) 2024-03-06T17:38:11.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:11.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:11.960248+0000) 2024-03-06T17:38:12.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:12.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:12.960514+0000) 2024-03-06T17:38:13.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:13.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:13.960764+0000) 2024-03-06T17:38:14.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:14.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:14.961023+0000) 2024-03-06T17:38:15.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:15.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:15.961282+0000) 2024-03-06T17:38:16.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:16.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:16.961526+0000) 2024-03-06T17:38:17.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:17.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:17.961786+0000) 2024-03-06T17:38:18.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:18.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:18.962025+0000) 2024-03-06T17:38:19.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:19.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:19.962286+0000) 2024-03-06T17:38:20.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:20.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:20.962521+0000) 2024-03-06T17:38:21.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:21.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:21.962760+0000) 2024-03-06T17:38:22.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:22.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:22.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:22.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:22.962939+0000) 2024-03-06T17:38:23.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:23.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:23.963184+0000) 2024-03-06T17:38:24.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:24.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:24.963443+0000) 2024-03-06T17:38:25.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:25.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:25.963740+0000) 2024-03-06T17:38:26.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:26.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:26.963980+0000) 2024-03-06T17:38:27.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:27.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:27.964231+0000) 2024-03-06T17:38:28.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:28.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:28.964476+0000) 2024-03-06T17:38:29.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:29.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:29.964773+0000) 2024-03-06T17:38:30.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:30.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:30.964967+0000) 2024-03-06T17:38:31.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:31.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:31.965229+0000) 2024-03-06T17:38:32.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:32.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:32.965486+0000) 2024-03-06T17:38:33.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:33.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:33.965757+0000) 2024-03-06T17:38:34.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:34.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:34.965982+0000) 2024-03-06T17:38:35.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:35.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:35.966252+0000) 2024-03-06T17:38:36.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:36.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:36.966494+0000) 2024-03-06T17:38:37.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:37.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:37.966735+0000) 2024-03-06T17:38:38.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:38.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:38.966993+0000) 2024-03-06T17:38:39.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:39.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:39.967239+0000) 2024-03-06T17:38:40.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:40.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:40.967505+0000) 2024-03-06T17:38:41.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:41.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:41.967774+0000) 2024-03-06T17:38:42.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:42.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:42.968019+0000) 2024-03-06T17:38:43.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:43.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:43.968269+0000) 2024-03-06T17:38:44.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:44.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:44.968573+0000) 2024-03-06T17:38:45.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:45.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:45.968832+0000) 2024-03-06T17:38:46.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:46.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:46.969082+0000) 2024-03-06T17:38:47.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:47.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:47.969338+0000) 2024-03-06T17:38:48.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:48.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:48.969592+0000) 2024-03-06T17:38:49.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:49.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:49.969853+0000) 2024-03-06T17:38:50.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:50.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:50.970114+0000) 2024-03-06T17:38:51.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:51.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:51.970371+0000) 2024-03-06T17:38:52.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:52.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:52.970661+0000) 2024-03-06T17:38:53.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:53.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:53.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:53.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:53.970944+0000) 2024-03-06T17:38:54.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:54.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:54.971203+0000) 2024-03-06T17:38:55.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:55.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:55.971459+0000) 2024-03-06T17:38:56.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:56.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:56.971765+0000) 2024-03-06T17:38:57.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:57.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:57.972022+0000) 2024-03-06T17:38:58.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:58.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:58.972207+0000) 2024-03-06T17:38:59.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:38:59.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:59.972461+0000) 2024-03-06T17:39:00.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:00.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:00.972758+0000) 2024-03-06T17:39:01.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:01.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:01.972998+0000) 2024-03-06T17:39:02.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:02.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:02.973237+0000) 2024-03-06T17:39:03.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:03.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:03.973505+0000) 2024-03-06T17:39:04.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:04.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:04.973747+0000) 2024-03-06T17:39:05.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:05.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:05.973990+0000) 2024-03-06T17:39:06.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:06.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:06.974251+0000) 2024-03-06T17:39:07.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:07.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:07.974497+0000) 2024-03-06T17:39:08.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:08.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:08.974740+0000) 2024-03-06T17:39:09.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:09.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:09.974989+0000) 2024-03-06T17:39:10.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:10.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:10.975229+0000) 2024-03-06T17:39:11.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:11.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:11.975470+0000) 2024-03-06T17:39:12.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:12.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:12.975799+0000) 2024-03-06T17:39:13.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:13.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:13.976052+0000) 2024-03-06T17:39:14.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:14.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:14.976306+0000) 2024-03-06T17:39:15.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:15.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:15.976584+0000) 2024-03-06T17:39:16.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:16.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:16.976888+0000) 2024-03-06T17:39:17.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:17.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:17.977160+0000) 2024-03-06T17:39:18.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:18.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:18.977445+0000) 2024-03-06T17:39:19.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:19.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:19.977742+0000) 2024-03-06T17:39:20.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:20.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:20.978017+0000) 2024-03-06T17:39:21.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:21.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:21.978298+0000) 2024-03-06T17:39:22.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:22.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:22.978564+0000) 2024-03-06T17:39:23.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:23.330 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:23.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:23.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:23.978862+0000) 2024-03-06T17:39:24.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:24.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:24.979122+0000) 2024-03-06T17:39:25.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:25.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:25.979401+0000) 2024-03-06T17:39:26.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:26.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:26.979706+0000) 2024-03-06T17:39:27.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:27.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:27.979919+0000) 2024-03-06T17:39:28.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:28.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:28.980133+0000) 2024-03-06T17:39:29.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:29.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:29.980388+0000) 2024-03-06T17:39:30.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:30.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:30.980679+0000) 2024-03-06T17:39:31.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:31.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:31.980962+0000) 2024-03-06T17:39:32.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:32.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:32.981235+0000) 2024-03-06T17:39:33.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:33.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:33.981502+0000) 2024-03-06T17:39:34.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:34.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:34.981771+0000) 2024-03-06T17:39:35.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:35.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:35.982045+0000) 2024-03-06T17:39:36.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:36.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:36.982325+0000) 2024-03-06T17:39:37.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:37.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:37.982723+0000) 2024-03-06T17:39:38.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:38.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:38.983004+0000) 2024-03-06T17:39:39.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:39.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:39.983277+0000) 2024-03-06T17:39:40.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:40.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:40.983552+0000) 2024-03-06T17:39:41.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:41.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:41.983810+0000) 2024-03-06T17:39:42.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:42.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:42.984055+0000) 2024-03-06T17:39:43.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:43.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:43.984337+0000) 2024-03-06T17:39:44.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:44.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:44.984613+0000) 2024-03-06T17:39:45.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:45.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:45.984930+0000) 2024-03-06T17:39:46.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:46.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:46.985209+0000) 2024-03-06T17:39:47.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:47.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:47.985463+0000) 2024-03-06T17:39:48.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:48.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:48.985751+0000) 2024-03-06T17:39:49.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:49.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:49.985998+0000) 2024-03-06T17:39:50.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:50.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:50.986267+0000) 2024-03-06T17:39:51.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:51.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:51.986543+0000) 2024-03-06T17:39:52.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:52.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:52.986799+0000) 2024-03-06T17:39:53.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:53.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:53.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:53.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:53.987061+0000) 2024-03-06T17:39:54.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:54.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:54.987313+0000) 2024-03-06T17:39:55.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:55.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:55.987541+0000) 2024-03-06T17:39:56.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:56.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:56.987807+0000) 2024-03-06T17:39:57.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:57.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:57.988063+0000) 2024-03-06T17:39:58.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:58.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:58.988306+0000) 2024-03-06T17:39:59.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:39:59.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:59.988565+0000) 2024-03-06T17:40:00.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:00.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:00.988801+0000) 2024-03-06T17:40:01.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:01.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:01.989049+0000) 2024-03-06T17:40:02.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:02.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:02.989326+0000) 2024-03-06T17:40:03.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:03.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:03.989616+0000) 2024-03-06T17:40:04.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:04.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:04.989943+0000) 2024-03-06T17:40:05.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:05.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:05.990213+0000) 2024-03-06T17:40:06.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:06.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:06.990473+0000) 2024-03-06T17:40:07.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:07.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:07.990750+0000) 2024-03-06T17:40:08.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:08.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:08.991047+0000) 2024-03-06T17:40:09.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:09.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:09.991357+0000) 2024-03-06T17:40:10.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:10.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:10.991713+0000) 2024-03-06T17:40:11.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:11.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:11.992023+0000) 2024-03-06T17:40:12.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:12.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:12.992275+0000) 2024-03-06T17:40:13.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:13.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:13.992544+0000) 2024-03-06T17:40:14.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:14.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:14.992834+0000) 2024-03-06T17:40:15.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:15.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:15.993119+0000) 2024-03-06T17:40:16.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:16.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:16.993405+0000) 2024-03-06T17:40:17.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:17.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:17.993765+0000) 2024-03-06T17:40:18.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:18.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:18.994042+0000) 2024-03-06T17:40:19.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:19.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:19.994335+0000) 2024-03-06T17:40:20.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:20.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:20.994683+0000) 2024-03-06T17:40:21.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:21.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:21.994993+0000) 2024-03-06T17:40:22.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:22.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:22.995304+0000) 2024-03-06T17:40:23.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:23.639 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:23.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:23.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:23.995682+0000) 2024-03-06T17:40:24.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:24.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:24.995935+0000) 2024-03-06T17:40:25.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:25.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:25.996298+0000) 2024-03-06T17:40:26.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:26.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:26.996584+0000) 2024-03-06T17:40:27.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:27.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:27.996929+0000) 2024-03-06T17:40:29.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:28.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:28.997230+0000) 2024-03-06T17:40:29.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:29.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:29.997505+0000) 2024-03-06T17:40:31.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:30.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:30.997811+0000) 2024-03-06T17:40:32.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:31.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:31.998118+0000) 2024-03-06T17:40:33.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:32.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:32.998445+0000) 2024-03-06T17:40:34.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:33.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:33.998828+0000) 2024-03-06T17:40:35.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:34.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:34.999067+0000) 2024-03-06T17:40:36.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:35.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:35.999300+0000) 2024-03-06T17:40:37.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:36.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:36.999608+0000) 2024-03-06T17:40:38.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:37.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:37.999971+0000) 2024-03-06T17:40:39.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:38.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:39.000276+0000) 2024-03-06T17:40:40.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:39.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:40.000503+0000) 2024-03-06T17:40:41.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:41.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:41.000815+0000) 2024-03-06T17:40:42.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:42.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:42.001105+0000) 2024-03-06T17:40:43.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:43.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:43.001349+0000) 2024-03-06T17:40:44.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:44.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:44.001715+0000) 2024-03-06T17:40:45.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:45.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:45.002025+0000) 2024-03-06T17:40:46.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:46.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:46.002323+0000) 2024-03-06T17:40:47.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:47.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:47.002620+0000) 2024-03-06T17:40:48.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:48.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:48.002978+0000) 2024-03-06T17:40:49.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:49.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:49.003239+0000) 2024-03-06T17:40:50.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:50.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:50.003538+0000) 2024-03-06T17:40:51.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:51.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:51.003832+0000) 2024-03-06T17:40:52.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:52.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:52.004123+0000) 2024-03-06T17:40:53.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:53.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:53.004383+0000) 2024-03-06T17:40:53.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:53.797 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:54.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:54.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:54.004601+0000) 2024-03-06T17:40:55.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:55.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:55.004924+0000) 2024-03-06T17:40:56.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:56.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:56.005234+0000) 2024-03-06T17:40:57.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:57.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:57.005553+0000) 2024-03-06T17:40:58.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:58.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:58.005848+0000) 2024-03-06T17:40:59.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:40:59.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:59.006126+0000) 2024-03-06T17:41:00.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:00.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:00.006424+0000) 2024-03-06T17:41:01.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:01.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:01.006706+0000) 2024-03-06T17:41:02.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:02.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:02.007117+0000) 2024-03-06T17:41:03.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:03.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:03.007381+0000) 2024-03-06T17:41:04.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:04.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:04.007698+0000) 2024-03-06T17:41:05.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:05.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:05.007948+0000) 2024-03-06T17:41:06.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:06.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:06.008190+0000) 2024-03-06T17:41:07.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:07.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:07.008456+0000) 2024-03-06T17:41:08.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:08.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:08.008735+0000) 2024-03-06T17:41:09.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:09.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:09.008997+0000) 2024-03-06T17:41:10.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:10.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:10.009238+0000) 2024-03-06T17:41:11.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:11.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:11.009432+0000) 2024-03-06T17:41:12.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:12.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:12.009760+0000) 2024-03-06T17:41:13.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:13.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:13.010019+0000) 2024-03-06T17:41:14.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:14.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:14.010291+0000) 2024-03-06T17:41:15.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:15.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:15.010543+0000) 2024-03-06T17:41:16.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:16.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:16.010808+0000) 2024-03-06T17:41:17.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:17.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:17.011053+0000) 2024-03-06T17:41:18.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:18.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:18.011240+0000) 2024-03-06T17:41:19.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:19.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:19.011488+0000) 2024-03-06T17:41:20.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:20.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:20.011730+0000) 2024-03-06T17:41:21.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:21.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:21.011973+0000) 2024-03-06T17:41:22.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:22.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:22.012178+0000) 2024-03-06T17:41:23.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:23.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:23.012417+0000) 2024-03-06T17:41:23.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:23.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:24.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:24.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:24.012666+0000) 2024-03-06T17:41:25.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:25.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:25.012881+0000) 2024-03-06T17:41:26.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:26.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:26.013194+0000) 2024-03-06T17:41:27.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:27.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:27.013483+0000) 2024-03-06T17:41:28.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:28.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:28.013802+0000) 2024-03-06T17:41:29.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:29.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:29.014095+0000) 2024-03-06T17:41:30.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:30.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:30.014395+0000) 2024-03-06T17:41:31.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:31.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:31.014758+0000) 2024-03-06T17:41:32.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:32.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:32.015021+0000) 2024-03-06T17:41:33.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:33.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:33.015278+0000) 2024-03-06T17:41:34.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:34.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:34.015526+0000) 2024-03-06T17:41:35.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:35.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:35.015771+0000) 2024-03-06T17:41:36.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:36.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:36.016043+0000) 2024-03-06T17:41:37.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:37.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:37.016311+0000) 2024-03-06T17:41:38.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:38.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:38.016574+0000) 2024-03-06T17:41:39.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:39.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:39.016830+0000) 2024-03-06T17:41:40.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:40.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:40.017083+0000) 2024-03-06T17:41:41.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:41.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:41.017338+0000) 2024-03-06T17:41:42.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:42.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:42.017592+0000) 2024-03-06T17:41:43.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:43.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:43.017853+0000) 2024-03-06T17:41:44.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:44.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:44.018099+0000) 2024-03-06T17:41:45.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:45.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:45.018350+0000) 2024-03-06T17:41:46.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:46.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:46.018594+0000) 2024-03-06T17:41:47.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:47.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:47.018807+0000) 2024-03-06T17:41:48.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:48.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:48.019040+0000) 2024-03-06T17:41:49.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:49.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:49.019299+0000) 2024-03-06T17:41:50.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:50.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:50.019541+0000) 2024-03-06T17:41:51.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:51.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:51.019797+0000) 2024-03-06T17:41:52.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:52.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:52.020048+0000) 2024-03-06T17:41:53.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:53.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:53.020290+0000) 2024-03-06T17:41:54.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:54.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:54.020562+0000) 2024-03-06T17:41:54.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:54.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:55.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:55.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:55.020816+0000) 2024-03-06T17:41:56.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:56.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:56.021072+0000) 2024-03-06T17:41:57.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:57.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:57.021324+0000) 2024-03-06T17:41:58.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:58.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:58.021587+0000) 2024-03-06T17:41:59.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:41:59.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:59.021791+0000) 2024-03-06T17:42:00.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:00.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:00.022043+0000) 2024-03-06T17:42:01.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:01.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:01.022305+0000) 2024-03-06T17:42:02.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:02.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:02.022578+0000) 2024-03-06T17:42:03.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:03.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:03.022837+0000) 2024-03-06T17:42:04.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:04.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:04.023104+0000) 2024-03-06T17:42:05.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:05.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:05.023296+0000) 2024-03-06T17:42:06.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:06.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:06.023536+0000) 2024-03-06T17:42:07.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:07.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:07.023858+0000) 2024-03-06T17:42:08.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:08.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:08.024098+0000) 2024-03-06T17:42:09.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:09.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:09.024348+0000) 2024-03-06T17:42:10.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:10.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:10.024532+0000) 2024-03-06T17:42:11.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:11.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:11.024792+0000) 2024-03-06T17:42:12.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:12.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:12.025055+0000) 2024-03-06T17:42:13.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:13.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:13.025319+0000) 2024-03-06T17:42:14.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:14.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:14.025605+0000) 2024-03-06T17:42:15.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:15.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:15.025939+0000) 2024-03-06T17:42:16.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:16.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:16.026196+0000) 2024-03-06T17:42:17.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:17.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:17.026460+0000) 2024-03-06T17:42:18.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:18.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:18.026740+0000) 2024-03-06T17:42:19.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:19.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:19.026950+0000) 2024-03-06T17:42:20.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:20.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:20.027183+0000) 2024-03-06T17:42:21.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:21.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:21.027445+0000) 2024-03-06T17:42:22.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:22.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:22.027744+0000) 2024-03-06T17:42:23.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:23.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:23.027982+0000) 2024-03-06T17:42:24.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:24.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:24.028249+0000) 2024-03-06T17:42:24.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:24.255 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:25.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:25.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:25.028510+0000) 2024-03-06T17:42:26.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:26.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:26.028778+0000) 2024-03-06T17:42:27.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:27.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:27.029047+0000) 2024-03-06T17:42:28.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:28.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:28.029309+0000) 2024-03-06T17:42:29.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:29.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:29.029556+0000) 2024-03-06T17:42:30.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:30.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:30.029814+0000) 2024-03-06T17:42:31.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:31.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:31.030077+0000) 2024-03-06T17:42:32.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:32.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:32.030342+0000) 2024-03-06T17:42:33.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:33.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:33.030600+0000) 2024-03-06T17:42:34.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:34.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:34.030921+0000) 2024-03-06T17:42:35.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:35.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:35.031175+0000) 2024-03-06T17:42:36.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:36.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:36.031422+0000) 2024-03-06T17:42:37.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:37.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:37.031831+0000) 2024-03-06T17:42:38.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:38.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:38.032095+0000) 2024-03-06T17:42:39.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:39.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:39.032289+0000) 2024-03-06T17:42:40.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:40.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:40.032554+0000) 2024-03-06T17:42:41.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:41.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:41.032815+0000) 2024-03-06T17:42:42.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:42.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:42.033054+0000) 2024-03-06T17:42:43.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:43.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:43.033312+0000) 2024-03-06T17:42:44.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:44.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:44.033564+0000) 2024-03-06T17:42:45.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:45.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:45.033800+0000) 2024-03-06T17:42:46.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:46.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:46.033993+0000) 2024-03-06T17:42:47.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:47.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:47.034238+0000) 2024-03-06T17:42:48.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:48.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:48.034496+0000) 2024-03-06T17:42:49.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:49.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:49.034745+0000) 2024-03-06T17:42:50.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:50.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:50.035004+0000) 2024-03-06T17:42:51.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:51.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:51.035199+0000) 2024-03-06T17:42:52.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:52.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:52.035454+0000) 2024-03-06T17:42:53.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:53.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:53.035763+0000) 2024-03-06T17:42:54.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:54.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:54.036030+0000) 2024-03-06T17:42:54.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:54.417 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:55.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:55.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:55.036339+0000) 2024-03-06T17:42:56.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:56.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:56.036582+0000) 2024-03-06T17:42:57.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:57.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:57.036841+0000) 2024-03-06T17:42:58.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:58.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:58.037104+0000) 2024-03-06T17:42:59.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:42:59.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:59.037371+0000) 2024-03-06T17:43:00.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:00.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:00.037667+0000) 2024-03-06T17:43:01.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:01.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:01.037937+0000) 2024-03-06T17:43:02.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:02.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:02.038183+0000) 2024-03-06T17:43:03.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:03.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:03.038449+0000) 2024-03-06T17:43:04.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:04.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:04.038753+0000) 2024-03-06T17:43:05.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:05.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:05.039014+0000) 2024-03-06T17:43:06.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:06.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:06.039274+0000) 2024-03-06T17:43:07.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:07.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:07.039530+0000) 2024-03-06T17:43:08.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:08.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:08.039774+0000) 2024-03-06T17:43:09.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:09.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:09.040004+0000) 2024-03-06T17:43:10.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:10.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:10.040255+0000) 2024-03-06T17:43:11.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:11.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:11.040511+0000) 2024-03-06T17:43:12.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:12.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:12.040741+0000) 2024-03-06T17:43:13.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:13.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:13.040983+0000) 2024-03-06T17:43:14.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:14.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:14.041234+0000) 2024-03-06T17:43:15.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:15.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:15.041483+0000) 2024-03-06T17:43:16.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:16.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:16.041745+0000) 2024-03-06T17:43:17.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:17.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:17.042000+0000) 2024-03-06T17:43:18.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:18.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:18.042255+0000) 2024-03-06T17:43:19.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:19.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:19.042490+0000) 2024-03-06T17:43:20.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:20.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:20.042747+0000) 2024-03-06T17:43:21.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:21.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:21.042990+0000) 2024-03-06T17:43:22.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:22.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:22.043249+0000) 2024-03-06T17:43:23.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:23.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:23.043503+0000) 2024-03-06T17:43:24.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:24.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:24.043768+0000) 2024-03-06T17:43:24.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:24.555 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:25.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:25.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:25.043966+0000) 2024-03-06T17:43:26.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:26.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:26.044227+0000) 2024-03-06T17:43:27.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:27.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:27.044464+0000) 2024-03-06T17:43:28.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:28.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:28.044770+0000) 2024-03-06T17:43:29.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:29.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:29.045024+0000) 2024-03-06T17:43:30.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:30.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:30.045275+0000) 2024-03-06T17:43:31.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:31.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:31.045455+0000) 2024-03-06T17:43:32.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:32.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:32.045740+0000) 2024-03-06T17:43:33.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:33.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:33.045983+0000) 2024-03-06T17:43:34.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:34.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:34.046248+0000) 2024-03-06T17:43:35.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:35.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:35.046494+0000) 2024-03-06T17:43:36.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:36.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:36.046751+0000) 2024-03-06T17:43:37.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:37.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:37.046997+0000) 2024-03-06T17:43:38.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:38.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:38.047265+0000) 2024-03-06T17:43:39.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:39.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:39.047500+0000) 2024-03-06T17:43:40.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:40.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:40.047758+0000) 2024-03-06T17:43:41.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:41.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:41.047944+0000) 2024-03-06T17:43:42.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:42.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:42.048128+0000) 2024-03-06T17:43:43.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:43.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:43.048375+0000) 2024-03-06T17:43:44.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:44.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:44.048624+0000) 2024-03-06T17:43:45.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:45.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:45.048881+0000) 2024-03-06T17:43:46.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:46.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:46.049125+0000) 2024-03-06T17:43:47.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:47.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:47.049355+0000) 2024-03-06T17:43:48.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:48.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:48.049659+0000) 2024-03-06T17:43:49.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:49.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:49.049902+0000) 2024-03-06T17:43:50.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:50.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:50.050157+0000) 2024-03-06T17:43:51.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:51.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:51.050422+0000) 2024-03-06T17:43:52.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:52.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:52.050694+0000) 2024-03-06T17:43:53.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:53.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:53.050938+0000) 2024-03-06T17:43:54.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:54.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:54.051211+0000) 2024-03-06T17:43:54.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:54.705 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:55.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:55.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:55.051410+0000) 2024-03-06T17:43:56.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:56.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:56.051715+0000) 2024-03-06T17:43:57.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:57.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:57.051976+0000) 2024-03-06T17:43:58.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:58.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:58.052169+0000) 2024-03-06T17:43:59.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:43:59.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:59.052430+0000) 2024-03-06T17:44:00.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:00.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:00.052737+0000) 2024-03-06T17:44:01.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:01.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:01.053179+0000) 2024-03-06T17:44:02.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:02.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:02.053449+0000) 2024-03-06T17:44:03.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:03.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:03.053776+0000) 2024-03-06T17:44:04.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:04.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:04.054025+0000) 2024-03-06T17:44:05.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:05.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:05.054283+0000) 2024-03-06T17:44:06.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:06.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:06.054541+0000) 2024-03-06T17:44:07.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:07.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:07.054792+0000) 2024-03-06T17:44:08.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:08.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:08.055055+0000) 2024-03-06T17:44:09.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:09.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:09.055294+0000) 2024-03-06T17:44:10.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:10.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:10.055551+0000) 2024-03-06T17:44:11.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:11.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:11.055811+0000) 2024-03-06T17:44:12.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:12.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:12.056052+0000) 2024-03-06T17:44:13.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:13.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:13.056309+0000) 2024-03-06T17:44:14.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:14.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:14.056567+0000) 2024-03-06T17:44:15.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:15.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:15.056831+0000) 2024-03-06T17:44:16.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:16.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:16.057109+0000) 2024-03-06T17:44:17.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:17.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:17.057364+0000) 2024-03-06T17:44:18.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:18.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:18.057623+0000) 2024-03-06T17:44:19.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:19.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:19.057930+0000) 2024-03-06T17:44:20.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:20.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:20.058183+0000) 2024-03-06T17:44:21.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:21.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:21.058423+0000) 2024-03-06T17:44:22.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:22.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:22.058728+0000) 2024-03-06T17:44:23.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:23.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:23.058991+0000) 2024-03-06T17:44:24.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:24.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:24.059263+0000) 2024-03-06T17:44:24.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:24.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:25.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:25.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:25.059529+0000) 2024-03-06T17:44:26.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:26.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:26.059774+0000) 2024-03-06T17:44:27.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:27.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:27.060018+0000) 2024-03-06T17:44:28.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:28.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:28.060286+0000) 2024-03-06T17:44:29.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:29.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:29.060528+0000) 2024-03-06T17:44:30.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:30.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:30.060778+0000) 2024-03-06T17:44:31.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:31.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:31.061021+0000) 2024-03-06T17:44:32.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:32.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:32.061288+0000) 2024-03-06T17:44:33.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:33.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:33.061485+0000) 2024-03-06T17:44:34.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:34.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:34.061756+0000) 2024-03-06T17:44:35.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:35.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:35.061939+0000) 2024-03-06T17:44:36.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:36.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:36.062124+0000) 2024-03-06T17:44:37.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:37.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:37.062432+0000) 2024-03-06T17:44:38.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:38.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:38.062744+0000) 2024-03-06T17:44:39.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:39.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:39.062944+0000) 2024-03-06T17:44:40.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:40.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:40.063197+0000) 2024-03-06T17:44:41.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:41.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:41.063386+0000) 2024-03-06T17:44:42.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:42.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:42.063673+0000) 2024-03-06T17:44:43.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:43.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:43.063885+0000) 2024-03-06T17:44:44.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:44.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:44.064145+0000) 2024-03-06T17:44:45.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:45.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:45.064375+0000) 2024-03-06T17:44:46.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:46.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:46.064658+0000) 2024-03-06T17:44:47.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:47.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:47.064893+0000) 2024-03-06T17:44:48.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:48.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:48.065137+0000) 2024-03-06T17:44:49.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:49.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:49.065394+0000) 2024-03-06T17:44:50.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:50.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:50.065697+0000) 2024-03-06T17:44:51.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:51.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:51.065936+0000) 2024-03-06T17:44:52.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:52.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:52.066196+0000) 2024-03-06T17:44:53.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:53.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:53.066438+0000) 2024-03-06T17:44:54.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:54.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:54.066737+0000) 2024-03-06T17:44:55.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:55.026 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:55.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:55.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:55.066933+0000) 2024-03-06T17:44:56.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:56.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:56.067195+0000) 2024-03-06T17:44:57.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:57.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:57.067451+0000) 2024-03-06T17:44:58.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:58.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:58.067762+0000) 2024-03-06T17:44:59.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:44:59.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:59.067992+0000) 2024-03-06T17:45:00.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:00.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:00.068246+0000) 2024-03-06T17:45:01.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:01.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:01.068505+0000) 2024-03-06T17:45:02.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:02.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:02.068742+0000) 2024-03-06T17:45:03.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:03.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:03.069044+0000) 2024-03-06T17:45:04.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:04.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:04.069360+0000) 2024-03-06T17:45:05.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:05.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:05.069728+0000) 2024-03-06T17:45:06.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:06.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:06.070037+0000) 2024-03-06T17:45:07.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:07.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:07.070352+0000) 2024-03-06T17:45:08.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:08.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:08.070715+0000) 2024-03-06T17:45:09.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:09.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:09.071011+0000) 2024-03-06T17:45:10.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:10.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:10.071249+0000) 2024-03-06T17:45:11.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:11.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:11.071495+0000) 2024-03-06T17:45:12.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:12.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:12.071755+0000) 2024-03-06T17:45:13.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:13.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:13.072013+0000) 2024-03-06T17:45:14.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:14.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:14.072281+0000) 2024-03-06T17:45:15.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:15.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:15.072514+0000) 2024-03-06T17:45:16.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:16.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:16.072754+0000) 2024-03-06T17:45:17.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:17.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:17.073009+0000) 2024-03-06T17:45:18.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:18.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:18.073254+0000) 2024-03-06T17:45:19.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:19.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:19.073501+0000) 2024-03-06T17:45:20.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:20.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:20.073735+0000) 2024-03-06T17:45:21.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:21.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:21.074000+0000) 2024-03-06T17:45:22.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:22.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:22.074259+0000) 2024-03-06T17:45:23.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:23.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:23.074450+0000) 2024-03-06T17:45:24.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:24.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:24.074756+0000) 2024-03-06T17:45:25.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:25.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:25.075013+0000) 2024-03-06T17:45:25.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:25.168 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:26.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:26.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:26.075272+0000) 2024-03-06T17:45:27.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:27.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:27.075522+0000) 2024-03-06T17:45:28.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:28.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:28.075784+0000) 2024-03-06T17:45:29.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:29.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:29.076043+0000) 2024-03-06T17:45:30.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:30.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:30.076355+0000) 2024-03-06T17:45:31.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:31.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:31.076595+0000) 2024-03-06T17:45:32.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:32.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:32.076859+0000) 2024-03-06T17:45:33.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:33.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:33.077122+0000) 2024-03-06T17:45:34.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:34.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:34.077390+0000) 2024-03-06T17:45:35.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:35.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:35.077671+0000) 2024-03-06T17:45:36.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:36.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:36.077904+0000) 2024-03-06T17:45:37.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:37.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:37.078142+0000) 2024-03-06T17:45:38.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:38.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:38.078388+0000) 2024-03-06T17:45:39.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:39.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:39.078660+0000) 2024-03-06T17:45:40.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:40.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:40.078919+0000) 2024-03-06T17:45:41.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:41.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:41.079172+0000) 2024-03-06T17:45:42.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:42.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:42.079431+0000) 2024-03-06T17:45:43.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:43.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:43.079719+0000) 2024-03-06T17:45:44.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:44.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:44.079980+0000) 2024-03-06T17:45:45.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:45.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:45.080234+0000) 2024-03-06T17:45:46.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:46.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:46.080475+0000) 2024-03-06T17:45:47.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:47.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:47.080741+0000) 2024-03-06T17:45:48.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:48.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:48.080990+0000) 2024-03-06T17:45:49.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:49.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:49.081249+0000) 2024-03-06T17:45:50.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:50.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:50.081482+0000) 2024-03-06T17:45:51.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:51.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:51.081741+0000) 2024-03-06T17:45:52.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:52.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:52.081994+0000) 2024-03-06T17:45:53.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:53.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:53.082183+0000) 2024-03-06T17:45:54.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:54.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:54.082450+0000) 2024-03-06T17:45:55.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:55.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:55.082749+0000) 2024-03-06T17:45:55.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:55.316 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:56.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:56.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:56.082950+0000) 2024-03-06T17:45:57.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:57.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:57.083203+0000) 2024-03-06T17:45:58.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:58.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:58.083461+0000) 2024-03-06T17:45:59.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:45:59.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:59.083750+0000) 2024-03-06T17:46:00.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:00.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:00.084002+0000) 2024-03-06T17:46:01.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:01.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:01.084263+0000) 2024-03-06T17:46:02.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:02.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:02.084530+0000) 2024-03-06T17:46:03.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:03.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:03.084804+0000) 2024-03-06T17:46:04.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:04.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:04.085054+0000) 2024-03-06T17:46:05.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:05.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:05.085301+0000) 2024-03-06T17:46:06.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:06.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:06.085541+0000) 2024-03-06T17:46:07.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:07.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:07.085782+0000) 2024-03-06T17:46:08.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:08.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:08.086027+0000) 2024-03-06T17:46:09.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:09.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:09.086275+0000) 2024-03-06T17:46:10.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:10.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:10.086516+0000) 2024-03-06T17:46:11.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:11.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:11.086749+0000) 2024-03-06T17:46:12.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:12.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:12.087006+0000) 2024-03-06T17:46:13.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:13.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:13.087314+0000) 2024-03-06T17:46:14.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:14.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:14.087582+0000) 2024-03-06T17:46:15.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:15.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:15.087835+0000) 2024-03-06T17:46:16.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:16.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:16.088072+0000) 2024-03-06T17:46:17.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:17.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:17.088332+0000) 2024-03-06T17:46:18.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:18.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:18.088584+0000) 2024-03-06T17:46:19.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:19.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:19.088840+0000) 2024-03-06T17:46:20.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:20.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:20.089074+0000) 2024-03-06T17:46:21.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:21.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:21.089313+0000) 2024-03-06T17:46:22.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:22.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:22.089565+0000) 2024-03-06T17:46:23.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:23.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:23.089830+0000) 2024-03-06T17:46:24.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:24.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:24.090080+0000) 2024-03-06T17:46:25.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:25.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:25.090318+0000) 2024-03-06T17:46:25.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:25.445 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:26.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:26.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:26.090577+0000) 2024-03-06T17:46:27.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:27.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:27.090833+0000) 2024-03-06T17:46:28.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:28.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:28.091085+0000) 2024-03-06T17:46:29.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:29.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:29.091350+0000) 2024-03-06T17:46:30.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:30.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:30.091607+0000) 2024-03-06T17:46:31.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:31.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:31.091882+0000) 2024-03-06T17:46:32.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:32.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:32.092121+0000) 2024-03-06T17:46:33.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:33.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:33.092313+0000) 2024-03-06T17:46:34.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:34.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:34.092592+0000) 2024-03-06T17:46:35.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:35.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:35.092854+0000) 2024-03-06T17:46:36.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:36.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:36.093109+0000) 2024-03-06T17:46:37.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:37.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:37.093349+0000) 2024-03-06T17:46:38.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:38.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:38.093609+0000) 2024-03-06T17:46:39.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:39.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:39.093912+0000) 2024-03-06T17:46:40.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:40.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:40.094146+0000) 2024-03-06T17:46:41.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:41.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:41.094380+0000) 2024-03-06T17:46:42.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:42.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:42.094625+0000) 2024-03-06T17:46:43.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:43.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:43.094932+0000) 2024-03-06T17:46:44.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:44.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:44.095202+0000) 2024-03-06T17:46:45.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:45.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:45.095459+0000) 2024-03-06T17:46:46.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:46.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:46.095743+0000) 2024-03-06T17:46:47.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:47.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:47.095979+0000) 2024-03-06T17:46:48.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:48.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:48.096241+0000) 2024-03-06T17:46:49.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:49.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:49.096498+0000) 2024-03-06T17:46:50.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:50.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:50.096757+0000) 2024-03-06T17:46:51.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:51.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:51.097015+0000) 2024-03-06T17:46:52.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:52.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:52.097270+0000) 2024-03-06T17:46:53.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:53.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:53.097512+0000) 2024-03-06T17:46:54.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:54.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:54.097777+0000) 2024-03-06T17:46:55.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:55.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:55.098030+0000) 2024-03-06T17:46:55.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:55.601 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:56.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:56.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:56.098287+0000) 2024-03-06T17:46:57.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:57.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:57.098526+0000) 2024-03-06T17:46:58.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:58.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:58.098768+0000) 2024-03-06T17:46:59.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:46:59.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:59.099049+0000) 2024-03-06T17:47:00.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:00.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:00.099297+0000) 2024-03-06T17:47:01.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:01.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:01.099558+0000) 2024-03-06T17:47:02.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:02.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:02.099803+0000) 2024-03-06T17:47:03.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:03.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:03.100071+0000) 2024-03-06T17:47:04.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:04.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:04.100338+0000) 2024-03-06T17:47:05.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:05.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:05.100580+0000) 2024-03-06T17:47:06.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:06.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:06.100869+0000) 2024-03-06T17:47:07.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:07.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:07.101116+0000) 2024-03-06T17:47:08.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:08.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:08.101368+0000) 2024-03-06T17:47:09.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:09.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:09.101663+0000) 2024-03-06T17:47:10.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:10.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:10.101900+0000) 2024-03-06T17:47:11.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:11.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:11.102137+0000) 2024-03-06T17:47:12.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:12.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:12.102330+0000) 2024-03-06T17:47:13.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:13.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:13.102568+0000) 2024-03-06T17:47:14.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:14.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:14.102831+0000) 2024-03-06T17:47:15.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:15.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:15.103084+0000) 2024-03-06T17:47:16.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:16.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:16.103263+0000) 2024-03-06T17:47:17.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:17.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:17.103518+0000) 2024-03-06T17:47:18.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:18.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:18.103771+0000) 2024-03-06T17:47:19.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:19.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:19.104029+0000) 2024-03-06T17:47:20.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:20.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:20.104276+0000) 2024-03-06T17:47:21.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:21.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:21.104532+0000) 2024-03-06T17:47:22.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:22.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:22.104775+0000) 2024-03-06T17:47:23.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:23.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:23.105035+0000) 2024-03-06T17:47:24.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:24.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:24.105302+0000) 2024-03-06T17:47:25.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:25.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:25.105542+0000) 2024-03-06T17:47:25.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:25.750 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:26.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:26.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:26.105806+0000) 2024-03-06T17:47:27.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:27.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:27.106000+0000) 2024-03-06T17:47:28.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:28.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:28.106246+0000) 2024-03-06T17:47:29.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:29.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:29.106475+0000) 2024-03-06T17:47:30.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:30.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:30.106718+0000) 2024-03-06T17:47:31.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:31.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:31.106983+0000) 2024-03-06T17:47:32.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:32.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:32.107244+0000) 2024-03-06T17:47:33.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:33.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:33.107482+0000) 2024-03-06T17:47:34.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:34.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:34.107765+0000) 2024-03-06T17:47:35.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:35.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:35.108014+0000) 2024-03-06T17:47:36.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:36.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:36.108256+0000) 2024-03-06T17:47:37.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:37.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:37.108509+0000) 2024-03-06T17:47:38.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:38.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:38.108763+0000) 2024-03-06T17:47:39.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:39.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:39.109018+0000) 2024-03-06T17:47:40.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:40.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:40.109259+0000) 2024-03-06T17:47:41.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:41.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:41.109514+0000) 2024-03-06T17:47:42.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:42.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:42.109775+0000) 2024-03-06T17:47:43.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:43.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:43.110031+0000) 2024-03-06T17:47:44.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:44.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:44.110297+0000) 2024-03-06T17:47:45.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:45.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:45.110546+0000) 2024-03-06T17:47:46.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:46.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:46.110783+0000) 2024-03-06T17:47:47.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:47.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:47.111022+0000) 2024-03-06T17:47:48.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:48.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:48.111279+0000) 2024-03-06T17:47:49.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:49.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:49.111525+0000) 2024-03-06T17:47:50.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:50.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:50.111760+0000) 2024-03-06T17:47:51.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:51.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:51.111953+0000) 2024-03-06T17:47:52.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:52.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:52.112190+0000) 2024-03-06T17:47:53.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:53.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:53.112444+0000) 2024-03-06T17:47:54.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:54.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:54.112791+0000) 2024-03-06T17:47:55.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:55.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:55.113026+0000) 2024-03-06T17:47:55.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:55.891 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:56.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:56.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:56.113272+0000) 2024-03-06T17:47:57.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:57.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:57.113511+0000) 2024-03-06T17:47:58.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:58.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:58.113764+0000) 2024-03-06T17:47:59.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:47:59.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:59.113957+0000) 2024-03-06T17:48:00.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:00.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:00.114193+0000) 2024-03-06T17:48:01.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:01.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:01.114492+0000) 2024-03-06T17:48:02.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:02.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:02.114799+0000) 2024-03-06T17:48:03.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:03.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:03.115105+0000) 2024-03-06T17:48:04.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:04.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:04.115419+0000) 2024-03-06T17:48:05.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:05.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:05.115752+0000) 2024-03-06T17:48:06.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:06.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:06.116055+0000) 2024-03-06T17:48:07.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:07.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:07.116361+0000) 2024-03-06T17:48:08.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:08.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:08.116708+0000) 2024-03-06T17:48:09.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:09.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:09.116947+0000) 2024-03-06T17:48:10.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:10.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:10.117245+0000) 2024-03-06T17:48:11.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:11.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:11.117534+0000) 2024-03-06T17:48:12.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:12.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:12.117848+0000) 2024-03-06T17:48:13.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:13.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:13.118135+0000) 2024-03-06T17:48:14.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:14.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:14.118430+0000) 2024-03-06T17:48:15.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:15.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:15.118769+0000) 2024-03-06T17:48:16.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:16.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:16.119063+0000) 2024-03-06T17:48:17.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:17.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:17.119377+0000) 2024-03-06T17:48:18.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:18.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:18.119737+0000) 2024-03-06T17:48:19.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:19.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:19.120012+0000) 2024-03-06T17:48:20.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:20.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:20.120271+0000) 2024-03-06T17:48:21.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:21.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:21.120529+0000) 2024-03-06T17:48:22.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:22.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:22.120764+0000) 2024-03-06T17:48:23.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:23.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:23.120981+0000) 2024-03-06T17:48:24.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:24.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:24.121299+0000) 2024-03-06T17:48:25.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:25.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:25.121597+0000) 2024-03-06T17:48:26.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:26.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:26.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:26.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:26.121901+0000) 2024-03-06T17:48:27.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:27.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:27.122207+0000) 2024-03-06T17:48:28.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:28.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:28.122502+0000) 2024-03-06T17:48:29.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:29.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:29.122759+0000) 2024-03-06T17:48:30.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:30.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:30.122991+0000) 2024-03-06T17:48:31.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:31.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:31.123234+0000) 2024-03-06T17:48:32.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:32.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:32.123460+0000) 2024-03-06T17:48:33.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:33.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:33.123743+0000) 2024-03-06T17:48:34.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:34.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:34.123996+0000) 2024-03-06T17:48:35.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:35.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:35.124252+0000) 2024-03-06T17:48:36.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:36.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:36.124456+0000) 2024-03-06T17:48:37.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:37.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:37.124760+0000) 2024-03-06T17:48:38.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:38.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:38.125021+0000) 2024-03-06T17:48:39.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:39.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:39.125284+0000) 2024-03-06T17:48:40.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:40.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:40.125523+0000) 2024-03-06T17:48:41.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:41.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:41.125778+0000) 2024-03-06T17:48:42.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:42.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:42.126050+0000) 2024-03-06T17:48:43.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:43.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:43.126363+0000) 2024-03-06T17:48:44.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:44.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:44.126733+0000) 2024-03-06T17:48:45.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:45.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:45.127046+0000) 2024-03-06T17:48:46.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:46.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:46.127350+0000) 2024-03-06T17:48:47.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:47.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:47.127681+0000) 2024-03-06T17:48:48.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:48.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:48.128001+0000) 2024-03-06T17:48:49.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:49.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:49.128298+0000) 2024-03-06T17:48:50.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:50.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:50.128589+0000) 2024-03-06T17:48:51.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:51.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:51.128935+0000) 2024-03-06T17:48:52.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:52.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:52.129240+0000) 2024-03-06T17:48:53.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:53.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:53.129529+0000) 2024-03-06T17:48:54.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:54.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:54.129826+0000) 2024-03-06T17:48:55.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:55.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:55.130133+0000) 2024-03-06T17:48:56.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:56.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:56.130385+0000) 2024-03-06T17:48:56.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:56.184 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:57.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:57.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:57.130727+0000) 2024-03-06T17:48:58.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:58.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:58.131027+0000) 2024-03-06T17:48:59.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:48:59.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:59.131324+0000) 2024-03-06T17:49:00.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:00.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:00.131675+0000) 2024-03-06T17:49:01.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:01.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:01.131993+0000) 2024-03-06T17:49:02.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:02.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:02.132301+0000) 2024-03-06T17:49:03.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:03.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:03.132575+0000) 2024-03-06T17:49:04.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:04.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:04.132927+0000) 2024-03-06T17:49:05.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:05.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:05.133235+0000) 2024-03-06T17:49:06.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:06.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:06.133545+0000) 2024-03-06T17:49:07.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:07.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:07.133836+0000) 2024-03-06T17:49:08.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:08.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:08.134143+0000) 2024-03-06T17:49:09.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:09.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:09.134431+0000) 2024-03-06T17:49:10.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:10.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:10.134775+0000) 2024-03-06T17:49:11.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:11.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:11.135058+0000) 2024-03-06T17:49:12.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:12.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:12.135378+0000) 2024-03-06T17:49:13.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:13.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:13.135735+0000) 2024-03-06T17:49:14.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:14.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:14.136073+0000) 2024-03-06T17:49:15.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:15.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:15.136384+0000) 2024-03-06T17:49:16.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:16.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:16.136745+0000) 2024-03-06T17:49:17.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:17.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:17.137039+0000) 2024-03-06T17:49:18.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:18.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:18.137348+0000) 2024-03-06T17:49:19.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:19.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:19.137667+0000) 2024-03-06T17:49:20.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:20.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:20.137905+0000) 2024-03-06T17:49:21.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:21.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:21.138143+0000) 2024-03-06T17:49:22.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:22.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:22.138384+0000) 2024-03-06T17:49:23.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:23.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:23.138663+0000) 2024-03-06T17:49:24.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:24.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:24.138931+0000) 2024-03-06T17:49:25.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:25.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:25.139194+0000) 2024-03-06T17:49:26.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:26.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:26.139450+0000) 2024-03-06T17:49:26.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:26.327 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:27.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:27.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:27.139754+0000) 2024-03-06T17:49:28.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:28.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:28.139998+0000) 2024-03-06T17:49:29.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:29.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:29.140255+0000) 2024-03-06T17:49:30.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:30.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:30.140481+0000) 2024-03-06T17:49:31.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:31.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:31.140742+0000) 2024-03-06T17:49:32.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:32.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:32.140983+0000) 2024-03-06T17:49:33.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:33.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:33.141236+0000) 2024-03-06T17:49:34.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:34.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:34.141506+0000) 2024-03-06T17:49:35.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:35.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:35.141763+0000) 2024-03-06T17:49:36.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:36.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:36.141955+0000) 2024-03-06T17:49:37.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:37.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:37.142192+0000) 2024-03-06T17:49:38.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:38.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:38.142429+0000) 2024-03-06T17:49:39.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:39.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:39.142737+0000) 2024-03-06T17:49:40.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:40.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:40.142972+0000) 2024-03-06T17:49:41.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:41.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:41.143270+0000) 2024-03-06T17:49:42.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:42.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:42.143525+0000) 2024-03-06T17:49:43.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:43.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:43.143763+0000) 2024-03-06T17:49:44.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:44.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:44.144034+0000) 2024-03-06T17:49:45.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:45.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:45.144292+0000) 2024-03-06T17:49:46.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:46.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:46.144543+0000) 2024-03-06T17:49:47.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:47.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:47.144800+0000) 2024-03-06T17:49:48.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:48.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:48.145068+0000) 2024-03-06T17:49:49.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:49.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:49.145330+0000) 2024-03-06T17:49:50.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:50.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:50.145564+0000) 2024-03-06T17:49:51.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:51.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:51.145804+0000) 2024-03-06T17:49:52.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:52.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:52.146035+0000) 2024-03-06T17:49:53.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:53.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:53.146293+0000) 2024-03-06T17:49:54.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:54.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:54.146569+0000) 2024-03-06T17:49:55.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:55.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:55.146813+0000) 2024-03-06T17:49:56.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:56.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:56.147067+0000) 2024-03-06T17:49:56.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:56.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:57.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:57.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:57.147315+0000) 2024-03-06T17:49:58.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:58.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:58.147500+0000) 2024-03-06T17:49:59.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:49:59.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:59.147748+0000) 2024-03-06T17:50:00.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:00.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:00.148000+0000) 2024-03-06T17:50:01.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:01.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:01.148227+0000) 2024-03-06T17:50:02.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:02.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:02.148470+0000) 2024-03-06T17:50:03.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:03.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:03.148737+0000) 2024-03-06T17:50:04.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:04.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:04.148994+0000) 2024-03-06T17:50:05.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:05.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:05.149233+0000) 2024-03-06T17:50:06.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:06.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:06.149477+0000) 2024-03-06T17:50:07.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:07.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:07.149753+0000) 2024-03-06T17:50:08.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:08.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:08.150014+0000) 2024-03-06T17:50:09.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:09.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:09.150277+0000) 2024-03-06T17:50:10.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:10.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:10.150531+0000) 2024-03-06T17:50:11.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:11.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:11.150791+0000) 2024-03-06T17:50:12.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:12.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:12.151046+0000) 2024-03-06T17:50:13.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:13.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:13.151299+0000) 2024-03-06T17:50:14.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:14.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:14.151560+0000) 2024-03-06T17:50:15.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:15.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:15.151820+0000) 2024-03-06T17:50:16.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:16.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:16.152078+0000) 2024-03-06T17:50:17.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:17.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:17.152310+0000) 2024-03-06T17:50:18.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:18.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:18.152573+0000) 2024-03-06T17:50:19.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:19.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:19.152835+0000) 2024-03-06T17:50:20.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:20.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:20.153068+0000) 2024-03-06T17:50:21.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:21.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:21.153251+0000) 2024-03-06T17:50:22.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:22.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:22.153489+0000) 2024-03-06T17:50:23.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:23.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:23.153740+0000) 2024-03-06T17:50:24.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:24.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:24.153997+0000) 2024-03-06T17:50:25.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:25.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:25.154252+0000) 2024-03-06T17:50:26.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:26.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:26.154510+0000) 2024-03-06T17:50:26.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:26.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:27.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:27.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:27.154752+0000) 2024-03-06T17:50:28.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:28.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:28.155006+0000) 2024-03-06T17:50:29.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:29.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:29.155200+0000) 2024-03-06T17:50:30.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:30.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:30.155437+0000) 2024-03-06T17:50:31.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:31.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:31.155670+0000) 2024-03-06T17:50:32.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:32.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:32.155926+0000) 2024-03-06T17:50:33.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:33.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:33.156181+0000) 2024-03-06T17:50:34.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:34.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:34.156454+0000) 2024-03-06T17:50:35.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:35.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:35.156723+0000) 2024-03-06T17:50:36.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:36.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:36.156959+0000) 2024-03-06T17:50:37.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:37.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:37.157225+0000) 2024-03-06T17:50:38.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:38.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:38.157483+0000) 2024-03-06T17:50:39.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:39.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:39.157757+0000) 2024-03-06T17:50:40.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:40.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:40.158013+0000) 2024-03-06T17:50:41.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:41.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:41.158276+0000) 2024-03-06T17:50:42.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:42.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:42.158540+0000) 2024-03-06T17:50:43.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:43.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:43.158796+0000) 2024-03-06T17:50:44.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:44.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:44.159066+0000) 2024-03-06T17:50:45.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:45.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:45.159316+0000) 2024-03-06T17:50:46.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:46.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:46.159575+0000) 2024-03-06T17:50:47.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:47.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:47.159831+0000) 2024-03-06T17:50:48.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:48.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:48.160070+0000) 2024-03-06T17:50:49.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:49.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:49.160323+0000) 2024-03-06T17:50:50.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:50.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:50.160564+0000) 2024-03-06T17:50:51.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:51.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:51.160809+0000) 2024-03-06T17:50:52.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:52.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:52.161057+0000) 2024-03-06T17:50:53.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:53.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:53.161309+0000) 2024-03-06T17:50:54.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:54.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:54.161562+0000) 2024-03-06T17:50:55.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:55.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:55.161814+0000) 2024-03-06T17:50:56.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:56.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:56.162071+0000) 2024-03-06T17:50:56.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:56.785 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:57.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:57.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:57.162339+0000) 2024-03-06T17:50:58.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:58.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:58.162601+0000) 2024-03-06T17:50:59.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:50:59.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:59.162824+0000) 2024-03-06T17:51:00.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:00.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:00.163061+0000) 2024-03-06T17:51:01.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:01.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:01.163309+0000) 2024-03-06T17:51:02.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:02.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:02.163537+0000) 2024-03-06T17:51:03.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:03.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:03.163803+0000) 2024-03-06T17:51:04.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:04.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:04.164069+0000) 2024-03-06T17:51:05.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:05.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:05.164331+0000) 2024-03-06T17:51:06.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:06.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:06.164592+0000) 2024-03-06T17:51:07.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:07.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:07.164854+0000) 2024-03-06T17:51:08.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:08.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:08.165106+0000) 2024-03-06T17:51:09.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:09.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:09.165374+0000) 2024-03-06T17:51:10.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:10.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:10.165613+0000) 2024-03-06T17:51:11.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:11.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:11.165943+0000) 2024-03-06T17:51:12.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:12.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:12.166215+0000) 2024-03-06T17:51:13.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:13.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:13.166469+0000) 2024-03-06T17:51:14.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:14.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:14.166755+0000) 2024-03-06T17:51:15.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:15.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:15.166948+0000) 2024-03-06T17:51:16.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:16.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:16.167205+0000) 2024-03-06T17:51:17.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:17.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:17.167467+0000) 2024-03-06T17:51:18.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:18.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:18.167737+0000) 2024-03-06T17:51:19.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:19.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:19.167982+0000) 2024-03-06T17:51:20.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:20.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:20.168226+0000) 2024-03-06T17:51:21.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:21.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:21.168480+0000) 2024-03-06T17:51:22.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:22.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:22.168740+0000) 2024-03-06T17:51:23.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:23.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:23.168937+0000) 2024-03-06T17:51:24.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:24.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:24.169200+0000) 2024-03-06T17:51:25.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:25.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:25.169440+0000) 2024-03-06T17:51:26.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:26.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:26.169736+0000) 2024-03-06T17:51:26.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:26.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:27.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:27.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:27.170001+0000) 2024-03-06T17:51:28.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:28.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:28.170244+0000) 2024-03-06T17:51:29.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:29.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:29.170506+0000) 2024-03-06T17:51:30.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:30.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:30.170763+0000) 2024-03-06T17:51:31.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:31.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:31.171019+0000) 2024-03-06T17:51:32.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:32.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:32.171218+0000) 2024-03-06T17:51:33.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:33.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:33.171486+0000) 2024-03-06T17:51:34.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:34.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:34.171762+0000) 2024-03-06T17:51:35.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:35.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:35.171997+0000) 2024-03-06T17:51:36.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:36.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:36.172237+0000) 2024-03-06T17:51:37.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:37.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:37.172492+0000) 2024-03-06T17:51:38.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:38.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:38.172755+0000) 2024-03-06T17:51:39.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:39.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:39.172998+0000) 2024-03-06T17:51:40.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:40.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:40.173233+0000) 2024-03-06T17:51:41.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:41.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:41.173497+0000) 2024-03-06T17:51:42.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:42.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:42.173751+0000) 2024-03-06T17:51:43.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:43.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:43.174005+0000) 2024-03-06T17:51:44.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:44.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:44.174267+0000) 2024-03-06T17:51:45.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:45.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:45.174503+0000) 2024-03-06T17:51:46.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:46.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:46.174759+0000) 2024-03-06T17:51:47.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:47.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:47.175002+0000) 2024-03-06T17:51:48.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:48.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:48.175188+0000) 2024-03-06T17:51:49.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:49.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:49.175429+0000) 2024-03-06T17:51:50.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:50.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:50.175702+0000) 2024-03-06T17:51:51.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:51.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:51.175953+0000) 2024-03-06T17:51:52.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:52.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:52.176181+0000) 2024-03-06T17:51:53.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:53.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:53.176438+0000) 2024-03-06T17:51:54.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:54.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:54.176737+0000) 2024-03-06T17:51:55.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:55.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:55.176990+0000) 2024-03-06T17:51:56.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:56.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:56.177230+0000) 2024-03-06T17:51:57.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:57.090 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:57.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:57.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:57.177418+0000) 2024-03-06T17:51:58.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:58.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:58.177724+0000) 2024-03-06T17:51:59.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:51:59.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:59.177988+0000) 2024-03-06T17:52:00.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:00.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:00.178240+0000) 2024-03-06T17:52:01.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:01.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:01.178503+0000) 2024-03-06T17:52:02.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:02.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:02.178741+0000) 2024-03-06T17:52:03.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:03.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:03.178999+0000) 2024-03-06T17:52:04.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:04.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:04.179250+0000) 2024-03-06T17:52:05.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:05.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:05.179499+0000) 2024-03-06T17:52:06.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:06.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:06.179735+0000) 2024-03-06T17:52:07.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:07.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:07.179989+0000) 2024-03-06T17:52:08.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:08.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:08.180226+0000) 2024-03-06T17:52:09.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:09.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:09.180469+0000) 2024-03-06T17:52:10.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:10.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:10.180736+0000) 2024-03-06T17:52:11.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:11.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:11.180975+0000) 2024-03-06T17:52:12.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:12.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:12.181231+0000) 2024-03-06T17:52:13.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:13.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:13.181495+0000) 2024-03-06T17:52:14.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:14.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:14.181757+0000) 2024-03-06T17:52:15.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:15.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:15.182013+0000) 2024-03-06T17:52:16.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:16.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:16.182269+0000) 2024-03-06T17:52:17.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:17.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:17.182458+0000) 2024-03-06T17:52:18.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:18.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:18.182670+0000) 2024-03-06T17:52:19.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:19.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:19.182910+0000) 2024-03-06T17:52:20.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:20.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:20.183154+0000) 2024-03-06T17:52:21.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:21.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:21.183413+0000) 2024-03-06T17:52:22.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:22.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:22.183685+0000) 2024-03-06T17:52:23.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:23.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:23.183940+0000) 2024-03-06T17:52:24.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:24.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:24.184205+0000) 2024-03-06T17:52:25.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:25.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:25.184460+0000) 2024-03-06T17:52:26.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:26.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:26.184737+0000) 2024-03-06T17:52:27.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:27.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:27.184987+0000) 2024-03-06T17:52:27.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:27.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:28.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:28.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:28.185253+0000) 2024-03-06T17:52:29.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:29.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:29.185507+0000) 2024-03-06T17:52:30.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:30.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:30.185757+0000) 2024-03-06T17:52:31.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:31.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:31.185953+0000) 2024-03-06T17:52:32.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:32.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:32.186198+0000) 2024-03-06T17:52:33.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:33.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:33.186434+0000) 2024-03-06T17:52:34.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:34.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:34.186734+0000) 2024-03-06T17:52:35.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:35.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:35.186928+0000) 2024-03-06T17:52:36.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:36.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:36.187180+0000) 2024-03-06T17:52:37.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:37.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:37.187439+0000) 2024-03-06T17:52:38.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:38.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:38.187731+0000) 2024-03-06T17:52:39.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:39.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:39.187923+0000) 2024-03-06T17:52:40.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:40.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:40.188165+0000) 2024-03-06T17:52:41.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:41.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:41.188401+0000) 2024-03-06T17:52:42.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:42.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:42.188675+0000) 2024-03-06T17:52:43.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:43.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:43.188904+0000) 2024-03-06T17:52:44.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:44.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:44.189170+0000) 2024-03-06T17:52:45.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:45.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:45.189426+0000) 2024-03-06T17:52:46.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:46.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:46.189730+0000) 2024-03-06T17:52:47.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:47.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:47.189994+0000) 2024-03-06T17:52:48.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:48.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:48.190231+0000) 2024-03-06T17:52:49.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:49.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:49.190470+0000) 2024-03-06T17:52:50.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:50.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:50.190682+0000) 2024-03-06T17:52:51.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:51.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:51.190935+0000) 2024-03-06T17:52:52.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:52.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:52.191154+0000) 2024-03-06T17:52:53.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:53.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:53.191416+0000) 2024-03-06T17:52:54.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:54.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:54.191720+0000) 2024-03-06T17:52:55.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:55.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:55.191974+0000) 2024-03-06T17:52:56.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:56.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:56.192226+0000) 2024-03-06T17:52:57.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:57.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:57.192462+0000) 2024-03-06T17:52:57.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:57.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:58.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:58.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:58.192786+0000) 2024-03-06T17:52:59.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:52:59.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:59.193042+0000) 2024-03-06T17:53:00.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:00.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:00.193295+0000) 2024-03-06T17:53:01.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:01.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:01.193534+0000) 2024-03-06T17:53:02.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:02.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:02.193794+0000) 2024-03-06T17:53:03.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:03.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:03.194066+0000) 2024-03-06T17:53:04.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:04.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:04.194353+0000) 2024-03-06T17:53:05.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:05.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:05.194599+0000) 2024-03-06T17:53:06.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:06.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:06.194853+0000) 2024-03-06T17:53:07.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:07.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:07.195068+0000) 2024-03-06T17:53:08.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:08.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:08.195320+0000) 2024-03-06T17:53:09.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:09.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:09.195616+0000) 2024-03-06T17:53:10.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:10.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:10.195922+0000) 2024-03-06T17:53:11.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:11.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:11.196163+0000) 2024-03-06T17:53:12.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:12.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:12.196418+0000) 2024-03-06T17:53:13.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:13.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:13.196715+0000) 2024-03-06T17:53:14.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:14.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:14.196976+0000) 2024-03-06T17:53:15.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:15.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:15.197237+0000) 2024-03-06T17:53:16.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:16.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:16.197475+0000) 2024-03-06T17:53:17.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:17.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:17.197753+0000) 2024-03-06T17:53:18.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:18.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:18.198018+0000) 2024-03-06T17:53:19.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:19.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:19.198289+0000) 2024-03-06T17:53:20.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:20.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:20.198508+0000) 2024-03-06T17:53:21.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:21.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:21.198783+0000) 2024-03-06T17:53:22.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:22.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:22.199058+0000) 2024-03-06T17:53:23.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:23.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:23.199337+0000) 2024-03-06T17:53:24.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:24.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:24.199622+0000) 2024-03-06T17:53:25.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:25.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:25.199855+0000) 2024-03-06T17:53:26.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:26.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:26.200114+0000) 2024-03-06T17:53:27.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:27.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:27.200370+0000) 2024-03-06T17:53:27.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:27.533 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:28.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:28.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:28.200661+0000) 2024-03-06T17:53:29.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:29.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:29.200923+0000) 2024-03-06T17:53:30.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:30.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:30.201177+0000) 2024-03-06T17:53:31.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:31.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:31.201433+0000) 2024-03-06T17:53:32.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:32.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:32.201713+0000) 2024-03-06T17:53:33.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:33.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:33.201981+0000) 2024-03-06T17:53:34.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:34.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:34.202247+0000) 2024-03-06T17:53:35.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:35.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:35.202510+0000) 2024-03-06T17:53:36.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:36.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:36.202778+0000) 2024-03-06T17:53:37.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:37.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:37.203010+0000) 2024-03-06T17:53:38.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:38.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:38.203197+0000) 2024-03-06T17:53:39.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:39.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:39.203459+0000) 2024-03-06T17:53:40.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:40.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:40.203727+0000) 2024-03-06T17:53:41.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:41.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:41.203972+0000) 2024-03-06T17:53:42.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:42.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:42.204208+0000) 2024-03-06T17:53:43.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:43.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:43.204442+0000) 2024-03-06T17:53:44.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:44.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:44.204725+0000) 2024-03-06T17:53:45.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:45.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:45.204962+0000) 2024-03-06T17:53:46.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:46.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:46.205203+0000) 2024-03-06T17:53:47.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:47.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:47.205448+0000) 2024-03-06T17:53:48.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:48.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:48.205721+0000) 2024-03-06T17:53:49.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:49.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:49.205974+0000) 2024-03-06T17:53:50.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:50.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:50.206225+0000) 2024-03-06T17:53:51.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:51.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:51.206521+0000) 2024-03-06T17:53:52.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:52.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:52.206771+0000) 2024-03-06T17:53:53.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:53.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:53.207028+0000) 2024-03-06T17:53:54.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:54.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:54.207286+0000) 2024-03-06T17:53:55.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:55.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:55.207544+0000) 2024-03-06T17:53:56.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:56.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:56.207787+0000) 2024-03-06T17:53:57.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:57.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:57.208028+0000) 2024-03-06T17:53:57.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:57.808 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:58.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:58.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:58.208289+0000) 2024-03-06T17:53:59.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:53:59.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:59.208514+0000) 2024-03-06T17:54:00.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:00.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:00.208777+0000) 2024-03-06T17:54:01.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:01.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:01.209014+0000) 2024-03-06T17:54:02.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:02.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:02.209257+0000) 2024-03-06T17:54:03.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:03.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:03.209493+0000) 2024-03-06T17:54:04.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:04.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:04.209757+0000) 2024-03-06T17:54:05.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:05.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:05.210020+0000) 2024-03-06T17:54:06.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:06.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:06.210282+0000) 2024-03-06T17:54:07.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:07.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:07.210539+0000) 2024-03-06T17:54:08.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:08.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:08.210793+0000) 2024-03-06T17:54:09.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:09.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:09.211048+0000) 2024-03-06T17:54:10.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:10.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:10.211305+0000) 2024-03-06T17:54:11.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:11.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:11.211565+0000) 2024-03-06T17:54:12.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:12.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:12.211826+0000) 2024-03-06T17:54:13.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:13.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:13.212090+0000) 2024-03-06T17:54:14.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:14.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:14.212336+0000) 2024-03-06T17:54:15.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:15.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:15.212578+0000) 2024-03-06T17:54:16.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:16.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:16.212832+0000) 2024-03-06T17:54:17.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:17.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:17.213075+0000) 2024-03-06T17:54:18.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:18.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:18.213331+0000) 2024-03-06T17:54:19.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:19.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:19.213589+0000) 2024-03-06T17:54:20.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:20.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:20.213848+0000) 2024-03-06T17:54:21.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:21.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:21.214079+0000) 2024-03-06T17:54:22.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:22.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:22.214312+0000) 2024-03-06T17:54:23.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:23.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:23.214573+0000) 2024-03-06T17:54:24.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:24.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:24.214837+0000) 2024-03-06T17:54:25.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:25.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:25.215103+0000) 2024-03-06T17:54:26.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:26.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:26.215354+0000) 2024-03-06T17:54:27.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:27.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:27.215614+0000) 2024-03-06T17:54:27.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:27.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:28.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:28.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:28.215912+0000) 2024-03-06T17:54:29.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:29.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:29.216178+0000) 2024-03-06T17:54:30.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:30.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:30.216405+0000) 2024-03-06T17:54:31.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:31.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:31.216672+0000) 2024-03-06T17:54:32.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:32.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:32.216916+0000) 2024-03-06T17:54:33.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:33.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:33.217169+0000) 2024-03-06T17:54:34.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:34.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:34.217444+0000) 2024-03-06T17:54:35.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:35.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:35.217738+0000) 2024-03-06T17:54:36.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:36.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:36.217977+0000) 2024-03-06T17:54:37.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:37.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:37.218241+0000) 2024-03-06T17:54:38.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:38.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:38.218482+0000) 2024-03-06T17:54:39.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:39.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:39.218820+0000) 2024-03-06T17:54:40.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:40.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:40.219084+0000) 2024-03-06T17:54:41.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:41.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:41.219350+0000) 2024-03-06T17:54:42.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:42.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:42.219705+0000) 2024-03-06T17:54:43.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:43.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:43.219998+0000) 2024-03-06T17:54:44.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:44.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:44.220251+0000) 2024-03-06T17:54:45.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:45.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:45.220553+0000) 2024-03-06T17:54:46.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:46.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:46.220868+0000) 2024-03-06T17:54:47.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:47.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:47.221159+0000) 2024-03-06T17:54:48.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:48.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:48.221459+0000) 2024-03-06T17:54:49.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:49.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:49.221789+0000) 2024-03-06T17:54:50.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:50.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:50.222107+0000) 2024-03-06T17:54:51.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:51.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:51.222424+0000) 2024-03-06T17:54:52.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:52.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:52.222758+0000) 2024-03-06T17:54:53.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:53.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:53.223069+0000) 2024-03-06T17:54:54.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:54.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:54.223394+0000) 2024-03-06T17:54:55.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:55.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:55.223748+0000) 2024-03-06T17:54:56.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:56.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:56.224064+0000) 2024-03-06T17:54:57.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:57.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:57.224364+0000) 2024-03-06T17:54:58.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:58.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:58.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:58.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:58.224672+0000) 2024-03-06T17:54:59.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:54:59.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:59.224893+0000) 2024-03-06T17:55:00.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:00.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:00.225215+0000) 2024-03-06T17:55:01.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:01.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:01.225506+0000) 2024-03-06T17:55:02.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:02.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:02.225775+0000) 2024-03-06T17:55:03.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:03.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:03.226087+0000) 2024-03-06T17:55:04.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:04.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:04.226422+0000) 2024-03-06T17:55:05.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:05.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:05.226778+0000) 2024-03-06T17:55:06.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:06.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:06.227057+0000) 2024-03-06T17:55:07.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:07.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:07.227352+0000) 2024-03-06T17:55:08.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:08.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:08.227689+0000) 2024-03-06T17:55:09.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:09.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:09.227983+0000) 2024-03-06T17:55:10.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:10.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:10.228290+0000) 2024-03-06T17:55:11.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:11.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:11.228610+0000) 2024-03-06T17:55:12.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:12.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:12.228967+0000) 2024-03-06T17:55:13.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:13.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:13.229285+0000) 2024-03-06T17:55:14.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:14.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:14.229601+0000) 2024-03-06T17:55:15.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:15.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:15.229943+0000) 2024-03-06T17:55:16.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:16.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:16.230255+0000) 2024-03-06T17:55:17.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:17.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:17.230558+0000) 2024-03-06T17:55:18.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:18.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:18.230855+0000) 2024-03-06T17:55:19.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:19.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:19.231084+0000) 2024-03-06T17:55:20.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:20.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:20.231397+0000) 2024-03-06T17:55:21.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:21.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:21.231749+0000) 2024-03-06T17:55:22.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:22.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:22.232101+0000) 2024-03-06T17:55:23.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:23.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:23.232396+0000) 2024-03-06T17:55:24.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:24.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:24.232766+0000) 2024-03-06T17:55:25.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:25.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:25.233078+0000) 2024-03-06T17:55:26.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:26.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:26.233366+0000) 2024-03-06T17:55:27.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:27.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:27.233703+0000) 2024-03-06T17:55:28.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:28.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:28.233995+0000) 2024-03-06T17:55:28.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:28.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:29.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:29.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:29.234296+0000) 2024-03-06T17:55:30.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:30.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:30.234617+0000) 2024-03-06T17:55:31.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:31.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:31.234975+0000) 2024-03-06T17:55:32.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:32.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:32.235258+0000) 2024-03-06T17:55:33.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:33.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:33.235567+0000) 2024-03-06T17:55:34.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:34.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:34.235886+0000) 2024-03-06T17:55:35.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:35.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:35.236170+0000) 2024-03-06T17:55:36.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:36.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:36.236483+0000) 2024-03-06T17:55:37.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:37.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:37.236800+0000) 2024-03-06T17:55:38.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:38.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:38.237107+0000) 2024-03-06T17:55:39.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:39.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:39.237417+0000) 2024-03-06T17:55:40.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:40.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:40.237776+0000) 2024-03-06T17:55:41.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:41.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:41.238069+0000) 2024-03-06T17:55:42.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:42.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:42.238294+0000) 2024-03-06T17:55:43.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:43.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:43.238581+0000) 2024-03-06T17:55:44.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:44.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:44.238914+0000) 2024-03-06T17:55:45.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:45.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:45.239228+0000) 2024-03-06T17:55:46.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:46.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:46.239505+0000) 2024-03-06T17:55:47.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:47.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:47.239748+0000) 2024-03-06T17:55:48.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:48.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:48.240004+0000) 2024-03-06T17:55:49.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:49.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:49.240264+0000) 2024-03-06T17:55:50.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:50.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:50.240534+0000) 2024-03-06T17:55:51.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:51.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:51.240775+0000) 2024-03-06T17:55:52.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:52.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:52.241032+0000) 2024-03-06T17:55:53.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:53.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:53.241268+0000) 2024-03-06T17:55:54.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:54.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:54.241541+0000) 2024-03-06T17:55:55.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:55.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:55.241811+0000) 2024-03-06T17:55:56.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:56.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:56.242069+0000) 2024-03-06T17:55:57.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:57.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:57.242308+0000) 2024-03-06T17:55:58.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:58.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:58.242544+0000) 2024-03-06T17:55:58.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:58.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:59.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:55:59.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:59.242789+0000) 2024-03-06T17:56:00.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:00.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:00.243026+0000) 2024-03-06T17:56:01.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:01.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:01.243224+0000) 2024-03-06T17:56:02.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:02.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:02.243476+0000) 2024-03-06T17:56:03.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:03.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:03.243747+0000) 2024-03-06T17:56:04.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:04.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:04.244024+0000) 2024-03-06T17:56:05.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:05.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:05.244322+0000) 2024-03-06T17:56:06.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:06.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:06.244579+0000) 2024-03-06T17:56:07.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:07.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:07.244820+0000) 2024-03-06T17:56:08.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:08.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:08.245073+0000) 2024-03-06T17:56:09.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:09.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:09.245272+0000) 2024-03-06T17:56:10.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:10.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:10.245466+0000) 2024-03-06T17:56:11.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:11.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:11.245753+0000) 2024-03-06T17:56:12.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:12.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:12.246007+0000) 2024-03-06T17:56:13.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:13.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:13.246261+0000) 2024-03-06T17:56:14.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:14.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:14.246513+0000) 2024-03-06T17:56:15.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:15.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:15.246750+0000) 2024-03-06T17:56:16.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:16.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:16.247017+0000) 2024-03-06T17:56:17.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:17.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:17.247253+0000) 2024-03-06T17:56:18.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:18.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:18.247457+0000) 2024-03-06T17:56:19.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:19.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:19.247757+0000) 2024-03-06T17:56:20.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:20.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:20.248023+0000) 2024-03-06T17:56:21.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:21.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:21.248279+0000) 2024-03-06T17:56:22.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:22.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:22.248519+0000) 2024-03-06T17:56:23.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:23.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:23.248776+0000) 2024-03-06T17:56:24.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:24.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:24.249039+0000) 2024-03-06T17:56:25.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:25.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:25.249292+0000) 2024-03-06T17:56:26.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:26.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:26.249557+0000) 2024-03-06T17:56:27.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:27.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:27.249804+0000) 2024-03-06T17:56:28.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:28.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:28.250039+0000) 2024-03-06T17:56:28.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:28.537 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:29.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:29.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:29.250295+0000) 2024-03-06T17:56:30.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:30.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:30.250496+0000) 2024-03-06T17:56:31.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:31.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:31.250753+0000) 2024-03-06T17:56:32.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:32.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:32.251011+0000) 2024-03-06T17:56:33.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:33.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:33.251247+0000) 2024-03-06T17:56:34.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:34.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:34.251497+0000) 2024-03-06T17:56:35.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:35.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:35.251792+0000) 2024-03-06T17:56:36.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:36.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:36.252073+0000) 2024-03-06T17:56:37.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:37.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:37.252331+0000) 2024-03-06T17:56:38.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:38.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:38.252510+0000) 2024-03-06T17:56:39.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:39.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:39.252766+0000) 2024-03-06T17:56:40.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:40.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:40.253020+0000) 2024-03-06T17:56:41.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:41.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:41.253278+0000) 2024-03-06T17:56:42.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:42.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:42.253540+0000) 2024-03-06T17:56:43.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:43.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:43.253794+0000) 2024-03-06T17:56:44.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:44.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:44.254043+0000) 2024-03-06T17:56:45.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:45.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:45.254296+0000) 2024-03-06T17:56:46.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:46.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:46.254537+0000) 2024-03-06T17:56:47.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:47.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:47.254793+0000) 2024-03-06T17:56:48.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:48.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:48.255027+0000) 2024-03-06T17:56:49.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:49.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:49.255267+0000) 2024-03-06T17:56:50.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:50.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:50.255511+0000) 2024-03-06T17:56:51.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:51.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:51.255770+0000) 2024-03-06T17:56:52.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:52.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:52.255962+0000) 2024-03-06T17:56:53.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:53.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:53.256218+0000) 2024-03-06T17:56:54.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:54.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:54.256482+0000) 2024-03-06T17:56:55.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:55.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:55.256732+0000) 2024-03-06T17:56:56.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:56.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:56.256971+0000) 2024-03-06T17:56:57.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:57.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:57.257214+0000) 2024-03-06T17:56:58.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:58.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:58.257456+0000) 2024-03-06T17:56:58.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:58.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:59.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:56:59.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:59.257760+0000) 2024-03-06T17:57:00.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:00.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:00.258006+0000) 2024-03-06T17:57:01.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:01.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:01.258268+0000) 2024-03-06T17:57:02.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:02.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:02.258503+0000) 2024-03-06T17:57:03.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:03.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:03.258744+0000) 2024-03-06T17:57:04.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:04.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:04.258988+0000) 2024-03-06T17:57:05.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:05.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:05.259180+0000) 2024-03-06T17:57:06.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:06.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:06.259366+0000) 2024-03-06T17:57:07.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:07.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:07.259622+0000) 2024-03-06T17:57:08.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:08.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:08.259911+0000) 2024-03-06T17:57:09.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:09.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:09.260226+0000) 2024-03-06T17:57:10.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:10.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:10.260460+0000) 2024-03-06T17:57:11.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:11.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:11.260684+0000) 2024-03-06T17:57:12.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:12.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:12.260870+0000) 2024-03-06T17:57:13.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:13.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:13.261108+0000) 2024-03-06T17:57:14.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:14.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:14.261377+0000) 2024-03-06T17:57:15.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:15.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:15.261657+0000) 2024-03-06T17:57:16.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:16.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:16.261901+0000) 2024-03-06T17:57:17.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:17.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:17.262157+0000) 2024-03-06T17:57:18.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:18.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:18.262420+0000) 2024-03-06T17:57:19.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:19.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:19.262725+0000) 2024-03-06T17:57:20.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:20.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:20.262973+0000) 2024-03-06T17:57:21.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:21.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:21.263213+0000) 2024-03-06T17:57:22.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:22.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:22.263475+0000) 2024-03-06T17:57:23.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:23.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:23.263752+0000) 2024-03-06T17:57:24.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:24.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:24.264018+0000) 2024-03-06T17:57:25.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:25.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:25.264269+0000) 2024-03-06T17:57:26.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:26.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:26.264530+0000) 2024-03-06T17:57:27.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:27.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:27.264766+0000) 2024-03-06T17:57:28.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:28.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:28.265003+0000) 2024-03-06T17:57:28.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:28.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:29.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:29.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:29.265247+0000) 2024-03-06T17:57:30.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:30.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:30.265498+0000) 2024-03-06T17:57:31.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:31.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:31.265759+0000) 2024-03-06T17:57:32.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:32.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:32.265994+0000) 2024-03-06T17:57:33.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:33.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:33.266218+0000) 2024-03-06T17:57:34.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:34.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:34.266488+0000) 2024-03-06T17:57:35.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:35.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:35.266744+0000) 2024-03-06T17:57:36.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:36.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:36.267003+0000) 2024-03-06T17:57:37.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:37.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:37.267188+0000) 2024-03-06T17:57:38.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:38.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:38.267436+0000) 2024-03-06T17:57:39.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:39.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:39.267658+0000) 2024-03-06T17:57:40.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:40.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:40.267896+0000) 2024-03-06T17:57:41.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:41.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:41.268293+0000) 2024-03-06T17:57:42.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:42.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:42.268565+0000) 2024-03-06T17:57:43.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:43.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:43.268822+0000) 2024-03-06T17:57:44.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:44.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:44.269064+0000) 2024-03-06T17:57:45.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:45.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:45.269301+0000) 2024-03-06T17:57:46.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:46.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:46.269562+0000) 2024-03-06T17:57:47.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:47.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:47.269818+0000) 2024-03-06T17:57:48.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:48.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:48.270055+0000) 2024-03-06T17:57:49.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:49.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:49.270306+0000) 2024-03-06T17:57:50.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:50.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:50.270563+0000) 2024-03-06T17:57:51.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:51.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:51.270824+0000) 2024-03-06T17:57:52.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:52.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:52.271088+0000) 2024-03-06T17:57:53.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:53.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:53.271332+0000) 2024-03-06T17:57:54.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:54.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:54.271595+0000) 2024-03-06T17:57:55.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:55.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:55.271854+0000) 2024-03-06T17:57:56.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:56.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:56.272104+0000) 2024-03-06T17:57:57.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:57.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:57.272399+0000) 2024-03-06T17:57:58.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:58.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:58.272675+0000) 2024-03-06T17:57:59.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:59.022 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:59.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:57:59.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:59.272937+0000) 2024-03-06T17:58:00.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:00.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:00.273189+0000) 2024-03-06T17:58:01.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:01.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:01.273445+0000) 2024-03-06T17:58:02.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:02.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:02.273689+0000) 2024-03-06T17:58:03.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:03.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:03.273938+0000) 2024-03-06T17:58:04.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:04.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:04.274206+0000) 2024-03-06T17:58:05.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:05.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:05.274459+0000) 2024-03-06T17:58:06.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:06.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:06.274737+0000) 2024-03-06T17:58:07.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:07.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:07.275004+0000) 2024-03-06T17:58:08.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:08.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:08.275238+0000) 2024-03-06T17:58:09.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:09.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:09.275499+0000) 2024-03-06T17:58:10.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:10.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:10.275756+0000) 2024-03-06T17:58:11.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:11.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:11.276016+0000) 2024-03-06T17:58:12.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:12.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:12.276258+0000) 2024-03-06T17:58:13.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:13.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:13.276505+0000) 2024-03-06T17:58:14.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:14.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:14.276753+0000) 2024-03-06T17:58:15.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:15.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:15.277003+0000) 2024-03-06T17:58:16.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:16.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:16.277200+0000) 2024-03-06T17:58:17.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:17.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:17.277441+0000) 2024-03-06T17:58:18.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:18.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:18.277736+0000) 2024-03-06T17:58:19.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:19.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:19.277993+0000) 2024-03-06T17:58:20.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:20.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:20.278182+0000) 2024-03-06T17:58:21.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:21.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:21.278442+0000) 2024-03-06T17:58:22.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:22.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:22.278733+0000) 2024-03-06T17:58:23.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:23.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:23.278993+0000) 2024-03-06T17:58:24.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:24.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:24.279234+0000) 2024-03-06T17:58:25.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:25.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:25.279485+0000) 2024-03-06T17:58:26.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:26.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:26.279757+0000) 2024-03-06T17:58:27.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:27.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:27.280001+0000) 2024-03-06T17:58:28.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:28.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:28.280235+0000) 2024-03-06T17:58:29.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:29.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:29.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:29.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:29.280493+0000) 2024-03-06T17:58:30.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:30.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:30.280763+0000) 2024-03-06T17:58:31.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:31.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:31.280972+0000) 2024-03-06T17:58:32.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:32.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:32.281214+0000) 2024-03-06T17:58:33.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:33.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:33.281479+0000) 2024-03-06T17:58:34.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:34.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:34.281747+0000) 2024-03-06T17:58:35.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:35.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:35.282011+0000) 2024-03-06T17:58:36.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:36.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:36.282288+0000) 2024-03-06T17:58:37.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:37.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:37.282532+0000) 2024-03-06T17:58:38.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:38.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:38.282796+0000) 2024-03-06T17:58:39.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:39.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:39.283049+0000) 2024-03-06T17:58:40.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:40.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:40.283245+0000) 2024-03-06T17:58:41.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:41.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:41.283495+0000) 2024-03-06T17:58:42.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:42.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:42.283752+0000) 2024-03-06T17:58:43.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:43.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:43.284029+0000) 2024-03-06T17:58:44.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:44.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:44.284282+0000) 2024-03-06T17:58:45.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:45.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:45.284539+0000) 2024-03-06T17:58:46.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:46.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:46.284808+0000) 2024-03-06T17:58:47.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:47.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:47.285060+0000) 2024-03-06T17:58:48.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:48.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:48.285302+0000) 2024-03-06T17:58:49.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:49.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:49.285569+0000) 2024-03-06T17:58:50.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:50.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:50.285857+0000) 2024-03-06T17:58:51.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:51.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:51.286133+0000) 2024-03-06T17:58:52.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:52.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:52.286369+0000) 2024-03-06T17:58:53.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:53.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:53.286612+0000) 2024-03-06T17:58:54.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:54.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:54.286932+0000) 2024-03-06T17:58:55.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:55.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:55.287181+0000) 2024-03-06T17:58:56.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:56.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:56.287412+0000) 2024-03-06T17:58:57.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:57.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:57.287710+0000) 2024-03-06T17:58:58.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:58.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:58.287938+0000) 2024-03-06T17:58:59.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:58:59.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:59.288197+0000) 2024-03-06T17:58:59.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:59.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:00.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:00.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:00.288443+0000) 2024-03-06T17:59:01.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:01.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:01.288733+0000) 2024-03-06T17:59:02.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:02.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:02.288969+0000) 2024-03-06T17:59:03.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:03.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:03.289224+0000) 2024-03-06T17:59:04.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:04.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:04.289462+0000) 2024-03-06T17:59:05.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:05.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:05.289753+0000) 2024-03-06T17:59:06.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:06.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:06.290001+0000) 2024-03-06T17:59:07.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:07.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:07.290236+0000) 2024-03-06T17:59:08.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:08.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:08.290485+0000) 2024-03-06T17:59:09.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:09.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:09.290745+0000) 2024-03-06T17:59:10.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:10.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:10.290984+0000) 2024-03-06T17:59:11.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:11.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:11.291260+0000) 2024-03-06T17:59:12.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:12.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:12.291509+0000) 2024-03-06T17:59:13.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:13.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:13.291746+0000) 2024-03-06T17:59:14.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:14.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:14.292007+0000) 2024-03-06T17:59:15.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:15.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:15.292278+0000) 2024-03-06T17:59:16.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:16.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:16.292530+0000) 2024-03-06T17:59:17.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:17.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:17.292816+0000) 2024-03-06T17:59:18.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:18.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:18.293062+0000) 2024-03-06T17:59:19.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:19.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:19.293334+0000) 2024-03-06T17:59:20.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:20.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:20.293599+0000) 2024-03-06T17:59:21.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:21.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:21.293951+0000) 2024-03-06T17:59:22.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:22.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:22.294147+0000) 2024-03-06T17:59:23.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:23.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:23.294339+0000) 2024-03-06T17:59:24.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:24.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:24.294581+0000) 2024-03-06T17:59:25.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:25.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:25.294839+0000) 2024-03-06T17:59:26.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:26.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:26.295104+0000) 2024-03-06T17:59:27.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:27.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:27.295420+0000) 2024-03-06T17:59:28.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:28.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:28.295747+0000) 2024-03-06T17:59:29.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:29.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:29.296063+0000) 2024-03-06T17:59:29.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:29.479 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:30.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:30.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:30.296380+0000) 2024-03-06T17:59:31.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:31.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:31.296746+0000) 2024-03-06T17:59:32.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:32.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:32.297063+0000) 2024-03-06T17:59:33.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:33.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:33.297411+0000) 2024-03-06T17:59:34.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:34.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:34.297761+0000) 2024-03-06T17:59:35.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:35.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:35.298022+0000) 2024-03-06T17:59:36.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:36.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:36.298260+0000) 2024-03-06T17:59:37.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:37.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:37.298512+0000) 2024-03-06T17:59:38.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:38.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:38.298817+0000) 2024-03-06T17:59:39.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:39.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:39.299127+0000) 2024-03-06T17:59:40.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:40.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:40.299408+0000) 2024-03-06T17:59:41.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:41.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:41.299767+0000) 2024-03-06T17:59:42.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:42.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:42.300056+0000) 2024-03-06T17:59:43.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:43.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:43.300356+0000) 2024-03-06T17:59:44.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:44.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:44.300699+0000) 2024-03-06T17:59:45.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:45.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:45.300994+0000) 2024-03-06T17:59:46.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:46.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:46.301278+0000) 2024-03-06T17:59:47.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:47.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:47.301534+0000) 2024-03-06T17:59:48.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:48.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:48.301842+0000) 2024-03-06T17:59:49.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:49.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:49.302126+0000) 2024-03-06T17:59:50.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:50.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:50.302424+0000) 2024-03-06T17:59:51.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:51.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:51.302764+0000) 2024-03-06T17:59:52.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:52.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:52.303009+0000) 2024-03-06T17:59:53.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:53.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:53.303324+0000) 2024-03-06T17:59:54.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:54.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:54.303569+0000) 2024-03-06T17:59:55.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:55.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:55.303882+0000) 2024-03-06T17:59:56.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:56.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:56.304127+0000) 2024-03-06T17:59:57.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:57.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:57.304450+0000) 2024-03-06T17:59:58.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:58.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:58.304785+0000) 2024-03-06T17:59:59.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T17:59:59.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:59.305070+0000) 2024-03-06T17:59:59.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:59.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:00.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:00.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:00.305392+0000) 2024-03-06T18:00:01.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:01.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:01.305746+0000) 2024-03-06T18:00:02.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:02.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:02.306030+0000) 2024-03-06T18:00:03.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:03.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:03.306346+0000) 2024-03-06T18:00:04.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:04.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:04.306614+0000) 2024-03-06T18:00:05.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:05.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:05.306890+0000) 2024-03-06T18:00:06.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:06.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:06.307122+0000) 2024-03-06T18:00:07.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:07.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:07.307383+0000) 2024-03-06T18:00:08.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:08.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:08.307670+0000) 2024-03-06T18:00:09.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:09.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:09.307938+0000) 2024-03-06T18:00:10.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:10.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:10.308195+0000) 2024-03-06T18:00:11.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:11.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:11.308391+0000) 2024-03-06T18:00:12.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:12.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:12.308672+0000) 2024-03-06T18:00:13.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:13.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:13.308917+0000) 2024-03-06T18:00:14.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:14.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:14.309159+0000) 2024-03-06T18:00:15.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:15.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:15.309432+0000) 2024-03-06T18:00:16.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:16.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:16.309727+0000) 2024-03-06T18:00:17.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:17.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:17.309985+0000) 2024-03-06T18:00:18.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:18.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:18.310234+0000) 2024-03-06T18:00:19.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:19.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:19.310488+0000) 2024-03-06T18:00:20.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:20.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:20.310743+0000) 2024-03-06T18:00:21.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:21.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:21.311018+0000) 2024-03-06T18:00:22.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:22.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:22.311270+0000) 2024-03-06T18:00:23.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:23.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:23.311450+0000) 2024-03-06T18:00:24.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:24.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:24.311740+0000) 2024-03-06T18:00:25.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:25.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:25.311971+0000) 2024-03-06T18:00:26.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:26.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:26.312252+0000) 2024-03-06T18:00:27.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:27.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:27.312496+0000) 2024-03-06T18:00:28.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:28.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:28.312749+0000) 2024-03-06T18:00:29.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:29.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:29.312985+0000) 2024-03-06T18:00:29.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:29.787 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:30.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:30.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:30.313238+0000) 2024-03-06T18:00:31.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:31.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:31.313525+0000) 2024-03-06T18:00:32.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:32.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:32.313794+0000) 2024-03-06T18:00:33.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:33.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:33.314060+0000) 2024-03-06T18:00:34.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:34.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:34.314324+0000) 2024-03-06T18:00:35.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:35.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:35.314580+0000) 2024-03-06T18:00:36.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:36.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:36.314811+0000) 2024-03-06T18:00:37.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:37.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:37.315074+0000) 2024-03-06T18:00:38.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:38.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:38.315343+0000) 2024-03-06T18:00:39.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:39.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:39.315620+0000) 2024-03-06T18:00:40.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:40.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:40.315876+0000) 2024-03-06T18:00:41.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:41.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:41.316179+0000) 2024-03-06T18:00:42.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:42.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:42.316432+0000) 2024-03-06T18:00:43.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:43.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:43.316748+0000) 2024-03-06T18:00:44.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:44.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:44.316988+0000) 2024-03-06T18:00:45.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:45.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:45.317227+0000) 2024-03-06T18:00:46.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:46.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:46.317478+0000) 2024-03-06T18:00:47.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:47.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:47.317757+0000) 2024-03-06T18:00:48.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:48.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:48.318004+0000) 2024-03-06T18:00:49.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:49.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:49.318308+0000) 2024-03-06T18:00:50.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:50.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:50.318557+0000) 2024-03-06T18:00:51.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:51.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:51.318808+0000) 2024-03-06T18:00:52.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:52.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:52.319043+0000) 2024-03-06T18:00:53.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:53.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:53.319296+0000) 2024-03-06T18:00:54.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:54.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:54.319537+0000) 2024-03-06T18:00:55.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:55.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:55.319785+0000) 2024-03-06T18:00:56.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:56.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:56.320036+0000) 2024-03-06T18:00:57.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:57.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:57.320279+0000) 2024-03-06T18:00:58.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:58.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:58.320480+0000) 2024-03-06T18:00:59.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:00:59.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:59.320758+0000) 2024-03-06T18:00:59.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:59.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:00.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:00.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:00.321012+0000) 2024-03-06T18:01:01.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:01.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:01.321283+0000) 2024-03-06T18:01:02.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:02.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:02.321527+0000) 2024-03-06T18:01:03.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:03.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:03.321800+0000) 2024-03-06T18:01:04.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:04.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:04.322096+0000) 2024-03-06T18:01:05.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:05.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:05.322412+0000) 2024-03-06T18:01:06.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:06.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:06.322704+0000) 2024-03-06T18:01:07.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:07.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:07.322981+0000) 2024-03-06T18:01:08.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:08.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:08.323171+0000) 2024-03-06T18:01:09.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:09.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:09.323366+0000) 2024-03-06T18:01:10.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:10.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:10.323619+0000) 2024-03-06T18:01:11.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:11.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:11.323915+0000) 2024-03-06T18:01:12.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:12.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:12.324163+0000) 2024-03-06T18:01:13.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:13.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:13.324404+0000) 2024-03-06T18:01:14.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:14.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:14.324684+0000) 2024-03-06T18:01:15.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:15.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:15.324935+0000) 2024-03-06T18:01:16.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:16.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:16.325176+0000) 2024-03-06T18:01:17.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:17.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:17.325449+0000) 2024-03-06T18:01:18.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:18.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:18.325735+0000) 2024-03-06T18:01:19.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:19.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:19.326049+0000) 2024-03-06T18:01:20.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:20.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:20.326300+0000) 2024-03-06T18:01:21.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:21.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:21.326562+0000) 2024-03-06T18:01:22.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:22.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:22.326813+0000) 2024-03-06T18:01:23.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:23.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:23.327079+0000) 2024-03-06T18:01:24.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:24.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:24.327338+0000) 2024-03-06T18:01:25.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:25.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:25.327601+0000) 2024-03-06T18:01:26.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:26.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:26.327868+0000) 2024-03-06T18:01:27.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:27.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:27.328132+0000) 2024-03-06T18:01:28.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:28.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:28.328385+0000) 2024-03-06T18:01:29.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:29.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:29.328672+0000) 2024-03-06T18:01:30.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:30.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:30.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:30.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:30.328861+0000) 2024-03-06T18:01:31.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:31.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:31.329057+0000) 2024-03-06T18:01:32.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:32.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:32.329314+0000) 2024-03-06T18:01:33.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:33.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:33.329576+0000) 2024-03-06T18:01:34.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:34.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:34.329816+0000) 2024-03-06T18:01:35.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:35.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:35.330054+0000) 2024-03-06T18:01:36.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:36.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:36.330301+0000) 2024-03-06T18:01:37.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:37.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:37.330546+0000) 2024-03-06T18:01:38.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:38.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:38.330806+0000) 2024-03-06T18:01:39.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:39.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:39.331064+0000) 2024-03-06T18:01:40.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:40.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:40.331312+0000) 2024-03-06T18:01:41.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:41.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:41.331548+0000) 2024-03-06T18:01:42.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:42.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:42.331789+0000) 2024-03-06T18:01:43.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:43.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:43.332033+0000) 2024-03-06T18:01:44.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:44.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:44.332279+0000) 2024-03-06T18:01:45.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:45.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:45.332517+0000) 2024-03-06T18:01:46.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:46.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:46.332777+0000) 2024-03-06T18:01:47.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:47.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:47.332963+0000) 2024-03-06T18:01:48.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:48.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:48.333140+0000) 2024-03-06T18:01:49.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:49.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:49.333326+0000) 2024-03-06T18:01:50.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:50.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:50.333609+0000) 2024-03-06T18:01:51.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:51.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:51.333910+0000) 2024-03-06T18:01:52.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:52.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:52.334164+0000) 2024-03-06T18:01:53.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:53.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:53.334403+0000) 2024-03-06T18:01:54.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:54.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:54.334719+0000) 2024-03-06T18:01:55.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:55.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:55.334955+0000) 2024-03-06T18:01:56.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:56.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:56.335205+0000) 2024-03-06T18:01:57.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:57.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:57.335461+0000) 2024-03-06T18:01:58.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:58.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:58.335741+0000) 2024-03-06T18:01:59.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:01:59.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:59.335999+0000) 2024-03-06T18:02:00.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:00.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:00.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:00.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:00.336196+0000) 2024-03-06T18:02:01.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:01.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:01.336449+0000) 2024-03-06T18:02:02.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:02.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:02.336751+0000) 2024-03-06T18:02:03.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:03.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:03.336940+0000) 2024-03-06T18:02:04.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:04.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:04.337182+0000) 2024-03-06T18:02:05.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:05.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:05.337444+0000) 2024-03-06T18:02:06.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:06.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:06.337739+0000) 2024-03-06T18:02:07.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:07.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:07.338050+0000) 2024-03-06T18:02:08.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:08.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:08.338301+0000) 2024-03-06T18:02:09.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:09.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:09.338559+0000) 2024-03-06T18:02:10.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:10.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:10.338815+0000) 2024-03-06T18:02:11.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:11.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:11.339007+0000) 2024-03-06T18:02:12.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:12.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:12.339229+0000) 2024-03-06T18:02:13.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:13.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:13.339474+0000) 2024-03-06T18:02:14.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:14.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:14.339733+0000) 2024-03-06T18:02:15.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:15.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:15.339989+0000) 2024-03-06T18:02:16.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:16.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:16.340219+0000) 2024-03-06T18:02:17.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:17.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:17.340413+0000) 2024-03-06T18:02:18.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:18.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:18.340721+0000) 2024-03-06T18:02:19.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:19.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:19.340964+0000) 2024-03-06T18:02:20.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:20.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:20.341214+0000) 2024-03-06T18:02:21.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:21.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:21.341472+0000) 2024-03-06T18:02:22.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:22.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:22.341752+0000) 2024-03-06T18:02:23.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:23.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:23.341949+0000) 2024-03-06T18:02:24.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:24.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:24.342228+0000) 2024-03-06T18:02:25.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:25.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:25.342408+0000) 2024-03-06T18:02:26.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:26.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:26.342724+0000) 2024-03-06T18:02:27.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:27.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:27.342986+0000) 2024-03-06T18:02:28.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:28.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:28.343241+0000) 2024-03-06T18:02:29.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:29.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:29.343502+0000) 2024-03-06T18:02:30.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:30.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:30.343749+0000) 2024-03-06T18:02:30.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:30.403 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:31.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:31.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:31.344010+0000) 2024-03-06T18:02:32.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:32.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:32.344259+0000) 2024-03-06T18:02:33.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:33.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:33.344520+0000) 2024-03-06T18:02:34.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:34.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:34.344765+0000) 2024-03-06T18:02:35.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:35.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:35.344992+0000) 2024-03-06T18:02:36.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:36.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:36.345243+0000) 2024-03-06T18:02:37.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:37.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:37.345484+0000) 2024-03-06T18:02:38.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:38.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:38.345750+0000) 2024-03-06T18:02:39.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:39.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:39.345992+0000) 2024-03-06T18:02:40.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:40.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:40.346252+0000) 2024-03-06T18:02:41.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:41.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:41.346491+0000) 2024-03-06T18:02:42.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:42.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:42.346764+0000) 2024-03-06T18:02:43.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:43.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:43.347002+0000) 2024-03-06T18:02:44.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:44.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:44.347285+0000) 2024-03-06T18:02:45.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:45.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:45.347552+0000) 2024-03-06T18:02:46.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:46.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:46.347803+0000) 2024-03-06T18:02:47.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:47.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:47.348037+0000) 2024-03-06T18:02:48.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:48.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:48.348309+0000) 2024-03-06T18:02:49.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:49.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:49.348499+0000) 2024-03-06T18:02:50.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:50.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:50.348736+0000) 2024-03-06T18:02:51.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:51.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:51.348935+0000) 2024-03-06T18:02:52.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:52.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:52.349202+0000) 2024-03-06T18:02:53.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:53.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:53.349450+0000) 2024-03-06T18:02:54.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:54.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:54.349737+0000) 2024-03-06T18:02:55.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:55.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:55.349974+0000) 2024-03-06T18:02:56.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:56.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:56.350230+0000) 2024-03-06T18:02:57.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:57.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:57.350418+0000) 2024-03-06T18:02:58.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:58.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:58.350713+0000) 2024-03-06T18:02:59.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:02:59.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:59.350977+0000) 2024-03-06T18:03:00.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:00.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:00.351223+0000) 2024-03-06T18:03:00.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:00.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:01.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:01.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:01.351482+0000) 2024-03-06T18:03:02.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:02.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:02.351757+0000) 2024-03-06T18:03:03.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:03.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:03.351977+0000) 2024-03-06T18:03:04.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:04.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:04.352286+0000) 2024-03-06T18:03:05.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:05.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:05.352524+0000) 2024-03-06T18:03:06.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:06.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:06.352821+0000) 2024-03-06T18:03:07.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:07.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:07.353148+0000) 2024-03-06T18:03:08.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:08.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:08.353430+0000) 2024-03-06T18:03:09.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:09.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:09.353771+0000) 2024-03-06T18:03:10.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:10.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:10.354081+0000) 2024-03-06T18:03:11.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:11.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:11.354368+0000) 2024-03-06T18:03:12.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:12.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:12.354707+0000) 2024-03-06T18:03:13.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:13.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:13.354960+0000) 2024-03-06T18:03:14.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:14.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:14.355286+0000) 2024-03-06T18:03:15.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:15.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:15.355584+0000) 2024-03-06T18:03:16.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:16.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:16.355926+0000) 2024-03-06T18:03:17.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:17.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:17.356236+0000) 2024-03-06T18:03:18.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:18.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:18.356546+0000) 2024-03-06T18:03:19.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:19.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:19.356853+0000) 2024-03-06T18:03:20.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:20.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:20.357155+0000) 2024-03-06T18:03:21.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:21.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:21.357454+0000) 2024-03-06T18:03:22.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:22.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:22.357778+0000) 2024-03-06T18:03:23.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:23.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:23.358099+0000) 2024-03-06T18:03:24.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:24.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:24.358415+0000) 2024-03-06T18:03:25.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:25.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:25.358745+0000) 2024-03-06T18:03:26.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:26.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:26.359036+0000) 2024-03-06T18:03:27.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:27.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:27.359340+0000) 2024-03-06T18:03:28.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:28.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:28.359668+0000) 2024-03-06T18:03:29.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:29.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:29.359971+0000) 2024-03-06T18:03:30.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:30.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:30.360281+0000) 2024-03-06T18:03:30.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:30.721 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:31.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:31.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:31.360601+0000) 2024-03-06T18:03:32.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:32.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:32.360951+0000) 2024-03-06T18:03:33.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:33.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:33.361251+0000) 2024-03-06T18:03:34.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:34.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:34.361558+0000) 2024-03-06T18:03:35.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:35.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:35.361848+0000) 2024-03-06T18:03:36.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:36.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:36.362148+0000) 2024-03-06T18:03:37.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:37.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:37.362407+0000) 2024-03-06T18:03:38.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:38.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:38.362710+0000) 2024-03-06T18:03:39.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:39.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:39.362950+0000) 2024-03-06T18:03:40.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:40.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:40.363201+0000) 2024-03-06T18:03:41.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:41.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:41.363461+0000) 2024-03-06T18:03:42.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:42.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:42.363691+0000) 2024-03-06T18:03:43.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:43.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:43.363946+0000) 2024-03-06T18:03:44.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:44.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:44.364195+0000) 2024-03-06T18:03:45.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:45.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:45.364456+0000) 2024-03-06T18:03:46.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:46.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:46.364728+0000) 2024-03-06T18:03:47.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:47.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:47.364932+0000) 2024-03-06T18:03:48.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:48.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:48.365191+0000) 2024-03-06T18:03:49.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:49.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:49.365461+0000) 2024-03-06T18:03:50.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:50.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:50.365745+0000) 2024-03-06T18:03:51.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:51.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:51.366017+0000) 2024-03-06T18:03:52.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:52.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:52.366290+0000) 2024-03-06T18:03:53.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:53.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:53.366572+0000) 2024-03-06T18:03:54.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:54.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:54.366896+0000) 2024-03-06T18:03:55.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:55.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:55.367225+0000) 2024-03-06T18:03:56.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:56.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:56.367518+0000) 2024-03-06T18:03:57.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:57.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:57.367835+0000) 2024-03-06T18:03:58.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:58.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:58.368146+0000) 2024-03-06T18:03:59.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:03:59.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:59.368445+0000) 2024-03-06T18:04:00.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:00.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:00.368772+0000) 2024-03-06T18:04:00.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:00.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:01.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:01.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:01.369061+0000) 2024-03-06T18:04:02.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:02.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:02.369296+0000) 2024-03-06T18:04:03.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:03.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:03.369556+0000) 2024-03-06T18:04:04.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:04.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:04.369825+0000) 2024-03-06T18:04:05.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:05.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:05.370082+0000) 2024-03-06T18:04:06.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:06.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:06.370330+0000) 2024-03-06T18:04:07.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:07.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:07.370580+0000) 2024-03-06T18:04:08.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:08.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:08.370841+0000) 2024-03-06T18:04:09.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:09.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:09.371082+0000) 2024-03-06T18:04:10.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:10.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:10.371311+0000) 2024-03-06T18:04:11.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:11.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:11.371555+0000) 2024-03-06T18:04:12.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:12.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:12.371803+0000) 2024-03-06T18:04:13.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:13.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:13.372053+0000) 2024-03-06T18:04:14.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:14.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:14.372318+0000) 2024-03-06T18:04:15.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:15.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:15.372577+0000) 2024-03-06T18:04:16.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:16.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:16.372832+0000) 2024-03-06T18:04:17.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:17.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:17.373105+0000) 2024-03-06T18:04:18.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:18.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:18.373368+0000) 2024-03-06T18:04:19.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:19.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:19.373614+0000) 2024-03-06T18:04:20.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:20.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:20.373913+0000) 2024-03-06T18:04:21.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:21.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:21.374161+0000) 2024-03-06T18:04:22.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:22.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:22.374412+0000) 2024-03-06T18:04:23.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:23.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:23.374723+0000) 2024-03-06T18:04:24.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:24.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:24.374968+0000) 2024-03-06T18:04:25.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:25.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:25.375238+0000) 2024-03-06T18:04:26.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:26.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:26.375510+0000) 2024-03-06T18:04:27.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:27.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:27.375802+0000) 2024-03-06T18:04:28.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:28.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:28.376054+0000) 2024-03-06T18:04:29.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:29.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:29.376309+0000) 2024-03-06T18:04:30.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:30.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:30.376556+0000) 2024-03-06T18:04:31.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:31.022 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:31.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:31.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:31.376816+0000) 2024-03-06T18:04:32.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:32.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:32.377059+0000) 2024-03-06T18:04:33.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:33.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:33.377334+0000) 2024-03-06T18:04:34.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:34.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:34.377595+0000) 2024-03-06T18:04:35.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:35.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:35.377921+0000) 2024-03-06T18:04:36.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:36.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:36.378184+0000) 2024-03-06T18:04:37.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:37.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:37.378510+0000) 2024-03-06T18:04:38.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:38.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:38.378744+0000) 2024-03-06T18:04:39.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:39.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:39.378931+0000) 2024-03-06T18:04:40.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:40.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:40.379109+0000) 2024-03-06T18:04:41.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:41.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:41.379357+0000) 2024-03-06T18:04:42.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:42.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:42.379595+0000) 2024-03-06T18:04:43.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:43.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:43.379865+0000) 2024-03-06T18:04:44.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:44.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:44.380125+0000) 2024-03-06T18:04:45.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:45.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:45.380373+0000) 2024-03-06T18:04:46.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:46.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:46.380621+0000) 2024-03-06T18:04:47.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:47.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:47.380929+0000) 2024-03-06T18:04:48.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:48.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:48.381188+0000) 2024-03-06T18:04:49.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:49.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:49.381447+0000) 2024-03-06T18:04:50.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:50.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:50.381741+0000) 2024-03-06T18:04:51.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:51.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:51.381995+0000) 2024-03-06T18:04:52.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:52.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:52.382194+0000) 2024-03-06T18:04:53.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:53.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:53.382385+0000) 2024-03-06T18:04:54.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:54.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:54.382700+0000) 2024-03-06T18:04:55.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:55.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:55.382951+0000) 2024-03-06T18:04:56.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:56.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:56.383190+0000) 2024-03-06T18:04:57.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:57.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:57.383435+0000) 2024-03-06T18:04:58.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:58.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:58.383723+0000) 2024-03-06T18:04:59.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:04:59.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:59.383985+0000) 2024-03-06T18:05:00.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:00.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:00.384218+0000) 2024-03-06T18:05:01.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:01.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:01.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:01.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:01.384506+0000) 2024-03-06T18:05:02.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:02.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:02.384771+0000) 2024-03-06T18:05:03.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:03.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:03.385032+0000) 2024-03-06T18:05:04.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:04.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:04.385293+0000) 2024-03-06T18:05:05.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:05.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:05.385526+0000) 2024-03-06T18:05:06.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:06.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:06.385758+0000) 2024-03-06T18:05:07.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:07.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:07.386031+0000) 2024-03-06T18:05:08.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:08.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:08.386329+0000) 2024-03-06T18:05:09.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:09.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:09.386614+0000) 2024-03-06T18:05:10.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:10.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:10.386969+0000) 2024-03-06T18:05:11.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:11.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:11.387284+0000) 2024-03-06T18:05:12.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:12.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:12.387578+0000) 2024-03-06T18:05:13.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:13.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:13.387867+0000) 2024-03-06T18:05:14.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:14.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:14.388177+0000) 2024-03-06T18:05:15.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:15.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:15.388462+0000) 2024-03-06T18:05:16.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:16.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:16.388775+0000) 2024-03-06T18:05:17.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:17.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:17.389095+0000) 2024-03-06T18:05:18.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:18.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:18.389418+0000) 2024-03-06T18:05:19.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:19.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:19.389777+0000) 2024-03-06T18:05:20.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:20.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:20.390003+0000) 2024-03-06T18:05:21.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:21.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:21.390304+0000) 2024-03-06T18:05:22.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:22.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:22.390601+0000) 2024-03-06T18:05:23.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:23.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:23.390942+0000) 2024-03-06T18:05:24.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:24.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:24.391173+0000) 2024-03-06T18:05:25.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:25.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:25.391421+0000) 2024-03-06T18:05:26.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:26.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:26.391691+0000) 2024-03-06T18:05:27.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:27.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:27.391983+0000) 2024-03-06T18:05:28.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:28.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:28.392274+0000) 2024-03-06T18:05:29.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:29.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:29.392567+0000) 2024-03-06T18:05:30.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:30.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:30.392916+0000) 2024-03-06T18:05:31.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:31.308 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:31.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:31.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:31.393212+0000) 2024-03-06T18:05:32.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:32.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:32.393510+0000) 2024-03-06T18:05:33.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:33.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:33.393825+0000) 2024-03-06T18:05:34.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:34.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:34.394074+0000) 2024-03-06T18:05:35.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:35.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:35.394385+0000) 2024-03-06T18:05:36.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:36.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:36.394729+0000) 2024-03-06T18:05:37.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:37.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:37.395003+0000) 2024-03-06T18:05:38.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:38.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:38.395274+0000) 2024-03-06T18:05:39.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:39.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:39.395523+0000) 2024-03-06T18:05:40.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:40.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:40.395756+0000) 2024-03-06T18:05:41.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:41.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:41.396016+0000) 2024-03-06T18:05:42.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:42.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:42.396313+0000) 2024-03-06T18:05:43.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:43.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:43.396596+0000) 2024-03-06T18:05:44.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:44.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:44.396894+0000) 2024-03-06T18:05:45.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:45.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:45.397152+0000) 2024-03-06T18:05:46.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:46.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:46.397407+0000) 2024-03-06T18:05:47.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:47.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:47.397708+0000) 2024-03-06T18:05:48.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:48.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:48.397946+0000) 2024-03-06T18:05:49.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:49.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:49.398264+0000) 2024-03-06T18:05:50.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:50.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:50.398571+0000) 2024-03-06T18:05:51.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:51.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:51.398856+0000) 2024-03-06T18:05:52.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:52.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:52.399146+0000) 2024-03-06T18:05:53.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:53.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:53.399441+0000) 2024-03-06T18:05:54.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:54.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:54.399798+0000) 2024-03-06T18:05:55.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:55.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:55.400075+0000) 2024-03-06T18:05:56.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:56.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:56.400322+0000) 2024-03-06T18:05:57.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:57.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:57.400584+0000) 2024-03-06T18:05:58.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:58.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:58.400828+0000) 2024-03-06T18:05:59.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:05:59.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:59.401074+0000) 2024-03-06T18:06:00.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:00.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:00.401332+0000) 2024-03-06T18:06:01.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:01.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:01.401559+0000) 2024-03-06T18:06:01.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:01.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:02.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:02.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:02.401845+0000) 2024-03-06T18:06:03.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:03.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:03.402181+0000) 2024-03-06T18:06:04.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:04.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:04.402499+0000) 2024-03-06T18:06:05.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:05.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:05.402822+0000) 2024-03-06T18:06:06.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:06.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:06.403047+0000) 2024-03-06T18:06:07.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:07.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:07.403356+0000) 2024-03-06T18:06:08.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:08.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:08.403713+0000) 2024-03-06T18:06:09.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:09.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:09.403975+0000) 2024-03-06T18:06:10.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:10.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:10.404215+0000) 2024-03-06T18:06:11.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:11.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:11.404475+0000) 2024-03-06T18:06:12.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:12.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:12.404752+0000) 2024-03-06T18:06:13.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:13.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:13.405012+0000) 2024-03-06T18:06:14.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:14.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:14.405277+0000) 2024-03-06T18:06:15.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:15.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:15.405498+0000) 2024-03-06T18:06:16.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:16.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:16.405744+0000) 2024-03-06T18:06:17.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:17.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:17.405936+0000) 2024-03-06T18:06:18.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:18.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:18.406239+0000) 2024-03-06T18:06:19.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:19.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:19.406496+0000) 2024-03-06T18:06:20.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:20.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:20.406740+0000) 2024-03-06T18:06:21.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:21.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:21.406984+0000) 2024-03-06T18:06:22.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:22.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:22.407221+0000) 2024-03-06T18:06:23.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:23.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:23.407462+0000) 2024-03-06T18:06:24.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:24.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:24.407739+0000) 2024-03-06T18:06:25.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:25.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:25.407980+0000) 2024-03-06T18:06:26.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:26.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:26.408211+0000) 2024-03-06T18:06:27.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:27.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:27.408464+0000) 2024-03-06T18:06:28.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:28.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:28.408709+0000) 2024-03-06T18:06:29.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:29.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:29.408967+0000) 2024-03-06T18:06:30.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:30.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:30.409230+0000) 2024-03-06T18:06:31.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:31.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:31.409414+0000) 2024-03-06T18:06:31.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:31.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:32.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:32.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:32.409707+0000) 2024-03-06T18:06:33.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:33.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:33.409963+0000) 2024-03-06T18:06:34.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:34.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:34.410227+0000) 2024-03-06T18:06:35.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:35.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:35.410474+0000) 2024-03-06T18:06:36.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:36.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:36.410743+0000) 2024-03-06T18:06:37.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:37.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:37.410980+0000) 2024-03-06T18:06:38.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:38.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:38.411234+0000) 2024-03-06T18:06:39.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:39.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:39.411490+0000) 2024-03-06T18:06:40.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:40.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:40.411741+0000) 2024-03-06T18:06:41.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:41.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:41.412003+0000) 2024-03-06T18:06:42.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:42.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:42.412251+0000) 2024-03-06T18:06:43.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:43.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:43.412506+0000) 2024-03-06T18:06:44.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:44.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:44.412762+0000) 2024-03-06T18:06:45.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:45.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:45.413011+0000) 2024-03-06T18:06:46.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:46.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:46.413259+0000) 2024-03-06T18:06:47.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:47.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:47.413487+0000) 2024-03-06T18:06:48.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:48.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:48.413735+0000) 2024-03-06T18:06:49.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:49.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:49.413973+0000) 2024-03-06T18:06:50.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:50.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:50.414205+0000) 2024-03-06T18:06:51.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:51.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:51.414441+0000) 2024-03-06T18:06:52.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:52.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:52.414726+0000) 2024-03-06T18:06:53.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:53.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:53.414986+0000) 2024-03-06T18:06:54.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:54.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:54.415248+0000) 2024-03-06T18:06:55.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:55.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:55.415496+0000) 2024-03-06T18:06:56.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:56.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:56.415733+0000) 2024-03-06T18:06:57.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:57.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:57.415981+0000) 2024-03-06T18:06:58.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:58.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:58.416235+0000) 2024-03-06T18:06:59.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:06:59.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:59.416489+0000) 2024-03-06T18:07:00.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:00.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:00.416738+0000) 2024-03-06T18:07:01.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:01.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:01.416995+0000) 2024-03-06T18:07:01.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:01.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:02.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:02.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:02.417223+0000) 2024-03-06T18:07:03.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:03.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:03.417494+0000) 2024-03-06T18:07:04.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:04.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:04.417754+0000) 2024-03-06T18:07:05.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:05.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:05.418007+0000) 2024-03-06T18:07:06.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:06.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:06.418294+0000) 2024-03-06T18:07:07.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:07.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:07.418552+0000) 2024-03-06T18:07:08.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:08.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:08.418802+0000) 2024-03-06T18:07:09.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:09.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:09.419057+0000) 2024-03-06T18:07:10.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:10.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:10.419306+0000) 2024-03-06T18:07:11.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:11.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:11.419546+0000) 2024-03-06T18:07:12.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:12.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:12.419778+0000) 2024-03-06T18:07:13.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:13.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:13.420017+0000) 2024-03-06T18:07:14.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:14.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:14.420276+0000) 2024-03-06T18:07:15.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:15.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:15.420526+0000) 2024-03-06T18:07:16.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:16.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:16.420760+0000) 2024-03-06T18:07:17.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:17.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:17.421015+0000) 2024-03-06T18:07:18.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:18.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:18.421195+0000) 2024-03-06T18:07:19.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:19.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:19.421437+0000) 2024-03-06T18:07:20.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:20.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:20.421703+0000) 2024-03-06T18:07:21.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:21.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:21.421947+0000) 2024-03-06T18:07:22.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:22.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:22.422194+0000) 2024-03-06T18:07:23.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:23.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:23.422447+0000) 2024-03-06T18:07:24.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:24.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:24.422657+0000) 2024-03-06T18:07:25.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:25.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:25.422912+0000) 2024-03-06T18:07:26.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:26.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:26.423144+0000) 2024-03-06T18:07:27.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:27.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:27.423409+0000) 2024-03-06T18:07:28.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:28.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:28.423666+0000) 2024-03-06T18:07:29.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:29.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:29.423927+0000) 2024-03-06T18:07:30.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:30.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:30.424174+0000) 2024-03-06T18:07:31.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:31.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:31.424426+0000) 2024-03-06T18:07:31.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:31.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:32.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:32.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:32.424703+0000) 2024-03-06T18:07:33.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:33.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:33.424964+0000) 2024-03-06T18:07:34.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:34.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:34.425224+0000) 2024-03-06T18:07:35.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:35.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:35.425455+0000) 2024-03-06T18:07:36.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:36.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:36.425745+0000) 2024-03-06T18:07:37.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:37.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:37.425985+0000) 2024-03-06T18:07:38.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:38.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:38.426231+0000) 2024-03-06T18:07:39.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:39.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:39.426486+0000) 2024-03-06T18:07:40.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:40.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:40.426754+0000) 2024-03-06T18:07:41.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:41.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:41.427008+0000) 2024-03-06T18:07:42.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:42.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:42.427237+0000) 2024-03-06T18:07:43.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:43.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:43.427482+0000) 2024-03-06T18:07:44.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:44.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:44.427742+0000) 2024-03-06T18:07:45.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:45.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:45.427980+0000) 2024-03-06T18:07:46.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:46.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:46.428228+0000) 2024-03-06T18:07:47.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:47.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:47.428467+0000) 2024-03-06T18:07:48.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:48.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:48.428733+0000) 2024-03-06T18:07:49.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:49.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:49.428974+0000) 2024-03-06T18:07:50.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:50.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:50.429204+0000) 2024-03-06T18:07:51.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:51.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:51.429461+0000) 2024-03-06T18:07:52.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:52.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:52.429729+0000) 2024-03-06T18:07:53.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:53.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:53.429997+0000) 2024-03-06T18:07:54.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:54.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:54.430258+0000) 2024-03-06T18:07:55.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:55.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:55.430514+0000) 2024-03-06T18:07:56.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:56.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:56.430763+0000) 2024-03-06T18:07:57.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:57.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:57.431020+0000) 2024-03-06T18:07:58.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:58.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:58.431272+0000) 2024-03-06T18:07:59.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:07:59.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:59.431540+0000) 2024-03-06T18:08:00.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:00.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:00.431791+0000) 2024-03-06T18:08:01.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:01.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:01.431991+0000) 2024-03-06T18:08:02.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:02.059 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:02.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:02.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:02.432246+0000) 2024-03-06T18:08:03.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:03.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:03.432514+0000) 2024-03-06T18:08:04.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:04.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:04.432758+0000) 2024-03-06T18:08:05.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:05.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:05.433022+0000) 2024-03-06T18:08:06.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:06.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:06.433256+0000) 2024-03-06T18:08:07.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:07.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:07.433499+0000) 2024-03-06T18:08:08.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:08.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:08.433747+0000) 2024-03-06T18:08:09.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:09.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:09.434008+0000) 2024-03-06T18:08:10.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:10.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:10.434258+0000) 2024-03-06T18:08:11.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:11.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:11.434516+0000) 2024-03-06T18:08:12.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:12.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:12.434755+0000) 2024-03-06T18:08:13.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:13.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:13.435016+0000) 2024-03-06T18:08:14.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:14.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:14.435263+0000) 2024-03-06T18:08:15.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:15.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:15.435514+0000) 2024-03-06T18:08:16.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:16.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:16.435767+0000) 2024-03-06T18:08:17.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:17.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:17.436029+0000) 2024-03-06T18:08:18.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:18.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:18.436262+0000) 2024-03-06T18:08:19.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:19.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:19.436510+0000) 2024-03-06T18:08:20.434 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:20.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:20.436755+0000) 2024-03-06T18:08:21.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:21.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:21.437009+0000) 2024-03-06T18:08:22.435 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:22.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:22.437243+0000) 2024-03-06T18:08:23.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:23.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:23.437510+0000) 2024-03-06T18:08:24.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:24.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:24.437774+0000) 2024-03-06T18:08:25.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:25.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:25.438024+0000) 2024-03-06T18:08:26.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:26.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:26.438268+0000) 2024-03-06T18:08:27.436 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:27.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:27.438523+0000) 2024-03-06T18:08:28.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:28.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:28.438741+0000) 2024-03-06T18:08:29.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:29.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:29.438976+0000) 2024-03-06T18:08:30.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:30.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:30.439209+0000) 2024-03-06T18:08:31.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:31.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:31.439469+0000) 2024-03-06T18:08:32.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:32.200 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:32.437 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:32.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:32.439748+0000) 2024-03-06T18:08:33.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:33.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:33.440025+0000) 2024-03-06T18:08:34.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:34.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:34.440275+0000) 2024-03-06T18:08:35.438 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:35.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:35.440526+0000) 2024-03-06T18:08:36.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:36.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:36.440756+0000) 2024-03-06T18:08:37.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:37.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:37.440995+0000) 2024-03-06T18:08:38.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:38.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:38.441185+0000) 2024-03-06T18:08:39.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:39.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:39.441422+0000) 2024-03-06T18:08:40.439 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:40.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:40.441709+0000) 2024-03-06T18:08:41.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:41.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:41.441965+0000) 2024-03-06T18:08:42.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:42.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:42.442219+0000) 2024-03-06T18:08:43.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:43.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:43.442491+0000) 2024-03-06T18:08:44.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:44.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:44.442717+0000) 2024-03-06T18:08:45.440 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:45.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:45.442969+0000) 2024-03-06T18:08:46.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:46.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:46.443151+0000) 2024-03-06T18:08:47.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:47.442+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:47.443387+0000) 2024-03-06T18:08:48.441 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:48.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:48.443674+0000) 2024-03-06T18:08:49.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:49.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:49.443918+0000) 2024-03-06T18:08:50.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:50.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:50.444104+0000) 2024-03-06T18:08:51.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:51.443+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:51.444328+0000) 2024-03-06T18:08:52.442 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:52.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:52.444566+0000) 2024-03-06T18:08:53.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:53.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:53.444823+0000) 2024-03-06T18:08:54.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:54.444+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:54.445081+0000) 2024-03-06T18:08:55.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:55.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:55.445328+0000) 2024-03-06T18:08:56.443 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:56.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:56.445581+0000) 2024-03-06T18:08:57.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:57.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:57.445838+0000) 2024-03-06T18:08:58.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:58.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:58.446082+0000) 2024-03-06T18:08:59.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:08:59.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:59.446319+0000) 2024-03-06T18:09:00.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:00.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:00.446548+0000) 2024-03-06T18:09:01.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:01.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:01.446793+0000) 2024-03-06T18:09:02.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:02.365 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:02.444 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:02.445+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:02.446990+0000) 2024-03-06T18:09:03.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:03.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:03.447273+0000) 2024-03-06T18:09:04.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:04.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:04.447544+0000) 2024-03-06T18:09:05.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:05.446+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:05.447779+0000) 2024-03-06T18:09:06.445 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:06.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:06.448027+0000) 2024-03-06T18:09:07.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:07.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:07.448288+0000) 2024-03-06T18:09:08.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:08.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:08.448549+0000) 2024-03-06T18:09:09.446 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:09.447+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:09.448788+0000) 2024-03-06T18:09:10.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:10.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:10.449017+0000) 2024-03-06T18:09:11.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:11.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:11.449276+0000) 2024-03-06T18:09:12.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:12.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:12.449526+0000) 2024-03-06T18:09:13.447 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:13.448+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:13.449768+0000) 2024-03-06T18:09:14.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:14.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:14.450015+0000) 2024-03-06T18:09:15.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:15.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:15.450262+0000) 2024-03-06T18:09:16.448 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:16.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:16.450492+0000) 2024-03-06T18:09:17.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:17.449+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:17.450750+0000) 2024-03-06T18:09:18.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:18.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:18.451010+0000) 2024-03-06T18:09:19.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:19.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:19.451249+0000) 2024-03-06T18:09:20.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:20.450+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:20.451479+0000) 2024-03-06T18:09:21.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:21.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:21.451756+0000) 2024-03-06T18:09:22.449 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:22.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:22.452002+0000) 2024-03-06T18:09:23.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:23.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:23.452273+0000) 2024-03-06T18:09:24.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:24.451+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:24.452513+0000) 2024-03-06T18:09:25.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:25.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:25.452769+0000) 2024-03-06T18:09:26.450 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:26.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:26.453004+0000) 2024-03-06T18:09:27.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:27.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:27.453255+0000) 2024-03-06T18:09:28.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:28.452+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:28.453493+0000) 2024-03-06T18:09:29.451 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:29.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:29.453780+0000) 2024-03-06T18:09:30.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:30.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:30.454024+0000) 2024-03-06T18:09:31.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:31.453+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:31.454280+0000) 2024-03-06T18:09:32.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:32.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:32.454532+0000) 2024-03-06T18:09:32.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:32.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:33.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:33.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:33.454816+0000) 2024-03-06T18:09:34.452 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:34.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:34.455078+0000) 2024-03-06T18:09:35.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:35.454+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:35.455367+0000) 2024-03-06T18:09:36.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:36.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:36.455602+0000) 2024-03-06T18:09:37.453 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:37.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:37.455907+0000) 2024-03-06T18:09:38.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:38.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:38.456185+0000) 2024-03-06T18:09:39.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:39.455+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:39.456424+0000) 2024-03-06T18:09:40.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:40.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:40.456701+0000) 2024-03-06T18:09:41.454 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:41.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:41.456956+0000) 2024-03-06T18:09:42.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:42.456+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:42.457204+0000) 2024-03-06T18:09:43.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:43.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:43.457457+0000) 2024-03-06T18:09:44.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:44.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:44.457757+0000) 2024-03-06T18:09:45.455 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:45.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:45.458021+0000) 2024-03-06T18:09:46.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:46.457+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:46.458204+0000) 2024-03-06T18:09:47.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:47.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:47.458448+0000) 2024-03-06T18:09:48.456 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:48.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:48.458759+0000) 2024-03-06T18:09:49.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:49.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:49.459019+0000) 2024-03-06T18:09:50.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:50.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:50.459259+0000) 2024-03-06T18:09:51.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:51.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:51.459508+0000) 2024-03-06T18:09:52.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:52.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:52.459755+0000) 2024-03-06T18:09:53.457 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:53.458+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:53.460032+0000) 2024-03-06T18:09:54.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:54.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:54.460292+0000) 2024-03-06T18:09:55.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:55.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:55.460490+0000) 2024-03-06T18:09:56.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:56.459+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:56.460745+0000) 2024-03-06T18:09:57.458 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:57.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:57.461007+0000) 2024-03-06T18:09:58.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:58.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:58.461241+0000) 2024-03-06T18:09:59.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:09:59.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:59.461508+0000) 2024-03-06T18:10:00.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:00.460+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:00.461759+0000) 2024-03-06T18:10:01.459 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:01.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:01.461997+0000) 2024-03-06T18:10:02.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:02.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:02.462251+0000) 2024-03-06T18:10:02.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:02.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:03.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:03.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:03.462521+0000) 2024-03-06T18:10:04.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:04.461+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:04.462780+0000) 2024-03-06T18:10:05.460 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:05.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:05.463035+0000) 2024-03-06T18:10:06.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:06.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:06.463284+0000) 2024-03-06T18:10:07.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:07.462+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:07.463543+0000) 2024-03-06T18:10:08.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:08.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:08.463795+0000) 2024-03-06T18:10:09.461 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:09.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:09.464061+0000) 2024-03-06T18:10:10.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:10.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:10.464313+0000) 2024-03-06T18:10:11.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:11.463+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:11.464569+0000) 2024-03-06T18:10:12.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:12.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:12.464820+0000) 2024-03-06T18:10:13.462 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:13.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:13.465081+0000) 2024-03-06T18:10:14.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:14.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:14.465344+0000) 2024-03-06T18:10:15.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:15.464+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:15.465604+0000) 2024-03-06T18:10:16.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:16.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:16.465907+0000) 2024-03-06T18:10:17.463 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:17.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:17.466144+0000) 2024-03-06T18:10:18.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:18.465+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:18.466388+0000) 2024-03-06T18:10:19.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:19.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:19.466681+0000) 2024-03-06T18:10:20.464 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:20.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:20.466926+0000) 2024-03-06T18:10:21.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:21.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:21.467171+0000) 2024-03-06T18:10:22.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:22.466+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:22.467425+0000) 2024-03-06T18:10:23.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:23.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:23.467769+0000) 2024-03-06T18:10:24.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:24.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:24.468028+0000) 2024-03-06T18:10:25.465 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:25.467+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:25.468284+0000) 2024-03-06T18:10:26.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:26.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:26.468514+0000) 2024-03-06T18:10:27.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:27.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:27.468749+0000) 2024-03-06T18:10:28.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:28.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:28.469012+0000) 2024-03-06T18:10:29.466 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:29.468+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:29.469277+0000) 2024-03-06T18:10:30.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:30.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:30.469507+0000) 2024-03-06T18:10:31.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:31.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:31.469757+0000) 2024-03-06T18:10:32.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:32.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:32.470007+0000) 2024-03-06T18:10:32.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:32.818 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:33.467 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:33.469+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:33.470271+0000) 2024-03-06T18:10:34.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:34.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:34.470517+0000) 2024-03-06T18:10:35.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:35.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:35.470765+0000) 2024-03-06T18:10:36.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:36.470+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:36.470992+0000) 2024-03-06T18:10:37.468 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:37.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:37.471229+0000) 2024-03-06T18:10:38.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:38.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:38.471484+0000) 2024-03-06T18:10:39.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:39.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:39.471745+0000) 2024-03-06T18:10:40.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:40.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:40.471991+0000) 2024-03-06T18:10:41.469 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:41.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:41.472250+0000) 2024-03-06T18:10:42.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:42.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:42.472483+0000) 2024-03-06T18:10:43.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:43.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:43.472739+0000) 2024-03-06T18:10:44.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:44.471+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:44.472998+0000) 2024-03-06T18:10:45.470 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:45.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:45.473229+0000) 2024-03-06T18:10:46.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:46.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:46.473461+0000) 2024-03-06T18:10:47.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:47.472+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:47.473740+0000) 2024-03-06T18:10:48.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:48.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:48.473985+0000) 2024-03-06T18:10:49.471 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:49.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:49.474246+0000) 2024-03-06T18:10:50.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:50.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:50.474484+0000) 2024-03-06T18:10:51.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:51.473+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:51.474737+0000) 2024-03-06T18:10:52.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:52.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:52.475001+0000) 2024-03-06T18:10:53.472 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:53.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:53.475242+0000) 2024-03-06T18:10:54.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:54.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:54.475488+0000) 2024-03-06T18:10:55.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:55.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:55.475768+0000) 2024-03-06T18:10:56.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:56.474+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:56.476010+0000) 2024-03-06T18:10:57.473 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:57.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:57.476269+0000) 2024-03-06T18:10:58.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:58.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:58.476525+0000) 2024-03-06T18:10:59.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:10:59.475+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:59.476759+0000) 2024-03-06T18:11:00.474 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:00.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:00.477011+0000) 2024-03-06T18:11:01.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:01.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:01.477281+0000) 2024-03-06T18:11:02.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:02.476+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:02.477551+0000) 2024-03-06T18:11:02.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:02.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:03.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:03.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:03.477833+0000) 2024-03-06T18:11:04.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:04.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:04.478100+0000) 2024-03-06T18:11:05.475 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:05.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:05.478288+0000) 2024-03-06T18:11:06.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:06.477+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:06.478528+0000) 2024-03-06T18:11:07.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:07.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:07.478793+0000) 2024-03-06T18:11:08.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:08.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:08.479030+0000) 2024-03-06T18:11:09.476 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:09.478+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:09.479295+0000) 2024-03-06T18:11:10.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:10.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:10.479551+0000) 2024-03-06T18:11:11.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:11.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:11.479806+0000) 2024-03-06T18:11:12.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:12.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:12.480057+0000) 2024-03-06T18:11:13.477 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:13.479+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:13.480318+0000) 2024-03-06T18:11:14.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:14.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:14.480585+0000) 2024-03-06T18:11:15.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:15.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:15.480848+0000) 2024-03-06T18:11:16.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:16.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:16.481083+0000) 2024-03-06T18:11:17.478 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:17.480+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:17.481382+0000) 2024-03-06T18:11:18.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:18.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:18.481625+0000) 2024-03-06T18:11:19.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:19.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:19.481854+0000) 2024-03-06T18:11:20.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:20.481+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:20.482114+0000) 2024-03-06T18:11:21.479 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:21.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:21.482373+0000) 2024-03-06T18:11:22.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:22.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:22.482678+0000) 2024-03-06T18:11:23.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:23.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:23.482929+0000) 2024-03-06T18:11:24.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:24.482+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:24.483195+0000) 2024-03-06T18:11:25.480 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:25.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:25.483469+0000) 2024-03-06T18:11:26.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:26.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:26.483753+0000) 2024-03-06T18:11:27.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:27.483+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:27.484016+0000) 2024-03-06T18:11:28.481 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:28.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:28.484286+0000) 2024-03-06T18:11:29.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:29.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:29.484551+0000) 2024-03-06T18:11:30.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:30.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:30.484820+0000) 2024-03-06T18:11:31.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:31.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:31.485079+0000) 2024-03-06T18:11:32.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:32.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:32.485342+0000) 2024-03-06T18:11:33.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:33.112 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:33.482 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:33.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:33.485618+0000) 2024-03-06T18:11:34.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:34.484+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:34.485950+0000) 2024-03-06T18:11:35.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:35.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:35.486210+0000) 2024-03-06T18:11:36.483 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:36.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:36.486485+0000) 2024-03-06T18:11:37.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:37.485+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:37.486768+0000) 2024-03-06T18:11:38.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:38.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:38.487042+0000) 2024-03-06T18:11:39.484 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:39.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:39.487325+0000) 2024-03-06T18:11:40.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:40.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:40.487592+0000) 2024-03-06T18:11:41.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:41.486+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:41.487917+0000) 2024-03-06T18:11:42.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:42.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:42.488204+0000) 2024-03-06T18:11:43.485 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:43.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:43.488486+0000) 2024-03-06T18:11:44.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:44.487+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:44.488754+0000) 2024-03-06T18:11:45.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:45.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:45.489003+0000) 2024-03-06T18:11:46.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:46.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:46.489253+0000) 2024-03-06T18:11:47.486 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:47.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:47.489512+0000) 2024-03-06T18:11:48.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:48.488+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:48.489749+0000) 2024-03-06T18:11:49.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:49.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:49.490005+0000) 2024-03-06T18:11:50.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:50.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:50.490245+0000) 2024-03-06T18:11:51.487 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:51.489+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:51.490501+0000) 2024-03-06T18:11:52.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:52.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:52.490738+0000) 2024-03-06T18:11:53.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:53.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:53.491016+0000) 2024-03-06T18:11:54.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:54.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:54.491298+0000) 2024-03-06T18:11:55.488 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:55.490+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:55.491507+0000) 2024-03-06T18:11:56.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:56.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:56.491765+0000) 2024-03-06T18:11:57.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:57.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:57.491980+0000) 2024-03-06T18:11:58.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:58.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:58.492175+0000) 2024-03-06T18:11:59.489 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:11:59.491+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:59.492407+0000) 2024-03-06T18:12:00.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:00.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:00.492728+0000) 2024-03-06T18:12:01.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:01.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:01.493010+0000) 2024-03-06T18:12:02.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:02.492+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:02.493282+0000) 2024-03-06T18:12:03.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:03.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:03.490 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:03.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:03.493568+0000) 2024-03-06T18:12:04.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:04.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:04.493891+0000) 2024-03-06T18:12:05.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:05.493+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:05.494173+0000) 2024-03-06T18:12:06.491 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:06.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:06.494441+0000) 2024-03-06T18:12:07.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:07.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:07.494729+0000) 2024-03-06T18:12:08.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:08.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:08.495003+0000) 2024-03-06T18:12:09.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:09.494+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:09.495283+0000) 2024-03-06T18:12:10.492 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:10.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:10.495607+0000) 2024-03-06T18:12:11.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:11.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:11.495900+0000) 2024-03-06T18:12:12.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:12.495+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:12.496150+0000) 2024-03-06T18:12:13.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:13.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:13.496423+0000) 2024-03-06T18:12:14.493 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:14.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:14.496740+0000) 2024-03-06T18:12:15.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:15.496+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:15.496975+0000) 2024-03-06T18:12:16.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:16.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:16.497235+0000) 2024-03-06T18:12:17.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:17.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:17.497485+0000) 2024-03-06T18:12:18.494 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:18.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:18.497741+0000) 2024-03-06T18:12:19.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:19.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:19.497981+0000) 2024-03-06T18:12:20.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:20.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:20.498231+0000) 2024-03-06T18:12:21.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:21.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:21.498467+0000) 2024-03-06T18:12:22.495 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:22.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:22.498744+0000) 2024-03-06T18:12:23.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:23.497+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:23.498981+0000) 2024-03-06T18:12:24.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:24.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:24.499242+0000) 2024-03-06T18:12:25.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:25.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:25.499498+0000) 2024-03-06T18:12:26.496 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:26.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:26.499739+0000) 2024-03-06T18:12:27.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:27.498+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:27.499982+0000) 2024-03-06T18:12:28.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:28.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:28.500220+0000) 2024-03-06T18:12:29.497 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:29.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:29.500495+0000) 2024-03-06T18:12:30.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:30.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:30.500746+0000) 2024-03-06T18:12:31.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:31.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:31.501007+0000) 2024-03-06T18:12:32.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:32.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:32.501260+0000) 2024-03-06T18:12:33.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:33.396 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:33.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:33.499+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:33.501536+0000) 2024-03-06T18:12:34.498 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:34.500+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:34.501793+0000) 2024-03-06T18:12:35.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:35.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:35.502028+0000) 2024-03-06T18:12:36.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:36.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:36.502283+0000) 2024-03-06T18:12:37.499 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:37.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:37.502511+0000) 2024-03-06T18:12:38.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:38.501+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:38.502748+0000) 2024-03-06T18:12:39.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:39.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:39.503017+0000) 2024-03-06T18:12:40.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:40.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:40.503272+0000) 2024-03-06T18:12:41.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:41.502+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:41.503509+0000) 2024-03-06T18:12:42.500 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:42.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:42.503766+0000) 2024-03-06T18:12:43.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:43.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:43.504011+0000) 2024-03-06T18:12:44.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:44.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:44.504280+0000) 2024-03-06T18:12:45.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:45.503+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:45.504517+0000) 2024-03-06T18:12:46.501 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:46.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:46.504761+0000) 2024-03-06T18:12:47.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:47.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:47.504958+0000) 2024-03-06T18:12:48.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:48.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:48.505206+0000) 2024-03-06T18:12:49.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:49.504+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:49.505475+0000) 2024-03-06T18:12:50.502 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:50.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:50.505747+0000) 2024-03-06T18:12:51.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:51.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:51.506013+0000) 2024-03-06T18:12:52.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:52.505+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:52.506243+0000) 2024-03-06T18:12:53.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:53.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:53.506499+0000) 2024-03-06T18:12:54.503 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:54.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:54.506765+0000) 2024-03-06T18:12:55.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:55.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:55.507019+0000) 2024-03-06T18:12:56.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:56.506+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:56.507278+0000) 2024-03-06T18:12:57.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:57.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:57.507540+0000) 2024-03-06T18:12:58.504 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:58.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:58.507832+0000) 2024-03-06T18:12:59.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:12:59.507+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:59.508105+0000) 2024-03-06T18:13:00.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:00.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:00.508359+0000) 2024-03-06T18:13:01.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:01.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:01.508601+0000) 2024-03-06T18:13:02.505 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:02.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:02.508889+0000) 2024-03-06T18:13:03.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:03.508+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:03.509145+0000) 2024-03-06T18:13:03.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:03.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:04.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:04.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:04.509402+0000) 2024-03-06T18:13:05.506 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:05.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:05.509671+0000) 2024-03-06T18:13:06.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:06.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:06.509908+0000) 2024-03-06T18:13:07.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:07.509+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:07.510144+0000) 2024-03-06T18:13:08.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:08.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:08.510397+0000) 2024-03-06T18:13:09.507 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:09.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:09.510711+0000) 2024-03-06T18:13:10.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:10.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:10.510945+0000) 2024-03-06T18:13:11.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:11.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:11.511120+0000) 2024-03-06T18:13:12.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:12.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:12.511361+0000) 2024-03-06T18:13:13.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:13.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:13.511619+0000) 2024-03-06T18:13:14.508 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:14.510+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:14.511907+0000) 2024-03-06T18:13:15.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:15.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:15.512170+0000) 2024-03-06T18:13:16.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:16.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:16.512425+0000) 2024-03-06T18:13:17.509 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:17.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:17.512692+0000) 2024-03-06T18:13:18.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:18.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:18.512942+0000) 2024-03-06T18:13:19.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:19.511+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:19.513197+0000) 2024-03-06T18:13:20.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:20.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:20.513455+0000) 2024-03-06T18:13:21.510 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:21.512+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:21.513755+0000) 2024-03-06T18:13:22.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:22.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:22.514010+0000) 2024-03-06T18:13:23.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:23.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:23.514250+0000) 2024-03-06T18:13:24.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:24.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:24.514519+0000) 2024-03-06T18:13:25.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:25.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:25.514764+0000) 2024-03-06T18:13:26.511 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:26.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:26.514993+0000) 2024-03-06T18:13:27.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:27.513+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:27.515185+0000) 2024-03-06T18:13:28.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:28.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:28.515439+0000) 2024-03-06T18:13:29.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:29.514+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:29.515769+0000) 2024-03-06T18:13:30.512 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:30.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:30.515991+0000) 2024-03-06T18:13:31.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:31.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:31.516240+0000) 2024-03-06T18:13:32.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:32.515+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:32.516470+0000) 2024-03-06T18:13:33.513 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:33.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:33.516776+0000) 2024-03-06T18:13:33.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:33.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:34.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:34.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:34.517021+0000) 2024-03-06T18:13:35.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:35.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:35.517278+0000) 2024-03-06T18:13:36.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:36.516+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:36.517526+0000) 2024-03-06T18:13:37.514 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:37.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:37.517787+0000) 2024-03-06T18:13:38.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:38.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:38.517975+0000) 2024-03-06T18:13:39.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:39.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:39.518232+0000) 2024-03-06T18:13:40.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:40.517+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:40.518471+0000) 2024-03-06T18:13:41.515 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:41.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:41.518761+0000) 2024-03-06T18:13:42.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:42.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:42.519002+0000) 2024-03-06T18:13:43.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:43.518+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:43.519298+0000) 2024-03-06T18:13:44.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:44.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:44.519591+0000) 2024-03-06T18:13:45.516 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:45.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:45.519948+0000) 2024-03-06T18:13:46.517 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:46.519+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:46.520245+0000) 2024-03-06T18:13:47.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:47.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:47.520568+0000) 2024-03-06T18:13:48.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:48.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:48.520879+0000) 2024-03-06T18:13:49.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:49.520+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:49.521188+0000) 2024-03-06T18:13:50.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:50.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:50.521491+0000) 2024-03-06T18:13:51.518 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:51.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:51.521801+0000) 2024-03-06T18:13:52.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:52.521+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:52.522116+0000) 2024-03-06T18:13:53.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:53.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:53.522419+0000) 2024-03-06T18:13:54.519 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:54.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:54.522788+0000) 2024-03-06T18:13:55.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:55.522+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:55.523075+0000) 2024-03-06T18:13:56.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:56.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:56.523365+0000) 2024-03-06T18:13:57.520 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:57.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:57.523593+0000) 2024-03-06T18:13:58.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:58.523+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:58.523961+0000) 2024-03-06T18:13:59.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:13:59.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:59.524286+0000) 2024-03-06T18:14:00.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:00.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:00.524580+0000) 2024-03-06T18:14:01.521 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:01.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:01.524877+0000) 2024-03-06T18:14:02.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:02.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:02.525189+0000) 2024-03-06T18:14:03.522 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:03.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:03.525496+0000) 2024-03-06T18:14:03.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:03.829 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:04.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:04.524+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:04.525815+0000) 2024-03-06T18:14:05.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:05.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:05.526105+0000) 2024-03-06T18:14:06.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:06.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:06.526408+0000) 2024-03-06T18:14:07.523 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:07.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:07.526756+0000) 2024-03-06T18:14:08.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:08.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:08.527050+0000) 2024-03-06T18:14:09.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:09.525+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:09.527309+0000) 2024-03-06T18:14:10.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:10.526+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:10.527623+0000) 2024-03-06T18:14:11.524 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:11.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:11.527978+0000) 2024-03-06T18:14:12.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:12.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:12.528283+0000) 2024-03-06T18:14:13.525 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:13.527+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:13.528596+0000) 2024-03-06T18:14:14.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:14.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:14.528955+0000) 2024-03-06T18:14:15.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:15.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:15.529262+0000) 2024-03-06T18:14:16.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:16.528+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:16.529567+0000) 2024-03-06T18:14:17.526 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:17.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:17.529882+0000) 2024-03-06T18:14:18.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:18.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:18.530180+0000) 2024-03-06T18:14:19.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:19.529+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:19.530502+0000) 2024-03-06T18:14:20.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:20.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:20.530804+0000) 2024-03-06T18:14:21.527 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:21.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:21.531079+0000) 2024-03-06T18:14:22.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:22.530+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:22.531394+0000) 2024-03-06T18:14:23.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:23.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:23.531738+0000) 2024-03-06T18:14:24.528 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:24.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:24.532014+0000) 2024-03-06T18:14:25.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:25.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:25.532208+0000) 2024-03-06T18:14:26.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:26.531+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:26.532439+0000) 2024-03-06T18:14:27.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:27.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:27.532746+0000) 2024-03-06T18:14:28.529 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:28.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:28.532998+0000) 2024-03-06T18:14:29.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:29.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:29.533245+0000) 2024-03-06T18:14:30.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:30.532+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:30.533481+0000) 2024-03-06T18:14:31.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:31.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:31.533743+0000) 2024-03-06T18:14:32.530 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:32.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:32.533976+0000) 2024-03-06T18:14:33.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:33.533+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:33.534239+0000) 2024-03-06T18:14:33.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:33.967 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:34.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:34.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:34.534539+0000) 2024-03-06T18:14:35.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:35.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:35.534799+0000) 2024-03-06T18:14:36.531 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:36.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:36.535057+0000) 2024-03-06T18:14:37.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:37.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:37.535315+0000) 2024-03-06T18:14:38.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:38.534+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:38.535538+0000) 2024-03-06T18:14:39.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:39.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:39.535767+0000) 2024-03-06T18:14:40.532 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:40.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:40.536013+0000) 2024-03-06T18:14:41.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:41.535+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:41.536275+0000) 2024-03-06T18:14:42.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:42.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:42.536525+0000) 2024-03-06T18:14:43.533 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:43.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:43.536782+0000) 2024-03-06T18:14:44.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:44.536+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:44.537093+0000) 2024-03-06T18:14:45.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:45.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:45.537377+0000) 2024-03-06T18:14:46.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:46.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:46.537683+0000) 2024-03-06T18:14:47.534 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:47.537+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:47.538000+0000) 2024-03-06T18:14:48.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:48.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:48.538311+0000) 2024-03-06T18:14:49.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:49.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:49.538675+0000) 2024-03-06T18:14:50.535 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:50.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:50.538991+0000) 2024-03-06T18:14:51.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:51.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:51.539306+0000) 2024-03-06T18:14:52.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:52.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:52.539620+0000) 2024-03-06T18:14:53.536 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:53.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:53.539988+0000) 2024-03-06T18:14:54.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:54.538+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:54.540293+0000) 2024-03-06T18:14:55.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:55.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:55.540579+0000) 2024-03-06T18:14:56.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:56.539+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:56.540878+0000) 2024-03-06T18:14:57.537 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:57.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:57.541164+0000) 2024-03-06T18:14:58.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:58.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:58.541464+0000) 2024-03-06T18:14:59.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:14:59.540+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:59.541770+0000) 2024-03-06T18:15:00.538 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:00.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:00.542052+0000) 2024-03-06T18:15:01.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:01.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:01.542353+0000) 2024-03-06T18:15:02.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:02.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:02.542685+0000) 2024-03-06T18:15:03.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:03.541+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:03.542944+0000) 2024-03-06T18:15:04.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:04.122 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:04.539 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:04.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:04.543263+0000) 2024-03-06T18:15:05.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:05.542+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:05.543557+0000) 2024-03-06T18:15:06.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:06.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:06.543855+0000) 2024-03-06T18:15:07.540 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:07.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:07.544168+0000) 2024-03-06T18:15:08.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:08.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:08.544482+0000) 2024-03-06T18:15:09.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:09.543+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:09.544791+0000) 2024-03-06T18:15:10.541 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:10.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:10.545112+0000) 2024-03-06T18:15:11.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:11.544+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:11.545422+0000) 2024-03-06T18:15:12.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:12.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:12.545750+0000) 2024-03-06T18:15:13.542 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:13.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:13.546053+0000) 2024-03-06T18:15:14.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:14.545+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:14.546367+0000) 2024-03-06T18:15:15.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:15.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:15.546730+0000) 2024-03-06T18:15:16.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:16.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:16.546967+0000) 2024-03-06T18:15:17.543 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:17.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:17.547270+0000) 2024-03-06T18:15:18.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:18.546+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:18.547567+0000) 2024-03-06T18:15:19.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:19.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:19.547881+0000) 2024-03-06T18:15:20.544 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:20.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:20.548184+0000) 2024-03-06T18:15:21.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:21.547+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:21.548474+0000) 2024-03-06T18:15:22.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:22.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:22.548750+0000) 2024-03-06T18:15:23.545 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:23.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:23.549047+0000) 2024-03-06T18:15:24.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:24.548+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:24.549342+0000) 2024-03-06T18:15:25.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:25.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:25.549688+0000) 2024-03-06T18:15:26.546 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:26.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:26.550009+0000) 2024-03-06T18:15:27.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:27.549+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:27.550313+0000) 2024-03-06T18:15:28.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:28.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:28.550623+0000) 2024-03-06T18:15:29.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:29.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:29.550978+0000) 2024-03-06T18:15:30.547 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:30.550+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:30.551243+0000) 2024-03-06T18:15:31.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:31.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:31.551532+0000) 2024-03-06T18:15:32.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:32.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:32.551828+0000) 2024-03-06T18:15:33.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:33.551+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:33.552107+0000) 2024-03-06T18:15:34.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:34.269 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:34.548 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:34.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:34.552416+0000) 2024-03-06T18:15:35.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:35.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:35.552743+0000) 2024-03-06T18:15:36.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:36.552+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:36.553023+0000) 2024-03-06T18:15:37.549 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:37.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:37.553278+0000) 2024-03-06T18:15:38.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:38.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:38.553533+0000) 2024-03-06T18:15:39.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:39.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:39.553790+0000) 2024-03-06T18:15:40.550 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:40.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:40.554047+0000) 2024-03-06T18:15:41.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:41.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:41.554302+0000) 2024-03-06T18:15:42.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:42.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:42.554549+0000) 2024-03-06T18:15:43.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:43.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:43.554849+0000) 2024-03-06T18:15:44.551 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:44.553+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:44.555139+0000) 2024-03-06T18:15:45.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:45.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:45.555457+0000) 2024-03-06T18:15:46.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:46.554+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:46.555784+0000) 2024-03-06T18:15:47.552 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:47.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:47.556068+0000) 2024-03-06T18:15:48.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:48.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:48.556375+0000) 2024-03-06T18:15:49.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:49.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:49.556734+0000) 2024-03-06T18:15:50.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:50.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:50.556980+0000) 2024-03-06T18:15:51.553 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:51.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:51.557233+0000) 2024-03-06T18:15:52.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:52.555+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:52.557479+0000) 2024-03-06T18:15:53.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:53.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:53.557797+0000) 2024-03-06T18:15:54.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:54.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:54.558017+0000) 2024-03-06T18:15:55.554 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:55.556+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:55.558197+0000) 2024-03-06T18:15:56.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:56.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:56.558413+0000) 2024-03-06T18:15:57.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:57.557+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:57.558685+0000) 2024-03-06T18:15:58.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:58.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:58.558938+0000) 2024-03-06T18:15:59.555 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:15:59.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:59.559194+0000) 2024-03-06T18:16:00.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:00.558+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:00.559447+0000) 2024-03-06T18:16:01.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:01.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:01.559741+0000) 2024-03-06T18:16:02.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:02.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:02.559994+0000) 2024-03-06T18:16:03.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:03.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:03.560256+0000) 2024-03-06T18:16:04.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:04.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:04.556 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:04.559+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:04.560540+0000) 2024-03-06T18:16:05.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:05.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:05.560745+0000) 2024-03-06T18:16:06.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:06.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:06.561002+0000) 2024-03-06T18:16:07.557 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:07.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:07.561243+0000) 2024-03-06T18:16:08.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:08.560+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:08.561480+0000) 2024-03-06T18:16:09.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:09.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:09.561748+0000) 2024-03-06T18:16:10.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:10.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:10.561999+0000) 2024-03-06T18:16:11.558 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:11.561+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:11.562263+0000) 2024-03-06T18:16:12.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:12.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:12.562516+0000) 2024-03-06T18:16:13.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:13.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:13.562770+0000) 2024-03-06T18:16:14.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:14.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:14.563033+0000) 2024-03-06T18:16:15.559 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:15.562+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:15.563277+0000) 2024-03-06T18:16:16.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:16.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:16.563517+0000) 2024-03-06T18:16:17.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:17.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:17.563764+0000) 2024-03-06T18:16:18.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:18.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:18.564013+0000) 2024-03-06T18:16:19.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:19.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:19.564247+0000) 2024-03-06T18:16:20.560 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:20.563+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:20.564482+0000) 2024-03-06T18:16:21.561 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:21.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:21.564795+0000) 2024-03-06T18:16:22.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:22.564+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:22.565050+0000) 2024-03-06T18:16:23.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:23.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:23.565309+0000) 2024-03-06T18:16:24.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:24.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:24.565578+0000) 2024-03-06T18:16:25.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:25.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:25.565835+0000) 2024-03-06T18:16:26.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:26.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:26.566089+0000) 2024-03-06T18:16:27.562 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:27.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:27.566345+0000) 2024-03-06T18:16:28.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:28.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:28.566599+0000) 2024-03-06T18:16:29.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:29.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:29.566837+0000) 2024-03-06T18:16:30.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:30.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:30.567068+0000) 2024-03-06T18:16:31.563 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:31.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:31.567324+0000) 2024-03-06T18:16:32.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:32.565+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:32.567558+0000) 2024-03-06T18:16:33.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:33.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:33.567828+0000) 2024-03-06T18:16:34.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:34.566+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:34.568059+0000) 2024-03-06T18:16:34.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:34.580 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:35.564 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:35.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:35.568324+0000) 2024-03-06T18:16:36.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:36.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:36.568573+0000) 2024-03-06T18:16:37.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:37.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:37.568811+0000) 2024-03-06T18:16:38.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:38.567+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:38.569035+0000) 2024-03-06T18:16:39.565 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:39.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:39.569294+0000) 2024-03-06T18:16:40.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:40.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:40.569525+0000) 2024-03-06T18:16:41.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:41.568+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:41.569782+0000) 2024-03-06T18:16:42.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:42.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:42.570018+0000) 2024-03-06T18:16:43.566 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:43.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:43.570280+0000) 2024-03-06T18:16:44.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:44.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:44.570525+0000) 2024-03-06T18:16:45.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:45.569+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:45.570764+0000) 2024-03-06T18:16:46.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:46.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:46.570996+0000) 2024-03-06T18:16:47.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:47.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:47.571243+0000) 2024-03-06T18:16:48.567 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:48.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:48.571428+0000) 2024-03-06T18:16:49.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:49.570+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:49.571610+0000) 2024-03-06T18:16:50.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:50.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:50.571887+0000) 2024-03-06T18:16:51.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:51.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:51.572137+0000) 2024-03-06T18:16:52.568 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:52.571+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:52.572371+0000) 2024-03-06T18:16:53.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:53.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:53.572665+0000) 2024-03-06T18:16:54.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:54.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:54.572925+0000) 2024-03-06T18:16:55.569 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:55.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:55.573164+0000) 2024-03-06T18:16:56.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:56.572+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:56.573419+0000) 2024-03-06T18:16:57.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:57.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:57.573718+0000) 2024-03-06T18:16:58.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:58.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:58.573965+0000) 2024-03-06T18:16:59.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:16:59.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:59.574202+0000) 2024-03-06T18:17:00.570 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:00.573+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:00.574446+0000) 2024-03-06T18:17:01.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:01.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:01.574746+0000) 2024-03-06T18:17:02.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:02.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:02.574996+0000) 2024-03-06T18:17:03.571 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:03.574+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:03.575326+0000) 2024-03-06T18:17:04.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:04.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:04.575713+0000) 2024-03-06T18:17:04.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:04.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:05.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:05.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:05.576018+0000) 2024-03-06T18:17:06.572 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:06.575+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:06.576343+0000) 2024-03-06T18:17:07.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:07.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:07.576615+0000) 2024-03-06T18:17:08.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:08.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:08.576984+0000) 2024-03-06T18:17:09.573 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:09.576+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:09.577267+0000) 2024-03-06T18:17:10.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:10.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:10.577579+0000) 2024-03-06T18:17:11.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:11.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:11.577898+0000) 2024-03-06T18:17:12.574 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:12.577+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:12.578191+0000) 2024-03-06T18:17:13.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:13.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:13.578479+0000) 2024-03-06T18:17:14.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:14.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:14.578779+0000) 2024-03-06T18:17:15.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:15.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:15.579075+0000) 2024-03-06T18:17:16.575 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:16.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:16.579383+0000) 2024-03-06T18:17:17.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:17.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:17.579612+0000) 2024-03-06T18:17:18.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:18.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:18.579983+0000) 2024-03-06T18:17:19.576 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:19.578+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:19.580282+0000) 2024-03-06T18:17:20.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:20.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:20.580595+0000) 2024-03-06T18:17:21.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:21.579+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:21.580936+0000) 2024-03-06T18:17:22.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:22.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:22.581233+0000) 2024-03-06T18:17:23.577 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:23.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:23.581534+0000) 2024-03-06T18:17:24.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:24.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:24.581850+0000) 2024-03-06T18:17:25.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:25.580+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:25.582111+0000) 2024-03-06T18:17:26.578 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:26.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:26.582408+0000) 2024-03-06T18:17:27.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:27.581+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:27.582760+0000) 2024-03-06T18:17:28.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:28.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:28.583054+0000) 2024-03-06T18:17:29.579 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:29.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:29.583357+0000) 2024-03-06T18:17:30.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:30.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:30.583717+0000) 2024-03-06T18:17:31.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:31.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:31.583995+0000) 2024-03-06T18:17:32.580 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:32.582+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:32.584288+0000) 2024-03-06T18:17:33.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:33.583+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:33.584591+0000) 2024-03-06T18:17:34.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:34.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:34.584968+0000) 2024-03-06T18:17:34.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:34.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:35.581 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:35.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:35.585287+0000) 2024-03-06T18:17:36.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:36.584+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:36.585593+0000) 2024-03-06T18:17:37.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:37.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:37.585930+0000) 2024-03-06T18:17:38.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:38.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:38.586227+0000) 2024-03-06T18:17:39.582 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:39.585+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:39.586531+0000) 2024-03-06T18:17:40.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:40.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:40.586840+0000) 2024-03-06T18:17:41.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:41.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:41.587125+0000) 2024-03-06T18:17:42.583 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:42.586+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:42.587419+0000) 2024-03-06T18:17:43.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:43.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:43.587752+0000) 2024-03-06T18:17:44.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:44.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:44.588083+0000) 2024-03-06T18:17:45.584 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:45.587+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:45.588387+0000) 2024-03-06T18:17:46.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:46.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:46.588725+0000) 2024-03-06T18:17:47.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:47.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:47.589007+0000) 2024-03-06T18:17:48.585 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:48.588+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:48.589311+0000) 2024-03-06T18:17:49.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:49.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:49.589624+0000) 2024-03-06T18:17:50.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:50.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:50.589996+0000) 2024-03-06T18:17:51.586 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:51.589+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:51.590329+0000) 2024-03-06T18:17:52.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:52.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:52.590670+0000) 2024-03-06T18:17:53.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:53.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:53.590976+0000) 2024-03-06T18:17:54.587 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:54.590+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:54.591314+0000) 2024-03-06T18:17:55.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:55.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:55.591686+0000) 2024-03-06T18:17:56.588 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:56.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:56.591993+0000) 2024-03-06T18:17:57.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:57.591+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:57.592296+0000) 2024-03-06T18:17:58.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:58.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:58.592588+0000) 2024-03-06T18:17:59.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:17:59.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:59.592839+0000) 2024-03-06T18:18:00.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:00.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:00.593122+0000) 2024-03-06T18:18:01.589 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:01.592+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:01.593360+0000) 2024-03-06T18:18:02.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:02.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:02.593681+0000) 2024-03-06T18:18:03.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:03.593+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:03.593989+0000) 2024-03-06T18:18:04.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:04.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:04.594295+0000) 2024-03-06T18:18:05.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:05.039 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:05.590 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:05.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:05.594623+0000) 2024-03-06T18:18:06.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:06.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:06.594983+0000) 2024-03-06T18:18:07.591 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:07.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:07.595280+0000) 2024-03-06T18:18:08.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:08.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:08.595568+0000) 2024-03-06T18:18:09.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:09.594+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:09.595859+0000) 2024-03-06T18:18:10.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:10.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:10.596156+0000) 2024-03-06T18:18:11.592 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:11.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:11.596464+0000) 2024-03-06T18:18:12.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:12.595+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:12.596789+0000) 2024-03-06T18:18:13.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:13.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:13.597081+0000) 2024-03-06T18:18:14.593 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:14.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:14.597402+0000) 2024-03-06T18:18:15.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:15.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:15.597789+0000) 2024-03-06T18:18:16.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:16.596+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:16.598050+0000) 2024-03-06T18:18:17.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:17.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:17.598334+0000) 2024-03-06T18:18:18.594 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:18.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:18.598677+0000) 2024-03-06T18:18:19.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:19.597+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:19.598925+0000) 2024-03-06T18:18:20.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:20.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:20.599230+0000) 2024-03-06T18:18:21.595 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:21.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:21.599521+0000) 2024-03-06T18:18:22.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:22.598+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:22.599826+0000) 2024-03-06T18:18:23.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:23.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:23.600134+0000) 2024-03-06T18:18:24.596 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:24.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:24.600442+0000) 2024-03-06T18:18:25.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:25.599+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:25.600788+0000) 2024-03-06T18:18:26.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:26.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:26.601084+0000) 2024-03-06T18:18:27.597 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:27.600+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:27.601378+0000) 2024-03-06T18:18:28.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:28.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:28.601725+0000) 2024-03-06T18:18:29.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:29.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:29.602014+0000) 2024-03-06T18:18:30.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:30.601+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:30.602334+0000) 2024-03-06T18:18:31.598 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:31.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:31.602681+0000) 2024-03-06T18:18:32.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:32.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:32.603006+0000) 2024-03-06T18:18:33.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:33.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:33.603245+0000) 2024-03-06T18:18:34.599 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:34.602+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:34.603569+0000) 2024-03-06T18:18:35.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:35.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:35.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:35.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:35.603896+0000) 2024-03-06T18:18:36.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:36.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:36.604203+0000) 2024-03-06T18:18:37.600 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:37.603+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:37.604525+0000) 2024-03-06T18:18:38.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:38.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:38.604820+0000) 2024-03-06T18:18:39.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:39.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:39.605139+0000) 2024-03-06T18:18:40.601 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:40.604+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:40.605464+0000) 2024-03-06T18:18:41.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:41.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:41.605776+0000) 2024-03-06T18:18:42.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:42.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:42.606086+0000) 2024-03-06T18:18:43.602 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:43.605+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:43.606382+0000) 2024-03-06T18:18:44.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:44.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:44.606745+0000) 2024-03-06T18:18:45.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:45.606+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:45.607157+0000) 2024-03-06T18:18:46.603 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:46.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:46.607461+0000) 2024-03-06T18:18:47.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:47.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:47.607779+0000) 2024-03-06T18:18:48.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:48.607+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:48.608095+0000) 2024-03-06T18:18:49.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:49.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:49.608378+0000) 2024-03-06T18:18:50.604 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:50.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:50.608740+0000) 2024-03-06T18:18:51.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:51.608+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:51.609044+0000) 2024-03-06T18:18:52.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:52.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:52.609334+0000) 2024-03-06T18:18:53.605 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:53.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:53.609622+0000) 2024-03-06T18:18:54.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:54.609+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:54.610011+0000) 2024-03-06T18:18:55.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:55.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:55.610347+0000) 2024-03-06T18:18:56.606 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:56.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:56.610706+0000) 2024-03-06T18:18:57.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:57.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:57.610993+0000) 2024-03-06T18:18:58.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:58.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:58.611270+0000) 2024-03-06T18:18:59.607 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:18:59.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:59.611560+0000) 2024-03-06T18:19:00.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:00.610+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:00.611899+0000) 2024-03-06T18:19:01.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:01.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:01.612203+0000) 2024-03-06T18:19:02.608 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:02.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:02.612509+0000) 2024-03-06T18:19:03.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:03.611+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:03.612823+0000) 2024-03-06T18:19:04.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:04.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:04.613124+0000) 2024-03-06T18:19:05.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:05.348 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:05.609 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:05.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:05.613448+0000) 2024-03-06T18:19:06.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:06.612+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:06.613793+0000) 2024-03-06T18:19:07.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:07.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:07.614079+0000) 2024-03-06T18:19:08.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:08.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:08.614389+0000) 2024-03-06T18:19:09.610 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:09.613+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:09.614726+0000) 2024-03-06T18:19:10.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:10.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:10.615038+0000) 2024-03-06T18:19:11.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:11.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:11.615329+0000) 2024-03-06T18:19:12.611 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:12.614+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:12.615671+0000) 2024-03-06T18:19:13.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:13.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:13.615965+0000) 2024-03-06T18:19:14.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:14.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:14.616260+0000) 2024-03-06T18:19:15.612 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:15.615+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:15.616575+0000) 2024-03-06T18:19:16.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:16.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:16.616891+0000) 2024-03-06T18:19:17.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:17.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:17.617208+0000) 2024-03-06T18:19:18.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:18.616+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:18.617502+0000) 2024-03-06T18:19:19.613 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:19.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:19.617806+0000) 2024-03-06T18:19:20.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:20.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:20.618059+0000) 2024-03-06T18:19:21.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:21.617+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:21.618366+0000) 2024-03-06T18:19:22.614 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:22.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:22.618697+0000) 2024-03-06T18:19:23.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:23.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:23.619017+0000) 2024-03-06T18:19:24.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:24.618+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:24.619345+0000) 2024-03-06T18:19:25.615 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:25.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:25.619702+0000) 2024-03-06T18:19:26.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:26.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:26.620022+0000) 2024-03-06T18:19:27.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:27.619+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:27.620317+0000) 2024-03-06T18:19:28.616 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:28.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:28.620679+0000) 2024-03-06T18:19:29.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:29.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:29.621003+0000) 2024-03-06T18:19:30.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:30.620+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:30.621301+0000) 2024-03-06T18:19:31.617 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:31.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:31.621588+0000) 2024-03-06T18:19:32.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:32.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:32.621925+0000) 2024-03-06T18:19:33.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:33.621+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:33.622222+0000) 2024-03-06T18:19:34.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:34.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:34.622547+0000) 2024-03-06T18:19:35.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:35.490 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:35.618 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:35.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:35.622860+0000) 2024-03-06T18:19:36.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:36.622+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:36.623169+0000) 2024-03-06T18:19:37.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:37.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:37.623462+0000) 2024-03-06T18:19:38.619 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:38.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:38.623842+0000) 2024-03-06T18:19:39.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:39.623+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:39.624133+0000) 2024-03-06T18:19:40.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:40.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:40.624450+0000) 2024-03-06T18:19:41.620 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:41.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:41.624792+0000) 2024-03-06T18:19:42.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:42.624+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:42.625075+0000) 2024-03-06T18:19:43.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:43.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:43.625388+0000) 2024-03-06T18:19:44.621 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:44.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:44.625746+0000) 2024-03-06T18:19:45.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:45.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:45.626050+0000) 2024-03-06T18:19:46.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:46.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:46.626364+0000) 2024-03-06T18:19:47.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:47.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:47.626711+0000) 2024-03-06T18:19:48.622 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:48.625+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:48.626950+0000) 2024-03-06T18:19:49.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:49.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:49.627251+0000) 2024-03-06T18:19:50.623 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:50.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:50.627573+0000) 2024-03-06T18:19:51.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:51.626+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:51.627867+0000) 2024-03-06T18:19:52.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:52.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:52.628168+0000) 2024-03-06T18:19:53.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:53.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:53.628471+0000) 2024-03-06T18:19:54.624 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:54.627+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:54.628781+0000) 2024-03-06T18:19:55.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:55.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:55.629072+0000) 2024-03-06T18:19:56.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:56.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:56.629363+0000) 2024-03-06T18:19:57.625 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:57.628+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:57.629723+0000) 2024-03-06T18:19:58.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:58.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:58.630028+0000) 2024-03-06T18:19:59.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:19:59.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:59.630315+0000) 2024-03-06T18:20:00.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:00.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:00.630581+0000) 2024-03-06T18:20:01.626 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:01.629+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:01.630823+0000) 2024-03-06T18:20:02.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:02.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:02.631054+0000) 2024-03-06T18:20:03.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:03.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:03.631296+0000) 2024-03-06T18:20:04.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:04.630+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:04.631554+0000) 2024-03-06T18:20:05.627 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:05.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:05.631813+0000) 2024-03-06T18:20:05.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:05.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:06.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:06.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:06.632065+0000) 2024-03-06T18:20:07.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:07.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:07.632320+0000) 2024-03-06T18:20:08.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:08.631+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:08.632573+0000) 2024-03-06T18:20:09.628 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:09.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:09.632828+0000) 2024-03-06T18:20:10.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:10.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:10.633034+0000) 2024-03-06T18:20:11.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:11.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:11.633292+0000) 2024-03-06T18:20:12.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:12.632+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:12.633538+0000) 2024-03-06T18:20:13.629 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:13.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:13.633782+0000) 2024-03-06T18:20:14.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:14.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:14.634025+0000) 2024-03-06T18:20:15.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:15.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:15.634228+0000) 2024-03-06T18:20:16.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:16.633+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:16.634404+0000) 2024-03-06T18:20:17.630 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:17.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:17.634683+0000) 2024-03-06T18:20:18.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:18.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:18.634940+0000) 2024-03-06T18:20:19.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:19.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:19.635195+0000) 2024-03-06T18:20:20.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:20.634+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:20.635443+0000) 2024-03-06T18:20:21.631 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:21.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:21.635730+0000) 2024-03-06T18:20:22.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:22.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:22.635987+0000) 2024-03-06T18:20:23.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:23.635+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:23.636266+0000) 2024-03-06T18:20:24.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:24.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:24.636523+0000) 2024-03-06T18:20:25.632 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:25.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:25.636761+0000) 2024-03-06T18:20:26.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:26.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:26.637019+0000) 2024-03-06T18:20:27.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:27.636+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:27.637270+0000) 2024-03-06T18:20:28.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:28.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:28.637520+0000) 2024-03-06T18:20:29.633 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:29.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:29.637777+0000) 2024-03-06T18:20:30.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:30.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:30.638012+0000) 2024-03-06T18:20:31.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:31.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:31.638328+0000) 2024-03-06T18:20:32.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:32.637+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:32.638558+0000) 2024-03-06T18:20:33.634 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:33.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:33.638834+0000) 2024-03-06T18:20:34.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:34.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:34.639084+0000) 2024-03-06T18:20:35.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:35.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:35.639320+0000) 2024-03-06T18:20:35.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:35.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:36.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:36.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:36.639572+0000) 2024-03-06T18:20:37.635 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:37.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:37.639832+0000) 2024-03-06T18:20:38.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:38.638+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:38.640055+0000) 2024-03-06T18:20:39.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:39.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:39.640321+0000) 2024-03-06T18:20:40.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:40.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:40.640558+0000) 2024-03-06T18:20:41.636 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:41.639+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:41.640815+0000) 2024-03-06T18:20:42.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:42.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:42.641072+0000) 2024-03-06T18:20:43.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:43.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:43.641380+0000) 2024-03-06T18:20:44.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:44.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:44.641624+0000) 2024-03-06T18:20:45.637 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:45.640+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:45.641916+0000) 2024-03-06T18:20:46.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:46.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:46.642166+0000) 2024-03-06T18:20:47.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:47.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:47.642425+0000) 2024-03-06T18:20:48.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:48.641+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:48.642702+0000) 2024-03-06T18:20:49.638 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:49.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:49.642971+0000) 2024-03-06T18:20:50.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:50.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:50.643206+0000) 2024-03-06T18:20:51.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:51.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:51.643445+0000) 2024-03-06T18:20:52.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:52.642+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:52.643735+0000) 2024-03-06T18:20:53.639 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:53.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:53.643981+0000) 2024-03-06T18:20:54.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:54.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:54.644243+0000) 2024-03-06T18:20:55.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:55.643+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:55.644504+0000) 2024-03-06T18:20:56.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:56.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:56.644756+0000) 2024-03-06T18:20:57.640 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:57.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:57.645009+0000) 2024-03-06T18:20:58.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:58.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:58.645261+0000) 2024-03-06T18:20:59.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:20:59.644+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:59.645522+0000) 2024-03-06T18:21:00.641 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:00.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:00.645782+0000) 2024-03-06T18:21:01.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:01.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:01.646044+0000) 2024-03-06T18:21:02.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:02.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:02.646297+0000) 2024-03-06T18:21:03.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:03.645+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:03.646565+0000) 2024-03-06T18:21:04.642 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:04.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:04.646806+0000) 2024-03-06T18:21:05.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:05.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:05.647049+0000) 2024-03-06T18:21:05.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:05.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:06.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:06.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:06.647231+0000) 2024-03-06T18:21:07.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:07.646+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:07.647474+0000) 2024-03-06T18:21:08.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:08.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:08.647731+0000) 2024-03-06T18:21:09.643 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:09.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:09.647991+0000) 2024-03-06T18:21:10.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:10.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:10.648244+0000) 2024-03-06T18:21:11.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:11.647+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:11.648495+0000) 2024-03-06T18:21:12.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:12.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:12.648730+0000) 2024-03-06T18:21:13.644 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:13.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:13.648982+0000) 2024-03-06T18:21:14.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:14.648+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:14.649222+0000) 2024-03-06T18:21:15.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:15.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:15.649486+0000) 2024-03-06T18:21:16.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:16.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:16.649739+0000) 2024-03-06T18:21:17.645 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:17.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:17.649996+0000) 2024-03-06T18:21:18.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:18.649+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:18.650228+0000) 2024-03-06T18:21:19.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:19.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:19.650487+0000) 2024-03-06T18:21:20.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:20.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:20.650743+0000) 2024-03-06T18:21:21.646 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:21.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:21.650980+0000) 2024-03-06T18:21:22.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:22.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:22.651159+0000) 2024-03-06T18:21:23.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:23.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:23.651354+0000) 2024-03-06T18:21:24.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:24.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:24.651595+0000) 2024-03-06T18:21:25.647 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:25.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:25.651859+0000) 2024-03-06T18:21:26.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:26.650+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:26.652091+0000) 2024-03-06T18:21:27.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:27.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:27.652346+0000) 2024-03-06T18:21:28.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:28.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:28.652600+0000) 2024-03-06T18:21:29.648 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:29.651+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:29.652904+0000) 2024-03-06T18:21:30.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:30.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:30.653176+0000) 2024-03-06T18:21:31.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:31.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:31.653418+0000) 2024-03-06T18:21:32.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:32.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:32.653707+0000) 2024-03-06T18:21:33.649 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:33.652+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:33.653973+0000) 2024-03-06T18:21:34.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:34.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:34.654232+0000) 2024-03-06T18:21:35.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:35.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:35.654492+0000) 2024-03-06T18:21:36.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:36.094 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:36.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:36.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:36.654745+0000) 2024-03-06T18:21:37.650 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:37.653+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:37.654930+0000) 2024-03-06T18:21:38.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:38.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:38.655163+0000) 2024-03-06T18:21:39.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:39.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:39.655401+0000) 2024-03-06T18:21:40.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:40.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:40.655681+0000) 2024-03-06T18:21:41.651 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:41.654+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:41.655945+0000) 2024-03-06T18:21:42.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:42.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:42.656195+0000) 2024-03-06T18:21:43.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:43.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:43.656448+0000) 2024-03-06T18:21:44.652 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:44.655+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:44.656738+0000) 2024-03-06T18:21:45.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:45.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:45.656997+0000) 2024-03-06T18:21:46.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:46.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:46.657252+0000) 2024-03-06T18:21:47.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:47.656+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:47.657497+0000) 2024-03-06T18:21:48.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:48.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:48.657766+0000) 2024-03-06T18:21:49.653 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:49.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:49.658024+0000) 2024-03-06T18:21:50.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:50.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:50.658204+0000) 2024-03-06T18:21:51.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:51.657+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:51.658461+0000) 2024-03-06T18:21:52.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:52.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:52.658751+0000) 2024-03-06T18:21:53.654 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:53.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:53.658990+0000) 2024-03-06T18:21:54.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:54.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:54.659241+0000) 2024-03-06T18:21:55.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:55.658+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:55.659501+0000) 2024-03-06T18:21:56.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:56.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:56.659744+0000) 2024-03-06T18:21:57.655 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:57.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:57.660006+0000) 2024-03-06T18:21:58.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:58.659+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:58.660255+0000) 2024-03-06T18:21:59.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:21:59.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:59.660510+0000) 2024-03-06T18:22:00.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:00.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:00.660743+0000) 2024-03-06T18:22:01.656 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:01.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:01.660997+0000) 2024-03-06T18:22:02.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:02.660+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:02.661297+0000) 2024-03-06T18:22:03.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:03.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:03.661561+0000) 2024-03-06T18:22:04.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:04.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:04.661799+0000) 2024-03-06T18:22:05.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:05.661+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:05.662057+0000) 2024-03-06T18:22:06.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:06.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:06.657 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:06.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:06.662309+0000) 2024-03-06T18:22:07.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:07.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:07.662569+0000) 2024-03-06T18:22:08.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:08.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:08.662822+0000) 2024-03-06T18:22:09.658 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:09.662+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:09.663076+0000) 2024-03-06T18:22:10.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:10.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:10.663325+0000) 2024-03-06T18:22:11.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:11.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:11.663575+0000) 2024-03-06T18:22:12.659 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:12.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:12.663806+0000) 2024-03-06T18:22:13.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:13.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:13.664072+0000) 2024-03-06T18:22:14.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:14.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:14.664330+0000) 2024-03-06T18:22:15.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:15.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:15.664589+0000) 2024-03-06T18:22:16.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:16.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:16.664750+0000) 2024-03-06T18:22:17.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:17.663+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:17.664945+0000) 2024-03-06T18:22:18.660 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:18.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:18.665182+0000) 2024-03-06T18:22:19.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:19.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:19.665432+0000) 2024-03-06T18:22:20.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:20.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:20.665736+0000) 2024-03-06T18:22:21.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:21.664+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:21.666004+0000) 2024-03-06T18:22:22.661 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:22.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:22.666239+0000) 2024-03-06T18:22:23.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:23.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:23.666495+0000) 2024-03-06T18:22:24.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:24.665+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:24.666750+0000) 2024-03-06T18:22:25.662 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:25.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:25.667007+0000) 2024-03-06T18:22:26.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:26.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:26.667261+0000) 2024-03-06T18:22:27.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:27.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:27.667500+0000) 2024-03-06T18:22:28.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:28.666+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:28.667790+0000) 2024-03-06T18:22:29.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:29.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:29.668049+0000) 2024-03-06T18:22:30.663 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:30.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:30.668280+0000) 2024-03-06T18:22:31.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:31.667+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:31.668543+0000) 2024-03-06T18:22:32.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:32.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:32.668789+0000) 2024-03-06T18:22:33.664 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:33.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:33.669046+0000) 2024-03-06T18:22:34.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:34.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:34.669320+0000) 2024-03-06T18:22:35.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:35.668+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:35.669524+0000) 2024-03-06T18:22:36.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:36.414 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:36.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:36.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:36.669799+0000) 2024-03-06T18:22:37.665 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:37.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:37.670049+0000) 2024-03-06T18:22:38.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:38.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:38.670304+0000) 2024-03-06T18:22:39.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:39.669+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:39.670561+0000) 2024-03-06T18:22:40.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:40.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:40.670795+0000) 2024-03-06T18:22:41.666 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:41.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:41.671070+0000) 2024-03-06T18:22:42.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:42.670+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:42.671306+0000) 2024-03-06T18:22:43.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:43.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:43.671565+0000) 2024-03-06T18:22:44.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:44.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:44.671810+0000) 2024-03-06T18:22:45.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:45.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:45.672006+0000) 2024-03-06T18:22:46.667 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:46.671+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:46.672275+0000) 2024-03-06T18:22:47.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:47.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:47.672515+0000) 2024-03-06T18:22:48.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:48.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:48.672765+0000) 2024-03-06T18:22:49.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:49.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:49.673004+0000) 2024-03-06T18:22:50.668 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:50.672+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:50.673265+0000) 2024-03-06T18:22:51.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:51.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:51.673537+0000) 2024-03-06T18:22:52.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:52.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:52.673795+0000) 2024-03-06T18:22:53.669 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:53.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:53.674038+0000) 2024-03-06T18:22:54.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:54.673+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:54.674299+0000) 2024-03-06T18:22:55.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:55.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:55.674560+0000) 2024-03-06T18:22:56.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:56.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:56.674838+0000) 2024-03-06T18:22:57.670 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:57.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:57.675096+0000) 2024-03-06T18:22:58.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:58.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:58.675347+0000) 2024-03-06T18:22:59.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:22:59.674+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:59.675580+0000) 2024-03-06T18:23:00.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:00.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:00.675817+0000) 2024-03-06T18:23:01.671 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:01.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:01.676084+0000) 2024-03-06T18:23:02.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:02.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:02.676333+0000) 2024-03-06T18:23:03.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:03.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:03.676593+0000) 2024-03-06T18:23:04.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:04.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:04.676902+0000) 2024-03-06T18:23:05.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:05.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:05.677157+0000) 2024-03-06T18:23:06.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:06.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:06.672 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:06.675+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:06.677362+0000) 2024-03-06T18:23:07.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:07.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:07.677615+0000) 2024-03-06T18:23:08.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:08.676+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:08.677923+0000) 2024-03-06T18:23:09.673 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:09.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:09.678179+0000) 2024-03-06T18:23:10.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:10.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:10.678434+0000) 2024-03-06T18:23:11.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:11.677+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:11.678759+0000) 2024-03-06T18:23:12.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:12.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:12.679012+0000) 2024-03-06T18:23:13.674 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:13.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:13.679255+0000) 2024-03-06T18:23:14.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:14.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:14.679540+0000) 2024-03-06T18:23:15.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:15.678+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:15.679812+0000) 2024-03-06T18:23:16.675 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:16.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:16.680082+0000) 2024-03-06T18:23:17.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:17.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:17.680338+0000) 2024-03-06T18:23:18.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:18.679+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:18.680592+0000) 2024-03-06T18:23:19.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:19.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:19.680834+0000) 2024-03-06T18:23:20.676 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:20.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:20.681078+0000) 2024-03-06T18:23:21.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:21.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:21.681353+0000) 2024-03-06T18:23:22.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:22.680+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:22.681599+0000) 2024-03-06T18:23:23.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:23.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:23.681885+0000) 2024-03-06T18:23:24.677 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:24.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:24.682129+0000) 2024-03-06T18:23:25.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:25.681+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:25.682391+0000) 2024-03-06T18:23:26.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:26.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:26.682700+0000) 2024-03-06T18:23:27.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:27.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:27.682937+0000) 2024-03-06T18:23:28.678 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:28.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:28.683170+0000) 2024-03-06T18:23:29.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:29.682+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:29.683421+0000) 2024-03-06T18:23:30.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:30.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:30.683729+0000) 2024-03-06T18:23:31.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:31.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:31.684010+0000) 2024-03-06T18:23:32.679 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:32.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:32.684244+0000) 2024-03-06T18:23:33.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:33.683+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:33.684506+0000) 2024-03-06T18:23:34.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:34.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:34.684771+0000) 2024-03-06T18:23:35.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:35.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:35.685034+0000) 2024-03-06T18:23:36.680 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:36.684+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:36.685299+0000) 2024-03-06T18:23:36.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:36.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:37.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:37.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:37.685541+0000) 2024-03-06T18:23:38.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:38.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:38.685779+0000) 2024-03-06T18:23:39.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:39.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:39.686036+0000) 2024-03-06T18:23:40.681 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:40.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:40.686282+0000) 2024-03-06T18:23:41.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:41.685+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:41.686509+0000) 2024-03-06T18:23:42.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:42.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:42.686785+0000) 2024-03-06T18:23:43.682 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:43.686+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:43.687077+0000) 2024-03-06T18:23:44.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:44.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:44.687396+0000) 2024-03-06T18:23:45.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:45.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:45.687705+0000) 2024-03-06T18:23:46.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:46.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:46.687968+0000) 2024-03-06T18:23:47.683 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:47.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:47.688231+0000) 2024-03-06T18:23:48.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:48.687+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:48.688466+0000) 2024-03-06T18:23:49.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:49.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:49.688813+0000) 2024-03-06T18:23:50.684 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:50.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:50.689061+0000) 2024-03-06T18:23:51.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:51.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:51.689337+0000) 2024-03-06T18:23:52.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:52.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:52.689572+0000) 2024-03-06T18:23:53.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:53.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:53.689831+0000) 2024-03-06T18:23:54.685 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:54.688+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:54.690098+0000) 2024-03-06T18:23:55.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:55.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:55.690356+0000) 2024-03-06T18:23:56.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:56.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:56.690605+0000) 2024-03-06T18:23:57.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:57.689+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:57.690899+0000) 2024-03-06T18:23:58.686 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:58.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:58.691146+0000) 2024-03-06T18:23:59.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:23:59.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:59.691405+0000) 2024-03-06T18:24:00.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:00.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:00.691693+0000) 2024-03-06T18:24:01.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:01.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:01.691952+0000) 2024-03-06T18:24:02.687 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:02.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:02.692190+0000) 2024-03-06T18:24:03.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:03.690+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:03.692439+0000) 2024-03-06T18:24:04.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:04.691+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:04.692735+0000) 2024-03-06T18:24:05.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:05.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:05.692983+0000) 2024-03-06T18:24:06.688 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:06.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:06.693234+0000) 2024-03-06T18:24:06.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:06.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:07.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:07.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:07.693478+0000) 2024-03-06T18:24:08.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:08.692+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:08.693729+0000) 2024-03-06T18:24:09.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:09.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:09.694013+0000) 2024-03-06T18:24:10.689 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:10.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:10.694285+0000) 2024-03-06T18:24:11.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:11.693+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:11.694540+0000) 2024-03-06T18:24:12.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:12.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:12.694790+0000) 2024-03-06T18:24:13.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:13.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:13.695069+0000) 2024-03-06T18:24:14.690 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:14.694+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:14.695346+0000) 2024-03-06T18:24:15.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:15.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:15.695700+0000) 2024-03-06T18:24:16.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:16.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:16.696019+0000) 2024-03-06T18:24:17.691 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:17.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:17.696321+0000) 2024-03-06T18:24:18.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:18.695+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:18.696555+0000) 2024-03-06T18:24:19.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:19.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:19.696857+0000) 2024-03-06T18:24:20.692 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:20.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:20.697159+0000) 2024-03-06T18:24:21.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:21.696+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:21.697456+0000) 2024-03-06T18:24:22.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:22.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:22.697798+0000) 2024-03-06T18:24:23.693 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:23.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:23.698091+0000) 2024-03-06T18:24:24.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:24.697+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:24.698407+0000) 2024-03-06T18:24:25.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:25.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:25.698730+0000) 2024-03-06T18:24:26.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:26.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:26.698976+0000) 2024-03-06T18:24:27.694 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:27.698+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:27.699293+0000) 2024-03-06T18:24:28.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:28.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:28.699610+0000) 2024-03-06T18:24:29.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:29.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:29.699957+0000) 2024-03-06T18:24:30.695 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:30.699+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:30.700255+0000) 2024-03-06T18:24:31.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:31.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:31.700571+0000) 2024-03-06T18:24:32.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:32.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:32.700816+0000) 2024-03-06T18:24:33.696 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:33.700+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:33.701134+0000) 2024-03-06T18:24:34.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:34.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:34.701457+0000) 2024-03-06T18:24:35.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:35.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:35.701782+0000) 2024-03-06T18:24:36.697 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:36.701+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:36.702093+0000) 2024-03-06T18:24:37.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:37.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:37.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:37.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:37.702403+0000) 2024-03-06T18:24:38.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:38.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:38.702734+0000) 2024-03-06T18:24:39.698 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:39.702+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:39.703056+0000) 2024-03-06T18:24:40.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:40.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:40.703357+0000) 2024-03-06T18:24:41.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:41.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:41.703689+0000) 2024-03-06T18:24:42.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:42.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:42.704060+0000) 2024-03-06T18:24:43.699 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:43.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:43.704371+0000) 2024-03-06T18:24:44.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:44.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:44.704729+0000) 2024-03-06T18:24:45.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:45.703+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:45.705032+0000) 2024-03-06T18:24:46.700 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:46.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:46.705330+0000) 2024-03-06T18:24:47.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:47.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:47.705593+0000) 2024-03-06T18:24:48.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:48.704+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:48.705956+0000) 2024-03-06T18:24:49.701 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:49.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:49.706258+0000) 2024-03-06T18:24:50.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:50.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:50.706574+0000) 2024-03-06T18:24:51.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:51.705+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:51.706896+0000) 2024-03-06T18:24:52.702 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:52.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:52.707206+0000) 2024-03-06T18:24:53.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:53.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:53.707498+0000) 2024-03-06T18:24:54.703 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:54.706+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:54.707791+0000) 2024-03-06T18:24:55.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:55.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:55.708085+0000) 2024-03-06T18:24:56.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:56.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:56.708392+0000) 2024-03-06T18:24:57.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:57.707+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:57.708741+0000) 2024-03-06T18:24:58.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:58.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:58.709054+0000) 2024-03-06T18:24:59.704 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:24:59.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:59.709364+0000) 2024-03-06T18:25:00.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:00.708+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:00.709597+0000) 2024-03-06T18:25:01.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:01.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:01.709956+0000) 2024-03-06T18:25:02.705 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:02.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:02.710273+0000) 2024-03-06T18:25:03.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:03.709+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:03.710562+0000) 2024-03-06T18:25:04.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:04.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:04.710894+0000) 2024-03-06T18:25:05.706 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:05.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:05.711214+0000) 2024-03-06T18:25:06.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:06.710+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:06.711518+0000) 2024-03-06T18:25:07.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:07.163 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:07.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:07.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:07.711824+0000) 2024-03-06T18:25:08.707 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:08.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:08.712134+0000) 2024-03-06T18:25:09.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:09.711+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:09.712440+0000) 2024-03-06T18:25:10.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:10.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:10.712777+0000) 2024-03-06T18:25:11.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:11.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:11.713073+0000) 2024-03-06T18:25:12.708 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:12.712+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:12.713371+0000) 2024-03-06T18:25:13.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:13.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:13.713716+0000) 2024-03-06T18:25:14.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:14.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:14.714000+0000) 2024-03-06T18:25:15.709 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:15.713+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:15.714313+0000) 2024-03-06T18:25:16.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:16.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:16.714623+0000) 2024-03-06T18:25:17.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:17.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:17.714978+0000) 2024-03-06T18:25:18.710 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:18.714+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:18.715287+0000) 2024-03-06T18:25:19.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:19.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:19.715579+0000) 2024-03-06T18:25:20.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:20.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:20.715794+0000) 2024-03-06T18:25:21.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:21.715+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:21.716091+0000) 2024-03-06T18:25:22.711 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:22.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:22.716395+0000) 2024-03-06T18:25:23.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:23.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:23.716741+0000) 2024-03-06T18:25:24.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:24.716+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:24.717059+0000) 2024-03-06T18:25:25.712 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:25.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:25.717371+0000) 2024-03-06T18:25:26.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:26.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:26.717715+0000) 2024-03-06T18:25:27.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:27.717+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:27.718022+0000) 2024-03-06T18:25:28.713 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:28.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:28.718323+0000) 2024-03-06T18:25:29.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:29.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:29.718622+0000) 2024-03-06T18:25:30.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:30.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:30.719036+0000) 2024-03-06T18:25:31.714 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:31.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:31.719347+0000) 2024-03-06T18:25:32.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:32.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:32.719683+0000) 2024-03-06T18:25:33.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:33.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:33.719997+0000) 2024-03-06T18:25:34.715 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:34.718+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:34.720299+0000) 2024-03-06T18:25:35.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:35.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:35.720593+0000) 2024-03-06T18:25:36.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:36.719+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:36.720937+0000) 2024-03-06T18:25:37.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:37.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:37.716 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:37.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:37.721251+0000) 2024-03-06T18:25:38.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:38.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:38.721561+0000) 2024-03-06T18:25:39.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:39.720+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:39.721877+0000) 2024-03-06T18:25:40.717 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:40.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:40.722181+0000) 2024-03-06T18:25:41.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:41.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:41.722481+0000) 2024-03-06T18:25:42.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:42.721+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:42.722791+0000) 2024-03-06T18:25:43.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:43.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:43.723101+0000) 2024-03-06T18:25:44.718 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:44.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:44.723395+0000) 2024-03-06T18:25:45.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:45.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:45.723747+0000) 2024-03-06T18:25:46.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:46.722+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:46.723971+0000) 2024-03-06T18:25:47.719 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:47.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:47.724277+0000) 2024-03-06T18:25:48.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:48.723+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:48.724591+0000) 2024-03-06T18:25:49.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:49.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:49.724925+0000) 2024-03-06T18:25:50.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:50.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:50.725211+0000) 2024-03-06T18:25:51.720 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:51.724+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:51.725516+0000) 2024-03-06T18:25:52.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:52.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:52.725832+0000) 2024-03-06T18:25:53.721 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:53.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:53.726133+0000) 2024-03-06T18:25:54.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:54.725+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:54.726457+0000) 2024-03-06T18:25:55.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:55.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:55.726797+0000) 2024-03-06T18:25:56.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:56.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:56.727107+0000) 2024-03-06T18:25:57.722 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:57.726+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:57.727398+0000) 2024-03-06T18:25:58.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:58.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:58.727688+0000) 2024-03-06T18:25:59.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:25:59.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:59.728012+0000) 2024-03-06T18:26:00.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:00.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:00.728298+0000) 2024-03-06T18:26:01.723 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:01.727+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:01.728531+0000) 2024-03-06T18:26:02.724 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:02.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:02.728826+0000) 2024-03-06T18:26:03.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:03.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:03.729135+0000) 2024-03-06T18:26:04.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:04.728+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:04.729440+0000) 2024-03-06T18:26:05.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:05.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:05.729782+0000) 2024-03-06T18:26:06.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:06.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:06.730078+0000) 2024-03-06T18:26:07.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:07.441 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:07.725 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:07.729+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:07.730378+0000) 2024-03-06T18:26:08.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:08.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:08.730747+0000) 2024-03-06T18:26:09.726 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:09.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:09.731063+0000) 2024-03-06T18:26:10.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:10.730+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:10.731359+0000) 2024-03-06T18:26:11.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:11.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:11.731724+0000) 2024-03-06T18:26:12.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:12.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:12.732042+0000) 2024-03-06T18:26:13.727 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:13.731+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:13.732412+0000) 2024-03-06T18:26:14.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:14.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:14.732755+0000) 2024-03-06T18:26:15.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:15.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:15.733049+0000) 2024-03-06T18:26:16.728 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:16.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:16.733356+0000) 2024-03-06T18:26:17.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:17.732+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:17.733604+0000) 2024-03-06T18:26:18.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:18.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:18.733906+0000) 2024-03-06T18:26:19.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:19.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:19.734161+0000) 2024-03-06T18:26:20.729 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:20.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:20.734400+0000) 2024-03-06T18:26:21.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:21.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:21.734692+0000) 2024-03-06T18:26:22.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:22.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:22.734931+0000) 2024-03-06T18:26:23.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:23.733+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:23.735126+0000) 2024-03-06T18:26:24.730 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:24.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:24.735388+0000) 2024-03-06T18:26:25.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:25.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:25.735662+0000) 2024-03-06T18:26:26.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:26.734+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:26.735899+0000) 2024-03-06T18:26:27.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:27.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:27.736147+0000) 2024-03-06T18:26:28.731 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:28.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:28.736378+0000) 2024-03-06T18:26:29.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:29.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:29.736666+0000) 2024-03-06T18:26:30.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:30.735+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:30.736898+0000) 2024-03-06T18:26:31.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:31.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:31.737144+0000) 2024-03-06T18:26:32.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:32.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:32.737381+0000) 2024-03-06T18:26:33.732 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:33.736+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:33.737660+0000) 2024-03-06T18:26:34.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:34.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:34.737939+0000) 2024-03-06T18:26:35.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:35.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:35.738196+0000) 2024-03-06T18:26:36.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:36.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:36.738417+0000) 2024-03-06T18:26:37.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:37.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:37.733 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:37.737+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:37.738760+0000) 2024-03-06T18:26:38.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:38.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:38.739020+0000) 2024-03-06T18:26:39.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:39.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:39.739284+0000) 2024-03-06T18:26:40.734 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:40.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:40.739520+0000) 2024-03-06T18:26:41.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:41.738+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:41.739759+0000) 2024-03-06T18:26:42.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:42.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:42.740013+0000) 2024-03-06T18:26:43.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:43.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:43.740263+0000) 2024-03-06T18:26:44.735 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:44.739+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:44.740514+0000) 2024-03-06T18:26:45.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:45.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:45.740746+0000) 2024-03-06T18:26:46.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:46.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:46.740980+0000) 2024-03-06T18:26:47.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:47.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:47.741252+0000) 2024-03-06T18:26:48.736 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:48.740+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:48.741495+0000) 2024-03-06T18:26:49.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:49.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:49.741755+0000) 2024-03-06T18:26:50.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:50.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:50.742015+0000) 2024-03-06T18:26:51.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:51.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:51.742271+0000) 2024-03-06T18:26:52.737 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:52.741+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:52.742528+0000) 2024-03-06T18:26:53.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:53.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:53.742789+0000) 2024-03-06T18:26:54.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:54.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:54.743041+0000) 2024-03-06T18:26:55.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:55.742+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:55.743281+0000) 2024-03-06T18:26:56.738 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:56.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:56.743529+0000) 2024-03-06T18:26:57.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:57.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:57.743787+0000) 2024-03-06T18:26:58.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:58.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:58.744027+0000) 2024-03-06T18:26:59.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:26:59.743+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:59.744265+0000) 2024-03-06T18:27:00.739 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:00.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:00.744508+0000) 2024-03-06T18:27:01.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:01.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:01.744769+0000) 2024-03-06T18:27:02.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:02.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:02.745023+0000) 2024-03-06T18:27:03.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:03.744+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:03.745219+0000) 2024-03-06T18:27:04.740 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:04.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:04.745477+0000) 2024-03-06T18:27:05.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:05.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:05.745739+0000) 2024-03-06T18:27:06.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:06.745+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:06.746047+0000) 2024-03-06T18:27:07.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:07.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:07.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:07.746315+0000) 2024-03-06T18:27:07.784 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:08.741 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:08.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:08.746557+0000) 2024-03-06T18:27:09.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:09.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:09.746803+0000) 2024-03-06T18:27:10.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:10.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:10.747043+0000) 2024-03-06T18:27:11.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:11.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:11.747287+0000) 2024-03-06T18:27:12.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:12.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:12.747537+0000) 2024-03-06T18:27:13.742 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:13.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:13.747786+0000) 2024-03-06T18:27:14.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:14.746+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:14.748030+0000) 2024-03-06T18:27:15.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:15.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:15.748285+0000) 2024-03-06T18:27:16.743 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:16.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:16.748518+0000) 2024-03-06T18:27:17.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:17.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:17.748788+0000) 2024-03-06T18:27:18.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:18.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:18.749016+0000) 2024-03-06T18:27:19.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:19.747+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:19.749210+0000) 2024-03-06T18:27:20.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:20.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:20.749471+0000) 2024-03-06T18:27:21.744 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:21.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:21.749736+0000) 2024-03-06T18:27:22.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:22.748+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:22.749984+0000) 2024-03-06T18:27:23.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:23.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:23.750242+0000) 2024-03-06T18:27:24.745 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:24.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:24.750500+0000) 2024-03-06T18:27:25.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:25.749+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:25.750741+0000) 2024-03-06T18:27:26.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:26.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:26.750984+0000) 2024-03-06T18:27:27.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:27.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:27.751257+0000) 2024-03-06T18:27:28.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:28.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:28.751490+0000) 2024-03-06T18:27:29.746 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:29.750+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:29.751750+0000) 2024-03-06T18:27:30.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:30.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:30.752006+0000) 2024-03-06T18:27:31.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:31.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:31.752263+0000) 2024-03-06T18:27:32.747 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:32.751+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:32.752539+0000) 2024-03-06T18:27:33.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:33.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:33.752799+0000) 2024-03-06T18:27:34.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:34.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:34.753042+0000) 2024-03-06T18:27:35.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:35.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:35.753281+0000) 2024-03-06T18:27:36.748 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:36.752+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:36.753530+0000) 2024-03-06T18:27:37.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:37.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:37.753800+0000) 2024-03-06T18:27:37.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:37.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:38.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:38.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:38.754059+0000) 2024-03-06T18:27:39.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:39.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:39.754312+0000) 2024-03-06T18:27:40.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:40.753+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:40.754554+0000) 2024-03-06T18:27:41.749 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:41.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:41.754809+0000) 2024-03-06T18:27:42.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:42.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:42.755051+0000) 2024-03-06T18:27:43.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:43.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:43.755295+0000) 2024-03-06T18:27:44.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:44.754+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:44.755541+0000) 2024-03-06T18:27:45.750 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:45.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:45.755798+0000) 2024-03-06T18:27:46.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:46.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:46.755986+0000) 2024-03-06T18:27:47.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:47.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:47.756261+0000) 2024-03-06T18:27:48.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:48.755+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:48.756445+0000) 2024-03-06T18:27:49.751 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:49.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:49.756722+0000) 2024-03-06T18:27:50.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:50.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:50.756986+0000) 2024-03-06T18:27:51.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:51.756+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:51.757234+0000) 2024-03-06T18:27:52.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:52.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:52.757488+0000) 2024-03-06T18:27:53.752 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:53.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:53.757738+0000) 2024-03-06T18:27:54.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:54.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:54.757987+0000) 2024-03-06T18:27:55.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:55.757+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:55.758224+0000) 2024-03-06T18:27:56.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:56.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:56.758475+0000) 2024-03-06T18:27:57.753 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:57.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:57.758793+0000) 2024-03-06T18:27:58.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:58.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:58.759047+0000) 2024-03-06T18:27:59.754 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:27:59.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:59.759361+0000) 2024-03-06T18:28:00.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:00.758+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:00.759601+0000) 2024-03-06T18:28:01.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:01.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:01.759878+0000) 2024-03-06T18:28:02.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:02.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:02.760120+0000) 2024-03-06T18:28:03.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:03.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:03.760379+0000) 2024-03-06T18:28:04.755 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:04.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:04.760625+0000) 2024-03-06T18:28:05.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:05.759+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:05.760901+0000) 2024-03-06T18:28:06.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:06.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:06.761157+0000) 2024-03-06T18:28:07.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:07.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:07.761415+0000) 2024-03-06T18:28:08.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:08.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:08.756 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:08.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:08.761697+0000) 2024-03-06T18:28:09.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:09.760+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:09.761958+0000) 2024-03-06T18:28:10.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:10.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:10.762213+0000) 2024-03-06T18:28:11.757 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:11.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:11.762467+0000) 2024-03-06T18:28:12.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:12.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:12.762767+0000) 2024-03-06T18:28:13.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:13.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:13.763026+0000) 2024-03-06T18:28:14.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:14.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:14.763228+0000) 2024-03-06T18:28:15.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:15.761+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:15.763487+0000) 2024-03-06T18:28:16.758 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:16.762+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:16.763711+0000) 2024-03-06T18:28:17.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:17.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:17.763979+0000) 2024-03-06T18:28:18.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:18.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:18.764233+0000) 2024-03-06T18:28:19.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:19.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:19.764487+0000) 2024-03-06T18:28:20.759 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:20.763+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:20.764758+0000) 2024-03-06T18:28:21.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:21.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:21.765002+0000) 2024-03-06T18:28:22.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:22.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:22.765252+0000) 2024-03-06T18:28:23.760 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:23.764+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:23.765492+0000) 2024-03-06T18:28:24.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:24.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:24.765739+0000) 2024-03-06T18:28:25.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:25.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:25.765987+0000) 2024-03-06T18:28:26.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:26.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:26.766234+0000) 2024-03-06T18:28:27.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:27.765+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:27.766504+0000) 2024-03-06T18:28:28.761 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:28.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:28.766761+0000) 2024-03-06T18:28:29.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:29.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:29.767018+0000) 2024-03-06T18:28:30.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:30.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:30.767271+0000) 2024-03-06T18:28:31.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:31.766+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:31.767532+0000) 2024-03-06T18:28:32.762 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:32.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:32.767801+0000) 2024-03-06T18:28:33.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:33.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:33.768070+0000) 2024-03-06T18:28:34.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:34.767+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:34.768331+0000) 2024-03-06T18:28:35.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:35.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:35.768570+0000) 2024-03-06T18:28:36.763 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:36.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:36.768814+0000) 2024-03-06T18:28:37.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:37.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:37.769070+0000) 2024-03-06T18:28:38.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:38.225 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:38.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:38.768+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:38.769323+0000) 2024-03-06T18:28:39.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:39.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:39.769579+0000) 2024-03-06T18:28:40.764 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:40.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:40.769817+0000) 2024-03-06T18:28:41.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:41.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:41.770070+0000) 2024-03-06T18:28:42.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:42.769+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:42.770273+0000) 2024-03-06T18:28:43.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:43.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:43.770539+0000) 2024-03-06T18:28:44.765 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:44.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:44.770769+0000) 2024-03-06T18:28:45.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:45.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:45.771025+0000) 2024-03-06T18:28:46.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:46.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:46.771275+0000) 2024-03-06T18:28:47.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:47.770+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:47.771583+0000) 2024-03-06T18:28:48.766 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:48.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:48.771845+0000) 2024-03-06T18:28:49.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:49.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:49.772103+0000) 2024-03-06T18:28:50.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:50.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:50.772358+0000) 2024-03-06T18:28:51.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:51.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:51.772595+0000) 2024-03-06T18:28:52.767 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:52.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:52.772854+0000) 2024-03-06T18:28:53.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:53.771+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:53.773095+0000) 2024-03-06T18:28:54.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:54.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:54.773357+0000) 2024-03-06T18:28:55.768 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:55.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:55.773605+0000) 2024-03-06T18:28:56.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:56.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:56.773886+0000) 2024-03-06T18:28:57.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:57.772+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:57.774127+0000) 2024-03-06T18:28:58.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:58.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:58.774387+0000) 2024-03-06T18:28:59.769 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:28:59.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:59.774668+0000) 2024-03-06T18:29:00.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:00.773+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:00.774922+0000) 2024-03-06T18:29:01.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:01.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:01.775180+0000) 2024-03-06T18:29:02.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:02.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:02.775421+0000) 2024-03-06T18:29:03.770 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:03.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:03.775726+0000) 2024-03-06T18:29:04.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:04.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:04.775970+0000) 2024-03-06T18:29:05.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:05.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:05.776228+0000) 2024-03-06T18:29:06.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:06.774+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:06.776471+0000) 2024-03-06T18:29:07.771 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:07.775+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:07.776764+0000) 2024-03-06T18:29:08.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:08.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:08.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:08.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:08.777026+0000) 2024-03-06T18:29:09.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:09.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:09.777282+0000) 2024-03-06T18:29:10.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:10.776+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:10.777517+0000) 2024-03-06T18:29:11.772 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:11.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:11.777767+0000) 2024-03-06T18:29:12.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:12.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:12.778021+0000) 2024-03-06T18:29:13.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:13.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:13.778263+0000) 2024-03-06T18:29:14.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:14.777+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:14.778532+0000) 2024-03-06T18:29:15.773 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:15.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:15.778783+0000) 2024-03-06T18:29:16.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:16.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:16.779034+0000) 2024-03-06T18:29:17.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:17.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:17.779288+0000) 2024-03-06T18:29:18.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:18.778+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:18.779541+0000) 2024-03-06T18:29:19.774 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:19.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:19.779799+0000) 2024-03-06T18:29:20.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:20.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:20.779983+0000) 2024-03-06T18:29:21.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:21.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:21.780221+0000) 2024-03-06T18:29:22.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:22.779+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:22.780462+0000) 2024-03-06T18:29:23.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:23.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:23.780756+0000) 2024-03-06T18:29:24.775 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:24.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:24.781009+0000) 2024-03-06T18:29:25.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:25.780+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:25.781278+0000) 2024-03-06T18:29:26.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:26.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:26.781547+0000) 2024-03-06T18:29:27.776 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:27.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:27.781809+0000) 2024-03-06T18:29:28.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:28.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:28.782062+0000) 2024-03-06T18:29:29.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:29.781+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:29.782304+0000) 2024-03-06T18:29:30.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:30.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:30.782565+0000) 2024-03-06T18:29:31.777 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:31.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:31.782819+0000) 2024-03-06T18:29:32.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:32.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:32.783079+0000) 2024-03-06T18:29:33.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:33.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:33.783320+0000) 2024-03-06T18:29:34.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:34.782+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:34.783569+0000) 2024-03-06T18:29:35.778 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:35.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:35.783860+0000) 2024-03-06T18:29:36.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:36.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:36.784123+0000) 2024-03-06T18:29:37.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:37.783+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:37.784495+0000) 2024-03-06T18:29:38.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:38.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:38.779 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:38.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:38.784808+0000) 2024-03-06T18:29:39.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:39.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:39.785068+0000) 2024-03-06T18:29:40.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:40.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:40.785252+0000) 2024-03-06T18:29:41.780 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:41.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:41.785506+0000) 2024-03-06T18:29:42.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:42.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:42.785739+0000) 2024-03-06T18:29:43.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:43.784+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:43.785997+0000) 2024-03-06T18:29:44.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:44.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:44.786260+0000) 2024-03-06T18:29:45.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:45.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:45.786517+0000) 2024-03-06T18:29:46.781 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:46.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:46.786751+0000) 2024-03-06T18:29:47.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:47.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:47.787007+0000) 2024-03-06T18:29:48.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:48.785+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:48.787241+0000) 2024-03-06T18:29:49.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:49.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:49.787483+0000) 2024-03-06T18:29:50.782 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:50.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:50.787736+0000) 2024-03-06T18:29:51.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:51.786+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:51.787968+0000) 2024-03-06T18:29:52.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:52.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:52.788208+0000) 2024-03-06T18:29:53.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:53.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:53.788456+0000) 2024-03-06T18:29:54.783 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:54.787+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:54.788761+0000) 2024-03-06T18:29:55.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:55.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:55.789014+0000) 2024-03-06T18:29:56.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:56.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:56.789266+0000) 2024-03-06T18:29:57.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:57.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:57.789518+0000) 2024-03-06T18:29:58.784 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:58.788+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:58.789790+0000) 2024-03-06T18:29:59.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:29:59.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:59.790063+0000) 2024-03-06T18:30:00.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:00.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:00.790329+0000) 2024-03-06T18:30:01.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:01.789+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:01.790602+0000) 2024-03-06T18:30:02.785 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:02.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:02.790905+0000) 2024-03-06T18:30:03.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:03.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:03.791183+0000) 2024-03-06T18:30:04.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:04.790+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:04.791510+0000) 2024-03-06T18:30:05.786 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:05.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:05.791739+0000) 2024-03-06T18:30:06.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:06.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:06.791992+0000) 2024-03-06T18:30:07.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:07.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:07.792250+0000) 2024-03-06T18:30:08.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:08.675 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:08.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:08.791+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:08.792457+0000) 2024-03-06T18:30:09.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:09.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:09.792737+0000) 2024-03-06T18:30:10.787 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:10.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:10.792991+0000) 2024-03-06T18:30:11.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:11.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:11.793249+0000) 2024-03-06T18:30:12.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:12.792+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:12.793503+0000) 2024-03-06T18:30:13.788 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:13.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:13.793770+0000) 2024-03-06T18:30:14.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:14.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:14.794012+0000) 2024-03-06T18:30:15.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:15.793+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:15.794265+0000) 2024-03-06T18:30:16.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:16.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:16.794535+0000) 2024-03-06T18:30:17.789 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:17.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:17.794771+0000) 2024-03-06T18:30:18.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:18.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:18.795001+0000) 2024-03-06T18:30:19.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:19.794+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:19.795237+0000) 2024-03-06T18:30:20.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:20.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:20.795474+0000) 2024-03-06T18:30:21.790 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:21.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:21.795758+0000) 2024-03-06T18:30:22.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:22.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:22.796017+0000) 2024-03-06T18:30:23.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:23.795+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:23.796221+0000) 2024-03-06T18:30:24.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:24.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:24.796509+0000) 2024-03-06T18:30:25.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:25.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:25.796747+0000) 2024-03-06T18:30:26.791 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:26.796+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:26.796998+0000) 2024-03-06T18:30:27.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:27.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:27.797233+0000) 2024-03-06T18:30:28.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:28.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:28.797484+0000) 2024-03-06T18:30:29.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:29.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:29.797747+0000) 2024-03-06T18:30:30.792 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:30.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:30.797995+0000) 2024-03-06T18:30:31.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:31.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:31.798250+0000) 2024-03-06T18:30:32.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:32.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:32.798491+0000) 2024-03-06T18:30:33.793 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:33.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:33.798801+0000) 2024-03-06T18:30:34.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:34.797+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:34.799128+0000) 2024-03-06T18:30:35.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:35.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:35.799441+0000) 2024-03-06T18:30:36.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:36.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:36.799805+0000) 2024-03-06T18:30:37.794 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:37.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:37.800052+0000) 2024-03-06T18:30:38.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:38.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:38.800313+0000) 2024-03-06T18:30:38.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:38.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:39.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:39.798+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:39.800596+0000) 2024-03-06T18:30:40.795 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:40.799+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:40.800964+0000) 2024-03-06T18:30:41.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:41.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:41.801277+0000) 2024-03-06T18:30:42.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:42.800+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:42.801597+0000) 2024-03-06T18:30:43.796 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:43.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:43.801954+0000) 2024-03-06T18:30:44.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:44.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:44.802280+0000) 2024-03-06T18:30:45.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:45.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:45.802581+0000) 2024-03-06T18:30:46.797 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:46.801+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:46.802877+0000) 2024-03-06T18:30:47.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:47.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:47.803171+0000) 2024-03-06T18:30:48.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:48.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:48.803427+0000) 2024-03-06T18:30:49.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:49.802+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:49.803744+0000) 2024-03-06T18:30:50.798 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:50.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:50.803984+0000) 2024-03-06T18:30:51.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:51.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:51.804219+0000) 2024-03-06T18:30:52.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:52.803+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:52.804458+0000) 2024-03-06T18:30:53.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:53.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:53.804744+0000) 2024-03-06T18:30:54.799 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:54.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:54.804995+0000) 2024-03-06T18:30:55.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:55.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:55.805255+0000) 2024-03-06T18:30:56.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:56.804+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:56.805528+0000) 2024-03-06T18:30:57.800 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:57.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:57.805849+0000) 2024-03-06T18:30:58.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:58.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:58.806133+0000) 2024-03-06T18:30:59.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:30:59.805+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:59.806449+0000) 2024-03-06T18:31:00.801 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:00.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:00.806775+0000) 2024-03-06T18:31:01.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:01.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:01.807094+0000) 2024-03-06T18:31:02.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:02.806+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:02.807401+0000) 2024-03-06T18:31:03.802 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:03.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:03.807735+0000) 2024-03-06T18:31:04.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:04.807+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:04.808256+0000) 2024-03-06T18:31:05.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:05.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:05.808538+0000) 2024-03-06T18:31:06.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:06.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:06.808812+0000) 2024-03-06T18:31:07.803 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:07.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:07.809083+0000) 2024-03-06T18:31:08.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:08.808+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:08.809360+0000) 2024-03-06T18:31:08.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:08.965 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:09.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:09.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:09.809659+0000) 2024-03-06T18:31:10.804 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:10.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:10.809919+0000) 2024-03-06T18:31:11.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:11.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:11.810162+0000) 2024-03-06T18:31:12.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:12.809+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:12.810399+0000) 2024-03-06T18:31:13.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:13.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:13.810688+0000) 2024-03-06T18:31:14.805 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:14.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:14.810943+0000) 2024-03-06T18:31:15.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:15.810+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:15.811196+0000) 2024-03-06T18:31:16.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:16.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:16.811430+0000) 2024-03-06T18:31:17.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:17.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:17.811771+0000) 2024-03-06T18:31:18.806 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:18.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:18.811960+0000) 2024-03-06T18:31:19.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:19.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:19.812158+0000) 2024-03-06T18:31:20.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:20.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:20.812404+0000) 2024-03-06T18:31:21.807 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:21.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:21.812731+0000) 2024-03-06T18:31:22.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:22.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:22.812980+0000) 2024-03-06T18:31:23.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:23.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:23.813182+0000) 2024-03-06T18:31:24.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:24.811+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:24.813376+0000) 2024-03-06T18:31:25.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:25.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:25.813623+0000) 2024-03-06T18:31:26.808 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:26.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:26.813919+0000) 2024-03-06T18:31:27.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:27.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:27.814180+0000) 2024-03-06T18:31:28.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:28.812+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:28.814379+0000) 2024-03-06T18:31:29.809 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:29.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:29.814623+0000) 2024-03-06T18:31:30.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:30.813+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:30.814899+0000) 2024-03-06T18:31:31.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:31.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:31.815160+0000) 2024-03-06T18:31:32.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:32.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:32.815398+0000) 2024-03-06T18:31:33.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:33.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:33.815711+0000) 2024-03-06T18:31:34.810 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:34.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:34.815966+0000) 2024-03-06T18:31:35.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:35.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:35.816227+0000) 2024-03-06T18:31:36.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:36.814+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:36.816422+0000) 2024-03-06T18:31:37.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:37.815+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:37.816723+0000) 2024-03-06T18:31:38.811 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:38.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:38.816975+0000) 2024-03-06T18:31:39.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:39.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:39.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:39.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:39.817239+0000) 2024-03-06T18:31:40.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:40.816+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:40.817476+0000) 2024-03-06T18:31:41.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:41.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:41.817749+0000) 2024-03-06T18:31:42.812 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:42.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:42.817984+0000) 2024-03-06T18:31:43.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:43.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:43.818252+0000) 2024-03-06T18:31:44.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:44.817+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:44.818444+0000) 2024-03-06T18:31:45.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:45.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:45.818744+0000) 2024-03-06T18:31:46.813 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:46.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:46.818922+0000) 2024-03-06T18:31:47.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:47.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:47.819159+0000) 2024-03-06T18:31:48.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:48.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:48.819430+0000) 2024-03-06T18:31:49.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:49.818+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:49.819616+0000) 2024-03-06T18:31:50.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:50.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:50.819890+0000) 2024-03-06T18:31:51.814 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:51.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:51.820135+0000) 2024-03-06T18:31:52.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:52.819+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:52.820373+0000) 2024-03-06T18:31:53.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:53.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:53.820665+0000) 2024-03-06T18:31:54.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:54.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:54.820929+0000) 2024-03-06T18:31:55.815 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:55.820+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:55.821182+0000) 2024-03-06T18:31:56.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:56.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:56.821432+0000) 2024-03-06T18:31:57.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:57.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:57.821723+0000) 2024-03-06T18:31:58.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:58.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:58.821923+0000) 2024-03-06T18:31:59.816 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:31:59.821+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:59.822197+0000) 2024-03-06T18:32:00.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:00.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:00.822435+0000) 2024-03-06T18:32:01.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:01.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:01.822734+0000) 2024-03-06T18:32:02.817 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:02.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:02.822992+0000) 2024-03-06T18:32:03.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:03.822+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:03.823180+0000) 2024-03-06T18:32:04.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:04.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:04.823435+0000) 2024-03-06T18:32:05.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:05.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:05.823733+0000) 2024-03-06T18:32:06.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:06.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:06.823947+0000) 2024-03-06T18:32:07.818 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:07.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:07.824185+0000) 2024-03-06T18:32:08.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:08.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:08.824437+0000) 2024-03-06T18:32:09.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:09.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:09.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:09.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:09.824725+0000) 2024-03-06T18:32:10.819 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:10.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:10.825043+0000) 2024-03-06T18:32:11.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:11.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:11.825281+0000) 2024-03-06T18:32:12.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:12.823+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:12.825528+0000) 2024-03-06T18:32:13.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:13.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:13.825787+0000) 2024-03-06T18:32:14.820 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:14.824+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:14.825996+0000) 2024-03-06T18:32:15.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:15.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:15.826287+0000) 2024-03-06T18:32:16.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:16.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:16.826546+0000) 2024-03-06T18:32:17.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:17.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:17.826806+0000) 2024-03-06T18:32:18.821 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:18.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:18.827065+0000) 2024-03-06T18:32:19.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:19.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:19.827322+0000) 2024-03-06T18:32:20.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:20.825+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:20.827546+0000) 2024-03-06T18:32:21.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:21.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:21.827786+0000) 2024-03-06T18:32:22.822 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:22.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:22.828019+0000) 2024-03-06T18:32:23.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:23.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:23.828273+0000) 2024-03-06T18:32:24.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:24.826+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:24.828524+0000) 2024-03-06T18:32:25.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:25.827+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:25.828783+0000) 2024-03-06T18:32:26.823 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:26.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:26.829036+0000) 2024-03-06T18:32:27.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:27.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:27.829283+0000) 2024-03-06T18:32:28.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:28.828+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:28.829539+0000) 2024-03-06T18:32:29.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:29.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:29.829805+0000) 2024-03-06T18:32:30.824 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:30.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:30.830039+0000) 2024-03-06T18:32:31.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:31.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:31.830297+0000) 2024-03-06T18:32:32.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:32.829+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:32.830558+0000) 2024-03-06T18:32:33.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:33.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:33.830831+0000) 2024-03-06T18:32:34.825 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:34.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:34.831093+0000) 2024-03-06T18:32:35.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:35.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:35.831330+0000) 2024-03-06T18:32:36.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:36.830+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:36.831567+0000) 2024-03-06T18:32:37.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:37.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:37.831824+0000) 2024-03-06T18:32:38.826 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:38.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:38.832090+0000) 2024-03-06T18:32:39.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:39.401 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:39.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:39.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:39.832342+0000) 2024-03-06T18:32:40.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:40.831+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:40.832577+0000) 2024-03-06T18:32:41.827 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:41.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:41.832836+0000) 2024-03-06T18:32:42.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:42.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:42.833079+0000) 2024-03-06T18:32:43.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:43.832+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:43.833354+0000) 2024-03-06T18:32:44.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:44.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:44.833618+0000) 2024-03-06T18:32:45.828 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:45.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:45.833907+0000) 2024-03-06T18:32:46.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:46.833+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:46.834163+0000) 2024-03-06T18:32:47.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:47.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:47.834410+0000) 2024-03-06T18:32:48.829 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:48.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:48.834670+0000) 2024-03-06T18:32:49.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:49.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:49.834936+0000) 2024-03-06T18:32:50.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:50.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:50.835208+0000) 2024-03-06T18:32:51.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:51.834+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:51.835436+0000) 2024-03-06T18:32:52.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:52.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:52.835733+0000) 2024-03-06T18:32:53.830 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:53.835+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:53.835994+0000) 2024-03-06T18:32:54.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:54.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:54.836297+0000) 2024-03-06T18:32:55.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:55.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:55.836552+0000) 2024-03-06T18:32:56.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:56.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:56.836788+0000) 2024-03-06T18:32:57.831 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:57.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:57.837060+0000) 2024-03-06T18:32:58.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:58.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:58.837401+0000) 2024-03-06T18:32:59.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:32:59.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:59.837715+0000) 2024-03-06T18:33:00.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:00.836+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:00.837920+0000) 2024-03-06T18:33:01.832 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:01.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:01.838162+0000) 2024-03-06T18:33:02.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:02.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:02.838410+0000) 2024-03-06T18:33:03.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:03.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:03.838726+0000) 2024-03-06T18:33:04.833 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:04.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:04.838983+0000) 2024-03-06T18:33:05.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:05.837+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:05.839253+0000) 2024-03-06T18:33:06.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:06.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:06.839540+0000) 2024-03-06T18:33:07.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:07.838+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:07.839793+0000) 2024-03-06T18:33:08.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:08.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:08.840042+0000) 2024-03-06T18:33:09.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:09.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:09.834 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:09.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:09.840300+0000) 2024-03-06T18:33:10.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:10.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:10.840550+0000) 2024-03-06T18:33:11.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:11.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:11.840809+0000) 2024-03-06T18:33:12.835 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:12.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:12.841064+0000) 2024-03-06T18:33:13.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:13.839+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:13.841308+0000) 2024-03-06T18:33:14.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:14.840+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:14.841599+0000) 2024-03-06T18:33:15.836 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:15.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:15.841885+0000) 2024-03-06T18:33:16.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:16.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:16.842137+0000) 2024-03-06T18:33:17.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:17.841+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:17.842396+0000) 2024-03-06T18:33:18.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:18.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:18.842694+0000) 2024-03-06T18:33:19.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:19.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:19.842966+0000) 2024-03-06T18:33:20.837 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:20.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:20.843160+0000) 2024-03-06T18:33:21.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:21.842+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:21.843419+0000) 2024-03-06T18:33:22.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:22.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:22.843723+0000) 2024-03-06T18:33:23.838 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:23.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:23.844008+0000) 2024-03-06T18:33:24.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:24.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:24.844272+0000) 2024-03-06T18:33:25.840 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:25.843+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:25.844531+0000) 2024-03-06T18:33:26.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:26.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:26.844798+0000) 2024-03-06T18:33:27.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:27.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:27.845072+0000) 2024-03-06T18:33:28.839 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:28.844+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:28.845293+0000) 2024-03-06T18:33:29.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:29.845+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:29.846234+0000) 2024-03-06T18:33:30.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:30.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:30.846687+0000) 2024-03-06T18:33:31.841 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:31.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:31.847000+0000) 2024-03-06T18:33:32.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:32.846+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:32.847310+0000) 2024-03-06T18:33:33.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:33.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:33.847585+0000) 2024-03-06T18:33:34.842 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:34.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:34.847938+0000) 2024-03-06T18:33:35.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:35.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:35.848247+0000) 2024-03-06T18:33:36.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:36.847+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:36.848492+0000) 2024-03-06T18:33:37.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:37.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:37.848796+0000) 2024-03-06T18:33:38.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:38.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:38.849064+0000) 2024-03-06T18:33:39.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:39.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:39.843 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:39.848+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:39.849280+0000) 2024-03-06T18:33:40.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:40.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:40.849537+0000) 2024-03-06T18:33:41.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:41.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:41.849796+0000) 2024-03-06T18:33:42.844 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:42.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:42.850023+0000) 2024-03-06T18:33:43.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:43.849+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:43.850263+0000) 2024-03-06T18:33:44.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:44.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:44.850533+0000) 2024-03-06T18:33:45.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:45.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:45.850755+0000) 2024-03-06T18:33:46.845 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:46.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:46.851201+0000) 2024-03-06T18:33:47.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:47.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:47.851444+0000) 2024-03-06T18:33:48.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:48.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:48.851739+0000) 2024-03-06T18:33:49.846 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:49.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:49.851999+0000) 2024-03-06T18:33:50.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:50.850+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:50.852472+0000) 2024-03-06T18:33:51.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:51.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:51.852754+0000) 2024-03-06T18:33:52.847 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:52.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:52.853012+0000) 2024-03-06T18:33:53.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:53.851+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:53.853210+0000) 2024-03-06T18:33:54.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:54.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:54.853470+0000) 2024-03-06T18:33:55.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:55.852+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:55.853768+0000) 2024-03-06T18:33:56.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:56.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:56.854023+0000) 2024-03-06T18:33:57.848 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:57.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:57.854270+0000) 2024-03-06T18:33:58.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:58.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:58.854525+0000) 2024-03-06T18:33:59.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:33:59.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:59.854786+0000) 2024-03-06T18:34:00.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:00.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:00.854978+0000) 2024-03-06T18:34:01.849 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:01.853+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:01.855166+0000) 2024-03-06T18:34:02.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:02.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:02.855401+0000) 2024-03-06T18:34:03.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:03.854+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:03.855724+0000) 2024-03-06T18:34:04.850 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:04.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:04.856037+0000) 2024-03-06T18:34:05.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:05.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:05.856312+0000) 2024-03-06T18:34:06.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:06.855+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:06.856577+0000) 2024-03-06T18:34:07.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:07.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:07.856862+0000) 2024-03-06T18:34:08.851 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:08.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:08.857137+0000) 2024-03-06T18:34:09.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:09.833 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:09.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:09.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:09.857360+0000) 2024-03-06T18:34:10.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:10.856+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:10.857613+0000) 2024-03-06T18:34:11.852 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:11.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:11.857933+0000) 2024-03-06T18:34:12.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:12.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:12.858189+0000) 2024-03-06T18:34:13.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:13.857+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:13.858451+0000) 2024-03-06T18:34:14.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:14.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:14.858747+0000) 2024-03-06T18:34:15.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:15.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:15.859006+0000) 2024-03-06T18:34:16.853 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:16.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:16.859258+0000) 2024-03-06T18:34:17.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:17.858+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:17.859456+0000) 2024-03-06T18:34:18.854 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:18.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:18.859749+0000) 2024-03-06T18:34:19.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:19.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:19.859992+0000) 2024-03-06T18:34:20.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:20.859+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:20.860227+0000) 2024-03-06T18:34:21.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:21.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:21.860471+0000) 2024-03-06T18:34:22.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:22.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:22.860748+0000) 2024-03-06T18:34:23.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:23.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:23.861020+0000) 2024-03-06T18:34:24.855 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:24.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:24.861215+0000) 2024-03-06T18:34:25.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:25.860+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:25.861406+0000) 2024-03-06T18:34:26.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:26.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:26.861700+0000) 2024-03-06T18:34:27.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:27.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:27.861962+0000) 2024-03-06T18:34:28.856 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:28.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:28.862243+0000) 2024-03-06T18:34:29.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:29.861+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:29.862506+0000) 2024-03-06T18:34:30.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:30.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:30.862760+0000) 2024-03-06T18:34:31.857 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:31.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:31.863020+0000) 2024-03-06T18:34:32.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:32.862+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:32.863292+0000) 2024-03-06T18:34:33.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:33.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:33.863544+0000) 2024-03-06T18:34:34.858 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:34.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:34.863791+0000) 2024-03-06T18:34:35.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:35.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:35.863985+0000) 2024-03-06T18:34:36.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:36.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:36.864239+0000) 2024-03-06T18:34:37.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:37.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:37.864466+0000) 2024-03-06T18:34:38.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:38.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:38.864743+0000) 2024-03-06T18:34:39.859 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:39.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:39.865039+0000) 2024-03-06T18:34:39.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:39.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:40.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:40.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:40.865285+0000) 2024-03-06T18:34:41.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:41.863+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:41.865473+0000) 2024-03-06T18:34:42.860 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:42.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:42.865755+0000) 2024-03-06T18:34:43.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:43.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:43.866008+0000) 2024-03-06T18:34:44.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:44.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:44.866274+0000) 2024-03-06T18:34:45.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:45.864+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:45.866529+0000) 2024-03-06T18:34:46.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:46.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:46.866774+0000) 2024-03-06T18:34:47.861 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:47.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:47.867030+0000) 2024-03-06T18:34:48.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:48.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:48.867269+0000) 2024-03-06T18:34:49.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:49.865+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:49.867544+0000) 2024-03-06T18:34:50.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:50.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:50.867783+0000) 2024-03-06T18:34:51.862 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:51.866+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:51.867983+0000) 2024-03-06T18:34:52.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:52.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:52.868234+0000) 2024-03-06T18:34:53.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:53.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:53.868489+0000) 2024-03-06T18:34:54.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:54.867+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:54.868735+0000) 2024-03-06T18:34:55.863 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:55.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:55.868979+0000) 2024-03-06T18:34:56.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:56.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:56.869233+0000) 2024-03-06T18:34:57.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:57.868+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:57.869485+0000) 2024-03-06T18:34:58.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:58.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:58.869739+0000) 2024-03-06T18:34:59.864 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:34:59.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:59.869981+0000) 2024-03-06T18:35:00.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:00.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:00.870222+0000) 2024-03-06T18:35:01.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:01.869+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:01.870468+0000) 2024-03-06T18:35:02.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:02.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:02.870747+0000) 2024-03-06T18:35:03.865 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:03.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:03.871003+0000) 2024-03-06T18:35:04.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:04.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:04.871267+0000) 2024-03-06T18:35:05.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:05.870+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:05.871531+0000) 2024-03-06T18:35:06.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:06.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:06.871800+0000) 2024-03-06T18:35:07.866 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:07.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:07.872071+0000) 2024-03-06T18:35:08.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:08.871+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:08.872276+0000) 2024-03-06T18:35:09.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:09.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:09.872537+0000) 2024-03-06T18:35:10.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:10.144 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:10.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:10.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:10.872806+0000) 2024-03-06T18:35:11.867 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:11.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:11.873078+0000) 2024-03-06T18:35:12.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:12.872+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:12.873287+0000) 2024-03-06T18:35:13.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:13.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:13.873569+0000) 2024-03-06T18:35:14.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:14.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:14.873850+0000) 2024-03-06T18:35:15.868 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:15.873+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:15.874114+0000) 2024-03-06T18:35:16.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:16.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:16.874380+0000) 2024-03-06T18:35:17.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:17.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:17.874671+0000) 2024-03-06T18:35:18.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:18.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:18.874917+0000) 2024-03-06T18:35:19.869 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:19.874+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:19.875131+0000) 2024-03-06T18:35:20.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:20.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:20.875402+0000) 2024-03-06T18:35:21.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:21.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:21.875715+0000) 2024-03-06T18:35:22.870 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:22.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:22.875964+0000) 2024-03-06T18:35:23.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:23.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:23.876247+0000) 2024-03-06T18:35:24.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:24.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:24.876512+0000) 2024-03-06T18:35:25.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:25.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:25.876778+0000) 2024-03-06T18:35:26.871 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:26.875+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:26.877046+0000) 2024-03-06T18:35:27.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:27.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:27.877362+0000) 2024-03-06T18:35:28.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:28.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:28.877613+0000) 2024-03-06T18:35:29.872 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:29.876+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:29.877941+0000) 2024-03-06T18:35:30.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:30.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:30.878204+0000) 2024-03-06T18:35:31.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:31.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:31.878476+0000) 2024-03-06T18:35:32.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:32.877+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:32.878731+0000) 2024-03-06T18:35:33.873 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:33.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:33.879009+0000) 2024-03-06T18:35:34.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:34.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:34.879267+0000) 2024-03-06T18:35:35.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:35.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:35.879487+0000) 2024-03-06T18:35:36.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:36.878+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:36.879760+0000) 2024-03-06T18:35:37.874 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:37.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:37.880038+0000) 2024-03-06T18:35:38.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:38.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:38.880316+0000) 2024-03-06T18:35:39.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:39.879+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:39.880577+0000) 2024-03-06T18:35:40.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:40.291 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:40.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:40.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:40.880852+0000) 2024-03-06T18:35:41.875 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:41.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:41.881107+0000) 2024-03-06T18:35:42.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:42.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:42.881352+0000) 2024-03-06T18:35:43.876 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:43.880+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:43.881677+0000) 2024-03-06T18:35:44.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:44.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:44.881978+0000) 2024-03-06T18:35:45.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:45.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:45.882249+0000) 2024-03-06T18:35:46.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:46.881+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:46.882495+0000) 2024-03-06T18:35:47.877 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:47.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:47.882774+0000) 2024-03-06T18:35:48.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:48.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:48.883052+0000) 2024-03-06T18:35:49.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:49.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:49.883300+0000) 2024-03-06T18:35:50.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:50.882+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:50.883531+0000) 2024-03-06T18:35:51.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:51.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:51.883760+0000) 2024-03-06T18:35:52.878 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:52.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:52.883955+0000) 2024-03-06T18:35:53.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:53.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:53.884201+0000) 2024-03-06T18:35:54.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:54.883+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:54.884450+0000) 2024-03-06T18:35:55.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:55.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:55.884741+0000) 2024-03-06T18:35:56.879 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:56.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:56.884998+0000) 2024-03-06T18:35:57.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:57.884+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:57.885261+0000) 2024-03-06T18:35:58.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:58.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:58.885504+0000) 2024-03-06T18:35:59.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:35:59.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:59.885776+0000) 2024-03-06T18:36:00.880 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:00.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:00.886012+0000) 2024-03-06T18:36:01.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:01.885+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:01.886252+0000) 2024-03-06T18:36:02.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:02.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:02.886523+0000) 2024-03-06T18:36:03.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:03.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:03.886776+0000) 2024-03-06T18:36:04.881 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:04.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:04.887078+0000) 2024-03-06T18:36:05.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:05.886+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:05.887310+0000) 2024-03-06T18:36:06.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:06.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:06.887567+0000) 2024-03-06T18:36:07.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:07.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:07.887810+0000) 2024-03-06T18:36:08.882 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:08.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:08.888004+0000) 2024-03-06T18:36:09.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:09.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:09.888207+0000) 2024-03-06T18:36:10.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:10.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:10.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:10.887+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:10.888419+0000) 2024-03-06T18:36:11.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:11.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:11.888732+0000) 2024-03-06T18:36:12.883 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:12.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:12.888922+0000) 2024-03-06T18:36:13.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:13.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:13.889151+0000) 2024-03-06T18:36:14.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:14.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:14.889436+0000) 2024-03-06T18:36:15.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:15.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:15.889766+0000) 2024-03-06T18:36:16.884 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:16.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:16.889961+0000) 2024-03-06T18:36:17.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:17.888+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:17.890229+0000) 2024-03-06T18:36:18.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:18.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:18.890484+0000) 2024-03-06T18:36:19.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:19.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:19.890757+0000) 2024-03-06T18:36:20.885 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:20.889+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:20.890999+0000) 2024-03-06T18:36:21.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:21.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:21.891260+0000) 2024-03-06T18:36:22.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:22.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:22.891498+0000) 2024-03-06T18:36:23.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:23.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:23.891764+0000) 2024-03-06T18:36:24.886 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:24.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:24.892042+0000) 2024-03-06T18:36:25.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:25.890+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:25.892242+0000) 2024-03-06T18:36:26.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:26.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:26.892482+0000) 2024-03-06T18:36:27.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:27.891+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:27.892754+0000) 2024-03-06T18:36:28.887 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:28.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:28.892999+0000) 2024-03-06T18:36:29.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:29.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:29.893276+0000) 2024-03-06T18:36:30.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:30.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:30.893514+0000) 2024-03-06T18:36:31.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:31.892+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:31.893775+0000) 2024-03-06T18:36:32.888 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:32.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:32.894012+0000) 2024-03-06T18:36:33.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:33.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:33.894270+0000) 2024-03-06T18:36:34.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:34.893+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:34.894533+0000) 2024-03-06T18:36:35.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:35.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:35.894795+0000) 2024-03-06T18:36:36.889 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:36.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:36.895031+0000) 2024-03-06T18:36:37.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:37.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:37.895260+0000) 2024-03-06T18:36:38.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:38.894+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:38.895520+0000) 2024-03-06T18:36:39.890 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:39.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:39.895783+0000) 2024-03-06T18:36:40.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:40.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:40.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:40.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:40.896052+0000) 2024-03-06T18:36:41.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:41.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:41.896316+0000) 2024-03-06T18:36:42.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:42.895+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:42.896552+0000) 2024-03-06T18:36:43.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:43.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:43.896755+0000) 2024-03-06T18:36:44.891 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:44.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:44.897033+0000) 2024-03-06T18:36:45.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:45.896+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:45.897309+0000) 2024-03-06T18:36:46.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:46.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:46.897586+0000) 2024-03-06T18:36:47.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:47.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:47.897901+0000) 2024-03-06T18:36:48.892 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:48.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:48.898153+0000) 2024-03-06T18:36:49.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:49.897+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:49.898383+0000) 2024-03-06T18:36:50.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:50.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:50.898670+0000) 2024-03-06T18:36:51.893 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:51.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:51.898935+0000) 2024-03-06T18:36:52.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:52.898+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:52.899123+0000) 2024-03-06T18:36:53.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:53.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:53.899384+0000) 2024-03-06T18:36:54.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:54.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:54.899668+0000) 2024-03-06T18:36:55.894 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:55.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:55.899911+0000) 2024-03-06T18:36:56.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:56.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:56.900171+0000) 2024-03-06T18:36:57.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:57.899+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:57.900373+0000) 2024-03-06T18:36:58.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:58.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:58.900619+0000) 2024-03-06T18:36:59.895 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:36:59.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:59.900936+0000) 2024-03-06T18:37:00.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:00.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:00.901185+0000) 2024-03-06T18:37:01.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:01.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:01.901442+0000) 2024-03-06T18:37:02.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:02.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:02.901731+0000) 2024-03-06T18:37:03.896 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:03.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:03.902047+0000) 2024-03-06T18:37:04.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:04.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:04.902317+0000) 2024-03-06T18:37:05.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:05.900+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:05.902605+0000) 2024-03-06T18:37:06.897 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:06.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:06.902893+0000) 2024-03-06T18:37:07.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:07.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:07.903169+0000) 2024-03-06T18:37:08.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:08.901+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:08.903432+0000) 2024-03-06T18:37:09.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:09.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:09.903760+0000) 2024-03-06T18:37:10.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:10.756 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:10.898 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:10.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:10.903953+0000) 2024-03-06T18:37:11.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:11.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:11.904222+0000) 2024-03-06T18:37:12.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:12.902+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:12.904478+0000) 2024-03-06T18:37:13.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:13.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:13.904749+0000) 2024-03-06T18:37:14.899 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:14.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:14.904984+0000) 2024-03-06T18:37:15.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:15.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:15.905240+0000) 2024-03-06T18:37:16.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:16.903+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:16.905490+0000) 2024-03-06T18:37:17.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:17.904+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:17.905722+0000) 2024-03-06T18:37:18.900 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:18.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:18.905951+0000) 2024-03-06T18:37:19.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:19.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:19.906213+0000) 2024-03-06T18:37:20.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:20.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:20.906475+0000) 2024-03-06T18:37:21.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:21.905+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:21.906736+0000) 2024-03-06T18:37:22.901 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:22.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:22.906972+0000) 2024-03-06T18:37:23.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:23.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:23.907211+0000) 2024-03-06T18:37:24.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:24.906+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:24.907468+0000) 2024-03-06T18:37:25.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:25.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:25.907739+0000) 2024-03-06T18:37:26.902 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:26.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:26.907962+0000) 2024-03-06T18:37:27.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:27.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:27.908222+0000) 2024-03-06T18:37:28.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:28.907+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:28.908460+0000) 2024-03-06T18:37:29.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:29.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:29.908754+0000) 2024-03-06T18:37:30.903 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:30.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:30.909004+0000) 2024-03-06T18:37:31.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:31.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:31.909248+0000) 2024-03-06T18:37:32.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:32.908+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:32.909480+0000) 2024-03-06T18:37:33.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:33.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:33.909745+0000) 2024-03-06T18:37:34.904 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:34.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:34.910014+0000) 2024-03-06T18:37:35.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:35.909+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:35.910258+0000) 2024-03-06T18:37:36.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:36.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:36.910513+0000) 2024-03-06T18:37:37.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:37.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:37.910771+0000) 2024-03-06T18:37:38.905 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:38.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:38.911024+0000) 2024-03-06T18:37:39.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:39.910+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:39.911291+0000) 2024-03-06T18:37:40.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:40.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:40.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:40.911549+0000) 2024-03-06T18:37:40.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:41.906 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:41.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:41.911859+0000) 2024-03-06T18:37:42.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:42.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:42.912111+0000) 2024-03-06T18:37:43.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:43.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:43.912374+0000) 2024-03-06T18:37:44.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:44.911+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:44.912578+0000) 2024-03-06T18:37:45.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:45.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:45.912791+0000) 2024-03-06T18:37:46.907 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:46.912+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:46.913073+0000) 2024-03-06T18:37:47.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:47.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:47.913326+0000) 2024-03-06T18:37:48.908 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:48.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:48.913577+0000) 2024-03-06T18:37:49.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:49.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:49.913849+0000) 2024-03-06T18:37:50.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:50.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:50.914098+0000) 2024-03-06T18:37:51.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:51.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:51.914283+0000) 2024-03-06T18:37:52.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:52.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:52.914521+0000) 2024-03-06T18:37:53.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:53.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:53.914765+0000) 2024-03-06T18:37:54.909 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:54.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:54.915035+0000) 2024-03-06T18:37:55.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:55.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:55.915222+0000) 2024-03-06T18:37:56.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:56.913+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:56.915457+0000) 2024-03-06T18:37:57.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:57.914+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:57.915787+0000) 2024-03-06T18:37:58.910 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:58.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:58.916035+0000) 2024-03-06T18:37:59.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:37:59.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:59.916286+0000) 2024-03-06T18:38:00.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:00.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:00.916547+0000) 2024-03-06T18:38:01.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:01.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:01.916754+0000) 2024-03-06T18:38:02.911 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:02.915+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:02.916989+0000) 2024-03-06T18:38:03.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:03.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:03.917233+0000) 2024-03-06T18:38:04.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:04.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:04.917511+0000) 2024-03-06T18:38:05.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:05.916+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:05.917752+0000) 2024-03-06T18:38:06.912 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:06.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:06.918001+0000) 2024-03-06T18:38:07.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:07.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:07.918250+0000) 2024-03-06T18:38:08.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:08.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:08.918503+0000) 2024-03-06T18:38:09.913 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:09.917+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:09.918777+0000) 2024-03-06T18:38:10.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:10.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:10.919058+0000) 2024-03-06T18:38:11.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:11.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:11.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:11.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:11.919314+0000) 2024-03-06T18:38:12.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:12.918+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:12.919568+0000) 2024-03-06T18:38:13.914 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:13.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:13.919845+0000) 2024-03-06T18:38:14.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:14.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:14.920120+0000) 2024-03-06T18:38:15.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:15.919+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:15.920407+0000) 2024-03-06T18:38:16.915 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:16.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:16.920728+0000) 2024-03-06T18:38:17.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:17.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:17.921007+0000) 2024-03-06T18:38:18.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:18.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:18.921278+0000) 2024-03-06T18:38:19.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:19.920+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:19.921492+0000) 2024-03-06T18:38:20.916 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:20.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:20.921765+0000) 2024-03-06T18:38:21.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:21.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:21.921985+0000) 2024-03-06T18:38:22.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:22.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:22.922237+0000) 2024-03-06T18:38:23.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:23.921+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:23.922493+0000) 2024-03-06T18:38:24.917 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:24.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:24.922764+0000) 2024-03-06T18:38:25.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:25.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:25.923043+0000) 2024-03-06T18:38:26.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:26.922+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:26.923306+0000) 2024-03-06T18:38:27.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:27.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:27.923552+0000) 2024-03-06T18:38:28.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:28.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:28.923820+0000) 2024-03-06T18:38:29.918 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:29.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:29.924093+0000) 2024-03-06T18:38:30.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:30.923+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:30.924343+0000) 2024-03-06T18:38:31.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:31.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:31.924602+0000) 2024-03-06T18:38:32.919 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:32.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:32.924939+0000) 2024-03-06T18:38:33.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:33.924+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:33.925217+0000) 2024-03-06T18:38:34.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:34.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:34.925500+0000) 2024-03-06T18:38:35.920 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:35.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:35.925761+0000) 2024-03-06T18:38:36.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:36.925+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:36.926020+0000) 2024-03-06T18:38:37.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:37.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:37.926284+0000) 2024-03-06T18:38:38.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:38.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:38.926548+0000) 2024-03-06T18:38:39.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:39.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:39.926811+0000) 2024-03-06T18:38:40.921 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:40.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:40.927100+0000) 2024-03-06T18:38:41.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:41.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:41.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:41.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:41.927385+0000) 2024-03-06T18:38:42.922 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:42.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:42.927721+0000) 2024-03-06T18:38:43.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:43.926+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:43.927996+0000) 2024-03-06T18:38:44.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:44.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:44.928284+0000) 2024-03-06T18:38:45.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:45.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:45.928563+0000) 2024-03-06T18:38:46.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:46.927+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:46.928828+0000) 2024-03-06T18:38:47.923 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:47.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:47.929075+0000) 2024-03-06T18:38:48.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:48.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:48.929326+0000) 2024-03-06T18:38:49.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:49.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:49.929592+0000) 2024-03-06T18:38:50.924 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:50.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:50.929942+0000) 2024-03-06T18:38:51.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:51.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:51.930146+0000) 2024-03-06T18:38:52.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:52.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:52.930349+0000) 2024-03-06T18:38:53.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:53.928+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:53.930591+0000) 2024-03-06T18:38:54.925 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:54.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:54.930908+0000) 2024-03-06T18:38:55.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:55.929+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:55.931165+0000) 2024-03-06T18:38:56.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:56.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:56.931441+0000) 2024-03-06T18:38:57.926 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:57.930+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:57.931755+0000) 2024-03-06T18:38:58.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:58.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:58.932019+0000) 2024-03-06T18:38:59.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:38:59.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:59.932284+0000) 2024-03-06T18:39:00.927 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:00.931+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:00.932552+0000) 2024-03-06T18:39:01.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:01.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:01.932800+0000) 2024-03-06T18:39:02.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:02.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:02.933054+0000) 2024-03-06T18:39:03.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:03.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:03.933312+0000) 2024-03-06T18:39:04.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:04.932+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:04.933589+0000) 2024-03-06T18:39:05.928 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:05.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:05.933907+0000) 2024-03-06T18:39:06.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:06.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:06.934147+0000) 2024-03-06T18:39:07.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:07.933+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:07.934392+0000) 2024-03-06T18:39:08.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:08.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:08.934710+0000) 2024-03-06T18:39:09.929 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:09.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:09.934957+0000) 2024-03-06T18:39:10.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:10.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:10.935218+0000) 2024-03-06T18:39:11.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:11.362 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:11.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:11.934+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:11.935481+0000) 2024-03-06T18:39:12.930 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:12.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:12.935737+0000) 2024-03-06T18:39:13.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:13.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:13.936007+0000) 2024-03-06T18:39:14.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:14.935+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:14.936235+0000) 2024-03-06T18:39:15.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:15.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:15.936518+0000) 2024-03-06T18:39:16.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:16.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:16.936778+0000) 2024-03-06T18:39:17.931 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:17.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:17.936988+0000) 2024-03-06T18:39:18.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:18.936+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:18.937229+0000) 2024-03-06T18:39:19.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:19.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:19.937463+0000) 2024-03-06T18:39:20.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:20.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:20.937733+0000) 2024-03-06T18:39:21.932 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:21.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:21.937960+0000) 2024-03-06T18:39:22.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:22.937+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:22.938191+0000) 2024-03-06T18:39:23.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:23.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:23.938451+0000) 2024-03-06T18:39:24.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:24.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:24.938751+0000) 2024-03-06T18:39:25.933 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:25.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:25.939018+0000) 2024-03-06T18:39:26.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:26.938+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:26.939219+0000) 2024-03-06T18:39:27.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:27.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:27.939460+0000) 2024-03-06T18:39:28.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:28.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:28.939765+0000) 2024-03-06T18:39:29.934 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:29.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:29.940015+0000) 2024-03-06T18:39:30.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:30.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:30.940263+0000) 2024-03-06T18:39:31.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:31.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:31.940514+0000) 2024-03-06T18:39:32.935 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:32.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:32.940779+0000) 2024-03-06T18:39:33.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:33.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:33.941036+0000) 2024-03-06T18:39:34.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:34.939+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:34.941286+0000) 2024-03-06T18:39:35.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:35.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:35.941525+0000) 2024-03-06T18:39:36.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:36.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:36.941764+0000) 2024-03-06T18:39:37.936 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:37.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:37.941995+0000) 2024-03-06T18:39:38.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:38.940+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:38.942302+0000) 2024-03-06T18:39:39.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:39.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:39.942530+0000) 2024-03-06T18:39:40.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:40.941+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:40.942793+0000) 2024-03-06T18:39:41.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:41.501 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:41.937 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:41.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:41.943047+0000) 2024-03-06T18:39:42.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:42.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:42.943294+0000) 2024-03-06T18:39:43.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:43.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:43.943547+0000) 2024-03-06T18:39:44.938 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:44.942+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:44.943808+0000) 2024-03-06T18:39:45.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:45.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:45.944051+0000) 2024-03-06T18:39:46.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:46.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:46.944284+0000) 2024-03-06T18:39:47.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:47.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:47.944516+0000) 2024-03-06T18:39:48.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:48.943+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:48.944764+0000) 2024-03-06T18:39:49.939 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:49.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:49.944999+0000) 2024-03-06T18:39:50.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:50.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:50.945235+0000) 2024-03-06T18:39:51.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:51.944+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:51.945416+0000) 2024-03-06T18:39:52.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:52.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:52.945705+0000) 2024-03-06T18:39:53.940 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:53.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:53.945953+0000) 2024-03-06T18:39:54.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:54.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:54.946202+0000) 2024-03-06T18:39:55.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:55.945+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:55.946459+0000) 2024-03-06T18:39:56.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:56.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:56.946739+0000) 2024-03-06T18:39:57.941 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:57.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:57.946995+0000) 2024-03-06T18:39:58.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:58.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:58.947229+0000) 2024-03-06T18:39:59.942 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:39:59.946+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:59.947459+0000) 2024-03-06T18:40:00.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:00.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:00.947754+0000) 2024-03-06T18:40:01.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:01.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:01.948018+0000) 2024-03-06T18:40:02.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:02.947+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:02.948259+0000) 2024-03-06T18:40:03.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:03.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:03.948504+0000) 2024-03-06T18:40:04.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:04.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:04.948769+0000) 2024-03-06T18:40:05.943 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:05.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:05.949040+0000) 2024-03-06T18:40:06.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:06.948+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:06.949280+0000) 2024-03-06T18:40:07.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:07.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:07.949524+0000) 2024-03-06T18:40:08.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:08.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:08.949763+0000) 2024-03-06T18:40:09.944 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:09.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:09.950016+0000) 2024-03-06T18:40:10.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:10.949+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:10.950268+0000) 2024-03-06T18:40:11.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:11.661 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:11.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:11.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:11.950539+0000) 2024-03-06T18:40:12.945 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:12.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:12.950795+0000) 2024-03-06T18:40:13.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:13.950+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:13.951045+0000) 2024-03-06T18:40:14.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:14.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:14.951312+0000) 2024-03-06T18:40:15.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:15.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:15.951578+0000) 2024-03-06T18:40:16.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:16.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:16.951808+0000) 2024-03-06T18:40:17.946 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:17.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:17.952044+0000) 2024-03-06T18:40:18.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:18.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:18.952291+0000) 2024-03-06T18:40:19.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:19.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:19.952540+0000) 2024-03-06T18:40:20.947 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:20.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:20.952738+0000) 2024-03-06T18:40:21.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:21.951+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:21.952968+0000) 2024-03-06T18:40:22.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:22.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:22.953195+0000) 2024-03-06T18:40:23.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:23.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:23.953444+0000) 2024-03-06T18:40:24.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:24.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:24.953746+0000) 2024-03-06T18:40:25.948 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:25.952+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:25.954007+0000) 2024-03-06T18:40:26.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:26.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:26.954314+0000) 2024-03-06T18:40:27.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:27.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:27.954556+0000) 2024-03-06T18:40:28.949 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:28.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:28.954785+0000) 2024-03-06T18:40:29.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:29.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:29.955035+0000) 2024-03-06T18:40:30.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:30.953+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:30.955274+0000) 2024-03-06T18:40:31.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:31.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:31.955523+0000) 2024-03-06T18:40:32.950 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:32.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:32.955774+0000) 2024-03-06T18:40:33.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:33.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:33.956030+0000) 2024-03-06T18:40:34.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:34.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:34.956276+0000) 2024-03-06T18:40:35.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:35.954+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:35.956505+0000) 2024-03-06T18:40:36.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:36.955+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:36.956762+0000) 2024-03-06T18:40:37.951 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:37.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:37.957006+0000) 2024-03-06T18:40:38.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:38.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:38.957258+0000) 2024-03-06T18:40:39.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:39.956+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:39.957504+0000) 2024-03-06T18:40:40.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:40.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:40.957770+0000) 2024-03-06T18:40:41.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:41.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:41.952 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:41.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:41.958013+0000) 2024-03-06T18:40:42.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:42.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:42.958268+0000) 2024-03-06T18:40:43.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:43.957+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:43.958515+0000) 2024-03-06T18:40:44.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:44.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:44.958776+0000) 2024-03-06T18:40:45.953 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:45.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:45.959045+0000) 2024-03-06T18:40:46.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:46.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:46.959271+0000) 2024-03-06T18:40:47.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:47.958+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:47.959495+0000) 2024-03-06T18:40:48.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:48.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:48.959746+0000) 2024-03-06T18:40:49.954 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:49.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:49.960006+0000) 2024-03-06T18:40:50.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:50.959+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:50.960263+0000) 2024-03-06T18:40:51.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:51.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:51.960509+0000) 2024-03-06T18:40:52.955 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:52.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:52.960746+0000) 2024-03-06T18:40:53.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:53.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:53.961002+0000) 2024-03-06T18:40:54.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:54.960+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:54.961243+0000) 2024-03-06T18:40:55.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:55.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:55.961490+0000) 2024-03-06T18:40:56.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:56.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:56.961732+0000) 2024-03-06T18:40:57.956 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:57.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:57.961989+0000) 2024-03-06T18:40:58.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:58.961+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:58.962243+0000) 2024-03-06T18:40:59.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:40:59.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:59.962484+0000) 2024-03-06T18:41:00.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:00.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:00.962756+0000) 2024-03-06T18:41:01.957 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:01.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:01.962987+0000) 2024-03-06T18:41:02.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:02.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:02.963178+0000) 2024-03-06T18:41:03.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:03.962+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:03.963407+0000) 2024-03-06T18:41:04.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:04.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:04.963727+0000) 2024-03-06T18:41:05.958 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:05.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:05.963997+0000) 2024-03-06T18:41:06.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:06.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:06.964250+0000) 2024-03-06T18:41:07.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:07.963+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:07.964481+0000) 2024-03-06T18:41:08.959 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:08.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:08.964761+0000) 2024-03-06T18:41:09.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:09.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:09.965015+0000) 2024-03-06T18:41:10.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:10.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:10.965277+0000) 2024-03-06T18:41:11.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:11.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:11.965530+0000) 2024-03-06T18:41:11.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:11.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:12.960 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:12.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:12.965791+0000) 2024-03-06T18:41:13.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:13.964+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:13.966034+0000) 2024-03-06T18:41:14.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:14.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:14.966332+0000) 2024-03-06T18:41:15.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:15.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:15.966583+0000) 2024-03-06T18:41:16.961 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:16.965+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:16.966818+0000) 2024-03-06T18:41:17.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:17.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:17.967055+0000) 2024-03-06T18:41:18.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:18.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:18.967295+0000) 2024-03-06T18:41:19.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:19.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:19.967548+0000) 2024-03-06T18:41:20.962 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:20.966+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:20.967815+0000) 2024-03-06T18:41:21.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:21.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:21.968052+0000) 2024-03-06T18:41:22.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:22.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:22.968304+0000) 2024-03-06T18:41:23.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:23.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:23.968534+0000) 2024-03-06T18:41:24.963 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:24.967+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:24.968800+0000) 2024-03-06T18:41:25.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:25.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:25.969071+0000) 2024-03-06T18:41:26.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:26.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:26.969326+0000) 2024-03-06T18:41:27.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:27.968+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:27.969552+0000) 2024-03-06T18:41:28.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:28.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:28.969809+0000) 2024-03-06T18:41:29.964 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:29.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:29.970009+0000) 2024-03-06T18:41:30.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:30.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:30.970282+0000) 2024-03-06T18:41:31.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:31.969+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:31.970457+0000) 2024-03-06T18:41:32.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:32.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:32.970751+0000) 2024-03-06T18:41:33.965 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:33.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:33.970999+0000) 2024-03-06T18:41:34.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:34.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:34.971196+0000) 2024-03-06T18:41:35.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:35.970+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:35.971476+0000) 2024-03-06T18:41:36.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:36.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:36.971734+0000) 2024-03-06T18:41:37.966 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:37.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:37.971987+0000) 2024-03-06T18:41:38.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:38.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:38.972245+0000) 2024-03-06T18:41:39.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:39.971+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:39.972487+0000) 2024-03-06T18:41:40.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:40.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:40.972755+0000) 2024-03-06T18:41:41.967 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:41.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:41.973005+0000) 2024-03-06T18:41:42.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:42.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:42.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:42.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:42.973262+0000) 2024-03-06T18:41:43.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:43.972+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:43.973526+0000) 2024-03-06T18:41:44.968 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:44.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:44.973790+0000) 2024-03-06T18:41:45.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:45.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:45.974004+0000) 2024-03-06T18:41:46.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:46.973+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:46.974242+0000) 2024-03-06T18:41:47.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:47.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:47.974479+0000) 2024-03-06T18:41:48.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:48.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:48.974756+0000) 2024-03-06T18:41:49.969 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:49.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:49.975019+0000) 2024-03-06T18:41:50.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:50.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:50.975285+0000) 2024-03-06T18:41:51.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:51.974+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:51.975533+0000) 2024-03-06T18:41:52.970 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:52.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:52.975792+0000) 2024-03-06T18:41:53.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:53.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:53.976055+0000) 2024-03-06T18:41:54.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:54.975+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:54.976324+0000) 2024-03-06T18:41:55.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:55.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:55.976616+0000) 2024-03-06T18:41:56.971 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:56.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:56.976939+0000) 2024-03-06T18:41:57.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:57.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:57.977190+0000) 2024-03-06T18:41:58.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:58.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:58.977439+0000) 2024-03-06T18:41:59.972 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:41:59.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:59.977734+0000) 2024-03-06T18:42:00.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:00.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:00.978019+0000) 2024-03-06T18:42:01.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:01.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:01.978301+0000) 2024-03-06T18:42:02.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:02.976+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:02.978570+0000) 2024-03-06T18:42:03.973 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:03.977+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:03.978836+0000) 2024-03-06T18:42:04.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:04.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:04.979116+0000) 2024-03-06T18:42:05.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:05.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:05.979390+0000) 2024-03-06T18:42:06.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:06.978+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:06.979698+0000) 2024-03-06T18:42:07.974 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:07.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:07.979948+0000) 2024-03-06T18:42:08.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:08.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:08.980191+0000) 2024-03-06T18:42:09.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:09.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:09.980427+0000) 2024-03-06T18:42:10.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:10.979+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:10.980734+0000) 2024-03-06T18:42:11.975 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:11.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:11.980957+0000) 2024-03-06T18:42:12.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:12.307 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:12.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:12.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:12.981218+0000) 2024-03-06T18:42:13.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:13.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:13.981470+0000) 2024-03-06T18:42:14.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:14.980+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:14.981757+0000) 2024-03-06T18:42:15.976 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:15.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:15.982032+0000) 2024-03-06T18:42:16.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:16.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:16.982306+0000) 2024-03-06T18:42:17.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:17.981+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:17.982575+0000) 2024-03-06T18:42:18.977 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:18.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:18.982866+0000) 2024-03-06T18:42:19.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:19.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:19.983129+0000) 2024-03-06T18:42:20.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:20.982+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:20.983399+0000) 2024-03-06T18:42:21.978 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:21.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:21.983741+0000) 2024-03-06T18:42:22.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:22.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:22.984016+0000) 2024-03-06T18:42:23.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:23.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:23.984281+0000) 2024-03-06T18:42:24.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:24.983+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:24.984551+0000) 2024-03-06T18:42:25.979 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:25.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:25.984812+0000) 2024-03-06T18:42:26.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:26.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:26.985050+0000) 2024-03-06T18:42:27.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:27.984+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:27.985299+0000) 2024-03-06T18:42:28.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:28.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:28.985547+0000) 2024-03-06T18:42:29.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:29.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:29.985780+0000) 2024-03-06T18:42:30.980 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:30.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:30.986056+0000) 2024-03-06T18:42:31.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:31.985+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:31.986299+0000) 2024-03-06T18:42:32.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:32.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:32.986536+0000) 2024-03-06T18:42:33.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:33.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:33.986790+0000) 2024-03-06T18:42:34.981 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:34.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:34.987054+0000) 2024-03-06T18:42:35.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:35.986+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:35.987324+0000) 2024-03-06T18:42:36.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:36.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:36.987560+0000) 2024-03-06T18:42:37.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:37.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:37.987806+0000) 2024-03-06T18:42:38.982 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:38.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:38.988058+0000) 2024-03-06T18:42:39.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:39.987+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:39.988300+0000) 2024-03-06T18:42:40.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:40.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:40.988572+0000) 2024-03-06T18:42:41.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:41.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:41.988822+0000) 2024-03-06T18:42:42.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:42.448 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:42.983 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:42.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:42.989061+0000) 2024-03-06T18:42:43.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:43.988+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:43.989296+0000) 2024-03-06T18:42:44.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:44.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:44.989557+0000) 2024-03-06T18:42:45.984 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:45.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:45.989779+0000) 2024-03-06T18:42:46.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:46.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:46.990036+0000) 2024-03-06T18:42:47.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:47.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:47.990292+0000) 2024-03-06T18:42:48.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:48.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:48.990554+0000) 2024-03-06T18:42:49.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:49.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:49.990814+0000) 2024-03-06T18:42:50.985 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:50.989+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:50.991068+0000) 2024-03-06T18:42:51.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:51.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:51.991324+0000) 2024-03-06T18:42:52.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:52.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:52.991552+0000) 2024-03-06T18:42:53.986 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:53.990+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:53.991806+0000) 2024-03-06T18:42:54.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:54.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:54.992050+0000) 2024-03-06T18:42:55.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:55.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:55.992325+0000) 2024-03-06T18:42:56.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:56.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:56.992587+0000) 2024-03-06T18:42:57.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:57.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:57.992850+0000) 2024-03-06T18:42:58.987 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:58.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:58.993088+0000) 2024-03-06T18:42:59.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:42:59.992+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:59.993344+0000) 2024-03-06T18:43:00.988 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:00.991+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:00.993606+0000) 2024-03-06T18:43:01.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:01.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:01.993996+0000) 2024-03-06T18:43:02.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:02.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:02.994252+0000) 2024-03-06T18:43:03.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:03.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:03.994513+0000) 2024-03-06T18:43:04.989 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:04.993+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:04.994761+0000) 2024-03-06T18:43:05.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:05.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:05.994979+0000) 2024-03-06T18:43:06.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:06.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:06.995240+0000) 2024-03-06T18:43:07.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:07.994+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:07.995470+0000) 2024-03-06T18:43:08.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:08.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:08.995767+0000) 2024-03-06T18:43:09.990 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:09.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:09.996008+0000) 2024-03-06T18:43:10.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:10.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:10.996223+0000) 2024-03-06T18:43:11.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:11.995+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:11.996469+0000) 2024-03-06T18:43:12.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:12.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:12.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:12.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:12.996763+0000) 2024-03-06T18:43:13.991 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:13.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:13.997015+0000) 2024-03-06T18:43:14.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:14.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:14.997270+0000) 2024-03-06T18:43:15.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:15.996+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:15.997506+0000) 2024-03-06T18:43:16.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:16.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:16.997770+0000) 2024-03-06T18:43:17.992 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:17.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:17.998016+0000) 2024-03-06T18:43:18.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:18.997+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:18.998256+0000) 2024-03-06T18:43:19.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:19.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:19.998510+0000) 2024-03-06T18:43:20.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:20.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:20.998774+0000) 2024-03-06T18:43:21.993 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:21.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:21.999035+0000) 2024-03-06T18:43:22.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:22.998+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:22.999323+0000) 2024-03-06T18:43:23.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:23.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:23.999587+0000) 2024-03-06T18:43:24.994 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:24.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:24.999860+0000) 2024-03-06T18:43:25.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:25.999+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:26.000123+0000) 2024-03-06T18:43:26.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:27.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:27.000361+0000) 2024-03-06T18:43:27.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:28.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:28.000673+0000) 2024-03-06T18:43:28.995 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:29.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:29.000924+0000) 2024-03-06T18:43:29.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:30.000+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:30.001168+0000) 2024-03-06T18:43:30.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:31.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:31.001433+0000) 2024-03-06T18:43:31.996 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:32.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:32.001744+0000) 2024-03-06T18:43:32.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:33.001+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:33.001985+0000) 2024-03-06T18:43:33.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:34.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:34.002248+0000) 2024-03-06T18:43:34.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:35.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:35.002514+0000) 2024-03-06T18:43:35.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:36.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:36.002777+0000) 2024-03-06T18:43:36.997 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:37.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:37.003035+0000) 2024-03-06T18:43:37.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:38.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:38.003303+0000) 2024-03-06T18:43:38.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:39.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:39.003571+0000) 2024-03-06T18:43:39.998 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:40.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:40.003798+0000) 2024-03-06T18:43:40.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:41.002+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:41.004059+0000) 2024-03-06T18:43:41.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:42.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:42.004314+0000) 2024-03-06T18:43:42.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:42.754 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:42.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:43.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:43.004577+0000) 2024-03-06T18:43:43.999 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:44.003+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:44.004893+0000) 2024-03-06T18:43:45.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:45.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:45.005149+0000) 2024-03-06T18:43:46.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:46.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:46.005347+0000) 2024-03-06T18:43:47.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:47.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:47.005612+0000) 2024-03-06T18:43:48.000 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:48.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:48.005944+0000) 2024-03-06T18:43:49.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:49.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:49.006205+0000) 2024-03-06T18:43:50.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:50.004+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:50.006451+0000) 2024-03-06T18:43:51.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:51.005+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:51.006759+0000) 2024-03-06T18:43:52.001 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:52.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:52.007003+0000) 2024-03-06T18:43:53.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:53.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:53.007270+0000) 2024-03-06T18:43:54.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:54.006+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:54.007553+0000) 2024-03-06T18:43:55.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:55.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:55.007822+0000) 2024-03-06T18:43:56.002 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:56.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:56.008067+0000) 2024-03-06T18:43:57.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:57.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:57.008314+0000) 2024-03-06T18:43:58.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:58.007+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:58.008574+0000) 2024-03-06T18:43:59.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:43:59.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:59.008838+0000) 2024-03-06T18:44:00.003 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:00.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:00.009106+0000) 2024-03-06T18:44:01.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:01.008+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:01.009350+0000) 2024-03-06T18:44:02.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:02.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:02.009620+0000) 2024-03-06T18:44:03.004 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:03.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:03.009960+0000) 2024-03-06T18:44:04.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:04.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:04.010198+0000) 2024-03-06T18:44:05.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:05.009+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:05.010459+0000) 2024-03-06T18:44:06.005 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:06.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:06.010767+0000) 2024-03-06T18:44:07.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:07.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:07.011001+0000) 2024-03-06T18:44:08.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:08.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:08.011243+0000) 2024-03-06T18:44:09.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:09.010+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:09.011485+0000) 2024-03-06T18:44:10.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:10.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:10.011738+0000) 2024-03-06T18:44:11.006 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:11.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:11.011983+0000) 2024-03-06T18:44:12.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:12.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:12.012232+0000) 2024-03-06T18:44:12.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:12.919 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:13.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:13.011+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:13.012402+0000) 2024-03-06T18:44:14.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:14.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:14.012699+0000) 2024-03-06T18:44:15.007 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:15.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:15.012953+0000) 2024-03-06T18:44:16.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:16.012+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:16.013191+0000) 2024-03-06T18:44:17.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:17.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:17.013457+0000) 2024-03-06T18:44:18.008 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:18.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:18.013751+0000) 2024-03-06T18:44:19.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:19.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:19.013983+0000) 2024-03-06T18:44:20.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:20.013+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:20.014181+0000) 2024-03-06T18:44:21.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:21.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:21.014435+0000) 2024-03-06T18:44:22.009 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:22.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:22.014804+0000) 2024-03-06T18:44:23.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:23.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:23.015048+0000) 2024-03-06T18:44:24.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:24.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:24.015248+0000) 2024-03-06T18:44:25.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:25.014+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:25.015500+0000) 2024-03-06T18:44:26.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:26.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:26.015746+0000) 2024-03-06T18:44:27.010 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:27.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:27.015992+0000) 2024-03-06T18:44:28.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:28.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:28.016229+0000) 2024-03-06T18:44:29.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:29.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:29.016481+0000) 2024-03-06T18:44:30.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:30.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:30.016750+0000) 2024-03-06T18:44:31.011 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:31.015+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:31.016976+0000) 2024-03-06T18:44:32.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:32.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:32.017283+0000) 2024-03-06T18:44:33.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:33.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:33.017537+0000) 2024-03-06T18:44:34.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:34.016+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:34.017801+0000) 2024-03-06T18:44:35.012 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:35.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:35.018068+0000) 2024-03-06T18:44:36.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:36.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:36.018294+0000) 2024-03-06T18:44:37.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:37.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:37.018535+0000) 2024-03-06T18:44:38.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:38.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:38.018794+0000) 2024-03-06T18:44:39.013 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:39.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:39.019041+0000) 2024-03-06T18:44:40.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:40.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:40.019250+0000) 2024-03-06T18:44:41.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:41.017+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:41.019452+0000) 2024-03-06T18:44:42.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:42.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:42.019706+0000) 2024-03-06T18:44:43.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:43.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:43.019905+0000) 2024-03-06T18:44:43.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:43.062 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:44.014 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:44.018+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:44.020110+0000) 2024-03-06T18:44:45.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:45.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:45.020321+0000) 2024-03-06T18:44:46.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:46.019+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:46.020504+0000) 2024-03-06T18:44:47.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:47.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:47.020740+0000) 2024-03-06T18:44:48.015 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:48.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:48.020929+0000) 2024-03-06T18:44:49.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:49.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:49.021132+0000) 2024-03-06T18:44:50.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:50.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:50.021320+0000) 2024-03-06T18:44:51.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:51.020+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:51.021503+0000) 2024-03-06T18:44:52.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:52.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:52.021766+0000) 2024-03-06T18:44:53.016 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:53.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:53.021963+0000) 2024-03-06T18:44:54.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:54.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:54.022151+0000) 2024-03-06T18:44:55.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:55.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:55.022344+0000) 2024-03-06T18:44:56.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:56.021+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:56.022581+0000) 2024-03-06T18:44:57.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:57.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:57.022854+0000) 2024-03-06T18:44:58.017 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:58.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:58.023043+0000) 2024-03-06T18:44:59.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:44:59.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:59.023233+0000) 2024-03-06T18:45:00.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:00.022+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:00.023421+0000) 2024-03-06T18:45:01.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:01.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:01.023736+0000) 2024-03-06T18:45:02.018 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:02.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:02.023931+0000) 2024-03-06T18:45:03.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:03.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:03.024121+0000) 2024-03-06T18:45:04.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:04.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:04.024311+0000) 2024-03-06T18:45:05.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:05.023+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:05.024508+0000) 2024-03-06T18:45:06.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:06.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:06.024738+0000) 2024-03-06T18:45:07.019 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:07.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:07.024933+0000) 2024-03-06T18:45:08.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:08.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:08.025123+0000) 2024-03-06T18:45:09.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:09.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:09.025309+0000) 2024-03-06T18:45:10.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:10.024+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:10.025556+0000) 2024-03-06T18:45:11.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:11.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:11.025805+0000) 2024-03-06T18:45:12.020 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:12.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:12.026009+0000) 2024-03-06T18:45:13.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:13.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:13.026204+0000) 2024-03-06T18:45:13.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:13.212 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:14.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:14.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:14.026409+0000) 2024-03-06T18:45:15.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:15.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:15.026749+0000) 2024-03-06T18:45:16.021 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:16.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:16.026960+0000) 2024-03-06T18:45:17.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:17.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:17.027154+0000) 2024-03-06T18:45:18.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:18.025+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:18.027402+0000) 2024-03-06T18:45:19.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:19.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:19.027699+0000) 2024-03-06T18:45:20.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:20.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:20.027973+0000) 2024-03-06T18:45:21.022 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:21.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:21.028166+0000) 2024-03-06T18:45:22.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:22.026+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:22.028369+0000) 2024-03-06T18:45:23.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:23.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:23.028618+0000) 2024-03-06T18:45:24.023 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:24.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:24.028908+0000) 2024-03-06T18:45:25.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:25.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:25.029173+0000) 2024-03-06T18:45:26.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:26.027+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:26.029454+0000) 2024-03-06T18:45:27.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:27.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:27.029769+0000) 2024-03-06T18:45:28.024 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:28.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:28.030003+0000) 2024-03-06T18:45:29.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:29.028+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:29.030186+0000) 2024-03-06T18:45:30.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:30.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:30.030453+0000) 2024-03-06T18:45:31.025 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:31.029+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:31.030747+0000) 2024-03-06T18:45:32.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:32.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:32.030996+0000) 2024-03-06T18:45:33.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:33.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:33.031248+0000) 2024-03-06T18:45:34.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:34.030+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:34.031501+0000) 2024-03-06T18:45:35.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:35.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:35.031764+0000) 2024-03-06T18:45:36.026 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:36.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:36.031995+0000) 2024-03-06T18:45:37.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:37.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:37.032268+0000) 2024-03-06T18:45:38.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:38.031+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:38.032523+0000) 2024-03-06T18:45:39.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:39.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:39.032782+0000) 2024-03-06T18:45:40.027 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:40.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:40.033037+0000) 2024-03-06T18:45:41.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:41.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:41.033292+0000) 2024-03-06T18:45:42.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:42.032+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:42.033539+0000) 2024-03-06T18:45:43.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:43.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:43.033771+0000) 2024-03-06T18:45:43.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:43.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:44.028 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:44.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:44.034020+0000) 2024-03-06T18:45:45.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:45.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:45.034286+0000) 2024-03-06T18:45:46.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:46.033+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:46.034519+0000) 2024-03-06T18:45:47.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:47.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:47.034789+0000) 2024-03-06T18:45:48.029 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:48.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:48.035037+0000) 2024-03-06T18:45:49.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:49.034+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:49.035291+0000) 2024-03-06T18:45:50.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:50.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:50.035547+0000) 2024-03-06T18:45:51.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:51.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:51.035801+0000) 2024-03-06T18:45:52.030 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:52.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:52.036078+0000) 2024-03-06T18:45:53.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:53.035+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:53.036315+0000) 2024-03-06T18:45:54.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:54.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:54.036576+0000) 2024-03-06T18:45:55.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:55.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:55.036839+0000) 2024-03-06T18:45:56.031 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:56.036+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:56.037071+0000) 2024-03-06T18:45:57.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:57.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:57.037345+0000) 2024-03-06T18:45:58.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:58.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:58.037584+0000) 2024-03-06T18:45:59.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:45:59.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:59.037827+0000) 2024-03-06T18:46:00.032 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:00.037+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:00.038088+0000) 2024-03-06T18:46:01.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:01.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:01.038341+0000) 2024-03-06T18:46:02.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:02.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:02.038613+0000) 2024-03-06T18:46:03.033 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:03.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:03.038931+0000) 2024-03-06T18:46:04.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:04.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:04.039185+0000) 2024-03-06T18:46:05.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:05.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:05.039449+0000) 2024-03-06T18:46:06.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:06.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:06.039749+0000) 2024-03-06T18:46:07.034 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:07.038+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:07.040010+0000) 2024-03-06T18:46:08.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:08.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:08.040277+0000) 2024-03-06T18:46:09.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:09.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:09.040520+0000) 2024-03-06T18:46:10.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:10.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:10.040785+0000) 2024-03-06T18:46:11.035 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:11.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:11.040987+0000) 2024-03-06T18:46:12.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:12.039+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:12.041180+0000) 2024-03-06T18:46:13.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:13.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:13.041429+0000) 2024-03-06T18:46:13.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:13.528 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:14.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:14.040+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:14.041740+0000) 2024-03-06T18:46:15.036 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:15.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:15.041998+0000) 2024-03-06T18:46:16.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:16.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:16.042254+0000) 2024-03-06T18:46:17.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:17.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:17.042527+0000) 2024-03-06T18:46:18.037 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:18.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:18.042785+0000) 2024-03-06T18:46:19.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:19.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:19.043084+0000) 2024-03-06T18:46:20.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:20.042+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:20.043321+0000) 2024-03-06T18:46:21.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:21.041+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:21.043573+0000) 2024-03-06T18:46:22.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:22.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:22.043847+0000) 2024-03-06T18:46:23.038 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:23.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:23.044083+0000) 2024-03-06T18:46:24.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:24.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:24.044346+0000) 2024-03-06T18:46:25.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:25.043+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:25.044610+0000) 2024-03-06T18:46:26.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:26.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:26.044904+0000) 2024-03-06T18:46:27.039 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:27.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:27.045177+0000) 2024-03-06T18:46:28.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:28.044+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:28.045441+0000) 2024-03-06T18:46:29.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:29.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:29.045762+0000) 2024-03-06T18:46:30.040 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:30.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:30.046005+0000) 2024-03-06T18:46:31.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:31.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:31.046243+0000) 2024-03-06T18:46:32.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:32.045+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:32.046504+0000) 2024-03-06T18:46:33.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:33.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:33.046757+0000) 2024-03-06T18:46:34.041 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:34.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:34.047002+0000) 2024-03-06T18:46:35.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:35.046+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:35.047273+0000) 2024-03-06T18:46:36.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:36.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:36.047534+0000) 2024-03-06T18:46:37.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:37.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:37.047810+0000) 2024-03-06T18:46:38.042 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:38.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:38.048043+0000) 2024-03-06T18:46:39.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:39.047+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:39.048281+0000) 2024-03-06T18:46:40.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:40.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:40.048522+0000) 2024-03-06T18:46:41.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:41.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:41.048775+0000) 2024-03-06T18:46:42.043 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:42.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:42.049042+0000) 2024-03-06T18:46:43.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:43.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:43.049327+0000) 2024-03-06T18:46:43.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:43.669 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:44.044 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:44.048+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:44.049607+0000) 2024-03-06T18:46:45.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:45.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:45.049974+0000) 2024-03-06T18:46:46.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:46.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:46.050277+0000) 2024-03-06T18:46:47.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:47.049+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:47.050597+0000) 2024-03-06T18:46:48.045 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:48.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:48.050946+0000) 2024-03-06T18:46:49.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:49.050+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:49.051256+0000) 2024-03-06T18:46:50.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:50.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:50.051571+0000) 2024-03-06T18:46:51.046 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:51.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:51.051887+0000) 2024-03-06T18:46:52.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:52.051+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:52.052212+0000) 2024-03-06T18:46:53.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:53.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:53.052461+0000) 2024-03-06T18:46:54.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:54.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:54.052777+0000) 2024-03-06T18:46:55.047 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:55.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:55.053027+0000) 2024-03-06T18:46:56.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:56.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:56.053337+0000) 2024-03-06T18:46:57.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:57.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:57.053721+0000) 2024-03-06T18:46:58.048 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:58.052+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:58.054006+0000) 2024-03-06T18:46:59.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:46:59.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:59.054265+0000) 2024-03-06T18:47:00.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:00.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:00.054522+0000) 2024-03-06T18:47:01.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:01.053+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:01.054762+0000) 2024-03-06T18:47:02.049 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:02.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:02.055026+0000) 2024-03-06T18:47:03.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:03.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:03.055285+0000) 2024-03-06T18:47:04.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:04.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:04.055530+0000) 2024-03-06T18:47:05.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:05.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:05.055794+0000) 2024-03-06T18:47:06.050 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:06.054+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:06.056013+0000) 2024-03-06T18:47:07.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:07.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:07.056272+0000) 2024-03-06T18:47:08.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:08.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:08.056524+0000) 2024-03-06T18:47:09.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:09.055+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:09.056784+0000) 2024-03-06T18:47:10.051 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:10.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:10.057026+0000) 2024-03-06T18:47:11.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:11.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:11.057281+0000) 2024-03-06T18:47:12.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:12.056+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:12.057587+0000) 2024-03-06T18:47:13.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:13.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:13.057848+0000) 2024-03-06T18:47:13.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:13.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:14.052 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:14.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:14.058121+0000) 2024-03-06T18:47:15.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:15.057+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:15.058399+0000) 2024-03-06T18:47:16.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:16.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:16.058700+0000) 2024-03-06T18:47:17.053 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:17.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:17.058970+0000) 2024-03-06T18:47:18.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:18.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:18.059218+0000) 2024-03-06T18:47:19.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:19.058+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:19.059450+0000) 2024-03-06T18:47:20.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:20.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:20.059755+0000) 2024-03-06T18:47:21.054 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:21.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:21.060015+0000) 2024-03-06T18:47:22.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:22.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:22.060265+0000) 2024-03-06T18:47:23.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:23.059+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:23.060526+0000) 2024-03-06T18:47:24.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:24.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:24.060783+0000) 2024-03-06T18:47:25.055 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:25.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:25.060976+0000) 2024-03-06T18:47:26.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:26.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:26.061236+0000) 2024-03-06T18:47:27.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:27.060+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:27.061485+0000) 2024-03-06T18:47:28.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:28.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:28.061749+0000) 2024-03-06T18:47:29.056 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:29.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:29.062014+0000) 2024-03-06T18:47:30.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:30.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:30.062284+0000) 2024-03-06T18:47:31.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:31.061+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:31.062554+0000) 2024-03-06T18:47:32.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:32.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:32.062843+0000) 2024-03-06T18:47:33.057 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:33.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:33.063105+0000) 2024-03-06T18:47:34.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:34.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:34.063351+0000) 2024-03-06T18:47:35.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:35.062+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:35.063599+0000) 2024-03-06T18:47:36.058 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:36.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:36.063904+0000) 2024-03-06T18:47:37.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:37.063+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:37.064162+0000) 2024-03-06T18:47:38.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:38.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:38.064425+0000) 2024-03-06T18:47:39.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:39.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:39.064730+0000) 2024-03-06T18:47:40.059 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:40.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:40.064988+0000) 2024-03-06T18:47:41.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:41.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:41.065249+0000) 2024-03-06T18:47:42.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:42.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:42.065526+0000) 2024-03-06T18:47:43.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:43.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:43.065773+0000) 2024-03-06T18:47:43.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:43.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:44.060 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:44.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:44.066032+0000) 2024-03-06T18:47:45.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:45.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:45.066290+0000) 2024-03-06T18:47:46.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:46.064+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:46.066537+0000) 2024-03-06T18:47:47.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:47.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:47.066801+0000) 2024-03-06T18:47:48.061 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:48.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:48.067054+0000) 2024-03-06T18:47:49.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:49.065+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:49.067254+0000) 2024-03-06T18:47:50.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:50.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:50.067517+0000) 2024-03-06T18:47:51.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:51.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:51.067777+0000) 2024-03-06T18:47:52.062 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:52.066+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:52.067981+0000) 2024-03-06T18:47:53.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:53.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:53.068228+0000) 2024-03-06T18:47:54.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:54.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:54.068483+0000) 2024-03-06T18:47:55.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:55.067+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:55.068805+0000) 2024-03-06T18:47:56.063 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:56.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:56.069070+0000) 2024-03-06T18:47:57.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:57.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:57.069331+0000) 2024-03-06T18:47:58.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:58.068+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:58.069593+0000) 2024-03-06T18:47:59.064 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:47:59.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:59.069863+0000) 2024-03-06T18:48:00.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:00.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:00.070129+0000) 2024-03-06T18:48:01.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:01.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:01.070393+0000) 2024-03-06T18:48:02.065 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:02.069+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:02.070726+0000) 2024-03-06T18:48:03.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:03.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:03.070968+0000) 2024-03-06T18:48:04.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:04.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:04.071226+0000) 2024-03-06T18:48:05.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:05.070+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:05.071488+0000) 2024-03-06T18:48:06.066 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:06.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:06.071744+0000) 2024-03-06T18:48:07.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:07.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:07.071981+0000) 2024-03-06T18:48:08.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:08.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:08.072182+0000) 2024-03-06T18:48:09.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:09.071+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:09.072450+0000) 2024-03-06T18:48:10.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:10.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:10.072757+0000) 2024-03-06T18:48:11.067 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:11.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:11.073025+0000) 2024-03-06T18:48:12.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:12.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:12.073281+0000) 2024-03-06T18:48:13.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:13.072+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:13.073550+0000) 2024-03-06T18:48:14.068 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:14.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:14.073818+0000) 2024-03-06T18:48:14.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:14.091 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:15.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:15.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:15.074055+0000) 2024-03-06T18:48:16.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:16.073+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:16.074359+0000) 2024-03-06T18:48:17.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:17.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:17.074715+0000) 2024-03-06T18:48:18.069 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:18.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:18.075036+0000) 2024-03-06T18:48:19.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:19.074+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:19.075318+0000) 2024-03-06T18:48:20.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:20.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:20.075625+0000) 2024-03-06T18:48:21.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:21.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:21.075914+0000) 2024-03-06T18:48:22.070 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:22.075+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:22.076163+0000) 2024-03-06T18:48:23.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:23.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:23.076397+0000) 2024-03-06T18:48:24.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:24.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:24.076676+0000) 2024-03-06T18:48:25.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:25.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:25.076960+0000) 2024-03-06T18:48:26.071 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:26.076+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:26.077207+0000) 2024-03-06T18:48:27.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:27.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:27.077445+0000) 2024-03-06T18:48:28.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:28.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:28.077759+0000) 2024-03-06T18:48:29.072 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:29.077+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:29.078010+0000) 2024-03-06T18:48:30.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:30.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:30.078264+0000) 2024-03-06T18:48:31.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:31.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:31.078502+0000) 2024-03-06T18:48:32.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:32.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:32.078750+0000) 2024-03-06T18:48:33.073 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:33.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:33.078995+0000) 2024-03-06T18:48:34.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:34.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:34.079255+0000) 2024-03-06T18:48:35.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:35.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:35.079519+0000) 2024-03-06T18:48:36.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:36.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:36.079784+0000) 2024-03-06T18:48:37.074 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:37.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:37.080042+0000) 2024-03-06T18:48:38.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:38.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:38.080292+0000) 2024-03-06T18:48:39.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:39.078+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:39.080536+0000) 2024-03-06T18:48:40.075 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:40.079+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:40.080796+0000) 2024-03-06T18:48:41.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:41.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:41.081053+0000) 2024-03-06T18:48:42.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:42.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:42.081314+0000) 2024-03-06T18:48:43.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:43.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:43.081622+0000) 2024-03-06T18:48:44.076 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:44.080+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:44.081927+0000) 2024-03-06T18:48:44.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:44.230 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:45.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:45.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:45.082190+0000) 2024-03-06T18:48:46.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:46.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:46.082445+0000) 2024-03-06T18:48:47.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:47.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:47.082741+0000) 2024-03-06T18:48:48.077 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:48.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:48.083005+0000) 2024-03-06T18:48:49.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:49.081+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:49.083246+0000) 2024-03-06T18:48:50.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:50.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:50.083440+0000) 2024-03-06T18:48:51.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:51.082+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:51.083770+0000) 2024-03-06T18:48:52.078 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:52.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:52.084040+0000) 2024-03-06T18:48:53.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:53.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:53.084278+0000) 2024-03-06T18:48:54.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:54.083+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:54.084518+0000) 2024-03-06T18:48:55.079 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:55.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:55.084759+0000) 2024-03-06T18:48:56.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:56.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:56.085013+0000) 2024-03-06T18:48:57.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:57.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:57.085264+0000) 2024-03-06T18:48:58.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:58.084+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:58.085527+0000) 2024-03-06T18:48:59.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:48:59.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:59.085765+0000) 2024-03-06T18:49:00.080 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:00.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:00.085947+0000) 2024-03-06T18:49:01.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:01.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:01.086188+0000) 2024-03-06T18:49:02.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:02.085+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:02.086416+0000) 2024-03-06T18:49:03.081 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:03.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:03.086716+0000) 2024-03-06T18:49:04.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:04.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:04.086977+0000) 2024-03-06T18:49:05.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:05.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:05.087221+0000) 2024-03-06T18:49:06.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:06.086+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:06.087464+0000) 2024-03-06T18:49:07.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:07.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:07.087755+0000) 2024-03-06T18:49:08.082 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:08.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:08.088019+0000) 2024-03-06T18:49:09.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:09.087+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:09.088256+0000) 2024-03-06T18:49:10.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:10.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:10.088511+0000) 2024-03-06T18:49:11.083 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:11.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:11.088763+0000) 2024-03-06T18:49:12.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:12.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:12.089003+0000) 2024-03-06T18:49:13.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:13.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:13.089236+0000) 2024-03-06T18:49:14.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:14.088+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:14.089479+0000) 2024-03-06T18:49:14.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:14.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:15.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:15.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:15.089746+0000) 2024-03-06T18:49:16.084 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:16.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:16.090002+0000) 2024-03-06T18:49:17.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:17.089+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:17.090254+0000) 2024-03-06T18:49:18.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:18.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:18.090511+0000) 2024-03-06T18:49:19.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:19.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:19.090777+0000) 2024-03-06T18:49:20.085 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:20.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:20.090971+0000) 2024-03-06T18:49:21.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:21.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:21.091226+0000) 2024-03-06T18:49:22.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:22.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:22.091460+0000) 2024-03-06T18:49:23.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:23.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:23.091746+0000) 2024-03-06T18:49:24.086 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:24.090+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:24.092004+0000) 2024-03-06T18:49:25.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:25.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:25.092251+0000) 2024-03-06T18:49:26.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:26.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:26.092509+0000) 2024-03-06T18:49:27.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:27.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:27.092765+0000) 2024-03-06T18:49:28.087 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:28.091+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:28.093005+0000) 2024-03-06T18:49:29.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:29.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:29.093241+0000) 2024-03-06T18:49:30.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:30.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:30.093429+0000) 2024-03-06T18:49:31.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:31.092+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:31.093738+0000) 2024-03-06T18:49:32.088 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:32.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:32.093985+0000) 2024-03-06T18:49:33.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:33.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:33.094244+0000) 2024-03-06T18:49:34.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:34.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:34.094484+0000) 2024-03-06T18:49:35.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:35.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:35.094756+0000) 2024-03-06T18:49:36.089 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:36.093+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:36.094945+0000) 2024-03-06T18:49:37.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:37.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:37.095207+0000) 2024-03-06T18:49:38.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:38.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:38.095493+0000) 2024-03-06T18:49:39.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:39.094+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:39.095747+0000) 2024-03-06T18:49:40.090 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:40.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:40.095985+0000) 2024-03-06T18:49:41.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:41.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:41.096240+0000) 2024-03-06T18:49:42.091 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:42.095+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:42.096502+0000) 2024-03-06T18:49:43.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:43.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:43.096764+0000) 2024-03-06T18:49:44.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:44.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:44.097022+0000) 2024-03-06T18:49:44.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:44.527 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:45.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:45.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:45.097298+0000) 2024-03-06T18:49:46.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:46.096+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:46.097565+0000) 2024-03-06T18:49:47.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:47.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:47.097823+0000) 2024-03-06T18:49:48.092 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:48.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:48.098074+0000) 2024-03-06T18:49:49.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:49.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:49.098331+0000) 2024-03-06T18:49:50.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:50.097+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:50.098588+0000) 2024-03-06T18:49:51.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:51.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:51.098837+0000) 2024-03-06T18:49:52.093 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:52.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:52.099077+0000) 2024-03-06T18:49:53.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:53.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:53.099311+0000) 2024-03-06T18:49:54.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:54.098+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:54.099581+0000) 2024-03-06T18:49:55.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:55.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:55.099856+0000) 2024-03-06T18:49:56.094 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:56.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:56.100114+0000) 2024-03-06T18:49:57.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:57.099+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:57.100378+0000) 2024-03-06T18:49:58.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:58.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:58.100616+0000) 2024-03-06T18:49:59.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:49:59.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:59.100920+0000) 2024-03-06T18:50:00.095 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:00.100+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:00.101174+0000) 2024-03-06T18:50:01.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:01.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:01.101425+0000) 2024-03-06T18:50:02.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:02.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:02.101725+0000) 2024-03-06T18:50:03.096 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:03.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:03.101991+0000) 2024-03-06T18:50:04.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:04.101+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:04.102230+0000) 2024-03-06T18:50:05.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:05.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:05.102508+0000) 2024-03-06T18:50:06.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:06.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:06.102757+0000) 2024-03-06T18:50:07.097 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:07.102+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:07.103013+0000) 2024-03-06T18:50:08.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:08.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:08.103282+0000) 2024-03-06T18:50:09.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:09.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:09.103541+0000) 2024-03-06T18:50:10.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:10.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:10.103809+0000) 2024-03-06T18:50:11.098 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:11.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:11.104062+0000) 2024-03-06T18:50:12.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:12.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:12.104307+0000) 2024-03-06T18:50:13.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:13.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:13.104549+0000) 2024-03-06T18:50:14.099 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:14.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:14.104791+0000) 2024-03-06T18:50:14.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:14.699 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:15.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:15.103+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:15.105065+0000) 2024-03-06T18:50:16.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:16.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:16.105324+0000) 2024-03-06T18:50:17.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:17.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:17.105599+0000) 2024-03-06T18:50:18.100 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:18.104+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:18.105907+0000) 2024-03-06T18:50:19.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:19.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:19.106151+0000) 2024-03-06T18:50:20.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:20.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:20.106404+0000) 2024-03-06T18:50:21.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:21.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:21.106684+0000) 2024-03-06T18:50:22.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:22.105+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:22.106910+0000) 2024-03-06T18:50:23.101 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:23.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:23.107158+0000) 2024-03-06T18:50:24.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:24.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:24.107423+0000) 2024-03-06T18:50:25.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:25.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:25.107742+0000) 2024-03-06T18:50:26.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:26.106+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:26.107945+0000) 2024-03-06T18:50:27.102 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:27.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:27.108183+0000) 2024-03-06T18:50:28.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:28.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:28.108443+0000) 2024-03-06T18:50:29.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:29.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:29.108742+0000) 2024-03-06T18:50:30.103 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:30.107+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:30.109001+0000) 2024-03-06T18:50:31.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:31.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:31.109264+0000) 2024-03-06T18:50:32.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:32.108+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:32.109515+0000) 2024-03-06T18:50:33.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:33.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:33.109801+0000) 2024-03-06T18:50:34.104 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:34.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:34.110084+0000) 2024-03-06T18:50:35.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:35.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:35.110340+0000) 2024-03-06T18:50:36.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:36.109+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:36.110595+0000) 2024-03-06T18:50:37.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:37.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:37.110841+0000) 2024-03-06T18:50:38.105 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:38.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:38.111095+0000) 2024-03-06T18:50:39.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:39.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:39.111351+0000) 2024-03-06T18:50:40.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:40.110+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:40.111595+0000) 2024-03-06T18:50:41.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:41.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:41.111839+0000) 2024-03-06T18:50:42.106 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:42.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:42.112072+0000) 2024-03-06T18:50:43.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:43.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:43.112343+0000) 2024-03-06T18:50:44.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:44.111+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:44.112594+0000) 2024-03-06T18:50:44.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:44.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:45.107 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:45.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:45.112929+0000) 2024-03-06T18:50:46.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:46.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:46.113194+0000) 2024-03-06T18:50:47.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:47.112+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:47.113452+0000) 2024-03-06T18:50:48.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:48.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:48.113753+0000) 2024-03-06T18:50:49.108 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:49.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:49.113999+0000) 2024-03-06T18:50:50.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:50.113+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:50.114261+0000) 2024-03-06T18:50:51.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:51.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:51.114532+0000) 2024-03-06T18:50:52.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:52.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:52.114794+0000) 2024-03-06T18:50:53.109 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:53.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:53.115046+0000) 2024-03-06T18:50:54.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:54.114+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:54.115287+0000) 2024-03-06T18:50:55.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:55.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:55.115531+0000) 2024-03-06T18:50:56.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:56.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:56.115779+0000) 2024-03-06T18:50:57.110 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:57.115+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:57.116024+0000) 2024-03-06T18:50:58.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:58.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:58.116297+0000) 2024-03-06T18:50:59.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:50:59.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:59.116548+0000) 2024-03-06T18:51:00.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:00.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:00.116789+0000) 2024-03-06T18:51:01.111 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:01.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:01.116982+0000) 2024-03-06T18:51:02.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:02.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:02.117243+0000) 2024-03-06T18:51:03.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:03.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:03.117515+0000) 2024-03-06T18:51:04.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:04.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:04.117774+0000) 2024-03-06T18:51:05.112 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:05.116+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:05.117969+0000) 2024-03-06T18:51:06.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:06.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:06.118238+0000) 2024-03-06T18:51:07.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:07.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:07.118475+0000) 2024-03-06T18:51:08.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:08.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:08.118770+0000) 2024-03-06T18:51:09.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:09.117+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:09.118997+0000) 2024-03-06T18:51:10.113 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:10.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:10.119262+0000) 2024-03-06T18:51:11.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:11.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:11.119513+0000) 2024-03-06T18:51:12.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:12.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:12.119767+0000) 2024-03-06T18:51:13.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:13.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:13.119960+0000) 2024-03-06T18:51:14.114 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:14.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:14.120219+0000) 2024-03-06T18:51:14.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:15.000 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:15.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:15.118+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:15.120484+0000) 2024-03-06T18:51:16.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:16.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:16.120740+0000) 2024-03-06T18:51:17.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:17.119+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:17.120939+0000) 2024-03-06T18:51:18.115 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:18.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:18.121208+0000) 2024-03-06T18:51:19.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:19.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:19.121461+0000) 2024-03-06T18:51:20.116 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:20.120+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:20.121766+0000) 2024-03-06T18:51:21.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:21.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:21.122023+0000) 2024-03-06T18:51:22.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:22.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:22.122283+0000) 2024-03-06T18:51:23.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:23.121+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:23.122601+0000) 2024-03-06T18:51:24.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:24.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:24.122818+0000) 2024-03-06T18:51:25.117 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:25.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:25.123066+0000) 2024-03-06T18:51:26.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:26.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:26.123308+0000) 2024-03-06T18:51:27.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:27.122+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:27.123555+0000) 2024-03-06T18:51:28.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:28.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:28.123806+0000) 2024-03-06T18:51:29.118 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:29.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:29.124068+0000) 2024-03-06T18:51:30.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:30.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:30.124320+0000) 2024-03-06T18:51:31.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:31.123+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:31.124561+0000) 2024-03-06T18:51:32.119 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:32.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:32.124801+0000) 2024-03-06T18:51:33.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:33.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:33.125063+0000) 2024-03-06T18:51:34.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:34.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:34.125315+0000) 2024-03-06T18:51:35.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:35.124+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:35.125574+0000) 2024-03-06T18:51:36.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:36.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:36.125839+0000) 2024-03-06T18:51:37.120 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:37.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:37.126101+0000) 2024-03-06T18:51:38.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:38.125+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:38.126360+0000) 2024-03-06T18:51:39.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:39.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:39.126669+0000) 2024-03-06T18:51:40.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:40.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:40.126933+0000) 2024-03-06T18:51:41.121 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:41.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:41.127160+0000) 2024-03-06T18:51:42.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:42.126+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:42.127415+0000) 2024-03-06T18:51:43.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:43.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:43.127733+0000) 2024-03-06T18:51:44.122 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:44.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:44.127999+0000) 2024-03-06T18:51:45.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:45.127+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:45.128272+0000) 2024-03-06T18:51:45.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:45.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:46.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:46.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:46.128539+0000) 2024-03-06T18:51:47.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:47.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:47.128774+0000) 2024-03-06T18:51:48.123 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:48.128+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:48.129036+0000) 2024-03-06T18:51:49.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:49.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:49.129294+0000) 2024-03-06T18:51:50.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:50.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:50.129536+0000) 2024-03-06T18:51:51.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:51.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:51.129794+0000) 2024-03-06T18:51:52.124 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:52.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:52.130054+0000) 2024-03-06T18:51:53.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:53.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:53.130322+0000) 2024-03-06T18:51:54.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:54.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:54.130565+0000) 2024-03-06T18:51:55.125 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:55.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:55.130831+0000) 2024-03-06T18:51:56.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:56.129+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:56.131032+0000) 2024-03-06T18:51:57.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:57.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:57.131295+0000) 2024-03-06T18:51:58.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:58.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:58.131532+0000) 2024-03-06T18:51:59.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:51:59.130+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:59.131789+0000) 2024-03-06T18:52:00.126 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:00.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:00.132036+0000) 2024-03-06T18:52:01.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:01.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:01.132288+0000) 2024-03-06T18:52:02.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:02.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:02.132548+0000) 2024-03-06T18:52:03.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:03.131+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:03.132806+0000) 2024-03-06T18:52:04.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:04.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:04.133052+0000) 2024-03-06T18:52:05.127 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:05.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:05.133311+0000) 2024-03-06T18:52:06.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:06.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:06.133569+0000) 2024-03-06T18:52:07.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:07.132+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:07.133826+0000) 2024-03-06T18:52:08.128 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:08.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:08.134076+0000) 2024-03-06T18:52:09.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:09.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:09.134336+0000) 2024-03-06T18:52:10.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:10.133+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:10.134592+0000) 2024-03-06T18:52:11.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:11.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:11.134927+0000) 2024-03-06T18:52:12.129 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:12.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:12.135196+0000) 2024-03-06T18:52:13.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:13.134+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:13.135468+0000) 2024-03-06T18:52:14.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:14.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:14.135741+0000) 2024-03-06T18:52:15.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:15.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:15.136007+0000) 2024-03-06T18:52:15.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:15.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:16.130 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:16.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:16.136213+0000) 2024-03-06T18:52:17.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:17.135+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:17.136442+0000) 2024-03-06T18:52:18.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:18.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:18.136752+0000) 2024-03-06T18:52:19.131 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:19.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:19.137013+0000) 2024-03-06T18:52:20.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:20.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:20.137266+0000) 2024-03-06T18:52:21.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:21.136+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:21.137519+0000) 2024-03-06T18:52:22.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:22.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:22.137753+0000) 2024-03-06T18:52:23.132 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:23.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:23.137994+0000) 2024-03-06T18:52:24.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:24.137+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:24.138235+0000) 2024-03-06T18:52:25.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:25.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:25.138497+0000) 2024-03-06T18:52:26.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:26.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:26.138791+0000) 2024-03-06T18:52:27.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:27.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:27.139026+0000) 2024-03-06T18:52:28.133 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:28.138+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:28.139274+0000) 2024-03-06T18:52:29.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:29.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:29.139532+0000) 2024-03-06T18:52:30.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:30.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:30.139789+0000) 2024-03-06T18:52:31.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:31.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:31.139970+0000) 2024-03-06T18:52:32.134 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:32.139+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:32.140239+0000) 2024-03-06T18:52:33.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:33.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:33.140514+0000) 2024-03-06T18:52:34.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:34.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:34.140772+0000) 2024-03-06T18:52:35.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:35.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:35.141038+0000) 2024-03-06T18:52:36.135 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:36.140+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:36.141233+0000) 2024-03-06T18:52:37.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:37.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:37.141474+0000) 2024-03-06T18:52:38.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:38.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:38.141779+0000) 2024-03-06T18:52:39.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:39.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:39.142035+0000) 2024-03-06T18:52:40.136 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:40.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:40.142282+0000) 2024-03-06T18:52:41.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:41.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:41.142539+0000) 2024-03-06T18:52:42.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:42.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:42.142783+0000) 2024-03-06T18:52:43.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:43.141+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:43.143032+0000) 2024-03-06T18:52:44.137 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:44.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:44.143271+0000) 2024-03-06T18:52:45.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:45.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:45.143531+0000) 2024-03-06T18:52:45.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:45.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:46.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:46.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:46.143797+0000) 2024-03-06T18:52:47.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:47.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:47.144050+0000) 2024-03-06T18:52:48.138 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:48.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:48.144294+0000) 2024-03-06T18:52:49.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:49.142+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:49.144533+0000) 2024-03-06T18:52:50.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:50.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:50.144751+0000) 2024-03-06T18:52:51.139 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:51.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:51.145005+0000) 2024-03-06T18:52:52.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:52.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:52.145273+0000) 2024-03-06T18:52:53.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:53.143+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:53.145509+0000) 2024-03-06T18:52:54.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:54.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:54.145817+0000) 2024-03-06T18:52:55.140 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:55.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:55.146068+0000) 2024-03-06T18:52:56.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:56.144+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:56.146306+0000) 2024-03-06T18:52:57.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:57.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:57.146559+0000) 2024-03-06T18:52:58.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:58.145+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:58.146818+0000) 2024-03-06T18:52:59.141 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:52:59.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:59.147087+0000) 2024-03-06T18:53:00.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:00.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:00.147328+0000) 2024-03-06T18:53:01.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:01.146+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:01.147584+0000) 2024-03-06T18:53:02.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:02.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:02.147843+0000) 2024-03-06T18:53:03.142 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:03.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:03.148087+0000) 2024-03-06T18:53:04.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:04.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:04.148281+0000) 2024-03-06T18:53:05.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:05.147+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:05.148547+0000) 2024-03-06T18:53:06.143 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:06.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:06.148796+0000) 2024-03-06T18:53:07.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:07.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:07.149051+0000) 2024-03-06T18:53:08.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:08.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:08.149253+0000) 2024-03-06T18:53:09.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:09.148+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:09.149508+0000) 2024-03-06T18:53:10.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:10.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:10.149744+0000) 2024-03-06T18:53:11.144 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:11.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:11.149978+0000) 2024-03-06T18:53:12.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:12.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:12.150236+0000) 2024-03-06T18:53:13.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:13.149+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:13.150479+0000) 2024-03-06T18:53:14.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:14.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:14.150735+0000) 2024-03-06T18:53:15.145 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:15.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:15.151009+0000) 2024-03-06T18:53:15.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:15.590 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:16.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:16.150+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:16.151273+0000) 2024-03-06T18:53:17.146 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:17.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:17.151537+0000) 2024-03-06T18:53:18.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:18.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:18.151839+0000) 2024-03-06T18:53:19.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:19.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:19.152097+0000) 2024-03-06T18:53:20.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:20.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:20.152339+0000) 2024-03-06T18:53:21.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:21.151+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:21.152587+0000) 2024-03-06T18:53:22.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:22.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:22.152753+0000) 2024-03-06T18:53:23.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:23.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:23.152941+0000) 2024-03-06T18:53:24.147 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:24.152+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:24.153190+0000) 2024-03-06T18:53:25.148 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:25.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:25.153445+0000) 2024-03-06T18:53:26.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:26.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:26.153747+0000) 2024-03-06T18:53:27.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:27.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:27.154002+0000) 2024-03-06T18:53:28.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:28.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:28.154318+0000) 2024-03-06T18:53:29.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:29.153+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:29.154609+0000) 2024-03-06T18:53:30.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:30.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:30.154968+0000) 2024-03-06T18:53:31.149 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:31.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:31.155276+0000) 2024-03-06T18:53:32.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:32.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:32.155586+0000) 2024-03-06T18:53:33.150 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:33.154+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:33.155946+0000) 2024-03-06T18:53:34.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:34.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:34.156264+0000) 2024-03-06T18:53:35.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:35.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:35.156556+0000) 2024-03-06T18:53:36.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:36.155+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:36.156838+0000) 2024-03-06T18:53:37.151 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:37.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:37.157134+0000) 2024-03-06T18:53:38.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:38.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:38.157435+0000) 2024-03-06T18:53:39.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:39.156+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:39.157805+0000) 2024-03-06T18:53:40.152 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:40.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:40.158079+0000) 2024-03-06T18:53:41.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:41.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:41.158341+0000) 2024-03-06T18:53:42.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:42.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:42.158600+0000) 2024-03-06T18:53:43.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:43.157+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:43.158895+0000) 2024-03-06T18:53:44.153 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:44.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:44.159152+0000) 2024-03-06T18:53:45.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:45.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:45.159414+0000) 2024-03-06T18:53:45.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:45.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:46.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:46.158+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:46.159718+0000) 2024-03-06T18:53:47.154 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:47.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:47.160032+0000) 2024-03-06T18:53:48.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:48.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:48.160272+0000) 2024-03-06T18:53:49.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:49.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:49.160512+0000) 2024-03-06T18:53:50.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:50.159+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:50.160752+0000) 2024-03-06T18:53:51.155 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:51.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:51.160998+0000) 2024-03-06T18:53:52.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:52.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:52.161187+0000) 2024-03-06T18:53:53.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:53.160+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:53.161424+0000) 2024-03-06T18:53:54.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:54.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:54.161731+0000) 2024-03-06T18:53:55.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:55.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:55.161994+0000) 2024-03-06T18:53:56.156 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:56.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:56.162244+0000) 2024-03-06T18:53:57.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:57.161+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:57.162515+0000) 2024-03-06T18:53:58.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:58.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:58.162753+0000) 2024-03-06T18:53:59.157 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:53:59.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:59.163005+0000) 2024-03-06T18:54:00.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:00.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:00.163260+0000) 2024-03-06T18:54:01.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:01.162+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:01.163516+0000) 2024-03-06T18:54:02.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:02.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:02.163775+0000) 2024-03-06T18:54:03.158 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:03.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:03.164016+0000) 2024-03-06T18:54:04.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:04.163+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:04.164279+0000) 2024-03-06T18:54:05.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:05.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:05.164521+0000) 2024-03-06T18:54:06.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:06.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:06.164750+0000) 2024-03-06T18:54:07.159 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:07.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:07.165063+0000) 2024-03-06T18:54:08.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:08.164+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:08.165345+0000) 2024-03-06T18:54:09.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:09.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:09.165611+0000) 2024-03-06T18:54:10.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:10.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:10.165908+0000) 2024-03-06T18:54:11.160 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:11.165+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:11.166165+0000) 2024-03-06T18:54:12.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:12.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:12.166424+0000) 2024-03-06T18:54:13.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:13.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:13.166725+0000) 2024-03-06T18:54:14.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:14.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:14.166974+0000) 2024-03-06T18:54:15.161 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:15.166+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:15.167216+0000) 2024-03-06T18:54:15.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:15.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:16.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:16.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:16.167482+0000) 2024-03-06T18:54:17.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:17.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:17.167757+0000) 2024-03-06T18:54:18.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:18.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:18.168001+0000) 2024-03-06T18:54:19.162 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:19.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:19.168258+0000) 2024-03-06T18:54:20.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:20.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:20.168499+0000) 2024-03-06T18:54:21.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:21.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:21.168746+0000) 2024-03-06T18:54:22.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:22.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:22.169003+0000) 2024-03-06T18:54:23.163 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:23.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:23.169259+0000) 2024-03-06T18:54:24.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:24.167+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:24.169522+0000) 2024-03-06T18:54:25.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:25.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:25.169769+0000) 2024-03-06T18:54:26.164 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:26.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:26.169966+0000) 2024-03-06T18:54:27.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:27.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:27.170160+0000) 2024-03-06T18:54:28.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:28.168+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:28.170355+0000) 2024-03-06T18:54:29.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:29.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:29.170596+0000) 2024-03-06T18:54:30.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:30.169+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:30.170832+0000) 2024-03-06T18:54:31.165 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:31.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:31.171084+0000) 2024-03-06T18:54:32.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:32.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:32.171353+0000) 2024-03-06T18:54:33.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:33.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:33.171610+0000) 2024-03-06T18:54:34.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:34.170+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:34.171893+0000) 2024-03-06T18:54:35.166 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:35.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:35.172160+0000) 2024-03-06T18:54:36.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:36.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:36.172404+0000) 2024-03-06T18:54:37.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:37.171+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:37.172699+0000) 2024-03-06T18:54:38.167 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:38.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:38.172957+0000) 2024-03-06T18:54:39.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:39.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:39.173198+0000) 2024-03-06T18:54:40.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:40.172+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:40.173491+0000) 2024-03-06T18:54:41.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:41.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:41.173754+0000) 2024-03-06T18:54:42.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:42.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:42.174007+0000) 2024-03-06T18:54:43.168 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:43.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:43.174254+0000) 2024-03-06T18:54:44.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:44.173+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:44.174526+0000) 2024-03-06T18:54:45.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:45.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:45.174774+0000) 2024-03-06T18:54:46.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:46.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:46.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:46.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:46.174960+0000) 2024-03-06T18:54:47.169 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:47.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:47.175202+0000) 2024-03-06T18:54:48.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:48.174+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:48.175448+0000) 2024-03-06T18:54:49.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:49.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:49.175745+0000) 2024-03-06T18:54:50.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:50.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:50.176002+0000) 2024-03-06T18:54:51.170 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:51.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:51.176248+0000) 2024-03-06T18:54:52.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:52.175+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:52.176512+0000) 2024-03-06T18:54:53.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:53.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:53.176752+0000) 2024-03-06T18:54:54.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:54.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:54.176955+0000) 2024-03-06T18:54:55.171 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:55.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:55.177191+0000) 2024-03-06T18:54:56.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:56.176+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:56.177448+0000) 2024-03-06T18:54:57.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:57.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:57.177733+0000) 2024-03-06T18:54:58.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:58.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:58.177993+0000) 2024-03-06T18:54:59.172 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:54:59.177+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:59.178259+0000) 2024-03-06T18:55:00.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:00.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:00.178510+0000) 2024-03-06T18:55:01.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:01.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:01.178783+0000) 2024-03-06T18:55:02.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:02.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:02.179055+0000) 2024-03-06T18:55:03.173 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:03.178+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:03.179316+0000) 2024-03-06T18:55:04.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:04.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:04.179564+0000) 2024-03-06T18:55:05.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:05.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:05.179828+0000) 2024-03-06T18:55:06.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:06.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:06.180039+0000) 2024-03-06T18:55:07.174 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:07.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:07.180302+0000) 2024-03-06T18:55:08.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:08.179+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:08.180489+0000) 2024-03-06T18:55:09.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:09.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:09.180767+0000) 2024-03-06T18:55:10.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:10.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:10.180968+0000) 2024-03-06T18:55:11.175 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:11.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:11.181228+0000) 2024-03-06T18:55:12.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:12.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:12.181466+0000) 2024-03-06T18:55:13.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:13.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:13.181752+0000) 2024-03-06T18:55:14.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:14.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:14.181993+0000) 2024-03-06T18:55:15.176 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:15.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:15.182255+0000) 2024-03-06T18:55:16.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:16.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:16.180+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:16.182448+0000) 2024-03-06T18:55:16.188 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:17.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:17.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:17.182745+0000) 2024-03-06T18:55:18.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:18.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:18.182987+0000) 2024-03-06T18:55:19.177 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:19.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:19.183191+0000) 2024-03-06T18:55:20.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:20.181+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:20.183384+0000) 2024-03-06T18:55:21.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:21.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:21.183657+0000) 2024-03-06T18:55:22.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:22.182+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:22.183907+0000) 2024-03-06T18:55:23.178 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:23.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:23.184154+0000) 2024-03-06T18:55:24.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:24.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:24.184424+0000) 2024-03-06T18:55:25.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:25.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:25.184706+0000) 2024-03-06T18:55:26.179 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:26.183+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:26.184903+0000) 2024-03-06T18:55:27.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:27.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:27.185146+0000) 2024-03-06T18:55:28.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:28.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:28.185464+0000) 2024-03-06T18:55:29.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:29.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:29.185771+0000) 2024-03-06T18:55:30.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:30.184+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:30.186007+0000) 2024-03-06T18:55:31.180 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:31.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:31.186256+0000) 2024-03-06T18:55:32.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:32.185+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:32.186441+0000) 2024-03-06T18:55:33.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:33.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:33.186739+0000) 2024-03-06T18:55:34.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:34.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:34.186999+0000) 2024-03-06T18:55:35.181 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:35.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:35.187220+0000) 2024-03-06T18:55:36.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:36.186+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:36.187500+0000) 2024-03-06T18:55:37.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:37.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:37.187736+0000) 2024-03-06T18:55:38.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:38.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:38.187973+0000) 2024-03-06T18:55:39.182 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:39.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:39.188269+0000) 2024-03-06T18:55:40.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:40.187+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:40.188502+0000) 2024-03-06T18:55:41.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:41.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:41.188755+0000) 2024-03-06T18:55:42.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:42.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:42.188999+0000) 2024-03-06T18:55:43.183 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:43.188+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:43.189248+0000) 2024-03-06T18:55:44.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:44.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:44.189508+0000) 2024-03-06T18:55:45.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:45.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:45.189772+0000) 2024-03-06T18:55:46.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:46.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:46.190010+0000) 2024-03-06T18:55:46.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:46.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:47.184 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:47.189+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:47.190250+0000) 2024-03-06T18:55:48.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:48.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:48.190496+0000) 2024-03-06T18:55:49.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:49.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:49.190772+0000) 2024-03-06T18:55:50.185 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:50.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:50.191025+0000) 2024-03-06T18:55:51.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:51.190+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:51.191256+0000) 2024-03-06T18:55:52.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:52.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:52.191511+0000) 2024-03-06T18:55:53.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:53.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:53.191766+0000) 2024-03-06T18:55:54.186 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:54.191+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:54.192036+0000) 2024-03-06T18:55:55.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:55.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:55.192296+0000) 2024-03-06T18:55:56.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:56.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:56.192550+0000) 2024-03-06T18:55:57.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:57.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:57.192806+0000) 2024-03-06T18:55:58.187 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:58.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:58.193062+0000) 2024-03-06T18:55:59.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:55:59.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:59.193307+0000) 2024-03-06T18:56:00.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:00.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:00.193547+0000) 2024-03-06T18:56:01.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:01.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:01.193801+0000) 2024-03-06T18:56:02.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:02.192+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:02.193985+0000) 2024-03-06T18:56:03.188 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:03.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:03.194232+0000) 2024-03-06T18:56:04.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:04.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:04.194475+0000) 2024-03-06T18:56:05.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:05.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:05.194744+0000) 2024-03-06T18:56:06.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:06.193+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:06.195002+0000) 2024-03-06T18:56:07.189 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:07.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:07.195237+0000) 2024-03-06T18:56:08.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:08.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:08.195497+0000) 2024-03-06T18:56:09.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:09.194+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:09.195757+0000) 2024-03-06T18:56:10.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:10.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:10.196007+0000) 2024-03-06T18:56:11.190 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:11.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:11.196269+0000) 2024-03-06T18:56:12.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:12.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:12.196521+0000) 2024-03-06T18:56:13.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:13.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:13.196772+0000) 2024-03-06T18:56:14.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:14.195+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:14.196974+0000) 2024-03-06T18:56:15.191 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:15.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:15.197233+0000) 2024-03-06T18:56:16.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:16.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:16.197525+0000) 2024-03-06T18:56:16.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:16.502 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:17.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:17.196+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:17.197766+0000) 2024-03-06T18:56:18.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:18.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:18.198014+0000) 2024-03-06T18:56:19.192 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:19.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:19.198264+0000) 2024-03-06T18:56:20.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:20.197+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:20.198513+0000) 2024-03-06T18:56:21.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:21.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:21.198751+0000) 2024-03-06T18:56:22.193 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:22.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:22.199007+0000) 2024-03-06T18:56:23.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:23.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:23.199262+0000) 2024-03-06T18:56:24.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:24.198+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:24.199504+0000) 2024-03-06T18:56:25.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:25.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:25.199768+0000) 2024-03-06T18:56:26.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:26.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:26.200022+0000) 2024-03-06T18:56:27.194 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:27.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:27.200281+0000) 2024-03-06T18:56:28.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:28.199+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:28.200520+0000) 2024-03-06T18:56:29.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:29.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:29.200781+0000) 2024-03-06T18:56:30.195 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:30.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:30.201031+0000) 2024-03-06T18:56:31.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:31.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:31.201273+0000) 2024-03-06T18:56:32.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:32.200+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:32.201516+0000) 2024-03-06T18:56:33.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:33.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:33.201769+0000) 2024-03-06T18:56:34.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:34.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:34.202036+0000) 2024-03-06T18:56:35.196 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:35.201+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:35.202300+0000) 2024-03-06T18:56:36.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:36.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:36.202556+0000) 2024-03-06T18:56:37.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:37.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:37.202803+0000) 2024-03-06T18:56:38.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:38.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:38.203042+0000) 2024-03-06T18:56:39.197 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:39.202+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:39.203290+0000) 2024-03-06T18:56:40.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:40.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:40.203542+0000) 2024-03-06T18:56:41.198 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:41.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:41.203775+0000) 2024-03-06T18:56:42.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:42.203+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:42.204020+0000) 2024-03-06T18:56:43.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:43.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:43.204297+0000) 2024-03-06T18:56:44.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:44.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:44.204571+0000) 2024-03-06T18:56:45.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:45.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:45.204884+0000) 2024-03-06T18:56:46.199 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:46.204+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:46.205141+0000) 2024-03-06T18:56:46.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:46.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:47.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:47.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:47.205415+0000) 2024-03-06T18:56:48.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:48.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:48.205707+0000) 2024-03-06T18:56:49.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:49.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:49.205975+0000) 2024-03-06T18:56:50.200 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:50.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:50.206236+0000) 2024-03-06T18:56:51.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:51.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:51.206493+0000) 2024-03-06T18:56:52.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:52.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:52.206745+0000) 2024-03-06T18:56:53.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:53.205+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:53.206979+0000) 2024-03-06T18:56:54.201 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:54.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:54.207253+0000) 2024-03-06T18:56:55.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:55.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:55.207519+0000) 2024-03-06T18:56:56.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:56.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:56.207762+0000) 2024-03-06T18:56:57.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:57.206+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:57.207990+0000) 2024-03-06T18:56:58.202 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:58.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:58.208244+0000) 2024-03-06T18:56:59.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:56:59.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:59.208492+0000) 2024-03-06T18:57:00.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:00.207+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:00.208749+0000) 2024-03-06T18:57:01.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:01.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:01.209003+0000) 2024-03-06T18:57:02.203 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:02.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:02.209267+0000) 2024-03-06T18:57:03.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:03.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:03.209507+0000) 2024-03-06T18:57:04.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:04.208+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:04.209772+0000) 2024-03-06T18:57:05.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:05.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:05.210028+0000) 2024-03-06T18:57:06.204 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:06.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:06.210287+0000) 2024-03-06T18:57:07.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:07.209+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:07.210545+0000) 2024-03-06T18:57:08.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:08.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:08.210789+0000) 2024-03-06T18:57:09.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:09.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:09.211053+0000) 2024-03-06T18:57:10.205 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:10.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:10.211291+0000) 2024-03-06T18:57:11.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:11.210+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:11.211545+0000) 2024-03-06T18:57:12.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:12.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:12.211807+0000) 2024-03-06T18:57:13.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:13.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:13.212043+0000) 2024-03-06T18:57:14.206 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:14.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:14.212318+0000) 2024-03-06T18:57:15.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:15.211+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:15.212567+0000) 2024-03-06T18:57:16.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:16.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:16.212826+0000) 2024-03-06T18:57:16.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:16.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:17.207 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:17.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:17.213061+0000) 2024-03-06T18:57:18.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:18.212+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:18.213306+0000) 2024-03-06T18:57:19.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:19.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:19.213571+0000) 2024-03-06T18:57:20.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:20.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:20.213811+0000) 2024-03-06T18:57:21.208 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:21.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:21.214067+0000) 2024-03-06T18:57:22.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:22.213+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:22.214310+0000) 2024-03-06T18:57:23.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:23.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:23.214565+0000) 2024-03-06T18:57:24.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:24.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:24.214835+0000) 2024-03-06T18:57:25.209 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:25.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:25.215108+0000) 2024-03-06T18:57:26.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:26.214+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:26.215347+0000) 2024-03-06T18:57:27.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:27.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:27.215584+0000) 2024-03-06T18:57:28.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:28.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:28.215837+0000) 2024-03-06T18:57:29.210 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:29.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:29.216040+0000) 2024-03-06T18:57:30.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:30.215+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:30.216279+0000) 2024-03-06T18:57:31.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:31.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:31.216521+0000) 2024-03-06T18:57:32.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:32.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:32.216784+0000) 2024-03-06T18:57:33.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:33.216+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:33.217037+0000) 2024-03-06T18:57:34.211 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:34.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:34.217304+0000) 2024-03-06T18:57:35.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:35.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:35.217568+0000) 2024-03-06T18:57:36.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:36.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:36.217814+0000) 2024-03-06T18:57:37.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:37.217+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:37.218021+0000) 2024-03-06T18:57:38.212 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:38.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:38.218298+0000) 2024-03-06T18:57:39.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:39.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:39.218578+0000) 2024-03-06T18:57:40.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:40.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:40.218850+0000) 2024-03-06T18:57:41.213 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:41.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:41.219126+0000) 2024-03-06T18:57:42.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:42.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:42.219405+0000) 2024-03-06T18:57:43.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:43.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:43.219728+0000) 2024-03-06T18:57:44.214 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:44.218+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:44.219998+0000) 2024-03-06T18:57:45.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:45.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:45.220269+0000) 2024-03-06T18:57:46.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:46.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:46.220534+0000) 2024-03-06T18:57:46.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:46.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:47.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:47.219+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:47.220798+0000) 2024-03-06T18:57:48.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:48.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:48.221075+0000) 2024-03-06T18:57:49.215 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:49.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:49.221334+0000) 2024-03-06T18:57:50.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:50.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:50.221598+0000) 2024-03-06T18:57:51.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:51.220+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:51.221859+0000) 2024-03-06T18:57:52.216 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:52.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:52.222123+0000) 2024-03-06T18:57:53.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:53.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:53.222382+0000) 2024-03-06T18:57:54.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:54.221+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:54.222663+0000) 2024-03-06T18:57:55.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:55.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:55.222950+0000) 2024-03-06T18:57:56.217 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:56.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:56.223212+0000) 2024-03-06T18:57:57.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:57.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:57.223453+0000) 2024-03-06T18:57:58.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:58.222+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:58.223809+0000) 2024-03-06T18:57:59.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:57:59.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:59.224065+0000) 2024-03-06T18:58:00.218 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:00.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:00.224304+0000) 2024-03-06T18:58:01.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:01.223+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:01.224513+0000) 2024-03-06T18:58:02.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:02.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:02.224813+0000) 2024-03-06T18:58:03.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:03.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:03.225125+0000) 2024-03-06T18:58:04.219 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:04.224+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:04.225393+0000) 2024-03-06T18:58:05.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:05.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:05.225684+0000) 2024-03-06T18:58:06.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:06.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:06.225946+0000) 2024-03-06T18:58:07.220 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:07.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:07.226207+0000) 2024-03-06T18:58:08.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:08.225+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:08.226459+0000) 2024-03-06T18:58:09.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:09.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:09.226788+0000) 2024-03-06T18:58:10.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:10.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:10.226984+0000) 2024-03-06T18:58:11.221 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:11.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:11.227226+0000) 2024-03-06T18:58:12.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:12.226+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:12.227468+0000) 2024-03-06T18:58:13.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:13.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:13.227762+0000) 2024-03-06T18:58:14.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:14.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:14.228019+0000) 2024-03-06T18:58:15.222 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:15.227+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:15.228267+0000) 2024-03-06T18:58:16.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:16.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:16.228521+0000) 2024-03-06T18:58:17.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:17.109 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:17.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:17.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:17.228776+0000) 2024-03-06T18:58:18.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:18.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:18.229024+0000) 2024-03-06T18:58:19.223 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:19.228+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:19.229290+0000) 2024-03-06T18:58:20.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:20.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:20.229526+0000) 2024-03-06T18:58:21.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:21.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:21.229767+0000) 2024-03-06T18:58:22.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:22.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:22.229965+0000) 2024-03-06T18:58:23.224 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:23.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:23.230227+0000) 2024-03-06T18:58:24.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:24.229+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:24.230425+0000) 2024-03-06T18:58:25.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:25.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:25.230719+0000) 2024-03-06T18:58:26.225 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:26.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:26.230979+0000) 2024-03-06T18:58:27.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:27.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:27.231222+0000) 2024-03-06T18:58:28.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:28.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:28.231483+0000) 2024-03-06T18:58:29.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:29.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:29.231755+0000) 2024-03-06T18:58:30.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:30.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:30.232009+0000) 2024-03-06T18:58:31.226 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:31.230+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:31.232204+0000) 2024-03-06T18:58:32.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:32.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:32.232443+0000) 2024-03-06T18:58:33.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:33.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:33.232733+0000) 2024-03-06T18:58:34.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:34.231+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:34.232997+0000) 2024-03-06T18:58:35.227 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:35.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:35.233264+0000) 2024-03-06T18:58:36.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:36.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:36.233527+0000) 2024-03-06T18:58:37.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:37.232+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:37.233779+0000) 2024-03-06T18:58:38.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:38.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:38.234049+0000) 2024-03-06T18:58:39.228 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:39.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:39.234288+0000) 2024-03-06T18:58:40.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:40.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:40.234546+0000) 2024-03-06T18:58:41.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:41.233+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:41.234785+0000) 2024-03-06T18:58:42.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:42.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:42.235035+0000) 2024-03-06T18:58:43.229 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:43.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:43.235294+0000) 2024-03-06T18:58:44.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:44.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:44.235546+0000) 2024-03-06T18:58:45.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:45.234+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:45.235809+0000) 2024-03-06T18:58:46.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:46.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:46.236068+0000) 2024-03-06T18:58:47.230 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:47.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:47.236334+0000) 2024-03-06T18:58:47.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:47.320 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:48.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:48.235+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:48.236580+0000) 2024-03-06T18:58:49.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:49.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:49.236843+0000) 2024-03-06T18:58:50.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:50.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:50.237043+0000) 2024-03-06T18:58:51.231 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:51.236+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:51.237355+0000) 2024-03-06T18:58:52.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:52.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:52.237620+0000) 2024-03-06T18:58:53.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:53.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:53.237919+0000) 2024-03-06T18:58:54.232 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:54.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:54.238184+0000) 2024-03-06T18:58:55.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:55.237+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:55.238435+0000) 2024-03-06T18:58:56.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:56.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:56.238729+0000) 2024-03-06T18:58:57.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:57.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:57.238995+0000) 2024-03-06T18:58:58.233 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:58.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:58.239264+0000) 2024-03-06T18:58:59.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:58:59.238+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:59.239522+0000) 2024-03-06T18:59:00.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:00.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:00.239779+0000) 2024-03-06T18:59:01.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:01.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:01.240036+0000) 2024-03-06T18:59:02.234 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:02.239+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:02.240297+0000) 2024-03-06T18:59:03.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:03.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:03.240537+0000) 2024-03-06T18:59:04.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:04.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:04.240783+0000) 2024-03-06T18:59:05.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:05.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:05.241067+0000) 2024-03-06T18:59:06.235 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:06.240+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:06.241334+0000) 2024-03-06T18:59:07.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:07.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:07.241607+0000) 2024-03-06T18:59:08.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:08.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:08.241841+0000) 2024-03-06T18:59:09.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:09.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:09.242104+0000) 2024-03-06T18:59:10.236 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:10.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:10.242319+0000) 2024-03-06T18:59:11.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:11.241+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:11.242577+0000) 2024-03-06T18:59:12.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:12.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:12.242794+0000) 2024-03-06T18:59:13.237 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:13.242+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:13.243091+0000) 2024-03-06T18:59:14.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:14.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:14.243368+0000) 2024-03-06T18:59:15.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:15.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:15.243610+0000) 2024-03-06T18:59:16.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:16.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:16.243894+0000) 2024-03-06T18:59:17.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:17.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:17.244126+0000) 2024-03-06T18:59:17.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:17.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:18.238 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:18.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:18.244393+0000) 2024-03-06T18:59:19.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:19.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:19.244626+0000) 2024-03-06T18:59:20.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:20.243+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:20.244912+0000) 2024-03-06T18:59:21.239 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:21.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:21.245165+0000) 2024-03-06T18:59:22.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:22.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:22.245424+0000) 2024-03-06T18:59:23.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:23.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:23.245703+0000) 2024-03-06T18:59:24.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:24.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:24.245959+0000) 2024-03-06T18:59:25.240 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:25.244+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:25.246207+0000) 2024-03-06T18:59:26.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:26.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:26.246466+0000) 2024-03-06T18:59:27.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:27.245+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:27.246747+0000) 2024-03-06T18:59:28.241 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:28.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:28.246989+0000) 2024-03-06T18:59:29.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:29.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:29.247233+0000) 2024-03-06T18:59:30.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:30.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:30.247485+0000) 2024-03-06T18:59:31.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:31.246+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:31.247764+0000) 2024-03-06T18:59:32.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:32.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:32.248017+0000) 2024-03-06T18:59:33.242 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:33.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:33.248270+0000) 2024-03-06T18:59:34.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:34.247+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:34.248529+0000) 2024-03-06T18:59:35.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:35.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:35.248808+0000) 2024-03-06T18:59:36.243 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:36.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:36.249061+0000) 2024-03-06T18:59:37.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:37.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:37.249314+0000) 2024-03-06T18:59:38.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:38.248+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:38.249558+0000) 2024-03-06T18:59:39.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:39.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:39.249996+0000) 2024-03-06T18:59:40.244 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:40.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:40.250243+0000) 2024-03-06T18:59:41.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:41.249+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:41.250489+0000) 2024-03-06T18:59:42.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:42.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:42.250727+0000) 2024-03-06T18:59:43.245 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:43.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:43.250975+0000) 2024-03-06T18:59:44.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:44.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:44.251233+0000) 2024-03-06T18:59:45.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:45.250+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:45.251490+0000) 2024-03-06T18:59:46.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:46.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:46.251750+0000) 2024-03-06T18:59:47.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:47.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:47.251984+0000) 2024-03-06T18:59:47.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:47.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:48.246 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:48.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:48.252224+0000) 2024-03-06T18:59:49.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:49.251+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:49.252482+0000) 2024-03-06T18:59:50.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:50.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:50.252740+0000) 2024-03-06T18:59:51.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:51.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:51.252987+0000) 2024-03-06T18:59:52.247 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:52.252+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:52.253242+0000) 2024-03-06T18:59:53.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:53.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:53.253480+0000) 2024-03-06T18:59:54.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:54.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:54.253741+0000) 2024-03-06T18:59:55.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:55.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:55.254007+0000) 2024-03-06T18:59:56.248 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:56.253+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:56.254260+0000) 2024-03-06T18:59:57.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:57.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:57.254506+0000) 2024-03-06T18:59:58.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:58.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:58.254761+0000) 2024-03-06T18:59:59.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T18:59:59.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:59.255022+0000) 2024-03-06T19:00:00.249 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:00.254+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:00.255266+0000) 2024-03-06T19:00:01.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:01.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:01.255503+0000) 2024-03-06T19:00:02.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:02.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:02.255761+0000) 2024-03-06T19:00:03.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:03.255+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:03.256002+0000) 2024-03-06T19:00:04.250 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:04.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:04.256240+0000) 2024-03-06T19:00:05.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:05.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:05.256486+0000) 2024-03-06T19:00:06.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:06.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:06.256750+0000) 2024-03-06T19:00:07.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:07.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:07.256989+0000) 2024-03-06T19:00:08.251 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:08.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:08.257231+0000) 2024-03-06T19:00:09.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:09.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:09.257474+0000) 2024-03-06T19:00:10.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:10.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:10.257755+0000) 2024-03-06T19:00:11.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:11.256+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:11.258012+0000) 2024-03-06T19:00:12.252 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:12.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:12.258248+0000) 2024-03-06T19:00:13.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:13.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:13.258505+0000) 2024-03-06T19:00:14.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:14.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:14.258768+0000) 2024-03-06T19:00:15.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:15.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:15.259035+0000) 2024-03-06T19:00:16.253 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:16.257+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:16.259277+0000) 2024-03-06T19:00:17.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:17.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:17.259518+0000) 2024-03-06T19:00:17.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:17.768 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:18.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:18.258+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:18.259761+0000) 2024-03-06T19:00:19.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:19.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:19.260013+0000) 2024-03-06T19:00:20.254 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:20.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:20.260273+0000) 2024-03-06T19:00:21.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:21.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:21.260526+0000) 2024-03-06T19:00:22.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:22.259+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:22.260793+0000) 2024-03-06T19:00:23.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:23.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:23.261028+0000) 2024-03-06T19:00:24.255 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:24.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:24.261291+0000) 2024-03-06T19:00:25.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:25.260+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:25.261570+0000) 2024-03-06T19:00:26.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:26.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:26.261832+0000) 2024-03-06T19:00:27.256 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:27.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:27.262125+0000) 2024-03-06T19:00:28.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:28.261+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:28.262375+0000) 2024-03-06T19:00:29.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:29.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:29.262659+0000) 2024-03-06T19:00:30.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:30.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:30.262927+0000) 2024-03-06T19:00:31.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:31.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:31.263117+0000) 2024-03-06T19:00:32.257 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:32.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:32.263357+0000) 2024-03-06T19:00:33.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:33.262+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:33.263611+0000) 2024-03-06T19:00:34.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:34.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:34.263894+0000) 2024-03-06T19:00:35.258 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:35.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:35.264145+0000) 2024-03-06T19:00:36.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:36.263+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:36.264386+0000) 2024-03-06T19:00:37.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:37.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:37.264621+0000) 2024-03-06T19:00:38.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:38.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:38.264891+0000) 2024-03-06T19:00:39.259 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:39.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:39.265139+0000) 2024-03-06T19:00:40.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:40.264+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:40.265400+0000) 2024-03-06T19:00:41.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:41.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:41.265690+0000) 2024-03-06T19:00:42.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:42.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:42.265926+0000) 2024-03-06T19:00:43.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:43.265+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:43.266185+0000) 2024-03-06T19:00:44.260 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:44.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:44.266441+0000) 2024-03-06T19:00:45.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:45.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:45.266743+0000) 2024-03-06T19:00:46.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:46.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:46.267003+0000) 2024-03-06T19:00:47.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:47.266+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:47.267230+0000) 2024-03-06T19:00:47.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:47.917 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:48.261 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:48.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:48.267481+0000) 2024-03-06T19:00:49.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:49.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:49.267746+0000) 2024-03-06T19:00:50.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:50.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:50.268008+0000) 2024-03-06T19:00:51.262 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:51.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:51.268243+0000) 2024-03-06T19:00:52.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:52.267+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:52.268481+0000) 2024-03-06T19:00:53.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:53.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:53.268692+0000) 2024-03-06T19:00:54.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:54.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:54.268934+0000) 2024-03-06T19:00:55.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:55.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:55.269179+0000) 2024-03-06T19:00:56.263 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:56.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:56.269384+0000) 2024-03-06T19:00:57.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:57.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:57.269670+0000) 2024-03-06T19:00:58.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:58.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:58.269939+0000) 2024-03-06T19:00:59.264 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:00:59.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:59.270199+0000) 2024-03-06T19:01:00.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:00.268+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:00.270390+0000) 2024-03-06T19:01:01.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:01.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:01.270699+0000) 2024-03-06T19:01:02.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:02.269+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:02.270952+0000) 2024-03-06T19:01:03.265 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:03.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:03.271227+0000) 2024-03-06T19:01:04.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:04.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:04.271486+0000) 2024-03-06T19:01:05.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:05.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:05.271749+0000) 2024-03-06T19:01:06.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:06.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:06.272006+0000) 2024-03-06T19:01:07.266 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:07.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:07.272260+0000) 2024-03-06T19:01:08.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:08.270+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:08.272521+0000) 2024-03-06T19:01:09.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:09.271+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:09.272776+0000) 2024-03-06T19:01:10.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:10.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:10.273037+0000) 2024-03-06T19:01:11.267 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:11.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:11.273295+0000) 2024-03-06T19:01:12.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:12.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:12.273551+0000) 2024-03-06T19:01:13.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:13.272+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:13.273809+0000) 2024-03-06T19:01:14.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:14.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:14.274082+0000) 2024-03-06T19:01:15.268 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:15.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:15.274349+0000) 2024-03-06T19:01:16.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:16.273+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:16.274622+0000) 2024-03-06T19:01:17.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:17.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:17.274962+0000) 2024-03-06T19:01:18.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:18.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:18.269 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:18.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:18.275250+0000) 2024-03-06T19:01:19.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:19.274+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:19.275510+0000) 2024-03-06T19:01:20.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:20.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:20.275820+0000) 2024-03-06T19:01:21.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:21.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:21.276017+0000) 2024-03-06T19:01:22.270 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:22.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:22.276292+0000) 2024-03-06T19:01:23.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:23.275+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:23.276563+0000) 2024-03-06T19:01:24.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:24.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:24.276845+0000) 2024-03-06T19:01:25.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:25.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:25.277128+0000) 2024-03-06T19:01:26.271 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:26.276+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:26.277412+0000) 2024-03-06T19:01:27.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:27.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:27.277696+0000) 2024-03-06T19:01:28.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:28.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:28.277954+0000) 2024-03-06T19:01:29.272 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:29.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:29.278226+0000) 2024-03-06T19:01:30.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:30.277+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:30.278428+0000) 2024-03-06T19:01:31.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:31.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:31.278738+0000) 2024-03-06T19:01:32.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:32.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:32.278992+0000) 2024-03-06T19:01:33.273 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:33.278+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:33.279265+0000) 2024-03-06T19:01:34.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:34.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:34.279521+0000) 2024-03-06T19:01:35.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:35.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:35.279797+0000) 2024-03-06T19:01:36.274 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:36.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:36.280054+0000) 2024-03-06T19:01:37.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:37.279+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:37.280307+0000) 2024-03-06T19:01:38.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:38.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:38.280574+0000) 2024-03-06T19:01:39.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:39.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:39.280839+0000) 2024-03-06T19:01:40.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:40.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:40.281095+0000) 2024-03-06T19:01:41.275 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:41.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:41.281349+0000) 2024-03-06T19:01:42.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:42.280+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:42.281597+0000) 2024-03-06T19:01:43.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:43.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:43.281832+0000) 2024-03-06T19:01:44.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:44.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:44.282116+0000) 2024-03-06T19:01:45.276 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:45.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:45.282379+0000) 2024-03-06T19:01:46.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:46.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:46.282621+0000) 2024-03-06T19:01:47.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:47.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:47.282887+0000) 2024-03-06T19:01:48.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:48.194 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:48.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:48.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:48.283089+0000) 2024-03-06T19:01:49.277 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:49.281+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:49.283325+0000) 2024-03-06T19:01:50.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:50.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:50.283565+0000) 2024-03-06T19:01:51.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:51.282+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:51.283830+0000) 2024-03-06T19:01:52.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:52.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:52.284090+0000) 2024-03-06T19:01:53.278 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:53.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:53.284345+0000) 2024-03-06T19:01:54.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:54.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:54.284575+0000) 2024-03-06T19:01:55.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:55.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:55.284824+0000) 2024-03-06T19:01:56.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:56.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:56.285086+0000) 2024-03-06T19:01:57.279 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:57.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:57.285345+0000) 2024-03-06T19:01:58.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:58.283+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:58.285587+0000) 2024-03-06T19:01:59.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:01:59.284+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:59.285849+0000) 2024-03-06T19:02:00.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:00.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:00.286112+0000) 2024-03-06T19:02:01.280 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:01.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:01.286369+0000) 2024-03-06T19:02:02.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:02.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:02.286616+0000) 2024-03-06T19:02:03.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:03.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:03.286895+0000) 2024-03-06T19:02:04.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:04.285+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:04.287122+0000) 2024-03-06T19:02:05.281 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:05.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:05.287384+0000) 2024-03-06T19:02:06.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:06.286+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:06.287683+0000) 2024-03-06T19:02:07.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:07.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:07.287942+0000) 2024-03-06T19:02:08.282 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:08.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:08.288254+0000) 2024-03-06T19:02:09.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:09.287+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:09.288530+0000) 2024-03-06T19:02:10.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:10.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:10.288813+0000) 2024-03-06T19:02:11.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:11.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:11.289072+0000) 2024-03-06T19:02:12.283 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:12.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:12.289312+0000) 2024-03-06T19:02:13.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:13.288+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:13.289556+0000) 2024-03-06T19:02:14.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:14.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:14.289818+0000) 2024-03-06T19:02:15.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:15.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:15.290061+0000) 2024-03-06T19:02:16.284 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:16.289+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:16.290327+0000) 2024-03-06T19:02:17.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:17.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:17.290597+0000) 2024-03-06T19:02:18.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:18.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:18.290869+0000) 2024-03-06T19:02:18.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:18.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:19.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:19.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:19.291132+0000) 2024-03-06T19:02:20.285 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:20.290+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:20.291397+0000) 2024-03-06T19:02:21.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:21.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:21.291692+0000) 2024-03-06T19:02:22.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:22.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:22.291953+0000) 2024-03-06T19:02:23.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:23.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:23.292189+0000) 2024-03-06T19:02:24.286 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:24.291+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:24.292421+0000) 2024-03-06T19:02:25.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:25.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:25.292706+0000) 2024-03-06T19:02:26.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:26.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:26.292906+0000) 2024-03-06T19:02:27.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:27.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:27.293149+0000) 2024-03-06T19:02:28.287 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:28.292+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:28.293394+0000) 2024-03-06T19:02:29.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:29.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:29.293670+0000) 2024-03-06T19:02:30.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:30.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:30.293935+0000) 2024-03-06T19:02:31.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:31.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:31.294131+0000) 2024-03-06T19:02:32.288 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:32.293+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:32.294359+0000) 2024-03-06T19:02:33.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:33.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:33.294599+0000) 2024-03-06T19:02:34.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:34.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:34.294896+0000) 2024-03-06T19:02:35.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:35.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:35.295159+0000) 2024-03-06T19:02:36.289 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:36.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:36.295400+0000) 2024-03-06T19:02:37.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:37.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:37.295678+0000) 2024-03-06T19:02:38.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:38.294+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:38.295935+0000) 2024-03-06T19:02:39.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:39.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:39.296198+0000) 2024-03-06T19:02:40.290 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:40.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:40.296437+0000) 2024-03-06T19:02:41.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:41.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:41.296737+0000) 2024-03-06T19:02:42.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:42.295+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:42.296980+0000) 2024-03-06T19:02:43.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:43.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:43.297251+0000) 2024-03-06T19:02:44.291 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:44.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:44.297513+0000) 2024-03-06T19:02:45.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:45.296+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:45.297746+0000) 2024-03-06T19:02:46.292 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:46.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:46.297990+0000) 2024-03-06T19:02:47.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:47.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:47.298243+0000) 2024-03-06T19:02:48.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:48.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:48.298502+0000) 2024-03-06T19:02:48.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:48.479 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:49.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:49.297+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:49.298764+0000) 2024-03-06T19:02:50.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:50.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:50.299019+0000) 2024-03-06T19:02:51.293 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:51.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:51.299290+0000) 2024-03-06T19:02:52.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:52.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:52.299552+0000) 2024-03-06T19:02:53.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:53.298+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:53.299793+0000) 2024-03-06T19:02:54.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:54.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:54.300090+0000) 2024-03-06T19:02:55.294 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:55.299+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:55.300360+0000) 2024-03-06T19:02:56.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:56.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:56.300618+0000) 2024-03-06T19:02:57.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:57.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:57.300918+0000) 2024-03-06T19:02:58.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:58.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:58.301177+0000) 2024-03-06T19:02:59.295 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:02:59.300+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:59.301363+0000) 2024-03-06T19:03:00.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:00.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:00.301622+0000) 2024-03-06T19:03:01.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:01.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:01.301919+0000) 2024-03-06T19:03:02.296 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:02.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:02.302164+0000) 2024-03-06T19:03:03.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:03.301+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:03.302411+0000) 2024-03-06T19:03:04.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:04.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:04.302704+0000) 2024-03-06T19:03:05.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:05.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:05.302969+0000) 2024-03-06T19:03:06.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:06.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:06.303218+0000) 2024-03-06T19:03:07.297 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:07.302+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:07.303462+0000) 2024-03-06T19:03:08.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:08.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:08.303742+0000) 2024-03-06T19:03:09.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:09.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:09.303986+0000) 2024-03-06T19:03:10.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:10.303+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:10.304249+0000) 2024-03-06T19:03:11.298 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:11.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:11.304516+0000) 2024-03-06T19:03:12.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:12.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:12.304781+0000) 2024-03-06T19:03:13.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:13.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:13.305038+0000) 2024-03-06T19:03:14.299 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:14.304+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:14.305289+0000) 2024-03-06T19:03:15.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:15.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:15.305541+0000) 2024-03-06T19:03:16.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:16.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:16.305789+0000) 2024-03-06T19:03:17.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:17.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:17.306035+0000) 2024-03-06T19:03:18.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:18.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:18.306298+0000) 2024-03-06T19:03:18.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:18.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:19.300 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:19.305+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:19.306543+0000) 2024-03-06T19:03:20.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:20.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:20.306875+0000) 2024-03-06T19:03:21.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:21.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:21.307154+0000) 2024-03-06T19:03:22.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:22.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:22.307352+0000) 2024-03-06T19:03:23.301 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:23.306+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:23.307549+0000) 2024-03-06T19:03:24.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:24.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:24.307806+0000) 2024-03-06T19:03:25.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:25.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:25.308053+0000) 2024-03-06T19:03:26.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:26.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:26.308300+0000) 2024-03-06T19:03:27.302 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:27.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:27.308542+0000) 2024-03-06T19:03:28.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:28.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:28.308785+0000) 2024-03-06T19:03:29.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:29.307+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:29.309049+0000) 2024-03-06T19:03:30.303 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:30.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:30.309284+0000) 2024-03-06T19:03:31.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:31.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:31.309532+0000) 2024-03-06T19:03:32.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:32.308+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:32.309783+0000) 2024-03-06T19:03:33.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:33.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:33.310044+0000) 2024-03-06T19:03:34.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:34.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:34.310316+0000) 2024-03-06T19:03:35.304 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:35.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:35.310581+0000) 2024-03-06T19:03:36.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:36.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:36.310840+0000) 2024-03-06T19:03:37.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:37.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:37.311087+0000) 2024-03-06T19:03:38.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:38.309+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:38.311278+0000) 2024-03-06T19:03:39.305 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:39.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:39.311526+0000) 2024-03-06T19:03:40.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:40.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:40.311780+0000) 2024-03-06T19:03:41.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:41.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:41.312046+0000) 2024-03-06T19:03:42.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:42.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:42.312308+0000) 2024-03-06T19:03:43.306 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:43.310+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:43.312496+0000) 2024-03-06T19:03:44.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:44.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:44.312873+0000) 2024-03-06T19:03:45.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:45.311+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:45.313123+0000) 2024-03-06T19:03:46.307 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:46.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:46.313348+0000) 2024-03-06T19:03:47.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:47.312+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:47.313590+0000) 2024-03-06T19:03:48.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:48.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:48.313897+0000) 2024-03-06T19:03:48.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:48.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:49.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:49.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:49.314141+0000) 2024-03-06T19:03:50.308 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:50.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:50.314345+0000) 2024-03-06T19:03:51.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:51.313+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:51.314607+0000) 2024-03-06T19:03:52.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:52.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:52.314846+0000) 2024-03-06T19:03:53.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:53.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:53.315080+0000) 2024-03-06T19:03:54.309 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:54.314+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:54.315327+0000) 2024-03-06T19:03:55.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:55.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:55.315603+0000) 2024-03-06T19:03:56.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:56.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:56.315890+0000) 2024-03-06T19:03:57.310 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:57.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:57.316144+0000) 2024-03-06T19:03:58.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:58.315+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:58.316406+0000) 2024-03-06T19:03:59.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:03:59.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:59.316670+0000) 2024-03-06T19:04:00.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:00.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:00.316914+0000) 2024-03-06T19:04:01.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:01.316+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:01.317198+0000) 2024-03-06T19:04:02.311 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:02.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:02.317466+0000) 2024-03-06T19:04:03.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:03.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:03.317749+0000) 2024-03-06T19:04:04.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:04.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:04.317961+0000) 2024-03-06T19:04:05.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:05.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:05.318244+0000) 2024-03-06T19:04:06.312 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:06.317+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:06.318494+0000) 2024-03-06T19:04:07.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:07.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:07.318765+0000) 2024-03-06T19:04:08.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:08.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:08.319034+0000) 2024-03-06T19:04:09.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:09.318+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:09.319220+0000) 2024-03-06T19:04:10.313 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:10.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:10.319456+0000) 2024-03-06T19:04:11.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:11.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:11.319763+0000) 2024-03-06T19:04:12.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:12.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:12.320004+0000) 2024-03-06T19:04:13.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:13.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:13.320184+0000) 2024-03-06T19:04:14.314 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:14.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:14.320439+0000) 2024-03-06T19:04:15.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:15.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:15.320715+0000) 2024-03-06T19:04:16.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:16.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:16.321002+0000) 2024-03-06T19:04:17.315 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:17.319+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:17.321189+0000) 2024-03-06T19:04:18.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:18.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:18.321455+0000) 2024-03-06T19:04:18.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:18.935 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:19.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:19.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:19.321743+0000) 2024-03-06T19:04:20.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:20.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:20.321997+0000) 2024-03-06T19:04:21.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:21.320+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:21.322232+0000) 2024-03-06T19:04:22.316 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:22.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:22.322425+0000) 2024-03-06T19:04:23.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:23.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:23.322712+0000) 2024-03-06T19:04:24.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:24.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:24.322986+0000) 2024-03-06T19:04:25.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:25.321+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:25.323241+0000) 2024-03-06T19:04:26.317 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:26.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:26.323521+0000) 2024-03-06T19:04:27.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:27.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:27.323771+0000) 2024-03-06T19:04:28.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:28.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:28.324022+0000) 2024-03-06T19:04:29.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:29.322+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:29.324271+0000) 2024-03-06T19:04:30.318 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:30.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:30.324535+0000) 2024-03-06T19:04:31.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:31.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:31.324823+0000) 2024-03-06T19:04:32.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:32.323+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:32.325024+0000) 2024-03-06T19:04:33.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:33.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:33.325280+0000) 2024-03-06T19:04:34.319 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:34.324+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:34.325538+0000) 2024-03-06T19:04:35.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:35.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:35.325803+0000) 2024-03-06T19:04:36.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:36.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:36.326053+0000) 2024-03-06T19:04:37.320 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:37.325+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:37.326310+0000) 2024-03-06T19:04:38.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:38.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:38.326678+0000) 2024-03-06T19:04:39.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:39.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:39.326935+0000) 2024-03-06T19:04:40.321 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:40.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:40.327188+0000) 2024-03-06T19:04:41.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:41.326+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:41.327434+0000) 2024-03-06T19:04:42.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:42.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:42.327721+0000) 2024-03-06T19:04:43.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:43.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:43.327978+0000) 2024-03-06T19:04:44.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:44.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:44.328238+0000) 2024-03-06T19:04:45.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:45.327+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:45.328489+0000) 2024-03-06T19:04:46.322 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:46.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:46.328693+0000) 2024-03-06T19:04:47.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:47.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:47.328914+0000) 2024-03-06T19:04:48.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:48.328+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:48.329218+0000) 2024-03-06T19:04:49.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:49.075 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:49.323 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:49.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:49.329504+0000) 2024-03-06T19:04:50.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:50.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:50.329761+0000) 2024-03-06T19:04:51.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:51.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:51.330035+0000) 2024-03-06T19:04:52.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:52.329+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:52.330247+0000) 2024-03-06T19:04:53.324 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:53.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:53.330506+0000) 2024-03-06T19:04:54.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:54.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:54.330761+0000) 2024-03-06T19:04:55.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:55.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:55.331003+0000) 2024-03-06T19:04:56.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:56.330+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:56.331276+0000) 2024-03-06T19:04:57.325 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:57.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:57.331511+0000) 2024-03-06T19:04:58.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:58.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:58.331754+0000) 2024-03-06T19:04:59.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:04:59.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:59.332005+0000) 2024-03-06T19:05:00.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:00.331+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:00.332253+0000) 2024-03-06T19:05:01.326 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:01.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:01.332498+0000) 2024-03-06T19:05:02.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:02.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:02.332757+0000) 2024-03-06T19:05:03.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:03.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:03.332989+0000) 2024-03-06T19:05:04.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:04.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:04.333256+0000) 2024-03-06T19:05:05.327 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:05.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:05.333522+0000) 2024-03-06T19:05:06.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:06.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:06.333839+0000) 2024-03-06T19:05:07.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:07.332+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:07.334094+0000) 2024-03-06T19:05:08.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:08.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:08.334350+0000) 2024-03-06T19:05:09.328 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:09.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:09.334607+0000) 2024-03-06T19:05:10.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:10.333+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:10.334902+0000) 2024-03-06T19:05:11.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:11.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:11.335181+0000) 2024-03-06T19:05:12.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:12.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:12.335445+0000) 2024-03-06T19:05:13.329 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:13.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:13.335736+0000) 2024-03-06T19:05:14.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:14.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:14.335993+0000) 2024-03-06T19:05:15.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:15.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:15.336251+0000) 2024-03-06T19:05:16.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:16.334+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:16.336457+0000) 2024-03-06T19:05:17.330 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:17.335+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:17.336738+0000) 2024-03-06T19:05:18.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:18.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:18.336977+0000) 2024-03-06T19:05:19.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:19.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:19.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:19.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:19.337249+0000) 2024-03-06T19:05:20.331 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:20.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:20.337512+0000) 2024-03-06T19:05:21.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:21.336+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:21.337771+0000) 2024-03-06T19:05:22.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:22.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:22.338037+0000) 2024-03-06T19:05:23.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:23.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:23.338282+0000) 2024-03-06T19:05:24.332 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:24.337+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:24.338541+0000) 2024-03-06T19:05:25.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:25.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:25.338823+0000) 2024-03-06T19:05:26.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:26.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:26.339129+0000) 2024-03-06T19:05:27.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:27.338+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:27.339392+0000) 2024-03-06T19:05:28.333 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:28.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:28.339623+0000) 2024-03-06T19:05:29.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:29.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:29.339873+0000) 2024-03-06T19:05:30.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:30.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:30.340098+0000) 2024-03-06T19:05:31.334 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:31.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:31.340344+0000) 2024-03-06T19:05:32.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:32.339+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:32.340586+0000) 2024-03-06T19:05:33.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:33.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:33.340839+0000) 2024-03-06T19:05:34.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:34.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:34.341090+0000) 2024-03-06T19:05:35.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:35.340+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:35.341335+0000) 2024-03-06T19:05:36.335 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:36.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:36.341596+0000) 2024-03-06T19:05:37.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:37.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:37.341861+0000) 2024-03-06T19:05:38.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:38.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:38.342056+0000) 2024-03-06T19:05:39.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:39.341+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:39.342309+0000) 2024-03-06T19:05:40.336 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:40.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:40.342546+0000) 2024-03-06T19:05:41.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:41.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:41.342805+0000) 2024-03-06T19:05:42.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:42.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:42.343074+0000) 2024-03-06T19:05:43.337 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:43.342+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:43.343311+0000) 2024-03-06T19:05:44.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:44.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:44.343581+0000) 2024-03-06T19:05:45.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:45.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:45.343845+0000) 2024-03-06T19:05:46.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:46.343+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:46.344109+0000) 2024-03-06T19:05:47.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:47.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:47.344368+0000) 2024-03-06T19:05:48.338 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:48.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:48.344665+0000) 2024-03-06T19:05:49.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:49.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:49.344913+0000) 2024-03-06T19:05:49.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:49.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:50.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:50.344+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:50.345168+0000) 2024-03-06T19:05:51.339 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:51.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:51.345420+0000) 2024-03-06T19:05:52.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:52.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:52.345702+0000) 2024-03-06T19:05:53.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:53.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:53.345964+0000) 2024-03-06T19:05:54.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:54.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:54.346209+0000) 2024-03-06T19:05:55.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:55.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:55.346453+0000) 2024-03-06T19:05:56.340 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:56.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:56.346750+0000) 2024-03-06T19:05:57.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:57.345+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:57.346951+0000) 2024-03-06T19:05:58.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:58.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:58.347190+0000) 2024-03-06T19:05:59.341 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:05:59.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:59.347447+0000) 2024-03-06T19:06:00.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:00.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:00.347745+0000) 2024-03-06T19:06:01.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:01.346+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:01.348202+0000) 2024-03-06T19:06:02.342 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:02.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:02.348461+0000) 2024-03-06T19:06:03.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:03.347+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:03.348752+0000) 2024-03-06T19:06:04.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:04.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:04.349018+0000) 2024-03-06T19:06:05.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:05.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:05.349282+0000) 2024-03-06T19:06:06.343 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:06.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:06.349544+0000) 2024-03-06T19:06:07.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:07.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:07.349806+0000) 2024-03-06T19:06:08.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:08.348+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:08.350132+0000) 2024-03-06T19:06:09.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:09.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:09.350407+0000) 2024-03-06T19:06:10.344 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:10.349+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:10.350709+0000) 2024-03-06T19:06:11.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:11.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:11.350956+0000) 2024-03-06T19:06:12.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:12.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:12.351200+0000) 2024-03-06T19:06:13.345 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:13.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:13.351440+0000) 2024-03-06T19:06:14.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:14.350+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:14.351736+0000) 2024-03-06T19:06:15.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:15.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:15.351924+0000) 2024-03-06T19:06:16.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:16.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:16.352188+0000) 2024-03-06T19:06:17.346 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:17.351+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:17.352450+0000) 2024-03-06T19:06:18.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:18.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:18.352761+0000) 2024-03-06T19:06:19.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:19.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:19.353088+0000) 2024-03-06T19:06:19.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:19.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:20.347 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:20.352+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:20.353369+0000) 2024-03-06T19:06:21.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:21.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:21.353683+0000) 2024-03-06T19:06:22.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:22.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:22.353904+0000) 2024-03-06T19:06:23.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:23.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:23.354159+0000) 2024-03-06T19:06:24.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:24.353+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:24.354431+0000) 2024-03-06T19:06:25.348 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:25.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:25.354744+0000) 2024-03-06T19:06:26.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:26.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:26.355029+0000) 2024-03-06T19:06:27.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:27.354+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:27.355288+0000) 2024-03-06T19:06:28.349 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:28.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:28.355562+0000) 2024-03-06T19:06:29.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:29.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:29.355836+0000) 2024-03-06T19:06:30.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:30.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:30.356116+0000) 2024-03-06T19:06:31.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:31.355+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:31.356386+0000) 2024-03-06T19:06:32.350 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:32.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:32.356672+0000) 2024-03-06T19:06:33.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:33.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:33.356916+0000) 2024-03-06T19:06:34.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:34.356+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:34.357182+0000) 2024-03-06T19:06:35.351 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:35.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:35.357450+0000) 2024-03-06T19:06:36.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:36.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:36.357756+0000) 2024-03-06T19:06:37.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:37.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:37.357987+0000) 2024-03-06T19:06:38.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:38.357+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:38.358222+0000) 2024-03-06T19:06:39.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:39.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:39.358458+0000) 2024-03-06T19:06:40.352 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:40.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:40.358760+0000) 2024-03-06T19:06:41.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:41.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:41.359008+0000) 2024-03-06T19:06:42.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:42.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:42.359195+0000) 2024-03-06T19:06:43.353 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:43.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:43.359453+0000) 2024-03-06T19:06:44.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:44.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:44.359715+0000) 2024-03-06T19:06:45.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:45.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:45.359964+0000) 2024-03-06T19:06:46.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:46.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:46.360179+0000) 2024-03-06T19:06:47.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:47.358+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:47.360407+0000) 2024-03-06T19:06:48.354 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:48.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:48.360721+0000) 2024-03-06T19:06:49.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:49.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:49.360954+0000) 2024-03-06T19:06:49.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:49.644 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:50.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:50.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:50.361192+0000) 2024-03-06T19:06:51.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:51.359+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:51.361442+0000) 2024-03-06T19:06:52.355 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:52.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:52.361732+0000) 2024-03-06T19:06:53.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:53.360+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:53.361918+0000) 2024-03-06T19:06:54.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:54.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:54.362175+0000) 2024-03-06T19:06:55.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:55.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:55.362425+0000) 2024-03-06T19:06:56.356 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:56.361+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:56.362701+0000) 2024-03-06T19:06:57.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:57.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:57.362958+0000) 2024-03-06T19:06:58.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:58.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:58.363200+0000) 2024-03-06T19:06:59.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:06:59.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:59.363449+0000) 2024-03-06T19:07:00.357 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:00.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:00.363743+0000) 2024-03-06T19:07:01.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:01.362+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:01.363945+0000) 2024-03-06T19:07:02.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:02.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:02.364203+0000) 2024-03-06T19:07:03.358 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:03.363+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:03.364455+0000) 2024-03-06T19:07:04.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:04.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:04.364748+0000) 2024-03-06T19:07:05.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:05.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:05.364992+0000) 2024-03-06T19:07:06.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:06.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:06.365256+0000) 2024-03-06T19:07:07.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:07.364+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:07.365550+0000) 2024-03-06T19:07:08.359 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:08.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:08.365811+0000) 2024-03-06T19:07:09.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:09.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:09.366059+0000) 2024-03-06T19:07:10.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:10.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:10.366311+0000) 2024-03-06T19:07:11.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:11.365+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:11.366515+0000) 2024-03-06T19:07:12.360 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:12.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:12.366777+0000) 2024-03-06T19:07:13.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:13.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:13.367027+0000) 2024-03-06T19:07:14.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:14.366+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:14.367276+0000) 2024-03-06T19:07:15.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:15.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:15.367534+0000) 2024-03-06T19:07:16.361 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:16.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:16.367785+0000) 2024-03-06T19:07:17.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:17.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:17.368045+0000) 2024-03-06T19:07:18.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:18.367+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:18.368280+0000) 2024-03-06T19:07:19.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:19.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:19.368536+0000) 2024-03-06T19:07:19.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:19.800 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:20.362 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:20.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:20.368769+0000) 2024-03-06T19:07:21.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:21.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:21.369012+0000) 2024-03-06T19:07:22.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:22.368+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:22.369262+0000) 2024-03-06T19:07:23.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:23.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:23.369511+0000) 2024-03-06T19:07:24.363 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:24.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:24.369749+0000) 2024-03-06T19:07:25.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:25.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:25.370010+0000) 2024-03-06T19:07:26.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:26.369+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:26.370251+0000) 2024-03-06T19:07:27.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:27.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:27.370517+0000) 2024-03-06T19:07:28.364 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:28.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:28.370814+0000) 2024-03-06T19:07:29.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:29.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:29.371081+0000) 2024-03-06T19:07:30.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:30.370+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:30.371313+0000) 2024-03-06T19:07:31.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:31.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:31.371534+0000) 2024-03-06T19:07:32.365 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:32.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:32.371773+0000) 2024-03-06T19:07:33.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:33.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:33.372024+0000) 2024-03-06T19:07:34.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:34.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:34.372290+0000) 2024-03-06T19:07:35.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:35.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:35.372536+0000) 2024-03-06T19:07:36.366 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:36.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:36.372806+0000) 2024-03-06T19:07:37.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:37.371+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:37.373050+0000) 2024-03-06T19:07:38.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:38.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:38.373316+0000) 2024-03-06T19:07:39.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:39.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:39.373556+0000) 2024-03-06T19:07:40.367 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:40.372+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:40.373791+0000) 2024-03-06T19:07:41.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:41.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:41.374042+0000) 2024-03-06T19:07:42.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:42.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:42.374316+0000) 2024-03-06T19:07:43.368 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:43.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:43.374568+0000) 2024-03-06T19:07:44.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:44.373+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:44.374817+0000) 2024-03-06T19:07:45.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:45.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:45.375080+0000) 2024-03-06T19:07:46.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:46.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:46.375340+0000) 2024-03-06T19:07:47.369 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:47.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:47.375597+0000) 2024-03-06T19:07:48.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:48.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:48.375848+0000) 2024-03-06T19:07:49.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:49.374+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:49.376044+0000) 2024-03-06T19:07:50.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:50.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:50.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:50.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:50.376290+0000) 2024-03-06T19:07:51.370 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:51.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:51.376525+0000) 2024-03-06T19:07:52.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:52.375+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:52.376775+0000) 2024-03-06T19:07:53.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:53.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:53.377033+0000) 2024-03-06T19:07:54.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:54.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:54.377263+0000) 2024-03-06T19:07:55.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:55.376+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:55.377579+0000) 2024-03-06T19:07:56.371 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:56.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:56.377809+0000) 2024-03-06T19:07:57.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:57.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:57.378066+0000) 2024-03-06T19:07:58.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:58.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:58.378310+0000) 2024-03-06T19:07:59.372 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:07:59.377+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:59.378559+0000) 2024-03-06T19:08:00.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:00.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:00.378815+0000) 2024-03-06T19:08:01.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:01.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:01.378985+0000) 2024-03-06T19:08:02.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:02.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:02.379246+0000) 2024-03-06T19:08:03.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:03.378+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:03.379495+0000) 2024-03-06T19:08:04.373 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:04.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:04.379754+0000) 2024-03-06T19:08:05.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:05.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:05.380017+0000) 2024-03-06T19:08:06.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:06.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:06.380280+0000) 2024-03-06T19:08:07.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:07.379+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:07.380486+0000) 2024-03-06T19:08:08.374 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:08.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:08.380748+0000) 2024-03-06T19:08:09.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:09.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:09.380983+0000) 2024-03-06T19:08:10.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:10.380+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:10.381234+0000) 2024-03-06T19:08:11.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:11.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:11.381494+0000) 2024-03-06T19:08:12.375 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:12.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:12.381763+0000) 2024-03-06T19:08:13.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:13.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:13.382011+0000) 2024-03-06T19:08:14.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:14.381+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:14.382243+0000) 2024-03-06T19:08:15.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:15.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:15.382502+0000) 2024-03-06T19:08:16.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:16.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:16.382745+0000) 2024-03-06T19:08:17.376 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:17.382+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:17.383011+0000) 2024-03-06T19:08:18.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:18.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:18.383271+0000) 2024-03-06T19:08:19.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:19.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:19.383524+0000) 2024-03-06T19:08:20.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:20.201 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:20.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:20.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:20.383776+0000) 2024-03-06T19:08:21.377 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:21.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:21.384015+0000) 2024-03-06T19:08:22.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:22.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:22.384268+0000) 2024-03-06T19:08:23.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:23.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:23.384524+0000) 2024-03-06T19:08:24.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:24.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:24.384780+0000) 2024-03-06T19:08:25.378 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:25.383+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:25.385039+0000) 2024-03-06T19:08:26.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:26.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:26.385302+0000) 2024-03-06T19:08:27.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:27.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:27.385569+0000) 2024-03-06T19:08:28.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:28.384+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:28.385827+0000) 2024-03-06T19:08:29.379 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:29.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:29.386063+0000) 2024-03-06T19:08:30.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:30.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:30.386315+0000) 2024-03-06T19:08:31.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:31.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:31.386548+0000) 2024-03-06T19:08:32.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:32.385+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:32.386792+0000) 2024-03-06T19:08:33.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:33.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:33.387046+0000) 2024-03-06T19:08:34.380 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:34.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:34.387316+0000) 2024-03-06T19:08:35.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:35.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:35.387561+0000) 2024-03-06T19:08:36.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:36.386+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:36.387801+0000) 2024-03-06T19:08:37.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:37.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:37.388059+0000) 2024-03-06T19:08:38.381 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:38.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:38.388321+0000) 2024-03-06T19:08:39.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:39.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:39.388553+0000) 2024-03-06T19:08:40.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:40.387+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:40.388785+0000) 2024-03-06T19:08:41.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:41.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:41.389039+0000) 2024-03-06T19:08:42.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:42.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:42.389281+0000) 2024-03-06T19:08:43.382 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:43.388+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:43.389520+0000) 2024-03-06T19:08:44.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:44.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:44.389772+0000) 2024-03-06T19:08:45.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:45.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:45.390034+0000) 2024-03-06T19:08:46.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:46.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:46.390291+0000) 2024-03-06T19:08:47.383 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:47.389+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:47.390558+0000) 2024-03-06T19:08:48.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:48.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:48.390819+0000) 2024-03-06T19:08:49.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:49.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:49.391071+0000) 2024-03-06T19:08:50.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:50.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:50.384 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:50.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:50.391265+0000) 2024-03-06T19:08:51.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:51.390+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:51.391497+0000) 2024-03-06T19:08:52.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:52.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:52.391753+0000) 2024-03-06T19:08:53.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:53.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:53.392010+0000) 2024-03-06T19:08:54.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:54.391+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:54.392255+0000) 2024-03-06T19:08:55.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:55.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:55.392536+0000) 2024-03-06T19:08:56.385 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:56.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:56.392797+0000) 2024-03-06T19:08:57.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:57.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:57.393080+0000) 2024-03-06T19:08:58.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:58.392+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:58.393360+0000) 2024-03-06T19:08:59.386 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:08:59.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:59.393666+0000) 2024-03-06T19:09:00.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:00.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:00.393920+0000) 2024-03-06T19:09:01.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:01.393+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:01.394174+0000) 2024-03-06T19:09:02.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:02.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:02.394421+0000) 2024-03-06T19:09:03.387 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:03.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:03.394708+0000) 2024-03-06T19:09:04.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:04.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:04.394968+0000) 2024-03-06T19:09:05.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:05.394+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:05.395238+0000) 2024-03-06T19:09:06.388 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:06.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:06.395478+0000) 2024-03-06T19:09:07.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:07.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:07.395745+0000) 2024-03-06T19:09:08.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:08.395+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:08.396003+0000) 2024-03-06T19:09:09.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:09.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:09.396262+0000) 2024-03-06T19:09:10.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:10.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:10.396522+0000) 2024-03-06T19:09:11.389 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:11.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:11.396775+0000) 2024-03-06T19:09:12.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:12.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:12.396986+0000) 2024-03-06T19:09:13.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:13.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:13.397281+0000) 2024-03-06T19:09:14.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:14.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:14.397495+0000) 2024-03-06T19:09:15.390 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:15.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:15.397786+0000) 2024-03-06T19:09:16.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:16.396+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:16.398061+0000) 2024-03-06T19:09:17.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:17.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:17.398344+0000) 2024-03-06T19:09:18.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:18.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:18.398606+0000) 2024-03-06T19:09:19.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:19.397+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:19.398844+0000) 2024-03-06T19:09:20.391 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:20.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:20.399109+0000) 2024-03-06T19:09:20.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:20.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:21.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:21.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:21.399396+0000) 2024-03-06T19:09:22.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:22.398+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:22.399716+0000) 2024-03-06T19:09:23.392 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:23.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:23.399993+0000) 2024-03-06T19:09:24.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:24.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:24.400267+0000) 2024-03-06T19:09:25.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:25.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:25.400551+0000) 2024-03-06T19:09:26.393 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:26.399+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:26.400825+0000) 2024-03-06T19:09:27.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:27.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:27.401110+0000) 2024-03-06T19:09:28.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:28.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:28.401368+0000) 2024-03-06T19:09:29.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:29.400+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:29.401680+0000) 2024-03-06T19:09:30.394 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:30.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:30.401959+0000) 2024-03-06T19:09:31.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:31.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:31.402224+0000) 2024-03-06T19:09:32.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:32.401+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:32.402503+0000) 2024-03-06T19:09:33.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:33.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:33.402756+0000) 2024-03-06T19:09:34.395 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:34.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:34.403016+0000) 2024-03-06T19:09:35.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:35.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:35.403262+0000) 2024-03-06T19:09:36.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:36.402+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:36.403522+0000) 2024-03-06T19:09:37.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:37.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:37.403758+0000) 2024-03-06T19:09:38.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:38.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:38.404030+0000) 2024-03-06T19:09:39.396 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:39.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:39.404273+0000) 2024-03-06T19:09:40.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:40.403+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:40.404527+0000) 2024-03-06T19:09:41.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:41.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:41.404764+0000) 2024-03-06T19:09:42.397 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:42.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:42.405096+0000) 2024-03-06T19:09:43.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:43.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:43.405354+0000) 2024-03-06T19:09:44.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:44.404+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:44.405600+0000) 2024-03-06T19:09:45.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:45.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:45.405892+0000) 2024-03-06T19:09:46.398 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:46.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:46.406137+0000) 2024-03-06T19:09:47.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:47.405+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:47.406387+0000) 2024-03-06T19:09:48.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:48.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:48.406672+0000) 2024-03-06T19:09:49.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:49.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:49.406919+0000) 2024-03-06T19:09:50.399 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:50.406+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:50.407170+0000) 2024-03-06T19:09:50.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:50.642 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:51.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:51.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:51.407426+0000) 2024-03-06T19:09:52.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:52.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:52.407743+0000) 2024-03-06T19:09:53.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:53.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:53.407943+0000) 2024-03-06T19:09:54.400 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:54.407+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:54.408200+0000) 2024-03-06T19:09:55.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:55.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:55.408461+0000) 2024-03-06T19:09:56.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:56.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:56.408756+0000) 2024-03-06T19:09:57.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:57.408+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:57.408999+0000) 2024-03-06T19:09:58.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:58.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:58.409258+0000) 2024-03-06T19:09:59.401 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:09:59.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:59.409517+0000) 2024-03-06T19:10:00.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:00.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:00.409764+0000) 2024-03-06T19:10:01.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:01.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:01.409987+0000) 2024-03-06T19:10:02.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:02.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:02.410227+0000) 2024-03-06T19:10:03.402 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:03.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:03.410497+0000) 2024-03-06T19:10:04.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:04.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:04.410750+0000) 2024-03-06T19:10:05.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:05.409+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:05.411015+0000) 2024-03-06T19:10:06.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:06.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:06.411320+0000) 2024-03-06T19:10:07.403 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:07.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:07.411606+0000) 2024-03-06T19:10:08.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:08.410+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:08.411948+0000) 2024-03-06T19:10:09.404 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:09.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:09.412254+0000) 2024-03-06T19:10:10.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:10.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:10.412570+0000) 2024-03-06T19:10:11.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:11.411+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:11.412870+0000) 2024-03-06T19:10:12.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:12.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:12.413179+0000) 2024-03-06T19:10:13.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:13.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:13.413464+0000) 2024-03-06T19:10:14.405 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:14.412+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:14.413793+0000) 2024-03-06T19:10:15.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:15.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:15.414106+0000) 2024-03-06T19:10:16.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:16.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:16.414407+0000) 2024-03-06T19:10:17.406 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:17.413+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:17.414746+0000) 2024-03-06T19:10:18.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:18.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:18.415031+0000) 2024-03-06T19:10:19.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:19.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:19.415322+0000) 2024-03-06T19:10:20.407 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:20.414+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:20.415667+0000) 2024-03-06T19:10:20.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:20.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:21.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:21.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:21.415981+0000) 2024-03-06T19:10:22.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:22.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:22.416272+0000) 2024-03-06T19:10:23.408 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:23.415+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:23.416589+0000) 2024-03-06T19:10:24.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:24.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:24.416998+0000) 2024-03-06T19:10:25.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:25.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:25.417262+0000) 2024-03-06T19:10:26.409 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:26.416+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:26.417570+0000) 2024-03-06T19:10:27.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:27.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:27.417832+0000) 2024-03-06T19:10:28.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:28.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:28.418101+0000) 2024-03-06T19:10:29.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:29.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:29.418360+0000) 2024-03-06T19:10:30.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:30.417+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:30.418549+0000) 2024-03-06T19:10:31.410 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:31.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:31.418786+0000) 2024-03-06T19:10:32.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:32.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:32.419052+0000) 2024-03-06T19:10:33.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:33.418+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:33.419294+0000) 2024-03-06T19:10:34.411 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:34.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:34.419543+0000) 2024-03-06T19:10:35.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:35.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:35.419800+0000) 2024-03-06T19:10:36.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:36.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:36.420042+0000) 2024-03-06T19:10:37.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:37.419+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:37.420266+0000) 2024-03-06T19:10:38.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:38.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:38.420526+0000) 2024-03-06T19:10:39.412 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:39.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:39.420779+0000) 2024-03-06T19:10:40.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:40.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:40.421024+0000) 2024-03-06T19:10:41.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:41.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:41.421211+0000) 2024-03-06T19:10:42.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:42.420+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:42.421459+0000) 2024-03-06T19:10:43.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:43.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:43.421731+0000) 2024-03-06T19:10:44.413 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:44.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:44.421981+0000) 2024-03-06T19:10:45.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:45.421+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:45.422239+0000) 2024-03-06T19:10:46.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:46.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:46.422510+0000) 2024-03-06T19:10:47.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:47.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:47.422762+0000) 2024-03-06T19:10:48.414 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:48.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:48.423027+0000) 2024-03-06T19:10:49.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:49.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:49.423303+0000) 2024-03-06T19:10:50.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:50.422+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:50.423574+0000) 2024-03-06T19:10:50.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:50.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:51.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:51.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:51.423840+0000) 2024-03-06T19:10:52.415 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:52.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:52.424123+0000) 2024-03-06T19:10:53.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:53.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:53.424387+0000) 2024-03-06T19:10:54.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:54.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:54.424672+0000) 2024-03-06T19:10:55.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:55.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:55.424951+0000) 2024-03-06T19:10:56.416 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:56.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:56.425238+0000) 2024-03-06T19:10:57.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:57.423+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:57.425512+0000) 2024-03-06T19:10:58.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:58.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:58.425781+0000) 2024-03-06T19:10:59.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:10:59.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:59.426031+0000) 2024-03-06T19:11:00.417 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:00.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:00.426218+0000) 2024-03-06T19:11:01.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:01.424+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:01.426447+0000) 2024-03-06T19:11:02.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:02.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:02.426738+0000) 2024-03-06T19:11:03.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:03.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:03.427005+0000) 2024-03-06T19:11:04.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:04.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:04.427190+0000) 2024-03-06T19:11:05.418 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:05.425+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:05.427382+0000) 2024-03-06T19:11:06.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:06.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:06.427661+0000) 2024-03-06T19:11:07.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:07.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:07.427899+0000) 2024-03-06T19:11:08.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:08.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:08.428101+0000) 2024-03-06T19:11:09.419 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:09.426+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:09.428287+0000) 2024-03-06T19:11:10.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:10.427+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:10.428468+0000) 2024-03-06T19:11:11.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:11.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:11.428728+0000) 2024-03-06T19:11:12.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:12.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:12.429007+0000) 2024-03-06T19:11:13.420 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:13.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:13.429197+0000) 2024-03-06T19:11:14.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:14.428+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:14.429458+0000) 2024-03-06T19:11:15.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:15.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:15.429738+0000) 2024-03-06T19:11:16.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:16.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:16.429992+0000) 2024-03-06T19:11:17.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:17.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:17.430242+0000) 2024-03-06T19:11:18.421 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:18.429+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:18.430497+0000) 2024-03-06T19:11:19.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:19.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:19.430754+0000) 2024-03-06T19:11:20.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:20.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:20.430978+0000) 2024-03-06T19:11:21.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:21.080 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:21.422 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:21.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:21.431240+0000) 2024-03-06T19:11:22.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:22.430+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:22.431486+0000) 2024-03-06T19:11:23.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:23.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:23.431745+0000) 2024-03-06T19:11:24.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:24.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:24.432002+0000) 2024-03-06T19:11:25.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:25.431+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:25.432245+0000) 2024-03-06T19:11:26.423 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:26.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:26.432499+0000) 2024-03-06T19:11:27.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:27.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:27.432773+0000) 2024-03-06T19:11:28.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:28.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:28.433020+0000) 2024-03-06T19:11:29.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:29.432+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:29.433273+0000) 2024-03-06T19:11:30.424 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:30.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:30.433506+0000) 2024-03-06T19:11:31.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:31.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:31.433753+0000) 2024-03-06T19:11:32.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:32.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:32.433951+0000) 2024-03-06T19:11:33.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:33.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:33.434145+0000) 2024-03-06T19:11:34.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:34.433+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:34.434337+0000) 2024-03-06T19:11:35.425 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:35.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:35.434587+0000) 2024-03-06T19:11:36.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:36.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:36.434831+0000) 2024-03-06T19:11:37.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:37.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:37.435023+0000) 2024-03-06T19:11:38.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:38.434+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:38.435221+0000) 2024-03-06T19:11:39.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:39.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:39.435474+0000) 2024-03-06T19:11:40.426 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:40.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:40.435735+0000) 2024-03-06T19:11:41.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:41.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:41.435971+0000) 2024-03-06T19:11:42.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:42.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:42.436166+0000) 2024-03-06T19:11:43.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:43.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:43.436413+0000) 2024-03-06T19:11:44.427 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:44.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:44.436716+0000) 2024-03-06T19:11:45.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:45.435+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:45.436973+0000) 2024-03-06T19:11:46.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:46.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:46.437235+0000) 2024-03-06T19:11:47.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:47.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:47.437487+0000) 2024-03-06T19:11:48.428 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:48.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:48.437747+0000) 2024-03-06T19:11:49.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:49.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:49.438007+0000) 2024-03-06T19:11:50.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:50.436+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:50.438255+0000) 2024-03-06T19:11:51.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:51.228 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:51.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:51.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:51.438530+0000) 2024-03-06T19:11:52.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:52.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:52.438770+0000) 2024-03-06T19:11:53.429 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:53.437+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:53.439001+0000) 2024-03-06T19:11:54.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:54.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:54.439264+0000) 2024-03-06T19:11:55.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:55.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:55.439525+0000) 2024-03-06T19:11:56.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:56.438+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:56.439757+0000) 2024-03-06T19:11:57.430 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:57.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:57.439994+0000) 2024-03-06T19:11:58.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:58.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:58.440231+0000) 2024-03-06T19:11:59.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:11:59.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:59.440485+0000) 2024-03-06T19:12:00.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:00.439+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:00.440751+0000) 2024-03-06T19:12:01.431 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:01.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:01.440986+0000) 2024-03-06T19:12:02.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:02.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:02.441248+0000) 2024-03-06T19:12:03.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:03.440+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:03.441493+0000) 2024-03-06T19:12:04.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:04.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:04.441738+0000) 2024-03-06T19:12:05.432 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:05.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:05.442003+0000) 2024-03-06T19:12:06.433 INFO:tasks.ceph.mds.b.smithi039.stderr:2024-03-06T19:12:06.441+0000 7fb8f9e00640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:12:06.442297+0000) 2024-03-06T19:12:07.000 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-06T19:12:07.001 DEBUG:teuthology.task.console_log:Killing console logger for smithi032 2024-03-06T19:12:07.002 DEBUG:teuthology.task.console_log:Killing console logger for smithi039 2024-03-06T19:12:07.002 DEBUG:teuthology.exit:Finished running handlers